Optimized Multi-Token Joint Decoding With Auxiliary Model for LLM Inference
Authors: Zongyue Qin
Ziniu Hu
Zifan He
Neha Prakriya
Jason Cong
Yizhou Sun
Venue: ICLR 2025 Poster (2025)
Keywords: LLM Inference
Speculative Decoding
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a novel decoding that improves output quality and downstream performance with 1.4 times faster and 1.5 times less energy cost compared to speculative decoding by considering joint probability of multiple tokens

Abstract: Large language models (LLMs) have achieved remarkable success across diverse tasks, yet their inference processes are hindered by substantial time and energy demands due to single-token generation at each decoding step. While previous methods such as speculative decoding mitigate these inefficiencies by producing multiple tokens per step, each token is still generated by its single-token distribution,
thereby enhancing speed without improving effectiveness. In contrast, our work simultaneously enhances inference speed and improves the output effectiveness. We consider multi-token joint decoding (MTJD), which generates multiple tokens from their joint distribution at each iteration, theoretically reducing perplexity and enhancing task performance. However, MTJD suffers from the high cost of sampling from the joint distribution of multiple tokens. Inspired by speculative decoding, we introduce multi-token assisted decoding (MTAD), a novel framework designed to accelerate MTJD. MTAD leverages a smaller auxiliary model to approximate the joint distribution of a larger model, incorporating a verification mechanism that not only ensures the accuracy of this approximation, but also improves the
decoding efficiency over conventional speculative decoding. Theoretically, we demonstrate that MTAD closely approximates exact MTJD with bounded error. Empirical evaluations using Llama-2 and OPT models ranging from 13B to 70B parameters across various tasks reveal that MTAD reduces perplexity by 21.2% and improves downstream performance compared to standard single-token sampling.
Furthermore, MTAD achieves a 1.42× speed-up and consumes 1.54× less energy than conventional speculative decoding methods. These results highlight MTAD’s ability to make multi-token joint decoding both effective and efficient, promoting more sustainable and high-performance deployment of LLMs.

Published as a conference paper at ICLR 2025
OPTIMIZED MULTI-TOKEN JOINT DECODING WITH AUXILIARY MODEL FOR LLM INFERENCE
Zongyue Qin"
Ziniu Hu
Zifan He
Neha Prakriya'
Jason Cong'
Yizhou Sun"
ABSTRACT
Large language models (LLMs) have achieved remarkable success across diverse tasks, but, due to single-token generation at each decoding step, their inference processes are hindered by substantial time and energy demands. While previous methods such as speculative decoding mitigate these inefficiencies by producing multiple tokens per step, each token is still generated by its single-token distribu- tion. Although this enhances the speed, it does not improve the output quality. In contrast, our work simultaneously boosts inference speed and improves the output effectiveness. We consider multi-token joint decoding (MTJD), which generates multiple tokens from their joint distribution at each iteration, theoretically reducing perplexity and raising task performance. However, MTJD suffers from the high cost of sampling from the joint distribution of multiple tokens. Inspired by spec- ulative decoding, we introduce multi-token assisted decoding (MTAD). a novel framework designed to accelerate MTJDD. MTAD) leverages smaller auxiliary model to approximate the joint distribution of a larger model, incorporating a verification mechanism that not only ensures the accuracy of this approximation, but also increases the decoding efficiency over conventional speculative decoding. Theoretically, we demonstrate that MTAD closely approximates exact MTJD with a bounded error. Empirical evaluations across various tasks reveal that MTAD improves downstream performance by 25% compared to standard single-token sampling. Furthermore, MTAD achieves a 1 1.42x speed-up and consumes 1.54x less energy than vanilla speculative decoding methods. These results highlight MTAD's s ability to make multi-token joint decoding both effective and efficient, promoting more productive and tegf-performance deployment of LLMs
INTRODUCTION
Large Language Models (LLMs) such as GPT-4 and Llama-2 Touvron et al. 2023) have demonstrated extraordinary capabilities across a wide range of tasks (Brown et al. 2020 Chowdhery et al. 2023 Thoppilan et al. 2022 Touvron et al. 2023). Despite their impressive performance, the deployment of LLMs is often constrained by substantial inference costs in terms of time and energy. This inefficiency primarily stems from the autoregressive nature of these models, where generating a sequence of K tokens requires K separate model calls. Fach call involves loading large weight matrices and intermediate results from GPU global memory to computing units, leading to repeated memory
accesses and limited hardware utilization (Samsi et al., 2023 Leviathan et al. 2023).
To tackle this challenge, researchers have delved into non-autoregressive decoding approaches. Early methods (Ghazvininejad et al.) 2019; Gu et al., 2017 Guo et al. (0000) aimed at reducing inference latency by concurrently generating multiple tokens. But these methods usually require task -eeeddenndd
techniques and information to match the performance of autoregressive decoding (Kim et al. 2023 Xiao et al. 2023). More recently, speculative decoding has emerged (Leviathan et al. 2023 Chen al.) 2023 Kim et al. 2023 Sun et al.) 2023). This method exploits the observation that most of the small model's prediction aligns well with that of a large model. It leverages a smaller auxiliary model to draft few future tokens autoregressively, which are subsequently validated in parallel by the
Depaarment of Computer Science, University of California, Los Angeles, USA. Correspondence to: giizonnyue@ccs www..aa..... edu California Institute of Technology, USA.
We release our code attttps:/ thub, http://ttt....ccoo////g/ggeeggig comp///oggguugggi/mmaaaaa

---

Published as a conference paper at ICLR 2025
larger model. As the smaller model operates significantly faster and parallel token verification incurs a similar time cost as generating a single token, speculative decoding attains an overall speed-up of 1-2x. Despite gains in speed, these methods still generate each token based on its single-token probability. Consequently, it does not enhance the effectiveness of the generated sequences.
In this work, first go beyond the conventional trade-off between efficiency and effectiveness and explore multi-token joint decoding (MTJD). Unlike traditional approaches, MTJD produces multiple tokens from their joint distribution each decoding step. Theoretically, show this joint generation can lead to lower perplexity and hence improved task performance. However, directly sampling from the joint distribution of multiple tokens poses significant computational challenges. rendering MTJD impractical.
Inspired by speculative decoding, we propose multi-token assisted decoding (MTAD), novel framework designed to approximate and accelerate MTJD. MTAD employs a smaller auxiliary model estimate the joint distribution of a larger model, significantly reducing computational demands. To ensure the accuracy of this approximation, MTAD incorporates verification mechanism that not only guarantees the accuracy the draft tokens but also enhances efficiency beyond conventional speculative decoding by maximizing the number of accepted tokens per iteration. We provide both theoretical and empirical analyses to demonstrate that MTAD boosts perplexity and downstream performance. Meanwhile, it significantly reduces the energy and time usage compared to existing decoding strategies.
Our contributions are as follows:
1. We introduce multi-token joint decoding (MTJD), multi-token, joint decoding approach that theoretically reduces perplexity by generating tokens from their joint distribution.
2. We develop multi-token assisted decoding (MTAD), an efficient approximation of MTJD with bounded error that leverages smaller model for distribution approximation.
3. We analyze the energy consumption of L.M inference. To our knowledge, we are the first to give quantified and empirical evidence that, despite the fact that MTAD and other speculative decoding algorithms increase the number of FLOPs needed during LLM inference, they actually use less energy with fewer accesses to the GPU global memory.
4. We conducted comprehensive evaluations across various tasks, demonstrating that MTAD improves downstream performance by 25% compared to standard single-token sampling, while, at the same time, realizing 1.42x speed-up and paring energy consumption by 1.54x compared to vanilla speculative decoding methods.
These advancements position MTAD as robust solution for making multi-token joint decoding both effective and efficient, thereby facilitating more sustainable and high- gegh-performncc deployment of large-scale language models.
PRELIMINARIES
In this section, we discuss preliminaries relevant to contextualizing our paper.
2.1 DECODINGS OF LLMS
Decoding and Perplexity. Let p denote the distribution defined by LLM model Mp. Given an input context input, a decoding algorithm generates a sequence of N tokens whose likelihood is designated as p( inpptt). (inputt. The likelihood of the sequence is directly linked to the perplexity of the sequence, which is the exponentiated average negative likelikel of all tokens. Based on autoregressive decomposition p( Iinput) = II p( imput the perplexity is defined as:
N
1 N
PPL( = exp
exxp
log logp(ss/0aa1))
(1)
II the paper, omit émput when there is ambiguity.
2

---

Published as a conference paper at ICLR 2025
Perplexity serves direct metric for assessing the effectiveness of a decoding algorithm. In practice, when a model is well-trained, lower perplexity often correlates with improved downstream performance. For example, beam sampling (explained below) aims to return output with lower perplexity and is empirically proven to have better downstream performance in general (Shi et al. 2024).
To further demonstrate the relationship between perplexity and downstream performance, we evaluate GPT-3.5-turbo on the spider (Yu et al. 2018) dataset. Employing temperature of the model gener- ated 10 outputs for each input. We measured the average perplexities and execution accuracies for the outputs with the highest, lowest, and median (the 5th lowest) perplexity. As seen in Table I lower perplexity correlates with improved downstream performance, even in one of today's 's largest models.
Table I: Relationship between perplexity and execution accuracy (EA, higher the better) for GPT-3.5-turbo
Now we introduce commonly used decoding approaches.
Multinomial Sampling. Multinomial sampling, also known as standarized sampling or single-token sampling, imput), where T is a warping operation applied to enhance the high probability region. Some common warping opera- tions include top-k warping. This limits the selection to the top k tokens, and top-p warping, where tokens are sampled from the smallest possible subset of the vocabulary whose cumulative probability mass exceeds specified threshold.
samples the next token T2 based on Top(( (11-11, 7
Output
Avg. PPL.
EA (%)
Highest PPL.
5-th Lowest PPL
1.40
Lowest PPL
1.07
The deterministic version of multinomial sampling special case with 1, also called greedy decoding.
Beam Sampling. Beam sampling is intended to decrease output perplexity over multinomial sampling. For each position t (1 t N), it maintains W > candidate sequences, which are also called beams. Assume we have already kept the W sequences IT- { {21a-1..... .i....,,1:.-11 11--1 at position t - 1. W sequences with length t are then sampled from T pbeam, where pleam I- V J0, 1] the beam sampling probability:
p itt Ie selppttt)
Pheam (I (3) (11:--1,
Σ SiSss p( s/(inputt
()
Notice that p( 24 linput) p( | imput) p( limput). In practice, beam sampling
(1)
stores the likelihood p(=f./_//ippe for each beam, and the computation complexity of Pbeam is O(W IV)). In deterministic beam sampling, the top W sequences with the highest likelihood Pheam (11a) will be kept.
2.2 VANILLA SPECULATIVE DECODING
Besides effectiveness, speculative decoding is proposed by (Leviathan et al. 2023 Chen et al. 2023) to accelerate the inference of LLMs. It utilizes small model to generate the next tokens and then uses the large model to verify the drafted tokens in parallel, which is summarized below:
1. Let inpul be the input context, the small model samples 7 draft tokens 31,-.. Byy with multinomial sampling based on [( =n, for I = 1,..., 7, where g = T o and q the small model's ''s output distribution.
2. The large model verifies the draft tokens in parallel by computing the conditional probability ( input) for 1,
3. Each draft token De is accepted with probability (1x,),,(x))//((1 before the first rejected token are kept as the decoding output. An additional token is sampled from a residual distribution as correction the first rejected token. Then the accepted tokens and the resampled token are appended to the context input as the input to the next iteration.
The draft tokens
4. Repeat step 1-3 until reaching the stopping criteria, e.g., reaching the length limit.
3

---

Published as a conference paper at ICLR 2025
Because the large model verifies y tokens in parallel with run, the time cost is smaller than calling y times. Moreover, the global memory access is also pared, which saves energy consumption, as we shall illustrate in Section Mcanwhile, although the small model still runs in autoregressive way, its inference speed is more efficient than the large model. As a result, speculative decoding maintains an identical sampling distribution while realizing speedup of 1-2x compared to multinomial sampling and using less energy.
3 METHODOLOGY
As discussed in Section 2 the goal of this work to design an algorithm that yields lower perplexity and better efficiency than multinomial sampling and vanilla speculative decoding. In this section, first introduce multi-token joint decoding (MTJD). This generates multiple tokens based on their joint likelihood. We prove it yield lower perplexity. Then present multi-token assisted decoding (MTAD), which approximates and accelerates MTJD by exploiting an auxiliary model.
3.1 MULTI-TOKEN JOINT DECODING
We first explain a new decoding algorithm to improve multinomial sampling in terms of perplexity. Definition 3.1. Multi-Token Joint Decoding, Let Mp be the large target model with distribution Different from single-token multinomial sampling, multi-token joint decoding (MTJD) produces the next % tokens at step i based on their joint conditional probability p(1++1:44 [=1:)), where % is an integer less than 1 and E Te i.e., the total tokens generated in the previous steps.
Multinomial sampling is a special case of MTJD where 7/- 1, Vi. When 71 N, MTJD generates the sequence directly based on their joint likelihood. So intuitively, output perplexity should improve % increases. Besides, gener- ating ?/ tokens simultaneously allows MTJD to
0.35
perplexe Perplexity d- Un
OPT
0.30
Llama-2 2
0.025
OPT
0.20
Llama
0.15
0.10
consider their interactions. In contrast, multino- mial sampling selects each token without consid- ering any future tokens. So MTJD is less prone to choosing local optima.
3
4
Figure 1: Perplexity and Rouge-L score of the output when "/ = for MTJD with OPT- 125M and Llama-2-68M fine-tuned ChatGPT. Prompts (Rashad 2023) dataset.
Theorem 3.2 demonstrates the limit of perplex- ity of MTJD when N approaches infinity. The proofs included in the Appendix
Theorem 3.2. Assume at the i-th (i 1,...,
iteration, MTJD generates "/ tokens. Let T denote the total number of tokens generated at the first iterations. Let FA.N denote the generated tokens. When N-
(PL,(( exp (-1E,L((7D)
where 7 is the expected number of y 5- T represents how sample the next % tokens from P (e.g., in deterministic sampling, ń = arg max op always returns the tokens with the highest joint likelihood), and T.p(2..) )) the expected log-likelihood of the ry tokens sampled from p:
Lp(1,p)
E
Σ
Here X is the space of all possible inputs.
Corollary 3.3. Based on Theorem 3.2 we can show that when N - DR, greedy MTJD (i.e., top-1 MTJD sampling) has lower perplexity than greedy decoding (top-1 single-token sampling).
Empirical evidence supports our claim. We tne-tune both a Llama and an OPT model on the ChatGPT. Prompts dataset and evaluate the output perplexity and Rouge-L. scores with example outputs. Figure
4

---

Published as a conference paper at ICLR 2025
1.
imput
small model
1, like, writing, codes
large model
1. like,
1. like, writing
1, like, writing, python
1, like, writing, codes X prefixes
draft tokens
decisions
accepted tokens
additional token
Figure 2: An example of MTAD's verification process. MTAD accepts the longest draft sub sub-sequence that passes verification based on joint likelihood.
depicts the output perplexity and Rouge-L scores of MTJD with 7/2 set to constant K, where K
I.....5. 5. Notice that setting K = is equivalent to multinomial sampling. We use beam sampling to approximate the arg max sampling from the joint distribution p(x++1=+* lx 1: imput). We can see that the perplexity keeps dropping when K increases. It confirms our claim that increasing 7/
will increase the output perplexity. Moreover, the Rouge-L. score also improves with K, supporting our claim that better perplexity reflects enhanced performance in downstream tasks.
3.2 MULTI-TOKEN ASSISTED DECODING
Unfortunately, the computation cost of MTJD is infcasible in practice. since the time and space complexity to compute the joint distribution % tokens [V]% Inspired by speculative decoding and the fact that "even when a small model is an order of magnitude smaller than large model, only a small fraction of the small model's prediction deviate from those of the large model" (Leviathan) al. 2023 Kim et al. 2023), we propose multi-token assisted decoding (MTAD), which exploits small auxiliary model M, to accelerate MTJD approximately. The core idea is to (1) use the joint distribution q(3++1:44% (r::)) output by Mg to approximate p(I
F and produce draft tokens from (4 (1.)). then (2) utilize the large model to validate draft tokens in parallel and accept the longest draft prefix sub-sequence that passes verification, and (3) sample an additional token from the distribution of the large model without extra overhead to ensure at least one token is generated at each iteration. However, it is still infeasible to directly generate draft tokens from the joint distribution g( [.1:)). ). So we propose to further approximate this process with beam sampling, which is an effective and efficient algorithm to generate sequences with high likelihood. In this way, MTAD decreases the number of runs of the large model to generate N tokens, thus accelerating the inference in the same way as vanilla speculative decoding does. Algorithm i in the Appendix illustrates the pseudocode of MTAD algorithm.
Draft Tokens Verification Figure 2 displays displays the verification process of MTAD. Let 1 Fr
be the draft tokens generated by beam sampling with the auxiliary model. Since beam sampling is widely recognized algorithm to produce sequences with high overall likelihood (Leblond al. 2021). it is reasonable to assume q ((1411444 [11.) is large. Also, since beam sampling works in an autoregressive way, we can also infer that Vj E {I...., 7). q(I++1:44) [+1;) large. To approximate MTJD, for each step i, MTAD needs to ensure the accepted tokens Z141447 (0 < y's S ?) also have high joint likelihood with the large model Mp. So MTAD first computes the joint likelihood p(=++ jatt (aa)) for 1, I. Then for each prefix sub-sequence 1e+1 it passes verification if and only if min(1, p(zenetilm > т, where T E [0,1) is pre-defined g(Ren144)1211))
(1044a44//110)) > which means g( > we have q(Pe41=4)) 1. Therefore, our acceptance policy guarantees that when ((1++1045 [1a) > (( the relative eтror is bounded. And if g(#c4144) < p(2++124+ [.1:), means the sub-sequence has higher likelihood in the large model, then it is reasonable to accept it. After verifying all the sub-sequences, MTAD accepts the longest prefix
threshold. Notice that if min(1, ((
p(re41445la ()))
ueq-se that passes verification.
The verification step of MTAD ensures that the accepted tokens have high joint likelihood with the large model. We have shown that selecting multiple tokens based on their joint likelihood leads
It is also valid to approximate with q. Without loss of generality, we consider non- non- warped distribution in the illustration of MTAD.
5

---

Published as a conference paper at ICLR 2025
to better output perplexity. Thus, MTAD is more effective than multinomial sampling and vanilla speculative decoding. Furthermore, since MTAD accepts the longest draft sub-sequence with high likelihood, it can tolerate low-quality tokens long the joint likelihood is high. So at each iteration, MTAD can admit more draft tokens than vanilla speculative decoding, which results in better efficiency.
Next, we theoretically analyze the approximation error of MTAD. Lemma 3.4 shows the upper bound of MTAD's perplexity. Theorem 3.5 reveals the upper bound of the ratio between the perplexity of approximate MTAD and exact MTJD. The proofs are given in Appendix A
Let us assume that when the small auxiliary model generates draft tokens with beam sampling, the beam width is large enough such that the returned log-likelihood is close to the maximum log-likelihood, i.e.,
Lemma 3.4.
EFTT, EX log ((F, 1+1:1:-1/11 .)2
(5)
(1 (1-6) (1-e)EELT,,,C EX
max 1111+
where is error term and E S 0 because logq < O.
Furthermore, let H (p. q) the single-token cross entropy between p and q, ie., H(p =
-EZLGex Exerp(e p(zc+1(x1a) log ((2++1
With the two assumptions above, when N - have
PPLg(D(TX) S exp( 1 1- En Ln 1, arg max oq) + H(.
(6)
where
Lg(7,, arg max oq) Er max log ((cc414t7/
(7)
Theorem 3.5. Let F1.Nx be the tokens generated by approximate MTAD, and virr be the tokens generated by deterministic exact MTJD. Assume Væia E X. || p(z((()) log ((r()1))) S U, where U is a constant. We have
lim (PL,(w1ax)) - exp (1 ((- + - - ++
(8)
N PPLp(rlny)
where H(p) is the entropy of and e < 0 is the error term of bean sampling (see Lemma 3.4.
Theorem 3.5
suggests the approximation error of MTAD is bounded by a factor related to the verification threshold average number of accepted tokens the difference between the large and small models (measured by U), the error beam sampling e, and the entropy of the large model itself. In addition, the following theorem analyzes 5. The proof is illustrated in Appendix A
Theorem 3.6. Following the assumption in Theorem 3.5 we have 7
log ml
With Theorem 3.6 we observe that when q p. have U + 0 and + Meanwhile, when the beam width for the auxiliary model is large enough, e - O, and the ratio bound in Theorem 3.5 converges to 1, This implies that MTAD converges to MTJD under these limiting conditions.
Similar to Spectr (Sun et al., 2023) and SpecInfer (Miao et al. 2023), it is possible to enhance the number of accepted tokens in MTAD by allowing the draft model produce multiple draft sequences and by applying tree- based attention (Miao et al.) (003)) for simultaneous verification. However, our preliminary experiment results suggest that since MTAD already selects the longest accepted prefix sub-sequence, the advantage of generating multiple draft tokens is less significant. Moreover, this approach increases the memory cost during inference and may affect the error bounds derived above. Therefore, we leave more detailed exploration of this extension as future work.
4 ENERGY EFFICIENCY ANALYSIS
Previous studies (Leviathan et al. 2023 Chen et al. 2023: Kim et al. 2023 Sun et al. 2023) only
focus the speed of speculative decoding. However, equally important consideration is energy

---

Published as a conference paper at ICLR 2025
Table 2: The effect batch size to inference speed and energy consumption. The number of inputs is the product of the number of LLM runs and input batch size.
Ratch Size
Energy (J)
Energy/ran (3)
Energy/Inuut ())
Time (6)
Time/run (s)
Timefinutt (s)
42,450
1,129
0.376
49.621
16.5
8.26
1.191
0.397
0.198
53,325
17.7
4.43
1,178
0.392
0.098
59,210
19.7
2.46
1,211
0.403
0.050
16
74,058
24.7
1.54
1,255
0.418
0.026
consumption. To our knowledge, there is no existing work evaluating the impact of speculative decoding on inference energy consumption. Although MTAD and speculative decoding raise the number of FLOPs due to the involvement of a small auxiliary model and the rollback operation, they concurrently reduce the inference time and memory operations, which are key factors of GPU (or TPU) energy consumption (Allen & Gel 2016 Chen et al., (001T) Consequently, it poses open question regarding whether speculative decoding increases or decreases overall energy consumption.
To understand the net effect of speculative decoding, we decompose the total energy consumption
into two parts following (Allen & Ge 2016):
Excaa= PW stop StopTTiton Stop + PW mem T meem
where PW slop: plotop PWmem denote the power (energy/second) of umit FLOP and memory operation, respectively, and Tiem are the total time spent on these operations. When input batch size increases, PW flop rises until it reaches the power of maximum FLOPs, designated as PW Hlop PW mem is irrelevant to the input batch size because it only depends on the memory hardware.
To determine the relative magnitude relationship between PW Htop and PW mems we first point out the fact that GPU memory operations in LLM inference are dominated by accessing off-chip global memory. This consumes about 100x of energy compared to accessing on-chip shared memory (Jouppi et al. 2021). Because each multiproces- sor on GPU usually has 64KB of on-chip memory shared by multiple threads, but to store a single layer of LLM, say 15-11b (Raffel et al. 2020), requires about 1GB of memory. Moreover, Allen and Ge showed that doing sequential read from off-chip memory consumes 20-30% more power than running maximum FLOPSs (Allen & Ge. 2016). So we have PW mem > PW; flop 2 PW flop- No-
Table 3: Speed and energy cost of multi- nomial sampling (ms) and speculative decoding (spec).
OPT
LLAMA
MS
SPEC
MS
SPEC
TOKENS/S
23.8
35.6
22.0
31.6
J/TOKEN
11.3
5.74
11.2
6.97
PW
PWrlop.
Lice that PW",,p flop = PW flop only if the batch size reaches the maximum parallelization capacity of GPUs. During multinomial sampling and speculative decoding, the batch size is generally small (Leviathan et al. 2023). So most of the computing power is not utilized (Leviathan et al. 2023 which means PW mem >> slop-
In addition, previous studies have revealed that during LLM inference Tmem >> Tytop (Leviathan al. (202). Therefore, the energy induced by memory operations, i.e., PW mem T mem dominates Erotal. Since speculative decoding lowers Tee by reducing the number of runs of the large model, should cut the inference energy consumption to similar extent as it reduces time consumption.
To validate our hypothesis, we conducted an experiment to evaluate how batch size influences energy consumption during inference. We ran OРT-13b models on Nvidia L40 GPUs with 48GB memory. Fixing the total number of runs of the large model while varying the input batch size b e (1,2,4, 4, 8, 16) for each run, we measured time and energy cost. The details of energy measurement are illustrated in the Appendix [D] Table shows the results. As batch size doubles, although the number of FLOPs doubles, the energy consumption per run goes up slightly. This observation demonstrates that consumption of rumning multinomial sampling with the large model and speculative decoding with OPT (125M, 13B) and Llama-2 (68M, 13B) models. The results, seen in Table 3 indicate that speculative decoding lowers the energy consumption and the time cost. This observation corroborates our claim to the energy efficiency of speculative decoding.
dominates Erotat. Moreover, we measured the speed and energy
7

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
Datasets and Models. In the main paper, we report results with three public datasets for evaluation: (I) Spider (Yu al. 2018), MTBench (Zheng al.] 2023). and HumanEval (Chen al.] (021). We use Llama-3-8B and -lamm-338B--img -Instruuccttt (Dubey et al. 2024) as target models, and Llama-3-I1 and Llama-3-IB-Instrucl as their draft models, respectively. We provide additional experiments with other datasets and model families in Appendix C
Baselines. We compare our method with six speculative decoding methods, including four loss- less decoding methods: vanilla speculative decoding (SpD) (Lee et al. 20188; Chen et al. (023). Spectr (Sun et al., 2023), SpecInfer (Miao al. 2023). .CSSS (Yang et al., 2024), and two lossy speculative decoding methods: BiLD (Kim et al. 2023) and typical decoding (Cai et al.) 2024). All the baselines and our method utilize the same pair of draft and target models without any fine-tuning. For each method, we let it generate at most 128 tokens for each input and run it for 1, 000 seconds. All the methods are stochastic with top-k and top-p sampling with the temperature = 1. The details of the hyper-parameters
parameters (e.g., and p) and machine configurations of the experiments can listed in the Appendix D and
Appendix reports additional experiments and ablation studies.
5.1 PERFORMANCE OF MULTI-TOKED JOINT DECODING
While most speculative decoding approaches focus on inference speed up, we want to design ap- proaches that can also improve inference quality. We propose multi-token joint decoding (MTJD, Section 3.1) to accomplish the goal, due to its capability to achieve lower perplexity and higher likelihood than single-token multinomial sampling. To validate that MTJD indeed betters output qual- ity, we test MTJD (k=4) and standard multinomial sampling using the Llama-3 series models. We follow the same way introduced Section 3.. to implement MTJD. For this process, the higher the scores, the better the downstream performance. Under all settings, MTJD realizes the highest scores and lower perplexity. These results show a clear advantage for MTJD in terms of output quality.
Spider, MTBench, and HumanEval
Table 4: Performance comparison of single-token sampling and multi-token joint sampling. We use Llama-3. 1-BB and 1-a--..-----Inttreee as target models, and Llama-3.2-1 and Llama-3.2- Instruct the draft models.
Llama-3 IB)
(8B,1B)
Spider
MTBench
HumanEval
Spider
MTBench
HumanEval
Single-token multinomial sampling
Score
22.0
3.40
15.9
36.0
4.11
28.0
PPI.
2.58
2.40
2.09
2.23
1.91
1.85
Multi-token joint sampling
Score
52.5
3.77
36.6
60.5
4.40
49.4
PPL
1.16
1.32
1.26
1.18
1.27
1.15
5.2 PERFORMANCE OF MULTI-TOKEN ASSISTED DECODING
Next, we evaluate the efficiency and effectiveness of MTAD, an approximate algorithm that accelerates MTJD while preserving its downstream performance advantages. Table |5 presents the decoding speed, energy consumption, and downstream performance of various decoding algorithms across different datasets.
Efficiency Analysis. We first observe that MTAD is the most efficient among all baselines in terms of both energy and time. On average, MTAD 17.2% faster than the most efficient lossless baseline, MCSS, while consuming 6.1% % less energy. Compared to lossy decoding algorithms, it attains 27.2% (15.1%) higher speed and 17.9% % (14.1 %) lower energy consumption than BiLD (typical decoding). respectively. Interestingly, despite utilizing only single draft sequence, MTAD
8

---

Published as a conference paper at ICLR 2025
outperforms baselines that employ multiple draft sequences, such Spectr, MCSS, and SpecInfer. In these methods, verification terminates immediately upon rejecting token. In contrast, MTAD continues verification even after rejection, searching for future tokens that may still pass. This mechanism results a greater acceptance length per iteration than the baselines.
Downstream Performance Comparison. Next, compare the downstream performance of differ- ent decoding algorithms. Notice that while lossless decoding algorithms theoretically sample from the target distribution, they exhibit slight variations in downstream performance. This discrepancy arises because, despite preserving the original distribution, differences the sampling process prevent them from generating identical sequences observe that lossy decoding algorithms can reach higher downstream performance at the expense of efficiency. This suggests that all lossy decoding methods can trade off efficiency for performance by adjusting verification strictness. Most notably, MTAD consistently achieves the highest downstream performance. On average, surpasses lossless decoding algorithms by 25.0% % BiLD by 11.8%. and typical decoding by 9.9 %. These results confirm our claim that MTAD offers superior effectiveness compared to conventional decoding methods that rely solely on single-token distributions.
when the random seed is fixed. Furthermore,
Table 5: Comparison of different speculative decoding methods across various models and metrics. Bold indicates best values, underline indicates second-best.
Lossy Decoding
Lossless Decoding
Ours
BiLD
Typical
SpD
Spectr
SpecInfer
MCSS
MTAD
HvmanEval
1lama-3-Instrune
tokens/s f
17.4
21.7
22.2
23.8
22.8
23.7
24.8
Jhoken +
10.0
8.1
7.8
7.8
7.9
7.8
7.6
pass@1 1
37.8
35.9
32.9
32.9
31.0
32.0
38.4
Llama-3
tokens/s
19.6
22.5
22.2
24.4
22.5
23.8
25.6
Jhoken
9.7
8.9
8.9
8.9
8.1
7.9
7.6
pass@1
19.5
20.0
15.9
16.0
17.7
17.0
22.0
Spider
-lama-3-Instreee
tokens/s
20.1
22.3
19.6
22.4
21.1
21.7
23.5
J/toeen +
10.2
9.5
10.5
9.6
10,2
10.0
9.2
Acc
35.0
36.0
35.5
37.0
35.0
44.0
Llama-3
tokens/s
23.3
32.3
31.1
32.1
32.6
32.7
33.3
Jtoken
8.2
7.9
7.5
7.1
8.1
8.0
7.8
Acc
30.5
29.5
21.5
23.0
21.5
24.0
35.0
MT-Bench
Llama-3- I-Intruutt
tokens/s
25.9
23.4
26.0
26.2
26.3
26.8
29.8
J/token +
10.8
12.2
10.0
9.9
10.0
9.9
9.2
score
4.15
4.26
4.10
4.11
4.01
4.02
4.40
Llama-3
tokens/s
24.5
22.3
24.1
24.5
24.5
25.7
28.2
Jhoken 11.5
12.4
11.0
11.6
11.7
11.1
10.0
score
3.41
3.24
3.39
3.41
3.35
3.36
3.75

---

Published as a conference paper at ICLR 2025
6 RELATED WORK
EFFICIENT DECODING INFERENCE. There are extensive studies on improving large model inference efficiency. Well-known methods include model quantization (Frantar et al. [022] Lin et al. 2023), model pruning (Gale et al. 2019; Sanh et al. 2020), and model distillation (Hinton et al.
2015). Despite achieving significant speed-ups, a common drawback of these methods is that they have to sacrifice the model's ' effectiveness.
Not-autooeerressive decoding more closely resembles our work. It is first proposed by (Gu et al. 2017) to generate multiple tokens in parallel. That is, the model simultaneously predicts p( (+1:) (k 1, Subsequent studies further improved the performance of parallel decoding by
incorporating additional information (Wang et al., Sun et al. [2019 Li et al. 2019) or employing
additional iterations refine predictions (Ghazvininejad et al. 2019 Lee al. 2018 Guo al. 2020). However, these works require continuous training of the model and generally either compromise the
model effectiveness or require task desk-dependent techniques to attain a comparable performance (Kim)
et al. 2023).
SPECULATIVE DECODING. Speculative decoding was recently proposed in (Leviathan et al. 2023 Chen et al. 2023) as a way to accelerate LLM inference. Spectr (Sun et al., 2023) enhances speculative decoding by letting the small model generate multiple i.i.d. draft sequences. While speculative decoding and Spectr use the large model to verify all the tokens drafted by the small model, BiLD (Kim et al. 2023) only calls the large model when the probability output by the small model is below a pre-defined threshold Ti. The large model rejects token if its negative log- riglliellhoodd is larger than threshold 72- SpecInfer (Miao et al. 2023) utilizes one or multiple small models to increase the average acceptance length for each iteration. MCSS (Yang) eet al.] (024) further strengthens SpecInfer via sampling without replacement. All these methods can be perceived as exact or approximate versions of sampling tokens from the conditional distribution (xx)). Therefore, their output perplexity is bounded by greedy decoding.
generate draft token
An orthogonal direction to boost speculative decoding is improve the effectiveness of the small draft model. It is obvious that if more draft tokens are accepted, the overall inference speed will increase. BiLD (Kim et al. 2023) employs model prediction alignment technique to better train the small model. Liu et al. (Liu et al. (2023) propose online speculative decoding to continually update the draft model based on observed input data. Instead, Rest (He et al. 2023) uses retrieval model to produce draft tokens. An alternative way is to train additional heads in the large model to predict future tokens. Representative works include EAGLE (Li al. (024) and MEDUSA (Cai et al. 2024). Importantly, these works are orthogonal to speculative decoding techniques, including our proposed method. This orthogonality means that the improvements offered by more accurate draft tokens could be combined with method for better effectiveness.
7 CONCLUSION
We introduce multi-token assisted decoding, a process that enhances output quality while improving time and energy efficiency A distinctive aspect of our work the exploration of speculative decoding's impact on inference energy consumption, an often neglected area in existing studies. This research contributes not only a novel decoding approach but also valuable insights for optimizing LLM deployment in real-world applications where considerations of both quality and efficiency are crucial.
8 ACKNOWLEGEMENT
This work was partially supported by NSF grants 2211557, 1937599, 21 2119643, 2303037, NSF 2312501, SRC JUMP 2.0 PRISM Center, NASA, Okawa Foundation, Amazon Research, Snapchat, and the CDSC industrial partners (htps::/ccsc..cla..dduaanaa thank Marci Baun for editing the paper.
The authors would also like to
REFERENCES
Tyler Allen and Rong Ge. Characterizing power and performance of GPU memory access. In 2016 4th International Workshop Energy Efficient Supercomputing (E2SC), pp. 46-53. IEEE, 2016.
10

---

Published as a conference paper at ICLR 2025
Feifei Bear.
LLMSSpcullaiveSampiing
LLMeeeculattivessppppp... 2024. Accessed: 2024-05-19.
Ning Bian, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, and Ben He. ChatAlpaca: A Multi- Turn Dialogue Corpus based on Alpaca Instructions.
https:/ /ttp://gttttb..... www.gttt..ooc..eeeeeeeoeoeeeeeee cttps/uwbccccccccccppcc//
ChatAlpaca 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 107744 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Yean-Baptist Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013 2023.
Jianmin Chen, Bin Li, Ying Zhang, Lu Peng. and Jih-kwon Peir. Tree structured analysis on GPU power study. In 2011 IEEE 29th International Conference on Computer Design (ICCD), pp. 57-64. IEEE, 2011.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Car,, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alee Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. 2021.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama herd of models. arXiv preprint 2rXiv:247..21783 2024.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 1XXi::2210.172... 2022.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: /902.00944.0 2019.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-Predict: Parallel decoding of conditional masked language models. arXiv preprint 1904:v9904009:20 2019..
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. Non-autoregresssve neural machine translation. arXiv preprint arXiv:1711.0228.1 2017.
Junliang Guo, Linli Xu, and Enhong Chen. Jointly masked sequence-to-sequence model for non- autoregressive neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 376-385, 2020.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. REST: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08222, 2023.

---

Published as a conference paper at ICLR 2025
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025.: 2015.
Norman P Jouupii Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas B Jablin, George Kurian, James Laudon, Sheng Li, Peter Ma, Xiaoyu Ma, et al. Ten lessons from three generations shaped Google's TPUv4i: Industrial product. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pp. 1-14. IEEE, 2021.
Schoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Rémi Leblond, Jean- Saan-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Jean-Baptinn Lespiau, loannis Antonoglou, Karen Simonyan, and Oriol Vinyals. Machine translation decoding beyond beam search. arXiv preprint arXiv:2104.05336 2021.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic anntoooggseeee neural sequence modeling by iterative refinement. arXiv preprint arXiv:1802.06901,, 2018.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from Transformers via Speculative Decoding. In International Conference on Machine Learning, pp. 19274-19286, PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858, 2024.
Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tin Liu. Hint-based training for non-autoregressive machine translation. arXiv preprint arXiv: 2019.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. AWQ: Activation-aware weight quantization for LLM compression and acceleration. arXiv preprint arXiv:2306.00975 2023.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Ion Stoica, Zhijie Deng, Alvin Cheung, and Hao Zhang. Online Speculative Decoding. arXiv preprint arXiv:2310.07177, 2023.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong. Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative LLM serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.0978, 1(2):, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1(1):5455-55555 -5551511,-55,,,,01 2020.
Mohamed Rashad. ChatGPT-prompts, 2023. URL http:::/hhugiingfcccco//// Pot.ameeRR.ssaad/hhatGtP---..
Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. From words to watts: Benchmark- ing the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1-9. IEEE, 2023.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems, 33:20378-2000 20388, 2020.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456-17473 2022.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 18.186 1111655//1/P1--I//1 1099. (tthts 7/ w/www.c ac 1 web. www. http://aattooolloooooll////
12

---

Published as a conference paper at ICLR 2025
Chufan Shi, Haoran Yang. Deng Cai, Zhisong Zhang. Yifan Wang, Yujiu Yang. and Wai Lam. A thorough examination of decoding methods in the era of LLMs. arXiv preprint arXiv:2402.06922, 2024.
Zhiqing Sun, Zhuohan Li, Haoqing Wang. Di He, Zi Lin, and Zhihong Deng. Fast structured decoding for sequence models. Advances in Neural Information Processing Systems, 32, 2019.
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. arXiv preprint arXiv:2310.15141, 2023.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201. 082299, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2XXi::330.13977 2023.
Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie- Yan Liu. Non-autoregressive machine translation with auxiliary regularization. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5377-5384, 2019.
Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie- Liu. A survey on non-autoregressive generation for neural machine translation and beyond. IEEE Transactions Patfern Analysis and Machine Intelligence, 2023.
Sen Yang, Shujian Huang. Xinyu Dai, and Jiajun Chen. hulti-candidate speculative decoding. arXiv preprint 20Xi:22001.06766, 2024.
Zeyu Yang, Karel Adamek, and Wesley Armour. Part-time power measurements: NVIDIA-SMI's lack of attention. arXiv preprint arXiv:2312.02741, 2023.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. arXiv preprint arXiv:/809.08887, 2018.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0n 000600 2022.
Lianmin Zheng, Wii-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging auddeee with MT-bench and 2023.
chatbot arena. Advances in Neural Information Processing Systems,
13

---

Published as a conference paper at ICLR 2025
A
PROOF
A.1
PROOF OF THEOREM 3.2
Proof.
PPL(M)T)) exp
logp(sg((1s-1)
TN
(10)
N
eex
log (T,, logp((rr,
Tx
N
When N ExLEX Ey E Exetiaty P(7)p(ze-att) T1a) log p +++1 114) = (,,p((7)))
and Di=l p.
A.2 PROOF OF COROLLARY 3.3
Proof. For deterministic multi token sampling, Pmoiti = arg max op, so we have
Lp(', Pmultt) Ex 1og:((2++10+xx1::s))
(11)
#441447
Notice that deterministic greedy sampling can be Paingle (21+114)11))))
special case of MJGD where 1 if and only if I+tz arg max, p(z[2144i-1) for = 1....,?. Let Zi41+ty be the tokens generated by deterministic MJGD and let 1/41:47 be the to- kens generated by deterministic greedy decoding. For any fixed and $1 we have log pog (11+). Therefore, Poingie) S Lp(') Pmului). Then with Theorem 3.2, know that the perplexity of greedy decoding will be higher.
logp(s"++1aa)122)) (114) I max log p(1e+1tt7)111))
A.3 PROOF OF LEMMA 3.4
We first prove the following Lemma.
Lemma A.1. Let PLLp and PLL denote the perplexity of tokens under distribution p and q. When
N + 0o, we have
PPLp(Ears) -T
(12)
(PL,(
where t is the verification threshold.
Proof. In the i-th iteration, the first % tokens are the accepted draft tokens and the last token is sampled from Based on our verification criteria, we know that for the accepted draft tokens, we have
2T. T.
(13)
(
So.
p(ZITx)
N p (1rr:1)
2 (x11xx
(14)
(0r, 21.
Notice that
$ = exp
i plar, (.rr,-)) ñi
N log
p(ar, (mir:-1)
(15)
g(Er, (1.r;-1)
g(0r, (1:1.-1)
When N + 00, since the last token at each iteration is sampled from p. we have
1
N
p ((1.) - E. log log
"p(rr p(rr,leur.-1)) KL(p. 20
q(ærr| )
(16)
N
14

---

Published as a conference paper at ICLR 2025
So
par, pr:r;-
21
(17)
(ar, (1:1.-1)
Therefore,
p(x1.1)) g(
2T
(18)
Thus,
PPL (PLG(TITT)
p(ZLIN) g(>l)w)))
NN
S Fx
(19)
Now, we prove Lemma 3.4
Proof.
1
llo (2gg((1111
IN Σ
log q /on
(20)
When N + oo, since the first 74 1 tokens are sampled with beam decoding, we have
N
log -nn
+ E, Exixx log 10
2 (1 E, EX 2r +1-1,-1
max
q(TT, - 1))
= (1 E)EE Ly - 1, arg max oq)
(21)
Since the last token at each iteration is sampled from p. we have
N
log (ar, (m)r,-1)
Ex Rp log (( -H
(22)
So
1
1 E
EX max g(12+1:47 H(p q)
log PPL PPLg(TIIW)
E
(23)
7
241144
7
(pp.g)
1 1- Ey Ln(7 1, arg arargaas max og)
PPLg(EINN) S sexp
(24)
7
7
A.4
PROOF OF THEOREM 3.5
Proof.
We have
lim (PLT()
PPL, (LemmoA.1) A.1
<T lim
PPLp(ri.TT)
lim PPLo(ZITx)
-
(Theorem3.2)
JE, Lp(7 arg max mp)
exp
H(p.g)
1 1- Ly - 1, arg max max oq)
- 1 exp
(Lemmod.4)
exp
- - E,,Lp( arg max op)
H(p.9)
1- 7
exp
E, L6 argmaxoo ( + E, Lp arg max op)
7
7 (25)
15

---

Published as a conference paper at ICLR 2025
(p arg max because for any 214. max logp(#c4144) (1.)) 2 (log Le + log + (2) max 11 log 11
Notice
that
Lp(7 + 1, aag op) for any
7.
This
max
So
PPIp(EIIN)
lim N-000
((p.q)
1 1- (TE,, Iep arg arg max op) Leg arg max oq))
<r --- exp
+ 7
L, arg max op)
(26)
Since < 0, and (7: arg max op) is the maximum liglliie,,oood, which is larger than the expected log-likelihood (i.c., negative entropy), we have
E Lp(y, arg oax
By E.EEEE EX max log pog s141it9
(27)
Ey ExLEE
Σ
p(Zs4taen/ log ton
(HH(p)
In addition
Ey arg max op) E,, Lu (7, arg max og) EL, arg max op) Lo (7, arg mag og))
=E, EEIEE max log [21:)) Retlitty
ExTE max log ((
- max log 10g max log (((44147))
#etlitt
(28)
< E, EmaE max (log (logp(z: - log 9 (2
Σ logg(ze - log 1 E, E.L.E max
$ E. Ex ExUy (because [ogg(r|zz)) - log (() $ U)
UR
And H(),)) = H(p) + KL(p)lu)
KL(p]lg) = Exiex Σ
(29)
E EmLex E((a(o=)U SU
So H(p.g) < H(p) + +U. Therefore,
lim (PL,(T1Tx) 5T exp N (PL,(
(1 (I-))) + +(1 7)0
(30)
7
A.5 PROOF OF THEOREM 3.6
Proof. Recall that we accept 1:41 if and only if (ogg((x41144)) [114) log 1(1 2 log Since log p(x]<1a) - log (1))oo < U, we have
log log (1a)) - log g( ((
-jU
(31)
Therefore #4414t is always accepted if [logrl So 7 2 log
16

---

Published as a conference paper at ICLR 2025
B PSEUDOCODE OF MTAD
See Algorithm I
Algorithm 1 One Iteration of MTAD Algorithm
1: Input: draft model Mg. target model Mp. input, threshold
2:
# Sample draft sequences from M, with beam sample.
3: æ, q t- beamSample( input)
# æ; the i-th draft token. qi q( 1:a ialinput)
4: P Mplinpul, X)
# P R(+1)x VI, Pa. p(x j input)
5:
# Select the longest accepted draft sequence
6: p 1, [
7: for i = =11= to do
8:
j x; p + p* P..j. q qi
9:
if min(1, ") then
11:
# longest accepted prefix so far
12: end if
13: end for
14:
# Sample the next token using results of M,
15: p' < P
\~ p'
17: return Eiy..., Eng
Table 6: Dataset Statistics
Dataset
Task
Avg. Input Len
BaaGGPT-Prommp
Instruction
25.2
ChatAlpaca
Chat
277.7
CNNDM
Summarization
3,967.1
Spider
Text-to-SQL
347.68
MT -TT-Becch
Various
N/A²
HumanEval
Coding
67
C ADDITIONAL EXPERIMENTS
C.1 ADDITIONAL DATASETS AND MODEL FAMILY
Here we report the additional experiment results with three more datasets: (1) ChatGPT Prompt (Rashad) 2023). (2) ChatAlpaca (Bian et al., 2023), (3) CNN Dailymail (See et al. 2017). We use two public L.M families our experiments: OPT (Zhang et al. 2022) and Ilama-2 (Touvron) al. (002). We set the large model to be OPT-13B and Llama-2- 13B as they the largest models that can run on a single 40GB GPU, and utilize Llama-68M (Miao et al. 2023) and OPT-125M as the small models.
Table shows the full evaluation results, and Table D displays the downstream performance. Table g depicts the average number of generated token per iteration for different algorithms. The experi- ment results demonstrate that MTAD achieves better efficiency, better perplexity, as well as better downstream performance.
17

---

Published as a conference paper at ICLR 2025
Table Inference efficiency and output perplexity different methods on ChatGPT-Prompl (CP), ChatAlpaca (CA), CNNDailyMail (CD), Spider (SP), and MTT-Bench (MT) datasets. Bold numbers mark the best result, underlined numbers mark the second best.
SpD
BILD
Spectr
Spoclnter
MTAD
speed (token/s)
36.84:0333
34.420.877
45 45.1211.22
29.7+ 0.40
63.0=0.20
Llama-2
energy (Mokenn)
6.62.10.91
5.17.1088
perplexity
3.64::0008
CP
speed (token/s)
33.84:2.47
31.5
38.04-2.20
32.8:: 0.58
05.8=0.30
OPT
energy (Phoken)
7,48:00077
8.7520.13
6.08-10111
1.49
3.61
perplexity 5.47:10.11
4.511:0.09
5.27:10.09
5.12::0.01
speed (token/s)
31,6+:0.55
28.8+0.20
27.7=0029
26.5=0049
44.1=0.33
Llama-2
energy (Mokken)
6.98
1.9910..15
7.20.10.08
7.52.1..32
4.72.10.00
perplexity
2.13:10033
1.95100.03
2.1510.011
2.15
1,88 0.05
CA
speed (token/s)
35.6:1:.45
38.5
28.410344
31.49...99
49.6
OPT
energy (Ihoken)
5.74::0.11
5.12::0.06
6.2420.11
8.68 1.83
4.03=00.22
perplexity
3.324:0.10
2.60:10.05
3.16:10006
3.42
2.07::
speed (token/s)
30.74:0.88
30.5::0.21
25.0=2.31
24.620006
44.2=0099
Llama-2
energy (Mokken)
7.07
1.416
8.22.10.19
1.085
4.80 0.12
perplexity
2.87+0008
2.93+00.3
3.06#0.11
2.92=0.09
2.63=00.10
speed (tokenfs)
31.710911
30.911..00
23.71:0.0
25.7...36
03.6
OPT
energy (Mtoken)
6.37:10.11
6.71:10.17
7,31
8.03
4,86: 0.03
perplexity
3.97:10.06
3.74:10.09
4.04
3.92:1: 0.34
3.17 ::17::10 0.06
speed (lokenfs)
24.0.1..88
26.211.0088
24.21.0229
23.810020
26.4
Llama-2
energy (Mtoken) L
10.75.10.02
9.84
11.010.08
11.010.76
9.01
perplexity
2.26::0.01
2.13:20.03
2.29:0004
2.29:20.03
0.03
SP
speed (token/s)
24.6:1:030
29.9110.55
19.8
24.1
34.4:10.46
energy (J/token)
15.64:3.55
13.6::3077
20.1+2.52
16.9::2.75
11.7::2.36
perplexity
2.30::0.07
1.90:00001
2.20::0.09
2.21: 0.01
1.63:: 0.03
speed (loken/s)
23.0:11100
19.1
energy (Mokken)
7.40.10.19
9.27.0.54
9.20
perplexity 3.644:0.11
3.44:0076
3.64+0.51
3.63+2.50
2.21=:0118
MT
speed (token/s)
34.041..00
44.7112.92
28.712.46
28.5
48.0
OPT
energy (Jooken)
12.1::0.36
6.23:10.67
12.91.733
13.241..88
6.11:00822
perplexity
2.02::::40
1.50::0277
Table 8: Average number of tokens generated each iteration across all datasets.
Llama-2
OPT
SpD
2.021:0.05
2.60:1:0.06
BiLD
188310.10
2.68
Speetr
2.73400.3
3.45+0042
Specinfer
2.7440.46
3.45#0040
MTAD
3.1710.43
4.30-10.03
5.5
80
5.0
opt
S 70
opt
4.5
Llama-2
60
Llama-2
4.0
50
40
3.5
3.0
20
2.5
0.0 0.2 0.4
0.6 0.8 1,0
0.0
0.2 0.4
0.6 0.8 1.0
acceptance threshold
acceptance threshold
Figure 3: Bffect of acceptance threshold on output perplexity and decoding speed.
18

---

Published as a conference paper at ICLR 2025
Table Downstream task scores speculative decoding and MTAD. All the scores higher the better.
SpD
MTAD
Rouge-L
0.114
11.5
13.0
Humanitics
2,95
3.15
Extraction
1,80
2,50
Roleplay
3.10
3.80
Math
1.10
Coding
Reasoning
STEM
2.85
3.10
Writing
3,80
3.65
Average
2.58
2.68
Table 10: Number of Generated Tokens Per Iteration Across Thresholds
Threshold
0.1
0.3
0.5
0.7
0.9
Llama-3
4.72
4.43
4.12
3.93
3.64
-lama-3-Instrue
4.74
4.51
4.28
4.06
3.79
C.2 ABLATION STUDIES ON ACCEPTANCE THRESHOLD
Next, evaluate the effect of acceptance threshold т. Intuitively, when increase from 0 to the acceptance criterion becomes stricter, the efficiency drops while the output perplexity increases. Surprisingly, this expectation only partially correct. As shown in Figure 3 the efficiency indeed drops when increases. However, the perplexity rises when is close to 1. When 7 1, all the draft tokens are rejected, which makes MTAD equivalent multinomial sampling. Similarly, when is close to 1, the advantage of multi-token joint decoding effectiveness disappears, hence MTAD's perplexity similar the perplexity of multinomial sampling. Moreover, when ranges from to 0.9, the performance of MTAD is relatively stable, suggesting that MTAD is not sensitive to the acceptance threshold.
In addition, Table exhibits how the number of accepted tokens change with thresholds. We can see that the number of accepted tokens grows smaller when the threshold increases, which explains the speed degradation.
C.3 ABLATION STUDIES ON NUMBER OF BEAMS
In Table 11 we investigate how the number of beams used in the beam decoding of the small model affects the inference performance. Increasing the number of beams boosts the quality of the draft tokens, which not only improves the output perplexity but also increases the average acceptance length and hence leads to better efficiency. But we can see that the increment slows down when the number of beams too large. In addition, when the number of beams is too large, the inference cost of the small model will become too high.
1 'The tasks of MT-Bench humanities, extraction, roleplay, math, coding, reasoning, stem, writing, and STEM.
"MT-Bench contains multi-turn tasks where the input includes the responses of L.I.Ms, so the input length is fixed.
19

---

Published as a conference paper at ICLR 2025
Table 11: Effect of beam count inference performance (ChaiGPT-Prompts dataset). Arrows indicate optimal direction (t=higher better, !-lower better).
Number of Beams
Speed (tok/ss
55.9
60.2
61.3
Llama-2
Energy (Jitok)
2.43
2.25
2.22
2.20
Perplexity
Speed (tok/s) 1
51.0
54.1
54.3
55.9
OPT
Energy (Ihok)
2.50
2.32
2.36
2.30
Perplexity
3.19
C.4 ABLATION STUDY OF TOP-K AND TOP-P SAMPLING
Table 12 demonstrates how the value of k and in top-k and top-p warping affects our method. We can see that by changing the value of and p, MTAD consistently performs significantly better.
Table 12: Ablation study of k and in top-k and top-p sampling
K
Multinomial
SpD
MTAD
PPL
Tokens/sec
PPL
Tokens/sec
PPL
Tokens/sec
20
0.9
3.74
22.6
3.64
36.8
2.06
63.0
20
0.8
3.06
22.7
3.10
38.5
1.93
58.8
10
0.9
3.03
22.7
3,22
38.5
1,95
62.5
10
0.8
2.56
22.7
2.53
40.0
1.80
62.5
C.5 RESUTLTS WITH OPT-30B AND LLAMA-2-705
Here report the performances of different methods for OPT (350M and 30B) and Llama-2-Chat (7B and 70B). Table 13 shows the average performances across all datasets. MTAD always realizes the lowest perplexity and the best efficiency.
Table 13: Inference efficiency and output perplexity different methods with OPT (350M,30B) and Llama-2-Chat (7B,70B). The mean and standard deviation are computed across all datasets. Bold numbers mark the best result, underlined numbers mark the second best.
SpD
BiLD
Spectr
SpecInfer
MTAD
speed (token/s) ?
8.37.13.07
8.64.13.50
9.11133.03
8.87.2.82
9.53.13322
Llama-2
energy (J/token) 4
138-+87.77
1424-99.7
1224-66.4
125-+65.4
119:-6..7
perplexity
1.77:t0.22
1.69:0025
1.73+0022
1.73::0.24
1.52:10.19
speed (token/s) ?
15.3.11.64
14.5.11.96
17.014.14
17.414..00
19.5
OPT
energy (Jitoken) 4
72.4+11.55
79.6++3.03
68.2+16.7
62.4+10.3
60.0++12.8
perplexity
1.74
3.50
4.55
4.49 1.95
2.74
C.6 ADDITIONAL EXPERIMENTS ON CNNDM AND SPIDER
Table 14 and Table 15 reveal the downstream effectiveness of method on CNNDM and Spider when the small model and large model are fine-tuned on the dataset. We can see that MTAD is consistently more effective and has faster decoding speed.
20

---

Published as a conference paper at ICLR 2025
Table 14: Comparison of ROUGE-L Scores and Tokens per Second under Different Fine-Tuning Conditions on CNNDM
No Fine-Tune
Fine-Tune 68M
Fine-Tune Both
Method
ROUGE-L
Tokens/sec
ROUGE-L
Tokens/sec
ROUGE-L
Tokens/sec
SpeD
0.114
37.7
0.114
20.4
0.164
24.3
MTAD
0.118
44.2
0.121
25.0
0.168
27.1
Table 15: Comparison of Execution Accuracy (EA) and Tokens per Second under Different Fine- Tuning Conditions on Spider
No Fine-Tune
Fine-Tune 68M
Fine-Tume Both
Method
EA
Tokens/sec
EA
Tokens/sec
EA
Tokens/sec
SpD
11.5
28.5
11.5
27.8
16.3
25.6
MTAD
13.0
30.3
14.8
32.3
18.3
29.4
C.7 VISUALIZATION OF PERPLEXITY AND OUTPUT QUALITY
To further illustrate the relationship between perplexity and downstream performance, we present scatter plot in Figure ! The plot depicts the correlation between relative downstream scores (normal- ized by the score of multinomial sampling) and relative perplexity (normalized by the perplexity of multinomial sampling) across 7 decoding algorithms, 3 datasets, and 2 model configurations. The results confirm that lower perplexity generally correlates with higher output quality.
C.8 CORRELATION BETWEEN ENERGY AND SPEED
Figure 5 shows there is correlation between speed and energy, whether we consider the entire table focus on a specific dataset and model. For fairness, all methods for given dataset and model were run on the same machine nodes. However, for fixed method (e.g.. Spectr), experiments on different datasets and models might be conducted on different nodes (all equipped with L40 GPUs). We did on different machines may have varied energy consumption.
notice that the same configuration This variation introduces some randomness, which could make the correlation appear less consistent across datasets and models.
2.2
relatve score
1.8
16
1.2
10-
0.6
0.7
0.8
0.3
2.0
relative perplexity
Figure 4: Relationship between relative perplexity (normalized by multinomial sampling's 's perplexity) and relative performance score (normalized by multinomial sampling's 's score).
21

---

Published as a conference paper at ICLR 2025
2
Jaraar
10.0
specalotive
Ch:OPT
BILD Spectt
Ckellama2
CA+OFT
175
Specinter
C24Jamaz
MIAD
35.0
Cleceet
15.0
energy (P9akes))
225
12.
-0.0
10.0
7.5
75
54
5.-
20
speed
speed
(a) group by dataset and model
(h) group by methods
Figure 5: Correlation between speed and energy
D ENERGY CONSUMPTION MEASUREMENT
To get GPU power every second, we run the command "nvidia-smi query-gpu=powe draw -format=csv". We add the results up to determine the to- tal energy consumption. We use average energy consumption per token to measure energy efficiency. There recent study pointing out the measurement error using nvidia-see (Yang et al. 2023). We follow the three principles proposed in (Yang al. 2023 to minimize this error.
E CONFIGURATION
The experiments are conducted on a machine with Nvidia L40 GPU (48 GB), 4 CPUs, and 50 GB main memory, using a batch size of 1, which is common for online serving (Schuster al. 2022). We set the maximum running time to be an hour for each baseline. We use average tokens/ to measure the inference speed and use average energy consumption per token to measure energy efficiency.
F EAPER-PARAMETER DETAILS
In the experiments, we follow the settings in (Bear 2024) to warp the sampling distribution and with the following steps, which are the default warping operations in a public speculative decoding implementation. Specifically, we first keep the probabilities of top 10 tokens unchanged, and set the probabilities of other tokens to O, then normalize the distribution. Then we sort the tokens based on their distributions in descending order and keep the first K tokens such that their cumulative probabilities larger than 0.9, while set the probabilities of other tokens to O.
For different methods, we choose their hyper-parameter based the set of ryper-parameter that make the corresponding method have best output perplexity. For MTAD, choose the beam width from (4, 8). the number of draft tokens from {3, 4), and the acceptance threshold from 10.1, 0.3, 0.5, 0.5,0.7,0..9 0.9).
small validation set. We select
G LICENSE OF DATASETS AND MODELS
Datasets:
ChatGPT-Promptss Non http://wgggngaaaaoooceccosooooen prompts)
ChatAllaca: Apache-2.0 License
CNN Dailymail: Apache-2.0 License
MTBench: Apache-2.0 License
22

---

Published as a conference paper at ICLR 2025
Spider: CC BY-SA 4.0 License
HumanEval: MIT License
Models
OPT-125M and
OPT-13B:
Model License
https:///ithub.com
LICENSE. md) LJama-68M: Apache-2.0 License
Llama-2-13B: Llama-2 Community License Agreement
Llama-3: the Llama 3.1/3.2 Community License
Codes
LLMSpcculativeSamping MiMppeluulai..eSsmmllingggg
https://githbb.cmmffaeieebee
23


Reviews:
[{"review_id": "b44aFozTYZ", "reviewer": "Reviewer_rQ6h", "summary": "The paper proposed multi-token assisted decoding by combining multi-token joint decoding and speculative decoding (SpD) to improve the quality of the tokens and inference speed.", "questions": "target model (larger model) is fine-tuned on chat-GPT data for proposed method, is the same fine-tuned model used for other Speculative Decoding (SpD) methods as well? \n\nWhat about the draft model? is that fine-tuned as well or used as is? \n\nCan qualitative generations be shown in the paper? There are cases when lower PPL is observed during SpD as the draft and target model are both generating and accepting repeated tokens.\n\nDid the author also check performances (PPL, speed-up) when using a larger draft model? \n\nFrom table 4., it seems like for past SpD methods there is no correlation between high speed (token/s) will lead to low energy (Joule/s), but that doesn't seem to make sense, as both high speed and low energy should be related to high acceptance rate which leads to small number of target model calls. Any comment/thoughts on this? \n\nCan the authors also add plots on acceptance threshold vs acceptance rate? Given the high token-rates shown in Table 4., I am curious to see how the acceptance rates look like.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper solves an interesting problem of sampling from a joint distribution which is challenging and proposes an alternate approximate method based on draft model from Speculative Decoding (SpD) \n\nIt uses ideas from SpD to also boost the inference speed.\n\nPaper uses an auxiliary model to predict joint distribution of multiple tokens w/ beam sampling to sample the tokens fast. \n\nThe power analysis provides some insight into the importance of using SpD\n\nAuthors provide a robust set of experiments", "weaknesses": "the experiment results can be re-checked, my concern is that lower PPL not always results in better quality, it could also be that the draft model is generating repeated tokens which the target model is accepting, given that there's no guarantee that the draft tokens accepted will follow the target distribution. \n\nthe reduction in energy consumption is a by-product of SpD and could have been mentioned in a sub-section rather a full section. \nThe acceptance rate of draft model should affect both the token-rate and the energy consumption.\n\nNo mention of draft model or target model training and if they are different from the baseline algorithms draft and target model in the experiments section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NeY6Qd51y8", "reviewer": "Reviewer_h4X4", "summary": "This paper introduces a new framework named multi-token assisted decoding (MTAD) by combining speculative decoding with multi-token joint decoding (MTJD). MTAD generates high quality draft tokens by estimating the joint distribution of the target model using draft model, while improving both the speed and quality of inference.", "questions": "1. The threshold for the verification phase of vanilla speculative decoding is sampled from U(0, 1) to ensure losslessness, and MTAD uses a fixed threshold (e.g. 0.1). I'm curious to see if the same speedup can be achieved with the vanilla speculative decoding method using a fixed threshold.\n\n2. The impact from loose thresholds has little effect with PPL and LM-Judger metrics, some more rigorous evaluation (e.g., Humaneval's Pass@K) is needed to validate the methodology.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed MTAD guides the generation and verification of draft tokens based on joint probabilities, providing a new idea for speculative decoding.\n\n2. This paper analyzes the speculative decoding technique from an energy perspective for the first time and verifies the energy efficiency of speculative decoding. \n\n3. In addition to the experimental analysis, the authors also theoretically demonstrate the bounded error as well as the effectiveness of the MTAD algorithm.", "weaknesses": "1. Despite generating texts with lower perplexity, MTAD is still a lossy speculative decoding method. It is unfair to compare MTAD with lossless speculative decoding methods. It might be useful to add some comparisons with other lossy verification methods (e.g. Medusa's Typical Sampling) to verify the superiority of MTAD.\n\n2. The main experiments in this paper use perplexity as an evaluation criterion for performance, and it is clear that MTAD based on PPL verification will have advantages. More experiments on downstream tasks are needed to verify whether MTAD can really improve the quality of text, since PPL and metrics are not strictly positively correlated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U5k8QuHyGB", "reviewer": "Reviewer_v4L1", "summary": "This paper focuses on improving the inference speed of large language models (LLMs). The authors propose a variant of speculative sampling that enhances both inference speed and output quality. Specifically, they explore multi-token joint decoding (MTJD), which generates multiple tokens from their joint distribution at each iteration, theoretically reducing perplexity and improving task performance. Building on this, they incorporate speculative sampling to approximate multi-token joint decoding and design an efficient algorithm, multi-token assistant decoding (MTAD), which achieves bounded error with respect to MTJD. Empirical evaluations demonstrate the effectiveness of the proposed method in terms of speed and perplexity.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The writing is good and easy to follow.\n- The addressed problem is of great importance to the community.\n- A new metric is provided to evaluate the effectiveness of speculative sampling, which is insightful.\n- Strong experimental results:\n  - Evaluated across a range of datasets\n  - Shows significant improvements in terms of perplexity, speed, and energy efficiency.", "weaknesses": "- In line 114, the authors claim, “As shown in Table 1, lower perplexity correlates with improved downstream performance, even in one of today’s largest models.” This assertion is questionable. The results in Table 1 are based exclusively on experiments with the Spider dataset, but in many cases, lower perplexity does not inherently lead to better downstream performance. For example, in tasks like machine translation and abstractive summarization, a lower perplexity score sometimes fails to correlate with improved downstream performance. This also can be validated by the right plot in Figure 1.\n- Lack of guarantee on quality: In vanilla speculative sampling and multi-draft speculative sampling, one of their advantages is the property of being lossless. This ensures that the quality of output is maintained, which further accelerates LLM inference. However, the proposed method does not have such guarantee. \n- The verification process in this work is similar to that in [1], as both aim to verify a sub-sequence of tokens in a single step rather than one token at a time. However, this study does not include a comparison between MTAD and the method in [1]. The author should further compare their work with it to further improve the quality.\n- The experiments also have some limitations:\n  - This paper places too much emphasis on perplexity. Since a lower perplexity score does not always correlate with improved downstream performance, the authors should provide a more comprehensive evaluation of downstream performance. I noticed that Table 6 contains some comparisons on downstream performance, but this is insufficient. I encourage the authors to add a \"downstream performance\" metric to Table 4 so that readers can better understand the effectiveness of the proposed method.\n  - I still do not fully understand why the proposed method significantly outperformed multi-draft speculative sampling methods such as Specter and SpecInfer. I suspect this is because (1) the selected baseline is weak, and (2) the configurations are not fair. For example, in Table 13, the number of sequences in Specter and SpecInfer does not match the number of beams in MTAD. I encourage the authors to do the following: (1) select more recent multi-draft speculative sampling methods for comparison, and (2) ensure consistent configurations.\n\n\n[1] Sun, et al. Optimal Block-Level Draft Verification for Accelerating Speculative Decoding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K0infMXU9X", "reviewer": "Reviewer_ds8N", "summary": "Considering the inference efficiency, this paper proposes multi-token joint decoding (MTJD), which generates multiple tokens from their joint distribution at each iteration. More importantly, this paper theoretically proves that their method can reduce perplexity and improve task performance compared to single token decoding. Besides, to mitigate the high cost of sampling from the joint distribution of multiple tokens, it introduce a small model to assist the decoding of a larger model. Empirical evaluations reveal that the method reduces perplexity\nby 21.2% and improves downstream performance compared to standard single-token sampling. It also achieves a 1.42× speed-up and consumes 1.54× less energy than conventional speculative decoding methods.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper theoretically proves that multi-token joint decoding (MTJD) can reduce perplexity and improve task performance compared to single token decoding.\n2. This paper introduces multi-token assisted decoding (MTAD), a novel framework designed to accelerate multi-token joint decoding. Theoretically, this paper demonstrate that MTAD closely approximates exact MTJD with bounded error.\n3. These results highlight MTAD’s ability to make multi-token joint decoding both effective and efficient.", "weaknesses": "1. This paper primarily conducted experiments on the OPT series and Llama 2 series models. However, the performance of these large models has already fallen far behind the recent Llama 3.1 and Llama 3.2. Could you provide more results on the latest models to demonstrate the generality of the results?\n\n2. The previous methods compared in this paper were all published last year. It would be beneficial to compare with some recently published methods, such as MEDUSA-2 and \"Better & Faster Large Language Models via Multi-token Prediction\", to prove the effectiveness of the results.\n\n3. In this paper, a major point is that multi-token joint decoding can reduce perplexity and improve task performance. However, the experimental results do not reflect related analytical experiments. Adding relevant experiments could enhance the self-consistency of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)