Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Authors: Jinze Li
Yixing Xu
Haiduo Huang
Xuanwu Yin
Dong Li
Edith C. H. Ngai
Emad Barsoum
Venue: ICML 2025 poster (2025)
Keywords: Speculative Decoding
Large Language Models
Primary area: deep_learning->large_language_models
TL;DR: Gumiho speeds up LLM text generation with a hybrid approach: it prioritizes accuracy for crucial early predicted tokens and uses faster parallel processing for later ones. This design leads to improved overall inference speed.

Abstract: Speculative decoding (SPD) aims to accelerate the auto-regressive token generation process of a target Large Language Model (LLM). Some approaches employ a draft model with multiple heads to predict a sequence of future tokens, where each head handles a token in the sequence. The target LLM verifies the predicted sequence and accepts aligned tokens, enabling efficient multi-token generation. However, existing methods assume that all tokens within a sequence are equally important, employing identical head structures and relying on a single-generation paradigm, either serial or parallel. To this end, we theoretically demonstrate that initial tokens in the draft sequence are more important than later ones. Building on this insight, we propose Gumiho, a hybrid model combining serial and parallel heads. Specifically, given the critical importance of early tokens, we employ a sophisticated Transformer architecture for the early draft heads in a serial configuration to improve accuracy. For later tokens, we utilize multiple lightweight MLP heads operating in parallel to enhance efficiency. By allocating more advanced model structures and longer running times to the early heads, Gumiho achieves improved overall performance. The experimental results demonstrate that our method outperforms existing approaches, fully validating its effectiveness. Our code is available at https://github.com/AMD-AIG-AIMA/Gumiho.

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Jinze Li 12 Yixing Xu' Haiduo Huang's 13 Xuanwu Yin Dong Li Edith C.H. Ngai Emad Barsoum
L.jinze-bkuéconnsee hk, (yixing.xu xu, Xuanwu. Yin, d.li, emad, barsoum)aaddd
hk, juanghdestu.xjtuu edu. en, chngaißeee hku.hk
Abstract
1. Introduction
While Large Language Models (LLMs) (Achiam al., 2023; Touvron et al., 2023) have demonstrated impres- sive capabilities, their auto-regressive inference introduces significant latency challenges, especially as the number of parameters continues to increase. Speculative decod- ing (SPD) (Leviathan al., 2023; Chen et al., 2023) has emerged promising solution to this problem. It lever- ages smaller models efficiently propose draft tokens for future steps, which are then verified in parallel by the L.M. Specifically, in each draft round, the draft model generates a sequence of multiple draft tokens, and the target LLM verifies these tokens in parallel. This innteration-verficaatt process constitutes draft round. These draft tokens are accepted only if they match the LLM's original output. If mismatch occurs, starting from the first divergent token, all subsequent draft tokens are discarded.
Speculative decoding (SPD) aims to accelerate the auto-regressive token generation process of target Large Language Model (LLM). Some approaches employ draft model with multi- ple heads to predict a sequence of future tokens, where each head handles a token in the sequence. The target LLM verifies the predicted sequence and accepts aligned tokens, enabling efficient multi-token generation. However, existing meth- ods assume that all tokens within sequence are equally important, employing identical head struc- tures and relying either serial or parallel. To this end, we theoreti- cally demonstrate that initial tokens in the draft se- quence are more important than later ones. Build- ing on this insight, we propose Gumiho, a hy- brid model combining serial and parallel heads. Specifically, given the critical importance of early tokens, we employ sophisticated Transformer architecture for the early draft heads in serial configuration to improve accuracy. For later to- kens, we utilize multiple lightweight MLP heads operating in parallel to enhance efficiency. By allocating more advanced model structures and longer running times to the early heads, Gumiho achieves improved overall performance. The ex- perimental results demonstrate that our method outperforms existing approaches, fully validat- ing its effectiveness. Our code is available at http:///hwbbbcoo//AAAAa---AAAAAC
single-generatin paradigm,
Recent advances (Cai et al., 2024; Li et al., 2024a;b; Ankner et al., 2024) have shown that smaller models, which leverage the hidden states of the LLM itself, can achieve substan- tial speedups inference while maintaining output quality. Medusa (Cai et al., 2024) predicts multiple future tokens in parallel using MLPs with the last verified token's 's hidden state, while Hydra (Ankner et al., 2024) and Eagle (Li et al., 2024a) generate tokens serially.
Although Medusa's parallel prediction paradigm runs faster through simultaneous draft token predictions, it relies solely on hidden states from previously verified tokens, making it blind earlier unverified predictions within the current draft round. Serial methods like Eagle, Eagle-2 and Hydra can fully utilize previously generated draft tokens, but their sequential paradigm tends to slow down the drafting pro- cess, making it less efficient. More critically, all approaches treat all tokens within a draft round as equally important, which unsuitable for SPD. In our view, the tokens gen- erated earlier in each draft round hold more importance than those generated later. This is because when the first incorrect token is encountered, it causes both that token and all subsequent draft tokens to be discarded, even if the following tokens are correct. We will formally present a
'Advanced Micro Devices, Inc., Beijing, China Department of Electrical and Electronic Engineering, The University of Hong Kong "Institute of Artificial Intelligence and Robotics, Xi'an Jiaotong University. Correspondence Edith C.H. Ngai <ehn- gai hkc.hku.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
theorem and mathematically prove that prioritizing the ini- tial tokens of sequence consistently improves the mean accepted tokens (7) in each draft round.
candidates, allowing tokens from longer candidates augment shorter ones. This approach further increases the acceptance length without incurring additional com- putational overhead.
Motivated by this, we propose Gumiho, a hybrid model that prioritizes the initial positions in the draft sequence, combin- ing both sequential and parallel architectures. We employ serial structure comprising a two-layer Transformer, en- abling comprchensive modeling of token dependencies and context for early tokens. The subsequent tokens, which are relatively less critical, are predicted in parallel through sim- ple MLPs, thereby enhancing computational efficiency. By combining the serial and parallel architectures, our hybrid design achieves higher acceptance lengths and less process- ing time in each verfiiieriionattoo cycle at the same time. The key idea of our hybrid model lies in two folds: (1) it allocates more parameters and leverages serial processing for crucial early token predictions, maximizing accuracy where it matters most, and (2) it employs efficient paral- lel computation with simple architecture for later tokens, reducing the overall computational cost.
We conduct comprehensive experiments to demon- strate Gumiho's 's superior performance compared to existing methods.
2. Related Works
With the widespread adoption of large language models (LLMs), significant research has been devoted to acceler- ating their inference through techniques such distilla- tion (Hinton, 2015; Bercovich et al., 2024; Zhao et al., 2024;
Fu et al., 2024), low-bit quantization (Hubara et al., 201 8;; Shen et al., 2020; Kim et al., 2021; Zadeh et al., 2020; Zafrir et al., 2019), pruning (Gale et al., 2019; Sanh et al., 2020; Kurtic et al., 2022: Voita et al., 2019), and innovative network architecture designs (Gu & Dao, 2023; Wu et al., 2020). These approaches aim to reduce the computational cost of each forward pass to improve efficiency. However, they often involve a trade-off (Donisch et al., 2024), as these optimizations can partially compromise model performance, requiring a balance between generation quality and compu- tational overhead. Speculative decoding, a draft-then-verify paradigm (Xia et al., 2023), achieves lossless acceleration by leveraging the original LLM for verification.
In addition, we enhance Eagle-2's s dynamic tree mechanism by introducing Full Tree Attention (FTA). While Eagle-2's dynamic tree selectively expands promising nodes and re- ranks draft tokens for optimal verification, this re-ranking process can result in shorter candidate sequences when later tokens are discarded due to low scores, potentially reducing the mean accepted tokens (T). We observed that tokens gen- erated by parallel heads exhibit correlations, as they share both input and purpose. Specifically, each n-th head de- signed to predict the token that should appear positions away from the input token. Inspired by this, our full tree attention mechanism supplements shorter paths with tokens from longer paths at corresponding positions, thereby in- creasing the mean accepted tokens (T) of shorter candidate paths. Since these supplementary tokens come from exist- ing longer candidates, their q. kt and have already been computed, incurring no additional computational overhead.
Drafting approaches can be broadly categorized into two paradigms (Xia et al., 2024): independent drafting which employs draft models that can be deployed without addi- tional training, and self-drafting which requires dedicated training processes to develop effective draft models.
Independent drafting typically uses separate, smaller model to generate multiple future tokens concurrently, thereby enhancing the efficiency of speculative decod- ing. For example, using T5-small to accelerate T5- XXL (Leviathan et al., 2023). These off-the-shelf drafters do not need extra training or architectural modifications and benefit from the inherent alignment in prediction behaviors due to shared tokenizers and pretraining processes. How- ever, independent drafting requires additional work to find or train a compatible model that matches the target LLM. This becomes more challenging when smaller versions of the LLM don't exist.
Our contributions are summarized as follows:
We propose Gumiho, a hybrid structure model for SPD, inspired by the observation that earlier tokens have more impact on the overall sequence length accepted, while later tokens are relatively less critical. We pri- oritize the generation of earlier tokens by allocating more computational resources and using a serial ap- proach to enhance accuracy, while simpler models are employed in parallel for the later tokens to improve computational efficiency.
Orthogonal to independent drafting, self-drafting typically uses the target LLM itself (Liu al., 2024a; Du et al., 2024;
Elhoushi et al., 2024; Gloeckle et al., 2024; Cai et al., 2024;
Li et al., 2024a; Zimmer et al., 2024; Xiao et al., 2024;
We demonstrate through theoretical analysis that to- kens appearing earlier in the draft sequence have a more significant impact on the overall accepted length.
Zhang et al., 2024: Brown et al., 2024; Liu et al., 2024b), utilizing features like its hidden states for more efficient drafting. Medusa (Cai et al., 2024) is one of the studies to leverage the hidden state of the original LLM as input for
We propose a full tree attention mechanism for tree
2

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
and ignore draft models. It employs K distinet MLPs as draft models, each predicting one of K future tokens. Since these K draft models operate independently, they can execute in parallel, enabling Medusa to generate K tokens in a single forward pass. Hydra (Ankner et al., 2024) is a sequential variant of Medusa, transforming the parallel MLPs into a serial architecture. feeds the unverified tokens from draft mod- els as imput subsequent ones. This sequential approach enables each draft model to leverage more contextual infor- mation when predicting later tokens, enhancing the quality of successive predictions. Eagle (Li et al., 2024a) advances the architecture by converting the serial MLPs into serial Transformers and introducing concatenated token-hidden state pairs as input. Eagle-2 (Li et al., 2024b) further inno- vates by implementing a dynamic tree candidate selection mechanism to enhance token prediction efficiency.
we define yt++ MM(()) - kuet for simplicity.
MM(yee kver)
Speculative Decoding Auto-regressive text generation in LLMs is time-consuming. Speculative decoding addresses this limitation by employing a smaller, faster draft model Ms to generate candidate tokens ahead of time. These candidate tokens, commonly referred as drafts, are then verified in parallel by the target LLM Mr using rejection sampling (Leviathan et al., 2023), i.e., if any token in the draft sequence is rejected, all subsequent tokens are dis- carded, and the draft-verification process resumes from the last accepted token.
During each drrfftveriiication iteration, Ms generates a sequence of D draft tokens (de++ Subsequently, Mr verifies these drafts in parallel according to Eq.
Our method falls within the self-drafting category. Different from previous self-drafting methods, we utilize the fact that the importance of a token decreases as its position moves back and propose different architectures for front-position positioos tokens and later ones.
M++1 Me(v)).
y142 Ma(0(41).
(2)
BID ~ Mi(
3. Method
The number of accepted draft tokens for each iteration is de- termined by comparing the draft sequence (U++1,-- --. Ht+ with the verified sequence (yc-1, Mt+D] using rejection sampling.
In this section, we begin by introducing the preliminaries of speculative decoding (SPD). We then present a theorem and provide a rigorous mathematical proof to validate that tokens at the beginning of the sequence are more crucial than those the end. Finally, we propose Gumiho, novel method derived from theorem.
Metrics We assess the method's performance by measur- ing its acceleration effect. Specifically, we use the speedup ratio as metric, which is calculated by dividing the speed of our proposed method by the speed of standard (vanilla) decoding:
3.1. Preliminaries
LLM Decoding The process of generating text from LLMs is termed decoding: the sequential production of tokens in response to input prompt. This generation pro- cess follows an auto-regressive pattern, where each token Ye is generated by sampling from a probability distribu- tion conditioned on both the initial prompt and all pre- viously generated tokens yer. In practice, the key-value cache (kvvzz) of previously generated tokens maintained, with the model taking both kvzt and the current token ye as inputs for efficient generation.
Speedup ratio speedoumihe
(3)
speedvanilla
The speed of each method is calculated by dividing the total number of generated tokens by the total processing time:
total tokens
speed
total time
mean accepted tokens x draft rounds
average time x draft rounds mean accepted tokens average time
(4)
Let Mr. denote the Large Language Model, which com- prises two components: the decoder layer fi(-) and -ead that maps the embeddings back to the vocab- ulary with size [V). The vanilla decoding process can formulated as:
Following existing works (Li et al., 2024b; Ankner et al., 2024), we primarily use mean accepted tokens (r) as the main metric. We also present the differences in draft time across different methods to demonstrate the effectiveness of our approach in improving efficiency.
ke het1 = fu(kuee, e(v:)),
Vt+1 So.tmax(T.M (heo)(
(1)
3.2. Theoretical Analysis
where h141 denotes the hidden states of the final decoder layer, and e(-) is an embedding function that maps tokens to their corresponding vector representations. In the following,
In this section, we prove that tokens at the beginning of the draft sequence are more crucial than those at the end.
3

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Medusa
Thetys)
sun(
Hidden
States
Head
M,
Head
MLP Head MC
aboue Obse
LM Head
Head
hess
Reez
MLP NG
the Uiee
Hydra/Eagle
Hidden States
Head
Head
Transformer Layers
Concat
MLP Head MM
mounturil (iuss
O2
Gr-1
Gco2
f
FC
Gumiho
through Uftee
Head MG
Hidden States
Embedding
hy
fices
heez
Head (Large)
Head (Large)
Ismaal
e(y,)
e(Po+s)
e(Poa))
MLP Mi
the (fee2)
2
Input
Embedding
[hesall
Describe besutiful scenee
Tho
sunt Stel
rose(
droft token (Serial)
draft taken (Serial)
The remaining draft toleens (Paraliel)
LLM Forward
Figure 1. Left: Differences between our proposed Gumiho and existing methods; Unlike existing approaches that similar models predict every token in sequence, we propose that initial tokens are more critical than later ones. So we employ a larger model with a serial structure to generate the early tokens, while leveraging smaller parallel models for the later ones. Right: Overview of Gumiho. Given an LLM input Describe a beautiful scene., Gumiho predicts the next 7 draft tokens (sun rose above the mountains through the). The first tokens (swn and rose) deemed critical and produced sequentially using the Transformer Mr for higher accuracy. The remaining tokens are generated simultaneously through the MLP heads, optimizing for computational efficiency.
Consider draft model that predicts 3 tokens at time with uniform acceptance probability of 0.8 at each position. The expected length EE [[[] Jorigiaal of accepted tokens per draft round is:
In the following, provide theorem to generalize the above example to broader scenario. Given draft se- quence with length D, we first have an original setting with acceptance probabilities {p:)__ and denote [F[L.) as its mean accepted tokens (T). Since errors accumulate when predicting a sequence, the acceptance probability of later to- kens tends to be lower than that of earlier tokens in practical scenarios. Based on this observation, we have:
ET oxiginal
1x P(L 22 2
Σ (1) ()
0.8 0.8 + 0 1.95.
1 12 pr 2 2 PD 2 0.
(5)
where I. represents the accepted length determined by the target LLM's verification after each draft round. E[|] de- notes the expectation operator, and PLL = i) represents the probability that the accepted length L is equal to
Then, we define an improved setting whose sequence is separated by index d with 1 < d D. In this setting, acceptance probabilities p; are modified follows:
P. = p. + G. 1sisd
Now, suppose redistribute the model's parameters to prioritize earlier positions, i.e,, allocating more parame- ters to predict the first position and fewer for subsequent positions. Assume this improved structure creates position- dependent acceptance probabilities of 0.85, 0.8, and 0.75 for the first, second, and third positions respectively. The expected length [[[]impdweed becomes:
p. Gi d d<igD"
s.f. S (c)) $
05 (p:)
51,
Σ E Σ So G
(6)
j=141
In this improved setting, we increase the acceptance prob- abilities for the first d tokens by a small amount Ç; and decrease those for the remaining tokens by the same total amount. In this way, the sum of the acceptance probabil- ities remains unchanged. We denote the mean accepted tokens (T) in this improved setting as E[L]improved- With these definitions above, we can derive the following theo- rem:
T3 [[]]innpooee Iispoooed Σ
P(I. 2 f)
0.85 + 0.85 0.8 0.85 0.8 0.75 2.04.
This example empirically demonstrates that when the overall token accuracy remains constant, improving the accuracy of the initial token can increase the mean accepted tokens (r).

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
The mean accepted tokens (T) under the im- proved probability distribution exceeds that of the original distribution:
Theorem 3.1.
The sun
0.3
shines (0.5)
glows (0.3)
rediutees (0.2)
E| Jimprovedd
2 Joriiiiinal.
0.3
0.4 warmly (0:12)
0.3
0.3 intervelly
warmly 10.2)
br brightly [0:35]
Intensely (0.15)
br shilllyy (0.09)
intensely (0.29)
warmly [0.08]
brightly (0.05)
[0.05]
This theorem shows that redistributing the acceptance proba- bilities across sequence tokens by increasing the accuracies of the initial tokens and decreasing those of the later tokens can improve the overall expected performance. A detailed proof of this theorem is provided in Appendix A.
0.6
0.3
0.1
0.6
0.3
0.1
0.5
0.3
0.3
now
hore [9.06]
above
now
here
above
now
here [0.018]
above 0.006)
(0.09)
[0.045]
[001
[0.036]
Select top 8 tokens as candidates by score
Tree Attention
Tree Attention
3.3. Gumiho
The sun
shines
warmly
now
The sun
shines
warmly
now
The sun
shines
brightly
The sun
shines
brightly
now
Inspired by Theorem 3.1, we propose Gumiho, a model that prioritizes the front part of the draft sequence. Gumiho consists of two main components: heads for generating draft tokens and full tree attention mechanism for verification.
The
shines
intensely
The
shines
Intersely
now
The sun
glows
warmly
The sun
plows
warmly
now
The sun
radiates
The sun
radiates
warmly
now
Figure 2. Our proposed Full Tree Attention enhances shorter can- didate paths by borrowing tokens from other tree nodes, thereby increasing the likelihood that candidates achieve longer acceptance lengths. Note that for each depth in the tree, we only have = 3 different tokens.
Gumiho Heads. As shown in Fig. 1, Gumiho introduces a hybrid architecture that distinguishes itself from existing methods. Unlike approaches that rely solely on a single serial or parallel structure and employ uniform head size across all positions, Gumiho combines large serial heads with small parallel heads to enhance accuracy and efficiency.
hu4244 Maleaa((r11 i = L,...... 5.
(10)
The serial component aims to increase the accuracy of the initial tokens and comprises a two-layer Transformer MT that predicts initial draft tokens sequentially. Specifically, MT generates the first two tokens of the draft sequence auto- regressively. Similar Eagle-2, our method concatenates the hidden state he E IRd with the corresponding output token embedding e(yx) Fd generated by LLM MM at time step f, and employs fully connected layer FC to reduce the dimension from 2d to d:
Given the hidden states {he we obtain the draft to- kens f0 using Eq. (1).
Full Tree Attention (FTA). A key distinction between our Gumiho and Eagle-2 lies in the use of parallel heads for generating subsequent tokens. To fully leverage this parallel paradigm, we introduce the full tree attention mechanism, which enhances the existing Tree Attention mechanism em- ployed by Eagle-2.
02 = FC(cat(e((s)) hc)).
(7)
In Eagle-2, tokens at each position are generated in an auto- regressive manner, meaning that each subsequent token is entirely dependent on the tokens generated in the previous. Conversely, our parallel heads Mis generate tokens for all positions simultaneously. This parallel generation paradigm removes dependencies between these tokens, as they are determined solely by the outputs of the preceding serial out- puts generated by .M.. The independence between tokens enables us to perform full traversal connection operation on the tokens generated in parallel. Specifically, any two tokens generated by two different MM can be connected to form a candidate path. As illustrated in Fig. 2, after the serial heads output the tokens the and sun, the three parallel heads generate s subsequent tokens for each position (s 3 in Fig. 2). These tokens at each position can be arbitrarily combined with tokens from other positions, resulting in a total of s candidate paths with only 3s different tokens by the time we complete the third MLP head, In Fig. 2, the score for each token is displayed, calculated by multiplying the score of the preceding token with the confidence of the
where cat denotes the concatenation operation, and or E Rdd Then, the concatenated result o, is fed into the serial component My. which sequentially generates the first two drafts:
hu+1 = MT(o))) 6141 = (C(e (8)
hetz -M 0ct2 = FC(cat(e((++>),
(9)
For simplicity, we omit the input and output of KV cache in My, and also the step of using Softmax to obtain O1-1, 0t+2, which is similar to Eq. (1).
The parallel component aims to speed up the generation of the remaining tokens while maintaining accuracy and consists of five different MLPs {M rumning concur- rently. These MLPs share the same architecture, consisting of two fully connected (FC) layers with a ReL.U activation function in between. They also share the same input, i.e., cat (ō.+ 1 2.42) which concatenate the two outputs gener- ated by the serial model My. The outputs of MLPs repre- sent the draft tokens at the following five positions:
5

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
current MLP in generating specific tokens.
To select candidate paths for verification, we choose the top eight tokens with the highest scores. As shown in Fig. 2, traditional tree attention often results in some candidate paths being very short because tokens on early positions usually have higher scores. To address this issue, our pro- posed FTA mechanism supplements shorter paths with to- kens from corresponding positions in longer paths. This is reasonable since any tokens from different positions among the parallel-generated tokens can be combined, and the borrowed tokens from longer candidate paths do not conflict with the original tokens in shorter paths. This en- sures the coherence of the final candidate paths, maintaining the integrity of the generated sequence. Consequently, this approach increases the average length of candidate paths, enhancing the overall performance of the model.
kmperature -
Saged
Corn
Figure 3. Comparison of average draft time (lower is better) with different lemperatures. Both results are based on Vicuna 7B.
It is worth noting that FTA incurs no additional compu- tational overhead, as the borrowed tokens already exist in other candidate paths within the tree, with their query, key, and value computations already completed. In the original attention tree, these tokens were simply discarded, while our approach unblocks them and allows shorter can- didates to access and utilize them. The rejection sampling method (Leviathan et al., 2023) for token selection ensures that the appending of borrowed tokens does not impact pre- viously selected tokens. Consequently, after applying FTA, each candidate's 's acceptance length equals or exceeds that of the original implementation.
GSM8K (Cobbe et al., 2021) for mathematical reasoning, Alpaca (Taori et al., 2023) for general esstructionfollowinng CNN/Daily Mail (Nallapati et al., 2016) for summarization, and Natural Questions (Kwiatkowski al., 2019) for ques- tion answering. We conduct model training using 8 AMD Instinct MI250 GPUs. For evaluation, we use single MI250 GPU for all models except the 70B variant, which requires 4 M1250 GPUs due to its larger size. Additionally, we include evaluation results using a single NVIDIA A100 GPU in Appendix B.
We compare our method against several existing approaches: Medusa (Cai et al., 2024) with multiple parallel MLP heads, Hydra (Ankner et al., 2024) with sequential MLP heads, Eagle (Li et al., 2024a) and Eagle-2 (Li et al., 2024b) with sequential single-layer Transformer head. Eagle-2 shares the same model parameters as Eagle but distinguishes itself by incorporating a dynamic tree to generate candidates.
4. Experiments
In this section, we compare our proposed Gumiho with other SOTA methods to show the priority of our approach. Then, we conduct several ablation studies to validate the effectiveness of each part of our method.
In line with Eagle-2 (Li et al., 2024b), also conduct experiments with temperature settings of 0 and 1. A tem- perature of () means that the target L.M uses greedy sampling method, where the token with the highest proba- bility is selected at each position. In contrast, a temperature of increases the diversity of the output by applying post- processing to the logits at the current position, rather than directly selecting the token with the highest probability, When the temperature is set to 1, Eagle-2 excludes methods like Medusa since their relaxed acceptance criteria under non-greedy sampling do not ensure lossless acceleration. We follow their experimental settings in this work.
4.1. Experimental Setup
We conduct experiments using seven target LLMs: Vicuna-7B/13B (Chiang et al., 2023), Llama2-chat- 7B/13B/70B (Touvron et al., 2023). and Alaam33-instruee 8B/70B (Meta, 2024). Target LLMSs are fixed during train- ing, with only the draft heads being trained. Following Eagle and Eagle-2 (Li et al., 2024a;b), we train our draft model on the ShareGPT dataset. Our Gumiho model comprises Transformer model and five MLPs to predict the next seven draft tokens: the Transformer autoregressively generates the first two tokens, and the remaining five are predicted in parallel by the MLPs. Training details and hyperparameters can be found in Appendix C.
4.2. Performance Comparison
Experimental results are shown in Tab. 1. The Speedup metric quantifies the actual and-to-end acceleration ratio of token generation speed compared to vanilla uugo-regressive generation, while 7 is the average number of tokens accepted
We evaluate the performance across multiple benchmarks: MT-Bench (Zheng et al., 2023) for multi-turn dialogue, HumanEval (Chen et al., 2021) for code generation,
6

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Table 1. Speeduu ratios and mean accepted tokens (r) of different methods. V represents Vieuna, 1.2 represents IJ aMA2-Chat, and 1.3 represents Insttt-Inttruct We present the results of different methods across six datasets. Mean represents the average performance across these six datasets.
Model
Method
MT--Bench Speedup
HumanEval Speedup
GSMMK
Alpuca Speedup
CNN/DM Speedup
Natural Ques. Speedup
Mean
Speedup
Speedup
Temperature=0
Medusa
1.96x
2.50
2.15%
2.69
2,01x
2.59
1.94x
2.48
1.60%
2.02
1.68x
2.05
1.89%
2.39
Hydru
x
3.59
2,65x
3.78
2.49x
3.67
2,44x
3.58
1.92x
2.70
2.01x
2.86
2.33x
3.36
7H
Fagle
2.61 x
3.82
2.96x
4.20
2.67x
4,00
x
3.66
2.35x
3.34
2.1
3.13
2.52x
3.69
Eugle-2
x
5.00
3.27
5.35
2.93x
4.94
x
4.85
2.45%
4.11
2.24x
3.84
x.74x
4.68
Gumihooours)
3.15%
5.29
3.65x
5.77
3.10%
5.06
2.83%
4.87
2.73x
4.48
2.34%
3.88
x
4.89
Medusa
2.03x
2.58
2.24x
2.77
2,08x
2.64
2.04x
2.44
1667x
2.10
1.70x
2.10
1.96x
2.44
Hydra
x
3.65
2.88 x
3.86
x
3.67
x.65
3.49
2.08x
2.82
2.16x
2.86
2.52
3.39
V 13B
Eagle
2.87x
3,90
3,25x
4.29
2.88
3.90
2.64x
3.50
2.58x
3.49
2.21
2.92
2.74x
3.66
Eagle-2
3.16x
4,93
3.68x
5,42
3.19x
4.82
3.01 x
4.89
2.79%
4.27
X.41
3.69
3.04x
4.67
Gumihhrours)
3.36x
5.16
4.11x
5.97
x.39x
5.04
3.07x
4.88
2.91x
4.41
2.52
3.76
3.23%
4.87
Eagle
2.22x
3.00
2.53x
3.58
2.21x
3.09
2.04x
2.88
2.08x
2.78
1,88K
2.64
2.16x
3.00
L. 7B
E
2.91x
4.76
3.30x
5.38
2.87x
4.76
2
4.65
2.53x
4.10
2.52x
4.16
2.82x
4.64
Gumihhoours)
x
4.90
3.55x
5.60
3.00
4.81
2.85x
4.55
2.66x
4.18
2.59
4.16
2.95x
4.70
Eugle
2.59x
3.30
2,96x
3,90
2.61x
3:45
2
3,16
x
3.09
2.15x
2.82
2.52x
3.29
1.2 13B
Fagle-2
3.17x
4.76
3.78
5.53
3,23x
4.88
3.03%
4.62
2.84x
4.27
2.76x
4.12
3.13x
4.70
Gumihocours)
3.34x
4.98
4.05
5.87
3.35x
5.02
3.12%
4.66
2.93x
4.40
2.84x
4.20
3.27x
4.85
2
4.52
2.98
5.24
2.63x
4.63
2.48%
4.42
2,04x
3.72
2.14x
3.88
2.47%
4.40
L2 70B
Gun)
2.83x
4.71
3.35x
5,43
2.90
4.69
2.70x
4.46
2.37x
4.08
2.35x
3.90
2.76x
4.54
Eugle-2
2.16x
4.36
2.51x
5,06
2.22x
4.45
2.25x
4.88
1,82x
3.81
1.75x
3.54
2.12x
4.35
1.3 8B
Gumiho(ours)
2,38x
4.48
2.77
5.18
x
4.63
2.44%
4.88
2.00
3.94
1.93x
3.64
2.34x
4.46
Eagle-2
2.94
4.17
3.65x
5.09
3.17x
4.34
3.12x
4.74
2.54x
3.66
2.48x
3.50
2.98x
4.25
L3 70B
Guriho(ours)
3,38x
4.28
4.28x
5.25
3.79x
4.58
3.48
4.58
2.91x
3.80
2.87
3.59
3.45%
4.35
Temperature
Eagle-2
2
4.30
2.67x
4.52
2.46x
4.47
2.38%
4.37
2.13x
3.70
2.02x
3.50
2.37x
4.16
V 7B
Gumihorours)
x
4.42
2.84x
4.62
2.73x
4.52
2.46x
4.40
2.38x
3.94
2.10%
3.51
2.52%
4.23
V 13B
Eagle-2
2
4.37
3.32x
4.96
2,80x
4.43
2.66x
4.46
2.51x
3.92
2.25x
3.50
2.73x
4.27
Gumihooours)
2.93%
4.54
3.55x
5,30
2.84x
4.59
2.77
4.54
2.58x
4.04
2.36x
3.72
2.84x
4.46
Eagle-2
2.66x
4.63
2,95x
5.15
2.70x
4.76
2.52x
4.40
2.34x
3.98
2.29x
4.02
2.58x
4.49
L2 7B
Gumiho(ours)
x
4.64
3.19x
5.27
2.78x
4.67
2.64x
4.40
2.47
4.05
2.44x
4.08
2.72x
4.52
L2 13B
Gumihhoourr)
Hagle-2
3.18%
3.01x x
4.82
4.60
3.86x
3.58x
5.71
5.34
3.24x
3.09x
4.94
4.76
2.91 x
2.98%
4.49
4.62
2.80x
2.71x
4.28
4.15
2.66x
2.76x
4.08
4.16
x.99
3.14x
4.75
4.57
Eagle-2
2,28x
4.41
2.73x
5.15
2.42x
4.59
2,31x
4.30
1.87%
3.67
2,00x
3.72
2.27x
4.30
1.2 70B
Guniho(ours)
x.60x
4.65
3.15x
5.46
2.66x
4.61
2.50%
4.43
2.15x
3.98
2.22%
3.95
2.55x
4.51
Eagle-2
1.3xx
4.04
2.32x
4.80
2.06x
4.27
2.03x
4.57
1.67x
3.55
1.59x
3.27
1,93x
4.08
L3 8B
Gumihooours)
2.13x
4.14
2.55x
4.95
2.29
4.42
2.19%
4.55
1.86x
3.64
1.72x
3.32
2.12x
4.17
L3 70B
Eagle-2
2,85x
4.07
3,57x
4.97
3.13x
4.31
3.00x
4.65
2.47x
3,58
2.42x
3.45
2
4.17
Gumihooours)
3,29x
4.20
4.20
5.17
3.69x
4.49
3.34x
4.43
2.84%
3.71
2.85
3.57
3,37x
4.26
by the target LLM per draft round after verification.
the superiority of our approach. This is primarily attributed to the efficiency of the parallel ML. heads in Gumiho.
Across diverse target I. L.Mss, model sizes, and temperature settings, Gumiho demonstrates superior performance. Over- all, Gumiho surpasses the existing SOTA method EAGLE-2 by 4.5%~15,8% The performance gains are particularly pronounced with 70B model variants. At temperature 0, Gumiho outperforms EAGLE-2 by 11 11.7% on LLaMA2 70B and 15.8% on LLaMA3 70B. This substantial improvement primarily attributed to enhancements in 7 and reduction in draft time. Specifically, the output hidden state for 70B models has dimension of 8 8192, compared to 4096 in the 7B and 13B models. While the larger hidden state increases computational complexity, it also amplifies the benefits of our model parallelization, significantly reducing drafting time and further boosting the speedup ratio.
4.3. Ablation Studies
In this section, we evaluate the contribution of each compo- nent in our method to the overall performance. Specifically, we conduct ablation studies focusing on three key aspects: (I) The depth of the serial head (Transformer head), where vary the number of Transformer layers to assess its im- pact; (2) The width of parallel heads (MLP heads), where we experiment with different numbers of MLP heads; and (3) the effectiveness of full tree attention (FTA). These ab- lation studies aim to provide a deeper understanding of the architectural design choices in our proposed method and their respective contributions the final performance. We also present detailed comparison of draft head accuracy in Appendix D. Unless otherwise specified, the ablation exper- iments are conducted using Vicuna 7B as the target LLM, with MT-Bench as the test dataset and the temperature set to O.
In Fig. 3, we present the time required by different models for a draft round. From Eq. (4), it is evident that shorter draft time leads to a higher speedup ratio, resulting in im- proved performance. Fig. 3 demonstrates that the draft time of Gumiho is consistently shorter than that of Eagle-2 2 across all datasets regarding different temperatures, highlighting
7

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Sperdup
Speeddup
Mean Accepted Tokenss (1)
Speeduu
0.0
50
3.0
49
2.9
55
Head Depth
Figure 4. Ablation study serial head depth. The serial head is Transformer model, whose depth represents the number of layers within the Transformer architecture.
Figure 5. Ablation study on parallel head width. Parallel heads refer to the MLPs in Gumiho, and the width indicates the number of MLP models.
Serial head depth. The serial head depth refers the number of layers in the Transformer model. This Trans- former serves as the initial head responsible for generating the first two tokens in draft sequence. In this study, we vary the number of layers in the Transformer model to examine how the depth of the initial head affects the model's overall performance. The experimental results shown Fig. 4 re- veal that reducing the number of Transformer layers results in decline in т, which underscores the significant impact of the initial heads. However, when the depth increases from 2 to 3 layers, although 7 improves further, the speedup ratio decreases. This is because the overall speedup depends not only on r but also on the time required to complete a single draft process. Using three-layer Transformer substantially increases the drafting time, which ultimately reduces the speedup effect.
Table 2. Ablation study the FTA mechanism.
Speedup
w/o Full Tree Attention 3.10x
5.18
w/ Full Tree Attention
3.15x
5.29
to generate more accepted tokens. Additionally, since MLP heads operate in parallel, increasing their number does significantly increase runtime. A higher number of mean accepted tokens with similar runtime leads to an improved performance. However, this improvement does not scale in- definitely. All MLP heads share the same input embedding, which is derived from the concatenation outputs from the preceding Transformer head. During training, this shared embedding serves as the input to every MLP head and is simultaneously shaped by the back-propagated losses from all of them. As the number of MLP heads increases, the em- bedding must encode growing amount of information to meet the requirements of each additional head. This leads to excessive information being compressed into the limited em- bedding space, which reduces the clarity and specificity of the information available to each MLP head and ultimately causes performance degradation.
It is worth noting that using varied depths to generate the first two tokens, such as two-layer Transformer for the first token and a one-layer Transformer for the second, may seem intuitive but proves inefficient during inference. Since Transformers with different architectures cannot share their key-value (KV) caches, each head must compute its cache independently. This prevents cache reuse between heads, increasing the computational overhead. Our approach em- ploys identical serial heads throughout the model, only trains one single Transformer model, and reuses it for auto- regressive token generation during inference. This architec- tural uniformity enables efficient KV cache sharing across the entire generation process.
Effectiveness of Full Tree Attention (FTA), We verify the effectiveness of this component by conducting exper- iments with or without using FTA our model. The ex- perimental results are presented Tab. They indicate that removing FTA impacts the mean accepted tokens and diminishes the speedup effect.
Parallel Head Width. Parallel head width refers to the number of MLP heads Gumiho, which run in parallel to generate subsequent tokens in the draft sequence. The exper- imental results are shown in Fig. 5. Note that increasing the number of MLP heads initially improves performance but eventually leads to a decline. This because increasing the number parallel heads enhances the model's 'ss capacity to predict longer draft sequences, thereby improving its ability
Wall Clock Time of Different Components. To provide a granular understanding of the model's performance, we report the wall clock time of key components in the pipeline, as shown in Tab. 3.
8

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087744 2023.
Table 3. Ablation study on the wall time.
Components
Wall Time (ms)
Ankner, Z.., Parthasarathy, R., Nrusimha, A., Rinard, C., Organ-Keeleyy J., and Brandon, W. Hydra: Sequentially- dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109, 2024.
1st Serial Head
2.80
2nd Serial Head
3.46
Parallel Head
2.02
Full Tree Attention
3.41
Other Computation
6.11
Bercovich, A., Ronen, T., Abramovich, T., Ailon, N., Assaf, N., Dabbah, M., Galil, I,, Geifman, A., Geif- man, Y. Golan, I,, et al. Puzzle: Distillation-basel nas for inference-optimized Ilms. arXiv preprint arX1v:441.11916, 2024.
Verification
45.50
5. Conclusion
The core idea of this paper is to rigorously prove that the accuracy of early tokens in draft sequence is more critical than that of later ones in speculative decoding. In other words, given fixed budget for model parameter size and overall execution time, prioritizing the heads responsible for generating the initial tokens can improve overall per- formance. Building on this insight, we propose Gumiho, a novel approach that employs a hybrid head design. Specifi- cally, Gumiho allocates a larger proportion of the parameter and execution time budget to the head responsible for gener- ating the initial tokens. This head is a Transformer model with serial structure designed for these initial tokens, en- suring higher accuracy. For those heads that generate later tokens, Gumiho employs lightweight MLPs and parallelizes their execution to produce multiple tokens simultaneously. This hybrid design achieves improved performance by bal- ancing accuracy and efficiency: the serial Transformer en- hances the accuracy of the initial tokens, while the parallel MLPs reduce overall generation time. Experimental results validate the effectiveness of our approach, demonstrating that Gumiho surpasses existing statc-of-the-a methods.
Brown, O., Wang, Z.,, Do, A., Mathew, N., and Yu, C. Dy- namic depth decoding: Faster speculative decoding for llms. arXiv preprint 0rXvv2249.000142, 2024.
Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 2024.
Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXlv:2302.01318, 2023.
Chen, M., Tworek, J., Jun, H., Yuan, Q.. Pinto, O.D.O. Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L.,, Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-sourge chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
6. Limitation
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4/68, 2021.
Our proposed method, while achieving significant speedup, utilizes more parameter-heavy draft model compared to architectures like Eagle and Medusa. Specifically, the incor- poration of a two-layer Transformer head alongside five par- allel MLP heads, trained concurrently, results in increased GPU memory consumption during the training phase.
Donisch, L., Schacht, S., and Lanquillon, C. Inference optimizations for large language models: Effects, chal- lenges, and practical considerations. arXiv preprint arXiv:2408.03/. 003.0.0..33,, 2024.
Impact Statement
Du, C., Jiang. J., Yuanchen, X., Wu, J., Yu, S., Li, Y., Li, S.. Xu, K., Nie, L., Tu, Z., et al. Glide with cape: A low- hassle method to accelerate speculative decoding. The International Conference on Machine Learning, 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layer skip: Enabling early exit inference and self-speculative decoding. The Association for Computational Linguistics, 2024.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Fu, Y., Bailis, P., Stoica, I., and Zhang, H. Break the se- quential dependency of IIm inference using lookahead decoding, The International Conference Machine Learning, 2024.
Liu, F,, Tang, Y., Liu, Z., Ni, Y., Han, K., and Wang, Y. Kangaroo: Lossless self-speculative decoding via double early exiting. Conference Neural Information Process- ing Systems, 2024a.
Gale, T., Elsen, E.,, and Hooker, S. The state of sparsity in deep neural networks. arXiv preprint arXiv: 1rXvv:192.009574. 2019.
Liu, Y., Li, W., Cui, L., and Yang, H. Cerberus: Ef- ficient inference with adaptive parallel decoding and sequential knowledge enhancement. arXiv preprint arXiv:2410.13344 2024b.
Gloeckle, F., Idrissi, B. Y., Rozière, B., Lopez-Paz, D., and Synnaeve, G. Better & faster large language models via multi-token prediction. The International Conference on Machine Learning, 2024.
Meta. Llama3. http://gghubb.coomyyyoobbbbbbhhh 2024.
Nallapati, R., Zhou, B., Gulcehre, C., Xiang, B., et al. Ab-
stractive text summarization using rnns and beyond. arXiv preprint arXiv: 1602.0660200623,, 2016.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752..00 2023.
Sanh, V., Wolf, T., and Rush, A. Movement pruning: Adap- tive sparsity by fine-tuning. Advances in neural informa- tion processing systems, 13:20378-20333 2020.
Hinton, G. Distilling the knowledge in a neural network. arXiv preprint 1 2015.
Shen, S., Dong, Z., Ye, J., Ma. L.., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pp. 8815-8821, 2020.
Hubara, I., Courbariaux, M., Soudry, D., Elnini,,, R., and Bengio, Y. Quantized neural networks: Training neu- ral networks with low precision weights and activations. Journal of Machine Learning Research, 18(187):1-30, 2018.
Taori, R., Gulrajani. I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, B. Stanford alpaca: An instruction-folle followwin llama model, 2023.
Kim, S., Gholami, A., Yao, Z.., Mahoney, M. W., and Keutzer, K. I-bert: Integer-only bert quantization. In International conference on machine learning, pp. 5506- 5518. PMLR, 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint
Kurtic, E., Campos, D., Nguyen, T., Frantar, E., Kurtz, M., Fineran, B., Goin, M., and Alistarh, D. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. arXiv preprint 27X.222203.07259. 2022.
230. 2023.
Voita, E., Talbot, D.,, Moiseev, F., Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv: 1905.09416 2019.
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: benchmark for ques- tion answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
Wu, Z., Liu, Z., Lin, J., Lin, Y., and Han, S. Lite trans- former with long-short range attention. arXiv preprint arXiv:2004. 1886, 2020.
Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z. Speculative decoding: Exploiting speculative execu- tion for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3909-3925, 2023.
Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In Inter- national Conference on Machine Learning, pp. 19274- 19286. PMLR, 2023.
Xia, H., Yang, Z., Dong, Q.. Wang. Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint urXiv:2401.0785.. 2024.
Li, Y., Wei, F., Zhang, C., and Zhang, H. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint ar arXiv:2401.1507 2024a.
Li, Y., Wei, F. Zhang, C., and Zhang, H. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 16858, 2024b.
Xiao, Z., Zhang, H., Ge, T., Ouyang, S., Ordonez, V., and Yu, D. Parallelspec: Parallel drafter for efficient specula- tive decoding. arXiv preprint /0.00509,, 2024.
10

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Zadeh, A. H., Edo, I., Awad, O. M., and Moshovos, A. Gobo: Quantizing attention-based nip models for low la- tency and energy efficient inference. In 2020 53rd Annual IEEE/ACM International Symposium on Microarchitec- ture (MICRO), pp. 811 511-824. IEEE, 2020.
Zafrir, O., Boudoukh, G., Izsak, P., and Wasserblat, M. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NuurlPS Edition (EMC2-NIPS), pp. 36-39. IEEE, 2019.
Zhang, L.,, Wang. X., Huang, Y., and Xu, R. Learning har- monized representations for speculative sampling. arXiv preprint arXiv:2408. 15766, 2024.
Zhao, Y., Xie, Z., Liang, C., Zhuang, C., and Gu, J. Looka- head: An inference acceleration framework for large language model with lossless generation accuracy. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6344-6355, 2024.
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z. Li, Z., Li, D., Xing, E., et al. Judging alm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Zimmer, M.. Gritta, M., Lampouras, G., Ammar, H. B., and Wang, Mixture attentions for speculative decoding. 2024.
arXiv preprint

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
A. Detailed Proof of Theorem 3.1
Given {p: 1=1 and { defined in Eq. (5) and Eq. (6) the main paper, the mean accepted tokens (r) for the original and improved settings are expressed as:
D Σ
D
k
E| L original
Pori
( = Σ
p.).
(11)
k=1
k=1
and
D E
L improved
Pimple (L 2 k)
P.).
(12)
k=1
We aim to prove that:
B jiprowed
2 R
original-
(13)
We introduce auxiliary probability sequence P// that concentrates the scattered changes Çi at two adjacent positions d and
+ 1. Specifically, we define:
Pi + i=
d D
P
Pi S. i=d+1 s.1. < = otherwise,
Σ Σ
(14)
Pii
1=1
s=d+1
Here, we assume
Pd + S
(15)
We will discuss the cases of Pa + >> 1 at the end.
With these assumptions hold, the corresponding mean accepted tokens (r) this concentrated setting is:
D
D
L
ponsentrate
Σ
Poon L
2 k))
2 P.).
(16)
In the following, we will prove that:
E [.] concentrate
E T.]
original
(17)
EE [ Improved
E [ concentrate-
(18)
Before proceeding with the main proof, let examine the special case where P4+1 1. this case, the ordering constraint
1 12 P1 2 P2 2 Pd 2 Pa+1 implies that all probabilities equal: P1 P2
Pd Pa+1 1. Given that
P: P for all {1, 2, ... d]. we must have Çi Ç2 Ça = 0. This leads c Σí= D Ei=d+1 Gi meaning that [pr)__ and [ are exactly the same. As result., E E[L) iimprveed = EE []]rgiinnal-
For the remainder of the proof, we assume Pd+1
A.1. The proof of [] comcentrate
[[Loorignnal original
Define
D
AE
E L comcentrate
E L
original
Σ km
A A
(19)
12

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
where A II P.- II P; represents the contributions to the difference of the expected value for each position k.
Step 1: Analyze of AE.
We separate AE into three parts, where AE = Ek=ı A A = AE and AE3 = Ek=d+2 A
For S < d, Pk = Pk and A = 0. Therefore:
AE Ed
d
I[ P i=1
II pi
i=1
d-1
Pr
(pa + ()
Pi
d-1
P
(20)
1=1
For = + 1, we have:
A AE
d41
P
pi
1=1
i=1
d-1
d+1
Ps (pa + ()p++1 -C)
pi
d-1
II 2=1
Ps (CP4+1 - Spa (?)
C(pst1
Pa -a ()
II
Ps- i=1
(21)
For k > d+ 1. the difference arises from the terms (pa + 5) and (pe+1 5) in the product. Thus:
D
AE3
Σ
A
k=d+2
D
Pr
k=
2
V=1
1=1
D
d-1
Σ
(ps + (p4411-
II p:)
=
k=
1=d+2
D
d-1
k
P -pa P) II
pi)
(22)
k
imd+2
By combining Eq. (20) Eq. (22), we have:
AE = AE + AE₃ + AE
D
d-1
h II p.)
= =((pa+ - Pa + 1-
+
Σ
II
p.)Cppa Pa- -)(
(23)
i=1
k=d+2
i=l
i=d+2
13

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Step 2: Scaling.
Define A I Piς and recall that in Eq. (5) we have 1
P
P2
2 PD
N 0, then the total difference AE
becomes:
AE A(pa+1 Pa Pa+1-- 1c) Σ
A(pat1
Pa -
II
pc)
k=d+2
i=d+2
D
= A
1 + (Pd+1 PP -)) (pd+1 PP
Σ
II
Ps
k= i=d+2
D
(oa - Pa+1 + () - (nd Pd+1 + +
Σ
II
Pi
k= i=
2 A
1 - (ps Pa+1 + +) (pa Pd+1 +
Σ
Pat1
k-d-1
k=d+2
D
1 (pa- - -P+1 + () +
Σ Part
(24)
k=d+2
Step 3: Simplifying the sum.
Note that the last term Ekma+z Pa+l
d-1 in Eq. (24) is geometric series. Therefore, we have:
D
Σ
k-
+ Pd+1 p2+1 + Ph41 + + Pd+1
D-d-1
Pat1
kmd+2
D-d 1 1 Pa+1
(25)
- Pd+1
Substitute this back into AE, and we have:
AE > A 1 1 (ps pa-+1 + + 1 - D-d
Pdt1)
(26)
1 Dd+1
Step 4: Proving AE 2 0
To ensure AE 2 0, we need:
D-d Pat1
(pa P4+1 5
< 1,
Pd+1
(pa P441 +1
P < pd+1.
(pa + () - - (pa Pa+1 + () <
(27)
(ps Eq. (15), the inequality holds. Thus, AE 2 0. Implying:
Since pa 2 Pa+1 2 and 0, it follows that P (pa Pd+1 () 2 0. Besides, since have Pa + Ç S 1 in
2 B.L Joriginaa
(28)
A.2. Proving that [[[[Impooeed 2 E [ comcentrate
We introduce a series of auxiliary sequences with the goal of proving the following inequality chain. Note that the first line and the last line in the following inequality chain represent the improved setting and the concentrate setting, respectively.
14

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Note that only elements different in every consecutive lines (except for the last two lines). Specifically, merge into Ça time for S S d 1. 1.
Epp + Ga, P2 + C2,
Ps + Cs.
wy Pa + Cas Pat1 Cdt1,
Pd+2 - Sa+2, "", PD - (D)
E(pi, P2 + C2, Ps + S3.
".. Pa + Sa + Pat1 Sat1, Pa+2 Sa+2. "H, PD (D)
EpI.
P2.
Pa + Sa.
Pd + Sa + G1 (2, Pat1 Sdt1, Pd+2 - Sa+2: "", PD - (D)
IV IV
(29)
E(PI,
p2:
Pa. "", Pa Ei Si, Pa+1 Catt, D442 Sa+2, "", pn - (n)
E(I,
P2:
Ps. ",, Pa + Ei Gi Pd+1 Eb G, Pd+2, "", pD)
First Part of the Inequality Chain
We begin by proving the first inequality in the chain, which involves transferring Ç1 from P1 to Pa
E(p1 + Sı, P2 + Co, Ps + Co. ".. Pa + Cas Pat1 Sat1, Pa42 - Sa+2, "", PD - (p)
(30)
E(pi, p2 + C2, pa + C3, "., Pa + La + Çi, P4+1 Sati, Pd42 - Sa+2, "", pn (n)
15

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Let AB} denote the difference between the left-hand side (LHS) and the right-hand side (RHS) of the above inequality:
AE =B(pi + (1,22 (2, Ps + S3, Pd + Ca, Pa+1 Ca+1,Pa+2 Ca+2, ,DD (D) E(p1, + 62, Ps + 53, -.. .. .Pd + Sa + (1,Pa+1 - Sat1; - Ca+2. .PD (D)
=(p +1 p1) +
- -M Pı II (ps + +()
+
i==1
i=2
d II (p + (1) - p1 ==1
(d-1
(pa + + [p + 5i)
1+
Σ
(p. -i) so)
1=2
h=d+1
i=d+1
261
II (pt Gi) -PI Pı [I +G) (i) (pa ==1
(1a+
1+
Σ
II
(pt - 4i)
k=d+1
i=d+1
d-1
D
20
II
(p= + Si) [(m + Gdppa (a) - pi(pa + Ci + Ca)l
1 Σ
11 (pr G))
1=2
k=d+1
i=d+1
d-1
D E (p. -()
= +
II 2
(p. + 5iSs (pa + Sa pı)
1+
(31)
k=d+1
i=++1
If Pa + Sa - 2 0, then A 2 0. This directly implies that the first inequality holds.
Pd + Sa - P1 0, then
d-1
AF 2 II 1=2
(p. + G)G (p1 - Pa (a)
1+
Σ (Wft))
k=d+1
Po+1
D-d+1
=(1 II (p. + (i)S1 (p1 Pa Ca)
Pd-1
1
29 1 (p) - G()
1 Pd+1
1
(p) Pa - Sa)
Pd+1
1 Pd+1) (p1 - Pd - Sa)
14
1 Pa+1
(1 pI) + (pa + Sa - Pa+1)
-4
(32)
1 Pa+1
Given that P1 and Pa+1 S Pd 1, the numerator (1 - p1) + (pa + Sa Pa+1) and the denominator Pd+1 are both positive. Hence, AF} 2 O.
In both cases, A > 0, thereby proving Inequality 30.
Similarly, consider further transferring & from Pa to Pa:
E(pI, p2 + C2. Pa + C3. ".. Pa + Ca + St, Pat1 - S441, Pa+2 - Sa+2, "", PD - (D)
(33)
E(pi, p2, Pa + Sa, ..,, Pa + Sa + Ci + C2, P441 Sati, Pa42 - Sa+2, "", pn - (n)
16

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Apparently, this equals to prove:
pi + P1Bi(pp2 + 62, pa + Sa. "y, Pa Sa + S1, Pa+1 Sa+1, Pa42 Ca+2. "" PD (D)
(34)
p1 + Bp22 p3 + Sa, Pa + + Ci C2, Pa+1 Ca+1, P442 C442: "", pn (D)
which can be degenerated to Inequality 30 by removing Pı from both auxiliary sequences.
Using the same method, we can iteratively transfer each S, from p; to Pa for i = 1, 2. d, ensuring that each step maintains the inequality. Consequently, all inequalities in the chain (29) hold, except the last.
To conclude, after transferring all & from d, we arrive at the following:
E(p Si,
Pe + (2,
p3 + t3.
".. pat Sd. Pa+1 Set1,
Pd+2 Ca+2.
"", PD (D)
(35)
E(pI,
p2:
Pa
pa P
Ei= So pa+1 Cd+1, Pd+2 Cä+2; "". PD CD)
This completes the proof for the first part of the inequality chain.
Second Part of the Inequality Chain
We now address the final inequality in the chain (29), specifically:
E(pı,
P2:
Ps: ",, Pa Eas Si Pa+1 Sot1, Pd+2 Sat2, "", PD (D)
(36)
E(p,,
P2.
P3 "., Pa + E Si Pa+1 Eindes Si Pa+2, "., PD)
Let AE; denote the difference between the left-hand side and the right-hand side of the above inequality:
A Es = =E(pp, P2,33. ... ,P +
Pi,P S441,P042
Ca42,---,,P0 - (D)
D
E(pi. P2, P3. Pdd +
Gi,Pa41
Σ
i=1
i=d+1
d
d
D Σ
pi)
(pa
(i)
(p. G))
E P!) i=d+1
(37)
Lk=d+1
i=d-1
k=d+1
Note that the multiplicative coefficient (II=-) (pa + E Gi) is always positive, as probabilities are non-negative. Therefore, the sign of Fis depends solely on the later bracketed difference. We denote this difference A E.J:
17

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
AE Σ
II (pi -()) Σ II P()
k=d+1 i=d+1
k=d+1 i=d+1
C (pr - (())- - (pa+11 - + Σ II po) mi -S1
k=d+1 1=++1
k=d+2 i=d+2
D
k
D
&
= (po+1 - Sa+1) +
Σ
II (p: 51) (pi-
(pu+1 - -5) Σ E
pi) pi)
k=
i=d+1
k=d+2 i=d-2
k
II (p. (i) - II pi)(po1 -5)
(5 --5+1) Σ
k=d+2
i=d+1
i=d+2
= (5 - Ca+1) +
Σ
5(
pe)
+
II (ps - G)
II Pi i=d+1
k=d+2
i=d+2
d=++
D Σ
k II j=d+1
= (C - (a+1)+
+
C(
po)
+
Σ (-C)) i=++1
Pÿ) + Ca42, ,6k)
(38)
k=d+2
i=d+2
where in the last two terms of Eq. (38), we split the result of II (P: Gi) Il= into two terms. The first term represents the summation of all elements with only one G, and R2(Ga+1, Sa+2, (k) denotes the sum of all possible products involving two or more distinct S, terms from {Sa+1, Sa+2, ... , Now we want to prove that R2 (Ca+1. S4+2.- (R) 2 O. Since R2(Sa+1, Sa+2. ". (kk only exists when & 2 d + 2, we define it as follow:
h II i=+1
R2(Sa41,Sa4ar S042. (x) = II
(p. - G)
Σ i=d+1
II
Pj
k
d
(39)
i=d+1
j=d+1 jtt
Then, can calculate the partial derivative of the function Re(Ss+1, Sa+2,
Skk with respect to Sm:
d( d(R2(Gat1, (c)) II (=d+1
(p: ()) +
II Pj.
(40)
d(Sm)
j=d-1 jpm
spm
Given S Di G S P: S 1, we observe that d(R21Se41 (ee2..e)1 2 is always true, which implies that .k..
d(Gm) R2 (G Sdt2...., (b) is monotonically non-decreasing with respect to Sm for all m E {d + 1, d + 2....,k, Note that R2(Sa+1, Satii,,442...5566 (x) is differentiable on (0, 1) and continuous on [0. 1] with respect to all Sm. Therefore, R2 (Sa41. ...2....,Ca (b) attains its minimum value when Sm = for all m {d (d++1 1....,kk. Since R2 0, ...... = we conclude that R2(Ga41 (G( (e) 2
Then
AFA 2 (S Ca+1)
Σ
C( p.)
Σ
Gi
II
pj)
k=d+2
k=d+2
1=d+1
s=d+1 j+f
k G)
= =(S
- - ((+1)
Σ
pd)
Σ
4(
II
pj)
(41)
k=d+2
b=d+1
1=d+2
i=d+1
f=d+1
jæi
18

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
When D d 2:
d+2
& Σ imd+1
Au 6441) +
Σ kmd+2
III p.)s
c( II Pj)
imd+2
j=d+1 f
Ca+2 + Pu+25 Cdt1Pa+2 Cd+2Pa+1
Sa+2(1 Pd+1) (S Sd411)paz2
2 (S - 1d+11
(42)
Since 0 S Ps & 1 and 0 2 Ç E 1, it is clear that A AES 2
Similarly, When D d 3:
d+3
d+3
d+33
A AET
3
E +
II
P.)S
Σ
5(
pj)
i=++
j=d+1 jfi
2 (< (a+1)pa-s + (p4426643 1 IPd-2P643 (442P+1P444 (443d-1P6442)
(Sa+2 (atz)Pa+2 (pa+2Po+3 - Cd+1P442Pa4s (4+2Pt+1Pu+3 S4t3Pa-1P44)
Sa+2(Pa42 - Pd+1P)43) + C443(Pat2 Pat1Pa+2) + -5- Pdt11
2 Sd+2(Po+2 P2+1P442) + C5+3(Pa42 Pa+1Pa42) + (C (G--G+1))442P00
2 (5 - (u+1)Po-2P643
N
(43)
Using induction, assume that for any D d = k (2) AE (S Ga+1) Ili=42 pi 2 0.
Then for D - d= k
d+b
d+i+1
d+k+1
d+k+1
AETh k-11
2 25 Sa+1) II p
II
P.C
Σ
5( II p,))
i=++2
i=d+2
d=
j=d+1
d+b
d+k+1
d+k+1
d++++1
d+k+1
= -s - Ca+1)
II i=d+2
+ +
<( II
p.) - C II
p,) C442(
II j=d+1
S046+1(
II j=d+1
p.)
i=d+2
j=d+2
o/d+2
f4d+k+1
d+
= (Ga42 + C443 +... (14k+1)
II pit
i=d+2
d4+++1
d+++1
d+-k+1
d+k++1
<( II pt) Sat1( (
II p3)
C442(
II j=d+1
Sotk+i(
II pJ)
imd+2
j=d+2
j=d+1 1dd+k41
3
d+-k
d+1+1
d+*
d++k+1
d+-
d+k+1
Sat2
II ps-
II j=d+1
Pj
+ C443
II
II
Pj
+ + Sd-k+1
Pi
Pj
k=
i=d+2
j=+++1
i=d+2
j=d+1
34d42
3-f-4433
1#d+k41
d++-k++-1
+ (S 14+1)
II #=d+2
Pi-
(44)
19

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
For the first term in Eq. (44), observe:
d++++1
II i=d+2
II Pj
II Pi
II f=d+1
Pi
Pj
Pd+&+1
j=d+1
i=d+2
j7d+2
j/d+2
d++
d-
II j=d+1
Pi
P5
Pa42
i=d+2
j7d42
d++k
d-k
II j=d+1
Pr
Pj
i=d+2
(1 - Pd41)
II d+2
Ps
2 0.
(45)
By the same reasoning, we can show that all coefficients in Eq. (44) are non-negative, and we have:
d4+-A-+-11
AEJ AEtkti]
2 (5 - Gd+1)
II
i=d-2
0.
(46)
Therefore, we can conclude that A Fi 2 and thus:
A Fg 2.
(47)
Conclusion:
Since:
AE 0, AE; 0.
(48)
conclude that the incquality chain (29) holds, which means:
E L Improved
EE concentrate-
(49)
A.3. Proving that IE [mpproved
2 E L] original
Given that:
L Improved
2 E
concentrate
E L concentrate
E(L]
Jorignnaall
(50)
establish that:
E E[L Jimpooedd
2
[|oooriginal- original-
E
(51)
A.4. Cases of pa + Ç
At the beginning of the proof, we assume Pa + S 1. In cases where Pa + + 1, the parameter can be distributed across multiple positions to satisfy the constraints. Specifically, we divide Ç into parts:
20

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
x= Ea + Ča-1 + Ea-2 + ... + Ea-nt1,
(52)
where
Pd + Sa 1,
(53)
Pa-1 + Sd-1 1,
(54)
(55)
Pd-n+2 + Gd-ntz 1,
(56)
Po-n+1 + Sd-n+1 1.
(57)
These adjustments are applied to n positions as follows:
Pt + Gis i= i=d n
d
d Σ
d
P
Pi S. 1+1
s.t. 5
Ci
Σ
Σ G.
(58)
Pi. otherwise,
1=1
i=d+1
i=d-n+1
Next, construct the following inequality chain:
B(pr,
Pa.
Ps.
"", Pd.
Pa+1,
Dd42s "", pD)
E(pi,
Pay
Ps.
Pa + Ea. pat1 - Sas Pa+2:
PD)
E(pl:
p2.
pa-1 + Ea-1, pa + Cas pat1 - Ee - -C--1, P4+2. "., PD)
(59)
E(p,,... Pa-n+1 + Ča-ati,
Pa-1 + Ea-1, pa Eo. pa+1 Ei &o ""+ .PD)
Each step in the inequality chain (59) can be proven using the same method as proving Next, we construct another inequality:
E L original-
E(p Pa-nt1 + Ea-nti, "*, pa-1 + Ea-i, pa Eas pat1 nd Eia-aat &o P442,'' ppp)
(60)
B(p..... Pa-n+1 + Ea-nti; Pa-1 + Ea-1, pa Ea, Pat1 - Cati, Po42 Sa42, .PD (D)
The inequality (60) can be proven using the same method proving inequality (36).
21

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Finally, construct third inequality:
B Pa-n+1 + Ea-nti, t., Pa-1 + Ča-1, pa + Ea, Pa+1 - Cet1, P442 "4
,DD (D)
(61)
E(pr + Ca, D2 + 62. "", Pa + Cds pa+1 Sati, Pd42 C442s
PD - (D)
To prove inequality (61), can follow the same method proving the first to the second-to-las line in inequality (29). In that proof, we observed that each step involved moving non-negative value Çi from an earlier position to later position, and the validity of the inequality was independent of the specific value of Çs or the positions involved. In the case of inequality (61), we have ",, Ça 2 0 and Sd-n+1,"" ... Sa S Čd Čd-n+1, ... Sa. This means that we are essentially moving positive values from earlier positions to later positions, similar to the process in inequality (29). Therefore, we can use same method establish inequality (61).
With inequalities (61), (60) and the inequality chain (59) established, conclude that E [ jmpreveed [[ E[[ Therefore,
we finish the proof of Theorem 3.1 in the main paper.
22

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
B. Experiment Results A100
This section gives the inference results on a single NVIDIA A GPU. It reasonable that there are different final speedups on MI250 and A100. However, note that the mean average tokens (r) are also slightly different. This is because of the use of FP16 precision during inference, and different FP16 processing mechanisms MI250 and A100 GPUs. Using FP32 precision yields identical values across both platforms, yet significantly increases the inference latency.
Table Speedup ratios and mean accepted tokens (r) of different methods on NVIDIA A represents Vicuna, L2 represents LLaMA2-Chat, and L3 represents LLaMA3-Instun We present the results of different methods across six datascts. Mean represents the average performance across these six datascts.
MT-
Humankival Speedup
GSMMKK
Alpaca
CNN/DM
Natural Ques. Speedup
Mean
Model
Method
Speedup
Speedup
Speedup
Speedup
Speedup
Temperature
Eugle-2
3.30x
5.03
3.59
5.36
3.21x
4.94
3.00x
4.86
2.57
4.10
2,48x
3.82
3.02
4,69
V 7B
Gumihooours)
3.62x
5.22
4
5.81
3.48x
5,07
3.13x
4.82
2.91
4.46
2.60
3.80
3.33x
4,86
Eagle-2
3
4.92
3.60%
5.42
3.24x
4.79
2.99%
4.90
2.47%
4.21
2.48x
3.71
2.98x
4,66
V 13B
Gumiho(ours)
3,33x
5.23
4.12%
6,01
3.35x
5,08
3.07%
4.97
2.68x
4.40
2.52
3.81
3.19x
4.92
E
3,28x
4.76
3.71x
5.38
3.28
4.77
3.17x
4.66
2.61x
4.09
2.78%
4.16
3.14x
4.64
Gumiho(ours)
3,44x
4.92
3
5.57
3,42x
4.83
3,19x
4.54
2.82x
4.20
2.91
4.19
3.28
4.71
E
x
4.77
3.60x
5.53
3.40x
4.89
3.10x
4.60
2.54x
4.26
2.77
4.12
3.07x
4.69
13B
Gumihooours)
3.24x
4.97
3.78
5.85
3.61
5,03
3.19x
4,63
2.72
4.38
2.85
4.20
3.23x
4.84
Eugle-2
x x
4.35
3.20%
5,06
2.79x
4.47
2.78x
4.87
2.24x
3.81
2.26x
3.53
2.67x
4.35
Gumiho(ours)
2.95x
4.48
3.62x
5.22
3.20
4.62
2.86x
4.88
2.39
3.90
2.41 x
3.62
2.91
4.45
Temperature=
7B
Eagle-2
2.73x
4.32
2.99x
4.65
2.59x
4.41
2.55x
4.25
2.28
3.87
2.21 x
3.52
x.56x
4.17
Gumihh(ours)
3.00
4.35
3.30x
4.82
2.98
4.54
2.61x
4.22
2.48
3.94
2.29 x
3.63
2.78%
4.25
Eugle-2
2.73x
4.35
3.14x
4.85
2.81x
4.54
2.75
4.57
2.34x
4.01
2.26x
3.53
2.68
4,31
V 13B
Gumiho(ours)
2.92
4.56
3.46x
5.31
2.95x
4.63
2.89
4.67
2.42x
4.05
2.43x
3.70
2.85
4.49
Fagle-2
x.94x
4.53
3.32x
5.10
3.01x
4.69
2.80x
4.61
2.50%
3.92
2.60Ix
4.04
2.86x
4.48
7B
Guuihhoours)
3.02x
4.65
3.41x
5.31
3.12x
4.74
2.97x
4.43
2.65x
4.07
2.83x
4.12
3.00
4.55
Eagle-2
2.80x
4.61
3.41x
5.37
3.19x
4.75
2.93x
4.50
2.41x
4.15
2.62x
4.06
2.90x
4.57
L2 13B
Gumihooours)
2
4.85
3.71
5.72
3.33x
4.88
2,99
4.50
2.54x
4.31
2.71x
4.12
3.04x
Eagle-2
2.34x
3.92
2.82
4.81
2.61x
4.36
2.53x
4.50
2.04x
2.02x
3.36
2.39x
4,08
L3 8B
Gumiho(ours)
3.13x
2.81
2.67
4.54
2.19x
2.57
23

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
C. Training Details and Hyper-parameters
Similar to Eagle-2 (Li et al., 2024b), we employ both the regression loss Ereg and the classification loss Cote train draft model. The total loss defined as weighted combination of these two components:
C Wreg" Creg
Wels
Loiss
where Wrey and Wale denote the weighting coefficients for the regression loss Creg and the classification loss Ceis- respectively.
Similar to EAGLE-2's tree attention, we select the top 10 output tokens from each Transformer head input for the subsequent head, i.e., topk=10. For FTA, we extract the top 35 output tokens from each MLP head, i.e., s=35. Hyper- parameters can be found in Tab.
Table 5. Hyper parameter configurations of Gumiho.
Vicuna 7B/13B
LLaMA2 70B
Hyper-parameters
LLaMA2 7B/13B
LLaMA3 70B
LLaMA3 8B
Learning rate
2c-4
lc-4
Transformer layer number
MLP head number
5
Batch size
4
Wels
0.1
Wreg
Training epoch
10
Optimizer
AdamW
(B1,82)
(0.9, 0.95)
Per MLP structure
[2*hiddensstate, #hidden_state]* + [1hidden_state, $
topk
10
S
35
24

---

Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
D. Ablation Study on Head Accuracy
MT-Bench
0.9
0.8
Accuracy
0.7
0.6
0.5
0.4
Gumiho
0.3
Eagle-2
GSMBK
0.9
0.8
Accuracy
0.7
0.6
0.5
1st Head
2nd Head
3rd Head
4th Head
5th Head
6th Head
Figure 6. Comparison of draft head accuracy on two datasets (MT-Bench and GSM8K). Both results are based on Vicuna with the temperature set to O.
We conducted comparative analysis of head wise accuracy between method and EAGLE-2 based on Vieuna 7B with temperature set to 0 on MT-Bench and GSM8K datasets. It should be noted that we have total of seven draft heads, while EAGLE-2 only has six heads. Therefore, to facilitate comparison, we have only conducted an accuracy comparison between the first six heads of ours and the six heads of EAGLE-2. As illustrated in Fig. 6, our approach enhances the accuracy of front heads, which are responsible for generating the initial tokens in the draft sequence. The precision of these early tokens substantially impacts the final mean accepted tokens(7). Our back heads employ a parallel MLP architecture, resulting in lower accuracy compared to EAGLE-2. This accuracy distribution aligns with our theoretical findings. Our theorem demonstrates that optimizing the accuracy distribution across heads, specifically through enhancing precision in front heads while proportionally reducing accuracy in back heads, leads to better overall mean accepted tokens(T).
25


Reviews:
[{"review_id": "7sCpKDYObZ", "reviewer": "Reviewer_xLKt", "summary": "This paper proposes Gumiho, a hybrid architecture to prioritize early draft tokens with large and autoregressive heads, and parallel decoding for the rest tokens. The experimental results strongly support the effectiveness of this method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions and support.\n\n**Q1. The paper lacks a direct comparison with a strong baseline using flashdecoding.**\n\nA1: We have evaluated the baseline with FlashDecoding (FD) on the MT-Bench dataset using Mi250 GPU, as shown in the table below. \n\n||L2 7B|L2 13B|\n|-|-|-|\n|FlashDecoding|2.11×|2.12×|\n|Gumiho|**3.07×**|**3.34×**|\n\nIt is important to clarify that FD is an algorithm specifically optimized for attention computation. Its acceleration mechanism for LLMs is orthogonal to SpD – meaning these two approaches can be combined to achieve further acceleration beyond standalone SpD or FD implementations. The results of the combination will be shown in the final version of our paper due to the limited time of rebuttal.\n\n**Q2. The authors seem to introduce two different draft models (one large and one small), which could result in additional memory usage.**\n\nA2: Yes, you are correct. Our method does require more memory usage compared to Eagle2, and Medusa. Here’s why:\n- Eagle2 use one single-layer Transformer head and reuse it serially across multiple speculative steps.\n- Medusa only maintains five parallel MLP heads.\n- Our approach, however, employs a two-layer Transformer head alongside five parallel MLP heads.\n\nThis architectural choice increases memory consumption. However, it enables a hybrid generation strategy: serial decoding for the first few tokens (to ensure accuracy) and parallel decoding for subsequent tokens (to maximize speed). While this introduces a memory-accuracy trade-off, it achieves a significant speedup improvement. We will explicitly discuss this trade-off in the final manuscript.\n\n**Q3. For the early draft model, using an approach similar to Eagle introduces an inconsistency between training and testing. I believe this issue becomes more pronounced as the network depth increases, as shown in Figure 4. Could the hass training method help mitigate this problem?**\n\nA3: Yes, our method could integrate Hass’s training strategy for further gains. Hass improves Eagle2 by addressing the training-inference inconsistency in serial transformer heads within SPD. While Hass focuses on training methodology, our work enhances performance via architectural modifications. These two directions are orthogonal, so our method could combine with Hass for further improvements.\n\nIn the table below, we compare the performance of our Gumiho with Hass. Additionally, we evaluate a variant of our model (denoted as Gumiho w/ Hass ) where Hass’s training method is applied to our Transformer Head. All results are conducted on an Mi250 GPU using the LLaMA3-8B model with a temperature of 0.\n\n||MT-Bench|HumanEval| \n|-|-|-|\n|Eagle2|2.16×|2.51×|\n|Hass|2.26×|2.59×|\n|Gumiho|2.38×|2.77×|\n|Gumiho w/ Hass|**2.43x**|**2.84x**|\n\nThe aforementioned discussions will be included in the final version of our paper.\n\n**Q4. throughput improvement.**\n\nA4: We evaluated Gumiho under bs > 1 scenarios with Vicuna 7B, as shown in the table below. Experimental results demonstrate that the speedup effect degrades as batch size increases, aligning with observations in prior works like Eagle. Throughput speedup of baseline and Gumiho is computed at their respective maximum bs, as Eagle do.\n\n|BatchSize | MT-Bench | HumanEval | \n|-|-|-|\n| 1| 3.15×| 3.65×|\n| 2| 3.10×| 3.61×|\n| 4| 2.87×| 3.34×|\n| Throughput| 2.03x | 2.11x |\n\nThe degradation in speedup with larger batch sizes can be primarily attributed to two key factors:\n\n**(1) Inter-Sequence Execution Time Imbalance.**\nWith larger batch sizes, the number of accepted tokens varies across sequences within a batch, leading to divergent completion times. Due to the \"bucket effect\" in batch processing — where the entire batch's completion time is determined by the slowest sequence — completed sequences must wait for unfinished ones. This waiting time exhibits growth as batch size increases.\nNotably, directly removing completed sequences and inserting new ones theoretically mitigates this issue but proves impractical in reality. Newly inserted sequences require full context re-computation (prefill phase), whereas existing sequences leverage pre-cached KV for efficient decoding. This computational asymmetry further increases overall latency.\n\n**(2) Computational Resource Contention.**\nLarge batch sizes push SpD into a computation-bound regime. During parallel verification, the target model must process a massive number of candidate tokens per step (quantity = draft tokens × batch size). Larger batch sizes prevent full parallelization of computing units, causing the verification process to degrade into partially sequential operations. This diminishes the theoretical acceleration benefits.\n\nGiven these constraints, most SpD solutions (e.g., Lookahead, Medusa, Hydra, Cerberus, Eagle2, Hass) primarily focus on optimizing single-sequence (bs=1) scenario. Our work follows them, prioritizing inference efficiency optimization for single-sequence processing.", "claims_and_evidence": "The experimental results strongly support the effectiveness of this method. Not only does it achieve a speedup in wall time, it also increases the acceptance rate—validating the authors’ approach from two different angles.", "methods_and_evaluation_criteria": "Yes, LLM acceleration is a very important topic for current applications. However, the authors should test over more advanced settings like flash decoding and bsz > 1.", "theoretical_claims": "I checked the correctness of the theory and it is correct.", "experimental_designs_or_analyses": "Yes.\n\nBut I think the authors should include more experimental results compared to the results of the implementation with flash decoding, i.e., flash_attn_with_kvcache and bsz > 1.", "relation_to_broader_scientific_literature": "na.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Advantages:\n1. The writing in this article is very clear.\n2. I think the idea presented here makes a lot of sense. Intuitively, because the model is autoregressive, the earlier tokens are definitely more important than the later ones. Moreover, the proposed solution is quite elegant.\n3. The experimental results strongly support the effectiveness of this method. Not only does it achieve a speedup in wall time, it also increases the acceptance rate—validating the authors’ approach from two different angles.\n4. Balanced Performance and Latency: Gumiho’s design leverages a serial Transformer head to boost the quality of early token predictions without significantly sacrificing speed. The authors validate this balance through ablation studies (Figure 4), showing that the performance gains outweigh the latency introduced by the serial component.\n\nDisadvantages:\n1. The paper lacks a direct comparison with a strong baseline using flashdecoding.\n2. The authors seem to introduce two different draft models (one large and one small), which could result in additional memory usage.\n3. For the early draft model, using an approach similar to Eagle introduces an inconsistency between training and testing. I believe this issue becomes more pronounced as the network depth increases, as shown in Figure 4. Could the hass training method help mitigate this problem?\n4. throughput improvement.\n\nIf the authors report results on 1 & 4, I will increase score from 2 to 3.", "other_comments_or_suggestions": "NA.", "questions_for_authors": "see cons", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tYX7SSWLxL", "reviewer": "Reviewer_E4Fx", "summary": "This paper improves the self-speculative decoding methods by combining the architecture of Eagle and Medusa. The paper uses sophisticated Transformer architecture for the early draft heads in a serial configuration to improve accuracy, and multiple lightweight MLP heads operating in parallel to enhance efficiency. The reason behind this combination is that the prior tokens in speculative decoding are more important. The paper gives a theoretical analysis of this idea. The experimental results demonstrate the effectiveness of this paper.\n\n## update after rebuttal\nI will keep my ratings since the rebuttal solve most of my concerns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions and support.\n\n**Q1. It is not very clear why full tree attention works better than the tree attention methods in Eagle2, especially why this method cannot be applied in Eagle2. The authors should give an example to illustrate this better.**\n\nA1: \n\n**(1) Why FTA cannot be applied to Eagle2:**\n\nLet us refer to Fig. 2 for illustration. In the upper half of Fig. 2, the tokens [shines, glows, radiates] are outputs of Head 3, while [warmly, brightly, intensely] are outputs of Head 4. In Eagle2, the heads operate serially—both Head 3 and Head 4 are single-layer Transformers, and the output of Head 4 depends on the output of Head 3. In our method, however, these heads operate in parallel (i.e., Head 3 and Head 4 correspond to MLP1 and MLP2 in Fig. 1). Since the outputs of MLP2 are independent of MLP1, the outputs of MLP1 and MLP2 can be freely combined.\nIn the lower half of Fig. 2, FTA leverages the independence between outputs of later heads to fill shorter paths in the tree with longer ones. In contrast, Eagle2’s Tree Attention (TA) requires strict dependency between tokens along each path (later tokens depend on earlier ones), making cross-path filling impossible.\n\n**(2) Why FTA outperforms TA:**\n\nThe final speedup improvement is achieved by either increasing the mean accepted tokens or reducing draft time. Compared to TA, FTA increases the mean accepted tokens by extending candidate path lengths without incurring additional computational overhead, thereby maintaining the same draft time as TA. This allows FTA to achieve better performance than TA.\n\n**Q2. The experimental setup on A100 should be further claimed.**\n\nA2: The requirement of 4 GPUs applies exclusively to the scenario that the parameter size of target model is 70B. We clarify that our experiments on the NVIDIA A100 GPU did not include the 70B variant. All A100-based evaluations were conducted with 7B/8B/13B-scale Target Models.\n\n**Q3. Compare and discuss Cerberus[1], DDD[2], Hass[3].**\n\nA3:\n\n**(1) Cerberus [1]:**\n\nThe observation in Cerberus that earlier heads exhibit higher accuracy aligns with our understanding. This is because, in parallel decoding, later heads predict tokens farther from the input (e.g., predicting the $i+2$-th token based on the $i$-th hidden state introduces greater error compared to predicting the $i+1$-th token). However, Cerberus still employs identical parameter scales and architectures across all heads, failing to prioritize earlier heads—a key distinction from our method. Their primary contribution lies in introducing sequential knowledge for parallel heads, whereas our approach explicitly optimizes model structure to prioritize early tokens.\n\n**(2) DDD [2]:**\n\nDDD improves Eagle2 by using a dynamic draft tree. Since DDD’s code is not open-sourced, we compare relative improvements over Eagle2. On the MT-Bench dataset, the performance gains of DDD and Gumiho are:\n\n|model|DDD|Gumiho|\n|-|-|-|\n|V 7B|3.5%|**9.4%**|\n|V 13B|4.2%|**6.3%**|\n|L2 7B|2.3%|**5.5%**|\n|L2 13B|2.3%|**5.4%**|\n\nOur method consistently outperforms DDD across all tested models, demonstrating our effectiveness. \n\n**(3) Hass [3]:**\n\nHass improves Eagle2 by addressing the training-inference inconsistency in serial transformer heads within SpD. While Hass focuses on training methodology, our work enhances performance via architectural modifications. These two directions are orthogonal: our method could integrate Hass’s training improvements for further gains. \n\n\nIn the table below, we compare the performance of our Gumiho with Hass. Additionally, we evaluate a variant of our model (denoted as Gumiho w/ Hass ) where Hass’s training method is applied to our Transformer Head. All results are conducted on an Mi250 GPU using the LLaMA3-8B model with a temperature of 0.\n\n| | MT-Bench | HumanEval | \n|----|----|---|\n| Eagle2 | 2.16×| 2.51×|\n| Hass| 2.26×| 2.59×|\n| Gumiho| 2.38×| 2.77×|\n| Gumiho w/ Hass| **2.43×**| **2.84×**|\n\nThe aforementioned discussions will be included in the final version of our paper.\n\n**Q4. Why $p_i - \\zeta < 0$ ($i=d+1$) is not assumed and discussed accordingly.**\n\nA4: Your concern is intuitively reasonable. From the perspective of probability definitions, it would indeed seem necessary to assume that probabilities should be greater than 0, even if this auxiliary variable is not a real probability but merely a constructed parameter for our proof. \n\nHowever, throughout the proof process, we observed that such a constraint was unnecessary, and the proof could be completed without invoking this additional condition. Therefore, to maintain conciseness, we did not explicitly discuss this case separately.\n\n**Q5. In Appendix D, the authors show the comparison of Eagle2 and Gumiho on two datasets. What about the other four datasets?**\n\nA5: The other four datasets also exhibit similar results, and we will include the comparisons of these datasets in the final version of the manuscript.", "claims_and_evidence": "The proposed method is evaluated on multiple architectures and settings, including experiments on Vicuna, LLaMA2 and LLaMA3. It is also compared against other similar competitor methods. The results demonstrate the effectiveness and efficiency of the proposed approach.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are reasonable and follow widely accepted evaluation standards. The benchmark datasets used are also standard.", "theoretical_claims": "I have reviewed the mathematical formulas in the paper and did not identify any issues.", "experimental_designs_or_analyses": "I have reviewed the experimental design in the paper, including experiments on multiple architectures and settings and the ablation study. In the ablation study, the authors explore FTA but show marginal improvement.", "relation_to_broader_scientific_literature": "The conclusion that tokens are more important in the previous part of the sequence in SpD is not surprising.\n\nHowever, this paper is still meaningful to society since it discusses the situation of SpD under a limited computational budget. Based on rigorously proven theory, it gives a useful solution to this situation.", "essential_references_not_discussed": "There are some discussions of the accuracies of different heads in paper [1], which shows that previous heads have higher accuracy, and I think the author should have a discussion in their paper.\n\n[1] Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement.", "other_strengths_and_weaknesses": "Strengths:\n\n1. I believe that this paper is meaningful to the research area of speculative decoding. The paper rigorously proves that improving the performance of the previous draft models with the expense of the later models is beneficial to the overall mean-accept-tokens of SpD.\n\n2. The paper gives a nice hybrid architecture solution that uses sequential models to improve the previous tokens’ performance and uses parallel models to accelerate the later ones.\n\n3. The paper is theoretically sound and easy to understand. The proof of the theorem is roughly correct.\n\n4. The experimental results show that this paper can achieve a new state-of-the-art speedup compared to the previous methods.\n\nWeaknesses:\n\n1. It is not very clear why full tree attention works better than the tree attention methods in Eagle2, especially why this method cannot be applied in Eagle2. The authors should give an example to illustrate this better.\n\n2. In lines 320 to 324. The authors claim that they use 1 MI250 GPU for evaluation except the 70B variant which requires 4 GPUs. However, you only use 1 NVIDIA A100 GPU for all models. The experimental setup on A100 should be further claimed.\n\n3. There are some discussions of the accuracies of different heads in paper [1] which shows that previous heads have higher accuracy, and I think the author should have a discussion in their paper. There are several papers that proposed better results than Eagle2, such as [2], [3]. The author should compare and discuss them.\n\n4. In the proof of theorem 3.1, the authors use an auxiliary probability sequence $P_i’$, and assume $p_d+\\zeta \\leq 1$ and discuss $p_d+\\zeta>1$ at the end. I am confused why $p_i-\\zeta < 0$ is not assumed and discussed accordingly. \n\n5. In appendix D, the authors show the comparison of Eagle2 and Gumiho on two datasets. What about the other four datasets?\n\n[1] Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement.\n\n[2] Dynamic Depth Decoding: Faster Speculative Decoding for LLMs.\n\n[3] Learning Harmonized Representations for Speculative Sampling.", "other_comments_or_suggestions": "See weaknesses above.", "questions_for_authors": "My major concerns are weaknesses 1, 3 and 4. The answers would likely change my evaluation of the paper.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u6Dm7dwHe2", "reviewer": "Reviewer_w5FY", "summary": "This paper proposes a new speculative decoding method called Gumiho. It combines the parallel draft head architecture and sequential draft head architecture to derive a hybrid architecture. The idea behind the paper is to prioritize the accuracy of the early tokens and make a rigorous mathematical proof to show that this can enhance the overall performance. The experiments on six datasets with different baseline models and temperatures show that Gumiho can achieve new state-of-the-art results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions and support.\n\n**Q1. The ablation study shows that using FTA has marginal improvement.**\n\nA1: The improvement contributed by our proposed FTA method is non-negligible in the context of the overall performance gain. In the FTA ablation study, the speedup ratio increases by 0.05, while the total improvement of our method over Eagle2 is 0.27. This means FTA accounts for approximately 18.5% of the total performance gain, thereby demonstrating its critical role in our approach.\n\nMoreover, FTA introduces a lossless improvement compared to existing TA  without incurring additional computational overhead. It only requires modifying the Tree Mask during verification by setting specific positions from 0 to 1 (i.e., unmasking certain tokens). This design ensures efficiency while enhancing performance.\n\n**Q2. The conclusion of the theorem is simple but it seems that the proof is rather complicated. Is there an easier way to give the proof?**\n\nA2: Note that although the conclusion of the theorem appears simple, the situation in reality is rather complicated since we have only imposed very few restrictions on the Theorem. Specifically, we only require that the sum of the increments in the former part and the decrements in the latter part must be equal. The probabilities $\\lbrace p_i\\rbrace_{i=1}^D$ in the original setting, the relative magnitude of increases and decreases $\\zeta_i$, the length of the original sequence $D$, and the divided location $d$ of the former part and latter part are all unbounded and can change freely.\n\nTherefore, although the theorem itself appears simple, the high degree of freedom results in highly complex scenarios requiring exhaustive case analysis. Under the current assumptions adopted in our paper, each step in the proof is indeed necessary.\n\n\n**Q3. The theorem assumes that the total amount of decrease to the second part is the same as that of increase to the first part of the sequence. However, this is hardly the case in reality. Is it able to give a more general assumption?**\n\nA3: Actually, the assumption that *\"the increment in the former part equals the decrement in the latter part\"* can be interpreted as an upper-bound condition. In fact, our conclusion holds not only when these quantities are equal, but also when the increment in the former part exceeds the decrement in the latter part, in which the Eq.6 can be modified from \n\n$\\sum_{i=1}^d \\zeta_i=\\sum_{j=d+1}^D \\zeta_j$ \n\nto \n\n$\\sum_{i=1}^d \\zeta_i\\geq\\sum_{j=d+1}^D \\zeta_j$.\n\nThis is very easy to prove that Theorem 3.1 still holds in this situation if we remove the excess portion in the former part without decreasing the mean accepted tokens.\n\nAs for the scenario where the increment is less than the decrement, we believe that this is a more complex situation and we will investigate it in our future research.\n\nThe aforementioned analysis will be added in the final version of our paper.\n\n\n**Q4. I wonder if the idea of ‘the prior tokens in speculative decoding are more important’ has been investigated by other papers since the conclusion is obvious and easy to derive. If so, the novelty of this paper will be decreased, and the author should give a further comparison to other papers.**\n\nA4: To the best of our knowledge, no prior work has proposed the idea presented in our paper. We are the first to introduce the idea that *\"prior tokens are more important\"* and provide rigorous theoretical proof for this concept. Furthermore, we leverage this insight to design a novel model architecture, Gumiho.\n\nMoreover, applying the principle of 'prior tokens are more important' to achieve speedup improvements is not straightforward. Simply enhancing prior token accuracy may increase computational overhead at the same time, which would prolong draft time and consequently undermine the overall speedup. In fact, we have no idea how this would affect the final speed-up ratio if we merely increase the computational budget on the prior tokens, since the mean accepted tokens and the draft time will increase together and they are two factors that play opposite roles.\n\nIn contrast, our Gumiho tries our best to fix the overall computational budget (by increasing the computational budgets of prior tokens and decreasing the latter) and shows that we can improve the mean accepted tokens. We achieve this by simultaneously boosting the accuracy of prior tokens while adopting simplified head structures (i.e., MLP) for subsequent tokens with parallel generation. This architectural innovation effectively balances accuracy and efficiency, ultimately achieving superior speedup.", "claims_and_evidence": "Theorem 3.1 in the paper as well as the proof in the appendix can support the claim.", "methods_and_evaluation_criteria": "I think the core idea of prioritizing the accuracy of the early tokens under a limited computational budget is important and meaningful to the research area of SpD, especially since a rigorous theoretical proof is provided. The evaluation criteria are the commonly used datasets and baseline models.", "theoretical_claims": "I roughly checked the proof of Theorem 3.1, and no problem was found.", "experimental_designs_or_analyses": "I checked the experimental designs and analyses. The datasets and baseline models are commonly used in other papers. The results are reasonable.", "relation_to_broader_scientific_literature": "There may have some findings on the conclusion ‘the prior tokens in speculative decoding are more important’ already.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper is well-organized, making it easy to follow.\n\nThe theoretical analysis is rich and the proof is rigorous.\n\nThe experimental section is comprehensive, covering multiple architectures and settings, demonstrates the validity and robustness of the proposed method. The overall speedup improvements over Eagle is non-negligible.\n\n\n\nWeaknesses:\n\nThe ablation study shows that using FTA has marginal improvement.\n\nThe conclusion of the theorem is simple but it seems that the proof is rather complicated. Is there an easier way to give the proof?\n\nThe theorem assumes that the total amount of decrease to the second part is the same as that of increase to the first part of the sequence. However, this is hardly the case in reality. Is it able to give a more general assumption?", "other_comments_or_suggestions": "No", "questions_for_authors": "I wonder if the idea of ‘the prior tokens in speculative decoding are more important’ has been investigated by other papers since the conclusion is obvious and easy to derive. If so, the novelty of this paper will be decreased, and the author should give a further comparison to other papers.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "URbDA9lTJc", "reviewer": "Reviewer_1s98", "summary": "Previous speculative decoding methods treat all tokens in a sequence equally, while this paper demonstrates that earlier tokens are more critical for success. To improve efficiency, the authors propose Gumiho, a hybrid model that prioritizes early tokens with a serial two-layer Transformer and uses lightweight parallel MLP heads for later tokens. This strategy boosts both the accuracy of early token predictions and the efficiency of later ones. Experimental results show that Gumiho outperforms existing methods in terms of speedup ratio and acceptance length.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions and support.\n\n**Q1. There is a lack of comparison to the latest state-of-the-art papers, which propose better results than Eagle2, such as HASS [1] and DDD [2]**\n\nA1:**(1) Hass** improves Eagle2 by addressing the training-inference inconsistency in serial transformer heads within SPD. While Hass focuses on training methodology, our work enhances performance via architectural modifications. These two directions are orthogonal: our method could integrate Hass’s training improvements for further gains. \n\nIn the table below, we compare the performance of our Gumiho with Hass. Additionally, we evaluate a variant of our model (denoted as Gumiho w/ Hass ) where Hass’s training method is applied to our Transformer Head. All results are conducted on a Mi250 GPU using the LLaMA3-8B model with a temperature of 0.\n\n|| MT-Bench|HumanEval| \n|-|-|-|\n|Eagle2|2.16×|2.51×|\n|Hass|2.26×|2.59×|\n|Gumiho|2.38×|2.77×|\n|Gumiho w/ Hass|**2.43×**|**2.84×**|\n\n**(2) DDD** improves Eagle2 by using a dynamic draft tree. Since DDD’s code is not open-sourced, we compare relative improvements over Eagle2. On the MT-Bench dataset, the performance gains of DDD and Gumiho are:\n\n||DDD|Gumiho|\n|-|-|-|\n|V 7B|3.5%|**9.4%**|\n|V 13B|4.2%|**6.3%**|\n|L2 7B|2.3%|**5.5%**|\n|L2 13B|2.3%|**5.4%**|\n\nThe aforementioned results and discussions will be included in the final version of our paper.\n\n**Q2. The method uses more parameters in the draft model compared to Eagle, Eagle2, and Medusa. Does the training process consume more GPU memory?**\n\nA2: Yes, you are correct. Our method does require more GPU memory during training compared to Eagle, Eagle2, and Medusa. Here’s why:\n- Eagle and Eagle2 use one single-layer Transformer head and reuse it serially across multiple speculative steps.\n- Medusa only maintains five parallel MLP heads.\n- Our approach, however, employs a two-layer Transformer head alongside five parallel MLP heads, all trained simultaneously.\n\nThis architectural choice increases memory consumption. However, it enables a hybrid generation strategy: serial decoding for the first few tokens (to ensure accuracy) and parallel decoding for subsequent tokens (to maximize speed). While this introduces a memory-accuracy trade-off, it achieves a significant speedup improvement. We will explicitly discuss this trade-off in the final manuscript.\n\n**Q3. Is it able to combine the proposed method with the state-of-the-art Spd method to further improve the speed up ratio? It is better if the conclusion of the theorem can be applied to other methods.**\n\nA3: As illustrated in A1, our method could integrate Hass’s training strategy for further gains. \n\nIn addition to Hass, our core theoretical principle—**prioritizing the initial tokens**—can also be generalized to serial multi-head draft models (e.g., Hydra or Eagle). By reallocating Head parameters to emphasize early tokens, this principle could further enhance the speedup ratio of existing SOTA methods. \n\nMoreover, existing methods typically use a single smaller model (e.g., LLaMA-3-3B) to accelerate larger counterparts like LLaMA-3-405B. However, our analysis reveals that adopting a differentiated draft model strategy—where a more capable model (e.g., LLaMA-3-8B) predict initial tokens, while a faster but lighter model (e.g., LLaMA-3-1B) handles subsequent tokens—could optimize the quality-speed trade-off beyond current homogeneous settings. \n\nExploring these extensions is a key direction for our future research.\n\n**Q4. What is the average draft time for each part of the method? In figure3 you show the overall draft time including the model forward time and full tree attention time. I would like to see each part of them.**\n\nA4:  The following table presents the draft time of the Vicuna 7B on Mi250 GPU. \n- `Serial Head Forward` represents `the time for a single serial head forward pass once`$\\times$`the number of forward passes`. In Eagle2, a single serial head corresponds to a single-layer Transformer, while in Gumiho it represents a two-layer Transformer.\n- `Parallel Head Forward` denotes the time for a parallel head forward pass once. Eagle2 does not have parallel heads, whereas in Gumiho, this refers to the parallel MLP heads.\n- `Tree Attention / Full Tree Attention` represents the time required to construct the tree structure in Eagle2 and Gumiho.\n- `Additional Computations` represents the time for additional computing. (e.g., using torch.top_k to retrieve current tokens).\n- `Total Time` means the total time consumed for a draft generation.\n\n||Eagle2|Gumiho| \n|-|-|-|\n|Serial Head Forward|9.8ms$\\times$6|21ms$\\times$2|\n|Parallel Head Forward|N/A|14.7ms|\n|Tree Attention / Full Tree Attention|3.3ms|3.4ms|\n|Additional Computations|3.1ms|3.2ms|\n|Total Time|65.2ms|63.3ms|", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "I checked the correctness of the proofs for theoretical claims, which is theorem 3.1 and appendix A in the paper.", "experimental_designs_or_analyses": "I checked the soundness/validity of experimental designs and analyses, which is section 4 in the paper.", "relation_to_broader_scientific_literature": "The proposed Gumiho combines the serial heads in Eagle2 and parallel heads in Medusa.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Pros:\n\n1. The finding of the proof is solid and inspiring.\n\n2. Given the proof, the approach in this paper is reasonable.\n\n3. The speed-up ratio gains, especially on large-scale LLMs such as LLaMA2 70B and LLaMA3 70B are obvious compared to Eagle2, which shows the effectiveness of the proposed method.\n\nCons:\n\n1. There is a lack of comparison to the latest state-of-the-art papers, which propose better results than Eagle2, such as HASS [1] and DDD [2].\n\n2. The method uses more parameters in the draft model compared to Eagle, Eagle2, and Medusa. Does the training process consume more GPU memory?\n\n[1] Learning Harmonized Representations for Speculative Sampling, ICLR, 2025.\n \n[2] Dynamic Depth Decoding: Faster Speculative Decoding for LLMs, arXiv, 2024.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "1. Is it able to combine the proposed method with the state-of-the-art Spd method to further improve the speed up ratio? It is better if the conclusion of the theorem can be applied to other methods.\n\n2. What is the average draft time for each part of the method? In figure3 you show the overall draft time including the model forward time and full tree attention time. I would like to see each part of them.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes Gumiho, a hybrid architecture for speculative decoding in Large Language Models. The core idea is motivated by the insight that early tokens in a draft sequence are more critical for the overall acceptance rate than later ones. Gumiho operationalizes this by using a more accurate, serial Transformer-based head for early tokens and faster parallel MLP-based heads for later tokens.

Strengths highlighted by reviewers include the clear motivation, the intuitive and elegant hybrid design, the theoretical justification, and the comprehensive experiments demonstrating speedup improvements over strong baselines (like Eagle2) across various models and benchmarks (please add the additional results from the rebuttal). The paper is generally well-written and sound (please update the paper to address all the points discussed below to provide further details and clarification on the architecture and experimental setup).
Decision: Accept (poster)