GRIFFIN: Effective Token Alignment for Faster Speculative Decoding
Authors: Shijing Hu
Jingyang Li
Xingyu Xie
Zhihui Lu
Kim-chuan Toh
Pan Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Model; Speculative Decoding; LLM Inference Acceleration; Token Alignment
Primary area: deep_learning
TL;DR: We propose GRIFFIN to accelerate the inference speed of LLM by addressing the token misalignment issue in speculative decoding.

Abstract: Speculative decoding accelerates inference in large language models (LLMs) by generating multiple draft tokens simultaneously. However, existing methods often struggle with token misalignment between the training and decoding phases, limiting their performance. To address this, we propose GRIFFIN, a novel framework that incorporates a token-alignable training strategy and a token-alignable draft model to mitigate misalignment.
The training strategy employs a loss masking mechanism to exclude highly misaligned tokens during training, preventing them from negatively impacting the draft model's optimization. The token-alignable draft model introduces input tokens to correct inconsistencies in generated features.
Experiments on LLaMA, Vicuna, Qwen and Mixtral models demonstrate that GRIFFIN achieves an average acceptance length improvement of over 8\% and a speedup ratio exceeding 7\%, outperforming current speculative decoding state-of-the-art methods. Our code and GRIFFIN's draft models will be released publicly in https://github.com/hsj576/GRIFFIN.

GRIFFIN: Effective Token Alignment for Faster Speculative Decoding
Shijing Hu' Jingyang Li
Kim-Chuan Toh2 Pan Zhou National University of Singapore "Singapore Management University s 8jhu240m. fudan. edu, cn 1zh@fudan edu.
Xingyu Xie
Zhihui Lu
Fudan University
1i li_jingyangeu. mus.ede xyxie0pku. edu. cn mattohkconus edu. ag panzhoudsmu edu. sg
Abstract
Speculative decoding accelerates inference in large language models (LLMs) by generating multiple draft tokens simultaneously. However, existing methods often struggle with token misalignment between the training and decoding phases, limit- ing their performance. To address this, we propose GRIFFIN, novel framework that incorporates a token-abie aheen-alignable training strategy and token-alle -akenn-liggaale draft model to mitigate misalignment. The training strategy employs a loss masking mechanism to exclude highly misaligned tokens during training, preventing them from negatively impacting the draft model's optimization. The aoken-alignable draft model introduces input tokens correct inconsistencies in generated fea- tures. Experiments LLaMA, Vicuna, Qwen and Mixtral models demonstrate that GRIFFIN achieves an average acceptance length improvement over 8% and speedup ratio exceeding 7%, outperforming current speculative decoding state-of the-art methods. Our code and GRIFFIN's draft models will be released publicly in https: h/ght::///ghhbbb cww/ssjj76...... http///s557///IFFII..
Introduction
Large Language Models (LLMs) like GPT-4 [1] and LLaMA [2, 3] have shown impressive capa- bilities diverse domains, including dialogue [4] and code generation [5]. However, the standard autoregressive decoding of LLMs generates tokens sequentially, with each token requiring a full forward pass through the entire model. Given the large size of LLMs, this process is both compu- tationally expensive and time-consuming, posing challenges for latency-sensitive applications. To accelerate generation, speculative decoding [6, 7] has become widely adopted and shown significant speed improvements. It leverages lightweight draft model to propose multiple tokens, verifies them in parallel with the target LLM, and accepts those aligned with the target's ' predictions. This enables multi-token generation per forward pass of the target LLM, substantially reducing latency.
However, the efficiency of speculative decoding depends critically on achieving high acceptance rate for draft tokens, while also minimizing the computational cost of generating them. Recent methods like EAGLE [8, 9] and Medusa [10] address this by utilizing shallow-layer hidden states the target LLM guide draft model's token predictions. Despite their improved efficiency, these methods face a fundamental limitation: misalignment between the training and decoding processes. During training, the draft model uses features from the target model and ground-truth tokens from training data, whereas in decoding, it relies on its own generated features and previously generated draft tokens. This discrepancy introduces two key issues: (1) feature misalignment, where the features generated by the draft model during decoding diverge from those features used during training, and (2) token misalignment, where ground -ruth tokens are replaced by draft tokens, often compounding
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

EAGLE2
EXGLE2
40-
HASS
GRIFFIN
3.8
4.01
length
5.4
(54)
43
GRIFFIN
Spoctop
2.5
3.4
31
2.0
2.8
3.0
3.2
3.0
3.2
3.1
1.8
EAGLE2 HASS
GRIFFIN
Mivalipoed
5%
15
24
15
20
24
2.6
2.5
1.2
7B
1.3
1.2 13B
Step
Step 2
Step 3 (b)
Step 4
Step
Forward 2 Forward 3 Forward 4 Forward
(1)
(c)
Figure I: Comparison between our GRIFFIN, EAGLE2, and HASS. (a) Speedup ratio comparison. (b) Acceptance length under different training steps, in which "Step n" denotes aligning draft model for n times in training. (c) Misaligned token rate under different forward passes in each drafting- verification cycle, where "Forward n denotes forwarding passes generate draft tokens.
errors over multiple steps. These misalignments, akin to exposure bias [11, 12], significantly degrade the acceptance rate of draft tokens and thus impair the overall speedup performance.
Efforts to address feature misalignment like HASS [13] use draft model's features to replace target model's features during training. While aligning training with decoding, it neglects token misalign- ment which is particularly problematic in decoding. Errors from carlier decoding steps propagate and amplify, further exacerbating token misalignment. For instance, as shown in Fig. 1 (c), EAGLE2 suffers from a token misalignment rate of 48% during training, resulting in suboptimal acceptance lengths and limiting its effectiveness. Similarly, while mitigating feature misalignment, HASS sees token misalignment escalate to 37% in later training steps, rendering it ineffective for deeper multi-forward harmonized training, e.g., training step n 2 3, as shown in Fig. 1 (b).
Contributions. We propose GRIFFIN, a novel speculative decoding framework that-unlike prior work-explieitly identifies and addresses the previously unobserved token misalignment issue, alongside feature misalignment. It introduces two core innovations: token-alignable training strategy and token-alignnn aalinnble draft model, working together significantly boost decoding efficiency.
Firstly, to mitigate token misalignment during training. GRIFFIN employs a dynamic loss masking mechanism that selectively backpropagates only through aligned tokens dokine-defined as those whose ground-truth tokens appear in the draft model's top-k predictions. This not only minimizes the disruptive effect of highly misaligned tokens, but also harmonizes training and decoding since draft trees decoding builds upon top-k predictions rather than exact matching to the highest- highest-probabiliy probability token. Unlike prior works [8, 9, 13] which train on exact targets, GRIFFIN embraces approximation while preserving signal fidelity, improving both alignment and generalization across decoding steps.
Secondly, further reduce token misalignment, GRIFFIN designs a token-alignable draft model by incorporating the architectural innovation of Moken-Guided Fusion (TGF) into draft model in EAGLE [8]. TGF performs two-step fusion to refine feature representations and mitigate incon- sistencies between the draft and target models. By incorporating input tokens twice- ince-illtllllyy with TGF module ensures that the draft model produces features more closely aligned with the target model, reducing feature and token misalignment.
features and later to refine them
These two components are mutually reinforcing. The token-alignable draft model reduces misalign- ment, increasing the number of aligned tokens available for effective training. In turn, the training strategy ensures that these aligned tokens contribute meaningfully to model optimization. Crucially, our approach is the first to expose and directly address token misalignment- an uncharted limitation in speculative decoding that hampers draft token acceptance and decoding speed. As shown in Fig. I (c), GRIFFIN consistently maintains much lower token misalignment rate compared to EAGLE2 and HASS across multiple forward steps. This yields longer accepted token sequences and greater speedups, particularly in deeper harmonized training settings where previous methods degrade.
Experimental results show GRIFFIN's superior performance over state-of state-of-the-ant the-arts (SoTAs) across di- verse tasks, including dialogue (MT-Bench [4]), code generation (vumanEval [S]), and mathematical reasoning (GSM8K [14]). For example, Fig. (a) and (b) show that on I.LaMA2-7/13B LLLMA3-8B, and Vicuna-7B, GRIFFIN improves the average acceptance length by 20% over EAGLE2 and 8% over HASS, while delivering speedup ratio of 18% over EAGLE2 and 7% over HASS.

---

Training Forward
Decoding l'orward
F
X
X1-1
Target Model
Draft Model
Head
Fy / F,
Fi+1
X
/ &
Figure 2: Token and feature misalignment in EAGLE.
Related Work
Speculative decoding [15-19] accelerates LLM inference by dividing each decoding step into a draft stage and a verification stage. Existing methods differ primarily in their draft model architectures strategies, each addressing specific challenges in speculative decoding.
Several methods enhance draft quality through context retrieval, e.g., PLD [20]. Lookahead [21], and CLLMs [22], which rely on prompt-based retrieval from similar contexts. However, their effectiveness is limited when relevant context is scarce or unavailable. Tree-based verification approaches like Sequoia [23] and SpecExec [24] use hierarchical structures to improve verification but incur high computational overhead, making them unsuitable for aatency-sensitiee
scenarios. Other works, including REST [25] and Ouroboros [26], reuse previous outputs or databases to guide drafting but constrained by the quality and accessibility of external resources. Chimera [27] and Glide [28] enhance token quality by integrating target model into draft model with extra computational cost.
Lightweight draft models have also been explored to improve efficiency. Medusa [10] employs MLPs for parallel candidate prediction, while Hydra [29] and Recurrent Drafter [30] use RNN-based models for regressive generation. EAGLE [8] and EAGLE2 [9] introduces transformer decoder for autoregression over feature sequences, balancing accuracy and complexity. FSPAD [3 constructs input sequences tailored for lightweight draft model predictions and introduces specialized training methods to improve draft quality. Additionally, methods like HASS [13] address feature misalignment during training and decoding but do not fully resolve token-level misalignment. In contrast, this work focuses on addressing token misalignment, a critical challenge in speculative decoding. We propose the GRIFFIN framework, which introduces token-alignable training strategy and token-alignable draft model. By tackling this issue, GRIFFIN improves both acceptance length and speedup ratio, offering complementary perspective to existing methods.
3 Motivation: Token Misalignment
Speculative decoding [6, 7] accelerates text generation by employing a "draft-and-verify strategy, Per cycle, a lightweight draft model M first generates multiple tokens through multiple forward passes, and stronger target model T then verifies and accepts subset in a single forward pass.
EAGLE [8] extends this paradigm by shifting autoregression from the token level to the feature level. As shown in Fig. 2, instead of predicting tokens directly, the draft model generates intermediate hidden state features that approximate those from the final layer of the target model T-just before the language modeling (LM) head H. At time step I, let x, and x, denote the I-th ground-truth and draft tokens, and F, and F, their respective hidden features from T and M. During training, as illustrated in Fig. 2, the draft model uses x; and F, to predict X and F.+1- However, during decoding, the draft model must rely solely on previously generated tokens X; and features F, F.-without access to x; or F,-since the target model is invoked only once per cycle after all draft tokens produced. This discrepancy introduces two fundamental issues: 1) feature misalignment where during decoding, for prediction, the draft model uses F, instead of F; as in training; and 2) token misalignment where the tokens X, used in decoding differ from ground-trute tokens X, seen during training.
Among these, token misalignment is particularly severe yet underexplored. Fig. 1(c) shows that for EAGLE2 and HASS, the rate at which x, * x, x,-the tte token misalignment rate-inccreaes sharply with the number of forward passes. For instance, EAGLE2 reaches misalignment rate of 48% when
3

---

generating five draft tokens per cycle. Even HASS, which partially mitigates feature misalignment, still suffers from 37% token misalignment rate. This degradation stems from error accumulation across passes, where early mistakes in token generation propagate and compound in subsequent steps.
Critically, high token misalignment undermines training effectiveness. As shown in Fig. 1(b), when the number of forward passes exceeds three in HASS, acceptance length platcaus-even with continued training. This suggests that draft models only generate fewer acceptable tokens, directly limiting decoding efficiency. Hence, solving token misalignment is not only important but necessary to unlock deeper multi-pass speculative decoding and greater speedups.
A seemingly simple fix reppaelan x, (ground-truth training tokens) with x; from the draft model during training- fails in practice. This is because 1) frameworks like EAGLE and HASS precompute and store F, for all x, before training which avoids the computational burden of regenerating training data; 2) swapping in X, leads to inconsistent input-feature pairs, which breaks the alignment needed for loss computation and degrades performance, as confirmed by Appendix. B in HASS. In fact, naive substitution significantly reduces acceptance length. In light of these challenges, we propose an effective solution to the token misalignment problem that preserves compatibility with existing training workflows and enables better alignment between training and decoding.
4 GRIFFIN: A Token-Alignable Framework
To address token misalignment challenges in Sec. 3, we propose GRIFFIN, a novel framework to mitigate token misalignment through two key components: 1) token-alignable training introduced in Sec. 4.1 and 2) token-alignable draft model elaborated in Sec. 4.2.
4.1 Token-Alignable Training
At the core of GRIFFIN is a progressive training strategy that mirrors how the draft model operates during decoding. Instead of relying on ground-truth tokens and features at every step-an an assumption that breaks down during inference-we gradually shift the model toward using its own outputs during training. This alignment is critical to mitigating token misalignment.
Concretely, GRIFFIN organizes training into multiple steps, where each training step involves draft model performing n forward passes to predict n future tokens and their corresponding features. With each additional pass, the model increasingly conditions on its own generated tokens and features from prior steps rather than ground-truth tokens and features from target model. This effectively aligns training process with decoding phase, as in training phase, the draft model simulates the similar input conditions encountered during decoding. Then we detail the first training step and its subsequent step.
First Forward Pass (n 1): Like vanilla autoregressive generation, draft model M predicts draft tokens which are then fed into target model T to verify and accept. Specifically, at time step f, draft model M and LM head H in target model predicts the t-th feature embedding F, and draft token Xy;
Fr M( F1a-1),
X (F.,,
where x denotes the token sequence [x from training dataset and F Fia-1 are the feature embedding sequence (F,) generated by target model 'T.
Token misalignment arises only from the second forward pass onward. In the first pass, both training and decoding share the identical ground-truth prefix X1i--1, so all predicted tokens X, are perfectly aligned. Therefore, no masking is needed in the first pass. The first-pass loss at step is
c Σ (xx,,,, x,, F..
where l combines a feature-level loss, i.e., the & distance between F, and F,, and a token-level loss, namely, cross-entropy between X, and x4. The detail implementation of é is summarized in Appendix. C.
n-th Forward Pass (n > 2). Draft model M would predict draft tokens at the n-th forward pass. From the second forward pass, during decoding, speculative decoding may reject draft token x,, in which case all later tokens X X in that draft sequence are also discarded. So, during
4

---

training, if X2 is unpredictable (rejected), the subsequent draft tokens X.+2.............. in this draft sequence are misaligned tokens. Training on those misaligned token provides no useful signal.
To address token misalignment challenge, we introduce a novel token-alignable training strategy that aligns draft model's 's training with its multi-pass behavior during decoding. Unlike prior approaches like EAGLE which use only top-1 predictions during training, our method incorporates the tree- structured decoding process directly into learning by supervising on top-k predictions. In EAGLE's decoding, each forward pass of draft model generates a top-k list of candidate tokens at each time step, forming a tree where alternative branches can be explored if the top-1 token is rejected. To match this. GRIFFIN considers draft token X, to be predictable if the ground-truth token x; appears within its top-k predictions Top-k(X;) This ensures that training reflects the decoding phase, where any top-k token may be valid. Accordingly, we introduce a binary predictable mask m, E {0, 1}. where m, if x, E TTp- and m, = otherwise. Since the current draft token x, is decided by previous predicted draft tokens X, x1=n+1:t-1 predicted in earlier 1)) forward passes, then if any draft token in x=-++1=-1 is unpredictable, the draft token x would likely be misalignment. To prevent the model from being penalised for such inevitably rejected positions, we introduce cumulative binary alignment mask m, adjusted by predictable masks mi-n+ii-1 of draft tokens
Xt-n+1a-13
These masks indicate whether token should contribute to the training loss, ensuring consistency between training and inference. Next, to further ensure alignment between training and decoding, we replace target-model features F, Fi-n41s-11 with seset--oodllgggerrate features FR-n+1s-1 from earlier passes. Then the draft model M and the LM head H. are used to generate the feature F, and the draft token X,:
F.- M(x Fir-ns Fo-n+1c-1), 1),
==(FF)).
(4)
Then, we define the following training loss to train the draft model M:
CM
mil(x, X1, Fx,,
(5)
Σ1=1 m
GRIFFIN's 's training strategy differs from priors like EAGLE and HASS that rely on ground- tokens during training. By progressively adapting draft model to operate under its own predictions and aligning its training with decoding, GRIFFIN addresses token misalignment issue via introducing top-k alignment masks, self-conditioning through generated tokens, and mask propagation.
4.2 Token-Alignable Draft Model
To enhance draft token accuracy and effectively address token misalignment, we propose a token- alignable draft model which systematically resolves feature inconsistency issues overlooked by prior draft models. While our architecture builds on EAGLE's draft model, it introduces two key extra modules: Token- Grieed Fusion (TGF) and insert TGF module before the autoregressive layer to fuse input features F, and tokens x,. After autoregression, we use TEH module, a dual-head design inspired by prior work [31], to output 1) a predict feature Ff for token prediction and 2) a regress feature F t++ to feed subsequent forward passes. Accordingly, TEH can separate and decouple the conflicting objectives of token prediction and feature generation within the draft model, improving draft token accuracy. Our ablation in Appendix. B confirm its effectiveness.
Head (TEH). As shown in Fig. 3(a), we
TGF module is designed to address a core challenge: feature representations in draft models often fail to match those of the target model, even after extensive training. Since feature-level losses can't 't be minimized to zero in practice, this gap leads to persistent misalignments, leading to the misaligned features between draft and target models which impairs draft token's accuracy. TGF tackles this by prioritizing token embeddings in the fusion process, guiding the feature generation toward better consistency with the target model. As illustrated in Fig. 3(), TGF operates in three steps:
(1) Embedding Fusion in Fig. 3 (b-i). Given input feature F and token embedding x (both in Rd))
we concatenate them and use lightweight MLP to project the result back 12d;
h C(F, x) W. + b..
(6)
5

---

Xr-2
Down Projector
LM Head & Sumpling
LM Head
SILU
Sampling
F
F
F
t+2
Up Projettor
TEH
TEH
cOn
Transfomeer Layers
One Autu-regressive Layer
layernorm
layemorm
X
Re+1
TGF
TGF
F
F
MLP
cresidual
connection
Furward
Forward
Forward
cooncttenate
Target Model
Token-Alignable Drafl Model
Token-Guided Fusion (TGF)
(a)
(b)
Figure 3: Structure of GRIFFIN's darft model. (a) Token- Aligna-li Draft Model. (b) TGF module. The diagram depicts the shared architecture used in both training and decoding thases-arrow indicating token flow correspond to valid data dependencies in both regimes.
Here, W. E 2ddd and by E Ed are the MLP weights and bias, and C(: is the concatenation operator. This produces unified feature that blends both token and feature information.
(2) Feature Normalization and Expansion in Fig. 3 (b-ii). We apply layer normalization both and x, then concatenate and expand the dimension to 4d using an Up Projector (a linear layer);
2 = C(C) + ba.
with Wa E R2dx4d and by E Rdd. Here, (-) denotes layer normalization. Operating in higher-dimensional space enables the model disentangle and align more complex token-feature relationships. This 4d expansion aligns with the intermediate size used in many transformer FFNs, and our ablations Sec. 5.2, Appendix and A.3 confirm its effectiveness.
(3) Refinement and Stabilization Fig. (b-iii). We apply a SiLU nonlinearity a to z and project back to Rd using a Down Projector (a liner layer). residual connection with h stabilizes training:
o (z) Wa ba + h,
(8)
where W. Ridxd and ba Rd. The nonlinearity enriches expressiveness, and the residual addition preserves essential fused information.
By explicitly integrating token embeddings into feature fusion, TGF ensures that generated features better reflect token distribution of target model. Combined with TEH, enables draft model to generate more accurate draft tokens and features, crucial for mitigating misalignment in multi-pas decoding. This token- and feature-aware design a key innovation that differs GRIFFIN from priors.
Experiments
Representative LLMs, including LLaMA2-Chat 7B/13B, LLaMA3-Instruu 8B/70B [3], Vicuna-1.5 7B [32], Qwen2-Instruct 7B [33] and Mixtral-88 8B 1 [34], are tested. For consistency, all inference runs use one NVIDIA A100 80G GPU, except for LLaMA3-70B and Mixtral-8x77 which require two GPUs. Vanilla auto-rogressive decoding is used as the baseline, serving as the benchmark for speedup ratios (1.00x). We compare GRIFFIN against recent SoTA speculative decoding methods, including SPS (standard speculative sampling with its draft model being Vicuna-68M) [6]. PLD [20], Lookahead [21], Medusa [10], EAGLE [8], EAGLE-2 [9], FSPAD [31], and HASS [13]. We follow priors and train our draft model on ShareGPT dataset, with token-alignment set to top-k (k 3)

---

Table 1: Comparison of different speculative decoding methods. This table presents evaluation results on standard LLM benchmarks with temperature T E {0, 1). including speedup ratio SR and acceptance lengths 7. Higher values indicate better performance.
Temperature
O
Temperature
Model
Method
MTT-bench HumanEval
GSM8K
Average
MT-bencch HumanEval GSM8K
Average
SR
SR
SR
SR
SR
SR
SR
SR
PLD
1.41
1.46
1.51
1.57
1.34
1.39
1.42
1.47
NIA, since the acceplance conditions are relaxed
Lookahcad
1.64
1.71
1.75
1.81
1.57
1.63
1.65
1.72
LLaMA2
EAGLE-2
2.69
4.50
3,22
2.77
4.72
2.89
4.82
2.41
4.29
3,00
5.01
2.63
4.66
2.68
Chat
FSPAD
2.89
4.82
3.38
5.62
2.95
4.99
3.07
5.14
2.61
4.53
3.14
5.35
2.84
4.88
2.86
4.92
7B
HASS
2.97
4.97
3,46
5.69
3,06
5.12
3.17
5.26
2.72
4.64
3.18
5.22
2.83
5.08
2.91
4.98
GRIFFIN
3.12
5.11
3.61
5.93
3.10
5.27
3.28
5.44
2.81
4.81
3.33
5.63
3.06
5,26
3.07
5.23
EAGLE
1.32
2,96
2.07
3.76
1.88
3.61
1.76
3.44
1.28
2.71
1.42
3.36
1.66
3.31
1.45
3.13
EAGLE-2
2.56
4.18
3.36
5.05
2.53
4.41
2.82
4.54
2.26
3.75
2.63
4.77
2.46
4,30
2.45
4.27
LLaMA3 Instruct
EAGLES3
2.93
3.59
3.17
5.01
3.23
5.15
2.51
4.18
3.27
5.47
ESPAD
2.72
4.52
3.40
5.39
2.95
4.77
3.02
4.89
2.43
4.09
3.04
5.18
2.75
4.60
2.74
4.62
8B
HASS
2.75
4.63
3,51
5.70
3.09
5.06
3.12
5.13
2.41
4.15
3.09
5.41
2.92
4.90
2.81
4.82
GRIFFIN
3,09
4.85
3.65
5.97
3.30
5.31
3.35
5.38
2.62
4.35
3.31
5.62
3.07
5.08
3.00
5.02
SPS
1811
2.34
2.04
2,68
1.73
2.28
1.86
2.43
1.49
1.85
1.57
1.99
1.52
1.80
1.53
1.88
Medusa
1.97
2.60
1,93
2.65
1.99
2.67
since
acceptance
conditions
relaxed
Vicuna1.5 7B
EAGLE-2
3.56
4.74
3.92
5.30
3.69
5.03
3.72
5.02
3.15
4.20
3.30
4.62
3.41
4.65
3.29
4.49
ESPAD
3.73
5.16
4.12
5.74
3.85
5.37
3.90
5.42
3.27
4.53
3.45
5.11
3.56
4.98
3.42
4.87
HASS
3.91
5.15
4.22
5.86
3.97
5.41
4.03
5.47
3.34
4.52
3.62
5.16
3,70
5.03
3.55
4.90
GRIFFIN
4.02
5.36
4.53
6.29
4.14
5.63
4.23
5.76
3.38
4.64
4.12
5.68
3.88
5.29
3.79
5.20
Qwen2
EAGLE-2
2.32
3.80
2.90
4.73
2.70
4.32
2.64
4.28
2.00
3.01
2.71
4.18
2.60
3.98
2.43
3.72
Instruct
HASS
2.59
4,23
3.18
5,46
2.91
4.86
2.89
4.85
2.17
3.23
2.83
4.52
2.79
4.38
2.59
4.04
7B
GRIFFIN
2.76
4.67
3.34
5.75
3,02
5.13
3.04
5.18
2.27
3,36
3.04
4.82
2.96
4.71
2.76
4.30
EAGLE-2
2.97
4.68
3.61
5.59
3.05
4.97
3,21
5.08
2.77
4.45
3.41
5.45
2.97
4.83
3.05
4.91
LLaMA2
ESPAD
3.09
5.05
3,91
5.98
3.32
5.35
3.44
5.46
3.03
4.85
3.51
5.71
3.21
5.25
3.25
5.27
Chat
HASS
3.11
5.05
4.16
6.05
3.38
5.33
3.58
5.47
3.05
4,90
3.66
5.85
3.22
5,30
3.31
5.35
13B
GRIFFIN
3.33
5.27
4.29
6.26
3.61
5.56
3.74
5.70
3.36
5.07
3.94
6.13
3.61
5.49
3.64
5.56
LLaMA3
EAGLE-2
2.96
4.13
3,21
4.42
3.40
4.54
3.65
5.01
4.32
4.46
Instruct
HASS
3.36
4.59
4.61
5.73
4.01
5.21
3.99
5.17
3.35
4.48
4.23
5.65
3.84
5.17
3.80
5.10
70B
GRIFFIN
3.52
4.66
4.71
6.03
4.09
5.39
4.11
5.36
3.49
4.54
4.33
5.94
3.90
5.30
3.91
5.26
Mixtral-vO.1
EAGLE-2
1.96
3.39
2.34
4.13
2.19
3.79
2.16
3.77
1.93
3.32
2.28
3.98
2.09
3.71
2.10
3.67
Instruct
HASS
2.17
3.67
2.63
4.76
2.39
4.58
2.39
4.33
2.09
3.61
2.53
4.58
2.24
4.46
2.28
4.21
8x7B
GRIFFIN
2.29
3.97
2.82
5.25
2.51
4.86
2.54
4.69
2.22
3,89
2.71
5.08
2.39
4.72
2.44
4.56
for N 3 steps. Other hyperparameters (e.g., optimizer) match EAGLE-2 for fair comparison. We proved the detailed description of GRIFFIN's hyperparameter settings in Appendix. C and implementation of baseline methods in Appendix. D.
We assess performance on three key tasks: multi-turn conversation (MT-Bench [4]), code generation (HumanEval [5]), and mathematical reasoning (GSM8K [14]). To align with prior work (e.g.. DistillSpec [35], EAGLE), we fix the batch size to and set the temperature T E (0. Like prior speculative decoding methods, GRIFFIN is also lossless, eliminating the need for additional quality evaluation of generation. Accordingly, we follow priors and focus on acceleration metrics: I) Speedup Ratio (SR) measure actual test speedup ratio over vanilla autoregressive decoding; and verification cycle.
2) Acceptance Length (7) which is average token number accepted per drafting-verification
5.1 Comparison with SoTAs
We present the acceptance lengths and speedup ratio of various methods across three datasets in Table 1. GRIFFIN consistently achieves the highest acceptance length and speedup ratio across all datasets and LLMs tested. Each GRIFFIN tokens, significantly exceeding other methods. This is roughly three times the amount of standard speculative sampling and 1.5 times the amount of EAGLE.
cycle generates approximately 5-6
For the multi-round conversation task (MT-Bench) with LLaMA3 8B (temperature = 0), GRIFFIN achieves an 8.7% higher speedup ratio compared to HASS. Even for temperature T 1, GRIFFIN
7

---

Table 2: Ablation study on Token-Alignabee Training (TAT) and Token-Alignable Draft Model (TAD). This table presents the evaluation speedup ratio SR and acceptance lengths 7 on L.LM benchmarks with temperature T E {0, Higher values indicate better performance.
Temperature =
||
Temperature
Method
MT-bench
HumanEval
GSMBK
Average
||
MT-bench
HumanEval
GSM8K
Average
SR
SR
SR
SR
SR
SR
SR
SR
GRIFFIN
3,12
5.11
3,61
5,93
3.10
5.27
3.28
5.44
2.81
4.81
3,33
5,63
3,06
5.26
3.07
5.23
w/o both
2.69
4.50
3.22
5.24
2.77
4.72
2,89
4,82
2.41
4.29
3.00
5.01
2.63
4.66
2.68
4.65
w/o
2,89
4.85
3,40
5.65
3.01
2.62
4.56
3.13
5.34
2.75
4,96
2.83
4.95
w/o TAD
2,95
4.94
3.45
5.68
3.08
5.14
3.16
5.25
2.73
4.65
3.24
5.31
2.82
5.05
2.93
5.00
Table 3: Comparison of different top-k parameter for GRIFFIN. This table presents evaluation of speedup ratio SR and acceptance lengths т standard LLM benchmarks with temperature E {0, 1). Higher values indicate better performance. NA represents do not align token.
Temperature
Temperature
Top-k
MTT-hench
HumanEval
GSM8K
Average
MTT-hench
HumanFvval
GSM8K
Avernge
SR
SR
SR
SR
SR
SR
SR
SR
3.01
5.03
3.56
5.85
3,05
5.17
3.21
5.35
2.75
4.73
3.27
5.52
2.85
5.10
2.96
5.12
3,12
5.11
3.61
5,93
3.10
5.27
3.28
5.44
2.81
4.81
3.33
5.63
3,06
5.26
3.07
5,23
5.09
3.59
3.08
5,23
3.25
2.78
4.78
3.32
5.61
3.01
3.04
5.19
10
3.03
5.05
3.55
5.84
3.06
5.19
3.21
5.36
2.74
4.72
3.30
5.60
2.89
5.12
2.98
5.15
NA
2.97
4.95
3.52
5.76
3.04
5.12
3.18
5.28
2.72
3.24
5.47
2.82
5.01
2.93
5.04
maintains 7.8% improvement over HASS. For the code generation task (HumanEval) with Vicuna 7B, GRIFFIN demonstrates a 7.3 7.3% increase in speedup ratios compared to HASS at temperature of and 13.8% improvement when the temperature
to For the mathematical reasoning task (GSM8K with LLaMA2 13B), GRIFFIN achieves a 6.8% increase in speedup ratios compared to HASS with temperature T 0, and an 12.1% improvement a temperature of GRIFFIN also outperforms EAGLE-3 across all benchmarks and temperature settings in LLaMA3 8B. GRIFFIN achieves 3.7% higher speedup ratio and 4.5% higher acceptance length compared with EAGLE-3 at temperature of O. GRIFFIN achieves 3.8% higher speedup ratio and 3.9% higher acceptance length compared with EAGLE-3 when the temperature is set
GRIFFIN also demonstrates consistently strong acceleration across LLMs with different architectures beyond LLaMA/Vicuna. On the Qwen2 7B model, GRIFFIN achieves 5.8% improvement in speedup over HASS. The speedup ratio for Qwen2 7B is slightly lower than LLaMA2 7B model. This discrepancy can be attributed Qwen2 7B's larger vocabulary size, which results in more substantial LM Head and subsequently slows down the draft model's 's decoding speed. For Mixtral-8 Mixtral-8x7B. the acceleration from speculative decoding, including that of GRIFFIN, less pronounced compared to other LLMs. This is primarily due to the inherent challenges of applying speculative decoding techniques
cexture-of-Experrr (MoE) architectures. In these settings, verifying multiple tokens simultaneously imposes additional computational overhead, which affects all speculative decoding methods such as GRIFFIN, HASS, and EAGLE-2. Nevertheless, even under the MoE scenario, GRIFFIN achieves more than 6.6% higher speedup ratio than HASS on Mixtral-8x7B further demonstrating its strong generalization ability across diverse large language model architectures.
The results across diverse tasks and models highlight the versatility and effectiveness of GRIFFIN. The consistent improvements over HASS, even at different temperatures, underscore GRIFFIN's robustness in handling varying levels of uncertainty in token predictions. Moreover, the performance gains in tasks like code generation and mathematical reasoning suggest that GRIFFIN's token- alignable speculative decoding framework is particularly advantageous for applications requiring high precision reasoning capabilities. These findings position GRIFFIN as strong candidate for accelerating LLM inference in real-world scenarios, where both speed and accuracy are critical.

---

Table 4: Comparision of varied training steps for GRIFFIN. This table presents evaluation results of speedup ratio SR and acceptance lengths T on standard LLM benchmarks with temperature T E (0. 1). Higher values indicate better performance.
Temperature
O
Temperature =
Step
MT bench
HumanBval
GSM8K
Average
bench
HumanEval
GSM8K
Average
SR
SR
SR
SR
Sit
SR
SR
SK
1
2.89
4.85
3,40
5.65
3.01
5.04
3.10
5.18
2.62
4.56
3.13
5.34
2.75
4.96
2.83
4.95
2.99
5.02
3.51
5.81
3.06
5.15
3.19
5.33
2.74
4.73
3.26
5.49
2.92
5.14
2.97
5.12
3.12
3,61
5.93
3.10
5.27
3.28
5.44
2.81
4.81
3.33
5.63
3.06
5.26
3.07
5.23
4
3.13
5.13
3.62
5.96
3.12
5.31
3.29
5.47
2.82
4.84
3.35
5.66
3.08
5.31
3,08
5.27
5
3.13
5.14
3.63
5.98
3.13
5.33
3.30
5.48
2.83
4.86
3.36
5.68
3.10
5,34
3,10
5.29
5.2
Ablation Study
Effectiveness of GRIFFIN Components. We evaluate the impact of GRIFFIN's two key Training (TAT) and the Token-Alignable Draft Model (TAD)-using LLLMA2-Chat 7B. As shown in Table 2. removing either component significantly reduces acceptance length and speed up ratio. Removing TAT leads marks, with average acceptance length reduced by 0.26 at T 0 and 0.28 T = 1. speed up ratio reduced by 0.18 at 7 and 0.24 at T = 1. This confirms the importance of TAT in aligning draft tokens during training. Similarly, removing TAD causes noticeable degradation, with acceptance length decreasing by 0.19 at = 0 and 0.23 at = 1, speed up ratio reduced by 0.12 at T = 0 and 0.14 T = 1, highlighting TAD's role in reducing misalignment during decoding. Notably, removing both components results in the steepest decline- 0.62 at T and 0.58 at T = 1- underscoring their complementary effects. Together, TAT and TAD ensure that draft tokens are aligned during both training and decoding, enabling GRIFFIN to achieve state-of-the-arr performance.
consistent performance drop across all bench-
Hlyper-parameters in Token-Alignable Training. We analyze the effect of the hyper-parameter which determines the number of top-k tokens to align. As shown in Table 3, aligning top-k tokens (from 1 to 10) consistently improves acceptance length and speed up ratio compared to no aakenn-liggmmnt. Notably, aligning only the top-1 token less effective, as neglects many other tokens that could benefit from alignment. The acceptance length and speed up ratio achieves its peak when & = 3, suggesting that aligning small but sufficient number of tokens provides the optimal trade-off between alignment and generalization.
We further analyze the effect of increasing the number of training steps N TAT. As shown in Table 4, increasing the training steps steadily improves GRIFFIN's acceptance length during the first steps. Unlike HASS which plateaus after step (see Fig. b)), GRIFFIN continues improve due to its token alignment mechanism. However, as the number of aligned tokens decreases with each additional training step, the improvements become less pronounced at steps and 5. To ensure fair comparison with HASS, we choose the number of training steps N 3 in our experiments.
Table 5: Ablation study on TGF. This table presents evaluation results of speedup ratio SR and acceptance lengths 7 standard LLM benchmarks with temperature T' [0, Higher values indicate better performance. "Feature" and "Fused" denotes using F and to replace x in Eqn. (7).
Temperature
O
Temperature = 1
Method
|
MT-bench
HumanEval
GSMBK
Average
||
MT-bench
HumunEval
GSM8K
Average
SR
SR
SR
SR
||
SR
7
SR
7
SR
SR
GRIFFIN
3112
5.11
3,61
5.93
3.10
5.27
3,28
5.44
2,81
4,81
3.33
5.63
3,06
5.26
3,07
5.23
Feature
2.63
4.44
3.06
4.78
2.71
4.60
2,80
4.61
2.35
4.23
2.86
4.47
2.54
4.50
2.58
4.40
Fused
2.91
4.87
3.42
5.68
5.07
3.10
5.21
2.64
4.59
3.15
5.36
2.83
4.97
2.87
4.97
Effectiveness of Token-Guided Fusion (TGF), To assess whether TGF's improvements stem from its token-aware design rather than just increased capacity via using more parameters, we conduct an ablation study in Table 5 by altering the secondary fusion input in Eqn. (7). To investigate this,

---

we replace token embeddings x with either raw features F or the initial fused features h in Eqn. (7), while keeping model size and training process fixed.
Replacing token embeddings x with raw features F in Eqn. (7) reduces acceptance length by 0.83 and speed up ratio by 0.48, indicating that features alone insufficient to resolve inconsistency. Replacing them with the initial fused features h in Eqn. (7) performs better-these retain some token bnformation--ut still lags behind the original design by 0.26 acceptance length and 0.2 in speed up ratio. These results confirm that TGF's effectiveness is not due to parameter scaling but stems from the explicit use of token embeddings, which are crucial for correcting inconsistent features and aligning the draft model with the target distribution.
We further investigate the impact of varying the expansion dimension of W. in Eqn. (7). As reported in Table 7 of Appendix A.3, decreasing the expansion dimension to 4,096 significantly hampers TGF's capacity to separate essential information from noise, leading marked reductions in both acceptance length and speedup ratio. Conversely, increasing the expansion dimension to 22,016 results slight improvement in acceptance length, attributable to greater representational capacity, but also introduces additional computational overhead, thereby reducing the speedup ratio. These findings validate the expansion dimension choice in GRIFFIN, demonstrating a well-balanced trade-off between performance and computational efficiency.
Conclusion
In this paper, we present GRIFFIN, a eoken-alignable speculative decoding framework. Prior methods have largely ignored the token misalignment problem between training and decoding. GRIFFIN addresses this by introducing a token-au aikn--alggnabl training strategy that excludes misaligned tokens from loss computation. It further incorporates token-alignable draft model that substantially reduces misalignment. Extensive evaluations across diverse LLMs and datasets show that GRIFFIN consistently outperforms SoTAs, achieving the highest speedup ratios and acceptance lengths.
Limitations. GRIFFIN adopts multi-step training process for ooken-alignable training, which incurs additional training overhead compared to EAGLE. However, since the draft model is trained only once, real-world applications prioritize decoding efficiency over training overhead, as inference is the primary bottleneck. GRIFFIN improves the specdup ratio by over 18% compared to EAGLE2, making the extra training cost a worthwhile trade-off for the significant inference accelcration delivers. Furthermore, GRIFFIN's 's overall training overhead remains comparable to that of HASS. Under the same training cost, GRIFFIN achieves an over 7% improvement in speedup ratio compared to HASS, further highlighting its effectiveness.
Broader Impact. GRIFFIN advances the efficiency of LLM inference by accelerating decoding speed without sacrificing output quality. This improvement can democratize access powerful LLMs by making real-time applications more feasible. Downstream, GRIFFIN could enable smoother, faster interactive Al for education, healthcare assistants, accessibility tools, and scientific research, broadening beneficial applications and reducing latency barriers for users worldwide.
Acknowledgement
This work was supported by the Yangtze River Delta Science and Technology Innovation Community Joint Research Project (YDZX20213100004001), the National Key Research and Development Program of China (2022YFC3302300) and the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant (Proposal ID: 25-SIS-SMM SIS-SMM--003). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the Ministry of Education, Singapore.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 07X1v:2303.08874, 2023. 1
10

---

[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023. 1
[3] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.09288, 2023. 1,6
[4] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lim-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, (6346595-446622 2023. 1,
2,
[5] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021. 2,
[6] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286.
PMLR, 2023. 3,
[7] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving. Jean- Saan-taptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01336 2023. 3.
[8] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint wrXv:22401.1507,, 2024. 1, 2, 3, 6
[9] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 16858, 2024. 1, 2, 3, 6
[10] Tianle Cai, Yuhong Li, Zhengyang Geng. Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 2024. 1,
[11] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015. 2
[12] Florian Schmidt. Generalization in generation: A closer look at exposure bias. arXiv preprint 11X10:191,.00222, 2019. 2
[13] Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized represen- tations for speculative sampling. arXiv preprint arXiv:2408. 15766, 2024. 2, 6
[14] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1410 2021. 2.7
[15] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024. 3
[16] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pages 932-949, 2024.
[17] Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chen-Chuan Chang, and Jie Huang. Cascade speculative drafting for even faster Um inference. arXiv preprint 212. 2023.
11

---

[18] Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36, 2024.
[19] Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, lon Stoica, and Hao Zhang. Online speculative decoding. arXiv preprint arXiv:2310.07177, 2023.
[20] Apoorv Saxena. Prompt lookup decoding, November 2023. 6
[21] Yichao Fu, Peter Bailis, lon Stoica, and Hao Zhang. Break the sequential dependency of Ilm
inference using lookahead decoding. arXiv preprint (7Xi::2402.0205//7 2024. 3, 6
[22] Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. arXiv preprint arXiv:2403.00833 2024. 3
[23] Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint TrXiv:2402.12374, 2024. 3
[24] Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive Ilm inference on consumer devices. arXiv preprini arXiv:2406.0255: 2024.
[25] Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval-based speculative decoding. arXiv preprint UrXiv:2311.08252 2023. 3
[26] Weilin Zhao, Yuxiang Huang, Xu Han, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. Ouroboros: Speculative decoding with large model enhanced drafting. arXiv preprint arXi::2402.33722 13720, 2024. 3
[27] Ziqian Zeng, Jiahong Yu, Qianshi Pang, Zihao Wang. Huiping Zhuang, Hongen Shao, and Xiaofeng Zou. Chimera: A lossless decoding method for accelerating large language models inference by fusing all tokens. arXiv preprint arXi::2402.1517 15758, 2024. 3
[28] Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yonggi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. arXiv preprint
20t.:2400.020882 2024. 3
[29] Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelll Kelley, and William Brandon. Hydra: seaueeniilll-deeedeeeee draft heads for medusa decoding. arXiv preprint arXiv:2402.05/09 2024.3
[30] Yunfei Cheng, Aonan Zhang, Xuanyu Zhang, Chong Wang. and Yi Wang. Recurrent drafter for fast speculative decoding in large language models. arXiv preprint arXiv:2403.099) 240.09999,999,, 2024.
[31] Lujun Gui, Bin Xiao, Lei Su, and Weipeng Chen. Boosting lossless speculative decoding via feature sampling and partial alignment distillation. arXiv preprint arXiv:2408. /5562, 2024. 3, 5,
[32] Zhenyi Lu Chenghao Fan and Jie Tian. Chinese-vicuna: A chinese instraction-followie based model. 2023, 6
[33] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023. 6
[34] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.0400 2024,
[35] Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Ros- tamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint 23Xiv:2310.08466 2023. 7
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Sec. we identifies the token misalignment issue, and in Sec. 4 we proposed token-alignable training strategy and token-alignable draft model. Experimental results in Sec. 5 verifies that our proposed methods effectively address the token misalignment problem.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of proposed method in Sec. 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification: Our paper focuses on practical methods to accelerate LLM inference and does not present theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide a detailed description the dataset, computational resources, training methods, and hyperparameter settings in Sec. 5 and Appendix. C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is included in https: / htt.:/ttt.....oooooee.et.eeet com/hs along with detailed guidelines for reproducing our experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.c cc/
httpg//dddeeeeeuuuuiiissssssssse www.ooooolllyyyl for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.co wwtppblcc/aacecccccccccccccc cabbiccgguidss gwwgge/ddeeeeee /ed/eeSeeu asiionooiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed description of the dataset, computational resources, training methods, and hyperparameter settings in Sec. and Appendix. C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Following the setting of the past papers the speculative decoding area, the core benchmarking process of speculative decoding involves running the same inference workloads multiple times each dataset, which yields highly consistent results with minimal variance due to the deterministic nature of the inference pipeline. Therefore, the reported numbers accurately reflect the acceleration performance without necessitating error bars, and confident that overall speedup ratio and acceptance length is statistically significant.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed description of the computational resources for reproduc- tion in Sec. 5 and Appendix. F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htt////eerrsseeeeee//ee//
Answer: [Yes]
Justification: The research conducted in this paper entirely conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
16

---

Justification:
We discuss the potential positive societal impacts of our proposed method in
Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper explicitly cites relevant sources for datasets, pre-trained models, and baseline code, and it clearly states compliance with the respective licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The new assets of GRIFFIN's draft models are well documented and made accessible alongside comprehensive documentation in https: //github com/csoc////ooooo js 5776 GRIFFIN.
Justification:
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper focuses LLM inference acceleration and does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
18

---

Justification:
This paper focuses on LLM inference acceleration and does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https:/ httpps///neurps...o http:/pps/e//oooeeen//////////// for what should or should not be described.
19

---

A Analysis for the Architecture of Token-Guided Fusion (TGF)
A.1 Motivation Behind TGF
The Token-Guided Fusion (TGF) module is motivated by the limitations of the standard concat-then- MLP strategy, as adopted in EAGLE, which does not fully capture the complementary information between token embeddings and draft model features. In practice, features generated by the draft model often remain misaligned with the target model's representations, a discrepancy that cannot be effectively eliminated with feature-Ievel loss minimization alone. TGF addresses this challenge by explicitly leveraging token embeddings to guide the fusion process, aligning feature distributions more closely to those of the target model. confirmed by ablation results (Table 5), this targeted architectural enhancement significantly reduces feature inconsistency, demonstrating that the modest complexity introduced by TGF provides strong empirical gains.
Key Architectural Enhancements in TGF:
Feature Normalization and Dimensional Expansion: Separate layer normalization is applied to both the initial fused features h and token embeddings x in Fig. 3 (b-ii), allowing for independent statistical scaling and improved stability during training. The Up Projector in Fig. (b-ii) expands the feature dimensionality, which helps disentangle relevant information and increase the expressiveness of fused representations.
Nonlinear Refinement and Consolidation: The SiLU activation function in Fig. 3 (b- iii) introduces nonlinearity, enhancing the module's 's capacity to recover complex feature interactions beyond linear operations. Afterwards, the Down Projector Fig. 3 (b-iii) compresses the representation back to the target dimension, aggregating enriched information into stable fused feature for downstream processing.
Overall, TGF enables the draft model to more accurately approximate the target model's output space, evidenced by measurable improvements in acceptance lengths and speedup ratio as shown Table. 1.
A.2 Ablation Study on the TGF Architecture
To systematically assess the contributions each component within the TGF module, we performed targeted ablation experiments, with each variant constructed by selectively removing modifying specific submodules:
Ablation 1: Simultaneous removal of the Up Projector (Fig. 3 (b-ii)) and the SiLU activation (Fig. (b-ii)).
Ablation 2: Exclusion of the token embeddings x from the secondary fusion step in Eqn. (7).
Ablation 3: Exclusion of the initial fused feature h from the secondary fusion step in Eqn. (7).
Table Ablation results for architecture of TGF. This table presents evaluation results on standard L.M benchmarks with temperature 7 E (0, including speedup ratio R and acceptance lengths 7. Higher values indicate better performance.
Temperature =
Temperature =
Method
MTT-bench
HumanFval
GSMBK
Mean
MT-hench
HumanFval
GSMBK
Mean
GRIFFIN
2.81
4.81
3.33
5.63
3,06
5.26
3,07
5.23
Ablation
2,93
4.85
3,36
5.73
3.04
5.07
3.11
5.21
2.68
4.55
3.20
3.39
2.69
4.96
2,86
4,96
Ablation
3.31
Ahlation 3
1.76
2.75
1.92
3.09
1.65
2.58
1.78
2.81
1.67
2.65
1.88
3.04
1.49
2.53
1.68
2.74
Table 6 presents the results of these ablation settings. The following key observations can be drawn:
20

---

Ablation 1: Omitting both the Up Projector and SiLU activation produces a marked decrease
performance, with acceptance length reduced by 0.23 (T and 0.27 (T' 1), and speedup ratio reduced by 0.17 (T = 0) and 0.21 21 (T = 1). This highlights the critical role these components play in enabling expressive and stable feature fusion.
Ablation 2: Removing token embeddings x from the secondary fusion step adversely affects the model's ability to inject token-specitfe information, resulting in lower acceptance length (by 0.10 at T = 0, 0.13 at T = 1) and speedup ratio (by 0.07 at T = 0, 0.08 at T = 1).
Ablation 3: Excluding the initially fused feature h from the secondary fusion produces the most severe degradation: acceptance length decreases by 2.63 (T 0) and 2.49
(T' -"=11 1), while speedup ratio drops by 1.50 (T 0) and 1.39 (T = 1). This underscores that the recurrent integration of fused features is indispensable for capturing high-quality representations and achieving effective aligmment.
Overall, these ablation results confirm the necessity of each architectural component within TGF for maximizing acceptance length and speed up ratio.
A.3 Ablation on the Expansion Dimension of TGF
In the TGF module. the expansion dimension refers the output dimensionality of the Up Projector in Fig. 3 (b-ii). For GRIFFIN, we set this dimension to 11,008, matching the intermediate size of the target model's feed-forward network (FFN). To evaluate the impact of this design choice, we perform ablation experiments by varying the expansion dimension, while holding all other components and training protocols constant.
Table Comparison of different expansion dimension of GRIFFIN. This table presents evaluation results on standard LLM benchmarks with temperature T E 1). including speedup ratio SR and acceptance lengths т. Higher values indicate better performance.
Temperature
0 GSMRK
Temperature =
Expansion Dimension
Model Size
MT-hench HumanEval
Mean
MT-bench
HumanEval
GSM8K
Mean
SR
SR
SR
SR
1RR
SR
SR
|RR
11,008 (GRIFFIN)
0.41B
3.12
5.11
361
5,93
3,10
5.27
3,28
5.44
2,81
4.11
3.33
5.63
3,06
5.26
3,07 5.23
0.33B
3.55
582
3.06
5.32
2.75
4.68
3.01
5.10
22,016
0,558
2.97
5:15
3.38
6.01
2.95
5.34
3.10
5,50
2.60
4.84
3.14
5.76
2.71
5.17
282
5.25
Table 7 summarizes the results, from which we draw the following conclusions:
Smaller expansion (4,096): Lowering the expansion dimension to 4,096 degrades TGF's capacity to extract and distinguish salient features, leading notable reduction in accep- tance length (by 0.12 at T 0, 0.13 at T 1) and speedup ratio (by 0.06 at both T = 0 and 1).
Larger expansion (22,016): Increasing the expansion dimension to 22,016 yields a slight
improvement in acceptance length (by 0.06 at T = O, 0.02 at = 1), suggesting marginal gains in representational power. However, this is offset by a decline in speedup ratio (reduced by 0.18 at T O, 0.25 at T = 1), primarily due to increased computational overhead and an additional 0. 14B parameters.
Overall, these results validate our chosen configuration: setting the TGF expansion dimension equal the target model's FFN intermediate size achieves an effective balance between fusion performance and computational efficiency.
B Effectiveness of Token-Alignable Draft Model (TAD) Components
We assess the individual contributions of Token-Guided Fusion (TGF) and TokeneEnhanced Head (TEH)-the two principal components of the Token- Alignalle Draft model (AAD)-using I.LaMA2- Chat 7B as the base model. Ablation results summarized in Table 8.
21

---

Table 8: Ablation results for TAD. This table presents evaluation results on standard LLM benchmarks with temperature {0, 1}. including speedup ratio SR and acceptance lengths т. Higher values indicate better performance.
Temperature
Temperature
Method
|
MT--bench
HumanEval
GSM8K
Mean
MTThhecc
HumanEval
GSM8K
Mcan
SR
SR
SR
SR
SR
SR
SR
SR
GRIFFIN
3,12
5.11
3,61
5,93
3,10
5.27
3.28
5.44
2.81
4.81
3,33
5.63
3,06
5.26
3.07
5.23
w/o TEH
3.09
5.07
3.56
5.85
3,08
5.22
3.24
5.38
2.77
4.75
3.29
5.55
3.03
5.21
3.03
5.17
w/o TGF
3,04
4,99
3.49
5.75
3.05
5.13
3.19
5.29
2.75
4.70
3.25
5.43
2.88
5.11
2.96
5.08
Removing either component leads clear and consistent reduction in both acceptance length and speedup ratio:
Token- Enhaneed Head (TEH); Excluding TEH results in a consistent performance drop across all benchmarks, with the average acceptance length reduced by 0.06 and speedup ratio decreased by 0.04. This highlights the critical role TEH boosting the draft model's token prediction accuracy.
Token-Guided Fusion (TGF): Excluding TGF leads to greater degradation: accep- tance length drops by 0.15, and speedup ratio decreases by 0.09 at T = 0 and by 0. at
T 1. These findings reinforce TGF's efficacy mitigating feature misalignment during speculative decoding.
Collectively, these results underscore that both TGF and TEH are indispensable for maximizing the effectiveness and efficiency of the TAD architecture.
C Implementation Details of GRIFFIN
C.1 Loss function
Per-step loss composition. The per-step loss (xx,, xx, Fr, F:) in Eq. (5) combines two comple- mentary components that supervise both the token prediction and the latent feature alignment of the draft model M with the target model T:
(xx, X1, F., Fi) Arkk (Xe, xt) + Aicar &teat (Fr, Ft),
(9)
where Ano and Aeent are scalar weights (default: Aook Afear 0.1 unless otherwise stated).
Token-level loss. The token-level supervision aligns the predicted token distribution of the draft model with the ground truth:
Lok (xx, x:) - log (x X1:t-),,
(10)
which corresponds to the standard cross-entropy between the predicted logits x, and the one-hot target X,.
Feature-level loss. To encourage the internal representations of the draft and target models to remain consistent, minmmize the E distance between their hidden features:
Eear (F,, F.) F., Felli-
(11)
This term regularizes the draft model toward the target model's latent space, facilitating stable alignment across multi-pass decoding.
C.2 Draft Tree Structure
For all experiments, we use dynamic tree structure with a total 60 draft tokens and set the draft tree depth to 6, closely following the optimal configuration established in EAGLE-2 and HASS.
22

---

C.3 Training Configuration
The draft model is trained using the Adam optimizer, with the following key settings:
Learning rate: 3e
Batch size: (per GPU)
Number of epochs: 20
Total training steps: 800,000
Warmup: 2,000 steps of linear warmup: learning rate scheduler enabled
Optimizer: AdamW, with betas (0.9, 0.95)
Gradient clipping: 0.5 (by value)
Maximum sequence length: 2,048 tokens
All hyperparameters are kept fixed for all reported experiments unless otherwise specified. Addi- tional hyperparameters and implementation scripts are provided in
GRIFFIN.
D Clarification of Baseline Methods
For EAGLE, EAGLE-2, and Medusa, we directly utilized the publicly released draft model parameters provided by the respective authors. For methods that do not require draft model training, such as PLD, Lookahead, and SPS, evaluated performance using official code from their GitHub repositories.
Regarding HASS, we used publicly released draft model parameters for LLaMA2-7B, LLaMA3- 8B, LLaMA2-13B, and LLaMA3-70B. However, at the time of submission, official draft model parameters for Vicuna-7B, Qwen2-7B, and Mixtral-8x7B were unavailable. To enable fair comparison, trained the HASS draft models ourselves using their official GitHub repository and strictly followed the configurations described the HASS paper. The experimental results we obtained closely corresponded those reported by the HASS paper. Similarly, draft model parameters for FSPAD were not publicly available at the time of submission. We therefore trained FSPAD's draft models with their official code and under the settings specified in the FSPAD paper. Our experimental outcomes showed high consistency with the results published by the original FSPAD paper.
Regarding EAGLE-3, since EAGLE-3 doesn't 't provide pre-trained draft models for LLaMA3-8B. we used their official code train on the ShareGPT dataset, maintaining all other hyperparameters consistent with their paper. Training EAGLE-3 on ShareGPT alone required over 300 A100-80G GPU hours. Following their paper's full protocol (UltraChat-2008 + ShareGPT) would require approximately 2,400 GPU hours, which exceeded our computing resources. However, both EAGLE-3 and GRIFFIN used identical ShareGPT training data, ensuring fair comparison.
To ensure the validity of our comparisons, we aligned all key training settings, including dataset, optimizer, and hyperparameters, with those used by EAGLE-2 and HASS. For example, we matched the training procedure to HASS's three-step schedule, ensuring consistency and reliability across all experiments.
E Parameter Sizes of GRIFFIN'S Draft Models
For 7B, 8B, 13B, and 70B scale target models, the corresponding GRIFFIN draft model sizes are 0.41B, 0.42B, 0.65B, and 2.07B parameters, respectively. For Mixtral-8x7B, the draft model size is 0.45B parameters.
By comparison, the draft model sizes for EAGLE-2 and HASS are 0.24B, 0.25B, 0.37B, and 0.99B, while those for FSPAD are 0.42B, 0.43B, 0.67B, and 2.09B, across corresponding target models. For Mixtral-8x7B. the EAGLE-2 and HASS draft model size is 0.28B. Therefore, GRIFFIN's draft model contains between 0.17B and 1.08B more parameters than those of EAGLE-2 and HASS, but remains similar in size FSPAD's.
Despite this modest increase in parameters, GRIFFIN consistently achieves an average speedup improvement exceeding 8%, shown in Table 1. The additional parameter count incurs only
23

---

marginal computational overhead, which is amply justified by the significant gains in inference efficiency and overall performance.
F
Training Overhead of GRIFFIN
All the methods(GRIIFINN, EAGLE-2, FSPAD, and HASS) utilize the ShareGPT dataset for draft model training, ensuring an equal number training tokens across methods.
In terms of computational resources, GRIFFIN employs the same multi-stage training strategy as the state-of-the-ar HASS method, with both adopting three-step training regimen. For 7B, 13B, and 70B parameter models, HASS typically requires approximately 130, 220, and 500 NVIDIA A100 80G GPU hours, respectively, whereas GRIFFIN'S requirements are about 150, 250, and 600 NVIDIA A100 80G GPU hours.
Crucially, the draft model is trained only once but leveraged extensively during inference. Thus, in practical scenarios, the primary computational cost lies in the decoding phase. GRIFFIN offers roughly 8% improvement in speculative decoding speed HASS, meaning that the slight increase in training overhead well justified by the substantial gains in inference efficiency.
G Throughput of GRIFFIN
To evaluate GRIFFIN's
performance under batch sizes greater than 1, integrated it into the open-source vLLM framework, following the same speculative decoding interface used by EAGLE. All experiments were conducted on the LLaMtu-SB-Instruet model using the MT-Bench dataset, with decoding temperature of O. We report throughput (tokens per second) relative to the baseline vLLM decoding without any speculative methods.
Results. Table 9 summarizes relative speedups across different batch sizes. GRIFFIN consistently achieves higher throughput than both EAGLE and HASS for all evaluated batch configurations.
Table 9:
Throughput comparison under different batch sizes. Numbers denote relative speedup (x) over vanilla vI.LM decoding (1.00x baseline). All speculative methods were evaluated using sequential speculation with a maximum chain length of 2.
Batch Size
4
8
EAGLE
1.37x
1.32x
1.28x
1.18x
HASS
1.40x
1.35x
1.30x
1.20x
GRIFFIN
1.52x
1.45x
1.37x
1.25x
Implementation constraints. These evaluations were conducted under certain restrictions imposed by the current speculative decoding support in vLLM. Specifically, the implementation does not support tree-based drafting, which is key component of our full decoding algorithm. Consequently, all measurements used sequential speculation with a maximum chain length of 2. Therefore, the throughput values in Table are not directly comparable to the main-text results, which were obtained using our native decoding backend configured for tree-structured speculation.
Analysis. The observed trend of decreasing relative speedup batch size increases is expected and consistent with theoretical expectations. Larger batch sizes improve GPU utilization for the target model, reducing redundant computations and narrowing the efficiency gap between speculative and standard decoding. Moreover, as batches grow, the memory footprint and compute overhead associated with additional draft-model evaluations become increasingly significant, diminishing net throughput gains.
Discussion. Despite these challenges, GRIFFIN maintains substantial caaantages-achievee 6-11% % higher throughput than EAGLE and 4-8% higher than HASS across the tested batch sizes. These improvements demonstrate that GRIFFIN's alignment mechanism continues to yield benefits even in largc-batch, high-throughput inference regimes that are typical in production deployments.

---

H Breakdown of Decoding Latency
Motivation. While speculative decoding yields substantial efficiency gains, the overall speedup bounded by the additional computation required by the draft model. In Table of the main text, the observed speedup ratio (SR) is notably lower than the corresponding acceptance length. This discrepancy arises primarily from the non-negligible latency overhead of draft model inference.
Latency formulation. Let N denote the total number of tokens generated during decoding. For standard autoregressive decoding, the total decoding latency is
T N t,
(12)
where is the average per-token latency of a single forward pass through the target model.
For speculative decoding, at each cycle the target model verifies 7 candidate tokens produced by the draft model with rollout depth The corresponding total latency can be approximated as
T, N (t+d.tt)
(13)
where is the draft model's average per-pass latency. The resulting theoretical speedup ratio therefore becomes:
To t t
Speedup Ratio (SR)
(14)
Empirical estimation. Using the LLaMA3-8B-Inttrus model A100-80G GPU as repre- sentative setup, we measure the forward-pan pass latency the target model as approximately t = 25 ms and of the draft model as i = 1.5 If set the acceptance length
and draft rollout depth
= 6, Eq. (14) gives:
25
SR
5 3.68x,
25 + 1.5
which closely matches our empirical results. This quantitative agreement confirms that the latency contributed by draft model inference is the primary factor limiting the achievable speedup.
Discussion. Although draft model latency constitutes relatively small portion of the total budget, its accumulation over multiple rollout steps can substantially reduce overall efficiency, particularly for deep or large-d speculative configurations. Future efforts will explore techniques further mitigate this cost, such as:
Draft-model distillation reduce forward-pass complexity;
Asynchronous drafting that overlaps draft and target evaluations;
Kernel fusion and caching to minimize memory transfer overhead.
These analyses confirm that the gap between acceptance length and speedup ratio is quantitatively explained by draft inference latency, and they motivate further system-level optimizations.
Discussion with EAGLE-3
Motivation. EAGLE-3 recently proposed simplified speculative decoding framework that removes cnature-peedcction loss from the draft model objective. To examine the practical effect this choice and its interaction with our token-alignment mechanisms, we conducted two complementary studies: (i) an ablation of GRIFFIN in which the feature-level loss term was removed, and (ii) head-to-head comparison between implementation of EAGLE-3 and full GRIFFIN under matched training conditions.
Experimental setup. All experiments were performed on Ma-MMA3--B-Instruee using three standard evaluation suites MT-BBench HumanEval, and GSM8K decoding temperatures t=0 and t-1. For the EAGLE-3 baseline, we trained a draft model following their official open-source repository and hyperparameter settings, including identical optimizer, learning-rate schedule, and architecture. Due to computational constraints, training used the ShareGPT dataset only (excluding the additional 2hro corpus), which would otherwise require roughly 2,400 GPU hours to reproduce fully.
25

---

Results. Table 10 reports speedup ratio (SR) and acceptance length (7) across the three benchmarks and two temperature settings.
Table 10:
Comparison between GRIFFIN, its feature-loss ablation, and EAGLE-3 LLaMA3-8B-nnstruct This table reports results on standard L.MM benchmarks (MT-Bench, Hu- manEval, GSM8K) for temperatures 7 E {0, including speedup ratio SR and acceptance length T. Higher values indicate better performance.
Temperature
Temperature 1
Method
MT-Bench
HumanEval
GSMBK
Mean
MT-Beach
HumanBval
GSMBK
Mcan
7
SR
1
SR
5R
f
SR
SR
GRIFFIN
3.09
4.85
3,65
5.97
3.30
5.31
3.35
5.38
2.62
4.35
3.31
5.62
3.07
5.08
3.00
5.02
w/o FeatLoss
2.61
4.33
3.32
5.15
2.76
4.58
2,89
4.69
2.32
4,02
2,85
4.97
2.49
4.36
2.55
4.45
EAGLE-3
4.71
3.59
5.72
3.17
5.01
3.23
5.15
2.51
4.18
3.27
5.47
2,90
Observation 1: Importance of feature-level loss.
Removing the feature-prediction term from GRIFFIN produces a consistent degradation of roughly 10-15% in both SR and across all evaluation settings. This confirms that the feature-level supervision remains critical for stabilizing token alignment by enforcing coherence between the hidden representations of the draft and target models.
Observation 2: Comparison with EAGLE-3, Under identical training data and inference settings, full GRIFFIN outperforms EAGLE-3 all metrics. This indicates that EAGLE-3's removal of the feature loss does not yield an advantage at this data scale and that GRIFFIN'S 's alignment mechanisms lead to efficient speculative decoding when controlling for model and data size.
Observation 3: Applicability of GRIFFIN techniques. EAGLE-3 does not explicitly address to- ken misalignment during training decoding. In contrast, GRIFFIN introduces the Token-Alignable Draft (TAD) architecture and the Token-Alignable Training (TAT) procedure, both designed to mit- igate this issue. Importantly, these techniques modular and could in principle be applied to EAGLE-3-style draft models without altering their external decoding interface, potentially improving alignment and throughput performance.
Summary. This study demonstrates that feature-level supervision remains beneficial even when the draft model is trained with large-scale data, and that GRIFFIN's token-alignment strategy provides complementary improvements beyond what EAGLE-3 achieves. We hope these findings clarify the design impact of feature prediction loss and encourage future integration of token-alignment principles into other speculative decoding frameworks.
26


Reviews:
[{"review_id": "eserqk0c7D", "reviewer": "Reviewer_wEUZ", "summary": "This paper introduces a speculative decoding method, building on top of EAGLE, which improves the alignment of the draft model with the target model. This is done by: (i) training the draft model in stages of increasing number of forward passes, where in each stage the draft model is conditioned on its own generations rather than the teacher models generations; and (ii) changing the architecture of the draft model to increase the coupling between tokens and features. \n\nExperimental results show significant improvements in speed and acceptance lengths over vanilla speculative decoding as well as EAGLE, across a variety of model sizes and eval benchmarks. Ablation studies establish the importance of both the training strategies, as well as the choice of hyperparameters.", "questions": "- Does the draft model need to be trained from scratch? Or can the architectural modifications be applied on top of existing pretrained models?\n- How much training data is sufficient for updating the draft model?", "limitations": "No comment", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "- Overall, the paper is technically strong -- it identifies an important limitation of prior SD methods, and proposes two detailed mechanims for combating that.\n- The biggest strength of the paper lies in its empirical results, which show significant bumps in speed across many different SD setups. This makes the method practically useful.\n- Ablation studies and thorough and clearly establish the importance of the proposed methods.\n\nWeaknesses include:\n- The architectural changes to the draft model may limit the applicability of the technique, and without the architectural change the stage-wise training strategy does not seem to be more effective than prior work.\n- The paper is hard to follow for someone not super familiar with related works -- I had to go over the methods section multiple times. Adding conceptual summary before the technical details might help. Also the introduction mentions terms like \"top-k\", \"forward n\" without describing them clearly first.", "was_revised": true, "final_justification": "As noted above, I think the paper is strong and the authors response addresses most of my comments. So I will recommend acceptance.", "rebuttal": "Thank you for your thoughtful feedback. Below, we provide a point-by-point response to address your concerns.\n\n**1. Architectural changes may limit the applicability**.  We would like to clarify that GRIFFIN’s draft model preserves identical input and output interfaces to those used in EAGLE and HASS, ensuring that our approach does not impose restrictive architectural modifications, at least in terms of input and output, and thus maintains broad applicability which means the same applicability as EAGLE and HASS. \n\nMoreover, as shown in Table 1, GRIFFIN consistently demonstrates strong performance across a diverse set of model families (LLaMA, Vicuna, Qwen, Mistral) and sizes (7B, 8B, 13B, 70B), confirming the robustness and generalizability of our method beyond any specific architecture or scale. \n\nImportantly, the two core components of our approach—Token-Alignable Training (TAT) and the Token-Alignable Draft Model (TAD)—are designed to be modular and independently integrable. TAT can be applied to any draft model training pipeline to effectively resolve input misalignment issues, while TAD’s architectural improvements can be adopted in existing draft model designs with minimal structural changes.\n\n**2. Adding conceptual summary for clarity.** Thank you for your valuable feedback.\nTo improve clarity, we will add a **“Conceptual Overview”** section at the start of Section 3 in the revised manuscript. This overview will provide a high-level intuition behind the token misalignment problem, outline the logical flow from problem identification to our proposed solutions, and clearly map technical components to their respective roles.\n\nAdditionally, we will carefully define key technical terms such as “top-k” and “forward n” upon their first introduction to ensure readers can follow the methods more easily.  We believe these changes will significantly enhance the paper’s readability and accessibility without sacrificing technical rigor.\n \n**3. Draft model trained from scratch?**  GRIFFIN’s draft model does indeed need to be **trained from scratch**, rather than fine-tuned from existing pretrained draft models. This is primarily due to the **architectural modifications** introduced in our Token-Alignable Draft Model (TAD), which alter the internal computation flow in a way that is not compatible with the weight initialization of standard pretrained draft models of EAGLE and HASS. However, training the draft model remains substantially less expensive than training full-scale language models, and we believe the efficiency gains during inference justify the added training cost.\n\n**4. Training data size for updating the draft model**.  In our experiments, we followed established practice by using the **entire ShareGPT dataset**, which aligns with the standard scale adopted in prior speculative decoding work such as EAGLE and HASS.\n\nOur current paper does not include a systematic analysis of the relationship between training data volume and draft model performance. This is primarily due to computational constraints—specifically, limited GPU availability during the rebuttal period—which prevented us from conducting large-scale data scaling experiments.\n\nWith additional resources, we would pursue a more thorough investigation on this good direction, as we believe it is a critical step toward making speculative decoding stronger."}, {"review_id": "zwtj8Bk8ag", "reviewer": "Reviewer_RdHq", "summary": "The authors propose a modified framework for speculative decoding which compared to EAGLE and HASS (1) modifies the training by masking the loss for tokens where the prediction would not lie in the top-k, training the draft model to align with tree-based decoding, and by a modified draft model that has a Token-Guided Fusion (TGF) component on the input side merging information from the previous token features F_t and the current token x_t, and a Token-Enhanced Head (TEH) module which splits the draft model's hidden state into one F^p representation used for prediction and one f^R representation used as input in the next drafting token.\nIn experiments on MT-Bench, HumanEval and GSM8k using different Llama/Qwen/Mixtral models, the new approach consistently outperforms EAGLE-2, FSPAD and HASS on speedup ratio as well as acceptance length.", "questions": "While EAGLE-3 has come out after the ArXiv draft of this paper, it would be interesting to have a comparison if possible. Do you expect its performance to be roughly at the same level as HASS?", "limitations": "the case of batch sizes > 1 is not considered at all, which is in line with previous work\nThe authors mention higher training overhead compared to EAGLE, which I think is an interesting point that would otherwise be lost", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- speculative decoding is a well-established task and the baselines provided are strong an meaningful\n- empirical results show that the approach brings a useful and significant improvement\n- the authors provide an ablation of the training loss and draft model parts of their contribution, showing how each novel element performs bz itself\n- the authors promise to make their implementation and draft models available\n\nWeaknesses:\n - the case of batch sizes > 1 is not considered at all, which is in line with previous work but also excludes a significant area of use-cases", "was_revised": true, "final_justification": "(8/3) After reading the comments and rebuttals I tentatively believe that the authors will be able improve the technical discussion in the paper to address the main points of the questions raised, but would be looking forward for the authors confirming this with a list of changes they want to make in the paper itself.", "rebuttal": "Thank you for your thoughtful feedback. Below, we provide a point-by-point response to address your concerns.\n\n**1. Batch sizes > 1**. To evaluate GRIFFIN’s performance under batch sizes greater than 1, we integrated GRIFFIN into **vLLM**, following the same speculative decoding interface as EAGLE. We then measured throughput on **LLaMA3-8B-Instruct** using the MT-bench dataset with temperature set to 0, across a range of batch sizes. As shown below, GRIFFIN consistently outperforms both EAGLE and HASS at all batch sizes:\n\n| Batch Size | 2     | 4     | 8     | 16    |\n| ---------- | ----- | ----- | ----- | ----- |\n| EAGLE      | 1.37× | 1.32× | 1.28× | 1.18× |\n| HASS       | 1.40× | 1.35× | 1.30× | 1.20× |\n| GRIFFIN    | 1.52× | 1.45× | 1.37× | 1.25× |\n\nHere, the baseline (1.00×) refers to standard vLLM decoding without any speculative decoding methods. It is important to note that these results were obtained under specific constraints imposed by vLLM’s current speculative decoding implementation. Specifically, the setup does **not support tree-based drafting**, so all evaluations used **sequential speculation** with a **maximum chain length of 2**. As a result, these results are not directly comparable to those reported in the main body of our paper, which uses a different decoding backend and configuration.\n\nThe observed trend—**declining speedup as batch size increases**—is theoretically expected. As batch size grows, the target model benefits from better GPU utilization, reducing computational redundancy and diminishing the marginal gain from speculative decoding. Furthermore, large batches begin to approach memory and compute bottlenecks, making the overhead from additional draft model calls more significant relative to the baseline.\n\nDespite these challenges, GRIFFIN shows robust performance, delivering **6–11% higher throughput than EAGLE** and **4–8% higher than HASS** across all batch sizes. These results suggest that GRIFFIN’s advantages are preserved even under high-throughput, batch-inference settings, which are common in real-world deployments.\n\nWe will include this analysis in **Appendix G** of the revised manuscript, along with: (1) full implementation details of the vLLM integration; (2) head-to-head comparisons with other speculative decoding methods in batch settings; and (3) a discussion of potential optimization strategies for improving speculative decoding under large-batch conditions.\n\n**2. Comparison with EAGLE-3.** We compared GRIFFIN with EAGLE-3 on **LLaMA3-8B-Instruct** across three benchmarks (MT-bench, HumanEval, GSM8K) at two temperature settings:\n\n|         | MT-bench |          | HumanEval |          | GSM8K    |          | Average  |          |\n| ------- | -------- | -------- | --------- | -------- | -------- | -------- | -------- | -------- |\n| t=0     | $ SR $     | $ \\tau $   | $ SR $      | $ \\tau $   | $ SR$     | $ \\tau$   | $ SR$     | $ \\tau$   |\n| GRIFFIN | **3.09** | **4.85** | **3.65**  | **5.97** | **3.30** | **5.31** | **3.35** | **5.38** |\n| EAGLE-3 | 2.93     | 4.71     | 3.59      | 5.72     | 3.17     | 5.01     | 3.23     | 5.15     |\n| t=1     | $ SR$     | $ \\tau$   | $ SR$      | $ \\tau$   | $ SR$     | $ \\tau$   | $ SR$     | $ \\tau$   |\n| GRIFFIN | **2.62** | **4.35** | **3.31**  | **5.62** | **3.07** | **5.08** | **3.00** | **5.02** |\n| EAGLE-3 | 2.51     | 4.18     | 3.27      | 5.47     | 2.90     | 4.85     | 2.89     | 4.83     |\n\n- $SR$: Speedup Ratio; $\\tau$ : Acceptance Length\n\nSince EAGLE-3 doesn't provide pre-trained draft models for LLaMA3-8B, we used their official code to train on the ShareGPT dataset, maintaining all other hyperparameters consistent with their paper. Training EAGLE-3 on ShareGPT alone required over **300 A100-80G GPU hours**. Following their paper's full protocol (UltraChat-200K + ShareGPT) would require approximately **2,400 GPU hours**, which exceeded our rebuttal period resources. However, both methods used identical ShareGPT training data, ensuring fair comparison.\n\nGRIFFIN outperforms EAGLE-3 across all benchmarks and temperature settings. At $t=0$, GRIFFIN achieves **3.7%** higher speedup ratio and **4.5%** higher acceptance length compared with EAGLE-3. At $t=1$, GRIFFIN achieves **3.8%** higher speedup ratio and **3.9%** higher acceptance length compared with EAGLE-3. \nInterestingly, EAGLE-3's performance is **roughly comparable to HASS** levels, which aligns with your expectation. This suggests that while EAGLE-3 introduces architectural improvements, the performance gains are modest when training data is limited.\n\nWe will include our experimental results of EAGLE-3 in the revised manuscript."}, {"review_id": "C3deHjylrK", "reviewer": "Reviewer_Cfb2", "summary": "This paper aims to address the feature and token misalignment issue in EAGLE. Specifically, during the training of EAGLE draft model, the draft model takes the feature and output tokens of the target model as inputs, but during inference it takes the feature and output tokens of the draft model as inputs. This misalignment results in less optimal accuracy for the draft model. To overcome this issue, it proposes (1) a new training method that discard training data that has mis-alignment issue; (2) propose a new module for the draft model to prioritize token embeddings over predicted features. Experiments over state-of-the-art baseliens confirm the effectiveness of the proposed method.", "questions": "1. Can authors help me with my confusion: I understand that the misalignment this paper aims to solve is that during the training of EAGLE draft model, the draft model takes the feature and output tokens of the target model as inputs, but during inference it takes the feature and output tokens of the draft model as inputs. But when designing the training mask (m_t), the value of m_t is determined by the prediction of step t, instead of the input of step t, which doesn't make sense to me. To my understanding, the m_t = 1 if the draft model makes a good prediction,  and it will be computed in the loss; and m_t = 0 if the draft model makes bad prediction (not in top-k prediction), and it will be discarded. So the training process only counted the training samples the draft model perform well? Why this approach can help address the misalignment issue? It seems more reasonable to use m_{t-1} to determine whether to include the loss in step t.\n\n2. In the results of Table 1, there are many cases where the acceptance length is over 5 but the speed up ratio is only around 3. Can authors report the detailed breakdown of decoding latency of the draft and target model, and analyze what is the main bottleneck for the speed-up ratio?\n\n3. I think running draft model for n-forward pass during training will slow down the training speed, perhaps quite significantly, which is also mentioned by the authors. Can authors report the training time comparision of the proposed method and the baseline?\n\n4. I do not need authors to compare with EAGLE-3. But I want to discuss it with the authors. In EAGLE-3, they choose to drop the feature prediction in the draft model, which is better when the training data increases. I wonder how the authors think of their choice. Will the techniques proposed in this paper still be applicable to EAGLE-3?", "limitations": "The authors have discussed the limitation in their conclusion.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "S1. I think the idea behind the proposed method is overall reasonable. \n\nS2. The experiments cover state-of-the-art baselines and lots of base models. I believe the experiments are strong.\n\nW1. The novelty is not excellent, but I believe it is still a solid work.\n\nW2. I have some trouble understanding the design of token-alignable training (Sec 4.1). Please see my detailed questions below.\n\nW3. I have some more questions about the details of the paper. Please see the questions below.", "was_revised": true, "final_justification": "The authors addressed my concerns. So I maintain the positive score.", "rebuttal": "Thank you for your thoughtful feedback. Below, we provide a point-by-point response to address your concerns.\n\n**1. Mask**. Thanks. The training mask $ \\mathbf{m} _t$ was unfortunately misrepresented in our original text due to a typesetting oversight. The correct definition of Equation (3) should be $ \\mathbf{m} _t = \\prod _{i=t-n+1}^{t-1} \\bar{\\mathbf{m}} _i$. This formulation means that the loss mask at time step $t$ is determined by the **prediction quality of the previous $n-1$** draft tokens—not by the current prediction at $t$. Specifically, $ \\bar{\\mathbf{m}} _i = 1$ if the ground truth token at position $i$ is within the top-$k$ predictions of the draft model, and 0 otherwise. Therefore, $ \\mathbf{m} _t = 1$ only if all previous draft tokens are well-aligned; otherwise, the loss at position $t$ is masked out.\n\nThis design is motivated by the behavior of **speculative decoding**: once a misaligned token is encountered, **all subsequent draft tokens are rejected**. By mirroring this decoding logic during training, we ensure the model is only updated on token sequences that would have been **accepted in actual decoding scenarios**. In this way, the loss focuses on segments of the draft that contribute valid outputs, thereby improving the alignment between training and inference distributions.\n\nWe will revise the equation and accompanying explanation in our manuscript for clarity. \n\n**2. Breakdown of Decoding Latency**. Thank you for the thoughtful question. The key reason the **speedup ratio** is lower than the **acceptance length** in Table 1 is due to the **non-negligible overhead of draft model inference**.\n\nTo elaborate, let $N$ be the total number of tokens generated. Then the total latency for **autoregressive decoding** is $\\mathbf{T}_a = N \\cdot \\mathbf{t}$, where $\\mathbf{t}$ is the latency per forward pass of the target model. By comparison, the total latency for **speculative decoding** is $\\mathbf{T}_s = \\frac{N}{\\tau} \\cdot \\left( \\mathbf{t} + \\mathbf{d} \\cdot \\bar{\\mathbf{t}} \\right)$,  where $\\tau$ is the average number of tokens accepted per cycle, $\\bar{\\mathbf{t}}$ is the draft model's per-pass latency, and $\\mathbf{d}$ is the draft tree depth. Thus, the **speedup ratio** becomes:\n$$\n\\text{Speedup Ratio (SR)} = \\frac{\\mathbf{T}_a}{\\mathbf{T}_s} = \\frac{\\mathbf{t}}{\\mathbf{t} + \\mathbf{d} \\cdot \\bar{\\mathbf{t}}} \\cdot \\tau\n$$\n\nUsing **LLaMA3-8B-Instruct on A100-80G** as a representative example, the latency per forward pass of target model is approximately **20-30ms**, for draft model is **1-2ms**.  So, if we set $\\mathbf{t} = 25ms$, $\\bar{\\mathbf{t}} = 1.5ms$, $\\tau = 5$ and $\\mathbf{d} = 6$, we could roughly calculate the speedup ratio $SR = 3.68$ , which is closely matches our reported experimental results, confirming that the drafting overhead is the primary factor limiting speedup below the acceptance length. We will include a breakdown of draft vs. target model latency in the **Appendix H** in the revision. \n\n**3. Training time comparision**. GRIFFIN’s progressive training introduces some overhead due to multiple forward passes per training example. To quantify this, we compare the total training cost of GRIFFIN against HASS, a strong state-of-the-art baseline that also uses a multi-stage training schedule:\n\n| Model Size | HASS Training Cost      | GRIFFIN Training Cost   | \n| ---------- | ----------------------- | ----------------------- |\n| 7B         | ~140 A100-80G GPU hours | ~160 A100-80G GPU hours | \n| 13B        | ~240 A100-80G GPU hours | ~270 A100-80G GPU hours |\n\nWhile the training cost increases modestly, this overhead is amortized over large-scale deployment. Once trained, the draft model serves millions of inference requests. GRIFFIN yields up to 7% higher decoding speed over HASS at inference (Table 1), making it significantly more efficient in production.\n\nFor example, in a system serving ~10,000 requests/day, let alone the billion requests/day of ChatGPTs and other popular LLMs, the additional training cost would be offset by inference gains within one week of deployment. We discuss this trade-off in detail in Appendix F, and believe the training overhead is well-justified by the lasting efficiency gains in real-world usage.\n\n**4. Discussion with EAGLE-3.** Thank you for opening this discussion about EAGLE-3’s decision to drop feature prediction loss in the draft model. To understand the practical impact of that choice and the possible interaction with our own techniques, we ran both (i) an **ablation on GRIFFIN that removes feature-level loss** and (ii) a head-to-head comparison with a faithfully re-implemented **EAGLE-3** draft model on LLaMA3-8B-Instruct.\n\n|         | MT-bench |          | HumanEval |          | GSM8K    |          | Average  |          |\n| ------- | -------- | -------- | --------- | -------- | -------- | -------- | -------- | -------- |\n| t=0     | $SR$     | $\\tau$   | $SR$      | $\\tau$   | $SR$     | $\\tau$   | $SR$     | $\\tau$   |\n| GRIFFIN | **3.09** | **4.85** | **3.65**  | **5.97** | **3.30** | **5.31** | **3.35** | **5.38** |\n| GRIFFIN + remove feature loss | 2.61     | 4.33     | 3.32      | 5.15     | 2.76     | 4.58     | 2.89     | 4.69     |\n| EAGLE-3 | 2.93     | 4.71     | 3.59      | 5.72     | 3.17     | 5.01     | 3.23     | 5.15     |\n| t=1     | $SR$     | $\\tau$   | $SR$      | $\\tau$   | $SR$     | $\\tau$   | $SR$     | $\\tau$   |\n| GRIFFIN | **2.62** | **4.35** | **3.31**  | **5.62** | **3.07** | **5.08** | **3.00** | **5.02** |\n| GRIFFIN + remove feature loss | 2.32     | 4.02     | 2.85      | 4.97     | 2.49     | 4.36     | 2.55     | 4.45     |\n| EAGLE-3 | 2.51     | 4.18     | 3.27      | 5.47     | 2.90     | 4.85     | 2.89     | 4.83     |\n\n- $SR$: Speedup Ratio; $\\tau$ : Acceptance Length\n\n**EAGLE-3 Implementation Details**:\nFor EAGLE-3, we used their official open-source codebase to train a draft model on the ShareGPT dataset. We made every effort to follow their protocol:\n- Training protocol and hyperparameters were kept fully consistent with those reported in the EAGLE-3 paper (e.g., learning rate schedule, optimizer, batch size, model configuration).\n- The only difference is the scale of training data: due to computational constraints, we used only ShareGPT (rather than the much larger UltraChat-200K + ShareGPT combination), as training on the full protocol would require approximately 2,400 A100-80G GPU hours—far beyond our resources for the rebuttal period. Both GRIFFIN and EAGLE-3 used identical ShareGPT training data, ensuring fair comparison.\n\n**Key observations**:\n- Removing feature-level loss from GRIFFIN causes a sizable drop (**10%-15%**) in both Speedup Ratio and Acceptance Length, confirming that feature-level loss is still important for our token-alignment objective.\n- Under identical ShareGPT training data and settings, GRIFFIN still **outperforms** EAGLE-3. This suggests that EAGLE-3’s “remove feature-level loss” design does NOT provide an advantage at this data scale.\n\n**Applicability of GRIFFIN techniques to EAGLE-3**:\n- EAGLE-3 does not explicitly address **token misalignment**, whereas GRIFFIN’s Token-Alignable Draft Model (TAD) and Token-Alignable Training (TAT) tackle this issue directly.\n- Both TAD and TAT can in principle be integrated with EAGLE-3-style draft models without altering their external interface.\n\nWe will include a detailed discussion of these experimental results and their implications in **Appendix I** of our revised manuscript."}, {"review_id": "8ldThVGfGb", "reviewer": "Reviewer_vyph", "summary": "GRIFFIN is a speculative decoding framework that enhances the efficiency of large language model inference by explicitly addressing token misalignment and feature misalignment, overlooked issues in prior work. It introduces a token-alignable training strategy that uses dynamic loss masking to exclude misaligned tokens during optimization, and two extra modules: TGF and TEH.", "questions": "1. During speculative decoding, all drafted token after misaligned token should be rejected right? This means that actually there is no point to correct the following tokens after a misaligned token. So the goal of token alignment is actually drop \"hard\" prediction right? If the ground truth is not in top-k of draft model's prediction, then this loss is not counted as this prediction is too hard for the draft model. In this sense, the goal of \"token alignment\" is totally different from \"feature alignment\", where \"feature alignment\" is using draft-model-generated features, indeed aligning train input distribution with test input distribution. Please correct me if my understanding is wrong. If my understanding is right, then I think it lacks a clear explanation and analysis about this in the article. I think it even shouldn't be called \"token misalignment\", as this mask doesn't actually \"align\" the token.\n2. I didn't understand about why \"we gradually shift the model toward using its own outputs during training\" in Line 139 and \"By progressively adapting draft model to operate under its own predictions and aligning its training with decoding\" in Line 180. How \"gradually\" and \"progressively\" work? Doesn't in Equation 4, the target-model features is directly replaced with draft-model-generated features? \n3. I'm confused about the term: step and pass. In Line 141, \"where each training step n involves draft model performing n forward passes to predict n future tokens and their corresponding features.\", is this \"n\" in \"each training step n\" an index or a total number? And I don't understand why acceptance lengths can be longer than Step in Table 4. Doesn't \"step\" represent the maximum length of drafted tokens be predicted within a cycle?\n4. What is the definition of loss in Equation 2? How did it combine the l1 distance between feature and cross entropy between tokens. If there is a weighted factor, what is the weighted factor? Why this is not included in hyperparameters?\n5. The Speedup Ratio is drafting + verification or just drafting? How many GPU does verification used for parallel verifying? Why Line 221 says \"For consistency, all inference runs use one NVIDIA A100 80G GPU\"? If drafting and verificaiton are on same GPU, speculative decoding should be slower than autogressive decoding, as FLOPs increased.", "limitations": "Limitation is addressed in section 6.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1. The experiments are thorough, in which GRIFFIN shows large improvements upon other baselines.\n2. The motivation is reasonable in concept.\n\nWeakness:\n1. Please check the question.", "was_revised": true, "final_justification": "Authors solved my concerns and I raised my score to positive.", "rebuttal": "Thank you for your thoughtful feedback. Below, we provide a point-by-point response to address your concerns.\n\n**1. Goal of Token Alignment**. There are some misunderstandings. We would like to clarify the distinctions and objectives of our method. \n\n**Feature Alignment vs Token Alignment.** As illustrated in Fig. 2, **feature misalignment** occurs because during decoding, the draft model uses draft features $ \\bar{\\mathbf{F}} _ t $ to generate $ \\bar{\\mathbf{F}} _ {t+1} $, while during training it uses target features $ \\mathbf{F} _ t $. **Token misalignment** arises because during decoding, the draft model uses draft tokens $ \\bar{\\mathbf{x}} _ t $ to generate $ \\bar{\\mathbf{F}} _ {t+1} $, while during training it uses ground truth tokens $ \\mathbf{x} _ t $.\n\nFeature alignment addresses feature misalignment by replacing $ \\mathbf{F} _t$ with $ \\bar{\\mathbf{F}} _t$ during training. However, token misalignment cannot be directly resolved by simply replacing $ \\mathbf{x} _t$ with $ \\bar{\\mathbf{x}} _t$ because we follow the fixed-dataset setting adopted by EAGLE and HASS. We provide the detailed explanations for this constraint in Section 3, lines 124-131. Therefore, GRIFFIN simultaneously incorporates a **token-alignable training strategy** and a **token-alignable draft model** to mitigate misalignment. The training strategy employs a loss masking mechanism to exclude highly misaligned tokens during training, preventing them from negatively impacting the draft model's optimization. The token-alignable draft model introduces input tokens to correct inconsistencies in generated features.\n\n**Why Token Alignment ≠ \"Dropping Hard Predictions\".** \nThe goal of our masking is **not** to discard tokens whose ground-truth falls outside the top-$k$ of the draft model. Here we take one example for illustration. Assume the draft model have generated five draft tokens $ \\bar{\\mathbf x} _{t+1:t+5}$. If $ \\mathbf x _{t+1}$ and $ \\mathbf x _{t+2}$ are in top-$ k(\\bar{\\mathbf{x}} _{t+1})$ and top-$ k(\\bar{\\mathbf{x}} _{t+2})$ respectively, but $ \\mathbf{x} _{t+3}$ is not, we still compute the loss for **all three** positions $ \\bar{\\mathbf x} _{t+1:t+3}$. Tokens **after** the first mismatch ($ \\bar{\\mathbf x} _{t+4:t+5}$) are guaranteed to be rejected in verification; training on them provides no useful signal. These positions are therefore **masked**, not because they are “hard,” but because they are **inevitably discarded** during speculative decoding.\n\n**Corrections and Clarifications.** We also checked the whole work to find the reason which leads to your misunderstanding. Firstly, in line 158, we use \"aligned\" to indicate when the ground-truth token $ \\mathbf{x} _t$ appears within top-$k$($ \\bar{\\mathbf{x}} _t$). We agree that \"predictable\" would be more accurate than \"aligned\" in this context, since this \"aligned\" does not have the same meaning with \"token alignment\" which refers to the consistency between training and inference in terms of token level. Secondly, Eqn (3) has an incorrect superscript, and should be $ \\mathbf{m}_t = \\prod _{i=t-n+1}^{t-1}\\bar{\\mathbf{m}} _i$. This means we do not directly mask out the loss for draft tokens where $ \\bar{\\mathbf{m}} _{t}=0$.  We will correct them to enhance clarity and readability.\n\n**2. How \"gradually\" and \"progressively\" work?** \nAssume the draft model aims to predict $N$ draft tokens $ \\bar{\\mathbf x} _{t+1:t+N}$ by performing forward pass $N$ times:\n\n1. **Forward pass 1 (predict $ \\bar{\\mathbf x} _{t+1}$)**\n\n   • Inputs: ground-truth tokens $ \\mathbf{x} _{1:t}$ and target-model features $ \\mathbf{F} _{1:t}$.\n\n   • Outputs: token $ \\bar{\\mathbf x} _{t+1}$ and feature $ \\bar{\\mathbf F} _{t+1}$.\n\n2. **Forward pass $n$ (predict $ \\bar{\\mathbf x} _{t+n}$), $2 ≤ n ≤ N$**\n\n   • Inputs:\n\n   – ground-truth tokens $ \\mathbf{x} _{1:t+n-1}$\n\n   – target-model features $ \\mathbf{F} _{1:t}$ (unchanged)\n\n   – self-generated draft features $ \\bar{\\mathbf F} _{t+1:t+n-1}$ from the previous $(n−1)$ passes\n\n   • Outputs: token $ \\bar{\\mathbf x} _{t+n}$ and feature $ \\bar{\\mathbf F} _{t+n}$.\n\nThus, with each additional drafted token, the proportion of **self-generated** draft features fed back into the draft model increases. This is what we mean by **gradually/progressively** shifting the model toward using its own outputs.\n\n**3. Step & Pass, and why acceptance lengths can be longer than step.** \nWe thank the reviewer for raising this important question regarding the usage of the terms *step* and *pass*, and for pointing out the potential confusion related to Table 4 and Line 141.\n\nIn Line 141, the variable *n* refers to the **index of the training step** in GRIFFIN’s progressive training schedule, rather than the total number of forward passes. Specifically, each training step *n* is designed to train the draft model to perform *n* consecutive forward passes, thereby predicting *n* future tokens and their corresponding features. This approach helps the draft model learn to operate under conditions that closely simulate those it will encounter during speculative decoding, where it must rely on its own previously generated tokens and features. For example, at training step 3, the draft model is trained to generate three tokens in a sequence, conditioning on the outputs of its own previous passes.\n\nIt is also important to clarify why the **acceptance length** reported in Table 4 can exceed the value of the training step. The *step* index defines how the model is trained, not a hard constraint on how many tokens it can generate or how many the target model may accept during **decoding**. At inference, the draft model is allowed to generate a sequence of any length (e.g., >*N*), and the target model verifies them all at once. The **acceptance length** refers to the average number of draft tokens accepted by the target model per draft-verify cycle and is influenced by the quality of the generated tokens.  Accordingly, acceptance length can exceed the \"step\" value used during training. \n\nIn our revised manuscript, we will explicitly clarify that \"step\" refers to the training schedule rather than a decoding limit, and we will add explanatory notes to Table 4 to prevent this misunderstanding. We appreciate the reviewer’s thoughtful feedback, which helps us improve the clarity and precision of our presentation.\n\n**4. Loss and Weight in Eqn 2.** \nWe follow exactly the same dual-objective loss introduced in EAGLE. The complete definition of loss $\\ell$ in Equation 2 is:\n$$\n\\ell(\\bar{\\mathbf{x}}_t, \\mathbf{x}_t, \\bar{\\mathbf{F}}_t, \\mathbf{F}_t) = \\ell_1(\\bar{\\mathbf{F}}_t, \\mathbf{F}_t) + \\omega * \\ell_2(\\bar{\\mathbf{x}}_t, \\mathbf{x}_t)\n$$\nwhere $\\ell_1$ is the Smooth L1 distance and $\\ell_2$ is the standard cross-entropy loss. For fair comparison, we follow EAGLE and HASS and set the weighted factor $\\omega$ to 0.1. \n\nBecause we did not tune $\\omega$ and simply inherited the canonical setting from prior work, it was omitted from the hyper-parameter by oversight. We will provide full implementation details of the loss function in **Appendix C.2** of the revised manuscript.\n\n**5. Speedup Ratio and hardware setup.**  To clarify, the Speedup Ratio reflects the entire speculative decoding process, including both drafting and verification. All evaluation experiments were conducted on a single NVIDIA A100 80G GPU, except for LLaMA3-70B, which required two GPUs due to memory constraints. Both drafting and verification are performed on the same GPU, consistent with prior work (e.g., EAGLE, HASS), ensuring a fair comparison.\n\nRegarding the concern that speculative decoding might be slower than autoregressive decoding on a single GPU: this is a misconception. Although speculative decoding adds some overhead from drafting, it significantly reduces latency by verifying multiple tokens in parallel via a single forward pass of the target LLM model using batched causal masking. In contrast, autoregressive decoding requires one forward pass per token.\n\nThis efficiency has been consistently validated in prior work (e.g., EAGLE, HASS), and our results confirm it: GRIFFIN achieves up to 18% speedup over EAGLE-2 and 7% over HASS, using the same GPU configuration. We will clarify this in the final version for clarity."}]

Author rebuttal:

Meta-review: This paper proposes GRIFFIN, which improves on previous self-drafting techniques such as EAGLE which are used for inference time speedup with speculative decoding via drafting and verification. The authors study the "token misalignment" problem, where the draft model is trained on ground-truth tokens that are being paired with actviations from previuos drafting steps that might have predicted different tokens. The authors proposed to address this alignment by modifying the training to do actual samling from the drafter. Noe that this makes the training more expensive since it allow teacher forcing and the authors use a curriculum where each epoch the drafter samples gradually more steps, but only the prediction from the last step in each epoch is used for the loss computation. In addition, the authors also propose to mask out trajectories after a misalignemnt event between the drafter and main model (defined by top-k), claiming that future tokens in this trajectory might introduce unnecesary signal. Finally, the authors add a “token-alignable draft” (TGF) module that through ablations they find to help improve the drafting accuracy without increasing the drafting inference-time latency, leading to overall inference speedups compared to existing baselines.

Strengths:
1. The empirical results are a major strength of the paper, improving over previous EAGLE/ HASS methods (comparison to EAGLE-3 was added during the rebuttal)
2. Studying and discussing the token alignemnt challenge in self speluclation models. While teacher-forcing works well for LLMS for enabling efficient training, in the studied architectures there is the added element of concatenating previous activations with token embeddings, which is worth studying further.
3. Positive scaling results

Weaknesses:

The main weakness of the paper is its current presentation that is unclear (even for readers with relevant background) and imprecise. The paper must be revised to carefully address all the points discussed below, including:
1. Going over all equations and notations to make sure they accurately describe the implementation.
2. Provide enough technical details to accurately describe all the techniques discussed in the paper 
3. Clearly position the paper against other related works such as HASS and EAGLE, and in the method section describe which parts are adopted and which parts are new to this paper.
4. Be careful about terminology and notation, especically when using terms like “outputs” or “predictions” in this context- to be clear about what is meant. 
5. Fully state all details of the the evalution setup

Other weaknesses of the proposed method are mostly around the more expensive training due to avoiding teacher forcing in favor of sampling, some gaps in claims like whether TGF truly helps specifically via token alignemnt as claimed or just generally helps generally with accuracy in other ways, ablations on whether masking helped or not, and justification for using top-k for masking.
Decision: Accept (poster)