Bridging Draft Policy Misalignment: Group Tree Optimization  for Speculative Decoding
Authors: Shijing Hu
Jingyang Li
Zhihui Lu
Pan Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Speculative Decoding; Draft Tree Reward; Tree Optimization
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce GTO, which optimizes draft-tree reward to provably increase acceptance length and achieve >7% faster speculative decoding than EAGLE-3, while fine-tuning existing draft models.

Abstract: Speculative decoding accelerates large language model (LLM) inference by letting a lightweight draft model propose multiple tokens that the target model verifies in parallel. Yet existing training objectives optimize only a single greedy draft path, while decoding follows a tree policy that re-ranks and verifies multiple branches. This draft policy misalignment limits achievable speedups.  	
We introduce **Group Tree Optimization** (GTO), which aligns training with the decoding-time tree policy through two components: (i) Draft Tree Reward, a sampling-free objective equal to the expected acceptance length of the draft tree under the target model, directly measuring  decoding performance; (ii) Group-based Draft Policy Training, a stable optimization scheme that contrasts trees from the current and a frozen reference draft model, forming debiased group-standardized advantages and applying a PPO-style surrogate along the longest accepted sequence for robust updates. We further prove that increasing our Draft Tree Reward provably improves acceptance length and speedup.  	
Across dialogue (MT-Bench), code (HumanEval), and math (GSM8K), and multiple LLMs (e.g., LLaMA-3.1-8B, LLaMA-3.3-70B, Vicuna-1.3-13B, DeepSeek-R1-Distill-LLaMA-8B, Qwen3-8B), GTO increases acceptance length by \(7.4\%\) and yields an additional \(7.7\%\) speedup over prior state-of-the-art EAGLE-3. By bridging draft policy misalignment, GTO offers a practical, general solution for efficient LLM inference. Code and draft models are available at https://github.com/hsj576/GTO.

Published as a conference paper at ICLR 2026
BRIDGING DRAFT POLICY MISALIGNMENT: GROUP TREE OPTIMIZATION FOR SPECULATIVE DECODING
Pan Zhou Fudan University National University of Singapore 3 Singapore Management University 1zh0 fudaf.udncccmm edu.. on
Shijing Hu'
Jingyang Li
Zhihui Lu
jhu244m. fudan. edu. li.jingyangéuu nus. edu panzhoul smu edu. .g
ABSTRACT
Speculative decoding accelerates large language model (LLM) inference by letting a lightweight draft model propose multiple tokens that the target model verifies in parallel. Yet existing training objectives optimize only single greedy draft path, while decoding follows a free policy that re-ranks and verifies multiple branches. This draft policy misalignment limits achievable speedups. We introduce Group Tree Optimization (GTO), which aligns training with the decoding- time tree pol- objective
icy through two components: (i) Draft Tree Reward, a sampling- equal to the expected acceptance length of the draft tree under the target model, directly measuring decoding performance; (ii) Group-based Draft Policy Training, stable optimization scheme that contrasts trees from the current and a frozen ref- erence draft model, forming debiased group-standardizes advantages and apply- ing PPO-style surrogate along the longest accepted sequence for robust updates. We further prove that increasing our Draft Tree Reward provably improves ac- ceptance length and speedup. Across dialogue (MT-Bench), code (HumanEval), and math (GSM8K), and multiple I.LMs (e.g.. I.LMMA33. 1-8B, L.LMMA-33-7705 Vicuna-1.3-133
Qwen3-8B), GTO increases acceptance length by 7. 7.4% and yields an additional 7.7% speedup over prior state- of-the-art EAGLE-3. By bridging draft policy misalignment, GTO offers prac- tical, general solution for efficient LLM inference. Code and draft models are available https: //tthub..bb/ cwm/hs (576/ss557//TTO
INTRODUCTION
Large language models (LLMs) like GPTs (Achiam et al., 2023) and LLaMAs (Touvron et al., 2023a;b; Dubey et al., 2024) have achieved remarkable success in dialogue (Zheng et al., 2023), coding (Chen et al., 2021), and reasoning (Cobbe et al., 2021). Yet their standard autoregressive de- coding remains inefficient: each token requires a full forward pass, making inference both compute- intensive and latency-bound. Speculative decoding (Leviathan et al., 2023; Chen et al., 2023a) mitigates this by introducing a lightweight draft model to propose multiple tokens, which the target LLM verifies in parallel. This enables multi-token generation per target step, substantially reducing inference time.
Recent work has improved speculative decoding by refining draft model training. For instance, HASS (Zhang et al., 2024) enforces feature consistency to reduce hidden-state mismatches, GRIF- FIN (Hu et al., 2025) resolves token-level misalignments, and EAGLE-3 (Li et al., 2025) incorpo- rates training-time rollouts to better mimic decoding. However, they face a fundamental limitation yet: draft policy misalignment between training and decoding. That is, the training objective of draft model does not align with how draft sequences are actually generated and used during decod- ing, ultimately weakening the effectiveness of training for improving decoding performance.
Specifically, during training, given a context, the draft model is optimized to maximize the likelihood of generating the same token the target model (Li et al., 2024a;b; 2025; Zhang al., 2024). It treats drafting as a single-panh sequence prediction problem, and its corresponding optimal training- time draft policy is greedy drafting: select the highest- probability token at each draft step to form a single draft sequence (e.g.. the leftmost draft path in Fig. (a)). However, the practice decoding differs from greedy drafting, and indeed adopts drasting (Li et al., 2024b): shown in Fig. (a), it uses draft model to expand a draft tree containing multiple draft sequences, then re-ranks
"ooresponding author.

---

Published as a conference paper at ICLR 2026
Greedy Path
Accept Path
(1.00) Selected
(1.00)
0.6
0.7
Ascept
Verified by Target Model
(0.60)
has (0.40)
(0.70)
has (0.30)
Selected
Algept
Diseard
0.6
0.4
0.95
0,05
0.2
0.8
0. 6
0.4
(0.36) Pruned
the (0.24)
to (0.38)
some (0.02)
(0.14)
the (0.56)
to (0.18)
some (0.12)
Pruned
Selected
Pruned
Discard
Accept
Discard
Discard
(a) Druft Tree Constructed by Druft Model
(b) Draft Tree Verified by Target Model
Draft policy misalignment between training and decoding. (a) The tree is built by draft model at decoding: number on edge is the token probability predicted by draft model, e.g., "is" (0.6), and number in parentheses is current path confidence, e.g., is" (0.6=1.0 x 0.6). Training enforces a training-time greedy draft policy, following the locally best child and yielding the path "It - is + a" (confidence 0.36). At decoding, top-4 re-ranking compares sibling paths, where "It - has - to" (0.38) outperforms the greedy branch which is thus pruned (red). Training signal concentrated on a single greedy path is wasted when sibling branches win. (b) Target model verifies the tree with its own probabilities. It compares the confidence of each sequence, and accepts the sequence "It - is + the". Even when the greedy branch survives, target model may accept a different sibling.
Figure 1:
sequences using prediction confidences, and finally selects top-g tokens which are then verified by the target LLM. This decoding-tine time policy is fundamentally different: unlike the training- time policy focusing on single greedy draft path (the most left one in Fig. 1 (a)), it leverages multiple high-quality branches (the whole tree in Fig. (a)) to maximize the expected acceptance length.
This draft policy misalignment leads two characteristic failure modes: (1) greedy path pruning; and (2) verification mismatch. For (1), due to re-ranking and top-g selection, the optimal training- time greedy path may be pruned decoding if sibling branches achieve higher overall confidence. For example, in Fig. I(a), the greedy sequence "It is a" (confidence 0.36) is discarded in favor of the sibling "It has to" (confidence 0.38). Regarding (2), even when the greedy path survives pruning, target model may accept different branch, e.g., accepting "It is the" rather than the greedy "It is a" in Fig. (b). In both cases, training effort spent on the greedy path yields little decoding benefit. These failures also reveal a structural bottleneck: training encourages convergence to a policy that is effective and optimal only under single-path greedy drafting, but suboptimal for the tree-based strategy used in practice, causing training-decodle misalignment and limiting decoding efficiency. Bridging this gap is therefore crucial for realizing the full potential of speculative decoding in LLMs.
We empirically validate this misalignment using the EAGLE-3 draft model LLaMA-3. 8B (Dubey et al., 2024). As shown in Fig. 2(a), 19-34% of greedy paths are pruned during draft tree construction, and the finally accepted path matches the greedy one only 36-49% of the time. Even when accepted, the greedy path averages 3 tokens, shorter than the 5-6 tokens of the full tree (Fig. 2(b).. This confirms that greedy training overlooks globally optimal sequences, highlighting the severity of draft policy misalignment and its direct impact on speculative decoding efficiency.
Contributions:
To address the draft policy misalignment, we propose Group Tree Optimization (GTO), novel training algorithm for speculative decoding that explicitly optimizes the tree-based draft policy rather than a single greedy path. By aligning training with the actual decoding proce- dure, GTO ensures that draft models learn policies that directly improve decoding-time efficiency.
First, we introduce a drafi-tree reward that directly aligns training with the decoding-time policy. Unlike prior methods that optimize token-level accuracy (Li et al., 2025; Hu et al., 2025; Zhang al., 2024), GTO adopts the same rollout strategy used during decoding: the draft model generates a tree of candidate sequences, which is then verified by the target LLM. We define the reward as the expected acceptance length of the tree, direct measure of decoding efficiency. This shifts the objective from "predicting the next token correctly" to "producing draft trees that survive verification and extend accepted prefixes as far as possible," aligning the training goal with real decoding.
Second, we develop stable and effective draft policy training algorithm to maximize this draft- tree reward and thus boost decoding efficiency. Training is challenging because rewards are sparse, and high variance. GTO addresses this with group-based approach tailored to deterministic draft-tree rollouts. We sample small groups of trees under both the current draft model and frozen reference, and use their contrasts to construct debiased tree-level rewards that cancel
2

---

Published as a conference paper at ICLR 2026
Greedy Pall Accept Rate
Greedy Path
50
49
6.5
Draft Tree 641
5.5
EAGLE-2
Greedy
EAGLE3
5111
Probability (%)
Acceptance Langth
5.77
5.0
GTO
40
.18
34
5,0
Speedap
0,5
4,0G
30-
28
4.25
3.73
6.0
3.5-
20
19
3.5
3,07
3,19
3.0
10 SIT-bench
HumanEval
GSMSK
2.5 MTT-eecch
HumanEEm!
GSMBK
2.5
LL SB Deepeckk Vi.0nal3-138 LLAMAA33-TE
(b)
(0)
Figure 2: Experimental Results Draft Policy Misalignment between Training and Decoding. (a) Fraction of training-time greedy paths that are pruned during draft tree construction (orange bars) and fraction where the accepted path coincides the greedy path (yellow bars). (b) Accepted greedy paths also shorter: their average acceptance length is 3 tokens, compared to 5-6 for the entire draft tree. (c) Speedup Ratio Comparison of GTO and EAGLE-3.
position-specific difficulty. Within each group, standardized advantages normalize rewards contexts, reducing variance and improving credit assignment by highlighting which branches truly drive longer accepted prefixes. Finally, we optimize PPO-style clipped objective, defined over the likelihood ratio along the longest accepted sequence, ensuring robust and efficient training.
Finally, we validate GTO across dialogue (MT-Bench (Zheng al., 2023)), code (HumanEval (Chen et al., 2021)),, and reasoning (GSM8K (Cobbe et al., 2021)) benchmarks on LLaMA-3. 1-BBBB LLLMA-3.3--78B,
and Vieuna- 13B. GTO consistently improves acceptance length by 7.4% over E-G,, translating into an additional 7.7% speedup (Fig. 2 (c)).
DiSSSek-RR--istil-LLLaaaa
RELATED WORK
Speculative decoding accelerates L.LM inference by splitting each step into lightweight draft and verification stage (Sun al., 2024; Miao et al., 2024; Chen et al., 2023b; Kim et al., 2024; Liu et al., 2023). Existing methods vary in how drafts are produced and verified: prompt- and retrieval- based approaches (PLD, Lookahead, CLLMs) improve draft quality but degrade with scarce context (Saxena, 2023; Fu et al., 2024; Kou et al., 2024); tree-based verification (Sequoia, SpecExec) boosts acceptance but often increases compute (Chen al., 2024; Svirschevski et al., 2024); REST and Ouroboros reuse outputs or databases but depend on resource quality (He et al., 2023; Zhao et al., 2024); hybrid designs (Chimera, Glide) partially integrate the target model at extra cost (Zeng al., 2024: Du et al., 2024). Efficiency-oriented drafters span Medusa, Hydra, and RNN/Transformer- based models such as EAGLE-3, with methods like HASS and GRIFFIN addressing feature- and token-level mismatches (Cai et al., 2024; Ankner et al., 2024; Cheng et al., 2024; Li et al., 2024a;b; Zhang al., 2024; Hu et al., 2025; Li et al., 2025). Despite these advances, key limitation remains: draft policy misalignment, where training optimizes single greedy path but decoding verifies a tree of candidates. We propose GTO to align the training objective with the decoding-time tree policy, improving acceptance length and speedup. GTO complements existing methods and provides general solution to policy mismatch speculative decoding.
3 GTO: GROUP TREE OPTIMIZATION
To address the draft policy misalignmeni highlighted in Section 1, we introduce Group Tree Op- timization (GTO), a training framework that explicitly aligns the draft policy with decoding. The central idea is evaluate and optimize draft policies not single greedy path, but entire draft trees, using the same drafting procedure deployed at decoding. To this end, GTO consists of two key components: (i) drafi-tree reward that faithfully measures expected decoding performance in terms of accepted draft sequence length (Section 3.1), and (ii) a stable group-based optimization algorithm for training with this reward (Section 3.2). Below we introduce them in turn.
3.1 DRAFT TREE REWARD
The effectiveness of speculative decoding is governed by the length of accepted draft sequence: the longer the draft sequence accepted by the target model, the fewer verification steps are needed, and thus the greater the decoding efficiency. With the same draft model, higher expected acceptance
3

---

Published as a conference paper at ICLR 2026
length directly translates to higher speedup. This makes expected acceptance length the most faithful measure of practical decoding performance when using the same draft model.
To capture this, GTO climinates the traditional mismatch between training and decoding: instead of optimizing token-level proxies along a greedy path, we construct draft trees during training using the same decoding-time expansion and pruning policy (e.g.. EAGLE-2- multi-branch expansion, reranking and selection). The draft model is then optimized with respect to a tree-level reward that directly reflects its expected decoding-ting utility.
Formally, given a training prefix ka. context) X1s. we follow EAGLE-2, and construct a depth-d draft tree T, with the draft model M
T. G(M, x1:),
where G denotes the decoding policy. The policy 9 grows the tree in two stages.
(i) Layer-wise expansion. At depth E [l....,d} consider all frontier expansions (token edges) from the current layer. For each candidate expansion we compute global acceptance score. We then select the top-k token expansions across the entire layer according to the global acceptance score and expand draft tree only on these children. This global competition allows promising siblings to outcompete locally greedy choices and prevents early commitment a single path.
(ii) Global pruming and re-ranking. After reaching the maximum depth, we collect all leaves and re-rank them by the global acceptance score. We retain the top-g leaves and prune the rest.
The tree consists of N candidate sequences T. (St. S.,N each of length 4 S d may be different due to selection (pruning):
St, ( x
where { (x141.)..... 8+th,}} denotes the draft sequence St, Then, for each sequence, we define its expected acceptance length under the target model T:
4
L = Σ P(x X1a. 8411445115)
(3)
1=1
with
P(x++ja X1st, 8 = II T (xx+ka Xua, 80-1 k=1
(4)
Here, Lr,; the expectation how many tokens of S., will be accepted by target model T. This definition is sampling-free, while remaining directly tied to decoding performance.
Accordingly, we can average the expected acceptance length of all sequences in the tree to measure the overall decoding performance of the tree. However, since decoding utility depends which sequences (branches) survive pruning, we aggregate the sequence-level expectations with a smooth max (log-sum-exp), balancing differentiability with focus on the strongest sequences:
Fr R(TGT)
1 log 7)
Σ eepp(n (e.)(
(5)
where the temperature 17 >> interpolates between the maximum (7) oo) and the average (1) + 0) branch acceptance length. We set 1 in experiments, which yields a stable and informative reward and works very well in our all experiments. Ablation results in Table show this strategy is better than average all expected length or use the maximum length.
By training the draft model maximize R(T,), GTO ensures that the draft policy and training objective are fully aligned with decoding. Unlike prior approaches that rely on token-level log- likelihoods or greedy-path proxies, GTO directly optimizes for the expected acceptance length that governs speculative decoding speedup.
Theoretical guarantee. Importantly, improving the Draft Tree Reward provably increases the ex- pected decoding acceptance length, regardless of the target model's sampling temperature:
4

---

Published as a conference paper at ICLR 2026
Theorem (Maximizing Draft Tree Reward Guarantees Improved Expected Acceptance Length). Consider a draft tree T, and target model temperature 2 Let L (T₂) denote the expected acceptance length at decoding. Then:
(a) For T >> 0, if the draft tree reward increases, then ((I.)(()))) strictly increases.
(b) For = 0, rf increases, then B[ (T+)] = max; Lu also increases.
See its proof in Appendix A. This result establishes expected acceplance length the key link between training and decoding: optimizing the draft-tree reward directly improves speculative de- coding efficiency practice.
3.2 TREE REWARD OPTIMIZATION
Directly optimizing the tree-level reward challenging, particularly early in training when the draft model is weak and draft-token acceptance rates are low. In this regime, the tree reward is small and high-variance, making naive optimization inefficient and unstable. To address this, following LLM's two-phase training (pretraining and fine-tuning), GTO adopts two-phase group- based ap- proach: an optional warmup obtain a competent draft model, followed by a structured group-wise optimization that stabilizes and accelerates training. This design improves sample efficiency and can skip the warmup if a strong pretrained draft model is available. For example, practice, we can directly use the draft model well trained by EAGLE-3, GRIFFIN and HASS as the reference draft model, which plays a role as the Phase training.
Phase I: Draft model warmup. We first train reference draft model Mg using standard token- level objectives like the ones in EAGLE-3 and GRIFFIN. This phase provides a baseline model to stabilize subsequent group-based updates and can skipped when a sufficiently strong draft model exists, e.g., draft model well trained by EAGLE-3 and GRIFFIN.
Phase II: Group-based optimization of the draft tree reward. We now optimize the draft tree reward while ensuring stability and robustness. Inspired by group-based reinforcement learning methods (e.g., GRPO (Shao et al., 2024)). we sample groups of related examples and use group- wise advantage estimation to reinforce high-performing samples while suppressing underperform- ing ones. However, unlike standard RL, for a fixed prefix X the draft-tree generation (MM, x::) 1a) is effectively deterministic given the policy, limiting the utility of multiple rollouts from the same state. To enable variance reduction and within-context comparisons, we form groups from nearby posi- tions in the same sequence and optimize clipped likelihood ratio surrogate with group group- normalized advantages.
Grouping,
Let the training sequence be Xı:> (II (I1...., Tg), where is the sequence length. We partition positions into K non-overlapping groups of adjacent indices. Each group is defined by start index t and fixed group size m (with m [4, 8] in practice):
G (k) = {te, t + t + - 1} S (1.....s)
(6)
subject to
S to 5
t++1 2 m (non-overlap).
The number of groups K is determined by the available compute budget and the sequence length (upper bounded by [s/m]).
For every position e G we construct a depth- limited draft tree with the current draft model M using the decoding policy G:
T. G(M,xis).
(8)
By construction, indices within group are adjacent: for any i,j E (k) we have - < 1.
Consequently, the corresponding prefixes X1ss and X1.5 differ by at most m trailing tokens and share a long common context. Comparing tree-level rewards only within group therefore: (i) matches examples under nearly identical contexts, (ii) reduces variance in reward comparisons caused by position-specific difficulty, and (iii) yields reliable credit assignment across nearby
5

---

Published as a conference paper at ICLR 2026
prefixes. Intuitively, we aggregate draft trees from adjacent prefixes so that the within- Sithin-group differ- ences are small, enabling stable and sample-efficien learning signals.
Reward shaping and standardization. A key challenge in draft tree reward optimization is that raw tree rewards R(T,) exhibit systematic difficulty bias: some prefixes X1: are inherently harder to continue than others, leading lower acceptance rates regardless of draft quality. For instance, pre- fixes ending with complex mathematical expressions or rare tokens may consistently yield shorter accepted sequences, while simple conversational prefixes may achieve high acceptance even with suboptimal drafts. This bias confounds the learning signal and can cause the model to avoid chal- lenging contexts rather than improving on them.
To remove systematic difficulty bias across prefixes, we construct reference trees T, = G[Mo x1.)) dcbias the tree reward:
R; = R - R(T,).
(9)
where is the tree-level reward from Section 3.1. Within each group, rewards are standardized to stabilize updates:
R. mean ((
A,
(10)
std (Ry) ((a())))))))))))))))))))))
with small 5 > for numerical stability. Our ablation study (Table 5) demonstrates that without debiasing, the model training will becomes unstable due to high variance in gradient magnitudes, leading to worser performance in decoding.
Clipped likelihood-ratio objective. Let S, be the longest accepted sequence in T; under T, with length 4, Define per-token likelihood ratio (geometric mean) between M and Mo on S,:
log M(, x1+) - log Mo(S, x1:)) ) max(4,,
= exp
(11)
We then optimize -tOlstyle clipped surrogate over each group (Schulman al., 2017):
1
Σ min ss A., clip( 8,, - 1+ A.).
LGTO
(12)
m (EG
where clip(s, b) = maxfa. min(s, b}} and > 0 controls update magnitude.
Overall training objective. We combine the group-tree objective with token-level loss Looken using a scalar weight w:
Leaken + w LGTO-
(13)
Lioken denotes the token-level cross-entropy loss introduced in EAGLE-3 (Li et al., 2025) that matches the draft model M to the target model T under the same prefixes.
This two-phase group-based procedure transforms the dacoding- draft tree reward into stable and effective learning signal, enabling the draft model to reliably maximize expected accep- tance length and align training with practical decoding performance. Details are summarized in Appendix. B and Algorithm
4 EXPERIMENT
Models & datasets. We test GTO on a representative set of LLMs, including 1.aMA--31-IIssrue. 8B (Touvron et al., 2023b), LLaMA-3.3 Inst--ntttt (Touvron et al., 2023b), Vicuna- Vicuma-1.3-13B (Fan et al., 2025), DeepSeek-R1- -iseee-R---isssll-LMMaae All experiments are conducted on a single NVIDIA A 100 80GB GPU, except for L.LMMA333--705 which requires two GPUs. We benchmark performance on three widely used evaluation suites: multi-turn conversation (MT-Bench (Zheng et al., 2023)), code generation (HumanEval (Chen al., 2021)), and mathematical reasoning (GSM8K (Cobbe et al., 2021)).
(Guo et al., 2025) and Qwen3-8B (Yang et al., 2025).
Baselines & implementations. Vanilla autoregressive decoding serves as the baseline (speedup ratio x.00x). For comparison, include recent SoTA speculative decoding methods: SPS (with Vicuna-68M as draft) (Leviathan et al., 2023), PLD (Saxena, 2023), Lookahead (Fu et al., 2024), Medusa (Cai al., 2024), EAGLE (Li al., 2024a), EAGLE-2 (Li et al., 2024b), HASS (Zhang

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of speedup ratio SR and acceptance length standard LLM benchmarks with temperature I E {0,
Temperature =0
Temperature =
Model
Method
MTI-bench
HumanEval
GSM8K
Averuge
MTT-each
HumanEyal
GSM8K
Average
SRR
t
SR
r
SR1
r
18R 1
SRt = SR T
t
PLD
1.53
1.61
1.69
1.73
1.79
1885
1.67
1.73
Lookahead
L.6]
1.67
1.72
1.78
1.84
1.93
1.72
1.79
the acceptance conditions relaxed
LLaMA-3.1
EAGLE
1.73
2.97
2.43
3.26
2.04
3.06
2.07
3.10
1.62
2.39
1,97
3.08
1.92
2.89
1.84
2.79
Instruct
EAGLE-2
2.52
4,02
3,31
4.70
2.83
4.21
2.89
4.31
2.04
3.13
2.62
4.37
2.37
3.71
2.34
3.74
SB
GRIFFIN
2.95
4.68
3.73
5.90
3.15
5.16
3.28
5.25
2.29
3,90
3.24
5.39
2.66
4.67
2.73
4.65
EAGLE-3
3,27
5.77
3.68
6.41
3.41
6.02
6.07
2.37
4.51
3.07
5.73
2.88
5.37
2.77
5.20
GTO
3.44
6.15
4.17
6.95
3,59
6.47
3,73
6.52
2.49
4.70
3.17
5.92
3.02
5.75
2.89
5.46
SPS
1.91
2.24
2.18
2.52
L.74
2,00
1.94
2.25
1.59
1.81
1.73
1.99
1.47
1.75
1.60
1.85
Medusa
2.48
2.37
2.74
Vicuna-1.3 13B
Hydra
2.57
3.25
3.02
3.68
2.61
3.37
2.73
3.43
N/A, since the accerlance conditions are relaxed
EAGLE
3.23
2.14
3,06
2,48
2.35
3.37
2.32
3.30
EAGLE2
3,79
4.78
4.71
5.37
3.83
4.72
4.11
4.96
3.47
4.33
3.84
4.87
3.15
4.36
3.49
4.52
EAGLE-3
6.59
4.03
4.16
5.79
4.27
5.93
GTO
5.23
7.01
6.06
7.95
5.55
6.92
5.61
7.29
4.10
5.71
4.77
6.52
4.90
6.05
4.59
6.09
PLD
1.34
1,42
1.53
1.62
1.48
1.54
1.45
1.53
N/A, since the acceptance conditions are relaxed
DeepSeck-R1 Distill BBail 8B
Lockahead
1.52
1.61
1.64
1.71
1662
1,68
1.59
1.67
GRIFFIN
2.71
4.24
3.19
5.23
3.42
5.58
3.11
5.02
2.38
3.93
2.83
4.68
3.13
5.23
2.78
4.61
EAGLE-3
3.34
5.32
3.59
5,88
3.78
6.16
3.57
5.79
2.71
4.54
3,15
5.10
3.49
5.82
3.11
5.15
GTO
3.49
5.60
3.98
6.58
4,20
6.92
3.89
6.37
2.76
4.59
3.34
5.44
3.71
6.50
3.27
5.51
1.43
1.51
LLaMA-3.3
Lockahead
1.58
1.66
1.71
1.79
1.73
1.82
1.67
1.76
N/A, since the acceptance conditions are relaxed
Instruct
70B
GTO
EAGLE-3
3,97
3.78
5.40
5.56
4.41
4.68
6.51
4.11
6.25
4.22
6.14
3,90
3.68
5.18
5.34
4.05
4.21
6.20
3.88
4.07
5.65
5.82
3.87
4.06
5.56
5.78
Qwen-3
EAGLE-3
2.37
4.53
2.56
4.88
2.59
5.02
2.50
4.81
2.24
4.23
2.32
4.77
2.51
4.84
2.36
4.61
8B
GTO
2.49
4.84
2,78
5.37
2.88
5.51
2.72
5.24
2.29
4.37
2.49
5.03
2.63
5.17
2,47
4.86
al., 2024), GRIFFIN (Hu al., 2025), and EAGLE-3 (Li al., 2025). Whenever available, we rely on public implementations and strictly reproduce their decoding policies and hyperparameters.
By default, GTO initializes its draft model from the one provided by EAGLE-3. To assess com- patibility, we also experiment with draft models trained by other approaches (see Table 2). The initialized draft models are then fine-tuned with GTO on the ShareGPT dataset (Chiang et al., 2023), except for the reasoning model on OpenThoughts- 1 14k-math dataset (Guha et al., 2025). See additional training details for GTO in Appendix B, and details for the baselines in Appendix C.
-aa-Seek-R--istill--..
8B, which is also fine-tuned
Metries. For fairness and consistency, we follow priors, e.g., HASS, GRIFFIN, and EAGLE-3, and fix the batch size to and evaluate under decoding temperatures T {0, 1}. Same as prior works like EAGLE-3, GTO is lossless and can preserve output quality. Thus, we focus on two efficiency metrics: (i) Speedup Ratio (SR)- the runtime acceleration relative vanilla decoding, and (ii) Acceptance Length (T) (r)- the average number of tokens accepted per draft-verification cycle.
4.1 MAIN RESULTS
Comparison with SoTAs. We report the acceptance lengths (7) and speedup ratios (SR) of GTO and all baselines across three benchmarks in Table One can observe that GTO consistently outper- forms all baselines, including SoTA EAGLE-3, across all datasets, models, and temperature settings. On average, each GTO trafting-verrificann
eycle accepts 6-7 tokens, compared 5-6 tokens for EAGLE-3. As a result, in terms of tangible wall-clock speedups, GTO improves the runner-up EAGLE-3 by 7.7% for temperature zero and 5.6% for temperature one an average across four evaluation models, while preserving the lossless property of speculative decoding.
Specifically, on the multi-turn conversation benchmark (MT-Bench), GTO achieves steady gains across all models. For example, with LLaMA-3.1 8B at T-0, GTO improves the speedup ratio by 5.2% over EAGLE-3, and by 5.1% at T=1. Vicuna- 1.3 13B shows even larger gains, reaching 8.1% T=0 and 1.7% at T=1. For code generation (HumanEval), the improvements pronounced. With LLaMA-3.1 8B, GTO yields a 13.3% speedup increase at T=0 and 3.3% at T=1. DeepSeek-RI 8B follows the same trend, achieving 10.9% and 6.0% improvements at T-0
more
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of speedup ratio (SR) and acceptance length (r) when respectively using draft models trained by GRIFFIN and HASS as initialization of GTO.
Temperature
Temperature
Model
Method
MTT-bench
HumanEval
GSMBK
Average
MT-beach
HumanEval
GSM8K
Average
19R1
SR1
SR1
ISR
SR
r
SRt
T
SR 1
[SR 1
T
LLaMA-3
GTO
GRIFFIN
3.28
3.09
5.17
4.85
3.65
4,03
5.97
6.44
3,53
3,30
5.73
5.31
3.61
3,35
5.78
5.38
2.62
2.74
4.35
4.54
3.47
3.31
5.95
5.62
3.23
3.07
5.41
5,08
3.00
3.15
5.30
5,02
Instract
8B
HASS
2.75
4.63
3.51
3.09
3.12
2.41
4.15
4.90
4.82
GTO
2.95
4.96
3,86
6.19
3.33
5.47
3.38
5.54
2.54
4.36
3.23
5.69
3.06
5.25
2.94
5.10
LLaMA-2
GRIFFIN
GTO
3.12
5.11
3.82
3.61
5.93
3.27
3.10
5:27
3,28
5:14
2.81
1.81
5.12
3.33
5.63
3.06
5.26
5.62
3.07
5.57
5.23
Chat
7B
HASS
2.97
4.97
3.46
5.69
3,06
5.12
3.17
5.26
2.72
4.64
3.18
5.22
2.83
5.0%
2.91
4.98
GTO
3,13
5.15
3,64
5.95
3,15
5.31
3.31
5.47
2.84
4.82
3.41
5.69
3,09
5.34
3.11
5.28
Table 3: Ablation of draft tree reward aggregation on LLaMA-3.1 8B.
Temperature 0
Temperature = 1
Method
MTT-bench
HumunEval
GSM8K
Average
MTT-bench
HumanEval
GSMBK
Average
SR1
Si 1
rt
SR1
rt
SR
SRR
SR1
T
SR 1
1
SS
GTO (LSE)
3,44
6.15
4.17
6.95
3.59
6.47
373
6.52
2.49
4,70
3.17
5.92
3.02
5.75
2.89
5.46
Max
3.38
6.05
4.06
6.80
3.52
6.36
3.65
6.40
2.46
4.65
3.12
5.84
2.97
5.66
2,85
5.38
Sum (Average)
3.29
5.92
3,95
6.62
3.42
6.18
3.55
6.24
2.41
4.56
3.04
5.72
2,90
5.55
2.78
5.28
and 7=1, respectively. These results highlight the effectiveness of GTO's tree- based optimization for structured generation tasks such as coding. On mathematical reasoning (GSM8K), GTO again surpasses EAGLE-3 across all configurations. For instance, with DeepSeek-RI 8B, GTO delivers an 11.1% speedup improvement at T-0 and 6.3% T1 The strong results on GSM8K suggest that GTO's draft-tree reward effectively captures sequential reasoning patterns critical for mathematical problem solving.
The results across diverse tasks and models highlight the versatility and robustness of GTO. The con- sistent improvements over the SoTA EAGLE-3, even at different temperatures, underscore GTO's effectiveness in handling varying levels of stochasticity in token predictions. Notably, the perfor- mance gains are more pronounced at temperature T across most settings, suggesting that GTO's deterministic tree optimization particularly benefits greedy decoding scenarios.
Compatibility evaluation. To further test compatibility and transferability, we evaluate GTO with draft models not initialized by EAGLE-3. Specifically, fine-tune the draft models from two ef- ficient speculative decoding -rtthodd-GRIFFNN and HASS using GTO, and evaluate them under identical configurations on LLLMM---IInstrucc-- and LLaMA-2-Chat--7
As shown Table 2, both GRIFFIN+GTO and HASS+GTO achieve consistent gains over their baselines. At TO, GRIFFIN+GTO improves the average speedup ratio (SR) and acceptance length (T) by 7.8% and 7.4%, respectively, while HASS+GTO improves them by 8.3% and 8.0%. At T==, GRIFFIN+GTO increases SR and by 5.0% and 5.6%, and HASS+GTO by 4.6% and 5.8%. These results validate GTO'' compatibility and transferability across distinct draft backbones, establishing tree-policy misalignment.
it as a general and effective approach for bridging the
4.2 ABLATION STUDY
Aggregation Operator. We ablate the aggregation operator in the Draft Tree Reward (Sec. 3.1) on 1-MMA--.-IInstrucc Our method employs the smooth maximum via log-sum-exy (LSE), which preserves differentiability while emphasizing strong branches (Eq. (5)). We compare against two alternatives under identical settings: (i) Sum (Average): r = + > Lr,i, treating all branches equally; (ii) Max: r xmax max; Lt,,, focusing only on the best branch but non-smooth.
Across all benchmarks and decoding temperatures, LSE aggregation (GTO) attains the best speedup ratio (SR) and acceptance length (r). At T=0, GTO improves the average SR by 2.1% over Max and 4.8% over Sum, with comparable gains in T. At T1,, the advantage remains, with SR gains of 1.4% over Max and 3.8% over Sum, again accompanied by consistent improvements in r.
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation of grouping size m LLaMA-3.1 8B.
Temperature
Temperature =
Method
HumanEval
GSMEK
Average
MTI-bench
HumanEval
GSMBK
Average
SR t
rf
SR 1
SR1
r.
S
SR1
rt
SR 7
r
SR1
SR
3.32
5.94
4,02
6.71
3.47
6.25
3.60
6.30
2,40
4.54
3.06
5.71
2.91
5.55
2.79
5.27
3.42
6.12
4.15
6.91
3,57
6.44
3.71
6.49
2.48
4,68
3.15
5.89
3.01
5.72
2.88
5.43
m
(GTO)
3.44
6.15
4.17
6.95
3,59
6.47
3.73
6.52
2.49
4.70
3.17
5,92
3,02
5.75
2.89
5.46
16
3.27
5.84
3,96
6.00
3.41
6.15
3.55
6.20
2.37
4.47
3.01
5.62
2.87
5,46
2.75
5.18
32
5.66
2,66
Table 5: Ablation of reward debiasing with a reference model on LLaMA-3.1 8B.
Temperature =
Temperature
Method
MT-heech
HumanEval
GSM8K
Average
MT-hench
HumanEval
GSM8K
Average
SS 1
SR
SR 7
T1
SR
SR
5R
8R
5R
GTO (Dehiased)
3.44
6.15
4.17
6.95
3.59
6.47
3.73
6.52
2.49
4.70
3.17
5.92
3
5.75
2.89
5.46
w/o Dehiasing
3,30
5,78
3,84
6.62
3,50
6,23
3,55
6.21
2.39
4.53
3.03
5.64
2,87
5.35
2,76
5.17
These results highlight the trade-offs of alternative operators: Sum dilutes signal by averaging weak branches, while Max is brittle and non-smooth, overfitting to a single path with poor gradient cov- crage. In contrast, LSE interpolates between them, providing a stable and selective objective that better aligns with decoding- timein re-ranking and pruning.
Group Size. We ablate the group size m in Tree Reward Optimization (Sec. 3.2) on LLaMA-3.1 Instruct 8B with E {1,4. 4. 8, 16, 32). As shown in Table 4, the default m= of GTO achieves the best average SR and т, while m= is within 1%, indicating a stable plateau. In contrast, m=11 and m=16 show clear degradation, and m=32 performs worst.
Small groups (e.g., m=1) suffer from noisy, context-misaligaee rewards, weakening credit assign- ment. Large groups (c.g., -,, 2 16) span longer contexts, introducing drift and bias that hurt learning. Thus, moderate sizes (m E [4, ()) strike the hest balance between variance reduction and context alignment, yielding the most reliable gains in SR and T.
These observations align with the theoretical insights from the GRPO (Shao et al., 2024), which shows that very small groups suffer from high-variance and unstable updates, while very large groups suffer from signal attenuation and slower convergence due to excessive averaging. In our GTO experiments,
also observe this qualitative pattern. Group sizes in the range of 4-8 consis- tently provide the best balance: they significantly reduce variance relative to size 1, yet still preserve enough reward contrast produce strong learning signals. Larger groups (e.g., size 32) remain stable but deliver noticeably weaker improvements due to the normalizationinnduee compression described above.
Reward Debiasing. We ablate the reward shaping and standardization step (Eq. (9)) in Tree Re- ward Optimization on
Debiasing computes a variaee reward by subtracting the tree-level reward of a frozen reference draft model MM (Phase I) from the current model M for matched prefixes, reducing variance and improving credit assignment. We compare our default GTO (Debiased) against a variant that omits this subtraction (w/o Debiasing), with all other settings fixed.
As shown in Table 5, debiasing consistently improves both SR and At T=0,, GTO achieves +5.0% SR and +5.1% t over w/o Debiasing; at T==,, the gains are +5.6% and -4.7%. Without debiasing, rewards are noisier and context-dependent, yielding weaker draft policies and shorter acceptance lengths.
Continual training. To ensure that the performance gains of GTO stem from our proposed al- gorithmic improvements rather than the additional computational budget, we introduce a continual training baseline. Specifically, we further fine-tune the vanilla EAGLE-3 draft model for addi- tional 200 A A100-80G GPU hours. This baseline employs the exact same training data and strictly follows the original EAGLES E--3 training recipe for the DeepSeek- -iDissl---L-ssss-LLLLaaM. target model.

---

Published as a conference paper at ICLR 2026
Table 6: Ablation of continual training draft model.
Temperature =
Temperature = 1
Method
MT-hench
HumanEval
GSMSK
Average
MT--benh
HumanEval
GSMSK
Average
SR t
SR
SR 1
522
r
SR
r
SR 7
rt
SR
r
S8R 1
r
GTO
5.60
6.92
3.34
5.44
327
5.51
EAGLE-3
3.34
5.32
3,59
5.88
3.78
6.16
3,57
5.79
2,71
4.54
3.15
5.10
3.49
5.82
3.11
5.15
continnal training
3.37
5.38
361
3.1
6.24
3.00
2,70
4.83
3.16
5.15
3.54
3.13
5.20
As reported in Table 6, GTO achieves consistently higher speedup (approximately 4%-7% on average) across various benchmarks and temperatures compared to this continued-triining base- line. Furthermore, we observe that the continual training baseline performs almost identically to the vanilla EAGLE-3, This behavior is expected: because the training dataset is already encompassed within EAGLE-3's original training mixture, the draft model is already near convergence on this distribution. Without the introduction of novel data, allocating additional compute the standard training objective merely reinforces an ddready-optimizee model, yielding negligible improvements.
These findings confirm that the efficacy of GTO does not derive from simply extending training time or data exposure. Instead, the improvements are fundamentally algorithmic, rooted in GTO's ability to explicitly align the draft tree policy with the target acceptance dynamics, thereby directly resolving the policy misalignment that typically bottlenecks speculative decoding.
CONCLUSION
In this paper, we proposed Group Tree Optimization (GTO) to bridge the draft policy misalign- ment between training and decoding. GTO introduces decodiingfaithhull faithful Draft Tree Reward that directly optimizes the expected acceptance length and stable group-based optimization that con- trasts current and reference trees, standardizes advantages across nearby contexts, and updates via a PPO-style clipped surrogate along the longest accepted sequence. Extensive evaluations across diverse LLMs and datasets show that GTO consistently outperforms SoTAs, achieving the highest speedup ratios and acceptance lengths.
Limitations. GTO increases training-time time compute due its two-phase procedure and the need to construct and evaluate grouped draft trees during training. Nevertheless, GTO is model-agnostie and complementary to existing speculative decoding methods: it can directly fine-tuned on top of pretrained draft models (e.g., EAGLE-3, GRIFFIN) without architectural changes or modifications to the verification stack. practice, the draft model is trained once, whereas decoding dominates the runtime in real-world deployments; the added training cost is therefore amortized by improved inference efficiency. our experiments, GTO improves the speedup ratio by more than 7% over EAGLE-3, making the extra training cost reasonable trade-off for sensitvvee applications.
ACKNOWLEDGEMENT
This work was supported by the Yangtze River Delta Science and Technology Innovation Com- munity Joint Research Project (YDZX20233 100004031) and the Singapore Ministry of Education (MOE) Academic Research Fund (AcRF) Tier 1 grant (Proposal ID: 25-SIS S-SSSSMS---0)) Any opin- ions, findings and conclusions or recommendations expressed in this material are those of the au- thor(s) and do not reflect the views of the Ministry of Education, Singapore.
ETHICS STATEMENT
GTO improves efficiency of large language model decoding. Nevertheless, faster generation could increase the throughput of undesirable content if deployed without safeguards. We recommend deploying GTO only with established safety measures (content filters, rate limiting, audit logging, and red-teaming) and within the original safety and usage policies of the underlying models.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Sequentially-depen draft heads for medusa decoding. arXiv preprint ar 24x::2402.05109 2024.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 2024.
Charlie Chen, Sehastian Borgeaud, Geoffrey Irving, Jean- Caan-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01338, 2023a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 3xXiv:2107.03374 2021.
Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and bardware-awaad speculative decoding. arXiv preprint arXiv:2402.12374. 2024.
Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chen-Chuar Chang, and Jie Huang. Cascade speculative drafting for faster Ilm inference. arXiv preprint arXiv:2312. 1462, 2023b.
Yunfei Cheng, Aonan Zhang. Xuanyu Zhang, Chong Wang, and Yi Wang. Recurrent drafter for fast speculative decoding large language models. arXiv preprint urXiv:2403.09919 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
/ wwl.mys..rg/blooogo..
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yonggi Li, Shenggui Li, Kai Xu, Liqiang Nie. Zhaopeng Tu, et al. Glide with a cape: A low-hassle method accelerate speculative decoding. arXiv preprint arXiv:2402.02008 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Chenghao Fan, Zhenyi Lu, and Jie Tian. Chinese-vicuna: A chinese instruction foolloiin llama- based model. arXiv preprint arXiv:2504.1227 2025.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm infer- ence using lookahead decoding. arXiv preprint arXiv:2402.02253 020507 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang, Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde. Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill,
11

---

Published as a conference paper at ICLR 2026
Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL https::////. wtwpaaiiivvvvoo/...//aaa org/ abs /et.//555....2.20
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: cetrieval-based speculative decoding. arXiv preprint arXiv:2311.08252, 2023.
Shijing Hu, Jingyang Li, Xingyu Xie, Zhihui Lu, Kim-Chuan Toh, and Pan Zhou. Griffin: Effective token alignment for faster speculative decoding. arXiv preprini arXiv:2502./1018. 2025.
Schoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36, 2024.
Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Clims: Consistency large language models. arXiv preprint arXiv:2403. 00835, 2024.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint ar arXhv:2401.15077 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 16858, 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Fagle-3: Scaling up inference acceler- ation of large language models via training-time test. arXiv preprint 201000500,,0840, 2025.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. arXiv preprint arXiv:2310.0717. 2023.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, al. Specinfer: Accelerating large lan- guage model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pp. 932-949, 2024.
Apoorv Saxena. Prompt lookup decoding, November 2023. URL https:://gtthtt com/ apoorvumang hwtp:p/ppppppppoee wwwoppttoookusute www.-pookkkceeeoo decluing/.d.dii///....
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1xXiv:/707.000 170..06347 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03330, 2024.
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024.
Ruslan Svirschevski, Avner May. Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive 1lm inference consumer de- vices. arXiv preprint (tt::22060025322 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023a.
12

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint w0Xiv:2505.09388 2025.
Ziqian Zeng, Jiahong Yu, Qianshi Pang, Zihao Wang, Huiping Zhuang, Hongen Shao, and Xiaofeng Zou. Chimera: A lossless decoding method for accelerating large language models inference by fusing all tokens. arXiv preprint urXiv:2402. 15758, 2024.
Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized representations for speculative sampling. arXiv preprint arXiv: 15766, 2024.
Weilin Zhao, Yuxiang Huang, Xu Han, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. Ouroboros: Speculative decoding with large model enhanced drafting. arXiv preprint arXiv:2402.13720, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
13

---

Published as a conference paper at ICLR 2026
A PROOF OF THEOREM
We first make explicit the objects in play. Let the draft tree at step é have N branches (root-to-lcaf paths) indexed by E [A]. For each branch i, let 2i,1:f, denote its token sequence up depth &, and let
L {0, (0.1,...,d)
denote the (random or deterministic) number of consecutive tokens, starting at the current prefix, that the target model would accept if branch were proposed. The draft-tree reward is the smooth maximum
N
log
enbe,s)
with
0,
which satisfies the standard bounds
max Lt, S I max Lt,s + log N.
(1)
17
For decoding, define for each j 2 the event
(T₂, fat least tokens are accepted decoding).
Then the expected acceptance length under target temperature I can be expressed as
B dec (To) = Σ (T)))
We will use the following elementary monotonicity fact.
Lemma (Coordinate-wise monotonicity of acceptance probability). Fix a draft tree topology and branch token sequences (Zi,1d;; H. For any j 2 the event (TT)) can be written the union
N
(T,) = U
Biss
Bij = {the target rollout matches 2i.1:5.-
1=1
If we increase a single coordinate Le,; by A E N (keeping other Lt, fixed), then for each j E [L, 1,...., Lu,i+ }}, the union gains a
Buj and hence
(E)() is non-decreasing.
Moreover, ifT > 0 (softmax sampling with strictly positive support over tokens), then TP (Bij) 0 and thus
increases strictly for those j.
Proof sketch. For each i, the event B,,, corresponds to the target producing the specific j-token prefix Z,,1;-- Increasing Lt,: by A adds new prefixes at depths L,,+1,.... LA,+A, hence enlarging the union. Under T > O, each concrete token sequence has strictly positive probability under softmax LM, the probability mass added is positive. Disjointness the level of exact token sequences follows from the tree structure: no two distinct branches share the same length-j token prefix, so B is not subset of Ukyz BR
We now prove the two cases in Theorem 1.
Proof of Theorem 1. (a) T > The function r; = 7 log Σ, entr.:)) is strictly increasing in each coordinate L,,, Because Lt,; are integer-valued lengths, any increase in r, implies that at least one coordinate Lr, increases by an integer A > 1.' By Lemma for each newly covered depth j E
Ly, + A) have (T))) increases strictly (because T confers strictly positive mass on the corresponding prefix event). Summing these strictly positive increases over
Forrallyy along any path that increases the first time changes must coincide with an increment in least discrete coordinate.
14

---

Published as a conference paper at ICLR 2026
j and possibly over multiple improved branches (if several coordinates increased) and invoking equation? yields
B LT des (T,))
(T,)
increases strictly whenever rę increases.
(b) set be the unique greedy target trajectory. Then L equals the longest common-prefix length between branch i and s*, and
E L dor (T:)] = max Lç,e-
Using the smooth-max bounds equation with M max; Lr we have
M S r. s M 1 log N. Consequently, if increases by more than the residual slack-to to-pttea.,
1
Ar,
M + log NN) rt.
7)
then the new reward r/ must satisfy ry M log N, which is impossible unless the new maximum increases to M' 2 M + 1. Hence, under T = 0,
r/ -r rt M + log log -
E dec (T.)) max L strictly increases.
This gives simple sufficient condition: increase in r, that exceeds the softmax slack 1 log N (I' M) necessarily raises the deterministic acceptance length.
")
Putting (a) and (b) together, we obtain the stated guarantees: for T >> 0, any increase in r, strictly increases the expected acceptance length; for T 0, increase r; that exceeds the smooth-mas slack forces an increase in max; Lt.
Remarks. (i) The case T relies only on the strictly positive support of the target sampler; it holds for any softmax temperature T > 0 (or any sampler with full support). (ii) The suffi- cient condition in stronger implication can be made from alone because can increase by raising only sub-maxima branches without changing the maximum.
tight with respect to the standard mmooth-max bounds equation 1; no
B IMPLEMENTATION DETAIL
B.1 DRAFT TREE STRUCTURE
Across all experiments, adopt dynamic draft tree with fixed budget of 60 draft tokens, maximum tree depth 7 and top-k of 10, following the configuration shown to be effective in EAGLE-3.
B.2 TOKEN-LEVEL LOss EQ. (13)
Let D be the training corpus over a vocabulary V. For a sequence = (xi...., IL) E D. denote the prefix X1:i-1 (Z),...,T--I). Let PT X1 and pM(( x1 be the next-token distributions produced by the target model 'T and the draft model M. respectively. under the same teacher-foree prefix. We define the token-level loss as the expected cross-entropy from the teacher to the student:
Lenken = ExxD
Σ H(pT(: X1s- PM( xx==
[(x)]
iEI(x)
where I(x) C {1,..., L} indexes supervised positions (e.g., sg.,a all non-padding positions) and - p(w) log (w))
H(p,q) - --me
vEV
15

---

Published as a conference paper at ICLR 2026
the cross-entropy. Equivalently, since (pr, P.M) (L(pT]IPM) ( and (pr) does not depend on M, minimizing Ltoken is equivalent (up to an additive constant) to minimizing
Ex~D
Σ
(olpT(-Ixxis- [I x1==1)))
[(x) (EI(x)
B.3 TRAINING CONFIGURATION
We fine-tune the draft model with AdamW and varmup-decay schedule under mixed precision and ZeRO optimizations. Key hyperparameters are summarized below:
Draft-tree construction: top-k for per-node expansion k 10.
Draft-tree reranking: top-g candidates per step set to g 60.
Smooth-max temperature in tree reward: 17
Number of groups per sequence: K 16.
Group size (prefixes per group): m 8.
Scalar weight on the GTO loss: 0.5 in Leoken + w LGTO-
Optimizer and scheduler.
Optimizer: Adam W with 31-0.9, 02009,82-0000 3290. weight decay = 0.
Learning rate: Warm up linearly from O to 5x10-6 over 1,000 steps, then decay over a total of 60,000 steps.
Gradient clipping: 0.5.
Precision and parallelism.
Mixed precision: FP16 autocast with dynamic loss scaling (initial scale 211, window 1000; hysteresis = 2; min scale = 1).
ZeRO: Stage-2 with overlapping communication, alllgaaher/rrddueee-cateerr enabled; bucket sizes 10
Gradient accumulation: 2 steps; per-GPU micro-batch size: 1.
Training loop.
Epochs:
max sequence length: 2048
dataloader workers: 2
Additional hyperparameters and scripts are available at
GTO.
The full GTO update is summarized in Algorithm
C CLARIFICATION OF BASELINE METHODS
For EAGLE, EAGLE-2, EAGLE-3, HASS, GRIFFIN, Medusa and Hydra, we directly utilized the publicly released draft model parameters provided by the respective authors. For methods that do not require draft model training, such as PLD, Lookahead, and SPS, we evaluated performance using official code from their GitHub repositories.
16

---

Published as a conference paper at ICLR 2026
Algorithm GTO Phase II: Group-based Optimization of Draft Tree Reward
Require: Draft model M, reference draft model Mo. target model T, group size m, clip e, std
floor ö, reward aggregator R.
for each minibatch training sequences do for each sequence x in batch do Sample {G(*) ===1 t- SampleChoups((,,1 m)
2:
3:
> G (tk...., . + m
4:
for each group ( do
5:
for each E ( do
6:
Build trees: T, + 9(M, x1.) ) T, G(Mo. x1:1) Compute rewards: r; + R(T,) ,, t R(T,)
7:
8:
Debiased reward: R, - r; P;
9:
Find longest accepted sequence S; in T, and its length
Likelihood ratio: 8; < exp ((log MM, (x:)) - log Mo (1.:))/4)
11:
end for Standardize within group: A, t RR - mean( (R H)) (std( (R,) +6) Compute group loss: LOTO +- 1 Liega min (s,Ai, clip(8;, 8;, - e, + +)A.))
12:
13:
end for
15:
end for
Update M by minimizing Ltoken + wLoo
17:
end for
D TRAINING OVERHEAD OF GTO
Compute budget. results were obtained on NVIDIA A100 GB GPUs under mixed preci- sion with ZeRO-2. The Phase-Il GTO fine-tuning requires approximately (i) 200 GPU-hours for 8B models, (ii) 400 GPU-hours for 13B models, and (iii) 900 GPU-hours for 70B models. These compute budgets cover end-to-end GTO training (including grouped tree construction and verifica- tion) and exclude any pretraining of the base or drafter models, as we fine-tune on publicly available pretrained drafters.
Why the overhead worthwhile.
Model-agnostic and complementary. GTO is model-agnostic and complementary ex- isting speculative decoding methods: it can be directly fine-tuned on top of pretrained draft models (e.g., EAGLE-3, GRIFFIN) without architectural changes or modifications verification stack.
Amortized cost in deployment. Train once, use everywhere: the draft model is trained single time, whereas decoding dominates the runtime in real-world deployments; the added training cost is therefore amortized by improved inference efficiency.
Measured gains. In our experiments, GTO delivers > 7% higher end-to-end speedup ratio than EAGLE-3, making the small additional training budget favorable trade-off for latency-sensitive applications.
E ABLATION ON TREE CONFIGURATION
In our primary experiments, we default to the standard EAGLE tree configuration (e.g., depth 7, 60 total tokens). This tree structure has emerged as the de facto standard in modern speculative decoding research and is widely adopted by recent ttate-of-the-ar frameworks (e.g., EAGLE-3, HASS, GRIFFIN). By deliberately adopting this configuration, we ensure that our results are strictly comparable to established baselines, accurately reflect realistic deployment settings, and maximize practical relevance for the community.
Tree-Agnostic Formulation. Importantly, GTO is not inherently constrained to any specific tree layout. Methodologically, GTO optimizes the drafter to maximize the expected acceptance length
17

---

Published as a conference paper at ICLR 2026
Table 7: Ablation of Tree Configuration LLaMA-3.1 8B.
Temperature =
Temperature
Depth
Total Tokens
MT-bench
HumanEval
GSM8K
Average
MT-bench
HumanKval
GSMBK
Average
SR
SR1
Y.
SRT
7
SR
7
SR
r
SRT
1
7
SR
3.29
5.81
3.71
6,45
3.46
6.12
3.49
6.13
2.39
4.54
3.10
5.75
2.90
5.40
2.80
5.23
3.19
3.34
5.93
3.73
6.52
3.51
6.27
3.53
6.24
2,41
4.56
3.12
5,80
2.93
5.52
2.82
5.29
7
50
3.39
6.05
3,95
6.87
3,56
6.39
3,63
5.44
2.44
4.50
3.15
5,86
2.83
5.30
2.80
5.25
3,44
6.15
4.17
3.59
3.73
6.52
2.49
4.70
3.17
5.92
3.02
5.75
5,46
3,46
6.23
4.17
7,00
3,63
6.57
3.76
2.51
4.74
3.24
6.10
5,86
2.95
5.57
8
50
3,56
6.22
4.22
7.24
3,61
6.59
3,80
5.58
2.49
4.72
3.29
6.31
2.80
5.21
2.86
5.43
3.54
6.34
4.24
7.33
3.66
3.81
6.79
2.46
4.73
3.20
6.12
2.95
5.56
2.87
5.47
3,49
4.20
7.39
3.71
3,80
4.74
6.28
5.26
5.43
under speculative decoding. Because the training procedure learns underlying acceptance-relate to- ken distribution patterns rather than memorizing a fixed tree shape, GTO is conceptually compatible with any tree configuration that satisfies two basic conditions: (i) the drafter can successfully con- struct the tree, and (ii) a tree-level reward can be computed via target model verification. This design ensures that GTO's optimization principles generalize well beyond the default EAGLE layout.
Interpolation and Extrapolation Performance. To empirically validate GTO's robustness to varying tree structures, we conduct an ablation study exploring the extrapolation and interpolation of the tree configuration at inference time. Specifically, standard EAGLE tree (depth 7, 60 total tokens) and evaluate it using modified configurations during inference. We vary the tree depth {6. 7, 8 and the total drafted tokens per step {50, 60, 70). The comprehensive results across different benchmarks and temperatures are presented in Table 7. We observe four key findings:
take drafter trained exclusively on the
Consistent Performance: GTO maintains strong speedup (SI)) and acceptance rate (7) improvements across all 9 tested configurations, demonstrating high robustness to both depth and width variations.
Graceful Interpolation:
Configurations closely interpolating the training setting exhibit minimal performance degradation, with speedup varying by only 1 1%-3% from the optimal setting.
Effective Extrapolation: When extrapolating to depth 8 (beyond the training depth of 7), GTO still achieves competitive or even superior performance. This indicates that the token- level alignment patterns learned by GTO transfer effectively deeper and larger trees.
Stability Across Conditions: The robustness holds consistently across MT-hench, Hu- manEval, and GSM8K, as well as under both greedy (T' = 0) and sampling (T > 0) de- coding settings. This confirms that the generalization is neither task-specific nor decoding- strategy-specific.
In conclusion, these results demonstrate that GTO does not overfit to a specific tree configuration during the training phase. The learned alignment strategies remain highly effective when the tree structure is modified at inference time. Consequently. GTO fully supports dynamic tree config- uration updates during inference while maintaining, and in some cases even improving, overall speculative decoding performance.
F LLM USAGE STATEMENT
Large language models were used minimally for proofreading and grammar checking. The research ideas, methodology, experiments, and analysis were entirely conceived and conducted by the au- thors.
18


Reviews:
[{"review_id": "hlW8138uFH", "reviewer": "Reviewer_MEFU", "summary": "This paper introduces Group Tree Optimization, a novel training framework designed to accelerate LLM inference by solving a fundamental issue in speculative decoding: draft policy misalignment. Existing methods typically optimize the draft model based on maximizing the likelihood of generating a single greedy path, whereas the actual decoding process uses a tree policy that re-ranks and verifies multiple branches. The authors empirically demonstrate that this misalignment is severe: the greedy path is pruned 19–34% of the time, and the accepted path matches the greedy path only 36–49% of the time. GTO aligns training with the tree-based decoding policy using two core components: draft tree reward and group-based draft policy.", "questions": "1. Expand the discussion in Section 4.2 to provide deeper insight into the qualitative differences between groups of size 1 and size 32.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper clearly identifies \"draft policy misalignment\" as a fundamental limitation that weakens the effectiveness of modern draft model training techniques.\n2. The group-based policy training effectively mitigates the typical challenges associated with optimizing sparse, tree-level rewards. The ablation studies confirm the necessity of the debiasing step (which yields a +5.0% SR gain at T=0) and highlight the effectiveness of the Log-Sum-Exp aggregator. Furthermore, GTO's compatibility is demonstrated by applying it as a fine-tuning step on draft models initialized by GRIFFIN and HASS, achieving consistent gains.", "weaknesses": "1. Clarity on Training Complexity Overhead: The authors acknowledge that GTO increases training-time compute due to the two-phase procedure and the need for grouped draft tree construction and verification. While they state this cost is amortized in deployment, the paper lacks a clear, quantitative discussion in the main body (Section 3) of the computational complexity added by Phase II compared to the standard token-level loss (L_{token}) calculation.\n2. The PPO-style optimization is performed using a likelihood ratio (s_i) calculated only along the longest accepted sequence (\\hat{S}_i). This is a critical design choice, as it focuses the gradient update entirely on the most successful path within the group. While this likely maximizes the speedup metric, a clearer justification is needed in Section 3.2 explaining why updating based solely on \\hat{S}_i (Eq. 11 and 12) is preferred over updating all accepted sequences or averaging their contributions, in the context of improving the overall tree reward R_t.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1fj7Saf5AF", "reviewer": "Reviewer_yvuE", "summary": "The paper addresses the draft policy misalignment problem in Speculative Decoding, where the drafter is trained on greedy paths, while drafting utilizes tree-expansion methods. The authors propose Group Tree Optimization (GTO), an RL framework designed explicitly for aligning training with tree-drafting procedures. It introduces a Draft Tree reward and a Group-based Draft Policy Training scheme, achieving approximately 7% improvements in acceptance length and surpassing prior state-of-the-art baselines, such as EAGLE-3.", "questions": "- W1: Can a drafter trained for EAGLE be used with HASS, or vice versa?\n- W2: Is it possible to dynamically update the tree configuration during the inference stage while maintaining performance in the trained settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Proposes a principled RL framework that directly optimizes the expected acceptance length in tree-based speculative decoding, eliminating the need for sampling.\n- Bridges the gap between training and inference by aligning the drafter’s objective with the tree decoding policy.\n- Demonstrates **strong empirical performance** across diverse LLMs and tasks, consistently outperforming prior methods like EAGLE-3 and HASS.\n- The approach is model-agnostic and computationally efficient.", "weaknesses": "- W1: Transferability. GTO trains the drafter explicitly on the drafting method, such as EAGLE, which may restrict the usage with other drafting methods.\n- W2: Robustness. It appears that the drafter is trained to fit a specific tree configuration during the training phase. The paper did not explore extrapolation or interpolation of the tree configuration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yZVDqxe9Z8", "reviewer": "Reviewer_AxA5", "summary": "This paper addresses the draft policy misalignment in speculative decoding for LLMs, where the draft model is trained using a greedy, single-path objective but deployed under a tree-based decoding policy. The authors propose Group Tree Optimization (GTO), a new training framework that explicitly aligns the training objective with the decoding-time policy. GTO introduces (1) a Draft Tree Reward, a sampling-free, differentiable objective measuring the expected acceptance length of the draft tree under the target model, and (2) a Group-based Draft Policy Training algorithm that leverages group-standardized advantages and a PPO-style surrogate to stabilize optimization. Theoretically, the paper proves that maximizing the Draft Tree Reward provably improves acceptance length and decoding efficiency. Experiments across dialogue (MT-Bench), code generation (HumanEval), and math reasoning (GSM8K) tasks on multiple LLMs (e.g., LLaMA-3.1-8B, Vicuna-13B) show that GTO increases acceptance length by 7.4% and yields 7.7% additional speedup over EAGLE-3, establishing it as a general and effective method for bridging training–decoding misalignment in speculative decoding", "questions": "- How sensitive is GTO to the choice of group size m, reference model quality, or the number of sampled trees per prefix? Could you elaborate on how these hyperparameters were chosen in Appendix B.3 and whether GTO remains stable across a wide range of values?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies and targets an important but previously overlooked research problem, draft policy misalignment between training and decoding in speculative decoding. This conceptual reframing is interesting and provides a unifying explanation for the inefficiencies of prior methods such as EAGLE, GRIFFIN, and HASS.\n- Based on the identified problem, the authors proposed GTO, which is sound and well-implemented with a theoretical guarantee.\n- Empirical results are strong, even at high decoding temperatures where existing methods may degrade. The analyses and interpretations of the results looks solid.\n- Overall the paper is clearly written and well-organized. Code and demos are available for reproducibility.", "weaknesses": "- All experiments are conducted on the LLaMA family and its derivatives (Vicuna, DeepSeek), leaving open whether GTO generalizes to other popular architectures such as Qwen and Gemma. Since speculative decoding behavior can vary significantly across model families due to differences in tokenization and calibration, including at least one non-LLaMA model would strengthen the empirical claims of generality.\n- While GTO introduces clear gains in decoding efficiency, the paper provides limited quantitative analysis of training-time overhead (I appreciate the statements in Appendix D). The two-phase group optimization and tree-level reward computation likely introduce nontrivial additional compute and memory costs. A clearer breakdown (e.g., training FLOPs or wall-clock comparison against EAGLE-3) would help justify the practical cost–benefit balance of adopting GTO in large-scale deployment settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JWupgwPi0X", "reviewer": "Reviewer_kt8J", "summary": "This paper investigates how to bridge evolving draft policies and model behaviors in large language models. The authors propose a two-stage method: a policy bridging module based on contrastive training between draft policy outputs and model outputs, and a misalignment diagnostic using representation-space probes. Experiments show modest improvements in alignment detection and behavior correction compared to direct fine-tuning and reward-model-based baselines.", "questions": "1. Why is there no baseline where the draft model is simply continued to train with new data for the same number of GPU hours as the bridging module?\n\n2. Can the authors clarify the exact sense in which their method is “sampling-free”? It seems draft model outputs are still needed.\n\n3. What would the bridging performance look like under stricter compute constraints (e.g., 10–20 GPU hours)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. Addresses a practical problem in alignment: bridging evolving policies and model behaviors.\n\n2. Conceptually coherent design combining contrastive alignment and probing.\n\n3. The theory is good.", "weaknesses": "1. Unfair experimental comparison: All proposed methods are trained on top of an existing draft model with additional compute (e.g., 200 GPU hours for a 7B model), whereas baselines are not given the same training budget. The most natural baseline should be continued training of the draft model with new data under the same GPU hours, and this is completely missing (e.g., further training eagle 3, hass, griffin for 200 GPU hours on a 7B model with new training data). This omission makes the reported improvements difficult to interpret.\n\n2. Misleading “sampling-free” claim: The method is not truly sampling-free. Although no RL trajectory sampling is used, model outputs are still required for every node in the policy tree. The phrasing gives a false impression that no generation or sampling is needed, which is inaccurate.\n\n3. Excessive compute for modest gains: Training a single-layer Transformer for 7B over a 5-layer policy tree costs about 200 GPU hours, resulting in about 7% improvement. This is too expensive, near the time consumption of training EAGLE from scratch.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)