Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning
Authors: Jiebin Zhang
Zhenghan Yu
Liang Wang
Nan Yang
Eugene J. Yu
Zheng Li
Yifan Song
Dawei Zhu
Xingxing Zhang
Furu Wei
Sujian Li
Venue: ICLR 2026 Poster (2026)
Keywords: speculative decoding
reinforcement learning
Primary area: foundation or frontier models, including LLMs
TL;DR: We use reinforcement learning to train two co-adaptive policies to dynamically coordinate the draft and verification phases, using throughput as the reward signal.

Abstract: Speculative decoding accelerates large language model (LLM) inference by using a small draft model to generate candidate tokens for a larger target model to verify. The efficacy of this technique hinges on the trade-off between the time spent on drafting candidates and verifying them. However, current state-of-the-art methods rely on a static time allocation, while recent dynamic approaches optimize for proxy metrics like acceptance length, often neglecting the true time cost and treating the drafting and verification phases in isolation. To address these limitations, we introduce Learning to Draft (LTD), a novel method that directly optimizes for throughput of each draft-and-verify cycle. We formulate the problem as a reinforcement learning environment and train two co-adaptive policies to dynamically coordinate the draft and verification phases.  This encourages the policies to adapt to each other and explicitly maximize decoding efficiency.
We conducted extensive evaluations on five diverse LLMs and four distinct tasks. Our results show that LTD achieves speedup ratios ranging from 2.24x to 4.32x, outperforming the state-of-the-art method Eagle3 up to
36.4\%.

Published as a conference paper at ICLR 2026
LEARNING TO DRAFT: ADAPTIVE SPECULATIVE DE- CODING WITH REINFORCEMENT LEARNING
Jiebin Zhang Zhenghan Yu Liang Wang
Nan Yang Eugene J. Yu
Zheng Li
Yifan Song Dawei Zhu Xingxing Zhang Furu Wei Sujian Li V
National Key Laboratory for Multimedia Information Processing, Peking University
School of Computer Science, Peking University
Microsoft Research Asia
{
Izhangjiebin. wisujiainnn htws:iian/pkuu.euuu.. en wangliang@micronmoo htw./aanggmmiioooottccomo
https://githin con//thaihaa//eeeaaann-----
ABSTRACT
Speculative decoding accelerates large language model (LLM) inference by using a small draft model to generate candidate tokens for larger target model to verify. The efficacy of this technique hinges on the trade-off between the time spent on drafting candidates and verifying them. However, current state-of-the- art methods rely on static time allocation, while recent dynamic approaches optimize for proxy metrics like acceptance length, often neglecting the true time cost and treating the drafting and verification phases in isolation. To address these limitations, we introduce Learning to Draft (LTD), novel method that directly optimizes for throughput of each haft-and-verify cycle. We formulate the problem reinforcement learning environment and train two co-adaptive policies dynamically coordinate the draft and verification phases. This encourages the policies adapt to each other and explicitly maximize decoding efficiency. We conducted extensive evaluations on five diverse L.LMs and four distinct tasks. Our results show that LTD achieves speedup ratios ranging from 2.24 to 4.32x, outperforming the state-of-the method Eagle3 up to 36.4%.
INTRODUCTION
Modern large language models (LLMs) are increasingly employed to perform deliberate, multi-step reasoning to solve complex tasks, paradigm that extends their performance capabilities inference time (Guo et al. 2025 Team, 2025b Agarwal et al. 2025). However, the complexity of these reasoning processes leads to significant latency, which has prompted a growing research focus on accelerating inference while preserving high-quality output (Sui al., 2025). Among these efforts, speculative decoding stands out as highly effective solution, using a smaller, faster draft model to generate token sequences that then verified in parallel by the larger target model. This approach ensures both accelerated inference and a preserved output distribution (Leviathan et al. (023).
The quality and structure of the draft token sequences ("draftss" for short) crucial the overall performance of speculative decoding. Early methods primarily employ simple chain-structured drafts (Chen al. 2023 Leviathan al. 2023). In contrast, recent studies have demonstrated that tree-structured drafts can significantly increase the number of accepted tokens (acceptance length) per draft-and-veriig cycle (Mioo al. 2023 Li et al. 20244) Cai et al. 2024). State-of-the-ar methods, such as Eagle3 (Li et al., 2025a). employs a static, one-size-fits all strategy, configured through manual tuning, define the size of the draft trees. While some works have attempted to dynamically adjust the size of the draft tree (Brown et al. 2024 Zhang et al. 2024c Huo et al.) 2025) Huang cet al. 2025a), their approaches typically rely on simple heuristics or train models to optimize for the metric of acceptance length-tie the average number of tokens guessed correctly per verification step. However, maximizing acceptance length does not necessarily lead to faster overall generation speed, it completely ignores the critical time cost of the drafting and verification processes themselves. The true objective should be maximizing the model's 's throughput. Otherwise, the pursuit of a longer
Contribution during Jiebin's 's internship MSR Asia. Sujian Li is the corresponding author.

---

Published as a conference paper at ICLR 2026
Draft Time
Verification Time
40030
Select Candidate TokEns
Geneoale Draft Tree
Reurganize
tolb.11
D3pth=3
40.12
Varify And Get
Target Model
A0.12
Verification Size=
Draft Model
Continue Stop
5 Tokens
Acceptance Lnnth+3
Depth Policy
Size Policy
Re Time + Verily Time)
Update Palicy Using Throughput Reward
The overview LTD method. We formulate the interaction between draft model and target model RL environment. We employ two policies to dynamically manage the allocation between draft generation and target verification by controlling the draft tree's depth and verification size. These policies are optimized together using the reward signal based on the throughput of each draft-and-verify cycle.
Figure 1:
acceptance length can result in a large draft tree whose generation and verification consume more time than is saved, ultimately slowing down the process. At the same time, key limitation identified in existing methods is the isolated optimization of drafting and verifying draft trees.
In this paper, we argue that drafting and verification interdependent and should be co-optimized. A truly optimal strategy must be both time-aware and adaptive to the interplay between the two stages. For example, while the verification stage remains fixed, a suboptimal outcome occurs. Even if the draft model can easily generate long and correct sequence, it forced truncate its output to accommodate the limited verification cost. This leads to waste of drafting capacity, preventing it from reaching its full potential for speedup. In an ideal joint optimization, the draft stage would produce a longer and more complete sequence, aware that the verification stage will dynamically adjust its resources. In turn, informed of the long drafts, the verification stage would allocate sufficient computational capacity to validate these tokens efficiently.
simple text generation scenario, if only the drafting stage is optimized
Based the idea above, we propose LTD (learning to draft), a novel method that synergistically optimizes both phases by using throughput the direct optimization objective. Specifically, formulate the interaction between the draft and target models as a reinforcement learning (RL) environment. As shown in Figure control draft cost by determining the draft tree's depth, and a size policy that manages verification cost by selecting the optimal number of candidate tokens for the target model. Crucially, these policies trained using the throughput of each draft-and-verify cycle as a direct reward signal. Furthermore, we train these two policies iteratively, which allows them to co-adapt to each other's ss evolving strategies. These enable them to learn how to adaptively manage the trade-off between acceptance length and time cost, and dynamically coordinate the two phases to maximize inference efficiency.
train two distinct, co-adaptive policies: depth policy to
We build our method upon the thie-oof-hhearr Eagle3 framework, training our policies to replace its static heuristies with dynamic, context-aware Decision-making process. As demonstrated through extensive evaluations under five distinct LLMs across four benchmarks, our method achieves significant performance gains. During greedy decoding, our method yields substantial throughput improvements over the highly optimized Eagle3 baseline: 36.4% on Qwen3-32B, 9.5% on Deepseek- Distilled- -ttilled-Llamaat Llama-8B 6.5% on Llama-3-88 3-BB, 5% on Vicuna- 1eeeaa--.31111 1.3, and 4% on Qwen3-14B. Notably, approach demonstrates robustness in nigh-temperattre scenarios, delivering approximate 5% throughput gain where most other dynamic speculative methods become ineffective. Our contributions be summarized follows:
2

---

Published as a conference paper at ICLR 2026
Direct throughput optimization: We propose time-aware optimization approach by formulating the speculative decoding process as an RL environment. By maximizing throughput of each draft- and-verify cycle as direct reward, our method learns to manage the trade-off between acceptance length and time cost.
Co-adaptive policy framework: Instead of treating the drafting and verification phases indepen- dent problems, we propose a jointly trained, co-adaptive policy framework where two policies learn dynamically coordinate to each phase.
SOTA performance and robustness: Our method achieves significant performance gains (up to 36.4%) over strong Eagle3 baseline in greedy decoding. Compared to prior work, it maintains its effectiveness in high-temperature sampling scenarios.
2
RELATED WORK
2.1 SPECULATIVE DECODING
Vanilla Speculative Decoding uses a chain-like structure to generate token candidate tokens (Stern cct al. 2018 Chen et al.] 2023 Leviathan et al. 2023). Moving beyond these initial chain-structured approaches, SpecInfer (Miao et al. 2023) introduced a tree-based drafting architecture. By incorporat- ing a tree attention mechanism, this method facilitates the parallel verification multiple candidate sequences, establishing highly efficient paradigm that has since been widely adopted (Cai et al. 2024] Li et al. 2024). Subsequent research has predominantly focused on refining the draft model through several key strategies, such as leveraging knowledge distillation (Liu et 2023 Zhou et al. 2023), utilizing more effective features for draft token prediction (Cai et al. 2024 Li et al.
2025a Ankner et al. 2024 Zhang et 2024a Li al. 20255b] Du [dl.|220)) and employing
self-s selfsspeculation, wherein components of the target model are repurposed for drafting (Zhang et al.
2023
Elhoushi et al., 2024; Hooper al. 2025).
2.2
ADAPTIVE ADJUSTMENTS FOR SPECULATIVE DECODING
While tree-based verification significantly enhances throughput, most methods adopt a static schedule for each draft-and- -eriif cycle (Cai et al. 2024; Li al. 2025a), failing to dynamically adapt the varying complexities of input contexts. Recognizing this limitation, recent efforts have increasingly focused on dynamic methods that adapt the speculative strategy based on the generation context. These works can be broadly categorized into two main streams: dynamically adjusting the draft time by controlling generation length draft depth (Brown et al. [2024, [Mamou al. 2024 Zhang et al.) 2024c HHang et al. 2025a Gautam et al. 2025). or dynamically adjusting the verification time by selecting candidate tokens for target models (Zhong et al. 2024 Wang et al., 2025 Huo et al. 2025). These approaches typically leverage contextual information such as token probabilities (Brown et al.) 2024), entropy (Mamou et al., 2024), or hidden states (Zhang et al. 2024b Huang et al. 2024) to adjust speculative decoding with the goal of maximizing the expected acceptance length. Further research explored other dimensions for optimization. Some methods focus on adjusting the width of the draft tree QQin et al.) 2025) Xiong et al. 2025). Other works explore dynamic methods for Self-Speculative Decoding (Zarch et al. 2025). Research has also targeted the improvement of overall Service Level Objectives (Huang al. 2025b) the optimization of parallel execution between the drafting and verification stages (Liu et al. 2025).
3 MODEL OVERVIEW
3.1
PRELIMINARY STUDY: TIME COST AND ACCEPTANCE LENGTH
We conducted preliminary experiments on the Llama and Vicuna models to validate our idea: that dynamic adjustment of the key parameters (draft depth and verification size) is necessary draft- and-verify inference to achieve optimal acceleration by maximizing throughput. Here, the throughput for each draft-and -vriffy cycle (Ae) is defined as the number of accepted tokens (Lx) divided by the total wall-clock time of the draft-and- verify cycle (Tuaa):
X LA/Tuane Tootal
(1)
3

---

Published as a conference paper at ICLR 2026
Draft Time
Verification Time
Eagle3 Verification Time
LTD Verification Time
Eagle3 Draft Time
Draft Time
50
100
Normalized Time (%)
32
Draft Time (ms)
16-
40
80
28
Verification Time
12-
30
60
20
71
64
82
20
D-y P020 Dng
Vidl
Dwg
yor
Verid
Vetto
io0
Vezatl
L31 88
V13
DSL SB
Owen3 148
Owen3 328
Figure 2: Analysis of Draft Time and Verification Time. Left: The average draft and verification latency of Vicuna-131 13B as function of draft depth D and verification size V on benchmarks. Right: The composition of the total inference time for Eagle3 and LTD on the benchmarks, with Eagle3's total time normalized to 100% as baselines.
Tittl can be decomposed into the draft time, Taran, and the verification time, Terttyy
Total
Taraf + Terify
The draft time Trra is primarily determined by the draft depth (D), which refers the number of forward passes performed sequentially by the draft model (i.c., the steps of generating candidate tokens). larger leads more candidate tokens being generated, but it also results in linear increase in the required time, as illustrated by the red bars in Figure 2 (Left). Conversely, the verification time mainly governed by the verification size (V), which is the total number of candidate tokens processed in one forward pass of the large target model. Due to the model's computational cost and the long input sequence, the verification time exhibits a step-wise increase V grows, shown by the blue bars in Figure (Left). V can be substantial (e.g., up 1000 tokens when D = 10), making a critical factor for verification time. With our LTD strategy, the verification time be significantly reduced, shown Figure S (Right).
Besides time latency, Figure alaso also indicates that draft depth and verification size critically affect the average acceptance length, given fixed draft model and target model (see Appendix), Increasing both draft depth (D) and verification size (V) can generate more candidate tokens, which has the potential raise acceptance length per cycle (La, the numerator in the throughput equation). However, increasing D and V inevitably leads to a rise in both draft time and verification time (i.c., an increase in the denominator, Tooal). Therefore, our goal is not to blindly maximize and V. but rather find an optimal balance that maximizes the overall throughput, defined as A L Tioal
3.2 DRAFT-AND- VERIFY CYCLE
Our work builds upon the state-of-the-ar tree-based approach, namely Eagle3 (Li ct al. this section, formally define the mechanics of single draft-and-verify cycle and the role of policies in guiding this process. As shown Figure the draft-and-verify cycle can be divided into three parts: Draft Tree Generation, Candidate Token Selection and Draft Tree Verification.
2025a). In
Draft Tree Generation The drafting process begins after the target model has output the token following the last accepted one; this token serves as the root for a new draft tree. The tree is constructed through an iterative, beam-search-like expansion process governed by two key hyper- parameters: the beam width, W. and the draft depth, D. The construction process commences with initial expansion from the root token. In this first step, the draft model performs forward pass to generate probability distribution over the subsequent tokens, and the top-W most probable candidate tokens selected to form the first level of leaf nodes in the tree. Following this, the tree undergoes iterative growth for D 2 subsequent rounds. In each round, the draft model takes the leaf nodes from the previous level as inputs. For each of these W nodes, generates W new child
4

---

Published as a conference paper at ICLR 2026
candidates, creating combined pool of W new nodes. A beam selection is then performed this pool, where the top-W candidates with the highest predicted cumulative path probabilities are chosen to become the new leaves of the tree, poised for the next round of expansion. The draft depth for Eagle3 is set default to 8, while in LTD our depth policy will decide whether to continue or stop after each draft model's forward pass. Once the depth policy decide to stop the expansion, the whole draft tree will pass to the size policy for candidate token selection.
Candidate Token Selection With draft depth of D, the final draft tree contains total of Vau = 11+ W + (D 2)- W candidate tokens, each with an associated predicted probability. Once the draft tree is generated. subset of V candidate tokens with the highest overall probabilities is selected for verification. While Eagle3 employs fixed verification size V, LTD utilizes size policy to dynamically determine this value based on the properties of the generated draft tree. These V selected tokens are then flattened into a single sequence, along with corresponding tree attention mask, is passed to the target model.
Draft Tree Verification The target model performs a single forward pass to validate these V tokens in parallel. It accepts the sequence of tokens up to the first point mismatch between its own prediction and the candidate tokens. At the position of this first error, target model discards the remainder of the candidate tokens, computes the correct token, and appends it to the sequence of accepted tokens. This newly confirmed token then becomes the root of draft tree for the next draft-and-verify cycle. Finally, the number of accepted tokens LA in a cycle (termed acceptance length) is divided by the sum of draft time and verification time to calculate the throughput for this cycle. This throughput is used as the reward signal to optimize our depth policy and size policy.
4 METHOD
We frame the interaction between draft and target models within a Reinforcement Learning (RL) framework, employing two co-adaptive policies to navigate the inherent trade-off between acceptance length and latency: a depth policy, ZD. which determines the draft depth, and a size policy, TT, which determines the verification size. The action space for the depth policy is binary, Anp {0, 1}. determining whether to continue drafting after each forward pass, while the action space for the size policy, Ar E [20, 240), is discrete range of verification sizes. To inform their decisions, the policies observe a state vector 8t E S at each time step i, defined as
8t - LD P
(3)
where D the current draft depth, is the context length, and PI....,P Pv. are the predicted probabilities for all tokens in the draft tree. This state representation is designed be highly informative, as token probabilities are strong predictors of acceptance length (Li et al. 2025a). while draft depth and sequence length are the important determinants of latency (Figure 2 To optimize decision latency, the size policy TV observes the entire state, whereas the depth policy TD utilizes a minimal subset: the probability of tokens at the last layer of the draft tree, along with 1 and L. Both policies are implemented as lightweight Multi-Layer Perceptrons (MLPs) to ensure the decision-making process introduces negligible computational overhead. More details about the two policies can be found in Appendix A.2
These policies are jointly optimized using the Proximal Policy Optimization (PPO) algorithm (Schul man et al. 2017) to maximize reward signal, RG which represents the throughput for each cycle:
R₂ (Tarrft + Tuerify)
This reward function directly incentivizes the policies to master the complex trade-off between generating more potentially correct tokens and the time cost incurred. We update the policy network Ty through the following objective function:
((0) = E., min Te(aelse) Ae, clip
To(ag(s)) Ta (at (st) 1-,,,,,,, + A. A.)].
(5)
Teay (Toay(at (at(st) (st)
where Tew is the behavioral policy and A, is the advantage estimate (Schulman et al. 2017 7). Following each optimization step, the old policy parameters updated with the new (fata 0),
5

---

Published as a conference paper at ICLR 2026
Size Reward Speeduup
Depth Policy Reward & Speediup
ng Ravard
16
singing tenard
3.00
1.2
1.20
Thiniig Seward (Gmoethre))
1.4
Valnol Specaur
1.15
3.43
12
Thipigg
1.05
5.00
1.00
3.4
20000
40020
52320
10000
120200
120002 200020 320003
403000 500002
602000
700302
302000
3.40
Gobbl Step
Global Step
Figure 3: We report the training reward curves for the Size Policy and Depth Policy of the Vicuna- 13B model, as well as evolution the speedup ratio on the validation set.
and the process of data collection and optimization is repeated. To ensure stable convergence, we employ a two-stage training procedure. The first stage involves training the initial depth and size policies independently. In the second stage, we use these initial policies as a starting point for iterative training, allowing them co-adapt and converge on a coordinated strategy.
Initial Policy Training. We leverage the HumanEval dataset (Chen et al. 2021 for training, as its diverse and complex code snippets provide rich environment for the policies to develop wide range of adaptive strategies. This initial phase involves training each policy independently while its counterpart operates on fixed, heuristic-based strategy. Specifically, the size policy, TV. trained for 100k steps with draft depths randomly sampled from the range [1, 12]. Conversely, the depth policy, n, is trained for IM steps using a constant verification size of 60, mirroring the heuristic used in the Eagle3 baseline. This initial training establishes robust, standalone policies that serve as stable foundation for the subsequent co- co-adaptation phase. We present the evolution of the reward for the Vicuna-133 13B policies during training in Figure 3] alongside the speedup ratios observed on the validation set. The checkpoint yielding the highest performance on the validation set was selected the basis for subsequent training. As illustrated in the figure, the number of training steps was sufficient to ensure model convergence. The training curves the other models exhibit patterns similar those of Vicuna-13B.
Iterative Optimizing for Policy Co-Adaptation. Although the policies are trained independently. achieving optimal performance inference hinges their synergistic operation. To address this co-adaptation challenge, we introduce an iterative optimization procedure. This process alternates between refining each policy while holding the other fixed. First, we freeze the parameters of the size policy (Ty) and optimize the depth policy (TD) with respect to the dynamic verification sizes my proposes. Subsequently, the newly updated depth policy (TD) is frozen, and the size policy (Tv) is optimized based on the dynamic draft depths determined by xD. This alternating optimization cycle allows the two policies progressively co-adapt each other's evolving strategies. Empirically, we found that only two rounds of this iterative process yield a highly synergistic strategy and substantial performance gains.
EXPERIMENTS
5.1
SETUPS
Models We conduct experiments with state-of-the-art open source chat and reasoning models, including Llama-3. --8-----snnstc (Dubey al. 2024), Vicuna-13B-vll (Chiang al. 2023), DeepSeek-RI-DD 11-Distill-LLaaee 8B (DeepSeek-A) All| 2025), Qwen3-14p 14B [Team] (025a) and Qwen3- 32B (Team 2025a). The Eagle3 models for the Qwen3 series are from AngelSlim (Contributors 2025), as the official implementation had not been released. For all other models, we used the official public releases of Eagle3 (Li al. 2025a).

---

Published as a conference paper at ICLR 2026
Tasks Following EAGLE (Li et al. 2024 and Spec-Bench (Xia al. 2024 evaluate four common tasks, using the same policy weights for all tasks. For multi-turn conversation, mathematical reasoning, instruction following, and QA, chose the MT-bench (Zheng et al. 2002)) GSM8K (Cobbe et al. 2021), Alpaca (Taori et al. 2023) and Natural Questions (Kwiatkowski et al. 2019) datasets, respectively.
Metries Speculative decoding will not change the output distribution for target model. Therefore, we do not evaluate the generation quality and instead use the following common used metrics (Li et al. 2025a) to assess acceleration performance:
Speedup: The actual test speedup ratio relative to vanilla auto-regressive decoding,
Average Acceptance Length т: The average number of tokens generated per draft-and- verify cycle on the benchmark dataset.
Implementation Details We implement the Proximal Policy Optimization (PPO) algorithm us- ing the Gaable-Baselinese library (Raffin et al.] 2021) and train our policies on the HumanFval dataset (Chen al. 202I). The size and depth policies are trained for 100k and 1M steps, respec- tively. Further analysis regarding the choice of training steps can be found in the Appendix C.I The PPO is configured with rollout buffer of 2048 steps and a minibatch size of 256. Due the lightweight nature of the two policies, the RL training phase is highly efficient, detailed can be found in Appendix(A.7 A comprehensive list of all hyperparameters can be found in Appendix AA.3
Comparison We use vanilla auto-regressive decoding as the baseline, which serves as the bench- mark for speedup ratios (1 (1.00x). We apply our methods to current SOTA Speculative Decoding method Eagle3 and compare with recent representative dynamic depth methods, including DDD (Brown et al.
2024), SVIP (Zhang et al. (024c). Gammatune (Gautam et al. 2025). Disco (Mamou et al. 2024)
and SpecDec++ (Huang et al.) (225a)) and dynamic size methods C2T (Huo et 2025). We set the
default parameters for dynamic depth methods with a verification size 60 and maximum depth of 12, dynamic size methods with a depth of 8. For the Eagle3 baseline, we set the verification size to 60, 50 for 8B, 13B original L.MMs respectively, with draft depth of 8, and select 10 nodes during the expansion phase as provided by the original implements (Li ct al. 2025a). We also employed Grid Search(GS) to find the optimal spper-paraeetess for Eagle3, as strong baseline. More details can be seen in Appendix A.
5.2 EFFECTIVENESS
The performance of our method is detailed in Table and Table 5 The experimental results demonstrate that our proposed method consistently outperforms all competing approaches across the evaluated models, datasets, and sampling temperatures. The stable speedup observed across diverse configurations highlights the strong capability of our framework.
Our method accelerates inference by an average of 6.5% on Llama-3 and 5.1% on Vicuna compared the default Eagle3 configuration. In long thinking model like Deepseek-R -1 --Distll-LLaMM 8B, our method can yield 10% accelerate, which is quite useful as the long generation time for these models. For Qwen3 series, as no default parameters were provided, we report the results of grid search as our baseline. Our method shows significantly greater improvement on the 32B model compared to the 14B model. This demonstrates the reliability of our method at larger scales. Notably, our method maintains the highest average speedup at a temperature of 1.0 as shown in Table [ despite being trained exclusively at greedy decoding. contrast, other dynamic methods that adjust draft depth or verification size, while effective at greedy decoding, exhibit significant performance degradation at the higher temperature. This highlights the strong robustness of our method.
Our method does not necessarily yield the longest acceptance length; its is often shorter than that of the Grid Search-optimized baseline. Nevertheless, it achieves the highest overall speedup. This outcome reveals the core mechanism of our approach: the reinforcement learning framework makes policies learn to optimize directly for decoding efficiency. It strategically forgoes candidate tokens that, while potentially correct, would incur disproportionately high time cost. By directly optimizing for the reward signal of throughput, our method achieves a superior speedup ratio by adaptively balancing acceptance length against time cost.

---

Published as a conference paper at ICLR 2026
Table 1: Speedup ratios and average acceptance lengths of different methods during greedy decoding. L31 8B represents 1-ma-3.1-8BBInsttnn V 13B represents Vicuna-13B-v1.3, Dpsk 8B represents -eepSeee-RR--issil---ee
8B. For the Qwen3- 14B and Qwen3-32B models, their corresponding Eagle3 models lack default settings. Therefore, only the results from Grid Search are presented. GT. Spec++, and Gs stand for Gammatune, SpecDec+t, and Grid Search, respectively. The results of Temperature=1 are presented in Table
MT-bench
Gsm8k
Alpaca
Qa
Mean
Model
Method
Speedup
Speedup
Speedup
Speedup
Speedup
Temperatureat)
Hagle3
3,70
6.39
3.58
6.44
4.26
6.93
3.17
5.35
3.68
6.28
Eagle3+DDD
3.67
5.41
3.55
5.48
3,97
5.58
3,07
4.21
3.57
5.17
Eagle3+Svip
3,65
6.21
3,47
6.21
4.04
6.73
3.12
4.78
3.57
5.98
Bagle33GT
3,79
5.86
3.64
5.77
4.37
6.49
3.38
4.92
3.80
5.76
Ragle3+C2T
3,80
6.39
3.57
6,44
4.19
6,93
3,25
5.36
3.70
6.28
131 88
Eaglc3+Disco
3.70
6.11
3.56
6.18
4.14
6.51
3.22
5.05
3.66
5.96
Specc3+
3.82
6.09
3.65
6.08
4.29
6,70
3,28
5,23
3,76
6.03
Bagle3+GS
3,83
6.75
3.78
6.91
4.38
7.42
3.33
5.67
3.83
6.69
Eaglc3+LTD
3.96
6.74
3.79
6.73
4.53
3.41
3,92
6.56
Eagle3
4.48
7.22
4.36
6.58
4.16
6.47
3.43
5.12
4.11
6.35
Bagle3+DDD
4.15
6.69
4,25
6.22
3.98
5,50
3.22
4.34
3.90
5.69
Fagle3+Svip
4.02
7.74
3.86
6.79
3.77
6.53
3.02
4,96
3.67
6.51
Eagle3+GT
4.15
6.85
4.07
6.00
3,94
6.09
3.35
3.88
5.94
V3B
Bagle3+C2T
4,33
7.22
3,97
6,58
3,93
6,47
3.23
5.12
3.87
6.35
Ragle3+Discn
4.48
7.08
4.13
6,49
4.02
6.29
3.31
5.05
3.99
6.23
Sapec3+Spec++
4.25
6.78
4,09
6.16
6.27
3.21
5.03
3.89
6.06
Eagle3+GS
4,53
7,99
4,47
7.27
4,38
7.16
3,51
5,48
4,22
6.98
bagle3+1:ID
4.64
8.24
4.59
7.38
4.40
7.01
3.66
5.37
4.32
7.00
Eagle3
3.67
5.82
4.73
7.39
3.65
5.63
3.14
5.02
3,80
5.53
Dpsk 8B
Bagle3+GS
3.69
6.42
4.83
8.37
3.54
6.24
3.20
5.55
3.82
6.08
Eagle3+LTD
4.05
6.32
4.98
8.53
3.98
6.00
3.62
5.37
4.16
6.08
Qwen3 14B
Eagle3+GS
2,34
3.40
2.75
3.95
2.15
2.90
1.86
2.65
2.28
3.23
Bagle34LTD
2.44
2.82
3.92
2.24
2.62
2.37
3.19
Eagle3+GS
1.76
3.30
2.26
4.10
1.75
2.96
1.45
2.67
1.81
3.26
Qwen3 328
LTgle3+LTD
2.42
3.17
3.02
3,90
2.38
2.85
2.06
2.60
2.47
3.13
To better verify generalization, we evaluated our method on MMLU, which consists of 57 subtasks across diverse domains (results detailed in Appendix B and Figure 7 Our method outperforms Eagle3 on 54 out of 57 subtasks, achieving an average speedup improvement of 5%. The top- performing categories vary widely, including not only Math, Logic, but also Human Aging, and Government Politics. These results strongly support the cross-domain generalization our approach.
5.3 DRAFT TIME AND VERIFICATION TIME
To provide granular analysis of our performance gains, we present the relative composition of the total inference time, breaking it down into its draft and verification components in Figure Right. It reveals that our method's s speedup stems from improvements in both drafting and verification. Universally, draft time is reduced due to our dynamic depth policy, which curtails speculation in low-confidence scenarios to save computation. Notably, our method can also reduce the verification time, implying lower token verification load. Paradoxically, this lower overhead corresponds to higher average acceptance length for Llama, Vicuna and Dpsk models as shown in Table This outcome highlights that our method's strength lies not just in reducing the size of draft trees, but in strategically allocating candidate tokens contexts where they most likely to be accepted, which validate the efficacy of our approach. For the Qwen series models, while our method exhibits a lower acceptance length compared the baselines, significantly reduces verification time, especially for the Qwen3-32B model, thereby achieving a substantially higher speedup ratio.
F
ABLATION STUDY
In this section, we aim to validate the effectiveness of LTD method through a series of analyses. We begin in Section 6.1 demonstrate the effectiveness of the iterative optimizing process. Then
8

---

Published as a conference paper at ICLR 2026
Dpsk Speetup Liama Speedup
Dpsk Acceptance Length
Ulama Acceptance Length
Exgle3 Baseline
Mean Speedup
4.2
-6.0
Acceptance Length
Mean Speedup
4.2
4.5
4.0
4.0
3.9
1.5
3.9
3.0
IterO
itell
iter2
iter3
110
-leefh
Figure 4: Left: The effectiveness of our iterative training strategy on Llama3-8B and Dpsk-8B models. Right: The contribution of each component on Vicuna- 13B model.
conduct an ablation study that isolates the contributions of the depth and size policies in Section 6.2 To justify our reinforcement learning design, we further experiment with other reward signals in Section 6.33 and observation spaces in Appendix A.4 Taken together, these experiments underscore the effectiveness of each component within our LTD method.
6.1 ANALYSIS OF ITERATIVE OPTIMIZING
We investigated the effectiveness of our iterative optimization strategy on the Llama3-88 and Dpsk- 8B models. The results presented in Figure a (Left), with comprchensive data available in Table and Table in Appendix A.5 The process begins at Iteration 0 (Iter0) with naive combination of apply the alternating optimization schedule described in Section Specifically, we first optimize the size policy I), then the depth policy (Iter2), and finally the size policy again (Iter3). During each stage, the parameters of the other policy are held frozen. The results affirm the benefits of this iterative approach.
the initially trained policies. Subsequently,
Each optimization step yields progressive improvements both mean speedup and acceptance length, confirming that the policy co-adaptation leads to more synergistic system. While the most significant gains occur in the initial transition from IterO to Iterl, the impact of subsequent iterations is model-dependent. For Llama3-8B: performance saturates after Iter2, with Iter3 offering only negligible benefits. In contrast, the Dpsk-8B model continues to show substantial improvement through Iter3. These findings validate the effectiveness of our iterative optimizing strategy.
6.2
IMPACT OF KEY COMPONENTS
To analyze the contribution of key components in our proposed method, we conducted compre- hensive ablation study on the Vicuna- 13BB model presented Figure (Right). The detailed results are presented in ia Appendix A.5 AA.5 We evaluated three primary variants: 1) Size: The size policy was ablated, relying solely on the depth policy. 2) Depth: The depth policy was ablated, relying solely on the size policy. 3) - Iterative Training: The initial token and depth policies from Section were directly combined without our proposed iterative optimizing.
The results clearly demonstrate that the depth policy plays important role to the overall acceleration. While the size policy also yields a speedup over the baseline, its impact is less pronounced. Fur- thermore, naive combination of the two policies leads to suboptimal performance. This highlights the efficacy of our iterative training scheme, which allows the policies to co-adapt and achieve a synergistic acceleration. In summary, these results validate the individual contributions of both policies and the essential role of the co-adaptive process that unites them.
6.3 EFFECTIVENESS OF THE REWARD SIGNAL
To validate the effectiveness of using throughput as the reward signal, we designed and compared three distinct reward functions optimize size policy alone Vicuma model: 1) the acceptance length per cycle, 2) the time cost per cycle, and 3) throughput per cycle. Our results in Table P demonstrate that employing throughput the reward yields the highest speedup across all datasets.

---

Published as a conference paper at ICLR 2026
Table The effectiveness of reward signal on size policy, evaluated Vicuna- model.
MT-bench
Gsm8k
Alpaca
Qa
Mean
Reward
Speedup
Speedup
Speedup
Speedup
Speedup
Acceptance Length
3.77
7.70
3.81
7.20
3.71
7.05
3.06
5.61
3.59
6,89
Time Cost
3.89
6.49
3.54
5.80
3,48
5.79
2.87
4.54
3,45
5.66
Throughput
4.40
7.68
4.43
7.10
4.23
7.00
3.47
5.56
4.13
6.84
While using acceptance length as the reward achieves the maximum т, its average speedup is substantially lower. This discrepancy occurs because this size policy proposes an excessive number of tokens for verification, leading to significant increase in verification time that outweighs the benefits of higher acceptance length. Conversely, using time cost as the reward leads to a lowest acceptance length, the policy is encouraged propose few tokens, which cannot benefit form the parallel verification from the target model, ultimately resulting in the lowest overall speedup. These findings confirm that throughput a superior reward signal it inherently balances the competing objectives of maximizing acceptance length and minimizing time cost.
7 CONCLUSION
In this paper, we introduce LTD, novel speculative decoding method that directly optimizes the throughput of each draft-and-veriff cycle using reinforcement learning. By employing two co- adaptive policies to dynamically coordinate the drafting and verification phases, LTD learns to manage the trade-off between acceptance length and time cost, achieving a synergistic acceleration. Extensive experiments five distinct LLMs across four benchmarks demonstrate that LTD consistently achieves state-of-the--e acceleration for both greedy decoding and high-temperature sampling. This superior performance underscores the effectiveness and robustness of our approach, positioning LTD as powerful solution for accelerating LLM inference.
REPRODUCIBILITY STATEMENT
In Section and Appendix A.2 we provide detailed description of our method and the training process; Section [.T and A Appendix A.3 we present the complete training parameters and evaluation datasets; and in Appendix I we document the hyperparameters used for baseline methods. Together, these details support the reproducibility our work.
ACKNOWLEEDGEMENT
We thank the anonymous reviewers for their helpful comments on this paper. This work was partially supported by National Natural Science Foundation of China projects (No. 92470205 and 62476010).
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. opt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508. 10925,, 2025.
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Scquentially-dependde draft heads for medusa decoding. arXiv preprint 4rXi::2402.05109 2024.
Oscar Brown, Zhengjie Wang, Andrea Do. Nikhil Mathew, and Cheng Yu. Dynamic depth decoding: Faster speculative decoding for 1lms. CoRR, 2024.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. In Proceedings of the 4 lsst International Conference Machine Learning, pp. 5209-5235, 2024.
10

---

Published as a conference paper at ICLR 2026
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Bean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL httpts
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemyy Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint
2021.
Tencent AngelSlim Project Contributors. AngelSlim, 6 2025. URL https://github.com
Tencent Cangeettt//glllll
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning,
2025. http:t/ps///aaxvv....o///////// 12948
Cunxiao Du, Jing Jiang. Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. arXiv preprint arXiv:2402.000082 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama herd of models. arXiv preprint arXiv:2407.21788 2024.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:240..16710, 2024.
Aayush Gautam, Susav Shrestha, and Narasimha Reddy. Token-driven gammatune: Adaptive calibration for enhanced speculative decoding. arXiv preprint arXiv:2504.00030, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12940 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 00Xi::2009.033000 2020.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Gene, Kurt Keutzer, Amir Gholami, and Yakun Sophia Shao. Speed: Speculative pipelined execution for efficient decoding. In En- hancing LLM Performance: Efficacy, Fine-Tuning, and Inference Techniques, pp. 19 32. Springer, 2025.
Shijing Hu, Jingyang Li, Xingyu Xie, Zhihui Lu, Kim-Chuan Toh, and Pan Zhou. Griffin: Effective token alignment for faster speculative decoding. arXiv preprint arXiv:2502.11013 2025.
Kaixuan Huang, Xudong Guo, and Mengdi Wang. Speedec++: Boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715 2024.
Kaixuan Huang, Xudong Guo, and Mengdi Wang. Specdec++: Boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715. 2025a.
Kaiyu Huang, Hao Wu, Zhubo Shi, Han Zou, Minchen Yu. and Qingjiang Shi. Specserve: Efficient and slo-aware large language model serving with adaptive speculative decoding, arXiv preprint arXiv:2503. 000.0000..0006. 2025b.
11

---

Published as a conference paper at ICLR 2026
Feiye Huo, Jianchao Tan, Kefeng Zhang, Xunliang Cai, and Shengli Sun. C2t: A classifier based tree construction method in speculative decoding. arXiv preprint arXiv:2502.13652, 2025.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguisties, 7:453 4663-466,20 2019.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286 PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 7421-7432, 2024.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference ac- celeration of large language models via training-time test. arXiv preprint arXiv:2503.00 840, 2025a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires 15077.
rethinking feature uncertainty, 2025b. URL http:/:://wwwww//////////
Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu, and Xiao Sun. Pearl: Parallel speculative decoding with adaptive draft length. arXiv preprint arXiv:2408. / 1850, 2025.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, lon Stoica, and Hao Zhang. Online speculative decoding. arXiv preprint arXi::2310.07177 2023.
Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz. Dynamic speculation lookahead accelerates speculative decoding of large language models. arXiv preprint arXiv:2405.04533 2024.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint arXiv:2305.09781, 2023.
Zongyue Qin, Zifan He, Neha Prakriya, Jason Cong, and Yizhou Sun. Dynamic-width speculative beam decoding for efficient IIm inference. arXiv preprint arXiv:2409. 16560, 2025.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. : Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268): 2021. URL /ttp:://mmai org/papers//22//
1364.31.22 mtmm
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (707.06377 2017.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. Advances in Neural Information Processing Systems, 31, 2018.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-followinng model. Stanford Center for Research on Foundation Models. https://erfm. stanford. edtu/2023//33//1///aaee himl, 3(6)77, 2023.
Qwen Team. Qwen3 technical report, 2025a. URL
09388
Qwen Team, Qwq-32b: Embracing the power of reinforcement learning, March 2025b. URL https://qwent thub.. oo/blog htp-///w---2 32b/
12

---

Published as a conference paper at ICLR 2026
Jikai Wang, Yi Su, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, and Min Zhang. Opt-tree: Speculative decoding with adaptive draft tree structure. arXiv preprint arXiv:2406.1722 2025.
Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge. Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. Lun-Wer Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 7655-66771 1, Bangkok, Thailand, August 2024. A Association for Computational Linguistics. doi: 10.18653/v120222 hww. fin.ingg--all.456. URL wtt.tttp://// www.caannnoolloo..oooooooooo www.oog..ooooooooooooooooooooooo 2024. f nd wiw-diinnna.... 4566
Yunfan Xiong, Ruoyu Zhang, Yanzeng Li, and Lei Zou. Dyspec: Faster speculative decoding with dynamic token tree structure. World Wide Web, 28(3):36, 2025.
Hossein Entezari Zarch, Lei Gao, Chaoyi Jiang, and Murali Annavaram. Del: Context-aware dynamic exit layer for efficient self-speculative decoding. arXiv preprint rrXiv:2504005598, 2025.
Jun Zhang. Jue Wang. Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.088 08108 2023.
Lefan Zhang, Xiaodan Wang. Yanhua Huang, and Ruiwen Xu. Learning harmonized representations for speculative sampling. arXiv preprint arXiv:2408. 15766, 2024a.
Situo Zhang, Hankun Wang, Da Ma, Zichen Zhu, Lu Chen, Kunyao Lan, and Kai Yu. Adaeagle: Optimizing speculative decoding via explicit modeling of adaptive draft structures. arXiv preprint arXiv:2412.18910. 2024b.
Ziyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Rui Wang, and Zhaopeng Tu. Draft model knows when to stop: A self-verification length policy for speculative decoding. arXiv preprint arXiv:2411.1846) 2024c.
Lianmin Zheng, Win-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623, 2023.
Shuzhang Zhong, Zebin Yang, Meng Li, Ruihao Gong, Runsheng Wang, and Ru Huang. Propd: Dynamic token tree pruning and generation for Ilm parallel decoding. arXiv preprint arXiv:2402. 13485, 2024.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, decoding via knowledge distillation. arXiv preprint arXhv:231000861,, 2023.
Kagy, and Rishabh Agarwal. Distillspec: Improving speculative
13

---

Published as a conference paper at ICLR 2026
A DETAILED SETTINGS
A.1 BASELINES SETTINGS
Table summmrizzs the key difference between methods and various dynamic draft tree methods. For DDD, cumulative rog-probabilities were monitored after the 5th, 7th, and 9th layers, with a minimum threshold of -0.3. In Gammatune, "min half the default depth, Tmax 1.5 times the default depth, and the adaptation rate was 0.2. For SVIP, we followed the original paper, using an entropy factor of 0.15 and a minimum entropy of 0.3. For DDD and SVIP, directly adopted the hyperparameters provided by the original authors, while we determined the hyperparameters ourselves based on experiments as the original paper did not specify them.
For FFN-based methods, samples from HumanEval generated by default Eagle3 were used to extract hidden states, token scores, entropy, and binary flag indicating whether draft token was accepted by the target model. Equal numbers of positive and negative samples were used for training. C2T was trained with hidden size of 64 for 50 epochs, while Disco used hidden size of 128 with the same training epochs; SpecDec++ employed ResNet with hidden size 1024 for 5 epochs. All models were trained with AdamW W (Ir = le-4). Interestingly, we observed that including entropy in the observation space can sometimes reduce speedup. This is due to tradeoff between the overhead of frequent entropy computation and the accuracy of the FFN decision. Based on our validation experiments, used entropy as observation feature for Disco, while omitting it for C2T.
We also performed a grid search to determine the optimal byperparameters for the Eagle3 model on the HvmanEval benchmark as strong baseline. Our search space included draft depths ranging from 4 to 12 and verification sizes from 40 to 240. The configuration that yielded the best performance was then selected for subsequent experiments.
A.2 LTD
A.2.1
S1ZE POLICY
The Size Policy governs the selection of the verification size, V-the number of candidate tokens from the speculative tree that are passed the target model. Its objective is to assess the aggregate quality of the draft tree and select value for V that optimally balances the computational cost of verification against the potential number of accepted tokens.
Architecture: Both the policy and value networks are implemented as two-layer feed- forward networks (FFNs) with hidden layer dimensions of [1024, 256].
Training: To promote robustness to varying draft tree structures, the policy is trained on episodes where the generation depth is randomized, sampled uniformly from integers between and 12.
The state representation provided to the agent is designed to include the key outcome of the drafting phase. It is fixed-size vector primarily composed of the log-probability scores of all candidate tokens from the generated tree. For a tree of depth D, this includes scores for up to 1 1+ W (D 2) x W tokens, with padding applied to ensure a consistent vector length. This score vector is then concatenated with scalar features representing the final draft depth (D) and the total length of the input sequence(L).
Observation Space:
Action Space: The agent's 's action space is discrete, comprising 12 actions that map linearly to a verification size V ranging from 20 to 240.
A.2.2 DEPTH POLICY
The Depth Policy executes sequential policy, deciding at each depth level whether to continue expanding the draft tree (CONTINUR) or to terminate (STOP). Given that this policy is executed multiple times within single generation step, its inference latency a critical performance factor.
the drafting phase and proceed to verification
Architecture: To minimize this latency, utilize lightweight architecture. We find that single-layer FFN(with hidden layer dimensions 1024) achieves an effective balance between predictive performance and inference speed.
14

---

Published as a conference paper at ICLR 2026
Table Comparison of Different Dynamic Draft Tree Methods
Method Nume
Predier Policy
Observazion Space
Dynamic Depth
Tiinning Methods
Objective
2024
Probubility
Depth
SVIP
Heuristic
Entropy
Depth
La
Ganmato
Cona Pertanireeii Ceum 2055
Heuristic
Token Acceptance Rate
Depih
La
Discn Momou
FFN
Probubility, Entropy, Draft Deyth
Depth
FT
La
SpecDea
FEN
Hidden States
Depth
Probubility Entropy, Draft Depth
FFN
Probubility, Total Length, Draft Depth
Both
Throughput
The state for this sequential decision-making process is updated at each depth level. The observation vector is primarily composed of the log-probabilities of the candidate tokens generated at the current expansion frontier. For beam search of width W, this comprises W candidate scores. This probability vector is then concatenated with two scalar features: the current draft depth (D) and the total length the input sequence(L).
Observation Space:
Training Strategy: establish stable training curriculum, we first trained an initial depth policy against a fixed verification size of V 60 tokens, which mirrors the heuristic used in the Eagle3 baseline.
Action and Reward:
The agent outputs a binary action at each step. A numerical reward, based on the final throughput of the episode, is dispensed only upon selecting the STOP action. Intermediate CONTINUE actions yield zero reward. To counteract the policy's potential bias towards premature termination, we employ a high discount factor (7 0.999), thereby encouraging the exploration of deeper draft trees to get more rewards potentially.
Table Contribution of each components of LTD, evaluated on Vicuna- 13B.
MT-hench
Gsm8k
Alpaca
QA
Mean
Method
Speedup
Speedup
Speedup
Speedup
Speedup
Eagle3
4.48
7.22
4.36
6.58
4.16
6.47
3.43
5.12
4.11
6.35
-Depth
4.40
7.68
4.43
7.10
4.23
7.00
3.47
5.56
4.13
6.84
-Size
4.77
7.75
4.46
6.82
4.36
6.59
3.55
5.12
4.29
6.57
-Iterative Training
4.54
8.29
4.48
7.41
4.28
7.08
3.51
5,45
4.20
7.06
LTD
4.64
8.24
4.59
7.38
4.40
7.01
3.66
5.37
4.32
7.00
A.3
TRAINING DETAILS FOR LTD
The policy (actor) and value (critic) functions are both implemented Multi- Perceptrons (MLPs).
Size Policy (TV) The policy and value networks for the size part share an architecture consisting of two hidden layers with 1024 and 256 units respectively, using ReLU activation functions. This policy is trained with standard discount factor of y = 0.9.
Depth Policy (πp) To minimize decision latency, the depth policy utilizes a more lightweight architecture with a single hidden layer of 1024 units, while value networks remains same to the size part. A key distinction is the use of high discount factor (7 0.999). This value is chosen specifically to mitigate the policy's potential bias towards premature termination, thereby encouraging the cxploration of deeper draft trees by prioritizing long-term rewards.
Shared Parameters For both policies, the PPO surrogate loss is optimized for 20 epochs per update cycle, and an entropy coefficient of 0.01 is applied to foster exploration. We use a learning rate of 1e-3, which decays linearly to zero following warm-up phase All other hyperparameters were kept at their default values as provided by the Stable-Baa Baselines3 library.
the initial I 1 of training steps.
15

---

Published as a conference paper at ICLR 2026
Table 5: Speedup ratios and average acceptance lengths T of different methods during Temperature= L31 8B represents - V 13B represents Vicuna- 1aatn--33Bv1... Dpsk 8B represents DeepSeek- Peapeeek-RI-isttiiee -i---Rsil-i-LLLMMA 8B. For the Qwen3-1 14B and Qwen3-32B models, their corresponding Eagle3 models lack default settings. Therefore, only the results from Grid Search are presented. GT, Spectt, and Gs stand for Gammatune, SpecDectt, and Grid Search, respectively.
MTT-bench
Gsm8k
Alpaca
Qa
Mean
Model
Method
Speedup
Speedup
Speedup
Speedup
Speedup
Temperature=1
Kagle3
2.85
3.43
Eagle3+DDD
2,89
3,40
3.35
4.09
3.38
4.59
2.36
2.95
3.00
3.76
Eagle3+Svip
2.74
3.41
3,21
4.92
3.19
4.95
2.31
3.20
2.86
4.12
Bagle3+GT
2.75
3.51
5.12
3.24
4.04
Fagle3+C2T
2.78
4.11
3.28
5.24
3.24
5.47
2.27
3.06
2.89
4.47
131
Eagle3+Disco
2.82
3.82
3.26
5.08
3.30
5.18
2.20
2.88
2,90
4.24
Spect3+
2,92
4.20
3.33
4.74
5.51
2.35
3.28
3.01
Fagle3+GS
2.79
4.08
3.36
5,50
3.46
5.79
2.31
3,40
2.98
4.69
Eagle3+1TD
2.85
4.02
3.35
5.09
3.43
5.68
2.39
3.26
3.01
4.51
Hagle?
3.601
5.55
3.68
5.64
3.40
5.49
3.08
4.64
3.44
5.33
Eaglc3+DDD
3.40
5.05
3.58
5,46
3.43
3.34
4.91
Eagle3+Svip
3.18
5.73
3.26
5.78
3.21
5,39
2.61
4,44
3.07
5,34
138
Bagle3+GT
3.43
5.36
3.32
5.26
3.39
4,95
2.99
4,44
3.28
5.00
Eagle3+C2T
3.38
3,41
5.63
3.27
3.22
Eagle3+Disco
3.48
5.55
3.67
5,74
3.36
5,51
2.88
4.59
3.35
5,35
Baglc3+Spec*+
3.61
5,58
3.64
5.63
3.34
5.39
3.02
4.55
3.40
5.29
Ragle3+GS
3.58
Eagle3+LTD
3.82
6.40
3.75
6.38
3.65
6.20
3,26
4.97
3.62
5.99
Eagle3
3.07
4.74
3.97
6.89
2.79
4.53
2.44
4.06
3.07
4.66
Dpsk 8B
Bagle3+GS
3.05
5.19
4.14
7.64
2.76
4.83
2.33
4.44
3.07
5.03
Eagle3+LTD
3.18
5.20
2.83
4.76
2.57
3.15
5.09
Bagle3+GS
2.05
3.13
2.62
3.83
2.04
2.78
1.68
2.57
2.10
3.08
Qwend
Eagle3+LTDD
2,16
3.11
2,73
3.85
2.11
2.76
1.82
2.55
2.21
3.07
Qwen3 328
Eagle3+GS
1.56
2.89
2,10
3.95
1.63
2.73
1.33
2.46
1.66
3.01
ETgle34LTD
2.17
2.78
2.83
3.76
2.16
2.59
2.41
2.26
2.89
Table 6: Ablation Study for Observation Space on size policy evaluated on Vicuna-13B.
MT-bench
Gsm8k
Alpaca
Mean
Reward
Speedup
Speedup
T
Speedup
Speedup
Speedup
P+L
4.35
7.68
4.30
7.10
4.18
7.00
3.35
5.55
4.05
6.83
P+D
4.31
7.68
4.35
7.10
4.22
7.00
3.41
5.56
4.07
6.84
P4D+L
4.40
7.68
4.43
7.10
4.23
7.00
3.47
5.56
4.13
6.84
P+D+L+H
4.40
7.53
4.32
6.97
4.22
6.94
3.44
5.51
4.10
6.74
P+D+L+B
4.36
7.68
4.36
7.10
4.28
7.00
3.43
5.56
4.11
6.84
A.4 ABLATION STUDY FOR OBSERVATION SPACE
We conducted an ablation study to determine the optimal composition of the observation state for our policy. While prior methods have incorporated variables such as predicted probabilities (P), hidden states (H), entropy (E), input length (L), and draft depth (D) into their state representation, the inclusion of such computationally expensive features is only justified if the resulting accuracy gains outweigh the associated latency cost.
Our results in Table indicatee that our proposed observation space configuration achieves the highest average speedup across these four benchmarks. Notably, we observe that many competing methods yield similar acceptance lengths. This suggests that while their proposed verification sizes are sufficient to capture the ground-truth token sequence, this often comes at the cost of excessive verification time. The superiority of our method stems from two key advantages. First, compared to approaches that rely computationally intensive inputs like entropy or full hidden states, method incurs significantly lower overhead when constructing its observation state. Second, contrast to baselines that omit draft depth total length information, model learns to propose more
16

---

Published as a conference paper at ICLR 2026
5.0
Acceptance Length
8.0
Acceptance Length
Acceptance Length
Acceptance Length
4.8
4.8
1.6
1.
D-1
D-2
D-3
D-4
D-5
D-6
D-7
D-8 D-9 D-10
V-10
V-20
V-40
V-80
V-160
Draft Depth
Verification Size
Figure 5: The average acceptance length for Llama-3.1 3.1BBBBBBBBBBBB Instrccttt using Eagle3 on HumanEval dataset. Left: Change the draft depth with verification size fixed to 60. Right: Change the verification size with draft depth fixed to 8.
3.1--B-IInsttr
conservative number of candidate tokens. This cautious strategy is crucial for reducing verification latency.
A.5 DETAILED RESULTS FOR ABLATION STUDIES
The detailed results for ablation studies are shown in the section. contains the results for contribution of each components of LTD in Table and the results for iterative optimizing in Table 7 and Table
Table 7: Llama3. 1-BB Iterative Optimizing
MTT-hench
Gsm8k
Alpaca
Qa
Mean
Method
Speedup
Speedup
Speedup
Speedup
Speedup
Eagle3
3.70
6.39
3.58
6.44
4.26
3.17
3.68
6.28
Iter0
3.88
6.46
3.76
6.40
4.34
6.82
3.50
5.12
3.87
6,20
Iterl
3.95
6,50
3.80
6.47
4.36
6.89
3,50
5.18
3.90
6,26
Iter2
3.96
6.74
3.79
6.73
4.53
7.31
3.41
5.44
3.92
6.56
Iter3
3.93
6.74
3.73
6.73
4.57
7.31
3.44
5.44
3.92
6.56
Table 8: Dpsk-8B Iterative Optimizing
MTT-benhh
Gsm8k
Alpaca
Mean
Method
Speedup
Speedup
Speedup
Specdup
Speedup
Eagle3
3.67
5.82
4.73
7.39
3.65
5.63
3.14
5.02
3.80
5.53
IterO
3.88
5,95
4.69
8.03
3,68
5.63
3.37
5.07
3.91
5.72
Iterl
3.96
5.98
4.92
8.14
3.86
5.67
3.46
5.08
4.05
5.78
Iter2
4.02
6.22
4.94
8.36
3.71
5.91
3.52
5.29
4.05
5.97
Iter3
4.05
6.32
4.98
8.53
3.98
6.00
3.62
5.37
4.16
6.08
A.6
INTERACTION BETWERN THE DEPTH AND SIZE POLICIES
To validate the effectiveness of our iterative training, we analyzed the test set distributions of the initial separately trained policies (" ('Iter0') ) and the final iteratively trained LTD policies, shown in Figure 6 Specifically, we examined the relationship between the number of candidate tokens and draft depth conditioned on acceptance length. As observed, IterO exhibits a static pattern regardless of acceptance length, predominantly clustering in the region where the draft depth is < 6 and the token count is < 60. Instances with a shallow depth (< 6) but large token budget (> 60) are negligible. In contrast, LTD demonstrates superior adaptability across different acceptance lengths. For large acceptance lengths, significant proportion (43%) of cases utilize both large token budget and deeper tree (both 60). Furthermore, when the acceptance length is malll-indicatin higher
17

---

Published as a conference paper at ICLR 2026
Distribution of depth and tokens in IterO
Acceptance Length
Acceptance Length Dmaft Depth
Draft Depth
2-6
266
100
100
Candidate Tokems
0.74
6.0
Canddidate Tokens
3.00
50
> 2.60
0.11
3.16
>.60-
0.00
428
Distribution of depth and tokens in LTD
Draft Depth
Draft Depth
6
26
26
6
100
100
Candidate Tokens
Candidate Tokens
5.60
0.35
3.00
$.00
922
3.00
>x 60-
0.38
2.27
25
035
043
25
Figure 6: We report the interaction between depth policy and size policy on Llama3-8B for IterO and LTD.
generation difficulty- LTD frequently adopts 'shallow but wide' strategy (depth < tokens 60). This strategy of deploying more candidate tokens effectively mitigates the reduction in acceptance length caused by increased difficulty.
A.7 LATENCY OF DEPTH AND SIZE POLICIES
We analyzed the latency overhead of our depth and size policies on both Llama-3.1 1--.-.....BBBB--- --8--Inttttt and Vicuna- 1aaa-13B-a. 33-v1.3. .3. reporss the average generation time per sample, along with the proportional time costs of the size and depth policies. The results demonstrate that the computational overhead introduced by our policies is minimal: the total overhead is less than 1.5% on Llama and even lower (1.2%) on the larger Vicuna model. Notably, the latency of the depth policy is approximately twice that of the size policy. Although we utilize lightweight single-layer network, the depth policy incurs a higher total cost due to its more frequent invocation compared to the size policy.
A.8 SIZE SELECTION FOR MLP POLICIES
There is an inherent off in MLP design: we aim for the accuracy gains from a larger MLP to outweigh the increased inference cost. To validate design choice. conducted ablation study on 1aama-3.1-8B-Instiii
comparing three architecture sizes for the Size Policy (Hidden dimensions: [1024), [1024, 256), and [1024, 512, 256]), the results reported in Table 10 The results show significant performance jump when increasing from to 2 layers. However, the marginal gain from layers minimal. Therefore, selected the 2-layer architecture ([1024, 256]) for size policy to strike the optimal balance between training simplicity and inference efficiency.
B TRAINING DATA GENERALIZATION
To evaluate the generalization capabilities of our model across diverse domains, employed the MMLU (Hendrycks et al.) 2020) benchmark, a widely adopted dataset covering 57 tasks ranging from elementary mathematics and US history computer science and law. We compared the speedup
18

---

Published as a conference paper at ICLR 2026
Table 9: Latency and overhead analysis of dynamic policies on Vicuna-131 and Llama31-8B. Policy latency is measured in milliseconds (ms), total generation latency in seconds (s), and overhead in percentage (%)
Time per Sample
MT-bench
Alpaca
Qa
Gsm8K
Mean
Llama-3, (llma-3..-8B--iee -n8-Irrtt
Size Policy Lateney (ms)
21.4
6.22
6.36
5.83
9.96
Depth Policy Latency (ms)
40.8
12.4
10.2
11.9
18.8
Total Latency per Sample (s)
4.48
1.28
1.27
1.24
2.07
Size Policy Overhcad (%)
0.478
0.485
0.500
0.469
0,483
Depth Policy Overhead (%)
0.910
0.969
0.803
0.955
0,909
Total Overhead (%)
1.39
1,45
1.30
1.42
1.39
Vicuna-13B-vI.3
Size Policy Latency (ms)
13.7
4.92
4.90
5.56
7.26
Depth Policy Latency (ms)
27.9
9.61
8.39
12.0
14.5
Total Latency per Sample (s)
3.59
1.16
1.10
1.37
1.81
Size Policy Overhead (%)
0.381
0.423
0.444
0.406
0.414
Depth Policy Overhead (%)
0.777
0.826
0.761
0.874
0.810
Total Overhead (%)
1.16
1.25
1.20
1.28
1.22
Table 10: Speedup of the Size Policy across three architecture configurations on Llama-3.1-88B Instruct.
Configuration
MT-bench
Gsm8k
Alpaca
Qa
Mean
[1024]
3.84
3.99
4.39
3.22
3.86
[1024, 256]
3.95
4.03
4.44
3.24
3.92
[1024, 512, 256]
3.96
4.07
4.43
3.25
3.93
ratios of our proposed method, LTD,, against the Eagle3 baseline the L model, as illustrated in Figure The results demonstrate that LTD outperforms Eagle3 in 54 out of the 57 domains, underscoring the robust generalization of our approach across differentt fields. In particular, LTD achieves speedup improvement of over 10% in mathematics and logic-related tasks. It worth noting that among the top five domains where LTD achieves the most significant gains over Eagle3, the baseline exhibits below-average speedup performance. This indicates that our method effectively compensates for the deficiencies of Eagle3 specific domains. Furthermore, efficacy of our approach is not limited to scenarios where Eagle3 underperforms: LTD maintains superior acceleration even in domains such as Human Aging and Astronomy, where Eagle3 already demonstrates performance significantly above its average.
C DETAILS OF REINFORCEMENT LEARNING
C.I ANALYSIS OF TRAINING STEPS
To validate the appropriateness of our training budget, analyze the relationship between the total training timesteps and the coverage of the dataset (HumanEval). Taking the Llama model as an example, the HumanEval dataset contains total of approximately Nioua 27829 new tokens (ground truth).
C.I.1
STEPS FOR DEPTII POLICY
For the depth policy, the total training budget set to Baepth 1, 000, 000 steps. In our reinforcement learning (RL) formulation, one step corresponds to single draft action or a verification action.
However, speculative decoding is stochastic; not all drafted tokens accepted. To estimate the effective number of dataset traversals (epochs), we must account for the acceptance rate a. Assuming moderate acceptance rate (e.g., 0.7), the number of draft steps required to generate the full
19

---

Published as a conference paper at ICLR 2026
Speedup Ratio LTD/Eagle3 (%)
Eagle3 Speedup Ratio
Logical Fallacies
Mcan Speedup
Computer Seuurity
Chemistr
Microcconomies (H)
Human Aging
Governmess And Polities
Acowanking (P)
Modical Centties
Medicine
Virology
Moral Seanarios
Jurisprudence
Blology (H)
Abstrut Algubma
Astronommy
Physies (H)
World History (H)
Chemistry (3I)
Interuatonaal Law
Formal Logic
Law
European History (3)
Machine Learning
Biology (C)
Macrocoonomies (H)
Computer Science (31)
Psychology (P)
Prolistory
World Religions
Psychology (H)
Miscdlancous
Computer Science (C)
Pusiness Ethics
Nurison
Socurit Studics
Us Histury (31)
Human Sexuality
Glabal
Moral Disputes
Clincal Knowledge
Pablic Relarions
Eovmmmetries
Conceptual Physics
Anatomy
Us Foreign Policy
Sociolugy
Managemast
Geography (H)
Mathematies (H)
Marketing
Elearical Engineering
Elemeszary Mathematics
Physies (C)
Eagle3 Spoodup
Medicine (C)
90
100
110
2.5
3.5
4.0
Speedup Ratio LTD/Eagle3 (%)
Eagle3 Speedup Rutio
Figure 7: Speedup comparison between LTD and Eagle3 on the Llama-3.1-88 In-truttct model. We evaluated performance across 57 subtasks from MML.(e.g.. mathematics, history, law), using 80 sampled examples per subtask. Domain levels are abbreviated as (High School), (College), and P (Professional). Left: The relative speedup of LTD normalized against Eagle3; values greater than 100% indicate LTD outperforms Eagle3. Right: The absolute speedup ratio of the Eagle3 baseline across different subtasks.
dataset is greater than the number of accepted tokens:
StePepoch Niotal 829
0.7 R3 39, 756 steps
(6)
Consequently, the number of dataset traversals Kuepth is calculated as:
Baepth
1, ..0000 000 ~ 25.1 passes
K Kdepth
(7)
Stepopoch
39, 756
In ideal scenario where 1 (or assuming 1 step R= 1 new token lower bound estimate for steps), the coverage would be approximately 1, 000, 000/27, 829 zo 35.9 passes. Thus, our setting of 1M steps corresponds roughly 25-40 traversals of the dataset.
20

---

Published as a conference paper at ICLR 2026
Given that the depth policy must learn to identify optimal termination points across varying contexts, and considering that PPO is an on-policy algorithm requiring repeated sampling of trajectories, this magnitude of exploration is standard and necessary for convergence, rather than excessive.
C.1.2 STEPS FOR SIZE POLICY
For the size policy, the budget is set to Broken 100, 000 steps. This policy determines the total draft budget (action space size: 12 configurations, ranging from 20 240 tokens). decision is made only once per speculative cycle.
Assuming an average acceptance length of tokens per cycle, the policy is invoked once every generated tokens. The steps required to traverse the dataset once are:
Niva
27. 829
Step Pepceh
~ 5, 566 steps
(8)
The total dataset traversals Kroken is:
Broken
100, 000
Kroken
2 18.0 passes
(9)
StePopoch
5, 566
This results in approximately 18-20 traversals. Considering the policy must explore discrete action space of 12 distinct configurations to optimize the trade-off between latency and generation length, this sampling volume is conservative and efficient for stabilizing the policy gradient.
C.2 CONVERGENCE OF THE THROUGHPUT-BANE BASED REWARD
To explicitly illustrate the dynamics of our throughput-based based reward during training, we report the reward curves and validation speedup ratios for the Vicuna- 13B size and depth policies, as shown in Figure 3 As observed, the rewards for both policies increase steadily as the number of training steps increases, and our allocated training budget is sufficient to ensure convergence. Regarding performance on the validation set, observe an initial improvement followed by declining trend. To prevent overfitting to the training set, we select the checkpoint that achieves the highest speedup our final model.
on the validation
C.3 TRAINING TIME COST
Due to the lightweight nature of the two policies, the RL training phase is highly efficient. For instance, training the Size Policy (100k steps) requires only 2.0, 2.5, and 3.5 hours for Llama3-8B, Vicuna-13B. and Qwen3-32B. respectively. Similarly, the Depth Policy (IM steps) takes 5, 5, and 7 hours on the same models. All experiments were conducted on a single NVIDIA 100 GPU. Even with three iterations of training, the total computational cost remains below 30 GPU hours. In practice, these policies are trained once, whereas decoding dominates the runtime real-world deployments; thus, the marginal training overhead is effectively amortized by the improved inference efficiency. Given that LTD improves the speedup ratio by up to 36.4% over Eagle-3, this additional training cost represents highly favorable trade-off for aatency-sensitivy applications.
C.4 PPO SELECTION
We deliberately select PPO since it can be integrated naturally into the online speculative decoding process, allowing us record rewards during standard inference steps. Other RL methods, such as the currently popular GRPO, are not suitable for our task. We know GRPO requires repeated sampling from the same state to calculate baselines. In the context of speculative decoding, this would necessitate either repeatedly reconstructing the KV cache for different samples (computationally expensive) or generating multiple continuations sequentially (which introduces timing mismatches with the real inference environment). Therefore, we select PPO to train our policies.
D LTD EFFECTIVENESS ON GRIFFIN
To further validate the effectiveness of our approach across different speculative decoding methods, selected Griffin (Hu al. 2025). another method based on the dynamic framework. Griffin
21

---

Published as a conference paper at ICLR 2026
Table 11: Speedup ratios and average acceptance lengths of different methods on Griflin using Meta-Llama-3-88 In-srucctt during greedy decoding.
MT-bench
Gsm8k
Alpaca
Qa
Mean
Method
Speedup
7
Speedup
Speedup
Speedup
Speedup
Griffin
1.83
4.82
3.18
5.16
2.98
4.79
2.20
3.96
2.55
4,68
Griffin+GS
2.81
5.11
3,19
5,47
2.95
5.04
2.18
4.10
2.78
GriflinLTD
2.88
4.98
3.13
5,45
2.98
5.01
2.23
4.10
features a distinct training loss and draft model architecture compared to Eagle3, but with a slightly inferior speedup. Using -itta-llmaa--SB-Innttte ratios of the default setting of GRIFFIN against our LTD method. The results in Table TT demonstrate that our method achieves the state-of-the-ar speedup on Griffin as well.
(Dubey et al., 2024), we compared the speedup
E STATEMENTS
USE OF LARGE LANGUAGE MODELS (LLMS)
The use of L.MSs were strictly limited to language polishing, such improving grammar, refining sentence structure, and enhaneing the overall readability of the text. All suggestions and modifications proposed by the LLMs were critically reviewed and manually edited by the authors. This process ensured that the final text accurately and faithfully represents authors' original ideas, arguments, and findings. The core intellectual contributions of this work-including the research ideation, experimental design, data collection, analysis, and interpretation of results are are exclusively the work of the human authors. The LLM played no role in these critical research activities. The authors take full responsibility for all content presented in this paper, including the accuracy of the data and the validity of the conclusions.
22


Reviews:
[{"review_id": "KJ6FJF1ufw", "reviewer": "Reviewer_WBfk", "summary": "The paper introduces Learning to Draft (LTD), a novel method for accelerating large language model (LLM) inference using adaptive speculative decoding. Unlike previous methods that rely on static configurations or optimize for proxy metrics like acceptance length, LTD directly optimizes for the throughput of each draft-and-verify cycle, accounting for the crucial time cost of both the drafting and verification phases. The authors solve this problem using a Reinforcement Learning (RL) environment and train two distinct, co-adaptive policies: a depth policy to dynamically control the draft tree's depth (draft cost) and a size policy to manage the verification size (verification cost). These policies are jointly optimized to maximize the throughput, defined as the number of accepted tokens divided by the total time ($L_A / T_{total}$). Through extensive evaluations, LTD demonstrated significant throughput improvements of up to 36.4% over the state-of-the-art Eagle3 baseline across various LLMs and tasks, proving a more robust and efficient dynamic strategy for speculative decoding.", "questions": "1. How does the computational cost of training and deploying the two co-adaptive policies using the RL environment compare to the total inference time savings achieved? Is the initial overhead of the RL training process outweighed by the long-term throughput gains in practical, high-volume production settings?\n\n2. If the small draft model's quality degrades (e.g., due to a mismatch between its training data and the target task) how effectively can the adaptive depth and size policies mitigate the resulting low token acceptance rate to maintain a high throughput? What is the minimum acceptable acceptance rate before LTD's advantages over static methods disappear?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method LTD directly optimizes the practical system throughput ($L_A / T_{total}$), i.e., the number of accepted tokens divided by the total time. This ensures the strategy is genuinely maximizing inference speed while prior works mostly focused on indirect metrics like acceptance length. \n\n2. The method uses a Reinforcement Learning (RL) environment to train two dynamic, co-adaptive policies: one for the draft tree's depth and one for the verification size. This dynamic approach allows the model to continuously adjust both the drafting cost and the verification cost for optimal performance in real-time.\n\n3. LTD demonstrates significant and robust performance gains, achieving speedup ratios up to 4.32x and improving throughput by up to 36.4% over state-of-the-art baselines like Eagle3. This indicates a highly effective and generalizable strategy for accelerating LLM inference across various models and tasks.", "weaknesses": "1. Training the adaptive policies requires setting up and running a complex RL environment. This adds significant computational overhead and complexity during the training phase, which is a major barrier to adoption compared to simpler, non-adaptive speculative decoding methods.\n\n2. The core of LTD relies on accurately modeling the time cost of both the drafting and verification phases to compute the throughput objective ($T_{total}$). If the real-world environment introduces variances or non-linearities in time cost that the model does not capture, the policies trained in the simplified environment may become sub-optimal in production.\n\n3. Like all speculative decoding methods, LTD's performance still fundamentally relies on the quality of the small draft model to accurately predict future tokens. While the adaptive policies optimize the use of the draft model, they cannot compensate for a draft model that frequently generates incorrect tokens, which would lead to low acceptance rates and minimal speedup.\n\n4. The paper only compares against Eagle3 as the base framework, but does not evaluate on other tree-based methods like SpecInfer or Medusa. The contribution would be strong if more comparisons with recent adaptive methods are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4et4CVeRAN", "reviewer": "Reviewer_c2Bw", "summary": "This paper presents Learning to Draft (LTD), a reinforcement learning (RL) framework for optimizing speculative decoding efficiency in large language models.\nUnlike prior methods, which use fixed or heuristic strategies to adjust the draft depth and verification size, LTD directly maximizes throughput (accepted tokens per total time) rather than surrogate metrics such as acceptance length.\nThe authors model the drafting–verification process as an RL environment with two co-adaptive policies: a Depth Policy (controls how deep the draft tree expands), and a Size Policy (controls how many candidate tokens are verified).\nBoth are trained via PPO using throughput as a direct reward.\nLTD achieves 2.24×–4.32× speedup across five LLMs and four benchmarks, improving upon EAGLE-3 by up to 36.4% without quality degradation, and maintains robustness even under high-temperature sampling.", "questions": "- Can you report sample efficiency or convergence statistics (e.g., number of steps to reach stable reward or variance across runs)?\n- What was the rationale for selecting PPO as the optimization algorithm? Were alternative RL methods considered?\n- Have you analyzed the interaction strength between the Depth and Size policies (e.g., correlation of their actions during co-training)?\n- What motivated the choice of HumanEval as the primary training environment?\n- How much additional inference-time latency is introduced by running the two policy networks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper formalizes speculative decoding as a reinforcement learning (RL) environment that directly optimizes throughput rather than proxy metrics such as acceptance length. It introduces two interacting policies (Depth and Size), and the co-adaptive training framework is conceptually clear and empirically validated.\n- The algorithmic structure is clearly explained, and the figures effectively illustrate the RL formulation and the draft–verify cycle.\n- The proposed method demonstrates robustness across different sampling temperatures and models, showing consistent performance improvements.\n- The approach can enhance real-world LLM inference speed without retraining or modifying the base model, which makes it practically valuable.", "weaknesses": "- The RL formulation lacks theoretical guarantees or convergence analysis under the throughput-based reward.\n- The paper does not report quantitative metrics related to training efficiency (e.g., sample efficiency, or convergence speed).\n- The paper does not discuss the rationale behind selecting PPO as the optimization algorithm for policy learning, nor whether alternative RL methods were considered.\n- While the ablation studies demonstrate co-adaptation effects, the analysis remains qualitative. A more quantitative study (e.g., correlation between policy actions or mutual information) could strengthen the argument for policy synergy.\n- The policies are trained on the HumanEval code dataset but evaluated on text-based benchmarks (e.g., GSM8K, MT-bench). The rationale or analysis explaining this generalization behavior is missing.\n- The experimental section lacks comparisons with other adaptive or self-speculative methods such as DEL, or LayerSkip which would provide a stronger baseline context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s6zvvnhkS2", "reviewer": "Reviewer_MWEV", "summary": "This paper proposes Learning to Draft (LTD), a reinforcement learning-based speculative decoding method that directly optimizes throughput, defined as the number of accepted tokens per unit time in this paper, by jointly training two co-adaptive policies to control draft depth and verification size. Unlike prior static or heuristic approaches (e.g., Eagle3), LTD dynamically balances drafting and verification costs, achieving up to 36.4% speedup across five LLMs and four benchmarks while maintaining robust performance under different decoding settings.", "questions": "- In Figure 3, any intuitions as to Llama saturates while DeepSeek continues to gain advantage from iterative training?\n- Some typos: (i) Make sure to add spacing before left brackets, e.g., lines 153 and 190. (ii) Figure 3 right, \"token\" should be \"size\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a principled reinforcement learning framework that directly optimizes throughput rather than proxy metrics like acceptance length, addressing a key limitation in prior speculative decoding methods.\n- Extensive experiments across multiple LLMs and tasks demonstrate consistent and significant speedups (up to 36.4%) over strong baselines such as Eagle3. The analyses are also comprehensive and well-designed.\n- The method generalizes well to high-temperature decoding scenarios where other speculative decoding methods typically degrade.", "weaknesses": "- The RL-based policy training (on HumanEval for 100K and 1M PPO steps) is expensive and tuned on a specific dataset. I am concerned about the training cost and the transferability to unseen domains or longer-context tasks. Could you provide the training details, including the actual GPU hours and why training for such extremely long steps?\n- I am also concerned about the iterative optimization part. From Table 7 and 8, we can see that some models/datasets benefit from iterative optimization, while others demonstrate marginalized effect. The orders for optimization (size first or depth first) also lacks investigation. It would be great if the authors could provide some guidance on the design choices for iterative optimization, so that LTD will be more useful in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3t7MubaRh9", "reviewer": "Reviewer_YsHJ", "summary": "This work proposes Learning to Draft (LTD), a novel speculative decoding method that directly optimizes for throughput using reinforcement learning. LTD trains two distinct policies -- a depth policy for controlling the tree depth in drafting and a size policy for deciding the number of candidate tokens to verify -- that are co-optimized to jointly improve drafting and verification. Experimental results using the Eagle3 framework shows that LTD can yield substantial throughput improvements for greedy decoding across the Eagle3 baseline with various model families and sizes.", "questions": "Q. How do the methods compare in case of sampling (as opposed greedy)?\n\nQ. The amount of compute needed to train the policies seems to heavily depend on the target model. Have the authors done any analysis on potential policy generalization, i.e., how the two policies trained for a smaller target possibly generalize to a larger target?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper's main premise, which is that optimizing for acceptance length is suboptimal, is well explain and is empirically supported. \n\n2. The proposed RL method, which jointly trains two policies to directly optimize throughput, is intuitive and demonstrates promising empirical results for greedy decoding. The initial policy training and iterative optimization offer practical implementation benefits.\n\n3. Experiments with multiple variants of the Eagle3 baseline with five different LLMs.", "weaknesses": "1. Missing discussion on training cost. While using sufficiently well trained policies have shown to often yield substantial throughput improvements, it is unclear if the cost of RL training is justified in all cases. Can one grid search be more efficient at the end than having to do RL training (which involves initial policy training + iterative optimization), especially for those cases where LTD doesn't lead to improvements (e.g., Dpsk 8B, Qwen3 14B, Qwe3 32B on several of the benchmarks)\n\n2. Missing analysis on inference overhead. The lightweight MLP policies may add only \"negligible overhead\", more discussion on inference cost of running the policies in isolation would be useful. Also, do MLP policies with a greater capacity give better predictions? If so, what are the trade-offs between using larger MLP policies for better predictions vs. inference overhead. How do the answers change for different target models?\n\n3. Training data generalization. It is somewhat surprising that the policies trained on HumanEval data give substantial improvements for non-coding benchmarks. There needs to be more discussion on the potential limits of generalization. Does training on more in-domain data lead to better results? How do policies trained on non-coding data transfer to other types of benchmarks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)