RepSpec: Structural Re-parameterized Draft Model Training for Speculative Decoding
Authors: Feiye Huo
Jianchao Tan
Jiahao Liu
Zixu Jiang
Jiacheng Li
Jingang Wang
Xunliang Cai
Shengli Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Reparameterization
Speculative Decoding
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce RepSpec, a training method for speculative decoding that uses structural re-parameterization to temporarily expand the draft model’s capacity during training—without adding inference cost.

Abstract: As the parameter size of large language models (LLMs) continues to grow, the latency of autoregressive inference increases due to memory-bound computational inefficiency. To address this, speculative decoding has been proposed, where a large target model verifies multiple tokens generated in parallel by a smaller draft model. However, the performance of speculative decoding is fundamentally limited by the draft model’s capacity, which stems from the parameter gap between the two models. To overcome this limitation, we propose RepSpec, which combines structural re-parameterization with draft model training. During training, redundant linear structures are introduced and later merged into the backbone network during inference, thus enhancing the draft model’s training effectiveness without increasing inference cost. By applying our method to improve the current state-of-the-art approach, EAGLE, we achieve a significant improvement in accepted sequence length. Furthermore, considering the specific characteristics of the speculative decoding scenario, we explore a hybrid training strategy that combines linear and nonlinear structures, which yields a further improvement in acceptance length.

Published as a conference paper at ICLR 2026
REPSPEC: STRUCTURAL RE-PARAMETERIZED DRAFT MODEL TRAINING FOR SPECULATIVE DECODING
Feiye Huo! 2,,
Jianchao Tan
Jiahao Liu?
Zixu Jiang?
Jiacheng Li²
Jingang Wang?
Xunliang Cai?
Shengli Sun! lif
ABSTRACT
As the parameter size of large language models (LLMs) continues to grow, the la- tency of autoregressive inference increases due to memory-bound computational inefficiency. To address this, speculative decoding has been proposed, where large target model verifies multiple tokens generated in parallel by a smaller draft model. However, the performance of speculative decoding is fundamentally lim- ited by the capacity of the draft model, which is due to the parameter gap between the models. To overcome this limitation, we propose RepSpec, which com- bines structural re-parameterization with draft model training. During training, redundant linear structures are introduced and then merged into the backbone net- work during inference, enhancing the training effectiveness of the draft model without increasing inference cost. By applying our method to improve the cur- rent state-of-the-ane approach, EAGLE, we achieve significant improvement in accepted sequence length. Furthermore, considering the specific characteristics of the speculative decoding scenario, explore hybrid training strategy that combines linear and non-lincar structures, yielding a further improvement in ac- ceptance length.
INTRODUCTION
Large Language Models (LLMs) (Achiam al.) 2023 Touvron al. 2023 Bai et al. 2023) have demonstrated impressive performance across a wide range of tasks. However, they encounter sub- stantial challenges in autoregressive token generation, primarily due to the high demands memory bandwidth and suboptimal utilization of GPU resources (Patterson) 2004 |Sazzeer 2019), since gen- erating each token necessitates accessing the entire set of model parameters (Radford et al. 2019 Brown et al.) 2020). To mitigate these limitations, Speculative Decoding (SD) (Chen et al. 2023 Leviathan et al. 2023) has emerged as a promising solution. SD accelerates inference by generating several candidate tokens using smaller draft model and validating them in parallel with the larger
target model, thereby enhancing GPU computational efficiency. This technique has already been
integrated into some of the most recent and influential LLMs (Liu et al., 2024 Team
2025
Team
ct al. 2025).
Apart from rule-based approaches that do not require draft model (Fu et al. 2024 Saxena 2023
Hu et al.) 2024), speculative decoding offers two main paradigms for draft model selection. The first training-fres free approach, where the draft model is constructed by directly extracting substruc- ture from the target model, commonly referred to as self-speculative decoding (Jun Zhang 2023).
The second paradigm pairs the target model with separately trained, external smaller model as the draft model, forming a two-model system that requires dedicated training for the draft compo- nent (Li al.] 2024a; Cai al. 2024). Rule-based approaches are generally limited to specific scenarios, while training free self self-speculative decoding typically yields shorter accepted sequence lengths compared standard dual-model speculative decoding most cases. Therefore, this paper primarily focuses on dual-model speculative decoding.
'Equal contribution; 'Corespponing Author:
s1 ass. ..k...pk....... pku.edu.........e... cn; Pekkng University;
Meituan;

---

Published as a conference paper at ICLR 2026
There two main directions for optimizing dual-model speculative decoding. The first focuses improving the sampling strategy (Miao et al. 2024 Li et al. 2024b), aiming to maximize the recall of correct candidate tokens generated by the draft model within limited verification budget. How- ever, such optimizations are still constrained by the inherent capacity of the draft model. Therefore, the second direction is enhance the capability of the draft model itself, either by modifying its architecture (Li et al., 2024a) or adopting improved training strategies (Zhang et al. 2025 [i et al. 2025). The primary focus of this work is on optimizing the training strategy of the draft model.
For dual-model speculative decoding methods that require training, the limited parameter size of the draft model is a key factor constraining upper-bound performance. For example, EAGLE uses a single decoder layer from the target model as the starting point for training. While increasing the size of the draft model can indeed improve its performance, also leads higher inference costs. Simply adding more layers to the draft model results in trade-off that is often not worthwhile. Therefore, a natural question arises: Is it possible to temporarily expand the parameter size of the draft model during training to achieve better performance, without introducing any additional inference cost?
To achieve this goal, we introduce the concept of structural re-parameterization into the training of draft models for speculative decoding. The original motivation of structural re-parameterization is to alleviate the vanishing gradient problem in simple deep models like VGG by adding multi-branch topologiaal bypasses similar to those in ResNet (He et al. 2013; Ding et al., 2019; 2021). Due to the additivity convolution, these additional branches can be merged with the main backbone during inference and thus incur no extra inference cost. Although single-layer decoder draft models typically do not suffer from vanishing gradients, the principle of decoupling the model structure during training and re-coupling it during inference remains highly relevant and beneficial in our setting.
We propose RepSpec, which replaces each linear module in the draft model with multiple redun- dant linear modules during training, thereby enhancing the training effectiveness. At inference time, these modules can be merged into single linear layer, ensuring no additional inference cost. Un- like previous works, considering the unique characteristics of speculative decoding becoding-namely.. modest increase in the draft model's computational cost can justified it leads to a substantial improvement in accepted sequence length and overall speedup-we go beyond vanilla structural re- parameterization. Specifically, we explore a hybrid linear-nonlinear training strategy, which further improves the performance of the draft model.
namely, that
In summary, the contributions of this work are as follows:
We propose RepSpee, novel training framework for draft model training in speculative decoding, which leverages structural pe-aramtterization to enhance training effectiveness without increasing inference cost.
We go beyond vanilla structural ce-parameterizaation by introducing hybrid linear nonlinear training strategies, resulting in further performance gains in speculative decoding.
Experimental results indicate that, after integrating RepSpec, various methods, includ- ing the statc-of-the-arr EAGLE-3, achieve improved training performance in multiple benchmarks and models. larger target models, employing hybrid structural re- parameterization further yields end-to-end speed gains.
2 PRELIMINARIES
2.1 SPECULATIVE DECODING
Speculative decoding (SD) (Chen et al., 2023 Leviathan al. 2023) accelerates model inference by leveraging the parallelism of attention mechanisms. It involves two stages: (1) a draft phase, where a smaller draft model Ma generates draft tokens, and (2) verification phase, where a larger target model M, verifies these tokens in parallel.
Let 1, denote the i-th token and Ts: {la, (,t1, ..., } subsequence, with Ti., as the prefix. For each 4; let p. and p; be the generation probability from M, and the verification probability from Mr. respectively. Define P..e ( .e. }e and Pa.e {Pe....,PA } as the sequences of these probabilities. Assume the draft model Ma has autoregressively generated k tokens T Tattiatk and
2

---

Published as a conference paper at ICLR 2026
obtained +m- We then imput (t,, T ++ into Mt to obtain P n.n++ (assuming the prefix's KV cache available). For each j = n. ,,..... 72 + k, we greedily accept the token if arg max(p,) matches fj+1, or alternatively, set the acceptance probability to min(1, PPpPP)) If accepted, the pro- cess proceeds to the next token; otherwise,
token is sampled from distribution proportional
to norm (nax(0,pp Pj - -))) and the process repeats. If all candidate tokens are accepted, Pr+tk is used sample 1-
2.2 STRUCTURAL RE- DE-PARAMETTRIZATION
Structural re- parameterization has been widely adopted in convolutional neural networks to decou- ple model architectures between training and inference. Notably, ACNet (Ding et al. 2019) and RepVGG (Ding et al., 202T) leverage this technique to enhance model capacity during training while maintaining inference efficiency.
In RepVGG, each convolutional block during training consists of multiple parallel branches, such as 3 convolution, 1 convolution, and an identity mapping. The overall output the sum of the outputs from all branches: the kernels for the 3 and 1 x convolutions, respectively, denotes the convolution operation, and x is the input. After training, these branches can be merged into single 3 x 3 convolution by appropriately padding and summing the kernels, exploiting the additivity property of convolution Kmarged K3x3 pad(Kix pad(1), where pad(-) denotes zero-padding to match the kernel size, and is the identity kernel.
(Kaxs x)) + (Kix1 x) + 2, where K3x3 and K1x1 are
Through this approach, the network benefits from richer representations and improved optimization during training, while preserving a simple and efficient structure for inference.
3 REPSPEC
To enhance the training of draft models in speculative decoding, we propose RepSpec, a novel draft model training approach that incorporates structural re-parameterization.
3.1 PURE LINEAR METHOD
The most straightforward manner to apply structural re-parameterization draft-model training is to augment every eligible linear block with mergeable redundant branches. For compact Transformer decoder, such blocks include the embedding layer, the projection layers within self-attention, and the linear layers inside the MLP submodule. Empirically, re- parameterizing all linear layers in both attention and MLP components is necessary and sufficient.
Denote the original linear layer by Main. During training, we insert (i) stack of n sequential layers {Pre,] before Main, (ii) a stack m sequential layers {Post,) /m after Main, and (iil) k parallel bypass layers [ The augmented mapping is
Aug Postmat (Main
Σ
Bypass,) Prej:-
1=
Let W., b. be the weight and bias of the corresponding module. With the shorthand
W pre
II p
Opre
Σ
(II
W pre, Oproes
(2)
==n
and analogous expressions for W Wostt Spost, the overall transformation reads
= W post
W
+
W
byypass W
pre
+ W post
Wmain +
Whypass, bpre
+ W post
Imain
Σ,
buypass, bpost-
3

---

Published as a conference paper at ICLR 2026
After training, the entire branch-wise structure is losslessly merged into a single linear layer New:
W Wnew
W Wost
W mainn
+
W Wbypass,
W prees
Unew
post
W Wmain + Σ, Whypass,
Upre
+ W post
Imain +
Σ, bbypass, + bpost.
Because no non-linearity is present, any deeper linear expansion is functionally equivalent to the above compact form; excessive depth only increases training cost without improving expressiveness.
By initializing Bypass to zero and Pre/Post to the identity, we ensure that the augmented model functionally equivalent the original model initialization (Identity Initialization), while still benefiting from the improved optimization landscape introduced by pe-aaaamttrriztionn dur- ing training.
The overall process is illustrated in Figure
Train
Marge/Inte
Main
Bypass,
Poely
Poeta
Prey
Prep
Bypaes,
Poely
Poski
Bypass,
Bypass,
Figure 1: Illustration of the RepSpec method, showing both the training and inference phases. Dur- ing training (left), inputs pass through a sequence of Pre-layers, then either the Main path or multiple Bypass paths, and finally through Post-layers. In the inference phase (right), the weights of the Main and Bypass layers are first summed clement-wise. This summation is then sequentially multiplied by the weights of the Pre and Post layers yield the final equivalent weights. This approach allows the model to leverage the benefits of multiple training paths without incurring additional computa- tional costs during inference.
3.2 HYBRID METHOD
For the pure linear method, the absence of nonlinear activation functions results in clear limita- tion on performance improvement. To address this, we consider introducing a minimal number of nonlinear modules to achieve maximal additional gains, even though these nonlinear components cannot be merged during inference. Vanilla structural re-parameterization has never considered nonlinear structures, standard inference is isolated process where architectural redundancy di- rectly increases overall computational cost. However, in the context of speculative decoding, the draft model's inference time constitutes only a small fraction of the total decoding time. This allows us to tolerate a slight increase in the draft model's 's inference cost, provided it leads to a significant improvement acceptance length and thus yields an overall speedup.
Inserting additional activation functions increases the number of non- mer-merge segments and thus the inference cost. Our experiments show that most of the performance gain is achieved by inserting just one activation function; further insertions yield diminishing returns. Therefore, inserting a single nonlinear module, following the pure linear design, at the Pre, Post, or Bypass po- sitions relative to the Main linear layer. Each nonlinear module is implemented two decomposed linear layers with an activation function in between, using a LoRA-like bottleneck to minimize the number of non-mergeable parameters. Since these modules are still bounded by linear layers, partial merging with other linear components remains possible.
consider
When the activation function is inserted in either the Pre or Post position, the entire linear system divided into two non mergeable linear segments. Suppose the Main layer has input dimension in feature and output dimension out feature, and the segment interrupted by the activation function
4

---

Published as a conference paper at ICLR 2026
has intermediate dimension mid miatuee. The more complex linear segment be merged ac- cording to the procedure described in Section 3.1, ultimately resulting in two linear layers. When the activation function is inserted in Bypass branch, it breaks one linear layer into two linear seg- ments separated by the activation function. These two linear segments can be merged with the Pre and Post layers, respectively. However, in this case, the first linear layer only be concatenated with the Main layer rather than directly summed, resulting in lower inference efficiency compared inserting the activation functiin the Pre or Post positions. Our experiments demonstrate that the optimal hybrid structure is to apply a LoRA-like nonlinearity to the Bypass branch.
4 EXPRIMENT
Models:
We used LLaMA-3.1 8B, LLaMA-2-Chat 13B (Tourron et al. (023) and Vicuna- 1.3 7B (Chiang et al. 2023) target model Mr
Tasks: To compare with EAGLE-2 (Li et al. 2024b), we aligned with it on the dataset. For tasks such multi-round dialogue, code generation, mathematical reasoning, instruction following, sum- marization, and Q&A, we selected the MT-bench (Zheng et al, (023), HumanEval (Chen et al. 2021), , GSM8K (Cobbe et al., 2021), Alpaca (Taori et al. 2023), CNN/Daily Mail (Nallapati al. 2016 and Natural Questions (wiatkowski et al. 2019) respectively.
Metrics: We mainly focus on the following indicators:
Decoding speed et The total number of new tokens divided by wall-clock time.
Acceptance length 7: The average length accepted by M, for each generation. Since the includes the token generated by the target model, which is always accepted, the percentage gain should be calculated by subtracting from T before computing the improvement.
Acceptance rate π-at The acceptance rate at the n-th step of autoregressive inference for Ma each generation. Here, we only consider the case of sequential speculative decoding, meaning that this acceptance rate specifically refers to the probability that, at each step, Ma's 's most probable candidate token is accepted.
Comparison:
We evaluate the improvements of our method when combined with the widely rec-
ognized industry baselines EAGLE-1/3 (Li et al. 2024a b 2025), Medusa (Cai et al. 2024). and Hydra (Ankner et al. 2024).
Environment: All training experiments were conducted on NVIDIA A 100 GPUs (80GB), and all inference experiments were performed on 2 NVIDIA A100 GPUs (80GB).
4.1 EFFECTIVENESS
We first evaluate the inference effectiveness of three standac training (Baseline), pure linear structural re- parameterization (Lincar), and hybrid structural re-parameterization (Hy- brid)-across models of different sizes and various benchmarks. The experiments primarily focus on the widely used EAGLE-I and EAGLE-3 methods, with inference performed in both chain mode and the dynamic tree mode of EAGLE-2. All experiments use a temperature of 0. For both re- parameterization methods, we adopt the optimal architectures identified in our ablation studies (see Section 4.2) The results are summarized in Table
On the LLaMA-3.1 8B model, the performance ranks as Linear > Hybrid > Baseline. The pure lin- ear method improves acceptance length without additional inference cost, while the hybrid method achieves greater gains in acceptance length but is slightly slower due to the inclusion of non- mergeable nonlinear components. On the larger 13B model, Hybrid outperforms the others, as the relative draft time decreases and the increased draft cost is offset by larger gains in acceptance length. Specifically, on LLaMA-3.1 8B, Linear accelerates EAGLE-1 by 70% and EAGLE-3 by 6%%6%% on LLaMA-2 13B, Hybrid accelerates EAGLE-1 by 5%%9%.
Beyond the EAGLE family, we also evaluated Medusa and Hydra on Vicuna-1.3 7B. The Linear method delivered the best acceleration, with speedups of 5% for Medusa and 8% for Hydra.
5

---

Published as a conference paper at ICLR 2026
Table 1: Inference performance of the standard training method (Baseline), the pure linear method (Linear), and the hybrid method (Hybrid) across different models and benchmarks, and tempera- ture=0. Here, denotes the acceptance length, denotes the inference speed (tokens/s), L31 8B refers to LLaMA-3. 1.MMA--..-Inntruue I-Instruct--8B, L2-13B refers to LLaMA-2-Chat-133 V 7B refers to Vicuna-1.3 7B, E1/3 refers to EAGLE-1/3, M refers to Medusa, refers to Hydra, Chain denotes sequential speculative decoding with draft length 5. For EAGLE-1 and EAGLE-3, Tree refers to the dynamic tree method of EAGLE-2. For Medusa and Hydra, Tree indicates the default static tree method. All parameters follow their default configurations.
MT
GSMSK
Alpaca
Human
QA
Sum
Avg
Target
Strategy
Draft
Method
Baseline
2.54
48
2.74
52
2.41
46
2.99
58
2.28
42
2.28
2.54
E1
Linear
2.67
52
2.93
56
2.54
48
3.22
63
2.38
44
2.45
46
2.70
52
Hybrid
2.79
52
3.00
53
2.68
47
3.39
60
2.48
45
2.56
45
2.82
50
Chain
Baseline
3.33
65
3.32
65
3.44
68
3.99
74
2.96
57
3.03
3.35
64
E3
Linear
3.58
68
3.60
69
3.75
74
4.25
78
3.08
59
3.30
61
3.60
68
L31 8B
Hybrid
3.62
65
3.65
66
3.78
70
4.32
75
3.15
55
3.42
59
3.66
6.5
Baseline
3,91
70
4.17
77
3.83
73
4.52
82
3.46
62
3.25
57
3.86
70
H1
Linear
4.16
74
4.42
81
4.11
75
4.82
88
3.65
67
3,46
62
4.10
75
Hybrid
4.34
72
4.62
79
4.28
71
5.05
83
3.87
65
3.68
4.31
Tree
Baseline 5.94
100
5.86
95
6.34
110
6.51
109
5.13
82
5.37
89
5.86
98
E3
Linear
6,00
102
5.95
99
6.48
115
6.60
114
5.25
87
5.41
91
5.95
101
Hybrid
6.08
98
6.01
95
6.61
114
6.65
110
5.32
85
5.55
6.03
Baseline 2.87
51
3.01
56
2.75
51
3.41
62
2.51
48
2.65
46
2.87
52
Chain
E1
Linear
2.97
54
3,09
58
2.84
54
3.56
64
2.60
50
2.73
49
2.97
55
Hybrid
3.09
56
3.20
60
2.96
55
3.78
67
2.73
53
2.93
3.12
1.2 13B
Baseline
4.38
76
4.64
80
4.32
78
5.09
89
3.94
70
3.48
63
4.31
76
Tree
Linear
4.54
79
4.75
81
4.47
80
5.21
92
4.10
72
3.77
4.47
78
Hybrid
4.71
4.95
84
82
93
4.25
75
4.02
4.67
Baseline
2.38
44
2.66
48
2.40
45
2.75
50
2.12
37
2.14
38
2.41
44
M
Linear
2.55
47
2.72
51
2.45
47
2.88
53
2.25
40
2.26
40
2.52
46
Hybrid
2.62
44
2.89
49
43
2.38
37
2.34
2.61
V 7B
Tree
Baseline
3.66
60
3.62
53
3.50
47
3.88
58
2.91
40
2.85
39
3.59
49
Linear
3.89
64
3.92
56
3.86
53
4.03
60
2.96
42
2.94
3.60
Hybrid
4.05
3.99
54
4.02
51
4.12
59
3.02
38
2.99
3.70
4.2 ABLATION STUDY
All experiments in this section conducted using EAGLE-1 LLaMA-3.1 8B and MT-bench, with chain mode inference and a draft length of
4.2.1 PURE LINEAR STRUCTURE
For the pure linear structure, the initial ablation study evaluates which modules in the Transformer decoder benefit most from structural praaamtterization. We insert single Bypass into the embed- ding layer, the q, k, v, o projections in the attention layers, and the gate, up, and down projections in the MLP layers, both individually and in combination, and assess shown in Figure [a the attention and MLP layers are the optimal targets for p-paramtterization. Notably, while re- re-parameterizin the embedding layer provides can degrade out-of-dommin (OOD) performance and, some benchmarks, even underperform the baseline. Inference results reported in Table 2
the validation set. As
evaluation improvement,
After identifying the optimal modules for structural re- parameterization (Attn MLP), we conduct ablation studies different combinations of linear blocks-Pre, Pre, Post, and Bypass Bypass-insertedd inserted before,

---

Published as a conference paper at ICLR 2026
Sandere
941-
3.79
0.75
0.75
9.70
0.70
6.65
0.60
0.55
0.55
0.55
n-d
(a) Module Ablation
(b) Block Ablation
(c) Size Ablation
Figure 2: Ablation studies for the pure linear structure method. (a) Structural re-parameterization
applied different modules in the decoder. (b) Ablation of different types and combinations of
linear blocks at the optimal insertion position from (a). (c) Ablation of different size configurations for structural re-parameterization based on the optimal position and blocks combination from (a) and (b), the numbers indicate the number of augmented blocks; for example, Pre1+Bypass denotes the addition of one Pre and one Bypass block.
Table 2: Comparison of acceptance lengths for ablation study on structural re-paramettrrization of the embedding layer.
MT Bench
GSM8k
Alpaca
Humaneval
QA
Sum
Baseline
2.51 2.55
2.74
2.41
2.99
2.27
2.28
Embedding
2.70
2.47
2.97
2.25
2.30
after, and in parallel with the original linear layer, respectively. As shown in Figure 2b while the combination of all three blocks achieves the best training performance, our benchmarking demon- strates that using only Pre and Bypass yields lower training cost and superior empirical results in benchmarks.
After identifying the optimal module positions and block combinations for structural parameterization, we further conduct ablation studies on the size of the current configuration. Since, theory, the pure linear structure can infinitely expanded, we experiment with increasing the number of Pre and Bypass blocks in the linear layers of both the attention and MLP modules. The experimental results are shown in Figure gle Bypass already constitutes the optimal pure linear structure; adding further redundant structures not only increases the training cost, but may even degrade training performance.
can be observed that using only single and sin-
In summary, our experiments show that the optimal pure linear structural re-parameterization method is to insert one Pre and one Bypass into each of the q, k, v, o projections of the attention layers and the gate, up, and down projections of the MLP layers.
4.2.2 HYBRID STRUCTURE
In the ablation experiments of the pure linear method, we found that unrestricted increases in linear structural complexity do not result in unlimited performance gains. This natu- rally motivates us explore the insertion of nonlinear activation functions to further en- hance model performance. We define insertion as splitting a linear block into two lin- layers separated by activation function. For example, given linear block X Linear ffctuure fetaura, inserting ReLU yields a new nonlinear block X' Sequential Cequential(Lineartt (Lincar(in feature, mid feature), RcLU(), Lincar(mide feattuurei out feature)).
Our first ablation study investigates activation functions inserted into the Bypass block, with mid_feature min(in featuree feature.. While GeLU achieves the best training results (see Figure Ba), benchmark evaluations show that ReLU offers the best speed and simplicity (see Table 3).
7

---

Published as a conference paper at ICLR 2026
Ner
0.80
0.50
0.80
0.75
0.90
0.78
0.70
Receetance
0.70
0.60
0.60
0.60
0.55
0-2
nd
(a) Activation Ablation
(b) Mid-feature Ablation
(c) Structure Ablation
Figure 3: Ablation studies for the hybrid structure method. (a) Ablation of different activation functions, (b) Using the best activation function, ablation of different insertion blocks, and differ- ent intermediate size configurations; for example, Pre+Mid0.5 indicates that the activation function inserted into the Pre block with mid feature tion of different overall structural designs, where Hybrid denotes the optimal hybrid structural parameterization method identified from the previous experiments, and Double Layer denotes sim- ply doubling the number of decoder layers in the EAGLE model.
(iiniinfeeatuee out featuur))))) x 0.5. (c) Abla-
Table 3: Comparison of acceptance rates and inference speeds for different activation functions
MT Bench
GSM8k
Alpaca
Humaneval
QA
Sum
Activation
7
1
T
V
ReLU
2.85
51
3,05
55
2.75
50
3.47
62
2.54
45
2.62
45
LeakyReLU
2.84
51
3.07
59
2.71
48
3.43
2.50
45
2.62
GeLU
2.85
50
3.08
56
2.75
48
3.46
60
2.50
43
2.62
44
SiLU
2.85
50
3.06
55
2.74
48
3.44
2.51
43
2.62
44
The second study examines the effect of inserting ReLU into either the Pre or Bypass block and varying mid feature. As shown in Figure 3b] using ReLU in the Bypass block with mid_feature ing fid_fce slightly lowers the acceptance rate but decreases the inference cost. Setting mid feature = min feature, out feature) x 0.5 achieves the best trade-off, as reported in Table
minttn_eeature, fut_feature) gives optimal training performance. Reduc-
Table 4: Comparison of acceptance rates and inference speeds for different mid feature sizes (with a ReLU inserted the Bypass block).
MT Bench
GSM8k
Alpaca
Humaneval
QA
Sum
Mid Feature
7
7
V
T
V
V
7
V
T
V
min(in, out)
2.85
51
3.05
55
2.75
50
3.47
62
2.54
45
2.62
45
0.5 * min(in, out)
2.78
53
3.00
57
2.64
51
3.35
63
2.46
45
2.55
45
The third ablation study compares our approach other methods that simultaneously increase in- ference cost and acceptance rate. For example, in EAGLE, the most straightforward approach is to double the number of decoder layers. As shown in Figure 3c this is compared with the optimal hybrid structural re- paaameterizzaion method identified in the previous studies. While doubling the decoder layers yields better training results, our empirical evaluations demonstrate that the hybrid structure achieves superior performance with significantly fewer parameters, as reported in Table
4.3 ADDITIONAL OVERHEAD
Although linear structural re- ce-parameterization does not introduce additional overhead during infer- ence, it does incur negligiii cost during training. Specifically, RepSpec expands the compu- tational graph by introducing multiple linear branches (Pre, Bypass) as well optional nonlinear
8

---

Published as a conference paper at ICLR 2026
Table Comparison of acceptance rates and inference speeds for different structures. Hybrid refers to the optimal hybrid structural re re-parameterization method, Base refers to only using one decoder layer as the draft model, Double refers to directly using two decoder layers as the draft model. refers to the number of trainable parameters.
Size
MT Bench
GSM8k
Alpaca
Humaneval
QA
Sum
Struc.
Size
7
v
V
Base 0.24B
2.51
48
2.74
52
2,41
46
2.99
58
2.27
42
2.28
42
Double
0.47B
2.77
49
3.00
54
2.60
48
3.31
60
2.42
42
2.55
43
Hybrid
0.40B
2.79
53
3.00
57
2.68
51
3.39
63
2.48
45
2.56
45
modules, significantly increasing the computational complexity during training. Here, provide a systematic analysis of this cost. For Medusa, Hydra, and EAGLE-1, we use native PyTorch dis- tributed training for simple data parallelism (DP). For EAGLE-3, we employ DeepSpeed's ZeRO-3 optimization for training. We specifically record the training speed and GPU memory usage for the Baseline, Linear, and Hybrid methods across different approaches. The experimental results shown in 6
Table 6: The additional training cost incurred by integrating RepSpec across different methods. All training is conducted in half-precision (FP16). Medusa and Hydra experiments are conducted on Vicuna-1.3 7B, while EAGLE-1 and EAGLE-3 experiments are performed LLaMA-3.1 8B. Medusa, Hydra, and EAGLE-1 use standard Py PoTch distributed training, whereas EAGLE-3 em- ploys DpepSpeed ZcRO-3 optimization. Here, denotes training speed (steps/s), and MEM denotes GPU memory usage during training (GB).
Medusa
Hydra
EAGLE-1
EAGLE-3
V
MEM
V
MEM
V
MEM
V
MEM
Bascline
1.42
50
0.45
60
1.88
28
1.29
38
Linear
1.32
0.34
76
1.57
55
1.09
72
Hybrid
1.29
54
0.32
73
1.54
53
1.06
70
In addition the inevitable training overhead, the Hybrid method also introduces a small amount of forward pass latency during inference. We conducted comparative experiments assess this aspect, primarily recording GPU memory usage and the latency each forward pass during inference. The forward pass latency is measured as the average latency over 1000 forward passes under the default inference configuration. The experimental results are shown in Table
Table 7: The additional inference overhead introduced by integrating the Hybrid method across different methods. All experiments conducted in half-precision (FP16). Medusa and Hydra experiments are conducted on Vicuna-1.3 7B, while EAGLE-1 and EAGLE-3 experiments are per- formed on LLaMA-3.1 8B, FPL (forward pass latency) denotes the average time per forward pass during inference (ms), and MEM represents GPU memory usage during inference (GB). Hybrid(n) denotes the Hybrid method with feature n x mintin.feature, out feature..
Medusa
Hydra
EAGLE-1
EAGLE-3
FPL
MEM
FPL
MEM
FPL
MEM
FPL
MEM
Baseline
1.38
16.4
2.04
21.5
0.87
6.0
0.95
6.4
(0.5)
1.41
16.9
2.45
24.4
1.24
8.3
1.37
8.8
H1brid(1)
1.44
17.2
2.61
26.8
1.57
9.0
1.64
9.8
The detailed configurations and re-parameterizatiin designs be found in Appendix A

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We propose RepSpec, a training-time structural re-paaameterization framework designed to enhance draft models for speculative decoding. By inserting mergeable linear branches during training, Rep- Spec increases model capacity without incurring any inference-time overhead; its hybrid variant offers further performance gains with only minimal additional cost. Extensive experiments across various trainable speculative decoding methods, target models, and datasets demonstrate that Rep- Spec consistently improves both acceptance length and end-to-end speed. RepSpec plug-and- play at training time (requiring only modifications to underlying substructures without altering the overall model design) and can effectively improve the training performance of draft models con- strained by parameter count. This flexibility also suggests broad potential for RepSpec's application in other scenarios in the future (including MTP, a pre-training approach based on speculative de- coding principles). Our experiments the 13B model also provide preliminary evidence that the Hybrid method offers greater potential on further larger target models.
ACKNOWLEDGEMENTS
This work was supported by the Natural Science Foundation of China(62372013),, the National Key Research and Development Program of Chin3(3224FCC3377901) and Meituan.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint Srdi::2303.0877772 2023.
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Sequentially-depende draft heads for medusa decoding. 2024.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterizatton. ICML, 2018.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chenggiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. /6609, 2023.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalizeed normalized margin bounds for neural networks. In NIPS, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 10774, 2024.
Charlie Chen, Sehastian Borgeaud, Geoffrey Irving, Lean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint vrXiv:2107.03374, 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.142233 2023.
Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the ker- nel skeletons for powerful enn via asymmetric convolution blocks. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vtg-style convnets great again. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 13733-13742, 2021.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm infer- ence using lookahead decoding. arXiv preprint 02255 2024.
11

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog nition. arXiv preprint 151.:315 2015.
Yuxuan Hu, Ke Wang, Xiaokang Zhang, Fanjin Zhang, Cuiping Li, Hong Chen, and Jing Zhang. Sam decoding: Speculative decoding via suffix automaton, 2024. URL https://arxs
org/abs 2411 10666
Huan Li Lidan Shou Ke Chen Gang Chen Sharad Mehrotra Jun Zhang, Jue Wang. Draft verify: Lossless large language model acceleration via self-speculative decoding. 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Yaniv Leviathan, Matan Kalman. and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE: Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of lan- guage models with dynamic draft trees. Empirical Methods in Natural Language Processing, 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3: Scaling up inference
acceleration of large language models via training-time test, 2025. URL https://arxiv. orq/abs/ 2503. 01840
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, al. Specinfer: Accelerating large lan- guage model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, pp. 932-949. 2024.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequeeee rmns and beyond. arXiv preprint arXiv:1662.06023, 2016.
David A Patterson. Latency lags bandwith. Communications of the ACM, 47(10):71 2004.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. EU- SIPCO, 2007.
Apoorv Saxena. Prompt lookup decoding, November 2023, URL https:///gtthbt com/
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint
arXiv:1911.02150, 2019.
Chao Wang Fan Yin Shuai Shi Yubo Wang Yi Zhang Yingyi Huang Haoshuai Zheng Yineng Zhang Shenggui Li, Yikai Zhu. Specforge: Train speculative decoding models effortlessly. https: 2025.
//g dlgithub.ccm com www.c/oooo/locce wtlppppojjtt/sppefooooooo
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instraction-followinn llaaa model, 2023.
12

---

Published as a conference paper at ICLR 2026
Meituan LongCat Team, Bei Li, Bingye Lei, Bo Wang, Bolin Rong, Chao Wang. Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, et al. Longcat-fian technical report. arXiv preprint arXiv:2509.01332, 2025.
Qwen Team. Qwen3 technical report, 2025. URL U.L.:ttss//aaxii.ooor//////22203
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 7655-7671, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10. 1/22///////2222. findings-acl.455 acl.s URL https: attpss///ccaaatholooogoo 224 finiingsiiininnnnineoanno.. 456
Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu. Learning harmonized represetttttss for speculative sampling. In International Conference on Learning Representations, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and 2023.
chatbot arena. Advances in Neural Information Processing Systems,
13

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A.1 TRAINING
All training experiments were conducted 8 NVIDIA A100 GPUs (80GB).
During the hyperparameter search, we found that when integrating RepSpec (1 x Bypass + x Pre), doubling the learning rate compared the baseline typically yields better results. The optimizer and scheduler settings also follow the default configurations. The key training parameters are as follows:
Medusa: learning rate x 10-3, batch size 8, epochs =
Hydra: learning rate 1 x 10-3, batch size 4, epochs 10
EAGLE-1: learning rate 6 x 10 batch size 4, epochs = 20
EAGLE-3: learning rate 1 x 10-1, batch size epochs
Medusa and Hydra also involve the design of the draft model head. We follow the officially released scripts and use the following parameters:
Medusa: medusa num heads = 3, medusa num layers
Hydra: hydra
hydra.num.layers = 4
In the Medusa method, there are medusa.num_heaad Medusa Heads, each consisting of a Se- quential block composed of medusa.nnm.llaers ResBlocks (square Linear layers with residual connections) and one LM head (a rectangular Linear layer). When integrating RepSpec, we add Pre and Bypass module both the Linear layers in the ResBlocks and the LM head.
Hydra is similar to Medusa, with hydra hydra-num _um heads Hydra Heads, each primarily composed of hydra-_um.layers ResBlocks and one LM head. The main difference is that each Hydra Head is preceded by LLaMA2 Decoder, which serves as the embedding layer. When integrating RepSpec, we apply the same structural re-parameterization as in Medusa to the Linear layers ResBlocks and the LM head. In addition, due to GPU memory constraints, apply structural re-parameterization only to the Q, K, V, and O projections in the Attention module of the Decoder.
For EAGLE-1/3 1/3, shown in Section 4.2 we perform structural re-parameterization on the Q. K. V. O projections the Attention module and the gate, up, and down projections in the MLP. Specifically. EAGLE-1 introduces only the Pre and Bypass modules. In contrast, due to the multi-step supervision (default 7) forward passes. In this case, simple structural re-parameterization can cause output explosion in the later stages of training. To stabilize training, we additionally introduce a Res skip connection module. That is, when in in_features out fua_uur direct residual skip connection is adopted; otherwise, linear block with the shape of fin-fcature, out_fcature) is added, i.c.,
EAGLE-3, each training step requires the draft model to perform n
Aug = Postme: (Main + Σ Bypass,) Prem. + Res. Regarding training data, Medusa, Hydra, and EAGLE- are trained on ShareGPT, while EAGLE-3 trained on both ShareGPT and UltraChat-200k (Ding et al. 2023), with the latter's responses generated by the target model.
-
The training process of EAGLE- can seen in Figure Considering only training performance, Hybrid Linear > Baseline.
A.2 INFERENCE
All inference experiments were performed on 2 NVIDIA A100 GPUs (80GB).
During inference experiments, follow the official implementations of each method. For Medusa and Hydra, we only perform static tree sampling, with the tree structure illustrated Figure of the Medusa paper.
For EAGLE-1/3, we evaluate RepSpec's performance from two perspectives: In the Chain mode, which is more commonly used in industry, set the draft length to 5. In the Tree mode, the dynamic tree sampling of EAGLE-2, following the official implementation:
EAGLE-1: depth 6, topk 10, total token 60
14

---

Published as a conference paper at ICLR 2026
0.5
08
0.78
0.72
0.2
0.88
0.22
0.54
12
10
18
20
18
(a) EAGLEI Stepl Epoch-Ace
(b) EAGLEI Step2 Epoch-Ace
(c) EAGLEI Step3 Epoch-Ace
0.86
0.666
0833
0.6
0.68
0820
0.600
0.70
0.646
0.5
10
(d) EAGLE3 Step1 Epoch-Ace
(c) EAGLE3 Step2 Epoch-Acc
(f) EAGLE3 Step3 Epoch-Ace
Figure 4: The Epoch-Ace curves for the n-th step during training of EAGLE-1/3 1/3 with RepSpec on LLaMA-3. I-Instruct 8B, where the vertical axis represents Ace (accuracy) and the horizontal axis represents Epoch. Here, Acc refers to cumulative accuracy, meaning the cumulative Ace at each step the product of the independent Acc values from all previous steps. Green denotes the EAGLE-1 baseline, orange denotes the EAGLE-3 baseline, blue represents the Linear method, and red represents the Hybrid method combined with RepSpec.
EAGLE-3: depth = 8, topk 10, total.token 60
Our evaluation code is built upon SpecBench (Xia et al., 2024).
B INITIALIZATION STRATEGY
To guarantee training stability and functional equivalence after merging, we initialize the re- parameterized structures as follows:
Main path (original linear layer): loaded from the pre-trained checkpoint when available; otherwise Xavier uniform.
Bypass branches:
W Waypass = 0, Dyypass = so initial forward behaviour equals the main
path alone.
Pre/Post sequential layers: identity matrices for weights, zeros for biases, yielding an initial identity transformation.
Hybrid nonlinear branch (LoRA-style): first matrix sampled from (0, 0.022), second matrix set to zero, giving zero initial output.
This method is referred
Consequently, the augmented network starts identically to the original model, avoids early gradient explosion, and produces numerically stable merged parameters after structural re-aaemtttrizanoon.
Identity Initialization.
To validate the effectiveness of this initialization strategy, we set up two commonly used alternative initialization strategies for comparison.
Random Initialization: both the Pre and Bypass modules initialized with Py PoTch'ss default random initialization.
Residual Scaling Initialization: each re- paraameterrized branch, including the Pre se-
quence, is initialized with scaled Gaussian weights whose variance is divided by the total
15

---

Published as a conference paper at ICLR 2026
number of mergeable branches N, ensuring that the merged transformation preserves the same activation variance as the original layer.
The experimental results are shown in Figure which indicates that identity initialization > residual scaling initialization random initialization in terms of performance.
025
0.40
0.2
0.34
0.20
011
(a) EAGLEI Stepl Epoch-Ace
(b) EAGLEI Step2 Epoch-Acc
(c) EAGLEI Step3 Epoch-Acc
Figure 5: The Epoch-Ace curves for the n-th step during training of EAGLE-1 with RepSpec on LLaMA-3.I-Instrute 8B, where the vertical axis represents Acc (accuracy) and the horizontal axis represents Epoch. Here, Acc refers to cumulative accuracy, meaning the cumulative Acc at each step the product of the independent Ace values from all previous steps. The blue curve represents identity initialization, the green curve represents residual scaling initialization, and the red curve represents random initialization.
C WARM START
We also evaluated the effectiveness of RepSpec in the warm-start scenario by using checkpoint from the EAGLE-1 baseline trained for 20 epochs as the starting point. Since RepSpec introduces new parameters, the optimizer state is reset in our experiments. During warm-start training. RepSpec still follows the identity initialization. The warm-start training process is shown in Figure 6 it can be observed that, although resetting the optimizer Ieads to fluctuations in the early stages of training, RepSpec ultimately outperforms continued training of the baseline once training stabilizes.
0.45
022
0.400
0.888
0.27
0.440
0.435
0.420
0.24
10
(a) EAGLEI Stepl Epoch-Ace
(b) EAGLEI Step2 Epoch-Acc
(c) EAGLEI Step3 Epoch-Ace
Figure 6: The Epoch-Ace curves for the n-th step during training of EAGLE-1 with RepSpec on LLLMA-3. Innstruct 8B, where the vertical axis represents Ace (accuracy) and the horizontal axis represents Epoch. Here, Acc refers to cumulative accuracy, meaning the cumulative Acc at each step the product the independent Ace values from all previous steps. The orange curve represents RepSpec with warm-start initialization, while the red curve represents continued training of the baseline.
D SAMPLING RESULTS
Due to space limitations in the main text, we only reported the effectiveness results for temperature = 0 in the experimental section. Here, we provide supplementary speculative sampling results for temperature 1. Since Medusa and Hydra are not lossless when temperature 1, exclude these methods and conduct experiments solely on the EAGLE family. The experimental results are shown in Table X
16

---

Published as a conference paper at ICLR 2026
Table 8: Inference performance of the standard training method (Baseline), the pure linear method (Linear), and the hybrid method (Hybrid) across different models and benchmarks, and tempera- ture=1. Here, denotes the acceptance length, denotes the inference speed (tokens/s), L31 8B refers to JLaMA-3.I-Instruucte 1/3, and Chain denotes sequential speculative decoding with draft length 5. For EAGLE-I and EAGLE-3,, Tree refers to the dynamic tree method of EAGLE-2. All parameters follow their default configurations.
L2-13B refers LLaMA-2-Cha-113B E1/3 refers EAGLE-
MT
GSM8k
Alpaca
Human
QA
Sum
Avg
Target
Strategy
Draft
Method
T
T
Baseline
2.52
48
2.72
51
2.40
46
2.98
56
2.23
40
2.26
41
2.52
47
E1
Linear
2.65
50
2.88
54
2.51
50
3.19
60
2.30
43
2.43
46
2.66
51
Hybrid
2.76
2,99
2.68
3.38
2.45
2.57
43
2.81
49
Chain
Bascline
3.32
61
3.34
65
3,40
61
3.73
72
2.91
52
3.03
58
3.29
61
Linear
3.61
65
3.52
70
3.77
67
4.13
76
3.11
59
3.27
61
3.57
66
1,31
Hybrid
3.65
63
3.58
66
3.82
63
4.27
75
3.15
55
3.33
58
3.63
63
Baseline
3.50 60
3,84
65
3.60
64
4.27
74
2.94
51
3.23
56
3.56
61
Linear
3.71
4.03
3.73
4.54
3.12
3.42
3.76
65
Hybrid
3.90
65
4,20
67
3.92
67
4.78
78
3.28
51
3.67
60
3.96
64
Tree
Bascline
4.44
4.56
77
4.95
80
5.27
91
3.70
60
4.22
70
4.52
76
E3
Linear
4.50
80
4.85
83
5.03
84
5.50
96
3.75
63
4.45
75
4.68
80
Hybrid
4.54
4.91
5.18
5.56
3.73
4.46
4.73
77
Baseline
2.87
51
2.98
53
2.76
51
3,43
63
2.53
45
2.64
45
2.87
51
Chain
E1
Linear
2.94
53
3.07
56
2.83
53
3.53
65
2.63
47
2.74
50
2.96
54
Hybrid
3.07
52
3.19
57
3.00
55
3.77
67
2.72
47
2.92
49
3.11
55
1.2 13B
Bascline
4.21
71
4.65
75
4.22
71
4.98
85
3.94
64
3.45
61
4.24
71
Tree
E1
Linear
4.39
75
4.69
76
4.40
76
5.09
87
4.00
66
3.76
65
4.39
74
Hybrid
4.57
76
4.70
78
4.59
78
5.28
89
4.15
68
4.00
67
4.55
76
E RESULTS ON LARGER MODELS
The following experiments were conducted on SpecForge (Shenggui Li 2025 ) with all training and inference parameters following the default configuration. The results of EAGLE-3 combined with RepSpec on I.L.MA3.3-700 3..3-70BB are shown in Table
Table 9: Inference performance of the standard training method (Baseline), the pure linear method (Linear), and the hybrid method (Hybrid) across different models and benchmarks, and tempera- ture=0. Here, denotes the acceptance length, denotes the inference speed (tokens/s), L33 70B refers to 1..MA--.3-Instruuee of EAGLE-2. All parameters follow their default configurations.
E3 refers to EAGLE-3. Tree refers to the dynamic tree method
MT
GSMSk
Human
Avg
Target
Strategy
Draft
Method
Baseline
5.42
46
6.14
55
6.32
56
5.96
52
L33 70B
Tree
E3
Linear
5.71
49
6.40
58
6.57
58
6.23
55
Hybrid
5.96
6.56
59
6.69
58
6.41
56
17

---

Published as a conference paper at ICLR 2026
F WHY PURE-LINEAR RE-PARAMETERIZATION HELPS
F.1 GRADIENT- VARIANCE REDUCTION: BYPASS
Let the original linear layer be y Wx b. During training we introduce Bypass branch: = (Wn + W2) + (bu + b1), where W1,b, are initialized to zero. After training we merge Wer Wo W1, beif by + bj. Both branches share the same upstream gradient g; hence
g = 9,
g (1) g.
Adam maintains separate moments for each branch:
m = 31m (1) + (1 Br)g.
u = B20, + (1 B2)g²²
The update for branch is
m
AW,
+
Since ( starts from zero, the Bypass branch receives larger adaptive learning rates early training. acting as high- high-resolution refinement of the main branch.
The total update is
(I)
m
m
AW total AWo AW, -2a
U/
(0)
+
VeL0
+
The term in parentheses is the average of two independennty-rreoondiiooi gradients; its variance is half that of a single branch:
(0)
(1)
m
mi
Var
2
adam
(0)
Uz
Thus the effective gradient noise is reduced, leading to stabler updates, larger learning rate and faster
convergence.
F.2 IMPLICIT SPECTRAL REGULARIZATION: PRE/POST
Consider adding only Pre and Post factors:
y = W₂ Wi Wo x, with WoERex WiER"xP, W2ERexr.
Under the balancing effect of gradient flow (Arora al. 2018 the singular values of the three factors converge to
(W)) (W1) (W2) [w)
where War W.WWW minimisation the log- spectral gaps
W. Wa the end-to-end matrix. This balancing behaviour is driven by implicit
1ogoo log o, (Wo) + Σ, llog σ, (W2)-logo
log (W:))
which gradient descent (and Adam) monotonically decrease. As result, the singular values of Waff satisfy
a, (Watt) o ay [o,(We))) 1/3 [W,)W) 1/3, [a,(WW+)))
This implicit regularization penalizes large singular values without any explicit penalty term, and produces a sharper spectral decay for Wat- A faster decay directly reduces the effective rank (Roy & Vetterli 2007). And smaller effective rank leads to tighter generalization bounds (Bartlett al.) 2017). Hence, our proposed Pre Post factorization method implicitly shrinks large singular values, accelerates spectral decay, and achieves tighter generalization bounds.
18

---

Published as a conference paper at ICLR 2026
F.3 SUMMARY
Bypass branch averages two ance and speeding up optimization.
Adam gradients, reducing vari-
2. Pre/Post factorization implicitly balances singular values, encourages low-rank solutions, and enhances generalization.
3. All branches are merged into a single matrix after training, incurring zero additional infer- ence cost.
G THE USE OF LARGE LANGUAGE MODELS
In the preparation of this paper, we utilized large language models (LLMs) to assist in the following aspects:
Language Polishing: We used LLMs (e.g., GPT-4) to improve the clarity, grammar, and
fluency of certain sections of the manuscript. The scientific content, technical contributions, and experimental results were entirely conceived, designed, and verified by the authors.
LaTeX Formatting Assistance: LLMs were used help generate LaTeX code for bles, figures, and appendix formatting. All content, structure, and data were provided and verified by the authors.
We emphasize that no novel scientific content or claims were generated by LLMs. All experi- ments, analyses, and conclusions were independently conducted and verified by the authors. The use of LLMs was limited to auxiliary tasks to improve presentation efficiency and clarity.
This statement is included in accordance with the transparency guidelines of the conference/journal, and to acknowledge the auxiliary role of LLMs in the writing process.
19


Reviews:
[{"review_id": "E4hqJygoIx", "reviewer": "Reviewer_cX35", "summary": "This paper introduces RepSpec, a training framework to fix the key bottleneck in speculative decoding: the draft model's weak capacity. The core idea is structural re-parameterization. During training, authors expand the draft model by adding mergeable linear structures (like Pre and Bypass layers) to boost its capacity. At inference time, these structures are merged back into the original layers, resulting in zero additional inference cost. This ``train-large, infer-small\" method improves the draft model's effectiveness, leading to better acceptance lengths and overall speedup. A Hybrid version also adds non-linear blocks for a minimal cost, which pays off on larger target models.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of merging the linear part of the model architecture during inference is novel and motivated. \n\n- The results shows speedups compared to the SOTA EAGLE. For example, there is the 7.3% improvement over EAGLE1 on LLaMA-3.1 8B (Table 1, T=0). The ablation studies are comprehensive.", "weaknesses": "- Pure linear method gives limited gains. Though the bypass path may make the training more effective, there is a performance ceiling for that. The ceiling can be related to the model size. The paper's own results show that while this method works well on the 8B model, the \"Hybrid\" method outperforms it on the larger 13B model. This implies that the zero-cost benefit comes with a performance ceiling that the authors themselves had to address with the costlier hybrid variant.\n\n- The benefits are not as simple as adding more layers. Will adding too much linear re-parameterization can actually degrade training performance? This brings up the question that whether the re-parameterization structure is a sensitive hyperparameter that must be carefully tuned, rather than a simple, scalable fix.\n\n- The training overhead is also a concern. The required training GPU memory increases and the training speed is also reduced.\n\n- How about larger models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eVFdfA6F0E", "reviewer": "Reviewer_Vyrc", "summary": "This article proposes RepSpec, a method of training draft models in speculative decoding using structural re-parameterization. This method enhances the draft model's capacity during training without increasing its inference cost. The core idea is to augment the draft model's linear layers with redundant, mergeable branches (Pre, Post, Bypass) during training, which are then fused into a single layer for inference. Furthermore, the authors introduce a hybrid method that incorporates a minimal, non-mergeable nonlinear component, justified by the fact that the draft model's inference time is a small fraction of the total SD latency. Experiments on various SD methods (EAGLE, Medusa, Hydra) and LLMs (LLaMA-3.1-8B, LLaMA-2-13B, Vicuna-7B) demonstrate that RepSpec consistently improves the accepted sequence length and end-to-end decoding speed.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The article applies the structural re-parameterization techniques previously used in convolutional neural networks to the training of the draft model for speculative decoding, perfectly adapting to the characteristics of the draft model that are insensitive to training costs and sensitive to inference costs.\n2. The experimental results fully demonstrate the effectiveness of its method, including end-to-end acceleration and draft acceptance rate.\n3. This method has certain value in practical applications.", "weaknesses": "1. Limited Theoretical Insight: The paper provides a solid empirical foundation but offers limited theoretical analysis of why the re-parameterization helps in this specific context (beyond general optimization benefits). The discussion in Appendix E is a good start but could be more integrated.\n2. Training cost (minor): Although the draft model is not very sensitive to training costs (as mentioned above), it still presents certain challenges in resource limited scenarios, especially when the base model is large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aBiljL0mCA", "reviewer": "Reviewer_XtQb", "summary": "This paper proposes a structured re-parameterization method, REPSpec, which enhances the training of draft models by introducing additional layers during training and merging them during inference. Extensive experimental results demonstrate that the proposed method significantly improves the performance of existing approaches.", "questions": "1. Although Appendix A provides some implementation details, the specific placement and strategy for incorporating nonlinear factors remain somewhat unclear.\n\n2. If my understanding is correct, is there a fundamental difference between introducing unmergeable nonlinear factors and directly increasing the size of the draft model?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The application of structured re-parameterization to the field of speculative decoding aligns well with the specific requirements of draft models.\n\n2. Extensive experiments have been conducted to explore the effectiveness of different architectural designs.", "weaknesses": "1. The proposed method introduces a significant increase in training overhead.\n\n2. Although it improves the acceptance rate, the hybrid approach also incurs additional inference costs, resulting in limited overall end-to-end acceleration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YxxxZxfl7k", "reviewer": "Reviewer_15fy", "summary": "The paper adapts structural re-parameterization to draft model training for speculative decoding. While the idea is technically sound, the work suffers from marginal gains, limited novelty, and insufficient practical impact. Below are the key criticisms supporting rejection.", "questions": "The paper focuses on comparing with EAGLE, Medusa, and Hydra, but how does RepSpec perform against other draft model optimization strategies such as knowledge distillation or dynamic architecture methods? Are there scenarios where training-free approaches (e.g., self-speculative decoding) might be more practical despite potentially shorter acceptance lengths?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1 The paper presents a well-motivated adaptation of structural re-parameterization techniques to the emerging domain of speculative decoding. While this technique has been widely used in convolutional networks, its application to draft model training in autoregressive decoding is timely and relevant. \n\n2 The method effectively decouples training-time complexity from inference-time efficiency, maintaining the lightweight nature of draft models while enhancing their capacity during training.t", "weaknesses": "1 The work directly adapts structural re-parameterization—a well-established technique in convolutional networks—to draft model training without conceptual innovation. The hybrid variant introduces non-mergeable nonlinear components but fails to justify the increased inference costs, contradicting the low-latency objective of speculative decoding.\n\n2 Performance improvements are modest: pure linear re-parameterization improves acceptance length by only 7–10% on LLaMA-8B, while the hybrid method incurs additional latency.\n\n3 Experiments are confined to small models (≤13B) and academic benchmarks. The paper lacks comparisons to larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)