SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism
Authors: Yuhao Shen
Junyi Shen
Quan Kong
Tianyu Liu
Yao Lu
Cong Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Speculative Decoding
LLM inference
Primary area: generative models

Abstract: Speculative decoding (SD) has emerged as a promising technique to accelerate LLM inference by employing a small, efficient draft model to propose draft tokens in advance, and subsequently validating them in parallel with the large target model. However, the existing SD methods still remain fundamentally constrained by their serialized execution, which inevitably causes mutual waiting bubbles between the draft and target models. To address this critical challenge, we draw inspiration from sophisticated branch prediction mechanisms in modern processors and propose a novel framework, \textbf{SpecBranch}, to fully unlock branch parallelism in SD. Specifically, we first conduct an in-depth analysis of the potential of branch parallelism in SD, and recognize that the key challenge lies in the intricate trade-offs between parallelization and token rollback. Based on this analysis, we introduce parallel speculative branches to preemptively hedge against likely rejections. Meanwhile, to significantly enhance parallelism, we jointly orchestrate adaptive draft lengths with a hybrid combination of the implicit draft model confidence and explicit reusing of target model features. Extensive experiments conducted across various models and benchmarks show that \textbf{SpecBranch} achieves impressive speedups of over \textbf{1.8}$\times \sim$ \textbf{4.5}$\times$ against the standard auto-regressive decoding and reduces rollback tokens by \textbf{50}\% for poorly aligned models, while maintaining an identical sampling distribution. Our code is available at \url{https://github.com/Sylvan820/Specbranch}.

Published as a conference paper at ICLR 2026
SPECBRANCH: SPECULATIVE DECODING VIA HYBRID DRAFTING AND COLLBACK-AWARD BRANCH PARAL- LELISM
Yuhao Shen Junyi Shen
Quan Kong*
Tianyu Liu
Yao Lu
Cong Wanga!
1 ZZejjiang University
NNational University of Singapore
Uniiversity Science and Technology of China
(riven, quankong, cwang85| & ju. edu. cn (jishen, Luyao) @comp nus edu. sg tianyu liu@mail ustu. edu.en
ABSTRACT
Speculative decoding (SD) has emerged as promising technique to accelerate LLM inference by employing a small, efficient draft model to propose draft kens in advance, and subsequently validating them in parallel with the large target model. However, the existing SD methods still remain fundamentally constrained by their serialized execution, which inevitably causes mutual waiting bubbles between the draft and target models. To address this critical challenge, we draw inspiration from sophisticated branch prediction mechanisms in modern proces- sors and propose novel framework, SpecBranch, to fully unlock branch paral- lelism in SD. Specifically, of branch parallelism in SD, and recognize that the key challenge lies in the intricate trade-off's between parallelization and token rollback. Based on this anal- ysis, we introduce parallel speculative branches to preemptively hedge against likely rejections. Meanwhile, significantly enhance parallelism, we jointly or- chestrate adaptive draft lengths with a hybrid combination of the implicit draft model confidence and explicit reusing target model features. Extensive experi- ments conducted across various models and benchmarks show that SpecBranch achieves impressive speedups of over 1.8x 4.5x against the standard auto- regressive decoding and reduces rollback tokens by 50% for poorly aligned models, while maintaining an identical sampling distribution. Our code is available at https:: /ttt:://ghhhhie com/ Sy1van820 Specbranch.
first conduct an in-depth analysis of the potential
INTRODUCTION
Recent advances in Large Language Models (LLMs) have revolutionized natural language processing (Achiam al., 2023; Team et al., 2023; Bai et al., 2023; Guo et al., 2025). However, their real- world deployment faces critical challenges inference latency due to auto-regressive token-by-toker generation, which restricts LLMs predicting token a time, creating fundamental bottleneck for real-time and large-scale applications.
To address this limitation, Speculative Decoding (SD) has emerged promising acceleration paradigm (Leviathan et al., 2023; Chen et al., 2023; Li et al., 2024a). SD uses a small draft model to proactively generate candidate tokens, which are then verified in parallel by the large target model, By replacing serialized token generation with parallel validation, SD decouples the computational workload from sequence length. However, a critical serial bottleneck still remains. As shown in Fig. 1(a), the draft and target models operate in strict alternation: the draft model idles during target model verification, but the target model cannot process new candidates until the draft model completes
"qqual contribution.
The Corresponding Authors.

---

Published as a conference paper at ICLR 2026
Speculative Decoding
(be distribttion accepted tokens iteration
X
X
Target model
X4
Normalized Distribution
0.40
Reject
Nocost (emy)
X
X
X3
X
X5
Idling
X5
X6
X,
Xa
Xx
0.30
Emrircial Distibution
Theoratioa Diatibution
Parallel Decoding (PEARL)
X
X
Target model
X1
X2
X3
X4
X5
X
X
Xia
Xg
K10
xgg
X6
X
Xg
Rollbasid
0.00
Parallel Decoding with Rollback Awareness (SpecBranch)
0
Accepted tokens per iteration
6
Unconfidenti Need more branches!
(o|The impact of Rollback on existing methods
Target model
x4
All tokies are correct!
300k
Total Tokens
Hoslback
83.2%
Drafing Tokens
250k
200k
78.3%
X2
Shortaned
66.3%
41.9%
100k
Waiting
Hybrid Drafting
Rollback
New
Draft
Target Model
50k
Bubbles
Tokens
Branch
Model
SD
Lookahead
AdaEDL Melhods
PEARL
SpecBranch
(a) Comparison of existing methods
Figure 1: Architectural comparison and empirical analysis of SD frameworks: a) Vanilla SD, Parallel SD (PEARL), and Parallel SD with Rollback Awareness (SpecBranch). When rejection occurs token 24,, PEARL's static pipeline forces verification those "doomed tokens" It Fin: SpecBranch dynamically terminates invalid branches and spawns new branches. b) Distribution accepted tokens generally follows a truncated geometric distribution with longer token length (Vicuna 68M& 13B, y 8); c) Percentage of rollback tokens under different mechanisms.
its proposal. This mutual dependency leads to pipeline bubbles (Narayanan et al., 2019) that neither model fully saturates the hardware resources.
Inspired by branch prediction in modern processors (Jiménez & Lin, 2001; Shi et al., 2019), we allow the draft model to proactively generate speculative branches concurrently with target model verification. Such parallel SD paradigm creates a two-stage pipeline in which the draft model token generation overlaps with the target model validation, effectively filling the inherent pipeline bubbles in vanilla SD. Prior works such as PEARL (Liu et al., 2024b) use the target model to verify the first draft token during the drafting phase (pre-verify), and use the draft model to continue generating draft tokens during the verification phase (post-verify). However, unlike lockstep exccution from the existing SD that discards tokens only with a local penalty (called "rollback tokens" henceforth), Parallel SD risks global invalidation if token is rejected which causes all subsequent tokens to be rejected, and in turn, stall parallel pipelining. This is exacerbated with longer draft sequence length since accepted tokens typically follow truncated geometric distribution as shown in Fig. ((b). This creates an important oafe-off between parallelism and rollback.
Unfortunately, PEARL inadequately addresses these challenges: 1) Pre-verify Rollback. PEARL overlooks a critical condition for parallel accelcration: the tokens during the verification need to be cce-tcc: otherwise, PEARL degenerates to serialized execution and loses its parallel capacity. It verifies only the first token by the target model, while the system remains oblivious to mid-sequence rollback until the parallel verification completes (e-g., when x4 is rejected in Fig. 1(b).. 2) Post- verify Rollback. The static draft length lacks sufficient awareness of rollback and rejected tokens, which also undermine the benefits of parallelism (shown in Fig. I(c) with high percentage of rollback). Consequently, it leads to redundant computation of those "doomed tokens" and makes the target model a bottleneck for processing unnecessary tokens from invalidated branches, despite inevitable rollbacks. This is exacerbated in resource-conssrained systems due to misalignment between parameter-imbalancee draft and target models (68M draft& 13B target).
Although recent dynamic drafting methods, categorized as implicit (confidence-driven early stop- ping (Li et al., 2024b; Liu et al., 2024a; Agrawal et al., 2024; Zhang et al., 2023)) or explicit (feature-based sequence modeling (Zhang et al., 2024)) partially mitigate the rollbacks, they face practical challenges from per-task threshold tuning, error compounding and low prediction accuracy. To this end, we propose SpecBranch with the following contributions:
2

---

Published as a conference paper at ICLR 2026
Methods
Parallel Drafting
Model-Trainee Model-Training-fee
Draft Structure Modeling
Speedup
Kangaroo (l.iu et al., 2024a)
Implicit (Confidence)
EAGLEE-2 (L al., 2024h)
Implicit (Confidence)
AdaEAGLE (Zhang et al., 2024)
Explicit (Feature)
Lookabead Decoding (Fu et al. 2024)
None
111 18x
AdaEDL (Agrawal et al., 2024)
Implicit (Entropy)
14xx3.0x
PEARL (Liu al,, 2024b)
None (Chunk-level))
16xx442x
SpecBranch (Ours)
Hybrid (Token-level)
1.8xx
Table 1: Comparison of SpecBranch with the existing SD methods. SpecBranch is the first parallel framework with hybrid drafting structures that does not require additional training of draft models.
+ Branch-Parallel Architecture: We first establish theoretical models to quantify ideal parallel speculation and extend it to consider rollback penalties in practice. Guided by these insights, we propose novel branch resampling mechanism that introduces parallel speculative branches to preemptively hedge against likely rejections, while preserving the original sample distribution.
Based on extensive empirical analysis of adaptive draft structures, are the first to unify the implicit (draft model confidence) and explicit (target model feature) methods into a hybrid framework that dynamically optimizes draft lengths. This effectively reduces the percentage of rollback and improves parallel efficiency.
Hybrid Adaptive Drafting:
Extensive Evaluation and Discussion: We conduct extensive experiments across various models and tasks, demonstrating that SpecBranch consistently achieves a 1.8 xto 4.5x speedup without draft-model training and reduces rollback tokens by 50% for poorly aligned draft/target models.
2 RELATED WORK
Speculative Decoding While SD has demonstrated significant acceleration and lossless generaliza- tion, increasing the acceptance rate of draft tokens by the target model remains a critical challenge. Existing approaches rely on training- based (draft model) and training-free methods align the draft and target models. For instance, Medusa introduces auxiliary decoding heads Cai et al. (2024), while EAGLE Li et al. (2024a) and Glide Du et al. (2024) reuse target model information to enhance accu- racy. SpecInfer uses tree-based attention to verify multiple draft candidates to improve acceptance rates (Chen et al., 2023). On the other hand, training-free -fre methods such as Lookahead decoding adopt trajectory caching mechanism to store 7-gram generation histories as draft candidates (Fu et al., 2024). However, all these methods follow a sequential draff-then-verijj paradigm, which is fundamentally limited by the mutual waiting bottleneck. PEARL (Liu et al., 2024b) introduces parallel framework that verifies the first draft token while allowing the draft model to simultancously generate additional tokens during verification. However, it overlooks the impact of rollback when verification fails, which negates the benefit of parallelism if not properly addressed.
Dynamic Drafting Structures Dynamic drafting structure is effective approach optimizing SD. It adapts to the draft sequence length or tree configuration (e.g., depth, width, shape) based on contextual speculation. Current methods model drafting boundaries fall into two categories as illustrated in Table 1: implicil and explicu. Implicit methods rely on output distribution metrics (e.g., confidence, entropy) dynamically terminate drafts (Li et al.. 2024b; Liu al., 2024a; Agrawal et al.. 2024: Zhang et al., 2023). However, these require manually tuned thresholds and struggle to balance flexibility with computational overhead, particularly trainable-hess variants (Huang et al., 2024; Mamou et al., 2024), which require extra time to predict tokens individually. By eliminating per-token prediction, explicit methods such as AdaEAGLE (Zhang et al., 2024) directly estimate draft lengths based on target model features. Unfortunately, the prediction accuracy declines sharply when the estimation sequence length becomes longer. Essentially, none of these works addresses rollback, key bottleneck where the draft model wastes tokens and stalls the parallelism. This work proposes a hybrid framework to combine the implicit confidence-barte termination with explicit sequence modeling that reduces rollback substantially and improves parallel efficiency.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
Notations We define the draft model as Mg and the target model as Mp. Given a prefix X1.j ;.....,,yj). q(-) and p(-) denote the probability distributions of the draft and target models, respectively. The speed ratio c c= Tp/T, quantifies the relative latency. Token generation maps Xıg to embeddings Eig which transforms latent features Fi.j and generates the next-token distribution Pj+1 from the final feature vector fj-
Speculative Decoding Speculative decoding accelerates autoregressive generation through par- allel token verification. The draft model M, proposes y candidate tokens X1 with probabílities {g(2; 1)
The target model M, computes true probabilities in one forward pass. The
acceptance probability for each candidate x; is ((4)) = min (1. q((.)X)-1))) We employ the
Match(p(t:) 11)-1), ((t:(X(;;=)) function (Zhao et al., 2024) to represent the verification process, which identifies the set of accepted or newly sampled tokens. If z; is rejected, subsequent candidates Xi+1:y discarded, and token is resampled from norm p(z;) ((;,))); if all y tokens are accepted, an additional token sampled from p(t,+1) (Leviathan et al., 2023).
4 ANALYSIS OF PARALLEL DECODING
4.1 THEORETICAL SPREDUP
We first quantify the theoretical speedup of parallel SD under different cireumstances. The draft model generates y candidate tokens X1.r, which are verified by the target model. Let Tg = denote the draft model's per-token generation time, and Т₁ at be the target model's verification time. Under full acceptance of y tokens, the baseline SD achieves TsD 7+1 r-
7+1
Parallel SD (Ideal) Under the ideal full acceptance condition, the theoretical per-token latency with parallel decoding (shown in Fig. ((a) can be derived
TPSD max(yt, ct)
72c 54 <<c
Then the speedup ratio from SD is, TSD/ TPsD 2tc or c
7+17. When y cand c > PD achieves an optimal 2x speedup against SD. For autoregressive decoding with y = c, PD represents ex speedup. Nevertheless, in practice, the actual performance depends on the draft acceptance rate, which has been largely overlooked in the prior work (Liu et al., 2024b).
>+
Parallel SD (with Rollback) Recall 3 as the acceptance rate and assume Bs are i.i.d. with a = B(3) as the expected accep- tance rate (how well My approximates Mp). When & S y, the draft accepted length can be approximated by a truncated geo-
a=0.2
a=0.6
Latency under Rollback
a=0.4
--50
metric distribution (Leviathan al., 2023) (shown in Fig. 1(b). detailed in Appendix F.6).
300
200
P(X k) - aa) at I[k <) + a² I[kk 7)
(=26
100
(-38.0
where 7 the probability of full acceptance and a is the probability of rollback. The rollback penalty becomes severe when the draft model has limited capacity, which would cause the subsequent tokens to be discarded and revert parallelism back to serialized execution.
(740.0
0
20 40
60
Draft length(c=400)
Figure 2: Latency under rollback (Theorem 1). For different a, the minimum values presented (the curves have very mild slopes).
Theorem (Latency under Rollback). The per-token latency of parallel SD under rollback is,
2 max(7t,
2et(1 a) ((1+aa)(100)
756 c
TesD.
(1 +++00 a?) a(1-07]
(((+a))1-a7))
2
7 >>
We defer the proofs to Appendix B. As visualized in Fig. 2, the minimum latency occurs at the segment of 7 S e, which theoretically validates the trade-off between parallelism and rollback: while
4

---

Published as a conference paper at ICLR 2026
(a) 8-Class Explict TSNE
(b) 3-Class Hybrid ISNE
[c] Prediction Accuracy
(d) Speedup Comparison
TSNE Dimension
8 B 8
Predicted Accurecy (%)
P-tal
Erio taplck
Carficence
10
Eelit
TYNE Sermentdeon
20-
-20
80
1.4
50 1SNE Dimension
50
20
40 80
100
Reasening Coding Extraction
Dimension
Iteration
Evalustion
Figure 3: Empirical results of different drafting length estimation strategies: (a.b) comparison of T-SNE visualization for the explicit and the proposed hybrid methods from the MLP activations; (c) both implicit and explicit drafting structures have limitations low prediction accuracy of the accepted draft length; (d) impact of different drafting schemes acceleration potentials.
small y underutilizes parallel resources, further increasing 7: beyond the minimum value leads to diminishing returns due to rollback accumulation. This trade-off is a-dependent: for well-aligned models (a + 1), a larger enjoys parallelism but for missaligeeddappciiyycossstiiiiii
draft models < 0.5, the penalties from rollback dominate. Though insightful, this theoretical analysis only reflects the statistical properties rather than run-time dynamics. The actual accepted draft length is ceneext-eepnneett and varies substantially across different iterations (Zhang et al., 2024) (detailed by Fig. 17 in Appendix), which necessitates adaptive control of rather than static configurations.
4.2 ANALYSIS OF ADAPTIVE DRAFT STRUCTURES
Thus, the next question is: "How to optimize draft structures to balance parallelism and rollback?" To answer this, we compare the implicit and explicit methods LLaMA 68M&7B across the MT-Bench datasets (Zheng et al., 2023) for dialogue tasks. Implicit methods evaluate the confidence max,, ((x;)) (Du et al., 2024) or entropy - v XH (x:4) (Agrawal et al., 2024) (positively correlated with acceptance rate) against pre-determined thresholds but the optimal are difficult to find across different tasks, models, temperatures (detailed in Appendix F.8). Additionally, token-level predictions would cause the eroor-compunndigg effect with higher instability across different tasks. On the other hand, explicit methods like AdaEAGLE (Zhang et al., 2024) predict accepted length directly using target model features. Unfortunately, the discriminative power of the explicit methods also declines with the increase of accepted length validated by the visually inseparable clusters in Fig. 3(a). As result, the explicit method results in lower prediction accuracy than the implicit method as indicated in Fig. 3(c), despite less overall variance of the ultimate speedup across different tasks (Fig. 3(d)) A partial reason for this difficulty due to the imbalanced geometric distribution of accepted lengths and the limited contextual information from single feature layer adopted by (Zhang et al., 2024). Based on these insights, we introduce hybrid rollback-aware branch parallelism.
SPECBRANCH: HYBRID ROLLBACK-AWAN BRANCH PARALLELISM
To address the intertwined challenges from pipeline bubbles and rollback cost in SD, we present SpecBranch with two novel components: 1) Hybrid Rollback-Awar Draft Structures (H- RAD), which combine the draft model confidence early stopping and target model feature reuse for adaptive draft lengths; 2) Branch Resampling, parallel drafting-verification mechanism that eliminates sequential bottlenecks through context-aware parallelism. We provide a profiling example in Appendix C.
5.1 H-RAD: HYBRID ROLLBACK-AWARE DRAFT STRUCTURE
As illustrated by Fig. (Case 1), H-RAD predicts the optimal draft lengths before branch resampling in the draft stage. Unlike PEARL's limited prriveiiii of only the first token by target model (Liu et al., 2024b), we leverage the insights from the truncated geometric distribution of accepted tokens in Fig. 1(b) to build hybrid and lightweight predictor reduce rollback.
Hybrid Drafting Length Prediction Given a prefix, our goal is to accurately predict the draft length y S Since direct regression multi-class classification of y suffers from low accuracy, the proposed hybrid design reduces the y-class classification into 3 class classification problem. This is inspired by intriguing bimodal phenomenon that target model features from multiple layers exhibit
5

---

Published as a conference paper at ICLR 2026
SpecBranch: Branch Speculative Decoding via Hybrid Rollback- Draft Structure
P1
P2
Ps
Ps
P6
P7
Pg
Target
verification
fr-
verification
do
great
work
love
roject
①
scoup?
(z.
H-RAD
cose2: All accept
cose3: All reject
casel: Use confidence
love
what
you
(0)
do
great
work
Is
Branch!
92
9s <
stay
true
go
fast
The only
to
good
Prefix
Continll
Continal
franci
work
hard
play
fair
Draft stage
(b) Branch stage
Bwowkl
X_1
Ag
X1
X2
X3
X4
X5
X
X10
X11
draft tokens
accepted tokens
branch points
target model
target tokens
H.RAD
H-RAD outputs st 1, indicating the branch point determined by the draft model's confidence. Case 2 (All Accept): H-RAD outputs 2, predicting that all tokens should be retained, and the branch point is the next round's first token, 'work', Case 3 (All Reject): H-RAD outputs s, = O, predicting that no tokens should be retained, and the branch point is the first token of this round, 'what'.
Figure 4: Architecture of SpecBranch. Case 1 (Use Confidence):
strong separability for the fully accepted and rejected cases, and the rest, intermediate cases can be resolved by the implicit approach. As illustrated Fig. 3(b), the hybrid method provides a more separable clustering compared to the overlapping distributions. Thus, we learn lightweight MLP,
Concat e = Concat (b/-1.-.. ... , 1, e() E R. - Lii-L-DDott
(4)
8; = arg (Soo///oaaaaMM((2))) E {0, 1, 2].
(5)
where extract K hidden states h4-1 from the target model's last K layers and concatenate them with the new token embedding e; to (ft-1, To capture richer context than single-layer approaches (Li et al., 2024a; Zhang et al., 2024), our method uses multiple layers for better length prediction. The output 8; initiates a new hybrid drafting strategy H4 with three classes,
if 8t = (Hard signal: All Reject), if st
74: = [ E X ((())) > })
(Soft signal: Confidence), if 8t 2 (Hard signal: All Accept).
(6)
X 1:7
H4 yields hybrid decisions of su and hard signals, and uses draft model confidence ((x) for the intermediate soft signal $1. Here, the hard and soft signals refer to direct and pending decisions, respectively. According to the empirical distribution, most tokens are handled by hard signals at once (all accept or reject), while small fraction is verified by soft signals later. Such hybrid approach improves the prediction accuracy of the explicit method as well as reduces the compounding errors from the implicit methods. H-RAD predicts the draft length and branch points, providing a dynamic decision for branch resampling, described in Section 5.2.
5.2 BRANCH RESAMPLING: PARALLEL DRAFTING DURING VERIFICATION
Branch Resampling H-RAD identifies unconfident tokens as branch points through the hybrid strategy H, in three cases as shown Fig. Then it splits the draft sequence at branch point x; and sends the prefix X1 to the target model to verify. Mcanwhile, it also spawns À parallel branches via Top-k resampling from the draft model's confidence distribution (2x)). For example, as illustrated by Fig. 4 case 1, in the sequence 'The only way to be' H-RAD outputs
1 to use confidence and the token 'be' (93 < c) triggers a new branch 'do' with 'way to be verified by the target model,
B TopK (g(x)), , k) { { 2 ... 2 where max (1, [imax (1- g(x))))), (7)

---

Published as a conference paper at ICLR 2026
in which & is adaptively controlled and scales inversely with zy's confidence ((z6). Lower confidence in 26, indicates a lower acceptance rate (Du et al., 2024), thus spawning more branches to help SD hedge against likely rejections. Each branch 2j E B generates subsequent tokens independently using the shared KV-Cache of the prefix 1 and branch token zi avoid redundant computation:
X branch X 1:b-1 @ x @ Ma (KV-Cache X 16-1 1 @
The maximum draft length per branch is constrained by the draft/target model speed ratio ensuring simultaneous verification and drafting to eliminate pipeline bubbles.
Branch Verification During token drafting, the target model Mp concurrently verifies the last round tokens X 1:6-1 1 using Match(p(z;] (i.=-)), ((xx ())) (Section 3). If any token v; rejected, all subsequent tokens are discarded, and the target model resamples a new token, back to the draft stage. Unlike vanilla SD, if all tokens in X1:0-1 are accepted, we do not need to resample a new token. Instead, we verify the branch point 26 using p(raa) derived from 2b-1, apply branch speculative sampling algorithm to sample or adjust the distribution (details in Algorithm 2), consistent with (Li al., 2024a; Miao al., 2024), ensuring that the distribution of the output branch point aligns with the target LLM. The branch point verification is formalized as:
V = Match
Drafl probability
Targel probubility
Here, V represents the accepted branch point or resampled token after verification. If 3 x V, we then remain the corresponding branch, discarding all non-selected branches and their associated KV-Cache. Unlike tree-based methods that spawn branches at every token (expensive KV-Cache growth) (Miao et al., 2024). SpecBranch limits branching to uncertainty points identified by H-RAD and avoids the need for complex tree attention verification. We provide more details in Appendix G.2.
As shown in Fig. 4 (Case 2/3), the parallel branch stage introduces a potential temporal mismatch between drafting and verification. During the drafting stage, H-RAD predicts the accepted length based on the previous features before the draft model generates tokens. However, in the branch stage, the tokens from the previous round have not been verified yet, which means H-RAD cannot immediately access those reliable target model features with sufficient guidance from the history. To address this mismatch, we propose a posterior approach for selecting the retained tokens after the branch generates tokens. Once parallel verification is completed, for the remaining V branch, we use (St-1, et) from the current round as input to H-RAD, and select He after the verification step. This ensures that tokens for the next round are selected based on the most up-to-date context, effectively resolving the mismatch as detailed in Appendix G.2.
Posterior Drafting in the Branch Stage
EXPERIMENTS
Implementation Details We evaluate the effectiveness of SpecBranch across diverse LLM configu- rations, particularly focusing on scenarios where draft models have significantly fewer parameters
(68M) and weak alignment with target models (7B-70B), and speedup ratios E [4, 15] (e is rounded up to the integer value). This includes LLaMA (Miao et al., 2024) (68M, 7B, c = 10). Vicuna (Yang et al., 2024) (68M. 13B, c 15), models with better alignment such as Srepseek-Coder (Guo et al.. 2024) (1.3B. 33B. 4) and LLaMA-3.1 (Grattafiori et al., 2024) (8B, 70B, = 5). We assess SpecBranch across several text generation tasks, including HvaanEval (Chen al., 2021), GSM8K (Cobbe et al., 2021), CNN/DM (Nallapati et al., 2).6), and Bench (widely-adopted six sub-tasks) (Xia et al., 2024b). More details are provided in Appendix E.
Baseline Methods We evaluate against the 4 model training free methods. (1) Speculative Decod- ing (SpS) (Chen et al., 2023): Standard implementation where a draft model generates y tokens for parallel verification. (2) AdaEDL (Agrawal et al., 2024): Early-stopping via entropy-based thresh- olds to terminate low- oow-prbbyyility drafts. (3) Lookahead Decoding (Fu et al., 2024): Token-level
speculation using cached n- n-gram matches without auxiliary draft models. (4) PEARL (Liu et al., 2024b): Pipeline parallelism with pre/post-
to overlap traft-target model execution.
Evaluation Metrics We report widely used metrics: Mean Accepted Length M, Wall-Time Speedup Ratio, and Speed (tokens/sec). In SpecBranch, M represents the continuously accepted length (Liu et al., 2024b). We also introduce new metric, Rollback Rate (RB), defined as RB- which quantifies computational waste from invalid drafts, detailed in Appendix E.3.
# Rollbback tokens
# Total tokens
7

---

Published as a conference paper at ICLR 2026
HumanEval
GSM8K
CNN/DM
Speed
Avg. Speedup
Models
Methods
M
Speedup
Speedup
Speedup
(tokens/s)
SpS
2.64
1.46x
3.32
1.74x
2.26
1.42%
63.04
1.54 x
LLaMA 68
AdaEDL
2.49
1.54 x
1.89 x
1,46 x
1.63 x
Lookahead
1.48
1.31
1.96
1.71
1.45
1.25 x
$7.63
I 1422x
PEARL
2.79
1.69
3.82
1.86
2.64
1.66
71.22
1.74
SpecBranch
3.24
2.04
4.46
2.12X
3.17
1.87 x
82.41
2.01X
SpS
2.87
x.79
1.56
2.07
1.45
48.79
1.60
Vicuna 63M&13B
AdaFDD
2.77
1.95x
2.46
1.68 X
2.01
1.53
$1.84
1.72X
Lookahead
1.76
1.57
1.83
1.59
1.52
1.23
43.95
1.66X
PEARL
3.11
2.02
2.83
1.61
2.89
1.68
53.31
1.77x
SpecBranch
3.69
2.47
3.29
1.95X
3.21
1.89
62.57
2.10X
SpS
4.45
2.16
3.85
1.86X
3.91
1.96
31.38
1.99
Deepseek L33B&33B
AdaEDL
4.12
2.35
3.57
2.01X
3.74
2.16X
34.22
2.17x
Lookshead
2.36
x.77
1.74
1.43X
1.89
1.65
25.55
1.62X
PEARL
16.97
3.39
8.28
2.78X
6.45
2.63
46.17
2.93
SpecBranch
22.52
3.71
3.02
2.97
50.94
3.23
SpS
5.25
2.41
5.15
2.31X
5.09
2.11X
16.21
2.28
LLaMA-3.1 8B&
AdaEDL
4.96
x
4.97
2.37X
4.85
2.19X
16.91
2.37 X
Lookahead
PEARL
17.28
3.75 4.02 X
14.33
3.35
7.51
3.04X
24.03
3.38 x
SpecBranch
21.74
18.08
3.67 X
9,41
3.37
26.27
3.69 x
Table 2: Comparison with existing baselines on HumanEval (Chen al., 2021), GSM8K (Cobbe al., 2021) 1) and CNN/DM (Nallapati et al., 2016). "_" indicate incompatibility: baseline implementations (transformers 4.36.2) conflict with L.LaMA 3s dependency on 2 4.43.0.
MT Bench
QA
Sum
Math
RAG
Trans
Models
Methods
M
Speedup
M
Speedup
M
Speedup
M
Speedup
M
Speedup
M
Speedup
Avg.
SpS
2.63
1.74x
2.47
1.64x
2.58
1.70x
2.37
1.55x
2.47
1.56x
2.57
1.65x
1.64x
AdaEDL
2.50
1.80x
2.43
1.67x
2.64
1.72x
2.31
1.62x
2.21
1.57x
2.45
1,75x
1.69x
Vicuna
Lookahead
x
1.41
1.23x
1.49
1.25x
1.71
1.46x
1.15x
1.32
1.10x
1.25x
PEARL
2.62
1.78x
2.45
1.64x
2,78
1.83x
2,63
1.67x
2.61
1.66x
2.89
2.05x
1.77x
SpecBranch
SpS
4.67
2.31x
4.57
2.27x
5.09
1.98%
5,01
2.44x
5.08
2.02x
5.52
2.57x
2.27x
AdaEDL
4.31
2.43x
4.23
2.30x
4,83
2.05x
4.94
2.46X
4.86
2.13%
5.24
2.65x
2.34x
LLaMA-3.1
Loukahead
PEARL
8.46
2,96
8.37
3.27
9.10
3.32x
12.53
3.39x
8.35
3.41x
12.59
4,22x
3.43x
SpecBranch
10.85
3.24%
10.59
3,45%
11.40
3,63x
15.76
3.78x
9.16
3.40x
16.64
4,51x
3,67x
Table 3: Comparison with the existing baselines Spec-Bench (Xia et al., 2024b).
H-RAD Training and Generalization The training data for H-RAD pairs the feature vector žt from Eq. (4) with the corresponding three-class labels 9t. We implement a lightweight three-layer MLP with ReLU activation. Training is performed offtine for 20 epochs and 32 batch size. The training converges within minutes on single NVIDIA A100 GPU and eliminates the need for costly online fine- -uningn We further conduct experiments demonstrating the cross-task generalization of H-RAD, where its performance only degrades by 5% (Table 8). We provide details Appendix E.4.
6.1 MAIN RESULTS
The main results from Tables 2, and Fig. 5 are explained be- low: (l) On HumanEval, GSM8K, and CNN/DM, SpecBranch shows superior efficiency over prior methods, achieving con- sistent speedups of 1.9x to 4.0x over vanilla autoregressive decoding. On Spec-Bench, our method also achieves signifi- cant acceleration from 1.8x to 4.5x on six diverse sub-taske indicating robustness and versatility. (II) For poorly aligned models (LLaMA, Vicuna), rollback dominates parallel accel- eration. SpecBranch improves over PEARL by 15%. As shown in Fig. 5, Spec reduces rollback from 66-90% to under 40%, yielding longer generated lengths M compared to the baseline (Tables 2, 3). This validates the capability of H-RAD to terminate those draft paths with ultimate failure. (III) For well-aligned models (Deepseek, I.aMM-3.1), SpecBranch improves parallelism and resource uti-
102
Speciranc
Locull
TEB:
Rulback
Figure 5: Comparison Rollback Rates on HvmanEval.
8

---

Published as a conference paper at ICLR 2026
lization through branch resampling. E.g., it achieves 3.23x speedup PEARL's 2.93 (10.2%). with a 4.14x improvement the average accepted length M against SpS on code generation tasks. It also reduces the rollback rate by 10% and improves speedup by 8% compared to PEARL. These results empirically validate the trade-off in Theorem and more details are provided in Appendix R.
6.2 ABLATION STUDIES
SpedBranch SpecBranch who H-RAD
SpecBranch H-RAD
SperBranch who branch
SpecBranch
3.5
Mean accepled longth
accepted length
2.
-3.0
Speedup
2.0
-2.5
Speedup
3.72
1.8-
375
1:55
72
1.71
-1.5
2.0
3.5
2.77
3.24
3.17
3.37
172
1.6
2.5
2.35
MT
QA
Sum
Math
RAG
Trans
MT
QA
Sum
Math
RAG
Trans
(a) Vicuna 6BM 138
(b) LLaMA 3.1 BB & 70B
Figure 6: Component Ablation on the Spec-Bende benchmark: (a) for poorly aligned model pairs like Vicuna 68M-13B, H-RAD provides higher contributions; (b) for better-aligned models such as LLaMA-3.1 8B-70B, branch resampling plays a more important role in acceleration.
Component Analysis Our ablation studies first isolate the two core components by removing: (1) branch resampling (SpecBranch w/o branch). or (2) H-RAD (SpecBranch w/o H-RAD). Fig. reveals that the absence of either component degrades performance. H-RAD dominates efficiency gains for misaligned model pairs like Vicuna 68M- 13B, which improves the speedup from 1.72x to 1.95x. On the other hand, branch resampling contributes more with better-aligned pairs like LLaMA-3.1 8B-70B. Both components offer complementary advantages to reduce rollback and improve parallelism by adapting to different model capacity. Meanwhile, branch parallelism and H-RAD are both modular designs, which are orthogonal to methods like EAGLE (Li et al., 2024a;b) and can be quickly adapted, which we leave for future exploration detailed in Appendix G.2.
Hyperparameter Sensitivity Since H-RAD still entails the confidence threshold from the implicit methods, we evaluate its sensitivity to such threshold and feature layer hyperparameters in Tables and by integrating H-RAD with vanilla SD. Table shows that while the speed for implicit methods drops from 64 49 tokens/s increases, H-RAD only decreases from 72 to 67 tokens/s, with less dependence on the threshold. The explicit variant in Table reveals diminishing returns with more feature layers (K), that increasing context layers K from 4 32 has marginal gains of I tokens/s but 8x more memory overhead. Thus, we choose =
4 to balance the speed and memory.
Implicit Confiden(on Implici((Entropy) Hybrid(H-RAD)
K
HumanEvalkcoding)
GSMSK/reason)
CNN/DMisum)
0.1
61.05
60.28
70.32
1
62.35
63.28
52.82
64.26
2
64.06
76.14
$7.46
61.12
71.02
4
72.15
79.24
63.46
0.6
56.43
$4.73
69.91
8
73.28
80.22
63.86
0.8
53.21
52.29
68.62
16
32
74.18
81.33
64.41
Table 4: Results of Stop thresholds of LLaMA 68M&7B on HumanEval. (tokens/sec)
Table 5: Results of feature layers K of LLaMA 68M& on H-RAD+SD. (tokens/sec)
Resource Consumption We further validate SpecBranch's 's effectiveness across three dimensions of resource consumption. For memory consumption, we test the LLaMM-3.1 models on HumanEval; for energy/time cost, we use poorly-aligned Vicuna on HumanEval. Fig. 7(a) shows that, due to the shared prefix KV-Cache, memory consumption for parallel branches with varying (from I to 16) only has a slight increment to 28% of the baseline model parameters. In SpecBranch, & is dynamically adjusted based on confidence at the branch point with kmax typically capped by 6. SpecBranch strategically spawns sparse branch points high-impat tokens, which improves computational and memory efficiency by avoiding unnecessary branching. By the rollback-awan aware designs, Fig. 7()) demonstrates that SpecBranch reduces energy consumption by 43% against PEARL. Fig. 7(c) shows the time cost

---

Published as a conference paper at ICLR 2026
for each SpecBranch module, where the H-RAD prediction cost is negligible (0.38% of total latency).
SpecBranch eliminates the mutual waiting bubbles between draft and target models with almost identical execution time (30.9 vs 31.4 ms per step). We provide more details in Appendix F.4, G.I.
Memory Used (GB)
Methods
Emergy ()
Modules
Latency (ms)
SpS
383k
Dmfit
30.9 + 1.36
16
PEARL
445k
Verifiattion
31.4 + 1.62
Branch
Spec-
251k
Prediet
0.24 0.02
160
250
260
[a] Memory consumption
(b) Energy consumption
(c) Time consumption
Figure 7: Resource consumption of SpecBranch through NVIDIA DCGM. (a) Trace of memory footprint of SpecBranch inference with different number of branches & on HumanEval using LLaMA- 3.1 8B-70B; (b)(c) Energy and time cost on HumanEval using Vieuna 68M-
Branch Length We define the max branch length as the default draft length y in each decoding round. Unlike PEARL, which statically fixes 7 = c (the theoretical speedup ratio) for all model pairs, SpecBranch introduces rollback-aware parallel SD architecture, optimizing y based on the parallelism-rolllbee tra As shown in Fig. 8, we evaluate two representative cases on 10) and LLaMA-3.1 8B/70B (high alignment,
HumanEval: LLaMA 68M/7B (low alignment, 5). For the low-alignment pair, the optimal 7 8 falls strictly below e, validating the trade-off analysis in Fig. 2. this regime, the latency penalty from rollbacks dominates parallelism gains, which is a critical nuance PEARL overlooks. Conversely, for high-alignment pairs, the optimal approaches as the benefits of parallelism prevail over rollback costs.
LLaMA 68m 78 (c=10)
LLa MA 3.1 8B 70B (c-5)
Speedup (x)
2.0
Max Speedup: 2.0x Optimal Capacity
4.0
Max Speedup: 4,0x Matches Capacity
Speedup
3.5
1.8
3.0
9
3
4
5
6
Max Branch Length
Max Branch Length (y)
Figure& 8: Impact of max branch length on inference speedup. Unlike PEARL's fixed setting (7 c). the optimal y varies by model alignment: (a) for the poorly aligned LLaMA pair (c 10), optimal 8 falls below due to dominant rollback costs; (b) for the well-aligned LLaMA-3.1 pair (c 5), optimal y approaches e as parallelism benefits prevail.
More Discussion Due to space, we include more results and discussions in the Appendix, including more comparisons with the tree methods (EAGLE) ( F.I,, G.10), high-batch performance ( (G.9), kmaz analysis ( (F2),, conttrcndd application (..1), tree structure and temporal mismatch ( (G.2) and more ablation results (F8, F.10).
7 CONCLUSION
Future Work Current parallel speculative decoding relies on off-the-shelf draft models, where semantic misalignment often limits verification efficiency. To address this bottleneck, we plan to explore on-polis on-policy online distillation (Lu Lab, 2025) dynamically adapt draft policies for superior alignment with the target model. Furthermore, we aim to synergize SpecBranch's 's parallel H-RAD mechanism with advanced feature-level speculative methods (e.g., EAGLE) and deploy these integrated engines into high-performance serving frameworks like vLLM and SGLang, thereby satisfying the rigorous demands of industrial environments.
We propose SpecBranch, rollback-aware parallel SD framework. By enabling concurrent branch drafting and verification, SpecBranch addresses the bottleneck of serialized execution via two key innovations of branch resampling and hybrid drafting structures. Experiments show that SpecBranch achieves 1.8 ~ 4.5x speedup and reduces computational waste up to 50% for poorly aligned models, while theoretically guaranteeing the preservation of the generated text's distribution.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is supported by Zhejiang Provincial National Science Foundation of China under Grant No. LZ25F020007 and National Science Foundation of China under grants 62576310, 62394341.
ETHICS STATEMENT
The datasets used in our experiments are publicly released and labeled through interaction with humans in English. In this process, user privacy is protected, and no personal information is contained in the dataset. The scientific artifacts that we used are available for research with permissive licenses. The use of these artifacts in this paper is consistent with their intended purpose.
REPRODUCIBILITY STATEMENT
We discuss the experimental settings in Section and Appendix E, including implementation details such models, datasets, inference setup, and evaluation metrics.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303.08774 2023.
Sudhanshu Agrawal, Wonseok Jeon, and Mingu Lee. Adaedl: Early draft stopping for speculative de- coding of large language models via an entropy-based lower bound on token acceptance probability, arXiv preprint arXiv:2410. 18351, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
Tianle Cai. Yuhong Li, Zhengyang Geng, Hongwu Peng. Jason Lee, Deming Chen, and Tri Dao. Medusa: Simple I1m inference accelcration framework with multiple decoding heads. arXiv preprint arXiv:2401. 10774, 2024.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer, Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013/6 18, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman et al. Evaluating large language models trained on code. 2021.
Zhuoming Chen, Avner May, Ruslan Svirschevski, Yu-Hsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable and robust speculative decoding. Advances in Neural Information Processing Systems, 37:119531-129566 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. Cornell University arXiv, Comell University- arXiv, Oct 2021.
Cunxiao Du. Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nic, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. arXiv preprint 20Xiv:2402.02082 2024.
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint arXiv:2402.088 02057. 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan,
11

---

Published as a conference paper at ICLR 2026
and Anirudh Goyal et al. The Ilama 3 herd of models, 2024. URL hwtpthtttaatiiiiieee URtsst/s///////.oog/ abs/ abs/2407, //////202222222
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y.K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseck-coder: When the large language model meets programming the rise of code intelligence, 2024. URL https:
//arxiv.org/ abs 2401 240..2120.....6......
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini arXiv:2501.1290 2025.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval-based speculative
decoding. arXiv preprint arXiv:2311.00 082522 2023.
Kaixuan Huang, Xudong Guo, and Mengdi Wang. Specdec++: Boosting speculative decoding via adaptive candidate lengths. arXiv preprini arXiv:2405. 2024.
Daniel A Jiménez and Calvin Lin. Dynamic branch prediction with perceptrons. In Proceedings
HPCA Seventh International Symposium on High-Performance Computer Architecture, pp. 197- 206. IEEE, 2001.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty, arXiv preprint arXiv:2401.15077, 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 16858, 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang. and Hongyang Zhang. Eagle-3: Scaling up inference ac- celeration of large language models via training-time test. arXiv preprint arXiv:2503.01 2025.
Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Duyu Tang, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding for accelerating ilms via double early exiting. Advances in Neural Information Processing Systems, 37:11946-1 1965, 2024a.
Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, and Winston Hu. Parallel speculative decoding with adaptive draft length. arXiv preprint arXiv:2408. / 18500 2024b.
lyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Kevin Lu and Thinking Machines Lab. On-policy distillation. Thinking Machines Lab: Connection-
ism, 2025, doi: 20.644344m1..225100000
Jonathan Mamou, Oren Pereg, Daniel Korat, Moshe Berchansky, Nadav Timor, Moshe Wasserblat, and Roy Schwartz. Dynamic speculation lookahead accelerates speculative decoding of large language models. arXiv preprint arXiv:2405.04304, 2024.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng. Zeyu Wang. Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support Programming Languages and Operating Systems, Volume pp. 932-949, 2024.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. Abstractive text summarization using sequence-to-sen to-quueence rnns and beyond. In Proceedings The 20th SIGNLL Conference on Computational Natural Language Learning, Jan 2016. doi: 18.13////v1/ k16-1028. URL http: //dx doi. org/ 110. 11653/v1/k166 wo0../1/1........ 1028..
Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia. Pipedream: Generalized pipeline parallelism for dnn training. In Proceedings of the 27th ACM symposium on operating systems principles, pp.
12

---

Published as a conference paper at ICLR 2026
1-15, 2019.
Zhan Shi, Xiangru Huang, Akanksha Jain, and Calvin Lin. Applying deep learning to the cache replacement problem. In Proceedings of the S2nd Annual IEEE/ACM International Symposium Microarchitecture, pp. 413-425, 2019.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:23/2./1805, 2023.
Zhepei Wei, Wei-Lin Chen, Xinyu Zhu, and Yu Meng. Adadecode: Accelerating Ilm decoding with adaptive layer parallelism. arXiv preprint arXiv:2506.037000 2025.
Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. Swift: On-the-ffy fly self-speculative decoding for Ilm inference acceleration. arXiv preprint arXiv:2410.06916, 2024a.
Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang, Yonggi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. Jan 2024b.
Sen Yang, Shujian Huang, Xinyu Dai, and Jiajun Chen. Multi-candidate speculative decoding. Jan 2024.
Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprini arXiv:2309. 20X.::3309008168, 2023.
Situo Zhang, Hankun Wang, Da Ma, Zichen Zhu, Lu Chen, Kunyao Lan, and Kai Yu. Adacagle: Optimizing speculative decoding via explicit modeling of adaptive draft structures. arXiv preprint 2:Xiv:24/2.189100 2024.
Weilin Zhao, Yuxiang Huang, Xu Han, Wang Xu, Chaojun Xiao, Xinrong Zhang, Yewei Fang. Kaihuo Zhang, Zhiyuan Liu, and Maosong Sun. Ouroboros: Generating longer drafts phrase by phrase for faster speculative decoding. arXiv preprint arXiv:2402. 13720, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric.P Xing, Hao Zhang, JosephE. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. Jun 2023.
Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. Jan 2024.
13

---

Published as a conference paper at ICLR 2026
A PROCEDURES OF SPECBRANCH
Algorithm 1: Algorithm of SpecBranch
Input: Prefix I, draft model Mg. target model Mp. max length L, target model feature f. target model embedding e. predicted output 8;. confidence threshold c, num branches B, max gamma T'm execution mode E.
Output: Generated sequence
x < 7 +- Tm + DRAFTING
while x L do
if E = =DRAFTING then
// Drafting phase: only drafting candidate tokens P Predicter(ff--- Ct.C, 7m)
for + to 7 qi M
[1 ........00 xi-
end
E + VERIFICAS ICATION Verification phase: evaluate branches
for b E (I,..., B} do for i 7 1 27 do
Q(6.1) Mask Mg (x [T(0.))) .... ....)))....0....)
((bb.) 4(b.))
if g(..) (2)().)) then
keep the invalid position
end end (PI...., PPy) &- (Mp [zi])..... M,(x (x1....,,,,) Retrieve (41...., 97) from cache for + to 1))
end
min(fi-1
pala 4: 22
U {+)
n = then //Al All drafted tokens accepted
if :6: S p[anas]
1).,+1)) [1] then
+ Fi -.t)).-..,,... --. m))
? Predictor(fs,ct.. ctt,, 7m)
E' + VERIFICATION
else
// Reject next token fallback target
y ~ N (max(0, Pa+1 ((b++1)))
x [y]
E + DRAFTING
end
else
// Rejection occurred position
y N (max(0,, Pa+1 9n+1)) x I1,..,, Ini y]
E + DRAFTING
end
end
return x
14

---

Published as a conference paper at ICLR 2026
B PROOF OF THEOREM
We re-state Theorem (Section 4.1) here for convenience with the basic assumption from (Leviathan et al., 2023) that the token acceptance is ii... with (accepp)) = Recall that the generation time Tg = t. the verification time Tр = ct, and any rollback would trigger full retry of ạ tokens. To prove Theorem I, we present Lemma the expected token draft length first.
Lemma (Expected Draft Accepted length) For truncated geometric distribution, X TruncGeola, 7),
E|x] = a(1 a(l-a)) aa7)
(10)
1
Proof.
E|X]
h P. x) Σ k (1- ty
k=
Let S Ei=o a
T-a Take differentiation regarding a, have,
ds da
1 11-70=1 + (7 1)x²
Σ ka k-1
(1 a)²
k==
ds
a(1 a?) + y a² +
E (1 a)a
da
1-a a
Theorem (Latency under Rollback) The latency of parallel SD under rollback is,
2ct(1 - aa) a + a²) a²)' 2yt(1 -
7 c
2 1-max(7t,ck
a(1-a7) TRO, (1 + a²)
1-a
a(1 + a²) a2-) 7>c
a²)"
Proof. Define the acceptance vector w (wi,...,w,) E (0, 1)? where wi = 1 if and only token i is accepted. The accepted token count is,
X
Wi
P(w: = =)= (i.i.d)..
To compute the total number of tokens with retry, define two rounds of: 1) tokens (accepted 1); 2) Retry if Round 1 fails, which yields E[X] tokens. Thus, the total expectation is
Faotal a + F[X]) + (1-a?) - (1 +??) F
(1)
- az
This implies that Parallel SD (with Rollback) achieves an acceleration factor of + a²) compared to the vanilla SD (with Rollback). As a approaches 1, the acceleration ratio reaches 2x, matching the acceleration of the Ideal Parallel SD in Eq. (1). Thus, the total time for the two rounds (parallel generaaion/verrffcattion verification):
Tistal max maxxtt,tn et)
T Total
2 (1ax((7(c)) cl)
TPSD.
(3)
Exoul
(1+a7)
al1-a?)
1-a
For different cases of length y, have,
Case 1: y < c (Verification time et 7t):
2cl
2cl - a)
TySD, (1+
a(l-a7)
a(1+ + a7)
1-a
Case 2: 7 (Generation time 76 > cl):
2nt
2yt(1 aa)
TPsD. + (1+a7) a(1-a²)
a(1 + a (1- a²)
(Simplified using 71 dt 1 > ol). This completes the proof of Theorem 1.
15

---

Published as a conference paper at ICLR 2026
C PROFILING EXAMPLE
We have illustrated the main process of SpecBranch in Fig. 4. Here, we provide more detailed step-by-step profiling example of SpecBranch with input prefix 'the only' in Fig. 9.
Step
Prefix
Input Signals
Draft model Output
Target
Judgement Tips
model Output
1
The only
draft stage (predict)
way to be
No
In the draft stage, II-RAD outputs st == I and uses confidence decide the branch point before generating tokens. q(be) eps, the branch is "be'.
q(be) cps
2
The only
branch stage
(1) be good man is to
way do
In the branch stage, 'be' and 'do' continue branches. After verification. 'do' is kept, and H-RAD oulputs st 2, retaining all tokens for the next verification.
(posterior)
(2) great work is
st=2 accept
The only way do
branch stage (posterior)
(1) work hard play fair
great work love
In the branch stage, "work" starts branchs, "love" After verification, 'love' is retained, and H-RAD outputs st = 0, discarding all tokens while branching to "what'.
(2) stay true go fast
(3) love what you do
st=0
all reject
The only way to do great work to love
branch stage (posterior)
(I) what you do throughout the (2) the efforts you devote during
the
In the branch stage, 'what" branches to 'the", After verification. 'the" remains, and H-RAD oulputs st 1, using confidence and q(devote) eps branch into 'devote'.
st-1 q(devote) eps
5
The only way to do great work is to love the
branch stage (posterior)
(1) devote during the whole research
the efforts you
In the branch stage, 'devote' branches to 'put". After verification, "put' is kept, and II-RAD outputs retaining all tokens for the verification.
(2) put into the whole project
st-2
accept
6
The only way to do great work is to love the efforts you
branch stage
(1)-but but not the fame (2) and the insight you
into the whole research
In the branch stage, 'but' branches to 'and" After verification, project rejected, and rollback 'whole' concatenated with "rescarch".
(posterior)
Rollback!
7
The only way to do great work is to love the efforts you put into whole rescarch
draft stage (predict)
but not the successes
NNo
Once token is rejected by the target model (cither branch point or as a normal token), the process returss to draft stage and repeats.
st-2
accept
Figure A step-by-step profiling example of SpecBranch with an input prefix 'the only".
The
cyan text represents the stage and the red text indicates branch points or rejected tokens. Underlined text signifies the tokens retained after each round of posterior drafting in the branch stage, which are sent to the target model for verification. The dashed lines represent the tokens discarded during rollback. We give some explanations about the whole process step by step.
16

---

Published as a conference paper at ICLR 2026
1) At step I, we extract features from the prefix 'The only' and input it H-RAD (assuming the preceding context is already established), which outputs the generation strategy & = 1. means that we use the implicit confidence ((x) to determine when to begin branching. Then, 'The only' is input to the draft model, and the target model does not operate during this stage. The draft model generates 'way to" until g(be) < c, which point branch at 'be' and send "way to' to the target model for parallel verification, transitioning from the draft stage to the branch stage.
2) step 2, 'be" and 'do' are generated parallel as two separate branches, while "way to' is verified by the target model. Once both drafting and verification are completed, the target model outputs 'way to do,' indicating that the second branch 'do" is selected and the 'be" branch is discarded. This allows SpecBranch successfully avoid the rollback of 'be' through H-RAD prediction and branch resampling. Then, H-RAD reuses the feature output 8 = 2. indicating that all tokens have high acceptance probability, and "great work is to' is sent for verification in the next round.
3) At step 3, since 8; = 2, 'word' is the first token of this round and considered as an unconfident token, prompting the generation of new branches 'stay" and "love'. After the target model verifies and accepts the sequence "great work is to," it also outputs 'love' to match the branch. By generating a new branch, the rejection of 'work' is avoided. Then, H-RAD reuses the feature output s = 0, indicating that all tokens have a low acceptance probability, and only 'love' is sent to the target model to match the branch point 'what'.
4) At step 4, 'what' branches to 'the' and generates tokens in parallel. Then, the target model outputs 'the", meaning 'what' rejected and the 'the' branch is retained, thus avoiding the rollback of 'what'. H-RAD and branch resampling combine to improve token rollback handling, while PEARL suffers from token rollback. Then, H-RAD outputs s; = 1, meaning we need to use confidence to determine the branch point 'devote', and send 'the efforts you' to the target model for the next round of verification.
5) At step 5, similarly, SpecBranch combats the unconfident token 'devote' through implicit confi- dence early stopping and branch resampling, improving parallel efficiency. Once 'the efforts you put' accepted, H-RAD outputs 8 to retain all tokens generated by the 'put' branch.
6) At step 6, provide rollback case. In the new round, the first token 'but' branches to 'and', drafting tokens. However, the target model rejects the previous round's token 'project' and resamples new token 'research', This means that all the tokens generated in parallel need be discarded, and we return to 'whole" and concatenate with 'research'. Meanwhile, if none of the branch points match the output of the target model, this also treated as rollback.
7) At step 7, once token is rejected by the target model (either at the branch point or normal token), the process returns to the draft stage and repeats the above process of draft branch in parallel. For those "doomed tokens" that cannot be avoided, they have to be rolled back to the draft stage. This completes the feedback loop.
predict RAD
posterior
H-
H- RAD
branching
branching
prefix
draft
branch
EOS
rollback
Figure 10: Illustration of the SpecBranch state transitions.
Meanwhile, to better illustrate the SpecBranch workflow, we present a simplified stage-transition logical loop in Fig. 10. Given a prefix, if target-model features are available, H-RAD reuses these features to predict the draft length during the draft stage. The transition from the draft stage to the branch stage occurs when a branch point is selected. which be one of three scenarios: (1) s; = 0 (all-reject), indicating that the branch point is the first token of this round; (2) s; = 1, where confidence is used to decide the branch point; and (3) St (all-accept), indicating that the branch
17

---

Published as a conference paper at ICLR 2026
point is the first token of the next round. Each scenario triggers a distinct branching ensure correct overlap between drafting and verification.
Similarly, in the branch stage, H-RAD uses posterior approach to select the retained tokens, leading to three branching scenarios for parallel exccution. Once a token is rejected by the target model (either the branch point or regular token), the branch stage reverts the draft stage and repeats the draft-branch process in parallel, thus maintaining the logical loop.
D BRANCH SPECULATIVE SAMPLING
As discussed in Section 5.2, we verify the branch point zs using p(z;) derived from Ib-1- and apply branch speculative sampling algorithms to sample or adjust the distribution of candidate branches. The branch point verification formalized as:
V Match (a(=)) (x()) ((ab))...,.p$$..... p(())))
(11)
Draft probability
Target probability
Single-round speculative sampling relies chain-structured draft, whereas SpecBranch adopts branch structure. Inspired by (Li et al., 2024a; Miao et al., 2024), SpecBranch introduces Branch Speculative Sampling. Specifically, given the target model's distribution p(ra) for the branch point, we perform speculative sampling token-by-token on the top-k branch point candidates. For each
candidate .rf, we evaluate it against the criterion Tb p(=()
q(={)" Once any branch point token zi.
is verified and accepted, the corresponding branch is selected and retained. In non-acceptance scenarios, branch speculative sampling recursively invokes single-round speculative sampling, instead of retaining the original naive sampling. The pscudocode corresponding to Branch Speculative Sampling is detailed in Algorithm 2.
Algorithm 2: Branch Speculative Sampling
Input: Branch points target model distribution p(as), top-k branch points æj, and distributions ((a[)) for each from k, where zi, is sampled from q(z;), Output: sample ~ p(xx);
for i < k do Tb + (0, if rb p(xt)//(<(() then Return x/ end if p(xs) nozm((mxx00,(aa)) ((x1))) 1ei+1 end for Sample a new token 2y ~ p(xs)
Return 2b
Unlike tree-based methods that spawn branches every token (expensive KV-Cache growth) (Miao et al., 2024), SpecBranch limits branching to uncertainty points identified by H-RAD and avoids the need for complex tree attention mask verification. This means SpecBranch only applies branch spec- ulative sampling at branch points, while employing the typical speculative sampling for subsequent draft tokens. This ensures an identical sampling distribution the target model, enabling lossless acceleration while also significantly reducing deployment complexity and computational overhead.
Our main experiments are conducted under greedy decoding conditions that are consistent across all baselines i.e., target model temperature 0. To further verify the lossless nature of our method, we perform additional experiments on GSM8K across different temperatures to evaluate accuracy. As shown in the Table 6, despite slight variation for Vicuna, SpecBranch achieves identical accuracy with the vanilla autoregressive decoding across various model pairs and temperatures. Branch Speculative Sampling theoretically guarantees the original output distribution of the target L.L.M.
18

---

Published as a conference paper at ICLR 2026
Temperature=0
Temperature-0.5
Temperature=1
Models
Methods
Acc.
Speedup
Acc.
Specdup
Acc.
Speedup
Vanilla
0.29
1.00 x
0.25
1.00
0.21
1.00
Vicuna
SpecBranch
0.29
1.95 x
0.24
1.83 x
0.22
1.75 x
Vanilla
0.93
1.00 x
0.92
1.00
0.90
1.00 X
LLaMA-3.1
SpecBranch
0.93
3.67 X
0.92
3.58
0.90
3.54
Table SpecBranch achieves lossless acceleration across various model pairs and temperatures on GSM8K (Cobbe al., 2021).
E EVALUATION DETAILS
For reproducibility, we discuss the experimental setup (Section 6) detail and source code of this project will be made available later time.
E.I DATA CONFIGURATIONS
In our experiments, we evaluate SpecBranch using the following dataset settings. The tasks include code generation, multilingual arithmetic reasoning, summarization, and Spec-Bench widely- adopted benchmark consisting of six sub-tasks) (Xia et al., 2024b). For code generation, we use HumanEval (Chen al., 2021). a widely recognized benchmark comprising 164 problems. For arithmetic reasoning and multilingual inference, we use GSMSK (Cobbe et al., 2021), presenting their results side by side. Specifically, we sample the first 100 examples from GSM8K. For summarization, we use CNN/DM (Nallapati et al., 201 6), and also sample the first 100 examples. The maximum generation length for these tasks is set 512 tokens.
For Spec-Bench, we define distinct templates for each subtask. The template for Vicuna follows the official format, while for LLaMA and DeepSeek, the templates are as follows:
MT-Bench: "A conversation between curious user and an AI assistant, where the assistant provides helpful, detailed, and polite answers to the user's questions."
QA: "A conversation between a curious user and Al assistant, where the assistant gives helpful, detailed, and polite answers to the user's questions."
Summarization: "Summarize: QUESTION TL;DR."
Translation: "Translate German to English. German: QUESTION English."
Math: "Leet's think step by step."
RAG: "A conversation between a curious user and an Al assistant, where the assistant gives helpful, detailed, and polite answers to the user's questions."
Specifically, due to the unique characteristics of LLaMA-3.1, we design specialized template as follows: "You are helpful, respectful, and honest assistant. Always answer helpfully possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If question does not make sense or is not factually coherent, explain why instead of providing an incorrect answer. you don't know the answer, please do share false information."
E.2 MODEL CONFIGURATIONS
To validate performance, we select state-of-the-an open-source model pairs such as the LLaMA series (JackFram /LLaMA-6MM, MuggyLLMMA/LLMMA+ Vicuna (double7/vicuna-68M Imsys/vieuna- 13b-v1.3), Deepseek-Coder (depsset-aiieeepsek--ooer 3.3-insrrutt, -eppsek-aideeessee-coeer 33b-instruct) and LLaMA-3.1 (Meta-LLaMAVLLLMAAA 1..1-88-IInstuuee Instruct, meta-LLaaMA/LLMMAA NOB-Instruct) for each task. All model weights loaded in bfloat16 format for optimized GPU inference without quantization. As draft model training- free method, SpecBranch does not modify any draft model parameters during evaluation. We summarize the model configuration in Table 7.
19

---

Published as a conference paper at ICLR 2026
Models
Layers
dim
FFN dim
Vocabulary size
LLaMA 68M
2
768
3072
32000
LLaMA 7B
32
4096
I 11008
32000
Vicuna 68M
2
768
3072
32000
Vicuna 13B
40
5120
13824
32000
Deepseek 1.3B
24
2048
5504
32256
Deepseek 33B
62
7168
19200
32256
LLaMA-3., -311 8B
32
4096
14336
128256
LLaMA-3. 70B
80
8192
28672
128256
Table 7: Model configurations.
E.3 EVALUATION DETAILS
We report widely used metrics for speculative decoding (SD): Mean Accepted Length M. Wall-Time Speedup Ratio, and Speed (tokens/sec). Additionally. introduce new metric, Rollback Rate (RB), defined as RB #Rollbuck tokens which quantifies computational waste due invalid drafts. In # Total tokens
SpecBranch, M represents the continuously accepted length (Liu al., 2024b), which is not the fixed length accepted in single round of y, but rather the higher accepted length achieved through multiple rounds parallel generation, surpassing the performance of Vanilla SD. RB specifically refers to the number of rollbacks during the draft model's forward times, excluding additional token loss due to branch and tree structures (since the impact of draft parallelism on acceleration negligible).
All experiments, including the main results and ablation studies, are conducted NVIDIA A100- PCIE-40G GPUs. Models with fewer than 8B parameters are run on a single device, 33B models on two devices, and 70B models four devices. For inference, set the batch size to match standard speculative decoding settings.
Temperature Sampling For SpecBranch and other baselines, temperature to 0 as greedy sampling. Since Top-k resampling is adopted in SpecBranch, we derive the draft token confidence temperature More discussions about temperature sampling detailed in Table 6.
the draft and target model
Resource Consumption In the resource consumption experiments, Center GPU Manager (DCGM) to monitor real-time GPU memory usage and power. Energy consumption calculated by multiplying the average power by the total inference time over the entire benchmark.
the NVIDIA Data
All baselines, such PEARL and SpS, are reproduced from their original papers and official codebases, using the optimal configurations reported by the authors. Experiments are conducted under identical conditions avoid introducing bias. Importantly, implementation of Lookahead requires transformers = 4.36.2, which conflicts with LLaMA 3.1's dependency on transformers 2 4.43.0. similar situation is also found (Liu et al., 2024b).
LLaMA-3. model pairs,
E.4 TRAINING DETAILS AND CROSS- TASK GENERALIZATION
H-RAD Training Our H-RAD training is performed offline on specific datasets, including our benchmarks. The training data for H-RAD pairs the feature vector z, from Eq. (4) with the corresponding three-class labels St. We implement a lightweight three-layer MLP with ReLU activation and dropout (rate= 0.4). The model architecture consists of two hidden layers (256 and 64 units) followed by a classification layer. Training performed offline for 20 epochs with 32 batch size using the AdamW optimizer (Loshchilov & Hutter, 2017) (learning rate- 5 x 10-5, weight decay= 1 x 10 10-4). To address class imbalance issues, we employ SMOTE (Synthetic Minority Over- sampling Technique) data augmentation (Chawla et al., 2002), specifically targeting underrepresented classes in H-RAD. The augmentation process involves standardizing features, applying SMOTE with k 5 nearest neighbors, and then inverse transforming the synthetic samples. Additionally, utilize label smoothing (smoothing= 0.1) in the loss function to prevent overconfident predictions and improve generalization. The training process incorporates several optimization strategies:
20

---

Published as a conference paper at ICLR 2026
Learning rate scheduling with ReduceLLROnPlaeau (factor= 0.5, patience= 2).
Early stopping with 5 epochs patience.
Gradient clipping with max norm of 1.0.
Weighted random sampling to balance class distributions.
The training converges within 5 minutes on single NVIDIA A GPU and avoids the need for costly online fine-tuning while maintaining adaptability to diverse domains. The model achieves balanced performance across the three classes, validated through confusion matrix analysis and t-SNE visualization of the learned feature space.
Cross-Task Generalization First, emphasize that H-RAD consists of lightweight MLP predictor, which involves very low offline training costs. Re-training and deployment on new datasets require minimal additional expenses. To better demonstrate its generalization capacity across different datasets and tasks, we conduct more experiments and analyses.
We define H-RAD pre-trained with Spec-Bench as the experimental module, referred to SpecBranch (Spec). As shown in Table 3, our method performs exceptionally well across six different sub-tasks of Spee-Bench, highlighting H-RAD's strong cross-task performance.
Additionally, we transfer pre-trained H-RAD (Spec) directly to HvaanEval, GSM8K, and CNN/DM and compare with the original SpecBranch trained on specific datasets. As shown in Table 8, we see little performance drop of less than 10%, with an average decrease of only 5 %, yet it still outperforms the baseline PEARL, showcasing H-RAD's generalization and robustness.
Models
Methods
HumanEval
GSMBK
CNN/DM
Speed (tokensss)
Avg.
M
Speedup
M
Speedup
Speedup
Speedup
PEARL
3,11
2
2,83
1.61
2,89
1.68
$3.31
1.77
SpeeBranch
3,69
2.47
3,29
1.95
3,21
1.89
62.57
2,10x
Vicuna
SpecBranch(Spec)
3
2.31 (6.48%)
3.13
(.88x)8)22%)
3.15
1 (81X(4.24%)
58.81
1,97 (6.2
PEARL
17.28
3.75
14.33
3.35
7,51
3.0
24.03
x
SpeeBranch
21.74
4.02X
18.08
3.67
9.41
3,37x
26.27
3.69
LLaMA-3.1
Speceranch(ppee)
20.53
(2.199)
17.38
3.48
9.15
(5)
25.03
3.52 (4.7119)
Table 8: Cross-Tass Generalization of SpecBranch (H-RAD pre-trained with Spec-Bench).
This generalization capability can be partially attributed to the hybrid prediction of H-RAD, where intermediate token results determined by confidence, rather than relying on dataset distributions. Thus, H-RAD achieves better generalization than a purely explicit method. Furthermore, the features of the target model provide guidance for token acceptance, which influenced more by the model pair. Across different tasks, the predictive effectiveness of these features remains highly consistent, demonstrating that H-RAD exhibits strong generalization across target tasks or domains for specific model pairs. Online training of H-RAD is a promising future direction since H-RAD can learn in real-time from historical draft lengths and target model features, thereby enhancing its generalization capacity in real-world deployments.
F
MORE EXPERIMENTAL RESULTS
F.1 MORE COMPREHENSIVE COMPARISONS WITH THE TREE- BASED METHODS
To further elucidate the differences between method and tree-based methods, conduct addi- tional comparisons as described the following.
Compared to training-required tree-based methods:
Since training-requee required methods such as EAGLE (Li et al., 2024a) incur substantial training costs, is unfair to directly compare with EAGLE since method is model training-free free and such cost is well-recognized in the field. To further highlight the efficiency of SpecBranch, we provide a detailed breakdown of the draft model and predictor training costs (refer Table ?? for additional details).
We compare the draft model and predictor training costs of SpecBranch with three representative training-required methods: EAGLE, Medusa (Cai al., 2024) and Kangaroo (Liu et al., 2024a). The
21

---

Published as a conference paper at ICLR 2026
Method
Training Cost Details
Model:
1-2 days (LLaMA-2-13B, 8 RTX 3090 GPUs)
EAGLE
Predictor: None
Model: hours (Vicuna 7B, NVIDIA A100-PCIE GPU)
Medusa
Predictor: None
Model: 24 hours (Vicuna-7B, 8 NVIDIA V100-PCIE GPUs)
Kangaroo
Predictor: None
Model:
None
SpecBranch
Predictor: Σ 5 mins (1 NVIDIA A100-PCIE GPU)
Table 9: Comparison of draft model and predictor training costs.
latter two also require tome-intensive fie-tuning on large datasets. In contrast, the draft model of SpecBranch does not require training. H-RAD only involves training a lightweight MLP predictor that takes few minutes on single NVIDIA A100 GPU and we can see the training cost is x lower than EAGLE.
To further highlight SpecBranch's s exceptional performance in terms of latency, we provide key metries including the mean accepted tokens (M) and speedup, for comparison between SpecBranch and the EAGLE family LLaMA-2 Chat 7B&7OB.
HumanEval
GSM8K
CNN/DM
Speed (tokens/s)
Avg.
Models
Methods
M
Speedup
M
Speedup
M
Speedup
Speedup
EAGLE
4.45
3.51
3.97
3.09
3.78
2,98
22.35
3.20
EAGLE-2
5.46
3.78 x
4.49
3.52 x
4.98
3.48
25.13
3,59 x
LLaMA-2
PEARL
15.34
3.21
13.29
3.07 x
12.98
2.96x
21.56
3.08 x
SpecBranch
18.27
3.59X
15.98
3.37
14.02
3.23x
23.73
3.39
Table 10: Comparison with training-required tree methods on LLaMA-2 7B&70B.
As shown in the Table 10, SpecBranch demonstrates an impressive speedup of 3.59x, surpassing EAGLE's 3.51 x. While it is slightly behind EAGLE-2 (Li et al., 2024b) 's 3.78 the performance closeness already indicates the significant potential of SpecBranch's 's branch-parallel framework and the H-RAD predictor.
Additionally, H-RAD is an independent component that can be "plug and play" into EAGLE. While EAGLE-2 dynamically adjusts the tree width through token confidence, H-RAD further optimizes the tree depth. We also point out that the parallel paradigm can be combined with EAGLE. For example, we can train 4-yayer EAGLE model with early-exit capability, enabling dynamic adjustment of the draft model's layers based on context, using parameters from {1, 2, 3, 4) layers. By combining this early-exit, multi-layer draft model with the target model for parallel prediction, we can form new branching parallel framework.
HumanEval
GSMSK
CNN/DM
Speed (tokens/s)
Avg.
Models
Methods
Speedup
M
Speedup
Speedup
Speedup
REST
1.97
1.78x
1.65
1.46x
1.68
1.57x
10.99
2.28 x
Ourohoros
5.16
2.10x
5.96
2.58
3.28
1.57
14.60
2.08 x
LLaMA-2
SWIFT
4.27
1.56
2.99
1,43 x
3.87
1.45 x
10.15
1.48 x
PEARL
15.34
3.21
13.29
3,07 X
12.98
2,96x
21.56
3.08 x
SpecBranch
18.27
3.59
15.98
3.37 x
3,23
Table 11: Comparison to model training-free free tree methods on LLaMA-2 7B&70B.
Compared to model training-free tree methods:
To justify SpecBranch against tree-structure methods, we have also conducted more experiments of the recent model training-free methods, such as REST (He et al., 2023) (retrieval tree structure
22

---

Published as a conference paper at ICLR 2026
self-SD), Ouroboros (Zhao et al., 2024) (lookahead tree structure), and SWIFT (Xia et al., 2024a) (layer-skip tree structure self-SD), evaluated on Spec-Beme Bench using LLaMA-2 Chat 7B&70B.
The comparisons in the Table 11 show that SpecBranch outperforms other tree-structure methods. Moreover, the parallel framework can orthogonally combined with these methods, offering significant future potential.
F.2 EVALUATION RESULTS OF VARYING kmar
The number of branches k is crucial hyperparameter, which is discussed in Section 6.2 Resource Consumption. Fig. 7 demonstrates that memory consumption for parallel branches increases only slightly with k ranging from to 6. Furthermore, we perform more discussion and new experiments to report how varying kmaz affects overall speedup and rollback rate (RB). As shown in Table 12, we conduct SpecBranch experiments on HumanEval for Vicuna 68M& 13B and LLaMA-3.1 &B&70B under varying kmaz.
Aan
Aisa
Raux
12
Ausa
18
Model
RB
Speedup
Speedup
RB
Speedup
RB
Speeduu
RB
Speodup
RB
Speedup
Vieuna 68M&13B
87.56%
x.05x
62.13%
2.16x
48.02%
2.28x
45.14%
2.36x
39.60%
2.47
38.92%
2/48x
L.MMA-311 8B&
19.63%
3.97x
Table 12: Performance comparison under different kmax values.
We observe that kmax increases, SpecBranch initially exhibits rapid acceleration, then slows down as speedup diminishes. Conversely, the rollback rate behaves in an opposite manner. When kmaz = 1, there is only one parallel branch. As the number of branches increases, more candidate branching points are available, improving the acceptance rate of unconfident tokens. However, this comes at the cost of higher memory consumption.
(1) For poorly aligned model pairs, the lower token acceptance rate necessitates more branches to increase the acceptance of unconfident tokens, thereby reducing rollback and improving parallel efficiency to achieve trade-off. Accordingly, a lightweight draft model can handle more memory, enabling more branches be processed in parallel.
(2) For well-aligned model pairs, the improved alignment between the target and draft models mitigates the impact of rollback, meaning that smaller kmaz sufficient to achieve the desired trade-off.
F.3 LATENCY STABILITY ACROSS SEQUENCES
In principle, for SpecBranch or any other SD methods, latency is highly correlated with acceptance. We define the acceptance for each iteration as the number of accepted tokens that round divided by (draft length one iteration). In practical scenarios, the acceptance for each iteration varies significantly, as shown in the Appendix E9. This is why finding an optimal draft length is challenging and only dynamic draft lengths can effectively enable acceleration.
However, when we shift the time scale from iterations to requests, we find that the average acceptance rate for each request is quite consistent, with being more dependent on model alignment. As shown in the Table 13 (HumanEval with LLaMA 68M&7B), when we treat a request as the unit of measurement, the latency of SpecBranch remains quite stable.
Method
Response Time per Request (s)
Std. Dev.
2
4
5
6
8
10
(n=100)
PEARL
4.55
4.80
5.21
4.92
4.74
5.90
5.58
4.98
5.77
6.04
0.589
SpecBranch
4.10
4.31
4.67
4.39
4.28
5.37
5.00
4.44
5.10
5.02
0.423
Table Response time and standard deviation for different methods across requests
23

---

Published as a conference paper at ICLR 2026
Compared to PEARL, by using H-RAD efficiently predict unconfident tokens, we reduce token rollback, leading to better stability in latency. Specifically, the standard deviation for 100 requests is only 0.423, much lower than 0.589 of PEARL.
F.4 TIME AND ENERGY CONSUMPTION
We have provided some evaluation results of time and energy consumption in Section 6.2. In this section, we conduct more experiments on the time consumption of various model pairs on HumanEval while we further test the energy consumption of various model pairs on HumanEval and GSM8K.
shown in Table 14, regardless of model size, the time spent on H-RAD prediction and communication between multiple GPUs is almost negligible compared to the total inference time for a single step. This indicates that the H-RAD module retains its lightweight nature with minimum resource consumption and the inter-GPU communications have low operational overhead as well. On the other hand, the time span for the draft stage and the target model verification stage is nearly equal. This is consistent with our prior results that SpecBranch effectively implements parallelism between these two stages to alleviate the mutual waiting bubbles.
Time consumption
Modules
LLaMA 68M&7B
Vicuna 63M Deepseek 1.3833B LLaMA-3.1 8B&70B
H-RAD Predict
0.26 ms
0.27 ms
0.31 ms
0.28 ms
Communication
0.21 ms
0.31 ms
0.26 ms
0.26 ms
Draft Stage
20.8
30.9 ms
58.3 ms
125.1 ms
Verification Stage
21.7
31.4 ms
56.1
128.0 ms
Table 14: Time cost of each module (per step) on the HumanEval dataset.
Energy consumption We use the NVIDIA Data Center GPU Manager (DCGM) toolkit to monitor the real-time power consumption and collect relevant traces. Energy consumption is calculated by multiplying the average power by the total inference time over the entire benchmark. The results are provided Table 15 and Table 16.
For poorly aligned models (LLaMA, Vicuna), SpecBranch with its rollback-awarg dynamic draft length through H-RAD, significantly reduces redundant tokens, thereby lowering energy consumption compared to PEARL and SpS. Unlike PEARL, which pre-verifies only the first token using the target model in parallel, SpecBranch utilizes H-RAD to predict all tokens during the draft stage. This approach reduces the number of forward passes required by the target model. For L.LaMA 68M&7B on HumanEval, SpecBranch reduces the target model forward passes to 32,285, compared to 44,949 in PEARL, resulting in lower energy consumption for target model inference. For better-aligned models (Deepseek, LLaMA-3.1), where token rollback is significantly reduced, parallel efficiency improves considerably. While energy savings less pronounced, SpecBranch still outperforms PEARL.
Theoretically, Parallel SD incurs negligible energy consumption compared Vanilla SD. The speedup achieved through parallelism offsets the additional power consumption caused by extra forward passes in the target model. However, the parallelism and communication overhead still contribute some energy consumption in real-world deployments. Overall, SpecBranch, with its hybrid rollback-aware draft structure, significantly optimizes energy consumption compared PEARL. Notably, for poorly aligned models, it outperforms SD in terms of energy efficiency.
Methods
LLaMA 68M&7B
Vicuna 68M& 13B
Deepseek 1.3&333B
LLaMA-3.1 SB&
SpS PEARL
KJ
KJ
565 KJ
1021KJ
287 KJ
445 KJ
631 KJ
1231 KJ
SpecBranch
156 KJ
251 KJJ
582 KJ
1092 KJ
Table 15: Energy cost of SpecBranch and baseline methods on the HumanEval dataset.
F.5 MORE EVALUATION RESULTS ON THE SPEC-BENCH BENCHMARK
As illustrated in Section 6, we provide more evaluation results of SpecBranch in Table F.5 with both LLaMA 68M&7B and Deepseek 1.3833B Spec-Bench. Notably, LLaMMA 3.1 is more
24

---

Published as a conference paper at ICLR 2026
Methods
LLaMA 68M&7B
Vicuna 68M& 13B
Deepseek 1.3833B
LLaMA-3.1 &B&
SpS
146 KJ
250 KJ
393 KJ
741 KJ
PEARL
193.6 KJ
295 KJ
428 KJ
901 KJ
SpecBranch
128 KJ
178 KJJ
395 K.J
791 KJ
Table 16: Energy cost of SpecBranch and baseline methods on the GSM8K dataset.
recent LLM version that requires the transformer version to be greater than 4.43.0. Due this incompatibility, we cannot reproduce the results of baseline Lookahead Decoding as described before.
MT Bench
QA
Sum
Math
RAG Speedup
Trans
Models
Methods
M
Speedup
M
Speedup
M
Speeduup
Speeduup
M
Speedup
Avg.
SpS
1.77x
4.43
2.57x
2.62
1.51x
2.50x
1.28x
2.90x
x,09x
AdaEDL
2.74
1,86x
4.25
2.59
2.56
1.54x
4.83
2.43x
2.27
1.39x
4.93
2.96x
2.13x
LLaMA
Lookahead
1.58
1.31x
1.62
1.34x
1.59
1.32x
1.73
1
1.31
1.15x
1.36
1.23x
1,31x
PEARL
2.83
2,14x
5.98
3.15x
2.79
1.85x
7.58
3,37x
2.48
1.43x
7.65
4.03x
2,66x
SpecBranch
3.01
2.34%
6.57
3,53x
2.87
1.76x
8.57
3.77x
1.64%
8.57
4.32x
2.89x
SpS
2.63
1.74x
2.47
1.64x
2.58
1.70x
2.37
1.55x
2.47
1.56x
2.57
1.65x
1.64x
AdaEDr.
1.80x
1.72%
1.57%
1.75x
x.69x
Vicuna
Lonkabead
1.56
1.31x
1.41
1.23x
1.49
1.25%
1.71
1.46X
1,38
1.15x
1.32
1.10x
1.25x
1.78x
2.45
1.64%
2.78
1.83%
1.66%
2.05%
1.77x
SpecBranch
3.11
2.09
2.67
1.83
2.72
1.78%
2.86
1.89
2.83
1.86X
3.32
2.30x
1.96x
SpS
3.99
2,03x
4.02
2.12%
4.08
2,02x
3.93
2.06x
3.95
1.94%
4.37
2.21x
2.06x
AdaEDL
3.54
2.26x
3.81
2.33x
3,74
2.03%
3.81
2.28x
3.69
2.03x
4.12
2.38x
2.22x
Deepseek
Lookahead
x.51x
1.81
1.64x
1.59
1.42%
1.75
1.53x
1.32x
1,50x
1.49x
PEARL
6.60
2.77
5.15
2.91
5.81
2.68x
7.14
2.79x
5.77
2.65x
6.20
3.09x
2
SpecBranch
3.02
2.87
6.67
2.95
3.21
2.76x
3.28x
3.02x
SpS
4.67
2.31x
4.57
2.27
5.09
1.98
5.01
2.44x
5.08
2.02
5.52
2.57x
2.27x
AdaEDL
4.31
2.43x
4.23
2.30%
4,83
2.05%
4.94
2,46x
4.86
2.13x
5,24
2.65x
2.34x
LLaMA-3.1
Lookahead
PEARL
8.46
2,96x
8.37
3,27
9.10
3.32%
12.53
3.39x
8.35
3.41x
12.59
4.22X
3.43x
SpecBranch
10.85
3.24%
10.59
3
11.40
3,63x
15.76
3.78x
9.16
3.40X
16.64
4.51x
3,67x
Table 17: More evaluation results on Spec-Bench. "_" indicate incompatibility: baseline implementa- tions (transformers = 4.36.2) and LLaMA 3.1's dependency on 2 4.43.0.
We draw several interesting findings from Table F.5: 1) SpecBranch demonstrates consistent speedups across six subtasks, regardless of the model alignment quality; 2) In particular, the LLaMA 68M&7B model combination performs exceptionally well, due to the capacity of the LLaMA model, which excels in the Math, QA, and Translation tasks, with an average accepted length even surpassing DeepSeck. However, there are significant performance variations across other tasks. This suggests that the aligmment between the small draft models and the large target model may vary across tasks, leading to highly task-specific effects. These findings also indicate incorporate multiple draft models for speculative sampling in future research. Different draft models can be tailored to specific tasks under the Mixture-of oftture-ff-Experts (MoE) principles to further enhance inference efficiency.
F.6 MORE EVALUATION RESULTS OF ROLLBACK RATIO
As discussed in Section 6. rollback is an important metric for the parallel efficiency. Here, we provide more evaluation results on HumanEval, GSM8K, CNN/DM, and Spec-Bench, as shown Fig. 11. The results demonstrate that SpecBranch achieves significantly lower rollback ratios across various datasets and subtasks compared to other methods. This is consistent with our prior justification that the H-RAD module plays an essential role in mitigating rollback - the additional evaluations in Fig. 11 further validate the generalization capacity of H-RAD in different tasks. It is worth mentioning that for poorly aligned model combinations (e.g.. Vicuna, LLaMA), SpecBranch reduces the rollback ratio by nearly 50% compared to PEARL. Even for better-aligned model pairs (e.g., Deepseek, LLaMA-3.1), it achieves about 10% reduction. These results indicate the potential of SpecBranch in essource-constraiin The proposed framework can reduce the percentage of rollbacks as well as save computation/cnerg) resources in the long run.
environments where the draft model sizes typically restrained.
25

---

Published as a conference paper at ICLR 2026
SpS
AdaEDL
Lookahead
PEARL
SpecBranch
(a) HumanEval
(b) GSM8K
(c) CNN/DM
80
850
80
Rollback Rate (%)
Rollback Rale (%)
Rollback Rate (%)
60
40
20
20
20
Vicuna
Deepseek
LLaMA
LLaMA-3.1
Vicuna Deepseek LLaMA LLaMA-3.1
Vicuna Deegseek LLaMA LLaMA-3.1
(d) MT Bench
(e) QA
f Summarization
80
80
80
Rollbakk Rate (%)
Rollback Rale (%)
Rollback Rate (%)
40
40
20
20
Vicuna
Deepseek
LLaMA
LLaMA-3.1
Vicuna
Deepseek
LLaMA LLaMA-3.1
Vicuna Deepseek LLaMA LLaMA-3.1
(g) Math
(h) RAG
Translation
80
80
80
Rollback Rate (%)
Rollback (%)
Rollback Rate (%)
40
40
20
20
Vicuna
Deepseek LLaMA
LLaMA-3.1
Vicuna Deepseek LLaMA LLaMA-3.1
Vicuna Deepseek LLaMA LLaMA-3.1
Figure 11: Comparison of Rollback Rates HumanEval, GSM8K, CNN/DM, Spec-Bench for different model combinations.
F.7
MORE RESULTT TOKEN DISTRIBUTION
Recall that in Section 4.1 and Fig. 1(b), we introduce truncated geometric distribution (Leviathan ct al., 2show( in Fig. 1(b),
P(X (1- t T[k: <) a² T(k = 7),
(12)
where a the probability full acceptance and a the probability of rollback. To validate this, present additional token distribution results for Vicuna 68M& 13B and Deepseek 1.3B&33B on HumanEval and GSM8K in Figs. 12 and 13. These results show that the token acceptance distribution closely follows the truncated geometric distribution, which is consistent with our prior statements a bimodal phenomenon extracted from the target model features. This has laid the foundation for the H-RAD module to perform token length predictions with high fidelity.
In particular, Fig. 12 shows that for poorly aligned models, the acceptance rate a is low, and the truncated geometric distribution peaks at All Ael-Reject. In this scenario, the rollback in SD is exacerbated, particularly in Parallel SD. since it only achieves parallelism under the All-Accept condition. Given the low proportion of All-Accept in this token distribution, the efficiency of parallelism is significantly
26

---

Published as a conference paper at ICLR 2026
[8] y= 4 on HumanEval
(b) y on HumanEval
(c) y = 16 on HumanEval
Normalized Disribution
Aroept-2nn beng
Reject ler))
Normalized Disiribulion
0.40
All Reje: ped:
Mikooe (mn)
Normalized Distribution
0.40
Reject Seng)
Aoant (em9g)
Errpice Diatt doon
Emperrall Dierrution
Errppioa Daith
0,00
Thacredod sobotion
6,30
Theorskel buttin
0.20
0.00
0.20
0.10
0.10
0.10
0.00
0.00
1 2
8
7
0.00
5 5 8 10 1213141516
Accepted tokens per iteration
Accepted tokens per iteration
Accepted tokens per iteration
(d) y on GSMBK
(e) y on GSM8K
(f) = 16 on GSMAK
Normalzzed Disttbution
0.40
Acrepl (e-4)
Reject (mn))
Normalzed Disirbbttin
Alcooot (-2)
All O
Nomallzed Distriuttion
0.40
Reject (ent)
Actezt (o-1)
Eripital Dik iituen
Emprial Dasu
Sreprical Oat Enkibtuion
Delttuatinn
0,30
0.30
0.20
0.00
0.20
0.10
0:10
0.10
4.5%
2
1
2
5
8
0.00
2 5 6 7 9 10 12131411516
Accepted tokens per iteration
Accepted tokens per Iteration
Accepted tokens per iteration
Figure 12: More evaluation results demonstrate that the distribution of accepted tokens generally follows truncated geometric distribution of different token length 7 of Vicuna 68M& 13B.
(a) - 4 on HumanEval
(b) y on HumanEval
(c) y 16 on HumanEval
Royot (wi))
All Rejoct (enn)
Al Rejoct (rid)
Nornalized Distributon
ABAiczppp (m4)
Nornelized Redieeieed Distribullin
Accest (er))
Normalized Distribution
Allecapp (e+10)
EmpircalD
Errpecal Dusibuuioon
Empiical
0.40
0.40
0.30
0.40
0.80
0.30
0.20
0.20
0.10
0:10
0.10
0.00
0.00
1
2
5
0
0.00
1 5 5
9 10 14
Accepted tokens per iteration
Accepled tokens per iteration
Accepted tokens per iterationn
(4) y= on GSMBK
(e) y= on GSM8K
(i) y 16 on GSMSK
0.40
feed bed)
Rejjett (e-z)
Rejedt (e0i
Normalized Distribution
Normalized Distrauuton
Normalizad Distribution
(e+13) Empllcal
0.00
0.30
0.30
0.20
0.20
11.1%
60.0
0.10
0.10
0.10
22
114
2
5
0.00
2 5 7
1011 1213141516
Accepted takens per iteration
Accepted tokens per iteration
Accepted tokens per iteration
Figure 13: Distribution of accepted tokens generally follows truncated geometric distribution of different token length y of Deepseek 1.3B &33B.
impacted by rollback. This highlights our motivation to jointly consider rollback with parallelism for traditional SDD
For better-aligned models, Fig. 13 shows that the acceptance rate a is high and the truncated geometric distribution peaks at All-Accept. In this scenario, the rollback effects in SD are undermined, with the balance between rollback and parallelism leaning towards parallelism. As result, the parallel framework shows significant acceleration compared to vanilla SD, even approaching the theoretical speedup limit with aell-aligned models. However, it is evident that the draft length increases, the impact of rollback becomes neglegglie SpecBranch effectively balances the trade-off between these factors.
27

---

Published as a conference paper at ICLR 2026
F.8 EVALUATION RESULTS OF IMPLICIT DISTRIBUTION
As discussed in Sections 4.2 and 6.2, we primarily analyze the sensitivity of hyperparameters in the implicit methods. Here, we further examine the top-1 implicit distribution under different experimen- tal setups. We conduct extensive experiments with LLaMA 68M&7B, Deepseek &3BB and LLaMA-3.1 8B&70B on HumanEval, GSM8K, and CNN/DM under various temperature settings.
Task Sensitivity We first explore the task sensitivity of two main implicit values:
confidence and entropy, which measure confidence by max,, g(z,) (Du et al., 2024) and entropy as 1 (AH(2;) (Agrawal et al., 2024) against pre-determined thresholds e. Fig. 14 illustrates that the implicit values have different distributions across tasks. In summarization tasks (CNN/DM), both the average accepted confidence (0.91) and entropy (0.77) are significantly higher than in other tasks. Mcanwhile, the rejected implicit values also vary notably (0.26 to 0.45), especially for entropy in summarization tasks. This indicates that the implicit distribution is highly task-sensitive, making the selection of the stop threshold static and finding optimal value difficult.
HumanEval-Confideer
GSMBK-Confidence
CNNIDM-Confderce
1.0
1.0
1.0
0.8
0.8
0.8
Implot
0.6
Implet
0.6
Implat
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
Calegory (X-Vllue)
Calegory (X-Value)
Calegory (X-Value)
HuntonvallEEtropy
GSMeK-Enkropy
CNNDM-EEtropy
1.00
1.00
0.76
yalue
0.76
Impliit Value
0.8
Imgice
0.50
0.50
0.4
0.25
0.25
0.00
0.00
0
Calegoory (X-Valae)
Calegory (X-Value)
Calegory (X-Vllue)
(a) Distribution Analysis
Figure 14: The top-1 implicit values (Confidence) distribution of LLaMA 68M&7B on HumanEval, GSM8k and CNN/DM with temperature = 1. Category 0 denotes the top-1 values of rejected draft tokens while denotes the corresponding values of accepted tokens.
On the other hand, we observe that compared to entropy, confidence has a clearer and more distinct distribution for accepted and rejected tokens. This is why confidence is chosen for H-RAD, as provides higher fidelity. Additionally, we note that around 0.5, both confidence and entropy show considerable overlap between accepted and rejected values, which indicates a key limitation of the implicit methods.
Model Sensitivity The distribution of entropy is less effective than that of confidence, so we further test the model sensitivity of implicit methods (confidence) on three different model sizes: LLaMA 68M& 7B, Deepseek 1.3B&33B, and LLaMA-3.1 8B&70B. As shown in Fig. 15, the average accepted confidence of better-aligned models (0.98) is higher than that of poorly aligned models (0.79), and the same holds for rejected confidence (0.27 against 0.11). This demonstrates that the confidence distribution is sensitive to model pairs, with lower average confidence poorly aligned models, resulting in higher rate of rollback.
Temperature Sensitivity Temperature plays an important role in the draft model's sampling process. Thus, we conduct additional analysis on the temperature parameter. As shown in Fig. 16. we observe a sharp variation in the confidence distribution with temperature, especially for rejected tokens. When temperature 1, the draft model generates tokens with higher randomness, leading to a more distinct and separated confidence distribution. At temperature = 0.7, the average rejected confidence rises to 0.55, overlapping more with accepted confidence. When temperature 0.2, the randomness of the draft model's s sampling decreases, causing the rejected and accepted confidence distributions to overlap, making it difficult to distinguish early stop tokens. From the above,
28

---

Published as a conference paper at ICLR 2026
Rumnaneva-Coofdeerce (LLaMA)
CannanEEal--offfdeeee (Deepseek)
HumnanEva-Confideeee (LLaMA-3.1)
1.00
0.76
0.75
Implott valua
0.75
0.50
0.50
0.50
0.00
0.00
0.00
Category (X-Value)
Category (X-Value)
Category (X-Valuu)
(a) Confidence Distribution Analysis
Figure 15: The top-1 implicit values (Confidence) distribution of LLaMA 68M&7B, Deepseek 1.3B&33B, and LLaMA-3,1 8B&70B on HumanEval with temperature 1. Category denotes top-1 values of rejected draft tokens, while 1 denotes the corresponding values of accepted tokens.
conclude that higher temperatures increase randomness and allow the confidence distribution to better reflect whether draft tokens are accepted by the target model or not. Notably, SpecBranch uses temperature 1 for top-k sampling and confidence selection, optimizing the use of implicit values for better representation.
HumanEval-Con (temperature=02
HlumanEvalCConfddeen Confidence (lemperature=O..
HumanEval--onlfdene (lemperature* 1.0)
1.00
1.00
1.00
Implict value
0.75
implett
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
Calegory Value)
Calegory (X-Vakue)
Calegory (X Value)
(a) Confidence Distribution Analysis
Figure 16: The top-1 implicit values (Confidence,, Entropy) distribution of LLaMA 68M&7B HumanEval with temperature = 0.2, 0.5 and 1.
Based on the above discussion, we summarize the advantages of H-RAD over implicit methods. H- RAD significantly reduces the frequency implicit confidence calls the token-level by leveraging explicit target model features. This mitigates eтrоr accumulation in implicit confidence, desensitizes and reduces the dependency on thresholds. Moreover, H-RAD improves the accuracy of dynamic draft structures, resulting in significant reduction of rollback tokens. Specifically, our H-RAD predictor is invoked only once during each draft process, incurring far lower overhead compared to the loken-level implicit training predictors (Huang et al., 2024).
F.9 THE OPTIMAL DRAFT LENGTH
Variation Accepted Draft Length
stalic optimal draft draft longth
langh
Draft Longth
20
10
20
80
100
Iteration
Figure 17: Variation the optimal draft length over different iterations.
In Section 4.1, we have discussed the theoretical solution under rollback. However, this theoretical analysis only reflects the statistical properties rather than the runtime dynamics. As shown in Fig. 17, the actual optimal accepted draft length context-depen -deeede- and varies significantly across different iterations and the same phenomenon is observed in (Zhang et al., 2024; Liu al., 2024b). Hence,
29

---

Published as a conference paper at ICLR 2026
fixed draft length would result in substantial rollback and token waste. This highlights the need for adaptive control of y rather than relying on static configurations.
F.10 MORE ABLATION STUDY OF THE ORIGINAL MODULE AND THE HYPERPARAMETER SENSITIVITY
To better demonstrate the effect of branch resampling by comparing with the original module, conduct more experiments comparing vanilla SD and SpecBranch without H-RAD on the Spec-Bench of LLaMA 3.1 8B& As shown in Table 18, SpecBranch shows a significant speedup (3.36x) x) over SD (2.27x).
Component Analysis
Methods
MTBench
QA
Sum
Math
RAG
Trans
Avg.
M
Speedup
M
Speedup
M
Speedup
M
Speedup
M
Speedup
M
Speedup
M
Speedup
Vanilla SD
4.67
2.31x
4.57
2.27x
5.09
1,98x
5,01
2.44x
5,08
2.02x
5.52
2.57x
4.99
2.27x
SpecBranch w/o H-RAD 8.14
291x
8.21
3.17x
8.87
3,37
11.49
3.35x
7.98
3,24
11.73
4.15x
9.40
3,36
Table 18: Comparing the original module of the vanilla SD and SpecBranch without H-RAD.
Hyperparameter sensitivity To improve alignment with established practices, we conduct more experiments with the Branch-Parallel structure of LLaMA 68M&7B on HumanEval and the explicit method. The results shown in Table 19 confirm H-RAD's high acceleration efficiency and lower sensitivity to hyperparameters.
More Discussion We further clarify connections among SpecBranch's 's components: 1) Branch- Parallel is holistic structure that covers draft parallelism well as branch resampling. Single- sequence parallelism (c.g.. PEARL (Liu et al., 2024b)) is special case with kmax = 1. Detailed results on SpecBranch's 's performance across knax values shown in Appendix F.2. The wranch-parallel framework uses top-k sampling for unconfident tokens (vs. top-1 in single-sequence parallelism) to mitigate potential token rollbacks. 2) Isolating branches from SpecBranch would significantly stall parallel efficiency. Combining single-sequence parallelism and H-RAD alone cannot counteract rejections reduce rollbacks. Early termination of the draft model would make the target model bottleneck, leaving overall latency unchanged. In summary, SpecBranch is a unified framework with two synergistic modules. H-RAD first identifies unconfident tokens that are prone rollback. Then the branch-paree parallel strategy counteracts rejections while accelerating inference in parallel.
(onfideeeC))))))))e)
Implicit(Entropy)
Hybrid(H-RAD)
(xpticit(Feattue)
M
Speedup
M
Speedup
M
Speedup
M
Speedup
0.1
2.39
1.67
2.37
1.65
3.02
1.98
2.21
1.51x
0,2
2.63
1.76x
2.51
1.74x
3.24
2.04x
0.4
2.31
1.59 x
2.27
1.56 x
3.19
2.02
0.6
1.98
1.47
1.96
1.45
2.97
1.95
Table 19: The hyperparameter sensitivity on the branch-parallel structure.
G FURTHER ANALYSIS AND DISCUSSION
G.1 MEMORY CONSTRAINED SCENARIOS
We have discussed the memory consumption of SpeeBranch in Section 6.2. Here, we provide more detailed discussion about memory memoory-onsttriied scenarios.
Clarification of the Application Scenarios First, we clarify the application scenarios of SpecBranch: most existing methods operate on "draft-then-verff"" sequential
lequueeii execution, which limits the ability to fully utilize the computational resources available. In contrast, the main appli- cation scenarios for SpecBranch focus environments with sufficient computational resources to enable parallel frameworks, where serialized execution is not able to adequately leverage available resources.
30

---

Published as a conference paper at ICLR 2026
SpecBranch in Memory-Abundan Scenarios. Specifically, SpecBranch is well-suited for multi- GPU parallel scenarios in cloud environments with ample GPU resources, as well as for cloud-edge collaborative settings. In such scenarios, the draft and target models are deployed the edge and cloud devices, respectively. Additionally, it can he applied to heterogeneous processor environments, including CPU/GPU configurations heterogeneous GPUs. We foresee significant potential for SpecBranch in these scenarios, where both the draft and target models are independently deployed across processors. This avoids slowdown caused by memory contention. Our main experiments are conducted under these settings. Furthermore, integrating tensor parallelism (TP) with SpecBranch in these environments (Zhong al., 2024) can further enhance acceleration in the future.
Methods
MT Bench
QA
Summarization
Math
RAG
Translation
Avg.
Sps
x
2.12x
2.02x
2.06x
1,94x
2,21x
2.06x
SpecBranch
3.02
2.87x
2.95x
3,21x
2.76x
3,28x
3.02
SpecBranch(PP)
2.80x
2.56x
2.57x
2.93x
2.41x
3.02x
2.73x
Performance retain
92.57%
89.03%
89.78%
91.38%
87.21%
91.93%
89.76%
Table 20: Comparisons of Deepseek 1.3B&33B on the Spec- Bhec-Bench tasks with the proposed Spec- Bench in
scenarios.
SpecBranch in narios, where resource contention between the draft and target model may arise. To mitigate this, we consider a common real-world scenario using the A A00 40GB GPU. In this case, a large target model (33B) is deployed across two GPUs, while smaller draft model (1.3B) is deployed on one of these GPUs. In such scenario, we employ a modified pipeline parallelism (PP) version of SpecBranch to alleviate resource contention based on PEARL (Liu et al., 2024b). Specifically, the target model's computation is sequential across multiple GPUs: while the target model runs on GPU O, the draft model can operate parallel on GPU to generate the first [ tokens; when the target model progresses to GPU 1, the draft tokens generated on GPU I are transferred to the idle GPU O, which continues generating the remaining [ tokens, thereby avoiding memory contention. Although this PP approach introduces additional communication overhead, it effectively mitigates memory contention in this specific scenario. We conduct experiments with Deepseek 1.3B&33B on Spec-Bench and find that SpecBranch maintains approximately 90% of the reliable performance through PP, which outperforms the vanilla SD and auto-regressive decoding in memory-constrained settings. The results are shown in Table 20.
Scenarios. We also consider memory conntro-e sce-
Meanwhile, SpecBranch conducts extensive experiments on poorly aligned lightweight model combi- nations (68M&BB, 68M& 13B). The experimental results demonstrate that, under H-RAD's rollback mitigation, SpecBranch exhibits better adaptability and energy efficiency for small draft models, making it more suitable for deployment resource-constraaned environments.
Methods
MT Bench
QA
Summarization
Math
RAG
Translation
Avg.
(PpS)
1.74x
1.64x
1.70x
1.55x
1.56x
1.65x
1.64x
SpecBranch w/o branch
1.87
1.73x
1.75x
1.71x
1.73x
2.03x
1.81
Table 21: Comparisons of Vieuna 68M& 13B the Spec-Bench tasks with the proposed Spec-Bench in single GPU scenarios.
SpecBranch in Single GPU Scenarios. In the extreme resource-constrained scenarios, where only a single GPU is available for inference deployment, we can still apply the pipeline parallelism (PP) strategy by offloading the draft model to the CPU (DRAM), enabling heterogencous parallelism between the CPU and GPU. This approach left as future work for SpecBranch. If deployment is limited to a single GPU, SpecBranch degenerates to a non-parallel framework, as discussed in Section 6.2 under SpecBranch w/o branch. In this case, the H-RAD component operates independently of the parallel framework and can be seamlessly integrated with the existing traft-then-verify methods. In contrast, PEARL's pre-verify and post-verify stages degenerate to vanilla SD under extreme resource constraints. We conduct experiments with Vicuna 68M and 13B on Spec- Bench using single A100 GPU and find that SpecBranch w/ branch outperforms PEARL (vanilla SD) in these single-GPU scenarios. The results are shown in Table 21.
31

---

Published as a conference paper at ICLR 2026
G.2 TREE STRUCTURE AND TEMPORAL MISMATCH
In this section, we provide more detailed discussions about the tree structure and temporal mismatch to gain an in-depth understanding of SpecBranch (Section 5).
Tree-based Structure We primarily focus the vanilla tree structures. For example, SpecIn- fer (Miao et al., 2024) constructs a token tree using k independent sequences, a topology that is constrained by the expected number of tokens it accept, regardless of the tree size, shown in Fig. 18(a). However, this structure is dense, as top-k sampling is applied to each token, which generates additional sequences. In the case where the draft length is 7 and the tree size is k, the number of tokens for each round is given by k -_1 We observe that the number of tokens in tree structure grows exponentially with 7, causing KV-Cache storage to increase exponentially well. To address this challenge, EAGLE2 (Li et al., 2024b) and SEQUOIA (Chen et al., 2024) employ dynamic draft tree adjustments to prune unnecessary branches, resulting in sparse tree structure. Our SpecBranch adopts a similar sparse branch structure. The difference is that, unlike traditional tree structures, where each token generates branch, our approach utilizes H-RAD predict high-impact token positions and spawns sparse branch points that minimize speculative divergence. As illustrated in Fig. 18(b), we branch at S 7 positions, resulting in the number of tokens per round in the branch structure being k (k- - 1) (6 + 1), which is significantly smaller than in vanilla tree structures. In general, vanilla tree structures maintain the KV-Cache for the entire search tree, with space complexity of O(k?), whereas SpecBranch reduces the complexity to O -7) by sparsely targeting specific branches.
next round starting
next round starting
prefix
draft tokens
bronch points
(a) Vanilla Tree structures (Dense)
(b) Branch structures (Sparse)
Figure 18: Comparison of the vanilla tree structures (dense) and branch structures (sparse).
Moreover, trees require complex attention mask verification to process the entire structure in parallel, whereas approach only needs to verify the branch point. From practical perspective, branch structures offer advantages in terms of deployment complexity and resource efficiency. Additionally, for sparse tree structures, SpecBranch However, a key limitation is that tree structures are not well-suited for parallel architectures as illustrated next.
be easily integrated, which we leave for future work.
Temporal Mismatch As shown in Fig. 19, we observe that the parallel branch stage introduces temporal mismatch between drafting and verification. In the draft stage (a), the previous tokens have already been verified by the target model (except for the first round, where the target model lacks feature information). As result, H-RAD can utilize the historical target model feature pairs to predict the token generation length for the next round. In this stage, H-RAD uses features in priori fashion for prediction. However, in the branch stage (b), the draft model proactively generates speculative branches coneurrently with target model verification. Such parallclization of drafting and verification leads to a unique challenge that tokens from the previous round have not been fully verified by the target model before tokens are generated. This temporal mismatch prevents H-RAD from obtaining reliable features from the target model before new branches generate tokens.
For vanilla tree structures, temporal mismatch limits the verification process to only determining the first token of the next round. In other words, the next round starting (Fig. 18) tree structures is not just token, but rather (7 1) k tokens from partial tree. This means that during the parallel
32

---

Published as a conference paper at ICLR 2026
phase, the number of tokens in dense tree structure will reach (7 11)-1)-a - Kー which is ( times higher than the KV-Cache in the previous intermediate steps, further exacerbating the memory overhead. the other hand, branch structures only retain one branch after each verification step due to sparsity. This ensures that each new round starts afresh and mitigates the memory overhead.
Temporal mismatch is a unique "byproduct" of parallelism, which is unavoidable. Its mistreatment would delay the validation of draft tokens by the target model, potentially forming bottlenecks without specialized treatment such as PEARL. In our work, SpecBranch explicitly addresses such temporal mismatch by introducing adjustments to mitigate such asynchronous challenge. This is done via posterior approach for token selection in case the target model's 's features are unavailable. In the next Section, also explore priori methods, which resolve this issue and enable feature reuse similar to EAGLE.
prefix
draft tokens
branch points
farget model
H-RAD
tokens have not been fully validated!
no reliable featurel
(fuem
())))e))
5z
predict the aceepted length
the retained after verification
(a) Draft Stage (predict)
(b) Branch stage (posterior)
Figure 19: Comparison of the Draft and Branch Stages. The parallel branch stage introduces temporal mismatch between drafting and verification.
Posterior Approach To address this temporal mismatch, introduce posterior approach in Section 5.2. Once the previous tokens are fully verified by the target model, since 7 represents the speed ratio e between the draft and target models, by the time this happens, the branch has also completed token generation. Then, for the remaining branch V, we use the features (fe. et+1) from the current round input to H-RAD and select the retained tokens H. after the verification step. This posterior approach effectively resolves the temporal mismatch between verification and drafting, ensuring that H-RAD always uses the most up-to-date and relevant context. It also leverages parallelism, making the time loss from the posterior approach negligible. However, on the other hand, unlike in the draft stage, we cannot implement early stopping. While the parallelization mostly mitigates time impact due to bottlenecks at the target model, this introduces additional token waste. Moreover, this feature utilization method does not fully align with the current mainstream methods such EAGLE (Li et al., 2024a).
Priori Approach We further explore priori methods to unify the verification processes between the draft and branch stages. We leverage the temporal locality of transformer hidden states, where features from earlier time steps, though slightly outdated, still retain sufficient predictive power. Thus, H-RAD can proactively reuse features from the target model, specifi- cally (ft-1, et), to predict H4, whereas the posterior approach uses (fe, (x+1) as input. In other words, if cannot reuse fea- tures from the previous round's 's token, we rely on token features from the two rounds back, which exhibits a temporal decay, where the predictive capability of the target model features diminishes across multiple rounds.
94.45%
Pradicted Accuracy (%)
01.30%
83.06%
80
8
5A21%
History Feature Rounds
To quantify such decay, we conduct evaluation of LLaMA 68M&7B on HumanEval as shown in Fig. 20. It measures the prediction accuracy of the MLP used in H-RAD. We observe that as the distance and number of rounds increase transition- ing from (Je, e (Ji-s c++1-n) the contextual informa-
Figure 20: Predictive capability of the features progressively decay of LLaMA 68M&7B on HymanEval.
33

---

Published as a conference paper at ICLR 2026
tion and predictive capability of the features decay gradually. However, note that the features from the previous two rounds (ft-1. ee) still exhibit some predictive capacity. In scenarios where speed is not as critical, these stale features may serve a viable surrogate for the more recent features (Je, e(+1). This a priori approach better unifies the operations of the draft and branch stages. While the acceleration effect is not as significant as the posterior approach, the early stopping mechanism reduces the number of tokens generated. Additionally, this method can be combined with EAGLE (Li al., 2024a) reuse features with the draft model in future.
G.3 MORE DISCUSSIONS ABOUT OUR CONTRIBUTION
The key novelty/difference from the prior methods is that Prior Work such as PEARL is Rollback- Oblivious and Static, but our approach Rollback-Aware and Dynamic. The previous works do methods would continue processing those
not take any active measures as the rollback-oblivian "doomed tokens" and make the target model a bottleneck with useless computations, especially under poorly aligned models (shown in Fig. 5). In sharp contrast, our approach makes it preemptive, since we are hedging against possible failures. This is accomplished by:
Anticipates Failure: At points of high uncertainty, proactively spawn alternative futures via "branch resampling", i.e., we do not just hope single path to succeed; we plan for its potential failure.
Once the branch point is verified, we commit to the correct path and immediately terminate the invalid ones, ceasing all computation and freeing their KV-Cache. This is fundamental difference from the static pipeline of prior work.
Instantly Prunes Bad Paths:
Maintains Sparsity and Efficiency: Unlike tree-based methods (e.g.. SpecInfer) that create a dense. memory memory-intensive intensive tree structure, our branches are more sparse, created only at critical junctures. This reduces KV-Cache complexity substantially, making our parallel approach practical and scalable (Appendix G.2, Fig. 18).
Furthermore, the second novelty is the unification of implicit and explicit drafting strategies since they have their own pros and cons. The hybrid approach we proposed makes the drafting problem simpler and more tractable: 1) reducing the difficult N-class length prediction into a 3-class problem where features are separable; 2) using the explicit methods for what it is good at by identifying the high-confidence hard signals of the two extreme cases, but for ambiguous "uncertain" cases where the MLP falters, we fall back to the implicit confidence score as a "soft signal". In fact, it turns out that such simple and elegant solution is effective and would be able cut the rollback rates by up to 50% based on our experimental validations.
The key differences between the existing architectures are highlighted by Table 1 (See. 2), which states that SpecBranch and PEARL are the only two parallel drafting techniques, but SpecBranch takes step forward to make the process rollback-aware.
G.4 MORE DISCUSSIONS ABOUT TRAINING-FREE
SpecBranch is the first parallel framework with hybrid drafting structures that DOES NOT require additional training of draft models. Similar to the previous works, define training as extra procedure with a non-trivial cost. For example, REST (He et al., 2023) replaces the draft model with trained retrieval model, advancing the development of Self-SD. AdaDecode (Wei al., 2025) introduces lightweight language model (LM) heads at intermediate layers, achieving more efficient layer skip compared to truly "training- free" methods like SWIFT (Xia et al., 2024a). In contrast, the training cost of three-layer MLP for our predictor module is indeed lightweight.
G.5 MORE DISCUSSIONS ABOUT COMPUTE AND COMPLEXITY
Real-time Compute While branch resampling accelerates wall time, it also introduces additional computational overhead. However, current LLM inference acceleration is more constrained by memory bounds. In contrast to Autoregressive Decoding, Speculative Decoding mitigates this limitation by increasing computational resources. Therefore, increasing computation to achieve higher speedups shows significant potential. SpecBranch fully leverages hardware resources to provide higher speedups.
34

---

Published as a conference paper at ICLR 2026
On the other hand, SpecBranch uses H-RAD to strategically spawn branches, which reduces token rollback by 50%, and the additional energy consumption remains lower compared to PEARL. Note that we have considered the energy cost in this work (see Section 6.2 and Appendix F.4) which has not been unveiled by contemporary works and we believe this is essential for real-world production environments.
Complexity In fact, the additive complexity turns out to be worth it. This is because the perfor- mance gains outweigh the cost: our system as a whole deliver 1.8x 38x-4.52 4.5x speedups with 50% less rollback, while the cost can be easily justified: 1) H-RAD is lightweight 3-layer MLP (trained offline minutes on single A100) with minimal overhead overhead-predict latency (0.38% of total latency), requiring no additional draft-model training; 2) Branch resampling uses adaptive logic to auto-tune branch counts based on draft tokens confidence, that alleviates manual hyperparameter tuning. For services like ChatGPT, Google's ' Bard/Gemini, or large enterprise API, inference is massive operational expense. If the performance margin compared to existing works could translate directly into millions of dollars in savings GPU infrastructure and energy costs, rather than being barrier to widespread adoption. The implementation of our framework is not complex either as we will make it publicly available later. The core software code only contains about 500 lines of Python, in which each component is modularized and easy to adapt to other techniques.
G.6 GROUP SPECULATIVE DECODING AND HETEROGENEOUS DEVICES
Group SD within Single GPU Cluster In SpecBranch, we rigorously validate the acceleration effectiveness for a single draft target model pair. However, as Model-as-a-Servies (MaaS) becomes more prevalent, serving multiple model instances within cluster is increasingly important for maximizing resource utilization. Meanwhile, we plan extend our system to support simultaneous SD for multiple draft-target pairs with more than one draft model per GPU and high throughput. For example, in an 8x 100 80G configuration, we can deploy different target models (33B) on GPUs and up to 15 draft models (1.3B) on the remaining GPUs. These draft models can be paired with the same or different target models through advanced communication and network design.
Speculative Decoding Heterogeneous Devices During validation, we find that large models often require multiple high-end GPUs to fit into memory (e.g.. a 70B-parames parammmmtttrr model experiment requires at least 4 x A100 40G GPUs). While one solution discussed in the previous section multiple draft models in a single device, the prerequisite for large-memory devices remains unchanged. However, we discover that under the setting of SID, homogeneous deployment is unnecessary, i.e., the draft models do not need to be co-located on the same device as the target models. Thus, we present a heterogeneous structure that runs smaller draft models on consumer-grade GPUs (c.g., RTX 3090) and connects them over the network to target models on data-center GPUs (c.g., A100). The challenges on the system level, including synchronization, communication latency, and workload balancing, remain as future work for SpecBranch.
pack
G.7 DETAILED ALGORITHM PERIOD AND DISCUSSION OF SPECBRANCH
While SpecBranch shares the fundamental parallelism principle with PEARL, it transcends simple combination of "PEARL+ + MLP." Instead, it is a synergistic co-design of the lightweight H-RAD module within rollback-aware parallel framework. Unlike PEARL, which neglects the cost of token rollbacks, SpecBranch explicitly addresses this limitation through two core architectural innovations:
The H-RAD module does not merely passively judge tokens; actively governs when to branch, where to locate the branch point, and how many branches to spawn. This transforms the draft-verify pipeline from a static parallel scheme into a dynamic, ancertainty-ddaptive process.
Active Topology Control:
Rollback Awareness: By explicitly modeling rollback risks, SpecBranch restricts branching to points and verifies specific branch points rather than the entire tree. This allows the system to immediately terminate those "doomed branches" upon verification failure, reducing rollback tokens by up to 50% compared to PEARL.
The detailed workflow of the SpecBranch algorithm is described below:
35

---

Published as a conference paper at ICLR 2026
(1) Drafting Phase. Given context prefix X1:-11, the draft model generates sequence of candidate tokens. Simultaneously, H-RAD analyzes the multi-layer target features (cached from the previous decoding step) to assess the entropy of the current generation. H-RAD classifies the current position into three distinct regions: (i) all-accept (high confidence), (ii) all-reject (certain failure), or (iii) confidence-based seft region (uncertainty).
(2) Branch Point Selection, the generation falls into the soft region, H-RAD marks the current token 2y branch point, At this juncture, the drafting process splits:
The stable prefix X1:6-1 is immediately sent to the target model for verification.
Meanwhile, the draft model samples the top-k alternatives for 26 and continues drafting parallel branches. Crucially, all branches share the same VV-cache for the prefix X1:6-1 to minimize memory overhead.
(3) Prefix Verification. Executed in parallel with branch drafting, the target model verifies the prefix segment 11 If any token within this segment is rejected, the system identifies a critical failure. All pending parallel branches are immediately discarded, and the process falls back to standard speculative decoding resampling (similar to the rollback mechanism in PEARL) to ensure correctness.
(4) Branch Verification and Selection. If the prefix X1:6-1 is fully accepted, the target model computes the probability distribution p(2;) at the branch point. We then apply a Branch Speculative Sampling strategy (detailed in Appendix D) to the set of k branch candidates { utilizing both the target distribution p and the draft distribution q to ensure the final output matches the target distribution. As soon as candidate is accepted, SpecBranch retains the corresponding path and discards all other branches (along with their K KV-caches). Consequently, the target model only ever continues with single valid sequence, eliminating the need for complex
attention masks.
(5) Posterior Drafting. Upon selecting the retained branch, the system updates the KV-cache and target features. H-RAD is then invoked again to determine the next draft length and identify potential future branch points, restarting the iterative process.
G.8 EXTENDED ANALYSIS ON DRAFT-TARGET MODEL SIMILARITY
To comprehensively assess the generalization capability of SpecBranch, we conducted extensive experiments across a wide spectrum of draft target similarity levels. Our evaluation benchmark spans from weakly aligned pairs (e.g.. LLaMA-68M/7B, Vicuna-68M/133) to strongly aligned configu- rations (e.g., 1...Seek--133/33 LLaMA-3. 1-B...--188//BB) We observe that SpecBranch exhibits dynamic adaptation mechanism where distinct components dominate performance depending on the alignment regime:
Low-Similarity Regimes: In scenarios with weak alignment, the performance gain is primarily driven by the rollback-aware H-RAD mechanism. By effectively identifying and terminating invalid branches early, H-RAD significantly mitigates the latency penalty associated with frequent corrections. Empirically, this results in a reduction of rollback rates by approximately 50% compared to PEARL, ensuring stable parallelism even when the draft model provides lower- quality predictions.
Conversely, for well-aligned model pairs, the contribution of Branch Resampling becomes dominant. The naturally high acceptance rate allows the parallel branches to be validated successfully more often, thereby maximizing token throughput and fully leveraging the system's parallel capacity.
High-Similarrty Regimes:
These findings confirm that SpecBranch is not limited to specific model combinations but automati- cally adapts to the underlying model alignment, maintaining a consistent performance advantage over baselines across diverse inference conditions.
G.9 HIGH-BATCH PERFORMANCE ANALYSIS
High-Batch Scalability To bridge the gap towards industrial deployment, we are currently integrat- ing SpecBranch into the nano-PEARI. framework (Liu et al., 2024b), lightweight, high-coneurrency
36

---

Published as a conference paper at ICLR 2026
engine based nano-vllm, while conducting pilot studies with industrial partners. Addressing the critical requirement for high-throughput scenarios, we further compare SpecBranch against EAGLE3 (Li et al., 2025) the Qwen3-32B model across varying batch sizes (bs). As shown in Table 22, EAGLE3 exhibits severe performance degradation as concurrency increases, dropping to near-baseline speeds (~ 1.05x) bs 32. This bottleneck stems from its tree-based design, where the computational load scales linearly with tree size x batch size, rapidly saturating hardware arithmetic intensity. In contrast, SpecBranch mitigates this by selectively branching only on low- confidence tokens and delaying compute saturation via reduced kmax. Furthermore, by employing lightweight verification step that bypasses the complex tree attention masks inherent prior methods, SpecBranch demonstrates superior robustness in high-load environments.
While mainstream speculative decoding (SD) research primarily benchmarks at batch size 1, industrial deployments require high throughput. In high-batch regimes, the system bottleneck shifts from memory bandwidth (memory-bounde to arithmetic intensity (compute-bound). This transition poses fundamental challenge for the existing SD methods; as bs increases, the additional computation required for drafting and verification often saturates hardware resources, nullifying the sitie-of-theaasd tree-based methods like EAGLE3 is
speed gain of SD unfortunately. For instance, found to degrade to baseline speeds at bs > 8. To demonstrate SpecBranch's 's industrial viability, we conduct comprehensive high-batch evaluation.
Implementation Details We integrate SpecBranch into nano-PEARL, (Liu et al., 2024b), a lightweight, high-concurrency inference engine based nano-vllm. This framework supports advanced optmizations including PagedAttention, FlashAttention, and CUDA graphs
Hardware: Experiments are conducted on 3 NVIDIA A100-80G (NVLink) GPUs.
Configuration: We employ Tensor Parallelism (TP). For the draft model (Qwen3-0.6B), we set TP=1; for the target model (Qwen3-32B), we set TP=2.
Parameters: To balance computational load in high-coneurrency settings, SpecBranch is config- ured with reduced maximum branch count of kmax = 3. The EAGLE3 baseline utilizes TP=2 following its official implementation. All baselines run with CUDA graphs enabled.
Batch Size (Qwen3-32B)
Dataset
Methods
batch = 1
batch = 4
batch = 8
batch = 16
batch 32
EAGLE3
1.98 x
1.67 x 2.11x
1.34 x
1.13 x
1.05 x
HymanEval
SpecBranch
2.34x
1.97 X
1.74
1,55 x
EAGLE3
2.03 x
1.74X
1.43
1.15
1.07 x
GSM8K
SpecBranch
2.57 X
2.34X
2.06
1.88%
1.63
Table 22:
High-Batel Batch performance comparison on Qwen3-32B, SpecBranch maintains robust accel- eration under high concurrency (bs 32) compared EAGLE3.
Results Analysis As shown in Table 22, we observe a distinct divergence in performance scaling. EAGLE3's acceleration collapses as batch size increases, dropping to near-negligible levels (1.05x) at bs 32. In contrast, SpecBranch maintains robust acceleration, achieving 2 1.55x speedup even at bs = 32. This performance gap is driven by two critical design advantages:
Selective vs. Dense Drafting: EAGLE3 employs a static tree structure where the computational load scales multiplicatively with Tree Size x Batch Size, causing rapid hardware saturation. SpecBranch, conversely, utilizes an entropy-based selective strategy. By spawning parallel branches only at high-uneertainty tokens and capping kimxx SpecBranch effectively manages the total FLOPs, delaying the onset of the compute-bound regime.
Lightweight Verification: High-batch verification introduces significant latency for the target model. Tree-based methods suffer from the overhead processing complex tree attention masks across the entire batch. SpecBranch bypasses this bottleneck by validating only specific, sparse branch points. This streamlined verification process minimizes the pre-fill latency penalty, making inherently more scalable for high-thooughput environments.
37

---

Published as a conference paper at ICLR 2026
These results confirm that SpecBranch offers competitive, robust solution for industry-scale de- ployment, effectively bridging the gap between theoretical SD gains and practical high-coneurrency demands.
G.10 EXTENDED COMPARISON WITH EAGLE-3
While comparisons with EAGLE and EAGLE-2 are detailed in Appendix F.1, we extend our evaluation include the recently proposed EAGLE-3 (Li et al., 2025). Due EAGLE-3's lack official support for the LLaMA-2 architecture, we conduct these comparative experiments using the Qwen3-32B model. Both implementations utilize the Transformers-baat based architecture for a fair comparison.
HumanEval
GSM8K
CNN/DM
Speed
Avg.
Model
Methods
M
Speedup
M
Speedup
M
Speedup
(tok/s)
Speedup
EAGLE-3
2.88
2.16X
3.17
2.36
2.63
2.07
28.61 33.02
2.20 2.54
Qwen3
SpeeBranch
5.62
2.57 X
7.64
2.69 X
4.96
2.35X
Table 23:
Comparison with EAGLE-3 Qwen3-32BB SpecBranch consistently outperforms EAGLE- 3 across all datasets despite Qwen3's i large vocabulary size.
As shown in Table 23, SpecBranch consistently outperforms EAGLE-3 across all datasets. Notably, EAGLE-3 exhibits limited acceleration on the Qwen3 model. We attribute this performance bottleneck to Qwen3's extensive vocabulary size, which imposes significant computational overhead during the tree verification phase inherent to EAGLE's design. In contrast, SpecBranch's 's sparse branching strategy remains efficient in cage--occbullay regimes. We acknowledge that EAGLE-3's dedicated draft model is a significant but orthogonal design choice. SpecBranch currently utilizes standard draft model; however, this opens promising avenue for future work integrate on- policy online distillation, developing specialized draft models to further enhance the parallel H-RAD framework.
of
H
USE OF LLMS STATEMENT
We do any Large Language Models (LLMs) in paper writing.
38


Reviews:
[{"review_id": "Zj1FGaZRKw", "reviewer": "Reviewer_mdwL", "summary": "This paper presents a novel and well-executed study on improving draft model training for speculative decoding using structural re-parameterization. The method is innovative, empirically thorough, and practically significant.", "questions": "The results on LLaMA-2-13B suggest that Hybrid RepSpec performs better relative to Pure Linear as the target model grows. Do the authors have any results or intuition about how RepSpec would behave with very large target models(e.g., 70B+)? Would the Hybrid method become even more favorable?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The paper introduces structural re-parameterization—commonly used in CNNs—into speculative decoding for the first time. The idea is clever and directly addresses the core limitation of draft model capacity. The dual design of Pure Linear (no inference cost) and Hybrid (better performance with minimal cost) offers flexible solutions for different deployment needs.\n2 The authors evaluate RepSpec across multiple target models (LLaMA-3.1-8B, LLaMA-2-13B, Vicuna-7B), tasks (dialogue, code, math, etc.), and draft frameworks (EAGLE, Medusa, Hydra). The consistent improvements in acceptance length and speed-up demonstrate the generality and effectiveness of the method.\n3 The ablation studies are systematic and insightful.", "weaknesses": "1 While Appendix E attempts to explain why re-parameterization helps (gradient variance reduction, implicit spectral regularization), the arguments are intuitive rather than rigorous. A more formal analysis—e.g., connecting the training dynamics to a well-defined optimization objective—would strengthen the foundation.\n\n2 The paper compares RepSpec against standard training. However, it would be more compelling to compare against a baseline that directly increases the model size(e.g., adding more layers or hidden units) to match the parameter count of RepSpec during training. This would better isolate the benefit of the re-parameterization structure itself.\n\n3 The inference overhead of the Hybrid method is well documented, but the additional training cost(time, GPU hours) is only briefly mentioned. A quantitative comparison (e.g., how much longer RepSpec takes to train than the baseline) would help users assess the overall cost-benefit trade-off.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IbAZZulCWv", "reviewer": "Reviewer_him1", "summary": "This paper seeks to address the pipeline bubble that occurs during the verification step in speculative decoding. While previous research has looked at speculating in parallel to performing verification, they have done so by making simplistic assumptions about draft acceptance length. SpecBranch on the other hand uses a hybrid predictor to intelligently decide how to spend parallel speculation resources. The SpecBranch method leads to substantial speed ups on multiple benchmarks.", "questions": "None right now.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The writing and figures used throughout the paper are very clean.\n* The theoretical model of parallel speculative decoding is well motivated and provides a good model for understanding tradeoffs.\n* The idea of rollback aware branched parallelism is novel and addresses a large gap with current speculative decoding methods.\n* The empirical analysis presented is robust as the method is tested across a large range of tasks and across multiple model sizes and model families. Furthermore, the speedup that was achieved is substantial compared to most improvements to speculative decoding.", "weaknesses": "* The largest weakness is that the analysis is only presented for the batch size = 1 setting. While this makes sense as the low batch setting is most relevant for speculative decoding methods, it would be helpful to profile the scaling behavior as the batch size is increased.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CjwPNXNJ4O", "reviewer": "Reviewer_MCa5", "summary": "Based on PEARL's parallel framework, SpecBranch introduces the Hybrid Rollback-Aware Draft Structure (H-RAD) and ingeniously integrates the branch resampling method, enabling branch-parallel execution of speculative decoding (SD). This design effectively addresses the serialized waiting bottleneck and high rollback issue inherent in traditional SD methods. The authors rigorously validate and elaborate on these two core innovations through comprehensive theoretical analyses—including Theorem 1 on latency under rollback and the truncated geometric distribution of accepted tokens—and extensive experimental evaluations.", "questions": "The authors have conducted numerous analytical experiments, and the paper is quite complexly written. However, I believe the method in the paper is actually very simple—it merely adds an MLP on top of PEARL for judgment. I need the authors to explain the timing sequence of the entire process, which should be described most clearly in the main text, especially how the subsequent tokens of the branches are generated and verified.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tComprehensive figures and textual descriptions, clear structure, and detailed experiments.\n2.\tStarting from the weaknesses of PEARL—including its rollback-oblivious nature, static draft length, and inadequate handling of pre-verify/post-verify rollbacks—the authors explain the rationale behind proposing the Hybrid Rollback-Aware Draft Structure (H-RAD). They further validate the design of H-RAD as a three-class classification framework through experiments, and combine it with in-depth theoretical analyses (e.g., Theorem 1 on latency under rollback, truncated geometric distribution of accepted tokens) to develop the SpecBranch method. This approach achieves excellent speedup ratios among training-free speculative sampling methods.\n3.\tDetailed comparative analyses are conducted on hyperparameters (e.g., maximum branch number kmax, confidence threshold ϵ, feature layer count K) and core components (H-RAD, branch resampling) that influence the speedup ratio.\n4.\tThe paper’s analysis of the Rollback Rate (RB) clearly identifies the time-consuming bottlenecks of speculative decoding (SD) methods, and effectively demonstrates the superiority of SpecBranch in reducing rollback tokens (up to 50% reduction for poorly aligned model pairs).", "weaknesses": "1. There are several errors in the paper, such as the incorrect labeling of \"Lookahead\" in Figure 1(c) and the syntax issue with \"foreach\" at Line 668.\n2. The superiority of the proposed method lies in its training-free characteristic, yet its acceleration performance is not as excellent as that of training-required methods like EAGLE-3. Furthermore, the so-called \"training-free\" essentially relies on pre-existing draft-target model pairs. Additionally, during inference, this parallel training-free method consumes more GPU resources than EAGLE and poses greater challenges in deployment. Therefore, it is anticipated that the authors will provide deployment implementations on frameworks such as vLLM or integrate SpecBranch into EAGLE, delivering more practical solutions to the industry.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pAdk0M3Cr1", "reviewer": "Reviewer_oJJE", "summary": "The method introduces parallel speculative branches, adaptively chooses draft lengths, and reuses features of the target model to hedge against incorrect drafts and reduce rollback tokens.", "questions": "I suggest adding ablation experiments exploring branch length, number of speculative branches, draft vs. target model similarity, and rollback frequency under different task/benchmark settings.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The idea of modelling speculative decoding as analogous to branch prediction in CPUs is clever, and the introduction of parallel speculative branches is conceptually interesting.", "weaknesses": "It is not fully clear how generally applicable the method is across diverse architectures, tasks, or deployment settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)