Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
Authors: Jinze Li
Yixing Xu
Guanchen Li
Shuo Yang
Jinfeng Xu
Xuanwu Yin
Dong Li
Edith C. H. Ngai
Emad Barsoum
Venue: ICLR 2026 Poster (2026)
Keywords: Speculative Decoding
Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) achieve strong performance across diverse tasks but suffer from high inference latency due to their autoregressive generation. 
Speculative Decoding (SPD) mitigates this issue by verifying candidate tokens from a smaller draft model in parallel, yet its strict exact-match verification discards many semantically valid continuations.
We propose Training-Free Loosely Speculative Decoding (FLy), a novel method that loosens the rigid verification criterion by leveraging the target model’s own corrective behavior to judge whether a draft–target mismatch remains semantically valid. 
FLy introduces a two-tier mechanism: an entropy-level gate that identifies whether the current token allows multiple plausible alternatives or is nearly deterministic, and a token-level deferred window that distinguishes genuine errors from differently worded yet semantically correct variants. 
To further reduce latency, we design a multi-level acceleration strategy that accelerates not only the target model but also the drafter itself.
Owing to its training-free design, FLy composes seamlessly with arbitrary draft–target pairs and generalizes across models and domains without hyperparameter re-tuning. 
Experiments show that FLy preserves $\geq$99\% of the target model’s accuracy while achieving an average 2.81$\times$ speedup on Llama-3.1-70B-Instruct and 5.07$\times$ speedup on the 405B variant. Notably, on out-of-domain datasets, our method remains highly effective and outperforms the training-based method EAGLE-3 by 1.62$\times$. 
Our code is available at
https://github.com/AMD-AGI/FLy.

Published as a conference paper at ICLR 2026
TRAINING-FREE LOOSELY SPECULATIVE DECODING: ACCEPTING SEMANTICALLY CORRECT DRAFTS BEYOND EXACT MATCH
Jinze Li Yixing Xu', Guanchen Li', Shuo Jinfeng Xu², Xuanwu Yin', Dong Li,, Edith C.H. N Emad Barsoum! Advvnnced Micro Devices, Inc., Beijing, China
"Department of Electrical and Electronic Engineering, The University of Hong Kong (lijinze-hku, shuo yang, jinfeng]@ fyixing. xu, Xuanwu. Yin, d.li, emad. bamrsoumm@add.coo amd. com chrigai @eee. hku.
hku. hk
ABSTRACT
Large language models (LLMs) achieve strong performance across diverse tasks but suffer from high inference latency due to their autoregressive generation. Speculative Decoding (SPD) mitigates this issue by verifying candidate tokens parallel from smaller draft model, yet its strict exact-malch verification discards many semantically valid continuations. Moreover, existing training-bass based SPD methods often suffer from performance degradation tasks. To this end, we propose Training- Lousely Speculative Decoding (FLy), a novel method that loosens the rigid verification criterion by leveraging the target model's self-corrective behavior to judge whether a draft-target mismatch remains semantically valid. FLy introduces a two-tier mechanism: entropy- level gate that identifies whether the current token allows multiple plausible alter- natives or is nearly deterministic, and token-level deferred window that distin- guishes genuine eπrоrs from differently worded yet semantically correct variants. To further reduce latency, we design a multi-level acceleration strategy that ac- celerates not only the target model but also the drafter itself. Owing to its training- free design, FLy composes seamlessly with arbitrary draft-target pairs and gener- alizes across models and domains without hyperparameter re-tuning, Experiments show that FLy preserves 299 of the target model's accuracy while achieving average 2.81 speedup Llama-3.1 3.1-700B-Instruu and 5.07 speedup the 405B variant. Notably, on fective and outperforms the training- based method EAGLE-3 by 1.62x. Our code is available at http:://gtttub.oomAMM-/AIAII
out-tf-distribution (OOD)
datasets, our method remains highly ef-
INTRODUCTION
While Large Language Models (LLMMs) (Vaswani et al. 2017) have demonstrated impressive ca- pabilities (Dubey et al. 2024 Anil et al. 2023 DeepSeek-Al et al., 2025), their auto-regressive inference strategy entails substantial latency, which becomes more pronounced as model size seales. Speculative decoding (SPD) (Stem et al. 2018) has emerged as promising solution to acceler- ate generation without compromising quality. In SPD, a lightweight draft model proposes multiple candidate tokens sequentially, which the larger target model verifies in parallel, accepting those sistent with its own predictions. This procedure provably preserves the target model's distribution while delivering significant throughput gains.
However, standard SPD is fundamentally constrained by its exact-match rule: the target accepts a draft token only if it is identical its own generation. This rigid requirement forces the rejection of many plausible continuations, even those semantically aligned, thereby discarding useful tokens and limiting speedup. Recent work (Bachmann et al. 2025) shows that even high-quality drafts
"oorreponndngg chnggai @eee. .ku.hk.

---

Published as a conference paper at ICLR 2026
Llama-3.33 Batma--..-70--Insse
Llama-3. - I-ntte-nntttuet
Llama-3. 1amma-3.1-40555 1-0-331---B---tttue
7777
EAGLES3 (raining-haiod)
REST
Token Recycling
Speculative Sampling
Fly (Ours)
Spoclap
2h
L.b.
LIN
MGSM
NIAH
ACP
MGSM
NIAH
ACP
MGSM
NIAH
ACP
Figure Speedup on out-of-domain (OOD) datasets. Training-based method EAGLE-3 suffers significant degradation under OOD conditions. Our approach surpasses existing methods across datasets and models by accepting more semantically valid tokens, achieving SOTA performance.
(e.g., human-written text) achieve low acceptance under this scheme. As draft models continue to improve, such strict rejection becomes an increasingly inefficient bottleneck, highlighting the importance of more flexible verification strategies. Existing works have proposed "loose" variants of SPD that relax the strict verification rule. A representative example is JudgeDecoding (Bachmann al. 2025). which trains an auxiliary classifier to decide whether a draft token is contextually valid. Although effective, this approach requires carefully curated training data and training. incurring high annotation costs. Moreover, the supervisively trained classifier often fails to generalize across multiple domains or tasks, making the method brittle in cattof-ddittibution (OOD) settings.
To address these limitations, we propose Training-Free Loosely Speculative Decoding (FLy), which relaxes verification by accepting semantically correct drafts without additional training. The central insight is that LLMs tend to exhibit self-corrective behavior when conditioned on genuinely erroneous tokens, but not when faced with merely worded differently yer semantically valid alter- natives (Pan et al.] 2023] Bachmaan et al.] 2025). Building on this property, FLy leverages the target model's behavior distinguish harmful mismatches from semantically equivalent tinuations. Specifically, FLy introduces a two-tier mechanism for mismatch handling. The first component is entropy-level gate, which classifies the mismatch position either ambiguous (multiple valid alternatives) or deterministic (a single plausible token) based on the target model's token-level entropy. If the entropy falls below a threshold, indicating near-deterministic case such as a numerical calculation, the mismatch is immediately rejected. Conversely, if the entropy ex- ceeds the threshold, FLy activates a token-level deferred window spanning the next several tokens. Within this window, the mismatch is provisionally accepted. If another mismatch emerges, it signals the model's attempt to course-correcl an earlier error, and the initial token retroactively rejected. Otherwise, the token is deemed semantically valid continuation and retained.
By accepting semantically correct mismatches, the average number of accepted tokens (r) rises markedly. Thus, the drafter needs to propose larger set of tokens per round, which raises the drafter's generation cost to the point where it becomes non-negligible compared to common SPD methods (Cai et al., 2024 Li et al., 2025b). thereby diminishing overall speedups. To mitigate this problem, we propose a multi-level acceleration scheme that not only accelerates the target model, but also speeds up the drafter itself, preventing the drafting stage from becoming the dominant bottleneck as T grows.
Notably, FLy is plug-and-play method requiring no data collection training, and it is model- agnostic. A single drafter can accelerate different targets, and distinct drafters can be paired with the same target without retraining. Moreover, FLy inherently robust to distribution shifts, since it does not depend on any training data whose generalization may fail on OOD test data (cf. Fig- ure D. Experimental results demonstrate that FLy achieves an average 2.53x speedup on out-of- domain datasets (1.62x faster than EAGLE-3) and 2.69x x on in-domain datasets (1.05x faster than EAGL...22 with 70B-scale models, and an average of 5.07 speedup with 405B-scale models. In all circumstances, FLyy preserves 299% of target accuracy on 70B and 405B models.
METHOD
In this section, we first introduce the preliminaries of Speculative Decoding (SPD) in Section 2.1 Then, we propose Training-Fres Loosely Speculative Decoding (FLy) in Section 2.2 a method that relaxes the rigid exact match verification rule used in standard SPD without requiring any train- ing, as shown in Figure _ Different from existing mechanisms, FLy can accept worded-difeed -iifedddiffeeently but semantically correct tokens, thereby improving the mean accepted tokens (7) and the speedup
2

---

Published as a conference paper at ICLR 2026
Draft & Target verification
Prompt What the overfinting phanomene??
testing
falls
perform
Aeter
overlited
pertorms
well
training
data
bul
Tails
to
generalize
specific
domains
Entropy- level Gate
Token-lever deferred window
GotenDefer
Deferral
Monitor following tokens from
After
Re-align no further divergence
Accept
Gete-Strict Reject
X
Monitor following tokkan
penoralize
Another mismatch
Reject
Oate=Defer
Deferral
specific
comotine
X
Same Taken
Mismatch Token
Final Generated:
Wieen
overfitted
per Formss
well
tnaining
Figure 2: Overview of our proposed FLy. (1) When the draft and target tokens differ, we do not immediately reject the case as in prior SPD methods. Instead, our two-tier scheme assesses whether the mismatch is semantically valid and rejects only truly invalid cases. (2) An entropy gate rejects deterministic target predictions where h < 0, deferring ambiguous mismatches. (3)A token-level deferral window (W = 6) then monitors for continued divergence. (4) The final generation demon- strates that FLy admits more semantically valid continuations, whereas standard SPD would reject at the first mismatch.
ratio with minimum accuracy drop. The detailed algorithm is provided in Appendix A Finally, multi-level acceleration method is used to further boost the final speed-up ratio in Section 2.3
2.1 PRELIMINARIES
In each SPD round drafter MMD sequentially proposes K draft tokens { and the target model Mr verifies them together with the last generated token in round t - (denoted as v in parallel by performing single forward pass on the draft tokens to produce logits (
E VI, where denotes the set of the vocabulary and [V] the vocabulary size. The token in position K + 1 is the bonus token. The target model then normalizes the logits with a softmax function, and selects the token with the highest normalized logit as the output token:
( softmax(Ef) (L))..
of arg max PMM (1))
NEV
where (.) denotes the normalized predictive distribution from the target model MM over the vocabulary set V at position E [1, + 1), and yf denotes the corresponding output token of Mr-
Then, we define the match indicator A to judge whether the prediction at position i of the draft model matches that of the target model:
A 1' v.J, 1 IsisK
where 1[:] denotes the indicator function, which equals when its condition is true and otherwise. Concretely, of = y means that the output tokens from draft and target models coincide at position i, leading to A = 1, and vice versa. We omit the superscript in the following paragraph and only use A; for simplicity.
The verification strategy of standard SPD accepts all draft tokens until the first position where mismatch occurs, and discards the rest of the generated draft tokens. Then, will accept one more token generated from the target model as a bonus. Formally, each draft round, the number of accepted tokens Sstandard can be defined as:
min | A, 0}, II A, =0
8standard
(3)
K K+1,
otherwise.
3

---

Published as a conference paper at ICLR 2026
2.2 TRAINING-FREE LOOSELY SPECULATIVE DECODING (FLY)
FLy modifies only the verification strategy of standard SPD. Given K draft tokens in a draft round, there will be several mismatches that occur at different positions:
J (j A = 0), ssisk.
For each mismatch position j E J, we compute a normalized entropy given the normalized logits from the target model at that position:
hj PMrey log P.g
(5)
log [
The entropy hy [0, 1] quantifies the target model's uncertainty position j. A large h; indicates that several tokens could serve as valid alternatives at this position, whereas a small hy indicates that the target model tends to accept only the top-rated token.
Entropy-level gate. Given Equation. 5 we apply an entropy gate at the mismatch position j to decide whether the current mismatch should be directly rejected. Specifically, define entropy threshold 0 E [0, 1], and the gate serves as a lightweight per-token ambiguity detector driven by the target model. If the target model confident (i.e., hy < 0), indicating strong preference for its top- rated token such as digits in arithmetic, we fall back to the standard SPD rule and reject all tokens K Otherwise, the token this position is ambiguous (i.e., h 2 8), meaning that several tokens are nearly interchangeable such as coreferent pronouns,
starting from the mismatch position {
and we activate a token-level deferred window mechanism which will introduced later.
Concretely, at the mismatch position j. we apply a Gate with threshold to decide whether deferral should be invoked:
Gate(j) Strict, by < 0, Deter, by 2 0,
where Strict refers to the standard SPD rule that rejects tokens after position j. and Defer means using the token-level deferred window, postponing the rejection decision and observing the target's s behavior over the following few tokens. This separation prevents deferral at positions where an incorrect token would likely corrupt exactness. When treating the Defer operation as temporal acceptance, the number of accepted tokens with entropy-level gate 8gate can be formulated as:
min [j Gate(j) St Bj E J: Gate(j) Strict, JEU
Sgate
K 1,
otherwise.
Token-level deferred window. When Defer is active the mismatch index j. FLy uses look- ahcad window spanning W tokens to decide whether the mismatch benign.
Over the next W tokens, we compute the total number of mismatches:
j+W
Nw = Σ (1-Ai).
(8)
(=j+1
Then, we can determine to accept or reject this mismatch at position j based on whether another mismatch appears in this window:
DeferDecide(f) = Accept,
h 2 0, Nw = 0 and j j+ W S K
(9)
Reject, otherwise.
Our decision rule leverages the target L.M''s own behavior. When conditioned on semantically invalid token, it typically exhibits corrective behavior in its subsequent generations. Specifically, if the target model can continue generating from the mismatched token without further divergence, the first divergent token is likely semantically correct and can be accepted. Otherwise, if the target keeps disagreeing, attempting to course-correct the mismatch and thereby causing another divergence, we reject the mismatch j and the following draft tokens.
Note that we only have a maximum of K draft tokens in each round. Thus, j + W > K indicates that we get into the boundary case where the first mismatch index j falls within the last W positions
4

---

Published as a conference paper at ICLR 2026
and the size of the look-ahead window would be less than W. Then, immediately reject this mismatch at position j since the rest the tokens are not enough to judge the semantic correctness of this mismatch. It is worth noting that this immediate rejection incurs negligible cost, since only few trailing tokens are discarded. We adopt this conservative rule to preserve accuracy and keep the implementation simple.
The number of accepted tokens with deferred window 8defer can be formulated as:
min j )CJ
DeferDecide(j) Reject), 3j E J DeferDecide(j)) = Reject,
Sdofer
(10)
1,
otherwise.
In each SPD round, the final output of FLy depends where the first rejection occurs. Therefore, the number of accepted tokens with our proposed FLy SFLy can be formulated as:
SFLy mintSgater Sdefer
(11)
As we index SPD rounds by / E (1, T). we define sTIy the number of tokens accepted at round t. The mean number of accepted tokens over the run is then given by:
FFL
(12)
T
Notably, FLy introduces additional forward passes. Instead, it computes per-token entropy di- rectly from the already-availabl logits. As a result, the computation overhead is negligible com- pared to the inference cost of the draft and target model, as shown in the last line in Table 1
2.3 MULTI-LEVEL ACCELERATION
By leveraging the entropy-level gate and token-level deferred window, FLy attains substantially longer acceptance length (T). This lengthening reduces the frequency of target validations and hence their propor- tion of total latency, which in turn amplifies the draft model's auto-regressive cost as the dominant runtime bottleneck (Table This observation motivates our further contribu- tion: a Multi-Level Accelcration (MIA) strategy that applies speculative acceleration not only to the target model (as in standard SPD) but also to the draft model. By reducing overhead at drafting stage, MLA achieves greater end-to-end efficiency.
Components
70B
1.3 405B
wro MLA
MLA
w/o MLA
w/ MLA
Draft time
245.51
197.45
428.30
360.01
Target verification
38.62
200.97
Gate & Window activation
0.45
0.57
Table I: Wall time (ms) of different components for each SPD round.
A few prior works (Georganas et al. 2025 Sun et al., 2024) have investigated multi-level acceler- ation, but they generally depend parameterized dini-draft models, such as quantized variants or small language models (SLMs). To ensure broader applicability, we design the araft-acceleratioon stage in MLA to be parameter-free free and plug -nd--aaddyyy allowing seamless integration with the pro- posed FLy. Specifically, we instantiate MLA with Prompt Lookup Decoding (PLD) (Saxenal 2023). simple n-gram retrieval method that is extremely fast and training-free, thereby avoiding additional bias and preserving cross-domain generalization. Consequently, MLA reduces draft-side overhead and achieves higher end-to-end speedups, as illustrated Table and Table
It is worth noting that MLA is orthogonal to the verification scheme and is primarily designed to speed up the drafting phase itself. is particularly beneficial in our setting, where the algorithm can safely propose long draft sequences, making drafting time rather than target-model verification time primary bottleneck. This stands in clear contrast to prior approaches, where the target model's verification phase often dominates the runtime.
3 EXPERIMENTS
We begin this section by outlining the experimental settings in Section 3.1
Then, we compare
proposed FL.y against current SOTA methods in Section 3.2 Next, underscore its capability
5

---

Published as a conference paper at ICLR 2026
Model
Training
Metbod
ACP prog-gem
NIAH millivauue
MGSM
th
Mecnn
Speedup
Speedup
Speedup
Speedup
Speedup
Speedup
Temperstren0
L333 70B
FLy (Ouus)
BAGILES
1/88
1.52X
10.95
3.92
2.15%
3,34%
13.53
2,43
2.64%
1.57%
11.41
2.23
2.18%
1.72%
10.29
2.65
0184%
2,60%
10.58
1,26
2.53%
1.36%
11.35
2.50
Sp8
1.69
10.94
1.79
9.23
1.62x
8.72
1.78
9.84
2.04x
10.01
1.35%
1.28%
1.62%
1.51
TelennReycllngg
3.31
2.66x
2.74%
12:41
Sp5 REST
2.32x
14.02
24,13
2.42%
11.01
2,63x
11:22
3,0x
12:51
2.83%
14.58
L31 405B
TokenRacycling
1.588
1.94%
2.09
3.52
2.77%
1.00%
3.02
3.12
1.72%
1.40%
3.31
1.46
1.4%%
1.79%
1.53
3.34
1.78%
1.30%
3.35
1.44
1,68%
1.79%
1.91
3.31
X
FLy (Ours)
2,72%
15.08
4,07x
24.15
5.16%
14.49
5,67%
15.75
6.37x
16.18
4,80%
17:13
Tomperstaen1
1,33 708
/
X
FLy (Ours)
BAGIEES
1.48X
1.89
11:02
3.00
3,37x
1,59%
13.78
2,27
2.72%
1.47%
11.97
2,45
2.21
1.58x
10.99
2.45
2,63%
0.81
11.01
1.18
2.56%
1.45%
11.75
2.31
SpS
1.71%
1.79%
2.00x
10.71
1.31
TokenRacycling
1.83%
1.27%
1.40
1.22X
3.36
1,36%
1.55%
1.73
2.56x
Sp5
1,97x
11.70
2.10%
9.39
2.20x
2.39
12.17
L31 405B
TekenRacycling FLy (Ours)
REST
1.538
1811
1.95
3.54
2,58%
1.99x
2,82
3.10
1.31x
1.73%
3.23
1.36
1.38%
1.79%
1.43
3.34
1.78%
1.30x
3.34
1.34
1.67%
1,68
3,31
1.78
2.74%
15:44
4.10%
24.09
6.08%
17.97
6.42%
18.84
6.71x
18.10
5.21%
18,89
Table 2: Speedup ratios and mean accepted tokens (T) on out-of-domain (OOD) datasets. L31 and L33 represents Llama-3. 3.-.-..111mm -Innttruct and Llama-3.3-Instruue respectively. Mean represents the average performance across these datasets. We use bold text to denote the best result. / indicates training-based methods, whereas X means training-free methods.
for maintaining the target model's performance, conduct accuracy preservation benchmark in Section[.3] Finally, we perform a series of ablation studies to validate the effectiveness individual components of our method in Section 3.4
3.1 EXPERIMENTAL. SETUP
Models. For training-free baselines, we employ L-ama-3.1-8B-Instii (Dubey et al. 2024) as the draft model, with Llama-3. 1.1-a.....-7------ InOt-nntttt and Llama-3. 1.ama-3.1-405B-. parisons with EAGLE-2, we use Deea-Llama---Instrueee 3-Instruct-8B as its draft model. For comparisons with EAGLE-3, we adopt 1aama-3.-70B--Isttuu as the target model and Llama-3.1-8B -----B-IInsttett its draft model.
as target models. For com- the target model and Meta-Llama-
Baselines. We compare our method against both training-basee and training-free SPD approaches. For training-based methods, primarily include the previous state-of-the-au EAGLE-2 (Li et al. 2024b) and EAGLE-3 (Li et af. 20256). For training-fre methods, we select Speculative Sam- pling (Xia et al., 2023) (SpS), REST (He al. 2023) and TokenRecycling (Luo et al. 2024 as representative baselines.
Benchmarks. To assess robustness under distribution shift, divide the evaluation datasets into out-of-domain (OOD) and in-domain (ID) categories according to the training data used for EAGLE- 3. Specifically, the OOD group includes ACP-prog-gen (Kokel et al., 2025) (Action, Change, and Planning) which focuses on planning-style reasoning, NIAH-multivalue (Hsich et al., 2024) which extends the classic Meedle-in-a-Haysttee MGSM (Shi et al. 2022). the Multilingual Grade School Math benchmark. For the ID group, we adopt the widely used GSM8K (Cobbe et al., (021). HumanEval (Chen et al. 2021). I), and
test to include varied needle types and counts, and
MBPP (Austin et al. 2021 benchmarks.
set the deferred window length to W = 6 and the entropy gate threshold to 0 = 0.3 for all experiments. The draft token number K at each round is set to 15 for the 70B target model and 25 for the 405B target model. All experiments are conducted on AMD Instinct MI355X GPUs. Specifically, the 70B model is evaluated single GPU, whereas the 405B model is distributed across four GPUs.
Implementation Details.
3.2 PERFORMANCE COMPARISON
On OOD datasets, as shown in Table 2 FLy demonstrates exceptional performance. For Llama- 3.1-70B- -L-7tBB-Instruct, FLy achieves an average speedup of 2.74x (2.62x) with temperature= (= 1), outperforming existing training-free baselines. On the 1lama-3.3 variant, FL.y surpasses the training-

---

Published as a conference paper at ICLR 2026
Model
Training
Method
GSMMK
HumanFval
MAPP
Mean
Speedup
Speedup
Specciup
Specdup
Temperatunee)
1,3 708
FLy (Ours)
EAGLE-2
2.51%
2.94%
11.67
3.74
2.56x
2,60x
12.74
4.24
2,61
2,53x
11:21
3.90
2.57x
2.69%
11.87
3.96
1,33 70B
FLy (Ours)
EAGLES
3.72x
2.68X
11,68
5.64
3.97x
2.75%
12.20
5.73
3,80x
2.47X
11.76
5.47
3,3x
2,63X
11.87
5.61
Sy8 REST
2.13%
9.87
1.6%
9.89
1.72x
10.47
1.83x
10.08
L31 70B
TeleenRecyclngg
1.79x
1100x
2,00
3,04
2.12X
1,89%
2.23
3.07
2.03%
2.13x
2.34
3.01
2.12%
1.00%
2.19
3.04
FLy (Ours)
2,98
12.57
2.86%
12.61
2.79%
12.84
2.88%
12.67
Sy8
2.59
10.98
2.81%
11.62
3,03%
12.67
2,82X
11.76
2,13
2.41
2.
Tecennecycling
1.66x
(Ours)
Temperatune
1.3 708
EAGLE-2
(Ours)
2.36x
2.91%
11.55
3,48
2.38x
2.53%
12.67
3.95
2.59x
2.43
2.39%
11.86
3,69
L33 70B
EAGLES A-G FLy (Ours)
3.50
2.85%
12.40
5.31
3.74x
2.75%
12.55
5.40
3.58
2.69%
12.33
5.15
2.76x
3.61%
12:43
5,20
SyS REST
2.13x
9.70
1.07%
6.29
1.0%
6.08
1.40%
7.36
133 70B
Tokennazycling
1.71x
2.08 x
3,02
1.91
2.15%
1.81x
2.13
3.06
2.15%
1.94%
2.24
3.01
2.13%
1.82x
3.03
2.09
Fly (Ours)
2.67
11.98
2,52%
11.96
2.59%
12:57
2.59%
12:15
SaS REST
2.18x
9.25
2,39x
9.78
2.38
10.67
2.37%
9.90
4058
TeyenRogycling FLy (Chirs)
1.78x
1
2.85
1,82
1.83x
1.56x
2.91
2.07
2.02
2.11
7.97
1.57%
1.88%
2.00
2,89
4.97%
19.40
5.00x
16.29
17.80
5.31%
17.83
Table 3: Speedup ratios and mean accepted tokens (r) on in-domain (ID) datasets. L3, L31 and L.3 represents ------m-m-------cae Llama-3. I-Inttruct and L.ama-3.3- respectively. Mean represents the average performance across these datasets. We use bold text denote the best result. indicates training-base baainngbasedd methods, whereas X means training-free free methods.
based SOTA method EAGLE-3 by 1.62x (1.77x). This advantage scales with model size. Flyy achieves 4.80x (5.21x) average speedup on the 405B variant, as its higher per-token latency allows greater time savings when draft tokens are accepted, reducing costly target model calls.
On ID datasets, presented in Table 3 FLy remains highly competitive. With the 70B model, FLy obtains 2.88 x (2.59x) average speedup. surpassing all other training-free methods. Notably, FLy outperforms the training- based EAGLE-2 by 1.05x (1.12x), Although a bit slower than the heavily optimized EAGLE-3, this is expected and reasonable for a On the larger 405B model, FLy achieves remarkable 5.34x (5.31 x) speedup, significantly ex- ceeding all baselines and highlighting its powerful potential for accelerating large-scale models. is particularly noteworthy that baiindg-based methods like EAGLE-3 require prohibitive resource costs to train a drafter for the 405B model, often resulting in no officially supported model at this scale. In contrast, our training-free free approach provides solution that delivers excellent performance without such overhead.
training-fre free approach.
Experimental results demonstrate that FLy is more robust under distribution shift. EAGLE-3 achieves an average speed-up of 3.83x on in-distribution (ID) datasets, but this drops to 1.56x on out-of-distribution (OOD) data. In contrast, when using Llama-3.3-70B
the target model, FLy attains an average acceleration of 2.63x on ID data and 2.53x on OOD data, indicating that its performance remains far more stable across the distribution shift.
3.3 ACCURACY PRESERVATION
Since our method is loosely SPD method, the output of the accelerated target model would not be exactly the same as the original. Thus, we report accuracy preservation relative to the original target model after applying FLy, as shown in Figure Concretely, we normalize the original target model's score to 100% and use a recovery ratio to quantify how much performance is retained. The original non normaazad scores are provided in Appendix B Across different datasets and model scales, our method consistently maintains accuracy with over 99% recovery score, and performs on par with the training-bass loosely SPD method JudgeDecoding (Bachmann al., 2025). The JudgeDecoding accuracy preservation results reported here are taken directly from the original pa- per, since JudgeDecoding has not been open-sourced. recognizes sensitivity to train-test domain misalignment. When coding examples are removed from their training data, performance on HumanEval drops substantially from 99.4% to 92.3%, under- scoring degradation under distribution shift.
worth noting that JudgeDecoding also
7

---

Published as a conference paper at ICLR 2026
Limma-o
judgehheeding--88
Segpelieeodnnggltllleeeee ht code training datse
Fly-70
Unme-708
Nartowery (8g
$2.6
GSMBK
HumanEval
NBPP
MGSN
NIAH
ACP
Lama-80
JudguOucoding 435b
Fly-45bb
Ulma-408b
Beccuary 186
100
961
100
100
100
100
323
100
03
60
Figure 3: Accuracy preservation results. The performance the target model normalized to 100, and the recovery ratio is used to quantify the extent of performance preservation.
3.4 ABLATION STUDIES
In this section, we evaluate the contribution of each component of our method to the overall perfor- mance. Specifically, we conduct ablation studies on four key aspects: (1) the core including window length (W), the entropy gate threshold (0), and the draft token number (K). (2) The multi-level acceleration mechanism, which evaluates the impact of accelerating the drafter stage itself. (3) The drafter model size, capturing the capacity-latency trade-off. (4) Cross-model traft-target combinations, evaluating robustness across LLM families and scales. Unless otherwise specified, experiments
conducted with Llama-3.1- 1-70------- Inst-nntt the target LLM, 1lama-3.1- 8B-Instruct as the drafter, HumanEval as the evaluation dataset, and the temperature fixed as 0.
Hypee--arameters. The window length (W) determines how many subsequent positions mon- itored in the deferred decision process to assess the target model's behavior. We vary W E (0, 4, 6, 8), shown in Table (1). When the window disabled (W 0), all deferred mis- matches are accepted, leading to maximal speedup and т, but at the expense of accuracy. A short window (W 4) yields a higher speedup with slight accuracy drop. W
restores perfect recovery with only a small loss in speedup and is our default, while W = 8 is overly strict, reducing both 7 and end-to-end speedup.
The entropy gate threshold (0) controls whether position is treated as a unique choice (low entropy, mismatches directly rejected) or multiple plausible candidates (high entropy, deferral allowed). We study E {0, 0.3, 0.6, 1}. as shown Ta- (2). Disabling the gate (0 = 0) defers all mismatches, maximizing speedup and but hurting accuracy. 0 0.3 achieves perfect recovery with strong speedup and our default. A stricter gate (0 0.6) reduces deferrals and rejects most mis- matches, lowering speedup and т. When 0 1, the method rejects all mismatches and degenerates to standard SPD.
Hyper-paranen
Val
Specdup
Recovery(S))
3.42x
15.59
93.7
2.91x
13.27
97.9
(1) Window length W
2.86x
12.61
100
2.61x
11.96
100
2,98x
12.76
97.7
Entropy threshold 0
0.3
2.86x
12.61
100
2.76x
12,20
100
1.64x
9.89
100
2.75x
9.04
100
(3) Draft token K
2.86x
12.61
100
2.59
15.60
100
25
2.58x
18.56
100
Table Ablation study on the hyper-parameters.
The draft token number K specifies how many tokens the drafter generates in a single round. We evaluate E {10, 15, 20, 25), as shown in Table 4 (3). While т grows monotonically with speedup increases then declines bottleneck. We therefore use K = 15 for the 70B target (best speedup) and = 25 for the 405B target, where verification is costlier. A detailed 405B ablation is shown in Appendix C
the drafter's ss cost for unaccepted tokens (K 7) becomes the
Multi-level acceleration effect. Ablation results on multi-level acceleration are presented in Ta- ble 5 Enabling MLA improves speedup from 2.69 to 2.86x, confirming that accelerating the drafter stage reduces end-to-end latency beyond what target-only acceleration can achieve.
Draft model size. We vary the draft model size from 1B 8B, shown in Table 6 The mean accepted tokens t grows monotonically with drafter capacity, whereas the overall speedup is non- monotonic since the drafter's computational cost also increases with model size, potentially out-
8

---

Published as a conference paper at ICLR 2026
weighing the benefit of a higher unless the gain from is substantial. We therefore adopt the 8B model as the default drafter, as it achieves the best balance between accuracy and throughput.
w/o MLA
w/ MLA
Draft model size
1B
3B
8B
Speedup
2.69x
2.86x
Speedup
2.80
2.47
2.86x
8.98
10.71
12.61
Recovery(%) (%)
99.23
100
100
Table 5: Ablation study on the multi- level acceleration.
Table 6: Ablation study on drafter size.
Cross-model draft-target pairing. To validate whether FLy generalizes beyond specific pair of models, we further evaluate it under three heterogeneous draft-target combinations, as shown in Table 7 They cover both cross-family and cross-scale settings: (1) lightweight Qwen2.5-Coder- Instuutt (Hui et al. 2024 Yang et al.) 2024) drafter with a Disstral-LaggeInsttue
target, (DeepSeek-Al (025) drafter with Mistral--arge-Instricc -eepSeec-R---isill--Peeeee Twen-I..BBB drafter accelerating a larger Llama variant. Across all settings, FLy achieves 1.85x 3.54x speedups while preserving over 99% of the target model's ' accuracy. This demonstrates FLy's 's model-agnostic property, which allows it to seamlessly compose with multiple targtt pairs without any retraining.
(2) a DeepSeek-RI-e -----isill--wenn
2411 target, and (3)
Draft model
Target model
Speedup
Recovery (%)
Owen2.5-Coder-D. -.B-st--....ttt SeepSeek-RI--isiil--eeeeeee
MisstrllLarge--Issrrutt-22 Dissral-Large--nsttuu---444
3.54x
12.34
99.1
2.35x
9.60
100
DeepSeek-RI ---iistill--wwn--ll
DeepSeek-RI Baaissek-----iiilll--------- -Dii------still-aaa--0oo
1.85x
10.28
100
Table 7: Ablation study cross-model draft target pairing.
4 RELATED WORKS
Speculative decoding (SPD) speeds up autoregressive LLMs by letting small drafter propose sev- eral next tokens sequentially and having the target model verify them parallel (Xia al. (023).
Training -baend SPD. A large body of research (Li et al. 2025a Liu et al. 2024 Xiao et al. 2024
Ankner et al., 2024) extends SPD by learning auxiliary predictors to better align drafter and target, thereby improving speculation accuracy. Typical strategies train additional modules that imitate the target's behavior to raise acceptance rates and speedups. Medusa (Cai et al. (024) uses hidden states from the base LLM as inputs to multiple lightweight MLP heads, each predicting a future token. GLIDE (Du et al., 2024) speeds up decoding by reusing the target model's KV cache via cross- attention and adaptively expanding proposals. HASS (Zhang et al. 2024) improves acceptance rates by aligning training objectives with inference behavior. EAGLE (Li et al. 2024a) generalizes this design by employing lightweight Transformer predictors and concatenated token-state pairs, while EAGLE-2 (Li et al. 2024b) improves efficiency with a dynamic tree-based candidate selection mechanism. EAGLE-3 (Li et al., 2025b) further leverages intermediate hidden states scale up decoding acceleration. Distinct from attaching auxiliary modules, LayerSkip (Elhoushi et al. (024) exploits the target itself by training early layers to propose drafts that are subsequently verified by the full model. Although effective, these approaches require task-specific supervision and often generalize poorly beyond the training distribution, limiting robustness out-of-distribuiie settings.
Training-fres free SPD. In contrast, training-free free approaches dispense with additional training. Specula- tive Sampling (Chen et al., 2023) pioneered the foundational draft-then- then-verify paradigm, employing modified rejection sampling scheme to accelerate decoding while ensuring lossless generation. Retrieval-basee methods (Zhao et al. 2024 He et al. 2023) maintain a datastore and retrieve n- gram continuations draft tokens, offering a aan--and--lyy accelerator that obviates a parametric drafter and can be applied to arbitrary targets. Draft& Verify (?) formalizes this by employing Bayesian optimization to identify optimal static skipped layer sets. SWIFT (?) further advances this by dynamically optimizing the skipped layer set on-the fly based on the input context. Additionally, KNN-SSD (?) utilizes nearest neighbor search retrieve domain-specifie skipping configurations
9

---

Published as a conference paper at ICLR 2026
to address sensitivity to data distribution shifts. Because the drafts generally weaker and less aligned with the target, these approaches tend to yield lower speedups than training- baaid ones.
Loosely SPD. Strict verification often rejects semantically valid drafts, limiting achievable speedups. To address this, "loosely" variants (Garipov et al., 2025; Wang et al.) (025c) relax the criterion to tolerate token-level deviations. JudgeDecoding (Bachmann al. 2025) trains lightweight auxiliary model to recognize semantically correct but mismatched drafts, but such su- pervised training limits out-of-domain generalization. Reflective Verification (Wang et al. 2025b) takes a training-free route, leveraging the reflective capacity of LLMs to semantically probe draft correctness. SPRINTER (Zhong et al.] 2025) employs a lightweight verifier to approximate tar- get acceptance only when mismatches occur, reducing target calls and further lowering latency. sigemennt-sappling methods (Wang et al. 2025a) exploit distributional information from the pre- filling phase to propose hetter-aligned drafts, and couple this with flexible thresholding strategies that adaptively accept high-quality but imperfect candidates. These approaches highlight the grow- ing interest in relaxing strict token-level equivalence, trading exactness for efficiency while explor- ing balances between training-based and training-free designs.
Our FLy follows the loose philosophy but differs from prior methods: instead of relying on trained verifiers or relaxed acceptance rules, it remains entirely training-fiee free and judges drafts by defer- ring verification, leveraging the target's own behavior and entropy signals to distinguish harmless divergences from true errors.
5 LIMITATIONS
Although FLy achieves substantial speedups while preserving high accuracy, it remains fully training-free and does not optimize its verification policy on task-specific benchmarks. On datasets tailored to toining-based speculative decoding methods such EAGLE-3, this can lead formance gap. Moreover, FLy is designed to preserve semantic consistency with the target model rather than exact token-level agreement, which can be suboptimal for applications that require ver- batim reproduction (e.g., repeating a long passage or reciting a poem).
per-
6 CONCLUSION
This paper introduces Training-Free Loosely Speculative Decoding (FLy), novel training-frre al- gorithm that replaces standard SPD's rigid exact-match criterion with loosely verified scheme to accept semantically correct tokens. When mismatch occurs, FLy applies two-tier schemn to distinguish genuine errors from semantically valid cases. Firstly, an entropy-level gate determines whether loosely verifying is appropriate (i.e., whether the current context is sufficiently uncertain such that multiple alternative tokens could be considered valid). Then, a token-level deferred win- dow monitors the target model's behavior over the subsequent tokens. If the generation proceeds without further divergence, the initial mismatch is treated as a differently worded yet semantically correct continuation and is accepted. Otherwise, additional mismatches indicate corrective behav- ior, and the initial mismatch is retroactively rejected. We further propose a multi-level acceleration mechanism, in which the drafter itself is accelerated to further reduce latency. Experimental results show that our approach significantly increases mean accepted tokens (r) and the overall speedup while preserving the target model's 's accuracy (2 99%), , while demonstrating strong generalization.
REFERENCES
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptin Alayrac. Jiahui Yu, Radu Soricut, Jo- han Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Tim- othy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Dani- helka, Becca Roclofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and al. Gemini: A fam-
10

---

Published as a conference paper at ICLR 2026
ily of highly capable multimodal models. CoRR, abs/2312.118055 2023. doi: 10.48550/ARXIV. 2312.11800 Utthttsss://// /tt.:/doiiioo 25tp://...4445000222222222222222
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Sequentially-dependee draft heads for medusa decoding. arXiv preprint arXiv:2402.05109, 2024.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 077332, 2021.
Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment. arXiv preprint arXiv:2501.19309 2025.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple IIm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013/8, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1441 2021.
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang. Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang, Meng Li. Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs 2025. doi: 10. 48550/ARXIV.2551 12249
https /oi...i... woog/llgll..........o 10. 48550/85544 arxiv 2501 12948
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. The International Conference on Machine Learning, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong.
11

---

Published as a conference paper at ICLR 2026
Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruy Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank 7hang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, liyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The URL
Ilama 3 herd of models. CoRR, 0/bb/2407.22183.. 2024. doi: 10:4850/ARXI:2240...00 48550/ ht httpps://dot http://....g/...../5//////////// www.a.iivvvv...... 2407. 21783
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layerskip: Enabling early exit inference and self-speculative decoding. arXiv preprint 1677:0 2024.
Roman Garipov, Fedor Velikonivtsev, Ruslan Svirschevski, Vage Egiazarian, and Max Ryabinin. Autojudge: Judge decoding without manual annotation, 2025. URL https://arxiv. org/
abs/2504. 20039
Evangelos Georganas, Dhiraj D. Kalamkar, Alexander Kozlov, and Alexander Heinecke. MI-specqd: Multi-level speculative decoding with quantized drafts. CoRR, abs/2503. 13565,, 2025, doi: 10. 48550/ARXIV.2500..3.
URL httph:://p/oott org/10
0e0111..855001 48554/555//00 arxiv, 2503. 230363
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval-based speculative 2023.
decoding. arXiv preprini
Cheng-Ping Hsieh, Simeng Sun. Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models?, 2024. URL https:// aawxiv. www.xiii.org/baa....oeee org/abs.////eoooooooo..ooo.o.s.s 2404. 06654
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12222499.11180 12186, 2024.
Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. Acphench: Reasoning about action, change, and planning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 26559-26568, 2025.
Jinze Li, Yixing Xu, Haiduo Huang, Xuanwu Yin, Dong Li, Edith CH Ngai, and Emad Barsoum. Gumiho: A hybrid architecture to prioritize early tokens in speculative decoding. arXiv preprint arXiv:2503. 10135, 2025a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint ar arXiv:2401.15077 2024a.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858 2024b.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3; Scaling up inference acceler- ation of large language models via training-time time test. arXiv preprint arXiv:2503.01884 01840 2025b.
Fangcheng Liu, Yehui Tang. Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding via double early exiting. Conference Neural Information Processing Systems, 2024.
Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, and Dongliang Xu. Turning trash into treasure: Accelerating inference of large language models with token recycling. arXiv preprint arXiv:2408.0860 08696, 2024.
12

---

Published as a conference paper at ICLR 2026
Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self- correction strategies. CoRR, abs/2308.03188, 2023. doi: 1044550/ARRXI.333X005132 URL
ht.pp:://df www./oorr....... wrrg///....50/rxxiv......o w03. 03188
Apoory Saxena. Prompt lookup decoding, November 2023. URL https://github. com/
apoorvumang/proome 1owpt--ooooooppdddiddii-
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Lan- guage models are multilingual chain-of-thought reasoners, 2022. URL hhtps://aara
org/abs/ 221 2210. 03057
Mitchell Sterm, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kris- ten Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 31: Anmual Conference on Neural Information Process- ing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 10107- 10116, 2018. URL https: httpp::s//ppoceednggs.... neurips. tttpcc//ppppaa ppwppr//c1//aaeeeeeeeepppp c4127b91 www. (225556266440000
(ww.:999900555220000000000000000 5bf Abst-cccbbbttttc...... html
Ryan Sun, Tianyi Zhou, Xun Chen, and Lichao Sun. Spechub: Provable acceleration to multi- draft speculative decoding. In Yaser AI-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 20620-20641. Association for Computational Linguisties, 2024. doi: 10. 18633/V /2000/..EE.P-MAI..IIII URL https:
//doi..rg// 10. 18653//1//0000
emnlp main. 1148
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL http:: /tt.tppe:/procoeediiiie neuri www.ies............... wttplpp..cccpppeee htcppppppe/221//aaaeeee
7 Sac Abst htm
Jikai Wang, Zhenxu Tian, Juntao Li, Qingrong Xia, Xinyu Duan, Zhe-Feng Wang. Baoxing Huai, and Min Zhang. Alignment-augmenne speculative decoding with alignment sampling and con- ditional verification. CoRR, abs/2505.1 13204, 2025a. doi: 10.48550/RRXIV2555...33. 13204. URL
https://doo
/CC 48550/ arXiv. 2505. .16...12444
Yixuan Wang, Yijun Liu, Shiyu Ji, Yuzhuang Xu, Yang Xu, Qingfu Zhu, and Wanxiang Che. Think before you accept: Semantic reflective verification for faster speculative decoding. CORR, doi: D044850AAXXV.2555.... 48550/arXiv. 2505.
18s/2505.18629, 2025b.
URL httph::ts//dltt org/10.
Yixuan Wang, Yijun Liu, Shiyu ji, Yuzhuang Xu, Yang Xu, Qingfu Zhu, and Wanxiang Che. Think before you accept: Semantic reflective verification for faster speculative decoding, 2025c. URL
https://aan http://axxvv..v. org/
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decod- ing: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 20233, pp. 19909-3925, 2023.
Zilin Xiao, Hongming Zhang, Tao Ge, Siru Ouyang, Vicente Ordonez, and Dong Yu. Parallelspec: Parallel drafter for efficient speculative decoding. arXiv preprint 1xXh::2410055589, 2024.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai
13

---

Published as a conference paper at ICLR 2026
Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren. Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint 240iv:2407. 10671, 2024.
Lefan Zhang, Xiaodan Wang. Yanhua Huang, and Ruiwen Xu. Learning harmonized represtttoos for speculative sampling. arXiv preprint arXiv:2408. 2024.
Yao Zhao, Zhitian Xie, Chen Liang, Chenyi Zhuang, and Jinjie Gu. Lookahead: inference accel- eration framework for large language model with lossless generation accuracy. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6344-6355, 2024.
Meiyu Zhong, Noel Teku, and Ravi Tandon. Speeding up speculative decoding via sequential ap- proximate verification. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Mod- els, 2025. URL URL (tthttps:/// http:///eaeeeeeeeeeeeeerrrrrmmee
14

---

Published as a conference paper at ICLR 2026
A DETAILED ALGORITHM
We provide a step-by-step exposition of our proposed Flyy in Section 2.2 detailing each component and procedure facilitate deeper understanding.
Algorithm 1: Training-Free Free Loosely Speculative Decoding (FLy)
Input: Prompt z, draft model Ma, target model Mr, draft token number K, thresholds 6, window size W. and vocabulary V.
Output: Generated sequence y, mean accepted token
// Initialize SPD round t, accepted token number
f 4- 0, 8
y 4-x
while not EOS do
// (Step 1) Sequentially draft tokens
(+)) M
// (Step 2) Verify by target model with single forward pass
( Mr.foraard(concat (
forit 1to do (M) softmax for v arg max, (M) A 1 vil PMr.K+1(1) f- softmax +1)) for V sk+1 + arg max gv (Mtp, (i -1 = then
10
11
// All matched, accept all
y 4 concat [y, (0.
12
t+t+1
14
15
continue accept all True
17
for j E do
1/ (Step 3) Entropy gate at the mismatch j E PMr.(() Pog
hy
log
19
if hy < then
// Strict mode, reject
accept_all +- False
У concat [v. (o v/]
22
8 - s
23
break
24
// (Step 4) Deferred window decision over next W tokens
Mw + +W (1 A!)
25
26
Mw and j + < K then
// Mismatch at j is do but semantically correct, accept it
continue
28
else
// Treat it as harmful, reject it accept_all +- False
y 4- concat [v. (1 v1]
8 5 8
32
break
if acccpptaall then
34
+ concat ly, K v1
35
8 86
36
37
s/t
15

---

Published as a conference paper at ICLR 2026
B RAw ACCURACY SCORES
This section presents the raw, non-normalized accuracy scores corresponding to the accuracy preser- vation results discussed in Section 3.3 Table provides the original evaluation metrics for each model and dataset, offering a direct view of the absolute performance.
Model Family
Model
GSMSK
HumanEval
MBPP
MGSM
NIAH
ACP
L31 8b
85.06
66.46
74.87
73.20
99.42
42,30
FLy 70b
94.69
79.26
89.15
88.41
99.93
60.37
Llama-3.1 -.ntt-ut
L31 70b
95.45
79.26
89.42
88.67
99.97
60.77
FLy 405b
96.21
87.20
90.48
92.27
100
69.31
L31 405b
96.51
87.80
91.00
92.80
100
70.00
y 70b
94.84
84.54
88.62
90.2
100
59.23
1lama-3.3-Instruue
1.33 70b
95.60
85.37
89.15
90.4
100
57.69
L3 8b
79.30
60.37
75,40
Meta-Llama-3-Instue
FLy 70b
91.75
78.55
84.39
L3 70b
92.34
79.27
84,93
Table 8: Raw accuracy scores. L31 represents -.-na--t,tI L33 represents Llama-3.3- Instruct, L3 represents Meta-Llama-33Instiee and FLy is our proposed method.
C ABLATION STUDY ON DRAFT TOKEN NUMBER K FOR THE 405B VARIANT
We investigate the effect of the draft token number K on the performance of FLy when accelerating the 1aama-31-400B--1 Instruct model. shown in Table 9 increasing K leads to a higher mean ac- cepted token count (r) and improved speedup, but at the cost of slightly reduced accuracy recovery. Based on the trade-off, we select K 25 for the 405B model, which achieves a strong speedup of 5.15x with minimal accuracy degradation (99.3% recovery).
Draft token K
Speedup
Recovery(S)) (%)
15
4.60x
11.43
99.7%
20
4.89x
13.64
99.7%
25
5.15x
15.83
99.3%
30
5.67x
17.57
97.9%
Table 9: Ablation study draft token number K for Llama-3. 1amma-3.1-40555 1-00S--Innsttt
D SPEEDUP ON LARGE BATCH SIZES BASED ON VLLM
We evaluate the impact of FLy throughput for large batch sizes using four AMD Instruct MI250 GPUs with vLL.M, a widely adopted production-gre framework. The results are shown Ta- ble 10
Batch size
8
16
Llama-3.1-70b-nnste
2.39
2.09
1.96x
1.78x
Table 10: Throughputs on HumanEval dataset under different batch sizes using vLLM. The infer- ence speed of the model without speculative sampling is used the baseline (1.00x).
E
CASE STUDY
Figure presents case from the GSM8K dataset using Llama-3, 1--15--0-nnsiiie where FLy rectly accepts semantically valid draft tokens that standard SPD would reject. As shown, disagree- ments between the drafter and target appear several tokens. Rather than rejecting the draft whole-
16

---

Published as a conference paper at ICLR 2026
sale standard SPD would, FLy preserves semantically valid alternatives, increasing acceptance rate per step and accelerating decoding.
Input Prompt:
4|start header, jdj-user=iend heanndd_eaeerraaee ii|>>
Given the following problem, reason and give final answer to the problem.
Problem: Ellen decided play prank on her friend. She got case 12 sodas and shock them up. Then she look unshaken soda for herself and left. Ellen's brother stopped and took the shaken sodas and 2 the unshaken sodas, then Elien's friend came along. What is the likelhhod, expressed as percentage, that Ellen's friend gets sprayed with soda from shaken can?
Your response should end with "The final answer is [answer]" where [answer] is the response to the problem.x/ect hdl=s(star__haddee saaaddrrddpasssssattseeedd haapaattteeedddehee______
Output:
Ellen (There) started with sodas. She shook of them. Then (So) she took unshaken soda, leaving there were are 12-1 = sodas left. Her brother took shaken soda
unshaken sodas (ones), so there are 11 31-33 sodas shaken sodas left. Ell The chance of Ellen's friend getting sprayed (a) is the number of shaken sodas divided by the total number of sodas.. There
left. There were originally (3) shaken sodas, but
taken
there
shaken sodas out (and) of 8 lotal sodas. 2 8 0.25. To idl=assistantt(jnd _eed____eddro multiply by 100. 0.25 100 = 25%. The final answer is 25%. </eot_ idj=<//sarrt header
(converi) this percenlage,
Figure 4: Case study of FLy sample from the GSM8K dataset using Llama-3.1 1.4400B-Instruee Blue tokens denote the drafter's output and green tokens denote the target's output. Under standard speculative decoding, all mismatches would be rejected. In contrast, FLy selectively rejects only the mismatches shown with strikethrough, while retaining the remaining ones because they are seman- tically valid.
F LLM USAGE
In the preparation of this manuscript, we used large language model (LLM) solely for language polishing and grammatical refinement of the text.
17


Reviews:
[{"review_id": "HZT1xqhQNl", "reviewer": "Reviewer_6AHf", "summary": "The authors propose a novel method to accelerate LLMs via speculative decoding. In particular, they relax the rigid assumption of verification in classic speculative decoding and propose a two-tier mechanism. The method comprises first an entropy gate to evaluate whether a mismatch should be directly rejected, and then a token-level defer window to find whether the mismatch is benign. The efficacy of the method is showcased on multiple benchmarks and across different draft and target models.", "questions": "I would appreciate if the authors address the questions raised in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The main idea is novel and clearly presented.\n\n- The studied problem is relevant and the proposed solution appropriately motivated.\n\n- There is a plethora of experimental evidence across different models that support the efficacy of the proposed method.\n\n- The proposed method is benchmarked against mutliple SOTA baselines and showcases superior performance while requiring no training.\n\n- The comparison with learning-based approaches, i.e., judge-decoding, is insightful and highlights the capabilities of the proposed method.\n\n- I appreciate the thorough ablations on cross-model drafting.", "weaknesses": "Although I do not see any major flaws, there are some issues that I believe would worth addressing:\n\n- I am missing some discussion and/or analysis of the shortcomings and failure modes of the proposed model.\n\n- For completeness, it would be interesting to see what are the speedups achieved by the proposed and judge-decoding in Fig. 3.\n\n- I am missing some related work, e.g., [1, 2]\n\n [1]. Learning harmonized representations for speculative sampling\n\n [2]. Glide with a cape: A low-hassle method to accelerate speculative decoding", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4yuYLgkGtL", "reviewer": "Reviewer_RKkU", "summary": "This paper studies speculative decoding with lossy verification, a topic that has recently garnered some attention in the community. The authors investigate training-free approaches that enable the target model to accept tokens it would usually reject under the standard regime, while maintaining high output quality. This is achieved through a two-stage process as follows: First, an entropy gate is designed, where the rejection of tokens that would get rejected according to the target logits, is deferred in case the entropy of the target distribution was very high, suggesting that multiple tokens could potentially be used instead of solely the most likely one. The deferred token is then put on “probation”, i.e. if another rejection within a certain window occurs within the continuation of said token, then it will be rejected. If no other rejections occur, then the token is accepted. This follows from the insight that target models tend to correct mistaken tokens in subsequent steps, making the likelihood of another rejection significantly higher in case the token truly should have been rejected.\n\nThe authors then evaluate their approach in the setup of draft / target speculative decoding, with models of size roughly 8B serving as the draft for targets of size 70B and 405B. The proposed approach is evaluated on a range of benchmarks and compared against recent models such as Eagle-3. On out-of-distribution datasets, the novel approach outperforms the baselines and remains competitive in-distribution as well, while maintaining the output quality.", "questions": "1. When looking for later mistakes in the window, you seem to not apply the entropy rule anymore, i.e. any direct mismatch counts as a mistake within the window, correct? \n2. The section on Multi-Level Acceleration confused me a bit initially. Of course when drafting longer candidate sequences with high acceptance rates, the generation process starts to spend most time on drafting. But from a pure speculative decoding perspective, this is of course inevitable (in the extreme case we draft the entire sequence once and accept everything with the target in a single forward pass). I would state more clearly that MLA is an idea to further speed-up the entire process, but it’s not a mechanism to improve the speculative decoding setup you’re studying, it’s a way to speed-up drafting itself. Or did I misunderstand something?\n3. Can one equip EAGLE-3 with your augmentations to enable it to have lossy verification? In general, does the approach extend to any speculative decoding setup?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Lossy verification in speculative decoding is a timely topic that enables more efficient decoding for large models. Making progress in this area is very relevant for the field to further push the efficiency gains.\n2. The paper is very nicely written and easy to understand. The proposed approach is very well motivated and the experiments are thorough, which is needed to assess its validity. \n3. The technique doesn’t require any training and any model can easily be enhanced with this approach, making it easy for users to experiment with it for their own model.", "weaknesses": "1. The out-of-distribution numbers are quite strong but in-distribution, training-based approaches like EAGLE-3 still keep the upper hand, with quite a discrepancy. What seems a bit problematic however in this experiment design is that different speculative decoding techniques are compared; standard small / large model speculation (Fly) and self-speculation (EAGLE-3) where the draft model is a very small model on top of the features. Usually, more involved like tree-attention are also involved. Thus enhancing small / large model speculation with the new techniques might not be the best comparison. Why not also equip EAGLE-3 with the same techniques as introduced in this paper, or am I missing an incompatibility? Indeed, the approach presented here does outperform the standard small / large model speculation setup without these enhancements.\n2. I could not find any tokens/s metrics, arguably the most important metric in LLM inference. What frameworks are the experiments based on in the main paper? I found some results in vLLM in the appendix but they’re unfortunately very limited. I find experiments in this setting way more important as they actually reflect real scenarios more accurately. Many conclusions in the literature have changed in the past when moving to more optimized frameworks, so it would be great to verify that the proposed approach remains superior even in more optimized settings like in vLLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TLsY4Zxp8y", "reviewer": "Reviewer_oayV", "summary": "The paper introduces Training-Free Loosely Speculative Decoding (FLy), a speculative decoding method that relaxes the strict exact-match verification criterion in standard SPD. FLy accepts semantically correct draft tokens even if they do not exactly match the target model’s output, thereby increasing the number of accepted tokens per round and improving speedup. The method uses a two-tier mechanism: an entropy-level gate to determine whether a mismatch is in a high-uncertainty context, and a token-level deferred window to monitor future tokens for signs of error correction. FLy is training-free, model-agnostic, and includes a multi-level acceleration strategy that also speeds up the draft model. Experiments show strong speedups (up to 5.07× on 405B models) while preserving ≥99% of the target model’s accuracy.", "questions": "- The reproducibility of the work may need improvement through open-source sharing. Would you provide your source code?\n- Regarding the performance of JudgeDecoding in the paper, please detail the reproduction of the results.\n- How do you explain the extremely high recovery rates (≥99.5%, including multiple 100%) reported in Figure 3, Tables 4, 6, and 7? Given that FLy explicitly accepts semantically mismatched tokens, some degradation in task-level accuracy is expected.\n  - What is the standard deviation across multiple runs?\n  - How many test samples were used per dataset?\n  - Could you provide confidence intervals or bootstrap estimates to demonstrate that these values are statistically stable, rather than artifacts of small test sets or coarse metrics?\n- Please address the issues mentioned in \"Weakness\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach Fly introduces a novel, training-free speculative decoding mechanism that accepts semantically valid mismatches.\n- Extensive experiments, ablations, case study and cross-model evaluations demonstrate robustness and effectiveness.\n- The paper is clearly written, with intuitive explanations and strong visual aids.\n- FLy addresses a real bottleneck in LLM inference and offers a clever, general-purpose acceleration method.", "weaknesses": "- For Figure 1, it is hard to tell whether Fly is more robust against OOD datasets as the degration rate on speedup is similar.\n- While outperforming training-based methods on OOD data is a key result, the paper notes that FLy is slightly slower than the heavily optimized EAGLE-3 on ID datasets. An in-depth discussion or analysis of this performance gap beyond the expected trade-off for \"training-free\" would improve the technical quality of the paper.\n- The Multi-level acceleration technique with PLD is borrowed from literature. Its technical contribution is not significant.\n- The paper may need a quantitative breakdown of how often FLy’s deferred acceptance mechanism is triggered, and what fraction of accepted tokens are due to semantic mismatch tolerance rather than exact matches. Reporting the percentage of tokens where FLy overrules vanilla SPD and the resulting speedup contribution would directly substantiate the claim that loose verification is the key driver of acceleration. Without this, it remains unclear how much of the speedup is due to FLy’s core innovation versus other factors (e.g., larger K, MLA).\n- The ablation studies in Table 4 vary only one hyper-parameter at a time (W, θ, K), while keeping others fixed. This one-at-a-time approach is insufficient for exploring the joint effect of hyper-parameters. A grid search over the hyper-parameter space is necessary to ensure that the reported configurations are globally optimal, not just locally tuned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wAk3nFDb6r", "reviewer": "Reviewer_Eqon", "summary": "This paper proposes Training-Free Loosely Speculative Decoding (FLy), which relaxes verification by accepting semantically correct drafts without additional training. Experiments show that FLy preserves ≥99% of the target model’s accuracy while achieving a considerable speedup.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a clearly written narrative and well-structured exposition that is easy to follow.\n\n2. The two-tier mechanism that relaxes verification for worded-differently yet semantically valid alternatives is reasonable, simple to implement, and empirically effective.", "weaknesses": "1. My primary concern is that with tree-structured verification, the paper’s targeted optimization cases may be diluted; this risk should be explicitly analyzed and validated with focused experiments.\n\n2. The main table integrates PLD’s multi-level acceleration while comparing against methods not adapted with hierarchical speculative decoding; this may be an unfair comparison.\n\n3. The related-work section mischaracterizes training-free SPD: LayerSkip requires training, whereas the earliest training-free approaches are Self-SD, Draft & Verify, and their extensions SWIFT and KNN-SSD; this should be corrected.\n\n4. Please add insights on task sensitivity to relaxed verification: which tasks are less sensitive (and typical improvement gaps between non-relaxed vs. relaxed), which are more sensitive, and why. This would substantively enrich the paper’s contribution.\n\n5. The paper contains interesting ideas; releasing code would make the work more promising and facilitate community adoption and verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)