Scaling Speculative Decoding with Lookahead Reasoning
Authors: Yichao Fu
Rui Ge
Zelei Shao
Zhijie Deng
Hao Zhang
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Reasoning
Speculative Decoding
Primary area: deep_learning

Abstract: Reasoning models excel by generating long chain-of-thoughts, but decoding the resulting thousands of tokens is slow. Token-level specualtive decoding (SD) helps, but its benefit is capped, 
because the chance that an entire $\gamma$-token guess is correct falls exponentially as $\gamma$ grows. This means allocating more compute for longer token drafts faces an algorithmic ceiling -- making the speedup modest and hardware-agnostic. We raise this ceiling with lookahead reasoning, which exploits a second, step-level layer of parallelism. 
Our key insight is that reasoning models generate step-by-step, and each step needs only to be semantically correct, not exact token matching.  In lookahead reasoning, a lightweight draft model proposes several future steps; the target model expands each proposal in one batched pass, and a verifier keeps semantically correct steps while letting the target regenerate any that fail. Token-level SD still operates within each reasoning step, so the two layers of parallelism multiply. We show lookahead reasoning lifts the peak speedup of SD both theoretically and empirically. Across GSM8K, AIME, and other benchmarks, lookahead reasoning improves the speedup of SD from 1.4x to 2.1x while preserving answer quality, and its speedup scales better with additional GPU throughput. Our code is available at https://github.com/hao-ai-lab/LookaheadReasoning

Scaling Speculative Decoding with LOOKAHEAD REASONING
Yichao Fu'
Rui Ge l
Zelei Shao
Zhijie Deng"
Hao Zhang'!
UCSD
Shanghai Jiao Tong University
Abstract
Reasoning models excel by generating long chain-of-thoughts. but decoding the resulting thousands of tokens is slow. Token-level speculative decoding (SD) helps, but its benefit is capped, because the chance that an entire y-token guess is correct falls exponentially Y grows. This means allocating more compute for longer token drafts faces an algorithmic ceiling making the speedup modest and hardware-agnostic. We raise this ceiling with LooK AHEAD REASONING, which exploits second, step-level layer of parallelism. Our key insight is that reasoning models generate step-by-step. and each step needs only to be semantically correct, not exact token matching. In LOOKAHEAD REASONING, lightweight draft model proposes several future steps; the target model expands each proposal in one batched pass, and verifier keeps semantically correct steps while letting the target regenerate any that fail. Token-level SD still operates within each reasoning step, the two layers of parallelism multiply. We show LOOKAHEAD REASONING lifts the peak speedup of SD both thcoretically and empirically. Across GSM8K, AIME, and other benchmarks, LOOKAHEAD REASONING improves the speedup of SD from 1 1.4x to 2.1x while preserving answer quality, and its speedup scales better with additional GPU throughput. Our code is available at httpp:: /e::::/gtttit thub com/hao-ai an att.aaaaaaaooooatteaaaaaonnnnoaa
Introduction
Large reasoning models (LRMs) have recently pushed the state of the art in math problem solving and program synthesis by generating explicit, long Chains of Thoughts (CoT) [ In these models, an answer unfolds sequence of intermediate reasoning "steps". and each step arrives token by token via autoregressive decoding. a solution needs N steps and each step needs T tokens, the model must generate (NNT) tokens, often running into tens of thousands of tokens and minutes of wall-clock time. For instance, OpenAl's ol model [2] may take more than minutes to solve single problem from the International Mathematical Olympiad (IMO) challenges.
Speculative decoding (SD) mitigates this token-level dependency by spending additional FLOPs to shorten the critical path of generation: cheap draft model proposes y future tokens and the expensive target model then verifies them in parallel: if every guess matches, the decoding can fast forward + positions at once. However, in the face of LRMs with long decode, two facts limit how far this idea can scale. First, the probability of an entire yttoeen sequence correct drops almost grows. exponentially with ($2). so the expected number of accepted tokens quickly saturates Second, the verifier must still verify the target logits for all 7 positions, and that cost grows linearly. This results in speedup curve that climbs with small 7, plateaus after few dozen tokens, and can even decline once the verification cost dominates. For example, in a real profiling, we observe SD's speedup caps at 1.4x (Figure 2. As this ceiling is algorithmic rather than hardware-bound, this means
Work done during an internship at UCSD. i Corresponning authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Draft
Prefix
Prefix
51
Draft Step
Target
Prefix
Target Step
5g
8g
Lookahead Reasoning
speeduy
Prefix
S
Autoregressive
Prefix
5,(5 fi)
(-
(y sy)
Decoding
Time
2t
2t+T
3T
One cycle LOOKAHEAD REASONING. The draft model proposes r steps (5), 82. $3). The target model then generate {$1, 82, 83) based prefixes and ($1, 62, 83). respectively. Verifier checks if draft and target steps are semantically equivalent (e.g., 81 R 81). If the first two steps are equivalent but the third is not, LOOKAHEAD REASONING outputs the verified draft steps (61, 82) followed by the target's correction (#3). This allows accepting multiple steps with only lowered latency (e.g.. 2t + T) compared to the sequential target calls in autoregressive decoding (e.g.. 37'), where is draft step time and target step time.
Figure 1:
Speedup
1.8
1.6
Ouiss NGRAM NGRAM
12
50
100
150
200
draft takens
Figure 2: Speedup vs Draft Tokens. Speedup over autoregressive decoding, comparing LOOKA- HEAD REASONING combined with token-level SD (NGram-based (red line) to SD alone (blue line). Our method orthogonal to token-level and improves the maximum speedup from 1.4x to 2.Ix.
that allocating more FLOPs in yields only diminishing returns, making SD's acceleration not scale with future accelerators. As LRMs produce ever-longer generations, the number of tokens SD can safely skip does not grow proportionally, so the end-to-end latency remains substantial.
This paper makes key observation that reasoning is naturally hierarchical: a full chain-of of-thu breaks into discrete steps, and each step unrolls tokens by token. To reach the correct answer, reasoning step requires only semantic correctness but not exact token matches. To illustrate, we replaced over 50% of DeepSeek-RI 32B's s reasoning steps with semantically equivalent ones from another smaller model. The impact on overall task accuracy was minimal, with deviations typically exceeding 2% ($4.T). This looser requirement exposes coarser unit for speculation: in addition to guessing the next few tokens, model can guess and verify the next few reasoning steps. These step proposals and verification are independent, so they can batched and executed parallel, making full use of GPU's batching capacity. At the same time, token-level speculation can still operate within each step, achieving two complementary layers parallclism rather than one.
This paper develops LOOKAHEAD REASONING based on this insight, with one operational cycle shown in Figure ! First, a lightweight draft model autoregressively generates several sequential, future reasoning steps {$1, $2, 83). Concurrently, the target LRM generates corresponding follow- up steps {81, $2, 83), where each s; generated based on a prefix formed by the initial context concatenated with the sequence of preceding draft steps $1..... $;-1. Notably, the generations of {81.82, 82, 83 } are issued as a batch running in parallel to exploit additional cross-request parallelism on GPUs. A lightweight verifier, implemented then begins with the first speculative step to determine if the draft's original speculative step & semantically aligns with this oracle step . If the step passes verification, we keep it and proceed
small M-L--as-a-Jude
an embedding model,

---

to the verification of &2 and 82, which are already available due to batched execution. If it fails, we drop it and revert to the target model's ss standard generation. Concurrently, token-level speculation can operate independently when the target/draft model generates the content of each step.
LOOKAHEAD REASONING operates at step level, axis orthogonal to token-level speculation. Because step-level speculation absorbs compute that would otherwise hit the speedup ceiling by token-level speculation, the method scales better with hardware. Additional FLOPs can draft more (or deeper) steps instead of lengthening token guesses, sidestepping diminishing returns faced by liken-level-only speculative decoding. For example, on GSM8K, LOOK AHEAD REASONING lifts token-level SD's peak speedup from 1.4x to 2.1x (combined), as depicted in Figure 2 Even when available compute is limited, we prove that the peak speedup given limited compute shall be achieved only by combining both levels of speculation ($3.3).
A key design consideration is the choice of verifier. While ideal semantic verifier ensures no accuracy loss, practical ones balance compute cost against judgment accuracy. For instance, a looser verification may boost draft acceptance (and speedup) but risks accuracy drop from erroneous steps. We finally opted for a 7B LLM-As-a-Judge, striking a balance between these competing factors
To sum up, our contributions can be listed as follows: (1) We develop LOOKAHEAD REASONING, a novel step-level speculation dimension to scale speculative decoding, orthogonal to existing token- level approaches. (2) We present theoretical analysis demonstrating significant speedups from our method, both as a standalone technique and when combined with token-level SD. (3) We conduct extensive experiments showing consistent performance improvements across diverse datasets.
2 Background
Speculative Decoding. LLMs autoregressively generate one token time, with each next token
2++1 sampled from the distribution P(æ++1 x1+)) This sequential dependency poses a funda- mental bottleneck inference speed. Speculative decoding (3 mitigates this challenge using "guess-and-verify" strategy with two models: a lightweight draft model @ and a strong target model p. Given context æ1:t. q autoregressively proposes sequence of ? candidates tokens, Øt 1st+ along with their draft probabilities Q. Subsequently, p verifies these tokens in single parallel forward pass, yielding the target probabilities P. A each proposed token ti+s- If a token is accepted, it is appended to the output; if rejected, the process halts, and final token is sampled from p's distribution based on the last accepted token. This allows for the acceptance of n S tokens in fewer steps than standard autoregression.
procedure then sequentially processes
The theoretical speedup achieved by this speculative decoding approach, assuming negligible verification overhead beyond the target model's 's single pass, can be characterized by:
1 1-a77 7++1
g(7) (1-
where a represents the average acceptance rate of each token drafted by q (assuming it's independent for each token in the sequence). Note that the probability of an entire sequence of y tokens being
accepted typically decreases exponentially with 7 as eтrors accumulate. = TT is the latency ratio of generating single token by the draft model relative to the target model, where Tg and Tр are step time of and p, respectively. Furthermore, for all c > 0 and 7 2 0, the speedup g(7) is upper-bounded by 1/(1 a), limit approached only in the idealized case where
O. This inherent upper bound on s(7) signifies critical limitation: beyond an optimal point, investing more computational resources by increasing the speculative length (7) yields diminishing or even negative returns speedup. Therefore, this algorithmic ceiling implies that the acceleration gains from loken-level speculative decoding do not scale with improvements in hardware, such as more powerful GPUs. Consequently, for reasoning models with longer CoTs, the bounded acceleration offered by token-level speculative decoding alone highlights an urgent need for more potent acceleration strategies.
LLM Reasoning. Large reasoning models (LRMs) 2 are increasingly pivotal for complex tasks such as math problem-solving and coding. These models often generate solutions by giving a "chain- of-thought" (CoT)-a sequence of intermediate reasoning steps, denoted as 8, produced step-by-step to derive a final answer [[] Each reasoning step (s;) typically conditions on the previous one (s;-1),
3

---

creating sequential dependency analogous to token-level autoregression but at a higher conceptual granularity. We observe that this step-wise structure itself presents significant opportunity for acceleration. Specifically, entire reasoning steps can be speculatively proposed by a draft model, denoted as 8. Our preliminary experiments ($4.1) show this potential. small 1.5B draft model can generate speculative steps & that semantically align with over 50% of ground-truth steps s from much larger 32B target model. Besides, this is achieved while maintaining comparable task accuracy.
3 Method
In this section, explain the details of LOOKAHIEAD REASONING, then provide theoretical analysis that shows its performance benefits. Furthermore, since both step-level and token-level speculative generation rely on increasing concurrency, we show that in real-world settings- where the two methods compete for limited concurrency resources peak performance gains can only be achieved when combining both speculative strategies together.
3.1 LOOKAHEAD REASONING: Step-Level Speculative Generation Solution
The core idea of LOOKAHIEAD REASONING to perform speculation and verification on entire steps rather than individual tokens. To put clear, first presented synchronous version of this approach in Algorithm[|] and then conceptually illustrate an optimized asynchronous variant in Figure
As detailed in Algorithm (sync version), one cycle of LOOKAHEAD REASONING proceeds as follows:
1. Draft Step Generation:
Given token prefix Tia, the draft model q first autoregressive generates a sequence of 7 candidate steps, denoted 60, 60...., 8, Each step 8j is generated conditioned on the prefix extended by all preceding draft steps: I1.t + ①i- sk- In practice, we simply use "In'n' the step break as we found I's common flag for step split in various reasoning models [5 4 6).
2. Parallel Target Step Generation: Same as in the standard speculative decoding [3] once all ? draft steps available, target model p generates following steps So. 8o....,88, 8, accordingly in parallel. 3. Verification and Output Construction: The algorithm then determines the longest prefix of accepted draft steps. Verification between each draft step 8, and its corresponding target step 8, is performed by a verifier V 8j), which assesses whether if 85 is an acceptable substitute for 8j. i.e. whether they are semantic similar.
Algorithm 1 LOOKAHEAD REASONING (Sync Version)
Input: Draft model q. Target model Prefix æi... Max lookahead steps 7, Verifier V(+.) (TTuue, False) Initialize empty step sequences $p...., 8y $o...., 8y
2: Zuurrent Zi:
do 4: 6, = q. genneaateSttpp [eureat); Teurrnt
3: for j 0 to
b Generate y draft steps sequentially
Teurrent @ 8j
5: in parallel do for
0 to 6: Let  = In.t e +i
Compute target steps ay in parallel based on draft prefixes > Prefix before draft step
1 & jf 2 1 else Cr.t
7: = (eeratteSSep)()
8: end parallel
9: min (0... 10..--13 V(*,, Palse) U {7)
> Find first unaccepted step using V > Append verified drafts decisive target
OutputSequence = (-1 8t) $ 83
Output: OutpusSequence
Verifier Selection. The choice of verifier (V) pivotal design consideration in LOOKAHEAD REASONING. While an ideal semantic verifier ensures accuracy loss, practical implementations face primary trade-off between judgment precision and computational overhead; Furthermore, the strictness of verification (e.g., threshold) presents secondary trade-off, potentially boosting draft acceptance and speedup at the risk of degrading task accuracy from erroneously accepted steps. We explore three common paradigms for semantic assessment-L.LMMM -LL.M-L--------d-e
7 for nuanced
evaluation, embedding- based verifier g for efficient similarity, and target model scoring [9 each
with distinct cost prscciccoiioon profiles, empirically evaluating their impact in $4.3

---

Asynchronous Generation (Illustrated in Figure 1 In Algorithm 1 the parallel verification steps launch only after all y draft steps 8, with j E {0... 1} are produced. An optimized asynchronous implementation, conceptually depicted in Figure I can begin generating a target step 8, as soon as its required prefix (containing æ1:t, 80; ..... 8 becomes available from the draft model. This asyne execution brings overlap for draft/target generation and verification phases, which can significantly reduce end-to-end latency compared with the synchronous version. Note that in this asynchronous mode implementation, both the draft and target models will "lookahead" steps, ensuring maximal utilization of parallel processing. This is different from the sync version that draft model generate y drafts while target model generate y + 1 steps in each cycle.
Multi-Branch Drafting. To further increase the number of the accepted reasoning steps, we explored tree-structure generation where the draft model proposes multiple candidate steps at each speculative position. Specifically, instead of generating a single candidate chain, the draft q can propose a set of W alternative steps for each position in the draft sequence. Once step is generated, the draft then proposes W child candidates in parallel for the subsequent position j + This branching process continues up to a maximum y steps, leading to an exponential growth in the total number of candidate sequences explores, i.e., W7 The target model p, however, still generate one single candidate continuation step for each position j (based on the draft prefix). The verifier V would then check if any of the W proposed draft branches for that position j semantically aligns with the target model's 'ss step. If such match is found, that branch is accepted and other branches are discarded. This multi-branch strategy aims to boost the likelihood of speculative success, albeit at the cost of increased computational effort in the drafting phase. We discussed its trade-off in $4.3
3.2 Theoretical Speedup Analysis for LOOKAHEAD REASONING
We now analyze the potential performance gains of LOOKAIIEA REASONING. We make simplifying assumptions for clarity: negligible verification overhead, constant cost for generating steps, and single draft branch each stage.
Notation:
Let r be the maximum number of draft steps generated sequentially, and k = mm be the number of target steps that generate in parallel. Let T be the wall-time cost for the target model p to generate one step, and ea T be the cost for the draft model (0 < a < 1). Let on E (0, 1) be the probability that a draft step is accepted.
Step-Level Speedup for Sync Version of LOOKAHEAD REASONING: The latency speedup for sync Lookahead Reasoning is
Sayne (k1)
an) 9 tc
Step-Level Speedup for Asyne Version of LOOKAHEAD REASONING: The speedup depends on
whether the draft generation is limited by the maximum depth i by the relative cost C₁. Let X; be the number of consecutively accepted draft steps in stage i. The expected number of accepted steps before a rejection is E[X,] = a1/(1 a1).
We define the asymptotic speedup S as the ratio of steps generated by LOOKAHEAD REASONING compared to a target-only baseline over the same wall-time. Two cases arise:
kn 2 [1/a]]: ]: The draft model is relatively slow, and generating kn drafts takes longer than one target step. The depth limit k1 is effectively inactive. The speedup converges to:
1 E[X.] S 1+ [
1
ci +1
- a1)
2. h1 < [1/ca]: The draft model is fast enough to generate k steps within time T. The maximum depth k limits the number of speculative steps per cycle. The speedup converges to:
E + X,]
S2 FI[(X, + 1)/k,] + en (X, mod kj)
(1 - a1) + a (0 afth k1 a1j K1
(Detailed derivations are provided in Appendix B Let fasyno(ki) denote the step-level speedup, where (k)) = S₁ or S2 depending on the case.
5

---

3.3 Optimal Speculation Strategies under Concurrency Constraints
Token-level speculative decoding 3 and step-level speculative decoding are orthogonal each other. If k2 is the number of additional tokens speculated by the draft model within each step generation (for both draft and target models, assuming they use internal speculation) and eg is the ratio of execution time of the draft model and target model in speculative decoding, its speedup is given by (kk2), based on the token acceptance rate 0:2:
1
(k22)
(1 - a1) c1
Since these mechanisms operate different granularities (inter-step intra-step), their speedups multiply, yielding a combined speedup h(k,, k2) = f(11) x g(k2). However, these two orthogonal parallelism dimensions compete for computational resources, making it crucial to determine the optimal resource allocation strategy to achieve maximum speedup. In this work, we focus on a fundamental question: is using both speculation methods superior applying only method?
Optimality of Hybrid Approach under Budget Constraint:
In real-world systems, memory and computational constraints necessitate capping the total degree of parallelism (M) available to the target model, i.e., ParallelDim, Parallel Dim, for step-level and leven-level speculative decoding methods, respectively. This constraint transforms our earlier question into resouree allocation optimization problem: given finite parallel budget (M), should we distribute resources across both parallelism dimensions or concentrate them on single method? Consequently, our design goal becomes:
max Parallel x Paa Dim SM
h(k1, k2) f(ki) g(k2).
(1)
max M
f()) 9(72)-
(2)
Where ParallelDimg = k2, and
Darm =
kii min
sync
1,
asyne
It's easy to see that if we set kn = 1, then we are using purely token-level speculative decoding, whereas if we set ko 1, then we are using purely lookahead reasoning.
Theorem (Hybrid Method Optimality for Asyne Algorithm LOOKAHEAD REASONING): Under the conditions of acceptance rates (0.52 < 0 022 < 0.8), reasonably efficient draft models (c) <
R C2 < )), and sufficient parallelism budget M 2 16, the maximum speedup la(ki, k2) is achieved and only if hybrid strategy employed, meaning both k > 2 and ka 2.
These conditions are broadly representative of real-world scenarios: acceptance rates (a1, 02)) in the 0.52-0.8 range common in speculative decoding [10] and our experiments ($4.1): draft model efficiency ratios (ci) below 1 and (c2) below 1 are also common; and the parallelism budget (M 2 16) reflects typical GPU capabilities. This analysis demonstrates that neither pure step-level nor pure token-level speculation optimal under a fixed parallelism budget. The highest theoretical speedup is obtained by judiciously combining both strategies, leveraging parallelism across steps (ki) and within steps (k2). This motivates architectures and systems that can effectively manage and exploit both levels of speculative execution. It is empirically validated in $4.2 We provide complete proof in Appendix B.1.2. Additionally, we analyze in detail the conditions under which single methods (either token-leves level step-level) outperform hybrid approaches, and conversely, when combining both methods yields superior performance (Appendix B.1.2).
4 Experiment
Models. We evaluate two popular open-source reasoning model series: DeepSeek-RI-Die 4 and Qwen3 (6). For the DeepSeck-RI-Dee -i--stiill series, the 1.5B version serves as the draft model and the

---

32B version as the target model. Similarly, for the Qwen3 series, the 1.7B model is used as the draft model and the 32B model as the target. Unless otherwise specified, employed as the judgement model. A deliberately designed judge prompt template allows our model to assess the semantic alignment between two sentences in just one prefill pass (Appendix [A).
-reen..5-BBIIstrruue [TT] is
Datasets. Our evaluation spans a suite of benchmarks, aligning with previous speculative decoding re- search [3] and reasoning model evaluations 6). For code generation, we use HvaanEEal [14 and LiveCodeBench [5]. Math reasoning tasks are assessed using GSM8K [6]. AIME' 24 [7.. and
AMC12'23 [8] For question answering, we include GPQA [9 and MT-BBench 77. Specific to dataset sampling, we utilize 40 out of 50 problems from AMC 1222, selected by Qwen2.5 Math (20]
and randomly sample 100 queries from the 1.3K GSM8K test set. For LiveCodeBench, We select 268 problems collected between August 2024 and Janaury 2025, following previous research [4]
General Parameters. LLM generation settings are configured specifically for each model series. For the DeepSeek-R 1-Distill series, we adhere to the official settings with a temperature of 0.6, top_p of 0.95, and maximum generation length of 32K. For the Qwen3 series, the temperature is set to 0.6, top_p to 0.95, min_p to O, top_k to 20, and the maximum generation length is 37K. These maximum generation lengths are chosen to ensure complete outputs. We use prompt-lookuu decoding (n-gram) 21 as a representative speculative decoding (SD) method: the max lookup tokens is set to 1 for GSM8K and for other datasets. The number of speculative tokens is set to for SD and the number of speculative steps is set to 6 for LOOKAHEAD REASONING by default.
Testhed. Experiments conducted on server equipped with eight NVIDIA H100 GPUs. Target models (32B) are deployed across two H100 GPUs using tensor parallelism. Draft models (1.5B/1.7B) and the default judge model (Qwen2.5-77) -.--....tt--ttt are each deployed on single H100 GPU. Our algorithm is built upon the vLLM v0.8.3, Both the baseline and our proposed method are evaluated using vLLM (22 to simulate real-world LLM serving conditions.
Evaluation Metrics. For code generation tasks (HumanEval, LiveCodeBench) and mathematical reasoning benchmarks (GSM8K, AIME'24 24, AMC12'23), we accuracy (or pass @ 1). In question answering, accuracy is used for GPQA, while MT--Benddd scores are obtained using Qwen2.5-72B- Instruct [I] as the judge. The accuracy livecodebench is averaged over 8 samples while the accuracy on other datasets are averaged over 16 samples. We calculate the acceptance rate over the entire generation process and the accuracy of the final generated text. The evaluation procedure works as follows: at each generation step, we obtain outputs from both the draft and target models, then use verifier to determine whether accept reject the draft output. The accepted result is added the history trajectory, and this iterative process repeats until the end of generation is reached.
4.1 End-to-End Performance of LOOKAHEAD REASONING
We evaluated the end-to-end performance of LOOKAHEAD REASONING (LR) across diverse bench- marks using DeepSeek-RI -D-istilll and Qwen3 pairs. The detailed results are presented in Table [ key finding is LR's 's consistent ability to preseve task accuracy. Across a variety of benchmarks, LR's s accuracy varies within a narrow range relative to the target model's 's autoregressive baseline, from approximately 1.0% above to 2.1 1% below baseline performance. This accuracy preservation contrasts with SpecReason, which exhibited more noticeable accuracy reductions (e.g., dropping from 91.8% to 85.9% on GSMSK with Deepseek-Ri 1, a ~ 6% decrease). This underscores LR's design principle of preserving output via robust semantic verification.
several tasks
Furthermore, LR achieves strong accuracy while maintaining high step acceptance rates, often above 50% and reaching up to 63%. These substantial acceptance rates empirically support our initial insight that a smaller draft model can effectively predict semantically correct reasoning steps for a larger target model. LR also delivers significant efficiency gains. Its step-level parallelism is orthogonal to teven-level speculative decoding, and their synergy produces substantial speedups. LR alone achieves speedups ranging from 1.04x to 1.71x across various benchmarks and model pairs. When combined with n-gram SD, the total speedup further amplified, reaching up to 2. Ix. This combined approach consistently outperforms n-gram SD alone, demonstrating the added value of step-level speculation. These results, consistent across both Deepseek-R and Qwen3 families, underscore the generalizable acceleration benefits of LR.
7

---

Table I: LOOKAHEAD REASONING's Performance Across Datasets. Speedup is relative to the Autoregressive Decoding of the respective Target Model. We use W here.
Dataset
Method
Metric
AIME24
AMC23
GSM8K
HumanEval
GPQA
MT-Bench
LiveCodeBench
Draft:
Deepseck-Rl-Distin
1.5// Target: Deepseck-RI. RI--Dstill 32B
Draft Model
Acc. (%)
2.9
716
77.6
2.4
9.6 1.2
6.23 1.9"
14.5
Target Mode
Acc. (%)
70.8 + 5.2
9.3
91.8 1.9
96.9 0.8
63.3 =
8.17 1.2
48.9 #: 1.3
SpecReason
Aec. (%)
58.3 4: 5.7
00.6 =: 2.6
85.9 2:2
94.5 1: 1.5
57.0 :: 2.8
40.6 1.5
Apt.
0.39
0.69
0.93
0.43
0.08
0.25
Acc.
69.2
91.1
92.8 +
95.5 +
61.2 = 2.8
8.13
49.5
LR(ours)
Apt.
0.47
0.58
0.63
0.44
0.36
0.48
0.47
Speodup
1.36x
1.48x
1.71x
1.27x
1.14x
1.27x
1.21x
SD
Speedup
1.53x
1.50x
1.39x
1.32x
1.48x
1.25x
1.45x
SD+LR(ours)
Speedup
1.82x
2.00x
2.11x
1.54x
1.63x
1.51x
1.58x
Draft: Qwen3 1.7B/ / Target: Qwen3 328
Draft Model
Acc. (%)
46.9 # 8.1
81.2 1.7
91.1 + 1.6
85.4 1.6
38.5 1.4
7,96 # 1.5"
28.8 1.6
Target Model
Acc. (%6)
80.0 :: 3.9
97.5 = 2.0
96.6 1.4
97.6
68.2 2.1
8.53 :: 1.1"
52.4
SpecReason
Acc.
68.3 5.3
90.5
94.5 1.5:::1..
92.0 2:00::2:00
66.3 ::
39.7
Apt.
0,75
0.92
0.95
0,91
0.46
0.65
Acc. (%) Apt.
80.4 + 4.1
96.4 2.0
96.4 1.2
97.1 0.8
68.5 2.4
8.46 1.15'
51.7 + 1.7
LR(ours)
0.53
0.50
0.39
0.30
0.38
0.40
Speedup
1.12x
1.22x
1.32x
1.13x
1.04x
1.10x
1.08x
SD
Speodup
1.40x
1.38x
1.32x
1.32x
1.10x
141x
1.25x
SD-LR((urs)
Specdup
1.49x
1.62x
1.68x
1.39x
144x
1.49x
1.32x
Note: LR (ours) refors proposed Lookahead Reasoning method. SD denotes token-level Speculative Decoding (N-gram based). Acc. stands for Accuracy (9) and Apt. for Acceptance Rate. For Benceench (marked with ), the reported metric is stundard score (0-9 scale) instead of ascuracy. "." indicates data not applicable. Speeduu relative the autoregressive decoding of the respective target model.
4.2 Combining LOOKAHEAD REASONING with Speculative Decoding
To empirically validate the orthogonality of LOOKAHEAD REASONING (LR) with speculative decoding, we conducted experiments using prompt-lookup decoding (n-gram) on the AIME dataset.
Figure 3 shows the orthogonality of LR and Speculative Decoding (SD). Subplot (a) shows that while LR alone with varying draft step number reaches speedup around 1.4x, adding SD boosts this to approximately 1.9x. Similarly, subplot (b) illustrates that SD alone with varying Speculative Token Numbers peaks around 1.55x speedup, but combining it with LR again achieves up to 1.9x. Collectively, these results highlight that while either method in isolation offers limited gains, their combination consistently yields the most significant performance improvements, aligning with our theoretical analysis in 3.2
2.0
1.8
Speedup
LR Only
Speedup
IR 5D
Combining Gain
Only
1.4
1.4
LR SD
Combining Gain
10
15
20
23
30
20
30
Speculative Steps
Speculative Tokens
(a) vary the draft step number for LR
(b) vary the speculative token number for SD
Figure 3: Orthogonality of Lookahead Reasoning and Speculative Decoding. When used alone, the speedup from both LR and SD is limited by their draft length (7). However, their combination consistently improves the max achievable speedup.

---

4.3 Ablation Study
Effectiveness of the Verifier. We conducted an ablation study to assess the impact of different verifier mechanisms on task accuracy, utilizing CrepSeek-R1-Diseee 32B as the target model and 1.5B parameter model as the draft model on GSM8K and AIME'24 datasets. We compare four verifiers: (1) Random Acceptance of drafts; (2) LLM-as-a-Judde (LLM--) with Qwen2.5 7B/32B []];; (3) Embedding-based Verification (Emb.) with all-mpnet-base-22 model [8] 0.85/0.95 similarity thresholds; and (4) Reasoning Model Scoring(Score) [9]. using the target model to assign 0-9 scores with acceptance thresholds of 7/9. Results are Table
LLM-J verifiers (both 7B and 32B) showed robust accuracy preservation across both datasets, with minimal performance difference observed between the two verifier sizes. On GSM8K, their performance closely aligned with the original baseline, indicating no accuracy degradation. On AIME, LLM-J verifiers also maintained accuracy very close to the original, with observed deviations within approximately 1-2%. This contrasts sharply with Random Acceptance. Despite comparable or lower acceptance rates (e.g., 0.50 on GSM8K and 0.40 on AIME), Random Acceptance led to significant accuracy degradation on both GSM8K (~ 3.5% lower) and AIME ( ~~ 11% lower). This underscores the necessity of a robust verification mechanism.
The Emmbedding-based verifier shows a trade-off: the stricter 0.95 threshold on GSM8K preserved accuracy (92.3 + 1.4%) at a lower acceptance rate (0.37), while the 0.85 threshold, despite higher acceptance rate (0.56), resulted in a ~ 2% accuracy drop (89.8 + 2.5%). This pattern was mirrored on AIME. This indicates that while semantic equivalence is promising criterion, its efficacy in preserving accuracy is highly dependent on the stringency (and precision) of the similarity judgement.
Reasoning Model Scoring, which assesses draft quality via target model scores rather than direct equivalence, consistently underperformed in accuracy preservation. For instance, even employing the stricter Threshold 9 resulted in notable accuracy reductions of approximately 5.9% GSM8K and 12.5% on AIME. The still relatively high acceptance rate on GSM8K with this threshold (e.g., 0.93) suggests that the scoring mechanism may possess limited discriminative power on simpler datasets, even at stricter thresholds. This highlights fundamental limitation: quality scores, even with high thresholds, do not reliably ensure alignment with the target model's output distribution, which is critical for LOOKAHEAD REASONING'S 's correctness.
These results reveal that verifiers grounded in semantic equivalence with the target model's likely output are most effeetive for preserving accuracy within LOOKAHEAD REASONING. LLM-as-a-Judde excels in this, provding nuanced judgment, though with potential computational overhead. Embedding models provide a lightweight alternative (e.g., all-mpnet-base baseetbbaeev2 is only ~100 parameters), where performance is tunable via the similarity threshold, offering cost-effective solution.
Table 2: Performance comparison with different verifiers on GSMSK and AIME datasets. Apt.: accept rate; Acc.: accuracy (%).
LLM-J (Qwen)
Emb. (Th.)
Score (Th.)
Dataset
Metric
Orig.
Rand.
7B
32B
0.85
0.95
GSMBK
Apt.
0.50
0.63
0.58
0.56
0.37
0.97
0.93
Acc.
91.8 1.9 88.3 3.7 92.8 # 1.8 92.3 + 1.2 89.8 + 2.5 92.3 + 1.4 82.1 = 2.4 85.9 : 2.2
Apt.
0.40
0.47
0.46
0.45
0.38
0.81
0.39
AIME
Acc.
70.8 5.2
59.6 = 5.4 69.2
= 8.1 69.0 + 4.7 64.0 + 6.5 66.7 + 6.3 37.9 = 7.5 58.3 + 5.7
Effect of Tree Width on Performance We investigate the impact of speculation tree width on LOOKAHEAD REASONING's accuracy, accept rate, and specdup. keeping depth y = 2. In LOOKA- HEAD REASONING, candidate sequences grow as W Wider trees (W 1) can boost accept rate but escalate FLOPs and, with imperfect verifiers, risk accuracy degradation due to erroncous acceptances. We hypothesize stronger verifier mitigates this. Experiments on GSM8K and AIME24 used Qwen2.5-775 2- -BInstruct and Qwen2.5-33 3-nttttutttt as judges. Results are demonstrated in Table 3
Increasing W consistently raised accept rate across datasets and judges (e.g., GSM8K with Qwen2.5- 7B: accept rate 0.63 for W = to 0.83 for W = 8). However, this rarely translated to better speedup beyond W = 2. Further widening often diminished speedup (e.g., AIME24 with Qwen2.5-
7B, W = 8 yielded no speedup), likely due to the exponential overhead outweighing accept rate gains. Accuracy trends highlight verifier importance. With the Qwen2.5-7B judge, increasing

---

W led to noticeable accuracy drop, especially on AIME24 (from 69.2% at W = to 64.6% W = 8), supporting our hypothesis. The stronger Qwen2.5-32B judge demonstrated greater resilience: accuracy remained more stable on GSMSK, and the degradation on AIME24 was less pronounced (69.0% at W = to 67.3% at W 8). This indicates stronger verifier is crucial for wider trees to manage the increased risk of incorrect speculation.
Table 3: Impact of Tree Width (W) on Performance Metries (Depth y = 2)
W=l
W=2
W=
W=8
Dataset
Judge
(%c.(%)
Apt.
Spd.
(cc(%)
Apt.
Spd.
AccX%)
Apt.
Spd.
Acc.
Spd.
GSMBK
Qwen7B
92.8 +
0.63
1.48x
91.2
0.73
1.49x
91.1 1.7
0.77
1.47x
91.5 1.8
0.83
1.25x
Qwen32B
92.3 = 1.2
0.58
1.40X
93.2
2.0 0.66
1.42X
92.8 0.73
1.39x 92.5 1.5 0.77
1.19x
AIME24
Qwen32B
Qwen7B
69.0 1 4.7 0.46
29.2
0.47
1.27x
1.23x
67.3
69.0 i. 6.7 0.54 1.23x
0.58
1.32x
6..4
68.1
0.67
0.59 1.17x 67.3
6.66
0.68
1.00x
0.98x
Note: Acc.:
Accuracy (%); Api.: Accept Rate; Spd.: Speedup (relative to torget model antoregressive decoding). Depth yy = Qwen2.3- 78/32B- Instruct are judge models.
5 Related Work
Speculative Decoding. There are many different types of speculative decoding approaches. Draft- head methods like Medusa 23 Hydra (24). and EAGLE (25 TO 13 integrate auxiliary heads into the target model to propose sequences. In contrast, Jacobi- based approaches such as Lookahead Decoding [2 and CLLM (26 enable paralfel n-gram generation without draft models. System- level efforts [27 28 further optimize SD's runtime efficiency in serving systems. LOOKALIED REASONING introduces a complementary form of step-level speculation tailored for reasoning models, enabling a new dimension of parallelism orthogonal to token-level methods.
to scaling inference-time compute 31 32 [3), enabling large reasoning models (LRMs) like OpenAl o3/o4 (5), Kimi- K1.5 34. and DeepSeck-RI 4 to generate longer CoT to solve more complex problems in many steps. Recent work has begun to leverage this inherent step-by-step structure to accelerate LRMs. For instance, Speculative Thinking 35 uses LRMs to guide a smaller draft model, while SpecReason [9] accelerates reasoning by employing the large model to score the output of small model, thereby deciding whether to accept its generated step. However, unlike LOOKAHIED REASONING, these methods do not pursue a step-level equivalent with the original target model to accelerate reasoning.
LLM Reasoning, Recent trends shift from scaling model size 29
30
Limitation and Conclusion
This paper introduces LOOKAHEAD REASONING, a novel method for accelerating large reasoning models during long CoT reasoning. LOOKAHEAD REASONING adds new step-level dimension of parallelism, complementing traditional token-level speculative decoding. Our approach uses a draft model to propose future reasoning steps, which then semantically verified by the target model. Evaluation on various datasets using two open open-source reasoning models show that it can achieve up to 2.1X speedup combined with speculative decoding. This highlights LOOK AHEAD REASONING's effectiveness in making large reasoning models faster. This work has limitations that suggest future improvements. First, using "Inin" to split reasoning steps is simple but may miss optimal breaks; smarter segmentation is needed. Second, current verifiers still trade speed for accuracy: designing lightweight but robust semantic verifiers remains an open challenge. Third, LOOKAHEAD REASONING introduces non-trivial implementation complexity, as integrating lookahcad exccution into production serving engines (c.g., SGLang or vLLM) requires additional scheduling logic to manage the step hierarchy. Some hyperparameters, such as W and 7 also require tuning to balance aggressiveness and stability, which may introduce additional engineering overhead in deployment. While LOOKAHEAD REASONING improves per-request latency by batching candidate steps. it increases instantaneous GPU utilization and KV cache usage. This trades computation for speed and may reduce overall throughput under fully saturated GPU conditions. However, like other speculative decoding methods, it particularly beneficial in latency serving scenarios or when spare compute capacity is available.
10

---

Acknowledge
We sincerely thank the anonymous reviewers for their insightful and constructive feedback. We gratefully acknowledge MBZUAI and NVIDIA for providing the computational resources that made this work possible, and we thank Yiming Zhao for his contributions in open-sourcing the project. Z. Deng is partially supported by NSF of China (Nos. 92470118 18, 62306176 76) and Natural Science Foundation of Shanghai (No. 237R 177711487700
References
[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24427774-24877, 2022.
[2] OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey,, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum. Angela Jiang, Ashvin Nair, Barret Zoph. Behrooz Ghorbani. Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzic, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong. Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang. Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo. Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng. Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O' Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglckar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe. Michael Malek, Michele Wang. Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino. Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu. Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card, 2024.
11

---

[3] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. International Conference on Machine Learning, pages 19274-19286.
PMLR, 2023.
[4] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
[5] OpenAl. Openai o3 and 04-mini system card. April 2025.
[6] Qwen Team. Qwen3 technical report. 2025.
[7] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595 46623, 2023.
[8] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Association for Computational Linguistics, 11 2019.
[9] Rui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro, Zhihao Jia, and Ravi Netravali. Specrea- son: Fast and accurate inference-time compute via speculative reasoning. arXiv preprint arXiv:250.007891, 2025.
[10] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406. 16858, 2024.
[11] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[12] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint arXiv:2402.02057 2024.
[13] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time time test. arXiv preprint arXiv:2503.01840
2025.
[14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel BeeberttVosss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
[15] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint Att::2403.079744 2024.
[16] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ /4168, 2021.
[17] AIME, Aime problems and solutions, 2025.
[18] AMC12. Ame 12 problems and solutions, 2025.
[19] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024.
12

---

[20] Qwen Team. Qwen2.5-math: The world's ' leading open-sourced mathematical 1lms, 2024.
[21] Apoorv Saxena. Prompt lookup decoding, November 2023.
[22] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-626, 2023.
[23] Tianle Cai, Yuhong Li, Zhengyang Geng. Hongwu Peng. Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple I1m inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774 2024.
[24] Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Began-Kelley, and William Brandon. Hydra: Sequentially-dependdnn draft heads for medusa decoding. arXiv preprint arXiv:2402.038 109, 2024.
[25] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077 2024.
[26] Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Cllms: Consistency large language models. In Forty-first International Conference on Machine Learning, 2024.
[27] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint mrXiv:2305.09788 2023.
[28] Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung, Zhijie Deng. Ion Stoica, and Hao Zhang. Optimizing speculative decoding for serving large language models using goodput. arXiv preprint arXiv:2406. 14066, 2024.
[29] Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 29X3: 2024.
[30] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08% 087744 2023.
[31] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 19393, 2025.
[32] Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf. Alex Xie, Graham Neubig, Ili Kulikov, and Zaid Harchaoui. From decoding meta generation: Inference-tine algorithms for large language models. arXiv preprint arXiv:2406.1683 2024.
[33] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling I1m test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03: 033144 2024.
[34] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501. 10t::299111199, 2025.
[35] Wang Yang, Xiang Yue, Vipin Chaudhary, and Xiaotian Han. Speculative thinking: En- hancing small-model reasoning with large model guidance at inference time, arXiv preprint vrXiv:2504.1232 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We list our contributions clearly at the end of the introduction, and the second paragraph of the abstract. We also clearly claimed our scoop in the beginning of the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of this paper discussed in $6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: Our assumptions are clearly list both in the paper and the appendix B The complete and correct proof was also placed in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provides open access to the data and code at
com/hao- ai- aitiao-aa-lab/LLooe wtw.-aaaboooaaaadaaaaaoooo
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Coaee//dddeSSbbmmcma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper does not train any model. The evaluation details are listed in $
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The error bars are suitably and correctly shown all our tables, like Table [
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detailed our experiments compute resources in
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:://wwe.ree..cccccccccctceea
Answer: [Yes]
Justification: Our research strictly adheres to the NeurIPS Code of Ethics as outlined in the official guidelines. Our work focuses exclusively on algorithmic improvements for reasoning efficiency in LLM-based serving, with no collection processing of sensitive personal data. We also ensure full transparency and reprodueibility of results, aligning with ethical standards for open scientific research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdictio)..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper does not discuss potential positive sociatal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., dentoyment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper doesn', 't have such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our paper utilizes various LLM serving engines (vLLM), LLMs (Qwen, Deepseek-R1 distilled models), algorithms (SpecReason, N-Gram), and datasets (MATH, AMC23, AIME24, LiveCodeBench, GSM8K, HumanEval, GPQA, MT-Bench). All assets are properly cited, and their corresponding copyright information is acknowledged. For each asset, we have ensured that the appropriate licenses and terms of use are respected and explicitly mentioned. Additionally, we have referenced the original paper, creators and specified the versions used where applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
18

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not provide new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: No core methodology is LLM- editing, and formatting that does not impact the core methodology, scientific rigorousness. originality of the research.
LLM- generated. LLM is only used for writing,
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non- non-standard components.
Please refer to our LLM policy (https: https:///nuuripe htt//onfeeennesee/eeeeeeeeceesee for what should or should not be described.
A Judgement Prompt Template
Semantic Equivalence Analysis Prompt
<im_startbssstem
You are Qwen, created by Alibaba Cloud. You are helpful assistant.clim_endee him_startbbuser
Evaluate whether the following two reasoning steps (sl and s2) convey exactly the same meaning. Focus on semantic similarity rather than exact wording.
Compare the main ideas, key points, overall message, logical structure, and numerical calculations/results of both reasoning steps.
If the reasoning steps convey essentially the results, respond with [aligned]. If the reasoning steps express different meanings, respond with [unaligned]. If is too hard determine, respond with [unaligned] Please directly provide the final result in [aligned] or [unaligned]. Reasoning step (sl): <start_st sl>
meaning and generate caleulation
<end_s>>
Reasoning step (s2):
<start s2>
<end s2><<im endi> <lim <im_startbassistant
This prompt template is specifically designed for Qwen2.5 Instruct models. It guides the model to directly output either "aligned" or "unaligned" as its judgment. Consequently, user can determine semantic equivalence between two sentences (s1 and s2) by simply checking if the model's initial output string begins with "ali" (the first three letters of "aligned")). Thus, only one forward pass needed to get the result and judgement latency can be largely saved.
20

---

B Detailed Speedup Analysis
B.1
Performace Gains Analysis
B.1.1
Speedup Analysis of Asyne LOOKAHEAD REASONING
For the analysis that follows, we assume all sequences are of equal length and that the draft tree contains exactly one branch at each layer. Moreover, we treat the verifier's 's overhead as negligible.
Notation.
? E N: maximum number of generations the large model performs in parallel.
T': cost (wall-time) of one sentence generated by target-model.
Cj cost of one traft-model run, measured in units of 'I' (so drafting costs e1 c1
[r (0, :cceppt rate of the drafts
X,: number of consecutive accepted drafts in stage i before the first rejection.
We view a full generation sequence of DRAFT STAGE, each which proceeds as follows:
1. If the number of generations the large model performs in parallel less than y. The draft model sequentially generate drafts.
2. Each time when we start to generate a draft step, we immediately ask the target model to generate target step.
3. After the target model finished generation, immediately ask the verifier to verify whether should accept the draft. If the draft was reject, fall back to the target model's original sentence and proceed to the next DRAFT STAGE.
Since each draft is accepted independently,
P(X, = a
E[X] = Q
Theorem 1. The latency speedup for sync LOOKAHEAD REASONING is 1-a²
(Ty (1-a)(1-cta)
Proof. The proof follows the same reasoning as in 3 The only difference is that our represents the maximum number of tokens the large model generates in parallel, whereas in their notation, corresponds to 7 +
Theorem 2. Let
total sentences generated by our algorithm in n DRAFT STAGE
total sentences generated by target-only model in n DRAFT STAGE
We can this as the latency speedup using async LOOKAHEAD REASONING algorithm. Then:
1. If 2 [2]. the draft tree never saturates. The parallel dimension of the target model
[1 and as co, the asymptotic speedup is
1
S1 (1- (1
2. 1 < 1. the draft tree is depth-limiten The parallel dimension of the target model is % and as oc, the asymptotic speedup is
a7 - a1 7(1 - a1)aa
S2 (1 - + a
(1
Proof. Over n stages, compare the total number of sentences generated by our algorithm that produced by the baseline (target-only) approach.
21

---

note that since each draft costs e the draft model can generate at most sentences during the time required for the target model to produce one sentence. Therefore, the draft tree never saturates, and the parallel dimension the target model effectively Moreover, in DRAFT STAGE i our algorithm spends
Case
When 7 2 01
T X
total time, Over that same interval, the baseline target-only model would have produced
T T+o TT
+ c1 X;
T
sentences, while algorithm emits 1 X, sentences. Thus stages, according the Law of Large Numbers,
E,(+,, n n+E, X (m(m) = (1 12 + E;
1 EX 1+ E Ci +1 (1-1))) a1)
Case 2 When y < []] the draft-tree saturates at depth y. So the parallel dimension of the target
model would be y To emit a total of X; + 1 sentences in stage i, we therefore proceed in
X
full intervals of length T, plus a final partial batch of size X; mod y. Hence the total wall-time for stage is
X T + T (X, mod 7).
Over that same interval, the baseline target-only model would have generated
X + C1 (X, mod 7)
sentences, whereas our algorithm emits X,, Therefore, + 00
n + - X, Ci EX, mod 7) (1-a)) + +allai aj -
a7
S2(n)
We put the calculation details in the appendix.
B.1.2 Optimal Speculation Strategies under Concurrency Constraints
In this section we show that under a fixed parallelism budget, the optimal inference speedup is always achieved by jointly applying step-level and token-level parallelism, rather than by using either in isolation. Concretely, let ?/2 denote the degree of parallelism used by speculative decoding and 02 be the ratio of the per-step execution time of the draft model to that of the target model. Then token-level speculative decoding alone yields 3
((12) (1 (2)
C2
02 72)
(Note: here the formula little different than the one in 33 due to different definition.) Next, pure step-level asynchronous parallelism scheme of depth 7 achieves speedup
a
S2 = -- + a (1 a c a a -1-60 (1-a1) 11
1
fasyno(11)
S1 + (1)11aa)
otherwise
and pure step-level synchronous parallelism scheme of depth 7 achieves speedup
(1)) = - (1-a))(1-0 +o004000 to m
(4)
22

---

When both schemes are combined, the resulting speedup factorizes:
f(m) x g(92)-
(5)
Since in real-world systems, due to memory or compute constraints, we often need to cap the total degree of parallelism 72 in hybrid speculative setting to M. In this case, may ask that given a finite parallel budget M, is it better that we combine these two parallel dimensions or we only use of them? Thus our design goal becomes
max ParallelDim, Paaraalellaaaaaaaaaaaaaaa S. h(m, f(m) g(2)).
(6)
Where ParallelDimg = 72, and
Dimm =
71. min 11h.
sync
async
It's easy to see that if we set 7/ 1, then whereas if we set 7/2 = 1, then we are using purely LOOKAHEAD REASONING.
using purely token-level speculative decoding,
Theorem 3. Under synchronous LOOKAHEAD REASONING with concurrency budget M M an even number, and the mild parameter constraint
1 a 1202)
mink 1+a 1
1.157,
(7)
Then at least one of the following must hold:
1 1+00
(1 + 02 [ 02 +
M/2
C2M
2
(8)
1 01
1- + M
M/2 [1 - + a (1 a1+
1 02
(9)
1+
+
Furthermore, if both ( and (9) hold simultaneously, then combining both speculative techniques strictly outperforms using either one alone. Conversely:
If fails, the optimal strategy use only token-level speculation.
1f fails, the optimal strategy is use only step-level speculation.
Proof. Step 1: At least one of 8 and (9) must hold Define, for i = 2,
c; c:M
D.(M) = (1 + a; M/2)
1 c, + C,M
Observe that both factors
1 1- G cM 2
M/2 and 1 + a
1 C + GM
are strictly decreasing in M. Hence each D.(M) decreases as M grows. In particular,
D2(2) 1 ++0; an
I c
Since either Di(2) 2 D2(2) (2 < D2(2), it follows that either
1 1 D2(M),
D2(2) > D2(M) =
1+a
or
1 02
D2(2) DM(M)
DM(M).
1+
In other words, at least one of 9 must hold.
23

---

Step 2: Token-level-only speculation is suboptimal if condition 8 holds; otherwise, it is the optimal strategy. Lemma shows that both (1)) and ( are unimodal, reaching their maxima at "i 2 2 and 7/2 2 2. Below, we analyze whether token-level-only speculation (i.e. " = 1) yields the best speedup for different ranges of the concurrency budget M.
Case 1: If M <3 In this case, when (8) holds, we have M)) & 1(2, M (transfform through formulas), then according to the monotonicity of h, have
M ) V1 S 72 22
h1,, 12) S h(1, M) S h(2, 2
Therefore, the overall maximum attained only by jointly employing both levels. However, when (8) fails, since D1(x) strictly decreasing, we know that
1 on
(1 02
x/2 (1 c Vx M
1+c
1 1-02 +2+ 02
In this case, according to Lemma 5
h(71 72) S h(1, 7122) S h(1, M), 1n, 72 2 1,722 S M
So the overall maximum is attained when we use token-leveloonly technique. Case 2: If 2/ $ M < 2 In this case, first prove that
h(1, (5) < h(2, 2
A1 S 72 S
Then we prove that is always hold. From above we can see
h(1. h(1, h(2,
12) L1 12 S M
And the theorem follows.
1.From Lemma we can know that
02 ln
28
75
C272
02(15)
-02
1 02 + 02 0272
Therefore
h22, )/((1,
22
1 1+
1 C2 + C272
1+cl (1+02
7//2 (1 C2 02 22
1 1+
1
1 1 1+ 02
72/2 {
Let y
-a a21/2 E =0,,
7//2
(1 + 02 (1
25/2
2 02 02(1))) = y + (y- 2
1 ) In y)
Then from Lemma have
6 we can see that its value less than 1.157. Then given 7 we can
22 h(2,
2 /1)h(1112)) >1
2. Frrrm previous step, we know that (2 is strictly decreasing, so
1+ay
D2(M) < D2(13) < 1.157
1
So 8 is always hold.
Case 3: If M 2 275 this case, same as in Case 2,
8 is always hold. Besides, we have
h(1, < ((1,75) h(2, 12), Y1 S 7/2
M
So it's obvious token-level-only speculation is suboptimal.
24

---

Step 3: If (9) holds, step-level-only speculation is suboptimal. Otherwise, it is the optimal strategy. Same the previous step.
Theorem Under asynchronous LOOKAHEAD REASONING with 0.52 < 01,02 < 0.8 and < 3, < 5 and the total parallelism budget M 2 16 and M is an even number. Then under
722 S Mthe overall speedup h("; 72) maximized if and only if both step-level and token-level parallelism employed (i.e. ? and '2 2 2).
constraint min
Proof. Step 1: Token-leveloolly not optimal. Lemma shows that 9(22) is unimodal and reaching their maxima 7/ 2 2.
1.M 27/ This case, have
121, S & 1A S 12 S M
(2,
Therefore, it's obvious that lesee-eevel-only is not optimal. 2.12 < M < 2n2 In this case, we only need to prove
h(1, 12) - h(2 72
Same as the proof in 3 we only need to prove that
1 + 1.157
+ c1
is hold, and it's easy to prove.
3.M < 7 This case, according to Lemma
we can see that
h(1, 12) < h(2, 72,
is always hold, so we can conclude that leve--level-only is not optimal.
Step Step-level-only is not optimal. From Lemma 2 we can know that fasyme is strictly increasing when S S - and will stay constant after that.
1. M <
This case, according to Lemma
M 2), tn $ M h(7, s 1) h( 20
So step-level-onll is not optimal.
2. - M 2
This case, according Lemma
1(7,1) 1.1) = M(M,1) < M M
2), An $ M
2
3. M. 2 This case, will have
MY,, S MT MI v1 s 7 SM
So step-level-only is not optimal.
Lemma 1. Let
w(7) f(>) dM/)),
where M is an even number, and
a-a7
f()) 1 - ai + -1 a1) ci 7 (1
25

---

- am g(7) (1- a2)a2)(11 - 02 +27)
. 0.5 < 02,00 < 0.8, 0 <<a < <
and M 2 16.
Then (2))> > w(1), wl x w(M).
Proof.
(1+a1)
- C2
w w22)/w(1)
(1 (1+ M/2
(1+ tajc((1 a1))
1- 02 c2M12
(1+aa)
x
(1+ a1 M/3 (1 - a1))
We can easily found that this function is monotonically decreasing with respect to et 02 M. monooonnicllly increasing with respect to 01. Therefore,
0.52
w(2)/w(1)> >
>1
0 x 0.48)
M w( - (M)
1+ 02
0 + t a1) Ma a1)
2
1 (1+
M/2 ((1 - 01 + +11a(11
aj
M/2
01
M M/2
a
(1-a1)
M/2
M/2
1+ 1 02 [1 - (1 - c,M a1
0
01
1402
2
1 aj
M/2
(1 - a1) ta(a + ... a M/2
c1 2 a
M M122)
Given that of 2 ai a/ k E (1,2,... M,/],, so we have
M/2
al +
+ al
M12)
c1 2 a1
M M/2
If
1 aj M/2
C
50
then
M
1 02
/w(MM)
2
>
1+ 02
Otherwise,
1 022
01 M/2
1 0.52
0.8
w(
//w(M)
>
[1
1 x
x 1] >
(1
2
1
a+a
1
M/2
1+ 1/5
1 + 0.8
Lemma 2. The speedup function the asyne LOOKAHEAD REASONING
1-a7
S2 1 - 01 + 01 (1 a1-a]) - (1- 1
faryne (7)
S ci (1 (1-c))(11-0
otherwise
is strictly increasing when 1 5 [1] 1, N+ and will stay constant after y -
Proof. We first see the function as a continuous function on R and prove that when 1 15 < 1 the speedup function is strictly increasing
26

---

Write
A(7) a1. D(7) a1 a (1 (1- ci a17a (1-a))
Then Lasyne (7) A(7)/D(r)) and by the quotient rule
Saeyne(?) A'()) D(7) A(r)
D(7)²
We compute
A'()) -a In
= an aaat)--aa (1 yhha)
Hence the numerator of fassyn (7) becomes
((()) P(7) A(7) = o (| (1)7)))))
+ (1
= ai (1 a1) < (alna))aarratttaa)aaaaaaaaaaa (1 (1+y|ma)
= ai (1 01) (-nna)
Since
In > 0, 1 - > 0, 1-a7>0 0, 1- a-a >0 0
each term is strictly positive for all 7 2 1. Therefore
A(()) ((7) - A(7) D' (7) >0
fasync 0,
1 Then we prove that S 2 S2 We only need to prove that when 7/ ej' we have S3 = S
When
1 a-a
S2 1- a- + (1 - 0.)
a177 -1)
a-a
1- + (1 (1- af
1 1-
1
S1
(1-ai) - +c
ci + (1 c))(1aa)) an) a1)
Therefore, So the lemma follows.
Lemma 3. The speedup function of token-level speculative decoding 3 and is unimodality function.
1 1- 0
9(7) (1 (1- (1-
c2 7
where 02 > C2 Then g increases on [1 and decreases on (f, 50) for a unique RR.. In praccticaal scenarios, where E NT, the maximum of g(x)) is attained at some integer point 7 E NN+,"* 2 Sync LOOKAHEAD REASONING has similar form, so also has this property.
Proof. Step 1. Compute g(()). Set
((7) 1 az.
D(7) = (1 az) (
so that g(7) = N())/D(()) By the quotient rule,
g'(r) (7 D(7) - - D' D(())
D(r)²
Since
N' = -0 In 02;
D'(7) (1 022) (2))02
g'(7) og (n) (1 02) (1
02 227) 02 (1 0z) (1 02)
(2)) (1 02 + 02 7)
27

---

Since (1 a2) > 0, the sign of g' (7) equals the sign of
F(*) a3 (- a2) c2 (1
Step 2. F is strictly decreasing. Differentiate F:
-1- F(()) = -0 (1- 02 +2 72
Since 1 c2 02 0, it follows
('()) < for y 2 1.
Thus F is strictly decreasing on [1, oo).
Step Signs of F at the ends.
At 1:
F(1) a2(- o2(1 -22)
since 02 > 0, lna > 1 - 02 > 0, we have F(1) > 0
As 00, az + 0,
F(7) = 03
- -
02) - -0 50
Step 4. Conclusion via the Intermediate Value Theorem. Because F is continuous, strictly
decreasing, F(1) > O, and limy-+o0 F(7) < 0, there exists a unique o E R, 2 such that F(f) 0. Moreover,
F(7) >
15 < ?
F(r) <0
+ 7> 4.
Since (7) and F())) share the same sign, it follows that
g'(7) > 0 for 1 S I <><9 9
g'(r) < 0 for y
Besides, noting that
1 a2
(22)/(())
1 02
so in practical scenarios where 7 E N+ we conclude that the maximum is achieved at some integer
point 7" 2 as claimed.
Lemma 4. Ler 0.5 < a < 0.8 and define
o a(a,x) = - -Ina 1- az
x x21 1.
Then:
1. For each fixed a E (0.5,0.8), the function + a(a, x)) is strietly decreasing on [1, oo).
2. For each fixed x 2 1, the function a - a(a,2) x)) is strictly increasing on (0.5, 0.8).
3. Consequently, for every x 2 10 and a E (0.5, 0.8).
10-0.8 0.
a(a,r) < a(0.8, 10) In.88))
0.26,
1 0.810
and for all (0.52,0.8)
ala, 2) (a(0.52,2), a(0.8.))) (0.48, 0.79).
28

---

Proof. (i) Monotonicity in x. Fix a E (0, 1) and write
I aa" N(x)
f(x) = 1- 1 a² D(x)²
N(x= - D(x) = 1 - -²".
Then
(2))) = a (² +xlaa),
D'(x)) = aa
and by the quotient rule
f(xx) (x)x) D(x) -(r) D(x)²
a²) -a))
a1-a²)
Since 0 1, setting a 0 we have by convexity of the exponential,
a = > -+u=l+a0 +naa, Ina,
hence
(1 =I+ - --)11--=)) x Ina) = 1 hna a <.
Thus (x) < O. Because - In 0, it follows immediately
a x) = In f'(x) <0,
8r so a(( x) is strictly decreasing in x 2 1.
(ii) Monotonicity in a. Fix x 2 and set
Ia V(a) 1 a-a"
U(a) Ina,
so z) V(a). Then
1 v'(a) = (1-a0 a=)z >0.
x² x-1
U'(a)= = a'
Hence
0a ((((a)V(a++(aarrr)
V(a) V(-hn))((a).
da
a
We claim this is > 0. Indeed,
V + (- V >0
b (-Ina)aV'> >V V.
Since
z a-1
x² a
V aa
a V' = a (1- a²)² a1=
(1 - (1-a2)²
1 l-
this inequality becomes
x² a
xa
(-Ina) )²) 1 a²
- x(-lna)> > 1- az
But for 0.5 <a < 0.8, the well-known bound - In > 1 - a and n² S imply
(- In) 2 -Ina 1 a 2 1- a"
so 0a/0a > 0. Thus a(a, 2) is strictly increasing in E (0.5, 0.8).
(iii) Numerical bounds. By (i), a(a, .) S a(a, 10) for all æ 2 10, and by (ii),
10 - 0.810
a(a, 10) < a(0.8, 10) = In(8.8)
~ 0.26.
I- 0.810
10- 0.86
a(a, 6) S a(0.8, 6) = In(0.))
R 0.48.
I 0.86
10 0.8
a(a,8) 8) S a(0.8,8 8) = - (n..8))
R 0.36.
0.88
Also by (ii), for any a E (0.52, 0.8),
This completes the proof. a(a, 2) E (a(0.52, 2), a(0.8, 2)) (0.48, 0.8).
29

---

Lemma 5. Let
w(r) fsyno( {M//),
here y E N+, .1/7 E N and assume M 2 4. Then:
(a) If both 8 and 9 hold, then w(r)) is unimodal on [1, M and attains its maximum
22 M
(b) If 8 fails, then the unique maximizer is (token-level only).
h(7),72) < h(1, V112 = M,n, 72 N+
(c) If 2 fails, then unique maximizer is 7 M (step-level only). i.e.
h(71.72) < h 1), Vy112 = M, N+
Proof. We first treat this function continuous function over R, analyze its derivative determine its monotonicity, and then restrict its domain to N to obtain the desired results. Step 1: Derivative. By the product and chain rules,
1 7 ( (())
(sunc()) foync(7)
M ((M/7))
w' (x)
(M/7))
Step Log-derivatives. Define
-In(as)
ax(x)) = x 1 aς
i= 1,2,
Then
8' (aagn( feync( (7)
C1T
g'(7) 9(7)
C2T
= a1(7)
= =22(7)
1- c1 + CIT"
1 02 + 027
Hence with 1) M/7,
w' (7) Seyno()) (()
a1 (7) a2(1))
ci7
C2n
I-c2+027
Step 3: Monotonicity. By Lemma Also
4 a1 (7) is strictly decreasing in y and (2(M/)) strictly increasing. 1-
C17
C27)
C2 M
1 +
1-
C27l
1-a c1- oz
-
strictly increasing in y. Therefore w' (c) is strictly decreasing on [1, [1] and so w(y) either strictly increasing, or strictly decreasing, or first increasing then decreasing on [1, M).
Step 4: Endpoint comparison. Now restrict the domain to be N+
If (8) fails, then (2) < w()),, so w is either strictly decreasing on [1, M). or first increasing then decreasing and the critical points was in (1, Therefore we can conclude that w is strietly decreasing on [2, M). Therefore we can conclude that there's 's unique maximize at
y = 1, so we have
72) < h(,, 7 = 71.172 E N
If (9) fails, then w(() < w(M), , so w either strictly increasing on M]. or first increasing then decreasing and the critical points was in ( M). Therefore we can conclude that w is strictly increasing on [1, MI. Therefore we can conclude that there's unique maximize at y M, so we have
hmm, 12) < h(M, m22 = M, E
If both 8 and 99 holds, then w=2) 2 (1)), w( ( 2 w(M) so w would achieve the max between 2,
M 2
30

---

So the lemma follows.
Lemma 6, When 0 < y <
F(y) y + (y - 2 + In (1 y) 1.157
Proof. Define
F(y) 2-y + 2+ 2 In(1 y), o<y<l.
First derivative. A direct calculation gives
(y) = _ dy -v- 1)( v)] - 1
In(1 - 3)
Second derivative and concavity of F.. Differentiating again,
(v) = d
dy (1 - y 3) In(1 - w)}
+ d
(-v
dy
In(1 - -y)
1 - y
1 = y(1 + y) + 21 - v)
1 +
y²
y²
y²
Set
(2) y(1 + y) + +22nn11 y), so that (v) Nv)/vv On (0,
(v)
_ u + 21 y)] = +1 2y
d
2
20² 1
0,
dy
y
y
and ((0) = 0. Hence Nwv) < 0 for all y E (0,1), 1), which implies
F" (y) < 0 on (0, 1).
Thus F' is strictly decreasing.
Sign-change of F". As - 0+, In(1 - y) ~ y, so
F'( ~ (-v) - - +0(y) > 0.
y + 1n,,,nn111000 + -o while (1 - 1/1/2)) + 1, so F' (y( + -00
By continuity and strict decrease, there is a unique y² E (0, 1) with F'()") = 0, and
F Fv) Hence F increases on (0, y²) and decreases on (y", 1), so its maximum on (0, 1) occurs at y" Numerical evaluation. Numerically one finds y ~ 0.3693971022,
(0 <0<y<ソ"),
F'(y) < (y" < < 1).
Fly*) ~ 1. 1.1562281731 < 1.157.
Conclusion. Therefore for all (0, 1),
F(y) F(y") < 1.157,
as claimed.
C Calculation of (2)))
We'll prove when n + 00
1==1 X;
o7
S2(n)
Σ 22
X
C1 E" Ei,(X, (X, mod 7)
(1 a) + 00 o
-7 (11-01)0
31

---

Step 1: Use the Law of Large Numbers According to the Law of Large Numbers
X;
+ E[X,]
S2(n)
X
Ei= (X; mod 7)
a mod 7
Here, the PMF of X; is:
P(X; = = af -x1)
And its expectation
E[X]]
1 α1
Step 2: Compute F
X
Let Y = X 7
Key Observation The ceiling function m
Xi+1
can be expressed in terms of integer thresholds. For
X; +
= m++1 if X, [m7, (m + 1)7 - 1]
Thus:
Y m +1 for X, (mn,
E (m 1)7 1], = 2,,,
Compute B[Y]
B|y] Σ (m + 1)P((17 X; (m 1)7 1)
The probability Pmmy S X (m 1)y - 1) is:
(m+1)--1
Σ
P(X, = k) =0 my (1 - at)
k=my
Therefore we can get:
X;
a-
Step Compute [X, mod 7] To calculate the expectation of X; mod 7, where X, follows given geometric distribution and y is an integer greater than 4, we proceed as follows:
Compute P(X, mod 7 r) For E I0,1,...,. ? 1}. we have:
20
P(X, = my + (1 an)
P(X; mod 7 = r) =
- of
m=D
Compute E[X; mod 7] The expectation is:
7-1 E[X, mod 7 = Σ r P(X, mod -))
(1 alaa)aa
ai - - 7+1 - 7 an)a
Σ r.
aj
(1 aj a1) a7)
p=
D Illustration of Hybrid Approach
Accept draft
Token-Level Speculation
Target
Therefore, bio 468
Therofore, have b'o a 488, Welt.
Step-Level Speculation
Thorefore, we have bio 468 Welt, let me
Therefors, have snin
doublo
Therefore: we have
Therefore, we have ahe Wis. ist me ventlylntniii haveinie
Therefore, WnitWall vertly wnw.: Alloln 2 Afudd02
Figure 4: Illustration of hybrid approach.
32

---

E Verifier Comparison on QwQ-32B for Direct SPECREASON Reproduction
To facilitate fair comparison with SPECREASON, we replicate their setup using QwQ-32B as the base model and -iipseel-----istil---- 1.5B as the draft model, which is the original model setting in SPECREASON. Table f summarizes the results across multiple verifier strategies, reporting accuracy (%) and acceptance rate (%) on GSM8K and AIME. All values are averaged over 16 runs.
Dataset
Base
Rand.
M.M.JJ 32B
1.M.J 7B
Emb 0.95
Emb 0.85
Score 0.7
Score 0.8
GSM8K
96.2
96.5 (40)
96.3
96.4 (50)
(17(
(3.)(3))
(766 95.4 (36)
AIME
77.1
75.8 (30)
77.5 (38)
75.0 (42)
77.7 (24)
(36)
66.9 (41)
74.6 (16)
Table 4: Accuracy and acceptance rate (%) of QwQ-32B under different verifir strategies. Each cell shows:
Accuracy (Acceptance Rate).
As shown above, both the LLM-as-a-luuge (LLM-J) verifier and the embedding-based verifier with high similarity threshold (0.95) maintain the baseline accuracy on AIME (77.5% and 77.7% respec- tively, compared to 77.1 % without verification). while the score-based verifiers used in SPECREASON result in notable degradation, dropping to as low as 66.9% when using the score threshold of 0.7. These findings emphasize that scoring drafts with the same model does not align well with its generation distribution, particularly under longer contexts. Our experiments use the full 32k context length, whereas the original SPECREASON paper reported results using truncated 8k context. This difference further amplifies the mismatch between score-basee based filtering and semantic correctness, especially complex tasks like AIME.
F
Verifier Latency and Speedup Analysis on QwQ-32B
To analyze the accuracy-efficiency trade-off, we report the end-to-end inference speedups for various verifiers on QwQ-32B model with the deepseek-r boneeck-rl-dsttiiee ri-distill-115B draft model. These experiments were conducted on an H200 server (with TP=2 for the 32B verifier) and include results with n n-gram speculation to show orthogonality.
LLM (32B)
LLMJ
LLMJ
LLMJ (7B)an-gam
Emh (0.85)
Emb (0.85)en-gram
Score (0.7)
Score (0.7)en-gram
Dataset
(32R)en-gram
(7B)
GSMSK
x
1.67x
1.45x
1.75x
1,22x
1.53x
1.57x
1.63
AIME
1.12x
1.42x
1.0x
1.57x
1.0xx
1.1xx
0.77x
0.93x
Table 5: End-to- inference speedup (x faster) of different verifiers with and without n-gram
speculation on QwQ-32B. Experiments run on H200 (TP=2 for 32B).
The results reveal key trade-offs between verifier latency, acceptance rate, and final speedup. For instance, the 7B LLM-J achieves greater overall acceleration than both the heavier 32B LLM-J and the more lightweight Embedding model. This is particularly interesting given their isolated per-step latencies: the Embedding model fastest (0.009s), followed by the 7B LLM-J (0.013s), and the 32B LLM-J (0.025s). We hypothesize that the final speedup is function of both low latency and high acceptance rate, and the 7B verifier strikes the most effective balance between these two factors.
We analyzed the performance of the SpecReason verifier (Score 0.7). While it achieves significant 1.57x speedup on the short-context GSM8K dataset (avg. output 1K tokens), we observed an overall slowdown (0.77x) on the long-co congect AIME dataset. Upon analyzing the generation traces, we found that SpecReason actually causes a deceleration on individual examples where the output length grows very long (often >10K tokens). We hypothesize this slowdown stems from a lack of system optimizations SeacReason, where per-step costs like tokenizer overhead become amplified for these long sequences. This finding is consistent with the original SpecReason paper, where experiments on shorter contexts (<8k tokens) would not have revealed this overhead. We also note that for practical applications of large reasoning models, context limit of 8k tokens can be overly restrictive.
Finally, the +n-gram results consistently show substantial boost across all methods, confirming that both our LOOKAHEAD REASONING and SpecReason are orthogonal to token-level speculation.
33


Reviews:
[{"review_id": "NvZaF4eyDQ", "reviewer": "Reviewer_jaoN", "summary": "This paper proposes LOOKAHEAD REASONING to speedup the speculative decoding of reasoning models. The key insight is that reasoning models generate step-by-step, and each step needs only to be semantically correct, not exact token matching. Therefore, LOAKAHEAD REASONING uses a small draft model to generate future steps while the large reasoning model (target model) expands each proposal in one batched pass. A verifier is used to keep semantically correct steps while letting the target regenerate any that fail. In reasoning benchmarks, LOAKAHEAD REASONING improves the speedup of SD from 1.4x to 2.1x while preserving answer quality.", "questions": "- As mentioned in weakness, discussion of computation resource usage and the effects to request throughput would be helpful.\n- There are many question marks in Figure 1a. Are they formatting error?", "limitations": "YES", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n- The idea is intuitive and reasonable.\n- The paper is well-written and easy-to-understand. The figures clearly demonstrate how the algorithm works and how it affects the inference efficiency.\n- The theoretical speedup analysis is solid and helpful.\n- Insightful ablation studies, especially the effect of tree width on performance.\n\nCons:\n- This paper only considers the inference speed of single sequence, while the usage of GPU resources is not well discussed. Take figure 1a as an example: there are 3 draft steps. Does this mean 3 copies of the target model are needed to implement the pipeline? It is good that this algorithm improves the ceiling of the inference speed of single sequence, but it is also critical to discuss whether the GPU consumption is increased or reduced. Suppose that we have an inference system with LOOKAHEAD REASONING to process lots of user requests. Will LOOKAHEAD improve the system throughput? I agree that this algorithm can improve user experiences when there are IDLE GPUs, but it is worthy discussing how important these scenarios are.", "was_revised": true, "final_justification": "The authors have well addressed my concerns. I would like to keep my positive rating.", "rebuttal": "We thank the reviewer for the insightful and helpful feedback! We would like to address your questions in the below response.\n\n**Q1.1:** This paper only considers the inference speed of single sequence, while the usage of GPU resources is not well discussed. Take figure 1a as an example: there are 3 draft steps. Does this mean 3 copies of the target model are needed to implement the pipeline? It is good that this algorithm improves the ceiling of the inference speed of single sequence, but it is also critical to discuss whether the GPU consumption is increased or reduced.\n\n**A1.1:** Thank you for this excellent point. A discussion on resource usage is critical, and we will add a detailed analysis to the paper to clarify this.\n\nTo answer your primary question, our method does not require multiple copies of the target model's weights; only a single instance is needed. We achieve this by leveraging the batching capabilities of modern inference engines (e.g., vLLM). The process works as follows: first, a lightweight draft model generates several candidate steps. These candidates are then grouped into a single batch and processed by the target model. This allows the target model to generate its own corresponding \"ground truth\" step for each candidate simultaneously. Finally, a verification check is performed between the original drafts and the target model's outputs to decide whether to accept the drafts (fast-forwarding) or fall back.\n\nOur method uses more GPU resources to lower latency. We process many candidate steps together in a large parallel batch. This approach increases both the KV cache memory and the computational load. As a result, the GPU is utilized more effectively and does not sit idle. This trade-off is the key to our speedup: we use more of the GPU's power at each moment to finish the entire sequence faster.\n\n**Q1.2:** Suppose that we have an inference system with LOOKAHEAD REASONING to process lots of user requests. Will LOOKAHEAD improve the system throughput? I agree that this algorithm can improve user experiences when there are IDLE GPUs, but it is worthy discussing how important these scenarios are.\n\n**A1.2:** Regarding system throughput, we acknowledge that Lookahead Reasoning, like all speculative decoding methods, trades computation for lower latency. While this reduces per-request latency, the computation spent on inevitably failed speculations may decrease overall system throughput. This effect is scenario-dependent: when GPUs are underutilized, our method provides a speedup; when GPUs are fully saturated, it may reduce throughput. This trade-off is not unique to our work but is an inherent characteristic of all speculative algorithms designed to leverage spare compute resources. We anticipate that ongoing hardware advances will make this trade-off even more favorable.\n\nFurthermore, we emphasize that for many online serving systems, metrics like per-request latency and user experience are often more critical than raw throughput. Our work focuses on optimizing for these latency-sensitive applications. We will add a detailed discussion on this trade-off and resource usage in the revised paper.\n\n**Q2:** There are many question marks in Figure 1a. Are they formatting error?\n\n**A2:** The question marks in Figure 1a are indeed a rendering issue that occurs with certain web browsers. The symbols display correctly when viewed in a standard PDF reader or most web browsers. We apologize for this formatting issue and will ensure it is corrected in the revised version."}, {"review_id": "Z8W1fhxkU8", "reviewer": "Reviewer_qPc4", "summary": "The authors propose \"Lookahead reasoning\" (LR) which extends speculative decoding (SD) from token-level speculation to reasoning step speculation. After analyzing the convergence rate of SD, the authors introduce LR which uses a draft model to build a $W$-width speculation tree, where a node is a speculated reasoning step. Similar to SD, a draft is confirmed/rejected by a verifier. The authors analyze theoretical properties in two scenarios: 1) Cost-limited, i.e., when the draft model generates $k\\_1$ drafts slower than a single step (which takes time $T$) in the target model, and 2) Depth-limited, When the draft model can create $k\\_1$ drafts in $T$. Finally, the authors show that when using SD and LR together, a maximum speed up can be achieved. In their experimental analysis, the authors show 1) increased speed ups on real-world data sets over different draft and target models, 2) an ablation study over the verifier, and 3) the effect of speculation tree width, demonstrating its benefits in terms of acceptance rates while highlighting diminishing returns in terms of speed ups for $W>2$.", "questions": "In no particular order:\n\n1. I think in 3.2, the expectation is derived from the connection to a geometric distribution. Could be good to make this connection concrete. \n2. Over which data set did you compute Figure 2? Is it a combination over all data sets? Would it be possible to show uncertainties?\n3. Are you planning on releasing the code to reproduce results? The community would certainly benefit from such a contribution.\n4. In 3.3, would it be possible to incorporate a dataset's complexity in terms of expected number of reasoning steps? I assume, you could show that with increasing complexity expected speedups increase. \n5. line 159: explores $\\rightarrow$ explored\n6. You mention and compare to n-gram speculative decoding but never formally introduce it. It would be helpful to do so.\n7. What's $W$ for the results in Table 1?", "limitations": "The authors mention two limitations at the end of Section 6. I wonder if you could also comment on the implementation complexity. Since LR also requires to set parameters such as $W$ and $\\gamma$, please mention that as a limitation which may require tuning or some other selection procedure.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The idea is simple yet intriguing, and executed very thoroughly and rigorously. The manuscript is written clearly and structured in a way that makes it easy to comprehend the cores ideas. I am particularly drawn to the experimental evaluation and ablation over 2 combinations of draft and target models, demonstrating the effort the authors put into evaluating their approach. Additionally, I like the evaluation over 7 data sets underlining that LR leads to speed improvements in various scenarios. Lastly, the analysis of different verification approaches (LLM-based vs. embedding based) and the width analysis make this manuscript well-rounded. If code made public, this proposal may significantly impact the field by reducing costs/increasing speed of reasoning models.", "was_revised": true, "final_justification": "The authors answered my questions and I still believe that this work meets the acceptance criteria.", "rebuttal": "We appreciate your time and suggestions in reviewing our work. We take this opportunity to clarify our key contributions and address the concerns raised.\n\n**Q1:** I think in 3.2, the expectation is derived from the connection to a geometric distribution. Could be good to make this connection concrete.\n\n**A1:** Thank you for the suggestion. You're absolutely right. In the appendix, we provide a detailed derivation of the expectation. Specifically, we define a random variable X as the number of consecutive accepted drafts before the first rejection, which follows a geometric distribution. We then apply the standard expectation formula for the geometric distribution, together with the law of large numbers, to arrive at the final result for the speedup.\n\n**Q2:** Over which data set did you compute Figure 2? Is it a combination over all data sets? Would it be possible to show uncertainties?\n\n**A2:** Figure 2 (section 4.2) is computed on the AIME dataset. We appreciate you pointing out this omission and will clarify this in the main text. Regarding uncertainties, we will add error bars in the revised version to provide a more complete picture of the results.\n\n**Q3:** Are you planning on releasing the code to reproduce results? The community would certainly benefit from such a contribution.\n\n**A3:** Yes, we are committed to open source and are actively preparing the code for release. We expect to make it publicly available soon to help the community reproduce our results and build upon our work.\n\n**Q4:** In 3.3, would it be possible to incorporate a dataset's complexity in terms of expected number of reasoning steps? I assume, you could show that with increasing complexity expected speedups increase. \n\n**A4:** Thank you for the thoughtful suggestion. We can use the acceptance rate to describe the lookahead reasoning’s performance on a given dataset. This rate then directly informs the expected speedup.\nThe acceptance rate is likely correlated with the dataset's complexity, so we can model the relationship between speedup and complexity through their shared connection to the acceptance rate.\n\n**Q5:** line 159: explores $\\rightarrow$ explored\n\n**A5:** Thank you for catching this typo. We will correct \"explores\" to \"explored\" in line 159 in the revised version.\n\n**Q6:** You mention and compare to n-gram speculative decoding but never formally introduce it. It would be helpful to do so.\n\n**A6:** Thank you for highlighting this gap. We will clarify this in the related work section. N-gram speculative decoding is a standard, draft-model-free method, notably implemented as one of the speculative decoding strategies in the vLLM framework. This technique operates by maintaining a cache of all historical tokens. At each decoding step, it uses the suffix of the current sequence to perform a string-matching lookup within this cache. Historical n-grams that begin with this suffix are then proposed as speculative drafts. Subsequently, the verification stage proceeds identically to conventional speculative decoding, where the target model validates the entire candidate sequence in a single forward pass.\n\n**Q7:** What's W for the results in Table 1?\n\n**A7:** For the results in Table 1, we used W=1. We will add this clarification to the table caption to avoid confusion.\n\n**Q8:** The authors mention two limitations at the end of Section 6. I wonder if you could also comment on the implementation complexity. Since LR also requires to set parameters such as $W$ and $\\gamma$, please mention that as a limitation which may require tuning or some other selection procedure.\n\n**A8:** Thank you for highlighting these important practical considerations. Regarding implementation complexity, full integration into production serving engines (e.g., SGLang and vLLM) requires non-trivial engineering effort, particularly for handling requests in tree hierarchy. However, implementing Lookahead Reasoning on the client side is relatively straightforward, though it may not achieve the best performance as a deeply integrated solution. We have already developed a working implementation and plan to release it as open source to facilitate further adoption and experimentation.\n\nBesides, we will mention tuning the hyperparameters as a limitation in a revised version."}, {"review_id": "565T4ipiC7", "reviewer": "Reviewer_qndN", "summary": "This paper proposes Lookahead Reasoning, a two‐level speculative decoding scheme that augments vanilla token‐level speculative decoding (SD) with step‐level reasoning speculation. A lightweight draft model proposes entire “reasoning steps,” which are then expanded and verified in parallel by the full‐power target model; token‐level SD runs within each accepted step, yielding multiplicative parallelism. The authors provide a theoretical analysis showing that combining step‐ and token‐level speculation breaks SD’s inherent speedup ceiling, and empirical results on benchmarks (including GSM8K, AIME, AMC12, HumanEval) demonstrate speedups rising from ~1.4× (vanilla SD) to up to 2.1× without degrading output quality.", "questions": "1. **Define and contextualize SpecReason.** Please include a concise description (algorithmic steps, compute budget, prior speedup results) so readers can gauge how Lookahead Reasoning differs and improves upon it.\n2. **Add an LLM-driven SD baseline.** Run vanilla speculative decoding using your same draft LLM (1.5 B or 1.7 B) to fairly compare token-only vs. step-plus-token speculation under equal compute.", "limitations": "Yes.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Introducing step-level lookahead on top of token-level speculative decoding is an insightful way to unlock additional parallelism and overcome SD’s theoretical speedup ceiling.\n\nConcerns:\n1. Lack of Context on Prior Work\nThe paper repeatedly contrasts its method with “SpecReason” but never explains what SpecReason actually does or why it is a meaningful point of comparison. Without at least a concise description of SpecReason’s algorithm or performance characteristics, readers unfamiliar with that work cannot judge what the novel contribution truly is.\n2. Inappropriate Baseline for Efficiency Claims\nThe paper claims that users should allocate more drafting budget away from token-level drafting and toward reasoning steps or chain-of-thought drafting. However, its choice of baselines is inappropriate: it only compares to SpecReason (which, as noted, is never introduced) and to speculative decoding with an n-gram model, which isn’t comparable. They should instead compare to speculative decoding using a draft model with the same compute budget as their draft model.\n3. Limited Practical Validation\nWhile the theoretical analysis is useful, its significance depends on demonstrating practical effectiveness. Otherwise, it does not stand on its own.", "was_revised": true, "final_justification": "The baseline is misleading. vLLM v0.10.0 has deprecated its support for having separate draft models (they currently only support PLD and Eagle). However, the documentation mentions that this is temporary, as they plan to eventually support separate draft models. The authors should include a proper baseline, even if it means implementing their proposed method over a previous stable version of vLLM or an alternative. \n\nPublishing the paper in its current format is irresponsible, in my opinion.", "rebuttal": "We thank the reviewer for the insightful comments! We would like to answer your questions and address your concerns in this response:\n\n**Q1:** Lack of Context on Prior Work The paper repeatedly contrasts its method with “SpecReason” but never explains what SpecReason actually does or why it is a meaningful point of comparison. Without at least a concise description of SpecReason’s algorithm or performance characteristics, readers unfamiliar with that work cannot judge what the novel contribution truly is. \n\n**A1:** Thank you for this critical feedback. We agree that a clear context for SpecReason is essential for readers. We have already briefly introduced it in the related work (section 5 line 343-345), but we acknowledge that a more detailed description is beneficial for comparison. We will expand on this in the paper to clarify its role as a key baseline and better frame the contributions of Lookahead Reasoning.\n\nWe chose SpecReason as a primary baseline because, as a concurrent work, it is one of the few methods that also addresses the challenge of step-level acceleration for large reasoning models, making it a highly relevant point of comparison. \n\nHowever, the two approaches are founded on fundamentally different principles:\n\n- SpecReason operates on a score-and-threshold framework. First, a draft model generates a complete reasoning step. Then, the large reasoning model is prompted to assign a numerical score to this draft. The draft is accepted only if its score surpasses a predefined threshold; otherwise, it is discarded, and the target model generates the step autoregressively.\n\n- In contrast, Lookahead Reasoning is grounded in the principles of speculative decoding. Its acceptance criterion is not based on a subjective score but on whether the draft step match what the target model would have generated itself. This ensures statistical faithfulness to the original model, avoiding the potential brittleness of SpecReason’s approach, which is highly dependent on the target model's instruction-following capabilities and the quality of handcrafted scoring prompts. Beyond this, our approach incorporates system-level optimizations like concurrent multi-step verification and asynchronous execution to maximize hardware throughput. We will expand this context in the revised manuscript.\n\nThese methodological and architectural distinctions lead to crucial differences in the compute budget and performance outcomes. SpecReason introduces the overhead of an additional scoring call to the large target model for every proposed step. As our results demonstrate, this reliance on an external scorer can cause significant accuracy degradation. In contrast, Lookahead Reasoning's adherence to the speculative paradigm preserves the target model's accuracy while achieving substantial speedups. We will add this detailed comparison to the paper to properly contextualize our work.\n\n**Q2:** Please include a concise description (algorithmic steps, compute budget, prior speedup results) so readers can gauge how Lookahead Reasoning differs and improves upon it.\n\n**A2:** We will add a dedicated subsection with the following details to clearly differentiate the two methods. It is important to note that Lookahead Reasoning and SpecReason are concurrent works addressing the same challenge. Therefore, our goal is not to present LR as an improvement upon SpecReason, but to clearly distinguish their fundamental approaches and performance characteristics.\n\n**Algorithmic Steps**: SpecReason's generation process follows these steps at each generation step:\n- Draft Generation: A draft model generates a candidate for a complete reasoning step.\n- Scoring Prompt: The large reasoning model is prompted with instructions to evaluate the generated draft and assign it a numerical quality score (e.g., on a scale of 1-10).\n- Thresholding: The assigned score is compared to a predefined acceptance threshold. If the score meets the threshold (e.g., ≥ 8), the draft is accepted; otherwise, it is discarded, and the target model generates the step autoregressively.\n\n**Compute Budget:** For every proposed step, it requires a full forward pass of the large target model simply to generate a score. In contrast, Lookahead Reasoning's verifier can be a much smaller model (e.g., a 7B model or a lightweight embedding model), drastically reducing verification overhead. Furthermore, our architecture incorporates system optimizations like concurrent multi-step verification and asynchronous execution to further mitigate the overhead.\n\n**Prior Speedup Results:** Regarding prior results, the original SpecReason paper reported speedups of approximately 1.4x - 3.0x on various reasoning tasks. However, it is crucial to note that these results were achieved by limiting the maximum output length to 8k tokens—a constraint that can significantly compromise model accuracy on complex tasks. In our fair comparison on the GSM8K and AIME datasets (using the Qwen2-32B / DeepSeek-Distill-1.5B models from their work), we show that without this restrictive limit, the SpecReason method can lead to slowdowns and a notable drop in accuracy in more demanding, long-context scenarios like AIME.\n\nWe believe this detailed comparison will effectively contextualize our work and clarify the novelty and practical advantages of Lookahead Reasoning.\n\n**Q3:** The baseline comparison is inappropriate - only comparing to SpecReason and n-gram SD. Should include LLM-based SD with same compute budget for fair comparison.\n\n**A3:** Thank you for your valuable feedback regarding our baseline comparisons. We would like to clarify our rationale and present evidence suggesting that **n-gram speculative decoding (SD) is not a weak baseline, but rather a strong, highly-optimized, and often more effective benchmark than LLM-driven SD in practical, system-level applications**.\n\nThe core assumption that an LLM-based draft model is inherently superior is not true once system-level overhead is considered. Our own experiments directly compare the end-to-end speedup of n-gram SD against a Qwen3-1.7B parameter LLM-drafter when using Qwen3-32B as the target model. The results below clearly show that the efficient, draft-model-free n-gram approach achieves a higher effective speedup.\n\n**Speedup comparison of different speculation methods on H200 GPUs using Qwen3-32B**  \n| Dataset | 1.7B as Draft | n-gram | n-gram + LR |\n|---------|---------------|--------|-------------|\n| AIME'24 | 1.16X | 1.40X | **1.58X** |\n| AIME'25 | 1.04X | 1.37X | **1.53X** |\n| HMMT | 0.88X (slowdown) | 1.40X | **1.55X** |\n\nThe reason for this outcome lies in system-level realities, a conclusion supported by other recent research that also highlights the practical inefficiencies of LLM-driven SD. The core issue is high overhead: LLM-driven speculative decoding introduces the significant cost of running a second neural network and the system complexity of managing two models. In contrast, n-gram speculation is a highly efficient, draft-model-free approach. To clarify its mechanism: this technique operates by maintaining a cache of all historical tokens. At each decoding step, it uses the suffix of the current sequence to perform a string-matching lookup within this cache. Historical n-grams that begin with this suffix are then proposed as speculative drafts. Subsequently, the verification stage proceeds identically to conventional speculative decoding, where the target model validates the entire candidate sequence in a single forward pass. This makes the draft generation cost virtually free compared to running a separate LLM.\n\nThis performance trade-off is directly reflected in the design of leading inference frameworks. For instance, the latest version of vLLM (v1.0) has explicitly removed support for external LLM drafters due to these practical challenges, while maintaining robust, optimized support for n-gram speculation. And while this latest vLLM version also supports methods like EAGLE, our preliminary tests confirmed that EAGLE's performance degrades significantly in the long-context scenarios our work focuses on. Therefore, our choice of n-gram as a baseline is a deliberate one, aligned with the design of state-of-the-art systems and empirical evidence.\nTherefore, we argue that n-gram SD is the most appropriate and challenging baseline for our work. It is a standard, industry-relevant method that our empirical results confirm is highly effective. By demonstrating that our hybrid Lookahead Reasoning + n-gram method outperforms both n-gram SD and the LLM-driven SD we tested, we are showing a clear improvement upon a strong and practical foundation. We will add this detailed discussion to the paper to better justify our choice of baselines.\n\n**Q4:** Limited Practical Validation While the theoretical analysis is useful, its significance depends on demonstrating practical effectiveness. Otherwise, it does not stand on its own.\n\n**A4:** We agree that practical validation is essential to substantiate our claims. Our paper already showed the empirical evidence that using both lookahead reasoning and speculative decoding can achieve speedups as in Section 4.1. Specifically, Table 1 demonstrates that our hybrid approach, which integrates Lookahead Reasoning with n-gram speculative decoding, achieves a speedup of up to 2.1X. This result significantly outperforms the use of a single speculative method alone, thereby confirming that our theoretical contributions translate into tangible performance gains in practical scenarios"}, {"review_id": "pohbbiSzyN", "reviewer": "Reviewer_WAjf", "summary": "This work proposes Lookahead Reasoning that uses a smaller draft model to propose reasoning steps for a larger reasoning models. A third model is used to evaluate whether the drafted steps should be accepted based on semantic similarity. When used together with token level speculative decoding, generation efficiency can be improved without losing too much accuracy.", "questions": "1. SpecReason uses QwQ-32B models as the base model and reported a much lower accuracy on AIME for the original model and using reasoning model scoring actually improved model accuracy for them. Is it possible to get some results (i.e. Table 2) using the same model with different verifiers so that the comparison is straight-forward? \n2. (Related to weakness 1) Can you report the speed-up of SpecReason and SpecReason+SD?\n3. (Related to weakness 2) Can you report the latency (or end to end speed-up) of using different verifiers?", "limitations": "1. Auto-regressive calls to the base model: If I understand the paper correctly, this work still involves auto-regressively generating from the large base model. This means that there is opportunity to gain further speed-up if we are able to cut down the amount of auto-regressive generation from the target model.\n2. Token level speculation: The reviewer understands that the method proposed is orthogonal to token level speculation. The token level speculation method (prompt look-up decoding) used in the experiment is far from SOTA. While not discrediting this work, it would be nice if we can see more recent speculative decoding methods used together with LR and their combined effect on efficiency.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength:\n1. Presentation is clear and understandable\n2. Experiments results are good. However, some key comparisons are missing. \n\nWeakness:\n1. Key comparison with baseline: The work mentioned and compared against SpecReason as a baseline. However, the speed-up obtained using SpecReason is not reported in Table1. Both methods involves a trade-off between end-to-end speed up (efficiency) and results accuracy. Comparing only with the results accuracy with your baseline seems unfair.  Meanwhile, SpecReason is also orthogonal to token-level speculation. Hence a complete comparison should be SpecReason+SD (Ngram) with LR+SD (Ngram) in terms of both speed-up and accuracy.\n2. Efficiency of different verifiers: Different verifiers has different runtime and it is insufficient to report the acceptance rate in Table 2 when comparing different verifiers. Verifier runtime (or even end-to-end runtime) should be reported in order for the readers to evaluate the trade-off more comprehensively.\n3. The methods employs an additional 7B model as a verifier to evaluate whether to accept the drafted steps. This puts a lot of pressure on the memory and adds a lot of latency to the critical path of execution.", "was_revised": true, "final_justification": "The author provided experiment results to complete the evaluation. Therefore, I raise my rating.", "rebuttal": "Thank you for your thoughtful review and the encouraging feedback. Below, we address your questions in detail.\n\n**Q1:** SpecReason uses QwQ-32B models as the base model and reported a much lower accuracy on AIME for the original model and using reasoning model scoring actually improved model accuracy for them. Is it possible to get some results (i.e. Table 2) using the same model with different verifiers so that the comparison is straight-forward? \n \n**A1:** For a direct comparison with SpecReason, we evaluated the QwQ-32B model with the deepseek-r1-distill-1.5B draft model, as used in their paper. The results below show that verifiers grounded in semantic equivalence are most effective at preserving the model's original accuracy.\n\n**Performance of QwQ-32B with Various Verifiers**  \n*Each cell shows: Accuracy% (Acceptance Rate %), Accuracy values are averaged over 16 runs.*\n| Dataset | QwQ-32B | Rand. | LLM J.(32B) | LLM J(7B) | Emb (0.95) | Emb (0.85) | Score (0.7) | Score (0.8) |\n|---------|---------|-------|-------------|-----------|------------|------------|-------------|-------------|\n| GSM8K   | 96.2%   | 96.5% (40%) | 96.3% (43%) | 96.4% (50%) | 96.4% (17%) | 96.2% (38%) | 95.6% (76%) | 95.4% (36%) |\n| AIME    | 77.1%   | 75.8% (30%) | 77.5% (38%) | 75.0% (42%) | 77.7% (24%) | 74.4% (36%) | 66.9% (41%) | 74.6% (16%) |\n\n\nOur LLM-as-a-Judge (LLM-J) verifier and the Embedding model with a high similarity threshold (0.95) both successfully maintained the baseline accuracy on the challenging AIME dataset (77.5% and 77.7% vs. 77.1% baseline). While effective, the Embedding model's lower acceptance rate may lead to lower speedups.\n\nIn contrast, the score-based verifier used by SpecReason consistently underperformed. While the original SpecReason paper reported an accuracy gain, they used a shortened context (<8k tokens). Our experiments, using a full 32k context, reveal a significant accuracy drop on AIME to as low as 66.9%. This highlights a fundamental limitation: assessing draft \"quality\" via a score from the original reasoning model does not guarantee alignment with that model’s true output distribution, which is critical for maintaining correctness on complex reasoning tasks.\n\n\n\n**Q2 & Q3:** Can you report the speed-up of SpecReason and SpecReason+SD? Can you report the latency (or end to end speed-up) of using different verifiers?\n\n**A2 & A3:** Thank you for these important questions. To analyze the accuracy-efficiency trade-off, we report the end-to-end inference speedups for various verifiers on QwQ-32B model with the deepseek-r1-distill-1.5B draft model. These experiments were conducted on an H200 server (with TP=2 for the 32B verifier) and include results with n-gram speculation to show orthogonality. We will add this discussion to our revised manuscript.\n\n**Speedup Comparison Across Different Verifiers**  \n*Values show speedup ratio (X times faster)*\n| Dataset | LLM J.(32B) | LLM J.(32B) + n-gram | LLM J(7B) | LLM J(7B) + n-gram | Emb (0.85) | Emb (0.85) + n-gram | Score (0.7) | Score (0.7) + n-gram |\n|---------|-------------|----------------------|-----------|-------------------|------------|-------------------|-------------|---------------------|\n| GSM8K   | 1.32X       | 1.67X                | 1.45X     | **1.75X**             | 1.22X      | 1.53X             | 1.57X       | 1.63X               |\n| AIME    | 1.12X        | 1.42X                 | 1.20X     | **1.57X**             | 1.10X      | 1.41X             | 0.77X       | 0.93X               |\n\nThe results reveal key trade-offs between verifier latency, acceptance rate, and final speedup. For instance, the 7B LLM-J achieves greater overall acceleration than both the heavier 32B LLM-J and the more lightweight Embedding model. This is particularly interesting given their isolated per-step latencies: the Embedding model is fastest (0.009s), followed by the 7B LLM-J (0.013s), and the 32B LLM-J (0.025s). We hypothesize that the final speedup is a function of both low latency and high acceptance rate, and the 7B verifier strikes the most effective balance between these two factors.\n\n\nWe analyzed the performance of the SpecReason verifier (Score 0.7). While it achieves a significant 1.57x speedup on the short-context GSM8K dataset (avg. output ~1K tokens), we observed an overall slowdown (0.77x) on the long-context AIME dataset. Upon analyzing the generation traces, we found that SpecReason actually causes a deceleration on individual examples where the output length grows very long (often >10K tokens). We hypothesize this slowdown stems from a lack of system optimizations in SpecReason, where per-step costs like tokenizer overhead become amplified for these long sequences. This finding is consistent with the original SpecReason paper, where experiments on shorter contexts (<8k tokens) would not have revealed this overhead. We also note that for practical applications of large reasoning models, a context limit of 8k tokens can be overly restrictive. We will include these new results and this discussion in our revised manuscript.\n\nFinally, the +n-gram results consistently show a substantial boost across all methods, confirming that both our lookahead reasoning and SpecReason are orthogonal to token-level speculation\n\n\n\n\n\n \n\n\n\n\n\n\n**Q4:** Auto-regressive calls to the base model: If I understand the paper correctly, this work still involves auto-regressively generating from the large base model. This means that there is opportunity to gain further speed-up if we are able to cut down the amount of auto-regressive generation from the target model.\n\n**A4:** Our framework is designed to reduce sequential autoregressive generation by introducing parallelism at two different levels.\n\nWe thank the reviewer for this insightful question and agree that minimizing sequential autoregressive calls is the primary path to achieving significant speedups. Our Lookahead Reasoning framework is designed to do precisely that by fundamentally reducing the amount of step-by-step generation.\n\nAt the step level, the core innovation of our approach is to verify multiple independent draft steps in parallel. Instead of generating one token at a time, it processes several candidate steps simultaneously by using them as parallel prefixes (Figure 1a/line 49-60 shows this process). In addition to this coarse-grained parallelism, our framework also can work together with the token level speculative decoding. As demonstrated in our LR+SD experiments, any remaining sequential generation is further accelerated by n-gram speculative decoding. This combination effectively tackles the autoregressive bottleneck at both a coarse (step) and fine (token) granularity.\n\nFurthermore, the reviewer raises an excellent point about exploring other opportunities to reduce generation costs. For example, we can shorten the output length to cut down the amount of auto-regressive generation. We view such potential methods as complementary, not mutually exclusive. Our LR framework is designed to accelerate the generation of any sequence of tokens. If another technique could produce a more concise intermediate output, our method would still be orthogonal and could be applied to accelerate the generation of that compressed sequence. We believe this synergy with other potential optimizations is a key strength of our approach.\n\n**Q5:** The reviewer understands that the method proposed is orthogonal to token level speculation. The token level speculation method (prompt look-up decoding) used in the experiment is far from SOTA. While not discrediting this work, it would be nice if we can see more recent speculative decoding methods used together with LR and their combined effect on efficiency.\n\n**A5:** We thank the reviewer for the point regarding our choice of token-level speculation. Our decision was guided by the practical observation that some state-of-the-art methods, such as EAGLE, do not yet seamlessly support today's large reasoning models, especially in long-context scenarios. For instance, the evaluation code in EAGLE's official repository does not readily scale up to 10K tokens for reasoning models. In our preliminary experiments, we also observed a significant degradation in speedups when attempting to apply these methods, making them unsuitable for our study.\n\nHowever, n-gram speculation is a highly relevant baseline. Firstly, it offers a compelling balance of performance and practicality. The speedup achieved by n-gram is substantial, while its implementation is significantly simpler and more robust for integration. This practicality is underscored by its explicit support in leading inference engines like vLLM, which often prioritize stable, easy-to-integrate solutions over the system-level complexities introduced by draft-model-based approaches. To illustrate its effectiveness, we present a speedup comparison on H200 GPUs, using Qwen3-32B as the target model, Qwen3-1.7B model as the draft, and a max output length of 32K tokens (see table below).\n\n**Speedup comparison of different speculation methods on H200 GPUs using Qwen3-32B**  \n| Dataset | 1.7B as Draft | n-gram | n-gram + LR |\n|---------|---------------|--------|-------------|\n| AIME'24 | 1.16X | 1.40X | **1.58X** |\n| AIME'25 | 1.04X | 1.37X | **1.53X** |\n| HMMT | 0.88X (slowdown) | 1.40X | **1.55X** |\n\n\nAs the table shows, n-gram can outperform a standard LLM draft model, particularly in challenging scenarios where draft model performance degrades. More critically for our paper, this allows us to cleanly demonstrate our core thesis: **Lookahead Reasoning is orthogonal to and combines effectively with token-level speculation**. The cumulative acceleration shown in the n-gram + LR results directly proves this synergy, with consistent improvements across all three datasets."}]

Author rebuttal:

Meta-review: This paper proposes a speculative decoding method with a lookahead mechanism. The LOOKAHEAD mechanism can speedup the speculative decoding of reasoning models. Experimental results show that it improves the decoding speed from 1.4x to 2.1x while preserving answer quality.
Strong points:
1. Reviewers recognized that the proposed lookahead method is reasonable, practical and useful.
2. Reviewers also agree that the presentation is clear
3. The experiment is well designed and the results show the gains of the proposed idea.
Weak points:
1. some reviewers pointed out that some important related papers were not included.
2. Given the related paper, the novelty of proposed approach became relatively smaller. 

Overall, the paper is a slightly above the bar.
Decision: Accept (poster)