SpecExit: Accelerating Large Reasoning Model via Speculative Exit

## Abstract

Abstract Despite their strong performance on reasoning tasks, large reasoning models (LRMs) often suffer from overthinking, producing unnecessarily long outputs and incurring high end-to-end latency, a significant limitation to their real-world deployment.
To address overthinking, early-exit mechanisms have been proposed to terminate reasoning before typical completion, showing that this approach can effectively shorten generation length with minimal impact on accuracy. However, their reliance on probing mechanisms introduces a detection overhead that limits their end-to-end latency gains and compromises their generalizability across diverse problems.
Inspired by the use of hidden states in speculative decoding, we propose SpecExit , a novel framework that predicts both future tokens and an early-exit signal directly from a lightweight draft model without probing overhead.
Our method offers significant improvements, reducing average generation length by 66% and achieving a 2.5x speedup in end-to-end latency compared to the speculative decoding baseline, without compromising accuracy.
Our method leverages the inherent signals from hidden states to provide effective early-exit signals, suggesting broader use of hidden states for efficient reasoning.
Our code is available at: https://github.com/Tencent/AngelSlim .

## 1 Introduction

Large reasoning models (LRMs) such as OpenAI-o1 (OpenAI, 2024), DeepSeek-R1 (DeepSeek-AI et al., 2025) and Qwen (Qwen et al., 2025) have recently achieved state-of-the-art performance in complex tasks.
These models follow the test-time scaling law (Snell et al.,; Brown et al.,; Muennighoff et al.,), where generating longer chain-of-thought (CoT) sequences (Wei et al.,) generally enhances model performance.
However, this reliance on extended reasoning often leads to an overthinking problem, where models produce unnecessarily verbose outputs.
This redundancy leads to both excessive token usage and high end-to-end latency, which limits LRMs’ practical deployment.

To mitigate overthinking, researchers have proposed both inference-time and training-based strategies (Sui et al., 2025).
Inference-time early-exit methods (Yang et al., 2025; Fu et al., 2024) rely on model-generated signals such as intermediate answers or output logits to terminate decoding once sufficient evidence is detected.
These methods can shorten reasoning length without harming accuracy, but the probing overhead they introduce limits the actual latency gains.
Training-based approaches, such as reinforcement learning (Aggarwal & Welleck, 2025; Yeo et al., 2025) and supervised fine-tuning (Ma et al., 2025b; Munkhbat et al., 2025), incur little runtime overhead during inference but risk altering the model’s output distribution.
As a result, existing methods struggle to deliver consistent improvements in end-to-end efficiency.

Speculative decoding (Chen et al., 2023; Leviathan et al., 2023) is a promising approach that improves efficiency without altering the target model’s outputs. A lightweight draft model proposes multiple candidate tokens, which the target model verifies in parallel, substantially reducing per-token latency. However, this strategy alone does not resolve the overthinking problem, as models still generate the full CoT.
Recent advanced speculative decoding methods (Li et al., 2024a; Zhang et al., b) exploit hidden states to predict several future tokens. Other studies (Lin et al.,; Dong et al.,) also show that hidden states encode richer predictive signals beyond next-token probabilities.

Figure: (a) Efficiency Comparison
Refer to caption: x1.png

In this work, we introduce SpecExit, a reasoning-aware early-exit framework that leverages draft model hidden states not only to anticipate future tokens but also to predict early-exit signals. Unlike prior probing-based approaches, SpecExit requires no modifications to the target model and incurs no additional detection overhead. Instead, it extends the lightweight draft model with auxiliary prediction heads, enabling it to jointly output token distributions and reasoning-related signals in a single forward pass. By exploiting the latent information embedded in hidden states, SpecExit provides reliable criteria for dynamically terminating chain-of-thought generation when sufficient reasoning has been achieved.

We validate SpecExit on state-of-the-art reasoning models across mathematical, scientific, and logical benchmarks. For Qwen3-4B-Thinking-2507 and DeepSeek-R1-Distill-Llama-8B models, SpecExit shortens generation length by 66%, and speeds up end-to-end latency by up to 2.5x compared with the speculative decoding baseline.
Our contributions can be summarized as follows:

- •
Signals Extracted for Early Exit. We derive early-exit signals from hidden features and integrate them into speculative decoding, enabling reliable early exit for efficient reasoning.
- •
General and Practical Framework. We implement SpecExit, a reasoning-aware early-exit framework, in both PyTorch and vLLM, making it easy to deploy across diverse inference environments.
- •
Substantial End-to-End Performance Gains. SpecExit reduces reasoning length by 66% and achieves up to 2.5$\times$ lower latency than speculative decoding while maintaining accuracy.

## 2 Motivation

Current early-exit methods face two challenges: runtime overhead from probing and limited generality from task-specific prompts.
Since hidden states already reflect reasoning sufficiency, we propose leveraging them to replace costly probing for faster and more reliable inference across diverse tasks.

Figure: Figure 2: Overall architecture of the proposed SpecExit framework. The Multi-Token Prediction (MTP) layer is augmented to output both token logits and auxiliary signals. Training is performed with Multi-Task Learning (MTL), while at inference these signals guide speculative early stopping without modifying the backbone model. The example illustrates how redundant reasoning steps can be pruned while preserving final answer quality.
Refer to caption: x3.png

Probing Overhead and Limited Generalizability.
Probing-based early-exit methods suffer from both latency overhead and poor generalizability across tasks. Shortened reasoning trajectories do not effectively translate into latency improvements. As shown in Figure [1(a)](https://arxiv.org/html/2509.24248v2#S1.F1.sf1),
DEER (Yang et al., 2025) reduces the generation length of GSM8K (Cobbe et al., 2021) and ARC-Challenge(Clark et al., 2018) by 32% and 44% respectively on Qwen3-4B-Thinking-2507, but even increases end-to-end latency than vanilla baseline.
The gap arises because probing introduces extra computation, and its effectiveness is highly sensitive to both the task and the model. A probing phrase like “Final Answer is” elicits effective intermediate answers in math problems, but fails in coding tasks where extra tokens are still generated. This drawback not only undermines generalizability across domains but also limits true latency savings.

Signals from Hidden States.
Our preliminary experiments show that models’ hidden states encode informative signals about its reasoning process.
As illustrated in Figure [1(b)](https://arxiv.org/html/2509.24248v2#S1.F1.sf2), we use a simple Multi-Layer Perceptron (MLP) trained on hidden states to predict reasoning progress. For complex tasks, the predicted signals appear darker at the beginning, reflecting the need for continued reasoning, but gradually shift to lighter colors as the model approaches a sufficient chain of thought. This progression suggests that hidden states provide fine-grained indicators of task complexity and reasoning sufficiency. Leveraging these internal signals offers an efficient alternative to costly probing, motivating our approach of utilizing signals from hidden states.

## 3 Method

### 3.1 Overall Architecture

SpecExit Framework. The overall design of this work aims to incorporate additional learnable signals into large model reasoning, thereby providing explicit decision-making criteria for early stopping. In the decoding process of large language models, hidden states not only encode semantic and contextual information for next-token prediction but also implicitly contain higher-level cues related to reasoning progress, generation quality, and content completeness. The Multi-Token Prediction (MTP) mechanism leverages these hidden states to project into the vocabulary space and predict multiple future tokens simultaneously, thereby improving inference efficiency. Inspired by this, we extend the MTP layer while keeping the backbone language model unchanged, introducing auxiliary prediction heads that allow the model to explicitly generate reasoning-related signals, including confidence, reasoning progress, and remaining reasoning length, alongside token distributions. This design preserves the original language modeling ability while providing learnable auxiliary variables for inference control, enabling efficient and dynamic reasoning regulation. The overall architecture of the SpecExit framework is shown in Figure [2](https://arxiv.org/html/2509.24248v2#S2.F2).

Model Structure. We propose a speculative sampling-based early stopping mechanism for reasoning. In the MTP layer of the model, we extend the linear projection to include additional dimensions. In standard decoding, the hidden state $h\in\mathbb{R}^{D}$ is projected into the vocabulary space to predict the next token distribution, where $D$ is the hidden size of the model. In our method, the linear layer output is extended as:

$$ $Wh=\big[W_{\text{tok}}h,\;W_{\text{conf}}h,\;W_{\text{prog}}h,\;W_{\text{rem}}h\big],$ (1) $$

where $W_{\text{tok}}h$ produces standard token predictions, while $W_{\text{conf}}h,W_{\text{prog}}h,W_{\text{rem}}h$ predict confidence, reasoning progress, and remaining reasoning length, respectively. These additional signals serve as observable indicators for deciding whether to stop reasoning early, thus reducing redundant computation without compromising output quality.

### 3.2 Signal-Extracted Training

Figure: Figure 3: Convergence of token classification loss and signal regression loss during training.
Refer to caption: x4.png

Data Construction. We first obtain the complete response generated by the base language model and extract the reasoning content enclosed within the <think> and </think> tokens. To identify the effective reasoning trajectory, we iteratively attempt to insert the closing marker </think> after each paragraph and verify whether the resulting final answer matches the original output. If the answer remains consistent, the subsequent reasoning content is regarded as redundant.
Consequently, only the minimal reasoning segment required to produce the correct answer is retained as training data.

Signal Annotation. Confidence is defined as the geometric mean of the logit probabilities across prediction steps, reflecting the reliability of the generation; remaining reasoning length is defined as the number of tokens from the initial <think> marker to the earliest valid insertion point of </think> that still yields the correct answer; progress is represented as a normalized value increasing from 0 to 1, capturing the relative progression of the reasoning CoT.

Signal Regression. We propose a cost-efficient extension by introducing a small number of additional dimensions into the linear projection layer of the MTP module for regressing reasoning signals. These dimensions are orthogonal to the vocabulary classification weights, ensuring that signal regression does not interfere with the convergence of speculative decoding training.

The Multi-Task Learning (MTL) overall training objective jointly optimizes token classification and signal regression, defined as:

$$ $\mathcal{L}=\mathcal{L}_{\text{cls}}+\lambda_{c}\mathcal{L}_{\text{conf}}+\lambda_{p}\mathcal{L}_{\text{prog}}+\lambda_{r}\mathcal{L}_{\text{rem}},$ (2) $$

where $\mathcal{L}_{\text{cls}}$ is the standard cross-entropy loss for vocabulary prediction, and $\mathcal{L}_{\text{conf}},\mathcal{L}_{\text{prog}},\mathcal{L}_{\text{rem}}$ correspond to the regression losses for confidence, progress, and remaining reasoning length, with $\lambda_{c},\lambda_{p},\lambda_{r}$ denoting dynamic weighting coefficients. Specifically:

Confidence and progress are optimized using mean squared error (MSE), remaining reasoning length is optimized with mean squared logarithmic error (MSLE):

$$ $\mathcal{L}_{\text{conf}}=\frac{1}{N}\sum_{i=1}^{N}(\text{sigmoid}(\hat{c}_{i})-c_{i})^{2},$ (3) $$

$$ $\mathcal{L}_{\text{prog}}=\frac{1}{N}\sum_{i=1}^{N}(\text{sigmoid}(\hat{p}_{i})-p_{i})^{2},$ (4) $$

$$ $\mathcal{L}_{\text{rem}}=\frac{1}{N}\sum_{i=1}^{N}\left(\hat{r}_{i}-\log(1+r_{i})\right)^{2},$ (5) $$

where $\hat{c}_{i}$, $\hat{p}_{i}$, and $\hat{r}_{i}$ represent the model-predicted confidence, progress, and remaining reasoning length, respectively; $c_{i}$, $p_{i}$, and $r_{i}$ denote their corresponding ground-truth values; and $N$ is the total number of samples.

Dynamic Weighting. Since the regression losses of signals converge faster than the token classification loss, we adopt a gradient-based dynamic weighting strategy to balance the contributions of different tasks. This mechanism assigns higher weights to tasks with smaller gradient magnitudes, preventing tasks with larger gradients from dominating the learning process and ensuring all tasks are effectively optimized. Formally, the mechanism is defined as:

$$ $\lambda_{j}=\frac{\|\nabla_{\theta}\mathcal{L}_{j}\|}{\sum_{k}\|\nabla_{\theta}\mathcal{L}_{k}\|},\quad\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{cls}}+\sum_{j}\lambda_{j}\mathcal{L}_{j},$ (6) $$

where $\mathcal{L}_{j}\in\{\mathcal{L}_{\text{conf}},\mathcal{L}_{\text{prog}},\mathcal{L}_{\text{rem}}\}$, $\nabla_{\theta}\mathcal{L}_{j}$ denotes the gradient of task $j$ with respect to the model parameters, $\mathcal{L}_{\text{cls}}$ is the cross-entropy loss for token classification, and $\lambda_{j}$ is the dynamically computed weight. This formulation ensures that the gradient contributions of all tasks are balanced, which facilitates stable convergence in multi-task optimization, as shown in Figure [4](https://arxiv.org/html/2509.24248v2#S3.F4).

### 3.3 Signal-Guided Inference

Overall Procedure. We build upon the speculative decoding framework, where a smaller draft model first proposes a sequence of candidate tokens, which are then verified in parallel by a larger target model. To evaluate feasibility and efficiency, the inference procedure is implemented on both PyTorch and vLLM frameworks. The central modification lies in the forward pass of the target model. Beyond computing the logits for the next token, we additionally extract the final hidden state corresponding to the last accepted token. This representation is processed through a lightweight linear layer to generate three signals: a confidence score, a progress indicator, and an estimate of the remaining reasoning length, as shown in Figure [4](https://arxiv.org/html/2509.24248v2#S3.F4).

Stopping Conditions. To ensure that early-exit decisions occur at semantically coherent boundaries, we introduce a class of special markers called step split tokens, which indicate natural segmentation points in the generated text.
Specifically, step split tokens can be divided into two categories: Paragraph Delimiters (e.g., .\n\n), which mark the end of a paragraph or reasoning unit, and Discourse Markers (e.g., ”Wait”, ”But”, or ”Therefore”), which often signal semantic transitions or logical shifts during reasoning. Since the segmentation strategy based on Paragraph Delimiters is more general, this strategy is adopted by default in subsequent experiments.
Examples of commonly observed discourse markers in reasoning trajectories are shown in Fig. [7](https://arxiv.org/html/2509.24248v2#A1.F7) (Appendix). When a sampled token belongs to the above set, the early-exit logic is triggered. If the smoothed signal exceeds the predefined threshold, the system determines that the reasoning process has been sufficiently explored. In this case, the accepted output length is truncated at the position of the step split token, and the target model’s recover token is replaced with a special reasoning-end marker (e.g., </think>), thereby ensuring that the termination point lies at a natural boundary while maintaining coherence of the generated text. The complete inference process is summarized in Algorithm [1](https://arxiv.org/html/2509.24248v2#algorithm1).

Figure: Algorithm 1 Inference procedure with signal-guided speculative exit

Signal Smoothing. Since raw signals may exhibit significant volatility, relying directly on them risks premature or unstable termination. To enhance robustness, we apply an Exponentially Weighted Moving Average (EWMA) to smooth the signals across steps. At each iteration, the smoothed value is updated as a weighted average of the current raw signal and the previous smoothed value, with the smoothing factor controlling the balance between recent and past observations. A smaller factor emphasizes historical stability, yielding smoother trajectories that are less sensitive to transient noise. This ensures that termination decisions reflect consistent trends rather than isolated fluctuations.

## 4 Experiments

### 4.1 Experimental Setup

To evaluate the effectiveness of our SpecExit framework, we conducted a comprehensive set of experiments across multiple domains. Specifically, we used the GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021) and AIME (MAA Committees,) datasets for mathematical reasoning, the HumanEvalPlus (Liu et al., 2023) dataset for coding, the GPQA Diamond (Rein et al., 2023) dataset for science, and the ARC-Challenge (Clark et al., 2018) dataset for logic. Experiments are conducted on two mainstream LRMs: Qwen3-4B-Thinking-2507 (Qwen et al., 2025) and DeepSeek-R1-Distilled-Llama-8B (DeepSeek-AI et al., 2025).

We compare our SpecExit method against several baselines: Vanilla, which represents full generation without any early-exit mechanism; NoThink (Ma et al., 2025a), which skips the reasoning phase; DEER (Yang et al., 2025), a dynamic early-exit method; and EAGLE3 (Li et al., 2025), a speculative decoding baseline, which is trained using the same dataset as SpecExit to ensure a fair comparison.
Our performance analysis is based on three key metrics, as detailed in Table [1](https://arxiv.org/html/2509.24248v2#S4.T1): Accuracy ($\uparrow$), Token ($\downarrow$) count and end-to-end Latency ($\downarrow$).
All experimental results are obtained by implementing our early-exit strategy in vLLM  (Kwon et al., 2023), and running inference on an 8×H20 GPU cluster.

### 4.2 Main Results

We first evaluate the proposed SpecExit against baseline reasoning approaches on mathematical, scientific, coding, and logical benchmarks. As shown in Table [1](https://arxiv.org/html/2509.24248v2#S4.T1), SpecExit consistently achieves substantial reductions in both output length and inference latency while maintaining comparable or even higher accuracy.

Across benchmarks, SpecExit significantly shortens reasoning trajectories, with up to 54% and 53% reduction on GSM8K and ARC-Challenge for Qwen3-4B-Thinking-2507, and up to 66% and 64% reduction for DeepSeek-R1-Distill-Llama-8B. The reduced reasoning length corresponds to measurable efficiency improvements: SpecExit achieves a 1.9x latency reduction with Qwen3-4B-Thinking-2507 and up to 2.5x speedup with DeepSeek-R1-Distill-Llama-8B on GSM8K, compared with the speculative decoding baseline EAGLE3. Importantly, these gains come only with marginal accuracy differences, confirming that early termination of redundant reasoning does not harm task performance. By contrast, prior inference-time methods primarily focus on reducing output length, but the latency gains they achieve are relatively modest. In some datasets, the additional computational overhead even leads to slower inference than the standard think mode. Notably, for Qwen3-4B-Thinking-2507, inserting the </think> token in the NoThink baseline still fails to suppress reasoning, resulting in even longer outputs than the original Think mode.

Overall, these results demonstrate that SpecExit achieves a favorable balance between efficiency and accuracy, highlighting the practicality of integrating reasoning-aware early-exit strategies into LRMs inference.

**Table 1: Performance comparison of various reasoning methods on mathematical, scientific, general, and coding benchmarks. “Acc” denotes accuracy, “Tok” denotes token count, and “Lat” denotes total end-to-end latency. $\uparrow$ indicates that higher values are better, while $\downarrow$ indicates that lower values are better.**
|  | Math | Coding | Science | Logic |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | GSM8K | MATH500 | AIME | HUMANEVAL+ | GPQA-D | ARC-Challenge |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Acc$\uparrow$ | Tok$\downarrow$ | \columncolorgray!10Lat$\downarrow$ | Acc$\uparrow$ | Tok$\downarrow$ | \columncolorgray!10Lat$\downarrow$ | Acc$\uparrow$ | Tok$\downarrow$ | \columncolorgray!10Lat$\downarrow$ | Acc$\uparrow$ | Tok$\downarrow$ | \columncolorgray!10Lat$\downarrow$ | Acc$\uparrow$ | Tok$\downarrow$ | \columncolorgray!10Lat$\downarrow$ | Acc$\uparrow$ | Tok$\downarrow$ | \columncolorgray!10Lat$\downarrow$ |  |
| Qwen3-4B-Thinking-2507 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Think | 95.3 | 1414 | \columncolorgray!10155.6 | 96.6 | 6719 | \columncolorgray!10530.1 | 86.7 | 19577 | \columncolorgray!10243.3 | 90.9 | 5079 | \columncolorgray!10175.3 | 68.7 | 9041 | \columncolorgray!10325.8 | 95.6 | 1812 | \columncolorgray!10156.5 |  |
| NoThink* | 95.2 | 1631 | \columncolorgray!10204.2 | 96.6 | 6395 | \columncolorgray!10488.5 | 86.7 | 19816 | \columncolorgray!10243.2 | 88.4 | 4480 | \columncolorgray!10131.5 | 67.2 | 8833 | \columncolorgray!10276.8 | 95.1 | 1889 | \columncolorgray!10159.8 |  |
| DEER | 94.3 | 960 | \columncolorgray!10230.3 | 94.4 | 4893 | \columncolorgray!10519.6 | 70.0 | 17838 | \columncolorgray!10218.6 | 86.6 | 4079 | \columncolorgray!10242.4 | 67.2 | 9053 | \columncolorgray!10505.2 | 94.6 | 1011 | \columncolorgray!10200.3 |  |
| EAGLE3 | 94.8 | 1408 | \columncolorgray!10140.3 | 96.6 | 6670 | \columncolorgray!10395.7 | 80.0 | 19792 | \columncolorgray!10187.3 | 87.2 | 5178 | \columncolorgray!1081.7 | 67.7 | 8975 | \columncolorgray!10212.2 | 95.7 | 1822 | \columncolorgray!10164.2 |  |
| SpecExit | 93.8 | 649 | \columncolorgray!1075.8 | 96.8 | 4777 | \columncolorgray!10367.9 | 90.0 | 17769 | \columncolorgray!10206.1 | 89.6 | 4319 | \columncolorgray!1058.4 | 68.7 | 7011 | \columncolorgray!10137.0 | 94.5 | 588 | \columncolorgray!1071.4 |  |
| DeepSeek-R1-Distill-Llama-8B |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Vanilla | 76.4 | 1008 | \columncolorgray!10629.4 | 81.8 | 6878 | \columncolorgray!10857.1 | 36.7 | 22170 | \columncolorgray!10307.0 | 74.4 | 6287 | \columncolorgray!10445.5 | 43.6 | 8857 | \columncolorgray!10574.0 | 49.9 | 1917 | \columncolorgray!10628.5 |  |
| NoThink | 54.6 | 233 | \columncolorgray!1022.2 | 55.2 | 1643 | \columncolorgray!10262.8 | 10.0 | 8744 | \columncolorgray!10184.1 | 46.3 | 472 | \columncolorgray!107.3 | 26.8 | 1200 | \columncolorgray!10166.6 | 12.6 | 135 | \columncolorgray!1013.6 |  |
| DEER | 74.7 | 710 | \columncolorgray!10484.8 | 80.8 | 3533 | \columncolorgray!10973.3 | 40.0 | 15619 | \columncolorgray!10272.3 | 79.3 | 4206 | \columncolorgray!10269.2 | 40.9 | 8492 | \columncolorgray!10521.5 | 47.5 | 1029 | \columncolorgray!10531.3 |  |
| EAGLE3 | 79.3 | 976 | \columncolorgray!10276.9 | 80.8 | 6172 | \columncolorgray!10593.6 | 30.0 | 25686 | \columncolorgray!10228.1 | 78.7 | 5312 | \columncolorgray!10346.5 | 43.9 | 8749 | \columncolorgray!10420.1 | 59.2 | 1378 | \columncolorgray!10496.4 |  |
| SpecExit | 75.3 | 333 | \columncolorgray!10112.6 | 80.6 | 1968 | \columncolorgray!10348.3 | 36.7 | 8160 | \columncolorgray!10176.0 | 81.7 | 3105 | \columncolorgray!10118.1 | 46.0 | 6849 | \columncolorgray!10307.5 | 50.3 | 500 | \columncolorgray!10253.7 |  |

### 4.3 Ablation Study

Signal Type. To investigate the impact of individual reasoning signals in SpecExit, we conduct ablation studies on confidence, progress, and remaining token length, along with a combined configuration (SpecExit*) that integrates all three.
As shown in Figure [5](https://arxiv.org/html/2509.24248v2#S4.F5), the confidence-only variant yields the largest token reduction but overestimates the model’s certainty, resulting in noticeable accuracy drops on complex benchmarks.
The predicted reasoning progress increases sharply in the early steps yet continues to fluctuate during iterative reflection. Remaining token length is generally high at the beginning of inference but often triggers premature exits on complicated problems. By integrating all signals, SpecExit* leverages their complementary strengths, preserving competitive accuracy while substantially reducing tokens, demonstrating that multi-signal integration mitigates individual biases and enables more reliable early stopping.

Figure: (a) Accuracy comparison.
Refer to caption: x6.png

Signal Smoothing. In order to investigate the influence of different smoothing strategies on the stability and performance of early-exit decisions, we conducted a series of ablation experiments comparing multiple approaches. As shown in Table [2](https://arxiv.org/html/2509.24248v2#S4.T2), removing smoothing increases the volatility of cognitive signals, leading to inconsistent early exits and increased token consumption. The momentum-based prediction strategy significantly reduces token usage, though it may slightly degrade accuracy due to overly aggressive early termination. Smoothing using sliding-window and paragraph-level averaging offers a better trade-off, maintaining accuracy while improving efficiency. Among all methods, Exponential Weighted Moving Average (EWMA) strikes the most consistent balance, providing both stability and reliability. These results demonstrate that appropriate smoothing is essential for reliable early-exit behavior, as it mitigates the influence of transient fluctuations in the raw cognitive signals.

**Table 2: Ablation study of different signal smoothing methods on Qwen3-4B-Thinking-2507.**
| Method | MATH500 | AIME | HUMANEVAL+ | GPQA-D | Average |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc$\uparrow$ | Tok$\downarrow$ | Acc$\uparrow$ | Tok$\downarrow$ | Acc$\uparrow$ | Tok$\downarrow$ | Acc$\uparrow$ | Tok$\downarrow$ | Acc$\uparrow$ | Tok$\downarrow$ |  |
| Vanilla | 96.60 | 6719 | 86.67 | 19577 | 88.40 | 5133 | 68.69 | 9041 | 85.09 | 10118 |
| NoSmooth | 94.20 | 3608 | 73.33 | 17832 | 92.10 | 2789 | 62.12 | 4066 | 80.44 | 7074 |
| Momentum | 91.80 | 2230 | 60.00 | 12427 | 83.50 | 2219 | 64.65 | 3406 | 74.99 | 5071 |
| Sliding Window | 95.20 | 4444 | 80.00 | 19184 | 86.60 | 4342 | 62.12 | 4738 | 80.98 | 8177 |
| Paragraph Mean | 95.40 | 4285 | 76.67 | 18231 | 84.80 | 4569 | 65.66 | 4726 | 80.63 | 7953 |
| SpecExit* (EWMA) | 96.80 | 4777 | 90.00 | 17769 | 89.60 | 4319 | 68.69 | 7011 | 86.27 | 8469 |

Step Split Tokens. To evaluate the influence of different step split strategies on early-exit performance, we conducted ablation experiments comparing paragraph delimiters, general discourse markers, and a contrastive subset of discourse markers. Discourse markers indicate semantic transitions or reasoning shifts, but are dependent on the underlying data and model, limiting their generality.
In prior work on dynamic early-exit methods, contrastive subsets of discourse markers (e.g., “Wait”, “But”, “Alternatively”) are frequently used to capture reasoning-relevant transitions.
In contrast, paragraph delimiters (\n\n) provide a more general segmentation that does not rely on model-specific or dataset-specific patterns. As shown in Figure [6](https://arxiv.org/html/2509.24248v2#S4.F6), using paragraph delimiters achieves competitive accuracy and token reduction, demonstrating that a general segmentation strategy can be effective for early-exit decisions while maintaining coherence in reasoning trajectories.

Figure: (a) Accuracy comparison.
Refer to caption: x8.png

In summary, our ablation studies on signal types, smoothing strategies, and step split methods provide key insights for improving early-exit decision-making. The integration of multiple signals strikes the best balance between accuracy and token efficiency, while appropriate smoothing methods stabilize cognitive signals and enhance the consistency of early exits. Additionally, using general segmentation strategies, such as paragraph delimiters, improves the generalizability of early-exit systems across diverse datasets. These findings emphasize the importance of a holistic approach, where complementary strategies jointly enhance both efficiency and reliability.

## 5 Related Work

Efficient Reasoning.
To mitigate unnecessary CoT generation in LRMs (Chen et al., 2025; Sui et al., 2025), prior work has explored both training-based and inference-time strategies.
Training-based approaches typically modify model behavior through reinforcement learning with length-sensitive objectives (Aggarwal & Welleck, 2025; Yeo et al., 2025; Shen et al., 2025) or supervised fine-tuning on reasoning traces of varying lengths (Ma et al., 2025b; Munkhbat et al., 2025). While effective in shortening outputs, these methods demand substantial retraining cost and can distort the model’s output distribution, raising concerns about reliability and generalization to unseen tasks. Inference-time methods avoid retraining and instead attempt to stop reasoning dynamically by monitoring model signals such as logits (Yang et al., 2025) or intermediate answers (Fu et al., 2024). Although these methods show that early stopping can reduce reasoning length without degrading accuracy, their reliance on probing introduces additional computation and often emphasizes token count reduction rather than true end-to-end latency improvements.

Speculative Decoding and Hidden States.
Speculative decoding (Chen et al., 2023; Leviathan et al., 2023) is a widely adopted technique for accelerating decoding speed, where a lightweight draft model proposes candidate tokens that a larger target model verifies in a single pass. Recent methods (Cai et al., 2024; Li et al., 2024a; b; 2025; Zhang et al., b) leverage hidden states to predict multiple future tokens. Beyond speculative decoding, several studies (Lin et al.,; Zhang et al., c; Dong et al.,; Zhang et al., a), have revealed that hidden states contain broader information about future outputs, including correctnes, response length, and reasoning paths.
Building on this insight, our method extends speculative decoding by training hidden states not only to forecast future tokens but also to produce an early-exit signal.

## 6 Conclusion

In this work, we propose SpecExit, a reasoning-aware early-exit framework that leverages latent signals from models’ hidden states to dynamically terminate reasoning processes in LRMs. By concatenating auxiliary prediction heads to a lightweight draft model, SpecExit simultaneously predicts future tokens and early-exit signals in a single forward pass, eliminating the probing overhead required by previous approaches. Our experiments across diverse tasks and models demonstrate that SpecExit substantially reduces reasoning length by 66% and achieves significant end-to-end latency improvements up to 2.5x without compromising accuracy. The proposed method highlights the potential of hidden states as informative signals for efficient reasoning and establishes a practical pathway for deploying LRMs in real-world scenarios.

## References

- Aggarwal & Welleck (2025)
Pranjal Aggarwal and Sean Welleck.
L1: Controlling how long a reasoning model thinks with reinforcement learning.
*arXiv preprint arXiv:2503.04697*, 2025.
- (2)
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini.
Large language monkeys: Scaling inference compute with repeated sampling.
URL [http://arxiv.org/abs/2407.21787](http://arxiv.org/abs/2407.21787).
version: 1.
- Cai et al. (2024)
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao.
Medusa: Simple llm inference acceleration framework with multiple decoding heads.
*arXiv preprint arXiv:2401.10774*, 2024.
- Chen et al. (2023)
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper.
Accelerating large language model decoding with speculative sampling.
*arXiv preprint arXiv:2302.01318*, 2023.
- Chen et al. (2025)
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu.
Do not think that much for 2+3=? on the overthinking of o1-like llms, 2025.
URL [https://arxiv.org/abs/2412.21187](https://arxiv.org/abs/2412.21187).
- Clark et al. (2018)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
*arXiv preprint arXiv:1803.05457*, 2018.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
- DeepSeek-AI et al. (2025)
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong
Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu,
Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- (9)
Zhichen Dong, Zhanhui Zhou, Zhixuan Liu, Chao Yang, and Chaochao Lu.
Emergent response planning in LLMs.
URL [http://arxiv.org/abs/2502.06258](http://arxiv.org/abs/2502.06258).
- Fu et al. (2024)
Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang.
Efficiently serving llm reasoning programs with certaindex.
*arXiv e-prints*, pp. arXiv–2412, 2024.
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.
Measuring mathematical problem solving with the math dataset, 2021.
URL [https://arxiv.org/abs/2103.03874](https://arxiv.org/abs/2103.03874).
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the 29th Symposium on Operating Systems Principles*, SOSP ’23, pp. 611–626, New York, NY, USA, 2023. Association for Computing Machinery.
ISBN 9798400702297.
doi: 10.1145/3600006.3613165.
URL [https://doi.org/10.1145/3600006.3613165](https://doi.org/10.1145/3600006.3613165).
- Leviathan et al. (2023)
Yaniv Leviathan, Matan Kalman, and Yossi Matias.
Fast inference from transformers via speculative decoding.
In *International Conference on Machine Learning*, pp. 19274–19286. PMLR, 2023.
- Li et al. (2024a)
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang.
EAGLE: Speculative sampling requires rethinking feature uncertainty.
In *International Conference on Machine Learning*, 2024a.
- Li et al. (2024b)
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang.
EAGLE-2: Faster inference of language models with dynamic draft trees.
In *Empirical Methods in Natural Language Processing*, 2024b.
- Li et al. (2025)
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang.
EAGLE-3: Scaling up inference acceleration of large language models via training-time test, 2025.
URL [https://arxiv.org/abs/2503.01840](https://arxiv.org/abs/2503.01840).
- (17)
Zhengkai Lin, Zhihang Fu, Ze Chen, Chao Chen, Liang Xie, Wenxiao Wang, Deng Cai, Zheng Wang, and Jieping Ye.
Controlling thinking speed in reasoning models.
URL [http://arxiv.org/abs/2507.03704](http://arxiv.org/abs/2507.03704).
version: 1.
- Liu et al. (2023)
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.
Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.
*Advances in Neural Information Processing Systems*, 36:21558–21572, 2023.
- Ma et al. (2025a)
Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia.
Reasoning models can be effective without thinking.
*arXiv preprint arXiv:2504.09858*, 2025a.
- Ma et al. (2025b)
Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang.
Cot-valve: Length-compressible chain-of-thought tuning.
*arXiv preprint arXiv:2502.09601*, 2025b.
- (21)
MAA Committees.
Aime problems and solutions.
[https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions](https://artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions).
- (22)
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto.
S1: Simple test-time scaling.
URL [http://arxiv.org/abs/2501.19393](http://arxiv.org/abs/2501.19393).
version: 1.
- Munkhbat et al. (2025)
Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang, Yujin Kim, and Se-Young Yun.
Self-training elicits concise reasoning in large language models.
*arXiv preprint arXiv:2502.20122*, 2025.
- OpenAI (2024)
OpenAI.
Learning to reason with llms, September 2024.
URL [https://openai.com/index/learning-to-reason-with-llms/](https://openai.com/index/learning-to-reason-with-llms/).
- Qwen et al. (2025)
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report, 2025.
URL [https://arxiv.org/abs/2412.15115](https://arxiv.org/abs/2412.15115).
- Rein et al. (2023)
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman.
Gpqa: A graduate-level google-proof q&a benchmark, 2023.
URL [https://arxiv.org/abs/2311.12022](https://arxiv.org/abs/2311.12022).
- Shen et al. (2025)
Yi Shen, Jian Zhang, Jieyun Huang, Shuming Shi, Wenjing Zhang, Jiangze Yan, Ning Wang, Kai Wang, and Shiguo Lian.
Dast: Difficulty-adaptive slow-thinking for large reasoning models.
*arXiv preprint arXiv:2503.04472*, 2025.
- (28)
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar.
Scaling llm test-time compute optimally can be more effective than scaling model parameters.
URL [http://arxiv.org/abs/2408.03314](http://arxiv.org/abs/2408.03314).
version: 1.
- Sui et al. (2025)
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, Hanjie Chen, and Xia Hu.
Stop overthinking: A survey on efficient reasoning for large language models, 2025.
URL [https://arxiv.org/abs/2503.16419](https://arxiv.org/abs/2503.16419).
- (30)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language models.
URL [http://arxiv.org/abs/2201.11903](http://arxiv.org/abs/2201.11903).
version: 1.
- Yang et al. (2025)
Chenxu Yang, Qingyi Si, Yongjie Duan, Zheliang Zhu, Chenyu Zhu, Zheng Lin, Li Cao, and Weiping Wang.
Dynamic early exit in reasoning models.
*arXiv preprint arXiv:2504.15895*, 2025.
- Yeo et al. (2025)
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue.
Demystifying long chain-of-thought reasoning in llms.
*arXiv preprint arXiv:2502.03373*, 2025.
- Zhang et al. (a)
Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He.
Reasoning models know when they’re right: Probing hid- den states for self-verification.
a.
- Zhang et al. (b)
Lefan Zhang, Xiaodan Wang, Yanhua Huang, and Ruiwen Xu.
Learning harmonized representations for speculative sampling, b.
URL [http://arxiv.org/abs/2408.15766](http://arxiv.org/abs/2408.15766).
- Zhang et al. (c)
Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang.
Soft thinking: Unlocking the reasoning potential of LLMs in continuous concept space, c.
URL [http://arxiv.org/abs/2505.15778](http://arxiv.org/abs/2505.15778).

## Appendix A Appendix

This section may contain supplementary materials such as additional experimental details, ablation studies, hyperparameter settings, and qualitative examples of generated CoT sequences.

### A.1 Implementation details

Stop Threshold: In the ablation study of SpecExit signal types, the following thresholds are applied as stopping conditions for the respective signal types: SpecExit-Confidence requires predicted confidence value greater than 0.9, SpecExit-Progress requires predicted progress value greater than 0.8, SpecExit-Remain requires predicted remaining reasoning length value less than 100, and SpecExit* combines thresholds with a predicted confidence value greater than 0.8, predicted progress greater than 0.3, and predicted remaining reasoning length less than 200.

Signal Smoothing: In the ablation study of signal smoothing strategies, the following methods are implemented to stabilize cognitive signals for early-exit decisions:

- •
Sliding Window: The sliding window approach smooths the signal by averaging the last $N$ predicted signal values, with $N$ set to 10. The mean score $x_{t}$ at decoding step $t$ is computed as:
$x_{t}=\text{Mean}(s_{t},N)=\frac{1}{N}\sum_{i=t-N+1}^{t}s_{i},$
(7)
where $s_{i}$ denotes the predicted signal value at decoding step $i$.
- •
Momentum-based Prediction: This method predicts the next score based on the momentum, which is calculated as the difference between $N$ consecutive signal values, with $N$ set to 10. The predicted score $x_{t}$ at decoding step $t$ is given by:
$x_{t}=\text{Predict}(s_{t},N)=s_{t-1}+\frac{1}{N-1}\sum_{i=t-N+1}^{t-1}(s_{i}-s_{i-1}).$
(8)
- •
Paragraph Mean: In this approach, the score $x_{t}$ is calculated as the average of all predicted signal values within the current paragraph:
$x_{t}=\frac{1}{T}\sum_{i=1}^{T}s_{i},$
(9)
where $T$ is the total number of steps in the current paragraph.
- •
Exponential Weighted Moving Average (EWMA): In this approach, the smoothing factor $\alpha$ is set to 0.1. The new score $x_{t}$ is updated based on the previous score $x_{t-1}$ and the current signal value $s_{t}$ as:
$x_{t}=\text{EWMA}(s_{t},x_{t-1},\alpha)=\alpha\cdot s_{t}+(1-\alpha)\cdot x_{t-1}.$
(10)

Discourse markers:
We collect high-frequency words appearing at the beginning of model-generated sentences as discourse markers. As shown in Figure [7](https://arxiv.org/html/2509.24248v2#A1.F7), we present examples of discourse markers extracted from Qwen3-4B-Thinking-2507 on the MATH dataset. Among them, transitional words such as ”Wait” and ”But” can be regarded as a subset of these high-frequency markers.

Figure: Figure 7: Discourse marker distribution in Qwen3-4B-Thinking-2507’s responses on the MATH500 (Hendrycks et al., 2021) dataset.
Refer to caption: figs/fig4b_wordcloud.png

### A.2 Predicted Signal Visualization

As shown in Figure [8](https://arxiv.org/html/2509.24248v2#A1.F8), for simple problems, the predicted confidence remains high, the predicted remaining reasoning length is relatively short, and the predicted progress rises rapidly within the first few sentences, with only minor drops on a few uncertain words. This indicates that for such problems, the model is able to establish a stable reasoning trajectory at an early stage. In contrast, as shown in Figure [9](https://arxiv.org/html/2509.24248v2#A1.F9), for complicated problems, the model also exhibits high confidence and a short remaining reasoning length in the initial summarization phase, but once it enters the detailed analysis stage, the predicted remaining reasoning length increases significantly, confidence drops, and progress, though rising in the beginning, fluctuates heavily during repeated self-reflection, making it difficult to stabilize at a high threshold.

These observations reveal the inherent limitations of relying on individual signals for early exiting. When depending solely on confidence, the model often exhibits overconfidence and terminates too early before sufficient reasoning has been completed, leading to substantial accuracy degradation. When depending solely on the predicted remaining reasoning length, the model may become overly optimistic in the early stages of complicated problems, resulting in premature exits before essential reasoning steps are accomplished. When depending solely on progress, the signal tends to fluctuate and remain unstable in complex reasoning tasks, making it difficult to trigger an appropriate early exit and thereby restricting achievable speedup. In summary, each single signal suffers from the inability to balance accuracy and efficiency across diverse problem types. By integrating multiple signals in a complementary manner, the model can achieve a smoother trade-off between reasoning accuracy and inference acceleration.

Figure: Figure 8: Visualization of reasoning signals for a simple problem, illustrated with an example from the MATH500 (Hendrycks et al., 2021) dataset, where darker colors denote insufficient reasoning and lighter colors denote sufficiency.
Refer to caption: x10.png

Figure: Figure 9: Visualization of reasoning signals for a complicated problem, illustrated with an example from the AIME (MAA Committees,) dataset, where darker colors denote insufficient reasoning and lighter colors denote sufficiency.
Refer to caption: x13.png

### A.3 Case Study Details

Figure [10](https://arxiv.org/html/2509.24248v2#A1.F10) presents an example from the GSM8K (Cobbe et al., 2021) dataset, where SpecExit is applied after an initial analysis. In this case, SpecExit inserts a decision to exit reasoning based on the signal magnitude after completing the first paragraph, thus preventing the continuation of redundant reasoning tokens. The process begins with the model evaluating the initial segment of the problem, analyzing the available context and producing intermediate reasoning steps. When SpecExit detects that all the signals exceeds a predefined threshold, it triggers the early exit mechanism. As a result, the model halts further reasoning and transitions directly to the answering phase, thus producing the correct result in a more efficient manner.

By shortening the unnecessary token generation and reducing redundant reasoning, SpecExit leads to significant performance improvements in terms of inference speed while maintaining the correctness of the answer. This early exit strategy not only accelerates the inference process but also avoids the potential confusion that the model might experience from excessive reflection or unnecessary computations. This case highlights the effectiveness of integrating early exit mechanisms in enhancing both the efficiency and reliability of large language model inference.

Figure: Figure 10: Discourse marker distribution in Qwen3-4B-Thinking-2507’s responses on the MATH500 (Hendrycks et al., 2021) dataset.
Refer to caption: x16.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How sensitive is SpecExit to the chosen thresholds (confidence > 0.8, progress > 0.3)? Could these be learned dynamically?
2. Have you tested **cross-model transferability** of exit heads (e.g., trained on Qwen, applied to DeepSeek)?
3. How would SpecExit behave with **long-context**?
4. Could you quantify **energy / GPU-hour savings** corresponding to the reported 2.5× latency reduction?
5. Can you provide a theoretical account—stating assumptions and guarantees—that explains why functions of intermediate hidden states serve as reliable proxies for ‘reasoning sufficiency’ (e.g., monotonicity/calibration of the confidence and progress heads)?
soundness: 3
strengths: * Novel integration of early-exit and speculative decoding via hidden-state supervision.
* Eliminates probing overhead while maintaining accuracy.
* Multi-signal design (confidence + progress + remaining length) improves robustness across tasks.
* Experiments on six benchmarks and two reasoning-optimized backbones.
* Ablations on signal types, smoothing strategies, and segmentation markers.
* Reproducibility and open-source release fully documented.
* Clear contextualization within reasoning-efficiency literature (DEER, EAGLE, RL-based control).
summary: This paper introduces **SpecExit**, a *reasoning-aware early-exit framework* designed to reduce overthinking in large reasoning models (LRMs).
Instead of relying on explicit probing phrases (e.g., “Final Answer is”), SpecExit leverages **hidden states** from a lightweight **draft model** to jointly predict (i) future tokens and (ii) early-exit signals—**confidence**, **reasoning progress**, and **remaining reasoning length**—in a single forward pass.
The method removes probing overhead, integrates smoothly with **speculative decoding**, and dynamically stops generation at semantically coherent boundaries such as paragraph delimiters.
Experiments on GSM8K, MATH500, AIME, GPQA, HumanEval+, and ARC-Challenge show up to **66 % shorter reasoning chains** and **2.5 × latency reduction** without accuracy loss.
Ablation studies confirm that multi-signal integration with EWMA smoothing yields the best balance between efficiency and correctness.
weaknesses: * Lacks a **formal theoretical analysis** of why hidden-state features correlate with reasoning sufficiency.
* Dependence on explicit *<think>* / *</think>* markers may limit generalization to free-form reasoning.
* Limited model family evaluation; Experiments are conducted on two reasoning model families.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: Please refer to Weaknesses.
soundness: 3
strengths: The primary strength of this work lies in its novel and elegant integration of two distinct lines of research: speculative decoding for per-token acceleration and early-exit mechanisms for reducing reasoning length. Instead of treating these as separate problems, the authors propose a unified framework. The key insight—to offload the exit-signal prediction to the draft model and leverage its hidden states—is highly original. This avoids the well-documented overhead of probing-based early-exit methods, which is a creative and impactful conceptual leap.
summary: This paper introduces SpecExit, a novel framework designed to accelerate inference in Large Reasoning Models (LRMs) by tackling the "overthinking" problem, where models produce unnecessarily long reasoning chains, leading to high latency. The core idea is to integrate an early-exit mechanism directly into the speculative decoding process. Unlike prior methods that rely on costly probing of the target model, SpecExit extends the lightweight draft model to predict not only future tokens but also three auxiliary signals: confidence, reasoning progress, and remaining reasoning length. These signals are learned via multi-task training on data where the minimal sufficient reasoning length is heuristically identified. During inference, these signals are used to dynamically terminate the reasoning process at natural breakpoints (e.g., paragraph ends) without any modification to the target model or extra probing overhead. Experiments on various reasoning benchmarks with models like Qwen3-4B and DeepSeek-R1-Distill show that SpecExit can reduce generation length by up to 66% and achieve end-to-end latency speedups of up to 2.5x compared to a standard speculative decoding baseline, with minimal impact on task accuracy.
weaknesses: 1. The author should introduce some basic knowledge of speculative decoding to facilitate readers who are not familiar with speculative decoding to understand the content of the paper.

2. Figure 2 only shows the forward process of one token. I don't understand how to predict the vocab logits and signals of multiple tokens simultaneously. The author needs to explain in detail how the MTP layer in the paper is designed. Is it exactly the same structure as Medusa or EAGLE?

3. The author combines speculative decoding with exit prediction, but I believe there is no coupling relationship between the two. According to the method designed in this paper, adding an MLP layer directly to the hidden feature of the token can also predict the exit signal without the need for an MTP layer.

4. The author can supplement the experimental results without speculative decoding (only predicting the exit signal of each token).

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: - Please provide detailed configurations of the draft model and its relation to the target model (architecture, size, tokenizer compatibility).

- Can the MLP exit heads trained on one model generalize to another of similar architecture?

- How large is the training overhead compared to runtime gains?

- Could the authors clarify whether the method assumes the draft and target models belong to the same family?

- Why were the three particular auxiliary losses (L_conf, L_prog, L_rem) chosen, and is there theoretical justification for their complementarity?
soundness: 2
strengths: 1. The paper combines speculative decoding, multi-task learning, and confidence-based exit criteria in a coherent way to address reasoning efficiency.

2. Demonstrates significant inference latency reduction with minimal accuracy degradation across tasks.

3. The analysis of different signal types (Confidence / Progress / Remaining Reasoning Length) and smoothing methods (No Smoothing / Momentum / Sliding Window / EWMA) is thorough and informative.

4. The method offers a general framework applicable to existing reasoning models without modifying their core architecture.
summary: The work introduces SpecExit, a reasoning-aware early exit framework for reducing the overall latency of the reasoning process. The method involves using a combination of speculative decoding and multi-task learning, to generate confidence signals from the draft model which guide the system to pre-empt the reasoning process eﬃciently. Evaluation across a range of tasks, the work claims to achieve 66% reduction in average generation length and 2.5x speedup in end-to-end latency, compared to the baseline.
weaknesses: - Line 267: The notion of “recover token” is unclear. How does it differ from the reasoning-end marker  ("\</think>")?
- Lines 363–364: The variant using combined signals (Confidence + Progress + Remaining Length) is denoted as SpecExit*, yet Table 1 reports results for SpecExit. Which configuration do these results correspond to?
- Table 1: Ambiguity exists between “Think” (line 333) and “Vanilla” (line 339). If they refer to the same baseline, HUMANEVAL+ accuracies (90.9 vs 88.4) are inconsistent.
- Table 1 caption should define what boldface indicates (best overall or best among comparable methods).

---
- Lines 313–314 mention that the speculative decoding baseline is trained on the same dataset as EAGLE3, but the draft model configuration is unspecified.
- What is the relationship between the draft and target models (same family, smaller variant, or identical)? Without this detail, reproducibility and interpretation of results are limited.

---
- Please quantify additional compute (e.g., FLOPs or GPU hours) for:
    - Dataset construction, which involves multiple reasoning traces per prompt to determine optimal lengths.
    - Multi-task learning with additional loss heads.
- These costs should be weighed against runtime improvements.
---

- Does the target LLM architecture affect training of the MLP heads? Since the dataset is derived from a specific model’s reasoning traces, how is overfitting avoided? Are exit heads transferable across models within the same family?
---

- The inference flow involving the MTP mechanism, lightweight draft model, MLP head, and target model is confusing. Please clarify whether:
    - The draft model includes MTP heads,
    - These predictions are passed to the target for verification, and
    - The target model then generates extended outputs via a linear layer.
- A schematic or algorithmic diagram would make the inference pipeline clearer.
---

- What is the motivation for three auxiliary losses (L_conf, L_prog, L_rem). The rationale for choosing these specific losses is unclear. What intuition or empirical observation led to defining these signals? Figure 5 shows dataset-dependent variation for Progress and Remaining signals, while Confidence remains consistently low. What happens if only Progress and Remaining signals are used?
- The abstract claims an “average generation length reduction of 66%,” but results suggest this is the maximum observed, not the mean. The phrasing should be corrected to “up to 66% reduction.”

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Since the MTP and signal heads must be retrained for each model due to architectural differences, how costly is this process in practice? Could a lightweight or partially shared training strategy (e.g., freezing most of the MTP layer or distilling from a pretrained head) reduce the retraining overhead across models?
-  Can the authors provide results or at least discussion on how SpecExit might generalize to other model architectures, such as Phi, Mistral, or LRMs?
- How sensitive is SpecExit’s performance to the chosen thresholds for confidence, progress, and remaining length? like use different thresholds to do some ablation studies. Why do you choose these thresholds in STEER method (like confidence > 0.8 && progress > 0.3 && remaining < 200)?
- Can the authors identify and analyze typical scenarios where the control signals conflict (e.g., rising confidence but low progress), and show how the EWMA controller behaves in these situations?
soundness: 3
strengths: - Elegant integration of early-exit prediction into speculative decoding with zero probing overhead.
- Uses hidden-state signals (confidence/progress/remaining) to dynamically determine reasoning sufficiency.
- Demonstrates consistent 2×–2.5× speedup and 66% token reduction without hurting accuracy.
- Framework is general, modular, and easily deployable on vLLM or PyTorch inference pipelines.
- Well-designed ablation studies confirm the effectiveness of multi-signal fusion and signal smoothing.
summary: This paper proposes SpecExit, an early-exit framework that integrates with speculative decoding to predict both future tokens and reasoning sufficiency signals from a lightweight draft model, achieving up to 2.5× faster inference and 66% shorter reasoning without accuracy loss.
weaknesses: - Model-specific training requirement: The MTP-based signal extraction head must be retrained for each target model due to differences in hidden-state representations, limiting plug-and-play generality.
- Limited model family evaluation: Experiments are conducted only on two reasoning model families, leaving generalization to other architectures or general-purpose LLMs unclear.
- Signal & smoothing dependency: The method relies on multiple signals and smoothing strategies (e.g., EWMA) to remain stable, indicating insufficient robustness of raw signals.


Decision: Reject