SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification

## Abstract

Abstract Speculative decoding accelerates LLM inference by verifying candidate tokens from a draft model against a larger target model. Recent “judge” decoding boosts this process by relaxing verification criteria by accepting draft tokens that may exhibit minor discrepancies from target model output, but existing methods are restricted by their reliance on human annotations or tasks with verifiable ground truths, limiting generalizability across diverse NLP tasks. We propose SelfJudge , which trains judge verifiers via self-supervision of the target model. Our method measures semantic preservation by assessing whether token-substituted responses preserve the meaning of original responses, enabling automatic verifier training across diverse NLP tasks. Our experiments show SelfJudge achieves superior inference-accuracy trade-offs than judge decoding baselines, offering a broadly applicable solution for faster LLM inference.

## 1 Introduction

Modern large language models (LLMs) have made significant strides in natural language processing, demonstrating competitive performance across a wide range of tasks (Radford et al., 2019; OpenAI et al., 2024). Empirical scaling laws establish a relationship between the number of parameters and model capability, as evidenced by models with hundreds of billions of parameters achieving state-of-the-art results on benchmarks (Kaplan et al., 2020; Grattafiori et al., 2024). However, the autoregressive generation process requires accessing all model parameters for each forward pass, creating a memory bandwidth bottleneck that impacts token generation latency. Furthermore, current trends toward more sophisticated LLM applications, such as multi-hop reasoning (Wei et al., 2022), tool integration (Patil et al., 2024), and reasoning capability (Yang et al., 2025; DeepMind, 2025), produce longer output sequences, amplifying the computational burden of inference.

One prominent approach to address inference latency is Speculative Decoding (SD), which achieves partial parallelization of the generation process (Leviathan et al., 2023; Chen et al., 2023). Standard SD operates by deploying a computationally efficient *draft* model to propose candidate token sequences, which are subsequently validated in parallel by the *target* model (the model of interest). The acceptance criterion for draft tokens relies on a probability-based alignment verification: draft tokens are accepted when their likelihood under the target model meets or exceeds their likelihood under the draft model. This probabilistic acceptance mechanism guarantees distributional equivalence with standalone target model generation, preserving output quality while achieving computational speedup (Li et al., 2024a; Sun et al., 2023; 2024; Hu et al., 2025; Zhou et al., 2024).

Recently, Bachmann et al. (2025) and Garipov et al. (2025) revealed that alignment-based verification approaches are overly conservative and lack contextual awareness. Specifically, when minor lexical variations occur between draft and target models without altering the contextual meaning, conventional SD rejects these draft tokens. The conservative behavior limits the speedup potential of SD even when output quality remains uncompromised. To address the limitation, Bachmann et al. (2025) proposed the judge decoding that relaxes strict token alignment by evaluating semantic compatibility through a learnable *judge verifier*, enabling greater inference acceleration.

Figure: Figure 1: Inference efficiency and task performance comparison of SD methods on GSM8K (a,b) and MMLU (c,d). AutoJudge shows domain-specific limitations, performing well on mathematical reasoning but poorly on general knowledge tasks, while SelfJudge maintains consistent performance across both domains. $\gamma$ represents the number of tokens generated by draft model per step.
Refer to caption: x1.png

However, judge verification approaches face substantial practical limitations in judge verifier training. Effective judge verifiers require token-level supervision data indicating acceptance decisions for individual tokens, a challenging annotation requirement. Recent approaches have attempted to address this through human annotation (Bachmann et al., 2025) or automated labeling based on task correctness (Garipov et al., 2025), but both strategies exhibit fundamental constraints. Human-annotated approaches introduce bias due to annotation subjectivity and divergence from target model decisions. In contrast, automated methods are constrained to tasks with verifiable ground truth, restricting applicability to specific domains such as mathematical reasoning and code generation. This domain constraint is demonstrated empirically in Figure [1](https://arxiv.org/html/2510.02329v1#S1.F1), where automated approaches such as AutoJudge achieve substantial speedup on mathematical reasoning tasks (GSM8K) but exhibit significant performance degradation on general knowledge evaluation (MMLU).

In this paper, we propose SelfJudge, a speculative decoding framework that enables judge verification across general NLP tasks without requiring human supervision or verifiable ground truth. Our approach exploits self-supervision from the target model to generate verifier training data. We introduce a criterion for token acceptability based on semantic preservation, quantified as the likelihood difference between the original target response and token-substituted variants under the target model’s distribution. Tokens whose replacement minimally affects the target model’s response likelihood are labeled as acceptable for training; tokens that substantially alter likelihood are rejected. This semantic preservation criterion enables automatic supervision generation across diverse task domains, including open-ended question answering and text summarization. The generated data is used to train a lightweight verifier that performs context-aware token evaluation during inference, accepting draft proposals based on semantic compatibility rather than distribution alignment.

We evaluate SelfJudge on diverse NLP tasks and datasets, including GSM8K, MATH-500, MMLU, CNN/DailyMail and LiveCodeBench. We demonstrate that SelfJudge outperforms existing verification methods by achieving superior accuracy-efficiency trade-offs across domains. While prior judge verification methods like AutoJudge achieve +1.96 accepted tokens per verification cycle with significant task performance degradation (-2.7% accuracy), our approach delivers enhanced +2.06 accepted tokens per verification cycle with only -1.0% accuracy. The improvement demonstrates SelfJudge’s superior applicability to general-purpose NLP scenarios, maintaining task quality while enabling efficient speculative decoding across diverse domains.

In summary, we make the following contributions:

- •
We propose SelfJudge, a novel judge verification approach that enables generalization across diverse NLP tasks by accepting semantically coherent output tokens for faster LLM inference.
- •
We introduce an automatic training data generation method for the judge verifier that labels tokens using a semantic preservation score, measured by the target model’s likelihood differences.
- •
We conduct comprehensive experiments across NLP tasks using the Llama-3 and Qwen-2.5 families, demonstrating that SelfJudge generally outperforms existing verification methods in terms of token generation speed and task performance.

## 2 Related Works

### 2.1 Speculative Decoding

SD is a partial parallelization approach that addresses the latency of autoregressive generation. The core idea is to use an efficient module to generate drafts, which are then verified in parallel by the target model. Existing SD methods can be categorized into parameterized and parameter-free approaches. First, several SD methods have used small language models from the same family to suggest drafts (Leviathan et al., 2023; Miao et al., 2024). To enhance the alignment between draft and target models, advanced approaches propose specialized drafting architectures that utilize parts in the target model (e.g., hidden representation, LM head) (Stern et al., 2018; Cai et al., 2024; Li et al., 2024a; 2025). Moreover, recent approaches tried to improve the distribution alignment through knowledge distillation, which improves the draft acceptance rate (Zhou et al., 2024; Liu et al., 2024). In contrast, parameter-free SD methods produce a draft using tokens from user prompts or generated history (Fu et al., 2024; Luo et al., 2025; Oliaro et al., 2025). While previous approaches have focused on improving the drafting phase to generate draft tokens that the target model is likely to accept, our work focuses on developing an improved verification method that allows the acceptance of these tokens without altering their semantic meaning. In this paper, our goal is to achieve speedup while maintaining task performance through our proposed verification approach.

### 2.2 Judge Speculative Decoding

Standard SD methods perform draft verification based on modified rejection sampling (Chen et al., 2023), which accepts tokens when the token probabilities from the target model meet or exceed those of the draft model. The advantage of using rejection sampling is that it preserves the distribution of the target model without degrading output quality. However, this alignment-based verification is overly conservative, which limits the speedup of SD when there are minor token discrepancies between draft and target models.
To address this limitation, JudgeDecoding (Bachmann et al., 2025) have been proposed.
JudgeDecoding utilizes a lightweight classifier as verifier, which examines the hidden representations of tokens. However, training such verifiers remains challenging as it requires labels indicating whether each token should be accepted or not.
Hence, AutoJudge (Garipov et al., 2025) automates the data generation process by using verifiable ground truth in math and coding tasks.
Specifically, AutoJudge adopts the answer-preservation approach, where it systematically tests token alternatives while maintaining correctness of math and coding problems. The method replaces individual tokens in a correct response and completes the modified response. If the final answer remains correct, it considers the replaced token as acceptable.
Despite these advances, current judge verification methods lack generalization ability across diverse NLP tasks, showing performance improvements primarily on math and coding tasks. In this paper, we propose a judge verification approach that trains the verifier using semantic coherence from the original response, aiming to improve generalization across diverse NLP domains.

## 3 Method

In this section, we propose SelfJudge: a novel speculative decoding approach that enables judge verification across general NLP tasks without requiring human annotations or ground truth answers. The key innovation of SelfJudge lies in leveraging the semantic information inherent in the target model to automatically generate verifier training data.

As shown in Figure [2](https://arxiv.org/html/2510.02329v1#S3.F2), SelfJudge introduces a self-data-generation paradigm where the target model assesses the semantic change of the original response with a token substituted. Specifically, we define and measure semantic preservation by comparing the likelihood in the target model’s original response versus token-substituted variants. Tokens whose replacement does not significantly affect the model’s overall response confidence are labeled as acceptable, enabling automatic generation of high-quality training data across diverse NLP domains.
This semantic preservation-based approach fundamentally shifts the verification paradigm from exact token matching or task-specific correctness to general semantic preservation. By obtaining the semantic preservation in the perspective of the target model, our approach makes it applicable to any NLP tasks, including open-ended tasks (e.g., creative writing) where ground truth is unavailable.

Figure: Figure 2: The training data generation process of SelfJudge for the verifier. Our approach compares the likelihood of the replaced response with the original response to measure the semantic preservation score. If semantic preservation score is higher than $\tau$, the replaced token is labeled as acceptable. After the token labeling process, we train the verifier that will be used during the inference phase for draft verification.
Refer to caption: x2.png

### 3.1 Conventional Speculative Decoding

We begin by introducing conventional SD. The core idea of SD is to efficiently generate drafts and verify them in parallel. Specifically, we generate $\gamma$ candidate tokens using a small language model (Leviathan et al., 2023) or a specialized autoregressive model (Li et al., 2024a) to create drafts. When these generated candidate tokens are fed into the target model, we can verify whether the tokens are sufficiently aligned with the target model’s output using the probability value of next token.

Draft Generation. Formally, let $\mathcal{M}_{\text{target}}$ be the large language model of interest, and $\mathcal{M}_{\text{draft}}$ be the efficient model with a smaller number of parameters compared to $\mathcal{M}_{\text{target}}$. In the drafting phase, given prefix $\mathbf{x}_{\leq i}$, we first generate $\gamma$ tokens using the efficient model autoregressively:

$$ $(d_{i+1},{q_{i+1}})=\mathcal{M}_{\text{draft}}(\mathbf{x}_{\leq i}),\quad\text{where }i=t,t+1,\ldots,t+\gamma-1$ (1) $$

where $d_{i+1}$ represents the generated draft token and $q_{i+1}$ denotes its corresponding probability.

Alignment-based Verification. Subsequently, we feed the draft tokens into the target model to verify whether each token is valid from the target model’s perspective:

$$ $({p_{t+1}},\ldots,p_{t+\gamma},p_{t+\gamma+1})=\mathcal{M}_{\text{target}}(\left[\mathbf{x}_{\leq t};d_{t+1},...,d_{t+\gamma}\right])$ (2) $$

where $p_{t}$ represents the target model’s probability value for each draft token $d_{t}$. The last probability $p_{t+\gamma+1}$ corresponds to additionally sampled token using the target model.

Conventional SD typically employs rejection sampling to verify draft tokens by comparing probability distributions from both the draft and target models for each token, accepting tokens only when the target model probabilities are high or similar to the draft probabilities as follows:

$$ $\displaystyle\text{Accept }d_{t}\text{ if }u_{t}<r_{t},\text{ otherwise reject }d_{t}\quad\text{where}\quad r_{t}$ $\displaystyle=\min\left(1,\frac{p_{t}}{q_{t}}\right),\quad u_{t}\sim\text{Unif}(0,1)$ (3) $$

From the draft sequence, we accept tokens sequentially until the first rejection occurs according to Eqn.([3](https://arxiv.org/html/2510.02329v1#S3.E3)). It ensures that the accepted draft tokens maintain the target distribution. Beyond rejection sampling, other alignment-based methods exist, such as strict speculative sampling (He et al., 2024), which accepts draft tokens only when they match the target model’s greedy predictions exactly.

### 3.2 Judge Verification

As alignment-based verification methods have strict criteria, they lack context-awareness. This strict criterion can be overly conservative, for instance, rejecting *“It’s a strong instrument”* when the target model produces *“It is a strong instrument”*, even though both expressions preserve the same semantic meaning in context. Such unnecessary rejections limit the potential speedup of SD.

To address this issue, JudgeDecoding (Bachmann et al., 2025) performs model-based verification. Given draft tokens $d_{t+1},\ldots,d_{t+\gamma}$, we first obtain the hidden representations of the target model:

$$ $(h_{t+1},\ldots,h_{t+\gamma})=\mathcal{M}_{\text{target}}([\mathbf{x}_{\leq t};d_{t+1},\ldots,d_{t+\gamma}])$ (4) $$

We then perform verification as follows:

$$ $\text{Accept }d_{t}\text{ if }\mathsf{Verifier}(h_{t})>\theta,\text{ otherwise reject }d_{t}$ (5) $$

where $\mathsf{Verifier}$ is a learned binary classifier. For simplicity, we employ a logistic regression model as our verifier, and $\theta$ is a hyperparameter whose value is determined using a holdout validation set with the best metric (e.g., AUROC, recall, and F1 score).

Finally, we employ a *two-stage verification process*: first applying the judge $\mathsf{Verifier}$ to all draft tokens in parallel, then using alignment-based verification (e.g., rejection sampling) for tokens rejected by the verifier. This ensures that no valid tokens are incorrectly discarded.

### 3.3 Generate Data for Judge Verifier

For efficient and accurate judge decoding, obtaining a high-quality judge verifier is paramount. Our core insight is to leverage the target model’s own semantic understanding to train the verifier. By doing so, we increase the accepted tokens by allowing semantically valid tokens that may have a lower probability in the target model while maintaining response quality. Consequently, the judge verifier must identify which tokens preserve the original semantic meaning within the given context.

#### 3.3.1 Semantic Preservation-based Token Labeling

To maintain response quality in judge verification, previous methods have relied on human annotators (Bachmann et al., 2025) or on whether the final answers are preserved when tokens are replaced (Garipov et al., 2025). However, human annotations suffer from subjectivity, and the answer-preservation-based method is limited to tasks with verifiable ground truths.

Rather than relying on external supervision, we utilize the target model’s self-evaluation. Since the goal of SD is to preserve the target model’s generation quality while improving latency, the target model itself is the best judge of semantic preservation. In our approach, the target model directly assesses the semantic impact of token replacements within its own responses. This enables automatic generation of high-quality training data across diverse NLP tasks without domain constraints.

Step 1. Identifying Mismatched Tokens. To improve the efficiency of SD with a verifier, we first choose which tokens to label, focusing on mismatched tokens where the target and draft models disagree. These tokens are candidates for relaxed verification, whereas matched tokens are already handled by the alignment-based verification phase of SD.

Formally, we generate response ${y}$ using the target model given an instruction. We then feed response ${y}$ to the draft model and obtain the draft model’s predictions at each position. We identify mismatched positions $i$ where the target token ${y}_{i}$ differs from the draft model’s top prediction $\arg\max P_{\text{draft}}(\cdot|{y}_{<i})$.

Step 2: Computing Semantic Preservation Score. When the draft model’s token does not match the token output by the target model, we need to determine if the alternative token from the draft model, i.e., $\arg\max P_{\text{draft}}(\cdot|{y}_{<i})$), is a semantically acceptable substitute to the target model. To this end, we introduce a *semantic preservation score*. Our core insight is to consider the alternative token as acceptable if it preserves the semantic meaning of the target model’s response. Therefore, our semantic preservation score quantifies how much of the semantic meaning of the target model’s response is preserved when the target model’s original token is replaced by the draft’s alternative.

Formally, given the target response ${y}$ and the mismatched index $i$, we define the following *semantic preservation score* for token replacement $z_{i}$ as:

$$ $s({y},z_{i})=\log P({z}_{i}|{y}_{<i},{y}_{>i})-\log P({y}_{i}|{y}_{<i},{y}_{>i})$ (6) $$

where $z_{i}$ is the alternative token from the draft model and ${y}_{>i}$ represents the future token sequence. In practice, our semantic preservation score uses $N$ future tokens as opposed to using all future tokens. This score measures how semantically similar the replacement token is to the target model’s established context. To be precise, when $z_{i}$ is semantically similar to the original token ${y}_{i}$, the target model will generate the same tokens in the original response after the replaced token. This results in a small probability difference and thus a high preservation score. Conversely, when $z_{i}$ is semantically inappropriate for the context, it receives a much lower probability than ${y}_{i}$, creating a large probability difference and a small preservation score, suggesting that this replacement should be rejected to maintain response quality.

As many modern LLMs have adopted autoregressive inference, directly computing the probabilities conditioned on the bidirectional context in Eqn.([6](https://arxiv.org/html/2510.02329v1#S3.E6)) is not possible. Therefore, we propose a method of transforming the bidirectional semantic difference into computationally tractable autoregressive operations as follows:

$$ $\displaystyle\log P({z}_{i}|{y}_{<i},{y}_{>i})-\log P({y}_{i}|{y}_{<i},{y}_{>i})$ $\displaystyle=\log P({z}_{i},{y}_{<i},{y}_{i>i})-\log P({y}_{i},{y}_{<i},{y}_{i>i})$ $\displaystyle=\cancel{\log P({y}_{<i})}P({z}_{i}|{y}_{<i})P({y}_{>i}|{y}_{<i},{z}_{i})-\cancel{\log P({y}_{<i})}P({y}_{i}|{y}_{<i})P({y}_{i>}|{y}_{\leq i})$ $\displaystyle=\underbrace{\log P({z}_{i}|{y}_{<i})-\log P({y}_{i}|{y}_{<i})}_{s_{\text{prefix}}}+\underbrace{\log P({y}_{>i}|{y}_{<i},{z}_{i})-\log P({y}_{>i}|{y}_{\leq i})}_{\text{suffix likelihood}}$ (7) $$

$s_{\text{prefix}}$ represents the relative probability given the prefix, computing the difference between the original and replaced token probabilities given the same prefix. The second term represents future context consistency, which computes the likelihood difference of future tokens in the target model’s response. Note that we obtain the probabilities conditioned on the replaced token by directly feeding the token-substituted response, i.e., $[y_{<i},z_{i},y_{>i}]$, into the target model.

Step 3: Token Labeling. Based on the semantic preservation score computed on the mismatched tokens, we finally obtain the label for each token. Specifically, a replaced token is labeled as acceptable if $s({y},{z}_{i})>\tau$. Otherwise, the replaced token is labeled unacceptable, which implies that this replaced token should be rejected if it appears during the inference phase. For data construction, we save the pairs of hidden representations and labels $(h_{z},label)$.

Our token label incorporates information about how closely the replaced token preserves the semantic information in the original response. We argue that SelfJudge is a more robust verification approach than the answer preservation-based approach, as our approach focuses on keeping the local semantics of the responses generated by the target model. Moreover, SelfJudge eliminates human bias when assessing the acceptable tokens.

Discussion on the Semantic Preservation Score.
A straightforward approach to accepting the token from the draft model (accepting the replaced token $z$) is to compare the confidence of $y$ and $z$. Some prior works (Kim et al., 2023; Narasimhan et al., 2025) have compared the probabilities solely on the prefix context, i.e., $s_{\text{prefix}}$ in Eqn.([3.3.1](https://arxiv.org/html/2510.02329v1#S3.Ex1)), to accept high-confidence draft tokens.

Our bidirectional context approach enhances the criterion by incorporating the suffix from the original response, which reduces the uncertainty of replacement decisions. We formally define our semantic preservation score, $s(y,z_{i})$, which can be decomposed to reveal its direct relationship with the prefix-only baseline. Through a Bayesian interpretation, our score can be expressed as:

$$ $s(y,z_{i})=s_{\text{prefix}}(y,z_{i})+\underbrace{\log\left(\frac{P({y}_{>i}\mid z_{i},{y}_{<i})}{P({y}_{>i}\mid y_{i},{y}_{<i})}\right)}_{\text{Log Bayes Factor}}$ (8) $$

This formulation demonstrates that our score begins with the baseline prefix score and refines it with an additional term, where the Log Bayes Factor quantifies the strength of evidence provided by the suffix. If the Log Bayes Factor is large, it indicates that the suffix is more probable given the replacement token, thus increasing the score. Conversely, if it is small, the suffix provides evidence for the original token, penalizing the replacement score. This transforms the decision-making process from a simple probability comparison into a principled, evidence-based update. Since obtaining the future token probabilities during the inference phase is not possible, we use the learned verifier to predict the semantic preservation score online.

### 3.4 Verifier Training

After collecting a dataset with semantically-preserved tokens, we can train a verifier that identifies acceptable tokens during the inference phase. We generate 69,432 token labels using 1,220 prompts in GSM8K train, LiveCodeBench and Dolly15k datasets. While the judge verifier could be any type of model, we primarily select simple linear models that utilize existing LLM hidden states as input features, following (Bachmann et al., 2025). Specifically, we employ basic logistic regression for recognizing semantic-preserved tokens in our approach. To prevent overfitting, we conduct a straightforward grid search across L2 regularization parameters.

## 4 Experiments

### 4.1 Experimental Settings

Evaluation Protocol. We comprehensively evaluate our proposed method by following the SD literature (Bachmann et al., 2025). We select Llama-3.1-8B/Llama-3.2-1B and Qwen-2.5-8B/Qwen-2.5-0.5B as the target and draft models, respectively. To measure inference efficiency, we report the average accepted length ($m$). However, relaxed verification methods can introduce the possibility of accepting incorrect tokens, which may lead to task performance degradation. Hence, we report task-specific performance metrics with the average accepted lengths. We conduct the experiments on five evaluation scenarios, including GSM8K (Cobbe et al., 2021), MATH-500 (Lightman et al., 2023), LiveCodeBench (Jain et al., 2025), CNN/DailyMail, and MMLU (Hendrycks et al., 2021). Note that we apply chain-of-thought prompting (Wang et al., 2025) while the original MMLU benchmark employs a multiple-choice prompt that encourages language models to directly answer among four choices. This modification allows us to simultaneously assess both QA task performance and inference efficiency, taking into account the trade-off between accuracy and the average accepted lengths. For more details of the experimental setup, refer to Appendix [C](https://arxiv.org/html/2510.02329v1#A3).

Baseline. For baseline comparisons, we include standard SD (Leviathan et al., 2023) with rejection sampling, which preserves the target model’s task performance. We also include top-k verification, which accepts draft tokens if they are among the target model’s top-k tokens, as a naive relaxed verification approach. Both methods represent alignment-based verification approach, as they determine acceptance based on token-level matching with the target model’s distribution. We also compare against judge verification approaches, specifically AutoJudge, which extends JudgeDecoding. We use AutoJudge instead of JudgeDecoding directly because the training data and implementation details of JudgeDecoding are not publicly available.

We train two versions of the AutoJudge verifier: AutoJudge and AutoJudge+. Specifically, AutoJudge generates token labels based on the importance of each token to the final answer, using the training split of the GSM8K and LiveCodeBench datasets. While AutoJudge can assess token importance by checking whether a token replacement preserves the correctness on math and coding problems, this approach has limited applicability to open-ended NLP tasks. To address this limitation, we implement AutoJudge+ by combining an LLMs-as-Judges framework (Li et al., 2024b) with AutoJudge. Specifically, we incorporate the instruction tuning dataset, Dolly15K, which encompasses diverse categories such as summarization, creative writing, and question answering. We train the verifier to act as a judge, asking it to determine if the original response and the token-substituted response are semantically equivalent.

Implementation Details. To set the threshold $\tau$ for semantic preservation labeling of SelfJudge, we use 100 sampled queries from GSM8K. In this math task, AutoJudge has identified ground-truth important tokens (i.e., tokens whose replacement changes the final answer). Specifically, we set $\tau=\textsf{quantile}(\textit{semantic preservation scores of unacceptable tokens determined by AutoJudge},0.1)$ to ensure that all tokens rejected by AutoJudge are also rejected by SelfJudge. This conservative threshold guarantees high recall for answer-critical tokens of our method. Based on the established threshold $\tau$, SelfJudge generates token labels for full training data. The suffix length for computing semantic preservation score is set to 20. For reproducibility, we set the temperature of the target model to 0 in all experiments. We used NVIDIA V100 GPUs for all experiments.

**Table 1: Performance comparison with SD baseline using Llama3.1-8B/Llama3.2-1B. $m$ denotes the average accepted length. Green = best performance gain compared to SD, Yellow = second place, Red = worst two. $\Delta_{m}$: Average change in accepted length from SD, $\Delta_{task}$: Average change in task performance from SD. More details on experimental settings can be seen in Appendix [C](https://arxiv.org/html/2510.02329v1#A3).**
| Method | GSM8K | MATH-500 | LiveCodeBench | MMLU | CNN/DM | Avg. Change |
| --- | --- | --- | --- | --- | --- | --- |
| $m$ / Acc. | $m$ / Acc. | $m$ / Pass@1 | $m$ / Acc. | $m$ / FC | $\Delta_{m}/\Delta_{task}$ |  |
| SD | 9.14 / 80.7 | 9.98 / 45.4 | 7.88 / 8.6 | 4.36 / 65.0 | 4.35 / 64.6 | - / - |
| Top-$k$ | 14.59 / 71.2 (-9.5) | 15.08 / 29.8 (-15.6) | 13.36 / 0.7 (-7.9) | 6.69 / 49.5 (-15.5) | 7.49 / 62.5 (-2.1) | +4.30 / -10.1 |
| AutoJudge-R | 9.71 / 80.1 (-0.6) | 11.23 / 41.0 (-4.4) | 15.78 / 0.7 (-7.9) | 4.47 / 64.6 (-0.4) | 4.36 / 64.5 (-0.1) | +1.96 / -2.7 |
| AutoJudge-F | 11.63 / 78.4 (-2.3) | 13.12 / 40.6 (-4.8) | 19.63 / 2.1 (-6.5) | 5.15 / 63.1 (-1.9) | 4.74 / 64.5 (-0.1) | +3.71 / -3.1 |
| SelfJudge-R | 10.09 / 80.7 (+0.0) | 10.91 / 43.0 (-2.4) | 8.12 / 9.7 (+1.1) | 5.14 / 64.4 (-0.6) | 4.92 / 65.5 (+0.8) | +0.69 / -0.2 |
| SelfJudge-F | 11.29 / 80.5 (-0.2) | 12.78 / 42.2 (-3.2) | 9.52 / 10.0 (+1.4) | 6.38 / 62.7 (-2.3) | 6.05 / 63.9 (-0.7) | +2.06 / -1.0 |

Figure: Figure 3: Speed/Performance comparison across different methods. We report the accuracy, with the corresponding average accepted length, by searching the threshold values of each method.
Refer to caption: x3.png

### 4.2 Performance Comparison Across NLP Tasks

We first compare SelfJudge with baseline verification methods in a practical setting where we apply judge verification in diverse NLP scenarios. As judge verification methods have a trade-off between task performance and inference efficiency depending on the threshold value $\theta$, we set a fixed threshold for the judge verifier using a holdout set in advance. {Method}-R and {Method}-F denote applying $\theta$ that achieved the best recall and best F1 score, respectively. From Table [1](https://arxiv.org/html/2510.02329v1#S4.T1), we make the following observations:

Observartion1. SelfJudge achieved significant speed-up over standard SD across various NLP tasks while demonstrating the smallest task performance degradation compared to SD. Particularly when we compare SelfJudge-F with AutoJudge-R, it shows much less performance degradation while achieving higher average accepted length across the GSM8K, MATH-500, MMLU, and CNN/DM datasets, resulting in greater inference speed improvements. Moreover, note that $\Delta_{m}$ of AutoJudge-R is inflated due to task accuracy dropping close to zero on LiveCodeBench. These suggest that our approach of accepting tokens based on semantic coherence outperforms AutoJudge’s strategy of accepting tokens based on final answer correctness.

Observation2. SelfJudge has the advantage of consistently generalizing across all tasks. Specifically, existing baselines show significant performance loss on certain tasks. Top-$k$ ($k=2$) verification achieves large speed-ups but substantially degrades task performance. AutoJudge-R partially achieves meaningful speed-up without significant performance drop compared to SD on GSM8K and MATH-500, but shows almost no speed-up on MMLU and CNN/DM. Moreover, it shows a severe decrease in Pass@1 on LiveCodeBench. In contrast, both SelfJudge-R and SelfJudge-F consistently achieve speed-ups across all datasets with minimal performance degradation among all compared methods.

We argue that these results stem from the superiority of our semantic coherence-based token acceptance strategy. Specifically, SelfJudge is trained to accept tokens if they are semantically coherent in context, regardless of the task. In contrast, AutoJudge trains the verifier to accept tokens even when semantically different tokens appear in context, as long as the final answer remains identical to the original. This fundamental difference significantly impacts the robustness when the verifier encounters out-of-distribution tokens during actual inference scenarios.

### 4.3 Trade-off Analysis: Accepted Length vs. Task Performance

Since judge verification exhibits a trade-off between task performance and inference speed depending on the threshold setting, we performed additional analyzes using the Llama-3.1-8B / Llama-3.2-1B and Qwen-2.5-7B / Qwen-2.5-0.5B models. Specifically, we varied the threshold $\theta$ of each judge verifier. From Figure [3](https://arxiv.org/html/2510.02329v1#S4.F3), we make two key observations.
First, SelfJudge shows competitive performance to AutoJudge on MMLU with CoT prompting, indicating that the token generation criteria of SelfJudge correctly labels the token compared to answer-preservation-based methods AutoJudge and AutoJudge+. These suggest that SelfJudge is consistently generalizing well in general reasoning tasks such as QA. Second, we observed that SelfJudge generally performs comparably to or better than AutoJudge methods on GSM8K. Although AutoJudge can determine the token label with the ground-truth answer for math/coding problems, our semantic-preserving approach remains competitive. This implies that determining token acceptance based on the semantic coherence in the local context is consistently better than judging token acceptance based on answer correctness, not only for general reasoning tasks but also for math and coding problems.

### 4.4 Semantic Preservation Score Analysis

When computing the semantic preservation score $s(y,z_{i})$, we can adjust the length of the suffix $y_{>i}$. In Figure [4](https://arxiv.org/html/2510.02329v1#S4.F4), we analyze the impact of the suffix length $N$ on performance. We observe that the highest accuracy is achieved when the suffix length $N=20$, while $N=0$ yields the worst performance. This implies that considering bidirectional context is beneficial for measuring token semantic preservation, resulting in improved judge decoding performance. Moreover, it indicates that suffix tokens provide stronger evidence for semantic preservation than prefix-only approaches, validating our approach.

### 4.5 Qualitative Analysis

We analyze the most frequently accepted token transformations by SelfJudge across three distinct domains: mathematical reasoning, coding, and general reasoning. First, across all task domains, we observed that SelfJudge consistently relaxes draft verification by accepting tokens, which change the format (“, $\rightarrow$
.”, “\n $\rightarrow$
\n\n”) and equivalent sentence starters (“Let $\rightarrow$ To”, “Next $\rightarrow$ Now”). Second, we observed the domain-specific transformations. For example, we find mathematical term substitutions like “solve $\rightarrow$ find” and “x $\rightarrow$ *”, and coding style variations (“:\n $\rightarrow$ :\n\n”) reflecting different programming styles with identical functionality. These patterns confirm that SelfJudge achieved great inference efficiency by accepting tokens with minor token discrepancy. Please refer to Appendix [D.1](https://arxiv.org/html/2510.02329v1#A4.SS1) for more examples.

Figure: Figure 4: Performance of SelfJudge over a range of suffix length $N$. We compute the semantic score by including the likelihood computed on $N$ future tokens.
Refer to caption: x4.png

## 5 Conclusion

In this paper, we introduce a paradigm shift for judge verification in SD, moving beyond the conventional reliance on human annotations and task-specific ground truths. We present SelfJudge, a novel framework that pioneers a self-supervised approach, leveraging the target model itself to assess semantic preservation. Our method automatically constructs a verifier training data by quantifying the contextual impact of token substitutions through a principled “semantic preservation score” derived from the model’s own likelihoods. Empirical validation across a diverse suite of NLP benchmarks confirms that SelfJudge achieves substantial inference acceleration while maintaining high fidelity to the target model, establishing our method as a practical and scalable foundation for efficient large language model inference. Our current work adopts a simple linear verifier, and a broader exploration of more complex architectures remains an important next step. Consequently, we have not yet investigated the scaling laws that relate the volume of training data to verifier performance, an understanding of which will be crucial for improving the performance of SelfJudge.

## References

- Bachmann et al. (2025)
Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas K Kohler.
Judge decoding: Faster speculative sampling requires going beyond model alignment.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Cai et al. (2024)
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao.
Medusa: Simple LLM inference acceleration framework with multiple decoding heads.
In *Forty-first International Conference on Machine Learning*, 2024.
URL [https://openreview.net/forum?id=PEpbUobfJv](https://openreview.net/forum?id=PEpbUobfJv).
- Chen et al. (2023)
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper.
Accelerating large language model decoding with speculative sampling, 2023.
URL [https://arxiv.org/abs/2302.01318](https://arxiv.org/abs/2302.01318).
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, , and John Schulman.
Measuring massive multitask language understanding, 2021.
URL [arXiv:2110.14168,2021.](arXiv:2110.14168,2021.)
- DeepMind (2025)
Google DeepMind.
Gemini2.5: Our newest gemini model with thinking., 2025.
URL [https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/](https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/).
- Devlin et al. (2019)
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
BERT: Pre-training of deep bidirectional transformers for language understanding.
In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
doi: 10.18653/v1/N19-1423.
URL [https://aclanthology.org/N19-1423/](https://aclanthology.org/N19-1423/).
- Fu et al. (2024)
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang.
Break the sequential dependency of llm inference using lookahead decoding, 2024.
URL [https://arxiv.org/abs/2402.02057](https://arxiv.org/abs/2402.02057).
- Garipov et al. (2025)
Roman Garipov, Fedor Velikonivtsev, Ruslan Svirschevski, Vage Egiazarian, and Max Ryabinin.
Autojudge: Judge decoding without manual annotation, 2025.
URL [https://arxiv.org/abs/2504.20039](https://arxiv.org/abs/2504.20039).
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang,
Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur,
Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher,
Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh
Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna
Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim
Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan,
Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi
Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma.
The llama 3 herd of models, 2024.
URL [https://arxiv.org/abs/2407.21783](https://arxiv.org/abs/2407.21783).
- He et al. (2024)
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He.
REST: Retrieval-based speculative decoding.
In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pp. 1582–1595, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.naacl-long.88.
URL [https://aclanthology.org/2024.naacl-long.88/](https://aclanthology.org/2024.naacl-long.88/).
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding, 2021.
- Hu et al. (2025)
Zhengmian Hu, Tong Zheng, Vignesh Viswanathan, Ziyi Chen, Ryan A. Rossi, Yihan Wu, Dinesh Manocha, and Heng Huang.
Towards optimal multi-draft speculative decoding.
In *ICLR*, 2025.
- Jain et al. (2025)
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.
Livecodebench: Holistic and contamination free evaluation of large language models for code.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Kaplan et al. (2020)
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
Scaling laws for neural language models, 2020.
URL [https://arxiv.org/abs/2001.08361](https://arxiv.org/abs/2001.08361).
- Kim et al. (2023)
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer.
Speculative decoding with big little decoder.
In *Proceedings of the 37th International Conference on Neural Information Processing Systems*, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc.
- Laban et al. (2022)
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst.
SummaC: Re-visiting NLI-based models for inconsistency detection in summarization.
*Transactions of the Association for Computational Linguistics*, 10:163–177, 2022.
doi: 10.1162/tacl˙a˙00453.
URL [https://aclanthology.org/2022.tacl-1.10/](https://aclanthology.org/2022.tacl-1.10/).
- Leviathan et al. (2023)
Yaniv Leviathan, Matan Kalman, and Yossi Matias.
Fast inference from transformers via speculative decoding.
In *Proceedings of the 40th International Conference on Machine Learning*, ICML’23. JMLR.org, 2023.
- Li et al. (2024a)
Li, Yuhui, Wei, Fangyun, Zhang, Chao, Zhang, and Hongyang.
Eagle: speculative sampling requires rethinking feature uncertainty.
In *International Conference on Macihne Learning*, ICML’24. JMLR.org, 2024a.
- Li et al. (2024b)
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu.
Llms-as-judges: A comprehensive survey on llm-based evaluation methods, 2024b.
URL [https://arxiv.org/abs/2412.05579](https://arxiv.org/abs/2412.05579).
- Li et al. (2025)
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang.
Eagle-3: Scaling up inference acceleration of large language models via training-time test, 2025.
URL [https://arxiv.org/abs/2503.01840](https://arxiv.org/abs/2503.01840).
- Lightman et al. (2023)
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step, 2023.
URL [https://arxiv.org/abs/2305.20050](https://arxiv.org/abs/2305.20050).
- Liu et al. (2024)
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang.
Online speculative decoding.
In *Proceedings of the 41st International Conference on Machine Learning*, ICML’24. JMLR.org, 2024.
- Luo et al. (2025)
Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, and Dongliang Xu.
Turning trash into treasure: Accelerating inference of large language models with token recycling, 2025.
URL [https://arxiv.org/abs/2408.08696](https://arxiv.org/abs/2408.08696).
- Miao et al. (2024)
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia.
Specinfer: Accelerating large language model serving with tree-based speculative inference and verification.
In *Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3*, ASPLOS ’24, pp. 932–949. ACM, April 2024.
doi: 10.1145/3620666.3651335.
URL [http://dx.doi.org/10.1145/3620666.3651335](http://dx.doi.org/10.1145/3620666.3651335).
- Nallapati et al. (2016)
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gu$\dot{}$lçehre, and Bing Xiang.
Abstractive text summarization using sequence-to-sequence RNNs and beyond.
In *Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning*. Association for Computational Linguistics, 2016.
doi: 10.18653/v1/K16-1028.
- Narasimhan et al. (2025)
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar.
Faster cascades via speculative decoding.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=vo9t20wsmd](https://openreview.net/forum?id=vo9t20wsmd).
- Oliaro et al. (2025)
Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao.
Suffixdecoding: Extreme speculative decoding for emerging ai applications, 2025.
URL [https://arxiv.org/abs/2411.04975](https://arxiv.org/abs/2411.04975).
- OpenAI et al. (2024)
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo
Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan
Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez,
Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia
Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.
Gpt-4 technical report, 2024.
- Patil et al. (2024)
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez.
Gorilla: Large language model connected with massive APIs.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=tBRNC6YemY](https://openreview.net/forum?id=tBRNC6YemY).
- Radford et al. (2019)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners.
*OpenAI*, 2019.
- Stern et al. (2018)
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit.
Blockwise parallel decoding for deep autoregressive models.
In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 31. Curran Associates, Inc., 2018.
URL [https://proceedings.neurips.cc/paper_files/paper/2018/file/c4127b9194fe8562c64dc0f5bf2c93bc-Paper.pdf](https://proceedings.neurips.cc/paper_files/paper/2018/file/c4127b9194fe8562c64dc0f5bf2c93bc-Paper.pdf).
- Sun et al. (2024)
Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen.
Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding.
In *First Conference on Language Modeling*, 2024.
URL [https://openreview.net/forum?id=HVK6nl3i97](https://openreview.net/forum?id=HVK6nl3i97).
- Sun et al. (2023)
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu.
Spectr: fast speculative decoding via optimal transport.
In *Proceedings of the 37th International Conference on Neural Information Processing Systems*, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc.
- Wang et al. (2025)
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen.
Mmlu-pro: a more robust and challenging multi-task language understanding benchmark.
In *Proceedings of the 38th International Conference on Neural Information Processing Systems*, NIPS ’24, Red Hook, NY, USA, 2025. Curran Associates Inc.
ISBN 9798331314385.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.
Chain-of-thought prompting elicits reasoning in large language models.
In *Proceedings of the 36th International Conference on Neural Information Processing Systems*, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc.
ISBN 9781713871088.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu.
Qwen3 technical report, 2025.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Zhou et al. (2024)
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal.
Distillspec: Improving speculative decoding via knowledge distillation.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=rsY6J3ZaTF](https://openreview.net/forum?id=rsY6J3ZaTF).

## Appendix A Proof of Uncertainty Reduction Using Bidirectional Context

In this section, we provide a formal, information-theoretic proof that incorporating bidirectional context strictly reduces the uncertainty of selecting acceptable tokens.

Theorem 1. When the suffix $\mathbf{x}_{>t}$ provides additional information about a token $X_{t}$ given the prefix $\mathbf{x}_{<t}$ (i.e., the conditional mutual information $I(X_{t};\mathbf{x}_{>t}\mid\mathbf{x}_{<t})>0$), the uncertainty about $X_{t}$, measured by conditional entropy, is strictly reduced when using the bidirectional context $\mathbf{c}_{t}=(\mathbf{x}_{<t},\mathbf{x}_{>t})$.

$$ $H(X_{t}\mid\mathbf{c}_{t})<H(X_{t}\mid\mathbf{x}_{<t})$ (9) $$

Proof. The proof follows directly from the definition of conditional mutual information.

- 1.
The definition of conditional mutual information between two random variables $A$ and $B$ given a third variable $C$ is:
$I(A;B\mid C)=H(A\mid C)-H(A\mid B,C)$
(10)
- 2.
We substitute our variables: $A=X_{t}$, $B=\mathbf{x}_{>t}$, and $C=\mathbf{x}_{<t}$.
$I(X_{t};\mathbf{x}_{>t}\mid\mathbf{x}_{<t})=H(X_{t}\mid\mathbf{x}_{<t})-H(X_{t}\mid\mathbf{x}_{>t},\mathbf{x}_{<t})$
(11)
- 3.
By our definition, the bidirectional context is $\mathbf{c}_{t}=(\mathbf{x}_{<t},\mathbf{x}_{>t})$. Therefore, the final term is equivalent to the entropy conditioned on the bidirectional context:
$H(X_{t}\mid\mathbf{x}_{>t},\mathbf{x}_{<t})=H(X_{t}\mid\mathbf{c}_{t})$
(12)
- 4.
The premise of our theorem states that $I(X_{t};\mathbf{x}_{>t}\mid\mathbf{x}_{<t})>0$. Substituting this into the equation from step 2 gives:
$H(X_{t}\mid\mathbf{x}_{<t})-H(X_{t}\mid\mathbf{c}_{t})>0$
(13)
- 5.
Rearranging the terms yields the final result:
$H(X_{t}\mid\mathbf{c}_{t})<H(X_{t}\mid\mathbf{x}_{<t})$
(14)

This concludes the proof, demonstrating that conditioning on bidirectional context logically leads to lower uncertainty.

### A.1 Discussion on the Premise

It is crucial to understand the role of the premise, $I(X_{t};\mathbf{x}_{>t}\mid\mathbf{x}_{<t})>0$. From a Natural Language Processing (NLP), this premise is not merely an assumption but a well-established empirical fact. Language is inherently structured, and for any non-trivial linguistic context, the suffix almost always contains information that helps resolve ambiguity or refine the prediction of a token (Devlin et al., 2019). Therefore, while our proof relies on a mathematical assumption, its real-world applicability is grounded in the fundamental nature of human language.

**Table 2: Data for training verifier of each judge decoding methods.**
| Target Model | Method | Dataset | Num. of samples | Label Rate |
| --- | --- | --- | --- | --- |
| Llama-3.1-8B | AutoJudge | GSM8K (train), LiveCodeBench (train) | 14896 | 13633 |
| AutoJudge+ | GSM8K (train), LiveCodeBench (train), Dolly15k | 39069 | 37261 |  |
| SelfJudge | GSM8K (train), LiveCodeBench (train), Dolly15k | 69432 | 42253 |  |
| Qwen-2.5-7B | AutoJudge | GSM8K (train), LiveCodeBench (train) | 24748 | 13068 |
| AutoJudge+ | GSM8K (train), LiveCodeBench (train), Dolly15k | 55757 | 44067 |  |
| SelfJudge | GSM8K (train), LiveCodeBench (train), Dolly15k | 95128 | 58482 |  |

Figure: Figure 5: ROC curve of our judge verifier. We train each verifier for Llama-3.1-8B/Llama-3.2-1B and Qwen-2.5-7B/Qwen-2.5-0.5-B, individually. Red point represents threshold value with the best F1 score on validation set.
Refer to caption: x5.png

## Appendix B Verifier Training

### B.1 Statistics

Table [2](https://arxiv.org/html/2510.02329v1#A1.T2) shows the training data of verifiers across judge decoding methods. Following previous works, AutoJudge uses math and coding datasets to train the judge verifier. It assigns a token label as acceptable if the newly generated response after replacing a token preserves the original final answer in math and coding problems. To further utilize general datasets like Dolly15k, we implement AutoJudge+ by generating token labels using LLMs-as-Judge approaches. Specifically, LLMs decide whether the original response and the alternative response are semantically [SAME] or [DIFFERENT]. We assign a token label as acceptable for tokens that results in [SAME].

### B.2 Training Logistic Regressor

We employed a logistic regression classifier to address the binary classification problem, which determines acceptable and unacceptable token. We optimized hyperparameters through grid search over the $L_{2}$ regularization parameter $C$ . We explored $C$ values from $0.001$ to $100$ across 10 logarithmically spaced points. Finally, we selected the verifier with the best ROC-AUC.

During inference phase, we apply the trained verifier our two-phase verification method. For the judge verification phase, we select the threshold $\theta$ to decide the acceptable tokens. We set the $\theta$ to value of the best recall and the best F1 as shown in Figure [5](https://arxiv.org/html/2510.02329v1#A1.F5).

### B.3 Dataset Generation Time

AutoJudge generates new responses by replacing each mismatched token and completing the text, which is computationally expensive. This requires generating responses $\textit{Num. of Queries}\times\textit{Num. of Mismatched Tokens}$ times. In contrast, our approach simply computes likelihood differences from the original response without generating new text. We only need to partially prefill a few tokens to calculate these likelihoods. As *partially prefilling* requires much less computational cost compared to *generation*, our approach is significantly efficient, making our method applicable to large models. Using Llama-3.1-8B on NVIDIA V100 GPUs, our SelfJudge method processed 69,432 data points in 8 hours, while AutoJudge processed only 14,896 data points in 82 hours.

## Appendix C Experimental Configuration

For mathematical reasoning evaluation, we utilize GSM8K (Cobbe et al., 2021) and MATH-500 (Lightman et al., 2023) using the standard train/test split and evaluation prompts in the benchmark. For code generation evaluation, we leverage LiveCodeBench (Jain et al., 2025). Since this benchmark does not provide predefined train/test splits, we implement a 4-fold cross-validation strategy. For the summarization task, we employ the CNN/DailyMail datasets (Nallapati et al., 2016) using the conventional test split. For general knowledge assessment, we evaluate on MMLU (Hendrycks et al., 2021), a multiple-choice QA benchmark. While the original MMLU benchmark employs a multiple-choice prompt that encourages language models to directly answer among four choices, we apply chain-of-thought prompting (Wang et al., 2025). This modification allows us to simultaneously assess both QA task performance and inference efficiency, taking into account the trade-off between accuracy and the average accepted lengths.

### C.1 Math Reasoning Benchmarks

GSM8K (Cobbe et al., 2021) is a dataset of high-quality, diverse grade-school math word problems. We used 1,319 test problems for our evaluation. The MATH-500 dataset contains a subset of 500 problems from the MATH benchmark, which was created by (Lightman et al., 2023). For our evaluation, we used a prompt that required the answer format to appear between [answer] and [\answer]. After extracting the answers, we normalized the format by removing uppercase letters and certain operators. We finally check the correctness of the answer with ground truths to measure accuracy.

### C.2 Coding Benchmark

LiveCodeBench (Jain et al., 2025) is a continuously updating coding benchmark for evaluating Large Language Models. It evaluates LLMs across multiple capabilities including code generation, self-repair, test output prediction, and code execution. We used release version 0.0.5, which contains 880 programming problems, and applied 4-fold cross-validation to create training and test datasets. We evaluated performance using the Pass@1 metric, which measures the percentage of problems successfully solved on the first attempt when evaluated against comprehensive test suites.

**Table 3: Summarization Task Performance Comparison on Llama3.1-8B/Llama-3.2-0.5B across various SD methods. $m$ is the averaged accepted length.**
| Method | $m$ | ROUGE1 | ROUGE2 | ROUGE-L | FC |
| --- | --- | --- | --- | --- | --- |
| SD | 4.35 | 27.1 | 10.8 | 22.4 | 64.6 |
| TopK | 7.49 | 26.8 | 10.5 | 22.0 | 62.5 |
| AutoJudge-R | 4.36 | 27.0 | 10.8 | 22.4 | 64.5 |
| AutoJudge-F | 4.74 | 27.1 | 10.9 | 22.5 | 64.5 |
| SelfJudge-R | 4.92 | 27.5 | 11.0 | 22.8 | 65.5 |
| SelfJudge-F | 6.05 | 26.2 | 10.3 | 21.5 | 63.9 |

### C.3 Summarization Benchmark

The CNN/DailyMail dataset (Nallapati et al., 2016) is an English-language dataset containing over 300,000 unique news articles written by journalists at CNN and the Daily Mail. We evaluated summarization quality on 11,490 test samples from version 2.0.0, which supports both extractive and abstractive summarization.
Since our evaluation focuses on relaxed speculative decoding methods that maintain token-level equivalence with standard decoding, traditional metrics like ROUGE cannot effectively differentiate between methods—all produce identical outputs in terms of content. Therefore, we instead report factual consistency scores to assess summarization quality (Laban et al., 2022).
We measure factual consistency using a Natural Language Inference (NLI) model to classify the relationship between source articles and generated summaries. Specifically, we identify sentences that are either entailed by or contradict the source material. The factual consistency score (FC) is computed as:

$$ $\text{Factual Consistency}=\frac{1}{M}\sum_{i=1}^{M}(\text{Entailments}_{i}-\text{Contradictions}_{i})$ (15) $$

where $M$ is the number of summary sentences evaluated. We employed the deberta-large-mnli model for NLI classification due to its strong performance on factual verification tasks. In Table [3](https://arxiv.org/html/2510.02329v1#A3.T3), we can observe the result of summarization task in more details. As we mentioned, ROUGE scores, which depend on token-level alignment, do not show distinctive difference across methods. Hence, we selected the FC score as our summarization task metric, as it reflect more semantic information by considering entailment of fact when evaluating results.

### C.4 Multiple Choices Benchmark

MMLU (Hendrycks et al., 2021) is a massive multitask benchmark containing 14,042 multiple-choice questions across 57 tasks. The benchmark spans diverse domains including humanities, social sciences, hard sciences, and other fields, covering subjects from elementary mathematics to US history, computer science, and law. While the original MMLU evaluation prompts models to directly select answers from ’A’, ’B’, ’C’, or ’D’, we adopted the chain-of-thought prompting approach from (Wang et al., 2025) to measure both inference efficiency and task performance simultaneously.

## Appendix D Additional Experiments

### D.1 Qualitative Analysis

Figure: Table 4: Freuquently appeared token acceptance by SelfJudge on Math Problem (GSM8K)

Tables [4](https://arxiv.org/html/2510.02329v1#A4.T4), [5](https://arxiv.org/html/2510.02329v1#A4.T5), and [6](https://arxiv.org/html/2510.02329v1#A4.T6) present the most frequently accepted tokens by SelfJudge while they are rejected by Standard SD across three distinct domains: mathematical reasoning (GSM8K), coding (LiveCodeBench), and general reasoning (MMLU). These results reveal several key insights about the flexible nature of our approach and domain-specific characteristics of semantically preserving token modifications that do not compromise response quality.
Domain-Agnostic Strengths. Across all three tasks, SelfJudge demonstrates consistent acceptance of tokens that preserve semantic meaning and logical equivalence. Mathematical terms like “solve $\rightarrow$ find” and calculate $\rightarrow$ find” in GSM8K, along with synonymous replacements such as
number $\rightarrow$ element” in coding tasks, highlight the model’s ability to identify semantically equivalent tokens that would not alter the fundamental meaning of the response. Moreover, SelfJudge successfully allows minor formatting discrepancies such as “, $\rightarrow$
.” and “\n $\rightarrow$
\n \n”. This flexibility enables speculative decoding to achieve significant latency reduction while preserving the semantic integrity and quality of the original target model response.

Mathematical Reasoning Characteristics. In GSM8K, SelfJudge exhibits a particular pattern of accepting mathematical vocabulary variations that preserve computational meaning. The frequent acceptance of transformations like x $\rightarrow$ *” and find $\rightarrow$
calculate” indicates that SelfJudge correctly identifies when mathematical operations remain functionally equivalent. Additionally, the acceptance of sentence starters like
“Let $\rightarrow$ To” and “Next $\rightarrow$
Now” demonstrates that these discourse markers do not fundamentally alter the mathematical reasoning process, allowing for flexible problem-solving exposition styles without compromising the logical flow.

Coding Domain Specificity. LiveCodeBench results reveal how SelfJudge achieves high acceptance rates while maintaining code functionality. SelfJudge correctly accepts functionally equivalent code transformations, such as whitespace normalization (“\n $\rightarrow$ \n\n”) and formatting changes that do not affect program execution. Notably, synonym acceptance for programming concepts (“max $\rightarrow$
maximum”, “count $\rightarrow$ number”) shows the model’s ability to recognize coding-specific terminology variations that preserve semantics. The acceptance of structural changes in brackets and punctuation effectively improve the efficiency of judge verification while preserving the target model’s intention.

General Reasoning Adaptability. Table 5 exhibits frequent acceptance of transformations involving punctuation and formatting variations, indicating that such changes do not alter the semantic content or factual accuracy of responses. Particularly noteworthy is the acceptance of articles/determiners changes (“a $\rightarrow$ the”, “the $\rightarrow$ a”) and logical term modifications (
“validity $\rightarrow$ correct”), which suggests SelfJudge can distinguish between semantically equivalent expressions that convey identical meaning. We argue that minor linguistic variations with these token changes do not compromise the informational content or reasoning quality of the original response.

### D.2 Hyperparameter Sensitivity Analysis

Prediction Threshold Analysis. To comprehensively analyze the relationship between inference efficiency and task performance across varying threshold values, we conduct a systematic investigation into the threshold sensitivity of each method with respect to parameter $\theta$. In Figure [3](https://arxiv.org/html/2510.02329v1#S4.F3), our experimental results demonstrate that SelfJudge consistently achieves superior performance compared to all baseline methods across diverse thresholds. This robust performance advantage indicates that SelfJudge represents a fundamentally superior approach, delivering optimal trade-offs between accepted sequence lengths and task-specific performance metrics while maintaining stability across different threshold settings. It implies that our method’s effectiveness is not contingent upon fine-tuned hyperparameter selection, thereby enhancing its practical applicability and robustness in real-world deployment scenarios.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - All experiments use 7–8B-scale targets (Llama-3.1-8B, Qwen-2.5-7B), with the argument that data generation is already much more efficient than AutoJudge. It would be helpful to at least discuss: How expensive would SelfJudge data generation be for a 70B model? Whether the cost remains acceptable if you want many more labels (e.g., to train a higher-capacity verifier).

- Could you add a short, concrete example in the main text showing how the semantic preservation score is computed for one token replacement, and explicitly highlight which part corresponds to the prefix vs suffix?
soundness: 3
strengths: - Using the target model itself to score token substitutions via a bidirectional likelihood difference is reasonable and avoids human annotation or task-specific ground truth. 

- Using the bidirectional context to score and verify the generated labels.
summary: The paper proposes SelfJudge, a verifier for speculative decoding (SD) that relaxes strict token alignment between draft and target models. Instead of requiring that the draft token exactly match the target token, SelfJudge accepts draft tokens that are semantically compatible with the target model’s own generations. This differs from some prior work, which relies on human annotations and accordingly performs well per task. The model (verifier) relies on data generated from the target model and is trained to preserve semantic compatibility.
weaknesses: - The bidirectional context is not easily understandable. The core scoring formula is non-trivial, and the paper jumps rather quickly from the conceptual definition. It would be helpful to include a concrete toy example in the main text, illustrating how the suffix likelihood behaves.
Also, it should be stated more explicitly that “bidirectional” is purely a training-time construction.

- The two-stage verification logic is underspecified / slightly inconsistent. Section 3.2 defines verification as “Accept (d_t)​ if Verifier(​h_t) > θ, otherwise reject”, but then later states that tokens rejected by the verifier are passed to alignment-based verification.

- There is no ablation on two-stage verification. It would be useful to see: (1) Judge-only verification (no fallback to alignment-based SD), (2) SelfJudge + fallback, as proposed,  to quantify how much the fallback contributes to quality and speed.

- Scaling to larger target models is not demonstrated.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: N/A
soundness: 2
strengths: The paper is clearly written and well-structured.

The integration of self-supervision into speculative decoding is well-motivated.

The proposed semantic preservation criterion is simple and implementation-friendly, requiring no human labels or task-specific supervision.

Experimental evaluation covers multiple domains and demonstrates general applicability across reasoning, coding, and summarization tasks.
summary: The paper proposes SelfJudge, a self-supervised framework for speculative decoding (SD).
Instead of relying on human-annotated correctness signals or task-specific ground truths (as in AutoJudge), SelfJudge trains a lightweight verifier using semantic preservation scores computed from the target model itself. The idea is to estimate whether token substitutions preserve meaning, and to accept draft tokens based on semantic coherence rather than strict probability alignment. Experiments on Llama-3 and Qwen models across GSM8K, MATH-500, MMLU, CNN/DailyMail, and LiveCodeBench show modest speed-accuracy trade-offs compared with previous judge-based decoding methods.
weaknesses: The main idea of leveraging a model’s own likelihood distribution to verify semantic coherence is conceptually very close to existing self-consistency or self-verification paradigms.
In essence, SelfJudge performs a token-level version of self-consistency decoding integrated into speculative decoding.
While the framing as “semantic preservation” is neat, the underlying mechanism does not appear fundamentally new or theoretically distinct from prior work on self-consistency, distribution alignment, or self-reflective verification.

The reported improvement is small and possibly within experimental noise.
Compared with AutoJudge, SelfJudge achieves only +0.1 higher accepted token length and slightly lower accuracy drop (–1.0% vs –2.7%).
No wall-clock runtime or throughput results are provided, making it difficult to confirm real acceleration.
Overall, the empirical evidence does not convincingly demonstrate that the proposed semantic verifier yields a substantial or practical speedup.

The paper also lacks analysis of how the self-supervised signal differs from conventional probability alignment beyond intuitive discussion.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1.	The accuracy seems to degrade faster on Qwen-2.5 compared to Llama-3.1. What explains this discrepancy? Does it indicate that the method is model sensitive?
2.	In Figure 4, the accuracy trends differ between GSM8K and MMLU for varying suffix lengths. What factors contribute to this difference?
3.	How are hyperparameters determined? Are they consistent across tasks and models, or do they require careful tuning for each scenario and model?
soundness: 3
strengths: The self-judging mechanism shows potential for better generalization and significantly reduces annotation effort.
summary: This paper proposes a model-based speculative decoding verifier training data generation method using the target model as self-judger, which can generalize across diverse tasks.
weaknesses: 1.	The paper states that its motivation is to avoid unnecessary fallbacks and improve decoding speed. However, the proposed model-based judger introduces additional verification time, and the paper does not provide evidence of end-to-end speedup.
2.	The method appears to address only token-level semantic matching. This limits its effectiveness, as phrase-level and sentence-level semantic matching remain unresolved.
3.	Experiments are conducted on only two relatively small models. Evaluating on larger models would strengthen claims of generalization.


Decision: Reject