AutoJudge: Judge Decoding Without Manual Annotation
Authors: Roman Garipov
Fedor Velikonivtsev
Ivan Ermakov
Ruslan Svirschevski
Vage Egiazarian
Max Ryabinin
Venue: NeurIPS 2025 poster ()
Keywords: LLM
Speculative Decoding
Lossy Speculative Decoding
Inference
Primary area: deep_learning
TL;DR: Automatically detecting task-specific important tokens to accelerate speculative decoding

Abstract: We introduce AutoJudge, a method that accelerates large language model (LLM) inference with task-specific lossy speculative decoding. 
Instead of matching the original model output distribution token-by-token, we identify the generated tokens that affect the downstream quality of the response, relaxing the distribution match guarantee so that the "unimportant" tokens can be generated faster.
Our approach relies on a semi‑greedy search algorithm to test which of the mismatches between target and draft models should be corrected to preserve quality and which ones may be skipped.
We then train a lightweight classifier based on existing LLM embeddings to predict, at inference time, which mismatching tokens can be safely accepted without compromising the final answer quality.
We evaluate AutoJudge with multiple draft/target model pairs on mathematical reasoning and programming benchmarks, achieving significant speedups at the cost of a minor accuracy reduction. 
Notably, on GSM8K with the Llama 3.1 70B target model, our approach achieves up to ${\approx}2{\times}$ speedup \textit{over speculative decoding} at the cost of a ${\le} 1\%$ drop in accuracy.
When applied to the LiveCodeBench benchmark, AutoJudge automatically detects programming-specific important tokens, accepting ${\ge}25$ tokens per speculation cycle at a$~ 2\%$ drop in Pass@1. Our approach requires no human annotation and is easy to integrate with modern LLM inference frameworks.

AutoJudge: Judge Decoding Without Manual Annotation
Roman Garipoy + HSE University, Yandex
Fedor Velikonivtsey HSE University, Yandex
Ivan Ermakov HSE University, Yandex
Ruslan Svirschevski Yandex
Vage Egiazarian" $ IST Austria
Max Ryabinin Together Al
Abstract
We introduce AutoJudge"] method that accelerates large language model (LLM) inference with task-specific lossy speculative decoding. Instead of matching the original model output distribution token-by-token, we identify the generated tokens that affect the downstream quality of the response, relaxing the distribution match guarantee so that the "unimportant" tokens can be generated faster. Our approach relies on semi-greedy search algorithm to test which of the mismatches between target and draft models should be corrected to preserve quality and which ones may be skipped. We then train a lightweight classifier based on existing LLM embeddings predict, at inference time, which mismatching tokens can be safely accepted without compromising the final answer quality. We evaluate AutoJudge with multiple draft/target model pairs mathematical reasoning and program- ming benchmarks, achieving significant speedups at the cost minor accuracy reduction. Notably, GSM8K with the Llama 3.1 70B target model, approach achieves up to =2x speedup over speculative decoding at the cost of a < <1% drop in accuracy. When applied to the LiveCodeBench benchmark, AutoJudge auto- important tokens, accepting 225 tokens
matically detects poggramming-speciinn per speculation cycle 2% drop in Pass@1 Our approach requires human annotation and is easy to integrate with modern LLM inference frameworks.
Introduction
Recent advances LLM capabilities, including chain-of-thougleee reasoning [Wii al., 2022 Kojima et al. 2022 Suzgun et al. 2022], writing complex software Rozière et al.] [2023 Li et 2003 Jiang
ct al. 2024 or interacting with external tools [Schick et al. [2023] Qin et al., 2023), increasingly rely on inference-time computation (Snell et al. 2024 Beeching et al. 2024 This progress is further accelerated with the release reasoning-capable models, both proprietary [OpenAl et al. 2024
Anthropic 2024 Google DeepMind] 2025) and open-access (DeepSeck-A) et al. 2025 Meta
2025
Qwen Team 2025 that were explicitly trained to perform these kinds of inference- time computation.
However, LLMs tackle harder problems, they also tend to generate longer sequences [Muennighofii
ct al. 2025 with tens of thousands of tokens [Yeo et al., 2025]. taking up tens of minutes (and
hundreds of dollars in costs) per task (RC Prize Foundation 2024
popular way speed up LLM inference is through speculative decoding [Leviathan al.) 2023 Chen et al. 2023). which uses a small "draft" model to propose the likely next tokens, then verifies these tokens with the main model in parallel. This method, along with its successors [Miao al. 2023
Our code is available at atggithub... com/ garr gaw..ppooroma/auuouuuee. 'Equal contribution. 'Corresponding author; pevilgardgmail.cm Work done during employment Yandex.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Cai et al. 2024 Li et al. 2024d]. can speed up LLM inference while guaranteeing that the generated outputs match those of the original model (for greedy inference) or follow the same distribution. To achieve this, speculative decoding algorithms check if the draft tokens match the original model predictions. If there is mismatch, they discard the incorrect token and all subsequent ones.
Speculative decoding can accelerate reasoning and other test-tima computations, but it can be overly strict in how discards tokens [Bachmann al., 2025 Pan et al. 2025 Tran-Thien 2024]. Intuitively, if a model generates a reasoning chain, not all mismatching tokens are equally important: errors in derivation should be fixed, while minor word choices should not. Judge Decoding (Bachmann et al., 2025] takes advantage of this by labeling which tokens are important for reasoning (and which are not) and allowing speculative decoding to accept more tokens by skipping the unimportant ones. However, their approach relies on human annotators to determine which tokens are important for reasoning. This complicates adoption and can be prone to human errors, particularly if the task requires expert knowledge (e.g., complex mathematical proofs or software engineering)
In this work, we look for ways to streamline this process. Instead of relying on human annotators, we propose AutoJudge, a search- based algorithm that detects which tokens are important for the task at hand based on how they affect the final answer. The algorithm based on the idea that token is deemed "important" not by itself, but in combination with other generated tokens. Thus, we propose a procedure that selects a small subset of important mismatching tokens that affect the final answer. Using this procedure, we can automatically mine a dataset to train an important token classifier that can then be used to accelerate speculative decoding.
The proposed search algorithm finds a small set of task-specific contextual "important tokens" - cases where target and draft models disagree on the next token in way that affects the final response quality. We then train a classifier to detect these important tokens and use it to improve traditional speculative decoding by relaxing its verification procedure.
Our experiments with Llama 3.x models demonstrate that the proposed approach can indeed identify important tokens and save time speculation, accepting average over 40 tokens per target model forward pass (approx. 2x that of speculative decoding), at the cost of a 51% drop in accuracy on GSM8K [Cobbe et al., 2021] and even more with minor accuracy drawdown. We obtain similar results with the Qwen2.51 Yang et al. 2024] family of models, observing comparable number of ac- cepted tokens and accuracy trade-offs. When applied to programming tasks BiveCodeBench (Jain) (et [ [024], our approach is able to determine different task-specific important tokens, showing similar performance gains. The proposed framework is simple and general, using classifier only when the original algorithm would reject a token, making it compatible with arbitrary speculative decoding algorithms. The main contributions of our work can be summarized follows:
We formulate AutoJudge, an algorithm for detecting which of the tokens generated in speculative decoding affect the downstream accuracy for a given task. Our algorithm requires no human annotation and can be applied to most popular LLM tasks.
We verify the efficacy of AutoJudge on mathematical reasoning and programming benchmarks for several speculative decoding setups (model pairs). Our evaluations demonstrate favorable tradeoffs between the accuracy and the inference speedup, generating 20-45 tokens per speculative decoding cycle at the cost of a slight accuracy drawdown.
We integrate AutoJudge with the vLLM framework [woon et al. 2023 and report the inference speed on A100 GPUs for both 8B and 70B target models, with up to 2 speedup over speculative decoding at a 51% quality decrease, and on H100 GPUs with 405B target model.
2
Background
Speculative Decoding, Our work builds on top of speculative decoding (Stern et al. 2018 Leviathan
ct al. 2023 Chen et al., 2023). a family of inference algorithms that accelerate token generation by improving hardware utilization. Speculative Decoding uses an auxiliary "draft" model to generate K>1 possible future tokens, then runs the main "target" model in parallel to verify the generated tokens. The drafted tokens that agree with the target model predictions are accepted by the algorithm. In turn the first mismatching token and all subsequent ones are rejected. This way, the method
For greedy decoding, it checks that the drafted tokens are the same the target model's own next token predictions. For sampling, it uses procedure that matches the sampling probabilities (Leviathan al. 2023

---

DATA GATHERING
INFERENCE
target (parallel)
target...
the result pecomes
12
34 56 102
the derivative becomes didx xfx x (1
Inx)
mismeich
mismatch
mbmmtcc
drafl:
34 56 34
draft (sequential)
draft:
equals 12 34 + 102
the derivative
equalls
didx x xhx
draft
50
the final sum of the (12, 56) 102
classifier
extra accepted tokens
discard
Figure I: Intuitive scheme of the proposed approach: (left) data collection: detecting mismatching tokens that affect final response quality; these tokens are then used train classifier (right) using the trained classifier to generate more tokens per cycle with speculative decoding.
guarantees that all generated tokens follow the same distribution sampling from the target model.
Subsequent works improve on this idea by generating draft trees instead of single sequences [Mao
al. 2023] Liu al.] 2023] Chen al. [2024] [Svirchhvvski et al.)
2024].
training specialized "heads"
to draft next tokens based on the model's 's hidden states [Cai et al.
2024
Ankner et al. 2024 Li et al.
2024d and more (Fu et al., 2023 Spector and Re
2023
Sun et al.
2023 et al.
2023
Lossy Speculative Decoding. The core guarantee of Speculative Decoding is that all generated
tokens follow the probability distribution of the original model. However, there are practical scenarios where this guarantee can be sacrificed in favor of faster inference, which is known as lossy speculative
decoding algorithms TrianTThie| 2024 Narasimhan et al., [2025 Kim al. 2023]. Our work extends
one such method: Judge Decoding [Bachmann et al. 2025]. The core idea of Judge Decoding is that speculative decoding should only reject the mismatching token if accepting it would harm the response quality. For instance, in mathematical reasoning, errors in the equations or logical fallacies important for the final quality, while minor style changes not. When writing code, algorithmic errors are important, while minor variable renames can be skipped in favor of faster inference.
The main challenge of Judge Decoding is determining which of the generated tokens can be skipped this way. Bachmann et al. 2025) address this problem by manually labeling a training dataset for the classifier. Judge Decoding requires human annotators to find the "mistake" - the first mismatching token that led the draft model to diverge from the original answer. The resulting dataset of high- quality training examples is then used to train a linear classifier that detects such "mistakes" during inference. Authors demonstrate that the collected dataset can, in principle, be reused across tasks and models. However, using the data from task for inference on different task results in substantial performance drawdown. Intuitively, different tasks (such as creative writing, math, or programming) have different criteria for which parts of the generated response matter most. Hence, it best train the important token classifier for the exact lask at hand. However, doing so with Judge Decoding would require relabeling the data by human annotators, which can be costly and time-consuming in specialized domains such as medicine or law. To alleviate this problem, we develop an automated search procedure for determining important tokens without external human (or LLM) annotators.
3 Method Overview
Our approach consists of three important stages. First, we detect which of the mismatching tokens affect the model quality using a semi-greedy search algorithm that we describe Section 3.1 We then use the gathered data to train a lightweight classifier that can detect important tokens at inference time (Section 3.2). Finally, we use the trained classifier augment a speculative decoding algorithm as described in Section 3.3
so that it can generate more tokens per speculation-verificattoo cycle.
3.1 Mining Important Tokens
In this section, we describe an algorithm to identify which draft tokens that mismatch with the target ones influence the final output quality. To achieve this, we systematically alter the generation output, swapping between draft and main model tokens and test how this affects the downstream task output, such as the final answer to a math problem or test outputs for a programming task. If replacing
3

---

Algorithm SEARCH FOR IMPORTANT TOKENS
1: Input: 2: prompt, Oama: draft model, Ouaggrr target model
2: Output: sequence of M mismatches, labeled as important or unimportant
3: M < 0
D A set of tuples (position, target token, draft token, important)
4: y < GENERATE(Z, (angg))
5: Q < EXTRACTANSWER
6: y < FORWARD(Z @ y, Odran) .rgmaxx (-1) [len -1:
7: I + {i 11 + }
> Indices where draft and target tokens mismatch
8: while I + #
9:
> The earliest position where mismatch happened D - FIr @ Dr & GENERATE(1 @ Bia @ Dt, Otarget) Replace De and continue with Ozarget
f s min(I)
10:
11:
â EXTRACTANSWER(G)
12:
if a = then
13:
M c +-M ((1, Vt. Nc, False)}
Equivalent answer, token ye is not important
14:
Y ++ D
> Continue search from the new response
15:
Y FORWARD(2 @ y, ddma) argmax (-1) [1en -x:
16:
else
17:
M t M U {(t, Vt. De, True))
Different answer, token ye is important, keep it
18:
end if
19:
I (ilys # Di n ni>1}
Continue with the remaining mismatches after /
20:
end while
21:
return .M..
target model token with its draft version does not change the final answer, we deem this token swap "unimportant" and allow it to be generated with the faster draft model. In turn, if swapping out the token changes the final answer, it is deemed "important" and should be generated by the main model.
In more formal terms, consider the task defined as a prompt with and models: the larger Brargez and the smaller frafft- Both models can generate a response y = (V1..--, UT) GENERATE(I, fdrafe) with up T T'STmaz total tokens. For simplicity, we first assume that the GENERATE procedure is deterministic (e.g., greedy) and generalize to sampling in Appendix A
Without the loss of generality, final answer from the model's 's response, a = EXTRACTANSWER (y). mathematical reasoning tasks such GSM8K [Cobbe et al., 2021]. the final answer is literally whatever the model puts after "the final answer (is)". In programming tasks, the "answer" would be the output from the testing system given the generated code either report about passing and failing tests a testing error (e.g., an Out Of Memory or Syntax Error). Finally, we say that two answers are the same from the downstream task perspective. Note that this does
also assume that there is a problem-specife way extract the
equivalent alt if they not require them to be exactly equal as strings: in math problems, 1. 1.53/2 whereas in programming tasks, two programs can be equivalent despite having different variable names. If the task at hand does not have a formalized evaluation procedure, e.g., general conversation agents, we can define EXTRACTANSWER(J)) y and detect if two answers are equivalent using an LLM or human judges.
Following this notation, let Mtarget = GENERATE(Z. (z, Qtarget) be the main model outputs. A token Yt Vharget is unimportant if swapping that token for the draft model's output results in an equivalent answer. Likewise, replacing yi (and continuing target generation from there) results in a different answer, then the original token "important" and the token should be generated with Starget-
Note that even Odraft is significantly smaller than Osarget, most of the individual tokens will match between the two. As such, we are only interested in the mismatches the cases where draft and target models produce different tokens given the same prefix:
I(x) = [1,T): arg max P(yneut (2. U1:t. Otarget) # arg max P(ynezt Va:t, Odraft))
where V/a 91............... .1..... Vi- denotes taking a prefix of y up to, but excluding index t. prac- tice, we can find these tokens quickly by re-encoding the target model response with the draft model: FORWARD(s @ y, Odraft)) arrgaa..gaaxxxx (dim=-11 [M-1:M+T-1]] where x @ y denotes concate- nation, FORWARD(".> is a parallel transformer forward pass that outputs next token logits, and the logits. art.arggga. argmax (dim=-1) [M-1:M+T-1] takes the most likely next tokens for every position, excluding the prompt and accounting for the shift from next token prediction.

---

When deciding if a mismatching token is important for the final response, we need to account for the fact that changing one token will most likely lead changes in subsequent tokens. A naive way to account for that change is by continuing the response after replacing one token yt:
V1at @ Yt + GENERATE(S (2 + y1.t + Dt, Itarget)
However, this approach has significant downside in that it assumes that all subsequent tokens will be generated by Atarget, whereas in reality, some of them may be generated by fdraft following the same algorithm. In preliminary experiments, we found that, with capable enough Strgett even significant generation errors can be detected and self-corrected (similar to the "Aha moment" from DeepSeek-A]] et al. [2025). Muennighoff et al. (2025]). However, if the model makes multiple mistakes, they eventually reach a critical mass, leading to an incorrect answer.
To address this, we reframe our task from detecting individual important tokens to finding com- binations of tokens that jointly affect the final answer. This changes our problem to finding the minimal set of mismatching tokens that need to be generated by Otarget while still producing an equivalent answer[] Since replacing a single mismatching token affects all subsequent token choices, the exact solution to this problem requires a tree search over possible token assignments. While this type of tree search is possible, it would take up significant runtime due to the large number of LLM forward passes required to try all mismatch combinations.
To simplify the procedure, we opt instead for a simpler, semi-greedy search that starts from the target model response and iteratively tries to replace mismatching target model tokens with their draft counterparts. If replacing a token affects the final answer, we consider this token important and keep the original (target model) version. If, however, replacing the token results in an equivalent answer, we deem this token unimportant, replace with the draft model version and continue the search from the new sequence, with different suffix and possibly different I. That way, we guarantee that the search algorithm is aligned with what happens during inference: the important tokens are generated with the target model and the unimportant ones are kept from the draft model. We summarize the resulting search procedure in Algorithm D and discuss some of its implications in Appendix A
3.2 Classifier Training
Once we gather dataset of task-specific important tokens with Algorithm can train classifier that would detect such tokens for use during inference. This classifier can, in principle, be any type of model, from simple linear model decision tree to fine-tuned transformer layer. However, in work, we default to training lightweight linear models with existing LLM hidden states as features, since those would introduce the least overhead during inference. There are several important design choices that can affect the effectiveness of such a classifier: we address each one separately.
1. Which token representations to use: the hidden states that predicted the mismatched token, or the next hidden states that encode the mismatched token itself? In our experiments, we found that using the latter representations results in substantially greater classifier accuracy (see Appendix B). However, obtaining these representations comes with a caveat.
Normally, when doing speculative decoding, one generates a draft "window" of W tokens with Oreaft, then verifies these tokens by processing them (in parallel) with Starget- This automatically computes the necessary hidden representations for all but for the very last token - the next token predicted from the last hidden state the window, which is not encoded. There are two ways to address this: either encoding the extra token alongside the window, or simply assuming that if the very last token mismatches between O Odruft and Ptarget, it is automatically discarded without the classifier. However, in practice, we found that the overhead from either strategy is negligible and is outweighed by greater classifier accuracy that translates to more accepted tokens.
2. Which token alternative to use? Since the classifier works best with the representations from encoding the mismatching token, it is natural to ask which token should be encoded: the draft token, the mismatching target token, or both? When analyzing this, we found that using both token representations comes with a slight increase in classifier accuracy (see Appendix B). However,
For notation simplicity, we assume that the GENERATE(:.) .) function can called with a prefix of a response. In that case, we assume that the total response length (and not just newly generated tokens) does not exceed T Tmaz. so that the response cannot grow indefinitely with each subsequent replacement.
Mooe precisely, find the sastesttto-gnnerae sequence, accounting for the differences in response length.
5

---

[GSM8K] Arnel had ten boxes of pencils ... how many pencils are in each box?
Arnel kept ten pencils and shared the remaining pencils with his 5 friends.
[equally] with ... v This means that the total number of pencils he shared is 10 x 10.
[.] He shared the ...
[Arnel]
V [-] x 10 ... X
[GSM8K] Adlai has 2 dogs and 1 chicken.
[LCB] Given string S of lowercase...
How many animal legs are there in all?
If there are adjacent occurren- ces of a and b in S, print Yes;
To find the total number of animal legs, we need to calculate the legs [total] of each animal and then add them up.
pyyhon
# -*-(YouR) coding: utf-8 -
def(u] solve (s)
2 dogs have 4 [2] legs each, so 2 dogs have 2 [times] 4 8 legs.
for i in range (1) -D] 1)
if s s[i] == 'a' and s [i+1] = Ab²: return "Yes"
1 chicken has 2 legs.
Now [Adding], let's add the legs together [of] we get 8 (from the dogs) + (from the chicken) 10 legs.
if s[i] == 'b(/ and s s[i+]] == 'a?:
return "Yes"
return "No"
The final answer is 10.
if (e) ___ee._.. "__maan__"""
Figure 2: Excerpts from GSM8K (top, left) and LiveCodeBench (right) labeled by Algorithm Important mismatching tokens that are in red, unimportant ones are in green. Alternative tokens are shown in [brackets]. Black tokens are where Garaft and Ssarget gave the same prediction. The top example additionally shows Otarget continuations after mismatching tokens (V if â. X if not).
obtaining these representations in practice would require running Otarget more than once during the verification stage, which would complicate inference and introduce performance overhead. For this reason, we opt to use only the draft token representations for the classifier, since those are already available in regular speculative decoding.
3. Which model provides feature representations? During the verification stage, we have access both the draft and the target model hidden states: we can use either or both of them as the input. In practice, we found that concatenated draft and target model representations give slightly better results than just those of the target model, and using draft model representations alone is substantially worse. Since both representations are already available during inference, we opt to use each of them.
Classifier model & training. In this work, we train a simple logistic regression to detect important to- kens. While a more complex model could achieve greater accuracy, logistic regression is significantly easier to deploy, has less runtime& & memory overhead and needs less training data. Furthermore, can be fused with the existing "LM head" layer of the draft and target LLMs, which would make its computation virtually free. To control overfitting. we perform a simple grid search over the L2 regularization coefficient ("C")) with logarithmic grid. We report additional details in Appendix B
3.3 Inference
The resulting classifier can be used with an arbitrary speculative decoding algorithm that has a verification stage. During said verification stage, the classifier is called when the original algorithm would reject token. If the would- berdd-be-rejected token is deemed to be unimportant, i.c. not to affect the response quality, then override the verification procedure and accept the token instead, proceeding to test subsequent tokens (if any) as per the original algorithm.
Generality. In our initial experiments, we focus on traditional speculative decoding [Leviathan et al. 2023 Chen al. 2023) for simplicity. However, our algorithm is compatible with arbitrary speculative decoding algorithms, including tree-based [Miao et al.] 2023 Svirschevski et al. 2024
Chen al., 2024) and single-model multi-heal algorithms [Cai et al.) 2024, et al. [024d|c]
This also means that our approach can be integrated into existing inference frameworks such as
vLLM [won et al. 2023), TensorRT-LLMM (NVIDIA] 2023 or TGI (Hugging Face 2023).
Thresholds. To balance computational efficiency and downstream performance, we select a decision threshold that achieves a high recall (290%) in order to retain quality. Since the classifier is accurate enough, this threshold can also achieve a reasonable rejection rate, i.e., the rate of tokens correctly predicted to be unimportant. This allows us to retain downstream accuracy while skipping large

---

Llama B.1 8B Instruct 85.97%
Llama 70B Instruct 92.19%
85%
79%
91%-
Test Accuracy
73%
Test Accuracy
89%
67%
87%
61%
Spec. Decoding
Spec. Decoding
AutoJudge (ours)
AutoJudge (ours)
55%
85%-
Top-K (bascline)
Top-K K (baseline)
Laaaa 3.1 8B Instruct 83.47%
49%-
Llama 3.2 1B Instruct 43.36%
43%
10
12
18 20
15
20
25
30
35
45
50
Average Accepted Tokens
Average Accepted Tokens
Figure 3: Accuracy and the number of accepted tokens on GSM8K for (Ieft) 8-shot Llama-3.2 IB
draft 1 8B target and (right) 0-shot Llama 3.1 8B draft / Llama 3.1 70B target (all Instruct)
portion of unimportant tokens, thus enabling efficient speculative decoding. In Section we evaluate various threshold values show their effect on accuracy and acceptance rate.
Comparison with Judge Decoding. As we discussed earlier, our approach can be seen as an extension of Judge Decoding that enables automatic dataset mining. As such, the dataset generation algorithm from Section 3.1 can be used in conjunction with the Judge Decoding training and inference protocol, which appears to be similar to ours up to possible minor details. In Appendix N we additionally compare against manual human annotation similar Judge Decoding. Unfortunately, the original source code and data from Judge Decoding are not available, making it difficult to compare directly.
4 Experiments
We organize our evaluations as follows: in Section 4.1
we evaluate AutoJudge on the GSM8K (Cobbe et al. 20221] mathematical reasoning benchmark. Next, in Section 4.2 we evaluate our approach on programming tasks from LiveCodelencc (Jain et al. 2024]. Finally, Section 4.3 contains GPU inference speed benchmarks with our vLLM implementation. We focus on two pairs of Llama 3.x models: 1) Llama-3.2 2-1IBIIntruc draft 1.ama-3.1 1-ttaa....-B-BBB- Instruct targe and 2) Llama-3. B.itna--.1-B---- --st-uuttttct draft / Llama-3,1 -lamma-3.1-70B
target. We also report results with Qwen2.5 models on the GSM8K benchmark to showcase the method's transferability across model families. Our main experiments run in of 1 precision (see Appendix C). We run AutoJudge on top of the standard speculative decoding algorithm [Leviathan et al. 2023 in main experiments and explore EAGLE-2 in Appendix[]
4.1
Mathematical Reasoning with GSMSK
Our first set of experiments is based on the GSM8K dataset with grade school mathematical problems. This dataset has natural split with 77.47K training samples and x1.32K test samples. Following the standard evaluation procedure, we use the training set to "mine" important tokens with Algorithm ] and train the classifier, then run inference and evaluate on the test with the recommended parameters [Gao et al.] [ for zero-shot and 8-shot evaluation: greedy inference with prompt that encourages chain-ol of -hhougghht reasoning. During training, we consider two responses equivalent (a=â in Algorithm T if the extracted final answers (numbers) are equal. For reference, we provide an example of important token assignments found by our algorithm in Figure
We train a classifier on the last hidden state embeddings from both draft and target models (concate- nated) for encoded draft tokens. The training dataset contains w130K mismatches, about 20% of which are deemed important. We train a logistic regression with the Le regularization coefficient tuned individually for each setup(for instance 10 -4 for 8B/70B), found by grid search over a log- arithmic grid between 10 ,0,, 10-?, During inference, we integrate the trained classifier into the
"The reason why the two models have different minor versions (i.e. I and 3.2) is that the 3,2 version does not have the larger 8B models, and the 3.1 version does not have the smaller IB models.
7

---

Owen2 5-7B I--nsstruct 87.33 87.33%
Owen2.5-32B. Instructtttttttt 91.73%
85%
Spec. Decoding
90%
Praeer www.
Spee. Decoding
80%
AutoJudge (ours)
85%
AutoJudge (ours)
Top-K (baseline)
80%
75%
Top-K (baseline)
75%
Test Accuracy
70%
Test Accuraay
70%-
65%
65%
60%-
60%
55%
55%-
50%
50%
45%
45%
40%
40%-
Qwen2.
50,5 Instruct 37.90%
Qwen2 2.00m2.300330 Instrcl %0%
35%
35%
30
40
50
60
10
20
30
40
50
Average Accepted Tokens
Average Accepted Tokens
Figure 4: Accuracy and the average number of accepted tokens on GSM8K 8-shot for (left) Qwen2.5- 0.5B draft Qwen2.5-7B target and (right) Qwen2.5-0.5B draft Qwen2.5-32B target (all Instruct).
speculative decoding loop from Leviathan et al. [2023] during verification. Whenever the original algorithm would reject token, we run the classifeer to determine if changing that token affects the final response quality, and if not not-aaa accept the token and continue verification for subsequent tokens (if any). Since the resulting algorithm can accept additional tokens, we use the increased draft window size of W-64 tokens for all evaluations.
We report two main metrics: downstream accuracy and the number of accepted tokens per speculative decoding cycle. The accuracy is measured as the exact match rate for the final answer extracted from the response as per standard GSM8K evaluation protocol. In turn, we report the decoding speed in terms of the number of tokens accepted per target model forward pass, aiming to decouple our results from the specific hardware configuration. We evaluate AutoJudge with different classifier thresholds, balancing between accuracy and speed. Our baselines are traditional speculative decoding, decoding with the draft model, and simpler lossy speculative decoding method. In the latter, we accept mismatching draft token if it is within top-K most likely tokens of the target model, similarly to how is defined in Bachmann et al. 2025). We report K K=2, 4, .,..., [V] for different tradeoffs: increasing K results in more accepted tokens but reduces accuracy.
........I.......
The results in Figure demonstrate that AutoJudge decoding can achieve substantial speedups. Notably, our algorithm can accept over 40 tokens per target model forward pass in 8-shot evaluations for the 8B draft / 70B target model pair with <1% change in accuracy. Varying the classifier threshold allows us to achieve even greater speedups at the cost of several percentage points drop in accuracy. The heuristic- based top-K bascline also achieves some speedups. but the cost of significantly higher accuracy drawdown. We report additional setups in Appendix D.I
4.2 Programming with LiveCodeBench
Next, test if the AutoJudge search algorithm is able to generalize between domains. For this purpose, we evaluate the same model family on the LiveCodeBench [Jain et al. 2024) programming benchmark. Here, The dataset contains 880 programming tasks; we evaluate on all three subsets: easy, medium um and hard. Since LiveCodeBench does not have a dedicated training split, evaluate using out-of-fold predictions. Namely, we split the dataset randomly into 5 folds. For each fold, we evaluate using the classifier trained the remaining folds. We use the standard evaluation protocol: extracting the generated code and evaluating it using the benchmark's built-in test suite. We follow the standard evaluation protocol for this benchmark and report Pass @ in zero-shot setting.
the code geweratiionnnn.nom dite"datasee with the version tag release _v5..
Similarly to Scction we use the training data to find important tokens - this time, in terms of the resulting program correctness, measured as passing tests. Since the calibration dataset is smaller and further subdivided into folds, we only have x27K mismatching tokens to train the classifier (with a

---

Llama 3 8B Instruct 16.81%
17%
34%
Pass (correctly solved rate)
Pass (correctly solved rate)
32%
Llama 3.1 708
Instruct 30, 9.%
15%
30%
13%
28%
11%
26%
24%
9%
Spee. Decoding
22%
7%
Spec. Decoding
AutoJudge (ours)
20%
AutoJudge (ours)
Top-K (baseline)
5%
Top-K (baseline)
18%
Llama 3.1 8B
Instruct 16.25%
Llama 3.2 1B Instruct 3,07%
16%
3%
10
15
20
25
30
35
40
45
15
20
25
30
35
40
Average Accepted Tokens
Average Accepted Tokens
Figure 5: Downstream Pass and the average number of accepted tokens LiveCodeBench for (left) Llama-3.2 IB draft Llama-3.1 8B target and (right) Llama 3.1 draft Llama 3.1 70B target. We use Instruct versions for both model pairs and report additional details in Section 4.2
slight <0.5K variation depending on the active fold). Furthermore, we found that only =55% of the mismatching tokens were deemed to affect the output quality. We provide example token assignments in Figure Figure[(rightt) (right): notably, the tokens deemed important that case would not appear in GSM8K in the same context. We otherwise follow the same training and evaluation protocol before.
The results in FigureBjare similar to what we observed in Section 4.1 AutoJudge decoding can accept over 22 tokens per speculative decoding cycle at the cost of a 2:% accuracy decrease. This results in approximately 2 increase over traditional speculative decoding [Leviathan et al. 2023 The top-/ K baseline similarly achieve some increase in the number of accepted tokens, but AutoJudge decoding offers significantly better quality-speed tradeoffs across all configurations. We report additional configurations and threshold values in Appendix D.2 including the setup for AutoJudge decoding fout-of-domalls i.c. using the classifier trained on GSM8K for LiveCodeBench evaluation. This out-of-domain configuration results in inferior performance, which aligns with hypothesis that the important tokens depend on the problem type and evaluation criteria.
4.3 Inference Benchmarks with vLLM
In this section, report the GPU inference speed of speculative decoding with AutoJudge classifiers. To benchmark real-world inference speed, we integrated AutoJudge into vLLM speculative decoding. We use the same GSM8K evaluation setup as Section 4.1 and evaluate model pairs from Llama 3.x family: 1B/BB, 8B/70B and 8B/405B draft/target respectively (all Instruct).
Table I: Inference speed benchmarks GSM8K 0-shot with vL.MM for (left) 1B draft / 8B target models with tuned window size (baseline = 8, AutoJudge 10) and (right) for 8B draft 70B target models (all Instruct) with tuned window size (baseline = 8, AutoJudge 32).
Llama 3.2 IB draft / 3.1 8B target (0-shot)
Llama 3.1 8B draft / 331 70B target (0-shot)
Threshold
0.06
0.12
0.15
0.16
Threshold
0.005
0.031
0.145
0.230
Accuracy, %
83.1
80.2
79.8
77.4
Accuracy, %
92.0
91.9
89.9
88.0
Speed, tokens/s
149.2
169.2
171.2
173.9
Speed, tokens/s
72.3
80.6
107.4
109.6
Speculative Decoding:
147.7 tokens/s
Speculative Decoding:
72.3 tokens/s
Speedup(ours)
1.01
1.14
1.15
1.17
Speedup(ours)
1.0
1.11
1.49
1.52

---

Table 2: Inference speed with vLLM for (left) Llama 3.1 8B draft /00BB target models on GSM8K O-shot with tuned window size (baseline=14, Autoludge=20). (right) Llama 3.1 8B draft / 70B target (all Instruct) on GSM8K 8-shot with offloading, tuned window size (baseline=10, Autoludge=48).
Llama 3.1 8B draft 3.1 405B target (0-shot)
Llama 3.1 8B draft / 3.1 70B target (offload)
Threshold
0.01
0.05
0.09
0.14
Threshold
0.03
0.05
0.11
0.28
Accuracy, %
96.1
93.4
92.5
91.5
Accuracy, % Speed, tokens/s
95.4
94.8
93.4
90.4
Speed, tokens/s
50.6
58.0
58.5
60.1
1.4
1.6
1.9
2.4
Speculative Decoding: 50.0 tokens/s
Speculative Decoding:
1.19 tokens/s
Speedup(ours) 1.01
1.16
1.17
1.20
Speedup(ours) 1.20
1.31
1.59
1.98
We compare AutoJudge decoding against lossless vLLM speculative decoding [Leviathan al. 2023
To ensure fair comparison, we tune draft window size for AutoJudge and baseline independently (see Appendix G In each configuration, we report the absolute speed in tokens per second (batch size 1), as well as the relative speedup over speculative decoding. We run 1 B/8B model pair on a single A 100-SXM4-00GB GPU; 8B/70B the 8B/405B runs on 8 10000005-80GGG GPUs with the 405B model loaded in FP8 precision. Additionally, we consider a setup where 8B/70B model pair runs on a single A A100-SXM4-80GH GPU H
A 100-SXM4-80G GPUs in tensor-parallel mode. Finally,
with RAM offloading (see Appendix
We provide additional configuration details in Appendix E
Our results in Tables D and demonstrate consistent improvements across all model pairs, with particularly high speedups for 8B/70B with and without offloading. This confirms that our earlier results in terms of accepted tokens (Sections 4.1 & 4.2) translate to real- world tokens per second. Namely, if we allow x 2% change in accuracy, our vLLM inference with AutoJudge classifier can achieve 107.4 tokens per second for 8B/70B model pair, which translates into about 1.5x speedup compared to regular speculative decoding. This improvement is even more noticeable for hybrid setup with offloaded target model (in Table 2. where AutoJudge achieves up 2x speedup.
Additional evaluations. To better explore the wide variety of tasks, methods and hardware setups, we report multiple series of additional experiments in supplementary materials. We evaluate Auto- Judge decoding with EAGLE-2 in Appendix J Next, we generalize to open- open-ended problems (question answering and creative writing) with LLM-as-as a-a-ueee evaluation in Appendix [[]
We also evaluate how AutoJudge classifiers transfer to adjacent problems in Appendix M Finally, we benchmark with equal window size in Appendix D and control for vLLM nondeterminism in Appendix [
Limitations. Our approach assumes that the downstream task has way to determine whether not two solutions are equivalent. While this is true for many tasks, we found that AutoJudge sometimes struggles in open-ended tasks with clear criteria for correct answers. It would be interesting to explore alternative algorithm designs more suitable for open-ended problems. Additionally, running Algorithm consumes compute resources, mostly LLM inference, in the form of local inference or API calls. We discuss this in more detail Appendices anddn Q
Conclusion
In this work, we propose and evaluate a fully automated protocol for task-specific acceleration of speculative decoding. Our experiments show that simple procedure can automatically determine which of the mismatching tokens in the LLM response affect the downstream quality tasks. Our runtime benchmarks demonstrate significant speedups on top of already tuned speculative decoding and EAGLE algorithms with minimal inference code modification. We hope that AutoJudge can facilitate the use of Judge Decoding across different task types, languages and modalities.
variety of
One promising direction for future research is to focus on open-ended problems such as instruction following or creative writing. While AutoJudge already demonstrates some speedups on open-ended tasks, it was primarily designed for technical problems such as math and programming. It would be interesting to investigate how can be extended for longform generations with noisy and informal quality criteria. Another possible direction is to combine AutoJudge with additional speculative decoding algorithms, such as speculative decoding with tree-based drafts MMio al.) 2023 Chen
et al. 2024 Svirschevski et al. 2024 or learned drafting heads [Cai et al. 2024
Li ct al. 2025
10

---

Acknowledgements
We would like to express our sincere gratitude Denis Mazur for his valuable contributions the implementation of API calls used in Algorithm and for supporting inference with the Llama 405B model. We are also thankful for his positive influence on the overall atmosphere and team morale throughout the course of this project.
References
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Sequeentaall--eeenddee draft heads for medusa decoding, 2024. https:tss://aatt
org/ /402 05109
Anthropic. Claude 3.7 sonnet and claude code, 2024, URL wtt.:aanthropcc.coo//eem
claude-3-7-sonnet Accessed: 0025.04.02.
ARC Prize Foundation, Openai's new o3 system scores breakthrough on arc-agi-pub, 2024. URL httpp://aarppriaa..orr//b/bg//// pwp.aa-----------bbbabbrrrrrgo Accessed: 2025.03.28.
Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, and Jonas Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment. In The Thirleenth Interna- tional Conference on Learning Representations, 2025.
www.ereeiiieneeefrureeeee net/forum?
id Id=mtSFFiqW6y
Edward Beeching, Lewis Tunstall, and Sasha Rush. Scaling test-time compute with
open models, 2024. URL https:/ httpsss///uginggace... wwwpggiinnaaee.cpppcccoooo /tt///aace//uggiggaaaaeeae//e
blogpost-scal wawlllinttttt tes-testtttie time- timp-ccutte
Tianle Cai, Xinyun Li, Zhiruo Wang, Yuhuai Wang, and Dawn Song. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401. 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, fean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013/88, 2023.
Zhuoming Chen, Avner May. Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-awant speculative decoding, 2024. URL
https: ftttps://arxiv.orr/ org/ abs/ 2402 12374
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05533 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
DeepSeek-AI, AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chenggi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao. Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou. Shanhuang Chen, Shengfeng Ye, Shiyu Wang,

---

Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye. Tao Yun, Tian Pei. Tianyu Sun, Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L.. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, 7. 7. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang. Zhipeng Xu, Zhongyu Zhang, and Zhen Incentivizing reasoning capability in ilms via reinforcement learning, 2025. www.///511...119911
Zhang. Deepseek-rl:
URL
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama herd of models. arXiv preprint arXiv:240..21783, 2024.
Yichao Fu, Peter Bailis, lon Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. Accessed: 2023-11-22 2023.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL httpt:///oo org/ 10. 200.5/22//emmoo537777
Google DeepMind Gemini 2.5: Our Newest Gemini Model with Thinking, https://blog google/ Bechnollogy/goggle-deepmp gemiiieeeemmnnmeeiiiiiiieieeinii -motiii-imool---- www.ttttiiiinnuuiteee
march-
2025. Accessed: 2025-04-07.
E. J. Gumbel. Statistical Theory of Extreme Values and Some Practical Applications, volume 33 of Applied Mathematics Series. National Bureau of Standards, Washington, D.C., 1954. U.S. Government Printing Office.
Horace He and Thinking Machines Lab. ence. Thinking Machines Lab: Connectionism, 2025. doi: 6644434/ml.20250099 httpss/ttinkingnachinnssss
Defeating nondeterminism in Ilm infer-
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval- speculative decoding. arXiv preprint NNii:233/.00255: 2023.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR. OeenReview.net, 2020. URL URLhttp: httt::/tppppppp httpo///oo///t////////aoaaattoaa iclr/ic1r2020. html #Hol ti11nn022mBB
Hugging Face. Text generation inference, 2023, URL
pext-generation-- inference GitHub repository.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. URL httth:t/ppsaaaee //tpiv///xvvvvv/a .rg. atgg/bss/4033 07974
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation, 2024. URL http:tttss//ari..rrg/////// wwtp:////////a////o//// 00/00/0002/2000005555
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada, July 2017. Association for
12

---

Computational Linguistics. doi: 10.18653//1/P17-114477 URL https::/acaaatttllllleee/ org/ P17- P17-1147/
Schoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Neural Information Pro- cessing Systems, 2023. URL https://apppp..... http:s////ppp.emmmnncccccoooooo
org. Terg/CorpusID-... :
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916, 2022. URL https://api
semanticscholar httpr///ccccccooo..ooo////ro/raa wttp:o//orpps:::::9001/7.0
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-6622, 2023.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. 2023.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj. Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Randy Zebaze. Ming-HH Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nourhan Fahmy, Urvashi Bhattacharyya, W. Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jana Ebert, Tri Dao, Mayank Mishra, Alexander Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean M. Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! ArXiv, abs/2305.06161, 2023. URL https: //ppppapp....a http/ppp..smmaiicscchhllaaat www.aaowggoooooaaa.e 258588247
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024a.
Tianle* Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024b. URL 2aalogg/22224 2024-0222202 04 19- areaa-reaaaaaa
https:///msys: www.oorgg////ooooooooooooooooo/o
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: 2: Faster inference of language
models with dynamic draft trees, 2024c. URL htttittts:/:/ariitt www...rggggbo... abs/
abs//406.. 2406. 16858
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, pages 1177-111 PMLR, 2024d.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference accel- eration of large language models via training-time test, 2025. URLhttppa chttts://arxiis //arxiv. org/ abs/
0084000840
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Ion Stoica, Zhijie Deng, Alvin Cheung, and Hao Zhang. Online speculative decoding. arXiv preprint 2rX1v:2310.07177, 2023.
Paul Joe Maliakel, Shashikant Ilager, and Ivona Brandic. Investigating energy efficiency and performance trade-offs in I1m inference across tasks and dvfs settings, 2025. URL https://
arxiv. org/abs/////eeoooooo/..ooeeeeera ab///5bb////5// 2201 08219
Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https www.immmmal--neeellllel
//ai meta. com/bloge
2025.
13

---

Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative IIm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.00781, 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, arXiv preprint arXiv:2501.1933 2025.
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascades via speculative decoding. In The Thirteenth International Conference on Learning Representations, 2025. URL https
NVIDIA. Tensorrt-llm, 2023. URL httppp//ggggthacccc////////aaaar
OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaicv, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy. Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy. Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann. Freddie Sulit, Geoff Salmon, Giambattista Parascandolo. Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere. Hadi Salman, Haiming Bao. Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang. Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang. Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowd- hury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay. Steph Lin. Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi. Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha. Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card 2024. httpst / arxiv. org/ abs/ www.a///.......... /412 16720
14

---

Rui Pan, Yinwei Dai, Zhihao Zhang, Gabriele Oliaro, Zhihao Jia, and Ravi Netravali. Specreason: Fast and accurate inference-time compute via speculative reasoning. 2025. URL https://arxiv org/abs/ 2504. 07891
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS). Neural Information Processing Systems Foundation, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, al. Scikit- Machine learning in python. Journal of machine learning research, 12(Oct):2825-22) 2830, 2011.
Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis. ArXiv, abs/2307.16789, 2023. URL https: //api. www.ssiesmaononlaa.o.mmo org/
:
Qwen Team. Qwq-32h: Embracing the power of reinforcement learning, March 2025. URL bttpss://qwett http://wwelm....tuu.b.. wwt.eiiggttbbbbbbbbeeeeee wwtioo/o/oo/aaeeeee qwq--2b///////// 32b/
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Cantón Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D' D'efossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. ArXiv, abs/2308. 2023. URL httpp://api semanti cscholar. org/ ID:
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themsel tools. ArXiv, abs/2302.04761, 2023. URL httts:///ppp semant cacnnnaatccchlla.....m org/ Corpus ID: 2256697342
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can more effective than scaling model parameters. arXiv preprint arXiv:2408.033 2024.
Benjamin Spector and Chris Re Accelerating Ilm inference with staged speculative decoding. arXiv preprint arXiv:2308. 20X0:22308.04623,, 2023.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. Advances in Neural Information Processing Systems, 31, 2018.
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. arXiv preprint 100iv:2310.15144 2023.
Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thoughh can solve them. In Anmual Meeting of the Association for Computational Linguistics, 2022. URL https: semanticscholar org/ Ing//orpusII: 252217648
Ruslan Svirschevski, Avner May, Zhuoming Chen, Beidi Chen, Zhihao Jia, and Max Ryabinin. Specexec: Massively parallel speculative decoding for interactive Ilm inference on consumer devices, 2024. URL httpst//::/// http///axivvv.. wwwwa.orgaabaaaaaaaa.o/aaaa www.//gg///beeeeeeeeeeeeeeeeeeee 02532
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: instruction- followiinnn llama model.
btttps:/ggite wtttchhuu...m/tttteeeeeeee wtw.aattttaaaaataaaarroaeeea alpaca_ 2023,
15

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Vivien Tran-Thien. An optimal lossy variant of speculative decoding, June 2024. URL https:// huggingface, co/blog/vivei www.entttmaaaeeeeeee wtw.i//eettttammloleeeeeeeeeeeee -ossy- -f deceding Hugging Face Blog.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the tet--hh-art art natural language processing. arXiv preprint 1111v:1910.03771, 2019.
Yang, Baosong Yang. Beichen Zhang. Binyuan Hui, Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of-thought reasoning Ilms, 2025. /ttps://aa/r www. wwwpg/gggaa......./// 03373
Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi. Xia Hu, and Zirui Liu. Give me fp32 or give me death? challenges and solutions for reproducible reasoning, 2025. URL https:tp::// //arx//aavvv///////////eee/aavv/ org/ abs 250. 09501
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that AutoJudge can accelerate decoding and support with experi- ments in Section 4 including vLI.M inference benchmarks in Section 4.3 We claim that our method works across different tasks, supporting it with an evaluation on both mathematical reasoning and programming benchmarks. Our claim that the method does not require human annotation (unlike Judge Decoding) and provide the search algorithm (Alg. [ that justifies this claim. Thus, the claims in the abstract do indeed reflect our main contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification; We summarize the limitations of our work on the last page, near the Discussion section. For instance, that our algorithm is so far limited to tasks where one can determine if answers are equivalent, which may not always be possible.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
17

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: While include some simple mathematical claims, none of them are signifi- cantly novel. The main contributions of our work are empirical, not theoretical.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the supplementary code that includes instructions for reproducing our experiments and details regarding hardware and library versions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
18

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the supplementary code with the paper and intend to release immediately upon publication. For convenience, also release the mined datasets of important tokens used in our work, though one can also reconstruct them from public datasets using our algorithm.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
mublic/guides/ CodeSubmissionoolooo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http:llccggiieeeeeeeeeeeee http://udddes/oeeuuubbiiiiii www.oodeuuiiiisonnn....
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide the important configuration details, within reason, in the experi- mental section and appendices. More importantly, we include the exact configurations with the supplementary code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We include error bars in some, but not all results. The main reason we do not provide error bars everywhere is that our main experiments require L.M inference with long sequences on large datasets, which is computationally expensive.
19

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We rum our experiments on a fairly homogenous hardware (servers with 8x A100 each) and describe details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that wtw.eeeteeennleeeeeeeneeneeeeeee
didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our work is focused on fundamental LLM inference efficiency and complies with the Code of Ethics. Notably, AutoJudge does not require human annotators, which means we do not use crowdsourcing.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
20

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work (stemming from more efficient LLM inference), none which feel must be specifically highlighted here.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All the models and datasets used in our work are either already open- open-access or can be generated automatically based on public models and datasets. Hence, we see no need to safeguard their release.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
21

---

Justification:
We do our best to cite all code, datasets and models used in our research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssi..iiipiiipa.... wncerswwthcodeeee thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our supplementary code contains a README with instructions for reproduc- ing our experiments.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: We only have one small-scale analysis that required a small amount human annotations (in Appendix Nj. This analysis was by acknowledged volunteers with no compensation over the course of a few hours. One of the main contributions of our work is that provide algorithm that bypasses the need for human annotators.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
22

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: We only used crowdsourcing with small group of volunteers in Appendix N over the course of several hours. The crowdsourced tasks had volunteers analyze LLM solutions simple non-sensitive problems (e.g. grade school math). Our institution's policy classifies this analysis as too small-scale to seek formal approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Our work uses repeated LLM inference with substituted tokens to detect which of the generated tokens are important for LLM inference. We describe Section 3 and evaluate it through the actual use of LLMs in Section
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy for what should or should not be described.
23

---

A Additional Considerations for Section 3.1
Generalization to sampling. In Section 3.1 we assume that the generation procedure is determin- istic, i.e. that the model performs "greedy inference". In practice, however, many applications work better with stochastic sampling (Holtzman et al. 2020]. However, this has an obvious caveat for Algorithm[] [ if the text generation process is stochastic, token can be deemed important based not on its actual impact on the model outputs, but on the randomness of the decoding procedure.
To generalize our approach for stochastic generation, we take advantage of the well-known Gumbel- max trick [Gumbel| 1954]. To recap, if we add independent Gumbel-distribuee random variables to each predicted logit and take the index of the maximum, the probability that certain index will be chosen is equal to the softmax of the original logits.
In case of Algorithm| we use the Gumbel-max trick to reparameterize stochastic sampling from the model deterministic sampling conditioned on predefined random state +- RANDDBTSS()). Given a prompt x, a response prefix V1:t and model parameters 0, we generate the next token as follows:
Unext = arg max log P(ilæ @ Via, 0) + GUMBELPRNG(( @ @ Vn.))
where GUMBELPRIG a function that samples a pseudo-randowe variable from standard Gumbel distribution based on an input seed s @ x 2 M1.t. To recall, @ denotes concatenation. This way, Unezt distributed as P(ynext) x y1:4: f), but it deterministic when conditioned on the random state s. Hence, we sample random state once at the beginning of Algorithn the entire procedure after that will also be conditionally deterministic (given 8).
Issues with naïve important token mining. As we described earlier, Algorithm is inher- ently sequential because it searches not for individual important tokens, but for important to- ken combinations. In principle, it is tempting ers each token replacement in isolation and can run in parallel. However, when considering cetrget_model_eee
consider a simpler algorithm that consid-
sequences only, sufficiently strong target model might recover from even low-quality token and still produce the correct answer. This results in fallure mode where all tokens are individually unimportant, but when all such tokens are jointly replaced with their draft versions, the model fails to produce the correct answer. In our preliminary experiments, when using Llama-3.1-706 1-B-...1-----ttruuee Instructt Touvron et al. [023] as the target model and Llama-3.2- 3.2- 1--1B-IIsse II-nrutttct as the draft model, fewer than 1% of the tokens were labeled as important with this simplified algorithm, whereas our main Algorithm | found substantially more. ovver One interesting guarantee of Algorithm over its naïve counterpart is that whenever draft and target models produce different (non-equivalent) answers to a given prompt, our algorithm will find at least one important token, whereas the naïve algorithm may find none.
draft_token, draft token, target_ saatttmmdeeegeeee
On starting conditions for the important token search. To recall, mining important tokens can be viewed shortest path search algorithm in tree of possible mismatch choices. When performing this type of search, there are two possible directions that one can search from. In Algorithm| [ we start from the target model outputs and iteratively (greedily) replace the mismatching tokens with their draft versions. However, one could also start from the draft model outputs and iteratively swap in target model outputs until the answer becomes equivalent that of the target model. If we were to use an exhaustive search algorithm, both approaches would converge to the same important token labeling. However, since we are using a semi-gra genndyy algorithm, is easier to start with an already correct solution and simplify it, as opposed to starting with a wrong one and attempting to fix it.
B Additional Details on Classifier Training
there are several important design choices that can affect the performance of an important token classifier our setting. Here, we report the experiments that led us to use a línear classifier based on draft token embeddings encoded with both Odraft and 0main- To that end, we compare the different classifier variants using the important token embeddings from the GSM8K [Cobee et al. 2021 training subset (see Section 4.1
As we discussed earlier in Section 3.2
24

---

To compare different classifier configurations, we further divide the GSM8K training set into classifier training (90%) and validation (10%) subsets. We perform this division the sample level, ie.. all labeled tokens from a given GSM8K sample are used either entirely for classifier training, or entirely for validation. We use the same training and validation subsets throughout this section.
For the first set of experiments (Figure6). we compare regularizer coefficients for Logistic Regression (left). We also report different classifier types: Logistic Regression, Random Forest with 128 trees and multi-layer perceptron (MLP) with a single hidden layer consisting of 128 hidden units with ReLU activation. For consistency, we run models using scikit-learn [Pedregosa et al. 2011 v1.4.2 with all other settings kept to their default values. For LogisticRegression we set max iter parameter 500. For MLP, we perform early stopping on yet another 10% subset of the training set with the built-in default MLPClassifier early stopping parameters. For RandomForest we use min_ @amples_less leaf set 0.001. For this evaluation, all classifiers use draft and target model hidden states (concatenated) encoding the draft token, which is our main setup from Section 3.2
ROC curves for varying regularizer cae fficients
ROC curves for varying classifier types
1.0
0.8
0.8
True Positive Rate
True Positive Rate
0.6
0.6
0.4
0.4
1e AUC-0.803
0.2
0.2
A- AUC-0.818
Logutetloprroo (CrSe-4), AL
C-1e-02, AUC-0.804
Razdoes Fa orimasos 1251, A1/C-4755
0.0
0.0
MEP thidden, loyer aisor (128IL 0.39
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
False Positive Rate
Figure 6: Receiver Operating Characteristics and the corresponding AUC values values for different Logistic Regression regularizers (left) and classifier types (right). Bold lines are validation curves and the dotted lines represent training curves. The AUC is reported in the legend (bottom right).
The results in Figure 6 demonstrate that the classifier quality is fairly robust to the choice of the regularization hyperparameter. It is also fairly robust to the choice of the classifier architecture, barring perhaps the Random Forest classifier, which is overfitting the training data more than other models. Note that this does not necessarily mean that the MLP or tree-based classifiers are generally worse than linear models only that linear model is enough in our exact setup with limited training set. We hypothesize that, if allowed train much larger dataset, the more complex models will be able to match and possibly outperform logistic regression.
Next, we compare classifier inputs. As we describe in Section 3.2 we use existing LLM hidden states from the last layer of Odraft and fearget since they are already computed during speculative decoding, This, however, leaves several possible choices about which hidden states should be used:
Previous token embeddings, last hidden states used to predict the mismatching token;
Draft token embeddings are the next embeddings, obtained by encoding the draft token;
Target token embeddings are the next embeddings, obtained by encoding the target token;
Both token embeddings are concatenations of the draft and target token embeddings;
We compare the four input configurations in Figure (left), using Logistic Regression with C=10-4 and both draft and target model hidden states (concatenated) for each case. The results suggest that classifier that uses mismatching token embeddings (for draft or target token) significantly more accurate than using the preceding token embeddings (the ones used to predict the mismatch). In turn,
25

---

ROC curves for varying token embeddings
ROC curves for varying model hiddens
Troe Positive Rate
True Positive Rate
0.6
0.6
0.4
0.4
0.2
0.2
Previcu token embocdlings, AUC--0815
tokes enbellängs, AUC 0.819
model hidders, 0.760.763
Target token eathedingg, AubesdingssAA2e099.000 AU0000504
Target medel hi hiddens, AUC
0.0
takess enboddiigs, AUC-0.326
0.0
loken enbeddling AVC-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.6 Falsc Positive Rate
0.4
0.8
1.0
False Positive Rate
Figure 7: Receiver Operating Characteristics and the corresponding AUC scores (in the plot legend) for different classifier imput tokens (left) and models (right). See Appendix B for details.
using both token embeddings results in somewhat better performance than either of them. However, using both token embeddings introduces complications during inference time.
In normal speculative decoding, the algorithm already computes hidden states for draft tokens with both fd (during draft generation) and Starget (during verification). However, it does not compute embeddings for target tokens since those tokens not known before the end of the verification stage and computing them already requires a forward pass with Orarget. As a result, computing target (or both draft & target) token emheddings would require two sequential forward passes with Otarget one to determine the target tokens and detect mismatches, and the other to compute embeddings for those mismatching target tokens. In principle, one could devise a more sophisticated algorithm that computes only the faraft embeddings for mismatching target tokens or guesses the target tokens prior to the verification stage, but doing so would greatly complicate the implementation. Since the increase in the AUC score compared to using just the draft token embeddings is relatively small (Figure 7 on the left), we default to using draft token embeddings.
Additionally, we also test three model hidden states configurations for draft token embeddings: draft model hidden states, target model hidden states, and concatenated hidden states from both models in Figure 7 (right). Here, using the target model hidden states results in superior accuracy to using the draft model. In turn, using both ft and Oarget produces an additional, if marginal, increase in accuracy. However, since both hidden states are already available during inference, using them both does not pose additional complications. Though, some real world inference systems may make more convenient to only use Orarger for classifier inputs since the AUC difference is within 1%.
C Precision Matters for Speculative Decoding
When validating the AutoJudge algorithm, we found a peculiar implementation detail that can affect the real world performance of speculative decoding. Namely, when using the LLM in half precision, token embeddings can differ significantly (up to 10%) between parallel and sequential forward passes on the same data. In other words, if we record model hidden states as it generates a sequence, then encode the same sequence in parallel to recompute said hidden states, the two sets of hidden states will not match exactly. We attribute this to the fact that encoding tokens in parallel has different summation order to encoding tokens one by one, which introduces small numeric errors. These errors compound over consecutive layers, resulting in larger errors in the final hidden states.
This is important for AutoJudge, since Algorithm I Truns sequential inference with ftrreet and parallel inference on Bara/t, whereas inference time speculative decoding does it the other way around: sequential calculations of Od during the draft generation phase, then parallel forward pass with Osarget during the verification phase. As a result, the classifier is trained on features that can be significantly different from what they would be during inference. In contrast, running in full precision (float32) does not have such problems.
26

---

80%
75%
Test Accuracy
70%
65%
60%
55%
Spoe. Decoding
50%
Spec. Decoding GSM8K Accuracy AutoJudge (FP32)
45%
AutoJudge (BF16)
40%
10
50 Average Accepted Tokens
20
30
40
60
Figure 8: Accuracy on GSM8K and the number of accepted tokens per speculative decoding cycle in float32 and bfloat16 precision. The setup is the same as in Section 4.1
In Figure 8 we compare accuracy and acceptance rate trade-offs for different classifier thresholds in the same setup in Section 4.1 There several ways to circumvent this problem. The most practical one would be to recompute target model embeddings for Algorithm in parallel forward pass and not using the draft model embeddings (since adding them has negligible effect on accuracy. see Figure 7 right). As result, the classifier would use Starget embeddings computed in parallel over draft tokens during both training and inference.
D Additional Evaluations
D.1 Additional Evaluations for Section 4.1
In Figure we report accuracy and the number of accepted tokens for Llama-3, 1-8B draft Llama- 3.1-70B target and Ilama-3. 1-8 draft Llama-3.1 1-70B target model pairs GSM8K 8-shot setup.
Llama 3.11 70B Instruct 94.84%
95%
95%
Llama B. 70B Instruct 94.84%
85%
93%
Test Aceuracy
Test Accuracy
75%
91%
Spec. Decoding AutoJudge (ours)
65%
Spee. Decoding
89%
AutoJudge (ours)
Top-K (hascline)
Top-K (baseline)
55%
87%
Llama 3.1 8B Instruct 85.97%
Llama $.2 1B Instruct 45.266%
45%
85% 1o
25
30
35
20
30
40
Average Accepted Tokens
Average Accepted Tokens
Figure 9: Downstream accuracy and the average number of accepted tokens for GSM8K 8-shot with Llama-3.1 3.B- target and Llama-3.2 1a-maa-.--B-IIstrruu I-Inttrucct draft models (left) and Llama-3.1-70B Instruct target and Llama-3.1 1.--a-..--B--- -.B---rtsttt draft models (right)
27

---

85%-
18%
Autolodge, GSMNK Heed
LCB Pass@: (correctly solved rate)
Artolaign, LCB
80%
Aunlldlg, LCBI Hea
Autalalg; GSMRK
GSM8K Accuracy (lest set)
Top-K (beszlins)
Top (buadiac)
75%
Spor. Decsding
Deculing
Spon. GSMSS, Acouacy
14%
Spec. Decoling LCB Accuracy
70%
65%
10%
60%
55%
6%
50%
45%-
2%
40%
10
20
30
40
50
60
10
20
30
40
50
60
Avg. Avg Accepted Tokens
Avg. Accepted Tokens
Figure 10: Downstream accuracy and the average number of accepted tokens for GSM8K (left) and LiveCodeBench (right) with Llama-3.1 1.1...1.88 1-8B-Instruct target and --amma3.2-IB-Instteeee In-strucct draft models.
Llama 3.1 8B Instrued 16.81%
34%
17%
32%
Llama 3.1 70B Instruct 30.90%
Pass@1 (correetly solved rate)
15%
Pass (correetly solved rate)
30%
Spec. Decoding
13%
28%
AutoJudge (ours)
Top-K (baseline)
11%
26%
24%
9%
22%
7%
Spec. Decoding
20%
5%
AutoJudge (ours)
18%
Top-K (baseline)
Llama 3.1 8B Instruct 16. 16.25%.
3%-
Llama 3.2 IB Instruct 3.07%
16%
30
50
60
30
50
Average Accepted Tokens
Average Accepted Tokens
Figure 11: Downstream accuracy and the average number of accepted tokens for LiveCodeBench with L I-8B-Instruct target and Llama-3.2-1 B--nstuttttt draft models (left) and 1lama-3.1-70B- Instruct target and --ama-3.1-B--Insse draft models (right)
D.2 Additional Evaluations for Sections 4.2
In this section, we provide additional classifier threshold evaluations for both model pairs in our LiveCodeBench setup. These results reported in Figure II with IB draft / 8B target models the left and 8B draft / 70B target on the right. We use the same setup as in Section 4.2 and the portion of the results at the top exactly the same values that reported in Figure 5
Additionally, we evaluate the AutoJudge classifier trained on LiveCodeBench on GSM8K and vice versa to gauge the effect of task-specific training. We repost results for Llama-3.2-1D draft / Llama-3.1 1-BB target models pair in Figure T0 Predictably, these out-of-domain classifiers perform significantly worse. We attribute this to the fact that the GSM8K-traine classifier likely did not any Python source code, whereas the LiveCodeBench classifier did not perform arithmetic operations and did not solve equations that are common in GSM8K.
E vLLM Inference Implementation Details
To measure time efficiency of AutoJudge, we incorporate into vLLLM inference library. The integra- tion is built upon v1lm==0.8 8.5, torch==2. 7. with CUDA 12.8 and transf c .1.3
28

---

We use window size of 32 and a batch size of 1. The implementation allows to perform batched inference without modifications. but choose evaluate with batch size 1 for accurate measure- ments. For efficiency, the current implementation predicts important tokens using only the hidden state of the target model, as we have shown that adding hidden states of the draft model does not substantially increase the accuracy. We run evaluations for the 1B/8B model pair and for the 8B/70B pair on and NVIDIA A A100-SXM4-800
GPUs respectively and on 8 NVIDIA H100-SXM5 80GB GPUs for 8B/405B in FP8 precision in the tensor parallel setup.
F Additional Inference Benchmarks: Equal Window Size
In this section, report additional runtime benchmarks that extend our evaluations from Section 4.3 For these experiments, we use a fixed window size 32 for both vanilla speculative decoding and AutoJudge (as opposed individually tuned window size in Section 4.3
We report our results for GSM8K with IB/8B model pair Table 3 8B/70B Table 4 and 1B/70B
in Table 5 All three model pairs show significant speedups relative to standard speculative decoding. Notably, our 8B/70B setup has up to 2.5x with about 2.5% loss accuracy.
Table 3: Inference speed benchmarks GSM8K for IB draft / 8B target model on: (left) GSM8K O-shot evaluation and (right) GSM8K 8-shot evaluation.
Llama 3.2 IB draft / 3.1 8B target (0-shot)
Llama 3.2 IB draft / 3.1 8B target (8-shot)
Threshold
0.04
0.09
0.12
0.16
Threshold
0.050
0.087
0.098
0. 0.133
Accuracy, %
83.9
82.0
80.2
77.4
Accuracy, % Speed, tokens/s
85.7
85.1
84.5
83.1
Speed, tokens/s
89.1
106.8
121.0
146.2
66.5
74.3
76.3
83.3
Speculative Decoding:
84.5 tokens/s
Speculative Decoding:
62.4 tokens/s
Speedup(ours) 1.05
1.26
1.43
1.72
Speedup(ours)
1.07
1.19
1.22
1.33
Table 4: Inference speed benchmarks GSM8K for 8B draft / 70B target model on: (left) GSM8K 0-shot evaluation and (right) GSM8K 8-shot evaluation.
Llama 3.1 8B draft / 3.1 70B target (O-shot)
Llama 3.1 8B draft / 3.1 70B target (8 (8-shot)
Threshold
0.005
0.031
0.145
0.230
Threshold
0.03
0.05
0.18
0.28
Accuracy, %
92.0
91.9
89.9
88.0
Accuracy: Accuracy, % 95.4
94.8
92.9
90.4
Speed, tokens/s
41.8
80.6
107.4
109.5
Speed, tokens/s
64.4
71.1
79.3
86.2
Speculative Decoding:
40.6 tokens/s
Speculative Decoding:
40.5 tokens/s
Speedup(ours)
1.03
1.98
2.64
2.70
Speedup(ours)
1.58
1.74
1.94
2.11
Table 5: Inference speed benchmarks GSM8K 8-shot with vLLM implementation for IB draft 70B target models.
Llama 3.2 IB draft / 3.1 70B target (8-shot)
Threshold
0.01
0.03
0.05
0.11
Accuracy, %
95.1
95.3
94.6
92.3
Speed, tokens/s
50.1
65.6
75.5
79.9
Speculative Decoding:
45.7 tokens/s
Speedup(ours)
1.10
1.44
1.65
1.75
29

---

G Evaluation with Individually Tuned Window Sizes
As we discussed in Section we tune the speculation window size individually for AutoJudge and vanilla speculative decoding provide speculative decoding accepts, on average, less tokens and does not benefit from having a larger draft window size. Thus, the two algorithms often work best with different draft sizes.
competitive baseline. This is because traditional
In this section, we account for this by evaluating AutoJudge and speculative decoding with optimal window sizes for every model pair. We consider window sizes between 6 and 64 with the following values: [6, 8, 10, 12, 14, 16, 20, 24, 26, 28, 32, 40, 48, 54, 64] and choose the best window size in terms of tokens per second. Note that this evaluation protocol suffers from the natural variance in latency (e.g. due to varying individual clock rates) since we report the highest value measured.
The results for 1B/8B models pair are reported in Table 6 The results for 8B/70B pair are reported in Table Overall, AutoJudge decoding still significantly outperforms standard speculative decoding.
Table 6: (left) Inference speed benchmarks on GSM8K O-shot with vLLMM implementation for 1B draft / 8B target models with tuned window size (Spec. Decoding tokens, AutoJudge 10 tokens) and (right) for IB draft 8B target models on GSM8K 8-shot with tuned window size (Spec. Decoding tokens, Autojudge 10 tokens).
Llama 3.2 IB draft 3.1 8B target (0-shot)
Llama 3.2 IB draft / 3.1 8B target (8-shot)
Threshold
0.06
0.12
0.15
0.16
Threshold
0.050
0.098
0.133
0.164
Accuracy, %
83.1
80.2
79.8
77.4
Accuracy, % Speed, tokens/s
85.7
84.5
83.1
81.2
Speed, tokens/s
149.2
169.2
171.2
173.9
114.4
125.0
132.1
139.3
Speculative Decoding:
147.7 tokens/s
Speculative Decoding:
116.8 tokens/s
Speedup(ours)
1.01
1.14
1.15
1.17
Speedup(ours)
0.98
1.07
1.13
1.19
Table 7: (Ieft) Inference speed benchmarks on GSM8K 0-shot with vLLM implementation for draft / 70B target models with tuned window size (Spec. Decoding = 8 tokens, Autojudge = 32 tokens) and (right) for draft 70B target models on GSM8K 8-shot with tuned window size (Spec. Decoding 8 tokens, Autojudge = 16 tokens).
Llama 3.1 8B draft / 3.1 70B target (0-shot)
Llama 3.1 8B draft / 3.1 70B target (8 (8-shot)
Threshold
0.005
0.031
0.145
0.230
Threshold
0.03
0.05
0.18
0.28
Accuracy, %
92.0
91.9
89.9
88.0
Accuracy, %
95.4
94.8
92.9
90.4
Speed, tokens/s
72.3
80.6
107.4
109.6
Speed, tokens/s
69.0
73.1
83.6
84.5
Speculative Decoding:
72.3 tokens/s
Speculative Decoding:
57.3 tokens/s
Speedup(ours)
1.0
1.11
1.49
1.52
Speedup(ours)
1.20
1.27
1.45
1.47
H
Offloading
As presented in Section 4 AutoJudge accept up 40 tokens per verification cycle, which makes it naturally well-suited for scenarios with a large draft window. In offloading setups, the drafting step is significantly cheaper than verification, yet speculative decoding suffers from being able to accept only a few tokens on average. AutoJudge avoids this limitation by accepting substantially more tokens per verification cycle, leading notable gains in offloading configurations.
As reported in Table g the accuracy drop with AutoJudge does not exceed 3% across thresholds, while achieving throughput between 1.4 and 2.4 tokens/s, compared to 1.19 tokens/s for speculative
30

---

Table 8: Inference speed benchmarks GSM8K 8-shot for 8B draft / 70B target models with offloading on a single NVIDIA A100-SXM4 GPU with tuned window size (Spec. Decoding 10 tokens, AutoJudge 48 tokens).
Llama 3.1 8B draft 3.1 70B target (8-shot)
Threshold
0.03
0.05
0.11
0.28
Accuracy, %
95.4
94.8
93.4
90.4
peed, tokens/s
1.4
1.6
1.9
2.4
Speculative Decoding:
1.19 tokens/s
Speedup(ours)
1.20
1.31
1.59
1.98
decoding. This corresponds relative speedups of x It turns out that the optimal window size for AutoJudge 48 tokens, whereas for speculative decoding it is only
I On the Instability of vLLM Inference for Accuracy Benchmarks
In Section we evaluate AutoJudge decoding in terms of the number of accepted tokens per phase and the real-world tokens per second. For convenience, we only use vLLM implementation to evaluate number of tokens per second and report all other metrics based on transformers pytorch implementation. This is because, in our preliminary experiments, we observed discrepancies between benchmark accuracy with vL..M inference and PyTorch. Upon further investigation, we found that vLLM inference sometimes produces inconsistent results. Namely, if we run vLLM greedy inference in bfloat16 precision with speculative decoding, changing "technical" hyperparameters such as window size can significantly affect the accuracy. To illustrate this, measure GSM8K accuracy 132 random test samples in the same setup as Section 4.3) We use standard vLLM implementation of speculative decoding Leviathan et al. 2023 without any modificatiins the vLLM codebase.
Table Llama 3.2 1B draft / 3.1 70B target, 10% random test samples from GSM8K. Accuracy is measured based on vLLM generations with varied window size.
Window Size
Accuracy, %
8
93,9
10
93.9
93.9
14
93.9
16
95,4
20
95,4
24
94.6
26
94.6
The results in Table iillustrate the observed inconsistency: changing window size can affect the accuracy to a significant degree. This coincides with concurrent observations about inconsistent LLM inference from Yuan et al. [2055] and He and Lab 2025). To make measurements more consistent, we report all accuracy and accepted token rates using more stable implementation based on Hugging Face Transformers (included in our repository), only using vLLM to measure the speed in terms of tokens/second.
J EAGLE experiments
Following the invention speculative decoding, several lines of work proposed follow-up algorithms that incorporate trained speculation "heads" (Cai i et al. 2024 Li et al. 2024d tree decoding MMao
jt al., 2023 Chen al., 2024 SSvirchevski et al. [2024] and many other improvements. In this section, we explore how AutoJudge generalizes to these more advanced decoding algorithms.
31

---

To that end, we integrate AutoJudge with the popular EAGLE-2 algorithm [Li et al. 2024c]. Unlike the original speculative decoding, EAGLE-2 does not have a separate draft model, but trains lightweight "head" to predict future tokens from target model hidden states. This allows EAGLE to draft tokens much faster, albeit less accurately than powerful standalone draft model.
We evaluate Llama 3.1 8B Instruct using the official pre-trained EAGLE head Since there is no separate draft model, the classifier is trained we only use target model hidden states when training, Ad- ditionally, since EAGLE draft model was not trained to produce long coherent drafts, we use a shorter draft window size of We integrate AutoJudge with the official EAGLE implementation and set parameters that are compatible with the vLLM EAGLE implementation bogthhwindow_sizee Aside from that, we use the same evaluation protocol as in Section[4.]]ee GSM8K.
In Figure 12 we report GSM8K accuracy and the average number of accepted tokens for AutoJudge with the official PyTorch implementationfff EAGLE 2. We also report real-world inference speed (tokens per second) using the vLLMM implementation on a single A 1100-80G GPU. The results suggests that integrating AutoJudge with EAGLE can produce additional speedups on top of the highly efficient speculative decoding algorithm.
Llama 3.1 8B Instruct + EAGLE head (0-shot)
90%
80%
Antolodge
Threshold
0.05
0.15
0.30
0.40
Agcuracy
Basclinc
70%
Accuracy, % Speed, tokens/s
81.4
81.3
81.0
78.1
60%
50%
91.1
96.8
102.6
107.5
40%
EAGLE:
89.8 tokens/s
2.5
3.0
3.5
4.0
4.5
5.0
Speedup(ours)
1.01
1.08
1.14
1.20
Avg. Accepted Tokens
Figure 12: Evaluating AutoJudge with EAGLE-2 draft head on GSM8K using Llama 3.1 8B Instruct model. (Left) Accuracy to accepted tokens in PyTorch, (right) vLLM inference speed on A A100-80GB.
K
Rule-Based Approaches Ablation
Table 10: Comparison between AutoJudge and simple rule-based heuristic focused mathematical tokens, GSM8K O-shot, Llama 3.2 1B Instruct draft / 3.1 8B Instruct target.
Method
Math Only
Math & Top-1024
Math & Top-128
Heuristic criterion
Accuracy, %
65.05
72.27
80.36 15.5
Accepted Tokens
39.7
22.3
AutoJudge (nearest threshold)
Accuracy, %
57.8
75.6
81.0
Accepted Tokens
37.4
22.3
15.33
To explore how AutoJudge decoding compares to rule-based methods for mathematical reasoning, compare it against two heuristic alternatives. The first heuristic approach is to consider only the mathematical symbol tokens as 'important". To that end, we filter the mismatching tokens that contain numbers, operations (e.g., */ /, etc.), well in these mathematical tokens are rejected, whereas mismatches non-mathematical tokens are allowed. As it turns out (Table this algorithm misses important planning and logical steps that do not contain computations explicitly, which results in poor accuracy. To address this, we introduce second, more complex heuristic approach that combines the mathematical rule with the Top-K baseline we use in Section 4. This works somewhat better, but still does not outperform the learned AutoJudge classifier.
common variables. Mismatches
1- I--nnsuucttttt
32

---

L Open-Ended Generation with LLM-as-a-Judge
As we discuss earlier in Section 3.1 the choice of important tokens largely depends on what counts as an "equivalent answer quality". For mathematical reasoning, we can check whether the alternative response leads to the same numerical answer (up to notation) or an equivalent formula. For programming. we compare how the two programs behave in testing. However, not all LLM tasks have formal quality criteria. Open-ended tasks like creative writing and question answering have implicit quality criteria that are difficult to evaluate. In this section, we investigate how AutoJudge generalizes to two open-ended problems: generative question answering and creative writing.
Case A: Question Answering, For this task, we evaluate Llama 3.x LLMs generative question answering on questions from the TriviaQA dataset (Joshi et al. 2017]. We use the "closed book" setup, where the LLM receives only the question itself as the prompt, without any additional information (i.e. no search results). Then, we use more powerful LLM-as-an j--juge to compare responses against target model outputs. We mine important tokens on 500 training samples and evaluate on subset of 100 validation samples. Since this dataset was intended for short answers (typically sentence), we stop generation on \n or after generating 120 tokens.
Case B: Creative Writing. We use the "Creative Writing" subset of Hace--Hard-Auto-v.... [Li itt al. 202 that contains 250 creative writing tasks sourced from Chatbot Arena. These tasks include requests write a poem on certain topic, an imaginary dialogue transcript, or similar, in several languages. These tasks are even harder to judge than question answering, often boiling down to personal preference. We set 100 queries aside for evaluations and use the rest to mine important tokens, using the standard generation prompt from Arena- -
LLM-as-a-judge- We evaluate generations using pairwise LLM-as-a-judge protocol inspired by Arena-Hard-Auto-v2 Under this protocol, the LLM "judge" does not test model answers against a pre-defined "correct" response, but compares two generations against each other. For our speculative decoding setup, we ask the L.LM judge to compare target model generation against the output of speculative decoding with AutoJudge head. When comparing two generations, the LLM judge sees both responses A and B and chooses one of 5 options:
1. Assistant is significantly better: [[A>>B]]
2. Assistant A is slightly better: [[A>B]]
3. Tie, relatively the same: [[A=B]]
4. Assistant B is slightly better: [[B>A]]
5. Assistant B is significantly better: [[>>>A]
When evaluating on creative writing, we found that the default LLM judge often produces different
answers between consecutive runs, both with sampling and greedy decoding, likely due numer- ical instability (Yuan et al. 2025) ). To make our results more reliable we switch to a stronger cl 4-22et-----222-1111 judge model and run it times with majority voting:
If there are more votes [[A>>B]] & [[A>B]] than the opposite, then A is better.
If there are more votes [[B>>A] & [[B>A]] than the opposite, then B is better.
If there are equal number of votes favoring A and B or all votes are [[A=B]], we declare A and B are equivalent. Note that this rule does not differentiate between [[A>B]] and [[A>B]] as we found them to be uninformative.
Mining important tokens. We mine important tokens using Algorithm with one change: in L12 (if' a = à then ...). we ask the LLM judge to compare the alternative response ğ (with the draft token) against y. If the new response is rated strictly worse, we label the corresponding draft token as important and roll back to the target token. Otherwise (better or equal), we keep the draft token and label it unimportant, same as in the original algorithm. Note that the LLM judge compares not against the original target model response (L4 Alg. J but against the current running response y that contains unimportant draft tokens from previous iterations (L) This change would have no effect for tasks with formal However, for LL-- audgee this results in more accurate labels and improves classifier accuracy. We only use this for training, not evaluation.
â criteria from math and programming benchmarks due to transitivity.
33

---

95%
Targel rate on Creative Writing set)
Targel rate on TriviaQA(va)) sel)
90%
Spee. Decoding
60%
Spee. Decoding
AutoJudge (ours)
55%
AutoJudge (ours)
85%
Top-K (baseline)
50%
Top-K (bascline)
80%
45%
75%
40%
70%-
35%
30%
65%-
25%
60%-
20%
X 55%
15%
50%
10%
10
15
20
25
30
10
12
14
16
18
20
22
24
Average Accepted Tokens
Average Accepted Tokens
Figure 13: AutoJudge performance on open-ended generation with L.LM-as-a-judge evaluation. (Left) closed-book generative question answering on TriviaQA questions (Right) creative writing on Arena-hard- Autrd--2 queries. Both use Llama-3.2-1B draft Llama-3.1-80 target models (Instruct). Creative writing was later found to have noisy labels (>70% mismatches judged non-equivalent).
Results. We report our evaluations on question answering and creative writing in Figure 13] left and right, respectively. Similarly to Sections (.1 and 4.2 we evaluate AutoJudge against the top-K baseline in terms of i) the average number of accepted tokens for greedy decoding and ii) the rate which our outputs
equal better than the target model. We the Claude Sonnet judge and best of 3 voting as specified above. Note that the lossless decoding results are not exactly 0.5 quality because
response pairs were deemed equal, i.e. [[A=B]]-
The results suggest that AutoJudge classifier outperforms traditional speculative decoding on question answering, but not creative writing. Upon closer inspection, we found that the creative writing benchmark has a very high rate important tokens - over 70% mismatching tokens result in non-equivalent answers, much larger than all other problems. We attribute this anomaly to creative writing being inherently subjective, resulting in noisy LLM judgements. With such high rate of important tokens, there is little room for speed up with AutoJudge. To summarize, we found that Autoludge can be indeed be used for open-ended problems with the L.L--as-a-uudge paradigm, outperforming the baseline in some but not all cases.
M Trained Classifier Transfer Between Tasks
M.1 From GSM8K to MATH-hard Subset
To evaluate the task transfer capability of AutoJudge, we test whether a classifier trained on one mathematical reasoning dataset can generalize different, more challenging dataset. For this evaluation, we use the AutoJudge classifier trained on GSM8K 0-shot with the Llama-3.22 2-1I-tnstttcct draft / Llama-3.1 1.1.3..--88---eee- -8B-Instruct target model pair (from Section 4.1) and apply it to the MATH-har -ard subset from the LLama Codebook based on the Im-evaluation-hanne 2021]. The MATH-hard dataset contains significantly more challenging mathematical problems compared to GSM8K, requiring more advanced mathematical reasoning. We follow the standard evaluation protocol from the Im-evaluation-han harness framework and report exact match accuracy as the downstream metric.
benchmark [Gao et al.
The results are presented in Figure 14 (left). While AutoJudge can still accept more tokens than standard speculative decoding, the accuracy-speed trade-off's are less favorable than the top-K baseline across most operating points. This suggests that the patterns of important tokens learned from grade- school math problems (GSM8K) might not transfer well to more advanced math reasoning tasks.
We reproduce the setup and get the problems from the official Meta evaluations repository
34

---

M.2 From GSM8K to Long Context
To AutoJudge on cong-contett tasks, we construct a synthetic long- cong-context mathematical reasoning benchmark based on GSM8K. For each test problem, we concatenate 250 randomly sampled GSM8K questions (without their solutions) as context, followed by a single target question that the model must solve. The resulting prompts contain approximately 8-10K tokens before applying the chat template, significantly longer than the standard GSM8K setup.
We evaluate the Llama-3, 1.2na--..-111 2-11 IB-nnsrruct draft Llama-3. 1-1--B.1111 1-8B-Instruct target model pair using the AutoJudge classifier trained on hiddens mined described long context GSM problems. The results presented in Figure 14 (right), AutoJudge maintains its ability to accept more tokens than standard speculative decoding this long-context regime, achieving better accuracy while maintaining similar number of accepted tokens compared to the Top-K baseline.
These results demonstrate that AutoJudge classifiers trained on standard- length examples general- ize to significantly longer contexts, suggesting that the notion of "important tokens" for mathematical reasoning remains consistent regardless of the input length.
Llama 3.1 8B Instruct 25.00%
Llama a 3.1 8B Instrucl 92. 18%
0.24
0.-
Test Aceurayy
Test Accuracy
0.0-
07-
Spec. Decoding AutaJudge (ours)
Spec. Decoding
0.14
AutoJudge (ours)
Top-K (baseline)
Top-K (baseline)
Llama 3.2 IB Instruct 53.90%
Llama 3.2 1B Instruct 10.49%
20
30
60
20
30
50
60
Average Accepted Tokens
Average Accepted Tokens
Figure 14: (Left) Task transfer: accuracy accepted tokens on MATH-hard using GSM8K-trained classifier. (Right) Long-contens performance: accuracy vs. accepted tokens on GSMSK with 8- 8-10K token prompts. Both use Llama-3.2-1B draft Llama-3. 1-1BB target setup (Instruct).
N
Manual Annotation
To compare our automated approach with the manual annotation procedure from Judge Decoding (Bachmann et al. (2025), we conduct an experiment following their methodology as closely as possible. Unfortunately, the original paper does not provide public access to their annotated dataset source code, which limits direct comparison. Therefore, we recreate their data collection and training procedure based on the methodology described in Section 4.1 of their paper.
Dataset construction. We curate a dataset of 167 high-quality questions from two sources: the ARC-Challent Challenge benchmark Clark et al. (2018) (25 questions) and the Alpaca dataset Taori et al. 2023) (142 questions, split between 71 mathematical reasoning problems and 71 coding problems). Since the exact question indices used in the original Judge Decoding work are not publicly available, we manually selected challenging and diverse questions from each dataset that represent a range of difficulty levels similar to those described in the original paper.
For each question, we generate responses using three different models from the Llama family: Llama-3.22 1 Instrrctt, Llama-3. 3.1.....--BBB Instuuns and 1i-ma-..--70--Instiii This produces 501 total generated responses with varying quality levels, providing a diverse set correct and incorrect continuations for training.
35

---

Llama 3.1 8B Instruct 83.47%
0.8
Spec. Decoding
AutoJudge on manual data
Top-K (baseline)
0.7
Test Accuracy
0.6
0.5
Llama 3.2
1B Instruct 43.36%
10
20
30
40
50
60
Average Accepted Tokens
Figure 15: GSM8K O-shot evaluation of Llama-3.2-1B draft / Llama-3. 1--8B target model pair comparing AutoJudge with manual annotation mining (following [Bachmann et al. 2025) against Top-K baseline.
Manual annotation. Following the procedure described in "Dataset Curation" " in Section 4.1 of Bachhann et al., 2025), we manually annotate each generated response identify the precise model to
location where errors occur. We encode all responses using the 1aama-3.1-8B-Inseee extract hidden state representations, then manually label each token either correct incorrect based on whether accepting that token would lead to a wrong final answer. This annotation process required approximately 8-10 hours careful manual review by the authors.
Training and evaluation. Using these manually annotated hidden states, we train AutoJudge classifier with the same architecture and hyperparameters described in Section and Appendix B We then evaluate the resulting "manuall--anotaaedd" AutoJudge classifier on GSMKK (-shot with the Llama-3.2- 1 draft Llama-3. I-8B-Instruct target setup, comparing against Top-K baseline. The results are presented in Figure T5 We observe that the manually-annotatee classifier performs substantially worse than Top-K baseline and, consecutively, our automatically-minn approach. The heanuall--annotaie classifier achieves lower acceptance rates while exhibiting greater accuracy degradation, suggesting that manual annotation is both less effective and more labor-intensive. We attribute this gap to several factors: (1) human annotators may inconsistently label edge cases or fail to account for how subsequent tokens interact, (2) the manual process is prone to errors over hundreds of annotations, and (3) our automated search procedure (Algorithm [ systematically tests token combinations in a way that better reflects actual inference conditions. These results validate that our automated mining procedure not only eliminates the substantial human effort required for manual annotation (8-10 hours in our case), but also produces more robust training data that leads to better classifier performance.
36

---

Correlation between Draft Model and AutoJuuge Classifier Probabilities
One reasonable heuristic for speculative decoding is to consider a token important if a model is certain in its generation, measured in terms of that token's probability. Here, we check whether the AutoJudge classifier is similar to that heuristic. To that end, we measure the correlation between the Llama 3.2 IB Instruct draft model's probability of the chosen draft token and the corresponding AutoJudge classifier probability on the IB/8B model pair using the GSM8K dataset (O-shot setting). The resulting Pearson correlation varies within approximately +0.3 per generation, and the full-sample covariance 0.073. This suggests that AutoJudge classifier does not act on that heuristic.
P Hardware Configurations
We run our experiments primarily on A100-SXM4 GPUs with 80GB DRAM servers with dual Epyc 7742 CPU and ITiB RAM. For 8B/405B model pair we used 8 NVIDIA H100-SXM5 80GB GPUs.
For model pairs that do not fit on single GPU, we use distributed inference with naive model parallelism srtvvice_ap="aattte" when using transformers and tensor parallelism for vLLM experiments.
The actual time per experiment varies by the dataset and model pair: 1B draft / 8B target model pair takes, on average, 65.6 seconds to process a GSM8K example, whereas the 8B draft 70B target takes up an average of 706.4 seconds per sample. On LiveCodeBench, the same 8B draft 70B target model takes up 449 seconds per sample. Since Algorithm T can run independently for each sample, we were able to run our code on low-priority preemptible hardware. However, this also makes hard to measure the exact amount of computations used in our experiments since some of them were lost due to preemption. For running on a single A100/H100 server, please refer to the time per sample above to estimate the total runtime requirements. Please also note that there are ways to mine important tokens more efficiently using APIs (below).
Power consumption
To estimate the energy consumption of AutoJudge, vanilla speculative decoding, and sequential decoding, we run each inference method on a 10% sample of the GSM8K test set using vLLM. We measure real-world power usage on Llama 3.2 1B draft / 3.1 8B target models with a single 100-SXM4-80GG GPU (Watts) as reported by nvidia-smi, and multiply it by the mean inference time. This represents the GPU-reported power consumption, before adjusting for PSU inefficiency (equally for AutoJudge and baselines) and will vary between GPU types. For convenience, we convert all results to kJ, similar Maliakel et
2025
Table 11: Estimated energy consumption of running inference on 10% of GSM8K test set.
Method
Autoregressive
Speculative Decoding
AutoJudge
Power Usage (kJ)
87
43
37
R
Scaling Up Algorithm I with API Calls
We would also like to mention that it is possible to scale up the important token discovery in AutoJudge by reframing it in terms of API calls. Note that the search algorithm only ever runs regular generation (greedy or sampling) with the target model and runs parallel forward pass on Odraft-
Hence, we can run Algorithm by replacing GENERATE(...) on lines 4 and 10 with call to an LLM generation API with the specified input. With this reframing, we can mine important tokens by querying LLM API providers even if they cannot inference the large target model locally. This can help in a number of use cases: for instance, when developing speculative decoding algorithm for use with offloading (Svirschevski al. 2024
Miao et al. 2023
37

---

Additionally, modern open-source inference libraries for LLMs often expose OpenAl-compatible API. This way, one can run the important token mining algorithm efficiently over pool of LLM inference servers, taking advantage of server-side batching and optimized kernels.
Note, however, that this regime requires tokenizing and detokenizing the target model's messages received from API calls, since most public services operate on non-tokenized text strings. Since there several ways spell the same text with given BPE merge table, this can sometimes lead to unintentional "token healing".
In repository, provide variant of the important token mining algorithm that leverages Opemalecompatible inference API to run the target model. This enables training the AutoJudge classifier even models that cannot be hosted locally. As demonstration, experiment using Llama 3.1 -.1-400B-Instructt (accessed via API) as the target model and Llama 3.1- 8B-Instruct the mining model. This example illustrates that Algorithm i applicable to models available exclusively through an API. Results obtained for this model pair are shown in Table (left).
conducted
Dataset and Model Licenses
In this section, we summarize our use of licensed models and datasets.
The GSM8K benchmark [Cobbe et al.
2021
is licensed under the MIT License;
The LiveCodeBench benchmark [Jain et al. 2024] is licensed under the MIT License;
Llama 3.1 and 3.2 models Dubey et al.
2024 are under the Llama Community License Agreement.
Our work would also be impossible without open-source software including (but not limited to) PyTorch (Paszke et al. 2019]. Hugging Face Transformers (Wolf et al. 2019), vLLMM [Kwon et al. 2023) and hundreds of other open-source packages in the Python data science & machine learning ecosystem. Enumerating and acknowledging all these individual packages would take up several pages, but they be recovered automatically from repository.
38


Reviews:
[{"review_id": "u6cM7eofSQ", "reviewer": "Reviewer_vcrD", "summary": "LLM inference is expensive. This method proposes a classifier that detects which tokens are important in the context of speculative decoding of LLM generations. If a token produced by speculative decoding doesn't exactly match the original one, the classifier can still tag it as \"unimportant\" and we can continue the speculative decoding without significantly affecting the generation quality. This is evaluated in relevant LLM benchmarks, showing improved decoding speed at the expense of small accuracy degradation.", "questions": "\"a semi-greedy search that starts from the target model response and iteratively tries to replace mismatching target model tokens with their draft counterparts\": does the method go token-by-token? Did you consider any heuristic to skipping tokens?\n\nThe way of extracting the final answer is task-specific. However, once the classifier has been already trained, would it work for other tasks?", "limitations": "The main limitation of this work is already acknowledged by the authors. An additional one I observe is that there aren't many benchmarks (or models) represented in the results, which might pose some doubts on whether this would work in other cases.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "### Strenghts\n\n- The method is novel and well-principled.\n- Automatically constructs training data.\n- The empirical results are compelling.\n- The application (speeding up LLM decoding in real-world tasks without degrading accuracy) is relevant. The integration with vLLM is useful and proof that authors take real-world applications seriously. This integration, if released, increases the likelihood that the method will be used.\n- The design choices are well-motivated. For example, \"2. Which token alternative to use?\" in line 196.\n\n### Weaknesses\n\n- The task-specificity of the method might limit its applications. Plus, as acknowledged by the authors, this method requires having a pre-determined way to extract the final answer, which might not work in less \"templatic\" scenarios.\n- Relatively small novelty with respect to Bachmann.", "was_revised": true, "final_justification": "I have updated the \"Significance\" results given the additional results provided by the authors, which show some potential cross-task generalization and results for additional tasks. The rebuttal is compelling and I recommend acceptance, and that they include the additional results in the camera ready version.\n\nHowever, my original global score was already strong, a 5, and while I do think it's a high-quality paper, I don't think it's enough for updating my global score to 6.", "rebuttal": "Thank you for your valuable feedback and insightful questions! Please find our responses below. For convenience, we answer the questions in the order they are raised in the review.\n\n> \"a semi-greedy search that starts from the target model response and iteratively tries to replace mismatching target model tokens with their draft counterparts\": does the method go token-by-token? Did you consider any heuristic to skipping tokens?\n\nAlgorithm 1 does indeed go token-by-token, but it only considers the “mismatching” tokens (L7 in Algorithm 1). These are tokens where the draft and target models would predict different next tokens given the same (current) prefix. Across different models and benchmarks, we found that only one in 10-20 generated tokens is a mismatch. This is because many of the tokens can be unambiguously predicted from the prefix (e.g. following grammar / punctuation rules or repeating an earlier sentence). Another contributing factor is that the draft and target models come from the same family (both Llama in the main paper, Qwen below) in the sense that they were trained on the same data and predict similar next tokens.\n\nIn a setup with substantially more mismatches (e.g. weak draft model), one way to speed up Algorithm 1 is to prune obviously bad draft tokens early. This could be implemented, for instance, by forbidding any draft tokens that have very low probability according to the target model (e.g. <0.001).\n\n> The way of extracting the final answer is task-specific. However, once the classifier has been already trained, would it work for other tasks?\n\nThe answer would likely depend on how similar this new task is. If the classifier was trained on GSM8K with no code in the training set, it will not generalize to coding problems. However, it might be able to generalize to other problems that involve mathematical reasoning.\n\nTo quantify this, we evaluate the AutoJudge classifier trained on the GSM8K dataset (from Section 4.1) on a subset of the MATH benchmark (256 problems from math_hard) using the protocol from the official “Llama-3.1-8B-Instruct-evals” dataset by Meta.\n\nWhile both benchmarks involve mathematical reasoning, the problems in MATH_hard are harder in the sense that they require longer chains of thought. Furthermore, the answer to these problems is a mathematical expression, whereas GSM8K problems have an integer answer.\n\nWe use Llama 3.1-8B Instruct as the target model and Llama 3.2-1B Instruct as a draft model. Since the NeurIPS author response rules prohibit us from uploading images, we reformat our results as a table, reporting the best results in every bracket of accepted token range.\n\n### MATH_hard (256 samples) accuracy and accepted tokens (cross-domain evaluation)\n\n|Accepted token bracket| <25 | 25-30 | 30-35 | >35 |\n|-|-|-|-|-|\n|AutoJudge (trained on GSM8K)|||||\n|Accuracy, %|17.6|16.4|14.1|12.1|\n|Accepted Tokens|24.8|28.16|33.89|36.72|\n|Top-K baseline|||||\n|Accuracy, %|15.5|15.2|11.3|9.8|\n|Accepted Tokens|24.36|28.24|32.59|37.77|\n\nWhile the advantage is not as high as on the task that the classifier was trained on, this suggests that AutoJudge can indeed generalize between these two similar tasks. We agree that this is an important evaluation scenario and will add these results and explore additional cross-task generalization further in the final version of the paper.\n\n> The main limitation of this work is already acknowledged by the authors. An additional one I observe is that there aren't many benchmarks (or models) represented in the results, which might pose some doubts on whether this would work in other cases.\n\nTo alleviate this concern, we report additional model evaluations and extra benchmarks in addition to the MATH_hard evaluation above. To see how our algorithm generalizes beyond the Llama model family, we evaluate AutoJudge with the Qwen 2.5 Instruct model family, with the 0.5B draft model and 7B target model. These evaluations follow the same protocol as in Section 4.1, except that we use the recommended system prompts for Qwen 2.5 models from the official technical report [1]. The results are summarized in the table below.\n\n|Accepted token bracket| <9 | 9-13 | 13-25 | >25 |\n|-|-|-|-|-|\n|AutoJudge (trained on GSM8K)|||||\n|Accuracy, %|87.4|87.6|83.2|79.8|\n|Accepted Tokens|8.95|12.30|24.00|27.53|\n|Top-K baseline|||||\n|Accuracy, %|86.2|82.0|52.8|46.8|\n|Accepted Tokens|8.96|9.13|21.95|27.30|\n\n\nThe results show similar speed-ups (up to 2.68x more accepted tokens) with minor accuracy drawdowns. We will include this and additional experiments from the Qwen model family in the final version of the paper. To further explore alternative use cases, we combine AutoJudge with the EAGLE decoding algorithm (see response to Reviewer DSqA). \n\nIf you have any further questions or suggestions, we are happy to address them in the Reviewer-Author discussion phase.\n\n[1] Qwen Team (Yang et al.), Qwen2.5 Technical Report, arXiv:2412.15115, 2024a."}, {"review_id": "plywqemX72", "reviewer": "Reviewer_znPD", "summary": "The paper proposes AutoJudge, a fully automated protocol for task-specific acceleration of speculative decoding. AutoJudge identifies tokens on which (small) surrogate and (large) target LLMs disagree and characterizes if that diagreement is critical or not. The paper proposes to reframe the inference from detecting individual important tokens to finding combinations of tokens that jointly affect the final answer, using a trained classifier (based on logistic regression). The evaluation shows 1) ~2x speedup over speculative decoding at the cost of under 1% drop in accuracy; 2) the improvement of AutoJudge over Top-k inference in terms of accuracy and run-time for both mathematical reasoning and programming tasks.", "questions": "None", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* Given the emergence of speculative LLM decoding, finding ways to improve correct answers and reduce the LLM inference time. The paper automates several steps in a recent strategy Judge Decoding (from ICLR'25). \n\n* The proposed approach presents promising initial results. The comparison on two standard datasets shows the improvement over the Top-K baseline (which was used in the ICLR'25 paper too). \n\n* The evaluation shows that the design of AutoJudge could generalize across different tasks. I encourage the authors to continue investigating this question. \n\nWeaknesses:\n\n* No comparison with JudgeDecoding and/or generalization to other advanced speculative decoding algorithms (as the authors mentioned in the Future work paragraph). The current baseline is weak, even though the initial results are encouraging. Having stronger alternative baselines would lead to a better comparison to state-of-the-art in this nascent problem domain. While I understand that the full comparison with Judge Decoding may not be possible without their annotations, can you re-create a smaller set of annotations based on that paper and run a study with those?\n\n* The proposed classifier design, a logistic regression, is rather simple and it is unclear how its training generalize for different tasks. Without comparing to Judge Decoding or other methods, it is unclear whether it yields sufficient accuracy. Further, some details from Appendix B would be useful to move to the main paper to motivate the model's use. Overall, a study of different families of classifiers would help understand this problem better. \n\n* (minor) Some of the existing results can be improved too: (1) The current results are on only several pairs of LLMs. More combinations would give a better picture of the tradeoff space between the overheads and model's accuracy. The results presented in the appendix (i.e., other sizes of Llamas) are good, but it would be even better if the impact of those (and some other combinations of small open source LLMs) on accuracy and run time are presented in a separate figure int eh main paper.  (2) In addition to the time speedups, include the estimates of consumed energy while running the target and surrogate models in parallel. The work is acceptable even if energy consumption is larger than a single model, but characterizing the tradeoff makes the paper stronger.", "was_revised": true, "final_justification": "I am satisfied with the authors' detailed response to my questions.", "rebuttal": "Thank you for your detailed review and suggestions for improving our work! To address these suggestions, we conducted additional experiments, reporting their results below.\n\n> The evaluation shows that the design of AutoJudge could generalize across different tasks. I encourage the authors to continue investigating this question.\n\nWe agree that cross-task generalization is an interesting direction. While we don’t expect AutoJudge to generalize between completely unrelated tasks, it is curious how it generalizes to adjacent task types. To that end, we evaluate the AutoJudge classifier trained on the GSM8K dataset (Section 4.1) on a subset of the MATH benchmark (math_hard from Llama evals). While both benchmarks involve mathematical reasoning, GSM8K has integer answers, whereas MATH_hard is answered with mathematical formulae and has generally harder problems.\n\nWe follow the evaluation protocol from the official “Llama-3.1-8B-Instruct-evals” by Meta. We use and Llama 3.1-8B Instruct as the target model and Llama 3.2-1B Instruct as a draft model. Since the author response rules prohibit us from uploading images, we reformat our results as a table, reporting the best results in every bracket of accepted tokens bracket.\n\n### MATH_hard (256 samples) accuracy and accepted tokens (cross-domain evaluation)\n\n|Accepted token bracket|<25|25-30|30-35|>35|\n|-|-|-|-|-|\n|AutoJudge (trained on GSM8K)|||||\n|Accuracy, %|17.6|16.4|14.1|12.1|\n|Accepted Tokens|24.8|28.16|33.89|36.72|\n|Top-K baseline|||||\n|Accuracy, %|15.5|15.2|11.3|9.8|\n|Accepted Tokens|24.36|28.24|32.59|37.77|\n\nWhile the improvement is not as high as on the dataset that the classifier was trained on, the results suggest that AutoJudge can indeed generalize between these two similar tasks. We agree that this is an important evaluation scenario and will add these results and explore additional cross-task generalization further in the final version of the paper.\n\n> No comparison with JudgeDecoding and/or generalization to other advanced speculative decoding algorithms (as the authors mentioned in the Future work paragraph).\n\nWhile we originally intended this as future work, we agree that testing AutoJudge with more advanced speculative decoding algorithms would be beneficial. To that end, we compare AutoJudge with two such advanced algorithms: EAGLE-2 and offloaded speculative decoding.\n### AutoJudge with EAGLE-2\n\nOur first advanced algorithm is EAGLE-2 [1,2] that trains a draft ‘head’ on top of target model hidden representations to predict future states. We run our experiments using both the official EAGLE codebase (for training, accuracy, and accepted tokens) and its vLLM integration (for tokens/s), using the hyperparameters that match the behavior of the vLLM version (i.e., depth = total_tokens-1).\n\nBelow, we report our evaluation results on the GSM8K dataset in the same evaluation setup as in Section 4.1, except with EAGLE instead of vanilla speculative decoding and a smaller window size of 8. This is because EAGLE was trained with shorter drafts and becomes incoherent after drafting 8-10 tokens. To comply with the author response rules, we format our results as a table.\n\n|Method \\ Tokens Bracket|2.5-3|3-3.5|3.5-4|4-5|5+|\n|-|-|-|-|-|-|\n|AutoJudge (threshold)|0.01|0.08|0.5|0.8|0.9|\n|Accuracy, %|82.34|80.36|73.77|60.12|46.70|\n|Accepted Tokens|2.872|3.128|3.656|4.458|5.119|\n|Baseline (nearest bracket)|||||\n|Accuracy, %|82.18|74.68|63.08|45.11|29.95|\n|Accepted Tokens|2.718|3.018|3.615|4.462|5.248|\n\nAdditionally, we report vLLM inference speed (tokens per second) for a single A100-SXM4 GPU and relative speed-ups for the same thresholds as above. The speed-ups are relative to Speculative Decoding (150.5 tokens/s), similar to how they are reported in Table 1.\n\n|AutoJudge Threshold|0.01|0.08|0.5|0.8|0.9|\n|-|-|-|-|-|-|\n|Accuracy, %|82.34|80.36|73.77|60.12|46.70|\n|Speed, tokens/s|158.8|170.5|195.7|221.0|225.5|\n|Speedup vs. Spec. Decoding|1.05x|1.13x|1.30x|1.47x|1.5x|\n\nThe results suggest that AutoJudge decoding can indeed generalize to this more advanced speculative decoding algorithm.\n\n### AutoJudge with Offloaded Speculative Decoding\n\nAnother advanced use case for speculative decoding is when the target model does not fit on the device. Recent works that target this scenario tend to generate significantly longer drafts [3,4]. This is because running the offloaded target model takes significantly longer, regardless of how many draft tokens it needs to verify.\n\nFor consistency, we use the same GSM8K benchmark with llama 3.1 8B draft / 3.1 70B target models as in the original Table 1 (right) from Section 4.3. The main purpose of this evaluation is to compare speedups in this more advanced technical scenario. We evaluate on a single A100-SXM4 GPU, which would be unable to run this without offloading. We evaluate AutoJudge with a threshold of 0.26 (see Table 1 in Section 4.3), and report the average inference speed (tokens per second) for different window sizes below.\n\n|Method \\ Draft Length |8|12|16|32|64|\n|-|-|-|-|-|-|\n|AutoJudge (offloading) (toks/s)|0.54|0.69|0.81|0.78|0.70|\n|Leviathan (offloading) (toks/s)|0.35|0.35|0.36|0.36|0.31|\n|Speedup|1.54x|1.97x|2.25x|2.17x|2.26x|\n\n> While I understand that the full comparison with Judge Decoding may not be possible without their annotations, can you re-create a smaller set of annotations based on that paper and run a study with those?\n\nWhile it would be difficult, we generally agree that it would be valuable to compare AutoJudge automated annotation against human-labelled data. However, unlike our previous experiments that only required integrating AutoJudge into existing benchmarks, human annotation requires a longer timeframe than the duration of the Author Response phase. Hence, we will only be able to conduct this experiment after the discussion to include it in the final version of the paper.\n\n> The proposed classifier design, a logistic regression, is rather simple and it is unclear how its training generalize for different tasks. Further, some details from Appendix B would be useful to move to the main paper to motivate the model's use. Overall, a study of different families of classifiers would help understand this problem better.\n\nWe explore several alternative classifier designs and feature sets in Appendix B (summarized and referenced in S3.2) and find that a simple linear model is sufficient. One possible explanation for this is that the linear classifier has access to the feature representations of the target LLM. This coincides with prior findings from Judge Decoding that also use a linear model. If the paper is accepted, we will be happy to describe these results in the main paper using the additional content page.\n\n> The current results are on only several pairs of LLMs. More combinations would give a better picture of the tradeoff space between the overheads and model's accuracy.\n\nTo alleviate this concern, we report results with another model pair, but this time from a different family: Qwen 2.5 Instruct models. We use Qwen 2.5 0.5B Instruct as the draft and 7B Instruct target model and summarize our results below.\n\n|Accepted token bracket| <9 | 9-13 |13-25 |>25|\n|-|-|-|-|-|\n|AutoJudge (trained on GSM8K)|||||\n|Accuracy, % |87.4|87.6|83.2|79.8|\n|Accepted Tokens|8.95|12.30|24.00|27.53|\n|Top-K baseline|||||\n|Accuracy, % |86.2|82.0|52.8|46.8|\n|Accepted Tokens|8.96|9.13|21.95|27.30|\n\nThe results show similar speed-ups (up to 2.68x more accepted tokens) with minor accuracy drawdowns. We will include this and additional experiments from the Qwen model family in the final version of the paper.\n\n> In addition to the time speedups, include the estimates of consumed energy while running the target and surrogate models in parallel. The work is acceptable even if energy consumption is larger than a single model, but characterizing the tradeoff makes the paper stronger.\n\nTo estimate the energy consumption of AutoJudge, vanilla speculative decoding, and sequential decoding, we run each inference method on a 10% sample of the GSM8K test set using vLLM. We measure real-world power usage on Llama 3.2 1B draft / 3.1 8B target models with a single A100-SXM4-80GB GPU (Watts) as reported by nvidia-smi, and multiply it by the mean inference time. This represents the GPU-reported power consumption, before adjusting for PSU inefficiency (equally for AutoJudge and baselines) and will vary between GPU types. For convenience, we convert all results to kJ, similar to [5].\n\n|Method|Autoregressive|Speculative Decoding|AutoJudge|\n|-|-|-|-|\n|Power Usage|87kJ|43kJ|37kJ|\n\nNote that both AutoJudge and vanilla speculative decoding turned out to be more energy efficient than simple autoregressive decoding. We attribute this to the fact that LLM inference power consumption is affected not just by ‘FLOPs’, but also by memory reads/writes to load the LLM parameters. Since the autoregressive decoding requires more sequential operations to generate a given sequence, it has a higher total GPU energy consumption per task.\n\nIn summary, we believe we addressed the main concerns raised in your review with additional experiments and clarifications. If our response answers your questions, we would kindly ask you to reevaluate your score for the submissions. If you have further suggestions, we are happy to address them during the Reviewer-Author Discussion phase.\n\n[1] Li et al. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. ICML 2024.\n\n[2] Li et al. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. EMNLP 2024\n\n[3] Svirschevski et al. SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices. NeurIPS 2024.\n\n[4] Miao et al. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. ASPLOS '24\n\n[5] Maliakel et al. Investigating Energy Efficiency and Performance Trade-offs in LLM Inference Across Tasks and DVFS Settings, arXiv:2501.08219"}, {"review_id": "x1n6CpUrMa", "reviewer": "Reviewer_DSqA", "summary": "This paper introduces AutoJudge, a lossy speculative decoding framework that leverages unimportant token classification for boosting draft model acceptance length. To accomplish this, AutoJudge identifies sets of tokens that don't alter a verifiable result and train a linear classifier with intermediate representations as inputs. Evaluation on two combinations of Llama-3 models on math and programming showcase 1.4 - 2x speedup at 1pp of accuracy degradation.", "questions": "* How does the proposed method perform on top of different speculative decoding methods, like EAGLE [a,b,c] or HASS [d] or multi-token prediction [e]?\n* Are there any bounds in the distribution shifts that can happen in the lossy AutoJudge approach?\n* How does the draft model confidence correlate with the classification module's output?\n* Is the semi-greedy search ordered somehow (e.g. left-to-right) or do replacements happen randomly?\n\n[a] Li, Y., Wei, F., Zhang, C., & Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In Forty-first International Conference on Machine Learning.  \n[b] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024, November). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 7421-7432).  \n[c] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2025). Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840.  \n[d] Zhang, L., Wang, X., Huang, Y., & Xu, R. (2024). Learning Harmonized Representations for Speculative Sampling. arXiv preprint arXiv:2408.15766.  \n[e] Mehra, S., Garcia, J. A., & Mauch, L. (2025). On multi-token prediction for efficient LLM inference. arXiv preprint arXiv:2502.09419.", "limitations": "The method's main limitation is its applicability on verifiable tasks. While the authors touch upon interesting topics of judge evaluation for non-verifiable tasks, this is left as future work.\n\nAdditionally, while the authors state that their method can be implemented on top of different speculative decoding schemes, this has not been shown or evaluated.\n\nLast, testing the limits of the method's generalisability on long-context tasks would have been an welcome and relevant addition in the evaluation.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* The paper is nicely motivated and tackles a very timely and important problem with a practical solution that is more scalable than existent baselines.\n* The paper showcases the generalisability of the trained classifier across different tasks.\n* The method has been integrated with vLLM and shows significant improvements over the selected baselines.\n\n### Weaknesses\n\n* The paper has only assessed a single family of models (Llama-3) on two verifiable tasks. It would have been interesting to see additional speculative decoding baselines and how autojudge can be applied on top of such setups.\n* It is unclear how the method would perform under large context sizes and different decoding strategies.\n* The overhead of the classifier module has not been quantified.", "was_revised": true, "final_justification": "I am happy with the paper and the rebuttal, which did not affect my score.", "rebuttal": "Thank you for your feedback! Below, we answer the questions raised in your review and provide additional experiment results to address your concerns.\n\n> The paper has only assessed a single family of models (Llama-3) on two verifiable tasks.\n\nWe agree that our analysis would benefit from exploring additional model families and tasks. To that end, we explore an extra model family of Qwen-2.5. Below, we report a pair of Qwen2.5-0.5B-Instruct draft model and Qwen2.5-7B-Instruct target model on GSM8K benchmark using the same evaluation setup as in Section 4.1. Due to the restrictions for this year’s author response (no images), we had to reformat our results as a table.\n\n|Accepted token bracket| <9 | 9-13 | 13-25 | >25 |\n|-|-|-|-|-|\n|AutoJudge (trained on GSM8K)|||||\n|Accuracy, % |87.4|87.6|83.2|79.8|\n|Accepted Tokens|8.95|12.30|24.00|27.53|\n|Top-K baseline|||||\n|Accuracy, %|86.2|82.0|52.8|46.8|\n|Accepted Tokens|8.96|9.13|21.95|27.30|\n\nThe results suggest that AutoJudge decoding offers significantly better accuracy to speed (token acceptance rate) trade-offs than the Top-K criterion. To further explore different tasks, we also evaluate on a subset of the MATH benchmark in our response to Reviewer vcrD. We will explore additional Qwen model pairs and tasks in the final version of the paper.\n\n> It would have been interesting to see additional speculative decoding baselines and how autojudge can be applied on top of such setups. … How does the proposed method perform on top of different speculative decoding methods, like EAGLE [a,b,c] or HASS [d] or multi-token prediction [e]?\n\nTo address this, we evaluate and compare AutoJudge with EAGLE-2 [a,b], a popular speculative decoding algorithm that trains a predictive draft ‘head’ instead of using a separate draft model. We evaluate Llama 3.1-8B Instruct model using the “EAGLE-LLaMA3.1-Instruct-8B” draft head published with the original paper [a]. We use the official PyTorch implementation to report accuracy and accepted tokens, and run vLLM EAGLE integration for real-world inference speed (tokens per second) with the same configuration as in Section 4.3. We configure the hyperparameters to match the behavior of the vLLM model (i.e. depth = total_tokens - 1).\n\n\n### AutoJudge with EAGLE, GSM8K (0-shot) accuracy and inference speed\n\n|Method \\ Tokens Bracket|2.5-3|3-3.5|3.5-4|4-5|5+|\n|-|-|-|-|-|-|\n|AutoJudge (threshold)|0.01|0.08|0.5|0.8|0.9|\n|Accuracy, %|82.34|80.36|73.77|60.12|46.70|\n|Accepted Tokens|2.872|3.128|3.656|4.458|5.119|\n|Baseline (nearest bracket)|||||\n|Accuracy, %|82.18|74.68|63.08|45.11|29.95|\n|Accepted Tokens|2.718|3.018|3.615|4.462|5.248|\n\nWe also report vLLM inference speed (tokens/s) on one A100-SXM4 GPU and the relative speed-up compared to Speculative Decoding (150.5 tokens/s). For this evaluation, we use the same 5 thresholds as above and otherwise follow the same protocol as in Section 4.3.\n\n|AutoJudge Threshold|0.01|0.08|0.5|0.8|0.9|\n|-|-|-|-|-|-|\n|Accuracy, %|82.34|80.36|73.77|60.12|46.70|\n|Speed, tokens/s|158.8|170.5|195.7|221.0|225.5|\n|Speedup vs. Speculative Decoding|1.05x|1.13x|1.30x|1.47x|1.5x|\n\nThe results above correspond to AutoJudge decoding with EAGLE draft model on the GSM8K dataset using the same setup as in Section 4.1, but with EAGLE draft model and a smaller window size of 8 since the EAGLE draft ‘head’ was not trained to generate longer drafts. We use the same window size and other hyperparameters for both AutoJudge classifier training and inference. The results suggest that AutoJudge can generalize to at least one significantly different speculative decoding setup.\n\n> It is unclear how the method would perform under large context sizes and different decoding strategies.\n\nIt would indeed be interesting to explore alternative decoding strategies. Note that some of our benchmarks (e.g., LiveCodeBench in the main paper) already require long-form reasoning with thousands of generated tokens, though there are tasks with significantly longer inputs. Due to the limited time for author responses, we prioritized your other suggestions, but we will explore more long-context tasks in the final version of the paper.\n\n> The overhead of the classifier module has not been quantified.\n\nSince we claim that the overhead of using a linear classifier is negligible (L194), we agree that it would be best to more formally quantify the overhead. Compute-wise, the classifier head can be merged with the existing LM “head” (L212) of draft and target models to reduce its overhead. In that case, the actual multiplication is done in the same kernel call as the next token prediction. However, using the predicted classifier outputs does incur a minor overhead.\n\nWe evaluate the runtime overhead of running the classifier on a A100-SXM4 80GB GPU in the same setup as in Section 4.3. We found that the compute overhead is within 0.3% even without merging (0.026ms classifier vs 6.992ms 8B model forward). Likewise, the memory overhead of storing the classifier weights is within 10^-5 of the target model weights (12KiB for 1B/8B pair and 24KiB for 8B/70B. The model weights take up tens of GiB).\n\n> Are there any bounds in the distribution shifts that can happen in the lossy AutoJudge approach?\n\n\nThe natural upper bound on the token distribution shift is the shift between draft and target models *on the tokens deemed important by the classifier*. Establishing a tighter guarantee on the distribution shift would be difficult due to the complex nature of LLMs. However, if you would like to explore this problem from a specific angle, we are happy to discuss this further.\n\n> How does the draft model confidence correlate with the classification module's output?\n\nThank you for suggesting this sanity check. When measuring this, we found that the instruct-tuned draft models are often overconfident, and the probability of the chosen draft token in these models is not well correlated with that token’s importance to the downstream task. More specifically, we measured the correlation between the Llama 3.2 1B Instruct draft model’s **probability of the chosen draft token** and the corresponding AutoJudge classifier probability on the 1B/8B model pair on the GSM8K dataset (0-shot). The resulting Pearson correlation varies within ±0.3 per generation, and the full sample covariance is -0.073.\n\n> Is the semi-greedy search ordered somehow (e.g. left-to-right) or do replacements happen randomly?\n\nIn Algorithm 1, we order the search from left to right, as you mentioned (line number 9 in the algorithm). This is because every time we consider a token, there is a chance that we will need to replace it and *re-generate all subsequent tokens* given the new prefix. Hence, changing an earlier token can invalidate the mismatches found in subsequent tokens. To avoid this, AutoJudge does indeed explore tokens left-to-right. We will state this more clearly in Section 3.1.\n\nWe hope that these new results address the concerns raised in your review. In the final version of the paper, we plan to use the extra space to further explore the directions you suggested, as detailed above."}, {"review_id": "eHFZ3xZQzp", "reviewer": "Reviewer_QVNr", "summary": "This paper proposes AutoJudge, a lossy speculative decoding framework that learns to selectively verify only the tokens deemed “important” for final answer quality. The key idea is to avoid full token-level validation by training a lightweight classifier to identify which mismatches matter. The method involves a semi-greedy search over tokens to label “importance,” trains a classifier on these, and integrates it into the speculative decoding pipeline. Experiments on GSM8K and LiveCodeBench show promising 2× speed-ups with minor quality degradation.", "questions": "See Weakness", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "I think the paper tackles a valuable and timely problem — making speculative decoding more efficient — and it introduces a relatively clean and automatable way to identify \"important\" mismatches. I also appreciate that the proposed method integrates smoothly into existing decoding systems and shows solid improvements on two benchmarks.\n\nI have several concerns:\n- I think the baselines chosen for comparison are somewhat weak. Speculative decoding is a very active area, and there are stronger follow-ups to the original algorithm that are not considered here. This makes it harder to gauge the real benefit of AutoJudge.\n- More critically, I feel that the experimental setup — focusing only on GSM8K and code generation — somewhat hides the underlying simplicity of the classification problem. These domains have very rigid output formats and answer structures, so the classifier might just be learning dataset-specific patterns. In fact, I suspect that a simple rule-based filter could likely perform similarly within-domain. This makes me question the generality and transferability of the approach, especially for more open-ended or subjective generation tasks.\n-  I also noticed that there’s no proper ablation on the classifier itself. Since the core claim is about learning to identify important tokens, I would have expected comparisons with alternative classifiers, feature sets, or even rule-based methods to understand what’s really driving performance.", "was_revised": true, "final_justification": "I have read the author rebuttal carefully and reviewed the other reviews and responses as well.\n\nThe concerns I initially raise, such as classifier ablations, and the possibility of overfitting to rigid tasks, were directly addressed with new results (e.g., evaluation on Qwen models, rule-based comparisons, and additional cross-dataset tests like GSM8K → MATH_hard). These updates alleviate my major reservations.\n\nThat said, I maintain a lower confidence score, as I am not deeply familiar with the latest developments in speculative decoding. In particular, I cannot fully assess whether the chosen baselines (e.g., Top-K) are sufficient, or whether comparison with more recent methods is critical for a fair evaluation.\n\nI raised my score accordingly (3 -> 4).", "rebuttal": "Thank you for your feedback and suggestions! We are glad that you appreciated the importance of more efficient speculative decoding and how our work contributes to improving it, as well as the simplicity of integrating AutoJudge into inference pipelines. Below, we discuss the concerns and suggestions you raised in the review, providing additional experiments to address them.\n\n> Speculative decoding is a very active area, and there are stronger follow-ups to the original algorithm that are not considered here. This makes it harder to gauge the real benefit of AutoJudge.\n\nWe agree that considering additional speculative decoding algorithms and setups would greatly strengthen the paper. To explore this direction, we integrated AutoJudge with the popular EAGLE-2 algorithm [1,2]. Unlike the original speculative decoding, EAGLE-2 does not have a separate draft model, but trains a lightweight “head” to predict future tokens, which makes it one of the few speculative algorithms fit for batched inference.\n\nFor AutoJudge decoding, we only use the target model hidden states (during verification) when training the classifier, but otherwise follow the same pipeline as in Section 4.1 for Llama 3.1 8B target model with two exceptions: 1) using EAGLE draft head instead of a separate draft model 2) using a shorter draft window size of 8. The shorter window size is because the EAGLE draft ‘head’ is not designed as a standalone model and does not generate long coherent text, as it is designed for speed rather than accuracy.\n\nBelow, we report GSM8K accuracy and the average number of accepted tokens when integrating AutoJudge with the official PyTorch implementation of EAGLE 2. We also report real-world inference speed (tokens per second) using the vLLM implementation on a single A100-SXM4-80GB GPU. Since PyTorch and vLLM implementations are slightly different, we set PyTorch hyperparameters to match the parameters used in vLLM integration (depth = total tokens - 1).\n\n|Method \\ Tokens Bracket|2.5-3|3-3.5|3.5-4|4-5|5+|\n|-|-|-|-|-|-|\n|AutoJudge (threshold)|0.01|0.08|0.5|0.8|0.9|\n|Accuracy, %|82.34|80.36|73.77|60.12|46.70|\n|Accepted Tokens|2.872|3.128|3.656|4.458|5.119|\n|Baseline (nearest bracket)|||||\n|Accuracy, %|82.18|74.68|63.08|45.11|29.95|\n|Accepted Tokens|2.718|3.018|3.615|4.462|5.248|\n\nNext, we measure the real-world inference speed (tokens per second) with vLLM on a single A100-SXM4-80GB GPU. Similarly to Section 4.3, we also report relative speed-ups compared to standard Speculative Decoding (150.5 tokens/s).\n\n|AutoJudge Threshold|0.01|0.08|0.5|0.8|0.9|\n|-|-|-|-|-|-|\n|Accuracy, %|82.34|80.36|73.77|60.12|46.70|\n|Speed, tokens/s|158.8|170.5|195.7|221.0|225.5|\n|Speedup vs. Speculative Decoding|1.05x|1.13x|1.30x|1.47x|1.5x|\n\nThe results suggest that AutoJudge can generalize to advanced follow-ups to speculative decoding algorithms. However, we agree that further exploration of various successors of Speculative Decoding would further improve our work. We will include these and additional experiments with speculative decoding follow-ups in the final version of the paper.\n\n> More critically, I feel that the experimental setup — focusing only on GSM8K and code generation — somewhat hides the underlying simplicity of the classification problem. These domains have very rigid output formats and answer structures, so the classifier might just be learning dataset-specific patterns. This makes me question the generality and transferability of the approach, especially for more open-ended or subjective generation tasks.\n\n\nThank you for your suggestion! For GSM8K and coding problems, it is indeed tempting to try simpler rule-based alternatives to AutoJudge (as you suggest later in the review), and we do that later in our response. We also agree that it would be interesting to evaluate AutoJudge with open-ended problems such as creative writing. In L135-137, we briefly address this scenario by describing how AutoJudge can detect important tokens in these problems using an LLM-as-a-Judge approach to compare answer quality. Due to the limited time of the author response phase, we prioritized your other suggestions, but we will explore AutoJudge for open-ended tasks in the final version of the paper.\n\n> I also noticed that there’s no proper ablation on the classifier itself. Since the core claim is about learning to identify important tokens, I would have expected comparisons with alternative classifiers, feature sets, or even rule-based methods to understand what’s really driving performance.\n\nWe perform the ablation of the AutoJudge classifier in Appendix B (L974 in supplementary material, referred to in L214). Overall, Figures 5 and 6 suggest that using specifically the target token hidden representations (as opposed to the previous representations that predicted said token) yields substantial benefits to classifier AUC. Using both draft and target hidden representations further yields a smaller (but essentially free) improvement. In turn, using “deeper” classifiers (e.g., MLP, RandomForest) led to no improvements and increased overfitting. We also consider alternative feature sets in Figure 6 (left).\n\n> (continued) … rule-based methods to understand what’s really driving performance.\n\nTo explore how AutoJudge decoding compares to rule-based methods for mathematical reasoning, we compare it against two heuristic alternatives. The first heuristic approach is to consider only the mathematical tokens as ‘important’. To that end, we filter the mismatching tokens that contain numbers, operations (e.g., + - * / =, etc), as well as some common variables. Mismatches in these mathematical tokens are rejected, whereas mismatches in non-mathematical text are allowed.\n\nAs it turns out, this algorithm misses important planning and logical steps that do not contain math explicitly, which results in poor accuracy. To address this, we introduce a second, more complex heuristic approach that combines the mathematical rule with the Top-K baseline we use in Section 4. This works somewhat better, but still does not outperform the learned AutoJudge classifier.\n\n### GSM8K (0-shot) rules-based methods, 1B draft / 8B target models\n\n\n|Heuristic method|Math Only|Math & Top-1024|Math & Top-128 |\n|-|-|-|-|\n|Accuracy, %| 65.05 | 72.27 | 80.36 |\n|Accepted Tokens| 39.7 | 22.3 | 15.5 |\n|AutoJudge (nearest)||||\n|Accuracy, %| - | 79.37 | 84.15 |\n|Accepted Tokens| - | 22.7 | 15.7 |\n\n\nTo summarize, we believe that we have discussed the concerns raised in the review and conducted additional experiments following your suggestions. We respectfully ask you to reevaluate your original score based on these additional results. If we missed something, please inform us in the follow-up response, so we can address it in the final version of the paper.\n\n\n[1] Li, Y., Wei, F., Zhang, C., & Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In Forty-first International Conference on Machine Learning.\n\n[2] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024, November). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 7421-7432)."}]

Author rebuttal:

Meta-review: Despite some limitations for the proposed approach, the paper proposes a novel, automatable, and practical solution to improving speculative decoding. Its demonstrated integration with real-world systems (vLLM), competitive speedups, and clean framing that  make it a valuable contribution to the efficient inference community.
The limitations are acknowledged and partially addressed during the rebuttal phase. While the method is simple in design, it is effective, and its simplicity enhances scalability and ease of deployment. I recommend acceptance
Decision: Accept (poster)