Title: AutoJudge: Judge Decoding Without Manual Annotation
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

Speculative Decoding

Lossy Speculative Decoding

Inference
Abstract: We introduce AutoJudge, a method that accelerates large language model (LLM) inference with task-specific lossy speculative decoding. 
Instead of matching the original model output distribution token-by-token, we identify the generated tokens that affect the downstream quality of the response, relaxing the distribution match guarantee so that the "unimportant" tokens can be generated faster.
Our approach relies on a semi‑greedy search algorithm to test which of the mismatches between target and draft models should be corrected to preserve quality and which ones may be skipped.
We then train a lightweight classifier based on existing LLM embeddings to predict, at inference time, which mismatching tokens can be safely accepted without compromising the final answer quality.
We evaluate AutoJudge with multiple draft/target model pairs on mathematical reasoning and programming benchmarks, achieving significant speedups at the cost of a minor accuracy reduction. 
Notably, on GSM8K with the Llama 3.1 70B target model, our approach achieves up to ${\approx}2{\times}$ speedup \textit{over speculative decoding} at the cost of a ${\le} 1\%$ drop in accuracy.
When applied to the LiveCodeBench benchmark, AutoJudge automatically detects programming-specific important tokens, accepting ${\ge}25$ tokens per speculation cycle at a$~ 2\%$ drop in Pass@1. Our approach requires no human annotation and is easy to integrate with modern LLM inference frameworks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AutoJudge, a lossy speculative decoding framework that learns to selectively verify only the tokens deemed “important” for final answer quality. The key idea is to avoid full token-level validation by training a lightweight classifier to identify which mismatches matter. The method involves a semi-greedy search over tokens to label “importance,” trains a classifier on these, and integrates it into the speculative decoding pipeline. Experiments on GSM8K and LiveCodeBench show promising 2× speed-ups with minor quality degradation.\n\nSTRENGTHS AND WEAKNESSES:\nI think the paper tackles a valuable and timely problem — making speculative decoding more efficient — and it introduces a relatively clean and automatable way to identify \"important\" mismatches. I also appreciate that the proposed method integrates smoothly into existing decoding systems and shows solid improvements on two benchmarks.\n\nI have several concerns:\n- I think the baselines chosen for comparison are somewhat weak. Speculative decoding is a very active area, and there are stronger follow-ups to the original algorithm that are not considered here. This makes it harder to gauge the real benefit of AutoJudge.\n- More critically, I feel that the experimental setup — focusing only on GSM8K and code generation — somewhat hides the underlying simplicity of the classification problem. These domains have very rigid output formats and answer structures, so the classifier might just be learning dataset-specific patterns. In fact, I suspect that a simple rule-based filter could likely perform similarly within-domain. This makes me question the generality and transferability of the approach, especially for more open-ended or subjective generation tasks.\n-  I also noticed that there’s no proper ablation on the classifier itself. Since the core claim is about learning to identify important tokens, I would have expected comparisons with alternative classifiers, feature sets, or even rule-based methods to understand what’s really driving performance.\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AutoJudge, a lossy speculative decoding framework that leverages unimportant token classification for boosting draft model acceptance length. To accomplish this, AutoJudge identifies sets of tokens that don't alter a verifiable result and train a linear classifier with intermediate representations as inputs. Evaluation on two combinations of Llama-3 models on math and programming showcase 1.4 - 2x speedup at 1pp of accuracy degradation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* The paper is nicely motivated and tackles a very timely and important problem with a practical solution that is more scalable than existent baselines.\n* The paper showcases the generalisability of the trained classifier across different tasks.\n* The method has been integrated with vLLM and shows significant improvements over the selected baselines.\n\n### Weaknesses\n\n* The paper has only assessed a single family of models (Llama-3) on two verifiable tasks. It would have been interesting to see additional speculative decoding baselines and how autojudge can be applied on top of such setups.\n* It is unclear how the method would perform under large context sizes and different decoding strategies.\n* The overhead of the classifier module has not been quantified.\n\nQUESTIONS:\n* How does the proposed method perform on top of different speculative decoding methods, like EAGLE [a,b,c] or HASS [d] or multi-token prediction [e]?\n* Are there any bounds in the distribution shifts that can happen in the lossy AutoJudge approach?\n* How does the draft model confidence correlate with the classification module's output?\n* Is the semi-greedy search ordered somehow (e.g. left-to-right) or do replacements happen randomly?\n\n[a] Li, Y., Wei, F., Zhang, C., & Zhang, H. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In Forty-first International Conference on Machine Learning.  \n[b] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2024, November). EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 7421-7432).  \n[c] Li, Y., Wei, F., Zhang, C., & Zhang, H. (2025). Eagle-3: Scaling up inference acceleration of large language models via training-time test. arXiv preprint arXiv:2503.01840.  \n[d] Zhang, L., Wang, X., Huang, Y., & Xu, R. (2024). Learning Harmonized Representations for Speculative Sampling. arXiv preprint arXiv:2408.15766.  \n[e] Mehra, S., Garcia, J. A., & Mauch, L. (2025). On multi-token prediction for efficient LLM inference. arXiv preprint arXiv:2502.09419.\n\nLIMITATIONS:\nThe method's main limitation is its applicability on verifiable tasks. While the authors touch upon interesting topics of judge evaluation for non-verifiable tasks, this is left as future work.\n\nAdditionally, while the authors state that their method can be implemented on top of different speculative decoding schemes, this has not been shown or evaluated.\n\nLast, testing the limits of the method's generalisability on long-context tasks would have been an welcome and relevant addition in the evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes AutoJudge, a fully automated protocol for task-specific acceleration of speculative decoding. AutoJudge identifies tokens on which (small) surrogate and (large) target LLMs disagree and characterizes if that diagreement is critical or not. The paper proposes to reframe the inference from detecting individual important tokens to finding combinations of tokens that jointly affect the final answer, using a trained classifier (based on logistic regression). The evaluation shows 1) ~2x speedup over speculative decoding at the cost of under 1% drop in accuracy; 2) the improvement of AutoJudge over Top-k inference in terms of accuracy and run-time for both mathematical reasoning and programming tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* Given the emergence of speculative LLM decoding, finding ways to improve correct answers and reduce the LLM inference time. The paper automates several steps in a recent strategy Judge Decoding (from ICLR'25). \n\n* The proposed approach presents promising initial results. The comparison on two standard datasets shows the improvement over the Top-K baseline (which was used in the ICLR'25 paper too). \n\n* The evaluation shows that the design of AutoJudge could generalize across different tasks. I encourage the authors to continue investigating this question. \n\nWeaknesses:\n\n* No comparison with JudgeDecoding and/or generalization to other advanced speculative decoding algorithms (as the authors mentioned in the Future work paragraph). The current baseline is weak, even though the initial results are encouraging. Having stronger alternative baselines would lead to a better comparison to state-of-the-art in this nascent problem domain. While I understand that the full comparison with Judge Decoding may not be possible without their annotations, can you re-create a smaller set of annotations based on that paper and run a study with those?\n\n* The proposed classifier design, a logistic regression, is rather simple and it is unclear how its training generalize for different tasks. Without comparing to Judge Decoding or other methods, it is unclear whether it yields sufficient accuracy. Further, some details from Appendix B would be useful to move to the main paper to motivate the model's use. Overall, a study of different families of classifiers would help understand this problem better. \n\n* (minor) Some of the existing results can be improved too: (1) The current results are on only several pairs of LLMs. More combinations would give a better picture of the tradeoff space between the overheads and model's accuracy. The results presented in the appendix (i.e., other sizes of Llamas) are good, but it would be even better if the impact of those (and some other combinations of small open source LLMs) on accuracy and run time are presented in a separate figure int eh main paper.  (2) In addition to the time speedups, include the estimates of consumed energy while running the target and surrogate models in parallel. The work is acceptable even if energy consumption is larger than a single model, but characterizing the tradeoff makes the paper stronger.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLLM inference is expensive. This method proposes a classifier that detects which tokens are important in the context of speculative decoding of LLM generations. If a token produced by speculative decoding doesn't exactly match the original one, the classifier can still tag it as \"unimportant\" and we can continue the speculative decoding without significantly affecting the generation quality. This is evaluated in relevant LLM benchmarks, showing improved decoding speed at the expense of small accuracy degradation.\n\nSTRENGTHS AND WEAKNESSES:\n### Strenghts\n\n- The method is novel and well-principled.\n- Automatically constructs training data.\n- The empirical results are compelling.\n- The application (speeding up LLM decoding in real-world tasks without degrading accuracy) is relevant. The integration with vLLM is useful and proof that authors take real-world applications seriously. This integration, if released, increases the likelihood that the method will be used.\n- The design choices are well-motivated. For example, \"2. Which token alternative to use?\" in line 196.\n\n### Weaknesses\n\n- The task-specificity of the method might limit its applications. Plus, as acknowledged by the authors, this method requires having a pre-determined way to extract the final answer, which might not work in less \"templatic\" scenarios.\n- Relatively small novelty with respect to Bachmann.\n\nQUESTIONS:\n\"a semi-greedy search that starts from the target model response and iteratively tries to replace mismatching target model tokens with their draft counterparts\": does the method go token-by-token? Did you consider any heuristic to skipping tokens?\n\nThe way of extracting the final answer is task-specific. However, once the classifier has been already trained, would it work for other tasks?\n\nLIMITATIONS:\nThe main limitation of this work is already acknowledged by the authors. An additional one I observe is that there aren't many benchmarks (or models) represented in the results, which might pose some doubts on whether this would work in other cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-06"}