Title: SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
Venue: NeurIPS 2025
Accepted: yes
Keywords: speculative decoding

LLM agents

model-free speculation

SWE-Bench

LLM inference
Abstract: Speculative decoding is widely adopted to reduce latency in large language model (LLM) inference by leveraging smaller draft models capable of handling diverse user tasks. However, emerging AI applications, such as LLM-based agents, present unique workload characteristics: instead of diverse independent requests, agentic frameworks typically submit repetitive inference requests, such as multi-agent pipelines performing similar subtasks or self-refinement loops iteratively enhancing outputs. These workloads result in long and highly predictable sequences, which current speculative decoding methods do not effectively exploit. To address this gap, we introduce \emph{SuffixDecoding}, a novel method that utilizes efficient suffix trees to cache long token sequences from prompts and previous outputs. By adaptively speculating more tokens when acceptance likelihood is high and fewer when it is low, SuffixDecoding effectively exploits opportunities for longer speculations while conserving computation when those opportunities are limited. Evaluations on agentic benchmarks, including SWE-Bench and Text-to-SQL, demonstrate that SuffixDecoding achieves speedups of up to 3.9$\times$, outperforming state-of-the-art methods -- 2.2$\times$ faster than model-based approaches like EAGLE-2/3 and 1.6$\times$ faster than model-free approaches such as Token Recycling. SuffixDecoding is open-sourced.


Peer reviews:
comments: SUMMARY:
This paper proposes a novel model-free speculative decoding algorithm for the LLM agent domain. By leveraging a prefix-tree, the proposed algorithm could effectively find previous tokens for the new speculation. Even though this method cannot be genealize to all domains, it serves well for the agent domain, where the generation often follows certain patterns. Overall, I think this paper proposes an efficient solution for this niche challenge.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The proposed method is sound and carefully designed for the target scenario
2. The paper is clearly written and easy to follow

Weakness:
1. The technical depth of this work can be further improved. For example, the current adaptive speculation length algorithm is quite straightforward. As the main contribution of the proposed algorithm, it is worth further investigation. For example, whether the past information in each trajectory can help select a better speculation length.

QUESTIONS:
1. How will the proposed algorithm impact the memory consumption
2. What is the performance of the proposed algorithm on a more general domain benchmark like MT-Bench?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SuffixDecoding, a novel, model-free speculative decoding method designed to reduce LLM inference latency for agentic AI applications that produce repetitive and predictable sequences. SuffixDecoding uses efficient suffix trees to cache and rapidly match long token sequences from prompts and past outputs, enabling fast draft token generation without GPU overhead. A key innovation is its adaptive nature; it speculates more tokens when pattern matches are long and confidence is high, and fewer when matches are short, thereby conserving computational resources. Evaluations on agentic benchmarks like SWE-Bench and a Text-to-SQL pipeline show that SuffixDecoding achieves speedups of up to 3.9x, outperforming SOTA model-based (EAGLE-2/3) and model-free (Token Recycling) methods on these agentic tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper addresses the critical and timely issue of LLM inference latency, focusing on the important and growing domain of agentic applications.\n2. The method proposed in the paper is simple and straightforward while being practically useful.\n3. Significant speedup on agentic workloads compared to SOTA model-based and model-free methods.\n4. The ablation experiments answer most of my questions about the method.\n\nWeaknesses\n1. The primary benefit of the proposed method is in repetitive workloads. While highly effective for agentic applications characterized by repetitive token sequences, its advantages may be less pronounced in purely open-ended or less repetitive scenarios when compared to specialized model-based methods designed for those tasks. However, the paper does show it can learn useful patterns even in open-ended workloads and can fall back to the SOTA model-based method.\n\nQUESTIONS:\nIt would be great if the authors could also briefly discuss how to trade off the number of draft tokens per request within a batch, especially when all requests might have high confidence in generating long draft sequences -- how the proposed method can be properly integrated with the online serving scenario.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SuffixDecoding, a model-free speculative decoding method to save compute at inference time for LLMs by reusing tokens predicted on previous prompts/contexts.  The novelty introduced is in the trie/suffix tree which allows for lookup of past sequences (from  prompts or completions) in a dynamic / adaptive system.  Its highlighted use in agent-based systems with structured outputs is highlighted due to the type of repetition present in those settings that allows for the differentiated improvement of this method in storing/retrieving past tokens in such a system.  Like standard speculative decoding, the inference result is exact, with no accuracy loss, and only gains from leveraging the previous cache.  The experiments show significant speedups compared to existing methods in agent/structured settings.  The method has also been open-sourced and integrated into vLLM.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* Paper is clear and well-written\n* Motivation is clear, and emphasized through the mention of patterns that are particularly useful for the priors of this method: self-consistency, self-refinement, multi-agent workflows\n* Method:\n  * Simple: easy to understand, no added complexity that isn’t justified, model-free\n  * Practical/general: doesn’t require strict homogeneity of LLMs within the multi-agent setting or training an auxiliary model; demonstrated use outside of multi-agent systems as well (with the caveat of reduced benefit in these settings)\n  * Open-sourced integration into vLLM – good community contribution\n  * Handles the tradeoff in number of consecutive draft tokens vs likelihood of them being incorrect in an adaptive rather than purely heuristic fashion, adapting specifically to the distribution driven by that agentic system.\n  * Mentions practical additions, like maintaining two trees – one that’s global and one that’s smaller, per-request\n  * Significant gains, including comparison to and performance against EAGLE\n  * Admits to limitations compared to existing methods in certain settings\n\n## Weaknesses\n* The scoring/lookup is the key challenge to overcome and could have been highlighted/explained better\n* The title is strange to me. 'Extreme' in the title seems a bit *extreme* -- I'd expect that word only with orders of magnitude difference.  I also didn't understand what 'Emerging' means here.  Are you referring to agents/structured outputs?\n* Proprietary benchmarks will make it hard for the community to reproduce/compare\n* Doesn't address scenarios when cpu memory is filled -- many practical scenarios at scale when this might happen.\n\nQUESTIONS:\n1. How does lookup time, speedup, and scoring approximation (compared to best lookup) scale with the number of records? (within some distributional assumptions)\n2. Can you explain the title?\n3. 'Related Works' section is in a bit of a weird spot. Why not together with ‘Background’?  If you were trying to point out tradeoffs or highlight most recent works in particular, I suggest renaming the section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces SuffixDecoding, a model-free speculative decoding framework tailored for agentic LLM pipelines where generations exhibit high repetitiveness and structured loops. Instead of relying on a smaller draft model, SuffixDecoding constructs a suffix tree over past prompts and completions to identify the longest matching context at each decoding step. It then greedily expands a “speculation tree” of frequent continuations, verifies the selected prefix in a single forward pass of the full LLM, and adjusts its maximum speculation length dynamically based on match depth. Evaluated on two real-world agentic benchmarks (SWE-Bench, AgenticSQL) and in live vLLM, SuffixDecoding achieves up to 3.9× speedups on offline tasks and 4.5× end-to-end gains, while preserving exact token fidelity.\n\nSTRENGTHS AND WEAKNESSES:\n### Quality\n* The authors compare against both model-based speculators (EAGLE-2/3) and other model-free baselines (Token Recycling, Prompt Lookup), demonstrating consistent speed and throughput gains. – The paper reports FLOP-based efficiency but omits wall-clock latency and energy measurements on GPU hardware, leaving system-level impact unquantified.\n\n### Clarity\n* The algorithmic pipeline is described clearly, with pseudocode guiding the reader through tree construction, speculation, and verification steps. Figures illustrate key data structures and control flow. – Details around memory management (suffix-tree storage, updates under concurrency) and fallback threshold tuning (τ) are only sketched in prose, making reproduction nontrivial.\n\n### Significance\n* By decoupling speculative decoding from auxiliary draft models, SuffixDecoding opens a new direction for model-free inference acceleration in settings where prompt/response patterns repeat. – Its applicability to more open-ended or non-agentic workloads remains unclear: acceptance rates vary and no clear decision rule is given for when to revert to traditional draft-model approaches.\n\n###Originality\n* The use of suffix trees for speculative decoding is novel and exploits structural redundancies in agent-driven loops. – Prior work has explored cache-based and retrieval-augmented inference; the paper could more sharply delineate how suffix-tree speculation fundamentally differs in algorithmic guarantees or failure modes.\n\nQUESTIONS:\n1. Latency and profiling\n- Please report real-world latency and energy consumption comparisons between SuffixDecoding and a standard draft-model speculative decoder on GPU. This will verify that FLOP savings translate into actual system-level speedups.\n\n2. Memory overhead\n- How large do the suffix trees grow in practice, and what are the RAM and update-time costs under concurrent agentic requests? Empirical profiling or a detailed complexity analysis would clarify deployment feasibility.\n\n3. Fallback threshold\n- The hybrid threshold τ governs when to abandon tree-based speculation. Can you characterize its sensitivity across mixed workloads? A small study varying τ on open-ended versus highly repetitive streams would help practitioners tune the system.\n\n4. Generalization beyond Agentic workloads\n- Have you evaluated SuffixDecoding on less structured or single-turn tasks (e.g., open-ended question answering)? Understanding its failure modes and identifying break-even points would guide broader application.\n\n**Note: In particular, providing detailed latency and memory-overhead measurements across diverse tasks, along with a thorough analysis of τ’s sensitivity, will likely eliminate any remaining concerns and could substantially raise the paper’s overall score.**\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}