SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
Authors: Gabriele Oliaro
Zhihao Jia
Daniel F Campos
Aurick Qiao
Venue: NeurIPS 2025 spotlight ()
Keywords: speculative decoding
LLM agents
model-free speculation
SWE-Bench
LLM inference
Primary area: deep_learning
TL;DR: A model-free speculative decoding method that accelerates agentic AI workloads using suffix trees. Achieves 5.3x speedup on multi-agent tasks.

Abstract: Speculative decoding is widely adopted to reduce latency in large language model (LLM) inference by leveraging smaller draft models capable of handling diverse user tasks. However, emerging AI applications, such as LLM-based agents, present unique workload characteristics: instead of diverse independent requests, agentic frameworks typically submit repetitive inference requests, such as multi-agent pipelines performing similar subtasks or self-refinement loops iteratively enhancing outputs. These workloads result in long and highly predictable sequences, which current speculative decoding methods do not effectively exploit. To address this gap, we introduce \emph{SuffixDecoding}, a novel method that utilizes efficient suffix trees to cache long token sequences from prompts and previous outputs. By adaptively speculating more tokens when acceptance likelihood is high and fewer when it is low, SuffixDecoding effectively exploits opportunities for longer speculations while conserving computation when those opportunities are limited. Evaluations on agentic benchmarks, including SWE-Bench and Text-to-SQL, demonstrate that SuffixDecoding achieves speedups of up to 3.9$\times$, outperforming state-of-the-art methods -- 2.2$\times$ faster than model-based approaches like EAGLE-2/3 and 1.6$\times$ faster than model-free approaches such as Token Recycling. SuffixDecoding is open-sourced.

SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
Gabriele Oliaro
Zhihao Jia Daniel Campos Aurick Qiao
SSnowflake Al Research *Carngge Mellon University
{goliaro, igoliaro,zhihhojii zhihaoj2) cmu. edu,
{daniel. campos, campos ataappps....rrt.....oooooooto...
Project Page: http:///rriiixeeeooi...-ttttoob. http://ppgggaaa...ooooooooooaaaa
Abstract
Speculative decoding is widely adopted to reduce latency in large language model (LLM) inference by leveraging smaller draft models capable of handling diverse user tasks. However, emerging Al applications, such as LLM- agents, present unique workload characteristics: instead of diverse independent requests, agentic frameworks typically submit repetitive inference requests, such as multi-agent pipelines performing similar subtasks self-refinement loops iteratively enhancing outputs. These workloads result in long and highly predictable sequences, which current speculative decoding methods do not effectively exploit. To address this gap, we introduce SuffixDecoding, a novel method that utilizes efficient suffix trees cache long token sequences from prompts and previous outputs. By adaptively speculating more tokens when acceptance likelihood is high and fewer when it is low, SuffiDecoding effectively exploits opportunities for longer speculations while conserving computation when those opportunities are limited. Evaluations on agentic benchmarks, including SWE-Bench and Text-to-SQL, demonstrate that Suffix achieves speedups of up to 5.3x, outperforming state-of-the-ard methods- 2.8% faster than model-based approaches like E.GLLE-2233 and 1.9x faster than model-free approaches such as Token Recycling. Suffix SecoinDg is open-sourced at hhttps: /ttpt/gttttt/ttt com/ coom/nooe swwwf wwwwwllkedb///oooo waateee//Arcticnffeeeeete
Introduction
Large language models (LLMs) are foundational agentic AI applications, such as automated coding assistants [Wang et al. 2025 Yang et al., 2024), multi-agent workflows [Wang et al.) 2024a Chen al., 2024a], and retrivaal systems [Wang et al. 2024d Gao et 2024b]. Unlike basic chatbots, these workloads issue repetitive and predictable inference requests. For instance, multi-agent systems repeatedly perform similar tasks, and reasoning loops (Wang et al., [203a]] |Mdaan al. 2023) regenerate similar token sequences. Despite this predictable repetition, existing methods fail to fully exploit recurring patterns, leaving latency as a bottleneck.
A poopuarr strategy for mitigating inference latency is speculative decoding [Leviathan et al. 2023
Chen al. 2023 Miao et al./ [2024] Cai et al. 2024 Lin al.) [2024 Zhang al. [202]]. While an LLM can only generate one token per forward pass, it can verify multiple tokens. Leveraging this phenomenon, speculative decoding methods small "draft" models additional decoding heads to predict multiple candidate tokens, which the LLM then verifies in parallel.
To efficiently handle the long repetitions common in agent-driven applications, speculative decoding methods must satisfy two critical requirements. First, they need to generate draft tokens rapidly and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

with minimal overhead, enabling maximal exploitation of long speculation lengths. Second, they must do so adaptively-on generating more draft tokens when acceptance likelihood is high and fewer tokens when acceptance likelihood is low, to prevent verification from becoming bottleneck.
However, existing speculative decoding approaches fall short in meeting these dual requirements. Model-based methods can use significant GPU time when speculating long sequences, and can incur memory contention and kernel-level transitions [Chen et al. 2024b Li al., 2024 that must be managed carefully. Conversely, existing model-free approaches, such as prompt-lookup decoding (PLD) Saxena 2023]. achieve low overhead and rapid token generation, but typically lack adaptivity. These methods speculate fixed number of tokens irrespective of acceptance likelihood, leading to wasted computational resources on verifying long and improbable draft sequences.
Ongoing
Infereasee
5
Expassion
Murify (LLM))
Indlxx
Speculation
Previous Outputs
Spoculation Tree Candidates
Verification
Suttix Trees
Generation
Figure I: Overview of SuffiDecoding's algorithm. Two suffix trees track ongoing inference (top-left) and previous outputs (bottom-left). SuooiDe uses these trees find matching patterns based on recently generated tokens. It constructs speculation tree (middle) by selecting the most likely continuations, scoring them based on frequency statistics. Finally, the best candidate verified by the LLM in single forward pass (right), with accepted tokens (shown in green) being added to the output and used for the next round of speculation.
To address these limitations, we introduce Suffixeecoding (Fig model-free speculative decoding method for repetitive, agent-driven workloads. SuffixDecoding uses efficient suffix trees to cache long token sequences from prompts and previous outputs. Each node represents a token, and paths encode previously observed subsequences, enabling rapid pattern matching to identify continuations based on prior occurrences. Draft tokens are generated extremely quickly (~20 microseconds per token) without GPU overhead.
At each inference step. Suffixeccoing adaptively limits its number of draft tokens based on the length of the pattern match, and uses frequency-based statistics captured within the suffix trees to score and select the best speculation candidate. Longer pattern matches enable confident speculation of longer token sequences, maximizing its effectiveness agentic workloads, while shorter pattern matches trigger conservative speculation to avoid computational waste. Moreover, SuffixDeoddng can seamlessly integrate with existing model-base speculative decoding methods. This flexibility enables a hybrid approach that leverages suffix-tree-baed speculation for repetitive, predictable agentic workloads, while exploiting the strengths of model--ased speculation methods for open-ended conversational tasks, thus achieving the best of both worlds.
We evaluate SuffixDecoding on two practical agent-driven workloads: SWE-Bench, an LLM- software engineering benchmark, and AgenticSQL, proprietary multi-agent pipeline application for SQL generation. We compare with state-of of-the-artttt--- baseel-based and model-free speculative decoding methods using Spec-Bench (Xia et al., [202], showing up to 2.8x faster decoding than EAGLE-2/3 [Li et al., 2025a], and 1.9x faster decoding than Token Recycling (Luo et al. 2024]. For SWE-Bench, we also measured the comprehensive, end-to-end task completion time- includinn prompt prefilling, token generation, and execution of external actions-and demonstrate speculative speedups of up to x.5x. These results highlight that Suffix substantially reduces latency for real-world agentic applications, addressing critical bottleneck in practical inference scenarios.

---

2 Background and Related Work
LLM Inference. LLM inference involves two stages: given prompt Z prompe = (I1, 32,,.. Fm)) the LLM first processes the prompt in parallel (prefill), then sequentially generates new tokens (decode), with each token Ztsm conditioned on previously generated tokens:
2++1 = Sample(z)/r.....
In greedy sampling, the highest-probability token is selected iteratively until stopping condition. Since each token depends on preceding outputs, generation is inherently sequential, requiring separate forward pass per token, limiting throughput and underutilizing parallel hardware accelerators.
Speculative Decoding. Speculative decoding (Leviathan et al., 2023) accelerates inference by generating multiple candidate tokens quickly using lightweight model, which can then be verified in parallel by the primary L.IM. The basic method has two core steps:
Speculation: A smaller "draft" model rapidly produces speculative tokens (#t+1...., Iz+n) based on the existing token prefix T<t.
Espee
2. Verification: The LLM verifies the draft tokens in parallel, accepting tokens up to the first discrepancy and discarding the rest.
This shifts computation from sequential generation parallel verification. However, draft models still require compute resources and add orchestration complexity. Model-based methods include Medusa [Cai et al. 2024]. SpecInfer [Miao al. 2024 (which introduced tree-based speculation).
multi-token prediction [Gloeckle et al. 2024]. and blockwise parallel decoding [Stern et al. 2018
Kim et al. 2024
Recent model-free methods include Prompt Lookup Decoding [Saxena 2023], Token Recycling [uo al.] 22024. LLMA (Yang et al. 2023), and ANPD [Ou al. 22224] These rely small reference texts and lack adaptive speculation. SuffiDDeoddin is uniquely designed for agentic applications with long repetitive sequences.
Agentic AI Algorithms. Agentic applications structure tasks as multiple LLM calls, generating long repetitive token sequences. Self-consistency (Wang al. 2023a] samples multiple reasoning paths from the same prompt, sharing similar shain-of-though steps. Self-refinmment [Madaan et al. 2023y iteratively fixes eπrоrs, revising small portions while preserving surrounding content. Multi- agent workflows (Khot et al. | use specialized agents with narrow functions, producing repetitive outputs. These patterns create opportunities for exploiting long repeated sequences.
Methods for Accelerating LLM Agents. Recent works target latency in agentic applications. ALTO [Santhanam al. 2024 optimizes multi- aalti-ageettt workflows through pipelining and scheduling. Dynasor [Fu et al.) 2024 early-terminate unlikely reasoning paths. Suffixecoddigg takes an orthogonal speculative decoding approach and can be used in combination.
3 SuffixDecoding
The goal of SuffixDecoding is to enable fast, adaptive speculative decoding over long sequences, particularly suited for agentic applications where repeated inference calls often contain highly predictable and overlapping token sequences. In such settings, long stretches of output can be accurately predicted from prior and ongoing requests.
To fully exploit these opportunitics, SuffiXecoding must address two key challenges. First, must support fast generation of speculative sequences including long inntinuations-wittee relying on draft models or expensive token-by-token prediction. Second, must be adaptive to the current prediction context: aggressively speculating long continuations only when they are likely to be accepted, and speculating shorter sequences when uncertain to avoid wasted verification compute.
To support fast speculation, Suffix builds suffix free [Weiner 1973 over the tokens in the current and prior requests, and uses the suffix tree to generate speculative tokens. The root node of the tree represents the beginning of suffix of any token sequence stored in the tree, each child of

---

a node represents a specific token which is possible continuation from its that node, and the path from the root each node represents a distinct subsequence.
For each request, we consider speculating token sequences from (1) the prompt and output of that request, and (2) the outputs of prior requests. Doing so captures the source of output token repetition from many agentic algorithms, including self-consistency, self-refinement, and multi-agent pipelines.
Suffix SuffixDecodigg leverages suffix trees to perform fast pattern matching and find possible continuations of token sequences. Suppose the prompt and output tokens an ongoing inference is T1.t- Consider suffix 2t-p+1:4 of length p, which starting from the root node, and at each step taking the child that corresponds to token Tt-p+s- If no such child exists, then the pattern is not found and SuffixDeodding reverts to standard non-speculative decoding. Otherwise, after p steps, we arrive at a node whose descending paths are the possible continuations of the pattern sequence.
will refer to the pattern sequence. We walk the suffix tree
Although this procedure can quickly find a (potentially large) set of candidate sequences, verifying all of them in speculative decoding may be cost-prohibitive. Instead, SuffixDecoding builds much smaller and more likely speculation tree through a greedy expansion and scoring procedure, and uses this smaller tree in tree-based speculation. An overall illustration of SuffiXDecdding is shown in Fig. T which we detail in the rest of this section.
Suffix Tree Construction. Building the suffix tree and updating it as part of an online inference service involves two stages. First, the previous inference outputs can be added to the tree a single offline processing step (e.g. from historical logs), or online during inference serving after each inference request completes. Second, the current ongoing prompt and out tokens are added online as new requests are received and as each new token is generated.
In reality, we found convenient to maintain two different suffix trees: a global tree for the previously generated outputs, and separate per-request tree for the current ongoing inference request. This circumvents the complexities and overheads due to synchronizing the suffix tree updates from multiple concurrent requests. The global tree can be constructed offline in O(n) time, while the per-request tree can be efficiently constructed and updated online [Ukkonen] 1995].
Although suffix trees are memory-efficient at O(n) space, the global tree can still become large when there are many previous outputs. However, they only require CPU memory, which typically plentiful and under-utilized in LLM serving scenarios. For example, AWS p5. 48xlarge are often used for LLM serving and have 2TB of main memory, which is easily enough to support suffix tree over millions of historical outputs and billions of tokens. Given typical server configurations SuffiDeecddig can cache approximately a month's 's worth of generated tokens before requiring cache eviction (see Appendix B for detailed memory overhead analysis).
Speculation Tree Expansion. Given a pattern sequence It-p+1: of an ongoing inference Ti:t- SuffixDecoding can quickly find node N, in the global per-request suffix tree whose descending paths are the possible continuations of the pattern sequence. To select a smaller more likely sub-tree that is of a more practical size for speculative verification, we start with the single node N, and grow a sub-tree greedily by expanding one leaf node at a time.
In particular, define:
COUNT(NV)
C(N) =
Σ MECHILOREE(/AREETTN) COUNT (M)
D(N)- ((PARENT(N)) x C(N), if 2
11
otherwise
where COUNT(N) is the number of occurrences of node N in the reference corpus, which can be computed when constructing the suffix tree. Starting with the single node N₂ in our speculation sub-tree, we consider all children of all of its leaf nodes, and add the node N with the highest D(N). This process is repeated until the sub-tree tree reaches a predetermined size limit, MAX_SPEC.
Intuitively, C(N) estimates the probability that TOKEN (N) would be the next observed token in a sub- sequence TOKEN (OKEN(N,),-.. ...,... TOKEN (PAENNPARRETT) PARENT(N)). and D(N) estimates the probability that TOKEN NN) would be ultimately accepted by the speculative tree verification, assuming the output tokens follow

---

historical patterns. Thus, SufoiDecoding builds the speculation tree by greedily adding leaf nodes that it believes to be the most likely be accepted during verification.
Algorithm Speculation Tree Generation
function EXPANUSPPECULATIONNEERN
MAX (PEC))
Input: Suffix tree node Np. MAX_SPEC
Initialize T <- (Np)
while [2'] < MAX_SPEC do
N f- arg DEXXVVCCIDDEENEEEEENYT) D(N)
T T-+77 TUTU {N}
end while
return
end function
function MATCHPATTERN(S. Xia. p)
Input: Suffix tree S, sequence Ii:4, length p
Initialize Np <- ROOT(S)
for 1 = 1 to do
if NfNO_ce WWO_CHILD(Yp.... Pt Pt-pt() then
return 0 end
N₂ < CHILD(N end for return Np
function
function GENERATECCANDDATETTRRSSS g. S.S... Xıa, a, P)
Input: Global suffix tree Sg. prompt suffix tree Sr, sequence 2ist. max spec factor max pattern size Initialize Thea +- 0, SCORE +-
for S in {Sg, S. do
for p =
N +- MatchPatterruS,,, IS,
T ExxpandSpeculation Tree [N, ap)
if SCORE(T) SCOREses then
Thest + T SCORE + SCORE(T)
end end for end for return Thesr
end function
Adaptive Speculation Lengths. While the procedure above allows SuffixDecoding to cache and quickly speculate long token se- quences based on empirical probability esti- mates, it also needs mechanism for adaptively controlling the number of tokens speculates. SuffixDecoding achieves this by dynamically ad- justing MAX_SPEC. Low values mean fewer but more likely tokens would be chosen for specu- lation, while higher values mean more but less likely tokens would be chosen. If too low, then the speedup from speculation can be limited, and if too high, then compute may be wasted on verifying unlikely tokens.
g-lcc
Constani Adaptive
Moan Accopted Tokena
Speedtp Pector
2.7
ex2.7
2.4
1.4
Langth
Acceptancc
(a) Avg accepted tokens vs (b) Using constant vs adap- pattern match length. tive MAX_SPEC
Figure (a) the mean number of accepted to- kens increases with the length of the pattern match, which motivates MAX_SPEC ap. (b) shows that this choice achieves better trade-off between ac- ceptance rate and speculative speedup.
To guide how to adaptively set MAX_SPEC, we observed that the number of accepted tokens in practice typically increases with longer pattern
sequence lengths p (Fig. 2ap. Thus, we define MAX_SPEC adaptively as
MAX_SPECH (X__SEC(p) app,
5

---

where a is a user-defined max speculation factor. Fig. 26 shows that setting UAX_SPEC adaptively according to the pattern length results in a better trade-off between acceptance rate and speculative speedup. In practice, we found that E [1, 4] works well for agentic applications.
Speculation Tree Scoring. So far, we have discussed how to obtain a speculation tree given a suffix tree and pattern length p. However, SuffixDecoding maintains two suffix trees, the global suffix tree and the per-request suffix tree, each with many choices for p. To obtain just à single speculation tree, build speculation trees for both the global suffix tree and the per-request suffix tree, and for range of values of Then, a single speculation tree is selected according to scoring function:
SCORE spee)
Σ
D(N).
NET
Intuitively, if D(N) estimates the probability that node N in a speculation tree Tapec would be accepted, then SCORE spee) estimates the expected number of accepted tokens. SuffixDecdding then selects the Tepec with the highest SCORE as the final speculation tree to be verified. The end-to-end candidate generation from speculation tree expansion to scoring is described in Alg. T
Hybrid Suffix Speculative Decoding. Lastly, we find that SCORE(Tsper spee)) can be used to dynamically decide between using SuffiDee or falling back to a model-basee speculation method, which is useful for practical scenarios when the workload can be mixed between agentic and more diverse applications. Specifically, for each decoding iteration, we always speculate using SuffiDDecoding first. If SCORE(T spec) > т, where is a configurable threshold, then Suffix draft tokens are used. Otherwise, we use a fall-back speculation method, such EAGLE-3 [Li al., 2025a
As a practical guideline, we find that setting t close to the mean accepted tokens of the fallback speculator works well for mixed workloads, while using SuffixDecoding alone (r 0) is optimal for highly repetitive agentic tasks. A detailed sensitivity analysis of r across different workload types is provided in Appendix B.I For batched serving scenarios. SuffixDecoding can be integrated with existing batch-level speculation control methods (see Appendix C
4 Evaluation
4.1 Evaluation Methodology
Baseline Comparisons. We compare with both model-based and model-free speculative decoding methods using Spec-Bende [Xia et al.,
Model
Strattrrd Generation
Usetmecttred Gcnaration
2024]. (1) EAGLE-2 and EAGLE-3 [Li et al.
2025a state-of-the-arn model-based specula-
Classity
Rariecal-Augmenttl Genertiion (RAG)
tors, (2), Prompt-Lookup Decoding (PLD) (Sax-
enaa 2023). a simple model-free speculator based on ngram-matching, and (3) Token Re- cycling [Luo et al. 2024], a more recent model- free speculator that sources token sequences from both the prompt and previous outputs. EAGLE-3 and Token Recycling both leverage tree speculation [Mao et al. 2024
Extrct
Firor ! Conrectoon
SQL
Quetiic Imcrpcptiion
Earich
:
Combine
SQL Quer
SQLN
Cocrection
Error
Dasihas
Figure 3: AgenticSQL is multi-agent workflow consisting of stuctured generation, unstructured generation, and steps across several different LLMs. Useful fea- tures are extracted from the user question (Classify and Extract) and supplemented with retrieved con- text (Enrich). Several text-to-SSO to-SQL steps propose solutions to the user question (SQL 1...N) N) in par- allel with feedback from an error corrector. A last Combine step synthesizes the proposed SQL can- didates into a final SQL query and text response.
generation
Datasets and Agentic Applications. We eval- uate on both agentic and non-agentic workloads. For agentic applications, we trace requests from two real applications: OpenHands (Wang et al. 2024c] on BWEE-Becch (Jimenez et all.[024) (a GitHub issue resolution benchmark), and Agen- ticSQL, a proprietary multi-agent SQL genera- tion workflow (Fig. 3). For non-agentic work- loads, we use Spec-Bench, consisting of open- ended, single-turn tasks across 13 categories, in-

---

Speculative Speedups over Vanilla Decoding
500%
400%
Spunduu
PLD
Riloid sive
1.46
200%
1.7%
156
15s
100%
176
1N%
Spec-Bench
AgenticSOL
SWE-Bench
Benchmarks
Mean Accepted Tokens per Step
Mean Accepted
1.1
3.5
Spac-Bench
AgenticSAL
Benchmarks
Figure 4: Speculative speedups (top) and mean accepted tokens per step (bottom) compared to vanilla decoding for SuffiDeecoding and baseline methods on three benchmarks: Spec-Benche AgenticSQL, and SWE-Bench. Experiments use Llama-3.1-88 --8----tsttt Speedup is measured as the ratio of wall-clock time-per-output--oiee Suffix (tree) and Hybrid (tree) SuffixDecoding's tree speculation algorithm, which constructs speculation tree from the suffix tree for parallel verification. Suffix (linear) and Hybrid (linear) use simpler linear speculation approach that only allows sequential token chains. The hybrid variants combine SuffixDecoding with EAGLE-3, dynamically selecting between suffix-based and model- based speculation based on pattern match confidence. Note that EAGLE-2/S and Token Recycling failed to run on several SWE-Bench tasks due to long context lengths (>81 tokens), indicated by missing bars. Spec-Bench represents non-agentic workload and included for comparison. Further sub-task breakdowns, including the raw teee-per-outpuutooon token and mean acceptance lengths, can be found in Appendix A.I
single H100 GPU with batch size 1. relative to vanilla decoding.
cluding 8 MT-Bench categories (Writing, Role-
play, Reasoning, Math, Coding, Extraction, STEM, Humanities). We include Spec-Bench to stress- test SuffixDecoding limitations and evaluate the hybrid fallback mechanism.
End-to-end System Evaluation. We implemented SuffiDeccodigg in vLLM [won et al. 2023 By running OpenHands live, show SuffixDecoding accelerates end-to-end task completion times, including prefill and code execution.
Hardware configuration. We conducted our experiments single p5. 48xlaarge AWS instance equipped with 8x NVIDIA H100 80G GPUs and 2TB of main memory.
Simulated Ablations. In addition a simulated verifier for additional experiments in Sec. [.4 and continued in Appendix [A.2 Given prompt i.m and example ground-truth response In+1::t verification for greedy sampling by verifying that speculated token Fn+i = Ynti-
main evaluation using real hardware, also leverage
accurately simulate speculative
4.2 Baseline Comparisons
We compare Suffixeeoddig with EAGLE-2, EAGLE-3, PLD, and Token Recycling on SWE- Bench and AgenticSQL. We also run the SpechBench standard dataset, which a more traditional non-agentic workload. Fig. T shows the results. First, the agentic workloads, SuffixDecoding

---

outperforms all baselines. In Age SuffixDecdding obtains mean speedup of 5.3x over vanilla decoding, a 2 improvement over EAGLE-2/3, and 1.9x higher than Token Recycling. In SWE--enchh EAGLLE-2/3 fail due to their maximum sequence length limitations. SuffixD obtains a mean speedup of 2.5x over vanilla decoding, 1.7x improvement over PLD, the next best baseline. Suffix SuffiDecoding's s superior performance in agentic workloads can be attributed to its consistently higher mean accepted tokens per decoding step. In AgenticSQL, Suffixecoddin reaches 6.3 mean accepted tokens per step- substannnall higher than EAGLE-3 (3.6 tokens) and Token Recycling (3.2 tokens). In SWE-Bench, SuffixDe achieves 7.8 mean accepted tokens per step, while PLD only accepts 3.2 tokens per step average.
On non-agentic workloads such as Spec-Bench (which includes open-ended single-turn tasks and MT-Bench categories), SuffixDecoding alone is outperformed by EAGLE-2/3 and Token Recycling, as expected for less repetitive scenarios. However, the hybrid approach of Suffix + EAGLLE- 3 achieves the best of both worlds: we speculate with the faster SuffiDDecoding method whenever possible and fall back to EAGLE-3 when the speculation score is too low. The Hybrid approach obtains a mean speedup of 2.5x over vanilla decoding, outperforming the 2.4x speedup from standalone EAGLE-3 and the 2.2x speedup from Token Recycling.
The hybrid approach also performs well in AgenticSQL, achieving a 4.1x speedup in the tree variant, significantly better than the 1.9x speedup from standalone EAGLE-2/3 and the 2.7x speedup from Token Recycling. These speedups are achieved thanks to the hybrid approach's 's impressive 7.5 mean accepted tokens per step. more than 2x higher than EAGLE-2/3 and Token Recycling. Decoddng has slightly lower mean acceptance length of 6.3, but its much lower speculation cost and higher acceptance rate make the winning solution in agentic tasks (5.3x average speedup compared to the 4.1x speedup of the hybrid approach).
A peek into a speculation tree. To gain some intuition into why Suf- fixDecoding performs so well for cer- tain tasks, we examine how it builds a speculation tree for the AgenticSQL. Extract task. The outputs of the Ex- tract task have many characteristies in common. First, they are all JSON doc- uments following the same format and key names, with keys often appearing in the same order. Second, many of the features are discrete values, and in particular, boolean true/false values. These patterns recorded in SuffiDDecoding's global suffix tree and guide its speculation tree construction.
Foloo: "featere_y": falon, "feature_ p faloe)
Spez hoarch
"teature_x"s
Profis
Spoc leanch
"Seatuce_y"
Spec brunch 2
"feature_
Spec branch
Figure 5: A SuffixDecoding speculation tree containing 66 tokens for the AgenticSQL Extract task.
Fig. 5 shows example of a speculation tree constructed by SuffixDecoding We observed many instances of large speculation trees that branch at each boolean true/false value of several consecutive features. These speculation tree always contains branch with high acceptance, advancing output generation by dozens of tokens or more in one step. Although this is one specific example of a speculation tree, it demonstrates that Suffixeecoding can find complex patterns in previous outputs, particularly for structured generation tasks, that help accelerate output generation.
4.3 End-to-End SWE-Bench on vLLMM
In this section, we show that SuffixDecoding can be efficiently integrated into vLLM, a popular inference system used in production deployments, and it can effectively accelcrate accelerate end-to- end agentic task complction time. For this experiment, we run OpenHands directly on vLLM with Suffixeecoding, the agent is solving each benchmark problem live. We also use the specially- trained LLM all-hands/ all-hands/openhandde -wwphhannd--m-32----1 and achieves 37.2% on SWE-Bench Verified. Since there are model-based methods with draft models trained for this LLM, we compare with vLLM's native implementation of PLD.
1-ep3, which was fine-tuned for SWE- E'E-Bench
Fig. 6 shows the results. First, we note that decoding time (i.e. output generation) takes a majority of the time across all SWE-Bench tasks, dominating both prefilling and agentic actions (i.e. code exccution). In this end-to-end scenario, SuffixDecoding outperforms PLD by 1.3-3x, leading to
8

---

Solving
Decoding
Solving Time per Instance
093
iatrepy Inttonyy
daangesHange
mappbatiimmmppen
pallow-
palreppeato
Pydathherray
pnlar-dovppllle
pytnt-deuippter
wilks femi-ermiiiiiiiicc from
apliion dos liphiooe
ayoapy syrmag
Repository
Legend
Vazzla Decading line
Przape Lookop Doooiigg Linz
Sudiis Devoding (Cnro)
Vonita Pwstiling
Prrnpe Lookop twililing
Suftic Poetiling
Viailla
Lackup
Figure 6: End-to-end task-completion time of the OpenHands agent on SWE-Bench Verified. The benchmarks are run with a concurrency of 8 tasks running simultaneously. vLLM is deployed on H100 GPUs configured with 4-way tensor parallelism and prefix caching enabled. The results are broken down by the different code repositories in SWE-Bench.
1.8 x speculative speedup over vanilla decoding. Since SuffiDecodding exactly preserves the output distribution of the LLM, it matches the original model's 37.2% score on SWE-Bench Verified.
4.4 Ablation Experiments
In this section, we present few ablation studies on SuffiDecoddin using simulated verifier on offline traces. Given ground-truth prompt-response pair from an LLM, we can verify the draft tokens proposed by SuffiDeecdding by comparing with the ground truth responses. Additional ablation studies can be found in Appendix A.2
Global vs per-request suffix trees. We study impact of the two suffix trees: the global suffix tree containing previous outputs, and the per-request suffix tree containing the prompt and generation of the current ongoing request. To do so, we ran the tasks in AgenticSQL using SuffixDecoding (I) with the global suffix tree only, (2) with the per-request suffix tree only, and (3) using both trees.
15
Buth Treee (Bheelnn)) Global Tree Only
80
Per-requent Tiwe Only
Spredig
20 as
Fig. 7 Isowss the results. First, we that with the exception of the Content Enrichment (En- rich) and Extract steps, using both suffix trees performs better than using just one. The small degradations on the enrich and extract steps sug- gest that, when both trees present, SeffixDe- coding may sometimes choose a speculation tree from the per-request suffix tree when the global suffix tree may have been the better choice. Im- provements to Suffix SucoixDecoiing's speculation tree scoring mechanism may help bridge this gap.
Avg. Sperutsted
Classiy
Combine
Eartch
SOL
80L
50L
AgcnticSOL
Figure 7: Speedup factor and number of speculated tokens for the tasks AgenticSQL. Suffix ing was run with only the global suffix tree, only the per-request suffix tree, and both (baseline).
Second, the global tree outperforms the per- request tree on tasks except for Combine.
This is because the Combine task heavily re-uses tokens from its context, which are the proposed SQL

---

solutions from the previous steps in the workflow. Although there is diversity of task characteristics, SuffiDeecoding able to achieve high speedups on all of them by combining both suffix trees.
SuffixDecoding in open-ended scenarios. Al- though SuffiDeeodding is designed for agentic workloads with long repeated token sequences, it is also interesting to evaluate it using more open-ended workloads like WildChat (open- ended chat) (Zhao et al. 20244 and Magicoder (code-oriented chat) [Wei et al. 2023 Details on these datasets can be found in Appendix A.2
1.7
0.25
1.6
Sgouday Factor
Acceplanio Baite
1.4
0.15
0.10
1:2
Musicader
0.05
Magicadar
WildChat
WiiChatt
In Fig. [ we show the speedup and acceptance rate of SuffixDecoding on WildChat and Magi- coder across range of suffix sizes between 256 and 10,000 output examples. First, we note promising pattern: the speedup consistently improves as the size of the suffix tree grows. This indicates that SucfinD can learn use- ful patterns even in workloads with lower token repetition, and may be a substitute for model- based methods when a draft model is not available.
1.0
0.00
16
104
Sufti Tree Size (# Examples)
Suffi Tree Size (# Examples)
Figure 8: Speedup (left) and acceptance rate (right) vs global suffix size for Magicoder and Wild- chat (a = 1). The speedup from Decoidd continues to increase with more previous output examples, while the acceptance rate holds steady.
Second, perhaps surprisingly, the acceptance rate does not change much even when the suffix tree size varies across almost two orders of magnitude. We believe this is primarily due to the effect of the adaptive speculation length MAX_SPEC op. Although less data may mean less certainty in the speculated tokens, the pattern matches are also shorter, which results in fewer speculated tokens.
5 Conclusion
In this paper, we presented SuffixDecoding, a model-free speculative decoding approach designed for emerging agentic applications. Using efficient suffix tree data structures, Suffix ef- fectively exploits long and repetitive token sequences found in many agentic algorithms, such as self-consistency, self-refinement, and multi-agent pipelines. Using two practical agentic applications, OpenHands and AgenticSQL, showed that Suffixeeoding significantly accelerates their decod- ing latency and completion times, and is also significantly faster than other model-based and model-frec speculative decoding baselines.
References
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lec, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads, 2024. URL https://aartt org/ ab/////a.22.2....// 2401 10774
Sahil Chaudhary. Code alpaca: An instruction followwig llama model for code generation. https://github
com/sahi sshi 12801220144 14/0114/caa codeappaaaappaaa 2023.
Charlie Chen, Schastian Borgeaud, Geoffrey Irving, Jean- Baptitatiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023. URI /tttps////axxii... orrg abs/2302.0133.
Guangyao Chen, Siwei Dong. Yu Shu, Ge Zhang, Jaward Sesay, Börje F. Karlsson, Jie Fu, and Yemin Shi. Autoagents: framework for automatic agent generation, 2024a. URL Sttpssttss//aaxii org/ abaaaba//a 2309 17288
Zhuoming Chen, Avner May, Ruslan Svirschevski. Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding, 2024b. htR.https:... /aattp::/aariv.. org/ aba abb/2402.. 12374
Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, and Hao Zhang. Efficiently serving Ilm reasoning programs with certaindex, 2024. URL http:tt////xxii.oorot www.orgg.////ooooooee/o...//aaee httpab////2.22/0 20993
10

---

Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding. and Jingren Zhou. Texl-to-sql empowered by large language models: A benchmark evaluation. Proc. VLDB Endow. 17(5):1132-1145, May 2024a. ISSN 2150-8097, doi: 10.14778/3644224.3442222 3641221
https://://doi org/1o.rg/111// wtt......4.77/331110000 14778/3641204
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmentea generation for large language models: survey, 2024b. URL
abs/ 2312. 10997
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Proceedings of the 41st International Conference on Machine Learning, ICML 24, JMLR.org. 2024.
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pai, Ofir Press, and Karthik Narasimhan. Swe-hench: Can language models resolve real-world github issues?, 2024, URI Wtl.hhtpa:///axiii www.arvww.r..rr.eeoooo /bs/ 23100 06770
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks, 2023. abs/2210 02406
org/
Tachyeon Kim, Ananda Theertha Suresh, Kishore A Papineni, Michael Riley, Sanjiv Kumar, and Adrian Benton. Accelerating blockwise parallel language models with draft refinement. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https://openrevaait wet aermf?
id=KT6P55w0eg
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention, 2023. https://
.rg. h/t///bb///// 2309 06180
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org. 2023.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang, Eagle-2: Faster inference of language models with dynamic draft trees, 2024. /RL. /406. 16858
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference accelcration of large language models via training- test, 2025a. URL http::ttt///ttt //arxivv///xxv....// org/ abs/a/b...//////////// 2000.00000001
Zikun Li, Zhuofu Chen, Remi Delacourt, Gabriele Oliaro, Zeyu Wang. Qinghan Chen, Shuhuai Lin, April Yang. Zhihao Zhang, Zhuoming Chen, Sean Lai, Xinhao Cheng, Xupeng Miao, and Zhihao Jia. Adaserve: Accelerating multi-slo Ilm serving with slo-customized speculative decoding, 2025b. URL https://arxix org/abs/2en 2501 12162
Feng Lin, Hanling Yi, Hongbin Li, Yifan Yang. Xiaotian Yu, Guangming Lu, and Rong Xiao. Bita: Bi-directional tuning for lossless accelcration in large language models, 2024. URL hattps://arxtt org/ abs/ 2401
12522
Junda Liu, Yilong Zhao, Zheyu Shen, and Yiying Zhang. Turhospec: Closed-loop speculation control system for optimizing Ilm serving goodput. 2024. URL http::: http//aawaa.rrrraaaaaaaaaaaaea/e wwtp/rgg///bbaaeeeeeeeeee 14066
Xianzhen Luo, Yixuan Wang. Qingfu Zhu, Zhiming Zhang. Xuanyu Zhang, Qing Yang. Dongliang Xu, and Wanxiang Che. Turning trash into treasure: Accelerating inference of large language models with token recycling, 2024. URL httpt:t//////xiiii org/
08696
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instract, 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback, 2023. URL htttt::////arrttt http://axxiivoor//////////
17651
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS '24, page 932 949, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400703867. doi: 10. 00.11457622066..6615533 URL https: //ti///oiiooo// org/10. 1145/3620666. 3651335

---

Jie Ou, Yueming Chen, and Wenhong Tian. Lossless acceleration of large language model via adaptive n-gram parallel decoding, 2024. http:://:///r wttp:///..rrrrab///bab//e// 08698
Keshav Santhanam, Deepti Raghavan, Muhammad Shahir Rahman, Thejas Venkatesh, Neha Kunjal, Pratiksha Thaker, Philip Levis, and Matei Zaharia. Alto: An efficient network orchestrator for compound ai systems, In Proceedings of the 4th Workshop on Machine Learning and Systems, EuroMLSys '24, page 117- New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400705410. doi: 145/444666.270.
3655844, URL https:://dooo org/10. 10. 1145/3642979000. 3655844
Apoory Saxena. Prompt lookup decoding, November 2023. URL. httpt::/ggttuub.cc.... htwp:/huwb.coomppppruunnaggg Soooppt-ooouppdecccoo
Mitchell Stern, Noam Shazeer. and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, page 11107-10116, Red Hook, NY, USA, 2018. Curran Associates Inc.
E. Ukkonen. On-line construction of suffix trees. Algorithmica, 14(3):249 260, September 1995. ISSN 0178-4617. doi: 10. 10.1007/BF01206311 hRtpsstts////iitoott org/ 10. 1007 /FF0120331
Junlin Wang, Jue Wang, Ben Athiwaratkum, Ce Zhang, and James Zou. ofxture-of-agents enhances large language model capabilitics, 2024a. URL http:::taa.....oo/reee/. http///wwww....rr////////////// abs/////.//22... 04692
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better 1lm agents, 2024b. UtLha:ta:://artt http://wxxxvvo..raaaa.a aba/ 2402. 01000
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang. Mingehen 7huge, Jiayi Pan, Yueqi Song. Bowen Li. Jaskirat Singh, Hoang H. Tran, Fuqiang Li. Ren Ma. Mingzhang Zheng. Bill Qian. Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng. Heng Ji. and Graham Neubig. OpenHands: An Open Platform for AI Software Developers as Generalist Agents, 2024c. URL https://aaraae org/ /egg/abee /200/22400.m2 16741
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for ai software developers as generalist agents, 2025. URL Nttps://arxiv.oort bbgg/abs//ae aa/aa/as/2///// 2207 16741
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le. Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, Self-consistency improves chain thought reasoning language models, 2023a. URL
https://arxiv. wttp:///rrr/bb////////// 11171
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Han- nanch Hajishirzi. Self-instruct: Aligning language models with self generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 13484-13508, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10, 16233 URL https: btttp::/aacanttoo//
org/ 2023/2223... acl long. 754
Zilong Wang, Zifeng Wang, Long Le. Huaixiu Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, Chen- Yu Lee, and Tomas Plister. Speculative rag: Enhancing retrieval augmented generation through drafting, 2024d. URL httpp:://aaxvv..rg//////
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint 22120023...0 2023.
Peter Weiner. Linear pattern matching algorithms. In Proceedings of the 14th Annual Symposium on Switching and Automata Theory (Swat 1973), SWAT '73, page 1-11, USA, 1973. IEEE Computer Suciety. doi: 10. 10.1109/SWTT.19...11 URL https://t::///// http:::://ao..oo//1111 10. 1109 1109//AAT.......11 /WAT. 1973. 13
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhisang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding, 2024. URL hhtpt::tt////aee http:/axxiw.../raaaaa wttpaaa/////..o../////////// 07851
John Yang. Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Otir Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024. URL https
//axxiv. arg//bbs/4455 15793
12

---

Nan Yang. Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang. Rangan Majumder. and Furu Wei. Inference with reference: Lossless acceleration of large language models, 2023. httpttt/p///aiii org/ abs/2300 04487
Liangsheng Yin. Yineng Zhang. Ying Sheng. and The SGLang Team. Achieving faster open-source Ilama3 serving with sglang runtime (vs. lensorrt-llm, vllm). https://1mst
2000lo//222400-25---2-2 2024-07-255 2 sglang- weanngg-llamnnn 1lama3/ LMSYS Org Blog.
11mmm1/mm///// July 2024. URL Sttps://lmsyp: org/ blog/
Tao Yu, Rui Zhang, Kai Yang. Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale numan-labeles dataset for complex and cross-domain semantic parsing and text-to-SQU task. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference Empirical Methods Natural Language Processing, pages 3911 1-3921, Brussels, Belgium, October-November 2018. Association for Computational 1425. URI.tttps. ptttpa://aalaanhhllogg.. wwwpatthhooooyoooooooggg Dtrg/D18-1 1425
Linguistics. doi: 10.
Zhang, Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding, 2024. URL httl/httt:///raae http:///xxvvvo////// abs/ abs/2309 001688
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Reprexentations, 2024.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. htttt:::://aarii wwwpaaawwww..aa///a//a////// 05685
13

---

A Technical Appendices and Supplementary Material
A.1 Details for Main Experiments
A.1.1 Experiment setup details
Setup of Spec-Bench experiments (Sec. 4.2). We conducted our Spec-Benchs experiments by rumning the Spec-Bench Bench codebase from the original repository with the following modifications. library, which is
First, we updated the code to work with the latest version of the required to run recent open-source L.LMs such as meta-llama/Le aataalllaaaaaaaaa-..... We also added support for arbitrary datasets (such as SWE, W--Benchh and AgenticSQL) and implemented SuffixDecoding within the framework. We ran the experiments on 8xH100 80GB GPU cluster, with ITB RAM. We ran each baseline using one GPU, and a batch size of 1, just like in the original SpecBench code.
Setup vLLM SWE-Bench experiment (Sec.
We conducted the end-to-end SWE- 8xH100 80GB GPU cluster, with 1TB RAM. We served the
Bench experiment h01-baands//ppnhaddds -am-anddd----mm--------- 1m-mmmmvv.....mm 1-ep3 model locally using vL..M, with a tensor parallelism degree of 4 and with prefix caching enabled. We used the flashinfer kernels for sampling. We made some minor modifications to vLLM to record the per-request and per-step statistics of interest (time-per-token latency, throughput, acceptance length, acceptance rate). We used the same settings for all baselines. We ran the OpenHands daemon on the same machine, and used the OpenAl API to interact with the vLLM server. We OpenHands with the CodeActAgent (Wang al. 2024b] with ITERATIVE_EVES MODEE and a maximum of 100 iterations, as recommended by the OpenHands authors. We used maximum of 16 concurrent workers to run the SWE-Benn BeEchBhcch tasks.
A.1.2 Detailed sub-task results
The following tables present detailed performance metrics across all evaluation benchmarks. For Spec-Bench, the 13 categories include both single-turn open-ended tasks (qa, rag, summarization, translation) and multi-turn tasks. Fight of these categories (Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, Humanities) from MTT-Bench [Zheng et al. 2023 allowing direct performance comparison on this widely-used benchmark.
14

---

SWE-Bench: Mean accepted tokens (tokens/step)
System
astropy
django
malplotlib
scaboen
requests
xarra
pylint
pylest
sphinxx
sympy
suffix (linear)
6.415
6.346
5.521
7.2017
3.772
6.480
7.137
4.635
5.412
5.933
17.140
5.165
7.821
sullix (tree)
6.262
6.221
4.992
7.064
3.708
5.922
6.764
4,452
5.311
5.600
16,876
5.020
7.552
pld
3.008
2.756
3,080
2.195
3.223
4.724
2.641
recycling
3.159
3,058
3.004
3,133
2.978
3.072
2.992
3.046
2.994
3,054
vanilla
1.000
1,000
1,000
1,000
1.000
1.000
1,000
1.000
1,000
1.000
1,000
1.000
1,000
engles
cuglc2
cagle
by brid
SWE-Bench: Mean Acceptance Rate
System
astropy
Ojango
matplotiib
scalorn
Dlak
requests
xarray
pylint
pylest
solkit-lcam
sphinx
-ympy
Overall
saffix (linear)
0.252
0.255
0.238
0.293
0.167
0.250
0.281
0.204
0.230
0.243
0.554
0.217
0.296
suffix (tree)
0.183
0.216
0.539
0.196
pld
0.191
0.210
0.184
0.215
0.128
0.208
0.231
0.174
0.199
0.175
0.379
0.175
0.225
recycling
0.028
04227
0.026
0.028
0.025
0.027
0.025
0.026
0.026
0.026
sullix-1 1.ixxx----trree O-trree
0.385
0.381
0.348
0.420
0.330
0.388
0.409
0.349
0.375
0.375
0.632
0.373
0.421
SWE-Bench: Time per output token (ms)
System
astropy
matplotibb
seabeen
flask:
requests
xarroy
pylint
pylest
scikit-lcss
sphinx
sympp
Overall
suftii (linear)
37:012
17,072
27.436
suffix (mree
30,800
238
26.278
37.095
22.728
23.998
23.847
28. 388
29.451
32.496
25.855
17.199
27.211
pld
41.675
31.029
34.447
47.588
31.549
31.155
31,330
37.386
38.238
41.531
41.877
25.142
37.758
recycling
41.328
31.847
34.155
49.670
33.434
34.304
38.438
39.667
78.153
43.774
43.019
57.069
39.268
41,213
46.061
77.815
32,527
50.074
engle3
eagle2
cagle
hybrid
SWE-Bench: Speculation time per generated token (ms)
System
astropy
django
matplotish
scahorn
flask:
requests
xammy
pylint
pytest
scikit-leam
sphinx
sympy
Overall
0.000
0.000
0.000
0.000
suffix (linear)
0.156
0.122
0.175
0.185
0.198
0.182
0.178
0.203
0.217
0.203
0.125
0.167
0.171
suflix (iree)
0.170
0.135
0.172
0.172
0.191
0.165
0.165
0.235
0.235
0.212
0.135
0.150
0.175
pld
0.191
0.197
0.208
0.176
0.266
0.200
0.191
0.222
0.208
0.228
0.126
0.217
0.191
recycling
9.298
6.023
7.982
13.731
6.607
7.665
8.081
11.748
19.653
10.582
cagic3
eagic?
cagle
byhrid
SWE-Bench: Speedup over vanilla decoding
System
astropy
django
matplotiib
flask
requests
xarray
pylint
pytest
scikit-leam
sphinx
sympy
Overall
suffix (linear)
2,213
2.442
2.039
1.951
1.792
1.962
2:210
1.792
1.858
2.356
4.453
2:292
2.452
suffix (tres)
2.158
2.412
1.991
1.985
1.759
1.972
2.190
1.754
1.912
2,296
4.417
2.280
2.433
pld
1,360
recycling
1.427
1.458
1.483
1,264
1.360
1.311
1.290
1.399
1.328
1.358
vanilla
1.000
1.000
1,000
1.000
1.000
1,000
1,000
1,000
1.000
1,000
1.000
1.000
L.000
eagle3
eaple?
cagle
bybrid
15

---

AgenticSQL: Mean accepted tokens (tokens/step)
System
Classify
Extract
Enrich
Combine
SQL1
SQL2
SQL3
Overall
hybrid (tree)
4.180
14.813
15.081
6.448
3,983
3,932
4.006
7.500
hybrid (linear)
4.008
13.473
15.304
6.362
3.876
3,842
3.913
7.262
suffix (linear)
3.577
11.833
12.395
5.924
3.665
3.005
4.005
6.349
suffix (tree)
3.470
11.724
12.137
5.834
3.633
2.914
3.904
6.236
eagle2
3.156
5.173
3.249
3.534
3.066
3.761
3.057
3.572
recycling
2.915
4.125
3.125
3.138
2.951
2.929
2.994
3.169
eagle3
2.529
3.374
4.198
3.179
2.142
4.622
2.056
3.160
eagle
2.305
4.062
2.877
3.127
2.166
3.295
2.109
2.851
pld
1.427
4.134
1.455
3.914
2.074
1.452
2.151
2.373
vanilla
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
AgenticSQL: Mean Acceptance Rate
System
Classify
Extract
Enrich
Combine
SQL1
SQL2
SQL3
Overall
vanilla
suffix (linear)
0.212
0.628
0.740
0.428
0.318
0.245
0.330
0.415
suffix (tree)
0.189
0.605
0.614
0.397
0.294
0.225
0.298
0.375
hybrid (tree)
0.131
0.642
0.683
0.249
0.138
0.143
0.140
0.304
hybrid (linear)
0.124
0.595
0.750
0.242
0.130
0.139
0.133
0.302
pld
0.061
0.365
0.069
0.355
0.137
0.076
0.144
0.173
eagle
0.052
0.122
0.075
0.085
0.047
0.092
0.044
0.074
eagle2
0.036
0.070
0.037
0.042
0.034
0.046
0.034
0.043
cagle3
0.025
0.040
0.053
0.036
0.019
0.060
0.018
0.036
recycling
0.025
0.041
0.028
0.027
0.025
0.024
0.026
0.028
AgenticSQL: Time per output token (ms)
System
Classify
Extract
Enrich
Combine
SQL1
SQL2
SQL3
Overall
suffix (lincar)
9.552
2.687
3.007
6.023
9.559
10.588
9.767
7.306
suffix (tree)
9.594
2.876
3,188
6.164
10.339
10.935
10.090
7.592
hybrid (tree)
11.975
3.491
3.633
7.883
14.329
11.600
14.399
9.604
recycling
10.316
7.314
9,470
9.724
10.981
9.981
10.702
9.782
hybrid (linear)
12.563
3.681
3.762
8.603
14.827
11.639
21.219
10.874
cagle2
16.814
9.210
15.078
13.877
17.967
12.312
17.531
14.677
pld
20.146
7.735
20.321
8.173
14.605
20.546
14.646
15.169
eagle
21.221
11.308
15.760
15.688
23.353
13.396
24.595
17.887
eagle3
23.334
14.729
12.513
16.863
26.997
10.747
27.728
18.966
vanilla
26.578
25.292
25.032
25,406
27.011
25.851
26.307
25.924
AgenticSQL: Speculation time per generated token (ms)
16

---

System
Classify
Extract
Enrich
Combine
SQL1
SQL2
SQL3
Overall
vanilla
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
suffix (linear)
0.060
0.015
0.015
0.033
0.059
0.058
0.061
0.043
suffix (tree)
0.064
0.017
0.020
0.035
0.064
0.062
0.064
0.047
pld
0.419
0.124
0.422
0.130
0.281
0.434
0.276
0.298
recycling
0.762
0,303
0.399
0.577
0,928
0.260
0.922
0.592
hybrid (tree)
1.728
0.249
0.376
1.112
2.350
0.927
2.372
1.299
hybrid (linear)
1.776
0.256
0.358
1.177
2.387
0.930
4.389
1.604
eagle
3.189
1.693
2.423
2.372
3.423
2.197
3.616
2,700
eagle2
4.118
2.006
3.292
3.308
4.599
2.597
4.509
3,487
eagle3
6.092
3.526
3.021
4.350
7.156
2.543
7.332
4.854
AgenticSQL: Speedup over vanilla decoding
System
Classify
Extract
Enrich
Combine
SQL1
SQL2
SQL3
Overall
suffix (linear)
3.016
9.854
10.406
4.848
3.205
2.839
3.211
5,345
suffix (tree)
2.998
9.545
10.009
4.765
3.008
2.733
3.133
5.175
hybrid (tree)
2.338
7.672
8.191
3.613
2.057
2.496
2.077
4.068
hybrid (linear)
2.243
7.137
7.965
3.327
1.993
2.483
1,405
3.799
recycling
2,588
3.472
2.672
2.640
2.502
2.604
2.492
2.710
pld
1.330
3.695
1.255
3.298
1.936
1.311
1.905
2.105
eagle2
1.591
2.751
1.673
1.855
1.527
2.119
1.527
1.864
cagle3
1.328
1.720
2.025
1.619
1.108
2.496
1.056
1.623
eagle
1.324
2,246
1.598
1.669
1.229
1.955
1.138
1.595
vanilla
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
17

---

Spec-Bench: Mean accepted tokens (tokens/step)
Sydtem
coaling
humeniitiss
math
math
mingly
translation
Overlll
hyhnn (tee)
0,325
7.418
4,990
5.507
5.181
4.558
4.512
6.031
4.556
5.155
4,55
3,432
5.300
4.086
oules
5,975
5.373
5.180
5.745
3.582
4.351
5,007
4.936
4.664
5.299
4.787
2.009
5.093
4.647
bybad (incar)
5.055
3.2.9
4.365
5.446
5.121
4.452
4.753
4.637
4.557
5.089
4.520
3345
5.158
4.553
1842
1.551
1.244
2800
cuple
4,149
3.728
2968
1331
23925
2352
recycling
2417
1.757
2.161
1331
1.252
1.455
1.725
1.705
1.765
1,900
1.753
2.138
14.38
1850
1.259
1,703
1,705
1.438
1,750
1.911
1.6.1)
1.800
1.430
1.302
1.3M
vanille
1,000
1.000
1,303
1,000
1.000
1,000
1,000
1.000
1,000
1.000
1,000
1.000
1.000
1,000
Spec- :reccBBnch: Mean Acceptance Rate
System
colling
Extraction
humatities
reatlreasonnng
scasoning
nieplay
sunmatizaion
trasslation
willig
Overall
sutiss (liac))
0.255
0.216
0.162
0.240
0.163
0.152
0.216
0.171
0.116
0.169
0.199
0.216
0.218
0.190
vanille
sutte (mro)
0:244
0:211
0.155
1.212
0.301
8.156
8.115
0.189
0.208
0.194
0.132
0.103
0:125
0.799
0.077
1.100
0.075
0.44
bybnd
0.107
0.072
0.101
0.04
0.99
0.022
0.08%
0.045
0,77
3,579
sagle3
0,083
0,019
0.063
0.51
0.088
culle2
0.063
0.847
0.044
0.054
0,053
0.839
0.044
0.048
0.039
0.946
0.088
0.027
0.040
0,941
recycling
0.005
0.030
0.019
0.027
01025
0.017
0.017
01000
0.017
0021
0.009
0.117
0.118
0,030
Spec-Bench: Time per output token (ms)
System
onding
Insrnnitce
mu3h
stath rossuting
neeuming
milephay
Irmelation
Writin
Overall
(lacary
9.749
13,475
12.057
13.373
11.947
9.221
11.256
12.822
11.316
14.905
13.631
14.239
12.451
14:473
19.116
13992
14:58T
cigle
10.201
13,193
13,570
11.455
11,835
16.299
14.222
13.215
14,539
13:022
14.542
19:415
14,290
14.925
collis (iad
13.83
16-384
18,770
13:547
16.33 10.306
19.819
18.363
21. 11.090
18034
15.825
16.50 08.955
18.15
16805
Sumti (inear)
13,595
16.245
18.212
13.292
16,447
19.999
37.
18,061
21.217
18941
15.925
16,670
18925
16,95
25,052
25.114
Spec-Bench: Speculation time per generated token (ms)
System
codting
ostraction
hrmmnittiss
roasining
933
mig
reasoning
rolelay
simmanizatis
tratslation
wnitig
vatille
0,000
0.010)
0,000
0.000
0.000
0.000
0.000
0.000
0.000
0,000
sulli (liaze)
0.071
0.079
0.099
0.067
0.087
0.001
0.087
0.084
0.101
0.024
0.092
0.076
0.084
0.886
suttix (tree)
0.074
01080
0.108
11089
11891
0.889
0.093
01058
0.110
01977
0.095
01779
0.086
9.011
recycling
0.283
0.318
0.291
11240
11255
0.391
0.419
0130
0.320
0.275
0.321
0.365
0.310
0.500
plá
0.291
0.357
0.421
0.277
0.392
0.480
0.326
0.391
0,492
0.407
0.335
0.466
0.28
0.395
hyhnid (incar)
1.808
2,455
23883
1995
3.118
3000
ciple
1.977
1.877
2.421
1.166
2385
bybnd
1.013
2.001
2.500
2,402
2.357
cxplc2
2.055
2.997
2.628
3.062
3.826
Spec-Bench: Speedup over vanilla decoding
codlng
humaniiicss
nocglay
byboid (incar)
3,441
2.755
2,500
3,139
2,856
2.404
2.542
2,406
2.461
2.871
2.575
1.250
2.833
2,500
hyhnd (tre)
3.442
2,717
2.544
3.132
2.730
2,611
2.445
2.344
2.330
2.732
2.485
1.213
2.736
2.455
eigle3
3.115
2A55
2.113
2.910
2.353
2.172
2.474
2,866
2.394
2.706
1.500
1.467
2,22
2.367
rocsdingg
2,619
2.554
2.309
2.600
2.577
1.551
2,087
2,150
1.799
2.334
2.215
1932
2,057
2.169
oglle
1.908
1.358
2.255
1.731
1.573
1:251
2.077
1:755
1.3%6
1:791
melle
2,427
1.9%
1:00
1933
1.305
1.620
1.628
1,350
1.386
1.327
1.712
1.484
1.753
1.320
1.255
1.219
vanillz
1,000
1.020
1.000
1.000
1.000
1,000
1.000
18

---

Spoodup Fautor
2.5
2.0
WilcChal - Spider
SpideršOL Baseline
100
200
300
400
500
Number of SpiderSigl. Exumpiee
Amtrgtaace Sttte
0.2
WildChat SpiderQLL
SpiderSQL Baselino
0.0
500
Number of SpidersIQL thamples
Figure 9: The performance DecoiiDg under input distribution shift. Suffixeeoding was trained on outputs from WildChat, while being evaluated on SpiderSQL. X axis: the number of SpiderSQL inputs, which are added to the global suffix tree after they are processed. Red line: the performance of SuffiDeecodig if trained on 500 output examples from only SpiderSQL offline.
A.2 Additional Ablation Experiments
In this appendix, we share ablation studies that reveal the impact of several design decisions in SuffixDecoding. The studies are conducted using the simulated verifier described in Sec. 4.1
A.2.1 Additional Dataset Details
We performed additional ablation experiments, which used additional datasets described below.
1. WildChat [Zhao et al. 2024]. We use instructions from the WildChat dataset, which consists real-world interactions between users and the ChatGPT service. WildChat represents most diverse and apen-domain dataset used in our evaluations.
2. Magicoder [Wei et al., 2023 Specifically, we use instructions from the Magicoder-Ewve -vol- Instruct- 10K dataset, which consists of code-related questions and instructions generated via Self-Instruct [Chaudhary 2023 Wang et al. 2023b and further augmented for difficulty
and diversity [Luo et al. 2023 by GPT-4.
3. SpiderSQI. Spider Yu et al. 2018 is dataset of manually-annotated questions and SQL responses over 200 different databases with multiple tables, covering 138 different domains. We use instructions from DAIL-SQL [Gao al. 2024a]. which consists of LLM prompts with instructions to answer questions from Spider using structured SQL code.
A.2.2 Effect of input distribution shift
In real-world LLM serving, the input characteristics of requests may change time, and may be out-of-distribution from the output examples that SuffixDecoding was trained on. To evaluate this scenario, run SuffixDecoding trained WildChat outputs, and begin to send it inputs from SpiderSQL, which represents a very sudden distribution shift.
Fig. 9 shows the results. SuffiDeeoding starts from having 4,000 output examples from WildChat, and begins to receive SpiderSQL inference requests. Without any adaptation, SuffixDecoding still achieves 1.5x speedup and 8% acceptance rate, but is far from the 2.6x speedup and 20% acceptance rate it would achieve if it were trained on 500 examples from SpiderSQL instead.
After processing each SpiderSQL inference request, SuffiDe can insert its output into its global suffix tree, which means can adapt in online fashion to the new input distribution. As Fig. shows, the performance of SuffixDecoding improves with the number of SpiderSQL inference requests processed. Perhaps surprisingly, after observing 500 SpiderSQL. and adapting online, SuffixDecdding' performance is almost indistinguishable to its performance if it were trained offline on the 500 SpiderSQI. examples alone. This suggests that Suffixeecoding is able adapt to input distribution shifts quickly and at loss in performance.
19

---

A.2.3 Predicting SuffixDecoding Effectiveness
SuffiDecoding tends to perform better on more structured tasks compared to very open-ended ones (e.g., AgenticSQL vs WildChat). We can measure this "structuredness" using empirical entropy. The steps are as follows: (1) create suffix tree from example model outputs (100 examples is typically enough), (2) calculate the entropy of each node's output distribution by determining how often each child node is accessed, and (3) compute weighted average of this entropy across all nodes. A low average entropy indicates that output tokens are more predictable based on their prefixes, which generally suggests that Suffix Decoding will perform better.
Table Measured empirical entropy of our various evaluation datasets.
Dataset
Average Entropy
AgenticSQI. (Enrich)
0.171
AgenticSQL (Classifyy
0.738
AgenticSQL (Extract)
0.0862
AgenticSQL (SQLI)
1.52
AgenticSQL. (SQL2)
1.49
AgenticSQL. (SQL3)
1.51
AgenticSQL (Combine)
1.49
Spider
2.50
WildChat
3.43
Magicoder
2.95
Table shows the empirical entropy measured samples from each of our evaluation datasets. We find that the average entropy is closely related to the intuitive understanding the "structuredness" of each dataset. Additionally, it correlates well with the performance of SuffiXecodiin those datasets. Therefore, practitioners can calculate this value using a small number of output examples to assess whether SuffiDecooding is appropriate for their specific tasks.
B Scalability and Memory Overhead
The suffix trees employed by Suffix Decoding are highly time and memory efficient. Table shows the per-token lookup time, update time, and memory consumption across various tree sizes (using requests from the SWE-Bench dataset).
Table 2: Performance metrics for different entry counts
Total entries
Total # tokens
Memory footprint (MB)
Avg update time per token (ps)
Avg lookup time
per speculated token (us)
K entries
27 M
1670 MB
3.95
12.18
5 K entries
143 M
2980 MB
4.06
10.93
K entries
285
4070 MB
4.05
12.00
15 K entries
429 M
5110 MB
4.07
11.62
20 K entries
572
6150 MB
4.04
11.64
Overall, the total memory consumption scales linearly with the size of the tree, and both the update and lookup times remain fast even with larger trees. Given typical server configurations, SuffixDecoding cache many weeks of generated tokens before hitting CPU memory limits. For example, optimized inference engines running Llama-8B on an A100 GPU can typically generate ~ 5000 tokens per second (1 Y et al., 2024)) )) 432M tokens per day. Typical A100 systems (e.g. AWS 24d.24xlarge) have 144GB of CPU memory per A100 GPU. This means that Suffix Decoding can potentially cache 31 days (see math below) of generated tokens per A100 GPU before hitting CPU memory limits. Beyond this limit, it is also straightforward to incorporate cache eviction (e.g. LRU) into Suffix Decoding avoid out-of-memory problems.
Time to fill up GPU memory:
1. Memory per token: 6.15 GB + 572M tokens 1.075 x 10 GB/token
20

---

2. Total token capacity: 144 GB + (1.075 x 10 GB/token) = 1.34 x 10 tokens
3. Time to fill: (1.34 x 10th tokens) (5000 tok/s) 2.68 x 10 seconds = 31.0 days
B.1 Hybrid Fallback Threshold Sensitivity Analysis
The hybrid Sucoing approach uses fallback threshold determine when to use suffix- tree-based speculation versus falling back to a model-based speculator (e.g., EAGLB-3). This section analyzes the sensitivity performance to different threshold values across both open-ended and agentic workloads.
Table shows the wall-clock speedup results on Spec-Bench for various threshold values. For open-ended generation tasks, we find that setting close to or slightly exceeding the mean accepted tokens (MAT) of the model-based speculator yields the best performance. For instance, EAGLE-3 achieves a MAT of approximately 4.65 tokens/step on Spec-Bench. The results show that r E [5, 7 produces the highest overall speedups (2.5x), with performance being relatively stable across this range.
Table 3: Hybrid SuffixDecoding speedup on Spec-Bench across different threshold values
coding
qa
math
reasoning
stem
writing
Overall
7
3.44
2.40
3.14
2.50
2.87
2.83
2.50
6
3.19
2.22
2.88
2.30
2.68
2.61
2.31
5
3.25
2.28
2.70
2.35
2.74
2.70
2.37
4
2.22
2.72
2.57
2.64
2.25
3
2.88
2.09
2.54
2.10
2.39
2.50
2.13
2
2.73
2.04
2.40
2.01
2.19
2.36
2.03
1
2.28
1.86
2.01
1.69
1.90
2.12
1.80
(suffix only)
1.83
1.77
1.39
1.39
1.33
1.66
Conversely, Table 4 shows results for AgenticSQL, highly repetitive agentic workload. Here, the standalone SuffixDecoding (r = 0) achieves the best overall speedup (5.35x), as it can confidently speculate much longer sequences than model-based methods. Higher threshold values progressively degrade performance as the system increasingly falls back to less effective model-based speculation.
Table 4: Hybrid SuffixDecoding speedup on AgenticSQL. across different threshold values 7.
Classify
Extract
Enrich
Combine
SQI.
Overall
0 (suffix only)
3.02
9.85
10.41
4.85
3,09
5.35
1
2,31
7.03
8.37
3.38
2.17
3.95
2
2.24
7.14
7.97
3.33
2.29
3.80
3
2.22
5.51
5.68
3.52
2.20
3.22
7
2.06
6.79
7.10
3.44
2.29
3.66
In summary, practitioners should set ze MAT for mixed open-ended workloads, where MAT the mean accepted tokens of the fallback model-based speculator. For highly repet- itive agentic workloads, using SuffixDecoding alone (r = O or very low values) yields the best performance.
C Batch-level Speculation Control
For batched serving scenarios, optimizing the speculation per request in batch is crucial for many practical online deployments. While Suffixeecdding focuses on what tokens to speculate, it is also compatible with existing works that explore how much to speculate per request. For example, TurboSpec [Liu et al.) 2024 uses "goodput" metric to balance speculation lengths and batch size, noting that optimal speculation decreases as batch size increases. AdaServe [Li et al. 2025b) formulates multi-$I.O serving constrained optimization problem and introduces SI.O-customized speculative decoding that constructs speculation trees tailored each request's latency target. By dynamically adjusting the number of speculative tokens generated by the Suffix Tree per request,
21

---

SuffixDecoding can be integrated with TurboSpec to maximize the batch-wise goodput metric. Furthermore, the statistics-based scoring used by Suffix Decoding can potentially better inform methods like TurboSpec. For example, choosing to speculate more from sequences that have a higher marginal score. These are interesting and important directions for future work.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The technical description is accurate, and the claimed numbers are exactly as presented in the main evaluation section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our paper evaluates SuffiDeeodiig Spec-Bench, workload that it is not designed for, to show its limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Appendix A.I.I
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ 'nipsscc/puullc//uuu /ttpe//gguudeeeeeeouuuuumbbbba ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Appendix A.I.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In Sec. 4.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::///aarrrp...cpppppp//hhhaa
Answer: [Yes]
Justification: This paper does not curate any new data, train any new models, use human subjects, or risk negative societal impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper does not curate any new data, train any new models, use human subjects, or risk negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release any models or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets and basclines used properly cited in the body of the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
28

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
29


Reviews:
[{"review_id": "I3m36iNJC0", "reviewer": "Reviewer_WAC7", "summary": "The paper introduces SuffixDecoding, a model-free speculative decoding framework tailored for agentic LLM pipelines where generations exhibit high repetitiveness and structured loops. Instead of relying on a smaller draft model, SuffixDecoding constructs a suffix tree over past prompts and completions to identify the longest matching context at each decoding step. It then greedily expands a “speculation tree” of frequent continuations, verifies the selected prefix in a single forward pass of the full LLM, and adjusts its maximum speculation length dynamically based on match depth. Evaluated on two real-world agentic benchmarks (SWE-Bench, AgenticSQL) and in live vLLM, SuffixDecoding achieves up to 3.9× speedups on offline tasks and 4.5× end-to-end gains, while preserving exact token fidelity.", "questions": "1. Latency and profiling\n- Please report real-world latency and energy consumption comparisons between SuffixDecoding and a standard draft-model speculative decoder on GPU. This will verify that FLOP savings translate into actual system-level speedups.\n\n2. Memory overhead\n- How large do the suffix trees grow in practice, and what are the RAM and update-time costs under concurrent agentic requests? Empirical profiling or a detailed complexity analysis would clarify deployment feasibility.\n\n3. Fallback threshold\n- The hybrid threshold τ governs when to abandon tree-based speculation. Can you characterize its sensitivity across mixed workloads? A small study varying τ on open-ended versus highly repetitive streams would help practitioners tune the system.\n\n4. Generalization beyond Agentic workloads\n- Have you evaluated SuffixDecoding on less structured or single-turn tasks (e.g., open-ended question answering)? Understanding its failure modes and identifying break-even points would guide broader application.\n\n**Note: In particular, providing detailed latency and memory-overhead measurements across diverse tasks, along with a thorough analysis of τ’s sensitivity, will likely eliminate any remaining concerns and could substantially raise the paper’s overall score.**", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Quality\n* The authors compare against both model-based speculators (EAGLE-2/3) and other model-free baselines (Token Recycling, Prompt Lookup), demonstrating consistent speed and throughput gains. – The paper reports FLOP-based efficiency but omits wall-clock latency and energy measurements on GPU hardware, leaving system-level impact unquantified.\n\n### Clarity\n* The algorithmic pipeline is described clearly, with pseudocode guiding the reader through tree construction, speculation, and verification steps. Figures illustrate key data structures and control flow. – Details around memory management (suffix-tree storage, updates under concurrency) and fallback threshold tuning (τ) are only sketched in prose, making reproduction nontrivial.\n\n### Significance\n* By decoupling speculative decoding from auxiliary draft models, SuffixDecoding opens a new direction for model-free inference acceleration in settings where prompt/response patterns repeat. – Its applicability to more open-ended or non-agentic workloads remains unclear: acceptance rates vary and no clear decision rule is given for when to revert to traditional draft-model approaches.\n\n###Originality\n* The use of suffix trees for speculative decoding is novel and exploits structural redundancies in agent-driven loops. – Prior work has explored cache-based and retrieval-augmented inference; the paper could more sharply delineate how suffix-tree speculation fundamentally differs in algorithmic guarantees or failure modes.", "was_revised": true, "final_justification": "Well structured clarification to my previous weakness and questions.", "rebuttal": "Thank you for your detailed review and feedback, our response is below.\n\n**Latency and profiling (the paper reports FLOP-based efficiency but omits wall-clock latency and energy measurements on GPU hardware).**\n\nWe believe there is a misunderstanding regarding the metrics reported in our paper. To clarify, all speedup measurements in our paper are based on wall-clock latency, not FLOP counts. The speedup in Fig 4 is the ratio of real-world wall-clock time per output token measured on H100 GPUs. For example, in SWE-Bench, Suffix Decoding achieves 31.093 ms per token vs vanilla's 49.836 ms per token (see Appendix A.1.2), resulting in a speedup of 49.836 / 31.093 \\= 1.6x.\n\nFLOPs and energy consumption are less relevant for evaluating speculative decoding methods because speculative decoding typically expends *more* FLOPs (via speculation) to obtain *lower* wall-clock latency (via parallel verification).\n\n**Memory overhead: How large do the suffix trees grow in practice, and what are the RAM and update-time costs under concurrent agentic requests?**\n\nThe suffix trees employed by Suffix Decoding are highly time and memory efficient. To illustrate this, we present a microbenchmark below showing the memory footprint and per-token update cost across various tree sizes (using requests from the SWE-Bench dataset). We will include this additional microbenchmark in the revised paper.\n\n| Total \\# entries | Total \\# tokens (millions) | CPU Memory (MB) | Avg update time per token over last 10 entries (μs) |\n| :---- | :---- | :---- | :---- |\n| 1 K entries | 27 M | 1670 MB | 3.95 |\n| 5 K entries | 143 M | 2980 MB | 4.06 |\n| 10 K entries | 285 M | 4070 MB | 4.05 |\n| 15 K entries | 429 M | 5110 MB | 4.07 |\n| 20 K entries | 572 M | 6150 MB | 4.04 |\n\nOverall, the memory consumption scales linearly with the size of the suffix tree, while the update time remains consistently low. For comparison, optimized inference engines running Llama-8B on an A100 GPU can typically generate \\~5000 tokens per second \\[1\\] or \\~432M tokens per day. Typical A100 systems (e.g. AWS p4d.24xlarge) have 144GB of CPU memory per A100 GPU. This means that Suffix Decoding can potentially cache 31 days of generated tokens per A100 GPU before hitting CPU memory limits. Beyond this limit, it is also straightforward to incorporate cache eviction (e.g. LRU) into Suffix Decoding to avoid out-of-memory problems. \n\n\n\n**Generalization beyond Agentic workloads: Have you evaluated SuffixDecoding on less structured or single-turn tasks (e.g., open-ended question answering)?**\n\nTo clarify, the Spec-Bench dataset used in our evaluation (Sec 4.1, Fig 4\\) consists of mainly non-agentic, open-ended tasks (e.g. single-turn question-answering). We included this dataset explicitly to stress-test Suffix Decoding in a scenario where *we did not expect it to perform well*, as evaluating performance under these extreme scenarios is important to understand the limitations and robustness of Suffix Decoding.\n\nThe tables in appendix A.1.2 detailed performance results across all 13 Spec-Bench categories. For example, the `qa` (question answering) column represents open-ended, single-turn questions. Among the 13 categories, `qa`, `rag`, `math_reasoning`, `summarization`, and `translation` are single-turn tasks.\n\nAlthough Suffix Decoding is not specifically designed for these open-ended tasks, it can be combined with other speculative decoding methods in a *hybrid* mode to achieve the best of both worlds. We have performed additional experiments evaluating this hybrid-mode, which we present below.\n\n**SpecBench \\- Speedup (x)**\n\n| System | coding | extraction | humanities | math | math\\_reasoning | qa | rag | reasoning | roleplay | stem | summarization | translation | writing | Overall |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| hybrid (τ=7) | 3.441 | 2.795 | 2.800 | 3.139 | 2.866 | 2.404 | 2.542 | 2.496 | 2.461 | 2.871 | 2.573 | 1.759 | 2.833 | 2.500 |\n| eagle3 | 3.115 | 2.634 | 2.713 | 2.910 | 2.888 | 2.172 | 2.474 | 2.446 | 2.394 | 2.764 | 2.520 | 1.447 | 2.622 | 2.367 |\n| recycling | 2.619 | 2.234 | 2.209 | 2.660 | 2.577 | 1.951 | 2.057 | 2.150 | 1.979 | 2.334 | 2.218 | 1.932 | 2.057 | 2.169 |\n| eagle2 | 2.548 | 1.998 | 1.958 | 2.215 | 2.220 | 1.733 | 1.877 | 1.968 | 1.751 | 2.007 | 1.756 | 1.336 | 1.791 | 1.825 |\n| eagle | 2.427 | 1.956 | 1.848 | 2.142 | 2.112 | 1.600 | 1.841 | 1.871 | 1.703 | 1.922 | 1.702 | 1.305 | 1.753 | 1.752 |\n| suffix | 1.828 | 1.630 | 1.383 | 1.967 | 1.544 | 1.773 | 1.890 | 1.389 | 1.156 | 1.386 | 1.618 | 1.618 | 1.327 | 1.659 |\n| pld | 1.712 | 1.484 | 1.231 | 1.753 | 1.323 | 1.326 | 1.807 | 1.320 | 1.055 | 1.275 | 1.628 | 1.219 | 1.232 | 1.448 |\n| vanilla | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |\n\n**AgenticSQL \\- Speedup (x)**\n\n| System | CATEGORIZATION | FEATURE\\_EXTRACTION | QUESTION\\_SUGGESTION | SQL\\_COMBINE | SQL\\_FANOUT1 | SQL\\_FANOUT2 | SQL\\_FANOUT3 | Overall |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| suffix | 3.016 | 9.854 | 10.406 | 4.848 | 3.205 | 2.839 | 3.211 | 5.345 |\n| hybrid (τ=1) | 2.309 | 7.025 | 8.368 | 3.382 | 2.024 | 2.421 | 2.072 | 3.947 |\n| recycling | 2.588 | 3.472 | 2.672 | 2.640 | 2.502 | 2.604 | 2.492 | 2.710 |\n| pld | 1.330 | 3.695 | 1.255 | 3.298 | 1.936 | 1.311 | 1.905 | 2.105 |\n| eagle2 | 1.591 | 2.751 | 1.673 | 1.855 | 1.527 | 2.119 | 1.527 | 1.864 |\n| eagle3 | 1.328 | 1.720 | 2.025 | 1.619 | 1.108 | 2.496 | 1.056 | 1.623 |\n| eagle | 1.324 | 2.246 | 1.598 | 1.669 | 1.229 | 1.955 | 1.138 | 1.595 |\n| vanilla | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |\n\nHere, we used a fallback threshold of τ=7 for SpecBench, and τ=1 for AgenticSQL. For more open-ended tasks such as QA (question answering), the hybrid approach with Suffix Decoding outperforms the speculative decoding method alone. This indicates that *while Suffix Decoding can greatly improve agentic applications, it does not have to come at the expense of other open-ended applications*.\n\nNote that all baselines are slightly improved compared to the table in our submission due to performance improvements in the benchmark framework. \n\nGiven space limitations, we are presenting only the wall-clock speedups compared to the baseline. A comprehensive breakdown of results, including per-token latency (TPOT), mean accepted tokens per step (MAT), mean acceptance rate, and speculation time per generated token, will be incorporated into the revised paper.\n\n**Hybrid Fallback Threshold: Can you characterize its sensitivity across mixed workloads?**\n\nWe report the results of SpecBench and AgenticSQL experiments using the Hybrid Suffix Decoding method with various thresholds. In the table below, we show the measured wall-clock speedups with respect to the vanilla (no speculation) baseline.\n\n**SpecBench \\- Speedup (x)**\n\n| System | coding | extraction | humanities | math | math\\_reasoning | qa | rag | reasoning | roleplay | stem | summarization | translation | writing | Overall |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| τ=7 | 3.441 | 2.795 | 2.800 | 3.139 | 2.866 | 2.404 | 2.542 | 2.496 | 2.461 | 2.871 | 2.573 | 1.759 | 2.833 | 2.500 |\n| τ=5 | 3.247 | 2.701 | 2.682 | 2.702 | 2.605 | 2.280 | 2.402 | 2.351 | 2.358 | 2.735 | 2.461 | 1.763 | 2.697 | 2.366 |\n| τ=6 | 3.186 | 2.595 | 2.614 | 2.875 | 2.567 | 2.216 | 2.360 | 2.296 | 2.304 | 2.675 | 2.380 | 1.719 | 2.610 | 2.314 |\n| τ=4 | 3.072 | 2.561 | 2.552 | 2.719 | 2.402 | 2.219 | 2.278 | 2.249 | 2.276 | 2.569 | 2.285 | 1.728 | 2.641 | 2.249 |\n| τ=3 | 2.875 | 2.407 | 2.352 | 2.536 | 2.195 | 2.086 | 2.166 | 2.099 | 2.161 | 2.387 | 2.161 | 1.733 | 2.503 | 2.126 |\n| τ=2 | 2.734 | 2.256 | 2.218 | 2.401 | 1.980 | 2.041 | 2.061 | 2.011 | 1.990 | 2.194 | 2.026 | 1.791 | 2.362 | 2.028 |\n| τ=1 | 2.284 | 1.977 | 1.935 | 2.014 | 1.620 | 1.858 | 1.870 | 1.694 | 1.745 | 1.897 | 1.786 | 1.723 | 2.117 | 1.802 |\n| τ=0 (suffix alone) | 1.828 | 1.630 | 1.383 | 1.967 | 1.544 | 1.773 | 1.890 | 1.389 | 1.156 | 1.386 | 1.618 | 1.618 | 1.327 | 1.659 |\n\n**AgenticSQL \\- Speedup (x)**\n\n| System | CATEGORIZATION | FEATURE\\_EXTRACTION | QUESTION\\_SUGGESTION | SQL\\_COMBINE | SQL\\_FANOUT1 | SQL\\_FANOUT2 | SQL\\_FANOUT3 | Overall |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| τ=0 (suffix alone) | 3.016 | 9.854 | 10.406 | 4.848 | 3.205 | 2.839 | 3.211 | 5.345 |\n| τ=1 | 2.309 | 7.025 | 8.368 | 3.382 | 2.024 | 2.421 | 2.072 | 3.947 |\n| τ=2 | 2.243 | 7.137 | 7.965 | 3.327 | 1.993 | 2.483 | 1.405 | 3.799 |\n| τ=7 | 2.059 | 6.794 | 7.096 | 3.440 | 1.692 | 2.812 | 1.714 | 3.663 |\n| τ=8 | 2.080 | 6.848 | 4.959 | 3.358 | 1.691 | 2.854 | 1.259 | 3.298 |\n| τ=3 | 2.217 | 5.505 | 5.679 | 3.517 | 1.913 | 2.256 | 1.424 | 3.220 |\n| τ=4 | 2.173 | 5.669 | 5.404 | 3.526 | 1.892 | 2.237 | 1.390 | 3.189 |\n| τ=5 | 2.023 | 5.211 | 5.139 | 3.575 | 1.868 | 2.376 | 1.326 | 3.078 |\n| τ=6 | 2.027 | 5.113 | 5.199 | 3.491 | 1.782 | 2.351 | 1.287 | 3.040 |\n\nFor open-ended generation, a good heuristic is to set the fallback threshold (τ) to a value close to or slightly exceeding the mean number of accepted tokens (MAT) for the model-based speculator used with the suffix-tree. For instance, we used EAGLE-3, which has a MAT of approximately 4.65 tokens/step in SpecBench. The table shows that τ values of 5, 6, and 7 yield the best overall speedups, with the results being quite similar across these values.\n\nConversely, in agentic workloads like AgenticSQL, the standalone SuffixDecoding implementation (τ=0) is optimal because it can confidently speculate much longer sequences than model-based speculators such as EAGLE-3.\n\n\\[1\\] (Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) | SGLang Team, Jul 25, 2024 | LMSYS Org)"}, {"review_id": "ETXHPGJBPs", "reviewer": "Reviewer_CbVA", "summary": "The paper introduces SuffixDecoding, a model-free speculative decoding method to save compute at inference time for LLMs by reusing tokens predicted on previous prompts/contexts.  The novelty introduced is in the trie/suffix tree which allows for lookup of past sequences (from  prompts or completions) in a dynamic / adaptive system.  Its highlighted use in agent-based systems with structured outputs is highlighted due to the type of repetition present in those settings that allows for the differentiated improvement of this method in storing/retrieving past tokens in such a system.  Like standard speculative decoding, the inference result is exact, with no accuracy loss, and only gains from leveraging the previous cache.  The experiments show significant speedups compared to existing methods in agent/structured settings.  The method has also been open-sourced and integrated into vLLM.", "questions": "1. How does lookup time, speedup, and scoring approximation (compared to best lookup) scale with the number of records? (within some distributional assumptions)\n2. Can you explain the title?\n3. 'Related Works' section is in a bit of a weird spot. Why not together with ‘Background’?  If you were trying to point out tradeoffs or highlight most recent works in particular, I suggest renaming the section.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n* Paper is clear and well-written\n* Motivation is clear, and emphasized through the mention of patterns that are particularly useful for the priors of this method: self-consistency, self-refinement, multi-agent workflows\n* Method:\n  * Simple: easy to understand, no added complexity that isn’t justified, model-free\n  * Practical/general: doesn’t require strict homogeneity of LLMs within the multi-agent setting or training an auxiliary model; demonstrated use outside of multi-agent systems as well (with the caveat of reduced benefit in these settings)\n  * Open-sourced integration into vLLM – good community contribution\n  * Handles the tradeoff in number of consecutive draft tokens vs likelihood of them being incorrect in an adaptive rather than purely heuristic fashion, adapting specifically to the distribution driven by that agentic system.\n  * Mentions practical additions, like maintaining two trees – one that’s global and one that’s smaller, per-request\n  * Significant gains, including comparison to and performance against EAGLE\n  * Admits to limitations compared to existing methods in certain settings\n\n## Weaknesses\n* The scoring/lookup is the key challenge to overcome and could have been highlighted/explained better\n* The title is strange to me. 'Extreme' in the title seems a bit *extreme* -- I'd expect that word only with orders of magnitude difference.  I also didn't understand what 'Emerging' means here.  Are you referring to agents/structured outputs?\n* Proprietary benchmarks will make it hard for the community to reproduce/compare\n* Doesn't address scenarios when cpu memory is filled -- many practical scenarios at scale when this might happen.", "was_revised": true, "final_justification": "The original review stands, and the decision is only strengthened by the degree to which the authors addressed each of my questions and concerns.", "rebuttal": "Thank you for your positive review and detailed feedback, our response is below.\n\n**Scalability of lookup time, speedup and scoring.**\n\nFor lookup time vs number of records, we provide an additional micro-benchmark of the suffix tree below, where we measure its lookup time, update time, and memory consumption for various tree sizes. We will include it in our final paper.\n\n| Total \\# entries | Total \\# tokens | Memory footprint (MB) | Avg update time per token (μs) | Avg lookup time per speculated token (μs) |\n| :---- | :---- | :---- | :---- | :---- |\n| 1 K entries | 27 M | 1670 MB | 3.95 | 12.18 |\n| 5 K entries | 143 M | 2980 MB | 4.06 | 10.93 |\n| 10 K entries | 285 M | 4070 MB | 4.05 | 12.00 |\n| 15 K entries | 429 M | 5110 MB | 4.07 | 11.62 |\n| 20 K entries | 572 M | 6150 MB | 4.04 | 11.64 |\n\nIn short, the total memory consumption scales linearly with the size of the tree, and both the update and lookup times remain fast even with larger trees.\n\nFor speculative speedup and accuracy vs number of records, we provided a brief ablation study in Fig 8\\. Speedup consistently improves as suffix tree size increases, demonstrating that more historical data leads to better performance. Importantly, acceptance rates remain stable (\\~10-20%) even as tree size varies dramatically, indicating that our adaptive speculation mechanism (MAX\\_SPEC \\= αp) effectively manages the quality-quantity tradeoff.\n\nIf there are specific questions beyond what’s provided in our ablation study, we are happy to answer during the discussion period.\n\n**Scenarios when CPU memory is filled up.**\n\nWe agree that the CPU memory can be filled up, but we would like to work through a practical scenario. Our micro-benchmark results above show that Suffix Decoding is highly memory efficient, caching 572M tokens using only 6.15 GB of memory, and scaling linearly. At the same time, optimized serving systems can generate on the order of \\~5000 tokens/s for Llama-8B on an A100 GPU \\[1\\]. Typical A100 systems (e.g. AWS p4d.24xlarge) have 144GB of CPU memory per A100 GPU. This means that Suffix Decoding can potentially cache 31 days of generated tokens per A100 GPU before hitting CPU memory limits (see math below). Beyond this limit, it is also straightforward to incorporate cache eviction (e.g. LRU) into Suffix Decoding to avoid out-of-memory problems.\n\nTime to fill up GPU memory: \n\n1. Memory per token: $$\\frac{6.15 GB}{572M tokens} = 1.075 \\times 10^{-8} GB/token$$\n2. Total token capacity: $$\\frac{144 GB}{(1.075 \\times 10^{-8} GB/token)} = 1.34 \\times 10^{10} tokens$$   \n3. Time to fill RAM: $$\\frac{1.34 \\times 10^{10} tokens}{5000 tok/s} = 2.68 \\times 10^{6} seconds =  \\boxed{31 days}$$ \n\n\n**Can you explain the title?**\n\nWe used the term \"extreme\" in reference to the significantly longer speculation sequences possible with suffix trees compared to typical speculative decoding, which is also crucial for exploiting repetition in agentic applications such as SWE-Bench and AgenticSQL. We used the term \"emerging\" in reference to the growing field of agentic AI applications that involve multiple steps in answering a user query, often with highly repetitive LLM calls. We will add additional explanations and a footnote to our introduction to explain these choices.\n\n**'Related Works' section is in a bit of a weird spot.**\n\nWe thank the reviewer for pointing out the non-standard location of the related works section and we will move it closer to the background section as suggested.\n\n\\[1\\] (Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) | SGLang Team, Jul 25, 2024 | LMSYS Org)"}, {"review_id": "9hoi0hFnIQ", "reviewer": "Reviewer_QNpe", "summary": "This paper introduces SuffixDecoding, a novel, model-free speculative decoding method designed to reduce LLM inference latency for agentic AI applications that produce repetitive and predictable sequences. SuffixDecoding uses efficient suffix trees to cache and rapidly match long token sequences from prompts and past outputs, enabling fast draft token generation without GPU overhead. A key innovation is its adaptive nature; it speculates more tokens when pattern matches are long and confidence is high, and fewer when matches are short, thereby conserving computational resources. Evaluations on agentic benchmarks like SWE-Bench and a Text-to-SQL pipeline show that SuffixDecoding achieves speedups of up to 3.9x, outperforming SOTA model-based (EAGLE-2/3) and model-free (Token Recycling) methods on these agentic tasks.", "questions": "It would be great if the authors could also briefly discuss how to trade off the number of draft tokens per request within a batch, especially when all requests might have high confidence in generating long draft sequences -- how the proposed method can be properly integrated with the online serving scenario.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses the critical and timely issue of LLM inference latency, focusing on the important and growing domain of agentic applications.\n2. The method proposed in the paper is simple and straightforward while being practically useful.\n3. Significant speedup on agentic workloads compared to SOTA model-based and model-free methods.\n4. The ablation experiments answer most of my questions about the method.\n\nWeaknesses\n1. The primary benefit of the proposed method is in repetitive workloads. While highly effective for agentic applications characterized by repetitive token sequences, its advantages may be less pronounced in purely open-ended or less repetitive scenarios when compared to specialized model-based methods designed for those tasks. However, the paper does show it can learn useful patterns even in open-ended workloads and can fall back to the SOTA model-based method.", "was_revised": true, "final_justification": "The paper addresses an important problem and demonstrates superior performance. The author further discussed the possibility of integrating the method with the common serving scenario and provide results for open-ended tasks.", "rebuttal": "Thank you for your positive review and detailed feedback, our response is below.\n\n**Performance on open-ended or less repetitive token sequences.**\n\nAs the reviewer mentioned, Suffix Decoding can still learn useful patterns even in open-ended workloads and can fall back to the SOTA model-based method. The hybrid suffix/speculative decoding (end of Sec 3\\) also helps address this scenario. We provide additional evaluations that show the performance of this hybrid method below:\n\n**SpecBench \\- Speedup (x)**\n\n| System | coding | extraction | humanities | math | math\\_reasoning | qa | rag | reasoning | roleplay | stem | summarization | translation | writing | Overall |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| hybrid (τ=7) | 3.441 | 2.795 | 2.800 | 3.139 | 2.866 | 2.404 | 2.542 | 2.496 | 2.461 | 2.871 | 2.573 | 1.759 | 2.833 | 2.500 |\n| eagle3 | 3.115 | 2.634 | 2.713 | 2.910 | 2.888 | 2.172 | 2.474 | 2.446 | 2.394 | 2.764 | 2.520 | 1.447 | 2.622 | 2.367 |\n| recycling | 2.619 | 2.234 | 2.209 | 2.660 | 2.577 | 1.951 | 2.057 | 2.150 | 1.979 | 2.334 | 2.218 | 1.932 | 2.057 | 2.169 |\n| eagle2 | 2.548 | 1.998 | 1.958 | 2.215 | 2.220 | 1.733 | 1.877 | 1.968 | 1.751 | 2.007 | 1.756 | 1.336 | 1.791 | 1.825 |\n| eagle | 2.427 | 1.956 | 1.848 | 2.142 | 2.112 | 1.600 | 1.841 | 1.871 | 1.703 | 1.922 | 1.702 | 1.305 | 1.753 | 1.752 |\n| suffix | 1.828 | 1.630 | 1.383 | 1.967 | 1.544 | 1.773 | 1.890 | 1.389 | 1.156 | 1.386 | 1.618 | 1.618 | 1.327 | 1.659 |\n| pld | 1.712 | 1.484 | 1.231 | 1.753 | 1.323 | 1.326 | 1.807 | 1.320 | 1.055 | 1.275 | 1.628 | 1.219 | 1.232 | 1.448 |\n| vanilla | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |\n\nAs shown, hybrid Suffix Decoding \\+ EAGLE-3 performs similarly to plain Suffix Decoding on more structured/repetitive tasks (e.g. AgenticSQL), while slightly improving upon plain EAGLE-3 on the open-ended tasks (Spec-Bench). We will include these additional results in the updated version of our paper. Given space limitations, we are including here only the wall-clock speedups compared to the baseline. A comprehensive breakdown of results, including per-token latency (TPOT), mean accepted tokens per step (MAT), mean acceptance rate, and speculation time per generated token, will be incorporated into the revised paper.\n\n**How to set the number of draft tokens per request within a batch.**\n\nWe agree that optimizing the speculation per request in a batch is crucial for many practical online deployments. While SuffixDecoding focuses on *what* tokens to speculate, it is also compatible with existing works that explore *how much* to speculate per request. For example, **TurboSpec** \\[1\\] uses a \"goodput\" metric to balance speculation lengths and batch size, noting that optimal speculation decreases as batch size increases. By dynamically adjusting the number of speculative tokens generated by the SuffixTree per request, SuffixDecoding can be integrated with TurboSpec to maximize the batch-wise goodput metric.\n\nFurthermore, the statistics-based scoring used by Suffix Decoding can potentially better inform methods like TurboSpec. For example, choosing to speculate more from sequences that have a higher marginal score. These are interesting and important directions for future work.\n\n[1] TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput | Liu et al. 2025 | arXiv:2406.14066"}, {"review_id": "A0ykSsimjZ", "reviewer": "Reviewer_Bnmw", "summary": "This paper proposes a novel model-free speculative decoding algorithm for the LLM agent domain. By leveraging a prefix-tree, the proposed algorithm could effectively find previous tokens for the new speculation. Even though this method cannot be genealize to all domains, it serves well for the agent domain, where the generation often follows certain patterns. Overall, I think this paper proposes an efficient solution for this niche challenge.", "questions": "1. How will the proposed algorithm impact the memory consumption\n2. What is the performance of the proposed algorithm on a more general domain benchmark like MT-Bench?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is sound and carefully designed for the target scenario\n2. The paper is clearly written and easy to follow\n\nWeakness:\n1. The technical depth of this work can be further improved. For example, the current adaptive speculation length algorithm is quite straightforward. As the main contribution of the proposed algorithm, it is worth further investigation. For example, whether the past information in each trajectory can help select a better speculation length.", "was_revised": true, "final_justification": "I have read the author's feedback and tend to keep my original rating of accepting this paper.", "rebuttal": "Thank you for your positive review and detailed feedback, our response is below.\n\n**How will the proposed algorithm impact memory consumption?**\n\nSuffix Decoding uses CPU memory on the host during serving, which is usually plentiful and under-utilized. To show Suffix Decoding’s memory consumption and other performance stats, we performed an additional micro-benchmark on its suffix tree, shown below:\n\n| Total \\# entries | Total \\# tokens | Memory footprint (MB) | Avg update time per token (μs) | Avg lookup time per speculated token (μs) |\n| :---- | :---- | :---- | :---- | :---- |\n| 1 K entries | 27 M | 1670 MB | 3.95 | 12.18 |\n| 5 K entries | 143 M | 2980 MB | 4.06 | 10.93 |\n| 10 K entries | 285 M | 4070 MB | 4.05 | 12.00 |\n| 15 K entries | 429 M | 5110 MB | 4.07 | 11.62 |\n| 20 K entries | 572 M | 6150 MB | 4.04 | 11.64 |\n\nAs shown, Suffix Decoding can cache 572M tokens using only 6.15 GB of CPU memory, and its memory usage scales linearly. In comparison, optimized serving systems can generate on the order of \\~5000 tokens/s for Llama-8B on an A100 GPU \\[2\\]. Typical A100 systems (e.g. AWS p4d.24xlarge) have 144GB of CPU memory per A100 GPU. This means that Suffix Decoding can potentially cache 31 days of generated tokens (see math below) before requiring cache eviction.\n\nTime to fill up GPU memory: \n\n1. Memory per token: $$\\frac{6.15 GB}{572M tokens} = 1.075 \\times 10^{-8} GB/token$$\n2. Total token capacity: $$\\frac{144 GB}{(1.075 \\times 10^{-8} GB/token)} = 1.34 \\times 10^{10} tokens$$   \n3. Time to fill RAM: $$\\frac{1.34 \\times 10^{10} tokens}{5000 tok/s} = 2.68 \\times 10^{6} seconds =  \\boxed{31 days}$$ \n\n**General domain (e.g. MT-Bench) performance.**\n\nSuffix Decoding's MT-Bench performance is reflected in SpecBench A.1.2 tables. These tables contain 8 MT-Bench subsets (Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, Humanities). As SpecBench equally samples all 8 categories, overall Suffix Decoding MT-Bench performance can be calculated by averaging these column results.\n\nAdditionally, we provide evaluation results that include hybrid Suffix/Spec Decoding (end of Sec 3\\) using Suffix Decoding with EAGLE-3:\n\n**SpecBench \\- Speedup (x)**\n\n| System | coding | extraction | humanities | math | math\\_reasoning | qa | rag | reasoning | roleplay | stem | summarization | translation | writing | Overall |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| hybrid (τ=7) | 3.441 | 2.795 | 2.800 | 3.139 | 2.866 | 2.404 | 2.542 | 2.496 | 2.461 | 2.871 | 2.573 | 1.759 | 2.833 | 2.500 |\n| eagle3 | 3.115 | 2.634 | 2.713 | 2.910 | 2.888 | 2.172 | 2.474 | 2.446 | 2.394 | 2.764 | 2.520 | 1.447 | 2.622 | 2.367 |\n| recycling | 2.619 | 2.234 | 2.209 | 2.660 | 2.577 | 1.951 | 2.057 | 2.150 | 1.979 | 2.334 | 2.218 | 1.932 | 2.057 | 2.169 |\n| eagle2 | 2.548 | 1.998 | 1.958 | 2.215 | 2.220 | 1.733 | 1.877 | 1.968 | 1.751 | 2.007 | 1.756 | 1.336 | 1.791 | 1.825 |\n| eagle | 2.427 | 1.956 | 1.848 | 2.142 | 2.112 | 1.600 | 1.841 | 1.871 | 1.703 | 1.922 | 1.702 | 1.305 | 1.753 | 1.752 |\n| suffix | 1.828 | 1.630 | 1.383 | 1.967 | 1.544 | 1.773 | 1.890 | 1.389 | 1.156 | 1.386 | 1.618 | 1.618 | 1.327 | 1.659 |\n| pld | 1.712 | 1.484 | 1.231 | 1.753 | 1.323 | 1.326 | 1.807 | 1.320 | 1.055 | 1.275 | 1.628 | 1.219 | 1.232 | 1.448 |\n| vanilla | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |\n\n**AgenticSQL \\- Speedup (x)**\n\n| System | CATEGORIZATION | FEATURE\\_EXTRACTION | QUESTION\\_SUGGESTION | SQL\\_COMBINE | SQL\\_FANOUT1 | SQL\\_FANOUT2 | SQL\\_FANOUT3 | Overall |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| suffix | 3.016 | 9.854 | 10.406 | 4.848 | 3.205 | 2.839 | 3.211 | 5.345 |\n| hybrid (τ=1) | 2.309 | 7.025 | 8.368 | 3.382 | 2.024 | 2.421 | 2.072 | 3.947 |\n| recycling | 2.588 | 3.472 | 2.672 | 2.640 | 2.502 | 2.604 | 2.492 | 2.710 |\n| pld | 1.330 | 3.695 | 1.255 | 3.298 | 1.936 | 1.311 | 1.905 | 2.105 |\n| eagle2 | 1.591 | 2.751 | 1.673 | 1.855 | 1.527 | 2.119 | 1.527 | 1.864 |\n| eagle3 | 1.328 | 1.720 | 2.025 | 1.619 | 1.108 | 2.496 | 1.056 | 1.623 |\n| eagle | 1.324 | 2.246 | 1.598 | 1.669 | 1.229 | 1.955 | 1.138 | 1.595 |\n| vanilla | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |\n\nAs shown, the hybrid Suffix Decoding and EAGLE-3 method performs similarly to plain Suffix Decoding on the more structured/repetitive tasks (e.g. AgenticSQL), while being faster than plain EAGLE-3 on the more general domain SpecBench. We will include these additional results in updating our paper.\n\n\\[1\\] TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput | Liu et al. 2025 | arXiv:2406.14066  \n\\[2\\] (Achieving Faster Open-Source Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) | SGLang Team, Jul 25, 2024 | LMSYS Org)"}]

Author rebuttal:

Meta-review: (a) Summary:
This paper proposes SuffixDecoding, a training-free speculative decoding method based on suffix tree. The method leverages the high repetitiveness and structured loops in agentic LLMs to achieve acceleration. Experiments on SWE-Bench and AgenticSQL show the power of the proposed method.

(b) Strengths:
1. Reviewers recognize the novelty in this paper.

2. The paper is well-written.

3. Experiments on agentic tasks show the advantages of the proposed method.

4. Training-free method is easier to be applied to more models with lower cost.

(c) Weaknesses:
1. The applicability of SuffixDecoding to more open-ended or non-agentic workloads remains unclear. Acceptance rates vary for different dataset. Multiple reviewers have concerns that for non-agentic tasks and non-repetitive workloads, the proposed method might lead to limited speedup.

2. How suffix-tree speculation fundamentally differs from cache-based and retrieval-augmented inference in algorithmic guarantees or failure modes remains unclear.

3. Some references are missing.

4. The reported numbers of baseline methods seem to be worse than the reported numbers in the original papers. It is known that the choice of hyper-parameters matters. It is unclear how this paper tunes the hyper-parameters for baseline methods (e.g., tree width, tree depth, tree size, etc when using tree attention).

(d) Why the decision:
Overall, the paper is of good quality. All reviewers vote for accept. AC stands with the reviewers and recommend accept with spotlight.

(e) Summary of discussions:
In the first round of review, all reviewers are positive about the paper. The rebuttal further strengthens the advantage of this paper. AC would recommend accept too.
Decision: Accept (spotlight)