SPECTRA: Faster Large Language Model Inference with Optimized Internal and External Speculation
Authors: Nguyen-Khang Le, Truong Dinh Do, Le-Minh Nguyen
Accepted at: acl25

SPECTRA: Faster Large Language Model Inference with Optimized Internal and External Speculation

Nguyen-Khang Le1*, Dinh-Truong Do1*, Nguyen Le Minh1

1Japan Advanced Institute of Science and Technology
Correspondence: {lnkhang, truongdo, nguyenml}@jaist.ac.jp

Abstract

Inference with modern Large Language Models (LLMs) is both computationally expensive and time-consuming. Speculative decoding has emerged as a promising solution, but existing approaches face key limitations: training-based methods require a draft model that is challenging to obtain and lacks generalizability, while training-free methods offer limited speedup gains. In this work, we present SPECTRA, a novel framework for accelerating LLM inference without the need for additional training or modification to the original LLM. SPECTRA introduces two new techniques for efficiently utilizing internal and external speculation, each outperforming corresponding state-of-the-art (SOTA) methods independently. When combined, these techniques achieve up to a 4.08x speedup across various benchmarks and LLM architectures, significantly surpassing existing training-free approaches. The implementation of SPECTRA is publicly available.

1 Introduction

Generating long sequences with low latency using Large Language Models (LLMs) is a critical requirement. Current LLMs rely on autoregressive decoding (Touvron et al., 2023; Bai et al., 2023; Jiang et al., 2023; OpenAI et al., 2024), which suffers from inefficiency because it generates text one token at a time. This results in generation time scaling linearly with the sequence length and underutilizes the parallel processing capabilities of modern GPUs. A widely studied approach to mitigate this issue is speculative decoding (Chen et al., 2023; Leviathan et al., 2023), which follows a guess-and-verify paradigm. In this approach, a smaller LLM (draft model) (Chen et al., 2023; Leviathan et al., 2023; Miao et al., 2024; Sun et al., 2023b; Zhou et al., 2024; Cai et al., 2024) or the original LLM trained in a specialized manner (self-speculative decoding) (Elhoushi et al., 2024; Liu et al., 2024a; Yang et al., 2024; Zhang et al., 2024a; Li et al., 2024b) predicts multiple tokens in advance. The original LLM then verifies these predictions in parallel, improving efficiency. However, these approaches require additional training, which demands substantial computational resources and may degrade the original model’s capabilities.

Another line of research focuses on speculating subsequent tokens without requiring additional training. This approach eliminates the need for training new models or modifying the original LLM, making it practical for off-the-shelf deployment. Some methods leverage specialized mechanisms to generate speculative tokens directly from the LLM’s predictions (Fu et al., 2024; Ou et al., 2024), while others rely on external information sources to derive these tokens (Yang et al., 2023; He et al., 2024; Li et al., 2024a). However, the speedup gain in these approaches remains limited due to the quality of the speculative guesses.

We introduce SPECTRA (Figure 1a), a speculative decoding method that improves generation speed without requiring any training or modifications to the original LLM. SPECTRA consists of two main components: a core module (SPECTRA-CORE, Figure 1c), which integrates seamlessly into LLMs in a plug-and-play manner, and an optional retrieval module (SPECTRA-RETRIEVAL, Figure 1e) that further enhances performance. The core module SPECTRA-CORE improves speculative decoding by leveraging the token distribution predicted by the LLM to generate high-quality guesses. Specifically, it employs two multi-level N-gram dictionaries that enable bi-directional search for dynamic-length guesses, balancing both quality and quantity. Additionally, SPECTRA optimizes a candidate pool to continuously update the N-gram dictionaries, ensuring broad token coverage. All updates to these resources, along with guess verifi-
Figure 1: Overview of Spectra and comparison with other non-training SOTA approaches. (a) Overview of SPECTRA. (b) Overview of Lookahead Decoding (Fu et al., 2024). (c) Overview of the SPECTRA-CORE module, which utilizes the knowledge inside LLM for obtaining guesses. (d) Overview of REST (He et al., 2024). (e) Overview of the SPECTRA-RETRIEVAL module, which is designed to be integrated efficiently with SPECTRA-CORE to boost the speedup. The results in the bar chart are measured on HumanEval.

cation, are performed efficiently in a single forward pass. The retrieval module, SPECTRA-RETRIEVAL, can be integrated to further enhance speedup. Existing approaches that rely on external sources for generating guesses (He et al., 2024) struggle to integrate with other speculative decoding methods, as the search time outweighs the speedup gains. SPECTRA-RETRIEVAL addresses this issue by reducing the search space, selecting only high-quality content from the corpus based on perplexity scores computed by the target LLM. This optimization enables seamless integration with SPECTRA-CORE, maximizing efficiency.

Empirical results on six tasks—including multi-turn conversation, code generation, and mathematical reasoning—across three LLM families (Llama 2 (Touvron et al., 2023), Llama 3 (Dubey et al., 2024), and CodeLlama (Rozière et al., 2024)) with model sizes ranging from 7B to 70B demonstrate that SPECTRA outperforms other non-training speculative decoding methods, achieving speedups of up to \( \mathbf{4x} \). We publicly release the code and data. The key contributions of this paper are as follows:

• We introduce SPECTRA, which improves speculative decoding by effectively leveraging the LLM’s predicted token distribution. SPECTRA is a plug-and-play solution that requires no modifications to the LLM (Section 3.1).

• SPECTRA’s retrieval module refines external corpora using perplexity scores computed by the target LLM, providing a general framework that enables speculative decoding approaches relying on external information to be seamlessly integrated with other speculative decoding techniques (Section 3.2).

• Extensive experiments across diverse tasks, LLM architectures, GPU types, and settings demonstrate the efficiency of SPECTRA, outperforming other non-training speculative decoding approaches (Section 5). SPECTRA also integrates with acceleration tools such as FlashAttention and pipeline parallelism (Section 5.2). The code and data are available.

2 Preliminaries

2.1 Autoregressive Decoding in LLMs

Given an input sequence \( x = (x_1, x_2, \ldots, x_s) \) of length \( s \), and a slice of length \( m \) as \( x_{1:m} = (x_1, x_2, \ldots, x_m) \), the output of an LLM represents a probability distribution over the next token. The probability of generating the \( s \)-th token, conditioned on all preceding tokens, is given by \( P_M(x_s \mid x_{1:s-1}) \). The next token \( x_s \) is sampled from this distribution using methods such as greedy, top-\( k \), or top-\( p \) sampling (see (Kool et al., 2020; Holtzman et al., 2020)). For greedy sampling, the next token is selected as \( x_s = \argmax\ P_M(x_s \mid x_{1:s-1}) \). Consequently, the LLM generates an output se-
quence \((y_1, y_2, \ldots, y_m)\) of length \(m\) autoregressively, where each token \(y_i\) is computed as \(y_i = \argmax P_M(y_i \mid y_{1:i-1}, \mathbf{x})\).

2.2 Speculative Decoding

Speculative decoding follows a guess-and-verify approach, where multiple candidate future tokens are speculated and subsequently verified in a single decoding step. With tree attention (Miao et al., 2024), multiple drafts can be verified simultaneously. Let \(G\) denote the number of guesses, and define the set of guesses as \(\tilde{Y} = \{\tilde{y}^{(1)}, \tilde{y}^{(2)}, \ldots, \tilde{y}^{(G)}\}\), where each guess sequence has length \(K\). The \(j\)-th token of the \(i\)-th guess is denoted as \(\tilde{y}_j^{(i)}\).

In the case of speculative decoding with greedy sampling, given the prompt \(\mathbf{x}\), a drafting method generates the draft sequences \(\tilde{Y}\). Using these drafts, the LLM computes the true tokens \((y_1', y_2', \ldots, y_K')\) in parallel. These tokens are then verified, and \(h\) is defined as the highest number of correctly guessed tokens across all guesses. Consequently, \(h + 1\) tokens are generated in a single forward step. Algorithm 2 outlines speculative decoding with greedy sampling, and additional details are provided in Appendix A.

3 SPECTRA DECODING

SPECTRA consists of two modules (SPECTRA-CORE and SPECTRA-RETRIEVAL) that can function independently or together. The core module (SPECTRA-CORE) improves speedup by leveraging the LLM’s predicted token distribution to generate high-quality guesses and integrates into LLMs in a plug-and-play manner. The retrieval module (SPECTRA-RETRIEVAL) derives guesses from a refined external information source and is designed to integrate with SPECTRA-CORE to further enhance performance.

3.1 SPECTRA-CORE

SPECTRA-CORE maintains an N-gram storage, which is used to obtain guesses, and a candidate pool, which is used to augment new N-grams in storage. The candidate pool \(\mathcal{C}\) contains \(W\) sequences, \(\{c^{(0)}, c^{(1)}, \ldots, c^{(W-1)}\}\), with each sequence consisting of \(N\) tokens. Let \(c_j^{(i)}\) represent the \(j\)-th token in the \(i\)-th sequence. The N-gram storage includes two dictionaries: the forward dictionary \(S_{fwd}\) and the backward dictionary \(S_{bwd}\). At each time step, guesses \(\mathcal{G}\) are obtained through a bidirectional search using \(S_{fwd}\) and \(S_{bwd}\). A single inference pass to the LLM retrieves all necessary distributions, which are used to generate new candidate tokens for \(\mathcal{C}\) and verify the guesses \(\mathcal{G}\). The dictionaries \(S_{fwd}\) and \(S_{bwd}\) are updated with N-grams from the candidate pool. The details of the SPECTRA-CORE decoding process are described in Algorithm 1.

Bi-directional Search for Guesses. At each step, SPECTRA generates \(G\) guess sequences \(\mathcal{G} = \{\tilde{y}^{(1)}, \tilde{y}^{(2)}, \ldots, \tilde{y}^{(G)}\}\). Unlike previous work (Fu

Algorithm 1 SPECTRA-CORE Decoding Process

Require: Sequence \(\mathbf{x} = (x_1, x_2, \ldots, x_n)\), model \(P_M\), max N-gram size \(N\), candidate pool size \(W\), max guesses \(G\), max number of new tokens \(m\). Refine threshold \(\tau\)
1: Initialize N-gram Forward-dictionary \(S_{fwd} \leftarrow \emptyset\)
2: Initialize N-gram Backward-dictionary \(S_{bwd} \leftarrow \emptyset\)
3: Random \(c_j^{(i)}, \forall j \in [0, N - 1], \forall i \in [0, W - 1]\)
4: \(t \leftarrow n + 1\)
5: while \(t \leq n + m\) do
6:   {Obtain the guesses}
7:     \(\mathcal{G} \leftarrow S_{fwd}[\mathbf{x}_{t-1}]\)
8:     \(u = \emptyset\)
9:     for \(j = 0\) to \(N - 1\) do
10:        for \(k = N - 1\) to \(1\) do
11:           \(u_j \leftarrow S_{bwd}[\mathbf{x}_{t+j-k:t-1} \oplus u_{0:j-1}]\)
12:           break if found value for \(u_j\)
13:        end for
14:     end for
15:     \(\mathcal{G}\).append(u)
16:     \(\mathcal{G} = \mathcal{G} \oplus \mathcal{G}_{retrieve}\) ▷ Retrieval Integration (Optional)
17:     \(\mathcal{G} \leftarrow G_0:G_{-1}\) ▷ Ensure the max guesses is \(G\)
18:     {Forward in LLM}
19:     Obtain necessary distributions of \(P_M\) in parallel.
20:     {Verification}
21:     {Greedy verify (Alg. 3) or Sampling verify (Alg. 4)}
22:     hits \leftarrow VerificationFunction(\mathbf{x}, P_M, \mathcal{G})
23:     \(\mathbf{x} \leftarrow \mathbf{x} \oplus hits\)
24:     \(t \leftarrow t + \text{size}(hits)\)
25:     {Predict Candidates}
26:     for \(i = 0\) to \(W - 1\) do
27:        \(r \sim \text{Uniform}[0, 1]\)
28:        \(P_c(c_{N-1}^{(i)}) \leftarrow P_M(c_{N-1}^{(i)} \mid c_{N-2}^{(i)}, \mathbf{x})\)
29:        if \(r > \tau\) then
30:           \(c_{N-1}^{(i)} \leftarrow \argmax_{c \notin S_{fwd}} P_c(c_{N-1}^{(i)})\)
31:        else
32:           \(c_{N-1}^{(i)} \leftarrow \argmax_{c} P_c(c_{N-1}^{(i)})\)
33:        end if
34:     end for
35:     {Update N-gram dictionaries}
36:     for \(i = 0\) to \(W - 1\) do
37:        for \(j = 0\) to \(N - 2\) do
38:           \(S_{fwd}[c_{j+1}^{(i)}].append(c_{j+1}^{(i)})\)
39:           \(S_{bwd}[c_{0:j}] \leftarrow c_{j+1}\)
40:        end for
41:     end for
42:     {Update Candidates}
43:     \(c_j^{(i)} \leftarrow c_{j+1}^{(i)}, \forall j \in [0, N - 2], \forall i\)
44: end while
45: Output: \(\mathbf{x}_{n+1:n+m} = (y_1, y_2, \ldots, y_m)\)
Figure 2: Details of how SPECTRA handles internal knowledge. The dashed arrow indicates interactions between the tokens, which are realized by the attention mask in the LLM.

et al., 2024), which enforces uniform guess lengths, SPECTRA supports variable-length guesses, improving both flexibility and efficiency. The forward dictionary \( S_{fwd} \) maps a token to a list of sequences, while the backward dictionary \( S_{bwd} \) maps a sequence to a single token. At time step \( t \), the set of guesses is obtained through a bidirectional search (Alg. 1, lines 7–17). This search operates in two directions: (1) the forward direction, which prioritizes the quantity of guesses, and (2) the backward direction, which prioritizes the quality of guesses. In the forward direction, the last generated token \( x_{t-1} \) is used to search \( S_{fwd} \) for guess sequences (Alg. 1, line 7). In the backward direction, a high-quality guess is constructed by iteratively predicting one token at a time using \( S_{bwd} \), repeating the process until a desired sequence length \( N \) is reached (Alg. 1, lines 8–14).

Verification. The verification step ensures the output distribution is preserved by validating the guesses (Alg. 1, lines 22–24). For greedy sampling, the process is detailed in Appendix H (Alg. 3). In general speculative decoding, verification involves sending draft tokens to the LLM to obtain outputs and progressively checking if the LLM-generated token matches the draft token. Following prior work (Fu et al., 2024), we verify multiple guesses in parallel, accepting the guess with the largest number of correctly predicted tokens. For advanced sampling methods, we adopt sampling verification from (Miao et al., 2024; Fu et al., 2024), whose correctness has been proven. Details on sample verification are provided in Appendix H (Algorithm 4), and its performance and speedups are verified in Appendix F.

Predict & Verify in One Forward Pass. All distributions required for predicting candidates and verifying guesses are obtained in a single forward pass to the LLM, leveraging parallel processing (Figure 2). This is achieved using a specially designed attention mask that specifies the allowed interactions between tokens. For instance, the token \( c_2^{(1)} \) attends only to \( c_1^{(1)}, c_0^{(1)} \), and the input.

Predict Tokens for Candidate Pool. We predict the next candidate tokens \( c_{N-1}^{(i)} \) for the candidate pool using the distribution obtained from the forward pass (Alg. 1, lines 26–34). A straightforward approach is to select tokens with the highest probability in the token distribution. However, we observe that when searching for guesses in the forward dictionary \( S_{fwd} \), it is crucial for the search token to exist in the dictionary; otherwise, no guesses can be retrieved. To address this, we introduce a randomness-based mechanism to increase the coverage of \( S_{fwd} \). Specifically, we probabilistically encourage the selection of unseen tokens in \( S_{fwd} \) using a hyperparameter \( \tau \in [0,1] \). Let \( r \) be a random draw from \([0,1]\). If \( r > \tau \), we select tokens with the highest probability that are not in \( S_{fwd} \); otherwise, we choose tokens with the highest probability regardless of their presence in \( S_{fwd} \). Although \( c_{N-1}^{(i)} \) does not immediately affect the coverage of \( S_{fwd} \), it contributes to coverage expansion in subsequent time steps through our candidate updating mechanism. At the end of each time step, all candidate sequences are shifted left by one token: \( c_j^{(i)} \leftarrow c_{j+1}^{(i)} \), leaving \( c_{N-1}^{(i)} \) empty and ready for prediction in the next time step (Alg. 1, line 43).

Update N-gram Dictionaries. At the end of each time step, candidate tokens from the pool \( C \) are used to update the N-gram dictionaries \( S_{fwd} \) and \( S_{bwd} \). While previous work (Fu et al., 2024) only adds the full N-gram \( (c_0^{(i)}, c_1^{(i)}, \ldots, c_N^{(i)}) \), we ob-
serve that subsequences within N-grams often appear later in the generation process. By including these subsequences in the N-gram storage, we improve both the quality of guesses and the coverage of the dictionaries. Specifically, we add subsequences to \( S_{fwd} \) using the first token as the key, and update \( S_{bwd} \) by mapping the preceding part of the sequence to the last token (Alg. 1, lines 35–41).

3.2 SPECTRA-RETRIEVAL

SPECTRA-RETRIEVAL leverages an external knowledge source to generate guesses. This involves processing a text corpus and indexing it into a structure that supports fast prefix search, such as a trie. At each time step, the last generated tokens are used as input to this structure to retrieve guesses for speculative decoding. However, we observe that using random texts from the corpus without selection can limit the speedup gain. To address this, we propose a method to identify and select high-quality, relevant texts from the corpus tailored to the specific LLM. This improves the speedup gain and enables seamless integration with other speculative decoding approaches, including SPECTRA-CORE.

Corpus Refinement by Perplexity. Given a text sequence \( u = (u_0, u_1, \ldots, u_t) \), perplexity quantifies the average uncertainty of the model when predicting the next token, conditioned on the preceding tokens. The perplexity is calculated as
\[
\mathrm{PPL}(u) = \exp \left\{ -\frac{1}{t} \sum_{i=1}^t \log P_M(u_i \mid u_{<i}) \right\}
\]
A lower perplexity indicates that the model assigns higher probabilities to the sequence, suggesting that the sequence is well-aligned with the model’s predictions and can produce high-quality guesses for speculative decoding. To optimize the retrieval process, we select texts with the lowest perplexity from the corpus to form a smaller, high-quality subset, which is then used to construct the Trie structure for generating guesses.

Integration with SPECTRA-CORE. Our experiments (Section 5.2, Table 2) demonstrate that naively integrating guesses from external sources (e.g., REST (He et al., 2024)) into other speculative methods (e.g., Lookahead (Fu et al., 2024)) can lead to a noticeable drop in speedup. This occurs because the forward pass in the LLM can only handle a limited number of guesses, and exceeding this limit increases memory usage and slows down generation. With a limited guess budget, guesses from external sources can only account for a fraction of the total guesses, causing the search time in the indexing structure (e.g., a trie) to outweigh the speedup gain. To address this, it is crucial to limit the size of the external knowledge while maintaining the quality of the guesses. By refining the corpus using perplexity, SPECTRA-RETRIEVAL seamlessly integrates with SPECTRA-CORE, further boosting the speedup gain. Specifically, we integrate SPECTRA-RETRIEVAL into SPECTRA-CORE by including its guesses (\( G_{retrieve} \)) in the set of SPECTRA-CORE’s guesses during the guess generation step (Alg. 1, line 16).

4 Experiments

Models. We evaluate LLaMA-2-Chat 7B, 13B, 70B (Touvron et al., 2023), CodeLlama 7B, 13B (Rozière et al., 2024), and LLaMA-3-Instruct 8B, 70B (Dubey et al., 2024).

Tasks. We conduct comprehensive evaluations on various generation tasks. MT-Bench (Zheng et al., 2023) for multi-turn conversation; GSM8K (Cobbe et al., 2021) for mathematical reasoning; HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021) and ClassEval (Du et al., 2023) for code generation.

Metrics. SPECTRA does not modify the original LLM and the acceptance conditions, making it a lossless acceleration method. Therefore, the generation quality remains the same as the original LLM. We only evaluate the acceleration performance using the following metrics.

• Speedup Ratio: The speedup ratio relative to autoregressive decoding.

• Compression ratio \( \tau \): The ratio of the total number of autoregressive steps to the number of Spectra decoding steps needed to produce the same sequence length.

Baselines. We use standard autoregressive decoding as the baseline (speed-up ratio = 1.00x). We further compare SPECTRA with leading non-training speculative decoding approaches, namely Adaptive N-gram (Ou et al., 2024), REST (He et al., 2024), and Lookahead (Fu et al., 2024). For details regarding implementation settings of both SPECTRA and these baselines, please refer to Appendix B.

5 Results

5.1 Main Results

Overall Performance. The top portion of Table 1 presents speedup ratios under greedy decod-
ing. SPECTRA consistently achieves the highest acceleration, with speedups up to 4.08× for Llama-3-8B on MBPP. For 7B models, SPECTRA often exceeds 3× acceleration, highlighting the effectiveness of multi-token compression. For 13B models, speedups are slightly lower (1.6×–3×). Overall, the model architecture and dataset characteristics significantly influence the speedup gains of speculative decoding methods. While some approaches excel in specific scenarios—such as tasks with repetitive patterns or predictable token distributions (e.g., repeated variable names or class definitions), they often struggle in diverse or open-ended contexts. In contrast, SPECTRA demonstrates robustness across a wide range of models and datasets, consistently achieving the highest speedup ratios.

Compression Ratio. Table 1 also reports each method’s compression rate, a measure agnostic to specific hardware configurations. Across every dataset and LLM tested, SPECTRA delivers the highest average compression ratio. Each of SPECTRA’s draft-and-verify iterations typically yields 2.1–4.8 tokens, substantially outpacing alternative approaches and nearly doubling the acceptance length achieved by REST.

Acceleration in Sampling Decoding. The lower section of Table 1 reports the performance of SPECTRA under sampling-based decoding with a temperature of 1.0. The results highlight how SPECTRA continues to accelerate generation relative to baselines, offering roughly 1.15–2.77× speedups over standard autoregressive decoding. These gains are more modest than in greedy decoding, reflecting the lower acceptance rate under the sampling-based verification phase, which is consistent with earlier findings (Fu et al., 2024; Leviathan et al., 2023).

5.2 Analysis

Ablation Study. We performed a detailed component-wise analysis to evaluate the contribution of each module to the overall performance (Table 2). On LLaMA2-7B-chat, removing components impacts GSM8K speedups differently. Using only SPECTRA-CORE, excluding multi-level n-grams reduces the speedup from 2.04× to 1.95×, omitting backward dictionary guesses lowers it to 1.94×, and removing forward dictionary guesses drops it further to 1.50×. For SPECTRA-RETRIEVAL, skipping perplexity-based filtering decreases the speedup from 1.18× to 1.16×. The full SPECTRA framework achieves a 2.14× speedup on GSM8K, underscoring the importance of integrating all components to maximize acceptance rates and performance. A similar trend holds for the MTBench dataset. Additionally, we compared SPECTRA with a naive combination of Lookahead and REST, where guess sequences from REST are appended to Lookahead. This approach performs significantly worse than SPECTRA, underscoring that a straightforward merger of two techniques is inadequate without our carefully optimized integration strategy and components.

![Bar chart showing acceptance rates for different guess sources on various datasets](page_1012_1042_388_180.png)

Figure 3: Acceptance rates of Llama2-7B-chat for different guess sources (from SPECTRA-CORE forward dictionary, backward dictionary, SPECTRA-RETRIEVAL). The acceptance rate is the fraction of guessed tokens that pass verification.

Priority for Source of Guesses. Since verifying too many candidate tokens at once can strain GPU resources and reduce speedups (Fu et al., 2024; Li et al., 2024b), SPECTRA limits the total number of guesses processed in each step (Appendix B). In order to assess the individual contributions of our two modules—SPECTRA-CORE and SPECTRA-RETRIEVAL—we temporarily remove the limit on the number of guess sequences in the verification branch and monitor the acceptance rates (Figure 3). We find that guesses generated by the SPECTRA-CORE module (via both forward and backward dictionaries) are accepted at a higher rate than those obtained from the external knowledge source via the SPECTRA-RETRIEVAL module. As a result, SPECTRA gives priority to internal guesses from SPECTRA-CORE over external guesses from SPECTRA-RETRIEVAL, as in Algorithm 1.

FlashAttention. Figure 4 shows that enabling FlashAttention consistently boosts the speedup of all methods, albeit to varying degrees. Notably, we observe an additional 0.24× speedup gain for SPECTRA on both GSM8K and MTBench. This
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="3">Classeval</th>
    <th colspan="3">GSM8K</th>
    <th colspan="3">Humaneval</th>
    <th colspan="3">MBPP</th>
    <th colspan="3">MTBench</th>
    <th rowspan="2">AVG Speedup</th>
  </tr>
  <tr>
    <th>Speedup</th><th>\( \tau \)</th><th>Speedup</th><th>\( \tau \)</th><th>Speedup</th><th>\( \tau \)</th><th>Speedup</th><th>\( \tau \)</th><th>Speedup</th><th>\( \tau \)</th><th>Speedup</th><th>\( \tau \)</th><th>Speedup</th><th>\( \tau \)</th><th>Speedup</th><th>\( \tau \)</th>
  </tr>
  <tr>
    <td colspan="19"><b>Greedy (temperature=0)</b></td>
  </tr>
  <tr>
    <td rowspan="4">CL-13B</td>
    <td>ANPD</td><td>1.94</td><td>2.52</td><td>2.81</td><td>3.72</td><td>2.08</td><td>2.50</td><td>2.71</td><td>3.58</td><td>2.61</td><td>3.41</td><td>2.43</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>2.25</td><td>3.61</td><td>2.80</td><td>4.24</td><td>2.30</td><td>3.16</td><td>2.91</td><td>4.44</td><td>2.59</td><td>4.04</td><td>2.57</td>
  </tr>
  <tr>
    <td>REST</td><td>1.28</td><td>2.14</td><td>0.93</td><td>1.54</td><td>1.58</td><td>2.31</td><td>0.85</td><td>1.40</td><td>0.94</td><td>1.53</td><td>1.12</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>2.38</b></td><td><b>4.06</b></td><td><b>2.91</b></td><td><b>4.65</b></td><td><b>2.63</b></td><td><b>3.95</b></td><td><b>3.29</b></td><td><b>4.46</b></td><td><b>2.65</b></td><td><b>4.40</b></td><td><b>2.77</b></td>
  </tr>
  <tr>
    <td rowspan="4">CL-7B</td>
    <td>ANPD</td><td>2.30</td><td>2.68</td><td>3.21</td><td>3.75</td><td>2.16</td><td>2.47</td><td>3.16</td><td>3.78</td><td>3.35</td><td>3.83</td><td>2.84</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>2.59</td><td>3.66</td><td>2.99</td><td>3.83</td><td>2.50</td><td>3.05</td><td>2.90</td><td>3.67</td><td>3.23</td><td>4.27</td><td>2.84</td>
  </tr>
  <tr>
    <td>REST</td><td>1.45</td><td>2.22</td><td>0.91</td><td>1.39</td><td>1.70</td><td>2.34</td><td>0.96</td><td>1.45</td><td>1.02</td><td>1.44</td><td>1.21</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>2.70</b></td><td><b>4.10</b></td><td><b>3.33</b></td><td><b>4.59</b></td><td><b>2.96</b></td><td><b>3.90</b></td><td><b>3.56</b></td><td><b>4.45</b></td><td><b>3.70</b></td><td><b>4.52</b></td><td><b>3.25</b></td>
  </tr>
  <tr>
    <td rowspan="4">L2-13B</td>
    <td>ANPD</td><td>1.36</td><td>1.78</td><td>1.47</td><td>1.72</td><td>1.34</td><td>1.61</td><td>1.12</td><td>1.32</td><td>1.17</td><td>1.37</td><td>1.29</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>1.81</td><td>2.76</td><td>1.46</td><td>1.87</td><td>1.73</td><td>2.32</td><td>1.38</td><td>1.69</td><td>1.51</td><td>2.04</td><td>1.58</td>
  </tr>
  <tr>
    <td>REST</td><td>1.22</td><td>2.01</td><td>0.94</td><td>1.46</td><td>1.25</td><td>1.94</td><td>0.95</td><td>1.44</td><td>1.14</td><td>1.90</td><td>1.10</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>2.00</b></td><td><b>3.24</b></td><td><b>1.83</b></td><td><b>2.62</b></td><td><b>1.96</b></td><td><b>2.91</b></td><td><b>1.63</b></td><td><b>2.24</b></td><td><b>1.75</b></td><td><b>2.60</b></td><td><b>1.83</b></td>
  </tr>
  <tr>
    <td rowspan="4">L2-70B</td>
    <td>ANPD</td><td>1.82</td><td>1.90</td><td>1.63</td><td>1.61</td><td>1.86</td><td>1.87</td><td>1.17</td><td>1.20</td><td>1.34</td><td>1.30</td><td>1.56</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>2.65</td><td>2.87</td><td>1.86</td><td>2.02</td><td>2.57</td><td>2.67</td><td>1.49</td><td>1.54</td><td>1.94</td><td>2.00</td><td>2.10</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>3.10</b></td><td><b>3.40</b></td><td><b>2.52</b></td><td><b>2.69</b></td><td><b>3.22</b></td><td><b>3.37</b></td><td><b>1.86</b></td><td><b>1.93</b></td><td><b>2.43</b></td><td><b>2.51</b></td><td><b>2.62</b></td>
  </tr>
  <tr>
    <td rowspan="4">L2-7B</td>
    <td>ANPD</td><td>1.62</td><td>1.95</td><td>1.52</td><td>1.68</td><td>1.54</td><td>1.67</td><td>1.19</td><td>1.33</td><td>1.30</td><td>1.37</td><td>1.43</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>2.19</td><td>2.94</td><td>1.66</td><td>1.93</td><td>2.06</td><td>2.42</td><td>1.46</td><td>1.69</td><td>1.73</td><td>2.05</td><td>1.82</td>
  </tr>
  <tr>
    <td>REST</td><td>1.36</td><td>2.12</td><td>1.01</td><td>1.47</td><td>1.41</td><td>2.04</td><td>1.01</td><td>1.46</td><td>1.25</td><td>1.90</td><td>1.21</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>2.40</b></td><td><b>3.43</b></td><td><b>2.14</b></td><td><b>2.64</b></td><td><b>2.40</b></td><td><b>3.05</b></td><td><b>1.77</b></td><td><b>2.16</b></td><td><b>2.02</b></td><td><b>2.59</b></td><td><b>2.14</b></td>
  </tr>
  <tr>
    <td rowspan="4">L3-70B</td>
    <td>ANPD</td><td>1.54</td><td>1.67</td><td>1.50</td><td>1.47</td><td>1.83</td><td>1.88</td><td>1.46</td><td>1.41</td><td>1.23</td><td>1.23</td><td>1.51</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>2.40</td><td>2.62</td><td>1.54</td><td>1.58</td><td>2.56</td><td>2.70</td><td>1.43</td><td>1.45</td><td>1.76</td><td>1.86</td><td>1.94</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>2.67</b></td><td><b>2.91</b></td><td><b>2.10</b></td><td><b>2.14</b></td><td><b>2.84</b></td><td><b>3.02</b></td><td><b>1.94</b></td><td><b>1.94</b></td><td><b>2.06</b></td><td><b>2.13</b></td><td><b>2.32</b></td>
  </tr>
  <tr>
    <td rowspan="4">L3-8B</td>
    <td>ANPD</td><td>2.11</td><td>2.49</td><td>3.86</td><td>4.57</td><td>1.83</td><td>2.09</td><td>3.36</td><td>3.58</td><td>1.14</td><td>1.23</td><td>2.46</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>2.59</td><td>3.44</td><td>3.71</td><td>4.61</td><td>2.49</td><td>2.89</td><td>3.79</td><td>4.65</td><td>1.53</td><td>1.85</td><td>2.82</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>2.83</b></td><td><b>3.49</b></td><td><b>3.89</b></td><td><b>4.77</b></td><td><b>2.57</b></td><td><b>3.02</b></td><td><b>4.08</b></td><td><b>4.76</b></td><td><b>1.69</b></td><td><b>2.10</b></td><td><b>3.01</b></td>
  </tr>
  <tr>
    <td colspan="19"><b>Sampling (temperature=1.0)</b></td>
  </tr>
  <tr>
    <td rowspan="4">CL-13B</td>
    <td>ANPD</td><td>1.15</td><td>1.46</td><td>1.07</td><td>1.31</td><td>1.05</td><td>1.30</td><td>1.00</td><td>1.24</td><td>2.31</td><td>2.89</td><td>1.31</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>1.38</td><td>2.00</td><td>1.08</td><td>1.43</td><td>1.29</td><td>1.75</td><td>1.02</td><td>1.34</td><td>2.33</td><td>3.48</td><td>1.42</td>
  </tr>
  <tr>
    <td>REST</td><td>1.14</td><td>1.87</td><td>0.82</td><td>1.35</td><td>1.27</td><td>1.96</td><td>0.84</td><td>1.39</td><td>0.93</td><td>1.50</td><td>1.00</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>1.68</b></td><td><b>2.22</b></td><td><b>1.20</b></td><td><b>1.75</b></td><td><b>1.65</b></td><td><b>2.12</b></td><td><b>1.15</b></td><td><b>1.70</b></td><td><b>2.37</b></td><td><b>3.80</b></td><td><b>1.61</b></td>
  </tr>
  <tr>
    <td rowspan="4">CL-7B</td>
    <td>ANPD</td><td>1.29</td><td>1.50</td><td>1.16</td><td>1.30</td><td>1.10</td><td>1.32</td><td>1.12</td><td>1.27</td><td>2.77</td><td>3.05</td><td>1.49</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>1.54</td><td>2.03</td><td>1.19</td><td>1.41</td><td>1.43</td><td>1.81</td><td>1.19</td><td>1.43</td><td>2.72</td><td>3.50</td><td>1.61</td>
  </tr>
  <tr>
    <td>REST</td><td>1.23</td><td>1.86</td><td>0.88</td><td>1.33</td><td>1.33</td><td>1.98</td><td>0.91</td><td>1.40</td><td>0.97</td><td>1.44</td><td>1.06</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>1.81</b></td><td><b>2.25</b></td><td><b>1.35</b></td><td><b>1.73</b></td><td><b>1.68</b></td><td><b>2.12</b></td><td><b>1.33</b></td><td><b>1.72</b></td><td><b>2.78</b></td><td><b>3.94</b></td><td><b>1.79</b></td>
  </tr>
  <tr>
    <td rowspan="4">L2-13B</td>
    <td>ANPD</td><td>1.20</td><td>1.52</td><td>1.24</td><td>1.46</td><td>1.17</td><td>1.40</td><td>1.03</td><td>1.22</td><td>1.17</td><td>1.35</td><td>1.16</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>1.52</td><td>2.22</td><td>1.32</td><td>1.69</td><td>1.48</td><td>2.00</td><td>1.18</td><td>1.48</td><td>1.49</td><td>2.01</td><td>1.40</td>
  </tr>
  <tr>
    <td>REST</td><td>1.18</td><td>1.96</td><td>0.93</td><td>1.45</td><td>1.19</td><td>1.88</td><td>0.92</td><td>1.44</td><td>1.12</td><td>1.88</td><td>1.07</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>1.70</b></td><td><b>2.75</b></td><td><b>1.55</b></td><td><b>2.23</b></td><td><b>1.69</b></td><td><b>2.59</b></td><td><b>1.34</b></td><td><b>1.89</b></td><td><b>1.74</b></td><td><b>2.57</b></td><td><b>1.60</b></td>
  </tr>
  <tr>
    <td rowspan="4">L2-7B</td>
    <td>ANPD</td><td>1.31</td><td>1.51</td><td>1.34</td><td>1.48</td><td>1.28</td><td>1.46</td><td>1.10</td><td>1.22</td><td>1.25</td><td>1.36</td><td>1.26</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>1.78</td><td>2.30</td><td>1.51</td><td>1.76</td><td>1.72</td><td>2.09</td><td>1.25</td><td>1.49</td><td>1.68</td><td>2.02</td><td>1.59</td>
  </tr>
  <tr>
    <td>REST</td><td>1.26</td><td>2.03</td><td>0.99</td><td>1.46</td><td>1.27</td><td>1.93</td><td>0.96</td><td>1.41</td><td>1.21</td><td>1.88</td><td>1.14</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>1.97</b></td><td><b>2.83</b></td><td><b>1.78</b></td><td><b>2.28</b></td><td><b>2.04</b></td><td><b>2.75</b></td><td><b>1.47</b></td><td><b>1.84</b></td><td><b>1.97</b></td><td><b>2.54</b></td><td><b>1.85</b></td>
  </tr>
  <tr>
    <td rowspan="4">L3-8B</td>
    <td>ANPD</td><td>1.25</td><td>1.37</td><td>1.97</td><td>2.18</td><td>1.43</td><td>1.65</td><td>1.89</td><td>2.07</td><td>1.15</td><td>1.21</td><td>1.54</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>1.48</td><td>1.78</td><td>2.07</td><td>2.41</td><td>1.79</td><td>2.21</td><td>1.99</td><td>2.40</td><td>1.57</td><td>1.81</td><td>1.78</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>1.94</b></td><td><b>2.84</b></td><td><b>2.27</b></td><td><b>2.78</b></td><td><b>1.92</b></td><td><b>2.51</b></td><td><b>2.19</b></td><td><b>2.78</b></td><td><b>1.70</b></td><td><b>2.05</b></td><td><b>2.01</b></td>
  </tr>
</table>

Table 1: Overall performance of speculative decoding methods across multiple tasks. “CL-xB” denotes CodeLlama with xB parameters, “L2-xB” denotes LLaMA-2-Chat of size xB, and “L3-xB” denotes LLaMA-3-Instruct of size xB. We report the speedup ratio (vs. autoregressive) and the compression ratio \( \tau \).

is because FlashAttention better exploits the parallel structure of speculative decoding by reducing attention overheads, especially when verifying multiple guessed tokens in parallel. Although smaller gains are also seen for other methods, SPECTRA benefits the most, as it presents the longest verification branches and thus stands to profit significantly from more efficient attention implementations.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">MTBench</th>
  </tr>
  <tr>
    <th>Speedup</th>
    <th>\( \tau \)</th>
    <th>Speedup</th>
    <th>\( \tau \)</th>
  </tr>
  <tr>
    <td>REST</td>
    <td>1.01</td>
    <td>1.47</td>
    <td>1.25</td>
    <td>1.90</td>
  </tr>
  <tr>
    <td>Lookahead</td>
    <td>1.66</td>
    <td>1.93</td>
    <td>1.73</td>
    <td>2.05</td>
  </tr>
  <tr>
    <td>Lookahead + REST</td>
    <td>1.08</td>
    <td>1.47</td>
    <td>1.27</td>
    <td>1.90</td>
  </tr>
  <tr>
    <th colspan="5">SPECTRA’s ablation</th>
  </tr>
  <tr>
    <td>CORE Module</td>
    <td>2.04</td>
    <td>2.50</td>
    <td>1.92</td>
    <td>2.35</td>
  </tr>
  <tr>
    <td>- w/o Forward Dict</td>
    <td>1.50</td>
    <td>1.68</td>
    <td>1.20</td>
    <td>1.37</td>
  </tr>
  <tr>
    <td>- w/o Backward Dict</td>
    <td>1.94</td>
    <td>2.21</td>
    <td>1.74</td>
    <td>2.12</td>
  </tr>
  <tr>
    <td>- w/o Sub-Ngram</td>
    <td>1.95</td>
    <td>2.34</td>
    <td>1.75</td>
    <td>2.18</td>
  </tr>
  <tr>
    <td>RETRIEVAL Module</td>
    <td>1.18</td>
    <td>1.31</td>
    <td>1.24</td>
    <td>1.50</td>
  </tr>
  <tr>
    <td>- w/o PPL refine</td>
    <td>1.16</td>
    <td>1.29</td>
    <td>1.20</td>
    <td>1.45</td>
  </tr>
  <tr>
    <th>SPECTRA (ours)</th>
    <th><b>2.14</b></th>
    <th><b>2.64</b></th>
    <th><b>2.02</b></th>
    <th><b>2.59</b></th>
  </tr>
</table>

Table 2: Ablation study of SPECTRA’s components (greedy decoding, LLaMA2-7B-Chat).

![Bar chart showing speedups for different methods on GSM8K and MTBench](page_1012_1042_388_246.png)

Figure 4: Effect of FlashAttention on speculative decoding speed: Measured speedups on GSM8K and MTBench (LLama2-7B-Chat, greedy decoding). “No Flash” uses standard attention; “With Flash” uses FlashAttention for faster parallel verification.

Other Analysis. Detailed throughputs from Table 1 are provided in Appendix D. Evaluations of SPECTRA on different GPU types and quantization settings are described in Appendix C, while its performance in distributed settings with multiple GPUs is discussed in Appendix E.

6 Related Works

Large language models (LLMs) are increasingly deployed in a range of applications, motivating ongoing research into more efficient inference (Liu et al., 2025). Common strategies include quantizing model weights into lower-precision formats (Liu et al., 2024b; Lin et al., 2024; Zhao et al., 2024; Park et al., 2024), pruning redundant parameters (Ma et al., 2023; Xia et al., 2023; Sun et al., 2023a; Le et al., 2025), and employing knowledge distillation (Gu et al., 2024; Friha et al., 2024; Zhang et al., 2024b). These techniques help reduce the computational load per forward pass, thereby lowering generation latency. However, they often introduce some degradation in model performance, forcing practitioners to balance quality with efficiency.

A growing line of work explores speculative decoding as a strategy for accelerating generation while maintaining the output distribution (Chen et al., 2023; Leviathan et al., 2023). Some speculative decoding approaches train a smaller LLM (draft model) (Chen et al., 2023; Leviathan et al., 2023; Miao et al., 2024; Sun et al., 2023b; Zhou et al., 2024; Cai et al., 2024), or train the original LLM itself in a special manner (self-speculative) (Elhoushi et al., 2024; Liu et al., 2024a; Yang et al., 2024; Zhang et al., 2024a; Li et al., 2024b) to guess several subsequent tokens and then verify them parallelly using the original LLM. As these approaches require training, they pose limitations, such as requiring heavy computational resources and losing the original model capabilities.

To avoid additional training, alternative speculative decoding methods leverage external resources or structural properties of language generation. Retrieval-based methods sidestep draft model training by using a datastore indexed with observed prefixes to retrieve guess sequences (Yang et al., 2023; He et al., 2024; Li et al., 2024a). Other approaches, such as Jacobi-like parallel decoding (Santilli et al., 2023) and lookahead decoding (Fu et al., 2024), mitigate left-to-right dependencies by generating and validating multiple candidate tokens in parallel. These training-free techniques achieve comparable speedups to learned methods without requiring model optimization, making them ideal for scenarios with computational constraints.

7 Conclusions

In this work, we have introduced SPECTRA, a new, training-free framework for accelerating large language model inference by harnessing both internal and external speculation. By integrating our plug-and-play SPECTRA-CORE module—which leverages multi-level N-gram storage and bidirectional search—with the refined SPECTRA-RETRIEVAL module that selects high-quality external cues via perplexity-based filtering, our approach achieves substantial speedups (up to 4.08x) across diverse tasks and model architectures while preserving the original model’s output quality. By offering a loss-
less speedup. SPECTRA provides a practical, high-impact solution for accelerating inference in LLMs.

8 Limitations

(1) Cost of Building External Datastores. While SPECTRA-CORE—our internal-knowledge module—relies solely on sequences observed during generation and thus requires no extra external data, SPECTRA-RETRIEVAL depends on constructing and indexing a sizeable external datastore from potentially large corpora. This process can be time-consuming and memory-intensive, particularly in domains where data updates frequently or storage is constrained. Although this additional investment can yield substantial speedups by boosting token acceptance rates, it may not be universally feasible or cost-effective.

(2) Limited Evaluation Scope. Our experiments center primarily on English-language benchmarks in conversational and coding tasks using LLaMA-based models. Although SPECTRA can, in principle, be applied to other models or languages, additional factors such as domain-specific tokenization or specialized textual structures may affect the acceptance rate and overall speedup. Future work is needed to assess the generality of SPECTRA across diverse linguistic settings (e.g., low-resource languages or specialized technical documents) and for a wider range of model families (beyond LLaMA-based architectures) to confirm and refine its applicability.

Acknowledgments

This work was supported by JST SPRING, Japan Grant Number JPMJSP2102.

References

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xinxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen Technical Report.

Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024. MEDUSA: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. Place: Vienna, Austria.

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating Large Language Model Decoding with Speculative Sampling.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukas Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3029–3051, Singapore. Association for Computational Linguistics.

Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. ClassEval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. 2024. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12622–12642, Bangkok, Thailand. Association for Computational Linguistics.
Othmane Friha, Mohamed Amine Ferrag, Burak Kantarci, Burak Cakmak, Arda Ozgun, and Nassira Ghoualmi-Zine. 2024. Llm-based edge intelligence: A comprehensive survey on architectures, applications, security and trustworthiness. IEEE Open Journal of the Communications Society.

Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of LLM inference using LOOKAHEAD DECODING. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. Place: Vienna, Austria.

Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minilmv: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations.

Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. REST: Retrieval-Based Speculative Decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1582–1595, Mexico City, Mexico. Association for Computational Linguistics.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B.

Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dmitry Bahdanau, Leandro Von Werra, and Harm de Vries. 2023. The Stack: 3 TB of permissively licensed source code. Transactions on Machine Learning Research.

Wouter Kool, Herke van Hoof, and Max Welling. 2020. Ancestral Gumbel-Top-k Sampling for Sampling Without Replacement. Journal of Machine Learning Research, 21(47):1–36.

Khang Nguyen Le, Ryo Sato, Dai Nakashima, Takeshi Suzuki, and Minh Le Nguyen. 2025. Optipruner: Effective pruning approach for every target sparsity. In Proceedings of the 31st International Conference on Computational Linguistics, pages 3600–3612.

Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. Place: Honolulu, Hawaii, USA.

Minghan Li, Xilun Chen, Ari Holtzman, Beidi Chen, Jimmy Lin, Wen-tau Yih, and Xi Victoria Lin. 2024a. Nearest Neighbor Speculative Decoding for LLM Generation and Attribution. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024b. EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7421–7432, Miami, Florida, USA. Association for Computational Linguistics.

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Weiming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87–100.

Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Duyu Tang, Kai Han, and Yunhe Wang. 2024a. Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Yiheng Liu, Hao He, Tianle Han, Xu Zhang, Mengyuan Liu, Jiaming Tian, Yutong Zhang, Jiaqi Wang, Xiaohui Gao, Tianyang Zhong, Yi Pan, Shaochen Xu, Zihao Wu, Zhengliang Liu, Xin Zhang, Shu Zhang, Xintao Hu, Tuo Zhang, Ning Qiang, Tianming Liu, and Bao Ge. 2025. Understanding llms: A comprehensive overview from training to inference. Neurocomputing, 620:129190.

Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2024b. LLM-QAT: Data-free quantization aware training for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 467–484, Bangkok, Thailand. Association for Computational Linguistics.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702–21720.

Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunlan Shi, Zhuoming Chen, Daianya Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS ’24, pages 932–949, New York, NY, USA. Association for Computing Machinery. Event-place: La Jolla, CA, USA.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. Association for Computational Linguistics.

Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Alteman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgrum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Justin Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhoun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hee-woo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondratiuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoon Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emry Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Shershakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroksi Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillett, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayaragha, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, C. J. Weinmann, Akila Welihinda, Peter Wellner, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barrett Zoph. 2024. GPT-4 Technical Report.

Jie Ou, Yueming Chen, and Prof. Tian. 2024. Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 10–22, Mexico City, Mexico. Association for Computational Linguistics.

Yeonhong Park, Jake Hyun, SangLyul Cho, Bonggeun Sim, and Jae W. Lee. 2024. Any-precision lilm: Low-cost deployment of multiple, different-sized llms. In Proceedings of the 41st International Conference on Machine Learning.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafori, Wenhan Xiong, Alexandre Défosséz, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2024. Code Llama: Open Foundation Models for Code. _eprint: 2308.12950.

Andrea Santilli, Silvio Severino, Emilian Postolache,
Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. 2023. Accelerating transformer inference for translation via parallel decoding. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12336–12355, Toronto, Canada. Association for Computational Linguistics.

Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. 2023a. A simple and effective pruning approach for large language models. ArXiv, abs/2306.11695.

Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. 2023b. SpecTr: fast speculative decoding via optimal transport. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. Event-place: New Orleans, LA, USA.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Alber, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenying Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molynog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rash Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models.

Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2023. Sheared llama: Accelerating language model pre-training via structured pruning. ArXiv, abs/2310.06694.

Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furui Wei. 2023. Inference with Reference: Lossless Acceleration of Large Language Models. _eprint_: 2304.04487.

Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. 2024. Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding. Transactions on Machine Learning Research.

Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2024a. Draft&Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11263–11282, Bangkok, Thailand. Association for Computational Linguistics.

Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. 2024b. Dual-space knowledge distillation for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18164–18181, Miami, Florida, USA. Association for Computational Linguistics.

Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems, 6:196–209.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. 2024. DistillSpec: Improving Speculative Decoding via Knowledge Distillation. In The Twelfth International Conference on Learning Representations.

A More on Speculative Decoding

Autoregressive decoding (Touvron et al., 2023; Bai et al., 2023; Jiang et al., 2023; OpenAI et al., 2024), suffers from inefficiency because it generates text one token at a time (Figure 5, Left). Speculative decoding (Chen et al., 2023; Leviathan et al., 2023) follows a guess-and-verify paradigm (Figure 5, Right). In speculative decoding, a smaller LLM (draft model) (Chen et al., 2023; Leviathan et al., 2023; Miao et al., 2024; Sun et al., 2023b; Zhou et al., 2024; Cai et al., 2024) or the original LLM trained in a specialized manner (self-speculative decoding) (Elhoushi et al., 2024; Liu et al., 2024a; Yang et al., 2024; Zhang et al., 2024a; Li et al., 2024b) predicts multiple tokens in advance. The original LLM then verifies these predictions in parallel, improving efficiency.

LLMs process discrete integer sequences as inputs, where each integer represents a token. We define the input sequence as \( x = (x_1, x_2, \ldots, x_s) \in \mathbb{N}^s \) of length \( s \), and denote a slice of length \( m \) at step \( t \) as \( x_{1:m} = (x_1, x_2, \ldots, x_m) \). The output of an LLM represents the probability distribution over the next token. The probability of generating the
Figure 5: Examples of Autoregressive decoding (Left) and Speculative Decoding (Right). While autoregressive decoding generates one token per forward step, speculative decoding generates three tokens with one forward step.

s-th token, conditioned on all preceding tokens, is given by \( P_M(x_s \mid x_{1:s-1}) \). The next token \( x_s \) is then sampled from this distribution using various methods (e.g., greedy, top-\( k \), and top-\( p \) sampling; see (Kool et al., 2020; Holtzman et al., 2020)). In the case of greedy sampling, the next token is selected as \( x_s = \arg\max P_M(x_s \mid x_{1:s-1}) \)

Let \( \mathbf{x} \) be the prompt tokens provided by the user. The LLM generates an output sequence of length \( m \), with each generated token \( y_i \) computed autoregressively. Assuming greedy sampling, the decoding process follows:

\[
\begin{cases}
y_1 = \arg\max P_M(y_1 \mid \mathbf{x}) \\
y_2 = \arg\max P_M(y_2 \mid y_1, \mathbf{x}) \\
\vdots \\
y_m = \arg\max P_M(y_m \mid y_{1:m-1}, \mathbf{x}).
\end{cases}
\] (1)

A.1 Speculative Decoding

Speculative decoding follows a Guess-And-Verify approach, where multiple candidate future tokens are speculated and subsequently verified in a single decoding step. With tree attention (Miao et al., 2024), multiple drafts can be verified simultaneously. Let \( G \) denote the number of guesses, and define the set of guesses as \( \tilde{Y} = \{\tilde{y}^{(1)}, \tilde{y}^{(2)}, \ldots, \tilde{y}^{(G)}\} \), where each guess sequence has length \( K \). The \( j \)-th token of the \( i \)-th guess is denoted as \( \tilde{y}_j^{(i)} \).

In the case of speculative decoding with greedy sampling, given the prompt \( \mathbf{x} \), a drafting method is used to generate the draft sequences \( Y \). Using these drafts, the LLM then computes the true tokens \( (y'_1, y'_2, \ldots, y'_K) \) in parallel. For instance, for the guess sequence \( \tilde{y}^{(1)} \), the true tokens are determined as:

\[
\begin{cases}
y'_1 = \arg\max P_M(y_1 \mid \mathbf{x}) \\
y'_2 = \arg\max P_M(y_2 \mid \tilde{y}_1^{(1)}, \mathbf{x}) \\
\vdots \\
y'_K = \arg\max P_M(y_K \mid \tilde{y}_{1:K-1}^{(1)}, \mathbf{x}).
\end{cases}
\] (2)

These generated tokens are then verified. Let \( h \) be the highest number of correct guessed tokens across all guesses. Consequently, \( h + 1 \) tokens are generated in one forward step. Algorithm 2 outlines speculative decoding with greedy sampling.

B Implementation Details

B.1 Frameworks and Libraries

We implement SPECTRA in Python using PyTorch 2.1.0 and the Hugging Face transformers library (version 4.36.2).

B.2 Models and Checkpoints

We run our experiments primarily with:

• LLaMA-2-Chat (Touvron et al., 2023) in sizes 7B, 13B, 70B.

• CodeLlama (Rozière et al., 2024) in sizes 7B and 13B.

• LLaMA-3-Instruct (Dubey et al., 2024) in sizes 8B and 70B.

All checkpoints are sourced from official repositories or Hugging Face without fine-tuning or modification. For the 7B and 13B models, we use 16-bit (FP16) precision with a pre-allocated key-value cache. For large-scale models such as LLaMA-2-70B and LLaMA-3-70B, we quantize them to 8-bit for the primary results presented in Table 1. Additionally, we evaluate the 70B models in FP16 precision, as reported in Appendix E. We also verify numerical consistency by comparing the 32-bit and 16-bit outputs of LLaMA-2-7B, detailed in Appendix F.

B.3 Hardware

Most experiments are conducted on a single NVIDIA A100 GPU with 80GB of memory. To analyze hardware-specific scaling (Appendix C), we also test on other NVIDIA GPUs, including the RTX 3090, RTX 8000, A40, and A6000. For the largest models (70B) that exceed single-GPU memory constraints under FP16 settings, we distribute
computation across multiple GPUs (2x, 4x, or 8x H100) using Hugging Face’s pipeline parallelism (Appendix E).

B.4 Hyperparameters

Lookahead, REST, and ANPD. We replicate each baseline using their publicly available GitHub code, keeping to the default settings and hyperparameters outlined in the original papers.

Spectra. By default, we use a 5-gram setup for forward/backward dictionaries. A candidate pool of size \( W = 15 \) is maintained per key to generate new n-gram records. After each forward pass, candidate sequences are shifted by one token and then re-populated. We introduce a threshold \( \tau \in [0, 1] \), set to 0.1 by default, to determine when to force the selection of a token not yet present in the forward dictionary. At each speculative decoding step, up to \( G = 15 \) guesses are allowed. Internal guesses receive priority, and if the guess limit is not reached, external guesses are added.

For external lookups, we implement a Trie structure for rapid prefix queries, following a design similar to REST (He et al., 2024). For conversation tasks (e.g., MT-Bench), we gather approximately 100k examples from the UltraChat dataset (Ding et al., 2023), focusing on those with minimal perplexity under the same LLM we aim to accelerate. For code tasks (e.g., HumanEval, MBPP), we draw from TheStack (Kocetkov et al., 2023) and again refine it to the 100k snippets with the lowest perplexity for memory efficiency. We measure perplexity by running a single forward pass (in streaming mode) over candidate samples and ranking them.

All speedup and throughput metrics are computed at a batch size of 1. In code generation tasks, the maximum generation length is typically 512 tokens, whereas for conversation tasks (MT-Bench, GSM8K), we allow up to 1024 tokens or stop early if the model outputs an end-of-sequence token. All random seeds are set to 0.

C Evaluating SPECTRA in Different GPU Types

Table 3 reports speedups on GSM8K and MT-Bench across four GPUs with varying memory throughput and compute capabilities. While absolute wall-clock times differ across GPUs, the relative accelerations remain consistent. SPECTRA consistently outperforms other baselines, including Lookahead, achieving higher speedups in all cases. On older GPUs (e.g., RTX 3090 or RTX 8000), the gap between Lookahead and SPECTRA narrows slightly due to less efficient parallelism, but SPECTRA maintains its lead. These results demonstrate that SPECTRA is robust to hardware variations and effective across both data-center and consumer-grade GPUs.

<table>
  <tr>
    <th rowspan="2">GPU</th>
    <th rowspan="2">Method</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">MTBench</th>
  </tr>
  <tr>
    <th>Speedup</th>
    <th>\( \tau \)</th>
    <th>Speedup</th>
    <th>\( \tau \)</th>
  </tr>
  <tr>
    <td>A40</td>
    <td>Lookahead</td>
    <td>1.49</td>
    <td>1.93</td>
    <td>1.53</td>
    <td>2.07</td>
  </tr>
  <tr>
    <td></td>
    <td>SPECTRA</td>
    <td><b>1.92</b></td>
    <td><b>2.46</b></td>
    <td><b>1.84</b></td>
    <td><b>2.36</b></td>
  </tr>
  <tr>
    <td>A6000</td>
    <td>Lookahead</td>
    <td>1.48</td>
    <td>1.92</td>
    <td>1.52</td>
    <td>2.06</td>
  </tr>
  <tr>
    <td></td>
    <td>SPECTRA</td>
    <td><b>1.92</b></td>
    <td><b>2.46</b></td>
    <td><b>1.84</b></td>
    <td><b>2.36</b></td>
  </tr>
  <tr>
    <td>RTX8000</td>
    <td>Lookahead</td>
    <td>1.33</td>
    <td>1.93</td>
    <td>1.34</td>
    <td>2.08</td>
  </tr>
  <tr>
    <td></td>
    <td>SPECTRA</td>
    <td><b>1.70</b></td>
    <td><b>2.46</b></td>
    <td><b>1.58</b></td>
    <td><b>2.35</b></td>
  </tr>
  <tr>
    <td>RTX3090</td>
    <td>Lookahead</td>
    <td>1.32</td>
    <td>1.92</td>
    <td>1.30</td>
    <td>2.06</td>
  </tr>
  <tr>
    <td></td>
    <td>SPECTRA</td>
    <td><b>1.84</b></td>
    <td><b>2.46</b></td>
    <td><b>1.74</b></td>
    <td><b>2.36</b></td>
  </tr>
</table>

Table 3: Hardware scalability of SPECTRA decoding on GSM8K and MTBench for various GPU architectures.

D Details Results with Throughputs

We provide a detailed throughput analysis to complement the speedup ratios reported in the main text. Our goal is to demonstrate how SPECTRA scales across various model sizes, datasets, and GPU architectures. We measure throughput using two key metrics:

• Macro Throughput (Mac-TP). Calculated as the average of per-generation token-processing rates—i.e., for each generation step \( i \), we compute \( token_i / time_i \) and then average over all steps.

• Micro Throughput (Mic-TP). Calculated as the total number of generated tokens divided by the total elapsed time

Table 5 focuses on GSM8K and MTBench performance across four different GPU models, while Table 4 provides more granular results on additional datasets and model configurations. In all cases, SPECTRA consistently achieves higher throughput than both non-speculative baselines and other training-free accelerators, as evidenced by improvements in both Mic-TP and Mac-TP. Notably, this performance advantage remains stable even on older GPUs (e.g., the RTX 3090 and RTX 8000), demonstrating SPECTRA’s robustness to varying hardware capabilities.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Method</th>
    <th colspan="4">Classeval</th>
    <th colspan="4">GSM8K</th>
    <th colspan="4">HumanEval</th>
    <th colspan="4">MBPP</th>
    <th colspan="4">MTBench</th>
  </tr>
  <tr>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
  </tr>
  <tr>
    <td colspan="19">Greedy (temperature=0)</td>
  </tr>
  <tr>
    <td rowspan="5">CL-13B</td>
    <td>Autoregressive</td><td>30.85</td><td>30.85</td><td>32.03</td><td>32.03</td><td>32.35</td><td>32.35</td><td>32.07</td><td>32.07</td><td>30.69</td><td>30.69</td>
  </tr>
  <tr>
    <td>ANPD</td><td>59.77</td><td>58.03</td><td>89.99</td><td>89.18</td><td>67.43</td><td>64.65</td><td>86.76</td><td>86.41</td><td>80.10</td><td>76.68</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>69.28</td><td>68.62</td><td>89.73</td><td>89.00</td><td>74.33</td><td>73.23</td><td>93.38</td><td>92.80</td><td>79.38</td><td>78.67</td>
  </tr>
  <tr>
    <td>REST</td><td>39.53</td><td>37.73</td><td>29.93</td><td>29.47</td><td>51.15</td><td>47.49</td><td>27.41</td><td>27.39</td><td>28.92</td><td>27.18</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>73.47</b></td><td><b>72.98</b></td><td><b>93.36</b></td><td><b>93.23</b></td><td><b>84.91</b></td><td><b>84.41</b></td><td><b>105.44</b></td><td><b>105.39</b></td><td><b>81.32</b></td><td><b>80.68</b></td>
  </tr>
  <tr>
    <td rowspan="5">CL-7B</td>
    <td>Autoregressive</td><td>41.17</td><td>41.17</td><td>41.17</td><td>41.17</td><td>41.41</td><td>41.41</td><td>41.60</td><td>41.60</td><td>38.91</td><td>38.93</td>
  </tr>
  <tr>
    <td>ANPD</td><td>94.76</td><td>93.02</td><td>132.26</td><td>131.30</td><td>89.26</td><td>87.13</td><td>131.35</td><td>130.99</td><td>130.41</td><td>126.64</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>106.51</td><td>105.95</td><td>123.04</td><td>121.90</td><td>103.45</td><td>103.51</td><td>120.75</td><td>120.23</td><td>125.58</td><td>124.77</td>
  </tr>
  <tr>
    <td>REST</td><td>59.49</td><td>56.61</td><td>37.61</td><td>37.21</td><td>70.38</td><td>65.22</td><td>40.11</td><td>40.09</td><td>39.64</td><td>36.70</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>111.09</b></td><td><b>110.68</b></td><td><b>137.24</b></td><td><b>136.86</b></td><td><b>122.54</b></td><td><b>122.41</b></td><td><b>148.32</b></td><td><b>148.07</b></td><td><b>143.98</b></td><td><b>144.32</b></td>
  </tr>
  <tr>
    <td rowspan="5">L2-13B</td>
    <td>Autoregressive</td><td>31.85</td><td>31.56</td><td>32.40</td><td>32.43</td><td>32.27</td><td>32.27</td><td>32.19</td><td>32.19</td><td>31.93</td><td>31.78</td>
  </tr>
  <tr>
    <td>ANPD</td><td>43.30</td><td>44.44</td><td>47.54</td><td>45.22</td><td>43.24</td><td>42.28</td><td>36.20</td><td>35.84</td><td>37.44</td><td>34.84</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>57.49</td><td>58.94</td><td>47.44</td><td>47.62</td><td>55.76</td><td>55.58</td><td>44.41</td><td>44.15</td><td>48.11</td><td>46.62</td>
  </tr>
  <tr>
    <td>REST</td><td>38.81</td><td>37.74</td><td>30.36</td><td>30.22</td><td>40.47</td><td>39.70</td><td>30.70</td><td>30.67</td><td>36.39</td><td>37.02</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>63.64</b></td><td><b>64.31</b></td><td><b>59.21</b></td><td><b>58.63</b></td><td><b>63.39</b></td><td><b>63.18</b></td><td><b>52.43</b></td><td><b>52.19</b></td><td><b>56.04</b></td><td><b>53.75</b></td>
  </tr>
  <tr>
    <td rowspan="5">L2-70B</td>
    <td>Autoregressive</td><td>2.60</td><td>2.60</td><td>2.61</td><td>2.61</td><td>2.61</td><td>2.61</td><td>2.63</td><td>2.63</td><td>2.60</td><td>2.60</td>
  </tr>
  <tr>
    <td>ANPD</td><td>4.72</td><td>4.80</td><td>4.25</td><td>4.10</td><td>4.85</td><td>4.76</td><td>3.07</td><td>3.07</td><td>3.47</td><td>3.30</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>6.90</td><td>7.16</td><td>4.87</td><td>5.12</td><td>6.71</td><td>6.73</td><td>3.92</td><td>3.93</td><td>5.05</td><td>5.02</td>
  </tr>
  <tr>
    <td>REST</td><td>8.07</td><td>8.35</td><td>6.58</td><td>6.75</td><td>8.41</td><td>8.41</td><td>4.88</td><td>4.88</td><td>6.32</td><td>6.22</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>96.88</b></td><td><b>98.75</b></td><td><b>86.51</b></td><td><b>85.50</b></td><td><b>98.77</b></td><td><b>98.38</b></td><td><b>72.39</b></td><td><b>73.22</b></td><td><b>81.93</b></td><td><b>79.20</b></td>
  </tr>
  <tr>
    <td rowspan="5">L3-70B</td>
    <td>Autoregressive</td><td>2.58</td><td>2.57</td><td>2.58</td><td>2.58</td><td>2.59</td><td>2.59</td><td>2.59</td><td>2.59</td><td>2.55</td><td>2.55</td>
  </tr>
  <tr>
    <td>ANPD</td><td>3.97</td><td>4.19</td><td>3.86</td><td>3.72</td><td>4.72</td><td>4.75</td><td>3.77</td><td>3.59</td><td>3.14</td><td>3.03</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>6.17</td><td>6.47</td><td>3.99</td><td>3.96</td><td>6.63</td><td>6.75</td><td>3.70</td><td>3.66</td><td>4.49</td><td>4.53</td>
  </tr>
  <tr>
    <td>REST</td><td>6.87</td><td>7.18</td><td>5.43</td><td>5.34</td><td>7.33</td><td>7.50</td><td>5.01</td><td>4.88</td><td>5.25</td><td>5.16</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>103.61</b></td><td><b>105.88</b></td><td><b>142.89</b></td><td><b>142.72</b></td><td><b>92.86</b></td><td><b>93.16</b></td><td><b>143.80</b></td><td><b>142.72</b></td><td><b>61.69</b></td><td><b>60.22</b></td>
  </tr>
  <tr>
    <td rowspan="5">L3-8B</td>
    <td>Autoregressive</td><td>36.59</td><td>36.58</td><td>36.74</td><td>36.74</td><td>36.20</td><td>36.21</td><td>35.24</td><td>35.20</td><td>36.55</td><td>36.69</td>
  </tr>
  <tr>
    <td>ANPD</td><td>77.21</td><td>78.76</td><td>141.89</td><td>141.36</td><td>66.31</td><td>65.57</td><td>118.47</td><td>112.95</td><td>41.77</td><td>40.20</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>94.92</td><td>97.09</td><td>136.32</td><td>135.92</td><td>89.99</td><td>90.47</td><td>135.67</td><td>133.12</td><td>56.09</td><td>55.49</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>103.61</b></td><td><b>105.88</b></td><td><b>142.89</b></td><td><b>142.72</b></td><td><b>92.86</b></td><td><b>93.16</b></td><td><b>143.80</b></td><td><b>142.72</b></td><td><b>61.69</b></td><td><b>60.22</b></td>
  </tr>
  <tr>
    <td colspan="19">Sampling (temperature=1.0)</td>
  </tr>
  <tr>
    <td rowspan="5">CL-13B</td>
    <td>Autoregressive</td><td>30.90</td><td>30.64</td><td>31.38</td><td>31.37</td><td>31.24</td><td>31.39</td><td>31.46</td><td>30.71</td><td>30.67</td>
  </tr>
  <tr>
    <td>ANPD</td><td>35.48</td><td>34.86</td><td>33.54</td><td>32.34</td><td>32.64</td><td>34.36</td><td>31.57</td><td>30.95</td><td>70.92</td><td>65.68</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>42.54</td><td>40.74</td><td>33.79</td><td>32.49</td><td>40.25</td><td>42.17</td><td>32.02</td><td>31.19</td><td>71.50</td><td>68.46</td>
  </tr>
  <tr>
    <td>REST</td><td>35.15</td><td>33.22</td><td>25.67</td><td>25.24</td><td>39.58</td><td>38.49</td><td>26.43</td><td>25.89</td><td>28.41</td><td>26.69</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>51.86</b></td><td><b>50.04</b></td><td><b>35.77</b></td><td><b>35.67</b></td><td><b>51.60</b></td><td><b>52.64</b></td><td><b>36.29</b></td><td><b>35.27</b></td><td><b>72.90</b></td><td><b>69.98</b></td>
  </tr>
  <tr>
    <td rowspan="5">CL-7B</td>
    <td>Autoregressive</td><td>39.60</td><td>39.58</td><td>40.85</td><td>40.87</td><td>40.05</td><td>40.10</td><td>40.81</td><td>40.49</td><td>40.50</td>
  </tr>
  <tr>
    <td>ANPD</td><td>50.89</td><td>51.76</td><td>47.44</td><td>46.68</td><td>44.14</td><td>46.34</td><td>45.86</td><td>45.81</td><td>112.29</td><td>103.57</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>60.87</td><td>60.29</td><td>48.54</td><td>47.64</td><td>57.12</td><td>61.14</td><td>48.64</td><td>48.27</td><td>110.07</td><td>105.00</td>
  </tr>
  <tr>
    <td>REST</td><td>48.64</td><td>46.41</td><td>35.98</td><td>35.46</td><td>53.35</td><td>52.26</td><td>37.04</td><td>36.57</td><td>39.36</td><td>36.51</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>71.70</b></td><td><b>71.78</b></td><td><b>55.24</b></td><td><b>52.81</b></td><td><b>67.27</b></td><td><b>69.20</b></td><td><b>54.48</b></td><td><b>52.91</b></td><td><b>112.43</b></td><td><b>108.49</b></td>
  </tr>
  <tr>
    <td rowspan="5">L2-13B</td>
    <td>Autoregressive</td><td>31.23</td><td>31.17</td><td>31.44</td><td>31.47</td><td>31.41</td><td>31.42</td><td>32.02</td><td>32.06</td><td>31.67</td><td>31.59</td>
  </tr>
  <tr>
    <td>ANPD</td><td>37.53</td><td>37.94</td><td>39.11</td><td>37.99</td><td>36.79</td><td>36.75</td><td>32.97</td><td>32.71</td><td>36.91</td><td>34.34</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>47.59</td><td>47.35</td><td>41.60</td><td>41.76</td><td>46.33</td><td>46.51</td><td>37.82</td><td>37.82</td><td>47.35</td><td>45.48</td>
  </tr>
  <tr>
    <td>REST</td><td>36.78</td><td>36.17</td><td>29.33</td><td>29.25</td><td>37.46</td><td>36.71</td><td>29.38</td><td>29.28</td><td>35.50</td><td>36.21</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>53.13</b></td><td><b>52.28</b></td><td><b>48.60</b></td><td><b>48.11</b></td><td><b>52.93</b></td><td><b>53.11</b></td><td><b>42.95</b></td><td><b>43.03</b></td><td><b>54.98</b></td><td><b>52.42</b></td>
  </tr>
  <tr>
    <td rowspan="5">L2-7B</td>
    <td>Autoregressive</td><td>39.89</td><td>39.88</td><td>40.58</td><td>40.59</td><td>40.09</td><td>40.10</td><td>40.59</td><td>40.66</td><td>40.65</td><td>40.70</td>
  </tr>
  <tr>
    <td>ANPD</td><td>52.14</td><td>52.78</td><td>54.23</td><td>52.90</td><td>51.40</td><td>50.97</td><td>44.73</td><td>43.77</td><td>50.92</td><td>48.24</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>70.82</td><td>71.17</td><td>61.15</td><td>61.34</td><td>68.78</td><td>69.01</td><td>50.84</td><td>51.83</td><td>68.27</td><td>66.77</td>
  </tr>
  <tr>
    <td>REST</td><td>50.35</td><td>49.99</td><td>40.19</td><td>40.09</td><td>50.86</td><td>50.06</td><td>38.94</td><td>38.18</td><td>49.12</td><td>50.54</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>78.46</b></td><td><b>78.74</b></td><td><b>72.13</b></td><td><b>71.68</b></td><td><b>81.71</b></td><td><b>81.76</b></td><td><b>59.77</b></td><td><b>60.09</b></td><td><b>80.21</b></td><td><b>77.00</b></td>
  </tr>
  <tr>
    <td rowspan="5">L3-8B</td>
    <td>Autoregressive</td><td>35.75</td><td>35.76</td><td>35.16</td><td>35.17</td><td>36.01</td><td>36.02</td><td>36.05</td><td>36.07</td><td>35.39</td><td>35.48</td>
  </tr>
  <tr>
    <td>ANPD</td><td>44.71</td><td>43.72</td><td>69.12</td><td>66.73</td><td>51.48</td><td>51.57</td><td>68.03</td><td>64.54</td><td>40.84</td><td>39.23</td>
  </tr>
  <tr>
    <td>Lookahead</td><td>53.05</td><td>50.57</td><td>72.68</td><td>69.11</td><td>64.59</td><td>63.79</td><td>71.88</td><td>68.90</td><td>55.46</td><td>53.74</td>
  </tr>
  <tr>
    <td>SPECTRA (Ours)</td><td><b>69.50</b></td><td><b>68.92</b></td><td><b>79.88</b></td><td><b>76.53</b></td><td><b>69.09</b></td><td><b>68.62</b></td><td><b>78.99</b></td><td><b>76.69</b></td><td><b>60.33</b></td><td><b>57.69</b></td>
  </tr>
</table>

Table 4: Micro throughput (Mic-TP) and Macro throughput (Mac-TP) across multiple tasks and models.
<table>
  <tr>
    <th rowspan="2">GPU</th>
    <th rowspan="2">Method</th>
    <th colspan="2">GSM8K</th>
    <th colspan="2">MTBench</th>
  </tr>
  <tr>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
    <th>Mac-TP</th>
    <th>Mic-TP</th>
  </tr>
  <tr>
    <td rowspan="3">A40</td>
    <td>Autoregressive</td>
    <td>32.66</td>
    <td>32.66</td>
    <td>32.14</td>
    <td>31.66</td>
  </tr>
  <tr>
    <td>Lookahead</td>
    <td>48.59</td>
    <td>48.73</td>
    <td>49.13</td>
    <td>47.96</td>
  </tr>
  <tr>
    <td>SPECTRA</td>
    <td><b>62.56</b></td>
    <td><b>61.52</b></td>
    <td><b>59.00</b></td>
    <td><b>56.80</b></td>
  </tr>
  <tr>
    <td rowspan="3">A6000</td>
    <td>Autoregressive</td>
    <td>39.15</td>
    <td>39.17</td>
    <td>38.78</td>
    <td>38.24</td>
  </tr>
  <tr>
    <td>Lookahead</td>
    <td>58.13</td>
    <td>58.30</td>
    <td>58.84</td>
    <td>57.40</td>
  </tr>
  <tr>
    <td>SPECTRA</td>
    <td><b>75.20</b></td>
    <td><b>74.16</b></td>
    <td><b>71.3</b></td>
    <td><b>69.28</b></td>
  </tr>
  <tr>
    <td rowspan="3">RTX8000</td>
    <td>Autoregressive</td>
    <td>34.03</td>
    <td>34.27</td>
    <td>34.21</td>
    <td>34.02</td>
  </tr>
  <tr>
    <td>Lookahead</td>
    <td>45.25</td>
    <td>45.42</td>
    <td>45.73</td>
    <td>44.16</td>
  </tr>
  <tr>
    <td>SPECTRA</td>
    <td><b>57.95</b></td>
    <td><b>57.09</b></td>
    <td><b>54.16</b></td>
    <td><b>52.32</b></td>
  </tr>
  <tr>
    <td rowspan="3">RTX3090</td>
    <td>Autoregressive</td>
    <td>40.67</td>
    <td>40.76</td>
    <td>41.17</td>
    <td>41.22</td>
  </tr>
  <tr>
    <td>Lookahead</td>
    <td>53.69</td>
    <td>53.75</td>
    <td>53.51</td>
    <td>52.09</td>
  </tr>
  <tr>
    <td>SPECTRA</td>
    <td><b>74.87</b></td>
    <td><b>73.88</b></td>
    <td><b>71.58</b></td>
    <td><b>69.79</b></td>
  </tr>
</table>

Table 5: Throughput results for different GPU types on GSM8K and MTBench.

E Evaluating SPECTRA in Multi-GPU Environments

A critical consideration for practical deployment is how SPECTRA scales when models are distributed across multiple GPUs—a common requirement for large LLMs exceeding single-device memory capacity. To evaluate this, we measure SPECTRA’s performance under three distributed configurations of LLaMA-2-70B: (1) 2xH100 with full precision, (2) 4xH100 with full precision, and (3) 8xH100 with full precision. We also include a baseline of 1xH100 with 8-bit quantization for memory-constrained single-GPU inference. Table 6 reports throughput and speedup metrics.

SPECTRA achieves consistent speedups of 2.00—2.03× across all multi-GPU configurations while maintaining a stable compression ratio (\( \tau \)) of 2.52. This demonstrates robust scalability—partitioning model weights introduces minimal overhead, and the speculative verification process remains efficient despite inter-GPU communication. Notably, even in the quantized single-GPU setting, SPECTRA provides a 2.43× speedup, outperforming standard autoregressive decoding. These results validate SPECTRA’s practicality for large-scale deployments where memory constraints necessitate distributed inference.

F Verifying Generation Quality with SPECTRA Decoding

Greedy Decoding Performance. To assess the quality of greedy decoding, we compare the inference results of the LLaMA-2-7B Chat model using SPECTRA Decoding against Hugging Face’s standard greedy search. Our baseline consists of single-precision (FP32) inference on 160 conversational turns from the MT-Bench dataset. Under FP32, SPECTRA Decoding produces identical outputs to the baseline.

However, when transitioning to half-precision (FP16), even Hugging Face’s native greedy search generates 25 discrepancies (out of 160) compared to the FP32 baseline. SPECTRA Decoding exhibits a similar discrepancy rate (26), confirming that it maintains the output distribution within the numerical error margins typically observed in standard half-precision inference libraries.

Sampling Decoding Performance. We also assess generation quality under a stochastic sampling setting (temperature = 1.0). As detailed in Table 7, SPECTRA Decoding produces ROUGE-1, ROUGE-2, and ROUGE-L scores on both the CNN/DailyMail (Nallapati et al., 2016) and XSum (Narayan et al., 2018) summarization datasets that are nearly identical to those of standard autoregressive sampling. At the same time, SPECTRA achieves notable speedups (1.60× on CNN/DailyMail and 1.69× on XSum) with compression ratios of 2.05 and 2.08, respectively. These results confirm that SPECTRA Decoding accelerates inference while preserving generation quality across diverse tasks.

These findings reaffirm that SPECTRA Decoding, does not degrade generation quality compared to conventional greedy or sampling-based methods.
<table>
  <tr>
    <th>GPU & Model Setting</th>
    <th>Method</th>
    <th colspan="2">MTBench</th>
    <th>Speedup</th>
    <th>\( \tau \)</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>Mac-TP</td>
    <td>Mic-TP</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>1xH100 - Quantized Int8</td>
    <td>Autoregressive<br>SPECTRA</td>
    <td>2.60<br><b>6.32</b></td>
    <td>2.60<br><b>6.22</b></td>
    <td>1.00<br><b>2.43</b></td>
    <td>1.00<br><b>2.51</b></td>
  </tr>
  <tr>
    <td>2xH100 - FP16</td>
    <td>Autoregressive<br>SPECTRA</td>
    <td>14.81<br><b>29.62</b></td>
    <td>14.70<br><b>28.91</b></td>
    <td>1.00<br><b>2.00</b></td>
    <td>1.00<br><b>2.52</b></td>
  </tr>
  <tr>
    <td>4xH100 - FP16</td>
    <td>Autoregressive<br>SPECTRA</td>
    <td>14.60<br><b>29.67</b></td>
    <td>14.48<br><b>28.89</b></td>
    <td>1.00<br><b>2.03</b></td>
    <td>1.00<br><b>2.52</b></td>
  </tr>
  <tr>
    <td>8xH100 - FP16</td>
    <td>Autoregressive<br>SPECTRA</td>
    <td>14.39<br><b>29.27</b></td>
    <td>14.28<br><b>28.55</b></td>
    <td>1.00<br><b>2.03</b></td>
    <td>1.00<br><b>2.52</b></td>
  </tr>
</table>

Table 6: Results in multi-GPU Environments on GSM8K and MTBench using LLama-2-chat-70B.

<table>
  <tr>
    <th>Dataset</th>
    <th>Method</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>Speedup</th>
    <th>\( \tau \)</th>
  </tr>
  <tr>
    <td rowspan="2">CNN</td>
    <td>Autoregressive</td>
    <td>9.77</td>
    <td>0.39</td>
    <td>7.20</td>
    <td>1.00</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>SPECTRA</td>
    <td>9.74</td>
    <td>0.41</td>
    <td>7.18</td>
    <td>1.60</td>
    <td>2.05</td>
  </tr>
  <tr>
    <td rowspan="2">XSUM</td>
    <td>Autoregressive</td>
    <td>18.12</td>
    <td>4.36</td>
    <td>12.43</td>
    <td>1.00</td>
    <td>1.00</td>
  </tr>
  <tr>
    <td>SPECTRA</td>
    <td>18.13</td>
    <td>4.40</td>
    <td>12.49</td>
    <td>1.69</td>
    <td>2.08</td>
  </tr>
</table>

Table 7: Evaluation of SPECTRA Decoding on CNN/DailyMail and XSum using a temperature of 1.0. ROUGE scores, speedups over autoregressive decoding, and compression ratio (\( \tau \)) are reported for LLaMA-2-7B-Chat.

G Token Acceptance Rate Analysis

Figure 6 plots the cumulative number of accepted tokens versus decoding steps for each dataset (MT-Bench, HumanEval, MBPP, and GSM8K) using LLama2-7B-chat with greedy decoding. The steeper ascent of the SPECTRA curve indicates that our method requires substantially fewer decoding steps compared to alternatives, for example, almost two times shorter than ANPD. This improvement is attributed to a higher token acceptance rate, which in turn reduces the overall number of decoding iterations and enhances the efficiency of the generation process.

H Algorithms

![Total number of accepted tokens across all samples at each decoding step.](page_1012_1202_388_388.png)

Figure 6: Total number of accepted tokens across all samples at each decoding step.
Algorithm 2 Speculative Decoding (Multiple guesses and Greedy Sampling)

Given guess size \( K \), number of guesses \( G \), and target length \( T \).
Given initial prompt sequence \( \mathbf{x} \).
while \( n < T \) do
    Obtain multiple drafts \( \tilde{Y} = \{\tilde{y}^{(1)}, \tilde{y}^{(2)}, \ldots, \tilde{y}^{(G)}\} \).
    In parallel, compute \( K + 1 \) verification tokens \( y' \):
        for \( i = 1 : K \) do
            \( y_i^{(g)} = \arg\max P_M(y_i \mid \tilde{y}_{i-1}^{(g)}, \mathbf{x}), \quad \forall g \in \{1, \ldots, G\} \)
        end for
    Identify the sequence \( \tilde{y}^{(g^*)} \) with the highest token matches and the corresponding \( y'^{(g^*)} \).
        for \( t = 1 : K \) do
            if \( y_t'^{(g^*)} = \tilde{y}_t^{(g^*)} \) then
                Set \( y_{n+t} \leftarrow \tilde{y}_t^{(g^*)} \) and \( n \leftarrow n + 1 \).
            else
                \( y_{n+t} \leftarrow y_t'^{(g^*)} \) and exit for loop.
            end if
        end for
end while
Algorithm 3 Greedy Verification with SPECTRA Decoding

Require: sequence x, model \( P_M \), guesses \( \mathcal{G} = \{g^i\} \) with \( i \in [0, G-1] \)
Ensure: o {accepted tokens of length 1 to N}

1: function GREEDYVERIFICATION(x, \( P_M \), \( \mathcal{G} \))
2: \( D \leftarrow \emptyset \) \( \triangleright \) Store the distributions
3: \( V \leftarrow \mathcal{G} \) \( \triangleright \) Store the current guesses
4: for \( i = 0 \) to \( G - 1 \) do
5:     \( D.\text{append}(P_M(g^{(i)}, x_{\text{next}}|g^{(i)}, x)) \) \( \triangleright \) Last token of x and \( g^{(i)} \) outputs – total N distributions
6: end for
7: for \( i = 1 \) to \( N - 1 \) do
8:     \( j \leftarrow 1 \)
9:     is_accept \( \leftarrow 0 \)
10:    \( \mathcal{P} \leftarrow D[1]_i \)
11:   while \( j \leq \text{size}(V) \) do
12:      \( s_j \leftarrow V[j]_i \)
13:     if \( s_j = \arg\max \mathcal{P} \) then \( \triangleright \) accepted, update all potential speculations and probabilities
14:        o.append(\( s_j \))
15:        is_accept \( \leftarrow 1 \)
16:        \( V_{\text{new}}, D_{\text{new}} \leftarrow \emptyset, \emptyset \)
17:        for \( k = j \) to \( \text{size}(V) \) do
18:            if \( s_j = V[k]_i \) then
19:                \( V_{\text{new}}.\text{append}(V[k]) \)
20:                \( D_{\text{new}}.\text{append}(D[k]) \)
21:            end if
22:        end for
23:        \( V, D \leftarrow V_{\text{new}}, D_{\text{new}} \)
24:        break
25:    else \( \triangleright \) rejected, go to next speculation
26:        \( j \leftarrow j + 1 \)
27:    end if
28: end while
29: if is_accept then
30:     continue
31: else \( \triangleright \) guarantee one step movement
32:     o.append(\( \arg\max \mathcal{P} \))
33:     break
34: end if
35: end for
36: if is_accept then
37:     o.append(\( \arg\max D[1]_N \))
38: end if
39: return o
40: end function
Algorithm 4 Sample Verification with SPECTRA DECODING

Require: sequence x, model \( P_M \), guesses \( g^i \) with \( i \in [0, G-1] \)
Ensure: o {accepted tokens of length 1 to N}

1: function SAMPLEVERIFICATION(x, \( P_M \), g)
2:     \( D \leftarrow \emptyset \)                                  ▷ Store the distributions
3:     \( V \leftarrow \mathcal{G} \)                             ▷ Store the current guesses
4:     for \( i = 0 \) to \( G-1 \) do
5:         \( D.\text{append}(P_M(g^{(i)}, x_{\text{next}}|g^{(i)}, \mathbf{x})) \)   ▷ Last token of \( \mathbf{x} \) and \( g^{(i)} \) outputs – total \( N \) distributions
6:     end for
7:     for \( i = 1 \) to \( N-1 \) do
8:         \( j \leftarrow 1 \)
9:         is_accept \( \leftarrow 0 \)
10:        \( \mathcal{P}_j \leftarrow D[j]_i \)
11:        while \( j \leq \text{size}(V) \) do
12:            \( s_j \leftarrow V[j]_i \)
13:            sample \( r \sim U(0, 1) \)
14:            if \( r \leq \mathcal{P}_j(s_j) \) then                ▷ accepted, update all potential speculations and probabilities
15:                o.append(\( s_j \))
16:                is_accept \( \leftarrow 1 \)
17:                \( V_{\text{new}}, D_{\text{new}} \leftarrow \emptyset, \emptyset \)
18:                for \( k = j \) to \( \text{size}(V) \) do
19:                    if \( s_j = V[k]_i \) then
20:                        \( V_{\text{new}}.\text{append}(V[k]) \)
21:                        \( D_{\text{new}}.\text{append}(D[k]) \)
22:                    end if
23:                end for
24:                \( V, D \leftarrow V_{\text{new}}, D_{\text{new}} \)
25:                break
26:            else                                              ▷ rejected, go to next speculation
27:                \( \mathcal{P}_j(s_j) \leftarrow 0 \)
28:                \( \mathcal{P}_{j+1} = \text{norm}(\mathcal{P}_j) \)
29:                \( j \leftarrow j + 1 \)
30:            end if
31:        end while
32:        if is_accept then
33:            continue
34:        else                                              ▷ guarantee one step movement
35:            sample \( x_{\text{next}} \sim \mathcal{P}_j \)
36:            o.append(\( x_{\text{next}} \))
37:            break
38:        end if
39:    end for
40:    if is_accept then
41:        o.append(sample \( x_{\text{next}} \sim D[1]_N \))
42:    end if
43:    return o
44: end function

Reviewer paper summaries: This paper introduces SPECTRA, a novel training-free framework to accelerate LLM inference. It applies forward and backward dictionaries from the lookahead window for better prediction and uses perplexity for external datastore selection. This method surpasses state-of-the-art training-free speculative decoding methods such as lookahead decoding and REST.
Summary of strengths: 1.	This framework performs well by utilizing the benefits of both internal and external training-free speculative decoding. The authors modify these methods and demonstrate remarkable speedups.
2.	The evaluation is well-designed and detailed, demonstrating the effectiveness of each component. The authors evaluated performance with state-of-the-art models across different GPU platforms, hyperparameter settings, multi-GPU configurations, and quantization techniques. SPECTRA consistently performs well in all scenarios.
3.	The paper is well written with a clear presentation of concepts and results.
Summary of weaknesses: 1.	The improvement upon lookahead decoding in the internal token prediction part appears somewhat simple. The addition of a backward dictionary, which uses multiple prefix tokens to predict the next tokens, represents an incremental rather than transformative advancement. However, this is not a significant issue that impacts the paper's core contributions.
Comments, suggestions and typos: 1.	Can you provide more details about the flash-attention integration? Does it simply adopt the flash-attention integration implementation from lookahead decoding?
2.	Algorithm 1, line 17 shows that the guess list is truncated from front to back. Does this mean guesses from the forward dictionary have the highest priority, followed by the backward dictionary, while the retrieval integration has the lowest priority? How does this prioritization influence the acceptance rates shown in Figure 3?
3.	I am curious about how the random sample and threshold mechanism in "Predict Tokens for Candidate Pool" (line 239) functions. Is there any ablation study showing the effect of this mechanism?
Overall assessment scores: 4
Metareview score: 4.5
Metareview: The paper presents a new state-of-the-art training-free method for increasing LLM inference speed, achieving up to 4x faster inference across various benchmarks and different architectures.
Summary of reasons to publish: All 3 reviewers agree that the paper is sound, interesting, and clear, while reporting improvements beyond the SoA in terms of inference speed across mutliple tasks.