SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration

## Abstract

Abstract Speculative decoding (SD) has emerged as a widely used paradigm to accelerate the inference of large language models (LLMs) without compromising generation quality. It works by first employing a compact model to draft multiple tokens efficiently and then using the target LLM to verify them in parallel. While this technique has achieved notable speedups, most existing approaches necessitate either additional parameters or extensive training to construct effective draft models, thereby restricting their applicability across different LLMs and tasks. To address this limitation, we explore a novel plug-and-play SD solution with layer-skipping, which skips intermediate layers of the target LLM as the compact draft model. Our analysis reveals that LLMs exhibit great potential for self-acceleration through layer sparsity and the task-specific nature of this sparsity. Building on these insights, we introduce \method , an on-the-fly self-speculative decoding algorithm that adaptively selects intermediate layers of LLMs to skip during inference. \method does not require auxiliary models or additional training, making it a plug-and-play solution for accelerating LLM inference across diverse input data streams. Our extensive experiments across a wide range of models and downstream tasks demonstrate that \method can achieve over a 1.3 × 1.3\times 1.3 × ∼ similar-to \sim ∼ 1.6 × 1.6\times 1.6 × speedup while preserving the original distribution of the generated text. We release our code in https://github.com/hemingkx/SWIFT .

## 1 Introduction

Large Language Models (LLMs) have exhibited outstanding capabilities in handling various downstream tasks (OpenAI, ; Touvron et al., ; ; Dubey et al., ). However, their token-by-token generation necessitated by autoregressive decoding poses considerable efficiency challenges, particularly as model sizes increase. To address this, speculative decoding (SD) has been proposed as a promising solution for lossless LLM inference acceleration (Xia et al., ; Leviathan et al., ; Chen et al., ). At each decoding step, SD first employs a compact draft model (e.g., a smaller LM) to efficiently predict multiple tokens as speculations for future decoding steps of the target LLM. These tokens are then validated by the target LLM in parallel, ensuring that the original output distribution remains unchanged. The effectiveness of SD hinges on the trade-off between drafting latency and speculation accuracy (Xia et al., ). By minimizing latency and maximizing accuracy, SD enables the efficient generation of multiple tokens per decoding step, thereby substantially enhancing the inference efficiency of LLMs without compromising output quality.

Recent advancements in SD have pushed the boundaries of the latency-accuracy trade-off by exploring various strategies, including incorporating lightweight draft modules into LLMs (Cai et al., ; Ankner et al., ; Li et al., ; ), employing fine-tuning strategies to facilitate efficient LLM drafting (Kou et al., ; Yi et al., ; Elhoushi et al., ), and aligning draft models with the target LLM (Liu et al., ; Zhou et al., ; Miao et al., ). Despite their promising efficacy, these approaches require additional modules or extensive training, which limits their broad applicability across different model types and causes significant inconvenience in practice. To tackle this issue, another line of research has proposed the Jacobi-based drafting (Santilli et al., ; Fu et al., ) to facilitate plug-and-play SD. As illustrated in Figure 1(a), these methods append pseudo tokens to the input prompt, enabling the target LLM to generate multiple tokens as drafts in a single decoding step. However, the Jacobi-decoding paradigm misaligns with the autoregressive pretraining objective of LLMs, resulting in suboptimal acceleration effects.

Figure: Figure 1: Illustration of prior solution and ours for plug-and-play SD. (a) Jacobi-based drafting appends multiple pseudo tokens to the input prompt, enabling the target LLM to generate multiple tokens as drafts in a single step. (b) \methodadopts sparsity-based drafting, which exploits the inherent sparsity in LLMs to facilitate efficient drafting. This work is the first exploration of plug-and-play SD using sparsity-based drafting.
Refer to caption: x1.png

In this work, we introduce a novel research direction for plug-and-play SD: sparsity-based drafting, which leverages the inherent, readily available sparsity in LLMs to enable efficient drafting (see Figure 1(b)). Specifically, we exploit a straightforward yet practical form of LLM sparsity – layer sparsity – to accelerate LLM inference. Our approach is based on two key observations: 1) LLMs possess great potential for self-acceleration through layer sparsity. Contrary to the conventional belief that layer selection must be carefully optimized using training data (Zhang et al., ), we surprisingly found that uniformly skipping layers to draft can still achieve a notable $1.2\times$ speedup, providing a strong foundation for plug-and-play SD. 2) Layer sparsity is task-specific. We observed that each task requires its own optimal set of skipped layers, and applying the same skipped layer configuration across different tasks would cause substantial performance degradation. For example, the speedup drops from $1.47\times$ to $1.01\times$ when transferring the configuration optimized for a storytelling task to a mathematical reasoning task, underscoring the limitations of static layer-skipping patterns during inference.

Building on these observations, we introduce \method, the first on-the-fly self-speculative decoding algorithm that adaptively optimizes the set of skipped layers in the target LLM during inference, facilitating the lossless acceleration of LLMs across diverse input data streams. \methodintegrates two key innovations: (1) a context-based layer set optimization mechanism that leverages LLM-generated context to efficiently identify the optimal set of skipped layers corresponding to the current input stream, and (2) a confidence-aware inference acceleration strategy that maximizes the use of draft tokens, improving both speculation accuracy and verification efficiency. These innovations allow \methodto strike an expected balance between the latency-accuracy trade-off in SD, providing a new plug-and-play solution for lossless LLM inference acceleration without the need for auxiliary models or additional training, as demonstrated in Table 1.

We conduct experiments using LLaMA-2 and CodeLLaMA models across multiple tasks, including summarization, code generation, mathematical reasoning, etc. \methodachieves a $1.3\times$$\sim$$1.6\times$ wall-clock time speedup compared to conventional autoregressive decoding. Notably, in the greedy setting, \methodconsistently maintains a $98\%$$\sim$$100\%$ token acceptance rate across the LLaMA2 series, indicating the high alignment potential of this paradigm. Further analysis validated the effectiveness of \methodacross diverse data streams and its compatibility with various LLM backbones.

Our key contributions are:

- (i)
We performed an empirical analysis of LLM acceleration on layer sparsity, revealing both the potential for LLM self-acceleration via layer sparsity and its task-specific nature, underscoring the necessity for adaptive self-speculative decoding during inference.
- (ii)
Building on these insights, we introduce \method, the first plug-and-play self-speculative decoding algorithm that optimizes the set of skipped layers in the target LLM on the fly, enabling lossless acceleration of LLM inference across diverse input data streams.
- (iii)
We conducted extensive experiments across various models and tasks, demonstrating that \methodconsistently achieves a $1.3\times$$\sim$$1.6\times$ speedup without any auxiliary model or training, while theoretically guaranteeing the preservation of the generated text’s distribution.

## 2 Related Work

##### Speculative Decoding (SD)

Due to the sequential nature of autoregressive decoding, LLM inference is constrained by memory-bound computations (Patterson, ; Shazeer, ), with the primary latency bottleneck arising not from arithmetic computations but from memory reads/writes of LLM parameters (Pope et al., ). To mitigate this issue, speculative decoding (SD) introduces utilizing a compact draft model to predict multiple decoding steps, with the target LLM then validating them in parallel (Xia et al., ; Leviathan et al., ; Chen et al., ). Recent SD variants have sought to enhance efficiency by incorporating additional modules (Kim et al., ; Sun et al., ; Du et al., ; Li et al., ; ) or introducing new training objectives (Liu et al., ; Kou et al., ; Zhou et al., ; Gloeckle et al., ). However, these approaches necessitate extra parameters or extensive training, limiting their applicability across different models. Another line of research has explored plug-and-play SD methods with Jacobi decoding (Santilli et al., ; Fu et al., ), which predict multiple steps in parallel by appending pseudo tokens to the input and refining them iteratively. As shown in Table 1, our work complements these efforts by investigating a novel plug-and-play SD method with layer-skipping, which exploits the inherent sparsity of LLM layers to accelerate inference. The most related approaches to ours include Self-SD (Zhang et al., ) and LayerSkip (Elhoushi et al., ), which also skip intermediate layers of LLMs to form the draft model. However, both methods require a time-consuming offline training process, making them neither plug-and-play nor easily generalizable across different models and tasks.

**Table 1: Comparison of \methodwith existing SD methods. “AM” denotes whether the method requires auxiliary modules such as additional parameters or data stores. “Greedy”, “Sampling”, and “Token Tree” denote whether the method supports greedy decoding, multinomial sampling, and token tree verification, respectively. \methodis the first plug-and-play layer-skipping SD method, which is orthogonal to those Jacobi-based methods such as Lookahead (Fu et al., ).**
| Methods | Drafting | Verification | Speedup |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Approach | AM | Plug&Play | Greedy | Sampling | Token Tree |  |  |
| Eagle (Li et al., ; ) | Draft Heads | Yes | ✗ | ✓ | ✓ | ✓ | - |
| Rest (He et al., ) | Context Retrieval | Yes | ✗ | ✓ | ✓ | ✓ | - |
| Self-SD (Zhang et al., ) | Layer Skipping | No | ✗ | ✓ | ✓ | ✗ | - |
| \hdashlineParallel (Santilli et al., ) | Jacobi Decoding | No | ✓ | ✓ | ✗ | ✗ | $0.9\times$$\sim$$1.0\times$ |
| Lookahead (Fu et al., ) | Jacobi Decoding | No | ✓ | ✓ | ✓ | ✓ | $1.2\times$$\sim$$1.4\times$ |
| \method (Ours) | Layer Skipping | No | ✓ | ✓ | ✓ | ✓ | $1.3\times$$\sim$$1.6\times$ |

##### Efficient LLMs Utilizing Sparsity

LLMs are powerful but often over-parameterized (Hu et al., ). To address this issue, various methods have been proposed to accelerate inference by leveraging different forms of LLM sparsity. One promising research direction is model compression, which includes approaches such as quantization (Dettmers et al., ; Frantar et al., ; Ma et al., ), parameter pruning (Liu et al., ; Hoefler et al., ; Liu et al., ), and knowledge distillation (Touvron et al., ; Hsieh et al., ; Gu et al., ). These approaches aim to reduce model sparsity by compressing LLMs into more compact forms, thereby decreasing memory usage and computational overhead during inference. Our proposed method, \method, focuses specifically on sparsity within LLM layer computations, providing a more streamlined approach to efficient LLM inference that builds upon recent advances in layer skipping (Corro et al., ; Zhu et al., ; Jaiswal et al., ; Liu et al., ). Unlike these existing layer-skipping methods that may lead to information loss and performance degradation, \methodinvestigates the utilization of layer sparsity to enable lossless acceleration of LLM inference.

## 3 Preliminaries

### 3.1 Self-Speculative Decoding

Unlike most SD methods that require additional parameters, self-speculative decoding (Self-SD) first proposed utilizing parts of an LLM as a compact draft model (Zhang et al., ). In each decoding step, this approach skips intermediate layers of the LLM to efficiently generate draft tokens; these tokens are then validated in parallel by the full-parameter LLM to ensure that the output distribution of the target LLM remains unchanged. The primary challenge of Self-SD lies in determining which layers, and how many, should be skipped – referred to as the skipped layer set – during the drafting stage, which is formulated as an optimization problem. Formally, given the input data $\mathcal{X}$ and the target LLM $\mathscr{M}_{T}$ with $L$ layers (including both attention and MLP layers), Self-SD aims to identify the optimal skipped layer set $\bm{z}$ that minimizes the average inference time per token:

$$ $\bm{z}^{*}=\underset{\bm{z}}{\arg\min}\frac{\sum_{\bm{x}\in\mathcal{X}}f\left( \bm{x}\mid\bm{z};\bm{\theta}_{\mathscr{M}_{T}}\right)}{\sum_{\bm{x}\in\mathcal {X}}|\bm{x}|},\quad\text{ s.t. }\bm{z}\in\{0,1\}^{L},$ (1) $$

where $f(\cdot)$ is a black-box function that returns the inference latency of sample $\bm{x}$, $\bm{z}_{i}\in\{0,1\}$ denotes whether layer $i$ of the target LLM is skipped when drafting, and $|\bm{x}|$ represents the sample length. Self-SD addresses this problem through a Bayesian optimization process (Jones et al., ). Before inference, this process iteratively selects new inputs $\bm{z}$ based on a Gaussian process (Rasmussen & Williams, ) and evaluates Eq (1) on the training set of $\mathcal{X}$. After a specified number of iterations, the best $\bm{z}$ is considered an approximation of $\bm{z}^{*}$ and is held fixed for inference.

While Self-SD has proven effective, its reliance on a time-intensive Bayesian optimization process poses certain limitations. For each task, Self-SD must sequentially evaluate all selected training samples during every iteration to optimize Eq (1); Moreover, the computational burden of Bayesian optimization escalates substantially with the number of iterations. As a result, processing just eight CNN/Daily Mail (Nallapati et al., ) samples for 1000 Bayesian iterations requires nearly 7.5 hours for LLaMA-2-13B and 20 hours for LLaMA-2-70B on an NVIDIA A6000 server. These computational demands restrict the generalizability of Self-SD across different models and tasks.

### 3.2 Experimental Observations

This subsection delves into Self-SD, exploring the plug-and-play potential of this layer-skipping SD paradigm for lossless LLM inference acceleration. Our key findings are detailed below.

Figure: Figure 2: (a) LLMs possess self-acceleration potential via layer sparsity. By utilizing drafts from the top-$k$ candidates, we found that uniformly skipping half of the layers during drafting yields a notable $1.2\times$ speedup. (b) Layer sparsity is task-specific. Each task requires its own optimal set of skipped layers, and applying the skipped layer configuration from one task to another can lead to substantial performance degradation. “X LS” represents the skipped layer set optimized for task X.
Refer to caption: x2.png

#### 3.2.1 LLMs Possess Self-Acceleration Potential via Layer Sparsity

We begin by investigating the potential of behavior alignment between the target LLM and its layer-skipping variant. Unlike previous work (Zhang et al., ) that focused solely on greedy draft predictions, we leverage potential draft candidates from top-$k$ predictions, as detailed in Section 4.2. We conducted experiments using LLaMA-2-13B across the CNN/Daily Mail (Nallapati et al., ), GSM8K (Cobbe et al., ), and TinyStories (Eldan & Li, ) datasets. We applied a uniform layer-skipping pattern with $k$ set to 10. The experimental results, illustrated in Figure 2(a), demonstrate a $30\%$ average improvement in the token acceptance rate by leveraging top-$k$ predictions, with over $90\%$ of draft tokens accepted by the target LLM. Consequently, compared to Self-SD, which achieved a maximum speedup of $1.01\times$ in this experimental setting, we revealed that the layer-skipping SD paradigm could yield an average wall-clock speedup of $1.22\times$ over conventional autoregressive decoding with a uniform layer-skipping pattern. This finding challenges the prevailing belief that the selection of skipped layers must be meticulously curated, suggesting instead that LLMs possess greater potential for self-acceleration through inherent layer sparsity.

#### 3.2.2 Layer Sparsity is Task-specific

We further explore the following research question: Is the skipped layer set optimized for one specific task applicable to other tasks? To address this, we conducted domain shift experiments using LLaMA-2-13B on the CNN/Daily Mail, GSM8K, TinyStories, and WMT16 DE-EN datasets. The experimental results, depicted in Figure 2(b), reveal two critical findings:
1) Each task requires its own optimal skipped layer set. As illustrated in Figure 2(b), the highest speedup performance is consistently achieved by the skipped layer configuration specifically optimized for each task. The detailed configuration of these layers is presented in Appendix A, demonstrating that the optimal configurations differ across tasks. 2) Applying the static skipped layer configuration across different tasks can lead to substantial efficiency degradation. For example, the speedup decreases from $1.47\times$ to $1.01\times$ when the optimized skipped layer set from a storytelling task is applied to a mathematical reasoning task, indicating that the optimized skipped layer set for one specific task does not generalize effectively to others.

These findings lay the groundwork for our plug-and-play solution within layer-skipping SD. Section 3.2.1 provides a strong foundation for real-time skipped layer selection, suggesting that additional optimization using training data may be unnecessary; Section 3.2.2 highlights the limitations of static layer-skipping patterns for dynamic input data streams across various tasks, underscoring the necessity for adaptive layer optimization during inference. Building on these insights, we present our on-the-fly self-speculative decoding method for efficient and adaptive layer set optimization.

## 4 SWIFT: On-the-Fly Self-Speculative Decoding

We introduce \method, the first plug-and-play self-speculative decoding approach that optimizes the skipped layer set of the target LLM on the fly, facilitating lossless LLM acceleration across diverse input data streams. As shown in Figure 3, \methoddivides LLM inference into two distinct phases: (1) context-based layer set optimization (§4.1), which aims to identify the optimal skipped layer set given the input stream, and (2) confidence-aware inference acceleration (§4.2), which employs the determined configuration to accelerate LLM inference.

Figure: Figure 3: Timeline of \methodinference. N denotes the maximum generation length per instance.
Refer to caption: x3.png

### 4.1 Context-based Layer Set Optimization

Layer set optimization is a critical challenge in self-speculative decoding, as it determines which layers of the target LLM should be skipped to form the draft model (see Section 3.1). Unlike prior methods that rely on time-intensive offline optimization, our work emphasizes on-the-fly layer set optimization, which poses a greater challenge to the latency-accuracy trade-off: the optimization must be efficient enough to avoid delays during inference while ensuring accurate drafting of subsequent decoding steps. To address this, we propose an adaptive optimization mechanism that balances efficiency with drafting accuracy. Our method minimizes overhead by performing only a single forward pass of the draft model per step to validate potential skipped layer set candidates. The core innovation is the use of LLM-generated tokens (i.e., prior context) as ground truth, allowing for simultaneous validation of the draft model’s accuracy in predicting future decoding steps.

In the following subsections, we illustrate the detailed process of this optimization phase for each input instance, which includes context accumulation (§4.1.1) and layer set optimization (§4.1.2).

#### 4.1.1 Context Accumulation

Given an input instance in the optimization phase, the draft model is initialized by uniformly skipping layers in the target LLM. This initial layer-skipping pattern is maintained to accelerate inference until a specified number of LLM-generated tokens, referred to as the context window, has been accumulated. Upon reaching this window length, the inference transitions to layer set optimization.

Figure: Figure 4: Layer set optimization process in \method. During the optimization stage, \methodperforms an optimization step prior to each LLM decoding step to adjust the skipped layer set, which involves: (a) Efficient layer set optimization. \methodintegrates random search with interval Bayesian optimization to propose layer set candidates; (b) Parallel candidate evaluation. \methoduses LLM-generated tokens (i.e., prior context) as ground truth, enabling simultaneous validation of the proposed candidates. The best-performing layer set is selected to accelerate the current decoding step.
Refer to caption: x4.png

#### 4.1.2 Layer Set Optimization

During this stage, as illustrated in Figure 4, we integrate an optimization step before each LLM decoding step to refine the skipped layer set, which comprises two substeps:

##### Efficient Layer Set Suggestion

This substep aims to suggest a potential layer set candidate. Formally, given a target LLM $\mathscr{M}_{T}$ with $L$ layers, our goal is to identify an optimal skipped layer set $\bm{z}\in\{0,1\}^{L}$ to form the compact draft model. Unlike Zhang et al. (), which relies entirely on a time-consuming Bayesian optimization process, we introduce an efficient strategy that combines random search with Bayesian optimization. In this approach, random sampling efficiently handles most of the exploration. Specifically, given a fixed skipping ratio $r$, \methodapplies Bayesian optimization at regular intervals of $\beta$ optimization steps (e.g., $\beta=25$) to suggest the next layer set candidate, while random search is employed during other optimization steps.

$$ $\bm{z}=\left\{\begin{array}[]{ll}\operatorname{Bayesian_Optimization}(\bm{l}) &\text{ if }o\text{ \% }\beta=0\\ \operatorname{Random_Search}(\bm{l})&\text{ otherwise }\end{array},\right.$ (2) $$

where $1\leq o\leq S$ is the current optimization step; $S$ denotes the maximum number of optimization steps; $\bm{l}=\binom{L}{rL}$ denotes the input space, i.e., all possible combinations of layers that can be skipped.

##### Parallel Candidate Evaluation

leverages LLM-generated context to simultaneously validate the candidate draft model’s performance in predicting future decoding steps. Formally, given an input sequence $\bm{x}$ and the previously generated tokens within the context window, denoted as $\bm{y}=\{y_{1},\dots,y_{\gamma}\}$, the draft model $\mathscr{M}_{D}$, which skips the designated layers $\bm{z}$ of the target LLM, is employed to predict these context tokens in parallel:

$$ $y^{\prime}_{i}=\arg\max_{y}\log P\left(y\mid\bm{x},\bm{y}_{<i};\bm{\theta}_{ \mathscr{M}_{D}}\right),1\leq i\leq\gamma,$ (3) $$

where $\gamma$ represents the context window. The cached key-value pairs in the target LLM $\mathscr{M}_{T}$ are reused by $\mathscr{M}_{D}$, presumably aligning $\mathscr{M}_{D}$’s distribution with $\mathscr{M}_{T}$ and reducing the redundant computation. The matchness score is defined as the exact match ratio between $\bm{y}$ and $\bm{y}^{\prime}$:

$$ $\texttt{matchness}=\frac{\sum_{i}\mathbb{I}\left(y_{i}=y^{\prime}_{i}\right)}{ \gamma},1\leq i\leq\gamma,$ (4) $$

where $\mathbb{I}(\cdot)$ denotes the indicator function. This score serves as the optimization objective during optimization, reflecting $\mathscr{M}_{D}$’s accuracy in predicting future decoding steps. As shown in Figure 4, the matchness score at each step is integrated into the Gaussian process model to guide Bayesian optimization, with the highest-scoring layer set candidate being retained to form the draft model.

As illustrated in Figure 3, the process of context accumulation and layer set optimization alternates for each instance until a termination condition is met – either the maximum number of optimization steps is reached or the best candidate remains unchanged over multiple iterations. Once the optimization phase concludes, the inference process transitions to the confidence-aware inference acceleration phase, where the optimized draft model is employed to speed up LLM inference.

### 4.2 Confidence-aware Inference Acceleration

Figure: Figure 5: Confidence-aware inference process of \method. (a) The drafting terminates early if the confidence score drops below threshold $\epsilon$. (b) Draft candidates are dynamically selected based on confidence and then verified in parallel by the target LLM.
Refer to caption: x5.png

During the acceleration phase, the optimization step is removed. \methodapplies the best-performed layer set to form the compact draft model and decodes following the draft-then-verify paradigm. Specifically, at each decoding step, given the input $\bm{x}$ and previous LLM outputs $\bm{y}$, the draft model $\mathscr{M}_{D}$ predicts future LLM decoding steps in an autoregressive manner:

$$ $y^{\prime}_{j}=\arg\max_{y}\log P\left(y\mid\bm{x},\bm{y},\bm{y}^{\prime}_{<j} ;\bm{\theta}_{\mathscr{M}_{D}}\right),$ (5) $$

where $1\leq j\leq N_{D}$ is the current draft step, $N_{D}$ denotes the maximum draft length, $\bm{y}^{\prime}_{<j}$ represents previous draft tokens, and $P(\cdot)$ denotes the probability distribution of the next draft token. The KV cache of the target LLM $\mathscr{M}_{T}$ and preceding draft tokens $\bm{y}^{\prime}_{<j}$ is reused to reduce the computational cost.

Let $p_{j}=\max P(\cdot)$ denote the probability of the top-1 draft prediction $y^{\prime}_{j}$, which can be regarded as a confidence score. Recent research (Li et al., ; Du et al., ) shows that this score is highly correlated with the likelihood that the draft token $y^{\prime}_{j}$ will pass verification – higher confidence scores indicate a greater chance of acceptance. Therefore, following previous studies (Zhang et al., ; Du et al., ), we leverage the confidence score to prune unnecessary draft steps and select valuable draft candidates, improving both speculation accuracy and verification efficiency.

As shown in Figure 5, we integrate \methodwith two confidence-aware inference strategies(^1^11These confidence-aware inference strategies are also applied during the optimization phase, where the current optimal layer set is used to form the draft model and accelerate the corresponding LLM decoding step.): 1) Early-stopping Drafting. The autoregressive drafting process halts if the confidence $p_{j}$ falls below a specified threshold $\epsilon$, avoiding any waste of subsequant drafting computation. 2) Dynamic Verification. Each $y^{\prime}_{j}$ is dynamically extended with its top-$k$ draft predictions for parallel verification to enhance speculation accuracy, with $k$ determined by the confidence score $p_{j}$. Concretely, $k$ is set to 10, 5, 3, and 1 for $p$ in the ranges of $(0,0.5]$, $(0.5,0.8]$, $(0.8,0.95]$, and $(0.95,1]$, respectively. All draft candidates are linearized into a single sequence and verified in parallel by the target LLM using a special causal attention mask (see Figure 5 (b)).

## 5 Experiments

### 5.1 Experimental Setup

##### Implementation Details

We mainly evaluate \methodon LLaMA-2 (Touvron et al., ) and CodeLLaMA series (Rozière et al., ) across various tasks, including summarization, mathematical reasoning, storytelling, and code generation. The evaluation datasets include CNN/Daily Mail (CNN/DM) (Nallapati et al., ), GSM8K (Cobbe et al., ), TinyStories (Eldan & Li, ), and HumanEval (Chen et al., ). The maximum generation lengths on CNN/DM, GSM8K, and TinyStories are set to 64, 64, and 128, respectively. We conduct 1-shot evaluation for CNN/DM and TinyStories, and 5-shot evaluation for GSM8K. We compare pass@1 and pass@10 for HumanEval. We randomly sample 1000 instances from the test set for each dataset except HumanEval. The maximum generation lengths for HumanEval and all analyses are set to 512. During optimization, we employ both random search and Bayesian optimization(^2^22[https://github.com/bayesian-optimization/BayesianOptimization](https://github.com/bayesian-optimization/BayesianOptimization)) to suggest skipped layer set candidates. Following prior work, we adopt speculative sampling (Leviathan et al., ) as our acceptance strategy with a batch size of 1. Detailed setups are provided in Appendix B.1 and B.2.

##### Baselines

In our main experiments, we compare \methodto two existing plug-and-play methods: Parallel Decoding (Santilli et al., ) and Lookahead Decoding (Fu et al., ), both of which employ Jacobi decoding for efficient LLM drafting. It is important to note that \method, as a layer-skipping SD method, is orthogonal to these Jacobi-based SD methods, and integrating \methodwith them could further boost inference efficiency. We exclude other SD methods from our comparison as they necessitate additional modules or extensive training, which limits their generalizability.

##### Evaluation Metrics

We report two widely-used metrics for \methodevaluation: mean generated length $M$ (Stern et al., ) and token acceptance rate $\alpha$ (Leviathan et al., ). Detailed descriptions of these metrics can be found in Appendix B.3. In addition to these metrics, we report the actual decoding speed (tokens/s) and wall-time speedup ratio compared with vanilla autoregressive decoding. The acceleration of \methodtheoretically guarantees the preservation of the target LLMs’ output distribution, making it unnecessary to evaluate the generation quality. However, to provide a point of reference, we present the evaluation scores for code generation tasks.

**Table 2: Comparison between \methodand prior plug-and-play methods. We report the mean generated length M, speedup ratio, and average decoding speed (tokens/s) under greedy decoding. ^† indicates results with a token acceptance rate $\alpha$ above 0.98. More details are provided in Appendix C.1.**
| Models | Methods | CNN/DM | GSM8K | TinyStories | Speed<br>(tokens/s) | Overall<br>Speedup |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| M | Speedup | M | Speedup | M | Speedup |  |  |  |  |
| LLaMA-2-13B | Vanilla | 1.00 | 1.00$\times$ | 1.00 | 1.00$\times$ | 1.00 | 1.00$\times$ | 20.10 | 1.00$\times$ |
| Parallel | 1.04 | 0.95$\times$ | 1.11 | 0.99$\times$ | 1.06 | 0.97$\times$ | 19.49 | 0.97$\times$ |  |
| Lookahead | 1.38 | 1.16$\times$ | 1.50 | 1.29$\times$ | 1.62 | 1.37$\times$ | 25.46 | 1.27$\times$ |  |
| \method | 4.34 | 1.37$\times$^† | 3.13 | 1.31$\times$^† | 8.21 | 1.53$\times$^† | 28.26 | 1.41$\times$ |  |
| LLaMA-2-13B<br>-Chat | Vanilla | 1.00 | 1.00$\times$ | 1.00 | 1.00$\times$ | 1.00 | 1.00$\times$ | 19.96 | 1.00$\times$ |
| Parallel | 1.06 | 0.96$\times$ | 1.08 | 0.97$\times$ | 1.10 | 0.98$\times$ | 19.26 | 0.97$\times$ |  |
| Lookahead | 1.35 | 1.15$\times$ | 1.57 | 1.31$\times$ | 1.66 | 1.40$\times$ | 25.69 | 1.29$\times$ |  |
| \method | 3.54 | 1.28$\times$ | 2.95 | 1.25$\times$ | 7.42 | 1.50$\times$^† | 26.80 | 1.34$\times$ |  |
| LLaMA-2-70B | Vanilla | 1.00 | 1.00$\times$ | 1.00 | 1.00$\times$ | 1.00 | 1.00$\times$ | 4.32 | 1.00$\times$ |
| Parallel | 1.05 | 0.95$\times$ | 1.07 | 0.97$\times$ | 1.05 | 0.96$\times$ | 4.14 | 0.96$\times$ |  |
| Lookahead | 1.36 | 1.15$\times$ | 1.54 | 1.30$\times$ | 1.59 | 1.35$\times$ | 5.45 | 1.26$\times$ |  |
| \method | 3.85 | 1.43$\times$^† | 2.99 | 1.39$\times$^† | 6.17 | 1.62$\times$^† | 6.41 | 1.48$\times$ |  |

### 5.2 Main Results

Table 2 presents the comparison between \methodand previous plug-and-play methods on text generation tasks. The experimental results demonstrate the following findings: (1) \methodshows superior efficiency over prior methods, achieving consistent speedups of $1.3\times$$\sim$$1.6\times$ over vanilla autoregressive decoding across various models and tasks. (2) The efficiency of \methodis driven by the high behavior consistency between the target LLM and its layer-skipping draft variant. As shown in Table 2, \methodproduces a mean generated length M of 5.01, with a high token acceptance rate $\alpha$ ranging from $90\%$ to $100\%$. Notably, for the LLaMA-2 series, this acceptance rate remains stable at $98\%$$\sim$$100\%$, indicating that nearly all draft tokens are accepted by the target LLM. (3) Compared with 13B models, LLaMA-2-70B achieves higher speedups with a larger layer skip ratio ($0.45$$\rightarrow$$0.5$), suggesting that larger-scale LLMs exhibit greater layer sparsity. This underscores \method’s potential to deliver even greater speedups as LLM scales continue to grow. A detailed analysis of this finding is presented in Section 5.3.

**Table 3: Experimental results of \methodon code generation tasks. We report the mean generated length M, acceptance rate $\alpha$, accuracy (Acc.), and speedup ratio for comparison. We use greedy decoding for pass@1 and random sampling with a temperature of 0.6 for pass@10.**
| Datasets | Methods | CodeLLaMA-13B | CodeLLaMA-34B |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| M | $\alpha$ | Acc. | Speedup | M | $\alpha$ | Acc. | Speedup |  |  |
| HumanEval (pass@1) | Vanilla | 1.00 | - | 0.311 | 1.00$\times$ | 1.00 | - | 0.372 | 1.00$\times$ |
| \method | 4.75 | 0.98 | 0.311 | 1.40$\times$ | 3.79 | 0.88 | 0.372 | 1.46$\times$ |  |
| HumanEval (pass@10) | Vanilla | 1.00 | - | 0.628 | 1.00$\times$ | 1.00 | - | 0.677 | 1.00$\times$ |
| \method | 3.55 | 0.93 | 0.628 | 1.29$\times$ | 2.79 | 0.90 | 0.683 | 1.30$\times$ |  |

Table 3 shows the evaluation results of \methodon code generation tasks. \methodachieves speedups of $1.3\times$$\sim$$1.5\times$ over vanilla autoregressive decoding, demonstrating its effectiveness across both greedy decoding and random sampling settings. Additionally, speculative sampling theoretically guarantees that \methodmaintains the original output distribution of the target LLM. This is empirically validated by the task performance metrics in Table 3. Despite a slight variation in the pass@10 metric for CodeLLaMA-34B, \methodachieves identical performance to autoregressive decoding.

### 5.3 In-depth Analysis

##### Illustration of Inference

As described in Section 4, \methoddivides the LLM inference process into two distinct phases: optimization and acceleration. Figure 6 (left) illustrates the detailed acceleration effect of \methodduring LLM inference. Specifically, the optimization phase begins at the start of inference, where an optimization step is performed before each decoding step to adjust the skipped layer set forming the draft model. As shown in Figure 6, in this phase, the matchness score of the draft model rises sharply from 0.45 to 0.73 during the inference of the first instance. This score then gradually increases to 0.98, which triggers the termination of the optimization process. Subsequently, the inference transitions to the acceleration phase, during which the optimization step is removed, and the draft model remains fixed to accelerate LLM inference. As illustrated, the instance speedup increases with the matchness score, reaching an average of $1.53\times$ in the acceleration phase. The overall speedup gradually rises as more tokens are generated, eventually approaching the average instance speedup. This dynamic reflects a key feature of \method: the efficiency of \methodimproves with increasing input length and the number of instances.

Figure: Figure 6: Illustration and latency breakdown of \methodinference. As the left figure shows, after the context-based layer set optimization phase, the overall speedup of \methodsteadily increases, reaching the average instance speedup during the acceleration phase. The additional optimization steps account for only $\bf{0.8\%}$ of the total inference latency, as illustrated in the right figure.
Refer to caption: x6.png

##### Breakdown of Computation

Figure 6 (right) presents the computation breakdown of different modules in \methodwith 1000 CNN/DM samples using LLaMA-2-13B. The results demonstrate that the optimization step only takes $\bf{0.8\%}$ of the overall inference process, indicating the efficiency of our strategy. Compared with Self-SD (Zhang et al., ) that requires a time-consuming optimization process (e.g., 7.5 hours for LLaMA-2-13B on CNN/DM), \methodachieves a nearly 180$\times$ optimization time reduction, facilitating on-the-fly inference acceleration. Besides, the results show that the drafting stage of \methodconsumes the majority of inference latency. This is consistent with our results of mean generated length in Table 2 and 3, which shows that nearly $80\%$ output tokens are generated by the efficient draft model, demonstrating the effectiveness of our \methodframework.

Figure: Figure 7: Comparison between \methodand Self-SD in handling dynamic data input streams. Unlike Self-SD, which suffers from efficiency reduction during distribution shift, \methodmaintains stable acceleration performance with an acceptance rate exceeding 0.9.
Refer to caption: x7.png

##### Dynamic Input Data Streams

We further validate the effectiveness of \methodin handling dynamic input data streams. We selected CNN/DM, GSM8K, Alpaca (Taori et al., ), WMT14 DE-EN, and Nature Questions (Kwiatkowski et al., ) for the evaluation on summarization, reasoning, instruction following, translation, and question answering tasks, respectively. For each task, we randomly sample 500 instances from the test set and concatenate them task-by-task to form the input stream. The experimental results are presented in Figure 7. As demonstrated, Self-SD is sensitive to domain shifts, with the average token acceptance rate dropping from $92\%$ to $68\%$. Consequently, it suffers from severe speedup reduction from $1.33\times$ to an average of $1.05\times$ under domain shifts. In contrast, \methodexhibits promising adaptation capability to different domains with an average token acceptance rate of $96\%$, leading to a consistent $1.3\times$$\sim$$1.6\times$ speedup.

Figure: Figure 8: In-depth analysis of \method, which includes: (a) Flexible optimization strategy. The maximum optimization iteration $S$ and Bayesian interval $\beta$ can be flexibly adjusted to accommodate different input data types. (b) Scaling law. The speedup and optimal layer skip ratio of \methodincrease with larger model sizes, indicating that larger LLMs exhibit greater layer sparsity.
Refer to caption: x8.png

##### Flexible Optimization & Scaling Law

Figure 8(a) presents the flexibility of \methodin handling various input types by adjusting the maximum optimization step $S$ and Bayesian interval $\beta$. For input with fewer instances, reducing $S$ enables an earlier transition to the acceleration phase while increasing $\beta$ reduces the overhead during the optimization phase, enhancing speedups during the initial stages of inference. In cases with sufficient input data, \methodenables exploring more optimization paths, thereby enhancing the overall speedup. Figure 8(b) illustrates the scaling law of \method: as the model size increases, both the optimal layer-skip ratio and overall speedup improve, indicating that larger LLMs exhibit more layer sparsity. This finding highlights the potential of \methodfor accelerating LLMs of larger sizes (e.g., 175B), which we leave for future investigation.

Figure: Figure 9: Speedups of \methodon LLM backbones and their instruction-tuned variants.
Refer to caption: x9.png

##### Other LLM Backbones

Beyond LLaMA, we assess the effectiveness of \methodon additional LLM backbones. Specifically, we include Yi-34B (Young et al., ) and DeepSeek-Coder-33B (Guo et al., ) along with their instruction-tuned variants for text and code generation tasks, respectively. The speedup results of \methodare illustrated in Figure 9, demonstrating that \methodachieves efficiency improvements ranging from $26\%$ to $54\%$ on these LLM backbones. Further experimental details are provided in Appendix C.2.

## 6 Conclusion

In this work, we introduce \method, an on-the-fly self-speculative decoding algorithm that adaptively selects certain intermediate layers of LLMs to skip during inference. The proposed method does not require additional training or auxiliary models, making it a plug-and-play solution for accelerating LLM inference across diverse input data streams. Extensive experiments conducted across various LLMs and tasks demonstrate that \methodachieves over a $1.3\times$$\sim$$1.6\times$ speedup while preserving the distribution of the generated text. Furthermore, our in-depth analysis highlights the effectiveness of \methodin handling dynamic input data streams and its seamless integration with various LLM backbones, showcasing the great potential of this paradigm for practical LLM inference acceleration.

## Ethics Statement

The datasets used in our experiments are publicly released and labeled through interaction with humans in English. In this process, user privacy is protected, and no personal information is contained in the dataset. The scientific artifacts that we used are available for research with permissive licenses. The use of these artifacts in this paper is consistent with their intended purpose.

## Reproducibility Statement

All the results in this work are reproducible. We provide all the necessary code in the Supplementary Material to replicate our results. The repository includes environment configurations, scripts, and other relevant materials. We discuss the experimental settings in Section 5.1 and Appendix C, including implementation details such as models, datasets, inference setup, and evaluation metrics.

## Appendix

## Appendix A Preliminary Details

We present the detailed configuration of Self-SD across four task domains in Figure 10, demonstrating that the optimal skipped layer configurations vary depending on the specific task.

Figure: (a) Summarization - CNN/DM
Refer to caption: x10.png

## Appendix B Experimental Setups

### B.1 Models and Datasets

Our experiments mainly evaluate the effectiveness of \methodon LLaMA-2 (Touvron et al., ) and CodeLLaMA series (Rozière et al., ). We provide empirical validation on a diverse range of generation tasks. For summarization, mathematical reasoning, storytelling, and code generation tasks, we chose the CNN/Daily Mail (CNN/DM) (Nallapati et al., ), GSM8K (Cobbe et al., ), TinyStories (Eldan & Li, ), and HumanEval (Chen et al., ) datasets, respectively. We perform 1-shot evaluation for CNN/DM and TinyStories, and 5-shot evaluation for GSM8K. The maximum generation lengths on CNN/DM, GSM8K, and TinyStories are set to 64, 64, and 128, respectively. We compare pass@1 and pass@10 for HumanEval. In our further analysis, we include three more datasets to validate the capability of \methodin handling dynamic input data streams. Specifically, we select Alpaca (Taori et al., ), WMT14 DE-EN, and Nature Questions (Kwiatkowski et al., ) for the instruction following, translation, and question answering tasks, respectively. The maximum generation lengths for HumanEval and all analyses are set to 512. We randomly sample 1000 instances from the test set for each dataset except HumanEval.

### B.2 Inference Setup

In the optimization phase, we employ both random search and Bayesian optimization to suggest potential skipped layer set candidates, striking a balance between optimization performance and efficiency. The context window $\gamma$ is set to 32. The maximum draft length $N_{D}$ is set to 25. For random sampling in code generation tasks, we apply a temperature of 0.6 and $top_p=0.95$. The maximum number of layer set optimization steps $S$ is set to 1000, with Bayesian optimization performed every $\beta=25$ steps. The optimization phase is set to be early stopped if the matchness score does not improve after 300 steps or exceeds 0.95. The layer skip ratio $r$ is fixed at 0.45 for the 13B model and 0.5 for the 34B and 70B models. All experiments were conducted using Pytorch 2.1.0 on 4$\times$NVIDIA RTX A6000 GPU (40GB) with CUDA 12.1, and an Intel(R) Xeon(R) Platinum 8370C CPU with 32 cores. Inference for our method and all baselines was performed using the Huggingface transformers package. Following prior work, we adopt speculative sampling (Leviathan et al., ) as our acceptance strategy, and the batch size is set to 1.

### B.3 Evaluation Metrics

This subsection provides a detailed illustration of our evaluation metrics, which are mean generated length $M$ and token acceptance rate $\alpha$. Specifically, the mean generated length $M$ refers to the average number of output tokens produced per forward pass of the target LLM; the acceptance rate $\alpha$ is defined as the ratio of accepted tokens to the total number of draft steps. In other words, it represents the expected probability of the target LLM accepting a potential token from a forward pass of the draft model. These two metrics are independent of computational hardware and, therefore considered as more objective metrics. Given the mean generated length $M$, acceptance rate $\alpha$, and the layer skip ratio $r$, the mathematical formula for the expected wall-time speedup during the acceleration phase is derived as follows:

$$ $\mathbb{E}(\text{Spd.})=\frac{M}{(M-1)\times\frac{c}{\alpha}+1}=\frac{M\alpha} {(M-1)c+\alpha},\quad c=1-r,$ (6) $$

where $c$ is defined as the cost coefficient in Leviathan et al. (). It is calculated as the ratio between the single forward time of the draft model and that of the target LLM. In summary, the ideal speedup will be higher with the larger $M$ and $\alpha$ and smaller $c$.

## Appendix C Experimental Details

### C.1 Details of Main Results

We present the detailed statistics of our main experimental results in Table 4. \methodconsistently achieves a token acceptance rate $\alpha$ exceeding $90\%$ across all evaluation settings, with the mean generated length M ranging from 2.99 to 8.21. These statistics indicate strong behavior alignment between the target LLM and its layer-skipping draft variant, as discussed in Section 5.2. Additionally, we report the expected speedup $\mathbb{E}(\text{Spd.})$ calculated using Eq (6), indicating that the current implementation of \methodhas significant potential for further optimization to boost its efficiency.

**Table 4: Detailed results of \methodon text generation tasks using LLaMA-2 series. We report the mean generated length M, token acceptance rate $\alpha$, and the expected speedup $\mathbb{E}(\text{Spd.})$ calculated by Eq (6) under the setting of greedy decoding with FP16 precision.**
| Models | Methods | CNN/DM | GSM8K | TinyStories | Expected<br>Speedup |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| M | $\alpha$ | $\mathbb{E}(\text{Spd.})$ | M | $\alpha$ | $\mathbb{E}(\text{Spd.})$ | M | $\alpha$ | $\mathbb{E}(\text{Spd.})$ |  |  |  |
| LLaMA-2-13B | Vanilla | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00$\times$ |
| \method | 4.34 | 0.99 | 1.52$\times$ | 3.13 | 0.98 | 1.43$\times$ | 8.21 | 1.00 | 1.65$\times$ | 1.53$\times$ |  |
| LLaMA-2-13B<br>-Chat | Vanilla | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00$\times$ |
| \method | 3.54 | 0.90 | 1.39$\times$ | 2.95 | 0.92 | 1.36$\times$ | 7.42 | 0.99 | 1.62$\times$ | 1.46$\times$ |  |
| LLaMA-2-70B | Vanilla | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00$\times$ |
| \method | 3.85 | 0.99 | 1.58$\times$ | 2.99 | 0.98 | 1.48$\times$ | 6.17 | 0.99 | 1.71$\times$ | 1.59$\times$ |  |

**Table 5: Experimental results of \methodon text generation tasks using Yi-34B series. We report the mean generated length M, token acceptance rate $\alpha$ and speedup ratio under the setting of greedy decoding with FP16 precision. The skip ratio $r$ is set to 0.45.**
| Models | Methods | CNN/DM | GSM8K | TinyStories | Overall<br>Speedup |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| M | $\alpha$ | Speedup | M | $\alpha$ | Speedup | M | $\alpha$ | Speedup |  |  |  |
| Yi-34B | Vanilla | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00$\times$ |
| \method | 2.74 | 0.94 | 1.30$\times$ | 2.65 | 0.97 | 1.28$\times$ | 3.25 | 0.98 | 1.34$\times$ | 1.31$\times$ |  |
| Yi-34B-Chat | Vanilla | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ | 1.00$\times$ |
| \method | 2.84 | 0.91 | 1.29$\times$ | 2.77 | 0.89 | 1.27$\times$ | 2.52 | 0.80 | 1.21$\times$ | 1.26$\times$ |  |

### C.2 Detailed Results of LLM Backbones

To further validate the effectiveness of \method, we conducted experiments using additional LLM backbones beyond the LLaMA series. Specifically, we select two recently representative LLMs: Yi-34B for text generation and DeepSeek-Coder-33B for code generation tasks. The experimental results are illustrated in Table 5 and 6, demonstrating the efficacy of \methodacross these LLM backbones. \methodachieves a consistent $1.2\times$$\sim$$1.3\times$ wall-clock speedup on the Yi-34B series and a $1.3\times$$\sim$$1.5\times$ on the DeepSeek-Coder-33B series. Notably, for the DeepSeek-Coder-33B series, \methodattains a mean generate length M ranging from 3.16 to 4.17, alongside a token acceptance rate $\alpha$ exceeding $83\%$. These findings substantiate the utility of \methodas a general-purpose, plug-and-play SD method, offering promising inference acceleration across diverse LLM backbones.

**Table 6: Experimental results of \methodon code generation tasks using DeepSeek-Coder-33B series. The skip ratio $r$ is set to 0.5. We use greedy decoding for pass@1 and random sampling with a temperature of 0.6 for pass@10. “DS” denotes the abbreviation of DeepSeek.**
| Datasets | Methods | DS-Coder | DS-Coder-Instruct |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| M | $\alpha$ | Speedup | M | $\alpha$ | Speedup |  |  |
| HumanEval (pass@1) | Vanilla | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ |
| \method | 4.97 | 0.99 | 1.54$\times$ | 3.80 | 0.88 | 1.39$\times$ |  |
| HumanEval (pass@10) | Vanilla | 1.00 | - | 1.00$\times$ | 1.00 | - | 1.00$\times$ |
| \method | 3.16 | 0.91 | 1.36$\times$ | 3.74 | 0.83 | 1.31$\times$ |  |

## Appendix D Further Analysis

### D.1 Ablation Study

Table 7 presents the ablation study of \methodusing LLaMA-2-13B on CNN/DM. The experimental results demonstrate that each component of \methodcontributes to its overall speedup of \method. Specifically, early-stopping drafting effectively reduces the number of ineffective draft steps, improving the token acceptance rate $\alpha$ by $55\%$. Dynamic verification further enhances efficiency by selecting suitable draft candidates from the top-$k$ predictions based on their confidence scores; removing this component leads to a decrease in both the mean generated length (M) and the overall speedup ratio. Additionally, the optimization phase refines the set of skipped layers, improving speedup by $34\%$ compared to the initial uniform layer-skipping strategy. In summary, these results confirm the effectiveness of each proposed innovation in \method.

**Table 7: Ablation study of \method. “ver.” denotes the abbreviation of verification.**
| Methods | M | $\alpha$ | Speedup |
| --- | --- | --- | --- |
| Vanilla | 1.0 | - | 1.000$\times$ |
| \method | 5.82 | 0.98 | 1.560$\times$ |
| w/o early-stopping | 11.16 | 0.43 | 0.896$\times$ |
| w/o dynamic ver. | 5.72 | 0.98 | 1.541$\times$ |
| w/o optimization | 2.15 | 0.90 | 1.224$\times$ |

### D.2 Context Window

In Table 8, we show a detailed analysis of context window $\gamma$, which determines the number of LLM-generated tokens used in the layer set optimization process. A smaller $\gamma$ introduces greater randomness in the matchness score calculation, resulting in suboptimal performance, while a larger $\gamma$ increases the computational overhead of the optimization step. The results indicate that $\gamma=32$ provides an optimal balance between optimization performance and computational overhead.

### D.3 Optimization Steps

We present the detailed configuration of \methodacross various optimization steps in Figure 10. As the process continues, the skipped layer set is gradually refined toward the optimal configuration.

Figure: (a) Optimization Step 0
Refer to caption: x14.png

### D.4 Case Study

We present examples from CNN/DM and Humaneval in Table 9 and 10, respectively, comparing token acceptance between \methodand prior plug-and-play approach, Lookahead (Fu et al., ). The results demonstrate that \methodaccepts a higher number of drafts, which are uniformly distributed in the outputs, highlighting the effectiveness of our approach.

**Table 9: CNN/DM example using LLaMA-2-13B; red tokens indicate accepted drafts.**
| User Input | Article: An Alberta student who’d accidentally left his headlights on all day was greeted by what may have been the world’s friendliest note from a stranger when he returned to his car. But Derek Murray, a University … Summary: |
| --- | --- |
| Lookahead | A Canadian law student was greeted by a friendly note from a stranger when he returned to his car after leaving the headlights on all day . |
| \hdashline\method | A Canadian law student was greeted by a friendly note from a stranger when he returned to his car after leaving the headlights on all day . |

**Table 10: Humaneval example using CodeLLaMA-13B; red tokens indicate accepted drafts.**
| User Input | from typing import List def has_close_elements(numbers: List[float], threshold: float) ->bool: ””” Check if in given list of numbers, are any two numbers closer to each other than given threshold. >>>has_close_elements … |
| --- | --- |
| Lookahead | for i in range(len(numbers) - 1): if abs(numbers[i] - numbers[i + 1]) <= threshold: return True return False def has_close_elements_2(numbers: List[float], threshold: float) ->bool: ””” Check if in given list of numbers, are any two numbers closer to each other than given threshold. >>>has_close_elements_2 |
| \hdashline\method | for i in range(len(numbers) - 1): if abs(numbers[i] - numbers[i + 1]) <= threshold: return True return False def has_close_elements_2(numbers: List[float], threshold: float) ->bool: ””” Check if in given list of numbers, are any two numbers closer to each other than given threshold. >>> has_close_elements_2 |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. How does SWIFT handle non-stationary input distributions during Bayesian optimization?

2. Could the authors provide insights into how SWIFT performs under extreme token count variations or highly domain-specific tasks?
soundness: 3
strengths: While the concept of layer-skipping is not really novel, its use of bayesian optimization can be a good idea for self-SD.
summary: By adaptively skipping intermediate layers during inference, SWIFT improves speedups of LLMs without compromising the quality of generation. The method integrates a Bayesian optimization-based layer selection mechanism to adapt to task-specific requirements dynamically.
weaknesses: 1. The reward design and its stability under distributional changes need more explanation. Open discussion with concurrent work, such as "A Unified Framework for Speculative Decoding with Multiple Drafters as a Bandit (Submitted at ICLR'25; https://openreview.net/forum?id=5haYLrlyGj)", could enhance understanding of these challenges. While the primary focus is different, the insight of using bandit approach is quite similar to this paper. And I recommend the authors to put the discussions for the assumption and extensions of Bayesian optimization for layer skipping inspired by this work.


2. More discussions on related work, such as Kim et al. (2024), Stern et al. (2018), and Gloeckle et al. (2024) on pretrained blockwise parallel language models, would position the contribution better within the existing literature. Because both papers are also a parallel line of work for self-speculative decoding, while they use the non-autoregressive heads instead. 

- Gloeckle et al. (2024),  Better & Faster Large Language Models via Multi-token Prediction.
- Stern et al. (2024), Blockwise Parallel Decoding for Deep Autoregressive Models
- Kim et al. (2024), Accelerating Blockwise Parallel Language Models with Draft Refinement. (https://openreview.net/forum?id=KT6F5Sw0eg)

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses section above. In addition, please consider the following questions:

1) Looking at the ablation studies in Appendix D (Table 7), it appears that *dynamic verification* does not bring much value as the loss in overall speed-up is minimal when one excluded dynamic verification (1.560x to 1.541x). Could authors comment on this?

2) Do the speedup numbers in Table 2 take into account the optimization phase? If yes, how many LLM generations are performed to obtain the results in Table 2?
soundness: 3
strengths: 1) The paper successfully demonstrates that the speculative decoding (SD) framework has the potential to speed up LLM inference even when one does not employ additional model parameters and task-specific training to support the drafting phase.

2) The paper makes two key observations about layer skipping during the drafting phase that highlights the need for adaptive (task-specific) selection of layers to skip during the drafting phase to maximize the benefit of layer skipping-based drafting approach. Subsequently, the paper proposes SWIFT - an effective SD approach that can identify a reasonable set of layers to skip for the underlying task with minimal training.

3) The paper further showcases the utility of leveraging the (draft) model's prediction confidence and top-k per-token predictions to improve the realized speed-up via SWIFT.

4) The paper is mostly well-written and conveys the key ideas in sufficient detail. The proposed ideas exhibit sufficient novelty over existing SD methods. The empirical results and in-depth empirical analysis highlight the gains realized by SWIFT over vanilla LLM inference.
summary: This paper aims to improve speculative decoding (SD) with a focus on eliminating the need for additional model parameters or extensive training to enable effective drafting in SD. In particular, the paper utilizes the same (target) model as the draft model by skipping a subset of model layers while generating draft tokens. Towards this, the paper proposes an SD method, namely SWIFT, that performs on-the-fly adaptive layer selection via an optimization phase to identify task-specific layers to skip. The optimization phase is followed by an inference acceleration phase that leverages the identified layers to perform skipping during drafting. During the inference acceleration phase, SWIFT additionally relies on 1) early stopping of the drafting process if (draft) model's confidence is not high enough; and 2) utilizing top-k predictions for each draft token position during parallel verification. The authors empirically validate the utility of SWIFT by showcasing 1.3-1.6x speed-up on CNN/DM, GSM8K, and TinyStories datasets.
weaknesses: 1) There is room for improvement in the discussion of related prior work. Given that Elhoushi et al. 2024 also leverage layer skipping during the drafting phase, a detailed discussion of this work is warranted. Furthermore, the authors may also want to cite https://openreview.net/pdf?id=yUmJ483OB0. 

2) The authors may want to make their empirical evaluation more comprehensive. Currently, the authors don't compare with existing approaches that rely on layer skipping during the drafting phase. Even though these existing methods might rely on extensive training, the authors should compare SWIFT with these methods. Such a comparison can highlight if there is any performance gap between these methods and their proposed plug-and-play approach.

3) The paper aims to eliminate the extensive training of existing layer skipping-based approaches via an efficient on-the-fly optimization phase. However, it's not clear if the existing methods can also perform well even when one limits the amount of offline training for these methods.

4) The authors repeatedly emphasize that their proposed method is a plug-and-play method. However, they don't seem to be evaluating their method in a dynamic setting where the underlying task (distribution) changes over time. In such a dynamic setting, would SWIFT have to interleave the optimization and acceleration phases? Would one still observe a good amount of speed up in such settings?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Table 2 presents results for Llama-2-13B, Llama-2-13B-Chat, and Llama-2-70B. Why are the results for Llama-2-70B-Chat and Llama-2-7B(-Chat) not included?
2. How does the overhead of the proposed layer searching algorithm compare to the overhead of training additional modules like Eagle?
soundness: 3
strengths: The method does not require training an additional model or module for drafting, making it applicable to most large language models.
summary: This paper proposes a plug-and-play self-speculative decoding method. The authors employ a layer-skipping approach to construct a draft model. Experimental results indicate that this method achieves a 1.3-1.6 times inference speedup on Llama-2 and Code-Llama models.
weaknesses: 1. The speedup is not as promising compared to other training-free methods like Lookahead. The authors should also present results for methods such as Mudusa and Eagle, which require minimal training overhead.
2. It is recommended that the authors test well-trained LLMs, such as Llama-3, as models with less effective performance might yield higher speedup ratios.
3. The method requires different settings for different tasks. However, in real-world LLM chat applications, it is often difficult to predict the corresponding tasks of user instructions. It is suggested that the authors evaluate the method's speedup performance on benchmarks like MT-Bench, which test the general capabilities of models.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.The author should compare their method with self-SD [1] in table 2, since their method is an improvement of the latter. 
2. The author only compared to the baseline on the Llama and CodeLlama models. I believe experiments should be conducted on larger models with different architectures to demonstrate the generalization of the method.
3.Moreover, compared with self-SD, the innovation is still insufficient, for example , the confidence-aware inference strategies are similar to some mechanism in [1],[2]
4.Despite SWIFIT does not require additional training, comparing with other method ,like EAGLE [2] ,Medusa [3], which can achieve over a 3.05-4.26x speedup, SWIFIT does’t show much value.As reported in [2], the draft models is trainable within 1-2 days for 70B models.

[1] Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. ACL 2024
[2] EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees, EMNLP 2024
[3] Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads, ICML 2024
soundness: 2
strengths: 1.The paper is well-written and flows very smoothly. 
2.The authors make effort to demonstrate the feasibility of their theory through experiments.
3.The method incorporates many of the latest techniques.
summary: This paper aims to accelerate the inference of LLMs. They introduce SWIFT, a self-speculative decoding algorithm that adaptively selects intermediate layer to skip without extra cost. They performed an empirical analysis of layer-skipping SD paradigm and show the potential of self-accelerate of LLMs through layer sparsity. They used some techniques like early-stop drafting to further speed up reasoning.
weaknesses: 1.The author should compare their method with self-SD [1] in table 2, since their method is an improvement of the latter. 
2. The author only compared to the baseline on the Llama and CodeLlama models. I believe experiments should be conducted on larger models with different architectures to demonstrate the generalization of the method.
3.Moreover, compared with self-SD, the innovation is still insufficient, for example , the confidence-aware inference strategies are similar to some mechanism in [1],[2]
4.Despite SWIFIT does not require additional training, comparing with other method ,like EAGLE [2] ,Medusa [3], which can achieve over a 3.05-4.26x speedup, SWIFIT does’t show much value.As reported in [2], the draft models is trainable within 1-2 days for 70B models.

[1] Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. ACL 2024
[2] EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees, EMNLP 2024
[3] Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads, ICML 2024

Meta-review: (a) Summary of Scientific Claims and Findings:

This paper introduces SWIFT, a plug-and-play self-speculative decoding (SD) method that accelerates large language model (LLM) inference by dynamically skipping intermediate layers during drafting. Layer-skipping is used as the compact draft model, avoiding additional training or parameters. A Bayesian optimization algorithm adaptively selects task-specific layers to skip during inference. SWIFT achieves speedups on various benchmarks without compromising text quality, which outperforms prior SD methods.

(b) Strengths of the Paper:

SWIFT does not require auxiliary models or additional training, enabling immediate deployment across various LLMs.

The authors introduce layer sparsity-based SD complements existing SD approaches and offers a practical alternative to training-heavy methods.

SWIFT is validated across multiple datasets and LLMs, with comparisons to existing methods like Self-SD and Lookahead.

(c) Weaknesses of the Paper and Missing Elements:

Limited evaluation against newer training-free methods, e.g., Lookahead, and methods with minimal training overhead, such as Mudusa and Eagle.

Needs more extensive testing in highly dynamic scenarios where task distribution changes rapidly.

Some reviewers noted limited speedup for well-optimized LLMs (e.g., Llama-3 series) compared to less efficient models.

(d) Decision and Rationale:

The paper has significant contributions, particularly its novel plug-and-play SD method. However, concerns about limited comparisons still remain. Positive reviewer feedback following the discussion phase suggests potential for acceptance.

Decision: Accept (Poster)