SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration
Authors: Heming Xia
Yongqi Li
Jun Zhang
Cunxiao Du
Wenjie Li
Venue: ICLR 2025 Poster (2025)
Keywords: Speculative Decoding
LLM Inference Acceleration
Efficient NLP
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Speculative decoding (SD) has emerged as a widely used paradigm to accelerate LLM inference without compromising quality. It works by first employing a compact model to draft multiple tokens efficiently and then using the target LLM to verify them in parallel. While this technique has achieved notable speedups, most existing approaches necessitate either additional parameters or extensive training to construct effective draft models, thereby restricting their applicability across different LLMs and tasks. To address this limitation, we explore a novel plug-and-play SD solution with layer-skipping, which skips intermediate layers of the target LLM as the compact draft model. Our analysis reveals that LLMs exhibit great potential for self-acceleration through layer sparsity and the task-specific nature of this sparsity. Building on these insights, we introduce SWIFT, an on-the-fly self-speculative decoding algorithm that adaptively selects intermediate layers of LLMs to skip during inference. SWIFT does not require auxiliary models or additional training, making it a plug-and-play solution for accelerating LLM inference across diverse input data streams. Our extensive experiments across a wide range of models and downstream tasks demonstrate that SWIFT can achieve over a $1.3\times$$\sim$$1.6\times$ speedup while preserving the original distribution of the generated text. We release our code in https://github.com/hemingkx/SWIFT.

Published as a conference paper at ICLR 2025
SWIFT: ON-THE-FLY SELF-SPECULATIVE DECOD- ING FOR LLM INFERENCE ACCELERATION
Heming Xia', Yongqi ; Zhang", Cunxiao Du', Wenjie Li' 'Department of Computing, The Hong Kong Polytechnic University "Colleee of Computer Science and Technology, Zhejiang University SSea Al Lab fhe-ming. xia laa.iiiaaaaaao.. @connect. polyu. hk; {zj. cs ez jww.z...u... ju.juueduu edd. cn
ABSTRACT
Speculative decoding (SD) has emerged as widely used paradigm to accelerate LLM inference without compromising quality. works by first employing a com- pact model to draft multiple tokens efficiently and then using the target LLM to verify them in parallel. While this technique has achieved notable speedups, most existing approaches necessitate either additional parameters or extensive training to construct effective draft models, thereby restricting their applicability across different LLMs and tasks. To address this limitation, explore a novel plug- and-play SD solution with layer-skipping, which skips intermediate layers of the target LLM the compact draft model. Our analysis reveals that LLMs exhibit great potential for self-acceleration through layer sparsity and the task-specific na- ture of this sparsity. Building on these insights, we introduce SWIFT, an on-the-fly self- eeff-speculative decoding algorithm that adaptively selects intermediate layers of LLMs to skip during inference. SWIFT does not require auxiliary models or ad- ditional training, making it ence across diverse input data streams. Our extensive experiments across a wide range of models and downstream tasks demonstrate that SWIFT can achieve over a 1.3xx~1.6x speedup while preserving the original distribution of the generated text. We release code in https:/ htt//ggtttt githhb.. com/ hwwp/hemingk//S// SWIFT.
solution for accelerating LLM infer-
INTRODUCTION
Large Language Models (LLMs) have exhibited outstanding capabilities in handling various down- stream tasks (OpenAI, 2023; Touvron et al., 2023a;b; Dubey et al., 2024). However, their token-by- token generation necessitated by autoregressive decoding poses efficiency challenges, particularly as model sizes increase. To address this, speculative decoding (SD) has been proposed as promising solution for lossless LLM inference acceleration (Xia et al., 2023; Leviathan al., 2023; Chen et al., 2023). At each decoding step, SD first employs compact draft model to efficiently predict multiple tokens as speculations for future decoding steps of the target LLM. These tokens are then validated by the target L.M in parallel, ensuring that the original output distribution remains unchanged.
Recent advancements in SD have pushed the boundaries of the latency-accuracy trade-off by ex- ploring various strategies (Xia et al., 2024), including incorporating lightweight draft modules into LLMs (Cai et al., 2024; Ankner et al., 2024; Li et al., 2024a;b), employing fine-tuning strategies to facilitate efficient LLM drafting (Kou et al., 2024; Yi al., 2024; Elhoushi et al., 2024), and aligning draft models with the target LLM (Liu et al., 2023a; Zhou et al., 2024; Miao et al., 2024). Despite their promising efficacy, these approaches require additional modules or extensive training, which limits their broad applicability across different model types and causes significant inconvenience in practice. To tackle this issue, another line of research has proposed the Jacobi-based drafting (San- tilli et al., 2023; Fu et al., 2024) to facilitate plug-and-ple -ad--pay SD. As illustrated in Figure (a)). these methods append pseudo tokens to the input prompt, enabling the target LLM to generate multiple tokens as drafts in single decoding step. However, the Jacobi-decoding paradigm misaligns with the autoregressive pretraining objective of LLMs, resulting in suboptimal acceleration effects.
"orresponding Author

---

Published as a conference paper at ICLR 2025
In this work, we introduce a novel research direc- tion for plug-and-play SD: sparsity-asse drafting, which leverages the inherent sparsity in LLMs to enable efficient drafting (see Figure (bb).. Specifi- cally, we exploit a straightforward yet practical form of LLM sparsity layer sparsity - to accelerate in- ference. Our approach is based on two key obser- vations: 1) LLMs possess great potential for self- acceleration through layer sparsity. Contrary to the conventional belief that layer selection must be carefully optimized (Zhang al., 2024), we surpris- ingly found that uniformly skipping layers to draft can still achieve a notable 1.2x speedup, provid- ing strong foundation for plug-and-play SD. 2) Layer sparsity is task-specific. We observed that each task requires its optimal set of skipped lay- ers, and applying the same layer configuration across different tasks would cause substantial performance degradation. For example, the speedup drops from 1.47x to 1.01x x when transferring the configuration optimized for storytelling task to reasoning task.
Refine
Full-parameter LLM
Sparse LLM
(a) Jacobi-based Drafting
(b) Sparsity- Drafting
Figure 1: Illustration of prior solution and
ours for alug-and-play SD. (a) Jacobi-based drafting appends multiple pseudo tokens to the input prompt, enabling the target LLM to generate multiple tokens as drafts in sin- gle step. (b) SWIFT adopts sparsity-based drafting, which exploits the inherent sparsity in LLMs to facilitate efficient drafting. This work is the first exploration of plug-and-play SD using sparsity-based drafting.
Building on these observations, we introduce SWIFT, the first on-the-fly self-speculative decoding algorithm that adaptively optimizes the set of skipped layers in the target LLM during inference, facilitating the lossless acceleration of LLMs across diverse input data streams. SWIFT integrates two key innovations: (1) a context-based layer set optimization mechanism that leverages LLM- generated context to efficiently identify the optimal set of skipped layers corresponding to the current input stream, and (2) confidence-awee
inference acceleration strategy that maximizes the use of draft tokens, improving both speculation accuracy and verification efficiency. These innovations allow SWIFT to strike expected balance between the latency-accuracy trade-off in SDD, providing a new plug-and- and-play solution for lossless LLM inference acceleration without the need for auxiliary models or additional training, demonstrated in Table 1.
We conduct experiments using LLaMA-2 and CodeLLaMA models across multiple tasks, including summarization, code generation, mathematical reasoning, etc. SWIFT achieves 1.3x~1.66x wall- clock time speedup compared to conventional autoregressive decoding. Notably, in the greedy set- ting. SWIFT consistently maintains a 18%%~00%% token acceptance rate across the LLaMA2 series, indicating the high alignment potential of this paradigm. Further analysis validated the effectiveness of SWIFT across diverse data streams and its compatibility with various LLM backhones.
Our key contributions are:
(i) We performed an empirical analysis of LLM acceleration on layer sparsity, revealing both the potential for LLM self-acceleration via layer sparsity and task-specific nature, un- derscoring the necessity for adaptive self-speculative decoding during inference.
(ii) Building on these insights, we introduce SWIFT, the first plug-and-play self-speculative decoding algorithm that optimizes the set of skipped layers in the target LLM on the fly, enabling lossless acceleration of LLM inference across diverse input data streams.
(iii) We conducted extensive experiments across various models and tasks, demonstrating that SWIFT consistently achieves a 1.3x~1.6xx speedup without any auxiliary model or train- ing, while theoretically guaranteeing the preservation of the generated text's distribution.
2 RELATED WORK
Speculative Decoding (SD) Due to the sequential nature of autoregressive decoding, LLM in- ference is constrained by memory- bound computations (Patterson, 2004; Shazeer, 2019), with the primary latency bottleneck arising not from arithmetic computations but from memory reads/writes of L.LM parameters (Pope al., 2023). To mitigate this issue, speculative decoding (SD) introduces
2

---

Published as a conference paper at ICLR 2025
Drafting
Verification
Methods
Speedup
Approach
AM Plug&Play
Greedy
Sampling
Token Tree
EAGLE (Li et al., 2024a;b)
Draft Heads
Yes
REST (He et al., 2024)
Context Retrieval
Yes
SELF-SD (Zhang et al., 2024)
Layer Skipping
No
PARALLEL (Santilli ct al., 2023)
Jacobi Decoding
No
0.0x
LOOKAHEAD (Fu et al., 2024)
Jacobi Decoding
No
1.2x~144x
SWIFT (Ours)
Layer Skipping
No
1.3xx~1.6x
Table I: Comparison of SWIFT with existing SD methods. "AM" denotes whether the method requires auxiliary modules such as additional parameters or data stores. "Greedy", "Sampling", and "Token Tree" denote whether the method supports greedy decoding, multinomial sampling, and token tree verification, respectively. SWIFT is the first alug-and-play layer-skipping SD method, which is orthogonal to those Jacobi-based methods such as Lookahead (Fu et al., 2024).
utilizing a compact draft model to predict multiple decoding steps, with the target LLM then validat- ing them in parallel (Xia et al., 2023; Leviathan et al., 2023; Chen al., 2023). Recent SD variants have sought to enhance efficiency by incorporating additional modules (Kim et al., 2023; Sun et al., 2023: Du et al.. 2024; Li et al., 2024a:b) or introducing new training objectives (Liu et al., 2023a; Kou et al., 2024; Zhou et al., 2024; Gloeckle et al., 2024). However, these approaches necessitate extra parameters or extensive training. limiting their applicability across different models. Another line of research has explored plug-and-play SD methods with Jacobi decoding (Santilli et al., 2023; Fu et al., 2024), which predict multiple steps in parallel by appending pseudo tokens to the input and refining them iteratively. As shown in Table 1, our work complements these efforts by investigating a novel plug-and-play SD method with layer-skipping, which exploits the inherent sparsity of LLM layers to accelerate inference. The most related approaches to include Self SD (Zhang al., 2024) and LayerSkip (Elhoushi et al., 2024), which also skip intermediate layers of LLMs to form the draft model. However, both methods require time-consuming offline training process, making them neither plug-and-play nor easily generalizable across different models and tasks.
Efficient LLMs Utilizing Sparsity LLMs are powerful but often over parameterized (Hu et al., 2022). To address this issue, various methods have been proposed to accelerate inference by lever- aging different forms of LLM sparsity. One promising research direction is model compression, which includes approaches such as quantization (Dettmers al., 2022; Frantar et al., 2023; Ma et al., 2024), parameter pruning (Liu et al., 2019; Hoefler et al., 2021; Liu et al., 2023b), and knowl- edge distillation (Touvron et al., 2021; Hsich et al., 2023; Gu et al., 2024). These approaches aim to reduce model sparsity by compressing LLMs into more compact forms, thereby decreasing mem- ory usage and computational overhead during inference. Our proposed method, SWIFT, focuses specifically spansity within LLM layer computations, providing more streamlined approach to efficient L.M inference that builds upon recent advances in layer skipping (Corro et al., 2023; Zhu al., 2024; Jaiswal et al., 2024; Liu al., 2024). Unlike these existing layer-skipping methods that may lead to information loss and performance degradation, SWIFT investigates the utilization of layer sparsity to enable lossless acceleration of LLM inference.
3 PRELIMINARIES
3.1 SELF-SPECULATIVE DECODING
Unlike most SD methods that require additional parameters, self-speculative (Self-SD) first proposed utilizing party of an LLM as a compact draft model (Zhang al., 2024). In each decoding step, this approach skips intermediate layers of the LLM to efficiently generate draft tokens; these tokens are then validated in parallel by the full-parameter LLM to ensure that the output distribution of the target LLM remains unchanged. The primary challenge of Self-SD lies in determining which layers, and how many, should be skipped - referred to as the skipped layer set during the drafting optimization problem. Formally, given the input data X' and the target LLM My with L layers (including both attention and MLP layers), Self-SD aims to identify the optimal skipped layer set that minimizes the average inference time per token:
stage, which is formulated
arg min ExEx I (x z; 0MT)
s.t. z E (0,
EBEX [
3

---

Published as a conference paper at ICLR 2025
where f(.) is black-box function that returns the inference latency of sample æ, 2, E {0. denotes whether layer of the target LLM is skipped when drafting, and |x] represents the sample length. Self-SD addresses this problem through Bayesian optimization process (Jones et al., 1998). Before inference, this process iteratively selects new inputs based on a Gaussian process (Rasmussen & Williams, 2006) and evaluates Eq (1) on the training of X. After specified numher of iterations, the best & is considered an approximation of x and is held fixed for inference.
While Self-SD has proven effective, its reliance on a time-intensive Bayesian optimization process poses certain limitations. For each task, Self-SD must sequentially evaluate all selected training samples during every iteration optimize Eq (1); Moreover, the computational burden Bayesian optimization escalates substantially with the number of iterations. As result, processing just eight CNN/Daily Mail (Nallapati et al., 2016) samples for 1000 Bayesian iterations requires nearly 7.5 hours for LLaMA-2-133 --3B-3 and 20 hours for LLaMA-2-708B on an NVIDIA A6000 server. These computational demands restrict the generalizability of Self-SD across different models and tasks.
3.2 EXPERIMENTAL OBSERVATIONS
This subsection delves into Self-SD, exploring the plug-and-pla aand-aaddpay potential of this layer-skipping SI paradigm for lossless LLM inference acceleration. Our key findings are detailed below.
(a] Speedups with a Unified Skipping Pattorn
(b) Spoodup Variations under Domain Shift
1.5
Stoy:
Acceptance
1.4
Rea.
Tram. LS
Speedup
Speedup
1.3
1:28
0.6
1.2
1.15
0.4
1
0.2
Top canditates
candilates
1.1
1.0
1.11
1.04
1.25
08
1.06
Summarization
Reasoning
StoryTeilng
Translation
Number of Sub-layers to Skip
Evaluation Tasks
Figure 2: (a) LLMs possess seff-acceleration potential via layer sparsity. By utilizing drafts from the top-k candidates, found that uniformly skipping half of the layers during drafting yields notable 1.2x speedup. (b) Layer sparsity is task-specific. Each task requires its own optimal set of skipped layers, and applying the skipped layer configuration from one task to another can lead to substantial performance degradation. "X LS" represents the skipped layer set optimized for task X.
3.2. LLMs POSSESS SELF-ACCELERATUS POTENTIAL VIA LAYER SPARSITY
We begin by investigating the potential behavior alignment between the target LLM and its layer- skipping variant. Unlike previous work (Zhang et al., 2024) that focused solely greedy draft pre- dictions, we leverage potential draft candidates from top-k predictions, as detailed Section 4.2. We conducted experiments using LLaMA-2- 13B across the CNN/Daily Mail (Nallapati et al., 2016), GSMSK (Cobbe et al., 2021), and TtoyStories (Eldan Li, 2023) datasets. We applied uniform layer-skipping pattern with tt set 10. The experimental results, illustrated in Figure 2(a), demon- strate a 30% average improvement in the token acceptance rate by leveraging top-k predictions, with over 90% of draft tokens accepted by the target LLM. Consequently, compared to Self-SD, which achieved a maximum speedup of 1.01 x in this experimental setting, we revealed that the layer-skipping SD paradigm could yield an average wall-clock speedup of 1.22x over conventional autoregressive decoding with uniform layer-skipping pattern. This finding challenges the prevail- ing belief that the selection of skipped layers must be meticulously curated, suggesting instead that LLMMs possess greater potential for ceff-acceleration through inherent layer sparsity.
3.2.2 LAYER SPARSITY IS TASK-SPECIFIC
We further explore the following research question: Is the skipped layer set optimized for one spe- cific task applicable to other tasks? To address this, we conducted domain shift experiments using -LaMA-22133B the CNN/Daily Mail, GSM8K, TinyStories, and WMT16 DE-EN datasets. The experimental results, depicted in Figure 2(b), reveal two critical findings: 1) Each task requires its optimal skipped layer set. As illustrated in Figure 2(b), the highest speedup performance is
4

---

Published as a conference paper at ICLR 2025
consistently achieved by the skipped layer configuration specifically optimized for each task. The detailed configuration of these layers is presented in Appendix A, demonstrating that the optimal configurations differ across tasks. 2) Applying the static skipped layer configuration across dif- ferent tasks can lead to substantial efficiency degradation. For example, the speedup decreases from 1.47x 1.01 when the optimized skipped layer set from storytelling task is applied to a mathematical reasoning task, indicating that the optimized skipped layer set for one specific task does not generalize effectively others.
These findings lay the groundwork for our plug-and-play solution within layer-skipping SD. Sec- tion 3.2.1 provides strong foundation for real-time skipped layer selection, suggesting that addi- tional optimization using training data may be unnecessary; Section 3.2.2 highlights the limitations of static layer-skipping patterns for dynamic input data streams across various tasks, underscoring the necessity for adaptive layer optimization during inference. Building on these insights, we present our on-the-fly self-speculative decoding method for efficient and adaptive layer set optimization.
SWIFT: ON-THE-FLY SELF-SPECULATIVE DECODING
We introduce SWIFT, the first plug-and-play self-speculative decoding approach that optimizes the skipped layer set of the target LLM on the fly, facilitating lossless LLM acceleration across diverse input data streams. As shown in Figure 3, SWIFT divides LLM inference into two distinct phases: (1) context-buse based layer set optimization ($4.1), which aims to identify the optimal skipped layer set given the input stream, and (2) confidence-aware inference acceleration ($4.2), which employs the determined configuration to accelerate LLM inference.
Context- based Layer Set Optimization
Confidence- Inference Acceleration
Generated Takens
2N
mN
(m-1)N
(m+
context accumulation
loyer optimization
acceleration
Figure 3: Timeline of SWIFT inference. N denotes the maximum generation length per instance.
4.1 CONTEXT-BASED LAYER SET OPTIMIZATION
Layer set optimization is a critical challenge in self-speculative decoding, as determines which layers of the target LLM should be skipped to form the draft model (see Section 3.1). Unlike prior methods that rely on time-intensive offline optimization, our work emphasizes on-the-fly layer set optimization, which poses greater challenge to the latency-accuracy trade-off: the optimization must be efficient enough to avoid delays during inference while ensuring accurate drafting of subse- quent decoding steps. To address this, we propose an adaptive optimization mechanism that balances efficiency with drafting accuracy. Our method minimizes overhead by performing only single for- ward pass of the draft model per step validate potential skipped layer set candidates. The core innovation is the use of LLM-ggnnrated tokens (i.e., prior context) ground truth, allowing for simultaneous validation of the draft model's accuracy in predicting future decoding steps.
In the following subsections, we illustrate the detailed process of this optimization phase for each input instance, which includes context accumulation 1.4 and layer set optimization 1.2..
4.1.1 CONTEXT ACCUMULATION
Given input instance in the optimization phase, the draft model is initialized by uniformly skip- ping layers in the target LLM. This initial layer-skipping pattern is maintained to accelerate infer- ence until specified number of LLM -eM-eeeettted tokens, referred as the context window, has been accumulated. Upon reaching this window length, the inference transitions to layer set optimization.
4.1.2 LAYER SET OPTIMIZATION
During this stage, as illustrated in Figure 4, we integrate an optimization step before each LLM decoding step refine the skipped layer set, which comprises substeps:
5

---

Published as a conference paper at ICLR 2025
(1) Efficient Layer Set Suggestion
(b) Parallel Candidate Evaluation
Guassian Update
Random Search
MLP
Original Outputs
pp zandon.
Attention
choice
Calculate
LLM Inputs
MLP
Parallel
bchneea
Attention
Draft
Bayes Optimization
MLP
Draft Tokens
if
up
Attention
Alter Skipped Layer Set
Accepted
Q
Target LLM Verification
Draft multiple
Compact Draft Model
tokens
steps
imput
LIM--geerraed tokens
tokens
accepted tokenss
Figure 4: Layer set optimization process SWIFT. During the optimization stage, SWIFT performs an optimization step prior to each LLM decoding step to adjust the skipped layer set, which involves: (a) Efficient layer set optimization. SWIFT integrates random search with interval Bayesian op- timization to propose layer set candidates; (b) Parallel candidate evaluation. SWIFT uses LLM- generated tokens (i.e., prior context) as ground truth, enabling simultaneous validation of the pro- posed candidates. The berf-perg layer set is selected accelerate the current decoding step.
Efficient Layer Set Suggestion This substep aims to suggest potential layer set candidate. For- mally, given a target LLM Mr with L layers, our goal is to identify an optimal skipped layer set {0, 1}2 form the compact draft model. Unlike Zhang al. (2024), which relies entirely on a time-consuming Bayesian optimization process, we introduce an efficient strategy that combines random search with Bayesian optimization. In this approach, random sampling efficiently handles most of the exploration. Specifically, given a fixed skipping ratio r, SWIFT applies Bayesian op- timization at regular intervals of 3 optimization steps (e.g., B 25) to suggest the next layer set candidate, while random search is employed during other optimization steps.
Bayesian Optimization(() if o % B = Random (tarch)Scarch(I)
(2)
otherwise
where 1 < 0 < S is the current optimization step; S denotes the maximum number of optimization L steps; 1 = ( denotes the input space, i.e., all possible combinations of layers that can be skipped.
Parallel Candidate Evaluation SWIFT leverages LLM--gnerated context to simultaneously val- idate the candidate draft model's 's performance in predicting future decoding steps. Formally, given an input sequence and the previously generated tokens within the context window, denoted as = (vi,- ... ). ; the draft model M.D. which skips the designated layers of the target LLM,, is employed to predict these context tokens in parallel:
1/4 arg max log P (y x, M.r 0MD), iiis.
where y represents the context window. The cached key-value pairs in the target LLM My are reused by MD, presumably aligning M.'s distribution with Mc and reducing the redundant computation. The mat chness score is defined the exact match ratio between y and y':
matchness Σ. T = 3/4 1sis
where I(() denotes the indicator function. This score serves as the optimization objective during optimization, reflecting MD's accuracy predicting future decoding steps. shown Figure 4, the matchness score at each step is integrated into the Gaussian process model to guide Bayesian optimization, with the highest-scoring layer set candidate being retained to form the draft model.
As illustrated in Figure 3, the process of context accumulation and layer set optimization alternates for each instance until termination condition met either the maximum number of optimiza- tion steps is reached or the best candidate remains unchanged over multiple iterations. Once the optimization phase concludes, the inference process transitions to the confidence-aware inference acceleration phase, where the optimized draft model is employed to speed up LLM inference.
4.2 CONFIDENCE-AWARY INFERENCE ACCELERATION

---

Published as a conference paper at ICLR 2025
During the acceleration phase, the optimization step is removed. SWIFT applies the best- pess-perfommed layer set to form the compact draft model and decodes fol- lowing the traft-thhn--erify paradigm. Specifically, at each decoding step, given the input and previous LLM outputs y, the draft model MD predicts future LLM decoding steps in an autoregressive manner:
Continue
Farly Stop!
0.86 Sr
Pun 0.86 <
A
will
the best
that
Mp
Mp
Attention
V/5 arg max log P (y x. y. U. OM.))
(5)
(a) Early-stopping Drafting (b) Dynamic Verification
where 1 i S ND is the current draft step, ND de- notes the maximum draft length, y'e represents pre- vious draft tokens, and P--) denotes the probability distribution of the next draft token. The KV cache of the target LLM Mr and preceding draft tokens y is reused to reduce the computational cost.
Figure 5: Confidence-aware inference pro- cess of SWIFT. (a) The drafting terminates early if the confidence score drops below threshold (b) Draft candidates are dynam- ically selected based on confidence and then verified in parallel by the target LLM.
Let Pj = max P(.) denote the probability of the top-1 draft prediction v/.- which can be regarded as a confidence score. Recent research (Li et al., 2024b; Du et al., 2024) shows that this score is highly correlated with the likelihood that the draft token if will pass verification - higher confidence scores indicate a greater chance of acceptance. Therefore, following previous studies (Zhang al., 2024; Du et al., 2024), we leverage the confidence score to prune unnecessary draft steps and select valuable draft candidates, improving both speculation accuracy and verification efficiency.
As shown in Figure 5, we integrate SWIFT with two confidence-awan inference strategies': 1) Early-stopping Drafting. The autoregressive drafting process halts the confidence p; falls below a specified threshold c, avoiding any waste of subsequant drafting computation. 2) Dynamic Ver- ification. Each 3/ is dynamically extended with its top-k draft predictions for parallel verification to enhance speculation accuracy, with & determined by the confidence score p5. Concretely, & is set to 10, 5, 3, and for p in the ranges of (0, 0.5), (0.5, 0.8], (0.8, 0.95]. and (0.95, 1]. respectively. All draft candidates are linearized into single sequence and verified in parallel by the target LLM using a special causal attention mask (see Figure 5 (b)).
5 EXPERIMENTS
5.1
EXPERIMENTAL SETUP
We mainly evaluate SWIFT on LLaMA-2 (Touvron et al., 2023b) and Codel_L.MAA series (Rozière et al., 2023) across various tasks, including summarization, mathemat- ical reasoning, storytelling, and code generation. The evaluation datasets include CNN/Daily Mail (CNN/DM) (Nallapati et al., 2016), GSM8K (Cobbe et al., 2021). TinyStories (Eldan & Li, 2023), and HumanEval (Chen et al., 2021). The maximum generation lengths on CNN/DM, GSM8K, and TinyStories are set to 64, 64, and 128, respectively. We conduct 1-shot evaluation for CNN/DM and TinyStories, and 5-shot evaluation for GSM8K. We compare pass@ and pass @ 10 for HvmanEval. We randomly sample 1000 instances from the test set for each dataset except HvaanEval. The maximum generation lengths for HumanEval and all analyses are set to 512. During optimization, employ both random search and Bayesian optimization' to suggest skipped layer set candidates. Following prior work, we adopt speculative sampling (Leviathan et al., 2023) as our acceptance strategy with batch size of 1. Detailed setups are provided in Appendix B.1 and B.2.
Implementation Details
Baselines In our main experiments, we compare SWIFT to two existing plug-and-play methods: Parallel Decoding (Santilli et al., 2023) and Lookahead Decoding (Fu et al., 2024), both of which employ Jacobi decoding for efficient LLM drafting. It is important to note that SWIFT, as a layer- skipping SD method, orthogonal to these Jacobi-based SD methods, and integrating SWIFT with them could further boost inference efficiency. We exclude other SD methods from our comparison as they necessitate additional modules or extensive training, which limits their generalizability.
'These confidence-aware inference strategies are also applied during the optimization phase, where the current optimal layer set is used to form the draft model and accelerate the corresponding L.LM decoding step. Pttpps: / i thub. , com homm/baassii. an- optiimiim zati Hayea anOpt i mi zati on
7

---

Published as a conference paper at ICLR 2025
CNN/DM
GSM8K
TinyStories
Speed (tokens/s)
Overall Speedup
Models
Methods
M
Speedup
M
Speedup
M
Speedup
VANILLA
1.00
1.00x
1.00
1.00x
1.00
1.00
20.10
1.00x
PARALLEL
1.04
0.95
1.11
0.99x
1.06
0.97
19.49
0.97
LLaMA-2-13B
LOOKAHEAD
1.38
1.16x
1.50
1.29x
1.62
1.37x
25.46
1.27x
SWIFT
4.34
1.37
3.13
1.31 x
8.21
1.53x
28.26
1,41 x
VANILLA
1.00
1.00
1.00
1.00
1.00
1.00x
19.96
1.00x
LLaMA-2-13B -Chat
PARALLEL
1.06
0.96x
1.08
0.97
1.10
0.98
19.26
0.97
LOOKAHEAD
1.35
1.15x
1.57
x
1.66
1,40x
25.69
1.29
SWIFT
3.54
1.28x
2.95
1.25x
7.42
1.50
26.80
1.34x
VANILLA
1.00
1.00x
1.00
1.00x
1.00x
4.32
1.00x
PARALLEL
1.05
0,95
1.07
0.97
1.05
0.96x
4.14
0.96x
LLaMA-2-70B
LOOKAHEAD
1.36
1.15x
1.54
1.30
1.59
1.35x
5.45
1.26x
SWIFT 3.85
1.43x
2.99
1.39x1
6.17
1.62
6.41
1.48
Table 2: Comparison between SWIFT and prior plug-and-play methods. We report the mean gener- ated length M, speedup ratio, and average decoding speed (tokens/s) under greedy decoding. indi- cates results with a token acceptance rate a above 0.98. More details are provided in Appendix C.I.
CodeILLMMA-13B
CodeLLaMA-341
Datasets
Methods
M
a
Acc.
Speedup
M
a
Acc.
Speedup
VANILLA
1.00
0.311
1.00x
1.00
0.372
1.00
HumanEval (pass@1)
SWIFT
4.75
0.98
0.311
1.40x
3.79
0.88
0.372
1.46x
VANILLA
1.00
0.628
1.00x
1.00
0.677
1.00x
HumanEval (pass 10) SWIFT
3.55
0.93
0.628
1.29
2.79
0.90
0.683
1.30x
Table 3: Experimental results of SWIFT code generation tasks. We report the mean generated length M, acceptance rate a, accuracy (Acc.), and speedup ratio for comparison. We use greedy decoding for pass@1 I and random sampling with temperature of 0.6 for pass 10.
Evaluation Metrics We report two widely-used metrics for SWIFT evaluation: mean generated length M (Stern al., 2018) and token acceptance rate a (Leviathan al., 2023). Detailed descrip- tions of these metrics can be found in Appendix B.3. In addition to these metrics, we report the actual decoding speed (tokens/s) and wall-time speedup ratio compared with vanilla autoregressive decoding. The acceleration of SWIFT theoretically guarantees the preservation of the target LLMs' output distribution, making unnecessary to evaluate the generation quality. However, to provide point of reference, we present the evaluation scores for code generation tasks.
5.2
MAIN RESULTS
Table 2 presents the comparison between SWIFT and previous aldg-and-play methods on text gener- ation tasks. The experimental results demonstrate the following findings: (1) SWIFT shows superior efficiency over prior methods, achieving consistent speedups of 1.3x~1166x over vanilla autore- gressive decoding across various models and tasks. (2) The efficiency SWIFT is driven by the high behavior consistency between the target LLM and its layer-skipping draft variant. As shown in Table 2, SWIFT produces a mean generated length M of 5.01, with a high token acceptance rate a ranging from 90% to 100%. Notably, for the LLaMA-2 series, this acceptance rate remains stable at 98%%,00%, indicating that nearly all draft tokens are accepted by the target LLM. (3) Compared with 13B models, LLaMA-2-700 achieves higher speedups with a larger layer skip ra- tio (0.45->0.5) suggesting that larger- scale L.MSs exhibit greater layer sparsity. This underscores SWIFT'S potential to deliver even greater speedups as LLM scales continue to grow. A detailed anal- ysis this finding is presented Section 5.3, while additional experimental results for LLaMA- models, including -LaMA-3-7705
presented in Appendix C.2.
Table 3 shows the evaluation results of SWIFT on code generation tasks. SWIFT achieves speedups of 1.3xx11.5x vanilla autoregressive decoding, demonstrating its effectiveness across both greedy decoding and random sampling settings. Additionally, speculative sampling theoretically guarantees that S WIFT maintains the original output distribution of the target L.L.M This is empirically validated
8

---

Published as a conference paper at ICLR 2025
by the task performance metrics in Table 3. Despite slight variation in the pass @ 10 metric for CodeLLaMA-34B. SWIFT achieves identical performance to autoregressive decoding.
5.3
IN-DEPTII ANALYSIS
1.6
1.0
Average
Medules
Latency (mns)
Ratio (%)
1.5
Speedup
Opeimine
0.24 0.02
6,8
0.6
1.4
Draft
19,33 1.36
64.4
0.4
Vendy
X.50
28.4
0.2
Optimization Stop!
Instance Specdup
Overall Speodup
1.3
Others
Toca
30.05 1 2.84
1.98 0.13
100.0
6.4
0.0
1.2
10
20
30
40
50
60
70
80
90
100
of Instances
Latency Breakdown per Token
Figure 6: Illustration and latency breakdown of SWIFT inference. As the left figure shows, after the context-based layer set optimization phase, the overall speedup of SWIFT steadily increases, reaching the average instance speedup during the acceleration phase. The additional optimization steps account for only 0.8% of the total inference latency, as illustrated in the right figure.
Illustration of Inference As described in Section 4, SWIFT divides the LLM inference process into two distinct phases: optimization and acceleration. Figure 6 (left) illustrates the detailed accel- eration effect of SWIFT during LLM inference. Specifically, the optimization phase begins at the start of inference, where an optimization step is performed before each decoding step to adjust the skipped layer set forming the draft model. As shown in Figure 6, in this phase, the mat chness score of the draft model rises sharply from 0.45 to 0.73 during the inference of the first instance. This score then gradually increases to 0.98, which triggers the termination of the optimization process. Subsequently, the inference transitions to the acceleration phase, during which the optimization step is removed, and the draft model remains fixed accelerate LLM inference. As illustrated, the instance speedup increases with the mat chtess score, reaching an average of 1.53x in the acceleration phase. The overall speedup gradually rises as more tokens are generated, eventually approaching the average instance speedup. This dynamic reflects key feature of SWIFT: the effi- ciency of SWIFT improves with increasing input length and the number instances.
Breakdown of Computation Figure 6 (right) presents the computation breakdown of different modules SWIFT with 1000 CNN/DM samples using LLaMA-2-13B, The results demonstrate that the optimization step only takes 0.8% of the overall inference process, indicating the efficiency of our strategy. Compared with Self-SD (Zhang et al., 2024) that requires a time-consuming opti- mization process (e.g.. 7.5 hours for LLaMA-2-133 13B CNN/DM), SWIFT achieves nearly 180x optimization time reduction, facilitating on-the the--hhe inference acceleration. Besides, the results show that the drafting stage of SWIFT consumes the majority of inference latency. This is consistent with our results of mean generated length in Table and 3, which shows that nearly 80% output tokens generated by the efficient draft model, demonstrating the effectiveness of our S SWIFT framework.
Vanilla
Self-D
SWIFT
SWIFT
Self-SD
1.6
1.0
Token Acceptanc eeeeeeencee
1.47k
Speedup
1.4
1.27x
0.8
1.2
1.d6x
1
1.02x
Aie
On
0.6
0.8
Summarization
Reasoning
Instruction
Translation
Figure 7: Comparison between SWIFT and Self-SD handling dynamic data input streams. Unlike Self-SD, which suffers from efficiency reduction during distribution shift, SWIFT maintains stable acceleration performance with an acceptance rate exceeding 0.9.

---

Published as a conference paper at ICLR 2025
Dynamic Input Data Streams We further validate the effectiveness of SWIFT in handling dy- namic imput data streams. We selected CNN/DM, GSM8K, Alpaca (Taori et al., 2023), WMT14 DE-EN, and Nature Questions (Kwiatkowski et al., 2019) for the evaluation summarization, rea- soning, instruction following, translation, and question answering tasks, respectively. For each task, randomly sample 500 instances from the test and concatenate them task-by-task form the input stream. The experimental results are presented in Figure 7. As demonstrated, Self-SD sensitive to domain shifts, with the average token acceptance rate dropping from 92% 68%. Consequently, it suffers from severe speedup reduction from 1.33 to an average of 1.05x under domain shifts. In contrast, SWIFT exhibits promising adaptation capability different domains with an average token acceptance rate of 96%, leading to a consistent 1.3x~1.6xx speedup.
(a) Flexible Optimization Strategy
(b) Scaling Law SWIFT
1.50
1.6
Speedup
1.45
Speedup
1.5
1.4
5 1-0000,8 8-25
5-500, 8-25
1.3
7B
708
1.30
5-1000,
8-50
138
1.25
5
10
15 of Instances
20
50
1.2
0.30
0.35
0.40
0.45
0.50
0.55
0.60
Layer Skip Ratio
Figure 8: In-depth analysis of SWIFT, which includes: (a) Flexible optimization strategy. The maximum optimization iteration S and Bayesian interval 3 can be flexibly adjusted to accommodate different input data types. (b) Scaling law. The speedup and optimal layer skip ratio of SWIFT increase with larger model sizes, indicating that larger LLMs exhibit greater layer sparsity.
Flexible Optimization & Scaling Law Figure 8(a) presents the flexibility of SWIFT in handling various input types by adjusting the maximum optimization step S and Bayesian interval B. For input with fewer instances, reducing S enables earlier transition the acceleration phase while increasing 3 reduces the overhead during the optimization phase, enhancing speedups during the initial stages of inference. In cases with sufficient input data, SWIFT enables exploring more op- timization paths, thereby enhancing the overall speedup. Figure 8(b) illustrates the scaling law of SWIFT: the model size increases, both the optimal layer-skip ratio and overall speedup improve, indicating that larger L.MMs exhibit more layer sparsity. This finding highlights the potential of SWIFT for accelerating LLMs of larger sizes (e.g., 175B), which we leave for future investigation.
Other LLM Backbones Beyond LLaMA, we as- sess the effectiveness SWIFT on additional LLM backbones. Specifically, include Yi-34B (Young et al., 2024) and ScepSeck-Coder-33 2024) along with their instruction-tuun tundd variants for text and code generation tasks, respectively. The speedup results of SWIFT are illustrated in Figure 9, demonstrating that SWIFT achieves efficiency im- provements ranging from 26% to 54% on these LLM backbones. Further experimental details are pro- vided Appendix C.3.
Vanilla
Base
Instruct
1.6
1.54
1.5
(Guo et al.,
1.4
1.39
1.31
1.2
1.1
1.0
1.00
1.00
Yi-34B
DeepSeek.Coden-30
Figure 9: Speedups of SWIFT LLM back- bones and their instruction-tume variants.
CONCLUSION
In this work, we introduce SWIFT, an on-the-fly self-speculative decoding algorithm that adaptively selects certain intermediate layers of LLMs to skip during inference. The proposed method does not require additional training or auxiliary models, making it a plug-and-play solution for accelerating LLM inference across diverse input data streams. Extensive experiments conducted across various LLMs and tasks demonstrate that SWIFT achieves over a 1.3xx1.66 speedup while preserving the distribution of the generated text. Furthermore, of SWIFT in handling dynamic input data streams and its seamless integration with various LLM backbones, showcasing the great potential of this paradigm for practical LLM inference acceleration.
in-depth analysis highlights the effectiveness
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
The datasets used in our experiments are publicly released and labeled through interaction with hu- mans in English. In this process, user privacy protected, and no personal information is contained in the dataset. The scientific artifacts that we used are available for research with permissive licenses. The use of these artifacts this paper is consistent with their intended purpose.
ACKNOWLEDGEMENTS
We thank all anonymous reviewers for their valuable comments during the review process. The work described in this paper was supported by Research Grants Council of Hong Kong (PolyU/15207122, PolyU/152097244 Po1yU/15207821, Pol1U//0732 PolyU/15213323) and PolyU internal grants (BDWP).
REPRODUCIBILITY STATEMENT
All the results in this work are reproducible. We provide all the necessary code in the Supplemen- tary Material to replicate our results. The repository includes environment configurations, scripts, and other relevant materials. We discuss the experimental settings in Section 5.1 and Appendix C, including implementation details such as models, datasets, inference setup, and evaluation metrics.
REFERENCES
Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan- Kelley, and William Brandon. Hydra: Sequentially-dependeen draft heads for medusa decoding. CoRR, abs/2402.05109, 2024. doi: 10.4555/ARXXV.2222.550000 URL https: //o/..///////////// .org//rr////..
10. 08..48550/a20 20.5500/axii... 2402 05109.
Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se-Young Yun. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings the 2023 Conference on Empir- ical Methods in Natural Language Processing, pp. 5910-5924, Singapore, December 2023. Association for Computational Linguistics. doi: 20.18653/v/20233 120.:///1.....mmmmmmam..... ma..... URL https::// http:///aaaaatttogg....... org/ 2023.002222 connl-maaie 362.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:ttp//ccc...t.. /ttp::////p/eeviiieeee ew net forum? PEpbUobfJv.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. CORR, abs/2302.01318,, 2023, doi: 1048850WaXXi.223220013388 URL https:/ //doi///oii.o org/10.
48550/arXiv 01318.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, et al. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL
.rgg /t//a///....11/111
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman, Training verifiers to solve math word problems. CoRR, abs/2110.14168,, 2021. URL
https:/ //tp////aavvv///ae///eeeeeeaaaea org/ alos/ 2110. 14168.
Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Subhabrata Mukherjee. Skipdecode: Autoregressive skip decoding with batching and caching for effi- cient LLM inference. CoRR, 0td//237.022223 2023. doi: 10.48550/ARXIV22307 026288. URL https://dof 415p://..855///// a .20.7........... 2307.02622

---

Published as a conference paper at ICLR 2025
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3.int8(1 8- bit matrix multiplication for transformers at scale. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems 35: Anmual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http:/ htap://papeee papprspppsspppppppppp/aaeaeeeeea .ips. cttpspsccppppp..pppppppppp saapper_files/ppppee fttess/paper/2222/aaaa c3ba49 0e044992cccc44 (22499666666622299262
Abstract- Conference html.
Cunxiao Du, Jing Jiang, Yuanchen Xu, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, and Yang You. Glide with a cape: A low-hassle method to accelerate speculative decoding. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: /tttp:://openreveew net/forum? mik.iddkkmkkkxxx22
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The llama 3 herd of models, 2024. URL https: http//awwvvv.ooga/a//a/oa/ abs/44/7 2tt.s///247..277755
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak
coherent english? CoRR, abs/2305.00759, 2023. doi: 10.4850/ARXIV23300075555 URL http:/oorg.............. 2tt.://.....0/.................. 20....075555555
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. LayerSkip: Enabling early exit inference and self-speculative decoding. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12622-12642, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https:
/ //aclanthology org / 2024. acl - longg 681.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https:t////// wttp//ppwww.e....eeee net forum? prrrummiddtcccbee wwt.idddeeeeeeeeesseeee
Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of LLM inference using lookahead decoding. In Forty-first International Conference on Machine Learn- ing, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https:
/
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. CoRR, 2t0.///444499777 2024. doi: 10.48550/ARXIV.244.1999 19737.
URL https::/ddoo
485500...55////// 2404.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
URL https: /ttpppppeeeiieeeeeeeeeeeeeeeeeee wewweeeiiwww.... hwwp:/eettfoootteeeeeeee forum? id=5hQqf7
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming the rise of code intelligence. CoRR, abs/2401.14196, 2024. doi: 0044550/ARXIV.241..444 14196. URL https: dooi orgg /10 48550/arxiv.. 2401. 14196.
Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. REST: Retrieval-based speculative decoding. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pp. 1582-1595, Mexico City, Mex- ico, June 2024. Association for Computational Linguistics. URL https:////// pettpp:///ccaanee haclaatooooooy...... org/ 2024 4.222444..... nwaclaaalll.....o long. 88.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 221241)::11124, 2021.
12

---

Published as a conference paper at ICLR 2025
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Gene, Kurt Keutzer, Amir Gholami, and Yakun Sophia Shao. SPEED: speculative pipelined execution for efficient decoding. CoRR, abs/2310.12072, 2023. doi: 1044550/ARXIV.3300.DD077 URL https: //ooi. 10. 48550/arXiv 2310 2072.
Cheng- Yu Hsich, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Anna Rogers, Jordan I.. Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 8003-8017. Association for Computational Linguistics, 2023. doi: 10. 36663//1/2023FIINDIIIN 10. ww..1853/v1/2000 windings-annnlnn.n.o 507.
ACL.507, URL https: //oo//do.//// org/
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: www.opeeiieeeee... www.wwwwww...................... net/ forum? donomm/idnneeeeeeFFF
Ajay Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Shiwei Liu, Tianlong Chen, and Aditya Akella, Ffn- skiplim: hidden gem for autoregressive decoding with adaptive feed forward skipping. CoRR, abs/2404.03865, 2024. doi: 10.4 10448550/ARXIV2444.0088
URL https: //doi///oii////ooo org/10.
48550/arXiv 2384.03865.
Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of expen- sive black-box functions. Glob. Optim., 13(4):455 1998. doi: 00.1023/A:1088306411 147.
1000/112331000 1tt.://:.10000001111111 URL https: /ttp::/:///o..og11..0000000
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Alice Oh, Tris- tan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Ad- vances in Neural Information Processing Systems 36: Annual Conference
Neural Infor- mation Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:/ http://papers ppppppsppssp/pppppppp wwwpsicppppppeeeeeeeeee.p htc/paper_fillee httplss/ppppe//2233111 7757adeefalc51cf66 00531cff6623559ca3dd11oo.. 59c99dd77 Abstraatttabttttttetttttt wwweee-ooooooeceeeccoomee
Abstract
Taehyeon Kim, Ananda Theertha Suresh, Kishore A Papineni, Michael Riley, Sanjiv Kumar, and Adrian Benton. Accelerating blockwise parallel language models with draft refinement. In The Tight Annual Conference on Neural Information Processing Systems, 2024. URL https: : opwnreviewweeeeeeeeeeeeeee net/ wwweet/oouumideeeeeeee
Siqi Kou, Lanxiang Hu, Zhezhi He, Zhijie Deng, and Hao Zhang. Clims: Consistency large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Aus- tria, July 21-27, 2024. OpenReview.net, 2024. URL https: httpp//ppwwweeeeeeeeeppee net forum? 8.=
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019, doi: 10.1162/tael URL .rgg 20tp//1911022222
https:
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via specula- tive decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19274-19286. PMLR, 2023. URL httpsttt////oootoe pwwceeeeeeccccccoooooooooccccccc ngs html.
m...., www..mml..rrsss. press/
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty, 2024a.
13

---

Published as a conference paper at ICLR 2025
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: faster inference of language models with dynamic draft trees. CoRR, 2os/2406.11888,, 2024b. doi: 10.48550/ARXIV.2466 16858. URL https: www.o.org//00/ 485.0045551101.. 2406.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Ion Stoica, Zhijie Deng, Alvin Cheung, and Hao Zhang.
Online speculative decoding. CoRR, abs/2310.07177,, 2023a. doi: 104855/ARXW..210001111 URL https:ttt////.ctt. /doi org/ www448555554... wtt.:riiii............ 2310.
Yijin Liu, Fandong Meng. and Jie Zhou. Accelerating inference in large language models with a unified layer skipping strategy. CoRR, abs/2404.069544 2024. doi: 1044550/ARXV..40006633
URL https: / org/ 10. www.4///.......2.. 2404 06954.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://ppenrevvieer net/forun i id= dB331mm.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference Machine Learning, pp. 22137-22176. PMLR, 2023b.
Shuming Ma, Hongyu Wang, Lingxiao Ma. Lei Wang, Wenhui Wang. Shaohan Huang, Li Dong, Ruiping Wang. Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in I 1.58 bits. CoRR, abs/2402. 17764, 2024. doi: 10.48550/ARXIV.24022111
URL https:
//doi. wrw/r0gg11... //dpi.org/10. 48550/arXis 2402. 17764.
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang. Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serv- ing with tree-based speculative inference and verification. International Conference on Architectural Support for Programming Languages and Operat- ing Systems, Volume 3, ASPLOS '24, pp. 932 932-949, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703867. doi: 1011145/322066.3351333 htttps://di..ore 10.
Proceedings of the 29th ACM
URL
3651335.
Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequenee to-sequence mns and beyond. In Yoav Goldberg and Stefan Riezler (eds.), Proceedings the 20th SIGNLL Conference on Computational Natural Language Learning. CoNLL 2016, Berlin, Germany, August 11-12. 2016, pp. 280-290. ACL, 2016. doi: 10.18653/V1/K16 1028. URL. htttt://:://do org/ 100. 11885//w//16///
OpenAl. technical report. CoRR, ab01sB330223333.7774...2 2023. doi: 10448550/RRXIV22333.0777 URL
David A. Patterson. Latency lags bandwith. Commun, ACM, 47(10) 2004. doi: 10.1145/ 1022596.
0022594.1022500 URL /tt.:ttpp///o....o///0...../
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. In Dawn Song, Michael Carbin, and Tianqi Chen (eds.), Proceedings of the Sixth Conference on Machine Learning and Systems, MISSs 2023, Miami, FL, USA, June 4-8, 2023. misys.org, 2023. URL https: / http:///oocceeecoeoooooooo ngs mwl.lyy mlsys.......mmmmm.mmm.mmmm...m.m http:/ppprr_ffll htt.lle//pppp///// pappr/20233 hash/c4be aadeeaaaaaaa44444e3443300000 wdt.5eeeedddddboeeee.eeeco wwt.t//tt--bttttttc............. sya2023 html.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learn- ing. Adaptive computation and machine learning. MIT Press, 2006. ISBN 026218253X. URL
https: www. porlddatt org/ oclc/ 61285753.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu. Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Fereer Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez,
14

---

Published as a conference paper at ICLR 2025
Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve, Code llama: Open foundation models for code. CoRR, abs/2308. habbs//33..11900,, 2023. doi: (0.48550/ARXIV.33088. 12950. URL https:/ http://dooi.oo /ttp///..........0//////aaa////. 12950.
48550/ wrw.i/ a.t...iiviv...............
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorea, Michele Mancusi, Ric- cardo Marin, and Emanuele Rodolà. Accelerating transformer inference for translation via parallel decoding. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 12336-1222 12355. As- sociation for Computational Linguistics, 2023. doi: 10.18853/w1/2223.aco aat.o/22.... URL https://do/ htt///oi.orgg///// org/10 w223//1//223..vaeeeeee
689.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/ obs//1911,02150, 2019. URL
abs 11119991111. 02150.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kris- ten Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 31: Annual Conference on Neural Information Process- ing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 10107- 10116, 2018. URL https::/proceeddee www.nnggsneerrpee www.ieiiiip. wwtpeeeps....pppee///.......eeee wttpppppp////////e/ec/p 2hash/hhhhhhh///// 01127511166522066005552222233333
html.
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:/ http:///pppeeeeeeeee new forum?
Sd-SSYHLTCC5J.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing llama model. https: /gthhtttt....... wtt.:/cttttttteeeeeeeee.ee tatsu-lab/ aawaord_allaaes.. 2023.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 2023a. doi: 10448550/axxw.2230..17777 URL
bttpps://oo. www.ddoi.orgg/1./ 48550/ /0.550//xiii wrw.i.iiiivv..... 22..2
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307 009288, 2023b. doi: 10.
48550/ARXIVV.337700228 200.289307..888.... URL https:: //o/i.//////////oo orq/10 10. 48550/ arxiv. 2307. 09288.
Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decod- ing: Exploiting speculative execution for accelerating seq2seq generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 3909-3925, Association for Computa- tional Linguistics, 2023. doi: 10. findin http///22...nnnnnnn--eeeeeeeee..
URL https:
//doi. org/ 11t.3/10..1633111000000
Heming Xia, Zhe Yang. Qingxiu Dong, Peiyi Wang. Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Lum- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp. 7655-7671, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https: //acl antho loy.o...................... wtcy.orrg/2220 fi Findinge Endingsaall 456.
15

---

Published as a conference paper at ICLR 2025
Seongjun Yang. Gibbeum Lee, Jaewoong Cho, Dimitris S. Papailiopoulos, and Kangwook Lee. Predictive pipelined decoding: A compute-latency trade-off for exact LLM decoding. CoRR, abs/2307.05908 2023. doi: 10.4850/AARXIV.230.0033 48550/arxiv 2307 05908.
URL https:://do org/10.
Hanling Yi, Feng Lin, Hongbin Li, Ning Peiyang, Xiaotian Yu, and Rong Xiao. Generation meets verification: Accelerating large language model inference with smart parallel auto-core cottect decod- ing. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties ACI. 2024, pp. 5285-5299, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https: /ttpps://aaaaatie http://aaaaaalooogoogoooooo org/2024. windinggs-nii acl. 313.
Alex Young, Bei Chen, Chao Li, Chengen Huang. Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang. Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by Ol.ai. CoRR, abs/2403.04652. 2024. doi: 10. URL https: wodooi.orggo// 48550 arxiv 2403. 04652.
Jun Zhang, Jue Wang. Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft& verify: Lossless large language model acceleration via self-speculative decoding. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguisticy (Volume 1: Long Papers), pp. 11263- 11282. Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https: http///aaaaaahtohoeeo/e
wwt.////2022 2t1.22244..111onnoe.. 607.
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. The Twelfth International Conference on Learning Rep-
resentations, 2024. URL https: / opperevveww
net httpr/iwwwweeetttueeeeeeeeeeeeee Forum21-=r3YYJ3221C
Yunqi Zhu, Xuebing Yang, Yuanyuan Wu, and Wensheng Zhang. Hierarchical skip decoding for efficient autoregressive text generation. CoRR, abs/2403. 2024. doi: 10.48550/ARXIV. 2403.14919. URL https:
2ttp::/.......000............000 1499.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A PRELIMINARY DETAILS
We present the detailed configuration of Self-SD across four task domains in Figure 10, demonstrat- ing that the optimal skipped layer configurations vary depending on the specific task.
HIP
(a) Summarization
CNN/DM
MUP
Reasoning
GSM8K
Storytelling
TinyStories
(d) Translation WMT16
Figure 10: Visualization of skipped layer set configurations of -LaMA-2-13B optimized by Self- SD (Zhang al., 2024) on different task domains. Gray squares indicate retained layers, red squares denote skipped attention layers, and blue squares signify skipped MLP layers.
B EXPERIMENTAL SETUPS
B. MODELS AND DATASETS
Our experiments mainly evaluate the effectiveness of SWIFT on LLaMA-2 (Touvron et al., 2023b) and CodeLLaMA series (Rozière al., 2023). We provide empirical validation on diverse range of generation tasks. For summarization, mathematical reasoning, storytelling, and code generation tasks, we chose the CNN/Daily Mail (CNN/DM) (Nallapati al., 2016), GSM8K (Cobbe et al., 2021), Tinrito (Eldan & Li, 2023), and HumanEval (Chen et al., 2021) datasets, respectively. We perform 1-shot evaluation for CNN/DM and TinyStories, and 5-shot evaluation for GSM8K. The maximum generation lengths CNN/DM, GSM8K, and TinyStories are set to 64, 64, and 128, respectively. We compare pass @ and pass @ 10 for HumanEval. In our further analysis, we include three more datasets to validate the capability of SWIFT in handling dynamic input data streams. Specifically, we select Alpaca (Taori et al., 2023), WMT14 DE-EN, and Nature Ques- tions (Kwiatkowski et al., 2019) for the instruction following, translation, and question answering tasks, respectively. The maximum generation lengths for HumanEval and all analyses are set to 512. We randomly sample 1000 instances from the test set for each dataset except HumanEval.
B.2 INFERENCE SETUP
In the optimization phase, we employ both random search and Bayesian optimization to suggest potential skipped layer set candidates, striking balance between optimization performance and ef- ficiency. The context window 7 is set to 32. The maximum draft length ND is set to 25. For random sampling code generation tasks, we apply a temperature of 0.6 and top-p 0.95. The maxi- mum number of layer set optimization steps S' is set to 1000, with Bayesian optimization performed every = 25 steps. The optimization phase is set to be early stopped if the mat chness score does not improve after 300 steps exceeds 0.95. The layer skip ratio is fixed at 0.45 for the 13B model and 0.5 for the 34B and 70B models. All experiments were conducted using Pytorch 2. on 4x NVIIIA RTX A6000 GPU (40GB) with CUDA 12.1, and an Intel(R) Xeon(R) Platinum 8370C
17

---

Published as a conference paper at ICLR 2025
CPU with 32 cores. Inference for method and all baselines performed using the Hugging- face transformers package. Following prior work, we adopt speculative sampling (Leviathan et al,, 2023) as acceptance strategy, and the batch size is set to 1.
B.3 EVALUATION METRICS
This subsection provides a detailed illustration our evaluation metrics, which are mean generated length M and token acceptance rate a. Specifically, the mean generated length M refers to the average number of output tokens produced per forward pass of the target LLM; the acceptance rate is defined as the ratio of accepted tokens to the total number of draft steps. In other words, represents the expected probability the target LLM accepting a potential token from forward pass of the draft model. These two metrics are independent of computational hardware and, therefore considered as more objective metrics. Given the mean generated length M, acceptance rate a, and the layer skip ratio the mathematical formula for the expected wall-time speedup during the acceleration phase is derived as follows:
M
Ma
B(Spd.) (M ( (M - +
c= 1-r,
(6)
where is defined as the cost coefficient in Leviathan et al. (2023). It is calculated as the ratio between the single forward time of the draft model and that of the target LLM. In summary. the ideal speedup will be higher with the larger M and and smaller
C EXPERIMENTAL DETAILS
C.I DETAILS OF MAIN RESULTS
We present the detailed statistics of our main experimental results in Table SWIFT consistently achieves token acceptance rate exceeding 90% across all evaluation settings, with the mean generated length ranging from 2.99 8.21. These statistics indicate strong behavior alignment between the target LLM and its layer-skipping draft variant, discussed in Section 5.2. Addition- ally, we report the expected speedup ((Spd.) calculated using Eq (6), indicating that the current implementation of SWIFT has significant potential for further optimization boost its efficiency.
CNN/DM
GSMSK
TinyStories
Expected Speedup
Models
Methods
M
E(Spd.)
M
E(Spd.)
M
B(Spd.)
VANILLA
1.00
1.00x
1.00
1.00x
1.00
1.00x
1.00
LLaMA-2-13B
SWIFT
4.34
0.99
1.52x
3.13
0.98
1.43x
8.21
1.00
1.65x
1.53x
LLaMA-2-13B
VANILLA
1.00
1.00x
1.00
1.00x
1.00
1.00x
1.00x
-Chat
SWIFT
3.54
0.90
1.39
2.95
0.92
1.36x
7.42
0.99
1.62x
1,46x
VANILLA
1.00
1.00
1.00
1.00x
1.00
1.00x
1.00
LLaMA-2-70B
SWIFT
3.85
0.99
1.58x
2.99
0.98
1.48x
6.17
0.99
1.71x
1.59
Table 4: Detailed results of SWIFT text generation tasks using LLaMA-2 series. We report the mean generated length M, token acceptance rate a. and the expected speedup E(Spd.) calculated by Eq (6) under the setting of greedy decoding with FP16 precision.
C.2 ADDITIONAL RESULTS ON LLAMA-70B MODELS
In addition to the main results presented in Table 2, we provide further experimental evaluations of SWIFT on LLaMA-70B models, including LLaMA-2-70B and LLaMA-3-70B, along with their instruction-tuned variants, under the same experimental settings. The results demonstrate that SWIFT consistently achieves a 1.4x~1.5x wall-clock speedup across both the LLaMA-2 and LLaMA-3 series. Notably, SWIFT achieves token acceptance rate exceeding 85% across var- ious evaluation settings, with the mean generated length M ranging from 3.43 to 7.80. Although differences in layer redundancy are observed between models (e.g., skip ratio for LLaMA-2-70BB
18

---

Published as a conference paper at ICLR 2025
LLaMA-3-7000 3-B'''), SWIFT demonstrates robust adaptability, maintaining consistent acceleration performance regardless of model version.
CNN/DM
GSM8K
Tiny Stories
Overall Speedup
Models
Methods
M
Speedup
M
Speedup
M
Speedup
VANILLA
1.00
1.00x
1.00
1.00x
1.00
1.00x
1.00x
LLaMA-2-70B
SWIFT
3.85
0.99
1.43x
2.99
0.98
x.39
6.17
0.99
1.62x
1.48x
LLaMA-2-708 -Chat
VANILLA
1.00
1.00x
1.00
1.00x
1.00
1.00x
1.00x
SWIFT
3,43
0.85
1.31x
3.12
0.89
1.32
5.45
0.95
1.53x
1.37x
VANILLA
1.00
1.00x
1.00
1.00
1.00
1.00x
1.00x
LLaMA-3-70B
SWIFT
5.43
0.99
1.41x
4.11
0.99
1.37
7.80
0.99
1.51x
1.43x
LLaMA-3-70B
VANILLA
1.00
1.00x
1.00
1.00x
1.00
1.00x
1.00x
-Instruct
SWIFT
3.76
1.33x
3.92
1.31x
5.87
0.97
1.43x
1.36x
Table 5: Experimental results of SWIFT on text generation tasks using the LLaMA-70B series. We report the mean generated length M, token acceptance rate a, and speedup ratio under the setting of greedy decoding. The skip ratio r is set to 0.5 for LLaMA-2 models and 0.4 for LLaMA-3 models.
C.3 DETAILED RESULTS OF LLM BACKBONES
To further validate the effectiveness of SWIFT, we conducted experiments using additional LLM backbones beyond the LLaMA series. Specifically, select two recently representative LLMs: Yi-34B for text generation and DeepSeek-Coder-333 for code generation tasks. The experimental results are illustrated in Table 6 and demonstrating the efficacy of SWIFT across these L.M backbones. SWIFT achieves a consistent 1.2x~1 wall-clock speedup on the Yi-34B series and 1.3x~1..5x on the DeepSeck-Coder-333 SWIFT attains mean generate length M ranging from 3.16 to 4.17, alongside token acceptance rate a exceeding 83%. These findings substantiate the utility of SWIFT as paneral-purpose, plug- and-play SD method, offering promising inference acceleration across diverse LLM backbones.
series. Notably, for the DeepSeek-Codes Coobeer---odr series,
CNN/DM
GSMSK
TinyStories
Overall Speedup
Models
Methods
a
Speedup
a
Speedup
a
Speedup
VANILLA
1.00
1.00x
1.00
1.00x
1.00
1.00x
1.00x
Yi-34B
SWIFT
2.74
0.94
1.30x
2.65
0.97
1.28x
3.25
0.98
1.34x
1.31x
VANILLA
1.00
1.00x
1.00
1.00x
1.00
1.00x
1.00x
Yi-34B-Chat
SWIFT
2.84
0.91
1.29x
2.77
0.89
1.27x
2.52
0.80
1.21x
1.26x
Table 6: Experimental results of SWIFT on text generation tasks using Yi-34B series. We report the mean generated length M, token acceptance rate a and speedup ratio under the setting greedy decoding with FP16 precision. The skip ratio is set 0.45.
D FURTHER ANALYSIS AND DISCUSSION
D.1 ABLATION STUDY
Table presents the ablation study of SWIFT using LLaMA-2-13B on CNN/DM. The experimen- tal results demonstrate that each component of SWIFT contributes to its overall speedup SwIFT. Specifically, early-stopping drafting effeetively reduces the number of ineffective draft steps, im- proving the token acceptance rate a by 55%. Dynamic verification further enhances efficiency by selecting suitable draft candidates from the top-k predictions based on their confidence scores; re- moving this component leads to a decrease in both the mean generated length (M) and the overall speedup ratio. Additionally, the optimization phase refines the set of skipped layers, improving
During the optimization phase, the layer skip ratio for I.IaMA3-700 was automatically adjusted from 0.5 to 0.4 the token acceptance remained helow the tolerance threshold of 0.7.
19

---

Published as a conference paper at ICLR 2025
speedup by 34% compared the initial uniform layer-skipping strategy. In summary, these results confirm the effectiveness of each proposed innovation in SWIFT.
DS-Coder
DS-
Datasets
Methods
M
Speedup
M
Speedup
VANILLA
1.00
1.00x
1.00
1.00x
HumanEval (pass@))
SWIFT
4.97 0.99
1.54x
3.80
0.88
1.39
VANILLA
1.00
1.00x
1.00
1.00x
HumanEval (pass@ 10)
SWIFT
3.16 0.91
1.36x
3.74
0.83
1.31x
Table Experimental results of SWIFT on code generation tasks using DeepSeek- Coodeee--Cde series. The skip ratio is set to 0.5, We use greedy decoding for pass@) and random sampling with temperature of 0.6 for pass@ 10. "DS" denotes the abbreviation of DeepSeek.
Methods
M
Speedup
7
M
Cx
Speedup
Latency
Vanilla
1.0
1.000
16
3.91
0.95
1.341 x
0.242ms
32
5.82
0.98
1,560 x
0.244ms
SWIFT
5.82
0.98
1,560
64
5.56
0.99
1.552x
0.312ms
w/o early-stopping
11.16
0.43
0.896x
5.61
0.98
1,550x
0.425ms
w/o dynamic ver.
4.39
0.90
1.342
w/o optimization
2.15
0.90
1,224x
Table 9: Speedups of SWIFT across differ- ent context window 7. The latency of the optimization step is reported to illustrate the associated overhead.
Table 8: Ablation study of SWIFT. "ver." denotes the abbreviation verification.
D.2 CONTEXT WINDOW
In Table 9, we show a detailed analysis of context window which determines the number of LLM-generated tokens used the layer set optimization process. smaller y introduces greater randomness in the mat chness score calculation, resulting in suboptimal performance, while larger increases the computational overhead of the optimization step. The results indicate that y = 32 provides an optimal balance between optimization performance and computational overhead.
D.3 COMPARISONS WITH PRIOR LAYER-SKIPPING METHOSS
In this subsection, we compare SWIFT with two representative layer-skipping speculative decoding (SD) methods: LayerSkip (Elhoushi al., 2024) and Self-SD (Zhang et al., 2024). Specifically, LayerSkip (Elhoushi et al., 2024) introduces an innovative approach to self-speculative decoding by implementing early-exit drafting, where the LLM generates drafts using only its earlier layers. How- ever, this method necessitates a cone-consuming pretraining or finetuning process, which modifies the original output distribution of the target L.M.. Such alterations may compromise the reliability of the generated outputs; Self-SD (Zhang et al., 2024) proposed to construct the compact draft model by skipping intermediate layers, using an extensive Bayesian Optimization process before inference to determine the optimal skipped layers within the target LLM. As illustrated in Section 3.1, while effective, Self-SD suffers from significant optimization latency (nearly 7.5 hours for LLaMA-2-133 This prolonged optimization process limits its practicality and
and 20 hours for LLaMA-2-7000 generalizability across diverse models and tasks.
Tables 10 and 11 summarize the comparative results in terms of acceleration performance and train- ing/optimization costs, respectively. Below, we detail the advantages of SWIFT over these methods:
LayerSkip achieves an aggressive skip ratio (r 0.8), resulting in an average generated length of 2.42 and token acceptance rate of 0.64. How- ever, its reliance on pretraining or finetuning alters the original distribution of the target LLM, potentially reducing reliability. In contrast, SWIFT preserves the original distribu- tion the target LLM while delivering a comparable 1.56 speedup without requiring additional training.
Comparison with LayerSkip:
20

---

Published as a conference paper at ICLR 2025
Methods
Plug&Play Original
M
Speedup
LIYERSKIP
0.80
2.42
0.64
1.64x
SELF-SD
0.43
4,02
"088
T.29
SELF-SD w/ dynamic ver.
0.43
5.69
0.98
1.52x
SWIFT (Ours)
0.45
5.82
0.98
1.56x
Table 10: Comparison of SWIFT and prior layer-skipping SD methods. We report the skip ratio r, mean generated length M, token acceptance a, and speedup ratio under greedy decoding. The results are obtained with LLaMA-2-15 13B on CNN/DM. "ver. denotes the abbreviation of verification.
Methods
Training Cost
Optimization Latency
LAYERSKIP
50 x 10 training steps with A100 (80GB)
SELF-SD
1000 Bayesian Optimization Iterations Before inference
775 hours 22 minutes
SWIFT (Ours)
N/A
Table 11: Comparison of SWIFT and prior layer-skipping SD methods in terms of training cost and optimization latency for LLaMA-2-13B. Training costs are sourced from the original papers, while optimization latency is measured from our re-implementation demonstrates a ~000 reduction in optimization latency compared to previous methods without requiring additional training, establishing an efficient plug-and-ple and-play SD method.
an A6000 GPU. SWIFT
Comparison with Self-SD: Self-SD relies on a time-intensive Bayesian Optimization pro- cess, which incurs substantial latency before inference. SWIFT eliminates this bottleneck through an on-the-fly optimization strategy, achieving an approximately x reduction in optimization latency while maintaining the same 1.56x speedup. We further augmented Self-SD with our Confidence-avare Inference Acceleration strategy (Self-SD w/ dynamic ver.). Even compared this augmented version, SWIFT achieves competitive speedups.
These findings highlight the efficiency and practicality of SWIFT over previous layer-skipping SD methods. As the first plug-and-play layer-skipping SD approach, we hope that SWIFT could provide valuable insights and inspire further research in this area.
D.4 DETAILED COMPARISONS WITH SELF-SD
In this subsection, we provide detailed compar- ison of SWIFT and Self-SD (Zhang et al., 2024). Figure presents the speedups of Self-SD across varying optimization latencies, reflecting the in- crease in Bayesian Optimization iterations. As shown, Self-SD achieves minimal speedup improve- ment almost equivalent to unified skipping with fewer than 50 Bayesian iterations, corresponding an optimization latency below 1474 seconds. At 100 Bayesian iterations, Self-SD achieves 1.19x speedup; however, its optimization latency is nearly 25 times longer than that of SWIFT (2898s vs. 116s).
1.55
1.35
SWIFT
1.15
0.95
6000
Optimization Latency (5)
Figure 11:
Comparison of SWIFT and Self- SD in terms of optimization latency and speedup. SWIFT achieves a 1.56x speedup with optimization lateney of 11 116 seconds.
Table 12 compares SWIFT and Self-SD (first two rows) under similar optimization latencies. The results highlight SWIFT'S superiority in both optimization efficiency (1s vs. 1558) and speedup (1.56x vs. x.. Even when compared to the augmented version of Self-SD (w/ dynamic verifi- cation), SWIFT achieves a substantial 30% relative improvement in speedup. Below, we analyze the factors contributing to this advantage (elaborated in Section 3.1):
Optimization Objective Granularity: Self-SD calculates its optimization objective at multi-sample level, requiring sequential decoding of all selected training samples (c.g., 8 samples with 32 tokens each) for every iteration optimize Equation 1. In contrast,
21

---

Published as a conference paper at ICLR 2025
#Random Optimization
#Bayesian Optimization
Optimization Latency (8)
Methods
M
Speedup
SELF-SD
155
0.50
1.80
0.57
0.97
SELF-SD w/ dynamic ver.
155
0.50
2.07
0.86
1.17x
SELF-SD.
30
199
0.45
2.08
0.70
1.04
SELF-SD, w/ dynamic ver.
30
199
0.45
2.44
0.93
1.22x
SWIFT (Ours)
552
23
116
0.45
5.82
0.98
1.56x
Table 12: Comparison of SWIFT and Self-SD at similar optimization latencies. We report the skip ratio r, mean generated length M, token acceptance rate a, and speedup under greedy decoding. The results are obtained with LLAMA-2-13B on CNN/DM, with "ver." indicating verification.
SWIFT adopts step-level optimization objective, optimizing the layer dynamically each decoding step.
Bayesian Optimization Complexity: The computational complexity of Bayesian opti- mization increases significantly with the number of iterations. SWIFT mitigates this bur- den by combining random search with interval Bayesian optimization, accelerating conver- gence of the optimization process while reducing computational overhead.
To further examine optimization trade-offs, we reduce Self-SD's sequential optimization require- ment single sample with 8 tokens, enabling more Bayesian Optimization iterations within comparable latency. The corresponding results, denoted as Self-SD: (rows 3-4), are presented in Table 12. Even with these optimized settings, SWIFT demonstrates substantial superior speedup and efficiency, highlighting the effectiveness of our proposed strategies.
D.5 THE NECESSITY OF PLUG-AND- SD METHODS
There has been surge of recent interest Speculative Decoding (SD), leading to the development of numerous promising strategies in the field, which can be broadly categorized into two directions:
requuied SD. These methods require additional pretraining or fine-tuning to im- prove speculative accuracy, often involving the integration of extra parameters. For stance, Medusa (Cai et al., 2024) and Eagle (Li et al., 2024a;b) incorporate lightweight draft heads into target L.LMS and fine-tune them, achieving 3x~44 speedups.
Plug-and- SD. These approaches offer immediate acceleration of LLM inference with- out relying on auxiliary models or additional training. Notable examples include Parallel Decoding (Santilli et al., 2023) and Lookahead (Fu et al., 2024), which leverage Jacobi- based drafting, achieving 1.2xx114x speedups across various LLMs.
While training-required SD methods generally deliver higher speedups, their reliance on additional training and parameters limits both their generalizability and practicality. This has sparked debate within the academic community regarding the value of plug-and-play SD methods. To address these concerns, we present detailed analysis below to highlight the necessity of plug-and-pla. SD approaches and underscore the contributions of our proposed SWIFT:
1) Training costs of training-required SD methods are often prohibitive. Frauiie methods such as Medusa (Cai et al., 2024) and Eagle (Li et al., 2024a;b), while achieving higher speedups, ineur substantial training costs. Despite efforts to reduce training overhead, these methods still require extensive computational resources (e.g., GPU time and datasets) to deliver valid accel- eration performance. For example, Eagle requires 1-2 days training with 8 RTX 3090 GPUs for LLaMA-33B up to 2 days on 4 A 110 (40G) GPUs for dialogues from ShareGPT. Such computational burdens introduce challenges several scenarios:
using dataset of 70k
Users must train new draft models for unsupported target LLMs. For example, the user's target L.LM is not among the released checkpoints or if the hase model updated (e.g..
22

---

Published as a conference paper at ICLR 2025
LLaMA-3.x), users are forced to train new draft model, which may exceed their available GPU resources (e.g., GPU time).
Users with small-scale acceleration needs face inefficiencies. For instance, researcher needing to evaluate a small set of samples (e.g.. 10 hours of evaluation) would find the 1-2 day training requirement disproportionate and hinder overall research efficiency.
2) Plug-and-play SD fills critical gaps unaddressed by training-required methods. Plug-and- play SD methods, including SWIFT, are model-agnosti and training-free, providing immediate ac- celeration without requiring additional computational overhead. These attributes are particularly critical for large models (70B-340B) and for use cases requiring rapid integration. The growing adoption of plug-and-play SD methods, such as Lookahead (Fu et al., 2024), further underscores their importance. These methods cater to scenarios where ease of use and computational efficiency are paramount, validating their research significance.
3) SWIFT pioneers plug-and-play SD with layer-skipping drafting. SWIFT represents the first plug-and-play SD method to incorporate layer-skipping drafting. It consistently achieves 1.3x~1.6x speedups over vanilla autoregressive decoding across diverse models and tasks. Ad- ditionally, demonstrates 10%~20% higher efficiency compared Lookahead (Fu et al., 2024). Despite its effectiveness, SWIFT introduces a complementary research direction for existing plug- and-play SD. Its approach is orthogonal to Lookahead Decoding, and combining the two could fur- ther amplify their collective efficiency. We believe this study provides valuable insights and paves the way for future SD advancements, particularly for practical and cost-effective LLM acceleration.
To sum up, while training-required SD methods achieve higher speedups, their high computational costs and limited flexibility reduce practicality. Plug-and-play SD methods, like SWIFT, offer training-free model-agnostic acceleration, making them ideal for diverse scenarios. We hope this clarification fosters greater awareness and recognition of the value of plug-and-play SD research.
D.6 ADDITIONAL DISCUSSIONS WITH RELATED WORK
In this work, we leverage the inherent layer sparsity of LLMs through layer skipping, which selec- tively bypasses intermediate layers within the target LLM to construct the compact draft model. In addition to layer skipping, there has been another research direction in SD that focuses on early ex- iting, where inference halts at earlier layers to improve computational efficiency (Yang et al., 2023; Hooper et al., 2023; Bae et al., 2023; Elhoushi et al., 2024). Particularly, LayerSkip (Elhoushi al., 2024) explores early-exit drafting by generating drafts using only the carlier layers of the target LLM, followed by verification with the full-parameter model. This approach requires training in- volving layer dropout and early exit losses. Similarly, PPD (Yang et al., 2023) employs early exiting but trains individual classifiers for each layer instead of relying on a single final-layer classifier. Although effective, these methods rely on extensive fine-tuning to enable early-exiting capabilities, incurring substantial computational costs. Moreover, the training process alters the target LLM's original output distribution, potentially compromising the reliability of generated outputs. In con- trast, our proposed SWIFT does not require auxiliary models or additional training, preserving the original output distribution of the target I.I.M while delivering comparable acceleration benefits.
There has been a parallel line of training-required SD research focusing on non-autoregressive draft- ing strategies (Stern et al., 2018; Cai et al., 2024; Gloeckle et al., 2024; Kim al., 2024). These methods integrate multiple draft heads into the target LLM, enabling the parallel generation of draft tokens at each decoding step. Notably, Kim et al. (2024) builds on the Blockwise Parallel Decod- ing paradigm introduced in Stern et al. 22018), accelerating inference by refining block drafts with indeeedeendd n- r-grams and lightweight rescorers using smaller LMs. While these approaches achieve notable acceleration. they also necessitate extensive training of draft models. SWIFT com- plements these efforts by pioneering plug-and-play SD that eliminates the need for auxiliary models or additional training, offering a more flexible and practical solution for diverse use cases.
D.7 OPTIMIZATION STEPS
We present the detailed configuration of SWIFT across various optimization steps in Figure 10. As the process continues, the skipped layer set is gradually refined toward the optimal configuration.
23

---

Published as a conference paper at ICLR 2025
HPP
ATT
(a) Optimization Step
HPP
Optimization Step
MUP
(c) Optimization Step 128
Optimization Step 512
Figure 12: Visualization of skipped layer set configurations of LLaMA-2-133 optimized by SWIFT on different optimization steps. Gray squares indicate retained layers, red squares denote skipped attention layers, and blue squares signify skipped MLP layers.
D.8
CASE STUDY
We present examples from CNN/DM and Humaneval in Table and 14, respectively, comparing to- ken acceptance between SWIFT and prior plug-and-play approach, Lookahead (Fu et al., 2024). The results demonstrate that SWIFT accepts a higher number of drafts, which are uniformly distributed in the outputs, highlighting the effectiveness of our approach.
Table 13: CNN/DM example using LLaMA-2-1368 red tokens indicate accepted drafts.
An Alberta student who'd accidentally left his headlights all day was greeted by what may have been the world's friendliest note from a stranger when he returned to his But Derek Murray, University ... Summary:
Article:
User Input
Canadian law student was greeted by a friendly note from a stranger when he returned to his car after leaving the headlights on all day
LOOKAHEAD
Canadian law student was greeted by a friendly note from a stranger when he returned to his after leaving the headlights all day.
SWIFT
Table 14: Humaneval example using CodeLLaMA-13BB red tokens indicate accepted drafts.
from typing import List def has_close_elementtte elmmeettsssuubbrrr: List[float], thresh-
User Input
old: float) booo:: Check if in given list of numbers, any two numbers closer to each other than given threshold. >> >>>>>has_coosc.eemmett
(ungerss 1): if abs(mbeerss(]]] numbbrs[i + 1J) <= thresh- for i in range(len( numbers)
old: return True return False def has has_elose_a .eeements deements.2nnumbeer 2(numbers: List[float], threshold: float) ->bool: *.*. Check if in given list of numbers, are any two numbers closer to each other than given threshold. >>>has wwwoe.eeeeeeee.eeeeee for in (umberess))- 1): if abbmnmbbers[i]] mummbers[i+]] < ihresh old: return True return False def has www._cloeeeeleee celements. 2(numbers: List[float], threshold: float) >bbool: Check if in given list of numbers, any two numbers closer to each other than given threshold. >>> has close elements
LOOKAHEAD
SWIFT
24


Reviews:
[{"review_id": "b9I43l3MnS", "reviewer": "Reviewer_rnoa", "summary": "By adaptively skipping intermediate layers during inference, SWIFT improves speedups of LLMs without compromising the quality of generation. The method integrates a Bayesian optimization-based layer selection mechanism to adapt to task-specific requirements dynamically.", "questions": "1. How does SWIFT handle non-stationary input distributions during Bayesian optimization?\n\n2. Could the authors provide insights into how SWIFT performs under extreme token count variations or highly domain-specific tasks?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "While the concept of layer-skipping is not really novel, its use of bayesian optimization can be a good idea for self-SD.", "weaknesses": "1. The reward design and its stability under distributional changes need more explanation. Open discussion with concurrent work, such as \"A Unified Framework for Speculative Decoding with Multiple Drafters as a Bandit (Submitted at ICLR'25; https://openreview.net/forum?id=5haYLrlyGj)\", could enhance understanding of these challenges. While the primary focus is different, the insight of using bandit approach is quite similar to this paper. And I recommend the authors to put the discussions for the assumption and extensions of Bayesian optimization for layer skipping inspired by this work.\n\n\n2. More discussions on related work, such as Kim et al. (2024), Stern et al. (2018), and Gloeckle et al. (2024) on pretrained blockwise parallel language models, would position the contribution better within the existing literature. Because both papers are also a parallel line of work for self-speculative decoding, while they use the non-autoregressive heads instead. \n\n- Gloeckle et al. (2024),  Better & Faster Large Language Models via Multi-token Prediction.\n- Stern et al. (2024), Blockwise Parallel Decoding for Deep Autoregressive Models\n- Kim et al. (2024), Accelerating Blockwise Parallel Language Models with Draft Refinement. (https://openreview.net/forum?id=KT6F5Sw0eg)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vaHtgvrEtH", "reviewer": "Reviewer_tWD9", "summary": "This paper aims to improve speculative decoding (SD) with a focus on eliminating the need for additional model parameters or extensive training to enable effective drafting in SD. In particular, the paper utilizes the same (target) model as the draft model by skipping a subset of model layers while generating draft tokens. Towards this, the paper proposes an SD method, namely SWIFT, that performs on-the-fly adaptive layer selection via an optimization phase to identify task-specific layers to skip. The optimization phase is followed by an inference acceleration phase that leverages the identified layers to perform skipping during drafting. During the inference acceleration phase, SWIFT additionally relies on 1) early stopping of the drafting process if (draft) model's confidence is not high enough; and 2) utilizing top-k predictions for each draft token position during parallel verification. The authors empirically validate the utility of SWIFT by showcasing 1.3-1.6x speed-up on CNN/DM, GSM8K, and TinyStories datasets.", "questions": "Please see the weaknesses section above. In addition, please consider the following questions:\n\n1) Looking at the ablation studies in Appendix D (Table 7), it appears that *dynamic verification* does not bring much value as the loss in overall speed-up is minimal when one excluded dynamic verification (1.560x to 1.541x). Could authors comment on this?\n\n2) Do the speedup numbers in Table 2 take into account the optimization phase? If yes, how many LLM generations are performed to obtain the results in Table 2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper successfully demonstrates that the speculative decoding (SD) framework has the potential to speed up LLM inference even when one does not employ additional model parameters and task-specific training to support the drafting phase.\n\n2) The paper makes two key observations about layer skipping during the drafting phase that highlights the need for adaptive (task-specific) selection of layers to skip during the drafting phase to maximize the benefit of layer skipping-based drafting approach. Subsequently, the paper proposes SWIFT - an effective SD approach that can identify a reasonable set of layers to skip for the underlying task with minimal training.\n\n3) The paper further showcases the utility of leveraging the (draft) model's prediction confidence and top-k per-token predictions to improve the realized speed-up via SWIFT.\n\n4) The paper is mostly well-written and conveys the key ideas in sufficient detail. The proposed ideas exhibit sufficient novelty over existing SD methods. The empirical results and in-depth empirical analysis highlight the gains realized by SWIFT over vanilla LLM inference.", "weaknesses": "1) There is room for improvement in the discussion of related prior work. Given that Elhoushi et al. 2024 also leverage layer skipping during the drafting phase, a detailed discussion of this work is warranted. Furthermore, the authors may also want to cite https://openreview.net/pdf?id=yUmJ483OB0. \n\n2) The authors may want to make their empirical evaluation more comprehensive. Currently, the authors don't compare with existing approaches that rely on layer skipping during the drafting phase. Even though these existing methods might rely on extensive training, the authors should compare SWIFT with these methods. Such a comparison can highlight if there is any performance gap between these methods and their proposed plug-and-play approach.\n\n3) The paper aims to eliminate the extensive training of existing layer skipping-based approaches via an efficient on-the-fly optimization phase. However, it's not clear if the existing methods can also perform well even when one limits the amount of offline training for these methods.\n\n4) The authors repeatedly emphasize that their proposed method is a plug-and-play method. However, they don't seem to be evaluating their method in a dynamic setting where the underlying task (distribution) changes over time. In such a dynamic setting, would SWIFT have to interleave the optimization and acceleration phases? Would one still observe a good amount of speed up in such settings?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VPtIcloQRo", "reviewer": "Reviewer_LGVh", "summary": "This paper proposes a plug-and-play self-speculative decoding method. The authors employ a layer-skipping approach to construct a draft model. Experimental results indicate that this method achieves a 1.3-1.6 times inference speedup on Llama-2 and Code-Llama models.", "questions": "1. Table 2 presents results for Llama-2-13B, Llama-2-13B-Chat, and Llama-2-70B. Why are the results for Llama-2-70B-Chat and Llama-2-7B(-Chat) not included?\n2. How does the overhead of the proposed layer searching algorithm compare to the overhead of training additional modules like Eagle?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method does not require training an additional model or module for drafting, making it applicable to most large language models.", "weaknesses": "1. The speedup is not as promising compared to other training-free methods like Lookahead. The authors should also present results for methods such as Mudusa and Eagle, which require minimal training overhead.\n2. It is recommended that the authors test well-trained LLMs, such as Llama-3, as models with less effective performance might yield higher speedup ratios.\n3. The method requires different settings for different tasks. However, in real-world LLM chat applications, it is often difficult to predict the corresponding tasks of user instructions. It is suggested that the authors evaluate the method's speedup performance on benchmarks like MT-Bench, which test the general capabilities of models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kn7YXzjX0u", "reviewer": "Reviewer_vnfL", "summary": "This paper aims to accelerate the inference of LLMs. They introduce SWIFT, a self-speculative decoding algorithm that adaptively selects intermediate layer to skip without extra cost. They performed an empirical analysis of layer-skipping SD paradigm and show the potential of self-accelerate of LLMs through layer sparsity. They used some techniques like early-stop drafting to further speed up reasoning.", "questions": "1.The author should compare their method with self-SD [1] in table 2, since their method is an improvement of the latter. \n2. The author only compared to the baseline on the Llama and CodeLlama models. I believe experiments should be conducted on larger models with different architectures to demonstrate the generalization of the method.\n3.Moreover, compared with self-SD, the innovation is still insufficient, for example , the confidence-aware inference strategies are similar to some mechanism in [1],[2]\n4.Despite SWIFIT does not require additional training, comparing with other method ,like EAGLE [2] ,Medusa [3], which can achieve over a 3.05-4.26x speedup, SWIFIT does’t show much value.As reported in [2], the draft models is trainable within 1-2 days for 70B models.\n\n[1] Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. ACL 2024\n[2] EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees, EMNLP 2024\n[3] Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads, ICML 2024", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The paper is well-written and flows very smoothly. \n2.The authors make effort to demonstrate the feasibility of their theory through experiments.\n3.The method incorporates many of the latest techniques.", "weaknesses": "1.The author should compare their method with self-SD [1] in table 2, since their method is an improvement of the latter. \n2. The author only compared to the baseline on the Llama and CodeLlama models. I believe experiments should be conducted on larger models with different architectures to demonstrate the generalization of the method.\n3.Moreover, compared with self-SD, the innovation is still insufficient, for example , the confidence-aware inference strategies are similar to some mechanism in [1],[2]\n4.Despite SWIFIT does not require additional training, comparing with other method ,like EAGLE [2] ,Medusa [3], which can achieve over a 3.05-4.26x speedup, SWIFIT does’t show much value.As reported in [2], the draft models is trainable within 1-2 days for 70B models.\n\n[1] Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. ACL 2024\n[2] EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees, EMNLP 2024\n[3] Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads, ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)