DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding
Authors: Hossein Entezari Zarch
Lei Gao
Chaoyi Jiang
Murali Annavaram
Venue: COLM 2025 (2025)
Keywords: Speculative Decoding
Efficient Large Language Model
Inference Acceleration
TL;DR: We propose DEL, a dynamic method that adaptively selects the exit layer and speculation length in self-speculative decoding to accelerate large language model inference.

Abstract: Speculative Decoding (SD) is a widely used approach to accelerate the inference of large language models (LLMs) without reducing generation quality. It operates by first using a compact model to draft multiple tokens efficiently, followed by parallel verification using the target LLM. This approach leads to faster inference compared to auto-regressive decoding. While there are multiple approaches to create a draft model, one promising approach is to use early-exit methods. These methods draft candidate tokens by using a subset of layers of the primary model and applying the remaining layers for verification, allowing a single model to handle both drafting and verification. While this technique reduces memory usage and computational cost, its performance relies on the choice of the exit layer for drafting and the number of tokens drafted (speculation length) in each SD round. Prior works use hyperparameter exploration to statically select these values. However, our evaluations show that these hyperparameter values are task-specific, and even within a task they are dependent on the current sequence context. We introduce DEL (Dynamic Exit Layer), a plug-and-play method that adaptively selects the exit layer and speculation length during inference. DEL dynamically tracks the token acceptance rate if the tokens are drafted at each layer of an LLM and uses that knowledge to heuristically select the optimal exit layer and speculation length. Our experiments across a broad range of models and downstream tasks show that DEL achieves overall speedups of $2.16\times$$\sim$$2.62\times$ over vanilla auto-regressive decoding and improves upon state-of-the-art SD methods, which peak at $2.43\times$, by up to $0.19\times$. The code is available at https://github.com/hoenza/DEL.

Published as a conference paper at COLM 2025
DEL: Context-Aware Dynamic Exit Layer for Efficient Self- Speculative Decoding
Hossein Entezari Zarch", Lei Gao", Chaaoi Jiang, Murali Annavaram University of Southern California {entezari, leig, chaoyij, annavaraJ@usc.cm edu
Abstract
Speculative Decoding (SD) is widely used approach to accelerate the inference of large language models (LLMs) without reducing generation quality. It operates by first using compact model to draft multiple tokens efficiently, followed by parallel verification using the target LLM. This approach leads to faster inference compared to auto-regressive decoding. While there are multiple approaches to create a draft model, one promising approach is to use early-exit methods. These methods draft candidate tokens by using subset of layers of the primary model and applying the remaining layers for verification, allowing single model to handle both drafting and verification. While this technique reduces memory usage and computational cost, its performance relies on the choice of the exit layer for drafting and the number of tokens drafted (speculation length) in each SD round. Prior works use hyperparameter exploration to statically select these values. However, our evaluations show that these hyperparameter values are task-specific, and even within a task they are dependent on the current sequence context. We introduce DEL (Dynamic Exit Layer), plug-and-play method that adaptively selects the exit layer and speculation length during inference. DEL dynamically tracks the token acceptance rate if the tokens are drafted at each layer of an LLM and uses that knowledge to heuristically select the optimal exit layer and speculation length. Our experiments across a broad range of models and downstream tasks show that DEL achieves overall speedups of 2.16x 2262 over vanilla auto-regressive decoding and improves upon state-of-the-art SD methods, which peak at 2.43x, by up to 0.19x. The code is available at https: / /hu.. com/ /DLL
1 Introduction
Large language models (LLMs) such Llama 2 (Touvron et al., 2023), Llama 3 (Dubey et al., 2024), GPT-4 (OpenAI et al., 2024), and Claude (Bai et al., 2022) have demonstrated strong performance across diverse tasks, driving adoption in search engines (Microsoft, 2023; Reid, 2023), chatbots (OpenAI, 2022), and virtual assistants (Wu et al., 2023a;b). However, LLM token generation remains slow due its auto-regressive nature, where each token is generated sequentially based on all previous tokens. This sequential dependency results in low generation speed, as every new token requires access to the full set of model parameters. The resulting T/O bottleneck limits hardware utilization during inference, especially in small-batch or low-latency scenarios, such as edge deployments or real-time applications.
Speculative decoding (SD) (Leviathan et al., 2023; Chen et al., 2023) addresses this bottleneck by using a smaller model draft sequence of tokens ruto-regressively, which are then verified in parallel by the target model. This reduces latency while preserving the output distribution of the main model. The performance of SD depends on two key factors: the speed and accuracy of the draft model, and the number of tokens drafted per SD round (speculation length r). Larger draft models increase acceptance rates but reduce draft speed.
"Equal contribution.

---

Published as a conference paper at COLM 2025
A larger "y increases the number of proposed tokens per round, which may lead to more accepted tokens, but it also increases the likelihood of rejection, potentially reducing overall efficiency, Early versions of SD relied on separate draft models with fixed ", tuned offline. Later work explored adaptive Y policies (Zhang et al., 2024).
LayerSkip (Elhoushi et al., 2024) is recent self-speculative decoding method that improves efficiency by removing the need for a separate draft model. Instead, it reuses the first E layers of the main model to draft tokens and the remaining layers for verification, reducing both memory and compute overhead. Prior works have relied on static configuration of and y, selected via offline grid search. This introduces two key limitations. First, the optimal& & and vary significantly across tasks; configurations tuned for one task often underperform on others. For example, using a language modeling -oaiiiiigooimidddddeeeee configuration on a summarization task reduces speedup from 2.65x to x.50 Second, even within single task, optimal configurations can shift during generation. In code generation, for instance, setting that performs well for early tokens may degrade performance later in the sequence. These findings indicate that fixed configurations are suboptimal and motivate the need for dynamic approach that adapts & and y throughout the generation process.
Coincidentally, we observe that LayerSkip is particularly well-suited for dynamic selection of & and y with minimal overhead. First, since any prefix of the model layers can serve as a draft model, LayerSkip naturally exposes a spectrum of sub-models defined by the choice of exit layer E, with no overhead when switching between them during generation. Second, for each draft token, LayerSkip computes hidden states of the first & model layers during the drafting stage, and then computes the remaining layers hidden states during verification in each SD round. Importantly, these intermediate hidden states computed can be reused to estimate the token acceptance rate under different choices of E. Together, these two properties enable real-time adaptation of È and "y during inference, guided by layer-wise acceptance rate estimates. To fully exploit this flexibility, however, one needs an algorithmic approach to adaptively select appropriate values for & and 7 at each SD round.
To address this challenge, we introduce DEL (Dynamic Exit Layer), an algorithmic approach that is implemented as a plug-and-play module for LayerSkip that adaptively selects & and Y at each round of speculative decoding. DEL tracks the acceptance rate of draft tokens for each layer using cached hidden states, and uses this information to estimate metric called Token-per-Layer (TPL), which reflects decoding efficiency. By selecting the configuration that maximizes TPL., DEL improves speed without sacrificing output quality. DEL also adjusts dynamically within each round using a context-aware confidence threshold to decide when to stop drafting.
In summary, our key contributions are:
We conducted an empirical study on the impact of È and y in LayerSkip, revealing that the optimal settings of these values are model- and input-dependent, motivating the need for dynamic selection of these parameters during inference.
Based on these observations, we propose DEL, plug-and-pla module for LayerSkip that adaptively selects & and y by evaluating cached hidden states and applying a dynamic, context-aware thresholding mechanism.
Through extensive evaluations across tasks and model sizes, we show that DEL achieves up 2.62x speedup over vanilla decoding and outperforms dynamic LayerSkip baselines, which peak at 2.43x, while maintaining output quality and incurring negligible overhead.
2 Background
Auto-regressive decoding. Given an input sequence (xo....,X441)) a transformer-base auto-regressive language model My generates the next token x; from the conditional probability distribution p(x Xer). Specifically, each transformer decoder layer processes the hidden states ky from the previous layer as follows:
h Transformer, l [1,
2

---

Published as a conference paper at COLM 2025
where h. is the embedding representation of X;-1- After the L-th layer of the model, the predicted token x; is determined by the probability output from a softmax classifier: p(x)) = LM-Head (h)) = softmax(W h.). We use the notation pxx+) to denote pxx+ xxt).
Speculative decoding. The SD algorithm generates tokens in two phases: drafting and veri- fication. During the drafting phase, a small and efficient draft model Mg is used to sample candidate token distributions (x()),,... ((())),,.,,(((+1)))> ((x++>=))}
In the verification phase, the input sequence concatenated with these y drafted tokens is passed to the target model Mp, which computes the probabilities (x(x)),..., p(x,,)))))) } in parallel. The verification step compares 1(x+1t) and q(x+1i). and accepts the drafted token with probability min p(x i) (x+
If a token x is rejected before all Y tokens are accepted, the remaining draft tokens are discarded, and X1+i is resampled from the residual distribution max (0, p(x (x)+))) Otherwise, if all drafted tokens are accepted, SD samples one additional token from p (x*++)) and appends it to the sequence. Each round of SD thus generates at least one and at most y+1 tokens. Leviathan et al. (2023) prove that the output sequence from SD follows the same distribution as that of the target model. In the special case of greedy decoding, the verification phase simply compares the top-1 prediction from p(x+ (i) and q i)- A key advantage of SD is that the target model verifies all draft tokens in parallel, offering higher efficiency than generating tokens sequentially.
Early-exit self-speculative decoding. LayerSkip (Elhoushi et al., 2024) removes the need for separate draft model My by leveraging a subset of the full model as the draft model. During decoding, only the first غ layers are executed, and the output is passed directly to the LM-Head to generate draft token. Once 'y tokens are drafted auto-regressively, they are passed through all L layers of the model for verification. LayerSkip's ' verification phase reuses the KV cache from the draft phase, resulting in lower memory usage and reduced computation compared to other SD approaches that rely on independent draft models. Nevertheless, the performance of LayerSkip is sensitive the choice of & and 7.
3 Motivating Observation
Exit layer E and speculation length y are task-dependent. We begin by investigating whether a single configuration of exit layer E and speculation length Y can generalize across tasks. To this end, we perform grid search to identify the optimal (E, 7) for each task, selecting the configuration that maximizes decoding speed (measured in tokens per second). We then apply each task-specific configuration to all other tasks to assess the generalizability of these settings. Figure 1(a) shows that applying a configuration tuned for one task to another often degrades performance. For instance, using the language modeling configuration for summarization reduces speedup from 2.65x to 1.50x, while applying the summarization configuration to language modeling drops speedup from 2.01x to 1.54x.
Even within a single task, optimal settings vary by prompt and generation stage. To investigate whether optimal YE settings remain stable across prompts within a task, or even across the course of generating output from a single prompt, we conduct a finer- grained analysis. We randomly sample two prompts from the coding task and divide the generation process into fixed-length segments of 64 tokens. For each segment, we run separate grid search to determine the optimal configuration for that specific generation window. Figure 1(b) visualizes the results as heatmaps, where the first row of three figures correspond the first prompt, and the second row corresponds the second prompt. The columns within each row correspond to the heat map generated for particular segment. The heatmaps reveal that the best-performing configuration shifts not only between prompts but also across different stages of token generation within the same prompt. Notably, the globally optimal configuration for the coding task, (8=7,7=6), often fails to achieve the best performance at finer granularity, several segments exhibit higher decoding speed with alternative settings.
These findings highlight critical insight: the optimal exit layer È and speculation length are not solely functions of the underlying task, and they also depend heavily on the specific prompt and the dynamic context during generation. In practice, the optimal configuration
3

---

Published as a conference paper at COLM 2025
(a) Cross-Task Generalization of Optimal Setting
(b) Speed (tokens/sec) Varlation Optimal Setting
L.ge7-ye
2.5
Speculation Langth
-00
Spoedup
2.16,,13
2.0
1.90
3.41
1.54
Modeling Sumarization
Reasoning
1asigrien!
aaf Saprien!
Evaluation
Layer
(a) Optimal (E, configurations of LayerSkip task-specific; there is uni- versally best setting. The legend shows configurations tuned for each task, with colors matching the corresponding task label the x-axis. Bars show the speedup achieved when applying each configuration to each task. Tasks perform best with their own configuration, while cross-task settings degrade performance. (b) Optimal (E,r) configurations vary across prompts and even within segments of a single prompt. Heatmaps show decoding speed over (E,7) grids. Shifting optimal regions motivate dynamic, context-aware adaptation.
Figure 1:
often shifts throughout the generation process, even within single prompt. This motivates our core contribution: dynamic, context-aware approach that adaptively selects & and y at fine granularity for each stage of generation, in order to fully realize the potential efficiency of self-speculative decoding.
DEL
As shown in the prior section the token generation speed function of exit layer & and speculation length A draft model with larger value of È may increase the likelihood that a drafted token is accepted, but it also introduces higher latency for drafting each token. Similarly, a larger value of y may reduce the total number of SD rounds but can also increase the number of redundant draft generations. Hence, one needs an appropriate token generation cost model to evaluate the trade-off introduced by & and y.
4.1 Defining TPL Metric
We introduce Token-per-Layer (TPL), metric that estimates the expected number of tokens generated per loaded layer in each SD round for the LayerSkip approach. TPL is defined as:
generated tokens
1 1- a d+1
TPL(e,d) =
(1)
# loaded layers
(1 az) (dl L)
where 0ę is the expected acceptance rate when using exit layer E to draft the tokens. As described in prior work (Leviathan et al., 2023) if the probability that a token proposed by the draft model with / layers accepted a, then the expected number of newly generated
1-4411
tokens when using speculation length of d is given by T Note that this equation is
simplification of (a,) We take this expected token generation length and divide that by the cost to generate these d tokens create the TPL metric.
Cost estimation. While there are various approaches to measure the cost of generating tokens, we rely on an easily measurable approximation. particular, we exploit a known observation that LLM token generation is heavily memory-bounded during inference, and the latency of generating token is proportional to the number of transformer layers loaded into the GPU (Xia et al., 2024). Loading a model layer and forwarding multiple tokens for verification takes roughly the same amount of time as loading a model layer and forwarding single token for drafting. By exploiting this observation, we estimate the cost of drafting
4

---

Published as a conference paper at COLM 2025
X
x
Eikce
LMd
Hoal
Calculate acceptance rates and confidence scores
next layer E such that masimizes TPI.
LOVs
Layer
Layer
Layer?
Tager?
Speceaciis Docsting Iound
Adjust threshold start the next SD round
(a) Draft
(b) Vertfy
(c) Forward all layers' hidden states to M-Head
Figure 2: DEL Overview. (a-b) In this LayerSkip example, Layer 2 serves as the exit layer, drafting 4 tokens uuto-reerrssvvely. During verification, 3 tokens are accepted, while x4 is rejected and becomes the bonus token. (c-d) The plug-and-plas hidden states from all layers, forwarding them through the LM-Head to generate shadow tokens used to estimate layer-wise token acceptance rates and confidence scores without incurring additional forward passes. (e-f) DEL selects the next exit layer & that maximizes Token-per-Layee (TPL) based on historical acceptance statistics, and updates the dynamic confidence threshold T to guide adaptive speculation length in the next SD round.
DEL module reuses the
and verifying d tokens at each SD round with exit layer l as (dl L), where the draft model with layers is loaded times, and the target model with L. layers is loaded once. This cost estimation used to compute the TPL metric discussed above.
4.2 Modeling Expected Acceptance Rate
To accurately estimate TPL, as defined in Eq. (1), our methodology requires modeling the expected token acceptance rate aę for each layer. Different layers yield different values of ag and incur different drafting costs, leading to different TPL values. To enable efficient and adaptive estimation of a; in LayerSkip, we make the following key observation. One can directly utilize the hidden states produced in each SD round to efficiently compute the ag for each layer, we discuss next.
Shadow tokens generation. Specifically, during the drafting and verification phases of LayerSkip (Figure 2(a) and (b)), we cache all generated hidden states across all layers, denoted as [h/..., 1i for l E L]. As shown in Figure 2(c), we then forward these hidden states through the LM-head in a single pass and apply greedy decoding to obtain [.,,..,,,(,,) producing L sets of tokens, one set for each layer. Each set represents the candidate tokens (referred to as shadow tokens) that would have been generated if the corresponding layer had been used as the draft model. These shadow tokens allow us to estimate the expected acceptance rate a for each possible exit layer.
ay formulation. Suppose layer E is used the draft model during the drafting phase of SD round We compare the tokens xx,,...,xfF
generated by the draft model, with generated by the target model. We then identify the index of the first mismatched token: u, = min((i E [0, xf; # x),,)), if a mismatch exists; otherwise, u, 7. The index u, indicates the point of divergence between the draft and target generations. Tokens beyond this point are discarded, as the context has diverged from the target model's generation. For each layer l E L), the number of matched shadow tokens within the valid context is computed as cf = Enr [(x x ,), where I[()) denotes the indicator function.
The above procedure provides of and u, for the current SD round Our goal is estimate a by tracking this information over multiple past rounds. To achieve this, we define weighted sum function S(a) E at Ialii, where E R" is the input vector and
5

---

Published as a conference paper at COLM 2025
E [0,1] is decay control factor. The expected acceptance rate is then defined as:
S(c))
0g = S(z)'
L E [1,L,
(2)
where c', и E R' represents the number of matched shadow tokens and length of the valid context for layer é across SD rounds up to r, respectively. The expected acceptance rate a, computed as weighted ratio of matched tokens to total tokens, with recent rounds weighted more heavily, As result, & reflects how well the predictions from given exit layer align with the target model over time.
4.3 Selecting Exit Layer Dynamically
With the estimated ag, we evaluate TPL(C, d) for all possible values of & [1,L) and d E [0, 4max]. The optimal setting for the next SD round is selected such that E maximized. To determine the initial values of & and y for the first SD round, we begin inference with the pre-filling step, where the target model generates initial hidden states for all layers over the prompt context evaluate TPL.
While & and y are selected at the SD round granularity based on contextual information and historical statistics, the speculation length 7 can still be suboptimal for individual tokens due to variation in token-level predictability within an SD round. To address this, we introduce dynamic draft-exiting mechanism that allows the draft process to be shortened or extended based on the confidence of individual draft tokens, enabling finer-grained and more accurate control within each SD round.
4.4 Dynamic Draft-Exiting
The probability of the top-1 draft token prediction is used as a confidence score, representing the likelihood that the token will be accepted during verification (Du et al., 2024; Xia et al., 2025). In DEL, drafting proceeds for up to dmax steps but may stop earlier if the confidence score falls below a threshold. While prior work uses fixed threshold, this approach may not generalize well across prompts. Challenging prompts might require higher thresholds, while simpler ones may tolerate lower values, and the optimal threshold can even vary within a single prompt. To address this, we adopt a dynamic threshold that adjusts based context-aware update rule.
Let fcsf,...,,csf c denote the confidence scores for the shadow tokens {xf,...,, layer L. During SD round r, the sum of confidence scores over matched tokens within the valid context from layer is defined as tcs; = - ost |i I(x = x j). Similarly, the sum of confidence scores over mismatched tokens from layer is fcst = E c5t+5 1(x * x x(41).
xf 1 } at
We then define the dynamic threshold as follows:
1 stcss
S(fcs') S(u (u E [1,
Ti 2 S(c²)
where tcs', fas' E R' denote the historical confidence score sums for matched and mis- matched tokens across SD rounds up to and c', u cf IR' represent number of matched and mismatched tokens over the same rounds. Therefore, the dynamic threshold 7; is computed as the midpoint between the weighted average confidence scores of matched and mismatched tokens, with greater emphasis on recent rounds. We provide the summarization of the DEL method in Appendix A.
Experiments
5.1 Experimental Setup
Implementation details. We evaluate DEL on top of LayerSkip (Elhoushi et al., 2024) using LLMMA-2, ILaMA-3, and CodeLLaMA models across range of tasks: CNN/DA, Maiilyllaail

---

Published as a conference paper at COLM 2025
AQuA-RAT (Reasoning)
CNN/DM (Lang. Mod.)
CNN/DM (Abs. Sum.)
XSUM (Abs. Sum.)
Speed (tokens
Overall
Models
Methods
Speedup
eTPL
Speedup
eIPL
Speedup
eIPL
Speedup
eTPL
Speedup
Vanilla
0.063
1.00x
0.063
1.00x
0.063
1.00x
0.063
1.00x
64.34
1.00x
LLaMA
-3.2-1B
PS(53-76)
LS(E3-y6)
0.146
0.158
2.16x
2.1x
0.154
0.184
2.11x
x
0.144
0.159
2.13x
1.93x
0.139
0.159
2.18x
1.89%
144.21
127.87
2.24x
1.99x
DV(33)
0.122
1.72%
0.138
1.93%
0.132
1.83%
0,099
1.43X
111.14
1.73x
DEL
0.170
2
0.198
2.56%
0.173
2.25%
0.183
2.37%
151.73
2.36x
Vanilla
0.031
1.00x
0.031
32.57
LLaMA -3-88
LS(E3-y6)
0.084
2
0.087
x
0.089
2.30x
0.077
2.08x
72.00
2.21x
(S(E3-16)
0.094
2,47
79.13
DV((3)
0.052
x.52x
0.052
1.50x
0,085
2.32
0.051
1.46x
55.40
1.70x
DEL
0.098
2.53x
0.109
2.81
0.105
2.72x
0.092
2.40
85.20
2.62
Vanilla
0.031
x.00x
0.031
1.00x
0,031
1.00X
0.031
1.00X
36,08
1.00
LLaMA -2-78
LS(E7-76)
0.064
1.83x
0.064
x
0.074
2.16%
0.070
2.09%
71.93
1.99
FSL27-76)
0.074
2.17x
0.073
x
0,080
2.33
0.074
2.25
80.48
2.23
DV(E7)
0.048
1
0.049
1
0.088
2.52
0.075
2.20x
67.86
1.88X
DEL
0.087
x
0.083
2.41x
0.097
2.75x
0.084
2.44
90.29
2
Vanilla
0.025
x.00x
0.025
x.00x
0.025
1.00x
0.025
1.00
27.08
1.00x
LLaMA -2-13B
LS(E7-14)
FS(E7-14)
0.062
0.053
x
x
0.054
0.049
2
1.89
0.049
0.048
1.72x
1.77x
0,051
1.89x
1.87
53.99
50.68
1.99
1.87
DV(77)
0.040
T
0.0G8
TA5x
0.038
1.36x
0.038
1.41x
38.68
1.43x
DEL
0.063
x
0.056
2.11%
0.067
2.29x
0,054
1.96x
58.39
2.16x
Vanilla
0.013
x
0.013
1.00x
0.013
x.00
0.013
1.00
9.75
1.00x
LLaMA -2-70B
PS(89-76)
LS(29-76)
0.036
0.033
2.76x
x
0.026
0.027
2
1.98x
0.028
0.029
2.22x
0.026
2.00x
1.99x
20.90
22.08
2.26x
DV(89)
0.021
1.59X
0.019
1.50X
1.51X
1.51X
14.86
1.52x
DEL
0.038
2.84
0.028
2.15%
0.035
2.57X
0.028
2.12%
23.49
2.41x
Table 1: Performance comparison of DEL against static and dynamic speculative decoding baselines on text generation tasks. We report empirically observed Liken-per-Layer (eTPL), per-task speedup, and overall decoding speed. The best results are bolded and the second- best are underlined. DEL consistently achieves the highest speedups by up 2.84 across different models and tasks.
(CNN/DM) (Nallapati et al., 2016) for language modeling and summarization, XSUM (Narayan et al., 2018) for abstractive summarization, Humanfival (Chen et al., 2023) for code generation, and AQuA-RAT-CoT (Ling et al., 2017) for arithmetic reasoning.
Following prior work (Elhoushi et al., 2024; Xia et al., 2025; Zhou et al., 2024), we use 1-shot summarization for CNN/DM and 0-shot for XSUM. AQuA-RAT-CoT is evaluated in 5-shot setting. The maximum generation length is set to 512 tokens. We randomly sample 1,000 test examples per dataset, except for HumanEval and AQuA-RAT, where full test sets are used. For HvmanEval, we report both pass@1 and pass@10. Speculative sampling (Leviathan et al., 2023) is used with a batch size of 1. Our implementation builds on the codebase from Elhoushi et al. (2024).
Baselines. Vanilla refers to ruto-regressive decoding without acceleration. LS(E->) uses LayerSkip with a fixed exit layer & and static speculation length y. FS(E-y) a LayerSkip variant with dynamic speculation. It initializes the speculation length to 7, then adjusts it using finite-state controller: it increases by if all draft tokens are accepted in the previous round, and decreases by 1 if any are rejected (Liu et al., 2025). DV(E) is another LayerSkip variant that applies the Draft Verify method (Zhang et al., 2024), adjusting speculation length dynamically based on a confidence threshold to maintain a target acceptance rate. For all baselines, and y are selected via grid search to maximize overall speedup. Detailed experimental setups and results are provided in Appendix B and C.
Evaluation metrics. We report the empirically observed Token foken-per-Layer (eTPL), defined as the total number of generated tokens divided by the number of loaded layers, averaged across all prompts. We also report decoding speed (tokens/s) and wall-time speedup, computed the relative increase in average inference throughput compared to the auto- regressive baseline under the same settings (Zhang et al., 2024). For output quality, we use ROUGE-2 (Ganesan, 2018), implemented via the TerchMetrics library (Detlefsen et al., 2022)
7

---

Published as a conference paper at COLM 2025
Models
CodeIlaMA-7B
a
HumanEval (pess@1)
HumanEval (pass@10)
HumanEval (pass81)
HumanEval (paas810)
Methods
eTPL
Speedup
eTPL
Speedup
ROUGE-2
Methods
eTPL
Speedup
eTPL
Speedup
ROUGE-2
Vanilla
0.031
100x
0.001
1.00x
0.093
Vanilla
0,021
1,0x
0.021
1.00x
0.0%
18(87-76)
0.063
1.96x
0.04
1.60x
1S(89-76)
0.042
1.86x
0.038
1.64x
FS187-16)
0.066
2,05x
0.057
1.70x
0.093
FS(89-)6)
0.045
1,99x
0.039
1.70x
0.093
(V()7)
0.069
2.08
0.048
143x
0.092
DV(E9)
0,042
L86x
0.002
143x
0.099
DEL
0.077
2,26x
0.062
1.73X
0.083
DEL
0.051
2,12%
0.045
1.79x
0.101
Table 2: Comparison of DEL and baselines on code generation tasks using CadeLL
and 34B. Pass@1 uses greedy decoding; pass@10 uses sampling with temperature 0.6. We report empirical Token-per-Layer (eTPL), speedup, and ROUGE-2. Best results are bolded; second-best underlined. DEL reliably achieves the highest speedups across both decoding methods. The results indicate consistent output fidelity across methods, including DEL.
5.2 Overall Speedup Results
Table 1 shows that DEL consistently outperforms both static (LS) and dynamic (FS, DV) baselines across LLaMA model sizes and text generation tasks. DEL achieves speedup of up to 2.84x on the AQuA-RAT task with the LLaMA-2-70B model and delivers overall speedups of 2. 2.16x to 2.62 across all models and tasks over vanilla auto-regressive decod- ing. On LLaMA-2-7B, DEL reaches an overall speedup of 2.50x, exceeding the best baseline (FS) at x.23x by 0.27x (a 12% speed gain). These gains are driven by higher e'IPL, reflecting more efficient layer utilization. DEL maintains strong performance across diverse tasks, including reasoning, summarization, and language modeling, and generalizes well from 1B to 70B models, demonstrating scalability across different tasks and model sizes.
Table reports results on code generation with CodeLLaMA-7B and 34B. DEL achieves the highest speedups under both greedy (pass@1) and compling-based (pass@10) 00) decoding, reaching up to 2.26x on 7B and 2.12 on 34B. In all cases, obtains the highest eTPL. con- firming its token-level effficiinccc. For greedy decoding, DEL produces exact token matches with the target model, ensuring identical outputs. In the sampling setting, speculative decoding theoretically preserves the target model's output distribution, which is supported by the ROUGE-2 scores in Table 2. These results confirm that DEL maintains output fidelity, scales effectively with model size, and remains robust across different decoding strategies.
5.3 In-depth Analysis
Ablation study of DEL. DEL includes two key components: dynamic exit layer selection and dynamic draft-exiting. To assess their individual contributions, we conduct an ablation study on LLaMA2-7B across all tasks (Figure 3(a)). We evaluate three variants: DEL-E disables dynamic draft-exiting and selects both the exit layer and speculation length solely based on the dynamic exit layer selection mechanism (i.e., optimized for TPL). DEL-y fixes the exit layer and applies dynamic draft-exiting to adapt the speculation length y at runtime. The fixed exit layer in DEL-- is chosen based on the best- posst-erforming baseline setting from Table 1. Full DEL, which combines both components, achieves higher speedup than either variant alone, confirming that both contribute to performance gains. DEL-E-lim enables both components, but restricts exit layers to 3 < E < 32. The rationale for this study to explore potential missed opportunities when exit layers are constrained. Even early layers (i.e., layers less than 3) can sometimes yield efficient drafts, and restricting their use reduces potential gains. Thus full flexibility in dynamically selecting exit layers is beneficial.
Runtime and memory breakdown. Recall that DEL forwards cached hidden states from all layers through the LM-Head to compute TPL. We quantify the runtime and memory overheads of this additional computation. Figure 3(b) shows the runtime breakdown for different speculation lengths on LLaMA-2-7B with E-8. The reported runtime is measured after the drafting, verification, and the DEL module has completed processing the corresponding y tokens. In terms of memory, DEL adds only 0.22%-2..66% over vanilla decoding, with minimal overhead compared to LayerSkip's 0.50%~1.42%% These results show DEL is lightweight and easy to integrate. More details are provided in Appendix D.1.
8

---

Published as a conference paper at COLM 2025
Wall-clook Time
0.25
Speedup
2.5
2.182.16.11
90144
2.43
0:20
2.0
1.8
Datasets
Speculation Length
Figure (a) Ablation on LLaMA2-7B shows both dynamic exit layer selection and dynamic draft-exiting components improve performance. (b) Runtime breakdown across shows DEL adds minimal overhead.
Evolution of confidence threshold and exit layer. Figure 4(a) shows how the confidence threshold 7 evolves over 200 SD rounds across different models and datasets. DEL adapts over time, with noticeable variation, showing the importance of using dynamic rather than fixed threshold. Figure 4(b) visualizes т for a sample prompt. The light blue cloud shows the range between confidence scores of matched (top) and mismatched (bottom) shadow tokens. The wide gap makes the midpoint a meaningful threshold for dynamic draft exiting. Figure 4(c) shows how DEL selects the exit layer and tracks acceptance rate a during inference. DEL updates & across rounds to keep a high and maintain efficiency.
Acceptance Rate a 25
0.7
0.7
Exit Layez E
Threshold
0.8
LE35RG010 ZCUM
Thrushold
0.8
0.5
0.8
9.8
0.3
50
100
130
200
50
1.00
100
100
1.5
200
(a) SD Round Index
(b) SD Round Index
(c) Round Index
Figure 4: (a) Confidence threshold 7 evolves over 200 SD rounds across models and datasets. (b) T for sample prompt, with accepted (top) and rejected (bottom) token scores shown as shaded range. (c) Evolution of selected exit layer & and acceptance rate a during inference.
6 Related Works
Early studies speculative decoding (Leviathan et al., 2023; Chen et al., 2023) introduced a rejection sampling method that preserves the target model's distribution and maintains output quality. Recent work has focused identifying effective stopping criteria by monitoring confidence scores to trigger verification. Kangaroo (Liu et al., 2024a) halts speculation when the draft model's confidence falls below fixed threshold, while AdaEDL. (Agrawal et al., 2024) estimates a lower bound on token acceptance rate using draft logit entropy. SpecDec++ (Huang et al., 2024) augments the draft model with trained prediction head to estimate token acceptance rate and dynamically adjust speculation length. CaPE (Du et al., 2024) and SWIFT (Xia et al., 2025) improve speculation accuracy by using confidence scores to expand draft sequences with informative tokens. However, these methods rely on fixed thresholds that do not adapt to varying prompts or tasks. Draft & Verify (Zhang et al,, 2024) addresses this limitation by proposing an adaptive, feedback- thresholding method enforce fixed acceplance rate, but it requires extensive hyperparameter tuning.
Self-speculative decoding methods eliminate the overhead of serving two models by using a subset or modified version of the target model to generate draft tokens. Draft Verify (Zhang et al., 2024), SWIFT (Xia et al., 2025), and Draft on the Fly (Metel et al., 2024) reduce draft generation time by selecting intermediate layers based on Bayesian optimization. Kangaroo (Liu et al., 2024a) uses shallow sub-network as the draft model, and lightweight adapter module is trained on it to align its representations with the full model. LayerSkip (Elhoushi et al., 2024) increases early exit accuracy without auxiliary modules through

---

Published as a conference paper at COLM 2025
specialized training strategy. By performing verification with the remaining layers, it reduces memory usage and benefits from shared computation between drafting and verification. The most relevant approach to ours is S3D (Zhong & Bharadwaj, 2024), which reduces speculation cost by selecting how many mid-layers to skip, but its time-consuming offline training makes it neither plug-and-play nor easily adaptable to different models and tasks. Like prior speculative decoding methods, DEL assumes sufficient alignment between the draft and target distributions; recent works have shown that under out-of-distributton prompts, performance may degrade due to reduced acceptance rates (Hong et al., 2025; Goel et al., 2024; Liu al., 2024b).
Conclusion
We introduced DEL, plug-and-play module for LayerSkip that dynamically selects the exit layer and speculation length to maximize decoding efficiency. DEL tracks token ac- ceptance rates across layers and estimates a Token-per-Layer (TPL) metric, leveraging context-aware feedback to identify the optimal configuration for each speculative decoding round. Extensive evaluations across models and tasks show DEL achieves overall speedups of 2.16 over auto-regressive decoding. DEL's lightweight design enables seamless integration with existing SD pipelines, enhancing performance without sacrificing quality.
Acknowledgment
We sincerely thank all the reviewers for their time and constructive comments. This material based upon work supported by NSF award number 2224319, REAL&USC-Meta center, and VMware gift. The views, opinions, and mddoo / findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies the Department of Defense or the U.S. Government.
References
Sudhanshu Agrawal, Wonseok Jeon, and Mingu Lee. Adaedl: Early draft stopping for speculative decoding of large language models via an entropy-based lower bound on token acceptance probability. arXiv preprint arXiv:2410.18351, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from feedback. arXiv preprint ar XrI::2212.080073, 2022.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, lean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculativ sampling, 2023. URL https:://rriin http://aawxi.../aabbbbsb/e wwt.oog////a/eooooooo/oooo.oo /b/////ss//.....///// 01318.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv orgg/bb/////// 14168.
Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Fal- con. Torchmetrics-ment meaiirinn reproducibility pytorch. Journal of Open Source Software, (70):400,,02)
Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: A low-hassle method to accelerate speculative decoding. ar preprint ar Xix::2442.00208, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xiv preprint at Xxx::2407.217783 2024.
10

---

Published as a conference paper at COLM 2025
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint ar arXiv:2404.16710, 2024.
Kavita Ganesan. Rouge 2.0:
Updated and improved measures for evaluation of summariza- tion tasks. Xrx preprint ar #xXx:1803.019337 2018.
Raghavy Goel, Mukul Gagrani, Wonseok Jeon, Junyoung Park, Mingu Lee, and Christopher Lott. Direct alignment of draft model for speculative decoding with chat-fine-tuned llms. arviv preprint ar 0rXit:2403.008. 20000008,, 2024.
Fenglu Hong, Ravi Raju, Jonathan Lingjie Li, Bo Li, Urmish Thakker, Avinash Ravichandran, Swayambhoo Jain, and Changran Hu. Training domain draft models for speculative decoding: Best practices and insights. arXivv preprint ar arXin:2503.0780; 2025.
Kaixuan Huang, Xudong Guo, and Mengdi Wang. Specdec++: Boosting speculative decod- ing via adaptive candidate lengths. ar Xiv preprint ar XiX::22405.1971, 2024.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by ratio- nale generation: Learning to solve and explain algebraic word problems. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 158-167, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/11/P17-0000 URL http:s///cclantholoy..rrggg
Fangcheng Liu, Yehui Tang, Zhenhua Liu, Yunsheng Ni, Kai Han, and Yunhe Wang. Kangaroo: Lossless self-speculative decoding via double early exiting. ar preprint arXit:2404.18911 2024a.
Jiesong Liu, Brian Park, and Xipeng Shen. A drop-in solution for on-the-fly adaptation of speculative decoding in large language models, 2025. URL https: //tp://////eeeeeeww/ net/
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024b.
Michael R. Metel, Peng Lu, Boxing Chen, Mehdi Rezagholizadeh, and Ivan Kobyzev. Draft on the fly: Adaptive self-speculative decoding using cosine similarity. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 2267-2272, Miami, Florida, USA, November 2024. Associ- ation for Computational Linguistics. doi: 10. 10.18653/v/ 20244.2.422 ai0.44.finings-emml...... URL https://acie https://aclanthology. org org/2024. findings-emnlp... 124/
Microsoft. Copilot. https://copilot.. wtt.llott..iooooooo www.mmmcccoot.ommmmmmm com/, 2023. Accessed: 2025-02-05.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequeeec RNNs and beyond. In Stefan Riezler and Yoav Goldberg (eds.), Proceedings of the 20th SIGNLL Conference Computational Natural Language Learning, pp. 280-290, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/vil ///////1/1111....... URL https:
//aclanthology. org/K1 1028/.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Pp. 1797-1807, Brussels, Belgium, Oetober-Novemble 2018, Association for Computational Linguistics. doi: 10.18653 v..18633/v1/D18-220 1206. URL https: http::/:s///aclaaoooooggs org/D1 10000011111 1206/.
11

---

Published as a conference paper at COLM 2025
OpenAI. Chatgpt. http://hhat.ppeaa.ccco
Accessed: 2025-02-05.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, and et al. Gpt-4 technical report, 2024. URL petpss//arrlls http://aaawxv..o.aa///////a wtgg/bb//303./000
Elizabeth Reid. Supercharging search with generative ai. https: //blgg. hoa.llggggoolll//ouuuttt search/generatiie searcshh 2023. Accessed: 2025-02-05.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024. URL https: //tp:v/aaavv////aa org/ /ttgg////////..../ 12950.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xiv preprint arXi::2307.09288 2023.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen Ilm applications via multi-agent conversation framework. ar preprint ar arXio:2308.08155 2023a.
Yiran Wu, Feiran Jia, Shaokun Zhang, Hangyu Li, Erkang Zhu, Yue Wang, Yin Tat Lee, Richard Peng, Qingyun Wu, and Chi Wang. An empirical study on challenging math problem solving with gpt-4. arXiv preprint ar arXin:2306.01337, 2023b.
Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wen- jie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, Pp.
Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10.18653//11 20/553/1122222 http//22...innnn---/iaaa findings-adl 4...nng-acl.456.. URL findings-acl. acl. 456/.
0r2/
Heming Xia, Yongqi Li, Jun Zhang, Cunxiao Du, and Wenjie Li. SWIFT: On-the-fly self- speculative decoding for LLM inference acceleration. In The Thirteenth International Conference on Learning Representations, 2025. URL EKJhH5D5WA.
wttpeenreeiieeneeffrumiiiii net/forumiid
Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11263- 11282, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ v1/ (2.18633/v120024.d2 222/222.allooong66000 607. URL httt::::/aalaanhhoogg..ooo wrg/ acl-onn 607/.
Wei Zhong and Manasa Bharadwaj. S3d: A simple and cost-effective self-speculative decoding scheme for low-memory gpus, 2024. URL http::p::/arrrt..oo/////////// 20314,
Yang Zhou, Zhuoming Chen, Zhaozhuo Xu, Xi Victoria Lin, and Beidi Chen. SIRIUS Contexual sparisty with correction for efficient LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: httpss/////eeeeeee.eeeeeeee net/ forum?id=5bR211 1b2eh.
12

---

Published as a conference paper at COLM 2025
A Algorithm
Algorithm 1 summarizes self-speculative decoding with DEL, where and y are dynami- cally adjusted to maximize efficiency and optimize token generation.
Algorithm 1 Self-Speculative Decoding with DEL.
Require:
1: Target model MM with layers
2: Initialize hidden states, exit layer &, and threshold 7 by Mp pre-filling >> Initialize SD round counter Lists to store DEL statistics
3: r r+0
4:
c, tcst 4
5:
while tokens remain be generated do
6:
D <-0
b Initialize token list Draft tokens
7:
for 0 to dmax do
8:
Draft token X1+i using the first & layers with confidence score ifs then
9:
10:
Break end if
>> Early exit if low confidence
11:
12:
Append X44i to D end for
13:
14:
Verify D by M,
15:
DEL.UPDATE(?)
r+
17:
end while
18:
function DEL.
19:
11 4, = min(fi [0, f1i # 1)) mismatch, else 4, =
(xf,, (xf y cst (s/11)) () + LM-Head ( LE [1,
20:
of Em 1(x x
Count of matched tokens in valid context
22:
Append of to c and M, to 5(d')
23:
at 5(n)
Calculate acceptance rate
24:
(E, 7) = arg max (PL(E,d)
25:
Ics, the cst [(x x
D Compute true confidence scores Compute false confidence scores
26:
fcst- = E. (5 I( # x
27:
Append tes to tes and fes, to fcs
S(tesf) S(fes')
28:
Ti x== S(ee)
ct)
Dynamic threshold update
29:
Set updated exit layer and threshold v for the next SD round
30:
end function
B Detailed Experimental Setups
B.1 Models and Datasets
We evaluate DEL on LLaMA-2, LLaMA-3, and CodeLLaMA models (Touvron et al., 2023; Dubey et al., 2024; Rozière et al., 2024), finetuned for LayerSkip by Elhoushi et al. (2024). Our experiments cover various generation tasks, including language modeling, summarization, arithmetic reasoning, code generation, and translation. For language modeling and summa- rization, we use ENN/DailyMan (CNN/DM) (Nallapati et al., 2016), while XSUM (Narayan et al., 2018) is used for abstractive summarization. HumanEval (Chen et al., 2023) is used for code generation, and GSMSK (Cobbe et al., 2021) and AQuA-RAT-CoT (Ling et al., 2017) are used for arithmetic reasoning. To cover multilingual generation, we evaluate on the WMT14 De-En dataset (German to English translation). Following prior work (Elhoushi et al., 2024;
13

---

Published as a conference paper at COLM 2025
Xia et al., 2025; Zhou et al., 2024), we use 1-shot summarization for CNN/DM and 0-shot for XSUM. GSM8K and AQuA-RAT-CoT are evaluated in a 5-shot setting. HumanEval is evaluated on CodeLLaMA-7B and 34B using pass@1 (greedy decoding) and pass@10 10 (random sampling), with pass@10 results averaged over 10 runs. The maximum generation length is set to 512 tokens. We randomly sample 1,000 test examples per dataset, except for HumanEval and AQuA-RAT, where the full test sets are used. Prompt shots for CNN/DM and GSM8K are randomly sampled from their respective training sets, while AQuA-RAT uses the specific CoT prompt shots provided by Zhou et al. (2024).
B.2 Inference Setup
To evaluate DEL, we used specific hyperparameter settings. For given input prompt, DEL initializes ag by pre-filling and considers the last 32 tokens of the context to determine the optimal exit layer & and speculation length y for the first initial SD round. We apply decay factor of w = 0.95 in the weighted sum function in Section 4.2, to gradually forget information from previous SD rounds. For instance, 0.9513 = 0.5 indicates that information from 13 rounds prior is given half the importance of the current round. DEL restricts the maximum speculation length to dmax = 18 tokens. For code generation tasks using random sampling, we set the temperature to 0.6 and top p to 0.95. All experiments were conducted using PyTorch 2.2.1. Models LLLMA-3322 1---3.2111BB LLaMA-2-7B, LLL CodeLLLaMA-BB were run on a single NVIDIA A100 (SXM4-00GB) GPU with CUDA 12.8 and 16 CPU cores from an AMD EPYC 7H12 64-Core processor. CodeLLaMA-344 used two instances of the same GPU, while LLaMA-70B ran on two NVIDIA H100 (PCIe-80GB) GPUs with the same CUDA version and 16 CPU cores from an Intel Xeon 6548Y processor. To eliminate hardware variability, we ensured that the same GPU/CPU instances were used consistently for each model-dataset pair. We adopted speculative sampling (Leviathan et al., 2023) as the acceptance strategy with a batch size of 1. Our implementation is based on the codebase from Elhoushi et al. (2024).
and
B.3 Baselines
Vanilla refers to standard auto-regressive decoding without acceleration. LS(E-y) applies LayerSkip with fixed exit layer & and static speculation length y This baseline has two kinds of configurations. LS(E-y)+ uses configurations that are proposed by the original LayerSkip paper, if available. Specifically, (E=8, 7=6) is applied for HumanByal generation on LLaMA-2-7B, (E=8, 121 for CNN/DM 1-shot and XSUM summarization on LLaMA-2- 7B, (E=7, for HumanEval on LLaMA-2-13B, and (E=15, 7=4) for XSUM summarization. The second configuration kind, LS(E-r), determines the exit layer and speculation length by conducting a grid search over 10 randomly sampled inputs from a calibration dataset, with maximum generation length of 256 tokens. Each configuration in the grid search space, consisting of potentially effective combinations of exit layers and speculation lengths for the underlying model, is evaluated on these 10 samples. This process identifies the most effective configurations for the given model and task. FS(E-y) is LayerSkip variant with a fixed exit layer & and dynamic speculation length. starts with an initial Y and adjusts it using finite-state controller: increasing 7 by if all draft tokens are accepted in the previous round and decreasing it by 1 if any are rejected (Liu et al., 2025). DV(E) is another LayerSkip variant that maintains fixed exit layer & but dynamically adjusts the speculation length using the Draft & Verify strategy (Zhang et al., 2024), where the speculation length is adapted based on a confidence threshold to maintain target acceptance rate.
B.4 Evaluation Metrics
We use the following served Token-per-Layer is defined as the total number of generated (verified) tokens divided by the number of loaded layers, averaged across all prompts. For each prompt, we count the total number of loaded layers during generation and compute eTPL by dividing the number of generated tokens by this value. The final reported eTPL is the average across all prompts in the test set. Speed (tokens/s): Speed is defined the total number of generated
metrics to evaluate the performance of DEL, eTPL: empirically ob-
14

---

Published as a conference paper at COLM 2025
(verified) tokens divided by the total time (seconds) taken to generate the output for a given prompt. This value is computed for each prompt, and the reported speed is the average across all test samples. Speedup: This refers to the ratio of the decoding speed achieved by the given method to that of standard rego-regressive (Vanilla) decoding.
C Extended Exeprienmental Results
We present the detailed statistics of our main experimental results in Table 3. DEL con- sistently outperforms both static (LS) and dynamic (FS, DV) baselines across range of LLaMA model sizes and text generation tasks. On LLaMA-2-70B, DEL achieves a remark- able speedup of 2.84x on AQuA-RAT (Reasoning), outperforming the best FS configuration at 2.78x. For CNN/DM (Abstractive Summarization) on the same model, DEL attains speedup of 2.57 x, while the strongest baseline, FS(E9-y6), achieves only 2.22x.
AQuA-RAT
CNN/DM (Lang, Mod)
CNN/DM
GSMSK (Rassningg
HumanLival (Coding)
XSUM
WMT De-Ii (Traneationn
Methods
(Ressoming)
(Abs. Sum)
[Abs, Sum.)
Speed
Speedup
Speed
Specdup
Speed
Speeduup
Speed Speedup
Speed
Speedup
Speed
Specdup
Speed
Speediup
L.aMAA332-18
Vanilla
63.17
1.00x
65:15
1,00X
64.23
1.30%
63.51
1,0%
64,84
1.00%
64.83
1.00x
63.92
1.00x
LS(83-13)
118.30
1.87
121.01
1.90x
116.75
1.82x
115.52
1.81%
120.09
1.85%
116.63
1.80%
119.23
1887
15(23-75)
127.09
2.01
137.65
2,11x
124.23
1,35x
117.05
1.83x
136.
1966
122.45
1.89%
127
2,00%
LS(24-73)
108.31
1.71%
115.48
1.77%
107:13
1,87%
103.97
1.68%
109.44
1,69%
108.2
1.67%
108.7
1.70%
PS453-73)
158.71
2
163 12
2,50x
137.20 26
2,14X
152.43
2,07%
134.68
2
141.88
2.19x
150.91
2.36
78453-76)
136.30
2.16x
161
2.48%
137.05 05
2,13x
130.98
2,26
135.05
2.08
141.34 14
2.18x
151.07 07
2.38%
(SE11-3)
121.20
1.2
112 35
2.18%
66
1,89
117.15
139.74
185x
1.92
132.1
2088
DN163)
108.58
1.72
125
1.93x
117.51
1.83x
90.10
1,41%
107.83
1.66
92.43
1.45x
91.27
1.47X
()
104.71
DEL
2.26%
ULMMA-5-8B
Vanilla
32.85
1.00x
32.57
1.00%
32.68
1,00k
31.90
1,00x
31,65
1.00
32.16
1.00
32.29
1.00
1))(3-13)
65.
1.99 x
67.77
2.08%
67.81
2.07%
55.51
1.77
63.02
62.13
1.93
66.71
LS((3-76)
71.25
LS(29-76)
57.11
1.78x
68.54
47.50
1.49%
56.91
1.79x
38.37
1811
(84)3-13)
81.95
62.23
76.16
2.64%
(8013-16)
237%
75597
ZTNK
262%%
(8458-16)
2100%
2,26
210xx
1.37
1.51%
63,08
91.56
2.81
64:13
78,01
2.73x
TLaMA-2-7B
37.08
1.00
35.98
35.59
18(57-75)
67.99
1.91%
58.76
71
2.01
LS(E8-15)
65.65
1.77x
65.91
1.83x
77.96
2.19%
59.83
1.68
66.81
1
70.55
1.98%
68.27
1.92
12(68-7123
57:27
1
58.22
1611
78.67
2.21
63.64
1.34x
59.12
1.64%
67.65
1.90
62.72
1766%
F8427-96)
60.50
2.17%
78.40
2.17%
82.98
2.33M
69
1,92
72.27
2.01 x
50.00
2.25%
82.75
2.31%
(5(3-166
76.11
200x
71.72
2007X
81.24
2.37%
68.59
150X
09.21
1.92*
77.89
219%
78.23
2200
15488-y12)
74.58
2.01
72.95
2.02
82.71
2.32x
66.31
1,80
66.07
1,88 x
75.68
2.13x
76.72
2.10x
DV((7)
$1.57
1.39
51.91
1.44x
89.65
2.52
49.76
1.37%
72.05
2,00
78.51
2.20x
61.61
1.73%
LVVES)
$1.27
1.38
$4.37
1.51x
91.78
2.85%
69.26
1.36X
76.06
2.11%
76.91
2.16x
65.61
1.84x
DET
89.43
2.41
86.86
2.41%
98.10
1.75%
72.31
1.99
76.31
86.77
2.44
94.42
2.68%
LLaMA-2-13U
Vanilla
27.65 69
1.00x
27.39
1,00X
25.97
1.00%
27.96
1.00%
27.60
1.00%x
27.25
1.00x
27.84
1.00%
1S(87-93)
52.95
1911x
51.48
1,88x
03.81
1.69%
47:27
1.69
50.11
1.51
34.20
1.84
51.20
1884%
LS(27-14)
58.19
1.99
51.86
1.89%
01.61
1.72x
47.57
1.70%
90.74
1.84x
51.01
1.87
52.19
LS((15-444
43.13
1.56x
1.71X
41.81
40.91
1.48 X
41.43
(8677-93)
63.54
2.11X
1.75X
52.29
1.91x
1477%
PS457-44)
7.06
1.77
51.51
54.87
T07X
P56515-644)
1.69%
1,81
41.20
1.63%
DN167)
11.03
1
39.77
1.15%
35.09
1.36x
38.72
1.38
1/11x
38.53
1.11x
10.75
1.16x
(N(E15)
40.02
1
36.33
1.35%
50.97
1,96x
32.12
1.30x
47.02
1,73x
46.36
1.77
46.04
DEL
62.18
2.25
57.55 30
2.11
59.50
2.29
$2.96
1.89
55.51
2.01
$3.97
1.98
55
1.98%
LLaMA-2-708
9.71
1.00X
1.00x
10.51
1.00x
1.00%
18(E8-15)
24.57
2
20.21
1.93%
18.66
2.10x
17.86
1,58
20.50
1.95
19:28
1.97
19.54
2.00x
(5(69-13)
21.55
2.22x
20.00
1.91x
17:02
1,90%
17.69
1
20.07
1991
16.00
1.93%
20.55
2.10x
LS(29-76)
24.35
2.51x
20.71
1.98%
19:04
2.11x
17:41
1,53%
20.53
1.95%
19.49
1.59%
19.32
197x
PS488-76)
26.11
2,69
20.88
1.99%
19.1
2,20%
17.85
1,57%
20.24
1.90
18.85
1.95x
19.51
2.00x
#5439-13)
27.00
2.78%
21.79
2.08%
19:57
2.20x
17.88
1,88
20.47
1956
19.51
1.97%
19.98
2.4%%
#5429-76)
26.81
276%
21.98
1.0%%
2005
2.22X
17.2
1,07%
20.47
195%
19.52
2000
20.15
2.06%.
DVIES)
15.37
1,98x
15.37
1.39%
13,66
T.11
14.17
1
15.65
14.66
T.UK
17.52
1.7%%
DN(69)
15.41
1.59
15/66
1.0%%
1366
1.51
14.12
1
15.57
148
14.73
1.51%
17.38
1.7%%
DFF
27.55
2.84
22.48
2.15%
23.20
2.77
19.0
2.00
22.90
2.18 x
20.73
2.12%
20.69
2.12%
Table 3: Detailed performance comparison of DEL against static (LS) and dynamic (F'S, DV) speculative decoding baselines across multiple tasks and model sizes. We report speed (in tokens/see sec) and speedup for each method. DEL achieves the highest speedup in most cases, demonstrating consistent gains across different task types and model scales. The best results are bolded, and the second-best results are underlined.
15

---

Published as a conference paper at COLM 2025
C.1 Extended Baseline Comparisons
We compare DEL. with SWIFT (Xia et al., 2025), a recent secetttsel-peeculativee decoding method, on the HumanEval (code generation) and CNN/DailyMail (language modeling) datasets using LLaMA-2-7B and 13B models. Like DEL, SWIFT is a plug-and-play method that requires no model retraining architectural modifications. However, unlike DEL, it operates without early-exit capability. To ensure a fair comparison, we evaluate DEL on top of LayerSkip and compare its speedup gain over LayerSkip to the gain SWIFT achieves over vanilla decoding. As shown in Table 4, DEL consistently delivers larger relative improvements across all settings. These results demonstrate that DEL offers stronger acceleration benefits when built atop early-exit self-speculation, highlighting its complementarity with LayerSkip.
Models
LLaMA-2-7B
LLaMA-2-13B
HumanEval
CNN/DM
HumanEval
CNN/DM
Methods
Methods
Speedup
Gain
Speedup
Gain
Speedup
Gain
Speedup
Gain
Vanilla
1.00x 1.01 x
1.00x
Vanilla
1.00x
1.00x
SWIFT
x.01x
1.06x
0.06x
SWIFT
1.13x
0.13x
1.26x
0.26x
(S(E7-76)
1.92x
1.88x
LS(E9-y4)
1.68x
1.67
DEL
x.07x
0.15x
2.30x
0.48x
DEL
1.85x
0.17x
2.28x
0.61
Table 4: Speedup comparison of DEL and SWIFT on HumanEval and CNN/DM using LLaMA-2-7B and 13B. "Speedup" is measured relative to vanilla decoding. "Gain" de- notes improvement over LayerSkip for DEL, and over Vanilla for SWIFT. DEL consistently achieves larger relative gains than SWIFT across tasks and model sizes.
D Further Analysis
D.1 Memory Breakdown
Table 5 shows the maximum allocated GPU memory during the inference randomly selected prompt from each dataset. The experiments were conducted on LLaMA-2-7B using three decoding methods: Vanilla auto-regressive decoding, LayerSkip with exit layer and speculation length (LS(E7-76)), and DEL. For each task, the maximum GPU memory used during the entire inference process is reported, along with the memory overhead compared to Vanilla decoding. The results demonstrate that both LS and DEL introduce minimal memory overhead. LS(E7-76) incurs an overhead of 0.50%~ 1.42% 124, depending on the task, with an average overhead of 0.91% across all tasks. DEL introduces a slightly higher overhead, ranging from 0.52% to 2.26% with an overall average of 1.54%. Overall, DEL maintains minimal memory overhead relative Vanilla decoding, ensuring scalability and making it easy to integrate into various scenarios without significant resource constraints.
HumanEval (Coding)
CNN/DM
CNN/DM (Abs. Sum.)
XSUM (Abs. Sum.)
CSMSK (Reasoning)
AQuA-RAT (Reasoning)
Methods
Overall
(Lang. Mod.)
Vanilla
Memory
13.184
13.200
14.348
13.288
14:252
14.015
13.715
Overhead
LS((7-96)
Memory
13.250
13.270
14.551
13.363
14.435
14.173
13,840
Overhead
0.50%
0.53%
1.42%
0.56%
1.28%
1.13%
0.91%
DEL
Memory 13.253
13.393
14.672
13.487
14.473
14.275
13.926
Overhead 0.52%
1.46%
2.26%
1.50%
1.55%
1.86%
1.54%
Table 5: Maximum allocated GPU memory (in GB) during inference of a randomly selected prompt from each dataset using LLaMA-2-7B. Memory overhead relative to Vanilla auto- regressive decoding is shown as a percentage. DEL maintains minimal memory overhead, ranging from 0.52% to 2.26%, with an overall average of 1.54%, ensuring scalability and ease of integration across different scenarios.
16

---

Published as a conference paper at COLM 2025
D.2 Decay Control Factor w Sensitivity Analysis
We analyze the impact of different values of w on the achieved speedup by running LLaMA- 2-7B the CNN/DM Summarization task. is the decay factor used in the weighted sum function, defined in Section 4.2. This function gradually forgets information from previous SD rounds, and w controls the rate of decay. Table 6 presents the results of varying @ E {0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 1.0}. For reference, the first column (with w -) shows the baseline results for Vanilla decoding with no speculative acceleration. The table reports eTPL,, Speed (tokens/ sec), and Speedup for each @ value. The results indicate that DEL is largely insensitive to w, with speedup improving from 2.54x at w 0.5 to a peak 2.60x at 0.9 and 1.0, and remaining stable (2.57x) even at 0.95.
Metrics
w 0.5
0.6
w
0.7
@ 0.8
0.9
0.95
eTPL 0.031
0.091
0.092
0.092
0.093
0.094
0.094
0.093
Speed 35.61
90.61
91.41
91.04
92.53
92.75
91.55
92.66
Speedup 1.00
2.54x
2.57%
2.56x
2.60x
2.60x
2.57x
2.60x
Table 6: Effect of varying eTPL, speed (tokens/ sec), and speedup during inference the CNN/DM Summarization task using LLaMA-2-7B. DEL remains largely insensitive to w, achieving near-optimal speedups (up to 2.60x) across a wide range of values.
17


Reviews:
[{"review_id": "dov2nVjDT7", "reviewer": "Reviewer_wiH3", "summary": "This paper aims to improve a subclass of speculative decoding methods that employ an early exit (sub)model of the target/verifier LLM as the draft model. The paper first conducts an empirical study to assess the effect of the choice of the early exit layer $\\mathcal{E}$ and speculation length $\\gamma$ -- the number of tokens produced by the draft model before invoking the verification phase. The empirical study shows that the optimal choice of $(\\mathcal{E}, \\gamma)$ is model- and input-dependent. Motivated by this, the paper proposes an adaptive approach, namely DEL, to select $(\\mathcal{E}, \\gamma)$. Towards this, the paper introduces a metric called token-per-layer (TPL) that characterizes the utility of selecting a particular exit layer to define the draft model. The paper proposes an efficient approach to estimate the expected acceptance rate for the tokens drafted by each layer, which in turn defines the TPL metric. Subsequently, DEL selects the early exit layer by optimizing the TPL metric. As for the dynamic selection of  $\\gamma$, the paper compares the prediction confidence with a dynamic threshold to decide if the speculation should be stopped and the verification phase needs to be invoked. The paper evaluates the utility of DEL on 4 benchmarks and shows that it outperforms various baselines. The paper also performs an ablation study to showcase the value of the dynamic selection of both $\\mathcal{E}$ and $\\gamma$.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper conducts a systematic study of the effect of $(\\mathcal{E}, \\gamma)$ on early exit-based drafting strategies for speculative decoding. \n- The paper proposes DEL -- an approach to dynamically select $(\\mathcal{E}, \\gamma)$  for speculative decoding. \n- The paper presents empirical results showcasing the utility of DEL on multiple benchmarks. In addition, the ablation results clearly establish the importance of dynamically adjusting both $\\mathcal{E}$ and $\\gamma$.", "reasons_to_reject": "- Given that multiple early exit-based speculative decoding methods already present in the literature, covering the selection of the early exit layer $\\mathcal{E}$ and utilization of model confidence-based speculation lengths, the novelty of the contributions is somewhat limited.\n- The gains realized by DEL are not significantly large compared to other baselines considered in the paper. \n- The empirical evaluation in the paper can be expanded to cover more baselines, such as S3D and Eagle.", "questions_to_authors": "- How is the TPL metric different from the improvement factor considered by Leviathan et al. (https://arxiv.org/pdf/2211.17192; Theorem 3.8) in a meaningful way?\n- Could the authors elaborate on \"Through extensive evaluations across tasks and model sizes,...and **0.27× improvement** over dynamic LayerSkip baselines\"?", "extra_scores": {}}, {"review_id": "PdPFRrzQUz", "reviewer": "Reviewer_TsAY", "summary": "This paper proposes DEL (Dynamic Exit Layer), a light‑weight, plug‑and‑play controller that augments LayerSkip self‑speculative decoding with fine‑grained, context‑aware decisions about where to exit the network (number of transformer layers E) and how long to speculate (token block size γ).  DEL monitors layer‑wise acceptance rates in real time by re‑routing cached hidden states through the LM head to create “shadow” predictions, from which it derives a Token‑per‑Layer (TPL) efficiency metric and an adaptive confidence threshold for early stopping.  Across six generation tasks, four LLaMA sizes (1 B–70 B) and CodeLLaMA (7 B/34 B), DEL delivers 2.16×–2.50× overall speed‑ups over vanilla decoding and up to 0.27× over the best existing dynamic speculative baselines while maintaining output quality, all with <2.3 % extra memory and negligible latency overhead.  The manuscript is clearly written, provides intuitive motivation, thorough algorithmic detail, and ablation studies that isolate the benefit of dynamic E and dynamic γ individually.\n\nConceptually, the work is an incremental yet meaningful advance: it shows that speculative‑decoding hyper‑parameters are highly instance‑dependent and offers a practical solution that requires no retraining or extra models.  While the ideas build on LayerSkip and existing confidence‑based heuristics, the unified, on‑the‑fly optimisation of both E and γ is original, and the empirical section is broad enough to suggest significance for both research prototypes and production LLM inference stacks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- DEL is a drop‑in module that piggy‑backs on hidden states already computed by LayerSkip, adding <2 % memory and minimal time overhead, which lowers the barrier to deployment in latency‑sensitive settings.\n- On every model‑task pair tested—spanning reasoning, summarisation, language modelling and code generation—DEL secures the best or second‑best speed‑ups, with empirical Token‑per‑Layer gains confirming that efficiency, not luck, drives the results (Tables 1–2).\n- The TPL metric and decayed statistics for acceptance‑rate estimation are simple, interpretable, and validated via ablations that show each component’s contribution.\n- Experiments cover models from 1 B to 70 B parameters and both greedy and sampling decoding, demonstrating robustness across sizes and decoding regimes.\n- The paper provides pseudo‑code, hyper‑parameter details, and an open‑source link, facilitating replication and extension.", "reasons_to_reject": "- DEL implicitly assumes access to a LayerSkip‑tuned model; it is unclear how well the controller would work with off‑the‑shelf checkpoints or other self‑speculative variants.\n- Apart from pass@k on HumanEval and reporting that output distributions are “unchanged,” the paper offers no human or automatic fidelity metrics (e.g., BLEU, ROUGE, MMLU), leaving open the risk of subtle quality degradation.\n- The TPL cost model relies on a memory‑bound assumption and a fixed decay factor ω=0.95; sensitivity analyses are absent, and no theoretical guarantee links maximising TPL to minimising latency.\n- All tasks are English and transformer‑decoder based; effects on multilingual prompts, encoder–decoder models, or dialogue safety filters are unexplored.\n- The controller could oscillate or choose poor configurations on out‑of‑distribution prompts, but stability under such conditions is not studied.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WDaMq9PStK", "reviewer": "Reviewer_SrG8", "summary": "This paper proposes a novel approach to adaptively select the exit layer in LayerSkip self-speculative decoding. The core idea is that number of layers required for a good draft prediction depends on the context so far, sometimes it has to be more or less depending on how complicated so far context was. Authors propose a way of how to estimate such complexity or trade off between speed and acceptance rate. The proposed DEL algorithm is using so called shadow tokens and cached layer activation to plan which exit layer should be used for a given prompt context. Experimental results confirm that the proposed approach optimizes such tradeoff very effectively that results in practical speed ups in different target tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Self speculation is very practical algorithm, and this DEL adaptive version has all chances to be widely used in the community, so its a good contribution.", "reasons_to_reject": "* Model selection is quite outdated given LLama2 and CodeLLama models. From my understanding that choice aligns well with baseline results from LayerSkip methods, but i will be happy to see how this method confirms its efficiency with newer models such as qwen3 and llama3.", "questions_to_authors": "* Do you know if self-speculative decoding can be applied to LoRA adapter trained models? I.e. if early exit work as well when we use lower rank approximation parameters during decoding?", "extra_scores": {}}, {"review_id": "zZZatR2wR8", "reviewer": "Reviewer_GrNu", "summary": "This paper proposes DEL, a plug-and-play module, in order to help dynamically select the exit layer for drafting ($\\epsilon$) and the number of tokens drafted ($\\gamma$), two important hyperparameters in speculative decoding. The authors begin with a preliminary study, showing that the optimal choice of exit layer and the speculation length varies in different tasks. Also, the optimal settings vary by prompt and generation stage. Motivated by this, the authors propose DEL, to dynamically select $\\epsilon$ and $gamma$ by first presenting a metric TPL as an estimation of the expected number of tokens generated per loaded layer. TPL serves as the criterion for selecting the parameters for the next decoding round. Experiment verified the effectiveness of DEL when plugged into LayerSkip across different models and tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is novel and interesting. The metric TPL could be useful for future works in speculative decoding.\n- The experiments are pretty solid on a wide range of tasks and models. \n- The paper is a joy to read and well-organized.", "reasons_to_reject": "- The linked anonymous repository is invalid, which might result in concerns with the reproducibility of this paper.\n- I am mainly concerned about the adaptability of this method as a \"plug-and-play\" module, especially when the experiments are only conducted with LayerSkip. I understand that there might be limited work in the research line of early-exit, but I would be happy to see some discussions on applying this DEL module to other possible methods. Otherwise, the scope of this paper might be limited.\n- The experiments are only conducted with the Llama family. Experiments on other model architectures would be beneficial in further strengthening the experimental results.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Speculative decoding has become an active area of work and this paper proposes some innovations in that direction. Reviewers generally consider that the proposed approach for determining an exit layer is an original contribution and has a chance to be widely adopted even in combination of other speculative decoding methods. There are some reservations with respect to similar competing approaches and hence the contribution being incremental however all reviewers seem to acknowledge still some contribution and weigh mostly on the positive side. Looking at the overall picture the AC is also inclined to recommend acceptance.
Decision: Accept