Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting
Authors: Fangcheng Liu
Yehui Tang
Zhenhua Liu
Yunsheng Ni
Duyu Tang
Kai Han
Yunhe Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: speculative decoding
LLMs
self-drafting
early exiting
Primary area: natural_language_processing

Abstract: Speculative decoding has demonstrated its effectiveness in accelerating the inference of large language models (LLMs) while maintaining an identical sampling distribution. However, the conventional approach of training separate draft model to achieve a satisfactory token acceptance rate can be costly and impractical. In this paper, we propose a novel self-speculative decoding framework \emph{Kangaroo} with \emph{double} early exiting strategy, which leverages the shallow sub-network and the \texttt{LM Head} of the well-trained target LLM to construct a self-drafting model. Then, the self-verification stage only requires computing the remaining layers over the \emph{early-exited} hidden states in parallel. To bridge the representation gap between the sub-network and the full model, we train a lightweight and efficient adapter module on top of the sub-network. One significant challenge that comes with the proposed method is that the inference latency of the self-draft model may no longer be negligible compared to the big model. To boost the token acceptance rate while minimizing the latency of the self-drafting model, we introduce an additional \emph{early exiting} mechanism for both single-sequence and the tree decoding scenarios. Specifically, we dynamically halt the small model's subsequent prediction during the drafting phase once the confidence level for the current step falls below a certain threshold. This approach reduces unnecessary computations and improves overall efficiency. Extensive experiments on multiple benchmarks demonstrate our effectiveness, where Kangaroo achieves walltime speedups up to 2.04$\times$, outperforming Medusa-1 with 88.7\% fewer additional parameters. The code for Kangaroo is available at https://github.com/Equationliu/Kangaroo.

Kangaroo: Lossless Self-Speculative Decoding for Accelerating LLMs via Double Early Exiting
Fangcheng Liu!
Yehui Tang"
Zhenhua Liu Yunhe Wang
Yunsheng Ni
Duyu Tangl
Kai Hant
*Consumer Business Group, Huawei {1 (Liufangcheng3, yehui tang, kai han, yunhe wang) @huawei. com
HHuawei Noah's 's Ark Lab
Abstract
Speculative decoding has demonstrated its effectiveness in accelerating the infer- ence of large language models (LLMs) while maintaining an identical sampling distribution. However, the conventional approach of training separate draft model to achieve a satisfactory token acceptance rate can be costly and impractical. In this paper, we propose novel self-speculative decoding framework Kangaroo with double early exiting strategy, which leverages the shallow sub-netwone and the LM Head of the well-trained target L.M to construct self-drafting model. Then, the self- verificciiin stage only requires computing the remaining layers over the early-exited hidden states in parallel. To bridge the representation gap between the sub-network and the full model, we train lightweight and efficient adapter module on top of the sub-network. One significant challenge that comes with the proposed method is that the inference latency of the self-draft model may no longer be negligible compared to the big model. To boost the token acceptance rate while minimizing the latency of the self-drafting model, we introduce an additional early exiting mechanism for both single-sequence and the tree decoding scenarios. Specifically, dynamically halt the small model's s subsequent prediction during the drafting phase once the confidence level for the current step falls below a certain threshold. This approach reduces unnecessary computations and improves overall efficiency. Extensive experiments on multiple benchmarks demonstrate our effec- tiveness, where Kangaroo achieves walltime speedups up 2.04x, outperforming Medusa-1 with 88.7% fewer additional parameters. The code for Kangaroo is available https: / giihhb.. con/ Equati oul wtw.aiiaanai//a///.../.ooo
Introduction
Large Language Models (LLMs) [1, 2, 4, 6] have demonstrated remarkable performance across various natural language processing tasks, such chain-of-thouge reasoning [7] and agents [8]. Beyond model performance, recent research has also focused on inference efficiency [9, 10], particu- larly in scenarios involving the deployment of LLMs for service applications. However, constrained by the bottleneck of memory bandwidth [11], the primary latency for autoregressive decoding of
LLMs arises mainly from memory read/write operations rather than arithmetic computations, leading to inadequate parallelism. For instance, decoding Vicuna-33B [12] on four NVIDIA V100 GPUs yields throughput of only seven new tokens per second.
To address this challenge, Speculative Decoding (SD) techniques [13, 14] have been developed, aiming accelerate autoregressive decoding by verifying multiple tokens generated by draft model
"ooresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

50%
Kangaroo
Kangaroo
Medusa
REST
REST
Lookahead
Token Acceptance
40%
Medusa
Speedup Ratio
Lookahead
30%
1.4
20%
1.3
10%
1.2
1.1
1
2
MT Bench
Math Benchmarks
RAG
Summarization
Token Position
(a) Token acceptance rate on mathematical reasoning subtask. Token position "2" denotes the task predict the next-next-token.
(b) End-to-end speedup ratio on four subtasks in Spec- Bench. "Math" and "RAG" denote mathematical reason- ing and retrieval-augmented generation, respectively.
Figure 1: Comparison of various self-drafting speculative decoding methods without free mask on Spec-Bench [22] for Vicuna-7B [12]. Kangaroo outperforms all other methods w.r.f end-to-ent speedup ratio across all the four subtasks. Specifically, Kangaroo (without tree) achieves speedups of 1.68x on MT-bench [23], outperforming Medusa with 88. 38.7% fewer additional parameters.
in parallel. Given y draft tokens, SD can generate to y + new tokens within each forward pass of the big LLM. Most of existing SD methods typically train a tiny draft model from scratch on large corpus accelerate LLMs from the same series, e.g., LLaMA-68M [15] for LLaMA-7B [2]. Distillspec [16] uses knowledge distillation to better align the draft model with the target model. However, the training of such task- specifie models can be costly [16, 17], limiting its application in real-world scenarios.
To mitigate these costs, several studies have proposed self-drafting methods that do not rely on external drafter models. LLMA [18] and REST [19] generate draft tokens by selecting text spans from references or retrieving relevant tokens from database. Notably, Medusa [20] trains multiple time-independent
-erd-Foowarrd neural Network (FFN) heads on top of the last decoder layer. Although Medusa and REST could generate draft tokens efficiently, however, the token acceptance rate is not always satisfactory (see Figure (a)).. On the other hand, focusing exclusively on the token acceptance rate without considering the latency of generating draft tokens can lead suboptimal speedup ratio. For instance, as shown in Figure 1, Lookahead [21] achieves significantly higher token acceptance rate than Medusa in the mathematical reasoning subtask. However, due its lower efficiency in the drafting phase when compared to Medusa, its end-to-end speedup ratio is slightly lower than that of Medusa (see Figure 1(b)).
In this paper, we propose novel self-speculatin decoding framework based on a double early-exiting mechanism. Rather than training the draft model from scratch, we enhance efficiency by inheriting part of the target model's knowledge through parameter sharing. Specifically, Kangaroo utilizes both the shallow sub-network and the LM Head of the target LLM to construct a self-drafting model. However, there is natural discrepancy in the representation capabilities between the shallow network and the final layer of the model. To bridge this gap, we incorporate a lightweight and efficient adapter module trained on top of the sub-network. Kangaroo's 's adapter network is streamlined yet effective, comprising only one multi-head attention layer [24] and two normalization layers [25], utilizing just 11.3% of the parameters used by Medusa's 's heads. This layer-level early-exiting strategy significantly reduces both the training and deployment costs typically associated with traditional self-speculative decoding methods that necessitate separate draft model.
Moreover, achieve an optimal balance between token acceptance rate and drafting efficiency, Kangaroo introduces an additional token-level dynamic drafting strategy via early exiting. Unlike existing methods that rely on a fixed drafting step and predefined static token tree, our approach dynamically adjusts the depth and width of the token tree based on the conditional probability distribution the self-drafting model. the case of single-sequence decoding, which is a specific scenario within the token tree, the drafting phase is immediately halted if the top-1 probability of the current sample token (in the self-drafting model) falls below predefined threshold. Extensive experiments conducted on the Spec-Bench demonstrate the effectiveness of our approach. Kangaroo

---

achieves speedups of up to 2.04x, significantly outperforming Medusa-1 while using 88.7% fewer additional parameters (67M compared to 591M.). These results highlight the efficiency and potential of Kangaroo in improving decoding processes without compromising performance.
2 Related Work
Inference Acceleration of Large Language Models With the rapid development of large language
models, significant research effort has been dedicated accelerating their inference speed [26].
Techniques such as knowledge distillation [27], model compression [28], and quantization [29]
have also been widely applied in this area. However, these approaches often require additional training of the backhone substantial modifications to the model architecture. FlashAttention [9]
and vLLM [10] frameworks accelerate the inference speed of large language models by optimizing memory management. Recent efforts have explored early exiting on models like the T5 series [30, 31, 32] and decoder-only architectures [33]. However, since early exiting accelerates inference by
saving subsequent computations, it inevitably incurs the issue of performance degradation [30].
Speculative Decoding Speculative decoding has gained significant attention due its ability to accelerate the inference of LLMs while maintaining the same sampling distribution. Generally, speculative decoding [13, 14] involves finding or training [16, 34] a small draft model closely aligned with the target LLM. Consequently, recent research has focused on more convenient self self-drafting
methods. For instance, approaches like blockwise parallel decoding [35] and Medusa [20] expedite the generation of draft tokens by training multiple time- üime-independere Feedforward Neural Networks at the second top layer. Several self-drafting acceleration techniques are inspired by early exiting. Draft & Verify [36], for example, generates draft tokens by skipping intermediate redundant layers of the target L.M. While this approach could achieve high token acceptance rate, the inference latency of the "small model" is exceptionally high, which can hinder end-to-end acceleration efficiency. SPEED [37] adapts early exiting to pipelined speculative execution for transformer decoders that employ parameter sharing. There are also several works [38, 39, 40] that make improvement on Medusa by introducing time dependency among the draft tokens. Unlike our approach, which utilizes early exiting from shallow layers, these methods typically extrapolate directly from the features of the penultimate layer. For more related works on speculative decoding, we refer readers to a recent survey [22] for detailed summarization.
3 Preliminaries
In this section, we introduce background and the formulations of standard speculative decoding under greedy decoding. We use x² to denote the discrete token sequence (.I,,... ,.,) and x to represent sequence (z,,..., ,I,). Let X' be discrete space over all possible tokens in the LLM's vocabulary, we model the autoregressive process of a language model M by the conditional distributions M x²) e R|22 where | is the vocabulary size. We denote the big target language
model and the speculative small model as Mr and M., respectively.
Single-Sequence Decoding Due the memory bandwidth [11] limitations of autoregressive decod- ing in large language models, the latency to generate one new token is approximately the same as the time required parallelly infer 7 tokens. Based on this characteristic, speculative decoding [13, 14] leverages a low-cost small model to quickly generate 7 candidates
Gi+i arg max Ma ((+i-)), i 1,2, 1,2,...,77 ZE
where 4² x. Subsequently, the big model Md only needs one forward pass over (ăz, -.. 2(t7) to generate + tokens (I++1." 214++1) The accept length over this single-sequence verification is
max {i f2 2+4e + 1, 10,-.
(2)
ranging from 1 to 7 1.
3

---

Frazen Parameters
#1
Treinable
LM Head
Draft Tokens
Remaining Layers:
Verified
Norm
Hiddon States
fo
Parallel Compute Unit
S
x
Sgg
Adapter-Networt Network
A
Neapterr-eework:
A
Imput Norm
States
Shallow Sub-Network:
Mat:
Shallow Sub-Network:
Ma
$3
AG
Round 1
Round 2
The framework of Kangaroo under single-sequence verification. The adapter network A consists of only
Figure 2:
multi-head attention and two normalization layers. The self-drafting model M, = A o Ma[: will reuse the LM Head of the target LLM MM. for better alignment, where I denotes the early exit layer. To avoid unnecessary costs on more difficult tokens, M, stops drafting once the top-1 probability of the current sampled token falls below a certain threshold, e.g., (. 20:2)
< 7). Note that we will concatenate the stopped token's 's next early feature fs with all previous exited features into a parallel compute unit [fo, fi.
fa), which will be verified by the remaining layers Ma :] in parallel. Once all drafted tokens are accepted (f; z; for i 2, 3), we could start the next round with rather than 33 if we have not calculated fa in advance. The decoding on parallel unit [Ja, [4] could save the latency a single forward pass A.
Kangaroo
In this section, we present Kangaroo, a novel self-speculative decoding framework based on double early-exiting mechanism. We leverage the shallow sub-network and the LM Head of the target LLM construct a self-drafting model. Subsequently, the self-verification stage only requires computing the remaining layers over the early-exited hidden states in parallel.
4.1 Early Exiting Self- Seff-Drafting Model
Training an additional small model from scratch is often costly and impractical, thus it is worth considering sharing a portion of the parameters from the target LLM. Drawing inspiration from early exiting, we directly extract hidden states from a fixed shallow sub-network of the target LLM
ft = Ma]: ((x)), E (1,2,.... L)
where Ma]: denotes the first layers from the target model Mr. Note that there is natural representation gap between the shallow sub-network and the full model. Therefore, we train a lightweight and efficient adapter A to bridge this gap. As shown in Figure 2, the architecture of the adapter A consists of only one multi-head attention and two normalization layers. Given an exited hidden states ft, the forward pass of the adapter model A can be represented
ft ft + Mult Head (LayerNorm(f.).
where we keep the residual connection and the multi-head attention module but remove the FFN in standard transformer block. Besides the shallow sub-network, Kangaroo also reuses the LM Head of the target model to get the final conditional distribution, i.e.
Ma(a x²) Softmax (W LayerNorm(ff))

---

Accept
0.94
A...4
8-
Reject
Probabillty lity Density
Title: "Hawalian Adventures: Discovering the Islands'
Rich Culture and Natural Wonders"
onders 0.98
0.82 \n 0.99
0:99
Description 0.16
Figure 3: The difficulty of token acceptance varies across different contexts. The values the right side of the rectangular blocks repre- sent the top-1 probability of the tokens on the self-drafting model M,. Green boxes indicate accepted tokens, while red boxes represent re- jected tokens. Blue tokens signify corrections made by the big target model M,.
0.0
0.2
0.6
0.8
Top-1 Probability
Figure 4: The density of top-1 conditional proba- bility on the mathematical reasoning subtask. "Accept" denotes the top-1 confidence of ac- cepted draft tokens while "Reject" denotes the corresponding confidence of rejected tokens.
where W E IR Nx is the frozen LM Head and N denotes the dimension of hidden states. The total parameters of A comes from the four projection matrix in the Mul t Head layer and two vectors in the LayerNorm layers, ie., 4N2 + 2N = 67. M when = 4096.
Single-Sequence Self-Speculative Decoding In the drafting phase, the self-drafting model M, generates 7 new tokens autoregressively. Similar to the standard speculative decoding in Equation (1), have Se+i arg max Ma(x a(++-1) and the exited hidden states
fiti= Mo|: ((+++)), i= 1,2,,..,7.
Subsequently, the remaining layers Mall :] will in charge of verifying the concatenated early-exited hidden features in parallel, i...
glitt def Concat (Ife, fi+1. ... f(+)) R Nx(r+1,
My(a Molx x²) Softmax (W LayerNorm (Moll (((l:((((tt)t))
where W is the frozen LM Head. In the verification procedure, we keep the same as the conventional speculative decoding by comparing the top-1 candidates of Ma x²) and ue+s for
i == 1,2. ... following Equation (2).
Training of the Adapter and the full model Mr, we train the adapter A with the conventional cross-entropy loss, i.e.
bridge the representation gap between the self-drafting model M,
A" arg min Σ Σ
-Mo(x Io) Ma(x x²); xEX
where the token positions t is averaged over the whole training set and the condition probability distribution of the target LLM M₁ serves as a distillation for the self-drafting model M,'s.
4.2 Dynamic Drafting Steps via Token-Level Early-Exiting
Speculative decoding typically employs a fixed drafting step during the drafting phase, but this often leads to local optima. Fixed-step drafting can result in unnecessary time spent on more challenging samples or missed opportunities to speculate on simpler tokens. As shown in Figure 3, the difficulty of predicting the next token varies across different contextual scenarios, necessitating the design of token-wise dynamic drafting strategy.
SengleeSequence Decoding During drafting phase, we do not have access to the full target model M3 and must rely solely on the information from the self-drafting model M.. to determine whether to continue or terminate the drafting process. Fortunately, we observe strong correlation between the small model's confidence level for the current sampled token and the likelihood that the token
5

---

Kangaroo Lookahead
Medusa
REST
Kangaroo Lookahead
Medusa SpS
REST
SpS
Summarizztioon
Translation
Summarization
Translation
2.4
1.8
2.0
6
Question Answering
1.3
Multi-tum ieesatton
Question
Multi-turn Conversation
Answering
Mathernatica! Reasoning
Retriguel augmented Generation
Mathemattical Reasoning
Rotrieval augmented Generation
(a) Compression rate.
(b) End-to-end speedup ratio.
Figure 5: Comparison of various speculative decoding methods on Spec-Bense Bench [22] for Vicuna-7B. where Kangaroo outperforms other approaches in most subtasks, especially in mathematical reasoning and aurrival-augmenter generation.
will be accepted by the big target model. In Figure 4, all conditional probability values of the tokens divided into two groups. The peak values for tokens accepted by the large model skew to the right, while the conditional probability values for tokens rejected by the large model cluster at lower confidence levels. This indicates that the top-1 probability on the Kangaroo's small model is reliable metric for determining when to stop drafting. Therefore, stop drafting once the top-1 probability on the self-drafting model falls below predefined threshold 17, i.c.
max
M2(a #EX
x²) S 17.
(3)
Extension to Tree Decoding Based on the single-sequence verification procedure, the big model MA can only verify one feasible path, composed of the top-1 candidates generated by the small model M,, Tree verification technology [15, 20] enhances parallelism and improves GPU utilization by designing sparse token tree to simultaneously verify multiple feasible paths. Most current works that adopt token tree verification rely on handcrafted or pre-searched [41] static tree. We generalize the early stopping mechanism of Kangaroo in single-sequenc' decoding tree decoding, where both the tree depth and node selection at each level of the tree are token- wise dynamic.
We define the first draft token from M, as the root node. To increase the token acceptance rate, we feed the Top-K² most probable tokens into the draft model M,, parallel at each drafting step. The confidence score of child node is computed as the product of its conditional probability in M, and its parent's conditional probability, Formally,
c(r:)
II
c(r,),
xy ay
where Path(root, .z.) denotes the path from root to node æ4, c(x;) is the probability confidence M, and c(root) = 1. At the first level, we begin with Top-K M,, root). For level 2 2, we consider the Top-K child nodes for each token in the previous level's Top-K tokens. In this way, the tokens with the Top-K largest confidence score will be selected from Top-K x Top- candidates. For the selected Top-K tokens, their parent nodes will be added into the token tree T directly. For tokens the previous level's Top-K set without any child node selected, half of those with the lowest confidence scores pruned from T. This process continues until the size [T] surpasses predefined maximum or the highest confidence score at the current level falls below a threshold v).
Thhe zero of the token tree. "Single-sqquent sequence decoding is special when Top-K equals

---

Table I: Speedup comparison of various speculative decoding methods Spec-Bench [22] for
Vicuna [12]. Speedup is the walltime speedup ratio and CR denotes the compression rate.
Translation
QA
Summarization
Math
RAG
MT Bench
Method
Avg.
CR Speedup
CR
Speedup
CR
Speedup
CR
Speedup
CR
Speedup
CR
Speedup
Lookahead [2]]
1.24
1.15x
1,56
1.21x
1.53
1.27x
1.66
1.51x
149
1.19x
1.70
1.40x
1.29x
Medusa wio Tree
1.58
1.41x
1.50
1,34x
1.49
1.32x
1.73
1.54x
1.51
1.29x
1.76
1,55x
1.41x
REST [19]
1.54
1.26x
1.91
1.63x
1.64
1.33x
1.53
1.23x
192
1.46x
2.00
1.58x
1.43x
7B
Kangaroo woo
1.41
1.24x
1.43x
1.50x
2.14
1.61x
2.05
1.52x
2.22
1.68x
1.50x
SpS [13]
1.45
1.17x
2.16
1.55x
2.43
1.76x
2.06
1.51x
2.31
1.67x
2.33
1.69x
1.56x
Medusa [20]
2.12
1.60%
2.08
1.64x
2.01
1.41x
2.48
1887
2.09
1.37x
2.51
1.84x
1.63x
Kangaroo
1.76
1.43x
2.32
1.71x
2.31
1.68x
2.76
2.04x
2.37
1.75x
2.67
1.93x
1.72x
Lookahead [2]]
1.25
1.02x
1.39
0.99
1.50
0,98
1.94
1.24x
1.52
0.94x
1.68
1.08x
1.04x
REST [19]
1.53
1.07x
1.92
1.41x
1.66
1.14x
1.5
1.0xx
1.87
1.34x
1.98
1.36x
1.23x
Medusa [20]
2.19
1.21
2.11
1.17x
2,08
1.21x
2.59
1.41x
2.12
1.12x
2,58
1,38x
1.24x
13B Medusa win Time
1.61
1.33x
1.49
1.25x
1.53
1.25x
1.80
1.48%
1.53
1.23x
1.82
1.48x
1.34x
Kangaroo
1.45
1.18x
1.79
1.34x
2.00
1.41X
2.42
1.63x
2.16
1.40x
2.44
1.66x
1.44X
Sp8 [13]
1.44
1.21x
1.83
1.49x
2.32
1.62x
2.15
1.63x
2.43
1.61x
2.25
1.65x
1.54x
Kangaroo
1.79
1.39x
2.25
1.66x
2.41
1.57x
2.82
187x
2.49
1.68X
2.71
1.79x
1.65x
5
Experiments
Models and Benchmarks We conduct experiments on Vicuna [12] models with size of 7B and 13B. We select several speculative decoding approaches for comparison including Lookahead [21] Medusa [20], REST [19] and SpS [13] using Vicuna-68M [17] as the draft model. For fair and comprehensive comparison, we evaluate the acceleration performance with the recently proposed Spec-Bench [22], which consists of six subtasks including Multi-turn Conversation, Translation, Summarization, Question Answering, Mathematical Reasoning and Retrieval-augmented Generation.
Evaluation Metrics Speculative decoding is often evaluated using two primary metrics: walltime speedup ratio and compression rate. Given speculative decoding algorithm, execute it to generate new tokens and record the accepted tokens per forward pass of the big model as a list [|sss] where Σn 8k T. The Compression Rate (CR) is defined
[81, 82,
1
CR
Σ SK: S
which does not accurately reflect the acceptance levels of the drafting algorithm for tokens at varying distances. Thus, we propose a new metric consistent token acceptance rate CTAR((w) (w), given prefix and following window with size w, the probability that the w guessed tokens from the draft model are all accepted by the target model:
CTAR(W) = S
Σ R
1(5x 11( - w 00,
(5)
which is a decreasing function w.r.f. the window size w. We plot the empirical CTARs of several self-drafting speculative decoding algorithms the mathematical reasoning subtask of Opec-Bench in Figure 1, where we observe a trade-off between the token acceptance rate and drafting efficiency.
Training and Inference As shown in Figure 2, the big target model is frozen and the trainable parameters are the multi-head attention and two normalization layers. For Kangaroo, we train the adapter A for 10 epochs with the AdamW [42] optimizer the ShareGPT dataset following Medusa [20]. The training of the adapter A for Vicuna-7B takes around 24 hours on 8 NVIDIA V100 GPUs. During the inference stage, set L = 2 for Vicuna-7B and f = 3 for Vicuna- 13B. For the single-sequence decoding in Kangaroo, we set y = 6 and r = 0.6. For the dynamic tree decoding scenario, set Top-K K as 10, and 0.4.
5.1 Effectiveness: Comparison with other Speculative Decoding Methods
The Spec-Bench results for Vicuna families* are available at Figure and Table 1, where we can conclude that:
Additional result for Llama2- 13B-1BBBBa [2] is available Table 4.
7

---

For both 7B and 13B model sizes, Kangaroo achieves the highest average speedup across all tasks, with 1.72x and 1.65 respectively. Furthermore, both model sizes, Kangaroo with dynamic tree verification demonstrates approximately 1 12% improvement in speedup compared to single-sequence decoding version, indicating its effectiveness.
In both single-sequence and tree verification scenarios, Kangaroo outperforms Medusa across most datasets, despite Kangaroo's is additional parameters comprising only 11.3% of Medusa's heads. Kangaroo shows consistent high performance across different tasks such as QA, Math, and MT Bench, indicating its robustness and versatility.
Under single-sequence decoding, Kangaroo (67M) achieves comparable results to the SpS method using Vicuna-68M, which is specifically pre-trained from scratch. Furthermore, with tree verification, Kangaroo significantly outperforms SpS across most datasets.
5.2 Ablation Studies
To fully explore the sensitivity of different hyperparameters the Kangaroo framework, we conducted comprehensive ablation experiments focusing aspects such as the depth of shallow sub-Network ł, the early stopping threshold 77, and the architectural choices of the adapter network A. In the settings of the ablation experiments described below, we focus single-sequence decoding scenario.
The Depth of Shallow Sub-Network. The capacity of the self-drafting model M, highly depends on the depth of the shared shallow sub- network. However, selecting deeper early iting layers, such as half layers of Ms. would result in excessively high inference latency. As shown in Figure 7(a), the choice of the opti- mal early exitlayer significantly influences the trade-off between token acceptance rate and drafting efficiency. For Kangaroo, we set ( for Vicuna-7B and 1 3 for Vicuna-13B. In general cases, the early exit layer can be based on an empirical ratio derived from the target model's depth N. To determine the optimal depth for shallow sub-networks relative to the full model depth, conducted multiple comparative experiments across models with different architectures and sizes. Figure records the average speedup achieved on Spec-Bench with early exits various depths. We recommend setting & between 1 and 1 practical applications.
1.5
Walltime Speedup
1.4
1.2
Vicuna-7B
Vicuna-13B
Vicuna-338
1.0
Llama2-18--hhat
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0.16
Early-Exit Ratio F/N
Figure 6: The optimal early-exit ratio &
N.
16 10
The Architecture of the Adapter Module. In transformer block, the Feed-Forward Network (FFN) component counts for 67% of whole parameters. As shown Table removing the FFN component and sharing the LM Head from the target Large Language Model (LLM) has been proven to highly effective and efficient. Specifically, Kangaroo's 's adapter module A for Vicuna-7B achieves higher "Speeddp" of 1.50 compared to 1.41 x for Medusa, while using only about one-tenth of the additional parameters as Medusa.
Table 2: Ablation studies on the architecture of the adapter module for Vicuna-7B. "Speedup" denotes the average speedup ratio on Spec- Bench [22].
Architecture
Input IN
Attention
Post LN
FFN
Linear
Last L.N
Head
Parameters
Speedup
Medusa
x
591M
1.41 x
Kangaroo
X
/
x
67M
1,50
Kangaroo Head
/
X
/
/
198M
1.44 x
I-Layer Transformer
7
/
X
/
X
202M
1.37 x
MLP Only
x
X
2
/
/
165M
1.22 x
Dynamic Exiting v.s. Fixed Step Drafting. Kangaroo uses a static threshold to determine timing of the second early stopping, based on the observation that the confidence of draft tokens in the small model strongly correlated with their acceptance by the large model. To validate the effectiveness of dynamic drafting steps with fixed threshold, we plot the comparison for various

---

1.50
3:25
2.3
Comprestion Rate
Walltime Speedup
Compression Rate
2.2
1.40
2.75
1.35
2.50
Walltime
1.30
222
2.0
y
1.20
y
1.75
Vicuna-78
Vicuna-78
19
Y-
1.15
y-
1.5
Vicuna-11 Vic
Vicuna-138
V-
1.10
10
0.1
0.5
0.1
0.5
0.7
Exit Layer
Exit Layer
Threshold
Threshold 7
(a) Optimal exit-layer
(b) Optimal threshold v).
Figure 7: Ablation studies hyper-parameters. The compression rate and walltime speedup is averaged across all sub-benchmarks in Spec-Bench [22]. It can be seen that there is a trade-off between token acceptance the expressive power of the equivalent draft model in Kangaroo, the increased inference latency can actually hinder the overall acceleration performance of the system.
and drafting efficiency. While deeper early-exit layer & can enhance
in Figure 7(b). The fixed step strategy (1) 0) achieves the maximum compression rate, however, leading to sub-optimal end-to-end walltime speedup. Overall, the optimal threshold r) is consistent across different maximum steps. For Kangaroo, we set ? 6 and 7 0.6. Besides, the static threshold is also robust across different subtasks and architectures. We visualized the conditional distributions of the small model's Top-1 confidence across different model architectures and subtasks in Figures 8 and 9 in the appendix, where the thereshold is stable, ranging from 0.6 to 0.8.
Temperature Sampling It's well known that speculative sampling suffers from decreased speedup ratio when increasing the sampling temperature [22]. We consider two sets of comparative ex- periments ion Kangaroo for the temperature sampling case. One strategy uses the original top-1 confidence as stopping criterion, while the other uses an adjusted top-1 confidence which is com- puted by applying softmax adjusted logits with temperature scaling factor. As shown in Table it can be seen that Kangaroo still achieves speedup effect very similar to that of greedy decoding when T 0.2. Moreover, the top-1 confidence used for the second early stopping mechanism should remain unadjusted because when T < 1, the adjusted top-1 confidence tends to be overestimated, making it more difficult to trigger the early stopping mechanism.
Table 3:
Speedup comparison of various temperature T on Spec- Bench [22] for Vieuna [12]. Speedup the walltime speedup ratio and CR denotes the compression rate.
Translation
QA
Summarization
Math
RAG
Bench
Confidence
Avg.
CR
Speedup
CR
Speedup
CR
Speedup
CR
Speodup
CR
Speedup
CR
Speedup
0.0
Original
1.41
1.24x
1.87
1.43x
1.87
1.50
2.14
1.61x
2.05
1.52x
2.22
1.68x
1.50x
0.2
Original
1.41
1.23x
1.88
1.41x
1.88
1.48x
2.19
1.58x
2.01
1.50x
2.21
1.67x
1.48x
0.2
Adjusted
1.45
1.04x
1.90
1.25x
2.00
1.31
2.32
1.48x
2.18
1.40x
2.41
1.53x
1.34x
0.5
Original
1.41
1.18x
1.86
1.38x
1.88
1.4xx
2.21
1.55x
2.01
1.46x
2.23
1.62x
1.4xx
Conclusion
In conclusion, Kangaroo presents novel seff-speculative decoding framework that significantly enhances the efficiency of autoregressive decoding for large language models. By leveraging double carly-exit mechanism, Kangaroo achieves remarkable speedups compared to existing methods, with significantly fewer additional parameters. Experimental results demonstrate the effectiveness of Kangaroo in balancing the token acceptance rate and drafting efficiency, making it promising approach for accelerating inference of LLMs in real-world scenarios.
Limitations Although we introduce lightweight adapter module to bridge the gap between the shallow network and the final layer of the model, the effectiveness of Kangaroo relies on the target model and the specific task. In certain complex tasks, this representation gap may still persist.

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2330. 139771, 2023. 1, 2, 13
[3] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[4] Yunhe Wang, Hanting Chen, Yehui Tang, Tianyu Guo, Kai Han, Ying Nie, Xutao Wang, Hailin Hu, Zheyuan Bai, Yun Wang, et al. Pangu-T: Enhancing language model architectures via nonlinearity compensation. arXiv preprint arXiv:2312.17276 2023.
[5] Yehui Tang, Fangeheng Liu, Yunsheng Ni, Yuchuan Tian, Zheyuan Bai, Yi-Qi Hu, Sichao Liu, Shangling Jui, Kai Han, and Yunhe Wang. Rethinking optimization and architecture for tiny language models. arXiv preprint arXiv:2402.0279) 2024.
[6] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji. Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arxiv:2309, 16609, 2023.
[7] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[8] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang. Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):1-26, 2024.
[9] Tri Dao, Dan Fu,, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing
Systems, 35:16344-16359, 2022. 1,3
[10] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 1-626, 2023. 1,3
[11] Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint arXiv:1911.02150, 2019. 3,3
[12] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April
2023), 2(3) 2023. 1, 2, 7, 9, 13
[13] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2300.001378 2023. 1, 7, 13
[14] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. International Conference on Machine Learning, pages 19274-19286. PMLR, 2023, 3,3
10

---

[15] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang. Xinhao Cheng. Zeyu Wang. Rae Ying Yee Wong. Zhuoming Chen, Daiyaan Arfeen. Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative IIm serving with speculative inference and token tree verification. arXiv preprint arXiv:2305.09781, 2023. 2,6
[16] Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Ros- tamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspee: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08466 2023. 2, 3
[17] Sen Yang, Shujian Huang, Xinyu Dai, and Jiajun Chen. Multi-candidate speculative decoding. arXiv preprint arXiv:240.006706 2024. 7.7
[18] Nan Yang. Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. Inference with reference: Lossless acceleration of large language models. arXiv preprint arXiv: 04487, 04487 2023. 2
[19] Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. Rest: Retrieval-based speculative decoding. arXiv preprint (2Xiv:2311.08253: 2023. 2,
[20] Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple Ilm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.
2024. 2, 3, 7
[21] Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of Ilm inference using lookahead decoding. arXiv preprint arXiv:2402.00057, 2024. 7,
[22] Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint Mxii::2401.088//1 2024. 2, 3, 6, 7, 8, 9, 13
[23] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. 2
[24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 2
[25] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[26] Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, et al. A survey on efficient inference for large language models. arXiv preprini arXiv:2404. 14294, 2024. 3
[27] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2023.
3
[28] Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, and Dacheng Tao. A survey transformer compression. arXiv preprint arXiv:2402.05964, 2024. 3
[29] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099, PMLR, 2023. 3
[30] Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani. Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 15:11766-1777 2022.
[31] Sangmin Bae, Jongwoo Ko, Hwanjun Song, and Se- Se-Young Yun. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. In Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5910-5924, 2023. 3
11

---

[32] Shengkun Tang, Yaqing Wang, Zhenglun Kong, Tianchi Zhang, Yao Li, Caiwen Ding, Yanzhi Wang, Yi Liang. and Dongkuan Xu. You need multiple exiting: Dynamic early exiting for accelerating unified vision language model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 10781-10791, 2023.
[33] Neeraj Varshney, Agneet Chatterjee, Mihir Parmar, and Chitta Baral. Accelerating llama inference by enabling intermediate layer decoding via instruction tuning with lite. arXiv e-prints, pages arXiv-2310, 2023. 3
[34] Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. arXiv preprint arXiv:2310. 15141, 2023.
[35] Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018. 3
[36] Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168, 2023. 3, 13
[37] Coleman Hooper, Schoon Kim. Hiva Mohammadzadch, Hasan Genc, Kurt Keutzer, Amir Gholami, and Sophia Shao. Speed: Speculative pipelined execution for efficient decoding. arXiv preprint arXiv:2310. 12072, 2023. 3
[38] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024. 3
[39] Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Kegann and William Brandon. Hydra: Sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.055109 2024.
[40] Aonan Zhang, Chong Wang, Yi Wang, Xuanyu Zhang, and Yunfei Cheng. Recurrent drafter for fast speculative decoding in large language models. arXiv preprint arXiv:2403.09911 2024.
[41] Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint arXiv:2402.12377 2024. 6
[42] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint N0X::1771.05100,2017 7
12

---

A
Appendix
Table 4: Speedup comparison of various speculative decoding methods Spec-Bench [22] for
Llama2-13B Chat [2]. Speedup is the walltime speedup ratio and CR denotes the compression rate.
Translation
QA
Summarization
Math
RAG
Bench
Method
Avg.
CR Speedup
CR
Speedup
Speedup
CR
Speedup
CR
Speedup
CR
Speedup
Draft& [36]
2.61
1.22x
2.36
1.02x
2.84
1.13x
2.47
1.8x
2.44
1.15x
2.46
1.12x
1.12x
SpS [13]
1.63
1.26x
1.69
1.34x
1.47
1.14x
1.77
1.34x
1.81
1.32x
1.67
1.28x
1.28x
Medusa
1.85
1.53x
1.55
1.27x
1.48
1.19x
1.76
1.36x
1.54
1.25x
1.72
1.43x
1,34x
Kangaroo
1.97
1.46x
1887
140xx
1.97
1.35x
2.22
1.52x
2.05
1.36x
2.28
1.58x
1.45x
Accepn Reject
Accept Rojoct
Acrent
Reject
beskky
Protsb May Densty
0.0
0.
0:4
3.6
3.5
1.5
6.
0.2
0.6
0.5
1.5
000
0.2
0.4
0.5
0.5
1.
Prossiiiity
Top
Probability
Accwot Reject
Accest Reject
Rejert:
Prosasit Denatt
trocasilte
6.5
0.0
Top-1 Probabiilyy for Math
Top-1 Probuability for BAG
1op-2 Phobebblllyy for MTT Bercch
Figure 8: The density of top-1 conditional probability on various subtasks for Vicuna-7B [12].
Accept
Accept
Accest
Reject
Probebkity saattyyy
6/2
3.4
1.
1.0
0.4
2.0
0.4
6.6
1.
Top 1 Probobility for Transtation
Top 1 Probosiityy for Q4
Top 1 Probability
Sunvmor zodion
Reject
Reject
Probubilt Dansty
4-
Probubilty
Probiadiitt
Math
Probability RAG
10001 Probubility
Bench
Figure 9: The density of top-1 conditional probability various subtasks for -Jama2-13B-Chh [2].
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction reflect the paper's contributions and scope accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Liimitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. provide closed captions for online lectures because fails to handle technical jargon.
speech-to-ted to-tetttt-tettt system might not used reliably to
The authors should diseuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, aren't 't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms
outcome might be that reviewers discover limitations that
14

---

that preserve the integrity of the community. Reviewers will be specifically instructed penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-eeference..
All assumptions should be clearly stated or referenced in the statement any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimen- tal results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the information needed to reproduce the main experimental results are provided in the Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case large language model), releasing of a model checkpoint, other means that appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
15

---

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer:
[Yes]
Justification:
The data is open-sourced and the code implementation will be released upon accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: futtps://nipa: http://nipp...e cc cc/pucliccccccccccc
guides (codeSubii ssionP olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is am acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for
open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /ttp///npp/./a///ee ec/ public quuidedess.. CCodeSubmi ss coioooioyyly for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details are provided in the Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
16

---

7. Experiment Statistical Significance
Question: Does the paper report eπror bars suitably and conrectly defined other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The error bar is not provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g.. Normally distributed errors).
should be clear whether the error bar the standard deviation or the standard error the mean.
It is OK report 1-sigma bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the computer re- sources (type compute workers, memory, time of execution) needed to reproduce experiments?
Answer: [Yes]
Justification: The details are provided in the Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:// /tt/r//erppppppp// httppubll.//.hhcccc....eeeee
Answer: [Yes]
Justification: It is conformed.
17

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to par- ticular applications, alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strate- gies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer:[NA]
Justification:
There is no risk in the paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions access the model implementing safety filters.
18

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: CC-BY 4.0 is included for each asset Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swt thoode com/ datasetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Sub- jects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer[[NA]
Question:
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
20


Reviews:
[{"review_id": "fb7JseR6cF", "reviewer": "Reviewer_NQcw", "summary": "The authors propose a new self-speculative decoding framework, Kangaroo, for accelerating large language models (LLMs) by leveraging a double early exiting strategy. Kangaroo is able to enhance inference efficiency without the need for a separate draft model. It utilizes the shallow sub-network and the LM Head of the target LLM to construct a self-drafting model and introduces a dynamic token-level early exiting mechanism to minimize latency. The proposed approach achieves significant speedups, up to 2.04x, compared to existing methods, with substantially fewer additional parameters. The paper provides empirical validation mainly on Spec-Bench for Vicuna, showing Kangaroo achieves better wall-clock speedup.", "questions": "1. According to Figure 4, it seems there is no clear separation between \"Accept\" and \"Reject\" draft tokens. This makes choosing an appropriate threshold difficult. Do different down-streaming tasks have different favorable threshold and distribution? \n\n2. How does the proposed double early exiting mechanism differ fundamentally from existing early exiting strategies in speculative decoding? Why starting from shallow layers works better? \n\n3. How robust is Kangaroo in terms of hyperparameters under different models and tasks?", "limitations": "1. Reproducibility: the paper doesn't provide source code, which makes it difficult to reproduce the results.  \n\n2. Evaluation: the paper mainly evaluates the proposed method on Spec-Bench for Vicuna. It would be great to see more evaluation on other benchmarks and models to provide a more comprehensive comparison.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and clearly presents the proposed framework. The related works are well-discussed, and the proposed method is well-motivated.\n\n2. Novelty: the proposed token-level early exiting mechanism is interesting. It dynamically adjusts the decoding process based on the confidence levels of the predictions, which leads to better inference efficiency. \n\n3. The authors conduct extensive experiments on multiple benchmarks, providing a thorough comparison with state-of-the-art methods.", "weaknesses": "1. Insufficient explanation and comparison to existing early-exiting methods. The authors suggest that their work differs from the previous early exiting methods [1] [2] by using early exiting from shallow layers. However, the paper lacks a detailed explanation of the differences and a comparison with these methods empirically. In Table 1, the main evaluation doesn't include the comparison with these early exiting methods. It would be great to have a detailed comparison.\n\n2. Double early exiting strategy involves additional hyperparameters, such as the exiting layer and the threshold for early exiting. Although the authors provide ablation studies on the selection of hyperparameters, it would be helpful to discuss their sensitivity across different models and tasks. \n\n[1] Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168, 2023.\n[2] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, and Sophia Shao. Speed: Speculative pipelined execution for efficient decoding. arXiv preprint arXiv:2310.12072, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and answer each of these comments below.\n\n### **1. Insufficient explanation and comparison to existing early-exiting methods.** \n\nWe highlight the **fundamental differences** between Kangaroo and existing early-exiting methods in the following table:\n\n|    $\\qquad$ Method    |     $\\qquad$  $\\quad$ Early-Exiting     |     $\\qquad$ Architecture | Plug and Play |\n| :----------------: | :-----------------------------: | :-------------------: | :-----------: |\n|      Kangaroo      |         shallow layers          |      General LLM      |     True      |\n| Draft & Verify [1] |    skipping redundant layers    |      General LLM      |     True      |\n|     SPEED [2]      | Speculative Pipelined Execution | Parameter sharing LLM |     False     |\n\nSPEED specifically targets parameter-sharing (block-reuse) architectures, combining speculative decoding with pipelined execution for inference acceleration. Parameter sharing allows deeper decoders to enhance model performance.  The combination of speculative decoding and pipelined execution offers SPEED a speed advantage over larger models without parameter sharing. However, Draft & Verify and Kangaroo are architecture-agnostic and can be plugged into mainstream large language models (LLMs). Draft & Verify employs Bayesian optimization to pre-select less critical layers in the target LLM for skipping, while Kangaroo implements early exits in the continuous shallow layers of the target LLM and uses a lightweight adapter to bridge the expressiveness gap between the self-speculative small model and the large model. Due to SPEED’s limited applicability and there is no released code, we focus on a detailed performance comparison between Kangaroo and Draft & Verify.\n\n|  Model   |    $\\quad$ Method     |   Translation  |   $\\qquad$ QA  |  Summarization  |     $\\quad$  Math   |     $\\quad$   RAG     |       MT Bench       |      Avg.      |\n| :-------: | :------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------: |\n| Vicuna-7B | Draft & Verify | $1.22\\times (2.61)$  | $1.02 \\times (2.36)$ | $1.13 \\times (2.84)$ | $1.08 \\times (2.47)$ | $1.15 \\times (2.44)$ | $1.12 \\times (2.46)$ | $1.12 \\times $ |\n| Vicuna-7B |    Kangaroo    | $1.24 \\times (1.41)$ | $1.43 \\times (1.87)$ | $1.50 \\times (1.87)$ | $1.61 \\times (2.14)$ | $1.52 \\times (2.05)$ | $1.68 \\times (2.22)$ | $1.50 \\times $ |\n\n|   Model   |  Method  |  Translation    |  $\\qquad$QA   | Summarization   |    $\\quad$  Math   |  $\\quad$   RAG   |   MT Bench  |      Avg.      |\n| :-------: | :------------: | :---------: | :--: | :-----------: | :--: | :--: | :------: | :--: |\n| Llama2-13B-Chat | Draft & Verify | $1.11 \\times (2.34)$ | $1.06 \\times (2.07)$ | $1.02 \\times (2.25)$ | $1.08 \\times (2.39)$ | $1.05 \\times (2.32)$ | $1.07 \\times (2.34)$ | $1.07 \\times $ |\n| Llama2-13B-Chat |  Kangaroo | $1.46 \\times (1.97)$ | $1.40 \\times (1.87)$ | $1.35 \\times (1.97)$ | $1.52 \\times (2.22)$ | $1.36 \\times (2.05)$ | $1.58 \\times (2.28)$ | $1.45 \\times $ |\n\n> We used the official code from the Draft & Verify repository to optimize the redundant layers for Vicuna-7B and used the official optimized layers for Llama2-13B-Chat. Values outside the parentheses indicate speedup, while those inside parentheses indicate the compression rate (CR).\n\nWhile Draft & Verify achieves a higher compression rate, indicating the retained layers in the draft model better approximate the full model, the high latency of the small model limits its end-to-end speedup.\n\n### **2. Why starting from shallow layers works better?** \n\nTheoretical analysis of speculative inference [3] highlights two key factors: the accuracy of the small model and its inference latency. Although Draft & Verify benefits from higher accuracy by leveraging deeper features, its small model incurs a high cost, almost half that of the large model (compared to about 1/10 in Kangaroo), limiting its overall speedup.\n\nThe reasons for the effectiveness of shallow layers in Kangaroo may include:\n\n* The initial layers directly connect token embeddings to subsequent decoder layers, playing a critical role in feature representation. References provided by Reviewer Tuhh [4, 5] suggest that the shallow layers of LLMs are sufficient for understanding simple tasks, aligning with the requirement for speculative decoding where the draft model needs only to align with the large model on simple tokens.\n* The autoregressive adapter used in Kangaroo enhances the expressiveness of shallow layers, and its lightweight design reduces inference latency.\n\n> [3] Leviathan Y, Kalman M, Matias Y. Fast inference from transformers via speculative decoding[C]. ICML 2023.\n>\n> [4] The Unreasonable Ineffectiveness of the Deeper Layers \n>\n> [5] Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?\n\n### **3. Do different down-streaming tasks have different favorable thresholds and distribution?**\n\nAs shown in Figure 4 in the main paper, \"Accept\" and \"Reject\" draft tokens cannot be perfectly separated by a fixed threshold. However,  selecting the intersection of these distributions as the threshold allows us to retain a high proportion of tokens likely to be accepted while minimizing the proportion of tokens likely to be rejected. To explore the sensitivity of this threshold, we visualized the conditional distributions of the small model's Top-1 confidence across different model architectures and subtasks (see **Figures in the global response PDF**). The confidence distributions across different subtasks suggest that selecting $\\eta \\in [0.6, 0.8]$ is a robust choice.\n\n### **4. Robustness and sensitivity of Kangaroo's hyperparameters**\n\nPlease refer to the common concerns in the global response.\n\n### **Reproducibility**\n\nWe will make our training and evaluation code publicly available in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KRobUnaVp1", "reviewer": "Reviewer_Tuhh", "summary": "The paper presents Kangaroo, a novel self-speculative decoding framework designed to accelerate the inference of LLMs while maintaining an identical sampling distribution: a new speculative decoding method: self-speculative decoding method using the model's own subnetwork as the speculative small model. The drafting stage is dynamic based on the confidence level for the current step. To fully utilize GPU, it proposes two decoding method: single-sequence and tree decoding for the verification phase. Extensive experiments are done to demonstrate the effectiveness of the method.", "questions": "Is the optimal layer dataset specific or domain specific? The early exit strategy adopted in self-drafting here reminds me of depth of learnt knowledge/concept in LLM [1, 2]. Thus in ablation study on the depth of shallow sub-network, I wonder how general the results can be seen? Also, is it also possible that the early exit layer also becomes dynamic?\n\n\n[1] The Unreasonable Ineffectiveness of the Deeper Layers\n[2] Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This is a pretty novel speculative decoding method, smartly utilizing self-drafting by the model's own layers. It resolves both the issue of having to accommodate another drafting model by reducing extra parameters needed and the accuracy of the self-drafted model.\n2. Extensive experiments on multiple tasks and baselines\n3. The proposed tree decoding method to enhance parallelism and improve GPU utilization is also pretty interesting\n4. Clear presentation and explanation", "weaknesses": "1. This paper seems largely similar to: Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. What is the main difference?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and answer each of these comments below.\n\n### **The main difference between Kangaroo and Draft & Verify**\n\nBoth Draft & Verify and Kangaroo are self-speculative decoding algorithms that construct a small model by reusing the parameters of the original model. However, the starting points differ: Draft & Verify aims to approximate the full model by removing redundant middle layers, leaving a small model with about half the layers of the original model. This creates a trade-off between the small model's accuracy and inference latency. In contrast, Kangaroo is designed to minimize the layers of shared parameters while maintaining small model accuracy. Therefore, Kangaroo reuses only the initial shallow layers of the LLM, which are crucial for feature extraction [3]. Additionally, the lightweight adapter in Kangaroo is designed to enhance the small model's accuracy without significantly impacting latency.\n\n> [3] Tang Y, Liu F, Ni Y, et al. Rethinking optimization and architecture for tiny language models[J]. ICML, 2024.\n>\n> For the benefits of reusing the initial layers, please also refer to the question \"Why starting from shallow layers works better?\" raised by Reviewer NQcw.\n\n### **Is the optimal layer dataset specific or domain specific?**\n\nThank you for pointing out this insightful question and for the valuable references. According to [2], LLMs learn different difficulty levels of tasks at different layers. This raises the question of whether the optimal early exit layer varies across datasets (domains). To address this, we summarized the changes in speedup across different datasets with varying early exit layers for Kangaroo@Vicuna-7B, as shown in the table below:\n\n| Exit Layer $\\ell$ |    Translation      |           QA          |    Summarization     |          Math         |           RAG          |       MT Bench       |      Avg.      |\n| :---------------: | :---------: | :--: | :-----------: | :--: | :--: | :------: | :--: |\n|         1         | 1.10x | 1.35x | 1.23x | 1.43x | 1.31x | 1.49x | 1.32x |\n|         2         | **1.24x** | **1.43x** | 1.50x | 1.61x | **1.52x** | **1.68x** | **1.50x** |\n|         3         |    1.19x    | 1.41x |     **1.53x**     | **1.62x** | 1.49x |  1.63x   | 1.47x |\n|         4         | 1.12x | 1.34x | 1.47x | 1.56x | 1.44x | 1.60x | 1.43x |\n|         5         | 1.11x | 1.29x | 1.39x | 1.46x | 1.37x | 1.49x | 1.35x |\n\nIt can be seen that the optimal early exit layers for different subtasks are not exactly the same but are generally close.\n\n\n### **Is it also possible that the early exit layer also becomes dynamic?**\n\nEarly exiting methods often exit at different layers based on the difficulty of the samples [4]. These methods save inference costs by exiting earlier for easier samples. However, in self-speculative inference, the small model only needs to align with the large model on simple tokens. Pursuing higher accuracy by exiting at deeper layers increases the small model's inference cost. Therefore, we chose to exit at a fixed shallow layer to balance inference cost and framework simplicity. Nevertheless, considering different exit layers within the shallow sub-network for different tokens is a valuable direction for future research.\n\n> [4] Teerapittayanon S, McDanel B, Kung H T. Branchynet: Fast inference via early exiting from deep neural networks[C]//2016 23rd international conference on pattern recognition (ICPR). IEEE, 2016: 2464-2469.\n\nSincerely,\n\nPaper 2124 Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dpiLgSUgdM", "reviewer": "Reviewer_oUCN", "summary": "Authors proposed a new method to speed up large language model inference called self-speculative decoding. Instead of training a separate, costly draft model to maintain token acceptance rates, Kangaroo uses a shallow sub-network of the large model itself as the draft model. A lightweight adapter module is trained to bridge the representation gap between the sub-network and the full model. To improve efficiency, the method introduces an early exiting mechanism, halting predictions if the confidence level for the current token is too low. Experiments show that Kangaroo achieves up to 1.68× speedup with significantly fewer additional parameters compared to another method, Medusa-1.", "questions": "* How frequently does the top-1 probability of small model goes below the predefine threshold? Do you have some observations on the efficacy of double early exiting?", "limitations": "* Requires some hyperparameters search (e.g., exit-layer l)", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* Compared to its prior work Medusa-1, the model is extremely lightweight (88.7% less parameter) but the performance is still outperforming. \n\n* The proposed auto-regressive self-drafting mode design is novel, and low cost. Their simple adapter network design sounds like a kick. Verification on a large language model can start where the self-drafting model has stopped and this can be beneficial as well. \n\n* Self-drafting models can share KV cache thus can be memory efficient. \n\n* Well written paper.", "weaknesses": "* It will be nice to evaluate more LLM families other than Vicuna. \n\n* Evaluation is not comprehensive. It misses evaluations on double-early exiting effect – see question", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and answer each of these comments below.\n\n### **Evaluate more LLM families other than Vicuna**\n\nTo validate the robustness of Kangaroo across different architectures, we additionally trained adapter networks on Llama2-13B-Chat and Llama-3-8B-Instruct and evaluated the performance on Spec-Bench. The results are shown in the tables below:\n\n|   $\\qquad $  Method   |   Translation   | $\\qquad $ QA  |  Summarization |   $\\quad $ Math   |  $\\quad $ RAG  |  MT Bench   |   Avg.    |\n| :-------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------: |\n|    Draft & Verify     | $1.22\\times (2.61)$  | $1.02 \\times (2.36)$ | $1.13 \\times (2.84)$ | $1.08 \\times (2.47)$ | $1.15 \\times (2.44)$ | $1.12 \\times (2.46)$ | $1.12 \\times $ |\n|          SpS          | $1.26\\times (1.63)$  | $1.34\\times (1.69)$  | $1.14\\times (1.47)$  | $1.34\\times (1.77)$  | $1.32\\times (1.81)$  | $1.28\\times (1.67)$  | $1.28 \\times $ |\n| Medusa$^*$ *w/o Tree* | $1.53\\times (1.85)$  | $1.27\\times (1.55)$  | $1.19\\times (1.48)$  | $1.36\\times (1.76)$  | $1.25\\times (1.54)$  | $1.43\\times (1.72)$  | $1.34 \\times $ |\n|  Kangaroo *w/o Tree*  | $1.46 \\times (1.97)$ | $1.40 \\times (1.87)$ | $1.35 \\times (1.97)$ | $1.52 \\times (2.22)$ | $1.36 \\times (2.05)$ | $1.58 \\times (2.28)$ | $1.45 \\times $ |\n\n> Speedup comparison of various speculative decoding methods on Spec-Bench [22] for Llama2-13B-Chat. Values outside the parentheses indicate speedup, while those inside parentheses indicate the compression rate (CR).  $^*$ denotes reproduction result.  SpS takes Llama-68M as the draft model.\n\n|      $\\quad $  Model      |   $\\qquad$   Method   |   Translation   | $\\qquad $ QA  |  Summarization |   $\\quad $ Math   |  $\\quad $ RAG  |  MT Bench   |   Avg.    |\n| :-----------------: | :-----------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------: |\n| Llama-3-8B-Instruct | Kangaroo *w/o Tree* | $1.46\\times (2.10)$  | $1.49 \\times (2.21)$ | $1.47 \\times (2.33)$ | $1.61 \\times (2.51)$ | $1.38 \\times (2.44)$ | $1.64\\times (2.44)$  | $1.51 \\times $ |\n| Llama-3-8B-Instruct |      Kangaroo       | $1.57 \\times (2.32)$ | $1.62 \\times (2.43)$ | $1.61 \\times (2.61)$ | $1.92 \\times (2.95)$ | $1.87 \\times (2.85)$ | $1.93 \\times (2.87)$ | $1.75 \\times $ |\n\n\n### **How frequently does the top-1 probability of small model goes below the predefine threshold? Do you have some observations on the efficacy of double early exiting?**\n\nThe frequency with which the small model's top-1 confidence falls below the predefined threshold varies depending on the model and the evaluation dataset. For instance, in Figure 4 of the main paper, about 40% of tokens have a confidence lower than the threshold $\\eta$, but most of these tokens are \"hard tokens\" that would not be accepted by the large model. More confidence distribution graphs are available in the **Figures in the global response PDF**.\n\nThe purpose of the second early exit mechanism is to reduce the time spent on difficult tokens, allowing the model to speculate further on simpler token sequences. As shown in Figure 6(b), when the stride $\\gamma = 6$ but the dynamic early exit mechanism is not used ($\\eta = 0$), the speedup is 1.2. When $\\eta = 0.6$, the speedup increases to 1.5, indicating that the second early exit mechanism can provide approximately a 25% performance improvement.\n\n### **Requires some hyperparameters search (e.g., exit-layer l)**\n\nWe argue that in practical applications, the early exit layer $\\ell$ can be set based on an empirical ratio derived from the target model's depth. To determine the optimal depth for shallow sub-networks relative to the full model depth, we conducted multiple comparative experiments across models with different architectures and sizes. The table below records the average speedup achieved on Spec-Bench with early exits at various depths, and the final column shows the optimal ratio of the early exit layer to the total model depth.\n\n|    $\\qquad $ Model  | $\\quad $  Group 1 |    $\\quad $  Group 2  |  $\\quad $  Group 3  |   $\\quad $   Group 4   |   $\\quad $  Group 5   |  Optimal Ratio  |\n| :-----------------------: | :------------: | :------------: | :------------: | :------------: | :-------------: | :-------------: |\n|    Vicuna-7B @32 layer    | 1.32x @1 layer | 1.50x @2 layer | 1.47x @3 layer | 1.43x @4 layer | 1.35x @5 layer  | 2 / 32 = 0.0625 |\n|   Vicuna-13B @40 layer    | 1.26x @1 layer | 1.36x @2 layer | 1.44x @3 layer | 1.34x @4 layer | 1.31x @5 layer  | 3 / 40 = 0.075  |\n|   Vicuna-33B @60 layer    |       -        | 1.44x @4 layer | 1.49x @5 layer | 1.38x @6 layer |        -        | 5 / 60 = 0.083  |\n| Llama2-13B-Chat @40 layer | 1.18x @1 layer | 1.42x @2 layer | 1.45x @3 layer | 1.39x @4 layer | 1.27x @5 layer- | 3 / 40 = 0.075  |\n\nIt can be seen that deeper models (with depth $N$) have larger optimal early exit layers $\\ell$, but the ratio $\\ell / N$ is relatively consistent. This implies that the latency cost for the self-speculative small model compared to the full model is fairly stable. Therefore, we recommend setting $\\ell / N$ between 1/16 and 1/10 in practical applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MZTxKruFgg", "reviewer": "Reviewer_J1nq", "summary": "The authors introduce \"Kangaroo\" a novel self-speculative decoding framework designed to accelerate LLMs using a double early exiting strategy. This approach leverages the shallow sub-network and LM Head of the target LLM to construct a self-drafting model and employs a dynamic early exiting mechanism to enhance token acceptance rates and overall efficiency. The method shows promising results, achieving significant speedups and outperforming existing techniques with fewer additional parameters", "questions": "1. **Adapter-Network Design**:\n   - **Question**: Can you provide more detailed justification and exploration for the design of the adapter network? Specifically, how did you determine the architecture (one multi-head attention layer and two normalization layers) and its suitability across different model architectures?\n   - **Suggestion**: Include additional experiments verifying the effectiveness of the adapter network on a broader range of LLMs beyond LLama2 and Vicuna. This could enhance the generalizability of your approach.\n\n2. **Size of Shallow Network**:\n   - **Question**: What criteria or experimental validations did you use to select the size of the shallow sub-network? Is there an optimal depth, and how does it vary with different models or tasks?\n   - **Suggestion**: Provide a more rational basis or empirical analysis for the chosen size of the shallow network. Include ablation studies or sensitivity analyses showing the impact of varying the depth on performance and efficiency.\n\n3. **Task-Specific Performance**:\n   - **Question**: Why does Medusa perform better on translation tasks compared to Kangaroo, and how does Kangaroo's performance in mathematical reasoning differ? Can you elaborate on the strengths and weaknesses of each method across different tasks?\n   - **Suggestion**: Include a more detailed discussion and comparative analysis of the performance of Kangaroo and other methods across various tasks. Highlight the specific attributes that make Kangaroo excel in mathematical reasoning and discuss potential improvements for translation tasks.\n\n4. **Training Time Justification**:\n   - **Question**: How did you justify the chosen number of training epochs (10 epochs) compared to previous methods? How does this training regimen impact efficiency and performance?\n   - **Suggestion**: Provide a comparative analysis of training times and their impact on performance. Include benchmarks or references to previous methods to justify the training duration and its effectiveness.\n\n5. **Speedup Verification**:\n   - **Question**: Have you verified the speedup claims over multiple GPUs? How does the performance scale with different hardware configurations?\n   - **Suggestion**: Conduct and report additional experiments verifying the speedup over multiple GPUs and diverse hardware setups. This would strengthen the robustness and reliability of your speedup claims.\n\n6. **Experimental Results with Temperature > 0**:\n   - **Question**: How does Kangaroo perform with different temperature settings during inference? Have you evaluated the impact of varying the temperature on token acceptance rates and overall efficiency?\n   - **Suggestion**: Include experimental results and analysis with different temperature settings (e.g., temperature > 0) to provide a more comprehensive evaluation of the method. Discuss how temperature variations influence the performance and stability of Kangaroo.\n\n7. **Generalizability and Scalability**:\n   - **Question**: How does the proposed method generalize to even larger models and more diverse application scenarios? Have you explored its scalability and potential limitations in real-world deployments?\n   - **Suggestion**: Extend your experimental evaluations to include larger models and a wider range of application scenarios. Discuss any potential scalability issues and provide insights into how the method can be adapted or improved for broader applicability.", "limitations": "See Weakness.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. New Self-Speculative Decoding: The Kangaroo framework introduces a novel double early exiting strategy that effectively combines self-drafting and verification stages within the same model, significantly enhancing decoding efficiency.\n\n2. Efficiency Gains: Experimental results demonstrate substantial walltime speedups over prev methods, with Kangaroo outperforming Medusa by using 88.7% fewer additional parameters.\n\n3. Dynamic Drafting: The introduction of a dynamic early exiting mechanism tailored to both single-sequence and tree decoding scenarios ensures optimal balance between token acceptance rate and drafting efficiency.\n\n4. Comprehensive Empirical Validation: Extensive experiments on Spec-Bench provide robust evidence of Kangaroo's superior performance across multiple tasks, including mathematical reasoning and retrieval-augmented generation.", "weaknesses": "1. Adapter-Network Design: The design of the adapter network is heuristic and seems verified only on LLama2 and Vicuna models. More justification and exploration across different model architectures would enhance the generalizability of the approach. (new results for Gemma1~2, Phi3, LaMMa3, etc..)\n\n2. Size of Shallow Network: The paper does not provide a rational basis for selecting the size of the shallow network. Clear criteria or experimental validation for this choice are needed.\n\n3. Task-Specific Performance: While Medusa performs better on translation tasks, Kangaroo excels in mathematical reasoning. A deeper discussion on the strengths and weaknesses of each method across different tasks would provide more insights.\n\n4. Training Time Justification: The justification for the chosen number of training epochs (10 epochs) compared to previous methods is unclear. A comparative analysis of training times and their impact on performance would be beneficial.\n\n5. Speedup Verification: Speedup claims are tricky and should be verified over multiple GPUs to ensure robustness. More comprehensive benchmarking in diverse hardware settings is required.\n\n6. Experimental Results with Temperature > 0: The paper lacks discussion on the impact of different temperature settings during inference. Additional experimental results with temperature > 0 would provide a more complete evaluation of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and valuable suggestions. We answer these comments below. \n\n### **How did you determine the architecture and its suitability across different model architectures.**\n\nThe iterative process of designing the adapter network in Kangaroo is reflected in Table 2 of our paper, as follows:\n\n1. Initial Approach with MLP: We initially considered using a simple MLP to map shallow features to the LM head. However, this approach did not yield satisfactory results as it failed to leverage token context effectively.\n2. Transformer Layer: Next, we replaced the MLP with a single transformer layer to enhance expressive power. This improved performance but still fell short of our expectations.\n3. Removing FFN Module: Observing that the FFN module in the LLM Decoder significantly contributes to latency without leveraging token context, we removed the FFN module from the transformer and introduced a new LM Head to maintain parameter count. This strategy increased the speedup from 1.37 to 1.44.\n4. Reusing LLM’s Language Head: Finally, we found that reusing the LLM’s own language head was highly effective, significantly boosting acceleration performance.\n\nTo demonstrate the generalizability of Kangaroo’s approach (reusing shallow parameters + learning a lightweight adapter), we conducted experiments on Llama2 and Llama3 as suggested by the reviewer. The results are as follows:\n\n|   $\\qquad $  Method   |   Translation   | $\\qquad $ QA  |  Summarization |   $\\quad $ Math   |  $\\quad $ RAG  |  MT Bench   |   Avg.    |\n| :-------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------: |\n|    Draft & Verify     | $1.22\\times (2.61)$  | $1.02 \\times (2.36)$ | $1.13 \\times (2.84)$ | $1.08 \\times (2.47)$ | $1.15 \\times (2.44)$ | $1.12 \\times (2.46)$ | $1.12 \\times $ |\n|          SpS          | $1.26\\times (1.63)$  | $1.34\\times (1.69)$  | $1.14\\times (1.47)$  | $1.34\\times (1.77)$  | $1.32\\times (1.81)$  | $1.28\\times (1.67)$  | $1.28 \\times $ |\n| Medusa$^*$ *w/o Tree* | $1.53\\times (1.85)$  | $1.27\\times (1.55)$  | $1.19\\times (1.48)$  | $1.36\\times (1.76)$  | $1.25\\times (1.54)$  | $1.43\\times (1.72)$  | $1.34 \\times $ |\n|  Kangaroo *w/o Tree*  | $1.46 \\times (1.97)$ | $1.40 \\times (1.87)$ | $1.35 \\times (1.97)$ | $1.52 \\times (2.22)$ | $1.36 \\times (2.05)$ | $1.58 \\times (2.28)$ | $1.45 \\times $ |\n\n> Speedup comparison of various speculative decoding methods on Spec-Bench [22] for Llama2-13B-Chat. Values outside the parentheses indicate speedup, while those inside parentheses indicate the compression rate (CR).  $^*$ denotes reproduction result.  SpS takes Llama-68M as the draft model.\n\n|      $\\quad $  Model      |   $\\qquad$   Method   |   Translation   | $\\qquad $ QA  |  Summarization |   $\\quad $ Math   |  $\\quad $ RAG  |  MT Bench   |   Avg.    |\n| :-----------------: | :-----------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------: |\n| Llama-3-8B-Instruct | Kangaroo *w/o Tree* | $1.46\\times (2.10)$  | $1.49 \\times (2.21)$ | $1.47 \\times (2.33)$ | $1.61 \\times (2.51)$ | $1.38 \\times (2.44)$ | $1.64\\times (2.44)$  | $1.51 \\times $ |\n| Llama-3-8B-Instruct |      Kangaroo       | $1.57 \\times (2.32)$ | $1.62 \\times (2.43)$ | $1.61 \\times (2.61)$ | $1.92 \\times (2.95)$ | $1.87 \\times (2.85)$ | $1.93 \\times (2.87)$ | $1.75 \\times $ |\n\n###  **Provide a more rational basis or empirical analysis for the chosen size of the shallow network.**\n\nPlease refer to section 1.1 in the global response, where we conducted multiple comparative experiments across models with different architectures and sizes. It can be seen that deeper models (with depth $N$) have larger optimal early exit layers $\\ell$, but the ratio $\\ell / N$ is relatively consistent. This implies that the latency cost for the self-speculative small model compared to the full model is fairly stable. Therefore, we recommend setting $\\ell / N$ between 1/16 and 1/10 in practical applications.\n\nRegarding whether different tasks should have different optimal early exit layers $\\ell$, we summarized the changes in speedup across different datasets with varying early exit layers for Kangaroo@Vicuna-7B, as shown in the table below:\n\n| Exit Layer $\\ell$ |    Translation      |           QA          |    Summarization     |          Math         |           RAG          |       MT Bench       |      Avg.      |\n| :---------------: | :---------: | :--: | :-----------: | :--: | :--: | :------: | :--: |\n|         1         | 1.10x | 1.35x | 1.23x | 1.43x | 1.31x | 1.49x | 1.32x |\n|         2         | **1.24x** | **1.43x** | 1.50x | 1.61x | **1.52x** | **1.68x** | **1.50x** |\n|         3         |    1.19x    | 1.41x |     **1.53x**     | **1.62x** | 1.49x |  1.63x   | 1.47x |\n|         4         | 1.12x | 1.34x | 1.47x | 1.56x | 1.44x | 1.60x | 1.43x |\n|         5         | 1.11x | 1.29x | 1.39x | 1.46x | 1.37x | 1.49x | 1.35x |\n\nIt can be seen that the optimal early exit layers for different subtasks are not exactly the same but are generally close.\n\n### **Verifying the speedup over multiple GPUs and diverse hardware setups**.\n\nWe conducted inference on Vicuna-33B using 4 NVIDIA V100 GPUs, achieving a speed of approximately 7.6 tokens per second. With Kangaroo, using early exit at the 5th layer and training the adapter, the inference speed increased to 11.4 tokens per second, resulting in a speedup of about 1.5x.\n\nSincerely,\n\nPaper 2124 Authors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c0BaWysD4J", "reviewer": "Reviewer_xb3n", "summary": "This paper proposes a novel self-speculative decoding framework Kangaroo with a double early exiting strategy for accelerating LLM inference. It addresses the challenge of inference latency and shows effectiveness through extensive experiments, achieving significant speedups and outperforming the competitors with fewer parameters.", "questions": "See questions in the weaknesses.", "limitations": "See questions in the weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel double early exiting strategy Kangaroo, which is a unique approach to improving the efficiency of LLMs without compromising performance.\n- The framework significantly accelerates inference by reducing the computational overhead associated with autoregressive decoding in LLMs, and achieves remarkable speedups compared to existing methods, with empirical results showing up to 2.04× wall-time speedups.\n- The proposed method can achieve lossless acceleration. The method maintains the same sampling distribution as the target LLM, ensuring that the quality of the generated text is not sacrificed for speed.", "weaknesses": "- This paper introduces extension to tree decoding for Kangaroo, but the implementation details are not fully provided. More explanation of that would be helpful for readers to understand the mechanism.\n- The optimal settings for the early exit layer and the dynamic threshold η may require careful tuning, which could be resource-intensive and may not generalize across different tasks or datasets.\n- This paper focuses on Vicuna-7B and Vicuna-13B models. More experiments on a wider range of model sizes could strengthen the evidence of Kangaroo's scalability and generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and valuable suggestions. We answer each of these comments below. \n\n### **More explanation of the extension to tree decoding for Kangaroo**\n\nThank you for your valuable suggestions. We will provide a more detailed formal expression and description of this extension in the final version. Kangaroo uses a static threshold to determine the timing of the second early stopping, based on the observation that the confidence of draft tokens in the small model is strongly correlated with their acceptance by the large model. Therefore, we approximate the probability of a token being accepted by the large model using the top-1 confidence of each token in the token tree. Considering the contextual dependency of tokens in speculative decoding, we model the probability of a token being accepted by the large model as the product of the top-1 confidences from the root node to that token. This approach is inspired by similar ideas in [1]\n> Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding.\n\n### **The optimal settings for the early exit layer and the dynamic threshold $\\eta$ may require careful tuning**\n\nPlease refer to section 1.1 in the global response.\n\n###  **More experiments on a wider range of model sizes.**\n\nTo demonstrate the generalizability of Kangaroo’s approach (reusing shallow parameters + learning a lightweight adapter), we conducted experiments on Llama2 and Llama3 as suggested by the reviewer. The results are as follows:\n\n|   $\\qquad $  Method   |   Translation   | $\\qquad $ QA  |  Summarization |   $\\quad $ Math   |  $\\quad $ RAG  |  MT Bench   |   Avg.    |\n| :-------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------: |\n|    Draft & Verify     | $1.22\\times (2.61)$  | $1.02 \\times (2.36)$ | $1.13 \\times (2.84)$ | $1.08 \\times (2.47)$ | $1.15 \\times (2.44)$ | $1.12 \\times (2.46)$ | $1.12 \\times $ |\n|          SpS          | $1.26\\times (1.63)$  | $1.34\\times (1.69)$  | $1.14\\times (1.47)$  | $1.34\\times (1.77)$  | $1.32\\times (1.81)$  | $1.28\\times (1.67)$  | $1.28 \\times $ |\n| Medusa$^*$ *w/o Tree* | $1.53\\times (1.85)$  | $1.27\\times (1.55)$  | $1.19\\times (1.48)$  | $1.36\\times (1.76)$  | $1.25\\times (1.54)$  | $1.43\\times (1.72)$  | $1.34 \\times $ |\n|  Kangaroo *w/o Tree*  | $1.46 \\times (1.97)$ | $1.40 \\times (1.87)$ | $1.35 \\times (1.97)$ | $1.52 \\times (2.22)$ | $1.36 \\times (2.05)$ | $1.58 \\times (2.28)$ | $1.45 \\times $ |\n\n> Speedup comparison of various speculative decoding methods on Spec-Bench [22] for Llama2-13B-Chat. Values outside the parentheses indicate speedup, while those inside parentheses indicate the compression rate (CR).  $^*$ denotes reproduction result.  SpS takes Llama-68M as the draft model.\n\n|      $\\quad $  Model      |   $\\qquad$   Method   |   Translation   | $\\qquad $ QA  |  Summarization |   $\\quad $ Math   |  $\\quad $ RAG  |  MT Bench   |   Avg.    |\n| :-----------------: | :-----------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------: |\n| Llama-3-8B-Instruct | Kangaroo *w/o Tree* | $1.46\\times (2.10)$  | $1.49 \\times (2.21)$ | $1.47 \\times (2.33)$ | $1.61 \\times (2.51)$ | $1.38 \\times (2.44)$ | $1.64\\times (2.44)$  | $1.51 \\times $ |\n| Llama-3-8B-Instruct |      Kangaroo       | $1.57 \\times (2.32)$ | $1.62 \\times (2.43)$ | $1.61 \\times (2.61)$ | $1.92 \\times (2.95)$ | $1.87 \\times (2.85)$ | $1.93 \\times (2.87)$ | $1.75 \\times $ |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful comments and valuable feedback. We are encouraged by the positive reception of Kangaroo, as reflected in comments like ``The proposed method is well-motivated and the proposed token-level early exiting mechanism is interesting`` from Reviewer NQcw, ``This is a pretty novel speculative decoding method, smartly utilizing self-drafting by the model's own layers.`` from Reviewer Tuhh and ``Their simple adapter network design sounds like a kick. `` from Reviewer oUCN. 

Below, we briefly summarize our response to the common concerns raised by reviewers.


### **1. Robustness and sensitivity of Kangaroo's hyperparameters across different models**

Both Reviewers xb3n and oUCN raise concerns about the optimal early exit layer $\ell$ and the early-stopping threshold $\eta$ potentially requiring careful tuning. We argue that in practical applications, the early exit layer $\ell$ can be set based on an empirical ratio derived from the target model's depth, while the optimal threshold $\eta$ is robust across different model architectures and datasets.

### 1.1 The choice of the exit layer $\ell$

To determine the optimal depth for shallow sub-networks relative to the full model depth, we conducted multiple comparative experiments across models with different architectures and sizes. The table below records the average speedup achieved on Spec-Bench with early exits at various depths, and the final column shows the optimal ratio of the early exit layer to the total model depth.

|    $\qquad $ Model  | $\quad $  Group 1 |    $\quad $  Group 2  |  $\quad $  Group 3  |   $\quad $   Group 4   |   $\quad $  Group 5   |  Optimal Ratio  |
| :-----------------------: | :------------: | :------------: | :------------: | :------------: | :-------------: | :-------------: |
|    Vicuna-7B @32 layer    | 1.32x @1 layer | 1.50x @2 layer | 1.47x @3 layer | 1.43x @4 layer | 1.35x @5 layer  | 2 / 32 = 0.0625 |
|   Vicuna-13B @40 layer    | 1.26x @1 layer | 1.36x @2 layer | 1.44x @3 layer | 1.34x @4 layer | 1.31x @5 layer  | 3 / 40 = 0.075  |
|   Vicuna-33B @60 layer    |  -  | 1.44x @4 layer | 1.49x @5 layer | 1.38x @6 layer |  -  | 5 / 60 = 0.083  |
| Llama2-13B-Chat @40 layer | 1.18x @1 layer | 1.42x @2 layer | 1.45x @3 layer | 1.39x @4 layer | 1.27x @5 layer- | 3 / 40 = 0.075  |


It can be seen that deeper models (with depth $N$) have larger optimal early exit layers $\ell$, but the ratio $\ell / N$ is relatively consistent. This implies that the latency cost for the self-speculative small model compared to the full model is fairly stable. Therefore, we recommend setting $\ell / N$ between 1/16 and 1/10 in practical applications.

### 1.2 The early-stopping threshold $\eta$

Kangaroo uses a static threshold to determine the timing of the second early stopping, based on the observation that the confidence of draft tokens in the small model is strongly correlated with their acceptance by the large model. For example, Figure 4 in the main paper shows that the Top-1 confidence distribution of the speculative small model on the mathematical reasoning subtask exhibits a clear bimodal distribution. Selecting the intersection of these distributions as the threshold allows us to retain a high proportion of tokens likely to be accepted while minimizing the proportion of tokens likely to be rejected. To explore the sensitivity of this threshold, we visualized the conditional distributions of the small model's Top-1 confidence across different model architectures and subtasks (see **Figures in the global response PDF**). Interestingly, we found this threshold to be stable (ranging from 0.6 to 0.8) across various models and subtasks.

### **2. More comprehensive comparison**

Following the suggestions from Reviewer xb3n and Reviewer oUCN, we conducted experiments on a wider range of model architectures and larger models on Spec-Bench by including Llama2-13B-Chat.

|   $\qquad $  Method   |   Translation   | $\qquad $ QA  |  Summarization |   $\quad $ Math   |  $\quad $ RAG  |  MT Bench   |   Avg.    |
| :-------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------------: | :------------: |
|    Draft & Verify     | $1.22\times (2.61)$  | $1.02 \times (2.36)$ | $1.13 \times (2.84)$ | $1.08 \times (2.47)$ | $1.15 \times (2.44)$ | $1.12 \times (2.46)$ | $1.12 \times $ |
|   SpS | $1.26\times (1.63)$  | $1.34\times (1.69)$  | $1.14\times (1.47)$  | $1.34\times (1.77)$  | $1.32\times (1.81)$  | $1.28\times (1.67)$  | $1.28 \times $ |
| Medusa$^*$ *w/o Tree* | $1.53\times (1.85)$  | $1.27\times (1.55)$  | $1.19\times (1.48)$  | $1.36\times (1.76)$  | $1.25\times (1.54)$  | $1.43\times (1.72)$  | $1.34 \times $ |
|  Kangaroo *w/o Tree*  | $1.46 \times (1.97)$ | $1.40 \times (1.87)$ | $1.35 \times (1.97)$ | $1.52 \times (2.22)$ | $1.36 \times (2.05)$ | $1.58 \times (2.28)$ | $1.45 \times $ |

> Speedup comparison of various speculative decoding methods on Spec-Bench [22] for Llama2-13B-Chat. Values outside the parentheses indicate speedup, while those inside parentheses indicate the compression rate (CR).  $^*$ denotes reproduction result.  SpS takes Llama-68M as the draft model.

From the experimental results in the table, we further validate the effectiveness and generalization of Kangaroo. Compared to Table 1 in the main paper, an interesting phenomenon emerges: most algorithms applied to Vicuna-13B show lower speedup in translation tasks relative to other datasets, but when the target model is switched to Llama2-13B-Chat, the speedup significantly improves. This may be related to the different SFT data used by these models.

We will incorporate the valuable suggestions provided by the reviewers in our final version and further refine the ablation experiments and model evaluations. We sincerely hope our response addresses the concerns raised by the reviewers.

Sincerely,

Paper 2124 Authors

Meta-review: For this paper, reviewers are in general positive about the novelty of the methods (i.e., using the first few layers of the pre-trained model to build a self-drafting model for Speculative Decoding). After rebuttals, the authors have also addressed most of the concerns (e.g., novelty against Draft & Verify / SPEED, more lightweight than existing works, and e.g., speed claim on different types of GPUs).
Decision: Accept (poster)