D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models
Authors: yikun jiang
Huanyu Wang
Lei Xie
Hanbin Zhao
Chao Zhang
Hui Qian
John C.S. Lui
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Dynamic Inference
Inference Acceleration
Adaptive Computing Resource Allocation
Primary area: natural_language_processing

Abstract: Large language models have shown an impressive societal impact owing to their excellent understanding and logical reasoning skills. However, such strong ability relies on a huge amount of computing resources, which makes it difficult to deploy LLMs on computing resource-constrained platforms. Currently, LLMs process each token equivalently, but we argue that not every word is equally important. Some words should not be allocated excessive computing resources, particularly for dispensable terms in simple questions. In this paper, we propose a novel dynamic inference paradigm for LLMs, namely D-LLMs, which adaptively allocate computing resources in token processing. We design a dynamic decision module for each transformer layer that decides whether a network unit should be executed or skipped. Moreover, we tackle the issue of adapting D-LLMs to real-world applications, specifically concerning the missing KV-cache when layers are skipped. To overcome this, we propose a simple yet effective eviction policy to exclude the skipped layers from subsequent attention calculations. The eviction policy not only enables D-LLMs to be compatible with prevalent applications but also reduces considerable storage resources. Experimentally, D-LLMs show superior performance, in terms of computational cost and KV storage utilization. It can reduce up to 45\% computational cost and KV storage on Q\&A, summarization, and math solving tasks, 50\% on commonsense reasoning tasks.

D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models
Yikun Jiang', Huanyu Wang's " Lei Xie Hanbin Zhao?t, Chao Zhang', Hui Qian", John C.S. Lui
FFI Lab, Huawei Technologies Co., Ltd.
ZZejiang University *Chinese University of Hong Kong {jiangyikune kun5, (iielei70)@huaeet com httpyy/hhlloo.zaohahhiee zww.jj.nnninnnjuuuuu.o qianhui) edu. en cslui@ces kuhk edu. hk
Abstract
Large language models have shown an impressive societal impact owing to their excellent understanding and logical reasoning skills. However, such strong ability relies on huge amount of computing resources, which makes it difficult to deploy LLMS on computing resource-constrained platforms. Currently, L.LMs process each token equivalently, but we argue that not every word is equally important. Some words should not be allocated excessive computing resources, particularly for dispensable terms in simple questions. In this paper, propose novel dy- namic inference paradigm for LLMs, namely D-LLMs, which adaptively allocate computing resources in token processing. We design a dynamic decision module for each transformer layer that decides whether a network unit should be executed skipped. Morcover, we tackle the issue of adapting D-LLMs to real-world ap- plications, specifically concerning the missing KV-cache when layers are skipped. To overcome this, we propose simple yet effective eviction policy exclude the skipped layers from subsequent attention calculations. The eviction policy not only enables D-LLMs be compatible with prevalent applications but also reduces considerable storage resources. Experimentally, D-LLMs show superior performance, in terms of computational cost and KV storage utilization. It reduce up to 45% computational cost and KV storage on Q&A, summarization, and math solving tasks, 50% on commonsense reasoning tasks.
1 Introduction
Large language modeis(LLMMs), have demonstrated amazing capability on generation tasks. Ben- efiting from existing parameter-efficien finetuning strategies [30, 94, 54 66, 84], LLMs can also be adapted to specific domains with extra training costs. However, such models containing bil- lions of parameters require expensive computation costs and memory overhead at the inference stage. Thus, deploying LLMs on resource-constrainee platforms becomes challenging especially considering the ever-growing requirements for offline using, model customizing, and data pri- vacy (IXX) 14, IGG. 50. 611. 13% X3). To alleviate this problem, various inference acceleration techniques have been proposed, e.g., quantization [L] 42 57 76. [X], distillation [
65 and prun- ing [511, 52 /Z 8L 82], Among these methods, pruning is typical structural compression method reduce the participating network layers. These methods usually remove redundant neurons based on hand-crafted metrics [37] [8], when optimizing or finetuning large language models.
3y 2
'Equal contribution. *Correspondence to Hanbin Zhao. Our code is available at https: htops:///ithann com/
Jyk- LLM
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Although previous methods can decrease computational cost, many essential characteristics of natu- ral languages are ignored. First, tasks of different difficulties should not be allocated with the same computing resource. Obviously, a response to "how are you" is much easier than "explaining math theorem". More importantly, not every word in a sentence is equally important. Thus, it not nec- essary to allocate too much computing resources to non-critical tokens, i.e,, articles and punctuation marks. In this paper, we introduce a novel dynamic inference mechanism, namely D-LLMs, which can boost inference speeds by dynamically skipping redundant network components. Specifically, we assign an execution decision module for each transformer layer. Before going through a trans- former layer, tokens need to pass through the decision module to obtain execution decision. Based on the execution decision, D-LLMs would adaptively decide whether the following layer should be exccuted skipped. Thus, dispensable tokens and simple tasks would utilize much fewer layers than relevant tokens and difficult tasks. Another critical issue that should be emphasized here is how to apply such dynamic inference mechanism LLMs, e.g., LLaMA [6%]] GPT [855 588, etc. Note that it is unacceptable to retrain or finetune entire models due to the high computational overhead and massive training data. Therefore, we combine strategy with finetuning methods by inserting low-rank adaptors, LoRA [EO], for well pre-trained large language models.
With the aforementioned method, each token execution by specific network topology can success- fully reduce the computational cost at inference time. However, to deploy such dynamic inference networks for realistic scenarios, making this dynamic inference compatible with KV-cache strate- gies [[T 2 [ is a crucial challenge. Compared with static inference networks which cache all KV embeddings of previous tokens, dynamic inference networks selectively skip layers would lead to KV-cache misses. To tackle this issue, we design a simple yet effective eviction strategy on KV-cache in our proposed D-LLMs framework. Specifically, culated transformer layers in previous tokens and simplify the regular causal masks to sparse ones. For example, if given token skips a transformer layer the inference time, would not be included in self-attention calculation of subsequent tokens. This way, the attention mask in each layer is also dynamically changed according the execution decisions of preceding tokens.
evict the KV embeddings of uncal-
To summarize, the main contributions of our paper are concluded as follows:
We propose a novel inference framework for LLMs, termed D-I.LMs, which can significantly reduce computational resource requirements with an adaptive allocation scheme. We design de- cision modules in LLMs, which dynamically decide to execute or skip transformer layers at the inference time.
We propose simple yet effective eviction policy by transforming the causal self-attention masks. As result, it not only makes the proposed D-LLMs compatible with KV-cache methods but also saves the storage overhead inference time, which particularly important for resource- constrained platforms.
We conduct extensive experiments on main-stream LLMs. reduce up to 45% computational cost and KV-cache storage on Q&A and math solving tasks, and 50% on commonsense reasoning tasks without performance degradation.
The results demonstrate that D-LLMs
2 Related Works
LLMs Finetuning Methods. Due to the large amount of computational cost of training LLMs, various parameter efficient finetuning methods have been explored, including prompt-based learning methods [0, [19, 25, 411, 80], adapter-
learning methods[27] 344, 67, [Z4] and reparametrization- based learning methods [ D.. CO, 255]. Prompt-based learning methods propose to insert and infer collection of trainable embeddings to existing tokens. For example, P-tuning [[]] adds a prefix to input, and P-tuning v2 [[G applies prompts to intermediate layers. Adapter- learning methods propose to insert specially designed adaptors to pre-trained LLMs. How to insert adaptors has also been explored, i.e., Adamix | and Compacter [[S=]] serially connect adaptors and [27, 671 parallelly connect adaptors with transformer. Reparametrization-ee based learning methods exploit finetuning models with low intrinsic dimension [M]] Specifically, LoRA [U] decomposes parameters into two low-rank trainable matrices before computation.
Different from these methods which focus on parameter-efficien finetuning, D-LLMS pay more attention to achieving high-speed inference via adaptive computing resource allocation.

---

LLMs Acceleration Methods. Extensive research works have been proposed to compress LLMs
and reduce the network parameters. Among these methods, quantization [9. 3G. 92. 57 62]. distilla-
G 65] and pruning [[%, 43. AX 511, X2] are mainstream approaches. Quantization methods propose to reduce the bits of each parameter by converting floating-point (FP32 FP1 parameters to integers (INT8/ / INT4) or other discrete forms. This line of methods preserves the structural characteristics of the network and compresses models in exchange for lowering accuracy. Distillation methods [24], R9. !!ل] attempt to transfer knowledge from LLMs to a lightweight student model. However, such methods usually require retraining models an end-to-end manner and may take up large computing resources. Model pruning methods propose to reduce the complexity of models by removing redundant parameters. These methods have two forms, static pruning [G. IT. 25] and dynamic pruning [24 I2. 21. 29, 49, 8I]. Static pruning methods usually remove redundant layers based on various relevant metrics. For example, Shortened-L_LaMe
tion [22], 39, 20.
[655] measures the importance of layers and prunes the unnecessary ones. However, with the increase of pruned parameters, static pruning poses inevitable performance degradation. Dynamic pruning methods tend to prune unim- portance layers based inputs. For example, Ada-Infer [[ introduces an early-exit mechanism to stop the inference at intermediate layers. Mixture-of-Dephe [50] measures the importance of each token and only calculates the top-k tokens different layers.
In essence, the proposed D-LLM belongs to dynamic pruning methods, which dynamically decide which layers should be executed for each input token.
Dynamic Inference in Computer Vision. Dynamic inference is promising technique skip layers at inference time to achieve acceleration [700 77I, 772 [ž3], which was initially proposed in computer visions. Research on dynamic inference mechanisms has two different approaches. The first one is layer skipping. This line of methods mainly focus on how to design a specific dynamic decision module or metric to reduce computational cost. Specifically, ConvNet-AIG [[X]] uses a convolutional module to define the inference graph conditioned on inputs. It proposes router to make the execution decision for each model layer. SkipNet [Z3] utilizes LSTM as the decision module to determine whether a layer should be executed or not. Besides, CoDiNet [[22] attempts to model the relationship between the inputs and their executing decisions to enhance the optimization. o The second line of methods is early prediction. While dynamic inference network has only one exit, an early prediction network is characterized by multiple exits. In early prediction networks, the execution would stop once a criterion for given sample is satisfied. MSDN [[I]] is a typical example, which introduces multiple early-exits, according to the allowed time budget. Instead of bypassing all units, DCP [20]] generated decisions reduce the computational cost channels.
Inspired by dynamic inference the CV task, we rethink the characteristics NLP and raise the following question. Should models allocate the same computing resource on tasks of different diffi- culties or tokens of different importance? ? Motivated by this, we introduce the D-LLMs framework, which will dynamically allocate fewer computing resources for unimportant tokens.
3 Methods
3.1 Preliminary: LLMs Architecture
As natural language processing task, a sequence of input words is first embedded into tokens fa _xx2.... , x } by a tokenizer. Each token would then go through a pre-defined large language model sequentially in autoregressive manner. That is say the target of {x ,x..... x²") is =+1 In this way, the inference paradigm of large language model is formulated as
x n+ = foutpu C fL 2
0 ... O 0
fnead( 22
x
where is is the total number of transformer layers, fread is the word embedding and foutpu is the word classifier. For convenience, represent the layer index of token embeddings in the following sections.
the superseript represent the token index and the subscript to
Transformer Layer Dominant architectures of LLMs are decoder-only transformer networks, which consist of L transformer layers. Each transformer layer contains a multi-head self-attention (MHA) and feed-forwate network (FFN). The inference of the (-th transformer layer can be de-
3

---

fined as,
by MHA(2f) +
xi+1 FFN( + h
where If is the n-th imput token embedding of the 1-th transformer layer and hy is the output of MHA. Specifically, a MHA divides token embeddings into multiple heads and perform self-attention operations in parallel.
Self Attention. Since large language models mainly process inputs sequentially, the calculation of former tokens should not be affected by latter ones. To tackle this problem, the self-attention module (SA) uses causal attention mask M on attention scores as
0, 1-00,
j S }
M
(4)
With the causal attention mask M, the features of the n-th token are processed without subsequent tokens. Under such framework, the formulation of MHA(-) is defined
MHA(x²) == concat((A(( SA( ... :S(((A)) o Wo.
(5)
where Wo is learnable parameter and is matrix multiplication. MHA(-) is essentially concatena- tion of self-attention operated features divided into heads { Based on learnable weights Wo: WK, Wv, self attention is formulated as
SA(x))) = softmax Ti. (xo + M) o oooooWi
(6)
vd
where X {- oi.
2 is input features of the sentence, d the dimension of each head and (.)T is transpose. In addition, FFN consists of several linear layers with activation functions following the MHA.
KV-Cache. The KV-Cache method [56] has been a standard acceleration technique for large lan- guage model inference. As shown in Eq. i. the query embedding of given token only caleulates the self-attention with previous keys and values. Since decoder-only LLMs process input tokens in an autoregressive way, the key and value embeddings can be reused to boost inference speed by storing calculated key and value embeddings. The KV-cache method reduces the computation complexity from (n²) (n), but brings extra memory overheads store the KV-cache when generating
long texts 313 MI 29, [7]].
3.2 D-LLMs: Dynamic Inference LLMs
In this section, illustrate the proposed dynamic inference mechanism in large language models. We assign a dynamic decision module before each single transformer layer. At the inference time, the dynamic decision module makes execution decision whether the related transformer layer should be executed or not as shown in Fig. Mā
Given token in an input sequence, it would go through dynamic decision module 0t. prior to the 1-th transformer layer fi in large language models. As shown Fig. IE a dynamic decision module is composed of two linear layers separated by an activation function. At the inference time, the dynamic decision module outputs (I;), the parameters of categorical distribution that represent the probability of skipping and executing based on the input feature xi to the [-th transformer layer. The execution decision for the 1-th layer is defined as
by = 1 (arg max(x(
where one-hot operation and by is -in-nninnsional vector. This way, the result the execution decision tuple for each token is either [o. 1] or [1, O]. However, arg max operation deterministically outputs the max argument without exploration and is not differentiable during training. To solve
this problem, we utilize the Gumbel-Softmax reparametrization technique [62] and straight-through estimator [5] to ensure the transformer networks can be optimized an end-to-end fashion during training. Thus, the execution decisions are organized as hard forms when forward computing:
by = (arg max(log((( (g: (x:)) + T)), K G(0, 1),
(8)

---

Trainable
Freezed
Transformer Layer
Transformer Layer 1+1
m
Decision Module
Q.K.V
m1--1
Decision Medule
Q.K.V
51/2
LORA
MHA
FFN
LORA
MHA
FFN
[0,1]
11.0
(a) The framework of D-LLMs.
Reserved
Lincar
T
Decision Medule
0
Norm Nonlinearity
MHA
Linear & Softmax
Eviction Mask
(b) The Dynamic Decision Module.
(c) The KV-cache Eviction Mask.
Figure I: The framework the proposed D-LLMs. The inference paradigm of dynamic decisions for transformer layers is shown in Fig. Ial The design of dynamic execution decision modules is shown Fig. DT. The mask in multi-head self-attention with eviction strategy shown in Fig, Irc
where Go, 1)is a Gumbel distribution. With sampled noise म, the decisions b, satisfy the categorical distribution (z))). The decisions calculated in soft forms when back sock-prpaagatin::
(x)(( + x,)/T) b E. exp((lo((ar + x;)/r)"
(0,
The temperature 7 controls the sharpness of softmax. As O. the softmax result converges the discrete form from the categorical distribution. Finally, the output of layer / is defined as
T1+1 b 21 + b f(()).
(10)
where by, by, 1) are the decisions of skipping executing the 1-th transformer layer.
3.3 Customizable Acceleration Rate
According to Eq. III. when b zero, f(())) would not be involved in calculation. Thus, it achieves reducing the computational cost of fi. The average acceleration rate wn for the token z" can be defined as
"
Σ' (6,))
(11)
where L is the total number layers, and b is the decision executing the n-th token at the I-th layer. To make the acceleration rate adaptive to platforms of different computing capabilities, design an acceleration ratio loss to customize the acceleration rate as
Cytee
-Q
(12)
N
where .11 is [ -norr and the Q is the user-defined target acceleration rate.
Based on the aforementioned method, we achieve the optimization and inference for D-LLMs in end-to-end manners. Note that in addition to training the L.LM model from scratch and obtaining generalized D-LLMs, our method can easily work on pre-trained LLMSs by employing LoRA [B]].
5

---

MaWPS
0BQA
SAMSuM
4.75
0.7
LoRA
X
4.50
(Tay))((Tas))
SP.LLM
0.7
6.25
Accurscy
0.4
0.3
D-LIMs
D-LLMS
3.75
LORA
Sh.LLMCTaa.)
Mh.LLMMTHy))
3.50
Sh.LLM(PPL)
L1.PPP)
Ada
Ads-infer
3.25
0.-
0.4
0.2
CLOPS
LOPS
ELOPs
(a) (f) FLOPSs MaWPS, (b) (fc.(1) FL.PPs OBQA. (c) PPI(L) v.s. FLOPs on SAMSum.
Figure 2: The performance against computational cost D-LLMs on three datasets. The figures show that reducing around 40% 60% computational cost achieves the best trade-off.
3.4 KV-Cache Eviction Strategy
In this section, elaborate how our proposed dynamic inference mechanism cooperate with the KV-cache strategy. Considering the inference of the n-th token the (-th transformer layer in eq. (66), all previous tokens involved in the calculation of at It means that the execution decision b E (1,.... , on token embedding of is zero, we still need to calculate its keys and values for self-attention calculation of z7' We conduct an aggressive strategy of self-attention, which bypasses the features of skipped tokens. We propose KV-cache eviction policy by designing mask the attention matrix ignore the uncalculated features. Thus, the corresponding KV- cache is unnecessary to keep, which reduces storage overhead during inference.
As shown in Fig. IIC. design eviction attention mask E each layer together with causal mask in Eq. & which supports training in batch. In addition, recent research on attention maps and KV-cache [26, [S] shows that tokens the beginning of sentence are crucial subsequent tokens. We also find that evicting the KV-cache of initial tokens at skipped layers would either leads causes unstable optimization. Therefore, we preserve the keys and values of the first m tokens at the beginning of texts. Hence, the eviction attention mask for the I-th layer is encoded
performance decreasing
0, Bij
i m or 1 =1
(13)
00,
otherwise,
where 1/11 = 1 represents the j-th token executing the I-th transformer layer, and m is the number of preserved tokens at beginning. Thus, the KV-cache of m initial tokens are kept in the prefilling phase. In method, simply set the decisions of initial tokens to be executed in a deterministic way, which demonstrates equivalent performance and is easier to implement practice.
Finally, we introduce the overall loss functions in D-LLMs. The objective function is composed of two parts: cross-entropy loss and acceleration ratio loss.
L LeE Lrate:
(14)
where Lce is the cross-entropy loss and is a hhper--prammnnn parameter of the acceleration ratio loss Crate
4 Experiments
4.1
Experimental Setup
Models and Benchmarks. We conduct experiment of the proposed D-LLMs on widely used LLMs. LLaMA2-7B and LLaMA3-8B. LLaMA2-7B and LLaMA3-8B both contain 32 transformer layers. We perform few-shot finetuning on nine different benchmarks to evaluate the effectiveness. The benchmarks are divided into three different tasks. The first task is Q&A and summarization, in- cluding Alpaca [68] and SAMSum [22], and perplexity (PPL) is used to measure their performance.

---

Table Performance Comparison on different tasks under Few-shot Settings based on LLaMA2-7B.
Sh. Lla. PPL refers to Mhortened-LLaMee applying PPL metric. Sh. Lla. Tay. refers to Shortened-
LLaMA applying Taylor metric. Ada-Inf. refers to Ada-Infer. For convenience, we mark the best performance in red and the lowest computational cost in blue.
Dataset
MoD [59]
Sh. Ila. PPI. [555]
Sh. IJa. Tay. (35) Ada-Inf. [1
D-ILM
Q&A
PPL(() (LOP)())
PPI.()) FLOPP(())
PPL(() FLOPs())
PPL(() FLOPS())
PPI.() FLOPs(L)
Alpaca
10:32
0.56
7.09
0.66
7.65
0.66
319
0.65
6.01
0.59
SAMSum
4.47
0.56
4.39
0.66
4.66
0.66
874
0.56
3.18
0.55
Math
Acc.(()) FLOPS(L)
Acc.(1) FLOPS(()
Acc.(1) FLOPs(()
Acc.(t) FLOPS(()
Acc.(1) FLOPs(L)
GSM8K
0.08
0.56
0.10
0.66
0.18
0.66
0.00
0.83
0.29
0.59
MaWPS
0.33
0.56
0.52
0.66
0.39
0.66
0.00
0.90
0.74
0.56
Com. Sen.
Acc.(f) FLOPs(L)
Acc.(t) FLOPs(()
Acc.(1) FLOPs(L)
Acc.(t) FLOPs())
Acc.(t) FLOPs(L)
BoolQ
0.64
0.56
0.67
0.66
0.73
0.66
0.71
0.61
0.73
0.52
PIQA
0.49
0.56
0.76
0.66
0.83
0.66
0.55
0.63
0.84
0.52
SIQA
0.58
0.56
0.75
0.66
0,81
0.66
0.80
0.64
0.82
0.54
OBQA
0.42
0.56
0.63
0.66
0,81
0.66
0.78
0.76
0.80
0.53
MMLU
0.28
0.56
0.47
0.66
0.53
0.66
0.41
0.60
0.53
0.55
are about math problem solving. The answers marked as correct only if the numerical difference is less than 10-5, Finally, BoolQ [8], PIQA [6], SIQA [60]]. OBQA [53], MMLU [28] are datasets about common sense reasoning. The performance on these benchmarks is evaluated by accuracy.
Second, GSM8K [9], MaWPS [33
Implementation Details. To avoid finetuning the entire parameters in D-LLMs, adopt the LoRA finetuning method as a baseline. We apply dynamic decision modules to transformer layers except the first two for training stability. The maximal context length of tokens is set to 1024. In the designing of the dynamic decision module, we use two linear layers with hidden dimension of 512. For the KV-cache eviction policy, reserve the first two tokens, i.e., the loss function is set for Q&A, summarization task, 1.0 for math problems tasks, and 0.1 for commonsense reasoning tasks. We finetune 20 epochs for GSM8K and 10 epochs for other datasets.
2. Finally, in
4.2 Performance Comparison
Inference Acceleration. We show the acceleration of our proposed D-LLMs. As defined in Eq. Im| D-LLMs can customize the acceleration rate by adjusting the target rate Q. In Fig. D. we show the performance under different FLOPs on three datasets. The performance on MaWPS and OBQA is evaluated by accuracy, while that on SAMSum is evaluated by PPL. With the increasing FLOPs on MaWPS and OBQA, the accuracy shows improvement. When only using 40% FLOPs and 30% FLOPs on MaWPS and OBQA, D-LLM exceeds the baseline with 100% FLOPs. In terms of SAM- Sum, which is Q&& task, the performance gets better first and then gets worse. The reason that performance gets worse when the cost over 60%, might be due to overfitting.
Comparison with state-of-the-art. We compare the proposed D-L.LMs with state-of-the-art meth- ods on nine benchmarks. These benchmarks are concluded in three tasks, i.e Q&A with summariza- tion, Math Solving, and Common Sense Reasoning, The performance of D-LLMs exceed MoD [59]] Chortened-LLaMe [[S]], and Ada-Infer [17]] in a large margin. In terms of computational cost, we denoted the FLOPS of LLaMA2-7B with LoRA as 1.00, the cost of others are percentages based on as shown in Tab. M1 It is worth noting that FLOPs are calculated on average cost over tokens. Specifically, with the same baseline, i.e., LLaMA2-7B with LoRA finetuning, D-LLMs achieve bet- ter performance with only using about 50% computational cost. Shortened-LLLave reports two different results in their paper, which utilize Taylor and PPL as metrics, respectively. D-LLMs sur- pass Shortened-L_Lawe on datasets, though adopting fewer computing resources. Besides, we also reproduce the MoD and Ada-Infer and show that D-I.MMs perform better as well.
7

---

Table Ablation on different tasks under Few-shot Settings based on LLaMA2-7B and LLaMA3-
8B. LoRA refers to using LLaMA3-8B and LLaMA2-7B as pretrained model and finetuning with LoRA, D-LLM w/o Evi. Str. refers to the proposed method without eviction strategy.
Datasets
Alpaca
SAMSum
GSM8K
MaWPS
BoolQ
PIQA
SIQA
OBQA
MMLU
Metrics
PPL()) FLOPS())
PBL() FLOPs(()
Ascr FLOP
Acc(T) FLOP
AccrT) FLON(L)
Acar") FLOP
Atc(tt) FLOPS(()
Acc FLOP
Acc/fh FLOPAL)
Backbone
LLaMA2-7B
LoRA
4.69
3.61
0.27
0.72
0.73
0.84
0.81
0.79
0.54
finetune
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
D-LLM w/o.
6.08
3.63
0.29
0.72
0.72
0.83
0.81
0.82
0.53
Evi. Str.
0.64
0.62
0.60
0.61
0.60
0.59
0.60
0.59
0.60
6.01
3.18
0.29
0.74
0.73
0.84
0.82
0.80
0.53
D-ILM
0.59
0.55
0.59
0,56
0.52
0.52
0.54
0.53
0.55
Backbone
LLaMA3-8B
LoRA
4.84
3,93
0.52
0.88
0.75
0.87
0.81
0.85
0.56
finetune
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
8.43
3.63
0.50
0.91
0.75
0.88
0.81
0.82
0.57
D-LLM
0.59
0.58
0.60
0.55
0.51
0.52
0.55
0.53
0.55
4.3
Quantitative Analysis
Ablation Study. We conduct ablation studies of our proposed method based on LLaMA2-7B and LLaMA3-8B and then perform the results three different tasks including Q&A and Summary, Math Problem, and Common Sense Reasoning. As shown in Tab. D we illustrate the performance of dynamic inference large language models over the baseline. Based on LLaMA3-SB, with less than 55% computational cost, our proposed D-LLM achieves better performance than that of the baseline on five datasets. In comparison, obtaining a comparable performance, D-LLMS on LLaMA2-7B only take about 55% computational cost. On Q&A and Summary tasks, D-LLMs sacrifice some accuracy for lower computational cost. The performance of baseline methods, i.e., LLaMA with LoRA finetuning, outperforms D-LLMs on Alpaca using 40% extra cost. Considering math-solving problems, D-LLMs spend more computing resources than common sense reasoning tasks, which difficult.
also demonstrates that math-solving problems
KV-Cache Overhead. In this part, we discuss the KV-cache storage benefits D-LLMs. With the KV-cache eviction strategy, some keys and values are not necessary, because these features useless to subsequent tokens. So the memory overhead on KV-cache reduced with the layers are skipped. The benefits of reducing KV-cache storage actually the skipped ratios of transformer layers in D-LLMs. As a result, we achieve almost 45% storage overhead reduction than the baselines on both LLaMA3-8B and LLaMA2-7B.
Table 3: The parameter analysis on numbers of reserved tokens not participate in dynamic inference.
Res. Tok.
0
m
T7a 2
m
8
Metrics
PPL()) FLOPs(()
PPL()) FLOPs())
PPL(.) FLOPs())
PPL()) FLOPs())
PPL(() FLOPs(()
SAMSum
3.33
0.55
3.32
0.55
3.18
0.55
3.29
0.55
3.35
0.55
Metries
AAc.(()) FLOPs(()
Acc.(t) FLOPs(L)
Acc.(t) FLOPs(L)
Acc.(t) FLOPs(L)
Acc.(t) FLOPs(L)
SIQA
0.80
0.53
0.81
0.55
0.82
0.53
0.81
0.55
0.80
0.55
Reserved Tokens (m). We conduct experiments to analyze the number m of initial tokens reserved our proposed eviction policy. For a fair comparison, we set the computational cost of all experi- ments at the same level with Ω = 0.5. We compare reserved tokens of 0, 1, 2, 4, and 8 on SAMSum and SIQA datasets. As shown in Tab. EL using about 55% computational cost of entire LLMs,, the perplexity decreases at first and then increases. The result demonstrates that keeping the first two

---

tokens is the best setting. Similarly, it shows the same tendency on SIQA, and the inflection point appears at the same time. As a result, we keep the first two tokens without skipping over layers.
4.4 Qualitative Analysis.
Execution Layers of Grammatical Terms. Here, we present a visualization of the execution ratios of different grammatical terms shown in Fig. E. From this figure, we observe that executing ratios of number and math symbols are much higher than others on layer #4 and layer #26. Meanwhile, layer #21 and layer #23 are utilized more by modal verbs and subject terms respectively. All terms show high execution ratios on the layer #18. Since different grammatical terms usually play the same role in sentences, thus, their execution ratios are similar. More importantly, different layers in D-LLMs are of different abilities, which are used by different grammatical terms.
Layer
Layer
Layer
Layer
Layer
Pei They We
de
2 They poodd Hemaas
We
cold
These
(a) Ratio on layer (b) Ratio on layer 18. (c) Ratio on layer 21. (d) Ratio on layer 23, (e) Ratio on layer 26.
Figure 3: Execution ratios different layers of three grammatical terms. Blue dots refer to number and math symbols, e.g., '1, +, x'. Red dots refer to subject terms, e.g., 'She, He, They'. Green dots refer to modal verbs, e.g. 'may, should, might'. The distance of dot to the center represents executing ratio. The red circle is probability of 100% and the blue circle is the average ratios.
Execution Decisions of Similar Tasks. We count up and visualize the executing ratios of different layers to illustrate the difference on Benchmarks and Questions. As shown in Fig. !R the first few layers are utilized by most Benchmarks. Q&A and summarization tasks and math solving show larger diversity than common sense reasoning. Common sense reasoning tasks utilize more shallow layers and show smaller variances. In addition, we visualize several questions about high school history, science, and engineering knowledge from MMLU. As shown in Fig. HER the execution ratios of historical knowledge similar, while that of science and engineering knowledge similar. In conclusion, the execution behavior of utilizing similar knowledge tends to adopt similar ratios in D-LLM.
Aipata
sastum
high secholchooeeeeee eursppan haseey high school as listory
guaer
38 0.5-
high world histery
Hasers
abstrct
Bosid
26 0.5
colinge_chemisryy
eleca cal engineering
tick
tockk
0.4
2
10
12
14
22
24
26
28
3b
15
16
17
18
190
22
27
(a) Execution ratios of layers on Benchmarks.
(b) Execution ratios of layers on Questions.
Figure 4: The execution ratios of different layers visualizations on Benchmarks and Questions.
Fig. E shows the execution ratios of different benchmarks on respective layers. A deeper color refers to higher execution ratios. Fig. AR shows six standard questions' execution ratios over layers from MMLU. The Y-axis is the execution ratios and X-axis is the layer index.
Conclusion
In this paper, we take the inference of large language models from a new perspective, that tasks of different difficulties should not require the same amount of computing resources and not every word is equally important in a sentence. Motivated by this, we propose a novel dynamic inference framework for LLMs, called D-LLMs, which achieves adaptive computing resource allocation for different tokens. We first propose a dynamic decision module to decide whether layer is necessary

---

for the given token. Then, we propose a KV-cache eviction strategy to enable D-LLMs' deployment in real-world upplications. Finally, we conduct experiments on nine benchmarks and show D-LLMs performing comparable accuracy with only about 50% computational cost over baselines. More importantly, the D-L.LMS can be deployed on almost all L.LMs with negligible training cost.
Limitation. Our proposed D-LLMS make the first attempt to adaptively allocate computing re- sources at inference time for LLMs. Although experiments show surprising performance with lower cost, there are still some interesting issues worthy of future research. First, the generalizability of D-LLMs for broader tasks still needs verification. Second, it is worth exploring the dynamic mech- anism on different granularities, e.g., sentences and tasks.
References
[1] Armen Aghajanyan. Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effective-
ness of language model fine- tinettuning arXiv:2012.13255, 2020.
[2] Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hofmann. Dynamic context pruning for efficient and interpretable autoregressive transformers. Adv. Neural Inform. Process. Syst. 2024.
[3] Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. In Emerg. Techn. Comput. Syst., 2017.
[4] Haoli Bai, Wei Zhang, Lu Hou, Lileng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Trwin King. Binarybert: Pushing the limit of bert quantization. In arXiv:2012.
2020.
[5] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1.308. 3432, 2013.
[6] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianleng Gao, and Yejin Choi. Piqa: Reasoning about
physical commonsense natural language. In Proc. AAAI Conf. Artif. Intell, 2020.
[7] Lester Brian, Al-Rfou Rami, and Constant Noah. The power of scale for parameter-efficient prompt tuning. In urXiv:2104.086)) 2021.
[8] Christopher Clark, Kenton Lee. Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficullt of natural yes/no questions. In NAACL. 2019. [9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Katser, Matthias Plappert. Jenry Tworek, Jacob Hilton, Reichhrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. In arxivv 10. 1416,4,8, 2021.
[10] Xiangxiang Dai, Jin Li, Xutong Liu, Anqi Yu, and John Lui. Cost-effective online multi-lim selection with versatile reward models. arXiv preprint ar Xiv:22400 16587, 2024.
[1]] Tim Dettmers, Artidoro Pagnoni, An Holizman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In Adv. Neural Inform. Process. Syst., 2023.
[12] Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejic Chi, and Beidi Chen. Get more with less: Synthesizing recurrence with kv cache compression for efficient I1m inferenee. In 00X3v:2402.09398 2024.
[13] Shichen Dong. Wen Cheng, Jiayu Qin, and Wei Wang. Qaq: Quality adaptive quantization for I1m kv 24Xx:22403.04643, 2024.
cache.
[14] Xin Luna Dong, Scungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. Towards next- generation intelligent assistants leveraging Ilm techniques. In Proc. Knowl. Disc. Data Mining, 2023. [15] Ali Edalati, Marzich Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James Clark, and Mehdi Rezagholizadeh.
Króna: Parameter efficient tuning with kronecker adapter. In arXiv:2212. 10650, 2022,
[16] Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Chukwunyere Osi, Prateek Sharma, Fan Chen, and Lei Jiang. I.I.Car Modeling the end-to-end carbon footprint of large language models. In Int. Conf. Learn. Represent, 2024.
[17] Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Pong Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not all layers of ilms are necessary during inference. In arXiv:2403. 021881 2024.
[18] Gongfan Fang, Xinyin Ma. Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. in IEEE Conf. Comput. Vis. Pattern Recog.. 2023.
[19] Peng Gao, Jiaming Han, Renrui Zhang. Zayi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-sdapter v2: Parameter-efficient visual instruction model. In arXiv:2304.1500
2023.
[20] Xitong Gao, Yiren Zhao, ukasz Dudziak, Robert Mullins, and Cheng zhong Xu. Dynamic channel prun- ing: Feature boosting and suppression. In Int. Conf. Learn. Represent, 2019.
[21] Suyu Ge, Yuman Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what
to discard: Adaptive kv cache compression for L.MMs. In Int. Conf. Learn. Represent., 2024.
[22] Bogdan Gliwa. Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue datasct for abstractive summarization. In Proc. Worksh. New Front. Summariz., 2019.
[23] Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the elfects of weight
pruning on transfer learning. In arXiv:2002.0883 08307, 2020.
[24] Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language
models. In Int. Conf. Learn. Represent, 2023.
10

---

[25] Qin Guanghui and Jason Eisner. Learning how to ask: Querying Ims with mixtures of soft prompts. In
arXiv:2104 10n.0999. 2021.
[26] Chi Han, Qifan Wang. Hao Peng. Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Zero- shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language
Technologies (Volume I: Long Papers), 2024.
[27] Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Kerg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In arXiv:21 10.04366, 2021.
[28] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding, In Int. Conf. Learn. Represent. 2021. [29] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kyquant: Towards 10 million context length IIm inference with kv cache quantization. In arXiv:2401. 18079, 2024.
[30] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. Int. Conf. Learn. Represent.. 2022.
[31] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q Weinherger. Multiscale dense networks for resource efficient image classification. In Int. Conf. Learn. Represent.,
2018.
[32] Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softtax. In Int. Conf. Iearn. Represent,, 2017.
[33] Marek Kadlik, Michal tefánik, Ondej Sotolá, and Vlastimil Martinek. Calc-x and calcformers: Empower- ing arithmetical chain-of-thoughh through interaction with symbolic systems. In Proc. Conf. Empir. Meth. Natural Langu, Process.. 2023.
[34] Rabeeh Karıi Mahahadi, James Henderson, and Schastian Ruder. Compacter: Efficient low-rank hyper-
complex adapter layers. In Adv. Neural Inform. Process. Syst., 2021.
[35] Bo-Kyeong Kim. Geonmin Kim, Tae-Ho Kim. Thibault Castells. Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened llama: A simple depth pruning for large language models. In arXiv:2402.02833 2024.
[36] Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. In Adv. Neural Inform. Process. Syst., 2023.
[37] Aaron Klein, Jacek Golebiowski, Xingchen Ma, Valerio Perrone, and Cedric Archambeau. Struc- tural pruning of large language models via neural architecture search. In AutoMI. Conference 2023 (Wrkshopp) 2023.
[38] Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large lan- guage models. In arXiv:22003.... 07259, 2022.
[39] Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. Dynamic knowledge distillation for pre-trained language models. In arXiv:2109.1 1100911295,... 1295, 2021.
[40] Shiyang Li. Jianshu Chen, Yelong Shen. Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian. Baolin Peng. Yi Mao, et al. Explanations from large language models make small reasoners better. In arXiv:2210006777
2022.
[41] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In arXiv:2101.00 00t-2101.00190 90, 2021.
[42] Yanjing IA, Sheng Xu, Xianhin Cao, Xiao Sun, and Baochang Zhang. Q-DM: An efficient low-bit quan-
tized diffusion model. Adv. Neural Inform. Process. Syst., 2023.
[43] Yixiao Li. Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoSparse: Structured compression of large language models based on low-rank and sparse approximation. In Int. Conf. Mach. Learn., 2023.
[44] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-effiiient fine-tuning is better and cheaper than in-context learning. In Adv. Neural Inform. Process. Syst., 2022.
[45] Shih- Liu, Chien-Yi Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In arXiv:2402.09353, 2024.
[46] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam. Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. In 100.022110007002. 2021.
[47] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt under- stands, too. In AI Open, 2023.
[48] Zichang Liu, Jue Wang. Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, and Beidi Chen. Contextual sparsity for efficient LLMs at infer- ence time. In Int. Conf. Mach. Learn., 2023.
[49] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for ky cache. In arXiv:2402.07550 2024.
11

---

[50] Ruilong Ma, Jingyu Wang, Qi Qi. Xiang Yang, Haifeng Sun, Zirui Zhuang, and Jianxin Liao. Pipellm: Pipeline Ilm inference heterogeneous devices with sequence slicing. In Proc. SIGCOMM Conf., 2023. Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Adv. Neural Inform. Process. Syst., 2023.
[51]
[52] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang. Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers large language models are more redundant than you expect. In
arXiv:2403,038: 03383333333333.. 2024.
[53] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity?
a new dataset for open book question answering. In Conf. Empir. Meth. Natur. Lang. Process., 2018. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong. Hailey Schoelkopf, Xiangru Tang. Dragomir Radev, Al- ham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, dward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning. In Proc. Assoc. Comput. Ling., 2023.
[54]
[55]
openai. Chatgpt. In chat.openai.com
2023.
[56]
Reiner Pope. Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 2023.
[57] Haotong Qin. Mingyuan Zhang. Yifu Ding. Aoyu Li. Zhongang Cai, Ziwei Liu, Fisher Yu, and Xianglong
Liu. Bibench: benchmarking and analyzing network binarization. In Int. Conf. Mach. Learn., 2023.
[58] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask leamners. In GitHub.com, 2019.
[59]
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depeee de-th:: Dynamically allocating compute in transformer-bar language models. In arXiv:2404.0 022588,02288888 2024.
[60]
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. In arXiv: Xiv 1904.09728, 2019.
[61]
Armin Sarabi, Tongxin Yin, and Mingyan Liu. An llm-based framework for fingerprinting internet- connected devices. In Proc. Intern. Mcasur. Conf., 2023.
[62]
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In Int. Conf. Learn. Represent., 2024.
[63]
Benjamin Frederick Spector and Christopher Re. Accelerating L.MM inference with staged speculative
decoding. In Int. Conf. Mach. Learn., 2023.
[64]
Siqi Sun, Yu Cheng. Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression.
In arXivv11988 10111998999555 033,59935522 2019.
[65]
Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang, Shuohang Wang, and Jingjing Liu. Contrastive distillation on intermediate representations for language model compression. In arXiv:2009, 14167, 2020.
[66] Zhiqing Sun, Yikang Shen. Qinhong Zhou. Hongxin Zhang. Zhenlang Chen, David Cox, Yiming Yang. and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. In Ady. Neural Inform. Process. Syst., 2023.
[67]
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. In Adv. Neural Inform. Process. Syst., 2022.
[68] Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li. Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model. GitHuh.com. 2023.
[69] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient
foundation language models. arXiv preprint XrXiv:2302. 13971, 2023.
[70] Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Eur. Conf.
Comput. Vis., 2018
[71] Huanyu Wang. Songyuan Li, Shihao Su. Zequn Qin, and Xi Li. Rdi-net: Relational dynamic inference networks. In lnt. Conf. Comput. Vis., 2021.
[72] Huanyu Wang, Zequn Qin, Songyuan Li, and Xi Li. Codinet: Path distribution modeling with consistency and diversity for dynamic routing. IEEE Trans. Pattern Anal. Mach. Intell., 2022.
[73] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Eur. Conf. Comput. Vis., 2018.
[74] Yaqing Wang. Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jian- feng Gao. Adamix: Mixture-of-adapte for parameter-efficient tuning of large language models. In arXiv:2205.124: 2022.
[75] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In Int. Conf. Learn. Represent., 2023.
[76] Z Yao, RY Aminabadi, M Zhang, X Wu, C Li, and Y Zeroquant He. Efficient and affordable post- post-training quantization for large-scale transformers, 2022. In arXiv:2206.01866 2022.
[77] Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. Tr-bert: Dynamic token reduction for acceler- ating bert inference. In arXiv:2 105. 1618,, 2021.
[78] Shuzhou Yuan, Ercong Nie, Bolei Ma, and Michael Farber. Why lift so heavy? slimming large language models by cutting off the layers. arXiv:2402. I 1700, 2024.
12

---

[79] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In Adv.
Neural Inform. Process. Syst. Worksh., 2019.
[80] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hong- sheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention.
In Int. Conf. Learn. Represent., 2024.
[81] Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, Yanzhi Wang. systematic dnn weight pruning framework using alternating direction method of multipliers. In Eur. Conf.
Comput, Vis., 2018.
[82] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song, Yuan- dong Tian, Christopher Re, Clark Barrett, Zhangyang Wang. and Beidi Chen. H2o: Heavy-hilter oracle for efficient generative inference of large language models. Adv. Neural Inform. Process. Syst., 2023. [83] Juntao Zhao, Borui Wan, Chuan Wu, Yanghua Peng. and Haibin Lin. Lim LIm- pq:serving Ilm on heteroge- clusters with phase-aware partition and adantive quantization. In Proc. Annu. Sympos. Princip. Pract. Program., 2024.
[84] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang
Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. Int. Conf. Learn, Represent, 2024.
13

---

A
Appendix
The Appendix is organized follows. A.I.A.II. we visualize the execution decisions over layers of tasks of different difficulties. In S.c.A. we show the execution ratios of each layer under differ- ent computational cost. Sec.A... we present the concrete accuracy and perplexity under different cost. In AAE.A we show the accuracy and computational cost under different hyper-parameter In A we provide information about extra overhead taken by our proposed decision modules in D-LLMs. In S.c.A.f. we elaborate details on the reproduction of state-of-the-an methods and im- plementation our D-LLMs. In Sec.A.J we present the comparisons of generated results between D-LLMs and LLaMA with LoRA. In S.C.A.N. we list the licenses for all assets used in this paper.
A.1 Visualization on Tasks of Different Difficulties.
To demonstrate the key motivation 'models should not allocate the same computing resource on tasks of different difficulties or tokens of different importance.' we illustrate transformer layers to be executed for tasks of different difficulties in D-LLMs as shown in Fig. E. We compare the execution decisions over layers on two questions. The first question "How can develop my critical thinking skills?", which is referred as simple one. The second question is "Can you explain Fermat's Last Theorem?", which is referred as difficult one. From Fig. we can observe that the simple question utilizes fewer tokens than the different question. Moreover, the skipping decisions usually happen on upper half and lower half, while the execution decisions happen on initial layers, middle layers, and last layers.
Ruc
Buy
Siaccn
Besgon
fuu
40
(a) Tokens in Simple Task.
Lase
alates
that
Gnd
(b) Tokens in Difficult Task.
Figure 5: The transformer layers to be executed for tasks different difficulties in D-LLMs. fig.
the execution decisions, when D-LLMM answers "How can I develop my critical thinking skills?" fig. is is the execution decisions, when D-LLM answers "Can you explain Fermat's 's Last Theorem?" The second question more diffieulty than the first one, therefore, utilizing transformer layers. A filled block refers to token executing the corresponding layer, while an empty block refers to token skipping the corresponding layer.
14

---

A.2 Execution Ratios under Different Cost.
We visualize the executing ratios of layers under different computational cost in Fig. As one can observe, with the reduced computational cost, the skipped layers are usually from the last layers. When we only utilize about 25% computational cost, most of the layers are skipped. Only the first three layers are executed by most tokens. This phenomenon illustrates that just a few layers are enough for most of the tokens' understanding and generation. Only a few tokens require more computational resources.
OBGA
42%
60%
15%
219
24%
(a) Execution ratios under different cost MaWPS. (b) Execution ratios under different OBQA.
Figure 6: Execution Decisions over Layers under different ratios on MaWPS and OBQA. The verti- cal coordinate the computational cost. The horizontal coordinate the indices of layers. A dark color refers to most tokens execute the layer, while a light color refers to most tokens skip the layer.
A.3 Performance and Cost Trade-off.
With the customizable acceleration rate loss, adjust the user-defined target acceleration rate Ω to achieve performance under different FLOPs on three datasets. The concrete performance in terms of accuracy and computational cost is shown in Tab. EL Under extreme conditions, our proposed D-LLMS obtain 57% and 77% accuracy with less than 25% percentage of computational cost on MaWPS and OBQA, respectively. Specifically, the performance ranges from 57% 73% under computational cost from 0.28 to 0.80 on MaWPS. And it ranges from 24% to 81 % under compu- tational cost from 0.24 to 0.81. In terms of SAMSum, our method achieves least 4.09 PPL with 76% computational cost reduction.
Table The accuracy against computational MaWPS, OBQA, and SAMSum datasets.
MaWPS
OBQA
SAMSum
Acc.(f)
FLOPs())
Acc.(t)
FLOPs(L)
PPL())
FLOPs(L)
0.73
0.80
0.81
0.81
3.44
0.81
0.72
0.72
0.81
0.72
3.25
0.72
0.74
0.64
0.80
0.61
3.17
0.64
0.74
0.55
0.80
0.52
3.18
0,56
0.72
0.47
0.81
0.44
3.23
0.48
0.73
0.40
0.80
0.39
3.50
0.38
0.71
0.36
0.80
0.34
3.73
0.32
0.57
0.28
0.77
0.24
4.09
0.24
A.4
Hyper-parameter Analysis.
We use hyper- peee-parameter the factor of Lrate in loss function Eq to control the importance of acceleration ratio loss during training. a influences the performance on specific tasks and the ability to control the acceleration rate in D-LLMs. We perform parameter analysis on the MaWPS. In Tab. & the results show that larger a provides better control over the acceleration rate. For example, with the target accelcration ratio Ω set to 80%, the trained model with 0.1 achieves only 47%, whereas
15

---

5, D-LLM achieves 72%. In addition, an excessively large can decrease the performance of D-LLM, even when trained models share the same computation cost. For example, comparing cases on (St = 0.7,a 1) with cases on (s? = 0.6, 5), trained D-I.LMs both achieve approximately 60% acceleration. However, the accuracy a 5 decreases by 11% compared to the a 1 case.
Table 5: The accuracy and computational cost against papee--arameter.ccmm MaWPS dataset.
a
Acc.(t)
FLOPs(L)
Acc.(1)
FLOPs(L)
Acc.(t)
FLOPs(L)
0.5
0.75
0.63
0.74
0.56
0.62
0.52
0.6
0.73
0.57
0.72
0.47
0.62
0.42
0.75
0.56
0.73
0.40
0.61
0.34
0.8
0.73
0.53
0.71
0.36
0.57
0,28
0.9
0.72
0.49
0.70
0.33
0.50
0.22
A.5
Overhead of Decision Modules.
We provide information about extra overhead taken by decision modules of D-LLM in Tab.B. The base LLM is LLaMA2-7B.
Params/FL.OPs overheads. Decision modules are parameter-effichee which only take 1.0% the size of parameters of LLM and 0.9% FLOPs compared with the forward computation of LLM.
Training memory overheads. We store parameters to be updated in float32 and others in bfloat during training. The running GPU memory usage of training LoRA on LLM is about 26.2 GB and training D-LLM about 32 GB, which means the extra memory cost taken by decision modules is only 5.8 GB.
Inference memory overheads. During inference, the additional GPU memory cost taken by de- cision modules is only 0.3 GB. Furthermore, D-LLM requires less memory when generating each token since layers decided to be skipped are unnecessary to load into GPU memory. For example, D-LLM with an acceleration rate 50% requires only 7.4 GB when inference, significantly less than the 13.9 GB needed by origin LLM using all layers.
Latency. the latency of inference, computing through one decision module costs only 0.1 ms, which is quite lightweight compared with computing through transformer block, which costs about 1.2 ms. The gain of acceleration brought by layer skipping significant.
Table 6: Overhead information of decision modules in D-LLMs.
Params
FLOPs
Training(GB)
Inference(GB)
Latency/Block(ms)
Decision Modules
0.9%
0.8%
5.8
0.3
0.1
LLM
1.0
1.0
26.2(LoRA)
13.9
1.2
A.6 Experimental Details
Reproduce of State-of-the-art. Shortened-LLaMA. The pruning strategy on Shortened-LLaMe is based on Taylor and PPL metries. According authors' implementation, the number of pruning lay- ers is set to 11. Taylor-based Shortened-LLaam prunes layers (27,24.26,25,29 2,,,,,,,,, 23.22,.....8,,2,,,222. 29,22,,3, and PPL-based
MoortneddLLLMMe prunes layers [24.26,25, 10,27, 1,,,9,,,,221,4,29921,
Ada-Infer. Ada-Infer is trained on LoRA-based pretrained LLMs. We use Gap and Top Prob as the input feature and reproduce the classifier with two linear layers of 512 hidden dimension.
Mixture-of-depth. MoD selects top-k tokens for calculation at dynamic layers. According to their paper, we deploy dynamic selections even layers and the number of & is set 12.5% of the length
16

---

of sentences. We reproduce the classifier with linear layer and incorporate the same LoRA module for finetuning.
Computational resources. We conduct experiments on a server with 8 NVIDIA 1.20 GPUs. Models implemented in Pytorch with a AdamW optimizer. It takes about 0.07s every step the training phase. The learning rate is set to 9 x 10-3, Furthermore, we warm up training in the first 2 epochs and then utilize cosine annealing schedule.
A.7 Comparison of Generated Results.
Instruction: Give students tips on how to keep their nerves under control during class presentations,
D-LLMs on LLaMA2-7B:
(58% FLOPs)
Practice your presentation in front of a mirror or with a friend.
2. Take deep breaths and relax before the presentation.
Make sure stay focused on the task and don't let your nerves get the better of you.
4. Make sure to smile and stay positive throughout the presentation.
Don't be afraid to make mistakes or stumble.
6. Make sure to thank your teacher and classmates for their support.
Ask questions and listen to your teachers feedback.
8. Don't be afraid to ask questions if you don't understand something.
LLaMA2-7B + LoRA: (100% FLOPs)
1. Practice presenting your material in front of a mirror or with a friend.
2. Make sure breathe deeply and relax your body before the presentation.
3. Visualize success and remind yourself of your strengths.
4. Speak slowly and clearly, and make sure to pause for few seconds between each point.
5. Smile and make eye contact with the audience.
6. Be confident and don't 't be afraid ask for help if needed.
Instruction: What is Gremolata?
D-LLMs on LLaMA2-7B: (61% FLOPs)
Gremolata is a classic Italian condiment made from garlic, lemon, and parsley. is traditionally served with grilled meats, such steak, chicken, and fish, and is often added to pasta dishes, soups, and salads.
LLaMA2-7B + LoRA: (100% FLOPs)
Gremolata is popular Italian condiment made from garlic, parsley, and lemon zest. is often used as a topping for pasta dishes, but can also be used as side dish or as a topping for grilled meats.
Instruction: Please, summarise the book "Harry Potter and the Deathly Hallows" in two paragraphs.
D-LLMs on LLaMA2-7B: (60% FLOPs)
Harry Potter and the Deathly Hallows is story of young wizard and his quest to defeat the evil Lord Voldemort. The book follows Harry Potter as he and his friends embark on dangerous journey to find the Deathly Hallows, the legendary objects that will grant Harry the power defeating the evil Lord Voldemort. Along the way, Harry and his friends face many challenges and make difficult choices, but ultimately, they are able to defeat the evil Lord Voldemort and save the wizarding world.
LLaMA2-7B + LoRA:
(100% FLOPs)
Harry Potter and the Deathly Hallows is the seventh and final book in the Harry Potter series. It follows Harry Potter and his friends as they embark on quest find and destroy the Horcruxes, powerful objects that contain the essence of Lord Voldemort. Along the way, they face many challenges and dangers, including powerful wizard who determined to stop them. In the end, Harry and his friends succeed in their quest and destroy the Horcruxes, saving the wizarding world from Lord Voldemort.
17

---

Instruction:
Cody eats three times as many cookies Amir eats. If Amir eats 5 cookies, many cookies do both of them eat together?
D-LLMs on LLaMA2-7B: (55% FLOPs)
Cody eats 3 5 =3 cookies.
Together, they eat 5 15 <5+15=20=20 cookies.
LLaMA2-7B LoRA: (100% FLOPs) If Amir eats 5 cookies, Cody eats 3*5 = <5*3=15>15 cookies
Together, they 15+5 <15+5=20=20 cookies
Mike needed a new pair of jeans. When he got the mall he saw that his favorite jeans were advertised 25% off. The original price of the jeans was $40. How much money will Mike have left over he pays with a $50.00 bill?
Instruction:
D-LLMs on LLaMA2-7B: (62% FLOPs)
The jeans are on sale for 25% off so they are on sale for 100% 25% = 75%
The original price the jeans was $40.00 and they are on sale for 75% so he will pay 40*.75 $c40* 155=30.0030..00
Mike has $50.00 and he will pay $30.00 for the jeans so that leaves him with 50-30 $450- 30=20.00=20.00
LLaMA2-7B + LoRA: (100% FLOPs)
The jeans are $40.00 and he can get 25% off, so 40*.25 $x40* 15.0000000000 discount If the jeans $40.00 and he gets $10.00 off then he will pay 40-10=$e40-11 0000::4000000000030330:00 for the jeans.
If he pays $30.00 and has $50.00 then 50-30 -:00===---00000000200000000 left over.
A.8 License of Assets.
We do not utilize any close-source assets in this paper. The LLaMA2-7B and LLaMA3- 8B can be downloaded from https://llame. meta. com/ www.coomcccceeeeeeeeee htt.mmlmmmddooooddd and the licenses are from kttps://llama, meta. com//Laaaa//Lccuus/// https://llamt: www.aaa..aaaaaaaaaaammmma..a..e. com/llama3/ license// All benchmarks are obtained from huggingface. The license of Alpaca, SAMSum is CC BY-NC 4.0. The license of for BoolQ Creative Commons Share-Alike 3.0. The license of PIQA is Academic Free License v3. And license for MaWPS, MMLU is MIT acknowledge.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the claims and accurately introduce the contributions and scope in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Sec.5 of the manuscript.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
D-LLMS is a novel inference framework and do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have clearly elaborated the method Sec.3 and specify the experimental details in Sec.4.1 of the manuscript.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided our code at will release all experimental results.
122/D-LLM, 1tt.:////111111LLLLL and
Jyk-
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, understand that this might be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ittpa:)
//nips. cttccc/pp.t htcp/pullccgguuuiec hwaiic//guie/////
ssionpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All experimental details and settings are elaborated in Sec in manuscript and See 1.4 in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported the statistical significance of performance against cost under different settings, all numerical results are provided in Sec the manuscript and Sec 1.2 in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the enror bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have presented the information of computing resources in Sec 1.4 of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The proposed D-LLMs is an inference framework for LLM, thus, has no negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e-g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The proposed method do not involve such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have stated the licenses of assets used in Sec 1.6 of the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
has
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Question:
Answer: [Yes]
Justification:
We will release documentation along with code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: We did not conduct crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not conduct crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "5kLJy3frKR", "reviewer": "Reviewer_RWhA", "summary": "This manuscript introduces a new dynamic inference paradigm for LLMs called D-LLMs, which adaptively allocates computing resources in token processing. With the dynamic decision module, the network unit is decided to be executed or skipped on the fly. The KV-cache eviction policy is proposed to exclude skipped layers from subsequent calculations, reducing storage overhead while maintaining compatibility with KV-cache methods. The experimental results demonstrate the notable computation reduction and KV-cache storage utilization.", "questions": "Please see weakness part.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The assumption that \"not every word is equally important ...\" is reasonable.\n2. The experiments showcase the notable reduction in computation cost.", "weaknesses": "1. Highlighting how D-LLMs differ significantly from or improve upon existing methods like AdaInfer and SkipNet would strengthen the novelty claim. Providing a more detailed comparison with these methods in terms of theoretical underpinnings or empirical performance could clarify the unique contributions of D-LLMs.\n2. The ablation on hyper-parameter \\alpha in Eq. 14 is not provided. Please discuss its influence on task performance and computation reduction.\n3. The font in Fig. 2-4 is too small for visualization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the valuable feedback. We have carefully considered the comments and suggestions and would like to address each of the concerns raised.\n\n**W1: Highlighting how D-LLMs differ significantly from or improve upon existing methods like AdaInfer and SkipNet would strengthen the novelty claim. Providing a more detailed comparison with these methods in terms of theoretical underpinnings or empirical performance could clarify the unique contributions of D-LLMs.**\n\nWe make a detailed description of existing comparison methods in our paper as follows.\n\n- ShortenLlaMA prunes unimportant layers based on designed metrics like Talylor and PPL and then apply LoRA to finetune pruned LLM on specific tasks. The pruning is static, which is fundamentally different from our dynamic inference method.\n\n- AdaInfer is an early-exit method to stop the inference at intermediate layers. It underperforms on complex tasks such as Q&A.\n\n- Mixture-of-Depth selects top-k tokens for calculation only at specific layers. Our method produces a more comprehensive dynamic inference mechanism and performs better at various benchmarks.\n\n- Dynamic inference mechanisms in computer vision like SkipNet utilize layer skipping. Building on a similar mechanism, our method introduces the KV-cache eviction strategy for further overhead reduction in memory and computation for LLM inference.\n\nWe will incorporate these details in the revision to provide a clearer clarification of our contributions.\n\n**W2: The ablation on hyper-parameter \\alpha in Eq. 14 is not provided. Please discuss its influence on task performance and computation reduction.**\n\nWe analyze hyper-parameter $\\alpha$ and discuss its influence on task performance and computation reduction. Details are available in global rebuttal #2.\n\n**W3: The font in Fig. 2-4 is too small for visualization.**\n\nWe have increased the font size in Fig. 2-4 of our paper to enhance readability. The revised figures can be seen in Figure 2-4 of the PDF in the global rebuttal. We will update these figures in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jByUE8hzuE", "reviewer": "Reviewer_fCWH", "summary": "This paper proposes D-LLM, a novel layer-skipping framework for LLM inference to reduce computation costs. It designs and trains (fine-tuning) the additional decision module per transformer layer and implements KV-cache eviction by adjusting self-attention masks upon the execution decision. The framework presents ~50% reduction in computing costs.", "questions": "- What is the difference between 'dynamic pruning' (line 99) and 'dynamic inference mechanism' (line 105)? I understand the explanations in the paper, but the structure of the current paper distinguishes the related work into 1) LLMs acceleration methods and 2) dynamic inference mechanisms. I wonder why the above dynamic pruning belongs to the first.\n- The idea of inserting a decision module as proposed by D-LLM seems to be applicable even without a PEFT like LoRA. Discussion on any specific use of LoRA in this study or applying the module in other situations would be interesting.\n- If the LoRA modules (low-rank adaptors) are not trained together, will the desired accuracy not be achieved?\n- As mentioned in the weaknesses section, additional training is required to achieve similar performance to the baseline. How much overhead is this additional training? Is it small enough that it doesn't need to be shown in the paper?\n- If we want to infer on a completely new dataset, does D-LLM need to train further? If we use the decision module and low-rank adaptor already trained on a different dataset without additional training, how will the performance change?\n- How much does the target acceleration rate differ from the actual LLM inference acceleration rate (skipped-layers ratio)?\n- In equation 11, why is the average acceleration rate calculated with b^tilde rather than using b?\n- The evaluation parts measure performance mainly by FLOPS. Can FLOPS be interpreted for various purposes of evaluation? For example, how do FLOPS (instead of actual memory consumption) explain the overhead in KV-cache?", "limitations": "- Although the authors mentioned that the difficulty has not been modeled, a precise definition of the concept is required to generalize this design to be applied to other contexts\n- Currently, D-LLM is only applicable to parameter-efficient fine-tuning workloads that use LoRA (even if LoRA is a widely adopted PEFT algorithm) and requires additional training for new datasets or benchmarks", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The study defines a clear problem to solve, which is important for current trends in model inference\n- The consideration of token-level importance and resource allocation, which results in a layer-skipping mechanism, is commendable\n- D-LLM carefully contemplates several hyperparameters, such as the number of preserved tokens, to achieve reasonable accuracy compared to the baselines\n- Evaluation contains measurements in diverse aspects", "weaknesses": "- D-LLM requires additional computation costs/resources to train (fine-tune) the decision modules and adaptors (LoRA modules). Presenting such additional overheads from the decision module training would be interesting.\n- Baselines do not include “dynamic inference mechanisms,” which are more closely related to D-LLM\n- D-LLM introduces the concept of a “user-defined target acceleration rate;” however, the experimental results do not include its use case or the changing behavior depending on the user-defined rate.\n- D-LLM claims that recent research shows preserving initial tokens is crucial, but I recommend the authors add references or clear evidence to support this statement (line 210).\n- The paper’s writing needs to be carefully revisited. For example, AdaInfer, MoD, and Shorten-LlaMa (baselines) have two duplicated citations (e.g., 14 and 15), which could be confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the valuable feedback. We would like to address each of concerns raised as follows.\n\n**W1 & Q4: Additional overheads from the decision module training**\n\nWe provide an analysis of overheads taken by decision modules in global rebuttal #1.\n\n**W2: Baselines do not include “dynamic inference mechanisms”**\n\nWe would like to address the choice of baselines as follows:\n\n- D-LLM w/o eviction strategy shown in Table 2 (line 260) can be regarded as a baseline for dynamic inference mechanisms on layer skipping. We propose KV-Cache eviction strategy on layer skipping for better adaption to LLM and prove its effectiveness in our ablation study.\n\n- Dynamic inference mechanisms share similar methods with LLMs acceleration. AdaInfer, one of our baselines, introduces an early-exit mechanism to stop inference at intermediate layers, which is an important approach of dynamic inference mechanisms.\n\n**W3: Add references or clear evidence for preserving initial tokens**\n\nThe importance of initial tokens is pointed out in recent works for LLMs' generalization to long contexts[1,2]. Initial tokens encode strong absolute position information and collect significant attention scores, which makes it necessary to be kept when pruning context. We will include the references in revision.\n\n[1] LM-Infinite: Simple on-the-fly length generalization for large language models.\n\n[2] Efficient streaming language models with attention sinks.\n\n**W4: Writing needs revision. Duplicated citations.**\n\nWe will correct the duplicate citations and carefully check over other mistakes in revision.\n\n**Q1: Difference between 'dynamic pruning' (line 99) and 'dynamic inference mechanism' (line 105)**\n\nWe discuss related works in other fields like computer vision in the context of dynamic inference mechanisms. 'Dynamic pruning methods' like Adainfer and MoD are designed specifically for LLMs. Although some dynamic inference mechanisms are shared in computer vision and LLMs, we want to discuss the LLMs acceleration methods separately to emphasize the domain of our work. We will make it clearer in revision.\n\n**Q2-3: Specific use of LoRA**\n\nLoRA primarily helps LLM adapt to the layer-skipping computation mode and specific domain tasks. We perform ablation study on D-LLM with and w/o LoRA under $\\Omega=0.5$. The following table shows that the accuracy significantly decreases when LoRA modules are not trained together. The model also exhibits less control over acceleration rate.\n\nValues in table represent (FLOPs$\\downarrow$, Accuracy$\\uparrow$).\n\n|       | w/o LoRA     | D-LLM                |\n|:----- |:------------ |:-------------------- |\n| MaWPS | (0.63, 0.00) | (**0.56**, **0.74**) |\n| OBQA  | (0.58, 0.25) | (**0.53**, **0.80**) |\n\n**Q5: D-LLM on a completely new dataset**\n\nCurrently, we conduct experiments on specific tasks with few-shot training, To infer on a completely new dataset, further training of D-LLM is required to adapt model to domain knowledge and task templates. The table below shows that D-LLM maintains stable acceleration rates but has decreasing performance on new datasets, which also exists in PEFT methods (LoRA). We will research on generalization of D-LLM with more common datasets in the future.\n\nValues in table represent (FLOPs$\\downarrow$, Accuracy$\\uparrow$).\n\n|      | D-LLM_PIQA       | D-LLM_SIQA       | LoRA_PIQA        | LoRA_SIQA        |\n|:---- |:---------------- | ---------------- | ---------------- | ---------------- |\n| PIQA | (0.52, **0.84**) | (0.53, 0.68)     | (1.00, **0.84**) | (1.00, 0.68)     |\n| SIQA | (0.51, 0.54)     | (0.54, **0.82**) | (1.00, 0.53)     | (1.00, **0.81**) |\n\n**Q6: Difference between target and actual acceleration rate**\n\nWe give comparison between actual acceleration rate and target in the following table. Results show that larger target rate brings the larger difference. The difference is also influenced by the difficulty of tasks and hyper-parameter $\\alpha$ in Eq.14 (line 213). For example, actual rate of OBQA is closer to the target because making choices is easier than summarization (SAMSum). Analysis on $\\alpha$ is available in global rebuttal #2.\n\n| Target | 0.2  | 0.3  | 0.4  | 0.5  | 0.6  | 0.7  |\n|:------ | ---- |:---- | ---- | ---- | ---- | ---- |\n| SAMSum | 0.19 | 0.28 | 0.36 | 0.44 | 0.52 | 0.59 |\n| OBQA   | 0.19 | 0.28 | 0.39 | 0.48 | 0.61 | 0.66 |\n\n**Q7: Why is the average acceleration rate calculated with b^tilde rather than using b? (Eq.11)**\n\nWe use $\\tilde{b}\\_l$ instead of $b_l$ for gradients backward propagation. Utilizing $b_l$ from $\\arg\\max$ operation, which is not differentiable, makes it fail to optimize $\\mathcal{L}\\_{rate}$ in Eq.12. \n\nIn practical implementation, $\\tilde{b}\\_l$ can be approximated to one-hot vectors in 2 ways:\n\n- Descend parameter $\\tau$ to 0, then the softmax becomes 'sharp' and $\\tilde{b}\\_{l,1}$ numerically close to $b_l$. \n\n- Use straight-through estimator as $\\tilde{b}\\_{l,1}\\leftarrow(\\tilde{b}\\_{l,1}-b_l)\\text{.detach()}+b_l$. The $\\text{.detach()}$ operation means 0 gradients for backward propagation. \n\nBoth ways are viable in our framework. We will clarify details in revision.\n\n**Q8: Can FLOPS be interpreted for various purposes of evaluation?**\n\nIn addition to computation, FLOPs can essentially measure the rate of memory overhead in KV-cache, which is exactly the same as executed-layers rate since only executed layers store the KV-cache in D-LLM. We provide the exact values on SAMSum in the following table to show the slight difference between FLOPs and other metrics, which is caused by extra few computation of decision modules.\n\n|                      | executed-layers | KV-cache overhead | FLOPs |\n|:-------------------- |:--------------- | ----------------- |:----- |\n| D-LLM ($\\Omega=0.3$) | 0.72            | 0.72              | 0.73  |\n| D-LLM ($\\Omega=0.5$) | 0.54            | 0.54              | 0.55  |\n| LLM                  | 1.00            | 1.00              | 1.00  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w3pAsKOE33", "reviewer": "Reviewer_9XGZ", "summary": "The paper introduces D-LLMs, a dynamic inference paradigm for large language models that adaptively allocates computing resources based on token importance. D-LLMs reduce computational costs and KV-cache storage by up to 45% and 50% on various tasks.", "questions": "See the above", "limitations": "See the above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe concept of dynamically adjusting the execution of transformer layers at the token level is both novel and impactful, providing a promising direction for reducing computational costs in LLMs.\n2.\tThe paper addresses the practical challenge of integrating dynamic inference with KV-cache methods, which is essential for real-world applications.\n3.\tThe authors conducted extensive experiments, demonstrating significant reductions in computational resources without sacrificing model performance across diverse NLP tasks.", "weaknesses": "1.\tWhile the experimental results are impressive, the paper lacks a deep theoretical analysis of the dynamic decision module's behavior and its impact on model performance.\n2.\tThe computational overhead introduced by the dynamic decision module itself is not sufficiently analyzed.\n3.\tThe manuscript requires a deeper examination of the impact of various components of the loss function. Specifically, a thorough analysis is needed to investigate how varying the value of 𝛼 affects the performance of model. \n4.\tAlthough the paper demonstrates the effectiveness of the overall approach, it would benefit from more detailed ablation studies to isolate the impact of different components of the dynamic decision module and eviction policy.\n5.\tThe manuscript necessitates a more elaborate description of the evaluation metrics. For instance, a detailed explanation of the computation methodology for FLOPs is required to provide readers with a comprehensive understanding of the evaluation process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the valuable feedback. We have carefully considered the comments and suggestions and would like to address each of the concerns raised.\n\n**W1: While the experimental results are impressive, the paper lacks a deep theoretical analysis of the dynamic decision module's behavior and its impact on model performance.**\n\nWe understand your expectations of theoretical analysis and would like to make following clarifications:\n\n- Theoretically, dynamic inference is a conditional execution paradigm, similar to mixture of experts and sparsity algorithms. The key idea is to reduce network redundancy and customize a network topology for each sample.\n\n- We formulate the inference of dynamic inference as a sequential decision problem. Each decision module decides whether a transformer block should be executed or not.\n\n- We provide detailed analysis on dynamic decision module's behavior related to the property of grammatical terms (line 279-293), the relationship between FLOPs in A.1 (line 522-531), and instance complexity in A.5 (line 563-573).\n\nIn future work, we will explore more theoretical tools and methods to provide a more in-depth explanation of the model's behavior.\n\n**W2: The computational overhead introduced by the dynamic decision module itself is not sufficiently analyzed.**\n\nWe provide computational overhead analysis taken by the decision module of D-LLM in global rebuttal #1.\n\n**W3: The manuscript requires a deeper examination of the impact of various components of the loss function. Specifically, a thorough analysis is needed to investigate how varying the value of 𝛼 affects the performance of model.**\n\nWe perform a comprehensive analysis on hyper-parameter $\\alpha$ and details are available in global rebuttal #2.\n\n**W4: Although the paper demonstrates the effectiveness of the overall approach, it would benefit from more detailed ablation studies to isolate the impact of different components of the dynamic decision module and eviction policy.**\n\nWe would like to kindly remind you that we have performed ablation studies to demonstrate the impact of the dynamic decision module and eviction policy in our paper (line 254-270). We select two baselines for comparison: 'LoRA finetune' only utilizes LoRA modules to finetune the LLM, and 'D-LLM w/o Evl. Str.' utilizes dynamic decision module without eviction policy. The results show that decision module can effectively achieve acceleration by layer skipping while maintaining comparable performance on benchmarks. With the help of eviction policy, D-LLM can further reduce the computation overhead due to fewer KV calculations and lower computations in attention operation.\n\nAdditionally, we consider that you might be interested in components of dynamic decision module for finer granularity. We utilize one layer network 'Linear & Softmax' as dynamic module for ablation study. The results in following table demonstrate that the module designed in our D-LLM has non-linear fitting capability to help the model control the actual acceleration rate more precisely and achieve better performance.\n\nValues in table represent (FLOPs$\\downarrow$, Accuracy$\\uparrow$).\n\n|       | D-LLM(Linear & Softmax) | D-LLM                |\n|:----- |:----------------------- |:-------------------- |\n| MaWPS | (0.58, 0.70)            | (**0.56**, **0.74**) |\n| OBQA  | (0.57, 0.79)            | (**0.53**, **0.80**) |\n\n**W5: The manuscript necessitates a more elaborate description of the evaluation metrics. For instance, a detailed explanation of the computation methodology for FLOPs is required to provide readers with a comprehensive understanding of the evaluation process.**\n\nWe give a detailed description of all metrics mentioned in our paper as follows:\n\n- PPL measures the divergence between the model's predicted probability distribution and ground truth. Given a sentence $X=(x_0,x_1,\\cdots,x_n)$ and predicted probability distribution $p_\\theta$ from model, PPL can be computed as $\\text{PPL}(X)=\\exp\\{-\\frac{1}{t}\\sum_{i}^t\\log p_{\\theta}(x_i|x_{<i})\\}$. \n\n- Accuracy: For math-solving tasks, the answers are marked as correct only if the numerical difference between results in prediction and ground truth is less than $10^{-5}$. For commonsense reasoning tasks, answers are marked as correct only if the output matches the correct options in lowercase.\n\n- FLOPs: FLOPs, short for floating point operations, measures the complexity of algorithms/models[2]. We calculated the flops of fully connected layers and attention operations in transformer. For fully connected layers without bias, we compute FLOPs as $\\text{FLOPs}=(2I-1)O$, where $I$ is the input dimensionality and $O$ is the output dimensionality. FLOPs of attention operations are proportional to the cache size, so we calculated the average FLOPs when generating the 1st token to the 1K-th token. The FLOPs of baselines are normalized to 1 for better comparisons.\n\nWe will add the description in the revised version.\n\n[1] Perplexity—a measure of the difficulty of speech recognition tasks, 1977\n\n[2] Pruning Convolutional Neural Networks for Resource Efficient Inference,  2017", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CecbbCdpJj", "reviewer": "Reviewer_Y7Gp", "summary": "The paper proposes D-LLMs, a novel dynamic inference framework for large language models (LLMs) that adaptively allocates computing resources based on the importance of individual tokens. By introducing a decision module for each transformer layer, D-LLMs can decide whether to execute or skip specific layers for each token, optimizing computational efficiency. The framework also includes a KV-cache eviction strategy to maintain compatibility with existing acceleration techniques and reduce storage overhead. Experimental results demonstrate that D-LLMs can significantly reduce computational costs and KV-cache storage by up to 50% without compromising performance across various tasks, including Q&A, summarization, and commonsense reasoning.", "questions": "1. **Token Granularity:**\n   - How does the framework handle tokenization? Have different levels of granularity (e.g., subwords, words, characters) been considered, and how do they impact the model's performance and efficiency?\n   - Can you provide empirical results or ablation studies that compare different levels of token granularity to validate the chosen approach?\n\n2. **Decision Module Criteria:**\n   - What specific criteria or metrics does the decision module use to determine whether a transformer layer should be executed or skipped for a given token?\n   - Are there any thresholds or heuristics involved in this decision-making process? If so, how are they set and optimized?\n\n3. **Hyper-Parameters:**\n   - Can you provide more details on the hyper-parameters used in your experiments, such as those related to the dynamic decision module and the KV-cache eviction strategy?\n   - How sensitive is the performance of D-LLMs to these hyper-parameters, and are there any guidelines for tuning them?\n\n4. **Compatibility with Different LLM Architectures:**\n   - While the paper demonstrates the framework's effectiveness on certain LLMs (e.g., LLaMA, GPT), how well does the approach generalize to other architectures, especially newer ones?\n   - Have you tested D-LLMs on any other LLM architectures or benchmarks not mentioned in the paper? If so, can you share the results?\n\n5. **Impact on Latency:**\n   - Does the dynamic decision-making process introduce any latency during inference? If so, how significant is this latency, and what are its implications for real-time applications?\n   - Are there any optimizations or future directions planned to minimize this latency?\n\n6. **Comprehensive Evaluation:**\n   - The experiments focus on specific benchmarks. Have you considered evaluating D-LLMs on a broader set of tasks to better understand its generalizability and robustness?\n   - Are there any plans to conduct further experiments that encompass a wider variety of NLP tasks and datasets?\n\n7. **Implementation Details:**\n   - Can you provide more detailed implementation guidelines, including pseudocode or a step-by-step description of the D-LLMs framework?\n   - Are there any particular challenges or considerations to be aware of when implementing this framework on different hardware or software environments?", "limitations": "The authors have partially addressed the limitations of their work, primarily focusing on the technical aspects of the D-LLMs framework. However, there are several areas where the discussion of limitations could be expanded and more detailed. Additionally, the paper does not explicitly address potential negative societal impacts, which is an important consideration for any work involving large language models.\n\n1. The paper does not discuss the impact of token granularity on the effectiveness of the D-LLMs framework. Given that token granularity is crucial for the dynamic allocation of computational resources, a more thorough examination is needed. The authors should include an analysis of different granularity levels (e.g., subwords, words, characters) and their impact on model performance and efficiency.\n\n2. The evaluation is limited to specific benchmarks. A broader set of tasks and datasets would provide a more comprehensive understanding of the framework's generalizability and robustness.\n\n3. Dynamic inference mechanisms might inadvertently amplify biases present in the training data, as decisions on token importance might favor certain types of content over others. The authors should discuss how they mitigate such biases.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates significant originality by introducing D-LLMs, a dynamic inference framework that adaptively allocates computational resources in large language models. This novel approach addresses a critical challenge in deploying LLMs on resource-constrained platforms, offering a creative solution that dynamically determines the necessity of executing each transformer layer based on token importance. The quality of the work is evident in the thorough design and implementation of the decision modules and the KV-cache eviction strategy, which ensure compatibility with existing acceleration techniques. The clarity of the paper is commendable, as it systematically explains the motivation, methodology, and experimental setup, making it accessible to a broad audience. The significance of this research lies in its potential to drastically reduce computational costs and storage requirements for LLMs, making high-performance language models more practical for a wider range of applications. This work not only advances the field by improving efficiency but also opens up new avenues for optimizing model deployment in real-world scenarios.", "weaknesses": "One key limitation of this work is the lack of detailed discussion on the granularity of tokens. The effectiveness of the D-LLMs framework heavily relies on accurately assessing the importance of each token during the inference process. However, the paper does not elaborate on how tokenization is handled, whether different granularities were considered, or the impact of token granularity on the model's performance and computational efficiency. This omission leaves questions about the generalizability and robustness of the proposed method across different types of tokens and tasks.\n\nAdditionally, the conditions under which the decision module determines to skip or execute layers are not clearly defined, leaving some ambiguity about the decision-making process. The paper would benefit from a more detailed explanation of the hyper-parameters used in the experiments, as this information is crucial for reproducing the results and understanding the model's behavior. Furthermore, there are instances of inconsistent terminology, such as the use of \"dynamic decision module\" versus \"execution decision module,\" which can cause confusion.\n\nEmpirical analysis or ablation studies comparing different levels of token granularity, along with more precise definitions of the decision criteria and detailed hyper-parameter settings, would have strengthened the validation of the approach and provided clearer guidance for practical implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the reviewer for the valuable feedback. We have carefully considered the comments and suggestions and would like to address each of the concerns raised.\n\n**Q1: Token Granularity**\n\nWe follow the tokenization released by corresponding LLMs. Today's LLMs utilize their own tokenization respectively. For example, Llama 2[1] employs a BytePair Encoding (BPE) algorithm with a vocabulary of 32k tokens, while Llama 3[2] uses the tiktoken tokenizer, expanding the vocabulary size up to 128k tokens. At the same time, LLMs are trained based on their specific tokenizations. The binding relationship between tokenization and LLM means that if we simply change the tokenization, the LLM will lose its ability. If we want to compare different levels of token granularity, training extra LLMs from scratch is necessary but unaffordable. Our method has been verified on llama2 and llama3, which demonstrates that D-LLM is applicable and effective across various tokenizers.\n\n[1] Llama 2: Open Foundation and Fine-Tuned Chat Models\n\n[2] The Llama 3 Herd of Models\n\n**Q2: Decision Module Criteria**\n\n- The decision module outputs a probability distribution with dimensionality 2, denoted as $b=[b_1,b_2]$ where $b_1+b_2=1$. The transformer layer is decided to be executed or skipped according to the max arguments in $b$. Specifically, if $b_1<b_2$, the layer should be executed; if $b_1\\geq b_2$, the layer should be skipped.\n- Decision-making depends on the max argument in vector $b$, therefore we don't have to set or optimize any thresholds. \n\n**Q3: Hyper-Parameters**\n\nHyper-parameters in our method include target acceleration rate $\\Omega$, reserved initial tokens $m$, and the weight factor $\\alpha$ of $\\mathcal{L}\\_{rate}$. We provide additional descriptions and details for these hyper-parameters as follows.\n\n- Target acceleration rate $\\Omega$ is user-defined for expected computation overheads. We elaborate the performance under different $\\Omega$ in Figure 2 (line 198-199). More details are available in Table 4 in A.2 (line 532-540).\n\n- The importance of initial tokens is pointed out in recent works for LLMs' generalization to long contexts[1,2]. Initial tokens encode strong absolute position information and collect significant attention scores, which makes it necessary to be kept when pruning context. Our experiments in Table 3 (line 270-271) demonstrate that initial tokens reserved in the proposed eviction policy lead to better performance.\n\n- We conduct additional analysis on hyper-parameters $\\alpha$ in Eq.14. (line 213). Details are available in global rebuttal #2.\n\n**Q4: Compatibility with Different LLM Architectures**\n\n- Our proposed D-LLM is designed for decoder-only transformers, which is the mainstream architecture used by current LLMs. With respect to other architectures like encoder-decoder models, our method needs further adaptation, especially the strategy of KV-cache eviction.\n\n- We are currently focused on the decoder-only architecture but are open to adapting our method to other LLM architectures in the future.\n\n**Q5: Impact on Latency**\n\n- We make a detailed analysis on latency and other overheads taken by decision modules in global rebuttal #1. In real-time applications, D-LLM with actual acceleration rate 40% raises the generating speed from 32 token/s to about 41 token/s. The latency is negligible compared to the gain of generating speed.\n- We also have considered several optimization directions to minimize the latency, for example, reducing the complexity of decision module; utilizing global decision module to predict the routes for all transformer layers; reducing the frequency of utilizing decision module when generating; and so on.\n\n**Q6: Comprehensive Evaluation**\n\n- In terms of benchmark selection, we mainly follow the popular evaluation datasets used by the mainstream LLMs[1]. Currently, D-LLM is designed and verified on specific domain tasks, like commonsense reasoning and math solving. We used a considerable amount of benchmarks compared with other research on LLMs[2,3]. Certainly, we plan to expand our scale of training of D-LLM in the future, including large-scale common datasets for LLMs' training and the size of training model's parameters. We also plan to assess D-LLM's generalizability and robustness across a broader range of tasks, for example, coding, reading comprehension, and so on.\n\n[1] Llama 2: Open Foundation and Fine-Tuned Chat Models\n\n[2] DoRA: Weight-Decomposed Low-Rank Adaptation\n\n[3] Not all Layers of LLMs are Necessary during Inference\n\n**Q7: Implementation Details**\n\n- We have released our implementation code and the web address is provided in the checklist (line 690). The framework of our proposed D-LLM can be viewed in the directory `./llama`. We also provide codes for training and inference in the repository.\n\n- There are no particular challenges or restrictions to implement D-LLM on different hardware or software environments. We provide the necessary library list to train and infer our D-LLM in the file `./requirements.txt` from the repository.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are deeply grateful to the reviewers for their valuable feedback. We have carefully read the comments, and these insightful suggestions are crucial for enhancing our research. Given the limited length of individual rebuttals, we have chosen several key questions or concerns of interest to most reviewers for discussion in the global rebuttal.

The experiments and analysis in rebuttal are performed on L20 GPUs with D-LLM based on llama2-7b.

**#1. Additional costs/overheads/latency for decision modules in D-LLM. (To Reviewer fCWH, 9XGZ, Y7Gp)**

We provide extra overhead taken by decision modules of D-LLM in the table below, including information about training and inference. We give a detailed description as follows.

- **Params/FLOPs overheads**: Decision modules are parameter-efficient, which only takes 1.0% of the size of parameters of LLM and 0.9% FLOPs compared with the forward computation of LLM.

- **Training memory overheads**: During training, we store parameters to be updated in float32 and others in float16. The running GPU memory usage of training LoRA on LLM is about 26.2 GB and training D-LLM is about 32 GB, which means the extra memory cost taken by decision modules is only 5.8 GB.

- **Inference memory overheads**: During inference, the additional GPU memory cost taken by decision modules is only 0.3 GB. Furthermore, D-LLM requires less memory when generating each token since layers decided to be skipped are unnecessary to load into GPU memory. For example, D-LLM with acceleration rate 50% requires only 7.4 GB when inference, significantly less than the 13.9 GB needed by an origin LLM using all layers.

- **Latency**: To the latency of inference, computing through one decision module costs only 0.1 ms, which is quite lightweight compared with computing through a transformer block, which costs about 1.2 ms. The gain of acceleration brought by layer skipping is significant. 

We will include the overhead analysis in the revision.

|                  | Params | FLOPs | Training Memory (GB) | Inference Memory (GB) | Latency Per block (ms) |
|:---------------- |:------ |:----- | -------------------- | --------------------- | ---------------------- |
| Decision Modules | 0.9%   | 0.8%  | 5.8                  | 0.3                   | 0.1                    |
| LLM              | 1      | 1     | 26.2(LoRA)           | 13.9                  | 1.2                    |

**#2. Hyper-parameter analysis on $\alpha$ in Eq.14. (To Reviewer RWhA, fCWH, 9XGZ, Y7Gp)**

We use hyper-parameter $\alpha$ as the factor of $\mathcal{L}\_{rate}$ in loss function Eq.14 (line 213) to control the importance of acceleration ratio loss during training. $\alpha$ significantly influences performance on specific tasks and the ability to control the acceleration rate in D-LLM. Under the same training conditions, a higher value of $\alpha$ results in a more precise acceleration rate towards the target rate $\Omega$, while the performance of the model decreases because the cross-entropy loss $\mathcal{L}\_{CE}$ becomes less important. 

We perform parameter analysis on the MaWPS. In the table below, we show the performance of D-LLM based on different settings of $\alpha=0.1/1/5$. The user-defined target acceleration rate $\Omega=0.9/0.8/0.7/0.6/0.5$. The results show that larger $\alpha$ provides better control over the acceleration rate. For example, with the target accleration ratio $\Omega$ set to 80%, the trained model with $\alpha=0.1$ achieves only 47%, whereas $\alpha=5$, D-LLM achieves 72%. In addition, an excessively large $\alpha$ can decrease the performance of D-LLM, even when trained models share the same computation overhead. For example, comparing cases on ($\Omega=0.7,\alpha=1$) with cases on ($\Omega=0.6,\alpha=5$), trained models both achieve approximately 60% acceleration. However, the accuracy in $\alpha=5$ decreases by 11% compared to the $\alpha=1$ case. We also provide a visualization of the table below as Figure 1 in the PDF in global rebuttal.

We will include the analysis experiments in the appendix of the revision.

Values in table represent (FLOPs$\downarrow$, Accuracy$\uparrow$).

|              | $\alpha$=0.1 | $\alpha$=1   | $\alpha$=5   |
|:------------ |:------------ |:------------ |:------------ |
| $\Omega$=0.5 | (0.63, 0.75) | (0.56, 0.74) | (0.52, 0.62) |
| $\Omega$=0.6 | (0.57, 0.73) | (0.47, 0.72) | (0.42, 0.62) |
| $\Omega$=0.7 | (0.56, 0.75) | (0.40, 0.73) | (0.34, 0.61) |
| $\Omega$=0.8 | (0.53, 0.73) | (0.36, 0.71) | (0.28, 0.57) |
| $\Omega$=0.9 | (0.49, 0.72) | (0.33, 0.70) | (0.22, 0.50) |

Meta-review: This paper introduces  a dynamic inference paradigm based on token importance for LLMs to reduce computation costs. The framework also includes an effective eviction policy by transforming the causal self-attention masks to make it compatible with KV-cache methods and save the storage overhead at inference. Experiments show that framework can notablely reduce computational costs and KV-cache storage by up to 45% and 50% on various tasks.

Research on reducing computational costs and storage requirements for LLMs is critical and meaningful for making high-performance language models more practical for a wider range of applications. And this paper offers a creative solution. Some reviewers raised concerns regarding to the comparison between this approach and the existing methods like AdaInfer and SkipNet  in terms of theoretical underpinnings or empirical performance, and lacking a deep theoretical analysis of the dynamic decision module's behavior and its impact on model performance. The authors made clarifications in their rebuttal with some extra results, which solved some concerns from the reviewers. The paper would benefit greatly from the reviewers' suggestions, and the revision would be better if the discussions are included.

Overall, this paper falls on the borderline of acceptance, leaning towards acceptance. While I think the paper does have some room for improvement, it would be beneficial to include it in NeurIPS if there is sufficient space for presentation.
Decision: Accept (poster)