Let LLM Tell What to Prune and How Much to Prune
Authors: Mingzhe Yang
Sihao Lin
Changlin Li
Xiaojun Chang
Venue: ICML 2025 poster (2025)
Keywords: large language models
model compression
network pruning
structured pruning
Primary area: deep_learning->large_language_models
TL;DR: We propose a pruning method that targets multiple LLM modules with dynamic pruning ratios by quantifying the complex interactions within LLMs,  achieving better trade-off between efficiency and performance.

Abstract: Large language models (LLMs) have revolutionized various AI applications. However, their billions of parameters pose significant challenges for practical deployment. Structured pruning is a hardware-friendly compression technique and receives widespread attention. Nonetheless, existing literature typically targets a single structure of LLMs. We observe that the structure units of LLMs differ in terms of inference cost and functionality. Therefore, pruning a single structure unit in isolation often results in an imbalance between performance and efficiency. In addition, previous works mainly employ a prescribed pruning ratio. Since the significance of LLM modules may vary, it is ideal to distribute the pruning load to a specific structure unit according to its role within LLMs. To address the two issues, we propose a pruning method that targets multiple LLM modules with dynamic pruning ratios. Specifically, we find the intrinsic properties of LLMs can guide us to determine the importance of each module and thus distribute the pruning load on demand, i.e., what to prune and how much to prune. This is achieved by quantifying the complex interactions within LLMs. Extensive experiments on multiple benchmarks and LLM variants demonstrate that our method effectively balances the trade-off between efficiency and performance.

Let LLM Tell What to Prune and How Much to Prune
Mingzhe Yang" Sihao Lin
Changlin Li
Xiaojun Chang
Abstract
veloping effective methods compress LLMs has become valuable research direction.
Large language models (LLMs) have revolution- ized various Al applications. However, their bil- lions of parameters pose significant challenges for practical deployment. Structured pruning is hardware-friende compression technique and receives widespread attention. Nonetheless, exist- ing literature typically targets a single structure of LLMs. We observe that the structure units of LLMs differ in terms of inference cost and func- tionality. Therefore, pruning single structure unit in isolation often results in imbalance be- tween performance and efficiency. In addition, previous works mainly employ prescribed prun- ing ratio. Since the significance of LLM modules may vary, it is ideal to distribute the pruning load to a specific structure unit according to its role within LLMss To address the two issues, we pro- pose pruning method that targets multiple LLM modules with dynamic pruning ratios. Specifi- cally, we find the intrinsic properties of L.MSs can guide us to determine the importance of each module and thus distribute the pruning load on de- mand, i.c., what to prune and how much to prune. This is achieved by quantifying the complex inter- actions within LLMs. Extensive experiments on multiple benchmarks and L.LM variants demon- strate that our method effectively balances the trade-off between efficiency and performance.
Network Pruning (Han et al., 2015) is promising method to reduce model complexity by pruning the non-essential net- work connections. Traditional pruning strategy (i.e., weight magnitude) may not be suitable to LLMs (Yin et al., 2024; Muralidharan et al., 2024), which typically requires fine- tuning (Gordon et al., 2020; Jaiswal al., 2023; Xia et al., 2023) to restore the performance. Since retraining LLMs could be prohibitively expensive, (Frantar & Alistarh, 2023) propose SparseGPT, the first accurate one-shot unstructured pruning method without any post-training, Compared to unstructured pruming, structured pruning (An et al., 2024; Song et al., 2024; Ashkboos al., 2024; Zhong et al., 2024; Ma al., 2023; Men al., 2024) is more friendly for de- ployment, which establishes criterion to determine the importance of specific structures, c.g., blocks and layers, and remove the uninformative components accordingly.
Nonetheless, existing literature on structured pruming only focuses single structure unit in LLMs, such as transformer block (Song et al., 2024; Men et al., 2024), attention & MLP layers (Zhong et al., 2024) and the rows columns of weight ma- trices (An et al., 2024;
Lo0 Parpinarty (PL)
30
RoweCoolunn
Black
Figure 1. Dilemma between per- formance and efficiency. A faster pruning strategy (speed f) comes with inferior performance (PPL 4). Ashkboos et al., 2024; Ma et al., 2023; Ling et al., 2024). Different structure units vary from parameter counts and functionality, contributing differently to inference speed versus performance. An example is illustrated Figure I where three structure units, accounting for 10% parameters, of LLaMa2-7B are randomly pruned. The result shows that there is free lunch for structured pruning when targeting certain LLM module. Specifically, faster strategy (e.g.. block-wise pruning) comes with worse performance and vice versa. Consequently, previous methods could lead to an imbalance between throughput versus performance, as shown in Figure 2.
1. Introduction
Large language models (LLMs) have demonstrated excep- tional performance in the field of natural language process- ing (Mann al., 2020; Thoppilan al., 2022; Zhu al., 2024). Nonetheless, the billions of parameters (Zhao al., 2023) characteristic of LLMs substantially increase storage requirements and computational overhead. Therefore, de-
Equal contribution 'University of Science and Technology of China FRMIT University Stannfor University. Correspondence to: Xiaojun Chang <xjchang@ edd...dd..e...
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In addition, a majority of arts (Ashkboos et al., 2024; Song et al., 2024; Zhong et al., 2024; Lin al., 2024; Men et al.,

---

Let LLM Tell What to Prune and How Much to Prune
measuring their interactions within the LLMs. We then dispatch the pruning load to the structured units within the block by entropy consideration.
OURS)al)
FLoPirowsscolumms
2000
Noccm/nnet/wppen) SLEB(sleck))
100
SLEBIblecks)
Throughout (Token/s)
Our method has been rigorously validated across dif- ferent benchmarks, demonstrating exceptional perfor- mance without relying on any fine-tuning or retraining.
1500
Perplexity (PPL)
60
1000
2. Related Work
40-
500
To reduce the inference cost of large language models and increase their practical applications, numerous recent stud- ies have focused on model compression techniques. there have been many recent works compressing models. A majority of model compression techniques fall into one of three categories: distillation (Gu et al., 2023; Agarwal et al., 2023; Ko et al., 2024; Padmanabhan et al., 2024), prun- ing and quantization (Frantar et al., 2022: Chee et al., 2024; Egiazarian et al., 2024; Zhang et al., 2024a). In the paper, we specifically focus the pruning of the LLMs. Pruning can be categorized into unstructured pruning, semi-structured pruning and structured pruning.
50%
Batch Size
Pruning Ratio
(a) Inference Speed
(b) Perplexity Result
Figure 2. Compare different structured pruning methods. (a) We compare the inference speed of pruned model using different struc- tured pruning methods. (b) We evaluate the perplexity performance of different structured pruning methods.
2024) prune LLMs with a prescribed ratio in each unit For instance, SLEB (Song et al., 2024) uses a binary operator to determine whether to prune an entire (100%) block. Al- though it achieves profound throughput (Figure 2a), it is less flexible scale up the pruning ratio (Figure 2b). In parallel, SliceGPT (Ashkboos et al., 2024) evenly prunes the matrix weights of each block. As the significance of each unit varies (Zhang et al., 2022), the ideal pruning ratio dis- tributed for each unit might vary. Hence, such a prescribed proposal might be less ficxible to scale up and limits the model efficiency considering performances.
Unstructured pruning and semi-structured pruning. Un- structured pruming involves removing individual elements from the weight matrix without regard to the network's structure, which allows the model achieve a high degree of sparsity. However, this approach does not alter the matrix size and the inference speed remains nearly identical to that of the original model. To address this limitation, unstruc- tured pruning can be combined with N ; M sparsity (Mishra et al., 2021), a technique known as "stmi-structured prun- ing." In N: M sparsity, at most N out of every M contiguous weights are allowed remain non-zero. SparseGPT (Fran- tar & Alistarh, 2023) takes different approach by employ- ing the OBS technique (Hassibi al., 1993) for pruning GPT-family models, marking it as the first method capa- ble of effectively pruming LLMs with 10-100 billion or more parameters. RIA (Zhang et al., 2024b), another semi- structured pruning method, also considers both weights and activation. However, unlike Wanda, it proposes using the relative magnitude of the weights rather than their absolute values. Although semi-structured pruning can largely pre- serve model performance at high sparsity, its acceleration benefits are still heavily dependent on NVIDIA Ampere GPU architecture (Mishra et al., 2021).
To address the two aforementioned issues, we propose hierarchical strategy to simultancously target differrntt struc- ture units with a dynamic pruning ratio. In the first stage, we allocate the pruning proportion for each block by con- sidering their importance. Specifically, we rank each block by measuring their interactions within the LLMs. Subse- quently, in a transformer block, we dispatch the pruning load to different structure units considering their entropy. We use simple clustering approach to mitigate the computation complexity. Our approach not only ensures significant im- provements in the inference speed of the pruned model but also maximally preserves the original model's performance. Our contributions are summarized as follows:
Structured pruning. Unlike unstructured pruning, which removes individual elements without regard to the network's structural integrity, structured pruning specifically targets and removes entire components modules within LLMs. L.L.M-runne (Ma et al., 2023) is structured pruning tech- nique that assesses the importance of module groups in LLMs by analyzing gradients and activations, systematically climinating less significant components. SliceGPT (Ashk- boos et al., 2024) employs a structured pruning strategy
We propose a structured pruning framework to simulta- neously target different structure units of LLMs, which balances the trade-off between inference speed and model performance.
We introduce a hierarchical strategy to prune different LLM modules with a dynamic ratio. In the first stage, we determine the pruning proportion of each block by
2

---

Let LLM Tell What to Prune and How Much to Prune
similar to weight slicing (Li et al., 2021; 2022), by intro- ducing embedding dimensionality reduction matrices for LLMs. These matrices facilitates the identification and elimination of less critical component. FLAP (An et al., 2024) introduces the concept of "fluctuation measurement" to assess the importance of rows&columns in the weight ma- trix. BlockPrune (Zhong et al., 2024) is a dynamic pruning method that masks attention and MLP layers sequentially, recording the PPL results. The layer with the lowest PPL is removed, and the process continues until the target pruning ratio is achieved. SLEB (Song et al., 2024) also proposes an iteration pruning method to assess the impact of cach block based on the token prediction results of the LLMs. In each iteration, the block with the least impact on the prediction results is removed. ShortGPT (Men et al., 2024) proposes metric called Block Influence (BI), which quantifies the changes in the hidden state of each block, to measure of block importance.
blocks and preserve the essential blocks. That is, want to formulate metric to measure the importance of a trans- former block.
Measuring interactions of blocks. We rank the blocks by capturing their complex interactions within the LLM. Intuitively, if the model output is highly correlated with block, this block is identified as important; otherwise uninformative. To this end, we resort to the concept of transfer entropy (TE) (Schreiber, 2000), which quantifies the information transfer between two network components. Transfer entropy is formally defined as:
TE H(Xoa)) H( out Mask (block)k)o
(I)
where (Xoot)) represents the original entropy of the fi- nal output layer, and (Xu Mask (Mlock))))))) denotes the entropy of final output layer after masking the 7th block. Specifically, when block is masked, Equation (1) provides comprehensive evaluation of the impact on the hidden state of the L.M output. Thus, TE can measure the sig- nificance of block; on the model output and thus reflect its importance, i.c., block is identified as important if the TE metric changes drastically. Consequently, we use the TE metric to guide the pruning proportion allocated to the blocks. Similarly, Equation (1) can be applied to attention or MLP layers. Figure 3a and Figure 3b illustrate the trans- fer entropy w.r.t. each block and layer of the LLaMA2-7B model. The transfer entropy of blocks varies dramatically across different orders of magnitude. Hence, rather than a static prescribed ratio, it is necessary to assign pruning ratio to each block based on its dynamic interaction within the LLM.
Yet, existing pruning methods typically focus on a sin- gle structure unit within LLMs, but such single-structure- oriented pruning strategies struggle to effectively balance model performance and efficiency after pruning. Moreover, existing pruning methods typically assign the same pruning ratio to each block without accounting for the varying im- portance of individual blocks within the LLMs. In contrast, we consider information entropy as a pruning criterion, en- abling a unified pruning approach across all structure units with a dynamic ratio. Our approach not only maximizes the retention of the model's 's original performance but also significantly enhances inference speed.
3. Method
In this section, we propose a systematic pruning framework grounded in information-theorete principles which simul- taneously targets all the LLM modules. In Sec. 3.1, we first introduce the metric of transfer entropy to analyze the interaction among blocks in LLMs. This enables to dy- namically determine the pruning ratio for each block, and perform pruning on coarse -raiiee structure units such as blocks and layers in Sec. 3.2. Then, we further allocate the pruning load within each block and perform fine- -raigraided pruning of the weight matrix rows&columms based on the information entropy of individual structure units in Sec. 3.3. Finally, in Sec. 3.4, our method use the bias compensation to enhance the performance of the pruned model without the need of post-training or any fine-tuning.
Entropy quantification. For a distribution p(z), its entropy can be written as:
H(X) = - log())) x EX,
(2)
where x represents the hidden state in the feature space of transformer unit X. In prior work, such as (Sirignano & Spiliopoulos, 2020; Sun et al., 2022), it assumed that p(x) follows Gaussian distribution, i.c. X N(H, Therefore, Equation (2) can be rewritten as:
H(X) -Ellog N o²]]
= -EE log (2Ta2)-1/2 exp (x - 11)2
3.1. Quantifying the Importance of Blocks
202
Since the functionalities of network units would vary (Zhang et al., 2022), we propose to assign pruning proportion to each block according to their role in the LLMs. We tend to assign a larger pruning proportion to the uninformative
log o + log(2x) + 1
1
(3)
2
2'
where a is the standard deviation of X.
3

---

Let LLM Tell What to Prune and How Much to Prune
30000
23880
Alterdien
Treeater Entmeey
20000
Trannfer Entraay
Taauter SHUiep
20000
1000
3001
Meck
(a) Block transfer entropy
(b) Layer transfer entropy
(c) Pruning ratio of each block
Figure The transfer entropy LLaMA2-7b's block and layer. (a) We sequentially mask each block the LLaMA2-7B model and compute the transfer entropy of the output layer for each masked configuration. (h) We sequentially mask each layer of the I.LaMA2-7B model and calculate the transfer entropy for each corresponding layer. (c) The pruning ratio of each block using Algorithm 1.
3.2. Block-wise Pruning Ratio
where represents the operation to obtain the number of elements in a set.
Grouping blocks based on TE. We store the transfer en- tropy associated with each block (Figure 3a) in set S and divide them into three sub-sets: St. SM and Sh standing for low, medium and high TE set, based on the statistic of the set S:
Depth-First Search compensation process. To compen- sate for the leftover, we should increase the pruning ratios of other blocks. As Figure 5 shown, initially, we select the top Shr| blocks with the least TE from set St. since they have least impact on the hidden state of LLM and place them into set F, assigned with the pruning ratio (i.e., completely pruned) and mapped Pr. The blocks assigned a prun- ing ratio of have their corresponding layers completely removed. Therefore, these layers in set A will be classified as unavailable part. Next, the top |Sar layers with the least TE are selected from the available part from set A. These layers will be completely pruned, and their corresponding blocks are divided to the extensively pruned set E, with their pruning ratio increased
S5 = {s < Ti};
S =
SM = {s Ti S s, S.T.
(4)
SH = {5/|5; > Th).
where s; represents the transfer entropy of 7th block, Ti Ps os and Th Ps + a os are two thresholds with the scaling factor a. Here Ma M and o, = - V - =1)² are the mean and the standard deviation of the set S, N represents the number of blocks in LLM. Similarly, we sort the transfer entropy of the attention and MLP layers (Figure 3b) in set A.
Plow mapped to Pg. For the remaining blocks that have not yet been assigned initial pruning ratio, we incorporate them into set R. Finally, we need to select according blocks from F and È to compen- sate for the leftover. How to select according blocks can be reformulated as the following optimization problem:
Hybrid pruning strategy. Let P denote the set of pruning ratios. We allocate the pruning ratio each block according to their importance in Equation (4): S + P. Since our pruning strategy involves multiple LLM modules simultane- ously, we have five pruning types in general: a) Pz: blocks are preserved with zero pruning ratio; b) Pp: blocks are fully pruned: <) Pg: attention or MLP layers are completely pruned; d) Pp: blocks are pruned with lower pruning ratio; e) PR: the resting blocks with an adjusted ratio.
arg
min ACF, MC || Peompensaaiinn - Piefover
s.t. Peompensation Pleftovers
(6)
where
Prompensation [C Piarget) + [M Pharger
Plowe)
Block-wise pruning. We preserve the blocks in Sy as they are the essential blocks in LLM identified by transfer entropy, ie.. SH + Pz. The blocks in SM would be assigned with a pruning ratio Plow lower than the target ratio Prorget, as they are the relatively essential blocks in LLM, i.e., SM + Pp- To accommodate the pruning requirement, there would be leftover Plefooer that is redistributed to the blocks in Sy:
C and M represent subsets of blocks selected from sets F and ε, respectively. This problem can be efficiently solvable using a Depth-First search algorithm. This opti- mization problem may yield multiple solutions To achieve optimal inference speed, we select the solution with the maximum [+| as the final result. The unselected blocks in F and E are returned to R. with their pruning ratios reassigned accordingly.
Dlefover
|SH| Diarger + [SM] (Puaget Piow):
Dynamic ratio assigning. For the remaining blocks in

---

Let LLM Tell What to Prune and How Much to Prune
W
X
2
3
-1
3
5
-2
2
4
H
W
-2
3
0
6
5
16
8
1.77
1.51
33611
2.46
1
2
3
1
4
132
0
Prune y
2
24
|X|_
2
1
8
4
Figure 4. Illustration of our proposed method based on information entropy. (Left): A weight matrix W and the Le norm of the activation value ||| (Middle): Multiply the absolute value of weight matrix by the To2 of the activation value. (Right): Calculate the information entropy of each column, perform clustering using the K-means algorithm, and represent different clusters with distinct colors, where lighter colors indicate lower information entropy for the corresponding cluster.
R unallocated
Algorithm Block-wise pruning ratio assignment
Stepl:
umav aiblle
Input: an original LLM, target pruning ratio Piarget-
Output: the pruning ratio for each block.
Calculate the transfer entropy of block and layer: for 0 to N do
the top
Sy top corresponding blocks
fayers with least
Store the TE values corresponding to layers and blocks in the set A and set S using Eq. (1).
end for
Step2:
unuvailuble
0
Divide S into three subsets St., Sas and Sn using Eq. (4). SH 0%, SM Plow. calculate Pletover using Eq. (5).
1-
S,
A
Depth- First search
Ft argmin Sz1 [SL], E argmin, Say {A}
Ft 1, 1 Plow- Depth- Search solve this optimization problem 6.
Remaii
Cumpensation
R
Z
M
Assign pruning ratios to the remaining blocks using Eq. (8).
Step3:
reallocated
M
Figure 5. Compensation process. Each O represents block, and its fill level visually corresponds to the pruning ratio. @ represents block that has not yet been assigned pruning ratio.
3.3. Row& Row&Colung &Colunnwise Pruning
Distributing the pruning load. After determining the appropriate pruning ratios for each block, we sign the pruning load to structured units within the block, i.c. finer-grained structure pruning by eliminating entire rows&columns in weight matrices. Building the ap- proach proposed in (Sun al., 2023), propose im- portance metric driven by information entropy that jointly weights W and the activation value X. Taking the j-th structured unit(column) W as an example, the impor- tance score IS, is calculated as follows:
R, we dynamically allocate these ratios based on transfer entropy. The pruning ratio for each block g is calculated as follows:
Wg
Pg
x Prarget
(8)
T 2g Wg
where
IS = H(( . (1,112),
(9)
Piarget = (G Piarget Peoopeesaaioo
+ Piefooer)
where H--) denotes the calculation of the information en- tropy, |- represents the absolute value operator, and ||X, is the Ig-norm of j-th channel across N x L different tokens. Upon obtaining the information entropy for each column in the weight matrix, we apply the K -meas algorithm (Mac- Queen et al., 1967) to cluster the units based on their entropy values and prune the cluster with the lowest average infor- mation entropy. Figure illustrates the process of assessing
Poompensation represents the result obtained by substituting the optimized solution { :M" into Equation (6). G 3
the number of blocks R, weight wy =
is scaling factor and average information entropy ( We summarize this assigning block-wise pruning ratio process, shown in Algorithm
5

---

Let LLM Tell What to Prune and How Much to Prune
Table 1. The perplexity results of pruned LLaMA2, LLaMA3 and Vieuna models.
METHOD
PRUNING UNIT
PRUNING RATIO
LLAMA2-78
LLAMA2-138
LLAMA2-70B
LLAMA3-8B
LLAMA3-708
VICUNA-7B
VICUMA-138
5.47
5.53
5.94
SPARSEGIP
2:4
50%
9.33
7.90
5.74
19.99
6.73
10.31
9.22
WANDA
2:4
11.00
8.23
5.26
10.89
7.77
12.41
8,96
FLAP
9.13
11.21
SLICEOPT
COLUMN
8,63
7.43
5.42
13.08
9.01
9.95
11.32
LLM-PRUNER
ROw& COLUMN
30%
17.86
19.38
22.88
21.17
14.27
BLOCKPRUNER
15.06
SLEB
BLOCK
27.45
23.48
5,93
29.17
9.08
OURS
ALE.
8,71
7.03
4.77
10.53
5.83
9.90
8.54
SLICEGPT
ROW&COLUMM
12.79
10.61
7.08
20.1
13,46
14.09
19.79
L.M--PRNNR
Row&CoLLUN
40%
46.33
45,10
$6.28
43.52
27.24
32.39
SLEB
BLOCK
45.36
27.49
7.57
78.66
OURS
ALL
12.55
8.85
5.64
14.72
6.93
13.91
10.54
FLAP
ROW&COLUMN
25.98
23.12
SLICEGPT
&ow&& COLUMN
21.08
17.51
10.75
35,64
28,69
23.01
45.01
LLM-PRUNER
ROWLCCOLMN
50%
286.07
89.31
185.86
189.08
70.37
BLOCKPRUNER
LAYER
58.86
29.13
12.81
479.23
61.47
80.16
36.07
SL.E
BLACK
OURS
ALL
20.07
13.44
6.76
21.97
8.92
22.05
15.20
the importance of columns in weight matrix using informa- tion entropy and subsequently removing the less important columns.
dimensions with higher activation variability through 0r and d2 thereby more effectively preserving critical features and maintaining the original activation statistics.
Once the rows&columms in the weight matrix that require pruning are identified, we can employ binary mask matrix M indicate the locations of pruning (0 for pruned, for retained). According to the binary mask matrix M and weighted average X, we can calculate the corresponding bias term by as follows:
Practical pruning process. In practical pruning process, due the attention head mechanism employed in LLMs, where each attention head corresponds to subset of rows columns, attention head are typically treated as the primary unit for pruning the attention layers. In the MLP layer, rows or columns are treated as the fundamental pruning units. The pruning procedure for different LLMs is provided in the Appendix A.
bo (1 X)
(12)
where denotes an all-ones matrix and O represents Hadamard product. Subsequently, we employ bias to com- pensate for the impact pruning on the model's output, as shown in the following equation:
3.4. Bias Compensation
Bias compensation a common compensatory mechanism used to recover the performance of models after pruning quantization (Gong et al., 2024; Finkelstein al., 2019; An et al., 2024). The compensation bias term for the j- th channel is computed as weighted average across all samples, formulated as follows:
WX ~
(M O W)X+ bo
(13)
4. Experiment
4.1. Experimental Setup
N Et (1
X
(10)
> EL_ (1)
Setup. In experiment, we apply our method to various large language models (LLMs), including LLaMA2 (Tou- vron et al., 2023), LLaMA3 (Meta, 2024), and Vicuna- 7B/13B (Chiang et al., 2023), all of which are available through the Hugging Face Transformers library (Wolf, 2019). All experiments are conducted on NVIDIA A800 GPUs with 80GB memory.
where the normalized weighting factors A (n) and Az modulate contributions along the sequence and channel di- mensions respectively:
(n) = (n)
(n))
(1 (1)
Datasets and Evaluation. For calibration data, we ran- domly select 128 samples from the WikiText2 training dataset (Merity et al., 2016). Since pruning can affect the model's ability, it is crucial to measure its impact using a reliable evaluation metric. Perplexity serves as the primary metric for evaluating model performance, with lower per- plexity indicating better performance. Therefore, we focus
(11)
(2(
Here, Jj denotes the standard deviation computed over the joint sequence-channel dimensions of X, while 02 repre- sents the standard deviation across the batch-channel di- mensions. This weighted approach adaptively prioritizes
6

---

Let LLM Tell What to Prune and How Much to Prune
Table 2. Zero-shot performance of the compressed LLaMA2-7B. Bold results highlight the best performance.
LLAMA2-7B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AVG
DENSE
0%
68.98
79.11
76.00
74.58
46.25
68.98
SPARSEGPT
2:4
61.03
68.77
52.80
56.69
30.63
53.99
50%
WANDA
2:4
60.93
69.80
54.16
58.08
30.89
54.77
FLAP
Row & COLUMN
61.25
70.67
56.79
54.84
32.08
55.12
SLICEGPT
&Ow& COLUMNN
62.67
64.80
49.19
50.38
31.48
51.70
LLM-PRUNER
ROW&COLUMN
30%
54.93
72.85
57.79
53.16
32.51
54.24
BLOCKPRUNER
LAYER
57.62
70.51
56.62
50.51
30.97
53.24
SLEB
BLOCK
52.33
69.80
$1.49
$1.85
30.03
51.10
OURS
ALL
60.77
70.95
59.18
58.16
34.22
56.65
FLAP
ROw&
56.51
67.57
49.00
37.71
27.30
47.62
SLICEGPT
ROW& COWLCOLUM
57.22
58.22
39.49
43.39
27.05
45.07
LLM-PRUNER
ROw&
40%
$2.80
66.32
43.44
39.23
29.95
46.34
BLOCKPRUNER
LAYER
53,99
63,00
45.70
42.51
29.01
46.84
SLEB
BLOCK
51.38
64.85
42.75
44.95
25.85
45.95
OURS
ALL
56,67
66.10
49,00
49.83
30.38
50.39
FLAP
Row&CoL
53.35
62.40
40.31
36.53
25.09
43.53
SLICEGPT
Row COLUMN
53.43
53.81
32.64
34.89
23.63
39.68
LLM-PRUNER
ROW&COLUMN
50%
50.59
56.96
31.23
31.23
26.62
39.32
BLOCKPRUNER
LAYER
53.35
58.16
36.32
37.50
26. 11
42.28
SLEB
BLOCK
50.75
57.34
34.41
33.92
26.28
40.54
OURS
ALL
$3.83
61.81
40,46
43.94
27.56
45.52
on perplexity and utilize the WikiText2 test dataset to evalu- ate the perplexity of the model after pruning with different pruning methods. Furthermore, to further validate the effi- cacy of our method, we report the Zero-Shot accuracy on five benchmark datasets: PIQA (Bisk et al., 2020); Wino- Grande (Sakaguchi al., 2021); HellaSwag (Zellers al., 2019); ARC-e and ARC-c (Clark et al., 2018).
outperforms SliceGPT, FLAP, and LLM-Pruner, even af- ter removing significant number of blocks and layers. Compared to pruning methods that focus primarily on re- moving blocks or layers, such as SLEB and BlockPruner, our method achieves significantly better results across all zero-shot tasks. Even compared to semi-structured pruning methods such as SparseGPT and Wanda, our approach de- livers superior performance the 30% pruning ratio. These results demonstrate that our method not only effectively reduces the model size but also maximally preserves the original performance of the LLMs. For further details, ad- ditional accuracy results for other LLMs across these tasks can be found Appendix
Baselines. We compare proposed approach with prun- ing methods designed for individual structure units: LLM- Prune (Ma et al., 2023), FLAP (An et al., 2024) (utilizing bias compensation) and SliceGPT (Ashkboos et al., 2024) prune the rows&colunns the weight matrices. Block- Prune (Zhong et al., 2024) and SLEB (Song al., 2024) separately focus on pruning the layers (attention&MLP) and the blocks in LLMs.
4.3. Inference Speed
In Table 3, we provide a detailed comparison of the infer- ence speed for generation the sequences of length 128 (batch size of under different pruning methods. All experiments conducted on NVIDIA A800 GPUs and observe that unstructured pruning methods like Wanda and SparseGPT do not significantly improve inference speed. Compared to row& row&colun pruning methods such as SliceGPT and FLAP, our approach achieves a significant improvement in inference speed while requiring less memory overhead. Fur- thermore, as the pruning ratio increases, the improvement in inference speed achieved by our method becomes more pronounced. This is because, at higher pruning ratios, approach tends to remove more blocks and layers within LLMs. For example, when pruning LLaMA2-7B model at a 50% ratio, we remove three blocks and three layers, respectively, further enhancing inference speed.
4.2. Pruning Results LLMs
Generation Tasks. We begin by presenting the WikiText2 performance results for the LLaMA2, LLaMA3, and Vicuna models at three distinct pruning ratios (30%, 40%, and 50%) in Table 1. As illustrated in Table 1, our approach, which prunes across all structure units, consistently achieves supe- rior perplexity performance compared to methods that prune based on rows and columns, such as SliceGPT, FLAP,, and LLM-Pruner, across most LLMs. Notably, pruning ratio of 30%, our method outperforms semi-structured pruning techniques, including SparseGPT and Wanda, across LLMs.
Zero-shot Tasks. Table 2 presents the zero-shot accura- cies of the pruned L.LMMA-7BB model across five different tasks, evaluated using various pruning methods. As ex- pected, at the same pruning ratio, our method consistently

---

Let LLM Tell What to Prune and How Much to Prune
Table Inference speed of LLaMA2-7B/13B compared with dif-
our method consistently outperform SliceGPT across the different number of samples. Moreover, method exhibits the smallest perplexity variation as the number of samples increasing, making it more robust compared to SliceGPT.
ferent pruning methods.
METHOD
SPARSITY
MEMORY
TOKENS/S
LLAMA2-7B
0%
12989MB
29.86
31.3
WANDA
50%
12989MB
28.60
32.
BliceOPT
SPARSEGPT
12989MB
30.34
SLICEGPT
10118MB
33.39
27.
FLAP
30%
9296MB
32.81
26.2
OURS
9297MB
34,88
2255
SLICEGPT
7172MB
34.38
23.3
FLAP
50%
6833MB
34.39
OURS
6800MB
38.78
1634
LLAMA2-13B
0%
251 4M1
25.64
Calibration Samples
WANDA
25114MB
26.85
SPARSEGPT
50%
2511 4MMB
25.35
Figure 7. Robustness to calibration samples in the LLaMA2-7B with 50% pruning ratio.
SLICEGPT
1963 IMB
29.11
FLAP
30%
17767MB
26.90
OURS
17701MB
29.22
4.6. Dependency on Calibration Dataset
SLICEGPT
12995MB
29.79
Different calibration datasets can affect the performance of the pruned model. Therefore, it is important evaluate how different calibration datasets affect the performance of different pruning methods. We use C4 and WikiText2 training datasets as different calibration datasets. Then, we utilize different structured pruning methods to evaluate the perplexity of the pruned LLaMAA-7B with 50% pruning ratio on both C4 and WikiText2 test datasets. In Figure regardless of the choice of calibration dataset, the perfor- mance variation of our method is much smaller than that of other methods. This indicates that our approach does not rely on the selection of the calibration dataset compared to other methods. Furthermore, regardless of the chosen test dataset, our method consistently outperforms other pruning methods
FLAP
50%
12922MB
27.91
OURS
12632MB
31.80
4.4. Baseline Bias Compensation
Our approach leverages bias compensation, which elimi- nates the need for retraining or fine-tuning, To evaluate its effectiveness, we conduct experiments on the LLaMA2-13B model with different pruning ratio. The results shown in Fig- highlight that bias compensation effectively restores the model's performance after pruning- As the pruning ratio increases, the impact of bias compensation becomes more pronounced, emphasizing its importance in maintain- ing performance. Notably, even without the use of bias compensation, our method outperforms other approaches without bias compensation in terms of PPL.
120
Compensation Without Compersation
(a) Perplexity results on Wiki- Text2.
(b) Perplexity results on C4,
30%
50%
Figure 8. Perplexity results LLaMA2-7B with 50% pruning ratio on the different calibration dataset.
Figure 6. Performance comparison of the LLM with and without bias compensation at various pruning ratios.
4.7. Ablation Studies
4.5. Robustness to Calibration Samples
In this section, we conduct comprehensive ablation studies to evaluate the effectiveness of our proposed method. Specif- ically, we compare various evaluation criteria for quantify- ing module interactions and different search methods, ana- lyze the performance differences between multi-structure units and single-structured unit pruning, and assess the im- pact of dynamic versus static pruning ratio allocation strate- gies on the performance of the pruned model. The detailed results of the ablation studies are provided in Appendix C.
The effectiveness of different pruning methods can be af- fected by the number of samples in the calibration dataset. Therefore, this makes it critical to verify whether our method remains robust and outperforms other pruning methods with calibration datasets of varying samples. In the section, we se- lect SliceGPT as the comparison method because, as shown in Table 1, it exhibits the second-best performance when pruning ILaMAA-7B at a 50% ratio. Figure? shown,
8

---

Let LLM Tell What to Prune and How Much to Prune
5. Conclusion
with 90%* * chatgpt quality. See https://vicua. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
In this paper, we propose a a structured pruning framework to simultaneously target different structure units of LLMs with dynamic ratios. Our method can effectively balance the trade-off between model performance and inference speed. We validate our proposed method across multiple LLMs and performe a comprchensive comparison with various struc- tured pruning methods. The experiment results demonstrate that our method can effectively maintain model performance after pruned and enhance inference efficiency.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C.,, and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: /803 0ot:::0083.0057,, 2018.
Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.00 000 18, 2024.
Acknowledgments
Finkelstein, A., Almog, U., and Grobman, M. Fighting quan- tization bias with bias. arXiv preprint 1/Xiv:196.003// 2019.
This work is partially supported by CAS Pioneer Hundred Talents Program. The authors thank Pumeng Lyu for his helpful discussion.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference on Machine Learning, pp. 10323- 10337. PMLR, 2023.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work. none which we feel must be specifically highlighted here.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. urXiv preprint urXiv:2210.1732.. 2022.
References
Gong, C., Zheng, H., Hu, M., Lin, Z., Fan, D.-P., Zhang, Y., and Li, Minimize quantization output error with bias compensation. arXiv preprint arXiv:2404. 01822, 2024.
Agarwal, R., Vieillard, N., Stanczyk, P., Ramos, S., Geist, Bachem, O. Gkd: Generalized knowledge distilla-
M., and and tion for auto-regressive sequence models. arXiv preprint arXiv:2306.13644 2023.
Gordon, M. A., Duh, K., and Andrews, N. Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307 2020.
An, Y., Zhao, X., Yu, T., Tang, M., and Wang, J. Fluctuation- based adaptive structured pruning for large language mod- els. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865-10873, 2024.
Gu, Y., Dong, L., Wei, F., and Huang, M. Knowledge distillation of large language models. arXiv preprint SKKY:2306.085444 2023.
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Ashkboos, S., Croci, M. L., do Nascimento, M. G., Hoefler, T., and Hensman, J. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024.
Hassibi, B., Stork, D. G., and Wolff, G. J. Optimal brain surgeon and general network pruning. In IEEE interna- tional conference on neural networks, pp. 293-299. IEEE, 1993.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1432-7439, 2020.
He, Y., Zhang, X., and Sun, J. Channel pruning for acceler- ating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389,1397,90 2017.
Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. M. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Jaiswal, A. K., Liu, S., Chen, T., Ding, Y., and Wang. Z. Instant soup: Cheap pruming ensembles in single pass can draw lottery tickets from large models. In Inter- national Conference on Machine Learning, pp. 14691 14701. PMLR, 2023.
Chiang, W.-L., Li, Z.,, Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., et al. Vieuna: An open-source chatbot impressing gpt-4

---

Let LLM Tell What to Prune and How Much to Prune
Ko, J., Kim, S., Chen, T., and Yun, S.-Y. Distillm: Towards streamlined distillation for large language models. arXiv preprint FrXiv:2402.03892 2024.
Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Ac- celerating sparse deep neural networks. arXiv preprint wrXi::210.08837, 2021.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Muralidharan, S., Sreenivas, S. T., Joshi, R. B., Chochowski, M., Patwary, M., Shocybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. In The Thirly-eighth Annual Conference Neural Information Processing Systems, 2024.
Li, C., Wang. G., Wang, B., Liang, X., Li, Z., and Chang, X. Dynamic slimmable network. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8607-8617, June 2021.
Padmanabhan, S., Onoe, Y., Zhang, M., Durrett, G., and Choi, E. Propagating knowledge updates to Ims through distillation. Advances Neural Information Processing Systems, 36, 2024.
Li, C., Wang, G., Wang, B., Liang, X., Li, Z., and Chang, X. Ds-net++: Dynamic weight slicing for efficient inference in cnns and vision transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4430- 4446, 2022.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
Lin, C.-H., Gao. S., Smith, S., Patel, A., Tuli, S., Shen, Y., Jin, H., and Hsu, Y.CC. Modegpt: Modular decomposition for large language model compression. arXiv preprint Ttt.:2408.096322 2024.
Schreiber, T. Measuring information transfer. Physical review letters, 85(2):461, 2000.
Ling, G., Wang, Z., and Liu, Q. Slimgpt: Layer-wise structured pruning for large language models. Advances in Neural Information Processing Systems, 37:107112-
Sirignano, and Spiliopoulos, K. Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics, 80(2):725 -5222,5--2222 2020.
Song, J., Oh, K., Kim, T., Kim, H., Kim, Y., and Kim, J.-J. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09023 2024.
Ma, X., Fang, G., and Wang. X. Lim-pruner: On the struc- tural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 16695, 1695, 2023.
MacQueen, J. et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pp. 281-297. Oakland, CA, USA, 1967.
Sun, Z.,, Ge, C., Wang, J., Lin, M., Chen, H., Li, H., and Sun, X. Entropy-driven mixed-precision quantization for deep network design. Advances Neural Information Processing Systems, 55:21508-21522 2022.
Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., et al. Language models are few-shot learners. arXiv preprint wxXv:2005..1416. 1, 2020.
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kul- shreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. Lamda: Language models for dialog appli- cations. arXiv preprint arXiv:2201. 088399 2022.
Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language mod- els are more redundant than you expect. arXiv preprint arXiv:2403.03855 2024.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv: 2016.
Wolf, T. Huggingface's transformers: State-of-the- art natural language processing. arXiv preprint arXiv:1910.0:23 2019.
Meta, A. Introducing meta llama 3: The most capable openly available Ilm to date. Meta AI, 2024.
10

---

Let LLM Tell What to Prune and How Much to Prune
Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning, arXiv preprint arXiv:2310.000
2023.
Yin, L., Jaiswal, A. K., Liu, S., Kundu, S., and Wang. Z. Junk DNA hypothesis: Pruning small pre-trained weights Irreversibly and Monotonically impairs "difficult" down- stream tasks in LLMs. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp. F. (eds.), Proceedings of the 41st Interna- tional Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 57053- 57068. PMLR, 21-27 Jul 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint I9Xiv 07830, 2019.
hhang, C., Bengio, S., and Singer, Y. Are all layers created equal? Journal of Machine Learning Research, 23(67);
1-28, 2022.
Zhang, C., Cheng, J., Constantinides, G. A., and Zhao, Y. Lqer: Low-rank quantization error reconstruction for Ilms. arXiv preprint 402102240..024446, 2024a.
Zhang, Y., Bai, H., Lin, H., Zhao, J.. Hou, L., and Can- nistraci, C. V. Blug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, 2024b.
Zhao, W. X., Zhou, K., J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., al. A survey of large language models. arXiv preprint arXiv:2303.18222 2023.
Zhong, L.,, Wan, F., Chen, R., Quan, X., and Li, L. Block- pruner: Fine-grained pruning for large language models. arXiv preprint arXiv:2406. 10594, 2024.
Zhu, X., Li, J., Liu, Y., Ma, C., and Wang, W. survey on model compression for large language models. Transac- tions of the Association for Computational Linguistics, 12:1556-1577 2024.

---

Let LLM Tell What to Prune and How Much to Prune
A. Implementation Details
A.1. Detailed Pruning Process for LLMs Using the Multi-Head Attention Mechanism
The LLaMA2-7B, LLaMA2-13B, and Vicuna models employ a multi-head attention mechanism (MHA), where cach attention head is comprised of set of rows or columns from the weight matrix, functioning as an independent computational unit. Directly pruning the rows columns of the weight matrix in the attention layer can compromise the integrity of the attention heads. Thus, to address this, treat entire attention heads the fundamental pruning units preserve the functionality and coherence of the attention mechanism.
We first prune the attention heads in the attention layer. As illustrated in Figure 9a, after multiplying the absolute value of the weight matrix by the L2 norm of the activation value, we perform an averaging operation on the columns of each attention head within the output projection weight matrix W.. and then compute the information entropy for these averaged values. Based on the calculated information entropy, we cluster and sort the attention heads. The cluster with the smallest average information entropy are pruned. When the columns of W. are pruned, the corresponding rows Wa (query projection weight matrix), W (key projection weight matrix), and W. (value projection weight matrix) should also be pruned accordingly, as illustrated in Figure 9b. To achieve the target pruning ratio for each block, the remaining portion is pruned in the MLP layer.
For the weight matrices in the MLP layer, we directly calculate the information entropy of the columns in W down (down- projection weight matrix). After clustering and sorting the columns based on their information entropy, we prune the smallest average entropy within each cluster. Consequently, the corresponding rows in Wup (up-projection matrix) and W gate (gate weight matrix) are also pruned maintain structure consistency.
heud,
head
head
head head
head, head, head,
Average
0000
Calculate entropy
Sort and Cluster
(a) Attention heads pruning for LLMs using MHA..
W
head.
head
column
now
W
Ipec
now
Attention Layer
MLL Layer
(b) Pruning process for LLMs using MHA.
Figure The detailed pruning process for LLMs using MHA. (a) Attention head pruning. (b) The pruning process attention layer and ML. layer.
12

---

Let LLM Tell What to Prune and How Much to Prune
A.2. Detailed Pruning Process for LLMs Using the Group-uuery Attention Mechanism
Group-Query Attention mechanism (GQA) is a variant of the multi-head self-attention mechanism (MHA), designed enhance computational efficiency by reducing computation and storage requirements. In GQA, the query heads divided into G groups, with each group sharing single key and value head. LLaMA2-70B, LLaMA3-8B and LLaMA3-70B models use the GQA, therefore, different from Appendix A.1, we treat the group as the pruning unit as Figure 10a shown. Consistent with Figure 9a, we calculate the information entropy of each group within the W.. perform clustering, and prune the cluster with the lowest information entropy. When the groups of W are pruned, the corresponding groups in W, g. and the corresponding heads in W and W. should also be pruned accordingly. as shown in Figure 10b. order to reach the desired pruning ratio for each block, the remaining portion pruned in the MLP layer accordingly. For the weight matrix in the MLP layer, the pruning process is consistent with that described in Appendix A.I.
Group,
Group
Group
Group
Group, Group.
Group, Group Group.
Average
Calculate entropy
Sort and Cluster
Attention heads pruning for LLMss using GQA.
W
Group,
W
Group
colann
row
W.
W
W
head,
pice
nuN
W.
head,
Attention Layer
MLP Layer
(b) Pruning process for LLMs using GQA.
Figure 10. The detailed pruning process for LLMs using GQA. Upper:Attention head pruning. Down: The pruning process for the attention layer and MLP layer.
13

---

Let LLM Tell What to Prune and How Much to Prune
B. Additional Experiment Results
this section, we provide the more detailed zero-shot tasks results of the different LLMs
Table 4. Zero-shot performance the compressed LLaMA2- 13B. Bold results highlight the best performance.
LLAMA2-13B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AVG
DENSE
72.30
80.52
79.37
77.53
49.06
71,75
SPARSEGPT
2:4
65.75
71.22
58.97
61.03
34.90
58.37
50%
WANDA
2:4
65.35
74.86
62.07
65.19
36.86
60.86
SLICEGPT
Row& COLUMNN
65.59
64.96
52.35
53.54
36.86
54.65
LLM-PRUNER
ROw&CoLUME
61.88
73.72
64.37
63.72
38.05
60.34
BLOCKPRUNE
LAYER
30%
63.22
75.08
65.28
66.37
40.02
61.99
SLEB
BLOCK
56.43
74.65
62.11
63.09
37.37
58.72
OURS
ALL
66.69
73.61
64.83
67.68
41.47
62.85
SLICEGPT
ROW&COLUMN
60.69
59.30
42.50
56.44
34.13
47.30
LIM-PRUNER
Row &COLUMN
54.70
68.23
50.50
49.45
31,40
50.85
BLOCKPRUNE
LAYER
40%
58.80
71.00
57.15
56.14
35.58
55.73
SLEB
BLOCK
55,49
70.24
54.08
54.25
32.34
53.27
OURS
ALL
62.90
69.48
54.25
59.05
34.56
56.04
SLICEGPT
ROW&COLUMN
55.17
55.22
34.33
37.08
24.49
41,25
LLM-PRUNER
Row &COLUMN
$4.06
62.68
40.85
39.02
27.22
44.76
BLOCKPRUNE
LAYER
50%
$4.62
61.48
45.08
41.71
28.16
46,02
SLEB
BLOCK
55.01
62.95
43.29
42.76
26.
46.02
OURS
ALI
55.72
65.72
47.30
45.54
29.10
48.68
Table 5. Zero-shot performance compressed LLaMM2-700. Bold results highlight the best performance.
LLAMA2-70B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AVG
DENSE
0%
77.98
82.70
83.82
80.98
57.42
76.57
SPARSEGPT
2:4
74.82
78.62
76.11
77.86
52.47
71.97
50%
WANDA
2:4
74.66
79.49
75.86
76.73
51.45
71.64
SLICEGPT
ROW&COLUMN
73.09
71.55
62.72
73.95
49.23
66.10
BLOCKPRUNE
LAYER
70.40
78.02
72.50
69.78
43.09
66.75
SLEB
BLOCK
30%
69.53
77.80
73.57
72.01
44.03
67.38
OURS
ALL
73.64
79.43
80.09
76.47
51.11
72.15
SLICEGPT
Row &COUMM
70.48
65.07
50.6
61.15
38.75
57.20
BLOCKPRUNE
LAYER
65.27
74.48
65.99
61.91
38.05
61.15
SLEB
BLOCK
40%
66.61
75.03
66.76
66.62
39.68
62.94
OURS
ALL
72.30
76.71
77.26
71.09
47.95
69.06
SLICEGPT
ROW&COLUMN
63.61
58.11
39.45
45.75
30.03
47.39
BLOCKPRUNE
LAYER
60.38
69.15
58.49
51.56
34.04
54.72
SLEB
BLOCK
50%
61.25
70.29
57.67
58.00
35.58
56.55
OURS
ALL
71.27
74.43
72.00
62.79
41.72
64.44
14

---

Let LLM Tell What to Prune and How Much to Prune
Table 6. Zero-shot performance of the compressed Vicuna-7B: Bold results highlight the best performance.
VICUNA-7B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AvG
DENSE
0%
68.38
78.02
73.77
71.30
45.82
67.65
SPARSEGPT
2:4
61.17
68.72
55.70
$9.60
33.70
$5.77
50%
WANDA
2:4
62.75
71.60
57.14
61.07
35.07
57.52
SLICEGPT
COLUMN
59.83
63.87
48.99
49.24
31.74
50.73
LLM-PRUNER
Row ROw&COLUMN
56.51
73.50
58.37
56.65
35.07
56.02
BLOCKKRUNTE
LaYER
30%
57.30
71.49
56.98
58.42
33.36
55.51
OURS
ALL
64.48
70.18
60.43
59.89
37.29
58.45
SLICEGPT
ROw&COLUMN
57.22
58.71
40.23
43.31
27.56
45.41
LLM-PRUNER
Row&CoLUNE
51.62
67.85
47.60
46.97
32.59
49.32
BLOCKPRUNER
LAYER
40%
53,43
63.55
45.97
49.37
28.84
48,23
OURS
ALL
58.64
65.78
50.63
53.32
30.97
51.87
SLICEGPT
ROw&COLUMN
50.28
53.26
32.67
34.76
23.98
38.99
L.MM-RRENER
ROw&COLUMN
54.06
59.14
34.74
34.81
28.07
42.65
BLOCKPRUNER
LAYER
50%
47.43
58.76
36.76
42.51
28.24
42.74
OURS
ALL
55.33
60.34
39.97
43.73
28.07
45.48
Table 7. Zero-shot performance of the compressed Vicuna-138 Bold results highlight the best performance.
VICUNA-13B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AVG
DENSE
0%
71.59
79.05
77.51
74.87
50.68
70.74
SPARSEGPT
2:4
64.48
72.14
60.08
63.72
37.03
59.49
50%
WANDA
2:4
67.17
74.10
64.07
66.20
42.66
62.84
SLICEGPT
Row & COLUMN
65.51
64.20
50.51
56.94
33.62
54.15
LLM-PRUNER
ROw&COLUMN
59.19
77.04
65.48
63.89
61.24
BLOCKPRUNER
LAYER
30%
62.43
73.67
64.90
64.94
40.78
61.34
OURS
ALL
65.35
73.39
63.04
69.99
40.44
62.44
SLICEGPT
&OW&COLUMN
58.72
58.27
38.22
44.49
27.05
45.35
I.M--RUNER
Row&CoLUMN
54.38
72.69
57.00
51.88
34.73
53.99
BLOCKPRUNER
LAYER
40%
58.88
67.68
$3.99
54.21
34.90
53.93
OURS
ALL
60.62
69.10
53.39
63.22
36.52
56.56
SLICEGPT
ROW&COLUMN
53.28
54.52
31.73
35.19
24.40
39.82
LLM-PRUNER
ROW&COLUMN
52.09
65.07
44.59
39.06
28.75
45.91
BLOCKPRUNER
LAYER
50%
52.41
62.02
40.81
42.59
28.24
45.21
OURS
ALL
56.12
64.04
45.10
50.08
32.25
49.52
15

---

Let LLM Tell What to Prune and How Much to Prune
Table 8. Zero-shot performance of the compressed L.aMA3-8B. Bold results highlight the hest performance.
LLAMA3-8B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AvG
DENSE
0%
72.69
80.79
79.19
77.69
53.41
72.75
SPARSEGPT
2:4
64.25
68.77
55.40
59.34
32.85
$6.12
50%
WANDA
2:4
59.67
65.94
47.93
50.04
28.67
50.45
SLICEGPT
Row& COLUMN
58.72
57.78
40.68
45.20
28.75
46.22
LLM-PRUNER
Row ROw&COLUMN
53.67
69.15
45.06
43.31
27.65
47.76
BLOCKPRUNTR
LAYER
30%
54.54
67.03
46.54
49.41
27.99
49.10
SLEB
BLOCK
61.01
67.74
55.53
47.10
32.25
52.72
OURS
ALL
60.30
69.10
54.40
54.76
33.19
54.34
SLICEGPT
Row&CoLUME
52.49
53.59
33.12
35.94
22.61
39.55
LLM-PRUNER
ROw&COLUMN
51.54
62.73
36.25
36.87
23.46
42.17
BLOCKPRUNER
LAYER
40%
50.67
58.76
36.18
36.41
24.06
41.21
SLEB
BLOCK
51.54
60.88
40.33
40.78
26.88
44.02
OURS
ALL
56.27
64.20
44.74
46.51
27.30
47.80
SLICEGPT
Row& ROW&COLUM
50.04
52.23
29.18
31.31
21.67
36.88
LLM-PRUNER
ROWUCOLUMN
51.22
60.34
31.74
33.00
24.74
40.20
BLOCKPRUNER
LAYER
50%
48.62
55.22
29.12
30.01
21.59
36.91
SLEB
BLOCK
50.67
55.98
31.54
31.19
24.74
38.82
OURS
ALL
54.14
59.96
38.36
37.84
24.15
42.89
Table 9. Zero-shot performance the compressed 1.MMM3-700.. Bold results highlight the best performance.
LLAMA3-70B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AvG
DENSE
0%
80.35
84.55
84.88
85.86
64.33
79.94
SPARSEGPT
2:4
75.37
78.35
72.30
78.37
51.37
71.15
50%
WANDA
2:4
71.67
79.27
73.33
76.01
49.06
69.86
SLICEGPT
ROW&COLUMN
69.14
65.72
55.01
64.69
42.58
59.42
SLEB
BLOCK
74.59
77.48
74.03
72.90
48.12
69.42
BLOCKPRUNER
LAYER
30%
71.90
76.12
73.81
69.40
47.44
67.73
OURS
ALL
75.53
80.69
80.38
77.78
52.05
73.28
SLICEGPT
ROw&COLUMN
61.09
58.49
43.93
47.85
32.85
48.81
SLEB
BLOCK
68.75
71.93
63.49
60.65
37.63
60.48
BLOCKPRUNER
LAYER
40%
66.38
71.06
64.76
59.60
37.80
59.91
OURS
ALL
72.93
77.26
73.27
73.23
47.44
68.82
SLICEGPT
ROW&COLUMN
54.70
54.95
33.11
36.74
25.77
41.05
SLEB
BLOCK
56.20
66.92
49.99
48.32
29.27
50.13
BLLOCKPRUNER
LAYER
50%
55.49
64.85
51.09
45.45
30.29
49.43
OURS
ALL
68.82
72.63
61.72
61.78
39.33
60.85
16

---

Let LLM Tell What to Prune and How Much to Prune
C. Ablation Studies
C.1. Ablation Study on Criteria for Quantifying Module Interactions
We use the Frobenius norm (He et al., 2017) instead of transfer entropy as the new criterion to measure the change LLM hidden state. We conduct evaluations on five different zero-shot datasets. As shown in Table 10, TE consistently preserves better model performance compared the Frobenius norm.
Table Different criteria for quantifying block interactions. Bold results highlight best performance.
LLAMA2-7B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AVG
FROBENIUS NORM
ALL
54.54
61.75
39.29 40.46
42.05
26.28
44.78
50%
TE(OURS)
ALL
53.83
61.81
43.94
27.56
45.52
C.2. Ablation Study on Search Method
We compare Depth-First Search (DFS) with an intuitive solution greedy search. We observe greedy search tends to prioritize removing entire blocks during the compensation phase to quickly satisfy the remaining pruning ratio in Equation (5). However, due to the lack of a backtracking mechanism, greedy search selects the seemingly optimal option at each step prone getting stuck in local optima. Our ablation study in Table 11 further supports this observation, demonstrating that DFS consistently outperforms greedy search in terms of pruning effectiveness.
Table Different search methods. Bold results highlight best performance.
LLAMA2-7B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AVG
GREEDY SEARCH
ALL
53.67
61.53
39.96 40.46
41.75
27.22
44.82
50%
DFS
ALL
53.83
61.81
43.94
27.56
45.52
C.3. Ablation Study on Hierarchical Pruning
In this ablation study, we adopt the Hybrid pruning strategy proposed in Sec. 3.2 to jointly prune both blocks and layers, thereby implementing hicrarchical pruning. However, for the remaining blocks, we implement the approach described in Sec. 3.3, pruning only the rows&columns of the weight matrices without applying dynamic ratio allocation. Instead, a uniform pruning ratio is used across all of them. To compare with pruning single structure unit, without using hierarchical pruning, apply uniform pruning ratio across all blocks and prune only the entire rows&columns of the weight matrices. The corresponding results are shown in Table 12.
Table 12. Zero-shot performance of the Hierarchical Pruning. Bold results highlight the best performance.
LLAMA2-7B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AvG
SINGLE UNIT
ROw&COLUMN ALL
51.07
$5.60
29.60
33.67
24.15
38.81
50%
MULTIPLE UNITS
55.09
61.32
40.50
43.22
27.13
45.45
C.4. Ablation Study on Dynamic Ratio
To validate the effectiveness of dynamic pruning ratios, we build upon the Hierarchical Pruming setup from Appendix C.3 by incorporating dynamic ratio assignment. The corresponding ablation results are presented in Table 13.
Table 13. Zero-shot performance of dynamic ratios. Bold results highlight the best performance.
LLAMA2-7B
PRUNING UNIT
SPARSITY
WINOGRANDE
PIQA
HELLASWAG
ARC-E
ARC-C
AVG
STATIC RATIO
ALL
55.09
61.32
40.50
43.22
27.13
45.45
50%
DYNAMIC RATIO
ALL
53.83
61.81
40.46
43.94
27.56
45.52
17


Reviews:
[{"review_id": "n2tR8vT5zy", "reviewer": "Reviewer_FHHv", "summary": "The paper introduces a new approach to pruning large language models (LLMs) that dynamically assigns pruning ratios to different components based on their importance. \n\nThere are two issues with the current pruning methods: (1) focusing on just one structure of the model;  (2) using a prescribed pruning ratio. To address these two issues, they developed a more flexible approach that targets multiple parts of the model simultaneously and varies how much to prune based on what's actually important. They use something called \"transfer entropy\" to figure out which transformer blocks matter most, and then \"information entropy\" to determine which parts within those blocks to keep or remove. Their method works in two main steps: first, decide how much to prune each block based on its importance to the model's overall function; then, distribute that pruning load across different components within each block. \n\nThey tested their method on various model sizes including LLaMA2-7B/13B/70B, LLaMA3-8B/70B, and Vicuna-7B/13B at different pruning levels (30%, 40%, 50%). Their pruned models had better perplexity scores and zero-shot accuracy than other methods. The models also ran faster and used less memory. Their approach worked well regardless of the training samples used or how many samples they had.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer FHHv: \n\nWe appreciate the reviewer’s constructive suggestions. Experimental results are available at the  link: https://anonymous.4open.science/r/5BDF/README.md.  We address the questions in “Questions for Authors” and “Other Strengths and Weaknesses” sections. For any additional concerns mentioned in other parts, we  provide detailed responses below:\n\n---\n\n### **Q1: _Clarification to Figure 8_**\nIn Fig. 8(a), we use the training sets of C4 and WikiText2 as calibration datasets and evaluate the models on WikiText2 test set. For Fig. 8(b), we use the same calibration datasets but evaluate the models on C4 test set. The result shows our approach is less sensitive to the calibration dataset.\n\n---\n\n### **Q2: _Motivation of K-means_**\nUsing average entropy as a threshold tends to mix the units with low entropy and those close to the mean, failing to distinguish between components of different importance levels.\n\n### **Q3: _K-means vs. Average threshold_**\nK-means allows us to partition structure units into distinct groups. Pruning is then applied to the group with the lowest entropy in each layer, enabling a more fine-grained pruning strategy. We validate the superiority of K-means over the averaging approach in Table 1 in the link.\n\n---\n\n### **Q4: _Number of K-means clusters_**\nAs shown in Table 2 in the link, for 30% and 50% pruning ratios, K is fixed at 3 and 6, respectively. For 40% pruning ratio, we conducted a hyperparameter search.\n\n---\n\n### **Q5: _Theoretical explanation of entropy_**\nWe learned from [1] that prior works [2,3] have found that in the over-parameterized LLM, network weights tend to remain close to their initialization throughout training. As a result, the magnitude of gradient updates is relatively small, and activation values remain nearly same in the ''lazy regime\", making it difficult to reflect the true contribution of each unit. Please refer to more details for the ablation results in Table 3 in the link. \n\n[1] Junk DNA hypothesis: Pruning small pre-trained weights Irreversibly and Monotonically impairs “difficult” downstream tasks in LLMs. ICML2024\n\n[2] Sparsity May Cry: Let us Fail Sparse Neural Networks Together. ICLR2023\n\n[3] A Kernel-Based View of Language Model Fine-Tuning. ICML2023\n\n---\n\n### **W1: _Theoretical foundations of entropy-based metrics_**\nTE [4] measures how much *unique* information a source process (blocks) provides about a target process (output layer). If removing a block dramatically reduces the unique information available to the downstream layers, the network’s overall performance should drop. Thus, blocks with high TE are deemed more critical for maintaining the model’s predictive capacity. A block that has low TE essentially replicates or redundantly encodes information in the network. Pruning such a block has minimal impact because it does not lose much unique signal. Information entropy as a key metric is introduced in **Q5**.\n\n[4] Measuring information transfer. Physical review letters, 2000.\n\n---\n\n### **W2: _Assumption of Gaussian distribution_**\nPlease refer to Reviewer nYT6's Q1.\n\n---\n\n### **W3: _Number of K-means clusters_**\nPlease refer to Q4.\n\n---\n\n### **W4: _Ablation studies_**\n\n_Entropy metric:_\nWe compare to activation and gradient magnitude in Table 3 in the link. \n\n_Hierarchical design:_\nHierarchical pruning targets multiple structures. To validate it, we implement a baseline that only prunes row&column of weight matrix in Table 4 in the link.\n\n_Dynamic ratios:_\nWe compare to fixed ratio in Table 5 in the link.\n\n---\n\n### **O1: _SparseGPT and Wanda_**\nSparseGPT and Wanda are two semi-structured methods that rely on special hardware, while our method is hardware-friendly. As a result, our model with 30% sparsity has faster inference while offering better results, as shown in Table 3.\n\n\n---\n\n### **O2: _Real-world tasks_**\nThe MathQA, OpenBookQA, and SciQ are included in the link.\n\n\n### **O3: _Noticeable fluctuations across different settings_**\nIn Table 1, our method outperforms other structured pruning methods in 18 out of 21 cases. Beyond the Perplexity, we evaluate the model in a zero-shot setting in Table 2, showing consistent improvement over prior works.\n\n### **O4:_Particularly well on LLaMA2-70B_**\nWe guess you were mentioning Table 5 of the Appendix. Our core motivation is that **pruning single unit with fixed ratio cannot handle complex LLM**. The result in Table 5 aligns with our insight: When it comes to larger model size and higher pruning ratio, our hierarchical strategy achieves a better balance between efficiency and performance.\n\n### **O5:_Missed references_**:\nThe two references you suggested are not proposed for LLM pruning. We compare to [5] targeting multiple structures in Table 6 in the link. \n\n[5] MultiPruner: Balanced Structure Removal in Foundation Models. arxiv 2025\n\nDecimal notation: We will fix it\n\nWe encourage the reviewer to ask questions on anything that may still be unclear.\n\nThank you so much,\n\nAuthors", "claims_and_evidence": "Supported Claims: \n\nThe authors claim their method outperforms existing approaches. This is well-supported by comprehensive experimental results in Tables 1-9, showing better perplexity scores and zero-shot accuracy across multiple models (LLaMA2, LLaMA3, Vicuna) and pruning ratios (30%, 40%, 50%). \n\nFigure 6 adequately demonstrates how their bias compensation method helps maintain model performance after pruning. \n\nPotentially Problematic Claims: \n\nThe paper claims that \"at a pruning ratio of 30%, our method outperforms semi-structured pruning techniques, including SparseGPT and Wanda, across LLMs.\" However, this comparison is misleading and unfair. Looking at Table 1, we can see that SparseGPT and Wanda were only evaluated at 50% sparsity, not at 30%. The authors are comparing their method at a lower pruning ratio (30%) against other methods at a much higher pruning ratio (50%). This creates an unfair advantage for their approach, since models with less pruning naturally tend to perform better. A fair comparison would require all methods to be evaluated at the same pruning ratio. \n\nThe paper assumes a Gaussian distribution for hidden states in Equation 3 without providing evidence that this assumption holds in practice for transformer blocks. This is problematic because the entire entropy calculation, which forms the foundation of their pruning strategy, depends on this distributional assumption. If the hidden states don't actually follow a Gaussian distribution, the entropy calculations could be inaccurate, potentially undermining the theoretical basis of their method.", "methods_and_evaluation_criteria": "Methods \n\nThe paper proposes a hierarchical pruning strategy that targets multiple structure units in LLMs with dynamic pruning ratios. While the approach is innovative, the theoretical foundation for using entropy as the key metric for importance could be strengthened. The authors don't sufficiently justify why entropy specifically is better than other potential metrics (such as activation magnitude or gradient-based importance) for identifying less important components. The transfer entropy concept for quantifying block importance is interesting, but lacks thorough theoretical connection to the actual functionality of transformer blocks in language modeling. \n\nEvaluation Criteria \n\nThe paper's use of perplexity as a primary evaluation metric makes sense for measuring language model performance after pruning. The authors also appropriately evaluate their method on common benchmarks including WINOGRANDE, PIQA, HELLASWAG, ARC-E, and ARC-C, which are standard datasets for measuring commonsense reasoning capabilities. However, the evaluation could be strengthened by including more diverse real-world benchmarks that specifically measure different capabilities such as mathematical reasoning, text completion, and other practical tasks.", "theoretical_claims": "The paper lacks substantial theoretical analysis to support its claims. While the authors introduce transfer entropy as a key metric for determining block importance in Section 3.1, they provide no theoretical proof establishing a relationship between transfer entropy and model performance after pruning. This is a significant gap, as the entire hierarchical pruning strategy depends on this correlation.  \n\nThe formulas presented (particularly Equations 1-3) appear mathematically correct in isolation, but the authors make assumptions without rigorous justification - for example, assuming Gaussian distribution for hidden states in Equation 3 without verifying this distributional assumption holds in practice for transformer blocks.", "experimental_designs_or_analyses": "Overall, the experimental design in this paper is sound, with appropriate evaluations across multiple models, pruning ratios, and datasets. However, there are notable inconsistencies in the results that warrant further explanation. In Table 1, the performance patterns across different LLMs and pruning methods appear somewhat inconsistent - in some cases, the proposed method outperforms all others, while in other cases, different methods perform better for specific model sizes or types. The authors don't adequately analyze or explain these variations, which raises questions about the generalizability of their approach. For instance, why does their method work particularly well on LLaMA2-70B but show less improvement over alternatives on some other model variants? A deeper analysis of why certain methods perform better on specific architectures would strengthen the paper's contribution and help readers understand when to apply which pruning strategy.", "relation_to_broader_scientific_literature": "This paper advances LLM pruning by introducing an entropy-based approach that quantifies information content across model components, allowing for dynamic, principled pruning decisions. While previous work like Wanda (2023) considered activation values and OWL (2023) implemented non-uniform layer pruning, this research provides a more sophisticated framework by using transfer entropy to determine component importance. Unlike single-structure methods such as SLEB (2024) or SliceGPT (2024), the authors' approach simultaneously targets multiple structural units (blocks, layers, weight matrices) with dynamically determined pruning ratios, creating a more holistic pruning strategy that better balances performance and efficiency.", "essential_references_not_discussed": "Recent work on efficiently pruning attention heads and MLP layers in transformers like \"Structured Pruning of Large Language Models\" (Wang et al., 2023) and \"Are Sixteen Heads Really Better than One?\" (Michel et al., 2019) should be included, as they directly relate to the paper's multi-structure pruning approach.", "other_strengths_and_weaknesses": "Strengths: The paper introduces a novel hierarchical pruning approach that targets multiple structural components simultaneously, which differentiates it from previous methods that focus on pruning a single structure. The dynamic pruning ratio allocation based on component importance is a significant innovation that allows for more flexible and efficient pruning. The bias compensation technique effectively helps maintain model performance after aggressive pruning. The experimental evaluation is comprehensive, covering multiple model families (LLaMA2, LLaMA3, Vicuna) and sizes, which demonstrates the broad applicability of their approach. \n\nWeaknesses: The paper lacks strong theoretical foundations for its entropy-based importance metrics. While the authors introduce transfer entropy and information entropy as key metrics, they don't adequately justify why these specific information-theoretic measures are optimal for pruning decisions compared to alternatives. The assumption of Gaussian distribution for hidden states needs validation. The description of the K-means clustering implementation lacks details on parameter selection (e.g., number of clusters). The paper would benefit from ablation studies isolating the impact of different components of their approach (hierarchical pruning, dynamic ratios, entropy-based importance) to better understand which elements contribute most to the performance improvements.", "other_comments_or_suggestions": "There is inconsistency in the decimal notation throughout the paper. For example, in Table 2 and other zero-shot performance tables, some accuracy percentages have two decimal places (e.g., \"68.98\"), while others have different formats (e.g., 76, 52.8, 62.40).", "questions_for_authors": "In Section 4.6 and Figure 8, there appears to be a discrepancy between the figure captions and the content. Figure 8(a) is labeled as \"Perplexity results on WikiText2\" and Figure 8(b) as \"Perplexity results on C4,\" yet both figures show bars for both WikiText2 and C4. Could you clarify what each subfigure is actually showing? \n\nIn Section 3.3, you mention using K-means clustering to group structure units based on their entropy values before pruning. What is the motivation for using a learning-based clustering algorithm like K-means instead of simpler approaches such as averaging or thresholding based on entropy values? Does K-means provide specific advantages for identifying pruning candidates compared to non-learning approaches? Additionally, how do you determine the optimal number of clusters for the K-means algorithm in your implementation? \n\nThroughout the paper, you use entropy as the foundation for measuring importance in different structure units. While entropy is a measure of information complexity, the paper lacks theoretical justification for why entropy specifically is an appropriate metric for determining what to prune in LLMs. Could you provide more theoretical explanation for why entropy is better than other potential metrics (such as activation magnitude, gradient-based importance, or contribution sufficiency) for identifying less important components in large language models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p8nkaZ3pwj", "reviewer": "Reviewer_nYT6", "summary": "The paper proposes a structured pruning framework for large language models (LLMs) that dynamically determines \"what to prune\" (specific modules) and \"how much to prune\" (pruning ratios) based on their importance. \n\nSpecifically, the method employ TE to quantify block-layer interaction and information entropy to guide pruning. A hierarchical strategy allocates pruning ratios to blocks and layers, balancing efficiency (inference speed) and performance (perplexity)", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer nYT6,\n\nThank you very much for your valuable feedback.  We first address the questions raised in the  “Questions for Authors” and “Other Strengths and Weaknesses” sections. For any additional concerns mentioned in other parts of the review(if they are distinct from those already covered), we also provide detailed responses and all experimental result tables have been compiled and are available at the following available link https://anonymous.4open.science/r/5BDF/README.md: \n\n---\n\n### **[Q1]: _How does the Gaussian assumption in Eq. 3 impact results if activations are non-Gaussian? Validation could strengthen the method’s generality?_**\n\nWe acknowledge that the true feature distribution in a LLM is unlikely to be perfectly Gaussian. Howeveer, the law of large number [1] suggests that the distribution can often be approximated by a Gaussian. We also find gaussian assuamption has been successfully applied to accelerating network like network quantization [2]. Empirically, we find that this approximation is sufficient to guide the network pruning in the ablation study of the pruning metrics. The detailed results are presented in Table 3 at the provided link.\n\n#### [1] Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics, 2020\n#### [2] Entropy-driven mixed-precision quantization for deep network design. Neurips2022\n\n---\n\n### **[Q2]: _Why weren’t dynamic pruning methods like ShearedLLaMA included as baselines? Their inclusion would clarify novelty against related work._**\n\nDynamic pruning iteratively identifies the component with the least impact before permanently removing it. In contrast, our method follows a **static pruning strategy**, which determines the pruning configuration for all structure units in a **single preprocessing stage**, avoiding repeated evaluations of the LLM’s internal state. Moreover, methods like **ShearedLLaMA** typically require **additional fine-tuning or retraining** following the pruning process while our method requires no post-training.  Our paper already includes comparisons with dynamic pruning methods such as **SLEB** and **BlockPruner**, and we have additionally compared to OWL and DSA using their settings. The corresponding results are shown in the Table 7 in the link.\n\n---\n\n### **[W1]: _This paper is generally well-written but algorithm steps (e.g., depth-first search compensation) could use more intuition._**\n\nWe provide a detailed explanation and additional ablation study in **W3.**\n\n---\n\n### **[W2]: _Also, this paper ignore the previous layer-wise assignment methods._**\n\nWe add the \"MultiPruner\" [3] pruning method as a baseline to evaluate its effectiveness on LLaMA2. The corresponding results are shown in the Table 6 in the link.\n\n#### [3] MultiPruner: Balanced Structure Removal in Foundation Models. arxiv 2025\n\n---\n\n### **[W3]: _The ablation study is missing:_**\n\n_over the metric TE:_\n\nWe use the Frobenius norm [4] as the new criterion to measure the change of LLM hidden state. The results are presented in Table 8 in the link.\n\n#### [4] Channel pruning for accelerating very deep neural networks. ICCV2017\n\n---\n\n_over the metric entropy:_\n\nWe compare the proposed method to activation-magnitude and gradient-magnitude methods in the Table 3 in the link.\n\n---\n\n_over the search method:_\n\nOur work adopts the depth-first search (DFS) strategy, with the goal of fully removing certain blocks/layers to fulfill the remaining pruning ratio defined in Equation (5), thereby solving the optimization problem formulated in Equation (6). Here we compare to an intuitive solution - greedy search. We observe greedy search tends to prioritize removing entire blocks during the compensation phase to quickly satisfy the remaining pruning ratio. However, due to the lack of a backtracking mechanism, greedy search selects the seemingly optimal option at each step but is prone to getting stuck in local optima. Our ablation study in Table 9 in the link further supports this observation, demonstrating that DFS consistently outperforms greedy search in terms of pruning effectiveness.\n\n---\n\n_Bias compensation:_\n\nWe have conducted an ablation study on the bias compensation strategy in Figure 6 of the paper.\n\n---\n\n###  **[Other1]：_Code availability is unclear, affecting reproducibility. (no additional supplementary provided)_**\n\nWe used the offical checkpoints (LLaMa2, LLaMA3&Vicuna) and followed the evaluation protocol of previous works like SliceGPT. Hence, our work warrants reproducibility. We will release the codebase upon publication.\n\n###  **[Other2]：_Typo in Table3: WADNDA → Wanda_**\n\nThank you for pointing this out. We apologize for the typo and will correct “WADNDA” to “Wanda” in Table 3 in the revised version.\n\n---\n\nWe would like to encourage the reviewer to ask questions on anything that may still be unclear in our responses or which we should clarify further.\n\nThank you very much,\n\nAuthors", "claims_and_evidence": "- The Gaussian assumption for entropy estimation (Eq. 3) is not validated, raising questions about its applicability to non-Gaussian activations.\n- Some baselines (e.g., ShearedLLaMA, LayerDrop, OWL, DSA) are missing, limiting context for dynamic pruning comparisons.\n\nOWL: https://arxiv.org/abs/2310.05175 \n\nDSA: https://proceedings.neurips.cc/paper_files/paper/2024/file/ff997469ac66cf893c4183efeb22212a-Paper-Conference.pdf", "methods_and_evaluation_criteria": "Methods and evaluation criteria seems reasonable.", "theoretical_claims": "the paper focuses on empirical validation. There is no theoretical proofs.", "experimental_designs_or_analyses": "Code availability is unclear, affecting reproducibility. (no additional supplementary provided)", "relation_to_broader_scientific_literature": "The work extends structured pruning literature (e.g., SliceGPT, LLM-Pruner) by introducing dynamic multi-unit pruning.", "essential_references_not_discussed": "Some baselines (e.g., ShearedLLaMA, LayerDrop, OWL, DSA) are missing, limiting context for dynamic pruning comparisons.\n\n\nDSA: https://proceedings.neurips.cc/paper_files/paper/2024/file/ff997469ac66cf893c4183efeb22212a-Paper-Conference.pdf \n\nOWL: https://arxiv.org/abs/2310.05175", "other_strengths_and_weaknesses": "This paper is generally well-written but algorithm steps (e.g., depth-first search compensation) could use more intuition.\n\nAlso, this paper ignore the previous layer-wise assignment methods.\n\nThe ablation study is missing:\n\n- over the metric TE and entropy\n- over the search method (DFS)\n- over the bias compensation", "other_comments_or_suggestions": "Typo in Table3: WADNDA → Wanda", "questions_for_authors": "- How does the Gaussian assumption in Eq. 3 impact results if activations are non-Gaussian? Validation could strengthen the method’s generality.\n- Why weren’t dynamic pruning methods like ShearedLLaMA included as baselines? Their inclusion would clarify novelty against related work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2pfJmpzRPM", "reviewer": "Reviewer_3YW3", "summary": "This paper propose a pruning method that targets multiple LLM modules with dynamic pruning ratios. It find the intrinsic of LLM can help to determine the importance of each module and\nthus distribute the pruning load on demand, i.e., what to prune and how much to prune. Extensive experiments on multiple benchmarks and LLM variants demonstrate that the proposed method effectively resolves the trade-off between efficiency and performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3YW3:\n\nThank you for your insightful review.  All experimental result tables have been compiled and are available at the following available link: https://anonymous.4open.science/r/5BDF/README.md. We have thoroughly considered your concerns and respond to them as follows :\n\n---\n\n### **[W1]: _While the core idea of the paper is effective, it does not introduce much novelty._**\nOur hierarchical pruning stretagy differs significantly from existing pruning approaches, particularly in its use of dynamically assigned pruning ratios and multi-structure pruning. Unlike methods such as SliceGPT, which necessitates a uniform and prescribed pruning ratio across all blocks, our approach dynamically allocates pruning ratios to each block based on the transfer entropy metric. Furthermore, we perform pruning across multiple structure units simultaneously, including blocks, layers, and rows or columns of weight matrices, while previous work mainly focus on a single unit. As shown in Table 1 and Table 2 in the paper, our method achieves strong performance across different pruning ratios in multiple LLMs. In terms of inference speed, our method outperforms approaches that only prune rows or columns of weight matrices, such as SliceGPT and FLAP. Compared to Table 2 (LLaMA2-7B), Table 5 of the Appendix shows our method achieves greater performance gains on larger models (LLaMA2-70B) and at higher pruning ratios, demonstrating its scalability. These findings demonstrates our insight: **pruning multiple structure units with dynamic pruning ratios can lead to balance between efficiency and effectiveness**.\nWe adopt information entropy as the pruning criterion, which distinguishes our method from commonly used activation-magnitude and gradient-magnitude approaches. Ablation study in Table 3 in the link demonstrates that entropy provides a more effective measurement of structural importance, leading to improved pruning performance compared to activation- or gradient-based methods.\n\n---\n\n### **[W2]: _The primary contribution appears to be a combination of several minor enhancements._**\nOur method is not a simple aggregation of several heuristics but a systematic pruning framework grounded in information-theoretic principles. Indeed, our approach fundamentally challenges the prevailing philosophy in prior works, which involve **pruning individual units based on a predefined ratio**. The pruning process under our framework is divided into two stages:\nIn the first stage, we introduce the metric of transfer entropy to analyze the interaction among blocks in LLMs. This enables us to dynamically determine the pruning ratio for each block and perform pruning on coarse-grained structure units such as blocks and layers.\nIn the second stage, we further allocate the pruning load within each block based on the information entropy of individual structural units, enabling fine-grained pruning of weight matrix rows and columns. \nFinally, our method introduces the bias compensation to enhance the pruned model without the need of post-training.\nThe ablation studies further validate the critical role and necessity of each component in the proposed framework, including the entropy-based importance metric, the hierarchical pruning strategy, and the dynamic ratio assignment. The results of these ablation studies are presented in Table 3,4 and 5 in the linked document. \nThis hierarchical pruning strategy not only significantly improves the inference speed of the pruned model, but also maximally preserves its original performance.\n\n---\n\nWe welcome any further questions or points of clarification the reviewer may have regarding our responses.\n\nThank you very much,\n\nAuthors", "claims_and_evidence": "YES", "methods_and_evaluation_criteria": "YES", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "YES", "relation_to_broader_scientific_literature": "This paper proposes an effective improvement to the existing LLM pruning method.", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "**Strengths:**  \n1. The paper is clearly written and easy to understand.  \n2. The proposed method outperforms existing LLM pruning techniques.  \n3. The experimental results are comprehensive, demonstrating the method's effectiveness across various LLM sizes.  \n4. The paper also shows actual inference speed improvements, highlighting its practical application value.\n\n\n**Weakness:**\n1. While the core idea of the paper is effective, it does not introduce much novelty.  \n2. The primary contribution appearing to be a combination of several minor enhancements.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see the weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a pruning method for LLMs which is based on two insights regarding weaknesses of current approaches in use. The paper does not make theoretical contributions, but the experiments are solid and convincing. All reviewers evaluated the work favourably. They also asked a large number of clarification questions that the authors were able to clear in depth — two out of the three reviewers raised their scores as a result. In light of this, I support the acceptance of this submission.
Decision: Accept (poster)