SlimLLM: Accurate Structured Pruning for Large Language Models
Authors: Jialong Guo
Xinghao Chen
Yehui Tang
Yunhe Wang
Venue: ICML 2025 poster (2025)
Keywords: structure pruning
large language models
machine learning
Primary area: deep_learning->large_language_models

Abstract: Large language models(LLMs) have garnered significant attention and demonstrated impressive capabilities in a wide range of applications. However, due to their enormous computational costs, the deployment and application of LLMs are often severely limited. To address this issue, structured pruning is an effective solution to compress the parameters of LLMs. Determining the importance of each sub-module in LLMs and minimizing performance loss are critical issues that need to be carefully addressed in structured pruning. In this paper, we propose an effective and fast structured pruning method named SlimLLM for large language models. For channel and attention head pruning, we evaluate the importance based on the entire channel or head, rather than merely aggregating the importance of individual elements within a sub-module. This approach enables a more holistic consideration of the interdependence among elements within the sub-module. In addition, we design a simple linear regression strategy for the output matrix to quickly recover performance. We also propose layer-based importance ratio to determine the pruning ratio for each layer. Based on the LLaMA benchmark results, our SlimLLM outperforms other methods and achieves state-of-the-art performance.

SlimLLM: Accurate Structured Pruning for Large Language Models
Jialong Guo Xinghao Chen
Yehui Tang' Yunhe Wang'
Abstract
translation to complex reasoning and problem-solving. With the enhancement of model capabilities, there is often cor- responding surge in computational expenses. This often limits the deployment and application of LLMs.
Large language models(LLMs)) have garnered sig- nificant attention and demonstrated impressive capabilities in a wide range of applications. How- ever, due to their enormous computational costs, the deployment and application of LLMs are of- ten severely limited. To address this issue, struc- tured pruning is an effective solution compress the parameters of LLMs. Determining the im- portance of each sub-module in L.MSs and min- imizing performance loss are critical issues that need to be carefully addressed in structured prun- ing. In this paper, we propose an effective and fast structured pruning method named SlimLLM for large language models. For channel and at- tention head pruning, we evaluate the importance based on the entire channel or head, rather than merely aggregating the importance of individual elements within a sub-module. This approach en- ables a more holistic consideration of the interde- pendence among elements within the In addition, we design a simple linear regression strategy for the output matrix to quickly recover performance. We also propose layer-based im- portance ratio to determine the pruning ratio for each layer. Based on the I.LaMA benchmark re- sults, our SlimLLM outperforms other methods and achieves state-of-the-arr performance.
Several techniques have been proposed to address the issue of computational cost in large language models, including model pruning (Ma et al., 2023: Chen et al., 2023), quantiza- tion (Frantar et al., 2022: Xiao al., 2023), and knowledge distillation (Saha et al., 2024). Among these, model pruning stands out as a particularly effective strategy. Model prun- ing (Anwar et al., 2017) involves the systematic removal of redundant or less important parameters from the model, which not only reduces the model's ' size but also decreases its computational requirements. Given these benefits, model pruning has emerged an ideal solution for large language models compression, particularly for deployment in envi- ronments with limited resources.
Pruning can be categorized into two main types: unstruc- tured pruning and structured pruning. Unstructured prun- ing involves removing individual weights from the model based on their magnitude or other criteria, which can lead to a sparse weight matrix. There are several representative works in this field, such SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2023). These methods are ef- fective in reducing the number of parameters but may not be optimal for hardware acceleration due to the irregular spar- sity pattern. Structured pruning, on the other hand, aims to remove channels, attention head, or layers, which results in more regular and maddware-friende sparsity pattern. For its better compatibility with existing hardware architectures, structured pruning has attracted much attention.
1. Introduction
Large language models(LLMs) (Achiam et al., 2023; Tou- vron et al., 2023), currently a popular area of research, have garnered significant attention due to their remarkable achievements in handling complex tasks. These models have not only demonstrated impressive capabilities in natural lan- guage processing but have also shown potential in a wide range of applications, from creative writing and language
Considering the enormous computational cost required for training LLMs, the method for L.MSs pruning typically em- ploys the strategy of post-training pruning. For instance, LLM-Pruner (Ma et al., 2023) measures the importance of weights based on gradient information and recovers the performance of the pruned model through efficient tuning techniques. Due to the necessity of computing gradients, significant amount of storage and computational resources is required. To address this issue, LoRAPrum (Zhang et al., 2023) proposes to estimate the importance of weights using LoRA (Hu et al., 2021). LoRAP (Li et al., 2024) designs gradient-free strategy to prune the unimportant channels,
'Hawei Noah's Ark Lab, China. Correspondence to: Xinghao Chen <xinghao.chenn@e www.chhwwweicomm...... Yunhe Wang <yunhe.wang huawei.com>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SlimLLM: Accurate Structured Pruning for Large Language Models
and combines Low-Rank matrix approximation to compress the weights of multi-head self-attention. As LoRAP es- timate the importance score of weight by calculating the product of the weight magnitude and the corresponding in- put activation's 1.2 norm, it ignore the direction of weight vector when pruning channels. To account for the influence of weight vector direction, we construct the feature space of the oulput and evaluate the importance of channels within this feature space. For MHA, we employ Pearson similarity to assess the significance of each head, which directly treats the head as a whole for evaluation. Besides, we design simple yet effective linear regression strategy to recover the performance loss due to pruning. At last, we propose a criterion to determine the pruning ratio for each layer.
consistently serve as indicators of filter importance. For LLMS pruning, Sheared LLaMA (Xia et al., 2023) adopts similar approach, compressing the model by learning set of pruning masks. This method can achieve performance comparable to models of equivalent sizes through sufficient training. To efficiently evaluate the importance of filters, several gradient- based methods have been proposed, such as -LM--runer and LoRAPrune. LLM-Prunet employs a first-order Taylor expansion evaluate the importance of weights. LoRAPrune uses the gradient of LoRA parameters to measure the importance of model's parameters. Addition- ally, LoRAP utilizes (Sun et al., 2023)'s 's methodology to assess the significance of weights, which achieves gradient- free importance estimation for weights. FLAP (An et al., 2024) proposed a fluctuation-based pruning criterion, tak- ing into account the fluctuations in activation values. Both of these methods calculate the importance score of filters by aggregating the importance scores of weights like sum- mation or L2 Norm. Furthermore, SlimGPT (Ling et al., 2024) incorporates the Optimal Brain Surgeon technique in structured pruning for LLMs, where filters are pruned by minimizing the squared error between the outputs before and after pruning.
Based on the LLaMA series of models, we extensively evalu- ate the effectiveness of our proposed method. As evidenced by the experimental results obtained from the Common- sense Reasoning datasets, our proposed method achieves 98.7% retention of the original performance on LLaMA-7B when the pruning ratio is set at 20% and outperforms current structured pruning methods.
Contributions. In this paper, our contributions can be sum- marized as follows:
Importance score of layers. For LLMs pruning, current methods typically employ a uniform ratio for pruning each layer. OWL (Yin et al., 2023) finds that non-uniform layer- wise sparsity typically obtain better performance and deter- mine the pruning ratio by Layerwise Outlier Distribution. Meanwhile, several works recognize the notable redundancy across the layers of L.MMs, and measure the importance of layers through the cosine similarity between inputs and out- puts. Some works (Men et al., 2024; Chen et al., 2024; Muralidharan et al., 2024) apply this strategy, using the cosine similarity to measure layer importance for pruning entire layers of LLMs. SlimGPT highlights that the error introduced during pruning one layer can accumulate and increase with model depth. To address this, SlimGPT de- signs an incremental pruning ratio for layers.
For the MHA sub-layer, we propose similarity im- portance for head pruning, which evaluates the entire head importance according to the pearson similarity between original output and output without correspond- ing head's contribution. Besides, we design greedy search algorithms to find better combination of heads.
For the FFN sub-layer, we devise a feature space im- portance method, which construct the feature space by output activations and assess the importance by consid- ering both direction and magnitude of the channel.
We find a simple yet effective strategy for performance recover. The method employs linear fitting of the out- puts before and after pruning each layer to fine-tune the parameters of the output matrix. Experimental results show that the method can recover the loss caused by pruning at extremely low cost.
Compression technologies. Based on the importance score of sub-module, there are different strategies to compression model. Specifically, unstructured pruning removes the least important elements in the matrix, while structured pruning typically removes entire rows (cols) or multiple rows (cols) in the matrix. Moreover, there have been some more coarse- grained pruning strategies recently. Some studies (Men et al., 2024; Kim et al., 2024; Song et al., 2024) compress models by removing entire layers, and others (Chen et al., 2024) replace transformer blocks with a lightweight net- work. Besides, low-rank approximation (Hsu et al., 2022; Yu & Wu, 2023) serves as an efficient technique for model compression. Low-rank approximation typically transforms matrices in the model into the product of two smaller ma- trices, thereby reducing the number of parameters. The
We design a non- non-uniform strategy to prune the different layers of LLMs, controlling the pruning ratio of each layer based on the cosine similarity between their input and output.
2. Related Work
Importance score of filters. The filter is the operational unit for structured pruning. In traditional structured prun- ing (Liu et al., 2017: Zhuang et al., 2020), scaling factors de- rived from Batch Normalization or learnable inserted masks
2

---

SlimLLM: Accurate Structured Pruning for Large Language Models
method proves to be highly effective when the matrix dis- plays pronounced low-rank property.
Algorithm Greedy Search for head pruning
Input: pruned attention heads set: S,=(head; if head, is pruned.), unpruned attention heads set: S (head, if head, unpruned.)
3. Preliminary
Output: left attention heads set Steft
Principal component analysis. Principal component anal- ysis (PCA) is a widely used dimension reduction tech- nique in data analysis. PCA can capture the directions of maximum variance in the original data. Given the data X E R=xm where is the number of data and repre- sents the dimension of data. PCA requires to calculate the covariance matrix of data, it can be describe as follows:
Steft S-p
O-p -p Output(S (S-p)
Oat = Output( Sp + S-p)
Sim - Pear son(O Onli)
for h; E Sp do
for h2 E S-p do
O Output(S
h2 + h4)
Pearson| (O, Onu))
if > Sim then
Cor = (x-) -4).
(1)
Sim =
Steft S - lig + h1
where J is the mean of data. Then, the covariance matrix can be decomposed into the following form:
end if end for
S-p = Steft
end for
Cou= = QAQ
(2)
where Q is the matrix composed of eigenvectors, A is the diagonal matrix, with each diagonal element representing an eigenvalue. The eigenvalues represent the magnitude of variance explained by each principal component. To obtain the
where W W K Wi, are query, key and value matrices in i-th head, dg is the dimension of head, h is the number of head, W is the weight matrix for the final linear projec- tion correspond to i-th head. For structured pruning, when the W is pruned, it effectively means that the i-th head's contribution to the final output is eliminated.
data, the original data is projected onto the selected principal components.
As we mentioned above, we consider the influence of W to evaluate the importance of each head, Given the input of output linear projection X, the score of i-th head can be described as follows:
4. Methods
We introduce novel approach to pruning large language models (LLMss). Our method involves compressing model parameters by reducing the number of attention heads and feed-forward network (FFN) channels. Specifically, we develop two distinct criteria for pruning different sub-layers. Then, we employ a linear regression strategy to finetune the weights of output matrix. At the end, we apply our non- uniform pruning ratio for each layer. We show our method in Figure 1.
Score; = - PewrronnXWWW....... XWo - X.W)).
(4)
We use pearson similarity to measure the linear correlation between the original output and the output without contri- bution of W Considering the interaction between outputs of each head, we design a greedy search algorithms to max- imize similarity. For each pruned head, we replace the unpruned heads in turn and evaluate the pearson similarity, Then we select combinations of heads with maximum simi- larity as heads left. The algorithm shows detail process of our greedy search for head pruning.
4.1. Similarity Importance for Attention Head Pruning
For multi-hean self-attention (MHA), given the input acti- vation X E NxC where N is the length of sequence and Cs the feature dimensions. The multi-head self-attention mechanism can be described as follows:
4.2. Feature Space Importance for Channel Pruning
When performing channel pruning, it means that remove row or column from the weight matrix, which represents a vector. Current methods typically first consider the impor- tance of individual elements in each row (or column), and then measure the importance of individual channels in an aggregated manner. This leads to the information loss of
= K, = XW2 V = XWL,
head, = (mmf//,,K,KK VV,
(3)
MHA(X) = Σ head, W
3

---

SlimLLM: Accurate Structured Pruning for Large Language Models
Pearson Similarity
0-hh
Norm
W.rrnned
Linear Regression
Attention
Feature Space Importance
y/A
W1
A
+ B
W2
Norm
x
x
FFN
W
Wrumned
Figure The overall framework our proposed SlimLLM. о-h, denotes the output excluding the head. For the MHA sub-layer. employ the Pearson similarity between 0-h, and the sum of all hcads' output to evaluate the importance of each head, and prune head with higher similarity when is inoperative. For FFN sub-layer, we map down matrix to feature space output activation, and calculate the channel importance based on the eigenvalues corresponding to the eigenvectors. Finally, we apply linear regression to fine-tune the output matrix of each sub-layer.
vector direction. Different from previous works, we propose a novel method that takes into account both the direction and magnitude of vectors when assessing importance.
feature space can be calculated as follows:
1 = 11 W2 WDDColl.
Inspired by principal component analysis, can construct feature space and evaluate the importance of each direction by calculating the eigenvectors and eigenvalues of the output matrix. Let Y E RN represents the output of final linear projection, where N is the sequence length and D the output dimension. We can calculate its cigenvectors and eigenvalues by PCA. Let Q RDxD represents the matrix composed of eigenvectors, M RD represents the vector composed of eigenvalues, and Q., represents the eigenvector corresponding the i-th eigenvalue M, Then, we can map the down matrix W of FFN to the space composed of cigenvectors using the following formula:
where W represents the element in the j-th row and i-th column of W".
Following (Li et al., 2024), we consider the dependencies between neurons and the influence of input activation. The formulation of j-th channel group importance is follows:
4, = 1|,,1211125 + 1|X W + 1| Wippll.
(8)
where 11X5122 is the 1.2 Norm of the input corresponding j- th channel. X 2.2 denotes a vector comprising the L2 Norm of the input activations corresponding to W W Watte and W are the elements in the j-th of gate matrix and up matrix in FFN.
W' W down Q,
(5)
gate or Wi up
Combining the eigenvalues corresponding to each eigenvec- tor, the importance score of i-th eigenvector can be calcu- lated as follows:
4.3. Linear Regression for Performance Recovery
Since we reduce the number of attention heads and chan- nels directly, adopted simple and fast linear regression strategy restore model performance. Let O represents the output of final linear projection of MHA or FFN, The i-th dimension of output before and after pruning are re- spectively expressed as O and Op We apply a simple linear fitting of the output before and after pruning. The
C (is.
(6)
where M is the mean of M. We use the function of sigmoid to smooth the eigenvalues, allowing channel importance to take into account the influence of more eigenvector direc- tions. Then, the importance score of j-th channel based on

---

SlimLLM: Accurate Structured Pruning for Large Language Models
leads suboptimal outcomes. Inspired by the work in (Men et al., 2024), which suggests that the impact of transformer block is determined by its ability to change hidden states, we utilize the cosine similarity between the input and output of a layer to determine the pruning ratio.
1.06
1.04
Mean value
1.02
1.00
R
Let Xi, represents the t-th row of input to layer The pruning ratio of layer is calculated by follows:
0.98
0.96
-9- MHA
0.94
FFN
X (11) [X,,,ll2 1.5.ll2-||4422.l
Jager -r--000 "ro-soffmma(( E
ri
5
10
15
20
25
30
Layer id
Both of ro and a are constant and their value are related to the pruning ratio. E represents the operation of expectation. Additionally, for smaller pruning ratios, a larger a can be used, whereas for larger pruning ratios, a should be reduced.
Figure 2. Different layers' mean value of the coefficients in MIAA and FFN LLaMA-7B. formulation as follows:
Based experimental observations, the cosine similarity between the first layer and the last layer is found to be relatively low. This observation is in accordance with the prevailing notion that the first and last layers tend to be of greater significance. Similar to many current methods (Ma et al., 2023; Kim et al., 2024), we skip the first and last layers during pruning. Moreover, at a pruning ratio of 20%, adopt the strategy from LLM-pruner and bypass addi- tional layers to achieve higher model performance. In the case of the LLaMA-7B model, it is observed that the co- sine similarity exhibits a gradual upward trend as the layer depth increases. Consequently, we employ a stratified prun- ing strategy, wherein the pruning ratios are systematically lower in the shallower layers and incrementally higher in the deeper layers.
O, = A.Oprmned + Bi:
(9)
Where A; and B; are coefficients of a first-order linear function corresponding i-th dimension of output. The coef- ficients can be calculated using least squares. Additionally, select the pearson similarity to determine the importance of attention heads, which helps maximize the linear corre- lation between O2 and Opruned and reduces fitting error. To mitigate the computational overhead, we implement our piecewise importance score within the FFN.
After the coefficients of A and B calculated, the formula- tion of final linear projection can be described as follows:
O= Xix (A (o-W))) + BB,
(10)
5. Experiments
where Xn is the input of final linear projection of MHA or FFN.
5.1. Experimental Settings
Evaluation and Datasets. We evaluate the performance of model by performing zero-shot task classification on common sense reasoning datasets, which follow the setting of LLM-pruner (Ma al., 2023), including BoolQ (Clark et al., 2019), PIQA (Bisk al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC- easy (Clark et al., 2018). ARC-challenge (Clark et al., 2018) and OpenbookQA (Mihaylov et al., 2018). Meanwhile, a zero-shot perplexity (PPL) evaluation is also conducted on the WikText2 (Merity et al., 2016) and PTB datasets (Mar- cus et al., 1993). We performed extensive experiments on LLaMA-1 and LLaMA-2 models rigorously validate the efficacy of our proposed method.
We list the mean values of A obtained by linear regression for each sub-layer under pruning ratio of 50% on LLaMA- 7B. The results are show in Figure 2. The results reveal that the mean values of A consistently hover around 1.0. This observation suggests that despite a substantial pruning ratio, the output magnitude of the sub-layer remains closely aligned with the original output, thereby preserving the essential characteristics of the model's performance. By making fine adjustments to the output matrix through linear regression, the model can reduce the output error caused by pruming and effectively maintain its performance.
Simultaneously, the values of also fluctuate around zero.
For the calculation of importance score, we randomly selected 32 samples from Bookcorpus, and the sequence length of each samples is 128. When calculating the pruning ratio of layers, we skipped some of
Implementation Details.
4.4. Pruning Ratio for Layers
Finding an appropriate pruning ratio for each layer is crucial. Maintaining a uniform pruning ratio across all layers often
5

---

SlimLLM: Accurate Structured Pruning for Large Language Models
Table Zero-shot performance of the compressed I.a.A- The average computed the Commonsense Reasoning datasets.
The "bolded" represents the best result under the same pruning ratio.
Ratio
Method
WikiText2
PTB
BoolQ PIQA HellaSwag WinoGrande ARC-e ARC-c OBQA Average
Ratio=0%
Llama-7B
12.62
22.14
73.18
78.35
72.99
67.01
67.45
41.38
42.4
63.25
LLM-pruner
19.09
34.21
57.06
75.68
66.8
59.83
60,94
36.52
40.0
56.69
Ratio=20% LoRAPrune
20.67
34.12
57.98
75.11
65.81
59.90
62.14
34.59
39.98
56.50
w/o tune
LoRAP
15.69
25.86
71.93
76.44
69.98
65.9
60.56
38.48
40.4
60.53
Ours
15.95
26.09
72.72
75.95
69.82
66.06
64.48
39.33
40.2
61.22
LLM-pruner
17.58
30.11
64.62
77.2
68.8
63.14
64.31
36.77
39.8
59.23
Ratio=20% LoRAPrune
16.80
28.75
65.62
79.31
70.00
62.76
65.87
37.69
39.14
60.05
w/ tune
LoRAP
16.35
27.06
72.94
76.93
70.9
65.75
64.31
39.93
41.2
61.7
Ours
15.55
26.66
74.71
76.61
71.23
66.54
66.96
40.61
40.2
62.41
I. L.-M-rrneer
112.44
255.38
52.32
59.63
35.64
53.20
33.50
27.22
33.40
42.13
Ratio=50% LoRAPrune
121.96
260.14
51.78
56.90
36.76
53,80
33.82
26.93
33.10
41.87
w/o tune
LoRAP
56.96
87.71
57.8
63.82
46.96
57.3
40.36
27.73
36.80
47.25
Ours
37.89
67.68
63.33
65.40
49.94
58.80
45.83
30.38
37.00
50.10
L.MM-pruner
38.12
66.35
60.28
69.31
47.06
53.43
45.96
29.18
35.60
48.69
Ratio=50% LoRAPrune
30.12
50.30
61.88
71,53
47.86
55.01
45.13
31.62
34.98
49.71
w/ tune
LoRAP
30.90
48.84
63.00
69.64
54.42
58.41
51.94
32.00
35.80
52.17
Ours
26.71
42.19
62.78
68.99
54.73
61.01
54.55
33.28
36.80
53.16
the most important layers. For example, the pruning ratio of first and last layer is set to zero when pruning model. when finetuning, we use a single GPU with 2 epochs on cleaned version of Alpaca (Taori et al., 2023), retaining the same settings as LLM-pruner We finetune the pruned model with LoRA. The learning rate set to lc-4, and the batch-size is 64. For the pruning ratio assigned to each layer, when the pruning ratio is set at 20%, the parameter in Equation I1 is configured to be 10. When the pruning ratio is increased to 50%, we correspondingly decrease the parameter value, setting it to 7.
rank approximation to compress the weight matrices of MHA, while adopting a group importance strategy for FFN. Based on this strategy, the method achieves state-of-the-ar performance.
5.2. Zero-shot Performance.
Table 1 shows the zero-shot performance of the pruned model. On the LLaMA-7B model, employing 20% prun- ing ratio without post-training results in an average score drop of 2.03%. However, through efficient post-training, the accuracy of the pruned model can be enhanced by 1.19%. Compared with other methods such as LoRAP, the score of our method is improved by 0.7% with 20% pruning ratio. When compared with PPL, LoRAP demonstrates slight advantage in performance on the WikiText2 dataset without finetuning. Conversely, it shows minor deficiency in per- formance PTB dataset. In the context of finetuning, approach attains optimal results on both PPL metrics. When the pruning ratio reaches 50%, our method still manages to retain more performance comparing to other methods. Specifically, in the absence of finetuning, our method at- tains an average accuracy on the Commonsense Reasoning datasets that surpasses that of the LoRAP by 2.85%. In the finetuning scenario, our method achieves a performance improvement of 1 % over other approaches. Specifically, our approach demonstrates superior performance across the majority of Commonsense Reasoning datasets.
Baselines. We select the following peshorerformncc struc- tured pruning methods in recent years benchmarks:
LLM-pruner (Ma et al., 2023), as the first structured pruning method applied to LLMs, has been widely used as a benchmark in numerous studies. This work groups the structures based on the dependencies be- tween weights and prunes the least important groups using Taylor expansion.
LoRAPrune (Zhang et al., 2023) is the method which reduces the gradient computation and memory con- sumption of LLMM-pruner by employing the weights of LoRA.
LoRAP (Li et al., 2024) employs distinct compression strategies for MHA and FFN components within the Transformer architecture, Specifically, it utilizes low-
Table presents our evaluation results on the LLaMA2-7B
6

---

SlimLLM: Accurate Structured Pruning for Large Language Models
Table Zero-shot performance of the compressed I.I.a ..M.7277B. The average score is computed on the Commonsense Reasoning datasets.
The "bolded" represents the best result under the same pruning ratio.
Ratio
Method
WikiText2
PTB
BoolQ PIQA HellaSwag WinoGrande ARC-e ARC-e OBQA
Average
Ratio=0%
Llama2-7B
12.18
47.25
71.04
78.40
72.96
67.17
69.32
40.53
40.80
62.89
Ratio=20% LoRAP
15.02
58.44
69.24
76.39
69.15
65.11
61.99
35.58
38.60
59.44
w/o tnee
Ours
15.70
56.33
69.79
76.28
68.88
63.54
65.74
39.08
39.80
60.44
Ratio=20% LoRAP
14.67
57.52
70.89
78.13
69.93
65.67
65.99
38.48
39.60
61.24
w/ tune
Ours
15.28
55.46
72.29
78.02
70.95
64.88
67.17
38.99
39.60
61.70
Ratio=50% LoRAP
60.89
282.22
61.86
62.23
43.98
55,41
38.51
27.6
33.00
46.09
w/o tune
Ours
38.64
141.06
62.69
64.74
45,91
53.28
39.73
29.01
33.2
46.93
Ratio=50% LoRAP w/ tune
26.26 27.29
101.22
63.27
70.78
55.14
57.85
52.15
30.97
36.00
52.31
Ours
88.28
64.19
69.04
53.60
55.33
52.53
32.08
37.40
52.02
Table 3. Inference latency of compressed model.
Table 4. Ablation Studies for the impact of our strategies at 50%
pruning ratio.
Ratio
#Params
Prefill(s)
decoding(s)
Method
WikiText2
PTB
Avg.
0%
6.7B
0,3008
13.12
SlimLLM
37.89
67.68
50.10
20%
5,4B
0.1429
11.48
Greedy search w/o feature space importance
40.89
82.60
48.35
50%
3,4B
0.1034
9.38
38.26
71.21
49.74
w/o non-uniform pruning ratio
66.37
123.52
42.16
model. We compare our method with LoRAP. As can seen from the results, our method performs better on the majority of datasets. At a pruning ratio of 20%, our method yields slightly higher PPL the Wiki'Text2 dataset com- pared to LoRAP, while achieving better results on the PTB dataset and greater average score on Commonsense Rea- soning datasets. At a 50% pruning ratio, without fine-tuning, our method attains an average score of 46.93% which is approximately 0.8% higher than LoRAP. After finetuning, LoRAP's average results are slightly higher, which is pri- marily due the inability to align the finetuning strategies.
5.4. Ablation Study
To verify the effectiveness of our proposed method, we con- ducted experiments on the LLaMA-7B model to validate each of the strategies we introduced. Unless otherwise spec- ified, all experiments were conducted without fine-tuning at pruning ratio of 50%.
Greedy search for head pruning. From Table can be observed that without greedy search for head pruning, the model's performance metrics generally decline across all evaluation criteria. Among these results, the average score of Commonsense Reasoning datasets dropped by nearly 2%, and the PPL metric of WikiText2 and PTB also deteriorated significantly. This indicates that there is interdependence among the outputs of different heads, and the linear corre- lation of outputs before and after pruning assess the importance of head combinations. Building on the basis of individual head importance, greedy search can rapidly and effectively identify superior head combinations, thereby enabling the pruned model to retain more of its original performance.
5.3. Latency of compressed model
Table illustrates the model size and inference latency of the pruned LLaMA-7B. The prefill latency was evaluated using an input sequence length of 256 and single-token genera- tion task. To test decoding latency, we set the batch size to 8 and measured the latency for generating 256 tokens. For each specified pruning ratio, the experiment was repeated 20 times to ensure statistical robustness, and the average latency was computed the definitive outcome. All latency measurements were conducted on single NVIDIA V100 GPU. As detailed in Table 3, pruning ratio of 50%, the latency for generating single token, given a consistent in- put sequence length of 256, decreased from 0.3008 seconds to 0.1034 seconds. Simultaneously, the decoding latency is reduced by 28.5% compared to the unpruned model when the pruning ratio is set at 50%.
effectively
It worth noting that, for FFN sub-layer, the outputs of dif- ferent channels also exhibit direct interdependencies. How- ever, considering that the number of channels in the interme- diate layer of FFN is considerably larger than the number of attention heads, and given the associated computational overhead, we refrained from employing the greedy search strategy in the channel pruning.

---

SlimLLM: Accurate Structured Pruning for Large Language Models
Table 5. Ablation Studies for the value of a.
Table 6. Ablation Studies for the impact of our Linear Regression for output matrix.
Value
WikiText2
PTB
Avg.
Ratio
Method
WikiTexx2
PTB
Avg.
54.17
96.57
44,42
a 1
43.36
83.91
44,95
Ratio=20%
w/ LR
15.95
26.09
61.22
Q 7
37.89
67.68
50.10
w/o tune
w/o LR
16.78
26.98
60.89
Q 10
44.82
75.80
46,86
Ratio=50%
w/ LR
37.89
67.68
50.10
w/o tune
w/o LR
48.66
81.64
46.66
The effective of feature space importance. Given that the output values are linear combinations of the corresponding weight vectors of each channel, we assess the importance of each channel based on the feature distribution of the output values. The results demonstrate that feature space importance enhances the selection of pruning channels for the FFN by integrating the significance of individual fea- ture directions. Benefiting from the incorporation of feature space importance, the pruned model achieves an average score improvement of nearly 0.4%. while also realizing notable enhancements in PPL results. The method primar- ily enhances the original approach based on the magnitude of weight vectors by incorporating the consideration of di- rectional importance. When the magnitude of the weight vectors is large, this strategy can combine vector direction to further optimize the selection of pruning channels.
models not using this strategy. The changes in PPL exhibit similar pattern, with the recovery becoming increasingly significant the pruning ratio is elevated. For example, at pruning ratio of 20%, the PPL on the WikiText2 dataset is decreased by 0.83 via the linear regression strategy. When the pruning ratio is increased to 50% the reduction in PPL reaches 10.77. This indicates that our linear regression strat- egy is effective in restoring performance under high pruning ratios. Moreover, since we only utilize a small calibration set for the least squares linear regression, the computational cost of this method is extremely low. Since it is only applica- ble when the output dimensions are identical, we currently employ the linear regression strategy solely on the output matrices of MHA and FFN. Further exploration is needed for rapid fitting of other parameter matrices.
The pruning ratio for different layers. As indicated in Table 4, the pruning ratio assigned to each individual layer exerts the most substantial influence on the model's overall performance. By incorporating layer importance into the pruning ratio allocation, the proposed strategy results in a pruned model with an average performance improvement of approximately 8 percentage points over uniform pruning, significantly alleviating the accuracy drop induced by model compression. Moreover, under the non-uniform pruning strategy, the pruned model's PPL of WikiText2 and PTB is nearly halved. In Equation 11, we introduce the parameter a to control the fluctuation range of the pruning ratios across layers. The Table 5 presents the impact of different values on model performance. As the results indicate, a pruning ratio of 50% combined with a leads improved model performance after pruning.
6. Conclusion
In this paper, we present SlimLLM, a novel and efficient pruning technique specifically designed for large-scale mod- cls. For both the Multi-Head Attention (MHA) and Feed- Forward Network (FFN) sub-layers, we adopt a holistic importance strategy that goes beyond mere aggregation of individual weight importance. Specifically, for head prun- ing, we directly assess the importance of each head by estab- lishing the linear relationship between the output of single head and the combined output of all heads. Furthermore, acknowledging the interdependencies among head outputs, we introduce a greedy search algorithm to explore more op- timal head pruning combinations. For channel pruning, we integrate both the magnitude and direction of each channel's corresponding weight vector and utilize Principal Compo- nent Analysis (PCA) to determine the importance of each feature direction in the output matrix, thereby deriving a fea- ture space importance that is based on both magnitude and direction. addition, we devise a simple linear regression strategy for the output matrix to rapidly and effectively re- store model performance. Considering that different layers of the model have varying degrees of parameter redundancy, we utilize the cosine similarity between the input and output of each layer to determine the pruning ratio for that layer. This approach further enhances the pruned model's 's per- formance. Results show that SlimLLM achieves excellent performance across different models and pruning ratios.
Linear regression for output matrix. Linear regression is the strategy we employ to rapidly restore the accuracy of pruned models. We perform individual linear fitting for each output dimension of the final output matrices of MHA and FFN within the Transformer blocks, aiming to make the model's 'ss output as close as possible to the output before prun- ing, As shown in Table 6, when the pruning ratio is 20%, the average score of the model without the linear regression strategy is 60.89, which is approximately 0.3% lower than that of the method using this strategy. When the pruning ratio reaches 50%, the linear regression strategy becomes even more effective in enhancing model performance, re- sulting in an approximate 3.4% improvement compared to
8

---

SlimLLM: Accurate Structured Pruning for Large Language Models
Impact Statement
Hsu, Y.-C., Hua, T., Chang, S., Lou, Q.. Shen, Y., and Jin,
H. Language model compression with weighted low-rank factorization. arXiv preprint arXiv:
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
2022.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L.,, and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09000 2021.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,, Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened Ilama: Depth pruning for large language models with comparison of retraining methods. arXiv preprint arXiv:2402.02883 02834, 2024.
An, Y., Zhao, X., Yu, T., Tang, M., and Wang, J. Fluctuation- based adaptive structured pruning for large language mod- cls. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865-10875. 2024.
Li, G., Tang, Y., and Zhang, W. Lorap: Transformer sub- layers deserve differentiated structured compression for large language models. arXiv preprint arXiv:2404.09695, 2024.
Anwar, S., Hwang, K., and Sung, W. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13(8):1-18, 2017.
Ling, G., Wang, Z., Yan, Y., and Liu, Q. Slimgpt: Layer- wise structured pruning for large language models. arXiv preprint arXiv:2412. 18/10, 2024.
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. Learning efficient convolutional networks through net- work slimming. In Proceedings of the IEEE international conference computer vision, pp. 2736-2744, 2017.
Bisk, Y., Zellers, R., Gao, Choi, Y., al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruming of large language models. Advances in neural information processing systems, 36:21702-21177 2023.
Chen, T., Ding, T., Yadav, B., Zharkov, I., and Liang, L. Lorashear: Efficient large language model struc- tured pruning and knowledge recovery. arXiv preprint arXiv:2310.18355 2023.
Marcus, M., Santorini, B., and Mareinkiewicz, M. A. Build- ing large annotated corpus of english: The penn tree- bank. Computational linguistics, 19(2):313-330, 1993.
Chen, X., Hu, Y., and Zhang, J. Compressing large language models by streamlining the unimportant layer. arXiv 2024.
preprint
Men, X., Xu, M., Zhang. Q., Wang, B., Lin, H. Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language mod- els are more redundant than you expect. arXiv preprint mmXhv:2403.0385. 2024.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T. Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 19055 10044, 2019.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.08 008400099077,, 2016.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1rXiv:1803.05455 2018
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 1809 2018.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference on Machine Learning, pp. 10323- 10337. PMLR, 2023.
Muralidharan, S., Turuvekere Sreenivas, S., Joshi, R., Cho- chowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. Advances in Neu- ral Information Processing Systems, 77:4107641100 2024.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arX1v:2210.17323, 2022.

---

SlimLLM: Accurate Structured Pruning for Large Language Models
Saha, S., Hase, P., and Bansal, M. Can language models teach? teacher explanations improve student performance via personalization. Advances in Neural Information Processing Systems, 36, 2024.
Zhuang, T., Zhang, Z., Huang, Y., Zeng. X., Shuang. K., and
Li, X. Neuron- structured pruning using polarization regularizer. Advances in neural information processing systems, 33:9865-9877 2020.
Sakaguchi, K., Bras, R. L.., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-11)) 106, 2021.
Song. J., Oh, K., Kim, T., Kim, H., Kim, Y., and Kim, J.-J. Sleb: Streamlining Ilms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09005. 2024.
Sun, M., Liu, Z.,, Bair, A., and Kolter, J. simple and effective pruning approach for large language models. arXiv preprint Ar3iv:206.. 1695, 2023.
Taori, R., Gulrajani, I. Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following Ilama model, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971 2023.
Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared Ilama: Accelerating language model pre-training via structured pruning, arXiv preprint arXiv:2310.0660
2023.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Yin, L.,, Wu, Y., Zhang, Z., Hsieh, C.-Y,, Wang, Y., Jia, Y., Li, G., Jaiswal, A., Pechenizkiy, M., Liang, Y., al. Out- lier weighed layerwise sparsity (owl): A missing secret sauce for pruning lims to high sparsity. arXiv preprint 10.005 2023.
Yu, H. and Wu, J. Compressing transformers: features are low-rank, but weights are not! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11007-11015, 2023.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 19050090 2019.
Zhang, M., Chen, H., Shen, C., Yang. Z., Ou, L., Yu, X., and Zhuang, B. Loraprune: Pruning meets low- rank parameter-efficien fine-tuning. arXiv preprint arXiv:2305. 18403, 2023.
10

---

SlimLLM: Accurate Structured Pruning for Large Language Models
A. Quantity of calibration data
Table illustrates the influence calibration set size on model accuracy. As shown in the results, perplexity tends to decrease with an increasing calibration set size. Notably, when the size is below 16, the average score on Commonsense Reasoning datasets shows considerable variation. The optimal performance observed when the calibration size is 32. Accordingly, adopt 32 as the default calibration set size in pruning framework.
Table 7. Ablation Studies for the quantity of calibration data. Avg, presents the average score Commonsense Reasoning datasets.
Number
4
64
WikiTexx24
58.8
45.89
45.98
37.89
39.56
PTB..
129.95
99.64
99.25
80.06
67.68
72.05
Avg.t
46.50
49.14
47.89
45.06
50.10
49.31
B. Experiments on other models.
In addition, we present pruning results on the Vicuna-7B and LLaMA-13B models. The corresponding experimental results are shown in Table 8 and Table 9, respectively. The results demonstrate that our method achieves competitive performance across various pruning ratios. Moreover, larger models tend to suffer less performance degradation under the same pruning ratio. Notably, compared with LoRAP, their fine-tuned models exhibit better performance in some cases under a 50% pruning ratio. We consider this due to the fact that LoRAP reconstructs the decomposed matrices in MHA during training, thereby enabling the utilization of more parameters in the optimization process.
Table 8. Zero-shot performance of the compressed Vicuna-7B
Method
Ratio
WikiText2
PTB
|BoolQ PIQA HellaSwag WinoGrande ARC-e ARC-c OBQA
Average
Vicuna-7B Ratio=0%
16.23
58.12
75.69
77.91
71.04
67.80
68.98
40.70
42.20
63.47
Ratio=20%
20.24
68.75
74.92
76.12
67.98
65.82
67.09
39.33
42.60
61.98
w/o tune
Ratio=50%
43.96
131.49
61.31
67.25
48.91
56.35
48.48
31.06
36.00
49.91
Ratio=20%
17.15
59.03
76.15
76.39
69.32
64.72
68.56
39.25
41.80
62.31
w/ tune
Ratio=50%
29.98
87.93
62.60 69.53
53.66
57.77
55.30
31.91
37.40
52.60
Table Zero-shot performance of the compressed LLaMA-13B.
Method
Ratio
WikiText2
BoolQ
PIQA
HellaSwag WinoGrande ARC-e ARC-c OBQA
Average
LLaMA-13B
Ratio=0%
11.58
68.47
78.89
76.24
70.09
74.58
44.54
42.00
64.97
Ratio=20%
13.35
74.13
77.53
74.73
69.30
70.45
42.32
41.00
64.21
w/o tune
Ratio=50%
25.64
62.31
72.20
60.84
60.62
55.60
33.87
37.80
54.75


Reviews:
[{"review_id": "99GwlhmdCy", "reviewer": "Reviewer_aFXN", "summary": "The paper presents SlimLLM, a structured pruning approach for large language models (LLMs) that tackles channel and attention head pruning through a unified importance evaluation framework. The paper introduces several novel techniques, including using Pearson similarity to identify redundant attention heads with a greedy search strategy, evaluating FFN channel importance via PCA to prioritize feature directions, and a layerwise pruning strategy that adjusts sparsity based on input-output cosine similarity. Evaluated on LLaMA-7B/2-7B, SlimLLM achieves better accuracy and lower perplexity than baselines like LoRAP.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The paper focuses on pruning-based acceleration of LLMs but lacks a comparison with quantization-based acceleration methods. Including such a comparison would better demonstrate the proposed approach's advantages of LLM compressing techniques.\n\nA1:  Thank you for your suggestion. Quantization is also a highly effective method for LLM compression. Since the directions of pruning and quantization as compression strategies are different and they can yield cumulative benefits, we have not compared our method with quantization at present.\n\nQ2: Table 2 only compares the proposed method with LoRAP. Expanding the evaluation to include more state-of-the-art pruning or compression baselines would strengthen the demonstration of the method's effectiveness and generality.\n\nA2: Thank you for your suggestion. In Table 2, we compared our method with the LoRAP method, which has excellent compression performance. However, due to memory limitations, we were unable to conduct experiments on LLMPruner with Llama2-7B. Below are the relevant results reproduced from other works, which show that our pruning strategy can better preserve model performance.\n\n|    Methods    | ratio | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|---------------|-------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| LLaMA2-7B     |  0%   | 71.04 | 78.40 | 72.96     | 67.17      | 69.32 | 40.53 | 40.08 | 62.89   |\n| LoRAP         | 20%   | 69.24 | 76.39 | 69.15     | 65.11      | 61.99 | 35.58 | 38.60 | 59.44   |\n| LLMPruner     | 20%   | 67.95 | 77.58 | 71.43     | 64.01      | 63.51 | 38.05 | 39.80 | 60.33   |\n| SLimLLM(ours) | 20%   | 69.79 | 76.28 | 68.88     | 63.54      | 65.74 | 39.08 | 39.80 | 60.44   |", "claims_and_evidence": "The claims made in the paper are well-supported by extensive comparisons and ablation studies.", "methods_and_evaluation_criteria": "The proposed method is evaluated on various benchmarks for LLaMA-7B and LLaMA2-7B. Evaluation focuses on zero-shot accuracy across commonsense reasoning datasets (e.g., BoolQ, PIQA), perplexity (WikiText2, PTB), and inference latency.", "theoretical_claims": "The paper does not provide too much theoretical analysis.", "experimental_designs_or_analyses": "The experimental setup is reasonable, with all comparative experiments conducted under the same conditions. Additionally, extensive ablation studies are provided to demonstrate the effectiveness of each proposed module.", "relation_to_broader_scientific_literature": "The paper focuses on LLM pruning, a critical problem in the field that has garnered significant attention from both the academic community and industry.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "Strengths\n\n1. SlimLLM introduces novel criteria for structured pruning, such as Pearson similarity-based head pruning and PCA-driven feature space importance for FFN channels. These methods move beyond element-wise aggregation, capturing interdependencies within sub-modules and providing a more comprehensive assessment of redundancy. The greedy search for head combinations further enhances this approach, addressing limitations of prior works like LoRAP that rely on weight magnitude or activation norms.\n\n2. The proposed linear regression strategy for output matrix fine-tuning is both simple and effective. It achieves significant performance recovery with negligible computational cost, avoiding complex retraining.\n\n3. The paper provides comprehensive comparative experiments and ablation studies to justify the effectiveness of the proposed method.\n\nWeaknesses:  \n\n1. The paper focuses on pruning-based acceleration of LLMs but lacks a comparison with quantization-based acceleration methods. Including such a comparison would better demonstrate the proposed approach's advantages of LLM compressing techniques.  \n\n2. Table 2 only compares the proposed method with LoRAP. Expanding the evaluation to include more state-of-the-art pruning or compression baselines would strengthen the demonstration of the method's effectiveness and generality.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kQzzlvMz33", "reviewer": "Reviewer_YsF4", "summary": "This paper proposes a structured pruning method for large language models (LLMs) that compresses both the feed-forward (FFN) and attention layers to accelerate inference. The pruning algorithm incorporates two key techniques: (1) removing redundant attention heads based on Pearson similarity and (2) pruning FFN layers using principal component analysis (PCA) on feature representations. After pruning, the method employs linear regression to efficiently recover the pruned weights by minimizing reconstruction error. The paper evaluates the approach on the LLaMA-7B model, assessing performance across benchmarks including WikiText, PTB, and BoolQ, among others. Experimental results demonstrate that SlimLLM outperforms baselines such as LLM-Pruner, LoRAPrune, and LoRAP.\n\n## update after rebuttal\n\nThanks for the response. I will keep my initial positive score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Although the proposed sub-methods have been proven effective, the relation between FFN pruning and attention layers pruning is no so clear. Are they independent methods?\n\nA1: In this method, both the head pruning and channel pruning strategies are designed to increase the linear correlation between the outputs before and after pruning, which benefits our linear regression strategy. Meanwhile, considering the significant difference in the number of heads and channels, we employ the PCA method in the FFN to make the pruned output as close as possible to the principal direction. In the MHA, we directly maximize the Pearson similarity to bring the similarity closer.\n\nQ2: The paper mainly focuses on LLaMA models. It is encouraged to conduct more experiments on other models.\n\nA2: Thank you for your suggestion. We have added the experimental results on Vicuna-7B.\n\n|      Methods       | ratio | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|--------------------|-------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| Vicuna-7B          |  0%   | 75.69 | 77.91 | 71.04     | 67.80      | 68.98 | 40.7 | 42.20 | 63.47   |\n| LLMPruner w/o tune | 20%   | 62.87 | 75.41 | 64.00     | 58.41      | 60.98 | 37.12 | 39.00 | 56.83   |\n| LLMPruner w/ tune  | 20%   | 60.40 | 75.63 | 65.45     | 63.22      | 63.05 | 37.71 | 39.00 | 57.78   |\n| LoRAP w/o tune     | 20%   | 76.42 | 76.38 | 68.31     | 64.96      | 65.82 | 37.29 | 38.60 | 61.11   |\n| LoRAP w/ tune      | 20%   | 75.81 | 76.77 | 68.39     | 65.04      | 70.08 | 39.33 | 39.20 | 62.09   |\n| SLimLLM w/o tune   | 20%   | 74.92 | 76.12 | 67.98     | 65.82      | 67.09 | 39.33 | 42.60 | 61.98   |\n| SLimLLM w/ tune    | 20%   | 76.15 | 76.39 | 69.32     | 64.72      | 68.56 | 39.25 | 41.80 | 62.31   |\n\n\nQ3: The performance gain appears to be not so significant at a 20% pruning ratio, as shown in Table 5. This suggests that SlimLLM might be more beneficial for higher compression levels but less impactful for moderate pruning. A deeper investigation into why performance varies across different pruning ratios would be valuable.\n\nA3: We posit that the impact on model performance is comparatively minimal at a pruning ratio of 20%, whereas the pruning error becomes significantly more pronounced at a pruning ratio of 50%. Our proposed strategy is capable of effectively mitigating this error, thereby yielding more substantial performance improvements.\n\nQ4: The layer-wise pruning ratio relies on empirical hyperparameter α, lacking theoretical justification and more ablation studies.\n\nA4: Thank you for your suggestion. We have added ablation experiments on Llama-7B, and the results are as follows. When the value of alpha increases to 7, the model achieves optimal performance. Further increment of alpha leads to a decline in model performance.\n|    alpha    | ratio | wikitext2$\\downarrow$ |  PTB$\\downarrow$  |   Avg.$\\uparrow$ |\n|-------------|-------|-----------|-------|-----------------|\n|     1       |  50%  |    54.17  | 96.57 | 44.42           |\n|     4       |  50%  |    43.36  | 83.91 | 44.95           |\n|     7       |  50%  |    37.89  | 67.68 | 50.10           |\n|     10      |  50%  |    44.82  | 75.80 | 46.86           |", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical results in this paper.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths:\n\n1. SlimLLM is a simple yet highly practical structured pruning method. It requires no additional training and can be easily applied to various large language models (LLMs), making it a scalable solution for real-world deployment.\n2. The proposed fast recovery method achieves performance levels comparable to low-cost fine-tuning. For example, at a 50% pruning rate, the LoRA-based LLM-Pruner attains a perplexity (PPL) of 38.12 on WikiText, while SlimLLM achieves a slightly lower PPL of 37.89, demonstrating its effectiveness in maintaining model accuracy despite significant compression.\n3. Unlike uniform pruning approaches, SlimLLM incorporates a non-uniform pruning ratio, dynamically adjusting layerwise pruning based on cosine similarity between input and output representations. \n\n## Weaknesses:\n\n1. Although the proposed sub-methods have been proven effective, the relation between FFN pruning and attention layers pruning is no so clear. Are they independent methods?\n2. The paper mainly focuses on LLaMA models. It is encouraged to conduct more experiments on other models.\n3. The performance gain appears to be not so significant at a 20% pruning ratio, as shown in Table 5. This suggests that SlimLLM might be more beneficial for higher compression levels but less impactful for moderate pruning. A deeper investigation into why performance varies across different pruning ratios would be valuable.\n4. The layer-wise pruning ratio relies on empirical hyperparameter α, lacking theoretical justification and more ablation studies.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fnrki0CZp0", "reviewer": "Reviewer_5cF2", "summary": "This paper proposes SlimLLM for pruning large language models (LLMs). The method uniquely combines Pearson correlation analysis for attention head redundancy detection with PCA-based directional importance for FFN channel pruning. A lightweight linear calibration technique minimizes post-pruning performance degradation, while layer-specific sparsity allocation leverages input-output alignment. Evaluations on LLaMA-7B and LLaMA2-7B demonstrate significant efficiency gains while maintaining competitive accuracy on commonsense reasoning tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Computational complexity of iterative head pruning (Algorithm 1) is unaddressed, raising concerns for larger models.\n\nA1: The computational complexity of Algorithm 1 is acceptable. First, the number of heads is generally small, which limits the number of iterations in the algorithm. Second, the outputs of each head can be precomputed and each iteration only involves the calculation of similarity, which reduces the number of redundant calculations during the iterations. For each layer, the time spent on greedy search is approximately 2 seconds when pruning Llama-7B.\n\nQ2: Linear regression is applied only to output matrices. Is it possible for extending it to intermediate layers for further reducing accuracy loss.\n\nA2: We compensate for the pruning error by performing linear fitting between the outputs before and after pruning. This requires that the outputs of the two stages maintain the same dimensionality. Currently, the output matrices of MHA and FFN meet this requirement. However, for other linear layers, since the output channels have been pruned, this method cannot be directly applied at present.", "claims_and_evidence": "The authors claim SlimLLM achieves state-of-the-art structured pruning for LLMs. This is substantiated by Table 1, where SlimLLM outperforms LoRAP by 2.85% average accuracy at 50% pruning without fine-tuning, and Table 3, which highlights a 3.4× speedup on LLaMA-7B.", "methods_and_evaluation_criteria": "The framework is evaluated on various tasks (BoolQ, PIQA etc.) using LLaMA-family models. Latency measurements on NVIDIA V100 GPUs confirm practical applicability.", "theoretical_claims": "None explicitly stated.", "experimental_designs_or_analyses": "This paper provides extensive experimental results and ablation studies.", "relation_to_broader_scientific_literature": "LLM pruning is a popular topic and this paper proposes a novel method that achieves strong performance.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "**Strengths**\n-  PCA-based feature importance for FFN channels addresses a critical limitation of magnitude-only pruning, preserving directional information.\n- The proposed method achieves great improvement for running speed, making it viable for real-time applications.\n- The linear regression strategy is an interesting idea and recovers performance with negligible computational overhead.\n**Weaknesses**\n1. Computational complexity of iterative head pruning (Algorithm 1) is unaddressed, raising concerns for larger models.\n2. Linear regression is applied only to output matrices. Is it possible for extending it to intermediate layers for further reducing accuracy loss.", "other_comments_or_suggestions": "See weakness part.", "questions_for_authors": "How does the computational cost of greedy search scale with model size?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WVBz17cpcG", "reviewer": "Reviewer_2F2C", "summary": "This paper proposes SlimLLM, a structured pruning method for LLMs that evaluates redundancy via Pearson similarity-driven head pruning and PCA-guided FFN channel pruning. A lightweight linear recalibration reduces post-pruning accuracy loss, while dynamic layer sparsity optimizes resource allocation. Experiments show strong performance on LLaMA models, surpassing prior methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: This paper should be carefully proofread. For example, what is $O_{-h_{i}}$ in Figure 1? $S_{-p}$ is not easy to understand in Algorithm 1 and should be replaced by better equation format.\n\nA1: Thank you for your suggestion. $O_{-h_{i}}$ denotes the output excluding the $i-th$ head. $S_{-p}$ represents the set of unpruned attention head. $S_{-p}$={$head_{i}$, $i \\in$ unpruned head index}. The detailed descriptions will be added in the paper.\n\nQ2: What is exactly the latency reported in Table 3? Prefill latency or Decoding latency? More details should be included.\n\nA2: Thank you for your suggestion. We show the prefill latency in Table 3 and we will add details in paper. Additionally, we conducted experiments to measure the decoding latency. The table below shows the latency results for generating 256 tokens with a batch size of 8.\n|    ratio    | #Param   | latency(s) |\n|-------------|----------|------------|\n|    0%       | 6.7B     | 13.12      |\n|    20%      | 5.4B     | 11.48      |\n|    50%      | 3.4B     | 9.38       |\n\nQ3: The proposed method is evaluated for LLMs. Is it possible that this method could also prune vision-language models?\n\nA3: Yes, it is. For VLM (Visual-Language Model) models, the pruning strategy needs to be adapted, such as the proportion of visual tokens to text tokens. We are currently conducting preliminary experiments on Qwen-VL-7B, and the experimental results are shown in the table. We will present more details in our future work.\n|    Datasets         |  prune ratio |  MME    | MMBench_dev_en |\n|---------------------|--------------|---------|----------------|\n|   Qwen-VL-7B        | 0%           |2292.88  |  77.9          |\n| Qwen-VL-7B w/o tune | 20%          |2047.691 |  74.9          |", "claims_and_evidence": "The claims made in the paper are well-supported.", "methods_and_evaluation_criteria": "The evaluation criteria and datasets are adequate for this task.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental designs and analyses appear to be solid. The proposed method is evaluated on various benchmarks and extensive ablation studies are provided.", "relation_to_broader_scientific_literature": "SlimLLM provides a novel and accurate method for pruning LLMs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n\n1. Eigenvector-guided PCA preserves critical feature directions in FFN layers, surpassing magnitude-only criteria.\n2. The proposed method demonstrates stronger performance than prior methods like LoRAP and LLM-Pruner.\n3. The linear regression strategy shows great impact for performance recovery, as demonstrated in Table 5.\n\n**Weaknesses**\n\n1.This paper should be carefully proofread. For example, what is $o_{-hi}$ in Figure 1? $S_{-p}$ is not easy to understand in Algorithm 1 and should be replaced by better equation format.\n\n2.What is exactly the latency reported in Table 3? Prefill latency or Decoding latency? More details should be included.\n\n3.The proposed method is evaluated for LLMs. Is it possible that this method could also prune vision-language models?", "other_comments_or_suggestions": "For LLaMA2-7B, the proposed method obtains slightly poorer performance than prior methods on pruning ratio of 50% with finetuning. More analysis and discussions are encouraged.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HxPb2LlTX1", "reviewer": "Reviewer_JstE", "summary": "This work involves compressing LLMs by width pruning of sublayers in transformer blocks, where MHA and FFN are treated differently. For MHA, head pruning is performed based on the similarity between outputs with and without specific heads. For FFN, channel pruning is carried out using a Wanda-based metric from LoRAP, which additionally considers feature direction along with feature magnitude. Performance recovery is achieved through a linear regression method, and a non-uniform pruning ratio is applied across different blocks. The experiments are exclusively conducted on the LLaMA family.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Proposed metric for MHA\n\nA1: We design similarity-based approach to enhance linear fitting by maximizing output similarity. To verify the effectiveness of our similarity-based approach, we employed the fluctuation metric from FLAP to evaluate the importance of heads, while aligning other configurations. From the results, It can be observed that our similarity-based strategy achieves a lower PPL.\n|    method              | ratio | wikitext2$\\downarrow$ |  PTB$\\downarrow$  |   Avg.$\\uparrow$  |\n|------------------------|-------|-----------|-------|------------------|\n| fluctuation metric     | 50%   | 46.34     | 87.25 |  44.44           |\n| similarity-basd metric | 50%   | 37.89     | 67.68 |  50.10           |\n\nQ2: Computations in Algorithm 1\n\nA2: The computational cost of greedy search is acceptable. The time spent on greedy search is approximately 2 seconds for each layer on Llama-7B.\n\nQ3: Importance metric for the FFN\n\nA3: Yes, there is another reason. When pruning the input channels of the down projection, each channel's input activation only affects the magnitude of its output vector, while the direction is determined by the weights. For up and gate projection, their outputs correspond to the input activations of the down projection, and thus, we have not considered their impact on the direction of the final output matrix.\n\nQ4: Linear regression\n\nA4:  A and B can be integrated into the output projection. Here, the weight matrix of the output projection is $A \\cdot W_{o}$, and the bias term is $B + B_{o}$.\n\nQ5: r_0​ and alpha\n\nA5: Under the pruning ratio $r$ r_0 is calculated according to the following formula:\n$ r_{0} = r * $ pruned_layers / total_layers \n\nFor alpha, we provide results for various values. Other models can use these results for adjustment.\n|    alpha    | ratio | wikitext2 |  PTB  |   Avg.$\\uparrow$  |\n|-------------|-------|-----------|-------|------------------|\n|     1       |  50%  |    54.17  | 96.57 | 44.42            |\n|     4       |  50%  |    43.36  | 83.91 | 44.95            |\n|     7       |  50%  |    37.89  | 67.68 | 50.10            |\n|     10      |  50%  |    44.82  | 75.80 | 46.86            |\n\nQ6: Performance\n\nA6: The experimental results show that the advantages of our model generally decrease after fine-tuning. This is mainly because the fine-tuning strategies of the two methods cannot be fully aligned. During compression, LORAP uses low-rank decomposition for MHA's linear layers, while LORA fine-tuning does not. LORAP reconstructs the decomposed matrices into a single matrix to keep MHA's parameter count unchanged. This increases the parameter count during LORA training. \n\nQ7: Calibration samples\n\nA7: We have added experiments to investigate the impact of the calibration set size on model performance. \n|    alpha    | ratio | wikitext2 |  PTB  |   Avg.$\\uparrow$  |\n|-------------|-------|-----------|-------|------------------|\n|     1       |  50%  |    58.8   | 129.95| 46.50            |\n|     4       |  50%  |    45.89  | 99.64 | 49.14            |\n|     8       |  50%  |    45.98  | 99.25 | 47.89            |\n|     16      |  50%  |    44.3   | 80.06 | 45.06            |\n|     32      |  50%  |    37.89  | 67.68 | 50.10            |\n|     64      |  50%  |    39.56  | 72.05 | 49.31            |\n\n\nQ8: Experiments\n\nA8: To demonstrate the generalizability of our strategy, we have included experimental results on the Vicuna-7B.\n|      Methods       | ratio | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|--------------------|-------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| Vicuna-7B          |  0%   | 75.69 | 77.91 | 71.04     | 67.80      | 68.98 | 40.7 | 42.20 | 63.47   |\n| SLimLLM w/o tune   | 20%   | 74.92 | 76.12 | 67.98     | 65.82      | 67.09 | 39.33 | 42.60 | 61.98   |\n| SLimLLM w/ tune    | 20%   | 76.15 | 76.39 | 69.32     | 64.72      | 68.56 | 39.25 | 41.80 | 62.31   |\n\nQ9: 13B model\n\nA9: Thank you for your suggestion. We have added experiments on LLaMA-13B.\n|      Methods       | ratio | BoolQ | PIQA  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA  | Average |\n|--------------------|-------|-------|-------|-----------|------------|-------|-------|-------|---------|\n| Llama-13B          |  0%   | 68.47 | 78.89 | 76.24     | 70.09      | 74.58 | 44.54 | 42.00 | 64.97   |\n| LoRAP w/o tune     | 20%   | 73.94 | 77.31 | 74.93     | 69.69      | 70.79 | 40.44 | 41.40 | 64.07   |\n| SlimLLM w/o tune   | 20%   | 74.13 | 77.53 | 74.73     | 69.30      | 70.45 | 42.32 | 41.00 | 64.21   |\n\nQ10: Latency\n\nA10: The framwork is Vanilla HuggingFace. To test decoding latency, we set the batch size to 8 and measured the latency for generating 256 tokens. The results are as follows: \n|    ratio    | latency(s) |\n|-------------|------------|\n|    0%       | 13.12      |\n|    20%      | 11.48      |\n|    50%      | 9.38       |\n\nOthers: Thank you for comments on writing, we will revise it carefully.", "claims_and_evidence": "This work is largely supported by empirical evidence, and the approach to pruning MHA and FFN differently seems interesting. Although additional hyperparameters are introduced, the type of performance recovery employed is novel to me.\n\nHowever, some aspects of FFN pruning remain unclear, and heuristics are used to determine non-uniform pruning ratios. The paper introduces one method for MHA pruning (A), another for FFN pruning (B), and a third for performance recovery (C). While (A) and (C) appear promising and interesting, the rationale and effectiveness of (A), (B), and (C) independently are not well justified. Additionally, several necessary ablation studies are missing.", "methods_and_evaluation_criteria": "The proposed framework aligns well with previous studies on LLM pruning. Given the emphasis on LLM compression in this work, the use of evaluation metrics seems adequate.", "theoretical_claims": "While the paper does not make theoretical claims, it does provide clear descriptions of the equations used in the method.", "experimental_designs_or_analyses": "* Measuring the impact of per-head removal as shown in Equation (4) is intriguing, yet its effectiveness is still in question. Regarding MHA pruning, is it more effective than gradient-based importance measures like those used in LLM-Pruner, AWSVD in LoRAP, or the fluctuation metric in FLAP, when all other conditions are identical? Although Tables 1 and 2 offer comparisons with several methods, the inclusion of other factors, such as FFN pruning and non-uniform ratios, makes it challenging to clearly evaluate the effectiveness of the proposed metric for MHA.\n  - [FLAP] https://arxiv.org/abs/2312.11983\n\n* Furthermore, the computations in Algorithm 1 appear to be intensive due to its greedy search nature. How long do these computations typically take, and is it possible to provide comments on this aspect?\n\n* I think the pruning importance metric for the FFN is quite similar to LoRAP, except it employs a sigmoid function for the down projection. Why is the down projection designed in this particular way while the gate and up projection do not follow the same design? Is it impossible to apply Eqn (7) to the gate and up projection matrices, or is there another reason?\n\n* Linear regression for performance recovery seems effective; however, are A and B in Eqn (10) additional, newly introduced parameters? It appears that A and B are necessary for the inference process. If this is the case, it may not integrate well with other inference engines, such as vLLM and TensorRT-LLM, because it requires modifications to the forward code and involves saving additional parameters. This could limit the practicality of the work.\n\n* How did the authors set r_0​ and alpha in Eqn (11)? The statement, 'Consequently, we employ a stratified pruning strategy, wherein the pruning ratios are systematically lower in the shallower layers and incrementally higher in the deeper layers,' (line 241 of page 5) seems entirely heuristic and lacks a systematic foundation. Furthermore, the generalizability to other LLMs beyond the LLaMA family is not guaranteed.\n\n* For the results for Llama-2 in Table 2, I am not convinced that SlimLLM outperforms LoRAP. The numerical results, such as WikiText2 PPL and downstream accuracy, appear very similar, and in some cases, LoRAP seems to perform better.\n\n* The number and type of calibration samples have not been investigated.\n\n* The experimental validation in this study seems restricted since only models from the LLaMA family are used. To better ascertain the widespread applicability and superiority of this method, I suggest conducting experimental comparisons with additional models such as OPT, Qwen, Phi, and MoE-based architectures.\n\n* The tests were limited to models with only 7B parameters. Including models with at least 13B parameters, as done in previous studies, is essential to provide a comprehensive assessment.\n\n* The methodology used to calculate latency gains in Table 3 needs further elaboration, especially since width pruning often faces challenges in achieving actual speedups in setups like ZipLM and Shortened LLaMA. Could you clarify which framework was used (e.g., Vanilla HuggingFace, TensorRT-LLM, vLLM)? It's also important to know if the method can accelerate both the prefill and decoding stages, as the current results seem to focus solely on prefill with single-token generation. Is a batch size of 1 appropriate?", "relation_to_broader_scientific_literature": "Reducing computational requirements for LLMs is widely discussed, and this work tackles it in a clear and effective manner.", "essential_references_not_discussed": "* The discussion lacks mention of certain pruning studies such as width pruning techniques like FLAP and Minitron, and depth pruning methods including Shortened LLaMA, SLEB, and Minitron. It seems essential to compare this work with FLAP, given its significant benefit of eliminating the need for retraining, and because analyzing the differences between original and pruned features is relevant.\n  - [FLAP] https://arxiv.org/abs/2312.11983\n  - [Minitron] https://arxiv.org/abs/2408.11796\n  - [Shortened LLaMA] https://arxiv.org/abs/2402.02834\n  - [SLEB] https://arxiv.org/abs/2402.09025", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "* I am uncertain about the relevance of the quantization and low-rank approximation methods discussed in the 'Compression technologies' part of Sect 2. Related Work to this study, as the authors do not test the compatibility of these methods with the proposed pruning technique. Additionally, there appears to be no citation for the studies mentioned in this subsection. \n* Figure 3, which illustrates the impact of each layer on LLMs, presents a concept that is already familiar in this field, as evidenced by similar graphs in ShortGPT, SLEB, and Shortened LLaMA. This raises questions about the value this graph adds to the paper. Furthermore, relevant citations are missing in line 274 on page 5, where the paper states, 'Similar to many current methods, we...'.", "questions_for_authors": "Please refer to the sections on <Claims And Evidence> and <Experimental Designs Or Analyses>.\n\nI like several aspects of this work and appreciate the authors’ efforts in compressing LLMs. However, several justifications and ablation studies appear to be missing, and the experimental validation seems weak compared to relevant papers.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper puts forward SlimLLM, a structured pruning approach for large language models (LLMs). It assesses redundancy through Pearson similarity-based head pruning and PCA-directed feed-forward network (FFN) channel pruning. A lightweight linear recalibration is used to decrease the accuracy loss after pruning, and dynamic layer sparsity helps optimize resource distribution. Experiments carried out on LLaMA models demonstrate that SlimLLM performs well and outperforms previous pruning methods.

All reviewers recommend acceptance of this work. All the questions are solved during the discussion phase.

Thus, the AC recommends acceptance of this work as poster.
Decision: Accept (poster)