DLP: Dynamic Layerwise Pruning in Large Language Models
Authors: Yuli Chen
Bo Cheng
Jiale Han
Yingying Zhang
Yingting Li
Shuhao Zhang
Venue: ICML 2025 poster (2025)
Keywords: Pruning
Large Language Models
Model Compression
Primary area: deep_learning->large_language_models
TL;DR: The importance of different layers is automatically determined by calculating weight and activation information, ensuring the model's performance is maintained under high-sparsity pruning.

Abstract: Pruning has recently been widely adopted to reduce the parameter scale and improve the inference efficiency of Large Language Models (LLMs). Mainstream pruning techniques often rely on uniform layerwise pruning strategies, which can lead to severe performance degradation at high sparsity levels. Recognizing the varying contributions of different layers in LLMs, recent studies have shifted their focus toward non-uniform layerwise pruning. However, these approaches often rely on pre-defined values, which can result in suboptimal performance. To overcome these limitations, we propose a novel method called Dynamic Layerwise Pruning (DLP). This approach adaptively determines the relative importance of each layer by integrating model weights with input activation information, assigning pruning rates accordingly. Experimental results show that DLP effectively preserves model performance at high sparsity levels across multiple LLMs. Specifically, at   70% sparsity, DLP reduces the perplexity of LLaMA2-7B by 7.79 and improves the average accuracy by 2.7% compared to state-of-the-art methods. Moreover, DLP is compatible with various existing LLM compression techniques and can be seamlessly integrated into Parameter-Efficient Fine-Tuning (PEFT). We release the code\footnote{The code is available at: \url{https://github.com/ironartisan/DLP}.} to facilitate future research.

DLP: Dynamic Layerwise Pruning in Large Language Models
Yuli Chen
Bo Cheng' Jiale Han 2 Yingying Zhang'
Yingting Li'
Shuhao Zhang'
Abstract
1. Introduction
Pruning (Jaiswal et al., 2023; Ma et al., 2023; Sun et al., 2024; Muralidharan et al., 2024; Cai et al., 2024; Men et al., 2024) has garnered significant attention in both academia and industry due to its ability to substantially reduce the parameter count of Large Language Models (LLMs) (Ope- nAI, 2023; Touvron et al., 2023a;b; Dubey et al., 2024). The core concept of pruning is to optimize resource utiliza- tion by eliminating redundant or less important parameters. SparseGPT (Frantar & Alistarh, 2023) implements a layer- by-layer and row-by-row greedy pruning strategy, ensuring that local optimizations have minimal impact global per- formance. Recent studies (Xiao et al., 2023; Lee et al., 2023; Lin et al., 2024) highlight the pivotal role of outliers in LLMs. Although outliers constitute a small fraction of the model, they exert a disproportionately large influence on predictive accuracy. Building on the emergence of outlier features in LLMs (Puccetti al., 2022; Lee al., 2023; Lin et al., 2024), Wanda (Sun et al., 2024) introduces novel approach to evaluate weight importance by integrating the absolute weight values with the norm of the corresponding input activations.
Pruning has recently been widely adopted re- duce the parameter scale and improve the in- ference efficiency of Large Language Models (LLMs). Mainstream pruning techniques of- ten rely on uniform layerwise pruning strate- gies, which can lead to severe performance degra- dation at high sparsity levels. Recognizing the varying contributions of different layers in LLMs, recent studies have shifted their focus toward non-uniform layerwise pruning. How- ever, these approaches often rely on pre-defined values, which can result in suboptimal perfor- mance. To overcome these limitations, we pro- pose a novel method called Dynamic Layerwise Pruning (DLP). This approach adaptively deter- mines the relative importance of each layer by integrating model weights with input activation information, assigning pruning rates accordingly. Experimental results show that DLP effectively preserves model performance at high sparsity lev- els across multiple LLMs. Specifically, at 70% sparsity, DLP reduces the perplexity of LLaMA2- 7B by 7.79 and improves the average accuracy by 2.7% compared state-of-the-art methods. More- over, DLP is compatible with various existing LLM compression techniques and can be seam- lessly integrated into Garameter-Efficien Fine- Tuning (PEFT). We release the code' to facilitate future research.
Although previous works (Frantar & Alistarh, 2023; Zhang et al., 2024; Sun al., 2024) have achieved satisfactory performance, they fail account for the varying importance of different layers within the model, instead assigning uniform sparsity rate to all layers. This limitation leads to significant performance drop under high sparsity condi- tions. Inspired by the presence of outliers, Outlier Weighed Layerwise Sparsity (OWL) (Yin et al., 2024) introduces novel pruning paradigm that leverages the criticality of layers with a higher proportion of outliers. Based on the principle that layers with higher proportion of outliers more critical, OWL assigns different sparsity rates to each layer of LLMs. In comparison to uniform layerwise pruning (Zhu & Gupta, 2018), OWL demonstrates superior perfor- mance in preserving model accuracy. However, OWL still has certain limitations in practical applications. Its reliance on predefined criteria for outlier selection not only limits its adaptability to the dynamic needs of the model but also hinders the achievement of optimal performance.
'State Key Laboratory of Networking and Switching Tech- nology, Beijing University of Posts and Telecommunications, Beijing, China 'Hong Kong University Science and Tech- nology, Hong Kong, China. Correspondence to: Bo Cheng <chengbo bww..cdu.cc... Jiale Han @jst.kkhanustthk>.. hus..kk>.
Proceedings of the 4rnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
To address the above issue, we compute the unimportance of each layer from an inverse perspective, which is then
'The code is available https://githie com/ ironart rt san/DLP.

---

DLP: Dynamic Layerwise Pruning in Large Language Models
transformed into the relative importance between layers. Based on the principle that layers with higher importance should have lower sparsity, we allocate layerwise sparsity rates, Some previous works (He et al., 2019; Zhang et al., 2023) use the median to identify redundant elements in a model, assuming that central elements can be replaced by other elements from the same layer. We demonstrate the effectiveness of the median in LLMs through three empirical studies. Additionally, we inherently place more emphasis on outliers. Due to the median's 's insensitivity to outliers (Huber et al., 2001), it provides a more accurate reflection of the central tendency of a layer when the weights contain outliers.
Overall, the contributions of our work follows:
We propose a novel method for measuring layerwise importance that does not rely on empirical values or model type. This method comprehensively considers both intra-layer and inter-layer elements automati- cally determine the relative importance of each layer.
We propose an effective method for unstructured prun- ing. Extensive experimental results consistently show that, at high sparsity levels, DLP not only outperforms sfate-of-the thet-rttpp pruning techniques for LLMs but also achieves significant end-to-end speedups on CPU.
In this paper, we propose a novel Dynamic Layerwise Prun- ing (DLP) method. DLP adaptively determines the impor- tance of each layer by combining model weights with input activation information, offering greater flexibility in sparsity allocation. Our goal is to determine the layerwise impor- lance of LLMs, which we first derive by identifying the layerwise unimportance and then applying an inversion op- eration to obtain relative importance. Specifically, we begin by calculating the unimportance of each Transformer block based on the median of model weights and input activation values in the same layer. We then evaluate the relative unim- portance across layers, which leads to the determination of the model's 's relative importance. Finally, pruning rates are assigned to each layer according to the principle that layers with higher importance should have lower sparsity. The pipeline of DLP is illustrated in Figure
Our method can not only be integrated with LLM com- pression techniques but also extended to PEFT.
2. Related Work
2.1. LLM Pruning
LLM Pruning aims improve computational efficiency by reducing redundant parameters while preserving model per- formance as much as possible. This technique is primarily categorized into structured pruning (Ma et al., 2023; An et al., 2024) and unstructured pruning (Frantar & Alistarh, 2023; Dong et al., 2024; Li et al., 2024a). Structured prun- ing removes specific dimensions of parameters, significantly simplifying the model architecture and accelerating the in- ference process. For example, L.MM-Pruner (Ma et al., 2023) uses sensitivity analysis and task-specific requirements to automatically identify and prune substructures with minimal impact on model performance, achieving efficient grouped structural optimization. Unstructured pruning, in contrast, operates at a finer granularity by directly removing indi- vidual weights enhance sparsity. Representative meth- ods include Magnitude (Jaiswal et al., 2023), SparseGPT (Frantar & Alistarh, 2023), and Wanda (Sun et al., 2024), Magnitude evaluates the importance of each weight based on its absolute value. SparseGPT performs layerwise lo- cal pruning and reconstructs losses to prune models to at least 50% sparsity in a single step, and Wanda determines which weights to prune by considering both weights and activations. Our work primarily focuses on unstructured pruning.
We conduct comprehensive experimental evaluations across multiple mainstream LLMs with varying parameter sizes (ranging from 7B to 30B) and architectures (e.g., LLaMA (Touvron et al., 2023a), Vicuna (Chiang et al., 2023), Mistral (Jiang et al., 2023)). The experimental results show that our method consistently outperforms the state-of-the-ard LLM pruming techniques. particularly at high sparsity levels. For instance, 70% sparsity, DLP reduces the perplexity of LLaMA2-7B by 7.79 and improves the average accuracy by 2.7%. When evaluated on the DeepSparse (Kurtic et al., 2023) inference engine, DLP achieves 2.8x-3.7x end-to-end acceleration on CPU with 70% 90% sparsity. Furthermore, we find that a brief period of fine-tuning can restore the LLM a reasonable range after high sparsity pruning.
As general method, our approach can be applied to un- structured pruning, as well as N M sparsity (Sun et al., 2021) and structured pruning (Ma et al., 2023), consistently outperforming layerwise methods. Our method is orthogo- nal to quantization (Dettmers et al., 2022; Xiao et al., 2023; Balanca al., 2024), and it can also be extended to Singular Value Decomposition (SVD) and Parameter-Efficiee Fine- Tuning (PEFT) (Liu et al., 2022a), achieving substantial performance improvements.
2.2. Layerwise Importance for Pruning
Layerwise importance has emerged as a powerful technique for pruning LLMs, enabling significant reductions in model size and computational cost while maintaining even im- proving performance. SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2024) uniformly apply the same spar- sity to every layer of the model. However, these methods do not account for the varying importance of layers within the
2

---

DLP: Dynamic Layerwise Pruning in Large Language Models
loped
Lipat
Laye20
Dusic
Prane Muld
Madd
Calcuraia Blec Dynamic Proning (30
Faaud Madi
Unifom
Figure 1. Illustration of Uniform Layerwise Pruning and Dynamic Layerwise Pruning (DLP): Blue squares represent unprumed weights, while white squares denote pruned weights. In uniform layerwise pruning, the same sparsity ratio is applied every layer. In contrast, DI.P calculates the unimportance each Transformer block compare the relative importance of layers, assigning different sparsity ratios based the principle that layers with higher importance should have lower sparsity.
model, which can result in suboptimal performance. To over- come this limitation, recent studies (Gao et al., 2019; Yin al., 2024) have explored non-uniform layerwise pruning approaches. Frankle & Carbin (2019) propose uniform global threshold for pruning based on overall sparsity. Lee et al. (2021) introduce Layer-Adaptive Magnitude-based Pruning (LAMP), which dynamically determines the spar- sity of each layer by calculating the relative importance of target connections. OWI (Yin et al., 2024) identifies strong correlation between activation outliers in LLMs and their performance. By determining the sparsity ratio for each layer based on the proportion of outliers, OWL effec- tively preserves critical outliers. However, OWL defines outliers based on empirically set thresholds, which vary across models and may lead to suboptimal performance. In contrast, our method uses the median to automatically determine layerwise importance, adjusting the sparsity of each layer while maintaining the global sparsity rate.
selecting the point with the smallest distance. This compu- tation is complex and often approximated through iterative methods. In contrast, our work uses the median measure the importance of layers, thereby reducing computational cost.
3. Methodology
In this section, we introduce our method, Dynamic Lay- erwise Pruning (DLP). We begin by defining the problem to be addressed, followed by discussion of preliminary results. Next, present three empirical studies to validate the proposed method, and finally, we provide detailed description of the algorithm design.
3.1. Problem Definition
A popular solution strategy for model pruning is to decom- pose the task into multiple layerwise subproblems, enabling hierarchical optimization. These subproblems are often formulated as minimizing the lz error. Consider neural network with L layers, where the weight matrix W of shape (Cout, Cin), with E 1,2, L. X is the input activation with shape of (N x L, Cm). where N and L are batch size and sequence dimension, respectively. Specifically, for each layer the goal is to determine target weights W, that achieve a predefined pruning ratio R while minimizing the squared error.
2.3. Median in Pruning
The median a statistical measure representing the middle value of dataset when ordered in ascending descend- ing order. In recent years, median-based approaches have garnered significant attention in the field of model prun- ing. He et al. (2019) propose Filter Pruning via Geometric Median (FPGM), which leverages the geometric median to identify and prune redundant filters. Gkalelis Mezaris (2020) utilize a geometric median-based criterion iden- tify and structurally prune the most redundant LSTM units. Zhang et al. (2023) introduce a pruning method based Learned Representation Median (LRMF), which identifies unimportant filters by calculating the median in the fre- quency domain. Filters with values near the median are replaced with alternative representations, resulting mini- mal impact on overall performance. However, the methods mentioned above all rely the geometric median, which involves calculating the Euclidean distance to all points and
argmin
W'x'
W
(1)
where Wi is the weight of the /-th layer, X is the input of the 1-th layer, and ||-11/ denotes &g norm squared.
3.2. Preliminaries
Inspired by the Optimal Brain Surgeon (Hassibi et al., 1993). SparseGPT (Frantar Alistarh, 2023) quantifies the sen-
3

---

DLP: Dynamic Layerwise Pruning in Large Language Models
sitivity of the weights the model error by means of the diagonal elements of the hessian matrix, with the less sensi- tive weights more suitable for pruning. The pruning metric of i-th layer in SparseGPT is:
model type, the optimal M values differ between LLaMAI- 7B and LLaMA1-13B, Similarly, for models with the same parameter scale, such as LLaMA1-7B and Vicuna-7B, the optimal M values also vary. This phenomenon indicates that the selection of M is influenced by both the model type and its parameter scale.
E = W/Pg/ (xxx++a)) +
2
(2)
Moreover, using fixed value may constrain the sparsity allocation to certain local patterns, overlooking the global importance distribution. To address this, adopt a global perspective, assigning sparsity rates based on the relative importance of each layer. Pruning is carried out following the principle that the more important a layer is, the lower its sparsity rate. To validate our approach, we conduct three empirical studies based on the relative importance distribution.
where 1 represents the square of the absolute value op- eration, À is the damping term, which serves to prevent the values from becoming unstable, is the unit matrix of the 1-th layer, x + AI' denotes the hessian ma- trix of the localized intra-layer reconstruction problem, diag (xx"x' + AT)) +
denotes the diagonal element
of the inverse of hessian matrix, indicating the reconstruc- tion sensitivity of each connection, and j represent the row and column indices of the matrix, respectively.
40.0
Wanda can be seen as a simplified version of SParseGPT, which avoids the complex computation of hessian matrix and reduces the pruning metric formula a first-order approximation form, measuring the importance only by weights and imput features. The score for the current weight of I-th layer is defined by:
Perplexity
30,0
LLMAIIB
Vicarn-7
20.0
3
7
M
1 =
(3)
where |. represents the absolute value operator, ||x]|12 evaluates the lg norm of the j-th feature vector in the input layer 1.
Figure 2. WikiText validation perplexity of LLaMA1-7B, LLaMA1-13B and Vicuna-7B pruned by various M at 70% sparsity using OWL.
SparseGPT and Wanda both the same sparsity rate for each layer and consider the contribution of each layer to be the same. However, these methods are not optimal for effectively allocating layerwise sparsity during the pruning of LLMs. Recently, OWL considers the retention rate of weight outliers and determines non-uniform sparsity rates across different layers based on the Layerwise Outlier Dis- tribution (LOD). LOD [D', D2 ...., Dd.. where D characterizes the outlier distribution of 1-th layer.
3.3. Empirical Study
Relative Importance Distribution (RID). We use RID the basis for assigning layer sparsity rates. RID takes into ac- count both intra-layer and inter-layer element importance within LLMs. For intra-layer analysis, RID compares the ab- solute unimportance of Transformer blocks within the same layer. This is then converted into relative importance across layers, ultimately deriving the RID of the LLM. Specifically, for a given layer we use A! as the metric to evaluate the importance of weights. The unimportance score of I-th layer can be expressed as:
Σ Cou
Σ
C. j=1
T(A > M. A.)
D'
(4)
Cout
where M a constant, typieally set to or 7, Al is the mean of A and II()) is the indicator function, returning I if A!, is larger than M A', else 0.
Cut Oi 8
F (A
(5)
However, M is determined empirically and is highly suscep- tible to variations in model parameters or types, which can result in suboptimal performance. In Figure 2, we investi- gate the performance relationship between model types, pa- rameter scale, and the value of M. Notably, under the same
where F(.) is a specific method, which is used to measure the absolute unimportance of the layer.
To make the importance across different layers comparable, we compute normalized relative unimportance and convert

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table 1. WikiText validation perplexity of pruning metries for LLaMAI-7B at 70% unstructured sparsity. The best performance result indicated in bold.
Table 2. Comparison of OWL. and Ours on LOD and Perplexity with LLaMA1-13B on the WikiText dataset at 70% unstructured sparsity. The best performance result indicated bold.
Method Sum
Mean Median
Max
Var
SD
Layerwise
Method
LOD(%))
Perplexity
Magnitude 3.7e3 3.7e3
3.4c3
2.9e4
4.7e5
2.5e5
Sparsity
SparseGPT 18.24
18.24
17.76
38.57
21.33 21.42
Dense
5.43
5.09
Wanda 21.03
21.03
20.40
931.89 43.31 38.03
Uniform
60.03
84511.48
Magnitude
OWL
64.70
18992.87
it into relative importance. For the [-th layer, the importance
Ours
77.58
7642.99
score is
Uniform
47.70
18.93
SparseGPT
OWL
51.97
14.02
Ours
64.46
12.63
St
1'
(6)
Uniform
55.14
56.26
EL= St
Wanda
OWI. 56.30
16.23
Ours
70.06
13.65
The importance scores of all layers constitute the RID, that is, RID .r........ I'.. Layers with lower importance have less impact on model performance, so they should be assigned higher sparsity.
higher redundancy are considered less influential to the overall model and are assigned higher sparsity rate during pruning. Moreover, the overall trend of increasing sparsity suggests that earlier layers are considered more important, likely because they play a fundamental role in capturing low-level and generalizable features. In contrast, deeper layers tend to focus on more specialized or task-specific information, which may be more redundant or more tolerant to pruning (Fan et al., 2024). This further highlights the necessity of non- non-uniform layerwise sparsity.
Empirical Study I: Evaluation of Unimportance Metries. FPGM (He et al., 2019) selects elements closest to the geo- metric median within a given layer for pruning, assuming that these elements are redundant and can be effectively represented by other elements in the same layer. To bet- ter evaluate the performance of pruning, we evaluate sev- eral common approaches, including Sum, Mean, Maximum (Max), Standard Deviation (SD) and Variance (Var). As pre- sented in Table 1, the median method performs better than other methods. This shows its effectiveness calculating layer unimportance. The median is more robust compared to other methods. is less influenced by outliers. This allows it better capture the performance of most layers. As re- sult, it calculates relative layer importance more accurately. Therefore, we choose F F-)) as the median.
Empirical Study III: Comparison between OWL and Ours. OWL aligns the sparsity ratio with the outlier ratio in each layer to preserve outliers. It also defines LOD as the ratio of the number of outlier weights to the total number of weights, including both zero and non-zero weights. We prune the LLaMA1-13B model using uniform layerwise pruning, OWL, and Ours, and compare the LOD after prun- ing. Following OWL's setup, We set M to 7. As shown in Table 2, our method achieves the highest LOD and the lowest perplexity. These results indicate that the proposed method outperforms OWL. Compared to uniform layerwise pruning, OWL and our method increase the proportion of outliers. Our method preserves outliers effectively even at high sparsity rates, maintaining better performance.
Empirical Study II: Relationship between Dense LLMs and Ours. To investigate whether the proposed method can achieve non-uniform layer sparsity for dense LLMs, we use RID to measure the differences between layers in the LLMs. If RID is highly balanced, it indicates that our method is not suitable for evaluating inter-layer importance. As shown in the bar chart in the background of Figure 3, the results show that not all layers contribute equally to the model's performance. Surprisingly, this finding aligns closely with recent studies (Li et al., 2024c: Sun et al., 2025; Gromov al., 2024), which show that deeper layers do not function as effectively as expected. Since the median is insensitive to extreme values, it better captures the central tendency. Elements near the center are easily represented by their neighboring elements, making their removal less detrimen- tal to performance. A lower median within a layer suggests minimal redundancy in its weights, whereas a higher me- dian implies greater redundancy. Consequently, layers with
3.4. Dynamic Layerwise Pruning (DLPP
Although we obtain the RID, its scale still influenced by the sparsity level. Therefore, it is necessary to further ex- plore the relationship between sparsity levels and the scale of importance. Following the principle that layers with higher importance should have lower sparsity, we introduce hyperparameter a to adjust this relationship. To mitigate the risk of severe performance degradation due to excessive pruming in a specific layer, we compress the range of im- portance into [0, 20.. Consequently, the sparsity of each
5

---

DLP: Dynamic Layerwise Pruning in Large Language Models
layer varies between [R a, R aJ, with an overall aver- age sparsity of R. The pseudocode of DLP is provided in Algorithm 1.
and zero-shot performance of sparse LLMs. Specifically, we measure language modeling performance using the per- plexity metric on the WikiText (Merity et al., 2017), PTB (Marcus et al., 1994), and C4 (Raffel et al., 2020) valida- tion datasets. For zero-shot evaluation, we assess accuracy on seven commonsense benchmarks from EleutherAl LM Harness (Gao et al., 2024), including BoolQ (Clark et al., 2019), RTE (Wang al., 2019). HellaSwag (Zellers et al.. 2019), WinoGrande (Sakaguchi al., 2020), ARC Easy and Challenge (Boratko et al., 2018), and OpenBookQA (Mihaylov et al., 2018).
Algorithm 1 Pseudocode of DLP
I: Input: Weight W and input X
2: Input: Deflation scale and sparsity rates p
3: Output: The dynamic pruning sparsity R of each layer
4: Obtain the score A via (3)(3)
5: Obtain the unimportance score St via (5)
6: Obtain the importance score I' via (6)
7: Store the scaled importance scores in d
8: for + 0 to length([)- do
Baselines. We apply DLP to three LLM pruning methods: Magnitude (Jaiswal et al., 2023), SparseGPT (Frantar & Al- istarh, 2023), and Wanda (Sun et al., 2024). Magnitude is a simple yet effective baseline that retains weights with larger absolute values to maintain model performance, SparseGPT and Wanda are two strong LLM pruning baselines capable of preserving reasonable performance under 50% sparsity. Our focus is primarily on high sparsity levels, no less than 50%. All three baselines adopt uniform layerwise sparsity rates. Additionally, we compare DLP with OWL to validate the performance of the proposed method. Furthermore, we evaluate it against other layerwise pruning methods, includ- ing Global (Frankle & Carbin, 2019), ER (Mocanu et al., 201 8), ER-Plus (Liu al., 2022b), and LAMP (Lee et al., 2021).
9: d2 + 1.- Immn x x22x C
10: end for
11: Calculate the average values m of d
12: for 0 length(d)- do d3
13: R P
14: end for
15: Obtain the final layerwise sparsity rates R
16: Return R
We compare the layerwise sparsity rates of OWL and DLP on LLaMA1 (7B/13B/30B) models. As shown in Figure 3, OWL and DLP exhibit similar overall trends. Notably, on the LLLMAA-300 model, DLP shows more significant fluc- tuations layer sparsity rates. This indicates greater differ- ences in relative importance between layers, resulting in more fine-grained sparsity distribution across the model.
Implementation Details. In experimental setup, utilize four NVIDIA A40 GPUs, each with 48 GB of mem- ory. To ensure a fair comparison, we follow OWL (Yin et al., 2024) by randomly selecting 128 samples from the C4 dataset, with each sample containing 2048 tokens as cali- bration data. In Appendix G, we present the hyperparameter configurations for various sparsity levels.
In addition, we investigate the relationship between pruning granularity and model performance Appendix C. When al- locating pruning rates, we choose to allocate them per layer rather than per Transformer block, as the former approach provides better performance. Furthermore, we also compare the performance of per-output pruning and per-layer prun- ing in Appendix D. During the actual pruning process, we perform pruning based on per output rather than per layer.
4.2. Main results
Language Modeling. We report the performance of var- ious LLM pruming methods on language modeling with WikiText dataset, as presented in Table 3. Additionally, we provide performance results for different sparsity levels, with details available in Appendix B. The results validate the effectiveness of DLP's layerwise sparsity strategy. When used in combination with other unstructured pruning meth- ods, DLP (ours) consistently achieves the lowest perplexity values across all model sizes, outperforming Uniform and OWL. For example, when the sparsity rate is 70%, DLP combined with Magnitude pruning on the LLaMA2- 13B model has a Perplexity of 52.41, which is significantly better than the baseline based on uniform layerwise pruning of 114.19. In addition, when combined with the SparseGPT and Wanda methods, DLP still achieves a lower Perplexity than OWL. When the sparsity 70%, SparseGPT reduces
4. Experiments
4.1. Experimental Setup
Models and Datasets. We evaluate the performance of DLP on various LLMs, including LLaMA] (7B/13B/30B)
(Touvron et al., 2023a), LLaMA2 (7B/13B) (Touvron et al.,
2023b), and other more advanced I.LMs such as LLLMM33
8B (Dubey et al., 2024), LLaMA3.1-8D (Ling et al., 2024), Vicuna-7B (Chiang et al., 2023), Mistral-7B (Jiang et al., 2023), and Qwen-7B (Bai et al., 2023). These models are available from the HuggingFace Transformers library². Ad- ditionally, we evaluate the language modeling capabilities
6

---

DLP: Dynamic Layerwise Pruning in Large Language Models
LL
(a) LLAMAA-308
1.00
0,90
1,000
1,000
GW1
[1A4)
C7NT. OuIS
OsrS
0.85
0.995
-085
0.99
Unifmm
Unikerr
Unilam
0.3
-0.0
0.955
Selage Disaltius
Aelalie (xpurtese Diesbbatess
0,990
0.98
-0.75
Lave sis-itt Rasio
Laveviie Spasitt
0.985
DA
0.990
0.91
1.200
0,000
-0.3 TV
0.00
0,975
-0.65
0.985
0.95
0.970
00.00
0.5
0.35
0.965
-0.55
0.34
30
12 Loyer Inles
25
0,50
0.960
20 Layer Itokx
3.30
0.915
23
30
50
0.4
Layer Irdox
Figure 3. Comparison of layerwise sparsity distributions between Ours (red) and OWL (orange). The bar chart in the background represents the Relative Importance Distribution (RID). In each subplot, the horizontal axis represents the layer index, the left vertical axis corresponds to the RID, and the right vertical axis corresponds to the layerwise sparsity ratio.
Table Perplexity results on WikiText. We produce the Uniform, OWL and DLP(Ours) with 70% unstructured sparsity on LLaMA1, LLaMA2 models. The best performance result is indicated in bold.
Table 4. Comparison of mean zero-shot accuracies (%) for pruned LLaMAI and LLaMA2 models at 70% unstructured sparsity. The best performance result is indicated in bold.
Layerwise Sparsity
LLaMA1
LLaMA2
Layerwise
LLaMA1
LLaMA2
Method
Method
7B
13B
30B
7B
13B
Sparsity
7B
13B
30B
7B
13B
Dense
5.68
5.09
4.10
5.47
4.88
Dense
64.33 66.78 69.72
64.42
67.04
Uniform
4.9c4 8.5e4
9.7c2
5.0e4
2.1c2
Uniform
34.80 37.09 35.06
35.65
36.23
Magnitude
OWL
2.0e4 1.9e4 2.4e2
1.5e4
57.55
Magnitude
OWL
36.40 39.45 35.73
36.44
40.67
Ours
3.4e3 7.6e3 98.05
8.7e3 52.41
Ours
38.21 40.11 42.03
40.84 44.02
Uniform
25.38 18.93 12.87 27.84 19.38
Uniform
45.32
48.34 55.87
44.72 47.99
SparseGPT
OWL
19.95 14.02 10.22 19.71 15.12
SparseGPT
OWL
47.84
50.78 56.82
48.02 51.70
Ours
17.76 12.63 9.43
18.58 13.30
Ours
48.32 53.06 57.84
49.65 53.47
Uniform
86.38 56.26 17.54
76.84 45.76
Uniform
39.91
41.62 54.59
37.04 40.44
Wanda
OWL
24.46 16.23 10.77
30.58
20.65
Wanda
OWL
46.32 49.59 55.93
43.55
48.11
Ours
20.46 13.65 9.93
22.79 16.19
Ours
48.62 52.03 56.83
46.25 51.11
it by 2.19 ILaMA1-7B, and Wanda reduces it by 7.79 on LLaMA2-7B.
to LLaMA2-7B-chat-ht (Touvron et al., 2023b) for pruning using Wanda, and test its end-to-end decoding latency using the DeepSparse (Kurtic et al., 2023) inference engine on Intel(R) Xeon(R) Gold 6248R CPU equipped with 24 cores, and the results are shown in Table 5. The model pruned by DLP achieves significant inference speedup compared to the dense model. is worth noting that the speedup ratio increases with sparsity and 3.5x when the sparsity 80%.
Zero-Shot Tasks. In Table 4, we present the average zero- shot accuracy of the pruned LLaMA1 and LLaMA2 models across seven zero-shot tasks. The performance for each specific task is provided in Appendix I. Notably, DLP con- sistently improves accuracy across all settings. For example, for the LLaMA2-13B model, DLP achieves average accu- racy improvements of 7.79, 5.48, and 10.67 over Magnitude, Wanda, and SparseGPT, respectively. Compared to OWL, DLP improves the average accuracy by 3.35, 1.77, and 3.00. These results clearly demonstrate the potential of DLP in tackling more challenging zero-shot downstream tasks.
Pruning Efficiency. To evaluate the computational com- plexity of our method, we compare the empirical pruning speed with baselines. Specifically, since non- unifurm layer sparsity can be pre-computed, we ignore the forward propa- gation and non non-uniform sparsity calculation processes, fo- cusing primarily on comparing the cumulative time spent on calculating pruning metrics for each layer between non- uniform and uniform layer pruning methods. The results
Inference Speedup. To verify the acceleration effect of sparse L.LM after pruning by our method, we apply DLP
7

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table End-to-end decoding lateney and throughput of I.aaMA277--cha-- DeepSparse inference engine using DL.P.
Sparsity
Dense
10
20%
30%
40%
50%
60%
70%
80%
90%
Latency (ms)
353.42
352.40
338.37
323.49
273.43
200.17
164.31
124.76
100.30
96.86
Throughput (tokens/sec)
2.83
2.84
2.96
3.09
3.66
4.99
6.08
8.01
9.97
10.32
Speedup
1.0x
1.0x
1.1x
1.1x
1.3x
1.8x
2.2x
2.8x
3.5x
3.7x
Table 6. Comparison of time overhead used for computing the pruning metric across layers of LLaMAI-7B (in seconds).
Table 8. WikiText validation perplexity of various LLMs pruned by Uniform and Ours using Wanda. The best performance result indicated in bold.
Layerwise
LLaMA1
Method
Sparsity
7B
13B
30B
Model
Method
60 %
70%
80%
Magnitude
Uniform
Ours
1.66
1.68
6.65
6.82
11.04
11.01
LLaMA3-8B
Uniform
Ours
23.50
19.21
122.96
96.31
687.11
676.16
SparseGPT
Uniform
254.32
511.30
1052.48
LLaMA3.1-8B
Uniform
21.84
118.18
1031.36
Ours
257.48
487.44
1051.55
Ours 18.58
84.30
786.19
Wanda
Uniform
Ours
0.95
0.98
5.30
5.50
8.50
8.64
Vicuna-7B
Uniform
Ours 11.49
12.89
60.60
28.79
1613.15
345.10
Uniform
11.28
60.62
331.04
Mistral-7B
Ours 9.91
29.83
199.19
Table 7. WikiText validation perplexity of various LLMs pruned by Ours with LoRA fine-tuning.
Qwen-7B
Uniform
14.76
91.99
23136.98
Ours
14.38
54.13
1122.54
Model
Method
Sparsity
Perplexity
4.3. More Corroborating Results of DLP
LLaMA1-P7 1-7B
Without FT
0.7
17.76
LLaMA1-27 1-7B
With FT
0.7
12.15
Comparison among Various Layerwise Sparsity Meth- ods. To evaluate the superiority of the DLP method, we also compare the performance of DLP with other methods in terms of assigning layerwise sparsity in Appendix A. When the sparsity exceeds 40%, DLP consistently outperforms other layerwise sparsity methods. Notably, at sparsity rate of 80%, the perplexity of DLP decreases by 56% compared to OWL.
LLaMA1-13B
Without FT
0.7
12.63
LLaMAI-13B
With FT
0.7
10.05
are shown in Table 6. In the case of uniform layer pruning, Wanda exhibits the lowest overhead compared to SparseGPT and Magnitude. As the number of model parameters in- creases, the efficiency of our method improves, with the time spent being lower than that of uniform layer pruning. This may be because method aligns better with the model distribution, enabling faster identification of pruning targets.
Performance on More Advanced LLMs. To evaluate the applicability of DLP, we also assess its performance on more advanced LLMs, with the results presented in Table 8. Notably, as a general method, DLP is applicable to LLMs with different architectures at higher sparsity rates and con- sistently outperforms uniform layerwise pruning methods. The experimental results further confirm the effectiveness of DLP.
Fine-Tuning Performance. In Table 7, we present the performance results after fine-tuning the model pruned with DLP. In order to expedite the model recovery process and improve its efficiency under limited data, we employ Low- Rank Adaptation (LoRA) (Hu et al., 2022) to post-train the pruned model. During fine-tuning, the pruning mask remains fixed, and the pretraining autoregressive loss is utilized. We fine-tun mine the LLaMA1-7B and LLaMA1-13B models pruned using SparseGPT on the C4 training dataset. The results indicate that the performance of highly sparse pruned models can be significantly restored with brief fine- tuning. The perplexity of LLMMA17BB decreased by 5.61, while that of LLLMA1 1-BBB decreased by 2.58.
Integration with Other Compression Methods. In the previous sections, we focus on the combination of RID with unstructured pruning methods. To demonstrate the generality of the proposed method, we also combine RID with structured pruning methods such LLM-Pruner, N M sparsily, and quantization. In Appendix E. we investigate the performance of non-uniform layerwise struc- tured pruning by combining RID with Appendix E.2, we investigate the application of RID within hybrid N and N sparsity configuration (Sun et al.,
In
8

---

DLP: Dynamic Layerwise Pruning in Large Language Models
2021). In Appendix E.3, we integrate our method with SVD to improve the effectiveness of low-rank compression. In Appendix E.4, we examine the performance of the pruned model after applying GPTQ (Frantar et al., 2022).
ployment of LLMs on resource-consttrineed devices, accel- erates the inference process, and promotes the sustainable development of LLMs.
References
Integration with PEFT. Recent studies (Pan et al., 2024; Li et al., 2024b) highlight significant imbalance in the dis- tribution of weight norms across different layers in LoRA during fine-tuning tasks. By leveraging importance sam- pling across LLM layers and selectively freezing most inter- mediate layers during optimization, this approach improves fine-tuning performance while maintaining memory usage comparable to that of LoRA. We apply the RID to PEFT. In Appendix F, we compare the accuracy on few-shot tasks of LLaMA2-7B, evaluating our method against other PEFT methods. Notably, our method demonstrates significant per- formance improvements compared to current state-of-the wuf-theeart art approaches. also further demonstrates the generalizability of our method.
An, Y., Zhao, X., Yu, T., Tang, M., and Wang, J. Fluctuation- based adaptive structured pruning for large language models. In Wooldridge, M. J., Dy, J. G., and Natara- jan, S. (eds.), Thirty-Eighth AAAI Conference on Artifi- cial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp. 10865-10873, AAAI Press,
2024, doi: 10.1609/AAALLV338100.88888 / //doi. www..orgg/1.... wtt.::109aaa........ v38110..00.. www.10....1............ 28960.
URL https:
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F. Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang. P., Wang. S., Wang, W., Wu, S., Xu, B., Xu, J.. Yang. A., Yang, H., Yang. J., Yang, S., Yao, Y., Yu, B., Yuan, H., Yuan, Z., Zhang, J., Zhang, X., Zhang, Y.. Zhang, Z.,, Zhou, C., Zhou, J., Zhou, X., and Zhu, T. Qwen technical report. CoRR, abs/2309. doi: 10.4650/ARKXIV2300166600 16609. URL https:/ doi orgg 100 48550 aa..5 2309. 16609.
5. Conclusion
In this paper, we propose a dynamic layerwise pruning method, DLP, which does not rely on empirical values or model architecture and can adaptively compute the relative importance of each layer. Specifically, we compute the me- dian of each Transformer block within a layer to determine the absolute unimportance of the layer, which is then con- verted into the relative importance between layers. Layers with lower importance are assigned higher sparsity. Exten- sive experimental results show that our method consistently maintains excellent performance under high sparsity, sig- nificantly outperforming existing state-of-the-art methods. Notably, our approach demonstrates strong potential, not only being compatible with other compression techniques but also integrating effectively with PEFT.
2023,
Balanca, P., Hosegood, S., Luschi, C., and Fitzgibbon, A. W. Scalify: scale propagation for efficient low-precision LLM training. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalabil- ity, and Resource Optimization (WANT@ ICML 2024), 2024. URL https: /pppeeeveeeeeeeeeeeeeeeeeeeeeeee net forum? i 1d=4 WCHW1b6K.
Acknowledgements
Boratko, M., Padigela, H., Mikkilineni, D., Yuvraj. P., Das, R. McCallum, A., Chang, M., Fokoue-Nkoutche, A., Ka- panipathi, P., Mattei, N., Musa, R.. Talamadupula, K., and Witbrock, M. A systematic classification of knowl- edge, reasoning, and context within the ARC dataset. In Choi, E.,, Seo, M., Chen, D., Jia, R., and Berant, J. (eds.), Proceedings of the Workshop on Machine Reading for Question Answering @ @AL 2018, Melbourne, Australia, July 19, 2018, pp. 60-70. Association for Computational Linguistics, 2018. doi: 10. 118653/VV1W18-2600 URL https: /tsss//aaaatttthhoc
We would like to thank the anonymous reviewers for their thoughtful comments and support on this work. This work was supported in part by the National Key Re- search and Development Program of China under Grant 2022YFF0902701; in part by the National Natural Science Foundation of China under Grants U21A20468, 62372058, U22A2026.
Impact Statement
/118 -2081/22677722
This paper focuses on pruning L.Mss. By automatically de- termining layerwise importance and assigning non-uniform sparsity, we can significantly reduce the number of param- eters in LLMs at high sparsity rates while preserving their performance. Therefore, this advancement aids in the de-
Cai, R., Muralidharan, S., Heinrich, G., Yin, H., Wang, Z., Kautz, J., and Molchanov, P. Flextron: Many-in- one flexible large language model. In Forty-forst In- ternational Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net,

---

DLP: Dynamic Layerwise Pruning in Large Language Models
2024. URL
J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasani,
K., Plawiak, K., Li, K., Heafield, K., Stone, K., and et al.
The Ilama 3 herd of models. CoRR, abs/2407. 217783, 2024. doi: 10.48550/ARXII.220.22222
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E.,, Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023.
URL https: / / ///pi. 48550 / 2407 227783.
Fan, S., Jiang, X., Li, X., Meng. X., Han, P., Shang, S., Sun, A., Wang. Y., and Wang, Z. Not all layers of Ilms are necessary during inference. CoRR, abs/2403.0218/) 2024. doi: 10.48550/ARXXI.22400.000
Clark, C., Lee, K., Chang, M., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surpris- ing difficulty of natural yes/no questions. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 2924-2936. Association for Computational Linguistics, 2019. doi: 10.18653/V 1118663//1/N119113 1/N19- 1300. URL https: //oii org DT..C..1663//// http://.....1////11111-----
URL https: 100 48550 aat.iv.iviiv.////v// www.i.21i3....... 02181.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net, 2019. URL https: / wpwnreveene.eeeeee///e/ee net/ forum? perum?id=rJin -b33RFF
Frantar, E. and A Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (cds.), International Conference on Ma- chine Learning, ICML 2023, 23-29 July 2023, Hon- olulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10323-10337. PMLR, 2023. v202 / Frantar23a html.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3.int8(): 8-bit matrix multiplication for transform- ers at scale. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Con- ference Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
mir mir.sssss/
Dong, P., Li, L.,, Tang, Z., Liu, X., Pan, X., Wang, Q., and Chu, X. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL httph::t//// httpp::///peeerevieeee / forum? id=1tRLxOzdep.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: accurate post-training quantization for genera- tive pre-trained transformers. CORR, abs/2210.17323, 2022. doi: 10.48550/ARXIV.2220.0. 17323, URL https: / Gao, L., Tow, Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang,
10. 485504554404 aaw.vv...vvvvvvv...vva .ttp:xvvi222. 2210 17323..
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A.,
J., Reynolds, L., Schoelkopf. H., Skowron, A., Sutawika,
Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku- mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A.. Gregerson, A.. Spataru, A., Rozière, B., Biron, B., Tang, B., Chemn, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C., Ferrer, C. C., Nikolaidis, C., Allonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choud- hary, D., Mahajan, D., Garcia-Olano, D., Perino, D., Hup- kes, D., Lakomkin, E., AlBadawy, E., Lobanova, E., Di- nan, E.. Smith, E. M.. Radenovic, F., Zhang, F.. Synnaeve, G., Lee, G., Anderson, G. L., Nail, G., Mialon, G. Pang, G. Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Tou- vron, H., Zarov, I., Ibarra, I. A., Kloumann, I. M., Misra, I. Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J., Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Bil- lock, J., Hong, J., Lee, J., Fu, J., Chi, J., Huang, J.. Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca,
L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation,
07 2024. URL https: / /tt.///eeooooo///o//////ooooooo org/ records/ 12608602.
Gao, X., Zhao, Y.. Dudziak, L., Mullins, R. D., and Xu, C. Dynamic channel pruning: Feature boosting and sup- pression. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https: / //openzeview. net wttett/forommmddddmd 10qYm.
Gkalelis, N. and Mezaris, V. Structured pruning of Istms via eigenanalysis and geometric median for mobile mul- timedia and deep learning applications. In IEFE Inter- national Symposium on Multimedia, ISM 2020, Naples, Ital, December 2-4, 2020, pp. 122-126. IEFE, 2020. doi:
10

---

DLP: Dynamic Layerwise Pruning in Large Language Models
10.1 10.1109/IMM.2220000003 URL https://doi.on 10. wtT.SS......0222...0
2023. doi: 104855WARXXVV230000222 URL https: / //doi //doi..rg/100 48050 231.000v...1........ 23100.6927.. 06927.
Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers. CoRR, abs/2403. 17887, 2024. doi: 00.48550/ARXI.22403... 17887. URL https: /doi. org /0/0 48550 /tt//vr.vvv./////////aa//aaaa 2403. 17887.
Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. OWQ: lessons learned from activation outliers for weight quantization in large language models. CoRR, abs/2306.022 02272, 2023. doi: 10:48550/RRXI..2366.02222
URL https: /
doi. ori.org/aa /0rr/ htt///5/////vvv.. 2306 02272.
Hassibi, B., Stork, D. G., and Wolff, G. J. Optimal brain surgeon and general network pruning. In Proceedings of International Conference on Neural Networks (ICNN'88) 88), San Francisco, CA, USA, March 28 April 1, 1993, pp- 293-299, IEEE, 1993. doi: 10.1109/CCNN... URL https: /twi//iiio/// wttp:r////........ 11t.111011111/111 11A.11119I/NN.... 19993 298572.
Lee, J., Park, S., Mo, S., Ahn, S., and Shin, I Layer-adaptive sparsity for the magnitude- based pruning. In 9th Inter- national Conference
Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net, 2021. URL http:: operrevierreeeeeeeeeeeeeee het.nnreiieeeee//ee///ee forum?id-H6Ar
Li, G., Tang, Y., and Zhang, W. LoRAP: Trans- former sub-layers deserve differentiated structured com- pression for large language models. In Salakhutdi- nov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 28657-28672. PMLR, 21-27 Jul 2024a. URL https: http:///ooocedeeeeeeeeeee ngs .mlr. html.
He, Y., Liu, P., Wang, Z., Hu, Z., and Yang, Y. Filter pruning via geometric median for deep convolutional neural networks acceleration. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 4340-4349, Computer Vision Foundation IEEE, 2019, doi: 10. CVPR.2019.00447.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adapta- tion of large language models. In The Tenth Interna- tional Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: http:pppwwwwwweweeeeeeeeeeeeeeee id-nzevKeeFYfs,
press/v235/1B /
Li, P., Yin, L.,, Gao, X., and Liu, Owlore: Outlier-weighed layerwise sampled low-rank projection for memory- efficient LLM fine-tuning. CoRR, abs/2405. 18380. 2024b. doi: 10. 10.48550ARXIV.24455.585 18380. URL https:/ doi. /ttrg//11..855//a2/// 20tp:/500002211222..1.100 2tt.al....20..8888.8
Huber, K. T., Moulton, V., Lockhart, P., and Dress, A. Pruned median networks: technique for reducing the complexity of median networks. Molecular phylogenetics and evolution, 19(2):302-33 -00-:00223100-00 2001.
Li, P., Yin, L., and Liu, Mix-In: Unleashing the power of deeper layers by combining pre-In and post-In. CoRR, abs/2412.13795 2024c. doi: 10.48550/ARXII, 2412.13795. URL https: http://iiooo///0/////// 1=xiv.2412.155
Jaiswal, A., Liu, S., Chen, T., and Wang, Z. The emergence of essential sparsity in large pre-trained models: The weights that matter. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Lin, J. Tang, J., Tang, H., Yang, S., Chen, W., Wang, W., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ: activation-aware weight quantization for on-device LLM compression and acceleration. In Gibbons, P. B., Pekhi- menko, G., and Sa, C. D. (eds.), Proceedings of the Sev- enth Anmual Conference on Machine Learning and Sys- tems, MLSys 2024, Santa Clara, CA, USA, May 13- 16, 2024. mlsys.org, 2024.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de Las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M., Stock, P.. Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. CoRR, abs/23 10.06825, 2023. doi: 10.48550/ARXII2231000222 URL https: //doi. 48550/0.4555////// /
Ling, Z., Li, Z., Romero, P., Han, L., and Nenadic, G. Bee- mane at the PLABA track of TAC-2024: roberta for task 1 1lamm3.1 and gpt-4o for task 2. CoRR, abs/241 1,07381,
2024. doi: 10.48550//AXXI.2211.007
URL https: /00 085508550//000 /awp/v/iivv///////v////aa///aaaa 2411 07381.
231 0..300000000000 06825.
/
Kurtic, E., Kuznedelev, D., Frantar, E., Goin, M., and Alistarh, D. Sparse fine-tuning for inference accelera- tion of large language models. CORR, abs/2310.06927,,
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. Few-shot parameter-efficien fine- tuning is better and cheaper than in-context learning. In

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Infor- mation Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, IA, USA, November 28- December 9, 2,2022,22 2022a.
Mihaylov, T.,, Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? A new dataset for open book question answering. In Riloff, E., Chiang, D., Hock- enmaier, J., and Tsujü, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 2381 2381-2391, Association for Computational Linguistics, 2018. doi: 10.18653/V 12600 URL attps://doi.oo 10. 1..00.865500 (011886/////111111
Liu, S., Chen, T., Chen, X., Shen, L., Mocanu, D. C., Wang, Z., and Pechenizkiy, M. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b.
Mocanu, D. C., Mocanu, E.,, Stone, P., Nguyen, P. H., Gibeseu, M., and Liotta, A. Scalable training of arti- ficial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9: 1-12, 2018.
Lu, H., Zhou, Y., Liu, S., Wang, Z., Mahoney, M. W., and Yang, Y. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. In Globersons, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomezak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, Decem- ber 10 - 15, 2024, 2024. URL nips. cc hitppppe_fiieepppppr//2////////
Muralidharan, S., Sreenivas, S. T., Joshi, R., Chochowski, M., Patwary, M., Shocybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. In Globersons, A., Mackey, L.,, Belgrave, D., Fan, A., Paquet, U., Tomezak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. Conference OpenAL. GPT-4 technical report. CoRR, abs/2303.08874 2023. doi: 10.48550/ARXXV2233300000 / / (dddi.oorgg 10. 48550/555//0 http//aavivvv... 08774.
Abstrrcct
html.
URL https:
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. In Oh, A., Nau- mann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10- - 16, 2023, 2023.
Pan, R., Liu, X., Diao, S., Pi, R., Zhang. J., Han, C., and Zhang, T. LISA: layerwise importance sampling for memory-efficient large language model fine-tuning, CoRR, abs/2403. 17919, 2024. doi: 10.48550/ARXIV. 1799.17919. URL https: o0o/10.rg111.oo 48550/ a arxiv. 2403.17911
Marcus, M. P., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. The penn treebank: Annotating predicate argument structure. In Human Language Technology, Proceedings ofa Workshop held at Plainsboro. New Jerey, USA, March 8-11, 1994. Morgan Kaufmann, 1994. URL https: / //aclanthologe www.o/ooo//9/10//0000
Puccetti, G., Rogers, A., Drozd, A., and Orletta, F. Outliers dimensions that disrupt transformers are driven by frequency. CoRR, abs/2205. 1380, 2022. doi: 10.48550/ARXIV.22005 13380. URL https://t///. /doii ////100.455//aaa0 /at.550//axii... w2w.:xiii.........mm..m 11380.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Explor- ing the limits of transfer learning with a unified text- to-text transformer. J. Mach. Learn. Res., 21:140:1- 140:67, 2020. URL https: http://mmmrrr.gggge/e/ v21/20- 074 html.
Men, X., Xu, M.. Zhang, Q. Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language models are more redundant than you expect. CoRR, abs/2403.03853, 2024. doi: 10.48550/ARXIV. 2403.03853. URL http:: htt:://doo.o..//// arXiv
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourte AAAI Conference on Ar- tificial Intelligence, AAAI 2020, The Thirty-Second In- novative Applications of Artificial Intelligence Confer- ence, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020,
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: http///pppeeeeieeeee/emeemee net/ forum? weerruummiiiyyyie 3=
12

---

DLP: Dynamic Layerwise Pruning in Large Language Models
New York, NY, USA, February 7-12, 2020, pp. 8732- 8740. AAAI Press, 2020. doi: 00.1609/AAAAI.3341055. 6399. URL https: https::/dol..or/11.110000. aaai. v34105. 6399.
analysis platform for natural language understanding. In 7th International Conference Learning Representa- tions, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https::t///ottt /oppneeveee..... net/ 2mt2
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A sim- ple and effective pruning approach for large language models. In The Tweifth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //oppnreviiiee ew. fere/fform? d=
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post- training quantization for large language models. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Hon- olulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 38087-38099, PMLR, 2023. URL ht.ps:::: / /ttp///ooeedeoooooooooooo ngs. mlr ..r.ess///// v202 xiwooaa.aa.......... html.
Sun, W. Zhou, A., Stuijk, S., Wijnhoven, R. G. J., Nelson, A., Li, H., and Corporaal, H. Dominosearch: Find layer- wise fine-grained N: M sparse schemes from dense neural networks, In Ranzato, M., Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 20721-20732, 2021.
Yin, L., Wu, Y., Zhang, Z., Hsich, C., Wang, Y., Jia, Y., Li, G., Jaiswal, A. K., Pechenizkiy, M., Liang, Y., Bendersky, M.. Wang, Z., and Liu, S. Outlier weighed layerwise sparsity (OWL)) A missing secret sauce for pruning llms to high sparsity. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, net, 2024. URL https: openreview net forum? aaemahhmm3m2 6w.
Sun, W., Song, X., Li, P., Yin, L., Zheng, Y., and Liu, S. The curse of depth in large language models. CoRR, abs/2502.05795, 2025. doi: 10.48550/ARXIV. 48550/
2502.05795. URL arXiv. 2..iv..502... 2502.05795.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Korhonen, A., Traum, D. R., and Mårquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 4791-4800. Association for Computational Linguistics, 2019. doi: 10..18653V11/19991444 URL. https:// 1472.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E.. and Lample, G. Llama: Open and efficient foundation lan- guage models. CoRR, abs/2302. 13971, 2023a. doi: 10. 10.4850/ARKXI.2332211990 org/10. 4 083588550////0 www..
URL https::///// /doi.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
Zhang, X., Xie, W., Li, Y., Lei. J., and Du Q. Filter pruning via learned representation median in the fre- quency domain. IEEE Trans. Cybern., 53(5):3165-3175 2023. doi: 10.1009/CCYB.2221312244
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferton-Feerre, C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J. Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshom, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. CoRR, fbs/2307.092883 2023b. doi: 10.48550/ARXIV22307. 09288.
URL https: //dbi. 100..11199/CC/C 2021 3124284.
Zhang, Y., Bai, H., Lin, H., Zhao, J., Hou, L., and Can- nistraci, C. V. Plug-and-plan An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRe- view.net, 2024. URL https: https////ppeeeiieeeeee n Forum?id=Tr01P
Zhu, M. and Gupta, S. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In 6th International Conference on Leurning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Workshop Track Proceedings. OpenReview.net, 2018. URL https: bettps://opezzeee /ppeneeveneeeeeeeeeeeeeeeeeeeeee net wwt.:/wwweteerer forum? i id=Sy1 IDKPM.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE; multi-rask benchmark and
13

---

DLP: Dynamic Layerwise Pruning in Large Language Models
A. Comparison among Various Layerwise Sparsity Methods
Table 9, we further compare the performance DLP with other layerwise sparsity methods on LLaMA1-7B. The details of these methods are as follows:
Global (Frankle & Carbin, 2019). A global threshold is set across all layers to automatically adjust the sparsity of specific layers while ensuring the overall sparsity requirement met.
Uniform (Zhu & Gupta, 2018). Each layer assigned the same sparsity rate for pruning.
d'' x where ef refers to the number
ER (Mocanu et al., 2018). The sparsity rate for each layer is proportional of neurons/channels in the layer.
ER-Plus (Liu et al., 2022b). Building on ER, the final layer is set as a dense layer if it is not, while maintaining the overall parameter count unchanged.
LAMP (Lee et al., 2021). Layer-adaptive sparsity is achieved by calculating the importance score of each weight relative to other weights in the same layer and globally pruning the connections with the lowest scores.
OWI. (Yin al., 2024). Non-uniform inter-layer sparsity is achieved by matching the sparsity rate of each layer with the proportion of outliers within that layer.
AlphaPruning (Lu et al., 2024). The importance of each layer in an LLM determined by analyzing the shape the empirical spectral density of its weight matrix.
The results indicate that all methods perform well when the sparsity rate is below 40%. However, when the sparsity rate is greater than equal to 40%, the performance differences become noticeable. Notably, DLP and OWL perform exceptionally well at higher sparsity rates. When the sparsity rate reaches or exceeds 70%, DLP consistently outperforms OWL. At a sparsity rate of 70%, DLP reduces the perplexity by 4.00 compared to OWL.
Table WikiText validation perplexity of LLaMA1-7B with various layerwise sparsity using Wanda. The best performance result indicated in bold.
Sparsity (Dense 5.68)
Method
10 %
20%
30%
40 %
50%
60%
70 %
80 %
Global (Frankle & Carbin, 2019)
14.11
3134
10293
10762
14848
17765
5147
39918.56
Uniform (Zhu & Gupta, 2018)
5.69
5.81
5.99
6.39
7.26
10.70
85.77
3499.88
ER (Mocanu et al., 2018)
5.69
5,80
6.02
6.55
7.74
12.16
112.03
11151.18
ER-Plus (Liu et al., 2022b)
5.70
5.82
6.05
6.62
8.00
14.04
229.17
6013.91
LAMP (Lee et al., 2021)
5.69
5.78
5.98
6.39
7.57
12.86
185.52
15647.87
OWL (Yin et al., 2024)
5.70
5.80
6.01
6.39
7.22
9.35
24.46
1227.24
AlphaPruning (Lu al., 2024)
5.69
5.81
6.00
6.37
7.18
9.47
24.00
698.56
Ours
5.70
5.81
5.99
6.38
7.17
9.35
20.46
534.42
B. Performance under Varying Levels of Sparsity
To evaluate the applicability of our method, we compare the perplexity of our method on the WikiText dataset with various sparsity rates, as presented in Table 10 and Figure 4. When sparsity is low, the differences among layerwise pruning methods relatively minor, with SparseGPT demonstrating the best performance and Magnitude showing the worst. As sparsity increases, the perplexity differences among these methods become more pronounced. In Figure 4, it is clear that DLP outperforms other methods. When the sparsity is 70%, DLP reduces perplexity by 65.92 compared to uniform layerwise pruning, using Wanda. As shown in Table 10, when the sparsity is 80%, DLP reduces perplexity by 110.32 compared to uniform layerwise pruning and by 19.02 compared to OWL, using SparseGPT, while maintaining performance within a reasonable range. These results clearly demonstrate the superiority of our method.
14

---

DLP: Dynamic Layerwise Pruning in Large Language Models
SparseGPT
(b) Wanda
Perplexity on Wikillext
Uniform
Perplexity WikiTeett
80
Uniform
OWL
OWL
Ours
Ours
40
20
60%
70%
50%
70%
Sparsity Rate
Sparsity Rate
Figure 4. Comparison different methods at high sparsity, using SparseGPT and Wanda.
Table 10. Perplexity results WikiText. We produce the Uniform, OWL, and DLP with various unstructured sparsity rates LLaMA1-
7B model.
Sparsity(Denss 5.68)
Method
Layerwise Sparsity
Weight Update
10%
20 %
30%
40%
50%
60%
70%
80
Uniform
X
5.81
6.02
6.67
8.59
17.26
562.03
48834.11 17
132881.94
Magnitude
OWL
X
8.11
6.01
6.55
8.13
13.86
82.75
19785.07
73458.74
Ours
X
5.73
6.03
6.80
8.69
18.12
133.65
3437.55
48933.69
Uniform
V
5.70
5.80
5.96
6.32
7.21
10.47
25.38
180.94
SparseGPT
OWL
V
5.71
5.81
6.00
6.32
7.19
9.66
19.08
89.64
Ours
V
5.70
5.81
5.97
6.32
7.15
9.44
17.76
70.62
Uniform
X
5.70
5.82
6.00
6.39
7.26
10.70
86.38
7784.48
Wanda
OWL
x
5.70
5.80
6.01
6.39
7.22
9.35
24.46
1227.24
Ours
5.70
5.81
5.99
6.38
7.17
9.35
20.46
534.42
C. Per-Block vs. Per-Layer
As we mentioned before, we assign a distinct pruning ratio for each layer instead of each Transformer block. We test the performance of these methods on LLaMAI-7B under 70% sparsity, using DLP pruming with Wanda. The perplexity values are 3463.32 and 20.46, respectively. Table 11 and Table 12 present the sparsity levels of seven fully connected layers, including q-proj. k k.proj. u-proj, o-proj, gate_proj. down-proj, and up-proj layers in layers 1, 2, 15, 30, and 31. It is noteworthy that comparing importance at the level of each Transformer block results in varying sparsity levels across blocks, leading to suboptimal performance. This is likely because such an approach creates significant sparsity discrepancies between blocks, potentially disrupting inter-layer information flow. In contrast, comparing importance at the layer level ensures uniform sparsity across Transformer blocks within each layer, which proves to be more beneficial for the performance of LLMs.
D. Per-Output vs. Per-Layer
We also compare the performance of per-output pruming and per-layer pruning. As shown in Table 13 and Figure 5, Wanda to compare the perplexity of the I.LMMA-7BB model at different sparsity levels for per-output pruning and per-layer pruning. Notably, the perplexity of the per-output pruning consistently outperforms per-layer pruning, with the performance gap becoming more pronounced as sparsity increases. Specifically, at 70% sparsity, the perplexity of the per-output method 18.56 lower than that of the per-layer method. These results demonstrate that, after obtaining lnter-layer importance through DLP, performing localized pruning on individual output neurons within each layer yields greater benefits.
15

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table 11. Sparsity of LLLMMA pruned with per-layer DL.P 70% unstructured sparsity, using Wanda.
Layer
q.proj
k.proj
v.proj
0.proj
gate.proj
down.proj
up.proj
0,548
0.548
0.548
0.548
0.548
0.548
0.548
2
0.565
0.565
0.565
0.565
0.565
0.565
0,565
5
0.609
0.609
0.609
0.609
0.609
0.609
0.609
30
0.832
0.832
0.832
0.832
0.832
0.832
0.832
31
0.813
0.813
0.813
0.813
0.813
0.813
0.813
Table 12. Sparsity of LLaMA1-7B pruned with per-block DLP at 70% unstructured sparsity, using Wanda.
Layer
q.proj
k.proj
v.proj
o.proj
gate.proj
down.proj
up.proj
I
0.788
0.786
0.829
0.842
0.811
0.813
0.841
2
0.759
0.755
0.815
0.839
0.793
0.798
0.838
5
0.669
0.666
0.743
0.829
0.755
0.766
0.827
30
0.578
0.574
0.563
0.743
0.606
0.681
0.626
31
0.624
0.619
0.639
0.747
0.621
0.624
0.628
E. Integration with Other Compression Methods
In the previous section, we primarily examine the effectiveness of combining our method with unstructured pruning methods. As general non-uniform layerwise approach, our method is inherently applicable to broader range of scenarios. To explore its potential, apply RID to structured pruning. N M sparsity, SVD and quantization, respectively.
E.1. Integration with Structured Pruning
In addition, we apply our method to structured pruning methods. Following the setup of LLM-Pruner (Ma et al., 2023), prune not individual weights but entire neurons and attention heads. This approach directly reduces the model's parameter size and enables acceleration. We replace the uniform layerwise sparsity in LLM-Pruner with the non-uniform sparsity provided by DLP. The results, shown in Table 14, indicate that applying non-uniform sparsity allows LLM-Pruner to better preserve performance across different sparsity levels.
E.2. Integration with N M Sparsity
To evaluate the potential of our method in hardware-friendly applications, we apply it to N . sparsity. Following the setup of DominoSearch (Sun et al., 2021). we investigate mixed 8 and 4 sparsity configurations. Unlike using a uniform N' value across all layers, we allocate different N' values based on layer importance while keeping the overall parameter count unchanged. The results, shown Table 15, demonstrate that our method achieves superior performance compared to uniform N .M sparsity. Notably, in high-sparsity scenarios of 4 and 2 our method reduces perplexity by 240x and 41x, respectively.
E.3. Integration with SVD
We further extend our method SVD to enhance low-rank compression. By leveraging RID, we assign different SVD compression rates to each layer. A higher RID score indicates greater layer importance, resulting in lower compression rate to preserve model performance. In Table 16, we present the perplexity of ILLMA1--7 under various compression rates. As the compression rate increases, the model's performance degradation becomes more pronounced. Notably, our method consistently outperforms uniform layerwise SVD.
E.4. Integration with Quantization
Finally, we apply the LLaMAI-7B model pruned with non-uniform layerwise sparsity to quantization techniques to evaluate whether can maintain pre- ve-pruning performance. Using the LLaMA1-77 model pruned to 70% sparsity with SparseGPT, we assess perplexity before and after quantization with GPTQ (Frantar et al., 2022) on the WikiText, PTB, and C4 datasets. The quantization bit widths are set 3, 4, 8, and 16. The results, presented in Table and Figure 6, reveal that the model
16

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table 13. Comparison of per-ouiput and per-layer perplexity different sparsity rates, using Wanda. indicated in bold.
The hest performance result
Method
10%
20%
30 %
40 %
50%
60%
70%
80%
Per-Output 5.70
5.81
5.99
6.38
7.17
9.35
20.46
534.42
Per-Layer 5.70
5.82
6.03
6.56
7.70
10.95
39.02
886.10 10
10²
Per-Output Per- Layer
Perplexity
10a
10
10%
20%
30%
40%
50%
60%
70%
80%
Sparsity Rate
Figure 5. Comparison of per-output and per-layer perplexity at different sparsity rates. Due to the significant difference in values between high and low sparsity rates, use the logarithmic results for comparison.
pruned with DLP consistently outperforms those with uniform sparsity during quantization. Notably, the performance of the 4-bit quantized model is nearly identical that of the 16-bit quantized model. This demonstrates that applying DLP enables 4x reduction in model size while maintaining performance.
F. Integration with PEFT
We apply the RID to PEFT. We mainly consider two state-of-the-at basclines: Layerwise Importance Sampled Adam (LISA) (Pan et al., 2024) and OwLore (Li al., 2024b). LISA selectively updates critical LLM layers based on importance sampling while keeping the remaining layers unchanged. Inspired by OWL, OwLore strategically assigns higher sampling probabilities to layers with
outliers, selectively sampling only few layers and fine-tuning their pre-trained weights. We assign higher sampling probabilities to layers with higher layerwise importance based RID. Following the settings of OwLore, all methods are first fine-tumed on commonsense 1 17kk and then evaluated separately on different tasks with 5 shots, the results are shown Table 18. Notably, our method achieves an average accuracy improvement of 4.88% compared to LISA and 0.8% compared to OwLore.
G. Hyperparameter Setting
Due to the potentially wide range of importance differences across layers, directly using unadjusted importance scores may re- sult in excessive pruning of some layers while under-pruning others. Adjusting the range of relative importance helps to make pruning decisions more balanced and precise across layers. To determine the matching relationship between different sparsity levels and relative importance, we conduct iterative experiments with a E [0.02, 0.04, 0.06, 0.08, 0.1, 0.12, 0.15, 0.2]. We provide the hyperparameter setting for different sparsity levels to facilitate the reproduction of our method's results, as shown in Table 19.
17

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table 14. Perplexity of structured pruning with IJaMA1-7B. WikiText. The best performance result indicated in hold.
Dataset
Method
Layerwise Sparsity
20%
40% %
60%
80% %
WikiText LLM-Pruner
Uniform
18.61
647.20
4074.48
33849.77
WikiText
LLM-Pruner
Ours
16.62
29.62
115.58
642.16
PTB LLM-Pruner
Uniform
90.02
538.65
2330.65
23447.05
PTB
LLM-Pruner
Ours
68.75
142.73
617.56
1932.18
Table 15. Perplexity of mixed N:M sparsity (N refers weights) with LLaMAI-7B WikiText. The best performance result indicated in bold.
N:M Sparsity Structure
Layerwise Sparsity
1:4
2:4
3:4
1:8
2:8
3:8
4:8
5:8
6:8
7:8
Uniform
7225.68
11.55
6.19
29790.41
3485.97
42.86
8.56
6.61
6.02
5.74
Ours
30.44
7.32
5.91
4990.69
83.50
10.42
7.19
6.27
5,87
5.71
H. Robustness across Various Validation Datasets
To evaluate the robustness of the proposed method, we also test the performance of LLaMA1 and LLaMA2 models on different validation datasets. In Table 20, we report the perplexity of LLaMAI (7B/13B/30B) pruned to 70% sparsity on WikiText, PTB, and C4. When using uniform layerwise sparsity, the Magnitude method performs the worst at high sparsity levels, while SparseGPT performs the best, primarily because SparseGPT updates weights post-pruning to recover performance. When combined with Wanda or SParseGPT, DLP consistently outperforms uniform layerwise pruning and OWL. In Table 21, present the perplexity results of LLaMA2 (7B/13B) pruned to 70% sparsity WikiText, PTB, and C4. Notably, DLP consistently outperforms the other layerwise methods. These experimental results strongly demonstrate the robustness of our method across different datasets.
I. Zero-shot Tasks Performance
In Table 22 and Table 23, we present the accuracy of the pruned models on seven commonsense benchmarks from the EleutherAI LM Harness (Gao al., 2024), including BoolQ (Clark al., 2019), RTE(Wang et al., 2019), HellaSwag (Zellers al., 2019), WinoGrande (Sakaguchi al., 2020), ARC Easy and Challenge (Boratko et al., 2018), and OpenbookQA (Mihaylov et al., 2018). Notably, the average accuracy of our method consistently outperforms other layerwise methods.
18

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table Perplexity of LLaMAI-7B across various compression rates. The best performance result indicated bold.
Method 0%
10 %
20 %
30%
40%
50%
Uniform 5.68
6.58
8.41
17.69
1917.76
19170.02
Ours 5.68
6.58
8.38
15.33
1252.03
16304.26
Table Perplexity of LLaMA1-7B on different validation datasets under varying quantization levels at 70% unstructured sparsity, pruned with DLP using SparseGPT. The best performance result is indicated in bold.
Bits Layerwise Sparsity Sparsity WikiText
PTB
C4
Dense
5.68
31.50
7.08
16
Uniform
70%
26.49
298.15
24.57
Ours
70%
18.38
182.69
17.72
Dense
0
5.68
31.46
7.08
8
Uniform
70%
26.50
298.70
24.51
Ours
70%
18.38
183.25
17.71
Dense
5.79
32.16
7.23
4
Uniform
70%
27.08
315.35
24.93
Ours
70%
18.92
178.28
18.13
Dense
6.23
36.38
8.02
3
Uniform 70%
30.73
428.32
29.92
Ours
70%
21.56
243.98
20.59
Table 18. Accuracy(%) of different Parameter-Ffficien Fine-Tuning (PFFT) methods few-shot tasks with I.aMMA-77. The hest
performance result is indicated in hold.
Method
BoolQ
HellaSwag WinoGrande
ARC-e
ARC-e
OBQA
Mean
LISA (Pan et al., 2024)
77.16
71.60
76.87
74.75
45.65
45.20
65.21
OwLore (Li et al., 2024b)
82.63
78.33
79.08
78.24
51.88
45.60
69.29
Ours
81.90
78.55
80.58
79.92
52.99
46.60
70.09
WiklTest
PT
(9)C4
Laskirm Oluxs
012
350
300
20
250
65
154
10-
Bals
Figure 6. Perplexity of LLaMA1-7B different validation datascts under varying quantization bits.
Table 19. Hyperparameter settings for different sparsity levels.
Sparsity 10% 20% %
30%
40 %
50%
60%
70 %
80%
0.06
0.02
0.04
0.02
0.04
0.1 0.15 0.12
19

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table 20. Perplexity of LLaMA1 models on different validation datasets at 70% unstructured sparsity. The best performance result indicated in bold.
Model
Method
Layerwise Sparsity
Weight Update WikiText
PTB
C4
Dense
5.68
41.15
7.34
Uniform
48838.41
141244.92
23374.60
Magnitude
OWL
19785.07
164741
29854.54
Ours
3437.55
798602.88
9694.97
Uniform
25.38
331.89
28.23
LLaMA1-7B
SparseGPT
OWL
19.08
221.95
20.61
Ours
17.76
202.06
19.34
Uniform
X
86,38
698.63
88.14
Wanda
OWL
X
24.46
398.58
27.36
Ours
20.46
302.15
22.71
Dense
5.09
28.10
6.80
Uniform
84511.48
389981.53
40205.27
Magnitude
OWL
18992.87
61249.96
19282.38
Ours
7642.99
14510.26
33386.63
Uniform
18.93
150.92
22.70
LLaMA1-13B
SparseGPT
OWL
14.02
99.02
16.19
Ours
12.63
80.19
14.47
Uniform
56,26
324.01
54.69
Wanda
OWL
16.23
146.35
18.84
Ours
13.65
100.94
15.82
Dense
4.10
23.51
6.13
Uniform
971.71
10452.38
5372.17
Magnitude
OWL
242.80
2495.37
808.44
Ours
98.05
491.10
114.72
Uniform
12.87
62.54
15.47
LLaMA1-30B
SparseGPT
OWL
10.22
48.42
12.79
Ours
9.43
40.54
11.69
Uniform
17.54
111.33
18,81
Wanda
OWL
10.77
60.24
13.62
Ours
9.93
46.51
12.36
20

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table 21. Perplexity of I.L.M22 models different validation datasets at 70% unstructured sparsity. The best performance result is indicated in bold.
Model
Method
Layerwise Sparsity Weight Update WikiText
PTB
C4
Dense
5.47
37.92
7.26
Uniform
49840.8
141244.92
27822.82
Magnitude
OWL
15480.39
76684.05
21543.82
Ours
X
8736.22
64708.39
3487.59
Uniform
27.84
8557.88
30.44
LLaMA2-7B
SparseGPT
OWL
19.71
3930.91
22.68
Ours
18.58
677.99
19,51
Uniform
76.84
778.75
78.75
Wanda
OWL
30.58
450.63
36.89
Ours
22.79
256.86
26.76
Dense
4.88
50.94
6.72
Uniform
214.19
3706.61
191.92
Magnitude
OWL
57.55
2125.47
50.79
Ours
52.41
1008.90
41.89
Uniform
19.38
450.85
23.41
LLaMA2-13B
SparseGPT
OWL
15.12
304.91
21.74
Ours
13.30
242.57
15.62
Uniform
45.76
548.29
56.10
Wanda
OWL
20.65
326.07
21.74
Ours
16.19
239.34
18.47
21

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table 22. Accuracy(%6) (6) of LLaMAl on seven zero-shot tasks at 70% unstructured sparsity. The best performance result is indicated in bold.
Layerwise Sparsity
Model
Method
BoolQ
RTE HellaSwag
WinoGrande
ARC-e
ARC-c OBQA Mean
Dense
75.05
66.79
76.22
70.09
72.94
44.80
44.40
64.33
Uniform
38.29
52.71
25.81
51.22
26.68
25.09
23.80
34.80
Magnitude
OWL
37.83
52.71
29.34
52.80
28.24
27.65
26.20
36.40
Ours
38.38
53.43
35.79
52.41
31.82
28.24
27.40
38.21
Uniform
65.32
54.15
42.59
58.80
41.71
26.45
28.20
45.32
LLMMA1-7B
SparseGPT
OWL
67.09
53.43
48.28
63.30
44.23
27.56
31.00
47.84
Ours
68.62
53.43
49.13
61.33
44.61
29.35
31.80
48.32
Uniform
57.92
57.76
31.25
50.75
32.62
21.67
27.40
39.91
Wanda
OWL
62.57
56.68
44.28
59.59
43.69
27.05
30.40
46.32
Ours
63.21
59.21
47.17
60.06
50.17
28.92
31.60
48.62
Dense
77.89
70.40
79.07
72.77
74.66
47.87
44.80
66.78
Uniform
52.91
50.54
27.52
50.83
28.03
24.83
25.00
37.09
Magnitude
OWL
55.87
49.10
30.27
50.43
31.48
26.62
32.40
39.45
Ours
58.41
50.54
29.68
51.14
30.77
27.65
32.60
40.11
Uniform
67.95
52.71
48.67
61.88
46.97
28.41
31.80
48.34
LLaMA1-13B
SparseGPT
OWL
66.30
52.71
54.14
66.61
50.08
30.20
35.40
50.78
Ours
68.47
54.87
57.92
69.14
53.70
31.74
35.60
53.06
Uniform
61.93
52.71
34.38
52.72
37.42
22.01
30.20
41.62
Wanda
OWL
62.78
52.71
51.08
63.06
52.31
30.20
35.00
49.59
Ours
66.36
52.71
56.19
64.56
56.90
31.66
35.80
52.03
Dense
82.63
66.79
82.64
75.93
78.96
52.90
48.20
69.72
Uniform
39.27
46.93
26.09
52.17
26.43
25.94
28.60
35.06
Magnitude
OWL.
39.02
56.68
26.30
49.33
27.86
24.49
26.40
35.73
Ours
61.62
47.29
34.83
52.72
39.14
27.82
30.80
42.03
Uniform
68.90
57.76
60.31
69.93
61.24
35.32
37.60
55.87
LLaMAI-30B
SparseGPT
OWL
66.61
55.96
63.15
71.74
63.51
36.77
40.00
56.82
Ours
68.87
53.79
65.68
72.38
64.48
38.65
41.00
57.84
Uniform
65.87
56.32
58.31
66.69
61.07
34.90
39.00
54.59
Wanda
OWL
66.30
53.79
62.12
69.69
64.06
35.32
40.20
55.93
Ours
64.95
48.74
65.39
70.24
67.72
38.74
42.00
56.83
22

---

DLP: Dynamic Layerwise Pruning in Large Language Models
Table 23. (ccuracy(%) of LLaMA2 zero-shot tasks 70% unstructured sparsity. The best performance result is indicated in bold.
Layerwise Sparsity
Model
Method
BoolQ
RTE HellaSwag
WinoGrande
ARC-e
ARC-c OBQA
Mean
Dense
77.71
62.82
76.00
69.30
74.58
46.33
44.20
64.42
Uniform
37.95
53.07
26.36
49.33
27.86
26.96
28.00
35.65
Magnitude
OWL
40.03
52.35
30.10
48.54
30.72
26.37
27.00
36.44
Ours
46.51
52.71
37.93
51.78
37.58
28.58
30.80
40.84
Uniform
65.35
53.43
41.07
58.01
40.66
24.74
29.80
44.72
LLaMA2-7B
SparseGPT
OWL
67.92
53.07
47.97
62.04
47.31
26.02
31.80
48.02
Ours
71.25
53.79
50.23
62.19
49.07
27.65
33.40
49.65
Uniform
48.23
52.71
30.28
49.96
30.30
21.42
26.40
37.04
Wanda
OWL
62.11
52.71
37.46
56.27
42.05
24.06
30.20
43.55
Ours
62.29
52.71
44.19
58.80
46.97
25.77
33.00
46.25
Dense
80.55
65.34
79.39
72.30
77.53
48.98
45.20
67.04
Uniform
38.62
52.71
29.56
49.41
32.11
24.57
26.60
36.23
Magnitude
OWL
38.65
52.71
43.89
54.54
37.63
28.84
28.40
40.67
Ours
40.55
52.71
50.34
59.43
44.57
31.14
29.40
44.02
Uniform
67.16
52.71
47.05
61.40
48.91
27.90
30.80
47.99
LLaMA2-13B
SparseGPT
OWL
69.45
54.87
52.86
65.27
53.24
30.38
35.80
51.70
Ours
74.22
54.15
55.80
65.67
54.84
33.02
36.60
53.47
Uniform
62.11
52.71
31.71
51.78
35.73
20.82
28.20
40.44
Wanda
OWL
63.67
52.71
46.30
60.85
51.01
28.24
34.00
48.11
Ours
67.06
52.71
52.98
64.64
54.59
30.97
34.80
51.11
23


Reviews:
[{"review_id": "5AwXcgIyA9", "reviewer": "Reviewer_AtTG", "summary": "The paper proposes a new method for pruning LLMs by determining the importance of each layer for determining the layer-wise sparsity ratio. The proposed method, DLP, adaptively determines the importance of each layer by combining model weights with input activation information. DLP uses a median to detect outlier values instead of setting a manual threshold value.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer AtTG:\n\nThank you for taking the time to read and review our paper! In the following, we would like to address your comments point by point. \n\n>Q1: Limited technical novelty and direct extension using median-based outlier filtering.\n\nThank you for your insightful comments. We would like to clarify that our proposed dynamic layerwise importance method is not merely a direct extension. We propose a dynamic layerwise pruning method for LLMs, which automatically evaluates the relative importance of each layer and adaptively assigns non-uniform pruning rates accordingly. The key innovations of our approach are as follows:\n\n(1) **Automatic Layerwise Importance Assessment without Predefined Thresholds:** Unlike OWL, which assigns layerwise sparsity based on a predefined threshold of outlier ratio, our method automatically evaluates layer importance using the median, eliminating the reliance on empirical values and enhancing generalization across different models and parameter settings.\n\n(2)  **Dynamic Adaptability:** By adjusting the pruning rate of each layer based on global relative importance, our approach preserves more parameters in critical layers, significantly improving performance at high sparsity levels (e.g., reducing perplexity by 7.79 and increasing accuracy by 2.7% for LLaMA2-7B at 70% sparsity).\n\n(3) **Compatibility and Scalability:** Our method seamlessly integrates with unstructured pruning, structured pruning, N:M sparsity, quantization, SVD, and PEFT, achieving up to a 3.7x speedup in CPU inference, promoting the practical deployment of LLMs in resource-constrained environments.\n\nAdditionally, we clarify that our method emphasizes retaining rather than filtering outliers, as OWL demonstrates that a higher proportion of outliers (LOD) positively correlates with performance. Leveraging the robustness of the median, our approach removes redundancy while effectively preserving crucial outliers, resulting in a higher LOD after pruning (see Table 2).\n\n>Q2: Limited model diversity, only LLaMA.\n\nThank you for raising this concern. As also noted by Reviewer zWSE, our experiments already include Vicuna, Mistral, and Qwen in addition to the LLaMA series. These models are introduced in Section 4.1 (Experimental Setup), and their results are reported in Section 4.3 (Performance on More Advanced LLMs) and Table 8. We will make this more explicit in the revised manuscript to avoid any confusion.\n\n>Q3: Comparison only with SparseGPT and Wanda, missing other LLM pruning baselines.\n\nThank you for your valuable comment. In our paper, we compare a range of pruning methods, including Magnitude, Wanda, SparseGPT, OWL, and LLM-Pruner. Based on your suggestion, we have further extended our experiments to include two recent pruning approaches: RIA [1] and AlphaPruning [2]. As shown in the **table 24** (available in the supplementary repository: https://anonymous.4open.science/r/DLP/rebuttal.md ), we evaluate the accuracy of LLaMA3-8B on seven zero-shot tasks at 70% sparsity. Experimental results demonstrate that our method outperforms others using RIA. Moreover, in our response to Reviewer EVqz's Q5, we also provide a comparison with Evopress. We will include such experimental results and discussion in our revised version.\n\n\n[1] Zhang Y, Bai H, Lin H, et al. Plug-and-play: An efficient post-training pruning method for large language models.ICLR 2024.\n\n[2] Lu H, Zhou Y, Liu S, et al. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. NeurIPS 2024.\n\n> Q4: a typo at line 263.\n\nThank you for pointing this out. We will carefully address this in our revised manuscript.\n\n>Q5: Clarify why the proposed method has lower overhead than uniform pruning.\n\nThank you for your insightful comment. Our intention in conducting the Pruning Efficiency evaluation is to demonstrate that, although our method incurs slightly more overhead than uniform layerwise pruning due to dynamic layerwise pruning, it does not lead to significant additional empirical pruning time. As shown in Table 6, our approach exhibits comparable time overhead to uniform pruning. In some cases, certain models even run faster with our method, likely because it better aligns with the model’s distribution, allowing for a more efficient selection of pruning units.\n\n>Q6: Do you have any insights as to why the red more or less increases consistently from the layer index in Fig. 6?\n\nThank you for your insightful comments. We assume you are referring to Fig.3. The overall trend of increasing sparsity indicates that earlier layers are considered more important, while deeper layers are less sensitive to pruning, suggesting that they do not contribute as effectively as expected. For brevity and to avoid redundancy, we have provided a detailed analysis of this topic in our response to Reviewer qTaq's Q2. We are happy to elaborate further if needed.", "claims_and_evidence": "The paper claims that median values are used to detect outlier values in the LLM layers. The authors present some evidence by comparing the proposed method with Wanda and SparseGPT, which suggests that the proposed method can achieve better performance at 70\\% sparsity; more experiments and comparison against more pruning methods would be required to fully substantiate the claim.", "methods_and_evaluation_criteria": "Yes,", "theoretical_claims": "No theoretical claim is introduced in the paper.", "experimental_designs_or_analyses": "The method is only compared with SparseGPT and Wanda. More extensive experiments should be conducted and the method should be benchmarked against more recent LLM pruning methods. However, the paper only uses LLaMA family of models, authors should use other families of open-source models to demonstrate the efficacy of the method.", "relation_to_broader_scientific_literature": "The key contribution of the paper is to explore the use of median values instead of mean values to filter out the outlier values and to determine the layerwise sparsity ratio.", "essential_references_not_discussed": "Related works are discussed and cited.", "other_strengths_and_weaknesses": "### Strenghts:\n\n1. The paper is well-written and easy to follow.\n\n### Weakness:\n\n1. The technical novelty of the proposed method is limited. The method is a direct extension of previous works and uses median to filter out the outlier values. \n\n2. The method is only compared with two model pruning baselines (SparseGPT, Wanda). Proposed method should be compared against with more recent LLM pruning methods.", "other_comments_or_suggestions": "There is a typo at line 263.\n\n## update after rebuttal:\n\nI would like to keep my original score. While the authors present extensive experiments, using the median value is not novel enough and there is no theory/analysis why this works.", "questions_for_authors": "It is not clear how the overhead timing for the proposed method is lower than uniform pruning in the layer. Could you please provide an explanation. \n\nDo you have any insights as to why the red more or less increases consistently from the layer index in Fig. 6?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qPAYIqnj7o", "reviewer": "Reviewer_9Taq", "summary": "This paper aims to improve LLM pruning by introducing a new layerwise sparsity ratio inspired by OWL. Unlike OWL, which assigns layerwise sparsity based on predefined threshold of outlier ratio, DLP measures layerwise importance based on the median of outlier ratios.  This simple modification not only avoids hyperparameter tuning of OWL, but also leads to stronger results. The effectiveness of DLP is extensively evaluated with structured/unstructured llm pruning, low-rank decomposition, quantization, and PEFT following OwLore.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 9Taq:\n\nWe sincerely appreciate your recognition of DLP’s median-based importance scoring and its extensive validation across multiple compression paradigms (pruning, quantization, etc.). Below, we address your comments point by point:\n> Q1: The paper presentation can be significantly improved.\n\n\nThank you for taking the time to review our paper and for your valuable feedback. We will carefully revise and enhance the presentation in the updated manuscript.\n\n\n>Q2: I would encourage authors to delve deep to provide intuitions why the median is better than OWL. Why does a low median mean more important layers? The linear decayed importance across layers also strikes me, which is different from my expression. This finding actually aligns well with the recent findings in the ''Curse of Depth'' https://arxiv.org/abs/2502.05795. A plausible understanding is that as the depth increases, deeper layers gradually suffer from large output variance, leading to less contribution to the training. Could the authors provide their intuitions here?\n\n\nWe appreciate your thoughtful comments and particularly thank you for highlighting the connection with recent findings such as the \"Curse of Depth\". We fully agree with your perspective. Recent research has highlighted the diminishing contribution of deeper layers to learning and representation. For instance, [1] finds that, compared to earlier layers, deeper layers contribute significantly less due to the exponential growth of output variance with model depth, leading to their degradation during training. [2] demonstrates that deeper layers contribute minimally to performance during fine-tuning. Similarly, [3,4] indicate that certain layers have a negligible impact on the overall network function and can be removed without significantly affecting model performance. [5] argues that, ideally, all layers in a model should be sufficiently trained, and feature representations across layers should exhibit enough diversity to maximize the utility of network parameters. \n\n\nOur experiments reveal a similar phenomenon, where deeper layers fail to contribute as effectively as expected. We posit that the median serves as an indicator of layer redundancy. Since the median is insensitive to extreme values, it better captures the central tendency. Elements near the center are easily represented by their neighboring elements, making their removal less detrimental to performance. A lower median within a layer suggests minimal redundancy in its weights, whereas a higher median implies greater redundancy. Consequently, layers with higher redundancy are considered less influential to the overall model and are assigned a higher sparsity rate during pruning.\n\n\nMoreover, the overall increasing trend in sparsity rates suggests that earlier layers are assigned greater importance, likely due to their foundational role in capturing low-level and generalizable features. In contrast, deeper layers often focus on more specialized or task-specific information, which can be more redundant or tolerant to pruning [6]. This aligns with findings in neural network pruning literature, where early layers tend to be more sensitive to pruning than later ones [7,8]. Therefore, our method adaptively allocates pruning budgets based on such structural and functional differences across layers.\n\n\nThank you for raising such a meaningful question. It inspires us to further explore the fundamental challenges of deeper layers in LLMs in our future work.\n\n\n\n\n[1] Sun W, Song X, Li P, et al. The Curse of Depth in Large Language Models. arXiv preprint arXiv:2502.05795, 2025.\n\n[2] Li P, Yin L, Gao X, et al. Owlore: Outlier-weighed layerwise sampled low-rank projection for memory-efficient llm fine-tuning. arXiv preprint arXiv:2405.18380, 2024.\n\n[3] Gromov A, Tirumala K, Shapourian H, et al. The unreasonable ineffectiveness of the deeper layers. arXiv preprint arXiv:2403.17887, 2024.\n\n[4] Men X, Xu M, Zhang Q, et al. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853, 2024.\n\n[5] Li P, Yin L, Liu S. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. arXiv preprint arXiv:2412.13795, 2024.\n\n[6] Fan S, Jiang X, Li X, et al. Not all layers of llms are necessary during inference[J]. arXiv preprint arXiv:2403.02181, 2024.\n\n[7] Chen X, Hu Y, Zhang J, et al. Streamlining redundant layers to compress large language models[J]. arXiv preprint arXiv:2403.19135, 2024.\n\n[8] Yang Y, Cao Z, Zhao H. Laco: Large language model pruning via layer collapse[J]. arXiv preprint arXiv:2402.11187, 2024.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. This paper evaluates their algorithms on almost all perspectives I could think about, just like OWL, including structured/unstructured llm pruning, low-rank decomposition, quantization, and PEFT following OwLore.", "methods_and_evaluation_criteria": "Both perplexity and zero-shot results on commonsense tasks are provided. While the reason behind the success of DLP is not clearly explained, their empirical performance is convincing enough. \n\nI would encourage authors to delve deep to provide intuitions why the median is better than OWL. Why does a low median mean more important layers? The linear decayed importance across layers also strikes me, which is different from my expression. This finding actually aligns well with the recent findings in the ''Curse of Depth'' https://arxiv.org/abs/2502.05795. A plausible understanding is that as the depth increases, deeper layers gradually suffer from large output variance, leading to less contribution to the training. Could the authors provide their intuitions here?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I am satisfied with the experimental designs.", "relation_to_broader_scientific_literature": "The linear decayed importance across layers also strikes me, which is different from my expression. This finding actually aligns well with the recent findings in recent works [1] [2]. A plausible understanding is that as the depth increases, deeper layers gradually suffer from large output variance, leading to less contribution to the training. Could the authors provide their intuitions here?\n\n[1] Li, P., Yin, L. and Liu, S., 2024. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. arXiv preprint arXiv:2412.13795.\n\n[2] Sun, Wenfang, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. \"The Curse of Depth in Large Language Models.\" arXiv preprint arXiv:2502.05795 (2025).", "essential_references_not_discussed": "[1] Li, P., Yin, L. and Liu, S., 2024. Mix-ln: Unleashing the power of deeper layers by combining pre-ln and post-ln. arXiv preprint arXiv:2412.13795.\n\n[2] Sun, Wenfang, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. \"The Curse of Depth in Large Language Models.\" arXiv preprint arXiv:2502.05795 (2025).\n\n[3] Gromov, Andrey, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. \"The unreasonable ineffectiveness of the deeper layers.\" arXiv preprint arXiv:2403.17887 (2024).\n\n[4] Men, Xin, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. \"Shortgpt: Layers in large language models are more redundant than you expect.\" arXiv preprint arXiv:2403.03853 (2024).", "other_strengths_and_weaknesses": "**Strengths**:\n\nUsing median to avoid hyperparameter tuning is important for OWL-like algorithms. \n\nI like the preliminary study in Figure 2, which provides good motivations for the paper. \n\nEvaluating on a wide range of tasks demonstrates the effectiveness of DLP.\n\n**Weaknesses**:\n\nThe paper presentation can be significantly improved.", "other_comments_or_suggestions": "The paper presentation can be significantly improved. \n\nFor instance, Figure 1 can be improved with aligned sub-figures. \n\nLine 20 has no space between text and citation. \n\nSome tables in the appendix lack a bottom line.", "questions_for_authors": "I would encourage authors to delve deep to provide intuitions why the median is better than OWL. Why does a low median mean more important layers? The linear decayed importance across layers also strikes me, which is different from my expression. This finding actually aligns well with the recent findings in the ''Curse of Depth'' https://arxiv.org/abs/2502.05795. A plausible understanding is that as the depth increases, deeper layers gradually suffer from large output variance, leading to less contribution to the training. Could the authors provide their intuitions here?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mMorD9HZbL", "reviewer": "Reviewer_EVqz", "summary": "The  paper focused on finding important layers to perform non-uniform pruning a.k.a to prune each layer in different rates. More important layers are pruned less aggressively, less important weights pruned more. The paper can be easily extended to any sparsification approach with just determining the pruning rate of each layer.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer EVqz:\n\nWe sincerely appreciate your time and thoughtful review of our work. Please find our detailed, point-by-point responses below.\n\n>Q1: Clarify LOD metric and its significance (Line 253) with a formula and explanation.\n\nThank you for your valuable comment. Followed by OWL, weight outliers are defined as weights with an outlier score at least M times greater than the average. LOD refers to the ratio of weight outliers, calculated as the number of outlier weights divided by the total number of weights (including both zero and non-zero weights), corresponding to Equation 4 in the paper. OWL demonstrates that LOD is positively correlated with model performance. In Table 2 of our paper, we compare the LOD after pruning with OWL. Our method achieves a higher LOD and delivers better performance.\n\n>Q2: Some of the model are outdated such as llama1, llama2; No zero-shot results on llama3 models; No results on bigger models f.e. Llama3 70B.\n\nThank you for your valuable comment. Due to hardware limitations, we are currently unable to conduct experiments on models larger than 30B parameters. Therefore, we choose the LLaMA1 model family (7B, 13B, and 30B) for our primary experiments. In our response to reviewer AtTG, we present the zero-shot results of LLaMA3-8B using SparseGPT. Additionally, we include two other methods, and the experimental results demonstrate that our approach outperforms the others. We hope to explore evaluations on larger models, such as LLaMA3-70B, in future work once sufficient computational resources become available.\n\n>Q3: Alpha hyper-parameter choice vs. OWL's M\n\nThank you for your insightful comments. In OWL, weight outliers are defined as weights with an outlier score at least M times greater than the average. In other words, M serves as the threshold for measuring outliers. Our method automatically determines layer importance using the median. Alpha controls the range of layer importance, preventing excessive pruning caused by overly large values, which could lead to significant performance degradation.\n\n>Q4: Limited practical impact despite improvements, underperforming at <60% sparsity compared to existing methods (see Tables 3 and 9).\n\nThank you for your insightful comments. When hardware resources are limited, such as in edge devices, deploying LLMs with a high number of parameters becomes a challenge. Pruning offers a solution, though it often requires a trade-off in performance to meet hardware constraints. Our method prunes models to 70% sparsity while maintaining reasonable perplexity. Moreover, we demonstrate that the perplexity gap can be significantly narrowed through very short time of fine-tuning. Notably, our approach can also be applied to SVD and PEFT. We believe our method holds promise for integration with future approaches, as long as they allow for layer-wise importance.\n\n>Q5: How your results compare to EvoPress: https://arxiv.org/pdf/2410.14649?\n\nThank you for your valuable suggestion. Following your advice, we reproduce Evopress and carefully align the relevant parameter settings. Notably, Evopress exhibits higher computational complexity and requires significantly longer pruning times. As shown in the table below, we evaluate the accuracy of LLaMA3-8B on eight zero-shot tasks at 70% sparsity. Experimental results clearly demonstrate that our method consistently outperforms Evopress. We include a detailed discussion of this baseline and its results in the revised version.\n\n\n|  Method  | BoolQ |  RTE  | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA |  PiQA |    Mean   |\n|:--------:|:-----:|:-----:|:---------:|:----------:|:-----:|:-----:|:----:|:-----:|:---------:|\n| EvoPress | 62.97 | 52.71 |   45.09   |    58.80    | 40.19 | 28.58 | 27.20 | 62.73 |   47.28   |\n|   **Ours**   | 72.54 | 53.07 |   47.17   |    60.14   | 45.16 | 29.18 | 31.20 | 67.25 | **50.71** |\n\n>Q6: Line 80: DLP reduces the complexity of LLaMA2-7B by 7.79 - what is complexity here?\n\nThank you for pointing out this issue. It actually refers to perplexity, and we will correct it in the final version.\n\n\n>Q7: What is \"m\" in Algorithm 1 line 13?\n\nThank you for your valuable comment. \"m\" represents the mean importance value. To control the overall sparsity while allowing sparsity of each layer to vary according to its importance, we introduce \"m\" . Its purpose is to centralize the sparsity adjustment, ensuring that the average change in sparsity across layers is zero, thereby maintaining the overall sparsity rate. We will make improvements in the final version to enhance clarity.", "claims_and_evidence": "Author claims:\n1. Novel method for measuring layer wise importance that does not rely on empirical values or model type.\n2. An effective method for unstructured pruning\n3. Wide integration with existing Sparsification methods and PEFT\n\nSecond contribution is included in the first one. To the best of my understanding, the paper is not introducing new unstructured pruning method, they are reusing existing ones with varying layer sparsification rate for each layer.", "methods_and_evaluation_criteria": "Yes,  the empirical analysis and the datasets are correctly chosen. \n\nAlthough, minor comments:\n1. I don't get the use of LOD metric. I don't understand what it contribute to?\n2. Some of the model are outdated such as llama1, llama2 \n3. No zero-shot results on llama3 models\n4. No results on bigger models f.e. Llama3 70B", "theoretical_claims": "There are no Theoretical Claims in the paper if I didn't miss them.", "experimental_designs_or_analyses": "Experiments are done thoroughly and there are a lot of empirical analysis of the results. In the paper  latency, throughput and speed up measurement are included in Table 5. \n\n1. Although in appendix: G author provided the set of alpha hyper-parameter, there is no analysis of this parameter: alpha hyper parameter dependence to metrics. This kind of analyze is done for  hyper parameter M in OWL in Figure 2. \n2. Table 2: Although the it is important to show results at the Magnitude pruning, PPL higher than 20-30+ is indicator that model were broken in some ways. So comparing ppl 7642.99 to 84511 is meaningless.", "relation_to_broader_scientific_literature": "The work is based on Wanda criteria. The method can be used on top of other sparsification techniques.", "essential_references_not_discussed": "The main paper that is missing is EvoPress: https://arxiv.org/pdf/2410.14649. I think the results in there are better than DLP. Technically the paper is from arxiv and it is from October 18, so maybe there is not a requirement to compare with it.", "other_strengths_and_weaknesses": "Additional strength: \n1. Paper is well written and easy to understand, there are a lot of experiments and good discussion about results.\n2. Authors provided the code.\n\nWeakness:\n1. The paper is very empirical without any theoretical justifications. Basically, this is some criteria that works empirically.\n2. Results are not groundbreaking, the improvement although not small at the 70% sparsity level, there are not useful in practice. \nFor example, improvement from 30 to 22 ppl seems massive, until you start paying attention to the dense model which is 5.4 (See table 3).\nAt the rate of less than 60% the method is not better than existing ones, please see table 9.", "other_comments_or_suggestions": "1.Table 2: Although it is important to show results at the Magnitude pruning, PPL higher than 20-30+ is an indicator that model were broken in some ways. So comparing ppl 7642.99 to 84511 is meaningless. Maybe consider moving this this results to appendix and don't discuss it in the main text.", "questions_for_authors": "1. How your results compare to  EvoPress: https://arxiv.org/pdf/2410.14649? \n2. Line 80: DLP reduces the complexity of LLaMA2-7B by 7.79  - what is complexity here?\n3. Line 253: Can you please provide formula for LOD and some explanation why this metric matters?\n4. What is \"m\" in Algorithm 1 line 13?\n5. How you are choosing alpha and if it is hyper-parameter why it is different of having hyperparameter m in OWL?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYucpW2P2z", "reviewer": "Reviewer_zWSE", "summary": "The paper relates to the weight pruning of LLMs and introduces a method to prune individual layers non-homogeneously, i.e. instead of applying a fixed sparsity ratio across all layers, every layer is sparsified according to its importance.\n\nThe method extends, and is compatible with previous importance estimation methods, e.g. Magnitude, SparseGPT, WANDA. The method is presented as a competing method to OWL, and is mainly compared against OWL throughout the paper.\n\nAccording to the authors, the main limitation of OWL is that it relies on a fixed outlier ratio (M), which is problematic as the optimal value of M depends on the model architecture and size, and that even within a given model architecture, they may not be a single optimal value of M for all layers.\n\nThe authors propose to do away with the fixed outlier ratio threshold, and instead propose non-parametric relative importance scores for every layer.\n\nThe method is backed by extensive empirical results.\n\n## update after rebuttal\n\nI think this is a strong paper that relates to a field that matters to the community, thus I would like to keep my Accept rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer zWSE:\n\n\nWe sincerely appreciate your positive assessment of DLP's non-parametric layer importance scoring and the rigor of our experimental design. We are also grateful for your acknowledgment of the paper’s detailed literature review and its clear presentation, which we hope will facilitate broader discussions on layerwise pruning strategies. In the following, we will respond to your comments point by point.\n\n\n>Q1: It would be nice to compare the method with recent works on pruning through distillation (https://arxiv.org/abs/2407.14679) and flexible models (https://arxiv.org/abs/2406.10260).\n\n\nWe sincerely appreciate the valuable suggestions from the reviewer. Following your recommendation, we have carefully read the two papers you mentioned. \n\n[[1](https://arxiv.org/abs/2407.14679)] considers multiple dimensions, including depth, width, attention, and MLPs, to perform pruning. It then applies knowledge distillation to retrain on a relatively small dataset, resulting in a powerful and compact model. The paper offers a comprehensive set of best practices for LLM compression and retraining, providing insightful perspectives on practical model pruning. \nFLEXTRON [[2](https://arxiv.org/abs/2406.10260)] dynamically adapts to different latency and accuracy targets during inference without requiring additional fine-tuning. It incorporates input-adaptive routing to maximize performance, significantly reducing training costs. \n\n\nWe find both papers highly insightful and valuable. Our paper primarily focuses on non-uniform layerwise pruning strategies, which represent a different contribution compared to these two papers. Since the implementations of these methods are currently not publicly available, we are unfortunately unable to reproduce their experimental results or perform direct comparisons within the limited rebuttal timeframe. We will include citations of these excellent works and discuss their relevance in the revised manuscript. We also look forward to exploring potential integrations with such advanced methods in future research.\n\n\n>Q2: Typos: line 50: Additionally, we inherently places more emphasis on outliers... (grammar);line 200: Howerver, M is ... (typo);line 265: As shown in Figure 1, OWL and DLP exhibit similar overall trends. (this is in fact figure 3)\n\n\nThank you very much for carefully pointing out these typos. Your suggestions are very helpful for improving the clarity and accuracy of our manuscript. We will correct these errors accordingly in the revised version.\n\n\n>Q3: Can you clarify something about equation 5? In my understanding $\\mathbf{A}_ {i j}^l$ is a scalar, yet the F function is meant to operate on a set of values. In practice are you taking the median value of $\\mathbf{A}_{i j}^l$ over all values of $i$ and $j$?\n\nYes, our method is consistent with what you described. For each layer $l$, we aggregate all $\\mathbf{A}_ {i j}^l$ values of Linear types into a one-dimensional tensor, compute the median, and then determine the unimportance. Thank you for your question. We will make improvements in the final version to enhance clarity.", "claims_and_evidence": "The claim about OWL's limitation (due to the fixed M value) is justified in Figure 2, which shows that the optimal value of M is highly dependent on the model architecture.\n\nThe paper also shows that DLP outperforms OWL according to the LOD metric, despite this being the main optimization criterion of OWL.\n\nAn empirical study justifies the particular choice of the function F used in Equation 5, and shows that the median value yields the best results.", "methods_and_evaluation_criteria": "The evaluation of the method is quite extensive and applies the method on top of Magnitude, SparseGPT and WANDA pruning, for a variety of models (LLaMA1, LLaMA2, LLaMA3, Vicuna, Mistral, Qwen) and a variety of tasks (language modeling, zero-shot accuracy), and systematically compares results against the uniform baseline, as well as the OWL method.", "theoretical_claims": "There is no theoretical claim being made here, as the paper is mainly an empirical study of the proposed method.", "experimental_designs_or_analyses": "As discussed above, the experimental setup is very sound.", "relation_to_broader_scientific_literature": "The paper provides a lot of interesting insights. The findings from Figure 3 align with other recent works which also show that the first and last layers in an LLM are the hardest to prune.", "essential_references_not_discussed": "It would be nice to compare the method with recent works on pruning through distillation (https://arxiv.org/abs/2407.14679) and flexible models (https://arxiv.org/abs/2406.10260).", "other_strengths_and_weaknesses": "The introduction section of the paper presents a detailed overview of the literature, and the style of the paper makes it an enjoyable read.", "other_comments_or_suggestions": "line 50: Additionally, we inherently places more emphasis on outliers...  (grammar)\n\nline 200: Howerver, M is ... (typo)\n\nline 265: As shown in Figure 1, OWL and DLP exhibit similar overall trends.  (this is in fact figure 3)", "questions_for_authors": "Can you clarify something about equation 5? In my understanding $A_{ij}^l$ is a scalar, yet the $F$ function is meant to operate on a set of values. In practice are you taking the median value of $A_{ij}^l$ over all values of i and j?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper aims to improve LLM pruning via a new layer-wise sparsity ratio inspired by OWL. Unlike OWL, which assigns layer-wise sparsity based on a predefined threshold of outlier ratio, DLP measures layer-wise importance based on the median of outlier ratios. This simple modification not only avoids hyperparameter tuning of OWL, but also leads to stronger results. The effectiveness of DLP is extensively evaluated with structured/unstructured llm pruning, low-rank decomposition, quantization, and PEFT following OwLore.

The paper received mixed reviews, with two reviewers proposing acceptance (score 4), and two reviewers proposing weak rejection (score 2).  

Some of the strengths of the paper include:
- The paper is well-written and relatively easy to follow (yet there is room for improvement, and some suggestions for improvements were made).
- Using median to avoid hyper-parameter tuning is important for OWL-like algorithms. 
- The preliminary study in Figure 2, used to provide a motivation for the paper, was appreciated. 
- Evaluating on a wide range of tasks demonstrates the effectiveness of DLP. Numerous experiments; good discussion.

Some of the weaknesses include:
- The method is a direct extension of previous works and uses median to filter out the outlier values. Hence, the technical novelty is marginal.
- The method is only compared with two model pruning baselines (SparseGPT, Wanda). 
- No theoretical insights or results.
- Improvements do not seem to be useful in industrial practice; the work seems mostly conceptual.
- Some missing discussions (ideas for improvements were provided by reviewers); some missing references (again, provided by reviewers).

I have read the reviews, rebuttals and the ensuing discussion. The authors have done a good job responding to the various questions.

In summary, I recommend acceptance.
Decision: Accept (poster)