Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective
Authors: Weizhong Huang
Yuxin Zhang
Xiawu Zheng
Fei Chao
Rongrong Ji
Venue: ICML 2025 spotlightposter (2025)
Keywords: Large language models
Network Sparsity
Layerwise sparsity
Primary area: deep_learning->large_language_models
TL;DR: We derive the layer-wise sparsity rate of LLMs through a theoretical perspective, which significantly enhances the performance of sparse LLMs.

Abstract: In this paper, we address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective. Specifically, we identify a critical issue of **"reconstruction error explosion"** in existing LLMs sparsification methods. This refers to the cumulative effect of reconstruction errors throughout the sparsification process, where errors from earlier layers propagate and amplify in subsequent layers. As a result, the overall reconstruction error increases significantly, leading to a substantial degradation in model performance. Through theoretical analysis, we derive a simple yet effective approach to layer-wise sparsity allocation that mitigates this issue. Our method uses a monotonically increasing arithmetic progression, reducing the process of determining sparsity rates for multiple layers to the determination of a single common difference hyperparameter. Remarkably, this allows for the optimal layer-wise sparsity rates to be identified with just a few trials. Both our theoretical analysis and experimental results demonstrate that this sparsity allocation scheme is near optimal. Extensive experiments show that our method significantly improves the performance of sparse LLMs across various architectures, outperforming existing layer-wise sparsity methods. Furthermore, it enhances the performance of various compression techniques and is applicable to vision and multimodal models. Notably, our method achieves a reduction of 52.10 in perplexity for the 70% sparse LLaMA2-7B model obtained via Wanda, improves average zero-shot accuracy by 10.50%, and delivers speedups of 2.63$\times$ and 2.23$\times$ on CPU and GPU, respectively. Code is available at https://github.com/wzhuang-xmu/ATP.

Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective
Weizhong Huang' Yuxin Zhang' Xiawu Zheng
123
Fei Chao
Rongrong Ji 122
Abstract
ers speedups of 2.63x and 2.23x on CPU and GPU, respectively. Code is available at https:
In this paper, address the challenge of de- termining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective. Specifically, we identify a critical issue of "reconstruction error explosion" in ex- isting LLMs sparsification methods. This refers to the cumulative effect of reconstruction errors throughout the sparsification process, where er- rors from earlier layers propagate and amplify in subsequent layers. As a result, the overall recon- struction error increases significantly, leading to a substantial degradation in model performance. Through theoretical analysis, we derive simple yet effective approach to layer-wise sparsity allo- cation that mitigates this issue. Our method uses monotonically increasing arithmetic progres- sion, reducing the process of determining sparsity rates for multiple layers to the determination of a single common difference hyperparameter. Re- markably, this allows for the optimal layer-wise sparsity rates be identified with just a few trials. Both theoretical analysis and experimental results demonstrate that this sparsity allocation scheme is near optimal. Extensive experiments show that our method significantly improves the performance of sparse LLMs across various ar- chitectures, outperforming existing layer-wise sparsity methods. Furthermore, it enhances the performance of various compression techniques and is applicable to vision and multimodal mod- els. Notably, our method achieves reduction of 52.10 in perplexity for the 70% sparse LLaMA2- 7B model obtained via Wanda, improves aver- age zero-shot accuracy by 10.50% and deliv-
/ / //githb.. hoom/whuang--a XTT/ATP.
1. Introduction
Large Language Models (LLMs) have demonstrated out- standing capabilities in various natural language processing tasks (Meta, 2024; Yang al., 2024; Liu al., 2024a). However, their vast number of parameters and high com- putational demands present significant challenges to model deployment, impeding further applications (Zhu et al., 2024; Wang et al., 2024). Network sparsity (Rao et al., 2021; Paul al., 2022; Huang al., 2025b) methods remove less important parameters from LLMs, enabling model compres- sion without sacrificing performance. This can reduce the model's memory footprint and computational complexity (Li et al., 2024a; An et al., 2024). Existing sparsity methods for LLMs, such as SparseGPT (Frantar & Alistarh, 2023) and Wanda (Sun et al., 2023), adopt a post-training approach which prune all weights in one-shot and can obtaín sparse LLMs without the need for additional fine-tuning.
However, these sparsity methods set a uniform layer- layer-wise wise sparsity rate for different layers, without considering the varying importance of each layer, which harms the accuracy of sparse L.MMs. To address the above issue, many studies have proposed various methods to determine the layer-wise sparsity rate of LLMs. Based on their methodological de- signs, we categorize these methods into two main groups:
Metric based methods. These methods determine the im- portance of each layer of LLMs through hand-crafted met- rics, thereby obtaining the sparsity rate of cach layer. For ex- ample, OWL (Yin et al., 2023) proposes an outlier weighted layer importance metric. By setting the sparsity rate to be inversely proportional to the outlier ratio, it effectively pro- tects the layers with higher ratio of outliers. AlphaPruning (Lu et al., 2024) utilizes the heavy-tailed self- regularization theory (Martin & Mahoney, 2019), especially the shape of the empirical spectral density (Martin et al., 2021) of the weight matrix, to determine the importance of each layer of LLMs. ALS (Li et al., 2024c) proposes an importance met- ric based on mutual information (Tschannen al., 2019),
'Key Laboratory of Multimedia Trusted Perception and Effi- cient Computing. Ministry of Education of China, Xiamen Uni- versity, 361005, P.R. China. "Institute of Artificial Intelligence, Xiamen University. 'Png Cheng Laboratory, Shenzhen, China. Correspondence to: Rongrong Ji <riji@ xmu.edu.en.>
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Metric based
14.
Uniform
OWL
Head
Metric values [m,
Sparsity rakes
Heuristics
Head
Reconstruction Error
DSA
AlphaPruning
m.]
[5 Sg Sp.
Search based
L-1
L-1
1..
Iterativee search
Sparsity rates
Time-consuming
82.... St]
Theoretical perspective
Error analysis
Sparsity reles
Bearslustenn fro
Embedding
Embedding
Layer Index
Dense LLM
20
33
10
23
30
Sparse LLM
Figure 1. (Left) shows the comparison of reconstruction error among different layer-wise sparsity methods. All methods face the prohlem of "reconstruction error explosion"; however, our method achieves lower reconstruction error compared to other methods, (Right) presents comparison between method and other layer- wise sparsity methods. The metrie- method calculates the importance of cach layer to obtain the sparsity rate. However, this method is heuristically designed by human experts and is not optimal. And the search-based method requires large number of iterative searches, which is time-consuming. In contrast, we analyze the causes of "reconstruction error explosion" from a theoretical perspective, and deduce theoretically that using a monotonically increasing arithmetic progression determine the layer-wise sparsity rate can alleviate the problem of "reconstruction error explosion".
and sets a higher sparsity rate for layers with higher mutual information. Although these metrics have proven their effec- tiveness experimentally, manually designing metric requires extensive validation and complex calculations are needed to obtain the sparsity rate of each layer. Most importantly, most of these methods lack theoretical analysis, making it impossible to ensure that the solutions obtained are optimal.
layer, but also leads to an increase in the reconstruction errors of all subsequent layers. As the network propagates forward, the reconstruction errors accumulate, causing the total reconstruction error to grow significantly, thus causing "reconstruction error explosion" (See the left in Figure 1),
Through the above theoretical analysis, we provide simple yet effective rule for determining the layer-wise sparsity rates of LLMs: the sparsity rate should be lower in ear- lier layers, and the layer-wise sparsity rales should follow increasing pattern. This approach effectively alleviates the issue of "reconstruction error explosion", resulting in a well vell-performing sparse LLM. To achieve this, we use a monotonically increasing arithmetic progression to deter- mine the sparsity rates for all layers of LLMs and employ grid search to find the common difference of the arithmetic progression. Since the range of valid values for the com- mon difference is narrow, our search is highly efficient, and after only few attempts, we can determine the common difference that yields the best accuracy.
Search based methods. In addition to these heuristics designed by humans, recently, there have also been some methods that adopt ceacch-based approach to determine the waie sparsity rate of LLMs. For example, DSA (Li et al., 2024b) develops an expression discovery framework to explore potential sparsity rate allocation strategies and obtains layer-wise sparsity allocation function by search- ing. However, the evolutionary search method employed by DSA requires 2000 iterations. For large-scale LLMs with a vast number of parameters, this demands a search process lasting several days, which incurs significant cost. In addition, in order to obtain the final allocation function, DSA designs complex search space and process, which means the effectiveness of the method heavily depends on the experience of human experts.
Furthermore, we prove that the total reconstruction error obtained from the monotonically increasing sparsity scheme is less than that of any non-monotonically increasing spar- sity scheme. This indicates that our method is theoretically close to the optimal solution. Additionally, we compare our sparsity rate scheme with the optimal solution obtained through Bayesian search, we find that our scheme is close to the optimal solution found by search. This indicates that our method is empirically close to the optimal solution.
In this paper, we rethink the approach of determining the layer-wise sparsity rate of LLMs, and derive the layer-wise sparsity rate of LLMs from reconstruction error perspective. Specifically, we first prove that increasing the sparsity rate leads to an increase in the reconstruction error of the cor- responding layer. Additionally, we show that an increase in the reconstruction error of one layer causes an increase in the reconstruction error of subsequent layers. This im- plies that increasing the sparsity rate of earlier layers not only increases the reconstruction error of the corresponding
To evaluate the effectiveness of our ATP method, we con-
Deterrmining layer-wise sparsity for LLMs through Theoretical Perspective (ATP).
2

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
duct extensive experiments on LLMs of various architec- tures, with parameter counts ranging from 6.7 billion to 70 billion. Our evaluation metrics include perplexity, average accuracy across seven zero-shot datasets, and performance on arithmetic and knowledge reasoning tasks. Our ATP method demonstrate substantial improvements over existing post-training sparsity techniques, significantly surpassing other layer-wise sparsity methods. Notably, ATP reduce the perplexity of the 70% sparse LLaMA2-7B pruned using Wanda (Sun et al., 2023) by 52.10 and increase average zero- shot accuracy by 10.50%, outperforming the state-of-the-ar AiphaPruning method (Lu et al., 2024) by 6.71 and 2.46%, respectively. Additionally, ATP achieved 2.63 x and x speedups on CPU and GPU, respectively, and require only 18 minutes to compute layer-wise sparsity rates. Further- more, we evaluate ATP's enhancements on various compres- sion techniques, including N:M sparsity, structured pruning, and network quantization, as well as its benefits for sparse multimodal and sparse vision models. These experimental results clearly demonstrate that ATP provides substantial performance improvements for compressed models.
DSA (Li et al.. 2024b) develops an expression discovery algorithm to explore potential sparsity allocation. However, the above metric and search based methods all lack theoreti- cal proof of effectiveness and require complex calculations or search to obtain layer-wise sparsity rate. In comparison, our method directly derives the layer-wise sparsity rates based on a monotonically increasing arithmetic progres- sion from the perspective of reconstruction error, requiring only the determination of the common difference to quickly obtain the sparsity rate for each layer. More importantly, we have validated the effectiveness of the above method through rigorous proof.
Reconstruction error is metric for measuring the difference between the output of a com- pressed network and that of the original network. A smaller reconstruction error generally implies that the compressed network can better preserve the performance of the original network (Yun & Wong. 2021; Hubara al., 2021; Ma et al., 2023b). In order to minimize the reconstruction error of sparse LLMs, SparseGPT (Frantar & Alistarh, 2023) pro- poses a mask selection and weight update algorithm based on Hessian inverse and DSno'T (Zhang et al., 2023) proposes a training-fiee fareingg-reeeaee dynamic weight pruning and growing algo- rithm. In this paper, we explore the "reconstruction error explosion" problem in sparse LLMs. Specifically, the re- construction error accumulates and magnifies across layers, leading to extremely large overall reconstruction error, which undermines the model's 's accuracy. Therefore, we pro- pose our layer-wise sparsity allocation method to alleviate the above "reconstruction error explosion" problem.
Reconstruction Error.
2. Related Work
LLMs Sparsity. Before the advent of LLM, a variety of sparsity techniques had been developed to compress models such as ResNet (Yu et al., 2022b; Zhang et al., 2024). BERT (Xia et al., 2022; Li et al., 2023), and ViT (Yu et al., 2022a; He et al., 2024). Meanwhile, researchers have developed several post- training sparsity methods specifically for LLMs. For example, SparseGPT (Frantar & Alistarh, 2023) uses the inverse of the Hessian matrix for pruning and pruned weight updates. Wanda (Sun et al., 2023) uses a metric that com- bines weight magnitude and input activation to prune LLMs, while Pruner-zero (Dong et al., 2024) searches for symbolic pruning metric using genetic programming. Additionally, ALPS (Meng et al.. 2024) uses an Alternating Direction Method of Multiplier (ADMM) (Boyd et al., 201 2011) baaed approach to prune LLMs in one-shot. The above methods focus on determining the mask within the layer of I.I.S and setting a uniform layer-wise sparsity rate. Our work study the layer-wist wise sparsity allocation problem in sparse LLMs from the perspective of reconstruction error, thereby effectively improving the accuracy of above methods.
3. Methodology
Notation. In this paper, we use bold typeface indicates matrices (e.g., W, X) and calligraphic font represents loss functions or models (c.
3.1. Preliminaries
Without loss of generality, we take each layer in the LLMs as the basic unit of analysis. These layers contain modules such as Attention (Vaswani et al., 2017), MLP (Popescu et al., 2009), LayerNorm (Lei Ba et al., 2016), and residual connections (He et al., 2016), etc. We represent a layer's ' computation
X, where W is the layer's weight and X is the input. Consider an LLM composed of L layers, we define the reconstruction error of the i-th layer (i 1, 2.2..... L) as follows:
Layer-wise Sparsity. Layer-wise sparsity rate determines the number of weights be retained in each layer of the network (Lee et al., 2020; Frankle et al., 2020; Liu et al., 2022a; Huang et al., 2025a). To determine the layer-wise sparsity rate in LLMs, OWL(Yin et al., 2023) proposes an outieer-wiighed metric. Alphapruning (Lu et al., 2024) uses heavy-tailed self- regularaaati theory (Martin& & Mahoney, 2019), ALS (Li et al., 2024c) proposes a layer redundancy metric based on mutual information (Kraskov et al., 2004),
C(W xi) = W.X W X
(I)
F
where W, E R and X; R xd are the weights and input of the i-th layer respectively. W, and X; are the
3

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
corresponding sparse versions, where Co, and Cout represent the number of input and output feature dimensions, d is the hidden dimension, and 1- is Frobenius norm.
inal weight matrix W obtain less sparse version W. (1)
The second term, (W, (1) W X quantifies
the additional error resulting from further sparsifying W; (1)
Existing post-training sparsity methods all attempt to minimize the reconstruction error of sparse LLMs (e.g. SparseGPT and Wanda) and large amount of experimental evidence in the these papers indicates that a sparse LLM with good accuracy often has low reconstruction error.
- (2) Since both er-
to derive an even sparser matrix W;
rors (W; W )X and ( W. ) are gen- erated through the same sparsification method, they point in similar directions within the vector space. This align- ment ensures that their Frobenius inner product satisfies
)X;
In the next section, we reveal that the existing post-training sparsity methods all have the problem of "reconstruction error explosion".
(WW W 8 X,, (W (1)
W. X;)r >
0. And since
(2)
W x; W; X;
> O, therefore:
3.2. "Reconstruction Error Explosion" in Sparse LLMS
( - (2) <0
(4)
First, we analyze the relationship between the sparsity rate and the reconstruction error. Briefly, higher sparsity rate leads to a higher reconstruction error. Formally, we propose Theorem 3.1.
Therefore, the reconstruction error (()) for lower sparsity is smaller than L(2) for higher sparsity. In other words, increasing the sparsity will lead to an increase in the re- construction error of this layer. So, we have completed the proof of the above theorem.
Theorem 3.1 (Effect of increased sparsity on reconstruc- tion error). When the input is the same, increasing the sparsity of the weights in the i-th layer will lead to an in- crease in the reconstruction error of this layer.
On the other hand, we find that there is an cumulative effect of reconstruction error during the sparsification process, It is manifested as the reconstruction error in L.Mss showing an increasing trend due to the influence of previous sparse layers. In other words, if the reconstruction error of the previous layer increases, the reconstruction error of the subsequent layers will also increase accordingly. Formally, propose Theorem 3.2.
Proof of Theorem 3.1. We only consider the effect of sparse weights on the reconstruction error, and we ignore the error caused by the imput at this time. Therefore, we consider the impact of the sparse weights of the i-th layer on the reconstruction error when the input is the same. That is, X; = X, Then, the reconstruction error of the i-th layer expressed as:
Theorem 3.2 (The cumulative effect of reconstruction error). When the reconstruction error of the i-th layer increases, it leads to an increase in the lower bound of the reconstruction error for the (i + 1)-th layer.
C= W.X. W. x.
(2)
Consider two weights W; and W; of different sparsity,
such that W (1) has lower sparsity (i.e., has fewer zero ele-
To prove Theorem 3.2, we need to define the following lemma:
ments). The difference in reconstruction error correspond-
ing to these two sparse weights is:
Lemma 3.3. Let E R.xx and B Raxp be arbitrary matrices. Then, it holds that || f 2 iii [
c0) - 2(2) = W W.X:
where mmin (A) denotes the smallest non-zero singular value of A.
F
2
WiX,- W X; =
W.X; - W: X,
The proof of the Lemma 3.3 can be found in Appendix B. Now we formally prove Theorem 3.2.
F
| (1)
x,) + (W; (1)
X; W. x.)
F
=-2((W, W i
(1)
Xx (W i
(1)
W X()
- (2)
Proof of Theorem 3.2. The reconstruction error for the (i + 1)-th layer can be expressed as:
-|W (1)
X; - W (2)
x;
2 L(W H1, X = W +1 W X ||=
(3)
F
( W11+1 - W + Wit(( Xx+1)
The first term of the inner product, (W,- X represents the error introduced by sparsifying the orig-

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
(Wi+1 i4 W W1+1)
1+1 x++1X++11 X+11
Proof of Theorem 3.4. According to Theorems 3.1 and 3.2, we have the following relationship:
+ - -X
sparsity of layer t C(W i, x;) 1
+ +2r((W i+1 (i+1)
X(1))
C(W (+1 x+1) > min ( ( (+1 (L)) X;) t.
(5)
(9)
Since the third term is the trace of the product of two matri- ces, the first and second term the square of the Frobenius norm, and considering that the weights and inputs in LLMs are matrices with large dimensions, the magnitude of the first and second term is much greater than the third term, we ignore the third term and get:
Therefore, we complete the proof of Theorem 3.4.
To summarize all the above, we can get the following logical chain of our paper: sparsity rate of 1-st layer & reconstruc- tion error of 1-st layer x reconstruction error of L-th layer x total error oc accuracy loss. That is, when the sparsity rate of the 1-st layer increases, it will lead to an increase in the reconstruction error of this layer. As the reconstruc- tion error accumulates continuously from the 1-st layer to the L-th layer, the reconstruction error of the L-th layer also increases accordingly. Then, the reconstruction errors of all layers of the model will exhibit an explosion phe- nomenon, resulting in a serious decline in the accuracy of the sparse model. We refer to the above phenomenon the "reconstruction error explosion". This phenomenon can be observed on the left side of Figure 1.
L(W 1+1, x ~0
( - W Wit1 X
+1
W (X+1((1+3) X 1
+
(Wi+1 ++2 X
2 | |=00 >0. 0. Therefore:
Since
i+1
(W1+,,X++1) W WX+1(X X X
According to Lemma 3.3, we get:
From the above theoretical analysis, we understand that the earlier layers are more important than the later layers. Setting a lower sparsity rate for the previous layers helps alleviate the problem of "reconstruction error explosion". In the next section, will introduce our method of deter- mining the layer-wise sparsity rate in detail.
L(W 41, x+) min ( (t) (X+11-Xxi x
= s+1 (W,X, - W W.X.)
= = (W (WW)) X
Since o W (+1)>> > O, we have proven that the increase of the reconstruction error of the i-th layer will lead to the increase of the lower bound of the reconstruction error of the (i + 1) layer.
3.3. Discussion about the Rationality of Theoretical Modeling
In Sec. 3.1, represent a Transformer layer's computa- tion as WX. The Transformer layer includes components such as Attention, MLP, nonlincarities, and layer normal- ization. Due to the more complex nonlinear calculations in a Transformer layer, there are differences between theo- retical analysis based on WX X and the actual architecture. However, our analysis remains reasonable for the following reasons:
Theorem 3.2 shows that an increase in the reconstruction error of the previous layer in sparse LLM usually leads to further increase in the lower bound of the reconstruction er- ror of the subsequent layer. In practice, this often means that an increase in the reconstruction error of the previous layer will lead to an increase in the reconstruction error of the sub- sequent layer. We have also observed this phenomenon in the left of Figure 1. We can see that when the reconstruction error of the earlier layers is smaller, the reconstruction error of the subsequent layers is also smaller. Conversely, when the reconstruction error of the earlier layers is larger, the reconstruction error of the subsequent layers is also larger.
1. The theoretical modeling of W X is sufficient to an- alyze the layer's reconstruction error. Our method sparsifies the linear layers in Attention and MLP mod- ules, while other components remain unaffected. These linear layers account for the majority of the parameter count and significantly influence the computation re- sults of the layer. The sparsified linear layers dominate the computation of the reconstruction error for each layer. Although various nonlinear operations exist in the actual architecture, they typically do not fundamen- tally alter the reconstruction error of each layer and have minimal impact on theoretical analysis. There- fore, modeling the primary computation of a layer as
According to Theorems 3.1 and 3.2, we can easily get the following Theorem 3.4:
Theorem 3.4 (Impact of the sparsity of the previous layer on the reconstruction error of the next layer.). Increasing the sparsity of the i-th layer will lead to an increase in the lower bound of the reconstruction error of the (i + 1)-th layer.
5

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
WXi is sufficient for analyzing the reconstruction er- ror of that layer. This is also sufficient for us analyze how reconstruction errors accumulate and propagate across the network.
We use grid search (Jiménez et al., 2008) to determine / for sparse LLMss. Specifically, since 0 S S0, 8L S 1 and considering the relatively low sparsity rate set for the ear- lier layers, the arithmetic progression should be increasing. Therefore, we can deduce that the possible range of values for 3 is 0 < 3 S min( 25 -.-1) ) This is small range for a sparse LLMs. For example, for a LLaMA3-8B (Meta, 2024) model with 32 layers and the average sparsity rate is S = 0.7, then the range is 0 < 8 < 0.019. In order to find the optimal value of 3 within the above range, we adopt an grid search method with step size of 0.002. The goal is to find the value of 3 that minimizes perplexity of the sparse LLMs on the WikiText-2 (Merity et al., 2016) dataset. Since the reasonable range of B is very small, this ensures that we can find the optimal [ very quickly. For ex- ample, for 0 < B I 0.019, we only need to make 9 attempts. Even for the largest 70B model, the reasonable range of B is < 3 < 0.0075, and only 3 attempts are required in this case. We present our ATP approach in Algorithm 1.
2. Transformer's linear modeling is supported by ex- isting research. Razzhigaev et al. (Razzhigaev al., 2024) employs Procrustes similarity analysis to dis- cover that the embedding transformations between se- quential layers in LLMs such as GPT, LLaMA, OPT, and BLOOM exhibit a near- pear-perfect linear relationship, with linear score is 0.99. This indicates that despite the non-linear operations within Transformer layers, the mapping between adjacent layers can still be ap- proximated as linear transformation. Therefore, it reasonable and natural to model Transformer layer computations using WX.
3. Modeling the computation of modules as W is common practice in many works. AdaRound (Nagel et al., 2020) and MRECG (Ma et al., 2023b) simplify the computation of the CONV-BatchNorn=RRIT modules in quantized convolutional neural networks as WX when analyzing reconstruction error. This approach of ignoring unnecessary computations and fo- cusing the core computations is common practice, which facilitates the derivation of theoretical results.
Algorithm 1 Using ATP method to obtain sparse LLMs
Input: Dense LLMs Maense: average sparsity rate S Output: Sparse LLMs . spprse
Use grid search to get 3 in Eq. 10;
Determine layer-wise sparsity rate according to Eq. 10; Combine with post-training sparsity method (e.g.. SpaeseGPT or Wanda) to obtain sparsity mask W; Apply W to Matense yields M Msparse.
3.4. Determining Layer-wise Sparsity Rates for LLMs
According to the theorem in Sec. 3.2, the reconstruction er- ror in sparse LLMs have "reconstruction error explosion" problem. Specifically, the error from the earlier layers will cause an increase in the eиror of the later layers. When the error from all earlier layers are accumulated, it will lead to a sharp increase in the error of the later layers. Meanwhile, this can lead to an increase in the total reconstruction er- ror of sparse I.LMs, thereby damaging the final accuracy of the sparse LLMs. Therefore, in order to mitigate the negative impact of the "reconstruction error explosion" of reconstruction eπrors sparse LLMs, we can set the earlier layers to have lower sparsity and the later layers to have higher sparsity. Therefore, we propose to determine the sparsity rate of each layer in L.MSS according to the following monotonically increasing arithmetic progression:
Although the above method of determining the layer-wise wise sparsity rate of LLMs by a monotonically increasing arith- metic progression very simple, our theoretical analysis in Sec. 3.5 fully proves its rationality, and we have fully demonstrated through a large number of experiments in Sec. 4 that our method can effectively improve the accuracy of existing post-training sparsity methods and significantly outperforms current layer-wise sparsity methods.
3.5. Analysis of the Proposed Determining Sparsity Method
We propose the following theorem to prove that the method for determining the layer-wise sparsity rate proposed in Eq. 10 is theoretically close to the optimal solution:
s; - 1( -- (1- 1,2, L (10) 2
Theorem 3.5. The total reconstruction error obtained from the monotonically increasing sparsity scheme proposed in Eq. 10 is strictly less than that obtained from any non- monotonically increusing sparsity scheme.
where s; the sparsity rate (fraction of zero entries) of the i-th layer, L is the total layer number of LLMs, and S is
the average sparsity rate of all layers. ,3 is a hyperparameter that controls the degree of difference in the sparsity rate of each layer of LLMs. The above formula means that we only need to determine the hyperparameter ,3 to get the sparsity rate of each layer of L.MMs.
The proof of the above theorem is detailed in Appendix C.
In addition, we compare the layer-wise sparsity rates deter- mined by Eq. 10 with those obtained by Bayesian search in Sec. 4.6. The experimental results show that our method
6

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Table Comparison the average zero-shot accuracy across tasks for 70% sparse LLMs obtained using various sparsity methods.
Layerwise sparsity
LLaMA
LLaMA2
LLaMA3
Method
7B
13B
30B
65B
7B
13B
70B
8B
Dense
61.74
63.84
67.41
68.57
61.88
65.00
69.14
65.62
Uniform
37.45
40.79
53.35
57.76
35.33
38.88
58.48
35,42
OWL
44.19
48.18
55.88
59.79
41.75
47.19
59.26
35,42
Wanda
DSA
43.72
48.64
55.21
58.24
36.55
43.36
58.25
37.85
AlphaPruning
44.99
49.81
56.67
61.05
43.37
49.11
59.44
35,61
ATP
47.03
51.60
57.69
61.92
45,83
52.11
60.91
41.28
Uniform
43.60
48.00
53.64
60.18
43.07
47.38
60.84
43.02
SparseGPT
OWI.
46.57
50.01
55.73
59.47
46.55
49.90
60.83
46.06
AlphaPruning
46.58
50.63
56.42
60.32
46.20
50.86
61.03
45.27
ATP
47.37
52.12
57.55
61.31
48.63
52.46
62.25
47.79
close to the optimal solution obtained by Bayesian search. Moreover, since our method doesn't 't require long time search, it is highly efficient compared to Bayesian search.
Baselines. We apply the layer-wise sparsity rates deter- mined by ATP to several statc-of- thrtt-r--rtttt post-training sparsi- fication methods, including SparseGPT (Frantar & Alistarh, 2023), Wanda (Sun et al., 2023), DSnoT (Zhang et al., 2023), Pruner-zero (Dong et al., 2024), and ALPS (Meng et al., 2024). Furthermore, we compare ATP with recent methods for determining layer- sparsity rates in LLMs, such as OWL (Yin et al., 2023), AlphaPruning (Lu et al., 2024), DSA (Li et al., 2024b), and ALS (Li et al., 2024c).
All all, the above theoretical analysis and experimental validation demonstrate the superiority of our method, indi- cating that the sparsity allocation scheme proposed in Eq. 10 is near-optimal both theoretically and empirically.
4. Experiments
More Models, Evaluations and Baselines. In Sec. F, we present additional experimental results, including the appli- cation of ATP to multimodal and vision models, integration with other compression techniques, LoRA fine-tuning and sparsity-preservie PEFT methods, and comparisons with an expanded set layer-wise sparsity baselines.
4.1. Experimental Setup
Models. We evaluate ATP across a diverse range of widely-used L.Mss, including ILaMA-1 (7B, 13B, 30B and 65B) (Touvron et al., 2023a), LLaMA-2 (7B, 13B and 70B) (Touvron et al., 2023b), LLaMA-3-8B (Meta, 2024), LLaMA-3.1-8B (Meta, 2024), LLaMA-3.2-1B and 3B (Meta, 2024), OPT-13B (Zhang et al., 2022), Vicuna- 13B (Chiang et al., 2023), Qwen2.5-7B (Yang et al., 2024), and Mistral-7B (Jiang et al., 2023) and Mixtral-8x7B (Jiang et al., 2024).
Implementation Details. Our pruning implementation builds upon the methods used by SparseGPT and Wanda, with the primary modification being the integration layer- wise sparsity rates generated by ATP.
Evaluation. Our evaluation protocol aligns with estab- lished sparsification methods for LLMs (e.g., SparseGPT (Frantar& & Alistarh, 2023) and Wanda (Sun et al., 2023)), encompassing both zero-shot learning and language model- ing capabilities. Specifically, we assess the perplexity of the models on the validation set of WikiText-2 (Merity et al., 2016) and evaluate zero-shot performance on seven down- stream tasks: BoolQ (Clark al., 2019), ARC Easy and Challenge (Clark al., 2018), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), OpenbookQA (Mihaylov et al., 2018) 8), and PIQA (Bisk et al., 2020). Addi- tionally. we measure performance on arithmetic and knowl- edge reasoning benchmarks, including 8-shot accuracy on the GSM8K dataset (Cobbe et al., 2021) and 5 5-shot accuracy on the MMLU dataset (Hendrycks et al., 2020).
4.2. Zero-shot Tasks
Quantitative Evaluation. We report the average perfor- mance of 70% sparse LLMs across seven zero-shot tasks in Table 1. The results demonstrate that our ATP method con- sistently improves accuracy compared to the uniform spar- sity baseline and significantly outperforms other ot the-art layer-wise sparsity methods. For instance, with the LLLMM338P model pruned using the Wanda method, ATP achieves a 3.43% higher accuracy than the best-performing DSA method, highlighting the effectiveness and superiority of ATP in enhancing sparse LLM performance.
Varying Sparsity Rates. We also evaluate the perfor- mance of sparse LLMSs under reduced sparsity constraints.
7

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Specifically, Table presents the zero-shot accuracy of LLMs pruned using the Wanda method 50% sparsity rate. Even under this lower sparsity setting, ATP demon- strates substantial improvements in accuracy across all mod- els, maintaining its performance advantage over existing layer-wise sparsity methods. This indicates that ATP is robust and effective optimizing LLM accuracy under varying sparsity rates.
tivated Mixture of Experts (MoE) models. We report the experimental results in Table 4. The results demonstrate that our ATP method consistently enhances the performance of LLMs across various architectures, further demonstrating its generalizability across different model architectures.
Table 4. The performance improvement ATP on sparse LLMs
acTOSs wider range of architectures.
Table Average zero-shot accuracy of sparse LLaMA2-7B/L3B models pruned using the Wanda method at 50% sparsity rate.
Method
Model
Perplexity (U)
Accuracy (f)
Dense
LLaMA3.1-88
6.18
65.93
Method
1-7B
1-13B
2-7B
2-13B
Wanda
1...MA31-88
109.99
36. 10
w. ATP
L.a.MA31--8
72.05
41.59
Dense
66.09
68.18
66.69
69.25
Dense
LLaMA3.2-1B
9.65
52.82
Uniform
60.06
65.22
63.03
65.73
SparseGPT
LLaMA3.2-1B
129.24
37.26
OWL
61.92
65.91
63.88
68.02
W. ATP
LLaMA3,2-1B
94.78
38.86
DSA
61.90
65.40
63.89
67.65
Dense
LLaMA3.2-3B
7.73
60.42
ALS
61.59
65.05
64.12
67.11
SparseGPT
LLaMA3.2-B8
65.97
39.50
AlphaPruning
61.91
66.19
64.20
67.80
ATP
LLaMA3.2-3B
46.64
42.51
ATP
62.72
66.39
64.49
68.18
Dense
OPT-
10.13
55,22
Wanda
OPT-13B
73.70
41.51
The zero-shot evaluation setting used here fol- lows the configuration outlined in the ALS paper. For details, refer Sec. D.
w.
OPT-13B
33.98
44.58
Dense
Vicuna-
5.94
65.53
Wanda
Vicuna- 13B
44.89
42.06
w. ATP
Vicuna- 13B
20.24
53.19
4.3. Language Modeling
Dense
Qwen2.55 2-222.5-7
6.77
65.36
Wanda
Qwen2.5-
75.16
41.10
We present the perplexity of 50% to 80% sparse LLaMA-7B and LLaMA2-7B models pruned using the Wanda method on the WikiText-2 dataset in Table 3. The results show that our ATP method achieves lower perplexity compared other layer-wise sparsity methods. Furthermore, this ad- vantage becomes increasingly significant at higher sparsity rates.
ATP
Qwen2.5-7B
43.28
42.82
Dense
Mistral-7B
5.28
66.17
Wanda
Mistral-7B
57.44
37.62
ATP
Mistral-7B
29.19
42.76
Dense Wanda
Mixtral-8x7B
3.86
69.35
Mixtral-xx7B
18.22
48.36
w. ATP
Mixtral-8x7B
14.30
50.67
Table 3. WikiText-2 perplexity of sparse LLaMA-7B/2-7B ob- tained by Wanda across varying sparsity rates.
4.5. More Results
LLaMA-7B
LLaMA2-7B
Method
50%
70%
80%
50%
60%
70%
80%
We provide more experimental results in the appendix. Specifically, in Sec. F.1, we demonstrate the enhanced per- formance of our ATP method arithmetic and knowledge reasoning tasks for sparse LLMs. Additionally, in Secs. F.2 and F.3, we present the performance gains of ATP on sparse multimodal and vision models, respectively. In Sec. F.4, we highlight the performance improvements when integrat- ing ATP with other compression techniques, including N:M sparsity, structured pruning, and quantization. Furthermore. we compare ATP with additional layer-wise sparsity base- lines (Sec. F.5), demonstrate its performance enhancements across various post-training sparsity methods (Sec. F.6), and showcase its effectiveness when combined with LoRA fine-tuning menitunnng for sparse LLMs (Sec. F.7). Furthermore, we show the zero-shot accuracy of LLMs at 60% sparsity in Sec. F.8.
Unilfor
7.26
10.63
84.52
5889.13
6.92
10.96
74.26
1980.85
OWL
24.56
629,99
7.17
24.52
1232.88
1638.81
AlphaPruning
7.18
9.47
23.86
698.56
6.88
9.78
28.87
1672.49
ATP
7,05
9.25
20.16
176.80
6.82
9.15
22.16
425.12
4.4. More LLM Architectures
We validate the effectiveness of our ATP method LLMs with broader range of architectures. Specifically, use ATP method to obtain 70% sparse models, including LLaMA3.1 1-BB, LLaMA3.2- 1-BB. OPT-13B, Vicuna-1333 Qwen2.5-7B. Mistral-7B, and Mixtral-8x77 -x77B These models cover a wide range of parameter sizes, from IB to 46.7B, including both ILaMA-like architectures and sparsely ac-
8

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
4.6. Ablation Study
Inference Speedup. We evaluate the acceleration perfor- mance of the sparse LLaMA2-7B model, with results sum- marized in Table 7. The end-to-end token generation time was measured using the DeepSparse (NeuralMagic, 2021) inference engine on an Intel Xeon Silver 4314 CPU and the nm-vllm (NeuralMagic, 2024) inference engine on an NVIDIA RTX 4090 GPU. Our method achieves significant speedups, ranging from 1.79x to 2.63x on the CPU and x.71 to 2.23 x on the GPU, compared to the dense model, at sparsity rates between 50% and 70%. In addition, there is discussion about which devices can be used with the DeepSparse and nm-vllm inference engines to deploy sparse LLMs for inference acceleration, please refer to Sec. E.
Searching Step. In Sec. 3.4, we perform grid search with step size of 0.002 to determine the optimal value of 3. Here, we analyze the impact of different step sizes the search results. As shown in Table 5, searches conducted with larger step sizes yield inferior results compared to those with a step size of 0.002. This is because larger step sizes fail to sufficiently explore the possible optimal values of B. Conversely, further reducing the step size for more fine- grained search shows limited improvement in perplexity. Therefore, to balance both accuracy and efficiency, we adopt grid search with a step size of 0.002.
Table The impact different step sizes on search results.
Table 7, End-to-ent inference accelcration of sparse LLaMA2-77 CPU and GPU,
Step size
0.0005
0.001
0.002
0.004
0.008
Perplexity (1)
22.14
22.16
22.16
23.09
23.09
Device
Sparsity
Dense
50%
60%
70%
Search Time (min)
76
38
18
8
4
Throughput
3.40
6.10
7.39
8,95
CPU
(tokens/s)
Speedup
1.00x
1.79x
2.17x
2.63x
Comparison with Bayesian Search. We compare the ATP method with layer-wise sparsity rates obtained through Bayesian search. Specifically, we use Bayesian search to determine the sparsity rates for the 50%. 60%, and 70% sparse LLaMA2-7B model. The search is conducted using the Optuna hyperparameter optimization framework (Ak- iba et al., 2019), performing 1000 iterations to optimize the sparsity rates across all 32 layers of LLaMA2-7B. The sparsity rate for each layer is constrained between 0 and 1, while ensuring that the average sparsity matches the target rates. We integrate Bayesian search with the Wanda method, with the objective of minimizing model perplexity on the WikiText-2 dataset. The comparison of perplexity and zero- shot accuracy between sparse I.LMs obtained using ATP and Bayesian search is presented in Table 6. The results indi- cate that the performance of our ATP method is comparable to the optimal solution obtained through Bayesian search, demonstrating that the layer-wise sparsity rates determined by our method are experimentally close to the optimal val- ues identified by the search approach. However, Bayesian search requires approximately 33 hours to complete on single NVIDIA A100 80GB GPU, whereas ATP only takes 18 minutes, demonstrating significantly higher efficiency.
Throughput (tokens/s)
57.29
97.92
111.86
127.67
GPU
Speedup
1.00x
1.71x
1.95x
2.23
More Ablation Study. We provide more ablation results in the appendix. Specifically, in Sec. G.I, we demonstrate the computational efficiency of our ATP method. In Sec. G.2, we analyze the layer-wise sparsity distribution gen- erated by ATP at different sparsity rates. In addition, we compare the sparsity rate distributions produced by ATP and other layer-wise sparsity methods in Sec. G.3. In Sec. G.4, we explore the impact of different B settings on the perplexity of sparse LLMs. Furthermore, we demonstrate the robustness of ATP under different random seeds in Sec. G.5.
5. Conclusion
In this paper, we propose a theoretically grounded method for determining layer-wise sparsity rates in LLMs,, effec- tively addressing the challenge of "reconstruction error explosion". Our approach utilizes an arithmetic progression to streamline sparsity allocation, reducing the complexity to a single hyperparameter while achieving near-optimal performance with minimal tuning. We have demonstrated through theoretical analysis and experimental validation that our sparsity allocation scheme is close to the optimal solu- tion. Extensive experiments demonstrate the effectiveness of our method, yielding significant improvements in model perplexity, accuracy, and inference speed. Furthermore, our approach exhibits strong generalization across diverse ar- chitectures and modalities, establishing it as a versatile and robust solution for optimizing compressed models.
Table 6. Comparison of ATP and Bayesian search.
Method
Sparsity
Perplexity (L)
Accuracy (t)
Bayesian Search
50%
6.81
59.67
ATP
50%
6.82
59.63
Bayesian Search
60%
9.20
54.73
ATP
60%
9.15
54.79
Bayesian Search
70%
22.10
45.90
ATP
70%
22.16
45.83

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Acknowledgments
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 11.1110..44666 2021.
This work was supported by the National Science Fund for Distinguished Young Scholars (No. NNo.62055603) the Na- tional Natural Science Foundation of China (No. U21B2037,
No. U22B2051, No. U23A20383, No. 62176222, No. 62176223, No. 62176226, No. 62072386, No. 62072387, No. 62072389, No. 62002305 and No. 62272401), and the Natural Science Foundation of Fujian Province of China (No. 2021J06003, No.2022106001).
Cunegatti, E.,, Custode, I.. I.., and Iacca, G. Zeroth-order adaptive neuron alignment based pruning without re- training. arXiv preprint arXiv:241 1,07066 2024.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Ieee. 2009.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Dong, P., Li, I.., Tang, Z.,, Liu, X., Pan, X., Wang, Q., and Chu, X. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. arXiv preprint 0tl::40060022444 2024.
References
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021. URL https://
Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 2623-2631 2019.
openreview net / wwtp:eeeeee/teeeeeeeeeeeee DorrumiddyYccbaaae
An, Y., Zhao, X., Yu, T., Tang, M., and Wang, J. Fluctuation- based adaptive structured pruning for large language mod- els. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10865-10873, 2024.
Frankle, and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint brXiv:1803.03633 2018.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Frankle, J., Dziugaite, G. K., Roy, D. M., and Carbin, M. Pruning neural networks at initialization: Why missing the mark? arXiv preprint 2xXiv:200.008576 2020.
Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J., al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning. 3(1):1-122, 2011.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Inter- national Conference on Machine Learning, pp. 10323- 10337. PMLR, 2023.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, I.,, Zhuang, S., Zhuang, Y., Gonzalez, J. F.,, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. (atp://gg2222--33---vvvve- 3(5), 2023.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L.,, Hsu, J., McDonell, K., Muennighoff, N., al. A framework for few-shot language model evaluation. Version v0. 0.I. Sept, 2021.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.1004 2019.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 150300557, 2018.
He, H., Cai, J., Liu, J., Pan, Z., Zhang, J., Tao, D., and Zhuang, B. Pruning self-attentions into convolutional lay- ers in single path. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
10

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Li, G., Tang, Y., and Zhang, W. Lorap: Transformer sub- layers deserve differentiated structured compression for large language models. arXiv preprint arXiv:2404.09695, 2024a.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint 0rXiv:2009.03300 2020.
Li, L., Dong, P., Tang, Z., Liu, X., Wang, Q., Luo, W., Xue, W., Liu, Q.. Chu, X., and Guo, Y. Discovering sparsity allocation for layer-wise pruning of large language mod- els. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Hu, E. J., Shen, Y. Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 (06,08,,,,,, 2021.
Li, W., Li, L., Lee, M. G., and Sun, S. Adaptive layer spar- sity for large language models via activation correlation assessment. In The Tiirtt-eighth Annual Conference on Neural Information Processing Systems, 2024c.
Huang, W. Zhang, Y., Zheng, X., Chao, F., and Ji, R. To- wards efficient automatic self self-pruning of large language models. arXiv preprint arXiv:2502. 14413, 2025a.
Li, Y., Yu, Y., Zhang, Q. Liang, C., He, P., Chen, W., and Zhao, T. Losparse: Structured compression of large lan- guage models based on low-rank and sparse approxima- tion. In International Conference on Machine Learning, pp. 20336-20350. PMLR, 2023.
Huang, W., Zhang, Y., Zheng, X., Liuyang, Lin, J., Yao, Y., and Ji, R. Dynamic low-rank sparse adaptation for large language models. In The Thirteenth International Confer- ence on Learning Representations, 2025b. URL https: //openreview / wtr/m//rummmm wwwe:ddd==xhhhh3h00
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:4412.19437, 2024a.
Hubara, I., Chmiel, B., Island, M., Banner, R., Naor, J., and Soudry, D. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in neural information processing systems, 34: 2 21099-211 21111, 2021.
Liu, H., Li, Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (0296-26306, 2024b.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXivv:3 10.3000000... 2023.
Liu, S., Chen, T., Chen, X., Shen, L.,, Mocanu, D. C., Wang, Z., and Pechenizkiy, M. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training, arXiv preprint arXiv: 22420043, 2022a.
Liu, Z., Mao, H., Wu, C.-Y,, Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference
Jing, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Computer Vision and Pattern Recognition (CVPR), pp. 11976-11986, June 2022b.
Lu, H., Zhou, Y., Liu, S., Wang, Z., Mahoney, M. W., and Yang. Y. Alphapruning: Using heavy-tailed self reg- ularization theory for improved layer-wise pruning of large language models. arXiv preprint arXiv:2410.10912. 2024.
Jiménez, Á. B., Lázaro, J. L., and Dorronsoro, J. R. Finding optimal model parameters by discrete grid search. In Innovations in hybrid intelligent systems, pp. 120-127. Springer, 2008
Kraskov, A., Stögbauer, H., and Grassberger, P. Estimat- ing mutual information. Physical Review E- Statisttical Nonlinear, and Soft Matter Physics, 69(6))66111)) 2004.
Lu, P., Mishra. S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 35:2507-2521, 2022.
Lee, Park, S., Mo, S., Ahn, S., and Shin, J. Layer-adaptive sparsity for the magnitude- pruning. arXiv preprint arXiv:2010.07611, 2020.
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. Advances in neural information processing systems, 36:21702-21177 2023a.
Lei Ba, J., Kiros, J. R., and Hinton, G. E. Layer normaliza- tion. ArXiv e-prints, pp. a60i, 2016.

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Ma, Y., Li, H., Zheng, X., Xiao, X., Wang, R., Wen, S., Pan, X., Chao, F., and Ji, R. Solving oscillation prob- lem in post-training quantization through theoretical perspective. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pp- 7950-7959, 2023b.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruc- tion tuning with gpt-4, arXiv preprint arXiv:2304003777, 2023.
Popescu, M.-C., Balas, V. E., Peeeeeee-PPoeeseeeeeeeee Popescu, L., and Mas- torakis, N. Multilayer perceptron and neural networks. WSEAS Transactions on Circuits and Systems, 8(7):579- 588, 2009.
Martin, C. H. and Mahoney, M. W. Traditional and heavy- tailed self regularization in neural network models. arXiv preprint arXiv: 00276, 2019.
Rao, Y., Zhao, W. Liu, B., Lu, Zhou, J., and Hsieh, C.-J. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:1 34:13937-139499 2021.
Martin, C.
H., Peng, T., and Mahoney, M. W. Predicting trends in the quality of state-of-the neural networks without access to training or testing data. Nature Commu- nications, :211) 2021.
Razzhigaev, A., Mikhalchuk, M., Goncharova, E.,, Gerasi- menko, N., Oseledets, I., Dimitrov, D., and Kuznetsov, A. Your transformer is secretly linear. arXiv preprint arXiv:2405.12550 2024.
Meng, X., Behdin, K., Wang, H., and Mazumder, R. ALPS: Improved optimization for highly sparse one-shot pruning for large language models. In The Tiihth- Annual Conference on Neural Information Processing Systems, 2024. URL https::t///// http:///ppeeeeeeee//m/oo http//ne//tteeeeee nwt.:/urmttooo forum? 1d-01Bx8 4 4upd.
Sakaguchi, K., Bras, R. L.,, Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99 106, 2021.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
Singh, A., Natarajan, V., Shah, M.. Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Meta. Llama3. https://github.ce deea--llame alammmmmmmmmmmmmmmmmmmmmm 2024.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: 180988000000 02789, 2018.
Sun. M.. Liu, Z., Bair, A., and Kolter, Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 16695 2023.
Mocanu, D. C., Mocanu, E., Stone, P. Nguyen, P. H., Gibescu, M., and Liotta, A. Scalable training of arti- ficial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9 (1):2383, 2018.
Sun, W., Zhou, A., Stuijk, S., Wijnhoven, R., Nelson, A. O., Corporaal, H., et al. Dominosearch: Find layer-wiss fine- grained n: M sparse schemes from dense neural networks. Advances in neural information processing systems, 34: 20721-20732, 2021.
Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up or down? adaptive rounding for post-training quantization. In International conference machine learning, pp. 7197-7206. PMLR, 2020.
Tang, C., Ouyang, K., Wang, Z., Zhu, Y., Ji, W., Wang, Y., and Zhu, W. Mixed piixed-precision neural network quanti- zation via learned layer-wise importance. In European Conference on Computer Vision, pp. 259-275. Springer, 2022.
MagralMagie. Neuralmagic deepsparse inference en- gine. https://gites cwm/neuraammgiidd
Touvron, H.,, Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transform- ers & distillation through attention. In International con- ference on machine learning, pp. 10347-10357. PMLR, 2021.
deepspa rse,, 2021.
NeuralMagic. Neuralmagic nm-vlim inference en- gine. https: gighthtthh..coggggoggg com/neura laaiiiammmmmaa//oomm nm-vl1m, 2024.
Paul, M., Chen, F., Larsen, B. W., Frankle, J., Ganguli, S., and Dziugaite, G. K. Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's ' mask? arXiv preprint arXiv:22 10.03444 2022.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302 /3971, 2023a.
12

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Bahaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Zhang, Q. ZHANA R., Sun, J., and Liu, Y. How sparse can we prune deep network: A fundamental limit perspec- tive. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: / / /openreview ew.....ww......................r ew net/ forum? d= d-TAPPOTAAc
Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lucic, M. On mutual information maximization for representation learning. arXiv preprint arXiv: 1907. 2019.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arwiv:2205.0011 00068, 2022.
Zhang, Y., Zhao, L., Lin, M., Sun, Y., Yao, Y., Han, X.,
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Tanner, J., Liu, S., and Ji, R. Dynamic sparse training: Training-free fine-tuning for sparse llms. arXiv preprint urXiv:2310.089) 2023.
Zhu, M. and Gupta, S. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv: 1710.01878, 2017.
Wang, W., Chen, W., Luo, Y., Long, Y., Lin, Z., Zhang, L., Lin, B., Cai, D., and He, X. Model compression and efficient inference for large language models: A survey. arXiv preprint arXiv:2402.0977
2024.
Zhu, X., Li, J., Liu, Y., Ma, C., and Wang. W. A survey on model compression for large language models. Transac- tions of the Association for Computational Linguistics, 12:1556-1577, 2024.
Xia, M., Zhong, Z., and Chen, D. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.0044 00408, 2022.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.55115 2024.
Yin, L., Wu, Y., Zhang, Z., Hsich, C.-Y., Wang, Y., Jia, Y., Pechenizkiy, M., Liang, Y., Wang. Z., and Liu, S. Out- lier weighed layerwise sparsity (owl): missing secret sauce for pruning Ilms to high sparsity. arXiv preprint arXiv:2310.05175. 2023.
Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., and Cui, L. Width & depth pruning for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 3143-3151, 2022a.
Yu, X., Serra, T., Ramalingam, S., and Zhe, S. The combi- natorial brain surgeon: pruning weights that cancel one another neural networks. International Conference on Machine Learning, pp. 16668-25683, PMIR, 2022b.
Yun, S. and Wong, A. Do all mobilenets quantize poorly? gaining insights into the effect of quantization on depth- wise separable convolutional networks through the eyes of multi-scale distributional dynamics. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2447-2456, 2021.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: met.. 2019.
13

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
A. Limitation and Future Work
In this paper, we derive that layer-wise sparsity rates should gradually increase based on the reconstruction error analysis and determine these rates using a monotonically increasing arithmetic progression. Extensive experiments validate the effectiveness of our method. However, the arithmetic progression configuration may be optimal, and we plan to explore more diverse sparsity rate schemes in the future. Additionally, while method significantly enhances the accuracy of existing post-training sparsity techniques for LLMs, there remains performance gap compared to lossless sparsification, particularly under high sparsity conditions. In future work, we aim to develop more advanced post-traine post-training sparsity methods to further improve the accuracy of sparse LLMs.
B. Proof of Lemma 3.3
Proof of Lemma 3.3. Let A Rmxx and B E Raxp be any matrices. Consider the singular value decomposition (SVD) of A:
A AEV
(11)
where:
U IR mxx is an orthogonal matrix (U" =),
E R" xx is an orthogonal matrix (v"V = I). and
Σ E mxx is diagonal matrix with non-negative singular values 2 02 2 ... 2 o min(m,,) 2 0 on the diagonal.
Substituting the SVD of A into the expression for AB, we have:
AB=UEV B
(12)
Therefore,
AB = UVV B
(13)
Since U is orthogonal matrix, the Frobenius norm is invariant under orthogonal transformations. Specifically:
U UX | |
VX
(14)
Applying this property:
|BB| ΣE B
(15)
The matrix Σ is diagonal with singular values a; on the diagonal. Let min mmnn (A) denote the smallest singular value of A. Then, for any matrix X, we have:
ΣX
Omin X
(16)
in the sense that each singular value scales the corresponding component of X. Therefore, applying this to our case:
min ||T
(17)
This inequality holds because scaling each component by at least amin results in the squared norm being scaled by at least
22 2
Similarly U, the orthogonal matrix V preserves the Frobenius norm:
x
= =|
VX
(18)
Applying this property:
VT B = B|
(19)
14

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Substituting back, obtain:
AB | p=
ΣV B 2 min
V
B
omin B r
(20)
Thus, we have shown that:
|AB
2
(A)
|
(21)
C. Proof of Theorem 3.5
Proof of Theorem 3.5. Recall that each layer i E 21, L in LLM has sparsity rate s; [0, 1], where s; indicates the fraction of zero entries in the weight matrix of layer i. Let L; denote the reconstruction error of layer i, and let 1= L be the total reconstruction error of the entire LLM.
L - Σ
To prove Theorem 3.5, we will make the following assumptions based on the preceding sections:
1. According to Theorem 3.1, The reconstruction error for each layer denoted as L,, is increasing function of the sparsity rate 8,, denote (s:).
2. According to Theorem 3.2, the reconstruction error propagates through layers such that the increase of reconstruction error i-th layer will lead to the increase of reconstruction error (i + 1)-th layer. Therefore we assume Li+1 = + where e 1. The above formula indicates that the reconstruction error of layer not only accumulates the reconstruction errors from previous layers but also that the current sparse layer contributes to the reconstruction error.
The monotonically increasing sparsity scheme given by Eq. 10 (in Sec. 3.4) implies that 81 < 82 < ... < 82, let us denote such a scheme as
st = (s] 2 $}),,
(22)
where of < < <...... <...
Consider different non-monotonically increasing sparsity scheme:
S (si, 2g si,,,
(23)
which is not monotonically increasing. That is, there exists least index k such that s
Moreover, suppose both st and s satisfy the constraint that their average sparsities match the same overall budget. Formally,
S,
(24)
T,
L
and each sf and s results in total reconstruction етгоr
1
ct
Σ
c
L
Σ
(25)
We will show that for any non-monotonic sparsity vector 8 can redundantly reorder it layer by layer get monotonically increasing vector st of the same average sparsity, such that the overall reconstruction error Lt is strietly smaller than L Ultimately, we will deduce
L
L.
(26)
Let us focus on any adjacent pair (5 s6+1) where $ > i1 Define s = + and s = $i. effectively swapping these two sparsities. We then compare the total contribution of layers & and l + to the overall reconstruction error, first in the con-monotonic case and then in the swapped case.
15

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
We restrict our attention layers & and & 1:
Layer l L = f(s)),
Layer & +
S = c + f(1++1).
(27)
Strictly speaking, Cz+1 depends partial errors from layer k and its own sparsity $ Intuitively, higher error S "propagates" or "magnifies" into layer k +++1 1. We keep the rest of the layer-wise sparsities fixed outside of these two positions to isolate their effect.
Now, consider swapping the two sparsities: (sh, sb+1) + (0 s1+1) = (5 s)). Let
C f(() f(6t+1), Ck+1= cli + f(5k+1) c cf + f($)).
(28)
Therefore, the total for these two layers is
Saappingg = 2i + Ca+1 = f($ +
= (($44) + f(()).
(29)
Meanwhile, the original total is
Coriginal Cx + C2+1
f(s)) (cf(s)) (cf(())+(((L+))) c1+c) + f(s+)))
(30)
The difference between the two is
Lariginal Lswapping
(1+ f(5)) + 16
[((h)) f(6i)
(1+t f(s -
- f((
(31)
= -
Since > 1 and f(()) ((6+11) it follows that []((g) [(b+1)) 0.
Hence we obtain
Lariginal
Lewappingg
(32)
This shows that locally swapping a pair (5% ( with 2t > s4t1 reduces the total reconstruction error contributed by layers l and le + 1. Globally across all T. layers, repeating such swap for each adjacent pair that breaks the monotonicity moves S to a strictly monotonically increasing sequence of sparsities.
By iterating this argument over each adjacent pair that fails monotonicity, we reorder s into st. Since every swap strictly reduces the local error, the resulting monotonically increasing scheme s has overall reconstruction error:
I
I.
Σ
C.
L.
(33)
Therefore, the total reconstruction error obtained from the monotonically increasing sparsity scheme proposed in Eq. 10 is strictly less than that obtained from any non-monotonically increasing sparsity scheme, under the same average sparsity constraint.
D. Zero-shot Evaluation Setting
We use the Im-eval-harness framework (Gao et al., 2021) to evaluate the zero-shot performance of LLMs. By default, we follow the settings used in Wanda (Sun et al., 2023) and AlphaPruning (Lu et al., 2024), reporting the "acc" metric across all datasets. However, Im-eval-harness provides multiple metrics depending on the dataset, including both "acc" and "acc "acc_norm". In contrast, ALS (Li et al., 2024c) employs different metrics for different datasets. The evaluation metrics are summarized in Table
16

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Table 8. Evaluation metrics for different datasets
Dataset
Metric
boolq
acc
hellaswag
acc_norm
are easy
acc
piqa
acc
are challenge
acc norm
winogrande
acc
openbookqa acc norm
E. Sparse Inference Engine Supported Devices
Due the sparsity of weights in unstructured pruning, we must use a specific sparse inference engine to accelerate inference. We DeepSparse and nm-vilm accelerate inference on general deployment environments, including CPUs and GPUs. For CPUs, DeepSparse supports architectures including: x86 AVX2, AVX-512, AVX-512 VNNI, and ARM v8.2+, which covers most Intel, AMD, and Apple M-series CPUs. For GPUs, as long the device supports CUDA, inference acceleration can be achieved using the nm-vllm. Similarly, if CUDA is supported for other deployment environments, nm-vllm can also be used to achieve acceleration.
F. More Results
F.1. Arithmetic and Knowledge Reasoning Tasks
We further evaluate the performance improvements of our ATP method arithmetic and knowledge reasoning tasks for sparse models. Specifically, We evaluate the 8-shot accuracy of 40% sparse models the GSM8K dataset and the 5-shot accuracy of 60% sparse models on the MMLU dataset. The results are presented in Table 9, Our ATP method significantly improves the accuracy of sparse models on both tasks, further demonstrating the generalization and effectiveness of our approach.
Table Accuracy of sparse LLaMA2 the GSM8K and MMLU datasets.
GSMSK
MMLU
Method
7B
13B
7B
13B
Dense
14.60
24.56
45.90
55.70
SparseGPT
11.07
16.91
29.40
39.60
Sparse w.TTP
12.01
19.02
32.40
47.70
Wanda
17.51
28.90
34.80
Wanda W.TTP
9.93
19.56
32.80
46.20
F.2. Multimodal Tasks
We demonstrate the applicability of our method to multimodal models by integrating it with Wanda sparsify the Vicuna-7B model (Chiang al., 2023) in LLaVA-115 (Liu et al., 2024b). We only sparsify the Vieuna model within it. Similar to ILLMA, we determine the sparsity rate for each layer and sparsify linear layers. The performance of the sparse model was evaluated on various visual question-answeringg and reasoning benchmarks, including VQA (Singh et al., 2019), VQAv2 (Goyal et al., 2017), and SQA (Lu et al., 2022). We compared our method with magnitude- based pruning, SparseGPT, Wanda, and the DSA method combined with Wanda, under a 50% sparsity rate. The results in Table 10 show that our method outperforms the magnitude- based, SparseGPT, and Wanda approaches and demonstrates superiority over the DSA method.
F.3. Vision Models
We compare our ATP method with other approaches for determining layer- wise sparsity rates on vision models. Experiments are conducted on both CNN and Transformer architectures, including ConvNeXt (Liu et al., 2022b), ViT (Dosovitskiy et al.,
17

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Table 10. The performance improvement of the ATP methad sparse multimodal models.
LLaVA-1.5
VQA
VQAv2
SQA
Dense
58.20
78.50
66.80
Magnitude
38.39
63.50
31.24
SparseGPT
53.69
75.86
63,92
Wanda
53.05
75.72
63.99
Wanda DSA
54.36
76.08
65.57
Wanda w. ATP
55.21
76.92
66.01
2021), and DeiT (Touvron al., 2021), and report the Top-1 accuracy the ImageNet-IK dataset (Deng et al., 2009). Sparse models are obtained using the Wanda (Sun et al., 2023) method, and ATP is compared against the following baselines: uniform sparsity rate, OWL (Yin et al., 2023), and AlphaPruning (Lu et al., 2024). Following Wanda, we only sparsify the linear layers within each block of the ConvNeXt, and we use our ATP method to determine the sparsity rate for each block. For VIT, we use ATP to determine the sparsity rate for each layer. Each layer contains modules such as attention and MLP. We use Wanda to sparsify linear layers. The results for ConvNeXt are presented in Table 11. These results demonstrate that our method is effective for determining layer-wise sparsity rates in vision models and outperforms existing methods. This further confirms the broad applicability of our monotonically increasing sparsity scheme in improving the accuracy of diverse sparse models.
Table 11. Comparison of ImageNet- IK accuracy across different layer-wise sparsity methods the ConyNeXt-Base model.
Method 50%
60%
70%
80%
Wanda
82.72
80.55
68,18
6.44
w. OWL
82.76
80.53
68.28
6.32
w. AlphaPruning
82.76
80.89
74,24
42.35
w. ATP
82.85
81.10
75.58
56.81
We also present the accuracy results for sparse ViT and DeiTT models in Table 12. Additionally, compare method with several widely-used non-uniform layer-wise sparsity strategies in computer vision, including ERK (Mocanu et al., 2018), 1),, Global (Frankle & Carbin, 2018), and LAMP (Lee et al., 2020), as well as AlphaPruning (Lu et al., 2024). The results indicate that our method outperforms all the aforementioned baselines.
Table 12. Comparison of ImageNet- 1K accuracy across different wise sparsity methods on the ViT-B and DeiT-B models.
VIT-B 16/224
DeiT-B 16/224
Method
40%
50%
60%
40%
50%
60%
Uniform
70.87
59.46
29.97
80.08
76.37
61.72
ERK (Mocanu et al., 2018)
70.89
60.49
33.15
80.05
76.22
63.49
Global (Frankle & Carbin, 2018,
66.81
45.75
8.09
79.94
75.09
57.01
LAMP (Lee et al., 2020)
69.45
57.51
26.99
80.19
76.35
63.32
AlphaPruning (Lu et al., 2024)
71.58
64.29
44.21
80.21
77.11
64.56
ATP
72.03
65.46
47.74
80.50
78.02
69.73
F.4. Integrate with other Compression Technologies
To demonstrate the generalization ability of our method for determining layer-wise sparsity rates, we combine ATP with other compression techniques, including N:M sparsity, structured pruning, and sparsity, we follow the mixed N:8 settings (Sun et al., 2021), using ATP to determine the value of N for each layer while maintaining average sparsity at 2:8, 3:8 and 4:8. In terms of structured pruning, we integrated ATP with L-M--runer (Ma et al., 2023a), where ATP determines the layer-wise sparsity rates, and LLM-Pruner applies pruning accordingly. For mixed-preciss precision quantization, we combine ATP with the LIMPQ method (Tang et al., 2022) to determine the quantization bits for each layer. We apply these compression techniques to the LLaMA-7B model and report the perplexity of the compressed models on the WikiText-2 validation set. The experimental results are presented in Table 13. It shows that ATP
quantization. For N:M
18

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
significantly enhances the performance of various compression methods and outperforms both the OWL and AlphaPruning approaches.
Table 13. Experimental results combining ATP with other compression technologics.
Method
4:8
3:8
2:8
Wanda
8.57
42.56
2962.00
OWL
8,55
22.77
331.37
AlphaPruning
8.55
21.49
585.81
ATP
8.15
16.08
265.96
Method
20%
40%
60%
LLM-Pruner
16.95
30.38
90.02
OWL
18.57
28.65
76.99
AlphaPruning
16.78
29.11
71.21
ATP
15.51
27.40
64.25
Method
Mixed
Mixed 2/3/4 Bit
Mixed 2/4 Bit
Random
12.04
11455.54
14817.12 12
I1 norm
14.61
13959.42
33679.21
OWL
9.54
311.95
8429.39
AlphaPruning
261.39
7630.14
ATP
8.52
198.65
5321.35
F.5. More Layer-wise Sparsity Baselines
We compare our method with additional approaches for determining layer-wise sparsity rates, including Uniform (Zhu Gupta, 2017), Global (Frankle Carbin, 2018), ER (Mocanu et al., 2018). ER-Plus (Liu al., 2022a), and LAMP (Lee al., 2020). These methods are combined with the Wanda pruning approach obtain sparse LLaMA-7B models. The experimental results presented in Table 14. It shows that models pruned using the ATP method is consistently lower than that of other baselines, further demonstrating the superiority of our approach.
sparsity rates ranging from 50% 80%, the perplexity of
Table 14. Comparison of WikiText-2 perplexity across various sparsity rates and methods.
Method/Perplexity
50%
60%
70%
80%
Global
14848
17765
5147
39918.56
LAMP
7.57
12.86
185.52
15647.87
LAMP (per-block)
7.25
10.95
98.77
7318.08
ER
7.80
12.41
119.66
6263.79
ER-Plus
8.00
14.04
229.17
6013.91
Uniform
7.26
10.63
84.52
5889.13
ATP
7.05
9.25
20.16
176.80
F.6. Integrate with More Post-training Sparsity Methods
We have demonstrated that our method improves performance over Wanda and SparseGPT methods. Notably, ATP be integrated with any post-training sparsity method to further enhance their effectiveness. In this section, showease the performance improvements achieved by combining ATP with other post-training sparsity methods. Specifically, we apply ATP to DSnoTT (Zhang et al., 2023), Pruner-Zero (Dong et al., 2024), and ALPS (Meng et al., 2024) to obtain a 70% sparse LLaMA2-7B model. The perplexity and zero-shot accuracy results of the sparse models are presented in Table 15. The results indicate that ATP significantly enhances the performance of DSnoT, Pruner-Zero, and ALPS methods.
F.7. Integrate with LoRA Fine-tuning
We further demonstrate the effectiveness of LoRA fine-tuning (Hu et al., 2021) in narrowing the performance gap between highly sparse LLMs and dense models. Specifically, we obtain a 70% sparse LLaMA2-7B model using the Wanda method and fine-tune on 10,000 samples from the Alpaca-GPTs (Peng al., 2023) dataset. We compare the results against models
19

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Table 15. The performance improvement ATP DSno'T, Pruner-zero and AL.S methods.
Method
Perplexity (4)
Accuracy (f)
Dense
5.12
61.88
DSnoT
77.83
35.11
ATP
24.31
44.51
Pruner-Zero
103.15
34.78
w. ATP
48.82
44.77
ALPS
19.31
46.75
ATP
17.99
49.82
sparsified using uniform sparsity, OWL, and AlphaPruning methods. The results in Table 16 show that LoRA fine-tuning significantly improves the accuracy of the sparse model, further reducing the gap with the dense model. Additionally, the sparse model obtained through the ATP method achieves higher accuracy, and this advantage is retained even after LoRA fine-tuning.
Table 16, Comparison of the perplexity and zero-shot accuracy of the 70% sparse ILaMA2-7B obtained by LoRA fine-tuning.
Method
Fine-tuning
Perplexity (1)
Accuracy (f)
N.A.
5.12
Uniform
N.A.
74.26
35.33
Uniform
LoRA
13.36
47.10
OWL
N.A.
30,38
41.75
OWL
LoRA
12.89
50.26
DSA
N.A.
63.71
36.55
DSA
LoRA
13.19
49.49
AlphaPruning
N.A.
28.87
43.37
AlphaPruning
LORA
12.76
50.37
ATP N.A.
ATP
22.16
45.83
12.28
50.79
E.8. Zero-shot Accuracy 60% Sparsity
We present the zero-shot accuracy results of L.L.MA-8BB 60% sparsity in Table 17. The sparse model is obtained using the Wanda method, and we compare ATP with other layer-wise sparsity methods. We can observe that ATP significantly improves the accuracy of Wanda, increasing the average zero-shot accuracy by 4. 1.3% and narrowing the performance gap between sparse and dense model. Furthermore, ATP outperforms OWL, DSA, and AlphaPruning, with 1.05% higher accuracy than the best-performing AlphaPruning.
Table 17. Zero-shot accuracy results of sparse ILaMA3-8B at 60%
sparsity obtained using the Wanda method.
Method
HellaSwag
Winogrande
BoolQ
OBQA
PIQA
ARC-e
ARC-c
Mean
Dense
60.19
72.77
81.35
34,80
79.71
80.09
50.43
65.62
Uniform
37.92
$9,90
68.16
19.80
67.55
59.63
27.43
48.63
OWL
40.71
62.90
70.34
22.90
69.80
62.28
31.39
$1.47
DSA
40.16
63.01
70.09
22.80
69.05
62.20
30.51
51.12
AlphaPruning
41.63
64.51
71.25
22.60
68.61
62.88
30.46
51.71
ATP
41.91
65.49
71.68
23.70
70.94
63.62
31.96
$2.76
G. More Ablation Study
G.1. Computational Efficiency.
In Table 18, we report the time required to determine the layer-wise sparsity rates for 70% sparse LLMs using our ATP method. The measurements were conducted on NVIDIA A100 80GB GPUs. The results show that only a few searches are
20

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
needed obtain the sparsity rates due the narrow range of reasonable values for the hyperparameter B3. Furthermore, the number of model parameters increases, this range narrows further, enabling the sparsity rates be determined with fewer searches. For example, for the largest 70B model, only three searches are necessary. This demonstrates that our method is highly computationally efficient.
Table 18. Computational efficiency of our ATP method minutes).
Model
7B
8B
13B
30B
65B
70B
Wanda
2.0
2.2
3.8
7.8
13.5
14.1
ATP
2.0x9
2.2x9
3.8x7
7.8x5
13.5x3
14.1x3
SparseGPT
6.6
7.3
10.0
35.0
60.0
67.5
W. ATP
6.6x9
7.3x9
7.3x7
10.0x5
60.0x3
67.5x3
G.2. Analyze the Layer-wise Sparsity Distribution.
We analyze the sparsity rate distribution across different average sparsity levels in Figure Our findings indicate that lower average sparsity rates, the differences in sparsity rates across layers minimal. In contrast, these differences become more pronounced at higher average sparsity rates.
0.22 4 0.31
0.8
70%
Sparsity Rate
0.7
0.6
0.03
0.5
10
15
20
25
35
Layer Index
Pigure 2. Comparison of layer-wise sparsity rate distributions at different average sparsity levels.
G.3. Comparison of Sparsity Rates Distribution with other Methods
Figure illustrates the sparsity rate distributions obtained from various layer-wise sparsity methods. We observe that the sparsity rates generally follow an increasing pattern from low to high, further validating the rationale behind our method.
OWL
0.8
AphaPruning
ATP
0.7
Sparsity
0.6
0.5
5
10
15
20
25
30
Layer Index
Figure 3. Comparison of layer-wise sparsity rate distribution with other methods.
21

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
G.4. Different 3 Settings.
Figure 4 shows the impact of different settings on the perplexity of the 70% sparse LLaMA2-7B model obtained using the Wanda method. We observe that as increases, the perplexity initially decreases and then rises. Furthermore, the model's perplexity under various 3 settings remains lower than that of the uniform sparsity rate scheme.
60
Perplexity
40
(0.010.
22.16)
30
Kon-unifam Sparsiyy
20
0.001
0.005
0.010 /
0.014
0.019
Figure 4. Impact of different 8 settings on the perplexity of the 70% sparse LLaMA2-BB.
G.S. Robustness of ATP under Different Random Seeds
Following OWL (Yin et al., 2023), DSA (Li et al., 2024b) and AlphaPruning (Lu et al., 2024), all experimental results conducted under single fixed random seed. We also report WikiText2 perplexity of 70% sparse LLaMA2-7B obtained by Wanda across five random seeds and different calibration sets in Table 19. The variance across random seeds is very low, suggesting the robustness of ATP.
Table 19. The WikiText2 perplexity of 70% sparse LLaMA2-7B obtained by Wanda across five random seeds.
Method
Perplexity
Dense
5.12 (+) 0.00
Uniffrrm
74.26 (+) 0.10
OWI.
30.38 (+) 0.09
DSA
63.71 (+) 0.08
AlphaPruning
28.87 (+) 0.06
ATP
22.16 (+) 0.05
H. Comparison with NEURONAL
The previous work NEURONAL (Cunegatti et al., 2024) also proposed adopting a monotonically increasing sparsity schedule for LLMs, but there are significant differences between our ATP and NEURONAL:
1. Significant differences in algorithms for determining monotonically inereasing sparsity schedules for LLMs.
Although both ATP and NEURONAL propose using monotonically increasing arithmetic progression to allocate sparsity rates for each block of L.Ms, there are significant differences in the algorithms by which ATP and NEURONAL derive the final arithmetic sequences.
ATP calculates the reasonable range values for the common difference 3 of the arithmetic progression, and then searches for the optimal value of within this reasonable range using a grid search with step size of 0.002. The goal of search find the value of that minimizes perplexity of the sparse L.LMs on the WikiText-2 dataset.
In contrast, NEURONAL defines a sparsity window, [s - X, s A], in advance. Given an average sparsity rate of s, NEU- RONAL searches for the optimal À value within a predefined range [0.01, 0.02, 0.03, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10, 0.12, 0515, 0.20, 0.25]. The optimal value is determined by maximizing the alignment between the activation
22

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
values of the sparse and dense models. Due the differences in these algorithms, the final sparsity allocation schemes obtained by ATP and NEURONAL are different.
2. Significant differences between the two sparsity methods beyond the monotonically increasing sparsity schedule.
ATP focuses on deriving that existing LLM sparsity methods have a reconstruction error explosion problem, thus proposing the use of monotonically increasing arithmetic sequence to determine the sparsity rates for LLMs to alleviate the above problem. Our method provides theoretical insights into the effectiveness and rationality monotonically increasing arithmetic sequence sparsity schemes. We only determined non-uniform sparsity rates for each block of LLMs, with linear layers within blocks using uniform sparsity rates.
In contrast, NEURONAL proposes utilizing functional information from dense pre-trained models, i.e., their activations, obtain sparse models that maximize alignment of activations with the corresponding dense model. NEURONAL adaptively selects the best hyperparameters for block sparsity rates and row-wise sparsity rates based on the model and desired sparsity level, to maximize neuron alignment between activations.
Therefore, ATP and NEURONAL also have significant differences at the more macro algorithmic level of sparsity algorithms.
I. Detailed Results for Zero-shot Tasks
In this section, we provide a detailed presentation the performance of each zero-shot task introduced in Sec. 4.
23

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Table 20. Zero-shot accuracy results of LLaMA-VI at 70% sparsity.
Model
Method
HellaSwag
Winogrande
BoolQ
OBQA
PIQA
ARC-e
ARC-c
Mean
Dense
56.92
69.93
75.05
34.40
78.67
75.34
41.89
61.74
SparseGPT
34.58
56,43
64.80
16.80
64.25
45.24
23.12
43.60
w. OWL
37.08
62.35
66.15
19.80
66.16
48.40
26.02
46.57
AlphaPruning
37.81
64.01
68.26
18.80
63.60
46.17
27.39
46.58
LLaMA-7B
w. ATP
37.88
64.22
66.51
20.00
66.17
49.37
27.47
47.37
Wanda
28.86
52.80
59.69
14.20
57.56
31.27
17.75
37.45
w. OWL
34.89
58.64
62.63
17.60
64.30
46.97
24.32
44.19
w. DSA
34.68
59.67
62.69
15.40
64.09
45.16
24.40
43.72
w. AlphaPruning
36.26
62.35
66.12
17.20
63.93
43,90
25.17
44.99
ATP
37.44
61.43
65.79
20.80
67.46
50.63
25.68
47.03
Dense
59.94
72.77
77.89
33.20
79.16
77.40
46.50
63.84
SparseGPT
37.51
63.30
68.78
20.80
67.63
52.78
25.17
48.00
OWL
40.36
66.22
66.82
20.80
69.86
57.37
28.67
50.01
w. AlphaPruning
42.43
67.80
67.58
22.00
70.01
55.47
29.10
50.63
LLaMA-13B
ATP
43.31
67.92
69.14
23.40
70.89
59.97
30.20
52.12
Wanda
31.06
54.38
61.59
16.20
62.68
42.05
17.58
40.79
OWL
38.57
63,46
62.81
20.40
68.61
57.07
26.37
48.18
W. DSA
38.84
62.04
63.03
23.20
68.98
58.29
26.10
48.64
AlphaPruning
41.04
64.96
64.83
21.20
69.01
59.39
28.24
49.81
w. ATP
43.11
64.98
66.64
25.80
69.80
59.97
30.89
51.60
Dense
63.35
75.69
82.69
36.00
81.01
80.30
52.82
67.41
SparseGPT
44.56
69.30
65.35
25.80
72.42
65.78
32,25
53.64
OWL
46.96
72.20
67.95
26.80
73.56
67.13
35.49
55.73
w. AlphaPruning
47.49
72.30
68.96
30.00
73.50
67.97
34.73
56.42
LLaMA-30B
ATP
48.77
72.83
69.28
30.60
73.94
68.81
36.68
57.55
Wanda
44.23
67.01
66.70
26.40
72.03
64.86
32.25
53.35
w. OWL
47.69
69.77
65.02
29.20
73.88
68.98
36.62
55.88
w. DSA
45.62
67.25
73.15
27.80
72.36
68.13
32.17
55.21
AlphaPruning
49.40
71.27
63.82
31.00
73.50
69.40
38.31
56.67
w. ATP
50.27
71.37
65.35
31.40
75.41
70.62
39.42
57.69
Dense
64.53
77.27
84.89
38.00
81.23
81.36
52.73
68.57
SparseGPT
49.98
74.74
81.03
29.20
74.76
72.05
39.51
60.18
w. OWL
51.25
74.98
81.30
27.00
75.68
68.35
37.71
59.47
AlphaPruning
52.40
74.59
83.80
28.42
75.68
69.30
38.05
60.32
LLaMA-65B
w. ATP
53.10
75.69
83.73
28.45
76.06
71.46
40.69
61.31
Wanda
46.29
70.79
77.37
27.20
74.21
70.66
37.79
57.76
w. OWL
50,90
74.11
78.50
30.60
75.68
70.70
38.05
59.79
DSA
48,60
73.08
71.77
28.80
75.35
71.88
38.22
58.24
w. AlphaPruning
52.55
75.06
81.85
30.40
75.84
71.60
40.02
61.05
ATP
52.92
75.70
82.73
31.80
75.98
73.53
40.78
61.92
24

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Table 21. Zero-shot accuracy results of LLaMA-V2/V3 at 70% sparsity.
Model
Method
HellaSwag
Winogrande
BoolQ
OBQA
PIQA
ARC-e
ARC-c
Mean
Dense
57.17
68.90
77.74
31.40
78.07
76.39
43.52
61.88
SparseGPT
33.08
58.41
64.89
17.40
62.46
43.22
22.01
43.07
OWL
36.57
63.06
66.94
21.60
63.89
49.33
24.49
46.55
w. AlphaPruning
36.83
62.19
65.93
19.80
64.58
49.62
24.43
46.20
LLaMA2-7B
ATP
38.94
63.54
69.54
21.70
68.12
52.74
25.85
48.63
Wanda
27.92
49.33
52.87
12.60
55.33
30.60
18.69
35.33
W. OWL
31.83
55.96
62.11
16.80
61.70
43.52
20.31
41.75
W. DSA
28.54
50.36
57.46
12.00
57.13
32.95
17.41
36.55
AlphaPruning
34.56
60.85
62.23
18.00
62.40
43.27
22.27
43.37
w. ATP
36.08
61.01
62.39
20.40
66.81
50.76
23.38
45.83
Dense
60.06
72.22
80.52
35.20
79.11
79.42
48.46
65.00
SparseGPT
36.90
61.64
66.02
21.00
67.57
52.61
25.94
47.38
w. OWL
39.31
65.75
68.04
22.80
67.89
57.70
27.82
49.90
AlphaPruning
41.26
68.03
68.13
24.00
68.28
57.15
29.18
50.86
LLaMA2-13B
w. ATP
42.81
68.09
72.91
24.40
69.74
58.25
31.06
52.46
Wanda
29.60
51.70
62.32
13.60
58.65
37.21
19.11
38.88
OWL
36.31
60.46
63.46
21.80
67.77
55.64
24.91
47.19
w. DSA
32.83
55.01
62.41
17.80
63.71
49.87
21.92
43.36
AlphaPruning
40.28
67.32
62.57
21.60
68.17
54.46
29.35
49.11
w. ATP
41.44
67.50
74.71
24.60
68.25
57.49
30.80
52.11
Dense
66.10
78.06
83.40
37.20
82.21
82.55
54.44
69.14
SparseGPT
50.98
75.45
80.06
30.00
75.24
73.57
40.61
60.84
w. OWL
51.95
74.98
79.25
30.40
75.68
73.00
40.53
60.83
AlphaPruning
51.90
75.06
80.40
29.20
75.68
74.10
40.87
61.03
LLaMA2-70B
w. ATP
53.44
76.56
80.42
31.00
76.71
75.38
42.23
62.25
Wanda
48.16
73.88
74.46
27.00
74.86
72.69
38.31
58.48
w. OWL
50.20
74.02
75.01
28.60
75.68
73.02
38.30
59.26
DSA
48.60
73.08
71.77
28.80
75.35
71.88
38.22
58.25
w. AlphaPruning
51.40
74.74
73.65
29.60
75.84
72.70
38.13
59.44
ATP
51.84
75.53
78.65
29.70
76.66
74.83
39.16
60.91
Dense
60.19
72.77
81.35
34.80
79.71
80.09
50.43
65.62
SparseGPT
34.26
56.75
66.51
16.80
63.28
42.09
21.42
43.02
OWL
36.78
58.96
69.54
18.20
65.45
49.46
24.06
46.06
w. AlphaPruning
35.54
61.56
71.02
17.00
63.92
46.17
21.67
45.27
LLaMA3-8B
ATP
38.19
63.22
71.12
19.00
66.59
50.20
26.19
47.79
Wanda
27.36
49.96
53.33
12.00
56.04
31.86
17.41
35.42
w. OWL
28.43
50.43
61.74
13.00
57.99
35.82
17.58
37.85
w. DSA
27.51
48.46
54.16
11.80
56.63
33.03
17.66
35.61
w. AlphaPruning
27.82
51.85
56.42
13.40
56.47
34.97
17.32
36.89
w. ATP
31.46
54.93
61.79
16.40
62.18
41.79
20.39
41.28
25

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Table 22, Zero-shot accuracy results of LLaMA-VI/V2 at 50% sparsity.
Model
Method
HellaSwag
Winogrande
BoolQ
OBQA
PIQA
ARC-e
ARC-c
Mean
Dense
76.18
70.09
75.11
44.40
79.16
72.98
44.71
66.09
Wanda
68.92
66.38
70.70
39.00
74.76
61.74
38.91
60.06
W. OWL
70.06
66.85
71.44
39.80
75.52
69.44
40.36
61.92
LLaMA-7B
DSA
69.51
67.95
71.31
39.40
75.46
69.65
40.01
61.90
W. ALS
69.59
66.30
73.70
38.60
77.26
65.66
40.02
61.59
AlphaPruning
69.60
67.64
73.33
39.20
75.57
69.65
38.39
61.91
w. ATP
70.18
68.06
73.92
39.90
76.51
69.74
40.70
62.72
Dense
79.06
72.77
77.89
44.80
80.14
74.79
47.78
68.18
Wanda
74.13
71.51
75.96
43.60
77.91
69.65
43.77
65.22
w. OWL
74.82
72.53
76.39
43.60
77.37
73.32
43.34
65.91
LLaMA-13B
DSA
74.02
70.79
76.05
43.20
76.98
72.60
44.19
65.40
w. ALS
74.34
71.35
75.17
43,00
77.37
69.70
44.45
65.05
AlphaPruning
74.58
72.77
76.63
44.00
76.82
74.20
44.36
66.19
w. ATP
74.86
72.78
76.47
44.20
77.46
74.36
44.60
66.39
Dense
75.98
69.06
77.74
44.20
79.11
74.49
46.25
66.69
Wanda
68.76
67.32
75.78
41.40
76.99
69.23
41.72
63.03
w. OWL
70.79
67.32
75.96
42.80
76.33
72.01
41.97
63.88
LLaMA2-7B
DSA
70.90
66.45
76.42
43.00
76.22
71.42
42.83
63.89
w. ALS
70.75
67.80
75.47
44.80
77.10
69.61
42.32
64.12
W. AlphaPruning
70.89
67.48
76.63
43,00
76.33
72.22
42.83
64.20
w. ATP
70.99
67.84
76.73
43.70
76.44
72.90
42.85
64.49
Dense
79.39
72.38
80.58
45.20
80.52
77.53
49.15
69.25
Wanda
75.02
69.39
80.34
44.10
78.13
70.37
42.76
65.73
w. OWL
76.11
71.19
81.65
45.40
78.67
76.85
46.24
68.02
LLaMA2-13B
w. DSA
75.86
71.03
80.83
45.00
78.62
76.22
45.99
67.65
w. ALS
75.67
72.06
81.35
45.80
78.51
70.33
46.08
67.11
w. AlphaPruning
76.19
71.58
80.97
45.00
78.34
76.38
46.16
67.80
ATP
76.26
72.09
81.66
45.20
78.74
76.86
46.42
68.18
26

---

Determining Layer-wise Sparsity for Large Language Models Through Theoretical Perspective
Table 23, Zero-shot accuracy results of more LLM architectures at 70% sparsity.
Model
Method
HellaSwag
Winogrande
BoolQ
OBQA
PIQA
ARC-e
ARC-e
Mean
Dense
59.98
73.32
82.05
33.20
79.98
81.57
51.45
65.93
LLaMA3.1-8B
Wanda 27.43
48.70
57.71
13.60
55.01
31.86
18.43
36.10
w. ATP
31.81
55.49
62.08
15.60
62.68
42.63
20.82
41.59
Dense
52.43
65.04
65.93
27.20
75.84
67.13
32.94
55.22
OPT-13B
Wanda
34.36
55.09
55.02
15.60
62.89
43.73
23.89
41.51
w. ATP
36.47
58.09
62.17
18.20
65.56
46.63
24.91
44.58
Dense
59.64
71.59
85.26
36.80
79.00
78.66
47.78
65.53
Vicuna-13B
Wanda
31.84
54.70
62.78
16.40
61.75
44.87
22.10
42.06
ATP
41.47
63.85
76.70
25.00
69.04
61.95
34.30
53.19
Dense
60.01
72.85
84.65
33.20
78.73
80.43
47.70
65.36
Qwen2.5-7B
Wanda
30.68
51.93
61.96
15.20
61.59
46.34
20.05
41.10
ATP
32.84
56.59
62.14
16.40
63.44
46.42
21.93
43,82
Dense
61.21
73.88
83.64
32.60
80.58
80.85
50.43
66.17
Mistral-7B
Wanda
28.86
51.07
60.03
12.60
57.56
34.60
18.69
37.62
ATP
34.44
58.96
62.20
15.60
63.93
42.68
21.50
42.76
Table 24. Zero-shot accuracy results of more post-training sparsity methods at 70% sparsity.
Model
Method
HellaSwag
Winogrande
BoolQ
OBQA
PIQA
ARC-e
ARC-e
Mean
Dense
57.17
68.90
77.74
31.40
78.07
76.39
43.52
61.88
DSnoT
27.80
51.78
49.66
12.60
55.66
30.89
17.41
35.11
w. ATP
34.98
58.01
62.23
18.20
65.78
50.42
21.93
44.51
LLaMA2-7B
Pruner-Zero
27.56
50.99
41.93
13.00
56.90
34.47
18.60
34.78
w. ATP
35.97
57.93
64.46
18.60
64.15
48.27
23.98
44.77
ALPS
38.35
61.96
64.59
22.20
66.82
48.37
24.95
46.75
ATP
41.58
64.25
64.78
24.20
68.34
56.65
28.92
49.82
27


Reviews:
[{"review_id": "BmRJSmDIbP", "reviewer": "Reviewer_vXo3", "summary": "The paper establishes a relationship between sparsity and reconstruction error in pruning LLMs, demonstrating that increased sparsity leads to higher reconstruction error, which propagates through linear layers. The authors support this claim through empirical evaluation on transforms and theoretical analyses on linear layers, showing that pruning strategies that do not account for this effect may result in higher total error, ultimately degrading model performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your careful review and comments!**\n> Theoretical Claims (Any assumptions should be included in the text of Theorem, not just appear in the proof.)\n\nWe will restate Theorem 3.1 as: *When the input is the same, increasing the sparsity of the weights in the $i$-th layer will lead to an increase in the reconstruction error of this layer.*\n> Experimental Designs Or Analyses (Report average and variance of results.)\n\nFollowing OWL, DSA and AlphaPruning, all experimental results are conducted under a single fixed random seed. We have reported WikiText2 perplexity of 70% sparse LLaMA2-7B obtained by Wanda across five random seeds and different calibration sets. The variance across random seeds is very low, suggesting the robustness of ATP.\n|Method|PPL|\n|-|-|\n|Dense|5.12(±0.00)|\n|Uniform|74.26(±0.10)|\n|OWL|30.38(±0.09)|\n|DSA|63.71(±0.08)|\n|AlphaPruning|28.87(±0.06)|\n|**ATP**|**22.16(±0.05)**|\n> Other Comments Or Suggestions (How do we determine that the $ β$ range is actually \"small\"?)\n\n1. In Sec. 3.3, we state that the reasonable range of $β$ is $0<β≤0.019$ for 70% sparse LLaMA3-8B. To find the best $β$, we use grid search with step size of 0.002, needing only 9 searches. We also test smaller step sizes in Table 4 but find no improvement in results. The step size of 0.002 balances search efficiency and performance well. Therefore, we claim that the reasonable range for $β$ is small, where \"small\" refers to the limited number of searches required.\n2. We analyze the impact of $β$ on perplexity of sparse model in Figure 4, finding that $β$ significantly affects performance. Figure 2 shows that for lower average sparsity, smaller $β$ values are optimal.\n\n> Question 1 (What is the relevance of theoretical analysis on linear networks when the focus is on transformers?)\n\nIn Sec. 3.1, we represent a layer's computation as $\\boldsymbol{WX}$, where $\\boldsymbol{W}$ is the layer's weight and $\\boldsymbol{X}$ is input. A layer includes components such as attention, nonlinearities, and layer normalization. We acknowledge that there are differences between the theoretical analysis based on $\\boldsymbol{WX}$ and the actual architecture, given the presence of more complex nonlinear computations in the network. However, we believe our analysis remains reasonable for the following reasons:\n1. **The theoretical modeling of $\\boldsymbol{WX}$ is sufficient to analyze the layer's reconstruction error.**\n\nOur method sparsifies the linear layers in Attention and MLP modules, while other components remain unaffected. These linear layers account for the majority of the parameter count and significantly influence the computation results of the layer. The sparsified linear layers dominate the computation of the reconstruction error for each layer. Although various nonlinear operations exist in the actual architecture, they typically do not fundamentally alter the reconstruction error of each layer and have minimal impact on theoretical analysis. Therefore, we believe modeling the primary computation of a layer as $\\boldsymbol{WX}$ is sufficient for analyzing the reconstruction error of that layer. This is also sufficient for us to analyze how reconstruction errors accumulate and propagate across the network.\n\n2. **Modeling the computation of modules as $\\boldsymbol{WX}$ is a common practice in many works.**\n\n[A1] and [A2] simplify the computation of the CONV+BatchNorm+RELU modules in quantized convolutional neural networks as $\\boldsymbol{WX}$ when analyzing reconstruction error. This approach of ignoring unnecessary computations and focusing on the core computations is a common practice, which facilitates the derivation of theoretical results.\n\nThank you for providing valuable insights, which are very important for improving our work. **We will include the above clarification into the final version.**\n\n[A1] Up or down? adaptive rounding for post-training quantization. ICML 2020.\n\n[A2] Solving oscillation problem in post-training quantization through a theoretical perspective. CVPR 2023.\n> Question 2 (Discuss Frankle et al.'s work [1])\n\nThank you for providing such awesome work!\n1. Frankle et al. [1] suggest that the effectiveness of initialization pruning mainly depends on each layer's pruning rate, rather than the specific selection of weights within layers. This highlights the importance of layer-wise sparsity rates and supports the value of our work.\n2. We have discussed and compared various layer-wise sparsity methods, including those for LLMs (methods in Table 1) and CNN/ViT (methods in Table 14). Unlike all previous methods, our ATP method discovers that using a simple monotonically increasing arithmetic progression for layer-wise sparsity can achieve excellent results.\n\nWe will include Frankle et al.'s work [1] in our final version.\n\n**We will include the above discussions in our final version. We hope this response has addressed your concerns and kindly ask for a more positive evaluation of our work. Thank you!**", "claims_and_evidence": "The authors argue that sparsity increases the reconstruction error between the output of a compressed layer and its dense counterpart. This claim is supported by both theoretical analysis of a linear layer and empirical evaluations (Figure 1). Additionally, they suggest that this reconstruction error propagates through the network—again backed by theoretical analysis of a linear layer and empirical evaluation —potentially compounding with each subsequent layer. As a result, the authors propose that per-layer sparsity should account for this propagation, with sparsity increasing monotonically as the layer number increases. To achieve this, they introduce a simple algorithm called APT, which determines layer-wise sparsity using an arithmetic progression. The soundness of the algorithm is verified by undergoing extensive evaluation to confirm its effectiveness. Furthermore, the authors provide theoretical analysis (again focusing on linear layers) to justify the necessity of a monotonically increasing sparsity-per-layer.   \n\nI find the empirical evaluations—and the claims based on them—quite strong and comprehensive, even though that idea behind APT is quite simple (i.e. the simplicity actually makes it interesting and compelling). However, my main concern lies with the theoretical analysis. While, to the best of my knowledge, it is correct, it is conducted exclusively on linear layers. Yet, the paper claims from the outset (Section 3.1) to be working with transformers, which incorporate multiple architectural components such as attention mechanisms, nonlinearities, and layer normalization. These elements could significantly influence the theoretical results, making it difficult to assess the practical relevance of these findings for the architectures in question. Moreover, the paper does not address this limitation, nor does it provide any discussion on its implications (See also “Theorems.”). Since theory is posed as one of the main contributions of the paper and it is clear that the authors gave it a lot of thought, I think that this issue of discrepancy (linear layer analysis versus using it to make claims on transformers) should be addressed, since otherwise half of the work seems a little disconnected from the empirical evaluations.", "methods_and_evaluation_criteria": "The majority of the experiments is conducted on the LLama model of various sizes. The studied algorithms for pruning include Wanda and SparseGPT, which are widely-used and well-recognized pruning approaches for zero-shot adaptation. Apart from that, additional experiments on other LLM architectures are provided in the appendix (Table 8). The paper proposes a new scheme for determining sparsity-per-layer densities, and as such compares with other commonly used schemes such as Uniform, OWL, DSA, AlphaPruning apple together with the selected pruning algorithms. In addition Table 1 includes an exploratory algorithm, ALS, and in the Appendix the authors also study some sparsity-per-layer schemes commonly used in non-LLM models (Table 14). In general, I consider the empirical evaluation solid, sound and quite extensive.", "theoretical_claims": "I have checked the correctness of Theorem 3.1, 3.2, Lemma 3.3, and Theorem 3.3, but only skimmed over Theorem 3.5. My general issue with the theorems is that they discuss linear layers (even with no nonlinearity), while the beginning of the Methodology sections promises transformer layers (Section 3.1, first paragraph). This is confusing - see \"Claims And Evidence\". Furthermore, In Theorem 3.1 and later we assume that the input to the pruned and unpruned network at layer $i$ did not change. In general I believe that any assumptions should be included in the text of the Theorem, not just appear in the proof.", "experimental_designs_or_analyses": "Checked Sections 4.2, 4.3, 4.4, 4.5, 4.6, the experimental design seems sound apart from the fact that I am not sure whether the reported values are averages from repeated experiments and what were the deviations.", "relation_to_broader_scientific_literature": "The paper discusses the problem of discovering the optimal sparsity-per-layer budget in pruning methods for LLMs. As such, I believe the work should also discuss related work on the importance of  sparsity-per-layer ratios on the outcome of pruning even for non-LLM models in the related work, eg. [1]", "essential_references_not_discussed": "[1] Frankle, Jonathan, et al. \"Pruning neural networks at initialization: Why are we missing the mark?.\" arXiv preprint arXiv:2009.08576 (2020).", "other_strengths_and_weaknesses": "Strengths:  \n- Work establishes a relationship between the sparsity and the reconstruction error in the pruning of LLMs, studying this aspect both from the empirical and theoretical perspective, showing that increased sparsity  leads to increased reconstruction error, which propagates through layers. Hence schemes that do not address this issue may results in increased total error and hinder the performance of the pruned model.   \n-  Based on this observation a simple method based on arithmetic linear progression is proposed. The method is easy to use (i.e. quite straightforward use of the established relation), but at the same time produces compelling results in comparison to other sparsity schemes.  \n-  The work contains numerous experiments on various architectures, as well as modalities and per-layer sparsity schemes that were used even in non-LLM context (Table 14).   \n- Overall, the paper is well written and easy to follow.  \n   \nWeaknesses:  \n\nDiscussed in paragraphs above (epsecially see \"Claims and Evidence\", \"Theorems\" and  and “Questions”)", "other_comments_or_suggestions": "How do we determine that this range is actually \"small\"? Such a claim shouldn't be based solely on the size of the interval—after all, any interval on RR has the same cardinality as RR. :) I see that the authors conducted a grid search over this parameter in the Appendix, but what I’m really curious about is how sensitive APT is to variations in this hyperparameter. In other words, beyond showing that the optimal β tends to be of the same magnitude across different models, it would be more convincing to demonstrate that small changes in β do not drastically impact performance. This would better support the claim that β is easily tunable.", "questions_for_authors": "1. What is the relevance of theoretical analysis on linear networks when the focus is on transformers? Could you revise Section 3.1 to clarify any potential confusion? Specifically, why introduce transformer layers only to shift to linear layers for theoretical analysis? Additionally, which part of the model is being sparsified? Clarifying these points is crucial to properly validate the theoretical contributions of APT. (See also Clarity and Evidence and Theorems for further discussion.). **Moreover, this is the point that guided my decision on the score of the paper.** \n\n2. Please address the concerns raised in the specific paragraphs above, aside from those already covered in Point 1. For example, the discussion on Relation to Broader Scientific Literature could be expanded to better position the paper within the context of related work. While I find this issue less critical than the concerns in Point 1, addressing it would still enhance the paper’s standing in relation to existing research.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ya7Yct9Er3", "reviewer": "Reviewer_2G3A", "summary": "This work proposes a relatively simple monotonically increasing layer-wise sparsity schedule for LLMs where the layers near the head are more sparse than earlier layers. The method is motivated by a detailed analysis of the effect of increasing sparsity on layerwise reconstruction errors and the propagation of these errors to deeper layers. The authors’ analysis is supported by extensive empirical experiments on one-shot pruned LLMs with a wide variety of leading methods. Across the board, the proposed method yields significant quality benefits compared to the pruning baselines considered and performs on par with bayesian search.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your careful review and comments!**\n\n> Weakness 1 (Missing sparsity preserving fine-tuning results.)\n\nThank you for your suggestions. The results of fine-tuning 70% sparse LLaMA2-7B obtained by Wanda using LoSA [1], LoRS [2] and SPP [3] are below.\n\n|Method|Fine-tuning|Wikitext2 PPL|HellaSwag|Winogrande|BoolQ|OBQA|PIQA|ARC-e|ARC-c|Mean|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Dense|N.A.|5.12|57.17|68.90|77.74|31.40|78.07|76.39|43.52|61.88|\n|Uniform|LoSA|12.68|43.23|58.55|67.04|23.00|69.07|59.06|28.58|49.79|\n|Uniform|LoRS|13.77|40.11|56.03|65.19|20.70|67.05|55.08|25.04|47.03|\n|Uniform|SPP|12.74|42.51|58.24|66.14|24.50|69.01|59.31|28.05|49.68|\n|OWL|LoSA|11.71|44.02|63.81|69.01|25.10|68.99|58.97|28.67|51.22|\n|OWL|LoRS|13.06|43.01|63.04|69.25|23.00|67.12|54.25|29.54|49.89|\n|OWL|SPP|11.55|44.43|64.25|68.38|24.60|69.02|59.04|29.03|51.25|\n|DSA|LoSA|12.09|43.71|63.01|69.35|24.30|68.07|58.06|28.71|50.74|\n|DSA|LoRS|13.14|43.15|62.89|69.02|22.80|66.80|53.01|29.51|49.59|\n|DSA|SPP|11.99|44.01|63.88|69.01|24.00|68.45|58.37|28.56|50.89|\n|AlphaPruning|LoSA|11.49|44.97|64.09|69.97|25.20|69.01|56.48|28.87|51.22|\n|AlphaPruning|LoRS|12.98|43.47|63.77|69.14|22.40|68.23|54.67|29.09|50.11|\n|AlphaPruning|SPP|11.54|45.05|63.01|68.02|25.40|68.93|57.91|30.55|51.27|\n|ATP|LoSA|**10.68**|45.88|62.95|70.84|25.50|70.98|59.58|29.94|**52.23**|\n|ATP|LoRS|**12.01**|44.31|61.10|68.62|24.90|69.39|59.93|29.42|**51.10**|\n|ATP|SPP|**10.87**|45.19|63.14|68.23|26.40|70.89|61.46|29.44|**52.11**|\n\nWe observe that:\n1. **After Fine-tuning, ATP still Outperforms other Methods**\n\nAfter fine-tuning using sparsity-preserving methods, the advantages of ATP have been further highlighted. After fine-tuning with LoSA, the perplexity of **ATP is 2.00 lower than Wanda and 0.81 lower than AlphaPruning**. In terms of average zero-shot accuracy, after fine-tuning with LoSA, **ATP achieves 2.44% higher than Wanda and 1.01% higher than AlphaPruning**.\n\nWe believe that since ATP better preserves the performance of the one-shot pruned model, the performance of the sparse model is easier to recover through fine-tuning in this case. Therefore, the layer-wise sparsity rate determined by ATP still retains its advantages after fine-tuning with the sparsity-preserving fine-tuning methods and remains the optimal layer-wise sparsity.\n\n2. **Significance of Relative Gains**\n\nWhile absolute improvements may appear smaller for fine-tuned models, the relative gains remain significant. For example, compared to dense model, Wanda w. LoSA shows an average zero-shot accuracy loss of 12.09%, while ATP w. LoSA shows a loss of 9.65%, **representing a relative improvement of 20.18%. In contrast, AlphaPruning w. LoSA only achieves a relative improvement of 11.83%.** Considering that the performance gap between sparse and dense LLMs further decreases after fine-tuning, the aforementioned accuracy improvement is likely to translate into more meaningful performance enhancements in practical applications.\n\n**We will include these results in our final version.**\n\n> Weakness 2 (Performance of the sparse model with 70% sparsity has a gap compared to dense model.)\n\nWe have observed that performance of 70% sparse LLMs obtained by ATP still underperform compared to dense models. This observation is consistent with the common pattern in model pruning and compression, where high sparsity rates lead to significant accuracy degradation. However, we believe our ATP method still offers the following advantages:\n\n1. **Pushing the Performance Limits of Sparse LLMs**\n\nOur ATP method addresses the challenge of performance degradation under high sparsity rates, advancing the frontier of sparse LLMs performance and narrowing the gap with dense models, which is beneficial to the community.\n\n2. **Good Performance at Lower Sparsity Rates**\n\nUnder lower sparsity settings, our ATP method further reduces the performance gap between sparse and dense LLMs. For example, the average zero-shot accuracy loss for 50% sparse LLaMA2-13B model obtained using the Wanda method is 3.52%, while our ATP method reduces this loss to 1.07%. This is highly advantageous for the practical deployment of lower-sparsity LLMs.\n\n3. **Further Narrowing the Gap with Dense Models When Combined with Fine-Tuning**\n\nAs demonstrated in Table 16 and response to Weakness 1, we show that combining ATP with PEFT techniques can further improve the accuracy of sparse LLMs. For example, the average zero-shot accuracy loss for 70% sparse LLaMA2-7B model obtained using Wanda is 26.55%. With ATP and LoSA optimizations, this accuracy loss is further reduced to 9.65%. Moreover, sparse LLMs obtained using the ATP method achieve higher accuracy, retaining this advantage even after fine-tuning.\n\nIn summary, although there remains a gap between sparse and dense LLMs under high sparsity settings, our ATP method significantly narrows this gap, greatly enhancing the practicality of sparse LLMs.\n\n**Finally, we hope our response has addressed your concerns. Thank you!**", "claims_and_evidence": "In general the claims made are supported by the empirical data and proofs provided. However, while I am convinced that ATP benefits the one-shot pruning setting, I remain unconvinced that ATP finds the best layer-wise sparsity distribution for LLMs in general after fine-tuning the pruned network. The fine-tuning experiments conducted with LoRA (Section F.8) are not sparsity-preserving and ATP only shows a modest benefit over AlphaPruning in this setting.", "methods_and_evaluation_criteria": "The method and evaluation criteria are typical for evaluating the performance of sparse LLMs. A wide range of downstream tasks were included as well as a perplexity analysis and performance using Neural Magic inference frameworks.", "theoretical_claims": "I reviewed the proofs and did not find any issues.", "experimental_designs_or_analyses": "* Generally, the experimental designs appeared to be sound. \n* However, one instance where I believe the authors could improve their results is to include fine-tuning with a *sparsity-preserving* method. LoRA adapters after fine-tuning yield a dense matrix which will destroy the sparsity introduced by pruning if merged with the original weight matrices – which is the typical approach. As such, I believe an important but missing experiment is to verify whether ATP provides any benefits after sparsity preserving fine-tuning. Examples of such methods include masked fine-tuning (simply masking weights and gradients with the sparsity introduced with pruning) or  sparsity-preserving PEFT methods such as those introduced in [1-3]. \n\n[1] W. Huang et al., “Dynamic Low-Rank Sparse Adaptation for Large Language Models,” Feb. 20, 2025, arXiv: arXiv:2502.14816. doi: 10.48550/arXiv.2502.14816.\n[2] Y. Hu, J. Zhang, X. Chen, Z. Zhao, C. Li, and H. Chen, “LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model,” Jan. 15, 2025, arXiv: arXiv:2501.08582. doi: 10.48550/arXiv.2501.08582.\n[3] X. Lu, A. Zhou, Y. Xu, R. Zhang, P. Gao, and H. Li, “SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models,” May 25, 2024, arXiv: arXiv:2405.16057. doi: 10.48550/arXiv.2405.16057.", "relation_to_broader_scientific_literature": "LLM efficiency is a crucial consideration given their high computational cost. Weight sparsity is one promising avenue to improve LLM performance as most models are memory-bound on current hardware. As such, reducing the number of non-zero elements in the model parameters yields potential performance gains but reducing the amount of HBM I/O and the overall VRAM required (depending on the fine-grained sparse structure and sparsity level).", "essential_references_not_discussed": "None noted.", "other_strengths_and_weaknesses": "## Strengths:\n* Important and timely topic\n* Strong empirical results\n* Intuitive method with theoretical justification\n* The method proposed is efficient and does not require significantly more time or compute than the considered baselines\n\n## Weakness: \n* The main weakness is the missing sparsity preserving fine-tuning comparison. Especially since the LoRA results show a convergence in accuracy between ATP and the considered baselines. A crucial finding will be whether ATP is primarily important for improving one-shot accuracy only or if these benefits also extend to the post fine-tuning setting when the sparse mask is fixed prior to fine-tuning. \n* While ATP improves the quality of the sparse LLMs compared to baselines, the quality still falls far short of the dense models at moderate sparsities such as 70%. Improving the quality of moderate to highly sparse LLMs remains a challenge, even with ATP.", "other_comments_or_suggestions": "None noted.", "questions_for_authors": "1. How does ATP compare with baselines when fine-tuned with a static mask or a sparsity preserving PEFT method? I would be willing to increase my score if ATP’s benefits remain after static mask fine-tuning.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AnVOfmCPqY", "reviewer": "Reviewer_JETH", "summary": "This paper identifies the issue of \"reconstruction error explosion\" in existing LLMs sparsification methods. Through theoretical analysis, it derives a layer-wise sparsity allocation method based on a monotonically increasing arithmetic progression to alleviate the above issue. Both theoretical analysis and experimental results indicate that the above sparsity allocation scheme is near optimal, and significantly improves the performance of sparse LLMs with various architectures, outperforming the existing layer-wise sparsity methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your careful review and comments!**\n> Weakness 1 (Smaller improvements for lower sparsity.)\n\nAs sparsity decreases, the returns on performance improvements tend to diminish, which is consistent with common patterns in model pruning and compression. However, we believe that our ATP method still demonstrates quite impressive performance:\n\n1. **Significance of Relative Gains**\n\nWhile absolute improvements may appear smaller for models with lower sparsity rates, the relative gains remain significant. For 50% sparse LLaMA-7B obtained using Wanda, our ATP method reduced the average zero-shot accuracy loss from 6.03% to 3.37%, showing a 44.11% relative improvement, compared to only 30.85% relative improvement with OWL.\n\n2. **Inherent Capabilities of Lower Sparsity Models**\n\nAt lower sparsity settings, the inherent capabilities of the model are better preserved. For a 50% sparse LLaMA2-7B obtained using Wanda, the average zero-shot accuracy loss is only 3.66%, compared to 26.55% at 70% sparsity, leaving less room for further improvement.\n\n3. **Alignment with Theoretical Insights**\n\nWang et al. [1] established scaling laws for sparse LLMs, revealing the relationship between sparsity rate and model performance, indicating that model performance loss is smaller at lower sparsity rates.\n\nOverall, our ATP method maintains effectiveness across all sparsity settings, offering critical stability and performance advantages.\n\n[1] Wang et al. Q-Sparse: All Large Language Models can be Fully Sparsely-Activated.\n> Weakness 2 (Implementation details of CNN, ViT and LLaVA-1.5.)\n1. Following Wanda, we only sparsify the linear layers within each block of the ConvNeXt, and we use our ATP method to determine the sparsity rate for each block.\n\n2. For ViT, we use ATP to determine the sparsity rate for each layer. Each layer contains modules such as attention and MLP.  We use Wanda to sparsify linear layers.\n\n3. For LLaVA-1.5, we only sparsify the Vicuna model within it. Similar to LLaMA, we determine the sparsity rate for each layer and sparsify linear layers.\n\n> Other Comments Or Suggestions (Typos.)\n\nThanks for this great and detailed comments. We will fix this typo in the final revision.\n> Question 1 (Lack of MoE experiments.)\n\nWe use Wanda to obtain 70% sparse MoE model Mixtral-8x7B.\n|Method|Wikitext2 PPL|HellaSwag|Winogrande|BoolQ|OBQA|PIQA|ARC-e|ARC-c|Mean|\n|-|-|-|-|-|-|-|-|-|-|\n|Dense|3.86|64.95|76.09|85.08|35.40|82.49|84.18|57.25|69.35|\n|Uniform|18.22|39.18|60.06|61.75|20.10|70.01|59.38|28.04|48.36|\n|OWL|16.15|40.31|60.54|62.66|21.00|70.40|60.27|29.22|49.20|\n|DSA|16.22|40.11|60.78|61.90|21.10|70.32|60.06|29.30|49.08|\n|AlphaPruning|15.77|40.52|62.12|61.78|21.40|70.89|60.34|29.35|49.49|\n|**ATP**|**14.30**|**41.98**|**63.89**|**62.82**|**22.50**|**71.76**|**61.12**|**30.60**|**50.67**|\n\nOur ATP method performs excellently on sparse MoE models, further demonstrating its generalizability across different model architectures.\n> Question 2 (Lack of Llama-3.2-1B/3B experiments.)\n\nWe use SparseGPT to obtain 70% sparse Llama-3.2-1B/3B.\n|Method|Wikitext2 PPL|HellaSwag|Winogrande|BoolQ|OBQA|PIQA|ARC-e|ARC-c|Mean|\n|-|-|-|-|-|-|-|-|-|-|\n|Llama3.2-1B|9.65|47.72|60.46|63.91|26.40|74.48|65.45|31.31|52.82|\n|Uniform|129.24|27.61|51.00|58.56|13.10|57.42|34.10|19.04|37.26|\n|OWL|111.02|28.05|51.20|61.39|13.40|57.78|35.36|19.51|38.10|\n|DSA|121.70|27.93|51.11|59.98|13.10|57.51|35.02|19.31|37.71|\n|AlphaPruning|115.67|28.22|51.38|60.76|13.70|57.64|34.90|19.24|37.98| \n|**ATP**|**94.78**|**28.73**|**52.37**|**62.08**|**14.40**|**58.71**|**35.40**|**20.35**|**38.86**|\n|Llama3.2-3B|7.73|55.28|69.93|73.27|31.00|76.71|74.54|42.24|60.42|\n|Uniform|65.97|30.01|51.78|62.17|14.40|60.66|37.79|19.71|39.50|\n|OWL|53.40|31.73|52.72|62.20|14.60|61.43|40.49|20.39|40.51|\n|DSA|57.81|30.79|53.89|62.65|14.60|60.99|38.90|20.87|40.38|\n|AlphaPruning|60.59|30.15|52.09|62.27|14.80|61.47|38.55|20.39|39.96|\n|**ATP**|**46.64**|**33.66**|**59.32**|**62.49**|**17.10**|**62.29**|**41.53**|**21.19**|**42.51**|\n\nOur ATP method demonstrates consistent improvements across all parameter levels, performing well for models ranging from 1B to 70B parameters.\n> Question 3 (On which devices can sparse LLMs be deployed?)\n1. Due to the sparsity of weights in unstructured pruning, we must use a specific sparse inference engine to accelerate inference. We use DeepSparse and nm-vllm to accelerate inference on general deployment environments, including CPUs and GPUs.\n2. For CPUs, DeepSparse supports architectures including: x86 AVX2, AVX-512, AVX-512 VNNI, and ARM v8.2+, which covers most Intel, AMD, and Apple M-series CPUs.\n3. For GPUs, as long as the device supports CUDA, inference acceleration can be achieved using the nm-vllm. Similarly, if CUDA is supported for other deployment environments, nm-vllm can also be used to achieve acceleration.\n\n**We will incorporate the above responses into the final version. We hope this addresses your concerns. Thank you!**", "claims_and_evidence": "1. The author claims that the existing post-training sparsification methods for LLMs suffer from the issue of \"reconstruction error explosion\". The author supports the above claim through proof of Theorems 3.1-3.4. Moreover, the author has depicted the \"explosive\" trend of the reconstruction error of sparse LLMs increasing with the layer number in Figure 1, which also supports the above claim.\n\n2. The author proposes that the issue of \"reconstruction error explosion\" can be alleviated through a layer-wise sparsity rate scheme based on a monotonically increasing arithmetic progression. The author has proven the effectiveness of the above method through extensive experiments. Moreover, through the proof of Theorem 3.5 and the experiment comparing with Bayesian search, the author supports the claim that the above scheme is near optimal both theoretically and experimentally.\n\n3. All in all, the author has effectively supported the proposed claims through theoretical analysis and experimental verification.", "methods_and_evaluation_criteria": "The author discovers the issue of \"reconstruction error explosion\" existing in the sparsity methods of LLMs. This refers to the cumulative effect of reconstruction errors. Throughout the entire sparsification process, the errors from the early layers propagate and amplify in the subsequent layers, leading to a significant increase in the overall reconstruction error and a substantial decline in the model's performance. This explains well that why the existing sparse LLMs have relatively low accuracy. Moreover, the author proposes to alleviate this issue by using a layer-wise sparsity scheme based monotonically increasing arithmetic progression, which is intuitive and reasonable. The author's theoretical analysis and experimental verification both show that this layer-wise sparsity scheme makes sense.", "theoretical_claims": "Yes. I have checked the proofs of Theorems 3.1-3.5. The author has proven the effect of the sparsity rate on the reconstruction error, the effect of the reconstruction error of the previous layer on that of the next layer, and that the total reconstruction error of the proposed monotonically increasing sparsity rate scheme is smaller than that of any non-monotonically increasing sparsity rate scheme. I think the author's proofs are correct, and I haven't found any issues.", "experimental_designs_or_analyses": "Yes. The author has conducted sufficient experiments, comparing with the SOTA baselines. Experiments have been carried out on different LLMs, many tasks have been evaluated, and experiments have also been conducted on multimodal models and vision models. The author has also demonstrated the effects of combining with different compression techniques. In addition, the author has analyzed the ablation experiments of different hyperparameters, the computational efficiency of the proposed method, as well as the distribution of the layer-wise sparsity rate. Both the experimental design and analysis are reasonable and effective.", "relation_to_broader_scientific_literature": "The author claims that the existing layer-wise sparsity methods for LLMs either determine the sparsity rate of each layer by calculating the importance metric of each layer, or obtain the layer-wise sparsity rate through a search method. The importance metric of LLMs is often heuristic and require a great deal of effort to verify their effectiveness, while the search method is very time-consuming. In contrast, the author proposes that the layer-wise sparsity rate of LLMs can be determined through a monotonically increasing arithmetic progression, eliminating the need for cumbersome layer-wise importance calculations or time-consuming search processes. The above scheme is simple and effective. It can quickly determine the sparsity rate of each layer of LLMs and improve the accuracy of sparse LLMs, which is of great significance to the compression community.", "essential_references_not_discussed": "No. The author compares the existing methods for determining the layer-wise sparsity rates of LLMs, including OWL, ALS, DSA, and AlphaPruning. These are the layer-wise sparsity methods customized for LLMs that were published at ICML or NeurIPS prior to the submission date of this paper. As far as I know, there were no other layer-wise sparsity rate methods specifically designed for LLMs before the submission date. Additionally, the author demonstrates the improvement effects of these layer-wise sparsity methods on the Wanda and SparseGPT methods, which are the latest and widely used sparsity methods for LLMs. In the appendix, the author also compares the methods for determining the layer-wise sparsity rates of CNNs and ViT models. Overall, the author has compared and discussed with many relevant and essential works.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Originality: The layer-wise sparsity rate scheme based on a monotonically increasing arithmetic progression is novel, simple and efficient, and it does not require complex layer-wise importance calculations or time-consuming searches. The author has proven its effectiveness through theoretical analysis and experimental verification.\n\n2. Significance: The author has conducted extensive experiments to verify the effectiveness of the proposed method, including experiments on LLMs with various architectures and parameters, as well as on multiple tasks. The proposed method has significantly improved the accuracy of sparse LLMs, and the accuracy improvement is obvious compared with the SOTA layer-wise sparsity methods. The proposed method also has a significant improvement effect on multimodal models and vision models, and can also improve the accuracy of compression techniques including quantization. In addition, sparse LLMs have an acceleration effect on CPU and GPU, which significantly improves the usability of sparse LLMs.\n\n3. Clarity: This paper is well-written, with clear logic, a well-organized structure, and rich content.\n\nWeaknesses:\n\n1. Under the setting of 50% sparsity rate, the proposed ATP method has a relatively limited improvement compared to the Uniform baseline, and also has a limited improvement compared to other layer-wise sparsity rate methods.\n\n2. Some implementation details are missing, for example, how is the proposed ATP method applied to CNN, ViT and LLaVA-1.5.", "other_comments_or_suggestions": "Typos:\nLine 833: Table 7. Evaluation Metrics for Different Datasets. -> Table 7. Evaluation metrics for different datasets.", "questions_for_authors": "1. The author has evaluated LLMs models with a wide range of architectures, but the experimental results of the Mixture of Experts (MoE) model are lacking. However, the MoE model has received increasing attention due to its powerful performance. Can the author evaluate the proposed method on LLMs based on the MoE architecture? I think this would be beneficial for enhancing the contributions of the paper.\n\n2. The author has presented the experimental results of models with more than 7 billion parameters. I'm wondering whether the ATP method can also bring about performance improvements for the smaller LLaMA-3.2-1B/3B models?  \n\n3. Sparse LLMs need to rely on specific sparse inference engines (such as DeepSparse and nm-vllm) to achieve acceleration effects. I am curious about the practicality of DeepSparse and nm-vllm. In other words, on which devices can I use these two inference engines to deploy sparse LLMs?\n\nOverall, I think this paper is a good paper and it is worthy of being accepted. However, it lacks some details and experiments. If the author can address the above weaknesses and questions, I will raise my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RLGBqoIC2D", "reviewer": "Reviewer_Jh19", "summary": "This paper determines the sparsity rate of each layer for LLMs from a theoretical perspective, proposes that there is a \"reconstruction error explosion\" problem in sparse LLMs, and proposes to use a monotonically increasing arithmetic progression to determine the sparsity rate of each layer, thereby alleviating the \"reconstruction error explosion\" problem. Experimental results on different models and datasets show the effectiveness of this method and that it outperforms existing methods.\n\n## update after rebuttal\n\nI am satisfied with the authors' response, including clarifications and additional experiments. I will keep my positive rating, and I hope the responses could be incorporated into the next revision.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thanks for your careful review and comments!**\n\n> Weaknesses (Smaller improvements for larger models.)\n\nWe have observed that larger models show less performance improvement with ATP. This observation aligns with common patterns in model pruning and compression, where the returns tend to diminish as model size increases. However, our ATP still demonstrates consistent improvements across different model sizes, especially notable in larger models:\n\n1. **Significance of Relative Gains**\n\nWhile absolute improvements may appear smaller for larger models, the relative gains remain significant. For example, for LLaMA-65B at 70% sparsity with Wanda, ATP reduces the zero-shot accuracy loss from 10.81% to 6.65%, representing a 38.48% relative improvement. However, the improvement of AlphaPruning is only 30.71%. Such gains can translate to meaningful performance enhancements in real-world applications.\n\n2. **Inherent Resilience of Larger Models**\n\nLarger models naturally exhibit greater resistance to pruning. As shown in Table 1, 70% sparse LLaMA2-70B with SparseGPT experiences an zero-shot accuracy loss of only 8.30%, compared to 18.81% for LLaMA2-7B, leaving less room for further improvement. Nevertheless, our method further reduces the loss of  LLaMA2-70B to 6.89%. This observation aligns with the findings of Li et al. [1].\n\n3. **Alignment with Theoretical Insights** \n\nThe behavior aligns with recent theoretical insights. Liu et al. [2] demonstrated that larger models can maintain performance even under random pruning, supporting the idea of their inherent pruning resistance.\n\nIn summary, our ATP method remains effective across all model sizes and still achieves considerable performance improvements in larger models, providing critical stability and performance benefits.\n\n[1] Li et al. Train big, then compress: Rethinking model size for efficient training and inference of transformers.\n\n[2] Liu et al. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training.\n> Other Comments Or Suggestions (Figure 1 needs to be enlarged.)\n\nThank you for your suggestion. We will enlarge Figure 1 in our final version to make it clearer for viewing.\n> Question 1 (What are the advantages of sparsity compared to quantization?)\n\nNetwork sparsity and quantization are both effective methods for improving inference speed and reducing memory footprints, with both techniques offering significant acceleration benefits across different hardware platforms (such as CPU and GPU). However, comparisons between these two approaches in terms of efficiency may vary.\n\nRecent research suggests that network sparsity can slightly outperform quantization in inference speedup. For example, SqueezeLLM [3] demonstrates that quantizing LLaMA2-7B to 3 bit achieves a 2.1$\\times$ speedup on GPU. Meanwhile, the latest sparse CPU and GPU kernels (DeepSparse and nm-vllm) provide better support for deploying sparse LLMs, achieving 2.63$\\times$ CPU inference acceleration and 2.23$\\times$ GPU inference acceleration for 70% sparse LLaMA2-7B, as shown in Table 6 of our paper. Additionally, compared to quantization, sparsity can better maintain and recover performance through fine-tuning, as described in [4].\n\nPruning and quantization are compatible and complementary approaches, combining these methods can further enhance efficiency. Table 3 in [4] shows that combining network sparsity with quantization (INT8) can achieve up to 9.08$\\times$ acceleration, significantly higher than using either method alone.\n\n[3] SqueezeLLM: Dense-and-Sparse Quantization.\n\n[4] Sparse Fine-tuning for Inference Acceleration of Large Language Models.\n> Question 2 (Experimental results with 60% sparsity.)\n\nWe present the zero-shot accuracy results of Llama-3-8B at 60% sparsity below. The sparse model is obtained using the Wanda method, and we compare ATP with other layer-wise sparsity methods.\n\n|Method|HellaSwag|Winogrande|BoolQ|OBQA|PIQA|ARC-e|ARC-c|Mean|\n|-|-|-|-|-|-|-|-|-|\n|Dense|60.19|72.77|81.35|34.80|79.71|80.09|50.43|65.62|\n|Uniform|37.92|59.90|68.16|19.80|67.55|59.63|27.43|48.63|\n|OWL|40.71|62.90|70.34|22.90|69.80|62.28|31.39|51.47|\n|DSA|40.16|63.01|70.09|22.80|69.05|62.20|30.51|51.12|\n|AlphaPruning|41.63|64.51|71.25|22.60|68.61|62.88|30.46|51.71|\n|**ATP**|**41.91**|**65.49**|**71.68**|**23.70**|**70.94**|**63.62**|**31.96**|**52.76**|\n\nWe observe that:\n\n1. ATP significantly improves the accuracy of Wanda, increasing the average zero-shot accuracy by 4.13% and narrowing the performance gap between sparse and dense model.\n\n2. ATP outperforms OWL, DSA, and AlphaPruning, with 1.05% higher accuracy than the best-performing AlphaPruning.\n\nIn conclusion, our ATP method demonstrates consistent improvements across various sparsity levels and significantly outperforms existing layer-wise sparsity methods.\n\n**We will incorporate the above responses into the final version. We hope that our response has addressed your concerns. Thank you!**", "claims_and_evidence": "The author claims that there exists a \"reconstruction error explosion\" problem in sparse LLMs which leads to severe accuracy degradation. This claim is supported by specific observations, and in Figure 1, we can see the trend of reconstruction error explosion. The author also analyzes the reasons for the \"reconstruction error explosion\" from a theoretical perspective, and I think there is no issue with the author's claim. Additionally, the author claims that using a monotonically increasing arithmetic progression can alleviate the above problem, thereby improving the accuracy of sparse LLMs. Extensive experimental validations support this claim made by the author.", "methods_and_evaluation_criteria": "The layer-wise sparsity method based on monotonically increasing arithmetic progression proposed in this paper makes sense, as it effectively improves the accuracy of sparse LLMs, outperforming existing layer-wise sparsity methods.", "theoretical_claims": "I have examined the author's proof of the theorem and did not find any issues. The author's proof mainly involves expanding and transforming the reconstruction error based on the Frobenius norm, thereby proving that increasing the sparsity rate in earlier layers leads to an increase in reconstruction error in later layers, which consequently results in the \"reconstruction error explosion\".", "experimental_designs_or_analyses": "The authors' experimental design follows established protocols for LLM sparsification methods, including Wanda, SparseGPT, and other layer-wise sparsity approaches for comparison. The paper's experiments encompass various LLM architectures and sizes across numerous datasets, demonstrating results at different sparsity rates. I find the experimental design to be well-reasoned and effective in evaluating the proposed method's efficacy.", "relation_to_broader_scientific_literature": "The author divides the previous layer-wise sparsity methods for LLMs into two categories, including metric based methods (OWL, AlphaPruning and ALS) and search based methods (DSA). The former requires complex calculations and lacks theoretical guarantees of optimality, while the latter demands time-consuming search processes spanning days and the search effect is heavily dependent on human expertise. Therefore, the author proposes to derive the layer-wise sparsity rate of LLMs from a theoretical perspective, so that only a simple monotonically increasing arithmetic progression is needed, and the author guarantees from a theoretical perspective that this scheme is near optimal. I think the analysis of the issues existing in the existing layer-wise sparsity methods is reasonable, and the motivation behind determining the layer-wise sparsity rate from a theoretical perspective is great.", "essential_references_not_discussed": "From the best of my knowledge, no essential references are not discussed. All the authors discuss are the latest sparsity methods for LLMs. The layer-wise sparsity methods that are most relevant to this paper have been discussed by the authors.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper demonstrates good organization, with a methodically presented approach supported by numerous figures, tables, and reproducible code.\n2. The idea of a monotonically increasing arithmetic progression to determine the layer-wise sparsity rate is interesting and novel, and the authors reveal the rationality of this sparsity rate scheme from the perspective of \"reconstruction error explosion\".\n3. The authors provide comprehensive experimental results on various tasks and models. Compared with the existing layer-wise sparsity baselines, APT outperforms the existing baselines. The efficiency of models make them good to deploy in real-world.\n\nWeaknesses:\nFor larger parameter LLMs, the improvements from the ATP method appear to be less substantial than those observed in smaller LLMs.", "other_comments_or_suggestions": "All the lines in the left picture of Figure 1 are close together and need to be zoomed in to be seen clearly. Can the author improve the clarity of this picture?", "questions_for_authors": "1. What are the practical advantages of LLMs sparsity as adopted in this paper compared to other acceleration techniques (such as quantization)?\n2. The authors show the zero-shot accuracy of LLMs at 50% and 70% sparsity. I would like to know what the zero-shot accuracy of the ATP method would be under the 60% sparsity setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work is clearly motivated, with sparsity in LLMs and specifically the layer-wise sparsity ratio being topics of particular interest. The reviewers commented consistently as to the comprehensiveness of the empirical results and soundness/novelty of the methodology. There were relatively few concerns from the reviewers, perhaps notably concerns on the clarity of the theoretical analysis presented, specifically the relevance of this to real-world Transformer architectures, and some implementation details. The author-reviewer discussion period was very productive (thanks to both the authors and reviewers on this), and I believe it is fair to summarize that the rebuttal appears to have largely addressed the reviewers' concerns. All reviewers recommend an accept post-rebuttal, with one weak accept, and three accepts, and given the discussion I would recommend for acceptance.
Decision: Accept (spotlight poster)