LSA: Layer-wise Sparsity Allocation for Large Language Model Pruning Based on Minimal Linear Reconstruction Error
Authors: Zhiguo Yang
Changjian Deng
Qinke Chen
Zijing Zhou
Jian Cheng
Venue: ICLR 2026 Poster (2026)
Keywords: Layer-wise Sparsity Allocation
Large Language Model Pruning
Linear Reconstruction Error
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Deploying large language models (LLMs) on platforms with insufficient computational resources remains a key challenge. Weight pruning is an efficient model compression technique that can reduce model size without retraining LLMs. However, due to the massive number of parameters, it is infeasible to estimate the importance of weights globally, and most prior studies assign a uniform sparsity ratio across all layers. Recent findings reveal that layers contribute unevenly to LLM performance, making it necessary to investigate Layer-wise importance. Existing Layer-wise sparsity allocation methods, such as OWL and DLP, rely on weight scoring and carefully designed score proxies to estimate Layer-wise importance and sparsity ratios, while enforcing identical sparsity to blocks and projection weights within a layer to avoid performance degradation. In this work, we propose Layer-wise Sparsity Allocation (LSA) for LLM pruning, which quantifies Layer-wise importance by evaluating the minimal linear reconstruction error (LSE) of each transformer layer under the assumption that 50\% of its least important weights are removed. Moreover, our method supports non-uniform sparsity allocation at block- or projection-level granularity within layers, without incurring catastrophic performance degradation. Experimental results demonstrate that LSA maintains high performance at high sparsity levels. At an overall sparsity ratio of 70\%, LSA surpasses state-of-the-art methods across language modeling tasks and seven zero-shot tasks.

Published as a conference paper at ICLR 2026
LSA: LAYER-WISE SPARSITY ALLOCATION FOR LARGE LANGUAGE MODEL PRUNING BASED ON MIN- IMAL LINEAR RECONSTRUCTION ERROR
Zhiguo Yang; Changjian Deng, Qinke Chen, Zijing Zhou, Jian Cheng'
School of Information and Communication Engineering, University of Electronic Science and Technology of China
com
cjdengestd. wwwtddeeeeeeeee.m. edu. cn
qinkechen7@gmail.co com
chouzzing meoouzzing@gmail... com
thengjian@uestc..emm edw.eesstteeeecccmm
ABSTRACT
Deploying large language models (LLMs) on platforms with insufficient compu- tational resources remains a key challenge. Weight pruning is an efficient model compression technique that can reduce model size without retraining LLMs. How- ever, due to the massive number of parameters, it is infeasible to estimate the importance of weights globally, and most prior studies assign uniform spar- sity ratio across all layers. Recent findings reveal that layers contribute unevenly LLM performance, making it necessary to investigate layer-wise importance. Existing layer-wise sparsity allocation methods, such as OWL and DIP, rely on weight scoring and carefully designed score proxies estimate layer-wise impor- tance and sparsity ratios, while enforcing identical sparsity to blocks and projec- tion weights within a layer avoid performance degradation. propose layer-wise Sparsity Allocation (LSA) for LLM pruning, which quanti- fies layer-wise importance by evaluating the minimal linear reconstruction error of each transformer layer under the assumption that 50% of its least important weights are removed. Moreover, our method supports non-uniform sparsity al- location block- projection-level gramularity within layers. without incurring catastrophic performance degradation. Experimental results demonstrate that LSA maintains high performance high sparsity levels. At overall sparsity ratio of 70%, LSA surpasses state-of-the-ar methods across language modeling tasks and seven zero-shot tasks. Code is available at https://github.com Yazii///..
this work, we
INTRODUCTION
Large Language Models (LLMs) have revolutionized the field of Natural Language Processing (Brown et al.) 2020 OpenAl 2024 Touvron et al. 2023a). Nevertheless, their enormous parame- ter counts impose severe constraints on deployment due limited memory and computational re- sources. Parameter pruning (Hassibi et al. 1993) LeCun et al. 1989 Han et al.) 2015) has emerged as widely used and effective compression technique to alleviate this issue. Recent studies further demonstrate that extensive pruning can be achieved without fine-tuning (Frantar and Alistarh] 2023) Sun al. 2024a), showing that the parameters of LLMS can be safely removed without retraining.
SparseGPT (Frantar and Alistarh, 2023) and Wanda (Sun et al. 2024a) are two representative one- shot pruning approaches that rely on limited calibration data. They impose uniform sparsity across all layers and enable the model to be deployed without full retraining after pruning, while still achieving competitive inference performance under unstructured sparsity. Nevertheless, these meth- ods overlook the varying importance of different layers within LLMs. Recent works (Kovaleval et al. 2021 Sun et al. 2024b) have shown that LLM activations contain a small number of very
"Equal contribution
1 "ooresponniig author

---

Published as a conference paper at ICLR 2026
Viggora LRC
Lane
E35
Leyer2
Levnd
LIEE Mip
Abwrlice
Delodiay
Entolling
Figure 1: (a) illustrates the weight with minimal linear reconstruction error (LRE) for linear layers within FFN and attention blocks. (b) denotes the layer-wise LRE across all Transformer layers, computed by assuming removing 50% of the weights that contribute least to the reconstruction error in each layer. (c) represents the allocation different sparsity rates based on the principle that layers with lower reconstruction error should exhibit lower sparsity.
large magnitude features that critically affect model behavior. Dettmers et al. (2022); Puccetti et al. 2022); Lee et al. (2024) indicate that ignoring these outliers leads to severe performance degrada- tion in model compression. Hence, Outlier Weilhhe- layer-wise Sparsity (OWL) (Yin et al. 2024) allocates non-uniform sparsity ratios across layers according the layer-wise Outlier Distribution (LOD), which reflects the retention rate of weight outliers. It posits that layers with a higher pro- portion of weight outliers should be assigned lower sparsity. However, the empirically determined multiplier of the mean pruning score does not generalize well across different model types. In contrast, Dynamic Layer-wise Pruning (DLP) (Chen al.) (025) demonstrates that using fixed hyperparameter is suboptimal for LLMss. DLP employs the median of weight scores to character- ize layer redundancy and translates this measure into relative layer importance. By leveraging the median, DLP dynamically adapts to both the model's weight distribution and the calibration data, thereby obviating the need for predefined outlier selection criteria.
Both OWL and DLP use Wanda-style weight scoring, computed the product of weight magni- tudes and activation norms, followed by an empirical reduction function to obtain a score proxy for layerwise importance. Although DLP demonstrates that using the median the reduction function outperforms OWI, which relies on an empirical multiplier on the mean weight score, it has not been established as the optimal reduction function since only limited set of alternatives has been evaluated. Meanwhile, sparsity allocation at finer granularity than the Transformer layers, such as block level (self-attention and feed-forward blocks) and projection level, has not been sufficiently addressed by either method. We find that when such finer-grained sparsity allocation is applied, the performance of both OWL and DLP degrades substantially.
Hence, we identify two important questions that prior work has underexamined:
Is it necessary to identify superior reduce function for assessing layer importance?
Can non com-fffform sparsity be applied at finer granularity without degrading performance?
To answer these questions, we measure each layer's redundancy by computing the minimal linear reconstruction error between the dense and sparse models. Intuitively, layers with higher emrors typically contain more parameters exhibiting high scores (product of weight magnitudes and activa- tion norm) (Yin et al., 2024, Chen et al., 2025) and demonstrate greater tolerance to their removal. This resilience, coupled with their tendency towards uniform error contributions (rather than large outliers) upon weight removal, signifies higher redundancy. Conversely, layers with lower errors, while having smaller average impact, are more prone to containing weight outliers whose removal causes significant performance drops, reflecting lower redundancy. Accordingly, we transform the redundancy characterized by the layer-wise linear reconstruction emrr into relative importance score and allocate lower sparsity ratios to layers with lower redundancy. Furthermore, we extend method to finer granularity, including biock-wise and projection-wisg allocation.
2

---

Published as a conference paper at ICLR 2026
In this paper, we propose novel layer-wise Sparsity Allocation (LSA) method. The LSA pipeline is illustrated in Figure [ Compared with other sparsity allocation strategies, our LSA avoids the computation of weight scores as well the empirically designed reduce functions. Our method further enables the assignment of distinct sparsity ratios to different projections within the same Transformer layer, providing finer-gred finer-grained allocation that can lead to performance improvements. To the best of our knowledge, this is the first work to explore projection-wis sparsity allocation, providing novel perspective for future research.
Overall, the contributions of our work are as follows:
We propose a novel metric for measuring layer-wise wise importance that evaluates layers di- rectly using linear reconstruction error. This approach removes the need for weight- wise scoring and the manual choice of a reduce function.
We present an effective sparsity allocation method. Comprehensive experiments show that LSA accurately captures layer importance and consistently outperforms state-of-the-an lay- erwise sparsity allocation methods for LLMs.
We demonstrate that LSSA applies not only to layer-wise pruning but also to finer-grained sparsity allocations. To the best of our knowledge, LSA is the first approach to achieve finer granularity without catastrophic performance degradation.
RELATED WORK
2.1
LLM PRUNING
LLM pruning be categorized into unstructured pruning and structured pruning. Unstructured
pruning (LeCun et al. 1989 Hassibi et al. 1993 Frantar and Alistarh 20233 Sun et al. 2024a
Zhang et al. (0244) removes weights one by one according to weight-wise importance score. SparseGPT (Fantaar and Alistarh 2023) combines the weight amplitude with the inverse Hessian to form an importance score and performs parameter updates accordingly. Wanda (Sun al. 2024a)
combines the weight amplitude with the L2 norm of input activation as an importance metric and prunes weights that have minimal impact on model performance. In contrast, structured pruning (Ma)
et al. 2023 Hou et al. 2020 Khaki and Plataniotis] Kurtić et al. [023 Li et al. 2023 Zhang al., 2024b) focuses on removing coupled components such as attention heads or neurons to achieve more regularized, hardware-friendly structure. LLM-Pruner (Ma et al. 2023) estimates weight importance using first-order gradient information and ranks coupled groups within each Transformer layer by their total scores. OSSCAR (Meng et al.) (024) uses the SparseGPT metric to compute channel importance and iteratively removes and updates channels in the pruning list via local search and low-rank inverse Hessian updates. Our work primarily focuses unstructured pruning.
2.2 LAYER- WISE SPARSITY FOR PRUNING
SparseGPT and Wanda both employ uniform pruning, where each layer share the same sparsity level. However, Frankle and Carbin (0088)) shown that different layers contribute unequally to model performance, suggesting that uniform pruning strategies may be suboptimal they ignore layer-wise importance. Recent studies (Yin et al. (0224] Chen et al., |025] Gao et al. [i al. 2024a]
Lu et al. 2024 Huang al.) 2025 Sieberling et al.] (025) have explored non-uniform layer-wise pruning strategies. OWL Yin et al., 2024) allocates sparsity based on the ratio of weight outliers per layer, linking the outlier distribution to model performance. DLP (Chen al.) 2025) instead uses the median of weight scores as a measure of redundancy. However, both methods rely on the Wanda metric for scoring weights and limited layer-wise granularity. In contrast, our method leverages minimal linear reconstruction error to directly estimate layer redundancy, eliminating the need for wight-wise scoring and careful selection of reduce function. Moreover, we explore finer- grained sparsity allocation while maintaining performance comparable to layer-wise.
2.3
LINEAR RECONSTRUCTION ERROR IN PRUNING
Our work is also related to methods that minimize linear reconstruction error during pruning (He)
et al. 2017 Luo et al. 2017 Ding et al. 2019 Zhuang et al. 2018 Meng al.] 2024). Thinet
3

---

Published as a conference paper at ICLR 2026
(Luo et al. 2017) prunes filters by minimizing the output error between selected and full channels. Zhuang et al. (0118) introduces a discrimination-awann loss to jointly reconstruct errors and assess channel importance. However, both approaches require repeated forward passes, often proportional to the square of channel count, making them computationally prohibitive for LLMs. In contrast, our method improves efficiency by precomputing the error caused by removing each channel, greedily pruning the least important one while updating the remaining errors. Moreover, our method is extended to unstructured pruning by partitioning input channels into groups to accelerate pruning.
PRELIMINARIES
3.1 LINEAR RECONSTRUCTION ERROR
Let W E R be the weight matrix of fully connected layer, where and denote the number of output and input channels, respectively. Given N calibration samples, the corresponding input activations are X E NV where L is the sequence length. Pruning is represented by binary mask M E (0, 1 that indicates which weights are removed or retained. The linear reconstruc- tion error after pruning is defined as:
E Wx - (M O w)
Pruning for LLMs can be seen as reconstruction task: given sparsity p. remove some pre-trained weights minimize the output difference between the sparse and dense models. For large models, use hierarchical (layer-wise) reconstruction to split the problem into per-layer subproblems.
3.2 WEIGHT OUTLIERS BASED LAYER-WISE SPARSITY ALLOCATION
OWL Yin et al., 2024 integrates input activation with weight magnitude to identify weight out-
liers. For the 1-th layer with input activations X E RXLLe the OWL score is defined as
A = |W,, 1.j x 2' where |W! represents the magnitude of the weight, | | denotes the L2 norm of the input activations connected to the weight.
OWL introduces the Layer-wise Outlier Distribution (LOD), which quantifies the proportion of weight outliers of layers. An outlier any weight whose OWL score exceeds the layer mean by factor of m. The LOD is the ratio of these outlier weights to the total number of weights.
D' D' E
I(A >
CCO
where m is constant, A' represents the mean of A' and I((-) is the indicator function that equals its argument is true and 0 otherwise.
After computing the LOD, OWL assigns non- non-uniform sparsity across layers on the principle that layers with higher LOD are more important and should receive lower sparsity. However, selecting an appropriate m is challenging. DLP (Chen et al. 2025) reports that the optimal m varies across model types and parameter scale. To resolve this, DLP proposes estimating each layer's redundancy and converting those estimates into relative importance scores. Rather than using a fixed threshold, DLP represents the redundancy of the 1-th layer by the median of A', It normalizes the redundancy scores, subtracts the normalized vector from vector of ones to obtain importance scores, and then follows OWL's procedure to assign non-unifon con-unffomm sparsity.
METHODOLOGY
4.1 REDUNDANCY REPRESENTATION MINIMAL LINEAR RECONSTRUCTION ERROR
Let - = O (W..). The set of indices corresponding to the pruned input channels denoted as P. The linear reconstruction enror after structured pruning can be written as (a detailed derivation is provided in Appendix
4

---

Published as a conference paper at ICLR 2026
Co NxL E
Co NxL
W x,)²²
W..
Sts = iEP jEP
EES
(3)
EE?
iEP jEP
Given a sparsity level p. the number of input channels to prune is C2 [c pj. To minimize linear reconstruction error, we select C2 X C2 submatrix is minimal. However, exhaustive search is not feasible for LLMs, as the total number of such submatrices is given by C(c,,,)), Therefore, we initialize vector E RR with the diagonal entries of S, which represents the error associated with pruning specific input channel.
whose entry sum
00
By updating e, the linear reconstruction error for prun- ing input channel j. denoted Ej. is calculated as the sum of entries of the submatrix indexed by } U IP minus the sum of entries of the submatrix indexed by where IP de- notes the set of indices of already pruned input channels. Initially, P empty and satisfies the constraints. In each step, greedily select the channel with the smallest e,, Subsequently, we update e using the corresponding row and column of the matrix S, and set the selected chan- nel's error to infinity so cannot be chosen again.
Figure 2: Submatrix selection. (a) S with 16 input channel. (b) Submatrix dexed by [6, 1, 13, 5) has the minimal entry sum at 25% structured sparsity.
Note that H does not depend on the output channels. Define b. WR.).. By minimizing the entry sum of the submatrices of (k), we can perform channel- by-channel pruning, thereby achieving unstructured pruning. The layer linear reconstruction error equals the sum of the per-channel errors, Ei= EiEp(N) EjEPo, $ where pp(k)
((2)) with S(R)
0}. However, computing the full matrix (Ak) for every channel is computationally and memory intensive. A more efficient approach is to compute only the rows of ((k) that are required for each update, rather than precomputing the entire matrix. Concretely, initialize the matrix E IR x with ek. W k. H... then update only the affected entries of ek,: using the corresponding rows and columns of H during each pruning step.
i = arg min ek:
C + Ck + + = Ck.: + 2W (We. @ H..)
(4)
C + 00
To accelerate error computation, we assign the same sparsity level to all output channels. This will be achieved through vectorized parallel pruning in each update step, resulting in an index vector of length c, that specifies the pruned weight indices for every output channel. To further speed up the computation, we partition the input channels into groups of size B and compute the linear recon- struction error group by group. After processing the current group, we update the corresponding scores for the remaining unprocessed groups. The pseudo-code to minimize the linear reconstruction error of L.A is shown in Algorithm Detailed implementation can found in Appendix B
After applying Algorithm[] the model remains dense we do not prune any weights. Moreover, we demonstrate that the linear reconstruction error not only allocates layer-wise sparsity but also can provide importance metric for unstructured pruning. Appendix H compares our resulting metric with Wanda's 's under a uniform sparsity setting. The results show that pruning by error yields substantially better performance than Wanda. This improvement arises since our metric accounts for the additional error that accumulates when multiple weights are pruned simultancously, suggesting that using error to compute layer importance is preferable to Wanda-hased scores.
5

---

Published as a conference paper at ICLR 2026
4.2 LINEAR RECONSTRUCTION ERROR BASED LAYER-WISE PRUNING
For each layer, measure redundancy using the minimal error calculated by Algorithm D with p = 0.5. Notably, we find that the sparsity ratio used to compute the linear reconstruction error is not sensitive hyperparameter. Experiments in Appendix [ show that setting p below 70% produces similar performance as the sparsity allocation results are similar. Consequently, we fix p 50% in all experiments. Although layers with low linear reconstruction error imply that the average error introduced by reducing individual weights is small, large outliers are more likely to appear. This occurs because lower value is more susceptible to yield an outlier that is m times greater, indicating low redundancy. Conversely, layers with high linear reconstruction error tend to have a uniformly large error when individual weights are removed, making difficult to identify additional weight outliers, indicating high redundancy. This conclusion is supported by OWL: layers with high error consistently exhibit high sparsity and low LOD in OWL, as shown in Figure
To properly compare the importance, we nor- malize the error and convert them into relative importance. The importance score of the 1-th layer is defined as I'
Algorithm 1 Pseudocode of Minimizing Linear Reconstruction Error algorithm
E'/ E.
Intuitively, layers with lower importance scores should receive higher sparsity. However, di- rectly applying raw importance scores often yields poor results since their scale should adapt different target sparsity ratios pr. In- spired by OWL and DLP, we introduce a hy- perparameter /3 to rescale importance values ac- cording to By compressing the importance range to [0, 2,3), the 1-th layer's sparsity st is adjusted within (pr B, pr + B] by allocat- ing sparsity as & = pr + mean d) d. where d = I x 23, thereby limiting excessive pruning and preventing performance collapse.
Input: weight W, Hessian = X'x, sparsity p. number of output channels co. group size B
Output: minimal linear reconstruction error E
1: E+0
2: e + W @ W diag(H) // initialize linear reconstruction error for each weight
3: C2 + [ * *p]
4: for i B,2.B do
5:
Z <- 0z.xB
6:
for 0 c, do
7:
v f arg min e B // get pruned index
for each output channel
To fully validate LSA, investigate how spar- sity allocation granularity affects model per- formance. OWL and DLP adopt layer-wise granularity, assigning single importance score to all projections within Transformer layer.
8:
E < E 2k=1 Ck,u, // accumulate lin- ear reconstruction error
9:
e.ii+ B <- e,,ii+ + 2W wi++ X W.,, @ Ho,ii // update for current group
In contrast, we introduce two finer granular- 10:
Z. + W. v // record pruned weight
ity: block- and projection-wise. Under block- wise granularity, all projections within block share the same importance score. Let N de- note the number of weights of projections. To maintain the overall sparsity ratio pr, we com- pute the sparsity for each block or projection as s (pr x N (mean(d) dd) x mcan(NN)/NN
11:
end for
12:
C.i+B: 5 c. + oZHHi+B.iBB // up-
date for dense groups
13:
end for
return E
4.3 EMPIRICAL STUDY
Evaluation of Finer Granularity. Although DLP suggests that projection-wisg granularity spar- sity allocation can cause catastrophic performance degradation by disrupting inter-layer information flow, Table I shows that the degradation produced by our projection-wil wise is less severe than DLP's. For Llama 1-mmal-B 7B and Llama2-7B at 70% sparsity with allocation reduce performance. However, the performance gap among LSA with different granular- ity is minimal, whereas OWL and DLP suffer substantially larger declines. As shown in Figure 3 LSA with block-wise granularity is markedly more robust than OWL and DLP. While OWL and DLP are effective only within a narrow 3 range [0, 0.07]. LSA remains stable across a much wider interval [0, 0.17]. with the resulting sparsity ratios stay within [0, 1]. Notably, LSA's block- wise results even surpass layer-wise performance, demonstrating superior tolerance to hyperparameter variation. Moreover, Table 2 shows that LSA with block-wise attains lower perplexity than with
0.15, both block- and projection -eise

---

Published as a conference paper at ICLR 2026
layer-wing at 70% sparsity for LLaMA3. These findings indicate that LSA successfully generalizes to grerulaaall sparsity allocation, whereas both OWL and DLP fail to do so.
Relationship between Linear Reconstruction Error and Layer Importance. To further assess the effectiveness of LSA, we measure layer-wise redundancy in LLMs using linear reconstruction error. If errors are balanced across layers, it suggests that our method fail to distinguish layer impor- tance. The bar chart in the background of Figure Isows that layers differ in redundancy. indicating unequal contributions overall model performance. Several recent studies (Gromov et al. 2025a Sun et al. 2025 Li et al. 2024b) support this observation, reporting that deeper layers are often less critical than expected compared to shallower layers. The overall trend of the sparsity curve across layers indicates that shallow layers are more crucial and therefore require more parameters to pre- serve the model's core capabilities. In contrast, deeper layers appear to have less impact on overall performance, probably because they process more specialized information (Fan et al. 2024).
Comparison of sparsity allocation tech- Table 3: niques across three types of granularity on Wiki- the WikiText dataset at 70% sparsity. The best Text dataset: layer-, block-, and projection-wise. performance result is indicated in bold.
Table I:
Comparison of LOD and Perplexity on
Method
Lasc Spundly
LLaMAL-7B
LLaMA2-7B
Layre-atio Spandity
Donas
10.25
5.04
Layer
Block
Projection
Layee
Block
Projection
Dizse
5.68
5.56
5:47
5.47
5.47
LIAMA
Spanorary
194 11
14.15
OWL
18.93
25.93
20:68
29.91
331.36
12.65
Sponettir
DLP
12.74
34.78
23.06
18.68
10.54
25.05
Oum
17.87
18.25
19:46
18A3
20.40
21.15
57.91
80.32
THL
133.30 14
17
20.66
21.40
24822
25.55
34.6
Dosec
25.36
4.55
Spanorary
312 06
11:11
Table 2: Results of block-wise (B) granularity on perplexity using the LLaMA3 model with the WikiText dataset 70% sparsity. The best per- formance results are highlighted in bold.
356.85
12.96
361.00
16.81
55
Layer-wise Sparsity
LLaMA 3-8B
LLaMA
LLaMA 3.2-B
Method
3,2-1B
Dense
6.14
9.75
7.81
Uniform
48.39
126.70
72.94
SparseGPT
OWL
37.06
164.42
$7.48
DLP
40.12
112.29
54.86
Ours
39.56
98.95
51.45
0.04
625
8.88
0.0
812
2M4
6,6
Ours (B)
32.94
87.08
46.24
Uniform
114.34
437,06
149.49
Wanda
OWL
89.72
388.79
132.15
Figure 3: Perplexity of the LLaMA3-8B
DLP
112.95
340.68
166.05
Ours
98.75
306.84
154.19
model on the WikiText dataset, pruned using
Ours (B)
80.25
202.01
126.01
various B 70% sparsity in block-wise (B).
(a) LLaMAI-7B
(b) LLaMAI-13B
(c) LLaMAI-30B
nugensooii Erer
0.04
0.05
0.88
Indaee
-0.8
R.0.-
DMLL
Layer Yater Spardtty SAR
Reconttudion cudioo breer
0.025
0.020
DUP
Sperdiyy 5at
D.7
0.02
0.02
0.1
0.005
0.00
0.00
0.5
0.00
Layerlnnde
Layer midex
Layer Indox
Figure 4: Comparison of layer- wise sparsity distributions. The background bar chart illustrates the normalized linear reconstruction error. In each subplot, the horizontal axis represents the layer index, the left vertical axis denotes the error, and the right vertical axis indicates the layer-wise sparsity.
Comparison between OWL, DLP and Ours. To evaluate LSA's ability to preserve weight outliers, compare the LOD metric of LSA sparsity allocation with Uniform, OWI, and DLP. LOD is
7

---

Published as a conference paper at ICLR 2026
defined as the ratio of weight outliers. For fair comparison, use LLaMA1-13B and LLaMA2- 13B with m = and aggregate LOD in all layers. Table 3 illlssrates that all layer-wise methods increase LOD, the proportion of outliers, compared outliers most effectively and performs best at high sparsity, achieving the highest LOD and the lowest perplexity. These results demonstrate that method outperforms both OWL and DLP.
uniform allocation. Our method preserves
5 EXPERIMENTS
5.1
EXPERIMENTAL. SETUP
Models and Datasets. To demonstrate the effectiveness of LSA, we evaluate its performance across various LLMs, including LLaMAl (7B/13B/30B) (Touvron et al. 2023a), LLaMA2 (7B/13B) (Tou-
vron et al. 2023b), LLaMA3-8B (Grttafiori et al. 2024), as well as LLMMA3.2- IB, LLaMA3 2 2-3B,
Vicuna-7B (Chiang et al. 2023) Mistral-7B (Tingg al. 2023), and Qwen-7B (Bai et al. 2023
Our evaluation encompasses language modeling proficiency and zero-shot capabilities of sparse LLMs. Language modeling is measured by perplexity on the WikiText (Merity et al. 2017). PTB Marcus et al. 1994), and C4 (Raffel et al. (020) validation datasets. For zero-shot evaluation, we report accuracy on seven commonsense benchmarks from EleutherAI LM Harness (Gao al., 2023), including BoolQ (Clark et al. (019), PIQA (Bisk et al.] 2201)), HellaSway (Zellers et al. 2019, WinoGrande (Sakaguehi et al.) 2019), ARC-easy (Boratko et al. 2018 ARC-challenge (Bo-
ratko et al., 2018) and OpenbookQA (Mihaylov et al. 2018).
5.2 BASELINES.
We select three baseline pruning methods for LLM: Magnitude (JAISWAL et al. 2023). SparseGPT FFrantar and Alistarh| 2023) and Wanda (Sun et al. 2024a). We then assess these baselines when combined with non-uniform sparsity allocation produced by OWL, DLP and LSA. Experiments concentrate high sparsity regime, no less than 50%. To ensure statistical reliability, each layer- wise experiment is run at least five times with different seeds, and the average result is reported. For calibration, each method randomly samples 128 examples of 2048 tokens from the C4 dataset.
Table 4: Perplexity results on WikiText dataset Table 5: Comparison of mean zero-shot accura- with 70% unstructured sparsity across the cies (%) for pruned LLaMAI and LLaMA2 mod- LLaMAI and LLaMA2 models. The best per- els at 70% unstructured sparsity. The best perfor- formance result is indicated in bold.
mance result is indicated in bold.
Method
Layer-wise Sparsity
LLaMA1
LLaMA2
Method
Layer-wise Sparsity
LLaMA1
LLaMA2
7B
13B
30B
7B
13B
7B
13B
30B
Dense
5,68
5.09
4.10
5.47
4.88
Dense
58.43
61.02
62.43
59.03
60.95
Uniform
4.9c4
8.5e4
9.762
5.0c4
2.1e2
Uniform
35.51
36.25
32.82
33.66
37.20
Magnitude
OWL.
1.964
2.1e4
2.4e2
6.0c4
59.31
Magnitude
OW1.
37.30
37.62
32.79
35.26
40.26
DLP
4.1e3
7.5e3
96.85
7.9e3
$2.16
DLP
38.35
38.28
40.49
37.80
44.22
Ours
4.4e3
8.7c3
93.42
9.0c3
45.16
Ours
38.66
38.65
41.00
39.17
44.38
Uniform
27.18
20.36
12.34
26.32
19.53
Uniform
42.75
45.46
52.54
41.51
45.44
SpurseGPT
OWL
18.98
14.15
10.69
20.68
14.31
SparseGPT
OWL
44.23
48.16
53.53
44.49
48.18
DLP
17.78
12.87
18.68
13.39
DLP
44.26
48.79
53.65
45.27
48.88
Ours
17.57
12.45
9.49
18.63
12.56
Ours
44.77
48.91
53.92
45.53
49.34
Uniform
83,45
53.47
17.67
75.01
45.26
Uniform
37.44
39.89
50.81
35.82
38,30
Wanda
OWL.
24.85
16.49
11.02
30.03
18.16
Wanda
OWL
42.95
46.98
53.42
41.11
46.94
20.89
13.94
22.85
16.80
DLP
44.31
48.84
53.67
43.77
47.66
Ours
20.66
13.65
9.77
22.88
15.36
Ours
44.19
48.99
53.34
43.86
47.96
5.3
MAIN RESULTS
Language Modelling. repoe perplexity of several LLM pruning methods at 70% sparsity on the WikiText dataset. Results for PTB and C4 appear in Appendix K Additionally, we provide results for different sparsity ratios in Appendix [D] Results indicate that LSA is an effective layer- wise pruning method that outperforms both Uniform and OWL across all model sizes. In most cases, LSA achieves lower perplexity than DLP. For example, on LLaMA2- 13B, LSA reduces perplexity compared to DL.P by 7.00, 0.83, and 1 ..44 for Magnitude, SparseGPT, and Wanda, respectively.
8

---

Published as a conference paper at ICLR 2026
Table 6: End-to-end decoding latency and throughput the LLaMA2-7B-chtthhf model using the DeepSparse inference engine with LSA.
Sparsity
Dense
10%
20%
30%
40%
50%
60%
70%
%
90%
Latency (ms)
828.76
876,48
805.76
795.49
809.78
390.84
319.61
278.24
251.17
218.23
Throughput (tokens/sec)
1.21
1.14
1.24
1.26
1.23
2.56
3.13
3.59
3,98
4.58
Speedup
1.0x
0.9x
1.0x
1.0x
1.0x
2.1x
2.6x
3.0x
3.3x
3.8x
Zero-Shot Tasks. 5 reports the average zero-shot accuracy of pruned models at 70% sparsity across seven zero-shot tasks. Additionally, we provide detailed accuracy for each specific task in Appendix L Overall, LSA achieves the best performance in nearly all settings, demonstrating its promise for more challenging zero-shot downstream tasks. For example, LSA improves average accuracy by 1.37% for Magnitude on ILaMMA-7B and by 0.51% % for SparseGPT on LLaMA1-7B,
Inference Speedup. To evaluate inference acceleration of sparse LLMs produced by our method, (Touvron et al., 2023b) to ONNX and then measure end-to-end
we convert ALaMA2--B-chatte decoding latency using the DeepSparse (Kurtic et al., 2023) inference engine on an Intel (R) Xeon (R) CPU ES-2699 with 44 cores. As shown in Table LSA yields substantial inference speedup compared to dense model. In particular, the speedup exceeds 3.0x at sparsities above 70%.
Pruning Efficiency. To evaluate the computational cost of our method, compare the empirical pruning time LSA and baselines. We focus exclusively on pruning time, excluding the forward propagation and sparsity allocation, since non-uniform sparsity can be pre-calculated. Table 7 trates that the differences between them are minimal. Moreover, as the model scale increases, the pruning time of LSA becomes even lower than that of the uniform baseline. These results indicate that LSA has the potential to accelerate the identification of weights for removal in pruning.
Performance on Small Models. We evaluate the performance of LSA and other layer-wise methods on Llama3.2 at 70% sparsity. As shown in Table 2 LSA achieves the lowest perplexity on the Wiki- Text dataset in nearly all cases. Notably, block-wise outperforms layer-wise with LSA, which suggests that the identical sparsity in layer-wise limits performance improvements derived from sparsity allocation. Detailed sparsity allocation appear in Ap- pendix F These results further support the effectiveness of LSA.
Table 7: Comparison of time LLaMAI for computing the pruning metric (seconds).
Gas
3.5
1.55
3,38
+53.90
001.35
119640
411.4
Wimds
Unierr
133
Fine- Tine-Tuning Performance. We investigate parameter-efficient fine-tuning (PEFT) techniques to recover the performance of LLMSs pruned with LSA. Specifically, we employ widely recognized PEFT method, LoRA (Hu et al. 202T), and fix pruning mask throughout fine-tuning. & that fine-tuning substantially restores the performance of the pruned models.
Table 8: Perplexity of vari- ous LLMs pruned by LSA using Wanda on the Wikitext datasets.
Madel
Mathed
Spunsity
Perplecity
LLSMAI-7
Wilhor FT
0.7
20 56
We evaluate LSA on additional advanced L.MMs. As shown in Table LSA reduces per- plexity at multiple sparsity levels and consistently outperforms uni- form sparsity. T0 zero-shot accuracy for three models. On Qwen3-8B, OWL and DLP suffer substantially greater degrada- tion than uniform sparsity when pruned with SparseGPT, whereas LSA exhibits only a minor drop. After applying a projection wise variant of LSA. performance improves further and exceeds that of uniform sparsity, indicating that our method better captures redun- dancy than OWI and DI.P. similar pattern appears for Qwen2.5- 5- 7B, we also adopt block-wise allocation scheme address the limitations of layer-wise granularity. Overall, these results demon- strate LSA's ability to generalize across LLM architectures.
Performance on More Advanced LLMs.
EJSMA1-11
LLAMA2-7B
With PT
Table 9: WikiText perplexity of various LLMs pruned by Uni- form and Ours using Wanda.
Vicoos77
Unitien
13.13
50.76
2195 m0
Okn
11.63
37.63
365.8
Oun
9.66
31.82
236.99
Queas-B
Unifoon
25.71
586.1
216755.20
$799.00
Integration with Other Compression Methods. Although our
method focused unstructured pruning in previous sections, it can also be applied to hardware- friendly settings. In Appendix C we combine it with structured pruning, N:M sparsity, and quanti- zation.

---

Published as a conference paper at ICLR 2026
Integration with More Advanced Pruning Methods We also evaluate LSA advanced LLM pruming methods. ADMM-Gras (Boža 2024) proposes using the Alternating Direction Method of Multipliers (ADMM) for pruming, which is more effective than prior state-of-the-ar methods such as SparseGPT and Wanda. Therefore, we combine ADMM-Grad with non-uniform sparsity to further validate LSA.. As shown in Table 11 LSA maintains superior performance relative to all baselines.
CONCLUSION
In this paper, we introduce linear reconstruction error based layer-wise pruning method that di- rectly measures layer importance without requiring wight-wise scoring or reduce function design. We compute the minimal reconstruction error at 50% sparsity assess redundancy and convert these values into relative layer importance. Layers with lower importance are assigned higher sparsity. Notably, method achieves substantial performance gains, outperforming existing state-of- of-thh--rt approaches. Moreover, our method successfully generalizes to finer granularity sparsity allocation, maintaining competitive performance compared to layer-wise approaches. These results highlight the promise of finer granularity sparsity allocation and suggest several directions for future work.
Table 10: Accuracy (%) of more advanced models on seven zero-shot tasks at 70% unstructured sparsity with various sparsity allocation methods, including block-wise (B) and projection-wise(( granularity. The best performance result is indicated in bold.
Model
Method
Layer-wisc Sparsity
WinoGrande HellaSwag BoolQ PIQA OBQA
ARC-e ARC-c Mean
Dense
72.61
60.17
81.59
79.65
34.8
80.09
50.17
65.58
Uniform
57.30
33.74
66.39
62.89
15.00
44.95
22.01
43.18
SparseGPT
OWL
60.93
36.67
70.58
65.40
16,20
48.27
23.81
45.98
LLaMA3-8B
Ours
62.12
37.93
74.53
65.89
18.60
48.95
25.09
47.59
Uniform
48.22
27.28
50.43
55.60
13,60
32.15
17.66
34,99
Wanda
OWL
49.49
28.40
61.50
57.83
13,40
35.52
17.66
37.69
52.41
29.51
58.53
60
14.00
Ours
53.91
30.45
60.49
60.83
15.00
41.04
19.03
40.11
Dense
73.01
60.04
85.11
78.78
33,20
80.47
47.78
65.45
Uniform
61.72
40.00
73.24
68.93
20.00
63.05
29.18
50.88
OWL
61.09
38.02
64.62
67.63
19.20
$9.95
27.39
48.27
SparseGPT
DLP
61.96
38.31
18.80
26.28
47.79
Qwen2.5-7B
Ours
62.67
39.04
77.40
65.78
19.00
56.10
27.65
49.66
Ourx(B)
64.33
41.68
72.69
68.99
23.00
63.34
29.69
51.82
Uniform
53.04
30.59
62.02
61.81
15.80
45.75
20.56
41.37
OWL
30.68
40.95
DLP
56.67
33.38
62.23
63.00
16.60
45.62
21.16
42.67
Ours
56.91
33.41
62.57
63.06
16.80
46.68
23.29
43.24
Dense
67.64
57.12
86.64
76.88
31.00
83.54
55.89
65.53
Uniform
62.12
38,57
73.30
68.77
21,20
61.45
29.52
OWT
60.22
37.17
66.54
67.41
2.1.40
58.75
27.73
48.46
SpanseGPT
$7.49
29.61
Ours
64.72
38.31
68.44
67.74
22.40
$9.93
31.31
50.41
Qwen3-8B
Ours(B)
61.57
Ours(C)
64.01
39.49
71.96
69.15
23.80
62.92
31.48
51.83
Uniform
Wanda
OWL
52.09
29.52
61.99
61.15
15,20
47.18
18.86
40,86
Ours
57.54
32.84
62.39
63.76
16.40
$1.60
24.15
44.10
Table 11: Perplexity on different validation datasets and accuracy (%) on seven zero-shot tasks of LLLMA3-8B models 70% sparsity using ADMM-Grad pruning with various layer-wise sparsity methods. The best performance result is indicated in bold.
Layer-wise Spursity
Perplexity.
Accuracy($5)1
WikiText
PTB
C4
WinoGrande
HellaSwag
BoolQ
PIQA
OBQA
ARC-e
ARC-c
Mean
Uniform
29.20
48.82
37.95
59,98
35.63
68.56
64.36
18.00
48.78
21.93
45.32
OWL
27.36
41.77
32.34
62.83
38.06
72.81
65.07
18.20
52.86
24.91
47.82
DLP
28.26
45.77
31.12
63.30
38.81
72.84
20.00
51.14
24.49
Ours
27.97
45.62
30.09
64.80
39.29
74.46
66.38
21.60
51.05
25.77
49.05
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are feo-shot learners, 2020. URL
http://arxiv/ org
abwpg/////////a/.aa//./////aaaea abs 2005 14165
OpenAl. GPT-4 technical report, 2024. URL UBLL:ttte//aarrr...rr/////2222.0
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023a. URLhttpp /tattp://arii....o/ /tt//aiivvv/..////// org/ 2ttp://bb/330221133 13971
Babak Hassibi, David Stork, and Gregory Wolff. Optimal Brain Surgeon: Extensions and perfor- mance comparisons. Advances in Neural Information Processing Systems, volume 6. Morgan- Kaufmann, 1993. URL https: http:://ppooeeeeigg..oooom 8 9acfe4 4f 9e Abst.bat.. oe httsssccppppppg/ http://ss..pppppee//////aaeeeee
Yann LeCun, John Denker, and Sara Solla. Optimal
Brain Damage. In Ad-
vances in Neural Information Processing Systems, volume 2. Morgan-Kauffamn 1989.
nwwrips........ cc wtt.ccpppppeee wwcpppper/eeeeeeeeeeeeeeeeeeeeee 989/hash/ efficient neural networks, 2015. URL Abstt htmml
URL https::/proceed/:
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for 62626
Elias Frantar and Dan Alistarh. SparseGPT: Massive Language Models Can be Accurately Pruned in One- One-Shot. In Proceedings of the 40th International Conference on Machine Learning, pages 10323 10337. PMLR, July 2023. URL https: petps://proceedee wtt..nnns......../2/2/2222 Frantar23a. html
mlr. press/v202/
ISSN: (6600-4498
Mingjic Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A SIMPLE AND EFFECTIVE PRUNING APPROACH FOR LARGE LANGUAGE MODELS. In The Twelfth International Conference on Learning Representations 2024a.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier di- mensions that disrupt transformers, 2021. URL U www.i.iivvvv///a./ wttp://wwwrrgabb///l/o/ 06990
Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models, 2024b. URLLtttpe///axxv.crrr/bbbb/
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. int8:n 8-bit matrix mul- tiplication for transformers at scale. In Advances in Neural Information Processing Systems, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell'Orleti Outliers dimensions that disrupt transformers are driven by frequency, 2022. URL http:///cxiw..rr//bb0
2205. 11380
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. OWQ: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models, 2024. URL http:://aaxiv... org/ abs abw. 02272
Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, AJAY KUMAR JAISWAL, Mykola Pechenizkiy, Yi Liang. Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. In Forty-first International Conference on Machine Learning, 2024, URL hetps: d=ahEm33 2P
//openrevi ew. vet/forrmm wwwtrrrwmmmmmmmammememmmm

---

Published as a conference paper at ICLR 2026
Yuli Chen, Bo Cheng, Jiale Han, Yingying Zhang. Yingting Li, and Shuhao Zhang. DLP: Dy- namic layerwise pruning in large language models. In Forty-second International Conference on Machine Learning. 2025. URLAttts://pppeeriiiee net. ferttt/oorummmdde- http://wwnnntooorrooooooodoooppo
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. PLUG-AND-PLAY: AN EFFICIENT POST-TRAINING PRUNING METHOD FOR LARGE LANGUAGE MODELS. In The Twelfth International Conference on Learning Representations,
2024a.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-Pruner: On the Structural Pruning of Large Language Models. Advances in Neural Information Processing Systems, 36:21702-21720, December 2023. URL TBLLattss//prrocedinss..ee meg..eeeddddnnne....... htcc/paper_flles/ppppee h0tp////044455555100007744410110 Abter ore.aaaeeeeeettttcccoooooreeeeee Conference. html
Lu Hou, Zhiqi Huang, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors. Advances in Neural Information Processing Systems, volume 33, pages 9782-9793. Cur- ran Associates, Inc., 2020. URL /tt////te///ppooeddng...eessse.. fil//fille.. 6f5216556662001 6 6 8d8 ott.aa/e...................626.3
files/
Samir Khaki and Konstantinos N. Plataniotis. The need for speed: Pruning transformers with one
recipe. URL http::tttp:aattaeee /ttp://aaxiv..v http://w..or/aab//20311.222
Eldar Kurtić, Elias Frantar, and Dan Alistarh. ZipLM: Inference-aware structured pruning of language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko,
M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 65597-65617. Curran Associates, Inc., 2023. URL
https::/procceeingg..eeeeee nwwrips. cwppppprrccppppprrreeeeeepppppe. ced4 aa5500aeraaaaaacoocooaaaaa222222
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation. In Proceedings of the 40th International Conference on Machine Learning, pages 20336-20350. PMLR. July 2023. URL htttps://proceeeeee
mlr wtl...press/2221
1i23ap.her html
ISSN: 2640-3498.
Mingyang Zhang. Hao Chen, Chunhua Shen, Zhen Yang. Linlin Ou, Xinyi Yu, and Bohan Zhuang. LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efffcien Fine-Tuning, August
2024b. URttp:////axxx
wttporgggaab////eeee 18403 arXiv:2305.18403 [es].
Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. OSSCAR: One-shot structured pruning in vision and language models with combi- natorial optimization. 2024.
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Train- able Neural Networks, March 2018a. URL htttp:://arxi...rt:: lww.hh/rx.x.....oc...eeeo.. http:/ghhn...eeet//////1....
03635v5
Chang Gao, Kang Zhao, Jianfei Chen, and Liping Jing. Maximum redundacyyn A principle- driven layerwise sparsity allocation for LLMs. URL
18377
version: I.
Lujun Li. Peijie Dong, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models. 2024a. URL /
Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang. Michael W. Mahoney, and Yao- qing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, Advances in Neural In- Curran Associates, Inc., httpeeee_eeeeeeeeeeeee222/ fi ffee:
formation Processing Systems. volume 37. pages 9117-9152. 2024. URL ott.ttoesrrpppeeedss...eerrr.o..
12

---

Published as a conference paper at ICLR 2026
Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Fei Chao, and Rongrong Ji. Determining layer- wise sparsity for large language models through a theoretical perspective. In Forty-second Interna- tional Conference on Machine Learning. 2025. beattps://openreveeet
forum?
Nd=ott
Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, and Dan Alistarh. Evopress: Accurate dynamic model compression via evolutionary search. In Forty-second International Conference on Ma- chine Learning, 2025. URL URL/ttps:: http::://////m http::::///eeeeeiiieee nnt forum 20
forum
Yihui Hc, Xiangyu Zhang, and Jian Sun. Channel Pruning for Accelerating Very Deep Neural Networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1398-
1406, Venice, October 2017. IEEE. ISBN 978-1-5386-10322 doi: 10.1 10.1109/CC222211..555 URL htttt::://eeeeeloorrtttee leee. htt.:///w.........ooocccoo/////a
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression. In 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated Oracle Filter Pruning for Destructive CNN Width Optimization. In Proceedings of the 36th International Conference on Machine Learning, pages 1607-1616. PMLR, May 2019. URL ISSN: 2640-3498.
mlr www.ppppws....ppp (ing//digg99... html
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Jun- zhou Huang, and Jinhui Zhu. Diserimination-aware Channel Pruning for Deep Neural Net- works. In Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https: 15577c/9Bc1111114444443333 -tt. html
/ttttp:://proceeee http:///o//eeee//eoeoooooooo ngs. ngs. nwwrips..p..... ptcc/paper/22155 B//hh/hhhh/// Abstract
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Damiel A. Roberts. The Unreasonable Ineffectiveness of the Deeper Layers, March 2025a. ARRLattt://axxxi abs/2403. 17887
arXiv:2403.1 17887 [es].
Wenfang Sun, Xinyuan Song, Pengxiang Li, Lu Yin, Yefeng Zheng, and Shiwei Liu. The Curse of Depth in Large Language Models, July 2025. URL http:/tp//aaxx..oo////a///////// 05795 arXiv:2502.05795 [cs].
Pengxiang Li, Lu Yin, and Shiwei Liu. Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN, December 2024b.
13795
arXiv:2412.13799 [cs].
Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, and Zhongyuan Wang. Not All Layers of LLMS Are Necessary During Inference, July 2024. URL
http://arxiv. org/ abs / 2403 02181 arXiv:2403.02188 [es].
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami. Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunoy, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https: hampss///axiiv org/ htsp/bbs/2330 2307 09288
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan,
13

---

Published as a conference paper at ICLR 2026
Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee. Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Ject Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lec, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, K., and et al. The Iaaaa 3 herd of models, 2024. URL httpt: /aattps org/ abs 2 2407 21783
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
-00-022---33--iii-000 30-vicuna/
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, October 2023. URL http://arxivv org/ /aobs/2310 2et.//2311..222555. arXiv:2310.068)) [cs].
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li. Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL
https: // arxiv. org/ abs 2309. 16609 16609
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mix- ture models. In International Conference Learning Representations, 2017. URL https:
//openreview wwweeeeoowemmeeeeeeeeeeeeeoe Byyjmmuuddd
Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Fer- guson, Karen Katz, and Britta Schasberger. The Penn Treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994, 1994. URL http::ttts/// http://cclatiin http:////caaathoooyo... org /H94 1020/
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/jmlr. wwwpll/orgppapers////22
074.000000 html
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL httttttp:///eeeeae org/records/ 10256836
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill
14

---

Published as a conference paper at ICLR 2026
Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computatioonal Linguistics: Human Lan- guage Technologies, Volume I (Long and Short Papers), pages 9924-2936, Minncapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 00.18653/11/119-133 1300. URL. https://acc antho ott.llo//...11110000000
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL arxia. www.xi..or//ab/11100
11641
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Märquez, editors. Proceedings the 57th Anmual Meeting of the Association for Computational Linguisties, pages 1790, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.186553v1//119-444 URL /tR./httss:///aaaneeeet http://aaaaaahooo//ooo
ogy. wwwphhhoogg..og.rrgooooo org.oogg
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale, 2019. URL
10641
Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj. Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, Ryan Musa, Kartik Talamadupula, and Michael Witbrock. A systematic classification of knowledge, reasoning, and context within the ARC dataset. In Eunsol Choi, Minjoon Seo, Danqi Chen, Robin Jia, and Jonathan Berant, editors, Proceedings the Workshop Machine Reading for Question Answering, pages 60-70, Melbourne, Australia, July 2018. Association for Computational Lin- guistics. doi: 18.18653/v 1/W18-2601 URL htt.s: //wc/aacaaaaaaaaooo ogy. org/ W1 W18-26077
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun 'chi Tsujii, editors, Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pages 2381 2331-2391, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/11/D18- 1260. URL
1250
AJAY KUMAR JAISWAL, Shiwei Liu, Tianlong Chen, and Zhangyang Wang. The emergence of essential sparsity in large pre-trained models: The weights that matter. In Seirty-svemtth Confer- ence on Neural Information Processing Systems, 2023. URL htttps:///peeeeeeeer..... net/
Forum?id=bUUhwbbs.
Eldar Kurtic, Denis Kuznedelev, Elias Frantar, Michael Goin, and Dan Alistarh. Sparse fine-tuning for inference acceleration of large language models, 2023.
2310 0. 06927
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021. URLhhttss: org abs / 22 200660000 09685
Vladimír Boža. Fast and effective weight update for pruned large language models. Transactions Machine Learning Research, 2024. ISSN 2835-8856. URL httpps///peenreview.. htteeerreieeweeeeeeeeeeee
forum? d= hhchhhppp1
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference Computer Vision (ICCV), December 2015.
Wei Sun, Aojun Zhou, Sander Stuijk, Rob Wijnhoven, Andrew Oakleigh Nelson, hong- sheng Li, and Henk Corporaal. DominoSearch: Find layer-wise fine-grained N:M sparse schemes from dense neural networks. In Advances in Neural Informa- fion Processing Systems. volume 34, pages 20721-20733 Curran Associates, Inc., 2021. URL httppp////pcccee........r.ooooo. /ptp://scc///ppppeeeeeeepp/////p hhtp://6672222003322222222050522
15

---

Published as a conference paper at ICLR 2026
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate Post- Training Quantization for Generative Pre-trained Transformers, March 2023. URL http: org/ hte//ab//at wwtpabbs....2..... 22220.13322 arXiv:2210.17323 [cs].
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Train- able Neural Networks, March 2018b.
URL hog.aceee//aaa....o.oo....e.eeee www.ongwww...eanoo/oootooaaoeaae
03635v5
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse con- nectivity inspired by network science. Nature Communications, 9(1), June 2018. ISSN http://xxxxxc..orggg.o. URL http::/dx.cccc.. dei 1038/
2041-1723. doi: 10.1088/41166-011--0455 6:1467-018-04516-
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training, 2022. URL http:::/aax.v..or/////aa///////a atg.ab///aab2222/// 02643
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Dan Roberts. The unreasonable ineffectiveness of the deeper layers. In The Thirteenth International Confer- ence on Learning Representations, 2025b. URL https:://penreviiew net/ fort/forrmmi ner
16

---

Published as a conference paper at ICLR 2026
A MINIMIZE LINEAR RECONSTRUCTION ERROR FOR STRUCTURED PRUNING
Efficient Representation of Linear Reconstruction Error. In structured pruning, each row or column of the mask matrix M is entirely composed of either of W by W the i-th column of X by X and the i-th column of M by M. By setting all elements of M as O, we can prune W.,- w,. The corresponding linear reconstruction error can be expressed as:
Denote the i-th column
E
[(W
W 1...... W. -1,,0 O, W Wi+1.., W W.,e))2 1x
NxL.
(W (W.). (2)
NxL
W2 x2
P. q.
NxL
X²
W.
(5)
By setting both M., and M., to 0, we prune the columns W. and W..j- The corresponding linear reconstruction error can be expressed as:
Gu NxL
E
Σ
(W.
F1
Nxt
(W p.i X q.i
W wy X )
6o NxL
Σ
Σ
W + 2W p.j W X,x
p.j
X.
+X XXX, W XX
(6)
Let Sw = W W. H = X and S H @ Sw, then the above expression can be rewritten as follows:
E= H.Swa + H,ySw,j + H.SS., HH,Sw Ss Sjj + Sij Sjz
Let P denote the set of indices corresponding to the pruned input channels. Based on Eq. 66 and
Eq. (7). it can be demonstrated that the linear reconstruction error can be represented
17

---

Published as a conference paper at ICLR 2026
NXL
E
W
NXL
W p. X g.)
EE
NXL
W W.. Xx
IET JEP
Sij
(8)
EE JEP
We call this an efficient representation of the linear reconstruction error since it enables computing the error for any index set without requiring multiple forward passes.
Greedy Strategy. Given target sparsity level p. the number of input channels to be pruned is de- fined as c, = [c pj. To minimize the linear reconstruction error, we select C2 C2 submatrix of S whose entries have the smallest total sum. In principle, recursion can enumerate all possible sub- matrices identify the one with the minimal entry sum, and the corresponding indices provide the optimal pruning set. However, this approach is impractical for large models, since the total number of candidate submatrices is given by C(e,, cs). For instance, for the smallest 7B model of LLaMA2 with hidden_size 4096, pruning just four channels would require examining 11,710,951, 11,11,,,8488, .88888889 960 submatrices.
One feasible strategy is to utilize the diagonal elements of matrix S as importance measure vector E 1R which reflects the linear reconstruction error when pruning a specific input channel. After removing channels in I., updating makes Ej equal to the of entries of the submatrix indexed by {j} UPP minus the sum entries of the submatrix indexed by IP.
Initially, the set P is empty and the importance vector satisfies the preseribed constraints. At each pruming step, we greedily remove the channel with the smallest linear reconstruction error score, and update by adding the corresponding row and column of S. We then mark the linear reconstruction error of the selected channel as infinite (i.e., adding this channel to the set IP), thus preventing its ce-selection during subsequent pruning.
arg min
(9)
c + e c S.. + S., =
(10)
c +
(11)
Eq. 100 utilizes the symmetric property of S. It is straightforward to verify that employing Eq. D.. (OO,. and (IT) select channels for inclusion in set and update e ensures that, whenever channel j is pruned, the linear reconstruction error E, equals to the sum of entries of the submatrix indexed by [i} U.P minus the sum of entries of the submatrix indexed by P.
B PYTORCH CODE IMPLEMENTATION: CALCULATE MINIMAL LINEAR RECONSTRUCTION ERROR
def fix po_upeuueee__ (weight, inputs, p=0.5, block_size=122 co, ci weight. shape inputs inputs. reshape (-1, )i))
sx inputs. inputs
# initialize linear reconstruction error for each weight score (weight 2) torch.dias (sx)
18

---

Published as a conference paper at ICLR 2026
re ___oosttuct 0.
prune_num (ci * p)
prune_ide []
for 11 in range (0, ei, block_sizo)
i2 = =min + block_size, ci)
w1 weight i1:i2] # weight in current group
w2 weightl:, 12:] $ weight in remain groups
sccrel = scorel:, i1:12] # error in urrent group
score2 scorel;, 12:1 # error in remain groups
(x(11:22 il:i2] # update hessian for current group
sx2 sx [xi1:12, i2:] update hessian for remain groups
err =
(wl)
for i in range (prune :
## get index of weight to be pruned for each output channel idx = torch. (scorel, . (usqueezo
## accumulatc linear reconstruction error
re_construct += terch, sum (score1. gather (1, idx)) prune_idx.appemn (idx il)
w - w1 w.l.gahher(( (1, idx) # gather selected weights sxl lidx. squeeze
change = w1
scorel += 2 change update
ecorel.scatter scdtterrddiiil1,,, index=idx, value=torch, inf) mark
|| record weight for updating error in remain groups err. scatterrddtmmll, index-idx, sre=w)
change w2 (err 0 sx2)
score2 += 2 * change update
return re_censtruct
C INTEGRATION WITH OTHER COMPRESSION METHODS
Experiments in the previous sections demonstrated LSA's ability to distinguish layer importance in unstructured pruning. To unleash the potential of LSA in more scenarios, explore the applica- tion of LSA in structured pruning, N:M sparsity and quantization by combining LSA with these compression methods.
C.1 INTEGRATION WITH STRUCTURED PRUNING
Although LSA is primarily designed for unstructured pruning. it extends naturally to structured prun- ing. Following the approach of LLM-Pruner (Ma et al. 2023), which removes coupled components such attention heads neurons to produce more regular structure and enable acceleration, use BookCorpus (Zhu et al. 2015) as the calibration dataset and assess the performance of LLaMA1- 7B combined with method. As shown in Table [2 LSA successfully improves performance of structured pruning by allocating non-uniform sparsity across layers.
C.2 INTEGRATION WITH N:M SPARSITY
We also apply LSA to N:M sparsity. Following the approach presented in DominoSearch (Sun al,, 2021], we adopt mixed N:8 configuration: rather than using the same N for every layer, we assign different values across layers while preserving the overall sparsity. Since layer-wise sparsity dif- ferences among allocation methods are statistically insignificant and the integer quantization effect of N causes all methods to yield near-identie cear-identical results, we omit OWL and DLP from the reported results. Table 13 shows that the mixed N:M configuration combined with LSA consistently outper- forms uniform N:M allocation. Notably, high sparsity regimes with 3:8 and 2:8 configurations, L.S yields substantial gains, approximately 2x and 15x improvements in performance, respectively.
19

---

Published as a conference paper at ICLR 2026
Table 12: Perplexity of the LLaMA1-7B models on WikiText and PTB dataset using LLM-Pruner. Best performance indicated in bold.
Dataset
wise Sparsity
20%
40%
60%
Uniform
11.04
269.36
1397.30
OWL
9.96
152.80
3792.64
WikiText
DLP
10.73
166.49
1043.75
Ours
9.23
157.92
371.97
Uniform
256.94
1036.08
3549.23
OWL
139.67
1418.18
4011.72
PTB
DLP
65.67
914.04
3093.68
Ours
161.69
752.84
1240.52
Table 13: Perplexity of mixed N:M sparsity (where N refers to non-zer zero weights) with LLaMA1-7B on the Wiki'Tett dataset. The best performance result is indicated in bold.
N:M Sparsity Structure
Layer-wise Sparsity
2:8
3:8
4:8
Uniform
7397.19
43.68
8.65
Ours
490.41
23.60
8.39
C.3 INTEGRATION WITH QUANTIZATION
We further compare quantized models pruned with uniform and nonuniform sparsity using L.A.. Specifically, We apply the advanced quantization approach, GPTQ (Frantar et al.) (003). to LLaMAI-7B at 70% sparsity level. We evaluate perplexity on Wikitext, PTB, and C4 datasets with bit widths of 2, 4, and 8. The results are presented in Table [4] After quantization, models pruned with LSA outperform those with uniform sparsity all bit widths. Notably, on WikiText the perplexity using LSA at 4 bits increases by only 1.04x relative to bits, whereas uniform sparsity yields 3.28x increase.
Table 14: Perplexity of LLaMA1-7B on different validation datasets under varying quantization levels at 70% unstructured sparsity, pruned with LSA using SparseGPT. The best performance result indicated in bold.
Bits
Layer-wise Sparsity
Sparsity
WikiText
PTB
C4
8
Dense
0
6.18
45.81
8.25
8
Uniform
70%
34.35
416.76
32.53
8
Ours
70%
24.22
370.32
24.52
4
Dense
0
6.64
48.24
8.85
4
Uniform
70%
112.65
820.77
94.71
4
Ours
70%
25.25
274.01
28.26
3
Dense
8.42
109.75
11.33
3
Uniform
70%
142.67
1246.90
112.66
3
Ours
70%
72.60
639.25
64.63
D PERFORMANCE AT VARIOUS SPARSITY RATIOS
To further evaluate the effectiveness of our method, we conducted extensive experiments on the WikiText dataset at various levels of sparsity, from low sparsity (< 50%) to very high sparsity (80%).
We compare our method with other layer-wiee wise sparsity techniques, including Global (Frankle and
Carbin 2018b), ER (Mocanu al. 2018). ER-Plus (Liu et al. (022). AlphaPruning al. 2024),
OWL and DLP. As shown in Table 15 all methods perform similarly at sparsity levels below 40%,
with only modest differences in perplexity. Notably, LSA consistently outperforms OWL and DL.P
20

---

Published as a conference paper at ICLR 2026
sparsity levels of 70% and above. These results strongly support the effectiveness and applicability of our method.
Perplexity of LLaMA1-7B on the WikiText validation dataset with various layer- wise sparsity using Wanda. The best performance result is indicated in bold.
Table 15:
Sparsity (Dense 5.68)
Method
10%
20%
30%
40%
50%
60%
70%
80%
Global
5.82
6.11
7.02
9.83
49.07
35477.43
28941.62
79951.92
Uniform
5.70
5.82
5.99
6.40
7.26
10.89
83.45
6548.71
ER
5.70
5.81
6.04
6.59
7.82
12.47
120.50
7831.16
ER-Plus
5.71
5.86
6.26
7.27
12.21
766.84
18190.85
185918.25
OWL
5.71
5.81
6.02
6.39
7.22
9.44
24.85
1024.15
AlphaPruning
5.69
5.81
6.01
6.39
7.25
9.51
24.02
775,43
DLP
5.70
5.81
6.00
6.38
7.19
9,45
20.89
552.53
Ours
5.70
5.81
6.01
6.39
7.20
9.38
20.66
484.93
The returns on performance improvements diminish with decreasing sparsity, which aligns with common patterns observed in model pruning and compression. Consequently, we focused our ex- periments on 70% sparsity to highlight performance differences between methods. Still, We further provide zero-shot performance results for Qwen2.5-7B at 60% sparsity in Table These results demonstrate that LSA still achieves noticeable improvements even at moderate sparsity levels.
Table 16: Accuracy (%) of more advanced models on seven zero-shot tasks at 60% unstructured
sparsity with various sparsity allocation methods, including block-wise (B). The best performance result is indicated in bold.
Model
Method
Layer-wise Spursity
WinoGrande HellaSwag BoolQ PIQA OBQA ARC-e ARC-e Mean
Uniform
68.03
49.47
82.
74.65
26.00
73.57
39,76
59.09
SparseGPT
DLP
68.67
48.24
81.65
72.63
26.20
71.76
37.71
58.12
Qwen2.502 5-7B
73.07
72.26
Ours(B)
69.93
49,88
83.03
73.50
28.00
73.82
40.53
59.81
Uniform
66.54
67
79.91
72.03
25
71.84
OWL
67.40
44.53
78.35
71.55
24.20
71.34
36.35
56.25
Wanda
66.77
70.95
25
69.95
36.60
55.87
Ours
66.61
45.38
79.24
71.49
25
70.03
37.54
56.50
Ours(B)
72.52
E COMPARISON WITH MORE LAYER-WISE SPARSITY METHODS
To further validate LSA, we compare it with two computationally efficient, state-of-the-art layer- wise sparsity methods: ATP (Huang al. 2025) and AlphaPruning (Lu al. (024). ATP theoret- ically shows that arithmetic sequences provide an effective sparsity allocation, while AlphaPruning applies Heavy-Tailed Self-Regularizatiion Theory assess layer importance. Since AlphaPruning published layer-wise metrics for LLaMA1 and LLaMA2, we run experiments on LLaMA2-133 For ATP, we perform a fixed-step grid search to find the tolerance that minimizes WikiText perplex- ity. Results in Table 17 show that LSA outperforms both ATP and AlphaPruning perplexity and zero-shot tasks.
F COMPARISON OF SPARSITY ALLOCATION BETWEEN DIFFERENT GRANULARITY
Since LSA achieves better performance with block-wer wise granularity than with layer-wise granular-
ity Llama3, compare their sparsity allocation results at 3 0.15 in Tables 18 and 19 On
21

---

Published as a conference paper at ICLR 2026
Table 17: Perplexity on different validation datasets and accuracy (%) on seven zero-shot tasks of LLaMA2-13B models at 70% unstructured sparsity. The best performance result is indicated in bold.
Method
Layer-wise Sparsity
Perplexity:
Accuracy (S]
Wik/Text
FTB
C4
WinoGrande
HcllaSwag
BoolQ
PIQA
OBQA
ARC-c
ARC-c
Mcan
Uniform
19.53
284.67
23.66
59.91
36,46
62.45
66.05
19.20
49.12
44.44
ATP
13.19
143.76
15.61
62.75
42.17
62.87
70.13
23.40
$1.64
29.69
48.95
SparsoGPT
ALphaPraning
OWT
14.31
196.07
195.73
17.74
65.04
40.12
65.05
67.63
21.40
51.39
28. 84
48.50
DLP
13.39
151.42
15.92
64.88
41.55
64.08
69. 69.10
22.87
50.87
28. 84
48.83
12.56
145
15.39
64.85
41.60
65.62
68.92
22.87
52.13
29.41
49.34
Uniform
45.26
445.76
58.45
52.57
29.28
62.
58,
12.00
35 15-48
18. 17.17
38.30
ATP
16.50
182.99
19.49
60.62
40.12
62.35
68.44
22.40
51.43
29.52
47.84
Wanda
ALphaPruning
17.12
230.28
21.87
67.52
19.40
51.30
27.73
46.84
OWL
18.16
225.65
22.30
61.14
38.23
62.21
67.99
20.93
50.45
27.64
46.94
DLP
61.37
62.27
50.70
66.66
Ours
15.36
135.03
17.73
62.61
41.42
62.21
68.82
21.75
49.63
29.31
47.96
LLMMA3-8B, block allocation significantly outperformed layer allocation. We attribute this primar- ily to architectural differences between LLaMA1-7B, LLaMA2-7B, and LLaMA3-8B.
LLaMA3-8B incorporates the Grouped-Quest Attention (GQA) mechanism, where multiple queries share key-value pairs. This technique was not employed in LLaMA1-7B or LLaMA2-7B, The outputs of the k.proj and v.proj layers are replicated multiple times based on the group count in GQA, increasing the effective importance of each weight in these projections. In layer-level allo- cation, each projection, q, k, V. uses the same sparsity rate, which is suboptimal for kv projections should be reduced. With block allocation, projections within the same block share a same score, but due to parameter count differences, the final sparsity for kv projections becomes lower than for the q projection.
sparsity rate
allocation, the last layer of LLaMA1-7B and LLaMA2-7B had lower sparsity than preceding layers. However, the sparsity results allocated for LLaMA3-8B indicate that its last layer had the highest sparsity level, contradicting findings in some literature (Gromov et al.] [2025b] Mal] eet al. 2023) suggesting the last layer's relative importance. Under block-wise allocation, the attention (attn) module in the last layer exhibits lower sparsity than the feed-forward network (ffn) modules in preceding layers. By suppressing excessive pruning of the last layer, fine-grained allocation enhances performance.
Table 18: Sparsity of LLaMA3-8B pruned with per-layer LSA at 70% unstructured sparsity, using GPT (Sparsity: 70%, Perplexity: 39.56).
Layer
q.proj
k.proj
v.proj
o.proj
gate.proj
up-proj
down.proj
0.5520
0.5520
0.5520
0.5520
0.5520
0.5520
0.5520
0.5595
0.5595
0.5595
0.5595
0.5595
0.5595
0.5595
0.6149
0.6149
0.6149
0.6149
0.6149
0.6149
0.6149
30
0.8342
0.8342
0.8342
0,8342
0,8342
0,8342
0,8342
31
0.8520
0.8520
0.8520
0.8520
0.8520
0.8520
0.8520
Table 19: Sparsity of LLaMA3-8B pruned with per-block LSA at 70% unstructured sparsity, using SparseGPT (Sparsity: 70%, P2321exity:32.58).
Layer
q.proj
k.proj
v.proj
o.proj
gate.proj
down.proj
up.proj
0
0.5488
0.0954
0.0954
0.5488
0.6611
0.6611
0.6611
0,5518
0.1074
0.1074
0.5518
0.6649
0.6649
0.6649
0,5978
0.2913
0.2913
0.5978
0.6834
0.6834
0.6834
30
0.6475
0,4900
0.4900
0.6475
0.8074
0,8074
0,8074
31
0.6555
0.5220
0.5220
0.6555
0.8160
0.8160
0.8160
22

---

Published as a conference paper at ICLR 2026
G HYPERPARAMETER SETTING
In this section, we report the hyperparameter 3 used to scale importance scores (their magnitudes vary with the sparsity ratio). To facilitate reproduction, we list the chosen values in Table 20
Table 20: Hyperparameter settings for different sparsity levels.
Sparsity 10%
20%
30%
40%
50%
60%
70%
80%
0.06
0.02
0.04
0.02
0.04
0.1
0.15
0.12
H METRIC COMPARISON WITH WANDA
To further validate the effectiveness our linear reconstruction error (RE) metric, we compare against the Wanda metric. For a fair comparison, apply uniform sparsity pruning to LLaMA models using Eq. 4 and evaluate both perplexity and zero-shot accuracy. As shown in Table 21 RE outperforms Wanda in nearly all cases. These results indicate that linear reconstruction error more reliably identifies layer importance than Wanda under uniform sparsity.
Table 21: Perplexity on different validation datasets accuracy (%) on seven zero-shot tasks of LLaMA models at 70% unstructured sparsity. The best performance result is indicated in bold.
Perplexity.
Accuracy(SOf
Model
Method
WikiText
PTB
C4
WinoGrande
HellaSwag
BoolQ
PIQA
OBQA
ARC-e
ARC-c
Moan
Wandaa
83.45
624.97
83.18
51.7
28.44
61.87
57.67
13.0
30.81
18.6
37.44
MLMMA-7B
62.24
527.86
45.20
54.14
30.85
62.17
60.01
14.2
35.61
21.67
39.81
LLaMA1-13B
Wanda
53.47
360.95
$5,79
52.49
30.72
62.17
61.53
15.4
37.75
19.2
39,89
34.02
283.67
23.26
57.77
37.11
62.42
68.28
18.6
50.0
25.34
45.63
HaMA1-33R
Wanda
RB
16.30
17.67
107.32
106.04
16.32
19.03
63.22
63.54
43.59
46.14
64.34
64.53
71.76
73.39
22.0
61.74
29.95
33.28
52.02
Wanda RE
75.01
308.89
83.53
49.01
27.84
59.14
55.82
11.4
28.79
18:77
35.82
LLaMA2-7B
69.26
403.48
55.64
52.96
30.02
62.05
60.34
15.0
36.78
20.99
39.73
Wanda R.F
45.26
445.76
58.45 45.45
52.57
29.28
62.2
38.38
12.0
35.48
38.30
LLaMA2-138
58.43
521.17
45.14
$7.54
33.43
62.17
65.67
20.8
47.94
24.57
44.59
I LINEAR RECONSTRUCTION ERROR PRUNING RATIO SETTING
LSA allocates sparsity to each layer by computing their minimal linear reconstruction error. The pruning ratio p used in this step is hyperparameter that directly affects the magnitude of the error. We evaluate the perplexity LLaMA models at 70% sparsity using Wanda, where the linear recon- struction error are obtained from LSA various pruning ratios. As shown in Table 22 perplexity values are similar for pruning ratios below 70%. Although layers' linear reconstruction errors in- crease with higher LSA pruning ratio p, each layer's error fraction of the total error remains stable for below 70%. These results indicate that the hyperparameter p is not critical for sparsity allocation, further supporting L.SA's robustness and effectiveness.
Perplexity the WikiText validation dataset with various linear reconstruction error pruning ratio setting using Wanda at 70% sparsity.
Table 22:
LSA Pruning Ratio
Model
10%
20%
30%
40%
50%
60%
70%
80%
90%
LLaMA1-7B
20.33
20.42
20.38
20.43
20.32
20.18
20.10
20.13
21.57
LLaMA1-138
13.42
13.42
13.41
13.37
13.39
13.38
13.39
13.47
13.96
LLaMA2-7B
22.62
22.75
22.62
22.52
22.62
22.35
22.27
23.40
29.48
LLaMA2-13B
14.89
14.86
14.87
14.87
14.91
14.96
15.28
15.84
17.01
Qwen2.5-7B
61.04
60.78
60.59
60.20
62.08
64.86
70.95
77.23
89.29
Qwen3-8B
74.17
73.89
74.11
74.64
73.39
73.90
73.98
71.06
68.97
23

---

Published as a conference paper at ICLR 2026
J COMPARED WITH LINEAR RECONSTRUCTION ERROR PRODUCED BY OTHER PRUNING METRIC
To fully validate ISA,, we compare it with layer-wise sparsity allocations derived from the linear reconstruction errors produced by SparseGPT and Wanda, and evaluate perplexity on Llamal-7B pruned with SPT As shown Table 23 LSA outperforms both SparseGPT and Wanda. These results indicate that LSA's linear reconstruction error more reliably identifies layer impor- tance.
Table 23: Perplexity on different validation datasets and accuracy (%) on seven zero-shot tasks of LLaMAI-7B model at 70% sparsity pruned by SparseGPT. Layer-wise sparsity allocation based on linear reconstruction error produced by SparseGPT, Wanda and LSA. The best performance result indicated in bold.
Layer-wise Sparsity
WikiText
PTB
C4
+RE(Wanda)
22.29
351.77
24.59
(E((SparsGGPT) +LSA
20.25
303.42
21.88
17.57
215.61
19.50
K. ROBUSTNESS ACROSS VARIOUS VALIDATION DATASETS
We further evaluate our method on WikiText (Merity al. 2017 PTB (Mareus al. 1994). and C4 (Raffel et al. 2020). We perform extensive experiments on LLaMA-I (7B, 13B, 30B), LLaMA-2 (7B, 13B), and LLaMA-3 (8B) 70% sparsity. As shown in Table 24 LSA combined with Wanda or SparseGPT outperforms Uniform, OWL, and DLP in nearly every case. These results demonstrate the robustness of our method across diverse validation datasets.
L ZERO-SHOW TASKS PERFORMANCE
For zero-shot evaluation, we measure accuracy on seven commonsense benchmarks from EleutherAl
LM Harness (Gao et al.) 2023). including BoolQ (Clark et al., 2019), PIQA (Bisk et al. 2019). HellaSwag (Zellers al.
2019), WinoGrande (Sakaguchi et al. 2019). ARC-Easy (Boratko et al.,
2018), ARC-Challenge [Boratko et al.) 2018) and OpenbookQA (Mihaylov et al. 2018). tar-task
accuracies are shown Table 25 Overall, LSA attains the highest average accuracy in nearly every setting.
M LARGE LANGUAGE MODELS USAGE OUR PAPER
We gratefully acknowledge the use of Large Language Models linguistic editing to enhance the manuscript's s read ability. The Al-generated edits were critically evaluated and modified by the author, ensuring the fidelity of the content.
24

---

Published as a conference paper at ICLR 2026
Table 24: Perplexity of LLaMA models on different validation datasets at 70% unstructured sparsity. The best performance result is indicated in bold.
Midel
Method
Layer-wise Sparsity
WikiText
PTB
C4
Dense
5.68
36.42
7.58
Uniform
48836.54
144955.48
22156.64
Magnitude
OWL
19481.68 1.2
110239.75
28196.23
DLP
4051.43
1873462.19
9332.01
Ours
4372.40
1232607.51 1
7608.37
Unitorm
27.18
388.70
28.53
LLaMAI-BB
SparseGPT
OWL
18.98
246.77
21.19
DLP
17.78
223
19.82
Ours
17.57
215.61
19.50
Unitorm
83.45
624.97
83.18
Wanda
OWL
24.85
363.84
27.38
20.89
276.45
22.90
Ours
20.66
265.05
22.76
Dense
5.09
25.27
7.06
Uniform
84514.06
83469. 16
37052.07
Magnitude
OWL
20846.71
98135.49
19514.49
DLP
7529.08
146070.26
32483.04
Ours
8682.50
105622.93
36390.94
Unitorm
20.36
164,90
23.14
LLMMA1-13B
SparseGPT
OWL
14.15
116.58
16.52
DLP
12.87
93.06
14.75
Ours
12.45
84.59
14.48
Unitorm
53.47
360.95
33.79
Wanda
OWL
16.49
162.51
19.40
DLP
13.94
100.94
16.19
Ours
13.65
91.39
15.94
Dense
4.10
21.21
6.39
Unitorm
972.99
7042.14
4807.79
Magnitude
OWI.
242.56
1165.50
765.41
DLP
96.85
554.96
115.10
Ours
93.42
544.71
108.18
Uniform
12.34
66.30
15.80
LLMMA1330B
SparseGPT
OWL
10.69
49,29
12.92
DL.P
9.62
43,16
11.88
Ours
9.49
41,31
11.80
Unitorm
17.67
107.32
19,03
Wanda
OWL
11.02
60.05
13.92
DLP
9.95
44,43
12.63
43.06
Dense
5.47
24.09
7.53
Uniform
49799.57 57
36727.36 36
29519.00 90
Magnitude
59566.99
24024.65
DLP
7888.52
38030.13
3646.19
Ours
8994.37
53501.48
3931.41
Uniform
26.32
32.58
ILaMA2-7B
SpanseGPT
OWI
20.68
712.58
23.64
DLP
18.68
212.96
19.97
18.63
Uniform
75.01
308.89
83.53
Wanda
OWL
30.03
198.37
37.39
22.85
Ours
22.89
133,08
26.30
Dense
4.88
34.41
6.99
Unitorm
Magnitude
OWL
59.31
1717.48
$1.03
DLP
52.16
825.59
42.27
Ours
45.16
595.34
40.35
Uniform
19.53
284.67
23.66
LLaMAZ-13B
SparseGPT
OWL
14.31
196.07
17.48
DLP
13.39
151.42
Ours
12.56
145.60
15.39
Unifom
45.26
445.76
58.45
Wanda
OWL
18.16
225.65
22.30
DIP
16.80
153,89
18.87
Ours
15.36
135.03
17.73
25

---

Published as a conference paper at ICLR 2026
Table 25: Accuracy (%) of LLaMA models on seven zero-shot tasks at 70% unstructured sparsity. The best performance result is indicated in bold.
Model
Method
Layer-wise Sparsaty
WizeGrande HellaSwag BoolQ PIQA
OBQA
ARC- ABC-c Mean
Desse
67.09
56.41
73.52
28.2
67.3
38.23
58.43
Unilvem
25.75
60.73
13,60
25.5
20.99
Magaitue OWL.
51.19
27.11
61.52
56.55
15.30
27
21.93
37.30
DLP
49.84
30.34
62.03
58.08
15.00
29 38
23,80
38.35
51.15
30.45
62.06
58.19
14.30
30.22
23.72
38.66
LLLMA--78
Unitcem
55.56
63.18
15.6
42.0 42.00
43.75
SparseGPT OWL
58.601
36.75
64,77
64.50
16.50
42
26.02
44.23
DLP
58.46
37.47
63.91
64.94
17.13
42 06
25. 23
44.26
Ours
58.49
38.02
64.60
65.72
17.27
43.15
26.14
44.77
Uniform
Winda
OWL
56.35
34.70
62.74
63.90
16:20
42.33
24.45
42.95
DLP
$6.50
36.72
62.49
66.40
15.48
44.77
24.79
44.31
Ours
56.43
37,06
62.36
66.26
0S
44.
24.97
44. 14.19
Dense
70.17
59.11
69.76
79.0
30.6
43.94
Uniform
50.75
26.24
62.05
53.59
14.6
26.64
19.88
36.25
Magnitdee
OWL
50.98
27.56
62:22
54.71
15.50
28, 66.66
23.42
37.62
DLP
51.10
27.18
62.14
53.78
19.10
30.79
23. 89
38.28
27.06
LLaMAI-13B
Uniform
60.77
36.39
66.49
19.8
47
24.91
SparseGPT OWL
63.53
40.94
62.77
69:35
20.47
52 05.05
28 OZ
16
DLP
63.71
42,26
62.28
70.66
22,00
52.42
28. 17
48.79
64.59
42.53
62.19
21.68
51 $1.47
29.03
48.91
Uniform
Winda
OWL
60.75
39.39
62:17
68.50
19:53
51
27. 53
46.98
DLP
62.04
42,51
62.17
70.38
23.40
52 32
29. 10
48.51
Ours
62.72
43.03
62.17
70.18
22,80
52.39
29.67
48.99
Dosse
72.69
62.67
29.4
46.59
Uniform
49.72
25.77
39.51
52.45
15.4
25.
21 33
32.82
Magnitdee
OWL
49.56
25.83
39.31
52.85
14.70
27 .244
19. 19.58
32.79
DLP
53.23
29.40
61.80
60.72
17,80
3.5
21.61
40.49
52.72
LLaMAI-30B
Uniform
67.72
44.27
61.71
72.96
23,4
61
32.76
SpurseGPT OWL
70.17
46
64. 58
73.69
24 90
61 93
32.77
53.53
DLP
68.94
48.25
64.68
73.75
25.30
60 04.844
33.88
53.65
70.48
48.09
64.29
73.28
34.61
53.92
Uniform
63.54
43.59
61.53
71.76
23.0
59
29.95
50.81
Winda
OWL
67.56
47.31
63.32
74.32
24.90
62.48
34.05
53.42
DIP
68.25
48.95
62.66
73.99
24 30
60 80.33
36,24
53.67
68.25
24.25
Dosse
67.17
56.68
70.15
31.8
Uniform
47.99
26.24
41.31
53.21
16.4
26 18
23 89
33.66
Magniude
OWL
48.86
27.63
43.15
56.75
16.60
30 22
23.63
35.26
50).48
45.16
23.98
31.94
51.59
39.17
LLaMA2-7B
Uniform
$5.96
33.31
62.
62.79
16.4
38
21. 16
41.51
SparseGPT OWL
$9.41
36.49
62.55
64.26
19.00
43.97
25 23
44.49
DLP
59.96
38.21
62 26
67.07
18.20
44.59
26 28
45.27
26.48
Uniform
49.01
27.84
59.14
55.82
11.4
28. 79
18.77
35.82
Winda
OWL
55.54
31.65
62.
62.28
15:47
39 38
21.27
41.11
DLP
57.82
35.00
62.16
65.69
18
44.45
23,60
43.77
$7.54
36.38
62.17
18.20
43.86
Dense
69.77
59.69
67.25
78.73
32.4
73:23
45.56
60.95
Uniform
49.8
27.47
61.87
55.11
14.8
30.3
21.08
37.20
Magnitde
OWL
52.21
33.68
62.52
60.60
16.75
33.
22. 27
40.26
65.93
$7.70
38.51
65.63
15.95
27.99
44.38
LLaMA2 13B
Unifcrm
64.09
40.33
63.39
68 19.39
21.2
51 .47
28. 16
4%.15
SparseGiTT OWL
64.46
40.04
64.33
69.55
21.07
51.00 $1.01
27. 79
48.18
DLP
64,88
41.55
64.08
22.87
50.87
28.
64,85
65.62
29.41
Uniform
52.57
29.28
62.2
58.38
12.0
35.48
18.17
38.30
Wanda
OWL
61.14
38.23
62.21
67.99
20.93
50.45
27 64
46.94
DLP
61.37
40.41
62.27
69.41
21.60
50.70
28.
6666
62.61
62:21
49.63
29.31
47.96
26


Reviews:
[{"review_id": "RVTF1trthC", "reviewer": "Reviewer_tytU", "summary": "This paper introduces LSA, a pruning method for large language models that measures each layer’s importance using *minimal linear reconstruction error* instead of weight-based scoring. By directly quantifying how much information each layer loses when half of its least important weights are removed, LSA assigns non-uniform sparsity ratios across layers, blocks, and even projection levels, achieving fine-grained pruning without performance collapse. Extensive experiments on LLaMA, Vicuna, Mistral, and Qwen models show that LSA consistently surpasses state-of-the-art methods like OWL and DLP in perplexity, zero-shot accuracy, and inference speed, demonstrating its robustness and generalization across architectures.", "questions": "It is necessary to conduct experiments on the LLaMA‑3 family or the Qwen 2.5/3 series.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed minimum linear reconstruction error is a meaningful and insightful contribution for both the application and analysis of pruning.\n\n2. The proposed LSA method achieves strong performance in high-sparsity pruning on the LLaMA-1/2 model series, and the experiments are solid and convincing.\n\n3. The paper is clearly written and easy to follow.", "weaknesses": "1. The LLMs used in this paper are outdated and do not reflect state-of-the-art LLMs. I think it is necessary to conduct experiments on the LLaMA‑3 family or the Qwen 2.5/3 series.\n\n2. The manuscript omits several relevant baselines and sparsity-allocation references (e.g., EvoPress, DSA).\n\n[1] Sieberling, O., Kuznedelev, D., Kurtic, E. & Alistarh, D. (2025). EvoPress: Accurate Dynamic Model Compression via Evolutionary Search. ICML. \n\n[2] Li, L., Dong, P., Tang, Z., Liu, X., Wang, Q., Luo, W., Xue, W., Liu, Q., Chu, X., & Guo, Y. (2024). Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models. NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pGToXvTNFC", "reviewer": "Reviewer_GCyG", "summary": "This paper proposes LSA (Layer-wise Sparsity Allocation), a novel method for pruning large language models (LLMs) that quantifies layer-wise importance by evaluating the minimal linear reconstruction error (LRE) under the assumption of removing 50% of the least important weights. LSA avoids weight scoring and empirical reduce functions, enabling non-uniform sparsity allocation at finer granularities (block- or projection-level) without performance degradation. Experiments show LSA outperforms state-of-the-art methods like OWL and DLP at high sparsity levels (e.g., 70%), achieving better performance on language modeling and zero-shot tasks while supporting efficient inference and fine-tuning.", "questions": "As shown in Tables 1 and 2, Layer-wise allocation outperforms Block-wise in most cases on LLaMA1-7B and LLaMA2-7B. However, on LLaMA3-8B, Block-wise allocation significantly surpasses Layer-wise. Do the authors provide any insights into this discrepancy?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.LSA introduces minimal linear reconstruction error as a direct measure of layer importance, eliminating the need for weight scoring or manual reduce function design.\n\n2.LSA consistently outperforms OWL and DLP across multiple models (e.g., LLaMA, Vicuna) and tasks (e.g., WikiText perplexity, zero-shot accuracy) at high sparsity levels (70%).", "weaknesses": "1.The authors claim that LSA is the first method to achieve projection-level sparsity allocation, but TRIM[1] had previously implemented even finer-grained allocation: assigning sparsity rates to rows and columns of matrices.\n\n2.These sparsity allocation methods cannot be applied to the currently most commonly used 2:4 semi-structured pruning, and are limited to unstructured pruning only. However, unstructured pruning cannot be accelerated by GPUs, which limits the practical applications of such methods.\n\n3.As shown in Table 6, LSA only shows slight improvements over DLP in most scenarios. Why didn't the authors conduct zero-shot task performance comparisons on Llama3? On more powerful models, the advantages of LSA over DLP might be more prominent.\n\n[1]Beck, Florentin, William Rudman, and Carsten Eickhoff. \"TRIM: Achieving Extreme Sparsity with Targeted Row-wise Iterative Metric-driven Pruning.\" arXiv preprint arXiv:2505.16743 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SaV4HjhZBP", "reviewer": "Reviewer_SSxf", "summary": "This paper investigates layer-wise Sparsity Allocation (LSA) for LLM pruning, which quantifies layer-wise importance by evaluating the minimal linear reconstruction error (LSE) of each transformer layer under the assumption that 50% of its least important weights are removed.", "questions": "See the weakness.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The method supports non-uniform sparsity allocation at block- or projection-level granularity within layers, without incurring catastrophic performance degradation. It enables the assignment of distinct sparsity ratios to different projections within the same\nTransformer layer, providing a finer-grained allocation that can lead to performance improvements.", "weaknesses": "The novelty of the proposed method may be limited. It follows traditional methods to compute the layer importance first and then allocate the sparsity for each layer according to the importance. The framework is very similar to previous works such as DLP, including the importance score and the sparsity allocation range between [pr - $\\beta$, pr + $\\beta$].  Some detailed part may be different, but the framework and other parts are similar to provious works. The technical contribution may be limited. \n\nThe experimental results with detailed baseline comparsion mainly focus on 70% sparsity. The results of other sparsity are very rare, and other sparsity does not have baseline results. It is better to compare with baselines under different sparsity ratios to demonstrate the general performance, not only under 70% sparsity. \n\nIt mentions to be more efficient. It is better to provide more detailed analysis about the efficiency such as complexity comparison. It is hard to see why it is more efficient, more discussions with detailed complexity demonstration can enhance this part. \n\nIn the experiemnts, the improvements seem to be marginal. The performance is very close to DLP with a very small gap. Some results are different from the original DLP paper. For example, in table 5, for sparsegpt on Llama2 7B, it reports LSA with PPL 18.63 and DLP with 18.68 PPL.  The gap is very small. And in the original DLP paper, the PPL is 18.58 under the same configuration, which is better than 18.63.  Different runs may lead to different results with small variantions. The current gap is minor, and another run may change the result leading to a better baseline. The appendix also shows that in many cases, the baselines can outperform the proposed method. The  experimental improvements  seem to be marginal, and it does not seem to be significantly better than baselines. \n\nThere are some other methods which also investigate the sparsity allocation for different layers, such as [R1,R2]. It is better to discuss the comparison with more baselines. \n\nSparseGPT and wanda are basic llm pruning methods. There are more advanced pruning methods such as [R3], which can outperform SparseGPT with large margins under uniform sparsity. It is better to combine the proposed method with more advanced pruning methods to demonstrate the advantages. \n\n[R1] Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models\n\n[R2] Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment\n\n[R3] Fast and Effective Weight Update for Pruned Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ju6fORxWm5", "reviewer": "Reviewer_ZGQH", "summary": "This paper proposes Layer-wise Sparsity Allocation (LSA), a new method for pruning large language models by measuring layer importance via minimal linear reconstruction error (LRE) rather than relying on heuristic weight scores such as those used in Wanda, OWL, or DLP. The method supports finer-grained sparsity allocation (block- and projection-level) without catastrophic performance degradation. Extensive experiments on multiple LLMs (LLaMA1/2/3, Mistral, Qwen) show that LSA achieves lower perplexity and higher zero-shot accuracy than prior methods at high sparsity (70%), while also providing measurable inference speedup.", "questions": "1. The proposed LRE offers a novel perspective for measuring layer importance. However, the paper lacks a thorough comparison or discussion of how LRE relates to previously established metrics, such as Hessian-based saliency or gradient magnitude. It would strengthen the work to analyze potential correlations between LRE and these existing indicators, clarifying whether LRE captures complementary or more comprehensive information. Furthermore, while the empirical performance of LRE-based pruning is impressive, the underlying intuition and theoretical motivation behind why LRE serves as a better proxy for layer importance remain insufficiently explained.\n\n2. It is great to see that the paper includes experiments on structured pruning and N:M sparsity. However, these results would be more convincing if they also included comparisons with existing layer-wise sparsity allocation methods such as OWL and DLP, to better contextualize the advantages of LSA in structured settings.\n\n3. The authors claim that the sparsity ratio $p$ used to compute the linear reconstruction error is not a sensitive hyperparameter. However, the provided evidence is limited to models of similar size and within the same family. A more convincing analysis would involve evaluating across different model families and scales to demonstrate broader generalization.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a new way to measure layer importance directly through reconstruction error, avoiding arbitrary scoring or reduction functions.\n2. LSA is conceptually simple and model-agnostic, applicable to unstructured pruning and extensible to block/projection levels.\n3. Extensive experiments on multiple LLMs (LLaMA1/2/3, Mistral, Qwen) show that LSA achieves lower perplexity than prior methods at high sparsity (70%), while also providing measurable inference speedup.", "weaknesses": "1. While the proposed method shows clear gains in perplexity, the improvements on the seven zero-shot benchmarks appear relatively marginal compared to OWL and DLP. The authors are encouraged to provide further discussion on this part.\n2. The notation in the Method section ( e.g. $S_w$, $S_x$ ) is not clearly defined, which makes the mathematical formulation somewhat difficult to follow. Additionally, the method description is overly verbose and could be streamlined for better readability and comprehension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)