Lua-LLM: Learning Unstructured-Sparsity Allocation for Large Language Models
Authors: Mingge Lu
Jingwei Sun
Junqing Lin
Zechun Zhou
Guangzhong Sun
Venue: NeurIPS 2025 poster ()
Keywords: Sparsity
Pruning
Efficient Inference
Large Language Models
Primary area: applications
TL;DR: We propose a global pruning framework that efficiently learns unstructured sparsity for LLMs.

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their extensive parameter scales pose significant challenges for practical deployment. Unstructured pruning has emerged as an effective model compression strategy with minimal performance loss, which introduces fine-grained sparsity for weight parameters. While existing methods employ a layer-wise pruning strategy to avoid the complexity of global pruning for billion-scale LLMs, they require appropriate sparsity allocation for the layer-wise pruning objectives and often lead to suboptimal solutions for the overall model. In this paper, we propose Lua-LLM ($\textbf{L}$earning $\textbf{u}$nstructured-sparsity $\textbf{a}$llocation in LLMs), a learning-based global pruning framework that explores the optimal unstructured sparsity allocation. Unlike existing pruning methods, which primarily focus on allocating per-layer sparsity, Lua-LLM achieves flexible allocation for both layer-wise and intra-layer sparsity. Furthermore, Lua-LLM leverages a soft Top-K operator to approximate the importance-based mask selection mechanism, enabling efficient binary mask learning. Experimental results on LLaMA and OPT families demonstrate significant performance improvements over existing methods.

Lua-LLM: Learning UnstructureddSpaariiy Allocation for Large Language Models
Mingge Lu Jingwei Sun" Junqing Lin Zechun Zhou
Guangzhong Sun'
University of Science and Technology of China
minggedmail ustc. edu.. en, sunjw@usta edu. en, linjunqing@mail. est.c.ddu.cm zhouzechun@mail. wst....st.........o..o.oo....... edu. en, gzsundustc. edwiee.dducccmm
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their extensive parameter scales pose significant challenges for practical deploy- ment. Unstructured pruning has emerged as an effective model compression strategy with minimal performance loss, which introduces fine-grained sparsity for weight parameters, While existing methods employ a layer-wise pruning strategy avoid the complexity of global pruning for billion-scale LLMs, they require appropriate sparsity allocation for the layer-wise pruning objectives and often lead suboptimal solutions for the overall model. this paper. we propose Lua-LLM (Learning eesstrutture-sparsiic allocation in LLMs), a learning- global prun- ing framework that explores the optimal unstructured sparsity allocation. Unlike existing pruning methods, which primarily focus on allocating per-layer sparsity, Lua-LLM achieves flexible allocation for both layer-wise and intra-layer spar- sity. Furthermore, Lua-LLM leverages a soft Top-K operator approximate the importance-basee based mask selection mechanism, enabling efficient binary mask learn- ing. Experimental results on LLaMA and OPT families demonstrate significant performance improvements over existing methods.
Introduction
Large Language Models (LLMs) 25 67
have demonstrated remarkable performance across a wide range of downstream tasks in natural language processing 9
However, their ever-inereasing parameter scales require substantial memory and computational resources, posing major challenges for their practical deployment on various platforms and applications. For instance, LLaMA-3.1 1-05BBBBBBBBB model [24] requires more than 754 GB memory store its parameters half- precision (FP16) format, far surpassing available memory on aisuucce-cosstaiie
68 69].
devices. To make L.MSs more accessible and efficient, considerable efforts have been made compress these models, including pruning 3 19
33, 48 (4), quantization (20 31 73) and knowledge distillation [36 56 63). Pruning an effective model compression approach and has been applied successfully
in various model structures 10
45 60)
Unstructured pruning 18 26 50 selectively removes less critical weight parameters the element
granularity. This process introduces element-wise sparsity in weight matrices while maintaining minimal degradation model performance. Conventional pruning methods 449 61
propose a global
pruning strategy, which solves an optimization problem to minimize the overall model loss. However, given the massive parameter scale LLMs. these methods become impractical due to the substantial computational overhead. To address this, recent studies like SparseGPT [99 and Wanda (64) split the global pruning objective into multiple local subproblems, each of which focuses on minimizing layer-wise pruning error that can be solved faster. Despite their efficiency gains, these methods
"Corresooniig authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LLaMA-78
LLaMA2-76
LLaMA3-88
OFT-138
Darse Model
Model
Dense Hodel
Dense
80
OWL
Wanda
90
Wanda
DIVIL
102
Wands
GWL
CWL
Wanda
C0
50
WilkTTest2
Lua LUM (Ours)
60
Lue LUM [Ours]
Lua-LUM (Ours)
32
Lue LLM (Ours)
20
19
20%
33%
50%
70%
20%
43%
50%
60%
70%
30%
45%
60%
30%
52%
70%
Sparsty
Sparsity
Figure 1: Model perplexity J. results on WikiText2 dataset with 2048 sequence length.
employ uniform pruning ratios across all layers and only focus on minimizing the local errors, posing the risk of removing important weights for more sensitive layers and leading to suboptimal solutions. Especially at high sparsity levels, these methods suffer severe performance degradation, undermining their practical applicability.
Recent studies like OWL (76 have thus focused on layer-adaptive sparsity allocation (29
37 38
(66 to fulfill the potential of LLM pruning. While these techniques have demonstrated noticeable performance improvements over uniform pruning strategies, two substantial challenges remain: First, existing layer-adaptive approaches overlook intra-layer sparsity allocation. Following the pruning strategy in prior work [19 64). these methods divide the weight matrix of a layer into finer- mner-grained comparison groups, cach employing uniform sparsity level. Although BESA [74] aims to learn sparsity for these fine- reeeggained groups, the substantial computational overhead caused by neural architecture search forces it to minimize the error of each transformer block, which still yields suboptimal solutions. Second, layer-wise importance statistics are insufficient to indicate the inherent distribution of unstructured sparsity. OWL uses layer outlier ratio heuristic proxy for per-layer sparsity allocations, which requires empirical sparsity mapping and lacks a solid theoretical foundation. DSA 37 conducts an evolutionary algorithm to find the optimal allocation function based on importance scores, yet the search process takes at least 12 hours for LLaMA-7B model, and the searched allocation function faces generalization issues for other models.
To address these challenges, we propose Lua-LLM, a gradient- -based global pruning framework that learns fine-gre fine-grained sparsity allocation minimize the overall model performance loss. Our key insight is that Wanda's pruning strategy primarily focuses on activation sparsity in the input dimension, while its uniform row-wise sparsity configuration overlooks output sparsity, leading to an imbalanced sparsity distribution. Thus, we develop our Lua-LLM method by building upon an adaptive row-wise sparsity allocation problem. Lua-LLM decomposes the selection of overall pruning masks into row-wise pruning subproblems, each of which leverages a Top-K selection operator represent the mask selection process based on existing element-wise importance metric. The non-differentiable Top-K operator is approximated using Sigmoid function, where the row-wise sparsity is transformed into a single pruning threshold parameter, enabling parameter- efficient mask learning. After optimizing the row-wise threshold parameters with end-to-end model performance loss, Lua-LLM identifies a sub-network within the original model, which achieves adaptive sparsity allocation for both layer-wise and intra-layer sparsity. For LLaMA-7B model, Lua-LLM learns sparsity allocation in only 1 hour on 2x NVIDIA A100 GPUs, which demonstrates superior efficiency compared to DSA.
We Lua on several LLMs, including LLaMA-7B/13B, ILAMAA-7B/113B LLaMA3-SB, and OPT-6.7B/133 Compared to existing sparsity allocation methods, Lua-LLM achieves significant performance improvements for compressed models, particularly at high sparsity ratios. As shown in Figure I Lua-LLM reduces the perplexity on LLaMA3-SB under a 70% pruning ratio by 99.5, 67.29, and 79.14 compared Wanda, OWL, and DSA, respectively. Under 60% pruning ratio. Lua-LLM improves the average accuracy on LLaMA-78 7B by 4.92%, 3.47%, 4.09%, compared to Wanda, OWL, and DSA, respectively, while incurring only a 3.18% accuracy degradation from the original model. When integrated with SpInfer [16), a GPU inference framework for sparse LLMs, Lua-LLM achieves end-to-end inference speedup for the compressed models with 50% 70% sparsity levels on an NVIDIA A100 80 GB GPU, ranging from 1.18x to 1.73x. Our experiment results demonstrate that Lua-LLM achieves more adaptive sparsity allocation, enhancing the practical applicability for L.M pruning at high sparsity levels.

---

2 Related Work
Unstructured Pruning for LLMs. Pruning has long history 227
34 and has been successfully
applied to compress neural networks of various structures [10] 28
45 60) Compared to structured and semi-structured [7 80 82 pruning strategies, unstructured introduces finer-grained sparsity and leads to minimal performance loss. proposes post-training pruning framework that computes Hessian metrics for weight elimination and update. Wanda [64]] designs novel pruning metric and pattern, which outperforms SparseGPT at 50% sparsity without any weight updates. SparseLLM 4 leverages auxiliary variables for the decomposition of the global pruning problem and achieves alternating optimization into the subproblems with global optimality. To achieve practical inference speedup for unstructured sparse neural networks on GPUs, multiple works [16] 22 41 71) have been proposed to optimize the kernel latency of sparse matrix-matrix multiplication (SpMM) operation.
32
2
6
21
48
53
72
78
pruning 4,
19
52
64
83
SparseGPT
19
Adaptive Layer-wise Sparsity. To address the limitation of prior uniform pruning methods [4] 19 [52 64]. recent studies have thus focused on layer-adaptive sparsity allocation techniques. Based on observation of the correlation between activation outliers and performance of LLMs, OWI. [76] adjusts the per-layer sparsity ratio according to layerwise outlier distribution. ALS 38 estimates inter-layer correlations using information orthogonality and then employs linear optimization to selectively prune features in intermediate layers. DSA [37] conducts an evolutionary algorithm to find an allocation function that establishes mapping from element-wise scores to sparsity ratios and generalizes across different models. AlphaPruning 447 uses heavy-tailed self regularization to allocate layerwise sparsity in theoretical manner. ATP (29) reduces the process of determining sparsity rates for multiple layers to the determination of a single common difference hyperparameter with monotonically increasing arithmetic progression.
NAS-based Pruning for LLMs. Neural Architecture Search (NAS) 15 39 43
51
58
66
is pivotal technique in machine learning that automates the design of optimal neural network architectures. Recent studies have applied NAS to automatically identify the optimal sparsity pattern in pre-trained LLMs. Using the evolutionary algorithm, Pruner-Zero 14] evolves symbolic pruning metrics. DSA (37] searches for effective sparsity allocation functions. Search-LLM [63] identifies structured sub- networks using mask mutation. The gradient- based method also serves an important NAS strategy. DISP-LLM (23) formulates dimension-independen structured pruning as an optimization problem and uses Straight-Throug! gradient estimator [44] to enable mask learning. MaskLLLM 17 incorporates Gumbel Softmax for differentiable sampling of semi-structured pruning masks. BESA (74) parameterizes local sparsity objectives with learnable combinations of candidate pruning rates and minimizes block-wise reconstruction error. Given the massive parameter scale of LLMs, searching for element-wise unstructured pruning masks is much more challenging due to prohibitively high computational costs.
3 Preliminary
3.1 Problem Formulation
Given a pre-trained large language model with parameter W, pruning is formulated as a constrained optimization problem, which utilizes inputs X to derive a sparse model with a binary pruning mask M and possibly updated weights W that minimizes the task performance loss:
min W. M(X;M W)
s.t. Sparsity(M) =
where the pruning mask M is constrained with target sparsity p. Since jointly optimizing both the pruning mask and the remaining weights is an NP-hard problem [8) popular approach is to separate the pruning problem into mask selection and weight reconstruction.
Challenges. The massive parameter scale of LLMs introduces significant computational overhead for global pruning optimization problems. To reduce the complexity of the global pruning problem, split the full-model pruning problem into layer-wise subproblems. Despite the efficiency gains, the layer-wise pruning strategy presents two primary limitations. Firstly, it focuses on minimizing the local pruning error of each linear layer, while the non-line linelinear operations
prior methods 4 19
52 64
3

---

in LLMs suggest that such a layer-wise approach may yield a suboptimal solution for the entire network [40]. Secondly, it requires handcrafting an appropriate sparsity ratio pe for each layer. as the individual contributions of layers to the final model performance exhibit significant variation
23 48 [6]. In this paper, we aim to address these challenges with an end-to-end learning method,
which achieves adaptive sparsity allocation while minimizing the overall performance loss.
3.2 Revisit Wanda Pruning
Wanda 64 is a one-shot layer-wise pruning method, which prunes model weights without the weight recon- struction process. However, it achieves superior model performance at 50% sparsity level compared to methods that require weight update, such as SparseGPT [19]. The success of Wanda underscores the effectiveness of a better mask selection strategy, including careful design of the weight importance metric and the comparison group.
Attention We
Preserved
Outllers
First, capture the importance of model weight, Wanda proposes novel pruning metric. Formally, given weight matrix W E RC XCin and its input activation X e RLxCin, where L is the input sequence length and Cout, Oin are the output and input dimension, respectively, the importance score for weight element Wi is computed as below:
Input Outliers
(2) where |,yl is the weight magnitude and ||,, is the Eg norm of the j-th column of input feature.
S.j W |Xازن
Figure 2: Visualization of the weight matrix Wo under Wanda sparsity pattern and the corresponding input and output magnitude outliers.
Second, Wanda chooses output channels, which refer to the rows in weight matrix, as the groups for weight im- portance comparison and applies uniform sparsity level to all rows. Through extensive experiments, Wanda demonstrated that using row-wise comparison groups outperforms alternative configurations, including the entire matrix, single columns, multiple columns, or multiple rows.
Observations. Prior works 13 42 76 have demonstrated that activations in LLMs exhibit heavy- tailed distributions, characterized by a subset of features with exceptionally large outliers. These outliers have been proven to play vital role in the remarkable performance of LLMs. From Figure [ we notice that Wanda's 's sparsity pattern maintains low sparsity in the input dimensions corresponding intrinsic explanation for its effectiveness. However, Wanda
to these outliers, which provides misses a potential issue that the output dimensions also contain outlier magnitudes, while using the same row-wise pruning ratio might eliminate important weights in the output dimensions that extremely sensitive, particularly at high sparsity levels.
Motivating Study. To verify our hypothesis, we perform a preliminary study for the output sparsity in weight matrices. Formally, given weight matrix W E R. and its input X E Lx the output Y Luuu is computed below:
Y XWT.
We collect the output magnitude y E Cour, where its j-th element y; = |Y --lll is the l2 norm of the j-th column of output feature.
We first apply the Wanda pruning strategy to uniformly prune the rows with 80% sparsity level for weight matrices in LLaMA-2-7B model. Then we adjust the row-wise sparsity ratios for each layer as follows: according to the output magnitude y, we reduce the sparsity to 60% for the top 128 rows, and increase the sparsity to 100% for the bottom 128 rows. Although completely removing some output channels will introduce structured sparsity, which empirically leads to lower performance than unstructured sparsity, we still observe that the perplexity on WikiText2 [53] dataset decreases from 2335 to 1532, indicating that our heuristic strategy enhances the model performance. This non-uniform importance distribution the output dimension motivates our pruning approach towards an adaptive row-wise sparsity pattern.

---

Layar Sparsity
Normalized Impoctanoe Score
Threshold
I ayerr-wise Sparsity
Intre- laye: Spansiy
BJB
Intre Sparsiy
Sigmoid
Soft Misii
3.30
0.08
0.81
0.10
Topk ppproximation
Hard Mask
(1) Unifomo Initialization
(2) Row Mask Leaminn
(3) Adaptive Unstructured Sparsity Alocaatoon
Figure 3: Overview of the proposed Lua-LLM pruning framework.
4 Lua-LLM: Learning UnstructureddSparsity Allocation for LLMs
In this section, motivate and describe pruning method, Lua-LLM, which learns more flexible and balanced sparsity allocation pattern from a global optimization perspective. Lua-LLM splits the global pruning mask selection problem into multiple row-wise subproblems, each of which is transformed into a single learnable threshold parameter using a soft Top-K operator, thereby enabling the end-to-end optimization of the model performance through gradient descent. An overview of our pruning framework is presented in Figure
4.1 Row- Wise Mask Selection with Weight Importance Metric
To address the limitations of layer-wise pruning, we aim to identify optimal pruning masks by solving the global pruning optimization problem. However, directly learning the binary pruning masks for LLMs prohibitively expensive. Motivated by our observation in Section we reformulate the global pruning problem weight matrices, we directly use the Wanda importance metric, which is introduced in Equation (2) and prune the less important weights according to the allocated sparsity. We first formulate the row-wise mask selection process as below.
adaptive row-wise sparsity allocation problem. Within each row of
Formally, let w E C denote row in the weight matrix of model layer, and let E R represent the corresponding importance scores derived from the Wanda pruning metric. For target sparsity ratio [0, 1J, define the pruning threshold / E IR as the K -tth largest value in where
K [(1 p) CinJ,
ensuring that exactly K weights retained. This threshold enforces the sparsity constraint:
I(s: <),,
(5)
where II(-) is the indicator function. A mask selection function f. which retains the Top-K most important weights, maps the importance score to a binary mask m E {0, 1 as follows:
f1, 0,
if
m,; = f(s, 0.5 sign(s: - 0.5--
+ 0.5
(6)
otherwise,
where m. 1 indicates that the corresponding weight will be preserved, and m, 0 indicates that the weight w; will be pruned.
4.2 Soft Approximation for Top-K Selection Function
After constructing pruning masks for each row of the weight matrices, we need solve the adaptive row-wise sparsity allocation problem. However, the non-difierentiable nature of the discontinuous mask selection function / poses major challenge for generating differentiable masks, preventing the use of common optimization solvers such as the gradient descent technique.
5

---

Algorithm The multi-stage Lua-LLM pruning algorithm.
Input: training dataset X, pre-trained LLM model, and target sparsity p.
Output: unstructured sparse model.
1: Initialization: integrate mask modules into Attn and MLP layers, prepare uniform importance scores within [0, 1]. initialize all threshold parameters to target sparsity p.
2: for fort+11 to I do:
1+
3: for each weight matrix W do:
4: Generate soft pruning mask M with row-wise thresholds (ty)/2r by Eqn.
5: end for
6: Forward propagation: Liotal = Crask (2'; @ W) + Areg (Mp:)),
7: Update row-wise threshold parameters during back- propagatonn
8: end for
9: Save the row-wise threshold parameters for each weight matrix,
10: Pruning: computing hard masks for pruning by Eqn.(o).
To address this challenge, employ the straightthrough-estiiie through-estimator (STE) technique [5], enabling gradient computation via soft approximation when processing non-differentiable functiion. Intu- itively, we employ the sigmoid function to construct soft approximation of the Top-K selection function, so that each row-wise pruning mask can be generated with a single learnable threshold parameter t. To mitigate the training instability caused by importance outliers (Section (..2)
map the importance scores to a uniform distribution within [0. 1]. Then, given a row of weights w E R with the uniformly distributed importance scores E [0, 1] ii
reparameterize the target sparsity p as learnable threshold parameter I, and compute the soft pruning mask ñ E [0, 1 C with a differentiable mask selection function f:
mh f(ss, 1) o>(ss
where (x) (1 + - is the sigmoid function and À controls the approximation level. We set the value of parameter A to Cin. which is large enough provide a precise approximation. The soft alternative for the Top-K selection function makes the loss function L differentiable with respect to the pruning threshold and the global pruning problem can be optimized with gradient descent method. Formally, the gradient for parameter f can be computed as:
OC Cin OL dm, Øt
am, Oa (X(ss Ot
(8)
0m, ōt
Remark. Equation 7 provides a row-wise differentiable mask constructed from the underlying pruning threshold t and importance score s. Since the operation patterns across all rows are identical, we can leverage the broadcast computation mechanism in PyTorch to compute row-wise pruning masks in parallel for each weight matrix. Another gradient- gradient-based unstructured pruning method for LLMs, BESA 74, requires customized CUDA operator to support the parallel generation for intra-layer probabilistic pruning masks, while Lua-L.LM does not induce any backend modification.
4.3 Learning Row-wise Sparsity Allocation
With the proposed soft mask selection technique,
formulate the global pruning optimization problem based on row-wise sparsity allocation and conduct the end-to-end learning process with the gradient descent method. To reduce our search and improve convergence, we employ uniform initialization strategy, and then enforce the target sparsity with a regularization term.
Initialization. Before training, we first employ Wanda pruning method to obtain the importance scores for weight parameters. As introduced Section 4.2 we map the importance scores to uniform distribution within [0, 1] mitigate the training instability caused by importance outliers. With this preprocessing approach, we can directly initialize the row-wise threshold parameter as the target sparsity p. leading to a uniform sparsity initialization pattern.
Learning Objective. For pre-trained large language model, we integrate learnable mask modules into the Attention and MLP layers while freezing the original model parameters, thus facilitating

---

Table 1: Model Perplexity 1. results of different unstructured sparsity allocation methods evaluated on WikiText2 dataset with 2048 sequence length.
WikiTex22 Perplexity
Methed
Sparstty
LLaMA-7B
LLMMA-138
HaMM22-B
1LMM221388
LJaMA3-8B
OPT-6.7B
OPT-138
5.47
Wanda
7.25
6.15
6.92
5.97
9.82
11.98
11.93
OWL
7.22
6.08
6.86
5.92
9.68
12.21
12.23
50%
DSA
7.26
6.11
7.07
6,11
9.81
12.40
13.00
Lua-LLM
7.12
6.05
6.85
5.89
8.87
11.96
10.94
Wanda
10.79
8.38
23.58
15,22
15.90
OWL
9.18
7.56
16.77
60%
DSA
9.48
7.71
10.36
8.27
23.80
16.65
18.69
Lua-L.LM
8,75
6.97
8.54
6.89
12.21
14.46
13.79
Wanda
91.83
56.26
75.42
45.63
122.02
157.48
55.26
OWL
24.46
16.95
30.23
20.57
89.81
42.92
30.23
70%
Lua-LLM
12.92
8.98
22.52
37.45
20.96
Wanda
2889.94
4008.75
2334.70
1134,94
894.99
4259.55
12516.03
OWL
1192.58
411.04
587.19
214.25
763.36
15370.94
5785.20
DSA
934.29
316.81
1436.02
864.85
894.01
9664.09
533.89
Lua-LLM
37.65
26.52
30.27
24.76
85.39
537.12
414.81
an end-to-end mask training process. To control the overall sparsity ratio, we introduce sparsity regularization loss Lyeg as below:
[oM(N((M)/(PNoooaa)
if N(M) > pNtotal:
Lreg
O.
if ((M) pNootal (NM))N(MM
if N(M) p/iotall
where (MM) the number of removed weight parameters with pruning mask M, p is the target sparsity ratio, and Ntotal is the number of weight parameters in the original model. The sparsity regularization loss encourages the sparsity of the overall model to converge to the target sparsity level The training objective is to minimize the language modeling loss Ltask of next token prediction computed via the cross-entropy loss function. Combining the regularization loss Lreg with the language modeling loss Ltosk, our total training loss in mask learning stage is:
Liotal
Lrask (X; M @ W) + Cre:
(10)
where X' denotes the input tokens from training data, W represents the original weights, and Areg is used to control the penalty for deviating from the target sparsity. Section 5.5 presents an ablation study on Areg. showing that our method achieves stable performance when Areg is large enough.
Experiments
5.1 Experimental Settings
Models. We evaluate our Lua-LLM method on several LLMS as follows: LLaMA family [67 LLaMA-7/13B, LLaMA-2-7/13B. LLaMA-3-8B; OPT family 779] OPT-6.7B, OPT-13B.
Datasets and Evaluation. To train the learnable pruning masks, we use 2048-token segments from C4 (59 dataset, which is also used to sample calibration data in previous works. We evaluate the language modeling perplexity on the validation set of aww-WikiTexr2 [53 dataset. To ensure fair comparison, the sequence length for all models is set to 2048. Following previous works, we also evaluate the zero-shot accuracy of pruned models on seven downstream tasks, including BoolQ [1I, PIQA 77. HellaSwag (77), WinoGrande [62], ARC-easy [2]. ARC-challenge [12] and
OpenbookQA 54 based on the EleutherAl LLM-Evluation-Hammee
35 framework.
Baselines. We compare our Lua-LLM method against Wanda 64 the uniform pruning method
introduced in Section 3.2] and four adaptive sparsity allocation methods, OWI. (76) DSA [7
AlphaPruning 47 and ATP 29
7

---

Table 2: Zero-shot accuracy 1 results seven downstream tasks for the pruned LLaMA-7B, LLaMA2- 7B and LLaMA3-8B models 70% sparsity level.
Model
Method
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-e f
ARC-c
OBQA
Mean
Dense
73.12
78.67
56.41
67.09
67.30
38.31
28.20
58.44
19.03
OWL
62.97
64.53
34.78
56.67
42.30
24.74
16.40
43.20
LLaMA-7B
DSA
62.45
63.44
34.59
55.80
42.09
24.83
16.60
42.83
ATP
65.79
67.46
37.44
61.43
50.63
25.68
20.80
47.03
Lua-LLM
63.74
69.15
42.02
58.62
$6,79
27.22
20.20
48.25
Dense
71.13
78.07
56.69
67.17
69.28
39.93
31.60
59.
Wanda
49.14
27.99
18.17
34.68
OWL
62.23
62.19
31.88
55.41
43.77
20.39
41.81
LLaMA2-7B
DSA
58.81
57.40
28 28.48
49.80
32.28
17.58
12.60
36.71
ATP
62.39
66.81
36.08
61.01
50.76
23.38
20.40
45.83
Lua-LLM
66.12
68.88
42.69
58.96
$8.50
26.79
22.00
49.13
Dense
81.25
79.71
60.18
72.69
80.09
50.51
34,80
65.
55.35
27.38
47.20
17.66
35.48
OWL
61.90
28.37
50.43
35.35
17.15
13.20
37.80
LLaMA3-8B
DSA
61.44
55.88
31.90
48.30
33.71
17.15
12.60
37.28
ATP
61.79
62.18
31.46
$4.93
41.79
20.39
16.40
41.28
Lua-LLM
65.23
65.83
37.29
$7.06
$0.08
23.38
17.20
45.15
Table 3: WikiText2 perplexity &. results
Table 4: End-ta-end inference throughput (token/s) and speedup of sparse models.
compared AlphaPruning and ATP.
aB
L. aMA--77
Model
Sparsity
Dense
50%
60%
70%
Method
70%
80%
70%
80%
Throughput
696.6
842.8
1012.7
1202.3
OPT-6.7B
AlphaPruning
23.86
698.56
28.87
Speedup
1.22x
1.73x
ATP
20.16
176.80
22.16
425.12
Throughput 1
401.6
472.8
563.4
685.1
OPT-13B
Lua-LLM
12.21
37.65
12.92
30.27
Speedup 1
1.18x
1.40x
1.71x
Implementation. We implement Lua-LLM in PyTorch 157 and use HuggingFace transformers library (70] for the evaluated L.MMs. We integrate learnable mask modules into the Attention and MLP layers while freezing the original model parameters, thus facilitating learning process. We utilize the Adam [46] optimizer with the learning rate set to x 10-3 and weight decay set to 0.05. The learnable threshold parameters are trained for 500 iterations, conducted on NVIDIA A100 80 GB GPUs.
end-to-end mask
5.2 Model Performance
In Table I we report the language modeling perplexity of pruned LLaMA and OPT models from 50% 80% sparsity levels. We also compare our method AlphaPruning and ATP in Table 3
Our Lua-LLM consistently outperforms the uniform Wanda pruning baseline and the state-of-the sparsity allocation methods, For example, under a 70% pruning ratio, Lua-LLM reduces the perplexity on LLaMA3-88 by 99.5, 67.29, and 79.14 compared to Wanda, OWL, and DSA, respectively. The performance improvement is particularly larger at higher sparsity levels. At the 80% sparsity level, Lua-LLM reduces the perplexity on LLaMA2-7B by 1642.22 and 394.85 compared to AlphaPruning and ATP, respectively. Notably, Lua-LLM achieves a perplexity result 30.27 for the LLaMA-2-7B model at the 80% sparsity level, which is comparable to the performance of other methods at lower 70% sparsity. Our experimental results demonstrate that directly extracting an unstructured sub-network from the original LLM can maintain acceptable performance with minimal degradation, even at higher sparsity levels.
In addition to model perplexity results, we report the zero-shot accuracy of LLaMA models at 70% sparsity level in Table More results for 60% and 80% sparsity levels are shown in Table [T and Table 12 of Appendix C.I respectively. Under 60% pruning ratio, Lua-LLM improves the average accuracy on LLaMA-7B by 4. 4.92%, 3.47%, 4.09%, compared to Wanda, OWL, and DSA, respectively, while incurring only 3.18% accuracy degradation from the original model. Our method achieves superior performance than baselines under various models, tasks, and sparsity levels, verifying our effectiveness in preserving important weights with adaptive sparsity allocation.

---

LLaM4:2-77
Traning Less Convergence Compor son
PPL with Differett Not
00%
Random
Denze Model
Unifom
Magnt.ade
Wonda
14-
304
219
Earation
Sparsiy
Figure 4: Layer-wise sparsity distribution for LLAMA2-7B.
Figure 5: Convergence for dif- ferent initialization strategies.
Figure 6: WikiText2 PPL under different importance metrics.
Figure 7: Intra-layer sparsity allocation for Attention layers in the 70% sparse LLaMA2-7B model.
5.3 Inference Speedup
Unstructured pruning achieves high sparsity levels while maintaining minimal performance degrada- tion, but the element-wise sparsity pattern it introduces cannot achieve a direct inference speedup on GPUs. Fortunately, a recent study proposes SpInfer [16), a sparse LLM inference framework specifically designed to optimize the kernel latency of sparse matrix-matrix multiplication (SpMM) operations at moderate sparsity levels on GPU Tensor Core architectures. We evaluate the pruned 7P and OPT-13B models on an NVIDIA A100 80GB GPU and report the end-to-end inference throughput across different sparsity levels in Table We use the testcase with batch size set to input sequence length set to 32, and output sequence length set to 256. The compressed models achieve practical speedups on the GPU at sparsity levels from 50% to 70%, ranging from 1.18x to 1.73x compared to the dense model.
5.4 Sparsity Allocation
In Figure 4 we visualize the layer-wise average sparsity distribution of the pruned LLaMA2-7B model under varying global sparsity levels. Our results show that the distribution maintains a consistent pattern across different sparsity ratios, like prior studies (29 37 76). with layer-wise sparsity generally increasing across layers. Notably, we identify a distinct pattern that the first layer demonstrates substantially higher redundancy, and then the layer-wise sparsity exhibits a sharp decline within subsequent layers. The potential reason is that the sparsity levels of MLP modules in the first few layers are significantly higher than the average level. Additionally, we observe that the final layer exhibits a decreasing trend, deviating from the overall pattern.
We also show the intra-layer sparsity pattern in Figure 7 TO Our method achieves adaptive intra- layer sparsity allocation for different types of modules, which provides an intuitive explanation for the significant performance improvement over prior methods. From Figure 7 we can observe the fine-gre one-nneie channel-wise sparsity pattern within each oeff-attention head, which is compatible with findings in prior literature [30, 75). Results for the layer-wise distribution of other models and the intra-layer sparsity pattern for MLP modules are shown in Appendix A We hope that these observations can help researchers to gain insights into the inherent sparsity pattern of L.MSs and design more adaptive one-shot model compression strategies in future work.
9

---

5.5 Ablation Study
Search Efficiency. We demonstrate the effectiveness of the initialization stage (Section 4.3) in helping Lua-LLM obtain the prior sparsity pattern of Wanda. We build baseline that, instead of uniformly initializing the threshold parameters with the target pruning ratio, adopts randomly initialized thresholds. Figure shows the convergence efficiency of the training loss within 500 iterations. We observe that training with uniformly initialized parameters significantly helps to reduce the training loss the beginning, which proves the effectiveness of our initialization strategy. We further report the training costs of our Lua-LLM method in Table 3
shows
Impact of Regularization Hyperparameter Areg- As discussed in Section 4.3 the regularization loss hyperparameter Àveg is used to control the penalty for deviating from the target sparsity. In Table 6 we show the perplexity at 70% sparsity level with various hyperparameter values for LLaMA2-7B model. The results show that the performance of the learned sparsity pattern is stable when the hyperparameter Àreg is large enough. Specifically, the regularization hyperparameter is not large enough, the regularization loss fails converge to zero. This indicates that the pruning mask learned by our method does not meet the target sparsity, and the overall sparsity level is actually lower than the target one, since this lead to better model performance and lower training loss. We mark the case as Not Converge, since it leads to an unfair comparison.
Importance Metric. We further explore whether an appropriate importance metric can enhance the performance of our methods. Specifically, we integrate Lua-LLM with three pruning metrics to prune I.AMA2-7B model, including Magnitude, Wanda[64), and Pruner-Zero [14]. In Figure we compare the perplexity of three pruning metrics across different sparsity ratios. We observe that Pruner-Zero consistently outperforms other metrics, which demonstrates that careful designs for weight importance metrics can further improve the resulting sparsity pattern.
Table 5:
Training costs for different model sizes.
Table 6: Impact of hyperparamter Areg
Training Cost (Time GPUs)
LLaMA-2-7B WikiText2 Perplexity (PPL) Dense:5.47
6.7B
7B
13B
Aveg
8.0
12.0
16.0
24.0
min
60.5
PPL
NC
NC
NC
13.18
12.92
13.11
12.96
6 Conclusion
In this paper, we propose Lua-LLM, gradient-based global pruning framework that learns un- structured sparsity allocation for large language models. Lua-LLM splits the global mask selection problem into multiple row-wise subproblems, and leverages soft Top-K operator to generate dif- ferentiable pruning masks for each row with single learnable threshold, which enables efficient end-to-end optimization for the model performance. Extensive experiments show that Lua-LLM outperforms existing methods in perplexity and accuracy, especially at high sparsity levels. Our pruning framework achieves adaptive allocation for both layer-wise and intra-layer sparsity, which reveals the inherent sparsity distribution in LLMs and enhances their practical applicability under high sparsity levels.
7 Limitation and Future Work
While Lua-LLM achieves significant improvements in model performance over existing methods, there are still some limitations for our methods. First, the end-to-end sparsity learning process for LLMs requires sufficient GPU resources to fulfill the training efficiency, and the learning-based pruning approach is computationally more expensive than metric- one-shot pruning. Second, similar other adaptive unstructured pruning methods, the element-wise sparsity pattern requires specific inference frameworks to achieve practical speedup on GPUs, and the SpInfer framework employed in our paper still faces limitations during the prefill phase when batch size and sequence length are large, which leads to higher memory access overhead for the SpMM operations. Future work will explore flexible one-shot pruning methods and advanced techniques for optimizing inference performance for sparse LLMs.
10

---

Acknowledgement
This research was supported by the 2023 Top-Notch Student Training Program 2.0 for Basic Disci- plines (20231008).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igee Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-basee adaptive structured pruning for large language models. In Proceedings of the AAAI Conference Artificial Intelligence. volume 38, pages 10865-10873, 2024.
[3] Salch Ashkboos, Maximilian I.. Croci, Marcelo Gennari Do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting and columns. In The Twelfih International Conference on Learning Representations. ICLR 2024, Vienna. Austria, May 7-11, 2024.
[4] Guangji Bai, Yijiang Li. Chen Ling. Kiback Kim, and Liang Zhao. Sparsellm: Towards global pruning for pre-trained language models. Advances neural information processing systems, 2024.
[5] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.34322 2013.
[6] Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. Advances in Neural Information Processing Systems, 37:18506-18534 2024.
[7] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical common- sense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1432-7439 2020.
[8] Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approximations. Journal of Fourier analysis and Applications, 14:629-654, 2008
19] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, ct al. Language models are fow-shot learerrs. Advances in neural information processing systems, 33:1877-1901, 2020.
[10] Christos Chataikonstantinou, Dimitrios Konstantinidis, Kosmas Dimitropoulos, and Petros Daras, Recurrent neural network pruning using dynamical systems and iterative fine-tuning, Neural Networks, 143:475-483 2021.
[11] Christopher Clark, Kenton Lee, Ming-Wes Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq:
Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chopter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLL HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers). pages 2924-2936.
[12] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord, Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 10.8800..555,200 2018.
[13] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O): 8-bit matrix multipli- cation for transformers at scale. Advances in Neural Information Processing Systems, 35:30318 8-3032,, 2022.
[14] Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner- zero: Evolving symbolic pruning metric from scratch for large language models. In Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11346 11374, PMLR, 21-27 Jul 2024.
[15] Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. Advances in Neural Information Processing Systems, 32, 2019.
[16] Ruibo Fan, Xiangrui Yu, Peijie Dong. Zeyu Li. Gu Gong, Qiang Wang. Wei Wang. and Xiaowen Chu. Spinfer: Leveraging level sparsity for efficient large language model inference on gpus. In Proceedings of the Twentieth European Conference Computer Systems, pages 243-260, 2025.

---

[17] Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnable semi semi-structured sparsity for large language models. Advances in neural information processing systems, 2024.
[18] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, L.A, USA, May 6-9, 2019.
[19] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337,. PMLR, 2023.
[20] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gplq: Accurate post- tost-traniin quantization for generative pre-trained transformers, arXiv preprint 1.XXx:22210.772333 2022.
[21] Yonggan Fu. Zhongzhi Yu, Junwei Li, Jiayi Qian, Yongan Zhang. Xiangehi Yuan, Dachuan Shi, Roman Yakunin, and Yingyan Celine Lin. Amoeballm: Constructing any-shape large language models for efficient and instant deployment. Advances in Neural Information Processing Systems, 37:78299-7553 193199 2024.
[22] Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-14. IEFE, 2020.
[23] Shangqian Gao, Chi-Heng Lin, Ting Hua, Zheng Tang, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Disp-llm: ceneesionnindeeeedeet structural pruning for large language models. Advances in Neural Information Processing Systems, 37:72219-72244 2024.
[24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten. Alex Vaughan. et al. The llama 3 herd of models. arXiv preprint 2xxiv2447.211788 2024.
[25] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepscek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 1..0..1944 /2948, 2025.
[26] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[27] Babak Hassibi. David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference neural networks, pages 293-299. IEEE, 1993.
[28] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision. pages 1389 1397, 2017.
[29] Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Fei Chao, and Rongrong Ji. Determining layer-wise sparsity for large language models through theoretical perspective. In Proceedings of the 42nd International Conference on Machine Learning. volume 267 of Proceedings of Machine Learning Research, pages 26175-26201, PMLR, 13-19 Jul 2025.
[30] Mingyu Jin, Kai Mei, Wujiang Xu, Mingjie Sun, Ruixiang Tang, Mengnan Du, Zirui Liu, and Yongfeng Zhang. Massive values in self-attention modules are the key to contextual knowledge understanding. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 28063 280066, PMLR, 13- 19 Jul 2025.
[31] Sehoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. In International Conference on Machine Learning, pages 13901-23922 PMLR, 2024.
[32] Woosuk Kwon, Sehoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. Advances in Neural Information Processing Systems, 35:24101-24116, 2022.
[33] Qi Le, Enmao Diao, Ziyan Wang, Xinran Wang, Jie Ding, Li Yang, and Ali Anwar. Probe pruning: Accelerating Iims through dynamic pruning via model-probing, In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025.
[34] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[35] Stella Biderman... Leo Gao, Jonathan Tow. A framework for feo-shot language model evaluation, 2022.
12

---

[36] Lei Li, Yongfeng Zhang, and Li Chen. Prompt distillation for efficient llm-based recommendation. In Proceedings the 32nd ACM International Conference on Information and Knowledge Management. pages 1348-1357, 2023.
[37] Lujun Li. Peijie Dong. Zhenheng Tang. Xiang Liu, Qiang Wang. Wenhan Luo. Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning of large language models, Advances in Neural Information Processing Systems, 37:141292-141317, 2024.
[38] Wei Li, Lujun Li, Mark Lee, and Shengjie Sun. Adaptive layer sparsity for large language models via activation correlation assessment. Advances in Neural Information Processing Systems, 37: 199000090-0093000 2024.
[39] Yanyu Li, Pu Zhao, Geng Yuan, Xue Lin, Yanzhi Wang, and Xin Chen. Pruning-as-searce Efficient neural architecture search via channel pruning and structural reparameterization. In Thirty-First International Joint Conference on Artificial Intelligence, 2022.
[40] Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song, and Yufa Zhou. Beyond linear approximations: A novel pruning approach for attention matrix. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025.
[41] Junqing Lin, Jingwei Sun, Xiaolong Shi, Honghe Zhang, Xianzhi Yu, Xinzhi Wang. Jun Yao, and Guangzhong Sun. Lo-spmm: Low-cost search for high-performance spmm kernels on gpus. ACM Transactions on Architecture and Code Optimization, 21(4):1-25, 2024.
[42] James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training-free activation sparsity in large language models. In The Thirteenth International Conference on Learning Representations. ICLR 2025, Singapore, April 24-28, 2025.
[43] Kai Liu, Ruohui Wang, Jianfei Gao, and Kai Chen, Differentiable model scaling using differentiable topk. In Proceedings of the st International Conference on Machine Learning, (CML'24, JMLR, 2024.
[44] Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, and Jianfeng Gao. Bridging discrete and backpropaga- tion: Straight-through and beyond. Advances Neural Information Processing Systems, 36: 36:122291111 12291-12311, 2023.
[45] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pages 9736-2744 2017.
[46] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
[47] Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W Mahoney, and Yaoqing Yang. Alphapruning: Using heavy-tailed self regularization theory for improved layer-wise pruning of large language models. Advances in Neural Information Processing Systems, 37:911 17-9122, 2024.
[48] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning large language models. Advances in neural information processing systems, 16:21722-21220, 2023.
[49] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765 7773, 2018
[50] Gabryel Mason Williams and Fredrik Dahlqvist. What makes a good prune? maximal unstructured pruning for maximal cosine similarity. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
[51] Joe Mellor, Jack Turner, Amos Storkey, and Elliot. J Crowley. Neural architecture search without training. In International conference on machine learning, pages 7588-7598. PMLR, 2021.
[52] Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. Alps: Improved optimization for highly sparse one-shot pruning for large language models. Advances in neural information processing systems, 2024.
[53] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 16X.v:1609.07433, 2016.
[54] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint 18X9:::80900270:0 2018
13

---

[55] Juan Pablo Muñoo. Jinjie Yuan, and Nilesh Jain. Mamba-shedder, Post-transformer compression for efficient selective structured state space models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL 2025 Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 3851 13663, Association for Computational Linguistics, 2025.
[56] Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang. Yaliang Li, and Jun Huang. Meta-kd: A meta knowledge distillation framework for language model compression across domains. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference on Natural Languoge Processing (Volume I: Long Papers), pages 3026-3036, 2021.
[57] A Paszke. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 2019.
[58] Hieu Pham, Melody Guan, Barret Zoph. Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pages 4095 4104. PMLR, 2018.
[59] Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21((10):116) 2020.
[60] Nitin Rathi, Priyadarshiní Panda, and Kaushik Roy. Stdp-based pruning of connections and weight quantization in spiking neural networks for energy -ffigy-efficient recognition. IEEE Transactions on Computer- Aided Design of Integrated Circuits and Systems, 38(4):668-677 2018.
[61] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02:5 0.23899 2020.
[62] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communicutions of the ACM, 64(9):99-106, 2021.
[63] Xuan Shen, Pu Zhao, Yifan Gong, Zhenglun Kong, Zheng Zhan, Yushu Wu, Ming Lin, Chao Wu, Xue Lin, and Yanzhi Wang. Search for efficient large language models. Advances in Neural Information Processing Systems, 2024.
[64] Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference Learning Representations, ICLR 2024. Vienna, Austria, May 1- 1, 2024.
[65] Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. Contrastive distillation on intermediate representations for language model compression. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 498-508, 2020.
[66] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 14. PMLR, 2019.
[67] Hugo Touvroa, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv: 13971, 2023.
[68] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgcaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[69] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824- 244377, 2022.
[70] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et al. Transformer:: Sfate-of-the-ar natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 45, 2020.
[71] Haojun Xia, Zhen Zheng, Yuchao Li. Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash- -lm: Enabling cost-effectie effective and highly-efficient large generative model inference with unstructured sparsity. Proceedings of the VLDB Endowment, 17(2)211-22) 2023.
14

---

[72] Mengzhou Xia. Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference Learning Representations ICLR 2024, Vienna, Austria, May 7-11. 2024.
[73] Guangxuan Xiao, Ji Lin, Mickael Seznec. Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[74] Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang. Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. BESA: pruning large language models with blockwise parameter- efficient sparsity allocation. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
[75] Yuhui Xu, Zhamming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. Think: Thinner key cache by query- query-driven pruning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025.
[76] Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yo Yu Hsich, Yaqing Wang, Yiling Jia, Gen Li, Ajay Kumar Jaiswal, Mykola Pechenizkiy, Yi Liang, al. Outlier weighed layerwise sparsity (owly: missing secret sauce for pruning ilms to high sparsity. In International Conference on Machine Learning, pages 57101-57115. PMLR, 2024.
[77] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 4791-4800. Association for Computational Linguistics, 2019.
[78] Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang, Lo- raprune: Structured pruning meets low-rank parameter-efficient fine-tuning. In Findings of the Association for Computational Linguistics ACL 2024, pages 3013-3026, 2024.
[79] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0100
2022.
[80] Yingtao Zhang. Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and- play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations. ICLR 2024, Vienna, Austria, May 7-11, 2024.
[81] Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao. Apt: adaptive pruning and tuning pretrained language models for efficient training and inference. In Proceedings of the 41st International Conference Machine Learning, pages 60812 60831, 2024.
[82] Kang Zhao, Tao Yuan, Han Bao, Zhenfeng Su, Chang Gao, Zhaofeng Sun, Zichen Liang, Liping Jing, and Jianfci Chen. Beyond 2: 4: exploring v: N: M sparsity for efficient transformer inference gpus. arXiv preprint arXiv:2410. 16135, 2024.
[83] Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, and Xue Lin. Pruning foundation models for high accuracy without retraining. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9681-9694, 2024.
[84] Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Zhuoqing Morley Mao, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models. Advances in Neural Information Processing Systems, 111110099--11999 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Introduction. The main claims made in the abstract and introduction clearly discuss the findings and challenges in prior work, as well as our method and achievement.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 7 We discuss the limitation of our method and provide insights for possible improvements future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
Justification:
Our paper does not include theoretical assumptions and results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: See Experiments. We provide the information needed for the reproduction of main experiment results, including setup, hyperparameters, libraries, and devices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: See supplemental material. We provide our code with sufficient instructions to enable reproduction for the experiment results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
wtt.g/ddeeeeeeeuuubbieiseoeeeeee Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle wwtpc//ggieeeeeeeeo Suci:///uuiieee/deeuuuubiiis
(CodeSubiisss wwtplubmmsssionoo ssiiiissicccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Experiments. We provide the experimental settings and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Experiments. We provide the computational cost of our method, including devices, memory, and execution time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httph//rhh/////rrrr....////////a
Answer: [Yes]
Justification: The research confirm with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper only focuses technical reports.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the code, datasets and models used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method proposed in our paper does not involve LLMs as important components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy ptttps://neuriiee /t//oonfeeencee//222222 for what should or should be described.
22

---

A Adaptive Unstructured Sparsity Allocation
LLaMA-7B Layerwise Sparsity
LLaMA3-88 Layerwise Sparsity
0.85
60%
70%
70%
0.55
80%
0.80
80%
0.80
Sparsity Ratio
Sparsity Ratio
0.75
9.70
0.70
0.65
0.55
0.55
Layer Index
Layer Index
Figure 8: Layer-wise sparsity distribution for LLaMA-7B.
Figure 9: Layer-wise sparsity distribution for LLaMA3-8B.
Wyste
MLP Wip
Figure 10: Intra-layer sparsity allocation for MI. layers in the 70% sparse LLaMA2-7B model.
B
Additional Ablation Studies
Pruning Granularity. To verify the scalability of the observation in Wanda that row-wise pruning the optimal choice for weight importance comparison, we conduct an ablation experiment under different pruning granularities, i.e. different comparison group selection strategies. The results in Table [ demonsttate that row-wise comparison remains the optimal choice for the adaptive sparsity allocation scenario, aligning with Wanda's observations for the uniform pruning strategy, which is discussed in Section 3.2 of our paper.
Training Datasets Size and Domain. Furthermore, we explore the sensitivity of our method the training dataset used (e.g., the size of the training set and its domain/distribttion)) We conduct an ablation study on the I.LMMA-77 model with different size of C4 training dataset across different sparsity levels. The results demonstrate that the performance improves with larger training datasets.
23

---

Table 7: Perplexity of sparse LLaMA2-7B on WikiText-2 under different pruning granularities.
Granularity 50%
60%
70%
80%
Row-wise 6.85
8.54
12.92
30.27
Column-wise 7.16
9.42
14.17
51.76
Layer-wise
7.32
14.40
210.48
28599.64 6.6
Table 8: Perplexity on Wikitext2 with different training samples and domains.
Samples
128
256
512
1024
Datasets
50%
60%
70%
70%
60.82
22.52
16.05
12.92
C4
6.85
8.54
12.92
80%
211.17
68.38
38.46
30.27
WikiText103
6.20
7.45
10.57
We also conduct ablation study on the LLaMA2-7B model with C4 and WikiText103 datasets across different sparsity levels. The results in Table(8rreeel that our method on two different training datasets achieve similar performance, demonstrating the robustness and generalization capability of our method across different datasets. Moreover, we notice that the perplexity of the sparsity pattern learned from WikiText better than C4. A potential reason is that the 1i training dataset has a more similar distribution to the validation data, which is obtained from WikiText2.
Speedups for Different Pruning Methods. We evaluate the inference speedups of Wanda and Lua-LLM for OPT-6.7B model at 50%-70% sparsity levels. The results in Table shows that the speedups of different pruning methods are quite the same for each sparsity level, which compatible with the parameter counts. The results demonstrate that although adaptive unstructured pruning methods introduce more irregular pattern for the sparse weight matrices, we can achieve meaningful speedup for the overall model with multiple performance optimization techniques employed in SpMM kernels like SpInfer, such as efficient sparse format and fine- grained execution pipeline.
Relative Order for Importance Metric. We conduct an ablation study on LLaMA2-78 BB (Multi- Head Attention architecture) and Mistral-7B (Grouped- Querry Attention architecture) models at 70% sparsity with different importance metrics (Magnitude, Wanda), and allocation strategies (with or without Lua-LLM). The results Table 10 demonstrate that: (1) When integrated with Lua-LLM, the relative order among Magnitude and Wanda importance metrics is preserved under different models and metrics. (2) For different importance metries and model architectures, our Lua-I.L. consistently outperforms the uniform pruning baseline Wanda.
Table 9: End-to-end inference throughput (token/s) speedups for different pruning methods.
Method
Sparsity
Dense
50%
60%
70%
Throughput t
696.6
844.9
1013.5
1200.6
Wanda
Speedup f
1.21x
1.45x
1.72x
Throughput
696.6
842.8
1012.7
1202.3
Lua-LLM
Speedup f
1.21
1.45x
1,73x
Table 10: Perplexity evaluation for importance metrics, model architectures and allocation strategies.
Metric
Magnitude
Wanda
Model
w/o Lua-LLM
Lua-LLM
w/o Lua-LLM
w/ Lua-LLM
LLaMA2-7B
141643
24.14
75.42
12.92
Mistral-7B
221.88
16.10
60.03
11.54
24

---

C Additional Evaluation Results
C.1 Zero-shot Accuracy Evaluation
Table 11: Zero-shot accuracy t results on seven downstream tasks for the pruned LLaMA-7B, LLaMA2-7B and LLaMA3-80 models at 60% sparsity level.
Model
Method
BoolQ 1
PIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA f
Mean
Dense
73.12
78.67
56.41
67.09
67.30
38.31
28.20
58.44
Wanda
67.92
72.20
43.45
59.75
56.57
29.95
24.20
50.58
LLaMA-7B
OWL
69.30
72.80
46.21
61.88
$6.36
31.57
25.20
51.90
68.26
45.97
54.46
31.83
51.33
Lua-LLM
70.89
73.88
52.82
64.17
62.04
35.24
26.20
$5.03
Dense
71.13
56.69
67.17
39.93
31.60
59.12
Wanda
65.29
71.70
43.77
64.17
64.60
30.97
25.80
52.33
LLaMA2-7B
OWL
66.85
72.74
46.64
66.77
67.76
32.34
27.60
54.39
DSA
71.71
72.63
45.39
65.75
65.32
31.91
28.80
54.50
Laa-LLM
70.28
73.29
52.69
65.51
65.70
37.29
30,00
56.39
Dense
81.25
79.71
60.18
72.69
80.09
50.51
34.80
65.60
Wanda
68.10
67.95
37.75
60.30
59.51
27.56
20.00
48.74
LLaMA3-8B
71.22
41.65
31.66
52.17
DSA
65.32
68.888
37.08
60.06
60.52
27.13
22.00
48.71
Lua-LLM
75.41
73.12
44.32
70.48
63.11
32.06
28.60
55.30
Table 12: Zero-shot accuracy t results on seven downstream tasks for the pruned LLaMA-7B, LLaMA2-7B and LLaMA3-SD models at 80% sparsity level.
Model
Method
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-e t
ARC-c
OBQA
Mean
Dense
73.12
56.41
67.09
38.31
58.44
Wanda
37.86
53.43
26.44
48.38
26.56
20.82
13:40
32.41
LLaMA-7B
OWL
49.82
53.70
26.54
50.67
26.35
19.71
11.00
33.97
DSA
37.83
26.68
49.80
27.61
20.73
32.45
Lua-LLM
61.71
64,36
35.03
$1.78
40.70
22.35
17.80
41.96
Dense
71.13
56.69
67.17
39.93
31.60
59.12
Wanda
37.83
52.61
25.87
48.54
26.64
19.54
13.40
32.06
LLaMA2-7B
OWL
37.86
54.35
26.43
49.96
27.65
19.28
32.65
DSA
37.83
53.86
26.38
49.09
27.06
19.80
12.00
32.29
Laa-LLM
61.38
61,32
32.18
55.01
38.93
19.62
17.80
40.89
Dense
81.25
79.71
60.18
72.69
80.09
50.51
34.80
65.60
Wanda
37.83
52.77
26.52
49.64
28.19
19.37
10.80
32.16
LLaMA3-8B
OWL
42,29
26.80
47.75
28.32
20.48
33.25
DSA
37.83
53.26
26.56
48.70
27.99
19.20
16.60
32.88
Lua-LLM
58.10
58.38
28.73
51.07
34.22
18.77
14.00
37.47
C.2 Few-shot Knowledge Reasoning Evaluation
We evaluate the 5-shot accuracy the MMLU dataset for LLaMA-7/13B and L.LMMA-7/133B models pruned by Wanda, OWI. and Lua-LLM, with 50%-70% sparsity levels as well as the dense baselines. The results show that:
(I) Compared to the dense LLaMA2-7B model (45.8%), the 60% sparse LLaMA2-133 model pruned with Lua-LLM demonstrates superior performance (46.6%), while Wanda (35.3%) and OWL (40.4%) cannot achieve this superiority at 60% sparsity level. A similar trend is observed for the LLaMA-VI pair (35.1% vs. 38.4%). These findings demonstrate that in terms of the so-called "large+sparse vs. small+dense" comparison, Lua-LLM achieves useful improvements at the higher sparsity level, which enhances its practical benefits.
(2) For larger models with 13B parameters, the performance gains of Lua-LLM are much larger than that of smaller models. This indicates that when pruning the smaller models (c.g., with 7B
25

---

Table 13: 5-shot MMLU accuracies (%) across different models, sparsity levels, and methods.
Method
Sparsity
LLaMA-7B
LLaMA-13B
LLaMA2-7B
LLaMA2-153
Dense
0%
35.1
47.0
45.8
55.7
Wanda
30.5
38.7
34.2
48.3
OWL
50%
30.9
40.6
33.6
48.7
Lua-LLM
31.3
41.2
34.8
49.4
Wanda
27.2
31.8
28.9
35.3
OWL
28.5
34.3
30.3
Lua-LLM
30.1
38.4
32.1
46.6
Wanda
24.6
25.3
24.5
26.8
OWL
70%
25.8
27.3
25.6
28.2
Lua-LLM
26.7
30.6
26.5
32.0
parameters), their MMLU accuracies relatively harder preserve. Moreover, OWL faces performance degradation (33.6%) compared to Wanda (34.2%) for the 50% sparse LLaMA2-7B model, while Lua-LLM (34.8%) consistently outperforms the Wanda baseline, which demonstrates the robustness of our method across different sparsity levels and models.
C.3 Integrate with Post-training Pruning Techniques
Lua-LLM is compatible with post-training pruning techniques like SparseGPT. Although SparseGPT iteratively selects block-wise masks and updates the other unpruned weights to compensate the pruning error, which is expensive integrate into the training process, we can use our Lua-LLM to search for block-wise sparsity, and then apply the sparsity allocation to SparseGPT.
To verify this, we adopt a block-wiss mask selection granularity for Lua-LLM, and search for the optimal sparsity allocation for these intra-layer groups. After the learning process, we consider the searched sparsity allocation as the block-wise sensitivity statistics to weight pruning, and apply it to SParseGPT, which uses the searched sparsity to select block-wise masks and updates the other weights. The results show that our allocation strategy achieves superior performance at different sparsity levels. However, the results also demonstrate that the weight update strategy adopted by SparseGPT is not sufficient to recover model performance at higher sparsity ratios, which underscores importance of mask selection granularity.
Table 14: Wikitext2 perplexity of sparse LLaMA2-7B models pruned with SparseGPT.
Method
50%
60%
70%
80%
SparseGPT
6.99
10.18
28.50
113.36
SparseGPT OWL
6.94
9.21
20.32
90.44
SparseGPT w. Lua-LLM
6.83
9.05
16.14
43.63
C.4 Integrate with Fine-tuning Process
LoRA fine-tuning. We conduct an ablation study on sparse LLaMA2-7B pruned with Wanda and Lua-LLM for LoRA fine-tune one-ttning with Alpaca training dataset. Each model is fine-tuned on A100 GPU for 3 epochs, which takes about 13 hours. We report the training loss and evaluate the model performance after fine-tuning. The results show that: (1) The loss converges in around 3 epochs, ensuring a meaningful comparison. (2) Compared to the final converged training loss of Wanda (in around 13 hours), our Lua-I.LM achieves comparable values in 0.5 epoch (in around 2 hours). (3) Compared to Wanda, our Lua-LLM improves model quality after fine-tuning.
Full Parameter Fine-tuning. We conduct full parameter fine-tuning on the auxiliary arriii subset of MMLU datasets for the sparse weights in 70% sparse LLaMA2-7B models, pruned by Wanda and Lua-LLM, respectively. Each model is fine-tuned on 4 A 1000 GPUs for 20000 steps (in around 25 hours). The results show that Lua-LIM achieves superior performance compared to Wanda under further fine-tuning, which consistent with the findings in prior literature that optimizing the mask selection strategy helps to improve the performance of pruned models under re
tunnng
26

---

process. Moreover, the experiment also demonstrates the substantial computational overhead of the full parameter fine-tuning process on large language models, and we would like explore other cost-effective weight reconstruction strategies with global optimality in our future study.
Table 15: Training loss convergence of sparse I.I LMMA2-7B during the LoRA fine-tuning process.
Epoch
0
0.5
1.5
2
2.5
3
Wanda (70%)
4.09
1.21
1.16
1.14
1.13
1.12
1.12
Lua-LLM (70%)
2.50
1.10
1.07
1.05
1.04
1.03
1.02
Wanda (80%)
7.03
1.92
1.78
1.72
1.68
1.66
1.66
Lua-LLM (80%)
3.10
1.42
1.35
1.32
1.31
1.29
1.28
Table 16: MMLU (%) evaluation with and without fine-tuning process.
Method
MMLU (%)
Wanda
24.5
Wanda W. LoRA FT
26.8
Wanda w. Full FT
31.3
Lua-LLM
26.5
Lua-LLM W. LoRA FT
30.3
Lua-LLM w. Full
35.6
27


Reviews:
[{"review_id": "MuO7Xzsfdg", "reviewer": "Reviewer_Gajc", "summary": "Lua-LLM is a novel pruning framework for LLMs. The method learns an optimal unstructured sparsity pattern on a layerwise and within-layer basis. Global pruning is prohibitively complex, and existing layerwise pruning metrics lead to suboptimal parameter allocations. Lua-LLM outperforms other LLM pruning methods, particularly at high sparsity levels, and can lead to inference speedups due to integration with a preexisting framework.", "questions": "1. What are the speedups of other pruning methods (with or without the SpInfer framework)? \n2. Could you compare the complexity of performing pruning on Lua-LLM vs other methods? I see section A addresses this, but some numbers would be nice", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, I think this is a solid paper with an interesting method and sufficient experimentation to prove the efficacy of the method.\n\nStrengths:\n- The paper introduces a technically novel method and clearly explains the process both intuitively and in rigorous detail.\n- The proposed method outperforms existing baselines.\n- The method leads to both lower parameter count and improved inference speed.\n- Generally clear writing, good use of mathematical formalism, nice figures and plots. \n\nWeaknesses:\n- (See questions) The method seems to be computationally more expensive than some of its comparison points. This is fine but a detailed analysis of the complexity would be appreciated. \n- An algorithm could help with clarity. The description of the method is a bit spread out, and I think a concise algorithm, even if it is in the appendix, could help provide a clear pointer to the steps in the method.", "was_revised": true, "final_justification": "I was satisfied with the authors’ response and think this is a solid paper.", "rebuttal": "We are glad that the reviewer found that our work is technically novel and clearly written. We thank the reviewer for the constructive feedback and appreciate the opportunity to address the questions you have raised.\n\n---\n\n> Q1: Detailed analysis of the complexity.\n\n> Q4: Could you compare the complexity of performing pruning on Lua-LLM vs other methods? \n\n**A1**: Following the suggestions, we compare the **pruning time costs of Wanda, OWL, DSA, and Lua-LLM for LLaMA-7B model**  (**See Table below**). We agree with the reviewer that NAS-based pruning methods (DSA, Lua-LLM) are computationally more expensive than proxy-based methods (Wanda, OWL), and we would like to consider this as a meaningful trade-off for the significant performance improvements. Besides, we would like to clarify that our Lua-LLM (1 hour) is significantly faster than evolutionary methods like DSA (12+ hours), since we leverage a soft Top-K selection function to enable an efficient row-wise mask learning process.\n\n*Table 1:* *Pruning* *time comparisons for different pruning methods.*\n\n| Method | Wanda | OWL   | DSA  | Lua-LLM |\n| ------ | ----- | ----- | ---- | ------- |\n| Time   | 4 min | 6 min | 12 h | 1 h     |\n\n> Q2: An algorithm could help with clarity.\n\n**A2**: We thank the reviewer for the meaningful suggestion and we will incorporate an algorithm into our revised manuscript. Furthermore, we summarize the algorithm of our Lua-LLM into three stages as follows:\n\n(a) **Initialization**. We compute the importance scores for weight parameters with Equation (2), and uniformly initialize the row-wise threshold parameter t as the target sparsity p.\n\n(b) **Mask Learning**. We set the training loss with Equation (10). In the training process, we generate row-wise pruning masks with Equation (7) for forward-propagation, and update the learnable threshold parameters with back-propagation. After the training process, we save the learned threshold parameters.\n\n(c) **Pruning**. We use the threshold parameters and mask selection function in Equation (6) to prune the original model.\n\n> Q3: What are the speedups of other pruning methods (with or without the SpInfer framework)?\n\n**A3**: We evaluate the **inference speedups of Wanda and Lua-LLM for OPT-6.7B model** at 50%-70% sparsity levels. The experiment (**See Table below**) shows that the speedups of different pruning methods are quite the same for each sparsity level, which is compatible with the parameter counts. The results demonstrate that although adaptive unstructured pruning methods introduce a more irregular pattern for the sparse weight matrices, we can achieve meaningful speedup for the overall model with multiple performance optimization techniques employed in SpMM kernels like SpInfer, such as efficient sparse format and fine-grained execution pipeline.\n\n*Table 2: Inference throughput (token/s) speedups for different* *pruning* *methods.*\n\n| Method  | Sparsity             | Dense | 50%   | 60%    | 70%    |\n| ------- | -------------------- | ----- | ----- | ------ | ------ |\n| Wanda   | Throughput (token/s) | 696.6 | 844.9 | 1013.5 | 1200.6 |\n|         | Speedup              | -     | 1.21x | 1.45x  | 1.72x  |\n| Lua-LLM | Throughput (token/s) | 696.6 | 842.8 | 1012.7 | 1202.3 |\n|         | Speedup              | -     | 1.21x | 1.45x  | 1.73x  |\n\n---\n\nThanks for the attentive reading of the manuscript and constructive feedback. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are glad to discuss further questions and suggestions."}, {"review_id": "bg1iBo39qE", "reviewer": "Reviewer_iAjF", "summary": "The paper introduces Lua-LLM, a gradient-based global pruning framework for LLMs that learns unstructured sparsity allocation. Existing layer-wise pruning methods for LLMs often lead to suboptimal solutions by focusing on local errors, so Lua-LLM aims to address this by decomposing global pruning into row-wise subproblems. It uses a soft Top-K operator with a sigmoid function to approximate binary mask selection, enabling end-to-end optimization of sparsity thresholds.", "questions": "1. How to address the train-test mismatch between soft masks and hard binary pruning?\n\n2. Please add ablation studies on different pruning granularities.\n\n3. How to address the output-side weight importance in your method?", "limitations": "Yes (Have the authors adequately addressed the limitations and potential negative societal impact of their work? If so, simply leave “yes”)", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The method consistently outperforms state-of-the-art techniques (e.g., ATP, DSA, OWL), particularly at extreme sparsity levels (e.g., 80%). Reductions in perplexity and improvements in zero-shot accuracy demonstrate its effectiveness in preserving model capability during compression.\n\n- Lua-LLM learns sparsity allocation in as little as 1 hour for LLaMA-7B on 2× A100 GPUs, significantly faster than evolutionary methods like DSA (12+ hours).\n\n**Weakness**\n\n- Lua-LLM uses a sigmoid function to approximate Top-K selection during training (soft masks), but inference requires hard binary masks. This creates a fundamental conflict: training optimizes probabilistic weight retention, while inference enforces deterministic pruning. This inconsistency is not adequately addressed.\n\n- The paper emphasizes row-wise pruning as a key advantage but fails to compare it against other granularities (e.g., column-wise, block-wise) in ablation experiments. (I understand this is something oberved in Wanda, but you should verify it in your experiments, since it is a key design).\n\n- The authors acknowledge that output dimensions contain critical weights and notes that Wanda’s input-driven metric risks removing sensitive output-side weights. However, Lua-LLM retains Wanda’s design without incorporating explicit output-side importance signals, failing to address this limitation in its own sparsity allocation strategy.", "was_revised": true, "final_justification": "Thank you for the authors to address the concerns in the first round of review, I would like to raise the scores towards boarderline accept.", "rebuttal": "We are glad that the reviewer found that our work effectively preserves the model capability and achieves superior search efficiency. We thank the reviewer for the constructive feedback and appreciate the opportunity to address the questions you have raised.\n\n---\n\n> Q1: The inconsistency that training with soft masks while inference with binary hard masks, is not adequately addressed.\n\n> Q4: How to address the train-test mismatch between soft masks and hard binary pruning?\n\n**A1**: Our Lua-LLM employs the **straight-through-estimator (STE) technique** [1], enabling gradient computation via a soft approximation during back-propagation when processing non-differentiable functions. As discussed in Section 2, STE techniques have also been adopted in other learning-based pruning methods such as MaskLLM [2] and DISP-LLM [3]. In this paper, we further propose a novel approximation method based on the sigmoid function, which provides an effective approximation for the Top-K mask selection function. For the sigmoid-based soft mask in Equation (7), we set the value of parameter *λ* to the number of elements per-row, which is large enough to provide a precise approximation. **During the training stage**, we use the sigmoid-based soft mask in Equation (7) and search for sparsity allocation with **learnable threshold parameters**. After the training process, we **store these threshold parameters and use the hard mask selection function in Equation (6) to generate hard masks with pruning thresholds for the inference stage**. We will incorporate the detailed interpretation of the straight-through-estimator technique into Section 4.2 of our revised manuscript.\n\n> Q2: The paper emphasizes row-wise pruning as a key advantage but fails to compare it against other granularities.\n\n> Q5: Please add ablation studies on different pruning granularities.\n\n**A2**: We conduct **ablation experiments under different** **pruning** **granularities**, i.e. different comparison group selection strategies. The results (**See Table below**) demonstrate that row-wise comparison remains the optimal choice for the adaptive sparsity allocation scenario, aligning with Wanda's observations for uniform pruning strategies, which is discussed in Section 3.2 of our manuscript.\n\n*Table 1: Perplexity of Lua-LLM with LLaMA2-7B on WikiText-2 under different pruning granularities.*\n\n| Granularity | 50%      | 60%      | 70%       | 80%       |\n| ----------- | -------- | -------- | --------- | --------- |\n| Row-wise    | **6.85** | **8.54** | **12.92** | **30.27** |\n| Column-wise | 7.16     | 9.42     | 14.17     | 51.76     |\n| Layer-wise  | 7.32     | 14.40    | 210.48    | 28599.64  |\n\n> Q3: Lua-LLM retains Wanda’s design without incorporating explicit output-side importance signals, failing to address this limitation in its own sparsity allocation strategy.\n\n> Q6: How to address the output-side weight importance in your method?\n\n**A3**: We thank the reviewer for raising this meaningful discussion for output-side importance signals. We would like to clarify that **Lua-LLM do not manually allocate sparsity based on** **heuristic** **statistics (e.g., output-side importance signals), in order to avoid suboptimal** **pruning** **results**. Instead, we adopt a NAS-based pruning strategy, which **automatically capture the output-side sensitivity to weight pruning, using gradient-based** **optimization** **techniques**. Our method decomposes the mask selection process into two components: intra-row and inter-row processing. (a) During **intra-row weight comparison**, all weights are within the same output channel. Consequently, calculating their importance metrics does not require considering output-side weight importance, and we can directly use Wanda's metric in Equation (2) to capture the weight importance with input-side signals. (b) For **inter-row sparsity allocation**, we do not manually allocate sparsity based on heuristic statistics, but with a NAS-based pruning strategy. This approach allows us to search for the optimal solution and control the overall sparsity via a regularization loss, eliminating the need for manual determination of per-row sparsity based on importance scores.\n\n---\n\nThanks for the attentive reading of the manuscript and constructive feedback. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are glad to discuss further questions and suggestions.\n\n---\n\n[1] Bengio, Yoshua, Nicholas Léonard, and Aaron Courville. \"Estimating or propagating gradients through stochastic neurons for conditional computation.\" *arXiv preprint arXiv:1308.3432* (2013).\n\n[2] Fang, Gongfan, et al. \"Maskllm: Learnable semi-structured sparsity for large language models.\" *Advances in Neural Information Processing Systems* 37 (2024): 7736-7758.\n\n[3] Gao, Shangqian, et al. \"Disp-llm: Dimension-independent structural pruning for large language models.\" *Advances in Neural Information Processing Systems* 37 (2024): 72219-72244."}, {"review_id": "GxQjL0HshR", "reviewer": "Reviewer_wteP", "summary": "Unstructured sparsity's generality is great for maintaining model quality when pruning, but this generality also complicates deciding where to put the sparsity.  This submission tackles this problem by directly learning the optimal sparsity for each row of the model's weights.  Each row is initialized with uniform sparsity (given some target p), and the mask is chosen with Wanda's importance score.  The mask is then optimized via gradient descent; through this process, the sparsity in each row is adjusted via a soft top-k operator given the learnable threshold and importance scores for each weight.  A regularization term seeks to control the model's overall sparsity, while standard language modeling loss adjusts the sparsity per-row.  Empirical evaluation shows modest perplexity benefit compared to baselines at 50% sparsity, but improvements are pronounced at 80% sparsity, and zero-shot accuracy shows meaningful improvements at 70% sparsity.  The authors performed ablation studies to demonstrate the importance of searching for the right sparsity distribution, their regularization term, and how different importance metrics affect final model quality.", "questions": "1. What batch size and sequence lengths were used for inference speedups in table 4?\n2. Is Lua-LLM robust to different importance metrics?  The ablation study shows that different importance metrics behave differently for one model and metric, but this does not show that the relative ordering among importance metrics is preserved under different models or metrics, or that Lua-LLM improves the results of each importance metrics without Lua-LLM.\n- Regularization:\n3. What was the final sparsity achieved by each model in Table 6?  (Does regularizer strength affect sparsity, or just mask quality?)\n4. What does it look like if this regularization term \"doesn't converge\" (line 322)?\n- Behavior under fine-tuning: the results presented aren't strong enough to convince me to use unstructured sparsity without adjusting weights to compensate for those that are removed.  Two questions follow: \n5. Is Lua-LLM inherently incompatible with techniques like SparseGPT?  It seems like it may be too computationally expensive to re-compute weight importances with changing masks, since other weights are updated as part of the sparsification process.\n6. Are these layer-wise sparsity reallocations necessary in the face of continued pre-training or fine-tuning?  If post-training techniques aren't sufficient to recover quality, and weight adjustments are necessary, it is important to show that improving the mask selection meaningfully improves model quality after weights are adjusted.\n\nThe most important questions are (6) and (5).  Questions (1), (3), and (4) are important for the submission's quality, and (2) is nice-to-have, but not crucial.", "limitations": "Unstructured sparsity is notoriously difficult to accelerate on GPUs.  The submission relies on SpInfer, which itself noted that when the batch size * sequence length is \"large\" (as is often the case in the prefill phase of model deployment), it becomes slower than dense operation due to the overheads.\nIf my understanding is correct, Lua-LLM is incompatible with SparseGPT and other techniques that adjust the weight values themselves.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\nThis submission's strengths are its clarity and originality.  \n- Clarity: I generally found the paper easy to read with clear motivation.  It answered most of my questions in the section after they occurred to me, and I feel like I could likely reproduce the results given the descriptions and equations, each of which was helpful.\n- Originality: I believe the ideas within are novel in this context -- soft top-k is not new, but its use here is unique, and the regularization to control overall sparsity seems like an elegant solution.\n\n## Weaknesses\nIts main weakness is in its significance.  While it does improve upon the baselines, Lua-LLM does not show that it's preferable to use one of these sparse models over a smaller, dense counterpart.  Take, for example, LLaMa2-7B and LLaMa2-13B in Table 1.  The dense 7B model has a perplexity of 5.47, and the 50% sparse 13B model, ostensibly around the same size, has a perplexity slightly higher, 5.89.  A similar trend is seen in LlAMA (1) 7B and 13B and OPT 6.7B and 13B.  Why should a user bother to start with a large version of a model and apply unstructured sparsity to achieve a smaller model, when directly training a smaller model gives similar (or better) quality?  Further, the speedups reported for the OPT models suggest that OPT-13B at 70% sparsity achieves only slightly lower throughput than the dense 6.7B model, but its perplexity is significantly worse - 20.96 vs. 10.86.\n\n\nI also have relatively minor qualms about the submission's quality.  Quality is otherwise good - claims are generally supported by evaluations, and while the limitations section should really be in the main text, I'll trust that this can be accommodated if the submission is ultimately accepted.  My qualms:\n- Evaluation is limited to only two model families: LLaMA and OPT.  Though different LLaMA versions were used, the results would be more compelling with a broader set of models with more varying architectures.\n- The regularization term's benefit is not clear.  What does it look like if this term \"doesn't converge\" (line 322)?  Does the final overall sparsity of the model change with regularization strength?\n- Details for the inference speedups are missing - what batch size and sequence lengths (input/output) are used?", "was_revised": true, "final_justification": "The authors addressed all of my concerns, and I'm now confident that the method is well-founded theoretically and can provide meaningful results in a wide range of scenarios as shown by the new empirical results provided during the rebuttal phase.", "rebuttal": "We are glad that the reviewer found that our work is clearly motivated and novel. We thank the reviewer for the constructive feedback and appreciate the opportunity to address the questions you have raised.\n\n---\n\n> Q1: Significance concerns: Lua-LLM does not show that it's preferable to use one of these sparse models over a smaller, dense counterpart.\n\n**A1**: We thank the reviewer for raising this meaningful discussion regarding performance comparisons between large sparse LLMs and small dense LLMs with similar parameter counts.\n\nFirst, we would like to clarify that large sparse LLMs with unstructured sparsity are often better than small dense LLMs with similar parameter counts on **downstream task performance**, even without any weight updates, which is also discussed in Section 4.1 of Wanda's paper [1]. To verify the practical usage of unstructured pruning methods without weight updates, we **evaluate the average zero-shot accuracies on 7 downstream tasks for the LLaMA2-13B model at 50% sparsity level with Wanda, OWL, and Lua-LLM**. The results (**See Table below**) demonstrate that unstructured 50% sparse LLaMA2-13B with OWL (60.03%) outperforms dense LLaMA2-7B (59.12%), and our Lua-LLM further improve the accuracy to 60.91%.\n\n*Table 1: Performance comparisons between large sparse LLMs and small dense LLMs*\n\n| Model                    | Zero-shot Accuracy |\n| ------------------------ | ------------------ |\n| LLaMA2-7B (Dense)        | 59.12              |\n| LLaMA2-13B (Dense)       | 61.84              |\n| LLaMA2-13B (Wanda 50%)   | 58.47              |\n| LLaMA2-13B (OWL 50%)     | 60.03              |\n| LLaMA2-13B (Lua-LLM 50%) | 60.91              |\n\nMoreover, we understand the reviewer's concern that the 50% sparse LLaMA2-13B model has a higher perplexity than the dense LLaMA2-7B model, yet a potential reason for this mismatch is that the perplexity metric cannot reflect the **practical performance difference between different structures of models**. For example, LLaMA3-8B shows superior zero-shot accuracy than LLaMA2-7B (65.60% vs. 59.12%), but it has a larger perplexity than LLaMA2-7B (6.14 vs. 5.47). Notably, perplexity remains a straightforward and efficient evaluation metric for **comparing different pruning methods on the same model type**.\n\n*Table 2: Performance comparisons with perplexity and downstream accuracy.*\n\n| Model     | PPL      | Zero-shot Accuracy |\n| --------- | -------- | ------------------ |\n| LLaMA2-7B | **5.47** | 59.12              |\n| LLaMA3-8B | 6.14     | **65.60**          |\n\n> Q2: Evaluation is limited to only two model families: LLaMA and OPT.\n\n> Q6: Is Lua-LLM robust to different importance metrics?\n\n**A2**: We conduct an **ablation study on LLaMA2-7B (Multi-Head Attention architecture) and Mistral-7B (Grouped-Query Attention architecture) models at 70% sparsity with different importance metrics (Magnitude, Wanda), evaluation metrics (PPL, MMLU), and allocation strategies (with or without Lua-LLM)**. The results (**See Table below**) demonstrate that:\n\n(1) When integrated with Lua-LLM, the relative order among Magnitude and Wanda importance metrics is preserved under different models and metrics.\n\n(2) For different importance metrics, model architectures, and evaluation metrics, our Lua-LLM consistently outperforms the uniform pruning baseline Wanda.\n\n*Table 3: Robustness evaluation for importance metrics, evaluation metrics, and allocation strategies.*\n\n|                       | LLaMA2-7B |          | Mistral-7B |          |\n| --------------------- | --------- | -------- | ---------- | -------- |\n|                       | PPL       | MMLU (%) | PPL        | MMLU (%) |\n| Magnitude w/o Lua-LLM | 141643    | 23.8     | 221.88     | 24.2     |\n| Magnitude w/ Lua-LLM  | 24.14     | 25.4     | 16.10      | 25.8     |\n| Wanda w/o Lua-LLM     | 75.42     | 24.5     | 60.03      | 24.9     |\n| Wanda w/ Lua-LLM      | 12.92     | 26.6     | 11.54      | 27.2     |\n\n> Q3: The regularization term's benefit is not clear.\n\n> Q7: What was the final sparsity achieved by each model in Table 6? \n\n> Q8: What does it look like if this regularization term \"doesn't converge\" (line 322)?\n\n**A3**: We thank the reviewer for raising this meaningful discussion about the effectiveness of the regularization term. If the regularization hyperparameter is not large enough, **the regularization loss fails to converge to zero**. This indicates that the pruning mask learned by our method does not meet the target sparsity, and **the overall sparsity value is actually lower than the target one**, since this can lead to better model performance and lower training loss.  We consider the case as \"doesn't converge\", since it leads to an **unfair comparison**. When the regularization hyperparameter is sufficiently large, the regularization loss rapidly converges to zero, which strictly enforces the target overall sparsity.\n\n> Q4&Q5: What batch size and sequence lengths (input/output) are used?\n\n**A4**: We use the testcase with batch size set to 8, input sequence length set to 32, and output sequence length set to 256. In this case, the inference throughput of SpInfer is not bottlenecked by the prefill stage, and the sparse model integrated can achieve practical speedup compared to the dense baseline. SpInfer faces limitations during the prefill phase when batch size and sequence length are large, which leads to higher memory access overhead for the SpMM operations. A potential solution [2] [3] is to extract dense blocks of non-zeros in the sparse weight matrices and use dense matrix-matrix multiplication to achieve high throughput, and we plan to leave this for our future study.\n\n> Q9: Is Lua-LLM inherently incompatible with techniques like SparseGPT? \n\n**A9**: Lua-LLM is compatible with techniques like SparseGPT. Although SparseGPT iteratively selects block-wise masks and updates the other unpruned weights to compensate the pruning error, which is expensive to integrate into the training process, we can use our Lua-LLM to **search for block-wise sparsity, and then apply the sparsity allocation to the pruning process of SparseGPT**. \n\nTo verify this, we adopt a block-wise mask selection granularity for Lua-LLM, and search for the optimal sparsity allocation for these intra-layer groups. After the learning process, we consider the searched sparsity allocation as the block-wise sensitivity statistics to weight pruning, and apply it to SparseGPT, which uses the searched sparsity to select block-wise masks and updates the other weights. The results (**See Table below**) show that our allocation strategy achieves superior performance at different sparsity levels. However, the results also demonstrate that the weight update strategy adopted by SparseGPT is not sufficient to recover model performance at higher sparsity ratios, which underscores the importance of mask selection granularity. In **A10**, we further show that improving the mask selection meaningfully improves model quality after fine-tuning.\n\n*Table 4: Wikitext2* *perplexity* *of* *sparse* *LLaMA2-7B models pruned with SparseGPT.*\n\n|                      | 50%      | 60%      | 70%       | 80%       |\n| -------------------- | -------- | -------- | --------- | --------- |\n| SparseGPT            | 6.99     | 10.18    | 28.50     | 113.36    |\n| SparseGPT w. OWL     | 6.94     | 9.21     | 20.32     | 90.44     |\n| SparseGPT w. Lua-LLM | **6.83** | **9.05** | **16.14** | **43.63** |\n\n> Q10: Are these layer-wise sparsity reallocations necessary in the face of continued pre-training or fine-tuning?\n\n**A10**: We understand the reviewer's concern with the necessity of improving the mask selection process when considering further fine-tuning. Thus, we conduct an **ablation study on** **sparse** **LLaMA2-7B pruned with Wanda and Lua-LLM for** **LoRA** **fine-tuning with Alpaca training dataset**. Each model is fine-tuned on 1 A100 GPU for 3 epochs, which takes about 13 hours. We report the training loss and evaluate the model performance after fine-tuning. The results (**See Table below**) show that: \n\n(1) The loss converges in around 3 epochs, ensuring a meaningful comparison.\n\n(2) Compared to the final converged training loss of Wanda (in around 13 hours), our Lua-LLM achieves comparable values in 0.5 epoch (in around 2 hours).\n\n(3) Compared to Wanda, our Lua-LLM improves model quality after fine-tuning.\n\n*Table 5: Training loss* *convergence* *of* *sparse* *LLaMA2-7B during the* *LoRA* *fine-tuning process.*\n\n| Epoch         | 0    | 0.5  | 1    | 1.5  | 2    | 2.5  | 3    |\n| ------------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Wanda (70%)   | 4.09 | 1.21 | 1.16 | 1.14 | 1.13 | 1.12 | 1.12 |\n| Lua-LLM (70%) | 2.5  | 1.10 | 1.07 | 1.05 | 1.04 | 1.03 | 1.02 |\n| Wanda (80%)   | 7.03 | 1.92 | 1.78 | 1.72 | 1.68 | 1.66 | 1.66 |\n| Lua-LLM (80%) | 3.10 | 1.42 | 1.35 | 1.32 | 1.31 | 1.29 | 1.28 |\n\n*Table 6: Wikitext2* *perplexity* *of fine-tuned LLaMA2-7B models for different* *pruning* *methods.*\n\n| Method  | 70%   | 80%   |\n| ------- | ----- | ----- |\n| Wanda   | 13.50 | 56.27 |\n| Lua-LLM | **11.23** | **26.48** |\n\n---\n\nThanks for the attentive reading of the manuscript and constructive feedback. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are glad to discuss further questions and suggestions.\n\n---\n\n[1] Sun, Mingjie, et al. \"A simple and effective pruning approach for large language models.\" *arXiv* *preprint* *arXiv:2306.11695* (2023).\n\n[2] Rumi, Masuma Akter, et al. \"Accelerating sparse cnn inference on gpus with performance-aware weight pruning.\" *Proceedings of the* *ACM* *International Conference on Parallel Architectures and Compilation Techniques*. 2020.\n\n[3] Yang, Tao, et al. \"Spmmplu: A compiler plug-in with sparse ir for efficient sparse matrix multiplication.\" *2023 60th* *ACM*/*IEEE* *Design Automation Conference (DAC)*. IEEE, 2023."}, {"review_id": "6977tqlpSk", "reviewer": "Reviewer_tzDL", "summary": "This paper proposes a new model pruning technique for LLMs. Specifically, instead of uniformly allocating the sparsity ration per layer, or using heuristic statistics like in prior work, the proposed method uses an end-to-end approach to learn the sparsity allocation. Empirical experiments demonstrate promising results at high sparsity with practical speed up observed.", "questions": "- Is perplexity a good evaluation metric for comparisons between different pruning strategies? Table 1 suggests that Llama3-8B has higher perplexity than Llama-7B (for both dense models). However, their performances on downstream tasks show the opposite in Table 2.\n- Could the authors discuss practical trade-offs between structured and unstructured pruning strategies in terms of end-to-end speed up and downstream performances?\n- How sensitive is the proposed method to the training dataset used (e.g., the size of the training set and its domain/distribution)?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written. The related work section also gives reader a clear context to LLM pruning literature.\n- The proposed method is intuitive and well-motivated. Previous pruning approach adopts uniform sparsity allocation or using heuristic statistics (like outlier ratio as in OWL) to determine sparsity allocation. Instead, it is natural to consider a learning based approach to determine the sparsity ratio as considered in this work. This learning is made possible by employing sigmoid approximation for top-k selection which is intuitive and shown to be effective in the experiments.\n- Empirical results are promising, especially at high sparsity ratio. Further ablation studies are done to help better understanding of the proposed method and its comparison to existing methods.\n\nWeaknesses:\n- While there are significant improvements on the perplexity evaluations, the improvements on downstream tasks are relatively small. Also, the downstream tasks considered here can be made more comprehensive by adding common evaluation datasets like MMLU.\n- The practical speed up observed is not very large even at 70% sparsity ratio. Could the authors discuss whether this limits the practical usage of unstructured pruning methods?", "was_revised": true, "final_justification": "The method is well-motivated and the paper is clearly written with promising empirical results.", "rebuttal": "We are glad that the reviewer found that our work is clearly written and well-motivated. We thank the reviewer for the constructive feedback and appreciate the opportunity to address the questions you have raised.\n\n---\n\n> Q1: The downstream tasks can be made more comprehensive by adding common evaluation datasets like MMLU.\n\n> Q3: Is perplexity a good evaluation metric for comparisons between different pruning strategies?\n\n**A1**: We agree with the reviewer that when comparing different types of models, there can be mismatchs between perplexity and downstream performance. However, we think that perplexity remains a straightforward and efficient evaluation metric for comparing different pruning methods **on the same model type**.\n\nTo demonstrate more comprehensive comparisons, we evaluate the 5-shot accuracy on the MMLU dataset for LLaMA2-7B and LLaMA3-8B model pruned with Wanda, OWL, and Lua-LLM, at 60% and 70% sparsity levels. The results (**See Table below**) show the superior performance of our method.\n\n*Table 1: Comparisons for 5-shot accuracy on the MMLU dataset.*\n\n| Sparsity | Method  | LLaMA2-7B | LLaMA3-8B |\n| -------- | ------- | --------- | --------- |\n| 60%      | Wanda   | 28.9      | 29.9      |\n|          | OWL     | 30.3      | 34.0      |\n|          | Lua-LLM | **32.1**  | **35.1**  |\n| 70%      | Wanda   | 24.5      | 26.1      |\n|          | OWL     | 25.6      | 27.3      |\n|          | Lua-LLM | **26.5**  | **28.6**  |\n\n> Q2: The practical speedup observed is not very large even at 70% sparsity ratio. Could the authors discuss whether this limits the practical usage of unstructured pruning methods?\n\n**A2**: We thank the reviewer for raising this meaningful discussion for the practical usage of unstructured pruning methods. Our inference evaluation is conducted with SpInfer framework. As discussed in its paper [1], SpInfer faces limitations during the **prefill phase** when batch size and sequence length are large, which leads to higher memory access overhead for the SpMM operations. Considering the relatively low performance degradation and substantial memory savings offered by unstructured pruning methods, optimizing SpMM operations on GPUs remains an important research topic for future work. Furthermore, future research directions include designing hardware-friendly sparsity patterns like V:N:M [2], which can leverage **Sparse** **Tensor** **Cores** on NVIDIA GPUs without significant performance degradation.\n\n> Q4: Could the authors discuss practical trade-offs between structured and unstructured pruning strategies in terms of end-to-end speed up and downstream performances?\n\n**A4**: **Structured** **pruning** removes entire components, such as attention-heads in the Attention layers and neurons in the MLP layers, which can directly generate a smaller \"dense\" model and achieve significant inference speedup on the GPU Tensor Core architecture. However, it typically suffers severe performance degradation. Even if employing weight reconstruction methods such as LoRA-finetuning [3] and ADMM-based reformation [4], the performance loss is still difficult to compensate, which limits its sparsity levels under 50% typically.\n\n**Unstructured** **pruning** removes less critical weight parameters at the element granularity, which offers the most flexibility and yields better post-training performance than structured pruning even at high sparsity levels. Despite the significant performance gains and memory savings, a major challenge for unstructured pruning is to achieve practical speedups on GPUs with substantial irregular memory access overhead, especially for large language models. Sparse inference framework like SpInfer [1] manages to optimize the SpMM operations for GPU Tensor Core architectures, yet its inference throughput is limited during the prefill stage.\n\nN:M sparsity serves as a trade-off between flexibility and efficiency, yet only 2:4 sparsity is supported by Sparse Tensor Cores on NVIDIA GPUs. Moreover, recent studies have focused on designing more **adaptive sparsity patterns**. For example, DISP-LLM [5] proposes a dimension-independent structured sparsity pattern, which prunes the channels for each weight matrix independently. To enhance the practical usage of model pruning, we will focus on other adaptive sparsity patterns in future exploration.\n\n> Q5: How sensitive is the proposed method to the training dataset used (e.g., the size of the training set and its domain/distribution)?\n\n**A5**: We conduct an **ablation study on the LLaMA2-7B model with different size of C4 training dataset across different sparsity levels**. The results (**See Table below**) demonstrate that the performance improves with larger training datasets. Notably, even with only 128 training samples, Lua-LLM demonstrates the ability to achieve superior results (60.82) compared to Wanda (75.42).  \n\n*Table 2:* *Perplexity* *of* *sparse* *LLaMA2-7B models on Wikitext2 with different training samples.*\n\n| samples | 128    | 256   | 512   | 1024  |\n| ------- | ------ | ----- | ----- | ----- |\n| 70%     | 60.82  | 22.52 | 16.05 | 12.92 |\n| 80%     | 211.17 | 68.38 | 38.46 | 30.27 |\n\nWe also conduct an **ablation study on the LLaMA2-7B model with C4 and Wikitext103 datasets across different sparsity levels**. The results (**See Table below**) reveal that our method on two different training datasets achieve similar performance, demonstrating the robustness and generalization capability of our method across different datasets.\n\n*Table 3: Performance of* *sparse* *LLaMA2-7B models with different training datasets.* \n\n|             | 50%  |      | 60%  |      | 70%   |      |\n| ----------- | ---- | ---- | ---- | ---- | ----- | ---- |\n|             | PPL  | MMLU | PPL  | MMLU | PPL   | MMLU |\n| C4          | 6.85 | 34.5 | 8.54 | 32.1 | 12.92 | 26.5 |\n| WikiText103 | 6.20 | 34.2 | 7.45 | 31.7 | 10.57 | 26.8 |\n\n---\n\nThanks for the attentive reading of the manuscript and constructive feedback. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are glad to discuss further questions and suggestions.\n\n---\n\n[1] Fan, Ruibo, et al. \"Spinfer: Leveraging low-level sparsity for efficient large language model inference on gpus.\" *Proceedings of the Twentieth European Conference on Computer Systems*. 2025.\n\n[2] Castro, Roberto L., et al. \"Venom: A vectorized n: M format for unleashing the power of sparse tensor cores.\" *Proceedings of the International Conference for* *High Performance Computing**, Networking, Storage and Analysis*. 2023.\n\n[3] Ma, Xinyin, Gongfan Fang, and Xinchao Wang. \"Llm-pruner: On the structural pruning of large language models.\" *Advances in neural information processing systems* 36 (2023): 21702-21720.\n\n[4] Shen, Xuan, et al. \"Search for efficient large language models.\" *Advances in Neural Information Processing Systems* 37 (2024): 139294-139315.\n\n[5] Gao, Shangqian, et al. \"Disp-llm: Dimension-independent structural pruning for large language models.\" *Advances in Neural Information Processing Systems* 37 (2024): 72219-72244."}]

Author rebuttal:

Meta-review: The authors propose a learning-based global pruning framework for optimal unstructured sparsity allocation in LLMs. The method leverages a soft Top-K operator to approximate the importance-based mask selection mechanism. The experimental results on LLaMA and OPT families demonstrate the effectiveness of the proposed approach.

The paper is technically sound and novel, and it clearly motivates and explains the methodology. The claims of the authors are supported by empirical evidence, showing that the method outperforms baselines (eg, in terms of number of parameters and inference speed). The authors also provided an extensive rebuttal and addressed all the concerns of the reviewers. For instance, the authors clarified the theoretical foundation of the paper and provided additional convincing empirical evidence. All reviewers voted for acceptance.
Decision: Accept (poster)