PROXSPARSE: REGULARIZED LEARNING OF SEMI-STRUCTURED SPARSITY MASKS FOR PRETRAINED LLMS
Authors: Hongyi Liu
Rajarshi Saha
Zhen Jia
Youngsuk Park
Jiaji Huang
Shoham Sabach
Yu-Xiang Wang
George Karypis
Venue: ICML 2025 poster (2025)
Keywords: Semi-structured sparsity
Large Language Models
Primary area: deep_learning->large_language_models

Abstract: Large Language Models (LLMs) have demonstrated exceptional performance in natural language processing tasks, yet their massive size makes serving them inefficient and costly. Semi-structured pruning has emerged as an effective method for model acceleration, but existing approaches are suboptimal because they focus on local, layer-wise optimizations using heuristic rules, failing to leverage global feedback. We present ProxSparse, a learning-based framework for mask selection enabled by regularized optimization. ProxSparse transforms the rigid, non-differentiable mask selection process into a smoother optimization procedure, allowing gradual mask exploration with flexibility. ProxSparse does not involve additional weight updates once the mask is determined. Our extensive evaluations on 7 widely used models show that ProxSparse consistently outperforms previously proposed semi-structured mask selection methods with significant improvement, demonstrating the effectiveness of our learned approach towards semi-structured pruning.

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
Hongyi Liu Rajarshi Saha? Zhen Jia Youngsuk Park Jiaji Huang? Shoham Sabach
Yu-Xiang Wang 24 George Karypis²
Abstract
cient, such model compression (Han et al., 2015; Frantar et al., 2022). By reducing memory footprint and accelerat- ing computation, model compression significantly improves the feasibility and cost-effectiveness of deploying LLMs at scale (Yuan al., 2024; Lin al., 2024; Tseng et al., 2025; Ozkara et al., 2025; Wei et al., 2025).
Large Language Models (LLMs) have demon- strated exceptional performance in natural lan- guage processing tasks, yet their massive size makes serving them inefficient and costly. Semi- structured pruning has emerged as an effective method for model acceleration, but existing ap- proaches are suboptimal because they focus on local, layer-wise optimizations using heuristic rules, failing to leverage global feedback. We present ProxSparse, a learning-based framework for mask selection enabled by regularized op- timization. ProxSparse transforms the rigid, non-differentiable mask selection process into a smoother optimization procedure, allowing grad- ual mask exploration with flexibility. ProxSparse does not involve additional weight updates once the mask is determined. Our extensive evalua- tions on widely used models show that Prox- Sparse consistently outperforms previously pro- posed semi-structured mask selection methods with significant improvement, demonstrating the effectiveness of our learned approach towards semi-structured pruning. Code available here.
Network pruning is commonly used reduce model size and lower computation cost by removing unimportant pa- rameters (Bai et al., 2024). Among various pruning patters,, semi-structured pruning (Mishra et al., 2021), or block-wise N:M sparsification, has emerged as a practical and effec- tive approach for LLM compression (Sun al., 2023; Fang et al., 2024). In this approach, only N non-zero elements are retained out of M consecutive elements within each parameter block. This semi-structured sparsity strikes bal- ance between model accuracy and hardware efficiency, and is well-supported by many hardware accelerators (Mishra et al., 2021), enabling efficient LLM serving.
Despite its advantages, finding an effective mask for LLMs remains challenging. Pruning must fol- low per-block structural restriction, making efforts other patterns hard to adopt. Additionally, extensive retraining after pruning is impractical due to LLMs' massive size (Ma et al., 2023; Chuang et al., 2024). Recent advances like Wanda (Sun et al., 2023) and SparseGPT (Frantar & Alis- tarh, 2023) improved cemi-structured pruning using minimal resources with only hundreds of calibration samples, but still struggle to maintain optimal performance after pruning. We identify two main challenges in finding effective semi- structured masks: 1. The heuristic rules used for mask se- lection cannot fully take advantage of the calibration dataset during pruning. Methods like SparseGPT and Wanda rely on the Hessian matrix and importance scores to select ele- ments to prune, but these lightweight criteria fail to effec- tively leverage learn from the calibration data. 2. Both methods focus solving "local" optimization problem associated with individual layer, without considering the broader, end-to-end optimization across the entire model. In those methods, pruning is based on localized informa- tion within each layer, without considering the connections across layers. Thus they cannot benefit from the global feedback, limiting the effectiveness of the pruning method.
1. Introduction
Large Language Models (LLMs) have demonstrated strong performance across a wide range of natural language pro- cessing (NLP) tasks (Achiam et al., 2023; Wei al., 2022). However, deploying and serving LLMs is not cost-efficient due to their massive size with billions of parameters (Fran- tar & Alistarh, 2023; Sui et al., 2025). To address the high computational demands and improve accessibility, various techniques have been proposed to make I.LMs more effi-
"Work done during internship at Amazon Web Service. 'Rice University 'Amazon Web Service "Technion *UCSD. Correspon- dence to: Hongyi <hongyi.liu@@ liu@@ricc.eduss Rajarshi <sahra- jar amazon.com>.. Yu-Xian Xiang W. <euxxxyyuu@ueeeeaccc.ee @ucsd.edu>>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMS
We advocate learning based solution for semi-structured mask selection that incorporates global feedback. We pro- pose ProxSparse, which learns to discover semi-structured masks through an end-to-end optimization process, rather than solely relying on local, heuristic-based decisions. Prox- Sparse enables a finetuning-lis procedure that learns the mask through only hundreds of calibration datasets with low resource utilization. The core of ProxSparse is the mask selection regularizer applied during learning, which trans- forms the rigid, non-differentiable mask-selection problem into gradual search process. ProxSparse progressively enforces semi-structured sparsity and frozen weight con- straints during training, and gradually shrinks unimportant weights to be pruned. ProxSparse does not involve addi- tional weight updates after determining the mask. One challenge in regularized learning is the efficiency of the solver, as slow solver makes end-to-end learning on LLMs impractical. To address this, we developed a fast solver us- ing iterative soft-thresholding, enabling efficient end-to-end learning at LLM scale.
effectively removes redundant parameters, improving effi- ciency. In LLMs, pruning has proven effective (Bai et al., 2024; Frantar & Alistarh, 2023; Huang et al., 2024), and can be categorized into three classes based on granularity.
Structured pruning (Ma et al., 2023; Xia et al., 2023) re- moves entire substructures like neurons or attention heads, reducing computation without extra overhead. However, its rigidity and lack of flexibility often lead to significant perfor- mance loss, requiring additional retraining recover accu- racy (Ma et al., 2023; Xia et al., 2023). Unstructured prun- ing (Frankle & Carbin, 2018) effectively preserves model accuracy by selectively removing unimportant weights in a fine- greiieraidd, non-uniform manner. However, its irregular pruning pattern is hardware-und unnfinedlll causing inefficient memory access. Semi-structured (block-wise N:M) spar- sity (Mishra et al., 2021) balances accuracy and efficiency by retaining N non-zero elements per M-sized block. Such patterns can be effectively leveraged by commercial hard- wares for real speedup (Fang et al., 2024; Sun et al., 2023; Mishra et al., 2021). while maintaining flexibility to remove unimportant weights. This work focuses semi-structured pruning for LLMs, introducing an end-to-end regularized learning framework towards optimal mask selection.
To comprehensively evaluate our method, we conducted extensive experiments on 7 widely used high-performance open-source models from four model families including Mistral (Jiang et al., 2023), Qwen (Yang et al., 2024), Open- Llama (Geng& & Liu, 2023) and Llama (Touvron et al., 2023) family. The benchmarks cover language modeling and seven widely used natural language reasoning tasks. The results show that our regularized learning significantly outperforms baselines consistently accross all evaluated models, produc- ing more effective pruning masks. Our contributions are summarized as follows:
2.2. Semi-Structured masks Selection for LLMs
Previous research has explored various mask-finding tech- niques for LLMs, with many showing success in semi- structured pruning. Here, review the most advanced methods for semi-structured mask selection.
Magnitude pruning (Han et al., 2015) is a standard tech- nique that removes individual weights based on their mag- nitudes with certain thresholds. Wanda (Sun et al., 2023) also avoids retraining or updating weights and introduces activation-aware pruning. The importance of each weight is evaluated using a per-output pruning criterion, where the weight magnitude is multiplied by its corresponding input activation using calibration data. SpanseGPT (Frantar & Alistarh, 2023) leverages the Hessian matrix to calculate the weight importance and reconstruction errors with the calibration data. These pruning methods typically solve a local optimization problems, providing efficient and low- resource compression techniques (Ma et al., 2023; Frantar & Alistarh, 2023; Frantar et al., 2022; Sun et al., 2023).
We propose to apply mask selection regularizer for end- to-end learning of semi-structured masks in LLMs. It allows gradual mask discovery with gradient feedback, enabling global optimization with flexibility, which leads to substantial improvements.
We developed an efficient proximal gradient descent solver for the semi-structured sparsity regularizer. This method is 10x faster than gradient descent-basee solvers and 100x faster than Interior Point Method (IPM) solvers, enabling end-to-end regularized learn- ing at LLMs scale efficiently.
Across all tested models, ProxSparse consistently im- proved perplexity (PPL) and accuracy on 7 common- sense reasoning tasks. It outperforms the previous SOTA pruning basclines at the same scale by up to 35% in PPL and 20% in zero-shot tasks, highlighting its effectiveness.
On the other hand, learning- based solutions for pruning have been explored in previous works, particularly in vision tasks. The main challenge the non-differentiable nature of mask selection, and techniques like Frraight-Througe Estimators (STE) (Bengio et al., 2013) have been proposed to overcome this. However, these methods typically require large-scale retraining, which is difficult for LLMs due to their enormous size. In our work, we propose to use the mask selection regularizer and efficiently identify the optimal mask in a
2. Preliminaries and Problem Setup
2.1. Large Language Model pruning
The massive size of LLMs has drawn attention to model compression to reduce serving overhead. Network pruning
2

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
learned manner with only hundreds of calibration samples without extensive retraining. A recently proposed learning- based method, MaskLLM (Fang et al., 2024), introduces a large-scale leamiig-basse based approach (~100,000 samples) to learn pruning masks using Gumbel Softmax sampling. Our approach employs a different design and operates with ~v 1000x smaller sample size (~1100 samples). We consider MaskLLM complementary to our approach, as it focuses on the regime that learns with large-scale data samples. We provide more comparison and discussion in Sec. 4.3.4.
2:4 sparsity, and we discuss the extension to other sparsity patterns in Appendix G.
To address the challenges posed by the non-convex and non-differemtiable nature of (1), our strategy for solving (I) involves (a) designing a relaxation of the problem with hard constraints into a (Lagrange) regularized form (b) devel- oping a principled optimization algorithm for solving the relaxed problem, thereby facilitating the learning process.
3.1. Relaxation and Structure-inducing regularization
2.3. Problem setup
We start by rewriting (1) into an equivalent form:
Let Wo E gd be the pre-trained weights of the model and L(W) be the (population) loss function for the model with weight W. We say a W E IRd is 4-sparse if for every block of 4 parameters in W only 2 are non-zero.
min W
L(W),
s.t.
W is 2:4 sparse,
(2a)
Maskw @ (W - Wo) = 0,
(2b)
Our goal is to solve the pruning problem by finding appropriate semi- cemi-structured sparse masks while keeping the weights of the pretrained model frozen.
where Maskw selects the non-zero elements of W, Wo denotes the original pretrained parameter weights.
We may express our task using the following stochastic optimization problem:
This seemingly trivial reformulation changes the variables to optimize from a Boolean mask to continuous weight vector which makes it more amenable to continuous optimization.
min M
L(Wo O M),
Next, we propose a relaxation of the two constraints (2a) and (2b) into regularized form that gradually induces these structures:
S.L. ME (0, . is 2:4 sparse,
where L denotes the loss, mask M E 10 denotes Boolean-valued with the same shape as the frozen model weights Wo. and c denotes element-wise multiplication. The problem is hard to solve because C is non-convex and the constraints are combinatorial. Moreover, we do not have access C directly (since it's the expected loss). Instead, we have a small calibration dataset that we can stream through that gives stochastic first-order (gradient) access, if we assume they are new data points drawn from the test-data distribution.
min L(W) + Regeza (W) + AgRegw, (W),
(3)
where Reg24 (W) promotes the structured sparsity con- straints and Regw, (W) penalizes the deviation away from the initial pretrained weight Wo.
Reg2.4 decomposes into every 4-parameter block, where we apply the following regularizer (Kübler et al., 2025) to enforce the sparse pattern.
Rega,4, WERs (w) =/wi || |ass| ||aa|| [ua]|ws|] |
Given these constraints, our goal is not to solve (1), but rather to find efficient heuristics that work in practice. In Section 3. we propose our approach and highlight the inter- esting aspect of it. In Section 4, thoroughly evaluate our method in semi-structured sparse pruning in number of open-source LLM models.
(4)
+ |wwss/wa |ws] I [w [|wa]-
Proposition 1. The following statements hold true.
1. Reg2:4, wERs (w) 0 if and only if w is 2:4 sparse.
2. Reg2,4, wERs (w) is invariant to permutation of the co- ordinates.
3. Methodology
We introduce ProxSparse, a learning-based pruning method guided by a mask selection regularizer that generates high- quality semi-structured masks for efficient LLM serving. ProxSparse enables mask exploration in a global perspec- tive by leveraging the gradient-based method, taking into account cross-layer connections with end-to-end feedback, rather than relying on localized, heuristic-bars based approaches for abrupt pruning. In this work, we focus specifically on
3. Reg2:1, WERt (w) is differentiable when restricting to the "active set" {i [4][[w]]] > 0}.
Observe that by the first property, if A + oo the solu- tion is guaranteed to be (2a). The non-smoothness of (4) ensures that it enjoys a "shrinkage" property (analogous to & regulaarizztion for sparsity) such that it induces exact 2:4-sparsity even if X is not tending to oc.
3

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMS
To promote the locality constraint (2b), we design the sec- ond regularizer as follows.
Algorithm 1 ProxSparse:
Proximal Gradient Descent for End-to-End 2:4-Sparsity Pruning
I: Input: Initial pretrained weights wi- Learning rate schedule TJo TI..... Stochastic gradient oracle & that takes w and outputs g such that Eig] VL(w).
W
Regw, (W)= Wo + csign (W))
(W-
Wo)
where the division is coordinate-wist and sign(-) outputs 1 when 2 0 and 0 otherwise.
2: for k = 0, 1, 2.... do
SGD (or Adam) update. 4: V + WR The (9x + A2 VRew, (W&))
3: gk <- ((m))
This regularizer can be viewed as a special weight decay towards Wo. but it imposes stronger penalty for coordi- nates of W that are larger and nearly no penalty for those coordinates that are nearly 0. e sigg is added to avoid the numerical instability associated with (near)-0 division. Proposition 2. 1. Regwo = if and only if [[[]: = for all coordinates s.t. W; + O.
5:
WK+1 arg minw 2 + X Regz2 (W).
6:
end for
7: Output: Wo @ Mask)
In our problem, /= Pseudocode of this algorithm is given in Algorithm 1.
:- L + Rege and h == Regeg
2. Regw, (W) Rog (W * 0)).
The main benefit of the proximal gradient approach is that it does not prematurely commit to particular sparsity mask, or fix the weights at the initialization. Instead, the regular- izers are soft constraints, allowing ample wiggling room around the rigid constraint set for the gradient descent-based al gorithhm to potentially jump out of a suboptimal local re- gion, and thereby converge to a better qualifying solution.
3. Regw, (W) is continuously differentiable.
Together with Proposition 1, we observe that the nullspace of the two regularizers together is the feasible region of the original problem, which allows solution that satisfies the original problem's 's constraints.
optimize towards
Corollary 3. Regw, (W)- = 0 and Regz. = 0 if and only if W satisfies (2a) and (2b).
One issue of not imposing the constraint is that the last iterate might not be feasible after the specified number of it- erations. For those cases, we simply project the solution Wi to a 2:4-sp solution basing on magnitude and snap the surviving weights to Wo. All our experimental results are based on solutions that are exactly 2:4 sparse with weights unchanged from initialization.
To say it differently, if λ1, + co, the relaxed problem (3) is identical to the original problem (1). We encode the rigid and non-differentiable mask selection constraints into the learning objectives, enabling a Icarnable optimization process. Another benefit of transitioning from hard con- straints to soft regularization is that introduces "wiggling room", enabling flexibility during exploration. This allows the learning to make smoother, more informed pruning de- cisions with larger exploration space, rather than making abrupt changes during optimization, which could cause early commitment to suboptimal state as we will show in exper- imental Section 4.3.2 later. The main challenge now lies in an effective solving algorithm for the regularizer with efficiency, which is crucial to facilitate end-to-end mask learning for LLMs with scale.
3.3. Efficient Proximal Operator
An efficient solver for the proximal operator is essential for enabling end-to-end learning at LLM scale. Since L and Regw, are both differentiable, the efficient implementation of ProxSparse boils down to solving the proximal operator associated with Reg244-
10" arg min |1w wC34
v112 + ARegza(w))
(6)
2
3.2. Proximal Gradient Descent for 2:4 Sparsity
This is non-convex optimization problem. Kübler et al. (2025) showed that can be solved with three convex sub- problems.
To optimize (3), propose use the proximal gradient descent (Nesterov, 2013)- popular method for solving composite optimization problems of the form min.: f(x + h(x)) where f is differentiable but h is not.
Theorem 4 ((Kübler et al., 2025)). To solve (6) for any y IR,, it suffices to solve:
Proximal gradient descent iteratively updates x by alternat- ing between a gradient descent step on ff and proximal operator (a generalization of "projection") on h:
1 ||w - =112 + (Regaa(w)
min webt
(7)
where 2 = sorted is non-negative and sorted in de- scending order, i.e., 2 2 22 2 23 2 24 2 0. Moreover, the
y = I2- ffr()
(5a)
T++1 = arg min a - 2112 + +xx)..
(5b)
optimal solution to (7) must be one of the following three candidates:
4

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
1. "--sparse solution" " [21; A2; 0, 0];
Algorithm 2 ALM: Alternating Minimization
I: Input: E R4 (sorted, nonnegative), parameter X,
2. "3-sparse solution", [W), vdz via 0]
tolerance e, desired sparsity-level S = 3 or 4.
3. "dense solution" " [ 122 Wig wān]
2: Initialize w' = 0, w = 0,
3: W1:S < -1:8
where ii = arg min WER (0x(w) s.t. gaga(w) = 0} with
4: while ||w/ - -wl > c do
for E {1, .... 9] do
(s(w)= = 2 I|0 21:311 + A(w1wy + wyly + wzw1),
1
6:
W3 f- max
2 X X Keeeenn wyme, of
1it
This is soft-thresholding operator
and w = arg min (9s(w) s.t. Vgg(w) = 0} with
8: end for
go (w) being the objective function of (7). Meanwhile, {w] tulV"gs(ve ga = 0} and [w|V2 (1(w) t/2 0} are convex sets, making the corresponding optimization problems convex.
9: w' <- w
10: end while
11: Output: w
This result suggests that we can simply enumerate the three candidate solutions and return the one with the smallest objective value. Kübler et al. (2025) thus proposed to solve for the "3- "pparrss"" and "dense" solutions using interior point method (IPM) with log-determinant barrier function, lead- ing to the EnumlPM algorithm, which optimally solves (7). However, Enum/PM incurs high computational cost (Table 1). A faster heuristic, EnumPGD, was introduced to replace IPM with projected gradient descent without impos- ing semidefinite constraints. While EnumPGD improves efficiency, sacrifices provably guarantees.
Algorithm 3 EnumALM for solving (6)
I: Input: y E RR, parameter X, tolerance e
2: s + sign(y)
> elementwise
2, idx < clyl 'descending') idx is reverse index.
4:
[21, 22,, 22, 0,0)
2- sparse solution.
5: E ALM(2, X, e, 3)
3- spaarse solution.
6: Ø = ALM(2, X, c, =
> dense solution.
7: < arg min (i (
b|| 21| + ARegz-1(w)
8: Output: s @ w idx
b + is elementwise product
We propose new method based on alternating minimiza- tion (ALM) with convergence guarantees. The resulting EnumALM is even more efficient than EnumPGD (see Table/ for an numerical comparison). Moreover, in all 20,000 experiments in Table 1, EnumALM provides more optimal solutions than those of EnumlPM. This enables us to scale up the proximal gradient method for handling LLMs with billions of parameters in practice. An example regularization paths is illustrated in Figure 3 in Appendix C.
EnumIPM
EnumPGD EnumALM (ours)
Total runtime (sec)
561.70
43.31
8.52
Max suboptimality
10
10
100 13
Table 1. Comparison of the runtime and accuracy of solvers of (6) for solving 100 randomly generated problem instances, each with 200 different choices of X. The second row shows the worst-case suboptimality. IPM is guaranteed to give the optimal solution up-to tolerance parameter of 10-15 ALM achieves better objective value in all experiments than IPM, while GD occasionally gives solutions with slightly suboptimal objective values.
Pseudocode for ALM and EnumALM are given in Algo- rithm 2 and 3 respectively. Algorithmically, ALM works by iterating over the coordinates of w and minimizing ga or 91 over the current coordinate while keeping other coordi- nates fixed. The solution of this one dimensional problem is soft-thresholding:
to a critical point. Then we will argue that if Algorithm 3 returns the correct solution (they do in all our experiments!), then under mild assumptions on training loss and bound- edness of the parameters Wk, the outer-loop Algorithm I also converges to stationary point. The proofs of both propositions below are deferred to Appendix A.
Fact 5. Assume 2 O, the optimal solution to min 1 2) + aw is w max(z- a, 0}.
Proposition (Convergence of ALM). When Algo- rithm 2 halts with no more than 3 /| iterations. Also, at the limit + 0, the output of Algorithm converges to critical point of 9s when 5 = 3 (or of 94 when S = 4).
Observe that
soft- soft-thresholding is commonly used in L.1 regularized optimization for inducing (unstructured) spar- sity. Our algorithm can thus be viewed as iterative soft- thresholding with adaptive chosen threshold that induces 2:4 structured sparsity rather than standard sparsity.
Proposition 7 (Convergence of ProxSparse). Assume L is continuously differentiable, and that there exists B > 0 such that [|,, < B for all t 1, 3. Then Algorithm 1 converges to a critical point in the sense of the limiting subdifferential of the regularized objective function of (3) (see (Rockafellar & Wets, 1998)).
3.4. Convergence guarantees
Next, we study the convergence theory of ProxSparse. We first prove that the inner-loop Algorithm always converges
5

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
4. Empirical evaluation
same scale. More specifically, ProxSparse achieves PPL. of 9.91 vs. Wanda's 's 13.81 on OpenI lama-7b-22 with 28% improvement. Similarly, ProxSparse achieves a PPL of 8.51 on Llama-2-7b, compared to Wanda's 11.42, reflecting a 35% improvement. In the Llama-3. experiments shown in Table 9, ProxSparse reduces PPL from Wanda's 20.91 to 13.63. Compare to the Llama-2-7b model, Llama-3.1-88 have more information encoded in the model weights as much larger training corpus was used during pretraining. This significant performance improvement highlights the potential of ProxSparse's 's effectiveness in handling dense model pruning mask selection. Even when compared to SparseGPT, which updates the weights to minimize error, ProxSparse still outperforms by up to an 18% margin, as demonstrated in the Llama-2-7b experiments. In summary, across different models, ProxSparse consistently achieves better PPL with significant gap compared to other base- lines.
In this section, we provide comprehensive evaluations of ProxSparse by addressing the following research questions: 1. End-to-end performance: how does ProxSparse com- pare to other state-of oft------thrrrr pruning methods? 2. The in- depth analysis of mask selection regularizer: how does the regularizer contribute to finding the effective mask? 3. Efficiency benefit: does sparsified models produced by ProxSparse improve efficiency?
4.1. Models, tasks and baselines
We evaluated Sporse on four most advanced and widely used open-source LLM families: Mistral (Jiang et al., 2023), Qwen (Yang et al., 2024), OpenI.ama (Geng & Liu, 2023) and an Llama (Touvron et al., 2023) family. The specific models used in our experiments include Mistral-v0. Mistral-v0.3-7bb Qwen2.5-14b, OpenLlama-7 Llama- 2-7b, Llama-2- 13b and Llama-3.1 1--b..
Zero-shot Task Performance We present the perfor- mance analysis on seven widely used zero-shot natural language reasoning tasks. Consistent with the language modeling results, ProxSparse significantly outperforms both magnitude pruning and Wanda. In the Mistral-v0. ex- periments, ProxSparse achieved an average accuracy of 52.7%, compared to Wanda's 's 44.1%, marking 20% im- provement in performance. Even with weight updates in SparseGPT, ProxSparse consistently achieves higher accu- racy. Similar trends hold for Qwen2.5-14p and other mod- cls as well. This highlights ProxSparse's 's effectiveness in finding an optimal semi-structured mask to maintain supe- rior performance, even compared to pruning methods with weight reconstruction for error reduction.
We assess the performance of pruned models from different pruning mechanisms on both zero-shot tasks and language modeling. For calibration, we followed Wanda (Sun al., 2023) and SparseGPT (Frantar& & Alistarh, 2023) to utilize the C4 (Raffel et al., 2020) dataset for calibration. Seoo performance was evaluated with the FleutherAl LM-Hval- Hamesss (Gao et al., 2024) on seven widely used tasks (Liu et al., 2024), while Wikitext (Merity et al., 2016) perplexity (PPL) was used as the language modeling metric, consistent with previous evaluation protocol (Sun et al., 2023; Frantar & Alistarh, 2023). The experiments use 400 data samples for calibration unless specified, with consistent counts across baselines for fair comparison. We discuss MaskLLM and present ablation studies on mask effectiveness with regards to calibration sample size in Section 4.3.4. Comparisons on additional pruning mechanisms (ADMMPrune (Boža, 2024), OWL (Yin et al., 2023) and AlphaPrune (Lu et al., 2024)) are further detailed in Appendix E. For hyperparame- ters and configurations, we detail them in Appendix B. Our experiments were done on Nvidia A100 GPUs.
Analysis of Better Performance ProxSparse consistently outperforms all baselines across evaluated models. Its ad- vantage stems from the global feedback in mask explo- ration, which enables ProxSparse to overcome localized constraints. By optimizing in an end-to-end manner, ProxS- parse achieves superior performance gains.
4.2. End to end performance evaluation
4.3. Deep dive into the regularizing mechanism
We first present end-to-end performance comparison against other baselines that enforce 2:4 sparsity: magnitude prun- ing (Han et al., 2015), SparseGPT (Frantar & Alistarh, 2023), and Wanda (Sun et al., 2023). Table 2 presents Wiki- text PPL and performance on seven widely used zero-shot reasoning tasks. Overall, ProxSparse consistently outper- forms all baselines across tested models.
This section explores the core properties of the mask selec- tion regularizer. The regularizer relaxes rigid mask selection constraints into differentiable optimization for end-to-end learning. In the meantime, its added flexibility with "wig- gling room" enhances exploration for better convergence. We ask the question: how does this flexibility aid in explor- ing the optimal mask during optimization?
4.3. 1. HARD CONSTRAINT V.S. SOFT REGULARIZATION
Language modeling We first evaluate language modeling. ProxSparse surpasses magnitude pruning and outperforms Wanda, the SOTA mechanism without weight updates at the
To showcase the effectiveness of soft regularization with flexibility, we compare it with strict constraints. Unlike
6

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
Table Experimental results Wikitext perplexity (PPI.) and commonly used zero-shot natural language reasoning tasks comparing ProxSparse to 3 other baselines on 7 widely used LLMs (Llama-3. 1-8b results are deferred to Table 9), Bold indicates the best pruning performance, while italic represents the original unprumed performance. SparseGPT updates weights minimize reconstruction error. while the other methods keep retained weights frozen. ProxSparse consistently yields better results compared to all other baselines.
Weight Update
Wikitext PPL
ARC-C
ARC-E
SIQA
HellaSwag
OBQA
PIQA
TruthfulQA
AVG
Mistral-v0.1-78
4.91
0,503
0.809
0.467
0.612
0.324
0.806
0.354
0.554
magnitude
14.18
0.310
0.666
0.417
0.488
0.204
0.732
0.314
0.447
SparseGPT
9,43
0.345
0.684
0.418
0.469
0.240
0.730
0.316
0.501
Wanda
11.49
0.336
0.665
0.408
0.444
0.214
0.716
0.307
0.441
ProxSparse
8.92
0.362
0.698
0.428
0.525
0.232
0.756
0.350
0.527
Mistral-v0.3-7b
4.95
0.490
0.797
0.458
0.609
0.336
0.803
0.353
0.549
magnitude
13.52
0.332
0.665
0.413
0.488
0.226
0.738
0.309
0.453
SparseGPT
9.23
0.353
0.687
0,421
0.470
0.248
0.733
0.308
0.458
Wanda
10.97
0.311
0.648
0.408
0.442
0.206
0.716
0.300
0.433
ProxSparse
8.68
0.362
0.697
0.429
0.525
0,242
0.751
0.321
0.475
Qwen2.5-14B
4.93
0.56
0.822
0.554
0.634
0.342
0.814
0.493
0.602
magnitude
48.87
0.359
0.638
0.405
0.418
0.256
0.680
0.356
0.444
SparseGPT
9.19
0.405
0.750
0.476
0.512
0.296
0.753
0.367
0.507
Wanda
11.69
0.389
0.729
0.440
0.491
0.286
0.740
0.331
0.485
ProxSparse
9.28
0.456
0.772
0.456
0.535
0,290
0.756
0.406
0.525
OpenLlama-77-v22
6.48
0.387
0.725
0.441
0.557
0.296
0.789
0.336
0.504
magnitude
36.15
0.230
0.498
0,380
0.360
0.162
0.683
0.306
0.374
SparseGPT
11.35
0.278
0.602
0.412
0.428
0.214
0.713
0.301
0.420
Wanda
13.81
0.261
0.575
0.409
0.409
0.196
0.703
0.310
0.409
ProxSparse
9.91
0.281
0.616
0.415
0.472
0.236
0.720
0.299
0.434
Llama-2-7b
5.12
0.433
0.763
0.461
0.571
0.314
0.781
0.321
0.527
magnitude
54.74
0.301
0.618
0,411
0.454
0,216
0.701
0.322
0.432
SparseGPT
10.30
0.326
0.655
0.412
0.435
0.246
0.713
0.304
0).41
Wanda
11.42
0.311
0.623
0.403
0.413
0.248
0.706
0.305
0.430
ProxSparse
0.331
0.656
0.407
0.478
0,242
0.716
0.328
0.452
Llama-2-13b
4.57
0.485
0.794
0.473
0.601
0.352
0.791
0.314
0.544
magnitude
0.319
0.623
0.408
0.501
0.232
0.717
0.309
0.444
SparseGPT
8.14
0.378
0.714
0.437
0.478
0.282
0.735
0.296
0.473
Wanda
8.35
0.340
0.683
0.424
0.464
0.246
0.739
0.292
0.455
ProxSparse
0,383
0.720
0.427
0.532
0.288
0.723
0.319
0.486
the gradually sparse regularizer, projected gradient descent (PGD) imposes hard thresholding during optimization. We conducted four experiments for mask selection, testing each with both soft and hard con- straints, shown in Table 3. In proximal gradient descent, "hard sparsity constraints" in the table enforce zeroing two of every four weights after each update, ensuring rigid 2:4 structural sparsity. "Hard frozen weights" reset the two largest- magentte weights to their original values, enfore- ing strict objectives for mask selection. With the relaxed regularizer, weights gradually shrink towards the 2:4 pat- tern (shown in Figure 4(a)), while the retained weights are encouraged to approximate their original values. This relax-
Table 3. Wikitext PPL under hard/soft constraints. Relaxing mask selection constraints improves performance hard thresholding. Bold indicates the best result.
evaluate both regularizers
Rech with
Fronzzon weight
Spuniiyy constminnts
Both with hand Constraints
relaxation
relaxation
relaxation
Mistral-v0.5.7 OpenL 7b-v2
8.68
13.23
11.24
13.6
9.91
34
33,07
35.28
ation meets both objectives under more flexible constraints. Table 3 indicates that hard constraints performs worst, while relaxed constraints enhance performance. Fully regular- izing both semi remi-structured and frozen weight constraints maximizes flexibility, achieving the best results.

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
1205
Eas (adarced)
24 suaseed Ieceiao
0Z5 16thhoci
INST Tee: Ratt
69
00
02
Hermalis Soech Prognss
24
00
09
32
Horea
14
06 00
32 82 04 C5 06
07
31
13
14
a. Two-four sparse ratio evolvement
b. Dense ratio evalvement
C. Initial/final two-four ratio vs.
b. Mask similarity evolvement
Figure 1. Evolution of sparsity ratio on Llama-2-70 based on the degree regularization. (a) Evolution of the 2:4 sparsity ratio over learning progress, where an insufficient regularization degree leads to under-leariing (b) With a larger X1 parameters shrink more quickly towards 2:4 sparsity, resulting in early commitment to a suboptimal mask. (c) Comparison of the 2:4 sparse block ratios early (0.1 epochs) and final stages of learning. (d) Mask similarity between the final mask and the early mask obtained after 10% epochs of learning, An excessively large X1 results premature mask commitment, causing mask selection to stagnate and hindering optimal mask discovery.
Table 4. Wikitext PPL across ÅL. Optimal performance occurs at balanced regularization. Bold indicates the best performance.
Table S. Wikitext PPL across λ2 A broad optimal plateau suggests that performance remains stable across robust range of λ2.
0.001
0.01
0.25
0.5
10
0
0.5
2
5
20
100
2500
inf
Mistral-vO 3-7h
9.66
8.68
9.30
11:33
Mistral-vO. v0.3-7b
8.68
8.88
8.9
8.82
8.99
8.85
9.11
13,23
A
0.1
2
10
OpenLJama- 7b-22
9.91
9.92
9,96
2
10.12
20
10.41
10.9
12.38
inf
34
OpenLlamm-7b-g 7b-v2
25.33
11.25
10.99
9.11
10.13
10.97
12.11
23.29
33.09
4.3.3. THE FROZEN WEIGHT RETENTION ENFORCER
4.3.2. THE SPARSITY PATTERN ENFORCER
We analyze the impact of the frozen weight regularizer, with its strength controlled by A2. Using the optimal X from previous analyses, vary X2 to assess its effect. Interest- ingly, Table 5 shows a broad optimal performance plateau, suggesting that a robust range of X2 values can be applied without significantly affecting performance.
In the following sections, we analyze the contribution of each regularizer individually, starting with the sparsity pat- tern regularizer, which encourages 2:4 sparsity. The regular- izer coefficient, Å1, controls the strength of regularization: higher values enforce more aggressive parameter shrinkage, approaching a harder projection with less flexibility. To isolate the effect of regularization, we only study the semi- structured regularizer in this analysis. We examine how varying its strength impacts mask learning. As shown in Table 4, optimal mask selection occurs at balance between gradual and aggressive regularization- -smaller values lead to conservative mask evolution, while larger values impose stricter constraints, both reducing performance.
We further plot the evolution of the relative norm gap across X2 in Figure 2. This gap quantifies the difference in norm between the learned model and the original model, with the mask applied. It assesses how closely retained weights preserve their original values. We see that even without the regularizer, the relative norm gap stays ~0%% Adding the regularizer incurs small impact until the strength reaches a high extent. This may result from implicit frozen weight con- straints imposed by the sparsity pattern regularizer, which we leave it to future investigation. As Xę increases toward in- finity, strict projection enforcement degrades performance, aligns with previous finding and reinforces the need for gradual and flexible mask optimization.
To better understand this phenomenon, we analyze the reg- ularizer's impact in detail. We show the evolution of 2:4 sparsity across different X1 values for Llama-2-7b (Fig- ures 1) and OpenI Jama--lam-77 (Figure 4 in Appendix F) with consistent trend. We use Llama-2-7b as the example. if X is too low, the model remains largely dense, as shown in Figures 4(a), (b) and (c). This suggests under-learning, where unimportant weights are not fully recognized by the end of learning, resulting in incomplete mask selection. Conversely, a high X value leads to early commitment to a specific mask. In Figure 4(d), the yellow line shows similarity to the "early mask" obtained after just 10% of learning. The final mask retains -9.55% similarity to the early one, indicating stalled optimization. In between, a bal- anced strength allows flexible mask exploration that avoids premature commitment, while also enabling more effective learning than excessively low values.
4.3.4. PERFORMANCE EVOLUTION WITH VARYING NUMBERS OF CALIBRATION SAMPLES
Our method enables effective remi-structured mask selec- tion with only hundreds of samples. Here we analyze per- formance based on the number of calibration samples with OpenI OpenI-lama-76e lama-7b-v2 and Llama-2-7b. We compare our re- sults with MaskLLM, SparseGPT, and Wanda using 100, 200, and 400 samples. MaskLLM struggles with small sam- ple sizes, making it a complementary method to ours in large-scale learning. We use the statisties reported in the
8

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
Table 7. Speeduup and memory utilization improvements achieved by ProxSparse induced 2:4 sparsity models(left: speedup, right: memory reduction). ProxSparse delivers a 1.3x-1.35x speedup for matrix multiplication and 1.26x end-to-end inference speedup on the Mistral-v0.3-7b model. Additionally, ProxSparse reduces memory consumption by 29.5%-37.33% across different models, demonstrating its efficiency in both computation and memory utilization.
Oparding fl
Lame 2:bb
20%
Nom Cenees
10%
14%
Module name
Speedup ratio
Model Family
Memory gain
24%
self_attn g/k/v/o
1.35x
Openllama.7bor
70.50%
mlp up'downsgate
1.30x
Qwen2.5-11bb
67.50%
End to end inference
1.26x
Mistral v0.3-7b
62.70%
20 93
60
60 100
Figure 2.
The relative norm difference over different λ2. The rela- tive norm gap measures how closely retained weights match their original values post-training. with the semi-structured mask ap- plied. The relative norm remained low (~~20%) with minimal change until a high lambda value was applicd.
of matrix multiplication in linear layers. The results of Mistral-v0.3-76 (batch size of 1) are presented in Table 7. As shown in the table, 2:4 semi-structured sparsity induced by ProxSparse provides significant inference speedup for linear layers in LLMs, achieving average speedup gains of 1.3 to 1.35. Additionally, we measured the end-to-end inference wall-clock speedup and observed 1.26x speedup, consistent with other sparsification methods evaluated in our experiments. We emphasize that the inference speedup is not specific to our pruning method but rather a result of the inherent computational efficiency enabled by semi- structured sparsity.
Table 6. Wikitext PPL across calibration sample sizes. ProxSparse outperformed all methods, with performance slightly improved as sample size increased, confirming its effectiveness in optimal mask lcarning. Bold indicates the best performance.
Openl.lama-7h.x
100
200
400
Llama-2-7b
100
200
400
MM
MasLLLM
> 13
> 13
>-1 11
SpunsGPP
11.581
11,478
SpurseGPT
10.36
10.32
10.298
Wanda
13.854
13,828
13.814
Wanda
11,46
11:45
11.42
ProxSparse
10.39
10.09
9.91
ProxSparse
9.24
8.99
8.51
4.4.2. MEMORY FOOTPRINT IMPROVEMENTS
Next, we evaluate the memory footprint reductions achieved by
MaskLLM paper (Fang et al., 2024). As shown in Table 6, MaskLLM performed worst on Llama-2-7b with low sam- ple sizes, likely due to its reliance on extensive training for effective masks. SparseGPT and Wanda showed minimal improvement with increased calibration samples, consistent with previous observations (Fang et al., 2024; Sun et al., 2023). ProxSparse achieved the best across these sample sizes, with slight performance gains as samples increased within our target range. This confirms the effectiveness of our method in learning towards an optimal mask for semi- structured sparsity.
models. The results of peak mem- ory utilization during inference time (batch size = 1) for dif- ferent models are presented in Table 7. ProxSparse reduces peak memory usage by 29.5% 37.3%, demonstrating sig- nificant memory savings with 2:4 sparsification. The exact reduction varies across different model architectures due to differences in model weight sizes, which influence activa- tion sizes and ultimately affect peak memory consumption. Overall, ProxSparse effectively reduces memory footprint during LLM inference, highlighting the system benefits of the 2:4 sparse operation.
4.4. Improved efficiency during inference
5. Conclusion
Finally, we evaluate the efficiency metrics of the sparsified model produced by ProxSparse. We present wall-clock infer- ence speedup and memory footprint improvements for the 2:4 semi-structured sparsified model induced by ProxSparse. Our experiments
LLMs excel in natural language processing tasks and down- streaming tasks. However, they suffer from high compu- tational costs due to the enormous parameter sizes. Semi- structured sparsity can improve inference efficiency, though it remains challenging due to the structural constraints dur- ing pruning. We propose a learning- based method with reg- ularized optimization, progressively explores optimal mask through end-to-end gradient feedback. Extensive evaluation shows that ProxSparse significantly outperforms previous methods, enabling better accuracy for LLM pruning, making model deployment more cost-effective.
conducted on Nvidia A A10 GPUs. We utilize the Nvidia CUTLASS library as the underlying im- plementation for 2:4 semi-structured sparse operations.
4.4. 1. INFERENCE SPEEDUP
We follow the evaluation setup of previous work (Frantar & Alistarh, 2023; Sun al., 2023) and measure the latency

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
Impact statement
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers, arXiv preprint arXiv:2210.17323, 2022.
This paper presents work with the goal is to advance the field of machine learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Gao, L.,, Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, L.,, Hsu, J., Le Noac'h, h. A., Li,
References
H., McDonell, K., Muennighoff, N., Ociepa, C.,, Phang,
J.. Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I,,
L., Tang, E., Thite, A., Wang, B., Wang. K., and Zou, framework for few-shot language model evaluation, 07 2024. URL https: //ee:d//////ooo/m recor//ccrr///////o/ooooooo 12608602.
Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087774 2023.
Bai, G., Li, Y., Ling, C., Kim, K., and Zhao, L. Sparsellm: Towards global pruning for pre- pre-trained language mod- els, 2024. URL httph::t///// httppss//aaxvv..o/ www.aaiww...r.oraaaa.a abs/ abtp///bb//2///// 17946.
Geng, X. and Liu, H. Openllama: An open reproduction of llama, May 2023. URL https: /ggthhb.....// com/ hannumm-esearrcoopen_iet 11ama.
Han, S., Mao, H., and Dally, W. J. Deep compres- sion: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv:1510.001 2015.
Beck. A., Sabach, S.. and Teboulle, M. An alternating semiproximal method for nonconvex regularized struc- tured total least squares problems. SIAM J. Matrix Anal. Appl., 37:1 1291111911111 1150, 2016.
Huang, W., Jian, G., Hu, Y., Zhu, J., and Chen, J. Prun- ing large language models with semi-structural adaptive sparse training. arXiv preprint arXh::2407.2055 2024.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint arXiv:1308.3433 2013.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., al. Mistral 7b, arXiv preprint arXh::2310.0682. 2023.
Boža, V. Fast and effective weight update for pruned large language models. arXiv preprint arXiv:2401.02938 2024.
Kübler, M., Wang, Y.-X., Sabach, S., Ansari, N., Klein- dessner, M., Budhathoki. K., Cevher, V., and Karypis, G, A proximal operator for inducing 2: 4-sparsity. arXiv preprint arXiv:2501. 18015, 2025.
Chuang, Y.-N., Xing, T., Chang, C.-Y., Liu, Z., Chen, X., and Hu, X. Learning to compress prompt in natural language formats, 2024. URL htttttt///axxiii 24022187000
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang. W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
Cohen, E., Hallak, N., and Teboulle, M. A dynamic alternat- ing direction of multipliers for nonconvex minimization with nonlinear functional equality constraints. J. Optim. Theory Appl., 193:324-353, 2022.
Liu, S.-Y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.C. F., Cheng, K.-T,, and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.093355 2024.
Fang, G., Yin, H., Muralidharan, S., Heinrich, G., Pool, J., Kautz, J., Molchanov, P., and Wang, X. Maskllm: Learnable semi-structured sparsity for large language models. arXiv preprint arXiv:2409.17881, 2024.
Lu, H., Zhou, Y., Liu, S., Wang, Z., Mahoney, M. W., and Yang, Y. Alphapruning: Using heavy-tailed self regular- ization theory for improved layer-wise pruning of large language models. Advances in Neural Information Pro- cessing Systems, 37:9117-9152, 2024.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635. 2018
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned one-shot. In Inter- national Conference on Machine Learning, pp. 10323- 10337. PMLR, 2023.
Ma, X., Fang. G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. Advances in neural information processing systems, 36:21702-2111 2023.
10

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
Yang, A., Yang. B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115, 2024.
Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Ac- celerating sparse deep neural networks. arXiv preprint arXiv:2104.08375, 2021.
Yin, L., Wu, Y., Zhang, Z., Hsieh, C.-Y., Wang, Jia, Y., Li, G., Jaiswal, A., Pechenizkiy, M., Liang, Y., et al. Out- lier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity. arXiv preprint arXiv:2310.05// 10.05175, 2023.
Nesterov, Y. Gradient methods for minimizing composite functions. Mathematical programming. 140(1):125-161, 2013.
Yuan, Liu, H., Zhong, S., Chuang, Y.-N., Li, S., Wang, G., Le. D., Jin, H., Chaudhary, V., Xu, Z., al. Kv cache compression, but what must give in return? a compre- hensive benchmark of long context capable approaches. arXiv preprint arXiv:2407.01522 2024.
Ozkara, K., Yu, T., and Park, Y. Stochastic rounding for IIm training: Theory and practice. arXiv preprint arXiv:2502.20566. 2025.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Rockafellar, R. T. and Wets, J.-B. Variational Analysis. Springer Verlag, Heidelberg, Berlin, New York, 1998.
Sui, Y., Chuang, Y.-N., Wang, G., Zhang, J., Zhang, T., Yuan, J., Liu, H., Wen, A.. Zhong, S., Chen, H., et al. Stop overthinking: survey on efficient reasoning for large language models. arXiv preprint arX1v.2503.16119 2025.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1169, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Tseng, A., Yu, T., and Park, Y. Training llms with mxfp4. arXiv preprint arXiv:2502.20586. 2025.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zier, D., Emergent abilities of large language models. arXiv preprint arXiv:2206.0760 07682, 2022.
Wei, Q. Yau, C.-Y., Wai, H.-T., Zhao, K., Kang, D., Park, Y., and Hong, M. Roste: An efficient quantization- aware supervised fine-tuning approach for large language models. arXiv preprint arXiv:2502.00 09003, 2025.
Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint 00Xiv:2310.0666 2023.

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
A. Proofs of Technical Results
Proof of Proposition 1. subsets, making the whole regularizer 0. If at least three parameters are non-zero, then there least one group that non-zero. The second statement follows by symmetry. The third statement valid because it is cubic function when in the strict interior of an orthant.
For the first statement, check that if at least two parameters are 0, there is at least one 0 in all (t)
Proof of Proposition 2. First observe that this regularizer applies pointwise to each coordinate of W. It suffices to prove the statements for one coordinate w, wp. w.l.o.g. assume wo > 0, then the regularizer gives [e(we-w)] 2
Observe that the nullspace is either 0 = wo. thus checking Statement 1. Statement 2 follows because all coordinates with 0 contributes 0 to the total. Statement 3 follows because this is fourth order polynomial of w. thus continuously differentiable.
(1+e)wo
Proof of Proposition 6. We start with S = 4. Define f := 94 as a shorthand.
should be noted that for all 1 S i.j < 4, the partial functions fú (w:) = f (w), for fixed wy with j o i, are strongly convex and quadratic. Therefore, when i = 1 for instance, we have for any w, and v; that
1
fi (w;) = fa (vt) + fl (vi) (wi - v.) + 2 (w: - ..
Therefore, for any fixed wy with j + i, let w = arg min w, w,20 20 fi(w). wa satisfies the following (when i = 1 for instance)
fww) fi (w1) = fi (wi) + fi (wi) (wi - ww) + 1 5((r - wî)²
1
2 fi (wi) +
2 (w) - w2²²²
1
= f [ W2, Wiy ws) + 2 (w1-ww) wi)?
(8)
The inequality due to the first-order optimality condition. This means that we have sufficient descent property with respect to each minimized variable.
Proposition 8. Let {wr) KEN be a sequence generated by the Alternating Minimization algoritim. Then, for all l E N. we have that
f (w*) 2 f (w++)) + 1 -1 - 1 -
(9)
Proof. Let k E N. Using (8) for all sis < 4 yields
f (w**) 2 f (w++) wz, wy w/) + (w x+1 - w///
I (wt+), uh w wx) > (w (wt+1 uu k+l w wx) (wtt] w//
1
f (uf+), ,uitt wg wh) 2 f (w wn
k+1
b+1 w w+) + 2
(w:3 k+1 2g)2
f (wft), (ugtt, ++1 w wxt) 2 f (w + 2 Adding all the inequalities yields the desired result.
(10 k+1 - -
Telescope (9), we get that
K T
++1 | 2011 < f(u0) f(w
4A/12/1
min RE(T]
k+1 21 <
T
T
k=1
The last inequality follows as we initialize at w = z, thus f(w0) < 41|12113. Also, since with E RR,, f(w is neg-ntgative. This completes the proof for the first statement.
12

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
Now take 0, the position this algorithm halt, || w²1 || < c² + 0.
By Theorem 1 of (Beck et al., 2016), w at k oo is critical point of the objective function with the non-negative constraints handled by adding an indicator function.
The argument for the S = 3 case follows analogously (hence omitted).
Proof of Proposition 7. Under the assumption, the function Regw, has locally Lipschitz continuous gradient, which implies that the function f also has a locally Lipschitz continuous gradient. Therefore, the convergence of the sequence fre heN convergence to a critical point of the function e = f + follows immediately from (Cohen et al., 2022) (since w is a semi-algebraie function and { JeN is bounded).
B. Hy Hyperparameters and configurations
Table 8 presents the configurations and hyperparameters used in our experiments. There are three key hyperparameters for learning an optimal semi-structured mask: sparsity regularization strength (A1), frozen weight regularization extent A2), and learning rate. As discussed in Section 4.3.3, the frozen weight regularization is robust across a wide range of values. Our learning procedure follows standard settings, using Adam as the optimizer with warmup ratio of 0.1.
Table 8. Configure of the parammeter used in the experiment
Leuming rate
Optimizer
Warmup ratio
Mistral
Adamw
Mistral v0.37bb
5.0E-08
Adauw
Qwen
QU8001
Adaaw
JpenL.lam-77b-v
0.0001
Adamw
Llama 2:7b
0.25
0.0001
Adamw
Llama 2 12b
0.5
0.25
0.0001
Adamw
Liama 3.1 Ab
0.33
5.00 08
Adamw
0.1
C. Regularization trajectory of the optimization algorithm.
We illustrate the regularization path for an example initialization with different À value using different optimization algorithms (EnumIPM, EnumPGD, and EnumALM) in Figure as explained in Section 3.3.
1.2
#2 (pm)
1.0
k3 (pm)
(pgd)
0.8
#2 (pgd)
0.6
(pgd)
m1 (23m)
(slm)
(23m)
0.0
0.0
0.4
1.0
Figure 3. Illustration of the solution (6) with an example input y 1.,, increases. Observe that (1) the regularizer shrinks different coordinates differently according to their relative magnitude (2) all three algorithms return the same solution path. (3) the dashed lines indicate the two thresholds of from KKT conditions above which the 3-sparse and 2-sparse solutions become critical points (a necessary condition for them to become global optimal).
13

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
Table Experimental results Wikitext perplexity (PPI.) and performance across 7 commonly used zero-shot natural language reasoning tasks comparing ProxSparse to 3 other baselines Llama-3.1-4 -.b-bb. Bold indicates the best pruning performance, while italic represents the original unpruned performance. SparseGPT updates weights to minimize reconstruction error, while other methods keep retained
weights frozen. Similar to the results in Table 2, ProxSparse consistently yields better results compared to all other baselines.
Weight Update
Wikitext PPL
ARC-C
ARC-E
SIQA
HellaSwag
OBQA
PIQA
TruthfulQA
AVG
Llama-3. ---
5.84
0,515
0,814
0.470
0.600
0.334
0,801
0.368
0.557
magnitude
X 766.91
0.257
0.454
0.365
0.335
0.154
0.634
0.319
0.360
SparseGPT
14.61
0,316
0.647
0.426
0.435
0.222
0,705
0.301
0.434
Wanda
X 20.91
0.269
0.573
0,400
0.380
0.192
0.686
0.309
0.401
ProxSparse
X 13.63
0.333
0.623
0,422
0.460
0.240
0.721
0.296
0.444
Table 10. Experimental results on Wikitext perplexity (PPL) and commonly used zero-shot natural language reasoning tasks comparing ProxSparse to 3 other baselines Llama-2-7b and Mistral-v0.3-76 model. Bold indicates the best pruning performance. while itaiic represents the original unpruned performance. AdmmPrune updates weights minimize reconstruction error, while OWT. and AlphaPrune uses dynamic sparse ratios across layers. ProxSparse consistently yields better results compared all other baselines.
Weight Update
Wikitext PPL
ARC-C
ARC-E
SIQA
HellaSwag
OBQA
PIQA
TruthfulQA
AVG
Mistral-v0.3-7b
4.95
0.490
0.747
0.458
0.609
0.336
0.803
0.353
01544
OWL
X
13.03
0.275
0.594
0.406
0.417
0.188
0.688
0.320
0,413
AlphaPrune
X
13.58
0.265
0.529
0.398
0.407
0.190
0.668
0.335
0.399
ADMMPrune
V
9.06
0.340
0.680
0,416
0.471
0.240
0.739
0.299
0,455
ProxSparse
X
8.68
0.362
0.697
0.429
0.525
0.242
0.751
0.321
0.475
Llama-2-7b
5.12
0.433
0,763
0,461
0.571
0.314
0.781
0.321
0.521
OWL
13.17
0,287
0,591
0.407
0.420
0.228
0.695
0.339
0.425
AlphaPrune
X
13.01
0.293
0.607
0.406
0.411
0.238
0.690
0,317
0.424
ADMMPrune
V
9.67
0.328
0.653
0.413
0.440
0.248
0.714
0.302
0.442
ProxSparse
X
8.51
0.331
0.656
0.407
0.478
0.242
0.716
0.328
0.452
D. End-to-end evaluation results on Llama-3.1-8b
In this section, further discuss the evaluation results from Table 2, focusing on Llama-3.1-8h We present results on Wikitext perplexity (PPL) and performance across seven commonly used zero-shot natural language reasoning tasks, comparing ProxSparse to three baselines in Table 9. In the Alama-3.1-88 experiments, ProxSparse significantly reduces perplexity from Wanda's 20.91 to 13.63. The overall results, compared to Magnitude Pruning, Wanda, and SparseGPT, follow the same trends discussed in Section 4.2, with ProxSparse consistently outperforming all other baselines.
E. Comparison with Additional Pruning Baselines (ADMMPrune, OWL, and AlphaPrune)
In this section, compare ProxSparse with three additional pruning baselines: ADMMPrune (Boža, 2024), OWL (Yin et al., 2023), and AlphaPrune (Lu et al., 2024). The experiments are done on Mistral-v0.3-7b and Llama-2-7b model. ADMMPrune introduces fast and effective weight update algorithm for layerwise pruning based using the Alternating Direction Method of Multipliers (ADMM). As shown table 10, ProxSparse outperforms ADMMPrune in both models, achieving lower PPL (8.51 vs. 9.67) and higher ace (47.6% vs. 45.5%), highlighting its effectiveness. We attribute the superority of ProxSparse to its end-to-end optimization process, which goes beyond solely relying on local layer-wised information.
OWL and AlphaPrune aim to determine layer-specfic ratio to protect important layers. Here argue that they are not the best-suited mechanism in semi-structured pruning, as the sparse operator supported by hardware typically requires all blocks to strictly adhere the pattern, making applying varying ratios hard. Nevertheless, we conduct experiments on AlphaPrune and OWL for comparison. We follow mixed sparsity proposed in OWL and AlphaPrune with Wanda, that layers can have varying ratios, while the overall ratio remains 2:4. We see ProxSparse outperforms OWL and Alphaprune on Llama-2-70 and -0s3 on PPL and accuracy, showing the strength of our end-to-end optimization. Further, as pruning patterns
14

---

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
6d0 (raffceri)
10%
22 Beanced
Norea rnd Exch Prppess
tamilbed tpack Prgees
Mereaired facth Prpeess b. Mask similarity evalvemmnt
sparse evalvement
ratio evolvement
c. Initial/final two-four sparse ratio vs. A
Figure 4. Evolution of sparsity ratio OpenLlama-7b-4e based on the degree of regularization. (a) Evolution of the 2:4 sparsity ratio over learning, where an insufficient regularization degree leads to lcarnnn. (b) With a larger A parameters shrink more quickly towards 2:4 sparsity, resulting in early commitment suboptimal mask. (c) Comparison of the 2:4 sparse block ratios early (0.1 epochs) and final stages of learning. (d) Mask similarity between the final mask and the early mask obtained after 10% of learning. An excessively large Å1 results in premature mask commitment, causing mask selection to stagnate and hindering optimal mask discovery.
become more fine-grained (e.g., 2:4), varying layer-wise pruning ratios become less effective as critical weights might still be removed within each block. This was reported in (Yin et al., 2023; Lu et al., 2024), where 4:8 pruning performed just similarly uniform pruning in Wanda. This highlights the benefits of ProxSparse in identifying fine-grained semi-structured masks.
F. Evolution of 2:4 sparsity across A on OpenLlama-77--22
In this section, we expand on the discussion from Section 4.3.2 and present the evolution trajectory of 2:4 sparsity across different λ values on OpenL Our findings on OpenL.lam--7b-v2 are consistent with the main paper's discussion: a balanced regularization strength enables flexible mask exploration, preventing premature commitment while also facilitating more effective learning compared to excessively low values.
G. Practical scenario of 2:4 sparsity and its extensibility discussion
To the best of our knowledge, commercially available hardware such as Nvidia Ampere GPUs, only supports the 2:4 sparsity pattern.. Our method directly aligns with the hardware features, making directly applicable to real-world use cases. Mcanwhile, our regularizer is flexible; extending to a 1:4 sparsity pattern is straightforward, as the regularizer can be reformulated and solved with even greater efficiency. On the otherhand, semi-structured patterns like 4:8 increase regularization terms, which could slow the solving process, Despite the longer but tolerable search time, inference gain remains unaffected once the optimal mask is found. A more efficient solver could further improve handling of such complex patterns, and we leave this for future exploration.
In the meantime, we note that increasing sparsity complexity (e.g.. 2:4 to 4:8) will expand the search space, which is a common challenge for learning- based methods, including MaskLLM (Fang et al., 2024). Nevertheless, our regularizer supports extensibility and shows practical benefits in wollworld scenarios.
'NVIDIA AMPERE GA GPU ARCHITECTURE Whitepaper
15


Reviews:
[{"review_id": "J6Wh7ooQfb", "reviewer": "Reviewer_hyi6", "summary": "This work introduces ProxSparse, a learning-based framework for mask selection via regularized optimization. The key design is a sparsity regularization $Reg_{2:4}$ that forces 2:4 sparsity and a weight regularization $Reg_{W_0}$  to avoid significant differences between the tuned parameters and the original parameters. The authors validate their approach through experiments on seven LLMs, demonstrating significant performance improvements over state-of-the-art pruning baselines such as SparseGPT and Wanda.\n\n## Update After Rebuttal\n\nThanks for all the clarifications. I will keep my initial & positive score for this submission.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the effectiveness and practicality of ProxSparse! Below, we address the questions raised regarding **Llama, ADMMPrune comparisons, clarification on weight updates and MaskLLM comparison**.\n\n# W1: `Lack of Llama results`: We anonymized Anon.model-1,2,3 due to IP constraints, and we believe the 7 models presented have broad coverage and ProxSparse exhibits consistent trends.\n\nWe didn't reveal the anonymous model family name due to internal policy constraints. However, we note that these anonymized models are among the most competitive LLMs, as evidenced with good PPL and accuracy in our benchmarks. We hope those models serve as additional data points to further support the effectiveness of our method. We appriciate the reviewer for the understanding! Meanwhile, we believe our experiments offer broad coverage on top-performing models, including Mistral, OpenLlama and Qwen family, plus the anonymous model family. The consistent results highlight the superiority of our method, which is robust and widely applicable to the top-tier LLMs.\n\n# W2: `Comparison with ADMMPrune`: we achieve better results!\n\nWe appreiciate the reviewer's notes on newer baseline named ADMMPrune[1]! We benchmarked ADMMPrune to compare with ProxSparse on Anon.Model-1 and Mistral model, using same evaluation settings discussed in paper. As shown, ProxSparse outperforms ADMMPrune in both models, achieving lower PPL (8.51 vs. 9.67) and higher acc (47.6% vs. 45.5%), highlighting its effectiveness. We attribute the superority of ProxSparse to its end-to-end optimization process, which goes beyond solely relying on local layer-wised information. We are happy to discuss ADMMPrune in our revised paper!\n\nIn the meantime, we include more baselines (OWL and AlphaPrune) discussion as proposed by reviewer 2EPP. ProxSparse consistently achieves better performance on them. Please kindly refer [here](https://openreview.net/forum?id=zkxe5vASi8&noteId=dogVmgWiVP) for more details!\n\n| Anon.Model-1      | Weight Update | Wikitext PPL | ARC-C | ARC-E | SIQA  | HellaSwag | OBQA | PIQA | TruthfulQA | Average |\n|-------------|----------------|---------------|--------|--------|--------|------------|------|------|-------------|---------|\n| ADMMPrune   | Yes            | 9.67          | 0.328  | 0.653  | **0.413** | 0.440      | **0.248**| 0.714| 0.302       | 0.442   |\n| ProxSparse  | **No**             | **8.51**          | **0.331**  | **0.656**  | 0.407 | **0.478**      | 0.242| **0.716**| **0.328**       | **0.452**   |\n| Mistral-v0.3-7b      | Weight Update | Wikitext PPL | ARC-C | ARC-E | SIQA  | HellaSwag | OBQA | PIQA | TruthfulQA | Average |\n| ADMMPrune   | Yes            | 9.06          | 0.340  | 0.680  | 0.416 | 0.471      | 0.240| 0.739| 0.299       | 0.455   |\n| ProxSparse  | **No**             | **8.68**          | **0.362**  | **0.697**  | **0.429** | **0.525**      | **0.242**| **0.751**| **0.321**       | **0.476**   |\n\nTable 1: Comparison between ADMMPrune and ProxSparse on Mistral-v0.3-7b and Anon.Model-1. ProxSparse consistently achieves better performance \n\n# W3: `Clarification on weight update`: our method does NOT update the unpruned LLM parameters.\n\nProxSparse is an learned method that identifies the optimal mask without further weight updates on retained weights. In other words, the retained weight after applying ProxSparse-selected mask remains identical to its initialization, similar to Wanda and magnitude prune. In ProxSparse, the end-to-end optimization with calibration data is sololy used to determine the mask. Even when compared to SparseGPT and ADMMPrune—which update weights after pruning—ProxSparse consistently achieves higher performance, underscoring its effectiveness in identifying high-quality pruning masks.\n\n# W4: `Comparison with MaskLLM on effectiveness`: ProxSparse consumes 25x less tokens compared to MaskLLM.\n\nWe are happy to provide more intutive and direct comparison between MaskLLM and ProxSparse. As discussed in our paper, MaskLLM employs a fundamentally different design with Gumbel Softmax sampling for learning the mask. We view MaskLLM as a complementary to ours as it focuses on large sample regimes, whereas ProxSparse operates effectively with much smaller sample size. This makes ProxSparse more practical and accessible in LLM era.\n\nHere we present direct comparison betwen MaskLLM and ProxSparse on Anon.Model-2. In terms of consumed tokens, ProxSparse achieves a PPL of 8.51 with 1638400 tokens, outperforming Wanda (11.42), SparseGPT (10.298) and MaskLLM (11). For MaskLLM to achieve a comparable PPL, it consumes 40960000 tokens, 25x larger than ProxSparse. This, in general, demonstrates the superiority of our pruning method with small-scale calibration data like ADMMPrune, Wanda and SparseGPT.\n\n# Thanks again!\n[1]ADMMPrune:Fast and optimal weight update for pruned large language models", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "The paper primarily focuses on a learnable approach for achieving N:M sparsity through regularization. The proposed technique is conceptually related to existing methods such as Lasso, SparseGPT, and Wanda, sharing similarities in its sparsification strategy.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n1. Unlike prior methods that rely on local heuristic-based mask selection, ProxSparse employs an end-to-end differentiable optimization that considers global feedback, leading to more effective and stable pruning results.\n2. The method achieves effective mask selection with only ~100 calibration samples, this makes the proposed method very practical.\n3. Across seven LLMs, ProxSparse consistently outperforms existing pruning baselines like SparseGPT and Wanda in both PPL and zero-shot accuracy. For example, on Mistral-v0.1-7b, ProxSparse improves PPL from 9.43 to 8.92. \n\n## Weaknesses\n1. The experimental section is not entirely convincing. First, it is unusual that the paper does not report results on LLaMA-1/2/3, which are crucial base models in benchmarks such as Wanda. Additionally, the selected SOTA baselines appear somewhat outdated, as several recent methods, such as [1], have demonstrated comparable or superior performance. For instance, [1] also achieves a ~1.00 improvement in PPL. It would be beneficial if the author could include a fair comparison to different methods.\n2. If my understanding is correct, the proposed method updates the parameters of the LLMs, implicitly influenced by regularization and end-to-end optimization. To ensure a fair comparison, it would be beneficial to include additional experiments where the remaining weights in SparseGPT, Magnitude, and Wanda models are fine-tuned on for example 400 samples. \n3. Although the paper claims that MaskLLM is resource-intensive, it remains unclear how significant the gap is between the proposed method, ProxySparse, and MaskLLM on LLaMA-2. Providing a direct comparison on the PPL, consumed tokens would help clarify the relative efficiency and effectiveness of ProxySparse.\n\n[1] Fast and optimal weight update for pruned large language models.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ecio0uX1kF", "reviewer": "Reviewer_8W4h", "summary": "The authors propose ProxSparse, a method for learn a semi-structured pruning mask using two regularisors, one is analogues to l1 regularisation and the other promotes a locality constraint for semi-structured pruning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments! Below, we address several questions raised including **sparsity pattern selection, Hessian-based pruning, anonymized citations, and the anonymized model family**.\n\n# W1: `The focus on 2:4 pruning sparsity`: 2:4 pruning is the most practical semi-structured sparsity pattern, and the only pattern currently supported by commercial hardware\n\nWe thank the reviewer for the consideration! In [Appendix G](https://https://openreview.net/pdf?id=zkxe5vASi8#page=13) in our paper, we have more discussion on the practical scenario of the 2:4 sparsity pattern and the extensibility of ProxSparse. We focus on the 2:4 sparsity pattern in this paper because it is the most practical semi-structured format and the only one currently supported by commercial hardware. To the best of our knowledge, existing hardware such as NVIDIA Ampere GPUs only support 2:4 sparsity [1]. ProxSparse aligns directly with this hardware feature, making it readily applicable to real-world use cases.\n\n# W2: `A misunderstanding on Why Hessian based method is local-wise`: Our claim is localized pruning (i.e. w/ Hessian) hinder the pruning. Meanwhile, the heavy hessian computation makes it hard for global optimization. \n\nWe thank the reviewer for raising the great point! We would like to first clarify on our claim, that we argue previous layer-wised pruning utilizes hessian metrics [2] or per-output based importance score [3] fail to select mask well because of the localized information constraints. Our method enables an end-to-end pruning mechanism that makes the pruning having well-informed solution. \n\nIn the meantime, while it is true as mentioned in [4], that Hessian can be evaluated through the global loss. We note that this is impractical in pruning LLM. Even in [4], which focuses on smaller CNN model, the authors report using Hessian incurs 30x inefficiency, leading to huge overhead. For an LLM with enormous parameter size (~B), it is even harder to calculate the Hessian, let alone to compute it round by round in end-to-end optimization. As further evidence, SparseGPT specifically highlights the computational burden of Hessian estimation. To overcome this, it proposes a Fast Approximate Reconstruction method to simulate Hessian more efficiently. Yet it is still limited in layer-wised pruning. In contrast, our proposed end-to-end optimization scheme delievers better performance compare against those layer-wised pruning methods.\n\n# W3: `Anonymous citation`: a private communication and we will update it upon acceptance\nThe anonymous citation is a private communication currently under a double-blind submission policy. We have provided detailed discussions of it in Section 3 and confirm that we will update the it upon acceptance.\n\n# W4: `Anonymous model family`: We anonymized some models (Anon.model-1,2,3) due to IP constraints, and we believe the 7 models presented have broad coverage and ProxSparse exhibits consistent trends.\nWe appriciate the reviewer for the understanding! We didn't reveal the anonymous model family name due to internal policy constraints. However, we note that these anonymized models are among the most competitive LLMs, as evidenced with good PPL and accuracy in our benchmarks. We hope those models serve as additional data points to further support the effectiveness of our method. Meanwhile, we believe our experiments offer broad coverage on top-performing models, including Mistral, OpenLlama and Qwen family, plus the anonymous model family. The consistent results highlight the superiority of our method, which is robust and widely applicable to the top-tier LLMs.\n\n# Thanks again!\n[1] [NVIDIA AMPERE GA102 GPU ARCHITECTURE](https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-whitepaper-v2.pdf#page=27)\n[2]SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot\n[3]A SIMPLE AND EFFECTIVE PRUNING APPROACH FOR LARGE LANGUAGE MODELS\n[4]Pruning Convolutional Neural Networks for Resource Efficient Inference. ICLR 2017", "claims_and_evidence": "From my understanding, the main claim is that previous methods, which rely on computing the Hessian, do not take into account the information between layers, whereas Proxy, which uses a global heuristic, does.", "methods_and_evaluation_criteria": "The main experiments are on zero-shot evaluations of the pruned models, which are trained using a small calibration dataset.", "theoretical_claims": "One of the main claims is that using a soft regularisation for structured pruning is effective for exploring a wider search space. Additionally, the local heuristics are ill-suited for pruning LLMs.", "experimental_designs_or_analyses": "Experimental design seems valid and follows previous works.", "relation_to_broader_scientific_literature": "This has broad applications in improving the efficiency of deployed LLMs across many areas.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "Why do the authors not consider other pruning ratios? i.e. not just 2:4 sparse.\n\nOne of the main claims is that a hessian based heuristic is local/layer-wise. This is not clear to me. When using hessian/sensitivty based pruning, the sensitivity of each weight takes into account the downstream loss [1]\n\nSmall concerns:\nTheorem 4. Why do the authors anonymise this citation? this is not done for any of the other citations and after following the reference, it is clear that this may be the authors of this submission..\n\nSimilarly, why are the authors using an anonymous model family for some experiments? is this just for reviewing purposes? it is very odd to me.\n\n[1] Pruning Convolutional Neural Networks for Resource Efficient Inference. ICLR 2017", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJsyEYN06M", "reviewer": "Reviewer_2Epp", "summary": "This paper introduces a learning-based approach for semi-structured pruning of LLMs using a structured sparsity regularizer and proximal gradient descent. It enables global mask optimization without retraining and improves efficiency. Experiments on seven models show superior perplexity and zero-shot accuracy over existing pruning methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback! Below we share responses on **comparison w/ layer-wise method and MaskLLM, inference efficiency.**\n\n# W1:`Comparing newer layer-wise method`: We achieve better results than OWL and AlphaPrune\nOWL[1] and AlphaPrune[2] are important works in pruning, aiming to determine layer-specfic ratio to protect important layers. We are happy to discuss them in our paper!\n\nIn the meantime, we would like to respectfully argue that they are not very well-suited in semi-structured pruning, as the sparse operator supported by hardware typically requires all blocks to strictly adhere the pattern, making applying varying ratios hard.\n\nNevertheless, we conduct more experiments on AlphaPrune and OWL for comparison. We follow mixed sparsity proposed in OWL and AlphaPrune with Wanda, that layers can have varying ratios, while the overall ratio remains 2:4. We see ProxSparse outperforms OWL and Alphaprune on Anon.Model-1 and Mistral on PPL and acc, showing the strength of our end-to-end optimization. Further, as pruning patterns become more fine-grained (e.g., 2:4), varying layer-wise pruning ratios become less effective as critical weights might still be removed within each block. This was reported in both paper, where 4:8 pruning performed just similarly to uniform pruning in Wanda. This highlights the benefits of ProxSparse in identifying fine-grained semi-structured masks.\n\nWe have more baseline (ADMMPrune) discussion as proposed by reviewer hyi6. ProxSparse achieves better results compared to ADMMPrune. Please kindly refer [here](https://openreview.net/forum?id=zkxe5vASi8&noteId=rlLpTKNcTH) for more details!\n\n| Anon.Model-1      | Weight Update | Wikitext PPL | ARC-C  | ARC-E  | SIQA   | HellaSwag | OBQA  | PIQA  | TruthfulQA | Average |\n|-------------|----------------|---------------|---------|---------|--------|------------|--------|--------|-------------|---------|\n| OWL         | No             | 13.17         | 0.287  | 0.591  | 0.407 | 0.420     | 0.228  | 0.695 | 0.339      | 0.425   |\n| AlphaPrune  | No             | 13.01         | 0.293  | 0.607  | 0.406 | 0.411     | 0.238  | 0.69e | 0.317      | 0.424   |\n| **ProxSparse**  | No             | **8.51**          | **0.331**   | **0.656**   | **0.407**  | **0.478**      | **0.242**  | **0.716**  | **0.328**       | **0.452**   |\n|Mistral-7b-v0.3|Weight Update | Wikitext PPL | ARC-C  | ARC-E  | SIQA   | HellaSwag | OBQA  | PIQA  | TruthfulQA | Average |\n| OWL         | No             | 13.03         | 0.275   | 0.594   | 0.406 | 0.417      | 0.188  | 0.688  | 0.320       | 0.413   |\n| AlphaPrune  | No             | 13.58         | 0.265   | 0.529   | 0.398 | 0.407      | 0.190  | 0.668  | 0.335       | 0.399   |\n| **ProxSparse**  | No             | **8.68**          | **0.362**   | **0.697**   | **0.429** | **0.525**      | **0.242**  | **0.751**  | **0.321**       | **0.476**   |\n\nTable 2: Comparison with OWL, AlphaPrune and ProxSparse on Mistral-v0.3-7b and Anon.Model-1. ProxSparse achieves the best.\n\n# W2:`Comparison w/ MaskLLM`: Complementary method with fundamentally different design, we excel in low-scale regime.\nThanks for the question! We note that ProxSparse and MaskLLM differs fundamentally in approaching semi-structured mask, and we view MaskLLM to be a complementary as it focuses on larger data regime\n\n- Mechanism difference: MaskLLM and ProxSparse take fundamentally different ways in pruning. We both tackle the non-differentiable task of selecting N out of M per block. MaskLLM sidesteps this with a probabilistic sampling approach, learning to sample the correct weights. In contrast, ProxSparse relaxes the hard constraint into a smooth optimization and performs optimization via proximal gradient descent, with theoretically provable properties for the proposed 2:4 regularizer.\n\n- Larger batch exps:\n| Method      | 1024 | 2048 |\n|-------------|------|------|\n| MaskLLM     | 10| 9.5 |\n| SparseGPT   | 10.18| 10.16|\n| Wanda       | 11.38| 11.4|\n| **ProxSparse**  | **8.38** | **8.23** |\n\nTable 3: PPL on Anon.Model-1 on extended sample size. ProxSparse achieves the best\n\nTable 3 evaluates ProxSparse with larger sample size. In this small-scale data regime, ProxSparse outperforms all baselines, demonstrating its superiority. We note that our targeted low-scale calibration regime is practical in LLM contexts, making our method more accessible in the real world. \n\n# W3:`Real world efficiency`: ProxSparse leads to 1.35x inference speedup and 37.3% reduced peak memory usage\nThanks for the comments! As discussed in [Appendix H](https://openreview.net/pdf?id=zkxe5vASi8#page=14), our analysis demonstrates that, beyond reduced FLOPs, ProxSparse achieves 1.35× inference speedup and 37.3% reduction in peak memory usage. These results highlight the practical efficiency gains enabled with semi-structured sparsity induced by ProxSparse.\n\n# Thanks again!\n[1]Yin et al. OWL\n[2]Lu et al. AlphaPruning", "claims_and_evidence": "Cons:\n1. the claim that the method achieves the SoTA performance is insufficient. It should be compared with learning-based methods like MaskedLLM [1], and layer-wise methods like OWL [2] and AlphaPruning [3].  \n2. The comparison between MaskLLM and ProxSparse has been made in Table 6, but it is only done with a limited sample size. It would be helpful tp provide evaluation on a larger sample size.\n\n\n\n\nReference:\n[1] Fang et al. Maskllm: Learnable semi-structured sparsity for large language models\n[2] Yin et al.  Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning llms to high sparsity\n[3] Lu et al.  AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models", "methods_and_evaluation_criteria": "It is necessary to compare how the method is different from MaskedLLM [1] since both use the learnable soft mask with thresholding to gain the binary mask. \n\nThe evaluation criteria are standard. \n\n\nReference:\n[1] Fang et al. Maskllm: Learnable semi-structured sparsity for large language models", "theoretical_claims": "I have checked the convergence proof and didn't catch any issues.", "experimental_designs_or_analyses": "It would be better to provide more inference cost measures like FLOPs or latency.", "relation_to_broader_scientific_literature": "It is related to LLM efficiency.", "essential_references_not_discussed": "Has been mentioned in the previous sections", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FnNLkbmHIH", "reviewer": "Reviewer_kksk", "summary": "The paper introduces ProxSparse, a learning-based framework designed to improve the efficiency of large language models (LLMs) through semi-structured pruning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for acknowledging the strength of our paper! Below we address the questions regarding **2:4 ratio, semi-structured benefits, assumption of the theoretical proof, ALM and EnumALM as well as model size justification.**\n\n# W1:`The focus on 2:4 pruning sparsity`: 2:4 pruning is the most practical semi-structured sparsity pattern and the only one currently supported by commercial hardware\n\nWe thank the reviewer for the consideration! In [Appendix G](https://https://openreview.net/pdf?id=zkxe5vASi8#page=13), we have more discussion on the practical relevance of the 2:4 sparsity pattern and the extensibility of ProxSparse. We focus on the 2:4 sparsity pattern in this paper because it is the most practical semi-structured format and the only one currently supported by commercial hardware. To the best of our knowledge, existing hardware such as NVIDIA Ampere GPUs only support 2:4 sparsity [1]. ProxSparse aligns directly with this hardware feature, making it readily applicable to real-world use cases.\n\n# W2:`Why semi-structured pruning considering its lack of flexibility`: Semi-structured pruning strikes a balance between efficiency and accuracy, and also benefiting from real hardware support.\n\nThe reviewer is correct that compared to unstructured pruning, semi-structured method cast more constraints, which is less flexible. However, unstructured pruning often does not directly translate to faster inference because it induces irregular access, and modern hardware exploits regularities in computation for faster speed. On the other hand, sturctured pruning typically offers the highest efficiency but suffers from huge accuracy loss due to its rigid constraints.\n\nSemi-structured pruning is an important problem to study [2][3] in pruning, as it strikes a balance between efficiency and accuracy. A key benefit is its direct support on commercial hardware like NVIDIA Ampere GPUs and high-performance libraries for sparse operator, enabling real-world speedups. In this paper, we focus on semi-structured pruning and propose a relaxed end-to-end mask selection approach to identify optimal pruning masks for LLMs.\n\n# W3: `relationship between ALM ans EnumALM`: ALM is the subrouting of the EnumALM.\nThanks for the question! ALM is a subroutine used within EnumALM. Specifically, EnumALM solves the proximal operator for the 2:4 regularizer by enumerating and evaluating three candidate sparsity patterns: a 2-sparse solution (selected directly by top-k), a 3-sparse solution, and a dense (4-sparse) solution. For the latter two cases (3-sparse and 4-sparse), EnumALM invokes ALM to efficiently solve the corresponding convex subproblems with convergence guarantees. \n\n# W4: `assumption of the proof`: in general cases, our assumption holds because the \"ReLU is weakly differentiable\" and \"weights are bounded\", as explained below.\nWe thank the reviewer for the discussion of the assumptions! We acknowledge that the convergence analysis assumes the loss function is continuously differentiable and the weights remain bounded during optimization.\n\nWhile the use of ReLU in the loss may technically violate differentiability at a single point (zero), this is a well-known and standard issue in deep learning. In practice, ReLU is differentiable almost everywhere, which is typically sufficient for convergence analyses in nonconvex optimization. Moreover, the population loss, as an expectation over a smooth data distribution, can remain continuously differentiable even when ReLU is used. \n\nThe other assumption—that the weights remain bounded—is rather mild and commonly used in convergence analyses. It is satisfied as long as the optimization does not diverge, which can often be ensured by using a sufficiently small learning rate. In our case, we observe stable behavior throughout, supporting the validity of this assumption. Meanwhile, ProxSparse's consistent performance across a variety of LLMs and tasks further supports the practical utility of our method.\n\n# W4: `Larger model size experiment`: we apologize for the current resource infeasibility on running experiments on larger (>30B) model. Our experiments show consistent good performance across different size of model.\nWe apologize for the lack of models larger than 30B due to limitations in our current resources. Notably, prior work on learning based LLM pruning [3] has also only conducted experiments on models up to ~15B in size. Nevertheless, our paper includes results on 14B model to demonstrate the effectiveness of our method. Across various model sizes, our approach consistently outperforms other baselines, highlighting its robustness and applicability.\n\n[1] [NVIDIA AMPERE GA102 GPU ARCHITECTURE](https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-whitepaper-v2.pdf#page=27)\n[2]Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch\n[3]MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models", "claims_and_evidence": "Yes.\nIt presents detailed experiments comparing ProxSparse with state‐of‐the‐art baselines across multiple LLM families and tasks. \nAdditionally, the authors offer convergence proofs and theoretical guarantees for the proximal gradient descent and the EnumALM solver.", "methods_and_evaluation_criteria": "The methods and evaluation criteria in the paper are well-suited.", "theoretical_claims": "Yes. Overall, the proofs are mathematically rigorous within their theoretical framework. However, the assumptions that are relied on for proof may need more considerations.", "experimental_designs_or_analyses": "The experimental design appears to be robust and well thought out.", "relation_to_broader_scientific_literature": "The paper's contributions are closely tied to existing work in model compression and network pruning.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Advantages:\n1.  The paper introduces a unique regularization framework that transforms the rigid mask selection problem into a differentiable one, enabling end-to-end learning.\n2. The development of the EnumALM solver and the efficient proximal gradient descent approach significantly improves the speed and scalability of finding the optimal mask.\n\nDisadvantages:\n1.  The work is specifically focused on 2:4 sparsity. Why are these hyperparmeters beneficial?\n2. The convergence proofs rely on assumptions which may be not possible, and what will that lead to?", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. M is 2:4 sparse. Why do we choose 2:4? Have the authors tested other hyperparameters like 3:6 or 4:8? Is there any explaination?\n2. What is the reason the semi structuring is better than other method? In my opinion, the semi structured method is less flexible?\n3. What is the relationship between ALM ans EnumALM?\n4. More tests should be conducted on larger model size.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposed ProxSparse to learn masks for N:M semi-structured pruning end-to-end. The non-differentiable binary mask selection process was reformulated and the constraints were relaxed into regularization terms in the end-to-end loss function. This could be optimized with proximal gradient descent and an efficient solver for the proximal operator was proposed and applied. This work was backend by strong theoretical analysis and proof of convergence, which has been proofread by a few reviewers. While ProxSparse only updates the pruning masks, the experiments also show that ProxSparse performs almost always better than the existing local or global pruning methods, some of which update model weights, including MaskLLM, SparseGPT, and Wanda, tested on a variety of well known open source LLMs and fine-tuning tasks.

Given both the theoretical and empirical contributions, I recommend accepting the paper.
Decision: Accept (poster)