A Proximal Operator for Inducing 2:4-Sparsity
Authors: Jonas M. Kübler
Yu-Xiang Wang
Shoham Sabach
Navid Ansari
Matthäus Kleindessner
Kailash Budhathoki
Volkan Cevher
George Karypis
Venue: Accepted by TMLR ()

Abstract: Recent hardware advancements in AI Accelerators and GPUs allow to efficiently compute sparse matrix multiplications, especially when 2 out of 4 consecutive weights are set to zero. However, this so-called 2:4 sparsity usually comes at a decreased accuracy of the model. We derive a regularizer that exploits the local correlation of features to find better sparsity masks in trained models. We minimize the regularizer jointly with a local squared loss by deriving the proximal operator for which we show that it has an efficient solution in the 2:4-sparse case. After optimizing the mask, we introduce masked-gradient updates to further minimize the local squared loss. We illustrate our method on toy problems and apply it to pruning entire large language models up to 70B parameters. On models up to 13B we improve over previous state of the art algorithms, whilst on 70B models we match their performance.

Published in Transactions on Machine Learning Research (08/2025)
A Proximal Operator for Inducing 2:4-Sparsity
Jonas M. Kübler Amazon
kucbj@amazon.coom com
Yu-Xiang Wang
Amazon and University of California Son Diego, USA
Shoham Sabach
Amazon and Technion Ismel Institute of Technology, Haifa, Israel
Navid Ansari
Huawei (work done during an internship at Amazon)
Matthäus Kleindessner
Amason
Kailash Budhathoki
Amazon
Volkan Cevher
Amazon and École Polytechnique Fédéral de Lausanne, Lousanne, Switzerland
George Karypis
Amazon
Reviewed on OpenReview:
https: // openreview. net/ form? Ad=
Abstract
Recent hardware advancements in AI Accelerators and GPUs allow to efficiently compute sparse matrix multiplications, especially when out of consecutive weights are set to zero. However, this so-called 2:4 sparsity usually comes at a decreased accuracy of the model. We derive regularizer that exploits the local correlation of features find better sparsity masks in trained models. We minimize the regularizer jointly with local squared loss by deriving the proximal operator for which we show that it has an efficient solution in the 2:4-sparse case. After optimizing the mask, we introduce masked-gradient updates to further minimize the local squared loss. We illustrate our method on toy problems and apply it to pruning entire large language models up to 70B parameters. On models up to 13B we improve over previous state of the algorithms, whilst on 70B models we match their performance.
1
Introduction
The extensive adoption of large language models has sparked renewed interest in post-training model compression to reduce inference cost and latency (Chitty-Venkata et al.) 2023 Park et al. 2024). The most notable techniques
quantization of model weights and activations (Frantar et al. 2023 Dettmers et al.
2022 Frantar et al. 2024) as well as model pruning, i.c., the removal of weights or structures (Frantar & Alistarh 2023 Sun el al. 2024 Ashkboos et al. 2024). However, pruning entire network structures like columns and rows of weight matrices leads to nonnegligible accuracy drops. Hence, the surge for more flexible sparsity patterns that allow to prune individual weights has sparkod. This accompanied by hardware and software support for efficient sparse matrix multiplications (Pool al. 2021). In this work we consider structured sparsity, that is attempt to prune weights of linear models (or linear layers in neural network)

---

Published in Transactions on Machine Learning Research (08/2025)
into structure where out of each consecutive M N weights N E N weights are set zero. Modern GPUs and AT Accelerators can efficiently represent structured sparsity patterns to compress the memory footprint of the model and accelerate the matrix multiplications. 2 4-sparsity is the most promising candidate for realizing actual speedups and thus the main focus of this work.
Recent work is generally designed for pruning to unstructured sparsity and has then been applied structured patterns. In this work, instead, we design family of regularizers that directly induce structured sparsity. The resulting regularized pruning problems are then solved using the celebrated proximal gradient method. Unlike existing pruning methods, our method induces sparsity gradually over the iterations, which can lead to better masks. The proximal gradient method requires solving a prozimal operator in cach iteration, which itself is a nonconvex problem.
The key theoretical contribution of this paper is to show that this non-convex proximal operator can be efficiently solved by solving three convex subproblems. We empirically show that these regularizers tend to identify more efficient sparsity masks, reduce the squared loss and can lead to better performance when applied to pruning entire large language models. Our approach natively applies gradient descent the layer level while finding the mask and after freezing. Our key empirical contribution is that apply such local gradient descent after masking to previous state-of-the-art methods (Wanda and SparseGPT) and find that we can improve those out of the box. Since Wanda and SparseGPT methods are extensively adopted, we expect that the masked gradient updates will have significant impact.
We defer proofs to Appendix
1.3
1.75
1.30
1.30
1.5
1.35
A1
1.0
47
0.25
42
13
0:58
3.0
01
02
0.3
0.4
05
0.6
0.7
0.0
0.2
3.8 0.6
0.5
3.0 0.2 1.4 2.5
05
022
0.6
2.8
Figure 1: Illustration of the regularization path, e.g., optimal solution (W1, u22 W3, W4) as a function of À in
Equation 7 From left to right, we are showing the result with (a) easy input [1.6, 1.1,0.8, 0.5] (b) nearly-tied 22:1 with 2 = [1.6, 1.11, 1.1, 1.09] (c) nearly-tied 21:3 with & = [1.6, 1.59, 1.58, 1.09] (d) All nearly tied with = [1.6, 1.59, 1.58, 1.57]. The two dashed lines all figures indicates the two critical thresholds Aj and A/ when the 3-sparse solution (and 2-sparse solution) become a critical point of Equation 7 thus
A >j (or >> X1) are necessary conditions for the solution to be 3-sparse (or 2-sparse), see Lemma 10 in
Appendix B Observe that in the "easy input" case, as increase X, the sparsification is gradual, rather than abrupt. In the "hard input" cases, decisions about those that are nearly tied are not made prematurely at smaller X, the transition to exact structured sparsity happened al a much larger X.
2 Problem and related work
Large Language Models (LLMs) based on the transformer architecture Vaswani al. 2017 have become the workhorse of language modelling. Recent architectures like Ilama Touvron et al. 2023 use decoder only models. Innovations on the attention calculation (Dao 2023) or its design as grouped-query attention (Ainslie et al.) 2023) have in few years already led lo notable efficiency improvements. This shifts the focus toward the classic matrix multiplication in linear layers. The idea here is to find an efficient approximation of the weight matrix such that the multiplication can be executed more efficiently as well as that the matrix can be stored efficiently in order to reduce the memory footprint of the matrix multiplication.
2.1 Matrix Compression
Traditional approaches involve low-rank approximations and sparsity. Modern hardware accelerators, which are used to run LLMs, cannot make of sparsity if the Os do not follow regular structure. On the other
2

---

Published in Transactions on Machine Learning Research (08/2025)
hand, if a lot of structure is imposed, for example entire rows columns are zeroed out, the accuracy drop is often too large.
A method aiming to strike a balance is so-called structured sparsity or N:M sparsity. Here, out of M consecutive weights N are set to zero (Pool et al., 2021). For the case of 2:4 sparsity, one stores only two weights as well as a two-bit index per weight. Thus, for example, if the weights are kept in 16 the memory footprint reduces from 4 16 = 64 bits to 2- (16 + 2) = 36 bits, ie., a compression to 56%. Also since those patterns are efficiently supported in hardware, it can reduce the FLOPs in matrix multiplications by factor of up to 2x, which is mostly relevant during prompt encoding (aka prefill). LLM inference during decoding is on the other hand memory-bound rather than compute bound (Park et al. 2024). There the speedup cornes from the memory compression. We note here that this speedup reduces when sparsity is combined with quantization (Jacob et al. 2018: Dettmers et al. 2022 Frantar et al., 2023), as the relative overhead of the position indices grows. With 2:4 sparsity the memory compression and decoding speedup for 16/8/4 bit datatype is 1.77x/1.6x/1.38 1.3x..
2.2 Pruning
Many recent works propose new algorithms and heuristics to prune the linear layers LLMs. Whilst some focus on sparsity (Frantar & Alistarh| 2023 Sun et al. 2024 Dong et al.) 2024 Wei al. 2023) others also remove entire structures, like heads, channels layers resolving the need for special hardware support (Xia
et al. 2024 Ma et al. 2023; Muralidharan et al. 2024 All of these methods have non-negligible drop in performance metries, requiring a delicate orf between latency/con and performance degradation of the models. It is known that fine tuning after compression recovers some of the performance drop (Sun ctt al. 2024 Frantar & Alistarh 2023 Dong ct al. 2024 and recent work have also studied more extensive distillation with a small percentage of pretraining lokens, for example Minitron (Muralidharan el al. 2024 or Llama-3.2
2.3
One-shot pruning with squared loss
In this work we focus on a local one-shot setting (Frantar & Alistarh 2023). which we introduce in the following. We consider linear layers in a deep neural network. Let N, M E N, N - and di, do E N be the feature input and output dimension, where
assume that d; is divisible by M. We call weight matrix W E Rdoxd. to be N:M sparse if each cell of M consecutive weights contains maximally N non-zero entries EN W M44l S N for all & E [d/M] and for cach row r E [do]. We will denote the set of N:M sparse matrices as SN:M, leaving the dependency on di, d, implicit.
We assume that the network has been trained to convergence resulting in dense weights W Assuming that we have inputs to the liner layer X e Rd our goal is find N:M sparse weights that maintain the output of the layer as good possible in terms of squared loss (LeCun el al. 1989 Hubara el al. 2021 Frantar &
Alistarh
2023
Sun et al. 2024
arg min 2 - W W. n
This problem can be solved for each row independently, because N:M sparsity imposes the samne level of sparsity across all rows and the Frobenius norm decomposes over the rows. Nonetheless, even for a single row, the problem becomes combinatorially hard. Whilst given a sparsity pattern, solving for the optimal weights is a convex problem, the number of masks that need to be searched (M) d//a4 Since N, M are determined by the hardware and fixed, the complexity grows exponentially with the matrix dimension and in practice we inevitably need to resort to heuristios.
We rewrite the loss in terms of the Hessian := X and using |A = Tr(AAT) as n
L(W): = IWX - x
(2)
Tr ((W - WW))(WWWW))
3

---

Published in Transactions on Machine Learning Research (08/2025)
We will also refer to this as local squared loss to emphasize that it is on per-matrix level. Notice that the diagonal of H we simply have the mean squared activations of the corresponding input channels.
The N:M sparse optimization problem simplifies significantly if the Hessian H is diagonal, meaning that the input features are uncorrelated. In this case the weights cannot compensate for each other, that is each weight is either pruned to zero or kept at its original value. Given a pruning mask M E {0, 1) 1)dxxd the loss is simply
(1- W. Hy =: 1(1-M13)
(3)
and we can solve the minimization problem efficiently and optimally by simply pruning those weights with smallest scores St, == |Wr, 31 Although not derived the way presented here, this preciscly the criterion that Wanda (Weights and Activations) (Sun et al.) 2024) proposes for pruning Large Language Models (LLMs) and use it as heuristic even when the Hessian is not diagonal.
Theorem 1 (Wanda is locally optimal for diagonal Hessians). Let the Hessian be diagonal, W arbitrary and define M E {0, 1]d,xd, such that for each M cell it has zeros for the N values with smallest score Sig- Then, M @ W* is minimizer of problem Equation 1
SparseGPT (Frantar & Alistarh, 2023) introduce an iterative heuristic that prunes the weight matrix in column blocks from left to right and takes off-diagonal elements of the Hessian into account. To determine which weights to prune, they use the criterion introduced by Hassibi & Stork (1992) and update the weights in the remaining right blocks, by using the inverse Hessian of the remaining sub-matrix. SparseGPT allows to efficiently prune multiple rows at once, without costly recomputations of the inverse Hessians.
Notice that both SparseGPT and Wanda only require forward passes through the full model to populate the Hessian matrix or mean squared activations for each linear layer, which we dub one-shot setting (Frantar & Alistarh 2023 . Both algorithms run in matter of minutes / hours on a single GPU on large language models up to scales of 100B parameters. We focus on aforementioned one-shot setting, where we only allow local updates, but no end-to-end tuning of the entire model. Further recent works one-shot pruning works include DSnoT (Zhang et al.] 20244. which provides small improvements in some cases over Wanda and SparseGPT, and ALPS (Meng ct al. 2024), which is particularly suited for high sparsity. Dong et al. (2024 investigated other criteria beyond one-shot pruming.
2.3.1 Applicability to arbitrary models with linear layers
While empirical experiments and investigations focus large language models, we emphasize that our methodology applics to any pretrained neural network having linear layers. The required information for our algorithm is solely the weight matrices as well as the local Hessians H, which are computed by aggregating input activations over forward passes on calibration data.
2.4 Proximal gradient
The proximal gradient (PG) method is well-known optimization algorithm designed for solving composite minimization problems, where the objective function can be decomposed into two parts: a smooth differentiable function and a nonsmooth. Such problems arise frequently in many modern applications. In this setting, the objective function typically has the form:
(indf(()+h(un)) +
where ʃ R. - R is smooth and differentiable, while h Rd - (-x0, oo) is a possibly nonsmooth stracture- inducing regularization term. To account for this nonsmooth component, the Proximal Gradient (PG) method incorporates a proximal operator alongside the standard gradient step for the smooth part, f fww). The proximal operator, designed to mainly tackle nonsmooth functions, is defined as follows:
1 I|w - =112 + n)w)) }
pr)xx = arg min
2
4

---

Published in Transactions on Machine Learning Research (08/2025)
Therefore, to sum-up, starting with an arbitrary wo, PG method with step-size >> generates iteratively sequence (w*) via the following iterative step:
whth p (w +V fw)").
The PG method is particularly efficient when the smooth part f has a Lipschitz-continnonn gradient and when the proximal operator for h has a closed-form solution (e.g., the hard-thresholding operator for fo-regularized problems). However, if the proximal operator has no closed-form solution it might require an additional failored optimization algorithm to solve the corresponding problem. Below, we will propose a new function to promote N:M sparsity and discuss in detail its 2:4 proximal operator.
Proximal gradient methods have been extensively studied, particularly in the context of convex optimization problems. For a thorough overview of their theoretical guarantees and extensions, refer to (Beck 2017) and references therein.
3 Proximal operator for N:M sparsity
Wanda (Sun et al. 2024) and SparseGPT (Frantar Alistarh, 2023) were both designed primarily for unstructured sparsity and then simply applied to the N:M sparse case by adding these additional constraints. Inspired by PG methods we propose regularizers that are explicitly designed for N:M sparsity and iterate gradient updates the squared loss (done the matrix level) with the solution of the proximal operator (which decomposes into the M cells). This leads to gradual emergence of the pruning mask, where the emerging pattern of one cell can in fact influence the pruning pattern of other cells and avoids committing to a sparsity pattern to early. Wanda and SParseGPT do not have this possibility. Our approach will consume more compute and time, but this is well invested. Compared against the cost of training the models and the inference cost, the additional cost to find better mask and model is well invested.
3.1 Gradient descent
Each step of PG roquires a gradient step on the unregularizod loss Equation (2) which has a simple closed-fore form, see Appendix A
W < W 1) 2(WH W*H),
(4)
where we can use the largest eigenvalue of the Hessian to set the stepsize 7 27mss guaranteeing convergence. While each gradient step has complexity Odddo), it allows to efficiently the full parallelization of modern GPUs.
We will use gradient descent during PG, but we also propose it as a method to improve any local pruning method. Once the pruning mask M is fixed, we can partially compensate for the pruning loss, by masked gradient updates:
War + WM 7 2M O (Wa.H- - H*
55)
Our first contribution is show that Wanda and SparseGPT do benefit from such gradient updates after they completed the pruning.
3.2 Regularization and proximal operator
To induce N:M sparsity, define the following family of regularizers
TN:M Σ
II
[uyl.
(6)
SCIM] JES
IS|=N+1
Fact 2 (The null space of the regularizer). For all E R we have
TN:Md (w) = if and only if ||lllo S N. 2In Appendix we propose few more, elaborate, regularizers 2:4 spansity.
5

---

Published in Transactions on Machine Learning Research (08/2025)
The corresponding proximal operator is
AM (2) == arg min
w z12² + Arwar(w))
weRM
2
The regularizer UN:M promotes N:M sparsity in that N sparse solutions are in the mull space of this regularizer. Moreover, as À + oc, there always exists X (as function of imput z) such that when X X X", the solution of the proximal operator becomes exactly N-sparse (see Figure for an illustration).
For 2:4 sparsity, the proximal operator of interest
arg min wER*
2 Ilw 2112 X([w2 [[wil|/22lle |/ws + |uy /
+ Iuy | ||1
|wa (oul|toyll /
The proximal operator can be used to induce 2:4 sparsity in all settings including: layerwise pruning, finetuning and pretraining using proximal gradient methods or straight-through gradient mothods. We focus on layerwise pruning via the squared loss Equation The combined loss function for the 2:4 sparse case is then
Lx(W) L(W) Σ r2a(w).
WEW
where W rums over all 2:4 cells of W. Following Fact 2 the matrix W is 2:4 sparse if and only if the regularizer is zero. Importantly the proximal operator corresponding to the regularizer of multiple cells decomposes into multiple 2:4 proximal operators and hence the complexity of solving the proximal operator at each iteration of PG only grows linearly with the dimensionality of the matrix.
3.3 Solution of the 2:4- 2:4-proximal operator
Solving the 2:4 proximal operator problem is tricky. is non-convex, non-smooth and appears to require an exhaustive search-style approach. Nonetheless, we have identified interesting mathematical structure this problem which led to discovery of very efficient solutions to this problem with provable guarantecs.
We first prove that
always reduce the proximal operator to the where is non-negative and sorted 21 22 2 2 0.
Lemma 3. To solve problem Equation 7 it suffices to solve
arg min (ww), with wER1
(9)
=12 + M(ur + w2W3W4 f(w) = (
+ W3 + w2wlw2)
(10)
where 21 > 22 i 2
IV IV
IV IV
2 Moreover, any optimal solution w of problem Equation 9 satisfies that
wy
Observe that f(w) is smoothly differentiable, but now we have constrained optimization problem. The same result can generally be stated for NiM-sparsiy.
Let us define the restricted version of problem Equation
9 when w4 is set to 0.
arg min g(w), WERS
with
(11)
g(w)
Σ (w 2,)² Aw w2t033
(12)
As side remark, the proximal operator for r2:3 be reduced to solving Equation 1

---

Published in Transactions on Machine Learning Research (08/2025)
Lemma 4 (Optimality conditions). Assume 21 22
23 2 > 0. The solution w" is one of the following
three cases.
2-sparse: in which case w' [21, 22, 22, 0, 0,
3-sparse: in which case w =0 and wis satisfies
wi= 21 uw wgg
0 w 22 Xw wÿ.
0 < Wi 23 Xwi WTz
Dense: in which case w [wf, w2, w3, wi] > 0 satisfies that
m 24
Σ
" W
In Appendix B.4 we also include necessary conditions on À for the solutions be 2-/3-sparse. Whilst the 2-sparse solution is trivial, optimizing for the 3-sparse and dense solution requires care. We next turn to the Hessians of f and 9, see Equation (23 and Equation (24 for their explicit forms.
Lemma 5 (Propertics of the Hessians).
The set C3 == {w E TR3 V2g(w) č 0} is conver.
The set C₁= {w R V f(w) ž 0} is conver.
This property of the Hessian is crucial, as allows us to focus on finding a single minimum.
Corollary 6 (No spurious local minima.). The set of all local minima of f'w) is conver. Moreover, if there erists w E arg win f(w) such that w > for all i E [4] (te., in the "Dense" case from Lemma 4), then all local minima of fw) global minima.
The set of all local minâma of g(w) is convex. Moreover, if there exists w E arg min g(w) such. that : > for all i E (3), then all local minima of g(w) are global minima.
To solve the proximal operator we can thus solve the following constrained optimization problems
min {9(w) w 3 w
g ž 00
(13)
min {f(w) E IR
2 0}
(14)
The optimal solutions to Equation 13 and Equation 14 are local minima Equation 11 and Equation
9
respectively the solutions have gradient can be scattered all over the place, but with Corollary we have shown that for these special functions f and 9, the local minima (if they exist at all) form convex set and have the same objective values.
In general, the set of local minima for a non-convex problem
Problens Equation 13 and Equation 14 are convex optimization problems due to Lemma They can be solved using interior point methods with self-concordant barrier functions. As barrier functions, we can use log det f²f(w) f(w) and log det V²g(w) respectively, as well as - log(w,) to encode the positivity constraints
Boyd & Vandenberghe 2004 Chapter 11.2).
We have now reduced our initial non-convex problem to three convex subproblems, out of which one has trivial solution. We can then solve those three problems and select the minimizer among the three cases
Theorem 7. Algorithm returns an optimal solution to Equation 9
3ff 4 in Algorithm local minimum of by Corollary 0 this suffices to know it the global optimum and there need compute (2),, For of notation and parallelize when solving multiple cells, nonetheless always compute the three cases.

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm Solve Prox with decreasing non-negative input
Require: 21 2 22 2 23 2 24 20
1: procedure PROXENUMERATE(2 A)
2: w((2 <- [2 22, 0,0]
3: (3)) Isoluion to Equation
13 |].
4: w(4) + solution to Equation
5: return arg min we (w()) anl9) a
(ww
end procedure
3.4 Generalization N:M sparsity.
2:4 sparsity currently the most relevant sparsity pattern. plausible that general N:M sparsity patterns also benefit from more gradual pruning algorithm. For 1:M sparsity, the proximal operator becomes very simple it is just quadratic function and it can be solved in closed form. However, for more general patterns when N > 2 solving the proximal operator becomes more challenging, Lemma still holds and it is enough to consider the non-negative sorted casc. However, finding the minimizers is harder. In particular it is not obvious how one could replace Lemma 5 i.e., the convexity of the space where the Hessian is convex. There we used that the Hessian in the 2:4 sparse case is linear in w, which will no longer hold for N > 3. There might be different arguments to show that those cases can be handled efficiently as well.
3.5 Parallelized Prox by gradient descent
We showed that Equations 13 and 14] are convex optimization problems that can be solved with guarantees for example with interior point methods with self-concordant barrier functions. However, in order to apply this algorithm at scale we need to optimize it further. For example when pruning the MLP down projection of a Llama 70B model (Dubey et al.) 2024). the matrix has dimensions 28672 and 8192, meaning that it has around 59 million cells of four weights. Thus at each iteration of Proximal Gradient, we have solve Algorithm 59 million times, which infeasible if not done a parallelized fashion using modern GPUs.
In all our experiments, will thus resort to gradient descent solver of which conjecture that it always correctly solves the problem. We will focus on the case of Equation (14), and the case for Equation (13) follows similarly. The constraints 2 O is casily enforced by projected gradient descent. The second constraint (²)(w) 2 0 is more computationally expensive and we do not want to compute it at every iteration. Luckily, empirically we find that we actually do not need to enforce this constraint at all. First note two facts
b) Let Tmoe denote the largest eigenvalue of matria. For all w E R4, we have V² f(w) 2 0 Tman 2 w S 4.
Fact 8. a) w= = [0, 0, 0, 0] is always a feasible point of Equation
Thus, whilst we are in the convex region of the loss, the gradients are Lipschitz with constant 4. Using 1/4 as step size, we are guaranteed to never cross a local minimum as long as we stay in the convex region. In our extensive ovaluations we observe that indeed gradient descont does never exit the region of convexity if the global minimum is dense. This implies that we can run gradient descent without the need of barrier function. Whilst we could not find any counterexamples we have not been able to formally prove this and hence state is explicitly
conjecture.
Conjecture 9. Let wt the minimizer of Equation 14 and suppose that wi > for all Consider the iterative scheme
w =, - (w* -"'VV"(VV")"
where the projection (.)+ is applied elementwise and denotes projection onto R4, and 1/² = 4.- Then the sequence {u*} converges to w².
Furthermore, each iterate w lies in the feasible set of Equation 14
The same considerations and empirical insights apply to the 3-sparse case. Hence, in practice when computing (3) and (4) in Algorithm 1 we the gradient descent based optimization without enforcing to remain
8

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm Matrix Prox Pruner 2:4
procedure PRUNEPROX(W") H, (a))a)
W < W 17 - 21max k 6++00
3: while W $ 2244 do
4:
W + W 11 2(H HW")
W, p + PosSort(W)))
W <- ProxEnumerate(W, Ap)
W <- InvPosSort(W,
8:
k < 11
end while
10: return W
11: end procedure
in the region of convexity. As soon as we witness that. we moved out of the convex region, we can stop the optimization as by Conjecture and Corollary the respective case can not be the optimal one. In practice, stop when witnessing that the gradient region with our choice of step size. Therefore, we slightly deviate from Algorithm [ in that in line 3 and 4 we do not compute the exact minima if we witness its not the minimum of the three cases anyways. If Conjecture g is true, then Theorem holds even with this modification.
increases, as this cannot happen whilst being in the convex
In our inplementation, the GD-based Algorithm is more than 10x faster than our implementation of the interior point method-based Algorithm and they always obtain numerically the same solution.
3.6 Full algorithm
We can now put together the full matrix pruning algorithm, see Algorithm 2 We define two functions: PosSort(W) that returns the weights such that each 2:4 is sorted and without sign as well as s,p that indicate the sign and original position of the weights. InvPosSort(W, s, which undoes above operation. Furthermore, we use a schedule {A for the regularization such that d + 30. In this work we consider exponential schedules Åς An. for do 0 and 8 1. Having exponential schedule guarantees that the regularizer eventually is large enough such that the solution to the proximal operator will be exactly 2:4 sparse, c.f. Section 3.2 which we need to ensure finite runtime of Algorithm 2 In other words: without making assumptions about the problem space, the regularizer eventually needs to be scheduled towards infinity.
4 Experiments
For our experiments unless otherwise stated we use A Ao,8h with Xo 0.01 and B == 1.01. Furthermore, we always use the solver Algorithm relying on Conjecture g After ending Proximal Gradient, we do 1000 steps of masked gradients according to equation 5 to minimize the local squared loss. Furthermore, we initialize all 1/2 H. 1/2 This helps the proximal pruming to not commit to weights that seem important solely in terms of magnitude, but takes the respective mean squared activations into account (Sun et al.] 2019). After pruning this transformation is reversed. All experiments can run on a single NVIDIA A100 GPU with 40GB of memory, but we used multiple GPUs in parallel to speed up the experimentation.
1/2 methods Wanda style by transforming W + W Wr," Hij + H. /
4.1 Toy experiments
We start with a simple experiment to illustrate the inner workings of our algorithm and the shortcoming of Wanda and SParseGPT. We consider the 2:4 sparse case of a single row with dense weights w" = (0, 5,3, 2, 0,5, 5, 2) We further assume that the Hessian has ones on the diagonal and zero elsewhere except that the fourth and eighth weight are perfectly correlated (value on the respective off-diagonal elements). In this case the optimal 2:4 sparse solution is (0,5,0, 0, 0, 0,5,6, 5, 0). The difficulty here is that the fourth

---

Published in Transactions on Machine Learning Research (08/2025)
relative WandA GD
0.9
0.8
0.7
0.6
prox + GD
WandA GD
0.5
SparseGPT + GD
0.0
0.2
0.4
0.6
0.8
1.0
Figure 2: Toy experiment with generated weights and Hessians. Without correlation (a= 1) considered pruning methods result in the same mask and loss. As we increase the correlations, our proximal approach leads to the best mask as shown by the smallest loss.
and eighth weight if looked at individually within their 2:4 cell, would be pruned, but instead we can merge them into one weight and prune another weight incurring a smaller loss.
Since Wanda (Sun et al. 2024) completely ignores the correlations, prunes to w (0, 0,35,, 0,5,5,01 which suboptimal. SparseGPT (Frantar & Alistarh 2023) would in first step prunc the first '2' because it can completely absorb it into the last weight due to their correlation w then in the second step, when pruning the second cell, it drops the combined '4' because it incurs lower loss than pruning one of the '5's w + w (0,5,3 5, 3,0, 0, 0, 5, Therefore, the SparseGPT heuristic can lead to deferred loss.
5,,5,3,0, 0,5, 4). However
Our proximal optimizer in turn adapts to the situation. In the second cell, the '2' is quickly pushed to 0. This in turn increases the gradient magnitude the first '2', pushing eventually over the value of the '3' the first cell. To simplify the understanding, we animated the evolution of the weights with gif LINK
Next consider the effect of correlation between input features structurally. Therefore generate synthetic Hessians and random weights:
d=1024
diag = torch. tiag(torch.rand
Z = torch randn (d, d) / float (d) ** (1/2)
2 = 2 a 2.T
hessian alpha diag (1 alpha) weights torch.randn(f. d)
We then prune with all the considered methods and after masking optimize the weights with gradient descent. Results are discussed and shown in Figure?
4.2 Scheduling A
During the course of proximal gradient, we increase the regularizer À with exponential schedule to guarantce, that no matter what input, eventually the weights will be 2:4 sparse. In this work we consider exponential schedules A = \n,k, for A > 0 and 3 > 1. Clearly, good choices of those hyperparameters depend on each problem and are expensive to ablate when done for each problem separately. We show the impact of different choices on the local loss and the runtime in the Appendix Figure 3 Notice that the optimization problem equation
not scale invariant when scaling z alone. However, u* the optimal solution to the problem 2, X, then ow" is the optimal solution for the problem (cz, A/c). Since is too
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Validation Perplexity of OpenLlama (3//7B/13B) and Llama3.1 (8//70B) (Pretrained/Instrurce All the methods with +GD are an innovation of the present work.
3b
v2
7b
v2
13b
8B
8B Inst
70B
70B Inst
Method
C4
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
dense
9.68
14.15
8.84
12.15
8.11
11.52
9.68
7.93
11.28
7.27
7.32
3.34
8.26
3.71
wanda
29.48
60.32
17.63
27.02
13.17
35.45
36.32
27.26
43.30
26.67
13.42
10.25
13.68
8.74
wanda+ GD
18.23
33.05
14.35
21.85
11.73
18.17
22.19
20.46
28.45
20.71
12.32
10.38
13.08
8.91
sp-gpt
18.76
33.78
14.24
22.29
11.91
19.51
24.95
23.18
35.55
26.19
13.00
10.93
14.05
9.37
sp-9pt+ GD
16.72
29.58
13.31
20.83
11.43
17.19
21.00
19.86
27.54
20.85
12.06
10.49
13.06
8.91
DSnoT
26.54
44.17
17.62
26.17
13.71
32.89
40.66
30.40
48.80
31.46
14.34
10.54
14.51
9.16
proz+ GD
16.27
28.59
13.19
20.55
11.40
17.71
20.86
19.83
26.46
20.20
12.33
10.38
13.12
8.91
expensive to extensively search the hyperparameters for each problem individually, we propose to consider
d0
A Mean W 8h.
(15)
4.3 Pruning large language models
To prune LLMs, we 2 million tokens from the c4 dataset (Raffel et al. 2020). which pack with end-of-sequence tokens and chunk into 1024 sequences of lengths 2048. We estimate all Hessians from the umpruned model, and do not propagate the pruning errors as we find that this has little impact (Appendix D.I). As basclines we run Wanda (Sun et al. 2024) and SparseGPT (Frantar & Alistarh, 2023) both with their original algorithrns as well as with our innovative additional 1000 gradient descent steps after masking, see Section 3.1 where we observed that the local loss has usually converged after 1000 steps as shown Appendix Figure We also run DSnoT (Wanda) (Zhang et al., 2024). For the proximal, for small models we used our default setting A Au with Åo 0.01 and 3 = 1.01. For the 70B models, observed that this would require more than 2000 iterations of PQ. We thus used the heuristic from Equattin (15) and the results of Figure 3 and selected / = 1.005, Ag = 1- 10-3 to strike a good balance between performance and runtime. Since, prox already uses gradient updates during pruning, we recommend to always use it logether with gradient updates after freezing the mask. For completeness though, in Appendix D.3 we also include a small ablation on the effect of masked gradient steps for prox.
We prune models from the openLlama family (Geng & Liu. 2023 and the Llama-3.1 models (Dbeey et al., 2024), see Table D We evaluate the models both on in-distribution validation data from c4, as well as out of distribution data from Wiki2ext2 (Merity et al. 2016
On models up to 13B parameters, we see that the proximal approach consistently outperforms the other methods at least in-distribution (C4) and mostly also on wikitext data (wiki). In the appendix Figure we show that this is related to an improvement in local squared loss. Furthermore, Wanda and SparseGPT both benefit clearly from our proposed masked gradient descent based updates after pruning with the respective strategy.
Interestingly, whilst prox tends to give the best perplexity, it turns out that our innovation using masked gradient updates (Section 3.1p is empirically more relevant, as it consistently improves the perplexity on C4. As a by-product of our work we thus identified a method that can be plugged on top of existing SOTA one-shot pruning methods. Table 3 we evaluate the pruned Llama-3.1 SB Instruct model on 6 Downstream tasks and find that on average we improve the SOTA more than 3%.
On the 70B models, whilst the gradient updates continue to improve the in-distribution loss it can harm the wiki perplexity in case of Wanda, which can indicate a slight overfitting to the domain of the calibration
We emphasize that all methods our experiments use exactly the same number of calibration samples. Appendix D.2 further ablate the effect using different number of calibration samples.
11

---

Published in Transactions on Machine Learning Research (08/2025)
data. Furthermore, on the 70B models we see that the method to find the mask has very minor impact on the final perplexity. On the 70B model scale, we thus recommend to use wanda for pruning instead of SparseGPTT or prox and rather invest resources into having a more balanced and larger calibration sample to estimate the Hessian from in order to mitigate domain speciliazation.
We can qualitatively define a hierarchy between the investigated methods by how much information of the Hessian matrix is used whilst identifying the mask. Wanda uses only the diagonal entries (see also Theorem 1), SparseGPT uses already the full Hessian, and prox makes even heavier use of the Hessian (see our toy experiment of Figure 2). We observe that on the smaller models, SparseGPT does quite consistently outperform Wanda, hence it scems that the off-diagonal elements of the Hessian are relevant. It is those cases, where doubling-down on using Hessian information (through prox or masked gradient updates) further improves performance. On the other hand, on the 70B scale using plain Wanda seems close to optimal neither SparseGPT, prox, nor masked gradient updates improve there relevantly. This scems to indicate that the bohavior on this scale largely determined by the diagonal entries of the Hessian matrix in which case wanda. (by our Theorem 1) is optimal. However, the difficulty in analyzing and predicting the end-lo-end behavior from the properties of the matrices is that the weight matrix and the Hessian need to be considered jointly. Existing methods to characterize matricos like Kruskal Rank (Kruskal & Wallis 1952), Mutual Coherence, or the restricted isometry property usually consider a single matrix in isolation.
Overall, we find that a 2:4 sparse model is consistently worse than the dense smaller model of the same class which is consistent with the results on Llama. 1 and models found by (Sun et al. (2024) Table 3). A potential way to mitigate this large drops is only partially prune the model, e.g., only certain modules or only a fraction of the model's layers. This can be considered for all pruning methods equally and is an orthogonal direction for future work. Additionally, pruned models can be further fine-tuned to specific downstream tasks, which can further mitigate the loss in quality.
5 Discussion
We have introduced a method to gradually induce structured sparsity in pretrained linear layers, for example of large language models. The immediate objective is to minimize the lincar squared loss at the matrix level. We showed that a local improvement of the squared loss leads to an improvement in model perplexity. On the theoretical side, we showed how to efficiently solve the complex proximal operator for 2:4 sparsity.
On the empirical side, we observe that the sparsity mask does not make huge difference for recent LLMs, but that our introduced local gradient based updates can also be used on top of existing local pruning methods and improve their performance. In particular, our proposed masked-gradien updates can easily be used to improve the widely adopted methods Wanda and SparseGPT. We believe that the masked gradient updates will be widely adopted.
Whilst we thus improved the state of the art on one-shot pruning, the resulting pruned models clearly are not good enough to be of direct practical relevance. Looking at Table [ we see that the pruned 13B/7B models fall clearly behind the dense 7B/3B model. However, for a 2x matrix compression this should be the lowest bar a model has to take in order to be useful. This is even more because whilst the structured sparsity reduces the latency and footprint of matrix multiplications, it does not decrease the sizes of in-/output activations the KV-cache.
One benefit of the proximal approach and the theoretical insight we provided is that they can also be applied during pretraining togother with a rogular pretraining objective D Other pruning approaches discretely interrupt the optimization, whilst the proximal operator is applied at every step, hence gradually pushing the model to sparsity. This could pave the road for more powerful sparse models with less quality degradation.
References
Joshua Ainslie, James Lee-Thorp Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training gencralized multi-query transformer models from multi-head chockpoints. In Proceedings of
5 Concurrent work Fang et al. 2024: found that end-to-end learning of masks is beneficial also post. training.
12

---

Published in Transactions on Machine Learning Research (08/2025)
the 2023 Conference Empirical Methods in Natural Language Processing, 2023.
Salch Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hocfler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024.
Amir Beck. First-order methods in optimization, volume 25. SIAM, 2017.
Stephen Boyd and Lieven Vandenberghe. Conver optimization. Cambridge university press, 2004.
Krishna Teja Chitty- Venkata, Sparsh Mittal, Murali Emani, Venkatram Vishwanath, and Arun K Somani. A survey of techniques for optimizing transformer inference. Journal of Systems Architecture, pp. 102990, 2023.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv:2307.08611, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.intS(): 8-bit matrix multiplication for transformers at scale. In Advonces in Neural Information Processing Systems, 2022.
Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zers Evolving symbolic pruning metric from scratch for large language models. In Proceedings of the 4 1st International Conference on Machine Learning, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv:2407.2178 2024.
Abhinav Dutta, Sanjeev Krishnan, Nipum Kwatra, and Ramachandran Ramjee. Accuracy is not all you need. arXiv preprint 07.002447,09141, 2024.
Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnable semi-structured sparsity for large language models. NewrIPS, 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefter, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. TCLR, 2023.
Elias Frantar, Roberto I, Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. Marlin: Mixed-precision auto-regressive parallel inference on large language models. arXiw:2408. 11743, 2024.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL https://githubb comn/oppenlmmrreeerrrcccpppp_p_p 11ama
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neurol information processing systems, 1992.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry, Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in newral information processing systems, 2021.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmeeiiconly inference. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
William H Kruskal and W Allen Wallis. Use of ranks in one-criterion variance analysis. Journal of the American statistical Association, 47(260):583 -218 1952.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 1989.
Xinyin Ma, Gongfan Fang, and Xinchao Wang, LIm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 2023.
Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumnder. ALPS: Improved optimization for highly sparse one-shot pruning for large language models. In The Thirty-eighth Anreal Conference on Neural Information Processing Systems, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv: arXiu:1609.078 609.07743, 2016.
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. orXiv:2407.1400 14679, 2024.
Youngsuk Park, Kailash Budhathoki, Liangfu Chen, Jonas M. Kübler, Jiaji Huang, Matthäus Kleindessner, Jun Huan, Volkan Cevher, Yida Wang, and George Karypis. Inference optimization of foundation models on ai accelerators. In Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2024.
Jelf Pool, Abhishek Sawarkar, and Jay Rodge. Accelerating inference with sparsity using the nvidia ampere architecture and nvidia tensorrt, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 (1140))1-667 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. ICLR, 2024.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression, 2019.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXit:2307.09288, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. Advances in neural information processing systems, 2017.
Xinying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization large language models by equivalent and optimal shifting and scaling, EMNLP, 2023.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations, 2024.
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwn Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-fre free fine-tuning for sparse LLMs. The Twelfth International Conference on Learning Representations, 2024.
14

---

Published in Transactions on Machine Learning Research (08/2025)
A Gradient and Hessian
We have the following loss function Equation 2
L(W)= = Tr ((W - - -W))))
The loss simply the sum of the loss of cach individual row and we can minimize this on per-row basis (w E IRd)
L(w) (w (ww w)).
(16)
From here we can simply compute the gradients:
VL(w)= 2H - w²) = 2Hw 2H
(17)
We can the stack this again for the whole matrix and obtain:
VL(W)= 2W H.
(18)
Notice that the second term can be precomputed once, and does not need to be computed again at each iteration. Computing the second order derivatives per row we find L2 = 2H, hence also our naming conventions (ignoring the factor of 2).
Proofs
B.1 Proof of Theorem
Proof. We first show that diagonal Hessian implies that for any mask, the optimal values the remaining weights match their dense counterparts. Therefore, let M be an arbitrary binary mask and assume that the Hessian is diagonal, i.., Hi = 0 whenever * j. We use the formulation of Equation where explicitly add the mask
L(W, M) = (MM W W*) ow W-W")
(19)
Wis
- &
(20)
Σ
(W W 2H +
Σ
(2
(21)
M
Notice that the second term solely depends the mask and not on the weights of W. Furthermore, the first. term can be set to zero by matching the original weights. By definition WandA selects mask that minimizes the second term whilst keeping the other weights fixed, hence minimizing the overall squared loss. ]
B.2 Proof of Fact 2
Proof. The "if" part is trivial. If || < N, then for any index subset S C [M] with |S| N 11,, at least one of the coordinate of ws is O. To see the "only if" part, if suffices to prove the contra-positive. Tf ||llo > N, let to be the first N +1 coordinates of w in S, thus (w) 2 Ilss |uy] > 0.
B.3 Proof of Lemma 3
Proof. First observe that the objective function of problem Equation 7 is invariant to joint permutations and signs of w and z. In other words, if v= an optimal solution of problem Equation 7 for a given input & e R4, then for any permutation and any diagonal matrix D with diagonal elements being -1 or 1, the DI is an optimal solution of problem Equation 7 when the input DITE and having the same optimal value. Therefore, without the loss of generalization, we can choose be the permutation that sorts the input into descending order by the absolute value of |2|, and D diag(sign(TIz)).
15

---

Published in Transactions on Machine Learning Research (08/2025)
Next, observe that for non-negative imput any optimal solution of problem Equation 7 is always non-negative. To see this, if w is negative and 2, > 0, replacing it with |we'l retains the regularizer but (lw:) z)² < (wf - z.)². If 2 = O, the corresponding optimal solution is wł = 0. which is non-negative. Thus, we can remove the absolute values and focus on solving problem Equation (9)
Moreover, we observe that a sorted (descent order) input implies that any optimal solution also sorted descent order. Indeed, we take w" be an optimal solution with w < w for some S i < 4. Since the regularizer is invariant to permutations,
focus the quadratic part and show that in this case i; which is defined by Wi wj. w wt and tū wg for all & of i,j, we have that
1l"" =|| - 1| z||² = (w] - =1)² + (w] 2j)² - (w; zi)2 (ūj 2z)²
(wş 21)² + (w) 2n)² - -ww - =,) (m) 2j) 2(w3z: 24 + uf - wf2t 4 w5²5)
= 2(w3 - (2s 22)
2 0,
where the last. inequality follows from the facts that w < wÿ and 2; 2 2j. This shows that iú, which is a sorted (descent order) vector, also an optimal solution.
Combining these two observations means that problem Equation 7 which is formulated for any input 2, can be rewritten only for non-negative sorted inpuls. Note that we can always invoke the above observations to recover a solution to the original problem by first solving problem Equation (9) to find w and then DI''D-!"" wn is an optimal solution of problem Equation (7)
B.4 Proof of Lemma
We first state slighlty extended version of Lemma
Lemma 10 (Optimality conditions Extended). Let the objective function in Equation 9
be f(w). Assume
21 2
22
2 23
2 24 > 0. The solution w" must be in one of the following three cases 6
spapree: in which case w' [21, 22, 0,0).
3-spars:: in which case wi = 0 and wis3 satisfies that
0 < wi 21 \w W.
0 < w Z2 mw wyj
< w 23 mu wy.
Dense: in which case w² = [wi, ww, wg. wi]] > O salisfies that
: = 2 A
Σ
wj, w
Moreover if 21, 22 > 0,
A 23/ (2122) is a necessary condition for the solution be in the 2-sparse regime.
> / [ w + wwwws++u + wg)) a necessory condition for the solution to in the 3-sparse regime, for wi.3 the solution Equation 11 weaker necessary condition 2 21/ 24/(212) 22 + 2123).
off we have equality in of the inputs, ie., 2, 2j. permuting those inputs also leads valid solutions. We do not give precedence either of those solutions but simply chose the that follows the order provided by the sorting algorithm. For the proof ignore this degeneracy keep simple.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Proof. If an optimal solution w' of problem Equation (9) is 2-sparse, it means that (2=a = 0. Therefore, w" is a minimizer of the quadratic term. Moreover, since w² is sorted in descent order it means that wi, wa > 0 and w3 = w = 0. Therefore, the desired result immediately follows.
If an optimal solution w" of problem Equation (9) is 3-sparse, it means that w = 0. Therefore, w" is also an optimal solution of problem Equation p1 Moreover, since w" is a non-negative 3-sparse vector follows that wi, w1: w3 > 0. This implies that the non-negative constraint of problem Equation (II) is inactive and therefore an optimal solution must be stationary point of the objective function. The desired result follows
by zeroing the gradient of the objective function of problem Equation
If an optimal solution w* of problem Equation
99 is dense, it follows that w" > 0. Then, the non-negative
constraint of problem Equation (9) is inactive al w". Therefore, w" must be a stationary point of the objective function of problem Equation
Since the optimization problem Equation (9) is a minimization of the (non-convex) differentiable objective function f over the non-negative constraint, is well known that the corresponding KKT conditions are necessary for critical points w", which are compactly given by Vf(w")= 2 0, w" 0 and Vf("")'""" w= = 0. Therefore, from the third condition it follows that for any positive clement of w* the corresponding partial derivative of must be zero. Moreover, by writing the gradient of the objective function of problem Equation we get that
Vf(w") w² = (w" 2 + AVr2:a (w") w² =
1|=0| =|w==""""""======"="""""" 2" "== + 3)"za(w"),
where the last equality follows from the fact that Vr2-aww) w 3 (w) for any R. From here we immediately see that = 2 w" and if (w=" * 0 then À = (2"w" w² - /l3 13)2 From the first condition, that is fw"" 2 0, we get that X 2 (2; -
for all i S 4 that Vrrza(w") + O.
Since the optimal solutions must be monotonically decreasing in i there are no other cases.
The two other necessary conditions about À stem from the KKT conditions of a critical point (with active constraints) for the constrained optimization problem. Specifically, the Lagrangian of ʃ [(w, s(w) vf w.
The conditions for critical points (including those at 0) are that there exists / E E such that
Stationarity: VwL(w, f((w) V
2. Complementary slackness: v,w;; = 0 for 2,2,44
3. Primal feasibility: w 2 0
4. Dual feasibility: v 1 0.
For any critical point to be 2-sparse i.c., w1 > 0, W2 > 0, w/s = w4 0, neods satisfy complementary slackness which implies v1 122 0. Moreover, stationarity condition gives wi 21, W2 22,
8 f(w) Wz3 23 + Aw W12 = -23 A21
1/3 Owa
and
① 0m f(w) = 24 + A21
14
Now the dual fcasiblity condition V3s 144 2 gives the condition that 2 23/ (21 Similarly, for any critical point to be 3-sparse, i.e., wi 0, w/ 0, W/4 0, we have v 122 L/3 0, and that there exists
a f(w)= - = + X(wny + wDd3 + w3w1) 2 0.
W1, W2, W3 can be solved by the following non-linear system of equation
Wi + Xwaw3 211
W2 + Aw W3 = 22
(22)
W3 + Xwn W22 = 23-
17

---

Published in Transactions on Machine Learning Research (08/2025)
For this reason, if we are able to first identáfy solution to Equation (22 then verify that the 24 2 0, then we certified that this solution is critical point a necessary condition for this sparse solution to be global optimal solution.
B.5 Proof of Lemma 5 Proof.
Aw3
Awcz
V ((w)
Awas
Awo1
and
(23)
Au02
Aw1
1
X(w's + wA)
X(w2 + w1)
M(wz + w3))
X(w's + wa)
X(w1 + wa)
N(w1 + W3)
f()(w)=
(24)
X(w2 w74)
X(wi wa)
1
X(wi + W2)
X(w2 wes)
X(w1 ws)
X(w1 w2)
Note that the conditions that 4²g ž and 2 f are Linear Matriz Inequalities, thus convex.
B.6 Proof of Corollary 6
Proof. It suffices to check the definition a convex set, i.e., for any pair of local minima, their convex combination is also local minimum.
Let C4 = (w TRd V fw) 2 0} in Lemma 5 where showed that it is convex. Let u², u² R be local minima of f. Then, since u", u* E C4 and C4 is convex, for any 0 S t S is convex at w(t) == tu" + +(1 (1-t))+*
By the definition of C4 when restricting to this set, f is a convex function. By the first order definition of convex function
ʃ(u") 2 f(v") + (u' "V) = s(v')
similarly
f(v") 2 f(u)(())))))))rr))) (0" - V(('") = f(u")
thus f(()) ((²)==(('))
By convexity of f on the line segment, f(w(t) S <tf(u") (1- t)((() f(u") for all t. On the other hand, as O, we also have fww(t)) 2 f(u") due to that u" is a local minima. These two conditions imply that f(w(t)) = f(u") for all E [0, 1], i.e. w(t) also a local minimum.
To prove the second part we first show that the minimum of over R exists. Therefore note that for any w E R4 if w; > 24, we have (w) > 0 for all i [4], implying that f is minimized within 2; which is closed set. Hence there exists w* E R4 such that w S 2 and f(w") min f(w). Now assume w > 0 for all E [4], then w" necessarily is a local minimum and V f(w") y 0. Conversely, since all local minima. of convex function attain the same value, the statement follows.
The statements for g follow in analogy.
B.7 Proof of Theorem 7
Proof. The optimal solution to Equation 9 has three possibilities: dense, 3-sparse, 2-sparse.
If the solution to Equation 9 is 2-sparse, (w) then (2) is the solution due to that it minimizes } || - 2|I2 subject to the 2-sparse constraint.
If the solution to Equation 9 is 3-sparse with wia > 0, then was is also the solution to Equation
moreover ( 2 0. Since the constraint is not active, and w1:3 is in the strict interior of the constraint,
thus Vg(w = Since waa is feasible, it also optimal for Equation (13). Any other solution /1.3
Equation
13 (if exists) must satisfy g (()1:) = g(wis) thus is also an optimal solution to Equation and
Equation
18

---

Published in Transactions on Machine Learning Research (08/2025)
Similarly, if the solution to Equation 9 is dense with 0 (for all coordinates), then v2f(w) 0 and (f(w) = 0 (due to stationarity and complementary slackness). This checks the feasibility of w in Equation 14h which implies that w is an optimal solution to Equation [4] Let w E IR be any other optimal
solution to Equation
[14], f()) = f(w), thus i is also an optimal solution to Equation (9)
To conclude, in each of the three cases, Line 2-4 of the algorithm returns the optimal solution.
B.8 Proof of Fact 8
Proof. a) At [0, 0, 0, O] the Hessian Equation 24 is just the identity and hence positive.
b) First note that as soon as one off aff-diagonal entry of Equation 24 is larger than one, there is 2 x 2 principal minor that becomes indefinite, thus the Hessian is not positive semidefinite anymore. Thus positivity of the Hessian implics that all off-diagonal entries are less or equal to 1. We can then use Gershgorin disc theorem and obtain that all eigenvalues are belween [-2, 4), hence "max S
C Other Regularizers for 2:4 Sparsity
Our main thcoretical contribution is to show that the non-convex proximal operator be solved efficiently by decomposing it into smaller subproblems. In principle, there also exist simpler regularizers that induce 2:4 sparsity and whose proximal operator can be solved more efficiently. Here we introduce three further proximal operators of simpler regularizers. To be concise, we assume Lemma 3 has alroady been applied, i.c., w1 2 W2 2 Wi 2 w4 2 0 and we define the regularizers
0 if w. W3- = 0,
Ro(wi, W2, W3, wa) =
1 if u4 = 0, W/3 > 0,
(25)
2 if wA > O.
(1(,, W2, uy, wa) = + W33
(26)
Re(wi, W2, wy, wa) = 2 (w] + wg²
(27)
Each of those regularizers equals 0 if and only if the four weights have a 2:4 sparse pattern. So those regularizers do also enforce 2:4 sparsity. Let's look at their proximal operators - again assuming 21 2 22
IN 23
2 24 2 0.
arg min f(w), with WER
(28)
1 f(w) = :1g100 2112 + ARp W22 Wa, wa)-
(29)
Note that the objective of those proximal operators can all be easily be decomposed for each of the variables and trivially wi = 21 and w = z2. The other optimizations are also simple. For Ro we have w = if X >> 2 else 23; for R₁ we have wi = max (23 - X, 0); For R2 we have w = 23/(1+ X). For wi it follows in analogy for all three cases.
Thus, depending on X, Ro forces weights to be zero if they below a threshold (hard thresholding), R does a noft-thresholding, and R2 leads to shrinkage. For large but finite X, Ro and R₁ lead to exact 2 sparsity, whilst R2 does not lead to exact 2:4 sparsity for finite X.
However, those regularizers commit very early to the sparsity pattern, and none of the behavior of Figure are observed for them. Furthermore, on the toy problem Section 4.] they cannot find the optimal solution. We hence focused the main part of this work on the more claborate proximal operator. For completeness we also report perplexity numbers for it in Table 2 as well as the runtime. Since the optimization problems are trivial and have close form solution, they significantly faster, but lead to slightly worse perplexity.
19

---

Published in Transactions on Machine Learning Research (08/2025)
1.010
5500
1.010
2250
1.008
1.008
2000
5450
1750
1.006
loss
1.006
time
5400
1500
1.004
1.004
1250
5350
1000
1.002
1.002
5300
750
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Åσ 10
Āσ 10a
Figure 3: Ablation for the regularizer schedule Equation 8B.
15
on the down projection of layer 18 of Llama-3.1
open llama 3b v2: mlp.down proj
20
1.00
19
Wiki
layer loss vs WandA + GD
0.98
Perplexily
18
0.96
0.94
Validation
0.92
15
0.90
14
prox + GD
0.88
WandA + GD
13
SparseGFT GD
0.86
0
5
10
15
20
25
250
500
750
1000
1250
1500
layer
Local GD teps after pruning
Figure 4: Layerwise local squared loss example matrix of openLlama 3bv2. We show the loss relative to the loss WandA (with masked gradient updates) incurs. As intended by design, the proximal approach of pruning leads to smaller local squared loss. In Table see that this also translates a better end to end performance in terms of perplexity.
Figure 5: Effect of number of GD steps Equation 5 after masking with Wanda on the 13B model. The perplexity converges quickly and the used 1000 steps the main paper suffice drive the local optimiza- tion convergence. These results confirm that the proximal operator identifies more effective mask, as prox achieves lower values of perplexity.
Experiment Details
In Figures and we show further ablations and insights into our main experiments. Table T shows the different runtime of the pruning methods and Table 3 shows results of the pruned models on downstream tasks.
D.1 Using the Hessian of the unpruned Model
For our studics the main paper we compute the Hessian matrix for each lincar layer once before the pruning. This allows prune layers in parallel and distribute the pruning on multiple GPUs, because the local Hessians do not depend each other. The alternative to prune the matrices sequentially and propagate
20

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2:
Wall clock runtime of different pruning approaches on Llama-3.1 8B Instruct. The proposed regularizer leads to the best perplexity, however, also incurs the highest pruning cost. Since this is onco-off effort, and negligibly small compared to pretraining costs, this can be disregarded in production settings.
Runtime Runtime [min]
C4 Perplexity
Wiki Perplexity
proposed regularizer |Section
3.2
11,170
186.17
26.44
20.15
LO regularizer, Equation
25
1,346
22.43
27.25
20.21
L1 regularizer, Equation
26
1,242
20.70
28.46
20.70
L2 regularizer, Equation
27
2,628
43.80
27.84
20.43
wanda
258
4.30
43.30
26.67
wanda+ GD sparsegpt sparsegpl+Gr1 GD
1,000
16.67
28.45
20.71
313
5.22
35.55
26.19
1,064
17.73
27.54
20.85
Table 3: Accuracy Evaluation of pruned versions of Llama-3.1 8B Instruct on 6 Downstream Tasks. The methods (+GD) introduced this work reduce the accuracy gap by around 3% to 5% against previous methods. Note that on GSMSK we see an almost complete breakdown of accuracy. This is because GSMSK is run in generative fashion, where the model can generate many tokens and then provides the final answer in specific format. It is known that such generative tasks much more susceptible to accuracy loss than pure classification lasks, where the answer options are already provided (Dulla el al. 2024). All tasks other
than GSMSK are run classification tasks, which explains that GSMSK is such an outlier.
model
Mean
MMLU
GSMSK
Winogrande Hellaswag TruthfulQA
ai2 are
original wanda
64.28%
67.70%
75.36%
73.79%
59.19%
37.58%
72.04%
35.78%
37.02%
2.50%
62.75%
38.94%
26.19%
47.27%
wonda + GD
40.76%
40.65%
11.83%
65.27%
43.98%
27.42%
55.44%
sparsegpt
37.59%
34.97%
6.07%
64.56%
41.50%
24.60%
53.86%
sparsegpt GD
40.98%
42.94%
8.95%
66.06%
44.74%
27.17%
56.00%
DSnoT 35.01%
36.98%
1.36%
59.75%
36.71%
26.32%
48.96%
prox + GD
40.81%
41.82%
10.01%
65.51%
44.82%
27.29%
55.41%
the pruned inputs. In our ablation Table 4 find however, that it's effect is overall very minor, and it not structurally better than using the unpruned inputs.
Table 4: The effect of propagating the pruning errors is negligible.
Model
Method Hessian
C4
Wiki
dense wanda Original wanda Pruned
9.68
14.15
3b v2
29.49
60.35
28.65
55.98
dense wanda Original
8.84
12.15
7b v2
17.61
27.00
wanda Pruned
17.59
26.88
dense
8.11
11.52
13b
wanda wanda
Original Pruned
13.17
35.39
13.30
37.61
D.2 Dependence on the number calibration samples
In this scction, we provide an ablation on the calibration samples for the 3B model, where we consider the same number of samples as in Sun et al. [2024] Table 17)). After pruning we again evaluate the perplexity the test sets of C4 and Wikitext2. We make three main observations: a/ Wanda. (which uses only the
21

---

Published in Transactions on Machine Learning Research (08/2025)
diagonal of the Hessian) converges quickly known from prior work. b/ Methods that use masked gradient updates can consistently improve their validation perplexity when using more calibration data. However, with diminishing returns (note that the number of samples is increased exponentially here). c/ Except when using only a single sample, our proposed prox+GD outperfors all other methods when using the same amount of calibration data.
Table 5: C4 perplexity after pruning with varying number of calibration samples
calibration samples
16
32
64
128
256
512
1024
2048
wanda
32.11
30.20
29.67
29.70
29.66
29.58
29.57
29.52
29.50
wanda + GD
35.72
21.01
19.51
18.99
18.70
18.50
18.36
18.23
18.16
sparsegpt
557.04
21.30
19.92
19.31
19.23
19.01
18.79
18.80
18.64
sparsogpt + GD prox +GD
302.03
19.77
18.01
17.44
17.12
16.89
16.81
16.74
16.62
70.96
18.85
17.53
17.11
16.73
16.38
16.34
16.28
16.19
Table 6: Wiki perplexity after pruning with varying number of calibration samples
calibration samples
1
16
32
64
128
256
512
1024
2048
Pruning Methods wanda
Wiki
Wiki
Wiki
Wiki
Wiki
Wiki
Wiki
Wiki
Wiki
72.89
63.03
60.16
61.63
61.56
61.51
60.82
60.47
60.54
wanda GD sparsegpt
100.41
40.92
36.43
34.86
33.96
33.71
33.55
33.05
32.88
2,859.53
41.01
35.97
33.92
33.60
35.03
33.83
33.95
33.38
sparsegpt + GD prox +GD
1,675.12
38.06
32.14
30.88
30.13
30.21
30.13
29.56
29.32
149.96
35.80
30.32
29.70
29.19
28.49
28.51
28.58
28.09
D.3
Prox without masked gradient steps
Since our proposed method intrinsically already uses gradient updates, we highly recommend always adding additional masked gradient steps after freezing the mask. This particularly advisable given its low computational cost.
vary the number of gradient steps after masking. Since prox already does gradient updates during the pruning, even without additional updates after freezing the mask, the perplexity is already fairly low. Doing further masked gradient updates monotonically decreases the perplexity (both in- and out- of distribution) until it reaches convergence.
However, to understand the effect of it, in Table 7 also include ablation the perplexity
Table Effect of masked gradient updates for prox on the OpenLlama 3B model.
steps after masking
50
100
200
400
800
1000
1600
16.92
16.70
16.56
16.41
16.31
16.27
16.27
16.26
wiki
30.63
29.99
29.58
29.12
28.84
28
28.62
28.63
22


Reviews:
[{"review_id": "1GyrGVh4Gl", "reviewer": "Reviewer_t4CL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The paper is well-written and easy to follow. \n2. The proposed algorithm is supported by both theory and simulations. \n3. The theoretical part is mathematically rigorous. \n\n\nWeaknesses:\n\n1. I am wondering that why this direction of $N:M$-sparsity is adopted for sparse model pruning. As shown in Table 1, all methods based on $N:M$-sparsity appear significant performance drops (compared to \"dense\").  \n2. The current investigation is tailored to a specific setting of $2:4$-sparsity, which is not intrinsic in the sense of difficulty to flexibly adjust the \"sparsity budget\" $N$. \n3. The time or efficiency evaluation seems lacking. I find runtime reports in Table 2 in the appendix for *only one* model (Llama-3.1 8B Instruct), and no accelerations are observed compared to baseline methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a theoretically grounded solution to the specific setting of sparse model pruning (i.e. the 2:4-sparsity), via introducing a family of regularizers and solving the induced proximal operator. Experiments on large language models validate this method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses above.", "broader_impact_concerns": "There are no significant broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dsmvy3dQOK", "reviewer": "Reviewer_oSSe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts: \n\n* The decomposition of the non-convex proximal operator into three convex sub-problems is interesting.\n\n* The method improves local squared loss and validation perplexity on models up to 13B parameters (and does so using a relatively small calibration set.\n\n* It seems that the method is broadly applicable to off-the self deep learning models. \n\nWeaknesses and Suggestions for Improvement\n1. Use-case explanation is buried: Although the paper applies the method to real models (OpenLLaMA, LLaMA3), the applicability to arbitrary pretrained models is not emphasized enough. A clear subsection explaining how to apply this method to an off-the-shelf model would significantly enhance impact.\n2. Is Algorithm 2 guaranteed to termite after a finite number of iterations?\n    1. How is InvPosSort defined?\n3. Conjecture 9 formatting: The update rule is not properly formatted and looks more like pseudocode than mathematical optimization notation.\n4. Ambiguity in practical scalability: The paper claims the method scales to 70B models but also mentions that computing exact prox solutions 59 million times is infeasible (page 8). Clarify whether the full method (Algorithm 1) is used or whether the GD-based approximation is always employed in practice.\n5. Comparison to structured pruning baselines: The method is primarily compared to unstructured pruning baselines (Wanda, SparseGPT). Including structured baselines like SparseHo or 2:4-aware training from NVIDIA would better contextualize the advantage of this method.\n6. Notation and language polishing:\n    * There are minor typos (e.g., “smootly differentialbe”, “not the the minimum”).\n    * Some expressions such as “dense solution” vs. “3-sparse solution” are introduced abruptly and could be better defined.\n    * Each section should start with an introduction that explains it subsections and how the section builds/serves other sections \n    * There is source code interleaved with natural language in Section 4.1.\n    * Unclear/vague/jargon formulations: \n\t\t“the accuracy drop is often too large”\n\t\t“mostly relevant during prompt encoding (aka prefill)”\n\t\t“one-shot setting”\n\t\t“masked gradient updates”\n\t\t“…allows us to efficiently use the full parallelization of modern GPUs….”\n\t\t“For the proximal, for small models we used…”\n\t\t“we recommend to always use it...\"\n\t\t“…and rather invest resources into having a more balanced and larger calibration sample”", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel proximal gradient method for inducing 2:4 structured sparsity in neural network weights. The key idea is to introduce a specially designed non-convex regularizer that promotes 2:4 sparsity and derive an efficient way to compute the corresponding proximal operator by decomposing it into three convex subproblems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Weaknesses and Suggestions for Improvement\n1. Use-case explanation is buried: Although the paper applies the method to real models (OpenLLaMA, LLaMA3), the applicability to arbitrary pretrained models is not emphasized enough. A clear subsection explaining how to apply this method to an off-the-shelf model would significantly enhance impact.\n2. Is Algorithm 2 guaranteed to termite after a finite number of iterations?\n    1. How is InvPosSort defined?\n3. Conjecture 9 formatting: The update rule is not properly formatted and looks more like pseudocode than mathematical optimization notation.\n4. Ambiguity in practical scalability: The paper claims the method scales to 70B models but also mentions that computing exact prox solutions 59 million times is infeasible (page 8). Clarify whether the full method (Algorithm 1) is used or whether the GD-based approximation is always employed in practice.\n5. Comparison to structured pruning baselines: The method is primarily compared to unstructured pruning baselines (Wanda, SparseGPT). Including structured baselines like SparseHo or 2:4-aware training from NVIDIA would better contextualize the advantage of this method.\n6. Notation and language polishing:\n    * There are minor typos (e.g., “smootly differentialbe”, “not the the minimum”).\n    * Some expressions such as “dense solution” vs. “3-sparse solution” are introduced abruptly and could be better defined.\n    * Each section should start with an introduction that explains it subsections and how the section builds/serves other sections \n    * There is source code interleaved with natural language in Section 4.1.\n    * Unclear/vague/jargon formulations: \n\t\t“the accuracy drop is often too large”\n\t\t“mostly relevant during prompt encoding (aka prefill)”\n\t\t“one-shot setting”\n\t\t“masked gradient updates”\n\t\t“…allows us to efficiently use the full parallelization of modern GPUs….”\n\t\t“For the proximal, for small models we used…”\n\t\t“we recommend to always use it...\"\n\t\t“…and rather invest resources into having a more balanced and larger calibration sample”", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "27Joct2jhN", "reviewer": "Reviewer_ZHFs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper addresses an important and practically relevant problem, particularly in the context of NVIDIA GPUs where structured sparsity (e.g., 2:4) is natively supported.\n2. The proposed method is simple, effective, and theoretically well-motivated.\n\nWeaknesses:\n1. Although the method outperforms existing baselines on language modeling and downstream tasks, there remains a substantial performance gap compared to the dense model. For example, accuracy on GSM8K drops dramatically from 75% to 11.8%, rendering the pruned model nearly unusable in such reasoning-heavy tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a 2:4 structured sparsity method for large language models (LLMs) using a proximal gradient approach. Specifically, it designs a new regularizer tailored for 2:4 sparsity and derives a corresponding proximal operator, enabling a gradual emergence of the pruning mask and avoiding premature decisions seen in heuristic methods like Wanda and SparseGPT. The method is evaluated on OpenLLaMA (3B/7B/13B) and LLaMA3.1 (8B/70B), using 2 million tokens from the C4 dataset for pruning. Experiments on C4, Wikipedia perplexity, and downstream tasks (MMLU, GSM8K) show improvements over Wanda and SparseGPT, though performance still lags behind the dense baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Could you provide any analysis or potential solutions for this kind of performance drop in downstream tasks like GSM8K?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)