The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM
Authors: Kwanhee Lee
Hyeondo Jang
Dongyeop Lee
Dan Alistarh
Namhoon Lee
Venue: ICLR 2026 Poster (2026)
Keywords: sparsity
pruning
LLMs
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We achieve extreme LLM sparsity via surrogate-free constrained optimization.

Abstract: Neural network pruning is a promising technique to mitigate the excessive computational and memory requirements of large language models (LLMs).
Despite its promise, however, progress in this area has diminished, as conventional methods are seemingly unable to surpass moderate sparsity levels (50-60\%) without severely degrading model accuracy.
This work breaks through the current impasse, presenting a principled and effective method called $ \text{Elsa}$, which achieves extreme sparsity levels of up to 90\% while retaining high model fidelity.
This is done by identifying several limitations in current practice, all of which can be traced back to their reliance on a surrogate objective formulation.
$ \text{Elsa}$ tackles this issue directly and effectively via standard and well-established constrained optimization techniques based on ADMM.
Our extensive experiments across a wide range of models and scales show that $ \text{Elsa}$ achieves substantial improvements over existing methods;
e.g., it achieves 7.8$ \times$ less perplexity than the best existing method on LLaMA-2-7B at 90\% sparsity.
Moreover, we show that $ \text{Elsa}$ remains stable even at extreme sparsity (e.g., 95\%), yielding up to $\times$3.98 inference speedup and $\times$7.80 memory compression over its dense counterpart.
We also present $ \text{Elsa}_ {-L}$, a quantized variant that scales to extremely large models (27B), and establish its theoretical convergence guarantees.
These results highlight meaningful progress in advancing the frontier of LLM sparsity, while promising that significant opportunities for further advancement may remain in directions that have so far attracted limited exploration.

Published as a conference paper at ICLR 2026
THE UNSEEN FRONTIER: PUSHING THE LIMITS OF LLM SPARSITY WITH SURROGATE-FREE ADMM
Kwanhee Lee', Hyeondo Jang', Dongyeop Lee', Dan Alistarh', Namhoon Lee!
POSTEEC ²STA (kwanhee Lee, hyeondo. jang, dongyeop. 1ee2, namhoon 1ee / dan.alistarh@ cetttarrhiist.c.... ac.at
ABSTRACT
Neural network pruning is promising technique to mitigate the excessive compu- tational and memory requirements of large language models (LLMs). Despite its promise, however, progress in this area has diminished, as conventional methods are seemingly unable to surpass moderate sparsity levels (50-60%) without severely degrading model accuracy. This work breaks through the current impasse, present- ing principled and effective method called EL.SA, which achieves extreme sparsity levels of up 90% while retaining high model fidelity. This is done by identifying several limitations in current practice, all of which can be traced back to their reliance on surrogate objective formulation. ELSA tackles this issue directly and effectively via standard and well-established constrained optimization techniques based on ADMM. Our extensive experiments across a wide range of models and scales show that ELSA achieves substantial improvements over existing methods; e.g., it achieves 7.8x less perplexity than the best existing method on I.aMM-2-7B at 90% sparsity. Moreover, we show that ELSA remains stable even at extreme sparsity (c.g., 95%). yielding up to x3.98 inference speedup and x7.80 memory compression over its dense counterpart. We also present ELSA. quantized variant that scales to extremely large models (27B), and establish its theoretical convergence guarantees. These results highlight meaningful progress in advancing the frontier of LLM sparsity, while promising that significant opportunities for further advancement may remain in directions that have so far attracted limited exploration.
INTRODUCTION
Large language models (LLMs) have become indispensable tools across various fields, from creative industries to scientific research, but their immense size incurs a tremendous amount of memory, computation, and energy consumption, posing significant challenge to their widespread deployment (Kaplan et al., 2020; Bommasani, 2021; Faiz et al., 2024). Neural network pruning can offer a viable solution this problem by removing redundant parameters without compromising performance (LeCun et al., 1989; Han et al., 2015; Hoefler et al., 2021). Indeed, the research community has responded this challenge with surge of innovative methodologies, demonstrating that LLMs can be made more compact and efficient through effective pruning techniques (Frantar & Alistarh, 2023; Sun al., 2024; Boža, 2024; Meng et al., 2024; Fang et al., 2024; Liu al., 2025; Lee et al., 2025).
However, the community is witnessing major roadblock: current methodologies are failing to push beyond moderate level of sparsity (roughly 50-60%) without significant decline in model performance; for instance, prior works have highlighted this limitation with rather incremental improvements high sparsity (Meng et al., 2024; Boža, 2024; Yin al., 2024a; Huang et al., 2025).
Have we truly reached a plateau, or is there a path to continued progress?
This work provides positive answer. We demonstrate that it is possible prune LLMs for very high sparsity levels- up to almost 9ithttt significant performance degradation (see Figure 1).
The key to our success is identifying and addressing potentially critical flaws in the current practice. Specifically, the majority of existing methods relies on the principle of sequential layerwise recon- struction error minimization, an approach proven effective in memory-constrainee environments.

---

Published as a conference paper at ICLR 2026
300
Dense
Magnitude
Wanda
SparseGPT
Perplexity
200
L-ADMM
ALPS
ELSA
100
Unseen
Frontier
50
90
Sparsity (%)
Figure 1: Perplexity (1) Sparsity (1) curves for different pruning methods; it is measured on the C4 dataset for pruned LLaMA-2-7B models. While existing methods start to fail as sparsity increases, our approach (ELSA) stays stable without losing much performance, revealing the unseen frontier. Previously it was considered nearly impossible to achieve such high sparsity for LLMs or go beyond the "sparsity wall" formed around 50-60% sparsity levels. The same trend is observed consistently across different architectures and scales we will show in Section S-EXPERIMENTS.
However, this approach is inherently prone propagating compounding errors while enforeing unnecessarily strong conditions and, in fact, seeks only local solutions by design based on a surrogate objective (Shin al., 2024; Bai al., 2024; Huang et al., 2025). On the other hand, we suggest finding more globally optimal solutions directly by formulating a causstt-constrained optimization problem and developing a robust solver as whole.
We show that our approach can be applied to a wide range of LLM models and scales from 125M 13B number of parameters. Across this range, ELSA remains stable in the highly sparse regime, whereas competing methods frequently collapse with order-of- magnitude perplexity blow-ups. Im- portantly, these gains translate into practical benefits: we obtain up to 7.80 memory reduction and x3.98 inference speedup without degrading model usability. We provide flexible implementation as well, which incorporates memory-efficient designs including quantized optimizer states and enables pruming even for 27B 77B-parameter models with 55% lower memory footprint, demonstrating extended potential at scale. Based on classic optimization theory, we also provide convergence guarantee for solver ensure theoretical soundness alongside empirical findings.
The full extent of its limits is not yet fully understood. However, our work clearly demonstrates significant potential for further advancements in LLM pruning. We believe that this finding calls for a renewed focus on alternative strategies that more faithfully preserve model fidelity, which could include better ways exchange efficiency for performance, providing practitioners with wider range of options.
PROBLEM STATEMENT
The long-standing research of neural network pruning, aimed at enhancing the efficiency of large models (LeCun et al., 1989; Han al., 2015), has recently made significant progress in its application to LLMs (Frantar & Alistarh, 2023; Sun et al., 2024; Boža, 2024; Liu et al., 2025). While effective, these methods decline sharply and fail maintain performance beyond a moderate level sparsity around 50-60%. For example, the recent study of Zhang et al. (2024) to evaluate these methods report that their performance begins to collapse after 70% sparsity. This deterioration is also evident in other recent works that, notwithstanding the relative advantage over existing methods, the majority still suffer from severely degraded performance high-sparsity regimes, with perplexity often increased more than order of magnitude (Boža, 2024; Meng et al., 2024). In fact, this stands in stark contrast to historical precedents, where extreme sparsity of say 90% or higher was commonly achieved (Frankle & Carbin, 2019; Lee al., 2019). Consequently, researchers has begun theorize the underlying causes, attributing the failure to compounding layer-wise errors and the explosion of reconstruction errrr (Shin al., 2024; Huang et al., 2025).
2

---

Published as a conference paper at ICLR 2026
These findings have collectively fostered a narrative that achieving high sparsity in language models is an illusional goal. We argue, however, that this "sparsity wall" is perhaps not an inherent limitation but rather an artifact of ill-defined problem formulation.
To analyze, let us begin by showing that pruning can be formulated most generally as a constrained optimization problem follows:
- arg min f(x) subject to | < k
where z E' refers to the optimization variable (i.e., parameters of a neural network), f denotes the minimization objective (e.g., tross-entropy loss for next token prediction), and & is the number of parameters to preserve after pruning. I.e., the successful processing of (1) will yield a solution that is sparse and keeps prediction performance.
However, the majority of LLM pruning methods takes an approach of the following form:
x= {a] for i = 1, I.....L) L} where xं = arg min (z;) subject to | < h; (2)
where L refers to the number of some modularized parts of the network model-most typically layers-and ff denotes module wise surrogate objective that measures reconstruction error; precisely, the reconstruction error here is defined to be
i == Ep Ep| 9(x;-1; - -x, (()
where ((x;-1;-) and E denote the activations of the previous layer and the i-th layer of the pre-trained dense model, respectively, and DD refers to some calibration data. Thus, the model is split into submodels, and each submodel is pruned so as to match or reconstruct the predictions of the dense counterpart on some data, sequentially until the last submodel. The solution is then obtained by simply stacking these sparse submodels.
We posit that this approach (2), so-called layer-wise reconstruction error minimization, introduces non-trivial and potentially critical limitations. Specifically, we highlight three potential pitfalls: (i) errors from approximate layer-wise solutions, (ii) suboptimality in model-wide reconstruction, and (iii) the surrogacy in the objective. We elaborate these as below.
First of all, it hard to solve (2) exactly without errors, in other words, the distance (3) cannot be zero realistically. This is due to the high cost of exactly solving sparse linear regression (Natarajan, 1995). In fact, this leads to layer-wise solvers relying on saliency- based heuristics to find approximate solutions (Frantar & Alistarh, 2023; Sun al., 2024; Meng al., 2024). Without zero layer-wise reconstruction errors, even small errors from each layer can compound into large overall errors, which has been observed to pose non-trivial harm to performance (Shin et al., 2024; Huang et al., 2025).
Also, its sequential, layer-wise design is naturally restrictive, potentially introducing suboptimality. By enforcing the layer-wise features match those of pre-trained network, it effectively restricts the search space of the potential solutions, even though no guarantee exists that the optimal sparse model would necessarily respect this requirement. Further concern stems from its independent and sequential nature; the layers are never jointly optimized, and notably, earlier layers will remain fixed even when subsequent layers change regardless of the potential suboptimality it introduces.
Lastly- and perhaps quite fundamentally- its reliance on a surrogate objective f implies that one cannot expect to obtain a solution on (1) even after perfectly solving (2). This stands in direct opposition to the underlying goal of achieving a perfect, zero error solution on (2), whereas, in reality, it may simply lead to overfitting, failing the true objective (1) of preserving the language modeling
capabilitics. We expect these core issues to act as a barrier as we seck higher sparsity levels.
METHOD
We propose ELSA (Extreme LLM sparsity via Surrogate-fre frree ADMM) to directly solve (1), We ground our approach in optimization from both first-principle and advanced techniques in order to better ensure that (1) is properly solved while enhancing effectiveness specifically for LLMs.
3.1 SURROGATE-FREE LLM SPARSIFICATION VIA ADMM
We solve (1) using the alternating direction method of multipliers (ADMM, Boyd et al. (201 strategy involving variable splitting decouple the intractable sparsity constraint S {v Rd
3

---

Published as a conference paper at ICLR 2026
|lu/o < k) from the training objective. This is done by introducing auxiliary variable in the
following manner:
min [(x Ts()) 2,2
s.t.
x =, 2,
where (z) is the indicator function for the set
S:
0
if E
Is(=)
(5)
8 otherwise.
In turn, we keep x constrained to be equal to z. This allows us to handle the model training and the sparsity satisfaction somewhat separated, making both much easier to handle.
To solve for this new formulation, the augmented Lagrangian can be used:
(x 2, f(x) + Ts(() + X +
u 1ul2
(6)
where À is the hyperparameter for adjusting the strength of the proximal penalty, and u is a scaled dual variable. ADMM solves this by alternating between minimizing the augmented Lagrangian over the primal variables (x, 2) and performing a dual ascent step on u. This decomposes the problem into three manageable subproblems that are iterated until convergence:
x arg min
f(x)
2 - x" + u u21
A
2
2 2+1 arg min
>| - 2' + u'11 IIs +1 + +(),
(8)
2
SES
u 1+ u' + x
1+1
t-1
The z-update (7) accounts for minimizing the training objective, and is iteratively minimized while x pushed closer to the sparse The z-update (8) can be expressed the projection IIs(x + w²). Here, the objective associated with its S is simplified to minimizing the Euclidean distance from ---1+u² u², effectively replacing the complex, non-convex with a tractable, convex quadratic function. As result, this has an exact closed-form solution computable by zeroing out the (d - k)-entries with the smallest magnitude (Lee et al., 2025). Finally, the scaled dual variable u updated in (9) to maximize the augmented Lagrangian via single step of gradient ascent.
3.2 OBJECTIVE-AWARE PROJECTION
Closely inspecting the projection step in the z-update (8), one can see that the Euclidean distance is far too removed from f. Thus, it reasonable to expect that the sparse parameters obtained may differ considerably from the actual sparse optima of f.
This motivates us align the projection step with f by modifying its objective into the following quadratic:
arg min (=- (a'+1 v()) H( (wt+1 + u²)),
(10)
26S
where H is the Hessian of f. Equivalently, the second-order geometry of f. Placed once again in the context of pruning research, its advantages would be akin to those of the family of approaches based on the Optimal Brain Surgeon algorithm (LeCun et al., 1989).
project in the H induced norm, aligning the step with
In practice, two approximations are introduced. We notice that the procedural simplicity in the Euclidean case stems from the objective being separable across entries. We found that using Diag (H) allows to retain this simplicity while still keeping the benefits by zeroing the entries with the smallest contribution to the objective rather than by their magnitudes. Also, we employ the Gauss- Newton approximation of the Hessian or the empirical Fisher information matrix F, which allows us to obtain a good approximation of the Hessian only by the outer products of the gradients (Martens, 2020). The results of these can be summarized into the following formula:
++1 arg min
Σ Fu (z. - (x + +z))²
(11)
26S idd
4

---

Published as a conference paper at ICLR 2026
where each coordinate i contributes independently to this new loss function. Luckily, the standard Adam optimizer has already made F available for free via its second-moment estimates, requiring no additional cost in implementing this enhancement (Li et al., 2025). Overall, this tailors our algorithm ELSA to better adapt the complex objective of LLMs, and in a way that incurs negligible additional cost.
3.3 SCALABLE ADMM VIA LOW-PRECISION STATES
We further enhance scalability by proposing L quantization operation, Q, that maps high-precision tensors to compact low-precision representation, and a dequantization operation, R, that rematerializes them.
Here, we rely on two core operations:
Formally, for a high-precision tensor R4, the Q operation produces storable pair (zq:8)
consisting of quantized tensor and a scale:
(2) (Zq. s), where s = max( Pmax and 2q round (z/s).
(12)
Here, Vmnax is the maximum representable absolute value of the target data type (e.g., 127 for signed INT8). Conversely, the R operation rematerializes the figh-precision tensor from the stored pair:
R(5q + Fqg-
(13)
These operations are applied cycle to manage the auxiliary variables. After a update yields an intermediate state, for instance z'+1 [s([x*+1 + w²), it is quantized for efficient storage: (af+1, , +1+
((2 This transition yields substantial memory savings; for instance, storing a state in FP8 (8 bits) reduces the memory footprint by 4x compared to the standard FP 32 representation (32 bits). The overhead from the scale factor negligible, as typically only a single 32-bit scale value is stored for the entire tensor. For the subsequent computation, the state is first rematerialized to high precision: gt+1 R(24+1 1.
This quant-dequant cycle, which bridges low- precision storage with high- precciieeiiion updates via dynamic, data-aware scale, a general and established principle in low- low-precision deep learning (Gholami al., 2022). The specific definitions in (12) can be adapted for various formats, including both 8-bit integers (TNT8) (Jacob et al., 2018) and modern floating-poin types like FP8, representing cornerstone of efficient numerical methods (Micikevicius al., 2022).
However, this introduces nontrivial changes into the algorithm, and thus, the guarantees of ADMM do not automatically extend. We therefore establish a proof to demonstrate that ELSA,,, alongside with ELSA, will converge to the solution of (1) in the following section.
CONVERGENCE ANALYSIS
We establish theoretical convergence for both ELSA and ELSA. to support their reliability in directly solving (1). Formally, we assume the following:
Assumption 4.1. (Lower bounded on constraint) The function f is lower bounded on S. That is, there exists constant smin := minles (a) and J > -tc.
Assumption 4.2. (B-smoothness) The function f differentiable, and its gradient is 8-smooth. That 1- -
is. [[)(()) Vs(v)) <
Assumption 4.3. (μ-weak comvexity) There exists a constant μl 2 0 such that f is p-weakly convex.
i.e., f(z) + 2 ||=] is convex.
Also, we rely on the notion of X-stationarity (Huang et al., 2021):
Definition 4.4. (>-stationary point) We say a point is X-stationary point of the optimization -- (
problem (1) if ī arg minges
ie., the point Vे cannot be locally improved using projected gradient descent with step-size - Given these, we present the convergence of ELSA and ELSA as follows:
Corollary 4.5. (Convergence of ELSA) Suppose that Assumptions 4.1-4.3 hold. Assume further that is chosen large enough so that - 3² (A 2)/2 < Let (ā, ž, a) be a limit point of ELSA algorithm. Then &. is X-stationary point of
5

---

Published as a conference paper at ICLR 2026
OPT-125M
OPT-1.38
Gemma-2-22
Perplexity
200
Perplexity
200
Perpleiity
200
1000
100
100
80
90
50
60
70
80
50
Sparsity
Sparsity
Sparsity (96)
LLaMA-3.2-38
LLaMA-2-78
LLaMA-2-138
300
Perplexity
200
Perplexity
Rerplexity
100
100
100
50
60
70
80
Sparsity (%)
Sparsity (%)
Sparsity (%)
Dense
Wanda
L-ADMM
SAFE
Magnitude
SparseGPT
ALPS
ELSA
Figure 2: Perplexity vs. Sparsity plots for different models and scales. ELSA preserves much lower perplexity at high sparsity compared to other methods, consistently across a wide range of settings, showing its advantage and robustness. All numerical results provided in Appendix C.
Theorem 4.6. (Convergence of ELSA.1) Suppose that Assumptions 4.1-4.3 hold, Also assume that the iterates of ELSA, are bounded, and the constant À and 7y are chosen such that
B2 B(A BY +2 2
(1- (1))))))1)))
-1) <0
A
2
Then, for any limit point (z, 2, ú) of the iterates, T is a Ststttionrry point of (1).
This demonstrates that ELSA and ELSA converge to the stationary point of the sparsity-constrained optimization problem (1). The detailed proof for ELSA provided in Appendix A.
EXPERIMENTS
We present series of concrete experiments validate EL.SA in this section. Specifically, we show that ELSA (i) effectively prunes models to extreme high sparsity levels across a wide range of models and scales (Section 5.1). ) (ii) can further push the limits to extreme sparsity (up to 99%) (Section 5.2), (iii) can accelerate inference while reducing memory requirements (Section 5.3), and (iv) scales efficiently large models up 27B (Section 5.4). Lastly, we analyze the cost of ELSA compared existing methods at (Section 5.5). We also provide extension to other sparsity patterns such as N:M semi-structured sparsity or non non-uniform sparsity, and an ablation study on the choice of objective functions and generalized projection on Appendix C.
We compare EL.SA to the following methods:
Magnitude (Han et al., 2015), SparseGPT (Frantar Alistarh, 2023), Wanda (Sun et al., 2024), ALPS (Meng et al., 2024), L-ADMM (Layer-wise ADMM) (Boža, 2024), SAFE (Lee et al., 2025), and SparseLLM (Bai et al., 2024). These methods are applied to four different architectures including OPT (Zhang et al., 2022), Gemma-2 (Team et al., 2024), and LLaMA-2/3 (Touvron et al., 2023; Grattafiori et al., 2024) across a wide range of scales from 125M to 27B. We report perplexity and zero-shot prediction accuracy of pruned models. All experiment settings can be found in Appendix B. The source code to reproduce results will be made available at https: / //g thub. com/ log--ostecch e els.

---

Published as a conference paper at ICLR 2026
Magnituse
Spanseor LDDMM
SparwLLM
ELS4
Perplexity
10a
New Pareto
Parato
Priar
1:38
Gemma 28
LLaMA 3.2 38
LLaMA 2 28
LLaMA 233
101
10
Number of non-zero parameters (B)
Figure 3: Pareto optimality of ELSA compared to prior works in terms of perplexity vs. number of non-zero parameters. ELSA displays its greater optimality across broad spectrum of effective scales.
Sparsity 70%
Sparsity 80%
Sparsity 90% ARC-C
ARC-C
ARC-C
Winogrande
ARC-E
Winogranda
ARC-E
Winogrande
ARC-E
RTE
RoolQ
RTE
RoolQ
RTE
RoolQ
DBQA
HellaSwag
OBQA
HellaSwag
OBQA
HellaSwag
Magnitude
SperseGPT
ALPS
SparseLUM
Wanda
L-DDMM
SAFE
ELSA
Figure 4: Zero-shot accuracy of pruned LLaMA-2-7B models. ELSA outperforms other methods for most tasks, with the performance gap widening sparsity increases, highlighting its strong generalization capability. Full numerical results are provided in Table of Appendix C.
5.1 MAIN RESULTS
Figure reports C4 perplexity for various models across different sparsity levels from 50% to 90%. Existing methods deteriorate rapidly beyond 70%; for instance, SparseGPT OPT-125M rises from 49.83 at 60% sparsity to over 1,000 at 80%. In contrast, EL.SA remains stable, increasing only from 42.99 to 47.45 over the same range, and 80% sparsity matches the perplexity of SparseGPT at 60%. This robustness holds across scales: on LLaMA-2-13B at 90% sparsity, ELSA achieves 27.84 perplexity, while most existing methods exceed the hundreds. Figure 3 further highlights this trend by plotting perplexity against the effective number of non-zero parameters. ELSA consistently sets the new Pareto frontier across scales, underscoring its robustness in extreme sparsity regimes.
This extends to downstream task performance, as shown in Figure 4. Each radar plot reports per-task accuracy at high sparsity (7090%% with the enclosed area reflecting the average accuracy across tasks. At 70% sparsity, EL.SA is competitive with leading methods, but clear gap emerges as sparsity increases. From 70% to 80% sparsity, other methods lose 10-20%p accuracy on tasks such as Winogrande and ARC-E, while ELSA degrades by less than half as much. At 90%, most methods collapse, whereas ELSA retains the highest accuracy on 6 out of 7 tasks, with an average margin of 6.06%p. This demonstrates that ELSA maintains generalization far better than existing methods at high sparsity. We believe that these results collectively establish the effectiveness ELSA for high sparsity.
7

---

Published as a conference paper at ICLR 2026
Table
Memory savings and inference accelerations of ELSA on LLaMA-2-7B.
Dense
50% %
70%
90%
95%
Latency Tokens/s
1.84
1.37
(x1.34)
0.95
(x 1x1,94)
0.72
(x2.50)
0.46
(x4.00)
54.47
72,81
(x33)
104.89
(x (x1)))
139.47
(x2.56)
216.98
(x3.98)
Memory (MB)
13596
8870
(x 1.53)
5603
(x2.42)
2918
(x4.60)
1743 (x7.80)
5.2 TOWARDS EXTREME SPARSITY
To assess whether ELSA remains effective under extreme spar- sity, we further evaluate LLaMA 2-7B 95% and 99% sparsity, We additionally consider retraining after pruning with Wanda (Sun al., 2024) for the baseline, and the experimental details can be found at Appendix B.3.
Table 2: Perplexity on Wiki/C4 at extreme sparsity.
Sparsity
Method
Wiki
C4
Wanda LoRA
92.66
65.56
As shown in Table 2, ELSA consistently achieves substan- tially lower perplexity on both WikiText and C4 dataset, and the margin increases as sparsity becomes more extreme, No- tably, at 99% sparsity, ELSA remains stable (55.94/40.10) whereas Wanda+full fine-tuning degrades (146.37/71.64) and Wanda+LoRA collapses (588.3/247.5) These results show that ELSA remains stable even in this extreme regime, while simple heuristies fail to preserve performance, highlighting the advantage of ELSA 's more principled approach.
Wanda Full
42.40
34.87
ELSA
26.97
23.14
Wanda LoRA
371.0
143.0
0.95
Wanda Full
84.30
53.62
ELSA
38.91
28.39
Wandu LoRA
588.3
247.5
0.99
Wanda Full
146.37
71.64
ELSA
55.94
40.10
5.3 REALIZING BENEFITS WITH EXTREME SPARSITY
Sparsity is practically meaningful only if yields real deployment gains beyond reducing parameter count. To quantify this, we benchmark end-to-end text generation on LLaMA 2-7B using MACKO (Macko & Boža. 2025), recent Sparse- Matrrx Vector multiplication (SpMV) kernel combined with memory-efficient MACKO format, supporting an acceleration and memory savings for sparse models. Following MACKO's 's standard end-to-end protocol, we convert the sparse models produced by ELSA and evaluate on single NVIDIA RTX 3090, reporting mean latency of token generation (s), throughput (tokens/s), and memory footprint (MB).
Table 1 shows that sparsity yields clear gains in both speed and memory, with improvements becoming pronounced from 70% sparsity onward. At 90% sparsity, ELSA achieves 2.50 reduction in end-to- end latency and 2.56x increase in throughput, together with a 4.60x memory reduction compared to the dense baseline. Pushing further to 95% sparsity provides even stronger efficiency gains (up to 4.00x latency reduction, 3.98x throughput increase, and 7.80x memory reduction). These results validate that the high/extreme-spaan sppasrstte regime targeted by ELSA is not merely an academic frontier: it enables tangible acceleration and substantial memory savings in cemory-constrainee settings, particularly benefiting the decoding phase where sparse matrix-vector computation dominates.
5.4 SCALING TO LARGE-R MODELS
We further validate the scalability of our approach by apply- ing ELSA,, to 27B-scale (Gemma-2-27B)) Specifically, we additionally employ the low-precision optimizer adam8bi (Dettmers et al., 2022) for r-update Equation (7), with ELSA where we use (bf16, 6, fp8) for auxiliary state (u, z), respec- tively. This design reduces the memory footprint of re- quired states (optimizer, auxiliary variables) by 55% com- pared to ELSA, enabling pruning 27B scale under limited resources. Additional implementation details can be found in Appendix B.4.
4.20e+322
Magnitude
Wanda
Perplexity
SparseGPT
1.P
L-DMM
ELSA,
10f
228.5
152.20
25.47
Magohuule warde
SparseGrr
L-DMM
Figure 5: Perplexity of Gemma-2- 27B at 90% sparsity.
8

---

Published as a conference paper at ICLR 2026
Table 3: Compute cost vs. perplexity of different pruners on LLaMA-2-7B at 90% sparsity,
Method
Wall-clock (h) #GPUs
Wiki
C4
Wanda
0.159
2.0 10*
1.0 10*
SparseGPT
0.251
1430
864.5
ALPS
12.57
248
180.9
Wanda+LoRA
4.03
92.66
65.56
Wanda+Full
1.64
42,40
34.87
ELSA
1.78
4
26.97
23.14
Figure 5 demonstrates that ELSA achieves the lowest perplexity among all compared methods, outperforming the strongest competing method by factor of 4x, supporting our main results at scale.
5.5 COST ANALYSIS
To assess effective efficiency (i.e., quality achieved per pruning compute at fixed sparsity), we measure end-to-end pruning cost on identical NVIDIA A100-80GB GPUs and report GPU-hours together with perplexity for LLaMA-2-7B at 90% sparsity (Table 3). One-shot methods (Wanda, SparseGPT) are cheapest (0. 160.25 hours with single GPU) but collapse this sparsity level, yielding unusable perplexities (> 10*). The best-performing baseline ALPS improves perplexity but still remains far from usable, while requiring substantial wall-clock time (12.57 hours). Overall, these layer-wise pruners are memory- by design, yet additional pruning compute does not reliably translate into commensurate quality gains under high sparsity.
In contrast, ELSA achieves substantially lower perplexity (26.97/23.14) 14) with moderate compute budget (7.12 GPU-hours), yielding markedly better cost- cost-quulity point in this regime. Moreover, even when augmented with matched-budget retraining Waadd+LLoRA/Ful one-shot+reerain baselines do not reach comparable quality, indicating that additional compute is more effectively converted into model quality by ELSA high sparsity.
DISCUSSION
In this work, we confront the problem of moderate sparsity in LLMs through critical inspection into the current practice, revealing that the prevailing reliance on the sequential layer-wire wise reconstruction surrogate may have been constraining the path toward more extreme sparsities. This led us develop ELSA and ELSA.,, enabling us to push the sparsity from 50-70% up to 80-90% while maintaining strong language modeling performance, where also observe tangible deployment benefits such as inference acceleration and memory savings, and further showing that our approach remains effective in extreme regimes (e.g., 95-99%). Grounding principle effectively solves the true I.LM objective as is, while also facilitating the development of advanced techniques that are both theoretically sound and effective for sparsifying LLMs, which believe were instrumental in attaining strong practical results.
optimization principles ensures that
Meanwhile, remark the memory demands associated with pruning LLMs. In particular, propose to reassess the widespread assumption that, given the limitations of commodity memory, the adoption of layer-wise surrogate strategy difficult cireumvent. First of all, worth questioning whether the underlying assumption itself is too restrictive -fter all, one would not typically attempt to prune an L.M without least the resources required run one. Also, we raise doubts about whether the layer-wise strategy provides clear memory advantages. Precisely, using the offloading technique allows one to optimize over the entire model with similar memory efficiency. In fact, quite the opposite may be the case case-they do not scale well with the size of calibration data, requiring the layer activations of the entire calibration data to be stored, while a single mini-batch usually suffices the surrogate- free principle. This calls into question whether our perception of its efficiency could be somewhat inflated, requiring the need for a careful assessment current practice and exploration of alternative strategies through more balanced lens.

---

Published as a conference paper at ICLR 2026
There many promising directions to pursue for future work: (i) alternative efficiency strategies through advanced memory-efficient and derivative-Fis optimizers, (ii) system-level advancements in memory offloading, and (iii) extensions advanced architecture such as coxture-of-Expere (Mu & Lin, 2025) and multi-modal large language models (Yin et al., 2024b). To conclude, our work validates that the frontier of LLM sparsity still be expanded by offering concrete strategy supported by strong empirical evidence. We hope it sets the stage for future breakthroughs and innovations in new directions that have thus far received relatively limited attention.
ACKNOWLEDGEMENT
This work was partly supported by the Institute of Information & communications Technology Planning & Evaluation (ITT)) grant funded by the Korean government (MSIT) ASS-2019-11199900
Artificial Intelligence Graduate School Program learning of Causal Inference in Vision and Language for Decision Making), the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) ( RS-2023-00210466. RS-2025-02264052).
(part2) Few-Shot
REFERENCES
Guangji Bai, Yijiang Li, Chen Ling, Kibaek Kim, and Liang Zhao. Sparsellm: Towards global pruning of pre-trained language models. NeurIPS, 2024.
Rishi Bommasani. On the opportunities and risks of foundation models. D:Xiv:2/08.072550 2021.
arXiv preprint
Stephen Boyd, Neal Parikh, Eric Chu, Borja Pelcato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine learning, 2011.
Vladimír Boža. Fast and effective weight update for pruned large language models. Transactions on Machine Learning Research, 2024.
Christopher Clark, Kenton Lee, Ming-Wos Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: arXiv: 1803.05457, 2018.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. 2022.
Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Chukwunyere Osi, Prateek Sharma, Fan Chen, and Lei Jiang. LImcarbon: Modeling the end-to-end carbon footprint large language models. ICLR, 2024.
Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnable semi-structured sparsity for large language models. NeurIPS, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2019.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. ICML, 2023.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey quantization methods for efficient neural network inference. In Low- Low-power computer vision, pp. 291-326. Chapman and Hall/CRC, 2022.
10

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv, 2024.
Song Han, Jeff Pool, John Tran, and William Dally, Learning both weights and connections for efficient neural network. NeurIPS, 2015.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241): 1-1224 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
Tianjian Huang, Prajwal Singhania, Maziar Sanjabi, Pabitra Mitra, and Meisam Razaviyayn. Alter- nating direction method of multipliers for quantization. A/STATS, 2021.
Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Fei Chao, and Rongrong Ji. Determining layer-wise sparsity for large language models through theoretical perspective. ICML, 2025.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anleger-arihhmetic-ollly inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wxXi:2201.08361,, 2020.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. NeurlPS, 1989.
Dongyeop Lee, Kwanhee Lee, Jinseok Chung, and Namhoon Lee. SAFE: Finding sparse and flat minima to improve pruning. ICML, 2025.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. ICLR, 2019.
YuXin Li, Felix Dangel, Derek Tam, and Colin Raffel. Fishers for free? approximating the fisher information matrix by recyeling the squared gradient accumulator. In Proceedings of the 2025 International Conference on Machine Learning (ICML), Jul 2025. doi: 10.485500/r2iv.2557.. 18807.
URL httph:t///// arxiv. org/ ab5/ 18807. Poster.
Hongyi Liu, Rajarshi Saha, Zhen Jia, Youngsuk Park, Jiaji Huang, Shoham Sabach, Yu-Xiang Wang, and George Karypis. Proxsparse: Regularized learning of semi-structured sparsity masks for pretrained 1lms. ICML, 2025.
Vladimír Macko and Vladimír Boža. Macko: Sparse matrix-vector multiplication for low sparsity. arXiv preprint arXiv:25/1.1306/ 2025.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21(146):1-76, 2020.
Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. Alps: Improved optimization for highly sparse one-shot pruning for large language models. NewrIPS, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. ICLR, 2017.
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenth- waite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.0543: 2022.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. EMNLP, 2018
11

---

Published as a conference paper at ICLR 2026
Siyuan Mu and Sen Lin. A comprchensive survey of mixture-of-experss: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137, 2025.
B.K. Natarajan. Sparse approximate solutions to linear systems. SIAM Journal on Computing, 24(2):
227-234, 1995.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
Sungbin Shin, Wonpyo Park, Jaeho Lee, and Namhoon Lee. Rethinking pruning large language models: Benefits and pitfalls of reconstruction error minimization. EMNLP, 2024.
Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, and Dan Alistarh. Evopress: Towards optimal dynamic model compression via evolutionary search. arXiv preprint wrXiv:2410116644 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. ICLR, 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprint 2rXiv:2408.00/
2024.
torchao. Torchao: aetorch-native waining-to-serving model optimization, oct 2024. httt:::: //github. wwwmpytteeeeoooooooooooooooooo.o ancyytrrchaaa aoo.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Lu Yin, You Wu, Zhenyu Zhang. Cheng-Y Yu Hsieh, Yaqing Wang. Yiling Jia, Gen Li, AJAY KUMAR JAISWAL, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. ICML, 2024a.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12):wwae033, 2024b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? ACI., 2019.
Wenyuan Zeng and Raquel Urtasun. Miprune: Multi-layer pruning for automated neural network compression. arXiv, 2018
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0000 2022.
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse L.MMs. ICLR, 2024.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint ar0iv:2304. /277, 2023.
12

---

Published as a conference paper at ICLR 2026
A PROOF OF THEOREM 4.6
Here we present the convergence proof of ELSAL. Formally, we prove the convergence of the following algorithm:
Algorithm ELSA.L
1: Input: Constant X >> 0; initial points To, uo E Rd
2: for r 1,,,..... do
3: Update y: 1ls(2' + -='=²")
4: Update x: by finding point r²+1 satisfying Vf(z++1) + Q(u + A(x - zf-1)]=0 and
| x & mmn {|x* - 24+1| || x'1}
5:
Update u: 4+1 = Qlu + A(x - 2
6: end for
First, let define:
e VaL(x', , ,
(14)
Vf(x') + -xt=1 + a(x² 2t
(15)
w'-l + N(x² - =²) - Q(u + A(x² - ==)]
(16)
Thus, can express the step in terms of el follows
u ++1 Q[u A(x - 2
(17)
u' + X(x z 2 e
(18)
Lemma A.1. Due to (A - -u--trong convexity and (3 + -ssmmoothssss of C(, ,0,"" 1 we know that
A-D)1e² - x'1 & |le'll & (A + 3]1|2² x'1l
(19)
Moreover, due to strong convexity we also know that:
xe',x'- - x².> - 2 (A - (A-H)]la* ø= - 12112
(20)
Lemma A.2. If 3 3 and we also assume that the iterates x' stay bounded. Then there exists non-negative number D s.t. |x - z11 & D. With this definition,
2(x',2',t 2²,u²) > Smin 7(A + 32
(21)
Proof. Note that
L(x', 2('',2,,') = f(x') (u', x² - -2²) 2212
(22)
= =f(x)) (V)(2x), xt - x²) - "|a" - 22112+(e +' -
(23)
3/({))
> f(z') | |e'"||='' 2'1l
(24)
> fmin 7A + D2
(25)
where the last inequality is due to the assumptions and Lemma A.I.
Now let us prove sufficient decrease C in each iteration.
Lemma A.3. Let the assumptions of Lemma A.2 be true. Also, assume that the parameters X and ? are chosen such that
82
3(++ Byy + (2+
(1- (1-))2(X-) 14) 0.
(26)
X
X
2
Note that X p 2 O. Then, we have
lim 70
| x² =
(27)
13

---

Published as a conference paper at ICLR 2026
Proof. Let 2(x
2t1, ,uttt [(x", ,',")'',') = D(x z , ,u(+1) L(xlt1 , uu)
(4)
+ L(x ,zth ,2)) [("2, xz)=²)
(B)
We want to show that + (B) & O.
=> - w²l2 + (eetl (ett), - -w))))
(A) = (u'+1 11 , =1 (u', 1 -(tt) = x-1
Using our definitions, we have
(A) -- 1 u112 + (el+1, ,u wi)
(28)
---
[ff(*+11 vf(x')]| + ( V vf(2'))
(29)
< ---
VV [((')|| + | (f(( vf(x'11)
(30)
E ---
8 11 2/12 3||e+1|||0 n'1)
(31)
< 1---
8 2112 3(A 3)||=(+1 +1 a²1
(32)
& -
11 2²12 3(A + 3)7111011 - x. 2¹12
(33)
=1 8 (A + /77) | 2²122
(34)
where the last inequality is due to Lemma A.1 and the way xf is chosen in Algorithm 1.
On the other hand:
(B) L(at+1, 2t11, u²) L(x',z',ul z²,u²) L(x 242 u²) L(x 2242, u²) L(x 240,, [(x 24,
2t,20 ut))
ED (due update of
L(x 2041 , - 1(xx,22t1 u²)
[xttt1 2t41, uu² - [(xt11 2/41 u²) + [(xtt1 2t11 1 ,,)),²) L(x", 2111 w²)
< x 12-
z (1 1oft1 10st1-2011 -21
8 ++
() - p))
x + 112
1+1 $12
2
x² and | ++ the update rules of Algorithm 1. Plugging in these, get
Now note that ||o"
x. 1+1
> (1 - >>)
x1 $ 71100 - x' because of
(+ 8) (1 7) 14) || |1a0t11110 ---41-000 x212
(B)
(35)
2
Now combining the inequalities for (A) and (B), we have
L(x 2!+1 11-1,=+11 x L(x", 2², u²)
(36)
82 3(A + 8)7 72 + B)
(1 - 7) - p))
1 2012
(37)
π
2
2
14

---

Published as a conference paper at ICLR 2026
Now for any T:
fmi - -(A+ B)D2 & [(x 2-+1 , +1)
g+1
(38)
L(x 12 u²) + Σ C(x't) 1 t+1 22,,,,, ,u
f(x' 2(x',"",u)
(39)
1=0
1 1²12 L(x 20.0))00)))
(40)
Now if the parameters are chosen appropriately such that O, then the right hand side of the above inequality is decreasing as T increases, while the left hand side is constant. Therefore, we have limr- Li || x²122 co. Thus, lim,- [x++1 x'1 = 0.
Theorem A.4. Assume that all the assumptions of Lemma A.3 is satisfied. Then, For any limit point (a, 2, X) of the Algorithm 1, z is X-stationary solution of the problem.
Proof. Consider a sub-sequence (x"', ,,,,,"))),, for t for
= 0,... which converges to (i, 2, ū). First of
all due Lemma A.3, we know that lime-co |x*c+1 - x*1 0 and lim,-ro | x || 0. Thus,
lim rit & lim xt
(41)
Moreover, due to the updates of the algorithm
lim | 1-
x- < lim , -=11=0 & lim ||0 x1'
lim 7/0" 1-x0
x
-x
(42)
Thus, lim e¹ lim e 0, which means
lim u
lim (VI(x")) ce') = -V((T)
(43)
(+0
lim u ret
lim [ - (F)
(44)
1-0
Thus, lim u u.
Also, as finite, there exists a large enough T. such that n" 7 for > T. Again due to the fact that S is finite, assume that these two conditions hold, i.e. 2t = ÿ and grc+1 = i for all for an appropriately refined sub-sequence. This means that
re-fine the sub-sequence such that g*++1 ÿ. Thus, without loss of generality
2 arg min | (x" + X x
(45)
Moreover, u=t11 = tht + Max+11 0). Taking the lim from both sides, we get
(46)
Combining the above with Equation 45 we can casily see that
(x" A - 2 & ||a (x" - 0,... .N
(47)
Taking the limits lim; from both hand sides of the inequality for all the points as we have
|aa (a x x= 0, N.
(48)
Thus,
E arg min
| -- -
(49)
S
where we used the fact that ū (i).
15

---

Published as a conference paper at ICLR 2026
Table 4: Global hyperparameters of ELSA shared across all models.
Hyperparameter
Value
LR schedule
Linear decay
Interval
32
Adam (ft, (8)
(0.9, 0,999)
Total datapoints
32768
Table 5: Learning rate (1), penalty (A), and penalty schedule configuration across models different sparsity levels.
Spursity
schee.
OPT-125M
OPT-1.38
Gemmnn
LLaMA-3.2.31
LLaMA-2-7B
LLaMA-2-1BR
50%
le.4
Se-5
Se.5
le-4
60%
2e-4
le-4
1o-4
2e-5
7
70%
le-4
5e-5
Se-5
5o-5
Se-5
80%
2c-4
1c-4
le-4
90%
50%
constant
le-4
le:3
2e-1
le:3
le:3
2e:3
60%
constant
le.3
2e-3
Se.3
le:3
70%
5e-3
2e-2
le:3
le:3
le:3
90%
cosine
Se-4
le.3
2e.3
Table 6: Batch size (BS) and number of steps used different sparsity levels.
Model(s)
50-60%
70-90 %
BS
Steps
BS
Steps
OPT-125M
2048
8
4096
OPT-LBB, Gemma-2-28. LLaMA-3.2-3B, LLaMA-2-7B, LLaMA-2-138
1024
B EXPERIMENTAL DETAILS
B.1 IMPLEMENTATION AND REPRODUCTION DETAILS
Our implementation is based on PyTorch (Paszke et al., 2019). using the HuggingFace trans formers and datasets libraries for model and data loading. ELSA is implemented over HuggingFace Tra Trainer, supporting distributed training via PyTorch FSDP-2 (Zhao et al., 2023) with HuggingFace Accelerate.
All experimental results in this work are obtained with unified codebase, while baseline methods are reproduced using their original implementations whenever available. The environment configuration (dependencies, versions, and training scripts) can be found at https:// com/ woom/oog-pose log--psttec/eeeee
Experiments are conducted on NVIDIA A100/H200 GPUs, with the number of GPUs scaled to model size: 2x GPUs for 1.3B-3B models, 4 A000 GPUs for 7B models, and 4x H200 GPUs for 13B and 27B models.
B.2 DETAILS FOR SECTION 5.1
Calibration/Trainnn data. To obtain baseline results (Wanda, SparseGPT, ALPS, L-ADMM, SAFE, SparseLLM), we follow the convention of Frantar Alistarh (2023), sampling 128 calibration sequences from the C4 dataset with sequence length 2048. For ELSA, adopt the same strategy, but use larger calibration sets to account for the iterative nature of our optimization.
Training details. We train ELSA with 32,768 data points, where each data point has sequence length of 2048. Batch size and number of steps differ across model, which is presented table Table 6. We use Adam as the base optimizer. The penalty parameter kept constant for moderate sparsity levels (50-60%), while cosine schedule, which gradually increases the penalty parameter from
16

---

Published as a conference paper at ICLR 2026
O at the start to À at the end of training. All model parameters and optimizer states uses full precision for training (except for memory-efficiet setting and ablations), and automatic mixed precision with bfl6 precision is used for efficient training. A full list of hyperparameter configurations is provided in Tables and 5.
Evaluation. Perplexity measured on the held-out (validation) C4 (Raffel et al., 2020) and Wiki- tex12 (Merity et al., 2017) datasets. Zero-shot performance is evaluated with 1m-eva harnssss across seven standard tasks: ERC-Fasy/Challenge (ARC-E/C) (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), OpenBookQA (OBQA) (Mihaylov et al., 2018), RTE (Zeng & Urtasun, 2018), and Winogrande (Sakaguchi al., 2021), and we report the average accuracy as in Section 5.1.
Ln-eval-harness
B.3 DETAILS FOR SECTION 5.2
For the baseline (Wanda+L LoRA Full), we first prune the pretrained model with Wanda, and then retrain the remaining parameters using either LoRA (Hu al., 2022) or full fine-tuning (Full). For a fair comparison, we use the same number data points with same sequence length as ELSA, with hyperparameters tuned separately for each method.
B.4 DETAILS FOR SECTION 5.4
We ran ELSA.1 Gemma-2-277 using H200 GPUs. Fp8 representations for ADMM states (u, 2) were implemented based on the torchao framework(torchao, 2024), where we further extended the implementation to fully support DTenssr,, required by the FSDP-2 framework for distributed training. For this setting, we used a learning rate of r = 2 x 10-6 and penalty parameter 0.002, using cosine penalty scheduling.
B.5 DETAILS FOR APPENDIX C.I
For N:M semi-structured sparsity, we use the same hyperparameter configuration as for 50% unstruc- tured sparsity.
For non-uniform sparsity comparisons, we evaluate ELSA on LLaMA-3-8B using the hyperparameters of ILLaMA--7B 70% sparsity, while the results of SparseGPT, OWI, and EvoPress are taken directly from Sieberling al. (2024), For ELSA (EvoPress), adopt the non-uniform sparsity configurations provided the official EvoPress repository, and initialize ELSA with these sparsity budgets while keeping the same training hyperparameters.
B.6 DETAILS FOR APPENDIX C.2
For objective ablation, we used the OPT-125M model at 90% sparsity, fixing the total number of optimization steps to 4,096 and varying the data count from 256 up 32,684, using the same hyperparameter configurations as in Table 5.
C ADDITIONAL RESULTS
Here we provide additional results for different sparsity patterns (Appendix C.I), ablation studies on objective function and projection step (Appendix C.2), and numerical results used to make visual plots in the main text followed by additional result reporting LLaMA-2-133 13B zero-shot task accuracy.
17

---

Published as a conference paper at ICLR 2026
Table 8: Perplexity and zero-shot prediction accuracy of LLaMA-2-7B under N:M semi-structured sparsity. ELSA compares competitively to other methods, demonstrating its adaptivity. Note that 2:4 and 4:8 patterns are only 50% sparsity levels.
Perplexity (L)
Tasks
Sparsity
Method
Wiki
ARC-C
ARC-E
BoolQ
HellaSwag
OBQA
Winogrande
Avg.
0%
Dense
5.47
7.26
43,35
76.26
77.68
57.14
31.40
62.82
69.06
59.67
Magnitude
37.76
74.66
30,12
61.87
59.85
43.45
21.80
52.35
61.01
47.49
Wanda
12.13
15.63
30.46
61.83
68.
41.28
24.20
53.07
62.31
48.80
SparseGPT
64.06
67.61
43.47
24.20
56.32
66.38
L-ADMM
10.19
12.51
32.85
66.04
68.81
45.05
25.40
56.32
66.38
51.55
2:4
ALPS
9,945
12.09
34,47
68.86
73.79
49.40
27.60
55.60
67.25
53.85
SAFE
9,914
12.53
30.46
63.43
66.42
44.66
21.60
53.07
61.80
48.78
SparseLLM
11.29
13.95
30.55
61.91
71.10
43.62
24:40
$7.40
65.82
50.69
66.36
Magnitude
15.91
31.60
36.01
64.81
63,09
50.05
26.00
$2.35
62.19
50.64
Wanda
34.47
67.05
72.87
46.98
26.80
54.15
SparseGPT
8,508
10.81
34,81
68,56
71.77
48.26
27.80
$6.68
68.11
53.71
4:8
L-ADMM
8.12
10.37
35.58
68.18
72.48
49.45
28.80
58.12
67.17
54.25
ALPS
8,103
10.29
33,28
65.19
68.75
45.96
26.20
55.96
65.98
51.62
SAFE
8,043
10.47
31.57
66.84
68.04
48.55
23.40
53.07
65.04
50.93
SpurseLLM
75.14
ELSA
9.20
11.47
32.25
64.69
69.42
49.90
27.40
$3.07
63.22
$1.42
C.1 OTHER SPARSITY PATTERNS
In this section, we analyze whether ELSA can adapt to other sparsity patterns including (i) N:M semi-structured sparsity and (ii) non-uniform sparsity over different layers.
Table 7: Perplexity of LLaMA-3- 8B 70% sparsity. ELSA outper- forms prior allocation methods.
We first evaluate ELSA for its adaptivity to N:M strisutructured sparsity, a setting designed for some current hardwares to ac- celerate computations (Sun al., 2024; Fang al., 2024). The results of both perplexity and zero-shot prediction accuracy are reported in Table 8. ELSA roughly on par with exist- ing methods, and yet, it is noteworthy that these 2:4 and 4:8 sparsity patterns only ensure 50% sparsity. More importantly, these results indicate that ELSA can easily adapt to arbitrary constraints moderate sparsity levels without much trouble.
Miki(1)
C4(1)
SparseGPT
85 84
98.35
OWL
48.07
52.32
EvoPress
ELSA (EvoPress)
26.11
29.33
ELSA
24.97
29.09
We also compare ELSA with non-uniform sparsity allocation based pruning methods. Specifically, we compare OWL (Yin et al., 2024a) that allocates sparsity based on outlier distributions and to EvoPress (Sieberling et al., 2024) that an evolutionary search strategy to determine the non-unine non-uniform sparsity levels over different layers. We further set up a method that overrides ELSA with the mask found by the evolutionary strategy of EvoPress. Note that the sparsity level is set to be 70%; it is simply because these methods only works or reports up to this level. The results presented in Table 7. One can that ELSA substantially outperforms OWL and shows an improvement over EvoPress as well: to elaborate, for instance, it achieves the C4 perplexity of 29.09, compared to 33.7 72 for EvoPress and 52.32 for OWL. Notably, adopting the non con-uniform mask found by EvoPress within ELSA yields some gains over the EvoPress itself, but it still falls short of the uniform allocation in ELSA, demonstrating the strength of our surrogate-fre free global formulation.
18

---

Published as a conference paper at ICLR 2026
C.2 ABLATIONS
In this section, we present two ablation analyses on (i) the choice of objective comparing the next token prediction (NTP) against the reconstruction error minimization (REM), and (ii) the projection step contrasting our objective-aware variant with the standard projection method.
Specifically, first up experiment where measure how effectiveeyy our surrogate-free approach with NTP make use of data to preserve the original model performance while vayring the number of data samples. We compare that to the existing REM approach.
REM
10
Rerptesity
The results are plotted Figure 6. While REM tend to perform better than NTP at low data regime, but soon starts to saturate data counts increases producing diminishing returns. This stark constrat NTP by which pruning performance keeps improving quite drastically with more data. Notably, REM requires memory to store dense model predictions, which can grow prohibitively large with large data. By contrast, NTP naturally benefits from additional data and continues to improve, enabling scalable LLM sparsity. This in part reveals the inherent limitation of surrogate objectives.
10a
Number Samples
Figure 6: Effect of NTP on data efficiency and perplexity.
We also evaluate the effectiveness of the objective-aware projection on high-sparsity regimes. Specifically, we measure the perplexity of LLaMA-3.2-33 model pruned for 70-90% sparsity levels by turning on and off of the projection and report the results in Table 9. The projection grows with sparsity: perplexity gap increases from 1.20 at 70% sparsity to 2.56 at 80%, and widens further at 90%. This demonstrates that incorporating objective-aware importance into the projection step can be beneficial particularly in high sparsity regimes.
Table 9: Effectiveness of geo- metric projection (V).
benefit of objective-
Sparsity
X
29.44
28.24
80%
40.06
37.50
90%
65.41
48.69
C.3 NUMERICAL RESULTS
Here we provide numerical results used to produce visual figures on the main text, followed by additional results reporting LLaMA-2-7B/13B zero-shot task accuracy.
19

---

Published as a conference paper at ICLR 2026
Table 10: Perplexity (L) of various models pruned with different methods across sparsity levels. Dense performance is shown under each model name (Wiki C4). Results for SparseLLM on Gemma-2-2B and LLaMA-2-13B are omitted due to implementation limitations (e.g., architectural incompatibility. out-of-memory errors). We could not obtain results of SparseLLM in Gemma-2-20. Llama-3.2-31 Llama-2-13B.
50%
60%
70%
80%
90%
Model
Method
Wiki
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
C4
Magnitude
193.4
141.0
920.0
598.2
3806
2263
4890
3213
6613
4475
Wanda
38.93
34.91
77.85
63.33
351.8
248.9
1912
1066
4940
3126
SparseGPT
37.02
33.51
60.90
49.83
239.2
156.3
2072
1050
6131
2443
OPT-125M
L-ADMM
33.02
31.21
45.04
38.49
100.5
74.61
580.8
315.8
3427
1350
(Desse: 26.56)
ALPS
32,70
30.91
43.07
36.94
90.85
66.28
484.8
267.7
2524
1094
SAFE
33.88
30.54
47.21
37.46
120.1
75.2
1254
726.8
5382
2331
SparseLLM
37.11
33.19
57.47
46.64
199.2
131.7
1576
752.2
4730
1825
ELSA
37.68
41.5
33.22
49.57
39.86
65.30
47.74
95.33
62.28
Magnitude
1712
403.3
9392
5066
9442
6498
1.6e4
1.1e4
2.9e4
1.8e4
Wanda
18.42
28.77
105.7
94.98
2504
1.3e4
8447
SparseGPT
17.45
19.25
24.02
23.30
50.52
46.11
947.9
406.7
6472
2843
OPT-1.3B
L-ADMM
26.62
26.26
32.35
30,28
61.10
49.52
595.9
289.5
56599
2298
(Dess: 14.52/ 16.07)
ALPS
16.78
18.59
20.58
21.552
35.77
34.09
285.7
158.4
45%0
1844
SAFE
16.38
17.75
19.63
19.93
31.17
27.52
387.1 1
222.3
1.3e4
7544
SparseLI.M
17.73
19.40
23.23
24.03
56.36
47.96
861.7
372.0
5535
2217
ELSA
18.99
18.45
22.11
20.2
27.13
24.43
36.89
31.51
61.52
45.39
Magnitude
51.66
57.68
2178
2064
4.4c7
3.5e6
2.5e9
2.4e8
5.0e9
2.3e9
Wanda
12.07
17.49
21.39
32.40
117.5
152.0
994.6
855.6
1.1c4
5524
SparseGPT
11.58
16.67
16.53
23.44
34.73
47.43
147.7
160.5
983.1
776.5
Gemma-2-2B
L-ADMM
11.02
15.84
14.65
20.84
26.91
38.32
86.64
110.8
308.2
300.3
(Dene:: 8371/13.16)
ALPS
10.93
15.77
14.42
20.32
24.96
35.08
73.50
238.5
254.3
SAFE
11.61
16.21
15.22
20.32
25.67
33.39
68.55
75.22
432.7
345.0
SparseLLM
ELSA
13.05
17.22
15.93
19.83
21.22
24.55
30.29
31,68
49.37
44.93
Magnitude
139.4
216.1
1.5e4
1.4e4
1.0c5
1.
3.Se5
3.5e5
3.0e5
2.4e5
Wanda
13.01
19.08
31.39
42.53
142.4
168.1
3859
1821
1.4c4
8766
SparseGPT
12.27
17.41
23.38
30.47
86.88
84.12
292.9
237.1
1807
1094
L.L.MA3,2--B
L-ADMM
11.56
16.32
19.06
24.84
45.48
53.30
160.4
126.9
760.5
509.5
(Dess: 718321
ALPS
11.31
15.88
18.
22.83
41.79
46.48
166.32
109.0
542.0
367.0
SAFE
16.76
22.57
50.78
$7.86
330.9
267.2
3410
2343
SparseLI.M
ELSA
12.06
17.34
16.65
21.73
24.07
28.24
36.25
37.50
50.88
48.69
Magnitude
16.03
1924
2063
5.0e4
2.8e4
NaN
NaN
NaN
Wanda
6.92
9.24
10.79
13.99
76.32
81.08
4096
2673
2.0e4
1.0c4
SparseGPT
7.01
9.23
10.20
12.93
27.12
30.94
107.3
100.8
1430
864.5
LLaMA-2-7B
L-ADMM
6.80
8.97
9.40
11.47
20.56
22.20
60.78
58.63
400.5
287.1
(Dnne: 5.7/7226)
ALPS
6.86
9.02
9.33
11.30
19.39
20.37
48.43
47.22
248.8
180.9
SAFE
6.72
8.87
9.02
11.40
86.80
48.54
8.1e5
5.3e5
1.6c4
1.6c4
SparseLLM
7.23
9.51
10.74
13.25
37.65
35.00
126.5
94.28
1267
648.0
ELSA
9.16
11.34
13.20
20.83
26.97
23.14
Magnitude
6.83
9.38
11.82
14.62
214.2
191.9
3,9c4
4.9c4
7.5c4
6.5c4
Wanda
8.30
8.40
11.53
45.37
56.27
1004
838.8
2.2c4
1.3c4
SparseGPT
8.27
10.93
19.79
23.47
97.82
79.17
1442
984.1
LLaMA-2-13B
L-ADMM
5.92
7.57
10.05
14.81
17.56
44.78
44.42
242.1
(Dnne:: 4.88 /677)
ALPS
5.90
7.99
7.56
9.92
14.17
16.28
38.44
36.78
231.3
152.1
SAFE
5.73
7.82
6.90
9.24
12.47
14.57
93.49
73.25
2122
1388
SparseLI.M
ELSA
6.54
8.78
7.96
9.93
11.14
12.20
17.21
16.60
30.19
25.07
20

---

Published as a conference paper at ICLR 2026
Table 11: Zero-shot accuracy (%) of Llama-2-7B across multiple tasks, in various sparsity regime (5%%-
Tasks
Sparsity
Method
ARC-C
ARC-E
BoolQ
HellaSwag
OBQA
RTE
Winogrande
Avg
0%
Dense
43.35
76.26
77.68
57.14
31.40
62.82
69.06
59.67
Magnitude
34.90
64.02
62.91
49.13
26.80
57.04
63.22
51.14
Wanda
39.25
72.22
75.17
52.64
30.60
53.43
67.17
55.78
SparseGPT
38.23
71.34
75.99
52.70
29.80
56.32
69.77
56.31
L-ADMM
39.68
72.77
76.24
53.35
31.40
61.37
69.30
57.73
ALPS
40.61
72.90
75.44
53.37
30.80
57.76
68.98
57.14
SAFE
38.14
72.14
74.83
52.15
26.00
57.04
66.77
55.30
SparseI.LM
38.05
71.25
75.14
52.66
29.60
53.43
69.30
55.63
ELSA
39.42
71.30
73.03
53.12
29.40
58.48
66.54
56.39
Magnitude
25.17
44.87
47.80
35.00
20.00
50.90
53.12
39.55
Wanda
30.63
64.44
65.51
43.51
25.80
$4.15
64.01
49.72
SpanseGPT
31.57
64.06
72.57
45.0
25.80
53.43
65.51
51.13
L-ADMM
34.13
66.50
70.43
47.29
26.60
$5.60
66.61
52.45
60%
ALPS
34.38
66.33
70.64
47.81
27.2
54.15
66.29
$2.40
SAFE
31.14
64.14
71.10
46.43
24.00
$4.15
62.98
50.57
SparseLLM
32.59
64.52
70.86
45.24
25
$3.79
66.14
51.28
FI.SA
32.08
65.83
67.52
49.19
26.00
53.80
63.38
51.41
Magnitude
22.87
27.82
37.95
25.90
17.20
53.07
49.25
33.43
Wanda
30.01
28.04
52.71
35,36
SparseGPT
22.01
42.34
33.04
52.71
57.7
41.39
L-ADMM
23.81
50.63
63.21
36.57
20.40
54.15
60.77
44.22
70%
ALPS
25.51
$2.78
37.54
$3.43
61.72
45.03
SAFE
24.23
45.62
43.76
34.74
18.40
52.71
53.12
38,94
SparseLLM
40.32
61.87
32.74
54.51
57.46
40.54
ELSA
27.13
55.81
63.61
43.16
$2.71
$8.64
46.21
Magnitude
22.35
25.38
43.67
25.72
46.57
$1.62
32.62
Wanda
20.82
26.98
37.83
25.89
15.0
$2.71
49.25
32.64
SparseGPT
17.92
27.95
38.07
27.51
12.0
$3.07
49.01
32.22
L-ADMM
18.26
29.29
57.49
28.33
13.00
53.07
51.22
35.81
80%
ALPS
19.37
32.07
61.1
29.06
12.6
$2.71
50,91
36.83
SAFE
21.76
25.80
37.83
26.01
14.00
$2.71
49,80
32.56
SparseLLM
18.09
28.70
43.55
27.57
11.6
$2.71
48.86
33.01
ELSA
20.99
44.61
60.67
34.02
16.80
$2.71
53.20
40.43
Magnitude
22.78
25.93
39.17
25.53
16.0
47.29
50.12
32.40
Wanda
21.67
25.46
37.83
25.83
15.2
47.29
49.33
31.8
SparseGPT
20.65
26.77
37.83
25.7
13.0
52.71
50,59
32.46
L-ADMM
19.97
26.14
37.83
26.46
13.60
51.62
47.51
31.88
90%
ALPS
19.45
26.89
37.8
26.81
12.8
53.79
46.65
32.03
SAFE
21.84
26.52
37.83
25.91
15.80
52.71
47.83
32.63
SparseLLM
20.56
25.72
37.83
25.94
13.8
52.71
46.96
31.93
ELSA
18.52
41.33
57.25
31.54
16.60
52.71
51.70
38.52
21

---

Published as a conference paper at ICLR 2026
Table 12: shoo accuracy (%) of Llama-2 13B across multiple tasks, under various sparsity levels. We could not obtain SparseLLM in Llama-2-13B.
Sparsity
Method
Tasks
ARC-C
ARC-E
BoolQ
HellaSwag
OBQA
RTE
Winogrande
Avg
0%
Dense
48.46
79.38
80.55
60.04
35.20
65.34
72.14
63.02
Magnitude
38.48
70.58
57.65
54.39
27.80
55.96
65.35
52.89
Wanda
43.09
76.30
80.95
$6.96
31.20
60.65
71.43
60.08
SparseGPT
42.41
74.96
81.53
55.95
31.00
64,26
71.35
60.21
50%
L-ADMM
43.17
75.84
82,29
56.51
32.00
63.18
71.98
60.71
ALPS
42.66
76.30
81.22
56.71
32.60
62.82
72.14
60.64
SAFE
41.64
75.84
80.40
56,59
30.60
60.65
69.
59.27
ELSA
43.68
75.12
77.71
57.45
33,80
70.00
Magnitude
27.13
56. 14
47.49
44.66
21.80
$2.71
57.46
43.93
Wanda
37.97
68.81
77.16
48.71
28.20
59.57
68.19
55.51
SparseGPT
36.01
69.40
78.72
49.38
27.4
57.76
70.56
55.60
60%
L-ADMM
39.76
72.98
80.70
51.49
29.8
59,93
70.32
57.86
ALPS
72.93
81.68
30.80
71.90
58.58
SAFE
36.95
72.43
78.38
$2.09
28.80
$7.40
67.88
56.28
ELSA
40.53
70.67
75.17
54.22
31.00
$3.43
67.80
56.12
Magnitude
20.65
31.31
38.65
27.53
14.60
52.71
49.25
Wanda
18.43
36.45
62.35
29.25
52.71
50.83
37.57
SparseGPT
25.34
49.58
67.86
36.27
20.2
52.71
60.93
44.70
70%
L-ADMM
27.56
59.64
69.76
40.05
24.00
53.43
65.35
48.54
ALPS
29.61
61.20
70.09
40.86
26.6
$3.07
64.56
49.43
SAFE
29.78
61.07
69,17
41.62
20.20
$2.71
58.96
47.64
ELSA
34.13
62.42
70.12
47.52
24.80
52.71
60.38
50.30
Magnitude
21.84
25.63
41.80
25.88
14.80
53.07
49.25
33.18
Wanda
20.48
26.26
37.83
26.81
12.6
52.71
50.04
32.39
SparseGPT
19.62
28.79
59.05
27.77
12.8
$2.71
49.33
35.72
80%
L-ADMM
19.11
33.80
62.14
29.67
14.60
$2.71
53.28
37.90
ALPS
20.05
35.99
62.17
30.65
14.0
52.71
54,93
SAFE
28.37
40.64
27.44
12.80
52.71
50.51
32.97
ELSA
24.32
50.97
63.52
38.03
19.60
52.71
53.75
43.27
Magnitude
21.42
24.87
44.16
25.72
15.0
46.57
$1.78
32.79
Wanda
21.33
25.93
37.83
25.80
13.8
$2.71
51,54
32.70
SparseGPT
25.76
58.62
25.87
$2.35
49.49
35.28
90%
L-ADMM
19.88
26.01
39.45
27.08
13.80
53.79
50.04
32.86
ALPS
18.94
26.94
43.52
27.37
13.4
$2.71
48.30
33.02
SAFE
22.10
25.76
37.83
26.02
14.20
52.71
53.43
33.15
ELSA
19.03
36.15
$8.44
28.65
16.20
$2.71
50.43
37.37
22


Reviews:
[{"review_id": "VfksOUPaMo", "reviewer": "Reviewer_v5xh", "summary": "This paper introduces ELSA (Extreme LLM Sparsity via Surrogate-free ADMM), a method to prune large language models to extreme sparsity levels (up to 90\\%) while preserving performance. It identifies limitations in existing pruning techniques, which rely on surrogate objectives like layer-wise reconstruction error minimization, leading to performance collapse beyond 50-60\\% sparsity due to compounding errors and suboptimality. ELSA directly optimizes a sparsity-constrained problem using ADMM, incorporating objective-aware projections and avoiding surrogates. A quantized variant, ELSA-L, scales to 27B-parameter models with reduced memory. Experiments across models (OPT, Gemma, LLaMA) show ELSA achieves 5-30× lower perplexity and up to 6\\% higher zero-shot accuracy at 90\\% sparsity compared to baselines. Theoretical convergence is proven, highlighting potential for further LLM efficiency advancements.", "questions": "1.\tWhat is the computational efficiency of ELSA? How much time does it take to prune LLMs with different parameter scales?\n\n2.\tCan ELSA be used to prune Mixture-of-Experts (MoE) models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper is well-written and highly readable.\n\n2.\tAt high sparsity rates, ELSA achieves significantly better accuracy than other baselines and successfully breaks through the performance limit of LLM sparsification.\n\n3.\tThe experiments cover models of various parameter scales and different benchmarks, and conduct comprehensive evaluations of the LLMs' performance, including perplexity and zero-shot accuracy.\n\n4.\tThe theoretical foundation is solid: convergence proofs for ELSA and ELSA-L are provided, based on standard assumptions (such as weak convexity and smoothness). This enhances the reliability of the method and aligns with the empirical results.", "weaknesses": "1.\tELSA's accuracy at low sparsity rates (50% and 60%) is lower than that of the baselines.\n\n2.\tIt remains unclear how ELSA performs on larger models. Although the parameter scales of the models tested in the experiments range from 125 million to 27 billion, there is a lack of experimental results on even larger models, such as Llama-3-80B.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BG6uMTS9jg", "reviewer": "Reviewer_KMnE", "summary": "This paper introduces ELSA, a surrogate-free ADMM-based method for pruning large language models (LLMs) to extreme sparsity levels (up to 90\\%) while preserving performance. It critiques existing layer-wise reconstruction approaches for their limitations and compounding errors. ELSA directly optimizes the sparsity-constrained objective, achieving significantly lower perplexity (e.g., 7.8× better on LLaMA-2-7B at 90\\% sparsity) and higher zero-shot accuracy across models like OPT, Gemma, and LLaMA.", "questions": "1. The authors used such a large number of GPUs to prune LLMs, so why not perform LoRA fine-tuning on the pruned model to obtain a better pruned model? It is suggested that the authors compare the accuracy, computational resources used, and time overhead for obtaining sparse LLMs between ELSA and the \"Wanda + LoRA\" approach.\n2. Can the LLM sparsification technique used in this paper improve the model's inference speed? What advantages does it have compared with quantization techniques?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper has a reasonable structure and is clearly written. The authors explain their method in detail and make substantial theoretical contributions.\n2. The authors conducted extensive experiments across various model architectures and scales, evaluating metrics including perplexity and zero-shot task performance (e.g., ARC, BoolQ) and performing fair comparisons with state-of-the-art methods (e.g., SparseGPT, ALPS, SAFE). The results show that ELSA exhibits significant performance advantages at high sparsity levels (80-90\\%).", "weaknesses": "1. Both ALPS and L-ADMM have also proposed ADMM-based pruning algorithms for LLMs. The authors should clearly elaborate on the differences between ELSA and ALPS to highlight the unique contributions of ELSA.\n2. At low sparsity rates (50% and 60%), the accuracy of ELSA is significantly lower than that of the optimal L-ADMM baseline (Tables 7 and 8).\n3. Although ELSA has advanced the performance of LLMs at high sparsity rates (70%-90%), there remains a substantial performance gap between ELSA and dense LLMs.\n4. Pruning a 7B-parameter model requires 4 A100 GPUs, while pruning 13B and 27B-parameter models requires 4 H200 GPUs. Compared with Wanda and SparseGPT, this constitutes a much higher computational overhead—since pruning a 27B-parameter model using the above two methods only requires at most 1 A100 GPU.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ot1uTRUNJ0", "reviewer": "Reviewer_oqwk", "summary": "This paper addresses the \"sparsity wall\" (50--60\\%) in large language model (LLM) pruning by introducing ELSA (Extreme LLM Sparsity via Surrogate-Free ADMM). ELSA formulates pruning as minimizing the true task loss $f(x)$ under an $\\ell_0$ sparsity constraint and solves it via ADMM, which alternates between weight optimization and projection onto the sparse set. Instead of layer-wise reconstruction surrogates, it performs an objective-aware projection using diagonal Fisher/Adam second-moment statistics to guide weighted top-$k$ selection. Experiments on OPT, Gemma-2, and LLaMA-2 models show stable perplexity up to 90\\% sparsity and consistent gains over SparseGPT, Wanda, ALPS, L-ADMM, and SAFE. The paper also mentions a quantized extension (ELSA-L) for 27B-scale models with theoretical guarantees, though detailed results appear later in the text.", "questions": "1. How does ELSA translate its high sparsity into real inference acceleration or memory savings on modern hardware?\n\n2. Can you quantify the additional compute cost (training or calibration time) introduced by ADMM compared to simpler methods like SparseGPT?\n\n3. How robust is the whole approach to the hyper-parameters? ADMM-based approaches performance could be quite sensitive to hyper-parameters and hard to tune in practice, especially for large-scale LLM pruning problem IMHO.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The curvature-weighted projection is simple yet effective, using readily available second-moment statistics to guide pruning decisions.\n\n2. Empirical results are strong and consistent, showing stable perplexity up to 90% sparsity across diverse LLM architectures.\n\n3. The paper is well written and conceptually cohesive. The additional theories round up a solid work.", "weaknesses": "1. The experiments convincingly show perplexity improvements, but omit practical metrics such as wall-clock time, memory footprint, and actual inference acceleration. In reality, pruning methods—whether extremely local (e.g., SparseGPT, Wanda) or global (e.g., this work, SparseLLM and etc)—ultimately represent different points on a performance–cost tradeoff curve. With sufficient GPU resources, even global pruning becomes computationally feasible, reducing the motivation for additional algorithmic complexity like ADMM. It would strengthen the paper to explicitly highlight the practical scenarios or deployment constraints where ELSA provides tangible benefits over simpler baselines.\n\n2. The evaluation focuses solely on perplexity and task accuracy, without measuring actual inference acceleration, FLOPs reduction, or memory throughput gains after pruning. This limits the practical impact, as sparsity alone does not guarantee real-world speed-ups. \n\n3. While the paper frames pruning as a constrained optimization problem and leverages ADMM elegantly, both the global-pruning perspective and ADMM-style decomposition have precedents (e.g., SparseLLM, L-ADMM, SAFE). The novelty lies mainly in unifying these ideas under a “surrogate-free” formulation rather than introducing a fundamentally new optimization principle.\n\n4. [minor] Despite claiming global sparsity, the projection step is performed per-tensor using diagonal curvature estimates, so cross-layer dependencies are not modeled. The approach therefore remains an efficient approximation to the ideal global objective rather than a full solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)