DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs
Authors: Ruokai Yin
Yuhang Li
Donghyun Lee
Priyadarshini Panda
Venue: NeurIPS 2025 poster ()
Keywords: Weight Pruning
Dual-sparsity
Activation Sparsity
Efficient Large Language Models
Post-training Calibration
Model Compression
Primary area: deep_learning
TL;DR: We propose DuoGPT, a training-free pruning framework that integrates activation sparsity into OBC framework to enable efficient dual-sparse LLM inference with state-of-the-art accuracy–efficiency trade-offs and scalability.

Abstract: Large language models (LLMs) deliver strong performance but are difficult to deploy due to high memory and compute costs. While pruning reduces these demands, most methods ignore activation sparsity observed at runtime. We reinterpret activation sparsity as dynamic structured weight sparsity and propose DuoGPT, a unified framework that constructs dual-sparse (spMspV) workloads by combining unstructured weight pruning with activation sparsity. To preserve accuracy, we extend the Optimal Brain Compression (OBC) framework with activation-aware calibration and introduce output residuals from the dense model as correction terms. We further optimize the solution for efficient GPU execution, enabling scalability to billion-parameter LLMs. Evaluations on LLaMA-2 and LLaMA-3 show that DuoGPT outperforms state-of-the-art structured pruning methods by up to 9.17\% accuracy at an iso-speedup of 1.39$\times$ compared to the baseline dense model. Code is available at GitHub.

DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs
Ruokai Yin, Yuhang Li, Donghyun Lee, Priyadarshini Panda
Yale University ruokai minaal.yn@yyle.c..
Abstract
Large language models (LLMs) deliver strong performance but are difficult to deploy due to high memory and compute costs. While pruning reduces these de- mands, most methods ignore activation sparsity observed runtime. We reinterpret activation sparsity as dynamic structured weight sparsity and propose DuoGPT, a unified framework that constructs dual-sparse (spMspV) workloads by combining unstructured weight pruning with activation sparsity. To preserve accuracy, we extend the Optimal Brain Compression (OBC) framework with activation-awer - calibration and introduce output residuals from the dense model as correction terms. We further optimize the solution for efficient GPU execution, enabling scalability to callion-parameter L.Mss. Evaluations on LLaMA-2 and LLaMA-3 show that DuoGPT outperforms state-of-the-an structured pruning methods by up to 9.1 1.7 accuracy at an iso-speedup of 1.39x compared to the bascline dense model. Code is available GitHub.
Introduction
Large language models (LLMs) have made significant advances in performance across range of complex, real-world language tasks (Brown et al. 2020, Touvron et 2023). However, their massive parameter counts present practical challenges for deploying these large pre-trained models during inference. For example, deploying LLaMA-2-70B (Touvron et al. 2023), an open-source pre-trained LLM with 70 billion parameters, typically requires 150 GB of GPU and CPU RAM, along with 60 GFLOPs of computation to decode single token. Consequently, there is widespread interest in reducing both the storage and computational requirements of LLMs through techniques collectively referred to as model compression. Existing model compression techniques for LLMs include quantization (Frantar al. 2022, Ashkboos et al. 2024b), tensor decomposition (Wang et al. 2024, Lin et al. 2024), and pruning (Frantar & Alistarh 2023, Ashkboos et al. 2024a). This work focuses pruning, specifically incorporating activation sparsity into the one-shot weight pruning framework.
One-shot pruning methods selectively zero out the elements in the weight matrices of an LLM during one forward pass, and optionally update the remaining non-zero weights compensate for the pruning error. Depending on the freedom in the zero patterns, the pruning can be further categorized into unstructured pruning and structured pruming. In general, the following tradeoff between the categories has long stood: more structured sparsity yields more speedup', while more unstructured sparsity associated with better accuracy performance. In this work, we strike a balance between the two by incorporating activation sparsity. We observe that during the decoding stage of LLMs (i.e., General Matrix- Matrix-Vector (GEMV) operations"), activation sparsity can be interpreted as structured weight sparsity, as only weight corresponding non-zero activations involved
We focus the speedup of running LLMs general purpose architectures. e.g., GPUs.
We primarily focus on single-batch decoding in this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Activation sparsity * unstructured sparsity (ours)
Unstructured sparsity
Structured sparsity
input
weight
Input
weight
input
weight
Accuracy 1
Speedup 1
SRAM fetch compute
zero values, storage HBMs
Figure 1: GEMV operation for the single-batch decoding stage under different types of sparsity.
in computation. Further, activation sparsity has been observed in most LLMs due to the non-linear activation functions (Liu et al. 2023, Li et al. 2022). Despite its acceleration potential, activation sparsity still requires storing the full dense model on the GPU, resulting in significant memory overhead. To reduce this burden, we propose leveraging activation sparsity on top of unstructured weight pruning. Our method thus produces dual-sparse L.M workloads that achieve favorable balance between accuracy and efficiency, as illustrated in Figure 1.
Although combining dynamic activation sparsity with static weight sparsity is appealing, doing so naively can degrade performance. To address this, we extend the Optimal Brain Compression (OBC) framework (Frantar & Alistarh 2023) compensate for runtime enrors introduced by activation sparsity. Inspired by asymmetric calibration (Li et al. 2025), we reconstruct a layer-wice wise objective using the sparse activation from the pruned model and the corresponding dense activation from the original model. This enables weight updates during pruning calibration, mitigating accuracy degradation caused by sparse activations. Our contributions as follows:
1. We reinterpret activation sparsity as a dynamic, structured form of weight sparsity. Combined with unstructured pruning, this perspective enables the construction of sparse matrix matrix-sparse vector (spMspV) workloads for compressing and accelerating LLMs during decoding.
2. To compensate for the runtime errors introduced by dynamic activation sparsity, we pro- pose DuoGPT, layer-wise iterative pruning method that extends the OBC framework to activation sparsity-awand unstructured pruning in LLMs. With an efficient implementation, DuoGPT can calibrate 70B- O0B-parameter LLaMA-3 model in under 130 minutes single A100 80GB GPU.
3. We conduct extensive experiments on LLaMA-2 and 3, demonstrating that DuoGPT sig- nificantly improves dual-sparse model performance across tasks. Under 50% dual weight- activation sparsity. DuoGPT outperforms state-of-the-ard structured prreeend-pninggggg (e.g., Short- GPT (Men et al. 2024)) by 9.17% accuracy at an iso-speedup of roughly 1.4x relative to the dense baseline. Compared to the aptrse-activation (e.g., R-Sparse (Zhang et 2025)) methods, DuoGPT achieves up to 1.97x x model size reduction at iso-accuracy.
2 Related Work
Activation Sparsity. Activation sparsity a well-established property in ReLU-based LLMs (Li et al. 2022, Mirzadeh et al. 2023) and also emerges with newer activations like SwiGLU (Zhang et al. 2024a, Liu et al. 2023, Zhang et al. 2021, Haziza et al. 2025), naturally inducing sparsity in Multi-layer Perceptron (MLP) activations. This has motivated methods such as MoF-style MIP reconfiguration (Zhang et al. 2021, 2024b, Liu et al. 2023) and threshold- activation pruning from calibration data (Lee et al. 2024a). Recent efforts have extended activation pruning to all transformer layers using thresholding (Liu et al. 2025) or hybrid SVD-based techniques (Zhang et al. 2025). Unlike these approaches-whine retain dense weights-oo work integrates activation sparsity with unstructured weight pruning, yielding a dual-sparse LLM workload. Notably, threshold- based techniques from prior work complementary and can be incorporated into framework. For reference, we also briefly compare with three recent activation sparsity works: TEAL (Liu et al. 2025), R-Sparse (Zhang et al. 2025), and CATS (Lee et al. 2024a), in Table 3b, Table 5, and Table 8.
One-shot Weight Pruning for L.MSs. One-shot pruning, originally proposed by Han et al. (2015), removes weights based on magnitude but performs poorly at LLM scale (Frantar & Alistarh 2023).

---

SparseGPT (Frantar & Alistarh 2023) improves this with an approximate regression solver, enabling accurate pruning of 100B+ parameter models on single GPU without fine-tuning. SlimGPT (Ling et al. 2024) extends this to structured pruning. Wanda (Sun et al. 2023) offers compensation-free unstructured pruning using the magnitudes of weights and activations. SliceGPT (Ashkboos et al. 2024a) prunes both weights and activations while preserving computational invariance, but suffers sharp accuracy drops beyond 30% sparsity. Other approaches prune at the block or submodule level using importance scores (Men et al. 2024, Zhong et al. 2024, Sandri et al. 2025). Different from recent work (Lee et al. 2024b) that combines the actual structured and unstructured pruning, our approach rather reinterprets activation sparsity as dynamic structured weight sparsity that actual structured pruning happens during calibration time.
Optimal Brain Compression (OBC). OBC builds on the Optimal Brain Surgeon (OBS) frame- work (LeCun et al. 1989, Hassibi & Stork 1992), which leverages second-order derivatives and the inverse Hessian to guide weight pruning. OBC (Frantar & Alistarh 2022) approximates this approach using an iterative solver, yielding closed-form solutions for optimal weight selection and updates. SparseGPT (Frantar & Alistarh 2023) was the first to scale this framework effectively to LLMs. Our work extends this line by incorporating activation sparsity into the OBC formulation and introducing correction term derived from dense inputs. This design is inspired by the asymmetric calibration strategy recently proposed by Li et al. (2025) for OBC-based quantization.
3 Preliminaries
Notations. Throughout this paper, row vectors are denoted by bold lowercase letters and matrices by bold uppercase letters. For example, E R1 represents the linear operation between the weight W E R and the input calibration matrices X E R is expressed as:
vector of the weight. Consequently,
WX Here, denotes the hidden dimension, n is the number of output channels, and m is the number of input tokens. Pruning is done through the element-wiss multiplication, for example, w w @ m" represents the pruning of one row of weight, Here, m E {0, 1} is bitmask. We use superscripts to specify different masks. Subscripts are used to specify the subsets of vectors and matrices. A negative subscript index on a matrix indicates the removal of the corresponding row and column. For example, A-, is the matrix A with its jth column and jth row removed.
A Activation Sparsity. We induce activation sparsity by element-wise multiplying the input activation x with a binary mask m², i.e., x @ m The activation sparsity level px is defined as the proportion of zeros in m². For a given linear layer / during the decoding stage, the computation becomes: y == W(x O m²) = Eie = W X, where x, denotes the sparse input vector. During calibration, m;=1 when the input calibration data is in matrix form, we enforce that each column of the input has a sparsity level of p². Activations are pruned based on their absolute magnitude on the fly. In contrast to prior works (Liu et al. 2025, Zhang et al. 2025) that search for layer-speeific optimal activation sparsity levels, we investigate a simpler setting in which uniform sparsity is applied across all transformer layers. At decoding time, our method is agnostic to how activation sparsity is induced (Details can be found in Section 5).
The OBC (Frantar & Alistarh 2022) pruning framework converts dense weight matrices W to sparse one W through a calibration process to preserve model performance. This calibration process is efficiently implemented by SparseGPT (Frantar & Alistarh 2023) iterative layer- wise pruning framework for LLMs. Formally, for each row of weight matrix, the calibration targets to minimize the differences between the original and pruned layer outputs: Min lI(Aw + w)X w s.t. Awe, + wp O, where e, is the unit vector that corresponds to Wp. the weight that is removed. The loss for wp and the updates to remaining weights are:
OBC Framework for Pruning.
W 2 H
2p
Aw
H
pp
Here -- (XXT)-1 denotes the inverse Hessian matrix. The weights are sorted by their loss value and the weight with the lowest loss value is removed. Then, we compensate the remaining weights according to Equation 1. Once the nd weight is removed, the inverse Hessian is updated
'We align our notation with prior work in our method description. In implementation or illustration, the weight matrices are stored in column-man major format, as shown in Figure. I and 2(a).
3

---

offline calibration
Calibration input stage
Cholesky Decomposition
SRAM
GPU
HBM
space
CPU
XX
X
AXX
H
m
runtime: loading
values
indices:
AXX
runtime: computing
O value
value
D
real value
sel
Iteration data process stage
(b)
Figure 2: (a) Illustration of how -pal-sparsity accelerates the decoding stage of LLMs by saving computation, memory loading, and storage. (b) Computing paradigm of the DuoGPT's efficient GPU implementtionn. We neglect the element-wise division of (lag(()) for e in the figure.
through Gaussian elimination: H (H-1
This process iterates until the preset p*
H
layer-wise weight sparsity is achieved. The resulting sparse model follows the closed-fone solution.
DuoGPT
4.1
Activation Sparsity in Sparse LLMs
By leveraging activation sparsity, only certain rows of weights are fetched from High Bandwidth Memory (HBM) into Static RAM (SRAM) and further into the computation cores. However, since known beforehand which specifie rows will be required at runtime, memory storage cannot be reduced. As result, the entire dense model must still be fully materialized in the GPU's HBM.
To address this limitation, we propose introducing sparsity into both actívations and weights- strategy referred to as dual-sparsity. Specifically, we prune the dense model into a sparse version during offline calibration process. During inference, only the compressed model parameters loaded and transferred from CPU to GPU, significantly reducing both GPU HBM bandwidth and memory storage requirements. During linear layer computations, only the of weight matrices corresponding to non-zero activations are fetched from HBM into SRAM. Moreover, because each weight row is stored in compressed format, additional savings are achieved in SRAM loading, SRAM storage, and GEMM core computation. The overall runtime execution flow for this dual-sparse GEMV workload is illustrated in Figure 2(a).
potential challenge with dual-sparsity lies in the overlap between activation and weight sparsity, Given activation and weight sparsity levels of pa and p", respectively, the fraction of weights loaded from HBM to SRAM could approach (I-p²) in the worst case- caio--partccllllyy if the weight sparsity distribution is highly skewed. Empirically, however, into the pruning calibration process helps distribute weight sparsity more uniformly across rows. For instance, for calibrated LLaMA-2-7B model with pd p" 0.5, the maximum fraction of weights loaded from HBM to SRAM is approximately 0.25 (More details in Section 5 and Appendix). The remaining challenge is to determine how activation sparsity be effectively integrated into the weight pruning calibration process.
found that incorporating activation sparsity
4.2 Activation Sparsity-Aware Pruning Calibration
A central challenge in leveraging activation sparsity for model pruning lies in incorporating this sparsity pattern directly into the calibration process. This section introduces a principled framework for solving this problem. Let X denote the output from the previously pruned layer during calibration. Applying magnitude-bris
pruning to X yields a sparse version of the calibration input, denoted as X, which maintains uniform sparsity px across its columns. The calibration objective thus becomes minimizing the expression [l(Aw w)X wX While utilizing sparse calibration data helps the model quickly adapt activation sparsity during inference, it inevitably reduces the information

---

density available for accurately compensating the remaining unpruned weights. This fundamental trade-off necessitates a more sophisticated approach.
The solution lies in asymmetric calibration (Li et al. 2025), which incorporates both sparse and dense data to mitigate information loss. Given X as the dense output from the dense model, the target becomes (| + w)X - w 1| Setting w(X - X) = r as the output residual between dense and sparse data, the target simplifies to | r rll With the constraint of A Awe, + w, 0, the Lagrangian formulation becomes:
min Cp = Aw - -r1 + X(Awe) + Wp).
(2)
A.A
Solving this Lagrangian with H XX p-th weight (if removed):
yields the induced error and optimal weight updates for the
4p
W P
FX H
2w p
H. Aw
W Hpp
H + H
(3)
pp
Equation 3 defines the iterative framework, DuoGPT, which prunes and calibrates the model using a closed-form solution. For each row w. the algorithm identifies the p-th weight for pruning by computing p arg (Fpp then calculates compensation term Aw to update the remaining weights. This process repeats until the target pruning ratio p" is achieved.
Despite its theoretically sound approach to effectively incorporating activation sparsity (X) into the calibration process and compensating the associated information loss with dense model's information (X), the direct implementation of this framework becomes computationally infeasible for large-scale LLMs with billions of parameters. Two critical bottlenecks emerge: (1) A separate Hessian inverse (H-+) must be maintained and updated for each weight row every iteration (Frantar & Alistarh 2023). (2) Selecting which weight to prune requires computing the loss L in Equation 3 for every candidate weight, followed by recalculating the corresponding compensation term Aw after pruning.
The naive implementation results in total computational complexity of Onmmk2 + nk*3) for single layer. In the context of LLM calibration, both and k reach substantial values (e.g.. m 128 2048- = 262,144 144 and k = 4096 for LLaMA-2-7B), far exceeding the practical limits of modern GPUs An efficient implementation is therefore essential, as detailed in the following section.
4.3 Efficient DuoGPT Algorithm
To overcome these challenges, we first make simplification: assume that all weights be pruned have already been selected, i.e., a fixed pruning mask M is provided. Under this assumption, the asynchronous Hessian issue can be addressed through Hessian synchronization and weight freezing, as introduced in SparseGPT (Frantar & Alistarh 2023). The key insight is to process all rows in parallel by iterating through columns fixed order, rather than strictly following the optimal pruning order. This synchronization enables all rows to share a common Hessian. requiring only a single copy of H in GPU memory. Moreover, all Hessian updates can be precomputed in single step using Cholesky decomposition, where H = LL" and L is the lower-triangular Cholesky factor. This formulation resolves the asynchronous Hessian challenge with minimal computational overhead while enhancing numerical stability (Frantar & Alistarh 2023).
The critical remaining problem is: how determine the pruning mask M More precisely, pruning score metric S is needed for all weights to derive the pruning mask, effectively decoupling mask selection from the pruning and calibration stages. This score must satisfy three essential criteria: (1) Efficiency: The score must be inexpensive to compute, expressible in vectorized form, and amenable precomputation. (2) Adaptivity: must incorporate update information across iterations to avoid the performance degradation associated with static pruning structures. (3) Fidelity: The score should closely approximate the original loss metric L ensure that the final result remains good approximation of the exact closed-form solution.
To begin, the pruning score S is constructed using the error L incurred when each weight is removed, as derived from Equation 3. For each weight column p, the pruning scores are defined as:
W S.p
2W
+ diag(RM d RXX H XR
RX
H
(4)
H pp
H Pp
5

---

where R = W denotes the output residual for the full weight matrix. An observation is that Equation 4 comprises orthogonal error terms capturing distinct aspects of the objective function's landscape. The first term, inherited from SparseGPT, quantifies the intrinsic information loss from removing each weight in isolation. The subsequent terms emerge from DuoGPT and operate in complementary subspaces. The second term captures the quadratic self-interaction of output residuals, representing errors from the sparse-dense activation discrepancy. The third term functions as negative correction, quantifying how each weight's 's removal affects wwwvatton-spassit-inneee errors. The final term represents a cross-correlation between weight importance and activation- sparsity effects, capturing dependencies that enable adaptive compensation for dual-sparsity effects. Together, these terms form a holistic metric that precisely quantifies each weight's 's contribution to model performance in the dual-sparse regime.
An important observation emerges from Equation although output residuals help compensate for information loss introduced by activation sparsity, they must be updated whenever weights are modi- fied: R < R A W(X- - X). This requirement not only eliminates the possibility of precomputing pruning scores but introduces a computational cost of O(mnk) per column, rendering the naive approach prohibitively expensive. An opportunity for simplification comes from decomposing R into a sum partial-sum matrices formed by the outer product between each weight column and the corresponding row of the input difference AX X X. Specifically, the residual matrix can be approximated as R = Σド= W., Since R quantifies the discrepancy between sparse and dense model outputs, it can be decomposed into contributions from individual weight-activation pairs.Thir This decomposition transforms Equation into:
W Hp
2W S Hp
S
R + R.R.D
W WX X XXXX W.
WWW. W AX P.
X H
(5)
P
.p
where R.p W W. AX p.:- Furthermore, since the pruning process independent across rows, the
term W .p can be factored out from the last three terms in Equation 5. This results in the following expression for the pruning score:
1 AAX
2 Hpp
S p W
AX
XAX +
(6)
pp
This reformulation reveals an important property: pruning scores now adaptively update through explicit contributions from previously processed weight columns. Simultaneously. output residual information is implicitly incorporated by fusing the R updates into Equation 6, allowing R to evolve across iterations without explicit recalculation.
The next challenge is compute Equation efficiently. Denoting the three new terms (annotated in red) as =,= AXJ.. b,=AX,, X H p.: and c,=AX,, X /H efficiently p.:
computing Equation 6 is now equivalent to efficiently computing a, b, and c.
First, all elements of a can be precomputed as a=diag( AXAX leveraging the independence across rows of AX. Second, the term bp can be reformulated by applying the Cholesky de- composition to H H-1 which yields H p.
Consequently, b, transforms into: AX,,. L. p I,XXXX p.:- Exploiting row independence again, the entire b can be precomputed b=diag (UU"), where Up.:=X,,. p. X Lpt
Finally, by again leveraging the Cholesky decomposition, H HH can be easily represented by Lt Lpp (Frantar & Alistarh 2023, Frantar et al. 2022). Thus c can be precomputed as c diag( AXX L) Ø diag(L), where @ denotes the clement-wise division (Proofs in Appendix).
further insight that U contains components from both the current iteration (the p-th column) and the subsequent one (the (p+1)-th column). Therefore, U can be re-expressed as: U= U=((AXX L) @
MD) where M E R is strictly upper-triangular masking matrix with ones above the diagonal.
This reformulation offers a significant computational advantage: the shared intermediate term Q=AXX L can be reused for computing both and c. Furthermore, U itself can be reused to precompute the new weight compensation terms in Equation 3, specifically the term rx H Denoting this quantity as D, it can be precomputed D=ULT demonstrated in Li et al. (2025).

---

Algorithm The DuoGPT activation sparsity-aware pruning for one layer with target of p" unstructured sparsity, Given lazy batch and mask selection block size B, each consecutive B columns will be p" sparse.
Input: Dense weight W, sparse calibration input X with p' sparsity, Dense input X.
H Xx L Inverse_ CenersseCholeskyll (H), AX X-X X
Q AXX L. U + Q Mu
diag AXAX
dUU ). c diag QQ) O diag(L), UL
4- 0,
O, M LxB, E 0,xB
for i = 0, B.2........ do
for j 1,i+
i+ B do
if mod B 0 then
Sugy. W Wisto @ [ diag(L))) ays.B - byg-B 20
M J5+B B f O-mask p" of weights E WagyeB with the lowest S B.
end if
P.j.
O M
E B
P..
W. ()(++) W W))+())) E- JJ000 W WiDDaattt
end for
W :)+BB): + W B)+BB - -EL EL + W. W.664 DR // lazy-batch updates
end for
W
The pruning score S for arbitrary column p is thus reformed into the following efficient form:
S.P W + + - by + 2cp).
All newly introduced terms are now computable via vectorized operations. Particularly, since both and b are expressed as the diagonal of inner products between matrices, they can be computed using in-place elementwise square operations followed by column-wiss summation. As result, computing the pruning scores for an entire layer reduces to (m.?² complexity-reducing the runtime of the naive implementation by a factor of min(n, nule
m
Moreover, since this efficient reformulation of S derives directly from the original error metric C-with the only approximation being the decomposition R-the pruning score remains a faithful surrogate to L, ensuring close approximation to the exact closed-form solution. The efficient implementation of DuoGPT illustrated Figure 2(b), with the complete algorithm presented in Algorithm 1. Full derivations are provided in the Appendix.
Compared to the naive implementation with complexity Onumk² + nk*)). DuoGPT's new com- plexity is sufficient to make the algorithm practical, even for extremely large models. Additionally, SParseGPT's iterative blocking and lazy-batch updates are adopted to mínimize data movement across iterations, further accelerating runtime. In practice, DuoGPT calibrates LLaMA-3-70B model on a single 80GB A100 GPU in under 140 minutes.
4.4 Theoretical Analysis
We provide a brief theoretical analysis of DuoGPT, aiming to derive a lower bound on its guaranteed loss improvement over SparseGPT under activation activation-sparriit- satr--sppprtttaaaaaaasaa aware pruning calibration. The complete proof is given in the Appendix, and our formulation builds upon the numerical eтror analysis framework in (Wu et al. 2016).
Theorem 1. Under the activation sparsity level p" and given Esparetrr and Lpwour as the loss functions of two methods during calibration. DuoGPT achieves the following guaranteed loss improvement over SpurseGPT:
ap"o? Amar (H)
AC LSparsePPT LDWOGPT rX T
where a > O is the stability constant measuring spectral gap preservation, Amar the maximum eigenvalue of the Hessian H X o, is the activation residual magnitude constant, Cu is the weight norm bound satisfying w | Ow. and m is the calibration batch size.
7

---

Theorem 1 predicts consistent and interpretable improvement of DuoGPT over SparseGPT that scales linearly with activation sparsity level p². Experimentally, shown in Tables 3a, and we observe consistent PPL. improvement of DuoGPT across all model sizes and sparsity settings, validating the bound's predictive trend. Moreover, the empirical results reveal a clear scaling behavior: higher dual-sparsity levels yield larger performance gains, in agreement with the theoretical analysis.
5
Experiments
We implement DuoGPT using PyTorch (Paszke 2019) and the HuggingFace Transformer library (Wolf et al. 2019) for efficient model and dataset management. All calibration procedures and experiments are performed on 80GB NVIDIA A100 GPUs with offloading (two GPUs are specifically employed for zero-shot evaluations of 70B models), Unless otherwise stated, the calibration dataset consists of 128 2048-token samples, randomly selected from the C4 training dataset (Raffel et al. 2020). To assess model performance, we evaluate the perplexity (PPL) of our DuoGPT -rruned models on the WikiText2 dataset (Merity al. 2016). Furthermore, we complement our evaluation by conducting O-shot task classifications using the LM Eval Harness (Gao et al. 2021) across widely recognized downstream benchmarks: PIQA (Bisk al. 2020), HellaSwag (Zellers et al. 2019), WinoGrande (Sakaguchi et al. 2021), ARC-easy, ARC-challenge (Clark et al. 2018), 8), OpenBookQA (OBQA) (Mihaylov et 2018), and BoolQ (Clark et al. 2019). More detailed setups can be found in the Appendix.
Experimental Setup.
Unstructured Pruning Baselines Comparison. We begin by comparing DuoGPT with dual- sparse baselines constructed using two widely adopted one-shot unstructured pruning methods: SparseGPT (Frantar& Alistarh 2023) and Wanda (Sun al. 2023), as shown in Table To ensure fair comparison, we enable 50% activation sparsity for both DuoGPT and SparseGPT during calibration, allowing each method perform activation-awar aware compensation. For Wanda, which lacks weight-adjusstentt compensation, the calibration data remains in dense format. All methods are evaluated with models pruned 50% weight sparsity and run with 50% activation sparsity during inference. Additionally, we compare to the unnruuursructrred (2:4) variants of these two approaches, which are widely regarded as achieving balance between speedup and accuracy (Mishra et 2021). In the 2:4 setting, activations remain dense during both calibration and evaluation.
Table I: Comparison against other one-shot unstructured pruning methods in their Qual-sparse variants (W 50% X50%) and 2:4 variants (W2:4 2:4 + Xdense). DuoGPT has 50% dual-sparsity We also report GPU hours required for calibration. Bold and underlined values indicate the best and second-best results, respectively.
|| Method
WikiZ())
PIQA
HelluSwug
ARC-Easy
IRC-Challenge
Avg(1)
Dense
6.14
80.63
79.13
77.53
53.33
72.93
72.71
SpaneGPT
0.21
14.05
72.96
61.97
62.29
37,80
62.51
59.51
SpusoGPT (2:4)
32.34
31.59
LLaMA-3-88
Wanda Wanda (2:4)
0.03
13.98
71.38
57.02
59.39
35.84
61.64
$7.06
0.03
25.13
67.85
47.97
50.80
29.95
58.96
$1.11
Dense
2,86
84.60
84.96
$6.07
64.25
80.58
80.09
SpanseGPT
1.62
7.54
80.03
74.87
78.49
50.85
74.03
71.65
L1aMA-3-70B
Spans(PP
46.33
72.77
66.61
Wanda
0.26
8.19
80.03
76,90
77.10
50.09
70.17
70.86
Wanda (2:4)
0.28
9.32
80.20
73.06
75.42
49.49
71.67
69. 90.97
DuoGPT
2.28
7.38
80.52
75:94
27.82
52.82
75.69
72.56
Dense
79.11
75.99
74.58
16.25
69.06
69. .0
SpaneGPT
0.18
8.98
73.88
63.87
67
35.75
63.38
60.02
LLaMA-2-7B
SpuneGPT (2:4)
0.20
12.1
71.65
57.41
59.18
32.17
64.80
57.04
58,
Wanda (2:4)
0.04
12.2
70.95
$4.94
$7.28
31.40
62.27
$5.37
DuoGPT
0.22
8.58
73.88
63.75
64.18
35.67
65.11
60.52
SpameGPT
0.32
7.39
76.39
69.21
68.27
41.04
67.64
64.50
LLaMA-2-13B
Sparse(PT (2:4)
0.33
9.56
74.16
62.95
63.09
37.46
66.22
60.78
Wanda (2:4)
0.07
9.05
75:41
69
64.73
37.12
67.09
61,47
DuoGPT
0.40
7.17
75.63
69.56
69.95
41.13
68.11
64.88
$7.34
SpansGPT
1.68
79.54
77.51
78.87
$2.73
75.37
72.80
LLAMA-2-70B
Wanda
SpansGPT (2:4)
0.27
1.50
5.03
5.94
80.52
78.84
78.39
74.00
75.34
77.15
53,16
48.25
75.53
75.53
70.40
72.95
(2:4)
71.88
DuoGPT
2,30
5.02
80.41
77.64
78.96
52.56
77.51
73.42

---

Across all LLaMA models, DuoGPT consistently achieves the lowest perplexity, validating its capability to enhance pruned model performance under sparse activations. For example, on LLaMA- 3-70B, DuoGPT reduces perplexity from 7.54 (SparseGPT) to 7.38. In zero-shot evaluations, it surpasses the second-best baseline results on WinoGrande by up to 2.14% and 2.29% for LLaMA-2 and 3, respectively. Overall, DuoGPT achieves the highest average accuracy across all evaluated tasks and models. Furthermore, compared to 2:4 weight-only sparse models, DuoGPT consistently yields improved performance. Notably, LLaMA-3-SB, DuoGPT enhances the average accuracy from 54.59% (best-performing 2:4 baseline) to 60.04%, marking an improvement of 5.45%. We additionally report GPU hours (single A100 GPU with offloading) required for calibration. Owing to our efficient implementation detailed in Section 4.3, calibration of 70B models can be completed in approximately 2.3 hours.
Table 2: Compare with other structured pruning methods on LLaMA-2-7B. To align the normalized GPU speedup relative to the dense model, we set SliceGPT with 30% and other three baselines with 31.25% weight-only sparsity. DuoGPT has 50% dual-sparsity.
Method
|| Model Size
Speedup
WikiZ
PIQA
HellaSwag
ARC-Easy
ARC-Challenge
WinoGrande
Avg(1)
ShonGPT
4.728/6.744 74B
1.44x
65.6
63.55
50.92
33.79
51.35
2SSP
4.72B/6.74B
1.31x
12.4
71.71
63.11
$5.05
31.83
64.17
57.34
BlockPruner
4.72B/6.22 74B
141x
20.5
69.53
54.77
51.89
32,00
60.85
SliceGPT
5,298/6 74B
1,26x
23.1
71.22
58.04
56.14
33.11
64.88
56.68
DuoGPT
3.50B/6.74B
1
8.58
73.88
63.75
64.18
35.67
68.11
60.52
Structured Pruning Baselines Comparison. We compare our dual-sparse DuoGPT framework with several popular structured pruning methods: ShortGPT (Men et al. 2024), 2SSP (Sandri et al. 2025), BlockPruner (Zhong et al. 2024), and SliceGPT (Ashkboos et al. 2024a). All methods are evaluated using 256 samples of calibration data (32 for 2SSP, following its original setup). Each model is pruned to approximately 30% structured sparsity, resulting in up to 1.44x end-to-end speedup over the dense baseline. As shown in Table 2. by leveraging activation sparsity, DuoGPT achieves a competitive end-to-end speedup (1.39x) while maintaining significantly better accuracy compared to the structured pruning baselines. Additionally, DuoGPT yields the smallest model size post-compression, due to its higher 50% unstructured weight sparsity. structured pruning methods were pushed comparable compression ratio of 50%, their performance would degrade sharply.
Table 3: Ablation studies of (a) different dual-sparsity levels and (b) other activation sparsity methods.
Method
30%
40%
55%
60%
65%
Method
Model
OBQA
Arc-C
BoolQ
Avg
SparseGPT
5.87
6.52
13.2
29.2
88.2
R-Sparse 50%
6.908/6 748
30.40
33.49
72.54
46.14
Wanda
5.85
6.49
16.0
67.1
248
DuoGPT + TEAL 50%
3,50B/6.74B
38.40
36.77
71.77
48.98
DaoGPT
5.84
6.48
12.3
26.3
77.3
DuoGPT 50%
3.50B1
39.00
35.67
72.48
(a) WikiText PPL of pruned LLaMA-2-
(b) Comparison and study of compatibility to other threshold- based activation sparsity work on LLaMA- models.
7B models with varying dual-sparsity.
Ablation Studies. In Table 3a, we evaluate the performance of our method compared to other dual- sparse baselines across varying levels of dual-sparsity. As shown, DuoGPT consistently improves the performance of dual-sparse models at all sparsity levels. Notably, the performance gains become more pronounced as the dual-sa qualssp increases.
We further investigate the compatibility of our method with threshold- activation pruning during evaluation. As shown in Table 3b, applying the activation thresholding technique from TEAL (Liu al. 2025) to 50% dual- sparse DuoGPT model yields comparable performance to on-the-fly fly magnitude-based pruning. We also compare against R- R-Sparse (Zhang et al. 2025), a prior approach that combines activation sparsity with SVD-based low- rank weight compression. As Table 3b shows, DuoGPT achieves higher average accuracy than R-Sparse under 50% sparsity. Moreover, while our method reduces the model size to 50% of the dense baseline, R-Sparse incurs 1% increase in memory usage due to the additional SVD branch.
Weight Loaded to SRAM
Portion of Dense (%)
70%
50% 49.5%
0.5
25.7% % 16.9%
Dense
30%
50%
30%
50%
60%
SWS AS
DS
DS
Figure Number of weights LLLMA 2-7B loaded to SRAM. SWS=structured weight sparsity, AS=activation sparsity, and DS-dual-sparsity,

---

Next, we analyze the number of weights loaded into SRAM across different sparsity paradigms, as shown in Figure 3. Consistent with our discussion Section 1. we observe that the -pal-sparse model loads approximately (1 (1-p") (1-p*) fraction of the weights into SRAM. This result shows potential for further improving of the efficieccy of dual-sparse L.MMs with custom GPU spMspV kernels or ASIC dual-sparse accelerators.
We also demonstrate DuoGPT's compatibility with weight-only quantization, a promising com- pression approach for accelerating memory-bounded LLM inference. We consider both 8-bit and 4-bit uniform integer quantization with channel-wise RTN (group size 128), which can be seam- lessly integrated into our pruning framework (Frantar et al. 2022, Frantar & Alistarh 2022), and report the perplexity of both LLaMA-2-7B and LLaMA-2-13B on WikiText2 in Table
Table 4: WikiText2 PPL with 50% dual- sparsity and varying weight precision.
Finally, evaluate the performance of DuoGPT rea- soning benchmark. Table 5 compares the accuracy of 50% activation-sparse TEAL, 30% dual-sparse SparseGPT, and DuoGPT on the 5-shot GSM8K (Cobbe et al. 2021) using LLaMA-3-8B, under the same number of SRAM weight fetches, DuoGPT achieves the highest accuracy and the smallest model size under iso-SRAM weight load.
Model
Method
FP16
INT8
INT4
SparseGPT
8,98
14.2
LLaMA-2-7B
DuoGPT
8.58
8.58
13.8
SparseGPT
7.39
7.41
10.0
LLaMA-2-138
DuoGPT
7.17
7.18
9.39
Limitations and Future Work. This work does not yet include full GPU kernel imple- mentation for accelerating the spMspV opera- tion. Our current kernel implementation mainly exploits activation sparsity (sp which en- abled the throughput measurements reported in our main paper. This demonstrates a lower bound on the real-world efficiency gains from our dual-sparsii approach. Unlike spMspM,, spMspV avoids costly index matching, making it strong candidate for efficient GPU acceleration an avenue we leave for future exploration. While DuoGPT offers promising approach to calibrate dual-sparse LLMs, this remains an open problem with substantial potential for further research.
Table 5: Comparison with other baselines on 5- shot GSM8K with LLaMA-3-8B.
Method (X, WApsoit
Model Size
SRAM Loads
GSM8K
TEAL (50%, dense)
8.03B/8.03B 5.948/8.03D
50.0%
36.47
SparseGPT (30%. 30%)
49.7%
36.39
DuoGITT (309, 20%)
5.94898.0B
49.6%
36.77
Furthermore, our work focuses on inducing dual-sparsity only for linear layers. Other efforts exist to induce single-sided sparsity in attention layers. Integrating dual-sparsity with these recent works on both unstructured (Joo et al. 2025) and structured (Xu et al. 2024) KV cache pruning remains relevant and promising direction.
Conclusion
In this paper, we present DuoGPT, an efficient one-shot dual-sparse pruning framework that unifies unstructured weight pruning with runtime activation sparsity. By extending the OBC framework with activation-awant calibration and residual correction, DuoGPT achieves -fate-of-the-art accuracy while significantly improving both memory and computational efficiency. Our method scales to large L.MSs and complements existing activation pruning techniques, offering practical path toward efficient and scalable LLM deployment.
7
Acknowledgement
This work was supported in part by CoCoSys, JUMP2.0 center sponsored by DARPA and SRC, the National Science Foundation (CAREER Award, Grant #2312366, Grant #2318152), the DARPA Young Faculty Award and the DoE MMICC center SEA-CROGS (Award #DE-SC0023198).
References
Albert, Q. J., Alexandre, S., Arthur, M., Chris, B., Devendra, S. C., Diego C., Florian, B.,
Gianna, L., Guillaume, L.,, Lucile, S., et al. Mistral 7b, arXiv preprint 241:2240 2024.
10

---

Ashkboos, S., Croci, M. L., Nascimento, M. G. d., Hoefler. T., and Hensman, J. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15022 2024a.
Ashkboos, S., Mohtashami, A., Croci, M., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier free bit inference in rotated 1lms. Advances in Neural Information Processing Systems, 37:100213 1-100440 2024b.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng. X., Fan, Y., Ge, W., Han, Y., Huang, F,, et al. Qwen technical report. arXiv preprint wrXiv:2309.166//0 2023.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1905,1004, 2019.
Clark, P., Cowhey, I., Etzioni, O.. Khot, T., Sabharwal, A., Schoenick, C., and Tafjord. O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:/803.0544 1803.05457. 2018.
Cobbe, K., Kosaraju, V., Bavarian, M.. Chen, M., Jun, H., Kaiser, L., Plappert, M.. Tworek, J., Hilton, J., Nakano, R., al. Training verifiers to solve math word problems. arXiv preprint 11X1::2110.14668, 2021.
Frantar, E. and Alistarh, D. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Frantar, E. and Alistarh, D. Sparsegpt: Massive language models be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative preitrained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., et al. A framework for few-shot language model evaluation. Version vO. 0.I. Sept, 10:8-9, 2021.
Han, S., Pool, J. Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. Advances neural information processing systems, 28, 2015.
Hassibi, and Stork, D. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Haziza, D., Chou, T., Choudhary, D., Wehrstedt, L., Massa, F., Yu, Jeong, G., Rao, S., Labatut, P., and Cai, J. Accelerating transformer inference and training with 2: activation sparsity. arXiv preprint arXiv:2503. 16672, 2025.
Joo, D., Hosseini, H., Hadidi, R., and Asgari, B. Mustafar: Promoting unstructured sparsity for ky cache pruning in Ilm inference. arXiv preprint 2rXiv22505.22113 2025.
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Lee, D., Lee, J., Zhang, G., Tiwari, M., and Mirhoseini, A. Cats: Context-aware thresholding for sparsity in large language models. In First Conference on Language Modeling, 2024a.
Lee, J., Qiao, A., Campos, D. F,, Yao, Z.,, He, Y., et al. Stun: Stnetured-tenn-nssttuuuurd pruning for scalable moe pruning, arXiv preprint 4rXhv:2409.0221,, 2024b.

---

Li, Y., Yin, R., Lee, D., Xiao, S., and Panda, P. Gptqv2: Efficient finetuning free quantization for asymmetric calibration. arXiv preprint TrXiv2254400292. 2025.
Li, Z.,, You, C., Bhojanapalli, S., Li, D., Rawat, A. S., Reddi, S. J., Ye, K., Chern, F., Yu, F., Guo, R., et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. arXiv preprint FrXiv:22100063111 2022.
Lin, C.-H., Gao, S., Smith, S., Patel, A., Tuli, S., Shen, Y., Jin, H., and Hsu, Y.-C. Modegpt: Modular decomposition for large language model compression. arXiv preprint 24Xi::2408.09632 2024.
Ling, G., Wang, Z., and Liu, Q. Slimgpt: Layer-wise structured pruning for large language models. Advances in Neural Information Processing Systems,
2024.
Liu, J., Ponnusamy, P. Cai, T., Guo, H., Kim, Y., and Athiwaratkun, B. Training-free free activa- tion sparsity in large language models. The Thirteenth International Conference on Learning Representations, 2025.
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. International Conference on Machine Learning, pp. 22137-221766 PMLR, 2023.
Men, X., Xu, M., Zhang, Q. Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403003853, 2024.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv: 1609.07843, 2016.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint (00iv:/80000077000 2018.
Mirzadeh, I., Alizadeh, K., Mehta, S., Del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.0450
2023.
Mishra, A., Latorre, J. A., Pool, J.. Stosic, D., Stosic, D., Venkatesh, G., Yu, C., and Micikevicius, P. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08374 2021.
Paszke, A. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 2019.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(14)):1-67 2020.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-006, 2021.
Sandri, F., Cunegatti, E., and Iacca, G. 2ssp: A two-stage framework for structured pruning of llms. arXiv preprint arXiv:2501. 2025.
Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint urXiv:2306. 1695, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Wang, X., Zheng, Y., Wan, Z., and Zhang, M. Svd-Ilm: Truncation- singular value decomposi- tion for large language model compression. arXiv preprint arXiv:2403.07378 2024.
Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arxiv: 1100::111 2019.
12

---

Wu, S., Bhojanapalli, S., Sanghavi. S., and Dimakis, A. G. Single pass pea of matrix products. Advances in Neural Information Processing Systems, 29, 2016.
Xu, Y., Jie, Z., Dong, H., Wang, L., Lu, X., Zhou, A., Saha, A., Xiong, C., and Sahoo, D. Think: Thinner key cache by query-driven pruning. arXiv preprini arXiv:2407.21018, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.07830 2019.
Zhang. S., Roller, S., Goyal, N., Arterxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V. al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205. 01068, 2022.
Zhang, Z., Lin, Y., Liu, Z., Li, P.. Sun, M., and Zhou, J. Moefication: Transformer feed-forward layers are mixtures of experts. arXiv preprini 1rX:v:21 2021.
Zhang, Z. Song, Y., Yu, G., Han, X., Lin, Y., Xiao, C., Song, C.,, Liu, Z., Mi, Z., and Sun, M. Relu wins: Discovering efficient activation functions for sparse llms. arXiv preprint arXfr:2402.03804, 2024a.
Zhang, Z., Xiao, C.,, Qin, Q., Lin, Y., Zeng. Z., Han, X., Liu, Z., Xie, R., Sun, M., and Zhou, J. Exploring the benefit of activation sparsity in pre-training. arXiv preprint arXiv:2410.03440 2024b.
Zhang, Z., Liu, Z., Tian, Y., Khaitan, H., Wang, Z., and Li, S. R-sparse: Rank-aware activation sparsity for efficient IIm inference. In The Thirteenth International Conference on Learning Representations, 2025.
Zhong, L.., Wan, F., Chen, R., Quan, X., and Li, L. Blockpruner: Fine-grained pruning for large language models. arXiv preprint arXiv:2406. 10594, 2024.
13

---

A Theoretical Derivation
A.1 Proof of Equation (Optimal Framework for Activation Sparsity-Aware Pruning)
We provide the detailed derivation of DuoGPT's optimal calibration framework (Equation 3) proposed Section 4.2.
Proof. The Lagrangian in Equation 2 is first expanded:
C = C TrT((AwX - r) (AwX - r)) + AAwe, + AW
Tr(X Aw Aww X Aw r AwX +r + AAwe, + Awp
To find the local minima of the Lagrangian in Equation 2, set the partial derivatives with respect to Aw and to zero. We first compute the partial derivative with respect to Aw:
OC
2AwXX 2rX Aep
dAw
(10)
2AwH 2rX + Xep.
and the partial derivative with respect to A:
0C
Awe, + Wy.
(11)
ax
Setting the first equation to zero yields:
A 2
Aw rX H
(12)
X
- H H
2
Setting Equation 11 to zero and substituting Aw from Equation 12, we obtain:
rX
H
e
H e + w, = 0.
H
(13)
2
By simplifying with Hp..o,
H pp:
we solve for as:
2
A
(rX
H
c + wp)
H
1
pp
(14)
2
H .p
wp)
H
pp
Substituting Equation 14 back into Equation 12, we solve for A w as:
Aw = rx
H
H p.
H
(15)
W
H
H
pp
pp
By recognizing that the expression enclosed in parentheses in the second term corresponds to the Gaussian climination operation introduced in Section 3, we obtain the final expression for Aw:
W. H +
Aw
(16)
which is identical to the weight update term in Equation 3.
The next step to derive the final expression for L. Substituting Equation 16 into the loss function = ||AwX - rli the loss becomes:
L
H
(17)
P.c
pp
14

---

This expression can be expanded as:
w
L
2(rX
H H
r)(
Wp
x
H
)2
Hpp
pp
2rX
H
X +
(18)
w
- Wp
r
H
HH.
+2
W
H.
H
H
pp
HH
2rX H
+rr
To simplify Equation 18, we observe that the second and third terms are nearly identical, differing only by the presence of the term H H. This same pattern appears between the fourth and fifth terms. We begin by simplifying H H
H
H H H
PP
P.i
H
H
H
H
(19)
Op-
pp
With the help of the identity matrix, we can more clearly see how the aforementioned nearly identical terms simplify. Substituting Equation 19 back into Equation 18:
C
+2
H
op H
H pp
(20)
H H
H Xr +
An important observation is that e,, H -1 is in fact an all-zero vector since the p-th row of H is eliminated. Consequently, the third term Equation 20 equals zero. Part of the second term can álso be further simplified: H He, ep = H., H The expression H HH in the first term can also be simplified to H H. Combining all these simplifications, Equation 20 reduces to:
w
C
P
- - T
(21)
H
pp
By reordering the second and fourth terms, we obtain the identical format of L as presented in Equation 3.
A.2 Proof of Cholesky Decomposition of H (Efficient calculation of b for pruning score S)
Recall that in Section 4.3, we leverage the equality - 1.p+1: L 1:p+1; to simplify the computation of term b for pruning score We provide the proof for this equality using mathematical induction below.
Proof. We begin the mathematical induction proof by establishing the base case: p 1.
Base Case: First, we rewrite the lower-triangular Cholesky factor L as:
L
0
(22)
L
L2.2:
The Cholesky decomposition of H can be written as:
Ln
Lu
LT
H
(23)
H2.2.
L
L2.2
L
22.2.
15

---

Based on this equation, we can construct the following linear system:
H 11 L
H Ln L
(24)
H2.2
L2, 22,1
L L
Solving the above equations, obtain the expression for L2,
2::2:
12 2.2
H
H
H 2.
2:2:
(25)
H
H
Recalling the expression for Gaussian elimination introduced in Section 3. Equation 25 essentially represents the removal of the first row and column from H after applying Gaussian elimination to zero out its first row and column. Therefore, we have proven the base case that H = L2 L
Inductive Step: Assume the statement holds for k 1. 1. Since we know (k--1) Lge, L.,k the lower-triangular Cholesky factor L for H (k--1) can be formulated as:
We show it also holds for k.
LEg L(--I);
(26)
L(++I).(++)))
This formulation is valid because, under our inductive hypothesis for p = k - 1, all rows and columns before & have already been removed. We can construct similar linear systems as in the base case:
H
Li Rb
(++1): (
(27)
H (++1):(--I): L ()+ L) 1) D Solving the above equations, we again obtain:
L(k-1),((+1)) (4-))) H (++1).(+1)
(K+1)ax
(1+1))))
H
(28)
H (K+1))((HI)
H ()+
R.(+++1):'
kk
Again, Equation 28 is equivalent to removing the first row and column of H (k-1) after applying Gaussian Elimination to zero out its first row and column. This operation is equivalent to computing H Therefore, we have proven that H_| L(++ (). 1): (I+ 1)+1))((+))) By the principle of mathematical induction, the statement holds for all with & 2 1.
A.3 Proof of H H = L..p/LL Lpp (Efficient calculation of c for pruning score S)
This equality has previously been leveraged to simplify the implementation of SparseGPT (Frantar Alistarh 2023) and GPTQ (Frantar et al. 2022). We also leverage this relationship to simplify the calculation of term in our pruning score S.
Proof. We assume that at the p-th iteration of the calibration, all columns and rows before have been removed through Gaussian elimination. Based this assumption, the Cholesky decomposition of the Hessian inverse at the p-th iteration can be formulated as:
H
H H
P.lo-1): (o+t)))tt))))
Lpp ((
(Lpp
L D
L 1)+())+(11)
LT
(p+1)ap
(29)
This equation also leverages the result proven in Section A.2. Next, we divide the expression H into
three parts: H H-1 1:(0-1).p p).p' pp. and H (p+1):p' For the first two parts, the equality is straightforward
16

---

to prove. Since H
1:(p-1), = -1:p- ppp p, so H 1:(p-1),p H - o = Lid(p- D.p/pp
And H pp Hpp Lpp/ Lpp-
The only equality that requires proof is: H (p+1):pp/ Hpp
Le 1).p/ Lpp. From Equation 29, construct the following linear relations:
H
pp pp
(30)
(p+ D).p Lpp
Substituting these relations back into the expression, it is straightforward to verify that H pp Lp 1):.p/ L pp ((p+
H (pt1):
A.4 Proof of U= ()
Recall that Section 4.2, first proposed to be precomputed as b = diag(UUU)) (UUT), where Up.:: = p. L'ptIII-111 To further improve the algorithmic efficiency, we propose to re- express U the form (AXXTL) c Mr). so that the term AXXT L can reused between b and The proof straightforward and provided below.
Proof. The p-th row of U equals AX,, 1p where L E R is the lower-triangle Cholesky factor. For any given vector Rixk and the Cholesky lower-triangle factor matrix 1pti:p+1: we have:
0
if ri<p+l else i 2 p +1
(zL (zLpt1:p+1:) =
(31)
2
Therefore, substituting z = AX p.. row of U as:
X
E ER 1xk into the above equation, we can reformulate the p-th
0
if i
U p.i
(32)
AX p. X else i p
Hence, Up. equals (AXp. p. X M where M" is strictly upper-triangulas masking ma- trix with ones above the diagonal. The full computation of U can thus be re-expressed as U= (AXXTL) O Mr. By leveraging the independence across rows AX, b = diag( UUT), where U = (AXX L) M..
A.5 Proof of Theroem
The key step in proving Equation 8 is to solve for LSpameGPP. We will follow the same procedure as Section A.I. According to Equation I, the optimal weight update for SparseGPT Aw We H Substituting this into the loss function = ||AwX - rl/ the loss for SparseGPT H
becomes:
SPpaseGPT
(33)
H
F²
Given the loss of DuoGPT in Equation 17, is straightforward to observe that the difference between the loss functions after equation expansion is simply AC rX THT Hr
The first observation we can make that, since H is positive definite, the loss improvement of DuoGPT (AL) will always be greater than or equal to zero. Then, since H is obtained from H via Gaussian elimination, it also follows the spectral bound that Amin (H
Xuma (H) where
the stability constant. We can then rewrite AL into its quadratic form to get AL a
Here,
Amaa [H)
it is reasonable to bound || 1|2 as || where o, is just constant to capture the intensity of the activation residual. This approximation reasonable due to the fact that we are always perturbing the smallest values during calibration, the term is not large overall. After assuming weight norm aCa, lower bound of Ow, we can get AC 2 (H) Finally, it is intuitive that the activation residual intensity is proportional to the number of "zerreddout" activations; thus, we include the term pmm to capture the context of global activation residual intensity. This yields the final error bound as shown in Equation 8.
17

---

B Detailed Experimental Setups
This section provides comprehensive implementation details and experimental configurations to ensure reproducibility of our results.
B.1 Calibration Settings for DuoGPT and Unstructured Baselines
We implement all unstructured baselines reported in Table using uniform evaluation settings to ensure fair comparison. Our implementation builds upon the calibration framework from SparseGPT (Frantar& & Alistarh 2023), using PyTorch (Paszke 2019).
For both SparseGPT and DuoGPT, we enable act_ oct_order, an option in SparseGPT that sorts weight columns based Hessian diagonal magnitude improve pruning performance. The dampening ratio for the Hessian is set to 0.1 for numerical stability. We apply the lazy batch updates and iterative mask blocking techniques from SparseGPT to DuoGPT for reducing 1/O overhead. The block size and lazy batch size are both set to B = 128. For both methods, we prune the input calibration data to each layer based magnitude before calculating each layer's Hessians.
We integrate the original pruning and mask selection procedures from Wanda (Sun et 2023) into evaluation framework. Since Wanda lacks compensation mechanism post-pruning, maintain dense activation throughout the calibration process (activation sparsity-aware pruning disabled).
For 2:4 structured versions of both SparseGPT and Wanda, we apply mask selection and pruning over every 4-column group with 50% weight sparsity. All other configurations remain identical to their unstructured counterparts. Calibration data remains dense throughout the pruning process for all 2:4 baselines.
B.2 Settings for Speedup Results
We report end-to-end GPU speedup results on an NVIDIA A A10 80GB GPU in Table 2 and 6. All speedups are normalized with respect the dense model baseline rather than reporting absolute running times to ensure fair comparison across different frameworks.
For DuoGPT, end-to-end GPU speedup is measured by inheriting TEAL's custom Triton kernel (Liu et al. 2025) and integrating it into the fast-gpt framework. The corresponding dense model bascline is also evaluated using the fast-gpt framework obtain normalized speedup ratios. All speedups for structured pruned models are measured using 2SSP's framework (Sandri et al. 2025), with dense model baselines similarly evaluated within the same framework compute normalized speedup.
B.3 Settings to Measure the Number of Weights Loaded to SRAM and Model Size
In Figure 3, we report the number of weights loaded into SRAM during inference. For dense models, structured-sparse models, and activation sparse models, the number of weights loaded into SRAM fixed. However, for DuoGPT, the dual-sparsity approach combined with unstructured weight distribution across makes the number of loaded weights dynamic quantity that depends which activation patterns are selected. To ensure a conservative and fair comparison, we report the worst-case SRAM loading numbers for DuoGPT. Specifically, we sort weight rows by their weight sparsity and assume that dynamic activation sparsity always selects the rows with the lowest weight sparsity (i.e., the densest rows requiring maximum SRAM access). For example, with 40% activation sparsity and 40% unstructured weight sparsity, we first sort all weight rows by their sparsity levels, then select the 40% of rows with the lowest weight sparsity-those requiring the largest number of weights to be loaded into SRAM. This conservative methodology ensures our reported SRAM access numbers represent an upper bound; in practice, DuoGPT's savings on SRAM accesses can be even higher.
Model size results reported in Table 3(b) and Table 5 include the full parameter count, including all model parameters beyond the transformer blocks (e.g., embedding tables and output projection layers).
18

---

C Extra Experimental Results
C.1 Extra Performance Comparisons with Structured Pruning Baselines
In Table 6, we provide additional comparison results against structured pruning baselines on LLaMA- 2-13B. All structured pruning baselines are pruned with 30% weight-only sparsity. The structured pruning methods use 256 calibration samples (except 2SSP, which uses 32 samples following its original setup) with 2048 token sequences. We employ 50% dual-sparsity for DuoGPT. The results demonstrate that DuoGPT achieves better performance across speedup, model size, and accuracy metrics. Notably, 50% dual-sparse DuoGPT achieves even higher speedup (1.51 x) compared the 30% structured pruned models, where the fastest baseline achieves 1.41 x speedup.
Table 6: Extra comparison result with other structured pruning methods on LLaMA-2-I3 -3-BBB
Model
Method
Model Size
Speedup
Wiki2(J)
PIQA
HellaSw
ARC-E
ARC-C
WinoG
Aug(1)
ShonGPT
9.21B/13.028
1
39.78
69.80
57.94
52.86
35.75
69.06
57.08
1LMMA-2-13B
9.218/13.022
1.29
76.66
70.51
65,28
38.74
68.90
BlockPruner
9.21B/13.00
1.38x
9.67
73.94
64.39
61.87
37.37
66.61
00.84
DusGPT
6.67B/13.028
LSIX
7.17
75.63
69.56
69.95
41.13
68.11
64.88
LLaMA 3 8B
LLaMA 7B
LLaMA 2 13B
70
70
70
Average Accuracy (%)
60
60
60
50
DusGPT
50
50
SparseGPT
40
Wanda
40
40
200
Wikitext
150 PPL
200
100
100
100
50
50
0
0
30%
40% 50% 60% 65%
30% 40% 50% 60% 65%
30% 40% 50%
60% 65%
Figure 4: Mean zero-shot accuracy and perplexity on LLaMA-3-80 LLaMA-2-7B, and LLaMA- 2-13B. The results are reported across different dual-sparsity levels. The perplexity is reported for WikiText2 dataset and the accuracy results are averaged across tasks.
C.2 Performance across Different Dual-Sparsity Levels
We provide a comprehensive visualization of our approach's performance compared to baselines across different dual-sparsity levels in Figure 4. The two baselines dual-sparse variants of SparseGPT and Wanda, using identical setups as those in Table We plot perplexity (PPL) perfor- mance the WikiText2 dataset and mean zero-shot accuracy across 7 tasks. We visualize sparsity levels of 30%, 40%, 50%, 60%, and 65%. Beyond 65% dual-sparsity, performance degrades too sig- nificantly from the dense model to be meaningful, while below 30%, efficiency gains insufficient to justify the complexity.
Several key trends emerge from Figure 4. At low dual-sparsity regimes (<500)), performance differences between methods are minimal regardless calibration settings. However, the benefits of activation sparsity-aware calibration become pronounced at high dual-sparsity regimes (250%). For example, Wanda's perplexity performance (without activation sparsity- calibration) begins to degrade exponentially beyond 60% dual-sparsity. The SparseGPT baseline partially recovers this performance loss through activation sparsity-awar calibration, though these benefits diminish at even higher sparsity levels (65%).
DuoGPT addresses this limitation through output residual compensation, using dense model outputs to compensate for information loss during sparse calibration. This enables superior performance
19

---

in extreme dpar-sparsity regimes, such as 65%. This trend is consistently observed across different metrics (perplexity and zero-shot accuracy) and model scales, as visualized in Figure Detailed perplexity and zero-shot accuracy results for each task are provided in Table 7 for reference.
Table Detailed zero-shot accuracy results for LLaMA-3-SB, LLaMA-2-7B, and LLaMA-2-13B
across different dual-sparsity levels. ARC-E stands for ARC-Easy, ARC-C stands for ARC-Challenge WinoG stands for WinoGrande, and OBQA stands for OpenBookQA, Bold and underlined values
indicate the best and second-best results, respectively.
Model
Method
W1ki2(1)
PIQA
HellaSwag
ARC-E
ARC-C
WinoG
BoolQ
OBQA
Avg(t)
LLaMA-3-8B
SparseGPT
7.00
78.94
77.20
76.43
49,06
72.85
81.07
43.40
68.42
Wanda
6.97
79.27
77.10
75.63
50.17
71.82
80.18
43,80
68.28
W aos5 Xa0%
DuoGPT
6.96
78.35
77.08
75.84
48.55
72.14
80.92
43:40
68.04
SparseGPT
8.57
77.26
73.07
71.68
44.97
70.88
78.81
40.80
65.35
Xoon
Wanda
8.46
76.28
72.61
71.42
46.59
69.
77.58
42.20
65.13
DuoGPT
8.47
76.88
73.21
70.88
44.28
68.11
78.39
41.20
64.74
SparseGPT
14.05
72.91
61.96
62.29
37.71
62.43
74.62
36.20
58.30
Woox WaoxXxxx
Wanda
15.98
71.44
57.01
59.34
35.84
61.33
70.80
36.60
56.05
DuoGPT
13.41
73.72
61.88
62.96
36,77
64.72
74.25
35.20
58.50
SparseGPT
59.85
58.32
33.96
39.39
23.38
$2.25
58.29
27.80
41.91
Mon X
Wanda
87,50
56.64
30.64
35.02
20.82
50.99
55.29
2..800
39.31
DuoGPT
47.33
61.32
36.95
42.76
24.83
52.57
62.29
27.60
44.05
SparseGITT
59.85
54.35
28.22
29.76
21.33
50.99
38.56
26.20
35.63
Wess
Wanda
87.50
53.37
28.18
28.79
22.44
50.51
38.04
26.00
35.33
DuoGPT
47.33
56.75
28.91
32,74
20.56
51.46
49.48
26.20
38.01
LLaMA-2-7B
SparseGPT
5.87
78.45
74.97
73.57
45.39
69.14
77.06
43.60
66.03
Wanda
5.85
78.29
75.30
73.32
46.16
67.56
76.30
42.40
65.62
Waux X
DuoGPT
5.84
77.91
74.66
73.06
45.65
68.43
77.34
44,00
65.86
SparseGPT
6.52
77.04
71.80
70.12
42,75
66. 30
72.12
41.80
63:70
W Wax Xax
Wanda
6.50
77.26
72.01
69.99
67.64
75.60
42.80
64.03
DuoGPT
6,48
76.61
71.57
71.13
43.34
67.17
76.18
42.00
64.00
SparseGPT
8.98
74.43
63.89
62.67
35.75
63.46
71.56
36.80
$8.37
Wou X00%
Wanda
9.13
62.30
66
35.32
62.75
70.24
39,80
57.87
DuoGPT
8.58
73.83
63.74
64.18
35.67
65.35
72.54
38.80
59.16
SparseGPT
29.21
61.15
38.20
42,38
24.57
52.41
63.67
30.20
44.65
WooxXoox
Wanda
67.13
55.93
29.67
34.01
22.70
50.59
58.59
27.000
39.78
DuoGPT
26.27
63.49
41.54
45.03
26.96
56.59
63.30
30.80
46.82
SparseGPT
88.17
53.43
29.04
29.25
22.53
49.49
51.38
25.60
37.25
Wask Xasn
Wanda
248.9
52.18
28.10
28.16
24.32
50.20
38.26
23.40
34.95
DuoGPT
77.34
55.44
29.80
32.62
50.83
58.90
28.60
39.68
LLaMA-2-13B
SparseGPT
5.20
79.71
78.62
75.88
49.66
71.90
80.83
46.80
69.00
Wanda
5.22
79.43
78.56
75.80
71.35
80.12
Waog Xyng
DuoGPT
5.18
80.36
78.57
75.46
49.15
71.74
81.77
48.20
69.32
SparseGiPT
5.70
79.33
76.45
73.36
47.10
71.98
81.10
43.20
67.50
Wanda
5.69
76.73
72.43
68.51
DuoGPT
5.66
78.67
75.97
73.02
47.01
70.80
80.83
44.80
67.30
SparseGPT
7.39
76,28
69. 14
68.52
41.81
68.67
79.30
42.00
63.67
Wao% WaoxXaox
Wanda
7.41
77.20
69.15
67.93
40.19
66.54
77.28
42.80
63.01
DuoGPT
7.17
76.12
69.43
69.49
40.96
67.40
77.83
43,40
63.52
SparseGPT
20.33
64.74
43.20
49.83
28.33
56.27
68.13
29.80
48.61
W Woo%: X00%
Wanda
35.33
63.00
36.90
44.65
25.09
$2.17
62.72
30.40
44.99
DuoGPT
17.74
45.74
50.29
67.25
SparseGPT
67.21
54,90
29.76
30.77
20,90
49.41
61.87
25.80
39.06
Waxx NaxxX
Wanda
129.22
52,50
29.31
29.04
22.44
$1.07
50.70
25.20
37.18
DuoGPT
48.49
58.16
31.76
34.34
22.35
$1.62
62.20
27.20
C.3 Sensitivity Analysis on the Calibration Set Size and Sequence Length
We present an ablation study to analyze the role the C4 calibration dataset. We focus on DuoGPT with 50% dual-sparsity using the LLaMA-2-7B model. For the calibration set size study, we fix the sequence length to 2048 tokens. For the calibration sequence length study, we fix the number of calibration samples to 128.
20

---

10
10
Wikitext PPL
9
8
7
32
64
128
256
512
256
512
1024
2048 4096
Calibration set size
Calibration sequence length
Figure 5: The effect of the C4 calibration set size and sequence length on PPL of Wiki'Tex22 dataset for LLaMA-2-7B.
Computing Performance Term:
Computing Performance Term:
#Neurons (4)
8.192
4,096
4,096
Ungarallel
2.048
Parallel
2,048
10
10
10
10
10
10a
10-3
10-1
10
10
10
10a
104
Time (seconds)
Time (seconds)
Computing Performance on Term:
Computing Performance on Term: S
16,384
16,304
#Neurons (k)
8.192
8.192
4,096
4,086
2,048
10
10"4
10
20"
10
109
10-1
10-1
10
10
10-
10"
10'
Time (secondss)
Time (seconds)
Figure 6: Latency visualization of algorithm under different size of input channel k.
Figure 5 (left) shows the effect of varying the number of calibration samples on WikiTeex2 perplexity. The results demonstrate that least 128 calibration samples provide reasonable performance for our calibration procedure.
We next explore the effect of different sequence lengths in the calibration dataset. Interestingly, find that beyond a sequence length of 2048 tokens, the perplexity does not continue to decrease, as shown in Figure (right). We conclude that for activation sparsity-awan calibration, sequence length of 2048 tokens is sufficient to achieve good perplexity performance.
C.4 Algorithm Effficieccy
Finally, we provide visualization of the GPU runtime for our efficient DuoGPT implementation (Equation 7) compared to the unparallelized implementation (Equation 6). We measure latency single A100 80GB GPU using PyTorrch 2.4.0. We provide 10 warm up runs. The token dimension (m) is fixed 2048. Figure compares the latency for computing a. c, and the overall pruning score S. Owing to highly optimized CUDA kernels, our vectorized precomputation of all quantities completes approximately ms for a 4096 x 4096 layer with m 2048 tokens, achieving roughly 350x speedup over the unparallelized implementation. Note that we do not compare against the naive implementation that follows the optimal pruning order, as is prohibitively slow and would not yield meaningful comparisons.
21

---

C.5 Extra Comparison with Prior Works
Given that CATS (Lee al. 2024a) is an important prior work that validated the use thresholding technique to induce activation sparsity in LLMs. is important to compare DuoGPT against it. We set the comparison to be at 40% global sparsity, which means that DuoGPT will have 40% dual-sparsity, Since CATS only explores activation sparsity in FFN layers, it requires 89.7% activation sparsity in those layers achieve an equivalent 40% global sparsity. The comparison between the two works on 5 downstream tasks is reported in Table 8. Since we directly report the results from CATS, for fair comparison, we do not report the normalized accuracy for DuoGPT in this comparison.
Table 8: Comparison with CATS LLaMA-2-7B at 40% global sparsity.
Method
PIQA
HellaSw
ARC-E
ARC-C
WinoG
Avg(1)
CATS
66.27
38.48
45.66
28.16
57.38
47.19
DuoGPT
76.66
53.42
74.37
40.78
67.17
62.48
For reference, also provide additional comparison with ReLUfication (Mirzadeh et al. 2023).
Since ReLU mostly yields around 50% activation sparsity, we compare it with DuoGPT at 22% global dual-sparsity. As discussed in (Lee et al. 2024a), ReLU-based methods perform poorly without enough epochs of fine-tuning.
Table 9: Comparison with Rel.Ufication on LLLMA-2-7B 22% global sparsity.
Method
PIQA
HellaSw
ARC-E
ARC-C
WinoG
Avg(1)
ReLUfication
54.08
25.86
27.95
24.06
48.93
36.18
DuoGPT 78.73
56.80
76.47
43.26
67.96
64.64
C.6 Isolated Effect of Asymmetric Calibration Weight-only Pruning
useful provide the isolated effect of asymmetric calibration (Li et 2025) on weight-only pruning, that we can get better idea of how activation sparsity-aware asymmetric calibration helps improve performance. We evaluate two scenarios: (I) weight-only pruning (no activation sparsity), and (2) 50% sparsity across different models in Table 10.
(with activation sparsity). We report the WikiText2 PPL results at
Table 10: Ablation study isolating residual correction effects.
Scenario
Method
LLAMA-2-7B
LLaMA-2-13B
LLaMA-3-8B
SparseGPT
7.11
6.15
9.98
Weight-only
DuoGPT
6.97
6.03
9.73
SparseGPT 8.98
7.39
14.05
Dual-sparsity
DuoGPT
8.58
7.17
13.41
The results demonstrate the following: 1. Asymmetric correction is effective its DuoGPT outperforms SparseGPT even in the weight-only pruning setting, showing that the second term in Aw (Equation 3) provides clear benefit. Activation sparsity amplifies the gain of asymmetric calibration. When activation sparsity is introduced, the magnitude of improvement increases by 2.9x, 1.8x, and 2.6x, respectively, across the three models. These results confirm that our closed-fome solution adapts effectively: the asymmetric calibration compensates for error from both weight pruning and activation sparsity.
C.7 Extra Results Different LLM Architectures.
Demonstrating our method's generalizability across L.LM model architectures crucial. We have thus evaluated DuoGPT multiple model families beyond LLaMA, which shows consistent
22

---

improvements across different architectures. We tested DuoGPT on Mistral (Albert et al. 2024), Qwen2 (Bai et al. 2023). and OPT (Zhang al. 2022) of varying model sizes. We report the WikiText2 PPL. at 50% dual-sparsity in Table 11.
Table II: WikiTex12 perplexity results across different LLM model families.
Method
Mist-7B
Qwen-2-7B
Qwen-2-1.5B
OPT-125M
OPT-1.3B
Dense
5.25
7.13
9.54
27.65
14.62
SparseGPT
8.40
10.91
19.50
55.08
35.25
DuoGPT 7.91
10.76
18.11
51.88
31.70
The results show that our method consistently outperforms SparseGPT- dual-sparse baselines, demonstrating DuoGPT's generalizability across different LLM model architectures.
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: our abstract and introduction, we talk about our contribution, the leveraging of dual-sparsity for efficient LLM decoding. And provide one-shot activation-awant pruning calibration framework together with it efficient implementation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide one paragraph limitation the end of experimental sections, which is about the missing of efficient GPU kernel for dual-sparse workloads.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
24

---

Answer: [Yes]
Justification: The formal proofs are provided in appendix. The informal proof sketch is provided in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the detailed experimental setup in the Appendix. For our proposed algorithm, the full pseudo algorithm description is provided in the main paper. Moreover, we will provide all source codes on GitHub.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
25

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will provide the source codes supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ helliic/gudes///eee Cooeeoomiiisiinooiii
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: // "twppsccpppplllia wttppc/bbbideeeeeeeepppppppppeee httpe/gguuddee/eeeuuubiieeeeeeee wwwodeuumisssiooooooo..oooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe the experimental setting and details in Section 5 and Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not include the error bars for reasons. (1) The calibration of large models (c.g., 70B) require non-negligible unit GPU hours, (2) we fix all random seeds of every experiments to O.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the type and amount of GPUs for reproducing the experiments in Section 5. All experiments can be reproduced with up 2 A100 80GB GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https :tttpp:///euure htt////eurrrp.eeeee//eee/ wwtpppplcccttiicceiiieeeeeeeeeee
Answer: [Yes]
Justification: Our research and experiments conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: This work focuses calibrating the dual-sparse LLMs in an efficient way. The pratical deployment of dual-sparse I.LMs have not yet been fully studied. There is no social impact in this field yet.
27

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymemt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our models and datasets the standard benchmarks that are widely used in academia. All models and datasets are cited properly.
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com comddatases has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs. LLM is only leveraged for writing and formatring.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: betpps//neurrps wwt////oe//oceeeeeeeeee cccc//ofeeeencss//22///L.. for what should or should not be described.
30


Reviews:
[{"review_id": "U8hfSdf38K", "reviewer": "Reviewer_nwtw", "summary": "This paper proposes DuoGPT, a compression method for LLMs that combines structured pruning and unstructured pruning using activation sparsity and weight pruning. It extend the OBC framework with residual correction to accommodate attention sparsity mechanisms. Experimental results demonstrate speedup over dense LLMs, with comparisons against baselines.", "questions": "See weaknesses above.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper proposes a framework that combines structured and unstructured pruning to balance performance and efficiency. It includes adaptive modifications to prior methods like OBC, which is practical.\n- Experiments demonstrates the effectiveness and applicability of DuoGPT.\n\n**Weaknesses**\n\n- The method of collecting activation statistics from calibration data and dynamically activating components during decoding is similar to CATS [1], which is also acknowledged in the paper and uses a threshold-based activation sparsity mechanism. A more detailed comparison in both methodology and performance may improve the clarity and originality of this work.\n- Why not use ReLU for activation sparsity, which offers a more straightforward and efficient way to reduce computation?\n- Combining structured and unstructured pruning seems simple and has been explored in prior works such as STUN [2]. Moreover, the motivation for introducing activation sparsity within the structured pruning component is unclear, since many structured pruning methods are already compatible with unstructured sparsity.\n- Some core results (e.g., Table 1, LLaMA-2-70B) show only a marginal improvement in perplexity (e.g., 0.07 better than SparseGPT), while requiring significantly more compute (e.g., +0.6 GPU hours for calibration). The effectiveness of the method seems heavily dependent on additional computation.\n- The comparison with SparseGPT does not isolate the effect of residual correction to OBC, as DuoGPT also includes activation sparsity. A separate ablation would help clarify its contribution.\n\n[1] CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models, COLM 2024\n\n[2] STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning, arxiv 2024.9", "was_revised": true, "final_justification": "The author’s response addressed most of the concerns.", "rebuttal": "#### We thank the reviewer for the valuable comments and insightful suggestions and have tried our best to address all of them in our rebuttal.\n\n> **Q1: Comparison with CATS**\n\n#### Thank you for the suggestion. We acknowledge that CATS is a pioneering work on activation sparsity in LLMs and provide a detailed comparison below.\n\n#### **Methodological Differences:**  \n#### CATS focuses on *output*-activation sparsity in gated MLPs by applying thresholds after non-linear activation functions (e.g., zeroing $\\text{SiLU}(xW_{\\text{gate}})$ neurons below a threshold). This approach targets SRAM savings specifically for $W_{\\text{up}}$ and $W_{\\text{down}}$.  \n#### In contrast, DuoGPT focuses on *input*-activation sparsity by thresholding the *inputs* to each layer. This design allows activation sparsity to be applied across **all** linear layers in LLMs, not just FFN blocks.\n\n#### **Performance Comparison:**  \n#### We evaluate both methods on five downstream tasks using LLaMA-2-7B at 40% global sparsity. DuoGPT consistently outperforms CATS across all tasks, as shown below.\n\n| $\\small{\\text{Method}}$       | $\\small{\\text{PIQA}}$  | $\\small{\\text{HSwag}}$ | $\\small{\\text{ARC-E}}$ | $\\small{\\text{ARC-C}}$ | $\\small{\\text{WinoG}}$ | $\\small{\\text{Avg}}$   |\n|------------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|\n| $\\small{\\text{CATS}}$     | $\\small{66.27}$        | $\\small{38.48}$        | $\\small{45.66}$        | $\\small{28.16}$        | $\\small{57.38}$        | $\\small{47.19}$        |\n| $\\small{\\text{DuoGPT}}$   | $\\small{76.66}$        | $\\small{53.42}$        | $\\small{53.56}$        | $\\small{40.78}$        | $\\small{67.17}$        | $\\small{58.32}$        |\n\n> **Q2: Why not ReLU?**\n\n#### Thank you for the question. While ReLU can indeed induce activation sparsity, our approach offers several key advantages:\n\n#### 1. ReLU provides output-activation sparsity only for FFN blocks, while our input-activation sparsity works across all linear layers in LLMs.\n\n#### 2. Our method requires no fine-tuning, operating in a one-shot calibration fashion. ReLU replacement requires extensive fine-tuning (e.g., 30B tokens for LLaMA-7B **[1]**).\n\n#### We further compare DuoGPT with the ReLU-based method [1] at 22% global sparsity on LLaMA-2-7B below.\n\n| $\\small{\\text{Method}}$           | $\\small{\\text{PIQA}}$  | $\\small{\\text{HSwag}}$ | $\\small{\\text{ARC-E}}$ | $\\small{\\text{ARC-C}}$ | $\\small{\\text{WinoG}}$ | $\\small{\\text{Avg}}$   |\n|----------------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|------------------------|\n| $\\small{\\text{ReLUfication}}$ | $\\small{54.08}$        | $\\small{25.86}$        | $\\small{27.95}$        | $\\small{24.06}$        | $\\small{48.93}$        | $\\small{36.18}$        |\n| $\\small{\\text{DuoGPT}}$       | $\\small{{78.73}}$ | $\\small{{56.80}}$ | $\\small{{76.47}}$ | $\\small{{43.26}}$ | $\\small{{67.96}}$ | $\\small{{64.64}}$ |\n\n> **Q3: Comparison with structured + unstructured pruning**\n\n#### Thank you for pointing out the STUN reference—we will include it in the related work section. However, we would like to clarify a key misunderstanding: **DuoGPT does *not* combine structured and unstructured pruning.**\n\n#### **Key Methodological Distinction**:  \n#### - **STUN** combines actual *structured pruning* (e.g., 5% for non-MoE models) with *unstructured pruning* (often >50%). While this achieves good accuracy, the unstructured component remains difficult to accelerate efficiently on SIMT architectures.  \n#### - **DuoGPT**, by contrast, performs only *activation-aware unstructured weight pruning* during calibration. At inference time, we reinterpret activation sparsity as a form of *dynamic structured sparsity* during GEMV operations. This allows for GPU acceleration without the accuracy degradation typically seen in actual structured weight pruning.\n\n#### **Core Innovation**:  \n#### We leverage the insight that, in single-batch decoding (GEMV), activation sparsity naturally behaves like structured sparsity: only the weight rows corresponding to non-zero activations are accessed. Building on this, our main technical contributions include extending the OBC framework with a novel closed-form solution that rigorously accounts for activation sparsity effects using asymmetric residual correction (Eq. 3), and develop an efficient GPU implementation that scales to 70B+ models in under 140 minutes.\n\n#### **Performance Comparison**:  \n#### On 5-shot GSM8K with LLaMA-2-7B, DuoGPT achieves **comparable accuracy** to STUN (e.g., 7.2% with 47.5% weight sparsity and 15% activation sparsity) while offering **3× greater acceleratable portion** (15% activation sparsity vs. 5% structured sparsity in STUN), demonstrating the practical effectiveness of our dynamic structured sparsity approach.\n\n> **Q4: Dependent on additional computation**\n\n#### We appreciate this concern and would like to clarify two important points:\n\n#### 1. **No Causality Between Compute and Effectiveness**:  \n#### One good example would be that the additional calibration time is a constant overhead, independent of the sparsity level. Meanwhile, the effectiveness of DuoGPT increases with dual-sparsity level. As shown in Table 6, our perplexity (PPL) improvement on LLaMA-2-7B grows from **0.5% (at 30% dual-sparsity)** to **12.5% (at 65% dual-sparsity)**—all at the same calibration cost.\n\n#### 2. **Expected Behavior for Large Models**:  \n#### Larger models are known to be more resilient to compression. For example, Wanda **[2]** reports PPL improvement dropping from 0.09 (7B) to 0.00 (70B) compared to SparseGPT. Our observed improvement of 0.07 on LLaMA-2-70B is thus reasonable and consistent with prior work.\n\n#### Importantly, the calibration cost is a **one-time offline expense** that scales sublinearly with model size. It is **negligible compared to training costs** and enables efficient inference over the model’s entire deployment lifecycle.\n\n> **Q5: Isolate the effect of residual correction without activation sparsity**\n\n#### Thank you for the suggestion. We conducted the requested ablation to isolate the effect of the residual correction effect by evaluating two scenarios: **(1) weight-only pruning** (no activation sparsity), and **(2) dual-sparsity** (with activation sparsity).\n\n#### The results demonstrate the following:\n\n#### 1. **Residual correction is effective on its own**: DuoGPT outperforms SparseGPT even in the weight-only setting, showing that the second term in Δw (Eq. 3) provides a clear benefit.\n\n#### 2. **Activation sparsity amplifies the gain**: When activation sparsity is included, the magnitude of improvement increases by **2.9×, 1.8×, and 2.6×** respectively across the three models.\n\n#### These results confirm that our closed-form solution adapts effectively: the residual correction term compensates for calibration error from both weight pruning and activation sparsity.\n\n#### The table below shows the results of the ablation study isolating residual correction effects (50% sparsity, WikiText2 PPL)\n\n| $\\small{\\text{Scenario}}$     | $\\small{\\text{Method}}$   | $\\small{\\text{LLaMA2-7B}}$       | $\\small{\\text{LLaMA2-13B}}$      | $\\small{\\text{LLaMA3-8B}}$       |\n|------------------------------|---------------------------|-------------------------------|-------------------------------|-------------------------------|\n| $\\small{\\text{Weight-only}}$ | $\\small{\\text{SparseGPT}}$ | $\\small{7.11}$                | $\\small{6.15}$                | $\\small{9.98}$                |\n|                              | $\\small{\\textnormal{DuoGPT}}$    | $\\small{6.97\\ (\\downarrow 0.14)}$ | $\\small{6.03\\ (\\downarrow 0.12)}$ | $\\small{9.73\\ (\\downarrow 0.25)}$ |\n| $\\small{\\text{Dual-sparsity}}$ | $\\small{\\text{SparseGPT}}$ | $\\small{8.98}$                | $\\small{7.39}$                | $\\small{14.05}$               |\n|                              | $\\small{\\text{DuoGPT}}$    | $\\small{8.58\\ (\\downarrow 0.40)}$ | $\\small{7.17\\ (\\downarrow 0.22)}$ | $\\small{13.41\\ (\\downarrow 0.64)}$ |\n\n> **References**\n\n#### **[1]** Mirzadeh, Iman, et al. *\"ReLU strikes back: Exploiting activation sparsity in large language models.\"* arXiv preprint arXiv:2310.04564 (2023)\n\n#### **[2]** Sun, Mingjie, et al. *\"A simple and effective pruning approach for large language models.\"* arXiv preprint arXiv:2306.11695 (2023)."}, {"review_id": "xYJ6sNJatg", "reviewer": "Reviewer_f2i7", "summary": "This paper proposes DuoGPT to improve weight pruning and activation sparsity for large language models. By treating activation sparsity as dynamic structured weight sparsity, DuoGPT constructs a dual-sparse method. Optimal Brain Compression, activation-aware calibration and dense-model output residuals are used for accuracy preservation. Experiments demonstrate that DuoGPT outperforms SOTA structured pruning methods, by up to 9.17% higher accuracy at an iso-speedup of 1.39X.", "questions": "- **More models are needed in the experiment:**\n  The current experimental evaluation is limited to the LLaMA architecture. It remains unclear how the proposed approach would perform on other model, such as Mistral, Qwen, DeepSeek, etc.\n\n- **Lack of latency measurements:**\n  While DuoGPT has dual sparsity from both activations and model weights, this paper does not report latency results. It will be great to understand how latency is affected as well, particularly under varying batch sizes (both large and small). Including latency measurements would strengthen this work.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- **Good novelty:**\n  Introducing activation sparsity into model pruning is a novel approach. With tailor-designed techniques, DuoGPT successfully improved model pruning performances. \n\n- **Good performance:**\n  The method outperforms the SOTA by up to 9.17% accuracy with an iso-speedup of 1.4×.\n\n- **Good experiments:**\n  The paper provides comprehensive experiments, including multiple baselines and various evaluation benchmarks.\n\n\n## Weaknesses\n\n- **No GPU kernel implementation:**\n  The paper does not provide a GPU kernel implementation, which make it unclear about the efficiency performance, such as latency, in real-world deployment.", "was_revised": false, "final_justification": "", "rebuttal": "#### We thank the reviewer for the valuable comments and suggestions on additional experiments. We have tried our best to address all of them in our rebuttal.\n\n> **Q1: The current experimental evaluation is limited to the LLaMA architecture.**\n\n#### Thank you for this important suggestion. We agree that demonstrating generalizability across architectures is crucial. We have evaluated DuoGPT on multiple model families beyond LLaMA, showing consistent improvements across different architectures. We tested on Mistral, Qwen2, and OPT with varying model size in the table below with 50\\% dual-sparsity on WikiText2 PPL.\n\n| $\\small{\\text{Method}}$   | $\\small{\\text{Mist-7B}}$ | $\\small{\\text{Qwen2-7B}}$ | $\\small{\\text{Qwen2-1.5B}}$ | $\\small{\\text{OPT-125M}}$ | $\\small{\\text{OPT-1.3B}}$ |\n|---------------------------|---------------------------|-----------------------------|-------------------------------|----------------------------|----------------------------|\n| $\\small{\\text{Dense}}$     | $\\small{5.25}$           | $\\small{7.13}$              | $\\small{9.54}$                | $\\small{27.65}$            | $\\small{14.62}$            |\n| $\\small{\\text{SparseGPT}}$ | $\\small{8.40}$           | $\\small{10.91}$             | $\\small{19.50}$               | $\\small{55.08}$            | $\\small{35.25}$            |\n| $\\small{\\text{DuoGPT}}$    | $\\small{{7.91}}$  | $\\small{{10.76}}$    | $\\small{{18.11}}$      | $\\small{{51.88}}$   | $\\small{{31.70}}$   |\n\n> **Q2: Latency results.**\n\n#### Thank you for this important suggestion as wel. We provide latency analysis below, noting that our method specifically targets single-batch decoding scenarios where activation sparsity can be effectively exploited through GEMV operations—the critical use case for interactive applications.\n\n#### In the table below, we measured end-to-end latency (% of reduction compared to dense baselines) across different context lengths using the setup described in Appendix B.2, showing consistent improvements that increase with model size (24.6\\% average for 7B vs 29.2\\% for 13B). This scaling behavior reflects the memory-bound nature of single-batch decoding, where our dual-sparsity approach effectively reduces SRAM loading costs.\n\n| $\\small{\\text{Model}}$         | $\\small{\\text{ctxt-50}}$ | $\\small{\\text{ctxt-100}}$ | $\\small{\\text{ctxt-200}}$ | $\\small{\\text{ctxt-500}}$ | $\\small{\\text{Average}}$ |\n|-------------------------------|---------------------------|----------------------------|----------------------------|----------------------------|---------------------------|\n| $\\small{\\text{LLaMA-2-7B}}$   | $\\small{24.5}$ %           | $\\small{25.0}$ %            | $\\small{25.0}$ %            | $\\small{23.7}$ %            | $\\small{24.6}$ %           |\n| $\\small{\\text{LLaMA-2-13B}}$  | $\\small{29.1}$ %           | $\\small{28.1}$ %            | $\\small{31.5}$ %            | $\\small{28.0}$ %            | $\\small{29.2}$ %           |\n\n#### We note that significant system optimization opportunities exist for dual-sparse LLM workloads, and we hope this work opens promising avenues for the ML-systems community to explore.\n\n> **Q3: GPU kernel implementation.**\n\n#### Thank you for bringing up this important implementation concern. We want to clarify our current kernel support and future work directions:\n\n#### We have implemented end-to-end triton-based GPU kernels for exploiting activation sparsity, which enables the latency measurements reported in our above response and the throughput results in the main paper. The key idea of the kernel is to use `triton.language.load` to selectively load weight rows based on a bitmask representing the activation sparsity pattern. This demonstrates a lower bound of the real-world efficiency gains from our dual-sparsity approach.\n\n#### The remaining kernel to implement is for compressed storage/loading of unstructured sparse weights. However, this weight compression kernel is orthogonal to our current hardware performance results, as the primary SRAM loading efficiency gains come from the activation sparsity exploitation during GEMV operations.\n\n#### Full kernel support for weight compression represents promising future work that would provide additional HBM loading reduction benefits on top of our current demonstrated improvements. We view this as an exciting direction that could further amplify the efficiency gains of dual-sparse LLM workloads."}, {"review_id": "fWwqvB8nsc", "reviewer": "Reviewer_MRdQ", "summary": "This paper introduces DuoGPT, a training-free pruning framework for large language models (LLMs) that combines unstructured weight pruning with activation sparsity to create dual-sparse (spMspV) workloads. By reinterpreting activation sparsity as a dynamic, structured form of weight sparsity, the authors enhance the Optimal Brain Compression (OBC) framework with activation-aware calibration and dense-model residual correction. The framework is computationally efficient, scalable to billion-parameter models like LLaMA-3-70B, and calibrated within ~2 hours on a single A100 GPU.", "questions": "Thanks for this submission. Could you please address the following concerns:\n1. Since Dual spasity has a closed-form and the accuracy is usually acceptable, I wonder if there is any more concise bound of error in terms of numeric values or higher-level semantic information. I mean, something like Section 3 in this paper: https://proceedings.neurips.cc/paper_files/paper/2016/file/0e55666a4ad822e0e34299df3591d979-Paper.pdf\n2. I feel like OBC is not squeezing bits; therefore, is it orthogonal to symbol quantisations like INT4/INT8? If so, what if they are placed together?", "limitations": "N.A. now", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengthens:\n1. Novel integration of activation sparsity and unstructured pruning.\n\n2. Efficient closed-form solution with GPU-optimized implementation.\n\nWeakness:\n1. There is no discussion about the error bound.\n2. Other LLM arches may be different from LLAMA, and theoretical clarifications or more evaluations may be required", "was_revised": true, "final_justification": "My concerns are well-addressed, and I'd like to keep the score.", "rebuttal": "#### We thank the reviewer for the insightful questions and suggestions on error bound, quantization compatibility, and different LLM architectures. We have tried our best to address all of them in our rebuttal.\n\n> **Q1: Since dual sparsity has a closed-form and the accuracy is usually acceptable, I wonder if there is any more concise bound of error in terms of numeric values or higher-level semantic information.**\n\n#### Thank you for this important question and the reference. We provide a concise improvement bound quantifying DuoGPT's guaranteed loss improvement over SparseGPT below.\n\n#### **Theorem (DuoGPT Improvement Bound):** Under activation sparsity level $p_x$, DuoGPT achieves guaranteed loss improvement over SparseGPT:\n\n#### ${\\Delta}L = L_{\\text{SparseGPT}} - L_{\\text{DuoGPT}} = r \\hat{X}^T H^{-1}_{-p} \\hat{X} r^T$.\n\n#### Then, the loss improvement satisfies the lower bound:  \n#### $\\geq \\dfrac{\\alpha p_x \\sigma_r^2 C_W^2 m}{\\lambda_{\\max}(H)}$.\n\n#### where $\\alpha > 0$ is the stability constant measuring spectral gap preservation, $\\lambda_{\\max}(H)$ is the maximum eigenvalue of the Hessian $H = \\hat{X} \\hat{X}^T$, $\\sigma_r > 0$ is the activation residual magnitude constant, $C_W$ is the weight norm bound that satisfies $\\||W\\||_F^2 \\leq C_W$, and $m$ is the calibration batch size.\n\n#### **Proof Sketch:**\n\n#### The key insight is that the weight update rules of SparseGPT and DuoGPT differ only in the residual correction term (Equation 3). Therefore we have the loss of SparseGPT as\n\n#### $L_{\\text{SparseGPT}} = || -\\frac{w_p}{H^{-1}_{pp}}||^2_F$, and the loss of DuoGPT from the Equation (20) in the appendix.\n\n#### When expanding both loss functions and subtracting $L_{\\text{DuoGPT}}$ from $L_{\\text{SparseGPT}}$, all terms cancel except one quadratic form: $r \\hat{X}^T H^{-1}_{-p} \\hat{X} r^T$.\n\n#### Since $H^{-1}_{-p}$ is positive definite, this guarantees $\\Delta L \\geq 0$ always. \n\n#### For the lower bound, we use the fact that $H^{-1}_{-p}$ is obtained from $H^{-1}$ via Gaussian elimination, which follows the spectral bound that\n\n#### $\\lambda_{\\min}(H^{-1}_{-p})$ is greater or equal to\n\n#### $\\frac{\\alpha}{\\lambda_{\\max}(H)}$. After applying the quadratic form bound where\n\n#### $r \\hat{X}^T H^{-1}_{-p} \\hat{X} r^T$ is greater or equal to\n\n#### $\\lambda_{\\min}(H^{-1}_{-p}) \\cdot \\||r \\hat{X}^T\\||^2$. Using activation sparsity properties yields our result.\n\n#### **Empirical Validation:** Our bound predicts meaningful improvement that scales linearly with activation sparsity $p_x$. Experimentally, we observe consistent improvement over SparseGPT across all models and sparsity levels in Table 1 and Table 3a, confirming the bound's validity. The linear scaling with $p_x$ explains why higher dual-sparsity yields greater improvements.\n\n#### We will add discussion of the bound to the revised version of the paper. The complete formal proof will be included in Appendix of the revised manuscript. We will make sure to reference the provided paper as our theoretical backup.\n\n> **Q2: Compatible with weight quantization?**\n\n#### Thank you for this question! You are correct about the orthogonal relationship between quantization and OBC-based pruning - they operate on different aspects of model compression and can be combined effectively.\n\n#### We evaluated the combination of INT8/INT4 uniform weight (group size 128) quantization with 50% dual-sparse DuoGPT on LLaMA-2-7B and 13B (WikiText2 PPL).\n\n#### DuoGPT maintains its performance advantage over SparseGPT across all precision levels, demonstrating successful orthogonal combination. Notably, our method shows better robustness to INT4 quantization, suggesting that the activation-sparsity-aware calibration helps preserve model quality under aggressive compression.\n\n#### This multi-dimensional compression approach is very promising for practical edge deployment-combining INT4 precision with 50% dual-sparsity achieves roughly 6% model density for SRAM loading, a compression ratio difficult to attain through any single compression method alone.\n\n| $\\small{\\text{Model}}$         | $\\small{\\text{Method}}$     | $\\small{\\text{FP16}}$ | $\\small{\\text{INT8}}$ | $\\small{\\text{INT4}}$ |\n|-------------------------------|------------------------------|------------------------|------------------------|------------------------|\n| $\\small{\\text{LLaMA-2-7B}}$   | $\\small{\\text{SparseGPT}}$  | $\\small{8.98}$         | $\\small{9.02}$         | $\\small{14.23}$        |\n|                               | $\\small{\\text{DuoGPT}}$      | $\\small{8.58}$         | $\\small{8.58}$         | $\\small{13.78}$        |\n| $\\small{\\text{LLaMA-2-13B}}$  | $\\small{\\text{SparseGPT}}$  | $\\small{7.39}$         | $\\small{7.41}$         | $\\small{9.95}$         |\n|                               | $\\small{\\text{DuoGPT}}$      | $\\small{7.17}$         | $\\small{7.18}$         | $\\small{9.39}$         |\n\n\n> **Q3: Different LLM architectures**\n\n#### We acknowledge this important concern about different LLM architectures. To address this, we have evaluated DuoGPT across multiple model families including Mistral, Qwen2, and OPT (as shown in our response to **Reviewer B, Q1**). The results demonstrate consistent improvements across all tested architectures, confirming that our theoretical framework and empirical benefits generalize beyond LLaMA models."}]

Author rebuttal:

Meta-review: This paper proposes to combine "dynamic" activation sparsity (which can be seen as per-token structured sparsity) with "static" unstructured weight sparsity. This involves using a Hessian-based correction mechanism (i.e., as in OBC/SparseGPT) that takes the activation sparsity into account in the weight pruning/recalibration stage. The approach is tested on the LLaMA family of models where it is found to result in a better sparsity-performance frontier than baseline methods such as SparseGPT/CATS/TEAL.

This is a well-executed paper that demonstrates solid improvements over strong baselines. The authors also provide a Triton-based implementation to enable actual wallclock speedups. On the negative side, there are some concerns with regard to novelty (in some sense, it is more or less just combining SparseGPT with CATS/TEAL), as well as whether this approach would generalize to other families of models. During the rebuttal phase, the authors showed that this method generalizes to non-LLaMA families as well.

Overall, despite some concerns about novelty, I believe that this is a solid contribution that would be of benefit to the community.
Decision: Accept (poster)