Title: DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Weight Pruning

Dual-sparsity

Activation Sparsity

Efficient Large Language Models

Post-training Calibration

Model Compression
Abstract: Large language models (LLMs) deliver strong performance but are difficult to deploy due to high memory and compute costs. While pruning reduces these demands, most methods ignore activation sparsity observed at runtime. We reinterpret activation sparsity as dynamic structured weight sparsity and propose DuoGPT, a unified framework that constructs dual-sparse (spMspV) workloads by combining unstructured weight pruning with activation sparsity. To preserve accuracy, we extend the Optimal Brain Compression (OBC) framework with activation-aware calibration and introduce output residuals from the dense model as correction terms. We further optimize the solution for efficient GPU execution, enabling scalability to billion-parameter LLMs. Evaluations on LLaMA-2 and LLaMA-3 show that DuoGPT outperforms state-of-the-art structured pruning methods by up to 9.17\% accuracy at an iso-speedup of 1.39$\times$ compared to the baseline dense model. Code is available at GitHub.


Peer reviews:
comments: SUMMARY:
This paper introduces DuoGPT, a training-free pruning framework for large language models (LLMs) that combines unstructured weight pruning with activation sparsity to create dual-sparse (spMspV) workloads. By reinterpreting activation sparsity as a dynamic, structured form of weight sparsity, the authors enhance the Optimal Brain Compression (OBC) framework with activation-aware calibration and dense-model residual correction. The framework is computationally efficient, scalable to billion-parameter models like LLaMA-3-70B, and calibrated within ~2 hours on a single A100 GPU.

STRENGTHS AND WEAKNESSES:
Strengthens:
1. Novel integration of activation sparsity and unstructured pruning.

2. Efficient closed-form solution with GPU-optimized implementation.

Weakness:
1. There is no discussion about the error bound.
2. Other LLM arches may be different from LLAMA, and theoretical clarifications or more evaluations may be required

QUESTIONS:
Thanks for this submission. Could you please address the following concerns:
1. Since Dual spasity has a closed-form and the accuracy is usually acceptable, I wonder if there is any more concise bound of error in terms of numeric values or higher-level semantic information. I mean, something like Section 3 in this paper: https://proceedings.neurips.cc/paper_files/paper/2016/file/0e55666a4ad822e0e34299df3591d979-Paper.pdf
2. I feel like OBC is not squeezing bits; therefore, is it orthogonal to symbol quantisations like INT4/INT8? If so, what if they are placed together?

LIMITATIONS:
N.A. now
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-02

comments: SUMMARY:
This paper proposes DuoGPT to improve weight pruning and activation sparsity for large language models. By treating activation sparsity as dynamic structured weight sparsity, DuoGPT constructs a dual-sparse method. Optimal Brain Compression, activation-aware calibration and dense-model output residuals are used for accuracy preservation. Experiments demonstrate that DuoGPT outperforms SOTA structured pruning methods, by up to 9.17% higher accuracy at an iso-speedup of 1.39X.

STRENGTHS AND WEAKNESSES:
## Strengths

- **Good novelty:**
  Introducing activation sparsity into model pruning is a novel approach. With tailor-designed techniques, DuoGPT successfully improved model pruning performances. 

- **Good performance:**
  The method outperforms the SOTA by up to 9.17% accuracy with an iso-speedup of 1.4×.

- **Good experiments:**
  The paper provides comprehensive experiments, including multiple baselines and various evaluation benchmarks.


## Weaknesses

- **No GPU kernel implementation:**
  The paper does not provide a GPU kernel implementation, which make it unclear about the efficiency performance, such as latency, in real-world deployment.

QUESTIONS:
- **More models are needed in the experiment:**
  The current experimental evaluation is limited to the LLaMA architecture. It remains unclear how the proposed approach would perform on other model, such as Mistral, Qwen, DeepSeek, etc.

- **Lack of latency measurements:**
  While DuoGPT has dual sparsity from both activations and model weights, this paper does not report latency results. It will be great to understand how latency is affected as well, particularly under varying batch sizes (both large and small). Including latency measurements would strengthen this work.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DuoGPT, a compression method for LLMs that combines structured pruning and unstructured pruning using activation sparsity and weight pruning. It extend the OBC framework with residual correction to accommodate attention sparsity mechanisms. Experimental results demonstrate speedup over dense LLMs, with comparisons against baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper proposes a framework that combines structured and unstructured pruning to balance performance and efficiency. It includes adaptive modifications to prior methods like OBC, which is practical.\n- Experiments demonstrates the effectiveness and applicability of DuoGPT.\n\n**Weaknesses**\n\n- The method of collecting activation statistics from calibration data and dynamically activating components during decoding is similar to CATS [1], which is also acknowledged in the paper and uses a threshold-based activation sparsity mechanism. A more detailed comparison in both methodology and performance may improve the clarity and originality of this work.\n- Why not use ReLU for activation sparsity, which offers a more straightforward and efficient way to reduce computation?\n- Combining structured and unstructured pruning seems simple and has been explored in prior works such as STUN [2]. Moreover, the motivation for introducing activation sparsity within the structured pruning component is unclear, since many structured pruning methods are already compatible with unstructured sparsity.\n- Some core results (e.g., Table 1, LLaMA-2-70B) show only a marginal improvement in perplexity (e.g., 0.07 better than SparseGPT), while requiring significantly more compute (e.g., +0.6 GPU hours for calibration). The effectiveness of the method seems heavily dependent on additional computation.\n- The comparison with SparseGPT does not isolate the effect of residual correction to OBC, as DuoGPT also includes activation sparsity. A separate ablation would help clarify its contribution.\n\n[1] CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models, COLM 2024\n\n[2] STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning, arxiv 2024.9\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}