Title: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Compression

Pruning

SSMs

Hybrid LLM
Abstract: Hybrid language models that combine Attention and State Space Models (SSMs) have been shown to achieve state-of-the-art accuracy and runtime performance. Recent work has also demonstrated that applying pruning and distillation to Attention-only models yields smaller, more accurate models at a fraction of the training cost. In this work, we explore the effectiveness of compressing Hybrid architectures. To this end, we introduce a novel group-aware pruning method for Mamba layers that preserves the structural integrity of SSM blocks and their sequence modeling capabilities. We combine this method with FFN, embedding dimension, and layer pruning, along with knowledge distillation-based retraining to obtain a unified compression recipe for hybrid models. Using this recipe, we compress the Nemotron-H 8B Hybrid model down to 4B parameters with up to $40\times$ fewer training tokens compared to similarly-sized models. The resulting model surpasses the accuracy of similarly-sized models while achieving $\sim2\times$ faster inference throughput, significantly advancing the Pareto frontier.


Peer reviews:
comments: SUMMARY:
This paper explores width, depth, and head pruning for hybrid Mamba architectures. In particular, it proposes a novel method for pruning SSM (State Space Model) modules.

STRENGTHS AND WEAKNESSES:
### **Strengths**:

- This is the first work, to the best of my knowledge, that conducts an in-depth analysis of different pruning strategies for the Mamba architecture. The pruning strategy is systematically explored across multiple architectural axes (depth, width, and heads).



### **Weaknesses**:
- The paper lacks sufficient **discussion of related work** and **stronger baseline methods**, such as [2]. Including these would improve the completeness of the evaluation.
- In addition, the **experimental scope is limited**—the evaluation is conducted solely on **Nemotron-H-8B-Base**, which makes it difficult to assess the general applicability of the proposed method. Broader experiments on additional models are recommended.

QUESTIONS:
I am confused about the **Group-Aware Head Permutation Constraints** and **Head Channel Consistency**. Prior work [1] suggests that the channel dimension in the Mamba architecture exhibits **permutation invariance**. Why, then, is it necessary to maintain **uniform pruning across all heads within a group**, rather than allowing permutation of heads across groups?

[1] Hung-Yueh Chiang et al.,  Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models

LIMITATIONS:
The authors transparently acknowledge all of the limitations I was able to identify within the paper itself.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a group-aware pruning method for the Mamba layer. It combines Mamba pruning with pruning of other network layers to produce smaller and more accurate models while reducing training costs.\nThis paper uses this method to compress the parameters of the Nemotron-H 8B hybrid model, reducing training tags, increasing throughput, and improving accuracy compared to models with the same parameters.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis paper focuses on pruning and distillation, and the proposed pruning method is intuitively reasonable. A relatively detailed experiment and analysis are conducted, and 7 findings are proposed and corresponding experimental results are provided. The compressed model of Nemotron-H 8B also achieves improvements over the same parameter model in knowledge, mathematics, encoding, common sense reasoning, and reading comprehension tasks.\n\n**Weaknesses**\n\nThis paper only distills Nemotron-H 8B, which makes people wonder whether the pruning scheme can be extended. I hope the authors can provide more than one compressed model and show similar results. Does the Mamba layer limit the range of LLMs that can be pruned?\nIn the main experiment, the Nemotron-H-4B compressed model is only used for comparison with the non-Nemotron series baseline. In order to eliminate the interference of model quality, can you provide the comparison results of the compressed model with the model with smaller parameters in the same series? For example, compress Qwen-7B to 3B and compare the compressed model with Qwen-3B?\nThe author mentioned that the compressed model achieved 2 times faster inference throughput, and experiments and analysis of this conclusion are needed.\n\nQUESTIONS:\n1. Can the pruning method show similar effects on models other than Nemotron-H 8B?\n2. Comparison of the compressed model with the model of the same series with smaller parameters.\n3. Experiments and analysis of inference throughput.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces a ​​compression framework​ for hybrid Transformer-State Space Models, the innovations include:
- A group-aware pruning method for Mamba layers that preserves SSM block structure and sequence modeling capabilities.
- A unified hybrid pruning recipe

The proposed method compresses the Nemotron-H 8B model to 4B parameters with much less training tokens and comparable performances.

STRENGTHS AND WEAKNESSES:
strength
- The paper is well written and well-motivated.
- The 7 findings provide meaningful and comprehensive guidance for SSM-based model compression in the community. 
- The performance and efficiency of the final model is well evaluated and achieves strong performances.

QUESTIONS:
- How knowledge distillation affects the performances? Results with pruning only before distillation can be ablated to showcase the effectiveness of proposed pruning techniques.
- I wonder the generalization ability of this method, what will the performance be if the proposed method is applied on other open-sourced models such as jamba.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a group-aware pruning method for the Mamba layers, which preserves the structural integrity of SSM blocks and their sequence modeling capabilities. The authors combine this with embedding, FFN, and layer pruning to further reduce the parameter count, and they conduct extensive ablations to study the effect of pruning different components on accuracy, latency, and throughput — while keeping the total parameter count fixed at 4B. The proposed pruning method enables compression model by 50% with minimal drop in performance.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. Hybrid attention is critical for long context performance; however, pruning hybrid architecture is not well studied. This paper presents a compression strategy that combines SSM pruning with the pruning of other network components such as FFN neurons, embedding channels, and network depth. Authors also conducted an in-depth architectural search for pruning different components (different layers, depth vs width, etc). \n\n2. As demonstrated in Figure 3, the proposed group-aware pruning method for the Mamba layer preserves the structural representation of the Mamba blocks and provides a simple method for pruning hybrid LLMs.\n\n3.  Extensive ablations are conducted in Section 3.1  to compare different pruning strategies and the impact of pruning different components on validation loss, latency and throughput (Table 1). \n\n4. As shown in Table 2, the proposed pruning strategy achieves 50% compression rate while retaining  over 96% of the original 8B model’s accuracy, while improving throughput by ∼1.4x – demonstrating the efficacy of the method.\n\n5. Pushes Pareto-frontier for throughput, though results are limited to the Nemotron family.\n\n### Weaknesses\n\n1. The major limitation of this work (as the authors also noted) is that all the experiments are conducted with the Nematron model; it is not clear if the observation and analysis will hold true for different model families. \n\n2. The model performance deteriorates after pruning, which is later recovered using distillation. \n\n3. There is no ablation on pruning heuristic used in sec 2.2. \n\n4. No discussion on the potential impact of pruning strategy proposed on the model bias, as prior work has highlighted that model compression exacerbates model bias.[1].\n\n[1]. Gonçalves and Strubell, Understanding the Effect of Model Compression on Social Bias in Large Language Models\n\nQUESTIONS:\n1. How well does the proposed pruning strategy generalize for different family architectures? \n\n2. How does the pruning affect the model bias for hybrid LLMs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-23"}