Title: PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Channel Permutation

N:M Sparsity
Abstract: Channel permutation is a powerful technique for enhancing the accuracy of N:M sparse models by reordering the channels of weight matrices to prioritize the retention of important weights. 
However, traditional channel permutation methods rely on handcrafted quality metrics, which often fail to accurately capture the true impact of pruning on model performance. 
To address this limitation, we propose PermLLM, a novel post-training pruning framework that introduces learnable channel permutation (LCP) for N:M sparsity. 
LCP leverages Sinkhorn normalization to transform discrete permutation matrices into differentiable soft permutation matrices, enabling end-to-end optimization. 
Additionally, PermLLM incorporates an efficient block-wise channel permutation strategy, which significantly reduces the number of learnable parameters and computational complexity. 
PermLLM seamlessly integrates with existing one-shot pruning methods to adaptively optimize channel permutations, effectively mitigating pruning-induced errors. 
Extensive experiments on the LLaMA series, Qwen, and OPT models demonstrate that PermLLM achieves superior performance in optimizing N:M sparse models.


Peer reviews:
comments: SUMMARY:
The paper proposes PermLLM, a novel framework for N:M sparse pruning of large language models (LLMs), introducing learnable channel permutation (LCP) to optimize sparsity and improve model performance post-pruning.

STRENGTHS AND WEAKNESSES:
**Strengths**:

1.The method can seamlessly integrate with existing one-shot pruning techniques like Wanda and RIA, extending their capabilities with adaptive channel optimization.

2.Extensive experiments across multiple models (LLaMA, Qwen, OPT) demonstrate the effectiveness of PermLLM in improving pruning results.

**Weaknesses**:

1.The experimental scope is limited, as only 2:4 semi-structured pruning experiments were conducted, which restricts its generalizability.

QUESTIONS:
1.All evaluations are performed under 50% sparsity (2:4 or 4:8). Could the performance be further compared under different sparsity levels, such as 1:4 and 3:4?

2.The experimental results in Table 2 are incomplete. A more comprehensive comparison using different methods on the same model should be conducted.

3.It would be beneficial to include some of the latest unstructured pruning methods, such as Pruner-Zero [1] and MaskLLM [2].

[1] Dong P, Li L, Tang Z, et al. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. arXiv preprint arXiv:2406.02924, 2024.

[2] Fang G, Yin H, Muralidharan S, et al. MaskLLM: Learnable semi-structured sparsity for large language models. arXiv preprint arXiv:2409.17481, 2024.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPermLLM is a post-training pruning framework that makes *channel permutation itself learnable*. The core idea is to relax the discrete permutation matrix into a soft, doubly-stochastic matrix produced by Sinkhorn normalization, then “harden” it back to a true permutation with the Hungarian algorithm while passing gradients through a straight-through estimator. To keep the search space tractable for LLM-scale layers, the permutation is applied **block-wise**, cutting the learnable parameters from $C_{\\text{in}}^{2}$ to $C_{\\text{in}}B$ and reducing the Hungarian step from $O(C_{\\text{in}}^{3})$ to $O(C_{\\text{in}}B^{2})$. Plugged into one-shot N:M pruning schemes such as Wanda and RIA, the learnable permutation is trained to minimize a cosine loss between dense and sparse outputs instead of maximizing a handcrafted importance score. On seven open-source LLMs (OPT-6.7B, LLaMA-1/2/3.1 7–13B, Qwen-2.5 7B) the 2:4 sparse models show consistent gains: perplexity on WikiText-2 drops from 11.59 to **9.41** for LLaMA-7B (vs. Wanda) and average zero-shot accuracy rises by 3–5 pp across five tasks. A custom CUDA kernel makes permutation overhead negligible (0.039 ms) and still delivers 1.67× faster inference than the dense model. Figure 1 illustrates how a naive “max-score” permutation can actually *hurt* accuracy, motivating the learnable approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \n* First to cast channel permutation as a differentiable layer for N:M sparsity and integrate it with existing one-shot pruning methods, replacing brittle heuristics with end-to-end loss minimization.  \n* Sinkhorn–Hungarian–STE pipeline is mathematically clean, and the block-wise design balances search power and cost.  \n* Broad experimental coverage (7 LLMs, language modeling plus reasoning tasks) with reproducible gains over SparseGPT, Wanda-CP, and RIA-CP in both 2:4 and 4:8 sparsity.  \n* Practical implementation: bespoke kernel yields an 84× speed-up over PyTorch permutation and preserves overall sparsity speed-ups.\n\n**Weaknesses**  \n* Training permutations adds wall-time (≈2.5 h for 7B, 5.5 h for 13B on multiple A100s), reducing the appeal of “one-shot” pruning.  \n* Block size is fixed at 64; larger or adaptive blocks are unexplored.  \n* Speed-ups are throughput-only; memory footprint and launch overhead are not discussed.  \n* Analysis focuses on 2:4 sparsity; 4:8 results are promising but less detailed.  \n* Calibration uses only 128 C4 samples, so robustness to domain shifts is unclear.\n\nQUESTIONS:\n1. How do accuracy and training time vary with block size beyond the single ablation shown? Could adaptive per-layer block sizes yield further gains?  \n2. Can learnable permutation be limited to the most sensitive layers (as hinted by a “partial PermLLM” experiment) without retraining dense layers and still match full performance?  \n3. Have the pruned models been evaluated on out-of-domain datasets or downstream fine-tuning to ensure the learned permutations do not overfit the calibration set?  \n4. Since many deployments combine sparsity with 4-bit or 8-bit quantization, does PermLLM remain effective when applied after GPTQ or similar schemes?  \n5. The custom CUDA kernel targets NVIDIA sparse tensor cores. How would the approach translate to hardware lacking these primitives (AMD MI-series, Intel Gaudi)?\n\nLIMITATIONS:\nPermLLM still requires a non-trivial GPU-heavy optimization loop—unlike heuristic channel permutation it is **not** a free post-processing step. Even with block-wise sharing, the method introduces additional parameters and hyperparameters (temperature schedule, Sinkhorn iterations) that may need tuning per model. The reported speed-up (≈1.67×) is far below the theoretical 2× for 2:4 sparsity, indicating residual inefficiencies. Accuracy gains, though consistent, are sometimes modest (<2 pp) on the hardest reasoning tasks, suggesting diminishing returns at higher sparsity. The method has only been validated on decoder-only transformer LLMs; applicability to encoder–decoder or vision transformers remains to be proven.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}

comments: SUMMARY:
PermLLM addresses the limitations of heuristic-driven channel permutation in N:M sparse LLMs by proposing a novel learnable channel permutation (LCP) framework that optimizes permutations end-to-end to minimize pruning-induced output errors. As a plug-in enhancement to existing one-shot methods (e.g., Wanda), PermLLM achieves outstanding results across LLaMA, OPT, and Qwen families—reducing perplexity degradation and improving zero-shot accuracy.

STRENGTHS AND WEAKNESSES:
### Strengths

The paper compellingly argues against handcrafted metrics (Fig. 1) and introduces a novel framework for learnable channel permutation (LCP) that directly minimizes pruning error, which is an original contribution to semi-structured pruning.

The block-wise permutation strategy is a clever solution to make LCP scalable for LLMs, reducing parameter count and conputational complexity.

The method is thoroughly evaluated on a wide range of modern LLMs, mostly outperforming baselines in both perplexity and zero-shot tasks.


### Weaknesses

The choice of block size (64) is justified anecdotally rather than empirically.

 An ablation study showing the trade-off between block size, performance, and training cost is missing.

The experimental results are not optimal in every setting.

QUESTIONS:
How robust is the learned permutation to the choice and size of the calibration dataset? Does performance vary significantly with different calibration data?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nChannel permutations have been shown to work well at reducing the lost weight magnitude or other particular metric, but these metrics do not always maximize model quality.  Further, existing work uses approximate heuristics or computationally expensive search procedures to find good permutations for N:M sparsity.  PermLLM addresses these two shortcomings by directly learning the permutations.  First, a standard permutation matrices are relaxed into \"soft\" permutation matrices in order to make them differentiable, using Sinkhorn normalization to convert the learned matrix into a doubly stochastic matrix, which can be hardened into a true permutation for forward propagation.  To reduce the complexity and number of learnable parameters, the permutation matrices are simplified into block-wise permutation matrices.  A soft mask is used in the backwards pass to allow differentiability when learning the permutations (since they will change the mask applied), and cosine similarity between the dense and sparse model outputs is used as the loss function.  Reasonably broad evaluations are performed, showing meaningful average improvements over the considered baselines, and top-quality results on most model/task pairs.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThe idea is well-motivated; figure 1 clearly shows how existing metrics are not ideal to maximize model quality.  The approach is also sound; each piece of the solution is clearly motivated and explained, and the effect of the whole process is proven successful by the evaluations.  I also appreciate the runtime analysis, as well as the ablation study (though it is somewhat limited).  I have little doubt that a dedicated practitioner could reproduce the results from studying the descriptions and equations, and that the reported model quality improvements would likely generalize beyond the presented results.\n\t\n## Weaknesses\nI want to believe in this submission's superiority over past work, but there are some missed references and baselines that affect the significance, originality, and quality of the submission.\n\nSome missing references should be included:\n- Mahajan et al. [1] formulate the permutation search as a discrete optimization problem.\n- Lyu et al. [2] learn permutation matrices directly.  This one is particularly relevant, and the authors should provide a comparison of this approach and the proposed approach, and, ideally, empirical comparisons.\n\nA simple baseline would be the approach from [42] applied to e.g. SparseGPT scores or Wanda importance metrics (as noted in that work, the technique is applicable to any metric, not just magnitude).  The authors note on line 45 that the large search space renders this approach infeasible for large sizes, but [42] includes less-expensive settings than the default.  Using the simplest from [42]'s Table 3 and the available code on a single A100, searching for a good permutation for a random 4096x4096 matrix (such as Llama2 7B's attention output projection) took only 125 seconds.  I continued timing random matrices for the other layers and came up with these results:\n\n| Layer | Size | Duration (s)|\n| ------ | ------ | -------|\n| QKV | 4096x12288 | 297 |\n| PROJ | 4096x4096 | 125 |\n| up| 4096x11008 | 268 |\n| gate | 4096x11008 | 268 |\n| down | 11008x4096 | 1318 |\n\nReplicated for 32 blocks, the total projected time to search for permutations for all of Llama2 7B is 20.25 hours.  If parallelized over 4 GPUs to match the submission's resources for this model, this is \"only\" double the 2.5 hours reported on line 298.  It may indeed be double the cost, but certainly not \"prohibitively high computation complexity\" as claimed on line 46.  If, as suggested above, these permutations are found using a higher-quality importance metric than weight magnitude, they could prove competitive with PermLLM.  Or, they may not.  I believe this is a useful baseline to correctly determine the significance of PermLLM as a technique as well as improve the quality of the submission.\n\nI'm also concerned about the interaction with another, non-orthogonal technique that seeks to reduce the quality impact of N:M sparsity: rotations.  They've been successfully used for quantization [3,4] as well as N:M sparsity[5].  It's not clear that PermLLM composes with or outperforms this technique.\n\nFinally, some small issues reduce the clarity and quality of the submission:\n- On line 269, the authors suggest that the additional permutation to handle residual paths has a negligible impact on deployment efficiency, citing [58].  In the next sentence, though, the authors describe a custom CUDA kernel to reduce the overhead.  Why do we care about reducing a negligible overhead?  (The experimental results on line 271 probably belong in Section 5, too.)\n- Sweeping the number of iterations used in Sinkhorn normalization in Section 5.3 would be informative.  (5 is better than 0, but is 5 enough?  More than enough?)\n- An ablation of the block size used (32, 64, 128) would be useful to see how sensitive the outcome is to this hyperparameter.\n- What software is used to gather the runtimes in Table 3?\n\nReferences\n1. Mahajan, M., Hwu, WM. & Nagi, R. Determining optimal channel partition for 2:4 fine grained structured sparsity. Optim Lett 18, 2079–2090 (2024). https://doi.org/10.1007/s11590-023-02084-8\n2. Jiancheng Lyu, Shuai Zhang, Yingyong Qi, and Jack Xin. AutoShuffleNet: Learning Permutation Matrices via an Exact Lipschitz Continuous Penalty in Deep Convolutional Neural Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20). https://doi.org/10.1145/3394486.3403103\n3. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, James Hensman, QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs, NeurIPS 2024\n4. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, Tijmen Blankevoort, SpinQuant: LLM Quantization with Learned Rotations, ICLR 2025\n5. Anonymous, ROTPRUNER: LARGE LANGUAGE MODEL PRUNING IN ROTATED SPACE  https://openreview.net/pdf?id=wV9iMiyQcc\n\nQUESTIONS:\n1. How does PermLLM compare to previous work (say, [42]) operating with a better importance metric than magnitude (say, Wanda's importance metric)?  \n2. How does PermLLM compare to learned rotations (RotPruner)?  Is there room for both techniques?\n3. How does PermLLM compare to past approaches to directly learning permutations (2, above)\n4. How does PermLLM perform with varying block sizes (32, 64, 128 (if possible!))?\n\nAddressing these questions will directly affect the quality, originality, and (most importantly) the significance of the submission.  Results on a single model should be enough to show the important trends.\n\nLIMITATIONS:\nLimitations are discussed in the appendix, but should really be in the main content.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}