PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Models
Authors: Lancheng Zou
Shuo Yin
Zehua Pei
Tsung-Yi Ho
Farzan Farnia
Bei Yu
Venue: NeurIPS 2025 poster ()
Keywords: Channel Permutation
N:M Sparsity
Primary area: deep_learning

Abstract: Channel permutation is a powerful technique for enhancing the accuracy of N:M sparse models by reordering the channels of weight matrices to prioritize the retention of important weights. 
However, traditional channel permutation methods rely on handcrafted quality metrics, which often fail to accurately capture the true impact of pruning on model performance. 
To address this limitation, we propose PermLLM, a novel post-training pruning framework that introduces learnable channel permutation (LCP) for N:M sparsity. 
LCP leverages Sinkhorn normalization to transform discrete permutation matrices into differentiable soft permutation matrices, enabling end-to-end optimization. 
Additionally, PermLLM incorporates an efficient block-wise channel permutation strategy, which significantly reduces the number of learnable parameters and computational complexity. 
PermLLM seamlessly integrates with existing one-shot pruning methods to adaptively optimize channel permutations, effectively mitigating pruning-induced errors. 
Extensive experiments on the LLaMA series, Qwen, and OPT models demonstrate that PermLLM achieves superior performance in optimizing N:M sparse models.

PermLLM: Learnable Channel Permutation for N:M Sparse Large Language Models
Lancheng Z Shuo Yin',
Zehua Pei', Tsung-Yi Yi Ho', Farzan Farnia', and Bei Yu!
The Chinese University of Hong Kong
Abstract
Channel permutation is a powerful technique for enhancing the accuracy of N:M sparse models by reordering the channels of weight matrices to prioritize the re- tention of important weights. However, traditional channel permutation methods rely on handcrafted quality metrics, which often fail to accurately capture the true impact of pruning on model performance. To address this limitation, we propose PermLLM, novel post-training pruning framework that introduces learnable chan- nel permutation (LCP) for N:M sparsity. LCP leverages Sinkhorn normalization transform discrete permutation matrices into differentiable soft permutation matri- ces, enabling end-to-end optimization. Additionally, PermLLM incorporates an efficient block-wiss channel permutation strategy. which significantly reduces the number of learnable parameters and computational complexity. PermLLM seam- lessly integrates with existing one-shot pruning methods to adaptively optimize channel permutations, effectively mitigating pruning-induced induced errors. Extensive experiments the LLaMA series, Qwen, and OPT models demonstrate that PermLLM achieves superior performance in optimizing N:M sparse models. The code is available at wtthhtt://// tutt..b.... com http://anchennzoo/Penln....
Introduction
The rapid advancements in large language models (LLMs) 6 61 52
have led to a notable enhancement in their capabilities across broad range domains. However, the growing scale of LLMs presents substantial challenges for efficient deployment. To address these challenges, model compression techniques, such as quantization promising solutions to reduce memory usage and computational overhead.
[57
and pruning 50 offer
In this paper, focus network pruning (28 22 21 particularly semi-structured pruning 46 42
The core idea of network pruning is to eliminate redundancies within the model by preserving only the essential weights while setting the less important ones zero. Semi-structured pruning takes this a step further by enforcing N:M sparsity, where N out of every M consecutive elements are set to zero. The N:M sparsity pattern is natively supported by Sparse Tensor Core in NVIDIA GPUs [45) to achieve speed-up, which makes semi-structured pruning a practical approach for efficient model inference.
Recent studies on LLM pruning primarily focus on designing a better pruning metric to obtain higher-quality masks to improve the accuracy of the sparse models [15] 50 62 RIA [62] introduces a novel pruning metric that avoids channel corruption while accounting for the effect of activations. Additionally, it proposes two-stage channel permutation strategy to maximize the sum of retained weight importance, which serves as the quality metric to evaluate channel permutation solution. However, is important note that discrepancy may exist between the handcrafted quality metric and the actual impact on output loss, as illustrated in Figure]] [ Moreover, it fails to fully capture the
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Original
Max. Score Channal Permutation
Min. Losa Channel Permutation
-4
W
LosE:14.375
W
Loss:475
21
21
10
-15
-25
23 16
Weight
Unpraned Weight
Figure 1: Effects of different channel permutation strategies on the outputs. Channel order is in purple. We use magnitude pruning 21 for 2:4 sparsity in this example. Score S denotes the sum of retained weight importance, which is used as the quality metric for channel permutation 46 62 Loss the mean square error between the original output y and the output the pruned one. The output loss of direct 2:4 sparsity (i.e., without channel permutation) is 12.375. The results demonstrate that channel permutation which maximizes the score may lead to performance degradation.
complex inter-layer interactions, thereby missing opportunities compensate for pruning errors and improve the overall performance of the sparse model.
To overcome the limitations of prior channel permutation methods, channel permutation (LCP) for N:M sparsity. Unlike previous approaches that rely on handcrafted quality metrics optimization proxies, the proposed post-training framework, PermLLM, directly minimizes the output errors between the dense model and the sparse model.
the first to present learnable
However, achieving feasible and practical permutation learning for pruning presents two major challenges: (1) the discrete nature and strict combinatorial constraints of permutation matrices render them non-differentiaale. hindering effective optimization; (2) the vast solution space of permutations, particularly in LLMs with high-ddmensional weight matrices, results in prohibitively high computational complexity.
To address these challenges, we first relax hard permutation matrices into soft permutation matrices using Sinkhorn normalization [48] enabling gradient-bart based optimization. Then, we introduce an efficient block-wise channel permutation strategy, which significantly reduces the number of learnable parameters and computational overhead. PermLLM is fully compatible with existing efficient one- shot pruning methods, such as Wanda (50) and RIA [62, enabling pruning-aware permutation learning that adaptively minimizes pruning-induces errors. Moreover, a customized CUDA kernel is developed to accelerate the channel permutation operation, achieving a significant speedup compared to the Pytorch implementation. Extensive experiments underscore the effectiveness of PermLLM. demonstrating its ability to enhance the performance of cxisting one-shot pruning methods across various LLMs, particularly for updated models such LLaMA-3.1 and Qwen-2.5.
2 Preliminaries
2.1 Large Language Models Pruning
The effectiveness of network pruning 22 20 21
has garnered significant attention from researchers, prompting extensive exploration for LLM pruning.
23
Based on the granularity of pruning. prior works can be categorized into three types: structured 38| [4], semi-structured pruning [62, 14] and unstructured pruning [15] 50
pruning [6 3 49
55,
Unstructured pruning is the most flexible approach, as it is not constrained by specific patterns. This flexibility often leads to improved accuracy; however, it comes at the expense of limited efficiency gains. contrast, structured pruning removes weights at coarse-grained
level, such channels [66,, layers 38 7 blocks 44 44), thereby enabling more substantial

---

improvements in computational efficiency. However, the structural removal often leads to significant accuracy degradation, necessitating retraining or fine-tuning to mitigate muning-induced errors. Semi-structured pruning serves an intermediate approach, introducing hardware-friendl patterns, such as N:M sparsity [42] which retains only N zero values within each group of M values. This method achieves compromise between the acceleration benefits of structured pruning and the flexibility of fine-grained sparsity.
pruning before training (PBT) 22 54]. pruning during training (PDP) [33 32 and post-training pruning (PTP) 28 22 PBT and PDP
In general, there are three pipelines to obtain sparse model 8
typically demand substantial training efforts, which makes PTP the widely adopted pipeline for LLM pruning due to its lower computational cost. The objective of PTP can be formulated follows:
arg min M
WX (MO () 1
S.t. | S k,
(1)
where W E RCouu X represents the pre- pre-trained weight with Cout output channels and Cnn input channels. The goal of PTP is to determine mask M that minimizes the reconstruction error under the given input X from calibration dataset and specific sparsity constraints (e.g., sparsity ratio and pruning granularity).
2.2
N:M Sparsity
leverages Sparse Tensor Core accelerate model inference with N:M sparsity [42). For instance, compressing the model with 2:4 sparsity can theoretically achieve 2x increase in compute throughput for sparse matrix multiplication compared to its dense counterpart. Thus, this approach has garnered significant attention for its ability to improve computational efficiency while maintaining model accuracy.
NVIDIA Ampere architecture
45
introduces a one-shot pruning method based on a handcrafted importance metric for semi- structured pruning. While one-shot pruning is highly efficient, it relies on handerafted importance metrics as proxies for true discrepancy, resulting in a significant gap with the actual pruning-inducee discrepancy. To address this issue, researchers have introduced various methodologies for learnable Estimator (SR-STE) (64) is to train N:M sparse models
RIA 62
N:M masks 664 34 244 26, 14]. Sparse-Refined Straight-Three
proposed by extending original Straight-Througg Estimator (STE) 5 from scratch.
2.3 Channel Permutation
Channel permutation 225 46, 37 has proven be an effective technique for improving the accuracy
of pruning with specific sparsity patterns (c.g., N:M sparsity) by reordering the input channels of the weight matrix. More recently, researchers have explored reordering to enhance quantization
performance [16] 59 30), highlighting channel permutation as a promising approach that merits
further investigation.
For a linear layer with Cin input channels, there are C.n! possible permutation candidates. Due the nature of N:M sparsity, channel permutation can be formulated as the following problem: distributing Ct. distinguishable balls into C,n/M /M indistinguishable boxes, where each box contains exactly M balls. In this case, the solution space is reduced to Cin!
[MI]O.-T where G = C../MM denotes the number of pruning groups. When C., 16 and M = 4, the reduced solution space still contains approximately 2.6 million candidates. The solution space grows rapidly with increasing C.n, leading to significant computational challenges for large values of Cin- Exhaustive search algorithm combined with a greedy incremental refinement strategy is applicd for channel permutation [46]. However, this approach is primarily suitable for models with small number of channels and becomes computationally expensive when applied to LLMs with large hidden dimensions. To address the computational overhead, RIA 62) adopts heuristic channel allocation method that iteratively assign important channels to different blocks efficiently. Subsequently, refinement process is applied, formulated linear assignment problem, to maximize the sum of retained weight importance scores.
Nevertheless, the handcrafted weight importance metric, used as a quality proxy in previous channel permutation methods [46, 62), fails to accurately capture the relationship between pruning error and channel permutation, resulting in suboptimal solutions. As illustrated in Figure I channel
3

---

permutation based on maximum importance score does not necessarily reduce pruning error and may even lead an increase in error.
To address the aforementioned challenges and limitations, this study pushes the boundaries of post- training semi-structured pruning for LLMSs by learnable channel permutation (LCP). This approach enables an end-to-end learning of channel reordering, eliminating the need for the handcrafted quality metrics. The proposed LCP serves an effective plugin for existing one-shot pruning methods (50) [62 by identifying appropriate channel reordering to mitigate mask quality limitations and reduce pruning errors.
3 Learnable Channel Permutation
The objective channel permutation to determine a permutation matrix E IRC.x for the weight matrix W E RCouxCCn such that the reordered weight matrix, W WP, can achieve improved accuracy after applying N:M sparsity.
However, there are two major challenges to learn the permutation matrix P: (1) P is a binary matrix containing only Os and 1s, which makes it inherently discrete and thus non-differentiable. The discrete nature of P poses a significant challenge for gradient-based learning methods. Moreover, P must satisfy the properties of permutation matrix-each row and column must contain exactly one "I" (with all other entries being "O"). This introduces strict combinatorial constraints that significantly increase the complexity of the learning process. (2) The number of possible permutation candidates increases factorially with C.n. In LLMs, Cin typically exceeds one thousand, leading to an extremely vast solution space and posing a significant challenge for the design of efficient algorithms.
3.1 Relaxation to Soft Permutation Matrix
Some existing mask learning methods assign learnable score 663 or probability 14] to each mask candidate identify the best option. Although permutation learning also be formulated combinatorial problem, the vast solution space of permutations renders these previously proposed methods impractical. Consequently, directly learning the permutation matrix tends to be more feasible.
To address the challenges associated with the discrete nature and properties of permutation matrix, common approach is to relax the hard constraints and represent the permutation using soft permuta- tion matrix. The soft permutation matrix, denoted as P, serves as a continuous and differentiable approximation of the discrete permutation matrix P, thereby enabling gradient- based learning method. doubly stochastic matrix can be used as P (2). where all entries are non-negative and each row and column sums to 1. This contrasts with P. in which each row and column contains exactly one "1". By leveraging Sinkhorn normalization [48 can be converted into doubly stochastic matrix through an iterative process of row and column normalization.
35), a nonnegative square matrix
Thus, any square matrix X can be transformed into a doubly stochastic matrix as follows:
(x = exp(X).
(2)
S'(X) (T (X))),
(3)
S(X) lim s (X),
(4)
t->ac
where non-negative square matrix is first obtained by Equation 2.. Then iterative row and column
normalization is performed by Equations (3) and (4. (x(X) = X @ (X1v1]) is the row-wise
normalization operation and (X) = 2 (1v1] is used for column normalization. Ø represents
element-wiss division and 1,v denotes column vector of one. Thus, the soft permutation matrix P can be obtained by
P S=WP(WP/T)
(5)
where WP P is a learnable matrix with the same shape as P. Since the limit in Equation cannot be computed exactly practice, truncated version with L is typically used for implementation [39
12 The temperature coefficient 7 controls the hardness of the soft permutation matrix: as approaches zero, the entries of P converge either or 1.

---

As P is not a strict permutation matrix, directly using it for channel permutation modifies both the channel order and the weight values. To avoid its impact mask selection, P is hardened into strict permutation matrix P during the forward pass. This hardening process can be formulated as linear sum assignment problem and solved by using Hungarian algorithm [27]. Specifically, this process identifies the hard permutation matrix P that is closest to the soft permutation matrix P. achieves this by solving the following optimization problem:
arg max Tr(P P), PEP
(6)
where P represents the set of all valid permutation matrices and Tr(-) denotes the trace operator. The objective is to maximize the alignment between P and P by selecting the entries of P that yield the highest overall score. Unfortunately, the hardening process is not differentiable. To address this limitation, STE [3] is employed to approximate the gradient in the backward pass, i.e., 0P/ AP/OPP- 1. By propagating gradients through this approximation, the STE preserves gradient flow across the computational graph, thereby ensuring end-to-end trainability of the permutation learning framework.
3.2 Bise Learnable Channel Permutation
According to Equation (5. if channel allowed to be permuted flexibly, the learnable param- eter matrix will be WP E @CcxxCCs which usually has the similar same shape with the weight matrix W. If each weight were to have its own learnable permutation, the learning bur- den would become prohibitively large.
Frozen Learnable
Full Matrix LCP
Block-wise LCP
Seithom
Blockvvbie
W P
P
WW5
ABCDE EO
To address this, we apply block-wise learnable channel permutation that only allows channel permutation operates within the block to reduce the training cost. It is inspired by the widely adopted block-wise operations in model com-
X
X
W
P
W
PB
HRODICFA
RAGCEFHR
pression 16 15 66 30].
Originally. the number of parameters in W W is C², for full matrix learnable channel permuta- tion. Let the block size be B. For permutation for a single block, the number of parameters in Wip P is B², With NB representing the total num- ber of blocks, the overall number of parameters
B
a
(b)
Figure 2: Illustration of learnable channel permuta- tion with different granularity: (a) full matrix LCP: (b) block-wise LCP.
given by NB x B² Can x B C.n B.
B
By block-wise learnable channel permutation.
B of the original, achieving significant parameter savings we reduce the number of parameters C.n
when & C.n.
Another advantage of block-wise learnable channel permutation is its enhanced computational efficiency when hardening the soft permutation matrix. This process is solved using the Hungarian algorithm (27), which has a time complexity of O(N²). For a full matrix permutation, the time complexity becomes O(C3 )- In contrast, by adopting block-wise manner, the time complexity for a single block is O(B 2n Given that there are Ng blocks in total, the overall complexity is O(NB Bd) O(Ct, B²). This demonstrates that it significantly reduces the computational cost of hardening process by utilizing block-wise learnable channel permutation, particularly when B & C.n.
To perform block-wise learnable channel permutation for W, each learnable matrix W is trans- formed into hard permutation matrix P, for the i-th block. Unlike the reordered weight matrix W WP obtained through full matrix permutation, the reordered weight matrix under block- wise permutation is given by Wn = WP where P = diag(P P2...., represents a block diagonal matrix and Nn is the number of blocks.
As illustrated in 2 an example of block-wise learnable channel permutation with NB = shown. In this case, the channels of W are partitioned into four blocks, with each block consisting of
5

---

consecutive C channels. P, only affects the channel permutation within the i-th block. Moreover, compared full matrix permutation, only the diagonal blocks are learnable, while all other entries are fixed to zero, which significantly reduces the training overhead.
Given the advantages of the block-wise manner, it is adopted as the default setting for the proposed learnable channel permutation in the following sections. The full matrix approach can be considered a special case when the number of blocks is set to one.
4 PermLLM: Pruning with Learnable Channel Permutation
In this section, we will introduce the proposed novel N:M semi-structured pruning framework that combines the existing one-shot pruning methods [50 62
with the proposed learnable channel permutation (LCP), which can further improve the performance of N:M sparse LLMs.
One-shot pruning eliminates weights by applying a predefined, handcrafted weight importance metric. For example, the weight importance metric proposed by Wanda (50] is defined as Sij = [Wel+ ||, where W RCoueXCiv and X is the input from calibration. Subsequently, the pruning mask M E @CCouxCin is determined to maximize the sum of the retained importance metrics, which can be formulated as
Cour CanMM
arg max M
Σ Σ I (MOS))MM(H+)M) s.t. |MM = M N, K=
(7)
0
where M IJM is constructed by setting the entries corresponding to the largest M - -N values in ( to I, while all other entries are set to O. This approach achieves N:M sparsity by ensuring that N out of every A consecutive elements are set to O, while preserving the most important weights based on their importance metrics.
With channel permutation, the order of the channels is rearranged, and consequently, the channels in the importance matrix S are permuted accordingly. The permuted importance matrix represented as S = SP a. where P3 B is the permutation matrix. As a result, the mask M varies depending on the specific permutation solution for the permuted weight W WPp:
Cant C.n/M
arg max M
EMM S) s.t. |M, [M, = M N. Σ K=
(8)
However, the non-differentiabiliyy of the argmax operation hinders gradient backpropagation,, render- ing unsuitable for gradient- based learning frameworks. To address this, STE (5) is employed to approximate the gradients during the backward pass. Specifically, while the forward pass uses the non-differentiable argmax operation to obtain a discrete hard mask M, the backward pass introduces soft mask M to enable gradient computation. The soft mask is defined as:
MI,kM.(--IMM = Softmax =Softmax ($1,MM((++))a))
where the softmax function provides a continuous and differentiable approximation. This approach allows the forward pass to retain the discrete selection behavior of argmax, while the backward pass leverages the smooth and differentiable properties of sof tmax to compute gradients effectively.
Existing channel permutation methods [46 622 are primarily designed to find the optimal permutation matrix P and the corresponding mask M that maximize the sum of retained importance score, as defined in Equation (8) However, the handcrafted quality metric used to evaluate channel permutation solutions often fails accurately reflect the true effectiveness of the permutation, potentially leading to suboptimal outcomes or even worse performance as illustrated in Figure []]
To address the aforemmntioned issue, PermLLM aims to directly minimize the output discrepancy between the dense model and the sparse N:M model by incorporating learnable channel permutations. Specifically, we utilize a cosine similarity loss to encourage alignment between the outputs of the two models, which is defined as:
Leosine (y, 8)
(10)
y

---

where y and y represent the outputs of the original dense model and the sparse N:M model.
During the proposed post-training pruning process, only Wip for each permutation matrix B is learnable, while all weight matrices remain fixed, as illustrated in Figure Additionally, each mask M is directly obtained from Equation (8), with its values dynamically updated based on changes in PB. By leveraging the proposed relaxation and gradient approximation techniques, the optimization of each Pa B is effectively guided toward solutions that maximize the preservation of the dense model's performance while adhering to the N:M sparsity constraint.
After training, weight W will be permuted and pruned by
W
M c (WP)),
(11)
where P3 denotes the learned channel permutation matrix and M is the corresponding pruning mask.
Notably, the channels of the input activations must also be permuted to align with the channel order of the weight matrix. It can be accomplished by permuting the output channels of the preceding layer.
Let Pī,B denote the permutation matrix for the current layer, and let Wi represent the permuted and pruned weight matrix of the preceding layer. The row of W 1-1 should be reordered for input activation permutation of its succeeding layer, which can be expressed as:
WLL1 Pi, 1-1-
(12)
Since it is a row-wise operation, it preserves the N:M sparsity of Wi To further reduce the runtime overhead introduced by channel permutations, we developed a customized CUDA kernel specifically for the channel permutation operation. Experimental results evaluated on LLaMA-2 7B demonstrate that this kernel achieves an average speedup of 84x compared the Pytorch implementation, thereby making pruning with channel permutations significantly more practical.
5 Experiments
5.1 Setups
We compare with three baselines in N:M sparsity, especially 2:4 sparsity: SparseGPT [5]. Wanda [50] and RIA (62]. Wanda/RIA-CPP enables channel permutation for N:M sparsity introduced in RIA.
PermLLM A indicates that Wanda or RIA is employed as the pruning metric in our
Le framework.
The proposed method is evaluated on various open source representative models: LLaMA 7B- 13B [51) LLaMA-2 7B-13B [52], LLaMA-3.1 8B 19. Qwen-2.5 7B [58] and OPT 6.7B [61]. We
randomly select 128 samples from the C4 dataset [47,, each comprising 1024 tokens, to serve as the
calibration data for all evaluated models. We utilize five zero-shot evaluation tasks: HellaSwag [60),
ARC-(Easy and Challenge) [9) OpenBookQA [41 and RTE (53) from amm-vvalutionn-harae
[8]
and one language modeling dataset: Wikitext2 40 evaluate the performance of the sparse models.
We implement PermLLM with Pytorch 43 and HuggingFace Transformers library [66. The experiments of PermLLM are conducted on A100 GPUs. We employ N:M ceni-structured pruning for linear layers, skipping the initial embedding layer and the final classification head. These linear layers constitute approximately 99% of the total parameters in LLMs.
For the proposed PermLLM framework, we utilize AdamW 33 as the optimizer, with the learning rate set from ( (1e-3, 5e-3) for all models. The iteration of Sinkhorn normalization is 5. The temperature 7 is linearly decayed from 1 to 0.1 to control the hardness of the soft permutation matrix in Equation (5). The block size for block-wise learnable channel permutation is set to 64, as it offers a balanced trade-off between performance and efficiency. Specifically, a block size of 64 is considered a more practical choice, as increasing the block size to 128 results in twofold increase in runtime. This is because larger block size not only raises computational complexity but also requires more iterations to achieve convergence due to the significantly expanded solution space. The pruning duration is about 2.5 hours for the 7B model with 4 GPUs and 5.5 hours for the 13B model with 8 GPUs, which is considered acceptable given the extremely large-scale nature of pruning-awan aware permutation problem. More efficient implementation scheme of PermLLMM discussed in Appendix A
7

---

Table 2:4 semi semi-structured pruning results on Wikitext2 with perplexity as the evaluation metric.
Method
OPT
L.aMA
LLaMA
11aMA-278
MaMA-22
Qwen--.5
4.89
7.74
SparseGPT
14.33
11.19
9.17
11.12
9.03
16.62
14.34
Wanda
16.29
11.59
9.60
12.16
9.05
23.42
24.44
Wanda+CP
15.28
11.07
8.69
11.00
8,51
21.09
18.76
14.27
8.06
8.20
13.58
RIA
15.93
11.14
8,96
11.30
8.51
22.62
22.67
RIA+CP
15.13
10.99
8.15
10.26
8.08
19.80
17.58
Peral.LMarA
7.81
7.97
15.93
Table 2: Zero-shot performance of 2:4 sparse models.
Model
Method
Weight Update
HellaSwag
ARC_E
ARC_C
OBQA
RTE
Average
Dense
50.46
65,49
30.12
26.80
55.23
45,62
Spars:GPT
43.40
60.82
26.62
24.40
52.71
41.59
OPT 6.7B
Wanda
41.56
57.62
24.83
23.00
53.43
40.09
Wanda+CP
42.87
59.51
26.02
22.00
52.71
40,62
PermL. M.Wand
X
44.27
59.43
27.22
24.00
54.15
41.81
Dense
56.95
75,38
41.89
34.80
65,34
54.87
SparseCiPT
43.55
61,78
27.90
22.80
$8.12
42.83
LLaMA 7B
X
42.33
61.57
28.07
23.60
51.26
41.37
Wanda+CP
44.21
63.51
29.86
24.00
58.12
43.94
PermLLMw
X
47.03
63.30
30.55
25.00
62.45
45.67
Dense
57.13
43.26
31.60
54.15
SparseGPT
44.11
64.14
31.31
24.20
58.84
44.52
LLaMA-2 7B
Wanda
X
41.59
61.74
30.20
24.00
$3.07
42.12
Wanda+CP
X
30.03
26.00
43.44
P
46.60
65.49
31.14
26,20
63.54
46.59
Dense
60.06
81,48
51.28
33.40
70.04
59.25
SparseGPT
44.25
63.76
30.55
24.20
53.79
43.31
LLaMA-3.1 8B
Wanda
38.45
58.00
26.37
19.40
52.35
38.91
Wanda+CP
39.32
62.25
28.92
20.40
$2.71
40.72
PermLL MimnLLMwend MWende
X
45.33
30.97
24.00
53.79
43.33
Dense
58.79
79.56
46.08
33.00
76.90
58.87
SparseGPT
7
46.20
71.13
37.46
26.00
75.45
51.25
Qwen-2.5
Wanda
X
67.17
33.45
25.40
72.92
47.91
Wanda+CP
42.92
70.50
36.09
25,20
72.20
49.38
PormLLMMw..
47.30
70.58
38.13
27.60
77.26
52.17
5.2 N:M Semi-structured Pruning for LLMs
Language Modeling, In Table T we evaluate the language modeling performance of the 2:4 sparse models on Wikitext2. Perplexity used as the evaluation metric, with lower values indicating better language modeling performance. SparseGPT updates the remaining unpruned weights during pruning to compensate the pruning error. Other pruning methods, including our proposed PermLL.M, do not modify weight values.
Empirical results demonstrate that channel permutations effectively mitigate performance degradation in pruned models. However, existing channel permutation algorithms rely on handcrafted heuristic metrics generate permutations, often yielding suboptimal solutions. In contrast, PermLLM employs end-to-end learnable optimization to derive superior permutations by directly minimizing the performance gap between the dense and pruned models. Compared SparseGPT, both Wanda and RIA initially demonstrate superior performance on LLaMA and LLaMA-2. The proposed PermLLM framework further unlocks their potential. On the other hand, for other models, Wanda and RIA underperform relative to SparseGPT, even with channel permutations. Specifically, significant performance degradations are observed L.LMA-3.1 and Qwen-2.5 even using Wanda+CP and RIA+CP However, with the incorporation of learnable channel permutations, PermLLM surpasses

---

Table 3: Runtime for the different layers and channel permutations in LLaMA-2 7B using 2048 tokens.
Method
Q/K/V/O_pray
Up/Gate proj
Down_pne proj
CP
Dense 2:4 sparsity + CP
1.513ms
2.607ms
2.614ms 1.535ms
0.927ms
1.526ms
0.039ms
Speedup
1.632x
1.708x
1.703x
Table 4: Evaluation on LimlLLMMaaaie for LLaMA-2 7B with different iteration number of Sinkhorn normalization.
Model
# of Iter.
HellaSwag
ARC_E
ARC_C
OBQA
RTE
Average
Wikitex12
45.28
64.65
29.86
21.20
53.79
42.96
14.12
Qwen-2.5
45.33
62.58
30.97
24.00
53.79
43.33
14.03
45.93
71.00
37.88
25.40
65.70
49.18
14.43
LLaMA-3.1 8B
47.30
70.58
38.13
27.60
77.26
52.17
13.58
SparseGPT due to its accurate and model-wise optimization, demonstrating the effectiveness and superiority of the proposed framework.
Zero-shot Performance. In Table we report the zero-shot performance the 2:4 sparse models on five evaluation tasks. The average accuracy across all tasks is presented in the last column. PermLLM significantly enhances the effectiveness of channel permutations for pruning, outperforming existing methods on the majority of tasks and achieving the highest average accuracy. This highlights the significant potential of channel permutation as an effective tool for semi-structured pruning. We also evaluate PermLLM for 4:8 sparsity on LLaMA-2 7B in Table g which shows PermLLM is not limited to 2:4 sparsity.
Inference Speedup. Inference runtime evaluation is crucial to validate the practicability of the proposed framework. In Table report the runtime speedup of 2:4 sparse LLaMA-2 model using batch of 2048 tokens following SparseGPT and RIA. The customized CUDA kernel of channel permutation reduces the total runtime from 3.288ms to 0.039ms, providing 84 speedup compared to Pytorch implementation. Thus, the overhead of channel permutations is minimal with the customized CUDA kernel. The overall acceleration across all linear layers, even with channel permutations, approximately 1.67x. x.
5.3 Ablation Study
We conduct an ablation study on the relaxation of the soft permutation matrix to evaluate its impact on framework. A larger iteration number in Sinkhorn normalization allows the soft permutation matrix to converge more closely to a doubly stochastic matrix (DSM). By default, we set the iteration number of Sinkhorn normalization to 5, which generally yields satisfactory performance. In Table 4 we evaluate the pruning performance of PermLLM under different Sinkhorn normalization iterations. When the iteration number is set to O, the soft permutation matrix deviates the most from a DSM. The results demonstrate the benefits of using DSM as the soft permutation matrix for permutation learning. It helps enhance the learning process by providing meaningful representation.
structured and
In Table 5 we evaluate PermLLM datasets: Pile [7], Wikitext2 [40], and C4 [47). Each dataset consists of 128 randomly selected samples. The results demonstrate that the learned permutation performs consistently well across different datasets, which indicates robustness of PermLLM.
the LLaMA-2 7B model using different calibration
Additionally, we conduct experiments to analyze the trade-off between performance and training cost under varying block sizes. As shown in Table 6 larger block sizes provide a greater optimization space. However, this increased space comes at the cost of longer exploration and convergence times. We select a block size 64 as the default, as strikes a good balance between pruning performance and training efficiency.

---

Table 5: Evaluation PermLLM for LlaMA-2 7B with different calibration dataset.
Dataset
HellaSwag
ARC_E
ARC_C
OBQA
RTE
Average
Wikitex12
Pile
45.83
64.31
32.08
26.60
54.87
44.74
8.96
Wikitext2
45.42
66.41
32.34
25.80
53.07
44.61
8.31
C4
46.60
65.49
31.14
26.20
63.54
46.59
9.39
Table 6: Evaluation on PermLLM wanda for LlaMA-2 7B with different block size.
Block size
HellaSwag
ARC_E
ARC_C
OBQA
RTE
Average
Wikitex12
Time
32
46. 13
64.39
29.69
24.60
53.07
43.58
9.50
2h
64
46.60
65.49
31.14
26.20
63.54
46.59
9.39
2.5h
128
46.47
66.08
32.08
27.40
64.43
47.09
9.07
6h
Conclusion
This paper introduces PermLLM, novel pruning framework leveraging learnable channel permuta- tions (LCP) to optimize N:M sparsity in large language models. By minimizing pruning errors through end-to-end optimization, PermLLM significantly enhances the performance N:M semi-structured pruning. Experimental results validate its superiority over existing methods.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint, 2023.
[2] Ryan Prescott Adams and Richard S Zemel. Ranking via sinkhorn propagation. arXiv preprint, 2011.
[3] Saleh Ashkboos. Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In International Conference on Learning Representations (ICLR), 2024.
[4] Guangji Bai, Yijiang Li, Chen Ling, Kibaek Kim, and Liang Zhao. Sparsellm: Towards global pruning of pre-trained language models. In Annual Conference Neural Information Processing Systems (NeurIPS), 2024.
[5] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint. 2013.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Annual Conference Neural Information Processing Systems (NeurIPS),
33:1877-1901, 2020.
[7] Xiaodong Chen, Yuxuan Hu, and Jing Zhang. Compressing large language models by stream- lining the unimportant layer. arXiv preprint, 2024.
[8] Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattem Analysis and Machine Intelligence (TPAMI), 2024.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint, 2018
[10] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 () 8-bit matrix multiplication for transformers at scale. In Annual Conference on Neural Information Processing Systems (NeurIPS), pages 303 0033333, 2022.
10

---

[11] Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In International Conference on Machine Learning (ICML), pages 11346-11374, 2024.
[12] Patrick Emami and Sanjay Ranka. Learning permutations with sinkhorn policy gradient. arXiv preprini Xrvi 1805.07010, 2018.
[13] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICML), pages 2943-2952. PMLR, 2020.
[14] Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Learnable semi-structured sparsity for large language models. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[15] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning (ICML), pages 10323-10337, 2023.
[16] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint, 2022.
[17] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint, 2020.
[18] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou, A framework for few-shot language model evaluation, 12 2023.
[19] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint, 2024.
[20] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint, 2015.
[21] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Annual Conference on Neural Information Processing Systems (NIPS), 28, 2015.
[22] Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks (ICNV), pages 293-299, IEEE, 1993.
[23] Yihui Hc, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In IEEE International Conference Computer Vision (ICCV), pages 1389-1397, 2017.
[24] Itay Hubara, Brian Chmiel, Moshe Island. Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: provable and efficient method to find n: m transposable masks. Annual Conference on Neural Information Processing Systems (NeurIPS), 34:21099- 21111, 2021.
[25] Yu Ji, Ling Liang, Lei Deng, Youyang Zhang, Youhui Zhang, and Yuan Xie. Tetris: Tile- matching the tremendous irregular sparsity. Annual Conference Neural Information Process- ing Systems (NeurIPS), 31, 2018.
[26] Cheng-Chun Kao, Amir Yazdanbakhsh, Suvinay Subramanian, Shivani Agrawal, Utku Evci, and Tushar Krishna. Training recipe for n: M structured sparsity with decaying pruning mask. arXiv preprint, 2022.
11

---

[27] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[28] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Annual Conference on Neural Information Processing Systems (NIPS), 2, 1989.
[29]
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint, 2018.
[30] Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. In Annual Conference on Neural Information P Processing Systems (NeurIPS), 2024.
[31] Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awe aware weight quantization for on-device Ilm compression and acceleration. Machine Learning and Systems (MLSys), 6:87-100, 2024.
[32] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Annual Conference on Neural Information Processing Systems (NeurIPS), 34:9908 9922,,99922222200 2021.
[33] I Loshchilov. Decoupled weight decay regularization. arXiv preprint, 2017.
[34] Yucheng Lu, Shivani Agrawal, Suvinay Subramanian, Oleg Rybakov, Christopher De Sa, and Amir Yazdanbakhsh. Step: learning n: M structured sparsity masks from scratch with precondition. In International Conference on Machine Learning (ICML), pages 22812-22824.
PMLR, 2023.
[35] Jiancheng Lyu, Shuai Zhang, Yingyong Qi, and Jack Xin. Autoshufflene: Learning permutation matrices via an exact lipschitz continuous penalty in deep convolutional neural networks. In ACM International Conference on Knowledge Discovery and Data Mining (KDD), pages 608-616, 2020.
[36] Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of large language models. Anmual Conference on Neural Information Processing Systems (NeurIPS), 36:21702-21720, 2023.
[37] Mohit Mahajan, Wen-Mei Hwu, and Rakesh Nagi. Determining optimal channel partition for 2:
4 fine grained structured sparsity. Optimization Letters, (89):2277-2000, 2024.
[38] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint, 2024.
[39] Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning latent permuta- tions with gumbel--inkhorn networks. International Conference on Learning Representations (ICLR), 2018
[40] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint, 2016.
[41] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint, 2018.
[42] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint, 2021.
[43] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Annual Conference on Neural Information Processing Systems (NeurIPS), 32, 2019.
12

---

[44] Zehua Pei, Hui-Ling Zhen, Xianzhi Yu, Sinno Jialin Pan, Mingxuan Yuan, and Bei Yu. Fusegpt: Learnable layers fusion of generative pre-trained transformers. arXiv preprint, 2024.
[45] Jeff Pool, Abhishek Sawarkar, and Jay Rodge. Accelerating inference with sparsity us- ing the nvidia ampere architecture and nvidia tensorrt. NVIDIA Developer Technical Blog, htpss//developer nvidia. and-tensorri, 2021.
[46] Jeff Pool and Chong Yu. Channel permutations for sparsity. Annual Conference Neural Information Processing Systems (NeurIPS), 34:13316-13327 2021.
[47] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(140):1-67, 2020.
[48] Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The annals of mathematical statistics, 35(2):776887, 1964.
[49] Jiwon Song. Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, al. Sleb: Streamlining Iims through redundancy verification and elimination of transformer blocks. In International Conference on Machine Learning (ICML), 2024.
[50] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In International Conference on Learning Represenlations (ICLR), 2024.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint, 2023.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arviv preprint, 2023.
[53] Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language under- standing. arXiv preprint, 2018
[54] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint, 2020.
[55] Zixiao Wang, Jingwei Zhang, Wenqian Zhao, Farzan Farnia, and Bei Yu. Moreaupruner: Robust pruning of large language models against weight perturbations. arXiv preprint, 2024.
[56] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics.
[57] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao WU, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning (ICML) pages 38087-38099. PMLR, 2023.
[58] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint, 2024.
[59] Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint, 2023.
[60] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint, 2019,
13

---

[61] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint, 2022.
[62] Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-playy An efficient post-training pruning method for large language models. In International Conference Learning Representations (ICLR), 2024.
[63] Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Rongrong Ji. Learning best combination for efficient n: M sparsity, Annual Conference on Neural Information Processing Systems (NeurIPS), 35:941-953, 2022.
[64] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: M fine -raiggr structured sparse neural networks from scratch. In International Conference on Learning Representations (ICLR), 2021.
[65] Lancheng Zou, Shuo Yin, Mingjun Li, Mingzi Wang, Chen Bai, Wenqian Zhao, and Bei Yu. Oiso: Outlier-isolated data format for low-bit large language model quantization. IEEE Trunsactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), 2025.
[66] Lancheng Zou, Wenqian Zhao, Shuo Yin, Chen Bai, Qi Sun, and Bei Yu. BiE: Bi-cxponent block floating-point for large language models quantization. In International Conference Machine Learning (ICML), volume 235, pages 22978-62999 PMLR, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's primary contribution, which is the proposal of learnable channel permutation method to improve N:M sparsity for the first time, is clearly and accurately articulated in both the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of this work are discussed in Appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
This paper dost not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The experimental setups and hyperparameters configurations are detailed in Section [.. and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release the code in the camera-ready version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iips.cc/ppll wttp:ccppppcccceeeeeeeeee..c.ett httpa//gggeeee/eeeeubbiisssssss /Cc////oddiiisssseeeeeee ssiiissiiciiiyyyycc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The experimental setups and hyperparameters configurations can be found in Section 3. and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bar is not used for experimental evaluation in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the sufficient information on the number of GPUs used and runtime in Scction[5.]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: peatpp::/neurrpeee httpp/////./ccctticiiiiiieeeeeee
Answer: [Yes]
Justification: The research in this paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts of this paper in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: For the existing assets used, we have cited the papers and repos.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is used only for editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: hattap///aaarpppeecc///offeerere for what should or should not be described.
21

---

A Implementations
Hyperparameters Configurations. The learning rate is set from (le-3, Se-3). Specifically, we use le-3 for PermLLMwondd and 5e-3 for PermLLM A- The iteration of Sinkhorn normalization is 5. The temperature is linearly decayed from to 0. to control the hardness of the soft permutation matrix in Equation (5). The block size for block-wise learnable channel permutation is 64, offers a balanced trade-off between performance and efficiency. We use 50 iterations for permutation learning.
More Efficient Implementation. PermLLM serves as an effective plugin to improve performance of existing zero-shot pruning methods. As observed in other studies, different layers have varying impacts on the output. To further enhance the efficiency of PermLLM, learnable channel permutation modules can be inserted into only a subset of layers, while the traditional channel permutation method is applied to the remaining layers. For instance, we apply learnable channel permutations only to the last six decoder layers of the LLLAMA-2-7B model. In this case, only single GPU is required for permutation learning, reducing the runtime to 0.4 hours, which is similar to the runtime of traditional channel permutation method.
The experimental results are shown in Table 7 Although partial PermLLM does not match the perfor- mance of full PermLLM due to its limited optimization space, it still provides notable improvements over traditional channel permutation methods. This approach also represents a balanced trade-off between performance and efficiency, making it particularly suitable for scenarios with relatively limited computational resources.
Table 7: Experimental results on LLaMA-2-7B with partial PermL.MM. We highlight the top-2 results.
Method
HellaSwag
ARC E
ARC C
OBQA
RTE
Average
Wikitex12
RIA+CP
42.86
64.69
30.29
24.40
54.87
43.42
10.26
PenaLLMRrA (partial)
44.46
64.10
31.74
24.80
53.79
43.78
10.10
PermLLMAIA (full)
45.15
64.77
32.25
24.80
$4.51
44.30
9.60
B PermLLM for 4:8 Sparsity
In Table we present the detailed results about the 4:8 sparsity on I.LAMA22 model with different pruning methods. The experimental results demonstrate that PermLLM is not limited to 2:4 sparsity and can still outperform traditional method for 4:8 sparsity.
Table 8: Evaluation on 4:8 sparse LLaMA-2-77 with different pruning methods.
Method
Weight Update
HellaSwag
ARC_E E
ARC_C
OBQA
RTE
Average
Wikitext2
Dense
57.13
76.30
43.26
31.60
62.45
54.15
5.47
SparseGPT
48.77
67.68
34.81
26.20
$3.79
46.25
8.56
Wanda
46.87
66.92
34.04
26.40
54.87
45.82
8.63
Wanda+CP
X
48.61
70.62
35.15
28.60
55.23
47.64
8.26
PermLLMay
49.02
70.20
36.35
29.40
$4.87
47.97
7.96
C Visualization of Mask
the pruned LLaMA-2-7B by differrnt methods. For methods involving channel permutations (e.g., RIA+CP and PermLLM A), the channels are permuted back to their original order for better comparison. We extract 128x 128 portion of the mask (i.e., mas8[[128., :188) for better visualization. It's observed that the retained weights differ between the previous channel permutation method and our proposed learnable channel permutation. This is because we utilize different strategies: previous one aims at maximizing the sum of the retained importance metrics and PermI_LM is to minimize the output discrepancy between dense model and the pruned model.
the masks of layer.30.down
22

---

(a) Wanda
(b) Wanda+CP
(c) PermLL.Meande
(d) RIA
(e) RIA+CP
(1) PermLLMRrA
Figure 3: Visualization of mask obtained by different pruning methods. The blue part means the pruned weights and the white part is the retained weights
D Limitations
This paper introduces innovative learnable channel permutation method enhance semi-structured pruning for the first time. Although the method tailored for semi-structured pruning, channel permutation channel reordering has also been shown be beneficial in other areas, such as quantization [30 59). This suggests that the broader applicability the proposed approach to tasks beyond pruming, such optimizing quantization performance, remains open area for future exploration. Moreover, while the proposed block-wise channel permutation scheme significantly reduces training overhead compared to the full matrix scheme, the training of PermLLMM still requires more computational resources compared to traditional channel permutation methods. Enhancing the training efficiency for pruning-aware permutation learning remains important direction for future research.
E Broader Impacts
The proposed learnable channel permutation for N:M sparsity not expected to have any negative societal impacts. Instead, it has the potential to advance the field of machine learning, particularly in the area of model compression.
23


Reviews:
[{"review_id": "kGV3yMkyw7", "reviewer": "Reviewer_uQ84", "summary": "Channel permutations have been shown to work well at reducing the lost weight magnitude or other particular metric, but these metrics do not always maximize model quality.  Further, existing work uses approximate heuristics or computationally expensive search procedures to find good permutations for N:M sparsity.  PermLLM addresses these two shortcomings by directly learning the permutations.  First, a standard permutation matrices are relaxed into \"soft\" permutation matrices in order to make them differentiable, using Sinkhorn normalization to convert the learned matrix into a doubly stochastic matrix, which can be hardened into a true permutation for forward propagation.  To reduce the complexity and number of learnable parameters, the permutation matrices are simplified into block-wise permutation matrices.  A soft mask is used in the backwards pass to allow differentiability when learning the permutations (since they will change the mask applied), and cosine similarity between the dense and sparse model outputs is used as the loss function.  Reasonably broad evaluations are performed, showing meaningful average improvements over the considered baselines, and top-quality results on most model/task pairs.", "questions": "1. How does PermLLM compare to previous work (say, [42]) operating with a better importance metric than magnitude (say, Wanda's importance metric)?  \n2. How does PermLLM compare to learned rotations (RotPruner)?  Is there room for both techniques?\n3. How does PermLLM compare to past approaches to directly learning permutations (2, above)\n4. How does PermLLM perform with varying block sizes (32, 64, 128 (if possible!))?\n\nAddressing these questions will directly affect the quality, originality, and (most importantly) the significance of the submission.  Results on a single model should be enough to show the important trends.", "limitations": "Limitations are discussed in the appendix, but should really be in the main content.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\nThe idea is well-motivated; figure 1 clearly shows how existing metrics are not ideal to maximize model quality.  The approach is also sound; each piece of the solution is clearly motivated and explained, and the effect of the whole process is proven successful by the evaluations.  I also appreciate the runtime analysis, as well as the ablation study (though it is somewhat limited).  I have little doubt that a dedicated practitioner could reproduce the results from studying the descriptions and equations, and that the reported model quality improvements would likely generalize beyond the presented results.\n\t\n## Weaknesses\nI want to believe in this submission's superiority over past work, but there are some missed references and baselines that affect the significance, originality, and quality of the submission.\n\nSome missing references should be included:\n- Mahajan et al. [1] formulate the permutation search as a discrete optimization problem.\n- Lyu et al. [2] learn permutation matrices directly.  This one is particularly relevant, and the authors should provide a comparison of this approach and the proposed approach, and, ideally, empirical comparisons.\n\nA simple baseline would be the approach from [42] applied to e.g. SparseGPT scores or Wanda importance metrics (as noted in that work, the technique is applicable to any metric, not just magnitude).  The authors note on line 45 that the large search space renders this approach infeasible for large sizes, but [42] includes less-expensive settings than the default.  Using the simplest from [42]'s Table 3 and the available code on a single A100, searching for a good permutation for a random 4096x4096 matrix (such as Llama2 7B's attention output projection) took only 125 seconds.  I continued timing random matrices for the other layers and came up with these results:\n\n| Layer | Size | Duration (s)|\n| ------ | ------ | -------|\n| QKV | 4096x12288 | 297 |\n| PROJ | 4096x4096 | 125 |\n| up| 4096x11008 | 268 |\n| gate | 4096x11008 | 268 |\n| down | 11008x4096 | 1318 |\n\nReplicated for 32 blocks, the total projected time to search for permutations for all of Llama2 7B is 20.25 hours.  If parallelized over 4 GPUs to match the submission's resources for this model, this is \"only\" double the 2.5 hours reported on line 298.  It may indeed be double the cost, but certainly not \"prohibitively high computation complexity\" as claimed on line 46.  If, as suggested above, these permutations are found using a higher-quality importance metric than weight magnitude, they could prove competitive with PermLLM.  Or, they may not.  I believe this is a useful baseline to correctly determine the significance of PermLLM as a technique as well as improve the quality of the submission.\n\nI'm also concerned about the interaction with another, non-orthogonal technique that seeks to reduce the quality impact of N:M sparsity: rotations.  They've been successfully used for quantization [3,4] as well as N:M sparsity[5].  It's not clear that PermLLM composes with or outperforms this technique.\n\nFinally, some small issues reduce the clarity and quality of the submission:\n- On line 269, the authors suggest that the additional permutation to handle residual paths has a negligible impact on deployment efficiency, citing [58].  In the next sentence, though, the authors describe a custom CUDA kernel to reduce the overhead.  Why do we care about reducing a negligible overhead?  (The experimental results on line 271 probably belong in Section 5, too.)\n- Sweeping the number of iterations used in Sinkhorn normalization in Section 5.3 would be informative.  (5 is better than 0, but is 5 enough?  More than enough?)\n- An ablation of the block size used (32, 64, 128) would be useful to see how sensitive the outcome is to this hyperparameter.\n- What software is used to gather the runtimes in Table 3?\n\nReferences\n1. Mahajan, M., Hwu, WM. & Nagi, R. Determining optimal channel partition for 2:4 fine grained structured sparsity. Optim Lett 18, 2079–2090 (2024). https://doi.org/10.1007/s11590-023-02084-8\n2. Jiancheng Lyu, Shuai Zhang, Yingyong Qi, and Jack Xin. AutoShuffleNet: Learning Permutation Matrices via an Exact Lipschitz Continuous Penalty in Deep Convolutional Neural Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20). https://doi.org/10.1145/3394486.3403103\n3. Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, James Hensman, QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs, NeurIPS 2024\n4. Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, Tijmen Blankevoort, SpinQuant: LLM Quantization with Learned Rotations, ICLR 2025\n5. Anonymous, ROTPRUNER: LARGE LANGUAGE MODEL PRUNING IN ROTATED SPACE  https://openreview.net/pdf?id=wV9iMiyQcc", "was_revised": true, "final_justification": "PermLLM builds on a solid foundation of past work, but is a novel advancement that meaningfully improves N:M model quality with relatively low overhead.  The authors satisfactorily addressed all of my concerns.", "rebuttal": "Thanks for your detailed review and valuable feedback.\n\n**W1**: Thanks for your reminding. We will include the missing references and discuss them in the final version. \n\n(a) Mahajan et al. [1] and [42] propose different methods to achieve a good permutation result for N:M sparsity. However, their major limitations lie in the biased optimization objective (focusing on maximum retained weight importance rather than the output loss between the original model and the pruned model) and the local optimality of the permutation matrix (performing layer-wise optimization, which fails to capture inter-layer dependencies). These issues, as highlighted in the motivation section, limit their ability to further improve performance. To further validate this, we implement the method from [42] combined with Wanda for a direct comparison against the proposed PermLLM. Please refer to **Q1** for detailed results. \"Prohibitively high computational complexity\" in line 46 refers to the solution space complexity of concurrently finding permutations for all layers. For example, if we need to determine two permutations for two linear layers in the model, and there are n possible permutations for layer 1 and m for layer 2, the complexity is n+m when performing layer-wise optimization. However, for model-wise optimization, the complexity increases to n*m. We will rewrite it to avoid misunderstanding.\n\n(b) Lyu et al. [2] employ an $\\ell_{1-2}$ penalty to constrain the learnable matrix, ensuring that, upon convergence, each row and column contains exactly one entry equal to 1, with all other entries being zero. They also utilize Sinkhorn normalization to facilitate convergence. The key difference between their approach and ours lies in the method used to achieve convergence to 1 or 0, which we think it is minor: while they rely on the $\\ell_{1-2}$ penalty, we adopt an annealing-based method. We believe that two approaches can achieve similar results with appropriate training configurations. Furthermore, the more valuable contributions of PermLLM are its abilities to achieve pruning-aware permutation learning for the first time and efficiently train the permutation matrix, which provides a new pruning framework for further exploration.\n\n**W2**: RotPruner [5] prunes the model in rotated space. As shown in Fig.2 of their paper, the rotation operation results in more values being 0 or small magnitude. We think it may be more benefical for unstructured pruning as N:M sparisty has more strict constraints. \n\nThere are some shortcomings compared with PermLLM: (a) memory overhead: RotPrune requires storing the rotation matrices in memory. For a weight matrix $\\mathbf{W} \\in \\mathbb{R}^{n \\times n}$, the corresponding rotation matrix $\\mathbf{Q} \\in \\mathbb{R}^{n \\times n}$ has the same size, leading to significant memory usage. In contrast, PermLLM only needs to store a vector $\\mathbf{v} \\in \\mathbb{R}^{n}$ to represent the column permutation order, resulting in much lower memory overhead. (b) runtime overhead: RotPrune requires online computation of rotations, which is less efficient compared to the lightweight permutation operation. Rotation can be seen as an additional dense linear layer, introducing extra computational costs (e.g., Q/K/V_proj vs. CP) in Table 3. \n\nPermLLM can also outperform RotPruner, the results can be found in **Q2**.\n\n**W3**: We will revise the paragraph in the final version for improved clarity. The original PyTorch implementation is inefficient, so we develop a custom CUDA kernel to achieve significant speedup. With this optimized kernel, the additional permutation incurs minimal overhead, resulting in a negligible overall impact. Thank you for bringing this to our attention!\n\n**W4**: We conduct additional experiments on LLaMA-2 7B by varying the number of iterations used in Sinkhorn normalization. With the exception of Wikitext2, iter=5 outperforms both iter=0 and iter=10 on the remaining benchmarks. The reason why iter=5 performs better is that introducing some perturbations during the early exploration helps in finding a better solution and avoiding being trapped in a local optimum.\n\n| Iter. | HellaSwag | ARC_E     | ARC_C     | OBQA      | RTE       | Average   | Wikitext2 |\n| ----- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| 0     | 46.23     | 64.48     | 30.80     | 25.00     | 54.51     | 44.20     | 9.15      |\n| 5     | **46.60** | **65.49** | **31.14** | **26.20** | **63.54** | **46.59** | 9.39      |\n| 10    | 46.37     | 63.17     | 28.50     | 25.80     | 55.23     | 43.81     | **9.07**  |\n\n**W5**: Block size is a critical hyperparameter that balances pruning performance and training efficiency. To explore this trade-off, we conduct additional experiments by varying the block size, which can be found in **Q4**.\n\n**W6**: We used SparseSemiStructuredTensor and to_sparse_semi_structured in Pytorch and enabled CUTLASS for sparse operations. Moreover, we utilized Timer utility from torch.utils.benchmark to measure runtimes.\n\n**Q1**: We apply [42] with Wanda to prune LLaMA-2 7B. The results below demonstrate the superiority of PermLLM. There is no doubt that our motivation is well-founded and our methodology highly effective. The reasons behind PermLLM's superior performance have been discussed in **W1**.\n\n|          Method          | HellaSwag |   ARC_E   |   ARC_C   |   OBQA    |    RTE    |  Average  | Wikitext2 |\n| :----------------------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|        [42]+Wanda        |   43.41   |   62.92   |   28.67   | **26.20** |   54.51   |   43.14   |   10.19   |\n| $\\text{PermLLM}_{Wanda}$ | **46.60** | **65.49** | **31.14** | **26.20** | **63.54** | **46.59** | **9.39**  |\n\n**Q2**: In RotPruner, 128 samples from Wikitext2 are used as calibration data. Thus, we also use 128 samples in Wikitext2 as calibration data for PermLLM. The results are shown below:\n\n|             Method             | HellaSwag | ARC_E | ARC_C | OBQA  |  RTE  | Average | Wikitext2 |\n| :----------------------------: | :-------: | :---: | :---: | :---: | :---: | :-----: | :-------: |\n|        RotPruner (2:4)         |     -     |   -   |   -   |   -   |   -   |    -    |   9.20    |\n| $\\text{PermLLM}_{Wanda}$ (2:4) |   45.42   | 66.41 | 32.34 | 25.80 | 53.07 |  44.61  | **8.31**  |\n\nWe didn't find the results for the zero-shot tasks under 2:4 sparsity in their paper. However, for Wikitext-2, PermLLM significantly outperforms RotPruner. Moreover, we believe there is room for both techniques, making it an interesting avenue for further exploration and improvement.\n\nFor example, $\\mathbf{x}$ = [[-1.0412],[-1.4108],[ 1.0199],[-1.7633],[-0.2013],[ 0.8784],[ 0.2520],[ 1.3273]], \n\n$\\mathbf{W}$=[[ 0.5143,  0.5205,  0.6937,  0.0350, -0.4507,  0.3604,  2.4873,  0.9460],[ 0.0755, -0.9656,  1.2620, -0.7471, -2.5672,  1.6677, -1.1052,  0.6418]], \n\nrotation matrix $\\mathbf{Q}$=[[-0.1583, -0.5283,  0.4140, -0.0586,  0.2760, -0.0372,  0.1789, -0.6414],\n        [ 0.6179, -0.2853, -0.1028, -0.5905, -0.0206,  0.2548, -0.3318, -0.0461],\n        [-0.2811, -0.3364, -0.3383, -0.2190,  0.4878,  0.2649,  0.3716,  0.4463],\n        [ 0.1433,  0.3674, -0.1850, -0.4794, -0.0594, -0.3628,  0.6207, -0.2449],\n        [ 0.0564, -0.2713, -0.7957,  0.3407, -0.0890, -0.1264, -0.0599, -0.3829],\n        [-0.2781, -0.3765,  0.0066, -0.2865, -0.2223, -0.7095, -0.2565,  0.2831],\n        [-0.3605, -0.1679, -0.0178, -0.1824, -0.7537,  0.4435,  0.1991, -0.0619],\n        [ 0.5321, -0.3844,  0.1894,  0.3723, -0.2377, -0.1262,  0.4749,  0.3110]],\n\n permutation order=[3, 1, 5, 4, 6, 0, 2, 7]. With magnitude pruning, direct 2:4 pruning loss=1.660, rotated 2:4 pruning loss=0.769, rotated and permutation 2:4 pruning loss=0.497. This example highlights the potential of combining rotation and permutation for more effective N:M pruning.\n\n**Q3**: Based on our analysis in **W1**, the differences between [2] and permutation learning in PermLLM are minimal. Consequently, with proper usage, the final performance is expected to be similar. We believe this aspect is not particularly critical for the evaluation of PermLLM, as, similar to mask learning in prior pruning works, there are various methods to learn the pruning mask, such as SR-STE [i], decaying-based methods [ii], and others. The more valuable contributions lie in implementing permutation learning to successfully enhance pruning performance and make this approach more practical—achievements that PermLLM fully realizes. At the same time, we hope that our motivation and approach can inspire more researchers to further advance this field. For example, it could be applied to quantization, combined with rotation as you mentioned, or lead to the development of better pruning-aware permutation learning framework.\n\n**Q4**: We conduct an ablation study about block size to see its impact on final pruning performance and training cost.\n\n| Block size | HellaSwag | ARC_E     | ARC_C     | OBQA      | RTE       | Average   | Wikitext2 |\n| ---------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| 32         | 46.13     | 64.39     | 29.69     | 24.60     | 53.07     | 43.58     | 9.50      |\n| 64         | **46.60** | 65.49     | 31.14     | 26.20     | 63.54     | 46.59     | 9.39      |\n| 128        | 46.47     | **66.08** | **32.08** | **27.40** | **64.43** | **47.09** | **9.07**  |\n\nThe training time is ~2h for block size=32, ~2.5h for block size=64 and ~6h for block size=128. A larger block size offers a greater optimization space. However, this increased space leads to longer exploration and convergence times.\n\n\n\nThank you again for your constructive suggestions!\n\n\n\nReferences\n\n[i] Zhou et al., LEARNING N:M FINE-GRAINED STRUCTURED SPARSE NEURAL NETWORKS FROM SCRATCH, ICLR 2021.\n\n[ii] Kao et al., Training Recipe for N:M Structured Sparsity with Decaying Pruning Mask, ICML2021 workshop."}, {"review_id": "sWFSn6T3vG", "reviewer": "Reviewer_gs8N", "summary": "PermLLM addresses the limitations of heuristic-driven channel permutation in N:M sparse LLMs by proposing a novel learnable channel permutation (LCP) framework that optimizes permutations end-to-end to minimize pruning-induced output errors. As a plug-in enhancement to existing one-shot methods (e.g., Wanda), PermLLM achieves outstanding results across LLaMA, OPT, and Qwen families—reducing perplexity degradation and improving zero-shot accuracy.", "questions": "How robust is the learned permutation to the choice and size of the calibration dataset? Does performance vary significantly with different calibration data?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\nThe paper compellingly argues against handcrafted metrics (Fig. 1) and introduces a novel framework for learnable channel permutation (LCP) that directly minimizes pruning error, which is an original contribution to semi-structured pruning.\n\nThe block-wise permutation strategy is a clever solution to make LCP scalable for LLMs, reducing parameter count and conputational complexity.\n\nThe method is thoroughly evaluated on a wide range of modern LLMs, mostly outperforming baselines in both perplexity and zero-shot tasks.\n\n\n### Weaknesses\n\nThe choice of block size (64) is justified anecdotally rather than empirically.\n\n An ablation study showing the trade-off between block size, performance, and training cost is missing.\n\nThe experimental results are not optimal in every setting.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time and effort to review our paper.\n\n**W1**: We conduct additional experiments about $\\text{PermLLM}_{Wanda}$ on LLaMA-2 7B to show the trade-off between performance and training cost with varying block size.\n\n| Block size | HellaSwag | ARC_E     | ARC_C     | OBQA      | RTE       | Average   | Wikitext2 |\n| ---------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| 32         | 46.13     | 64.39     | 29.69     | 24.60     | 53.07     | 43.58     | 9.50      |\n| 64         | **46.60** | 65.49     | 31.14     | 26.20     | 63.54     | 46.59     | 9.39      |\n| 128        | 46.47     | **66.08** | **32.08** | **27.40** | **64.43** | **47.09** | **9.07**  |\n\nThe training time is ~2h for block size=32, ~2.5h for block size=64 and ~6h for block size=128. A larger block size offers a greater optimization space. However, this increased space leads to longer exploration and convergence times. We choose block size=64 as the default configuration because it can achieve good balance between pruning performance and training efficiency.\n\n**W2**: As the benchmarks we used in the evaluation are diverse, so it is extremely challenging  to achieve optimal results in every benchamark. Thus, we believe the evaluation results are good enough to demonstrate the superiority of the proposed PermLLM.\n\n**Q1**: We perform extra experiments about $\\text{PermLLM}_{Wanda}$ on LLaMA-2 7B with different calibration data: (a) 128 samples from Pile[1], (b) 128 samples from Wikitext2 [2]and (c) 128 samples from C4 [3]. The learned permutation performs consistently well across different datasets, it indicates robustness of PermLLM.\n\n| Dataset   | HellaSwag | ARC_E | ARC_C | OBQA  | RTE   | Average | Wikitext2 |\n| --------- | --------- | ----- | ----- | ----- | ----- | ------- | --------- |\n| Pile      | 45.83     | 64.31 | 32.08 | 26.60 | 54.87 | 44.74   | 8.96      |\n| Wikitext2 | 45.42     | 66.41 | 32.34 | 25.80 | 53.07 | 44.61   | 8.31      |\n| C4        | 46.60     | 65.49 | 31.14 | 26.20 | 63.54 | 46.59   | 9.39      |\n\nWe also use different size of calibration dataset (C4) to check the performance of PermLLM.\n\n| Size | HellaSwag | ARC_E | ARC_C | OBQA  | RTE   | Average | Wikitext2 |\n| ---- | --------- | ----- | ----- | ----- | ----- | ------- | --------- |\n| 32   | 46.08     | 65.11 | 31.66 | 25.40 | 53.13 | 44.28   | 9.60      |\n| 64   | 46.61     | 64.14 | 32.00 | 25.20 | 53.43 | 44.28   | 9.46      |\n| 128  | 46.60     | 65.49 | 31.14 | 26.20 | 63.54 | 46.59   | 9.39      |\n\nThe performance of PermLLM improve gradually with increased size of calibration data, which also demonstrate the robustness of the proposed methods.\n\n\n\nThank you again for your helpful advice!\n\n\n\nReferences\n\n[1] Gao, Leo, et al. \"The pile: An 800gb dataset of diverse text for language modeling.\" *arXiv preprint arXiv:2101.00027* (2020).\n\n[2] Merity, Stephen, et al. \"Pointer sentinel mixture models.\" *arXiv preprint arXiv:1609.07843* (2016).\n\n[3] Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer.\" *Journal of machine learning research* 21.140 (2020): 1-67."}, {"review_id": "g0RdhbXbgO", "reviewer": "Reviewer_m4zW", "summary": "PermLLM is a post-training pruning framework that makes *channel permutation itself learnable*. The core idea is to relax the discrete permutation matrix into a soft, doubly-stochastic matrix produced by Sinkhorn normalization, then “harden” it back to a true permutation with the Hungarian algorithm while passing gradients through a straight-through estimator. To keep the search space tractable for LLM-scale layers, the permutation is applied **block-wise**, cutting the learnable parameters from $C_{\\text{in}}^{2}$ to $C_{\\text{in}}B$ and reducing the Hungarian step from $O(C_{\\text{in}}^{3})$ to $O(C_{\\text{in}}B^{2})$. Plugged into one-shot N:M pruning schemes such as Wanda and RIA, the learnable permutation is trained to minimize a cosine loss between dense and sparse outputs instead of maximizing a handcrafted importance score. On seven open-source LLMs (OPT-6.7B, LLaMA-1/2/3.1 7–13B, Qwen-2.5 7B) the 2:4 sparse models show consistent gains: perplexity on WikiText-2 drops from 11.59 to **9.41** for LLaMA-7B (vs. Wanda) and average zero-shot accuracy rises by 3–5 pp across five tasks. A custom CUDA kernel makes permutation overhead negligible (0.039 ms) and still delivers 1.67× faster inference than the dense model. Figure 1 illustrates how a naive “max-score” permutation can actually *hurt* accuracy, motivating the learnable approach.", "questions": "1. How do accuracy and training time vary with block size beyond the single ablation shown? Could adaptive per-layer block sizes yield further gains?  \n2. Can learnable permutation be limited to the most sensitive layers (as hinted by a “partial PermLLM” experiment) without retraining dense layers and still match full performance?  \n3. Have the pruned models been evaluated on out-of-domain datasets or downstream fine-tuning to ensure the learned permutations do not overfit the calibration set?  \n4. Since many deployments combine sparsity with 4-bit or 8-bit quantization, does PermLLM remain effective when applied after GPTQ or similar schemes?  \n5. The custom CUDA kernel targets NVIDIA sparse tensor cores. How would the approach translate to hardware lacking these primitives (AMD MI-series, Intel Gaudi)?", "limitations": "PermLLM still requires a non-trivial GPU-heavy optimization loop—unlike heuristic channel permutation it is **not** a free post-processing step. Even with block-wise sharing, the method introduces additional parameters and hyperparameters (temperature schedule, Sinkhorn iterations) that may need tuning per model. The reported speed-up (≈1.67×) is far below the theoretical 2× for 2:4 sparsity, indicating residual inefficiencies. Accuracy gains, though consistent, are sometimes modest (<2 pp) on the hardest reasoning tasks, suggesting diminishing returns at higher sparsity. The method has only been validated on decoder-only transformer LLMs; applicability to encoder–decoder or vision transformers remains to be proven.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "**Strengths**  \n* First to cast channel permutation as a differentiable layer for N:M sparsity and integrate it with existing one-shot pruning methods, replacing brittle heuristics with end-to-end loss minimization.  \n* Sinkhorn–Hungarian–STE pipeline is mathematically clean, and the block-wise design balances search power and cost.  \n* Broad experimental coverage (7 LLMs, language modeling plus reasoning tasks) with reproducible gains over SparseGPT, Wanda-CP, and RIA-CP in both 2:4 and 4:8 sparsity.  \n* Practical implementation: bespoke kernel yields an 84× speed-up over PyTorch permutation and preserves overall sparsity speed-ups.\n\n**Weaknesses**  \n* Training permutations adds wall-time (≈2.5 h for 7B, 5.5 h for 13B on multiple A100s), reducing the appeal of “one-shot” pruning.  \n* Block size is fixed at 64; larger or adaptive blocks are unexplored.  \n* Speed-ups are throughput-only; memory footprint and launch overhead are not discussed.  \n* Analysis focuses on 2:4 sparsity; 4:8 results are promising but less detailed.  \n* Calibration uses only 128 C4 samples, so robustness to domain shifts is unclear.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort the reviewer has invested in evaluating our work. \n\n**W1**: While permutation training adds wall-time, it enables significant performance improvements in pruning. Unlike naive one-shot pruning, which often sacrifices performance, PermLLM achieves a better accuracy, especially on the updated LLMs, such as LLaMA-3.1 and Qwen 2.5. \n\n**W2**: We conduct additional ablation study about block size and we provide a discussion about adaptive block, which can be found in **Q1**. \n\n**W3**: The memory overhead of PermLLM is minimal. For example, if a weight matrix $\\mathbf{W} \\in \\mathbb{R}^{n\\times n}$, PermLLM only need to store the permutation order index $\\mathbf{p} \\in \\mathbb{R}^n$, which is derived from the learned permutation matrix $\\mathbf{P}$. For LLaMA-2 7B, the additional memory of the permutation will be 16KB for each decoder block if INT16 is used for index data. Thus, the overall memory overhead introduced by PermLLM is negligible. Moreover, the kernel launch time is already accounted for in the runtime evaluation.\n\n**W4**: 2:4 sparsity is the primary sparsity pattern recommended by NVIDIA, as it strikes an excellent balance between efficiency and accuracy, making it the key focus of prior research on N:M sparsity. Consequently, our work also primarily focuses on the evaluation and application of this sparsity pattern. We provide additional experimental results on Qwen-2.5 7B under 4:8 sparsity. Due to time limitation, we plan to include more evaluations on 4:8 sparsity in the final version. As the results indicate, PermLLM enhances pruning performance under 4:8 sparsity, demonstrating its strong generalization capability. We also conduct experiments under 1:4 sparsity and 3:4 sparsity, the results can be found in the response of **Q1** by Reviewer fovA.\n\n|          Method          | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  | Wikitext2 |\n| :----------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: | :-------: |\n|          Wanda           |   47.05   | **74.24** |   40.96   |   27.20   |   74.01   |   52.69   |   14.13   |\n|         Wanda+CP         |   47.76   | 73.91     |   40.61   | **28.00** |   76.53   |   53.36   |   12.98   |\n| $\\text{PermLLM}_{Wanda}$ | **48.95** | 73.65     | **41.89** | **28.00** | **76.81** | **53.86** | **12.30** |\n\n**W5**: We just follow our baselines (i.e., SparseGPT, Wanda and RIA) for a fair comparison, which all use 128 samples from C4 dataset as the calibration data. \n\n**Q1**: The results about varying block size are shown as follows:\n\n| Block size | HellaSwag | ARC_E     | ARC_C     | OBQA      | RTE       | Average   | Wikitext2 |\n| ---------- | --------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| 32         | 46.13     | 64.39     | 29.69     | 24.60     | 53.07     | 43.58     | 9.50      |\n| 64         | **46.60** | 65.49     | 31.14     | 26.20     | 63.54     | 46.59     | 9.39      |\n| 128        | 46.47     | **66.08** | **32.08** | **27.40** | **64.43** | **47.09** | **9.07**  |\n\nThe training time is ~2h for block size=32, ~2.5h for block size=64 and ~6h for block size=128. A larger block size provides more optimization potential but also results in longer exploration and convergence times. We set the block size to 64 as the default configuration, as it strikes a good balance between pruning performance and training efficiency. \n\nFurthermore, we believe that using an adaptive, per-layer block size could lead to further improvements. By measuring the sensitivity or importance of each layer, larger block sizes could be assigned to the more critical layers, ensuring better preservation of the most important weights. However, as adaptive block size is not the primary focus of this paper, we leave it as a direction for future work.\n\n**Q2**: As shown in Table 5, partial PermLLM only inserts learnable channel permutation modules to the most sensitive layers, and applies traditional channel permutation method to the rest layers. Partial PermLLM outperforms the traditional channel permutation method but falls short compared to full PermLLM, highlighting a performance gap between the two. Because partial PermLLM leverages learnable channel permutations for the most sensitive layers, it sacrifices a significant portion of the optimization space in model-wide optimization. However, the advantage of partial PermLLM lies in its training efficiency, making it a viable option when computational resources are limited. Moreover, it is important to note that we did not perform any retraining or fine-tuning on the dense layers in any of the experiments, meaning the weight values remained unchanged and fixed. The only component we trained was the permutation matrix, which aims to identify a permutation for generating a better pruning mask for the dense layers.\n\n**Q3**: We follow the experimental settings used by our baselines (i.e., SparseGPT, Wanda and RIA) for a fair comparison. We think C4 is an out-of-domain dataset as the evaluation benchmarks are diverse and different from C4. We also conduct additional experiments using 128 samples from Pile [1] or 128 samples from Wikitext2 [2]. When using Wikitext2 as the calibration data, the pruned model performs better on Wikitext2 compared to using Pile or C4. Since we use C4 as the calibration data and apply the same dataset across all baselines, we believe the settings are fair and appropriate. And if overfitting the calibration data, we believe the results will be much worse on the evalution benchmarks than the existing results. \n\n| Dataset   | HellaSwag | ARC_E | ARC_C | OBQA  | RTE   | Average | Wikitext2 |\n| --------- | --------- | ----- | ----- | ----- | ----- | ------- | --------- |\n| Pile      | 45.83     | 64.31 | 32.08 | 26.60 | 54.87 | 44.74   | 8.96      |\n| Wikitext2 | 45.42     | 66.41 | 32.34 | 25.80 | 53.07 | 44.61   | 8.31      |\n| C4        | 46.60     | 65.49 | 31.14 | 26.20 | 63.54 | 46.59   | 9.39      |\n\n**Q4**: Yes, PermLLM remains effective when applied after 8bit GPTQ, and the results are shown as follows. Here, we perform pruning after quantization and we believe an interesting future work will be pruning- and quantization- aware permutaion learning, which can concurrently find a suitable permutation for improved pruning and quanzation effect.\n\n|            Method             | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  | Wikitext2 |\n| :---------------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: | :-------: |\n|          Wanda+GPTQ           |   41.50   | 61.41     |   30.55   |   23.20   |   53.07   |   41.95   |   12.08   |\n|         Wanda+CP+GPTQ         |   43.94   | 64.52     |   30.80   |   24.00   |   53.07   |   43.27   |   10.92   |\n| $\\text{PermLLM}_{Wanda}$+GPTQ | **46.42** | **65.99** | **31.83** | **27.20** | **53.43** | **44.97** | **9.04**  |\n\n**Q5**: Since NVIDIA GPUs are the dominant platform for research, our efforts primarily focus on developing custom kernels optimized for NVIDIA GPUs, as has been the case with most previous work.  Just as unstructured sparsity does not achieve significant acceleration on GPUs, this has not stopped researchers from extensively exploring it. Similarly, N:M sparsity was initially proposed by NVIDIA, so it is reasonable that it achieves better adaptation on NVIDIA GPUs. Even if acceleration is currently limited to NVIDIA GPUs, there is still a great deal of ongoing research focused on optimizing N:M sparsity.\n\nThe permutation kernel we developed is not strictly limited to NVIDIA GPUs, as the operation itself is essentially just reordering a matrix. If hardware such as AMD MI-series or Intel Gaudi supports parallel operations, we can adapt the kernel accordingly to make it compatible with those platforms, extending the applicability of our approach.\n\n**L1**: \"The reported speed-up (≈1.67×) is far below the theoretical 2× for 2:4 sparsity, indicating residual inefficiencies.\" $2\\times$ speedup is theoretical, and it does not mean everyone can achieve under any cases. SparseGPT and RIA also report speedup under 2:4 sparsity, which can not achieve the theoretical speedup either. Thus, this is not our limitation; it is a limitation of the hardware itself, as well as the gap between theoretical analysis and practical application. The only limitation about runtime is the additional channel permutation operations we introduce. With the developed CUDA kernel, the overhead is minimal.\n\n**L2**: Our methodology is not restricted to decoder-only transformers. As demonstrated in the paper, it does not rely on any specific components exclusive to LLM. We will open source our framework so that other researchers or users are able to extend it to any architectures. Furthermore, due to the structural similarities across model architectures used by encoder-decoder models or vision transformers, it can be easily adapted for them.\n\n\n\nThanks again for your valuable suggestions to improve the quality of our paper!\n\n\n\nReferences\n\n[1] Gao, Leo, et al. \"The pile: An 800gb dataset of diverse text for language modeling.\" *arXiv preprint arXiv:2101.00027* (2020).\n\n[2] Merity, Stephen, et al. \"Pointer sentinel mixture models.\" *arXiv preprint arXiv:1609.07843* (2016)."}, {"review_id": "02WskjfJSd", "reviewer": "Reviewer_fovA", "summary": "The paper proposes PermLLM, a novel framework for N:M sparse pruning of large language models (LLMs), introducing learnable channel permutation (LCP) to optimize sparsity and improve model performance post-pruning.", "questions": "1.All evaluations are performed under 50% sparsity (2:4 or 4:8). Could the performance be further compared under different sparsity levels, such as 1:4 and 3:4?\n\n2.The experimental results in Table 2 are incomplete. A more comprehensive comparison using different methods on the same model should be conducted.\n\n3.It would be beneficial to include some of the latest unstructured pruning methods, such as Pruner-Zero [1] and MaskLLM [2].\n\n[1] Dong P, Li L, Tang Z, et al. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. arXiv preprint arXiv:2406.02924, 2024.\n\n[2] Fang G, Yin H, Muralidharan S, et al. MaskLLM: Learnable semi-structured sparsity for large language models. arXiv preprint arXiv:2409.17481, 2024.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1.The method can seamlessly integrate with existing one-shot pruning techniques like Wanda and RIA, extending their capabilities with adaptive channel optimization.\n\n2.Extensive experiments across multiple models (LLaMA, Qwen, OPT) demonstrate the effectiveness of PermLLM in improving pruning results.\n\n**Weaknesses**:\n\n1.The experimental scope is limited, as only 2:4 semi-structured pruning experiments were conducted, which restricts its generalizability.", "was_revised": true, "final_justification": "The newly added experiments provide a more comprehensive demonstration of the method's effectiveness, so I have raised my score to 4.", "rebuttal": "Thanks for your valuable feedback.\n\n**Q1**: We conduct extra experiments on LLaMA-2 7B under 1:4 sparsity.\n\n|          Method          | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  | Wikitext2 |\n| :----------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: | :-------: |\n|          Wanda           |   55.13   | 75.08     |   42.83   |   30.60   | **58.12** |   52.35   |   5.91    |\n|         Wanda+CP         |   55.56   | 75.38     |   42.06   |   32.00   |   55.70   |   52.14   |   5.87    |\n| $\\text{PermLLM}_{Wanda}$ | **55.79** | **75.97** | **43.34** | **32.80** |   56.32   | **52.84** | **5.81**  |\n\nWe also conduct extra experiments on LLaMA-2 7B under 3:4 sparsity.\n\n|          Method          | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  | Wikitext2  |\n| :----------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: | :--------: |\n|          Wanda           |   26.25   | 26.39     |   19.54   |   11.80   |   52.71   |   27.61   |  2041.50   |\n|         Wanda+CP         |   26.05   | 27.53     | **20.48** | **14.60** |   52.71   |   28.27   |  2128.15   |\n| $\\text{PermLLM}_{Wanda}$ | **26.42** | **29.08** |   20.31   | **14.60** | **53.07** | **28.70** | **744.38** |\n\nThe results demonstrates that PermLLM can further improve the pruning performance under 1:4 and 3:4 sparsity.\n\n**Q2**: In Table 2, we list the better results between $\\text{PermLLM}_{Wanda}$ and $\\text{PermLLM}_{RIA}$ due to the page limitation. \n\nHere, we list more detailed results.  And we will list the complete results in the final version. \n\nFor LLaMA-3.1 8B,\n\n|         Method         | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  |\n| :--------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: |\n|          RIA           |   38.88   | 59.81     |   26.54   |   19.20   |   53.43   |   39.57   |\n|         RIA+CP         |   40.17   | 61.57     |   28.33   |   20.40   |   52.71   |   40.64   |\n| $\\text{PermLLM}_{RIA}$ | **43.44** | **62.63** | **31.83** | **21.20** | **56.32** | **43.08** |\n\nFor LLaMA 7B,\n\n|         Method         | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  |\n| :--------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: |\n|          RIA           |   42.74   | 61.11     |   28.42   |   24.00   |   57.40   |   42.73   |\n|         RIA+CP         |   44.34   | 62.63     |   29.95   | **25.00** |   55.23   |   43.43   |\n| $\\text{PermLLM}_{RIA}$ | **45.49** | **64.48** | **30.72** |   24.00   | **58.12** | **44.56** |\n\nFor LLaMA-2 7B,\n\n|         Method         | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  |\n| :--------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: |\n|          RIA           |   41.81   | 62.12     |   28.07   |   22.60   |   54.15   |   41.75   |\n|         RIA+CP         |   42.86   | 64.69     |   30.29   |   24.40   | **54.87** |   43.42   |\n| $\\text{PermLLM}_{RIA}$ | **45.15** | **64.77** | **32.25** | **24.80** |   54.51   | **44.30** |\n\nFor OPT 6.7B\n\n|         Method         | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  |\n| :--------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: |\n|          RIA           | **42.56** | 59.55     |   26.45   |   22.60   | **54.51** |   41.13   |\n|         RIA+CP         |   44.45   | **60.44** |   26.54   | **24.80** |   51.26   |   41.50   |\n| $\\text{PermLLM}_{RIA}$ |   44.27   | 59.43     | **27.22** |   24.00   |   54.15   | **41.81** |\n\nFor LLaMA 13B,\n\n|          Method          | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  |\n| :----------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: |\n|          Wanda           |   47.16   | 66.12     |   33.11   |   27.40   |   54.15   |   45.59   |\n|         Wanda+CP         |   49.49   | **69.99** | **36.01** | **28.40** |   52.71   |   47.32   |\n| $\\text{PermLLM}_{Wanda}$ | **51.27** | 68.52     | **36.01** |   28.00   | **55.96** | **47.95** |\n\nFor LLaMA-2 13B,\n\n|          Method          | HellaSwag | ARC_E     |   ARC_C   |   OBQA    |    RTE    |  Average  |\n| :----------------------: | :-------: | --------- | :-------: | :-------: | :-------: | :-------: |\n|          Wanda           |   46.77   | 68.43     |   35.75   |   24.60   |   56.68   |   46.45   |\n|         Wanda+CP         |   49.24   | 70.16     |   36.60   | **29.40** |   55.60   |   48.20   |\n| $\\text{PermLLM}_{Wanda}$ | **51.88** | **70.41** | **37.46** |   28.40   | **59.93** | **49.62** |\n\nThese results demonstrate that PermLLM is able to enhance pruning performance regardless of the one-shot pruning method applied.\n\n**Q3**: Yes, we will include Pruner-Zero and MaskLLM for comparison in the final version.\n\nFor LLaMA 7B under 2:4 sparsity,\n\n|          Method          | HellaSwag |   ARC_E   |   ARC_C   |   OBQA    |    RTE    | Wikitext2 |\n| :----------------------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|       Pruner-Zero        | **56.29** |   60.56   | **30.63** | **34.40** |   53.79   |   10.61   |\n| $\\text{PermLLM}_{Wanda}$ |   47.03   | **63.30** |   30.55   |   25.00   | **62.45** | **9.41**  |\n\nFor LLaMA-2 7B under 2:4 sparsity,\n\n|          Method          | HellaSwag |   ARC_E   |   ARC_C   |   OBQA    |    RTE    | Wikitext2 |\n| :----------------------: | :-------: | :-------: | :-------: | :-------: | :-------: | :-------: |\n|       Pruner-Zero        | **54.68** |   61.57   | **32.17** | **32.60** |   53.43   |   10.52   |\n| $\\text{PermLLM}_{Wanda}$ |   46.60   | **65.49** |   31.14   |   26.20   | **63.54** | **9.39**  |\n\nAs the results show, PermLLM can outperform Pruner-Zero on certain tasks even when using Wanda (which is originally much worse than Pruner-Zero) as the pruning metric. Pruner-Zero is designed to automatically search for the optimal pruning metric. Therefore, PermLLM can be integrated with Pruner-Zero, resulting in $\\text{PermLLM}_{Pruner-Zero}$, which utilizes the pruning metric identified by Pruner-Zero. In this way, PermLLM can further improve the performance of Pruner-Zero.\n\nAs for MaskLLM, they use a blended training set and the training overhead is significant, which is different from the experimental settings by us and the baselines we used. We implement MaskLLM using 128 samples from C4 for training, resulting in a perplexity exceeding 5k on LLaMA-3.1 8B and Qwen-2.5 7B models. This demonstrates that the performance of MaskLLM heavily depends on sufficient training data, which is consistent with the findings presented in Figure 4 of their paper. If available, one can perform PermLLM first to find a relative good channel order, which will be beneficial for subsequent mask learning by MaskLLM.\n\n\n\nThank you once again for your valuable suggestions to enhance the quality of our paper!"}]

Author rebuttal:

Meta-review: The paper proposes PermLLM, a novel post-training pruning framework that introduces Learnable Channel Permutation (LCP) for achieving N:M sparsity. The method first learns a soft permutation matrix—specifically, a doubly stochastic matrix—and then refines it into a discrete permutation (as described in Equation 6). This results in an end-to-end trainable pipeline. Additionally, the authors introduce block-wise weight learning, which further reduces training overhead. The paper presents experiments across multiple models to demonstrate the effectiveness of the approach.

All reviewers responded positively to the core idea. Initial concerns focused on the need for more clarification and additional experimental results. During the rebuttal phase, the authors addressed many of these concerns, and the discussion reflected meaningful engagement. Reviewers acknowledged the improvements and expressed increased confidence in the submission.

I recommend acceptance. The proposed framework is technically sound and well-motivated, and the authors have made a good effort to clarify and strengthen the paper during the review process. For the camera-ready version, the authors should include the additional results and explanations discussed during the rebutta
Decision: Accept (poster)