FlexHiNM-GP: Flexible Hierarchical Pruning via Region Allocation and Channel Permutation
Authors: YI XIAODIE
Hayun Lee
Dongkun Shin
Venue: ICLR 2026 Poster (2026)
Keywords: Model Pruning
N:M spasity
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: FlexHiNM-GP: Flexible Hierarchical Pruning via Region Allocation and Channel Permutation

Abstract: N:M sparsity has emerged as a hardware-friendly pruning strategy, notably supported by NVIDIA’s Sparse Tensor Cores. While efficient, its fixed sparsity ratio restricts flexibility, making it difficult to adapt pruning granularity to varying weight importance across layers and architectures.
To overcome this limitation, we propose FlexHiNM, a hybrid framework that adaptively partitions each layer into three regions: dense, vector-pruned, and N:M sparse, enabling finer-grained control while preserving hardware compatibility. To better preserve salient weights, we extend this to FlexHiNM-GP, which incorporates Gyro-Permutation, an iterative channel-rearrangement algorithm. Through successive sampling, clustering, and assignment, Gyro-Permutation aligns high-importance weights with structured sparsity patterns and mitigates suboptimal configurations in multi-level pruning.
During gradual pruning, FlexHiNM-GP further employs a differentiable masking mechanism based on the Hard Concrete distribution, enabling gradient-based mask learning and preventing over-aggressive early pruning. Experiments on vision and language benchmarks demonstrate that FlexHiNM-GP consistently surpasses strong structured baselines and approaches the performance of unstructured pruning, validating the effectiveness of combining hybrid sparsity with learned masks and permutation strategies.

Published as a conference paper at ICLR 2026
FLEXHINM-GP: FLEXIBLE HIERARCHICAL PRUNING VIA REGION ALLOCATION AND CHANNEL PERMUTA-
TION
Xiaodic Yi', Hayun Lee², Dongkun Shin
DDpartment of Computer Science and Engineering, Sungkyunkwan University Department of Electrical and Computer Engineering, Sungkyunkwan University cindy. 135luvêg skku. edu, (1hy920806, Bongkun)@skkun edu
ABSTRACT
N:M sparsity has emerged as a hardware-fricndly pruning strategy, notably sup- ported by NVIDIA's Sparse Tensor Cores. While efficient, its fixed sparsity ra- tio restricts flexibility, making it difficult to adapt pruning granularity to vary- ing weight importance across layers and architectures. To overcome this limita- tion, we propose Mle hybrid framework that adaptively partitions each vector-prumed, layer into three regions: dense, vector prumed, and N:M sparse, enabling finer- grained control while preserving hardware compatibility. To better preserve salient weights, we extend this to FlexHiNM iterative ahaanne--rearrngeenee tering, and assignment, Gyyro-Permutan structured sparsity patterns and mitigates suboptimal configurations in multi-level pruning. During gradual pruning, FlexHiNM-GP further employs a differentiable masking mechanism based on the Hard Concrete distribution, enabling gradient- based mask learning and preventing over-aggressive early pruning. Experiments on vision and language benchmarks demonstrate that FlexHiNM-GG consistently surpasses strong structured baselines and approaches the performance of unstruc- tured pruning, validating the effectiveness of combining hybrid sparsity with learned masks and permutation strategies.
which incorporates Gyro- pyro-Permutation, algorithm. Through successive sampling, clus-
aligns high-importance weights with
INTRODUCTION
Parameter pruning, zeroing out less important weights in a neural network, has become a key tech- nique for managing the growing size of modern deep neural networks (DNNs). Pruning reduces both the number Moating-point operations (FL.Os) and the memory footprint required for model ex- ecution. A wide range pruning strategies with varying granularity has been explored (Han al.) 2015 He et al. 2017;; Tan et al. 2022 Li et al. 2016). In general, fine-graine methods such as unstructured pruning achieve minimal accuracy degradation at given sparsity level. However, their highly irregular weight patterns lead to inefficient memory access, making it challenging to acceler- ate sparse model execution in practice.
To address this issue, N:M pruning (Mishra et al, 2021 has been proposed, which retains only the important N elements every group M elements within a vector, setting the rest to zero. N:M sparsity enables the removal of arbitrary elements within each group, similar to unstructured prun- ing, and thus helps preserve model accuracy after pruning. NVIDIA's Ampere architecture intro- duces Sparse Tensor Cores (STC) that support hardware-level sparse indexing for N:M operations, eliminating the need for additional software-level indexing overhead. specialized instruction for sparse matrix multiplication- iatpipiiicciioacccmllattiooo mma sp, performs the N:M sparse operation by access- ing only the relevant input data as specified by the NM index.
However, N:M sparse pruning suffers from the limitation of fixed sparsity. Current hardware sup- ports only specific N:M patterns (e.g. 2:4), resulting in a fixed 50% sparsity. To support variable
"Coresponding author.

---

Published as a conference paper at ICLR 2026
sparsity levels, Venom (Castro et al. 2023) introduces hierarchical compression technique that combines N:M sparsity with vector wise sparsity. Specifically, several unimportant column vectors first pruned entirely, and then row-wise N:M pruning is applied to the remaining vectors. By adjusting the column vector pruning ratio, the overall sparsity can exceed 50%, allowing for more flexible sparsity control. This composite sparsity scheme is referred to as hierarchical N:M (HiNM) sparsity.
When executing HiNM-pruned models, GPU kernel code uses vector index of the remaining vec- tors to load only the corresponding inputs from global memory into shared memory. Subsequently, N:M sparse operations are performed using the associated NM index.
As illustrated in Figure Da. HiNM such as Venom (Castro et al. 2023) partitions column vectors based on their importance scores (R), defined the fraction of significant elements within each column. Vectors with low scores are entirely pruned (i.e., 0:4 pruning). while the rest undergo 2:4 pruning. However, the wide variance importance scores among retained vectors may reduce the effectiveness of uniform 2:4 pruning. For instance, even if all four elements in vector are important, the 2:4 constraint requires that two them be pruned, potentially discarding useful information.
Raboo imporrat elemens (R)
044 04<R<1
0.45
0.8 < R
0.95 R
0:4
0:4
0.5
(mmi..sp
2-4
R
0.4
(mmiii.sp)
2:4
< 0.25
0:4
0.2
44 (mmaa
(mmaa sp)
R 0.1
0.55
Sorted column vectors (a) HiNM
Sorted column vectors
Sorted column vectors
Sorted columa vectors (d) FlexHiNM-P
(b) HiNM-P
(c) FlexHiNM
Figure 1: HiNM variants
To address the score variance problem, we consider two complementary approaches. The first ap- proach is channel permutation (Pool & Yu 2021), which rearranges the order of column vectors to more uniformly distribute important elements across row vectors. As illustrated Figure [b. the permuted variant, termed HiNM-P, reduces the range of importance scores among vectors subject to N:M pruning. However, channel permutation alone cannot perfectly balance the score distribution across all vectors.
The second approach introduces three-level pruning scheme. Instead of uniformly applying 2:4 pruning to all vectors retained after vector-level pruning, the retained vectors are divided into two regions: a dense region and a 2:4 sparse region (Figure [p). This scheme, called FlexHiNM (Flexible HiNM), preserves entire vectors with high importance scores, while vectors with mid-range scores are pruned under the 2:4 sparsity constraint. During execution, dense vectors are processed using standard matrix multiply-accumulate instructions (mma), while 2:4 sparse vectors are executed using specialized sparse instructions (mma. atm) A key challenge in FlexHiNM is determining the region boundaries (i.e., the proportions of 4:4, 2:4, and 0:4 regions) for a given target sparsity.
While the three-level sparsity scheme allows vectors to adopt different sparsity levels, it does not ensure that the 2:4 sparse region conforms well to the rigid pruning constraint, To address this, Flex- HiNM is extended with channel permutation, forming FlexHiNM-P (Figure It). This arrangement guarantees that dense-region vectors consist entirely of high-score elements, while in the 2:4 sparse region roughly half of the elements in each vector are high-score. Combining three-level sparsity with channel permutation is thus critical for fine-grained control and structural compatibility with 2:4 pruning. To realize this, we propose gyro-permutation (GP), FlexHiNM-aware channel permu- tation method tailored to the HiNM sparsity structure.
The FlexHiNM scheme enhanced with GP, referred to FlexHiNM-GP, integrates four key com- ponents:
1. Region boundary search for three-level sparsity: Given target sparsity level, this algo- rithm optimizes the proportion of each region (4:4, 2:4, and 0:4) to maximize the total importance score of the preserved weights.
2. seroppermutation technique: A channel reordering method that jointly optimizes the der of imput and output channels to better align with both vector-level and 2:4 sparsity constraints.
2

---

Published as a conference paper at ICLR 2026
3. Gradual pruning and fine-tuning flow: We propose gradual pruning framework for HiNM-GP in which weight updates and 2:4 mask learning co-evolve during stepwise fine- tuning. As sparsity increases, both the weights and the pruning boundaries change, ren- dering fixed heuristic selections suboptimal after fine-tuning. To address this, we jointly optimize the 2:4 masks and the weights using the Hard Concrete distribution, enabling the masks to adapt to evolving weight importance while strictly preserving structural con- straints.
4. Custom GPU kernel for FlexHINM-GP: We design an optimized GPU kernll that dy- namically reorders input channels at runtime to ensure cross-layer consistency, thereby eliminating the need for static offline permutation. The kernel processes the 2:4 sparse re- gion and the denser region using separate specialized kernels, and subsequently merges their outputs to produce the final result.
Experimental results on various DNN models including Deit, Bert, and Ilama2, demonstrate that FlexHiNM-GP significantly improves accuracy, achieving performance comparable to that of un- structured sparsity.
2 RELATED WORK
Weight pruning Numerous pruning strategies have been developed, incorporating various sparsity
patterns (He et al. 2017 Han et al.
2017|Han
a 2016 Li et al. 2016), salience estimation methods (Sanh al.)
2020
Frantar et al. 2021 Kurtic et al. 2022), and fine-tuning approaches such as one-shot (Lee)
ct al.
2019) and gradual pruning (Kurtic et al. 2022). Irregular fine-gr grained sparsity often mini-
mizes
accuracy loss but increases indexing overhead. Vector-wise sparsity (Huang et al. 2022 Tan
et al. tween accuracy and efficiency. N:M sparsity has also been advanced with training methods (Zhou
2022 Park et al. 2023), which prunes vectors of shape V x 1, provides a better balance be-
iet al.
|021] Hubara et al.) 2021), layer-wise allocation assuming variable N:M support (Sun et al.,
2021 and optimized GPU kernels (Sun et al. (022). Hierarchical sparsity has been explored in both hardware (Wu et al.)
2023 Liu et al.) (023) and software. Venom (Castro et al.
2023) com-
bines vector- wise sparsity handled in software with N:M sparsity supported by hardware.
Channel permutations Channel permutation techniques (Huang et al. 2022 Tan et al. 2022 Pool
& Yu 202T Ji al., 2018) reorder weight matrices by adjusting input output channels. This aligns weights with the required sparsity pattern and enables efficient removal of unimportant ele- ments. Prior works mainly target single-level sparsity: for column- wise vector sparsity, output chan- nel permutation groups unimportant vectors for removal (Tan et al.) 2022]; for N:M sparsity, input channel permutation redistributes significant elements across rows (Pool & Yu 202T). Tetris (Ji et al. 2018) extends this to block sparsity (c.g., 4 x 4) by swapping channels in both dimensions and introducing index translation across layers, though at high GPU overhead. Our Gyro- perommermttttionn in-
stead integrates index translation into the native indexing of HiNM sparsity during memory transfer, eliminating extra runtime cost.
Learnable semi-structured sparsity Recent works have also explored learnable sparsity, where pruning patterns are optimized jointly with training. MaskLLM (Fang et al. (024) formulates N:M sparsity discrete distribution via Gumbel Softmax, enabling end-to-end mask learning but suf- fering from scalability and costly sampling. S2HPruner (Lin et al., 2024) introduces a soft-to-hard mask distillation framework that couples a continuous mask with discretized version. However, its stability and final patterns depend heavily on hyperparameter tuning and heuristic schedules, limiting generalization across architectures and sparsity ratios.
3 FLEXHINM SPARSE SYSTEM
3.1 MULTI-LEVEL PRUNING
FlexHiNM sparsity pruning adopts a three-level pruning strategy that applies vector wise pruning and N:M structured pruning to different regions of the weight tensor. As illustrated in Figure 2 the original weight matrix is first partitioned into multiple tiles, each corresponding to a group of column vectors associated with the same output channel index. The height of each tile is determined
3

---

Published as a conference paper at ICLR 2026
by the vector size used for vector-level pruning. In FlexHiNM-GIP output channel permutation is applied across tiles, while input channel permutation is performed independently within each tile.
Within each tile, vectors with extremely low importance scores fully pruned using outer-vector- wise pruning (OVW) TTan et al.) 2022). Among the remaining vectors, those with high importance scores selected and grouped into separate dense matrix (shown in red), with their corresponding vector indices recorded. It important to note that the number of selected vectors must be consistent across all tiles to maintain structural alignment during execution. The remaining vectors (shown in blue) are pruned according to the 2:4 sparsity pattern. These pruned vectors are stored in a compact sparse format containing only non-zero elements, along with the corresponding NM indices.
ontpat channel
1OTI 133445
0.1.2.4 424.5 1012131415
Cereen to
Ourgoi
ippr channel
From
Get go
Original Weight
outpur channel
Prunng
22131455
ispui
PreVee Oeene' and (MM part
ippet
Pricing +
Proned wectors is DVW
Prourved nadere aller OVW
veclor index
verlor indes
compressed
NM Index
exsput channel
Sorted Wright
fur boundar seoeching
opeinal ps value
Proned winghte NM
Uiplte
weigh
pd
254
ippr channel
Princl Prined wegtzs NM
Final Weight with regioas
Figure 2: Pruning pipeline
Figure 3: Adaptive region-wise sparsity allocation
3.2 BOUNDARY SEARCH
Our method introduces unified framework that jointly allocates structured sparsity across vector and element levels. As illustrated in Figure 3] the weight matrix is partitioned into three disjoint regions:
Dense region (4:4), N:M sparse region (2:4), and Pruned region (0:4).
To achieve target sparsity 1s, we derive a closed-form relationship between two boundary parame- ters: the vector sparsity boundary (us) and the partial sparsity boundary (ps). Here, us denotes the fraction of fully pruned vectors, while ps represents the fraction of the remaining vectors assigned to N:M sparsity after vector pruning.
Consider weight matrix comprising K column vectors. Vector pruning leaves fraction (1 us) of vectors intact. Of these remaining vectors, a fraction ps is further subjected to the 2:4 structured sparsity constraint, which retains 50% of their elements. Consequently, the sparsity contributed by 2:4 pruming is us) ps x 0.5. Combining this with vector pruning (us) yields the total sparsity ts 178 (1 us) ps 0.5. Rearranging, obtain:
2(1s v8)
ps
U8
For example, in Figure with iss = 50% and us = 25%, we compute ps 66.7%. This means 4 out of 16 vectors (25%) are fully pruned (i.e., channels 3, 6, 7, and 11 tile0), 8 vectors (12 x 66.7%) are placed in the 2:4 sparse region (i.e., channels 1, 2, 5, 8, 9, 10, 13, and 14), and the remaining vectors (i.e.. 0, 4. 12, and 15) are retained for dense computation.
The feasible space of Equation is shown in Figure 4 where each curve depicts the trade-off be- tween vs and ps for fixed target sparsity ts. To select the optimal boundary pair for a given fs, we employ iterative search strategy, illustrated as steps @-- in Figure (right blue box). The steps are described below, with the overall pruning process summarized Algorithm
① Output Channel Permutation & Boundary Initialization: Prior to lector-level pruning, apply an output channel permutation using Pero-Permttttion to enhance pruning effectiveness, cor- responding to the function Gore (we ghts, s, axi sxis=uutptt) in Step 1 of Algorithm For

---

Published as a conference paper at ICLR 2026
target ts, begin with ps meaning N:M sparse region. For instance, when ts 50% (red curve in Figure A the initial values are us = 0.5 and ps =
@
Vector Pruning: The column vectors sorted in descending order of importance to identify those with minimal impact on model capacity. Column-vector pruning is subsequently performed via vectorrrust gwtig us) in Step 4 of Algorithm 1
Algorithm FlexHiNM Sparse Pruning Pipeline
Require: Weight matrix W, target sparsity is, vector size
Ensure: Updated weight W
Split W into tiles {W') of size v, then (W*3
2: for each W do
3:
for us < is step -a do
4:
Compute ps using Equation
W + VectorFrune us);
5: W f- NMP rune (GyroPerri (W', axis=input)) ps);
6:
Evaluate Roxal Ritense + Rz4:
7:
Riotal > Roest then USeest f- U8; p8best f- ps; Riest + Ryotal
8:
else break
9: end
10: end for
11:
end for
W pruuedd +- Get runed (VectorPrune(W, useeet))))
13: W dense &- GW W prined, pShest):
W24 NMPrune (W Wedese" W Wrruned) pSbes)
15: Return W = Wdense U W24 U W Wpruned
N:M Pruning: Among the remaining weight vectors, a subset is designated for N:M sparsity. Specifically, the less important column vectors (a fraction ps of the vectors) undergo row-wise 2:4 pruning, generating the corresponding N:M indices. Since N:M pruning follows row-wise 2:4 pat- tern, apply input-channel permutation via GyroP erm (weights, axi ieisssnnput before performing pruning. is important to note that the input channel dimension must be divisible by satisfy the hardware constraints of NVIDIA's Sparse Tensor Core architecture.
To guide the search for the optimal (vs, ps) pair, we maximize Riotal Raease R24, where Raense and R24 denote the cumulative importance scores of the dense and N:M sparse regions, respectively, computed using the second-order importance metric (Kurtic al. 2022).
Repetition or Termination:
Rather than exhaustively exploring all (us, pairs, we exploit the concavity of the objective (proof in Appendix B). The search proceeds by decreasing with step size a and terminates when the objective value begins to decrease. To prevent overly aggressive decline, the step size is updated as Bos >RMSProp(us, ps), where 3 penalizes excessively large values and 7 adjusts the soft constraint via the RMSPPop term. The configuration at the last monotonic improvement is chosen as the final solution, and pruning then executed according to the resulting boundary values.
0.40
4%
U24
pruned
t 0.50
keep
0.5
0.60
new mask by
0.75
Hard Concrete Module
Dense region
0.0
0.2
0.4
0.6
1.0
0.90
N:M sparse region
is
6%
Pruned region
Figure 4: Relationship between and ps
Figure 5: Mask reconfiguration between sparsity
levels
5

---

Published as a conference paper at ICLR 2026
3.3 FLEXHINM KERNEL
Figure C.1 in Appendix illustrates the kernel design for executing models pruned with FlcxHiNM- GP. As illustrated in F the weight tensor is separated into sparse and dense tiles, each assigned to different CUDA stream: Streamo for sparse tiles and Streaml for dense tiles. It is crucial to note that different tiles use different vector indices. Initially (Ø), both the weight and input matrices stored in global memory. These matrices transferred to the shared memory of each Stream Multiprocessor (2:3), and subsequently loaded into the registers of the processing cores (+))) To fully utilize GPU parallelism, multiple thread blocks across different SMs process tiles concurrently. Additionally, each thread block loads the relevant rows of the input matrix into shared memory, specifically the input row vectors that correspond to the column vector indices of the weight tile.
Stream0: Sparse Tile Execution Each thread block loads its pruned weight tile and only relevant input vectors (e.g. indices 1, 2, 5, 8, etc.) into shared memory. For each output row, the STC engine fetches pruned weights and corresponding inputs via precomputed N:M indices (c.g. 1,3; 0,1) and performs mma sp registers, minimizing memory access and enhancing throughput.
Streaml:
Dense Tile Execution In parallel, dense tiles (e.g. thread blocks 2 and 3) load full tiles with all vector indices (e.g. 0, 12. 15) and use standard mma for dense GEMM computation via tensor cores.
Final Synchronization Results from StreamO and Stream are written directly into the global out- put matrix, where their partial contributions are accumulated using at comccddo without requiring inter-stream synchronization. The complete kernel workflow and execution timeline are illustrated in Appendix
MASK LEARNING
4.1 GRADUAL PRUNING WORKFL.OW
We adopt gradual pruning, progressively increasing sparsity to avoid abrupt accuracy degradation, similar to oBERT (Mgic 2023). At each stage, both pruning boundaries and the set of weights assigned to the 2:4 region may change. A static greedy selection becomes suboptimal once weights updated through fine-tuning. Therefore, we jointly optimize both remaining weights and 2:4 masks during each pruning stage. We parameterize masks using the Hard Concrete distribution, enabling differentiable optimization under strict structural constraints. which allows masks to adapt to evolving weight importance while maintaining monotonic sparsity progression.
As shown in Figure 5) when the target sparsity increases from a% to 5% during gradual pruning, the pruning boundaries are updated accordingly. Channel permutation is performed only during the boundary search stage and is kept fixed throughout training. As result, some vectors in the previous N:M sparse region become fully pruned (3 3) while new vectors transition from the dense region the N:M sparse region (① + @). where fresh masks are generated using the Hard Concrete module. This design ensures monotonic and irreversible pruning (see Appendix D for proof) by restricting mask updates to newly introduced weights only.
4.2 HARD CONCRETE MODULE
To enable differentiable mask learning under the N:M sparsity constraint, we adopt the Hard Con- crete distribution (Maddison al. 2016) parameterize binary masks in a continuous and learnable form.
Step.1 Mask sampling : Each weight w; within a 4-element group assigned learnable logit a and paired with uniform noise 6; ~ 24(0, 1) to generate soft masks 2,, from which exactly two elements are retained per group
Step.2 Generate soft mask Given the sampled noise and logits from Step 1, we compute an intermediate variable s; that serves temperature-scaled activation.
=
(log - -lo((1 - -4)) + loo ox))
(2)

---

Published as a conference paper at ICLR 2026
where (-) is the sigmoid function and E (0.1,1.0] is the annealed temperature. To enable mask learning over a broader range, & is first stretched to the range (7, <) = (-0.1,1.1), and then clipped to [0, 1] to obtain the final soft mask 2 8; s; (S 7) 7, 2, min(1, max(0,
Then 2; is applied to the corresponding weights through element-wise multiplication during forward propagation WA w;; 2; Both the model weights and the sampling logits a, updated via backpropagation, guided by a composite loss:
Σ
= Lrack + Lsqurse + Lhard; Lsparse As mean(z:), Lnant = A mean
(3)
where Lunk denotes the standard task loss, Ixpanse encourages overall sparsity by reducing the average mask value, and Lhard enforces strict 2:4 sparsity within each group avoid invalid pruning patterns; A and A control the relative strengths of these two regularization terms, balancing global sparsity against strict pattern validity during training. Implementation details are provided in Appendix [
Step.3 Hardening and binary mask generation During training, the temperature т is gradually annealed every epochs, making the soft mask to hard mask.
1, 2/ L0,
2; > 0.5
(4)
otherwise
Finally, at the end of each 20-epoch interval, we convert the soft mask into a hard binary mask by applying a 0.5 threshold 2,, as defined in Equation 4
GYRO-PERMUTATION
Channel permutation aligns salient weights with structured sparsity patterns by rearranging chan- nels. Given a pre-trained model and target sparsity, pruming applies a binary mask M to each layer's weight matrix W _mxn, retaining important weights while enforcing structural con- straints (M, 0 indicates pruning). We assume an importance score matrix D E R aligned with W. The pruning mask M must satisfy three constraints: (1) C,,, enforcing binary pruning the column-vector level; (2) C2.4, restricting each four-aligned row vector to two nonzeros; and (3) C.. meeting the global target sparsity. The channel permutation problem can thus formulated as the following optimization:
argmax
II o D[Ao: s.t. M satisfies Co, C2.1. and C.,
(5)
Ao..9....,A7-
Here, Ao and A denote the output channel order and input permutations for the k-th tile, respec- tively. The notation D[Ao; Ay] indicates the reordered score matrix, and || denotes the L.1 norm.
We decompose Equation 5 into two sub- nub-problems: output channel permutation and tile-wise input channel permutation. Accordingly. the Gyro-Pert Permmuterttaioon framework follows a pipeline of output permutation, column-vector pruning, and input permutation. Each iteration comprises three stages: sampling, clustering, and assignment, as illustrated Figure and can be repeated to progressively refine the permutation by capturing interactions between output and input structures.
weight
Sampling
Clastering
Assignment
cost maltix St
cost function
Sy
Po
50
P
1.7
Pr
P
0.9
0.5
0.2
Apply VIDDOE
52
51
0.8
Pa
smaller better
Hunparian marcht
pruned
Pa
5g
52
Figure 6: Cyro-Permmtttion flow
The original weight matrix is partitioned into N tiles (Pa. P.. P2), each with height equal to the vector size. From each tile, M vectors sampled and then grouped into N clusters (so, 81, 82)
7

---

Published as a conference paper at ICLR 2026
through balanced K-means clustering, with each cluster containing M vectors. Pruning costs Ci. are computed for assigning cluster 8, to tile P. (e.g.. C1 = 0.2 for assigning $2 to PD). and the optimal assignment is obtained via the Hungarian algorithm. Implementation details are provided in Appendix F
6 EXPERIMENTS
We conduct experiments three families of models, Deit, Bert, and LLaMA-2. For image classi- fication, we evaluate Deit-Small and Deit-Base on ImageNet- using vector sizes of 64 and 128. For NLP benchmarks, we test Bert-Base on QQP, SST-2, and SQuAD v1.1, and LLaMA-2 7B on six 5-shot reasoning datasets. Each model is pruned at target sparsity levels of 75% to 95%, and evaluated under both vector sizes where applicable.
All experiments are run on NVIDIA RTX 4090 GPU with gradual pruning schedule starting from 30% sparsity, increasing the target in four stages. Each stage begins with mask reinitialization, followed by 20 epochs of joint optimization of weights and Hard Concrete masks.
6.1 ACCURACY EVALUATION
Figure 7 presents the results for transformer models using vector-size of 64: Unstructured corresponds to element-wise pruning: HiNM-GP represents the HiNM pruning with Gyro- permutation; F FlexHiNM-GP denotes the flexible Hierarchical pruning via region allocation and Gyro-permutation: OVW denotes the traditional out-vector-wise pruning TTan et al. 2022). as a base- line since it be regarded as special case of our framework, representing a lower bound of accuracy for FlexHiNM as illustrated in Figure 9 HiNM-V is a variant of our method that omits the Pyyo--Permutatonn process, which is mainly equivalent to Venom, to isolate the effect of channel permutation.
FledHINM GP
Acc, 5% (tpp1)
Deit-Small
Deit-Base
Sparsity
Sparsity 87.5%
78.64
73,52
79.46
Nobue
5535
BLSS
$7.25
63
75%
87.5%
80%
87.5%
Sparsity
Sparsity
QQP
SST-2 SQJAD
QQP
SSTE2 SQUAD
(a) Results Deit family
(b) Results Bert-Base
Figure 7: Gradual pruming for transformer models (V=64)
FlexHiNM GP achieves the highest top-1 accuracy among structured pruning techniques Deit- Base under moderate sparsity, reaching 81.13% and 79.46% at 75% and 80% sparsity, respectively (Figure 7p. Comparing HiNM-V and HiNM-GP highlights the role of channel permutation: with- out it, Hi NMN suffers from misalignment between salient weights and preserved tiles, causing greater degradation as sparsity increases; Hi NM-GP alleviates this by redistributing important ele- ments across row groups through pyro-permutation, improving compatibility with the 2:4 constraint, while FlexHiNM-GP further improves performance by coupling permutation with flexible region allocation.
As shown in Figure 7p. FlexHiNM GP achieves the best structured pruning performance Bert, attaining 88.55 Fl on SQuAD and 91.65% accuracy on SST-2 at 75% sparsity, closely matching the unstructured baseline (89.04 and 91. 11.86% respectively). At 87.5% sparsity, where all methods exhibit noticeable degradation, Fl FlexHi NM-GP remains the most robust, reaching 87.23 on SQuAD and 87.11 on SST-2 and consistently outperforming HiNM-GP by small margins.
Table 1 reports the performance of different pruning strategies on six downstream benchmarks on LLaMA2-7B at sparsity levels of 75% and 87.5%. Unstructured pruning yields the highest raw scores but suffers from large task variance and lacks hardware support, while structured meth- ods such as OVW and HiNM-V incur substantial degradation. Incorporating channel permutation further improves average accuracy by
in HiNM-GP alleviates part of the drop, and F FlexHiNM- +1.33% over HiNM-GP at 75% sparsity, with the most pronounced gains on BoolQ and HellaSwag where semantic sensitivity is critical. When sparsity increases to 87.5%, all methods decline pre-
8

---

Published as a conference paper at ICLR 2026
serve the same ranking, and FlexHiNM-GP consistently outperforms HiNM-GP (e.g., 65.72 64.18 on PIQA, 54.38 vs. 52.71 on ARC-E).
Method
75%
87.5%
OBQA
ARC-C
PIQA
BoolQ
HellaSwag
ARC-E
OBQA
ARC-C
PIQA
BoolQ
HellaSwag
ARC-E
Dense
32,06
43.08
79.62
77.74
57.03
76.39
32.06
43.08
79.62
77.74
57.03
76.39
Unstructured
23.84
71.75
20.55
OVW
19.66
27.94
65.13
52.27
39,86
53.77
15.88
23.75
61.32
48.56
34.28
48, 88.92
HINM-V
20.33
28.58
66.16
53.18
40.63
54.96
16.72
24.56
62.45
49.61
35.17
$0.34
HINM-GP
22.04
68.87
55.91
42.72
57.84
18.06
26.33
64.18
51.37
36.82
52.71
FlexHINM-GP
23.12
31.62
70.14
57.23
44.58
59.22
19.23
27.58
65.72
52.46
38.11
$4.38
Table 1: Llama2-7B Downstream task performance (%)
Consequently, these results indicate that FlexHiNM-OG GP narrows the gap to unstructured pruning while retaining full hardware compatibility, achieving more favorable trade-off between accuracy and structured sparsity, stabilizing performance under high compression while preserving task gen- eralization in large-scale language models. Additional evaluations under extended sparsity levels (up to 95%) and different vector sizes (64 and 128) are reported in Appendix to further validate robustness across model scales and compression regimes.
6.2 INFERENCE EVALUATION
We evaluate the inference efficiency of the proposed sparsity schemes using our custom Tensor Core kernel NVIDIA RTX 4090 GPU. The benchmark is conducted Deit-Base and Bert-Base with batch size set to 1, averaged over 30 post-warm-up runs.
The observed latency behavior is consistent with the memory-accese characteristics of each pruning rule, as shown Figure 8 OVW achieves the fastest execution, as it loads only the input elements required for computation. HiNM-GB GP, in contrast, loads full input vectors but uses only a subset of elements determined by the 2:4 pattern, resulting in additional unused loads that reduce efficiency relative to OVW.
FlexHiNM-GP processes both dense (4:4) vectors and 2:4-pruned vectors, resulting in the compute pattern that lies between OVW and HiNM-GP, with latency naturally interpolating between the two baselines. This trend is consistent across both Deit-Base and Bert-Base. Notably, flexHíNM-- maintains these inference gains while preserving partial dense regions, which contributes to its higher accuracy. The small latency overhead introduced by these dense vectors is thus well justi- fied when considering the trade-off of accuracy-efficiency.
OVW
HENM-GP
FIMMGGP
0.50
Is 0.75
0.50
Deit-Base
Bert-Base
3,05
1.00
HINAGG
Speed-un up
2.46
2.09
0.0.5
75%
87.5%
90%
75%
87.5%
90%
0.0
0.2
0.4
0.6
0.8
1.0
Sparsity
Sparsity
Figure 8: Speed comparison across sparsity levels
Figure 9: Variants based on boundaries
6.3 ABLATION STUDY
6.3.1 COMPARISON WITH GUMBEL SOFTMAX
Table 2 illustrates the ablation study over six variants to compare different noise functions used for mask learning- Variant @ is our proposed Fll TexHiNM-GS which achieves the highest overall aver- age accuracy among all variants, while Variant @ corresponds to the HiNM-GP bascline. Between them we construct four intermediate variants by toggling the use Flexible Distribution (F), and switching between Hard Concrete (H) and Gumbel-Softmax (G) for mask generation.
While Gumbel- Softmaxx enables differentiable 2:4 pruning by selecting from six fixed patterns per group (Fang al. 2024), its limited search space restricts fine- finennrained saliency modeling. In trast, Hard Concrete generates continuous, near-bine binary masks independently for each element, offer- ing greater flexibility while still supporting structured enforcement. It also facilitates stable gradient

---

Published as a conference paper at ICLR 2026
flow through hard-sigmoid activation and avoids categorical sampling, making it scalable choice for structured sparsity learning.
G
75%
80%
87.5%
90%
95%
Variant
75%
80%
87.5%
90%
95%
V
81.13
79.46
75.23
69.26
61.43
V
V 81.10
79.27
75.55
68.66
59.77
HiNM-GP
81.04
78.28
74.35
67.64
58,94
V
81.08
79.12
75.34
68.50
59.64
OVW
78.67
76.34
71.26
64.87
56.82
4
V
81.07
78.49
74.68
68.
59.19
BalFlexHiNN
80.38
78.51
75.09
68.45
57.26
80.82
78.77
74.91
68.29
59,42
OptFlexHiNM
81.13
79.46
75.23
69.26
61.43
81.04
78.28
74.35
67.64
58.94
Table 3: Ablation results under boundaries on
Table 2: Ablation results under variants on Deit- Base
Deit-Base
6.3.2 COMPARISON WITH BALANCED FLEXHINM-GP
Figure compares four variants under different boundary settings defined by ps and es. OVW (pink) applies only vector pruning ps 0: H NM-GP (orange) uses only N:M sparse the whole dense part after OVW since ps = 1; BalF 1 LexHiNM------ (Balanced FlexHiNM, purple) fixes at = 0.5; and OptFlexHiNM-- GP (Optimal FlcxHiNM, blue), representing our core method, adaptively searches both boundaries to achieve the best performance.
Table demonstrated that OptFlexHiNg consistently achieves the highest accuracy across all sparsity levels, which outperforms HiNM-GP by +1.62% at 90% sparsity and +2.49% at 95% spar- sity, indicating that greedy boundary optimization effectively preserves important information while enforcing structured sparsity. Unlike manually fixed strategies such as OVW BalF FlexHiig the adaptive approach delivers a better trade-off between accuracy and structural regularity, which is particularly crucial for VIT models sensitive to pruning design.
6.3.3 EFFECT OF PYRO-PERRUTATION
Finally, we provide a direct ablation comparing FlexHiNM (FlexHiNM-GP without Gyro- Permutation) with FlexHiNM-GG GP at 75% sparsity (vector size 64), isolating the effect of Gyro- Permutation. As shown in Table GP consistently improves performance across all Bert-Base tasks; for example, FlexHiNM-GP improves QQP Fl from 84.78 to 85.35 and SQuAD Fl from 88.04 to 88.55.
Table 4: Impact of Gyro-Permutation on FlexHiNM
QQP
SST-2
SQuAD
Method
F1
Acc
Acc
Fi
FlexHiNM-GP
85.35
87.82
91.65
88.55
FlexHiNM
84.78
86.42
90.60
88.04
The accuracy gaps between FlexHiNg NM and Fl elexHinee NM-GP show that three-region partitioning alone does not fully resolve channel misalignment introduced by structured N:M sparsity. Gyro- Permutation provides the additional alignment needed for stable improvements, and its effect per- sists across tasks, even at moderate pruning ratios. These results confirm that the benefit of GP is setting-specific, but stems from correcting the structural mismatch remaining after hierarchical partitioning.
7 CONCLUSION
We present FlexHiNM-GP, a unified framework for multi-level sparsity that combines adaptive region partitioning with permutation-aware pruning. It learns dynamic boundaries for column- wise and N:M sparsity, while Pero-Permttation efficiently aligns salient weights through coordi- nated channel permutation. With Hard Concrete- based differentiable mask learning, FlexHiNM-GP achieves higher accuracy across diverse sparsity levels, demonstrating its effectiveness for large- scale model compression.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by Samsung Research Funding & Incubation Center of Samsung Electron- ies under Project Number SRFC-IT2402-15.
REFERENCES
Roberto L Castro, Andrei Ivanov, Diego Andrade, Tal Ben-Nun, Basilio B Fraguela, and Torsten Hoefler. VENOM: A vectorized N:M format for unleashing the power of sparse tensor cores. In Proceedings of the International Conference for High Performance Computing. Networking, Storage and Analysis. pp. 1-14, 2023.
Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. MaskLLM: Learnable semi-structured sparsity for large lan- guage models. Advances in Neural Information Processing Systems, 37:7736-7758, 2024.
Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-orde information. Advances in Neural Information Processing Systems, 34: 14873-114888, 2021.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances neural information processing systems, 28, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. In International Conference on Learning Representations, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Proceedings of the IEEE international conference computer vision, pp. 1389-1399 2017.
Guyue Huang, Haoran Li, Minghai Qin, Fei Sun, Yufei Ding, and Yuan Xie. Shfl-bw: accelerating deep neural network inference with tensor-core aware weight pruning. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pp. 1153-1158, 2022.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find N:M transposable masks. Advances in neural information processing systems, 14:21099-22111 2021.
Yu Ji, Ling Liang, Lei Deng, Youyang Zhang, Youhui Zhang, and Yuan Xie. Tetris: Tile-matching the tremendous irregular sparsity. Advances in neural information processing systems, 31, 2018.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal bert surgeon: Scalable and accurate second-order pruning for large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4163 4181, 2022.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for
efficient convnets. In International Conference on Learning Representations, 2016.
Weihao Lin, Shengji Tang, Chong Yu, Peng Ye, and Tao Chen. S2HPruner: Soft-to-hard distillation
bridges the discretization gap in pruning. Advances in Neural Information Processing Systems, 37:116547-1166 2024.
Jun Liu, Guohao Dai, Hao Xia, Lidong Guo, Xiangsheng Shi, Jiaming Xu, Huazhong Yang, and Yu Wang. Tste: Two-level sparsity tensor enabling both algorithm flexibility and hardware efficiency. In 2023 JEEE/ACM International Conference on Computer Aided Design (ICCAD), pp. 1-9. IEEE, 2023.
11

---

Published as a conference paper at ICLR 2026
Chris J Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relax- ation of discrete random variables. arXiv preprint 4xxi::1611.07122.2006
Neural Magic.
2023.
Accessed: 2024-11-15.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.088: 08378. 2021.
Cheonjun Park, Mincheol Park, Hyun Jac Oh, Minkyu Kim, Myung Kuk Yoon, Subyun Kim, and Won Woo Ro. Balanced column-wiss block pruning for maximizing gpu parallelism. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pp. 9398-9407, 2023.
Jeff Pool and Chong Yu. Channel permutations for N:M sparsity. Advances in neural information processing systems, 34: 14:13316-11333 2021.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine- tuning. Advances in neural information processing systems, 33:20378-20389 2020.
Wei Sun, Aojun Zhou, Sander Stuijk, Rob Wijnhoven, Andrew O Nelson, Henk Corporaal, et al. Dominosearch: Find layer-wise fine- ane-ggrined N:M sparse schemes from dense neural networks. Advances in neural information processing systems, 34:20721-20777 2021.
Wei Sun, Ang Li, Tong Geng, Sander Stuijk, and Henk Corporaal. Dissecting tensor cores via microbenchmarks: Latency, throughput and numeric behaviors. IEEE Transactions on Parallel and Distributed Systems, 34(1):246 261, 2022.
Yijun Tan, Kai Han, Kang Zhao, Xianzhi Yu, Zidong Du, Yunji Chen, Yunhe Wang, and Jun Yao. Accelerating sparse convolution with column vector-wise sparsity. Advances in Neural Informa- tion Processing Systems, 35:30307-30317, 2022.
Yannan Nellie Wu, Po-An Tsai, Saurav Muralidharan, Angshuman Parashar, Vivienne Sze, and Joel Emer. Highlight: Efficient and flexible dnn acceleration with hierarchical structured sparsity. In Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture, pp. 1106-1120, 2023.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning N:M Fine- tineggraieed Structured Sparse Neural Networks From Scratch. In International Conference on Learning Representations, 2021.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
No but not all. We only used LLMs to polish some sentences.
B CONCAVE OPTIMAL TARGET
We consider the remaining weight magnitude Rt as function of vector-wise sparsity and pre- served proportion ps after pruning. Our goal is to demonstrate that R(us,ps) is concave, implying the existence of global optimum when jointly searching and ps.
Let W E R=x" be the original weight matrix with column-wiss wise importance scores sorted in de- scending order. The vector pruning stage removes proportion of column vectors, preserving a fraction (1 vs). The following 2:4 pruning stage retains exactly 50% of the weights in each preserved column. For given target sparsity ts, the total preserved portion is constrained such that:
2(ts vs) and thus, R(vs) = W(ls vs).
This formulation is consistent with HiNM-style hierarchical pruning, where ps is a function of us under fixed glohal sparsity.
Assume that the weight importance scores across columns follow a monotonically decreasing curve (c)), where [0, C*] is the column index after sorting. We then express the remaining weight magnitude as:
R(vs) W (c(us)) (Is uss,
where c(us) C(1 us). and C is the total number of column vectors.
Let us locally approximate ((c) as a linear function: (c)) = -ac + b, where a >> 0, b > 0. Substituting into R(us), obtain:
R(es) = (-aC(1 - u8) + b) (ts - us) = (aCus + b - aC) .ts v8).
Taking the second derivative:
R -2C < O.
dus
Hence, R(us) is concave under this approximation. More generally, because W(c) decays sub- linearly in practice (due to long-tailed weight distribution), the composite function R(vs) = ((c(ss) (ts us) maintains concavity.
Conclusion. Since R(us) is a smooth, concave function on compact domain es E [0, ts), it admits unique global maximum. This guarantees the existence of an optimal solution for selecting us (and indirectly ps), which can be efficiently searched. Therefore, the hybrid pruning target is formally concave objective.
C FLEXHINM KERNEL
Figure C.1 illustrates the kernel design for model pruned with FlexHiNM-GP. The weight tensor is separated into sparse and dense tiles, each assigned different CUDA stream: Streamo for sparse tiles and Streaml for dense tiles. It is crucial to note that different tiles use different vector indices. Initially (Q), both the weight and input matrices are stored in global memory. These matrices are transferred to the shared memory of each Stream Multiprocessor (23), and subsequently loaded into the registers of the processing cores (00). To fully utilize GPU parallclism, multiple thread blocks across different SMs process tiles concurrently. Additionally, each thread block loads the relevant rows of the input matrix into shared memory, specifically the input row vectors that correspond to the column vector indices of the weight tile.
Stream0: sparse tile execution Each thread block (e.g. thread block 0 and 1) loads its corresponding weight tile and only the relevant input vectors into shared memory. These input vectors are selected
13

---

Published as a conference paper at ICLR 2026
Memory
00
SCW dend Indening
SrW-land ladeding
Smese!]
(deam))
vectur
Sharee Mimurr
Sttrral Mumerry
lppt Ireakkert
Thread Black
Thread Block
(regiter)
Irgun
Starred Menury
Sliredd Menory
Vocker lades
Block
STC
selpai (regkter)
Reel valpet
NML Eudos
5:34 10003
XJH ia0cs
MEESHT (reabter)
lepat Orgitte)
neiget
devel Indenteg
Jevel Indenteg
Strewnd (p)
Figure C.I: FlexHiNM kernel
by analyzing the column vector indices retained in the sparse pattern (e.g. vector indices: 1, 2, 5, 8, 9, 10. 13, and 14 for thread block 0). During calculating (Ø), the STC engine loads each pruned weight row and its corresponding input row from registers, based on the precomputed N:M index (e.g. 1, 3. O, 1). It performs sparse matrix multiplication using the mma sp instruction in regis- ter space. This fine-grained activation-awan loading reduces unnecessary memory access, thereby maximizing throughput.
Streaml: dense tile execution parallel, Streaml handles dense tiles ($), such as thread blocks and 3. These blocks load complete tiles from global memory using full vector indices (e.g. 0, 4, 12, 15 for thread block 2). Because no pruning is performed in these tiles, mma is applied using standard tensor ()) directly computing dense outputs.
Final synchronization Upon completion (#G), results from Streamo and Streaml are written directly into the global output matrix, where their partial contributions accumulated using at atomic cAdd without requiring inter-stream synchronization.
D MONOTONICITY CONSTRAINTS
To ensure theoretical consistency and implementation safety of the gradual pruning schedule, establish two necessary monotonicity conditions on the pruning variables. These properties serve as foundational guarantees for the correctness of our mask evolution mechanism.
Case 1: Monotonicity of us Let us denote the proportion of input vectors that have been en- tirely pruned. When the target sparsity iss increases to drop would imply that already-pruncd vectors are reactivated, which violates pruning principles. Therefore, must be non-decreasing with respect to ts:
time, must not decrease. Allowing
dus dts
2 0.
Case 2: Decreasing trend remaining dense weights Let N denote the as the number total vectors in one weight tensor, and M denote the number of dense weights remaining in the tensor. Given the analytical relationship:
M N N us 2N us) ((1 -tts).
Then take us (2ts - ps)
(2- - ps) into and we get:
ts + ps ts
M = 2N
2 ps
14

---

Published as a conference paper at ICLR 2026
derive that M is decreasing function of ts, provided that increases ts increases. Specifi- cally:
dM
ps
(for ps 1, always satisfied),
dts
2 ps
and
d M
(2 - ps)? > 0.
dts²
According to the conclusion in Case.1 that es must increase, ps will also increase monotonicity.
This confirms that the number of remaining non-zero weights decreases smoothly with increas- ing sparsity target, avoiding oscillation or recovery of previously pruned weights. These two cases together justify the design of our pruning schedule and guarantee its progressive and irreversible nature.
E DETAILS OF THE LEARNABLE MASK ALGORITHM
To enable differentiable mask learning under the N:M sparsity constraint, adopt the Hard Con- crete distribution to parameterize binary masks in a continuous and learnable form.
Step.1: Mask sampling To support gradient based mask optimization, update the binary mask every 20 epochs using a differentiable sampling procedure over grouped weights. We treat every row-wise group of four consecutive weights (wf, w³, wy. w 0, 0, T) as unit g to enforce the 2:4 sparsity constraint, which the set of all these T groups is denoted by G. Within each group, each element w is associated with learnable logit o;,. Independently. noise variable e; (1, 1) sampled for each element. These logits and sampled noise values are used to generate per-element soft masks 2,, which are later constrained the group level to retain exactly two out of four elements.
Algorithm 2 Sub-training for 20 epochs
Require: Weights W. Model f()), inputs X, epochs E-20,
Require: Required hyper-params (To, Ty), (7, (), de A
Ensure: Mask M
1: Establish 4-element groups g
b initial logits
2: for all g do
3: 0gg x {0)
4: end for
5: Initialize the temperature T + Tf
6: for epoch
E do
7: for all E G do
8: Uniform sample: = {e:} ((, for each g
9: Generate the middle variable: t a ((log - log (1 c)+ + log a)) /T) Strech and normalize s: + s(5 7) + 7 2ooa min (1, max (0, s))
11:
end for
Z (ZxoR)
J:W <- } W
13:
Cusk t loss (f(W, X))
14:
Cyparse t Ag- mean (2)
15:
Chard +- An mean
E Zsotlg -2l)
16:
L+ Liack + Lsparse + Lhand
17:
Back-propagate L (W, a)) +- (W,a) - v/Wa.h
T T (T) To)/Ei epoch
19: end for
for all g do
21: Zhard + owwhot (Esot(g)) .4)
end for
23: Return M (ahand)
15

---

Published as a conference paper at ICLR 2026
Step.2: Generate soft mask 2, Given the sampled noise and logits from Step compute intermediate variable 8, that serves as temperature-scate activation.
s; (+ (logee - log(1 (1 (.) log o:))
where (-) is the sigmoid function and r E [0.1, 1.0] is the annealed temperature. To enable mask learning over broader range, & first stretched to the range (7, = (-0.1, 1.1), and then clipped to [0, to obtain the final soft mask
s; = s; (S 7) + 7
min(1, max(0, 5,)).
Then 2; is applied to the corresponding weights through element-wise multiplication during forward propagation W; wi Both the model weights and the sampling logits as are updated via backpropagation, guided by a composite loss:
I = Lunk + Tiguare Lhard
Esprre >a mean (2))
4
Liard X mean
23 2
where Lusk denotes the standard task loss. Lopasse encourages overall sparsity by reducing the erage mask value, and Liar enforces valid 2:4 sparsity within each four-element group g. In our implementation, the regularization weights set to X mask variables 2 E [0, 1] remain continuous during training, so Taparse typically produces gradi- ents of larger magnitude than Laard Setting Åe one order of magnitude larger than A balances the influence of the two terms: A softly drives the model toward a target sparsity level, while Åe penal- izes deviations from the strict 2:4 pattern, ensuring that each group converges valid structured configuration before binarization.
10-1 and A 1 x 10-2, The
Step.3: Hardening and binary mask generation During training, the temperature is gradually annealed every 5 epochs, making the soft mask to hard mask.
1, if > 0.5
24
0, otherwise.
(10)
Finally, at the end of each 20 epoch interval, we convert the soft mask into a hard binary mask by applying a 0.5 threshold to 2;, as defined in Equation 10 The procedure repeated every 20 epochs, as summarized in Algorithm
F GYRO-PERMUTATION
F.1 OVERALL ALGORITIMM
Eyro-Permutation is our pruning pipeline consisting of two sequential components: output channel permutation and input channel permutation. It first globally reorders output channels by importance to facilitate efficient vector removal, then locally adjusts input channels to enhance the effectiveness of N:M pruning.
HiNM pruning requires an effective permutation algorithm that accounts for both sparsity patterns across different channel dimensions. To design effective permutation algorithm for HiNM, the following issues must be addressed:
Hierarchical Pruning Awareness: Since pruning is applied to both output and input chan- nel dimensions in the HiNM method, optimal channel permutation for one dimension may not the best overall solution. For example, while an output channel permutation may cluster important elements into the same vector, subsequent N:M pruning might elim- inate some of these preserved elements. This interdependence adds complexity to channel permutation in HiNM pruning.
16

---

Published as a conference paper at ICLR 2026
2. Cross-Layer Consistency: Channel permutation must ensure that the channel orders consistent between adjacent layers. Specifically, if the output channel order of the i-th layer is rearranged, the input channel order of the (i + 1)-th layer must be adjusted accordingly. Ensuring cross-layer consistency is straightforward when applying channel permutation to only one dimension. However, in HiNM, where both input and output channels are per- muted in each layer, it becomes crucial to address the potential interference between layers during the permutation process.
3. Local Minima Problem: We observed that existing permutation techniques easily fall into local minima. Addressing this challenge essential for enhancing the performance of HiNM pruning.
For given pre-trained model and target sparsity, pruning method applies a mask M, adhering to specific structural constraints, to the weight matrix W E m in each layer, with the goal of maximizing the retention of important weight elements. A value of 0 in element of M indicates that the corresponding element in W will be pruned. Channel permutation is an effective strategy to achieve this goal, as it rearranges channels position important elements in the weight matrix in alignment with the structural constraints.
We assume that importance the importance score of the corresponding element in the pre-trained weight matrix. The pruning mask constraints, specifically the vector mask constraint C, and the 2:4 mask constraint C2.4, must be taken into consideration. The constraint C. requires that all elements in column vector must be either or while C2.1 mandates that only two out of every four elements a value of in the mask. Additionally, the target sparsity constraint C' must be addressed. Since the sparsity achieved by 2:4 pruning is fixed 50%, C, refers to the corresponding vector sparsity that satisfies the target sparsity. With these constraints in mind, the optimization problem for channel permutations in HiNM pruning can be defined as follows:
matrix D R is given, where each element represents
row vector can have
argmax Ao Ao.A% A7. A7-1
| o LDDDAAS D[Ao:Ay]]
(11)
s.t.
M satisfies Cuy C2:4. and Cgs
Here, Ao represents the order of the output channels, while A, denotes the set of input channel orders for tiles, specifically Aq...., A where A is the channel order for the k-th tile, The notation D[Ao: Az] refers to the matrix D rearranged according the permutations Ao and A₁, || represents the L1 norm.
The mask search problem encompasses vast search space. To streamline the search for effective permutations, we decompose the problem into two sub-problems: output channel permutation and tile-wise input channel permutation. The output channel permutation aims group less important elements within the same vector, thereby reducing the likelihood that important elements are pruned simply because they share vector with unimportant ones.
However, it is not necessary to completely segregate important and unimportant elements into differ- column vectors, as elements retained after vector-wise pruning may still be removed during the subsequent N:M pruning. Instead, the focus is on creating sufficient number of vectors containing only unimportant elements. The output channel permutation finds the vector pruning mask M, and can be formulated as follows using M., which denotes the inverted mask of Mg:
argmin M, D[Ao!]| s.t. M, satisfies Cp and C...
(12)
After the output channel permutation, the importance score matrix D is transformed into D' by reordering channels and removing the pruned vectors (D' M, O D[Ao]). The next step is to opti- mize the arrangement of column- volumn-wise vectors within each tile to ensure that the weight distribution aligns with the desired uniform sparsity across row-wise vectors. It finds the 2:4 pruning mask M2.4 and can be formulated as follows:
17

---

Published as a conference paper at ICLR 2026
argmax A
|M2.1 + D'[A)]l s.t. M2.1 satisfies C..
(13)
F.2 THE DETAILED ALGORITHM
Syro-Permutation consists of three steps: output channel permutation, column vector pruning, and input channel permutation. The optimal permutation scheme will be obtained through finite num- ber of iterations, with each iteration process divided into three steps: sampling, clustering, and as- signment, as illustrated Figure Gyro-Permutation can be applied iteratively to account for the interactions between output and input channel permutations.
Sampling, For the output channel permutation, the weight matrix is first divided into several tiles, each containing column vectors within the same (note that each tile corresponds to tile in GPU execution). A fixed number channels from all tiles are then selected and rearranged. In Figure 66 there are 9 rows of the original weight, while it is divided into 9/3=3 tiles, Po, PI, because the vector-size is 3. Then we assume that 2 samples are extracted from each tile.
Step.1:
The number of sampled channels plays role analogous to the learning rate in optimization. Larger sample sizes promote global exploration and help escape poor local minima, whereas smaller sam- ple sizes emphasize local refinement but risk premature convergence. To balance these behaviors, Gyro- Syro-Permutation adopts a sumpling-and-clustere and small-sample steps. The sampling schedule follows a coarse-to- pattern (e.g., 1, 4, 1, 2, 1, 1), where large-sample phases broaden the search space, and small-sample phases stabilize the configuration after each perturbation. This alternating update prevents both oscillation and early stagnation.
strategy that alternates between large-sample
The effectiveness of the permutation significantly influenced by the number of samples extracted from each tile, much like the role of learning rates in model training. Generally, extracting larger number of samples helps avoid local minima but may make it harder to reach the absolute optimum. Conversely, extracting fewer samples can facilitate reaching the optimum but increases the risk of encountering local minima. iterative Pyyro-Permutatinn the number of samples can be dynami- cally adjusted in each iteration, much like how learning rates are adapted during model training.
The input channel permutation for N:M sparsity is applied on a tile-wise basis, meaning each tile can have unique permutation. The tiles used in the output channel permutation further divided into sub-tiles, each consisting of M column vectors, where M is the basic vector size defined by N:M sparsity. Due the limited size of these sub-tiles, only one column vector from each sub-tile is selected for permutation.
Step.2: Clustering, Clustering is used group channels with similar importance scores, creating T' equally sized groups, where 7 is the number of tiles.
After Sampling step, importance scores for each candidate channel are calculated features input into the Clustering step. We use the Balanced K-means clustering algorithm as it effectively ad- dresses the issue of uneven sample sizes, ensuring that the number of channels in each cluster approximately equal. As shown in Figure 80. 82, represent the sample groups after clustering.
Step.3: Assignment. During this phase, sample groups after clustering will be assigned into dis- tinct tiles. The target tile for each group is determined by a cost function Ci,j, which represents the pruning cost when the j-th sample group is assigned to the i-th tile. This value derived by implementing the target pruning. For example, in Figure C1,2 has value of 0.2.
After calculating the cost for all combinations of tiles and samples, the Hungarian algorithm used to find the combination that minimizes the total cost. These three steps will be repeatedly executed until the cost of a given iteration exceeds that of the previous one. In the example shown in Figure the samples So, $1, and 82 assigned tiles Pz, Po. and Pt, respectively.
To facilitate the finite iterations, a list J = Jo. ,J is given, wherein each el- ement specifies the number of channels to be extracted during the sampling phase of each it- eration. The overall permutation process illustrated in Algorithm 3 By changing the direc- tion of channel reordering, combine functions GyroPerm (wei wweights axi s=output) and
Overall
18

---

Published as a conference paper at ICLR 2026
GyroPerm (weights, axis=innut), which are applied respectively before the vector pruning and the N:M sparse pruning stages.
Algorithm 3 Permutation Iteration Process
Require: Weight matrix W, iteration scheme list J, number of tiles T
Ensure: Updated channel assignments after permutation
1: Initialize partitions Po. Pi..... Pr derived from W and T
2: for each number of extracted channels j in J do
3: Sampling:
4: for each in do
5: Extract j channels from p to form sample groups
6: end for
7: Clustering:
8: So. .1,...; ST t Belance (smple groups)
9: Cost matrix Cij + CostFunction
10: Assignment:
11:
current.com t- Hungarian(C))
12:
if current.cost previous_cost then
13:
break
14:
else
15:
previous_cost < current.cost
end if
17:
end for
G SUPPLEMENTARY EXPERIMENTS
We evaluate FM-XHINM-GP Deit-Small and Deit-Base under various sparsity levels, and com- pare it with Unstructured OvW, HiNM-V, and HiNM-GP baselines.
Figure G.I presents the results for the Deit-Small and Deit-Base models using vector-size of 128, with the baseline accuracy of 79.80% and 81.80% respectively: Unstructured corre- sponds element-wise pruning: HiNM GP represents the HiNM pruning with Gyro-permutation; FlexHiNM-GP denotes the flexible Hierarchical pruning via region allocation and Gyro- permutation; OVW denotes the traditional out-vector-win pruning, as a baseline since can be re- garded as special case of our framework, representing lower bound of accuracy for FlexHiNM; HiNM-V is variant of our method that omits the GyrroPerrmuation process, which is mainly equiv- alent to Venom, to isolate the effect of channel permutation.
1% (top1)
Deit-Small
Deit-Base
Unstruetured
76.35
74.92
68.84
77.22
76.17
71.18
OVW
62.94
62.36
HINM-V
55
55.06
HINM-GP
45
FlexHiNM-GP
87.5%
90%
95%
75%
80%
87.5%
90%
95%
Sparsity
Sparsity
Figure G.1: Gradual pruning for Deit family (V=128)
As shown in Figure G.I with vector-size of 128, FlexHiNM-GP outperforms HiNM-GP by 1.45% on Deit-Small and by 1.28% on Deit-Base at 90% sparsity,. The advantage becomes more pro- nounced under extreme sparsity: at 95%, FlexHi NM-GP surpasses NM-GP by 2.19% on Deit- Small and by 3.16% Deit-Base. It demonstrates that F FlexHiNM--P is able to maintain higher accuracy under aggressive compression, indicating the effectiveness of its hybrid sparsity design and permutation-avwan optimization.
Moreover, we evaluate FlexHiNM-CP on Bert-Base across three standard NLP benchmarks: QQP, SST-2, and SQuAD. Experiments conducted under two total sparsity levels (75% and 87.5%) under the vector-size of 128, as shown Figure G.2 FlexHiNM-GG consistently delivers compet- itive superior performance compared structured baselines across all settings.
19

---

Published as a conference paper at ICLR 2026
95
90
84.10
89.63
88.28
87.04
86.01
OVW
Unstructured
Jalue
79,92
HINM-V
80
75
HiNM-GP
70
FexxHiNM-GP
Ace
Acc SST-2 SQuAD
Ace QQP Sparsity
Acc
QQP
SST-2 SQuAD
Sparsity
75%
87.5%
Figure G.2: Gradual pruning for Bert-Base (V=128)
Under 75% sparsity, PLexHiNM-GG achieves high task performance while maintaining structured constraints. When increasing vector size to 128, the model remains stable, achieving 88.28 on SQuAD and 89.63% SST-2, again par with HiNM-GP and outperforming OVW across the board. As sparsity increases to 87.5%, performance degradation is observed all methods, yet FlexHiNM-GE GP maintains strong robustness. In the most challenging setting (87.5% sparsity with vector size 128), MeLexiiMM-GN retains competitive performance: 86.01 of F1 score on SQuAD and 87.04% of accuracy outperforming OVW by a large margin. The results highlight FlexH P1exHiNM- GP's ability to preserve task performance under aggressive compression, showcasing the benefits of its permutation-guidee design and differentiable masking in structured sparsity regimes.
SST-2, exceeding Hi NiMM-GP by 0.4 and 0.42 points, and significantly
20


Reviews:
[{"review_id": "3dLzbnoDLs", "reviewer": "Reviewer_Cpf6", "summary": "This paper aims to address the effectiveness degradation caused by wide variance of important scores in N:M sparsity patterns. The authors propose a three-level hierarchical pruning framework that partitions model weights into dense, vector-pruned, and N:M-sparse regions, enabling balanced sparsity and accuracy. A boundary search algorithm is introduced to determine the optimal configuration. This paper also introduces Gyro-Permutation, a channel reordering method designed to enhance information aggregation and improve pruning performance. Combined with iterative pruning, the approach achieves further performance gains. \nTo mitigate the computational overhead introduced by permutation, a custom GPU kernel is developed to execute dense and sparse regions concurrently, ensuring hardware efficiency. The proposed framework is evaluated on both vision and language models, showing strong pruning performance.\n\nContributions\n\n1. A three-level hierarchical pruning framework and a dynamical boundary searching algorithm.\n\n2. The Gyro-Permutation algorithm that improves pruning efficiency through channel reordering.\n\n3. A hardware-aware co-design, integrating a custom GPU kernel to offset the online computational cost of permutation.", "questions": "In Figure 1, why is the N:M range of R for the HiNM-P model bounded at 0.7?\n\nThe Figure 2 and Figure 3 are confusing to me. Which part in figure 2 corresponds to figure 3? There is a sorting logic in figure 3, but no corresponding illustration in figure 2.\n\nThe description of Figure 7 is confusing. It states that HiNM-GP is HiNM with gyro-permutation, but it is unclear which “HiNM” this refers to.\nIf HiNM-V is defined as FlexHiNM without gyro-permutation and is said to be “equivalent to Venom,” then the relationship among HiNM, Venom, HiNM-V, and HiNM-GP becomes ambiguous.\nDoes HiNM here correspond to Venom, or is it a different baseline?\nIf HiNM-GP = HiNM + gyro-permutation, and HiNM-V = Venom, then logically HiNM-GP should be equivalent to HiNM-V + gyro-permutation, which seems to describe FlexHiNM.\nPlease clarify the naming and hierarchical relationship among these variants.\n\nWhat is the motivation behind gyro-permutation’s sampling and clustering?  In the appendix, it states that the number of samples is dynamically chosen without any detail. What is the method for choosing this number of samples hyperparameter? Also, there is no detail about how balanced K-means works.\n\nFor CUDA kernel, it seems when you move the input and weight from global memory to shared memory, you need to do indexing to select the corresponding partition. However, I wonder whether this indexing will cause an uncoalesced problem or not which may degrade the actual running speed? Since you have implemented the CUDA kernel, it would be more convincing if you provide experiment results on speed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Originality.**\nThe paper provides original and meaningful improvements over prior structured pruning methods. It introduces an effective boundary search algorithm to adaptively determine three-level pruning ratios, a more robust channel permutation scheme (Gyro-Permutation) to enhance information concentration, and an efficient kernel implementation that avoids costly online computation.\nBeyond these individual components, the work’s novelty lies in how it integrates algorithmic and hardware aspects into a unified co-design framework, achieving both flexibility and practical deployability.\n\n**Quality.**\nThe technical quality of the paper is strong, particularly in the boundary search algorithm and the GPU kernel design. Both components are clearly formulated, well-motivated, and experimentally validated.\n\n**Clarity.**\nThe motivation and high-level logic are clearly stated, helping readers understand how each module contributes to the framework.\n\n**Significance.**\nThe work addresses a practically important challenge, improving the effectiveness of N:M sparsity under high variance, through a coherent combination of pruning and permutation.\nThe cross-domain applicability to both vision and language models increases its potential impact.", "weaknesses": "**Clarity of figures and presentation.**\nSeveral figures (e.g. figure 6) are difficult to interpret and in some cases make the workflow harder to follow. A few textual descriptions are also inaccurate or ambiguous, which reduces readability and clarity (i.e. lambda in equation 3, what is the definition and what is the value in experiment).\n\n**Insufficient detail on Gyro-Permutation.**\nAs a core component of the proposed framework, the Gyro-Permutation method is described too briefly in the main text and lacks sufficient algorithmic detail even in the appendix. Providing a clearer formulation and rationale would significantly strengthen the technical contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eCqdNWrgPE", "reviewer": "Reviewer_LCAU", "summary": "This paper addresses the lack of flexibility in N:M sparsity by proposing the FlexHiNM framework. The method partitions weights into three regions: dense (4:4), sparse (2:4), and pruned (0:4). To accommodate this partitioning, the authors design the Gyro-Permutation (GP) algorithm to reorder channels. The authors claim their method outperforms the compared structured pruning baselines on DeiT, BERT, and LLaMA-2 in terms of accuracy.", "questions": "1. Why did you not provide any end-to-end inference latency or throughput data? This is the most critical metric for evaluating a hardware-friendly pruning method. Can you provide the actual speedup of FlexHiNM-GP versus HiNM-V at 75% and 87.5% sparsity, measured on an RTX 4090 using the kernel design from Appendix D.1?\n\n2. Can you quantify the total computational overhead of the complete FlexHiNM-GP pruning pipeline (including boundary search, GP reordering, and mask learning)? For example, how many GPU-hours are required to prune the LLaMA-7B model once?\n\n3. Can you add an ablation study comparing only FlexHiNM (without GP) versus FlexHiNM-GP? This is essential to justify the necessity of the complex Gyro-Permutation algorithm.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and practically relevant. For hardware-supported N:M sparsity, the fixed 50% sparsity ratio is indeed a severe limitation in practice. The paper's direction in addressing this is correct.\n\n2. The \"three-region\" partitioning (4:4, 2:4, 0:4) is a logical and incremental improvement over existing HiNM (2:4 and 0:4), providing finer-grained control for layers with different redundancies.\n\n3. The paper conducts extensive experiments on accuracy metrics. The experimental results show that the proposed FlexHiNM-GP method does consistently surpass baselines like OVW and HiNM-V in accuracy on vision and language tasks, and it narrows the gap with unstructured pruning.", "weaknesses": "1. The paper is lack of practical inference latency evaluation. The paper's premise is hardware-friendliness and efficiency, yet the entire manuscript contains no data on end-to-end latency, throughput, or actual wall-clock speedup. The custom kernel proposed in the appendix is merely a theoretical design; its practical overhead from mixed execution and cross-stream synchronization is unknown. Lacking this core speed evidence, the paper's accuracy improvements are unconvincing, as they might be achieved at the cost of sacrificing inference performance.\n\n2. The algorithmic proposal appears overly complex, like a hodgepodge of techniques. Its core innovation is limited to the three-region partitioning, which is a minor incremental improvement. Other parts of the framework, such as Hard Concrete mask learning (Algorithm 3) and gradual pruning, are off-the-shelf techniques in the field. The core Gyro-Permutation (GP) algorithm (Algorithm 2) feels like an over-designed product, seemingly designed as an extremely convoluted process (sampling + K-means + Hungarian matching) just to force the weight distribution to fit the three-region partitioning. This combination of boundary search, iterative GP, and gradual learning leads to extremely high pruning process overhead. The authors do not quantify this cost, making its practical feasibility questionable.\n\n3. The ablation studies fail to sufficiently demonstrate the necessity of its core components. The most critical comparison is missing: a comparison between FlexHiNM (with only three-region partitioning) and FlexHiNM-GP (which adds Gyro-Permutation). We cannot ascertain whether the accuracy gain comes from the (relatively simple) three-region partitioning or from the (extremely complex) GP algorithm. If the improvement from the GP algorithm is negligible, its existence only serves to add unnecessary complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DpvkcQin4a", "reviewer": "Reviewer_mA3Q", "summary": "This paper presented FlexHiNM, a framework to consider a hierarchical N:M sparsity (HiNM) using Gyro-permutation. Difference from HiNM that has 0:4 and 2:4 pruning, the proposed method considers 4:4 pruning that preserves entire vectors with high importance scores. Gyro-permutation based channel reordering optimizes the order of input and output channels so that important weights are well aligned with the n:M constraint. Hard concrete module enables differentiable mask learning under the N:M sparsity constraint. By applying DeiT, BERT, and LLaMA-2, the proposed method shows better performance than other previous methods such as HiNM and OVW (Outer-vector-wise Pruning)", "questions": "- Gyro-permutation for N:M sparsity is proposed in arxiv paper, Toward Efficient Permutation for Hierarchical N:M Sparsity on GPUs (2024). Why did not you cite and compare this paper?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Compared with other methods dealing with N:M sparsity, this paper introduces a hierarchical structure of 0:4, 2:4, 4:4 pruning to provide variable sparsity. This ratio can be automatically adjusted based on the importance of each layer and channel to minimize accuracy loss.\n- Gyro-permutation makes better aligned input and output channels with N:M sparsity. \n- Hard concrete module shows better importance representation and stable convergence.\n- By redesigning the GPU kernel to handle permutation operation, the gyro-permutation can be applied efficiently in real hardware environments.\n- It shows better performance than HiNM when applying to DeiT, BERT, LLaMA2.", "weaknesses": "- The learning scheme in this paper has complex structure so that model training and implementation may be difficult. Due to various hyperparameters and boundary search (vs, ps), the training time should be increased.\n- The additional memory and computational overhead are necessary for the process of permutation and learnable masks.\n- Comparison with other methods considering N:M sparsity is necessary to show the effectiveness of the method, for example, Venom (Castro et al., 2023)\n- There is no comparison of inference time or throughput, FLOPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)