Learning Best Combination for Efficient N:M Sparsity
Authors: Yuxin Zhang
Mingbao Lin
ZhiHang Lin
Yiting Luo
Ke Li
Fei Chao
YONGJIAN WU
Rongrong Ji
Venue: NeurIPS 2022 Accept (2022)
Keywords: Network sparsity
Efficient Inference

Abstract: By forcing N out of M consecutive weights to be non-zero, the recent N:M fine-grained network sparsity has received increasing attention with its two attractive advantages over traditional irregular network sparsity methods: 1) Promising performance at a high sparsity. 2) Significant speedups when performed on NVIDIA A100 GPUs. Current implementation on N:M sparsity requires a tedious pre-training phase or computationally heavy from-scratch training. To circumvent these problems, this paper presents an efficient solution for achieving N:M fine-grained sparsity from scratch. Specifically, we first make a re-formulation to convert the N:M fine-grained sparsity into a combinatorial problem, in which, the object falls into choosing the best weight combination among $C_M^N$ candidates. Then, we equip each combination with a learnable importance score, which can be jointly optimized along with its associated weights. Through rigorous proof, we demonstrate that the magnitude of the optimized score well reflects the importance of its corresponding weights combination to the training loss. Therefore, by gradually removing combinations with smaller scores till the best one is left, N:M fine-grained sparsity can be efficiently optimized during the normal training phase without any extra expenditure. Comprehensive experimental results have demonstrated that our proposed method for learning best combination, dubbed as LBC, consistently increases the efficacy of the off-the-shelf N:M methods across varying networks and datasets. Our project is released at https://github.com/zyxxmu/LBC.


Learning Best Combination for Efficient N:M Sparsity
Yuxin Zhang'
Mingbao Lin
Zhihang Lin'
Yiting Luo'
Ke Li
Fei Chao'
Yongjian Wu² Rongrong Ji ,3,4
'MAC Lab, School of Informatics, Xiamen University, Xiamen, China
Teencett Youtu Lab, Shanghai, China 3 Institute of Artificial Intelligence, Xiamen University, Xiamen China *Pengcheng Lab, Shenzhen, China
Abstract
By forcing most N out of M consecutive weights to be non-zero, the recent N:M network sparsity has received increasing attention for its two attractive advantages: 1) Promising performance at high sparsity. Significant speedups on NVIDIA A100 GPUs. Recent studies require an expensive pre-training phase or heavy dense-gradient computation. In this paper, we show that the N:M learning can be naturally characterized as a combinatorial problem that searches for the best combination candidate within a finite collection. Motivated by this characteristic, we solve N:M sparsity in an efficient devidd-and-conquee manner. First, we divide the weight vector into CN combination subsets of fixed size N. Then, we conquer the combinatorial problem by assigning each combination learnable score that is jointly optimized with its associate weights. We prove that the introduced scoring mechanism can well model the relative importance between combination subsets. And by gradually removing low-scored subsets, N:M fine-grained sparsity can be efficiently optimized during the normal training phase. Comprehensive experiments demonstrate that our learning best combination (LBC) performs consistently better than off -he-h N:M sparsity methods across various networks. Our project is released at http:://gihhub..oo/yyyxxum/
Introduction
Recent years have witnessed the popularity of convolutional neural networks (CNNs) in visual problems such as image classification 34 12 object detection [1] g] and semantic segmentation 9
3. However, the remarkable performance comes the cost of huge computation burdens and memory footprint, resulting in great challenges for deploying CNNs resource-limited devices. Model compression has received extensive attention from both academia and industries. By reducing the number of parameters in CNNs, network sparsity has become one of the most representative techniques to alleviate storage and computation burdens. According
the sparsity granularity. traditional methods can be divided into unstructured sparsity [18] 6 [T] and structured sparsity (20
29 27]. As shown in Fig. [a), unstructured sparsity removes arbitrary weights CNNs. Dozens of
earlier studies [23] 17 have demonstrated its ability to reach negligible performance degradation
under a large compression rate. Unfortunately, unstructured sparsity often results an irregular sparse matrix, which causes heavy index storage and very little acceleration. On the contrary, structured sparsity removes all weights in filter as illustrated in Fig. (b).
result, the sparse weights are still in hardware- frienddy format, leading to notable speedups. Nevertheless, the removal of the whole filters also severely damages the accuracy performance. Thus, future progress expected to find out new sparsity pattern to tradeoff the performance and acceleration.
Cooresponding author: rrji @ xmu.edu.cn
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unstructured sparse weights
Structured sparse weights
N:M sparse weights
Deploy
Deploy
Deploy
Indices
data
None-zero
(a) Unstructured sparsity
Structured sparsity
N:M sparsity
Figure 1: Mainstream sparsity granularity in the literature. (a) Unstructured sparsity removes individual weights at arbitrary locations. (b) Structured sparsity removes entire convolutional filters. (c) N:M sparsity (2:4 case) requires at most N out of M consecutive weights to be non-zero.
More recently, the N:M fine- anee-graied sparsity has shown a promising power for the research community. As depicted in Fig. (c), it divides the weights into several blocks of size M. By preserving at most out of weights in each block, it not only achieves a high sparsity rate but also retains a regular sparse matrix structure for effective acceleration supported by the NVIDIA Ampere Sparse Tensor Core [33 The pioneering ASP (apex's ' automatic sparsity) 30 preserves two langeste weights in every four-element block and then re-trains the sparse network. However, the time- scenarios. To the pre-training burden, Zhou al. [40] learned N:M sparsity from scratch by leveraging the straight-through estimator (STE) M to approximate the gradients of removed weights. Although an additional regularization is proposed alleviate the weight error accumulation, dense gradients are required to be computed in each training iteration to update both the removed and preserved weights. Consequently, a heavy training burden is unavoidable, barricading the generalization of N:M sparsity to gradient and activation [ Thus, how to realize an efficient N:M sparsity remains unexplored.
consuming re-training prohibits the deployment of N:M sparsity in resource-constraiieg
In this paper, propose learn the best combination in order reach efficient N:M sparsity (LBC). Our LBC motivated by the fact that the N:M learning can be modeled as a combinatorial problem which consists in finding, among a finite collection of combinations, one that satisfies all the given conditions. Considering a single block, the purpose of N:M sparsity is to select N non-repeated elements as combination from the M continuous weights. Thus, N:M sparsity can be simplified as finding the best combination candidate from total of CN combinations loss can be minimized on the observed dataset. Given this analysis, our LBC solves the N:M sparsity from the perspective of combinatorial problems in an efficient divide-and conquer manner. We first divide the weight block into a collection of CN combinations with fixed size N. Then, our LBC aims to learn the best combination from this collection. This is achieved by learnable score variable, which, as we prove in Sec.B.4 can well tell the relative importance between different combinations. As results, the best combination can be located by gradually removing the low-scored combinations during normal training process without the necessity a dense-gradient computation.
condition that the training
We perform extensive experiments to demonstrate the efficacy of our LBC. For instance, our LBC achieves 77.2% top-1 accuracy for training 2:4 sparse ResNet-50 on ImageNet, which takes the leading position in comparison with existing N:M sparsity methods [30] 40]. It is also worth oftte-of-theaarr methods that fail to
mentioning that our LBC even surpasses some unstructured achieve speedups. For example, under sparsity rate of around 95%, our LBC still achieves top-1 accuracy of 71.8% with 1:16 sparse pattern, surpassing the recent competitor STR 17 by 1.2%.
2 Related Work
2.1 Network Sparsity
Tradi- tional methods can be roughly categorized into unstructured sparsity and structured sparsity. The former category removes individual weights at any position of the network. Prevailing studies remove
Network sparsity has become an active research topic ever since the last decade 18 10 25

---

unimportant weights using heuristic criteria such as gradient [19] momentum [6), and magnitude [[]] Recent advances implement unstructured sparsity training adaptive manner. RigL 7 alternately removes and revives weights according to their magnitudes and dense gradients. Ding et al. [6] gradually zeroed out the redundant weights by categorizing CNN weights into two parts at each training iteration and updating them using different rules. Despite the progress, the irregular sparse weight tensors rarely support speedups on common hardware unless the sparse rate increases to 95% or even higher [37 33). On the other hand, structured sparsity gains noticeable speedups thanks to its coarse-gras gaarseegrainedddee removal of the entire filter.. Two issues are widely discussed in existing studies including the pruned network structure and the filter importance measurement. Typical works solve the former issue by aule-ff-ne thumb designs [13 5] or laier-wise sparse ratio searching based on evolutionary algorithms 21 [4] As for the second issue, most works resort to devising certain criterion to estimate the filter importance including output activation (39 ranks of feature map (20].
geometric median of filters [14), etc.
2.2
N:M Sparsity
The N:M fine-grained sparsity 30 40 35 advocates N-out-of-M non-zero sparse tensors in the input channel dimension. Supported by the NVIDIA Ampere Core RK], N:M sparsity leads to attractive storage and computation efficiency. For example. N:M sparsity can achieve 2x speedups on the NVIDIA A100 GPU in the 2:4 case, while unstructured sparsity even decreases the inference speed under a similar sparse rate. NVIDIA [0) follows a traditional three-step pipeline to implement N:M sparsity, unfolded as pre-training, pruning and fine-tuning. Pool et al. (32) further leveraged channel permutations to maximize the accuracy of N:M sparse networks. Sun et al. (35] proposed layerwise fine-griined N:M scheme achieve higher accuracy than the uniform N:M sparsity. Recently, there are also efforts in exploring the training efficiency of N:M fine ome-grained sparsity. Hubara et al. (155 devised transposable masks to accelerate the backward phase of N:M sparsity training. Chmiel el al. 2 further extended N:M sparsity to activation, which further boosts the training efficiency. Despite the well-retained performance [30], the expensive pre-training process still hinders the deployment of N:M sparsity. Sparse-refined
estimator (SR-STE) [40] learns the N:M sparsity from scratch. Detailedly, the removed weights are also revived during the backward phase using the straight-througe estimator [ and a specially-designel sparse penalty item. Therefore, by choosing -fout-of-M weights with the largest magnitudes in each forward propagation, the N:M sparsity can be obtained in dynamic and end-to sndett--nd training manner. However, it requires dense gradients in order revive the weights, thus can not be combined with these studies on training efficiency 21]
3 Methodology
3.1 Preliminaries
Denoting the parameters of an L-layer convolutional network as W (W", W ,..., W where W' is K'-dimension F N:M fine-grained sparsity divides parameters in each layer into groups along the input channel dimension, leading to each group containing consecutive weights. It saves computation and retains performance by requiring at most N out of M consecutive parameters to be non-zero values. To this end, the W is firstly divided into G' k'
M groups. For ease of the following
representation, we reorganize W as w' E go XM And then, the optimization of N:M sparsity can be formulated as:
min L(W: D)
| 1. =N,
where / = 1, L and y y= 1,2, G¹. The L()) denotes training loss function and 'D represents the observed dataset. An illustrative case of 2:4 sparsity is given in (c).
Previous N:M methods [30 (0] follow the traditional unstructured sparsity [10] to remove weights in a global manner by looking at the weight magnitude. However, they all require huge training costs
either from pre-training phase 30 or dense dense-gradient calculation [40] to select the most important weights from the large number of parameters in deep networks. Hence, how to achieve N:M sparsity in more efficient manner remains an open issue so far.
Batch norm layers and bias items omitted here for simplicity.
3

---

S2 S3 S4
S5
56
1
W
Bz
MMMI
W
Loss
B3
W
B4
W
forward
backward
Figure 2: The training process for learning the best combination candidate of N:M sparsity (2:4 case).
3.2 Combinatorial Problem in N:M Sparsity
Fortumately, by making full use of the intrinsic particularity in N:M sparsity, the solution space can be well reduced pursue an efficient N:M sparsity. Concretely, given a weight group W g.: the target of N:M sparsity is to select N out of the M weights in W IR. and diseard the remaining M-N ones. 9,1
This leads to combination collection e = {
where each combination @ is a subset of w! and consists of N non-repeated elements from W It is intuitive derive the collection size |0 CN As a toy example, we have C2 considering the 2:4 case. Also, we have:
C
9.4 1= i=1
g.:"
0 (W 9.00
W 9.1 W.ih (1W: 9.0: W (W! 9.0, W
9.3h (W: 9.1, W W. [W 9.11 W 9.3h (Wi 9.2;
W 9.3
2.4
2.0
According to the above analyses, realize that the N:M sparsity be naturally characterized combinatorial problem, which aims to, given finite collection of combinations e search for the best candidate within this collection. Herein, we reformulate the realization of N:M sparsity from the perspective of combinatorial problem as:
arg min L(W)) 0,i D), s.l. 10,a1= e..
where = 1,2, ..., L,, 1,2, G and 1,2. ON
Discussions. Eq. [ implements N:M sparsity by globally sparsifying a weight vector W g.:" In
contrast carlier implementations, our redefined Eq. (3) solves N:M sparsity manner. We first divide the weight vector into CN subsets of a fixed size N. Then, we conquer the combinatorial problem by scoring each subset and finally picking up the highest-scored one, details of which are given in the following contents.
annide-and-oonquer
3.3 Learning Best Combination
Based on the above analyses, we further propose to learn the best combination for each weight vector W g.:" Following [40), consider learning the N:M sparsity from scratch avoid the heavy dependency on an expensive pre-trained model. A learning example of the 2:4 case is shown in Fig 2 Specifically, denoting the input feature map of the 1-th convolution layer Z the output feature map Z is calculated as:
Z' = - W!
where @ denotes the convolutional operation. The earlier study 40 chooses zero out low- magnitude N weights for every weight vector W! g.: in the forward propagation. The straight-through estimator (STE) used to dynamically remove and revive weights, which however leads to error accumulation on the weights. Besides, dense-gradien computation is of heavy cost since both removed and preserved weights are required to update. Given the combinatorial property, we are to the opportunity to reach N:M sparsity by gradually removing the low-scored candidate subsets the combination collection é', during the training procedure until the one with the smallest loss can be found. The gradual sparsity is demonstrated perform better in many traditional network
entitled

---

sparsity 42 41 23 To that effect, we consider the gradual pruning schedule 42 to remove
candidate subsets considered unimportant, and the remaining collection at the t-th training epoch is:
e 9,t = - - 9,
(5)
g,--1
where 6/,, E 0z,t-1 g.t-1 contains unimportant candidates at the t-th training epoch and its size is:
= =
{
0, if = ty. 1)
10 g.al
- 10, otherwise, ty- -1--
where 1 E [4, fy.. 1 and ły denote the beginning and ending of the training epochs in the gradual pruning schedule E2]. When & = (,, we remove total of CM - candidate subsets. In other words, only one combination, which is considered to be the best candidate, preserved in the end.
To measure the candidate's importance, we invent a learnable score matrix S R x In our design, each element Sy, reflects relative importance of candidate 6 in comparison to others in Øt and reduce the size of collection O by removing the low-scored candidates. In the forward propagation, an individual weight W 9.1 often exists in multiple candidate subsets as shown in Eq. 2).
We impose a binary mask B' {0, 1) xM upon the weight W and the output in Eq. becomes:
z' z (B' W
where @ represents the element- -iise multiplication. The zero elements in B' indicate the removal of corresponding weights that contribute least to the network. The binary mask B' is derived as:
By g.4 = I((E(()))) I(W', 9.8 E =.g))) # #0).
(8)
where I(.) returns 1 if its input is true, and 0 otherwise. That is to say, choose to preserve weights within any preserved candidate subset. Note that if W g,s does not exist in any preserved candidate subset, it will be automatically removed since B g.i = 0 according to Eq. (8)
The use of a binary mask originated from many traditional unstructured sparsity methods Z 16] However, our best knowledge, it is the first time to be introduced in N:M sparsity. Moreover, very different from these studies that directly optimize the binary mask, our mask is obtained according to the candidate subsets the corresponding weight falls into. Finally, since the derivative of the I(.)
function is zero almost everywhere, we adopt the STE [
to update St as:
ac
OC
OC
as! 9.
0S 9. 9,i 9.j
AB
O g,i
Consequently, the STE approximation of weight gradient 40 becomes the approximation of the gradient of the introduced score matrix, avoiding the error accumulation on the weights. Intuitively, removing S does not influence the result of Eq. (8) in the forward propagation. However, our motive of retaining S is actually to derive the gradient in the backward propagation, i.e., Eq. ( to learn the importance of each combination. In Sec. 3.4] reflect the relative importance between different candidate subsets. Further, we conduct experiments in Sec. 4.4 show that the learned score S well outperforms existing pruning criteria [10 28
formally prove that the learned score S can well
Our learning the best candidate, dubbed as LBC, is outlined in Alg. Different from traditional methods [3| 26 that conduct network sparsity in two-step manner including mask learning and a weight tuning, LBC enables the optimization of W and S in a joint framework. leading to a well-optimized N:M sparsity during normal training phase. It is worthy to highlight the efficiency of LBC since casts off the dense gradient calculation in (40). Also, LBC can cooperate with techniques that extend N:M sparsity gradient and activation to further boost the training efficiency 2 15).
3.4 Tractability of Optimization
W! ) in 2:4 sparsity of Eq. 2 to show that our introduced mask score We take the candidate (W! 9.0: 9.1 can well reflect the relative importance of each candidate subset. The analysis can be applied to other
5

---

Algorithm 1:
Learning Best Candidate (LBC).
Require: Observed data D; Randomly initialized weights W; Loss function L:; Training epochs
T: Initial and final epoch for removing combinations 4,, Ly; Candidate collection O
Output Trained weights W; Binary mask B.
Sg t VI, Vg. Vi; // Learnable Importance Score
for t E .......T].... do
| 9.1 * then
Calculate (6'u| via Eq. (6:
for E [l..... , L] do
for g E [1, [..... G do
O Top- smallest S in O 9
o 0e g , // Remove Unimportant Candidate Subsets
O
end
10
end
11
Get B via Eq.
8):
12
end
Forward via Eq.
14
Update via the SGD optimizer.
15 end
N:M patterns as well. The gradient of S
9.1
is derived using STE [[]
as:
dc
dc
OB
OC
9.0
9.1
as!
OS!
as
9.1
9.0
9.1
9.1
9,1
OL
UL OB
Wi
9.0
+ W
as g.1 9.1
(10)
ow
as
OW! a.1
9.0
9.1
OL
ac
W
W
9.0 owl
9.1
ow
8.0
9.1
where W
9.0 aw!
meets the criterion
28 that estimates the loss changes after removing Wi
9.0 as:
AC(W 9.0) = L(W! = 0) L(W!
3:0
2.03
aC
= L(W!
W
+ (W 9.0 = 0) -L(WW) 9.0)
5:07
0.0
a W
9.0 = =0)
(11)
O
W
9.0
W
9.0
which leads Eq. 10
0c
-(AC(W)) 0.) AC(W' The updating rule of $ using
as
11
stochastic gradient descent (SGD) becomes:
S = Sh,, + (AC(W) () + AC(WW
(12)
where learning rate and momentum are ignored here for simplicity. The candidate score s 9.1 is the accumulation of loss change for removing weights in the corresponding candidate. A small score denotes less loss increase thus the corresponding candidate can be safely removed. Thus, our introduced score matrix be used measure the relative importance between candidate subsets, and preserving the highest-scored combination well solves the combinatorial problem of Eq. (
Experiments
4.1 Experiment Settings and Implementation Details
Our proposed LBC evaluated on three computer vision tasks including image classification, object detection, and instance segmentation. We implement LBC using the Py Porch [31] upon 2 NVIDIA

---

Table 1: Results of different methods for training the N:M sparse ResNet-18, ResNet-55 and DeiT- small on ImageNet. implies our reproduced results.
Model
Method
N:M
Top-1 Accuracy (%)
Epochs (Train)
FLOPs
Pattern
(Train)
ResNet- 18
Baseline
70,9
120
(x(144c18)
ResNet-18 18
ASP
2:4
69.9
200
1.24x
ResNet-18
SR-STE
2:4
71.2
120
0.83 x
ResNet-11
LBC
2:4
71.5
120
0.70
ResNet-50
Baseline
77.1
120
(x(222118)
ResNet-50
ASP
2:4
76.8
200
1,24x
ResNet-50
SR-STE
2:4
77.0
120
0.83
ResNct-50
LBC
2:4
77.2
120
0.72x
ResNet-50
SR-STE
1:4
75.3
120
x.74
ResNet-50
LBC
1:4
75.9
120
0,51 x
ResNet-50
SR-STE
2:8
76.2
120
0.74x
ResNet-50
LBC
2:8
76.5
120
0.53
ResNet-50
SR-STE
1:16
71.5
120
0.69 x
ResNet-50
LBC
1:16
71.8
120
0.38
DeiT-small
Baseline
79.8
300
(x.99c18)
DeiT-small
SR-STE*
2:4
75.7
300
0.83
DeiT-small
LBC
2:4
78.0
300
0.71
Tesla A 100 GPUs. Following Zhou al. 44). we train the ResNets for 120 epochs with an initial learning rate of O, which is linearly increased to 0.1 during the first 5 epochs and then decayed to O scheduled by the cosine annealing. The SGD is adopted to update parameters with the weight decay and momentum setting 0.0005 and 0.9. Besides, with 300 epochs. To implement LBC, we set l to O and y to 1/2 of the total training epochs. Top-1 classification accuracy, training epochs and training FLOPs reported.
adopt the Timm framework (38) to train DeiT
In Sec. 4.2 compare our LBC with existing N:M sparsity methods including ASP and SR-STE 40 under different N:M patterns. In Sec. 4.3 we further show the performance of LBC and traditional unstructured sparsity methods including Rigl. (7], GMP [42], and STR [17] under similar total compression rates. We give performance analyses of LBC w.r.f. its components in Sec. 4.4
4.2 Comparison with N:M sparsity methods
Image classification. We first apply our LBC to train N:M sparse ResNet 12 and DeiT 36 on ImageNet [4. Tab. shows that the proposed LBC takes the lead in all N:M patterns with even less training burden for sparsifying ResNet with depth of 18 and 50. For ResNet-18, LBC outperforms SR-STE 40 at 2:4 pattern at the Top-1 classification accuracy by 0.3% (71.2% for SR-STE and 71.5% for LBC). For ResNet-50, LBC again demonstrates its superiority against both ASP 30 and SR-STE for all sparse patterns. Specifically, ASP achieves Top-1 accuracy of 76.8% for training
2:4 sparse ResNet-50 using 200 training epochs that contain pre-training and fine-tuning phase, while LBC obtains a significantly higher accuracy of 77. 7.2%% with fewer epoch of 120. When comes to 1:4 sparse case, IBC surpasses SR-STE 440 by a large margin (75, 75.9% 75.3% in Top-1 accuracy and 0.49 x v.s. 0.74x in training FLOPs). Although SR-STE also alleviates the pre-training process, the dense backward propagation greatly decreases its training efficiency in comparison with LBC. From Fig (a), LBC still advantagcous in all aspects when compared with SR-STE in the 1:4 case.
We further conduct experiments on the recent advanced Vision Transformer (ViT) model DeiT 36 to validate the efficacy of LBC. Similar to the strategy for compressing CNNs, we divide all parameters including the multi-head attention and feed-forwart module in Deil-small and train the N:M sparse networks from scratch. Tab. T shows
results of LBC and our re-produced SR-STE for training 2:4 sparse DeiT' on ImageNet. As can be seen, LBC again surpasses SR-STE by noticeable margin of 2.3% (78.0% for LBC and 75.7% for SR-STE) using fewer training FL.OPs (0.71 x for LBC and 0.83 x for SR-STE) Therefore, the ability of LBC is well validated for compressing ViT models.

---

77
LBC 5A.5TE
Top 1 Accuracy
76
Top-1 Accuracy
75
73-
234
72-
ASP
SR
1:16
2:4
033
0/4
N:M Pattern
FLOPs(Train)
(a)
(b)
Figure 3: (a) Comparison results under different N:M pattern for sparsifying ResNet-50. (b) Top-1
accuracy v.s. training FLOPPs for SR-STE and our proposed LBC at 1:4 case.
Table 2: Results on object detection.
Table 3: Results on instance segmentation.
Model
Method
N:M Pattern
mAP.
Model
Method
N:M Pattern
Box mAP
Mask mAP
F-RCNN Baseline
37.4
M-RCNN
Baseline
38.2
34.7
F-RNNN SR-STE
2:4
38.2
M-RCNN
SR-STE
2:4
39.0
35.3
F-RCNN LBC
2:4
38.5
M-RCNN
LBC
2:4
39.3
35.4
F-RCNN
SR-STE
2:8
37.2
M-RCNN M-RCNN
SR-STE
2:8
37.6
33,9
F-RCNN
LBC
2:8
37.3
LBC
2:8
37.8
34.0
Object detection and instance segmentation. Next, exploit the efficacy of LBC object detection and instance segmentation. Tab. and Tab. respectivel display the comparison results of LBC against SR-STE (40] for training 2:4 sparse Faster-RCNN [8] and Mask-RCNN [I]] with the backbone of ResNet-50 on the COCO benchmark [22]. The results further demonstrate the robustness and superiority of LBC on other computer vision tasks.
4.3 Comparison with unstructured sparsity methods
We also compare the performance of LBC with state-of-the-art unstructured sparsity methods in Tab. By training 1:16 sparse ResNet-50 ImageNet, LBC achieves Top-1 accuracy of 71.8%, with 1.4% and 1.2% improvements over STR and GMP, respectively. Furthermore, the compressed models generated by unstructured sparsity remain irregular format, which brings great challenges for the apalaappliottion deployment. Nevertheless, under a similar sparse rate, our LBC achieves practical compression and acceleration thanks to the NVIDIA Ampere Core while still bringing higher accuracy performance, which validates the advantages of exploring N:M sparsity.
Table Results of the N:M and unstructured sparsity methods for sparsifying ResNet-50. Method
Top-1
Sparsity
Params (Test)
FLOPs
Structured
Accuracy (%)
(%)
(Test)
Baseline
77.3
0.0
25.6M
4.10G
DNW
68.3
95
1.28M
0.20G
X
RigL
70.0
1.28M
0.49G
GMP
70.6
95
1.28M
0.20G
X
STR
70.4
1.27M
0.16G
SR-STE
71.5
94 (1:16)
3.52M
0.44G
LBC
71.8
94 (1:16)
3.52M
0.44G
/
4.4 Performance analyses
Training efficiency. In this part, we investigate the property of LBC for improving the N:M sparse training efficiency. All the experimental results are conducted on ImageNet by training 1:4 sparse ResNet-50. We first point out that the total training FLOOPs of LBC are proportional to the initial epoch 1, and final epoch ly. To explain, larger I; and l indicate more smooth curve for removing combinations and more weights be sufficiently trained before the epoch reaches ły. For fair

---

Table 5: Experimental results for SR-STE and LBC the same gradual sparsity schedule.
Table 6: Experimental results for different cri- teria train N:M sparse ResNet-50.
Method ti
ts
Top-1
FLOPs (Train)
Method
N:M
Top-1 Accuracy (%)
Accuracy (%)
Pattern
SR-STE LBC
0
75.3
0.74x
Baseline
77.3
0
75.2
0.48x
Magnitude
2:4
75.8
SR-STE
30
75.5
0.76x
Gradient
2:4
76.
LBC
30
75.4
0.49
S-inverse
2:4
72.9
S SLBCC)
2:4
77.2
SR-STE
60
75.3
0.78x
LBC
60
75.9
0.51
SR-STE
60
75.6
0.87
LBC
60
76.0
0.54x
comparison, also apply the same gradual sparsity schedule SR-STE 40) Tab. more training FLOPs consistently improve the performance of LBC as the importance score be jointly trained with weights more sufficiently. Fig. 3)) further shows that our LBC takes the lead in the trade-off between training cost and Top-1 accuracy performance when compared with SR-STE [30 40]. It is worthwhile to note that, by selecting different t; and ty. LBC can provide satisfying solution between training expenditure and performance under different resource constraints. Thus, the efficiency and applicability of LBC are significant.
As illustrated in
Combination Score. To investigate our combination score S, we introduce three other criteria including 1) inversely sorted learned score S, 2) the magnitude-base based criterion [IO], and 3) the gradient-based criterion [28] under the same experiment settings for sparsifying ResNet-50 on ImageNet. Results in Tab. show that the proposed learnable score significantly outranks other criteria. Thus, the efficacy of LBC for recommending best weight combinations is well demonstrated.
5
Limitation
Firstly, LBC still needs relatively dense computation in the early stage of N:M training as there still plenty of left candidate subsets. Thus, despite the advantages pre-trained models or dense-grand dense-gradient calculation over the entire training phase, there are still room for improving the efficiency of LBC. Secondly, this paper only explores N:M sparsity on computer vision tasks. We expect to show more results on other tasks such as natural language processing (NLP) in future work. Finally, we mainly focus N:M sparsity on weights, while adding N:M sparsity for gradient [15 and activation in the workflow of LBC remains to be excavated in the near future.
approaches that rely
6 Conclusion
N:M fine-grained sparsity is an important technique that allows fast inference and training on the next-generation hardware and platforms. In this paper, we have presented LBC towards efficient N:M sparsity from scratch. We first transfer the N:M learning into a combinatorial problem, which motivates to select the best weights combination from C.N candidates. Further, we propose to assign Icarnable importance scores for each combination, which is jointly optimized with the associate weights. We perform an in-depth analysis that the best combination can be identified by aggressively removing less important combinations. Extensive experiments have demonstrated the efficacy of LBC in reducing the N:M training cost, and its superiority over several SOTAs. Our LBC characterized by end-to-end implementation, efficient training, the tractability of optimization, and state-of-the-art performance in sparsifying modern networks.
Acknowledgement
This work supported by the National Science Fund for Distinguished Young (No.62025603), the National Natural Science Foundation of China (No.62025603, No. U1705262, No. 62072386, No. 62072387, No. 62072389, No, 62002305, 61.6477, No. 61802324 and No. 61702136) and
Guangdong Basic and Applied Basic Research Foundation 2No.2019BI0 1511915111120499

---

References
[1] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1rXivv1308.344 2013.
[2] Brian Chmiel, Itay Hubara, Ron Banner, and Daniel Soudry. Optimal fine-grained n: M sparsity for activations and neural gradients. arXiv preprint arXiv:2203. 10991, 2022.
[3] Ioana Croitoru, Simion-VIas Bogolin, and Marius Leordeanu. Unsupervised learning of foreground object segmentation. International Journal of Computer Vision (IJCV), 127(9):1279- 1302, 2019.
[4] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248 248-255, 2009.
[5] Xiaohan Ding, Guiguang Ding, Yuchen Guo, Jungong Han, and Chenggang Yan. Approximated oracle filter pruning for destructive enn width optimization. In International Conference on Machine Learning (ICML). pages 1607-1616. PMLR, 2019.
[6] Xiaohan Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, Ji Liu, al. Global sparse momentum sgd for pruning very deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 3382-6394, 2019.
[7] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICML), pages 2943-24222222 2952, 2020.
[8] Ross Girshick. Fast r-cnn. In IEEE International Conference on Computer Vision (ICCV), pages 1440-1448, 2015.
(9] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In IEEE International Conference on Computer Vision (ICCV), pages 580-587, 2014.
[10] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NeurIPS), pages 11 1135-1143, 2015.
[11] Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cnn. In JEEE International Conference Computer Vision (ICCV), 2017.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[13] Yang He, Guoliang Kang. Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks. International Joint Conference on Artificial Intelligence (IJCAI), pages 2234-2240, 2018.
[14] Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4340 4349-4349,0000 2019.
[15] Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry, Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[16] Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances Neural Information Processing Systems (NeurIPS), pages 20744-20754, 2020.
[17] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Learning (ICML), pages 5544-5555, 2020.
10

---

[18] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In Advances in Neural Information Processing Systems (NeurIPS), pages 598 0888605.50 1989.
[19] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr Snip: Single-shot network pruning based connection sensitivity. In International Conference on Learning Representations (ICLR), 2019.
[20] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1529-1538, 2020.
[21] Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian. Channel pruning via automatic structure search. In International Joint Conference on Artificial Intelligence (IJCAI), pages 673-679, 2020.
[22] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), pages 740-755, 2014.
[23] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[24] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Tim Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In IEEE International Conference on Computer Vision (ICCV), pages 3296-3305, 2019.
[25] Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through |_0 regularization. In International Conference on Learning Representations (ICLR), 2017.
[26] Jian-Hao Luo and Jianxin Wu. Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference. Pattern Recognition (PR), page 107461, 2020.
[27] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In IEEE International Conference on Computer Vision (ICCV), pages
2017.
[28] Pavlo Molchanov, Stephen Tyree. Tero Karras, Timo Aila, and Jan Kautz, Pruning convolutional neural networks for resource efficient inference. In International Conference Learning Representations (ICLR), 2017.
[29] Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang. Dsa: More efficient budgeted pruning via differentiable sparsity allocation. European Conference on Computer Vision (ECCV), pages 592-607. Springer, 2020.
[30] Nvidia. Nvidia a100 tensor core gpu architecture.
2020.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), pages
2019.
[32] Jeff Pool and Chong Yu. Channel permutations for n: M sparsity. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[33] Olivier Giroux et al. Ronny Krashinsky. Nvidia ampere sparse tensor core. http::/devvloppr wwwdla..ombblgg//idaaaaaaaeeeeee ampaaa-ppaaaaaa www.aaaaeeeeeeeeeeeaeeeeee hwtpreeeecccccetettr----dddpppti 2020.
[34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
11

---

[35] Wei Sun, Aojun Zhou, Sander Stuij,, Rob Wijnhoven, Andrew O Nelson, Henk Corporaal, et al. Dominosearch: Find layer-wise fine-grained n: M sparse schemes from dense neural networks. Advances in Neural Information Processing Systems (NeurlPS), 34, 2021.
[36] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training cata-efficient image transformers & distillation through attention. In International Conference Machine Learning (ICML), pages 10347-10357, 2021.
[37] Ziheng Wang. Sparsert: Accelerating unstructured sparsity on gpus for deep learning inference. In Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques (ICPACT), pages 31 -42
(38) Ross Wightman. Pytorch image models. http://ghhh...ocooooomammmmmmooa 2019.
[39] Yuxin Zhang, Mingbao Lin, Chia-Wer Wen Lin, Jie Chen, Yongjian Wu. Yonghong Tian, and Rongrong Ji. Carrying out cnn channel pruning in white box. IEEE Transactions on Neural Networks and Learning Systems (TNNLSS), 2022.
[40] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: M fine -ere-grinndd structured sparse neural networks from scratch. In International Conference on Learning Representations (ICLR), 2021.
[41] Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-pre www-peecision weights. In International Conference on Learning Representations (ICLR), 2017.
[42] Michael Zhu and Suyog Gupta. To prune, or not prune: exploring the efficacy of pruning for model compression. In International Conference on Learning Representations Workshop (ICLRW), 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
12

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "w4wOswr4Bxx", "reviewer": "Reviewer_biEJ", "summary": "This paper deals with the problem of N:M sparsity and proposes an effective method with structure parameter to weigh the importance of different combination. The final result is appealing. ", "questions": "The problem of training cost saving should be corrected.", "limitations": "The problem of training cost saving should be corrected.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Good final pruning results\n\n\nWeakness:\n1. From the analysis of [1] Efficient Neural Network Training via Forward and Backward Propagation Sparsification, the gradient-based sparse training method cannot exceed the accelearation above a upper bound of around 1.5x since the gradient to structure parameters require dense gradient calculation, even though the gradient to weights is sparsified and the forward propagation is sparsified. I think this is a fundamental problem and unless the discussions around training costs are corrected, it should not be published. Otherwise it will again mislead the community. If this issue can be corrected, I am happy to raise the rating score to be positive. \n\nAfter rebuttal: I got the difference. The gradient calculation to auxiliary parameters only finally cares about the gradient to weights. I raise the score.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "l5U-UHBcYey", "reviewer": "Reviewer_VQRc", "summary": "This paper formulates the N:M sparsity as the combinatorial optimization problem. \nThis formulation is a well-motivated and promising direction. Then the authors use a learnable score matrix to measure \nthe candidate's importance. The authors use the gradients of group elements to update the score matrix.", "questions": "See Strengths And Weaknesses\n\n\n###############################################\n\nPost-rebuttal: I have read the rebuttal carefully, and all of my comments/questions were addressed. Thanks.", "limitations": "Na", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n\nThis paper is well-written and easy to follow.\n\nThe performance gain is consistent on various sparse ratios and models.\n\nAt present, reducing the sparse model training cost can motivate the research community to study sparse training acceleration.\n\nThe source code is available.\n\nWeaknesses:\nThe authors are encouraged to conduct experiments on other tasks.\n\nThe authors only report the training  FLOPs, the authors are encouraged to report the training time.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aXWq5W3qXP_", "reviewer": "Reviewer_378D", "summary": "Authors tackle the problem of a learnable structural sparsity parametrized by N:M pattern. The best sparsity patterns per layer are found via divide-and conquer like algorithm where from a large set of proposals only one is selected in the end. At first, all possible candidate combinations are proposed, then, during training they are ranked according to the importance variable. At every algorithm step, the combination with the lowest score is removed from the pool. Finally, the last standing combination is selected. Experiments are performed on Resnet50 and DIET model on Imagenet.  \n", "questions": "Formulation of the implementation is a bit confusing. Does it mean that during training all combinations contribute to the output of the layer? If true it will mean higher memory cost during training and higher computational complexity. \nFrom another perspective, if S is implemented via straight through estimator than it is sampled during the training and the gradient propagates only to the elements with S = 1. Please clarify this point. \nScore matrix. Does it mean that the score is a real value learned by the model? Is there any evidence that it does reflect importance? Authors provide derivation to end up with magnitude times gradient equation. However, providing some empirical evidence will be beneficial. \nCan authors explain how to set newly introduced hyper-parameters and what is the intuition behind the choice? \nWas the training recipe exactly the same when compared to SR-STE? ", "limitations": "At my current understanding, the cost of initial epochs is going to be higher due the requirement of forward0backward to be performed over all candidates. It will be nice to have some latency study as authors primary focused on efficiency of N:M sparsity. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Paper provide a comprehensive overview of existing pruning paradigms (structured and unstructured). Ampere sparsity N:M is also clearly introduced. \nThe problem of structural pruning is usually not formulated a combinatorial problem. The fact that authors do makes the paper stronger and the approach sound. Particularly, formulation in section 3.2 is sufficient. The implementation of learning score matrix S is a bit unclear and answering questions from the section below will help to understand the paper better. \nThe code is available to reviewers, I went over it but didn't run. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "GBInIo0p3p9", "reviewer": "Reviewer_11EU", "summary": "This work is focused on efficient learning of N:M sparsity from scratch. Finding the optimal N:M sparsity pattern is characterized as jointly solving a series of combinatorial problems with finite collections of candidates. The authors proposed to associate a learnable score parameter with each possible combination and learn it from data with the help of straight-through-estimator (STE). The experiments showed the proposed training method yields slightly better results with much fewer FLOPs compared to stronger baselines and much better performance than others.", "questions": "Please see the weaknesses part and address my questions there.\n\nThe superscript of $\\mathcal{L}^l$ in the last term of eq. (9) seems to be in the wrong place.", "limitations": "Please see the weaknesses part and address my questions there.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ This work successfully showed that N:M sparsity under a kind of gradual pruning manner can work well on large modern neural networks. The gradual pruning enables the possibility of a efficient method that trains N:M sparse models from scratch so that the expensive dense training process can be avoided.\n\n+ Experiments are promising in terms of performance and FLOPs.\n\n--------------------------------\n\nWeaknesses:\n\n- The biggest issue of this work if over-claiming. Characterizing the problem as jointly solving a series of combinatorial problems is trivial, and more importantly, does not simply the original problem in any sense.  The authors claimed that they adopted a \"divide-and-conquer\" method. But according to my understanding of \"divide-and-conquer\", this claim is not true. The problem is not simplified into simpler forms with lower complexity. It is still a complex joint optimization problem. And the authors uses a simple method that learns a score for each combination. This type of methods is not new and can date back to network slimming (Liu et al., 2017).\n- The only place that the learnable scores are used is eq. (8). But if I understand correctly, removing the S score in eq. (8) does not influence the whole methods at all. If that is the case, I am quite doubtful of what scores are learnt finally.\n- For a paper that emphasizes its training efficiency, there is no energy or wall-time comparison of the proposed method and other baselines. This is related to another of my question that --- whether Nvidia A series GPU can exploit the gradually increasing N:M sparsity to improve training efficiency?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept