Learning N:M  Fine-grained Structured Sparse Neural Networks From Scratch
Authors: Aojun Zhou
Yukun Ma
Junnan Zhu
Jianbo Liu
Zhijie Zhang
Kun Yuan
Wenxiu Sun
Hongsheng Li
Venue: ICLR 2021 Poster (2021)
Keywords: sparsity
efficient training and inference.

Abstract: Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. It can be generally categorized into unstructured fine-grained sparsity that zeroes out multiple individual weights distributed across the neural network, and structured coarse-grained sparsity which prunes blocks of sub-networks of a neural network. Fine-grained sparsity can achieve a high compression ratio but is not hardware friendly and hence receives limited speed gains. On the other hand, coarse-grained sparsity cannot simultaneously achieve both apparent acceleration on modern GPUs and
decent performance. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network, which can maintain the advantages of both unstructured fine-grained sparsity and structured coarse-grained sparsity simultaneously on specifically designed GPUs. Specifically, a 2 : 4 sparse network could achieve 2× speed-up without performance drop on Nvidia A100 GPUs. Furthermore, we propose a novel and effective ingredient, sparse-refined straight-through estimator (SR-STE), to alleviate the negative influence of the approximated gradients computed by vanilla STE during optimization. We also define a metric, Sparse Architecture Divergence (SAD), to measure the sparse network’s topology change during the training process. Finally, We justify SR-STE’s advantages with SAD and demonstrate the effectiveness of SR-STE by performing
comprehensive experiments on various tasks. Anonymous code and model will be at available at https://github.com/anonymous-NM-sparsity/NM-sparsity.

Published as a conference paper at ICLR 2021
LEARNING N:M FINE-GRAINED STRUCTURED SPARSE NEURAL NETWORKS FROM SCRATCH
Aojun Zhou Yukun Ma Junnan Jianoo
Jianbo Liu?, Zhijie Zhang', Kun Yuan'
Wenxiu Sun' Hongsheng Li
SSenseTime Research, 'CUHK-Sensetime Joint Lab, CUHK, NNortwwestern University, *NLPR, CASIA ao junzheu@gmail com gmayukun@gmail.. com hsli@ee. cuhk. edu.hh .h
ABSTRACT
Sparsity in Deep Neural Networks (DNNs) has been widely studied compress and accelerate the models on poourcc--oostrainee environments. can be gen- erally categorized into unstructured fine-graines sparsity that zeroes out multiple individual weights distributed across the neural network, and structured coarse- grained sparsity which prunes blocks of sub-networks of a neural network. Fine- grained sparsity can achieve a high compression ratio but is not hardware friendly and hence receives limited speed gains. On the other hand, coarse-grained sparsity cannot concurrently achieve both apparent acceleration on modern GPUs and de- cent performance. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network, which can maintain the advantages of both unstructured fine-grained sparsity and structured coarse- grainee sparsity si- multancously on specifically designed GPUs. Specifically, a 2 4 sparse network could achieve 2 x speed-up without performance drop on Nvidia A100 GPUs. Furthermore, we propose a novel and effective ingredient, sparse-refined straight- through estimator (SR-STE), to alleviate the negative influence of the approxi- mated gradients computed by vanilla STE during optimization. We also define metric, Sparse Architecture Divergence (SAD), to measure the sparse network's s topology change during the training process. Finally, We justify SR-STE's ad- vantages with SAD and demonstrate the effectiveness of SR-STE by performing comprehensive experiments on various tasks. Source codes and models are avail- able at
INTRODUCTION
Deep neural networks (DNNs) have shown promising performances various tasks including com- puter vision, natural language processing, speech recognition, etc. However, a DNN usually comes with large number of learnable parameters, ranging from millions of to even billions of (e.g., GPT-3 (Brown et al.] (2200), making the DNN model burdensome and difficult to be applied to real-world deployments. Therefore, researchers began investigate how speed up and compress DNNs via various methods such as knowledge distillation (Hinton et al.) 2015), quantization (Jacob)
al. 2018 Zhou et al. 2017), designing efficient model architectures (Howard al. 2017). and
structured sparsity (Wen et al. 2016 Li et al. 2016).
In this paper, we focus on the problem of sparsifying DNNs. Sparsity in DNNs can be categorized into unstructured sparsity and structured sparsity. Unstructured sparsity prunes individual weights any location, which is fine-grained and can achieve extremely high compression ratio (Han al., 2015 Guo et al. 2016). However, unstructured sparsity struggles to take advantage of vector- processing architectures, which increases latency due to dependent sequences of reads (Nvidia) 2020). Compared with unstructured sparsity, structured sparsity is more friendly to hardware, espe- cially for block pruning (Wang et al., 2019). kernel shape sparsity (Tan et al., 2020) or channel and filter pruning (Li et al. 2016 Wen et al. 2016). Although structured sparsity can speed up DNNs on commodity hardware, hurts model performance more significantly than unstructured fine- ine-grained sparsity. For example, Re network generated by unstructured pruning can achieve 5.96x compression ratio, with the same accuracy as the original network, but it can only achieve x com-
"The first authors equally contribute this paper.

---

Published as a conference paper at ICLR 2021
pression in the case of structured sparsity (Renda et al. 2020). Therefore, how to combine the unstructured sparsity and structured sparsity to accelerate DNNs on modern hardware (e.g., GPU) becomes challenging yet valuable problem. Recently, Nvidia Ampere A100 is equipped with the Sparse Tensor Cores to accelerate 2:4 structured fine-grained sparsity. Here, N:M sparsity indicates the sparsity of DNNs in which only N weights are non-zero for every continuous M weights. To the best of our knowledge, A A100 is the first commodity sparse hardware, where the sparse tensor core can support several common operation operations including linear, convolutional, recurrent cells, transformer blocks, etc. Specifically, suppose typical matrix multiplication X' x W in DNNs, X' and W denote input tensor and parameter tensor respectively. The Dense Tensor Cores implement X16x32 x W32x8 matrix multiplication by 2 cycles while the Sparse Tensor Cores only need cycle if the parameter tensor W satisfies the 2:4 structured sparse pattern.
Nvidia has proposed an ASH (APEX's Automatic Sparsity) solution (Nvidia 2020) to sparsify a dense neural network to satisfy the 2:4 fine-grained structured sparsity requirement. The recipe contains three steps: (I) training dense network until converge; (2) pruning for 2:4 sparsity with magnitude-bant single-shot pruning; (3) repeating the original training procedure. However, ASP is computationally expensive since it requires training the full dense models from scratch and fine- tuning again. Therefore, we still lack a simple recipe to obtain a structured sparse DNN model consistent with the dense network without extra fine-tuning-
This paper addresses this question: Can we design a simple yer universal recipe to learn N:M sparse neural networks from scratch in an efficient way?
is difficult to find the optimal sparse architecture (connections) and optimal parameters Evvii al. 2019b) simultaneously during training sparse CNNs and Transformers although SET-MLP could easily outperform dense MLP (Bourgin et al. 2019). There are two schemes to obtain such sparse models. One is a two-stage scheme, which discovers a sparse neural architecture by pruning a well-trained dense network and then uses the same or even greater computational resources to retrain
the sparse models (Nvidia 2020 Evci al.) 20196 Han et al., 2013 Frankle & Carbin| 2018). The other is one-stage scheme, which adopts the dynamic method to alternatively optimize parameters and prunes network architectures based on different criteria (Bellec et al. al.1 |
|Moaanu et al.) 2018)]
Mostafa & Wang 2019: Evci et al. [2019b] Kusupati et al. [020)] Dettmers & Zettlemoyer| 2019). Compared with the two-stage scheme, the one-stage scheme can save training time and cost however usually obtains lower performance.
To overcome the aforemennioned trade-off between training cost and performance, we present a simple yet effective framework to train sparse neural networks from scratch. Specifically, em- play the magnitude-based pruning method (Renda et al. 2020 Gale et al., 2019) during the forward process. Considering that the pruning operation is -in-differentiable operator (a similar dilemma in model quantization (Courbariaux et al.) 2016)), we extend the widely used Straight-through Es- timator (STE) (Bengio et al.] 2013) in model quantization to aid sparse neural network's back- propagation. However, perturbations are introduced during the back-propagation (Yin et al., 2019) Bengio et al.] 2013). Hence we define Sparse Architecture Divergence (SAD) to further analyze N:M sparse networks trained by STE methods so that we can identify the impact of perturbations on sparse neural networks training. Based on SAD analysis, to alleviate the negative impact, we propose sparse-refined term mitigating the approximated gradients' influence.
We also compare the performance of neural networks with different granularities of fine- -ne-grained structured sparsity (i.e., 1:4, 2:4, 2:8, 4:8) and conduct thorough experiments on several typical deep neural networks with different N:M sparsity levels, covering image classification, detection, segmentation, optical flow estimation, and machine translation. Experimental results have shown that the models with our proposed structured sparsity can achieve neglectful performance drop and can even sometimes outperform the dense model.
The main contributions of this paper are summarized as three-fold. (1) To the best of our knowledge, this is the first systematic study into training N:M structured sparse neural networks from scratch without performance drop. The N:M structured sparsity is a missing yet promising ingredient in model acceleration, which can be a valuable supplement with various compression methods. (2) We extend STE to tackle the problem of training N:M sparse neural networks. To alleviate the limita- tions of STE on sparsifying networks, we propose a sparse refined term to enhance the effectiveness
2

---

Published as a conference paper at ICLR 2021
on training the sparse neural networks from scratch. (3) We conduct extensive experiments on vari- ous tasks with N:M fine-graing aeeegraandd sparse nets, and provide benchmarks for N:M sparse net training to facilitate co-development of related software and hardware design.
RELATED WORK
Unstructured and Structured Sparsity. Sparsity of DNNs is a promising direction to compress
and accelerate deep learning model. Among all sparsity types, unstructured sparsity can achieve a significantly high compression ratios (e.g. 13x (Han et al. 2013) and 108x (Guo et al. (0166) while ensuring decent accuracy by pruning. Many different pruning criterions and pruning meth- ods are proposed for unstructured sparsity, e.g., magnitude-
pruning (Han et al., 2013 Frankle & Carbin 2018). Hessian based heuristics (LeCun et al., 1990), and pruning with connection sen- sitivity (Lee et al. 2018). However, unstructured sparsity's 's ability to accelerate is highly limited since it takes lot of overhead to store the irregular non-zero index matrix. On the other hand, [Wen et al. (2016) introduces the structural sparsity to speed up deep models on GPUs. Existing structural sparsity contains filter-wise sparsity (Li et al. 2016), channel-wise sparsity (Li al. 2016), filter- shape-wise sparsity. Different from existing sparsity patterns (fine-gn fine-grained unstructured sparsity and coarse-graing structured sparsity), this paper presents an N:M fine- fine-graine grained structured sparsity, a sparsity type that has both high efficiency and lossless performance.
One-stage and two-stage methods. There are mainly two types of techniques to obtain a sparse neural network, one-stage methods and two-stage ones. The two-stage method first prunes a trained dense neural network and then retrains fixed sparse network to recover its performance. Typical two-stage methods include single-shot pruning (Lee al., 2018) and iterative pruning (Han al., 2013] Guo et al. (016). Later, the lottery ticket hypothesis (Frankle& & Carbin] 2018) shows that the sparse sub-network (winning tickets) can be trained from scratch with the same initialization while the winning tickets are discovered by dense training. Deep-Rewiring (Bellec et al.) (017)), on the other hand, is a typical one-stage method, which takes a Bayesian perspective and samples sparse network connections from a posterior, however is computationally expensive and challenging to be applicd to large-scale tasks. Sparse Evolutionary Training (Mocanu et al. (SE8) is proposed as a simpler scheme where weights are pruned according to the standard magnitude criterion used in pruning and growing connections in random locations. Dettmers & Zettlemoyer (2019) uses the momentum of each parameter the criterion for growing weights and receives improvement in test accuracy. GMP (Gale et al. 2019) trains the unstructured sparse net using variational dropout and lo regularization from scratch, and shows that unstructured sparse architectures learned through pruning cannot be trained from scratch to have the same testing performance as dense models do. Recently proposed state-of -fee------he-ttttt method STR (Kusupati et al. 2020) introduces pruning learnable thresholds to obtain a non-uniform sparse network. RigL (Evci et al., 2019a) uses the magnitude- based method to prune and the periodic dense gradients to regrow connection. However, compared with training dense neural networks from scratch, to achieve the same performance, Rigl. needs 5x more training time.The most closely related work to ours may be DNW ONrtWott et al. 2019) which uses a fully dense gradient in the backward run to discover optimal wiring on the fly.
METHOD
3.1
N:M FINE-GRAINED STRUCTURED SPARSITY
Here we define the problem of training a neural network with N:M fine-grained structured sparsity. A neural network with N:M sparsity satisfies that, in each group of M consecutive weights of the network, there are at most N weights have non-zero values. Fig. Iillustrates a 2:4 sparse network.
Generally, our objective is to train an N:M sparse neural network as
min L(W:D), M(W,N.M
(1)
where D denotes the observed data, L represents the loss function, W = (W" : 0 < 1 S L} indicates the parameters of an L--layer neural network, and S(W, N,M) is the N:M sparse neural network parameters.
3.2 STRAIGHT-THROUGE ESTIMATOR ON TRAINING N:M SPARSE NETWORKS
A straightforward solution for training an N:M sparsity network is to simply extend Straight- through Estimator (STE) (Bengio et al. 2013) perform online magnitude- pruning and
3

---

Published as a conference paper at ICLR 2021
Spanse weightt
Dense weights
Grosping
Deplny
R
Pruning
R
.V
Q
O
0
Q
Non 2000 dasa values
log; M hos indiccss
Sparse subnetworkk madi
Figure 1: Illustration of achieving N:M structure sparsity. (Left) In a weight matrix of 2:4 sparse neural network, whose shape R C (e.g., R output channels and C input channels linear layer), at least two entries would be zero in each group of 4 consecutive weights. (Middle & Right) The process that the original matrix is compressed, which enables processing of the matrix to be further accelerated by designated processing units (e.g., Nvidia A100).
sparse parameter updating, which is depicted in Fig. 2(a) STE is widely used in model quanti- zation (Rastegari al. 2016), since the quantized function is non-differentiable without STE and the networks optimized with STE has decent performance under careful settings (Yin et al.) 2019). In STE, dense network maintained during the training process. During the forward pass, we project the dense weights W into sparse weights W = S(W, N, M) satisfying N:M sparsity. Let w C W be group of consecutive M parameters in W and w C W be the corresponding group in W. The projection of w can be formulated as:
W: if W. 3 for i .,2,...,AA ....M
O if W. <E
where & is the N-th largest value in {|wil, /w/...., / Intuitively, this projection func- tion S(-) produces sparse parameters W by setting N parameters that have the least significant absolute values to zero in each consecutive M-parameter group, while keeping the other parameters the same as before. The computation of an N:M sparse sub-network on-the-fly fly in the forward pass is illustrated in Fig.
The projection function S(-), which non-differentiable during back-propagation, generates the N:M sparse sub-network on the fly. To get gradients during pockppropagation, STE computes the gradients of the sub-network g(W) CWW; D) based on the sparse sub-network W, which can be directly back-projected to the dense network as the approximated gradients of the dense parame- ters. The approximated parameter update rule for the dense network (see Fig. 2(a) Appendix) can be formulated
W411 < W, -W..).
where Wi represents dense parameters at iteration and Y indicates the learning rate.
3.2.1 ANALYSIS OF DYNAMIC PRUNING USING STE
To validate the performance of STE on N:M sparse networks, perform pilot experiment with STE. The results are shown Fig. ((a) From Fig. 3(a) N:M neural network trained with the above mentioned STE shows significant performance drop compared with the dense network. We conjecture that this drop results from unstable neural architecture updates caused by the approxi-
mated gradients of the dense parameters from the mod-ffdified chain rules. As Eq 3hows,, g(W)) rough estimate of the gradients for W, due to the mismatch between the forward and back- ward passes. When conducting gradient descent to W with the rough gradients estimated by STE, discrepancies between the accurate gradients and approximated ones may lead to erroneous param- eter updates. These imprecise value updates the pruned parameters may further produce unsta- ble alternations of the architecture of the on-the-fly pruned sparse neural networks in the forward pass, which causes the notable performance drop. To demonstrate the possible relationship between sparse network architecture updates and performance drops, we define SAD (Sparse Architecture Divergence) and measure the network architecture change with this metric.

---

Published as a conference paper at ICLR 2021
Fediows
2011
(a) STE
(b) SR-STE
Figure 2: In this figure. O represents clement-wise multiplication and & indicates matrix multipli- cation. (a) This figure shows the forward and backward pass during training an N:M sparse network. In the forward stage, W is obtained by pruning W. And in the backward stage, the gradient w.r.. W will be applied W directly.
(b) This figure illustrates the training process with SR-STE. The forward pass is the same as in (b). However, in the backward pass, the weights of W are updated by not only 0C but also & @ W. where & is the mask matrix for the pruned weights in W. ow
101
SparSn/STE1
A.5
Spurse woglia Riznward Nemo
5-
Derso weagrta Witewward Kas!
Epoch
(a) top-1 accuracy v.s. epoch
(b) SAD v.s. layer number
Figure We compare two networks respectively trained with regular SGD method and STE- modified gradient descent. (a) This figure shows sparse networks trained with STE has significant performance drop in top-1 accuracy compared with dense networks.
(b) This figure illustrates the layer-wise SAD between the weights after certain number of iterations and the initial weights, for two networks trained with STE (sparse forward) and regular SGD(dense forward). Compared with networks trained with sparse forward gradient, the one with dense forward gradient displays smaller SAD, indicating fewer updates in its sparse network architectures.
Before formally defining SAD, we first define the binary parameter mask produced in the magnitude- based pruning process as ε = {E' E {0, 1 : 0 | S L} where Nj represents the length of W Specifically, if the i-th parameter of W survived (not pruned) in the pruned sub-networf otherwise. Thus, the sparse sub-network can be represented as W (W' @ : < < L). where @ represents element-wise multiplication. For convenience, define 8 1
W. we set E 1, and E
For a single training run, we propose Sparse Architecture Divergence (SAD) to measure the change of the binary mask & from W; (the weights after the i-th itcration) to W, (the weights after the j-th iteration). We define SADaj = ||ε Eill, where &, and Ej are the binary masks for W; and W, respectively. This formula measures the number of connections(weights that are pruned in
W, = W, 3 & and not pruned in W,- = W2 @ Ey or pruned in W; while not pruned in Wy. Smaller SAD indicates less discrepancy between the network architectures of W; and Wy-
To reveal the impact of STE to the obtained sparse network architecture, we analyze SAD between the weights after different number training iterations of two training schemes. Our quantitative results are shown in Fig. ((b The first scheme applies forward pass with dense weights and updates the net weights by W4+1 W;-- (W). Then the corresponding sparse sub-network obtained by pruning the trained dense network. The other updating scheme carries out forward pass with
sparse layers and uses STE-modified chain rule, W <- W. (W,) in backward pass. We
5

---

Published as a conference paper at ICLR 2021
define PW; to be the sparse model pruned from the model trained for iterations of regular gradient descent, and $ $W; be the sparse model pruned from the network trained with i iterations of STE- modified gradient descent. Let SSA' denote SAD between the i-th layer of x "Wo and $W trained with sparse forward. Similarly, SAD represents the SAD value between the I-th layer of PWo and DW when trained with dense forward. As depicted in Fig. 3(b) for each layer number SSA' o.t SAD' holds both when 1 and t 10. Meanwhile, SADD PSAD grows as t increases from 1 to 10. This phenomenon indicates a positive correlation between the poor performance of a sparse neural network and high SAD.
Before our defined SAD, (Liu et al., 2020) proposes Neural Network Sparse Topology Distance (NNSTD) measure topological distances between two sparse neural networks. It is worth noting that NNSTD reorders the neurons in each layer, based on their connections to the previous layer, to maximize the similarities between the two compared networks' topologies. Hence, NNSTD can only measure general topological differences between two networks but fails to reflect the transitions of individual connections' states (pruned or not pruned). However, when calculating SAD, the mask for each connection is directly computed without neurons being ordered, hence SAD could provide more precise estimation of actual state changes (from being pruned to not pruned, and vice versa) of network connections, which is what most concern about in this paper. It is also worth noting that SAD has been implicitly adopted in existing published research papers. In Rigl. (Evei et al., 2019a), the authors consider the case of SAD;- = 2k: where & is dynamically calculated for each layer during training. Rigl. can achieve state-of-the-art results on training unsturctured sparse networks from scratch. Another example is that, when we prune the network at initialization and don't update the connections, according to the recent work (Frankle et al. 2020), the performance drops significantly. This can be regarded as setting SAD = 0 during the whole training phase.
3.3
FIRIIGH-TTHROUGH ESTIMATOR (SR-STE)
Inspired by above observations made from SAD analysis, we aim reduce SAD improve the sparse network's 's performance. Since the magnitude of parameter |w| are used as a metric to prune the network weights, we need alternate the weight updating process in order to prevent high SAD. Two choices are left to us to achieve this goal: (1) restricting the values of weights pruned in W., (2) promoting the non- -nnupundd weights in W,. It is worth noting that although gradients of parameters calculated from STE are all approximated, the pruned parameters' gradients are more coarse than the non-pruned ones. Because for pruned parameters, the values to compute gradients in W; and the values to be updated in W; distinct, however for non-pruned parameters those two stages use the same values. These statements make penalizing weights pruned in W; a natural choice for restraining SAD.
Hence, we propose a sparse-refined term in the STE update formulation. We denote this new scheme as SR-STE. Compared to STE, when utilizing SR-STE in N:M sparse model training, the backward pass is carried out with a refined gradients for pruned weights, as illustrated in Fig, 2 The purpose of the regularization term is to decrease the magnitude of the pruned weights, which are determined in the forward pass. Intuitively, we encourage the pruned weights at the current iteration to be pruned also in the following iterations so that the sparse architecture is stabilized for enhancing the training efficiency and effectiveness.
Formally, the network parameter update rule changes from Eq. to the following formulation with a sparse-refined regularization term,
W = Wi - (((W)) + (E, @ W))),
(4)
where &, denotes the mask for the pruned weights, @ denotes Hadamard product, Aw denotes the relative weight for the sparse-refined term, and 7t denotes the learning rate.
When Aw = O. Eq. is equivalent to Eq. which is the STE update rule. In general, we set Aw > 0. SR-STE terms with a positive Aw set a constraint on and only on the pruned parameters, to prevent them from 1) being unpruned due to the different levels of mismatch between pruned parameter gradients and non-noonnnnnne nrunprd parameter gradients, and 2) ineffectively alternating the pruned network architecture. When fewer sparse connections in the network alternated, more stable training process and a higher validation accuracy would be expected, which has been demonstrated in the analysis above and manifested in following experiments.

---

Published as a conference paper at ICLR 2021
We perform extensive experiments with SR-STE, and these results be found in Fig. 4 The experiments here are conducted with ResNet-18 (He et al. 2016) on ImageNet (Deng et al. )009). In Fig. 4(a) 4 different settings of Aw, namely 0,0.0002, 0.00045, -0.00002, are inspected. With Aw < O. the potential negative impact of pruned weights' coarse gradients are enlarged, which leads the poorest top-1 accuracy (68.5%) and the most significant SAD. For smaller values of Aw corresponding to the standard version of SR-STE, SAD shrinks meanwhile the top-1 accuracy receives clear increase. Furthermore, we examine performances of three neural networks: 1) dense neural network trained with regular SGD method: 2) an N:M sparse network optimized with STE; 3) an N:M sparse network trained with SR-STE. The curves of their top-1 accuracy for all training epochs are illustrated in Fig. 4(b) The accuracy curve of STE is consistently below the other two curves, and has more turbulence between different training epochs. Note that the SAD value is associated with the learning rate 7. For instance, SAD grows rapidly during the first 5 epochs in Fig. since the learning rate increases from O to 0.1 in the so-called "warm-up" process, Besides, also present other formations of sparse-refined term in Appendix A.S
x1
2.2
STE/ 5)
5.0002
Sce (5TT)
Daras
2.0
5 0.00045
See (0158-5TE)
- -0,00022
0.6-
1.5
SAD
1.0
Top-
0.4-
0.0
0.1
20
40
60
50
100
120
20
60
80
100
120
Epoch
Epoch
(a) SAD v.s. epoch
(b) top-1 accuracy v.s. epoch
Figure 4: (a) This figure illustrates SAD as a function of training epoch number with different settings of Aw in the SR-STE term. When Xw < 0. the perturbations brought by coarse gradients of sparse wights are widened, SAD gets higher and the top-1 accuracy becomes lower. When Aw is set to a reasonable positive value, sparse nets received high performance and low SAD. (b) This figure compares the top-1 accuracy curves of sparse net trained with STE, sparse net trained with SR-STE, and dense net. Sparse networks naively trained with STE have significant performance drop compared with dense ones. After introducing the SR-STE term into optimization process, the sparse network's performance jumps to comparable level with dense networks.
4 EXPERIMENTS
In this section, we demonstrate the effectiveness of our proposed N:M fine-grained structured spar- sity neural network on computer vision tasks (e.g.. image classification, object detection, instance segmentation, and optical flow prediction) and machine translation tasks. For these experiments, the implementation details, including dataset settings, training schedules, and evaluation metrics, are listed in the Appendix A.44 Meanwhile, we set Xw as 0.0002 in all experiments because this value gives good performance in our experiments.
4.1 IMAGE CLASSIFICATION
In this section, we first conduct several experiments to evaluate the effects of different N:M sparse patterns and different training methods on the image classification benchmark ImageNet- 1K (Deng al.) 2009) with different backbones. Then, we provide the performance of our proposed N:M fine fine-grained structured sparsity network compared with the state-of-the-arr sparsity methods.
Different N:M Sparse Patterns. To investigate the performance of different N:M sparse patterns, we exploit the popular ResNet50 (He et al. 2016) model with four different N:M structural sparse patterns: 2:4, 4:8, 1:4, and 2:8. The baseline model is the traditional dense model. For the designed patterns, the 2:4 and 4:8 have the same sparsity 50%. The sparsity of 1:4 and 2:8 are both 75%. In Table I we observe that the 4:8 structural sparsity outperforms 2:4 with the same computational
7

---

Published as a conference paper at ICLR 2021
Table I: ImageNet validation accuracy on ResNet with different N:M sparse patterns.
Model
Methed
Sperse Pattern
Top-1 Acc(S)
Params((M)
Flops(G)
ResNet50
Donse
77.3
25.6
4.09
ResNet50
SR STE
2:4
77.0
13.8
2,15
ResNet50
13.8
2.15
ResNet50
SR STE
1:4
75.3
7,93
1.17
ResNet50
SR-STE
2:8
76.2
7,93
1.17
ResNei50 x1.25
SR-STE
11.8
1.79
Table 2: Experimental results of different training methods for training the N:M sparse network.
Model
Method
Sparse Pattern
Top-I Ace
Epochs
ResNer18
ASP (Nvidia
2020
2:4
70.7
200
ResNer18
STE
RosNet18
SR-STE
2:4
71.2
120
ResNer50
ASPiNvidia
2020
76.8
200
ResNer50
STE
2:4
76.4
120
ReaNe150
SR-STE
Table Experimental results of the proposed N:M sparse pattern with SR-STE and state-of state-of-the-arr sparsity methods. imply that the first layer keep dense.
Methed
Top-1 Acc(S)
(pursity(%)
Parums(M)
Flop((G)
Structured
Uniform
ResNe150
77.3
25.6
4.09
DSR"
71.6
80
5.12
0.82
x
RigL
74.6
GMP
75.6
80
5.12
0.82
X
STE
76.2
80
5.12
0.82
SR-STE
76.2
7.93
1.17
RigL
67.5
GMP
70.6
95
1.28
0.20
STR
70.2
95
1.24
0.16
STE
68.4
95
1.28
0.20
SR-STE
1:16
cost, and 2:8 also performs better than 1:4. the training curve in Fig. 6(a) It shows that with the same sparsity for N:M structural sparse patterns, larger will lead to better performance since provide more abundant convolution kernel shape (we visualize and analysis the convolution ker- nel shape in Appendix(A.2) For the fixed M, we can adjust N to obtain the different sparsity ranges. With the same M, it is reasonable that the performance of the larger N is better due to more pa- rameters and computational cost. Meanwhile, the performance of the ResNet50 with 1.25 width can achieve 77.5% top-1 accuracy about only 71% sparsity of original dense ResNet50. We also conduct several experiments with RegNetXs (Radosavovic et al. (020) to evaluate the effectiveness of our proposed :: fine-grained structural sparse patterns the compact models Appendix A.3
Different Training Methods. We also verify the effectiveness of the proposed SR-STE for training the N:M sparse pattern neural network. In Table 2] we find that SR-STE outperforms the NVIDIA ASP method and STE with less training epochs and better accuracy.
Comparison with State-of-the-arte Before the advent of N:M fine-grained structured sparsity, there exist many state-of-the toeic-----he-an- art methods to generate sparsity models, including DSR (Mostafa & Wang 2019,, Rigl. (Evei et al., (2019)), GMP (Gale et al. 2019). and STR (Kusupati et al.] 2020). SR-STE is compared to those methods on ResNet50 in mid-level sparsity( 80%) and ultra-level sparsity 95%). Table Table showw that SR-STE can outperform all state-of-the-art methods, even if other methods are unstructured sparsity. And STR (Kusupati et al.) 2020) shows that training the model with non-uniform sparsity can improve the performance consistently, thus the SR-STE can extend the sparsity with non-uniform structural sparsity setting (e.g.. mixed N:M fine- ine-grained structural sparsity). We believe that the mixed N:M sparsity could further improve the results and we leave this exploration for the future work.
8

---

Published as a conference paper at ICLR 2021
4.2 OBJECT DETECTION AND INSTANCE SEGMENTATION
We further conduct experiments on the challenging COCO dataset (Lin et al. 2014) to evaluate the efficiency of the proposed approach for two important computer vision tasks, i.e. object detection and instance segmentation. We exploit the classical model Faster RCNN (Ren et al., 2013) for object detection and Mask RCNN (He al. 2017) for instance segmentation. All the experiments are conducted based on MMDetection (Chen et al. 2019). Table and Table 3 show that 2:8 (25%) structured sparsity can achieve comparable result with dense baseline models. Furthermore, 4:8 (50%) sparsity can provide even better result than dense models. These results also illustrate that the N:M sparsity pre-traines model gives a similar or better feature transfer ability.
Table 4: Object detection results on COCO.
Table 5: Instance segmentation results on COCO.
Modal
Method
Sparee
Schi.
Model
Method
Spurie Patem
Schd
Box MAP
Mask mAP
F-RCNN-RSO
M-RCNN-RSO0
36.1
K-RCNN-RSO
SR-STF
M-RCNN-R50
SR-STR
E-UNNN
MERCNKK3O
Dense
2X
39.4
35.4
E-RCNN-RS0
2:4
39.2
M-RCNN-R30
SR-STE
244
2x
39.8
35.9
PRCNN R50
SR-STE
28
2x
38.9
M RCNN R50
SRSTEE
28
28
39.4
35.4
4.3 OPTICAL FLOW AND MACHINE TRANSLATION
Optical flow prediction is one representative dense pixel-level prediction task in computer vision. We verify our proposed method recent state-of-the-art model RAFT (Teed & Deng 2020) model on FlyingChairs (Dosovitskiy et al. 2015). The smaller value the metric end-point-error (EPE) represents better performance. Compared with the dense model for optical flow prediction, Table shows that our proposed method can achieve comparable accuracy with half of the parameters.
Besides the computer vision tasks, we investigate the effectiveness of our method on one of the most common tasks in natural language processing, i.e., machine translation. We conduct our ex- periments based on Transformer, which employs number of linear layers. We train our models with transformer base adopted by (Vaswani et al.] 2017p, which contains a 6-layer encoder and 6-layer decoder with 512-dimensional hidden representations. The larger value of the metric BLUE represents better performance. Compared with the dense model, Table 7 shows that our proposed method can achieve the negligible accuracy loss.
Table 6: RAFT results on FlyingChairs.
Table 7: MT Results on the EN-DE WMT 14.
Sparse
Puzmm(M) FloperG
Model
Methed
Sporse potten
BLUF
PanmerM)
Fleps(C)
Dense
134
Trandommer
Denie
27.31
63
10.2
RAFT
SRASTE
2:4
0.88
Transtomer
SRSTE
2:4
27.23
31.5
5.1
DISCUSSION AND CONCLUSION
In this work, we present SR-STE for the first time to train N:M fine- ane-grained structural sparse net- works from scratch. SR- extends Straight-Throus viate ineffective sparse architecture updates brought by coarse gradients computed by STE-modified chain rules. We define metric, Sparse Architecture Difference (SAD), analyze these architecture updates, and experimental results show SAD correlates strongly with pruned network's 's performance. We hope this work could shed light on machine learning acceleration and SAD could inspire more theoretical and empirical studies in sparse network training.
Estimator with regularization term to alle-
Acknowledgements We thank all the anonymous reviewers, Qingpeng Zhu and Tim Tsz-Kit LAU for their feedback this work. This work is supported in part by Sense Group Limited, in part by Centre for Perceptual and Interactive Intellgience (CPII) Limited, in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants CUHK 14207319/14208417.
REFERENCES
Guillaume Bellec, David Kappel. Wolfgang Maass, and Robert Legenstein. rewiring: Training very sparse deep networks. arXiv preprint 171 2017.

---

Published as a conference paper at ICLR 2021
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 130.iv:330..34. 2013.
David Bourgin, Joshua Peterson, Daniel Reichman, Stuart J Russell, and Thomas L Griffiths. Cognitive model priors for predicting human decisions. In International conference on machine learning, pp. 5133-5141, 2019.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv: /4/65, 2020.
Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 0etp//906007555 2019.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint 1602.02600 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 245. leee, 2009.
Tim Dettmers and Luke Zeutlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 197.::1907.048400 2019.
Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 2758-2766, 2015.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. arXiv preprint 1rXiv:1911.111... 2019a.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint 1rXiv:1906.1072 2019b.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:/803.0366 /80300663, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576, 2020.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.009: 099744, 2019.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. In Advances in neural information processing systems, pp. 1379-1387, 2016.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NIPS), pp- 1135-1143, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 558-567, 2019.
10

---

Published as a conference paper at ICLR 2021
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint D.Xiv:5503.025:: 2015.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 744666 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- pengger-rithmetic-onn inference. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2704-2713, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference Learning Representations (ICLR), 2015.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. arXiv preprint arXiv:2002.03231, 2020.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598 198-605,, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.0000 1810.02340 2018.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.087110 2016.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Shiwei Liu, Tim Van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada Sokar, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Topological insights into sparse neural networks. arXiv preprint arXiv:2006. 14085, 2020.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 911::1112, 2018.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. arXiv preprint arXiv:1902.05967. 2019.
Nvidia. Nvidia a100 tensor core gpu architecture. 7/SSoltionsDData--Cmttrvvvvi
http://wwww.iiddd..comooommeeee
2020.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-315 2002.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428-10436, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525 542. Springer, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003.02380 2020.

---

Published as a conference paper at ICLR 2021
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1715-1725, 2016.
Zhanhong Tan, Jiebo Song, Xiaolong Ma, Sia-Huat Tan, Hongyang Chen, Yuanqing Miao, Yifu Wu, Shaokai Ye, Yanzhi Wang, Dehui Li, et al. Penn: Pattern-based fine-grained regular pruning towards optimizing enn accelerators. arXiv preprint wrXiv:200.04997, 2020.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. arXiv preprint vrXiv:2003./2033 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yanzhi Wang. Shaokai Ye, Zhezhi He, Xiaolong Ma. Linfeng Zhang, Sheng Lin, Geng Yuan, Sia Huat Tan, Zhengang Li, Deliang Fan, et al. Non-structured dnn weight pruning considered harmful. arXiv preprint arXiv: /907 02 124, 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In Ad- vances Neural Information Processing Systems, pp. 2684-2694, 2019.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- estimator in training activation quantized neural nets. arXiv preprint arXiv:1903.05662, 2019.
standing
Aojun Zhou, Anhang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. arXiv preprint arXiv: 011.//022..4.. 2017.
A APPENDIX
A.1 ALGORITHM
Algorithm Training N:M sparse Neural Networks from Scratch of SR-STE
Require: N, M, Aw, dataset D.
randomly initilized model W,
2: learning rate
3: for each training iteration do
4: Sample mini batch of data (X) ~ D
5: W N,
> Eq.
6: Obtain corresponding mask E
7:
t L(W, (}))))
Forward pass
8:
g(W)= ow
ae
Backward pass
9:
W+ W no((WV + (w W)),
> Eq
A.2 KERNEL SHAPE
illustrates six learnt convolution kernels which are picked up from the trained ResNet50 2:8 sparse model. Note that, for these six convolution kernels, the shape of non-zero elements under the 2:8 sparsity constraints cannot be acquired or learnt in the case 1:4 sparsity.
Fig. 5
A.3 REGNETXS ON IMAGENET-1K
We further verify whether N:M sparsity can boost the compact models. Recent RegNetXs (Ra- dosavovic al., 2020) are state-of-the-art and hardware-friendee models, which are the best models out of a search space with 10 candidate models. The Table shows that SR-STE can improve RegNetX002 performance than STE significantly, and the 2:4 structured sparsity outperform
12

---

Published as a conference paper at ICLR 2021
0
0
0
0U68)
IavrnLowa (072)
(1147)
0
IaverLLoooa
60:00
LyentLond
Figure 5: Illustration of kernel shape in ResNer50 with 2:8 structured sparsity trained model, layerI.I.conv:: (0.32) denotes layer name: (index of input channel, index output channel).
the dense NegNetXs model with the same Flops. Therefore, we can obtain N:M fine-grained struc- tured sparse model with SR-STE easily and we believe that the proposed N:M fine-grained structured sparsity method could become a standard technique on model deployment.
Table 8: ImageNet validation accuracy on RegNet with different N:M sparse patterns.
Model
Method
Spanse Pattern
Top-I acc(3)
Flops(G)
RegNetX002
SR-STE
2:4
66.7
0.1
RegNerXX04
RegNetX006
SR-STE
2:4
72.8
0,3
RegNetX008
SR-STE
2:4
74.1
0.4
RegNetX006
SR-STE
2:4
76.7
0.8
RegNerX032
SR-STE
2:4
77.8
1.6
RegNetX002
Dense
68.3
0,2
RegNetX004
Dense
72.3
0.4
RegNetX0006
Dense
73.8
0.6
RegNeeXX008
Dense
74.9
0.8
RegNerX006
Donse
76.8
A.4 IMPLEMENTATION DETAILS
A.4.1 CLASSIFICATION
Dataset. ImageNet-IP K (Deng et al. 2009) is a large-scale classification task, which is known as the challenging image classification benchmark. ImageNet-IK dataset has about 1.2 million training images and 50 thousand validation images. Each image is annotated as one of 1000 object classes.
Training scheduler. All ImageNet-1K -K experiments trained images of 224 x 224, the dense models
models baselines following the hyperparameter setting in (He et al. 2019). Specifically, are trained with batch size of 256 over 120 epochs and learning rates are annealed from 0. to 0 with a cosine scheduler and first 5 epochs the learning rate linearly increases from 0 to O.I.
Evaluation Metric. All reported results follow standard Top-1 accuracy.
A.4.2 OBJECT DETECTION AND INSTANCE SEGMENTATION
Dataset. All experiments performed on the challenging MS COCO 2017 dataset (Lin et al.) 2014) of 80 categories. It consists I15K images for training (train-2017) and 5K images for validation (val-20/7). We train models on the training dataset train-20/7 and evaluate models the validation dataset val-2017.
Training scheduler. For object detection and instance segmentation, conduct these experiments on MMDetection (Chen et al. 2019). The Ix and 2x training schedule settings follow the settings in MMDetection (Chen al. 2019).
13

---

Published as a conference paper at ICLR 2021
Evaluation Metric. All reported results follow standard COCO-style Average Precision (AP) met- ric, i.e., mAP of IoUs from 0.5 to 0.95.
A.4.3 OPTICAL FLOW
Dataset. The optical flow prediction is conducted the FlyingChairs (Dosovitskiy al.] 2015) dataset, which is a synthetic dataset with optical flow ground truth. This dataset consists of 22,872 image pairs and corresponding flow fields. The training dataset contains 22,232 samples and the validation dataset contains 640 test samples. We train the RAFT (Teed & Deng 2020) model on the training dataset and report the final results on this validation dataset.
Training scheduler. We employ the original standard open framework to run the RAFT Teed & Deng 2020) model. And the training settings for the FlyingChairs (Dosovitskiy al. 2015 dataset have been listed in Teed & Deng (2020).
Evaluation Metric. We choose the endpoint error (EPE) evaluate the predicted result. EPE the Euclidean distance between the predicted flow vector and the ground truth, averaged over all pixels.
A.4.4 MACHINE TRANSLATION
Dataset. For English-German translation, the training set consists of about 4.5 million bilingual sentence pairs from WMT 2014. We use newstest2013 as the validation set and newstest2014 the test set. Sentences are encoded using BPE, which has a shared vocabulary of about 37,000 tokens.
Training scheduler. We use subword method (Sennrich et al. 2016) encode source side sentences and the combination of target side sentences. The vocabulary size 37,000 for both sides. Each mini-batch
GPU contains set of sentence pairs with roughly 4,096 source and 4,096 target
tokens. We use Adam optimizer (Kingma & Bal 2015) with 3 = 0.9 and ß32 = 0.98. For our model, we train for 300,000 steps. We employ four Titan XP GPUs train both the baseline and our model.
Evaluation Metric. All reported results follow standard BLUE (Papineni al. 2002) tokenized, truecase output.
0.8
2.0
0.7
Sparse Architacture Divergence!
Ac 0,000001
demma Loaining
0.5
0.4
0.3
0.1
20
Epoch
Epach
(a) Top-1 Accuracy v.s. epoch
(b) SAD v.s. epoch
Figure 6: (a) This figure compares the top-1 1 accuracy curves of different N:M patterns with SR- STE. (b) This figure illustrates the SAD curves as each training epoch number with different
sparse-refined formulation
A.5 OTHER REFINED FORMULATION
we present another sparse-refined regularization term. i.e., sign constant, as follow:
W W; (x()(W)) + (e( @ sign (W:))),
(5)
14

---

Published as a conference paper at ICLR 2021
Apart from the parameter of model, modified the Eq. 4 to apply approximated gradient di- rectly, as follow:
W41 = We - ((((((W) + ( @ (M9((W))))
(6)
Fig. 6(b) depicts the curve of SAD of Eq. Eq. and Eq. The refined term on gradients' SAD value is smaller than the STE in early stage of training while larger in the late training period, leading to worse performance (69.4%) than the performance of STE (69. 99).. The sign constant SAD value converges similar to the Eq. which improves the performance about 0.9%.
15


Reviews:
[{"review_id": "jtt8MEeeY-Z", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed a new method for training N:M fine-grained structured sparse networks from scratch. The authors found that the SAD metric, which measures the number of weights whose pruning state is changed, became higher if the existing STE is used to train sparse networks and this metric had the positive relationship with accuracy drop. To reduce the SAD, SR-STE which gives higher weight decay coefficient to the pruned weights is applied to train sparse networks, improving the accuracy of sparse networks trained from scratch.\n\nSR-STE which is penalizing the pruned weights is somewhat simple, but the results in the paper are strong. The proposed method supports to increase the accuracy of N:M fine-grained structured sparse networks, and it can be applied in many diverse tasks regardless of pruning method like general unstructured pruning. I think it is a good technique to easily apply when pruning states of each weight are continuously changed.\n\nHowever, weight-sorting process which occurs in every training step can decrease the training speed compared to other threshold-based pruning method. Could you compare the actual consuming time to train sparse network for each training method? \n\nComment: Below are some typos I think.\n\n1. 3.2. Analysis: I think |^{S}{SAD}^{l}_{0:t}-^{D}{SAD}^{l}_{0:t}| instead of |^{D}{SAD}^{l}_{0:t}-^{D}{SAD}^{l}_{0:t}| is likely to be the right expression in context.\n2. 3.2. Analysis: The sentence \"This formula measures the number of neurons that are pruned~\" is unfamiliar to me, because \"neuron\" is usually used when depicting activation, not weight of artificial neural network. I think \"weight\" is more matched expression.\n3. The stated GFLOPs of ResNet-50 (2:8 sparse pattern, SR-STE method) are different in Table 1 (1.02) and Table 3 (0.1). I think the former is right. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Vo5CVedPWZ-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose to train N:M structured sparse neural networks from scratch.  To improve the effectiveness of the training process, a sparse-refined straight through estimator (SR-STE) is proposed based on vanilla STE. Finally, the authors define Sparse Architecture Divergence (SAD) to indicate the topology change of the sparse network. The novelty of the proposed SR-STE is not enough, and the defined SAD lacks relevant theoretical support. Despite the good performance, I am concerned about the technical novelty of this paper. \n(1) The main contribution of the proposed SR-STE lies in a sparse-refined regularization term. It seems like a mitigation measure, which cannot completely eliminate the negative impact caused by the approximate measures taken in the STE. \n(2) And in my opinion, the analysis of SAD lacks relevant theoretical support and is not very convincing. I expect more explanations for SAD.\n\nThere are still several issues that need to be addressed. Firstly, the layout of this paper is not very suitable, and it is not very convenient in the process of finding the corresponding figures. Secondly, there are several narrative errors in this papere, which need to be checked carefully, e.g., in the third-to-last line of Section 1, I think it should be 'we propose a sparse refined term to enhance the effectiveness on training the sparse neural networks from scratch'; in the second-to-last line of Section 3.2, it should be '\\left | ^{S}\\textrm{SAD}_{0:t}^{l}- ^{D}\\textrm{SAD}_{0:t}^{l}\\right |'.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UIRFFFDj0HK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To maintain advantages of both unstructured and structured sparsity, the paper improves previous STE method based on the proposed SAD. The experiments on five types of applications show its outstanding performance.\n\npros.\n1. The motivation of this paper, maintaining advantages of both unstructured fine-grained sparsity and structured coarse-grained sparsity, is clear and persuasive.\n2. To validate outstanding performance of the proposed method, the authors did experiments on five different types of applications: image classification, object detection, instance segmentation, optical flow, and machine translation.\n3. The authors provide detailed explanations on physical meaning of their proposed SAD.\n\ncons.\n1. In Section Introduction, I think the authors should formally explain meaning of \"N:M\" when \"N:M\" is mentioned for the first time.\n2. In your github code train_imagenet.py, implementation of \"devkit\" package on 19th/20th/22nd line can not be found. I think your proposed method is mainly implemented in devkit.sparse_optimizer, while you do not include the most important code.\n3. In Figure 1, physical meaning of variables R and C is not introduced.\n4. The original STE is used for unstructured sparsity, while you improved STE for N:M structured sparsity. Why does your improvement encourage structured sparsity? \n5. According to Table 2, the classification performance gap between STE and SR-STE is minor. So I think SR-STE is not a significant improvement compared to the original STE.  I am not 100% sure about my opinion on performance, since comparison between STE and SR-STE is not provided for the experiments on other applications (detection, segmentation, optical flow, and machine translation).", "extra_scores": {}, "extra_text": {}}, {"review_id": "IM0fcQlyhhM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper introduces a new sparse training algorithm (SR-STE) based on the straight through estimator which is specially designed for the hardware constraints of Nvidia A100 GPU. Auxiliary, in order to study better this algorithm, the paper introduces also a metric (SAD) to measure the changes in the sparse network topology during training. The contributions have real added value as they show that sparse neural networks can actually benefit of hardware designed to consider sparsity. The experiments on CNNs and Transformers support the claims.\n\nStrong Points\n\n•\tSR-STE can take real advantage of A100 sparse capabilities.\n\n•\tThe algorithm is designed for the general form of n:m sparsity and, likely, can be used also for the next generation of GPUs with sparse networks support.\n\n•\tIt is the first study which, up to my knowledge, shows consistently that sparse networks can outperform dense networks also for convolutional layers.\n\n•\tThe paper is written in a clear manner and well structured.\n\nWeak points\n\n•\tThe proposed SAD metric seems to work just on the same network during its own training process, as it does not consider the fact that at different training runs the hidden neurons may develop differently.\n\n•\tThe related work is not complete\n\n•\tThe paper has a number of sloppy written passages and inconsistencies in the mathematical formalism.\n\nDuring the rebuttal, I would suggest to the authors to consider the following comments:\n\n1) Improve the mathematical formalism and perform a careful proof-read to make sure that all notations are consistent. For instance page 5,  2nd paragraph, E_f is actually \\epsilon_f?;  page 5, third paragraph, the last equation has two equal terms, and so on.\n\n2) I believe that it would be informative to discuss the SAD metric limitations and advantages in comparison with other state-of-the-art metrics suitable to measure the distance between two sparse topologies, e.g. NNSTD https://arxiv.org/abs/2006.14085 .\n\n3)  Related work. the 4th paragraph on page 2 about the one-stage scheme to obtain efficient sparse networks is missing the first two references from 2017 which introduced sparse training with dynamic topology: https://arxiv.org/abs/1707.04780, https://arxiv.org/abs/1711.05136 . Auxiliary, I believe that also this work has to be cited https://arxiv.org/abs/1907.04840 . \n\n4) The last phrase of the above mentioned paragraph and the next one are, up to my knowledge, accurate just for CNNs. It has been shown already that for other types of neural networks (e.g. MLPs), sparse networks can easily and constantly outperform dense networks (e.g. https://arxiv.org/abs/1905.09397). Please make this discussion more accurate and informative.\n\n5)  Please read the paper thoroughly to address all typos and small inconsistencies. E.g., In Tables 1 and 2 “ours” appear just in some cases of SR-STE; In Table 3, STR appears twice with exactly the same sparsity level, but different number of parameters (more clarifications would make easier the reader job).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers like the simplicity of the approach and the fact that it works well.  
Decision: Accept (Poster)