HRBP: Hardware-friendly Regrouping towards Block-wise Pruning for Sparse Training
Authors: Haoyu Ma
Chengming Zhang
lizhi xiang
Xiaolong Ma
Geng Yuan
Wenkai Zhang
Shiwei Liu
Tianlong Chen
Dingwen Tao
Yanzhi Wang
Zhangyang Wang
Xiaohui Xie
Venue: Submitted to ICLR 2023 (2023)
Keywords: efficient training
sparse training
fine-grained structured sparsity
grouping algorithm

Abstract: Recently, pruning at initialization and training a sparse network from scratch (sparse training) become increasingly popular. However, most sparse training literature addresses only the unstructured sparsity, which in practice brings little benefit to the training acceleration on GPU due to the irregularity of non-zero weights. In this paper, we work on sparse training with fine-grained structured sparsity, by extracting a few dense blocks from unstructured sparse weights. For Convolutional Neural networks (CNN), however, the extracted dense blocks will be broken in backpropagation due to the shape transformation of convolution filters implemented by GEMM. Thus, previous block-wise pruning methods can only be used to accelerate the forward pass of sparse CNN training. To this end, we propose the Hardware-friendly Regrouping towards Block-based Pruning (HRBP), where the grouping is conducted on the kernel-wise mask. With HRBP, extracted dense blocks are preserved in backpropagation. We further propose HRBP++ to reduce zero kernels by extracting common sparse kernel patterns on all kernels within one block. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate that HRBP (HRBP++) can almost match the accuracy of unstructured sparse training methods while achieving a huge acceleration on hardware.  

Under review as a conference paper at ICLR 2023
HRBP: HARDWARE-FRIENDLYY REGROUPING TO- WARDS BLOCK-WISE PRUNING FOR SPARSE CNN TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, pruning at initialization and training sparse network from scratch (sparse training) become increasingly popular. However, most sparse training lit- erature addresses only the unstructured sparsity, which in practice brings little ben- efit to the training acceleration on GPU due to the irregularity of non-zero weights. In this paper, we work on sparse training with fine- grainerridd structured sparsity, by extracting few dense blocks from unstructured sparse weights. For Convolu- tional Neural networks (CNN), however, the extracted dense blocks will be broken in backpropagation due to the shape transformation of convolution filters imple- mented by GEMM. Thus, previous block-wise pruning methods can only be used accelerate the forward pass of sparse CNN training. To this end, we propose Hardware-friendly Regrouping towards Block-based Pruning (HRBP), where the grouping is conducted on the kernel-wise mask. With HRBP, extracted dense blocks are preserved in backpropagation. We further propose HRBP++ to reduce zero kernels by extracting common sparse kernel patterns on all kernels within one block. Extensive experiments on CIFAR-10, 10, CIFAR-100 100, and ImageNet demon- strate that HRBP (HRBP++) can almost match the accuracy of unstructured sparse training methods while achieving huge acceleration on hardware.
INTRODUCTION
Convolutional Neural Networks (CNN) have accomplished enormous progress on many computer vision tasks, such classification, detection, and segmentation. However, most successful models are overparameterized and computationally extensive. The excessive computation usually requires tedious training and makes it difficult to deploy cumbersome models into real-world applications. Network pruning (LeCun et al. 1990 Han et al., 2015a bj Li et al. 2016), which removes unneces- sary weights from the heavy dense model, stands as of the most effective methods to compress a heavy model into lightweight counterpart while maintaining its accuracy.
pruning follows three-step paradigm: 1) training dense network vergence; 2) identifying a subset of weights (sparse network) by pruning unnecessary connections; 3) retraining or finetuning the sparse network to recover accuracy. However, dense training still inevitable in this paradigm. The recent Lottery Ticket Hypothesis (LTH) (Frankle & Carbin 2019) suggests that sparse network can be trained from scratch (sparse training) to the same accuracy as its original dense model. Consequently, the tedious dense training is unnecessary. During the training process, The sparse structure (sparse mask) can either be static (Lee et al., 2019 Wang et al. 2020
Traditionally, network
Tanaka et al. 2020) or dynamic (Mocanu et al. 2018 Evei et al.
2020
Liu et al. 2021).
Tanaka et al. 2020: Mocanu al,, 2018 Evei et al. 20200 Liu et al.) 2021 explore unstructured sparsity only, where zero weights distribute irregularly. Although unstructured sparsity can maintain accuracy at a high sparsity ratio, it brings little training time reduction modern hardware because the irregular mask leads to poor data locality and low parallelism (He et al.] 2017: Mao et al. (2017) Wen et al.) 2016). An alternative approach, structured sparsity (He al.) 22017] Liu et al. 2017). where the entire filter channel is pruned, is more hardware-frient and computationally efficient. However, it usually leads to more accuracy drop compared to unstructured pruning.
Most sparse training methods (Lee et
2019 Wang at 2020

---

Under review as a conference paper at ICLR 2023
Recently, fine fine-grained structured pruning becomes popular, which is trade-off between structured pruming and unstructured pruning. On the one hand, the N:M sparsity (Zhou et al. 2021 Sun et al. 2021) defines blocks to meet requirements that only N weights are non-zero for every continu- ous M weights, which allows acceleration in the inference phase on modern hardware. The N: M transposable mask (Hubara al.) (02T) further ensures that both the weight matrix W and its trans- pose V follow the same sparsity pattern. Thus it can accelerate both forward pass and backward pass. However, these methods require specialized hardware, i.e., the sparse tensor cores (Zhu al.) 2019). Moreover, the transpose matrix W does not describe the backward of CNN accurately. As shown in Fig. | the convolution operation is usually implemented with general matrix multiplica- tion (GEMM) on hardware. In this case, calculating the gradient w.r.t inputs requires rotating each kernel first, then conducting kernel-wise transpose, rather than a simple transpose operation (See Sec. 2.T for more detail). Thus, the transposable masks may not always achieve the expected ac- celeration on the backward pass of CNN. On the other hand, the regrouping algorithm (block-wist wise pruning) (Rumi et al. 2020) Yuan et al. 2021] Chen et al., 2022) finds dense blocks by grouping unstructured sparse weights, which can accelerate sparse training on general hardware. However, as shown in Fig. 2 the extracted blocks in forward pass usually cannot be maintained in backward pass. Thus, these methods cannot accelerate the backpropagation as well.
In this paper, we propose the Hardware-friendlf Regrouping towards Block-wise Pruning (HRBP) for sparse CNN training. HRBP performs the regrouping algorithm on the kernel-wise mask. Thus, has the ability maintain the same dense blocks both forward and backward pass. Mean- while, all blocks extracted by HRBP have the same shape. which can alleviate unbalanced workload issues in many-core graphies processing units (GPUs) (Chen et al., 2010). Furthermore, we pro- pose HRBP++ to reduce the number of zero kernels, where all kernels in one group share the same sparse pattern. Specifically, sparse training with fixed HRBP++ can almost match the accuracy of unstructured pruning methods such SNIP (Lee et al., 2019) and GraSP (Wang et al.) [2020), but brings 1.4x and 1. 1.6x overall training acceleration with 90% and 95% sparsity for ResNet. Our main contributions are summarized as follows:
We detailed analyze the implementation of CNN's forward and backward pass with GEMM, and find that current fine fineggrained structured pruning methods cannot guarantee the backward acceleration.
We propose a novel prrdwwrre-frenddy Regrouping Block-wise Pruning (HRBP/HRBP++) algorithm that extracts dense blocks from the non-zero weights, while maintaining the spa- tial regularity of the blocks from the weight transformation of the backward propagation, therefore accelerating both forward and backward of CNN training.
We conduct extensive experiments on CIFAR- 10/100 and ImageNet-IK and demonstrate that sparse training with HRBP can achieve better trade-off between accuracy and hard- ware acceleration.
PRELIMINARIES
2.1 CONVOLUTION OPERATION AND ITS IMPLEMENTATION
The weights of 2D convolutional layer can be defined by K E XCooCC where Co. C₁. K2, and K2 are the number of output channels, the number of input channels, kernel height, and kernel width, respectively. In convolution operation, each filter K2 slides over the input feature map I RCixHixW, and computes a weighted sum of the mapped input values time, which generates one activation map O, E R Hooo Thus, all Co filters conduct Co times of convolution operations and produce the output map O E xCoxHoxWW
Forward pass with GEMM. On hardware, the convolution operation is usually implemented with general matrix-matrix multiplication (GEMM) (Chetlur et al., 2014), where tensor is laid out in the memory in the NCHW the NHWC format (See Appendix [A for more details). We take the NCHW format as an example. As shown in Fig. [a), for the input I, the im2col() operation flattens each convolution window of the input and stacks them as columns in a matrix. Thus, the 2D input feature map I is unrolled into an input matrix X iml E (x, Wo), Meanwhile, K is reshaped and stored in the weights matrix W E (C K.K),, To this end, the forward pass is
2

---

Under review as a conference paper at ICLR 2023
X-im2eol (1)
dV im-ooooo
Impot
1
DO
12
IO E1
im2col
14
padding
imnleol
BCC
18
16
15
12 64
0011
Kernel
K
15
17
Kernels K
11 13
Lnjal
W dY
ho
hS
h3
17
b2
bo
bd h3
reshupe
X
180P
reshape
reshape
3
07
01
62
62
61
oc
kernel-
8:3 20
dl
reshape
wise
transpose
[W WT
Forward
Backward
(c) W
Implementation of forward and backward pass of convolution operation with GEMM in NCHW layout. Different color represents different channels, the forward pass, kernels are reshaped to matrix W. In the backward pass, each filter is rotated 180 firstly, then the kernel-wise transpose is conducted to obtain the new kernel layout K', Then K' is reshaped to matrix W which is different from the transpose W
Figure 1:
calculated by Y WX E RCxx(HoW)) Then the 2D output map O is obtained by reshaping Y. See Appendix D for summary of all notations in this paper.
Backward pass with GEMM. Given the gradients of the 2D output map dO E RCoxHoxWo the backpropagation involves two matrix multiplications. 1) Calculate the gradients w.r.t. the filters dK, which implemented by dW dY d'-X' following dk reshape(dW)
Calculate the gradients w.r.t the input dI, which can be obtained by full convolution between the kernel K and dO (LeCun et al. 1989). In detail, as in Fig. [b). we conduct padding and im2col operation on dO, and obtain dY im2cc co (dO) IR Co X Meanwhile, flip each kernel first vertically and then horizontally (i.e., 180 rotation) and perform the kernel-wise transpose to get the new kernel layout K'. Then we reshape K' to matrix W' E C (Co (Co Kw), Thus, the gradient is calculated by dX W'dY. Finally, reshape dX obtain dI.
Discussion. For the gradient w.r.t the input, previous works (Hubara et al. 2021 caleulate it with dx = WT for simplicity, where WT E (C, K=Ko)xCo, This simple form is applicable linear layers. However, cannot describe CNN accurately in general. As in Fig. (c). W different from the matrix W If and only if K₁, K 1, they the same, which downgrades the CNN to a linear layer. Consequently, sparse patterns based on WT may not always obtain the expected acceleration arbitrary CNN backpropagation. Besides, this algorithm has additional
requirements on the shape of CNN and the layout in the memory. See Appendix B for more details.
W
2 3
W
KT
122
KO
K1
K3
K0
4
11
KO
5
K3
15
grouping
K3
K4
grouping
C.K
G C. K
pruned convolutional
weight regrouping
(c) mask after transformation for backward
Figure 2: Example of weight regrouping on convolutional layer. Indexed cells are non-zeros. Different color represents different block groups. The extracted dense blocks cannot be kept in the backward pass.
2.2 WEIGHT REGROUPING ON UNSTRUCTURED SPARSITY
Given CNN fe(:) with weights 8, a sparse subnetwork is defined by fesm..) (-), where m {0. 10 the binary mask and @ is the element-win wise product. unstructured pruning, zeros distribute irregularly in m. Thus, the convolution operation still needs to maintain the entire weight matrix
3

---

Under review as a conference paper at ICLR 2023
as dense network, making it difficult to reduce the computation of unstructured sparse weights on hardware. Even with the help of the dedicated sparse matrix representation technique such as CSR format (Buluç et al.] 2009). the unstructured sparsity still expects to have over 85% sparsity ratio to acquire limited acceleration since the irregular weight distribution causes significant computation overhead due poor data locality (Yuan et al. 202T). The recent weight regrouping (reorganization) (Rumi et al. 2020) can accelerate the unstructured sparse weights on hardware by extracting multiple smaller dense blocks in a large sparse matrix, which can improve the throughput with GEMM.
Implementation. Denote the binary mask of the weights of CNN layer as E {0. 1)(Wi, The regrouping algorithm (Rumi et al. 2020) finds similar rows and columns from the sparse weights matrix W @ m and brings them together into several dense blocks. In detail, it firstly clusters Co rows of m into several groups based on the Jaccard similarity among non-zeros columns. For each group, it then picks out columns with the most non-zeros weights from all Cy K₁ Kw columns and generates one dense block. For example, in Fig. 2(a). we can group filter Ko, 11, and K3 together, and select columns with at least two non-zero (i.e., 0th, 2th and 5th columns), which leads the dense block with orange color in Fig. (b).. The remaining sparse weights (cells with green color) are usually discarded (Chen et al. 2022).
Limitations. However, the regrouping algorithm cannot be applied to the training directly. Firstly, the extracted dense blocks are fragmentary in backward pass due to the transformation from W to W', making the backward acceleration unfeasible. For example, in Fig. (c),, the cells of the block with orange color in W m are scattered irregularly in W One naive solution extracting blocks separately in forward and backward passes. However, it is possible that specific weights in the blocks of forward are removed in the blocks of backward, which obstructs the backpropagation of the weight. Besides, the shapes of different dense blocks are arbitrary, which introduces imbalanced memory access and data locality. Thus, it makes the GPU suffer a great workload imbalance.
METHODOLOGY
3.1 HARDWARE-FRIENDAA REGROUPING FOR BLOCK-WISE PRUNING (HRBP)
Motivation. The reason for fragmentary blocks in the backward of CNN is that all C2 Kw columns are considered independently. Thus, one dense block can select arbitrary locations on the kernel of different input channels. Ideally, the grouping algorithm should locate identical elements for different input channels within one block. When K₁, = K = 1, the CNN reduces to linear layer, and the issue of the fragmentary block can be solved naturally. Inspired by this, we propose the HRBP, which extracts dense blocks that can also be kept in CNN backpropagation.
Different from (Rumi et al. 2020) that conducts regrouping on the mask of weights matrix m {0. Co Kú, kw), HRBP conducts regrouping on the kernel-wise mask m' E {0. 1) CoxCr, Thus, the entire weights of one kernel K. E IR K, Kw are kept or discarded based on m'. In detail, we count the number of non-zero cells within each kernel based on m, and we keep rCyCo kernels with the most number of non-zero cells and discard the remaining kernels, which generates the kernel- wise mask m'. Here r the density ratio of the sparse weights. We then cluster C. rows of m' into t groups with equal size. For each group, we select r C2 columns with the most non-zero weights to generate one dense block. Lastly, we discard the remaining elements and refill zero slots in dense blocks. To this end, we have t dense blocks with identical shape Co (rC,K, As in Fig. 3 3(a), (), the same dense blocks are preserved in backpropagation, making the sparse training with dense blocks applicable in forward and backward passes. See Appendix C for detailed implementation.
HRBP.
The vanilla HRBP inevitably introduces too many zero kernels, which may result in layer collapse and heavy accuracy drop (Tanaka et al., 2020). Thus, we further propose HRBP++ to reduce the number of zero kernels. We define s as the density ratio of each kernel K In practice, s needs to satisfy 8 2 O E (i/(K, K.)) [=1 Kw, Similarly, we select : CC kernels with the most number of non-zero cells to generate m². For each group, we extract a common sparse pattern from all kernels within the group based
HRBP++.
In detail, for each group, we count the number of non-zero weights in each cell of the kernel and choose Kw, cells with the most non-zero weights as common pattern for the dense block. For example, in Fig. B].). one kernel has 4 cells. Given the 6 kernels from the blue group, the number of zor- weights is 5, 4, 5, 2 for the cells, respectively. Thus, we select the 1, 2,2,3-th 3-th cell common pattern and apply it to all kernels
4

---

Under review as a conference paper at ICLR 2023
1
Aw
Forward
Apply
111
Backward
Grouping
Co Xit, Kw)
(a) HRBP
m Kw
Pattem Finding
Forward
Co
Apply
m'
Grouping
Co (K, Kw)
(b) HRBP-
Figure 3: Illustration of HRBP and HRBP++. White cell means zero weights. Given the unstructured mask (matrix with yellow cell). HRBP(++) firstly extracts the kernel-wise mask (matrix with lavender cell) by count- ing non-zero weights of each kemel. We then group the kernel-wiss mask into several equal shape blocks (we have three groups marked with blue, orange, and green). HRBP, the entire weights kernel are kept or discarded. HRBP+,, we extract common patterns each block and apply it to all kernels within the group.
within the blue group. To this end, HRBP+ allows for zero values within kernels, resulting in fewer entirely discarded kernels for the same overall sparsity ratio.
3.2 SPARSE TRAINING WITH HRBP (++)
HRBP(++)-based Random Mask. With HRBP(++), we can extract dense blocks from sparse weights and accelerate both forward and backward of sparse training. For simplicity, we apply HRBP(++) to random pruning mask, which randomly chooses connections to satisfy the sparsity ratio requirements. To this end, we design strategy to create random pruning mask in HRBP (++) style, named HRBP(++ ++ + baase Random Mask. For the sparse ratio each layer, we follow the Endōs-Rényi-Kemed (ERK) (Evci et al. (020) where larger layers are allocated with higher sparsity than smaller layers within a network. For HRBP, we randomly split the Co rows into t equal groups. For each group, randomly keep rC, channels and zero out the rest. For HRBP++, a random unstructured mask m and apply the proposed HRBP++ algorithms to slightly modify m. replace the sparse weights matrix with
start from
Thus, based the HRBP(++)-based random mask, several dense blocks on GPU.
Block-wise Prune
Block-wise Grow
Block-wise HRBP
Prumed weights
Grown weights
Figure 4: Illustration of Block-wise updating for Dynamic Sparse Training.
Block-wise updating for Dynamic Sparse Training (DST) DST starts from a random sparse network. After optimizing several iterations, it prunes portion of weights based the pruning criterion and grows new connections according to the grow criterion. Then the new sparse network trained until the next update. Typically, any location of the entire weight can be pruned or activated
5

---

Under review as a conference paper at ICLR 2023
(i.e., grown). However, this default setting is not aligned with HRBP, since updating connections at arbitrary locations destroys the extracted blocks. To this end. we propose block-wise updating for DST with HRBP++, which follows the nrune-grow-regeee maintain the cluster of rows (filters). For each row, we prune d% weights based on the magnitude and grow back new d% weights based on its gradient. Then we perform the HRBP++ the new weights to group the new mask into several dense blocks.
scheme. As shown in Fig. 4
3.3 HRBP HARDWARE IMPLEMENTATION
We implement our method with CUDA and measure the acceleration rate on GPU, In detail, we rewrite the nn. Conv2d module of Pytorch with CUDA to CNN with sparse weights. The thread is the basic programmable unit that allows GPU programmers to use massive numbers of CUDA cores. CUDA threads are grouped at different levels such warp, block, and grid. Given the dense blocks found by HRBP/HRBP++, the original convolution computation will be decomposed into several computations corresponding to dense blocks, and these computations will be performed in parallel by threads. The extracted dense blocks from HRBP(++) have the same shape. To this end, in our kernel design, thread block only processes the output channels of the same group which enables us to have better reuse of the input data (e.g., the input data can be loaded to shared memory which can be access by all the threads inside a thread block). Inside thread block, each thread is responsible for one portion of the output (X, Y) dimension which further helps us to achieve good data reuse of the kernel weight (kernel weight can also be put into shared memory).
We also applied tiling across both height and width dimension of the input channel to increase the number of launched thread blocks, which further increases the parallelism level. The tiling size plays a significant role of the performance. A small tiling size can increase the parallelism level, while a big tiling size will provide better reuse of the weight data. Thus, finding a good tile size for given problem is non-trivial. A simple but powerful method is brute force search, it guarantees to find out the best tiling size for specific hardware. In our design, input data are put into shared memory and can be accessed by all thread block improve the data reuse. In the meantime, the cost by using brute force search is very low because most of the undesired tiling size candidates are discarded in order to satisfy the usage requirement of the shared memory.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Dataset & Networks. We follow the experimental settings in (Wang et al.) 2020 Yuan et al,
2021) and conduct experiments CIFAR-IO, CIFAR-100 and ImageNet-IK IK (Deng et al. 2009). As the common setting in sparse training, we apply the WideRess and VGG-19 (Simonyan
& Zisserman 2014) for CIFAR-10/100 and ResNet-50 (He al. 2016) for ImageNet.
Training. For 10/10- we use batch size of 128 and train networks with SGD optimizer for 160 epochs by default. The learning rate is set to 0.1 initially and is decayed by factor of 0.1 at the 80th and 120th. Moreover, we run each experiment 3 times and report the mean value and standard derivation. For ImageNet, networks for 100 epoch as (Wang et al., 2020). The learning rate is 0.1 initially and is decayed at 30-th, 60-th, and 90-th epoch with factor 0.1. For sparse training, set the number of dense blocks t to 8 RRumi et al. 2020) and the dense ratio of kernel s in HRBP++ to ş for all 3 x 3 kernels. The minimum size of a block B1 is set (Rumi et al. 2020).
adopt the Pytorch official implementation and train the
Hardware We evaluate our method on NVIDIA Ampere A100 (108 SMs, 40GB). The versions of CUDA and cuDNN are 11.0.0 and 8.0.4. respectively. For bascline implementations of convo- lution operations for full sparse weights, we adopt the widely used GEMM-based convolution, i.e., SUDNN_CONVOLL
___ Latp_ALGG___ _GEMMM (Chetlur al.) [014).. We report the overall training time acceleration rate compared with the dense baselines, which includes both forward and back- ward calculations on hardware.
4.2 SPARSE TRAINING WITH HRBP(++)-BASS) RANDOM MASK
Settings. We compare HRBP(++) static sparse training against random unstructured pruning (RU), random channel-win -winel-wise pruning (RC) with an ERK ratio, and unstructured pruning with well- defined criteria such SNIP (Lee et al. 2019) and GraSP (Wang et al. 2020). We also evaluate the

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of static sparse training methods on CIFAR-I0 and CIFAR-100. The number in brackets the relative training time speedup compared to a dense model.
Dataset
CIFAR-10
CIFAR- 100
Pruming ratio
90%
95%
90%
95%
ResNet-32
Dense: 94.80
Dense: 74.64
RU
92.81+0.19 (1.0x)
91.38 +0.04 (1.0x)
69.48-0.21 (1.0x)
67.03+40.66 (1.0x)
LTH Franklc Carbin 2019
92.31 (1.0x)
91.06 (1.0x)
68.99 (1.0x)
65.02 (1.0x)
SNIP eral 2019
92.59 (1.0x)
91.01 (1.0x)
68.89 (1.0x)
65.22 (1.0x)
GraSP Wang 2020
92.38 (1.0x)
91.39 (1.0x)
(9.24
66.50 (L.0))
RC
90.27.1..44 (1.5x)
87.1910.48 (1.6x)
62.71 (1.x)
56.33 10.37 (16x)
Regrouping Rumi et 2020
91.6340.11 (1.Ix)
90.77+0.07 (1.2x)
66.97-0.18 x1.1x)
64.35+0.45 45.25113.. (1.2x)
HRBP
91.76+0.17 (1.4x)
89.80=0.02 (1.xx)
67.20-0.28 (1.4))
63.1940.73 (16x)
HRBP+
92.3010.20 (1.4x)
90.84:0.41 (1.6x)
0.52 (1.4x)
65,9410.25 (1.6))
VGG-19
Donse: 94.23
Dense: 74.16
RU
92,9110,1101.000 (1.0x)
91.9110.13 (1.0x)
70.39 0.4((.0x)
68.63.1..4
ITH Frankle & Carhin 2019
93.51 (1.0x)
92.92 (1.0x)
72.78 (1.0x)
71.44 (1.0x)
SNIP Lze el 2019
(1.0x)
93.43 (1.0x)
72.84 (1.0x)
(1.0x)
GmSPP Wang et al 2020
93.30 (1.0x)
93.04 (1.0x)
(1.0x)
71,23 (1.0x)
RC
90.8410.38 (1.7x)
87,12=0.18 (1.8x)
59.61=0.52 (1.7x)
49.31 (L8X)
Regropiig Rumn al. 2020
92,81-10,25 (1.2%)
(..6..0.44
70.52-0.36 (1.2x)
68.6010.08 (1.2x)
HRBP
92.48-+0.19 (1 (1)x)
90.81=0.16
68 (1.4x)
64.98+00.47 (19x)
HRBP++
92.88
(1.66.)).44(1.x))
0.29 (1.4x)
67.8910.49 (1.9)
conventional regrouping method (Rumi et al. 2020) as a baseline, even if it fails to accelerate the backpropagation. Following (Wang et al. 2020), we set the sparsity ratio to 90% and 95%.
Results. The results are summarized in Table [ From the accuracy perspective, HRBP++ achieves similar performance to unstructured pruning methods and much better performance than channel-wise pruning. For example, on CIFAR-I0 with ResNet-32, HRBP++ achieves an accuracy of 90.84% 95% sparsity, which is 3.65% higher than the RC and just 0.17% negligible drop to SNIP (Lee et al. 2019). HRBP, as our baseline with more zero kernels, usually suffers 1 1% accu- racy drop compared HRBP++, but it still performs much better than RC. From the training time acceleration perspective, unstructured pruning cannot be accelerated due to its irregularity of non- zero weights. However, our HRBP can almost match the acceleration of channel-wise pruning, with the benefit of our optimized convolution implementation. Moreover, HRBP obtains more acceler- ation compared to the convention regrouping (Rumi al.] (2220). as the latter can only accelerate the forward pass. In summary, our HRBP(++) can successfully achieve comparable accuracy as unstructured pruning and bring training time acceleration on hardware.
Layer-wise speedup. We also explore the overall training time acceleration of each convolutional layer. We take VGG-19 with 90% overall sparsity CIFAR-10 10 example and report the speedup of each layer in Figure At shallow layers with a small sparsity ratio, channel-wise pruning achieves slightly better acceleration. However, deeper layers, HRBP achieves comparable or even better results, with at most 7.5x acceleration on Conv14 4. This is consistent with the conclusion in (Rumi al. 2020) that the grouping algorithm prefers large kernels and large sparsity. Noticeably, cuDNN is only optimized for kernel matrices with multiple of 32 rows, thus structured pruning with an arbitrary number of channels does not guarantee better acceleration (Rumi al. 2020).
4.3 DYNAMIC SPARSE TRAINING WITH HRBP++
We further explore the dynamic sparse training with HRBP++. which prunes and grows the con- nections based on the proposed block-wise updating during the training time. We compare DST with HRBP++ to several unstructured DST methods, including DeepR (Bellec et al. 2018). DSR (Mostafa & Wang 2019), SET (Mocanu et al., 2018), and Rigl. (Evci et al.) 2020). We follow all sparse training hyperparameters in (Evci et al. 2020). As the speedup similar to static sparse 2 simplicity. Noticeably, our HRBP++
training, we only show the accuracy comparison can still match the accuracy of unstructured DST methods. Nevertheless, when comparing Table and 2 the advantage of DST from static training in HRBP is not much that in unstructured pruning. One potential reason is that the mask updating mechanism of HRBP+ has a smaller mask diversity (Hubara et al. 2021) than the unstructured pruning.
7

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of dynamic sparse training methods on CIFAR-10 and CTFAR-I00
Network
ResNet-32
VOG-19
Doisset
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
Pruning ratio
90%
95%
90%
95%
90%
95%
90%
95%
Dasp-R Rollecetal
DSR (Mstna w. SET
92.97
91.61
69.83
68.20
90.86
72.31
71.98
92.30
90.76
69, 19.66
67.41
92.46
91.73
72.36
89.81
Rigl. [5 45 2020
92.44:0.13
92.02290229
70.98:00300
68.50011.15
93.15:0000
92 12.30:00.00 48
71.6320,28
69.13490.6
DST-HIBBP-+
91.72.0.25
91.25110.12
69.51
66,4110.30
93.07..0.12
91.79.00:18
70.49.0.19
68.01.1..57
Table 3: Static sparse training ImageNet. The dense ResNet-50 has 75.70% top-1 accuracy.
Pruning ratio
60%
30%
Accurscy
top-1
top-5
top-1
top-5
Lee
(1.0x)
69.87 (1.8x)
GruSp CruSPwwing
M020
(1,06)
(1.06)
HRBP++
(1.173
92.35
(1.26X)
89.93
93.0
Channdlwiie HRBP
92.5
Dense
92.0
Accurcyy
Speeduu
91.0
90.5
90.0
Network Sparsity: 10% Sparsity:
89.5
89.0
0.5 05 1.0
Kernel Sparse Ratio
Cony (51%) (33%) (166%) (8%) Cany (8%)
Figure 6: Different kernel dense ra- tio for HRBP+ with ResNett-2..
Figure 5: The layer-wise speedup of VGG-19.
4.4 RESULTS IMAGENET
We evaluate the static sparse training with HRBP(++) on ImageNet. Follow Lee et al. (2019); Wang et al. (2020). set the sparse ratio 60% and 80%. As shown in Table 3 HRBP+ can also achieve similar performance as well-designed unstructured pruning methods. Mcanwhile, provides 1.26x and 1.44x training time acceleration hardware when the pruning ratio is 80% and 90% respectively. To this end, our HRBP++ is still effective on complicated tasks like ImageNet.
RocNa-ti
Dan
Dit
3XB
5817
Ratina Seacasdiraki
55
16
0837
Sraczudd
1K3
Spanily
Spunity
Spanity
(a) VGG-19
(b) ResNet-32
(c) ResNet-56
Figure 7: The trade-off between sparsity and accuracy. All models evaluated on CIFAR-10.
4.5 ABLATION STUDIES
Dense Ratio of Kernel Pattern We take the ResNet-32 on CIFAR-10 as example to explore the effect of the kernel sparse ratio s in HRBP+ Typically, a smaller introduces more non-zero kernels but inevitably reduces the number of non-zero weights within each kernel. As the shape of most kernels x 3. we explore s from 1 to 3. As shown in Fig. 6. a smaller s usually brings slightly better accuracy, especially for networks with large sparsity. This is reasonable since
8

---

Under review as a conference paper at ICLR 2023
smaller can reduce the number of zero kernels, while many zero kernels can render network untrainable (Tanaka et al. 2020).
Sparsity and deeper nets. We also vary the sparsity from 50% to 95% to explore the trade-off between sparsity and accuracy. Besides, we also add experiments with deeper nets such as ResNet- 56. As in Figure(7] our method can achieve comparable accuracy as unstructured pruning methods at all sparsity levels and all kinds of networks. This suggests the generalization ability of our method. For the acceleration rate on both training and inference, please see Figure 14 in Appendix.
RELATED WORK
5.1 NETWORK PRUNING
Pruning aims to compress overparameterized networks into lightweight ones. Based on the dis- tribution of zero weights, can be divided into three types: 1) Unstructured pruning, where zero weights are distributed at arbitrary locations based on the importance score of each weight. The
score can be obtained from magnitude (LeCun al.. 1990 Han et al. 201 by Xu et al. 202T
Zhu & Gupta 2017), gradient (Molchanov et al.) 2017 1 2019 or Hessian (LeCun et al.) 1990p. The unstructured pruning can match the accuracy of dense networks a high sparsity ratio (more than 90%). However, it is difficult to achieve better hardware speedup due to irregularity. 2) Struc- tured pruning, where the weights of entire channels are pruned. Earlier works (Li et al. [2016] (He) al. 2017: Wen et al. 2016 Liu et al. 2017) adopt mathematics-oriee gorithms to generate sparsity. Other works such as HRank (Lin et al. 2020 SCOP (Tang et al.) 2020). DMCP (Guo et al. 2020 MetaPruning (Liu et al. 2019) use complicated rules to generate the sparsity distribution in the channel level. As the sparse model preserves the spatial regularity, the pruned convolution layers can be transformed size and accelerate computation on the hardware level. However, structured pruning suffers from significant accuracy loss as one entire activation map be zero. 3) fine-grained strucfured pran- ing, which includes block-based pruning and pattern-based pruning. In block- pruning (Rumi al. 2020), the unstructured weights are partitioned into several dense blocks. However, all the above works aim to accelerate the inference stage, our work differs substantially from them such that we do not rely on a pre-trained dense model.
regularization- -based al-
full matrix multiplication with reduced matrix
5.2 SPARSE TRAINING
Sparse training aims to train a sparse network from scratch. Based on the mask updating schemes, is usually divided into two categories: 1) Static sparse training, where the sparse mask is obtained at the early stage of training and is fixed during the course of training. Previous works obtain the
sparse mask by random pruning utilizing some saliency criteria. The SNIP (Lee al. 2019)
uses the gradients of the training loss as connection sensitivity to prune the network at initialization
Later on, many criteria have been proposed, such as gradient flow in GraSP (Wang et al. 2020). synaptic strengths in SynFlow (Tanaka et al.] 2020), Fisher information (Sung et al.] 202 etc. 2) Dynamic sparse training, which starts from a random sparse mask and dynamically updates connections during training. DST avoids the pruning-at-inice at-intiializatoonn process that usually involves the full dense model computation. Specifically, SET (Mocanu et al. 2018) update sparse masks by pruning weights that have the least magnitude and grow back the same amount of inactivated weights in a random fashion. RigL (Evci et al. 2020) proposes to update sparse mask by magnitude- pruning and grow back inactivated weights by their gradients. DSR (Mostafa & Wang 2019) and STR (Kusupati et al., (020) design a dynamic reparameterization method that allows weights to be re re-distributed across layers by providing global sparsity allocation dynamics. DeepR (Bellec et al., 2018) combines dynamic sparse parameterization with stochastic parameter updates for training, but primarily targets small and shallow fully-connected networks. However, these works focus on unstructured sparsity.
CONCLUSION
We analyze the implementation of CNN with GEMM and point out that current fine-grained struc- tured pruning methods cannot promise backward acceleration. We then propose the HRBP, aiming to accelerate the sparse training at both forward and backward passes by extracting dense blocks from the unstructured mask. Extensive results suggest that sparse training with HRBP(++) can achieve comparable accuracy with unstructured pruning methods. Meanwhile, it brings significant training acceleration. We believe our method can integrate the advantage of both unstructured pruning and structured pruning and make sparse training more frreware-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep net-works. International Conference on Learning Representations (ICLR), 2018.
Aydin Buluç, Jeremy T Fineman, Matteo Frigo, John R Gilbert, and Charles E Leiserson. Parallel sparse matrix-vector and matrix-transpose-vettee multiplication using compressed sparse blocks. In Proceedings of the twenty-first annual symposium on Parallelism in algorithms and architec- fures, pp. 233-244, 2009.
Long Chen, Oreste Villa, Sriram Krishnamoorthy, and Guang R Gao. Dynamic load balancing on single-and multi gpu systems. In 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS), pp. 1-12. IEEE, 2010.
Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. ICML, 2022.
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catan- zaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint 1rXiv:1410.0759, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2952. PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2019.
Yanwei Fu, Chen Liu, Donghao Li, Xinwei Sun, Jinshan Zeng, and Yuan Yao. Dessilbi: Exploring structural sparsity of deep networks via differential inclusion paths. In International Conference on Machine Learning, pp. 3315-3326. PMLR, 2020.
Yanwei Fu, Chen Liu, Donghao Li, Zuyuan Zhong, Xinwei Sun, Jinshan Zeng, and Yuan Yao. Exploring structural sparsity of deep networks via inverse scale spaces, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Shaopeng Guo, Yujie Wang, Quanquan Li, and Junjie Yan. DMCP: Differentiable Markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1539-1547, 2020.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, arXiv preprint arXiv: 100.00044, 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015b.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. Proceedings of the IEEE International Conference on Computer Vision, pp. 1389-1397, 2017.
10

---

Under review as a conference paper at ICLR 2023
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accel- erated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in Neural Information Processing Systems, 34:21099 21119,21111,,1111111 2021.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference on Machine Learning (ICML), pp. 5544-5555, 2020.
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hub- bard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551 1989.
Yann LeCun, John S Denker, and Sara Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-6598--555 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based connection sensitivity. In International Conference on Learning Representations, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 2016.
Minghao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and I Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1529-1538, 2020.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense overrparameterizzaton? in-time over-parameteriizaion in sparse training, In International Conference on Machine Learning, pp. 6989-7000. PMLR, 2021.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3296-3305, 2019.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang. Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp. 7734, 2017.
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William Dally. Exploring the regularity of sparse structure in convolutional neural networks. arXiv preprint arXiv:1705.08922, 2017.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):1-12, 2018
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pp. 2498-2507. PMLR, 2017.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1264-112722 2019.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In Proceedings of the International Conference Ma- chine Learning (ICML), pp. 4646 4655, 2019.
Masuma Akter Rumi, Xiaolong Ma. Yanzhi Wang, and Peng Jiang. Accelerating sparse enn infer- ence on gpus with performance-aware weight pruning. In Proceedings of the ACM International Conference Parallel Architectures and Compilation Techniques, pp. 267-278, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.

---

Under review as a conference paper at ICLR 2023
Wei Sun, Aojun Zhou, Sander Stuijk, Rob Wijnhoven, Andrew O Nelson, Henk Corporaal, et al. Dominosearch: Find layer-wise fine- ime-grained n: M sparse schemes from dense neural networks. Advances in Neural Information Processing Systems, 34:20721 2021.
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, 2021. URL http:: hatpps://openreviiwws net www.nneetooutttooo forum?idt
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Pro- cessing Systems, 33:6377-6338 2020.
Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376 2020.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Advances in neural information processing systems, pp. 2074-2082, 2016.
Dongkuan Xu, Ian EH Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning-unde under the pre-train and fine-tune paradigm. NAACL, 2021.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-economil sparse training framework on the edge. Advances in Neural Information Processing Systems, 34, 2021.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning n:m fine-grained structured sparse neural networks from scratch. In Interna- tional Conference on Learning Representationss 2021. URL htttps://ppereeviee net/ Eorum?id=K9bx 7vqp.
Maohua Zhu, Tao Zhang, Zhenyu Gu, and Yuan Xie. Sparse tensor core: Algorithm and hardware co-design for vector-wise sparse neural networks on modern gpus. In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pp. 359-371, 2019.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv: 171.01771..0188. 2017.
12

---

Under review as a conference paper at ICLR 2023
A FORWARD AND BACKWARD PASS IN NCHW AND NHWC LAYOUT
NCHW and NHWC data layout formats are two common types of cuDNN tensors arrangement in memory (Chetlur al. 2014). These two layouts produce the same shape matrix and the same outputs. The only difference is the order element in the flattened tensor, as shown below:
NCHW layout As shown in Fig. B(a), in forward pass, the flattened tensor W begins with the first imput channel (green color), the clements arranged contiguously in row-major order (i.e., a0, al, a2, a3 with green color for the kernel K forward). Then, continues with second (orange color) and subsequent channels until the elements of all the channels laid out.
NHWC layout As shown in Fig. c), in forward pass, the flattened tensor W begins with the first element of the first imput channel (i.e., a0 with green color for K), then proceed the first element of the second imput channel (i.e., a0 with orange color for K), and on, until the first elements of all the C' channels are laid out. Next, select the second element of the first input channel (i.e., a1 with green color for K), then proceed the second element of the second input channel (f.e., al with orange color for K), and so on, until the second element of all the channels are laid out.
X-im2col
dY (Y-im2col(dO)
do
Input
im2col
im2col
84
15
13
15
14
d0
dOg I....
14
Kernels K
a0
Kermels K'
K0
rolation
hv
transpose)
6O
B0 bi 60
reshape
W
X
reshape
dX W d'
Ni bi
IC
Ni
02
W
X1
M
(a) Forward pass with NCHW layout
Backward pass with NCHW layout
X=im2col (I)
dO
Input
im2col
im22oll
84
15
E3 34
13
15
12
16
X
d0
d0O I...
14
16
Kernels
K
all
Iv]
Kermess K"
K0
b3
rotation
transpose)
M
15
reshape
W
X
reshape
dX
W
dY
XC
OL
Dc
W
x1
M
(c) Forward pass with NHWC layout
(d) Backward pass NHWC layout
Figure Illustration GeMM with NCHW and NEWC format. Different colors represent different channels.
B N:M TRANSPOSABLE MASK IN THE BACKWARD OF CNN
As suggested in Zhou et al. 202I. , the N:M sparsity The N:M transposable mask (Hubara et al. 2021 guarantees that both the weights matrix W and its transpose W follow the same sparsity
13

---

Under review as a conference paper at ICLR 2023
pattern. This design works well for linear layer. whose forward pass is calculated with Y WX and backward pass is calculated with dX- = WWTdY, However, the implementation of CNN with GEMM different. Specifically, we have W E [RCox(C for the weights matrix of CNN. In the backward pass, CNN flips each kernel first vertically and then horizontally and performs the kernel-wise transpose. Thus, as in Fig. [c), W
Ku) is different from the matrix W E RR (Co Kw in the backward pass of CNN in most cases. Nevertheless, this N:M transposable mask can still accelerate the CNN backward under some conditions.
G = 4, Co
4,
K
K
al
92
26
D4
a5
b5
C5 04
06
A7
57
aB
b2
c2
02
bo
b
b2
50
67
6:
60
b1
62
50
h2
88
x7
66
bs
bG
c5
C7 CG
d'
66
54
14
b:
63
b4
65
a5
155
b4
b5
de
43
N7
a2
n0
b0
CO
40
c5
Rotation & kernel-wise transpose
de
c7
c9
:0
b0
d1 40
.1 d2
d0 d1 42
A7 a6
68 c7 e6
d6
d5
d5
a3
K
K
c2
reshape
reshape
A
B
C
D A
B C D A B
C
D
A
B
C
D
11
41
30
Jki
d0
41
d1
41
d8
d8
4%
68
68
406)
41
WM
NO
(a) NHWC layout forward
(b) NHWC layout backward
B
33
15
a0
a8
W7
a6
50
cD
JB
B
b0
b
b2
63
84
h5 16
68
b0
68
140
68
10
68
Co
38
a6
33
a0
bS
50
CD
JB
c0
cl
c2
c5
44
06
05%
611
08
A8
B7
a6
04
D
a0
BS
c0
d8
D
40
d2
43
45
dG
AR
48
k7 a6
K,K,
K,K,,
(c) NCHW layout forward
(d) NCHW layout backward
Figure 9: Comparison ofNHWC and NCHW layouts of N:M mask in both forward and backward pass.
Layout. The first requirement is the layout of the weights in the memory. For example, suppose N = 2, M = 4, the size of the input channel is C₁ = 4 and the size of the output channel is Co = 4. Thus, we can divide the kernel matrix W E [x4((x333)) into several x 4 blocks. As shown in Fig. (a)),), when we store kernels with NHWC format, we collect a0 with green color, a0 with orange color, a0 with blue color, and a0 with red color into a row of a block (i.e., the blue box) for the forward pass, and this row meets the N:M constraint. Based on the N:M transposable mask, the column of a0, 60, c0, and d0 with green color also satisfies the N:M constraint. In the backward pass, this blue block can still be kept. Meanwhile, a0, 60, cO, and d0 with green color is collected as a row. Thus, the same sparse pattern can be maintained in the backward with N:M transposable design. When the weights is stored with NCHW format, as shown in Fig. (d),,,, the green cell a0, al, a2, and a3 form a row of the blue block. Again, the column of a0, 50, c0, and d0 with green color also satisfies the N:M constraint. However, in the backward pass, the same block cannot be kept. Thus, the sparse pattern may not meet the N:M constraint. For example, in forward pass, green cells a2 and a3 are kept, and the green cells a4 and a7 are kept. In the backward, the green cell al, a2, a3, and a4 form new vector with the size M 4, but this vector has three cells a2, a3, and a4 with mask "1". To this end, additional operations such as re- re-indexing and regrouping are required to make the weight matrix in the backward meet the N:M constraint. In summary, the N:M transposable mask can accelerate the CNN backward pass with NHWC layout GEMM implementation, but it cannot be directly applied to the NCHW layout.
14

---

Under review as a conference paper at ICLR 2023
A
B
C
4
3,
K
K
2
60
80
81
80
81
80
B1 A3
82
+3
A2
83
82
A3
n2
b2
60
bl
60
61
60 b1
60 b1
b1
C
c0
Rotation
00
c1
60 cl
60) cl
K
kernel-wise transpose
e2 G3
C2 033
c2 c3
c2 c3
K
b1
reshape
reshape
C
B
C
A
B
C
aD
30
40
80
21
11
A2
R2
83
n3
K3
63
h2 62
a
b1
30
00
00
b0
bd
60
60
61
61
62
b2
62
20
50
02
W
W
(a) NHWC layout forward
(b) NHWC layout backward
Figure 10: Example of N:M transposable mask when Co mod * O.
Shape. Moreover, even if the weights are arranged in NHWC layout, we show some cases that the N:M transposable mask may not bring the acceleration backward well. Considering the example with Co = 3 and = 2, as shown 2 block (i.e., the blue box) for the forward pass, and this meets
Fig. TO we collect a0 with green color, o0 with
orange color into row of the N:M constraint. Based on the N:M transposable mask, the column of a0, 60 with green color also satisfies the N:M constraint. However, the backward, the a0 is grouped with cl, and the 60 is grouped with c0. Thus, the same sparsity pattern cannot be guaranteed. To this end, the output channel Co should be divisible by M, i.e., Co mod M
15

---

Under review as a conference paper at ICLR 2023
C ALGORITHM SUMMARY OF HRBP/HRBP++
In this section, we summarize the detailed implementation of HRBP/HRBP++ below:
Algorithm 1: HRBP/ HRBP++
unstructured mask density ratio of mask number of clusters t, density ratio of kernel s, number of input channel Cz. the number of output channel Co. the width the kernel Kw, the height of the kernel Kg, and the minimum number of rows of each group B1
Input:
Output: Dense groups
Obtain kernel-wise mask m'' based on the number of non-zero cells within each kernel of m: Divide the rows in mś' into i equal-shape groups {91, 92, 9r3 with hypergraph partitioning; for Di E {91 92. do if 9, has no less than B1 rows then Sort columns of Ji from high low based on the number of non-zen con-zero cells of each column: Select the C columns with the most number of non-zero; Extract the corresponding kernels based on selected columns in Jr if then
Count the non-zero weights of each cell based on all kernels in K:; // HRBP++ Select Kw cells with the most number of non-zero weights as the pattern P: Apply pattern P all kernels IC and output them as a dense group;
else
Output all kernels I. with all K2 K cells as a dense group; HRBP end if
end if' end for
16

---

Under review as a conference paper at ICLR 2023
D NOTATIONS
In this section, summarize all mathematical notations in this paper to help understand the moti- vation and the operation clearly.
Table 4: Summary of notations in this paper.
Notation
Range
Definition
Equation
C₁
Number input channels
Co
4
Number of output chunnels
K.
3
Kernel height
Kw
N+
Kernel width
II,
24
Height input 2D feature map
Wr
34
Width of input 2D feature map
Ilo
N+
Height of output 2D feature mup
Wo
N
Width of output 2D feature map
C
S+
Number input channels simplicity form
Cj C2 Ky x K.
K
Weights of 2D convolutional layer
K.
X
One filter
K'
RIICOOX,xX,
2D convolutional weights in hackward pass 21) input feature mup
T
ROIH,
O
RoOMOOWO
2D output feuture mup
dl RiXHIW,
Gradients w.r.. the 2D input feature mup
dO
RoxMooхW XO
Gradients w.r.. the 2D output feature map Gradients w.r.t the kernels
dK
(OXGGN
Convolutional weights GeMM (weights matrix) Transpose weights matrix
W reshape(K)
WT
W
((X(OOAA
Convolutional weights of backward pass GeMM
W reshape(K))
X
Input feature mup in GeMM Output feature map in GeMM Gradients w.r.. the output feature map in GeMM Gradients w.E.. the Convolutional weights in GeMM Gradients the input feature map GeMM
im2col(I)
Y
[oo(HooWo
Y WX
dY
((0
dY 1m2col(dO)
dW RooHHoXWo
dw dY -X
dx RCX(MWW))
dX W
m
(CONICIN,KK.)
binary mask weights matrix W Bernel-wise mask
m'
RCOXC,
U.
convolutional kernels number of groups (dense blocks)
t
S+
(s/(K K.
ratio each kernel with size Ki
E MATHEMATICAL ANALYSIS OF CNN BACKWARD PROPAGATION
In this section, we give a detailed mathematical analysis the CNN backward pass to show the reason for the transformation in the calculation of gradient w.r.t. the input. For simplicity, assume that the number of input channels C₁ is 1, the number of output channels Co the stride is 1, and there is padding operation in the forward pass. Thus, given the input X "I and the filter E IR K1 Kw. in the forward pass of CNN, the output O E MoxWoo is calculated by:
Ky
K..
Oi,j
Km. Xitmgias
mum
where 1 1ii5 Ho and j Wo. As there is padding operation and the stride is 1, have Ho 11, - Kr, + and Wo W₁ Kw + 1. In the backward pass, given the gradients of the 0L
output d0
based on the chain rule, the gradient w.rtt. each input pixel is calculated by:
aL
Ho Wo
AL 00.5
Σ 1=1
30
0X.5
1=1
17

---

Under review as a conference paper at ICLR 2023
where I S Hr, and s I Wr, and d0i, can be obtained by taking the difference on aX..b
aL Thus, when input pixel X., contributes to the output pixel Oz a0
both sides of Equation
OL In practice, Equation 2 can be represented as full convolution contributes to the gradient of aXa,b
between 180-degree rotated filter K' and the gradient on the output.
Considering a specific example where the imput has shape H₃ Wi and the kernel has size K₁ = K = 2, as shown in Figure 11 For simplicity, we set CI Co = 1. In the forward, based
on Equation we have:
O to ag
01
13 02 4 a3
0 * ag
11 01
4 02 I5 a3
O₂ I3 * og
4 01
18
02 1 a3
03 = 1 * ag
Is 01
1, 02 1s a3
80u
80g
d0,
Take On an example, the gradient w.r.t. To. J1, 13, and 7
OI,
do,
Ø11
01
0Ig
02
80p
and
0lg = ag respectively. Based on Equation
we can obtain the gradient of each input pixel:
0L aL.
* dg
O1, 80p
0L AL
ØL
10 00p
* dD
+
00,
0L OL
01, 801 #0
OL aL
0L
01 80,
d2
002 dg
OL OL
0L
öL
0L
0L 80p
03
001
02 +
00₂
01
00g
00
OL AL
aL
01s 801
03
803 * #1
OL AL
0 002
* 02
OL aL.
ØL
01 802
03
003
02
OL 0L
03
0T. 00,
As in Figure we can perform a convolution operation between the 180-degree rotated kernels and ØL
the gradient w.r.t output 80 with zero padding. Note that, zero padding on the output is necessary as we need to ensure the product of this full convolution has the same shape as the input.
F RESULTS ON TRANSFORMERS
Recently, the VIT (Dosovitskyy et al. 2021) shows that transformers (Vaswani et al., 2017) also play important role in computer vision tasks. The transformer encoder contains self-attention module and ML.P layer. Given input sequence X, the self-attention applies three linear transformations
18

---

Under review as a conference paper at ICLR 2023
Input X
Kernels K
Output
10
XI
I2
A2
16
18
Kemels K (180" rotation)
Gradient oulput
200
dil
612
padding
aon 2011
405
x2
al
aD
Gradient w.c.t. input
Figure 11: Illustration of the full convolution between the rotated kernels K'. For simplicity, we use d0 to 0L 80
represent
to obtain the query Q Wq the key K WKX, and the value V W,X, respectively. Then
QKT
the output obtained by Y Softmax(
VV, where d is the embedding dimension. Thus, all
calculations of transformers with learnable weights are linear projections. To this end, the general form dX w Y for the gradient w.r.t. the input is suitable for transformers in the backward pass and we can keep the same dense blocks at both forward and backward pass for sparse training of transformers.
Although the transformer does not encoder the shape transformation issues like CNN, we further apply our method on DeiT-Tiny (Dosovitskiy al.) 202T Touvron et al.] [02T) show that the dense-block-baag -based method is also effective in the training of transformers. The dense model of DeiT-Tiny can achieve 72.2% accuracy on ImageNet. With HRBP, we can achieve 72.7% accuracy with 60% sparsity.
G COMPARISON WITH THE GROUP LASSO PENALTY
The DessiLBI (Fu et al. 2020 2022) proposes use the group lasso penalty each convolutional filter to prune less important filters. When first training a dense network, DessiLBI can find effective subnetworks (i.e., "winning tickets" Frankle & Carbin 2019) at different epochs of the dense train- ing (i.e., early stopping). By retraining from scratch, the structural sparsity can achieve comparable performance as dense networks. We follow the setting in DessiLBI and apply HRBP+ to VGG- 16 on CIFAR-10. We compare DessiL.Bl with Lasso since baseline DessiL.B LBI with group lasso penalty ineffective extreme sparsity level. The results are shown in Figure [2] Our method can also achieve comparable or even better accuracy than DessiLBI at the same sparsity level. Note that, DessiLBI follows the LTH setting which still needs dense training at the first step, while our method directly trains the sparse subnetwork from scratch without dense training.
H ABLATION STUDIES ON INITIALIZATIONS.
We by default apply our method to random unstructured sparse mask with ERK sparse ratio distri- bution (Evei et al. (020) of each layer. In this ablation study, we explore the effect of different types of unstructured sparse mask initialization. In detail, we compare the default setting with 1) Random unstructured sparse mask with uniform sparse ratio distribution. 2) Unstructured mask from SNIP (Lee et al.) 2019). 3) Unstructured mask from GraSP (Wang et al. 2020). We then apply HRBP++ to these unstructured masks and show the results Figure 13) Noticeably, with SNIP and GraSP masks, HRBP++ can also achieve comparable accuracy at the same sparsity level. With uniform sparsity, the accuracy is slightly lower. This consistent with the conclusion in (Evci et al. 2020) that ERK sparsity is beneficial. Thus, HRBP+ is robust with different types of mask initialization.
19

---

Under review as a conference paper at ICLR 2023
Comparison between HRBP-- and ((Laso)
94.5
94.0
93.5
Accuracy
93.0
(Stati))
DessiDDI Epoch
92.5
DessiL Epoch
DessiLBB Epoch
92.0
Dessil BII Epoch
DessiLBI Epoch 32
91.5
DessiLBI Epoch 64
DessiLBI Epoch 128
91.0
60
70
80
Sparsity
Figure 12: Comparison between HRBP and DessiLBI.
Ablation study on the initialization of mask
96
94
Accurey
92
90
Dense
HRBP++ with SNIP mask
HRBP++ with uniform random mask
88
HRBP++ with GraSP mask
HRBP++ with ERK random mask (default)
86
50
60
70
80
90
Sparsity
Figure 13: Ablation study with different types of mask initialization of ResNet-32 on CIFAR-10.
Acceleration rate on training and inference with different sparsity ratios
Training
Testing
Acceleration rate
2.0
1.5
1.0
0.5
50
60
70
80
90
Sparsity
Figure 14: Training and inference acceleration on hardware of ResNet-32 on CIFAR-10.
20


Reviews:
[{"review_id": "r-sQPporVYC", "reviewer": "Reviewer_WyN2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the hardware-friendly regrouping in order to make structured sparsity. It aims at brining the training acceleration on GPU. The algorithm is conducted  on the kernel-wise mask. Some experiments are validated the show the efficacy. ", "summary_of_the_review": "Please clarify my questions about the weakness.", "strength_and_weaknesses": "Strength:\n\n1, generally this paper is well written,\n2, the presented method is interesting.\n\nweakness:\n\n1, I had some doubts whether the compared baselines are reasonable. Why not directly using the group lasso penalty on the each convolutional filters like [r1,r2]. This method should be the naive way of pruning the filters.\n[r1]Exploring Structural Sparsity of Deep Networks via Inverse Scale Spaces, TPAMI 2022\n[r2] DessiLBI: Exploring Structural Sparsity on Deep Network via Differential Inclusion Paths. ICML2020\nThese should be compared, as group lass is exploring the structure sparsity of CNN filters.\nI think the method of these papers is pretty strong competitor to the proposed method in this paper.\n\n2, Page 4, HRBP paragraph, the third line: \"weights, which is obtained by counting Thus, the\". It seems missing something here?\n\n3, As transformers make very good performance in many task, is it possible to apply the algorithm to transformer?  CNNs are a bit out of date (less attractive).\n\n4, As this paper advocates the hardware-friendly algorithm. It should give some more implementation details: is the HRBP/HRBP++ implemented by cuda? How's the implementation of baselines in Fig. 5? Is the comparison apple-to-apple? Are the authours using C++ or Python to run the baselines in Fig. 5? \n\n5. Fig.6 should be results from HRBP++. However the caption refers to HRBP instead.\n6. It would be better if the authors could have some discussion about the practical usage of the proposed method. It seems that it is not so worth it  having only 2 times faster speed while dropping almost all of the original parameters and suffering from significant performance drop.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FW-JAkuNOlh", "reviewer": "Reviewer_oxCu", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the Hardware-friendly regrouping towards block-based pruning (HRBP) method, and the HRBP++ method. It tests the method on CIFAR-10, CIFAR-100 and ImageNet dataset. ", "summary_of_the_review": "This paper presents the Hardware-friendly regrouping towards block-based pruning (HRBP) method, and the HRBP++ method. It tests the method on CIFAR-10, CIFAR-100 and ImageNet dataset. The paper can be further improved by clarifying the notations, and clarifying the propose method impact on data structure, compression rate and accuracy. It is also better to illustrate the initialization impact on the proposed method. ", "strength_and_weaknesses": "Strength: \n1. The network pruning is an important topic, and the author has given sufficient background introduction.\n2. The paper illustrate the implementation flow chart of HRBP and HRBP++ clearly.\n\nWeaknesses:\n1. There are spelling errors in the paper.\n2. The paper mix up the programming languages and the math notations. It makes the presented methods difficult to follow. The definitions of symbols and math notations are missing.\n3. The advantages of using the presented methods are unclear. What is the compression rate by using the methods? How is the impact on the accuracy? What is the current state of the art? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3JlYYndywG", "reviewer": "Reviewer_fvk3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to accelerate the training workflow of convolution operators through sparsity. While recent work has explored structured/semi-unstructured sparsity to accelerate inference, this work targets training. The main challenge in leveraging sparsity in both forward and backward pass is crafting a sparsity mask that is transposable (or in the case of convolution the sparsity patterns are preserved after reforming the kernels). To achieve that, the paper suggests to apply sparsity mask in a kernel-wise manner, which arguable the main difference between this work and Rumi et al.). Leveraging this form of sparsity enables to preserve a hardware-friendly sparsity pattern for both forward and backward pass unlocking the opportunity to accelerate convolutional neural network training.", "summary_of_the_review": "The paper tackles an interesting problem, accelerating sparse training. However, the domain and target workloads seem to be very narrow and not generally applicable to different neural network operations and workloads (or at least is not supported by the results). Even for the models covered in the paper, they are arguable old models and dataset, which still valuable, but raises the concerns of application of such method to a broad range of applications.", "strength_and_weaknesses": "**Strength**\n\n- Unifying sparsity for both forward and backward pass that enables better utilization of existing hardware platforms for acceleration.\n\n- Results on few convolutional neural network shows promising results both in terms of accelerating training and preserving model accuracy, comparable to unstructured sparsity accuracy.\n\n**Weaknesses**\n\n- The paper lacks discussion on how the proposed form of sparsity can be extended or applied to other (convolution/non-convolution) operations.\n\n- The paper did not cover (or discuss the limitations) other form of neural networks from prior work (recurrent neural network in SNIP or deeper ResNet models).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0B6zKlXXCi", "reviewer": "Reviewer_5xYn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first claimed that element arrangement in forward's feature/parameter multiplication is different from the counterpart in backward, which leads to the inefficiency in previous sparse training acceleration method. Based on this observation, the paper proposed a block grouping method to aggregate non-zero elements in feature/parameter.", "summary_of_the_review": "The submission take effort to visualize the idea and make itself understood but it fails on me. Hope I can be clarified in the following discussion. ", "strength_and_weaknesses": "Weaknesses:\n1. From my point of view, the most important matter of the submission is whether $dW=dY  X^\\top$ can be applied in backward directly. Though it is claimed and visualized in the content (for example, Figure 1) that the resulting $W^{'}$ is not $W^\\top$, I have conducted gradient derivation by manually multiplying gradient of output and activation and my experience implied that dW can be arrived by matrix multiplication between dY and X. My past work may not be fully solid. However, one very obvious question will be: why padding is necessary in backward ? In another words, I am still confused about the submission's foundational motivation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject