Dynamic Sparsity Is Channel-Level Sparsity Learner
Authors: Lu Yin
Gen Li
Meng Fang
Li Shen
Tianjin Huang
Zhangyang Wang
Vlado Menkovski
Xiaolong Ma
Mykola Pechenizkiy
Shiwei Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: dynamic sparsity
dynamic sparse training
sparse training
TL;DR: We propose a new method dubbed Chase, which can effectively transfer the promise of unstructured sparse training into the hardware-friendly channel sparsity with comparable or even better performance on common GPU device.

Abstract: Sparse training has received an upsurging interest in machine learning due to its tantalizing saving potential for both the entire training process as well as the inference. Dynamic sparse training (DST) as a leading approach can train deep neural networks at high sparsity from scratch to match the performance of their dense counterparts. However, most if not all DST prior arts demonstrate their effectiveness on unstructured sparsity with highly irregular sparse patterns, which receives limited support in common hardware. This limitation hinders the usage of DST in practice. In this paper, we propose Channel-aware dynamic sparse (Chase), that for the first time seamlessly translates the promise of unstructured dynamic sparsity to GPU-friendly channel-level sparsity (not fine-grained N:M or group sparsity) during one end-to-end training process, without any ad-hoc operations. The resulting small sparse networks can be directly accelerated by commodity hardware, without using any particularly sparsity-aware hardware accelerators. This appealing outcome is partially motivated by a hidden phenomenon of dynamic sparsity: off-the-shelf unstructured DST implicitly involves biased parameter reallocation across channels, with a large fraction of channels (up to 60%) being sparser than others. By progressively identifying and removing these channels during training, our approach transfers unstructured sparsity to channel-wise sparsity.  Our experimental results demonstrate that Chase achieves 1.7x inference throughput speedup on common GPU devices without compromising accuracy with ResNet-50 on ImageNet. We release our code in https://github.com/luuyin/chase.

Dynamic Sparsity Is Channel-Level Sparsity Learner
Lu Yin', Gen Li", Meng Fang", Li Shen', Tianjin Huang', Zhangyang Wang', Vlado Menkovski', Xiaolong Ma², Mykola Pechenizkiy', Shiwei Liu Eindhoven University Technology, ?Clemson University University of Liverpool, 'JD Explore Academy, 'University of Texas at Austin, f1. yin, t. hwanguuggguuu m. pw.peeeenizkk,, v. menkovski, s 1iu3)@tue nl genög - clemson. edu, xiaolom@clemson. edu, Fangel Faan@eliverpprn iveeeool..ppol..oo. ac..poool....... sathshenli@gmaile com, atlaswangdutexas.. edu
Abstract
Sparse training has received upsurging interest in machine learning due its tantalizing saving potential for the entire training process as well as inference. Dynamic sparse training (DST), leading sparse training approach, can train deep neural networks at high sparsity from scratch to match the performance of their dense counterparts. However, most if not all DST prior arts demonstrate their effectiveness on unstructured sparsity with highly irregular sparse patterns, which receives limited support in common hardware. This limitation hinders the usage of DST in practice. In this paper, we propose Channel-awe -wwrr dynamic sparse (Chase), which for the first time seamlessly translates the promise of unstructured dynamic sparsity to GPU-friendly channel-level sparsity (not fine-grat fine-grained N:M or group sparsity) during one end-to-end training process, without any ad-hoe operations. The resulting small sparse networks can be directly accelerated by commodity hardware, without using any particularly sparsity-aware hardware accelerators. This appealing outcome is partially motivated by hidden phenomenon of dynamic sparsity: off the-shelf unstructured DST implicitly involves biased parameter reallo- cation across channels, with large fraction of channels (up to 60%) being sparser than others. By progressively identifying and removing these channels during training, approach translates unstructured sparsity channel-wise sparsity. Our experimental results demonstrate that Chase achieves 1.7x inference throughput speedup common GPU devices without compromising accuracy with ResNet-50 on ImageNet. We release our codes in https: //ggttutb com cwwccomm////////eccc luuyin/ chase.
Introduction
Deep neural networks (DNNs) have recently demonstrated impressive breakthroughs with increasing scales [2; 8; 36). Besides the well-known scaling, i.e., test accuracy scales as power law regarding model size and training data size in quantity [18; 26], recent work has observed that massive increases in quantity can imbue models with qualitatively new behavior [49]. However, the memory and computation required to train and deploy these large models can be heavy burden on the environment and finance [13; 43]. Therefore, people start to probe possibility of training sparse neural networks from scratch without involving any dense training steps (dubbed sparse training [39; 35]). As the memory requirement and multiplications (which dominate neural network computation) associated with zero weights can be skipped, sparse training is becoming promising direction due to their "end-to-end" saving potentials for both efficient training and efficient inference.
Sparse training can be categorized into two groups, static sparse training and dynamic sparse training according to the dynamics of the sparse pattern during training. Static sparse training
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

75
75
74
Top-1 Avernay (5)
73
Top-1 Accuaey (5)
Choor
Clase
69
KET
KTT
INFS
SNTS
68
MIRATHEA
ACESTHEM
67
Chass
X1.3
20.9
22.5 25.0
21. 30.0 32.5 35,0
80
90
100
110
120
Inforence Latency
Throughput
Figure 1: Inference latency and throughput of various DST methods. The sparsity level is 90% for all approaches. All models are trained for 100 epochs with the ResNet-50/ImageNN benchmark. Each dot of Chase and Chase (prune skip) corresponds to model with distinct channel-wise sparsity. The results of latency are obtained on NVIDIA 2080TI GPU with a batch size of 2.
(SST) [28; 56; 53; 33; 22]. namely, draws sparse neural network at initialization before training and train with a fixed sparse pattern (sparse connections between layers) without further changes. Dynamic sparse training (DST) [39; 10; 35], on the contrary, jointly optimizes the weights and sparse pattems during training, usually delivering better performance than the static ones. DST quickly evolves as a leading direction in sparse training due to its compelling performance and training/inference efficiency. For instance, sparse ResNet-34 with only 2% parameters left can be dynamically trained to match the performance of its dense counterpart without involving any pre-training dense training [35].
While showing promise in performance and efficiency, so far the real speedup of DST has only been demonstrated on CPU [34; 6] or IPU [6]. Most sparse training methods produce unstructured sparse neural networks with extremely irregular sparse patterns, which can not be directly accelerated in common hardware (i.e., GPU and TPU), compared to the straightforward and serdware-frienddee sparse pattern produced by channel pruning [17; 37].
Many endeavors strive to solve this issue by coarsening the sparsity granularity, which can be loosely categorized into two groups. Grouping nonzero weights into blocks. As GPU performs very fast on contiguous memory operations, block-wise sparsity enjoys much more speedups than unstructured sparsity in practice. Group lasso regularization [41; 14] is a widely-used technique to induce block sparsity the network. Ad-hoe grouping operations can also be utilized to build dense blocks from unstructured sparse weights [46; 3]. ii. Seeking fine-grained structured sparse patterns. For instance, inspired by the recent support of 2:4 fine-grained sparsity in NVIDIA Ampere [42], previous arts attempt find sweet spot between structured and unstructured sparsity by learning N:M sparsity patterns [60; 20; 45]. However, these methods either rely on specialized sparse-aware accelerators [9; 42] to enable speedups or suffer from significant performance degradation due the constraint location of nonzero values [25].
In this paper, propose new method dubbed Channel- dynamic sparse (Chase). which can effectively transfer the promise unstructured sparse training into the hardware- friendly channel sparsity with comparable even better performance common GPU devices. The roadmap of our exploration is as follows:
We first present an emerging characteristic DST: off-the- the--hh--helf DST approaches implicitly involve biased parameter reallocation, resulting in large proportion channels (up to 60%) that rapidly become sparser than their initializations at the very early training stage. We term them "sparse amenable channels" for the sake of convenient reference.
Observation 1:
Observation 2: We examine the prunability (i.e., the accuracy drop caused by pruning) of the sparse amenable channels, we find that these channels cause marginal damages to the model performance than their counterparts when pruned.

---

We propose a new, inspiied,, channel pruning metric- - Unmasked Mean Magnitude (UMM) that can be used to precisely discover sparse amenable channels during training by monitoring the quantity and quality of weight sparsity.
A New Metric:
A New Approach: Based on the above findings, we propose Channel-ane aware dynamic sparse (Chase), a first sparse training framework that can favorably transform unstructured sparsity into channel-wise sparsity on the fly. Chase starts with an unstructured sparse neural network and dynamically trains while gradually eliminating sparse amenable channels with the lowest UMM scores. During training, we globally grow and shrink parameters to strengthen performance further.
Chase inherently can be tailed into an unstructured sparse training approach and structured sparse training approach. Our unstructured variant establishes of-the-art art accuracy bar for sparse training. More impressively, our structured approach is able to maintain even surpass SoTA performance with ResNet-50 on ImageNet, while achieving 1.2x 1.7x inference throughput speedups on common GPU devices.
Performance:
state-
2 Sparse Amenable Channels in DST
We first describe the basis and notations of the prior sparse training arts. Afterward, we provide evidence for the existence of the sparse amenable channels during the dynamic sparse training across different architectures and demonstrate that pruning of such channels leads to marginal performance damage than their counterparts. Based on this interesting finding, we introduce Chase, a sparsity- inspired sparse training method that for the first time translates the theoretical promise of sparse training into GPU-friendly speedup, without using any specialized CUDA implementations.
2.1 Prior Sparse Training Arts
Let denote the sparse neural network as fax; 0.). O₂ refers subset of the full network (..lo and || lo represents the fn-nomm.
parameters at a sparsity level of (1 100
is common to initialize sparse subnetworks 0g randomly based on the uniform [40; 5] or non- uniform layer-wise sparsity ratios with Erdös-Rény Rényi (ER) graph [39; 10; 35; 31]. In the case of image classification, sparse training aims optimize: 0, argming, ΣN E [zf 0,), y:) us- ing data {( M/ ) where L is the loss function. Static sparse training (SST) maintains the same sparse network connectivity during training after initialization. Dynamic sparse training (DST), on the contrary, allows the sparse subnetworks to dynamically explore new parameters while sticking
fixed sparsity budget. Most of the DST methods follow simple prune-and-gne scheme [39] to perform parameter exploration, i.e., pruning proportion of the least important param- eters based on their magnitude, and immediately grow the same number of parameters randomly [39] or using the potential gradient [10]. Formally, the parameter exploration can be formalized as the following two steps:
0. = (O,,, r),
(1)
0. 02 U ( :e0.: r).
where is the specific pruning criterion and & is growing scheme. These metries may vary from sparse training method to another. In addition to prune-and-grow previous work [23; 47] dynamically activates top-K parameters during forward-pass while keeping larger number of parameters updated in backward- pass to get rid of dense calculation of gradient. At the end of the training, sparse training can converge to a performant sparse subnetwork. Since the sparse neural networks are trained from scratch, the memory requirements and training/inference FLOPs are only a fraction of their dense counterparts.
One daunting drawback of sparse training is the resulting subnetworks are usually imbued with extremely irregular sparsity patterns, therefore, receiving very limited support from common hardware like GPU and TPU.
3

---

Dec aase stere Baa N
Decraas by rooie Bun 2020
Decimas by zere Eas y-30
Decruad by suee Dan 40
T
0.2
12
0.2
0.2
0.2
0.1
Dic mm
Dicrioa bymtem 2-20
Dec by zon 23
Decnonsby roo Dan
04
0.2
02-
03
Spocks
Rigl Nadel camd
Rig. Hed-3 Landd
SET Modil 100e0
Figure 2: The portion of sparse amenable channels justified by two metrics, the Unmasked Mean Magnitude (UMM) and the Weight Sparsity (WS), of ResNet-50 trained on CIFAR-100.
2.2 Sparse Amenable Channels
Here, we introduce the most important cornerstone concept for this work "sparse amenable channels" which defined as the channels whose sparsity becomes higher than their initial values caused by dynamic sparse training.
To provide empirical evidence for this interesting observation, we visualize the training dynamics of DST by monitoring two specific metrics of channels, Weight Sparsity and Unmasked Weight Magnitude, which are defined below.
Weight Sparsity (WS) (Quantity): Weight Sparsity directly quantizes the emergence of the Sparse Amenable Channels in quantity. Larger weight sparsity means more elements in the channel are becoming zero. Consequently, channels with fewer non-zero weights than their initial status are justified as Sparse Amenable Channels in this case.
Unmasked Mean Magnitude (UMM) (Quantity and Quality): Instead of solely quantitatively monitoring the weight sparsity, it is preferable take the quality (i.e., magnitude) of the nonzero weights into consideration due to the crucial role of magnitude to dynamic sparse training [39; 10; 35]. Here, Unmasked Mean Magnitude refers to the mean magnitude of all the weights (including zero and nonzero) in the channel without considering masking. Smaller Unmasked Mean Magnitude represents the channels that come to be more sparse both in quantity and quality. Specifically, channels with fewer non-zero parameters but larger magnitudes will be excluded from the Sparse Amenable Channels. Therefore, the number of Sparse Amenable Channels justified here will be smaller than WS. We formalize these two metries in Table I for a better interpretability. For comparison, we also evaluate the Masked Mean Magnitude (MMM), i.e., the mean magnitude of the non-zero weights.
We determined channels at the training iteration are amenable if their values of Weight Sparsity are larger than their ini- tialized values by ratio or their values of Unmasked Mean Magnitude are smaller than their initialized values by ratio ot WS WSS
Metrics that are introduced to measure the dynamics of the Sparse Amenable Channels. The weight tensor and the binary mask of channel is represented with @ and m, respectively. And || llo stands for the norm.
Table I:
or UMM,-UMM, > v. In UMMg
WS.
other words, we say channel becomes more sparse than its initial status if its WS, surpasses WS, by v, or its UMM; is smaller than UMMg by c.
Weight Sparsity (WS)
m0 la @llo
Unmasked Mean Magnitude (UMM)
Σ [e] 0
Masked Mean Magnitude (MMM)
Σ |m@@e] |m@ ll
Taking the most representative DST ap- proaches SET [39] and RigL [10] as exam- ples, we measure the number of the Sparse
Amenable Channels across layers in Figure 2, with e equals 0%, 20%, 30%, and 40%. We summarize
our main observations here. Overall, we observe that large part of channels (up to 60%) tend to be sparse amenable. While the number of amenable channels tends to decrease as v increases, there still exists around 10% 40% amenable channels becoming 40% more sparse than their initializations

---

Algorithm 1: Pseudocode of Chase
Input:
Sparse neural network with initialized sparse weight os target sparsity 6p, current sparsity 8t. parameter update frequency ATp, sparsity mutation factor se, target channel sparsity Se, channel pruning frequency AT, current channel sparsity S₁, training steps т. exploration stop
steps Tatop- total training steps Thota!
Output: A sparse model satisfying the target sparsity sp and channel sparsity requirement Se.
while т < Tatop do
if Mo,(t, AT) 0 and I Tmme then
0, f- Global channel prune O., St) & Perform gradual amenable channel pruning using
3 3
if Mod(t, ATp) 0 and < Tour then
0s(s: = se) <- Global Parameter Grow(O.)
Training for Ar epochs, + T AT,
0,(3: sp) + Global Parameter Prune(O.)
Chase adopts global parameter
expioration and soft memory bound.
Continue sparse training from the epoch Tatop to Tend.
across layers. @ Fewer channels are justified sparse amenable channels using the UMM metric than WS, as we expected. @ Deeper layers suffer from more amenable channels than shallow layers. 4 RigL tends extract more amenable channels than SET at the very early training phase. A possible reason is that the dense gradient encourages RigL to quickly discover and fill weights to the important (non-amenable) channels compared the random growth used in SET.
Sparse amenable channels enjoy better prunability' than their counterparts. So far, we have unveiled the existence of the sparse amenable channels. It is natural to conjecture that these amenable channels can be good indicator for channel prun- ing. To evaluate our conjecture, we choose the above proposed
Table 2:
Top-1 test accuracy (%) of various channel pruning criteria with ResNet-50 on CIFAR-100. "Reverse" refers to pruning with the reversed metric.
Channel Sparsity
Method
20%
30%
metrics, Weight Sparsity (WS) and Unmasked Mean Mag- nitude (UMM), our pruning criteria and perform a simple one-shot global channel pruning after regular DST training in com- parison with their reversed metrics as well as several commonly-used principles, in- cluding random pruning [33]. network slim-
Standard Rigl. [10)
76.89
76,992043
76.89:50:43
Random Pruning [33]
43.01=+9.62
11.74:22.79
3.79+11.32
Network Slimming [37]
76.82 10.43
76.67.10.39
66.57.12..5
MMM
62.31118666
19.34 14.88
5.32-:2.88
MMM Reverse
5.28:22.52
2.04=20.30
1.72::0.40
WS
76.86:20.43
76.79:20.99
62.79-13344
Revense
2.9.10911
2.43.10.07
2.03.10.38
UMM UMM Reverse
76.88:10.43
76.90:10.42
71.7742.1
ming [37], and Masked Mean Magnitude (MMM). Channels with the highest values
3.18+00.48
0.23
1.51
are pruned for WS, and the ones with the smallest values are pruned for UMM. Table 2 shows that both WS and UMM achieve good performance and UMM performs the best. Meanwhile, their reversed metries perform no better than random pruning. Perhaps more interestingly, the resulting hybrid channel-level sparse models favorably preserve the performance of the unstructured RigL. with no accuracy drop when pruned with mild channel sparsity.
In addition, we also observe the existence of "sparse amenable channel" in a broad range of settings, including BesNeet32//GG: 16 on CIFAR-100, MLP Model on CIFARIO, and ViT Small, 50 on ImageNet in Appendix. Hence, we believe that sparse amenable channels is a very general phenomenon that widely exists across different architectures and datasets.
This encouraging result confirms our conjecture and demonstrates the promising potentials of sparse amenable channels (UMM) as strong metric remove channels during training. In the next section, we will explain in detail how we leverage Sparsity Amenable Channels and UMM to translate the promise of unstructured sparse training to the hardware-friendly sparse neural networks.
'rrunablityy here refers the accuracy drop caused by the channel removal.
5

---

3 Methodology Chase
Inspired by the above encouraging findings of sparse amenable channels, we introduce Chase-aware dynamic sparse (Chase) in this section. We follow the widely-used sparse training framework used in [39; 10]. The technical novelty of Chase mainly lies in two aspects. On the structured sparsity level, adopt the gradual sparsification schedule [61] to gradually remove Amenable Channels during training with smallest UMM scores. The gradual sparsification schedule provides us with a moderate sparsification schedule, favorably relieving the accuracy drop caused by aggressive channel pruning. On the unstructured sparsity level, we globally redistribute parameters based on their magnitude and gradient, which significantly strengthens the sparse training performance. The overall Pseudocode of Chase is illustrated in Algorithm 1. We provide technical details of the above components below.
Gradual Amenable Channel Pruning. The gradual sparsification schedule is widely-used in the unstructured sparse literature to produce strong unstructured sparse subnetworks [61; 11;31], We explore it to the channel pruning regime with several ad-hoc modifications. Let denote the initial and target final channel-wise sparsity level as S; and Sy, respectively; gradual pruning starts at the training step ia with pruning frequency AT, performing over span of pruning steps. The sparsity level Sr at pruning step f. is:
t- nAT
We globally collect UMM (see Section 2.2 for the definition) of each channel as the pruning criterion and progressively remove the sparse amenable channels with the smallest UMM according to Eq 3. We observe that layer collapse occurs sometimes without setting layer-wise pruning constraints. To avoid layer collapse, we use 3 to control the minimum number of channels remaining in layer / be (1 - Sy) 3 wy, where wi is the number of channels in layer 1. We empirically find that smaller B tends to yield better performance. We report more details Appendix A.2.
To maintain the overall number of parameters the same during training, we redistribute the overly pruned parameters back to the remaining channels at the next parameter grow phase using Eq 2. We find that without doing this will significantly hurt the model's performance.
Global Parameter Exploration. Global parameter exploration was introduced in previous arts [40; 5]. However, with the popularity of RigL [10], is common to use a fixed set of layer-wise sparsities. Here, we revisit global parameter exploration in DST. To be specific, we globally prune parameters that have the smallest magnitudes and grow parameters with highest gradient magnitude. This small adaption brings a large performance benefit to RigL (up to 2.62% on CIFAR-100 and 1.7% on ImageNet), reportee as "Chase (S.- = 0)" in Table and Table 4.
Soft Memory Bound. Soft memory bound was proposed in [59], which allows the parameter growing operation happens before the parameter pruning, improving the performance the cost of slight increase of memory requirements and FLOPs. We borrow the idea of soft memory bound to allow parameters firstly being added to the existing parameters followed by AT, iteration of training, then remove the less important parameters including the newly added ones. This can avoid forcing the existing weights the model to be removed if they are more important than newly grown weights.
After training, Chase slims down the initial "big sparse" model to a "small sparse" model with a significantly reduced number of channels. We completely remove the pruned channels in the current layer as well as the corresponding input dimensions of the next layer, so that the produced small sparse models can directly enjoy the acceleration GPU.
4 Experimental Evaluation of Chase
In this section, we comprehensively evaluate Chase in comparison with the various state-of of-the- art (SOTA) unstructured sparse training methods as well as the state-of-the channel-pruning algorithms. At last, we provide a detailed analysis of hyperparameters and perform an ablation study to evaluate the effectiveness of the components of Chase.
Our evaluation is conducted with two widely used model architectures VGG-19 [48] and ResNet- 50 [15] on across various datasets including CIFAR-10/100 and ImageNet, We summarize the

---

Table 3: Test accuracy (%) of the sparse VGG-19 and ResNet-20050
Dataset
CIFAR 10
CIFAR-100
Sparsity
90%
95%
98%
90%
95%
98%
VGG-19 (Dense)
93.85-+-0005
93.855=0000
93 03.855-0.05
73.43-4-0.08
73.43-40.08
73.43-20.08
SynFlow [53]
93.35
93.45
92.24
71.77
71.72
70.94
GraSP [56]
93.30
93.04
92.19
71.95
71.23
68.90
SNIP [28]
93.63
93.43
72.84
71.83
Chase+CraSPP 0.5)
94.06-4-0.22
93.89--0.20
73.17:40.09
72.81+00111
7.66
Chase+SNIP (S. 0.5)
94.83-1-0.06
95.08:10.14
78.26:100.66
77.16::0.04
Deep-R [1]
90.81
89.59
86.77
66.83
63.46
59.58
SET [39]
93.61:1:0113
93.09:10.25
91.81:0.04
72.58:10.12
7144::0.12
69.04=20.15
RigL [10]
93.60:10.09
93.05:0.006
91.5510115
72.9240.31
71.5510.33
69.57=0.24
MEST [59]
93.61:10.36
93.4610.41
92.30:10.44
72.52:10.37
71.2110.41
69.02=0.34
Chase (S.
94.02440113
93.89:10.12
93.60=#0.05
73.5410.12
73.05::0.25
0..19=0.33
Chase (8.
94.03-1-0.11
93.84:00.08
93.69-0.033
73.43+0.12
73.04::..22
08
ResNet-50 (Dense)
94.75-H0.01
94.75-+0.01
94.75++0.01
78.23--0.18
78.23+-0.18
78.23--0.18
SynFlow [53]
92.49
91.22
88.82
73.37
70.37
62.17
SNIP [28]
92.65
90.86
73.14
69.25
GraSP [56]
91.32
88.77
73.28
70,29
62.12
Chase+SNIP (S. 0.5)
03.99:1:009
00
73,441:0.02
72. 72.80-00005
Chase+GraSP (S. 0.5)
94.78:1:0.35
94.71:10.07
94,36:10.15
77.70:10.24
7..65
7..74=0.24
Deep-R R [1]
91.62
89.84
86.45
66.78
63,90
58.47
SET [39]
94.65-+0.01
94.05:00.06
92.98-10.18
76. 16.1444
75.90:00.19
73.21=00.06
Rigl-[10)
94.42:1:.117
94,22:1:.23
93.20:10.08
77.18:10.42
76.50:10.26
7..84=00113
Chase (S. =
94.95100022
94.87-10.02
94.15:0.177
78.111:0.11
78.141:0.28
76.88-10.31
Chase (8. 0.5)
94.88:10.03
94.85
94,20:10.18
77.52+10.0
77.48:10.2
07.03::.229
ResNet-20 (Dense)
92.55-H0.02
92.55++0.02
92.55-+0.02
68.654:0.19
68.65
68.65=b.19
SNIP [28]
88.06:10.07
84.21
74.61=00.40
54.40:0.09
42.45:00.65
24.55=0.56
GraSP [56]
88.35+00.12
84.95-+0.30
78.25-+0.22
59.40.0.088
45.96-+0.11
30.67-00.44
SET [39]
90.16:1:.09
87.70:10.09
83.41:10.04
62.08:10116
54.77-10.74
43.70:0092
[1gl
89 89.82++0.10
87.44-0.33
79.16++0.96
60.49+0.17
52.97-40.23
31.94+1.52
Chase (S: =
90.43:100.16
88.65-10.29
85,26:10.29
62.18:10.05
$7.38:10.41
47.06:0065
Chase 0.5)
89.98-10044
88.654-0.02
85.24440.18
60.88-10.19
55.78-L0.37
46.91
implementation details for Chase in Appendix B. To show the superior performance of Chase on unstructured and structured sparsity, report two variants of Chase: Chase (S,- represents the unstructured version without channel pruning and Chase (S. 0.5) stands for the structured version with 50% channel-level sparsity.
4.1 Comparison with off-the-shelf SOTA DST
CIFAR-10/100. Our method is naturally versatile and can be applied to both static sparse training and dynamic sparse training regimes. Therefore, for each model, we categorize the results into two groups and report the results in Table 3.
Chase dramatically improves the performance SST. As shown in the upper panel of each group, Chase significantly boosts the accuracy (up to 13.62% for GraSP with ResNet-50 on CIFAR- 10FR-100) of SST methods like SNIP and GraSP. @ Chase (S.- 0) establishes new state-of-the bar for unstructured sparse training. Compared with the SOTA DST methods such as RigL and MEST, we clearly see that Chase (S.- = 0) universally outperforms all the presented DST methods by large margin. Specifically, Chase (S.- = 0) achieves 3.67 % and 3. 1.5%% performance gains compared with SET on Re and VGG-19 19. We also notice that the performance gain on CIFAR- 100 is larger than the ones CIFAR-10, which is expected since CIFAR-100 has larger improvement space than CIFAR-IO. Chase (S.- = 0.5), with only 50% channels remaining, matches the performance of its unstructured variant, demonstrating the promise of the unstructured DST can be favorably transferred to the structured regime.
performance
ImageNet. We reported the results on ImageNet in Table 4. Again, Chase (S. 0) dominates the performance the unstructured sparsity regime, achieving 1.7% and 2.1 accuracy improvements over Rigl. and MEST, respectively. While the accuracy of Chase (S. = 0.4) slightly decreases by 0.67% compared to Chase (S. 0), still outperforms Rigl. by a good margin (1.03%), while enjoying 1.5x x real inference speedup common GPU.
7

---

Table 4: Test accuracy (%) of sparse ResNet-50 ImageNet trained with 100 epochs and 150 epochs (1.5x). Training FLOPs of sparse training methods are normalized with the FLOPs used to train a dense model. "GPU-supportee FLOPs" refers to the real FLOPs that are required to calculate common GPU which usually does not support irregular sparsity patterns.
Method
Top-1
Theuretical FLOPs (Trin)
Theoretical FLOPs (Tes))
GPU Suppopee
TOP-1 Accuary
Theoretical FLOPs (Tain)
Theuretical FLOP (Tes)
Gap-Stpo FLOPs (Test)
Accuracy
FLOPs (Tes)
ResNet- (Deess)
76.8100.9
(3.2.18)
Ix (8.2cc)
(8.2c))
76810009
Ix (3.2e18)
1x (8.269)
Ix (8.2c9)
Sparsity
80%
90%
72.910039
023x
69.610023
DSR (40]
73.3
0.40%
0.4xx
71.6
0,30x
0,50x
1,00x
SNFS [5]
75.24+0.11
0.61x
0.42x
1.00x
72.9+0.06
0.50x
0.24x
1,00x
RigL [10]
75,1110.05
0442
0.42x
100%
73.0110.44
0.25
0.24x
1,0%
MEST [59]
75.39
0,23x
0.21x
1000
72.58
0.12X
0.11x
1,00x
RigL-ITOP [35]
75.84110.05
0.42x
0.42x
73,8210.08
0.25
0.24x
1.00x
75.87
0.34x
1,0%
Chase 18. 0..3)
15.62
0,39x
0.36x
0.5xx
74.35
0.74x
Chase (S. 0.43
73.27
0.39%
0.37x
0.68x
74.03
0.26x
0.23x
0.67
MEST
75.73
0.40x
0.21x
1.00
75.00
0.20x
0.11x
1,00
Chase (5.
76.67
0.55%
0.34%
100%
75.77
0.36X
0.21%
1,00%
Chaseias 0.3)
76.23
0.57x
0.36x
0.75x
75.20
0.37
0.22x
0.74x
Ju (3. 0:4)
76.00
0.37x
0.68x
0.67
When increasing the training time to 1.5x (150 epochs), Chase also demonstrates a promising scaling trend. Chase (S.- = 0) matches the dense performance with only 0.55 x training FLOPs and consistently outperforms the off-the off-the-shelf best DST method, MEST. Again, the promising results of Chase (Se- 0) can be effectively translated to channel-level sparsity. Chase (Se- 0.4) is 1.5x faster than MEST, while still performing on par or even better than MEST.
Real Inference Speedups. We further compare the actual inference throughput and latency of our model against Rigl. Table All results are averaged from 100 individual runs with one NVIDIA 2080TI GPU in float32 on PyTorch. We set the batch size to 128 for CIFAR-100 and 2 for ImageNet, when evaluating the latency. We empirically find that pruning the skip connection leads to significant accuracy drop while providing benefits speedups. Therefore, the standard Chase keeps the skip connection layers untouched for optimal accuracy. To fully unleash Chase's ' potential on real inference speedups, we also provide another variant of Chase that prunes skip connection layers, dubbed Chase (prune skip). Compared with SoTA RigL. Chase (prune skip) is able to prune 50% channels with ResNet-50 on ImageNet, leading to a notable 68% of throughput gain, while only losing 0.29% accuracy. Even without pruning skip connections, our model is about to provide 31 % throughput speedups, while outperforming RigL. by 0.39%.
Table 5: Inference throughput and latency. The best results are marked in bold.
Method
Datases
Model
Sy.
5.
Accuracy (f)
Throughput (h)
Latency (ms) (1)
RigL[100 [10]
CIFAR-100
VGG-19
0.9
0.0
72.92:+0031
15274.31
Chase
CIHAR-100
VGG-19
0.9
0.5
73.43:20.12
24981.77 (64%1)
5.37 (36% ()
Rigl. [10]
CIFAR-100
ResNet.50
0.9
0.0
17.18:10.42
3095.13
44,23
Chase
CIFAR-100
55
0..52:10.30
3958.67 (28%
34.76 (21%)
RigL [10]
ImageNNt
R5
0.9
0.0
73.00+00.04
59.50
35.79
Chase
ImageNet
RcsNei-50
0.9
0.5
73.39=20.04
78.19 (31%t)
27.55 (23%) ()
Chase (peune skip)
ImageNNt
ResNet-50
0.9
0.5
72.71++0.00
99.97 (68% f
21.55 (40% L)
Chase
ImngeNet
ReaNet-50
0.9
0.4
74.03-20003
71.54 (20%))
30.13 (16%)
Chise (prune skip)
ImageNet
ResNet-50
0.9
73.24-10.03
87.38 (47%
24.53 (319-4
4.2 Extensive Analysis
Performance under different channel sparsity. To investigate the performance of different channel sparsity with the same parameter count, we maintain 2%, 5% parameters and prune the model to different channel sparsity ranging from 20% to 70%. The results are reported in Figure 3. The same training scheme is adopted as Section 1. Two DST baselines, Rigl. and SET are adopted for comparison. Not surprisingly, the more channels remain the better performance of the model archives. Notably, in all settings, Chase archives better performance than the baselines with just 50% channels, and Chase outperforms SET and RigL with just 30% channels on 50 at 98% sparsity.
8

---

98 % Sparsity
95 % Sparsity
98 % Sparsity
95 % Spuriity
77
78
Accumay
21
76
70
71
75
76
73
20
96
30
40
96
50
96
36 %
20 46
t6
96
50
9b
60
99 90
a0 96
50 90
60
10 %
20
90 90
20,00
Channel-wise spursity
Chase
Righ (5, =
(S,
Figure 3: Performance of Chase under different channel sparsity. For Rigl and SET, we keep the channels un-pruned as baselines.
Apeiity
High -GCC-M0
REL SMOUE (Chae)
Figure 4: Ablation Study of Chase. GACP denotes gradual amenable channel pruning (50% channel sparsity), SM indicates soft memory bound, GE represents global parameter exploration.
Effect of the channel pruning frequency. We also study how the channel pruning frequency AT affects Chase's performance. For all experiments, we fixed the ending time Tstop for gradual amenable channel pruning as 130 epochs, the total training epochs Trotel 160 epochs and the minimum channel ratio factor as 0.5, while altering AT 1000, 4000, 8000, and 16000 iterations. We report the results in Appendix A.4. Overall, the largest AT 16000 leads to worse performance. This observation is as expected, as we aim to achieve the same channel sparsity and larger AT results in more removed channels in each punning operation. Consequently, larger performance degradation will be introduced during each pruning which could degrade the training stability.
Ablation study. In Figure 4, we study the effectiveness of different components in Chase, namely, the soft memory constraint (SM) and global parameter exploration (GE) on CIFAR-10/101 with ResNet-50 and VGG-19. We denote the RigL as our baseline, as RigL applies magnitude- based pruning and gradients-based growth like Chase. We apply the same training recipe as described in Section 4. Gradually amenable channel pruning safely removes 50% channels from RigL, while only suffering from minor or even no performance degradation. As for SM and GE, we found these techniques all bring universal performance improvements. Surprisingly, adding SM results in a 1.26% accuracy increase on CIFAR- 100 with ResNet-50 at 98% sparsity. With GE, we can obtain a more optimal ayyerwwise ratio, which also consistently improves the accuracy from SM.
5 Related Work
Recently, as the financial and environmental costs of model training grow exponentially [50; 43], endeavors start to pursue training efficiency by investigating training sparse neural networks from scratch. Most Sparse training works can be divided into two categories, static sparse training, and dynamic sparse training. Static sparse training determines the structure of the sparse network at the initial stage of training by using certain pre-detined layer-wise sparsity ratios [38; 39; 10; 33].
Dynamic sparse training is designed reduce the computation as well as memory footprint during the whole training phase. It trains sparse neural network from scratch while allowing the sparse mask to be updated during training. SET [39] update sparse mask at the end of each training epoch by magnitude-basee pruning and random growing. DSR [40] develops dynamic reparameterizztioon method that allows parameter reallocation during dynamic mask updating. DeepR [1] combines dynamic sparse parameterization with stochastic parameter updates for training. RigL [10] and

---

SNFS [5] propose to uses gradient information to grow weights. ITOP [35] studies the underlying mechanism of DST and discovers that the benefits of DST come from searching across time all possible parameters. GraNer [31 introduces the concept of "pruning plasticity" and quantitatively studies the effect of pruning throughout training. MEST [59] proposes a memory-friendly training framework that could perform fast execution on edge devices. AC/DC [44] co-trains the sparse and dense models to return both accurate sparse and dense models. [23] dynamically activates top-K parameters during forward-pass while keeping a larger number of parameters updated in backward- pass to get rid of dense calculation of gradient. Top-KAST [23] preserves constant sparsity throughout training in both the forward and backward passes. Built upon Top-KAST, Powerpropagation [47] leaves the low-magnitude parameters largely unaffected by learning, achieving strong results. CHEX [19] applied dynamic prune and regrow channels strategies to avoid pruning important channels prematurely. Very recently, SLaK [32] leverages dynamic sparse training to successfully train intrinsically sparse 51x kernels, which performs par with better than advanced Transformers. A concurrent work [21] discovers that a tiny fraction of channels (up to 4.3%) of Rigl. become totally sparse after training.
To enable acceleration of sparse training in practice, [34] build a truly sparse framework based on SciPy sparse matrices [55] that enables efficient sparse evolutionary training [39] in CPU. [6] fulfill group-wise DST on Graphcore IPU [24] and demonstrate its efficacy on pre-training BERT. Moreover, some previous work develops sparse kernels [12; 9] to directly support unstructured sparsity in GPU. DeepSparse [27] deploys aarge-scale BERT-level and YOLO-level sparse models on CPU.
6 Conclusions
In this paper, we have presented Chase, new sparse training approach that seamlessly translates the promise of unstructured sparsity into channel-level sparsity, while performing on par or even often better than state-of-the-art DST approaches. Extensive experiments across various network architectures including VGG-19 and ResNet-50 on CIFAR- 10/100 and ImageNet demonstrated Chase can achieve better performance with 1.2x 1.7x x real inference speedup on common GPU devices while performing on par or even better than unstructured SoTA. The results in this paper strongly challenge the common belief that sparse training typically suffers from limited acceleration support in common hardware, opening doors for future work to build more efficient sparse neural networks.
7 Acknowledgement
S. Liu and Z. Wang are in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML). Part of this work used the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. NWO2021.066 00600, EINF-2694 and EINF-2943/L11 It is also supported by the NSF CCF-23122166 Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF.
References
[1] G. Bellec, D. Kappel, W. Maass, and R. Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018. 7,9
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, F. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[3] т. Chen, X. Chen, X. Ma, Y. Wang, and Z. Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. arXiv preprint arXiv:2202.04733 2022. 2
[4] T.WW. Chin, R. Ding, C. Zhang, and D. Mareulescu. Legr: Filter pruning via learned global ranking. 2019. 17
10

---

[5] T. Dettmers and L. Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.0484/ 2019. 6, 8, 10
[6] A. Dietrich, F. Gressmann, D. Orr, I. Chelombiev, D. Justus, and C. Luschi. Towards structured dynamic sparse pre-training of bert. arXiv preprini
2021. 2, 10
[7] X. Dong and Y. Yang. Network pruning via transformable architecture search. Advances in Neural Information Processing Systems, 32, 2019. 17
[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani,
M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[9] E. Elsen, M. Dukhan, T. Gale, and K. Simonyan. Fast sparse convnets. In Proceedings of the IEFE/CVE conference on computer vision and pattern recognition, pages 14629-14638, 2020.
2, 10
[10] U. Evci, Т. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952, PMLR, 2020. 2, 3, 3, 5, 6, 7, 8, 9, 16
[1]] T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. arXiv preprint
arXiv:1902.09574, 2019. 6
[12] T. Gale, M. Zaharia, C. Young, and E. Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006. 2020. 10
[13] E. Garefa-Martin, F. Rodrigues, G. Riley, and H. Grahn. Estimation of energy consumption in machine learning. Journal of Parallel and Distributed Computing, 134:75-88, 2019.
[14] S. Gray, A. Radford, and D. P. Kingma. Gpu kernels for block-sparse weights. arXiv preprint arXiv:1711.09222 3:2, 2017. 2
[15] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-
778, 2016. 6
[16] Y. He, P. Liu, Z. Wang. Z. Hu, and Y. Yang. Filter pruning via geometric median for deep con- volutional neural networks acceleration. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019. 17
[17] Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1389-1397, 2017.
2
[18] Hestness, S. Narang, N. Ardalani, G. Diamos, Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712: 00409, 2017.
[19] Z. Hou, M. Qin, F. Sun, X. Ma, Yuan, Y. Xu, Y.-K Chen, R. Jin, Y. Xie, and S.-Y. Kung. Chex: Channel exploration for cnn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12287-12298, 2022. 10
[20] I. Hubara, B. Chmiel, M. Island, R. Banner, S. Naor, and D. Soudry. Accelerated sparse neural training: A provable and efficient method to find n: M transposable masks. arXiv preprint arXiv:2102.08. /24, 2021. 2
[21] E. Iofinova, A. Peste, M. Kurtz, and D. Alistarh. How well do sparse imagenet models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12266-12266 2022. 10
[22] A. K. Jaiswal, H. Ma, T. Chen, Y. Ding, and Z. Wang. Training your sparse neural network better with any mask. International Conference on Machine Learning, pages 9833-9844. PMLR, 2022." 2
11

---

[23] S. Jayakumar, R. Pascanu, J. Rae, S. Osindero, and E. Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744-20754, 2020. 10
[24] Z. Jia, B. Tillman, M. Maggioni, and D. P. Scarpazza. Dissecting the grapheore ipu architecture via microbenchmarking. arXiv preprint arXiv: urXiv1112.034133 2019, 10
[25] P. Jiang, L. Hu, and S. Song. Exposing and exploiting fine-grained block structures for fast and accurate sparse training. In Advances in Neural Information Processing Systems, 2022. 2
[26] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.00 0003611 2020.
[27] M. Kurtz, J. Kopinsky, R. Gelashvili, A. Matveev, J. Carr, M. Goin, W. Leiserson, S. Moore, B. Nell, N. Shavit, and D. Alistarh, Inducing and exploiting activation sparsity for fast inference on deep neural networks. In H. D. IIII and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5533-5545 Virtual, 13-18 Jul 2020. PMLR. 10
[28] N. Lee, T. Ajanthan, and P. H. Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810002440. 2018. 2, 7
[29] M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1529-1538, 2020. 17
[30] L. Liu, S. Zhang. Z. Kuang, A. Zhou, J.-H. Xue, X. Wang. Y. Chen, W. Yang, Q. Liao, and W. Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pages 7021-7032. PMLR, 2021. 17
[31] S. Liu, T. Chen, X. Chen, Z. Atashgahi, L. Yin, H. Kou, L. Shen, M. Pechenizkiy, Z. Wang, and D. C. Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems, 34:9908 9922, 2021. 3, 6, 10
[32] S. Liu, T. Chen, X. Chen, X. Chen, Q. Xiao, B. Wu, M. Pechenizkiy, D. Mocanu, and Z. Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprint arXiv:2207.03622 2022. 10
[33] S. Liu, T. Chen, X. Chen, L. Shen, D. C. Mocanu, Z. Wang, and M. Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv preprini arXiv:2202.02666
2022. 2, 5,9
[34] S. Liu, D. C. Mocanu, A. R. R. Matavalam, Y. Pei, and M. Pechenizkiy, Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications, 33(():2589- 2604, 2021. 2, 10
[35]= S. Liu, L. Yin, D. C. Mocanu, and M. Pechenizkiy. Do we actually need dense over- parameterization? in-time pramettrrtttiiiizaioonn in sparse training. In Proceedings of the 39th International Conference on Machine Learning, pages 6989-7000. PMLR, 2021. 1, 2,2, 3, 4, 8, 10
[36] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang. L. Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12009-12019, 2022. 1
[37] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pages 2736-2744, 2017. 2,5
[38] Z. Mariet and S. Sra. Diversity networks: Neural network compression using determinantal point processes. In International Conference Learning Representations, 2016. 9
[39] D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. 177.v.170..44880. Nature communications. 9():23383, 2018. 1, 2, 3, 4, 6, 7, 8, 9, 10, 16
12

---

[40] H. Mostafa and X. Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. International Conference on Machine Learning, 2019. 3, 8, 8,9,1 19
[41] S. Narang, E. Elsen, G. Diamos, and S. Sengupta. Exploring sparsity in recurrent neural networks. arXiv preprint urXiv:/704.055: 19, 2017.
[42] Nvidia. Nvidia a100 tensor core gpu architecture. =/Solutions/DataaCCnttreree Caataaaa--eeneeevviiiiaaaaa Center/mviiii-aapeee-aarriiiiii
http::/wwwwaadddd.cccc//ttttttaa 2020 2020.
[43] D. Patterson, Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean. Carbon emissions and large neural network training, arXiv preprint arX15:2104.10350, 2021. 1,9
[44] A. Peste, E. Iofinova, A. Vladu, and D. Alistarh. Ac/de: Alternating compressedddecopressed training of deep neural networks. Advances in Neural Information Processing Systems, 34:8557- 8570, 2021. 10
[45] Pool and C. Yu. Channel permutations for n:m sparsity. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 13316-13327. Curran Associates, Inc., 2021.
[46] M. A. Rumi, X. Ma. Y. Wang, and P. Jiang. Accelcrating sparse emn inference on gpus with performance- weight pruning. Proceedings of the ACM International Conference on Parallel Architecturs and Compilation Techniques, pages 267-278, 27200. 2
[47] Schwarz, S. Jayakumar, R. Pascanu, P. E. Latham, and Teh. Powerpropagation: sparsity inducing weight reparameterisation. Advances in Neural Information Processing Systems, 01228899-28900 2021. 3, 10
[48] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2014.
[49] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonss et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0405
2022.
[50] E. Strubell, A. Ganesh, and A. McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint 1w0.iv::966000
2019. 9
[51] X. Su, S. You, T. Huang, F. Wang, C. Qian, C. Zhang, and Xu. Locally free weight sharing for network width search. arXiv preprint 2rXiv:2102.0522 2021. 17
[52] Y. Sui, M. Yin, Y. Xie, H. Phan, S. Aliari Zonouz, and Yuan. Chip: Channel independence- based pruning for compact neural networks. Advances in Neural Information Processing Systems, 34:24604 24616, 2021. 17
[53] H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances Neural Information Processing Systems. ar 2mx5:20006.05467, 2020. 2,7
[54] Y. Tang, Y. Wang, Xu, Tao, C. Xu, C. Xu, and Xu. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems, 13:10936-10047, 2020.
[55] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, Peterson, W. Weckesser, J. Bright, S. van der Walt, M. Brett, J. Wilson, K. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientifie Computing Python. Nature
Methods, 17:261-272, 2020. 10
13

---

[56] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. International Conference Learning Representations, 2020. 7
[57] Z. You, K. Yan, Ye, M. Ma, and P. Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems, 32, 2019. 17
[58] Yu and T. Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint 1933:11033.11798 2019. 17
[59] G. Yuan, X. Ma. W. Niu, Z. Li. Z. Kong, N. Liu, Gong, Z. Zhan, C. He, Q. Jin, et al. Mest: Accurate and fast memory- recoooy sparse training framework on the edge. Advances in Neural Information Processing Systems, 14:20838-20850 2021. 6, 7, 10
[60] A. Zhou, Y. Ma, Zhu, Liu, Zhang, K. Yuan, W. Sun, and Li Learning n:m fine-grained structured sparse neural networks from scratch. In International Conference Learning Representations, 2021.
[61] M. H. Zhu and S. Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression, 2018.
14

---

A Remaining Experimental Analysis
A.1 Effeet of the Initial Sparsity
Chase starts from a subnetwork with unstructured sparsity to produce channel-level sparsity during end-to-end training process. Here we fix the target channel-level sparsity S. 0.4 and study how the initialized unstructured sparsity impacts the model performance. The results are reported in Figure 5. It could be seen that, in general, initialization with more parameters leads better performance. Counter-intuitively, the best accuracy is achieved using 0.6 unstructured sparsity, which 0.26% higher than initialized as a dense model.
76.8
Accuracy
16.0
34.4
demse
Inirialized unstructured sparsity
Figure 5: Performance under different initial unstructured sparsity ResNet-18, CIFAR-100.
A.2 Effect of the Minimum Channel Ratio Limitation 3
We alter the ratio of the minimum channel ratio 0.2, 0.5, 0.8 and show the performance in Table The channel pruning frequency AT is fixed 4000 iterations. Apparently, large ratio 3 0.8 archives the worst performance while 3 = 0.2 outperforms other settings out of 6 cases. In view of the fact that smaller 3 provide more channel exploration space.
Table 6: Test accuracy (%) CIFAR-100 of Chase at 50% channel-wise sparsity using different minimum layer limitation factor B. The best results are marked in bold.
Minimum
Sparity
layer ratio
90%
95%
98%
VGG-19
0,20
73.4510.227
72.98:10.22
71.69:10.1
0.50
13.16+00.06
72.39++0.17
71.74+t0.06
72.26.1..24
72.20.10.27
10.26.
ResNer-50
0.20
77.47.10.40
77.43
76.68 10.27
0,50
77.54.10.40
17.31
76.6 76.6410.26
76.76:10.66
77.2610.73
76.67:10.9
A.3 Ablation of Gradual Amenable Channel Pruning
Here, we perform more university ablation study of Gradual Amenable Channel Pruning (GACP) on CIFAR10/I00 ResNet-50 and VGG-19, with RigL and SET. The results are reported in Table 7.Sur- prisingly, GACP brings performance increases in most cases. To be specific, we found that GACP could boost the performance of RigL in 9 out of 12 cases and output SET in out of 12 cases with just 50% remaining channels.
A.4 Effect of the Channel Pruning Frequency
In this Appendix, we study how the channel pruning frequency AT affects Chase's performance. For all experiments, we fixed the ending time Tatop for gradual amenable channel pruning as 130 epochs, the total training epochs Trotal as 160 epochs and the minimum channel ratio factor as / as 0.5, while altering AT to 1000, 4000, 8000, and 16000 iterations. We report the results in Table 8. Overall, the largest AT 16000 leads to worse performance. This observation as expected, as we aim achieve the same channel sparsity and larger AT results in more removed channels in each
15

---

Table 7: Ablation of gradual amenable channel pruning (GACP). The best results marked in bold.
Dataset
CIFAR-10
CIFAR-100
Spursity
90%
95%
98%
90%
95%
98%
VGG-19
SET [39]
93.61 10013
93.09:10.25
91-81-10.044
72.58-10.11
71.48:0.12
69.04:0.115
SET+GACP (8. 0.50
93.78:00.16
93.56:20.05
92.66=0.10
72.93=+0.18
71,89:0.20
70.03=0.13
RigL,
93.60:20.00
92005+00.06
91.95=00155
72.92:00.3
69.57=0.24
RigL=GGACE (S, 0.5)
9..11:0.144
93.70:00006
92.80=10.00 (1.06
72.97:10.09
71,80+0.117
69.83
[39]
92.98-10.118
75.90:10.19
SET SETAGACP (8, = 0.50
94.52:40.17
94.51::0.19
93.23=20.13
76.75:t0.47
75.67::0.64
73.86=0.112
[10]
94.42-40.17
94,2240.23
00.00::0.08
77.18-+0.42
76.50++0.6
74.84=0.13
Rigl.+GACP (5. 0.5)
94.45:10.10
94.41:10.13
93.50:10.16
76.86:40.56
76.33-40.63
74.92:0.27
punning operation. Consequently, larger performance degradation will be introduced during each pruning which could degrade the training stability.
Table 8: Test accuracy (%) on CIFAR- 100 of Chase at 50% channel-wise sparsity using different channel pruning frequency AT, The best results are marked in bold.
AT
Sparsity
(brratoos))
90%
95%
98%
VGG-19
73.07-4-0.26
72.72:4-0.1
4000
73.16:2:0:06
72.39:50.17
71.74:00006
K000
73.15:::0.3
72.01
71.94220.13
72.88::0.24
72.66:20.044
71:79:0::4
ResNe-50
1000
77.52:::0
77.48::0.62
77.03-10.29
4000
77.54:200.40
77.31
76.64220.26
16000
77.02::0.25
76.96:00.97
76.77:20.35
B Implementation Details of Chase
In this appendix, we report the implementation details for Chase, including total training time (Frotal), exploration stop time (Tstop), gradual channel pruning frequency (AT',, parameter update frequency ATp. minimum layer limitation factor (3), learning rate (LR), batch size (BS), learning rate drop (LR Drop), weight decay (WD), SGD momentum (Momentum), sparse initialization (Sparse Init), target sparsity (sp), target channel-wise sparsity (Sz), etc.
Table 9: Implementation hyperparameters of Chase in Table 3, on CIFAR-10/100.
Model (epocla))
(sprcle)) 37 (lenttio))
22, (icrationn))
8
BS
LR
LR Doup. Epuchs
Optiniser
Momeatum
WD
Spane fait
VGG 19
160
3000
1000
0.2
0.1
10s, (80, 120]
SGD
09
ERK
RcsNss90
1000
120
Table 10: Implementation hyperparameters of Chase in Table 4, on ImageNet.
Trees (opochs)
Tatss (epocts)
AT, (berations)
Drop
Optimizer
Sparse
ResNes-0 50
100
50
1000
1000
0.2
512
0.512
Cusine Decay
SGD
0.9
Ic-4
ERK
ResNes 30
150
120
1000
1500
0.2
512
0,512
Cosite Deaay
SGD
0.9
1e4
ERK
Table 11: Implementation hyperparameters of Chase in Table 13, ImageNet.
5
8
Made
Thes repectel
Teo (epact))
All Homatono
AN, carmaten))
Urmp
Unimier
Mlenemm 6M
Spore
Class-
305
495
Redioe
750
170
1000
2000
03. 518
0413
Crdra Twey
krmh
1ed
906
49%
RoNa.50
Coite
SGD
BRK
16

---

C Real Inference Speedups
We report the real inference latency and throughput of Chase on various sparity Table 12.
Table 12: Real inference latency and throughput of Chase on the ResNee-50/lmageNe benchmark. The best results are marked in bold.
Method
Sc
Accuracy (%) (f)
Throughput (1)
Latency (ms) (1)
Chase
0.9
74.40
69.30
30.89
Chase
0.3
74.35
70.07
30.39
Chase
0,9
0.4
74.03
71.54
30.13
Chase
0,9
0.5
73.39
78.19
27.55
Chase
0.9
0.6
72.85
82,98
26.09
Chase
0.7
71.98
86.26
25.10
Chase (prune skip)
0.9
0.2
74.15
72.57
29.38
Chase (prune skip)
0,9
0.3
74.06
78,30
27.33
Chase (prune skip)
0.9
0.4
73.24
87.38
24.53
Chase (prune skip)
0.5
72.71
21.55
Chase (prune skip)
0,9
0.6
71.62
107.99
20.03
Chase (prune skip)
0.9
0.7
67.15
123.30
17.56
Chase
75.82
32.05
Chase
0.3
75.62
69.30
30.81
Chase
0.8
0.4
75.27
73.32
29.22
Chase
0.8
0.5
74.76
76.68
28.05
Chase
0.8
0.6
73.77
80,51
26.81
Chase
0.7
72.88
86.12
25.15
Chase (prune skip)
0.8
0.2
75.27
72.47
29.38
Chase (prune skip)
0.8
0.3
74.96
78.60
27.20
Chase (prune skip)
0.8
0.4
74.58
87.91
24.38
Chase (prune skip)
0.5
73.53
98.43
21.86
Chase (prune skip)
0.8
0.6
71.70
104.82
20.58
Chase (prune skip)
0.8
0.7
67.53
123.46
17.57
D Comparisons with SOTA Channel Pruning Methods
We further compare Chase with various state-of-the-arr channel pruning approaches in Table 13. is encouraging to see that Chase performs on par with state-of-the-art SOTA channel pruning approaches, such as Group Fisher [30], CafeNet-R [51], and CHIP [52], without the need for the costly dense pretraining step. The implementation details are reported in Table 13.
Table 13: Comparison with state-of-the-an art channel pruning methods popular benchmark: ResNet- 50 on ImageNet.
Methods
FLOPs
Top-1
Epochs
[57]
2.4G
76.2%
350
LEGR [4]
2,4G
75.7%
FPGM [16]
2.4G
75.6%
200
TAS [7]
2.3G
76.2%
240
Hrank
2.3G
75.0%
570
SCOP [54]
2,20i
76.0%
230
CHIP [52]
2.2G
76.3%
Group Fisher [30]
2.0G
76.4%
AutoSlim [58]
2.0G
75.6%
Uniform
2.0G
75.1%
300
Random
CafeNet-R [51]
2.0G
76.5%
300
Chase-1
1.5G
76.6%
250
Uniform
L.0G
73.1%
300
Random
1.0G
72.2%
300
Group Fisher [30]
1.0G
73.9%
CafeNet-R [51]
1.0G
74.9%
300
CafeNet-E [51]
1.0G
75.3%
300
Chase-2
0.9G
75.7%
250

---

E Existence of Sparse Amenable Channel in Various Settings
To demonstrate the broad prevalence sparse amenable channels across diverse architectures and datasets, we have evaluated their ratio in multiple scenarios, including ResNet-32/VGG-11 on CIFAR-100. 100, ResNet-50/ViT-smalll on ImageNet, an MLP model on CIFAR-10.
The MLP model consists of two hidden layers, each with 512 neurons. For the ViT small model, we focus attention the neurons within the MLP layers that exhibit suitability for pruning. In all cases, the sparse amenable channels are identified by Unmasked Mean Magnitude (UMM), with the threshold, v, set to 20%.
The results, shown in the corresponding tables, underline the consistent presence sparse amenable channels across various architectures and datasets, reinforcing the argument that the phenomenon is both significant and widespread.
Table 14: Sparse amenable channel portion during training on various settings
Semings
Layer
Epoch
20 Fjoch
40 Fpoch
Epoch
32CNEA 100
blocks.3.com/
0.23
0.38
0.38
0.63
blocks.5.coom 5.coev
0.15
0.19
0.28
0.32
VGG-16CTTAR--I0
Tentures D
0.11
0.22
0.30
0.39
0.19
0.20
Settings
Layer
Epoch
10 Epoch
Epoch
50 Epoch
RoNNe--OOtmgeNNet
layer3.L.oww Loom?
0.14
0.18
0.21
0:33
laycri.Leom Leoms
0:44
0.49
0.19
0.56
ViT-Small/EngeNN
moocks...ml..fc/
blocks. &mmlp.fl
0.20
0.22
0.21
Settings
Layer
5 Epoch
10 Epoch
20 Epoch
50 Epoch
ML. ModelCifarrO
Tel
0.21
0.21
0.30
0.37
f/2
0.42
0.49
0.63
18

---

Addressing the Memory Limitation of Global Parameter Exploration
During global parameter exploration, directly loading all the parameters for gradients/magnitude sorting is comooryconsuumig. Inspired by [40], we layer-wisely select parameters with the largest gradients for growth and the lowest magnitude for pruning by an adaptive global threshold H, until reaching the target sparsity. H is determined by a set point negative feedback loop to maintain an approximate parameter amount during each reallocation step, as reported below.
Algorithm 2:
Overview of Global Parameter Exploration
Input: Network with sparse weight 0s. target sparsity Sp, current sparsity 8t, prune magnitude threshold Hp. grow gradient threshold Hg. threshold incremental value H., sparsity tolerance
Output: A sparse model 0g satisfying the target sparsity 8p.
Initialize grow threshold H while not 8, + 85 28t >8p 86 do for each sparse tensor 0: of layer do
Begin global parameter growing
L (6', + grow.by.threshold((... Hg)
D gigi is the number of pruned weights in layer I G+ G+E, -. 5.... < calculate weatcelatecuurrnt..co wwrrent.sparsity(C. b. total number of grown weights in all layers
if <sp 88 then
L Hg (Hg Hi)
st >8p + 85 then Hg - (H,- H:)
b Update the grow threshold
Initialize prune threshold Hp while not Sp 85 2St 28p 95 do
Begin global parameter pruning > pi is the number of pruned weights in layer P is the total number of pruned
for each sparse tensor 6¹ of l
(0.pv) pi) + www.eebyythreeeteeee.. (0), H,)
P + Σ. Pt 8t +- calculate current sparsity(P)
weights in all layers
if st <s 84 then
Hp (Hp H.)
if st >8p + 85 then H, (Hp H.)
b Update prune threshold
19

---

G Learned Layerwise Sparsity
Table 15 summarize the final learned layer-wise sparsity on ResNet-50 under 80% sparsity and 40% channel-wise sparsity. The model is obtained by training 100 epochs on ImageNet-IK, The parameter- wise sparsity represents the sparsity budgets for all the CNN layers without the last fully-connected layer. The channel-wise sparsity denotes the sparsity of all the CNN internal layers (the first two convolution layers in the bottleneck blocks).
Table 15: ResNet-50 learnt budgets using Chase at 80% and channel-wise sparsity at 40%.
ResNee-50
Phrass
Weigits Dimension ofter
Dimension
Prrsmitat-aise Sparitty
Channel-wisc Spuritty
Rackbene
23454012
5.11
Layer layer
Layes layer 10
Layer layer! 0
16384
256
256 64
0.00
Layer layerl Adhonnample.
16384
256
64
65.5
0.00
Layer layer1. Looavl
16384
64 x 256
63 x 256
x
53.13
1.56
Laya: 7- loyee/ 1ooow2
36864
x 64
3
62 63 3
42.51
3.13
Layer 8- loyerl 3
16384
256 x 61
256 x 62
36-48
0.00
Layer 9- 0-yearl 2.00nv]
16384
64:5 x 256
63 x 256
48.64
1.56
Layer 10 layer] 1.com22
36864
64 x 61
64 63 x
52.23
0.00
Laya: 11- layer1 2.cond
16384
256
256 x 64
48.68
0.00
Layer 12- 1.yer20..omv
32766
256
128 258
32.51
0.00
Layer 13- 1.om220.omm
147456
127 128
71.73
0.78
layer20..omv3
65536
512
54.16
0.00
1aya -.w.ar2...wwwampp..........
86.72
8.00
layer2.1.com.
Layes 1ayer2.1.cow2
117
73.93
8.59
Layer layer2.1.cmw
512
69.08
Layer layer222.com comr
100
17.19
Layer kayer222.cmm cone2
147456
128 125
108 106
74.98
1562
Laye: 21 1ayea22222...mm coar3
63533
512 x 125
512
108
74.511
11000
Layer 22 layer2.3.com ponml
65536
128 x 512
122 512
78.87
4.69
Layor 23-12gr2.311111 l1w.ee22....... 1om0
147456
128 x 126
5 3
96 x 122 X x3
78.88
25.00
Layer 24111-y-123..omm comm)
65536
512 125
512 96
71.11
0.00
Laye: 25 -syyy333...oor
131072
256 512
250 122
1
57.36
0.39
Layer 26 laycr3.0.omm2
589824
266
X 236
261 255
85.19
5,86
Laye 27- 2 ..yer3..ownc
262144
1024
256
004 241
65.11
0.00
Layer 28 1xye330.dnncmppp
524288
1024 512
1024
97.36
0.00
262144
1022
1024
K5.45
1ayer3.1.comv2
589424
256
108 151
75.4H
46.14
1sya -byee33..oow
76.19
layer3.2.omm conv
x 1024
63.22
Layas layer3.2.oom
589424
108
77.29
57.81
Layer layer32.com3 cone
262146
1024
70.97
8100
1ayer 1ayer3.1.om conel
262144
1021
84.04
Layer 36 1ayer33.pomr2
589824
256 256
x
74 91 x
63.20
71.89
Layo: 57- 3.per3.1.omm
262144
10214 x 256
x
1024
x
78.42
8.00
Layer 38 11yer33334oo
2621
256 x 1021
76 1024
x
84.05
70.31
Layer 59- 11--1yr33...om
589924
266 x 256
33 x
41 76
x
56.02
83.08
Layer 60 come
262144
1024 x 256
1024 41
73.70
0.00
Leya 41. layar..5.00 co
262144
256 1024
100 X 1122
83.33
$8.59
Layer 42-
1.yer35 5.0m2
589824
256 256 x
41 105
63.34
R2,81
Layer 43
lyyer35 3
262144
1024 256
1024 44
66.12
0.00
Layer 64 layert.0.covv
524288
1024
74.52
2.54
Laya D.yw24.
2399296
88.06
Lager layers Dooome)
Layer layee4 d.wwenndd
99.55
0.00
x 20445
2045
84.56
71.09
Layos layes4
88
62.39
82.23
Layer layers. comd
1088576
2048
2065
72.09
0.00
Layer layerrk convl
1048576
312 x 2015
x 2045
82.31
4.49
Layer 52 layert-2 cons2
2599296
512 x 512
x
292 089
87.01
42.97
Layor 53- 33-1aycs4 layccsss22 comec
1048576
2048 x 512
2045 x 252 x
62.42
0.00
Layer 54-1c
2055000
1000 2045
1000 2018
61.82
0.00
20


Reviews:
[{"review_id": "fwwXrteCwT", "reviewer": "Reviewer_bsGW", "summary": "The paper introduces a novel approach called Channel-aware dynamic sparse (Chase) for training deep neural networks with high sparsity. Unlike previous techniques that focus on unstructured sparsity with irregular patterns, Chase seamlessly translates the benefits of dynamic sparsity to channel-level sparsity, which is more hardware-friendly. This is achieved through an end-to-end training process without requiring additional ad-hoc operations or specialized hardware accelerators. By identifying and removing sparser channels during training, Chase enables small sparse networks that can be efficiently accelerated on commodity GPUs. Experimental results show that Chase achieves a significant 1.7× inference throughput speedup without sacrificing accuracy when applied to ResNet-50 on ImageNet. The authors plan to release their code to facilitate further research and adoption of their approach.", "questions": "Please carefully address the weaknesses listed above in the rebuttal.", "limitations": "The paper lacks an adequate discussion on the limitations or potential negative societal impact.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper exhibits exceptional clarity and a well-structured format. I particularly appreciate the bullet-point summary in the introduction, which effectively outlines the paper's roadmap. However, it must be noted that the paper can be somewhat dense and requires some additional time to fully comprehend.\n\nThe proposed method is technically sound and yields commendable empirical results. The authors offer ample implementation details that would greatly facilitate other researchers and practitioners in reproducing the key findings outlined in the paper. It is worth mentioning that the authors have also committed to releasing the code upon the acceptance of the paper, further enhancing its accessibility and potential for future advancements.", "weaknesses": "The paper includes experimental results that demonstrate the improvement in inference speed. However, it would be valuable to clarify whether the proposed method also enhances the training efficiency of the model. From my understanding, the primary advantage of employing dynamic sparse training lies in improving training efficiency. Otherwise, one could simply train a dense model initially and subsequently prune it to achieve sparsity. By addressing this aspect, the authors can provide a more comprehensive understanding of the benefits of their proposed method.\n\nThis paper primarily conducts its main experiments and ablation studies using CIFAR-10, which is widely regarded as a small-scale dataset. Consequently, the results obtained from this dataset may not be entirely representative. It would significantly enhance the robustness of the findings if the authors could replicate these experiments using the larger-scale ImageNet dataset. Additionally, it would be highly advantageous if the authors could extend their validation to encompass other models and tasks, such as object detection and semantic segmentation, in order to establish the general effectiveness of their proposed method across diverse domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing that our proposed method is technically sound and yielding commendable empirical results. And we are glad that the reviewer appreciates the exceptional clarity and well-structured format of our paper. \nTo further address all the weaknesses pointed out by you, we would like to address them point-by-point as below. \n\n**Comment 1: . Clarify whether the proposed method also enhances the training efficiency of the model.**\n\n- Please refer to our global responses.\n\n\n**Comment 2: Replicate these experiments using the larger-scale ImageNet dataset**\n\n- Thank you for your constructive feedback. We fully agree with you that evaluating our approach on large-scale ImageNet is significant. In addition to the CIFAR-level experiments, we actually have conducted numerous experiments with ResNet-50 on ImageNet, such as  Tables 4 \\& 5 in the main paper, and Figure 5 and Table 8 in the appendix. Specifically, the primary evaluation considering various sparsity levels against other strong baselines is presented in Table 4 and the corresponding inference speed-ups are shared in Table 5. Besides, we've included detailed results capturing a wider range of channel-wise sparsity (from 0.2 to 0.7) with ResNet-50/ImageNet in Figure 5 and Table 8,  in Appendix B.\n\n- Moreover, as you requested, we further added the ablation study on ImageNet with ResNet-50 to further enhance the robustness of our approach. The results are highly in line with the results of CIFAR, demonstrating the strong robustness of our approach.\n| Sparsity | 90%     | 80%     |\n|:----------|---------|---------|\n| RigL     | 73.00   | 75.10   |\n| RigL+GVCP ($S_c=0.4$) | 72.11  | 73.75  |\n| RigL+GVCP+SM ($S_c=0.4$) | 73.08 | 74.18  |\n| RigL+GVCP+SM+GE (Chase) ($S_c=0.4$) | **74.03** | **75.27** |\n\n**Comment 3: Extend the validation to encompass other models and tasks.**\n\n- In response to this valuable feedback, we have expanded our evaluation to the domain of image segmentation. Specifically, we employed the DeepLab V3+ on the PASCAL VOC 2012 dataset. Staying in line with the original paper's approach that pre-trained a dense ResNet-50  on ImageNet to derive dense feature maps via atrous convolution,  we pre-trained an 80\\% sparse ResNet-50 with Chase at various channel-wise sparsity levels, choosing RigL as our most direct baseline. The results are presented below, $Sc$ indicates channel-wise sparsity:\n| Method              | M-IoU (↑)    |\n|---------------------|--------------|\n| RigL                | 0.693       |\n| Chase (Sc=0.2)      | **0.697**   |\n| Chase (Sc=0.3)      | 0.696       |\n| Chase (Sc=0.4)      | 0.693       |\n| Chase (Sc=0.5)      | 0.686       |\n| Chase (Sc=0.6)      | 0.683       |\n\n- Our results show that Chase can match RigL's performance in DeepLab V3+ segmentation using only 60\\% channels. This result again provides strong evidence to support the versatility of our method.\n\n\n\n**Comment 4: Lacks an adequate discussion on the limitations.**\n\n       \n- We appreciate the reviewer pointing this out. One limitation of our method is its dependence on Chase's gradual channel pruning, which gradually prunes channels to reach the desired channel-sparsity. As a result, the peak memory requirement remains equivalent to that of a dense network. Moving forward, we're considering the integration of one-shot channel pruning prior to training, which we anticipate could greatly enhance efficiency.\n\n- Additionally, for the sake of convenience and ease of implementation, we currently employ masks for channel pruning during training. To truly realize training speed-up, more engineering efforts are warranted, which we plan to undertake in future iterations\n"}, {"review_id": "JqX5Uh1pWa", "reviewer": "Reviewer_VN42", "summary": "This paper proposes Channel-aware dynamic sparse, which translates unstructured dynamic sparsity to GPU-friendly channel-level sparsity during one end-to-end training process without using any particularly sparsity-aware hardware accelerators. The authors also observe the sparse amendable channels and proposed a new metric - Unmasked Mean Magnitude. Experiments demonstrate that the proposed method achieves 1.2x - 1.7x inference throughput speedups on common GPU devices with less to no accuracy degradation. ", "questions": "The experiments use 50% channel sparsity. Why 50%? ", "limitations": "I don't find a discussion on the limitations. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and gives sufficient experiments.\n2. This paper observes the channel-wise sparsity during dynamic sparse training and proposes a framework Chase to translate unstructured dynamic sparsity to channel-level sparsity with less to no accuracy loss. ", "weaknesses": "1. The proposed method does not seem to have much performance benefit for training compared to unstructured methods. The computation is still unstructured in most iterations. \n\n2. In Figure 3-b, the accuracy of RigL+GACP and RigL+GACP+SM is lower than RigL for ResNet-50 with sparsity 90% on CIFAR-100. While I understand GACP improves efficiency, it is not a good way to show the advantage of GACP as accuracy drops. It will be more convincing if you can show the runtime at the same accuracy or the accuracy with the same runtime.  \n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe would like to thank you for the time to review our work and are glad that you find our work is well-written with sufficient experiments. We would like to address all the weaknesses pointed out by you below:\n\n**Comment 1: . The proposed method does not seem to have much performance benefit for training compared to unstructured methods. The computation is still unstructured in most iterations..**\n\nThank you for providing us with such valuable and constructive comments, which certainly contributed to enhancing the positioning of our paper. We provide our discussion below.\n\n\n- Please allow us to clarify that the channel pruning of Chase starts to perform at the very beginning of training, i.e., the **first epoch** for ResNet-50/ImageNet, thanks to the emergence of channel-wise sparsity implicitly produced by DST. To provide further evidence, we accumulate the reduction of training FLOPs solely caused by channel pruning, without considering unstructured/weight sparsity. With a 40\\% channel sparsity, Chase is able to achieve a 1.4$\\times$ training speedup when training ResNet-50 on ImageNet for 100 epochs. We expect to receive a higher speedup if we consider the benefits of unstructured sparsity together, which is widely accepted by the  community.\n\n\n**Comment 2: In Figure 3-b, the accuracy of RigL+GACP and RigL+GACP+SM is lower than RigL for ResNet-50 with sparsity 90\\% on CIFAR-100. While I understand GACP improves efficiency, it is not a good way to show the advantage of GACP as accuracy drops. It will be more convincing if you can show the runtime at the same accuracy or the accuracy with the same runtime.**\n\n\n- We really appreciate your constructive comment, which indeed helps us demonstrate the advantages of our approach. **As you suggested, we've detailed the throughput-accuracy trade-offs for different methods in Figure 1 of the rebuttal pdf.** High throughput indicates shorter runing time. To draw a more strong conclusion, we added a new ablation study with ResNet-50 on ImageNet. We can see that at the same accuracy, RigL+GACP offers around 30\\% throughput gains compared to RigL, while maintaining comparable performance. RigL+GACP+SM is able to push the throughput gains to 37\\% and the full version of Chase leads to a 60\\% reduction eventually, demonstrating the advantages of our approach.\n\n**Comment 3 : The experiments use 50\\% channel sparsity. Why 50\\%?**\n\n\n- Thank you for your question. I'd like to clarify that our submission goes beyond a mere selection of a 50\\% sparsity level. In fact, we have reported a diverse spectrum of sparsity values, ranging from 20\\% to 70\\%, across various experiments involving CIFAR and ImageNet, depicted in Figure 2, Figure 5 (Appendix), and Table 8 (Appendix).\n\n- We choose 50\\% channel sparsity in Table 4 as we firmly believe that a 50\\% sparsity level effectively showcases Chase's capability to eliminate a significant proportion of channels, thereby substantiating its efficiency, which is widely used in channel pruning papers [1,2,3].\n\n[1] He, Yihui, Xiangyu Zhang, and Jian Sun. \"Channel pruning for accelerating very deep neural networks.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n[2] Luo, Jian-Hao, Jianxin Wu, and Weiyao Lin. \"Thinet: A filter level pruning method for deep neural network compression.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n[3] Raihan, Md Aamir, and Tor Aamodt. \"Sparse weight activation training.\" Advances in Neural Information Processing Systems 33 (2020): 15625-15638.\n\n\n**Comment 4: I don't find a discussion on the limitations**\n\n- We appreciate the reviewer pointing this out. One limitation of our method is its reliance on Chase's gradual channel pruning, which gradually trims channels until the target channel sparsity is achieved. Consequently, the maximum memory usage is on par with a fully dense network. In future iterations, we are exploring the incorporation of one-shot channel pruning before training, which could significantly improve efficiency.\n\n"}, {"review_id": "Kb5oSw6uaF", "reviewer": "Reviewer_G8BR", "summary": "This paper proposes a new approach to dynamic sparsity called Channel-aware dynamic sparse (Chase), which gradually translates unstructured dynamic sparsity to GPU-friendly channel-level sparsity during one end-to-end sparse training process. ", "questions": "Please see above weakness. ", "limitations": "The authors shall discuss more limitations of their method ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe technical novelty of Chase mainly lies in two aspects: on the structured sparsity level, it adopts the gradual sparsification schedule to gradually remove Amenable Channels during training with smallest UMM scores, and on the unstructured sparsity level, it globally redistributes parameters based on their magnitude and gradient, which significantly strengthens the sparse training performance. I am a fan of this main idea.\n\n-\tChase differs from other approaches to dynamic sparsity in that it seamlessly translates unstructured dynamic sparsity to GPU-friendly channel-level sparsity during one end-to-end training pass, without any ad-hoc operations. Therefore, it is designed to be a more efficient and effective approach to dynamic sparsity that can be accelerated by commodity hardware. This is in contrast to other approaches demonstrated on unstructured sparsity which receive limited support in common hardware. \n\n-\tOne of the main new observations is that sparse amenable channels, which are channels that rapidly become sparser than others at the very early training stage, cause marginal damages to the model performance than their counterparts when pruned. The paper proposes a new metric called Unmasked Mean Magnitude (UMM) that can be used to precisely discover sparse amenable channels during training by monitoring the quantity and quality of weight sparsity. This finding reveals that a more nuanced approach to channel pruning can be more effective in achieving sparsity in neural networks.\n\n-\tChase achieves up to 1.7× inference throughput speedup on common GPU devices without compromising accuracy with ResNet-50 on ImageNet. Even without pruning skip connections, the model is able to provide 31% throughput speedups, while outperforming the SOTA RigL by 0.39%. Additionally, Chase establishes a new SOTA performance bar for unstructured sparse training, outperforming all the presented DST methods by good margins. \n", "weaknesses": "-\tChase appears to be a mixture between “unstructured sparse training” and “structured sparse inference”. So, what is the benefit of Chase over early-stage structured pruning methods, such as Early Bird lottery ticket? How about its comparison to N:M dynamic sparse training, which can be done now from end to end? And how about instead just training a slimmer dense network from scratch? Those important questions are not discussed.\n\n-\tExperiment choices seem flawed to me. Both VGG19 and ResNet50 are known to be (ridiculously) over-parameterized for CIFAR-10/100. Comparison results on those testbeds cannot be reliable for sparsity algorithms: for CIFAR perhaps ResNet-32 or Res-20 would have made more sense. The only persuasive setting in this paper is ResNet 50 + ImageNet.\n\n-\tImportant detail is missing on how “GPU-supported FLOPs” is computed? On what GPU? By channel level or N:M level? Also, how is the “Throughput” computed in Table 5?\n\n-\tChase requires additional memory to store the mask for each channel. This can be a limiting factor for models with a large number of channels, as the memory requirements can become prohibitively large.\n\n-\tAdditionally, the paper does not provide a detailed analysis of the end-to-end computational cost of Chase compared to other dynamic sparse training methods, which could be an important consideration for practical applications.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe greatly appreciate your positive feedback and we are glad that you like our idea. We provide the following response to address your concerns point-by-point.\n \n\n**Comment 1: Chase appears to be a mixture between “unstructured sparse training” and “structured sparse inference”. So, what is the benefit of Chase over early-stage structured pruning methods, such as Early Bird lottery ticket? How about its comparison to N:M dynamic sparse training, which can be done now from end to end? And how about instead just training a slimmer dense network from scratch? Those important questions are not discussed.**\n\nThank you for providing us with such valuable and constructive comments, which certainly contributed to enhancing the positioning of our paper. We provide our discussion below.\n\n- Chase is essentially a sparse-to-sparse training approach that does not require any pre-training or dense training steps, i.e., starting from unstructured sparse and ending up with structured sparse. In contrast, the Early Bird Lottery Ticket (EB tickets [1]) follows a dense-to-sparse training paradigm. Despite the fact that the winning ticket can be discovered much earlier than the original Lottery Ticket approach, a certain duration of dense training is necessitated—typically ranging from 6\\% to 20\\% of the full training time [1]. In addition, when initiating from an 80\\% unstructured ResNet-50, Chase surpasses EB tickets by a significant margin of 2.01\\% (achieving 75.87\\% accuracy compared to 73.86\\%) under a 30\\% channel sparsity, showing stronger results.\n- Chase presents a more hardware-friendly alternative in comparison to the N:M sparsity technique. While the N:M approach holds substantial promise, it remains confined to a select range of hardware, specifically the NVIDIA Ampere Architecture. Furthermore, only the 2:4 pattern is currently amenable to acceleration. Conversely, Chase employs a channel-wise sparse pattern, which seamlessly interfaces with readily available hardware, obviating the need for specialized sparsity-aware accelerators.\n    \n- It is commonly observed that a slimmer dense network, often referred to as a ``small dense`` model, typically falls short in performance compared to a larger sparse model trained through RigL [2]. However, Chase consistently performs on par, or even better than RigL with up to 1.7× inference throughput speedup, as exemplified in our submission. Consequently, Chase offers substantial accuracy advantages over a slimmer dense network.\n\n\n\n[1] You, Haoran, et al. \"Drawing early-bird tickets: Towards more efficient training of deep networks.\" arXiv preprint arXiv:1909.11957 (2019).\n\n[2] Evci, Utku, et al. \"Rigging the Lottery: Making All Tickets Winners. arXiv e-prints, art.\" arXiv preprint arXiv:1911.11134 (2019).\n\n\n**Comment 2: For CIFAR perhaps ResNet-32 or Res-20 would have made more sense**\n\n- Thanks for your suggestion.  We want to highlight that we have actually included the results of ResNet-20 on CIFAR-10/100 in our submission at the bottom of Table 3. **We also added extra experiments of ResNet-32 and report all the results in Table 1 of the rebuttal pdf for your convenience.** As we can see, Chase again significantly outperforms RigL at all sparsity levels, demonstrating its effectiveness on smaller architectures. We will highlight the results of ResNet-20 and ResNet-32 in the camera-ready version.\n\n\n\n**Comment 3: Important detail is missing on how “GPU-supported FLOPs” is computed? On what GPU? By channel level or N:M level? Also, how is the “Throughput” computed in Table 5**\n\n\n- Thanks for your asking.  \"GPU-supported FLOPs\" refers to the Floating Point Operations (FLOPs) that are measured on common GPUs without any specific sparsity-aware accelerators where channels containing unstructured sparsity are usually treated as a dense channel. This approach mirrors the inference of sparse models on common GPUs without any specific sparsity-aware accelerators.\n\n- The throughput, denoted as images/second, indicates the number of images processed by models per second during inference [1, 2]. It was measured on an NVIDIA 2080TI GPU device, using float32 precision within PyTorch. \n\n\n[1] Liu Z, Mao H, Wu C Y, et al. A convnet for the 2020s, CVPR. 2022\n\n[2] Liu Z, Lin Y, Cao Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows, CVPR 2021: \n\n**Comment 4: Chase requires additional memory to store the mask for each channel. This can be a limiting factor for models with a large number of channels, as the memory requirements can become prohibitively large.**\n\n- You are correct that Chase requires storing the mask for each channel. Fortunately, since additional masks are already used by other sparse training approaches such as RigL and SET, we directly leverage those masks to perform channel pruning without using additional memory. Therefore, we would like to clarify that Chase does not need extra memory to store masks compared to existing sparse training approaches.\n\n\n\n\n**Comment 5: Detailed analysis of the end-to-end computational cost of Chase compared to other dynamic sparse training methods**\n\n- Please refer to our global responses.\n\n\n**Comment 6:  The authors shall discuss more limitations of their method**\n\n- We appreciate the reviewer pointing this out. One limitation of our method is its dependence on Chase's gradual channel pruning, which gradually prunes channels to reach the desired channel sparsity. As a result, the peak memory requirement remains equivalent to that of a dense network. Moving forward, we're considering the integration of one-shot channel pruning prior to training, which we anticipate could greatly enhance efficiency.\n\n- Additionally, for the sake of convenience and ease of implementation, we currently employ masks for channel pruning during training. To truly realize training speed-up, more engineering efforts are warranted, which we plan to undertake in future iterations\n"}, {"review_id": "VznqcOglQX", "reviewer": "Reviewer_DQ24", "summary": "This paper investigates practical deployment and speedup of Dynamic Sparse Training methods on general hardwares. The author's motivation stems from observations that many channels naturally exhibit high sparsity rates. Based on this finding, the authors propose a sparse-inducing method to gradually remove entire channels during sparse training. This ultimately produces a network with certain structured sparse characteristics, enabling favorable acceleration. The authors validate the effectiveness of their proposed method through experiments on extensive datasets.", "questions": "Regarding weight regrowing, I understand that the author selects weights with the highest magnitude globally for restoration. How about restoring weights with the highest UMM in each channel? Would this allow the network to gradually learn to become completely structured, i.e., some channels are automatically removed, while important channels are all preserved?", "limitations": "n/a", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The motivation is highly reasonable, and the proposed method has great application prospects, as it overcomes the bottleneck of unstructured sparsity being difficult to deploy on off-the-shelf hardware.\n2. The observation on the sparsity distribution across channels of DST is novel and provides inspiration for the community.\n3. The proposed method is easy to implement and even surpasses traditional non-structured pruning methods while achieving practical GPU acceleration.\n", "weaknesses": "1. Given the author's focus on practical acceleration in the pruning field, it would be interesting to compare this method with structured pruning methods [1,2] even after obtaining accelerated and deployable networks.\n\n[1] Pruning filters for efficient ConvNets. In ICLR, 2017.\n\n[2] Channel Pruning via Automatic Structure Search. In IJCAI, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe sincerely appreciate your encouraging feedback. It's heartening to hear that you find the motivation compelling and recognize the potential application of our method. We are especially grateful for your acknowledgment of the novelty in our observation on sparsity distribution and its potential to inspire the community.  We provide the following response to address your concerns point-by-point: \n\n**Comment 1:  Given the author's focus on practical acceleration in the pruning field, it would be interesting to compare this method with structured pruning methods [1,2] even after obtaining accelerated and deployable networks.**\n\n\n- Thanks for bringing these two interesting works, we compare Chase with these two baselines in the following table. It could be seen that Chase consistently outperforms these two baselines while achieving much fewer inference FLOPs, even though being a sparse-to-sparse training method.\n| Model     | Approach                     | Top-1 Acc (\\%) | Inference FLOPs ×(10e9) |\n|:-----------|------------------------------|----------------|-------------------------|\n| ResNet-50 | Chase ($S_c=0.4$)            | **74.34**      | 1.80                    |\n|  ResNet-50   | ABC Pruner [Lin et al.]  | 73.52          | 1.79                    |\n| ResNet-34 | Chase ($S_c=0.4$)            | **72.61**      | 1.70                    |\n|  ResNet-34  | ABCPruner [Lin et al.]       | 70.98          | 2.17                    |\n| ResNet-34  | Efficient Convnets [Li et al.] | 72.56        | 3.08                    |\n\n[1] Li H, Kadav A, Durdanovic I, et al. Pruning filters for efficient convnets.In ICLR, 2017.\n\n[2] Lin M, Ji R, Zhang Y, et al. Channel pruning via automatic structure search .In IJCAI, 2020.\n\n\n**Comment 2: Regarding weight regrowing, I understand that the author selects weights with the highest magnitude globally for restoration. How about restoring weights with the highest UMM in each channel? Would this allow the network to gradually learn to become completely structured, i.e., some channels are automatically removed, while important channels are all preserved?**\n\n\n- We appreciate your insightful suggestion. Targeting channels with higher UMM values could indeed be a promising strategy for restoring weights.\n\n- After seeing your great suggestion, we make a deep dive into how the new weights are restored during training and surprisingly found that the majority of new weights are actually restored from the channels with high UMM scores.  To demonstrate this, we conducted an experiment with the following settings: we utilized Global gradients as the criteria to train a 90\\% sparse ResNet-50, emulating the Chase methodology. We first rank channels based on their UMM scores and split them into two categories: Large UMM channels: top-50\\% channels with the highest UMM scores; Small UMM channels: the rest 50\\% channels with the lowest UMM scores. Subsequently, we compute the proportion of newly restored weights within these two categories at different training epochs respectively.\n\n- Our results, detailed in the following table, reveal that Chase actually regrows the majority of the regrown weights (up to 91\\%) in channels with higher UMM scores. This result highly aligns with your idea of restoring weights with the highest UMM scores. We sincerely appreciate your knowledgeable contribution and will leave this as promising future work.\n| Epoch | Weights restored in small UMM channels (\\%) | Weights restored in large UMM channels (\\%) |\n|-------|---------------------------------------------|---------------------------------------------|\n| 20    | 39.06                                       | 60.94                                       |\n| 40    | 38.31                                       | 61.69                                       |\n| 60    | 40.76                                       | 59.24                                       |\n| 80    | 31.60                                       | 68.40                                       |\n| 100   | 25.77                                       | 74.23                                       |\n| 120   | 08.28                                        | 91.72                                       |\n| 140   | 14.27                                       | 85.73                                       |\n| 160   | 19.39                                       | 80.61                                       |\n\n\n\n"}]

Author rebuttal:



We express our gratitude to all reviewers for their insights and will endeavor to address each comment separately.  The reviewers tend to have a concern regarding the training speedup, to which we aim to provide a global response:

- For the sake of convenience and easy implementation, we adopt masks
to perform channel pruning during training. Therefore, our current implementation does not have training speedups. During inference, we reshape the network and completely remove all the channels that are masked out during training, to obtain the real inference speed-ups.  We believe that enabling real inference speed-up of dynamic sparse training (DST) itself is a solid contribution as the existing DST approach usually yields unstructured sparsity which can not be directly accelerated on common hardware. 

- However, our mask implementation strictly follows the real channel pruning setting, that is, when the output channels in
the current layer are removed, the input channels for the next convolutional layer as well as the BatchNorm layers are also masked. In addition, in contrast to soft channel pruning [1], our channel pruning adopts hard channel pruning, i.e., weights of masked channels are completely zeroed-out and receive no gradient updates
during the backward passes. Therefore, we are confident that our approach also holds the promise to achieve real training speed-ups and we just need more engineering effort. We will leave it for future work. 

- To help better understand the benefits of training speedup, we measure the reduction of training FLOPs **solely caused by channel pruning**, without considering any unstructured sparsity. In this context, channels with unstructured sparsity are deemed dense. This evaluation mirrors computational costs on common GPUs, thereby providing a true representation of actual training expenses. With a 40\% channel sparsity, Chase is able to achieve a 1.4$\times$ training speedup when training ResNet-50 on ImageNet for 100 epochs. We expect to receive a higher speedup if we consider the benefits of unstructured sparsity together, which is widely accepted by the community.

[1] He, Yang, et al. "Soft filter pruning for accelerating deep convolutional neural networks." arXiv preprint arXiv:1808.06866 (2018).

Lastly, we again thank all the reviewers and hope our additional results with rebuttal responses can clarify their doubts. We are more than happy to provide any further explanations required.

Sincerely yours

Authors

Meta-review: The paper makes a great case in favor of channel-wise sparsity andthe AC and reviewers all feel that the additional experiments and constructive discussion will greatly strengthen the submission. We therefore urge the authors to incorporate these materials. We also appreciate the honesty of the authors regarding training speedup. Please also include your points on this topic in the manuscript.
Decision: Accept (poster)