Title: Dynamic Sparsity Is Channel-Level Sparsity Learner
Venue: NeurIPS 2023
Accepted: yes
Keywords: dynamic sparsity

dynamic sparse training

sparse training
Abstract: Sparse training has received an upsurging interest in machine learning due to its tantalizing saving potential for both the entire training process as well as the inference. Dynamic sparse training (DST) as a leading approach can train deep neural networks at high sparsity from scratch to match the performance of their dense counterparts. However, most if not all DST prior arts demonstrate their effectiveness on unstructured sparsity with highly irregular sparse patterns, which receives limited support in common hardware. This limitation hinders the usage of DST in practice. In this paper, we propose Channel-aware dynamic sparse (Chase), that for the first time seamlessly translates the promise of unstructured dynamic sparsity to GPU-friendly channel-level sparsity (not fine-grained N:M or group sparsity) during one end-to-end training process, without any ad-hoc operations. The resulting small sparse networks can be directly accelerated by commodity hardware, without using any particularly sparsity-aware hardware accelerators. This appealing outcome is partially motivated by a hidden phenomenon of dynamic sparsity: off-the-shelf unstructured DST implicitly involves biased parameter reallocation across channels, with a large fraction of channels (up to 60%) being sparser than others. By progressively identifying and removing these channels during training, our approach transfers unstructured sparsity to channel-wise sparsity.  Our experimental results demonstrate that Chase achieves 1.7x inference throughput speedup on common GPU devices without compromising accuracy with ResNet-50 on ImageNet. We release our code in https://github.com/luuyin/chase.

Dynamic Sparsity Is Channel-Level Sparsity Learner

Abstract

Sparse training has received an upsurging interest in machine learning due to its tantalizing saving potential for the entire training process as well as inference. Dynamic sparse training (DST), as a leading sparse training approach, can train deep neural networks at high sparsity from scratch to match the performance of their dense counterparts. However, most if not all DST prior arts demonstrate their effectiveness on unstructured sparsity with highly irregular sparse patterns, which receives limited support in common hardware. This limitation hinders the usage of DST in practice. In this paper, we propose Ch annela ware dynamic s pars e ( Chase ), which for the first time seamlessly translates the promise of unstructured dynamic sparsity to GPU-friendly channel-level sparsity (not fine-grained N:M or group sparsity) during one end-to-end training process, without any ad-hoc operations. The resulting small sparse networks can be directly accelerated by commodity hardware, without using any particularly sparsity-aware hardware accelerators. This appealing outcome is partially motivated by a hidden phenomenon of dynamic sparsity: off-the-shelf unstructured DST implicitly involves biased parameter reallocation across channels, with a large fraction of channels (up to 60%) being sparser than others. By progressively identifying and removing these channels during training, our approach translates unstructured sparsity to channel-wise sparsity. Our experimental results demonstrate that Chase achieves 1.7 × inference throughput speedup on common GPU devices without compromising accuracy with ResNet-50 on ImageNet. We release our codes in https://github.com/luuyin/chase .

1 Introduction

Deep neural networks (DNNs) have recently demonstrated impressive breakthroughs with increasing scales [2; 8; 36]. Besides the well-known scaling, i.e., test accuracy scales as a power law regarding model size and training data size in quantity [18; 26], recent work has observed that massive increases in quantity can imbue models with qualitatively new behavior [49]. However, the memory and computation required to train and deploy these large models can be a heavy burden on the environment and finance [13; 43]. Therefore, people start to probe the possibility of training sparse neural networks from scratch without involving any dense training steps (dubbed sparse training [39; 35]). As the memory requirement and multiplications (which dominate neural network computation) associated with zero weights can be skipped, sparse training is becoming a promising direction due to their 'end-to-end' saving potentials for both efficient training and efficient inference.

Sparse training can be categorized into two groups, static sparse training and dynamic sparse training according to the dynamics of the sparse pattern during training. Static sparse training

17.5

20.0

22.5

25.0

27.5

30.0

32.5

35.0

67

68

69

70

71

72

73

74

75

Top-1 Accuracy (%)

Chase (prune skip)

Chase

SET

DSR

SNFS

RigL

Chase (unstructured)

60

70

80

90

100

110

120

Inference Throughput

67

68

69

70

71

72

73

74

75

Top-1 Accuracy (%)

Chase (prune skip)

Chase

SET

DSR

SNFS

RigL

Chase (unstructured)

(SST) [28; 56; 53; 33; 22], namely, draws a sparse neural network at initialization before training and train with a fixed sparse pattern (sparse connections between layers) without further changes. Dynamic sparse training (DST) [39; 10; 35], on the contrary, jointly optimizes the weights and sparse patterns during training, usually delivering better performance than the static ones. DST quickly evolves as a leading direction in sparse training due to its compelling performance and training/inference efficiency. For instance, a sparse ResNet-34 with only 2% parameters left can be dynamically trained to match the performance of its dense counterpart without involving any pre-training or dense training [35].

While showing promise in performance and efficiency, so far the real speedup of DST has only been demonstrated on CPU [34; 6] or IPU [6]. Most sparse training methods produce unstructured sparse neural networks with extremely irregular sparse patterns, which can not be directly accelerated in common hardware (i.e., GPU and TPU), compared to the straightforward and hardware-friendly sparse pattern produced by channel pruning [17; 37].

Many endeavors strive to solve this issue by coarsening the sparsity granularity, which can be loosely categorized into two groups. i. Grouping nonzero weights into blocks. As GPU performs very fast on contiguous memory operations, block-wise sparsity enjoys much more speedups than unstructured sparsity in practice. Group lasso regularization [41; 14] is a widely-used technique to induce block sparsity in the network. Ad-hoc grouping operations can also be utilized to build dense blocks from unstructured sparse weights [46; 3]. ii. Seeking fine-grained structured sparse patterns. For instance, inspired by the recent support of 2:4 fine-grained sparsity in NVIDIA Ampere [42], previous arts attempt to find a sweet spot between structured and unstructured sparsity by learning N:M sparsity patterns [60; 20; 45]. However, these methods either rely on specialized sparse-aware accelerators [9; 42] to enable speedups or suffer from significant performance degradation due to the constraint location of nonzero values [25].

In this paper, we propose a new method dubbed Ch annela ware dynamic s pars e ( Chase ), which can effectively transfer the promise of unstructured sparse training into the hardware-friendly channel sparsity with comparable or even better performance on common GPU devices. The roadmap of our exploration is as follows:

■ Observation 1: We first present an emerging characteristic of DST: off-the-shelf DST approaches implicitly involve biased parameter reallocation, resulting in a large proportion of channels (up to 60%) that rapidly become sparser than their initializations at the very early training stage. We term them as 'sparse amenable channels' for the sake of convenient reference.

■ Observation 2: We examine the prunability (i.e., the accuracy drop caused by pruning) of the sparse amenable channels, we find that these channels cause marginal damages to the model performance than their counterparts when pruned.

■ A New Metric: We propose a new, sparsity-inspired, channel pruning metric - Unmasked Mean Magnitude (UMM) - that can be used to precisely discover sparse amenable channels during training by monitoring the quantity and quality of weight sparsity.

■ A New Approach: Based on the above findings, we propose Ch annela ware dynamic s pars e ( Chase ), a first sparse training framework that can favorably transform unstructured sparsity into channel-wise sparsity on the fly. Chase starts with an unstructured sparse neural network and dynamically trains it while gradually eliminating sparse amenable channels with the lowest UMM scores. During training, we globally grow and shrink parameters to strengthen performance further.

■ Performance: Chase inherently can be tailed into an unstructured sparse training approach and a structured sparse training approach. Our unstructured variant establishes a new stateof-the-art accuracy bar for sparse training. More impressively, our structured approach is able to maintain or even surpass SoTA performance with ResNet-50 on ImageNet, while achieving 1.2 × - 1.7 × inference throughput speedups on common GPU devices.

2 Sparse Amenable Channels in DST

We first describe the basis and notations of the prior sparse training arts. Afterward, we provide evidence for the existence of the sparse amenable channels during the dynamic sparse training across different architectures and demonstrate that pruning of such channels leads to marginal performance damage than their counterparts. Based on this interesting finding, we introduce Chase, a sparsityinspired sparse training method that for the first time translates the theoretical promise of sparse training into GPU-friendly speedup, without using any specialized CUDA implementations.

2.1 Prior Sparse Training Arts

Let us denote the sparse neural network as f ( x ; θ s ) . θ s refers to a subset of the full network parameters θ at a sparsity level of (1 -∥ θ s ∥ 0 ∥ θ ∥ 0 ) and ∥ · ∥ 0 represents the ℓ 0 -norm.

It is common to initialize sparse subnetworks θ s randomly based on the uniform [40; 5] or nonuniform layer-wise sparsity ratios with Erd˝ os-R´ enyi (ER) graph [39; 10; 35; 31]. In the case of image classification, sparse training aims to optimize: ˆ θ s = argmin θ s ∑ N i =1 L ( f ( x i ; θ s ) , y i ) using data { ( x i , y i ) } N i =1 , where L is the loss function. Static sparse training (SST) maintains the same sparse network connectivity during training after initialization. Dynamic sparse training (DST), on the contrary, allows the sparse subnetworks to dynamically explore new parameters while sticking to a fixed sparsity budget. Most of the DST methods follow a simple prune-and-grow scheme [39] to perform parameter exploration, i.e., pruning r proportion of the least important parameters based on their magnitude, and immediately grow the same number of parameters randomly [39] or using the potential gradient [10]. Formally, the parameter exploration can be formalized as the following two steps:

where Ψ is the specific pruning criterion and Φ is growing scheme. These metrics may vary from sparse training method to another. In addition to prune-and-grow, previous work [23; 47] dynamically activates top-K parameters during forward-pass while keeping a larger number of parameters updated in backward-pass to get rid of dense calculation of gradient. At the end of the training, sparse training can converge to a performant sparse subnetwork. Since the sparse neural networks are trained from scratch, the memory requirements and training/inference FLOPs are only a fraction of their dense counterparts.

One daunting drawback of sparse training is the resulting subnetworks are usually imbued with extremely irregular sparsity patterns, therefore, receiving very limited support from common hardware like GPU and TPU.

1.0

0

10

20

30

40

50

60

70

80

0.0

0.1

0.2

0.3

0.4

UMM

Amenable channels

%

Decrease by more than v=0

%

0

10

20

30

40

50

60

70

80

0.0

0.1

0.2

0.3

0.4

Decrease by more than v=20

%

0

10

20

30

40

50

60

70

80

0.0

0.1

0.2

0.3

0.4

Decrease by more than v=30

%

0

10

20

30

40

50

60

70

80

0.0

0.1

0.2

0.3

0.4

Decrease by more than v=40

%

0

10

20

30

40

50

60

70

80

0.0

0.2

0.4

0.6

WS

Amenable channels

%

Decrease by more than v=0

%

0

10

20

30

40

50

60

70

80

0.0

0.2

0.4

0.6

Decrease by more than v=20

%

0

10

20

30

40

50

60

70

80

0.0

0.2

0.4

0.6

Decrease by more than v=30

%

0

10

20

30

40

50

60

70

80

0.0

0.2

0.4

0.6

Decrease by more than v=40

%

0.0

0.2

0.4

0.6

0.8

Epochs

0.0

0.2

0.4

0.6

0.8

1.0

RigL block1.1.conv2.

RigL block3.1.conv2.

SET block1.1.conv2.

SET block3.1.conv2.

2.2 Sparse Amenable Channels

Here, we introduce the most important cornerstone concept for this work - 'sparse amenable channels' - which is defined as the channels whose sparsity becomes higher than their initial values caused by dynamic sparse training.

To provide empirical evidence for this interesting observation, we visualize the training dynamics of DST by monitoring two specific metrics of channels, Weight Sparsity and Unmasked Weight Magnitude, which are defined below.

Weight Sparsity (WS) (Quantity): Weight Sparsity directly quantizes the emergence of the Sparse Amenable Channels in quantity. Larger weight sparsity means more elements in the channel are becoming zero. Consequently, channels with fewer non-zero weights than their initial status are justified as Sparse Amenable Channels in this case.

Unmasked Mean Magnitude (UMM) (Quantity and Quality): Instead of solely quantitatively monitoring the weight sparsity, it is preferable to take the quality (i.e., magnitude) of the nonzero weights into consideration due to the crucial role of magnitude to dynamic sparse training [39; 10; 35]. Here, Unmasked Mean Magnitude refers to the mean magnitude of all the weights (including zero and nonzero) in the channel without considering masking. Smaller Unmasked Mean Magnitude represents the channels that come to be more sparse both in quantity and quality. Specifically, channels with fewer non-zero parameters but larger magnitudes will be excluded from the Sparse Amenable Channels. Therefore, the number of Sparse Amenable Channels justified here will be smaller than WS. We formalize these two metrics in Table 1 for a better interpretability. For comparison, we also evaluate the Masked Mean Magnitude (MMM), i.e., the mean magnitude of the non-zero weights.

We determined channels at the i training iteration are amenable if their values of Weight Sparsity are larger than their initialized values by a ratio v or their values of Unmasked Mean Magnitude are smaller than their initialized values by a ratio v : WS i -WS 0 WS 0 > v or UMM 0 -UMM i UMM 0 > v . In other words, we say a channel becomes v more sparse than its initial status if its WS i surpasses WS 0 by v , or its UMM i is smaller than UMM 0 by v .

Taking the most representative DST approaches SET [39] and RigL [10] as examples, we measure the number of the Sparse

Amenable Channels across layers in Figure 2, with v equals 0%, 20%, 30%, and 40%. We summarize our main observations here. ❶ Overall, we observe that a large part of channels (up to 60%) tend to be sparse amenable. While the number of amenable channels tends to decrease as v increases, there still exists around 10% ∼ 40% amenable channels becoming 40% more sparse than their initializations

Algorithm 1: Pseudocode of Chase

across layers. ❷ Fewer channels are justified as sparse amenable channels using the UMM metric than WS, as we expected. ❸ Deeper layers suffer from more amenable channels than shallow layers. ❹ RigL tends to extract more amenable channels than SET at the very early training phase. A possible reason is that the dense gradient encourages RigL to quickly discover and fill weights to the important (non-amenable) channels compared to the random growth used in SET.

Sparse amenable channels enjoy better prunability 1 than their counterparts. So far, we have unveiled the existence of the sparse amenable channels. It is natural to conjecture that these amenable channels can be a good indicator for channel pruning. To evaluate our conjecture, we choose the above proposed two metrics, Weight Sparsity (WS) and Unmasked Mean Magnitude (UMM), as our pruning criteria and perform a simple one-shot global channel pruning after regular DST training in comparison with their reversed metrics as well as several commonly-used principles, including random pruning [33], network slimming [37], and Masked Mean Magnitude (MMM). Channels with the highest values

Table 2: Top-1 test accuracy (%) of various channel pruning criteria with ResNet-50 on CIFAR-100. 'Reverse' refers to pruning with the reversed metric.

are pruned for WS, and the ones with the smallest values are pruned for UMM. Table 2 shows that both WS and UMM achieve good performance and UMM performs the best. Meanwhile, their reversed metrics perform no better than random pruning. Perhaps more interestingly, the resulting hybrid channel-level sparse models favorably preserve the performance of the unstructured RigL with no accuracy drop when pruned with mild channel sparsity.

In addition, we also observe the existence of 'sparse amenable channel' in a broad range of settings, including ResNet-32/VGG-16 on CIFAR-100, MLP Model on CIFAR10, and ViT Small, ResNet-50 on ImageNet in Appendix. Hence, we believe that sparse amenable channels is a very general phenomenon that widely exists across different architectures and datasets.

This encouraging result confirms our conjecture and demonstrates the promising potentials of sparse amenable channels (UMM) as a strong metric to remove channels during training. In the next section, we will explain in detail how we leverage Sparsity Amenable Channels and UMM to translate the promise of unstructured sparse training to the hardware-friendly sparse neural networks.

3 Methodology - Chase

Inspired by the above encouraging findings of sparse amenable channels, we introduce Ch asea ware dynamic s pars e ( Chase ) in this section. We follow the widely-used sparse training framework used in [39; 10]. The technical novelty of Chase mainly lies in two aspects. On the structured sparsity level, we adopt the gradual sparsification schedule [61] to gradually remove Amenable Channels during training with smallest UMM scores. The gradual sparsification schedule provides us with a moderate sparsification schedule, favorably relieving the accuracy drop caused by aggressive channel pruning. On the unstructured sparsity level, we globally redistribute parameters based on their magnitude and gradient, which significantly strengthens the sparse training performance. The overall Pseudocode of Chase is illustrated in Algorithm 1. We provide technical details of the above components below.

Gradual Amenable Channel Pruning. The gradual sparsification schedule is widely-used in the unstructured sparse literature to produce strong unstructured sparse subnetworks [61; 11; 31]. We explore it to the channel pruning regime with several ad-hoc modifications. Let us denote the initial and target final channel-wise sparsity level as S i and S f , respectively; gradual pruning starts at the training step t 0 with pruning frequency ∆ T , performing over a span of n pruning steps. The sparsity level S t at pruning step t is:

We globally collect UMM (see Section 2.2 for the definition) of each channel as the pruning criterion and progressively remove the sparse amenable channels with the smallest UMM according to Eq 3. We observe that layer collapse occurs sometimes without setting layer-wise pruning constraints. To avoid layer collapse, we use β to control the minimum number of channels remaining in layer l to be (1 -S f ) · β · w l , where w l is the number of channels in layer l . We empirically find that smaller β tends to yield better performance. We report more details Appendix A.2.

To maintain the overall number of parameters the same during training, we redistribute the overly pruned parameters back to the remaining channels at the next parameter grow phase using Eq 2. We find that without doing this will significantly hurt the model's performance.

Global Parameter Exploration. Global parameter exploration was introduced in previous arts [40; 5]. However, with the popularity of RigL [10], it is common to use a fixed set of layer-wise sparsities. Here, we revisit global parameter exploration in DST. To be specific, we globally prune parameters that have the smallest magnitudes and grow parameters with highest gradient magnitude. This small adaption brings a large performance benefit to RigL (up to 2.62% on CIFAR-100 and 1.7% on ImageNet), reported as 'Chase ( S c = 0)' in Table 3 and Table 4.

Soft Memory Bound. Soft memory bound was proposed in [59], which allows the parameter growing operation happens before the parameter pruning, improving the performance at the cost of a slight increase of memory requirements and FLOPs. We borrow the idea of soft memory bound to allow parameters firstly being added to the existing parameters followed by ∆ T p iteration of training, then remove the less important parameters including the newly added ones. This can avoid forcing the existing weights in the model to be removed if they are more important than newly grown weights.

After training, Chase slims down the initial 'big sparse' model to a 'small sparse' model with a significantly reduced number of channels. We completely remove the pruned channels in the current layer as well as the corresponding input dimensions of the next layer, so that the produced small sparse models can directly enjoy the acceleration in GPU.

4 Experimental Evaluation of Chase

In this section, we comprehensively evaluate Chase in comparison with the various state-of-theart (SOTA) unstructured sparse training methods as well as the state-of-the-art channel-pruning algorithms. At last, we provide a detailed analysis of hyperparameters and perform an ablation study to evaluate the effectiveness of the components of Chase.

Our evaluation is conducted with two widely used model architectures VGG-19 [48] and ResNet50 [15] on across various datasets including CIFAR-10/100 and ImageNet, We summarize the

implementation details for Chase in Appendix B. To show the superior performance of Chase on unstructured and structured sparsity, we report two variants of Chase: Chase ( S c = 0 ) represents the unstructured version without channel pruning and Chase ( S c = 0 . 5 ) stands for the structured version with 50% channel-level sparsity.

4.1 Comparison with off-the-shelf SOTA DST

CIFAR-10/100. Our method is naturally versatile and can be applied to both static sparse training and dynamic sparse training regimes. Therefore, for each model, we categorize the results into two groups and report the results in Table 3.

❶ Chase dramatically improves the performance of SST. As shown in the upper panel of each group, Chase significantly boosts the accuracy (up to 13.62% for GraSP with ResNet-50 on CIFAR-100) of SST methods like SNIP and GraSP. ❷ Chase ( S c = 0 ) establishes a new state-of-the-art performance bar for unstructured sparse training. Compared with the SOTA DST methods such as RigL and MEST, we clearly see that Chase ( S c = 0 ) universally outperforms all the presented DST methods by a large margin. Specifically, Chase ( S c = 0 ) achieves 3.67 % and 3.15% performance gains compared with SET on ResNet-50 and VGG-19. We also notice that the performance gain on CIFAR-100 is larger than the ones on CIFAR-10, which is as expected since CIFAR-100 has a larger improvement space than CIFAR-10. ❸ Chase ( S c = 0 . 5 ), with only 50% channels remaining, matches the performance of its unstructured variant, demonstrating the promise of the unstructured DST can be favorably transferred to the structured regime.

ImageNet . We reported the results on ImageNet in Table 4. Again, Chase ( S c = 0 ) dominates the performance in the unstructured sparsity regime, achieving 1.7% and 2.12% accuracy improvements over RigL and MEST, respectively. While the accuracy of Chase ( S c = 0 . 4 ) slightly decreases by 0.67% compared to Chase ( S c = 0 ), it still outperforms RigL by a good margin (1.03%), while enjoying a 1.5 × real inference speedup on common GPU.

When increasing the training time to 1.5 × (150 epochs), Chase also demonstrates a promising scaling trend. Chase ( S c = 0 ) matches the dense performance with only 0.55 × training FLOPs and consistently outperforms the off-the-shelf best DST method, MEST. Again, the promising results of Chase ( S c = 0 ) can be effectively translated to channel-level sparsity. Chase ( S c = 0 . 4 ) is 1.5 × faster than MEST, while still performing on par or even better than MEST.

Real Inference Speedups. We further compare the actual inference throughput and latency of our model against RigL in Table 5. All results are averaged from 100 individual runs with one NVIDIA 2080TI GPU in float32 on PyTorch. We set the batch size to 128 for CIFAR-100 and 2 for ImageNet, when evaluating the latency. We empirically find that pruning the skip connection leads to a significant accuracy drop while providing benefits on speedups. Therefore, the standard Chase keeps the skip connection layers untouched for optimal accuracy. To fully unleash Chase's potential on real inference speedups, we also provide another variant of Chase that prunes skip connection layers, dubbed Chase (prune skip). Compared with SoTA RigL, Chase (prune skip) is able to prune 50% channels with ResNet-50 on ImageNet, leading to a notable 68% of throughput gain, while only losing 0.29% accuracy. Even without pruning skip connections, our model is about to provide 31% throughput speedups, while outperforming RigL by 0.39%.

4.2 Extensive Analysis

Performance under different channel sparsity. To investigate the performance of different channel sparsity with the same parameter count, we maintain 2%, 5% parameters and prune the model to different channel sparsity ranging from 20% to 70%. The results are reported in Figure 3. The same training scheme is adopted as Section 4.1. Two DST baselines, RigL and SET are adopted for comparison. Not surprisingly, the more channels remain the better performance of the model archives. Notably, in all settings, Chase archives better performance than the baselines with just 50% channels, and Chase outperforms SET and RigL with just 30% channels on ResNet-50 at 98% sparsity.

20

%

30

%

40

%

50

%

60

%

70

%

69

70

71

72

%

98

%

Sparsity

20

%

30

%

40

%

50

%

60

%

70

%

70

71

72

73

95

%

Sparsity

20

%

30

%

40

%

50

%

60

%

70

%

73

74

75

76

77

98

%

Sparsity

20

%

30

%

40

%

50

%

60

%

70

%

76

77

78

95

%

Sparsity

0.0

0.2

0.4

0.6

0.8

1.0

Channel-wise sparsity

0.0

0.2

0.4

0.6

0.8

1.0

Chase

RigL

(

Sc

=0)

SET

(

Sc

=0)

90%

95%

98%

92.0

92.5

93.0

93.5

94.0

94.5

95.0

Accuracy (%)

94.42

94.22

93.2

94.45

94.41

93.5

94.62

94.63

93.9

94.88

94.85

94.2

ResNet-50 (CIFAR-10)

90%

95%

98%

74.5

75.0

75.5

76.0

76.5

77.0

77.5

78.0

77.18

76.5

74.84

76.86

76.33

74.92

76.92

77.03

76.18

77.52

77.48

77.03

ResNet-50 (CIFAR-100)

90%

95%

98%

91.5

92.0

92.5

93.0

93.5

94.0

93.6

93.05

91.95

93.91

93.7

92.8

94.0

93.72

92.94

94.03

93.84

93.69

VGG-19 (CIFAR-10)

90%

95%

98%

68

69

70

71

72

73

74

72.92

71.85

69.57

72.97

71.8

69.83

73.06

72.69

71.07

73.43

73.04

71.85

72.97

71.8

69.83

73.06

72.69

71.07

73.43

73.04

71.85

VGG-19 (CIFAR-100)

0.

0.2

0.4

0.6

0.8

1.0

Sparsity

0.

0.2

0.4

0.6

0.8

1.0

RigL

RigL+GACP

RigL+GACP+SM

RigL+GACP+SM+GE (Chase)

Effect of the channel pruning frequency. We also study how the channel pruning frequency ∆ T affects Chase's performance. For all experiments, we fixed the ending time τ stop for gradual amenable channel pruning as 130 epochs, the total training epochs τ total as 160 epochs and the minimum channel ratio factor as β as 0.5, while altering ∆ T to 1000, 4000, 8000, and 16000 iterations. We report the results in Appendix A.4. Overall, the largest ∆ T 16000 leads to worse performance. This observation is as expected, as we aim to achieve the same channel sparsity and larger ∆ T results in more removed channels in each punning operation. Consequently, larger performance degradation will be introduced during each pruning which could degrade the training stability.

Ablation study. In Figure 4, we study the effectiveness of different components in Chase, namely, the soft memory constraint (SM) and global parameter exploration (GE) on CIFAR-10/100 with ResNet-50 and VGG-19. We denote the RigL as our baseline, as RigL applies magnitude-based pruning and gradients-based growth like Chase. We apply the same training recipe as described in Section 4.1. Gradually amenable channel pruning safely removes 50% channels from RigL, while only suffering from minor or even no performance degradation. As for SM and GE, we found these techniques all bring universal performance improvements. Surprisingly, adding SM results in a 1.26% accuracy increase on CIFAR-100 with ResNet-50 at 98% sparsity. With GE, we can obtain a more optimal layer-wise ratio, which also consistently improves the accuracy from SM.

5 Related Work

Recently, as the financial and environmental costs of model training grow exponentially [50; 43], endeavors start to pursue training efficiency by investigating training sparse neural networks from scratch. Most Sparse training works can be divided into two categories, static sparse training, and dynamic sparse training. Static sparse training determines the structure of the sparse network at the initial stage of training by using certain pre-defined layer-wise sparsity ratios [38; 39; 10; 33].

Dynamic sparse training is designed to reduce the computation as well as memory footprint during the whole training phase. It trains a sparse neural network from scratch while allowing the sparse mask to be updated during training. SET [39] update sparse mask at the end of each training epoch by magnitude-based pruning and random growing. DSR [40] develops a dynamic reparameterization method that allows parameter reallocation during dynamic mask updating. DeepR [1] combines dynamic sparse parameterization with stochastic parameter updates for training. RigL [10] and

SNFS [5] propose to uses gradient information to grow weights. ITOP [35] studies the underlying mechanism of DST and discovers that the benefits of DST come from searching across time all possible parameters. GraNet [31] introduces the concept of 'pruning plasticity' and quantitatively studies the effect of pruning throughout training. MEST [59] proposes a memory-friendly training framework that could perform fast execution on edge devices. AC/DC [44] co-trains the sparse and dense models to return both accurate sparse and dense models. [23] dynamically activates top-K parameters during forward-pass while keeping a larger number of parameters updated in backwardpass to get rid of dense calculation of gradient. Top-KAST [23] preserves constant sparsity throughout training in both the forward and backward passes. Built upon Top-KAST, Powerpropagation [47] leaves the low-magnitude parameters largely unaffected by learning, achieving strong results. CHEX [19] applied dynamic prune and regrow channels strategies to avoid pruning important channels prematurely. Very recently, SLaK [32] leverages dynamic sparse training to successfully train intrinsically sparse 51 × 51 kernels, which performs on par with or better than advanced Transformers. A concurrent work [21] discovers that a tiny fraction of channels (up to 4.3%) of RigL become totally sparse after training.

To enable acceleration of sparse training in practice, [34] build a truly sparse framework based on SciPy sparse matrices [55] that enables efficient sparse evolutionary training [39] in CPU. [6] fulfill group-wise DST on Graphcore IPU [24] and demonstrate its efficacy on pre-training BERT. Moreover, some previous work develops sparse kernels [12; 9] to directly support unstructured sparsity in GPU. DeepSparse [27] deploys large-scale BERT-level and YOLO-level sparse models on CPU.

6 Conclusions

In this paper, we have presented Chase , a new sparse training approach that seamlessly translates the promise of unstructured sparsity into channel-level sparsity, while performing on par or even often better than state-of-the-art DST approaches. Extensive experiments across various network architectures including VGG-19 and ResNet-50 on CIFAR-10/100 and ImageNet demonstrated Chase can achieve better performance with 1.2 ×∼ 1.7 × real inference speedup on common GPU devices while performing on par or even better than unstructured SoTA. The results in this paper strongly challenge the common belief that sparse training typically suffers from limited acceleration support in common hardware, opening doors for future work to build more efficient sparse neural networks.

References

G. Bellec, D. Kappel, W. Maass, and R. Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations , 2018. 7, 9

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020. 1

T. Chen, X. Chen, X. Ma, Y. Wang, and Z. Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. arXiv preprint arXiv:2202.04736 , 2022. 2

T.-W. Chin, R. Ding, C. Zhang, and D. Marculescu. Legr: Filter pruning via learned global ranking. 2019. 17

T. Dettmers and L. Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840 , 2019. 3, 6, 8, 10

A. Dietrich, F. Gressmann, D. Orr, I. Chelombiev, D. Justus, and C. Luschi. Towards structured dynamic sparse pre-training of bert. arXiv preprint arXiv:2108.06277 , 2021. 2, 10

X. Dong and Y. Yang. Network pruning via transformable architecture search. Advances in Neural Information Processing Systems , 32, 2019. 17

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020. 1

E. Elsen, M. Dukhan, T. Gale, and K. Simonyan. Fast sparse convnets. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14629-14638, 2020. 2, 10

U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning , pages 2943-2952. PMLR, 2020. 2, 3, 4, 5, 6, 7, 8, 9, 16

T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574 , 2019. 6

T. Gale, M. Zaharia, C. Young, and E. Elsen. Sparse gpu kernels for deep learning. arXiv preprint arXiv:2006.10901 , 2020. 10

E. Garc´ ıa-Mart´ ın, C. F. Rodrigues, G. Riley, and H. Grahn. Estimation of energy consumption in machine learning. Journal of Parallel and Distributed Computing , 134:75-88, 2019. 1

S. Gray, A. Radford, and D. P. Kingma. Gpu kernels for block-sparse weights. arXiv preprint arXiv:1711.09224 , 3:2, 2017. 2

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016. 6

Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 17

Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 1389-1397, 2017. 2

J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 , 2017. 1

Z. Hou, M. Qin, F. Sun, X. Ma, K. Yuan, Y. Xu, Y.-K. Chen, R. Jin, Y. Xie, and S.-Y. Kung. Chex: Channel exploration for cnn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12287-12298, 2022. 10

I. Hubara, B. Chmiel, M. Island, R. Banner, S. Naor, and D. Soudry. Accelerated sparse neural training: A provable and efficient method to find n: M transposable masks. arXiv preprint arXiv:2102.08124 , 2021. 2

E. Iofinova, A. Peste, M. Kurtz, and D. Alistarh. How well do sparse imagenet models transfer? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12266-12276, 2022. 10

A. K. Jaiswal, H. Ma, T. Chen, Y. Ding, and Z. Wang. Training your sparse neural network better with any mask. In International Conference on Machine Learning , pages 9833-9844. PMLR, 2022. 2

S. Jayakumar, R. Pascanu, J. Rae, S. Osindero, and E. Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems , 33:20744-20754, 2020. 3, 10

Z. Jia, B. Tillman, M. Maggioni, and D. P. Scarpazza. Dissecting the graphcore ipu architecture via microbenchmarking. arXiv preprint arXiv:1912.03413 , 2019. 10

P. Jiang, L. Hu, and S. Song. Exposing and exploiting fine-grained block structures for fast and accurate sparse training. In Advances in Neural Information Processing Systems , 2022. 2

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020. 1

M. Kurtz, J. Kopinsky, R. Gelashvili, A. Matveev, J. Carr, M. Goin, W. Leiserson, S. Moore, B. Nell, N. Shavit, and D. Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 5533-5543, Virtual, 13-18 Jul 2020. PMLR. 10

N. Lee, T. Ajanthan, and P. H. Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340 , 2018. 2, 7

M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1529-1538, 2020. 17

L. Liu, S. Zhang, Z. Kuang, A. Zhou, J.-H. Xue, X. Wang, Y. Chen, W. Yang, Q. Liao, and W. Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning , pages 7021-7032. PMLR, 2021. 17

S. Liu, T. Chen, X. Chen, Z. Atashgahi, L. Yin, H. Kou, L. Shen, M. Pechenizkiy, Z. Wang, and D. C. Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems , 34:9908-9922, 2021. 3, 6, 10

S. Liu, T. Chen, X. Chen, X. Chen, Q. Xiao, B. Wu, M. Pechenizkiy, D. Mocanu, and Z. Wang. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprint arXiv:2207.03620 , 2022. 10

S. Liu, T. Chen, X. Chen, L. Shen, D. C. Mocanu, Z. Wang, and M. Pechenizkiy. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv preprint arXiv:2202.02643 , 2022. 2, 5, 9

S. Liu, D. C. Mocanu, A. R. R. Matavalam, Y. Pei, and M. Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications , 33(7):2589-2604, 2021. 2, 10

S. Liu, L. Yin, D. C. Mocanu, and M. Pechenizkiy. Do we actually need dense overparameterization? in-time over-parameterization in sparse training. In Proceedings of the 39th International Conference on Machine Learning , pages 6989-7000. PMLR, 2021. 1, 2, 3, 4, 8, 10

Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12009-12019, 2022. 1

Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision , pages 2736-2744, 2017. 2, 5

Z. Mariet and S. Sra. Diversity networks: Neural network compression using determinantal point processes. In International Conference on Learning Representations , 2016. 9

D. C. Mocanu, E. Mocanu, P. Stone, P. H. Nguyen, M. Gibescu, and A. Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. arXiv:1707.04780. Nature communications. , 9(1):2383, 2018. 1, 2, 3, 4, 6, 7, 8, 9, 10, 16

H. Mostafa and X. Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. International Conference on Machine Learning , 2019. 3, 6, 8, 9, 19

S. Narang, E. Elsen, G. Diamos, and S. Sengupta. Exploring sparsity in recurrent neural networks. arXiv preprint arXiv:1704.05119 , 2017. 2

Nvidia. Nvidia a100 tensor core gpu architecture. https://www.nvidia.com/content/dam/enzz/Solutions/Data-Center/nvidia-ampere-architecture-whitepaper.pdf , 2020. 2

D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350 , 2021. 1, 9

A. Peste, E. Iofinova, A. Vladu, and D. Alistarh. Ac/dc: Alternating compressed/decompressed training of deep neural networks. Advances in Neural Information Processing Systems , 34:85578570, 2021. 10

J. Pool and C. Yu. Channel permutations for n:m sparsity. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 13316-13327. Curran Associates, Inc., 2021. 2

M. A. Rumi, X. Ma, Y. Wang, and P. Jiang. Accelerating sparse cnn inference on gpus with performance-aware weight pruning. In Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques , pages 267-278, 2020. 2

J. Schwarz, S. Jayakumar, R. Pascanu, P. E. Latham, and Y. Teh. Powerpropagation: A sparsity inducing weight reparameterisation. Advances in Neural Information Processing Systems , 34:28889-28903, 2021. 3, 10

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations , 2014. 6

A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 , 2022. 1

E. Strubell, A. Ganesh, and A. McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.02243 , 2019. 9

X. Su, S. You, T. Huang, F. Wang, C. Qian, C. Zhang, and C. Xu. Locally free weight sharing for network width search. arXiv preprint arXiv:2102.05258 , 2021. 17

Y. Sui, M. Yin, Y. Xie, H. Phan, S. Aliari Zonouz, and B. Yuan. Chip: Channel independencebased pruning for compact neural networks. Advances in Neural Information Processing Systems , 34:24604-24616, 2021. 17

H. Tanaka, D. Kunin, D. L. Yamins, and S. Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems. arXiv:2006.05467 , 2020. 2, 7

Y. Tang, Y. Wang, Y. Xu, D. Tao, C. Xu, C. Xu, and C. Xu. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems , 33:10936-10947, 2020. 17

P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, ˙ I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020. 10

C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations , 2020. 2, 7

Z. You, K. Yan, J. Ye, M. Ma, and P. Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems , 32, 2019. 17

J. Yu and T. Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728 , 2019. 17

G. Yuan, X. Ma, W. Niu, Z. Li, Z. Kong, N. Liu, Y. Gong, Z. Zhan, C. He, Q. Jin, et al. Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems , 34:20838-20850, 2021. 6, 7, 8, 10

A. Zhou, Y. Ma, J. Zhu, J. Liu, Z. Zhang, K. Yuan, W. Sun, and H. Li. Learning n:m fine-grained structured sparse neural networks from scratch. In International Conference on Learning Representations , 2021. 2

M. H. Zhu and S. Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression, 2018. 6

A Remaining Experimental Analysis

A.1 Effect of the Initial Sparsity

Chase starts from a subnetwork with unstructured sparsity to produce channel-level sparsity during one end-to-end training process. Here we fix the target channel-level sparsity S c as 0.4 and study how the initialized unstructured sparsity impacts the model performance. The results are reported in Figure 5. It could be seen that, in general, initialization with more parameters leads to better performance. Counter-intuitively, the best accuracy is achieved using 0 . 6 unstructured sparsity, which is 0 . 26% higher than initialized as a dense model.

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

dense

74.4

75.2

76.0

76.8

%

0.0

0.2

0.4

0.6

0.8

1.0

Initialized unstructured sparsity

0.0

0.2

0.4

0.6

0.8

1.0

A.2 Effect of the Minimum Channel Ratio Limitation β

We alter the ratio of the minimum channel ratio β to 0.2, 0.5, 0.8 and show the performance in Table 6. The channel pruning frequency ∆ T is fixed at 4000 iterations. Apparently, large ratio β = 0 . 8 archives the worst performance while β = 0 . 2 outperforms other settings in 4 out of 6 cases. In view of the fact that smaller β provide more channel exploration space.

A.3 Ablation of Gradual Amenable Channel Pruning

Here, we perform a more university ablation study of Gradual Amenable Channel Pruning (GACP) on CIFAR10/100, ResNet-50 and VGG-19, with RigL and SET. The results are reported in Table 7.Surprisingly, GACP brings performance increases in most cases. To be specific, we found that GACP could boost the performance of RigL in 9 out of 12 cases and output SET in 10 out of 12 cases with just 50% remaining channels.

A.4 Effect of the Channel Pruning Frequency

In this Appendix, we study how the channel pruning frequency ∆ T affects Chase's performance. For all experiments, we fixed the ending time τ stop for gradual amenable channel pruning as 130 epochs, the total training epochs τ total as 160 epochs and the minimum channel ratio factor as β as 0.5, while altering ∆ T to 1000, 4000, 8000, and 16000 iterations. We report the results in Table 8. Overall, the largest ∆ T 16000 leads to worse performance. This observation is as expected, as we aim to achieve the same channel sparsity and larger ∆ T results in more removed channels in each

punning operation. Consequently, larger performance degradation will be introduced during each pruning which could degrade the training stability.

Table 8: Test accuracy (%) on CIFAR-100 of Chase at 50% channel-wise sparsity using different channel pruning frequency ∆ T . The best results are marked in bold.

B Implementation Details of Chase

In this appendix, we report the implementation details for Chase, including total training time ( τ total ), exploration stop time ( τ stop ), gradual channel pruning frequency ( ∆ T ), parameter update frequency ∆ T p , minimum layer limitation factor ( β ), learning rate (LR), batch size (BS), learning rate drop (LR Drop), weight decay (WD), SGD momentum (Momentum), sparse initialization (Sparse Init), target sparsity ( s p ), target channel-wise sparsity ( S c ), etc.

C Real Inference Speedups

We report the real inference latency and throughput of Chase on various sparity in Table 12.

D Comparisons with SOTA Channel Pruning Methods

We further compare Chase with various state-of-the-art channel pruning approaches in Table 13. It is encouraging to see that Chase performs on par with state-of-the-art SOTA channel pruning approaches, such as Group Fisher [30], CafeNet-R [51], and CHIP [52], without the need for the costly dense pretraining step. The implementation details are reported in Table 13.

E Existence of Sparse Amenable Channel in Various Settings

To demonstrate the broad prevalence of sparse amenable channels across diverse architectures and datasets, we have evaluated their ratio in multiple scenarios, including ResNet-32/VGG-16 on CIFAR-100, ResNet-50/ViT-small on ImageNet, an MLP model on CIFAR-10.

The MLP model consists of two hidden layers, each with 512 neurons. For the ViT small model, we focus our attention on the neurons within the MLP layers that exhibit suitability for pruning. In all cases, the sparse amenable channels are identified by Unmasked Mean Magnitude (UMM), with the threshold, v , set to 20%.

The results, shown in the corresponding tables, underline the consistent presence of sparse amenable channels across various architectures and datasets, reinforcing the argument that the phenomenon is both significant and widespread.

F Addressing the Memory Limitation of Global Parameter Exploration

During global parameter exploration, directly loading all the parameters for gradients/magnitude sorting is memory-consuming. Inspired by [40], we layer-wisely select parameters with the largest gradients for growth and the lowest magnitude for pruning by an adaptive global threshold H , until reaching the target sparsity. H is determined by a set point negative feedback loop to maintain an approximate parameter amount during each reallocation step, as reported below.

G Learned Layerwise Sparsity

Table 15 summarize the final learned layer-wise sparsity on ResNet-50 under 80% sparsity and 40% channel-wise sparsity. The model is obtained by training 100 epochs on ImageNet-1K. The parameterwise sparsity represents the sparsity budgets for all the CNN layers without the last fully-connected layer. The channel-wise sparsity denotes the sparsity of all the CNN internal layers (the first two convolution layers in the bottleneck blocks).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates practical deployment and speedup of Dynamic Sparse Training methods on general hardwares. The author's motivation stems from observations that many channels naturally exhibit high sparsity rates. Based on this finding, the authors propose a sparse-inducing method to gradually remove entire channels during sparse training. This ultimately produces a network with certain structured sparse characteristics, enabling favorable acceleration. The authors validate the effectiveness of their proposed method through experiments on extensive datasets.\n\nSTRENGTHS:\n1. The motivation is highly reasonable, and the proposed method has great application prospects, as it overcomes the bottleneck of unstructured sparsity being difficult to deploy on off-the-shelf hardware.\n2. The observation on the sparsity distribution across channels of DST is novel and provides inspiration for the community.\n3. The proposed method is easy to implement and even surpasses traditional non-structured pruning methods while achieving practical GPU acceleration.\n\nWEAKNESSES:\n1. Given the author's focus on practical acceleration in the pruning field, it would be interesting to compare this method with structured pruning methods [1,2] even after obtaining accelerated and deployable networks.\n\n[1] Pruning filters for efficient ConvNets. In ICLR, 2017.\n\n[2] Channel Pruning via Automatic Structure Search. In IJCAI, 2020.\n\nQUESTIONS:\nRegarding weight regrowing, I understand that the author selects weights with the highest magnitude globally for restoration. How about restoring weights with the highest UMM in each channel? Would this allow the network to gradually learn to become completely structured, i.e., some channels are automatically removed, while important channels are all preserved?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach to dynamic sparsity called Channel-aware dynamic sparse (Chase), which gradually translates unstructured dynamic sparsity to GPU-friendly channel-level sparsity during one end-to-end sparse training process.\n\nSTRENGTHS:\n-\tThe technical novelty of Chase mainly lies in two aspects: on the structured sparsity level, it adopts the gradual sparsification schedule to gradually remove Amenable Channels during training with smallest UMM scores, and on the unstructured sparsity level, it globally redistributes parameters based on their magnitude and gradient, which significantly strengthens the sparse training performance. I am a fan of this main idea.\n\n-\tChase differs from other approaches to dynamic sparsity in that it seamlessly translates unstructured dynamic sparsity to GPU-friendly channel-level sparsity during one end-to-end training pass, without any ad-hoc operations. Therefore, it is designed to be a more efficient and effective approach to dynamic sparsity that can be accelerated by commodity hardware. This is in contrast to other approaches demonstrated on unstructured sparsity which receive limited support in common hardware. \n\n-\tOne of the main new observations is that sparse amenable channels, which are channels that rapidly become sparser than others at the very early training stage, cause marginal damages to the model performance than their counterparts when pruned. The paper proposes a new metric called Unmasked Mean Magnitude (UMM) that can be used to precisely discover sparse amenable channels during training by monitoring the quantity and quality of weight sparsity. This finding reveals that a more nuanced approach to channel pruning can be more effective in achieving sparsity in neural networks.\n\n-\tChase achieves up to 1.7× inference throughput speedup on common GPU devices without compromising accuracy with ResNet-50 on ImageNet. Even without pruning skip connections, the model is able to provide 31% throughput speedups, while outperforming the SOTA RigL by 0.39%. Additionally, Chase establishes a new SOTA performance bar for unstructured sparse training, outperforming all the presented DST methods by good margins.\n\nWEAKNESSES:\n-\tChase appears to be a mixture between “unstructured sparse training” and “structured sparse inference”. So, what is the benefit of Chase over early-stage structured pruning methods, such as Early Bird lottery ticket? How about its comparison to N:M dynamic sparse training, which can be done now from end to end? And how about instead just training a slimmer dense network from scratch? Those important questions are not discussed.\n\n-\tExperiment choices seem flawed to me. Both VGG19 and ResNet50 are known to be (ridiculously) over-parameterized for CIFAR-10/100. Comparison results on those testbeds cannot be reliable for sparsity algorithms: for CIFAR perhaps ResNet-32 or Res-20 would have made more sense. The only persuasive setting in this paper is ResNet 50 + ImageNet.\n\n-\tImportant detail is missing on how “GPU-supported FLOPs” is computed? On what GPU? By channel level or N:M level? Also, how is the “Throughput” computed in Table 5?\n\n-\tChase requires additional memory to store the mask for each channel. This can be a limiting factor for models with a large number of channels, as the memory requirements can become prohibitively large.\n\n-\tAdditionally, the paper does not provide a detailed analysis of the end-to-end computational cost of Chase compared to other dynamic sparse training methods, which could be an important consideration for practical applications.\n\nQUESTIONS:\nPlease see above weakness.\n\nLIMITATIONS:\nThe authors shall discuss more limitations of their method", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes Channel-aware dynamic sparse, which translates unstructured dynamic sparsity to GPU-friendly channel-level sparsity during one end-to-end training process without using any particularly sparsity-aware hardware accelerators. The authors also observe the sparse amendable channels and proposed a new metric - Unmasked Mean Magnitude. Experiments demonstrate that the proposed method achieves 1.2x - 1.7x inference throughput speedups on common GPU devices with less to no accuracy degradation.

STRENGTHS:
1. This paper is well-written and gives sufficient experiments.
2. This paper observes the channel-wise sparsity during dynamic sparse training and proposes a framework Chase to translate unstructured dynamic sparsity to channel-level sparsity with less to no accuracy loss.

WEAKNESSES:
1. The proposed method does not seem to have much performance benefit for training compared to unstructured methods. The computation is still unstructured in most iterations. 

2. In Figure 3-b, the accuracy of RigL+GACP and RigL+GACP+SM is lower than RigL for ResNet-50 with sparsity 90% on CIFAR-100. While I understand GACP improves efficiency, it is not a good way to show the advantage of GACP as accuracy drops. It will be more convincing if you can show the runtime at the same accuracy or the accuracy with the same runtime.

QUESTIONS:
The experiments use 50% channel sparsity. Why 50%?

LIMITATIONS:
I don't find a discussion on the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach called Channel-aware dynamic sparse (Chase) for training deep neural networks with high sparsity. Unlike previous techniques that focus on unstructured sparsity with irregular patterns, Chase seamlessly translates the benefits of dynamic sparsity to channel-level sparsity, which is more hardware-friendly. This is achieved through an end-to-end training process without requiring additional ad-hoc operations or specialized hardware accelerators. By identifying and removing sparser channels during training, Chase enables small sparse networks that can be efficiently accelerated on commodity GPUs. Experimental results show that Chase achieves a significant 1.7× inference throughput speedup without sacrificing accuracy when applied to ResNet-50 on ImageNet. The authors plan to release their code to facilitate further research and adoption of their approach.\n\nSTRENGTHS:\nThe paper exhibits exceptional clarity and a well-structured format. I particularly appreciate the bullet-point summary in the introduction, which effectively outlines the paper's roadmap. However, it must be noted that the paper can be somewhat dense and requires some additional time to fully comprehend.\n\nThe proposed method is technically sound and yields commendable empirical results. The authors offer ample implementation details that would greatly facilitate other researchers and practitioners in reproducing the key findings outlined in the paper. It is worth mentioning that the authors have also committed to releasing the code upon the acceptance of the paper, further enhancing its accessibility and potential for future advancements.\n\nWEAKNESSES:\nThe paper includes experimental results that demonstrate the improvement in inference speed. However, it would be valuable to clarify whether the proposed method also enhances the training efficiency of the model. From my understanding, the primary advantage of employing dynamic sparse training lies in improving training efficiency. Otherwise, one could simply train a dense model initially and subsequently prune it to achieve sparsity. By addressing this aspect, the authors can provide a more comprehensive understanding of the benefits of their proposed method.\n\nThis paper primarily conducts its main experiments and ablation studies using CIFAR-10, which is widely regarded as a small-scale dataset. Consequently, the results obtained from this dataset may not be entirely representative. It would significantly enhance the robustness of the findings if the authors could replicate these experiments using the larger-scale ImageNet dataset. Additionally, it would be highly advantageous if the authors could extend their validation to encompass other models and tasks, such as object detection and semantic segmentation, in order to establish the general effectiveness of their proposed method across diverse domains.\n\nQUESTIONS:\nPlease carefully address the weaknesses listed above in the rebuttal.\n\nLIMITATIONS:\nThe paper lacks an adequate discussion on the limitations or potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}