SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lu Yin<sup>1</sup>, Gen Li<sup>2</sup>, Meng Fang<sup>3</sup>, Li Shen<sup>4</sup>, Tianjin Huang<sup>1</sup>, Zhangyang Wang<sup>5</sup>, Vlado Menkovski<sup>1</sup>, Xiaolong Ma<sup>2</sup>, Mykola Pechenizkiy<sup>1</sup>, Shiwei Liu<sup>1,5</sup>
$^{1}$ Eindhoven University of Technology,  $^{2}$ Clemson University
<sup>3</sup>University of Liverpool, <sup>4</sup>JD Explore Academy, <sup>5</sup>University of Texas at Austin,
{1.yin,t.huang,m.pechenizkiy,v.menkovski,s.liu3}@tue.nl
gen@g.clemson.edu, xiaolom@clemson.edu, Meng.Fang@liverpool.ac.uk
mathshenli@gmail.com, atlaswang@utexas.edu
Sparse training has received an upsurging interest in machine learning due to its tantalizing saving potential for the entire training process as well as inference. Dynamic sparse training (DST), as a leading sparse training approach, can train deep neural networks at high sparsity from scratch to match the performance of their dense counterparts. However, most if not all DST prior arts demonstrate their effectiveness on unstructured sparsity with highly irregular sparse patterns, which receives limited support in common hardware. This limitation hinders the usage of DST in practice. In this paper, we propose Channel-aware dynamic sparse (Chase), which for the first time seamlessly translates the promise of unstructured dynamic sparsity to GPU-friendly channel-level sparsity (not fine-grained  $N:M$  or group sparsity) during one end-to-end training process, without any ad-hoc operations. The resulting small sparse networks can be directly accelerated by commodity hardware, without using any particularly sparsity-aware hardware accelerators. This appealing outcome is partially motivated by a hidden phenomenon of dynamic sparsity: off-the-shelf unstructured DST implicitly involves biased parameter reallocation across channels, with a large fraction of channels (up to  $60\%$ ) being sparser than others. By progressively identifying and removing these channels during training, our approach translates unstructured sparsity to channel-wise sparsity. Our experimental results demonstrate that Chase achieves  $1.7\times$  inference throughput speedup on common GPU devices without compromising accuracy with ResNet-50 on ImageNet. We release our codes in https://github.com/luuyin/chase.
Deep neural networks (DNNs) have recently demonstrated impressive breakthroughs with increasing scales [2; 8; 36]. Besides the well-known scaling, i.e., test accuracy scales as a power law regarding model size and training data size in quantity [18; 26], recent work has observed that massive increases in quantity can imbue models with qualitatively new behavior [49]. However, the memory and computation required to train and deploy these large models can be a heavy burden on the environment and finance [13; 43]. Therefore, people start to probe the possibility of training sparse neural networks from scratch without involving any dense training steps (dubbed sparse training [39; 35]). As the memory requirement and multiplications (which dominate neural network computation) associated with zero weights can be skipped, sparse training is becoming a promising direction due to their "end-to-end" saving potentials for both efficient training and efficient inference.
Sparse training can be categorized into two groups, static sparse training and dynamic sparse training according to the dynamics of the sparse pattern during training. Static sparse training
Figure 1: Inference latency and throughput of various DST methods. The sparsity level is  $90\%$  for all approaches. All models are trained for 100 epochs with the ResNet-50/ImageNet benchmark. Each dot of Chase and Chase (prune skip) corresponds to a model with distinct channel-wise sparsity. The results of latency are obtained on NVIDIA 2080TI GPU with a batch size of 2.
(SST) [28; 56; 53; 33; 22], namely, draws a sparse neural network at initialization before training and train with a fixed sparse pattern (sparse connections between layers) without further changes. Dynamic sparse training (DST) [39; 10; 35], on the contrary, jointly optimizes the weights and sparse patterns during training, usually delivering better performance than the static ones. DST quickly evolves as a leading direction in sparse training due to its compelling performance and training/inference efficiency. For instance, a sparse ResNet-34 with only  $2\%$  parameters left can be dynamically trained to match the performance of its dense counterpart without involving any pre-training or dense training [35].
While showing promise in performance and efficiency, so far the real speedup of DST has only been demonstrated on CPU [34; 6] or IPU [6]. Most sparse training methods produce unstructured sparse neural networks with extremely irregular sparse patterns, which can not be directly accelerated in common hardware (i.e., GPU and TPU), compared to the straightforward and hardware-friendly sparse pattern produced by channel pruning [17; 37].
Many endeavors strive to solve this issue by coarsening the sparsity granularity, which can be loosely categorized into two groups. i. Grouping nonzero weights into blocks. As GPU performs very fast on contiguous memory operations, block-wise sparsity enjoys much more speedups than unstructured sparsity in practice. Group lasso regularization [41; 14] is a widely-used technique to induce block sparsity in the network. Ad-hoc grouping operations can also be utilized to build dense blocks from unstructured sparse weights [46; 3]. ii. Seeking fine-grained structured sparse patterns. For instance, inspired by the recent support of 2:4 fine-grained sparsity in NVIDIA Ampere [42], previous arts attempt to find a sweet spot between structured and unstructured sparsity by learning  $N:M$  sparsity patterns [60; 20; 45]. However, these methods either rely on specialized sparse-aware accelerators [9; 42] to enable speedups or suffer from significant performance degradation due to the constraint location of nonzero values [25].
In this paper, we propose a new method dubbed Channel-aware dynamic sparse (Chase), which can effectively transfer the promise of unstructured sparse training into the hardware-friendly channel sparsity with comparable or even better performance on common GPU devices. The roadmap of our exploration is as follows:
■ Observation 1: We first present an emerging characteristic of DST: off-the-shelf DST approaches implicitly involve biased parameter reallocation, resulting in a large proportion of channels (up to  $60\%$ ) that rapidly become sparser than their initializations at the very early training stage. We term them as "sparse amenable channels" for the sake of convenient reference.
■ Observation 2: We examine the pruning ability (i.e., the accuracy drop caused by pruning) of the sparse amenable channels, we find that these channels cause marginal damages to the model performance than their counterparts when pruned.
A New Metric: We propose a new, sparsity-inspired, channel pruning metric - Unmasked Mean Magnitude (UMM) - that can be used to precisely discover sparse amenable channels during training by monitoring the quantity and quality of weight sparsity.
A New Approach: Based on the above findings, we propose Channel-aware dynamic sparse (Chase), a first sparse training framework that can favorably transform unstructured sparsity into channel-wise sparsity on the fly. Chase starts with an unstructured sparse neural network and dynamically trains it while gradually eliminating sparse amenable channels with the lowest UMM scores. During training, we globally grow and shrink parameters to strengthen performance further.
■ Performance: Chase inherently can be tailed into an unstructured sparse training approach and a structured sparse training approach. Our unstructured variant establishes a new state-of-the-art accuracy bar for sparse training. More impressively, our structured approach is able to maintain or even surpass SoTA performance with ResNet-50 on ImageNet, while achieving  $1.2 \times -1.7 \times$  inference throughput speedups on common GPU devices.
2 Sparse Amenable Channels in DST
We first describe the basis and notations of the prior sparse training arts. Afterward, we provide evidence for the existence of the sparse amenable channels during the dynamic sparse training across different architectures and demonstrate that pruning of such channels leads to marginal performance damage than their counterparts. Based on this interesting finding, we introduce Chase, a sparsity-inspired sparse training method that for the first time translates the theoretical promise of sparse training into GPU-friendly speedup, without using any specialized CUDA implementations.
2.1 Prior Sparse Training Arts
Let us denote the sparse neural network as  $f(\pmb{x}; \pmb{\theta}_{\mathrm{s}})$ .  $\pmb{\theta}_{\mathrm{s}}$  refers to a subset of the full network parameters  $\pmb{\theta}$  at a sparsity level of  $(1 - \frac{\|\pmb{\theta}_{\mathrm{s}}\|_0}{\|\pmb{\theta}\|_0})$  and  $\| \cdot \|_0$  represents the  $\ell_0$ -norm.
It is common to initialize sparse subnetworks  $\theta_{\mathrm{s}}$  randomly based on the uniform [40; 5] or nonuniform layer-wise sparsity ratios with Erdős-Rényi (ER) graph [39; 10; 35; 31]. In the case of image classification, sparse training aims to optimize:  $\hat{\theta}_{\mathrm{s}} = \operatorname*{argmin}_{\theta_{\mathrm{s}}} \sum_{i=1}^{\mathrm{N}} \mathcal{L}(f(x_i; \theta_{\mathrm{s}}), y_i)$  using data  $\{(x_i, y_i)\}_{i=1}^{\mathrm{N}}$ , where  $\mathcal{L}$  is the loss function. Static sparse training (SST) maintains the same sparse network connectivity during training after initialization. Dynamic sparse training (DST), on the contrary, allows the sparse subnetworks to dynamically explore new parameters while sticking to a fixed sparsity budget. Most of the DST methods follow a simple prune-and-grow scheme [39] to perform parameter exploration, i.e., pruning  $r$  proportion of the least important parameters based on their magnitude, and immediately grow the same number of parameters randomly [39] or using the potential gradient [10]. Formally, the parameter exploration can be formalized as the following two steps:
$$
\boldsymbol {\theta} _ {\mathrm {s}} = \Psi \left(\boldsymbol {\theta} _ {\mathrm {s}}, r\right), \tag {1}
$$
$$
\boldsymbol {\theta} _ {\mathrm {s}} = \boldsymbol {\theta} _ {\mathrm {s}} \cup \Phi \left(\boldsymbol {\theta} _ {i \notin \boldsymbol {\theta} _ {\mathrm {s}}}, r\right). \tag {2}
$$
where  $\Psi$  is the specific pruning criterion and  $\Phi$  is growing scheme. These metrics may vary from sparse training method to another. In addition to prune-and-grow, previous work [23; 47] dynamically activates top-K parameters during forward-pass while keeping a larger number of parameters updated in backward-pass to get rid of dense calculation of gradient. At the end of the training, sparse training can converge to a performant sparse subnetwork. Since the sparse neural networks are trained from scratch, the memory requirements and training/inference FLOPs are only a fraction of their dense counterparts.
One daunting drawback of sparse training is the resulting subnetworks are usually imbued with extremely irregular sparsity patterns, therefore, receiving very limited support from common hardware like GPU and TPU.
Figure 2: The portion of sparse amenable channels justified by two metrics, the Unmasked Mean Magnitude (UMM) and the Weight Sparsity (WS), of ResNet-50 trained on CIFAR-100.
2.2 Sparse Amenable Channels
Here, we introduce the most important cornerstone concept for this work - "sparse amenable channels" - which is defined as the channels whose sparsity becomes higher than their initial values caused by dynamic sparse training.
To provide empirical evidence for this interesting observation, we visualize the training dynamics of DST by monitoring two specific metrics of channels, Weight Sparsity and Unmasked Weight Magnitude, which are defined below.
Weight Sparsity (WS) (Quantity): Weight Sparsity directly quantizes the emergence of the Sparse Amenable Channels in quantity. Larger weight sparsity means more elements in the channel are becoming zero. Consequently, channels with fewer non-zero weights than their initial status are justified as Sparse Amenable Channels in this case.
Unmasked Mean Magnitude (UMM) (Quantity and Quality): Instead of solely quantitatively monitoring the weight sparsity, it is preferable to take the quality (i.e., magnitude) of the nonzero weights into consideration due to the crucial role of magnitude to dynamic sparse training [39; 10; 35]. Here, Unmasked Mean Magnitude refers to the mean magnitude of all the weights (including zero and nonzero) in the channel without considering masking. Smaller Unmasked Mean Magnitude represents the channels that come to be more sparse both in quantity and quality. Specifically, channels with fewer non-zero parameters but larger magnitudes will be excluded from the Sparse Amenable Channels. Therefore, the number of Sparse Amenable Channels justified here will be smaller than WS. We formalize these two metrics in Table 1 for a better interpretability. For comparison, we also evaluate the Masked Mean Magnitude (MMM), i.e., the mean magnitude of the non-zero weights.
We determined channels at the  $i$  training iteration are amenable if their values of Weight Sparsity are larger than their initialized values by a ratio  $v$  or their values of Unmasked Mean Magnitude are smaller than their initialized values by a ratio  $v$ :  $\frac{\mathrm{WS}_i - \mathrm{WS}_0}{\mathrm{WS}_0} > v$  or  $\frac{\mathrm{UMM}_0 - \mathrm{UMM}_i}{\mathrm{UMM}_0} > v$ . In other words, we say a channel becomes  $v$  more sparse than its initial status if its  $\mathrm{WS}_i$  surpasses  $\mathrm{WS}_0$  by  $v$ , or its  $\mathrm{UMM}_i$  is smaller than  $\mathrm{UMM}_0$  by  $v$ .
Taking the most representative DST approaches SET [39] and RigL [10] as examples, we measure the number of the Sparse
Amenable Channels across layers in Figure 2, with  $v$  equals  $0\%$ ,  $20\%$ ,  $30\%$ , and  $40\%$ . We summarize our main observations here. ① Overall, we observe that a large part of channels (up to  $60\%$ ) tend to be sparse amenable. While the number of amenable channels tends to decrease as  $v$  increases, there still exists around  $10\% \sim 40\%$  amenable channels becoming  $40\%$  more sparse than their initializations
Table 1: Metrics that are introduced to measure the dynamics of the Sparse Amenable Channels. The weight tensor and the binary mask of a channel is represented with  $\theta$  and  $m$ , respectively. And  $\| \cdot \|_0$  stands for the  $\ell_0$ -norm.
Weight Sparsity (WS)	1 - ||m ⊙θ||0/||θ||0
Unmasked Mean Magnitude (UMM)	∑ |θ|/||θ||0
Masked Mean Magnitude (MMM)	∑ |m ⊙θ|/||m ⊙θ||0
Algorithm 1: Pseudocode of Chase
Input: Sparse neural network with initialized sparse weight  $\theta_{s}$ , target sparsity  $s_p$ , current sparsity  $s_t$ , parameter update frequency  $\Delta T_p$ , sparsity mutation factor  $s_e$ , target channel sparsity  $S_c$ , channel pruning frequency  $\Delta T$ , current channel sparsity  $S_t$ , training steps  $\tau$ , exploration stop steps  $\tau_{stop}$ , total training steps  $\tau_{total}$
Output: A sparse model satisfying the target sparsity  $s_p$  and channel sparsity requirement  $S_c$ .
while  $\tau <  \tau_{stop}$  do
if Mod(t,  $\Delta T) = 0$  and  $t\leq T_{max}$  then  $\theta_{\mathrm{s}}\gets \mathrm{Global~channel~prune}(\theta_{\mathrm{s}},S_t)\triangleright \mathrm{Perform~gradual~amenable~channel~pruning~using}$  Eq.3
if Mod(t,  $\Delta T_{p}) = 0$  and  $t\leq T_{max}$  then  $\theta_{s}(s_{t} = s_{p} - s_{e})\gets \mathrm{Global~Parameter~Grow}(\theta_{s})$  Training for  $\Delta \tau$  epochs,  $\tau \leftarrow \tau +\Delta T_{p}$ $\theta_{\mathrm{s}}(s_t = s_p)\gets \mathrm{Global~Parameter~Prune}(\theta_{\mathrm{s}})$ $\triangleright$  Chase adopts global parameter exploration and soft memory bound.
Continue sparse training from the epoch  $\tau_{stop}$  to  $\tau_{end}$ .
across layers. Fewer channels are justified as sparse amenable channels using the UMM metric than WS, as we expected. Deeper layers suffer from more amenable channels than shallow layers. RigL tends to extract more amenable channels than SET at the very early training phase. A possible reason is that the dense gradient encourages RigL to quickly discover and fill weights to the important (non-amenable) channels compared to the random growth used in SET.
Sparse amenable channels enjoy better prunability than their counterparts. So far, we have unveiled the existence of the sparse amenable channels. It is natural to conjecture that these amenable channels can be a good indicator for channel pruning. To evaluate our conjecture, we choose the above proposed two metrics, Weight Sparsity (WS) and Unmasked Mean Magnitude (UMM), as our pruning criteria and perform a simple one-shot global channel pruning after regular DST training in comparison with their reversed metrics as well as several commonly-used principles, including random pruning [33], network slimming [37], and Masked Mean Magnitude (MMM). Channels with the highest values
Table 2: Top-1 test accuracy  $(\%)$  of various channel pruning criteria with ResNet-50 on CIFAR-100. "Reverse" refers to pruning with the reversed metric.
Method	Channel Sparsity
10%	20%	30%
Standard RigL [10]	76.89±0.43	76.89±0.43	76.89±0.43
Random Pruning [33]	43.01±9.62	11.74±2.79	3.79±1.32
Network Slimming [37]	76.82±0.43	76.67±0.39	66.57±2.95
MMM	62.31±8.66	19.34±14.88	5.32±2.88
MMM Reverse	5.28±2.52	2.04±0.30	1.72±0.40
WS	76.86±0.43	76.79±0.39	62.79±5.42
WS Reverse	2.9±0.91	2.43±0.07	2.03±0.38
UMM	76.88±0.43	76.90±0.42	71.77±2.31
UMM Reverse	3.18±0.48	2.23±0.26	1.51±0.35
are pruned for WS, and the ones with the smallest values are pruned for UMM. Table 2 shows that both WS and UMM achieve good performance and UMM performs the best. Meanwhile, their reversed metrics perform no better than random pruning. Perhaps more interestingly, the resulting hybrid channel-level sparse models favorably preserve the performance of the unstructured RigL with no accuracy drop when pruned with mild channel sparsity.
In addition, we also observe the existence of "sparse amenable channel" in a broad range of settings, including ResNet-32/VGG-16 on CIFAR-100, MLP Model on CIFAR10, and ViT Small, ResNet-50 on ImageNet in Appendix. Hence, we believe that sparse amenable channels is a very general phenomenon that widely exists across different architectures and datasets.
This encouraging result confirms our conjecture and demonstrates the promising potentials of sparse amenable channels (UMM) as a strong metric to remove channels during training. In the next section, we will explain in detail how we leverage Sparsity Amenable Channels and UMM to translate the promise of unstructured sparse training to the hardware-friendly sparse neural networks.
<sup>1</sup>Prunability here refers to the accuracy drop caused by the channel removal.
3 Methodology - Chase
Inspired by the above encouraging findings of sparse amenable channels, we introduce Chase-aware dynamic sparse (Chase) in this section. We follow the widely-used sparse training framework used in [39; 10]. The technical novelty of Chase mainly lies in two aspects. On the structured sparsity level, we adopt the gradual sparsification schedule [61] to gradually remove Amenable Channels during training with smallest UMM scores. The gradual sparsification schedule provides us with a moderate sparsification schedule, favorably relieving the accuracy drop caused by aggressive channel pruning. On the unstructured sparsity level, we globally redistribute parameters based on their magnitude and gradient, which significantly strengthens the sparse training performance. The overall Pseudocode of Chase is illustrated in Algorithm 1. We provide technical details of the above components below.
Gradual Amenable Channel Pruning. The gradual sparsification schedule is widely-used in the unstructured sparse literature to produce strong unstructured sparse subnetworks [61; 11; 31]. We explore it to the channel pruning regime with several ad-hoc modifications. Let us denote the initial and target final channel-wise sparsity level as  $S_{i}$  and  $S_{f}$ , respectively; gradual pruning starts at the training step  $t_0$  with pruning frequency  $\Delta T$ , performing over a span of  $n$  pruning steps. The sparsity level  $S_{t}$  at pruning step  $t$  is:
$$
S _ {t} = S _ {f} + \left(S _ {i} - S _ {f}\right) \left(1 - \frac {t - t _ {0}}{n \Delta T}\right) ^ {3}. \tag {3}
$$
We globally collect UMM (see Section 2.2 for the definition) of each channel as the pruning criterion and progressively remove the sparse amenable channels with the smallest UMM according to Eq 3. We observe that layer collapse occurs sometimes without setting layer-wise pruning constraints. To avoid layer collapse, we use  $\beta$  to control the minimum number of channels remaining in layer  $l$  to be  $(1 - S_{f})\cdot \beta \cdot w_{l}$ , where  $w_{l}$  is the number of channels in layer  $l$ . We empirically find that smaller  $\beta$  tends to yield better performance. We report more details in Appendix A.2.
To maintain the overall number of parameters the same during training, we redistribute the overly pruned parameters back to the remaining channels at the next parameter grow phase using Eq 2. We find that without doing this will significantly hurt the model's performance.
Global Parameter Exploration. Global parameter exploration was introduced in previous arts [40; 5]. However, with the popularity of RigL [10], it is common to use a fixed set of layer-wise sparsities. Here, we revisit global parameter exploration in DST. To be specific, we globally prune parameters that have the smallest magnitudes and grow parameters with highest gradient magnitude. This small adaption brings a large performance benefit to RigL (up to  $2.62\%$  on CIFAR-100 and  $1.7\%$  on ImageNet), reported as "Chase ( $S_{c} = 0$ )" in Table 3 and Table 4.
Soft Memory Bound. Soft memory bound was proposed in [59], which allows the parameter growing operation happens before the parameter pruning, improving the performance at the cost of a slight increase of memory requirements and FLOPs. We borrow the idea of soft memory bound to allow parameters firstly being added to the existing parameters followed by  $\Delta T_{p}$  iteration of training, then remove the less important parameters including the newly added ones. This can avoid forcing the existing weights in the model to be removed if they are more important than newly grown weights.
After training, Chase slims down the initial "big sparse" model to a "small sparse" model with a significantly reduced number of channels. We completely remove the pruned channels in the current layer as well as the corresponding input dimensions of the next layer, so that the produced small sparse models can directly enjoy the acceleration in GPU.
4 Experimental Evaluation of Chase
In this section, we comprehensively evaluate Chase in comparison with the various state-of-the-art (SOTA) unstructured sparse training methods as well as the state-of-the-art channel-pruning algorithms. At last, we provide a detailed analysis of hyperparameters and perform an ablation study to evaluate the effectiveness of the components of Chase.
Our evaluation is conducted with two widely used model architectures VGG-19 [48] and ResNet-50 [15] on across various datasets including CIFAR-10/100 and ImageNet, We summarize the