SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aleksandra I. Nowak*
Jagiellonian University, Doctoral School of Exact and Natural Sciences;
IDEAS NCBR
aleksandrairena-nowak@docotral.uj.edu.pl
Bram Grooten
Decebal Constantin Mocanu
University of Luxembourg;
Eindhoven University of Technology;
Jacek Tabor
Jagiellonian University,
Faculty of Mathematics and Computer Science
Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network's sparse connectivity. While the growing criterion's impact on DST performance is relatively well studied, the influence of the pruning criterion remains overlooked. To address this issue, we design and perform an extensive empirical analysis of various pruning criteria to better understand their impact on the dynamics of DST solutions. Surprisingly, we find that most of the studied methods yield similar results. The differences become more significant in the low-density regime, where the best performance is predominantly given by the simplest technique: magnitude-based pruning.
Modern deep learning solutions have demonstrated exceptional results in many different disciplines of science [5, 14, 26]. However, they come at the cost of using an enormous number of parameters. Consequently, compression methods aim to significantly reduce the model size without introducing any loss in performance [20, 6].
One approach to sparsifying neural networks is pruning, in which a portion of the weights is removed at the end of the training based on some predefined importance criterion [31, 21, 20]. More recently, [15] have found that iterative pruning joined with cautious parameter re-initialization can identify sparse subnetworks that are able to achieve similar performance to their dense counterparts when trained from scratch. This result, known as the lottery ticket hypothesis, has launched subsequent research into methods for identifying and training models that are sparse already at initialization [32, 60, 66, 48, 54].
*Work started while visiting University of Twente.
An especially promising direction in obtaining well-performing sparse neural networks is the Dynamic Sparse Training (DST) framework. Inspired by the neuroregeneration in the brain, DST allows for plasticity of the initial sparse connectivity of the network by iteratively pruning and re-growing a portion of the parameters in the model [42]. This relatively new concept has already gained increasing interest in the past few years. Most notably, in computer vision, DST demonstrated that it is sufficient to use only  $20\%$  of the original parameters of ResNet50 to train ImageNet without any drop in performance [12, 38]. Even more intriguingly, in Reinforcement Learning applications, DST is able to significantly outperform the classical dense models [18, 58]. At the same time, general sparse networks have been reported to surpass their dense counterparts in terms of adversarial robustness [47]. All those results demonstrate the incredible potential of DST not only in increasing the model's efficiency but also in providing a better understanding of the features and limitations of neural network training.
Motivated by the above, we take a closer look at the current DST techniques. Most of the research concerning design choices in DST focuses on the analysis of methods where variations occur in the growth criterion, while the pruning criterion becomes intertwined with other design considerations [3, 12, 11, 1]. In this study, we address this issue by taking a complementary approach and focusing on the pruning criteria instead. A pruning criterion in DST serves as a measure of weight importance and hence is a proxy of the "usefulness" of a particular connection. Note that the importance of a connection in DST can differ from standard post-training pruning, as the role of weights can change throughout training due to the network's plasticity and adaptation. Weights deemed unimportant in one step can become influential in later phases of the training.
Our goal is to provide a better understanding of the relationship between pruning criteria and the dynamics of DST solutions. To this end, we perform a large empirical study including several popular pruning criteria and analyze their impact on the DST framework on diverse models. We find that:
- Surprisingly, within a stable DST hyperparameter setup, the majority of the studied criteria perform similarly, regardless of the model architecture and the selected growth criterion.
- The difference in performance becomes more significant in a very sparse regime, with the simplest magnitude-based pruning methods surpassing any more elaborate choices.
- Applying only a few connectivity updates is already enough to achieve good results. At the same time, the reported outcomes surpass those obtained by static sparse training.
- By analyzing the structural similarity of the pruned sets by each criterion, we assert that the best-performing methods make similar decision choices.
The insights from our research identify that the simplest magnitude pruning is still the optimal choice, despite a large amount of alternatives present in the literature. This drives the community's attention to carefully examine any new adaptation criteria for Dynamic Sparse Training.2
Pruning and Sparse Training. A common way of reducing the neural network size is pruning, which removes parameters or entire blocks of layers from the network. In its classical form, pruning has been extensively studied in the context of compressing post-training models [25, 44, 31, 21, 20, 46, 33, 43, 17] – see e.g. [23, 4] for an overview and survey. Interestingly, [40] demonstrated for the first time in the literature that a sparse neural network can match and even outperform its corresponding dense neural network equivalent if its sparse connectivity is designed in a sensitive manner. Recently, a similar result was obtained by the lottery ticket hypothesis and follow-up research, which showed that there exist sparse subnetworks that can be trained in isolation to the same performance as the dense networks [15, 66, 62]. In an effort to find these subnetworks without the need for dense training, different techniques have been proposed over the years [32, 60, 59], including random selection [48, 34]. Such approaches are commonly referred to as Sparse Training, or Static Sparse Training to emphasize that the sparse structure stays the same throughout the training. Interestingly, [16] find out that the sparse initializations produced by the static approaches are invariant to parameter reshuffling and reinitialization within a layer and that even pre-training magnitude pruning performs quite well in such a setup.
2The code is provided at https://github.com/alooow/fantasticweights_paper.
Table 1: An overview of the existing pruning criteria in DST. Our work analyzes the differences and similarities between all the methods and fills the gaps  $(\times)$  in the literature. Each column lists a pruning criterion, while each row presents a growing method. SNIP [32] was not designed to grow weights, but we investigate whether it can be applied in DST.
CMagnitude |θ|	CSET |θ+,|θ-|	CMEST |θ| + λ|∇θL(D)|	CSensitivity |∇θL(D)|/|θ|	CSNIP |θ||∇θL(D)|
random growth gradient growth	× RigL[12]	SET[42] ×	MEST[64] ×	Sensitivity[45] × ×	× ×
Dynamic Sparse Training. In DST, the neural network structure is constantly evolving by pruning and growing back weights during training [42, 3, 11, 12, 64, 61, 1, 9, 28]. The key motivation behind DST is not only to provide compression but also to increase the effectiveness and robustness of the deep learning models without the need for overparametrization [38]. The capability of DST has recently been an area of interest. In [13], authors indicate that DST is able to outperform static sparse training by assuring better gradient flow. DST has also been reported to achieve high performance in Reinforcement Learning [18, 53, 19] and Continual Learning [52], with ongoing research into applicability in NLP [35]. Some attention has also been given to the topological properties of the connectivity patterns produced by DST. In particular, [36], investigated the structural features of DST. However, they focus only on one method, the Sparse Evolutionary Training (SET) procedure [42].
To the best of our knowledge, this work is the first to comparatively study a large number of pruning criteria in DST on multiple types of models and datasets. We hope that our analysis will increase the understanding within the dynamic sparse training community.
3.1 Dynamic Sparse Training
Dynamic sparse training is a framework that allows training neural networks that are sparse already at initialization. The fundamental idea behind DST is that the sparse connectivity is not fixed. Instead, it is repeatedly updated throughout training. More precisely, let  $\theta$  denote all the parameters of a network  $f_{\theta}$ , which is trained to minimize a loss  $L(\theta; \mathcal{D})$  on some dataset  $\mathcal{D}$ . The density  $d^{l}$  of a layer  $l$  with latent dimension  $n^{l}$  is defined as  $d^{l} = ||\theta^{l}||_{0} / n^{l}$ , where  $||\cdot||_{0}$  is the L0-norm, which counts the number of non-zero entries. Consecutively, the overall density  $D$  of the model is  $D = \frac{\sum_{l=1}^{L} d^{l} n^{l}}{\sum_{l=1}^{L} n^{l}}$ , with  $L$  being the number of layers in the network. The sparsity  $S$  is given by  $S = 1 - D$ . Before the start of training, a fraction of the model's parameters is initialized to be zero in order to match a predefined density  $D$ . One of the most common choices for initialization schemes is the Erdős-Rényi (ER) method [42], and its convolutional variant, the ER-Kernel (ERK) [12]. It randomly generates the sparse masks so that the density in each layer  $d^{l}$  scales as  $\frac{n^{l-1} + n^{l}}{n^{l-1} n^{l}}$  for a fully-connected layer and as  $\frac{n^{l-1} + n^{l} + w^{l} + h^{l}}{n^{l-1} n^{l} w^{l} h^{l}}$ , for convolution with kernel of width  $w^{l}$  and height  $h^{l}$ . The sparse connectivity is updated every  $\Delta t$  training iterations. This is done by removing a fraction  $\rho_{t}$  of the active weights accordingly to a pruning criterion  $\mathcal{C}$ . The selected weights become inactive, which means that they do not participate in the model's computations. Next, a subset of weights to regrow is chosen accordingly to a growth criterion from the set of inactive weights, such that the overall density of the network is maintained. The pruning fraction  $\rho_{t}$  is often decayed by cosine annealing  $\rho_{t} = \frac{1}{2} \rho(1 + \cos(t\pi / T))$ , where  $T$  is the iteration at which to stop the updates, and  $\rho$  is the initial pruning fraction.
The used pruning and growth criterion depends on the DST algorithm. The two most common growth modes in the literature are random growth and gradient growth. In the first one, the new connections are sampled from a given distribution [42, 45, 64]. The second category selects weights based on the largest gradient magnitude [12]. Other choices based on momentum can also benefit the learning [11].
3.2 Pruning Criteria
Given a pruning fraction  $\rho_{t}$ , the pruning criterion  $\mathcal{C}$  determines the importance score  $s(\theta_{i,j}^{l})$  for each weight and prunes the ones with the lowest score. We use local pruning, which means that the
criterion is applied to each layer separately. $^3$ . For brevity, we will slightly abuse the notation and write  $\theta$  instead of  $\theta_{i,j}^{l}$  from now on. Below, we discuss the most commonly used pruning criteria in DST that are the subject of the conducted study.
SET. To the best of our knowledge, SET is the first pruning criterion used within the DST framework, which was introduced in the pioneering work of [42]. The SET criterion prunes an equal amount of positive and negative weights with the smallest absolute value.
Magnitude. The importance score in this criterion is given by the absolute value of the weight  $s(\theta) = |\theta|$ . Contrary to SET, no division between the positive and negative weights is introduced. Due to its simplicity and effectiveness, the magnitude criterion has been a common choice in standard post-training pruning, as well as in sparse training [15, 12].
MEST. The standard magnitude criterion has been criticized as not taking into consideration the fluctuations of the weights during the training. The MEST (Memory-Economic Sparse Training) criterion [64], proposed to use the gradient as an indicator of the trend of the weight's magnitude, leading to a score function defined as  $s(\theta) = |\theta| + \lambda |\nabla_{\theta}\mathcal{L}(\mathcal{D})|$ , where  $\lambda$  is a hyperparameter.
Sensitivity. The role of gradient information in devising the pruning criterion has also been studied by [45]. Taking inspiration from control systems, the authors propose to investigate the relative gradient magnitude in comparison to the absolute value of the weight, yielding  $s(\theta) = |\nabla_{\theta}\mathcal{L}(\mathcal{D})| / |\theta|$ . In our study, we consider the reciprocal version of that relationship  $s(\theta) = |\theta| / |\nabla_{\theta}\mathcal{L}(\mathcal{D})|$ , which we call RSensitivity, as we found it to be more stable.
SNIP. The SNIP (Single-shot Network Pruning) criterion is based on a first-order Taylor approximation of the difference in the loss before and after the weight removal. The score is given by  $s(\theta) = |\theta| \cdot |\nabla_{\theta} \mathcal{L}(\mathcal{D})|$ . The criterion has been originally successfully used in static sparse training [32] and post-training pruning [43]. Motivated by those results, we are interested in investigating its performance in the dynamic sparse training scenario.
We denote the above-mentioned criteria by adding a subscript under the sign  $\mathcal{C}$  in order to distinguish them from the algorithms that introduced them. We summarize their usage with random and gradient growth criteria in the DST literature in Table 1.
This work aims to understand the key differences and similarities between existing pruning techniques in DST by answering the following research questions:
Q1 What is the impact of various pruning criteria on the performance of dynamic sparse training?
Q2 How does the frequency of topology updates influence the effectiveness of different pruning methods?
Q3 To what extent do different pruning criteria result in similar neural network topologies?
Q1: Impact on the Performance of DST. In the first research question, we are interested in investigating the relationship between the pruning criterion and the final performance of the network. We compare the results obtained by different pruning criteria on a diverse set of architectures and datasets, ranging from a regime with a small number of parameters  $(< 100K)$  to large networks  $(\sim 13M)$ . We examine multi-layer perceptrons (MLPs) and convolutional neural networks (ConvNets). Within each model, we fix the training setup and vary only the pruning criterion used by the DST framework. This allows us to assess the benefits of changing the weight importance criterion in isolation from other design choices. As the growing criterion, we choose the uniform random [42] and gradient [12] approaches, as they are widely used in the literature. In addition, we also perform a separate comparison on the ResNet-50 model ( $25M$  parameters) on ImageNet.
The common sense expectation would be that the more elaborate pruning criteria, which combine the weight magnitude with the gradient information, will perform better than the techniques that rely only on the weight's absolute value. Since the gradient may provide information on how the weight will change in the future, it indicates the trend the weight might follow in the next optimization updates.
<sup>3</sup>We adapt this setup as it is commonly used in DST approaches, and has a reduced risk of entirely disconnecting the network – see Appendix I for a more detailed discussion.
See Appendix B.
Figure 1: The test accuracy versus density computed on the studied models for different pruning criteria (due to space limits we only present here six plots, see Figure 10 in Appendix D for the remaining two setups). The first row represents the results obtained by random growth, while the second corresponds to gradient growth. Note the logarithmic scale in the x-axis. The performance of the dense model is indicated by the horizontal dashed line. In almost every case all pruning criteria perform well, regardless of the chosen model and growth criterion. At the same time, they surpass the static initialization (in light blue).
Therefore it may be more suitable in the DST approach, where connectivity is constantly evolving, making insights into future updates potentially beneficial. This is especially promising considering the effectiveness of the gradient-based approach in estimating the importance in the growth criterion.
Q2: Update Period Sensitivity. In the second research question, we focus on one of the most important design choices: the topology update period  $\Delta t$ . The setting of this update period determines how often the network structure is adapted during training. When considered together with the dataset and batch-size, this hyperparameter can be interpreted as the number of data samples the model trains on before performing the mask update. Therefore, within a fixed dataset and batch size, using a relatively low value means that the topology is changed very frequently in comparison to the optimization update of the weights. This poses a potential risk of not letting the newly added connections reach their full potential. On the other hand, a high value gives the weights enough opportunity to become important but may not allow the sparse network to adjust its structure based on the data characteristics. Finding a balance between this exploitation-exploration conflict is an imperative task in any DST framework.
Q3: Structural Similarity. Finally, we take a deep dive into the structures produced by the various pruning criteria. Please note that while the previous questions, Q1 and Q2, may explain how the pruning criterion choice affects performance and update frequency in DST, they do not indicate whether the mask solutions obtained by those criteria are significantly different. In order to assess the diversity of the studied methods in terms of the produced sparse connectivity structures, we need to compare the sets of weights selected for pruning by each criterion under the same network state. In addition, we also investigate the similarity between the final masks obtained at the end of training and how close they are to their corresponding sparse initializations. This allows us to assess whether the DST topology updates are indeed meaningful in changing the mask structure. For both cases, we incorporate a common score of the proximity of sets, known as the Jaccard index (or Intersection-Over-Union). We compute it separately for each layer and average the result:
$$
\bar {J} \left(I _ {a}, I _ {b}\right) = \frac {1}{L} \sum_ {l = 1} ^ {L} J \left(I _ {a} ^ {l}, I _ {b} ^ {l}\right), \quad J \left(I _ {a} ^ {l}, I _ {b} ^ {l}\right) = \frac {\left| I _ {a} ^ {l} \cap I _ {b} ^ {l} \right|}{\left| I _ {a} ^ {l} \cup I _ {b} ^ {l} \right|}, \tag {1}
$$
where  $I_{a}^{l}$  and  $I_{b}^{l}$  are the sets selected by pruning criteria  $a$  and  $b$  in layer  $l$ . A Jaccard index of 1 indicates that sets overlap perfectly, while 0 implies they are entirely separate.
We expect to see some overlap in the pruning methods' selection of weights since all of them incorporate the use of the magnitude of the weight. We also hope to see if the difference between the connectivities produced by scores that give different performances is indeed large or whether a small adjustment of the weights would render them equal.
In this section, we present the results of our empirical studies. We start with the description of the experimental setup and then answer each of the posed questions in consecutive sections.
5.1 Setup of the Experiments
We perform our analysis using eight different models, including small- and large-scale MLPs and Convolutional Nets. The small-MLP is a 4-layer network with hidden size of at most 256, (trained on the tabular Higgs dataset [2]). The large-MLP also consists of 4 layers (latent dimension size up to 1024 neurons) and is evaluated on the CIFAR10 [27]. The convolutional architectures are: a small 3-layer CNN (CIFAR10), LeNet5-Caffe [30] (FashionMNIST [63]), ResNet56 [22](CIFAR10, CIFAR100), VGG-16 [50](CIFAR100) and EfficientNet [57](Tiny-ImageNet [29]). In addition, on a selected density, we also consider the ResNet50 model on ImageNet [49]. We summarize all the architectures in Appendix A. While our primary focus is on vision and tabular data, we also evaluate the pruning criteria on the fine-tuning task of ROBERTa Large [39] using the CommonsenseQA dataset[56] adapted from the Sparsity May Cry (SMC) Benchmark [35]. In contrast to the setup studied throughout this work, this experiment involves a fine-tuning type of problem. In consequence, we discuss it in Appendix C and focus on training models from scratch for the remainder of the paper.
We train the models using the DST framework on a set of predefined model densities. We use adjustment step  $\Delta t = 800$  and initial pruning fraction  $\rho = 0.5$  with cosine decay. As the growth criterion, we investigate both the random and gradient-based approaches, as those are the most common choices in the literature. Following [64], we reuse the gradient computed during the backward pass when calculating the scores in the gradient-based pruning criteria. In all cases, we ensure that within each model, the training hyperparameters setup for every criterion is the same. Each performance is computed by averaging 5 runs, except for ImageNet, for which we use 3 runs. The ResNet50 model is trained using density 0.2 and gradient growth (we select the gradient-based growth method since it is known to provide
Figure 2: Left: Validation accuracy of the different pruning criteria on ImageNet obtained for density 0.2. The dashed black line indicates the best result of the dense model. Right: The training loss versus the number of epochs. We see that all methods perform similarly, except for  $\mathcal{C}_{\mathrm{SNIP}}$ , which suffers already at the beginning of the training.
good performance in this task [12]). Altogether we performed approximately 7000 runs of dense, static sparse, and DST models, which in total use 8 (9 including ImageNet) different dataset-model configurations, and jointly took approximately 5275 hours on GPU.
5.2 Impact on the Performance of DST
In this section, we analyze the impact of the different pruning criteria on the test accuracy achieved by the DST framework. We compare the results with the original dense model and static sparse training with ERK initialization. The results are presented in Figure 1.
In this work, we are interested in how the frequency of the topology update impacts different pruning criteria. Intuitively, one could anticipate that the pruning criteria incorporating the gradient
For training details please refer to Appendix A.2.
(a) all densities
(b) density  $\leq 0.2$
Figure 3: The critical distance diagram of the studied methods for (a) all densities, (b) densities smaller or equal 0.2, and (c) densities larger than 0.2. For each model we compute its average rank (the lower the better) and plot it on the horizontal line. The methods with ranks not larger than the critical distance (denoted as CD, displayed in the left upper corner of each plot) are considered not statistically different and are joined by thick horizontal black lines.
(c) density  $>0.2$
information will perform better with a smaller update period, as the gradient may indicate the trend in the weight's future value.