SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Recall from Section 3.2 that the MEST criterion is computed as  $s(\theta) = |\theta| + \lambda |\nabla_{\theta}\mathcal{L}(\mathcal{D})|$ . Since  $\mathcal{C}_{\mathrm{MEST}}$  depends on the  $\lambda$  hyperparameter, we verify how the change of that value impacts the overall results for various densities. We present the results on the large-MLP model in Figure 7. We observe that for  $\lambda$  close to 0, the MEST criterion is, as expected, close to the magnitude criterion. Increasing  $\lambda$  does not seem to benefit the performance. In particular, depending highly on the gradient is disadvantageous.
Figure 7: The performance of the MEST criterion with random growth for different parameters of the hyperparameters  $\lambda$ . We observe that for low values of  $\lambda$ , the criterion behaves similarly to magnitude, while for very large values, the performance drops. There does not seem to be a good value between those two extremes.
B Sensitivity and Reciprocal Sensitivity
We investigate the relation between the Sensitivity and Reciprocal Sensitivity on the large-MLP model on CIFAR10. The results are presented in Figure 8. We observe that not only is the Reciprocal criterion better in terms of the attained test accuracy, but it also seems to be more stable throughout the training. We argue that this is due to how those two criteria deal with weights that are large in magnitude but have small gradients. Note that such weights can be considered stable and hence
These are rounded percentages. We use the standard split of 45,000 - 5,000 - 10,000 images.
This is the average resolution. Images are randomly cropped to size 3x224x224, as is standard practice.
16 Percentages are rounded. We use the standard train/validation/test split. We do not use the test set and evaluate on the validation set.
17Percentages are rounded. We apply the default train/test split (60000/70000) and move a random  $10\%$  of the train set as a validation set.
18We use the standard train/Validation/test split. We use the validation set as the test set (we do not optimize any hyperparameters using the validation set).
be desired to keep active. While the Reciprocal criterion would assign them a very high score, the Sensitivity, having an inverse relation to the gradient, would most likely mark them for removal.
Figure 8: Left: The density versus test accuracy for the MLP dataset on CIFAR10 obtained by the Reciprocal Sensitivity and Sensitivity pruning criteria. Right: The validation accuracy throughout the training for density 0.2 (denoted by a vertical dashed line in the first plot). We may observe that the Reciprocal Sensitivity is more stable and gives much better results.
C Results On Text Data
In addition to the tasks studied in the main text, we conduct an evaluation of the pruning criteria on the fine-tuning task of ROBERTa Large [39] with approximately 354 million parameters, utilizing the CommonsenseQA dataset [56] adapted from the Sparsity May Cry (SMC) Benchmark [35].
We use the exact same setup as in the SMC-Benchmark by first performing magnitude pruning for the sparse initialization and then using DST during the fine-tuning phase. We also use the same hyperparameters. Please note that this is a different configuration from the one used in the main text, in which we trained all the models from scratch (in spirit, the CommonsenseQA study is more similar to the EfficientNet fine-tuning experiment from Appendix H). We report the results in Table 5 and Table 6 for the random and gradient growth, respectively.
Similar to the findings of [35], we observe that in this scenario, the achievable sparsity without a significant performance decline is notably lower than in vision or tabular data scenarios. Concerning random growth, the  $\mathcal{C}_{MEST}$  criterion appears to be the most suitable choice. For gradient growth with a density of 0.8, the  $\mathcal{C}_{R\text{Sensitivity}}$  and  $\mathcal{C}_{SET}$  criteria initially demonstrate strong performance but are eventually surpassed by the  $\mathcal{C}_{\text{magnitude}}$  and  $\mathcal{C}_{MEST}$  criteria at a density of 0.7. Additionally, we note a significantly higher variance in the outcomes across all criteria compared to vision tasks.
Furthermore, we conducted a brief exploration of update period values for this problem (refer to Figure 9). Notably, we observe that overly frequent updates do not yield beneficial results. Concerning gradient growth, the most effective update period value is consistently around  $\Delta t = 500$  for nearly all criteria. For random growth higher update period values generally lead to improved outcomes.
At the same time, please note that the studied pruning criteria had been evaluated either on tabular or vision datasets in most of the works that have introduced them or used them (e.g., [42, 64, 45, 12, 11]). Large language models typically have a different structure and are based on the attention mechanism. In consequence, the applicability of sparse training in such a setup is still an open area of research [35, 55]. This motivated our choice of fixing our focus on tabular and vision models in the main text.
D Additional Performance Plots
In this section we present the performance of the different pruning criteria on the two remaining architectures from the main text - the small-CNN trained on CIFAR10, and ResNet56 trained on CIFAR100. The results are in Figure 10 (see also Figure 22 for all the models in one plot).
Table 5: The mean accuracy results for the CommonsenseQA task on ROBERTa Large Model from the SMC-Benchmark (Liu et al. 2023). We use the same hyperparameter setup as in (Liu et al. 2023) and vary only the density and pruning criterion. The results below were obtained for random growth. In brackets, we report the standard deviation from 3 runs. We bold out the best result for each density and underline the second-best one.
random growth
0.9	0.8	0.7	0.6	0.5
Cmagnitude	73.80 (0.52)	37.57 (25.25)	26.53 (6.71)	19.80 (3.13)	20.60 (0.69)
CSET	74.40 (0.60)	49.50 (26.98)	21.00 (3.99)	20.13 (1.95)	18.63 (1.44)
CMEST	74.77 (0.67)	52.30 (29.79)	26.90 (7.20)	19.77 (0.49)	20.73 (0.96)
CRSensitivity	25.87 (8.36)	22.13 (1.69)	23.27 (0.85)	21.37 (1.66)	21.20 (0.53)
CSNIP	56.57 (1.82)	34.20 (9.11)	19.57 (1.31)	21.60 (0.95)	19.17 (1.47)
Table 6: The mean accuracy results for the CommonsenseQA task on ROBERTa Large Model from the SMC-Benchmark (Liu et al. 2023). We use the same hyperparameter setup as in (Liu et al. 2023) and vary only the density and pruning criterion. The results below were obtained for gradient growth. In brackets, we report the standard deviation from 3 runs. We bold out the best result for each density and underline the second-best one.
gradient growth
0.9	0.8	0.7	0.6	0.5
Cmagnitude	76.00 (0.46)	55.80 (29.51)	37.37 (18.23)	24.27 (2.94)	19.33 (0.35)
C_SET	76.30 (0.75)	68.43 (6.15)	28.57 (1.85)	22.60 (1.47)	22.17 (0.93)
C_MEST	74.73 (0.15)	53.90 (29.27)	37.53 (8.45)	24.27 (2.21)	19.63 (1.12)
CRSensitivity	75.83 (0.55)	69.60 (6.70)	25.43 (5.65)	20.67 (2.81)	21.57 (1.82)
CSNIP	65.50 (2.27)	48.20 (7.79)	32.70 (5.98)	23.17 (4.82)	20.73 (0.61)
Figure 9: The study of the effect of the change of update period  $\Delta t$  on the performance in the CommonsenseQA task on ROBERTa Large. Left: random growth, right: gradient growth.
Figure 10: The performance of the pruning criteria on the CNN-CIFAR10 and ResNet56-CIFAR100 datasets for different densities. Please note the logarithmic scale on the x-axis. We observe that the differences between the methods are larger for the low densities.
We observe that again the differences between the studied methods seem to be more visible in the low-density regime. In that setup, the  $\mathcal{C}_{SNIP}$  criterion is achieving worse test accuracy than the  $\mathcal{C}_{MEST}$  and the magnitude-based methods. The  $\mathcal{C}_{RSensitivity}$  also performs poorly on the CNN-CIFAR10 model. We can see that the behavior for the VGG-16 network observed in the plots in the main text is not due to the change of dataset, but rather the change of the model (since ResNet56 on CIFAR100 has similar results to ResNet56 on CIFAR10 from the main text).
E In-Time-Over-Parametrization (ITOP) Study
A known component that has been used to explain sparse training performance together with update-period sensitivity is the ITOP (In Time Over-Parametrization) ratio [38]. The ITOP measures the number of 'explored' parameters divided by the total number of parameters. The parameter is 'explored' if it has been included in the mask at least once during the whole training.
We study the relationship between ITOP, update period, and the pruning criteria on the example of the large-MLP model on CIFAR10 with density 0.1. The results are shown in Figure 11. There are some interesting insights gained by this analysis.
First, we observe that when paired with random growth, the choice of pruning criteria leads to almost identical final ITOP ratios. (Note that the points for different pruning criteria overlap in the left plot, yet we verify that they are not exactly equal). This is expected because, in such a case, the growth
Figure 11: The update period  $\Delta t$  versus the ITOP ratio obtained at the end of the training for different pruning criteria on the large MLP. The left plot corresponds to random growth, the right plot corresponds to gradient growth.
criterion is independent of the chosen set of weights to prune. Hence, the increase in ITOP is only due to the growth criterion, which (being random selection) is not affected by the pruning criterion. This is in contrast to gradient growth, in which the change in the pruned weights can impose a change to the gradient flow in the model and hence impact the rate of the exploration. Aside from this observation, we confirm that gradient growth has, in general, a lower ITOP ratio than random (which was shown in [38]).
Secondly, in the gradient growth setup, the magnitude criteria and  $\mathcal{C}_{MEST}$  have a much higher ITOP ratio for small update periods than the gradient-based pruning criteria. This indicates that they perform more exploration, which perhaps may be the reason behind their better performance observed in Figure 4 in the main text.
Finally, we notice that final ITOP ratio close to 1 is not necessary for good generalization. Observe that for  $\Delta t = 800$ , we get the best validation accuracy for random growth, MLP, and CIFAR10 (Figure 5 in our paper), but it only reaches an ITOP ratio of 0.7. Even with a final ITOP ratio of just 0.2 (for  $\Delta t = 6400$ ), sparse models can beat the performance of dense models.
F Sparse Hardware and Software Support
As discussed in [53], research on sparsity is being pursued in three main areas to create faster, more efficient neural networks. The first area is the development of hardware that can take advantage of sparse networks, such as NVIDIA's A100, which supports 2:4 sparsity (a restricted version of  $50\%$  layer sparsity) [65]. The second area is the creation of software libraries that can implement truly sparse networks [37, 8]. The third area is the development of algorithms that can produce sparse networks that perform just as well as dense networks [23]. All these steps are being undertaken as a collaborative community effort, aiming to create faster, memory- and energy-efficient deep neural networks. Additional information on this topic can be found in [24, 41].
G Gradient Based Methods and Batch Size
In Section 5.2 in the main text, we observe that, in general, the magnitude-based pruning criteria perform better in sparse regimes than the gradient-based ones. Note that the gradient is computed for a batch (using SGD), and hence, small batch sizes might introduce more significant errors in Taylor-approximation based pruning (e.g., SNIP). At the same time, computing the gradient over the entire training set is not feasible in DST since it would significantly increase the time complexity of each mask update.
Figure 12: The performance of different pruning criteria on LeNet-5-Caffe on FashionMNIST. The first row corresponds to random growth, the second row corresponds to gradient growth. The left column indicates a batch of size 128, and the right column indicates a batch of size 1024. The black dashed line in both plots is the performance of the dense model from the main text.
To investigate the impact of the batch size on our results, we perform a small study on the LeNet-5-Caffe model (trained on FashionMNIST), in which we increase the batch size almost 10 times up to 1024. At the same time, we increase the learning rate up to 0.08 from 0.01 so that the learning-rate-to-batch-size ratio stays approximately the same (as suggested by the scaling laws, for instance, in [51]). We compare the obtained results with the standard setup with a batch size of 128. We also decrease the update period from 800 to 100 so that in both setups, the DST updates are made after seeing the same number of training examples. We report the results in Figure 12.
Interestingly, we observe that for batch size 1024  $\mathcal{C}_{SNIP}$  indeed seems to perform slightly better.  $\mathcal{C}_{RSensitivity}$  and  $\mathcal{C}_{SET}$ , on the other hand, become significantly unstable, and  $\mathcal{C}_{MEST}$  is still indistinguishable from magnitude pruning. Those results may indicate that the impact of the batch size, although existent, is not significant. Furthermore, as we will see in the next paragraph, increasing only the batch size of the DST update is also not beneficial.
Increasing Only the Batch Size of the DST update. In addition to the above-conducted experiments, we examine an ablation in which we increase the batch size to 1024 only when computing the gradient used for the DST update. During normal steps of optimization, we use the standard choice from previous experiments (128). The results are in Figure 13. We observe that it does not enhance performance. Let us note, however, that, in general, the impact of the batch size still needs additional investigation in larger models and datasets in order to be evaluated with certainty.
H Fine-Tuning EfficientNet and the Choice of Pruning Criterion
In all experiments in the main text, we perform the training from scratch (no pretraining is applied). This includes the EfficientNet on the Tiny-ImageNet from the main text. In this section, we also analyze the difference in the performance of the pruning criteria when evaluated in a fine-tuning
Figure 13: The performance of MLP-CIFAR10 and LeNet5-Caffe-FashionMNIST from the main text experiments (first and third column) and the performance when using an increased batch only for the DST update (second and last column).
scenario. In such a setup, we initialize the weights of the EfficientNet to use the pretrained parameters from ImageNet. We only re-initialize the linear projection in the head of the model (which needs to have the output dimension changed from 1000 to 200 since there are 200 classes in Tiny-ImageNet). We keep all hyperparameters the same as in the training-from-scratch setup (see Appendix A). We compare the results in Figure 14
We observe that in the setup that uses the pretrained weights, the  $C_{SNIP}$  criterion performs much better than when training from scratch. We hypothesize that perhaps the use of the Taylor approximation of the change in the Loss function (used in the SNIP criterion) allows not to diverge far from the already quite well-established solution from the pretraining. The purely magnitude-based methods, on the other hand, disregard the potential change in the loss they may cause and hence, do not successfully leverage the information from pretraining. However, at the same time, the general performance of all DST methods when starting with pretrained parameters on ImageNet is much worse in the low-density regime compared to their DST counterparts trained from scratch. This may suggest that the DST framework in the fine-tuning scenario is heavily biased by the pretrained weights and potentially does not efficiently explore possible connectivity patterns. These results suggest that one should be careful not to transfer the findings of the DST training to fine-tuning scenarios. In the second case, more research is needed to understand the behavior of the DST algorithms.
I Global versus Local Pruning
In our study, we have decided to focus on local pruning as this is the typical approach in the DST literature. Examples include RigL [12] (Algorithm 1), SNFS [11] (Figure 1, where pruning and redistribution occur layer-wise), SET [42]. In contrast, global pruning computes and compares the importance scores associated with the pruning criterion with all the parameters in the network. Hence, global pruning may change the sparsity distributions in each layer in an unsupervised manner. Note that global pruning may also have one small disadvantage: It may carry a higher risk of disconnecting the model if the importance scores  $s(\theta_{i,j}^{l})$  are not balanced across different layers. Using local pruning may avoid this risk and omit the need for investigating layer-wise normalization schemes.
However, we do consider the question of how the studied pruning criteria perform when paired with global pruning an interesting one. We perform a small experiment on ResNet-56 with CIFAR10 in which we compare the different pruning criteria in the global- and local-pruning setup on densities
Figure 14: The performance of different pruning criteria on the EfficientNet architecture trained from scratch (left) and fine-tuned on the weights from training on ImageNet (right). The first row corresponds to random growth, and the second row corresponds to gradient growth. We can observe that in the setup that uses the pretrained weights (right column), the  $\mathcal{C}_{\text{SNIP}}$  criterion performs much better than in training from scratch. However, at the same time, the general performance of all DST methods is much worse in the low-density regime compared to the training from scratch.
0.05, 0.1, and 0.15. We present the results in Figure 15. $^{19}$  We observe that the results obtained in both global- and local- setups are indeed similar, with a slight preference for global pruning. By analyzing the plots, we can see that, in this case, the ranking results of the pruning criteria from the main text are indeed preserved. We consider the subject of using global pruning a very intriguing topic to investigate further in future research.
J Effect of Batch Size and Pruning Schedule on Update Period
When considered together with the batch size and dataset size, the update period can be interpreted as the number of data samples the model sees (and trains on) before performing the mask update. In consequence, one may expect that with the increase of the batch size, the best choice of the update period will shift to lower values, keeping the total number of seen examples similar. To visualize this effect, we run a comparison on the MLP-CIFAR10 setup and observe just this behavior (see Figure 16(c) and Figure 16(d)).
Another potential confounding factor is the pruning rate schedule. In all our main experiments, we used the cosine schedule, which is known to perform well in the DST setup [12]. In this section, we additionally test this schedule on the MLP-CIFAR10 setup against two different schedulers: linear and constant. The linear schedule multiplies every 600 optimization steps the current decay rate by a constant factor (0.99). The constant schedule keeps the prune fraction fixed at 0.5 throughout the whole training. The results are presented in Figure 16(a) and Figure 16(b). We observe that the choice of the pruning factor schedule does not significantly influence the best value of the update period.
19We leave the  $\mathcal{C}_{\mathrm{SET}}$  out due to its high similarity to the magnitude pruning.
Figure 15: The test accuracy of ResNet-56 model using different criteria for densities 0.05, 0.1, 0.15. We plot the test accuracy obtained with global pruning (y-axis) versus the test accuracy obtained with local pruning (x-axis). Note that if the performance in both setups was the same, the values would perfectly lie at the gray, dashed diagonal line. Points above that line indicate that global pruning performed better.
Figure 16: Additional results for the sensitivity to hyperparameters in the update-period study. We perform the study on random growth. From the left, we plot the results for (a) linear scheduler and default (i.e., 128) batch size (BS), (b) constant scheduler and default batch size, (c) cosine scheduler and default batch size (the same setup as in main paper), (d) cosine scheduler and batch size 256.
K Additional Jaccard Index Plots
We present the Jaccard index values computed after the first topology update of the model for different initial pruning ratios  $\rho$  in Figure 17. Naturally, for smaller  $\rho$ , the difference becomes more visible. The  $C_{\mathrm{Magnitude}}$ ,  $C_{\mathrm{MEST}}$  and  $C_{\mathrm{SET}}$  criteria choose similar sets of weights to prune, while  $C_{\mathrm{SNIP}}$  and  $C_{\mathrm{RSensitivity}}$  are more diverse in their selections.
In addition, we also study how the similarity between the chosen sets for removal changes with time. To this end, we run each criterion with the DST framework until the end of training. Next, every 3200 iterations, we examine what weights for removal would be chosen by the remaining pruning criteria at this point in training. Note that such a relationship does not need to be symmetric. We report the results in Figure 18.
Moreover, we analyze the statistics of the never-removed connections. To be precise, we consider the masks at the end of training for the MLP, CNN, and ResNet-56 models on CIFAR10. Next, we measure the number of weights that are always retained by every pruning criterion for the same seed and we divide it by the number of all remaining (unpruned) weights in the model. We do the same on the negation of the masks to get the number of the always removed weights and divide it by the number of all removed weights of the model. We present the results in Figure 19. For the MLP model, around  $23\%$  of all the retained weights are shared across all the models. For CNN, this number is higher and goes up to  $37\%$ , while for the ResNet-56 it is equal to  $11\%$ . In general, the smaller overlap of the pruning methods for the ResNet model is also something that we observe in the main paper in Figure 6b.
Finally, we report the standard deviations for the Jaccard Index plots from the main text in Figures 20 and 21. Note that the mean Jaccard Index for the sets chosen for removal in the main text was computed by first taking a mean over different seeds for each layer and then averaging the results over all layers. Hence, in Figure 20, the standard deviation is calculated for each layer separately and then averaged. For the end-similarity, the Jaccard Index was computed by first taking the mean over all layers for a given seed and then taking the mean over all random seeds. Hence, in Figure 21, we report the standard deviation for averaging over the different random seeds.
(a) MLP
(b) CNN
(c) ResNet-56
Figure 17: The Jaccard index between the selected pruning criteria computed for the same network state. The values are averaged over 5 different seeds. We observe that the  $C_{\mathrm{Magnitude}}$ ,  $C_{\mathrm{MEST}}$  and  $C_{\mathrm{SET}}$  criteria select similar sets of weights to prune.
L Learning Curves
In this section, we plot the validation accuracy during the training for different densities and pruning criteria for the small and large MLPs and ConvNets. We notice that  $\mathcal{C}_{\mathrm{SNIP}}$  and  $\mathcal{C}_{\mathrm{R Sensitivity}}$  criteria often lead to instabilities during the learning - see Figures 23, 24, 25, 26, 27, 28, 29, 30.
(c) ResNet-56