SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Firstly, we observe that the differences between the pruning criteria are usually most distinctive in low densities. Furthermore, in that setting, the dynamic sparse training framework almost always outperforms the static sparse training approach. This is important, as in DST research, high sparsity regime is of key interest. Surprisingly, the more elaborate criteria using gradient-based approaches usually either perform worse than the simple magnitude score, or do not hold a clear advantage over it. For instance, the Taylor-based  $\mathcal{C}_{\mathrm{SNIP}}$  criterion, despite being still better than the static initialization, consistently achieves results similar or worse than  $\mathcal{C}_{\mathrm{Magnitude}}$ . This is especially visible in the DST experiments using gradient growth (note ResNet56, MLP-CIFAR10, and EfficientNet). The only exception is the VGG-16 network. In addition, we also observe that  $\mathcal{C}_{\mathrm{SNIP}}$  gives better performance when used in a fine-tuning setup for EfficientNet - see Appendix H. Consequently, this Taylor-approximation-based criterion, although being well suited for standard post-training pruning [43] and selecting sparse initialization masks [32] seems not to be the optimal choice in dynamic sparse training approaches on high sparsity. The  $\mathcal{C}_{\mathrm{RSensitivity}}$ , on the other hand, can outperform other methods but is better suited for high densities. Indeed, it is even the best choice for densities larger than 0.15 for large-MLP on CIFAR10 and ResNet models. Finally, the  $\mathcal{C}_{\mathrm{MEST}}$  criterion also is not superior to the  $\mathcal{C}_{\mathrm{Magnitude}}$ , usually leading to similar results. We also present the results obtained for ResNet-50 on ImageNet with density 0.2 in Figure 2. We observe that, again, the  $\mathcal{C}_{\mathrm{SNIP}}$  criterion leads to the worst performance. The  $\mathcal{C}_{\mathrm{RSensitivity}}$ ,  $\mathcal{C}_{\mathrm{SET}}$ , and  $\mathcal{C}_{\mathrm{MEST}}$  methods achieve high results but not clearly better than the  $\mathcal{C}_{\mathrm{Magnitude}}$  criterion.
Figure 4: Left: The loss gap (validation loss - training loss) over time for different pruning criteria, the static sparse model, the dense model, and the dense model with dropout  $p = 0.05$  on the large-MLP. Right: The test accuracy obtained for each criterion.
Within each studied density, growing criterion, and model, we rank the pruning criteria and then calculate their average ranks. To rigorously establish the statistical significance of our findings, we compute the Critical Distance Diagram for those ranks, using the Nemenyi post-hoc test [10] with p-value 0.05 and present it in Figure 3. We observe that in the low-density regime (Figure 3b), the  $C_{\text{Magnitude}}$  criterion achieves the best performance, as given by the lowest average rank. Furthermore, in such a case, we also note that other predominately magnitude-based criteria, such as  $C_{\text{SET}}$  and  $C_{\text{MEST}}$  are not significantly different than  $C_{\text{Magnitude}}$ , while the remaining gradient-based approaches are clearly worse. This confirms the overall effectiveness of the
<sup>6</sup>We study the impact of the  $\lambda$  hyperparameter of the  $C_{\mathrm{MEST}}$  criterion in Appendix A.3. In general, we see that  $\lambda \rightarrow 0$  leads to results equal to  $C_{\mathrm{Magnitude}}$ , while  $\lambda \rightarrow \infty$  prioritizes only the gradient magnitude and degrades the performance. We do not see any clear improvement for values lying between those two extremes.
(a) density 0.2
(b) density 0.1
Figure 5: The update period  $\Delta t$  versus validation accuracy on the CIFAR10 dataset on the MLP, ConvNet, and ResNet-56 models for different pruning criteria with density 0.2 and 0.1. The top row corresponds to random growth, while the bottom row corresponds to gradient growth. Note the logarithmic scale on the x-axis. We see that the methods are most sensitive to the update frequency in the MLP setting. For all setups, performing the update later ( $\Delta t > 400$ ) seems to be beneficial.
weight's magnitude as an importance score in sparse regimes. See also Appendix G for the discussion of the effect of batch size on the obtained results. Interestingly, when larger densities are involved, the best choice is given by  $\mathcal{C}_{\mathrm{RSensitivity}}$ . Additionally, the average rank of  $\mathcal{C}_{\mathrm{magnitude}}$  deteriorates. This may suggest that the information from the gradient is more reliable in such a case. The gradient-based methods seem to perform better in denser setups.
Finally, we notice that in certain situations, the DST framework can lead to better results than the original dense model (see small-MLP, or ResNet56 on CIFAR100 in Figure 1). We argue that this is due to a mix of a regularization effect and increased expressiveness introduced by removing and adding new connections in the network - see Figure 4. Moreover, we also observe that within the setup studied in Figure 1, there is almost no difference between choosing the random or gradient growth criterion. The second one achieves slightly better accuracy for the convolutional models (see, e.g., density 0.2 on ResNet-56, as well as improved stability for EfficientNet). The similarity between those two growth criteria and the surprising effectiveness of DST in comparison to dense models have also been previously observed in the context of Reinforcement Learning [18].
5.3 Sensitivity to Update Period
In this section, we analyze the sensitivity of each pruning criterion to the update period  $\Delta t$ . This value describes how often the adjustments are made in training and hence can affect the weights' importance scores. We fix the density to 0.2 and 0.1 and the initial pruning fraction  $\rho$  to 0.5. We choose the densities 0.2 and 0.1 as they correspond to rather high sparsity levels and generally provide reasonable performance. Next, we investigate different update periods on the models using CIFAR10 dataset. We start from  $\Delta t = 25$  and increase it by a factor of 2 up to  $\Delta t = 6400$ .
The results are presented in Figure 5. For the MLP model, we clearly see the behavior described in Section 4: if the update period  $\Delta t$  is too small, the performance will suffer. The best update period setting seems to be  $\Delta t = 800$  for the random growth, regardless of the pruning method. For gradient growth, this value also gives a good performance. However, larger pruning periods (e.g.  $\Delta t = 1600$ ) are even better. $^{8}$  Note that we use a batch size of 128, hence for  $\Delta t = 800$  the connectivity is changed approximately once per 2 epochs. $^{9}$  Similarly, less frequent updates are also beneficial in the ResNet56 model, although there is not as much consensus between the different pruning criteria. Even performing just one topology update every  $\Delta t = 6400$  iterations (approximately once per 16 epochs) still gives excellent performance. At the same time, not performing any topology updates at
The update period is only meaningful when considered together with the size of the training set. Therefore it is important to evaluate models of different sizes on the same dataset.
See also Appendix J for an analysis of the effect of batch size and pruning schedule on these results.
${}^{9}$  Since  ${800} \cdot  {128} = {102},{400}$  samples,which is just over twice the training set size of CIFAR10 (50,000).
(b)
Figure 6: (a) The mean normalized Jaccard index between the sets of weights chosen for removal during the first update of the sparse connectivity, computed for the MLP, CNN, and ResNet-56 models on CIFAR10. The  $J_{r}$  indicates the expected overlap of random subsets and serves as a reference point. We estimate this value by computing the mean pair-wise Jaccard index of the random pruning criterion with different sparse initialization. (b) The same index computed between the masks obtained by different pruning criteria at the end of training. The rows and columns represent the pruning criteria between which the index is computed. The  $J_{init}$  denotes the expected overlap of random masks and serves as a reference point. We estimate this value by computing the mean pair-wise Jaccard index of the different sparse initializations.
all (i.e., static sparse training) deteriorates performance, as shown by the cyan dashed line in Figure 5. Furthermore, we observe that the gradient-based pruning criteria seem to suffer much more than the magnitude criteria when paired with a small update period (consider the plots for  $\Delta t = 25$ ), especially when combined with the gradient growth. $^{10}$  This suggests that it is safer not to use those techniques together or be more rigorous about finding the right update period in such a case.
Most interestingly, the results from Figure 5 strongly suggest that frequent connectivity adjustment is not needed. Even few pruning and re-growing iterations are enough to improve the static initialization.
5.4 Structural Similarity
Finally, we investigate whether the studied criteria are diverse in terms of the selected weights for removal. We fix the dataset to CIFAR10 and analyze the moment in which the sparse connectivity is updated for the first time. At such a point, given the same sparse initialization and random seed, all the pruning criteria share the same network state. Consequently, we can analyze the difference between the sets selected for removal by each criterion. We do this by computing their Jaccard index (Equation 1) and present the averaged results of 5 different runs in Figure 6a (See also Appendix K standard deviations and more plots).
We observe that the  $C_{\mathrm{SET}}$  and  $C_{\mathrm{Magnitude}}$  mainly select the same weights for pruning. This indicates that the absolute values of the positive and negative weights are similar. In addition, for the simpler models with smaller depths (MLP and CNN) the  $C_{\mathrm{MEST}}$  criterion also leads to almost the same masks as  $C_{\mathrm{Magnitude}}$ . On the other hand, the  $C_{\mathrm{SNIP}}$  and  $C_{\mathrm{RSensitivity}}$  criteria produce sets that are distinct from each other. This is natural, as  $C_{\mathrm{SNIP}}$  multiplies the score by the magnitude of the gradient, while  $C_{\mathrm{RSensitivity}}$  uses its inverse. The experiment suggests that early in training, pruning criteria such as  $C_{\mathrm{Magnitude}}$ ,  $C_{\mathrm{SET}}$ , and  $C_{\mathrm{MEST}}$  lead to similar sparse connectivity. Together with the similarity in performance observed in Section 5.2, the results indicate that these three methods are almost identical, despite often being presented as diverse in DST literature. At the same time, the updates made by  $C_{\mathrm{SNIP}}$  and  $C_{\mathrm{RSensitivity}}$  produce different pruning sets, and result in lower performance (recall Figure 3b). However, note that for each entry (except  $C_{\mathrm{SNIP}}$  with  $C_{\mathrm{RSensitivity}}$ ), the computed overlap is larger than random, suggesting that there is a subset of weights considered important by all the criteria.
In addition, we also compare how similar are the final sparse solutions found by the methods. To this end, we fix the sparse initialization to be the same for each pair of criteria and compute the Jaccard Index of the masks at the end of the DST training (performed with gradient growth and density 0.2 on the CIFAR10 experiments from Section 5.2). For each pair, we average the score over 5 initializations. The resulting similarity matrix is presented in Figure 6b. We observe that pruning criteria that selected similar sets of weights for removal from the previous experiment still hold some resemblance to each other in terms of the end masks. Let us also note that in Section 5.3, we observed
10In Appendix E we also study the exploration imposed by the pruning criteria using the ITOP ratio [38].
that applying just a few connectivity updates is sufficient for good results. This raises the natural question of whether DST masks are genuinely diverse from their sparse initializations. By analyzing the bottom row in Figure 6b, we see that the masks obtained at the end of the DST training are distinct from their corresponding initializations, having the Jaccard Index close to that of a random overlap.
In consequence, we conclude that the best-performing methods from Section 5.2 indeed make similar decision choices while having a smaller overlap with the less efficient criteria. This renders the magnitude-based criteria almost equivalent, despite their separate introduction in the literature. At the same time, the DST mask updates made during the training are necessary to adapt and outperform the static initialization. We would like to highlight that such insights could not have been derived solely from performance and hyperparameter results. We hope that our insights will raise awareness of the importance of performing structural similarity experiments in the DST community.
We design and perform a large study of the different pruning criteria in dynamic sparse training. We unveil and discuss the complex relations between these criteria and the typical DST hyperparameters settings for various models. Our results suggest that, overall, the differences between the multiple criteria are minor. For very low densities, which are the core of DST interest, criteria based on magnitude perform the best. This questions the effectiveness of gradient-based scores for pruning during the training. We propose to incorporate the selected models and datasets as a baseline in future works investigating the pruning criteria for DST to avoid the common case where methods are overfitted to given DST hyperparameters or tasks. We hope that our research will contribute to the understanding of the sparse training methods.
Limitations & Future Work. Our research was conducted mainly on datasets from computer vision and one tabular dataset. It would be interesting to verify how our findings translate to large language models. The computed structural insights are based on set similarity and disregard the information about the value of the parameters. Additional topographic insights could be provided to incorporate this information and analyze the graph structure of the found connectivity. Finally, much work still needs to be done on the hardware side to truly speed up training times and decrease memory requirements (see Appendix F). We do not see any direct social or ethical broader impact of our work.
The research of Aleksandra Nowak and Jacek Tabor has been supported by the flagship project entitled Artificial Intelligence Computing Center Core Facility from the DigiWorld Priority Research Area under the Strategic Programme Excellence Initiative at Jagiellonian University. The work of J. Tabor was supported by the National Centre of Science (Poland) Grant No. 2021/41/B/ST6/01370. The research of Bram Grooten was funded by the project AMADeuS (with project number 18489) of the Open Technology Programme, which is partly financed by the Dutch Research Council (NWO). We gratefully acknowledge Polish high-performance computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2022/015887.
[1] Zahra Atashgahi, Joost Pieterse, Shiwei Liu, Decebal Constantin Mocanu, Raymond Veldhuis, and Mykola Pechenizkiy. A Brain-inspired Algorithm for Training Highly Sparse Neural Networks. Machine Learning, pages 1-42, 2022. URL: https://arxiv.org/abs/1903.07138.
[2] Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for Exotic Particles in High-Energy Physics with Deep Learning. Nature communications, 5(1):1-9, 2014. URL: https://arxiv.org/abs/1402.4735.
[3] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep Rewiring: Training very sparse deep networks. International Conference on Learning Representations, 2018. URL: https://arxiv.org/abs/1711.05136.
A Experiments Setup
For the small-MLP setup, we use an MLP with 2 hidden layers with dimension sizes of 256. For the large MLP, we increase the number of neurons in the first hidden layer to 1024, and to 512 in the second. The small-CNN is a standard ConvNet architecture consisting of three convolutional layers with kernel size 3, stride 1, and padding 1, inspired by the standard ConvNet presented in section 8.2.3 of [7]. The spatial dimension reduction is implemented by MaxPooling with size 2. In all cases, we use ReLU activations. The architectures are summed up in Table 2. The ResNet-56 and ResNet-50 models follow their standard specifications [22]. The EfficientNet is the EfficientNet-B0 model [57] (we use the implementation from pytorch models), and follows the standard architecture. The VGG-16 model is modified to be used with CIFAR-10 and uses BatchNorm. The implementation of VGG-16, as well as the LeNet-5-Caffe model, is adapted from [34].
Table 2: The small-MLP, large-MLP, and small-CNN architectures used in this study. The first two pairs next to a Linear layer indicate the input and output dimensions. The tuple  $(c_{in}, c_{out}, k, s)$  after the Conv2d described the input channels, output channels, kernel size, and stride, respectively. The ResNet-50, and ResNet-56 architectures follow the standard specifications [22] (25,502,912 and 855,770 parameters, respectively). All networks use a Softmax nonlinearity after the last mentioned layer.
Layer	small-MLP	large-MLP	small-CNN
1	Linear(24, 256)	Linear(3 * 32 * 32, 1024)	Conv2d(3, 32, 3, 1)
2	ReLU	ReLU	ReLU
3	Linear(256, 256)	Linear(1024, 512)	MaxPool, size 2
4	ReLU	ReLU	Conv2d(32, 64, 3, 1)
5	Linear(256, 1)	Linear(512, 10)	ReLU
6	MaxPool, size 2
7	Conv2d(64, 128, 3, 1)
8	ReLU
9	Global Average Pool
10	Linear(128, 10)
Num. Parameters (Dense)	72,449	3,676,682	94,538
Num. Params. (95% Sparse)	3,622	183,834	4,727
A.2 Training Regime
We provide an overview of the datasets we used in our experiments is shown in Table 4. Below, we report the training regime.
Small- and large- MLPs We use a batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0005, and Nesterov  $=$  True. The learning rate starts at 0.01 and is decayed twice during training: after  $50\%$  and  $75\%$  of the epochs. We use the decay of 0.1. We train for 100 epochs.
Small-CNN We use a batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0005, and Nesterov= True. The learning rate starts at 0.01 and is decayed twice during training: after  $50\%$  and  $75\%$  of the epochs. We use the decay of 0.1. We train for 100 epochs.
LeNet-5-Caffe We use a batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0005, and Nesterov = True. The learning rate starts at 0.01 and is decayed twice during training: after  $50\%$  and  $75\%$  of the epochs. We use the decay of 0.1. We train for 100 epochs.
ResNet-56 We use a batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0001, and Nesterov = True. The learning rate starts at 0.1 and is decayed twice during training: after  $50\%$  and  $75\%$  of the epochs. We use the decay of 0.1. We train for 200 epochs.
ResNet-50 We use a batch size of 256, the SGD optimizer with momentum 0.9, weight decay 0.0001, and Nesterov = True. The learning rate starts at 0.1 and is decayed twice during training: after  $50\%$  and  $75\%$  of the epochs. We use the decay of 0.1. We train for 90 epochs.
VGG-16 We use a batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0001, and Nesterov = True. The learning rate starts at 0.01 and is decayed twice during training: after  $50\%$  and  $75\%$  of the epochs. We use the decay of 0.1. We train for 200 epochs.
EfficientNet We use a batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0001, and Nesterov= True. The learning rate starts at 0.01 and is decayed twice during training: after  $50\%$  and  $75\%$  of the epochs. We use the decay of 0.001. We train for 100 epochs. We train on the 64x64 Tiny-ImageNet images (we do not re-scale them to 224x224 and we do not use pre-training in the main text experiments).
DST Settings specific for dynamic sparse training: We use the ER initialization for the distribution of sparsity levels in MLP, and the ERK initialization for the convolutional models. We start with an initial pruning fraction  $\rho$  of 0.5, which is decayed during training using a cosine decay schedule (see the main text). The default topology update period  $\Delta t$  is set to 800, but in the experiments for Q2 we vary this hyperparameter. Note that we also keep the update period  $\Delta t = 800$  for the ImageNet, since together with batch size 256 it matches the number of training examples seen by the model before the next topology update reported in [12]. For completeness, we also provide the DST hyperparameter setups used to obtain the Figures in the main text in Table 3.
Table 3: The DST hyperparameters used to obtain the results presented in the main-text experiments.
Experiment	Prune Fraction	Prune Fraction Decay Scheduler	Density	Update Period
Fig. 1	0.5	cosine	0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.15, 0.2, 0.3, 0.4, 0.5	800
Fig. 2	0.5	cosine	0.2	800
Fig. 4	0.5	cosine	0.2, 0.1	25, 50, 100, 200, 400, 800, 1600,3200,6400
Fig. 5	0.5	cosine	0.5	800
Fig. 6a	0.5	cosine	1.0	800 (measured only once)
Fig. 6b	0.5	cosine	0.2	800
Other Setup and Infrastructure Details We perform 5 runs for each setup, except for the ImageNet dataset, in which we use 3 runs. We do not fine-tune anything specific for any model. In all the experiments performed, at the beginning of the code, we fixed all the random seeds (random.seed, numpy.random, torch_manual_seed, torch.cuda_manual_seed), and set the cudnn backend to deterministic. The DST pruning criteria (if not used with random growth) are deterministic and for the same seed start with the same sparse initialization. While computing the Jaccard Index between the end masks we use only one worker for data-loading. In terms of reproducibility, we provide the environment setup but also perform our experiments in a Singularity container, for which we will provide the definition file in the repository. We provide the hyperparameter setups for all the main-text experiments in the code.[11]
We used GPUs either of the NVIDIA Tesla V100 or NVIDIA A100 type. We used external servers with job scheduling. All experiments (except for ImageNet) were performed on a single GPU. No other computations could access that GPU at such time. The ImageNet experiments were run using distributed computing on eight NVIDIA A100. Each task was using a batch size of 32 (so that the total batch size is 256) and the hyperparameter configuration described for the ResNet50 above.
Our code is accessible at https://github.com/alooo/fantasticweights_paper. It is based on the publicly available repositories for [34] and [11] available at https://github.com/VITA-Group/Random_Pruning and https://github.com/TimDettmers/sparse_learning, respectively. The used sparse-learning library (of [11]) is published under the MIT license. $^{12}$
11Under the specs directory in the code.
12We include the copy of that licence in the sparse-learning directory in our code.
Table 4: Datasets overview. The validation and training set on ImageNet follow the standard division. For CIFAR-10, we use 5000 of the training examples to form the validation test and fit the models on the remaining 45000. The test set for CIFAR-10 follows the standard division.
dataset	type	samples	features	classes	train-valid-test	source	URL
Higgs	tabular	940,160	24	2	70%-15%-15%	[2]	openml.org/d/44129
CIFAR10	image	60,000	3x32x32	10	75%-8%-17%13	[27]	cs.toronto.edu/~kriz/cifar.html
CIFAR100	image	60,000	3x32x32	100	75%-8%-17%14	[27]	cs.toronto.edu/~kriz/cifar.html
ImageNet	image	1,431,167	3x469x38715	1000	90%-3%-7%16	[49]	image-net.org/challenges/LSVRC/2012/
FashionMNIST	image	70,000	1x28x28	10	77%-9%-14%17	[63]	github.com/zalandoresearch/fashion-mnist
Tiny-ImageNet	image	100000	3x64x64	200	80%-10%-10%18	[29]	cs231n.stanford.edu/tiny-imagenet-200.zip
A.3 Tuning MEST