Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training
Authors: Aleksandra Nowak
Bram Grooten
Decebal Constantin Mocanu
Jacek Tabor
Venue: NeurIPS 2023 poster (2023)
Keywords: Dynamic Sparse Training
Pruning
Deep Learning
TL;DR: We study the impact of different pruning criteria on the performance of Dynamic Sparse Training.

Abstract: Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training.  It has been shown that under specific conditions, DST is able to outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process to adjust the network’s sparse connectivity. While the growing criterion's impact on DST performance is relatively well studied, the influence of the pruning criterion remains overlooked. To address this issue, we design and perform an extensive empirical analysis of various pruning criteria to better understand their impact on the dynamics of DST solutions. Surprisingly, we find that most of the studied methods yield similar results. The differences become more significant in the low-density regime, where the best performance is predominantly given by the simplest technique: magnitude-based pruning.

Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training
Aleksandra Nowak"
Jagiellonian University, Doctoral School of Exact and Natural Sciences;
IDEAS NCBR
al eelasandraireeaa.noaaaoooooooo. uj edu.
Bram Grooten
Decebal Constantin Mocanu
Eindhoven University of Technology
University of Luxembourg:
Eindhoven University of Technology;
University of Twente
Jacek Tabor
Jagiellonian University,
Faculty of Mathematics and Computer Science
Abstract
Dynamic Sparse Training (DST) is a rapidly evolving area of research that seeks to optimize the sparse initialization of a neural network by adapting its topology during training. has been shown that under specific conditions, DST is able outperform dense models. The key components of this framework are the pruning and growing criteria, which are repeatedly applied during the training process adjust the network's sparse connectivity. While the growing criterion's 's impact on DST performance is relatively well studied, the influence of the pruning criterion remains overlooked. To address this issue. we design and perform an extensive empirical analysis of various pruning criteria to better understand their impact on the dynamics of DST solutions. Surprisingly, we find that most of the studied methods yield similar results. The differences become more significant in the low-density regime, where the best performance is predominantly given by the simplest technique: magnitude- baased pruning.
Introduction
Modern deep learning solutions have demonstrated exceptional results in many different disciplines of science [5] [1, 26 However, they come at the cost of using an enormous number of parameters. Consequently, compression methods aim significantly reduce the model size without introducing any loss in performance [20] 6].
One approach sparsifying neural networks is pruning, in which portion of the weights is removed at the end of the training based on some predefined importance criterion [1 21 20) More
recently, [15] have found that iterative pruning joined with cautious parameter re-initialization can identify sparse subnetworks that are able to achieve similar performance to their dense counterparts when trained from scratch. This result, known as the lottery ticket hypothesis, has launched subsequent research into methods for identifying and training models that are sparse already at initialization [32
60
66
48
54]
Work started while visiting University of Twente.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

An especially promising direction in obtaining well- verl-oerfonnnng sparse neural networks is the Dynamic Sparse Training (DST) framework. Inspired by the neuroregeneration in the brain, DST allows for plasticity of the initial sparse connectivity of the network by iteratively pruning and re-growing a portion of the parameters in the model 42]. This relatively new concept has already gained increasing interest in the past few years. Most notably, in computer vision, DST demonstrated that it is sufficient to use only 20% of the original parameters of ResNet50 to train ImageNet without any drop in performance 12 38 Even more intriguingly, in Reinforcement Learning applications, DST is able to significantly outperform the classical dense models 18 58). At the same time, general sparse networks have been reported to surpass their dense counterparts in terms of adversarial robustness [47]. All those results demonstrate the incredible potential of DST not only in increasing the model's 's efficiency but also in providing better understanding of the features and limitations of neural network training.
Motivated by the above, we take closer look at the current DST techniques. Most of the research concerning design choices DST focuses on the analysis of methods where variations occur in the
growth criterion, while the pruning criterion becomes intertwined with other design considerations [3] о D. In this study, we address this issue by taking a complementary approach and focusing on the pruning criteria instead. A pruning criterion in DST serves as a measure of weight importance and hence is a proxy of the "usefulness" of particular connection. Note that the importance of a connection in DST can differ from standard post-train training pruning, as the role of weights can change throughout training due to the network's 's plasticity and adaptation. Weights deemed unimportant in one step can become influential in later phases of the training.
Our goal is to provide a better understanding of the relationship between pruning criteria and the dynamics of DST solutions. To this end, we perform a large empirical study including several popular pruning criteria and analyze their impact on the DST framework on diverse models. We find that:
Surprisingly, within a stable DST hyperparameter setup, the majority of the studied criteria perform similarly, regardless of the model architecture and the selected growth criterion.
The difference in performance becomes more significant in very sparse regime, with the
simplest magnitude-basee pruning methods surpassing any more elaborate choices.
Applying only a few connectivity updates is already enough to achieve good results. At the
same time, the reported outcomes surpass those obtained by static sparse training.
. By analyzing the structural similarity of the pruned sets by each criterion, we assert that the best-performing methods make similar decision choices.
The insights from our research identify that the simplest magnitude pruning is still the optimal choice, despite a large amount of alternatives present in the literature. This drives the community's attention to carefully examine any new adaptation criteria for Dynamic Sparse Training.
2 Related Work
Pruning and Sparse Training, A common way of reducing the neural network size is pruning,
which removes parameters or entire blocks of layers from the network. In its classical form, pruning
has been extensively studied in the context of compressing post-training models [25] 44 31 21
20 46 33
4 17 see e.g. 23 for an overview and survey. Interestingly. 40 demonstrated
for the first time in the literature that a sparse neural network can match and even outperform its corresponding dense neural network equivalent if its sparse connectivity is designed in sensitive manner. Recently, similar result was obtained by the lottery ticker hypothesis and follow-up research, which showed that there exist sparse subnetworks that can be trained in isolation to the same performance as the dense networks [15 66. 62]. In an effort to find these subnetworks without the need for dense training, different techniques have been proposed over the years [32 60) 59],
including random selection [48] 34]. Such approaches are commonly referred to as Sparse Training, or Static Sparse Training to emphasize that the sparse structure stays the same throughout the training. Interestingly, [16] find out that the sparse initializations produced by the static approaches are invariant to parameter reshuffling and reinitialization within a layer and that even pre-training magnitude pruning performs quite well in such setup.
"The code is provided at /ttttt////ttthub... http://ww......laooo////aaatttta

---

Table 1: An overview of the existing pruning criteria in DST. Our work analyzes the differences and similarities between all the methods and fills the gaps (x) in the literature. Each column lists a pruning criterion, while each row presents growing method. SNIP [32 was not designed to grow weights, but we investigate whether it can be applied in DST.
Magginnde [
CSET |0 10:1.10
CMEST
Senvttvvit VeC(P1 [+]
CSNIP [
| + X(V)C(D)
|0
random growth
X
SET 42
MEST 64
Sensitivity
X
gradient growth R2 [2)
x
X
X
Dynamic Sparse Training, In DST, the neural network structure is constantly evolving by pruning and growing back weights during training [42 11 [2
1 9 28 The key motivation behind DST is not only to provide compression but also increase the effectiveness and robustness of the deep learning models without the need for overparamerrization (88 The capability of DST has recently been an area of interest. In [13), authors indicate that DST is able to outperform static sparse training by assuring better gradient flow. DST has also been reported to achieve high performance in Reinforcement Learning [8
64
53 19 and Continual Learning [52], with ongoing research into applicability in NLP 35 Some attention has also been given to the topological properties of the connectivity patterns produced by DST. In particular, (36), investigated the structural features of DST. However, they focus only on one method, the Sparse Evolutionary Training (SET) procedure 42)
To the best of our knowledge, this work is the first comparatively study large number of pruning criteria in DST on multiple types of models and datasets. We hope that our analysis will increase the understanding within the dynamic sparse training community.
3 Background
3.1 Dynamic Sparse Training
Dynamic sparse training is framework that allows training neural networks that are sparse already at initialization. The fundamental idea behind DST is that the sparse connectivity is not fixed. Instead, it is repeatedly updated throughout training. More precisely, let 0 denote all the parameters of a network fa. which is trained to minimize loss L(;; D) on some dataset D. The density d' of layer I with latent dimension n' is defined as d' = |00' where || llo is the L0-norm, which counts the number of non-zero entries. Consecutively, the overall density D of the model is D = Σl_ d'n''
ST
with I. being the number of layers the network. The sparsity given by = 1 D. Before the start of training, fraction of the model's parameters is initialized to be zero in order to match predefined density D. One the most common choices for initialization schemes is the Frdös-Rénye (ER) method 42]. and its convolutional variant, the ER- TR-Kenlel (ERK) [[2]. It randomly generates the sparse masks so that the density in each layer d' scales as 4n' for a fully-connected layer and ,
nf 4pp' for convolution with kernel of width wt and height ht' The sparse connectivity is nt An updated every At training iterations. This is done by removing a fraction Pt of the active weights accordingly to a pruning criterion C. The selected weights become inactive, which means that they do not participate in the model's 's computations. Next, a subset of weights to regrow is chosen accordingly to growth criterion from the set of inactive weights, such that the overall density of the network maintained. The pruning fraction Pt is often decayed by cosine annealing P: 1 2 + cos (tm/ (T),, where T is the iteration which to stop the updates, and p is the initial pruning fraction.
The used pruning and growth criterion depends on the DST algorithm. The two most common growth modes in the literature are random growth and gradient growth. In the first one, the new connections are sampled from a given distribution 42 45 64]. The second category selects weights based on the
largest gradient magnitude [12.. Other choices based on momentum can also benefit the learning
3.2 Pruning Criteria
Given a pruning fraction Pt, the pruning criterion determines the importance score s(Ol,) for each weight and prunes the ones with the lowest score. We use local pruning, which means that the
3

---

criterion is applied to each layer separately For brevity, we will slightly abuse the notation and write I instead of Bi, from now on. Below, we discuss the most commonly used pruning criteria in DST that are the subject of the conducted study.
SET. To the best of our knowledge, SET is the first pruning criterion used within the DST framework, which was introduced in the pioneering work of [42). The SET criterion prunes an equal amount of positive and negative weights with the smallest absolute value.
Magnitude. The importance score in this criterion is given by the absolute value of the weight s(0) (0). Contrary to SET, no division between the positive and negative weights is introduced. Due to its simplicity and effectiveness, the magnitude criterion has been a common choice in standard post-training pruning, well in sparse training [ [2].
MEST.
The standard magnitude criterion has been criticized as not taking into consideration the fluctuations of the weights during the training. The MEST (Memory-Fconome
Sparse Training) criterion [64]. proposed to use the gradient as an indicator of the trend of the weight's magnitude, leading to a score function defined as s(0) |0| + ()V))(D)), where À is a hyperparameter.
Sensitivity. The role of gradient information in devising the pruning criterion has also been studied by [45) Taking inspiration from control systems, the authors propose to investigate the relative gradient magnitude in comparison the absolute value of the weight, yielding s(t) = /oC(D)) /|0).
In our study, we consider the reciprocal version of that relationship s(0) = | ( which we call RSensitivity, as we found it to more stable
SNIP. The SNIP (Single-shot Network Pruning) criterion is based on a first-order Taylor approx- imation of the difference in the loss before and after the weight removal. The score is given by s(0) 0 (DL(D)) The criterion has been originally successfully used in static sparse train- ing 32 and post-training pruning [43) Motivated by those results, we are interested in investigating its performance in the dynamic sparse training scenario.
We denote the above- mentioned criteria by adding subscript under the sign C in order distinguish them from the algorithms that introduced them. We summarize their usage with random and gradient growth criteria in the DST literature in Table
4 Methodology
This work aims to understand the key differences and similarities between existing pruning techniques in DST by answering the following research questions:
Q1 What is the impact of various pruning criteria on the performance of dynamic sparse training?
Q2 How does the frequency of topology updates influence the effectiveness of different pruning methods?
Q3 To what extent do different pruning criteria result in similar neural network topologies?
Q1: Impact on the Performance of DST. In the first research question, we are interested in investigating the relationship between the pruning criterion and the final performance of the network. We compare the results obtained by different pruning criteria on a diverse set of architectures and datasets, ranging from regime with small number of parameters (< 100K) to large networks (~ 13M). We examine multi-layer perceptrons (MLPs) and convolutional neural networks (ets) Within each model, we fix the training setup and vary only the pruning criterion used by the DST framework. This allows us to assess the benefits of changing the weight importance criterion in isolation from other design choices. As the growing criterion, we choose the uniform random +p and gradient 12] approaches, as they are widely used in the literature. In addition, we also perform a separate comparison on the ResNet-50 model (25M parameters) on ImageNet.
The common sense expectation would be that the more elaborate pruning criteria, which combine the weight magnitude with the gradient information, will perform better than the techniques that rely only on the weight's absolute value. Since the gradient may provide information on how the weight will change in the future, it indicates the trend the weight might follow in the next optimization updates.
We adapt this setup as it is commonly used in DST approaches, and has a reduced risk of entirely discon- necting the network- see Appendix [I for a more detailed discussion. "See Appendix
B

---

Calest-Calie CUNISI
MLP-HIGGS
1oNWeE6-C1FAR10
MLP-CIFARIO
Efficien Net-Thy
VOCIG-IPAARO
rardom rpreeth
tcs acearuy (%)
73
65
70
92
65
93
64
88
70
85
020
10
62
grodien growth lsst uccrzaa (56)
65
61
30
80
66
20
88
A06
22
3.00
0.65
3.00
10
0.90
Aemsity
donsity
dorsity
densiity
dossity
Statin
Mogattode
Canr
Caxip
Figure The test accuracy versus density computed on the studied models for different pruning criteria (due to space limits only present here six plots, Figure I0 in Appendix D for the remaining two setups). The first row represents the results obtained by random growth, while the second corresponds to gradient growth. Note the logarithmic scale in the x-axis. The performance of the dense model is indicated by the horizontal dashed line. almost every case all pruning criteria perform well, regardless of the chosen model and growth criterion. At the same time, they surpass the static initialization (in light blue).
Therefore it may be more suitable in the DST approach, where connectivity is constantly evolving, making insights into future updates potentially beneficial. This is especially promising considering the effectiveness of the gradient- approach in estimating the importance in the growth criterion.
Q2: Update Period Sensitivity. In the second research question, we focus on one of the most important design choices: the topology update period A.. The setting of this update period determines how often the network structure is adapted during training. When considered together with the dataset- and batch-size, this hyperparameter can be interpreted as the number of data samples the model trains on before performing the mask update. Therefore, within a fixed dataset and batch size, using relatively low value means that the topology is changed very frequently in comparison to the optimization update of the weights. This poses a potential risk of not letting the newly added connections reach their full potential. On the other hand, high value gives the weights enough opportunity to become important but may not allow the sparse network to adjust its structure based on the data characteristics. Finding a balance between this exploitation eupporation-expllraatonn conflict is an imperative task in any DST framework.
Q3: Structural Similarity. Finally, we take a deep dive into the structures produced by the various pruning criteria. Please note that while the previous questions, QI and Q2. may explain how the pruning criterion choice affects performance and update frequency DST, they do not indicate whether the mask solutions obtained by those criteria are significantly different. In order to assess the diversity of the studied methods in terms of the produced sparse connectivity structures, we need to compare the sets of weights selected for pruning by each criterion under the same network state. In addition, we also investigate the similarity between the final masks obtained at the end of training and how close they are to their corresponding sparse initializations. This allows to assess whether the DST topology updates are indeed meaningful changing the mask structure. For both cases, we incorporate a common score of the proximity of sets, known the Jaccard index (or Cetrsectioo-Over-nniie We compute it separately for each layer and average the result:
17 0 1 U
J(I., Ib)
L Σ J(L 1)) J(TZ. 15)
(1)
where F and 7/ are the sets selected by pruning criteria and b in layer A Jaccard index of indicates that sets overlap perfectly, while 0 implies they are entirely separate.
5

---

We expect to see some overlap in the pruning methods' selection of weights since all of them incorporate the use of the magnitude of the weight. We also hope to see if the difference between the connectivities produced by scores that give different performances is indeed large or whether a small adjustment of the weights would render them equal.
5
Experiments
In this section, we present the results of our empirical studies. We start with the description of the experimental setup and then answer each of the posed questions in consecutive sections.
5.1 Setup of the Experiments
We perform our analysis using eight different models, including small- and large-scale MLPs and Convolutional Nets. The small-MLP is a 4-layer network with hidden size of at most 256, (trained on the tabular Higgs dataset (2). The large-MIPP also consists of 4 layers (latent dimension size up to 1024 neurons) and is evaluated on the CIFAR 10 (2). The convolutional architectures are: small 3-layer CNN 10),, LeNet5-Caffi [3) (FashionMNIST (33), ResNet56 (CIFARIII CIFAR100), VGG-16 16 (CIFARI0O)) and EfficientNet [57 (Tiny-Imaage
[29].. In addition, a selected density, we also consider the ResNe150 model on ImageNet 499, We summarize all the architectures in Appendix A While our primary focus is vision and tabular data, also evaluate the pruning criteria on the fine-tuning task of ROBERTa Large [39 using the CommonsenseQA [56 adapted from the Sparsity May Cry (SMC) Benchmark [35]. In contrast to the setup studied throughout this work, this experiment involves fine-tuning type of problem. In consequence, we discuss it in Appendix [ focus on training models from scratch for the remainder of the paper.
We train the models using the DST frame- work set of predefined model densi- ties. We use adjustment step At = 800 and initial pruning fraction p 0.5 with
78.0
validartie aceuracy (9%)
76.0
cosine decay. As the growth criterion, we investigate both the random and gradient- based approaches, as those are the most common choices in the literature. Follow-
74.36
71
Te.31
74.0
Traitinn
72.0
71.31
71,11
ing [64). we reuse the gradient computed during the backward pass when calculat- prun-
68.0
ing the scores in the gradient-basse ing criteria. In all cases, we ensure that within each model, the training hyperpa- rameters setup for every criterion is the same. Each performance is computed by averaging runs, except for ImageNet, for which we use 3 runs. The ResNet50 model is trained using density 0.2 and gra- dient growth (we select the gradient-based growth method since is known to provide good performance in this task (I2)). Altogether we performed approximately 7000 runs of dense, static sparse, and DST models, which in total use (9 including ImageNet) different dataset-model configurations, and jointly took approximately 5275 hours on GPUSS
Dense
Coar
Cansr
Cisseeritrity
Static
Care
Figure 2: Left: Validation accuracy of the different prun- ing criteria ImageNet obtained for density 0.2. The dashed black line indicates the best result of the dense model. Right: The training loss versus the number of epochs. We see that all methods perform similarly, ex- cept for CSNIP, which suffers already the beginning of the training.
5.2 Impact on the Performance of DST
In this section, analyze the impact of the different pruning criteria the test accuracy achieved by the DST framework. We compare the results with the original dense model and static sparse training with ERK initialization. The results are presented in Figure []
In this work, we are interested in how the frequency of the topology update impacts different pruning criteria. Intuitively, one could anticipate that the pruning criteria incorporating the gradient
For training details please refer Appendix A.2

---

CD
CD
2
CRSP
Caterinn
Cittoaatety Cissectiitty
CaxI
CET
Cusst
Coar
Cass
CAT
Cansi
(a) all densities
(b) density S 0.2
(c) density > 0.2
The critical distance diagram of the studied methods for (a) all densities, (b) densities smaller equal 0.2, and (c) densities larger than 0.2. For each model compute its average rank (the lower the better) and plot it on the horizontal line. The methods with ranks not larger than the critical distance (denoted as CD, displayed in the left upper corner of each plot) considered not statistically different and are joined by thick horizontal black lines.
Figure 3:
information will perform better with a smaller update period, as the gradient may indicate the trend the weight's future value.
Firstly, we observe that the differences between the pruning criteria are usually most distinctive in low densities. Furthermore, in that setting, the dynamic sparse training framework almost always outperforms the static sparse training approach. This is important, as in DST research, high sparsity regime is of key interest. Surprisingly, the more elaborate criteria using gradient- based approaches usually either perform worse than the simple magnitude score, do not hold clear advantage over For instance, the Taylor-based CSNIP criterion, despite being still better than the static initialization, consistently achieves results similar or worse than Maggitude This is especially visible in the DST experiments using gradient growth (note ResNet56, MLP-CIFAR 10, and EfficientNet). The only exception is the VGG-16 network. In addition, we also observe that CSNIP gives better performance when used in a fine-tuning setup for EfficientNet see Appendix approximation- based criterion, although being well suited for standard post-training pruning [43] and selecting sparse initialization masks [32] seems not to be the optimal choice in dynamic sparse training approaches high sparsity. The CRSensitivity the other hand, can outperform other methods but is better suited for high densities. Indeed, is even the best choice for densities larger than 0.15 for large-MLP CIFARIO 10 and ResNet models. Finally, the CAEST criterion also is not superior to the Maggitude usually leading to similar results [ol We also present the results obtained for ResNet-50 on ImageNet with density 0.2 in Figure 2 We observe that, again, the CSNIP criterion leads the worst performance. The CRSensitivity- CSET. and CMEST methods achieve high results but not clearly better than the Caagnitue criterion.
Consequently, this Taylor-
Within each studied density, growing crite- rion, and model, rank the pruning crite- ria and then calculate their average ranks. To rigorously establish the statistical sig- nificance of our findings, we compute the Critical Distance Diagram for those ranks, using the Nemenyi post-hoc test [10 with p-value 0.05 and present it in Figure We observe that in the low-density regime (Fig- ure 3b). the CMagnitdo criterion achieves the best performance, as given by the low- est average rank. Furthermore, in such case, we also note that other predominately magnitude based criteria, such as CSET and CMEST are not significantly different than CMagmiudee while the remaining gradient- based approaches are clearly worse. This confirms the overall effectiveness of the
Lbet accurary (X6)
07
lee grg
65
65
Statk: Gonar
Cusir
CAT
Denee- Despors
Cascunner
The loss gap (validation loss training loss) over time for different pruning criteria, the static sparse model, the dense model, and the dense model with dropout p 0.05 the large-MLP. Right: The test accuracy obtained for each criterion.
Figure Left:
"We study the impact of the hyperparameter of the CAEST criterion in Appendix A.3 In general, we see that A + leads to results equal to Cstagannde. while À + oo prioritizes only the gradient magnitude and degrades the performance. We do not any clear improvement for values lying between those extremes.

---

CONSCIPME
CuIST
Castr
Stutio
Cauer
Statio
Car
Curr
Cas abiik
(a) density 0.2
(b) density 0.1
Figure 5: The update period At versus validation accuracy on the CIFARIO dataset on the MLP, ConvNet, and ResNet-56 models for different pruning criteria with density 0.2 and 0.1. The top row corresponds to random growth, while the bottom row corresponds to gradient growth. Note the logarithmic scale on the x-axis. We see that the methods are most sensitive to the update frequency in the MLP setting. For all setups, performing the update later (At 400) seems be beneficial.
weight's magnitude as importance score in sparse regimes. See also Appendix G for the discussion of the effect of batch size on the obtained results. Interestingly, when larger densities are involved, the best choice is given by CRSeasitivity- Additionally, the average rank of Cmagnitude deteriorates. This may suggest that the information from the gradient is more reliable in such a case. The gradient- based methods seem to perform better in denser setups.
Finally, we notice that in certain situations, the DST framework can lead to better results than the original dense model (see small-MLP, ResNet56 on CIFARI00 in Figure []. We argue that this due to a mix of a regularization effect and increased expressiveness introduced by removing and adding new connections in the network- see Figure Moreover, we also observe that within the setup studied in Figure [ there is almost no difference between choosing the random or gradient growth criterion. The second one achieves slightly better accuracy for the convolutional models (see, e.g., density 0.2 on ResNet-56, as well as improved stability for EfficientNet). The similarity between those two growth criteria and the surprising effecttivneess of DST in comparison to dense models have also been previously observed in the context of Reinforcement Learning [8]
5.3 Sensitivity to Update Period
In this section, we analyze the sensitivity of each pruning criterion to the update period At. This value describes how often the adjustments are made in training and hence can affect the weights' importance scores. We fix the density to 0.2 and 0.1 and the initial pruning fraction to 0.5. We choose the densities 0.2 and 0.1 as they correspond rather high sparsity levels and generally provide reasonable performance. Next, we investigate different update periods on the models using CIFARIO dataset We start from At 25 and increase it by a factor of up to At 6400.
The results are presented in Figure 5 For the MLP model, we clearly see the behavior described in Section 4 if the update period At is too small, the performance will suffer. The best update period setting seems to be At = 800 for the random growth, regardless of the pruning method. For gradient growth, this value also gives a good performance. However, larger pruning periods (e.g. At 1600) are even better R Note that we use a batch size of 128, hence for At = 800 the connectivity is changed approximately once per 2 epochs | Similarly, less frequent updates are also beneficial in the ResNet56 model, although there is not as much consensus between the different pruning criteria. Even performing just one topology update every Al- 6400 iterations (approximately once per 16 epochs) still gives excellent performance. At the same time, not performing any topology updates at
'The update period is only meaningful when considered together with the size of the training set. Therefore it is important evaluate models of different sizes the same datasct.
$ $See also Appendix 7 for an analysis of the effect batch size and pruning schedule on these results.
Since 800- 128 102, 400 samples, which is just over twice the training size of CIFARIO (50, 000).
8

---

Cole
Coln
Cest
Gaper
Cour
Quacia
.20
20
0.550
CNN 4, 0.312
BosNes92 A
0.200
(a)
(b)
Figure 6: (a) The mean normalized Jaccard index between the sets of weights chosen for removal during the first update of the sparse connectivity, computed for the MI.,, CNN, and ResNet-56 models CIFAR1O. The J, indicates the expected overlap of random subsets and serves reference point. We estimate this value by computing the mean pair-wise Jaccard index of the random pruning criterion with different sparse initialization. (b) The same index computed between the masks obtained by different pruning criteria at the end of training. The rows and columns represent the pruning criteria between which the index is computed. The Jinit denotes the expected overlap of random masks and serves as a reference point. We estimate this value by computing the mean pair-wise Jaccard index of the different sparse initializations.
all (i.e., static sparse training) deteriorates performance, as shown by the cyan dashed line in Figure 5 Furthermore, we observe that the gradient-based pruning criteria seem to suffer much more than the magnitude criteria when paired with a small update period (consider the plots for A = 25), especially when combined with the gradient growth [ This suggests that it is safer not to use those techniques together or be more rigorous about finding the right update period in such a case.
Most interestingly. the results from Figure 5 not needed. Even few pruning and re-growing iterations are enough to improve the static initialization.
strongly suggest that frequent connectivity adjustment is
5.4 Structural Similarity
Finally, we investigate whether the studied criteria are diverse in terms of the selected weights for removal. We fix the dataset to CIFAR 10 and analyze the moment in which the sparse connectivity is updated for the first time. At such a point, given the same sparse initialization and random seed, all the pruning criteria share the same network state. Consequently, between the sets selected for removal by each criterion. We do this by computing their Jaccard index (Equation]] ID and present the averaged results of 5 different runs in Figure 6a (See also Appendix K standard deviations and more plots).
analyze the difference
We observe that the CSET and CMagninde mainly select the same weights for pruning. This indicates that the absolute values of the positive and negative weights are similar. In addition, for the simpler models with smaller depths (MLP and CNN) the CMEST criterion also leads to almost the same masks as Magnitudee On the other hand, the CSNIP and CRSensitivity criteria produce sets that are distinct from each other. This is natural, CSNIP multiplies the score by the magnitude of the gradient, while CRSensitivity uses its inverse. The experiment suggests that early in training, pruning criteria such as CMagnitude- CSET, and CAEST lead to similar sparse connectivity. Together with the similarity in performance observed in Section 5.2, the results indicate that these three methods are almost identical, despite often being presented as diverse in DST literature. At the same time, the updates made by CSNIp and CRSensiivity produce different pruning sets, and result in lower performance (recall Figure (3)). However, note that for each entry (except CSNIP with CRSensitivity). ). the computed overlap is larger than random, suggesting that there is a subset of weights considered important by all the criteria.
In addition, also compare how similar are the final sparse solutions found by the methods. To this end, we fix the sparse initialization to be the same for each pair of criteria and compute the Jaccard Index of the masks at the end of the DST training (performed with gradient growth and density 0.2 on the CIFARIO experiments from Section 5.2). For each pair, we average the score over 5 initializations. The resulting similarity matrix is presented in Figure 6b) We observe that pruning criteria that selected similar sets of weights for removal from the previous experiment still hold some resemblance each other in terms of the end masks. Let us also note that in Section 5.3, observed
II Appendix E we also study the exploration imposed by the pruning criteria using the ITOP ratio [8

---

that applying just a few connectivity updates is sufficient for good results. This raises the natural question of whether DST masks are genuinely diverse from their sparse initializations. By analyzing the bottom row in Figure [6b] we see that the masks obtained at the end of the DST training are distinct from their corresponding initializations, having the Jaccard Index close to that of a random overlap.
In consequence, we conclude that the best-performing methods from Section 5.2 indeed make similar decision choices while having smaller overlap with the less efficient criteria. This renders the magnitude-basee criteria almost equivalent, despite their separate introduction in the literature. At the same time, the DST mask updates made during the training are necessary to adapt and outperform the static initialization. We would like to highlight that such insights could not have been derived solely from performance and hyperparameter results. We hope that our insights will raise awareness of the importance of performing structural similarity experiments in the DST community.
6 Conclusion
We design and perform a large study of the different pruning criteria in dynamic sparse training. We unveil and discuss the complex relations between these criteria and the typical DST hyperparameters settings for various models. Our results suggest that, overall, the differences between the multiple criteria are minor. For very low densities, which are the core of DST interest, criteria based on magnitude perform the best. This questions the effectiveness of gradient-based scores for pruning during the training. We propose incorporate the selected models and datasets as baseline in future works investigating the pruning criteria for DST to avoid the common case where methods are overfitted to given DST hyperparameters or tasks. We hope that our research will contribute to the understanding of the sparse training methods.
Limitations & Future Work. and one tabular dataset. It would be interesting to verify how our findings translate to large language models. The computed structural insights are based on set similarity and disregard the information about the value of the parameters. Additional topographic insights could be provided to incorporate this information and analyze the graph structure of the found connectivity. Finally, much work still needs to be done on the hardware side to truly speed up training times and decrease memory requirements (see Appendix We do not see any direct social or ethical broader impact of our work.
Our research was conducted mainly on datasets from computer vision
Acknowledgements
The research of Aleksandra Nowak and Jacek Tabor has been supported by the flagship project entitled Artificial Intelligence Computing Center Core Facility from the DigiWorld Priority Research Area under the Strategic Programme Excellence Initiative at Jagiellonian University. The work of J.
Tabor was supported by the National Centre of Science (Poland) Grant No. 2021/41/BSST/01337 The research of Bram Grooten was funded by the project AMADeuS (with project number 18489) of the Open Technology Programme, which is partly financed by the Dutch Research Council (NWO). We gratefully acknowledge Polish perforeeform computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no.
PLG/2022/015887.
References
[1] Zahra Atashgahi, Joost Pieterse, Shiwei Liu, Decebal Constantin Mocanu, Raymond Veldhuis, and Mykola Pechenizkiy. A Brain-inspired Algorithm for Training Highly Sparse Neural Networks. Machine Learning, pages 1-42, 2022. URL: http::/tts:///t http///axvvv... saxxvv.rg//ae httg//bs/19993
07138
[2] Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for Exotic Particles in High- Energy Physics with Deep Learning. Nature communications, 5(1):1-9, 2014. URL: https
/ //arxiv. www./ogg///bbbbeeeeo 1402/1402.004 4735
[3] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep Rewiring: Training very sparse deep networks. International Conference on Learning Representations, 2018. URL: https: //tpx/ivvvv///////////////aaatv/ wwwgg/////g......./t.t... 05136
10

---

[4] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the State of Neural Network Pruning? Proceedings of Machine Learning and Systems, 2:129-146, 2020. URL:: http:////wwwr.....aaaa//////
200.//3333300333333 03033
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models Few-Shot Learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
URL: htt.ttt/ta//aaiiii org/ aba/ 2005 14165
[6] Yu Cheng, Duo Wang. Pan Zhou, and Tao Zhang. A Survey of Model Compression and Acceleration for Deep Neural Networks. IEEE Signal Processing Magazine, 2017. URL:
https://arxiv. org/ abs 09282
[7] François Chollet. Deep Learning with Python. Manning Publications, 2nd edition, 2021. URL: https://fchollet.. com com/#books
[8] Selima Curci, Decebal Constantin Mocanu, and Mykola Pechenizkiyi. Truly Sparse Neural Networks at Scale. arXiv preprint arXiv:2102.01732, 2021. URL: https::::::/ paattps://arxiy http///xxvvvorrree// org/ abs/
2102.01737
[9] Xiaoliang Dai, Hongxu Yin, and Niraj K Jha. NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm. IEEE Transactions on Computers, 68(10):1487-149) 2019. URL: pettps://arxivv
abs 1011171110000
[10] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(1):1-30, 2006.
[11] Tim Dettmers and Luke Zettlemoyer. Sparse Networks from Scratch: Faster Training without Losing Performance. arXiv preprint arXiv:1907.044: 1907.04840, 2019. URL: https://arxii http//oxxxvoo////////// abs/1907 .08800
[12] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the Lottery: Making All Tickets Winners. In International Conference on Machine Learning, pages 2943-2952. PMLR, 2020. URL: htttps:/aaax...or//aaaaaa/ htt//bbs/1911.. 11134
[13] Utku Evci, Yani Ioannou, Cem Keskin, and Yann Dauphin. Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 6577-6586, 2022. URL: https:/ sttps://arxiv. pexiv.org// aa///bb//11000 2010
03533
[14] Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszez, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-33, 2022. URL: htttps://wwwt nature.comm. wwt.:/tcc/aa/caaccee..ee.o....ca cles/
111586-022-05172-11
[15] Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, International Conference on Learning Representations, 2019. URL: https
//arxiv. wwwg/b///////.../////////// 1803. 03635
[16] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neu- ral networks at initialization: Why are missing the mark? arXiv preprint (XXxv:200.008770 2020.
[17] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity deep neural networks. arXiv preprint
2019,
[18] Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The State of Sparse Training in Deep Reinforcement Learning. In International Conference Machine Learning, pages 7766-7792. PMLR, 2022. URL: htttss://axxiiiii org/ abs/ /20.//2266.222 10369
[19] Bram Grooten, Ghada Sokar, Shibhansh Dohare, Elena Mocanu, Matthew Taylor, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Automatic noise filtering with dynamic sparse training deep reinforcement learning. Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 1932- 1941, 2023.
11

---

[20] Song Han, Jeff Pool, John Tran, and William Dally. Learning both Weights and Connections for Efficient Neural Networks. Advances in Neural Information Processing Systems, 28, 2015.
URL:: httpt:ttt/ssat /ttp:v////a/v.ee/a wwwpw/wwww/w///////////// http://ba/115..0060 02626
[21] Babak Hassibi, David G Stork, and Gregory Wolff. Optimal Brain Surgoon and General Network Pruming. In IEEE international conference on neural networks, pages 293-299. IEEE,
1993. URL: httt:tttts:://atthaee http://auuttos..b.a..r.a.aa www.echlhhhhc.ce wwwpeeehhhhh.eeeeeeeeeeeeeeeeeee edu/54981/
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. URL: hhttss://aaxtiii http///r..iiorrg/..////oo abs/ 1512. 03385
[23] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity Deep Learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-122 2021. URL: httpp:://aaaaiai org/
abs/2102.0055
[24] Sara Hooker. The Hardware Lottery. Communications of the ACM, 64(12):58-65) 2021. URL: https://dl. acm. org//oi/10..ooo 10011114445555 3467017
[25] Steven Janowsky, Pruning versus clipping neural networks. Physical Review A, (96)))66000 1989. URL: htttttt///o//rraalsssss aps. org/ aarrabbbstrcct/1... 1w11111//Pyseee// 3966000
[26] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron-
neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583-588,, 2021.
URL: htt:::tttt::://buie ncbi nlm. nih, gov gov/34265844/
[27] Alex Krizhevsky, Geoffrey Hinton, Hintonn et al. Learning Multiple Layers of Features from Tiny Images.
2009. URL: https://www.c c8. toronto. edu httaa//kriizlearning--ttt -eatt-ef-ttteeeeeeee -00---009--7 pdf
[28] Abhisek Kundu, Naveen K Mellempudi, Dharma Teja Vooturi, Bharat Kaul, and Pradeep Dubey. Autosparse: Towards automated sparse training deep neural networks. arXiv preprint 2rX1v:2304.06941, 2023.
[29] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 7(7):3, 2015.
[30] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):220 (1:1128228-2244 1998.
[31] Yann LeCun, John Denker, and Sara Solla. Optimal Brain Damage. Advances in Neural Information Processing Systems, 2, 1989. URL: htttss://pppe::re nips . cc/ wwtpappr////eceecccccccccce.eeet 1989/ hash/ haaaa//68882bbe..177992222221111
Abstract. html
[32] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Tor. SNIP: Single-shot Network Pruning
based on Connection Sensitivity. International Conference on Learning Representations, 2019. URL: httpp:://aaxiv.orr//////
02340
[33] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning Filters for Efficient ConvNets. International Conference on Learning Representations, 2017. URL:
http://aai....rr////////0/// 08710
[34] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The Unreasonable Effectiveness of Random Pruning: Re- turn of the Most Naive Baseline for Sparse Training. International Conference on Learning Representations, 2022. URL: https://arxix: orggabgg./...oooo..e /202, 02643
[35] Shiwei Liu, Tianlong Chen, Zhenyu Zhang, Xuxi Chen, Tianjin Huang, Ajay Jaiswal, and Zhangyang Wang. Sparsity may cry: Let us fail (current) sparse neural networks together! ICLR 2023, 2023.
12

---

[36] Shiwei Liu, Tim Van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada Sokar, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Topological Insights into Sparse Neural Networks. In Joint European conference machine learning and knowledge discovery in databases, pages 279-294. Springer, 2020. URL:: wtw.ttttss///xiiiie http///vvvvor/////////./ abs/ 2006 14085
[37] Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learning with over million artificial neurons on commodity hardware. Neural Computing and Applications, 33(7):2589-260 2604,, 2021.
URL: httph::ttaaaaae wwweeiii...rgeee absa http:/9911..9911
[38] Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do We Actually Need Dense OverrParrameterization" In-Time Pver-Parameterizzaton in Sparse Training. In International Conference Machine Learning, pages 6989-7000. PMLR, 2021. URL: https:
//arxiv. //axxiv.rggaall /bg//b////t 2102. 02887
[39] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
[40] Decebal Constantin Mocanu, Elena Mocanu, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. A topological insight into restricted boltzmann machines. Machine Learn- ing, 104())243-270, 2016. 110994-016-55000
URL: https://link: http://iiinnk.sinngerr... wwt.coooaait.eee http///rrcce/11110....//
[41] Decebal Constantin Mocanu, Elena Mocanu, Tiago Pinto, Selima Curci, Phuong H Nguyen, Madeleine Gibescu, Damien Ersst, and Zita A Vale. Sparse Training Theory for Scalable and Efficient Agents. International Conference on Autonomous Agents and Multiagent Systems URL: https://arxix.. http:///axxxvv.oaaabboe /ttp////iii..rr/////////...000 httpa/bb//////1111
(AAMAS), 2021.
[42] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science. Nature communications, 9(1):1-12, 2018.
URL: https:///aaxxii org/
04780
[43] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance Estimation for Neural Network Pruning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. URL: https://arxiv. org/ wwwporggggbbb..oeeeeeeeeeeegge 1911 10771
[44] Michael C. Mozer and Paul Smolensky. Using relevance reduce network size auto- matically. Connection Science, 1(1):3-16, 1989. URL:
09540098908915622)
[45] Seyed Naji, Azra Abtahi, and Farokh Marvasti. Efficient Sparse Artificial Neural Networks. arXiv preprint arXiv:2103.07674, 03 2021. URL:https://arxiv http:/awwwv.....raaa..a. Baargg/ab////// 0tt///113.0000
[46] Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring Sparsity in Recurrent Neural Networks. International Conference on Learning Representations, 2017.
URL: helL:httss://aart
arg/abs//7000 05119
[47] Ozan Özdenizci and Robert Legenstein. Training Adversarially Robust Sparse Networks via Bayesian Connectivity Sampling.
International Conference on Machine Learn- ing, pages 8314-8324. PMLR, 2021. URI.: https::/prrocedings..o mml.ainnnmm.......mmmmmmmmmmmmmm holr.press/v13//
ozdenizci21a html
[48] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's Hidden in Randomly Weighted Neural Network? In Conference on Computer Vision and Pattern Recognition, pages 11893-11902, 2020. URL: httpps://axxii
org/abs/1911. 13999
[49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211122)) 2015. URL: https: /ttp///avvvvv... org/ abs/ 1109.49905555
13

---

[50] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large- scale image recognition. arXiv preprint arXiv:1409.156,, 2014.
[51] Samuel L Smith and Quoc Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprini arXiv:1710.064511 2017
[52] Ghada Sokar, Decebal Constantin Mocanu, and Mykola Pechenizkiy. SpaceNet: Make Free Space For Continual Learning. Neurocomputing, 439:1-11, 2021. URL: https://arxiv org/ aa/abbbs20007/ 07661
[53] Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. Dynamic Sparse Training for Deep Reinforcement Learning. arXiv preprint arXiv:2106.04211
2021. URL: htt.:htttt::/araaee http:/::/aaxvvv..rro./ http///ba///./...eee 04217
[54] Kartik Sreenivasan, Jy-yong Sohn, Liu Yang, Matthew Grinde, Alliot Nagle, Hongyi Wang, Kangwook Lee, and Dimitris Papailiopoulos. Rare Gems: Finding Lottery Tickets at Initializa- tion. arXiv preprint arXiv:2202. 12002, 2022. URL: bottps://arx/ae org/ /org/abbe abs/ 2202. 12002
[55] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint (xXiv:2330.11699)) 2023.
[56] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. arXiv preprint 17X09:1811.00937, 2018.
[57] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning, pages 6105-6114 14. PMLR, 2019.
[58] Yiqin Tan, Pihe Hu, Ling Pan, and Longbo Huang. RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch. arXiv preprint 2rXiv:2205.15043 2022. URL: https://arxiv 15043
[59] Hidenori Tanaka, Daniel Kunin, Daniel I. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377-6389, 2020.
[60] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking Winning Tickets Before Training by Preserving Gradient Flow. International Conference on Learning Representations, 2020. URL: http::ttt////aat
org/ ht////b////00 2002 07376
[61] Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering Neural Wirings. Advances in Neural Information Processing Systems, 32, 2019, URL: (tt::httts//arxiiv org/
abs/1906.005mn
[62] Huikai Wu, Junge Zhang, and Kaiqi Huang. SparseMask: Differentiable Connectivity Learning for Dense Image Prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages
2019. URL: https: //tp:i/a//iv////aa//aa httaaaaaivv.rrroe/////// htt///////199 1904. 07642
[63] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: I 708.07747, 2017.
[64] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang I.i, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. MEST: Accurate and Fast Memory-Fconoonic Sparse Training Framework on the Edge. Advances in Neural Information Processing Systems, 34:20838-208 20850, 2021. URL: /nttps://////ttt
org/ abaa 2110 14032
[65] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning N:M Fine- fmeeggrined Structured Sparse Neural Networks From Scratch. International Conference on Learning Representations, 2020. URL: abs/2102. 0tt1//222200000000
[66] Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask. Advances in Neural Information Processing Systems, 32, 2019. URL: https:/ / /awxxixivvv.../ org/ httpa//////1.00000
14

---

A Experiments Setup
A.1 Model Architectures
For the small-MI.P setup, we use an MLP with hidden layers with dimension sizes of 256. For the large MIP, we increase the number of neurons in the first hidden layer to 1024, and to 512 in the second. The small-CNN is a standard ConvNet architecture consisting of three convolutional layers with kernel size 3, stride 1, and padding 1, inspired by the standard ConvNet presented in section 8.2.3 of 77] The spatial dimension reduction is implemented by MaxPooling with size In ReLU activations. The architectures
summed up in Table The ResNet-56 and ResNet-50 models follow their standard specifications [22]. The EfficientNet is the EfficientNet-B BO model [57] (we use the implementation from pytorch models), and follows the standard architecture. The VGG-16 model is modified to be used with CIFAR-10 10 and uses BatchNorm. The implementation of VGG-16, as well as the LeNffef-Caffe model, is adapted from R4
all cases,
Table The small-MLP. large-MLP, and small-CNN architectures used in this study. The first two pairs next to a Linear layer indicate the input and output dimensions. The tuple (Cin; Cout, k 8) after the Conv2d described the input channels, output channels, kernel size, and stride, respectively. The ResNet-50, and ResNet-56 architectures follow the standard specifications [22] 2,5,02,912 and 855,770 parameters, respectively). All networks use Softmax nonlinearity after the last mentioned layer.
Layer
small-MLP
large-MLP
small-
1
Linear(24, (24, 256)
Linear(3 32 32, 1024)
Conv2d(3, 32,
2
ReLU
ReLU
ReLU
Linear(256, 256)
Linear( 1024, 512)
MaxPool, size Conv2d(32, 61,, 3, 1)
ReLU
ReLU
5
Linear(250) (256, 1)
Linear( 10)
ReLU MaxPool, size 2
6
7
Conv2d(64, 3, 1) ReLU Global Average Pool Lincar 10)
8
10
Num. Parameters (Dense)
72,449 3,622
3,676,682 183,834
94.538 4.727
Num. Params. (95% Sparse)
A.2 Training Regime
We provide an overview of the datasets we used in our experiments is shown in Table 4 Below, we report the training regime.
Small- and large- MLPs We use a batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0005, and Nesterov= True. The learning rate starts 0.01 and is decayed twice during training: after 50% and 75% of the epochs. We use the decay of 0.1. We train for 100 epochs.
Small-CNN We use batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0005, and Nesterov= True. The learning rate starts at 0.01 and is decayed twice during training: after 50% and 75% of the epochs. We use the decay of 0.1. We train for 100 epochs.
LeNet-5-Caffe We use batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0005, and Nesterov= True. The learning rate starts at 0.01 and is decayed twice during training: after 50% and 75% of the epochs. We use the decay of 0.1. We train for 100 epochs.
ResNett-56 We use batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0001, and Nesterov= True. The learning rate starts at 0.1 and is decayed twice during training: after 50% and 75% of the epochs. We use the decay of 0.1. We train for 200 epochs.
ResNet-50 We use batch size of 256, the SGD optimizer with momentum 0.9, weight decay 0.0001, and Nesterov= True. The learning rate starts at 0.1 and is decayed twice during training: after 50%
and 75% of the epochs. We use the decay of 0.1. We train for 90 epochs.
15

---

DST Settings specific for dynamic sparse training: We use the ER initialization for the distribution of sparsity levels in MLP, and the ERK initialization for the convolutional models. We start with an initial pruning fraction p of 0.5, which is decayed during training using a cosine decay schedule (see the main text). The default topology update period At is set to 800, but in the experiments for Q2 vary this hyperparameter. Note that we also keep the update period At = 800 for the ImageNet, since together with batch size 256 it matches the number of training examples seen by the model before the VGG-16 We use batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0001, and Nesterov= True. The learning rate starts at 0.01 and is decayed twice during training: after 50% and 75% of the epochs. We use the decay of 0.1. We train for 200 epochs.
EfficientNet We use batch size of 128, the SGD optimizer with momentum 0.9, weight decay 0.0001, and =esterov= True. The learning rate starts at 0.01 and is decayed twice during training: after 50% and 75% of the epochs. We use the decay of 0.001. We train for 100 epochs. We train on the 64x64 Tiny-ImageNet images (we do not re-scale them to 224x224 and we do not use pre-training in the main text experiments).
next topology update reported in [12). For completeness, we also provide the DST hyperparameter setups used to obtain the Figures the main text Table
Table 3: The DST hyperparameters used to obtain the results presented in the main-text experiments.
Experiment
Prune Fruction
Prune Fraction Decay Scheduler
Density
Update Period
Fig.
0.5
cosine
0.05. 0.06, 0.07, 0.08, 0.08, 0.09.
800
0.1. 0.15, 0.2, 0.4, 0.5
Fig.
0.5
cosine
0.2
800
Fig.
0.5
cosine
0.2, 0.1
25, 50, 100, 200, 400, 1600,3200,6400
Fig.
0.5
cosine
0,5
800
Fig. 6a
0.5
cosine
1.0
800 (measured only once)
Fig. 6b
0.5
cosine
0.2
800
Other Setup and Infrastructure Details We perform 5 runs for each setup, except for the ImageNet dataset, in which we use 3 runs. We do not fine-tune anything specific for any model. In all the experiments performed, at the beginning of the code, we fixed all the random seeds (random. seed, numpy random, torch. www.maannnaeeeeeee to deterministic. The DST pruning criteria (if not used with random growth) are deterministic and for the same seed start with the same sparse initialization. While computing the Jaccard Index between the end masks we use only one worker for data-loading. In terms of reproducibility, we provide the environment setup but also perform our experiments in Singularity container, for which we will provide the definition file in the repository. We provide the hyperparameter setups for all the main-text experiments in the code I1
torch. cuda. maanallsseedae
and set the cudnn backend
We used GPUs either of the NVIDIA Tesla V100 or NVIDIA A100 type. We used external servers with job scheduling. All experiments (except for ImageNet) were performed on a single GPU. No other computations could access that GPU at such time. The ImageNet experiments were run using distributed computing on eight NVIDIA A100. Each task was using a batch size of 32 (so that the total batch size is 256) and the hyperparameter configuration described for the ResNet50 above.
Our code is accessible at based on the publicly available repositories for (34] and [I]] available at httpp:://ttthtist com/ VITA-Group/ Sikando_PPrunt and https://gii ht////tthubt com http///immeetttrss/ee/mm http//ppaaeeeaaeee learning respectively. The used sparse-learning library (of 11] is published under the MIT license 12
It is
11 Under the specs directory the code. We include the copy of that licence in the sparse-learnie directory in our code.
16

---

Table 4: Datasets overview. The validation and training set on ImageNet follow the standard division. For CIFAR- 10, we use 5000 of the training examples to form the validation test and fit the models on the remaining 45000. The test set for CIFAR-10 10 follows the standard division.
dataset
type
samples
features
classes
train-valid-test
source
URL
Higgs
tabular
940,160
24
2
70%--155-
15%
opennml oppenml.rppddi44111 1r
CIFARIO
image
60,000
3x32x32
75%--8%-177
17
P2)
co.toronto cowi/-kizziiffrr
CIFAR100
image
60,000
3x32x32
100
75%-88%-179
Co.foronto caul krzzciiar himd
ImageNet
image
1,331,677
3x469x38
1000
90%-33%-9%
49
imare-ne mere-neet.n coane-nee.rriccaee orpichal hennech NV RC/2012
FashionMNIST
image
70,000
Lx28x28
77%--9%-149
eithub, com' comizallasooreseee wrrhottsshoonn
Tiay Net
image
100000
3x64x64
200
80%-1099-10
29
12251n.stanlor cula lany -magendd--------- 200
A.3 Tuning MEST
Recall from Section 3.2 that the MEST criterion computed as s(0) = (0] + X[VoC(D)) Since CMEST depends on the À hyperparameter, we verify how the change of that value impacts the overall results for various densities. We present the results on the large-MLP model in Figure 7 We observe that for Ả close to 0, the MEST criterion is, as expected, close to the magnitude criterion. Increasing does not seem to benefit the performance. particular, depending highly on the gradient disadvantageous.
64
validation accuracy (%)
62
60
58
1060
011
100
0001
56
0.06
0.08
0.30
0.15
0.20
0:30
0.40
0.50
density
Figure 7: The performance of the MEST criterion with random growth for different parameters of the hyperparameters A. We observe that for low values of X, the criterion behaves similarly to magnitude, while for very large values, the performance drops. There does seem to be good value between those two extremes.
B Sensitivity and Reciprocal Sensitivity
We investigate the relation between the Sensitivity and Reciprocal Sensitivity on the large-MLP model on CIFARIO. The results are presented in Figure We observe that not only is the Reciprocal criterion better terms of the attained test accuracy, but it also seems be more stable throughout the training. We argue that this is due to how those two criteria deal with weights that are large in magnitude but have small gradients. Note that such weights can be considered stable and hence
Theeee are rounded percentages. We use the standard split of 45,000 5,000 10,000 images.
"These are rounded percentages. We use the standard split of 45,000 5,000 10,000 images.
19This is the average resolution. Images are randomly cropped to size 3x224x224, as is standard practice.
"Peercettagss are rounded. We use the standard train/validation/test split. do not use test set and evaluate on the validation set.
PPrcentages rounded. We apply the default train/test split (60000/70000) and move random 10% of the train set validation set.
We use the standard train/validation/eest split. We use the validation set as the test set (we do not optimize any hyperparameters using the validation set).
17

---

be desired to keep active. While the Reciprocal criterion would assign them a very high score, the Sensitivity, having an inverse relation the gradient, would most likely mark them for removal.
accuracy
tost accurey
30
Mayy
86
my
86
Casewisity
8
Caunndy
8
3,06
0.00
3.30
0.15
3.20
3.30
0.10
20
epoch
density by
Figure 8: Left:: The density versus test accuracy for the MLP dataset CIFARTO 10 obtained by the
Reciprocal Sensitivity and Sensitivity pruning criteria. Right: The validation accuracy throughout the training for density 0.2 (denoted by vertical dashed line in the first plot). We may observe that the Reciprocal Sensitivity more stable and gives much better results.
C Results On Text Data
In addition to the tasks studied in the main text, we conduct an evaluation the pruning criteria on the fine-tuning task of ROBERTa Large [39] with approximately 354 million parameters, utilizing the CommonsenseQA dataset [56) adapted from the Sparsity May Cry (SMC) Benchmark [35].
We use the exact same setup in the SMC-Benchnark by first performing magnitude pruning for the sparse initialization and then using DST during the fine-tuning phase. We also use the same hyperparameters. Please note that this is different configuration from the one used in the main text, which we trained all the models from scratch (in spirit, the CommonsenseQQ study is more similar to the EfficientNet fine-tuning experiment from Appendix H). We report the results in Table and Table 6 for the random and gradient growth, respectively.
Similar to the findings of B5,, we observe that in this scenario, the achievable sparsity without sigmificant performance decline notably lower than in vision or tabular data scenarios. Concerning random growth, the EST criterion appears to be the most suitable choice. For gradient growth with a density of 0.8, the CnSemsitivity and CSET criteria initially demonstrate strong performance but are eventually surpassed by the Emagnitude and CM EST criteria at density of 0.7. Additionally, we note a significantly higher variance in the outcomes across criteria compared vision tasks.
Furthermore, we conducted brief exploration of update period values for this problem (refer to Figure Notably, we observe that overly frequent updates do not yield beneficial results. Concerning gradient growth, the most effective update period value is consistently around At = 500 for nearly all criteria. For random growth higher update period values generally lead improved outcomes.
At the same time, please note that the studied pruning criteria had been evaluated either on tabular vision datasets most of the works that have introduced them used them (e.g.. 42 45
12 [1). Large language models typically have different structure and are based on the attention mechanism. In consequence, the applicability of sparse training in such setup is still open of research [35] (55) This motivated our choice of fixing our focus on tabular and vision models in the main text.
D Additional Performance Plots
In this section we present the performance of the different pruning criteria the two remaining architectures from the main text the small-CNN trained on CIFARIO 10, and ResNet56 trained on CIFAR 100. The results are in Figure (see (see also Figure 22 for all the models in one plot).
18

---

Table 5: The mean accuracy results for the CommonsenseQA task on ROBERTa Large Model from the SMC-Benchmark (Liu al. 2023). We use the same hyperparameter setup (Liu et al. 2023) and vary only the density and pruning criterion. The results below were obtained for random growth. In brackets, we report the standard deviation from 3 runs. We bold out the best result for each density and underline the second-bess one.
random growth
0.9
0.8
0.7
0.6
0.5
magnitude
73.80 (0.52)
37.57 (25.25)
26.53 (6.71)
19.80 (3.13)
20.60 (0.69)
CSET
74.40 (0.60)
49.50 (26.98)
21.00 (3.99)
20.13 (1.95)
18.63 (1.44)
CMEST
74.77 (0.67)
52.30 (29.79)
26.90 (7.20)
19.77 (0.49)
20.73 (0.96)
CRSensitivity
25.87 (8.36)
22.13 (1.69)
23.27 (0.85)
21.37 (1.66)
21.20 (0.53)
CSNIP
56.57 (1.82)
34.20 (9.11)
19.57 (1.31)
21.60 (0.95)
19.17 (1.47)
Table 6: The mean accuracy results for the CommonsenseQA task on ROBERTa Large Model from SMC-Benchmark (Liu et al. 2023). We use the same hyperparameter setup as (Liu et al. 2023) and vary only the density and pruning criterion. The results below were obtained for gradient growth. In brackets, we report the standard deviation from 3 runs. We bold out the best result for each density and underline the second-bess one.
gradient growth
0.9
0.8
0.7
0.6
0.5
moggistuee
76.00 (0.46)
55.80 (29.51)
37.37 (18.23)
24.27 (2.94)
19.33 (0.35)
CSET
76.30 (0.75)
68.43 (6.15)
28.57 (1.85)
22.60 (1.47)
22.17 (0.93)
CMMET
74.73 (0.15)
53.90 (29.27)
37.53 (8.45)
24.27 (2.21)
19.63 (1.12)
CRSenaitsuity
75.83 (0.55)
69.60 (6.70)
25.43
20.67 (2.81)
21.57 (1.82)
CSNIP
65.50 (2.27)
48.20 (7.79)
32.70 (5.98)
23.17 (4.82)
20.73 (0.61)
RoBERTa- Commonsense(AA
RoBERTa CommonsenseQA
20
random growth
test accuracy (%
60
gradient growth
test accuracy (%)
60
50
50
40
40
20
20
10
10
0
400
update period At
update period At
Coagnituee
CAEST
Cnsensitivity
CSXIP
CSET
Figure The study of effect of change of update period the performance CommonsenseeQA task on ROBERTa Large. Left: random growth, right: gradient growth.
19

---

CNN-CIFARIO
ResNet56-CIFAR100
85
72
70
random growth
test accuracy (%)
80
70
61
65
62
60
60
85
72
70
gradient growth
test accuracy (%)
68
75
66
64
62
60
0.05
15
10:20
0.50
60
0.08
0.10
1115
0:200
0.30
density
density
Static
CSET
C Rrensitivity
C Magnitude
CMSST
CSNIP
The performance of the pruning criteria the CNN-CIFARI0 and BesNet56-CIFAL 100 datasets for different densities. Please note the logarithmic scale on the x-axis. We observe that the differences between the methods are larger for the low densities.
Figure 10:
We observe that again the differences between the studied methods seem be more visible the low- density regime. In that setup, the CSNIP criterion is achieving worse test accuracy than the CMEST and the magnitude- based methods. The CRSensitiesty also performs poorly on the CNN-CIFAR10 model. We can see that the behavior for the VGG-16 network observed in the plots in the main text is not due to the change of dataset, but rather the change of the model (since ResNet56 on CIFARI00 has similar results to ResNet56 on CIFAR10 10 from the main text).
E Prraae-trra (ITOP) Study
known component that has been used to explain sparse training performance together with update- period sensitivity is the ITOP (In Time Paramerttttria)ion))) ratio (38). The ITOP measures the number of "explored' parameters divided by the total number of parameters. The parameter is 'explored" it has been included in the mask least once during the whole training.
We study the relationship between ITOP, update period, and the pruning criteria on the example of the large-MLP model on CIFAR10 with density 0.1. The results shown in Figure II] There some interesting insights gained by this analysis.
First, we observe that when paired with random growth, the choice of pruning criteria leads to almost identical final ITOP ratios. (Note that the points for different pruning criteria overlap in the left plot, we verify that they not exactly equal). This is expected because, in such case, the growth
20

---

Randor Growth
Gradient Growth
140
145
1000
008
87
8
1000
200
updale period
updatee
Calenitule
CEET
CXEST
CSMI
The update period At versus the ITOP ratio obtained at the end of the training for different pruming criteria on the large MLP. The left plot corresponds to random growth, the right plot corresponds to gradient growth.
Figure 11:
criterion is independent of the chosen set of weights to prune. Hence, the increase in ITOP is only due to the growth criterion, which (being random selection) not affected by the pruning criterion. This is in contrast to gradient growth, in which the change in the pruned weights can impose a change to the gradient flow in the model and hence impact the rate of the exploration. Aside from this observation, confirm that gradient growth has, in general, a lower ITOP ratio than random (which was shown (38).
Secondly, in the gradient growth setup, the magnitude criteria and CMEST have much higher ITOP ratio for small update periods than the gradient-based pruning criteria. This indicates that they perform more exploration, which perhaps may be the reason behind their better performance observed in Figure 4 in the main text.
Finally, we notice that final ITOP ratio close to 1 is not necessary for good generalization. Observe that for At 800, we get the best validation accuracy for random growth, MLP, and CIFARIO (Figure in our paper), but it only reaches an ITOP ratio of 0.7. Even with final ITOP ratio of just 0.2 (for At 64400), sparse models can beat the performance of dense models.
F Sparse Hardware and Software Support
As discussed in [53] research on sparsity is being pursued three main areas to create faster, more efficient neural networks. The first area is the development of hardware that can take advantage of sparse networks, such as NVIDIA's A A100, which supports 2:4 sparsity (a restricted version 50% layer sparsity) [65] The second area is the creation of software libraries that can implement truly sparse networks 37 g. The third area is the development of algorithms that can produce sparse networks that perform just well dense networks (23] All these steps being undertaken collaborative community effort, aiming to create faster, memory- and networks. Additional information this topic be found in 24 41
deep neural
G Gradient Based Methods and Batch Size
In Section 5.2 in the main text, we observe that, in general, the magnitude- based pruning criteria perform better in sparse regimes than the gradient-basee ones. Note that the gradient is computed for a batch (using SGD), and hence, small batch sizes might introduce more significant errors in ahylor-approximation based pruning (e.g.. SNIP). At the same time, computing the gradient over the entire training set is not feasible DST since it would significantly increase the time complexity of each mask update.
21

---

Batch Size 128, Ir 0.01
Batch Size 1024, Ir 0.08
94
random growth
test aceuracy (%
.92
92
90
90
86
86
34
81
0.015
0.06
0.08
0.10
0.15
0.20
0.30
0.40
0.50
0.01
0.06
0.08
0.10
0.15
0.20
0,90
0.40
0,30
test accuracy (%)
gradient growth
.22
92
90
90
88
88
84
84
0.11
0.06
0.08
0.10
0.15
0.20
0.30
0.40
0.50
0.115
0.06
0.08
0.15
1..20
0,30
0.10
0.50
density
density
Static
CSET
RSensitivity
Magnitude
CMMET
CSNIP
The performance of different pruning criteria on LeNet-5-Cafii on FashionMNIST, The first row corresponds random growth, the second row corresponds to gradient growth. The left column indicates a batch of size 128, and the right column indicates a batch of size 1024. The black dashed line in both plots is the performance of the dense model from the main text.
Figure 12:
To investigate the impact of the batch size on our results, we perform a small study on the LeNet-5- Caffe model (trained on FashionMNIST), in which increase the batch size almost 10 times up to 1024. At the same time, we increase the learning rate to 0.08 from 0.01 so that the learning-rate- lo-batch-size ratio stays approximately the same (as suggested by the scaling laws, for instance, in [5I). We compare the obtained results with the standard setup with batch size of 128. We also decrease the update period from 800 to 100 so that both setups, the DST updates are made after seeing the same number of training examples. We report the results in Figure 12
Interestingly, observe that for batch size 1024 CSNIp indeed seems perform slightly better. Cnsensitivity and CSET, on the other hand, become significantly unstable, and CAESTT is still indistinguishable from magnitude pruning. Those results may indicate that the impact of the batch size, although existent, is not significant. Furthermore, as we will see in the next paragraph, increasing only the batch size of the DST update is also not beneficial.
Increasing Only the Batch Size of the DST update. In addition to the above-conducted experi- ments, we examine an ablation which we increase the batch size to 1024 only when computing the gradient used for the DST update. During normal steps of optimization, from previous experiments (128). The results are in Figure [3 We observe that does not enhance performance. Let note, however, that, in general, the impact of the batch size still needs additional investigation in larger models and datasets in order to be evaluated with certainty.
the standard choice
H Fine-Tuning EfficientNet and the Choice of Pruning Criterion
In all experiments in the main text, we perform the training from scratch (no pretraining is applied). This includes the EfficientNet on the Tiny-Imagel from the main text. In this section, we also analyze the difference in the performance of the pruning criteria when evaluated in fine-tuning
22

---

MEP-CIFARIO
MLP-CIFARIO
LoNeTS- JINIST
LoNeT5-Ca Caa---ClEE--SSSSS
Train DST
DST
128, DST
94
test accuracy (%%
92
92
random growth
66
90
90
64
64
88
62
86
86
60
60
84
84
68
68
94
94
gradient growis
lest accuracy (1)
7666
92
92
64
64
88
88
62
62
86
86
60
80
84
0.50
0.05
0.10
0.15
0.20
0.50
0.15
0:20
0.50
0.05
0.10
0.15
0..00
0.10
0.15
0.20
0.50
density
density
density
donsity
Staie
Gaagotade
CSET
GHST
Casenatiity
Coup
Figure 13: The performance of MLP-CIFAR 10 and LeNet5- Caffe-PashionMMIIST from the main text exper-
iments (first and third column) and the performance when using an increased hatch only for the DST update (sccond and last column).
scenario. In such setup, initialize the weights of the EfficientNet the pretrained parameters from ImageNet. We only re- re-initialize the linear projection in the head of the model (which needs to have the output dimension changed from 1000 to 200 since there are 200 classes in Tiny-ImageNel) setup (see Appendix A). We
We keep all hyperparameters the same as in the compare the results in Figure 14
We observe that the setup that uses the pretrained weights, the CSNIP criterion performs much better than when training from scratch. We hypothesize that perhaps the use of the Taylor approx- imation of the change in the Loss function (used the SNIP criterion) allows not to diverge far from the already quite well-established solution from the pretraining. The purely magnitude- based methods, on the other hand, disregard the potential change the loss they may cause and hence, do not successfully leverage the information from pretraining. However, the same time, the general performance of all DST methods when starting with pretrained parameters on ImageNet is much worse in the low-density regime compared to their DST counterparts trained from scratch. This may suggest that the DST framework in the fine-tuning scenario is heavily biased by the pretrained weights and potentially does not efficiently explore possible connectivity patterns. These results suggest that one should be careful not to transfer the findings of the DST training to fine-tuning scenarios. In the second case, more research is needed understand the behavior of the DST algorithms.
I Global versus Local Pruning
In our study, we have decided to focus on local pruning as this is the typical approach in the DST literature. Examples include RigL [12 (Algorithm 1), SNFS (I] (Figure I, where pruning and redistribution occur layer-wise), SET [42] In contrast, global pruning computes and compares the importance scores associated with the pruning criterion with all the parameters in the network. Hence, global pruning may change the sparsity distributions in each layer in unsupervised manner. Note that global pruning may also have one small disadvantage: It may carry a higher risk of disconnecting the model if the importance scores s((,,) are not balanced across different layers. Using local pruning may avoid this risk and omit the need for investigating layer-wise normalization schemes.
However, we do consider the question of how the studied pruning criteria perform when paired with global pruning an interesting one. perform small experiment on ResNet-56 with CIFAR 10 in which we compare the different pruning criteria in the global- and local-pruning setup densities
23

---

From Scratch
Pretrained
60
test accuricy (19)
50
30
40
40
30
30
0.05
0.35
02.00
10.00
0.05
0.06
EL06
0.10
0.35
0.200
100
11.0
60
test accuracy
20
20
0.05
0.06
0,08
0.10
I130
20
0.05
0.00
0.00
0.30
(140
density
density
Static
CMagnituude
CSET
CMEST
Casensitivity
CSNIP
Figure 14: The performance of different pruming criteria the EfficientNet architecture trained from scratch (left) and fine-tuned on the weights from training on ImageNet (right). The first row corresponds to random growth, and the second corresponds gradient growth. We observe that in the setup that uses the pretrained weights (right column), the CSNTP criterion performs much better than in training from scratch. However, the same time, the general performance of all DST methods is much worse in the low-density regime compared to the training from scratch.
0.05, 0.1, and 0.15. We present the results in Figure 15 both global- and local- setups are indeed similar, with slight preference for global pruning. By analyzing the plots, we can see that, in this case, the ranking results of the pruning criteria from the main text are indeed preserved. We consider the subject of using global pruning very intriguing topic to investigate further in future research.
We observe that the results obtained in
J Effect of Batch Size and Pruning Schedule on Update Period
When considered together with the batch size and dataset size, the update period can be interpreted as the number of data samples the model sees (and trains on) before performing the mask update. In consequence, one may expect that with the increase of the batch size, the best choice of the update period will shift lower values, keeping the total number of seen examples similar. To visualize this effect, Figure [6 c) and Figure [)).))
comparison the MLP-CIFARIO 10 setup and observe just this behavior (see
Another potential confounding factor is the pruning rate schedule. In all our main experiments, we used the cosine schedule, which known to perform well in the DST setup [1]. In this section, we additionally test this schedule on the MLP-CIFAR10 setup against two different schedulers: linear and constant. The linear schedule multiplies every 600 optimization steps the current decay rate by constant factor (0.99). The constant schedule keeps the prune fraction fixed at 0.5 throughout the whole training- The results are presented Figure a) and Figure 16(b). We observe that the choice of the pruning factor schedule does not significantly influence the best value of the update period.
We leave the CSET out due its high similarity the magnitude pruning.

---

Density 0.05
Density 0.1
Density 0. 15
Categitata
clobal val. soeurayy (96)
Curget
91
90
CaNar
90
89
Ostegitace
Cygginer
GREET
Cxest
86
Cuseermntty
Cexur
Casur
87
87
68
90
92
90
92
88
90
92
local val aceuracy (16)
local sceuracy (%)
local val accuracy (%)
Figure 15: The test accuracy of ResNet-56 model using different criteria for densities 0.05, 0.1, 0.15, We plot the test accuracy obtained with global pruning (y-axis) versus the test accuracy obtained with local pruning (x-axis). Note that if the performance in both setups was the same, the values would perfectly lie the gray, dashed diagonal line. Points above that line indicate that global pruning performed better.
(4) seta 188128
(b) schedlitk: constant, B88288
(C) adisstolecenddnn BS 18:128
(d) scheclile cositel, H8S22e
02
gorth
38
58
26
0200
2010
0024
200
Figure 16: Additional results for the sensitivity hyperparameters in the update-perion study. We perform the
study on random growth. From the left, we plot the results for (a) linear scheduler and default (i.e., 128) batch size (BS), (b) constant scheduler and default batch size, (c) cosine scheduler and default batch size (the same setup as in main paper)(() cosine scheduler and batch size 256.
K Additional Jaccard Index Plots
We present the Jaccard index values computed after the first topology update of the model for different initial pruning ratios p in Figure 17 Naturally, for smaller p. the difference becomes more visible. The CMagnitude- CMEST and CSET criteria choose similar sets of weights to prune, while CSNIP and CRSeasitivity are more diverse in their selections.
In addition, we also study how the similarity between the chosen sets for removal changes with time. To this end, we run each criterion with the DST framework until the end of training. Next, every 3200 iterations, we examine what weights for removal would be chosen by the remaining pruning criteria at this point training. Note that such a relationship does not need to be symmetric. We report the results in Figure 18.
Moreover, we analyze the statistics of the never-remove rever-removed connections. To be precise, we consider the masks at the end of training for the MLP, CNN, and ResNet-56 models on CIFARIO. Next, we measure the number of weights that are always retained by every pruning criterion for the same seed and we divide it by the number of all remaining (unpruned) weights in the model. We do the same on the negation of the masks to get the number of the always removed weights and divide by the number of all removed weights of the model. We present the results in Figure [9 For the MLP model, around 23% of all the retained weights are shared across all the models. For CNN, this number is higher and goes up to 37%, while for the ResNet-56 it is equal to 11%. In general, the smaller overlap of the pruning methods for the ResNet model is also something that we observe in the main paper in Figure 6b.
25

---

Finally, we report the standard deviations for the Jaccard Index plots from the main text in Fig- ures and Note that the mean Jaccard Index for the sets chosen for removal the main was computed by first taking mean over different seeds for each layer and then averaging the results over all layers. Hence, in Figure 20 the standard deviation calculated for each layer separately and then averaged. For the end-similarity. the Jaccard Index was computed by first taking the mean over all layers for a given seed and then taking the mean over all random seeds. Hence, Figure 21 we report the standard deviation for averaging over the different random seeds.
Cstanimi
Chssustertr
Colagiitale
Cxnat
Chusston
CSET
Csusy
Cauaaian
CSET
Cynst
Casur
Canasio
CeET
CArST
CENTr
Caandon
CSET
CMEST
CsxI
Caseualastty
02
CRandum
10.00%
0= 25.00%
12.500%
0
(a) MLP
Copt
C Calggitale
ChieST
Coou
Crekesiiitty
CSET
Calegitahs
C
CSNP
CreSensnant
Chankon
CsET
Calogiteuts
CRBST
CosP
Crstersiisty
Crandoo
CArT
Calogittad
CARST
Caxp
CSET
0.8
Categitade
CAEST
CSNIP
02
Candva
p 50.00%
D 12.50%
N CNN
(b)
Cstapinnia
Cstessitoi
Casenntove
Cotenitnc
Cser
GrEST
Chendeon
CSRT
CansT
CANIF
Chuuaonn
CSET
CaSUP
Cataato
Ceen
CAEST
CENTF
Citandom
2.0
CSET
Catouarade
008
CAEST
0.6
CSxI
Casmatadt
Craviom
50.0%%
2=25.00%
12.50%
6= 25%
0
(c) ResNet-56
Figure 17: The Jaccard index between the selected pruning criteria computed for the same network state. The values are averaged over different seeds. We observe that the CMagnitude- CMEST and CSET criteria select similar sets of weights to prune.
L Learning Curves
In this section, we plot the validation accuracy during the training for different densities and pruning criteria for the small and large MLPs and ConvNets. We notice that CSNIP and CRSensitivity criteria often lead instabilities during the learning see Figures 23
24 25
29 30
26

---

step
6400
step 12800
step 19200
step 25600
step 32000
1.0
GRRT
0.8
CunsT
0.6
Caar
0.1
Co
0.2
Chentio
Cest
Cataaini
Couur
Casesitirs
Chusion
Cestr
Coui
Cascsilias
Chuskon
Cert
Catagnnat
CAIST
Cssi
Chuskon
Cerr
Calganate
Casi
Crocelliias
Chuskon
Cero
Calganats
Cueer
Csen
Cascolltrss
Chanka
(a)
step 6400
step 12800
step 19200
step 25600
step 32000
1.0
GEET
0.8
CURST
0.6
CanP
0.4
0.2
Chash
0
Car
Cane
Cosu
CaLi
Catt
CANt
Carl
Csteg
Canss
CANI
Carl
Cars
CENI
Chusbb
Car
Chres
Cruber
(b)
step 3200
step 16000
step 32000
step 48000
step 64000
L.0
CEET
0.8
CAEST
0.6
CamP
0.4
We
0.2
Cent
Catantiake
Covur
Cnseaaliity
Charkon
Cent
Catantidl
Canis
Casur
Onsestthiy
Chuskon
Cent
Cstantade
Chrrs
Chusken
Ceet
Caren
Chusken
Coer
Colasnude
Caren
Opseattry
Chasia
0
(c) ResNet-56
Figure 18: The Jaccard index between the selected pruning criteria in time. For each of the methods
in the rows, we run the standard DST procedure. Then, every 3200 iterations, we verify what set of weights would be chosen such a point by the remaining pruning criteria. Note that this relation does not need to be symmetrical.
1.0
Retained
5% Renowed
Removed
0.80
5 Benoweed
0.75
0.6
0.55
0.37
0.23
0.2
0.11
MLP
CNN
ResNet-56
Figure 19: The number of weights retained by all the pruning criteria relative to the number of all unpruned weights (in blue). The number of weights removed by all the pruning criteria relative the number of all pruned weights (in pink).
27

---

Calgaitaaio
CRSemsttviity
Cagauttoo
CRSemstivviy
Magaitodo
Ctandean
Random
CSET
MEST
CSNI
CEET
MEST
CNIP
CSET
CMEST
CSNI
0.033
CSET
000
.007
8007
.016
.000
Maggittue
002
1000
001
004
004
002
1007
0000
.003
.15
012
008
1007
000
.07
008
.000
0009
0.02
CMEST
CSNIP
001
1001
1003
1000
.H16
0012
.013
.015
.014
4000
X222
.010
.009
1109
.110
000
.014
0009
0.01
CuSenstivity
005
.012
CRaddmm
003
002
0002
.002
0002
0.00
1010
.009
000
.010
0007
0000
.009
000
000
008
010
000
MLP
CNN
ResNet-56
Figure 20: The mean standard deviation computed for the Jaccard index between the sets of weights chosen for removal during the first update of the sparse connectivity.
Calgnatale
Cangnit dl
Calguitale
CaET
CLEST
CSNA"
CGET
CAEST
CANI
CaET
CMSTT
CSNP
Init
bit
Init
0.2
CaRT
017
.016
Caaggatoi
.001
0000
.00
400
1033
1011
.014
1000
011
.15
J077
002
.016
000
.017
.015
001
0033
0.15
CansT
1044
.005
.00
.001
004
0000
.011
.0.1
000
.015
.007
0002
.016
.017
.000
.013
.003
0.1
CSNIP
0015
.015
.034
J00
.015
XX1I
.117
.0'
.05
(11
J112
J001
.018
.119
Cnsemeristty
004
003
004
005
000
001
300
.007
007
012
800
.004
010
011
.013
.008
000
.005
0.05
001
000
001
002
001
.017
002
003
MLP
CNN
ResNet-56
Figure 21: The standard deviation of the same index computed between the masks obtained by different pruning criteria at the end of training. The rows and columns represent the pruning criteria between which the index computed.
28

---

LeNott-Cadi--MMN PMNIST
naniom growth
graddintt growth
tost arerrrayy (96)
test aeenneey (96)
MLP-HIGOS
tost acuurrncyy (36)
CNN-CIFARIO
RosNotiC-- CIEAARO
test mcennaey (5)
tred arviracy (9%)
MLP-CIFARIO
testt aceuneey 1%6
New-TaiX ThrTN Bost ncerrayy (96)
Elfecian Nec-
30
VGGI6-CII CIFARLOO
frost nostuneyy (36)
0.03
0110
0.10
82
11:0
density
deusity
Statie
CagT
Cpeennttnryy
Calagnitnde
CAEST
CSMI
Figure 22: The performance of the pruning criteria with respect to the density on different datasets. Each row corresponds to different mnoee-ddatsset pair. The first column corresponds to the random growth, the second one corresponds to the gradient growth. Please note the logarithmic scale in the x-axis. We may observe that the differences between the methods become more visible predominantly the low-density regime. DST almost always outperforms the static sparse training (cyan line).
29

---

denalty 0.05
dematry 0.0
denstty 0.07
density 0.8
valiciation
60
40
demsity
Amsityy
desity 0.15
dlsiity
80
valiciation
%
4D
demsity
deaeity
density
80
valiilction
600
100
50
100
100
epoch
opoch
spoch
Maagnituii
Csir
CAEST
Casennitiiiity
CSNIP
Figure 23: The validation accuracy the training epoch for different pruning criteria and densities on the LeNet-5-Caffe (FashionMNIST dataset).
density 0.05
density 0.06
density 0.07
density 0.08
vall-ation
40
dlenstyy
demsiry
demiity
dimiity
vuliiieoon
%
40
denery 0.3
denelty
density 0.5
ball ioliciation
4D
100
50
100
100
spoch
epoch
epoch
MMagnitaric
CSITT
CXEST
Cpsensitivily
CSNIP
Figure 24: The validation accuracy versus the training epoch for different pruning criteria and densities on small-MLP (Higgs dataset).
30

---

denalty 0.05
demaiyy 0.0G
denstty 0.07
density 0.88
valeilation
50
demsity
Almsity L
dmsity
dimsity
valicitioon
%
80
25
density
deneity
density
75
viliiictoon
25
100
50
100
100
epoch
opoch
spoch
CMagnituii
CsEr
CAEST
Cissnniitivity
CSNIP
Figure 25: The validation accuracy versus the training epoch for different pruning criteria and densities on the small CNN (CIFARIO dataset).
density 0.05
demsity
density 0.07
density 0.08
sal-dation
50
25
density
demairy
demiity
dlmsity
75
valibieoon
%
50
demity 0.3
denelty
dencity 0.5
75
vall ialiciation
50
100 spoch
200
100
200
epoch
epoch
MMagnitaric
CSITT
CXEST
Cpsensitivvlyy
CSNIP
Figure 26: The validation accuracy versus the training epoch for different pruning criteria and densities on ResNet-56 (CIFARIO dataset).
31

---

denalty 0.055
demaryy O.GG
desstty 0.07
density 0.8
60
valiilation
40
20
demsity
Almiity L.]
dmsityy
Almsity
valiciaioon
%
40
demsitt
deaeity
density
60
valiilction
40
20
100 spoch
200
100
200
100 spoch
200
spoch
Caagnituii
Csir
CAEST
Cissnniitivityy
CSNIP
Figure 27: The validation accuracy versus the training epoch for different pruning criteria and densities on the ResNet-56 (CIFAR100 dataset).
density 0.05
density 0.06
density 0.07
density 0.08
vallcationn
40
20
density
demoiyy
demiity
dlmsity
valiiieoo
40
20
denity 0.3
deaelty
deneity 0.5
60
Daliciation
40
20
100
50
100
100
spoch
epoch
epoch
CMagnitaric
CSITT
CXEST
Cpsensitivilyy
CSNIP
Figure 28: The validation accuracy versus the training epoch for different pruning criteria and densities on large MLP (CIFARIO dataset).
32

---

denalty 0.d5s
demaiyy O.GG
demiity 0.07
density 0.88
40
valiiiation
20
demsity
dlmiity L.1
dmsityy 0.15
dimsity
40
valicitioo
density
deusity
density
40
valiilction
20
100
50
100
100
epoch
spoch
spoch
CMagnitati
CSET
CAEST
Casennitiiiity
CSNIP
Figure 29: The validation accuracy versus the training epoch for different pruning criteria and densities on the EfficientNet (TinyImagenet dataset).
density 0.05
density 0.06
density 0.07
density 0.98
vallhation
40
density
demoity
demiity
dlmiity
valiiieoon
%
denity 0.3
deaelty
dencity 0.5
600
Ialiciation
40
20
100 spoch
200
100
200
200
epoch
epoch
MMagnitaric
CSITT
CXEST
Cpsensitivilyy
CSNIP
Figure 30: The validation accuracy versus the training epoch for different pruning criteria and densities on VGG (CIFARION 100 dataset).
33


Reviews:
[{"review_id": "oXjwEve4ZH", "reviewer": "Reviewer_Q7vN", "summary": "This paper provides a comparison of several dynamic sparse training methods and conclude that they are perform similar unless in the ultra-sparse regime, in which case the magnitude-based pruning performs best.", "questions": "N/A", "limitations": "N/A", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The comparison is through and useful for new researchers in this field.", "weaknesses": "This paper only provides empirical experimental results for known dynamic sparse training. No novelty is the main issue for this paper. I think it is more proper for it to be publish in a report than in a conference proceeding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer Q7vN**\n\nWe thank the Reviewer for the feedback. We are happy to hear that the Reviewer finds our experiments thorough and sees the usefulness of our research. If we understand correctly, the Reviewer’s only concern is in the novelty of the paper.\n\nWe kindly request the Reviewer to acknowledge that scientific research can encompass novelty in various forms. We strongly believe that thoroughly exploring existing solutions holds equal significance to the pursuit of new methods or architectures. Our paper presents a robust and comprehensive study that systematizes the current knowledge in the field while challenging common assumptions made in dynamic sparse training (DST), such as the perceived need for a high number of updates or the superiority of more elaborate pruning criteria. Our study establishes a strong baseline for future research endeavors. We wish to highlight that, prior to our work, there has been no comprehensive study of different pruning criteria in DST. Previous studies either examined them in isolation or intertwined them with other design choices like the growth criterion. We are the first to investigate the impact of pruning criteria on DST performance and the selection of hyperparameters. Additionally, as the Reviewer has also noted, our work provides valuable insights that can greatly benefit researchers in the field. The originality of our paper and the significance of its findings have been praised by Reviewers xwEQ and mfmv. Therefore, we sincerely ask the Reviewer to consider our work from a broader perspective of novelty and reconsider the possibility of adjusting the score accordingly. Your thoughtful consideration of these aspects will be greatly appreciated."}, {"review_id": "out8hx9j18", "reviewer": "Reviewer_aJDG", "summary": "This empirical paper looks into the setting of adapting neural network architecture during training (dynamic sparse training), repeatedly pruning and growing the network. Authors try to draw conclusions about the performance and topology of various pruning criteria. They show that in high density regimes, most of criteria give similar results, with methods that incorporate gradients performing best, but in low density regime, simple magnitude based pruning performs the best", "questions": "This is comprehensive and well executed and I am willing to raise my score if the authors have good intuition for the questions below. \n\n1) My conceptual question is related to line 218 - you are using the same hyper parameters for all pruning criteria. I wonder whether some pruning methods that are more crude for example can benefit from smaller learning rate, etc - basically I am worried that the reuse of all the same hyper parameters is what gives you the results that are all similar to each other. Because in reality, I assume when you train with DST and you set the pruning criteria, you tune the hyper parameters for that chosen criteria. \nAlternatively - any indication (for example, from earlier experiments) that the best hyper parameters don’t depend on the pruning criteria?\n\n2) RE: conclusion that gradient information is useful (for pruning criteria) in high density and not so useful in low density regimes (e.g. magnitude based pruning wins in this case) - can’t it be also a function of the batch size? I would think that for very sparse networks, increasing the batch size will improve the quality of the gradient estimates => this might result in pruning that takes gradient into account to perform better\n\n3) Are the time complexities for all the pruning criteria the same too (I don’t think so, e.g. you need to calculate gradients for some while e.g. magnitude based pruning uses just the weights). This would be nice to include somewhere - if they all perform similar, the cheapest to calculate is preferable\n\nMinor:\nLine 21 cautions->  careful or cautious\nAlso I kinda find the title somewhat misleading - it suggestted to me that you will be coming up with a new criteria/method of \"finding fantastic weights\". Something that makes it clear that it is empirical study of pruning criteria is probably more appropriate (but I do appreciate snappy titles in general)", "limitations": "n/a", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- well written, easy to follow\n- author tackle the question that has not been investigated before\n- the emprical study is rigorous/well executed (modulo my questions/remarks) \n-assuming authors opensource their benchmark, it can serve as a test bed for new pruning and /or growth criteria\n", "weaknesses": "- Limited novelty (empirical study of existing methods). Not very deep insights\n- mostly vision data and only 1 tabular dataset. No text data\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer aJDG**\n\nWe are grateful for the Reviewer’s feedback. We are happy to hear that the Reviewer finds that the questions we ask in our study have not been investigated before. We are also pleased to learn that our study has been recognized as rigorous and well-executed by the Reviewer. We will, of course, make our code open source (the code is also in the supplementary materials). Below, we provide answers to the raised concerns:\n\n*Limited novelty (empirical study of existing methods). Not very deep insights* - We kindly request the Reviewer's thoughtful consideration of the multifaceted nature of novelty within scientific research. A researcher's role extends beyond the creation of new methods; it encompasses an understanding of existing ones as well. Exploring established solutions serves to organize current knowledge and offers valuable insights and directions for future endeavors. Our paper exemplifies this approach by focusing on the DST framework and conducting an analysis of how diverse pruning criteria from the literature influence model performance. As aptly noted by the Reviewer, this inquiry has not been investigated before.\n\nOur work, as underscored by the Reviewer, employs a rigorous experimental setup. Notably, we demonstrate that the more intricate pruning criteria do not necessarily hold a distinct advantage over the fundamental magnitude-based approaches. Furthermore, our study indicates that in various scenarios, achieving commendable outcomes necessitates only a modest number of connectivity updates. We kindly implore the Reviewer not to confuse the simplicity of these insights with their lack of significance. Through our analysis, we cast a critical lens on the efficacy of current pruning solutions. Additionally, our results suggest that we can potentially accommodate more computationally intensive solutions for DST, given the infrequent need for updates. These contributions hold value for both present and future researchers in the field. It is noteworthy that the originality and significance of our findings have garnered praise from Reviewers xwEQ and mfmv. Considering the arguments outlined above, we sincerely invite the Reviewer to revisit our work with a broader perspective on novelty. \n\n*Mostly vision data and only 1 tabular dataset. No text data* - We add a text dataset on the task of fine-tuning ROBERTa Large (please refer to “Joint Response” for the details). \n\n**Questions:**\n\n1. *My conceptual question is related to line 218 …* - The way the hyperparameter setup influences DST is indeed an interesting question. Our choice to use the same hyperparameter setup has two motivations. Firstly, using the same setup for all experiments allows us to assess the improvement that comes from changing only the pruning criterion in isolation from any other changes. Secondly, this choice was on par with the typical DST research in which the training hyperparameters (such as learning rate, etc.) are kept fixed, and the same as in the dense models (see for instance the MLP settings on page 4 of SET in Mocanu et al. 2018, Section 4.1 of RigL in Evci et al. 2019, Section 3.3 of SNFS in Dettmers & Zettlemoyer 2019 or page 8 of MEST in Yuan et al. 2021). We do however study a fair amount of different DST-specific hyperparameter choices. For instance, the entire section 5.3 is devoted to studying the impact of the update period hyperparameter. There we observe that the best setting of that parameter is typically common among all the different pruning criteria (Figure 4). We also study some design choices in DST such as local vs. global pruning, the impact of batch size (please see point below), and the MEST lambda hyperparameter in Appendix H, F, and A.3, respectively. \n\n2. *RE: conclusion that gradient information is useful …* - We agree that this is an interesting question and we study it to some extent in Appendix F (we will highlight this matter more in the main text). When increasing the batch size (up to 1024 from the initial 128) we indeed observe an improvement in the performance of gradient-based criteria, especially when paired with random growth. However, the overall maximal performance of all methods decreases -- this may be the result of training with a too-large batch, as a certain level of stochasticity is considered beneficial in deep learning optimization. Note that we reuse the gradients computed in the backward pass (see also point “3” below) and hence the batch size of the training is the same as the one used to compute gradients for the pruning criterion. We have also run for the rebuttal a quick experiment where we only increase the batch size for the DST update (from 128 to 1024). The results are in Figure R5 in the Rebuttal PDF. We observe, that in general, it does not enhance performance.\n\n3. *Are the time complexities for all the pruning criteria the same too …* - We adapt the setting from MEST (Yuan et al. 2021), in which the current gradient is used to compute the score. In consequence, the computation of such a gradient happens naturally during the backward pass and does not induce any additional cost. We will make this clear in the main text. \n\nWe hope that all our answers provide the necessary clarifications to the issues raised by the Reviewer. We have also followed the Reviewer’s suggestions to include text data in the comparison. Hoping that our answers have addressed the Reviewer's concerns, we kindly ask the Reviewer to consider increasing the rating. \n\nReferences:\n\nMocanu et al. \"Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.\" Nature communications 9.1 (2018).   \nEvci et al. \"Rigging the lottery: Making all tickets winners.\" ICML 2020.  \nYuan et al. \"Mest: Accurate and fast memory-economic sparse training framework on the edge.\" NeurIPS (2021).   \nDettmers et al. \"Sparse networks from scratch: Faster training without losing performance.\" arXiv:1907.04840 (2019).\n\n"}, {"review_id": "2rpLU1HcDF", "reviewer": "Reviewer_xwEQ", "summary": "This paper does a large scale study of dynamic sparse training, primarily on vision datasets. For a variety of models and hyperparameter settings, pruning using magnitude is found to be as or more effective when compared to pruning using more complex criteria proposed in the literature. The performance findings are reinforced by a study of the weights different pruning criteria select, and the impact of topology update frequency is studied. ", "questions": "1. In your related work, it is probably worth mentioning that Frankle et al. (2021) did an analysis of similar pruning criteria for static sparse training, also finding that magnitude was as good as more complex criteria. \n\n2. Line 266: An alternative explanation might be that removing a smaller amount of weights leads to less disruption of the batch norm statistics with non-magnitude criteria. After pruning, do you update the batch normalization statistics (e.g., by running the model on the training data without updating its weights, see page 16 of [Zimmer et al., 2023](https://arxiv.org/pdf/2306.16788.pdf))? I ask because removing small magnitude weights may have little effect on these statistics, but removing larger weights (e.g., because their gradients were small) could have more substantial effects on the relevance of the old batch norm statistics to the new model. In which case, it would be difficult to conclude that the criterion is responsible for a certain performance level as that performance level could be attributable (at least in part) to incorrect batch statistics. It might be worth running an experiment to check this (e.g., on EfficientNet).\n\n3. The networks considered cover a broad range but leave out architectures that have become more relevant in recent years. Also, the paper claims to cover large scale convolutional networks, but the largest network is a ResNet-50. Consider running on larger models and different architecture styles, e.g. ViTs. \n\n4. Figures are too far from where they are discussed in the text. Consider adjusting their placement.\n\n5. Lines 163-169 might benefit from being rephrased. As written, they do not provide a clear intuition for the expectation that a gradient based method would do better, which is what I think you are offering in these lines. In contrast, lines 184-185 (in conjunction with lines 177-178) give clear intuition for why I might expect a gradient based method to perform better. \n\n6. Lines 188-189: wouldn't the mask solutions have to be \"genuinely different\" if the performances associated with the different criteria differed? I would say the prior experiments \"do not necessarily indicate\" instead of \"do not indicate\".\n\n7. Line 270 and Figure 5 clash. You say that pruning adds regularization, but Figure 5 shows the dense model has the largest training loss - validation loss. Maybe you meant to define the loss gap as val - training instead of training - val.", "limitations": "The identified limitations are appropriate.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality:\n\n1. This paper provides the most comprehensive study of DST pruning criteria that I am aware of, providing a valuable insight about the relevance of the magnitude criterion.\n\nQuality:\n\n2. The experiments are designed well and support the contention that magnitude is a strong criterion in a variety of DST scenarios.\n\nClarity:\n\n3. The submission is well written.\n\nSignificance:\n\n4. The performance comparisons of criteria that have been shown to be effective in the literature (e.g. MEST and magnitude) are valuable for the community.\n", "weaknesses": "This paper does not have any major weaknesses. I believe it's significance to the community (and correspondingly my score) would be best enhanced by ensuring the experiments cover a broad range of relevant architectures. I address this and some more minor issues, along with my suggestions, below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer xwEQ**\n\nWe appreciate the feedback from the Reviewer and the positive comments about both the originality and the value of our work. We are happy to see that the Reviewer finds our experiments well-designed and supportive of our main claims. We were also glad to learn that the Reviewer does not see any major weaknesses in our work. We address the remaining questions of the Reviewer below:\n\n1. *In your related work, it is probably worth mentioning that Frankle et al. (2021)*  - For clarity, did the Reviewer mean the Pruning neural networks at initialization: Why are we missing the mark?.\" arXiv:2009.08576, ICLR (2021) paper? If so, it is indeed an important and related study, that contrary to our setup focuses on the **static** sparse pruning. Thank you for raising our attention to it, we will update the Related Work section accordingly. \n\n2. *Line 266: An alternative explanation might be that…* - Thank you for raising our attention to this work. We agree that the bach normalization statistics may influence the pruning during evaluation. Please note, however, that in DST we always prune during training, where the batch normalization statistics are not used. During training, the normalization is computed on the statistics directly from the mini-batch (see Algorithm 1 of Ioffe & Szegedy 2015, and the documentation of TensorFlow and PyTorch - links in the references below). The running normalization statistics only become relevant when we compute the final evaluation accuracy on the test set. But before that happens, the model still uses the iterations between the last mask update and the end of the training to update the running statistics. Therefore at the time of evaluation, they should already be adjusted according to the new mask structure.\n\n3. *The networks considered cover a broad range but…* - We have added a task on a transformer architecture (ROBERTa Large, ~354M Parameters) on the CommonsensQA dataset. Please refer to the “Joint Response” for details. There we also explain why we initially focused on convolutional and MLP-styled architectures. \n\n4. *Figures are too far from where they are discussed…* - Thank you, we will fix that. \n\n5. *Lines 163-169 might benefit from being rephrased…* - Thank you for noticing that. We will rephrase it to: “Since the gradient may provide information on how the weight will change in the future, it indicates the trend the weight might follow in the next optimization updates. Therefore it may be more suitable in the DST approach, where connectivity is constantly evolving and hence the insights into future updates may be beneficial”. We hope this clarifies the statement.\n\n6. *Lines 188-189: wouldn't the mask solutions have to be \"genuinely different\" if the performances associated with the different criteria differed? …* -  Not necessarily. Performance (i.e., accuracy) is only a measure of the end result of the model. A large difference in performance does not have to imply (a priori) a large difference in the mask (e.g., hypothetically, the mask could differ in only a few connections, but those connections could be very significant to the performance). Without studying to what extent the masks of different pruning criteria differ (but considering only performance), we are not able to identify such a situation.\n\n7. *Line 270 and Figure 5 clash …*  - Thank you for catching that. We had a typo in the text. Yes, Figure 5 shows the validation loss - training loss. \n\nWe have to the best of our effort, addressed all the concerns raised by the Reviewer and followed the suggestions to include more contemporary and large architectures by adding the ROBERTa Large model. We hope that our answers provide the necessary clarifications needed to enhance our paper. We would greatly appreciate if the Reviewer could re-evaluate our work taking into account our response and the fact that the Reviewer finds that there are no major weaknesses in our paper. \n\nReferences:\n\nFrankle, Jonathan, et al. \"Pruning neural networks at initialization: Why are we missing the mark?.\" arXiv:2009.08576, ICLR (2021).    \nIoffe, Sergey, and Christian Szegedy. \"Batch normalization: Accelerating deep network training by reducing internal covariate shift.\" International conference on machine learning. pmlr, 2015.    \ntensorflow batch norm: https://www.tensorflow.org/api_docs/python/tf/keras/layers/BatchNormalization     \npytorch batch norm: https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html.    "}, {"review_id": "ajw0AHqP8N", "reviewer": "Reviewer_mfmv", "summary": "This paper performs a systematic study of pruning strategies for dynamic sparse training (DST) methods, comparing their performance and structural decisions across backbone architectures, datasets, structural change frequencies, connection densities, and batch sizes.", "questions": "\nIn addition to the batch size studies, I would be interested to see (in this or future work) how much the batch used for gradient evaluation affects the connections selected for pruning and/or growth in gradient-based pruning/growth methods. This could be studied by performing the training as normal until immediately before the 1st pruning step, then repeatedly resampling a batch and performing a backwards pass (without updating weights) in order to record which connections would be pruned and which would be grown.\n\nIs Figure 2 aggregated over multiple trials per protocol? If so, please add error bars. If not, please run multiple trials.\n\nIs there any way to add error intervals to Figure 6? I like the graphical presentation but would also appreciate error intervals to determine how noisy the metrics are and how significant the differences between strategies are.\n\nTypos and tips (not considered negatively against paper):\n* Line 13: “cautions”?\n* Footnotes referring to appendices are not necessary: placing the references directly in the main text may save space.\n* The paper layout and figure placement was a bit hard to follow. Figures that span across the whole width seem to be placed rather randomly, while they are conventionally placed at the top of a page. This helps the reader read the main text more continuously and find the appropriate figure when referenced. \n* Many captions ended with an analysis of the figure, like the last two sentences of the caption of Figure 1: I would prefer to see this in the main text rather than the caption.\n* The Jaccard index equation could be moved to an appendix to save space.\n", "limitations": "Limitations and broader impacts are explicitly addressed. \n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: This work is the first systematic study on DST methods, combining various pruning and growing strategies that have thus far only been studied in isolation. It is also the first work to structurally compare the results of various methods. It does not introduce any new DST strategies, although it adapts a well-known pruning metric (SNIP) for the DST context and makes novel combinations of known DST strategies.\n\nQuality: This paper backs up claims with adequate experimentation and analysis, although the scope is limited to mostly image classification tasks (plus one tabular dataset). \n\nClarity: The paper is easy to understand overall, with a clear writing style. \n\nSignificance: This paper provides a significant study on the interplay not only of growing and pruning strategies but also factors such as backbone architecture, dataset, structural change frequency, connection density, and batch size.\n", "weaknesses": "I think random pruning (paired both with random or informed gradient-based growth) would be an interesting baseline in addition to dense and statically sparse architectures. \n\nAs noted by the authors, expanding to tasks beyond image classification (and one tabular dataset) would broaden the scope of this work.\n\nYou claim that growing is more thoroughly studied than pruning, but I’m not sure this is true. In Line 40, Dettmers and Zettlemoyer (2019) is cited as emphasizing the growth criterion but rather, to my understanding, uses random growth with informed pruning. There seems to be a roughly equal number of cited DST works that use simple pruning with more informed growing to those using simple growing with more informed pruning. This paper still has significance by focusing on the pruning method (although more growing methods could be included, but this would quickly increase the experimentation cost), but I think this language should be clarified. \n\nThe analysis on update period in Section 5.3 is interesting, but I think should be qualified by noting how the pruning factor schedule and batch size may be confounding factors. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response to Reviewer mfmv**\n\nWe thank the Reviewer for the review. We are very happy to see that the Reviewer appreciates the originality and significance of our work, recognizing it as the first systematic study on pruning criteria in DST methods. Furthermore, we are glad to hear our experimental evaluation and analysis have been acknowledged as adequate. We were mostly thrilled to receive such positive feedback. In the following, we address the remaining questions raised by the Reviewer:\n\n*I think random pruning (paired both with random or informed gradient-based growth) would be an interesting baseline…* - We have observed that random pruning paired either with random or gradient growth performs much worse than all of the other criteria and baselines (we have added a visualization of that effect on ResNet56 on CIFAR-10 in Figure R3 in the rebuttal PDF). The performance drops especially dramatically for the very low densities, making it a less interesting baseline. While random may be a quite reasonable choice for the growth (due to influencing the exploration), in pruning it effectively leads to removing arbitrary connections in a trained network.   For the above reasons, we initially decided to use only dense and static sparse baselines. We will add the random performance for all the Figures for the camera-ready version.  \n\n*As noted by the authors, expanding to tasks beyond image classification (and one tabular dataset) would broaden the scope of this work…* - We have enhanced our empirical evaluation with results on the ROBERTa Large model (~354M parameters) on the CommonsenseQA text dataset. We kindly ask the Reviewer to refer to the “Joint Response” for details. \n\n*You claim that growing is more thoroughly studied than pruning, but I’m not sure this is true…* - Thank you for this comment. Indeed, that phrase can take benefit of some clarifications. What we intended to convey is that pruning criteria have not ever been rigorously compared in DST. At the same time, the interplay between some growth criteria is rather better understood (as demonstrated, for example, by Evci et al., 2019). Additionally, we would like to kindly note that we believe that Dettmers and Zettlemoyer (2019) do, in fact, employ magnitude pruning alongside momentum-based growth (as evidenced by Algorithm 1 in their paper). We think rephrasing lines 38-41 from our paper to \"Much of the research concerning design choices in DST focuses on the analysis of methods where variations occur in the growth criterion or the pruning criterion becomes intertwined with other design considerations [3, 10, 9, 1] (...).\" will clarify our point. We hope that this adjustment effectively addresses the concerns raised by the Reviewer. \n\n*The analysis on update period in Section 5.3 is interesting, but I think should be qualified by noting how the pruning factor schedule and batch size may be confounding factors.* - Thank you for raising this point. The update period, when considered together with the batch size, can be interpreted as the number of data samples the model sees (and trains on) before performing the mask update. We will adjust the text in the Camera Ready version of the paper to make sure it includes that information. In consequence, we would expect that once we increase the batch size, the best choice of the update period shifts to lower values, keeping the total number of seen examples similar. We have run a quick comparison on the MLP-CIFAR10 setup and observed this behavior (see Figure R1c and R1d in the rebuttal PDF). Regarding the pruning factor schedule: we again run a quick comparison on the MLP-CIFAR10 setup, and try two different schedulers, apart from the default cosine one: the linear one, and the constant one. The linear schedule multiplies every 600 optimization steps the current decay rate by a constant factor (0.99). The constant schedule simply keeps the prune fraction fixed at 0.5 throughout the whole training. The results are in Figures R1a, R1b, and R1c in the Rebuttal PDF. We observe that the choice of the pruning factor schedule does not significantly influence the best value of the update period. We will add those results to the Appendix. \n  \n**Questions:**\n\n*I would be interested to see (in this or future work) how much the batch used for gradient evaluation affects the connections selected for pruning and/or growth in gradient-based pruning/growth methods...* - We have run for the rebuttal a quick experiment where we only increase the batch size for the DST update (from 128 to 1024). The results are in Figure R5 in the Rebuttal PDF. We observe that in general, it does not enhance the performance (and works significantly worse with gradient growth than with random growth for the MLP-CIFAR10). We consider the study on the interplay between DST update batch size and the model performance as an interesting direction for future work.   \n\n*Is Figure 2 aggregated over multiple trials per protocol?* - Yes, we have run 3 experiments in each setting, and we did include the error bars in the Figure  - they are just very small (all the methods perform very stable). We apologize, we will enlarge them in the Figure to make them more visible.\n\n*Is there any way to add error intervals to Figure 6?* - Thank you, this is a valid point. We will add the error intervals to the Appendix. \nTypos and tips (not considered negatively against the paper): Thank you for catching the typos and providing the tips. We will enhance our paper accordingly. \n\nWe once again appreciate the Reviewer’s favorable rating and constructive feedback, helping us to further enhance the quality of the paper."}, {"review_id": "O1Y7xBLgOq", "reviewer": "Reviewer_3Q28", "summary": "The paper conducts an extensive empirical study including several popular pruning criteria and analyzes their impact on the DST framework on diverse models. They found that within a stable DST hyperparameter setup, the majority of the studied criteria\nperform similarly, regardless of the model architecture and the selected growth criterion. In addition, in very sparse regimes, the simplest magnitude-based pruning methods surpass any more fancy choices.", "questions": "The paper's readability will benefit by including important hyperparameters like prune and growth ratios etc in tabular format in the main draft. When written with experimental discussion, it becomes difficult to follow up.", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and the key research questions examined are well-supported by extensive experiments and appendix results.\n2. Experiments related to how the frequency of topology updates influences the effectiveness of different pruning methods are interesting.\n", "weaknesses": "The primary concern related to the submission is the lack of novelty. Although some observations will be very beneficial to the sparsity community, the supremacy of magnitude-based weight removal in the high sparsity range is not novel. I believe extending this paper beyond small-scale models, to mid-to-large-scale models will make the paper more empirically strong and relevant in the current state of deep learning. In addition, some analysis of growth trajectory, connections which were never removed or always remained pruned across different pruning staretgies will add more value and uplift the quality of the work. Overall, it is still a strong empirical paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Response Reviewer 3Q28**\n\nWe are grateful to the Reviewer for the thoughtful comments and feedback. We are pleased to hear the reviewer considered our research questions well-examined by the extensive experiments we performed. We are also thrilled to see that the Reviewer found our experiments on the topology updates interesting and recognizes our work as a strong empirical paper. Below we address the raised concerns:\n\n*The primary concern related to the submission is the lack of novelty.* - We respectfully ask the reviewer to recognize the diverse possibilities for novelty within scientific research.  While introducing new algorithms to address problems is undoubtedly one facet of it, comprehending existing solutions holds equal significance. Our thorough and comprehensive study encompasses novelty by asking questions that have not been previously studied, such as to what extent the performance of DST is guided by the pruning criterion, and how the pruning criterion interacts with the choices of DST hyperparameters. \n\nWe kindly encourage the Reviewer to consider that the significance of a weight (and thus the pruning criterion) can diverge from that in standard post-training pruning due to the parameter changes inherent in the DST framework (elaborated upon in lines 43-46 of the paper). Therefore it could be potentially dangerous to transfer the knowledge from standard (or static) pruning into DST without careful consideration. It is also worth noting that no definitive superiority of the magnitude criterion for DST has been previously firmly established, considering the emergence of various pruning criteria for DST in recent years. Our research offers valuable insights into the DST framework, suggesting that updates need not occur at such high frequencies and that more intricate pruning criteria do not necessarily outperform simpler magnitude-based ones. The originality and novelty of our work have been acknowledged by Reviewers xwEQ and mfmv. Therefore, we earnestly appeal to the Reviewer to assess our work within the broader context of innovation. \n\n*I believe extending this paper beyond small-scale models, to mid-to-large-scale models …* - We consider the VGG-16, ResNet-50, and Efficient-Net models as large enough to consider them at least “mid-scale” models (all models have above 10M parameters, with ResNet50 having 25M parameters). Both ResNets and Efficient-Nets are also commonly used in modern research, and we have chosen them in our evaluations because they have been included as the largest models in most of the works that used the studied pruning criteria (see Yuan et al. 2021, Evci et al. 2019). However, following the Reviewer’s request, we also include an evaluation on the ROBERTa-Large model (354M parameters) on the CommonsenseQA text dataset (please see the “Joint Response” for more details).\n\n*In addition, some analysis of growth trajectory ...* - Thank you for this suggestion. In order to improve our work, we provide additional statistics on the never removed connections. We consider the masks at the end of training for the MLP, CNN, and ResNet-56 models on CIFAR10. Then we measure the number of weights that are always retained by every pruning criterion for the same seed and we divide it by the number of all remaining (unpruned) weights in the model. We do the same on the negation of the masks to get the number of the always removed weights and divide it by the number of all removed weights of the model. We present the results in the Rebuttal PDF in Figure R4. For the MLP model around 23% of all the retained weights are shared across all the models. For CNN, this number is higher and goes up to 37%, while for the ResNet-56 it is equal to 11%. In general, the smaller overlap of the pruning methods for the ResNet model is also something that we observe in the main paper in Figure 6b in the main text. Additionally, we would like to note that in Figure 9 in Appendix I we report how the Jaccard Index between the sets of weights selected for removal changes during the training - perhaps the Reviewer will also find this plot interesting. \n\n**Questions:**\n\nThe paper's readability will benefit by … - Thank you, we will add the below table to the Appendix: \n\n| Experiment | Prune Fraction  | Prune Fraction Decay Scheduler | Density                                                   \t| Update Period                          \t|\n|------------|-----------------|--------------------------------|---------------------------------------------------------------|--------------------------------------------|\n| Fig. 1   | 0.5         \t| cosine                     \t| 0.05, 0.06, 0.07, 0.08,  0.09, 0.1, 0.15, 0.2,  0.3, 0.4, 0.5 | 800                                    \t|\n| Fig. 2   | 0.5         \t| cosine                     \t| 0.2                                                       \t| 800                                    \t|\n| Fig. 4   | 0.5         \t| cosine                     \t| 0.2, 0.1                                                  \t| 25, 50, 100, 200, 400, 800, 1600,3200,6400 |\n| Fig. 5   | 0.5         \t| cosine                     \t| 0.5                                                       \t| 800                                    \t|\n| Fig. 6a  | 0.5         \t| cosine                     \t| 1.0                                                       \t| 800 (measured  only once)              \t|\n| Fig. 6b  | 0.5         \t| cosine                     \t| 0.2                                                       \t| 800                                    \t|\n\nOnce again we grateful for the feedback provided by the Reviewer and we hope that our responses have addressed the raised concerns. We kindly ask the Reviewer to reconsider adjusting the score accordingly.  \n\nReferences:    \nEvci, Utku, et al. \"Rigging the lottery: Making all tickets winners.\" ICML. PMLR, 2020.  \nYuan, Geng, et al. \"Mest: Accurate and fast memory-economic sparse training framework on the edge.\" NeurIPS 34 (2021). "}]

Author rebuttal:
**JOINT RESPONSE**

We thank all the Reviewers for the time and effort taken to provide valuable insights and comments on our work. We are very glad that our research has been recognized as useful and beneficial to the community (Reviewers Q7vN, xwEQ, mfmv). Moreover, Reviewers xwEQ and mfmv appreciated the novelty and originality of our work, recognizing our comprehensive study of the DST pruning criteria. Our empirical evaluations have been praised as being thorough (Reviewers Q7vN, aJDG), well-designed, and supportive of our main claims (Reviewers mfmv, 3Q28, xwEQ).  We are also pleased to learn that the reviewers found our paper to be well-written and easy to read (Reviewers aJDG, xwEQ, mfmv, 3Q28) and did not find any major weaknesses (Reviewer xwEQ). Overall, we are delighted to receive such an encouraging response from the Reviewers regarding the aforementioned aspects of our research.

We have noted a recurrent comment from the Reviewers, suggesting an extension of our results using transformer models and/or text data. In response to this feedback, we have conducted an additional evaluation of the pruning criteria on the fine-tuning task of ROBERTa Large (Liu et al., 2019) with approximately 354 million parameters, utilizing the CommonsenseQA dataset (Talmor et al., 2018) adapted from the Sparsity May Cry (SMC) Benchmark (Liu et al., 2023). The corresponding results are presented in Table R1 within the rebuttal PDF.

We have used the exact same setup as in the SMC-Benchmark, by first performing magnitude pruning for the sparse initialization and then using DST during the fine-tuning phase.  We also use the same hyperparameters. Please note that this is a different configuration from the one used in our paper, in which we trained all the models from scratch (in spirit, the CommonsenseQA study is more similar to the EfficientNet fine-tuning experiment from Appendix G). Similar to the findings of Liu et al. (2023), we observed that in this scenario, the achievable sparsity without a significant performance decline is notably lower than in vision or tabular data scenarios. Concerning random growth, the MEST criterion appears to be the most suitable choice. For gradient growth with a density of 0.8, the RSensitivity and SET criteria initially demonstrate strong performance but are eventually surpassed by the magnitude and MEST criteria at a density of 0.7. Additionally, we noted a significantly higher variance in the outcomes across all criteria compared to vision tasks. Furthermore, we conducted a brief exploration of update period values for this problem (refer to Figure R2 in the Rebuttal PDF). Notably, we observed that overly frequent updates do not yield beneficial results. Concerning gradient growth, the most effective update period value is consistently around Δt = 500 for nearly all criteria. For random growth higher update period values generally lead to improved outcomes.

We will include these results in the paper. However, we would like to emphasize that our work already contains nine different model-dataset pairs, with model sizes varying from ~72K to ~25M parameters. We believe that our more than 7000 experiments are already a strong basis that allows us to make reasonable conclusions about the impact of the pruning criteria in DST. At the same time please note that the studied pruning criteria had been evaluated either on tabular or vision datasets in most of the works that have introduced them or used them (e.g., Mocanu et al. 2018, Yuan et al. 2021, Naji et al. 2021, Evci et al. 2020). Large language models typically have a different structure and are based on the attention mechanism, for which the applicability of sparse training is still an open area of research (Liu et al. 2023, Sun et al. 2023). This motivated our initial choice of fixing our focus on tabular and vision models.  

For detailed responses to the individual concerns raised by each reviewer, please refer to our separate comments posted in response to their reviews. We have made a significant effort to address each issue and query raised by the reviewers and have included results for every suggestion made to improve our empirical evidence. 

References:  

Liu, Shiwei, et al. "Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!." arXiv:2303.02141, ICLR (2023).   
Sun, Mingjie, et al. "A Simple and Effective Pruning Approach for Large Language Models." arXiv preprint arXiv:2306.11695 (2023).  
Liu, Yinhan, et al. "Roberta: A robustly optimized bert pretraining approach." arXiv preprint arXiv:1907.11692 (2019).  
Talmor, Alon, et al. "Commonsenseqa: A question answering challenge targeting commonsense knowledge." arXiv preprint arXiv:1811.00937 (2018).  
Mocanu et al. "Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science." Nature communications 9.1 (2018): 2383.  
Evci, Utku, et al. "Rigging the lottery: Making all tickets winners." International Conference on Machine Learning. PMLR, 2020.  
Yuan, Geng, et al. "Mest: Accurate and fast memory-economic sparse training framework on the edge." Advances in Neural Information Processing Systems 34 (2021): 20838-20850.  
Naji, Seyed Majid, Azra Abtahi, and Farokh Marvasti. "Efficient Sparse Artificial Neural Networks." arXiv preprint arXiv:2103.07674 (2021).  


Meta-review: This paper provides an empirical survey of pruning and growing criteria in the dynamic sparse training framework. Notably, the simple technique of magnitude-based pruning works well. While some reviewers were concerned that this worked lacked novelty due to its focus on empirically comparing existing methods, this is not a valid concern because the paper provides a meaningful and useful contribution in the form of better understanding of the field.
Decision: Accept (poster)