The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training
Authors: Shiwei Liu
Tianlong Chen
Xiaohan Chen
Li Shen
Decebal Constantin Mocanu
Zhangyang Wang
Mykola Pechenizkiy
Venue: ICLR 2022 Poster (2022)
Keywords: random pruning
sparse training
static sparse training
layer-wise sparsities
dynamic sparse training

Abstract: Random pruning is arguably the most naive way to attain sparsity in neural networks, but has been deemed uncompetitive by either post-training pruning or sparse training. In this paper, we focus on sparse training and highlight a perhaps counter-intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks. Without any delicate pruning criteria or carefully pursued sparsity structures, we empirically demonstrate that sparsely training a randomly pruned network from scratch can match the performance of its dense equivalent. There are two key factors that contribute to this revival: (i) $the network sizes matter$: as the original dense networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios; (ii) $appropriate layer-wise sparsity ratios$ can be pre-chosen for sparse training, which shows to be another important performance booster. Simple as it looks,  a randomly pruned subnetwork of Wide ResNet-50 can be sparsely trained to outperforming a dense Wide ResNet-50, on ImageNet. We also observed such randomly pruned networks outperform dense counterparts in other favorable aspects, such as out-of-distribution detection, uncertainty estimation, and adversarial robustness. Overall, our results strongly suggest there is larger-than-expected room for sparse training at scale, and the benefits of sparsity might be more universal beyond carefully designed pruning. Our source code can be found at https://github.com/VITA-Group/Random_Pruning.


Published as a conference paper at ICLR 2022
THE UNREASONABLE EFFECTIVENESS OF RANDOM PRUNING: RETURN OF THE MOST NAIVE BASELINE FOR SPARSE TRAINING
Shiwei Liu', Tianlong Chen', Xiaohan Chen', Shen
Decebal Constantin Mocanu 14, Zhangyang Wang', Mykola Pechenizkiy'.
Eindhoove University of Technology. UUniversity of Texas at Austin IJD Explore Academy, University of Twente, www.pppccheeiieyyyttee.....
[tianleng. chen, xiaohan chen, at laswang] a Autexas.ede d. c. www.mmuuutennte .n., mathshen gfgmaill...........mm...mm com
ABSTRACT
Random pruning is arguably the most naive way to attain sparsity in neural net- works, but has been deemed uncompetitive by either post-training pruning or sparse training. In this paper, focus sparse training and highlight per- haps counter-intuitive finding, that random pruning at initialization can be quite powerful for the sparse training of modern neural networks. Without any delicate pruning criteria or carefully pursued sparsity structures, we empirically demon- strate that sparsely training randomly pruned network from scratch can match the performance of its dense equivalent. There are two key factors that con- tribute this revival: (i) the network sizes matter: as the original dense networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios; (ii) appropriate layer-wise sparsity ratios can be pre-chosen for sparse training, which shows to be another important performance booster. Simple it looks, randomly pruned subnetwork of Wide ResNet-50 can be sparsely trained to outperforming a dense Wide ResNet-50, on ImageNet. We also ob- served such randomly pruned networks outperform dense counterparts in other favorable aspects, such as out-of-distribution detection, uncertainty estimation, and adversarial robustness. Overall, our results strongly suggest there is larger-than- expected room for sparse training at scale, and the benefits of sparsity might be more universal beyond carefully designed pruning. Our source code can be found at https: / cwm/cI TA//VVIITAGOOOOA Batp///andddnaaeeeeeeeeee Prenonn_rrrinn...
INTRODUCTION
Most recent breakthroughs in deep learning are fairly achieved with the increased complexity of over-parameterized networks (Brown al., 2020; Raffel al., 2020: Dosovitskiy et al., 2021; Fedus et al., 2021. 10009101::3961 Jumper et al., 2021; Berner et al., 2019). It is well-known that large models train better (Neyshabur et al., 2019; Novak al., 2018; Allen-Zhu et al., 2019), generalize better (Hendrycks & Dietterich, 2019; Xie & Yuille, 2020; Zhao et al., 2018), and transfer better (Chen al., 2020b;a; 2021b). However, the upsurge of large models exacerbates the gap between research and practice since many real-life applications demand compact and efficient networks.
Neural network pruning, since proposed by (Mozer & Smolensky, 1989; Janowsky, 1989), has evolved as the most common technique in literature to reduce the computational and memory requirements of neural networks. Over the past few years, numerous pruning criteria have been proposed, including magnitude (Mozer & Smolensky, 1989; Han et al., 2015; Frankle & Carbin, 2019; Mocanu et al., 2018), Hessian (LeCun et al., 1990; Hassibi & Stork, 1993), mutual information (Dai et al., 2018), Taylor expansion (Molchanov et al., 2016). movement (Sanh et al., 2020), connection sensitivity (Lee et al., 2019), etc. Motivated for different scenarios, pruning can occur after training (Han et al., 2015; Frankle & Carbin, 2019; Molchanov al., 201 2016; Lee et al., 2021), during training (Zhu & Gupta,

---

Published as a conference paper at ICLR 2022
2017; Gale al., 2019; Louizos al., 2018; You et al., 2020; Chen et al., 2021c;a), and even before training (Mocanu et al., 2018; Lee et al., 2019; Gale et al., 2019; Wang et al., 2020; Tanaka et al., 2020). The last regime can be further categorized into "static sparse training" (Mocanu et al., 2016; Gale et al., 2019; Lee et al., 2019; Wang et al., 2020) and "dynamic sparse training" (Mocanu et al., 2018; Bellec et al., 2018; Evci al., 2020a; Liu et al., 2021b:a).
While random pruning is a universal method that can happen at any stage of training, training a randomly pruned network from scratch is arguably the most appealing way, owing its "end-to-end" saving potential for the entire training process besides the inference. Due to this reason, we focus on random pruning for sparse training this paper. When new pruning approaches bloom, random pruning naturally becomes their performance's 's empirical "lower bound" since the connections are randomly chosen without any good reasoning. Likely due to the same reason, the results of sparse training with random pruning (as "easy to beat" baselines to support fancier new pruning methods) reported in the literature are unfortunately vague, often inconsistent, and sometimes casual. For instance, it is found in Liu et al. (2020b) that randomly pruned sparse networks can be trained from scratch to match the full accuracy of dense networks with only 20% parameters, whereas around 80% parameters are required to do so in Frankle et al. (2021). The differences may arise from architecture ratios, and so on.
choices, training recipes, distribution hyperparameters/layert
In most pruning literature (Gale et al., 2019; Lee et al., 2019; Frankle al., 2021; Tanaka et al., 2020), random pruning usually refers to randomly removing the same proportion of parameters per layer, ending up with uniform layer-wise sparsities. Nevertheless, researchers have explored other pre-defined layer-wise sparsities, e.g., uniform+ (Gale et al., 2019), Erdös-Rényi random graph (ER) (Mocanu et al., 2018), and Erdös-Rényi-Kens wise sparsities also fit the category of random pruning, as they require no training to obtain the corresponding sparsity ratios. We assess random pruning for sparse training with these layer-wise sparsity ratios, in terms of various perspectives besides the predictive accuracy.
(ERK) (Evci al., 2020a). These layer-
Our main findings during this course of study are summarized below:
We find that the network size matters for the effectiveness of sparse training with random pruning. With small networks, random pruming hardly matches the full accuracy even at mild sparsities (10%, 20%). However, as the networks grow wider and deeper, the performance of training randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios.
We further identify that appropriate layer-wise sparsity ratios can be an important booster for training randomly pruned network from scratch, particularly for large networks. We investigate several options to pre-define layer-wise sparsity ratios before any training; one of them is able to push the performance of completely random sparse Wide ResNet-50 over the densely trained Wide ResNet-50 on ImageNet.
We systematically assess the performance of sparse training with random pruning, and observe surprisingly good accuracy and robustness. The accuracy achieved by ERK ratio can even surpass the ones learned by complex criteria, e.g., SNIP or GraSP. In addition, randomly pruned and sparsely trained networks are found to outperform conventional dense networks in other favorable aspects, such as out-of-distribution (OoD) detection, adversarial robustness, and uncertainty estimation.
RELATED WORK
2.1 STATIC SPARSE TRAINING
Static sparse training represents a class of methods that aim to train sparse subnetwork with fixed sparse connectivity pattern during the course of training. We divide the static sparse training into random pruning and non-random pruning according to whether the connection is randomly selected.
Random Pruning. Static sparse training with random pruning samples masks within each layer in a random fashion based on pre-defined layer-wise sparsities. The most naive approach is pruning each layer uniformly with the same pruning ratio, i.e., uniform pruning (Mariet & Sra, 2016; He et al., 2017; Suau et al., 2019; Gale et al., 2019). Mocanu et al. (2016) proposed a non-uniform and scale-free topology, showing better performance than the dense counterpart when applied to restricted
2

---

Published as a conference paper at ICLR 2022
Boltzmann machines (RBMs). Later, expander graphs were introduced to build sparse CNNs and showed comparable performance against the corresponding dense CNNs (Prabhu et al., 2018; Kepner & Robinett, 2019). While not initially designed for static sparse training, ER (Mocanu et al.. 2018) and ERK (Evci et al., 2020a) are two advanced layer-wise sparsities introduced from the field of graph theory with strong results.
Non-Randond Pruning. Instead of pre-choosing a sparsity ratio for each layer, many works utilize the proposed saliency criteria to learn the layer-wise sparsity ratios before training, also termed as pruning at initialization (Pal). Lee et al. (2019) first introduced SNIP that chooses structurally important connections at initialization via the proposed connection sensitivity. Following SNIP, many efficient criteria have been proposed to improve the performance of non-random pruning at initialization, including but not limited to gradient flow (GraSP; Wang et al. (2020)), synaptic strengths (SynFlow; Tanaka et al. (2020)), neural tangent kernel (Liu & Zenke, 2020), and iterative SNIP (de Jorge et al., 2021; Verdenius et al., 2020). Su et al. (2020); Frankle et al. (2021) uncovered that the existing Pal methods hardly exploit any information from the training data and are very robust to mask shuffling, whereas magnitude pruning after training learns both, reflecting a broader challenge inherent to pruning at initialization.
2.2 DYNAMIC SPARSE TRAINING
In contrast to static sparse training, dynamic sparse training stems from randomly initialized sparse subnetworks, and meanwhile dynamically explores new sparse connectivity during training. Dynamic sparse training starts from Sparse Evolutionary Training (SET) (Mocanu et al., 2018; Liu et al., 2020a) which initializes the sparse connectivity with Erdōs-Rémy (Erdōs & Rényi. 1959) topology and peri- odically explores the sparse connectivity via a prune-and-gres grow scheme during the course of training. While there exist numerous pruning criteria in the literature, simple magnitude pruning typically performs well in the field of dynamic sparse training. On the other hand, the criteria used to grow weights back differs from method to method, including randomness (Mocanu al., 2018; Mostafa & Wang, 2019), momentum (Dettmers& & Zettlemoyer, 2019), gradient (Evci et al., 2020a: Jayakumar al., 2020; Liu et al., 2021b). Besides the prune-and-grow scheme, layer-wise sparsities are vital to achieving high accuracy. (Mostafa & Wang, 2019; Dettmers & Zettlemoyer, 2019) reallocates weights across layers during training based on reasonable heuristics, demonstrating performance improvement. Evci et al. (2020a) extended ER to CNNs and showed considerable performance gains to sparse CNN training with the Serris-Reni--eeeeeee (2021a) started from a subnetwork at smaller sparsity and gradually pruned it the target sparsity during training. The denser initial subnetwork provides a larger exploration space for DST at the early training phase and thus leads to performance improvement, especially for extreme sparsities. Even though dynamic sparse training achieves promising sparse training performance, it changes the sparse connectivity during training and is thus out of the scope of random pruning.
(ERK) ratio. Very recently, Liu et al.
While prior works have observed that random pruning can be more competitive in certain cases (Mo- al., 2018; Liu al., 2020b; Su et al., 2020; Frankle al., 2021), they did not give principled guidelines on when and how it can become that good; nor do they show it can match the performance of dense networks on ImageNet. Standing on the shoulders of those giants, our work summarized principles by more extensive and rigorous studies, and demonstrate the strongest result so far, that randomly pruned sparse Wide ResNet-50 can be sparsely trained to outperform a dense Wide ResNet- 50, on ImageNet. Moreover, compared with the ad-hoc sparsity ratios used in (Su et al., 2020), we show that ERK (Evei et al., 2020a) and our modified ERK+ are more generally applicable sparsity ratios that consistently demonstrate competitive performance without careful layer-wise sparsity design for every architecture. Specifically, ERK+ ratio achieves similar accuracy with the dense Wide ResNet-50 on ImageNet while being data free, feedforward free, and dense initialization free.
METHODOLOGY
We conduct extensive experiments to systematically evaluate the performance of random pruning. The experimental settings are described below.
3.1 LAYER-WISH SPARSITIES RATIOS
Denote s as the sparsity of layer 1. Random pruming, namely, removes weights or filters in each layer randomly to the target sparsity s!. Different from works that learn layer-wise sparsities and model
3

---

Published as a conference paper at ICLR 2022
weights together during training using iterative global pruning techniques (Frankle & Carbin, 2019), soft threshold (Kusupati et al., 2020), and dynamic reparamcterization (Mostafa & Wang, 2019), etc., the wise sparsities of random pruning is pre-defined before training. Many pre-defined layer-wise sparsity ratios in the literature are suited for random pruning, while they may not initially be designed for random pruning. We choose the following 6 layer-wise sparsity ratios to study. SNIP ratio and GraSP ratio are two layer-wise ratios that we borrow from Pal.
ERK. Introduced by Mocanu et al. (2018), Erdös-Rényi (ER) sparsifies the Multilayer Perceptron (MLP) with a random topology in which larger layers are allocated with higher sparsity then smaller layers. Evci al. (2020a) further proposed convolutional variant Lendos-Rényi-Keeeee Kernel (ERK)) that takes the convolutional kernel into consideration. Specifically, the sparsity of the convolutional layer is scaled proportional to n²' n'=+tn' +w' +w'+ where n' refers to the number of neurons/channels in layer I; wt and h' are the corresponding width and height.
==- xn' x X
ERK+, We modify ERK by forcing the last fully-connected layer dense if it is not, while keeping the overall parameter count the same. Doing so improves the test accuracy of Wide ResNet-50 on ImageNet' shown in Appendix F.
Uniform. Each layer is pruned with the same pruning rate so that the pruned network ends up with a uniform sparsity distribution, e.g., Zhu & Gupta (2017).
Uniform+. Instead of using a completely uniform sparsity ratio, Gale et al. (2019) keep the first convolutional layer dense and maintain at least 20% parameters in the last fully-connected layer.
SNIP ratio. SNIP is a Pal method that selects important weights based on the connection sensitivity score Ig O w/ where w and g is the network weight and gradient, respectively. The weights with the lowest scores in one iteration are pruned before training. While not initially designed for random pruning, we adjust SNIP for random pruning by only keeping its layer-wise sparsity ratios, while discarding its mask positions. New mask positions (non-zero elements) are ra-sappled through a uniform distribution ~ Uniform(O, 1) with probability of 1 s Such layer-wise sparsities are obtained in (slightly) data-driven way. yet very efficiently before training without any weight update. The SNIP ratio is then treated as another pre-defined (i.e., before the training starts) sampling ratio for random pruning.
GraSP ratio. GraSP is another state-of-the- method seeking to pruning at initialization. Specif- ically, GraSP removes weights that have the least effect on the gradient norm based on the score of -w @ Hg. where H is the Hessian matrix and g is the gradient. Same with SNIP, we keep the layer-wise sparsity ratios of GraSP and ra-sample its mask positions.
3.2
EXPERIMENTAL SETTINGS
Table 1: Summary of the architectures, datasets and hyperparameters used in this paper.
Model
Mode
Data
#Epoch
Batch Size
LR
Momentum
IR Decay, Epoch
Weight Decay
Dense
CIFAR-IO0100
160
128
0.1
0.9
10x,180, 120]
0.0005
ResNets
Sparse
CIFAR-10/100
160
128
0.1
0.9
10x, [80, 120]
0.0005
Wide ResNets
Dense
ImageNet
90
192*4
0.4
0.9
10x, [30. 60, 80)
0.0001
Sparse
ImageNet
100
192*4
0.4
0.9
[30, 60,
0.0001
Architectures and Datasets. Our main experiments are conducted with the CIFAR version of ResNet (He et al., 2016) with varying depths and widths on CIFAR-10/11 (Krizhevsky et al.. 2009), the batch normalization version of VGG (Simonyan Zisserman, 2014) with varying depths on 10/AR-10/10) and the ImageNet version of ResNet and Wide ResNet-50 (Zagoruyko & Komodakis, 2016) on ImageNet (Deng et al., 2009). For ImageNet, we follow the common setting in sparse training (Dettmers& & Zettlemoyer, 2019; Evei et al., 2020b; Liu et al., 2021b) and train sparse models for 100 epochs. All models are trained with stochastic gradient descent (SGD) with momentum. We share the summary of the architectures, datasets, and hyperparameters in Table 1.
Measurement Metrics. In most pruning literature, test accuracy is the core quality that researchers consider. However, the evaluation of other perspectives is also important for academia and industry
Foor datasets with fewer classes, e.g, MNIST, CIFAR-10, ERK+ scales back to ERK as the last fully- connected layer already dense.
4

---

Published as a conference paper at ICLR 2022
ResNet-20
ResNet-32
ResNet-44
Accuracy aaaryyy.....
92
90
92
88
86
88
0.1
0.2
0.3
0.4
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Sparsity
Sparsity
Sparsity
ResNet-56
ResNet-110
92
92
90
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Sparsity
Sparsity
ERK
Uniform
Ratio
Figure 1: From shallow to deep. Test accuracy of training randomly pruned subnetworks from scratch with different depth on CIFAR-IO. ResNet-A refers to a ResNet model with A layers total.
ResNet-20-8
ResNet-20-16
ResNet-20-24
90
92
Accuracy
87
93
91
81
78
0.1
0.2
3.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9
ResNet-20-32
ResNet-20-40
ResNet-20-56
Accuracy
95
93
94
92
92
91
0.1
0.2 0.3
0.4 0.5 0.6 0.7 0.8 0.9 Sparsity
0.1
0.2 0.3 0.4 0.5 0.5 0.7 0.8 0.9 Sparsity
0.1 0.2
0.3 0.4 0.5
0.5 0.7 0.8 0.9 Sparsity
ERK
Uniform
Uniform+
SNIP Ratio
Dense
Figure 2: From narrow to wide. Test accuracy of training randomly pruned subnetworks from scratch with different width CIFAR-10. ResNet-A-B refers ResNet model with A layers in total and B filters in the first convolutional layer.
before replacing dense networks with sparse networks on a large scope. Therefore, we thoroughly evaluate sparse training with random pruning from a broader perspective. Specifically, we assess random pruning from perspectives of OoD performance (Hendrycks et al., 2021), adversarial robust- ness (Goodfellow et al., 2). and uncertainty estimation (Lakshminarayanann al.. 2016). See Appendix A for full details of the measurements used in this work.
4 EXPERIMENTAL RESULTS ON CIFAR
In this section, report the results of sparse training with random pruning CIFAR-10/100 under various evaluation measurements. For each measurement, the trade-off between sparsity and the corresponding metric is reported. Moreover, how the performance alters as the model size alters. The results are averaged over 3 runs. We report the results CIFAR-10 of ResNet in the main body of this paper. The results CIFAR-100 of
also alter the depth and width of the model to check
5

---

Published as a conference paper at ICLR 2022
ResNet, and CIFAR- 10010000000 of VGG shown in Appendix D and Appendix E, respectively. Unless otherwise stated, all the results are qualitatively similar.
4.1 PREDICTIVE ACCURACY OF RANDOM PRUNING ON CIFAR-10
We first demonstrate the performance of random pruning on the most common metric- test accuracy. To avoid overlapping multiple curves, we share the results of GraSP in Appendix The main observations are as follows:
① Performance of random pruning improves with the size of the network. We vary the depth and width of ResNet and report the test accuracy in Figure 1. When operating small networks, e.g., ResNet-20 and ResNet-32, we can hardly find matching subnetworks even at mild sparsities, i.e., 10%, 20%. With larger networks, e.g., ResNet-56 and ResNet-1 10, random pruning can match the dense performance at 60% ~ 70% sparsity. Similar behavior can also be observed when we increase the width of ResNet-20 20 in Figure 2. See Appendix B for results deeper models.
② The performance difference between different pruning methods becomes indistinct as the model size increases. Even though uniform sparsities fail to match the accuracy achieved by non- uniform sparsities (ERK and SNIP) with small models, their test accuracy raises to comparable level as non-uniform sparsities with large models, e.g., ResNet- 1 10 and ResNet-20-56.
3 Random pruning with ERK even outperforms pruning with the well-versed methods (SNIP, GraSP). Without using any information, e.g., gradient and magnitude, training randomly pruned subnetwork with ERK topology leads to expressive accuracy, even better than the ones trained with the delicately designed sparsity ratios, i.e., SNIP, and GraSP (shown in Appendix C).
4.2 BROADER EVALUATION OF RANDOM PRUNING ON CIFAR-10
In general, sparse training with random pruning achieves quite strong results on CIFAR-10 in terms of uncertainty estimation, OoD robustness, and adversarial robustness without any fancy techniques. We summary main observations as below:
① Randomly pruned networks enjoy better uncertainty estimation than their dense counter- parts. Figure shows that randomly pruned ResNet-20 matches or even improves the uncertainty estimation of dense networks with full range of sparsities. ECE of random pruning grows with the model size, in line with the finding of dense networks in Guo et al. (2017). Still, random pruning capable of sampling matching subnetworks with high sparsities (e.g., 80%) except for the largest model, ResNet-1100 10. Results of NLL are presented in Appendix G.2, where randomly pruned networks also match NLL of dense networks extremely high sparsities.
② Random pruning produces extremely sparse yet robust subnetworks on OoD. Figure 4 plots the results of networks trained on CIFAR-1O, tested on CIFAR-100. The results tested on SVHN are reported in Appendix G.I. Large network sizes significantly improve the OoD performance of random pruning. As the model size increases, random pruning can match the OoD performance of dense networks with only 20% parameters. Again, SNIP ratio and ERK outperform uniform sparsities in this setting.
3 Random pruning improves the adversarial robustness of large models. The adversarial robustness of large models (e.g.. ResNet-56 and ResNet-110) improves significantly with mild sparsities in Appendix G.3. One explanation here is that, while achieving high clean accuracy, these over-parameterized large models are highly overfitted on CIFAR-10, and thus suffer from poor performance on adversarial examples (shown by Tsipras et al. (2019); Zhang al. (2019) well). Sparsity induced by random pruning serves as a cheap type of regularization which possibly mitigates this overfitting problem.
5 EXPERIMENTAL RESULTS ON IMAGENET
We have learned from Section 4 that larger networks result in stronger random subnetworks on CIFAR-10/109 We are also interested how far we can go with random pruning on ImageNet (Deng al., 2009), non-saturated dataset which deep neural networks are less over- paratearaeteri than on 1IFAR-10/100 In this section, we provide a large-scale experiment on ImageNet with various ResNets from R8 to 10sNet-101 and Wide ResNet-50.

---

Published as a conference paper at ICLR 2022
ResNet-20
ResNet-32
ResNet-44
0.03
0.04
ECE
0.03
0.02
0.03
0.02
0.02
0.1
0.2 0.3
0.4 0.5
0.6
0.7 0.8 0.9
0.1
0.2
0.3
0.4 0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Sparsity
Sparsity
Sparsity
ResNet-56
ResNet-110
0.045
0.04
0.040
ECE
0.030
0.1
0.2
0.3
0.4
0.5 Sparsity
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5 Sparsity
0.6
0.7
0.8
0.9
Uniform
Uniform+
Ratio
Figure 3:
Uncertainty estimation (ECE). The experiments are conducted with various models on CIFAR-10. 10. Lower ECE values represent better uncertainty estimation.
ResNet-20
ResNet-32
ResNet-44
0.87
0.86
0.86
ROC-AUC
0.84
0.85
0.84
0.84
0.82
0.83
0.83
0.7
0.8 0.9
0.2 0.3
0.5
0.6
Sparsity
Sparsity
Sparsity
ResNet-56
ResNet-110
0.87
0.87
0.86
ROC AUC
0.86
0.85
0.84
0.85
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5 Sparsity
0.6
0.7
0.8
0.9
Sparsity
ERK
Uniform
Uniform+
SNIP Ratio
Dense
Figure 4: Out-of-distribution performance (ROC-AUC), Experiments are conducted with models trained on CIFAR-10, tested on C00.R-100. Higher ROC-AUC refers to better OoD performance.
Same with CIFAR, we evaluate sparse training with random pruning from various perspectives, including the predictive accuracy, OoD detection, adversarial robustness, and uncertainty estimation. We choose SNIP and ERK+ sparsity ratios for Wide ResNet-50, and SNIP ratios for the of the architectures. The sparsity of randomly pruned subnetworks is set as [0.7, 0.5, 0.3]. We first show the trade-off between the parameter count and the accuracy for all architectures Figure Setop-leftt Moreover, to better understand the computational benefits brought by sparsity, we report the trade-off between test FL.PPs and each measurement metric the rest of Figure
Predictive Accuracy. The parameter count-accuracy trade-off and the FLOPS-accuracy trade-off reported Figure 5-top-left and Figure 5-top-middle, respectively. Overall, we observe a very similar pattern as the results of CIFAR reported in Section 4. On smaller models like ResNet-13 18 and ResNet-34, random pruning can not find matching subnetworks. When the model size gets considerably larger (ResNet-101 101 and Wide ResNet-50), the test accuracy of random pruning quickly improves and matches the corresponding dense models (not only the amall-dense models) at 30% ~ 50% sparsities. While the performance difference between SNIP ratio and ERK on CIFAR- is

---

Published as a conference paper at ICLR 2022
Accuracy (#Param)
Accuracy (#FLOPs)
Adversarial Rebustness
20.0
Accuracy
76-
Test Accuracy
76
17.
15.0
74
74
Robustness
10.0
70
70
5.0
68
68
2.5
66
66
20
40
60
10
15
20
10
15
20
#Parametar Count [10*)
#F1OP%
(10']
#FLOPS [10"
OoD Performance
Uncertainty Estimation (ECE)
Uncertainty Estimation (NLL)
0.55
0.100
1.5
0.50
0.055
1.4
(OC-UU
0.45
0.090
1.3
0.085
0.35
0.080
0.075
1.1
0.25
0.070
0.065
15
10
15
20
15
20
[10"]
#FLO (107)
[10*]
SNIP Ratio ResNet-18
Ratio ResNet-101
Dense ResNet-18
Dense ResNet-101
ResNet-34
Wide ResNet-50
ResNet-34
Dense ResNet-50
SNIP Ratio ResNet-50
ERK+ Wide ResNet-50
Dense ResNet-50
Figure 5: Summary of evaluation on ImageNet. Various Evaluation of ResNets on ImageNet, including predictive accuracy on the original ImageNet, adversarial robustness with FGSM, OoD performance ImageNet-O, and uncertainty (ECE and NLL). The sparsity of randomly pruned subnetworks set as [0.7, 0.5, 0.3] from left to right for each line.
vague, SNIP ratio consistently outperforms ERK+ with Wide Resnet-50 on ImageNet with the same number of parameters (see Appendix F for more details), Besides, we observe that random pruning receives increasingly larger efficiency gains (the difference between x-axis values of sparse models and dense models) with the increased model size on ImageNet.
While in Figure 5-top-left, randomly pruned Wide ResNet-50 with SNIP ratios (purple line) can easily outperform the dense ResNet-50 (blue star) by 2% accuracy with the same number of parameters, the former requires twice as much FLOPs as the latter in Figure 5-top-middle. This observation highlights the importance of reporting the required FLOPS together with the sparsity when comparing two pruning methods.
Broader Evaluation of Random Pruning on ImageNet. As shown in the rest of Figure 5, the performance of the broader evaluation on ImageNet is extremely similar to the test accuracy. Random pruning can not discover matching subnetworks with small models. However, as the model size increases, it receives large performance gains in terms of other important evaluations, including uncertainty estimation, OoD detection performance, and adversarial robustness.
5.1 UNDERSTANDING RANDOM PRUNING VIA GRADIENT FLOW
The performance gap between the ERK and SNIP ratio on ImageNet raises the question what benefits do the layer-wise sparsities of SNIP provide to sparse training? Since SNIP takes gradients into account when determines the layer-wise sparsities, finding some insights on this question. The form of gradient norm we choose is the effective gradient flow (Tessera et al., 2021) which only calculates the gradient norm of active weights. We measure the gradient norm of sparse networks generated by SNIP and FRK2 during the early training phase. The results are depicted in Figure 6.
turn gradient flow in the hope of
Overall, we see that the SNIP ratio indeed brings benefits to the gradient norm at the initial stage compared with ERK. Considering only SNIP (green lines) and ERK (orange lines), the ones with higher gradient norm at the beginning always achieve higher final accuracy on both CIFAR-10 and ImageNet. This result is on par with prior work (Wang et al., 2020), which claims that increasing the gradient norm at the beginning likely leads to higher accuracy. However, this observation does not
The gradient norm patterns of ERK and ERK+ are too similar to distinguish. After specifically zooming in the initial training phase, find that the gradient norm of ERK+ is slightly higher than ERK.
8

---

Published as a conference paper at ICLR 2022
ResNet-20, Sparsity=50%
ResNet-56, Sparsity 50%
ResNet-110, Sparsity 50%
Gradlent Norm
Top-)
Top-]
Top-1
0.100
Top-1
0.020
0.04
0.075
0.018
0.050
0.02
0.016
0.025
0.014
0.000
5000
10000
15000 20000
5000
10000 15000 20000
5000
10000 15000 20000
Wide ResNet-50, Sparsity=85%
Wide ResNet-50, Sparsity=70%
Wide ResNet-50, Sp%
Gradlent Nor
0.006
Top-1
0.005
Top-]
0.005
Tep-1
78.11%
78.11%
0.004
0.004
0.004
0.003
0.003
0.002
0.002
0.002
10000 20000 30000 40000 Training Iterations [#]
10000 20000 30000 40000
10000 20000 30000 40000 Training Iterationn [4]
Training Iterations [#]
Figure 6: Gradient norm of randomly pruned networks during training. Top: Comparison between gradient norm of SNIP and ERK with 50% sparse ResNet-20, ResNet-56, and ResNet- on CIFAR-10. Botton: Comparison between gradient norm of SNIP and ERK with Wide ResNet-50 on ImageNet at various sparsities.
hold for dense Wide ResNet-50 on ImageNet (blue lines). Even though dense models have lower gradient norm than SNIP, they consistently outperform SNIP in accuracy by a large margin.
Instead of focusing on the initial phase, we note that the gradient norm early in the training (the flat phase after gradient norm drops) could better understand the behaviour of networks trained under different scenarios. We empirically find that final performance gaps between sparse networks and dense networks is highly correlated with gaps of gradient norm early in training. Training with small networks (e.g., ResNet-20) on CIFAR- 10 results in large performance and gradient norm gaps between sparse networks and dense networks, whereas these two gaps simultaneously vanish when trained with large networks, e.g., ResNet-56 and ResNet- 10. Similar behavior can be observed in Wide ResNet-50 on ImageNet in terms of sparsity. Both the performance and gradient norm gaps decrease gradually as the sparsity level drops from 85% to 60%. This actually makes sense, since large networks (either large model size large number of parameters) are likely to still be over- parameterized after pruning, so that all pruning methods (including random pruning) can preserve gradient flow along with test accuracy equally well.
Our findings suggest that only considering gradient flow at initialization might not be sufficient for sparse training. More efforts should be invested to study the properties that sparse network training misses after initialization, especially the early stage of training (Liu et al., 2021a). Techniques that change the sparse pattern during training (Mocanu et al., 2018: Evci et al., 2020a) and weight rewinding (Frankle et al., 2020; Renda et al., 2020) could serve as good starting points.
CONCLUSION
In this work, we systematically revisit the underrated baseline of sparse training random pruning. Our results highlight a counter-intuitive finding, that is, training a randomly pruned network from scratch without any delicate pruning criteria can be quite performant. With proper network sizes and layer-wise sparsity ratios, random pruning can match the performance of dense networks even at extreme sparsities. Impressively, a randomly pruned subnetwork of Wide ResNet-50 can be trained to outperforming strong benchmark, dense Wide ResNet-50 on ImageNet. Moreover, training with random pruning intrinsically brings significant benefits to other desirable aspects, such as out-of- distribution detection, uncertainty estimation and adversarial robustness. Our paper indicates that in addition to appealing performance, large models also enjoy strong robustness to pruning. Even if we pruning with complete randomness, large models can preserve their performance well.
9

---

Published as a conference paper at ICLR 2022
7 REPRODUCIBILITY
We have shared the architectures, datasets, and hyperparameters used in this paper in Section 3.2. Besides, the different measurements and metries used in this paper are shared in Appendix A. We have released our code at ttttps://githhi com/VITA Group Cacoopp/Rando_Prruniiii
ACKNOWLEDGEMENT
This work have been done when Shiwei Liu worked as an intern at JD Explore Academy. This work partially supported by Science and Technology Innovation 2030 "Brain Science and Brain-like Research" Major Project (No. 2021ZD0201402 and No. 2021ZD02014057 Z. Wang is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML).
REFERENCES
Zeyuan Allen- Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via parameterization. International Conference on Machine Learning, pp. 242-252. PMLR, 2019.
Maximilian Augustin, Alexander Meinke, and Matthias Hein. Adversarial robustness on in-and out- distribution improves explainability, In European Conference on Computer Vision, pp. 228-245 Springer, 2020.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018. URL https: http://pppteeeeeeeeeeeeeeeeeeeep net/ www.:neettoouttteeeeeeoem BarpuumiidBJJ__Be
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dkebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.0660 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https: //t:://rroccc ptw.//oooeeeeeeee ngs neurips. 967 Bbfbaacc 42 o
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. Advances in neural information processing systems, 33: 15844 -58466, 2020a.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34, 2021a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16306-16316. 2021b.
Ting Chen, Simon Korbblihh, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi- sumi-supervised learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 22223-222222222 22255. Curran Associates, Inc., 2020b. URL https: tttp:ppoooeii.nnnnn.nn..ereeeeee cc/paper/ /2020 /e/ febe 95ccdd5 da c0 4000::0000440000555550000e-
10

---

Published as a conference paper at ICLR 2022
Xiaohan Chen, Yu Cheng, Shuohang Wang. Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pp. 2195-22077 2021c.
Bin Dai, Chen Zhu, Baining Guo, and David Wipf. Compressing neural networks using the variational information bottleneck. In International Conference on Machine Learning, pp. 1135-1144. PMLR, 2018.
Pau de Jorge, Amartya Sanyal, Harkirat Behl, Philip Torr, Grégory Rogez, and Punect K. Dokania. Progressive skeletonization: Trimming more fat from a network at initialization. In International Conference Learning Representations, 2021. URL https: pttttt:://opereeeeee http///ppppeee.e/e/eeee net net/forum? forum? Ad=9GsFOUyUPi.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprini 1rXivv:907.048440 2019.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL / / net forum? idd aedd=YichFooa cicTTyy
Paul Erdós and Alfréd Rényi. On random graphs i. Publicationes Mathematicae (Debrecen), 6: 290-297, 1959.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. PMLR, 2020a.
Utku Evci, Yani A Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural networks and how lottery tickets win. arXiv preprint arXiv:2010.03533, 2020b.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Advances in Neural Information Processing Systems., 2021. arXv:22101.03961
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations, 2019. URL https:
//tpppnevieww.....c// net forrmmouumm/////m ?id d--d--rr------ (-11-b3eeF.,
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 32669, PMLR, 2020.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? In International Conference Learning Representations, 2021. URL https: http:::s////peeeviewwe net/ fort/m/rmmmee
Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al. The elements of statistical learning, volume 1. Springer series in statisties New York, 2001.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity deep neural networks. arXiv preprint arXiv: 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2014.
11

---

Published as a conference paper at ICLR 2022
Shupeng Gui, Haotao Wang, Haichuan Yang, Chen Yu, Zhangyang Wang, and Ji Liu. Model compression with adversarial robustness: unified optimization framework. Advances Neural Information Processing Systems, 32, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pp. 1321-1330. PMLR, 2017.
Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. Sparse dnns with improved adver- sarial robustness. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, Curran As- sociates, Inc., 2018. URL https: http:///ppceeeeeeeeeeeeee neurips wwcp/pppr//0000 8/ file/ 4c5bde7 a
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations, 2015.
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann, 1993.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41-50, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. International Conference on Learning Representations, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ut-of-distribution generalization. arXiv preprint arXiv:2006. ICCV, 2021.
Ting-Kuei Hu, Tianlong Chen, Haotao Wang, and Zhangyang Wang. Triple wins: Boosting ac- curacy, robustness and efficiency together by enabling anput-adaptive inference. arXiv preprint arXiv:2002.10025, 2020.
Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A, (11))66000 1989.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(78733:583--88) 2021.
Jeremy Kepner and Ryan Robinett. Radix-nee Structured sparse matrices for deep neural networks. In 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), pp. 268-274 274. IEFE, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateck Jain, Sham Kakade. and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Learning, pp. 5544 5555. PMLR, 2020.
12

---

Published as a conference paper at ICLR 2022
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint 1rXiv:1612.01444. 2016.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. In International Conference Learning Representations, 2021. URL https: /owpnreviee.eeeeeeoooee net forum?
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHON NETWORK
PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference on Learning Representations, 2019, URL /twpr////ep/eeeeeeee//eeeee wwt. formm? wttp::enneeeetnieteeeeeeeo.m Fid=B1VZajjcXX.
Shiwei Liu, Decebal Constantin Mocanu, Amarsagar Reddy Ramapuram Matavalam, Yulong Pei, and Mykola Pechenizkiy. Sparse evolutionary deep learning with over one million artificial neurons on commodity hardware. Neural Computing and Applications, 2020a.
Shiwei Liu, TT van der Lee, Anil Yaman, Zahra Atashgahi, Ferrar, Ghada Sokar, Mykola Pech- enizkiy, and DC Mocanu. Topological insights into sparse neural networks. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECMLPKDD, 2020b.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneraaion. Advances in Neural Information Processing Systems (NeurIPs). 2021a.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over-parannn ?aarraarmmttriiiooon? in-time over- parameteeeeziiioonnn in sparse training. In Proceedings of the 39th International Conference on Machine Learning, pp. 6989-7000. PMLR, 2021b.
Tianlin Liu and Friedemann Zenke. Finding trainable sparse networks through neural tangent transfer. In International Conference on Machine Learning, pp. 6336-6347. PMLR, 2020.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. International Conference on Learning Representations, 2018
Zelda Mariet and Suvrit Sra. Diversity networks: Neural network compression using determinantal point processes. In International Conference Learning Representations, 2016.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't 't know. In International Conference Learning Representations, 2020. URL https: // //openreview wttp:/iwwwwee//ouueeeeeoo
worrmmiiiii idd Byw.mmii--yyyyyaeeeeea
Decebal Constantin Mocanu, Elena Mocanu, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. topological insight into restricted boltzmann machines. Machine Learning, 104(2): 243-270, Sep 2016. ISSN 1573-0565. doi: 10. 1007/s109944-0065557--2 URL https: /doi. 0rg/10.11077 199944 6 5570-2.
Decebal Constantin Mocanu, Flena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9()):2383, 2018.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. International Conference on Learning Represen- tations, 2016.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646 4655. PMLR, 2019.
Michael Mozer and Paul Smolensky. Using relevance to reduce network size automatically. Connection Science, 1(1):3-16, 1989.
13

---

Published as a conference paper at ICLR 2022
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL htttp:///ffiee www.soad..deeeeoo.oomo. edu/ housenumbers ni https//nnnnnnnppoooaaaaaaa harr1_ohoouueeerrrsssss pdff.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-paramettrization in generalization of neural networks, In International Conference Learning Representations, 2019. URL https: www.reeiew..nee.eeeeeee.ee. wwwpweiwwwwweeeeerurreee.m hid.mmmmggdgyyygyyyycycc.
Roman Novak, Yasaman Bahri, Daniel A. Abolañia, Jeffrey Pennington, and Jascha Dibh-Dickstein. Sensitivity and generalization in neural networks: an empirical study. In International Confer- ence on Learning Representations, 2018. URL https://////. http///ppee/ereeeeee net forum? iid= HJC2SzZCR.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on security and privacy (EuroS&P), pp. 372 372-387. IEEE, 2016.
Ameya Prabhu, Girish Varma, and Anoop Namboodiri. Deep expander networks: Efficient deep networks from graph theory. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 20-35, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang. Michael Matena. Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http:// jmmr aoggpaperrs,..aeee /21//221220// htm.l...
Alex Renda, Jonathan Frankle, and Michael Carbin. neural network pruning. In International Conference on Learning Representations, 2020. URL.
Comparing rewinding and fine-tuning in
net forum? JNNK...
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 20378-20389, Curran As- sociates, Inc., 2020. URL / http://pppoooeeddnnnneeee...eeee neurips wttpeccccpppppcpeeeeeeeeeeeee file// aaaaaaaaaaeaaaa555555aaaaaeaeee.
.pf.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:1409.1555 ICLR., 2014.
Jingtong Su, Yihang Chen, Tianle Cai, Tianhao Wu, Ruiqi Gao, Liwei Wang, and Jason D Lee. Sanity-checking pruning methods: Random tickets can win the jackpot. Advances in Neural Information Processing Systems. arXiv:2009.11094 2020.
Xavier Suau, Luca Zappella, and Nicholas Apostoloff. NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES, 2019. URL htttps://openreviii fet/Eorum? dd=
Christian Szegedy, Wojeieeh Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 13 2013.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances Neural Information Process- ing Systems. 2rXiv:2006.05467 2020.
Kale-ab Tessera, Sara Hooker, and Benjamin Rosman. Keep the gradients flowing: Using gradient flow to study sparse network optimization. arXiv preprint arXiv:2102.01670 2021.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Ro- bustness may be odds with accuracy. In International Conference Learning Representations, 2019. URL https:: opwneevieee net/ forum? fttp///dsmmmm..aaooooe d=Syxxb30cY7.
Stijn Verdenius, Maarten Stol, and Patrick Forré. Pruning via iterative ranking of sensitivity statistics. arXiv preprint XXXx:2006.0089 2020.
14

---

Published as a conference paper at ICLR 2022
Chaoqi Wang, Guodong Zhang. and Roger Grosse. Picking winning tickets before training by preserving gradient flow. International Conference on Learning Representations, 2020. URL
https: http:://pppeeeeeeeeeeeeee net
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. In International Conference on Learning Representations, 2020. URL https: http:///ppneeeeeeeeeeeeeee net. forum?
id-HyxJhCEPDS.
Shaokai Ye, Kaidi Xu, Sijia Liu, Hao Cheng, Jan-Henrik Lambrechts, Huan Zhang, Aojun Zhou, Kaisheng Ma. Yanzhi Wang, and Xue Lin. Adversarial robustness vs. model compression, both? In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 111-120, 2019.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang. Xiaohan Chen, Zhangyang Wang, Richard G Baraniuk, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. In International Conference on Learning Representations 2020 (ICLR 2020),
2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C. 30.87. URL https: dxx doi /10. 52 C0.30.8.0..
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent EI Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. In International Conference on Learning Representations, 2018. URL httpa::/openeeeeiiee net/ htt.:/ooomnndd-lllggaaaa
Michael Zhu and Suyog Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. Xiviv preprint arXiv:1710.01878 2017.
15

---

Published as a conference paper at ICLR 2022
A MEASUREMENTS AND METRICS
Formally, let's denote a (sparse) network = + y trained on samples from distribution D. f(x (2,00 0) and f(x, 0 O m) refers to the dense network and the pruned network, respectively. The test accuracy of the pruned subnetworks is typically reported as their accuracy on test queries drawn from D, i.c., ((a.)))D(ff( (fr 8 m) = v)- In addition the test accuracy, we also evaluate random pruning from the perspective of adversarial robustness (Goodfellow et al., 2014). OoD performance (Hendrycks al., 2021), and uncertainty estimation (Takssminnarayanan et al., 2016).
Adversarial Robustness. Despite of the remarkable ability solving classification problems, the predictions of deep neural networks can often be fooled by small adversarial perturbations (Szegedy et al., 2013; Papernot et al., 2016). Prior works have shown the possibility of finding the sweet point of sparsity and adversarial robustness (Guo et al., 2018; Ye et al., 2019; Gui et al., 2019; 99; Hu et al., 2020). As the arguably most naive method of inducing sparsity, we also interested in if training randomly pruned subnetwork can improve the adversarial robustness of deep networks. We follow the classical method proposed in Goodfellow al. (2014) and generate adversarial examples with Fast Gradient Sign Method (FGSM). Specifically, input data is perturbed with esign(V (V,C(O x))) where refers the perturbation strength, which chosen & in paper.
255
-itt----iittiittion performance. The investigation -utt---distri (OoD) generalization is of importance for machine learning both academic and industry fields. Since the i.i.d. assumption can hardly be satisfied, especially those high-risk scenarios such as healthcare, and military. We evaluate whether random pruning brings benefits to OoD. Following the classic routines (Augustin et al., 2020; Meinke & Hein, 2020), SVHN (Netzer et al., 201 I), CIFAR- 100, and CIFAR-10 with random Gaussian noise (Hein al., 2019) are adopted for models trained on CIFAR- 10; ImageNet-O as the OoD dataset for models trained on ImageNet.
Uncertainty estimation. In the security-critical scenarios, e.g., self-driving, the classifiers not only must be accurate but also should indicate when they are likely to be incorrect (Guo et al., 2017). To test effects of the induced spansity on uncertainty estimation, we choose two widely-used metries, expected calibration error (ECE) (Guo et al., 2017) and negative log likelihood (NLL) (Friedman et al., 2001).
B PREDICTIVE ACCURACY OF RESNET WITH VARYING WIDTH
ResNet-20
ResNet-32
ResNet-44
92.5
92.5
92.5
90.0
90.0
90.0
87.5
87.5
87.5
85.0
85.0
85.0
82.5
82.5
82.5
80.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
Wide Multiplier
Wide Multiplier
Wide Multiplier
Dense Training
9p%
Sparsity=30%
Sparsity=50%
Sparsity=70%
Figure 7: Test accuracy of training randomly pruned ResNet-20, ResNet-32, and ResNet-44 from scratch with ERK ratios when model width varies CIFAR-10.
16

---

Published as a conference paper at ICLR 2022
C PREDICTIVE ACCURACY OF RESNET ON CIFAR- 10 INCLUDING GRASP
To avoid multiple lines overlap with each other, we report the results of GraSP ratio on CIFAR-10 10 separately in this Appendix. is surprising to find that random pruning with GraSP ratio (red lines) consistently have lower accuracy than ERK and SNIP, except for extremely high sparsites, i.e., 0.8 and 0.9.
ResNet-20
ResNet-32
ResNet-44
Test Accuracy
92
93
90
92
90
91
90
86
0.2
7.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2 0.3
0.6
0.7
0.8
0.9
Sparsity
Sparsity
Sparsity
ResNet-56
ResNet-110
94
Test Accuracy
93-
93
92
91
91
90
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Sparsity
Sparsity
ERK
Uniform
Uniform+
SNIP Ratio
GraSP
Dense
Figure From shallow to deep. Test accuracy of training randomly pruned subnetworks from scratch with different depth on CIFAR-IO. ResNet-A refers to ResNet model with A layers in total.
ResNet-20-8
ResNet-20-16
ResNet-20-24
Agcuracy
87
93
92
84
91
88
90
78
0.1 0.2
0.6
0.7
0.3
0.4
0.5
0.6
0.1
0.2
0.6
0.7
0.8
ResNet-20-32
ResNet-20-40
ResNet-20-56
94
95
Test Accuracy
94
93
94
92
93
91
0.2
0.4 0.5
0.1
0.2
Sparsity
Sparsity
Sparsity
ERK
Uniform
Uniform+
SNIP Ratio
GraSP Ratio
Dense
Figure 9: From narrow to wide. Test accuracy of training randomly pruned subnetworks from scratch with different width on CIFAR-I0. ResNet-A-B refers ResNet model with A layers in total and B filters in the first convolutional layer.
17

---

Published as a conference paper at ICLR 2022
D PREDICTIVE ACCURACY OF RESNET ON CIFAR-100
ResNet-20
ResNet-32
ResNet-4
Test Accuracy
70
65
65
60
65
55
60
60
0.1
0.2
0.3
0.4
0.5 Sparsity
0.6
7 0.8
0.9
0.1
0.2
0.3
0.4 0.5 0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Sparsity
Sparsity
ResNet-56
ResNet-110
72.5
Test Accuracy
70.0
67.5
65.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Sparsity
Sparsity
ERK
Uniform
Uniform+
SNIP Ratio
Dense
Figure 10: From shallow deep. Test accuracy of training randomly pruned subnetworks from scratch with different depth on CIFAR-100, ResNet-A refers to ResNet model with A layers in total.
ResNet-20-8
ResNet-20-16
ResNet-20-24
70
72.5
Accuracy
70.0
50
67.5
60
40
65.0
30
0.1
0.2
0.3
0.4
0.6 Sparsity
0.5
0.7
7...0.7.00
0.8 0.9
0.1 0.2 0.3
0.4 0.5 0.6 0.7 0.8
0.9
0.1
0.2
0.3
0.4
0.5 0.6 0.7 0.8 Sparsity
Sparsity
ERK
Uniform
Uniform+
SNIP Ratio
Dense
Figure 11: From narrow wide. Test accuracy of training randomly pruned subnetworks from scratch with different width CIFAR-100, ResNet-A-B refers to a ResNet model with A layers in total and filters the first convolutional layer.
18

---

Published as a conference paper at ICLR 2022
E PREDICTIVE ACCURACY OF VGG ON CIFAR-10/100
VGG-11
VGG-13
VGG-16
93.0
94.00
94.0
TestAccurasy
92.5
93.75
93.5
93.50
93.25
93.00
01.0
92.75
0.1 0.2
0.3
0.4
0.5 Sparsity
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5 Sparsity
0.6
0.7
0.8
0.9
Sparsity
ERK
SNIP Ratio
Dense
Figure 12: From shallow to deep. Test accuracy of training randomly pruned VGGs from scratch with different depth on CIFAR-10.
VGG 11
VBG-15
VGG 16
75
74.5
Test Accuray
71.0
74.0
70.5
73.5
70.0
73.0
73
72.5
69.0
72.0
68.5
0.1
0.2
0.3
0.4
0.5 Sparsity
0.6
0.7
0.8
0.9
0.1 0.2
0.3
0.4 0.5 0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.5 Sparsity
0.7
Sparsity
ERK
SNIP Ratio
Dense
Figure 13: From shallow to deep. Test accuracy of training randomly pruned VGGs from scratch with different depth on CIFAR-100.
19

---

Published as a conference paper at ICLR 2022
F COMPARING RANDOM PRUNING WITH ITS DENSE EQUIVALENTS WITH VARIOUS RATIOS.
70% Sparse Wide ResNet-50
Wide ResNet-50
75.2
78-
Rediet-30
Test Accuracy [%)]
panse RosNet
74.8
72
74.4
68
0.5
17.5 22.5 27.5
Sparsity of the Last Layer
Number Parameters [Millions]
SNIP
Figure 14: Test accuracy of Wide ResNet-50 ImageNet. Left: Performance of ERK with various sparsity of the last fully-connected layer. We vary the sparsity of the last fully-connected layer while keeping the overall sparsity fixed 70%. Results of the original ERK indicated with dashed red lines. Right: Performance comparison between randomly pruned Wide ResNet-50 and the corresponding dense equivalents with similar parameter count.
We demonstrate the performance improvement caused by ERK+ on ImageNet. As mentioned earlier, ERK naturally allocates higher sparsities to the larger layers while allocating lower sparsities the smaller ones. As a consequence, the last fully-connected layer is very likely be dense for the datasets with only few classes, e.g.. CIFAR-10 10. However, for the datasets with larger number of classes, e.g., ImageNet, the fully-connected layer is usually sparse. We empirically find that allocating more parameters the last fully-connected layer while keeping the overall parameter count fixed leads to higher accuracy for ERK with Wide ResNet-50 on ImageNet.
ERK+.
We vary the last layer's sparsity of ERK while maintaining the overall sparsity fixed and report the test accuracy achieved by the corresponding sparse Wide ResNet-50 on ImageNet in Figure 14-left. We can observe that the test accuracy consistently increases the last layer's sparsity decreases from 0.8 to O. Consequently, we keep the last fully-connected layer of ERK dense for ImageNet and term this modified variant as ERK+.
Compare random pruning with its dense equivalents. To draw a more solid conclusion, we train large, randomly pruned Wide ResNet-50 on ImageNet and compare the same number of parameters on ImageNet. As shown in Figure 14-right, all randomly pruned networks outperform the dense ResNet-34 with the same number of parameters. ERK+ consistently achieves higher accuracy than ERK, even closely approaching the strong baseline dense ResNet-50.. More interestingly, the layer-wise sparsities discovered by SNIP boost the accuracy of sparse Wide ResNet-50 over dense ResNet-50, highlighting the importance of layer-wise sparsity ratios on sparse training Given the fact that the performance gap between SNIP ratio and ERK on CIFAR-10 is somehow vague, results highlight the necessity of evaluating any proposed pruning methods with large-scale models and datasets, e.g., ResNet-50 on ImageNet.
the dense equivalents with
20

---

Published as a conference paper at ICLR 2022
G BROADER EVALUATION OF RANDOM PRUNING ON CIFAR-10
G.1 OUT-OF-DISTRIBUTION PERFORMANCE (ROC-AUC).
ResNet-20
ResNet-32
ResNet-44
0.950
0.96
0.925
0.94
ROC-AUC
0.900
0.900
0.92
0.875
0.875
0.90
0.850
0.850
0.7
0.7
0.5 Sparsity
Sparsity
Sparsity
ResNet-56
ResNet-110
0.96
0.94
ROC-AUC
0.92
0.90
0.88
0.88
0.2
0.3
0.4
0.5
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Sparsity
Sparsity
ERK
Uniform
Uniform+
SNIP Ratio
Dense
Figure 15: Out-of-distr performance (ROC-AUC). The experiments are conducted with various models trained CIFAR-10, tested on SVHN. Higher ROC-AUC refers to better OoD performance.
21

---

Published as a conference paper at ICLR 2022
G.2
UNCERTAINTY ESTIMATION (NLL)
ResNet-20
ResNet-32
ResNet-44
0.34
0.40
0.35
0.325
0.30
NLL
0.30
0.275
0.26
0.25
0.250
0.3
0.4
0.5
0.6
0.7
0.8 0.9
0.1
0.2
0.3
0.4
0.6
0.7 0.8 0.9
0.1
0.2
0.3
0.4
0.6 Sparsity
0.5
0.7
0.8
Sparsity
Sparsity
ResNet-56
ResNet-11
0.32
0.30
0.28
0.26
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.1
0.2
0.3
0.4
0.5 Sparsity
0.6
0.7
0.8
0.9
Sparsity
ERK
Uniform
Uniform+
SNPP Ratio
Dense
Figure 16: Uncertainty estimation (NLL). The experiments are conducted with various models on CIFAR-10. Lower NLL values represent better uncertainty estimation.
G.3 ADVERSARIAL ROBUSTNESS
ResNet-20
ResNet-32
ResNet-44
25
Robust Accuracy
20:
6
15
10
4-
5
5
0.1
0.2
0.3
0.4
0.6 Sparsity
0.7 0.8 0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7 0.8
0.1
0.2
0.3
0.4
0.5
0.6 0.7
Sparsity
Sparsity
ResNet-56
ResNet-110
Robust Accuracy
20
20
10
10
Sparsity
Sparsity
ERK
Uniform
Uniform+
SNIP Ratio
Dense
Figure 17: Adversarial robustness. The experiments are conducted with various models on CIFAR-
10. Higher values represent better adversarial robustness.
22


Reviews:
[{"review_id": "q4vJTvPV-ka", "reviewer": "Reviewer_WXbx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I will go sequentially and start with strengths and proceed to weaknesses:\n\nThe writing of the is really good and I appreciate the authors for their efforts. \nI also loved the extensive related work which will act as a great stepping stone for new researchers the same way Gale et al 2019 did in the past 2 years. \n\nStrengths:\n1) The revisiting of Random Pruning makes a lot of sense give the more complex pruning methods based on Hessians etc becoming a norm. The evaluation of this simple baseline should give perspective to new readers about what components affect the final deliverables most. \n2) Very well crafted related work section \n3) THe methodology is very clear and makes a clear distinction of where Random Pruning falls into the literature of learnable sparsity etc.,\n4) The sparsity ratios being discussed are well explained are extensive. \n5) The experimental setup is extremely clear and reproducible. \n6) The results on CIFAR are extensive with various architectures and variants among them along with a very nice section 4.1 listing the observations that come of these evaluations. \n7) While not as extensive similar observations were made on ImageNet with Wide ResNet50 as the base architecture. \n8) The observations made on the importance of pruning ratios, the indifference of pruning methods at larger model sizes are valuable (while not novel). \n9) The experiments trying to understand random pruning via gradient flow is interesting and give some perspective as to why certain pruning ratios might be doing better. \n10) While only on CIFAR, I love the extent of broader evaluation on uncertainty, OOD, Robustness, etc for these random pruning models with different architectures and pruning ratios.\n11) The plots are very clean and authors should be appreciated. The takeaways are explicitly mentioned making it easy for first-time readers. \n12) The appendices are well laid out and detailed.\n\nWeaknesses:\n1) As I mentioned earlier, the method itself isn't novel and the certain observation of pruning larger models being better than dense counterparts of the same size isn't novel. Gale et al., 2019, Wallace et al 2020, Kusupati et al 2020 make the same observation but even Zhu & Gupta 2018 make similar observations and it is often observed that pruning large models is called large-sparse is better than small-dense. That is the same observation being made here. \n2) The argument about the sparsity ratios being super important is not new either. There are papers like Su et al NeurIPS 2020 which argue that sparsity ratios are all that matter even in the LTH space and randomly changing the weights still gives them the tickets needed to win the lottery. This paper is missing in the related work but I am pretty sure (I am probably blanking out) there are papers that talk about the importance of sparsity ratios. Even papers like RigL and STR make a case for why sparsity ratios are important in general. \n3) While on the topic of sparsity ratios, I also noticed that the benchmarked pruning results don't have inference cost involved in these cases. For example, I would love to see the FLOPs for Fig 4 (right) to see what the FLOPs are for pruned WRN vs a Dense ResNet. \n4) I have no qualms about the broader evaluation except that I would love to see them on ImageNet even if it is for a single architecture. Is that something possible?\n\nOverall, I think I don't see as much novelty in this paper (which is fine) however even the observations made are not completely novel. The evaluation is thorough and the observations are clean. However, I want to hear from other reviewers and the authors during the discussion before I move my decision towards acceptance. This is a great benchmarking paper, but at times looks like stitching across insights from other papers. Looking forward to the discussion. \n\n", "summary_of_the_paper": "This paper revisits the OG and the simplest pruning baselines out there: Random Pruning. Random Pruning randomly zeros out the weights in a network with a target budget and is often used at a layer level based on the chosen sparsity ratios. \n\nThis paper is a detailed study into Random Pruning. As put forward in contributions, the discussion includes the dependence on network size, the importance of sparsity ratios, and lastly the general-purpose aspects like robustness and OOD, etc of these pruned networks. \n\nThe observations seem to point to the fact that as the model size increases the pruning method becomes obsolete and everything converges to the same solution. They also show that sparsity ratios matter more than anything else for pruning techniques. These two observations have been made in the past in different settings but still are valid observations. \n\nLastly, the paper extensively tries to evaluate the robustness aspects of these models + sparsity ratios for CIFAR. \n\n", "summary_of_the_review": "As mentioned in the paper, the technique isn't novel however, the detailed study can benefit quite a few in the community given the potential ramifications of such observations. I would say it is a great benchmarking paper that would help people like \"The State of Sparsity\" (Gale et al., 2019) did, but again it doesn't make a super-strong case for acceptance. I am on the fence with a 5.  However, I am willing to change my mind after rebuttal and discussion with other reviewers. \n\nI still think this paper is a good asset and will be a great one if the same scale of experiments is done on ImageNet instead of CIFAR and will prove to be a solid exploratory work in that case.\n\nOverall, I am happy with the rebuttal. I want to see these updates and results in the main paper and not the appendix and you can do this as revision in the next few days. \n\nI am increasing my score to 6 and will talk to other reviewers but push for an acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_XTvUvdKzu1", "reviewer": "Reviewer_nAhU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n-\tThis paper systematically revisits the underrated baseline of sparse training – random pruning. The authors present a series of very interesting findings with solid experimental supports. \n-\tThe findings first highlight a hidden gem role of larger models – their high tolerance to sparse training. As the networks grow wider and deeper, the performance of training a randomly pruned sparse network will quickly grow to matching that of its dense equivalent, even at high sparsity ratios. This is an inspiring new insight of broad potential impact in current ML trend, especially how to train larger models more efficiently.\n-\tAnother meaningful discovery is that, echoing prior arts, the layer-wise sparsity ratio or “effective width” seems to be a critical factor for high performant sparse training. Borrowing ratios from SNIP could lead to pushing the performance of a completely random sparse Wide ResNet-50 over the strong baseline of densely trained ResNet-50 on ImageNet. However, this part lacks deep understanding of its underlying cause – see first bullet under “questions”.\n-\tThe effectiveness of ERK on ImageNet (with the help of sparsity ratios) is surprising and impressive – perhaps the strongest I’ve seen in relevant empirical literature. It can outperform SNIP/Grasp for many cases, both in accuracy and in robustness/other aspects. \n-\tCodes are included as supplementary, and experiments have included sufficient details to be reproducible. \n\n\nQuestions or Suggestions:\n-\tOne weakness I feel about this paper is its interpretation part in Section 4.3. Especially, I find it hard to comprehend what “early-stage gradient gaps” mean or can reflect for sparse network optimization. The authors basically described the phenomenon then stopping without digging more insights.\n-\tMoreover, if early training dynamics are indeed essential for improving sparse training, can this be combined with dynamic sparse training (e.g. Evci 2020a)? As one possible way to balance randomness with learned connectivity, could there be some dynamic sparsity ratio re-allocation during training? The authors are encouraged to discuss more.\n-\tA minor limitation of this work is that random sparsity will only work so well when the network is large/wide enough, as the authors also pointed out. \n-\t(Optional) With SNIP/Grasp being evaluated, it is recommended that the authors can compare with SynFlow and some more recent sparse training baselines too. Including comparison to RigL would be interesting too.\n", "summary_of_the_paper": "Sparse training has traditionally relied on carefully crafted sparse patterns or pruning criteria. This paper reports a novel discovery that without any delicate pruning criteria or careful sparsity structures, training a randomly pruned sparse network from scratch can match the performance of its dense equivalent.", "summary_of_the_review": "The work questions the common wisdom of sparse training and found properly trained randomly pruned sparse networks from scratch can match dense networks in many ways. This finding is counter-intuitive yet important. The paper could yet be strengthened further by providing more interpretations of its observations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OaQ20aWkF8F", "reviewer": "Reviewer_Gr4w", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1.\tAs said in the paper, random pruning is the very baseline for all pruning methods. Its performance was believed to be quite bad in the past, while this paper shows evidence on the contrary. This appears quite surprising and interesting.\n2.\tThey have extensive experiments trying to support their claim. \n3.\tThey further show random pruning networks can outperform dense counterparts in other favorable aspects. Potentially, this can make random pruning more useful in practice.\n\nWeaknesses:\n\nThis paper appears to present surprising results (as indicated by the title -- The “Unreasonable” Effectiveness). However, the claims and contributions in this paper need to be rectified and re-evaluated.  \n\nFirst, a more rigorous way to treat the terminologies is needed. Pruning can be split into pruning a pretrained model (which is the traditional way), and pruning at initialization (i.e, pruning a randomly initialized network). What this paper really discussed and compared is pruning at initialization (PaI). PaI is only a relatively small track in pruning currently. And importantly, it has been shown PaI seriously underperforms the traditional pruning (pruning a pretraiend model), see [*1]. So the title should be “The Unreasonable Effectiveness of Random Pruning at Initialization”, not the most general pruning case. \n\nOtherwise, if you want to claim random pruning can be on par with other pruning methods (let’s take the most simple magnitude pruning as example) under the traditional pruning case, please try to add this result: Prune a pretrained ResNet50 with 90% sparsity on ImageNet, compare random pruning with magnitude pruning (extensive previous methods actually have shown the latter is better).\nThen, if we are talking about pruning at initialization, the so-called “unreasonable effectiveness” is actually not that unreasonable, provided you’ve seen [*1], which is cited as Frankle et al. (2020b) in the paper but I don’t think it is well discussed. [*1] has quite a lot of overlap with this paper.\n\nSpecifically, in [*1], they propose several sanity-check rules to evaluate whether a kind of pruning criterion is really effective or not: Randomly shuffling, Reinitialization, Inversion, in their Sec. 5. Particularly, for the random shuffling, they shuffled the masks obtained by some PaI methods (like SNIP, SynFlow, GraSP) and found  after mask random shuffling, these methods can achieve comparable accuracy (“All methods maintain accuracy or improve when randomly shuffled”). Randomly shuffling masks is essentially the same as random pruning. That is, in [*1], it has been shown that random pruning can perform on par with other PaI methods like SNIP. So the major point of this paper is actually reiterating what has been found in [*1]. \n\nEven similarly, in [*1], after discovering randomly shuffling masks does not hurt accuracy, they thus proposed “In other words, the useful information these techniques extract is not which individual weights to remove, but rather the layerwise proportions by which to prune the network”. Then look at the 2nd contribution claimed in this paper: “We further identify that appropriate layer-wise sparsity ratios can be an important booster for the performance of random pruning”. They are basically the same.\nGiven above, this paper has non-trivial overlaps with [*1]. Although it presents more empirical evidence to show random pruning can be useful in some cases (out-of-distribution detection, uncertainty estimation, and adversarial robustness), yet the major claims and points are already established in [*1].\n\nAlso, in the experiments, some are questionable. E.g., they compare pruned WideResNet50 to ResNet50 (Sec. 4.2). Although they have similar #params, notably they are two different networks. The comparison is not an apple-to-apple comparison (not scientifically valid), no matter how surprising the result may look. To claim random pruning is effective, what should be done is to compare randomly pruned WideResNet50 to a dense WideResNet50, not ResNet50. If the authors can provide this result and still show they have similar accuracy, I will improve my rating.\n\n[*1] Pruning Neural Networks at Initialization: Why are We Missing the Mark?. ICLR, 2021.\n", "summary_of_the_paper": "This paper discusses pruning at initialization (PaI) and they argue random pruning can be quite strong actually, whose performance was under-rated in the past. They present abundant experiments to show random pruning can perform on par with other PaI methods with dedicated pruning criteria. They further show random pruning networks can outperform dense counterparts in other favorable aspects, such as out-of-distribution detection, uncertainty estimation, and adversarial robustness.", "summary_of_the_review": "This paper has substantial overlap with a previous paper. Also, they have unfair comparisons in their experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J6zqp-YGcUP", "reviewer": "Reviewer_GL4s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper draws attention to the commonly used baseline method-random pruning and finds that it can be very effective for training a sparse network.\n2. This paper explores and evaluates various random pruning methods with different layer-wise sparsity ratios. Adopting these layer-wise sparsity ratios as standard sparse initialization for future work seems necessary.\n3. Experiments are promising, and analyses are very detailed.\n4. The findings of the gradient norm seem consistent and correct, making good connections with previous works.\n5. The paper also assesses random pruning from other perspectives for broader evaluation.\n\nWeaknesses:\n1. The effectiveness of random pruning has been investigated in previous works before, although not as detailed as in this paper.\n2. There is not much novelty in the method itself. Mainly using multiple existing pruning methods to support their claim.\n\nQuestions for Author: \n1. In the paper, you mentioned two categories (static and dynamic sparse training), and states that random pruning lies in the static sparse training. In the dynamic sparse training prune-and-grow scheme, if you randomly prune and grow the weights, can it be considered random pruning? \n2. Can these observations and conclusions potentially apply to other popular models such as VGG or Transformer, etc.? \n3. Reference styles are inconsistent, for example (Evci et al. , 2020a ) and Lee et al. (2018)", "summary_of_the_paper": "This paper investigates the effectiveness of random pruning for sparse training. Specifically, the paper adopted several random pruning methods(ERK, Uniform, Uniform+) and compare the results with non-random Pruning methods(SNIP,GraSP). They are conducted on different ResNet scaling models, with  CIFAR-10/100 and ImageNet dataset under different  sparsity ratios. Many observations and conclusions were proposed in the experiments. Results show that random pruning can be quite effective for sparse training especially on larger or wider models.  In some cases, it can outperform well-versed pruning approaches and match the performance of dense networks. ", "summary_of_the_review": "This paper re-evaluates the underrated baseline of random pruning in sparse training. The paper is well-written and well-structured. Although the paper does not provide new pruning techniques,  I believe the conclusions from the paper can contribute to the sparse training domain.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ### Summary

This paper builds on previous work on sparse training that shows the many modern sparse training techniques do no better than a random pruning technique that selects layer wise rations, but otherwise randomly selects which weights within a layer to remove.  The key difference in this work is to take these existing results and scale the size of the network to show that as the size of the network increases, the smaller -- as measured in pruning ratio -- a matching subnetwork becomes.

### Discussion 

#### Strengths

Places an emphasis on simple techniques

#### Weakness

Significant overlap with previous work. Prior already demonstrated the equivalence of random pruning and contemporary pruning at initialization techniques.

### Recommendation

I recommend Accept (poster). However, I do want to stress that there is a significant overlap with previous work. The paper does appropriately attribute observations to previous work. However, there is some risk that readers may misinterpret the title and claim results as a wholly new observation about random pruning, where the reality is instead much more nuanced. Given that the work points to new methodological directions on considering scaling the network as an additional parameter to consider in pruning observations, I do believe these results -- even if narrower in scope that can be interpreted -- provide value to the community.
Decision: Accept (Poster)