An empirical study of a pruning mechanism
Authors: Minju Jung
Hyounguk Shon
Eojindl Yi
SungHyun Baek
Junmo Kim
Venue:  (2021)

Abstract: Many methods aim to prune neural network to the maximum extent. However, there are few studies that investigate the pruning mechanism. In this work, we empirically investigate a standard framework for network pruning: pretraining large network and then pruning and retraining it. The framework has been commonly used based on heuristics, i.e., finding a good minima with a large network (pretraining phase) and retaining it with careful pruning and retraining (pruning and retraining phase). For the pretraining phase, the reason for which the large network is required to achieve good performance is examined. We hypothesize that this might come from the network relying on only a portion of its weights when trained from scratch. This way of weight utilization is referred to as imbalanced utility. The measures for weight utility and utility imbalance are proposed. We investigate the cause of the utility imbalance and the characteristics of the weight utility. For the pruning and retraining phase, whether the pruned-and-retrained network benefits from the pretrained network indded is examined. We visualize the accuracy surface of the pretrained, pruned and retrained networks and investigate the relation between them. The validation accuracy is also interpreted in association with the surface. 

Under review as a conference paper at ICLR 2021
AN EMPIRICAL STUDY OF A PRUNING MECHANISM
Anonymous authors
Paper under double-blind review
ABSTRACT
Many methods aim to prune neural network to the maximum extent. However, there few studies that investigate the pruning mechanism. In this work, we empirically investigate a standard framework for network pruning: pretraining large network and then pruning and retraining it. The framework has been com- monly used based on heuristics, i.c., finding good minima with large network (pretraining phase) and retaining it with careful pruming and retraining (pruning and retraining phase). For the pretraining phase, the reason for which the large net- work required to achieve good performance is examined. We hypothesize that this might come from the network relying on only portion of its weights when trained from scratch. This way of weight utilization referred to as imbalanced utility. The measures for weight utility and utility imbalance proposed. We investigate the cause of the utility imbalance and the characteristics of the weight utility. For the pruning and retraining phase, whether the muned-and-rrerainee network benefits from the pretrained network is examined. We visualize the accu- racy surface of the pretrained, pruned, and retrained networks and investigate the relation between them. The validation accuracy is also interpreted in association with the surface.
INTRODUCTION
Deep learning is currently one of the most powerful machine learning methods. It requires neural network to train, which usually takes a few hundreds times more weights than training data (He) et al. 2016, Zagoruyko & Komodakis 2016, Huang et al., 2017 Karen & Andrew (0155) Usu- ally, in common regimes, greater number weights leads to better performance (Zagoruyko & Komodakis 2016 However, paradoxically, neural networks
also compressible. Many of the recent pruning methods aim to maximally compress the networks (Han et al. 2013 Liu et al.. 2017] He al. 2019: You et al.] 2019), however, there few works that investigate why and how the pruning mechanism works (Frankle et al.) 2019 Elesedy et al., 2020).
In this work, we empirically investigate standard framework for network pruning: pretraining large network and then pruning and retraining it. The framework has been commonly used based on heuristics, i.e. finding a good minima with larger network and retaining with careful pruning and retraining (Han et al., 2015) Liu et al. (0077). We investigate the heuristic in two parts, i.c., one for the pretraining phase and the other for the pruming and retraining phase.
For the pretraining phase, the reason for training the large network to obtain a good minima is in- vestigated. Since the neural network is generally compressible, the pretrained large network can be pruned to a smaller one. However, a network with the same number of weights as that of the pruned network cannot achieve similar performance when trained from scratch (Frankle Carbin 2018). We conjecture that this comes from the networks not utilizing all of their weights. Thus we hypoth- esize: if trained from scratch, there a utility imbalance among the weights neural network. For investigation, the measures for the weight utility and the utility imbalance are proposed. Thereafter, the cause of the utility imbalance and the characteristics of the weight utility in various conditions examined.
For the pruning and retraining phase, we verify the heuristic that once a good minima is obtained with the large network, it can be retained by careful pruning and retraining (Han et al. 2015 Renda jet al. 2020). Our investigation based on the loss surface visualization on a two dimensional plane formed by three points in the weight space where each point represents the pretrained network, the

---

Under review as a conference paper at ICLR 2021
100
Average Accuracy (%)
80
Accuracy (16)
69-
60
0.15
0.25
20
01.55
0.1
0.15
0.2
0.25
Ablation Ratio
Magnitade
Figure 1: The examples of the imbalance in weight utilization. (left) We trained a network follow- ing the procedure for 1x network in Section [.2.1] The weights network and the training accuracy was measured for 500 times. The average accuracy is plotted with the minimum and maximum values in the bar. Note that the accuracy difference about 60% when the ablation ratio is 0.1. This implies the utility imbalance among the weights. (right) The training accuracy with respect the ablation regarding the magnitude the weights. Each legend indicates the ablation ratio. Note that the training accuracy drops only when the weights with the
randomly ablated from the
large magnitude is ablated. The utility of the weight is biased with regard to the magnitude.
pruned network, and the pruned and retrained network. We examine (1) the dynamics of the network on the loss surface throughout pruning and (2) the validation accuracy of the networks over varying pruning methods and retraining methods.
Contributions.
The utility imbalance among the weights increases during optimization.
The neural networks utilize the weights in proportion their size.
If pretrained network is carefully pruned and retrained, then the pruned-and-retrained network shares the same loss basin with the pretrained network.
2 WEIGHT UTILITY ANALYSIS FOR THE PRETRAIN MECHANISM
Then why do we have to train a large network and then prune to a smaller one? Why not just train the smaller one to get the performance we need? Why is it difficult? The investigation about the questions starts with hypothesis: let Niarge be large network that does not utilize all of its weights, and thus can be easily compressed into a smaller network Npruned with minimal loss change. And let Nsmall be network trained from scratch, whose number of weights is comparable to that N Npruned, which sufficient to achieve a similar level of loss to those of Narge or N proned- However, Namall generally performs worse, because Namall does not utilize all of its weights either. Therefore, we hypothesize that the neural network does not utilize all of its weights when trained from scratch in general. And we refer to the phenomenon which the neural network utilize the weights unevenly as utility imbalance. Thus,
Main Hypothesis. If trained from scratch, there is utility imbalance among the weights in neural network.
And empirically measure the utility of weights
Definition (Utility measure). Let W be a set of total weights a network N, W, be a subset of W, w. (x) probability mass functions resulting from softmax layer, where ~ X is input and fwo,w w. (x) is obtained by zeroing out the weights in E [dKr. (fw(x), fw(,, (x))]. where x~X
and X be dataset. Suppose fw (z) and fw,w,
W. Then, the utility of W, can be measured U U(W.)
dxr. is KL-divergence.
For reference, the way of the measurement. i.c., network ablation, was similarly done in (Casper al. 2019 2020 Meyes et al. 2019 Cheney al. 2017). We also define the utility imbalance
2

---

Under review as a conference paper at ICLR 2021
SamlardDDeian 0.6
Mean
Misisunn
30
100
200
Egucc
Epoch
Rpech
Figure 2: The weight utility characteristics during optimization. Each legend indicates the ablation ratio. (left) The measured value of the utility imbalance. i.e., the standard deviation of the weight utility of the sample set. set. weight utility in the sample set. The corresponding training accuracy is in Figure 7
(middle) The average weight utility of the sample set. (right) The minimum
Definition 2 (Utility imbalance). For W2 W and W3 C W, say there is d utility imbalance between the subsets of weights when (U(W) - U(W,) 5.
Where we empirically measure the utility imbalance by:
Definition 3 (Utility imbalance measure). For set of the randomly drawn subsets of W, i.e., (W:) W; C. W. which refer to sample set, we empirically measure the utility imbalance of the by the standard deviation of (U(W))
We empirically show that the utility imbalance among the weights exists in Figure | Hereafter, the cause of the utility imbalance and the characteristics of the weight utility discussed in Section 2.1
and 2.2 respectively.
2.1
THE CAUSE OF THE UTILITY IMBALANCE
In this section, we investigate the cause of the utility imbalance among the weights. The cause driven from initialization and optimization is discussed in Section 2.1.1 and 2.1.2 respectively.
2.1.1 FROM INITIALIZATION
The utility imbalance among the weights can given by initialization. The Lottery Ticket Hypoth- esis (Frankle & Carbin 2018) showed that there is a fortuitous subset of weights at initialization, namely winning tickets, which achieves commensurate accuracy with the original network within the commensurate training time when trained in isolation. Since they uncovered the winning tickets by pruning the weights with the smallest magnitude after pretraining the whole network, we can say that the weights in the winning ticket are the ones that become the largest when trained whole. And since the weights with the larger magnitude tend to be more utilized than the others (Han et al.] 2015), it can be inferred that the weights in the winning ticket the well-initialized ones that become the most utilized ones after trained as a whole. However, the winning tickets become less effective when the subnetworks
trained be far apart from the initialization, e.g., trained with a larger learning rate or using ResNet (He et al.) (016) or Batch Normalization (Toffe & Szegedy) 2015 Hoffer al. 2018), where the gradients flowing through the network are larger (Frankle & Carbin 2018).
2.1.2 DURING OPTIMIZATION
The utility imbalance among the weights also be intensified during optimization. Although Frankle & Carbin (218)) conjectured that SGD may seek out and train the winning tickets, they did give further evidence. Here, conduct experiment to measure the utility imbalance during training.
We classified CIFARI0 dataset (Krizhevsky) 2009) using vanilla convolutional neural network (vanilla CNN) (the architecture of the network is specified Section [A.2.I). trained by SGD with momentum and weight decay. The network trained for 200 epochs with initial learning rate

---

Under review as a conference paper at ICLR 2021
of 0.1, which is reduced by 10x at 50% and 75% of the training epochs. When measured the utility of a subset (refer to Definition 1), the weights in the subset were zeroed out for the exclusion, and KL-divergence
used to measure the distance between the probability outputs of the original network and the ablated network. For each subset, the distance was measured by averaging over the total dataset. To measure the utility statistics, e.g., utility imbalance, we used a set of 500 randomly selected subsets (N = 500 in Definition 3) which we refer to as sample set. We only used the training data in the experiment for two reasons: (I) it is more straightforward to interpret and (2) in our experiments, the validation accuracy was higher when the training loss was lower, thus we regarded that the probability output of the training data represented the validation accuracy to some degree.
From the left figure Figure 2 we can see that the utility imbalance, i.e., the standard deviation of the utility of the subsets in the sample set, increases during training. Additionally, the mean of the utility increases (Figure middle), which implies that the network is utilizing the weights more effectively and is becoming sensitive to the ablation as training proceeds. Because the utility given
the amount of ablation (18)) can be interpreted as AS(W + it is also inferred that the loss surface Ad
is becoming sharper. It corresponds with the previous work (refer to Ghorbani et al. [2119] Figure 3), where the scale of the eigenvalues of the Hessian with respect to the weights grows significantly over training. The increase in the average utility can be the reason for the increase in the utility imbalance statistically analyzed, the standard deviation of data grows larger when the data is scaled to be larger; ablation sets differ to a greater extent as the loss surface becomes sharper. Moreover, we can see that the minimum of the utility in the sample set also increases (Figure 2 right), which infers that each of the weights is struggling to be utilized more, rather than SGD purposely differ the utility among trained the network much longer, i.e., 5000 epochs, where we put the result in Figure [1] We also conducted the experiments with the most popular architecture for classification, i.e., ResNet20 (He et al., 2016) and Batch Normalization (loffe & | and more advanced optimization techniques, i.c., momentum and weight decay. Please refer to Figure 12 and Figure [13 for the results. The results also hold for the above statements.
if intuitively interpreted, the network outputs with respect to the different
the weights. This still holds when
2.2 THE FEATURES OF THE UTILITY IMBALANCE
In this section, we investigate how neural networks utilize weights under different conditions. We show the characteristics of the weight utilization in different-sized networks in Section [.2.1] and in pruned networks in Section 2.2.2
2.2.1 UTILITY IMBALANCE AND NETWORK SIZE
Here, we investigate how the different-sized networks utilize their weights. For fair comparison, instead of adjusting the width of the layers, i.e., the number of output channels of the layers, we randomly pruned the network at initialization and re-scaled the variance of the remaining weights according to the original initialization method, i.c. He initialization (He et al. 2015). This to control the number of feature maps and thus avoid any architectural bias. We compared ResNet20 (He et al. 2016) with Batch Normalization (loffe & Szegedy 2015) and its sampled subnetworks whose number of weights are 0.5x and 0.25x of that of the original network. The network was trained by SGD with momentum (a- 0.9) and weight decay (A 10-4). Other conditions are the same as the experiment in Section 2.1.2] We performed the ablation experiments under two different settings: where the number of ablated weights are (1) proportional the number of weights in each network (which is notated as Ablation Ratio in Figure and (2) the absolute number of weights that are equally applied to the networks (Ablation Number in Figure B Each scale of the ablation number in Figure is proportion to the number of weights in the x network.
From Figure 3 we observe that no matter how large the network is, there utility imbalance among the weights. Moreover, the three networks show the similar responses when the weights are ablated in proportion to the network size (Figure topp, but quite different when ablated by the same number (Figure 3 bottom). This is quite remarkable in that the network utilizes the number of weights proportional to its size rather than the absolute number of weights required to map a certain function. Casper et al. mentioned this way of weight utilization initialization (refer to Casper et al. 2020 Appendix B). Thus, it is inferred that the tendency endowed from initialization remains to the end of training.
4

---

Under review as a conference paper at ICLR 2021
100
1.%
Sunsaard Deviatbon
1.5%
0.5%%
0.5
0.5x
0.86
8.1
0.15
0.2
0.25
AMation Ratão
100
8.5%
Stansard De ioitie
1.5%
100
6.2%
5%
8.1
0.15
Ablation Number
Abblation Number
Figure 3: The characteristics of the weight utility of the networks with the various sizes. Each legend indicates the size of the network. (top) The characteristics when the weights are ablated by ratio. (bottom) The characteristics when the weights ablated by the absolute number. Note that the characteristics of the networks are similar when ablated by ratio.
200
150
Nintoun
50
325
8.05
6.1
0.15
0.2
Abbrlion Ratio
(Xillry
Figure 4: The other characteristics of the weight utilization. Each legend indicates the size of the network. (left) The maximum weight utility the sample set. (middle) The minimum weight utility in the sample set. (right) The histogram of the values of the weight utility when the ablation ratio is 0.05. Please refer to the histograms with other ratios in Figure 14
By taking closer look, even when compared by ratio where the similar tendency is shown, the actual number of weights still matters to a lesser degree. For example, the maximum weight utility among the random weight sets (Figure 4 left) is generally higher in the smaller network. This implies that the most utilized weights the smaller network are more capable than those in the larger one, even if more weights considered for the larger network. This is counterexample for the conjecture in the Appendix in Frankle & Carbin 2018). which says the larger network shows superior performance because it has better winning tickets. If the larger network had better ticket with the same amount (ratio) of weights, its most utilized weights should have been more critical than those of the smaller one. Another interesting result comes from the average weight utilization (Figure B] top middle). It is shown that the output of the larger network differs from that of the original network by the same level by the larger ablation ratio. This implies that the larger network relies on the larger portion of the weights. In addition, the larger network tends to less utilize the weights when ablated by the same ratio if the network has the ability to classify. This is in line with the findings of Casper al.) where the larger network was regarded to be more robust to the ablation Casper et al., 2019). In summary, the larger network uses the less-utilized weights by the larger portion. We conjecture this may be reason for the larger network showing the better performance. The tendency is also consistent with the maximum and the minimum of the utility, and confirmed by the histogram of the utility values in Figure
5

---

Under review as a conference paper at ICLR 2021
14
Frused Prstrained
1.2
Standard S Deviation
Seratch
0.6
0.4
Prunel
Prvtrnited
Scratch
0.05
0.1
0.15
0.25
0.05
0.1
Ablation
Ablation Ratio
Figure 5: The characteristics of the weight utility in the pruned-and-retrainn network. The legends indicate the mameed-andreetaiine network, the pretrained network and the small network, respec- tively. Note that the characteristies of the comed-and-retraann muned-and-retrainee one, even though we used the large learning rate for retraining.
network are similar that of the
2.2.2 UTILITY IMBALANCE PRUNED NETWORK
Here, we investigate the utility characteristics pruned network. The network architecture and optimization scheme are the same as the experiments in Section 2.2. We used the one-shot global magnitude pruning with respect to the individual weights, and only the weights in the convolution layers were pruned for convenience. We retrained the pruned network for another 200 epochs with the same learning rate schedule used for the pretraining (Renda et al. 2020). For comparison, we trained small network which is trained from scratch and has the of the pruned network. Especially, we trained the small network for 400 epochs to match the total training epochs of the pruned network. The learning rate schedule was also the same for the pruned- and-retrained network, which is twice the repetition of that of the pretraining. The utility statistics were acquired as the same as the experiments in Section 2.1.2] For the pretrained network, only the weights with the larger magnitude than pruning threshold were considered for the subsets for the utility measure (W, in Definition 1). This was to control the weights of interest, since only the weights larger in magnitude than the pruning threshold remain in the pruned network. Figure 5 is the result for the pruning ratio 0.5, and Figure 15 is for 0.25 and 0.75.
number of weights that
In Figure 5 the the pretained network. surprising that we used large learning rates for retraining the pruned network, which were large enough for the pretrained network to achieve 100% training accuracy from a random state. Although the statistics of the pruned-and and-retrained network differ from those of the pretrained network as the pruning ratio increases (Figure 13). still the former resembles the latter, even when the pruning ratio is 0.75. From the results, we conjecture that the high performance of the anddrri-ndd network comes from the weight utility characteristics endowed by the large pretrained network, c.g., relying on the larger portion of the less-utilized weights, as in Section 2.2.
network exhibits the similar utility characteristics to those of
3 Loss SURFACE ANALYSIS OF THE PRUNE-RETRAIN MECHANISM
Canonical pruning methods begin with training a large neural network, followed by identifying and removing portion of less important parameters for compressing the network, yet preserving the accuracy. Such pretrain- and-prune paradigm is motivated by an implicit assumption that: (1) large network trained with SGD optimization has higher chance of converging to minima with less generalization gap, and (2) pruned and retrained network converges to a minima with a close relation to that of the pretrained network. In this regard, we investigate the relation between the pretrained, pruned and retrained networks in this section.
Empirically, the generalization gap is known to be closely related to the geometry of the loss basin. Hochreiter& & Schmidhuber (1995) first proposed that flat loss minima corresponds to less overfitting. Jiang al. (2020) showed that measures based flatness of minima such sharpness (Keskar et al. (2117) are highly correlated with generalization performance. Also, Li et al. (2018) visualized and showed that larger networks converge flatter
the loss minima of family of

---

Under review as a conference paper at ICLR 2021
Fine-tuning
Rewinding
10
12
10
12
0-
100
0-
100
W
W
Weight Pruming
5-
W
W
W'
W
WP
48.09
W?
49.28
14
14-
10
12
10
100
2-
800
W
W
W
W
Channel
Pruning
8-
8-
W
W
10-
W
W
26
83
I W'1
66.99
W* 25.60
w>| 62.32
Figure 6: The visualization of the pretrained, pruned and retrained weights on the accuracy surface. The color map indicates training accuracy (%). Refer to Section A.5.2 for the trajectory change over the pruning cycles.
minima. This is consistent with the empirical observations that larger neural networks have smaller generalization gaps.
Likewise, we investigate the relation between the networks by visualizing the loss surface. To the best of our knowledge, there has been been clear verification on whether the assumptions for the heuristic hold. In order verify such assumptions and shed some light on what happens through pruning, we visualize the trajectory of the pruning mechanism on a fow-dimensional loss surface.
Pruning and retraining methods. The typical pruning algorithms prune and retrain the network iteratively to achieve high sparsity and avoid loss in accuracy. The iterative pruning cycle consists of two alternating phases: the pruning phase and the retraining phase. At the pruning phase, portion of weights or channels are removed. The weight pruning methods aim to identify the importance of the individual weight of neural network, while the channel pruning methods target the channels in a convolutional layer. Therefore, the channel pruning methods operates in a higher constraint, which often results in greater damage to the network. At the retraining phase, the pruned network is commonly fine-tuned using small and fixed learning rate, which is usually the last learning rate (2020) has shown that rewinding the learning
used the pretraining stage. In addition, Renda rate scheduling to an earlier time step can boost the validation accuracy of the retrained network.
Visualization technique. For straightforward understanding, we plot the accuracy surface instead of the loss surface in this section. To visualize the points of interest, we follow the 2D planar loss visualization used in Garipov et al.] (2018). We construct the 2D plane by affine combinations of three points in the weight space the pretrained weight (W"), the pruned weight in the last iterative cycle (WP), and the final retrained weight (W").
Experiment details. For the experiments, CIFAR10 dataset (Krizhevsky, 2009) and ResNer20 (He) jet al., 2016) were used. At the pretraining stage, we use the same training hyperparameters and data augmentation strategy described in He et al. (2016); with SGD momentum decay X = 10 The pretrained network was trained for 200 epochs, with an initial learning rate of 0.1 divided by 10 50% and 75% of the total training epochs. The number of retraining epochs was also 200 for each cycle. For comparison, the small networks with the same number weights as that of the corresponding pruned networks trained from scratch. The small networks are set by
0.9 and weight

---

Under review as a conference paper at ICLR 2021
Table 1: Validation accuracy comparison. "Prune" refers to the gruned-and-retraiin network and "Small" refers to the small network trained from scratch. Note that the aggressive channel pruning with fine-tuning shows the low performance, where often collapsed to 10% accuracy.
Pruning method
Retraining method
Pruned
Small
fine-tuning rewinding
90.870
weight pruning
91.897
89.290
fine-tuning rewinding
43.375
channel pruning
87.138
88.020
randomly pruning the large network at initial and rescaled the variance of the weights regarding the original initialization, i.e. (He et al., 2013). To be fair, we used the same number of training epochs and the same learning rate schedule as those of the 2018) for training the small network.
network (L.iu et al.)
For the weight pruning, we used the magnitude pruning method (Han et al. 2015). And for the channel pruning, we used the criteria proposed in Network Slimming (Liu et al.) 2017). To show the failure case of pruning, we applied the method much more aggressively (Section A.5.I). For each experiment, of weights per cycle for the weight and the channel pruning, respectively. Only the weights in the convolution layers were pruned for convenience. For retraining the pruned network, two learning rate scheme were used: fine-tuning and learning rate rewinding (Renda et al. (020).
performed the iterative pruning over five cycles. We removed 30% and 20%
Results. In the case of the weight pruning, consistently observe that the pretrained weight and the pruned-and-retrainee weight are connected by the high-accuracy region on the accuracy surface (Figure [6] top). This indicates that the final weight placed in the original loss basin (accuracy peak). Moreover, we find that the retrained weight vector remains in the same loss basin even with a aggressive retraining method, i.e., learning rate rewinding (Renda al. 2020p. Table 2 also shows that the pruned network has consistently better performance than the small network trained from scratch, which in line with our observation in the visualization. It is implied that the pruned- and-retrained network is likely to remain in the flat loss basin reached at the pretraining stage. This observation is in agreement with the assumption that the pretrain-and and-nrune practice takes advantage of the large network to attain higher generalization performance.
On the other hand, in the case of the aggressive channel pruning, we observed that the two points are possibly in the separate loss basins when retrained by learning rate rewinding (Figure bottom right). We conjecture this came from the aggressive channel pruning and the large learning rate of the learning rate rewinding. It is provable that the pruned weight shifted too much from the pre- trained weight W such that the retrained point W locates in a different loss basin. To give further evidence, compare the accuracy of the aruned-and-retrrained network and that of the small net- work (Table 2 The result shows that the sruned-and-retrained network has a higher generalization gap than the small network, implying that it did not benefit from the pretrained network. Moreover, when using the fine-tuning method, the retrained network could not even recover from the pruning and remains in the low accuracy region (Figure 6 bottom left). Overall, for the aggressive pruning, the result does not seem to work as the initial assumptions.
DISCUSSION
A standard pruning framework, i.e., pretraining a large network and then pruning and retraining, was examined. To investigate the pretraining phase, we defined the measures for the weight utility and the imbalance of the weight utility. The cause of the weight imbalance and the characteristics of the weight utility were discussed. For the pruming and retraining phase, the relation between the pretrained network and the pruned and retrained network was investigated using the accuracy surface and the validation accuracy of the networks. The various conditions were examined to verify the heuristic.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Stephen Casper, Xavier Boix, Vanessa D' A'mario,, Christopher Rodriguez, Ling Guo, Kasper Vinken, and Gabriel Kreiman. Robustness and/or redundancy emerge in overparametrized deep neural networks. 2019.
Stephen Casper, Xavier Boix, Vanessa D' Amario, Ling Guo, Martin Schrimpf, Kasper Vinken, and Gabriel Kreiman. Frivolous units: Wider networks are not really that wide, 2020.
Nicholas Cheney, Martin Schrimpf, and Gabriel Kreiman. On the robustness of convolutional neural networks to internal architecture and weight perturbations, 2017.
Bryn Elesedy, Varun Kanade, and Yee Whye Tch. Lottery tickets in linear models: An analysis of ilerative magnitude pruning. arXiv preprint N.KYv:2007.082444 2020.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Brich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv: 10732, 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. arXiv preprint arXiv: 0.677, 2019.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems, 2018.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density. arXiv preprint 1xXiv:9901.10155 2019.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323, 2011.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034. 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference computer vision and patlern recognition, pp- 770-778, 2016.
Yang He, Ping Liu, Ziwei Wang. Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4340-4349 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Advances in neural information processing systems, pp. 529-536, 1995.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. Advances in Neural Information Processing Systems, pp. 2160 2170-2177, 2018.
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015.

---

Under review as a conference paper at ICLR 2021
Yiding Jiang. Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. In International Conference on Learning Representations, 2020.
Simonyan Karen and Zisserman Andrew. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 04 2009.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Neural Information Processing Systems, 2018.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference Computer Vision, pp. 2736-2744, 2017.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2018.
Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen. Ablation studies in artificial neural networks, 2019.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference Learning Representations, 2020.
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. In Advances in Neural Information Processing Systems, pp. 2133-2144 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), pp. 87.1-87.12, 2016.
A APPENDIX
A.1 RELATED WORKS
Pruning methods. Network pruning attempts to compress pretrained deep neural networks by iden- tifying and removing parameters that estimated to be less important for the network. Unstruc- tured pruning methods remove parameters at a fine- fine-graing -oee-graind level by removing each of the weights individually. (Han et al. (2015) proposed method that assesses the importance of weights based on their norm, and also performs an additional fine-tuning step after pruning in order to compensate for the loss of parameters. Structured pruning on the other hand, removes parameters filter or kernel level. [iu et al. (2017) came up with a method that prunes channels with small BatchNorm (loffe Szegedy] 2015) scaling parameters, which have been pushed towards zero with LI regularization. Recent methods You et al.) 2019 He et al., 2019) focus on pruning networks with ikip-connections (He et al.] (2066). He et al. (2019) prunes 53.4% of parameters with slight accuracy drop. In this work, we want to reveal some of the reasons why deep neural networks can be successfully pruned.
larger,
Loss surface. A streamline of works focus on the loss surface of neural networks and its properties. Garipov et al. (2018) introduce the concept of mode connectivity; that simple linear pathways be- tween local minima can be discovered. They demonstrate ways of finding and visualizing the loss landscape along such pathways on various modern architectures. (Evci et al., 2019; Frankle et al.)
2019) analyze sparse neural networks in terms of mode connectivity, Frankle et al. 2019) discov- ered that the existence linear paths is a key indicator of whether lottery ticket networks (Frankle
& Carbin 2018) can be discovered, On the other hand, Evei et al. (2019) found that linear paths
10

---

Under review as a conference paper at ICLR 2021
between a pruned network and network randomly initialized from the same sparsity pattern separated by high energy barriers. In this context, our paper brings new insight by exploiting mode connectivity to analyze the minima obtained before and after pruning.
Lottery ticket hypothesis. The lottery ticket hypothesis(Frankl & Carbin 2018 explores the existence of sparse subnetwork that reach the commensurate performance of dense network within the commensurate time when trained isolation. The subnetwork discovered by pruning pretrained network based the magnitude of its weights, whereas the surviving weights restored back to their values at initialization. Such weights are called winning tickets, and a pruned network can be obtained by retraining them. The authors verified the existence lottery tickets on the CIFAR-10 10 dataset with fully connected and convolutional networks. However, currett methods of finding lottery tickets are only restricted to certain settings, i.e, learning rate warmup and iterative pruning. Moreover, the mechanism of the lottery tickets is still open area of research. |Elesedy al. (2020) is work that provides an insight to the mechanism with linear models, where they reformulated the iterative magnitude pruning
process of feature alignment.
Learning rate rewinding. Existing literature on pruning adopt a retraining phase after pruning to compensate for the loss of the network. The most commonly used method is fine-tuning, which is process of additional training with a small and fixed learning rate (Han et al., 2015 Liu et al. 2018), i..., typically the last learning rate used for pretraining. Mcanwhile, Renda et al. 2020) suggested a new learning rate scheme which is referred to as learning rate rewinding. Unlike fine-tuning, they propose to rewind the learning rate schedule the earlier phase in pretaining. In addition, they explored whether the weight rewinding (Frankle & Carbin 2018) is beneficial. On various datasets such as CIFAR-IO 10, ImageNet, WMT-16 dataset, they verified that both of the rewinding methods were better than fine-tuning.
A.2
ADDITIONAL MATERIAL FOR SECTION 2.1
A.2.1 THE ARCHITECTURE OF THE VANILLA CNN
The vanilla CNN used for the experiment was composed of (conv3 3x 3x16)-(conv3 16x 16x (coonv3 (mxppoo-maoool)cccccoo 16x32)-(conv3 <mx3)ommaapoo 32x64) 64x64)-(global average (fo 64x10). A convolution layer with spatial size of 3 x 3 is indicated as 'conv3', And 'mxppool' and 'fc' indicate a max pooling layer and a fully-connected layer, respectively. There is a Rectified Linear Units (ReLU) (Glorot et al. 2011) after each convolution layer.
A.2.2 THE TRAINING ACCURACY FOR EACH ABLATION IN FIGURE
80
Average Accuracy
0.01
50
0.075
8.1
0.25
20
50
150
200
Epoch
Figure 7: The training accuracy for each ablation in Figure Each legend indicates the ablation ratio.
11

---

Under review as a conference paper at ICLR 2021
A.2.3 THE DETAILED FIGURES FOR FIGURE
Since the data do not clearly show the tendency in the combined manner, we present the detailed The figures are ordered with respect the corresponding
figures for Figure in Figure g T0 ablation ratio of each datum, I.e., 0.01 0.1.0......07550111155.....22..2 In the figures, the statements in Section 2.1.2 generally hold.
Ablation ratin: [UIl
Ablation ratio: 0.85
Abiotion ratlo: 0,055
0.04
0.5
Shundard Devistion
0.83
Stuneard Deviaaiies
0.3
Stondaed
0.2
0.4
0.81
100
233
100 Epoch
1.98
1.8
200
Epoch
Ablition
Ablation
Ablation ratie: 0.25
Sambard Deviattis
Standard Deviation
Stondard Deviaation
0.5
50
150
200
50
150
200
Egorh
Epoch
Epuch
Figure The standard deviation values the weight utility the sample set.
Ablation ralinc 8,01
1.4
2.5
1.2
0.15
Moon
Mean
0.6
0.6-
0.4
0.2
59
100
150
201
50
100 Egooch
158
200
90
100 Egoch
159
200
Epoch
Ablution rutio: 0.15
Ablation ratio: 0.25
Mean
90
100 Epoch
159
200
90
180 Fpoch
150
100 Epoch
150
200
Figure 9: The average values of the weight utility in the sample set..
12

---

Under review as a conference paper at ICLR 2021
Ablation
0.04
0.5
0.4
Misium
0.02
Mis
0.2
0.01
50
100
1.0
200
50
100
150
200
30
100 Eigoch
1.9
200
Epuch
Fijuch
Ablation ratio: 0.25
Sininunn
Minimu
Minimuss
1.4
0.5
20
100
1.0
203
50
100 Epoch
150
200
150
Epoch
Epoch
Figure 10: The minimum values of the weight utility in the sample set..
13

---

Under review as a conference paper at ICLR 2021
A.2.4 THE CHARACTERISTICS OF THE WEIGHT UTILITY DURING OPTIMIZATION VARYING CONDITIONS.
The longer training epochs With the same setting as the experiment in Section 2. we trained the network much longer than the conventions, i.e., 5000 epochs. It about 25x longer than the typical settings. The initial learning rate of 0.1 was reduced by 10x at 50% and 75% of the total training epoch.
1.2
Standard Deviation
0.8
0.01
0.01
0,85
6
0,05
Mean
0.6
0.1
0.1
0.4
2
0.2
0-
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Epoch
Epoch
90
0.01
Average Accuracy
Minimum
60
0.05
0.075
0.075
0.15
40
0.15
30
10
2000
5000
1000
2000
3000
4000
5000
Epoch
Epoch
The utility characteristics of the vanilla network when trained for 500 epochs. Note the severe accuracy drop compared to that of Figure 7 This corresponds with the increase in the average weight utility, the increase in the difference between the outputs of the ablated and the original network might cause the accuracy drop. In particular, the tendency is shown after the early phase of the training where the network undergoes an abrupt transition (approximately before the epoch 1000). Each legend indicates the ablation ratio.
Figure 11:
14

---

Under review as a conference paper at ICLR 2021
An experiment with the advanced settings Here, the experiments done with the advanced
settings. We used ResNet20 (He et al. 2016) with Bathch Normalization (loffe Szegedy) 2015))
trained by SGD with momentum (a 0.9) and L2 weight decay (A 10 *), Other settings are the
same as the experiment Section 2.1
1.2
8
Standard Deviation
0.01
0.01
0.8
0.075
Mean
0.075
0.1
0.6
0.25
0.25
0.4
0.2
100
200
50
150
200
Epoch
Epoch
10
100
Average Accuracy
Minimum
0.05
0.05
0.075
0.1
0.15
0.15
0
50
100
150
200
50
100
150
200
Epoch
Epoch
Figure 12: The weight utility of the ResNer20 (He et al. 2016) with Batch Normalization (loffe Szegedy 2015), trained by SGD with momentum and weight decay. Each legend indicates the ablation ratio.
15

---

Under review as a conference paper at ICLR 2021
The longer training epochs with the advanced settings Here, the experiments done with the advanced settings and the larger training epochs. We used ResNet20 (He et al. 2016) with Bathch Normalization (loffe & Szegedy 2015), trained by SGD with momentum (a 0.9) and L2 weight decay (A = 10-)). The number of training epochs is 5000. The initial learning rate of 0.1 was reduced by 10x at 50% and 75% of the total training epoch. Other settings are the same as the experiment in Section 2.1
8
Standard Deviation
0.8
0.01
0.01
0.6
0.075
Mean
0.075
0.1
0.15
0.4
0.25
0.25
1000
2000
4000
1000
2000
3000
4000
5000
Epoch
Epoch
8
100
Minimum
5
0.05
Average Accuracy
0.01
0.075
0.1
0.1
0.25
0.25
20-
1000
2000
3000
4000
5000
2000
Epoch
Epoch
Figure 13: The weight utility the ResNet20 (He et al. 2016) with Batch Normalization (loffe & Szegedy 2015), trained by SGD with momentum and weight decay for 5000 epochs. Each legend indicates the ablation ratio.
16

---

Under review as a conference paper at ICLR 2021
A.3 ADDITIONAL FIGURES FOR SECTION 2.2.
A.3.1 THE HISTOGRAMS OF THE WEIGHT UTILITY FOR THE DIFFERENT-SIZED NETWORKS.
Here, we show the histograms of the weight utility for the different-sized networks. The figures are ordered as the corresponding ablation ratio, i.e., 0.01, 0.05, 0.075, 0.1, 0.15, 0.32.
500
200
400
Frequency
Frequeney
100
100
0
2
3
4
2
Utility
Udility
(a) Ablation ratio = 0.01
(b) Ablation ratio = 0.05
150
120
0.5%
0.5g
0.25k
0.25
100
Frequency
Frequeney
100
80
60
40
20
Udility
Utility
(c) Ablation ratio 0.075
(d) Ablation ratio 0.1
200
340
6.25g
Frequency
Froquency
200
100
1.50
50
5
8
10
7
10
Utility
Utility
(c) Ablation ratio = 0.15
(f) Ablation ratio 0.25
The histograms of the weight utility for the different-sized networks. Note that average and the standard deviation of the weight utility are higher in the smaller network when the networks have the ability to classify. And the distributions become similar as the networks output random chances. Each legend indicates the size of the network.
Figure 14:
17

---

Under review as a conference paper at ICLR 2021
A.4 ADDITIONAL FIGURES FOR SECTION 2.2.2
A.4.1 THE CHARACTERISTICS OF THE WEIGHTT UTILITY IN THE BRUNED-AND-RETRANNG
NETWORK WITH VARYING PRUNING RATIO
Here, we show the characteristics of the weight utility in the pruned-and seneed--nd-rttrainee network with varying pruning ratio.
1.4
10
Pruned
Pretrained
Serutch
Standard Deviation
0.8
Meun
0.4
Pruned
0.2
Pretrained Seratch
0.25
Ablation Ratio
Ablation Ratio
(a) Pruning ratio 25%
1.4
Pruned
Pretrained
Standard Deviation
0.8
Meun
0.6
Pruned
0.2
Pretrained
Seratch
0.15
0.25
0.05
0.1
0.15
0.2
0.25
Ablation Ratio
Ablation Ratio
(b) Pruning ratio 75%
Figure 15: The characteristics of the weight utility in the pruned-and and--anddi----ee network with varying pruning ratio. (Top) the characteristies of the networks when the pruning ratio was 25%. (Bottom) the characteristics of the networks when the pruning ratio was 75%. Although the pruned-and- retrained network shows the similar characteristics to those of the pretrained network, gets differ- ent the pruning ratio is increased.
18

---

Under review as a conference paper at ICLR 2021
A.4.2 THE TRAINING LOSS AND THE VALIDATION ACCURACY FOR THE EXPERIMENTS.
Here, show the training loss and the validation accuracy for the experiments used for Figure and Figure 15
Table 2: Validation accuracy comparison. "prune" refers the pruned-and-reerained network, and "small" refers to the small network with the same number of weights as that the pruned network, which is trained from scratch.
Pruned ratio (%) Training type
Validation accuracy (%
Train loss
prune
92.43
0.008
25
small
91.14
0.016
prune small
91.84
0.009
50
90.63
0.033
prune small
91.41
0.026
75
89.17
0.105
A.5 ADDITIONAL MATERIAL FOR SECTION 3
A.5.1 THE DIFFERENCE BETWEEN NETWORK SLIMMING AND OUR CHANNEL PRUNING METHOD
Here, we describe the difference between network slimming (Liu al. 2017) and our channel pruning method. Most importantly, the sparsity constraint was not used for fundamental analysis. The pruning ratio constraint for each layer was also ignored (refer to Liu et al., 2017 Section 4.5), so that our pruned-and-retrained network often collapsed when the small learning rate used for retraining (fine-tuning, please refer to Table 2]. Also, we pruned the channel by 67%, which is by far above the recommended level (refer to Liu et al., 2017 caption of Figure 1). We did not use the best hyperparameter for the method either, e.g., the scaling factors of the Batch Normalization layers (loffe & Szegedy 2015), which used the conventional value, i.e., 1, unlike the value used for the method, i.e., 0.5.
19

---

Under review as a conference paper at ICLR 2021
A.5.2
WEIGHT TRAJECTORY ON LOSS SURFACE OVER ITERATIVE PRUNING PROCESS
W
W
W
W
W
W
W?
W
W
10
12
|
Wi
20
12-
IW-
-00
12-
W"
5466
W.
48,
7366
1000
W2
48.
5.08522
14
14-
(a) Weight pruning, fine-tuning
W
W
W
W
W
W2
W3
W
W
IW'
I"
49.5095
10
54.54
IIW
W3]
48 8605
|
W
49.2611
14
(b) Weight pruning, rewinding
100
50
W
W
W
WT
W
4
W
W
W
|
Will
8.7481
I00
22 5400
W
W W
26.
W 5..174
18.00000
|00"
25.5950
(c) Channel pruning, fine-tuning
100
100
100
50
W
W
W
WT
W
W
60
WP
W
W
||
W|||
40, 80.8877
I
WT 1= 58 2783
I
Wil
65 9854
W8
I
W3
(d) Channel pruning. rewinding
Figure 16: Loss surface visualization of each pruning method and retraining method. On each plot, W*, W.P, WT refers to the pretrained weight, the pruned weight, and the retrained weight. The number of pruning iteration is indicated by the subscript i. Note the transitions in the accuracy surface over the cycles.
20


Reviews:
[{"review_id": "nwL0qw6KYNZ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# After Rebuttal\n\nI have read the response to my review and the responses to the other reviews. The summaries of the paper in the other reviews helped to clarify my understanding of the research question the authors were aiming to answer and how they went about doing so. I have re-read the paper and the revisions have helped to make this story clearer.\n\nWith that said, I remain concerned with many technical aspects of the paper, for example:\n* The scale of the networks studied.\n* I still don't understand why this particular definition of utility imbalance is well-motivated.\n* I generally don't think that two dimensional loss landscape visualizations are informative since they discard an enormous amount of information from the full loss landscape. To show that minima are related, I think it is better to use interpolation (mode connectivity).\n\nI am also still concerned about the writing. The revisions, alongside the other reviews, were enough for me to get a sense for the research question and technical story, but I still struggled to make sense of the details.\n\nSince the other reviewers appear to have better understood the paper, I have raised my score to a 4 and decreased my certainty to a 2. I suggest that the AC weight my review much less heavily than the other reviewers, who seem to have better understood the technical details.\n\n# Overall\n\nThis paper is inscrutable. The research question is unclear and the hypothesis is vague and imprecise. The metric being examined (\"utility imbalance\") is never justified in name or in definition, and broad, sweeping, unsubstantiated claims are made that it has to do with how the network uses its weights or the sharpness of the minima (among other aspects of deep learning). Section 3 seems completely unrelated to the previous analysis, and I don't understand how Sections 2 and 3 relate to form a cohesive story. I can't make sense of this paper - its question, its claims, or its method - and I'm an expert reader on topics of pruning and lottery tickets. I can't imagine what a non-expert reader would be able to glean from this paper.\n\n# Score\n\nI therefore strongly recommend rejection (score of 2).\n\n# To Improve My Score\n\nTo Improve my score, the authors need to:\n* Explain what the research question is, precisely.\n* Explain what the hypothesis is, precisely.\n* Explain (in detail) what the justification is for \"utility imbalance\" and why it corresponds to meaningful behaviors of a neural network. The authors will need to include evidence that this corresponds to the claimed attributes of a neural network, which will include a precise definition of what it means for a network to \"utilize\" a weight and how this metric measures that.\n* Explain why this is called \"utility imbalance\"\n* Explain jhow the results in sections 2 and 3 related to each other and the larger narrative/takeaway that they provide.\n* Explain why the network under study changes mid-way through the paper. Is there a reason for this? When I see this in papers, it's usually because the authors tried this on both networks and are only showing the positive results and are hiding bad results; to assuage this concern, the authors should show all results for both networks.\n\nFrom there, I'll need to re-read the experiments to see if I can make sense of them. In its current state, I do not believe that this paper makes a contribution to the scientific literature.\n\n# Notes\n\n## Abstract\n\nWhat do the authors mean by \"the pruning mechanism\"?\n\n## Intro\n\nWhat prior work has claimed that \"the reason for training the large network [is] to obtain a good minimum\"? I don't recall having previously seen this specific claim, nor the implied negation of this claim: that training a pruned network finds a \"bad minimum.\" What makes a minimum \"good\" or \"bad\" and how can we measure this?\n\n\"Cannot achieve similar performance when trained from scratch\" - this only occurs with a new random initialization, as Frankle & Carbin 2018 show (at least for the small-scale networks they examine).\n\nWhat do you mean by a \"utility imbalance\"? Which networks aren't \"utilizing all their weights,\" the unpruned networks or the pruned networks?\n\nAfter reading the introduction, I really have no idea what the paper is about. I don't understand the research question, the main hypothesis, the methodology, or the findings. the authors should make efforts to clarify this story. I'm confused, and that means I'm going to have a hard time making sense of the rest of the paper.\n\nI also can't make sense of Figure 1. I don't know which network this refers to, and I still don't know what a \"utility imbalance\" is.\n\n## Section 2\n\nWhy is there an assumption that a large network \"does not utilize all its weights and thus can easily be compressed?\" There are many possible reasons that one can prune a network, and we often find in the literature that pruning reduces accuracy. The entire point of retraining is to recover this accuracy. But since accuracy went down, it seems that these weights did serve some purpose.\n\nIs N_small the same pruned architecture, or is it a different pruned architecture, a smaller dense network, etc.?\n\nWhat does it mean for a network to \"utilize all its weights\"?\n\nIn Definition 1, does the pruned network include retraining?\n\nWhy is this measure of utility imbalance meaningful or useful? Networks whose outputs have very different distributions may get similar accuracy, so KL divergence doesn't seem to be esxpecially meaningful here. I also don't see what this has to do with utility.\n\n#### Section 2.1.1\n\nI don't follow this logic, and I'm an exceedingly well-informed reader when it comes to lottery tickets. What is the purpose of this section? Is utility being used in a formal way or an informal way? If formal, I don't see a proof or derivation to support the claims here. This seems like an argument without any evidence.\n\n#### Section 2.1.2\n\nIt is unacceptable that the name and details of the network used for this paper are buried in an appendix. The network in question, a small convolutional network, appears to be similar to the small convolutional networks used by Frankle & Carbin. These networks are not representative of larger-scale settings for lottery ticket behavior see (\"Linear Mode Connectivity and the Lottery Ticket Hypothesis\" (Frankle et al., 2020)), and the results in this paper should not be taken to be general.\n\nThe first paragraph of 2.1.2 suggests that the \"utility imbalance\" metric has anything to do with the specified claim made by Frankle & Carbin, and I see no reason to believe that.\n\nI don't know how to interpret the \"utility imbalance.\" It has a fancy name and it has a fancy definition, but I have no idea what it means. Graphs of the utility imbalance therefore aren't very enlightening.\n\nWhy does the utility have anything to do with the network \"utilizing the weights more effectively\"?\n\nWhat does the utility imbalance have to do with the sharpness of the loss landscape?\n\nWhy are the weights and SGD being anthropomorphized? e.g., \"The weights is struggling to be utilized more, rather than SGD purposely differ the utility among the weights?\" \n\nI'm completely lost in this section, as the above comments hopefully convey. I have no idea what is being measured, why it is being measured, or why the results have anything to do with the claims. These experiments are also being conducted on a single, non-standard network, so I don't understand why these results should generally be true.\n\n## Section 2.2\n\nWhat does it mean to \"utilize a weight\"?\n\nWhy were the weights rescaled? I don't understand what \"This was to control the number of feature maps and thus avoid any architectural bias.\" means.\n\nWhy did the experiment suddenly switch networks from what was being used in 2.1?\n\nI made a good-faith effort to read the rest of the section from here, but I wasn't able to make much sense of why the evidence substantiated the claims. Please see my general comment at the beginning of the paper.\n\n## Section 3\n\n\"Empirically, the generalization gap is known to be closely related to the geometry of the loss basin.\" This is hotly contested in the literature. I'm not sure you can say this so strongly.\n\nFigure 6 needs a legend to explain what the networks are. The text never explains what W*, Wp, and Wr are.\n\nI have no idea how this section relates to the rest of the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PDmhyQfHb79", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper dives into why small pruned networks don’t train as well as large networks. They come up with a measure of weight utilization and claim that networks of all sizes only use a portion of their weights during training, and the imbalance increases during optimization. Additionally, they visualize the accuracy surface on the plane defined by the pretrained, pruned, and retrained networks and find that the retrained networks end up in the same basin as the pretrained networks.\n\n\nOverall: The paper is going in an interesting direction, but I find the paper to be unclear and I do not see how their chosen measurements lead to their claims. Thus, I do not recommend acceptance. \n\n\nPros:\n* It is important to understand the mechanisms of how pruning works, as well as the role of overparameterization in neural networks.\n* It is also interesting to see how the ratio of weights used change with respect to network size or sparsity.\n* Visualizing the loss/accuracy landscape between the three networks (pretrained, pruned, pruned+retrained) is interesting and I have not seen that before.\n\nCons:\n* The need for overparameterization to achieve good performance has been studied before (https://arxiv.org/abs/1805.12076, https://arxiv.org/abs/1804.08838), though I do see the benefit of analyzing this from a different angle.\n* There are also studies of how neural networks do not use all of their weights during training (https://arxiv.org/abs/1909.01440)\n* The given utility measure is not the best way to measure how useful certain weights were. Some weights could have been very important for moving around during training, even if they end up unimportant or at a low magnitude. For instance, overparameterization provides more degrees of freedom that may be necessary in optimization (e.g. as discussed in https://arxiv.org/abs/1906.10732). You could include additional measures such as LCA (from https://arxiv.org/abs/1909.01440) to measure how “useful” certain weights were throughout training.\n* Suggestion: rather than choosing random subsets of weights to prune/ablate in definition 3, why not prune based off of some existing measure? E.g. prune by magnitude, using the n% lowest for $W_i$ (what magnitude pruning would choose) and perhaps the n% highest for $W_j$ for comparison. Your current measure won’t be able to pinpoint if the network heavily relies on, say, a specific 10% of its weights, because pruning a random subset of weights means that you prune both the important weights and the unimportant weights. Currently, this measure only evaluates the network’s sensitivity to random pruning.\n* Section 3 results: while it is useful to visualize the accuracy surface in 2D, a 2D picture does not tell the full story for high dimensional spaces: the existence of a connector in one particular dimension does not mean that two points are necessarily in the same basin. Further, if the pretrained model and the pruned+retrained model are in the same basin, that would mean the pruned+retrained model achieves the exact same accuracy as the full model, which is not the case unless the sparsity is low. Perhaps it would be interesting to see these visualizations at different pruning levels, where accuracy begins to drop. Also, why use accuracy rather than loss?\n\nThere are also several points in the writing that are unclear. While I do not want to penalize the paper for writing style, these issues make it difficult to understand the ideas in the paper. If it is just my misunderstanding, I will gladly read any explanations that the authors can provide.\n* Definition 3: don’t you need a $\\delta$ for utility imbalance? How do you convert standard deviation to utility imbalance? How do you “show that the utility imbalance among the weights exist” if utility imbalance is not a binary measure?\n* Please give a more intuitive description of utility and utility imbalance. What does it mean for networks to have high utility imbalance given the definition you provided? I don’t agree that having a large standard deviation in utility of the random $W_i$ signifies that the neural network utilizes the weights unevenly; the implication I see is that the network’s output is very sensitive to which random subset of weights you prune.\n* Figure 1 left: why would a drop in accuracy after removing weights imply the utility imbalance? Right: what does “ablation regarding the magnitude of weights” mean? How do you choose which weights to ablate?\n* Section 2.1.1: this explains why magnitude pruning works well but does not explain how initialization causes utility imbalance.\n* Section 2.1.2: how is $|\\Delta\\delta|$ the “amount of ablation”? How can $\\delta$ be added to $W$ if $\\delta$ is used in $|U(W_i) - U(W_j)| \\geq \\delta$? Also, my guess would be that utility increases over training because at the beginning of training, the network is close to random, so removing some subset of weights won’t change it as much (can’t get much worse than random) as ablating a more trained network. Thus it is only natural that it increases - I don’t see how this supports the claim that the loss landscape became sharper. Also, what does “each of the weights is struggling to be utilized more” mean?\n* Section 3 paragraph 3: what are the “assumptions for the heuristic”?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IhHoT9AZjQF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the following important question related to \"pruning after training\" framework: why, in general, smaller network trained from the scratch does not achieve the same accuracy as the pruned network (obtained from pretrained net)? For that matter, authors introduce a new mechanism (\"weights' utility measure and utility imbalance\") to measure a discrepancy between subnetworks (subset of weights) within each network and it seems that having a large value of that measure accounts for the networks not utilizing all of their weights.\n\nIn general, I liked the approach chosen by authors to study the effects of pretraining and weight utilization in pruning. For example, it is interesting to observe that the accuracy difference can be monotonically increasing during neural net training and that the neural networks utilization is proportion to their size. Moreover, such difference (described by utility imbalance measure) is observed independent of network size. As far as I know, such study is novel. However, I don't think that the utility imbalance measure is a unique mechanism for such analysis and one can choose other possible options. In fact, a huge variation of the neural net accuracy depending on which subnetwork (subset of weights) you choose is so obvious that there is no need to show it explicitly (fig. 1). Moreover, the core idea behind pruning is to find such best subset of weights. Therefore, I don't think that such variability can explain why smaller nets cannot achieve the same accuracy as that of pruned net. \n\nAs for the second part of the paper (loss function surface), the accuracy surface is drawn based on only THREE points, specifically the loss of the pretrained net, pruned net and retrained net after pruning. I don't think that this information is sufficient to obtain an accurate visualization. Moreover, again, I believe that the pruned-then-retrained and trained net are lying on similar surface is trivial to show since: 1) pruned-then-retrained almost always perform better than just pruning; 2) if we don't prune too much then we can expect similar performance as the original net. Therefore, I don't see much contribution here.\n\nMinor concerns: \\\n . some parts of the paper is relatively hard to follow (e.g. in Section 2, jumping from experiment setting to discussing results and vice versa);\\\n . colors in some figures (e.g. fig. 1 and fig. 2) are really hard to distinguish since they are very similar.\\\n . section 2.2.2 describes utility imbalance in a pruned network (before or after retraining). I'm just curious what would be that measure for iterative pruning schemes. For example, as in (Han et al., 2015) and [1,2].\n\n[1] Zhang et al. Adam-ADMM: A unified, systematic framework of structured weight pruning for DNNs. 2018.\\\n[2] Carreira-Perpin˜an, M. and Y. Idelbayev. Learning-compression algorithms for neural net pruning. 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FDVmqRmfEZE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to answer the question why \"a network with the same number of weights as that of the pruned network cannot achieve similar performance when trained from scratch\". Then it proposes an hypothesis that the small model \"does not utilize all of its weights either\". To prove this hypothesis, it goes on to define and study the \"utility imbalance\" of the weights and its changing with the pretraining, pruning, etc. Some visualization analysis was provided too.\n\nI appreciate the detail empirical studies conducted in this paper, but I have some serious concerns:\n\n1. The paper cites [1] for the assumption that \"a network with the same number of weights as that of the pruned network cannot achieve similar performance when trained from scratch\". This is the basis for the whole study of the paper. This assumption seems to be in contradiction with conclusion in [2] for some cases, including the experimental setting this paper studies. In fact, it is well documented both in [1] and [2] that a network trained from scratch can reach a similar performance as a pruned-retrained network, under relatively large learning rates, in the case of unstructured pruning for CIFAR-10, or any structured pruning. The learning rate used is 0.1 for the CIFAR-10, which this paper also uses. This basically makes the assumption of this study (and thus the goal of proving the hypothesis) at least not valid in some cases. I am aware that this issue on learning rate was mentioned in section 2.1.1, but that still doesn't address the basic assumption and aimed goal of this study. \n\n2. The paper demonstrated there is imbalance in the network weights' \"utility\", but did not try to investigate which part of the weights are of higher utility. I think this is a more important problem than its relation with network size. For example, in [2], the authors showed the imbalance usage within 3x3 convolutional kernels.\n\n3. The utility imbalance is not really a surprising phenomenon. After or during training, only with these utility imbalance we can prune network according to various criteria, and beat random pruning. Thus this should not be surprising to the literature. For imbalance at initialization, the paper directly cites [1] without much further study.\n\n4. The paper only discusses the conclusion at [1] for the utility imbalance at initialization. I think some experiments could help better understand the cause of imbalance at initialization. For example, do shallower layer's weights on average has larger utility? or those weights with large magnitudes? I wouldn't be surprised if the imbalance can be explained by the differences with initialized weight magnitudes. So again the question is how much of the imbalance is explained by the lottery hypothesis, and how much is due to other reasons. This goes back to point 2, where my concern is the paper did not investigate which weights are of higher utility, either at initialization, during optimization or after.\n\n5. The paper title is a bit too vague. The conclusions are kind of vague too. The conclusion section mentioned a lot of content was studied/discussed/defined/investigated, but didn't give what the findings are. This is also a reflection that the paper/study itself didn't have a strong conclusion either.\n\nIn total, there are some interesting analysis on the changing process of the defined imbalance. However, the paper seems to lack a central conclusion that can help future practices, or help people gain deeper understanding. Network weight imbalance is expected, not surprising. The paper did not dig further with the imbalance by investigating which subsets of weights are more useful, and did not give much explanation with experiments on why the imbalance exists. Last but not least, the assumption and the attempted hypothesis does not always hold in the first place, in particular even in paper's experimental setting (large lr). \n\n[1] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks [2] Rethinking the Value of Network Pruning\n\n\nI appreciate the author response but unfortunately they are still a bit vague (1,3), or not supported with experiments (2,4). I still maintain my rating of 4.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Pruning is an important problem in practice. The angle of this study is also interesting. The key concept proposed by this submission is called the "utility imbalance" of the weights.  There are many concerns raised by the reviewers. Let us summarize some of them here: (1) hard to follow even for the domain experts; (2) the definition and motivation on "utility imbalance" are unclear ; (3) loss landscape visualizations are too much simplified to be informative.  There are also lots of concerns on writings. The rebuttal did help clarifying some details. However, most of the concerns still remain. We hope the detailed comments from the reviewers will be useful for the authors to polish this work. 
Decision: Reject