Title: RETHINKING THE VALUE OF NETWORK PRUNING

Abstract: Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited "important" weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our results suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin, 2019) , and find that with optimal learning rate, the "winning ticket" initialization as used in Frankle & Carbin (2019) does not bring improvement over random initialization.

1. INTRODUCTION:

Over-parameterization is a widely-recognized property of deep neural networks (Denton et al., 2014; Ba & Caruana, 2014) , which leads to high computational cost and high memory footprint for inference. As a remedy, network pruning (LeCun et al., 1990; Hassibi & Stork, 1993; Han et al., 2015; Molchanov et al., 2016; Li et al., 2017) has been identified as an effective technique to improve the efficiency of deep networks for applications with limited computational budget. A typical procedure of network pruning consists of three stages: 1) train a large, over-parameterized model (sometimes there are pretrained models available), 2) prune the trained large model according to a certain criterion, and 3) fine-tune the pruned model to regain the lost performance.

Training:

Pruning Fine-tuning Generally, there are two common beliefs behind this pruning procedure. First, it is believed that starting with training a large, over-parameterized network is important (Luo et al., 2017; Carreira-Perpinán & Idelbayev, 2018) , as it provides a highperformance model (due to stronger representation & optimization power) from which one can safely remove a set of redundant parameters without significantly hurting the accuracy. Therefore, this is usually believed, and reported to be superior to directly training a smaller network from scratch (Li et al., 2017; Luo et al., 2017; He et al., 2017b; Yu et al., 2018 ) -a commonly used baseline approach. Second, both the pruned architecture and its associated weights are believed to be essential for obtaining the final efficient model (Han et al., 2015) . Thus most existing pruning techniques choose to fine-tune a pruned model instead of training it from scratch. The preserved weights after pruning are usually considered to be critical, as how to accurately select the set of important weights is a very active research topic in the literature (Molchanov et al., 2016; Li et al., 2017; Luo et al., 2017; He et al., 2017b; Liu et al., 2017; Suau et al., 2018) .

In this work, we show that both of the beliefs mentioned above are not necessarily true for structured pruning methods, which prune at the levels of convolution channels or larger. Based on an extensive empirical evaluation of state-of-the-art pruning algorithms on multiple datasets with multiple network architectures, we make two surprising observations. First, for structured pruning methods with predefined target network architectures (Figure 2 ), directly training the small target model from random initialization can achieve the same, if not better, performance, as the model obtained from the three-stage pipeline. In this case, starting with a large model is not necessary and one could instead directly train the target model from scratch. Second, for structured pruning methods with autodiscovered target networks, training the pruned model from scratch can also achieve comparable or even better performance than fine-tuning. This observation shows that for these pruning methods, what matters more may be the obtained architecture, instead of the preserved weights, despite training the large model is needed to find that target architecture. Interestingly, for a unstructured pruning method (Han et al., 2015) that prunes individual parameters, we found that training from scratch can mostly achieve comparable accuracy with pruning and fine-tuning on smaller-scale datasets, but fails to do so on the large-scale ImageNet benchmark. Note that in some cases, if a pretrained large model is already available, pruning and fine-tuning from it can save the training time required to obtain the efficient model. The contradiction between some of our results and those reported in the literature might be explained by less carefully chosen hyper-parameters, data augmentation schemes and unfair computation budget for evaluating baseline approaches.

Predefined: prune x% channels in each layer Automatic: prune a%, b%, c%, d% channels in each layer

A 4-layer model

Figure 2 : Difference between predefined and automatically discovered target architectures, in channel pruning as an example. The pruning ratio x is userspecified, while a, b, c, d are determined by the pruning algorithm. Unstructured sparse pruning can also be viewed as automatic.

Our results advocate a rethinking of existing structured network pruning algorithms. It seems that the over-parameterization during the first-stage training is not as beneficial as previously thought. Also, inheriting weights from a large model is not necessarily optimal, and might trap the pruned model into a bad local minimum, even if the weights are considered "important" by the pruning criterion. Instead, our results suggest that the value of automatic structured pruning algorithms sometimes lie in identifying efficient structures and performing implicit architecture search, rather than selecting "important" weights. For most structured pruning methods which prune channels/filters, this corresponds to searching the number of channels in each layer. In section 5, we discuss this viewpoint through carefully designed experiments, and show the patterns in the pruned model could provide design guidelines for efficient architectures.

The rest of the paper is organized as follows: in Section 2, we introduce background and some related works on network pruning; in Section 3, we describe our methodology for training the pruned model from scratch; in Section 4 we experiment on various pruning methods and show our main results for both pruning methods with predefined or automatically discovered target architectures; in Section 5, we discuss the value of automatic pruning methods in searching efficient network architectures; in Section 6 we discuss some implications and conclude the paper.

2. BACKGROUND:

in some resource-constrained settings. Those large models can be infeasible to store, and run in real time on embedded systems. To address this issue, many methods have been proposed such as low-rank approximation of weights (Denton et al., 2014; Lebedev et al., 2014) , weight quantization (Courbariaux et al., 2016; Rastegari et al., 2016) , knowledge distillation (Hinton et al., 2014; Romero et al., 2015) and network pruning (Han et al., 2015; Li et al., 2017) , among which network pruning has gained notable attention due to their competitive performance and compatibility.

One major branch of network pruning methods is individual weight pruning, and it dates back to Optimal Brain Damage (LeCun et al., 1990) and Optimal Brain Surgeon (Hassibi & Stork, 1993) , which prune weights based on Hessian of the loss function. More recently, Han et al. (2015) proposes to prune network weights with small magnitude, and this technique is further incorporated into the "Deep Compression" pipeline (Han et al., 2016b) to obtain highly compressed models. Srinivas & Babu (2015) proposes a data-free algorithm to remove redundant neurons iteratively. Molchanov et al. (2017) uses Variatonal Dropout (P. Kingma et al., 2015) to prune redundant weights. Louizos et al. (2018) learns sparse networks through L 0 -norm regularization based on stochastic gate. However, one drawback of these unstructured pruning methods is that the resulting weight matrices are sparse, which cannot lead to compression and speedup without dedicated hardware/libraries (Han et al., 2016a) .

In contrast, structured pruning methods prune at the level of channels or even layers. Since the original convolution structure is still preserved, no dedicated hardware/libraries are required to realize the benefits. Among structured pruning methods, channel pruning is the most popular, since it operates at the most fine-grained level while still fitting in conventional deep learning frameworks. Some heuristic methods include pruning channels based on their corresponding filter weight norm (Li et al., 2017) and average percentage of zeros in the output (Hu et al., 2016) . Group sparsity is also widely used to smooth the pruning process after training (Wen et al., 2016; Alvarez & Salzmann, 2016; Lebedev & Lempitsky, 2016; Zhou et al., 2016) . Liu et al. (2017) and Ye et al. (2018) impose sparsity constraints on channel-wise scaling factors during training, whose magnitudes are then used for channel pruning. Huang & Wang (2018) uses a similar technique to prune coarser structures such as residual blocks. He et al. (2017b) and Luo et al. (2017) minimizes next layer's feature reconstruction error to determine which channels to keep. Similarly, Yu et al. (2018) optimizes the reconstruction error of the final response layer and propagates a "importance score" for each channel. Molchanov et al. (2016) uses Taylor expansion to approximate each channel's influence over the final loss and prune accordingly. Suau et al. (2018) analyzes the intrinsic correlation within each layer and prune redundant channels. Chin et al. (2018) proposes a layer-wise compensate filter pruning algorithm to improve commonly-adopted heuristic pruning metrics. He et al. (2018a) proposes to allow pruned filters to recover during the training process. Lin et al. (2017) ; Wang et al. (2017) prune certain structures in the network based on the current input.

Our work is also related to some recent studies on the characteristics of pruning algorithms. Mittal et al. (2018) shows that random channel pruning (Anwar & Sung, 2016) can perform on par with a variety of more sophisticated pruning criteria, demonstrating the plasticity of network models.

In the context of unstructured pruning, The Lottery Ticket Hypothesis (Frankle & Carbin, 2019) conjectures that certain connections together with their randomly initialized weights, can enable a comparable accuracy with the original network when trained in isolation. We provide comparisons between Frankle & Carbin (2019) and this work in Appendix A. Zhu & Gupta (2018) shows that training a small-dense model cannot achieve the same accuracy as a pruned large-sparse model with identical memory footprint. In this work, we reveal a different and rather surprising characteristic of structured network pruning methods: fine-tuning the pruned model with inherited weights is not better than training it from scratch; the resulting pruned architectures are more likely to be what brings the benefit.

3. METHODOLOGY:

In this section, we describe in detail our methodology for training a small target model from scratch.

Target Pruned Architectures. We first divide network pruning methods into two categories. In a pruning pipeline, the target pruned model's architecture can be determined by either a human (i.e., predefined) or the pruning algorithm (i.e., automatic) (see Figure 2 ).

When a human predefines the target architecture, a common criterion is the ratio of channels to prune in each layer. For example, we may want to prune 50% channels in each layer of VGG. In this case, no matter which specific channels are pruned, the pruned target architecture remains the same, because the pruning algorithm only locally prunes the least important 50% channels in each layer. In practice, the ratio in each layer is usually selected through empirical studies or heuristics. Examples of predefined structured pruning include Li et al. (2017) , Luo et al. (2017) , He et al. (2017b) and He et al. (2018a) When the target architecture is automatically determined by a pruning algorithm, it is usually based on a pruning criterion that globally compares the importance of structures (e.g., channels) across layers. Examples of automatic structured pruning include Liu et al. (2017) , Huang & Wang (2018) , Molchanov et al. (2016) and Suau et al. (2018) .

Unstructured pruning (Han et al., 2015; Molchanov et al., 2017; Louizos et al., 2018) also falls in the category of automatic methods, where the positions of pruned weights are determined by the training process and the pruning algorithm, and it is usually not possible to predefine the positions of zeros before training starts.

Datasets, Network Architectures and Pruning Methods. In the network pruning literature, CIFAR-10, CIFAR-100 (Krizhevsky, 2009) , and ImageNet (Deng et al., 2009) datasets are the de-facto benchmarks, while VGG (Simonyan & Zisserman, 2015) , ResNet (He et al., 2016) and DenseNet (Huang et al., 2017) are the common network architectures. We evaluate four predefined pruning methods, Li et al. (2017) , Luo et al. (2017) , He et al. (2017b) , He et al. (2018a) , two automatic structured pruning methods, Liu et al. (2017) , Huang & Wang (2018) , and one unstructured pruning method (Han et al., 2015) . For the first six methods, we evaluate using the same (target model, dataset) pairs as presented in the original paper to keep our results comparable. For the last one (Han et al., 2015) , we use the aforementioned architectures instead, since the ones in the original paper are no longer state-of-the-art. On CIFAR datasets, we run each experiment with 5 random seeds, and report the mean and standard deviation of the accuracy.

Training Budget. One crucial question is how long we should train the small pruned model from scratch. Naively training for the same number of epochs as we train the large model might be unfair, since the small pruned model requires significantly less computation for one epoch. Alternatively, we could compute the floating point operations (FLOPs) for both the pruned and large models, and choose the number of training epoch for the pruned model that would lead to the same amount of computation as training the large model. Note that it is not clear how to train the models to "full convergence" given the stepwise decaying learning rate schedule commonly used in the CIFAR/ImageNet classification tasks.

In our experiments, we use Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B to denote training for the same amount of computation budget (on Ima-geNet, if the pruned model saves more than 2× FLOPs, we just double the number of epochs for training Scratch-B, which amounts to less computation budget than large model training). When extending the number of epochs in Scratch-B, we also extend the learning rate decay schedules proportionally. One may argue that we should instead train the small target model for fewer epochs since it may converge faster. However, in practice we found that increasing the training epochs within a reasonable range is rarely harmful. In our experiments we found in most times Scratch-E is enough while in other cases Scratch-B is needed for a comparable accuracy as fine-tuning. Note that our evaluations use the same computation as large model training without considering the computation in fine-tuning, since in our evaluated methods fine-tuning does not take too long; if anything this still favors the pruning and fine-tuning pipeline.

Implementation. In order to keep our setup as close to the original papers as possible, we use the following protocols: 1) ff a previous pruning method's training setup is publicly available, e.g. Liu et al. (2017) , Huang & Wang (2018) and He et al. (2018a) , we adopt the original implementation; 2) otherwise, for simpler pruning methods, e.g., Li et al. (2017) and Han et al. (2015) , we re-implement the three-stage pruning procedure and generally achieve similar results as in the original papers; 3) for the remaining two methods (Luo et al., 2017; He et al., 2017b) , the pruned models are publicly available but without the training setup, thus we choose to re-train both large and small target models from scratch. Interestingly, the accuracy of our re-trained large model is higher than what is reported in the original papers. This could be due to the difference in the deep learning frameworks: we used Pytorch (Paszke et al., 2017) while the original papers used Caffe (Jia et al., 2014) . In this case, to accommodate the effects of different frameworks and training setups, we report the relative accuracy drop from the unpruned large model.

We use standard training hyper-parameters and data-augmentation schemes, which are used both in standard image classification models (He et al., 2016; Huang et al., 2017) and network pruning methods (Li et al., 2017; Liu et al., 2017; Huang & Wang, 2018; He et al., 2018a) . The optimization method is SGD with Nesterov momentum, using an stepwise decay learning rate schedule. For random weight initialization, we adopt the scheme proposed in (He et al., 2015) . For results of models fine-tuned from inherited weights, we either use the released models from original papers (case 3 above) or follow the common practice of fine-tuning the model using the lowest learning rate when training the large model (Li et al., 2017; He et al., 2017b) . For CIFAR, training/fine-tuning takes 160/40 epochs. For ImageNet, training/fine-tuning takes 90/20 epochs. For reproducing the results and a more detailed knowledge about the settings, see our code at: https://github. com/Eric-mingjie/rethinking-network-pruning.

4. EXPERIMENTS:

In this section we present our experimental results comparing training pruned models from scratch and fine-tuning from inherited weights, for both predefined and automatic (Figure 2 ) structured pruning, as well as a magnitude-based unstructured pruning method (Han et al., 2015) . We also include a comparison with the Lottery Ticket Hypothesis (Frankle & Carbin, 2019) , and an experiment on transfer learning from image classification to object detection in Appendix, due to space limit. We also put the results and discussions on a pruning method (Soft Filter pruning (He et al., 2018a )) in Appendix.

4.1. PREDEFINED STRUCTURED PRUNING:

L 1 -norm based Filter Pruning (Li et al., 2017) is one of the earliest works on filter/channel pruning for convolutional networks. In each layer, a certain percentage of filters with smaller L 1 -norm will be pruned. Table 1 shows our results. The Pruned Model column shows the list of predefined target models (see (Li et al., 2017) for configuration details on each model). We observe that in each row, scratch-trained models achieve at least the same level of accuracy as fine-tuned models, with Scratch-B slightly higher than Scratch-E in most cases. On ImageNet, both Scratch-B models are better than the fine-tuned ones by a noticeable margin. 

4.2. AUTOMATIC STRUCTURED PRUNING:

Network Slimming (Liu et al., 2017) imposes L 1 -sparsity on channel-wise scaling factors from Batch Normalization layers (Ioffe & Szegedy, 2015) during training, and prunes channels with lower scaling factors afterward. Since the channel scaling factors are compared across layers, this method produces automatically discovered target architectures. As shown in Table 4 , for all networks, the small models trained from scratch can reach the same accuracy as the fine-tuned models. More specifically, we found that Scratch-B consistently outperforms (8 out of 10 experiments) the finetuned models, while Scratch-E is slightly worse but still mostly within the standard deviation. (Liu et al., 2017) . "Prune ratio" stands for total percentage of channels that are pruned in the whole network. The same ratios for each model are used as the original paper.

Sparse Structure Selection (Huang & Wang, 2018 ) also uses sparsified scaling factors to prune structures, and can be seen as a generalization of Network Slimming. Other than channels, pruning can be on residual blocks in ResNet or groups in ResNeXt (Xie et al., 2017) . We examine residual blocks pruning, where ResNet-50 are pruned to be ResNet-41, ResNet-32 and ResNet-26. (Huang & Wang, 2018 ).

In the original paper no fine-tuning is required so there is a "Pruned" column instead of "Fine-tuned" as before.

4.3. UNSTRUCTURED MAGNITUDE-BASED PRUNING (HAN ET AL., 2015):

Unstructured magnitude-based weight pruning (Han et al., 2015) can also be treated as automatically discovering architectures, since the positions of exact zeros cannot be determined before training, but we highlight its differences with structured pruning using another subsection. Because all the network architectures we evaluated are fully-convolutional (except for the last fully-connected layer), for simplicity, we only prune weights in convolution layers here. Before training the pruned sparse model from scratch, we re-scale the standard deviation of the Gaussian distribution for weight initialization, based on how many non-zero weights remain in this layer. This is to keep a constant scale of backward gradient signal as in (He et al., 2015) , which however in our observations does not bring gains compared with unscaled counterparts. (Han et al., 2015) . "Prune Ratio" denotes the percentage of parameters pruned in the set of all convolutional weights.

As shown in Table 6 , on the smaller-scale CIFAR datasets, when the pruned ratio is small (≤ 80%), Scratch-E sometimes falls short of the fine-tuned results, but Scratch-B is able to perform at least on par with the latter. However, we observe that in some cases, when the prune ratio is large (95%), fine-tuning can outperform training from scratch. On the large-scale ImageNet dataset, we note that the Scratch-B result is mostly worse than fine-tuned result by a noticable margin, despite at a decent accuracy level. This could be due to the increased difficulty of directly training on the highly sparse networks (CIFAR), or the scale/complexity of the dataset itself (ImageNet). Another possible reason is that compared with structured pruning, unstructured pruning significantly changes the weight distribution (more details in Appendix G). The difference in scratch-training behaviors also suggests an important difference between structured and unstructured pruning.

5. NETWORK PRUNING AS ARCHITECTURE SEARCH:

While we have shown that, for structured pruning, the inherited weights in the pruned architecture are not better than random, the pruned architecture itself turns out to be what brings the efficiency benefits. In this section, we assess the value of architecture search for automatic network pruning algorithms (Figure 2 ) by comparing pruning-obtained models and uniformly pruned models. Note that the connection between network pruning and architecture learning has also been made in prior works (Han et al., 2015; Liu et al., 2017; Gordon et al., 2018; Huang et al., 2018) , but to our knowledge we are the first to isolate the effect of inheriting weights and solely compare pruningobtained architectures with uniformly pruned ones, by training both of them from scratch.

Parameter Efficiency of Pruned Architectures. In Figure 3 (left), we compare the parameter efficiency of architectures obtained by an automatic channel pruning method (Network Slimming (Liu et al., 2017) ), with a naive predefined pruning strategy that uniformly prunes the same percentage of channels in each layer. All architectures are trained from random initialization for the same number of epochs. We see that the architectures obtained by Network Slimming are more parameter efficient, as they could achieve the same level of accuracy using 5× fewer parameters than uniformly pruning architectures. For unstructured magnitude-based pruning (Han et al., 2015) , we conducted a similar experiment shown in Figure 3 (right). Here we uniformly sparsify all individual weights at a fixed probability, and the architectures obtained this way are much less efficient than the pruned architectures. We also found the channel/weight pruned architectures exhibit very consistent patterns (see Table 7 and Figure 4 ). This suggests the original large models may be redundantly designed for the task and the pruning algorithm can help us improve the efficiency. This also confirms the value of automatic pruning methods for searching efficient models on the architectures evaluated. (Liu et al., 2017) . Middle and Right: Results for PreResNet-110 and DenseNet-40 pruned on CIFAR-100 by unstructured pruning (Han et al., 2015) .

More Analysis. However, there also exist cases where the architectures obtained by pruning are not better than uniformly pruned ones. We present such results in Figure 5 , where the architectures obtained by pruning (blue) are not significantly more efficient than uniform pruned architectures (red). This phenomenon happens more likely on modern architectures like ResNets and DenseNets.

When we investigate the sparsity patterns of those pruned architectures (shown in Table 18 , 19 and 20 in Appendix H), we find that they exhibit near-uniform sparsity patterns across stages, which might be the reason why it can only perform on par with uniform pruning. In contrast, for VGG, the pruned sparsity patterns can always beat the uniform ones as shown in Figure 3 and Figure 6 . We also show the sparsity patterns of VGG pruned by Network Slimming (Liu et al., 2017) in Table 21 of Appendix H, and they are rather far from uniform. Compared to ResNet and DenseNet, we can see that VGG's redundancy is rather imbalanced across layer stages. Network pruning techniques may help us identify the redundancy better in the such cases.

Generalizable Design Principles from Pruned Architectures. Given that the automatically discovered architectures tend to be parameter efficient on the VGG networks, one may wonder: can we derive generalizable principles from them on how to design a better architecture? We conduct several experiments to answer this question.

For Network Slimming, we use the average number of channels in each layer stage (layers with the same feature map size) from pruned architectures to construct a new set of architectures, and we call this approach "Guided Pruning"; for magnitude-based pruning, we analyze the sparsity patterns (Figure 4 ) in the pruned architectures, and apply them to construct a new set of sparse models, which we call "Guided Sparsification". The results are shown in Figure 6 . It can be seen that for both Network Slimming (Figure 6 left) and unstructured pruning (Figure 6 right), guided design of architectures (green) can perform on par with pruned architectures (blue).

Interestingly, these guided design patterns can sometimes be transferred to a different VGG-variant and/or dataset. In Figure 6 , we distill the patterns of pruned architectures from VGG-16 on CIFAR-10 and apply them to design efficient VGG-19 on CIFAR-100. These sets of architectures are denoted as "Transferred Guided Pruning/Sparsification". We can observe that they (brown) may sometimes be slightly worse than architectures directly pruned (blue), but are significantly better than uniform pruning/sparsification (red). In these cases, one does not need to train a large model to obtain an efficient model as well, as transferred design patterns can help us achieve the efficiency directly.

Discussions with Conventional Architecture Search Methods. Popular techniques for network architecture search include reinforcement learning (Zoph & Le, 2017; Baker et al., 2017) and evolutionary algorithms (Xie & Yuille, 2017; Liu et al., 2018a) . In each iteration, a randomly initialized network is trained and evaluated to guide the search, and the search process usually requires thousands of iterations to find the goal architecture. In contrast, using network pruning as architecture search only requires a one-pass training, however the search space is restricted to the set of all "subnetworks" inside a large network, whereas traditional methods can search for more variations, e.g., activation functions or different layer orders. Figure 6 : Pruned architectures obtained by different approaches, all trained from scratch, averaged over 5 runs. "Guided Pruning/Sparsification" means using the average sparsity patterns in each layer stage to design the network; "Transferred Guided Pruning/Sparsification" means using the sparsity patterns obtained by a pruned VGG-16 on CIFAR-10, to design the network for VGG-19 on CIFAR-100. Following the design guidelines provided by the pruned architectures, we achieve better parameter efficiency, even when the guidelines are transferred from another dataset and model.

Recently, Gordon et al. (2018) uses a similar pruning technique to Network Slimming (Liu et al., 2017) to automate the design of network architectures; He et al. (2018c) prune channels using reinforcement learning and automatically compresses the architecture. On the other hand, in the network architecture search literature, sharing/inheriting trained parameters (Pham et al., 2018; Liu et al., 2018b) during searching has become a popular approach for reducing the training budgets, but once the target architecture is found, it is still trained from scratch to maximize the accuracy.

6. DISCUSSION AND CONCLUSION:

Our results encourage more careful and fair baseline evaluations of structured pruning methods. In addition to high accuracy, training predefined target models from scratch has the following benefits over conventional network pruning procedures: a) since the model is smaller, we can train the model using less GPU memory and possibly faster than training the original large model; b) there is no need to implement the pruning criterion and procedure, which sometimes requires fine-tuning layer by layer (Luo et al., 2017) and/or needs to be customized for different network architectures (Li et al., 2017; Liu et al., 2017) ; c) we avoid tuning additional hyper-parameters involved in the pruning procedure.

Our results do support the viewpoint that automatic structured pruning finds efficient architectures in some cases. However, if the accuracy of pruning and fine-tuning is achievable by training the pruned model from scratch, it is also important to evaluate the pruned architectures against uniformly pruned baselines (both training from scratch), to demonstrate the method's value in identifying efficient architectures. If the uniformly pruned models are not worse, one could also skip the pipeline and train them from scratch.

Even if pruning and fine-tuning fails to outperform the mentioned baselines in terms of accuracy, there are still some cases where using this conventional wisdom can be much faster than training from scratch: a) when a pre-trained large model is already given and little or no training budget is available; we also note that pre-trained models can only be used when the method does not require modifications to the large model training process; b) there is a need to obtain multiple models of different sizes, or one does not know what the desirable size is, in which situations one can train a large model and then prune it by different ratios.

APPENDIX:

A EXPERIMENTS ON THE LOTTERY TICKET HYPOTHESIS (FRANKLE & CARBIN, 2019) The Lottery Ticket Hypothesis (Frankle & Carbin, 2019) conjectures that inside the large network, a sub-network together with their initialization makes the training particularly effective, and together they are termed the "winning ticket". In this hypothesis, the original initialization of the sub-network (before large model training) is needed for it to achieve competitive performance when trained in isolation. Their experiments show that training the sub-network with randomly re-initialized weights performs worse than training it with the original initialization inside the large network. In contrast, our work does not require reuse of the original initialization of the pruned model, and shows that random initialization is enough for the pruned model to achieve competitive performance.

The conclusions seem to be contradictory, but there are several important differences in the evaluation settings: a) Our main conclusion is drawn on structured pruning methods, despite for smallscale problems (CIFAR) it also holds on unstructured pruning; Frankle & Carbin (2019) only evaluates on unstructured pruning. b) Our evaluated network architectures are all relatively large modern models used in the original pruning methods, while most of the experiments in Frankle & Carbin (2019) use small shallow networks (< 6 layers). c) We use momentum SGD with a large initial learning rate (0.1), which is widely used in prior image classification (He et al., 2016; Huang et al., 2017) and pruning works (Li et al., 2017; Liu et al., 2017; He et al., 2017b; Luo et al., 2017; He et al., 2018a; Huang & Wang, 2018) to achieve high accuracy, and is the de facto default optimization setting on CIFAR and ImageNet; while Frankle & Carbin (2019) (Frankle & Carbin, 2019) for iterative/one-shot unstructured pruning (Han et al., 2015) with two initial learning rates 0.1 and 0.01, on CIFAR-10 dataset. Each point is averaged over 5 runs. Using the winning ticket as initialization only brings improvement when the learning rate is small (0.01), however such small learning rate leads to a lower accuracy than the widely used large learning rate (0.1).

In this section, we show that the difference in learning rate is what causes the seemingly contradicting behaviors between our work and Frankle & Carbin (2019) , in the case of unstructured pruning on CIFAR. For structured pruning, when using both large and small learning rates, the winning ticket does not outperform random initialization.

We test the Lottery Ticket Hypothesis by comparing the models trained with original initialization ("winning ticket") and that trained from randomly re-initialized weights. We experiment with two choices of initial learning rate (0.1 and 0.01) with a stepwise decay schedule, using momentum SGD. 0.1 is used in our previous experiments and most prior works on CIFAR and ImageNet. Following Frankle & Carbin (2019) , we investigate both iterative pruning (prune 20% in each iteration) and one-shot pruning for unstructured pruning. We show our results for unstructured pruning (Han et al., 2015) in Figure 7 and Table 8 , and L 1 -norm based filter pruning (Li et al., 2017) (Frankle & Carbin, 2019) for one-shot unstructured pruning (Han et al., 2015) with two initial learning rates: 0.1 and 0.01. The same results are visualized in Figure 7b . Using the winning ticket as initialization only brings improvement when the learning rate is small (0.01), however such small learning rate leads to a lower accuracy than the widely used large learning rate (0.1).

From Figure 7 and Table 8 , we see that for unstructured pruning, using the original initialization as in (Frankle & Carbin, 2019) only provides advantage over random initialization with small initial learning rate 0.01. For structured pruning as Li et al. (2017) , it can be seen from Table 9 that using the original initialization is only on par with random initialization for both large and small initial learning rates. In both cases, we can see that the small learning rate gives lower accuracy than the widely-used large learning rate. To summarize, in our evaluated settings, the winning ticket only brings improvement in the case of unstructured pruning, with small initial learning rate, but this small learning rate yields inferior accuracy compared with the widely-used large learning rate. Note that Frankle & Carbin (2019) also report in their Section 5, that the winning ticket cannot be found on ResNet-18/VGG using the large learning rate. The reason why the original initialization is helpful when the learning rate is small, might be the weights of the final trained model are not too far from the original initialization due to the small parameter updating stepsize. Figure 8 : Weight distribution of convolutional layers for different pruning methods. We use VGG-16 and CIFAR-10 for this visualization. We compare the weight distribution of unpruned models, fine-tuned models and scratch-trained models. Top: Results for Network Slimming (Liu et al., 2017) . Bottom: Results for unstructured pruning (Han et al., 2015) .

G WEIGHT DISTRIBUTIONS:

Accompanying the discussion in subsection 4.3, we show the weight distribution of unpruned large models, fine-tuned pruned models and scratch-trained pruned models, for two pruning methods: (structured) Network Slimming (Liu et al., 2017) and unstructured pruning (Han et al., 2015) . We choose VGG-16 and CIFAR-10 for visualization and compare the weight distribution of unpruned models, fine-tuned models and scratch-trained models. For Network Slimming, the prune ratio is 50%. For unstructured pruning, the prune ratio is 80%. Figure 8 shows our result. We can see that the weight distribution of fine-tuned models and scratch-trained pruned models are different from the unpruned large models -the weights that are close to zero are much fewer. This seems to imply that there are less redundant structures in the found pruned architecture, and support the view of architecture learning for automatic pruning methods.

For unstructured pruning, the fine-tuned model also has significantly different weight distribution from the scratch-trained model -it has nearly no close-to-zero values. This might be a potential reason why sometimes models trained from scratch cannot achieve the accuracy of the fine-tuned models, as shown in Table 6 .

H MORE SPARSITY PATTERNS FOR PRUNED ARCHITECTURES:

In this section we provide the additional results on sparsity patterns for the pruned models, accompanying the discussions of "More Analysis" in Section 5. 3 (left) under different prune ratio. The top row denotes the total prune ratio. The values denote the ratio of channels to be kept. For each prune ratio, the latter stages tend to have more redundancy than earlier stages.

:



Figures and Tables:

Figure 1: A typical three-stage network pruning pipeline.

Results (accuracy) for L1-norm based filter pruning(Li et al., 2017). "Pruned Model" is the model pruned from the large model. Configurations of Model and Pruned Model are both from the original paper.

Table content: <table><tr><td>Dataset</td><td>Model</td><td>Unpruned</td><td>Pruned Model</td><td>Fine-tuned</td><td>Scratch-E</td><td>Scratch-B</td></tr><tr><td/><td>VGG-16</td><td>93.63 (±0.16)</td><td>VGG-16-A</td><td colspan="3">93.41 (±0.12) 93.62 (±0.11) 93.78 (±0.15)</td></tr><tr><td>CIFAR-10</td><td colspan="2">ResNet-56 93.14 (±0.12)</td><td colspan="4">ResNet-56-A 92.97 (±0.17) 92.96 (±0.26) 93.09 (±0.14) ResNet-56-B 92.67 (±0.14) 92.54 (±0.19) 93.05 (±0.18)</td></tr><tr><td/><td colspan="2">ResNet-110 93.14 (±0.24)</td><td colspan="4">ResNet-110-A 93.14 (±0.16) 93.25 (±0.29) 93.22 (±0.22) ResNet-110-B 92.69 (±0.09) 92.89 (±0.43) 93.60 (±0.25)</td></tr><tr><td>ImageNet</td><td>ResNet-34</td><td>73.31</td><td>ResNet-34-A ResNet-34-B</td><td>72.56 72.29</td><td>72.77 72.55</td><td>73.03 72.91</td></tr><tr><td colspan="7">ThiNet (Luo et al., 2017) greedily prunes the channel that has the smallest effect on the next layer's</td></tr><tr><td colspan="7">activation values. As shown in Table 2, for VGG-16 and ResNet-50, both Scratch-E and Scratch-</td></tr><tr><td colspan="7">B can almost always achieve better performance than the fine-tuned model, often by a significant</td></tr><tr><td colspan="7">margin. The only exception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively</td></tr><tr><td colspan="7">from VGG-16 (FLOPs reduced by 15×), and as a result, drastically reducing the training budget for</td></tr><tr><td colspan="7">Scratch-E. The training budget of Scratch-B for this model is also 7 times smaller than the original</td></tr></table>

Results (accuracy)  for ThiNet(Luo et al., 2017). Names such as "VGG-GAP" and "ResNet50-30%" are pruned models whose configurations are defined inLuo et al. (2017). To accommodate the effects of different frameworks between our implementation and the original paper's, we compare relative accuracy drop from the unpruned large model. For example, for the pruned model VGG-Conv, -1.23 is relative to 71.03 on the left, which is the reported accuracy of the unpruned large model VGG-16 in the original paper; -2.75 is relative to 71.51 on the left, which is VGG-16's accuracy in our implementation.

Table content: <table><tr><td>Dataset</td><td>Unpruned</td><td>Strategy</td><td>Pruned Model</td></tr><tr><td/><td>VGG-16</td><td/><td>VGG-16-5x</td></tr><tr><td/><td>71.03</td><td>Fine-tuned</td><td>-2.67</td></tr><tr><td>ImageNet</td><td>71.51 ResNet-50</td><td>Scratch-E Scratch-B</td><td>-3.46 -0.51 ResNet-50-2x</td></tr><tr><td/><td>75.51</td><td>Fine-tuned</td><td>-3.25</td></tr><tr><td/><td>76.13</td><td>Scratch-E Scratch-B</td><td>-1.55 -1.07</td></tr></table>

Results

Table content: <table/>

Results (accuracy) for Network Slimming

Table content: <table><tr><td>Dataset</td><td>Model</td><td>Unpruned</td><td>Prune Ratio</td><td>Fine-tuned</td><td>Scratch-E</td><td>Scratch-B</td></tr><tr><td/><td>VGG-19</td><td>93.53 (±0.16)</td><td>70%</td><td colspan="3">93.60 (±0.16) 93.30 (±0.11) 93.81 (±0.14)</td></tr><tr><td>CIFAR-10</td><td colspan="2">PreResNet-164 95.04 (±0.16)</td><td>40% 60%</td><td colspan="3">94.77 (±0.12) 94.70 (±0.11) 94.90 (±0.04) 94.23 (±0.21) 94.58 (±0.18) 94.71 (±0.21)</td></tr><tr><td/><td>DenseNet-40</td><td>94.10 (±0.12)</td><td>40% 60%</td><td colspan="3">94.00 (±0.20) 93.68 (±0.18) 94.06 (±0.12) 93.87 (±0.13) 93.58 (±0.21) 93.85 (±0.25)</td></tr><tr><td/><td>VGG-19</td><td>72.63 (±0.21)</td><td>50%</td><td colspan="3">72.32 (±0.28) 71.94 (±0.17) 73.08 (±0.22)</td></tr><tr><td>CIFAR-100</td><td colspan="2">PreResNet-164 76.80 (±0.19)</td><td>40% 60%</td><td colspan="3">76.22 (±0.20) 76.36 (±0.32) 76.68 (±0.35) 74.17 (±0.33) 75.05 (± 0.08) 75.73 (±0.29)</td></tr><tr><td/><td>DenseNet-40</td><td>73.82 (±0.34)</td><td>40% 60%</td><td colspan="3">73.35 (±0.17) 73.24 (±0.29) 73.19 (±0.26) 72.46 (±0.22) 72.62 (±0.36) 72.91 (±0.34)</td></tr><tr><td>ImageNet</td><td>VGG-11</td><td>70.84</td><td>50%</td><td>68.62</td><td>70.00</td><td>71.18</td></tr></table>

Table5shows our results. On average Scratch-E outperforms pruned models, and for all models Scratch-B is better than both. Results (accuracy) for residual block pruning using Sparse Structure Selection

Table content: <table><tr><td>Dataset</td><td>Model</td><td colspan="5">Unpruned Pruned Model Pruned Scratch-E Scratch-B</td></tr><tr><td/><td/><td/><td>ResNet-41</td><td>75.44</td><td>75.61</td><td>76.17</td></tr><tr><td colspan="2">ImageNet ResNet-50</td><td>76.12</td><td>ResNet-32</td><td>74.18</td><td>73.77</td><td>74.67</td></tr><tr><td/><td/><td/><td>ResNet-26</td><td>71.82</td><td>72.55</td><td>73.41</td></tr></table>

Results (accuracy)  for unstructured pruning

Table content: <table><tr><td>Dataset</td><td>Model</td><td>Unpruned</td><td>Prune Ratio</td><td>Fine-tuned</td><td>Scratch-E</td><td>Scratch-B</td></tr><tr><td/><td/><td/><td>30%</td><td colspan="3">93.51 (±0.05) 93.71 (±0.09) 93.31 (±0.26)</td></tr><tr><td/><td>VGG-19</td><td>93.50 (±0.11)</td><td>80%</td><td colspan="3">93.52 (±0.10) 93.71 (±0.08) 93.64 (±0.09)</td></tr><tr><td/><td/><td/><td>95%</td><td colspan="3">93.34 (±0.13) 93.21 (±0.17) 93.63 (±0.18)</td></tr><tr><td/><td/><td/><td>30%</td><td colspan="3">95.06 (±0.05) 94.84 (±0.07) 95.11 (±0.09)</td></tr><tr><td>CIFAR-10</td><td>PreResNet-110</td><td>95.04 (±0.15)</td><td>80%</td><td colspan="3">94.55 (±0.11) 93.76 (±0.10) 94.52 (±0.13)</td></tr><tr><td/><td/><td/><td>95%</td><td colspan="3">92.35 (±0.20) 91.23 (±0.11) 91.55 (±0.34)</td></tr><tr><td/><td/><td/><td>30%</td><td colspan="3">95.21 (±0.17) 95.22 (±0.18) 95.23 (±0.14)</td></tr><tr><td/><td colspan="2">DenseNet-BC-100 95.24 (±0.17)</td><td>80%</td><td colspan="3">95.04 (±0.15) 94.42 (±0.12) 95.12 (±0.04)</td></tr><tr><td/><td/><td/><td>95%</td><td colspan="3">94.19 (±0.15) 92.91 (±0.22) 93.44 (±0.19)</td></tr><tr><td/><td/><td/><td>30%</td><td colspan="3">71.96 (±0.36) 72.81 (±0.31) 73.30 (±0.25)</td></tr><tr><td/><td>VGG-19</td><td>71.70 (±0.31)</td><td>50%</td><td colspan="3">71.85 (±0.30) 73.12 (±0.36) 73.77 (±0.23)</td></tr><tr><td/><td/><td/><td>95%</td><td colspan="3">70.22 (±0.38) 70.88 (±0.35) 72.08 (±0.15)</td></tr><tr><td/><td/><td/><td>30%</td><td colspan="3">76.88 (±0.31) 76.36 (±0.26) 76.96 (±0.31)</td></tr><tr><td>CIFAR-100</td><td>PreResNet-110</td><td>76.96 (±0.34)</td><td>50%</td><td colspan="3">76.60 (±0.36) 75.45 (±0.23) 76.42 (±0.39)</td></tr><tr><td/><td/><td/><td>95%</td><td colspan="3">68.55 (±0.51) 68.13 (±0.64) 68.99 (±0.32)</td></tr><tr><td/><td/><td/><td>30%</td><td colspan="3">77.23 (±0.05) 77.58 (±0.25) 77.97 (±0.31)</td></tr><tr><td/><td colspan="2">DenseNet-BC-100 77.59 (±0.19)</td><td>50%</td><td colspan="3">77.41 (±0.14) 77.65 (±0.09) 77.80 (±0.23)</td></tr><tr><td/><td/><td/><td>95%</td><td colspan="3">73.67 (±0.03) 71.47 (±0.46) 72.57 (±0.37)</td></tr><tr><td>ImageNet</td><td>VGG-16 ResNet-50</td><td>73.37 76.15</td><td>30% 60% 30% 60%</td><td>73.68 73.63 76.06 76.09</td><td>72.75 71.50 74.77 73.69</td><td>74.02 73.42 75.70 74.91</td></tr></table>

Network

Table content: <table><tr><td/><td>94.0</td><td colspan="6">Channel Pruned VGG-16 on CIFAR-10</td><td/><td>94.0</td><td colspan="2">Weight Sparsified VGG-16 on CIFAR-10</td></tr><tr><td>Test Accuracy (%)</td><td>92.5 93.0 93.5</td><td/><td/><td/><td colspan="3">Network Slimming</td><td>Test Accuracy (%)</td><td>91.5 92.0 92.5 93.0 93.5</td><td/><td>Unstructured Pruning</td></tr><tr><td/><td>92.0</td><td/><td/><td/><td colspan="3">Uniform Pruning</td><td/><td>91.0</td><td/><td>Uniform Sparsification</td></tr><tr><td/><td/><td>0.2</td><td>0.4</td><td colspan="2">0.6 #Parameters 0.8</td><td>1.0</td><td>1.2 ×10 7</td><td/><td/><td>0.2</td><td>0.4</td><td>0.6 #Parameters 0.8</td><td>1.0</td><td>1.2</td><td>×10 7</td></tr><tr><td colspan="3">Layer Width</td><td/><td>Width*</td><td colspan="2">Layer Width</td><td colspan="3">Width*</td><td/></tr><tr><td>1</td><td/><td>64</td><td colspan="2">39.0±3.7</td><td>8</td><td>512</td><td colspan="3">217.3±6.6</td><td/></tr><tr><td>2</td><td/><td>64</td><td colspan="2">64.0±0.0</td><td>9</td><td>512</td><td colspan="3">120.0±4.4</td><td/></tr><tr><td>3</td><td/><td>128</td><td colspan="2">127.8±0.4</td><td>10</td><td>512</td><td colspan="3">63.0±1.9</td><td/></tr><tr><td>4</td><td/><td>128</td><td colspan="2">128.0±0.0</td><td>11</td><td>512</td><td colspan="3">47.8±2.9</td><td/></tr><tr><td>5</td><td/><td>256</td><td colspan="2">255.0±1.0</td><td>12</td><td>512</td><td colspan="3">62.0±3.4</td><td/><td>Stage 3</td><td>Stage 4</td><td>Stage 5</td></tr><tr><td>6</td><td/><td>256</td><td colspan="2">250.5±0.5</td><td>13</td><td>512</td><td colspan="3">88.8±3.1</td><td/></tr><tr><td>7</td><td/><td>256</td><td colspan="2">226.0±2.5</td><td>Total</td><td>4224</td><td colspan="3">1689.2</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Figure 4: The average sparsity pattern of all</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">3×3 convolutional kernels in certain layer</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">stages in a unstructured pruned VGG-16.</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">Darker color means higher probability of</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">weight being kept.</td></tr></table>

Pruned architectures obtained by different approaches, all trained from scratch, averaged over 5 runs. Left: Results for PreResNet-164 pruned on CIFAR-10 by Network Slimming

Table content: <table><tr><td/><td colspan="5">0.950 Channel Pruned PreResNet-164 on CIFAR-10</td><td colspan="6">Weight Sparsified PreResNet-110 on CIFAR-100</td><td/><td colspan="3">74 Weight Sparsified DenseNet-40 on CIFAR-100</td></tr><tr><td>Test Accuracy (%)</td><td>0.925 0.930 0.935 0.940 0.945</td><td/><td colspan="3">Network Slimming Uniform Pruning</td><td>Test Accuracy (%)</td><td>72 74 76 70</td><td/><td colspan="3">Unstructured Pruning Uniform Sparsification</td><td>Test Accuracy (%)</td><td>69 70 71 72 73 68</td><td/><td>Unstructured Pruning Uniform Sparsification</td></tr><tr><td/><td>0.25</td><td>0.50</td><td>0.75 #Parameters 1.00 1.25</td><td>1.50</td><td>×10 6</td><td/><td>0.25</td><td>0.50</td><td>0.75 #Parameters 1.00</td><td>1.25</td><td>1.50 ×10 6</td><td/><td>0.2</td><td>0.4</td><td>0.6 #Parameters</td><td>0.8</td><td>1.0 ×10 6</td></tr><tr><td colspan="2">Figure 5:</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

mostly uses Adam with much lower learning rates. d) Our experiments include the large-scale ImageNet dataset, while Frankle & Carbin (2019) only considers MNIST and CIFAR. Comparisons with the Lottery Ticket Hypothesis

Table content: <table><tr><td/><td>94.0</td><td/><td/><td>VGG-16</td><td/><td>94</td><td/><td/><td>ResNet-50</td></tr><tr><td>Test Accuracy (%)</td><td>91.5 92.0 92.5 93.0 93.5</td><td/><td colspan="2">Winning Ticket lr=0.1 Random Init lr=0.1 Winning Ticket lr=0.01</td><td>Test Accuracy (%)</td><td>93 89 90 91 92</td><td/><td colspan="2">Winning Ticket lr=0.1 Random Init lr=0.1 Winning Ticket lr=0.01</td></tr><tr><td/><td/><td/><td colspan="2">Random Init lr=0.01</td><td/><td>88</td><td/><td colspan="2">Random Init lr=0.01</td></tr><tr><td/><td>91.0</td><td>20</td><td>40</td><td>60 %Prune ratio</td><td>80</td><td/><td>20</td><td>40</td><td>60 %Prune ratio</td><td>80</td></tr><tr><td/><td/><td/><td/><td/><td colspan="3">(a) Iterative Pruning</td><td/></tr><tr><td/><td>94.0</td><td/><td/><td>VGG-16</td><td/><td/><td/><td/><td>ResNet-50</td></tr><tr><td/><td/><td/><td/><td/><td/><td>93</td><td/><td/></tr><tr><td>Test Accuracy (%)</td><td>91.5 92.0 92.5 93.0 93.5</td><td/><td colspan="2">Winning Ticket lr=0.1 Random Init lr=0.1 Winning Ticket lr=0.01</td><td>Test Accuracy (%)</td><td>88 89 90 91 92</td><td/><td colspan="2">Winning Ticket lr=0.1 Random Init lr=0.1 Winning Ticket lr=0.01</td></tr><tr><td/><td>91.0</td><td>20</td><td colspan="2">40 Random Init lr=0.01 60 %Prune ratio</td><td>80</td><td>87</td><td>20</td><td colspan="2">40 Random Init lr=0.01 60 %Prune ratio</td><td>80</td></tr><tr><td/><td/><td/><td/><td/><td colspan="3">(b) One-shot Pruning</td><td/></tr><tr><td colspan="2">Figure 7:</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

in Table9. Comparisons with the Lottery Ticket Hypothesis

Table content: <table><tr><td>Dataset</td><td>Model</td><td>Unpruned</td><td colspan="2">Prune Ratio Winning Ticket Random Init</td></tr><tr><td/><td/><td/><td>20%</td><td>93.66 (±0.20) 93.79 (±0.11)</td></tr><tr><td/><td/><td/><td>40%</td><td>93.79 (±0.12) 93.77 (±0.10)</td></tr><tr><td/><td colspan="2">VGG-16 93.76 (±0.20)</td><td>60%</td><td>93.60 (±0.13) 93.72 (±0.11)</td></tr><tr><td/><td/><td/><td>80%</td><td>93.74 (±0.15) 93.72 (±0.16)</td></tr><tr><td>CIFAR-10</td><td/><td/><td>95% 20%</td><td>93.18 (±0.12) 93.05 (±0.21) 93.38 (±0.18) 93.31 (±0.24)</td></tr><tr><td/><td/><td/><td>40%</td><td>92.94 (±0.12) 93.06 (±0.22)</td></tr><tr><td/><td colspan="2">ResNet-50 93.48 (±0.20)</td><td>60%</td><td>92.56 (±0.20) 92.69 (±0.11)</td></tr><tr><td/><td/><td/><td>80%</td><td>91.83 (±0.20) 91.69 (±0.21)</td></tr><tr><td/><td/><td/><td>95%</td><td>88.75 (±0.18) 88.59 (±0.09)</td></tr><tr><td/><td colspan="4">(a) One-shot pruning with initial learning rate 0.1</td></tr><tr><td>Dataset</td><td>Model</td><td>Unpruned</td><td colspan="2">Prune Ratio Winning Ticket Random Init</td></tr><tr><td/><td/><td/><td>20%</td><td>92.78 (±0.11) 92.52 (±0.15)</td></tr><tr><td/><td/><td/><td>40%</td><td>92.80 (±0.18) 92.52 (±0.15)</td></tr><tr><td/><td colspan="2">VGG-16 92.69 (±0.12)</td><td>60%</td><td>92.72 (±0.16) 92.44 (±0.19)</td></tr><tr><td/><td/><td/><td>80%</td><td>92.75 (±0.07) 92.07 (±0.25)</td></tr><tr><td>CIFAR-10</td><td/><td/><td>95% 20%</td><td>92.58 (±0.25) 91.83 (±0.11) 91.28 (±0.15) 90.93 (±0.34)</td></tr><tr><td/><td/><td/><td>40%</td><td>91.16 (±0.07) 90.92 (±0.10)</td></tr><tr><td/><td colspan="2">ResNet-50 91.06 (±0.28)</td><td>60%</td><td>91.00 (±0.15) 90.43 (±0.16)</td></tr><tr><td/><td/><td/><td>80%</td><td>90.92 (±0.08) 89.71 (±0.18)</td></tr><tr><td/><td/><td/><td>95%</td><td>87.76 (±0.19) 87.20 (±0.17)</td></tr><tr><td/><td colspan="4">(b) One-shot pruning with initial learning rate 0.01</td></tr></table>

Sparsity patterns of PreResNet-164 pruned on CIFAR-10 by Network Slimming shown in Figure5(left) under different prune ratio. The top row denotes the total prune ratio. The values denote the ratio of channels to be kept. We can observe that for a certain prune ratio, the sparsity patterns are close to uniform (across stages). .904 0.900 0.454 0.475 0.454 0.043 0.059 0.048 0.885 0.891 0.889 0.409 0.420 0.415 0.032 0.033 0.035 0.898 0.903 0.902 0.450 0.468 0.458 0.042 0.055 0.046

Table content: <table><tr><td/><td>10%</td><td>20%</td><td>30%</td><td>40%</td><td>50%</td><td>60%</td><td>70%</td></tr><tr><td/><td colspan="7">Stage 1 0.879 0.729 0.557 0.484 0.421 0.349 0.271</td></tr><tr><td/><td colspan="7">Stage 2 0.959 0.863 0.754 0.651 0.537 0.428 0.320</td></tr><tr><td/><td colspan="7">Stage 3 0.889 0.798 0.716 0.610 0.507 0.403 0.301</td></tr><tr><td/><td>10%</td><td/><td/><td>50%</td><td/><td/><td>90%</td></tr><tr><td/><td colspan="7">0.905 0.905 0.909 0.530 0.561 0.538 0.129 0.171 0.133</td></tr><tr><td>Stage 1</td><td colspan="7">0.900 0.912 0.899 0.559 0.588 0.551 0.166 0.217 0.176</td></tr><tr><td/><td colspan="7">0.903 0.913 0.902 0.532 0.563 0.547 0.142 0.172 0.163</td></tr><tr><td/><td colspan="7">0.906 0.911 0.906 0.485 0.523 0.503 0.073 0.102 0.085</td></tr><tr><td>Stage 2</td><td colspan="7">0.912 0.911 0.915 0.508 0.529 0.525 0.099 0.114 0.111</td></tr><tr><td/><td colspan="7">0.911 0.916 0.912 0.502 0.529 0.519 0.080 0.113 0.096</td></tr><tr><td/><td>0.901 0</td><td/><td/><td/><td/><td/></tr><tr><td>Stage 3</td><td/><td/><td/><td/><td/><td/></tr></table>

Average sparsity patterns of 3×3 kernels of PreResNet-110 pruned on CIFAR-100 by unstructured pruning shown in Figure5(middle) under different prune ratio. The top row denotes the total prune ratio. The values denote the ratio of weights to be kept. We can observe that for a certain prune ratio, the sparsity patterns are close to uniform (across stages).

Table content: <table><tr><td/><td>10%</td><td>50%</td><td>90%</td></tr><tr><td/><td colspan="3">0.861 0.856 0.858 0.507 0.495 0.510 0.145 0.129 0.142</td></tr><tr><td>Stage 1</td><td colspan="3">0.843 0.844 0.851 0.484 0.486 0.479 0.123 0.115 0.126</td></tr><tr><td/><td colspan="3">0.850 0.854 0.857 0.509 0.490 0.511 0.136 0.131 0.147</td></tr><tr><td/><td colspan="3">0.907 0.905 0.906 0.498 0.487 0.499 0.099 0.088 0.100</td></tr><tr><td>Stage 2</td><td colspan="3">0.892 0.888 0.892 0.442 0.427 0.444 0.064 0.043 0.065</td></tr><tr><td/><td colspan="3">0.907 0.906 0.905 0.497 0.485 0.493 0.095 0.082 0.098</td></tr><tr><td/><td colspan="3">0.897 0.901 0.899 0.470 0.475 0.472 0.060 0.060 0.064</td></tr><tr><td>Stage 3</td><td colspan="3">0.888 0.890 0.889 0.433 0.437 0.435 0.040 0.040 0.042</td></tr><tr><td/><td colspan="3">0.898 0.900 0.899 0.473 0.477 0.473 0.060 0.061 0.063</td></tr></table>

Average sparsity patterns of 3×3 kernels of DenseNet-40 pruned on CIFAR-100 by unstructured pruning shown in Figure5(right) under different prune ratio. The top row denotes the total prune ratio. The values denote the ratio of weights to be kept. We can observe that for a certain prune ratio, the sparsity patterns are close to uniform (across stages). .969 0.914 0.883 0.875 0.844 0.836Stage 2 1.000 1.000 1.000 1.000 1.000 1.000 Stage 3 0.991 0.975 0.966 0.957 0.947 0.947 Stage 4 0.861 0.718 0.575 0.446 0.312 0.258 Stage 5 0.871 0.751 0.626 0.486 0.352 0.132

Table content: <table><tr><td>10%</td><td>20%</td><td>30%</td><td>40%</td><td>50%</td><td>60%</td></tr><tr><td>Stage 1 0</td><td/><td/><td/><td/><td/></tr></table>

Sparsity patterns of VGG-16 pruned on CIFAR-10 by Network Slimming shown in Figure

Table content: <table/>

Peer review (iclr2019):
 This paper reinvestigate several recent works on network pruning and find that the common belief about the necessity to train a large network before pruning may not hold. The authors find that training the pruned model from scratch can achieve similar, if not better, performance given enough time of training. Based on these observations, the author conclude that training a larger model followed by pruning is not necessary for obtaining an efficient model with similar performance. In other words, the pruned architecture is more important than the weights inherited from the large model. It reminds researchers to perform stronger baselines before showing complex pruning methods. The paper is well organized and written. It re-evaluate the recent progresses made on this topic. Instead of comparing approaches by simply using the numbers from previous paper, the authors perform extensive experiments to verify whether training the pruned network from scratch would work. The results are very interesting, it suggests the researchers to tune the baseline “hardly” and stick to simple approach. However, here are some places that I have concerns with: 1. The two “common beliefs” actually state one thing, that is the weights of a pre-trained larger model can potentially help optimization for a smaller model. 2. I don’t quite agree with that “training” is the first step of a pruning pipeline as illustrated in Figure 1. Actually the motivation or the common assumption for pruning is that there are already existing trained models (training is already finished) with good performance. If a trained model does not even exist, then one can certainly train various thin/smaller model from scratch as before, this is still a trial and error process. 3. “The value of pruning”. The goal of pruning is to explore a “thin” or “shallower” version of it with similar accuracy while avoiding the exhaustive architecture search with heavy training processes. Thus the first value of pruning is to explore efficient architecture while avoiding heavy training. Therefore, it should be fast and efficient, ideally with no retraining or little fine-tuning. When the pruning method is too complex to implement or requires much more time than training from scratch, it could be an overkill and adds little value, especially when the performance is not better enough. Therefore, it is more informative if the authors would report the time/complexities for pruning/fine-tuning . 4. The second value of pruning lies at understand the redundancy of the model and providing insights for more efficient architecture designs. 5. Comparing to random initialization, pruning simply provide an initialization point inherited from the larger network. The essential question the author asked is whether a subset of pre-trained weights can outperform random initialization. This seems to be a common belief in transfer learning, knowledge distillation and the studies on initialization. The authors conclude that the accuracy of an architecture is determined by the architecture itself, but not the initialization. If this is true, training from scratch should have similar (but not better) result as fine-tuning a pruned model. As the inherited weights can also be viewed as a “random” initialization. Both methods should reach equivalent good solution if they are trained with enough number of epochs. Can this be verified with experiments? 6. The experiments might not be enough to reject the common belief. The experiments only spoke that the pruned architectures can still be easily trained and encounter no difficulties during the optimization. One conjecture is that the pruned models in the previous work still have enough capacity for keeping good accuracy. What if the models are significantly pruned (say more than 70% of channels got pruned), is training from scratch still working well? It would add much value if the author can identify when training from scratch fails to match the performance obtained by pruning and fine-tuning. 7. In Section 4.1, “scratch-trained models achieve at least the same level of accuracy as fine-tuned models”. First, the ResNet-34-pruned A/B for this comparison does not have significant FLOPs reduction (10% and 24% FLOPs reduction). Fine-tuning still has advantage as it only takes ¼ of training time compare to scratch-E. Second, it is interesting that fine-tuning has generally smaller variance than stratch-E (except VGG-19). Would this imply that fine-tuning a pruned model produce more stable result? It would be more complete if there is variance analysis for the imagenet result. 8. What is the training/fine-tuning hyperparameters used in section 4.1? Note that in the experiment of Li et al, 2017, scratch-E takes 164 epochs to train from scratch, while fine-tuning takes only 40 epochs. Like suggested above, if we fine-tune it with more epochs, would it achieve equivalent performance? Also, what is the hyperparameter used in scratch-E? Note that the original paper use batch size 128. If the authors adopts a smaller batch-size for scratch-E, then it has in more iterations and could certainly result in better performance according to recent belief that small batch-size generates better. 9. The conclusion of section 5 is not quite clear or novel. Using uniform pruning ratio for pruning is expected to perform worse than automatic pruning methods as it does not consider the importance difference of each layer and. This comes back to my point 3 & 4 about the value of pruning, that is the value of pruning lies at the analysis of the redundancy of the network. There are a number of works worked on analyzing the importance of different layers of filters. So I think the “hypothesis” of “the value of automatic pruning methods actually lies in the resulting architecture rather than the inherited weight” is kind of straightforward. Also, why not use FLOPs as x-axis in Figure 3? Minor: It might be more accurate to use “L1-norm based Filter Pruning (Li et al., 2017)” as literally “channels” usually refers to feature maps, which are by-products of the model but not the model itself. I will revise my score if authors can address above concerns. --------- review after rebuttal---------- #1#2 It would be great if the authors can make it clear that training is not the always the first step and the value of pruning in introduction rather than mentioning in conclusion. Saving training time is still an important factor when training from scratch is expensive. #5 “fine-tuning with enough epochs”. I understand that the authors are mainly questioning about whether training from scratch is necessarily bad than pruning and fine-tuning. The author do find that “training from scratch is better when the number of epochs is large enough”. But we see that fine-tuning ResNet-56 A/B with 20 epochs does outperform (or is equivalent to) scratch training for the first 160 epochs, which validates “fine-tuning is faster to converge”. However, training 320 epochs (16x more comparing to 20 epochs fine-tuning and 2x comparing with normal training from scratch) is not quite coherent with the setting of “scratch B”, as ResNet-56 B just reduce 27% FLOPs. The other part of the question is still unclear, i.e., the author claimed that the accuracy of an architecture is determined by the architecture itself, but not the initialization, then both fine-tuning and scratch training should reach equivalent solution if they are well trained enough, regardless of the initialization or pruning method. The learning rate for scratch training is already well known (learning rate drop brings boost the accuracy). However, learning rate schedule for fine-tuning (especially for significantly pruned model as for reply#6) is not well explored. I wonder whether that a carefully tuned learning rate/hyperparameters for fine-tuning may get the same or better performance as scratch training. Questions: - Are both methods using the same learning rate schedule between epoch 160 and epoch 320? - The ResNets-56 A/B results in the reply#8 does not match the reported performance in reply#5. e.g., it shows 92.67(0.09) for ResNet-56-B with 40-epochs fine-tuning in reply5, but it turns out to be 92.68(±0.19) in reply#8. - It would be great if the authors can add convergence curves for fine-tuning and scratch training for easier comparison. #6 The failure case for sparse pruning on ImageNet is interesting and it would be great to have the imageNet result reported and discussed. The authors find that “when the pruned ratio is large enough, training from scratch is better by a even larger margin than fine-tuning”. This could be due to following reasons: 1. When the pruning ratio is large, the pruned model with preserved weights is significantly different from the original model, and fine-tuning with small learning rate and limited number of epochs is not enough to recover the accuracy. As mentioned earlier, tuning the hyperparameters for fine-tuning based on pruning ratio might improve the performance of fine-tuning. 2. Though the pruning ratio is large, the model used in this experiment may still have large capacity to reach good performance. How about pruning ResNet-56 with significant pruning ratios? Finally, based on above observations, it seems to me that the preserved weights is more essential for fast fine-tuning but less useful for significant pruning ratios. -------- update ---------------- The authors addressed most of my concerns. Some questions are still remaining in my comment “Review after rebuttal”, specifically, fine-tuning a pruned network may still get good performance if the hyperparameters are carefully tuned based on the pruning ratios, or in other words, the preserved weights is more essential for fast fine-tuning but less useful for significant pruning ratios. The authors may need to carefully made the conclusion from the observations. I would hope the authors can address these concerns in the future version. However, I think the paper is overall well-written and existing content is inspiring enough for readers to further explore the trainability of the pruned network. Therefore I raised my score to 7.