Title: Pruning Filters for Efficient ConvNets
Authors: Hao Li
Asim Kadav
Igor Durdanovic
Hanan Samet
Hans Peter Graf
Venue: ICLR 2017
Keywords: Computer vision
Deep learning

Abstract: The success of CNNs in various applications is accompanied by a significant increase in the computation and parameter storage costs. Recent efforts toward reducing these overheads involve pruning and compressing the weights of various layers without hurting original accuracy.  However, magnitude-based pruning of weights reduces a significant number of parameters from the fully connected layers and may not adequately reduce the computation costs in the convolutional layers due to irregular sparsity in the pruned networks. We present an acceleration method for CNNs, where we prune filters from CNNs that are identified as having a small effect on the output accuracy. By removing whole filters in the network together with their connecting feature maps, the computation costs are reduced significantly. In contrast to pruning weights, this approach does not result in sparse connectivity patterns. Hence, it does not need the support of sparse convolution libraries and can work with existing efficient BLAS libraries for dense matrix multiplications. We show that even simple filter pruning techniques can reduce inference costs for VGG-16 by up to 34% and ResNet-110 by up to 38% on CIFAR10 while regaining close to the original accuracy by retraining the networks.  


Review discussion:
This paper proposes a very simple idea (prune low-weight filters from ConvNets) in order to reduce FLOPs and memory consumption. The proposed method is experimented on with VGG-16 and ResNets on CIFAR10 and ImageNet.
Pros:
- Creates *structured* sparsity, which automatically improves performance without changing the underlying convolution implementation
- Very simple to implement
Cons:
- No evaluation of how pruning impacts transfer learning
I'm generally positive about this work.
While the main idea is almost trivial, I am not aware of any other papers that propose exactly the same idea and show a good set of experimental results.
Therefore I'm inclined to accept it. The only major downside is that the paper does not evaluate the impact of filter pruning on transfer learning.
For example, there is not much interest in the tasks of CIFAR10 or even ImageNet.
Instead, the main interest in both academia and industry is the value of the learned representation for transferring to other tasks. One might expect filter pruning (or any other kind of pruning) to harm transfer learning. It's possible that the while the main task has about the same performance, transfer learning is strongly hurt. This paper has missed an opportunity to explore that direction.
Nit: Fig 2 title says VGG-16 in (b) and VGG_BN in (c). Are these the same models?
Thanks for your comments. We demonstrate how to apply the structured sparsity to the most commonly used Resnets (which was non-trivial). Most prior work demonstrate their results on Alexnet or VGG which have lots of free parameters.
However, these networks are considered largely inefficient and are no longer state of the art.
Hence, we hope our methods and experience to prune Resnets can be helpful to the community.
We agree that transfer learning results are important since no one uses to test on Imagenet in real world problems.
However, transfer learning can be pretty robust. With transfer learning, new classes often get mapped to some other class from the original dataset and the discriminative power is often maintained. Recent work has shown that transfer learning is robust to fewer classes and examples[1]. Training on fewer classes may even improve performance in some cases[1].
Furthermore, they find that removing target classes from Imagenet leads to small reduction in performance (Removing PASCAL classes from ImageNet leads to an insignificant reduction in performance for the PASCAL TL task" [Table 3]).
However, from our experiments it is not clear if filters are removed at the class/concept level.
We performed a simple transfer learning experiment using our pre-trained and pruned resnet-34-b (from Table 1). We use the network to perform action recognition on the UCF-11 dataset. We use a simple CNN+LSTM model and connect the second last (pooling) layer as in input to a two layer LSTM. We achieve 82.7% using Resnet-34 and 83.6% using the pruned resnet-34b on the UCF-11 dataset.
Additionally, during feature generation we have proportional time savings. In our simple transfer learning model, the feature generation is a one-time task.
However, when using optical flow or spatial attention based models where the features are extracted from the convolution layer, the CNN is re-trained along with the LSTM. In these cases, the time savings can be significant over multiple iterations.
While this is just one experiment where the TL is not affected and we get a slightly better accuracy, we can expand our experiments to other datasets to understand how the transfer learning performance is affected.
[1] What makes ImageNet good for transfer learning? https://arxiv.org/abs/1608.08614
This paper proposes a simple method for pruning filters in two types of architecture to decrease the time for execution.
Pros:
- Impressively retains accuracy on popular models on ImageNet and Cifar10
Cons:
- There is no justification for for low L1 or L2 norm being a good selection criteria. There are two easy critical missing baselines of 1) randomly pruning filters, 2) pruning filters with low activation pattern norms on training set.
- There is no direct comparison to the multitude of other pruning and speedup methods.
- While FLOPs are reported, it is not clear what empirical speedup this method gives, which is what people interested in these methods care about. Wall-clock speedup is trivial to report, so the lack of wall-clock speedup is suspect.
We thank the reviewer for the kind suggestions to improve the paper. We have updated the paper with more results added in the appendix.
1. “There is no justification for for low L1 or L2 norm being a good selection criteria. There are two easy critical missing baselines of 1) randomly pruning filters, 2) pruning filters with low activation pattern norms on training set.”
In Appendix 6.1, we compared our approach with pruning random filters and largest filters for VGG-16 on CIFAR-10. We find that pruning smallest filters works better than random pruning for most of the layers at different pruning ratios, e.g., all layers have better performance at the pruning ratio of 90%. It also clearly outperforms pruning filters with largest L1-norm, indicating the importance of large filters for certain layers.
Updates on Jan.18: In Appendix 6.2, we compared our approach with activation-based feature map pruning, which needs sample data as input to determine which feature maps to prune. We compute 5 statistics over the feature map with sampled training data, including: 1) the mean value. 2) the standard deviation 3) L1-norm 4)  L2-norm and 5) the contribution variance (Polyak & Wolf (2015)).  We find that L1-norm filter pruning has clear advantage over criterion 1, 3, 4 and 5. The mean-std criterion has better or similar performance by up to pruning ratio of 60% but drops quickly after 60% for certain layers.  We think L1-norm is good heuristic for filter selection considering that it is data free.
2. “There is no direct comparison to the multitude of other pruning and speedup methods.”
We had some discussions about the relationship with recent work in the related work and section 2.1. Past work on weights pruning (Han et al 2016) and recent group sparse based methods (Zhou et al 2016, Wen et al 2016) need to specify layer-wise thresholds or parameters, which is in the continues space and could be hard to set especially for CNNs with many layers. Filter-based pruning can be used in addition to other speedup methods such as low-rank approximation, quantisation and binary networks.
We choose to report results on ResNet as it is the state-of-the-art model for tasks such as image/video recognition rather than AlexNet or VGGNet. It is also a quite thin network and does not have large free parameters in the FC layer that can be pruned away as in AlexNet/VGGNet, which is challenging to make further speedup. We are not aware of other pruning work done for deep ResNets at the time of submission.
3. "Wall-clock speedup is trivial to report, so the lack of wall-clock speedup is suspect.”
In Appendix 6.4, we report the wall-clock inference time on the test set of CIFAR-10 and the validation set of ImageNet 2012 with the latest GPU (Titan X Pascal) and library (cuDNN 5.1). As can be seen in Table 3, the saved wall-clock time is close to the FLOP reduction.
Since we physically prune the filters by creating a smaller model and then copy the weights, there are no masks or sparsity introduced to the original dense BLAS operations. The wall-clock time of the pruned model is the same as creating a model with less number of filters from scratch, which is dependent on the underlying hardware and software implementations.
Thanks for the timings and additional experiments which do indeed shed more light on the efficacy of L1 pruning. Very interesting to see the mean-std criterion being effective -- it makes sense that low variance filter responses are disposable as they act more as biases, and I agree that it seems L1 norm is a good heuristic then for data-free selection. I think the comparisons to all these other selection criteria should be part of the main paper as it is empirical justification for your method, and then this paper becomes a good resource for pruning criteria comparisons.
This paper prunes entire groups of filters in CNN so that they reduce computational cost and at the same time do not result in sparse connectivity. This result is important to speed up and compress neural networks while being able to use standard fully-connected linear algebra routines.
The results are a 10% improvements in ResNet-like and ImageNet, which may be also achieved with better design of networks. New networks should have been also compared, but this we know it is time-consuming.
A good paper with some useful results.
The idea of "pruning where it matters" is great. The authors do a very good job of thinking it through, and taking to the next level by studying pruning across different layers too.
Extra points for clarity of the description and good pictures. Even more extra points for actually specifying what spaces are which layers are mapping into which (\mathbb symbol - two thumbs up!).
The experiments are well done and the results are encouraging.
Of course, more experiments would be even nicer, but is it ever not the case?
My question/issue - is the proposed pruning criterion proposed? Yes, pruning on the filter level is what in my opinion is the way to go, but I would be curious how the "min sum of weights" criterion compares to other approaches.
How does it compare to other pruning criteria? Is it better than "pruning at random"?
Overall, I liked the paper.
We thank the reviewer for the kind feedback and comments. We have updated the paper with comparisons with other criteria in the appendix.
1. “is the proposed pruning criterion proposed?”
In the level of filter or feature map pruning, we are not aware of similar approaches. Past work on feature map pruning used to evaluate the importance of a feature map based on the variance of activation-norm [1] or the loss of validation accuracy if gets pruned [2, 3], but not based on the norm of the filter.
2. "Is it better than "pruning at random”?
Yes, it is better than “pruning at random”. In Appendix 6.1, we compared with pruning random filters and largest filters. Pruning smallest filters outperforms both approaches.
3. How does it compare to other pruning criteria?
In Appendix 6.2, we compared with feature map pruning based on activation patterns. We compute 5 criteria on the feature map over all training data, including: 1) the mean value. 2) the standard deviation 3) L1-norm 4)  L2-norm and 5) the contribution variance (Polyak & Wolf (2015)).  We find that L1-norm based filter pruning has clear advantage over criterion 1, 3, 4 and 5. The mean-std criterion has better or similar performance by up to pruning ratio of 60% but drops quickly after that for certain layers. We think the L1-norm is a good heuristic for filter selection considering its data-free property.
In Appendix 6.3, we also compared with L2-norm for filter selection and observe a slightly better performance over L2-norm.
[1] Polyak & Wolf, Channel-Level Acceleration of Deep Face Representations, IEEE Access, 2015
[2] Zheng et al, Compact Deep Neural Networks for Device Based Image Classification, ICME workshop, 2015
[3] Anwar et al, Structured Pruning of Deep Convolutional Neural Networks, arXiv, 2015
We would like to thank all reviewers for their comments and suggestions. We have added new experiments based on reviewer suggestions (and will continue to add more as necessary). The new experiments from Hao in Appendix A report fprop times for the pruned networks and random pruning results that address Reviewer 4's question about random pruning and Reviewer 2's concerns.

Meta-review:
role: PC
data: comment: The paper presents a simple but effective approach for pruning ConvNet filters with extensive evaluation using several architectures on ImageNet and CIFAR-10.

Decision: Accept (Poster)