Title: Pruning Convolutional Neural Networks for Resource Efficient Inference
Authors: Pavlo Molchanov
Stephen Tyree
Tero Karras
Timo Aila
Jan Kautz
Venue: ICLR 2017
Keywords: Deep learning
Transfer Learning

Abstract: We propose a new formulation for pruning convolutional kernels in neural networks to enable efficient inference. We interleave greedy criteria-based pruning with fine-tuning by backpropagation-a computationally efficient procedure that maintains good generalization in the pruned network. We propose a new criterion based on Taylor expansion that approximates the change in the cost function induced by pruning network parameters. We focus on transfer learning, where large pretrained networks are adapted to specialized tasks. The proposed criterion demonstrates superior performance compared to other criteria, e.g. the norm of kernel weights or feature map activation, for pruning large CNNs after adaptation to fine-grained classification tasks (Birds-200 and Flowers-102) relaying only on the first order gradient information. We also show that pruning can lead to more than 10x theoretical reduction in adapted 3D-convolutional filters with a small drop in accuracy in a recurrent gesture classifier. Finally, we show results for the large-scale ImageNet dataset to emphasize the flexibility of our approach.

TL;DR: New approach for removing unnecessary conv neurons from network. Work is focused on how to estimate importance fast and efficiently by Taylor expantion.

Review discussion:
- What is the time needed for the pruning procedure? I assume that it gets faster as more and more filters are removed, however it would be useful to show some integral value.
Furthermore, one can point out that the additional fine-tuning at the end is much faster than fine tuning of the original network.
- Were you able to observe some over-fitting behavior? E.g. that more thorough fine-tuning only at the end would lead to inferior results compared to increased number of mini-batches while tuning? (this can be easily shown in Figure 8, for example).
- What are roughly the FLOPS per layer? These values would give a bit of an intuition for the lambda parameter...
- Is it possible that the abs-loss is a better oracle indicator due to increased accuracy being a sign of over-fitting? This is more an open question...
Please see answers in-line:
Q:What is the time needed for the pruning procedure? I assume that it gets faster as more and more filters are removed, however it would be useful to show some integral value.
Furthermore, one can point out that the additional fine-tuning at the end is much faster than fine tuning of the original network.
A:We prune a single neuron at every pruning iteration. Pruning is performed sequentially after every K mini-batch updates. I.e., the parameter K defines the how frequently a neuron is pruned from the network. E.g., 10 neurons will be removed after 10*K mini-batch updates.
The comment that pruning gets faster is absolutely right. We remove the entire neuron (convolutional feature map) and the forward and backward passes get faster as a result. From an implementation point of view we found that it is faster to maintain the same model during training and apply pruning by setting the gate element g to 0 in eq. 2 rather than recompiling the model after each iteration. After pruning is completed, we generate a new model according to the learned weights and pruning gates.
Fine tuning after pruning as a separate step was performed only for R3DCNN on nvGesture and VGG16 on ImageNet. For R3DCNN pruning took 100 epochs, and additional fine tuning took 14 additional epochs. For VGG-16 the entire pruning shown in Fig. 9 was computed during 7 epochs over the ImageNet dataset and additional fine-tuning took 7 more epochs. After profiling the networks we observed that for VGG16 pruning from 31 to 12 and to 8 GFLOPs speeds up fine-tuning 1.83x and 2.35x, respectively. Pruning R3DCNN from 38 to 3 GFLOPs sped-up fine-tuning 4.7x.
Q:Were you able to observe some over-fitting behavior? E.g. that more thorough fine-tuning only at the end would lead to inferior results compared to increased number of mini-batches while tuning? (this can be easily shown in Figure 8, for example).
A:During pruning for transfer learning we observed that during first iterations the training and testing losses decrease because pruning acts as a regularizer. With more pruning, the training loss stays low but testing loss increases. It is not clear if behaviour can be explained by overfitting or reduced generalization due to the pruning.
The question of balance between increasing the number of mini-batches versus more thorough fine-tuning is indeed interesting. We did not run this comparison.
However, our intuition is that with a larger number of mini-batches, the network can stabilize better after the last pruning iteration and we are able to collect more statistics for the criteria and their estimate will be more accurate.
Q:What are roughly the FLOPS per layer? These values would give a bit of an intuition for the lambda parameter…
A:The lambda parameter used in our experiments is always equal to 1e-3.
For VGG-16: FLOPs per convolutional neuron in every layer: [3.1, 57.8, 14.1, 28.9, 7.0, 14.5, 14.5, 3.5, 7.2, 7.2, 1.8, 1.8, 1.8, 1.8], fully connected neurons: [0.05, 0.008], linear layer before softmax is not included.
For AlexNet: FLOPs per convolutional neuron in every layer: [2.3, 1.7, 0.8, 0.6, 0.6] and per fully connected neuron: [0.018, 0.008], linear layer before softmax is not included.
For R3DCNN: FLOPs per convolutional neuron: [5.6, 86.9, 21.7, 43.4, 5.4, 10.8, 1.4, 1.4] and for fully connected [0.016, 0.008], linear layer before softmax is not included.
Q:Is it possible that the abs-loss is a better oracle indicator due to increased accuracy being a sign of over-fitting? This is more an open question...
A:Intuitively, the oracle-loss is prone to overfitting as we remove neurons that potentially reduce the loss of the training set. With oracle-abs we remove neurons that have the smallest impact on the output of the network. For the experiment on Fig. 3, we do not recompute the oracle after each pruning iteration (this would be computationally infeasible), therefore oracle-loss-based pruning will cause more instability into the network.
We will revise the paper to reflect these answers.
- Is it possible to combine criterions to increase pruning confidence?
- To be sure what you mean by feature map, do you mean the resulting map from one convolution kernel, or the accumulation of multiple kernels into one feature map (this is the definition that is usually meant)? If it's the latter, have you tried pruning by kernel rather than by a set of kernels?
- It is not clear to me how many samples are used for each criterion evaluation? The entire training set or K mini-batch updates?  If K, what is the impact of the choice of K on the results? Is using the entire training set impractical? Could it lead to better results?
Thank you for asking these clarification questions, we will incorporate answers into the paper for better explanation.
Q: - Is it possible to combine criterions to increase pruning confidence?
A: Interesting suggestion. During our greedy pruning we are estimating importance of neurons (feature maps for convolutional layers) by selected criterion. The goal is to have criterion being highly correlated with oracle (true change in loss by pruning, which is computationally infeasible to compute) as shown in Table 1.
Indeed, several criteria can be efficiently combined to improve correlation with oracle and better pruning as a result.
Simple averaging of criteria didn’t lead to improvement, for example Spearman's rank correlation:
VGG-16/Birds 200 (similar to Table 1)
Acivation (mean) + Taylor: per layer 0.56; all layers 0.64
Weight + Taylor: per layer 0.72; all layers 0.73
Taylor alone: per layer 0.73; all layers 0.73
For a linear combination, we will need to compute weights for each criterion on a separate subset of data given oracle which will be unfair comparison to oracle-free pruning.
Q: - To be sure what you mean by feature map, do you mean the resulting map from one convolution kernel, or the accumulation of multiple kernels into one feature map (this is the definition that is usually meant)? If it's the latter, have you tried pruning by kernel rather than by a set of kernels?
A: By feature map we mean z_l^(k) described in equation (2), computed by applying individual convolutions, summing them together, adding bias and applying activation function.
For example VGG-16 has 64 feature maps as the output of the first convolutional layer, 64 as second, 128 for the third convolutional layer etc.
By convolutional kernel we mean a 3D dimensional tensor (nymber_of_input_channels x kernel_width x kernel_height) and after pruning we remove it completely (together with corresponding biases). If pruning by kernel you mean removing individual connections (e.g. kernel 1 uses only input channels 2,5,9; kernel 2 uses input channels 1,3,9) then we don’t do it. It is absolutely possible with framework described in the paper, however, such pruning doesn’t result in inference speedup on current architectures of GPGPUs with available SW. We will need to change HW and/or SW to see speedup given by such pruning.
With pruning described in the paper we get speedup on current HW and SW without any modifications which is the focus of the paper.
Q: - It is not clear to me how many samples are used for each criterion evaluation? The entire training set or K mini-batch updates? If K, what is the impact of the choice of K on the results? Is using the entire training set impractical? Could it lead to better results?
A: For criterion evaluation we only use samples from K mini-batch updates.
Indeed, choice of K has impact on results, please see Figure 6 and Figure 8. We jointly apply fine-tuning and criteria evaluation on the same data for K mini-batches. Selecting K to be larger (even up to the entire training set) will lead to better approximation of importance.
However, pruning will take longer; user needs to select this parameter according to available resources.
Authors propose a neural pruning technique starting from trained models using an approximation of change in the cost function and outperform other criteria. Authors obtain solid speedups while maintaining reasonable accuracy thanks to finetuning after pruning. Comparisons to existing methods is weak as GFLOPS graphs only show a couple simple baselines and no prior work baselines. I would be more convinced of the superiority of the approach with such comparison.
We appreciate comment of the reviewer to consider more baselines.
We implemented 3 extra methods: 1) Optimal Brain Damage, 2) APoZ, 3) pruning by regularization.
Our conclusions on comparing with them:
1) OBD shows good results that are slightly worse compared to Taylor criteria. OBD requires computation of diagonal of Hessian which takes extra 50% of computations as pointed in the original papers, or extra 3x to 30x computations as in our implementation. We putted details of our implementation into appendix A.6.
2) Proposed Taylor criteria shows significantly better performance than criterion based on Average percentage of zeros (APoZ).
3) Taylor criteria shows better accuracy when compared with second norm regularization based pruning as shown in appendix A.4.
Thank you for the added baselines, they make the work more convincing, I bumped my rating up to 6.
This paper presents a novel way of pruning filters from convolutional neural networks with a strong theoretical justification. The proposed methods is derived from the first order Taylor expansion of the loss change while pruning a particular unit. This leads to simple weighting of the unit activation with its gradient w.r.t. loss function and performs better than simply using the activation magnitude as the heuristic for pruning. This intuitively makes sense, as we would like to remove not only the filters with low activation, but also filters where the incorrect activation value would not have small influence on the target loss.
Authors thoroughly investigate multiple baselines, including an oracle which sets an upper bound on the target performance even though it is computationally expensive. The devised method seems to be quite elegant and authors show that it generalizes well on multiple tasks and is computationally more than feasible as it is easy to combine with traditional fine tuning procedure.
Also, the work clearly shows the trade-offs of increased speed and decreased performance, which is useful for practical applications.
It would be also useful to compare against different baselines, e.g. [1].
However this method seems to be more useful as it does not involve training of a new network (and thus is probably much faster).
Suggestion - maybe it can be extended in the future towards also removing only parts of the filters(e.g. for the 3D convolution)? This may be more complicated as it would need to change the implementation of convolution operator, but can lead to further speedup.
[1] https://arxiv.org/abs/1503.02531
We would like to thank reviewer for comments and suggestions.
We added 3 more comparisons as baseline methods: Optimal Brain Damage, APoZ and pruning by regularization.
Taylor criterion performs slightly better than OBD and requires no change to back-propagation algorithm, OBD on the other hand requires back-propagation of diagonal of the Hessian which adds extra computations and is at least 3 times slower in our implementation.
Proposed criterion demonstrates better performance than APoZ and pruning by regularization on network transfer tasks.
"Dark Knowledge" or network distillation is a very interesting approach to reduce computations by training a smaller network.
However it requires search of network architecture, which is done automatically for the proposed approach.
Proposed criterion and method can be used for pruning network parameters at different levels: individual connections, part of filters or entire filter.  Modern GPU hardware exploits regularities in computation for high throughput and only pruning the entire filter provides immediate speed up. Considering other levels of pruning is our goal for future work.
Analysing pre-review questions we added section 3.6 Speed up measurements where we measured wall-clock time of inference.
Also we added FLOPs numbers per layer used in FLOPs regularization for pruning.
Authors propose a strategy for pruning weights with the eventual goal of reducing GFLOP computations. The pruning strategy is well motivated using the taylor expansion of the neural network function with respect to the feature activations. The obtained strategy removes feature maps that have both a small activation and a small gradient (eqn 7).
(A) Ideally the gradient of the output with respect to the activation functions should be 0 at the optimal, but as a result of stochastic gradient evaluations this would practically never be zero. Small variance in the gradient across mini-batches indicates that irrespective of input data the specific network parameter is unlikely to change - intuitively these are parameters that are closer to convergence. Parameters/weights that are close to convergence and also result in a small activation are intuitively good candidates for pruning. This is essentially what eqn 7 conveys and is likely to be reason why just removing weights that result in small activations is not as good of a pruning strategy (as shown by results in the paper). There are two kind of differences in weights that are removed by activation v/s taylor expansion:
1. Weights with high-activations but very low gradients will be removed by taylor expansion, but not by activation alone.
2. Weights with low-activation but high gradients will be removed by activation criterion, but not by taylor expansion.
It will be interesting to analyze which of (1) or (2) contribute more to the differences in weights that are removed by the taylor expansion v/s activation criterion. Intuitively it seems that weight that satisfy (1) are important because they are converged and contribute significantly to network's activation. It is possible that a modified criterion - eqn (7) + \lambda feature activation, (where \lambda needs to be found by cross-validation) may lead to even better results at the cost of more parameter tuning.
(B) Another interesting comparison is with the with the optimal damage framework - where the first order gradients are assumed to be zero and pruning is performed using the second-order information (also discussed by authors in the appendix). Critically, only the diagonal of the Hessian is computed. There is no comparison with optimal damage as authors claim it is memory and computation inefficient. Back of envelope calculations suggest that this would result only in 50% increase in memory and computation during pruning, but no loss in efficiency during testing.
Therefore from a standpoint of deployment, I don't think this missing comparison is justified.
(C) The eventual goal of the authors is to reduce GFLOPs. Some recent papers have proposed using lower precision computation for this. A comparison in GFLOPs with lower precision v/s pruning would be a great.
While both these approaches are complementary and it is expected that combining both of them can lead to superior performance than either of the two - it is unclear when we are operating in the low-precision regime how much pruning can be performed. Any analysis on this tradeoff would be great (but not necessary).
(D) On finetuning, authors report results of AlexNet and VGG on two different datasets - Flowers and Birds respectively. Why is this the case? It would be great to see the results of both the networks on both the datasets.
(E) Authors report there is only a small drop in performance after pruning. Suppose the network was originally trained with N iterations, and then M finetuning iterations were performed during pruning. This means that pruned networks were trained for N + M iterations. The correct comparison in accuracies would be if we the original network was also trained for N + M iterations. In figure 4, does the performance at 100% parameters reports accuracy after N+M iterations or after N iterations?
Overall I think the paper is technically and empirically sound, it proposes a new strategy for pruning:
(1) Based on taylor expansion
(2) Feature normalization to reduce parameter tuning efforts.
(3) Iterative finetuning.
However, I would like to see some comparisons mentioned in my comments above. If those comparisons are made I would change my ratings to an accept.
We would like to thank reviewer for comments and interesting suggestions. Please see answers on comments listed below:
(A) It is a very interesting observation of how gradient and activation can contribute to the Taylor criteria. Reviewer’s observation that gradient of the cost function with respect to the activation functions tends to zero for a converged network is on point.
As a result of stochasticity in the training procedure, there is a small variance of the gradient which after multiplication with activation value produces Taylor criterion.
Reviewer's suggestion to verify assumptions (1) and (2) sounds interesting and we are on the last stage of verifying obtained results.
(Jan 16) This analysis was added as appendix A.7. Our conclusions is that low activations seem to correlate more strongly with the Taylor criteria for less important neurons.
However, both activation and gradients contribute to the set of neurons with higher Taylor scores.
We evaluated the idea of combining Taylor and activation criteria in appendix A.5 and found that it does not provide any significant improvement. It seems that Taylor criteria already has all information contained in activations.
(B) We agree that optimal brain damage (OBD) is an important baseline for any pruning algorithm.
Also, it is interesting to see how much second order information can provide for pruning. Reviewer’s comment on that OBD takes 50% more memory and computations is confirmed in the “Efficient backprop” paper by LeCun et al. 1998.
However, we found that efficiently implementing backpropagation of the diagonal of the Hessian for a large network is a rather challenging task that requires significant changes to the framework.
We implemented computation of the diagonal of the Hessian in a different way explained in the section 2.2, in the paragraph related to OBD and in appendix A.6.
Comparison of OBD and Taylor criteria demonstrated slightly better results of the latest in terms of correlation coefficient with oracle and actual iterative pruning. OBD is 50% (theoretically) to 300% (our implementation) slower than Taylor criteria that relies on first order gradient only.
These observations are added to section 3.3.
(C) We agree that using math with lower precision speed ups inference time, and that it can be applied complementary to pruning. Combining both techniques leads to even better speed up as shown in [1]. Pruning helps with 100% of the hardware installed today, while lowering precision is likely to become more important in the future when hardware with faster low-precision math will be widely available.
[1]S.Han, H. Mao, W. Dally, “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding”, ICLR 2016
(D) Originally we did not want to clutter paper to much with results that are similar in nature, therefore decided to show detailed analysis of 2 networks on 2 different datasets. To make our contribution stronger we added analysis of Spearman’s rank coefficient correlation for missing combinations (VGG16/Flowers and AlexNet/Birds) into Table 1. There is a surprising result of OBD on VGG16/Flowers which is quite low. We observed significant overfitting of VGG16 (Flowers training set is 3x smaller than Birds) and the sparsest oracle out of all tests what might be the reason.
(E) Observation is correct, we refer to the result of unpruned network after fine-tuning for N iterations only. Resulting classification accuracies can be improved with further fine-tuning of pruned or unpruned networks. We added the last paragraph in section 3.2 addressing this comment.  Unpruned networks tested on ImageNet datasets do not show improvement after further fine-tuning because of long original training till convergence.
Thanks for reporting results of additional experiments.
(A) Figure 13 look great. Agreed with the conclusion that modified loss doesnot help much.
In addition to plots in Figure 14, I would suggest making a plot with x and y axis as gradient and activation and then color each point with whether it was pruned or not based on Taylor criterion. This single figure would convey all the information in the 4 subplots in Fig. 14 and could be more insightful.
(B) I am curious about the comparison with OBD - it seems from figure 4, that the tradeoff between accuracy v/s number of parameters is similar between OBD and the proposed method.
However, GFLOP performance of the proposed method is significantly better than OBD. Why is this the case?
The paper thoroughly evaluates many aspects of network pruning.
While the results show minor improvement over OBD in terms on % of pruned parameter v/s accuracy, the paper would be a source of useful knowledge in the community.
Thank you for thoughtful suggestions.
(A) We have looked at different forms of Fig. 14, and converged on what is currently in the paper. We are happy to try your suggestion to improve the figure.
(B) The main reason is that reducing FLOPs is biased towards pruning of the first layers, whereas reducing % of parameters is biased towards the last convolutional layers. The first 4 layers have 30% of total FLOPs and only 6% of total convolutional parameters, whereas the last 4 layers have 3% of total FLOPs and 48% of total convolutional parameters.
That is why for pruning focused on reducing parameters (network compression) it matters to prune the last 4 layers correctly, whereas for reducing the number of FLOPs, it is the first 4 layers.
We provided a detailed comparison of Spearman’s rank correlation for different layers in Table 3. It can be seen that for the last 4 layers, OBD has mean correlation of 0.66 versus 0.74 for Taylor, considering the low median importance of this layers (Figure 2), the difference is not significant and therefore both techniques prune this layers with the same success and show similar results for pruning focused on reducing % of parameters.
On the other hand, the first 4 layers have a mean correlation of 0.59 for OBD and 0.85 for Taylor. The difference here is significant, especially when taking into account of how important these layers are. Our Taylor-based criterion ranks the neurons in these layers better and therefore shows much better performance of pruning wrt. to FLOPs. Since the focus of the paper is on speeding up inference, we consider FLOPs to be more significant.
To the best of our knowledge, none of the recent pruning related research papers compare against OBD primarily because of the increased computational complexity. OBD is 50% to 300% slower than our Taylor criterion and even though it has similar results on pruning for compression, ours has a significant advantage in terms of speed and pruning wrt. FLOPs.
This a good paper on network pruning specially with the extensive evaluations. I'm just curious about the reasons behind having Figure 14. Do you have any explanation on how activation is correlated to the gradient in Taylor criterion?
Thanks.
Thanks for the clarifications. The contributions of the paper are much clearer now. I have revised my ratings.
We would like to thank reviewers for their comments and suggestions. We added a new revision which addresses main points requested for clarification.
List of changes:
- Added comparison with Optimal Brain Damage. See section 2.2, paragraph "Relation to Optimal Brain Damage" for discussion, appendix A.6 for implementation details, Tables 1 and 3, Figures 4 and 5 for results and comparisons
- Comparison with average percentage of zeros criterion (APoZ) proposed by Hu et al. (2016)
- Comparison with pruning by regularization in appendix A.4
- Wall-clock time measurements for inference of pruned networks in section 3.6
- Results of combining Taylor and activation criteria in appendix A.5
- (Jan 16) Correlation of Taylor criterion with gradient and activation, appendix A.7

Meta-review:
role: PC
data: comment: The paper presents a method for pruning filters from convolutional neural networks based on the first order Taylor expansion of the loss change. The method is novel and well justified with extensive empirical evaluation.

Decision: Accept (Poster)