Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: - What is the time needed for the pruning procedure? I assume that it gets faster as more and more filters are removed, however it would be useful to show some integral value. Furthermore, one can point out that the additional fine-tuning at the end is much faster than fine tuning of the original network.
- Were you able to observe some over-fitting behavior? E.g. that more thorough fine-tuning only at the end would lead to inferior results compared to increased number of mini-batches while tuning? (this can be easily shown in Figure 8, for example).
- What are roughly the FLOPS per layer? These values would give a bit of an intuition for the lambda parameter...
- Is it possible that the abs-loss is a better oracle indicator due to increased accuracy being a sign of over-fitting? This is more an open question...
review_title: Few practical questions

final_score: rating: 6: Marginally above acceptance threshold
review_content: Authors propose a neural pruning technique starting from trained models using an approximation of change in the cost function and outperform other criteria. Authors obtain solid speedups while maintaining reasonable accuracy thanks to finetuning after pruning. Comparisons to existing methods is weak as GFLOPS graphs only show a couple simple baselines and no prior work baselines. I would be more convinced of the superiority of the approach with such comparison.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: Authors propose a strategy for pruning weights with the eventual goal of reducing GFLOP computations. The pruning strategy is well motivated using the taylor expansion of the neural network function with respect to the feature activations. The obtained strategy removes feature maps that have both a small activation and a small gradient (eqn 7). 

(A) Ideally the gradient of the output with respect to the activation functions should be 0 at the optimal, but as a result of stochastic gradient evaluations this would practically never be zero. Small variance in the gradient across mini-batches indicates that irrespective of input data the specific network parameter is unlikely to change - intuitively these are parameters that are closer to convergence. Parameters/weights that are close to convergence and also result in a small activation are intuitively good candidates for pruning. This is essentially what eqn 7 conveys and is likely to be reason why just removing weights that result in small activations is not as good of a pruning strategy (as shown by results in the paper). There are two kind of differences in weights that are removed by activation v/s taylor expansion:
1. Weights with high-activations but very low gradients will be removed by taylor expansion, but not by activation alone. 
2. Weights with low-activation but high gradients will be removed by activation criterion, but not by taylor expansion. 
It will be interesting to analyze which of (1) or (2) contribute more to the differences in weights that are removed by the taylor expansion v/s activation criterion. Intuitively it seems that weight that satisfy (1) are important because they are converged and contribute significantly to network's activation. It is possible that a modified criterion - eqn (7) + \lambda feature activation, (where \lambda needs to be found by cross-validation) may lead to even better results at the cost of more parameter tuning. 
  
(B) Another interesting comparison is with the with the optimal damage framework - where the first order gradients are assumed to be zero and pruning is performed using the second-order information (also discussed by authors in the appendix). Critically, only the diagonal of the Hessian is computed. There is no comparison with optimal damage as authors claim it is memory and computation inefficient. Back of envelope calculations suggest that this would result only in 50% increase in memory and computation during pruning, but no loss in efficiency during testing. Therefore from a standpoint of deployment, I don't think this missing comparison is justified. 

(C) The eventual goal of the authors is to reduce GFLOPs. Some recent papers have proposed using lower precision computation for this. A comparison in GFLOPs with lower precision v/s pruning would be a great. While both these approaches are complementary and it is expected that combining both of them can lead to superior performance than either of the two - it is unclear when we are operating in the low-precision regime how much pruning can be performed. Any analysis on this tradeoff would be great (but not necessary).

(D) On finetuning, authors report results of AlexNet and VGG on two different datasets - Flowers and Birds respectively. Why is this the case? It would be great to see the results of both the networks on both the datasets. 

(E) Authors report there is only a small drop in performance after pruning. Suppose the network was originally trained with N iterations, and then M finetuning iterations were performed during pruning. This means that pruned networks were trained for N + M iterations. The correct comparison in accuracies would be if we the original network was also trained for N + M iterations. In figure 4, does the performance at 100% parameters reports accuracy after N+M iterations or after N iterations? 

Overall I think the paper is technically and empirically sound, it proposes a new strategy for pruning:
(1) Based on taylor expansion
(2) Feature normalization to reduce parameter tuning efforts. 
(3) Iterative finetuning. 
However, I would like to see some comparisons mentioned in my comments above. If those comparisons are made I would change my ratings to an accept. 








review_title: Empirically justified pruning strategy, a few missing comparisons

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper presents a novel way of pruning filters from convolutional neural networks with a strong theoretical justification. The proposed methods is derived from the first order Taylor expansion of the loss change while pruning a particular unit. This leads to simple weighting of the unit activation with its gradient w.r.t. loss function and performs better than simply using the activation magnitude as the heuristic for pruning. This intuitively makes sense, as we would like to remove not only the filters with low activation, but also filters where the incorrect activation value would not have small influence on the target loss.

Authors thoroughly investigate multiple baselines, including an oracle which sets an upper bound on the target performance even though it is computationally expensive. The devised method seems to be quite elegant and authors show that it generalizes well on multiple tasks and is computationally more than feasible as it is easy to combine with traditional fine tuning procedure. Also, the work clearly shows the trade-offs of increased speed and decreased performance, which is useful for practical applications.

It would be also useful to compare against different baselines, e.g. [1]. However this method seems to be more useful as it does not involve training of a new network (and thus is probably much faster).

Suggestion - maybe it can be extended in the future towards also removing only parts of the filters(e.g. for the 3D convolution)? This may be more complicated as it would need to change the implementation of convolution operator, but can lead to further speedup.

[1] https://arxiv.org/abs/1503.02531
review_title: Strong experimental evaluation of a theoretically justified pruning method

final_score: rating: null
review_content: - Is it possible to combine criterions to increase pruning confidence?
- To be sure what you mean by feature map, do you mean the resulting map from one convolution kernel, or the accumulation of multiple kernels into one feature map (this is the definition that is usually meant)? If it's the latter, have you tried pruning by kernel rather than by a set of kernels?
- It is not clear to me how many samples are used for each criterion evaluation? The entire training set or K mini-batch updates?  If K, what is the impact of the choice of K on the results? Is using the entire training set impractical? Could it lead to better results?
review_title: clarifications

Meta-review: The paper presents a method for pruning filters from convolutional neural networks based on the first order Taylor expansion of the loss change. The method is novel and well justified with extensive empirical evaluation.
Decision: Accept (Poster)