Adaptive Activation-based Structured Pruning

Abstract: Pruning is a promising approach to compress complex deep learning models in order to deploy them on resource-constrained edge devices. However, many existing pruning solutions are based on unstructured pruning, which yield models that cannot efficiently run on commodity hardware, and require users to manually explore and tune the pruning process, which is time consuming and often leads to sub-optimal results. To address these limitations, this paper presents an adaptive, activation-based, structured pruning approach to automatically and efficiently generate small, accurate, and hardware-efficient models that meet user requirements. First, it proposes iterative structured pruning using activation-based attention feature maps to effectively identify and prune unimportant filters. Then, it proposes adaptive pruning policies for automatically meeting the pruning objectives of accuracy-critical, memory-constrained, and latency-sensitive tasks. A comprehensive evaluation shows that the proposed method can substantially outperform the state-of-the-art structured pruning works on CIFAR-10 and ImageNet datasets. For example, on ResNet-56 with CIFAR-10, without any accuracy drop, our method achieves the largest parameter reduction (79.11%), outperforming the related works by 22.81% to 66.07%, and the largest FLOPs reduction (70.13%), outperforming the related works by 14.13% to 26.53%.

Reviews and author responses:

This paper proposes iterative structured pruning methods using activation-based attention feature maps and an adaptive threshold selection strategy. Inspired by attention transfer, Activation-based attention feature maps are constructed as the important evaluation of filters in each layer. Adaptive threshold selection strategy decides the number of removed filters, which satisfies one of three adaptive pruning policies. Experimental results on CIFAR-10 and ImageNet with ResNet architectures show performance gains over several state-of-the-art methods. Strength: 
+ The proposed method is simple and easy to follow.
+ This paper proposes three modes for structured pruning, including accuracy-guaranteed adaptive pruning, memory-constrained adaptive pruning, and FLOPs-constrained adaptive pruning. 

Weakness:
- The contribution of the evaluation for the importance of filter, since activation-based attention maps are similar to attention score in [Sergey Zagoruyko, ICLR 16] and the activation-based score is also proposed in many related works, such as APoZ [1] and HRank. Please clarify the difference between these works and the proposed methods.
- The rationale of calculating the threshold T in each layer. T in layer-aware threshold adjustment is directly related to the percentage of each layer on parameters or FLOPs. Moreover, the typo in line 3 of algorithm N^i[r]->T^i[r].
- Lack of some important experimental evaluation: 1. The models selected to compress are all based resnet architectures, how about other architectures, especially light backbones (e.g. mobilenets); 2. The effect of attention score and pruning strategy should be discussed in ablation study; 3. The actual speedup of the pruned models should be evaluated if minimizing FLOPs.

[1] Network trimming: A data-driven neuron pruning approach towards efficient deep architectures, arXiv preprint arXiv:1607.03250, 2016.
 Although the proposed method is simple yet effective on ResNets, the contribution is limited and experimental evaluation is not comprehensive. I tend to reject this paper in this version.


-------------------POST-REBUTTAL COMMENTS-------------------

I thank the authors for the response and the efforts in the updated draft. After this rebuttal, the authors well answer my comments about the comparison to SOTA methods. However, I still believe that the proposed measurement of filter importance is a limited novelty, as it is a simple revision of attention score from [Sergey Zagoruyko, ICLR 16]. Moreover, the rationale of threshold T is based on the assumption that a layer is more likely to have redundant filters to prune if it contains more remaining parameters. I think it is not a correct assumption, as the entire filters of a layer with a smaller number of parameters may be redundant to be removed safely, especially for ResNets. Thus, I still keep my original rate to reject it.

This paper proposed an activation-based, adaptive threshold, iterative structured pruning method, combining several existing techniques together to perform a comprehensive pruning.  Strengths:
The proposed method has relatively good results on cifar10 and imagenet.  The method can automatically meet users' several requirements generally, like accuracy, latency, memory, and so on. 

Weakness:
1. Each part of the technique is not that novel. It is like a combination of several existing tricks to perform comprehensive pruning, not enough analysis of the convergence or correctness. For example, how to guarantee the convergence of algorithm 3? 
2. The comparisons on Imagenet are too weak. There are many SOTA pruning methods on Imagenet. The paper only lists four of them. I suggest comparing the paper with more recent SOTA pruning papers and comparing on more benchmark models besides Res50. 
for example, DMCP: Differentiable Markov Channel Pruning for Neural Networks
eagleeye: fast sub-net evaluation for efficient neural network pruning. 
gdp: neural network pruning via gates with differentiable polarization.
and so on. 

 Due to the concerns on the novelty and the weak comparisons on Imagenet. I temporarily think this paper is marginally below the acceptance threshold. 

This work proposes a technique for iterative structured pruning, without necessarily requiring too much manual human intervention. There are two parts to this paper that are important:

1. It is argued that we should prune channels based on the activation maps generated, rather than focusing on the weights of the channel.
2. They propose an iterative procedure that automatically backtracks if it has made a poor pruning decision. I did not like the way that this paper is written. It is extremely waffley, and is not upfront about what is going on. It is not completely clear (to me at least) what the overall strategy for pruning will be in this work until half way through page 4 (!). The paper needs to be (completely) rewritten to match the expected format in the computer science literature.

**Strengths**
- I am not a total expert on the pruning literature, but I do find the idea of pruning based on the activation maps rather than the weights intuitive and sensible.
- The results are compelling relative to other works they compare to. NOTE: again, I will not claim to be an absolute expert on the state of the pruning literature.
- The automated approach is relatively simple, and far more interpretable than approaches such as AMC. This matters a lot in many real-world applications.
- ImageNet results are included, which is relatively rare for pruning papers.

**Weaknesses**:
- The writing of this paper is unacceptable, as I mentioned earlier.
- A few really important ablation studies are missing. Firstly, there is no direct reason to couple the pruning technique, with the method for iteratively pruning; can these not be compared separately. It is misleading to conflate the two. Secondly, there is no comment on runtime for this method, which is important if this is an automated technique.
- Another important issue is that the FLOPs reduction doesn't seem that impressive. Works like Eigendamage (Wang et al.) achieve far more impressive reductions in FLOPs, which is arguably far more important than parameters in the real world. As an aside, would you be able to compare to Eigendamage?
- The work only focuses on ResNets. I know that they're harder to prune, but I'd also like to see some experiments on VGG models, for example. I think this work has some merit, but it is not ready for publication at this time. I can be swayed at rebuttal time if there is convincing new experimental evidence.

Meta-review: The paper proposes a strategy for incrementally pruning deep learning models based on activation values. The approach can satisfy different kinds of requirements, trading off between accuracy and sparsity.

The approach seems promising and seems to have competitive performance. However, the method is described by reviewers as a combination of ideas that have been proposed in the literature, and the experimental evaluation relies too much on a dataset considered too small to be reliable in such experiments --- CIFAR10. We do not expect substantial experiments within the rebuttal period: such comparisons with relevant SOTA methods should have been present in the submission. Moreover, the strategy proposed for selecting a threshold seems to rely on some doubtful assumptions, and there are no benchmarks on actual runtime.

The writing has improved based on reviewer input, and the reviewers are satisfied with this aspect. I would still add that I would prefer some clarity in the method presentation: is there a quantity being optimized? is there a value we can monitor to ensure our reimplementation is correct? etc. In addition I would like to ask authors in the next revision to be mindful to the difference between `\citet` and `\citep` in author-year citations -- see e.g. the first two ones in 3.1.

Decision: Reject