Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a structured pruning method which include correlations among structures by second order information. The author also improved the computational efficiency by fast Hessian-vector products. The method is demonstrated to achieve comparable/better results than previous methods. And it can also be applied to pruning-at-initialization and reveal architecture bottlenecks.
main_review: Strength:
1. The method is well-motivated. The author provided thorough derivations to explain the method.
2. The method can be applied both after-training and at-initialization.
3. The application of identifying architecture bottleneck and correspondingly further improve the architecture is interesting.

Weakness:
1. In Table 1, it seems that the method only achieves comparable results with previous methods. In some cases it is better but the improvements are also very marginal.
2. The compared methods didn't seem to be the most recent ones. Most of them are in 2019 and before. Could the author conduct some comparison with the most recent state-of-the-art?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work proposes a second-order structured pruning algorithms (SOSP), which can drastically reduce training time and further improve pruned network performance by removing architecture bottlenecks
main_review: This work is well-explained and easy to follow. This paper has very detailed background overview on the state-of-the-art pruning techniques and recent findings. This paper presents proposed method in good detail and complexity analysis. Extensive experiments on CIFARs with multiple architectures effectiveness of the proposed approach in pruning and further conducted experiments demonstrated its effectiveness in removing architectural bottlenecks.

limitations_and_societal_impact: This paper does discuss its limitations.

Questions and suggestions:
 
What would be the scale limits to choose SOSP-I over SOSP-H? Can we have more experiment in performance comparison between those two as network scale increases?
 
 
It would be interesting to compare with more recent pruning works such as  [1,2].
 
[1] AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates.
[2] EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This work proposes a structured pruning method that removes feature maps (channels) of convolutional neural networks without losing accuracy while increasing computational efficiency (MACs). The proposed method (SOSP-I/H) is based on a sensitivity based saliency and involves what they call global correlations through the second order partial derivatives with respect to different structures. SOSP shares a lot in common with previous works including OBD and C-OBD for the use of second order information except that they approximately incorporate all elements in the Hessian matrix. Then the paper evaluates the effectiveness of the proposed method for image classification task using VGG, ResNet, and Desnet on CIFAR and ImageNet datasets and achieve accuracies on par with previous works on both global and local (layerwise) pruning. The paper further presents that SOSP works on pruning at initialization and can improve further by performing expand-pruning based on skewed distribution of retained parameters over layers.

main_review: Using second order information to locally approximate the changes in the loss function can be useful, and in fact it has been used in many prior works as mentioned in the paper.
As it is quite expensive to compute the entire Hessian accurately for a deep network with a large number of parameters, the paper proposes to approximate based on (6) which is using the Hessian vector products.
Here, they make a rather strong assumption on the wide network referring to the NTK regime and the local first order approximation (linearization) of the network computation.
The paper attempts to demonstrate the effectiveness of the proposed method on large-scale experiments using ImageNet.

limitations_and_societal_impact: - The paper often misses clear explanations of why their approximation should work as intended. For instance, from (1) to (2) they take absolute values individually by saying that it is to avoid cancellations (L95-L98) however they clearly measure saliency differently from each other, and presumably the resulting pruned structures will most likely look very different.
- The method is not evaluated clearly how efficient the proposed algorithm is compared to other approaches; the computational cost for a high pruning ratio seems to be estimated not quite tight or realistic.
- A clear limitation of SOSP is that the method does not compare very strongly or improves over other existing prior works. Only in a few cases of global pruning seems to improve very marginally.
- The evaluations and analysis performed seem a bit insufficient to support claims such as the argument that SOSP at initialization works on par with SOSP after training while performing training twice and claim against other pruning approaches at initialization.
- Generalizing and automating expand-pruning rather than widening by a factor of two would make the process of finding bottleneck appeal stronger.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents two novel approximation techniques for approximating the importance of a filter (the estimated loss difference between pruned and pre-trained networks) using second order approximations. The main difficulties of second order approximation lie in the computational burden of computing the full Hessian matrix. To circumvent this issue, this paper presents two novel ways to do so. The first approach, dubbed SOSP-I, linearize the neural network and only consider the coupling of the loss function. The second approach, dubbed SOSP-H, approximate the loss difference using a Hessian-vector product. Both approaches have much better efficiency compared to using the Hessian and empirically perform a little better than existing approaches. Overall, the approximations seem novel to me and I find the appendix helpful in understanding the derivations. However, their empirical advantages seem rather limited. Additionally, I think some of the related papers are missed and they need to be discussed and compared against.
main_review: ## Strengths

- Novel approaches for approximating the Hessian for filter pruning
- The manuscript is well-written and easy to follow
- Filter pruning is a practically relevant research direction
- Comparisons with EigenDamage, NN-Slimming, and C-OBD are comprehensive for CIFAR

## Weaknesses

- The improvement over direct competitors (EigenDamage, NN-Slimming, and C-OBD) is relatively limited. Specifically, SOSP-H achieves 0.4% improvements over EigenDamage with similar MAC reduction for VGG on CIFAR-10 while being worse on CIFAR-100. Compared to EigenDamage, it seems only ResNet32 for CIFAR-100 works well. Not sure how to interpret this and an immediate question from the readers would be: When should we use EigenDamage and when should we use the proposed method? While I appreciate the results on ImageNet, direct competitors are not shown on ImageNet. More specifically, I do not think it is informative to compare local pruning methods with global pruning methods as the resulting pruned architectures are very important. It would be great if global pruning methods (e.g., EigenDamage, NN-Slimming, and C-OBD) are shown in the ImageNet table.

- Missing global pruning methods. Specifically, both [1,2] approximate the loss differences using Taylor expansion as well. Although missing second order information, I think it is not clear how much second order approximation adds to the picture if these first-order methods are not compared. Additionally, as noted by the authors in line 215, different layers contribute to MAC differently while the approximation only considers loss but not MAC. Hence, I think it is also critical to compared to learned global ranking such as [3] which factors the heterogeneous MAC into the ranking learning process.

- Missing comparisons with local pruning methods that have learned layerwise pruning ratio [4].

- Missing analysis on compact networks such as MobileNets and EfficientNets. Accelerating networks is important, but if the proposed method does not perform well on already fast/small networks, it is less significant.

[1] Theis, Lucas, et al. "Faster gaze prediction with dense networks and fisher pruning." arXiv preprint arXiv:1801.05787 (2018).

[2] Molchanov, Pavlo, et al. "Importance estimation for neural network pruning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

[3] Chin, Ting-Wu, et al. "Towards efficient model compression via learned global ranking." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[4] He, Yihui, et al. "Amc: Automl for model compression and acceleration on mobile devices." Proceedings of the European conference on computer vision (ECCV). 2018.
limitations_and_societal_impact: I do not think this paper has negative societal impacts
review_title: null

Meta-review: This paper proposes a pruning technique that exploits second order information. In order to circumvent the high computational cost, an approximate computation is proposed. The improvements brought by the presented method are mostly marginal, and the method is not really compared to the most recent state of the art. Also, some of the critical parts of the paper lack clarity, for instance when motivating the approximation scheme. Overall, I recommend the rejection of this paper and encourage the authors to take the points raised by the reviewers into account when re-working the manuscript and preparing a submission to another venue. 
Decision: Reject