Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method to prune deep neural networks. The aim is to reduce the computational cost and inference time while maximally maintaining the classification performance. The motivation of this work is to consider second-order structured pruning (SOSP), which considers the correlation information among the structures and layers when conducting network pruning. The key part of this work is the development of a method called SOSP-H that can have better scalability while considering the second-order correlation information for pruning. Experimental study is conducted to compare the proposed SOSP-H with its variant and other existing related methods, demonstrating its effectiveness.
main_review: On the strengths

1. Network pruning is important for deep learning approach and its wide application to practical tasks.
2. It is sound to consider the second-order correlation information among the structures of neural networks to conduct pruning. 
3. Experimental study demonstrates the overall better performance of the proposed SOSP-H with respect to the relevant methods proposed in the literature;
4. This papers provides detailed discussion and supplementary materials to explain the proposed work. 

On the weaknesses
1. Some parts on proposing SOSP can be made clearer. For example, it mentions Eq. (2) as "a modification of Eq.(1)." However, the relationship between Eq.(1)  and Eq.(2) is not clearly discussed. It seems that Eq.(2) is a (much) relaxed upper bound of Eq.(1)? In this case, how accuracy can Eq.(2) approximate Eq.(1)? The similar issue applies to the relationship between Eq. (6) to Eq.(1). Please clarify. 
2. The logic behind deciding the architectural bottleneck could be made more precise. Say, if some layers are barely pruned, does this actually mean they are so important that they shall not be removed? In this case, how to arrive at the idea in this work that "removing these could further improve the performance of the model?” Please clarify.  
3. This paper provides detailed discussion in Section 4, which is appreciated. Meanwhile, the summary of this discussion could be put before  Section 2 as Related Work. This will help to introduce the existing literature on second-order information based pruning and highlight the differences of the proposed work from the existing ones. This helps to better capture the novelty of this work.   

summary_of_the_review: This is a piece of work of good quality, with detailed information. A rigorous and principled approach is taken by this work to address the second-order pruning issue. Experimental study demonstrates the overall competitive performance of the proposed method. At the same time, some parts of this work can be further clarified to make the argument more precise and clearer. The novelty and contribution with respect to the existing relevant methods can be better described. Also, the structure of the presentation can be improved a bit.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper address the problem of neural network single-shot structured-based model pruning. 

Deep convolutional neural networks grow to achieve higher performance, which also means slower inference and higher computational cost. Model pruning can help with that. Model pruning can be unstructured, which means to remove individual weights, or structured, which means to remove entire substructures, e.g., nodes or channels. This paper focuses on structured pruning.

A key challenge for global saliency-based structured model pruning is to find a good objective that can be efficiently calculated to make the approach scalable to various modern convolutional neural networks. Existing saliency-based pruning methods such as OBD, C-OBD evaluate the effect of removing a single weight or structure on the loss of the neural network in isolation. This work is to devise a 2nd order pruning method that considers all global correlations for structured sensitivity pruning.

The basic idea in this work is to search for the pruning mask M to minimize the joint effect on the network loss approximately. The mathematical approximation focuses on the second-order approximation to the loss. 

The key contributions of this work are the proposed 2nd-order structured pruning (SOSP). There are two variants of that. The first one is based on fast hessian-vector products, and it has the same complexity as that with first-order methods. A second one is based on the Gaussian-newton approximation. The first one with fast hessian-vector products do better in terms of scale.

Experiments on VGG, ResNets, PlainNet, DenseNet are shown promising results across various image classification datasets, including Cifar10/100 and ImageNet.

main_review: Strengths

1. This paper makes a good contribution in presenting the idea around structured-based model pruning. The mathematical derivation is very well presented and easy to follow, and the literature review is also very well presented.

2. The experiments are also promising, and they show the superior performance of the proposed approach in terms of effectiveness in reducing the parameters while keeping the classification performance.

Weakness
1. The second-order approximation seems to be specialized for squared loss and cross-entropy loss. I am not sure if the proposed approximation can be generalized to the other losses. This limits the impact of the proposed approach, as modern neural networks might involve many other forms of losses.e.g. MSELoss, NLLLoss, CTCLoss, PoissonNLLLoss. How can one adopt the proposed approximation for the wide range of loss functions?

2. Some key implementation details are missing out in the paper.

a) The fast hessian-vector product seems to be the critical component of the proposed SOSP-H method. However, it is not clear how this is exactly implemented. It is not clear how the fast hessian-vector product is implemented to achieve the second-order approximation.
Using the hessian seems to be a great idea, but I am unsure how the proposed approach addresses the storage problems. For example, storing the hessian matrix for a standard ResNet50 would take at least 2.5 Petabytes of memory. For larger-scale neural networks, how would the proposed approach be able to address them? What are the theoretical limits on the proposed method that the SOSP-H cannot handle? Given that the SOSP-I results are not as great as those with SOSP-H, I think audiences might be more interested in the SOSP-H. But the high space cost seems to be a problem.

b) The presentation arrangement of this paper could be improved significantly. Some technical terms are presented without explanation. For example, what does it mean “individual sensitives”?

3. Some comparisons are missing out. How does the proposed approximation approach compare against the empirical Fisher approximation? I only get to know more about the mathematical details of this after reading the supplementary, it would be great to add some key approximation equations to the main paper.

4. The main evaluation metrics used in this work are MAC numbers and the model Top-1 accuracy. It would be interesting to see the speed of the pruned networks on the standard devices, though this is missing in the current paper.
summary_of_the_review: 1. This paper is an exciting work, and it would have a solid impact on the structured-based model pruning.

2. This submission also has a lot of minor issues. For example, the second-order approximation seems to be developed towards addressing particular two losses used in classification. It is not clear if the proposed approach would work for the other losses.

3. Some technical terms are not well explained.

4. The evaluation metrics used in this work should also include speed, but it is missing in the current submission.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents saliency-based second-order structured pruning methods, namely SOSP-I and SOSP-H. The proposed methods are designed to capture the correlations among all structures and layers, known as second-order structure (Hessian). In particular, SOSP-I employs Hessian approximation while SOSP-H employs exact Hessian. Overall, the idea of this paper is very clear, and I kinda like the discussion part of it. The expand-pruning is also interesting. 
main_review: 
For the SOSP-H, it assumes the pruning ratios should be high and then the approximation would be held. In Tab.1 and Tab.2, I can see that the reduction ratios of weights for Densenet40 on Cifar10/100 are around 50% and similarly for ResNet 18/50 on ImageNet. So I suggest the authors clarify what ratios are considered to be high.

In Tab.2, when comparing with the other methods, the performance of the proposed methods is on par with the SOTA. Whether the pruning efficiency will outperform the other methods. I can see in Fig. 1 the authors demonstrated results on Cifar, but how about on ImageNet?

I am not an expert on network pruning. However, I am wondering why the compared methods are from 2019 and 2020. Whether some references published in 2021, such as Liu et al. 2021, Hayou et al. 2021, Su et al. 2021, can be compared?

In Fig. 3, the orange bars are not clear to me. The vertical axes are representing different meanings that are not easy for readers to understand. For c and f, whether the pruning mainly focuses on input and low-level layers? Why did those layers have low pruning ratios originally? Maybe the authors could explain it and visualise filters before and after pruning?

I think the comparison with CCP is also needed in Tab. 1 as it is also based on second-order correlations.

Last but not least, when I recap the formulation of this paper, from Eq. 1 to 6, I also wonder if in general eq. 6 is the main contribution of this paper while eq.5 is not scalable (though it has very nice deduction). Whether eq. 5 is not necessary for practical problems?
summary_of_the_review: Overall, I think this paper has its merits. However, as I am not an expert in this field but a general reader. I am lean to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops two novel saliency-based pruning methods for second-order structured pruning which consider correlations among all structures and layers and have good scalability to large-scale neural network structures and datasets. The proposed method can also be used to find and remove architectural bottlenecks, which further improves the performance of the pruned networks.
main_review: Strengths(+):
1. The proposed approximation schemes for second-order structured pruning is interesting with theoretical derivations and rigorous experimental test. 
2. The paper is well written and the presentation is clear.

Weaknesses(-):
1. I am concerned on the advantages of using second-order information to improve the pruning performance for networks with large number of parameters, as can be seen from Fig. 4, the performance gap between first-order method and SOSP is quite small when #parameters reach 750K. Modern large-scale networks usually have even more parameters. 
2. The Fig.5 also raises some concerns since the gap on test accuracy between first-order and SOSP is only around 0.1%~0.3% percent, which has quite limited improvement. 
3. It seems that the merit of capturing global correlation and good scalability is not achieved by a single approximation method, but by the SOSP-I and SOSP-H respectively. This may decrease the technical attractiveness of the paper.

Minors:
1. Maybe due to space limitation, some important Figures are put into supplementary materials, the readers need to read back and forth to get a clear understanding.
summary_of_the_review: Due to the limited potential performance improvement in practical benchmark database, my overall rating would be (weak) rejection
review_title: null

Meta-review: Four reviewers have evaluated this submission with one score 6 and three scores 8. Overall, reviewers like the work and note that *a rigorous and principled approach is taken by this work*. AC agrees and advocates an accept.
Decision: Accept (Spotlight)