Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a new pruning scenario using regularization to better prune the network. The scenario has two-component, the first one proposes a new pruning schedule that does not directly remove the neurons that need to prune from the network. It removes the neurons by adding an L2 regularization and makes the neurons that need to remove gradually decrease to zero. The second one gives the importance score to the neurons. It uses the L2 regularization and studies how the coefficient \lambda of the regularization term can influence the weight change to derive the neuron's importance in the neuron network. By perturbing the penalty term to the converged network, the algorithm can get the Hessian information to score the neurons but uses less time than calculating the Hessian. The paper also shows many empirical results on various benchmarks to show their advantages when using the new schedule and scoring criterion during the pruning process. The result shows that their method can get better at a fast speed.

Pros:
I think the paper is interesting, both two steps make sense to me.  For the importance criterion statement, the paper also gives the theoretical analysis. Plus, the paper in good writing and easy to understand.

Cons:
Currently, GReg-1 is based on L1-norm pruning. It would be interesting if the author can show the more empirical result of applying GReg-1 on other pruning methods that directly removes the neurons. It can better support your statement that the pruning schedule is important.


===============================After response ======================
Thanks for stressing my concern, the additional experiment makes the empirical result more convincing. Overall I think this is a good paper, I prefer to keep my score and rate it as accept.
review_title: Official Blind Review #1

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper deals with regularisation based model compression for DNNs focusing on filter pruning. In particular they focus on growing regularisation during training instead of using a high regularisation parameter at the beginning. Growing regularisation for unimportant weights and negative penalty for important weights help them in improving the expressive power of the important weights.

+ve

- The paper is easy to follow and the authors make it quite clear about the problems related to filter pruning that they are tackling in this work.

- The experiments conducted are quite elaborate and the authors have provided detailed description about the experimental setup, hyper-parameters tuning, and training strategy.

Concerns

- The core idea of growing the regularisation parameter is not quite novel. The paper “Structured pruning for efficient convnets via incremental regularization“ by Wang et. al. also uses different learning rates for different set of parameters as well as they also grow the regularisation during training.

- In table 1, it would good to see a competitive method other than one-shot pruning because it is well established (this paper also mentions it correctly) that pruning a large number of filters in one-shot usually results in significant drop in accuracy. Comparing with iterative pruning methods like “Importance estimation for neural network pruning” by Molchanov et.al. would make the comparison fair especially for higher pruning percentages.

- While comparing GReg1 & GReg2 with other methods in Table 2&3, it would be good to see comparison with other regularisation based methods like “Structured pruning for efficient convnets via incremental regularization“ by Wang et. al. 

- In Table 2, the authors should include comparison with Ding et.al. for VGG as well. It seems that for ResNet56 the performance of GReg-1 and GReg-2 is quite comparable to Ding et.al.  
- The paper needs thorough proof reading as there are a lot of grammatical mistakes.

- Lastly, the style files used by the authors doesn’t match with the ICLR2021 style files. 
review_title: Official Review

final_score: rating: 7: Good paper, accept
review_content: This paper explores how the basic L2 regularization can be exploited in a growing fashion for better deep network pruning. The authors proposed two algorithms in this work: (1) The first (called GReg-1) is a variant of the L1-norm based filter pruning method [Ref1]. The important/unimportant filters are decided by their L1-norms. Later the unimportant ones are forced to zero through the proposed rising penalty scheme. (2) The second algortihm (called Greg-2) imposes the rising L2 regularization on all the filters. It is theoretically shown in the paper that this makes the parameters to separate to different degrees according to their local curvatures (ie, Hessian values). The method takes advantage of this by driving the weights into two groups with stark magnitude difference and then prunes by the simple L1-norm criterion.  The two methods are demonstrated effective on CIFAR10/100 and ImageNet benchmarks in the comparison with many state-of-the-art methods. 

Significance:
The paper focuses on deep neural network pruning, which is of interest to both the academia and industry. The empirical results look significant when compared with related works.

Clarity and quality:
The writing of paper is clear. Important formulas are proved. Related works are properly discussed.

Originality:
The growing regularization method looks novel to me. It further consists of two branches. The first one is simple but the results are inspiring. The second one has a good theoretical basis.

Pros:
The regularization process introduced in GReg-1 allows the network to adapt and recover before some weights are eventually removed. This idea is easy to understand and the investigation of the effect of regularization pruning schedule is insteresting. It is quite surprising that when pruning the same weights, regularization schedule can be quite better than the one-shot manner.

The proposed strategy and its theory basis of GReg-2 seem novel to me. It presents a new way to exploit the Hessian information like no other works before. Although the analyses are two simplified cases, the intuition is straight and also verified empirically on modern deep networks.

The empirical study looks strong on CIFAR10/100 and the large scale ImageNet benchmark. 

In addition to structured pruning (filter pruning), they also evaluate their methods in the unstructured pruning case and also achieve good performance compared with the recent pruning paper based on advanced Hessian-approximation.

Cons:
The two methods work pretty well on ImageNet from table 3 and 4, yet one small concern is they are outperformed by C-SGD on CIFAR10 in table 2. Is there more explanation and analysis for this?

Can the proposed methods be generalized to other regularization forms like L1 and L0 [Ref2]? This should be discussed properly in the paper.

Other questions:
How the layer-wise pruning ratios are decided in table 3 and 4 for the proposed methods?

For the Greg-1, it shows pruning the same weights with different schedule schemes can lead to quite different results. One concurrent paper [Ref3] in this venue has a similar finding (different pruning methods with different performances turn out pruning very similar weights). How is your work related with theirs?

[Ref1] Pruning filters for efficient convnets. In ICLR, 2017.
[Ref2] Learning sparse neural networks through l0 regularization. In ICLR, 2018¬¬¬
[Ref3] Rethinking the Pruning Criteria for Convolutional Neural Network, Anonymous, ICLR 2021 submission: https://openreview.net/forum?id=ZD7Ll4pAw7C

In all, I think it’s a good paper and rate as ‘accept’. If most of my concerns and questions are addressed in the rebuttal, I would like to upgrade my rating score.
review_title: Interesting investigation; good performance 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The authors propose regularization-based pruning methods with the penalty factors uniformly increased over the training session. The first algorithm (GReg-1) sorts the filters by L1-norm and only applies the increasing regularization to the “unimportant” filters; the second one (GReg-2) applies the increasing regularization to all the filters. The experiments are very extensive and convincing to support the claimed contributions.

Strengths:
1.The idea of utilizing Hessian without knowing its values is interesting to me. Theoretical analysis in Sec. 3.3 looks sound, which should be the main theoretical contribution of this work.

2.Empirical performances are promising, especially the ImageNet one in Tab. 3 and 4. 

3.Good performance. On CIFAR-10 and ImageNet, they evaluate the proposed methods with popular deep neural networks, reporting encouraging performances.

4.The methods are easy to implement with current deep learning tools and can work in both the structure pruning and unstructured pruning cases, which is a plus. 

5.The authors present theory analyses to show this will cause the filters to separate owing to their different underlying Hessian structure, thus achieving “exploiting the Hessian information without knowing their specific values”. 

6.Generally, this paper is well-written with sound proofs for their formulas, although there are some small problems (see the weakness below) that the authors may want to resolve.

Weakness:
1.In Sec. 3.3, it says h11 > h22 leads to r1 > r2, where r is defined by the new magnitude over the old one. But in Fig. 1, the plot is the normalized magnitude stddev. How these two are related to each other is not so straight to me. Could the authors give more explanation about this since it is said in the paper that Fig.1 is an “empirical validation” for the analysis in Sec. 3.3?

2.Some typos and small glitches: 
“importance-based one focus more” -> focuses
Appendix: In the last sentence of A.1 “our results (Tab. R3 and R4)” (I didn’t find R3, R4). Please clarify it.
Appendix: Footnote in Tab. 6, the references of [7] [54] seem pointing to nowhere.

===============================After response ====================== The authors have addressed all my concerns.  I would like to keep my initial score.
review_title: A nice idea for neural pruning with extensive experiments

Meta-review: This paper introduces a novel pruning algorithm for neural networks, gently regularizing the weights away (through weight decay) and using Hessian information instead of simple magnitude. All in all an idea that is simple and effective, and could be of interest to a large audience. 

AC
Decision: Accept (Poster)