Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper first proposes the zero-invariant group for neural network. Zero-invariant group is the disjoint group of trainable parameters which results in corresponding output to be zero. Then, this paper formulates a optimization problem with group sparsity regularizer on the zero-invariant groups. Then, authors alternate SGD and Half-space projection to optimize the problem. This optimization method is called Half-Space Stochastic Projected Gradient (HSPG) which benefits from the larger projection region. This method does not require fine-tuning  and pruned model is directly used on the inference. On various dataset (Cifar-10 and ImageNet), this paper achieves state-of-the-art performance. 

main_review: Weakness
1.  I believe the novelty of the group sparsity regularizer on the zero-invariant group is not enough. The idea of a group sparsity regularizer has received attention for a long time. Please refer to [1], [2]. The only extension of this paper is to leverage the group sparsity regularizer on the zero-invariant group which includes the parameter in the batch normalization. Therefore, authors need to compare these related works and this paper, then show the superiority of the proposed method with the experiments.

2. The paper insists on the superiority of the HSPG method. However, there is no intuitive explanation for this. The only explanation is the larger projection space. But, readers might not agree that this point directly contributes to the performance of the pruning method. The authors need to provide the intuitive explanation that most of the readers and reviewers agree with. I suggest the authors reduce the ZIG section and add an ablation study and explanation from the appendix. 

3. Table1 is quite weird. It seems that the most of FLOPs reduction comes from the first channel reduction (64->3). There should be sufficient analysis and explanation for this.

4. The baselines on the experiment results are outdated and there are not enough baselines in Tables 2 and 3. Authors need to compare the proposed method with recent pruning papers. For example, GBN [3] achieves 75.18 with 50% FLOPS. 

Therefore, I conclude that this paper is not ready for submission. The novelty of the proposed method is hard to find and the superiority of the proposed method is hard to believe since there is no intuitive explanation and not enough experiment results.

[1] Learning Structured Sparsity in Deep Neural Networks, NeurIPS 16
[2] DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures, ICLR 20
[3] Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks, NeurIPS 20.


Post : 

I have read all the feedback. Some issues are resolved. However, still other critical issues are not fully resolved. Also, I could not agree with the novelty from ZIG and do not fully agree with importance of ZIG. Furthermore, I still could not fully understand the correlation between the large projection area and group sparsity. Furthermore, I could not see the updated experiment results of important baselines in the main table. Therefore, I keep my score.


Post Post: 

I have read authors' responses. Most of critical issues which I pointed are resolved. Please update the experiment results (after fine-tuning) + other baselines. I wish to see them in the final version. Some parts are still hard to understand (HSPG) and I'm not sure that claims on ZIG are impressive. However, clear experiment results support the novelty and importance of the proposed method. There is no reason to reject this paper. Therefore, I raise my score.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a group sparsity optimisation approach for training pruned models. This is done by splitting the optimisation into an initial phase where the non-differentiable regularisation objective is optimised using subgradients, followed by a second stage where channels/units (plus their corresponding bias/BN/residual path) are mapped to exact zero by augmenting the sub-gradients with an indicator function.  For each parameter group, this function compares the direction of subgradient with the direction you'd get as a result of pruning that group, and if the angle between the two is larger than 90 it prunes them away.
main_review: I think this is an interesting work and the results seem significant but I have a few concerns and suggestions before I can increase my score.

* I feel the way OTO and ZIG are presented as novel ideas is a bit misleading. In the list of contributions OTO is presented as a separate thing in addition to ZIG and HSPG, but it's essentially just the combination of two? The idea of pruning corresponding parameters attached to a channel/weight/unit is also not exactly novel and in some cases (for instance with residuals) is necessary to get shapes right. Similarly, framing this work as "one-shot" pruning is a little bit misleading. The sparsity-inducing objective is present in both initialisation and the projection steps for the entirety of the training so I'm not sure what's "one-shot" about this approach.
* At the beginning of Section 3.1, the authors claim that the _root cause_ that other pruning methods fail is the fact that saliency scores are computed for units/channels in isolation, and their corresponding bias/BN/residual terms are not included. I think this is a bold claim and is not supported by any evidence in the paper. Many pruning objectives are trivial to expand to include the corresponding parameters so if authors want to make this claim this claim I suggest they support it with some ablation experiments with some baselines where this change can be easily tested.
* Section 3.3:  
    * I found this section difficult to read and follow. Unless I missed something I felt this section explains the same idea three times with slightly different notations. The beginning of this section includes an "intuition" for doing HSPG but after reading that I was expecting to see the half-space _perpendicular_ to the direction of the gradient that takes you from $x_k$ to $x_{k+1}$. But instead, the half-spaced is based on the origin. Could the authors pehaps make this bit clearer? 
    * I was also a bit confused by Fig3a as I was expecting to see projections of $x_{k+1}$ on [x]1 and [x]2 axes?
    * Am I right in thinking that the optimization using subgradient stage doesn't actually stop, but is simply augmented with this additional projection? If yes I think the writing could be improved to include this fact early on.
    * if epsilon is non-zero in all of the experiments why is mentioned at all?
* Section 3.4
    * I'm not sure what's the point of this section. Surely by the end of the optimisation the zero groups have either emerged or not. So I'm not sure what's left to do and what authors mean by "full model" vs "pruned model". Are you talking about physically removing those rows/columns from weight/bias tensors?

Other questions and suggestions
* I'd be curious to see an ablation study to analyse the effect of initialization stage vs the projection stage. What's the split (in terms of the number of epochs) between the two and how did the authors choose these numbers? what is the performance of the neural network _before_ the projection phase?
* I don't find the ResNet-50 experiments with CIFAR-10 convincing. Why have you decided on two baselines (AMC and ANNC) that achieve such low pruning rates (and report no FLOPS numbers)? The fact that these two methods also use quantisation only muddies the waters furthermore. I suggest including some good old pruning baselines with similar pruning ratios.
* how do you control the sparsity target?
* using $x$ to denote parameters is kind of confusing. Perhaps use $\theta$?

limitations_and_societal_impact: NA
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a one-shot pruning strategy, in which the pruned subnetwork achieves good performance on several benchmarks without post procedures or iterations. They make use of the ZIG and propose HSPG to enhance the group sparsity exploration. Good performance across several benchmarks supports  the validity of OTO.


main_review: Strength

1. The pruning method no longer requires fine-tuning and achieves impressive results on several benchmarks, little performance drop with high compression ratio. 

2.  The compression experiments are designed for both vision  and NLP tasks. 

Weakness
1.  Missing one important baseline ResRep[1],  which can make this work weaker if included. Note that [1] is a lossless pruning method.

2.  The paper claims that it achieves SOTAs compression benchmark such as R50. However, [1] also prunes without fine-tuning. And the latter slims down a standard ResNet-50 with 76.15% accuracy on ImageNet to a narrower one with only 45% FLOPs.  Morever, ResRep[1] achives 75.3% under similar FLOPS reduction (OTO 74.7%). 
 

Others

1. It's interesting to see the performance of OTO on vision transformers such as ViT.

Reference

[1]Lossless CNN Channel Pruning via Decoupling Remembering and Forgetting, https://arxiv.org/abs/2007.03260 （ICCV21）

Post:

Thanks for the reply. My concern is not well addressed. And I have carefully read the reviews of other reviewers. 

I still feel that ignoring some strong baselines  can be misleading and greatly weaken the contribution of this paper. Therefore, I lower my score to below the acceptance threshold.
limitations_and_societal_impact: See the weakness.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a structured pruning method called Only-Train-Once (OTO). OTO first partitions the trainable parameters into zero-invariant groups called ZIGs. This way allows one to prune without affecting the output and therefore it requires no additional training steps after pruning. Once they define ZIGs, they perform Half-space Stochastic Projected Gradient (HSPG), starting from a pre-trained estimate using SGD, it performs a form of projected gradient descent while gradually increasing group sparsity. The authors demonstrate the effectiveness of OTO on CIFAR, ImageNet, SQuAD by showing an impressive gain in terms of FLOPs while maintaining the performance of resulting slim network to the original dense counterparts.

main_review: This work is motivated to remove the subsequent fine-tuning process after pruning (i.e., making some parameters zero-valued) that is often performed for various pruning approaches, in particular those in a similar fashion such as Group Lasso or any group sparsity inducing regularizer based methods. The idea of first creating zero invariant groups is interesting and new in the context of pruning as far as I know, and designing these groups for major components in modern neural networks is done in an easy and nice way. This can actually be very pragmatic in practice. Their actual pruning and training algorithm, HPGS, differs from rather aggressive filter pruning methods that work based on heuristics and lack theoretical guarantees on convergence, and combined with ZIGs, it seems that the idea is very effective as supported by experimental evidence. Overall, the paper is written with clarity and organized well.

limitations_and_societal_impact: - Can the process of creating ZIGs be made automatic? I feel there should be a generalized way to perform the process. Discussing this may make the manuscript stronger. 

- The idea of finishing the pruning step in one shot (and training only once) seems to be shared with other “single-shot” pruning methods either before such as SNIP by Lee et al. 2019, GraSP Wang et al. 2020, or any potential pruning method relying on a saliency criterion. While OTO is based on an optimization method with some convergence guarantee (I haven’t checked this in detail; is it applied for non-convex objectives?) and the above methods seem to rely on heuristics, crediting the former for algorithmic simplicity (L94: “OTO is simple”) over the others seems a bit stretched, considering that OTO requires additional step of creating ZIGs, performing a projected gradient descent, especially without any analysis on the computational cost of HSPG in comparison to others.

- Related to this, the paper defines pruning approaches to be those that mostly require fine-tuning (or re-training) and claims their contribution for removing it, and yet, this doesn’t seem to be a fair scope setting. Any sparsity inducing regularization based method (including those cited in this work such as [50]) can theoretically remove fine-tuning step.

- Discussing these properly in the paper seems necessary. Having said that, the title as well as the name of the method are not quite representative of the main idea.

- Any idea how to perform HSPG without ZIGs? Would it basically reduce to [58]?

- How do you control the level of sparsity (set hyperparameters in general) found by HSPG?

- Any analysis on the cost of HSPG and its comparison to other methods?
review_title: null

Meta-review: The authors propose a fine-tuning free structured pruning method (OTO). The idea is to first partition the parameters into zero-invariant groups, pruning the zero groups, and solving for a structured sparsity optimization problem with projections. The experiment results on CIFAR10, ImageNet, and SQuAD show the competitive performance on FLOPs and number of parameters reduction.

The paper initially received a split review. I would like to thank the time and effort the authors and reviewers spent engaging in the active discussion during the rebuttal phase.

I strongly recommend the authors to revise the submission and include the ablation study and the baseline comparisons the reviewers requested.

Specifically, please include

1. The ablation study of initialization stage vs projection stage

2. Baseline comparison with related methods [1,2,3,4,5].

[1] Learning Structured Sparsity in Deep Neural Networks, NeurIPS 16 
[2] DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures, ICLR 20
[3] Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks, NeurIPS 20.
[4] Operation-Aware Soft Channel Pruning using Differentiable Masks, ICML2020
[5] Lossless CNN Channel Pruning via Decoupling Remembering and Forgetting. ICCV 2021.


Decision: Accept (Poster)