Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This submission proposes a way to prune neural networks using a continuous penalty function. 

== Pros ==

The experiments include some interesting illustrations beyond the basic accuracy comparisons. This includes illustrations of how weights are distributed between the layers (Fig 5), ablations on different components of the penalty function, and experiments on the transferability of the selected sparsity pattern.

There is a very complete description of the training procedure. Writing overall is clear. Code is included with submission. The code is readable and easy to get running.

== Cons ==

Many previous works propose a new sparsity penalty function, and associated optimization, for neural network training. As well as (Lemaire et al) and (Louizos et al) cited in the submission, there's is works such as:

  * Srinivas et al. "Training Sparse Neural Networks." CVPR 2017.
  * Manessi et al. "Automated Pruning for Deep Neural Network Compression." ICPR 2018.
  * Zhu et. al. "Improving Deep Neural Network Sparsity through Decorrelation Regularization." IJCAI 2018.
  * Yang et. al. "DeepHoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures." ICLR 2020

Within this general class of approaches, though, the specifics of the penalty function and optimization in this paper are novel. Whether that is a valuable novelty depends entirely on the empirical results: since the higher-level idea is well-explored this adds to the literature iff the submission has made the best design choices among these other methods.

Despite their overall heft, the experiments have some missing pieces that make it hard to evaluate this. To start, the evaluation is on a less common (within the sparse neural network literature) choice of backbones for each dataset. See:

Blalock et. al. "What is the State of Neural Network Pruning?" Sections 3.3 & 4.2

Wider-ResNet would be expected to have greater redundancy than ordinary ResNet, perhaps making an easier problem for sparsifying. However, this choice is inherited from BAR.

Ordinary ResNet-50 is a more common choice, but less so on CIFAR-10. This choice differs from the corresponding experiments in the (Liu et al) work that is compared against, that uses VGGNet, DenseNet-40, and ResNet-164.

The code appears to have partial support for a much wider variety of models (see line 19 of pruning.py), though maybe not all are actually supported: for instance r32 and r152 don't seem to be included in the branches in models/resnet.py lines 421-428.

And generally there are few comparisons made. This is possibly unavoidable due to the overall fragmentation of metrics. Choices of comparisons are the more closely similar papers in the literature (this is good), but lacks more some of the simple baselines (that can often do surprisingly well):

Gale et. al. "The State of Sparsity in Deep Neural Networks"

I don't see experiments that show proposed optimization has consistent convergence across runs. Experiments dealing with "stability" are either highlighted single comparisons, or look at stability w.r.t. hyperparameters. Some of the reported accuracy differences in comparison are quite small, especially at higher levels of sparsity, so seeing the standard deviation of multiple runs of each method compared might be especially important.

== After Rebuttal ==

I had previously missed that Lemaire et. al. had some of the key comparisons (simple baselines done for the same architecture/dataset pairs) that are necessary to judge this method. I thank the authors for pointing this out. It is also helpful that these baselines are now included for completeness.

C.6 still seems like an indirect measure of optimization stability, but it is reasonable to conclude from it that the differences from BAR are real, given the very low standard deviation in accuracy.

I remain on the side that this is still a relatively incremental addition to the pruning literature, but I am now satisfied that it is well-validated.
review_title: Incremental change to pruning methods with limited validation

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a new deterministic pruning strategy that employs continuous Heaviside function and crispness loss to identify a sparse network out of an existing dense network. Experiments show its effectiveness and robustness. Generally, it is well-written and easy to follow. Some minor issues are shown below.

1.	Pseudo codes for the functions mentioned in Algorithm 1 should be provided to show them clearly. 
2.	The proposed algorithm is a deterministic algorithm and may fail in complex network pruning problems. An in-depth analysis of its limitation is needed.
3.	It is recommended to use different labels for different algorithms in the figures.


review_title: ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations

final_score: rating: 7: Good paper, accept
review_content: Summary

This paper presents a new method for structure pruning called ChipNet. The ChipNet employs continuous Heaviside function with commonly used logistic curve and crispness loss to estimate sparsity masks. A combination of above three components is helpful to obtain approximately discrete solutions for a continuous optimization scheme.  As a result, it is possible to get a highly sparse network out of an existing pre-trained dense network. Experimentally, ChipNet outperforms other previous structured pruning methods by a large margin. 

Overall,  I don't think this paper has any critical drawbacks, and there are only a few comments. Therefore, I recommend accepting this paper.

Strength

The motivation for this paper is clear and quite interesting. Also, the performance of the proposed pruning method is much better than previous baseline methods, and experiments were performed in various ways to show the effectiveness of it. 



Comments and Weakness

I have not much to criticize this paper, but I have some questions and comments as follows.

Is it possible to get a similar result by applying the proposed method to the originally lightweight network such as Mobile Net?

Are there comparisons with NAS-based methods?

Is the proposed method applicable to networks for other tasks such as detection and segmentation?


There is a submitted paper (to this ICLR) that includes continuous relaxation of discrete network structure optimization for network growing (not pruning). 

“Growing Efficient Deep Networks by Structured Continuous Sparsification”

The direction of the paper is totally different, but there seems to be a similar part in terms of continuous relaxation. It would be good to mention the difference with this paper w.r.t. continuous relaxation.

Also, I would like that more related works are mentioned in recent techniques using continuous relaxation(approximation).

review_title: Review on Paper1752 

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################


Summary:

The paper provides a budget-aware regularizer method to train the network to be pruned. Existing methods based-on the regularizer method suffer from satisfying the user-specified constraints and resort to trial-and-error approach. The authors leverage logistic function and continuous heaviside function for continuous relaxation of the discrete variable to select the channels to be pruned.  There are four types of budget constraints (channel, volume, parameter, FLOPs).  
The experiment results on several different budget constraints (even in extreme condition) are quite promising. Also, these methods can be applied in transfer learning (transferrable network structure).

##########################################################################


Reasons for score:

Overall, I vote for marginal acceptance. I like the budget-aware constraints concept and its tightness which is shown in figure 3.
However, I have several concerns about the papers (see cons) and authors need to consider these concerns in the rebuttal period. 


##########################################################################

1. The budget-aware constraint is explicitly introduced and easy to implement.
 
2. Ablation study in figure 5 provides insights into the network channel pruning.

##########################################################################


Cons: 
 
(1) Experiment results about the other several pruning experiments are omitted. Therefore, I concern that the other readers are confused about the performance of the proposed method over the other pruning methods. Particularly, I wish to check the performance with [1].  Also, Experiments on the ImageNet (large dataset experiments) should be added for future research.

(2) Details on the forward steps are omitted. (See 'Forward(x,W,z)' in Algorithm 1) Readers should have prior knowledge to guess the details on the forwarding steps.

(3) Justification of the usage of two functions (logistic and heaviside function) is not enough. Two separate functions for continuous approximations are quite confusing. For example, '(Logistic(\psi)+1)/2' can be used for continuous relaxation or 'Gumbel trick' can be used. 
 
(4) This might be the most critical one. Since the budget constraint is given based on the l2 loss with the target budget, '\alpha2' should be tuned like the other regularizer based method. From a critical perspective,  there is no difference with the other regularized-based method in the sense that the trial-and-error on the tuning '\alpha2' is stilled required and very large '\alpha2' could degrade the performance of the network.


=======================================================================================================

After rebuttal : 

Sorry for the delay. The authors address my concerns and reflect them on the revised version. I'm keeping my rating.


[1] DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures, ICLR 2020.
review_title: Official review3 : interesting approach but required clear justifications and experiments

Meta-review:  This paper proposed a new method to prune neural networks using a continuous penalty function. All reviewers suggest acceptance (some are on borderline though) as the authors did a good job in the rebuttal phase. AC also could not find any particular reason to reject the paper (in particular, the overall writing is clear) and thinks that this paper is a meaningful addition to ICLR 2021. 
Decision: Accept (Poster)