Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
 
In this paper, the author propose a provable pruning method, and also provide a bound for the final pruning error.  Among most heuristics prune method,  pruning with mathematics guarantee is indeed more convincing. We expect this work can help people devoting some effort into more solid theoretical study in understanding the over-parameterized training.

Intuitively speaking, the sensitive neuron has greater contribution for the final output, reusing the corresponding filter and carefully rescale its value require many empirically attempts. To achieve a more reasonable algorithm, author prune the redundant channel by controlling the deviation of the summation statistically small,  and reusing the filter by important sampling the given channel. Experiment show that this method can reach a competitive prune radio against other pruning algorithm, and show robustly in retained parameters vs error experiment.


Weakness:

1. experiment is too weak

ImageNet model has great impact on most CV problem, and the current release models are flooding in the open source world. Author should at least provide a imagenet model and make this work more convincing. Besides, Author should also consider an experiment in modern lightweight network, vgg and resnet like model are out of fashion and so big that any one can make a sound result on it. 

2. lack of a comparing experiment for random select the top-k norm. 

Important sampling require an input of probability [p1, p2, p3, ... pn],  if those probabilities are nearly uniform, important sampling will behave like a random sampling method. In most case, if we want to prune the large channel network,  picking the top-1 significant filter or random sampling top-k filter will almost do the same thing.  

3. lack of further theory consideration

author only consider the single layer reconstruction, without discussing the overall accumulative error. Unlike the other deterministic method, sampling skill suffer variance propagation problem, the pre-layer variance will affect the sampling probability of next layer, how this pruning work if we change status of the pre-layer,  I didn't find any theoretical guarantee and only find a proof of single layer reconstruction bound.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper attacks the problem of pruning neural networks to obtain sparser models for deployment. In introduces a principled importance sampling approach for which independence of samples allows one to obtain bounds easily. These bounds can be used to control the accuracy of the method. 

The proposal mechanism is very smart. The authors use a measure of the sensitivity of the network outputs to the channels in a particular layer (eqn 1).

The paper is very well written, but it would help to add a picture where all the symbols in section 2.1 appear. At times it is hard to keep track of the channels and features. It might alternatively be a good idea to specify the equation of a layer (eg what eventually ends up happening at the bottom of page 3) in section 2.1 and then explain the symbols in the equation. This will make life easier for anyone reading the paper for the first time.  

The experiments are well execute and include reasonable baselines. I would in addition recommend this recent paper:
https://arxiv.org/abs/1902.09574  

It would also be nice to relate the work to this best paper this year.
https://openreview.net/forum?id=rJl-b3RcF7
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies the tasks of pruning filters, a provable, sampling-based approach for generating compact Convolutional Neural Networks (CNNs). This paper gives rise to a fully-automated procedure for identifying and preserving the filters in layers that are essential to the network’s performance.  In general, this paper is very well written and organized.

1, The key concerns come from the Lottery papers [1,2]. One can find sparse structure from an overparameterized model.  The results of pruned network should be improved, rather than getting worse, since some redundant filters/params are removed from original network. In contrast, all the results of this method gets worse results; this is less desirable.

2, the theoretical analysis is very good;  it is worthy publishing themselves. But ever since the "lottery" papers, I think it makes sense in locating the sparse and representative pruned structure, which can achieve better performance than full overparameterized model. 
so it’s quite a borderline paper.  


[1] THE LOTTERY TICKET HYPOTHESIS: FINDING SPARSE, TRAINABLE NEURAL NETWORKS. ICLR 2019.
[2] RETHINKING THE VALUE OF NETWORK PRUNING. ICLR 2019.


review_title: Official Blind Review #2

Meta-review: This paper presents a sampling-based approach for generating compact CNNs by pruning redundant filters. One advantage of the proposed method is a bound for the final pruning error.

One of the major concerns during review is the experiment design. The original paper lacks the results on real work dataset like ImageNet. Furthermore, the presentation is a little misleading. The authors addressed most of these problems in the revision.

Model compression and purring is a very important field for real world application, hence I choose to accept the paper.

Decision: Accept (Poster)