Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a masking process to improve the pruning of DNN. In addition, the algorithm proposes to automatically allocate the pruning rates over layers. 

On the positive side:
- I do believe the main contribution is automatically allocating the pruning rates over the network. 





Related works:
- How does this relate to methods using gating to prune in the presence of residual layers or BN?

- Paper claims multiple times that related works need a train-prune-retrain process which is only valid for post-processing works. 

- I missed sparsity promoting works related to training from scratch where similar masks are implicitly used during training (even not explicitly stated) to maintain the zeros in the network. At least in [1,2] the training time is the same as the time used for training a network from scratch (same claim as in this paper). 


Method:
- The mask is trained using a sigmoid function and claims this will be representative of the relevance of the 'neuron' within the layer. How is this really related?
- Why the initialization of the mask is to the mean? I think I am confused there. For initialization, I would argue all the neurons/parameters are relevant, right?

- The claim that this method is 'much simpler' is a bit subjective. I do not see why. Please elaborate. 
- I am not sure if the claim of pruning filters and not considering the bn layer is correct. I would tend to think that, if pruning a neuron, the corresponding BN module should be modified (that is, propagating the zero to subsequent layers). 

- I do like the extension to residual connections. It would be great to have more and clearer details on how is this done. Would this also apply to the UNET type of architecture?

- What is the intuition behind Eq 4 and how it is related to the relevance of a parameter? 

- The extension to multiple metrics is of interest, however, there is little detailed there. How is the automatic allocation done? This is repeated in section 3.3 but details are missing. 


- My understanding is that the regularization multiplier is affected by the learning rate, therefore their effect is lower as the training progresses. In this case, seems the opposite, right? (page 6 before 3.4). 


- The part with the sparsity budget is interesting. What guarantees are that the newly enabled neurons are actually useful? Could it be possible that the budget suggested is not the right for the task at hand and, therefore, the additional parameters are not really relevant / needed? 





Experiments:

- There is loss with little sparsity when it comes to Imagenet (15 and 17% pruning) does not seem very promising even the training time is similar. 

- Seems like the experiments and comparisons to L1-pruning are not very surprising. It would be nice to have more comprehensive numbers. For imagenet, if using Resnet-50, would be easier to compare to other numbers. 
- In the imagenet comparison, L1-pruning is the version-A of the paper. What about the others or why that particular model?
- How are the actual groups made? 

Minor details:
- Please improve figure 1. It is not easy to understand. Same with Figure 3. What is seen in Figure 4. 
- I do believe the WideResNet-28-10 number of parameters for BAR is not correct. 
- Section 4.2 is a bit overselling. I do not see 'much-higher' parameter sparsity. The claims are mostly valid for VGG type of networks in this particular setting. 




[1] Learning the Number of Neurons in Deep Networks, NeurIps 2016
[2] Compression-aware training of DNN, NeurIps 2017

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this work, the authors propose a network pruning method to learn a pruned network during training. Specifically, they add a pruning mask for each layer and induce a sparisity loss on the mask variables during training. The pruned network is obtained by applying the learned mask to the networks.

The paper seems to be well contained. However, my assessment of this paper is weak reject. I am mainly concerned with the novelty of this method. Also i think some more evaluation is needed to fully understand the effectiveness of this method. My questions are summarized as follows:

Q1: In the methods part, the authors said that “Previous pruning approaches often prune Conv filters with their successive Batch Normalization layer unchanged.” Can the authors give some reference here as to which pruning approaches?

Q2: Did the authors compare the proposed approach to training the pruned networks from scratch as done in [1]? Also can the authors analyze the sparsity patterns of the pruned networks as done in section G in the appendix of [1]?

Q3: What is the difference of your approach to [2]? They seem to be very similar. I think it is necessary to add some discussion in the related work. Is there any experimental results for comparison with [2]?

[1] Rethinking the Value of Network Pruning. Liu et al. ICLR 2019
[2] AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference. Luo et al. Arxiv, 2018.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a framework for training time filter pruning for convolutional neural networks. The main idea is to use a trainable soft binary mask to zero out convolutional filters and corresponding batch norm parameters.

Pros:
+ The proposed method seems relatively easy to implement.
+ The quantitative results in the paper indicate that MaskConvNet achieves performance competitive with previously proposed pruning methods.

Cons:
- Writing of the paper could be significantly improved. See some examples below.
- The main thing that bothered me about the method was the usage of hard sigmoid. If a mask component ever gets into one of the flat regions it won’t be able to escape. The authors propose a workaround which they call “mask decay update”. This approach looks quite hacky and I’m not sure how easy it is to make it work in practice.

Notes/questions:
* Abstract: “elegant support” -> “support”
* Everywhere in the text: Back-to-back citations should have the form (citation1; citation2; …)
* Section 1, paragraph 3: “suffer one” -> “suffer from one”
* Section 1, paragraph 4: “above mentioned” -> “above-mentioned”
* Figure 1: The figure would greatly benefit from a detailed description. What’s IF, OF and OF? The reader shouldn’t be guessing. 
* Section 2, paragraph 3: “corresponded” -> “corresponding”
* Section 3.1, paragraph 2: “W \in R” – W is probably not a scalar value therefore it’s in R^n. The same goes for the mask.
* Section 3.1, paragraph 2: “It’s easy to know ...” – this sentence needs to be rewritten, e.g., “One can see that …”
* Section 3.1, paragraph 2: “sparser” -> “more sparse”
* Section 3.2, “Extension to Multi-metrics”: “FLOPs” are never defined in the paper. How is this quantity computed exactly? I’m also not entirely sure how useful it is to introduce multiple lambdas – it seems that this case corresponds to a single lambda which is a sum \lambda_i.
* Section 3.3, paragraph 1: “undersparsed”, “oversparsed” – not sure if these words exist. Maybe rephrase instead of introducing new terms?
* Section 3.3, paragraph 1: “very laborious” -> “laborious”
* Figure 3: Why not show points all the way to 0 sparsity?
* Section 4.2, CIFAR-10: The authors mention that (Lemaire et al., 19) achieve better FLOP sparsity due to usage of Knowledge Distillation. From this sentence alone it’s not clear how exactly KD helps. Why can’t KD be applied in the proposed framework? I’d appreciate if the authors could elaborate on this.

I must admit that I’m not an expert in the field of NN pruning but I’m surprised that training-time masking of filters has not been tried before. Even if it’s really the case I’m not entirely confident that the paper should be accepted: the “unsparsification” looks more like a hack than a principled approach and the overall quality of writing needs to be improved. I’m giving a borderline score for now but I’m willing to increase it provided that the rebuttal addresses my concerns.
review_title: Official Blind Review #3

Meta-review: This paper presents a method to learn a pruned convolutional network during conventional training.  Pruning the network has advantages (in deployment) of reducing the final model size and reducing the required FLOPS for compute.  The method adds a pruning mask on each layer with an additional sparsity loss on the mask variables. The method avoids the cost of a train-prune-retrain optimization process that has been used in several earlier papers.  The method is evaluated on CIFAR-10 and ImageNet with three standard convolutional network architectures.  The results show comparable performance to the original networks with the learned sparse networks. 

The reviewers made many substantial comments on the paper and most of these were addressed in the author response and subsequent discussion.  For example, Reviewer1 mentioned two other papers that promote sparsity implicitly during training (Q3), and the authors acknowledged the omission and described how those methods had less flexibility on a target metric (FLOPS) that is not parameter size.  Many of the author responses described changes to an updated paper that would clarify the claims and results (R1: Q2-7, R2:Q3).

However, the reviewers raised many concerns for the original paper and they did not see an updated paper that contains the proposed revisions.  Given the numerous concerns with the original submission, the reviewers wanted to see the revised paper to assess whether their concerns had been addressed adequately. Additionally, the paper does not have a comparison experiment with state-of the art results, and the current results were not sufficiently convincing for the reviewers.  Reviewer1 and author response to questions 13--15 suggest that the experimental results with ResNet-34 are inadequate to show the benefits of the approach, but results for the proposed method with the larger ResNet-50 (which could show benefits) are not yet ready. 

The current paper is not ready for publication.

Decision: Reject