Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
Main contribution of the paper
- The paper proposes a new pruning method that dynamically updates the sparse mask and the network weight.
- Different from the other works, the proposed method does not require post-tuning.
- A theoretical explanation of the method is provided.

Methods
- In this method, the weight of the baseline network is updated not by the gradient from the original weight but pruned weight.
- Here, pruning can be conducted by (arbitrary) a pruning technique given the network weight (Here, the author uses the magnitude-of-the-weight given method from Han.et.al).


Questions
- See the Concerns

Strongpoints
- The author provides the simple and effective pruning method and verifies the performance with a sufficient amount of experiments.
- The author argues that the method is applicable to various pruning techniques.

Concerns
- It seems that the paper omits the existing work (You.et.al - https://arxiv.org/pdf/1909.08174.pdf), which seems to share some contribution. The reviewer wants the author to clarify the differences and the strongpoints compared to the work.
- The main pruning&update equation (DPF) does not seem to force the original network w to become sparse, such as by l1-regularization. So, the reviewer worried that the method might not produce sparsity if the initial weights are not that sparse.
If the reviewer missed the explanation about this, clarify this.
- Regarding the above concern, what if we add regularization term in training the original network w? 
- As far as the reviewer knows, the proposed method improves the sparsity of the network, but most works choosing the strategy actually cannot meaningfuly enhance the operation time and just enhances the sparsity. Does the author think that the proposed method can enhance the latency? If so, a detailed explanation or experiment will be required.

Conclusion
- The author proposes a simple but effective dynamic pruning method.
- The reviewer has some concerns regarding the novelty, real speed up, and guarantee of the sparsity. 
However, the reviewer thinks that this work has meaningful observations for this field with a sufficient amount of verification, assuming that the author's answers for the concerns do not have much problem.

Inquiries
- See the Concerns parts.
review_title: Official Blind Review #2317

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors proposed a novel model compression method that uses error feedbacks to dynamically allocates sparsity patterns during training. The authors provided a systematic overview of a good number of existing model compression algorithms depending on the relative order of pruning and training processes. The effectiveness of the proposed algorithm is illustrated by comparing its generalization performance with 6 existing algorithms (and their variants) with two standard datasets and various networks of standard structures. The authors also showed the convergence rate and the fundamental limit of the proposed algorithm with two theorems. 

This paper is well-written and very pleasant to read. I would like to accept this paper. But since I have never actually done research in model compression, I would say this is my 'educated guess'. 

Some quick comments:
1. I did not go through the proofs of the two theorems. But it seems that there is a typo in the definition of strong convexity on Page 4: '\Delta f(w)' should be '\Delta f(v)'. I assume that this is just a typo. 
2. Sorry again for not knowing the details of the baseline algorithms. According to Table 1 and Table 2, the proposed method (DPF) outperforms all the baseline algorithms, without a single exception, which looks suspicious for me. After reading the paper, I still don't understand why this should be the case. Is this due to some implementation details? Can you think of some scenarios that the proposed algorithm may not be the one to go with? In other words, when the experiment seems to show that one algorithm absolutely outperforms all the other existing algorithms, there should be some take-home message on why, or some known limitations of the proposed method. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This work proposes a simple pruning method that dynamically sparsifies the network during training. This is achieved by performing at fixed intervals magnitude based pruning for either individual weights or entire neurons. While similar methods have been explored before, this work proposes a slight twist; instead of updating the weights of the model by following the gradient of the parameters of the dense model, they update the parameters of the dense model according to the gradients of the sparse model. Essentially, this corresponds to a variant of the straight-through estimator [1], where in the forward pass we evaluate the compressed model, but in the backward pass we update the model as if the compression didn’t take place. The authors argue that this process allows for ``feedback” in the pruning mechanism, as the pruned weights still receive gradient updates hence they can be ``re-activated” at later stages of training. They then provide a convergence analysis about the optimization procedure with such a gradient, and show that for strongly convex functions the method converges in the vicinity of the global optimum, whereas for non-convex functions it converges to the neighbourhood of a stationary point. Finally, the authors perform extensive experimental evaluation and show that their method is better than the baselines that they considered.

This work is in general well written and conveys the main idea in an effective manner.  It is also a timely contribution as sparse models / compression are important topics for the deep learning community. The overall method seems simple to implement, doesn’t introduce too many hyper-parameters and seem to work very well. For this reason I tend towards recommending for acceptance, provided that the authors address /comment on a couple of issues I found in the draft. 

More specifically:
- The connection to the error feedback is kind of loose and not well explained. After skimming Karimireddy et al. I noticed that 1. it evaluates the gradient at a point (i.e. the current estimate of the parameters), 2. compresses said gradient, 3. updates the parameters while maintaining the difference of the original w.r.t. the compressed gradient. In this sense, it seems a bit different that DPF, as your notation at the first equation of page 4 implies that you take the gradient of a different point, i.e. w_t + e_t instead of w_t. I believe that expanding a bit more about the connection would help in making the manuscript more clear.
- There seems to be a typo / error on your definition of an m-strongly convex function at the “convergence of Convex functions” paragraph. I believe it should be <\nabla f(v), w-v> <= f(w) - f(v) - 0.5 m ||w - v||^2, instead of <\nabla f(w), w-v> <= f(w) - f(v) - 0.5 m ||w - v||^2.
- The proposed gradient estimator seems to be an instance of the STE [1] estimator, that, as the authors mention, has been using at the Binary Connect algorithm. It would be interesting to see some more discussion about this similarity perhaps also expanding upon recent work that discusses the STE gradient as a form of coarse gradient [2].
- At section 5.2 the authors mention that “dynamic pruning methods, and in particular DPF, work on a different paradigm, and can still heavily benefit from fine-tuning”. This claim seems to contradict the results at Figure 4; there it seems that the masks have “converged” in the later stages of training, hence one could argue that the fine-tuning already happens thus it wouldn’t benefit DPF. I believe it would be interesting if the authors provide a similar plot as the one in Figure 4 but rather for the ResNet-20 network on CIFAR 10 (which seems to benefit heavily from FT). Do the masks still settle at the end of training (as it was the case for WideResNet-28-2) and if they do, why is fine-tuning still increasing the accuracy?
- Minor: Try to use consistent coloring at Figure 6 as while (a), (b) share the same color-coding, (c) is using a different one hence could be confusing.

[1] Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, Yoshua Bengio, Nicholas Léonard, Aaron Courville, 2013
[2] Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets, Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, Jack Xin, 2019
review_title: Official Blind Review #1

Meta-review: The paper proposes a new, simple method for sparsifying deep neural networks.                                                      
It use as temporary, pruned model to improve pruning masks via SGD, and eventually                                                 
applying the SGD steps to the dense model.                                                                                         
The paper is well written and shows SOTA results compared to prior work.                                                           
                                                                                                                                   
The authors unanimously recommend to accept this work, based on simplicity of                                                      
the proposed method and experimental results.                                                                                      
                                                                                                                                   
I recommend to accept this paper, it seems to make a simple, yet effective                                                         
contribution to compressing large-scale models.                                                                                    
                             
Decision: Accept (Poster)