Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents a novel network pruning algorithm  -- Dynamic Sparse Training. It aims at jointly finding the optimal network parameters and sparse network structure in a unified optimization process with trainable pruning thresholds.  The experiments on MNIST, and cifar-10 show that proposed model can find sparse neural network models, but unfortunately with little performance loss.
The key limitation of the proposed model come from the experiments.
(1) Nowadays, the nature and important question is that, one can not tolerate the degraded performance, even with sparse neural network. Thus, it is important to show that the proposed model can find sparse neural network models, and with increased performance. 
(2) Another weakness is that proposed model has to be tested on large scale dataset, e.g. ImageNet-2012.current two datasets are too small to support the conclusive results of this proposed model. 
(3) As for the model itself, I donot find very significant novelty. For example, Sec. 3.3 (TRAINABLE MASKED LAYERS) in general is quite following previous works’ designing principle. Thus, the novelty should be summarized, and highlighted in the paper.

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes an algorithm for training networks with sparse parameter tensors. This involves achieving sparsity by application of a binary mask, where the mask is determined by current parameter values and a learned threshold. It also involves the addition of a specific regularizer which encourages the thresholds used for the mask to be large. Gradients with respect to both masked-out parameters, and with respect to mask thresholds, are computed using a "long tailed" variant of the straight-through-estimator.

The algorithm proposed in this paper seems sensible, but rather ad hoc. It is not motivated by theory or careful experiments. As such, the value of the paper will be determined largely by the strength of the experimental results. I believe the experimental results to be strong, though I am not familiar enough with this subfield to be confident there are not missing baselines.

There are many minor English language problems (e.g. with articles, prepositions, plural vs. singular forms, and verb tense), though these don't significantly interfere with understanding.

Rounding up to weak accept, though my confidence is low because I am basing this positive assessment on experimental results for tasks on which I am not well calibrated.

more detailed comments:

"using the same training epochs" -> "using the same number of training epochs"
"achieves prior art performance" -> "achieves state of the art performance"
"the inference of deep neural network" -> "inference in deep neural networks"

This paper considered only sparsity of weights -- it might have been nice to also discuss/run experiments exploring sparsity of activations.

eq. 4 -- Can you say more about why this particular form is the right one for the regularizer? It seems rather arbitrary. (it will tend to be dominated by the smallest thresholds, and so would seem to encourage a minimum degree of sparsity in every layer)

I appreciate the analysis in section 4.3.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: ## Update after the rebuttal
I appreciate the author's clarification in the rebuttal and the additional result on ImageNet, which addressed some of my concerns.

# Summary
This paper proposes a trainable mask layer in neural networks for compressing neural networks end-to-end. The main idea is to apply a differentiable mask to individual weights such that the mask itself is also trained through backpropagation. They also propose to add a regularization term that encourages weights are masked out as much as possible. The result on MNIST and CIFAR show that their method can achieve the highest (weight) compression rate and the lowest accuracy reduction compared to baselines. 

# Originality
- The idea of applying trainable mask to weights and regularizing toward masking out is quite interesting and new to my knowledge. 

# Quality
- The performance seems to be good, though it would be more convincing if the paper showed results on larger datasets like ImageNet. 

- The analysis is interesting, but I am not fully convinced by the "strong evidence to the efficiency and effectiveness of our algorithm". For example, the final layer's remaining ratio is constantly 1 in Figure 3, while it starts from nearly 0 in Figure 4. The paper also argues that the final layer was not that important in Figure 4 because the lower layers have not learned useful features. This seems not only contradictory to the result of Figure 3 but also inconsistent of the accuracy being quickly increasing up to near 90% while the remaining ratio is nearly 0 in Figure 4. 

- If the motivation of the sparse training is to reduce memory consumption AND computation, showing some results on the reduction of the computation cost after sparse training would important to complete the story. 

# Clarity
- The description of the main idea is not clear. 

- What are "structure gradient" and "performance gradient"? They are not mathematically defined in the paper.

- I do not understand how the proposed method can "recover" from pruned connection, although it seems to be indeed happening in the experiment. The paper claims that the use of long-tailed higher-order estimator H(x) makes it possible to recover. However, H(x) still seems to have flat lines where the derivative is 0. Is H(x) in Equation 3 and Figure 2d are showing "derivative" or step function itself? In any cases, I do not see how the gradient flows once a weight is masked out. 

# Significance
- This paper proposes an interesting idea (trainable mask), though I did not fully get how the mask is defined/trained and has a potential to recover after pruning. The analysis of the compression rate throughout training is interesting but does not seem to be fully convincing. It would be stronger if the paper 1) included more results on bigger datasets like ImageNet, 2) described the main idea more clearly, and 3) provided more convincing evidence why the proposed method is effective. 
review_title: Official Blind Review #2

Meta-review: The paper lies on the borderline. An accept is suggested based on majority reviews and authors' response.
Decision: Accept (Poster)