Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents a method for iteratively pruning redundant weights in deep networks. The method is primarily based on a 3-step pipeline to achieve this objective. These three steps consist of pruning, weight sharing and quantization. The authors demonstrate reduction in model size and number of parameters significantly while only undergoing minor decrease in accuracy.

Some of the main points of concern are below :

 - Computational complexity - The proposed method of iterative pruning seems quite computationally expensive. In the conclusion, it is mentioned that it takes 35 days of training for MNIST. This seems extremely high, and given this, it is unclear if there is much benefit in further reduction in model sizes and parameters (by the proposed method) than those obtained by existing method such as Han etal.

 - The novelty in the paper is quite limited and is mainly based on combining existing methods for pruning, weight sharing and quantization. The main difference from existing method seems to be the inclusion of layerwise threshold for weight pruning instead of using a single global threshold.

 - The results shown in Table 2 do not indicate much difference in terms of number of parameters between the proposed method and that of Han etal. For instance, the number of overall remaining parameters is 6.5% for the proposed method versus 8% for Deep Compression. As a result, the impact of the proposed method seems quite limited. 

 - The paper in the title and abstract refers to segmentation as the main area of focus. However, there does not seem to be much related to it except an experiment on the CityScapes dataset.
review_title: iterative deep compression

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper inherits the framework proposed by Han[1]. A pruning, weight sharing, quantization pipeline is refined at each stage. At the pruning stage, by taking into account difference in the distribution across the layers, this paper propose a dynamic threshold pruning, which partially avoids mistakenly pruning important connections. As for the weight sharing stage, this paper explores several ways to initialize the clustering method. The introduction of error tolerance gives us more fine-grained control over the compression process.

Here are some issues to be paid attention to:

1. The overall pipeline including the last two stage looks quite similar to Han[1]. Though different initialization methods are tested in this paper, final conclusion does not change.

2. The dynamic threshold pruning seems to be very time-consuming. As indicated from the paper, only 42 iterations for MNIST and 32 iterations for Cityscapes are required. Whether these number works for each layer or total network should be clarified.

3. Fig 7(a) says it's error rate while it plots accuracy rate.

4. Experiments on popular network structure such as residual connection should be conducted, as they are widely used nowadays.


References:
[1] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015 

review_title: Good work but much similarity with an existing work

final_score: rating: 6: Marginally above acceptance threshold
review_content: quality: this paper is of good quality
clarity: this paper is very clear
originality: this paper combines original ideas with existing approaches for pruning to obtain dramatic space reduction in NN parameters.
significance: this paper seems significant.

PROS
- a new approach to sparsifying that considers different thresholds for each layer
- a systematic, empirical method to obtain optimal sparsity levels for a given neural network on a task.
- Very interesting and extensive experiments that validate the reasoning behind the described approach, with a detailed analysis of each step of the algorithm.

CONS
- Pruning time. Although the authors argue that the pruning algorithm is not prohibitive, I would argue that >1 month to prune LeNet-5 for MNIST is certainly daunting in many settings. It would benefit the experimental section to use another dataset than MNIST (e.g. CIFAR-10) for the image recognition experiment.
- It is unclear whether this approach will always work well; for some neural nets, the currently used sparsification method (thresholding) may not perform well, leading to very little final sparsification to maintain good performance.
- The search for the optimal sparsity in each level seems akin to a brute-force search. Although possibly inevitable, it would be valuable to discuss whether or not this approach can be refined.

Main questions
- You mention removing "unimportant and redundant weights" in the pruning step; in this case, do unimportant and redundant have the same meaning (smaller than a given threshold), or does redundancy have another meaning (e.g. (Mariet, Sra, 2016))?
- Algorithm 1 finds the best sparsity for a given layer that maintains a certain accuracy. Have you tried using a binary search for the best sparsity instead of simply decreasing the sparsity by 1% at each step? If there is a simple correlation between sparsity and accuracy, that might be faster; if there isn't (which would be believable given the complexity of neural nets), it would be valuable to confirm this with an experiment.
- Have you tried other pruning methods than thresholding to decide on the optimal sparsity in each layer?
- Could you please report the final accuracy of both models in Table 2?

Nitpicks:
- paragraph break in page 4 would be helpful.
review_title: Pruning NNs with layer-dependent parametrization, combined with other approaches for space reduction (weight sharing, quantization) significantly decrease space requirements without hurting accuracy. However, this comes at the cost of significant pruning time.

Meta-review: This paper presents a new pipeline for nn compression that extends that of Han et. al, but show that it reduces parameters further, maintains higher accuracy and can be applied to methods behind classification (semantic segmentation). While the authors found the paper clearly written, excepting for some typos, and potentially useful, there were questions about originality, and significance.

- Reviewers were not completely convinced the method was different enough from deep compression: "The overall pipeline including the last two stage looks quite similar to Han[1].", or that enough focus was paid to the differences inherent with classification focused work: "The paper in the title and abstract refers to segmentation as the main area of focus. However, there does not seem to be much related to it except an experiment on the CityScapes dataset."

- In terms of impact, the additional benefits from pruning seem to require a significant amount of computation, and the reviewers were not convinced these were worth a small gain in compression. Furthermore, authors felt that this approach was not being applied to the most state-of-the-art approaches to demonstrate their use.
Decision: Reject