Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a block sparsity pruning approach to compress RNNs. There are several ways. One is using  group LASSO to promote sparsity. The other is to prune, but with a very specialized schedule as to the pruning and pruning weight, motivated by the work of Narang et al 2017 for non-group sparsity.  The block sizes used in experiments are about 4x4, 8x8, up to 32 x 32. The relative performance degradation ranges between 10% to 96%, depending on the method, severity of compression, and task. The speedup for a matrix multiply is between 1.5x to 4x, and varies according to batch size.

This is certainly a well-motivated problem, and the procedure is simple but makes sense. Also, the paper contains a good overview of related work in compression, and is not hiding anything.  One paper that is missing is

Scardapane, S., Comminiello, D., Hussain, A., & Uncini, A. (2017). Group sparse regularization for deep neural networks. Neurocomputing, 241, 81-89.

A major complaint is the lack of comparison of results against other compression techniques. Since it is a block sparsity approach, and the caching / fetching overhead is reduced, one does not need to have competitively superior results to basic pruning approaches, but one should come close on the same types of problems. This is not well presented. Additionally, the speedup should be superior to the sparse methods, which is also not shown (against previously published results, not personally run experiments.) 

Another issue I find is the general writing, especially for the results section, is not entirely clear. For example, when showing a relative performance degradation of 96%, why is that happening? Is that significant? What should an implementer be aware of in order to avoid that? 

Finally, a meta issue to address is, if the block size is small (realistically, less than 64 x 64) usually I doubt there will be significant speedup. (4x is not considered terribly significant.) What we need to see is what happens when, say, block size is 256 x 256? What is the performance degradation? If you can give 10x speedup in the feedforward part (testing only) then if you have a 10% degradation in performance that might be acceptable in certain applications. 

Overall, I believe this is a very promising and well-motivated work, but needs to be "marinated" further to be publishable. Actually, with the inclusion of 2-3 tables against known, previously published results, and clearly stated benefits, I would change my review to accept. 

Minor complaints:

The axis labels/numbers in figure 2 are too small. 

Also, please reread for some grammar / writing issues (last paragraph of 1st page, caption of figure 2, for example)

I think also figure 2 should be rerun with more trials. The noise in the curves are not showing a highly interpretable trend. (Though, actually, batch size = 8 being super low might have a significance; can this be explained?)


review_title: Good start, but needs better comparison against existing work. 

final_score: rating: 7: Good paper, accept
review_content: Compressing/pruning of neural networks is required to enable running on devices with limited compute resources. While previous works have proposed to 0 out weights, especially for the case of RNNs, in an unstructured way, the current paper proposes to 0 out weights blocks at a time via thresholding. The process is further aided by utilizing group lasso regularization. The resulting networks are sparse, memory efficient and can be run more efficiently while resulting in minimal loss in accuracy when compared to networks learned with full density. The proposed techniques are evaluated on RNNs for speech recognition and benefits clearly spelled out. Further experiments thresh out how much benefit is provided by thresholding (block sparsity) and regularizing via group lasso.

The paper quality seems high, presentation clarity sufficient, the ideas presented (especially the use of group lasso) well thought out and original, and the work seems significant. If I were to nitpick then I would suggest trying out these techniques on RNNs meant for something other than speech recognition (machine translation perhaps?).
review_title: The paper proposes to 0 out blocks of weights while training RNNs and further aid the process by utilizing group lasso regularization. As demonstrated empirically, the learned networks are sparse and can be run efficiently while showing minimal loss of accuracy.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Thanks to the authors for their response.

Though the paper presents an interesting approach, but it relies heavily on heuristics (such as those mentioned in the initial review) without a thorough investigation of scenarios in which this might not work. Also, it might be helpful to investigate if there ways to better group the variables for group  lasso regularization. The paper therefore needs further improvements towards following a more principled approach.

=====================================
This paper presents methods for inducing sparsity in terms of blocks of weights in neural networks which aims to combine benefits of sparsity and faster access based on computing architectures. This is achieved by pruning blocks of weights and group lasso regularization. It is demonstrated empirically that model size can be reduced by upto 10 times with some loss in prediction accuracy.

Though the paper presents some interesting evaluations on the impact of block based sparsity in RNNs, some of the shortcomings of the paper seem to be :

- The approach taken consists of several heuristics rather than following a more principled approach such as taking the maximum of the weights in a block to represent that block and stop pruning till 40% training has been achieved. Also, the algorithm for computing the pruning threshold is based on a new set of hyper-parameters. It is not clear under what conditions the above settings will (not) work.

 - For the group lasso method, since there are many ways to group the variable, it is not clear how the variables are grouped. Is there a reasoning behind a particular grouping of the variables. Individually, group lasso does not seem to work, and gives much worse results. The reasons for worse performance could be investigated. It is possible that important weights are in different groups, and group sparsity is forcing some of them to be zero, and hence leading to worse results. It would be insightful to explain the kind of solver used for group lasso regularization, and if that works for large-scale problems.

 - The results for various kinds of sparsity are unclear in the sense that it is not clear how to set the block size a-priori for having minimum reduction in accuracy and still significant sparsity without having to repeat the process for various choices.

Overall, the paper does not seem to present novel ideas, and is mainly focused on evaluating the impact of block-based sparsity instead of weight pruning by Han etal. As mentioned in Section 2, regularization has been used earlier to achieve sparsity in deep networks. In this view the significance over existing work is relatively narrow, and no explicit comparison with existing methods is provided. It is possible that an existing method leads to pruning method such as by Han etal. leads to 8x decrease in model size while retaining the accuracy, while the proposed method leads to 10x decrease while also decreasing the accuracy by 10%. Scenarios like these need to be evaluated to understand the impact of the method proposed in this paper.
review_title: Methods for inducing sparsity in terms of blocks of weights in neural networks.

Meta-review: Pros
-- Interesting approach to induce sparsity, trains faster than alternative approaches
Cons
-- Fairly complex set of heuristics for pruning weights
-- Han et al. works well, although the authors claim it takes more time to train, which may not not hold for all training sets and doesn’t seem like a strong enough reason to choose an alternative appraoch

Given these comments, the AC recommends that the paper be rejected.

Decision: Reject