Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Have you tried prunning on LSTM models? 
review_title: Have tried on LSTM?

final_score: rating: null
review_content: In introduction: "... unlike previous approaches
such as in Han et al. (2015). State of the art results in speech recognition generally require between
days and weeks of training time, so a further 3-4× increase in training time is undesirable."

But, according to Han et al. (2015), "Huffman coding doesn’t require training and is implemented
offline after all the fine-tuning is finished."

Both yours and Han et al. (2015) use a weight pruning technique. Intuitively, they should have similar training time for LSTM models.
Where does 3-4x extra training time comes from Han et al. (2015) but doesn't have in your approach?
review_title: Training time

final_score: rating: 7: Good paper, accept
review_content: Summary: The paper presents a technique to convert a dense to sparse network for RNNs. The algorithm will increasingly set more weights to zero during the RNN training phase. This provides a RNN model with less storage requirement and higher inference rate. 

Pros:
Proposes a pruning method that doesn’t need re-training and doesn’t affect the training phase of RNN. The method achieves 90% sparsity, and hence less number of parameters.

Cons & Questions:
Judiciously choosing hyper parameters for different models and different applications wouldn’t be cumbersome? In equation 1, is q the sparsity of final model? Is there a formula to know what is sparsity, number of parameters and accuracy of final model given a set of hyper parameters, before going through training? (Questions answered)

In table3, we see a trade-off between number of units and sparsity to achieve better number of parameters or accuracy, or in table5 better speed. Good, but where are the results for GRU sparse big? I mean, accuracy must be similar and still get decent compression rate and speed up. Just like RNN Sparse medium compared with RNN Dense. I can’t see much advantage of pruning and getting high speed-up if you are sacrificing so much accuracy. (Issue fixed with updated data)

Why sparsity for table3 and table5 are different? In text: “average sparsity of 88%” but in table5 is 95%? Are the models used in table3 different from table5? (Issue fixed)

review_title: sparsity vs accuracy

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a method for pruning weights in neural networks during training to obtain sparse solutions. The approach is applied to an RNN-based system which is trained and evaluated on a speech recognition dataset. The results indicate that large savings in test-time computations can be obtained without affecting the task performance too much. In some cases the method can actually improve the evaluation performance.

The experiments are done using a state-of-the-art RNN system and the methodology of those experiments seems sound. I like that the effect of the pruning is investigated for networks of very large sizes. The computational gains are clearly substantial. It is a bit unfortunate that all experiments are done using a private dataset. Even with private training data, it would have been nice to see an evaluation on a known test set like the HUB5 for conversational speech. It would also have been nice to see a comparison with some other pruning approaches given the similarity of the proposed method to the work by Han et al. [2] to verify the relative merit of the proposed pruning scheme. While single-stage training looks more elegant at first sight, it may not save much time if more experiments are needed to find good hyperparameter settings for the threshold adaptation scheme. Finally, the dense baseline would have been more convincing if it involved some model compression tricks like training on the soft targets provided by a bigger network.

Overall, the paper is easy to read. The table and figure captions could be a bit more detailed but they are still clear enough. The discussion of potential future speed-ups of sparse recurrent neural networks and memory savings is interesting but not specific to the proposed pruning algorithm. The paper doesn’t motivate the details of the method very well. It’s not clear to me why the threshold has to ramp up after a certain period time for example. If this is based on preliminary findings, the paper should mention that.

Sparse neural networks have been the subject of research for a long time and this includes recurrent neural networks (e.g., sparse recurrent weight matrices were standard for echo-state networks [1]). The proposed method is also very similar to the work by Han et al. [2], where a threshold is used to prune weights after training, followed by a retraining phase of the remaining weights. While I think that it is certainly more elegant to replace this three stage procedure with a single training phase, the proposed scheme still contains multiple regimes that resemble such a process by first training without pruning followed by pruning at two different rates and finally training without further pruning again. The main novelty of the work would be the application of such a scheme to RNNs, which are typically more tricky to train than feedforward nets.

Improving scalability is an important driving force of the progress in neural network research. While I don’t think the paper presents much novelty in ideas or scientific insight, it does show that weight pruning can be successfully applied to large practical RNN systems without sacrificing much in performance. The fact that this is possible with such a simple heuristic is a result worth sharing.


Pros:
The proposed method is successful at reducing the number of parameters in RNNs substantially without sacrificing too much in performance.
The experiments are done using a state-of-the-art system for a practical application.

Cons:
The proposed method is very similar to earlier work and barely novel.
There is no comparison with other pruning methods.
The data is private and this prevents others from replicating the results.


[1] Jaeger, H. (2001). The “echo state” approach to analyzing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, 148, 34.


[2] Han, Song, Pool, Jeff, Tran, John, and Dally, William J. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems, 2015.
review_title: Review: Exploring Sparsity in Recurrent Neural Networks

final_score: rating: null
review_content: When weights are added again, which value do they initially have? In Section 3, the text describes that weights will be used again during the forward pass if their gradient is larger than the threshold for that layer. Assuming that the updated weights would be much smaller than the gradient because of the learning rate scaling and the threshold growing over time, they are quite likely to be removed again at the next iteration. This step is not precisely defined and I'm curious how common it is for weights to reappear after being pruned.
review_title: When weights are added again, which value do they initially have?

Meta-review: Here is a summary of the reviews:
 
 Strengths
 Experiments are done on state-of-the-art networks, on a real speech recognition problem (R3, R1)
 Networks themselves are of a very large size (R3)
 Computational gains are substantial (R3, R4)
 Paper is clear (R1)
 
 Weaknesses
 Experiments are all done on a private dataset (R3)
 No comparison to other pruning approaches (e.g. Han et al.) (R3); AC notes that reviewers added new results which compare to an existing pruning method
 No comparison to distillation techniques (R1)
 Paper doesn't present much novelty in terms of ideas (R3)
 
 The AC encouraged feedback from the reviewers following author rebuttal and paper improvements. Reviewers stated that the improvements made to the paper made it publishable but was still closer to the threshold. R1 who had originally rated the paper 3: a "clear reject" updated the score to 6 (just above acceptance).
 
 Considering the reviews and discussions, the AC thinks that this paper is a poster accept. There are no serious flaws, the improvements made to the paper during the discussion paper have satisfied the reviewers, and this is an important topic with practical benefits; evaluated on a real large-scale problem.
Decision: Accept (Poster)