Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents the observation that it is possible to utilize sparse operations in the training of LSTM networks without loss of accuracy. This observation is novel (although not too surprising) to my knowledge, but I must state that I am not very familiar with research on fast RNN implmentations.

Minor note:
The LSTM language model does not use a 'word2vec' layer. It is simply a linear embedding layer. Word2vec is the name of a specific model which is not directly to character level language models.

The paper presents the central observation clearly. However, it will be much more convincing if a well known dataset and experiment set up are used, such as Graves (2013) or Sutskever et al (2014), and actual training, validation and test performances are reported.

While the main observation is certainly interesting, I think it is not sufficient to be the subject of a full conference paper without implementation (or simulation) and benchmarking of the promised speedups on multiple tasks. For example, how would the gains be affected by various architecture choices?

At present this is an interesting technical report and I would like to see more detailed results in the future.
review_title: Detailed analysis/implementation needed

final_score: rating: 4: Ok but not good enough - rejection
review_content: The findings of applying sparsity in the backward gradients for training LSTMs is interesting. 

But the paper seems incomplete without the proper experimental justification. Only the validation loss is reported which is definitely insufficient. Proper testing results and commonly reported evaluation criterion needs to be included to support the claim of no degradation when applying the proposed sparsity technique. 

Also actual justification of the gains in terms of speed and efficiency would make the paper much stronger.

review_title: null

final_score: rating: null
review_content: Why are detailed results from the sensitivity, captioning and translation experiments not provided?
review_title: Results of Section 4.2

final_score: rating: null
review_content: 1) Any results to show the sparsified SGD does speedup the training? 

2) How does the final model perform on testing sets, not just the losses? 
review_title: training speed comparisons and final model performance

final_score: rating: 4: Ok but not good enough - rejection
review_content: CONTRIBUTIONS
When training LSTMs, many of the intermediate gradients are close to zero due to the flat shape of the tanh and sigmoid nonlinearities far from the origin. This paper shows that rounding these small gradients to zero results in matrices with up to 80% sparsity during training, and that training character-level LSTM language models with this sparsification does not significantly change the final performance of the model. The authors argue that this sparsity could be exploited with specialized hardware to improve the energy efficiency and speed of recurrent network training.

NOVELTY
Thresholding gradients to induce sparsity and improve efficiency in RNN training is a novel result to my knowledge.

MISSING CITATIONS
Prior work has explored low-precision arithmetic for recurrent neural network language models:

Hubara et al, “Quantized Neural Networks: Training Neural Networks with
Low Precision Weights and Activations”, https://arxiv.org/abs/1609.07061

Ott et al, “Recurrent Neural Networks With Limited Numerical Precision”, https://arxiv.org/abs/1608.06902 

Low-precision arithmetic for recurrent networks promises to improve both training and inference efficiency. How does the proposed sparsification method compare with low-precision arithmetic? Are the ideas complementary?

EXPERIMENTS
The main experimental result of the paper (Section 4.1) is that training LSTM language models with sparse gradients does not affect convergence or final performance (Figure 5). This result is promising, but I do not think that this single experiment is enough to prove the utility of the proposed method.

I also have some problems with this experiment. Plotting validation loss for character-level language modeling is not a standard way to report results; it is much more typical to report bits-per-character or perplexity on a held-out test set. These experiments also lack sufficient details for replication. What optimization algorithm, learning rate, and regularization were used? How were these hyperparameters chosen? Is the “truncated Wikipedia dataset” used for training the standard text8 dataset? In addition, the experiments do not compare with existing published results on this dataset.

In the OpenReview discussion, the authors remarked that the “The final validation loss for the sparsified model is [...] almost the same as the baseline.” Comparing validation loss at the end of training is not the proper way to compare models. From Figure 5, it is clear that all models achieve minimal validation loss after around 10k iterations, after which the validation losses increase, suggesting that the models have slightly overfit the training data by the end of training.

In Section 4.2 the authors claim to obtain similar experimental results with other network architectures, on other datasets (tiny-shakespeare and War and Peace), and for other tasks (image captioning and machine translation). However, the details and results of these experiments are not included in the paper, making it difficult to assess the utility of the proposed method and the significance of the results.

ENERGY EFFICIENCY AND TRAINING SPEED
One of the main claims of the paper is that sparse gradients can be exploited in hardware to reduce the training speed and improve the energy efficiency of recurrent network training, but these benefits are neither quantified nor demonstrated experimentally. Even without actually implementing custom hardware, would it be possible to estimate the expected improvements in efficiency through simulation or other means? Such results would significantly strengthen the paper.

GRADIENT TRUNCATION AS REGULARIZATION
In Figure 5 all models appear to reach a minimum validation loss at around 10k iterations and then overfit; at this point the Low model achieves even lower loss than the baseline. This is an interesting experimental result, but it is not discussed in the paper. Perhaps a low truncation threshold acts as a weak regularizer to prevent overfitting? Is this a general phenomenon of training recurrent networks with sparse gradients, or is it just a quirk of this particular experiment? This idea deserves more investigation, and could strengthen the paper.

SUMMARY
The core idea of the paper (thresholding gradients to induce sparsity and improve efficiency of RNN training) is interesting and practically useful, if a bit incremental. Nevertheless with thorough and deliberate experiments quantifying the tradeoffs between task performance, training speed, and energy efficiency across a variety of tasks and datasets, this simple idea could be the core of a strong paper.

Unfortunately, as written the paper provides neither theoretical arguments nor convincing experimental results to justify the proposed method, and as such I do not believe the paper is ready for publication in its current form.

PROS
- The proposed method is simple, and seems to be a promising direction for improving the speed of training recurrent networks.

CONS
- No discussion of prior work on low-precision recurrent networks
- Experimental results are not sufficient to validate the method
- Many experimental details are missing
- Results of key experiments (image captioning and machine translation) are missing

review_title: Review: Training Long Short-Term Memory with Sparsified Stochastic Gradient Descent

Meta-review: The main point of the paper was that sparsifying gradients does not hurt performance; however, this in itself is not enough for a publication in this venue. As noted by R1 and R2, showing how this can help in more energy efficient training would make for a good paper; without that aspect the paper only presents an observation that is not too surprising to the practitioners in this area.
 
 Further, while the main point of the paper was relatively clear, the scientific presentation was not rigorous enough. All the reviewers point out that several details were missing (including test set performance, reporting of results on the different sets, etc). 
 
 Paper would be strengthened by a better exploration of the problem.
Decision: Reject