Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a way to use dropout induced sparsity for LSTMs to reduce run-time in general purpose SIMD hardware and systolic arrays, by structuring dropout patterns. Experiments on LM (PTB dataset), MT (IWSLT De-En and En-Vi), and NER (CoNLL-2003) shows that the proposed approach can induce 1.23x to 1.64x speedup without sacrificing accuracy.

main_review: * Pros
  * The core analysis of how to use dropout for efficient LSTM training is quite comprehensive
  * The general research direction to improve efficiency of NLP models is important
* Cons
  * This method seems only limit to RNN/LSTM models, yet RNN/LSTM is almost outdated. Can this method extends to Transformer?
  * The speedup of the proposed method is not significant (around 1.5x). How does it compare with constant optimization (i.e. better implementation)?


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work is mainly concerned with the time cost of LSTM computation. Inspired by the sparsity of neural networks, the authors propose to structure dropout patterns by discarding the same collections of neural units within a training batch. They have also carefully studied how dropout-induced sparsity propagated through different training phrases and how it was utilized in each of them. The experiments are very solid, with the investigations on LM, NMT, and NER.
main_review: The whole paper is really good. Accelerating the computation of sequential models is indeed important and the proposed method is awesome. However, the paper lacks a discussion on Transformer. For example, is your method applicable to improve its run-time efficiency? Transformer models are becoming more and more important in NLP. The authors should consider this.
limitations_and_societal_impact: 1, Excellent paper, regarding its topic, idea, method, and experiments.

2, A discussion about Transformer is necessary. I may change the score if no convincing response is given.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method to speed up training the LSTM networks. By introducing the structure into the dropout operation, authors are able to use sparse matrix multiplications. The paper carefully classifies different approaches drop out recurrent and feed-forward connections in LSTMs. Based on this classification, the paper chooses an approach to drop out both feed-forward and recurrent connections to get the most benefit from the sparse operations.

Second, the paper conducts experiments on series of tasks showing similar performance as baselines with speed-ups in training.
main_review: # Originality

The work looks novel enough. The motivation is a bit lacking, though. A usual practical model is trained once and used for thousands of clients in production after. I would like to see a more extensive motivation why speeding up the training 1.2-1.5 times is important. 
 
# Quality

In general, the work is focused on experimentation. The presented experiments look thorough. There are not enough details on how the training time was benchmarked. The work would benefit from computing number of FLOPS and comparing it the wall clock time. Second, all the experiments are focused on the language input. It would be beneficial to explore continuous tasks, such as time series.

It seems that the proposed approach depends on the mini-batch size. With a large mini-batch the speed up can be higher, but it might not be enough randomness in the mask sampling. With the small mini-batch I would expect small to no speed up. I am interested to see the discussion on this and experiments.

# Clarity

Section 3 is generally well written and easy to follow. The narrative is great. It is concise, but contains a lot of information. Section 4 is bit harder to follow, especially 4.3 looks rushed.

# Significance

The work looks significant enough.

# Suggestions for improvement

- Motivate the importance of this work.
- Compute FLOPS and describe the methodology of time benchmarking. Report the variance in time.
- Test with higher variety of data, such as time series, video, speech.
- Discuss and test the significance of the mini-batch size. Perhaps, use a toy dataset for this.

EDIT: the paper was improved, I increase the rating
limitations_and_societal_impact: The work is missing the discussion on the limitations of the approach. Is there any downsides or cases when it is not applicable?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present a generic framework to applying dropout as a sparsity technique to LSTM to reduce training time and as a plug-in technique w/o hyperparameters adjustment when comparing to previous similar techniques that focus on regularization. The dropout technique is randomized in timesteps but structured in the batch as a dropout mask that saves computation time on the dropout mask (does not compute), as well as inducing the regularization effect. This technique is applied to both the non-recurrent part of LSTM and the recurrent-hidden layer.
main_review: The paper is well written with a nice literature review. The author also provides a clear discussion about the difference of the proposed method and other literature. The computation acceleration across forward, backward, and gradient computation is well analyzed and verified with meaningful experiments. 

Testing on different types of GPUs could make the results more convincing.

Minor things:
Line 72, remove extra "that"
Line 164, "ST" appears for the first time, please define it. 

limitations_and_societal_impact: N/A
review_title: null

Meta-review: Paper presents results on structured dropout in recurrent models that can be used to speed up computation without affecting results. Reviewers found sufficient depth of exploration in results, and the results looked decent. Reviewers wanted to hear about the applicability to transformers, and the authors addressed that in the rebuttal.

Decision: Accept (Poster)