Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a way to speed up the inference time of RNN via Skim mechanism where only a small part of hidden variable is updated once the model has decided a corresponding word token seems irrelevant w.r.t. a given task. While the proposed idea might be too simple, the authors show the importance of it via thorough experiments. It also seems to be easily integrated into existing RNN systems without heavy tuning as shown in the experiments. 

* One advantage of proposed idea claimed against the skip-RNN is that the Skim-RNN can generate the same length of output sequence given input sequence. It is not clear to me whether the output prediction on those skimmed tokens is made of the full hidden state (updated + copied) or a first few dimensions of the hidden state. I assume that the full hidden states are used for prediction. It is somehow interesting because it may mean the prediction heavily depends on small (d') part of the hidden state. In the second and third figures of Figure 10, the model made wrong decisions when the adjacent tokens were both skimmed although the target token was not skimmed, and it might be related to the above assumption. In this sense, it would be more beneficial if the skimming happens over consecutive tokens (focus on a region, not on an individual token).

* This paper would gain more attention from practitioners because of its practical purpose. In a similar vein, it would be also good to have some comments on training time as well. In a general situation where there is no need of re-training, training time would be meaningless, however, if one requires updating the model on the fly, it would be also meaningful to have some intuition on training time.

* One obvious way to reduce the computational complexity of RNN is to reduce the size of the hidden state. In this sense, it makes this manuscript more comprehensive if there are some comparisons with RNNs with limited-sized hidden dimensions (say 10 or 20). So that readers can check benefits of the skim RNN against skip-RNN and small-sized RNN.

review_title: Neural Speed Reading via Skim-RNN

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes a skim-RNN, which skims unimportant inputs with a small RNN while normally processes important inputs with a standard RNN for fast inference.

Pros.
-	The idea of switching small and standard RNNs for skimming and full reading respectively is quite simple and intuitive.
-	The paper is clearly written with enough explanations about the proposal method and the novelty.
-	One of the most difficult problems of this approach (non-differentiable) is elegantly solved by employing gumbel-softmax
-	The effectiveness (mainly inference speed improvement with CPU) is validated by various experiments. The examples (Table 3 and Figure 6) show that the skimming process is appropriately performed (skimmed unimportant words while fully read relevant words etc.)
Cons.
-	The idea is quite simple and the novelty is incremental by considering the difference from skip-RNN.
-	No comments about computational costs during training with GPU (it would not increase the computational cost so much, but gumbel-softmax may require more iterations).

Comments:
-	Section 1, Introduction, 2nd paragraph: ‘peed’ -> ‘speed’(?)
-	Equation (5): It would be better to explain why it uses the Gumbel distribution. To make (5) behave like argmax, only temperature parameter seems to be enough.
-	Section 4.1: What is “global training step”?
-	Section 4.2, “We also observe that the F1 score of Skim-LSTM is more stable across different configurations and computational cost.”: This seems to be very interesting phenomena. Is there some discussion of why skim-LSTM is more stable?
-	Section 4.2, the last paragraph: “Table 6 shows” -> “Figure 6 shows”

review_title: A review of "Neural Speed Reading via Skim-RNN"

final_score: rating: 7: Good paper, accept
review_content: Summary: The paper proposes a learnable skimming mechanism for RNN. The model decides whether to send the word to a larger heavy-weight RNN or a light-weight RNN. The heavy-weight and the light-weight RNN each controls a portion of the hidden state. The paper finds that with the proposed skimming method, they achieve a significant reduction in terms of FLOPS. Although it doesn’t contribute to much speedup on modern GPU hardware, there is a good speedup on CPU, and it is more power efficient.

Contribution:
- The paper proposes to use a small RNN to read unimportant text. Unlike (Yu et al., 2017), which skips the text, here the model decides between small and large RNN.

Pros:
- Models that dynamically decide the amount of computation make intuitive sense and are of general interests.
- The paper presents solid experimentation on various text classification and question answering datasets.
- The proposed method has shown reasonable reduction in FLOPS and CPU speedup with no significant accuracy degradation (increase in accuracy in some tasks).
- The paper is well written, and the presentation is good.

Cons:
- Each model component is not novel. The authors propose to use Gumbel softmax, but does compare other gradient estimators. It would be good to use REINFORCE to do a fair comparison with (Yu et al., 2017 ) to see the benefit of using small RNN.
- The authors report that training from scratch results in unstable skim rate, while Half pretrain seems to always work better than fully pretrained ones. This makes the success of training a bit adhoc, as one need to actively tune the number of pretraining steps.
- Although there is difference from (Yu et al., 2017), the contribution of this paper is still incremental.

Questions:
- Although it is out of the scope for this paper to achieve GPU level speedup, I am curious to know some numbers on GPU speedup.
- One recommended task would probably be text summarization, in which the attended text can contribute to the output of the summary.

Conclusion:
- Based on the comments above, I recommend Accept
review_title: A model that makes intuitive sense, with solid experimentation

Meta-review: this submission proposes an efficient parametrization of a recurrent neural net by using two transition functions (one large and one small) to reduce the amount of computation (though, without actual improvement on GPU.) the reviewers found the submission very positive.

please, do not forget to include all the result and discussion on the proposed approach's relationship to VCRNN which was presented at the same conference just a year ago.
Decision: Accept (Poster)