Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a sequential algorithm that tackles text classification while introducing the ability to stop the reading when the decision to make is confident enough. This sequential framework -reinforcement learning with budget constraint- will be applied to document classification tasks. The authors propose a unified framework enabling the recurrent network to reread or skip some parts of a document. Then, the authors describe the process to ensure both a good classification ability & a reduced budget.
Experiments are conducted on the IMDB dataset and the authors demonstrate the interest to select the relevant part of the document to make their decision. They improve both the accuracy & decision budget.


In the architecture, fig 1, it is strange to see that the decision to stop is taken before considering the label probability distribution. This choice is probably made to fit with classical sequential decision algorithms, assuming that the confidence level can be extracted from the latent representation... However, it should be discussed.

The interest of rereading a word/sentence is not clear for me: we simply choose to overweight the recent past wrt the further. Can it be seen as a way to overcome a weakness in the information extraction?

At the end of page 4, the difference between the early stopping model & partial reading model is not clear for me. How can the partial reading model be overcome by the early-stopping approach? They operate on the same data, with the same computational cost (ie with the same algorithm?)

At the end of page 6, authors claim that their advantage over Yu et al. 2017 comes from their rereading & early stopping abilities:
- given the length of the reviews may the K-skip ability of Yu et al. 2017 be seen as an early stopping approach?
- are the authors confident about the implementation of the Yu et al. 2017' strategy?
- Regarding the re-reading ability: the experimental section is very poor and we wonder:
  -- how the performance is impacted by rereading?
  -- how many time does the algorithm choose to reread?
  -- experiments on early-stopping VS early-stopping + skipping + rereading are interesting... We want to see the impact of the other aspects of the contribution.

On the sentiment analysis task, how does the model behave wrt the state of the art?

Given the chosen tasks, this work should be compared to the beermind system:
http://deepx.ucsd.edu/#/home/beermind
and the associated publication
http://arxiv.org/pdf/1511.03683.pdf
But the authors should also refer to previous work on their topic:
https://arxiv.org/pdf/1107.1322
The above mentioned reference is really close to their work.


This article describes an interesting approach but its main weakness resides in the lack of positioning wrt the literature and the lack of comparison with state-of-the-art models on the considered tasks.

review_title: Interesting model but strong lacks in related works

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to augment RNNs for text classification with a mechanism that decides whether the RNN should re-read a token, skip a number of tokens, or stop and output the prediction. The motivation is that one can stop reading before the end of the text and/or skip some words and still arrive to the same answer but faster.

The idea is intriguing, even though not entirely novel. Apart from the Yu et al. (2017) cited, there is older work trying to save computational time in NLP, e.g.: 
Dynamic Feature Selection for Dependency Parsing.
He He, Hal Daumé III and Jason Eisner.
Empirical Methods in Natural Language Processing (EMNLP), 2013
that decides whether to extract a feature or not.
However, what is not clear to me what is achieved here. In the example shown in Figure 5 it seems like what happens is that by not reading the whole text the model avoids passages that might be confusing it. This could improve predictive accuracy (as it seems to do), as long as the model can handle better the earlier part of the text. But this is just an assumption, which is not guaranteed in any way. It could be that the earlier parts of the text are hard for the model. In a way, it feels more like we are addressing a limitation of RNN models in understanding text. 

Pros:
- The idea is intersting and if evaluated thoroughly it could be quite influential.

Cons:
- the introduction states that there are two kinds of NLP problems, sequence2sequence and sequence2scalar. I found this rather confusing since text classification falls in the latter presumably, but the output is a label. Similarly, PoS tagging has a linear chain as its output, can't see why it is sequence2scalar. I think there is a confusion between the methods used for a task, and the task itself. Being able to apply a sequence-based model to a task, doesn't make it sequential necessarily.

- the comparison in terms of FLOPs is a good idea. But wouldn't the relative savings depend on the architecture used for the RNN and the RL agent? E.g. it could be that the RL network is more complicated and using it costs more than what it saves in the RNN operations.

- While table 2 reports the savings vs the full reading model, we don't know how much worse the model got for these savings. 

- Having a trade-off between savings and accuracy is a good idea too. I would have liked to see an experiment showing how many FLOPs we can save for the same performance, which should be achievable by adjusting the alpha parameter.

- The experiments are conducted on previously published datasets. It would be good to have some previously published results on them to get a sense of how good the RNN model used is.

- Why not use smaller chunks? 20 words or one sentence at the time is rather coarse. If anything, it should help the model proposed achieve greater savings. How much does the choice of chunk matter?

- it is stated in the conclusion that the advantage actor-critic used is beneficial, however no experimental comparison is shown. Was it used for the Yu et al. baseline too?

- It is stated that model hardly relies on any hyperparameters; in comparison to what? It is better to quantify such statements,
review_title: Unclear focus: is it the speed gains or the improved accuracy? Experimental comparison not convincing

final_score: rating: 7: Good paper, accept
review_content: The paper present a model for fast reading for text classification with mechanisms that allow the model to reread, skip words, or classify early before reading the entire review. The model contains a policy module that makes decisions on whether to reread, skim or stop, which is rewarded for both classification accuracy and computation cost. The entire architecture is trained end-to-end with backpropagation, Monte Carlo rollouts and a baseline for variance reduction. 

The results show that the architecture is able to classify accurately on all syntactic levels, faster than a baseline that reads the entire text. The approach is simple and seems to work well and could be applied to other tasks where inference time is important.  
review_title: You don't need to read the entire review to classify it.

Meta-review: this is an interesting approach that applies the idea of dynamically controlling the amount of information from the input fed into the classifier (some of the earlier approaches have used this idea for, e.g., parsing, real-time translation, online speech recognition, and so on...) this is also related to some of the recent work on hierarchical recurrent nets [Chung et al.]. unfortunately, two of the reviewers and other commenters found this manuscript needs more work to clarify motivation, implication and relationship to other existing works, with which i don't necessarily disagree. 
Decision: Invite to Workshop Track