Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Contributions:

This paper introduces BookTest, a new and large training data for machine comprehension. While CBT has ~200K examples, this dataset has ~14M examples. Authors also take the AS Reader model and show the performance gain on CBT test set when trained with BookTest training data. Additional human study shows that humans can answer more than 50% of the questions which the model fails to answer which means there is room for improvement.

Novelty: 

Adding more training data helps. There is nothing novel in this. However, authors provide a bigger dataset to train machine comprehension model which I think is valuable.

Clarity: 

Paper is extremely well written (given the page constraints).

Significance: 

This is a significant contribution since I believe that this data can be used to pre-train any machine comprehension model.

Pros:
+ new dataset which is very large and useful
+ well written paper

Cons:
- authors could have benchmarked using more models (than just AS Reader)

review_title: Introduces a large dataset for training machine comprehension models

final_score: rating: 7: Good paper, accept
review_content: This paper introduces the BookTest dataset, which is a huge dataset of reading comprehension questions compared to existing datasets. While the paper does not contain any new methodological contributions, the dataset has potential to be extremely valuable to the research community, and as such I hope it is accepted to the workshop. The authors perform a human evaluation to show that, unlike the CNN dataset, there is room for models to improve on this larger dataset. 

With that said, it is interesting that the human performance is actually below that of some of the models; I wonder what types of questions humans are answering incorrectly, especially since human performance on the questions incorrectly answered by ASReader is relatively high.  It would be great if the authors could add some discussion on this in future versions of the paper. If the types of questions machines struggle with are different than those humans struggle with, this could potentially impact the sorts of architectural decisions made by future researchers to improve accuracy on this task.
review_title: review

Decision: Accept