Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper collects a cloze-style fill-in-the-missing-word dataset constructed manually by English teachers to test English proficiency.  Experiments are given which are claimed to show that  this dataset is difficult for machines relative to human performance.  The dataset seems interesting but I find the empirical evaluations unconvincing.  The models used to evaluate machine difficulty are basic language models.  The problems are multiple choice with at most four choices per question.  This allows multiple choice reading comprehension architectures to be used.   A window of words around the blank could be used as the "question".  A simple reading comprehension baseline is to encode the question (a window around the blank) and use the question vector to compute an attention over the passage.  One can then compute a question-specific representation of the passage and score each candidate answer by the inner product of the question-specific sentence representation and the vector representation of the candidate answer.  See "A thorough examination of the CNN/Daily Mail reading comprehension task" by Chen, Bolton and Manning.


review_title: This is an interesting dataset but the baselines are not very compelling.

final_score: rating: 7: Good paper, accept
review_content: This paper presents a new dataset for cloze style question-answering. The paper starts with a very valid premise that many of the automatically generated cloze datasets for testing reading comprehension suffer from many shortcomings. The paper collects data from a novel source: reading comprehension data for English exams in China. The authors collect data for middle school and high school exams and clean it to obtain passages and corresponding questions and candidate answers for each question.

The rest of the paper is about analyzing this data and performance of various models on this dataset. 

1) The authors divide the questions into various types based on the type of reasoning needed to answer the question, noticeably short-term reasoning and long-term reasoning. 
2) The authors then show that human performance on this dataset is much higher than the performance of LSTM-based and language model-based baselines; this is in contrast to existing cloze style datasets where neural models achieve close to human performance. 
3) The authors hypothesize that this is partially explained by the fact that neural models do not make use of long-distance information. The authors verify their claim by running human eval where they show annotators only 1 sentence near the empty slot and find that the human performance is basically matched by a language model trained on 1 billion words. This part is very cool.
4) The authors then hypothesize that human-generated data provides more information. They even train an informativeness prediction network to (re-)weight randomly generated examples which can then be used to train a reading comprehension model.

Pros of this work:
1) This work contributes a nice dataset that addresses a real problem faced by automatically generated datasets.
2) The breakdown of characteristics of questions is quite nice as well.
3) The paper is clear, well-written, and is easy to read.

Cons:
1) Overall, some of the claims made by the paper are not fully supported by the experiments. E.g., the paper claims that neural approaches are much worse than humans on CLOTH data -- however, they do not use state-of-the-art neural reading comprehension techniques but only a standard LSTM baseline. It might be the case that the best available neural techniques are still much worse than humans on CLOTH data, but that remains to be seen. 
2) Informativeness prediction: The authors claim that the human-generated data provides more information than automatically/randomly generated data by showing that the models trained on the former achieve better performance than the latter on test data generated by humans. The claim here is problematic for two reasons:
   a) The notion of "informativeness" is not clearly defined. What does it mean here exactly?
   b) The claim does not seem fully justified by the experiments -- the results could just as well be explained by distributional mismatch without appealing to the amount of information per se. The authors should show comparisons when evaluating on randomly generated data.

Overall, this paper contributes a useful dataset; the analysis can be improved in some places.
review_title: Promising dataset; needs better experiments to analyze

final_score: rating: 4: Ok but not good enough - rejection
review_content: 1) this paper introduces a new cloze dataset, "CLOTH", which is designed by teachers. The authors claim that this cloze dataset is a more challenging dataset since CLOTH requires a deeper language understanding and wider attention span. I think this dataset is useful for demonstrating the robustness of current RC models. However, I still have the following questions which lead me to reject this paper.

2) I have the questions as follows:
i) The major flaw of this paper is about the baselines in experiments. I don't think the language model is a robust baseline for this paper.  When a wider span is used for selecting answers, the attention-based model should be a reasonable baseline instead of pure LM. 
ii) the author also should provide the error rates for each kind of questions (grammar questions or long-term reasoning). 
iii) the author claim that this CLOTH dataset requires wider span for getting the correct answer, however, there are only 22.4 of the entire data need long-term reasoning. More importantly, there are 26.5% questions are about grammar. These problems can be easily solved by LM. 
iv) I would not consider 16% percent of accuracy is a "significant margin" between human and pure LM-based methods. LM-based methods should not be considered as RC model.
v) what kind accuracy is improved if you use 1-billion corpus trained LM? Are these improvements mostly in grammar? I did not see why larger training corpus for LM could help a lot about reasoning since reasoning is only related to question document.

review_title: Not convinced

Meta-review: Meta score: 4

The paper presents a manually-constructed cloze-style fill-in-the-missing-word dataset, with baseline language modelling experiments that aim to show that  this dataset is difficult for machines relative to human performance.  The dataset is interesting but the fact that the experiments are confined to baseline language models
Pros:
 - interesting dataset
 - clear and well-written
 - attempt to move the field forward in an important area
Cons:
 - limited experimentation
 - language modelling approaches not appropriate baseline


Decision: Reject