Paper Summary:  This paper presents a new comprehension dataset called NewsQA dataset, containing 100,000 question-answer pairs from over 10,000 news articles from CNN. The dataset is collected through a four-stage process -- article filtering, question collection, answer collection and answer validation. Examples from the dataset are divided into different types based on answer types and reasoning required to answer questions. Human and machine performances on NewsQA are reported and compared with SQuAD.  Paper Strengths:  -- I agree that models can benefit from diverse set of datasets. This dataset is collected from news articles, hence might pose different sets of problems from current popular datasets such as SQuAD. -- The proposed dataset is sufficiently large for data hungry deep learning models to train.  -- The inclusion of questions with null answers is a nice property to have. -- A good amount of thought has gone into formulating the four-stage data collection process. -- The proposed BARB model is performing as good as a published state-of-the-art model, while being much faster.      Paper Weaknesses:  -- Human evaluation is weak. Two near-native English speakers' performance on 100 examples each can hardly be a representative of the complete dataset. Also, what is the model performance on these 200 examples? -- Not that it is necessary for this paper, but to clearly demonstrate that this dataset is harder than SQuAD, the authors should either calculate the human performance the same way as SQuAD or calculate human performances on both NewsQA and SQuAD in some other consistent manner on large enough subsets which are good representatives of the complete datasets. Dataset from other communities such as VQA dataset (Antol et al., ICCV 2015) also use the same method as SQuAD to compute human performance.  -- Section 3.5 says that 86% of questions have answers agreed upon by atleast 2 workers. Why is this number inconsistent with the 4.5% of questions which have answers without agreement after validation (last line in Section 4.1)? -- Is the same article shown to multiple Questioners? If yes, is it ensured that the Questioners asking questions about the same article are not asking the same/similar questions? -- Authors mention that they keep the same hyperparameters as SQuAD. What are the accuracies if the hyperparameters are tuned using a validation set from NewsQA? -- 500 examples which are labeled for reasoning types do not seem enough to represent the complete dataset. Also, what is the model performance on these 500 examples? -- Which model's performance has been shown in Figure 1? -- Are the two "students" graduate/undergraduate students or researchers? -- Test set seems to be very small. -- Suggestion: Answer validation step is nice, but maybe the dataset can be released in 2 versions -- one with all the answers collected in 3rd stage (without the validation step), and one in the current format with the validation step.   Preliminary Evaluation:  The proposed dataset is a large scale machine comprehension dataset collected from news articles, which in my suggestion, is diverse enough from existing datasets that state-of-the-art models can definitely benefit from it. With a better human evaluation, I think this paper will make a good poster.
Output: en