Title: NEWSQA: A MACHINE COMPREHENSION DATASET
Authors: Adam Trischler
Tong Wang
Xingdi Yuan
Justin Harris
Alessandro Sordoni
Philip Bachman
Kaheer Suleman
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: We present NewsQA, a challenging machine comprehension dataset of over 100,000 question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting in spans of text from the corresponding articles. We collect this dataset through a four- stage process designed to solicit exploratory questions that require reasoning. A thorough analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (25.3% F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available at datasets.maluuba.com/NewsQA.

TL;DR: Crowdsourced QA dataset with natural language questions and multi-word answers

Review discussion:
Dear Authors,
Could you please clarify the following --
1. How many workers answered each question in the answer sourcing step?
2. How many workers validated each (article, question, answer) triple in the validation step?
3.
When classifying the questions based on reasoning required, what percentage of questions fall into more than one category?
Also, what would be the new distributions across different reasoning types, if the questions falling into more than one category were defaulted to the least challenging type?
4. The human performance on NEWSQA is reported on only 100 questions. How were these 100 questions selected? It seems unlikely that these 100 questions will provide a reliable measure of the human performance on the entire dataset (which consists of thousands of questions).
5. Since every question in the NEWSQA dataset was answered by more than one worker, why was the same evaluation method as SQuAD not used for computing interhuman agreement for NEWSQA?
6. In table 3 (for SQuAD), are the human (ours) numbers on 100 sentences?
Thanks.
1. In answer sourcing, each question is answered by an average of 2.73 workers.
2. In validation, each worker is asked to either pick the best answer or reject all answers for each (article, question) tuple, rather than validating an (article, question, answer) triple. The average number of workers per (article, question) tuple is 2.48. We have add both stats to the paper.
3. The five reasoning types are implicitly considered "ordered" in the sense that by the order of their appearance in the paper, the later types subsumes the earlier types.
As a result, the annotators were asked to choose only the most subsuming category when more than one category apply. Essentially all questions require word matching and paraphrasing, as can be seen in the examples in Table 2. In this sense, defaulting to the least challenging type would be somewhat degenerate---all questions would be classified into word matching. It would probably be clearer for us to discuss these reasoning types as "stacking" on top of or subsuming each other. The synthesis questions certainly involve word matching, and often paraphrasing, and sometimes inference (as defined) as well.
4. Human performance was tested on 200 questions drawn uniformly at random from the development set (100 questions each for two humans).
It is true that this is a relatively small sample. We could increase the sample size for human evaluation given more time.
5. We did not use SQuAD's evaluation method because our process for collecting (and keeping) answers was different.
Rather than maintaining a set of distinct answers for each question, we keep only answers with interworker agreement whenever possible.
Including the validation stage, this amounts to 86% of the answers in the dataset, so this could be considered our human performance according to the SQuAD metric. The other 14% of answers are specially marked and considered ambiguous.
However, we wanted to get a more accurate measure of human performance on these agreed answers, hence our in-house testing.
6. They are on 200 questions from SQuAD's dev set.
For the BARB model, the authors mention -- "Contrasting the multiplicative application of attention vectors, this annotation matrix is to be concatenated to the encoder RNN input in the re-encoding stage." What is the reason behind this?
The intuition behind the annotation mechanism is that we hope the model (in particular the re-encoding RNN) can learn how to best use this multi-dimensional annotation, rather than imposing a particular operation as the way the attention information is used  (e.g., input weighting by multiplication, as in conventional attention models).
Empirically, the mechanism is observed to improve model performance.
- The crowdsourcing instruction required workers to mark a single continuous span. I wonder for answers that require synthesis, how often the annotated span covers multiple sentences that are far apart. It'd be helpful if the authors can provide the average length of the span for each answer type.
- Can authors provide statistics on question types? e.g., Who, When, Where, What, Which, Why, How?
1. The crowdsource workers are encouraged instead of required to mark single, continuous spans. Possibly as a result, the average number of spans is only slightly greater than 1 (around 1.05), and only about 0.6% of the answers span across multiple sentences.
2. Due to limited space, we omitted some statistics from the paper.
However, these stats (include the wh-word questions type distribution) are available on the dataset website, to which we will add a reference in the paper.
Summary: The paper proposes a novel machine comprehension dataset called NEWSQA. The dataset consists of over 100,000 question answer pairs based on over 10,000 news articles from CNN. The paper analyzes the different types of answers and the different types of reasoning required to answer questions in the dataset. The paper evaluates human performance and the performance of two baselines on the dataset and compares them with the performance on SQuAD dataset.
Strengths:
1. The paper presents a large scale dataset for machine comprehension.
2. The question collection method seems reasonable to collect exploratory questions. Having an answer validation step is desirable.
3. The paper proposes a novel (computationally more efficient) implementation of the match-LSTM model.
Weaknesses:
1. The human evaluation presented in the paper is not satisfactory because the human performance is reported on a very small subset (200 questions).
So, it seems unlikely that these 200 questions will provide a reliable measure of the human performance on the entire dataset (which consists of thousands of questions).
2. NEWSQA dataset is very similar to SQuAD dataset in terms of the size of the dataset, the type of dataset -- natural language questions posed by crowdworkers, answers comprising of spans of text from related paragraphs. The paper presents two empirical ways to show that NEWSQA is more challenging than SQuAD -- 1) the gap between human and machine performance in NEWSQA is larger than that in SQuAD.
However, since the human performance numbers are reported on very small subset, these trends might not carry over when human performance is computed on all of the dataset.
2) the sentence-level accuracy on SQuAD is higher than that in NEWSQA.
However, as the paper mentions, the differences in accuracies could likely be due to different lengths of documents in the two datasets.
So, even this measure does not truly reflect that SQuAD is less challenging than NEWSQA.
So, it is not clear if NEWSQA is truly more challenging than SQuAD.
3. Authors mention that BARB is computationally more efficient and faster compared to match-LSTM.
However, the paper does not report how much faster BARB is compared to match-LSTM.
4. On page 7, under "Boundary pointing" paragraph, the paper should clarify what "s" in "n_s" refers to.
Review summary: While the dataset collection method seems interesting and promising, I would be more convinced after I see the following --
1. Human performance on all (or significant percentage of the dataset).
2. An empirical study that fairly shows that NEWSQA is more challenging (or better in some other way) than SQuAD.
1. We acknowledge that human performance is currently measured on a small, potentially insignificant subset. This limitation arose from time constraints on the original submission. To remedy the issue, we are currently evaluating human performance on a larger subset of the corpus with in-house data analysts. We will update the paper accordingly once this process is completed.
2.
It is true that NewsQA is similar to SQuAD in terms of the characteristics cited.
On the other hand, there are differences we believe to be significant: namely, NewsQA's larger article coverage (12,744 documents vs. 536), longer articles (31 sentences on average vs. 5), and unique collection methodology (i.e., questioners do not see the full article). These are in addition to the raw difficulty level. To address the difficulty issues raised in the review:
(a) The expanded human evaluation will give a better estimate of NewsQA's challenge level with respect to SQuAD.
(b) We believe that in reading comprehension, document length may be a challenge in and of itself. Increasing length decreases the 'signal-to-noise' ratio for salient information; it is harder to select an answer span when the number of non-answer spans is higher.
Nonetheless, to address the above-mentioned concerns, we are conducting an additional experiment where document length is artificially increased in SQuAD by concatenating consecutive paragraphs from the same Wikipedia article. Preliminary results suggest that even with comparable document length, sentence-level accuracy with simple methods is still significantly lower on NewsQA than on SQuAD, confirming our original beliefs on the level of difficulty in NewsQA.
3. The lack of a numerical comparison was an oversight. We hence added a speed comparison between BARB and mLSTM (footnote 2, page 6) for the experiments reported in Section 6.2.
4. This was a typo and has been corrected. The value in question ("n_s") represents the number of tokens in the document (previously denoted by "n").
Thank you.
Paper Summary:
This paper presents a new comprehension dataset called NewsQA dataset, containing 100,000 question-answer pairs from over 10,000 news articles from CNN. The dataset is collected through a four-stage process -- article filtering, question collection, answer collection and answer validation. Examples from the dataset are divided into different types based on answer types and reasoning required to answer questions. Human and machine performances on NewsQA are reported and compared with SQuAD.
Paper Strengths:
-- I agree that models can benefit from diverse set of datasets. This dataset is collected from news articles, hence might pose different sets of problems from current popular datasets such as SQuAD.
-- The proposed dataset is sufficiently large for data hungry deep learning models to train.
-- The inclusion of questions with null answers is a nice property to have.
-- A good amount of thought has gone into formulating the four-stage data collection process.
-- The proposed BARB model is performing as good as a published state-of-the-art model, while being much faster.
Paper Weaknesses:
-- Human evaluation is weak. Two near-native English speakers' performance on 100 examples each can hardly be a representative of the complete dataset.
Also, what is the model performance on these 200 examples?
-- Not that it is necessary for this paper, but to clearly demonstrate that this dataset is harder than SQuAD, the authors should either calculate the human performance the same way as SQuAD or calculate human performances on both NewsQA and SQuAD in some other consistent manner on large enough subsets which are good representatives of the complete datasets. Dataset from other communities such as VQA dataset (Antol et al., ICCV 2015) also use the same method as SQuAD to compute human performance.
-- Section 3.5 says that 86% of questions have answers agreed upon by atleast 2 workers. Why is this number inconsistent with the 4.5% of questions which have answers without agreement after validation (last line in Section 4.1)?
-- Is the same article shown to multiple Questioners? If yes, is it ensured that the Questioners asking questions about the same article are not asking the same/similar questions?
-- Authors mention that they keep the same hyperparameters as SQuAD. What are the accuracies if the hyperparameters are tuned using a validation set from NewsQA?
-- 500 examples which are labeled for reasoning types do not seem enough to represent the complete dataset.
Also, what is the model performance on these 500 examples?
-- Which model's performance has been shown in Figure 1?
-- Are the two "students" graduate/undergraduate students or researchers?
-- Test set seems to be very small.
-- Suggestion: Answer validation step is nice, but maybe the dataset can be released in 2 versions -- one with all the answers collected in 3rd stage (without the validation step), and one in the current format with the validation step.
Preliminary Evaluation:
The proposed dataset is a large scale machine comprehension dataset collected from news articles, which in my suggestion, is diverse enough from existing datasets that state-of-the-art models can definitely benefit from it. With a better human evaluation, I think this paper will make a good poster.
1. As mentioned in an earlier response from us, we acknowledge that human performance is currently measured on a small and potentially insignificant subset. This limitation arose from time constraints on the original submission. To remedy the issue, we are currently expanding the human evaluation to a much larger subset using in-house data analysts. We will add model performance on human-evaluated data and update the paper accordingly once the process is completed.
2. The way human performance is calculated on SQuAD is not applicable to NewsQA since NewsQA has only one answer per question after validation, and the level of noise before validation discourages any agreement-based evaluation. The current human evaluation on SQuAD in our paper is conducted the same way as on NewsQA, and we do plan to expand this as well as the NewsQA human evaluation.
3. This was an oversight and has been corrected. We accidentally discounted null answers with agreement before validation (3.7%), with which the two percentages in Section 4.1 shall sum to 14.0% — the complement of the percentage (86.0%) in Section 3.5.
4. Yes, the same article is presented to multiple questioners. We applied an array of techniques to minimize redundancy among questions collected for the same article, like measuring edit distance between the questions and between a question and the summary points, etc. These methods were applied in pilot studies and/or as post-hoc filtering steps throughout the data collection process to eliminate repeat questions.
5. We will shortly start hyper-parameter tuning on NewsQA for both models and update the paper accordingly upon completion.
6. We will expand the reasoning-type stratification results together with our planned expansion on human evaluation (and add model performance on this). The model used in Figure 1 is BARB and this information has been added to the paper.
7. They are researchers.
8. Considering the increased difficulty of NewsQA, we adopted a 90-5-5% split on NewsQA simply to maximize the training data size (in contrast to the 80-10-10% split on SQuAD). The resulting 5k+ testing samples are at least on par with several existing QA datasets (e.g., training/test size for CNN: 380k / 3.2k; for CBT-CN: 121k / 2.5k; for CBT-NE: 109k / 2,5k).
9. The released version of the dataset (https://github.com/Maluuba/newsqa) in fact contains annotations for the pre-validation answer spans, so it is possible to recover the two versions of the dataset that the reviewers suggests.
Thank you.
It would seem that the shelf life of a dataset has decreased rapidly in recent literature. SQuAD dataset has been heavily pursued as soon as it hit online couple months ago, the best performance on their leaderboard now reaching to 82%. This is rather surprising when taking into account the fact that the formal conference presentation of the dataset took place only a month ago at EMNLP’16, and that the reported machine performance (at the time of paper submission) was only at 51%. One reasonable speculation is that the dataset may have not been hard enough.
NewsQA, the paper in submission, aims to address this concern by presenting a dataset of a comparable scale created through different QA collection strategies. Most notably, the authors solicit questions without requiring answers from the same turkers, in order to promote more diverse and hard-to-answer questions.
Another notable difference is that the questions are gathered without showing the content of the news articles, and the dataset makes use of a bigger subset of CNN/Daily corpus (12K / 90K), as opposed to a much smaller subset (500 / 90K) used by SQuAD.
In sum, I think NewsQA dataset presents an effort to construct a harder, large-scale reading comprehension challenge, a recently hot research topic for which we don’t yet have satisfying datasets.
While not without its own weaknesses, I think this dataset presents potential values compared to what are available out there today.
That said, the paper does read like it was prepared in a hurry, as there are numerous small things that the authors could have done better.
As a result, I do wonder about the quality of the dataset. For one, human performance of SQuAD measured by the authors (70.5 - 82%) is lower than that reported by SQuAD (80.3 - 90.5%). I think this sort of difference can easily happen depending on the level of carefulness the annotators can maintain.
After all, not all humans have the same level of carefulness or even the same level of reading comprehension. I think it’d be the best if the authors can try to explain the reason behind these differences, and if possible, perform a more careful measurement of human performance. If anything, I don’t think it looks favorable for NewsQA if the human performance is only at the level of 74.9%, as it looks as if the difficulty of the dataset comes mainly from the potential noise from the QA collection process, which implies that the low model performance could result from not necessarily because of the difficulty of the comprehension and reasoning, but because of incorrect answers given by human annotators.
I’m also not sure whether the design choice of not presenting the news article when soliciting the questions was a good one. I can imagine that people might end up asking similar generic questions when not enough context has been presented. Perhaps taking a hybrid, what I would like to suggest is to present news articles where some sentences or phrases are randomly redacted, so that the question generators can have a bit more context while not having the full material in front of them.
Yet another way of encouraging the turkers from asking too trivial questions is to engage an automatic QA system on the fly — turkers must construct a QA pair for which an existing state-of-the-art system cannot answer correctly.
Annotation noise is an unfortunate but inevitable feature of all crowd-sourced resources, so although this issue no doubt affects NewsQA, it is reasonable to assume that it is no more prominent in NewsQA than in SQuAD. We employed two measures to counteract annotation noise, i.e., answer agreement between separate turkers (in the answer-sourcing stage) and the validation process.
Consequently, we believe that the measured difference in human performance should reflect the difference in complexity and level of difficulty between the two datasets.
As mentioned in an earlier response, we are also in the process of expanding our human evaluation to a much larger subset using in-house data analysts. We will update the paper accordingly once this process is completed.
There are on average 3.56 summary points for each CNN article, which, in combination with the headline, offer a fairly informative context summarizing the full article (see examples in Appendix B) — informative at least for the purpose of soliciting interesting, non-generic questions.
Meanwhile, judging from the variety in the reasoning types involved in answering the questions, the design appears to have limited factoid-oriented questions that can be answered by simple string matching (a limitation of the SQuAD dataset we aim to overcome).
Overall, the choice appears to have struck a good balance between avoiding generic and trivial questions.
Nonetheless, the suggestion of random sentence/phrase redaction is interesting and well received among the authors. We will certainly consider the hybrid approach in future work for improving the dataset.
The suggestion of engaging a QA system in crowd-sourcing is also an interesting direction we might consider in the future.
Thank you.
Dear readers and reviewers,
Happy holidays! Before the final revision, we'd like to give you a quick update on our latest efforts and progress. We have currently completed 95% of the planned expansion on human evaluations (on both NewsQA and SQuAD), and we're in the process of hyperparameter tuning for both MatchLSTM and our BARB model on NewsQA. Our plan is to update relevant statistics and results no later than the end of next week.
Thank you!
Thank you everyone for your patience. Here's a list of updates we've made per the comments and suggestions of our reviewers.
Major updates:
* Increased our human evaluation of NewsQA from 100 to 1000 samples with reasoning type annotation.
* Performed the same human evaluation on 1000 SQuAD with reasoning type annotation.
* Updated stratification results (by reasoning type) for both NewsQA and SQuAD on the expanded subsets evaluated by human annotators.
* Added more sentence-level scoring experiments for a more conclusive demonstration of NewsQA's increased level of difficulty over SQuAD (eliminating article length as a factor in performance difference).
* Performed hyper-parameter tuning for both BARB and MatchLSTM model on the NewsQA dataset.
Minor updates:
* Added machine performance on the human-evaluated subsets for both NewsQA and SQuAD.
* Updated various statistics to be consistent with NewsQA's public release.
* Various minor updates, corrections, and additional statistics per requested in the reviews.
Thank you.
Thanks for carrying out the above studies and experiments.

Meta-review:
role: PC
data: comment: The program committee appreciates the authors' response to concerns raised in the reviews. All reviewers agree that the paper is not convincingly above the acceptance threshold. The paper will be stronger, and the benefit of this dataset over SQuAD will likely be more clear once authors incorporate reviewers' comments and finish evaluation of inter-human agreement.

Decision: Reject