Introduction
[color=red!20,size=,fancyline,caption=,disable]ben:It is a little weird that RECORD is not spelled out in the abstract, but especially odd that it isn't spelled out in the Introduction. I would remove the footnote, put that content in the Introduction
[color=red!20,size=,fancyline,caption=,disable]ben:@kev agree. ... Human and Machine Commonsense Reading Comprehension
[color=red!20,size=,fancyline,caption=,disable]ben:Methods in machine reading comprehension (MRC) are driven by the datasets available – such as curated by deepmind-cnn-dailymail, cbt, squad, newsqa, and msmarco – where an MRC task is commonly defined as answering a question given some passage. However ...
Machine reading comprehension (MRC) is a central task in natural language understanding, with techniques lately driven by a surge of large-scale datasets BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , usually formalized as a task of answering questions given a passage. An increasing number of analyses BIBREF5 , BIBREF6 , BIBREF7 have revealed that a large portion of questions in these datasets can be answered by simply matching the patterns between the question and the answer sentence in the passage. While systems may match or even outperform humans on these datasets, our intuition suggests that there are at least some instances in human reading comprehension that require more than what existing challenge tasks are emphasizing. [color=red!20,size=,fancyline,caption=,disable]ben:This "thus" claim is far too strong. You haven't cited anything that says humans *don't* rely on simple pattern matching, you just rely on an implicit assumption that 'surely humans must be doing something complicated when they read'. If a system performs as well as a human on a task, the conclusion shouldn't immediately be that the task is too easy, it should more subtly be that new datasets are then needed to see if the inference mechanisms hold up, where the creation of the datasets can be based based on an explicitly stated intuition that humans may rely on more than pattern matching. It is a hypothesis at this point in the Introduction, that systems doing well on earlier datasets won't also do well on yours. You expect they will fail, and even design the dataset specifically around their failure cases. [color=red!20,size=,fancyline,caption=,disable]ben:I would say: While systems may match or even outperform humans on these datasets, our intuition suggests that there are at least some instances in human reading comprehension that require more than what existing challenge tasks are stressing. One primary type of questions these datasets lack are the ones that require reasoning over common sense or understanding across multiple sentences in the passage BIBREF2 , BIBREF3 . [color=red!20,size=,fancyline,caption=,disable]ben:This statement is given without citation: why do you claim that common sense is missing? Do you provide an analysis later in this paper that supports it? If so, provide a forward reference. If you can cite earlier work, do so. Otherwise, remove or soften this statement, e.g., "We hypothesize that one type of question ...". And then in next sentence, rather than "To overcome this limitation", which you haven't proven yet actually exists, you would say: "To help evaluate this question, we introduce ..." [color=red!20,size=,fancyline,caption=,disable]ben:rather than "most of which require", say "most of which seem to require some aspect of reasoning beyond immediate pattern matching". The SWAG / BERT case should be fresh in your mind as you write this introduction, and where-ever you are tempted to declare things in absolute terms. The more you go on the record as THIS DATASET REQUIRES COMMONSENSE then the more you look silly later if someone finds a 'trick' to solve it. A more honest and safer way to put this is to exactly reference the SWAG/BERT issue at some point in this paper, acknowledging that prior claims to have constructed commonsense datasets have been shown to either be false, or to imply that commonsense reasoning can be equated to large scale language modeling. You can cite Rachel's Script Induction as Language Modeling paper, JOCI, and the reporting bias article, perhaps all in a footnote, when commenting that researchers have previously raised concerns about the idea that all of common sense can be derived from corpus co-occurrence statistics.
To overcome this limitation, we introduce a large-scale dataset for reading comprehension, ReCoRD (), which consists of over 120,000 examples, most of which require deep commonsense reasoning. ReCoRD is an acronym for the Reading Comprehension with Commonsense Reasoning Dataset.
fig:example shows a ReCoRD example: the passage describes a lawsuit claiming that the band “Led Zeppelin” had plagiarized the song “Taurus” to their most iconic song, “Stairway to Heaven”. The cloze-style query asks what does “Stairway to Heaven” sound similar to. To find the correct answer, we need to understand from the passage that “a copyright infringement case alleges that `Stairway to Heaven' was taken from `Taurus'”, and from the bullet point that “these two songs are claimed similar”. Then based on the commonsense knowledge that “if two songs are claimed similar, it is likely that (parts of) these songs sound almost identical”, we can reasonably infer that the answer is “Taurus”. [color=purple!20,size=,fancyline,caption=,disable]kev:This example is good, but you might need to make sure the reader reads the whole passage first or else it may be hard to follow. Maybe add a few more sentences to explain Figure 1 in the paragraph here.
Differing from most of the existing MRC datasets, all queries and passages in ReCoRD are automatically mined from news articles, which maximally reduces the human elicitation bias BIBREF8 , BIBREF9 , BIBREF10 , and the data collection method we propose is cost-efficient. [color=purple!20,size=,fancyline,caption=,disable]kev:You should have one of these comparison tables that lists multiple MRC datasets and compares different features Further analysis shows that a large portion of ReCoRD requires commonsense reasoning.
Experiments on ReCoRD demonstrate that human readers are able to achieve a high performance at 91.69 F1, whereas the state-of-the-art MRC models fall far behind at 46.65 F1. Thus, ReCoRD presents a real challenge for future research to bridge the gap between human and machine commonsense reading comprehension. [color=red!20,size=,fancyline,caption=,disable]ben:this is a bulky URL: I will pay the small fee to register some domain name that is more slick than this [color=red!20,size=,fancyline,caption=,disable]ben:about the leaderboard on the website: I think it a little misleading to have Google Brain and IBM Watson, etc. as the names on the leaderboard, if it is really you running their code. Better would be "JHU (modification of Google Brain system)", "JHU (modification of IBM Watson system)", ... .
Task Motivation
A program has common sense if it automatically deduces for itself a sufficiently wide class of immediate consequences of anything it is told and what it already knows. – mccarthy59
Commonsense Reasoning in MRC As illustrated by the example in fig:example, the commonsense knowledge “if two songs are claimed similar, it is likely that (parts of) these songs sound almost identica” is not explicitly described in the passage, but is necessary to acquire in order to generate the answer. Human is able to infer the answer because the commonsense knowledge is commonly known by nearly all people. Our goal is to evaluate whether a machine is able to learn such knowledge. However, since commonsense knowledge is massive and mostly implicit, defining an explicit free-form evaluation is challenging BIBREF11 . Motivated by mccarthy59, we instead evaluate a machine's ability of commonsense reasoning – a reasoning process requiring commonsense knowledge; that is, if a machine has common sense, it can deduce for itself the likely consequences or details of anything it is told and what it already knows rather than the unlikely ones. To formalize it in MRC, given a passage $\mathbf {p}$ (i.e., “anything it is told” and “what it already knows”), and a set of consequences or details $\mathcal {C}$ which are factually supported by the passage $\mathbf {p}$ with different likelihood, if a machine $\mathbf {M}$ has common sense, it can choose the most likely consequence or detail $\mathbf {c}^*$ from $\mathcal {C}$ , i.e.,
$$\mathbf {c}^* = \operatornamewithlimits{arg\,max}_{\mathbf {c} \in \mathcal {C}}P(\mathbf {c}\mid \mathbf {p},\mathbf {M}).$$   (Eq. 2)
[color=purple!20,size=,fancyline,caption=,disable]kev:What are the properties of $o$ ? What can be a consequence? Be more specific or give examples.
Task Definition With the above discussion, we propose a specific task to evaluate a machine's ability of commonsense reasoning in MRC: as shown in fig:example, given a passage $\mathbf {p}$ describing an event, a set of text spans $\mathbf {E}$ marked in $\mathbf {p}$ , and a cloze-style query $Q(\mathbf {X})$ with a missing text span indicated by $\mathbf {X}$ , a machine $\mathbf {M}$ is expected to act like human, reading the passage $\mathbf {p}$ and then using its hidden commonsense knowledge to choose a text span $\mathbf {e}\in \mathbf {E}$ that best fits $\mathbf {X}$ , i.e.,
$$\mathbf {e}^* = \operatornamewithlimits{arg\,max}_{\mathbf {e} \in \mathbf {E}}P(Q(\mathbf {e})\mid \mathbf {p},\mathbf {M}).$$   (Eq. 3)
Once the cloze-style query $Q(\mathbf {X})$ is filled in by a text span $\mathbf {e}$ , the resulted statement $Q(\mathbf {e})$ becomes a consequence or detail $\mathbf {c}$ as described in eq:csr-in-mrc, which is factually supported by the passage with certain likelihood.
[color=purple!20,size=,fancyline,caption=,disable]kev:There's a disconnect between this paragraph and the previous one. How do you jump from $o$ to Q(e) and the ineqality to argmax? Also, I'm not sure if "cloze" is defined anywhere: you might need a one-sentence explanation in case the reader is not familiar.
Data Collection
[color=purple!20,size=,fancyline,caption=,disable]kev:First add motivation about general philosophy of data collection We describe the framework for automatically generating the dataset, ReCoRD, for our task defined in eq:task, which consists of passages with text spans marked, cloze-style queries, and reference answers. We collect ReCoRD in four stages as shown in Figure 2 : (1) curating CNN/Daily Mail news articles, (2) generating passage-query-answers triples based on the news articles, (3) filtering out the queries that can be easily answered by state-of-the-art MRC models, and (4) filtering out the queries ambiguous to human readers.
News Article Curation
We choose to create ReCoRD by exploiting news articles, because the structure of news makes it a good source for our task: normally, the first few paragraphs of a news article summarize the news event, which can be used to generate passages of the task; and the rest of the news article provides consequences or details of the news event, which can be used to generate queries of the task. In addition, news providers such as CNN and Daily Mail supplement their articles with a number of bullet points BIBREF12 , BIBREF13 , BIBREF0 , which outline the highlights of the news and hence form a supplemental source for generating passages.
We first downloaded CNN and Daily Mail news articles using the script provided by BIBREF0 , and then sampled 148K articles from CNN and Daily Mail. In these articles, named entities and their coreference information have been annotated by a Google NLP pipeline, and will be used in the second stage of our data collection. Since these articles can be easily downloaded using the public script, we are concerned about potential cheating if using them as the source for generating the dev./test datasets. Therefore, we crawled additional 22K news articles from the CNN and Daily Mail websites. These crawled articles have no overlap with the articles used in BIBREF0 . We then ran the state-of-the-art named entity recognition model BIBREF14 and the end-to-end coreference resolution model BIBREF15 provided by AllenNLP BIBREF16 to annotate the crawled articles. Overall, we have collected 170K CNN/Daily Mail news articles with their named entities and coreference information annotated.
Passage-Query-Answers Generation
All passages, queries and answers in ReCoRD were automatically generated from the curated news articles. fig:example-for-stage2 illustrates the generation process. (1) we split each news article into two parts as described in sec:news-curation: the first few paragraphs which summarize the news event, and the rest of the news which provides the details or consequences of the news event. These two parts make a good source for generating passages and queries of our task respectively. (2) we enriched the first part of news article with the bullet points provided by the news editors. The first part of news article, together with the bullet points, is considered as a candidate passage. To ensure that the candidate passages are informative enough, we required the first part of news article to have at least 100 tokens and contain at least four different entities. (3) for each candidate passage, the second part of its corresponding news article was split into sentences by Stanford CoreNLP BIBREF17 . Then we selected the sentences that satisfy the following conditions as potential details or consequences of the news event described by the passage:
[itemsep=0pt,topsep=6pt,leftmargin=10pt]
Sentences should have at least 10 tokens, as longer sentences contain more information and thus are more likely to be inferrable details or consequences.
Sentences should not be questions, as we only consider details or consequences of a news event, not questions.
Sentences should not have 3-gram overlap with the corresponding passage, so they are less likely to be paraphrase of sentences in the passage.
Sentences should have at least one named entity, so that we can replace it with $\mathbf {X}$ to generate a cloze-style query.
All named entities in sentences should have precedents in the passage according to coreference, so that the sentences are not too disconnected from the passage, and the correct entity can be found in the passage to fill in $\mathbf {X}$ .
Finally, we generated queries by replacing entities in the selected sentences with $\mathbf {X}$ . We only replaced one entity in the selected sentence each time, and generated one cloze-style query. Based on coreference, the precedents of the replaced entity in the passage became reference answers to the query. The passage-query-answers generation process matched our task definition in sec:task, and therefore created queries that require some aspect of reasoning beyond immediate pattern matching. In total, we generated 770k (passage, query, answers) triples.
Machine Filtering
As discussed in BIBREF5 , BIBREF6 , BIBREF18 , BIBREF7 , existing MRC models mostly learn to predict the answer by simply paraphrasing questions into declarative forms, and then matching them with the sentences in the passages. To overcome this limitation, we filtered out triples whose queries can be easily answered by the state-of-the-art MRC architecture, Stochastic Answer Networks (SAN) BIBREF19 . We choose SAN because it is competitive on existing MRC datasets, and it has components widely used in many MRC architectures such that low bias was anticipated in the filtering (which is confirmed by evaluation in sec:evaluation). We used SAN to perform a five-fold cross validation on all 770k triples. The SAN models correctly answered 68% of these triples. We excluded those triples, and only kept 244k triples that could not be answered by SAN. These triples contain queries which could not be answered by simple paraphrasing, and other types of reasoning such as commonsense reasoning and multi-sentence reasoning are needed. [color=purple!20,size=,fancyline,caption=,disable]kev:Briefly mention why you use SAN, i.e. it's competitive on current benchmarks like SQuAD. Also mention whether this may cause some bias in the filtering, compared to using some other system, and why your methodology is still ok.
Human Filtering
Since the first three stages of data collection were fully automated, the resulted triples could be noisy and ambiguous to human readers. Therefore, we employed crowdworkers to validate these triples. We used Amazon Mechanical Turk for validation. Crowdworkers were required to: 1) have a 95% HIT acceptance rate, 2) a minimum of 50 HITs, 3) be located in the United States, Canada, or Great Britain, and 4) not be granted the qualification of poor quality (which we will explain later in this section). Workers were asked to spend at least 30 seconds on each assignment, and paid $3.6 per hour on average.
fig:hit shows the crowdsourcing web interface. Each HIT corresponds to a triple in our data collection. In each HIT assignment, we first showed the expandable instructions for first-time workers, to help them better understand our task (see the sec:hit-instructions). Then we presented workers with a passage in which the named entities are highlighted and clickable. After reading the passage, workers were given a supported statement with a placeholder (i.e., a cloze-style query) indicating a missing entity. Based on their understanding of the events that might be inferred from the passage, workers were asked to find the correct entity in the passage that best fits the placeholder. If workers thought the answer is not obvious, they were allowed to guess one, and were required to report that case in the feedback box. Workers were also encouraged to write other feedback.
To ensure quality and prevent spamming, we used the reference answers in the triples to compute workers' average performance after every 1000 submissions. While there might be coreference or named entity recognition errors in the reference answers, as reported in BIBREF20 (also confirmed by our analysis in sec:data-analysis), they only accounted for a very small portion of all the reference answers. Thus, the reference answers could be used for comparing workers' performance. Specifically, if a worker's performance was significantly lower than the average performance of all workers, we blocked the worker by granting the qualification of poor quality. In practice, workers were able to correctly answer about 50% of all queries. We blocked workers if their average accuracy was lower than 20%, and then republished their HIT assignments. Overall, 2,257 crowdworkers have participated in our task, and 51 of them have been granted the qualification of poor quality.
Train / Dev. / Test Splits Among all the 244k triples collected from the third stage, we first obtained one worker answer for each triple. Compared to the reference answers, workers correctly answered queries in 122k triples. We then selected around 100k correctly-answered triples as the training set, restricting the origins of these triples to the news articles used in BIBREF0 . As for the development and test sets, we solicited another worker answer to further ensure their quality. Therefore, each of the rest 22k triples has been validated by two workers. We only kept 20k triples that were correctly answered by both workers. The origins of these triples are either articles used in BIBREF0 or articles crawled by us (as described in sec:news-curation), with a ratio of 3:7. Finally, we randomly split the 20k triples into development and test sets, with 10k triples for each set. tab:statistics summarizes the statistics of our dataset, ReCoRD.

Question: Which models do they try out?
Answer: DocQA, SAN, QANet, ASReader, LM, Random Guess