Abstract: Machine comprehension of texts longer than a single sentence often requires coreference resolution. However, most current reading comprehension benchmarks do not contain complex coreferential phenomena and hence fail to evaluate the ability of models to resolve coreference. We present a new crowdsourced dataset containing more than 24K span-selection questions that require resolving coreference among entities in over 4.7K English paragraphs from Wikipedia. Obtaining questions focused on such phenomena is challenging, because it is hard to avoid lexical cues that shortcut complex reasoning. We deal with this issue by using a strong baseline model as an adversary in the crowdsourcing loop, which helps crowdworkers avoid writing questions with exploitable surface cues. We show that state-of-the-art reading comprehension models perform significantly worse than humans on this benchmark---the best model performance is 70.5 F1, while the estimated human performance is 93.4 F1.

Introduction: Paragraphs and other longer texts typically make multiple references to the same entities. Tracking these references and resolving coreference is essential for full machine comprehension of these texts. Significant progress has recently been made in reading comprehension research, due to large crowdsourced datasets BIBREF0, BIBREF1, BIBREF2, BIBREF3. However, these datasets focus largely on understanding local predicate-argument structure, with very few questions requiring long-distance entity tracking. Obtaining such questions is hard for two reasons: (1) teaching crowdworkers about coreference is challenging, with even experts disagreeing on its nuances BIBREF4, BIBREF5, BIBREF6, BIBREF7, and (2) even if we can get crowdworkers to target coreference phenomena in their questions, these questions may contain giveaways that let models arrive at the correct answer without performing the desired reasoning (see §SECREF3 for examples). We introduce a new dataset, Quoref , that contains questions requiring coreferential reasoning (see examples in Figure FIGREF1). The questions are derived from paragraphs taken from a diverse set of English Wikipedia articles and are collected using an annotation process (§SECREF2) that deals with the aforementioned issues in the following ways: First, we devise a set of instructions that gets workers to find anaphoric expressions and their referents, asking questions that connect two mentions in a paragraph. These questions mostly revolve around traditional notions of coreference (Figure FIGREF1 Q1), but they can also involve referential phenomena that are more nebulous (Figure FIGREF1 Q3). Second, inspired by BIBREF8, we disallow questions that can be answered by an adversary model (uncased base BERT, BIBREF9, trained on SQuAD 1.1, BIBREF0) running in the background as the workers write questions. This adversary is not particularly skilled at answering questions requiring coreference, but can follow obvious lexical cues—it thus helps workers avoid writing questions that shortcut coreferential reasoning. Quoref contains more than 15K questions whose answers are spans or sets of spans in 3.5K paragraphs from English Wikipedia that can be arrived at by resolving coreference in those paragraphs. We manually analyze a sample of the dataset (§SECREF3) and find that 78% of the questions cannot be answered without resolving coreference. We also show (§SECREF4) that the best system performance is 49.1% $F_1$, while the estimated human performance is 87.2%. These findings indicate that this dataset is an appropriate benchmark for coreference-aware reading comprehension.

Question: What is the strong baseline model used?
Answer: Passage-only heuristic baseline, QANet, QANet+BERT, BERT QA