Venue: NAACL 2019

# A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation

Massimo Poesio

Queen Mary University

m.poesio@qmul.ac.uk &Jon Chamberlain

University of Essex

jchamb@essex.ac.uk &Udo Kruschwitz

University of Essex

udo@essex.ac.uk &Silviu Paun

Queen Mary University

s.paun@qmul.ac.uk &Alexandra Uma

Queen Mary University

a.n.uma@qmul.ac.uk &Juntao Yu

Queen Mary University

juntao.cn@gmail.com

###### Abstract

We present a corpus of anaphoric information (coreference) crowdsourced through a game-with-a-purpose. The corpus, containing annotations for about 108,000 markables, is one of the largest corpora for coreference for English, and one of the largest crowdsourced NLP corpora, but its main feature is the large number of judgments per markable: 20 on average, and over 2.2M in total. This characteristic makes the corpus a unique resource for the study of disagreements on anaphoric interpretation. A second distinctive feature is its rich annotation scheme, covering singletons, expletives, and split-antecedent plurals. Finally, the corpus also comes with labels inferred using a recently proposed probabilistic model of annotation for coreference. The labels are of high quality and make it possible to successfully train a state of the art coreference resolver, including training on singletons and non-referring expressions. The annotation model can also result in more than one label, or no label, being proposed for a markable, thus serving as a baseline method for automatically identifying ambiguous markables. A preliminary analysis of the results is presented.

## 1 Introduction

A number of datasets for anaphora resolution / coreference now exist [6], including ontontotes that has been the de facto standard since the conll shared tasks in 2011 and 2012 [35], and the just introduced and very substantial PreCo corpus [4]. None of these datasets however take into account the research challenging the idea that a 'gold standard' interpretation can be obtained through adjudication, in particular for anaphora [7, 9, 10]. Virtually every project devoted to large-scale annotation of discourse or semantic phenomena has reached the conclusion that genuine disagreements are widespread. This has long been known for anaphora [7, 11, 12] (see also the analysis of disagreements in ontontotes in [35]) and wordsenses [10], but more recent work has provided evidence that disagreements are frequent for virtually every aspect of language interpretation, not just in subjective tasks such as sentiment analysis [11], but even in the case of tasks such as part-of-speech tagging [36]. In fact, researchers in the CrowdTruth project view disagreement as positive, arguing that "disagreement is signal, not noise" [1]. In this paper we present what to our knowledge is the largest corpus containing alternative anaphoric judgments: 20.6 judgments per markable on average (up to 90 judgments in some cases) for about 108,000 markables. We are not aware of any comparable resource for studying disagreement and ambiguity in anaphora or indeed any other area of nlp. We present some preliminary analysis in the paper.

The corpus presented in this paper is also the largest corpus for anaphora / coreference entirely created through crowdsourcing, and one of the largest corpus of coreference information for English in terms of markables. So far, only fairly small coreference corpora have been created using crowdsourcing [12, 13]. The corpus presented here provides annotations for about 108,000 markables, 55% of the number of markables in ontontotes. Another novelty is that the corpus was created through a 'quasi' Game-With-A-Purpose (gwap) [14, 15], _Phrase Detectives_[7], and is, to our knowledge, the largest gwap-created corpus for nlp. (So far, the success of GWAS in other areas of science [31],

2011; Cooper et al., 2010) has not been replicated in nlp.) Finally, the corpus is notable for a richer annotation scheme than the other large coreference corpora. Singletons were marked as well as mentions participating in coreference chains (the omission of singletons being one of the main problems with ontonotes). Non-referring expressions were also annotated: both expletives (not annotated either in ontonotes or PreCo) and predicative NPs. Finally, all types of plurals were annotated, including also **split-antecedent** plurals as in _John met with Mary, and they went to dinner_, which again are not annotated either in ontonotes or PreCo.

Turning a crowdsourced corpus into a high-quality dataset suitable to train and evaluate nlp systems requires, however, an aggregation method appropriate to the data and capable of achieving sufficient quality, something that simple majority voting typically cannot guarantee Dawid and Skene (1979); Hovy et al. (2013). What made it possible to extract such a dataset from the collected judgments was the recent development of a probabilistic method for aggregating coreference annotations called mpaPaun et al. (2018b). mqa extracts silver labels from a coreference annotation and associates them with a probability, allowing for multiple labels in cases of ambiguity. As far as we know, ours is the first use of mpa to create a large-scale dataset. We show in the paper that mpa can be used to extract from the judgments a high quality coreference dataset that can be used to develop standard coreference resolvers, as well as to investigate disagreements on anaphora.

## 2 Background

### Datasets for Anaphora/Coreference

Since the two conll shared tasks Pradhan et al. (2012), ontonotes has become the dominant resource for anaphora resolution research Fernandes et al. (2014); Bjorkelund and Kuhn (2014); Martschat and Strube (2015); Clark and Manning (2015, 2016a, b); Lee et al. (2017, 2018). ontonotes contains documents in three languages, Arabic (300K tokens), Chinese (950K) and English (1.6M), from several genres but predominantly news. One frequently discussed limitation of ontonotes is the absence of singletons De Marneffe et al. (2015); Chen et al. (2018), which makes it harder to train models for mention detection Poesio et al. (2018). Another limitation is that expletives are not annotated. As a consequence, downstream applications such as machine translation Guillou and Hardmeier (2016) that require pronoun interpretation have to adopt various workarounds. Because of these two restrictions, ontonotes only has 195K markables, and a low markable density (0.12 markable/token).

A number of smaller corpora provide linguistically richer information Poesio et al. (2016). Examples include ancora for Spanish Recasens and Martí (2010), tuba-d/z for German Hinrichs et al. (2005), the Prague Dependency Treebank for Czech and English Nedoluzhko et al. (2009), and arrau for English Uryupina et al. (To Appear). In arrau, for example, singletons and expletives are annotated as well, as are split antecedent plurals, generic coreference, discourse deixis, and bridging references. The arrau corpus is relatively small in terms of tokens (350K), but has a higher markable density than ontonotes (0.29 markable/token), so it has around 100K markables, half the number ofontonotes. arrau was recently used in the crac 2018 shared task Poesio et al. (2018) to evaluate a number of anaphora resolution tasks.

The recently introduced PreCo corpus Chen et al. (2018) is the largest existing coreference corpus, consisting of 35,000 documents for a total of 12.5M tokens and 3.8M markables, half of which are singletons. However, the corpus is not intended as a general purpose dataset as only the 3000 most common English words appear in the documents (the majority - 2/3 - of the documents are from Chinese high-school English tests). The corpus's annotation scheme mainly follows the ontonotes guidelines, with a few important differences: singleton mentions and generic coreference are annotated, event anaphora is not, and predicative NPS are annotated as co-referring with their argument, as previously done in the mucGrishman and Sundheim (1995); Chinchor (1998) and aceDoddington et al. (2004) corpora.1 As one could expect, the corpus is relatively easy for coreference systems. The Peters et al. (2018) system trained and tested on PreCo achieves an av 

erage conll score of 81.5%, whereas the same system trained and tested on ontonotes only achieves a score of 70.4%.

### Crowdsourcing and GWAPs for NLP

A revolution in the way language annotation tasks are carried out was achieved by **crowdsourcing**Howe, 2008; Snow et al., 2008). Crowdsourcing comes in many forms, including **citizen science** and **microworking**. A third approach is to use a **game-with-a-purpose (GWAP)** to aggregate data from non-expert players for collective decisions similar to those from an expert (von Ahn, 2006). The game-based approach to collecting language data is initially costly, but once a game is deployed it can continue to collect data with very little financial support, especially if there is an active community. gwaps such as _Phrase Detectives_Poesio et al., 2013), _JeuxDesMots_Joubert and Lafourcade, 2008) and _Zombie Lingo_Fort et al., 2014) have been used in nlp to collect data on specific linguistic features; broader platforms such as Wordrobe (Venhuizen et al., 2013) to gamify the entire text annotation pipeline.

Crowdsourcing is the most realistic approach to collect a large number of judgments about phenomena such as anaphora. Games in particular are the one type of crowdsourcing scalable to the goal of, for example, a 100M word corpus. So far, however, only small and medium scale resources for nlp have been created via crowdsourcing. For coreference we are only aware of two, both around 50K tokens in size Chamberlain et al., Guha et al., 2015). The Groningen Meaning Bank being collected through the Wordrobe platform (Bos et al., 2017) includes many more documents, but so far only very few interpretations have been obtained through the games (e.g., only around 4K judgments have been collected for anaphora).

### Collecting Multiple Judgments

In most of the best known efforts at creating anaphoric corpora for English and other languages substantial disagreements between the coders were observed, but none of the resulting resources contains multiple anaphoric interpretations. Systematic analyses of the disagreements among coders observed in such annotation efforts were provided for ancora by Recasens et al. (2011) and for tuba-d/z by Versley (2008). The entire ontonotes corpus was double annotated, finding disagreements on around 20% of the markables, i.e., around 40,000 cases. An analysis of such disagreements can be found in Pradhan et al. (2012), but ultimately only the result of adjudication was included in the corpus. Most of the PreCo corpus was doubly annotated and the results adjudicated, but only the result of adjudication is released.

We are aware of only two corpus annotation schemes which explicitly allowed the annotation of anaphoric ambiguity: arrau and the Potsdam Commentary Corpus Krasavina and Chiarcos, 2007). Most of the arrau corpus was single-annotated by a highly experienced annotator, who was allowed to mark a variety of cases of ambiguity Poesio and Artstein (2005b). It is known, however, that such explicit marking of ambiguity is difficult Poesio and Artstein (2005b); Recasens et al., 2012), and indeed not many cases of ambiguity were marked in this way in arrau.

## 3 Collecting the Judgments

In this Section we discuss what type of judgments were collected, and how.

### A gamified approach

The gamified online platform _Phrase Detectives2_Chamberlain et al., 2008; Poesio et al., 2013) was used to collect the judgments about anaphoric reference included in the corpus. _Phrase Detectives_ is articulated around a number of tasks centered around the detective metaphor and uses scoring, progression and a variety of other mechanisms to make the activity enjoyable. In **annotation mode** (_Name the Culprit_), the participant provides an anaphoric judgment about a highlighted markable (the possible judgments according to the annotation scheme are discussed next). If different participants enter different interpretations for a markable then each interpretation is presented to other participants in **validation mode** (_Detectives Conference_), in which the participants have to agree or disagree with the interpretation.

Footnote 2: [http://www.phrasedetectives.com](http://www.phrasedetectives.com)

One of the key differences between _Phrase Detectives_ and gwaps such as those developed by von Ahn and his lab von Ahn (2006) is the much greater complexity of judgments required. Yet clearly we cannot expect participants to be experts about anaphora, or to be willing to read a manual explaining the annotation scheme, so all the training still has to be done while playing the game.

 

Therefore, we developed a number of mechanisms that could help in this respect: giving suggestions and tips (global, contextual and faq), comparing decisions with the gold standard, and showing agreement with other players in Validation Mode. When participants begin to play they are shown training texts (in which the answer is known from a gold standard) and get feedback as to whether their decisions agree with the gold standard. Once the player has completed all training tasks they are given a user rating (the percentage of correct decisions out of the total number of training tasks).

As of 17th of March 2019, 60,822 individuals have participated in _Phrase Detectives_ over ten years and using different platforms, providing over 4.26 million judgments, about half of which are included in the present release.

### Types of Judgments

The judgments asked to the participants to _Phrase Detectives_ follow a simplified version of the arrau annotation scheme, which is the result of extensive tests for intercoder agreement [35]. The participants are asked to make two basic distinctions: whether a markable is referring or not, and if referring, whether it is **Discourse-Old** (do), i.e., it refers to an entity already mentioned (in which case the players were asked to indicate the latest mention of that entity), or **Discourse-New** (dn), i.e., it introduces a new entity in the discourse. Anaphoric reference marked include **split antecedent** anaphora, as in _John met Mary, and they went out for drinks_, where the antecedent for _they_ is the set consisting of the separately introduced _John_ and _Mary_. Two types of non-referring expressions were marked: **expletives**, as in _It's five o'clock_ or _There was a fireplace in the room_; and **predicative NPs**, as in _The temperature is 24 degrees_. In the case of predicative nps, players were asked to mark the nearest mention of the entity that the predication applied to, following in this case the ontonotes approach instead of arrau's.

The key difference between this corpus and any other existing corpus for anaphora / coreference with the exception of arrau is that the corpus was designed to collect information about disagreement. The main difference from arrau is that no attempt was made to ask players to identify ambiguity, as that has proven hard or impossible to do [35]. Instead of **explicit (marking of) ambiguity**, the developers relied on **implicit ambiguity**: that genuine ambiguity would emerge if enough players supplied judgments. All the judgments produced by the players were therefore stored, without attempting to choose among them at collection.

The differences between the four corpora being compared are summarized in Table 1, modelled on a similar table in [31]. In the _Phrase Detectives_ corpus predication and coreference are clearly distinguished, as in ontonotes and arrau but unlike in PreCo. Singletons are considered markables. Expletives and split antecedent plurals are marked, unlike in either ontonotes or PreCo. Most importantly, ambiguity of anaphoric interpretation (as in the example from the trains corpus [35]) is marked, but implicitly, i.e., by asking the judgment of at least 8 players per markable, as opposed to explicitly, as attempted in arrau (with little success).

### Markable identification

Following standard practice in anaphoric annotation and gwaps, the markables to be annotated were not identified by the participants themselves; instead, markable identification was carried out semi-automatically. Each document would first be processed by a pipeline combining off-the-shelf tools (sentence splitting and tokenization using the OpenNLP pipeline3 and parsing using the Berkeley Parser [36]) and custom preprocessing and post-processing heuristic steps to correct the output. [See [35] for more details about the pipeline and its performance.) Then one of the administrators would carry out a quick check of the document removing the most obvious mistakes before uploading it. After the document was uploaded, participants could report markable errors, which would then be corrected by hand.4

Footnote 3: [http://opennlp.apache.org](http://opennlp.apache.org)

Footnote 4: As participants report over 10,000 errors per year, it became quickly apparent that carrying out the corrections ourselves was unfeasible. In subsequent work, we developed a gamified approach to markable identification and correction centered around the TileAttack! GWAP [10].

## 4 The corpus

### Basic statistics

This second release of the _Phrase Detectives_ corpus consists of a total of 542 documents contain 

ing 408K tokens and 108K markables from two main genres: Wikipedia articles and fiction from the Gutenberg collection. This corpus is divided in two subsets. The subset we refer to as PDsilver consists of 497 documents, for a total of 384K tokens and 101K markables, whose annotation was completed-i.e. 8 judgments per markable were collected, and 4 validations per interpretation-as of 12th of October 2018. In these documents, an aggregated ('silver') label obtained through mpa (see next Section) is also provided. 45 additional documents were also gold-annotated by two experts annotators. We refer to the subset of the corpus for which both gold and silver annotations are available as PDgold, as it is intended to be used as test set.5 The gold subset consists of a total of 23K tokens and 6K markables. The contents of the corpus are summarized in Table 2.

Footnote 5: PD_goljs the dataset released in 2016 as _Phrase Detectives_ corpus, Release 1 (Chamberlain et al.).

By comparison, the English corpus used for the conll 2011 and 2012 shared tasks consists of 3493 documents, for a total of 1.6M tokens and 194480 markables. In other words, although the current release of the corpus is only about 25% of the conll corpus in terms of tokens, it is 55.5% of its size in terms of annotated markables, i.e., actual training / testing items.

### Number of Judgments

In total, 2,235,664 judgments from 1958 players are included in the current release, of which 1,358,559 annotations and 867,844 validations. On average, 20.6 judgments were collected per markable: 12.6 annotations and 8 validations. In addition, around 10K expert judgments were collected for the gold portion of the corpus from two expert annotators. This compares with 600K estimated judgments for the entire ontonotes corpus, about 3 per markable (total number of annotators not known), and around 10M for PreCo, also 3 per markable, from about 80 annotators.

### Disagreement: a preliminary analysis

The 'raw' statistics about disagreement in the corpus are shown in Table 3. In total, only 35.7% of the markables in the corpus (38,579) were assigned only one interpretation by the participants, whereas 64.3% received more than one interpretation. This figure would seem to suggest massive ambiguity, but we are not saying that 64.3% of markables in the corpus are ambiguous. As already pointed out e.g. in Pradhan et al. (2012),

\begin{table}
\begin{tabular}{l l c r r r} \hline \hline
**Type** & **Example** & \multicolumn{1}{c}{ontontones} & \multicolumn{1}{c}{PreCo} & \multicolumn{1}{c}{arrau} & Present corpus \\ \hline predicative nps & [John] is a teacher & Pred & Coref & Pred & Pred \\  & [John, a teacher] & & No & Yes & Yes & Yes \\ singletons & & No & No & Yes & Yes \\ expletives & It’s five o‘clock & No & No & Yes & Yes \\ split antecedent plurals & [John] met [Mary] & No & No & Yes & Yes \\  & and they ... & & & & \\ generic mentions & [Parents] are usually busy. & Only with & Yes & Yes & Yes \\  & Parents should get involved & pronouns & & & \\ event anaphora & Sales [greyn] 10\%. & Yes & No & Yes & No \\  & This growth is exciting & & & & \\ ambiguity & Hook up [the engine] & No & No & Explicit & Implicit \\  & to [the boxcar] & & & & \\  & and send it to Avon & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison between the annotation schemes in ontonotes, PreCo, arrau and the present corpus

\begin{table}
\begin{tabular}{r p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline  & & Docs & Tokens & Markables & & & & \\ \multirow{4}{*}{PDgold} & Gutenberg & 5 & 7536 & 1947 (1392) & & & & \\  & Wikipedia & 35 & 15287 & 3957 (1355) & & & \\  & GNOME & 5 & 989 & 274 (96) & & & \\  & Subtotal & 45 & 23812 & 6178 (2843) & & & \\ \hline \multirow{4}{*}{PDisher} & Gutenberg & 145 & 158739 & 41989 (26364) & & & & \\  & Wikipedia & 350 & 218308 & 57678 (19444) & & & \\ \cline{1-1}  & Other & 2 & 7294 & 2126 (1339) & & & \\ \cline{1-1}  & Subtotal & 497 & 384341 & 101793 (47147) & & & \\ \hline All & Total & 542 & 408153 & 107971 (49990) & & & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Summary of the contents of the current release. The numbers in parentheses indicate the total number of markables that are non-singletons.

 

there are a number of reasons for disagreements among coders / players apart from ambiguity. In the case of ontonotes, the causes for the  20,000 observed disagreements include:

* Ambiguity proper, i.e., unclear interpretation ('Genuine Ambiguity' in Pradhan et al. (2012)) and/or disagreement on reference (31% of the disagreements in ontonotes, around 7% of all markables);
* Annotator error (another 25% of the cases of disagreement in ontonotes);
* Various limitations of the coding scheme: unclarity in the guidelines, inability to mark certain types of coreference e.g., between generics, etc. (36.5% of the cases of disagreement in ontonotes).
* Interface limitations (around 7.5% of the disagreements in ontonotes).

Some of the disagreements due to other causes- and in particular annotation errors-can be filtered through validation, i.e., by excluding those interpretations of a markable for which the **validation score** (annotations + agreements - disagreements) falls below a threshold. For example, if only interpretations with a validation score \(>0\) are considered, we find that 51,075 / 107,971 markables have at least two such interpretations, i.e., 47.3% of the total, which is considerably less than the 64.3% of markables with more than one interpretation, but it's still a large number.

We will discuss a more sophisticated method for automatically identifying plausible interpretations, as well as the results of a preliminary hand-analysis of the disagreements in a few documents in our corpus, in Section 7.

## 5 Aggregation

### Probabilistic Aggregation Methods

The data collected via _Phrase Detectives_ require an aggregation method to help choose between the different interpretations provided by the players. Simple heuristics such as majority voting are known to underperform compared to probabilistic models of annotation Whitehill et al. (2009); Raykar et al. (2010); Quoc Viet Hung et al. (2013); Sheshadri Lease (2013); Hovy et al. (2013); Passoneau and Carpenter (2014); Paun et al. (2018a). The models offer a rich framework of interpretation and can employ distinct prior and likelihood structures (pooled, unpooled, and partially pooled) and a diverse set of effects (annotator ability, item difficulty, or a subtractive relationship between the two). However, most work on models of annotation assumes that the set of classes the annotators can choose from is fixed across the annotated items, which is not the case for anaphoric annotation. More specifically, in _Phrase Detectives_ the participants can classify a markable as non-referring (expletive or predicative); as introducing a new discourse entity; or as discourseold, in which case they link it to the most recent mention of its antecedent-and coreference chains are document-specific and not fixed in number (see Section 3.2 for more details on the annotation scheme). Recently, however, Paun et al. (2018b) developed a probabilistic model (mpa) able to aggregate such crowdsourced anaphoric annotations.

### Mpa

In mpa, the term **label** is used to refer to a specific interpretation provided by a player, and the term **class** to refer to general interpretation categories such as discourse old, discourse new, expletive, or predicative np. Please note that under this formalism each label belongs to a class: the antecedents belong to the discourse old category, while the other possible labels (e.g., discourse new) coincide with the classes they belong to. The model assumes a preprocessing step in which the markable-level annotations are transformed into a series of binary decisions with respect to each candidate label. mpa then models these (label-level) decisions as the result of the sensitivity (the true positive rate) and specificity (the true negative rate) of the annotators which it assumes are class dependent. This latter assumption allows inferring different levels of annotator ability for each class (thus capturing, for instance, the fact that whereas most participants are generally able to recognize discourse-new mentions, they are much less good at identifying correct antecedents).

### Aggregating the game data

We use the mpa model as a component in a standard mention-pair framework to extract coreference clusters: 1) link each markable with the most likely label as identified by the model, and 2) follow the link structure to build the coreference 

chains. We next evaluate both of these components against expert annotations.

Table 4 shows a per class evaluation of the aggregated interpretations from the pdgold subset. The results indicate an overall better agreement with the expert annotations of mpa compared with a simple majority voting (majvote) baseline. This is because majvote makes the an implicit assumption that the annotators have equal expertise, which is not true in general even with data crowdsourced on microwoking platforms, and even more so with data collected through gwaps[3].

After inferring the mention pairs, coreference chains can be extracted and their quality assessed using standard coreference metrics. Table 5 presents the evaluation against gold chains in pdgold. We compare the chains produced from the mention pairs inferred by mpa and by majvote, and the chains produced by the stanford deterministic coreference system [12] (for which we switched off post-processing to output singleton clusters). The results indicate a far better quality of the chains produced using mpa over the alternative methods. Another interesting result is that even a simple majvote baseline based on crowdsourced annotations performed far better than the stanford system, underlining the advantage of crowdsourced annotations for coreference over automatically produced annotations.

## 6 Using the corpus for coreference resolution

Some nlp researchers may question the usefulness of the information about disagreements for coreference resolution (or other llp tasks). In this Section, we demonstrate that even those purely interested in conll-style coreference resolution can use the _Phrase Detectives_ corpus aggregated with mpa as a dataset. We use pbsilver to train a coreference system able to simultaneously identify non-referring expression and build coreference chains (including singletons). As no other system of this type exists at the moment, we developed one ourselves.

### Our system

The system trained and tested on the corpus is a cluster ranking system that does mention detection and corefence resolution jointly. The system uses the mention representation from the state-of-the-art [12] system, but replaces their mention-ranking model with a cluster ranking model. Our cluster ranking model forms clusters by going through the candidate mentions in their text order and adding them to the clusters, which take into consideration the relative importance of the mentions. An attention mechanism is used to assign mentions within the clusters salience scores, and the clusters are represented as the weighted sums of the mention representations. Separate classifiers are used to identify non-referring markables and singletons.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{Method} & \multicolumn{3}{c}{Discourse old class} & \multicolumn{3}{c}{Discourse new class} & \multicolumn{3}{c}{Predictive NPs class} & \multicolumn{3}{c}{Expletives class} & Avg. \\ \cline{2-17}  & P & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 & Accuracy \\ \hline MAJVote & **94.5** & 62.8 & 75.4 & 79.1 & **99.0** & 87.9 & 53.9 & 9.7 & 16.4 & **97.2** & 71.4 & 82.4 & 65.5 & 82.9 \\ MPA & 90.4 & **87.3** & **88.8** & **94.5** & 96.0 & **95.3** & **64.0** & **72.4** & **68.0** & 94.1 & **98.0** & **96.0** & **87.0** & **92.2** \\ \hline \hline \end{tabular}
\end{table}
Table 4: A per class evaluation of aggregated interpretations against expert annotations.

\begin{table}
\begin{tabular}{l l|c c c c c c c c c c} \hline \hline \multirow{2}{*}{Method} & \multirow{2}{*}{Method} & MUC & \multicolumn{3}{c}{BCUB} & \multicolumn{3}{c}{CEAFE} & Avg. \\ \cline{2-11}  & p & R & F1 & P & R & F1 & P & R & F1 & P & R & F1 \\ \hline \multirow{2}{*}{Singletons included} & majvote & **96.0** & 63.9 & 76.7 & **95.7** & 78.7 & 86.4 & 77.1 & **94.9** & 85.1 & 82.7 \\  & mpa & 91.6 & **82.4** & **86.8** & 94.8 & **87.8** & **91.2** & **92.4** & 93.8 & **93.1** & **90.3** \\  & stanford & 65.4 & 62.4 & 63.8 & 78.9 & 76.1 & 77.5 & 78.4 & 85.2 & 81.7 & 74.3 \\ \hline \multirow{2}{*}{Singletons excluded} & majvote & **96.1** & 64.8 & 77.4 & **93.8** & 45.0 & 60.8 & 66.3 & 48.5 & 56.1 & 64.8 \\  & mpa & 92.2 & **89.2** & **90.7** & 88.1 & **77.8** & **82.6** & **79.5** & **80.2** & **79.8** & **84.4** \\  & stanford & 65.7 & 62.1 & 63.9 & 50.3 & 42.5 & 46.1 & 42.7 & 49.8 & 46.0 & 52.0 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The quality of the coreference chains for the Pdgold subset.

 



### Evaluation Methodology

We randomly chose 1/20 of pdsilver as a development set and use the rest as the training set; PDgold was used as test set.

To get a baseline, we compare the results of our system on a simplified version of the corpus without singletons and expletives with those obtained by the current state-of-the-art system on ontonotes, Lee et al. (2018) trained and tested on the same data.

### Results

Table 6 shows the results of both systems on the simplified corpus. Our cluster ranking system achieved an average conll score of 60.5%, outperforming the Lee et al. (2018) system by 2 percentage points. Note that the Lee et al. (2018) system achieved a higher score on the conll data, which suggests that the present corpus is different from that dataset.

In the same Table, we also report the results obtained by training our system on the full corpus including both non-referring expressions and singletons. This version of system achieves an average conll score of 72.7%.6 We will note that although this score is on system mentions, it is very close to the score (74.3%) achieved by the Stanford deterministic system evaluated with gold mentions (see Table 5 in Section 5). Also, this model trained on the full corpus including singletons achieves a gain of 1 percentage point when compared with the model trained on the simplified corpus even when evaluated in a singleton excluded setting. This indicates that the availability of the singletons is also helpful for resolving nonsingleton clusters. In total, this model achieved a conll score on singletons excluded of 3 percentage points 3% better than our baseline.

Footnote 6: The Extended Coreference Scorer developed for the 2018 crac shared task Poesio et al. (2018) was used to evaluate coreference chains on a corpus using singletons and to assess non-referring expressions identification.

Regarding the task of identifying non-referring mentions, our model achieved a F1 score of 54.6% (see Table 7). The scores of the system on distinct types of non-referring expressions is presented in the following two rows of Table 7. Our model achieved a higher F1 score of 72.3% on expletives, and a lower score (48.7%) on predicative nps.

Overall, these results-the first results on system mentions for Pdgold-suggest that the silver corpus is sufficient to train a state-of-the-art system and achieve a reasonably good performance. Also, that training a model on a corpus enhanced by singletons and non-referring markables results in a better conll score when compared with a model trained on the simplified corpus.

## 7 Disagreements, revisited

In the previous Section we showed that mpa can be used to extract a silver standard out of the annotations that is suitable to train a conll-style coreference resolver or an extended coreference resolver also attempting identification of singletons and non-referring expressions. The key property of the corpus however is the information it provides about disagreements. The second useful contribution of mpa is that it can be used to get an assessment of the ambiguity of markables which is more refined than that discussed in Section 4.3. For each markable, mpa assigns a probability to each interpretation. Given that the model does not assume the existence of a 'gold', there are three possible cases for each markable: either only one interpretation has a probability above a

\begin{table}
\begin{tabular}{l l r r r r r r r r r} \hline \hline \multirow{2}{*}{Singletons} & \multirow{2}{*}{Method} & MUC & \multicolumn{5}{c}{BCUB} & \multicolumn{5}{c}{CEAFE} & Avg. \\ \cline{3-10}  & & P & R & F1 & P & R & F1 & P & R & F1 \\ \hline Included & Our Model & 79.3 & 72.5 & 75.7 & 72.1 & 69.3 & 70.7 & 70.5 & 73.2 & 71.8 & 72.7 \\ \hline \multirow{3}{*}{Excluded} & Our Model & 79.3 & **72.5** & **75.7** & 58.3 & 52.4 & **55.2** & **58.3** & **49.5** & **53.5** & **61.5** \\  & Our Model* & 77.8 & 71.8 & 74.6 & 55.4 & **53.7** & 54.6 & 56.2 & 49.0 & 52.4 & 60.5 \\  & Lee et al. (2018)* & **80.8** & 66.1 & 72.7 & **63.3** & 45.1 & 52.7 & 56.7 & 44.7 & 50.0 & 58.5 \\ \hline \hline \end{tabular}
\end{table}
Table 6: The CoNLL scores for our systems trained on PDsilver and tested on PDgold. * indicates the models trained on the simplified corpus.

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & P & R & F1 \\ \hline Non-referring & 55.2 & 54.0 & 54.6 \\ \hline Expletives & 62.3 & 86.0 & 72.3 \\ Predicative Nps & 49.7 & 47.7 & 48.7 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Non-referring scores for our model

certain threshold-say, 0.5; or more than one interpretation is above that threshold; or none is. This assessment of ambiguity according to mpa is summarized in Table 8.

This assessment appears to suggest a similar prevalence of ambiguity in our corpus than found in ontonotes in the already mentioned analysis by Pradhan et al. (2012). In order to verify this, two experts hand-analyzed 2 documents in pdgold containing a total of 900 markables: _Little Red Cap (LRC)_ and _Robber Bridgeroom_ (RG). Given that each markable has on average 20 interpretations, and that player errors are frequent (there is at least one player error for almost every markable) it wasn't possible to use the same categories as Pradhan et al. Instead, we simply attempted to assign markables to one of the categories: **Genuine ambiguity (GA)**, **Interface or Coding Scheme Problem (ICP)**, **Other (O)**. The results are summarized in Table 9. The Table has one row per document. The first column lists the total number of markables in a document; the second (Dis) the percentage of markables on which there is disagreement; the third (GA) the percentage of the _total_ number of markables which are cases of genuine ambiguity; and the fourth (ICP) the percentage which are cases of Interface or Coding Scheme Problem. As we can see from the Table, 9% of the total number of markables in these documents (80 out of 865) are genuinely ambiguous, i.e., that 12.6% of the disagreements (80 out of 633) are cases of genuine ambiguity. These are only preliminary figures, and we suspect that the ultimate figures on the prevalence of ambiguity are going to be much higher, given that Recasens et al. (2012) report that 12-15% of coreference relations in their corpus are cases of quasi-coreference, and that Poesio and Artstein (2005a) report a figure of 42.6% once ambiguity on discourse deictic reference are taken into account.

We next checked the extent to which mpa can correctly predict genuine ambiguity. The results suggest that mpa is good at removing spurious ambiguity, but as a predictor of ambiguity it only has a recall of around 20% and a precision of slightly under 50%. Improving these results is one of the objectives of our current research.

## 8 Conclusions

We presented a novel resource for anaphora that, because of its annotation scheme and size, at the very least should be useful to those in the community interested in developing systems able to perform a more comprehensive form of anaphora resolution, including for instance expletive detection and split antecedent resolution. The key property of this new resource however is that it provides a large number of judgments about each anaphoric expression, thus enabling the development of systems that do not make the assumption that a 'gold standard' exists, an assumption questioned by all studies associated with the creation of the current resources for the task. The dataset is also to our knowledge the first solid evidence that the gameswith-a-purpose approach can be successfully deployed to obtain substantial resources for nlp.

The corpus is freely available from the Linguistic Data Consortium and from [http://www.dali-ambiguity.org](http://www.dali-ambiguity.org). It is distributed in three formats: an xml format including all the judgments, suitable for analysis of disagreements and/or the development of systems taking disagreement into account; and in conll and crac18 format, with only the gold annotation or the silver label extracted, for those interested in using the corpus as an alternative resource for developing coreference systems only.

### Acknowledgements

This research was supported by the dali project, funded by the European Research Council (erc), Grant agreement ID: 695662.

\begin{table}
\begin{tabular}{c c r r r} \hline  & None & One & Two & Zero or more \\ \hline PDgold & 2.3\% & 93.4\% & 4.3\% & 6.6\% \\ PDhigher & 3.5\% & 94\% & 2.4\% & 5.9\% \\ \hline \end{tabular}
\end{table}
Table 8: Ambiguity in the corpus according to mpa

\begin{table}
\begin{tabular}{c c c c c} \hline  & Total & Dis & GA & ICP \\ \hline LRC & 401 & 79.1\% & 7\% (28) & 7.7\% (31) \\ RG & 464 & 68.3\% & 11.2\% (52) & 12.9\% (60) \\ \hline Average & & 73.7\% & 9.1\% & 10.3\% \\ \hline \end{tabular}
\end{table}
Table 9: Analysis of disagreements in two corpus documents 



## References

* [von Ahn2006] Luis von Ahn. 2006. Games with a purpose. _Computer_, 39(6):92-94.
* Aroyo and Welty (2015) Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. _AI Magazine_, 36(1):15-24.
* Bjorkelund and Kuhn (2014) Anders Bjorkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In _Proceedings of the 52nd ACL_, volume 1, pages 47-57.
* Bos et al. (2017) Johan Bos, Valerio Basile, Kilian Evang, Noortje J. Venhuizen, and Johannes Bjerva. 2017. The groningen meaning bank. In N. Ide and J. Pustejovsky, editors, _The Handbook of Linguistic Annotation_, chapter 18, pages 463-496. Springer.
* Chamberlain et al. (2008) Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. phrase detectives corpus.
* Chamberlain et al. (2008) Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. 2008. Phrase Detectives: A web-based collaborative annotation game. In _Proceedings of I-Semantics 2008_.
* Chen et al. (2018) Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In _Proceedings of EMNLP_, pages 172-181, Brussels, Belgium.
* Chinchor (1998) Nancy Chinchor. 1998. Overview of MUC-7. In _Proceedings of the Seventh Message Understanding Conference (MUC-7)_.
* Clark and Manning (2015) Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In _Proceedings of the ACL_.
* Clark and Manning (2016a) Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In _Proceedings of EMNLP_.
* Clark and Manning (2016b) Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In _Proceedings of the ACL_.
* Clery (2011) Daniel Clery. 2011. Galaxy evolution. Galaxy Zoo volunteers share pain and glory of research. _Science_, 333(6039):173-5.
* Cooper et al. (2010) Seth Cooper, Firsas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popovic, and the Foldit Players. 2010. Predicting protein structures with a multiplayer online game. _Nature_, 466:756-760.
* Dawid and Skene (1979) Alexander P. Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer errorrates using the EM algorithm. _Applied Statistics_, 28(1):20-28.
* De Marneffe et al. (2015) Marie-Catherine De Marneffe, Marta Recasens, and Christopher Potts. 2015. Modeling the lifespan of discourse entities with application to coreference resolution. _Journal of Artificial Intelligence Research_, 52(1):445-475.
* van Deemter and Kibble (2000) Kees van Deemter and Rodger Kibble. 2000. On coreferring: Coreference in MUC and related annotation schemes. _Computational Linguistics_, 26(4):629-637. Squib.
* Doddington et al. (2004) George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In _Proceedings of LREC_, volume 2, page 1.
* Fernandes et al. (2014) Eraldo R. Fernandes, Cicero N. dos Santos, and Ruy L. Milidiu. 2014. Latent trees for coreference resolution. _Computational Linguistics_, 40(4):801-835.
* Fort et al. (2014) Karen Fort, Bruno Guillaume, and H. Chastant. 2014. Creating Zombilingo, a game with a purpose for dependency syntax annotation. In _Proceedings of the 1st International Workshop on Gamification for Information Retrieval (GamifIR'14)_, pages 2-6. ACM.
* Grishman and Sundheim (1995) Ralph Grishman and Beth Sundheim. 1995. Design of the muc-6 evaluation. In _Proceedings of the 6th Conference on Message Understanding_, pages 1-11. Association for Computational Linguistics.
* Guha et al. (2015) Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In _Proceedings of NAACL_.
* Guillou and Hardmeier (2016) Liang Guillou and Christian Hardmeier. 2016. Protest: A test suite for evaluating pronouns in machine translation. In _Proceedings of LREC_, Paris, France.
* Hinrichs et al. (2005) Erhard W. Hinrichs, Sandraubler, and Karin Naumann. 2005. A unified representation for morphological, syntactic, semantic and referential annotations. In _Proc. of the ACL Workshop on Frontiers in Corpus Annotation II: Pie in the Sky_, Ann Arbor, Michigan.
* Hovy et al. (2013) Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In _Proceedings of NAACL_, pages 1120-1130.
* Howe (2008) Jeff Howe. 2008. _Crowdsourcing: Why the power of the crowd is driving the future of business_. Crown Publishing Group.
* Joubert and Lafourcade (2008) Alan Joubert and Mathieu Lafourcade. 2008. Jeavdc-mots : Un prototype ludique pour l'emergence de relations entre termes. In _Proceedings of JADT_.
* Kenyon-Dean et al. (2018) Kian Kenyon-Dean, Eisha Ahmed, Jeremy Georges-Filteau, Christopher Glasz, Barleon Kaur, Augustate Lalande, Shruti Bhanderi, Robert Belfer, Nimal Kanagasabai, Roman Sarrazingendron, Rohit Verma, and Derek Ruths. 2018. Sentiment analysis:

it's complicated! In _Proceedings of NAACL_, pages 1886-1895. ACL.
* Krasavina and Chiaros (2007) Olga Krasavina and Christian Chaircos. 2007. The Potsdam coreference scheme. In _Proceedings of the 1st Linguistic Annotation Workshop_, pages 156-163.
* Lafourcade et al. (2015) Mathieu Lafourcade, Alain Joubert, and Nathalie Le Brun. 2015. _Games with a Purpose (GWAS)_. Wiley.
* Lee et al. (2011) Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford's multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In _Proceedings of CONLL: Shared Task_, pages 28-34, Stroudsburg, PA, USA.
* Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In _Proceedings of EMNLP_.
* Lee et al. (2018) Kenton Lee, Luheng He, and Luke S. Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In _Proceedings of ACL_.
* Madge et al. (2017) Chris Madge, Jon Chamberlain, Udo Kruschwitz, and Massimo Poesio. 2017. Experiment-driven development of a gwap for marking segments in text. In _Proceedings of CHI PLAY_, Amsterdam.
* Martschat and Strube (2015) Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. _Transactions of the Association for Computational Linguistics_, 3:405-418.
* Nedoluzhko et al. (2009) A. Nedoluzhko, J. Mirokvsky, and P. Pajas. 2009. The coding scheme for annotating extended nominal coreference and bridging anaphora in the prague dependency treebank. In _Proceedings of the Linguistic Annotation Workshop_, pages 108-111.
* Passonneau et al. (2012) Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf SallebAouissi, and Nancy Ide. 2012. Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations. _Language Resources and Evaluation_, 46(2):219-252.
* Passonneau and Carpenter (2014) Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. _Transactions of the Association for Computational Linguistics_, 2:311-326.
* Paun et al. (2018a) Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018a. Comparing bayesian models of annotation. _Transactions of the Association for Computational Linguistics_.
* Paun et al. (2018b) Silviau Paun, Jon Chamberlain, Udo Kruschwitz, Juntao Yu, and Massimo Poesio. 2018b. A probabilistic annotation model for crowdsourcing coreference. In _Proceedings of EMNLP_, pages 1926-1937, Brussels, Belgium. Association for Computational Linguistics.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of NAACL_.
* Petrov and Klein (2007) Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In _Proceedings of HLTNAACL_.
* Plank et al. (2014) Barbara Plank, Dirk Hovy, and Anders Sogaard. 2014. Linguistically debatable or just plain wrong? In _Proceedings of EACL_.
* Poesio and Artstein (2005a) Massimo Poesio and Ron Artstein. 2005a. Annotating (anaphoric) ambiguity. In _Proc. of the Corpus Linguistics Conference_, Birmingham.
* Poesio and Artstein (2005b) Massimo Poesio and Ron Artstein. 2005b. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In _Proceedings of the ACL Workshop on Frontiers in Corpus Annotation_, pages 76-83.
* Poesio et al. (2013) Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase Detectives: Utilizing collective intelligence for internet-scale language resource creation. _ACM Transactions on Interactive Intelligent Systems_, 3(1):1-44.
* Poesio et al. (2018) Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roeisger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Ola Vryupina, Juntao Yu, and Heike Zimmester. 2018. Anaphora resolution with the arnva corpus. In _Proc. of the NAACl Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC)_, pages 11-22, New Orleans.
* Poesio et al. (2016) Massimo Poesio, Sameer Pradhan, Marta Recasens, Kepa Rodriguez, and Yannick Versley. 2016. Annotated corpora and annotation tools. In M. Poesio, R. Stuckardt, and Y. Versley, editors, _Anaphora Resolution: Algorithms, Resources and Applications_, chapter 4. Springer.
* Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Urvupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoCNotes. In _Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012)_, Jeju, Korea.
* WISE 2013_, pages 1-15, Berlin, Heidelberg. Springer Berlin Heidelberg.
* Raykar et al. (2010) Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. _Journal of Machine Learning Research_, 11:1297-1322 

* Recasens et al. (2011) Marta Recasens, Ed Hovy, and M. Antonia Marti. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. _Lingua_, 121(6):1138-1152.
* Recasens and Marti (2010) Marta Recasens and M. Antonia Marti. 2010. AnCorAO: Coreferentially annotated corpora for Spanish and Catalan. _Language Resources and Evaluation_, 44(4):315-345.
* Recasens et al. (2012) Marta Recasens, M. Antonia Marti, and Constantin Orasan. 2012. Annotating near-identity from coreference disagreements. In _Proceedings of LREC_.
* Sheshadri and Lease (2013) Aashish Sheshadri and Matthew Lease. 2013. SQUARE: A benchmark for research on computing crowd consensus. In _Proceedings of the 1st AAAI Conference on Human Computation (HCOMP)_, pages 156-164.
* but is it good? Evaluating non-expert annotations for natural language tasks. In _Proceedings of EMNLP_, pages 254-263.
* Uryupina et al. (2013) Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J. Rodriguez, and Massimo Poesio. To Appear. Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU corpus. _Journal of Natural Language Engineering_.
* Venhuizen et al. (2013) Noortje Venhuizen, Valerio Basile, Kilian Evang, and Johan Bos. 2013. Gamification for word sense labeling. In _Proceedings of the 10th International Conference on Computational Semantics (IWCS'13)_.
* Versley (2008) Yannick Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. _Research Language and Computation_, 6:333-353.
* Whitehill et al. (2009) Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In _Advances in Neural Information Processing Systems 22_, pages 2035-2043. Curran Associates, Inc.
* Wong and Lee (2013) Billy T. M. Wong and Sophia Y. M. Lee. 2013. Annotating legitimate disagreement in corpus construction. In _Proceedings of IJCNLP_, pages 51-57, Nagoya, Japan.