A Mention-Pair Model of Annotation with Nonparametric User Communities
Authors: Silviu Paun
Juntao Yu
Jon Chamberlain
Udo Kruschwitz
Massimo Poesio
Venue:  (2020)
Keywords: model of annotation
coreference resolution
anaphoric annotation
mention pair model
bayesian nonparametrics

Abstract: The availability of large datasets is essential for progress in coreference and other areas of NLP. Crowdsourcing has proven a viable alternative to expert annotation, offering similar quality for better scalability. However, crowdsourcing require adjudication, and most models of annotation focus on classification tasks where the set of classes is predetermined. This restriction does not apply to anaphoric annotation, where coders relate markables to coreference chains whose number cannot be predefined. This gap was recently covered with the introduction of a mention pair model of anaphoric annotation (MPA). In this work we extend MPA to alleviate the effects of sparsity inherent in some crowdsourcing environments. Specifically, we use a nonparametric partially pooled structure (based on a stick breaking process), fitting jointly with the ability of the annotators hierarchical community profiles. The individual estimates can thus be improved using information about the community when the data is scarce. We show, using a recently published large-scale crowdsourced anaphora dataset, that the proposed model performs better than its unpooled counterpart in conditions of sparsity, and on par when enough observations are available. The model is thus more resilient to different crowdsourcing setups, and, further provides insights into the community of workers. The model is also flexible enough to be used in standard annotation tasks for classification where it registers on par performance with the state of the art.

Under review as a conference paper at ICLR 2020
A MENTION-PAIR MODEL OF ANNOTATION WITH NONPARAMETRIC USER COMMUNITIES
Anonymous authors Paper under double-blind review
ABSTRACT
The availability of large datasets is essential for progress in coreference and other areas of NLP. Crowdsourcing has proven a viable alternative to expert annotation, offering similar quality for better scalability. However, crowdsourcing require adjudication, and most models of annotation focus on classification tasks where the set of classes is predetermined. This restriction does not apply to anaphoric annotation, where coders relate markables to coreference chains whose number cannot be predefined. This gap was recently covered with the introduction of a mention pair model of anaphoric annotation (MPA). In this work we extend MPA to alleviate the effects of sparsity inherent in some crowdsourcing environments. Specifically. we use nonparametrie partially pooled structure (based on a stick breaking process), fitting jointly with the ability of the annotators hierarchical community profiles. The individual estimates can thus be improved using infor- mation about the community when the data is scarce. We show, using a recently published large-scale crowdsourced anaphora dataset, that the proposed model performs better than its unpooled counterpart in conditions of sparsity, and on par when enough observations are available. The model is thus more resilient to dif- ferent crowdsourcing setups, and, further provides insights into the community of workers. The model also flexible enough to be used in standard annotation tasks for classification where it registers on par performance with the state of the art.
INTRODUCTION
Identifying and resolving anaphoric reference to discourse entities, a task known in NLP as coref- erence resolution, has long been considered a core aspect of language interpretation (Peesio al., 2016). Ever since the MUC evaluation campaign in the 1990s (Grishman & Sundheim 1995 Chin- chor 1998), larger and richer datasets have heen made available, pushing the state of the art to new heights. In the last few years, the ONTONOTES corpus (Pradhan et al.) |007 Weischedel et al.)
2011], used for the CONLL 2011 and 2012 shared tasks (Pradhan et al./ 2012), has become the de facto standard resource for coreference resolution research (Fernandes et al., 2014 Björkelund & Kuhn 2014 Martschat & Strube 2015 Clark& & Manning 2015 201 6a bj Lee et al. [017]] (018)]
The corpus hand-annotated by experts, and remained the largest available dataset up until the recent publication of PRECO (Chen et al., 2018). But there are still many languages and domains for which such resources are available, or where the annotation scheme is limited (e.g., the lack of singletons in ONTONOTES, no expletives in either PRECO nor ONTONOTES).
Annotating data on the scale required to train state of the art systems using traditional expert annota- tion can quickly get unaffordable. But in recent years crowdsourcing has proved a viable alternative to expert annotation, with studies indicating that expert-level quality can be achieved with much lower costs (Snow et al., 2008 Raykar et al., 2010). Crowdsourced data however require aggrega- tion methods to choose the most likely label(s) among the interpretations provided by the crowd. Past research suggests probabilistic models of annotation are one of the most promising approaches to aggregation (Dawid& Skene 1979, Carpenter 2008 Whitehill et al. 2009 Raykar et al. 2010] Hovy al.) [2013) Quoe Viet Hung et al. 2013 Sheshadri & Lease 2013 Passonneau & Carpenter, 2014| Venanzi et al. [2014] Kamar et al.| 2015 Paun et al. (018a)). These models offer a rich frame- work of interpretation and employ distinct prior and likelihood structures (pooled, unpooled, and partially pooled) and a diverse set of effects (annotator ability, item difficulty).
Motivation Most work models of annotation assume the set of classes the annotators choose from is fixed across the annotated items, an aspect not appropriate for anaphoric annotation where coders relate markables to anaphoric chains. Recently, Paun et al. (2018b) developed a probabilis-

---

Under review as a conference paper at ICLR 2020
tic model able aggregate crowdsourced anaphoric annotations. The model was later applied to adjudicate the interpretations from the Phrase Detectives 2 corpus with comparable quality to that of expert amnotators (Poesio et al.) 2019). The model of Paun et al. 2018b) assumes an unpooled structure, i.e., it models individual annotator parameters. Such models typically require a larger number of observations properly estimate the ability profile of the coders. In a crowdsourcing en- vironment this requirement may not always be satisfied, c.g., in the intial stages of a crowdsourcing campaign, or in the commonly encountered scenario where the workload of the annotators resem- bles power law curve ([peirotis] 2010)] Chamberlain 2016); both examples describe sparse data environment which may prove difficult to handle for an unpooled model. One intuitive solution to this problem is to exploit the similarities found in the behaviour of the annotators. Simpson et al. 2011 2013) identified, after fitting model of annotation, distinctive clusters the ability of the workers; more generally, typical annotator communities found in a crowdsourcing setup include spammers, adversarial, biased, average or high quality players. Knowledge of these communitics would allow regularize the ability of the annotators towards the profile of the community they are part of. This partially pooled structure can prove effective in conditions of sparsity where there are not enough observations accurately estimate the ability of the annotators in isolation. The level of pooling would be dictated by the data, such that, when enough observations are gathered, the partially pooled and the unpooled models would perform similarly.
Contributions In this work we extend the unpooled mention pair model of annotation (MPA) pro- posed by Paun et al. (2018b) with hierarchical communities of annotators. We let the number of communities grow with the data, a flexibility that we achieve using a Dirichlet process mixture based on stick-breaking representation of the underlying Dirichlet process. We conduct the eval- uation on the Phrase Detectives 2 corpus, in various levels of sparsity, assessing the accuracy of the inferred mention pairs, the quality of the post-hoc constructed silver chains, and the viability of using silver chains an alternative to expert-annotated chains when training state of the art coref- erence system. We discuss the inferred community profiles of a few known spammers and honest players of the game used to collect the Phrase Detectives 2 corpus. We conclude the evaluation with a performance check on traditional crowdsourcing datasets against several state of the art models.
A COMMUNITY MODEL FOR ANAPHORIC ANNOTATIONS
We use the same annotation scheme as Paun et al. 2018b): annotators mark mentions as discourse new if a new entity is being introduced into the discourse (represented internally DN), with prop- erty for predicative noun phrases (PR), with non-referring for expletives (NR) and with discourse old if an already introduced entity is being mentioned, in which case the annotators must also specify the mention's 's most recent antecedent (DO(ante-id)). We also follow the same notation as in Paun al. (2018b) and use the term label refer to given annotation (i.e., DN, DO(ante-id), PR, NR) and the term class to refer to general category a given label belongs to (DN, DO, PR, NR).
2.1 MODEL SPECIFICATION
Similar to MPA, the model we propose - which we will henceforth refer to as COMMUNITYMPA
assumes a pre-processing step in which the mention-level annotations are transformed into series of binary decisions with respect to each (distinct) candidate label. Both models assume a similar generative process of these decisions, although for COMMUNITY MPA we used different parame- terization to later accommodate the hicrarchical structure:
For every mention i {1,2, I):
For every (distinct) candidate label m E 21. M,}:
*
Draw true label indicator Ct. ~ Bern(T
* For every position E 21, J:
-f = then draw decision based on the sensitivity of the annotator yi,m,n ~
Bern (Fr:jfaaa.....mm........
"m is the class of the m-th candidate label for mention i.
2Y() is the standard logistic function.
[] returns the index of the annotator who made the n-th decision the m-th label of mention i.
2

---

Under review as a conference paper at ICLR 2020
Otherwise, draw decision based annotator's specificity ys,m,n Bern(1 T (f
The annotators belong to different communities and have abilities that depend on the class of the mentions they annotate:
For every annotator j E [1.
Draw a community 2j Cat( (A))
For every class {1, ..., K}:
* Draw sensitivity j ~ Normal(Taa..a Tz5.
Draw specificity B9.h Normal(Va, (v,,al (22,,h)
The communities serve hierarchical priors the ability of the annotators regularizing it towards their mean as strongly as evidenced by the data (effect captured by the variance parameters):
For every community E {1,2, oc):
Draw stick proportion A, ~ Beta 6)
For every class (1.,
Draw mean sensitivity Tr.b Normal(do, d1)
*
Draw sensitivity variance ,r Inverse Gamma(eo,es)
* Draw mean specificity Vr, ~ Normal(ta, (1)
* Draw specificity variance 2r,, ~ Inverse Gamma(ug, uz)
Finally, the model completed with conjugate priors:
For every class h E (1,2....,K) K):
Draw class specific true label likelihood Ta Beta(ap, a1)
Draw a scale b Gamma(su, $1)
Both MPA and its extension COMMUNITY MPA function as components in standard mention pair framework: each mention is assigned the most likely candidate label based on the posterior of the label indicators, and the coreference chains are built from the mention pair links.
2.2 PARAMETER ESTIMATION NOTES
We estimate the parameters of the proposed model using variational inference, which is determinis- tic, typically fast, and benefits from clear convergence criterion (Blei et al. (017).
We parameterized the model with conjugacy mind, making sure the complete conditionals are part of the exponential family. In this case the corresponding variational distributions take the same form and have the natural parameters equal to the expected value (under the variational distribu- tion) of the natural parameters of the complete conditionals (Blei & Jordan 2006 Hoffman et al. 2013). Conjugacy was not directly obtained in those cases involving the standard logistic function; addressed that using the bound from Jaakkola & Jordan (2000). Lastly, the stick breaking rep- resentation of the Dirichlet process nicely complies with conjugacy as well (Blei & Jordan 20066. We approximate the infinite mixture of user communities with truncated variational distributions.
The derivations are somewhat standard the machine learning literature and for space constrains are omitted from the main paper but included in Appendix
EVALUATION
We conducted the experiments on the recently made available Phrase Detectives corpus (Poesio et al. 2019) The dataset was annotated in game with a purpose setting, where 98.67% of the
(B) ) is vector of stick lengths, where the length of the r-th stick is o),(A) A, I1 A.r). The different stick lengths represent the prevalence of the communities.
"The length of the stick broken at that proportion gives the prevalence of the community.
TThe scaling parameter affects the growth of the number of communities with the data.
3

---

Under review as a conference paper at ICLR 2020
CommunityMPA
-MA
70
95
60
85
50
40
30
20
55
10
5
Maximum coder workload
Maximum coder workload
(a) Property results (predicative NPs)
(b) Non referring results (expletives)
96
90
89
95
88
94
93
85
92
84
83
90
5
all
40
all
Maximum coder workload
Maximum coder workload
(c) Discourse new results
(d) Discourse old results
Figure 1: A per class evaluation of the inferred mention pairs matched against expert annotations.
The "all" configuration uses the dataset it is (does not alter the player workloads).
collected judgements come from players who produced more than 40 annotations cach. To simulate more sparse evaluation environment break the larger player workloads into smaller batches which we further assume each was produced by different player such that the workload of the players does not exceed fixed threshold. Under this procedure the annotations are kept unchanged, offering larger confidence when assessing whether the differences in performance between the partially-pooled COMMUNITYMPA model and its unpooled counterpart MPA come from the number of annotator observations that is required for each of these models properly estimate their ability.
3.1 MENTION PAIR ACCURACY
This subsection presents the results for the agreement between the inferred mention pairs and the gold pairs. Both COMMUNITY MPAM and MPA models were trained on the full Phrase Detectives corpus and evaluated on the expert-annn annotated subset. Both model implementations produce posterior point estimates for each candidate interpretation; we assign each mention the interpretation with the most mass under the posterior.
Figure I shows the results obtained by the models, for each class, under different workload con- figurations (i.c., maximum player workload), The trend lines clearly indicate a better performance of COMMUNIYMMA., across all classes. The gap in performance between the two models is the largest when the maximum number of observations (annotations) a player can have is capped at and closes the number of available observations increase. reaching on par performance when the dataset is used as is (the "all" configuration). Remember that when the corpus is used as it is, al- most all the data (98,67%) is produced by players with more than 40 annotations each, having plenty
4

---

Under review as a conference paper at ICLR 2020
CommunityMPA
-PA
86
84
91
82
Average F1
Average F1
90
80
78
88
76
74
87
5
40
all
10
20
40
all
Maximum coder workload
Maximum coder workload
(a) Singletons excluded.
Singletons included.
Figure 2: The quality of silver chains evaluated against gold chains.
of observations for unpooled model like MPA to properly profile their ability. We saw however that MPA suffers in conditions of sparsity which the partially-pooled COMMUNITYMPA model leviates through the hierarchical structure (the annotators' ability is pooled towards the ability of the community they are part of).
3.2 SILVER CHAIN QUALITY
We build the silver coreference chains from the inferred mention pairs by following the link structure inherent in the pairs. We assess the quality the constructed chains by comparing them against gold (expert annotated) chains using standard coreference metrics. For this evaluation we used the scorer introduced by Poesio et al. (2018) to assess the quality of the chains both traditional CONLL style (without singletons) and having singletons included into the evaluation.
Figure shows the average Fl obtained when assessing the silver chains produced by the two mod- els. The trend lines are similar those from the mention pair evaluation: better performance of COMMUNITYMMM MPA that increases with the maximum player workload, registering the largest gap MPA the lowest number of observations considered in the evaluation, and on par performance when the dataset was used as it is. Again, we could see the benefits in low count data brought by the hierarchical structure of COMMUNTYMPP.
3.3 TRAINING ON SILVER CHAINS
In this Section we assess the viability of using silver chains as an alternative to expert- expert-annotated chains when training a state of the art coreference system. We use the system used by Poesio et al. 22119, as other coreference systems cannot classify singletons and non referring expressions (i.e., expletives and predicative NPs), of the unique characteristics of the Phrase Detectives corpus. We used as test data the gold chains that come with the corpus, while for training and development used silver chains constructed posteriori from aggregated mention pairs. We report the results obtained using the scorer introduced in the previous section which can handle both singletons and non-referring expressions addition to the traditional CoNLL evaluation.
The results, presented in Figure 3 paint a similar picture to those from the previous evaluation sections: better performance of the MOMMUNITYMPA model in conditions sparsity, with MPA closing the gap as more observations are allowed for each coder.
3.4 INFERRED USER COMMUNITIES
The Phrase Detectives 2 dataset also includes anonymized list of spammers and one with honest, well-established players from the game the corpus was collected with. We use these two lists to introduce and discuss of the inferred communities these players were assigned to.
5

---

Under review as a conference paper at ICLR 2020
CommunityMPA
MPA
62
73
59
43
38
5 53
56
23
55
Maxmmum coder wocklood
Maximum coder wocklood
Meximum coder weeklood
(a) Singletons excluded.
(b) Singletons included.
Predicative NPs and expletives.
Figure 3: Results of a state of the art coreference system trained on silver chains.
Specificity
Sensitivity
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.5
0.5
0.3
0.2
0.1
0.1
DN
PR
DO
NR
DN
PR
DO
NR
(a) "DN spammers" (4.5%)
(b) "Top players" (14.1%)
0.9
0.9
0.8
0.8
0.7
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
DN
DO
NR
DN
PR
DO
NR
(c) "Average players" (57.8%)
(d) "DN biased players" (9%)
Figure 4: Examples of inferred user communities from the Phrase Detectives 2 corpus. The horizon- tal line refers to the average ability for DN discourse new interpretations, PR property (predicative NPs), DO discourse old and NR non referring (expletives). Each community is given an intuitive name and is presented together with its prevalence.
Figure presents the community profiles the players from the aforementioned lists belong to. In- terestingly, all the spammers were assigned to the community from Figure 4a The model assigns almost full probability mass to the average specificity of this community and negligible mass to the sensitivity for all classes with the exception DN, where this mass assignment is reversed. For that objectively be the case, the players under this community would have to (almost) always

---

Under review as a conference paper at ICLR 2020
Bluebird
RTE
Valence
Temp
Majority vote
75,93 (82/108)
91.88 (735/800)
80.00 (80/100)
93.94 (434/462)
iBCC
89.81 (97/108)
92.88 (743/800)
85.00 (85/100)
94.35 (436/462)
cBCC
88.89 (96/108)
93.12 (745/800)
88.00 (88/100)
94.37 (436/462)
heBCC
88.89 (96/108)
93.12 (745/800)
89.00 (89/100)
94.37 (436/462)
MPA
88.89 (96/108)
93.00 (744/800)
86.00 (86/100)
94.16 (435/462)
COMMUNITYMPA
89.81 (97/108)
93.25 (746/800)
85.00 (85/100)
94.16 (435/462)
Table 1: Accuracy (correctly adjudicated items / total number of items) results on traditional crowd-
sourcing datasets. The results of the first 4 methods are as reported in (Moreno et al. 2013).
choose the DN interpretation, an aspect confirmed by an inspection of their annotations. The an- notators from the list of honest players, have more diverse ability profiles. Most of them belong to the pool of "Top players" presented in Figure 4b) The players from this community have solid understanding of anaphoric annotation and generally a large workload: for example the 10 people from the supplied list that the model assigned to this community have provided 35% of the entire corpus annotations. Figure [guure/ccshowe shows the profile of the average players, named so based on the large prevalence of this community in the population of annotators. Looking their sensitivity, the esti- mates confirm the general intuition that PR cases (predicative NPs) are the hardest to spot, mostly because they easily be confused with DN (discourse new); indefinite NPs (e.g., a policeman) are the most common type of mention in both classes. Expletives (NR) and the introduction of new entities into the discourse (DN) are the easiest understand by most people. Another intuitive re- sult is that discourse old mentions are more difficult compared with discourse new ones; the former requires the identification of the most recent antecedent, which could go wrong for various reasons (e.g., ambiguity, negligence). Finally, we could also find in the supplied list players who are biased towards DN, but make the effort occasionally supply other interpretations as well (see Figure 4d)
3.5 TRADITIONAL CROWDSOURCING TASKS
Both COMMUNITYMPA and its unpooled counterpart MPA can be applied to traditional crowdsoure- ing datasets where the set of classes the coders choose from is the same across the annotated items. Under the modeling framework described in this work (Section for this type of data, the labels coincide with the classes they belong to. We compare the aforementioned models against the methods from Moreno et al. (2015) which include a majority vote baseline, the iBCC model of Kim & Ghahramani (2012) which is Bayesian version of the model presented in the seminar work Dawid & Skene 1979), and two nonparametric community models: the cBCC model where annotators are assigned into communities and annotate according to the profile of the community they belong to, and the hBCC model which assumes both annotator and community level structures. The latter model is a nonparametric extention of the Dawid & Skene COMMUNITY MPA is to MPA.
1979) model, similarly how
The evaluation was conducted on 4 datasets commonly used in the crowdsourcing literature (Moreno et al. [2015] Raykar& & Yu 2012) Snow et al.] 2008 Hovy et al., 2013: Paun et al.) 2018a). Table I presents the accuracy results, indicating on par performance between the probabilistic models and a slight advantage of these models against the majority vote baseline. The simple majority vote baseline implicitly assumes an equal expertise among annotators, a well known shortcoming, previous studies reporting more significant differences in performance to the probabilistic models of annotation, on larger datasets (Venanzi et al. 2014 Paun et al.) 2018a).
4 RELATED WORK
To our knowledge, the model presented in this work and its unpooled counterpart from Paun et al. (2018b) are the first models designed for anaphoric annotation. Both models however draw in- spiration from the mention-pair model of coreference (Soon et al., 2001 Hoste 2016) and from traditional models of annotation.
7

---

Under review as a conference paper at ICLR 2020
The model we introduced in this work functions as component in rather abstract mention mention--piir framework: the mention pairs are inferred in an unsupervised fashion from the crowd annotations and the clustering to coreference chains is done by simply following the unique link structure from the inferred pairs.
Traditional models of annotation assume the set of classes the annotators can choose from is fixed across the annotated items, an aspect not appropriate for an anaphoric annotation task. However, the model we introduced is after all model of annotation, and thus shares many of the characteristics found in this area of research. Back in the late 70's, in a seminal work, Dawid & Skene (1979) introduced a model of annotation that would take into account in its inference for the correct in- terpretations the ability of the annotators. The model has found wide application and inspired the state of the art through the years (Smyth et al. 1995 Albert & Dodd 22044 Whitehill et al., 2009, Raykar et al. 2010, Kim & Ghahramani, 2012 Hovy et al. 2013] Simpssinspnnn et al. (0133] Passonnes
& Carpenter 2014, Felt al. [0144] 2015| (000155KKama Kamar et [015| Venanzi et al. 2014] Moreno al,,
2015 Nguyen et al. 2017 Paun et al. 2018a| inter alia). Two of the models proposed in recent years assume a partially pooled structure, having the annotators clustered into communities. This type of structure uses information about the communities to improve the estimates of the individ- uals by regularizing towards the community mean. One of the models assumes a fixed number of clusters (Venanzi al. 2014), while the other takes a nonparametric approach letting the number of clusters grow with the data (Moreno et al. 2013). Both of these models are community extensions of the Dawid & Skene (1979) model. The model we introduced in this work also assumes par- tially pooled structure, but it was built as an extension of the MPA model for anaphoric annotations (Paun et al. 801 Compared to the nonparametric community model Moreno al. (2015), the closest of the two community models to ours, besides the differences in the data these models were designed for, there are also differences in the parameterization and inference. Moreno et al. (2015) use a Chinese restaurant process to allow for a latent number of communities and Gibbs sampling for inference, whereas we use a stick breaking process with variational inference, aiming for conjugacy throughout our parameterization. Since our model can also be used traditional crowdsourcing datasets, we showed back in Section 3.5 that we get comparable performance to the state of the art which included the nonparametric model of Moreno al. (0155)
5 CONCLUSIONS
The development of more powerful and versatile coreference resolution systems relies on the avail- ability of larger and linguistically richer datasets, and crowdsourcing has been identified as a viable alternative to expert annotation, offering comparable quality fraction of the costs and larger scalability. Although the study of models of annotation, necessary to adjudicating crowd labels, re- ceived much attention over the years, it mostly aimed at standard classification tasks where the set of classes the coders can choose from is the same across the annotated items. The literature on models of anaphoric annotation is scarce, the only previous effort in this direction being the recently introduced mention pair model of Paun et al. (2018b).
In this work we extended the unpooled model proposed by Paun al. (0188) with hierarchical communities of annotators, using information about the community to improve the estimates of the individuals. The partially pooled extension is nonparametric, letting the number of communities grow with the data, flexibility we achieved with the help of a Dirichlet process mixture based on a representation of the underlying Dirichlet process. The hierarchical structure offers a better resilience to sparsity, making the model a better fit (compared to its unpooled counterpart) for a larger number of crowdsourcing setups. We demonstrated this across a number of coreference resolution related tasks, in various levels of sparsity, assessing the accuracy of the inferred mention pairs, the quality of the post-hoc constructed silver chains, and the viability of using silver chains as alternative to expert-annotated chains when training state of the art coreference system. We also included a discussion of the inferred community profiles.
The model, although developed for anaphoric annotation, is also flexible enough to be used in tradi- tional crowdsourcing setups where the set of classes the coders can choose from is the same across the annotated items. We showed, in this context, the model is par with the state of the art.
The paper also includes guidance for the estimation of the parameters using variational inference (see appendix) and is accompanied by the code implementing the proposed model.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Paul S. Albert and Lori E. Dodd. A cautionary note on the robustness of latent class models for es- timating diagnostic error without gold standard. Biometrics, 60(2))427-433, 2004. ISSN 1541- 00100000000000004444440
0420. doi:
1111/] 1111/j-
0006 341x. 2004. 00187.x
Anders Björkelund and Jonas Kuhn. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), volume 1, pp. 47-57, 2014.
David M. Blei and Michael I. Jordan. Variational inference for Dirichlet process mixtures. Bayesian Analysis, 1(1):121-143, 03 2006. doi: 10.1214/06-BA10k URL https:
2214/05-BA104
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statis- ticians. Journal of the American Statistical Association, 112(518):559 877, 2017. doi: 10.
(08001622559.29771127777 URL https:
1285773
Bob Carpenter.
Multilevel bayesian
models of categorical data annotation.
Available
http://lingpiper files. wordpress. stw.:pppp...ieeeeeooooorr..ossoo ct00s/s0.co/2000111111 aaap-dareeearrmllllleeeeaaaaaaoo
as
2008.
Jon Chamberlain. Harnessing Collective Intelligence on Social Networks PhD thesis, University of Essex, School of Computer Science and Electronic Engineering, 2016.
Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. Preco: A large-scale dataset
in preschool vocabulary for coreference resolution. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 172-181, Brussels, Belgium, October-
November 2018. Association for Computational Linguistics. URL http: /ttp:///www.... //wwww acl web.
org/antho ogy http//antolooyyDIIII
1016
Nancy A. Chinchor. Overview of MUC-7/MET-2. In Proceedings ofe the 7th Message Understanding Conference (MUC7), April 1998.
Kevin Clark and Christopher D. Manning. Entity-centric coreference resolution with model stack- ing. In Association for Computational Linguistics (ACI.), 2015.
Kevin Clark and Christopher D. Manning. Deep reinforcement learning for mention-ranking coref-
erence models. In Empirical Methods on Natural Language Processing (EMNLP), 2016a.
Kevin Clark and Christopher D. Manning. Improving coreference resolution by learning entity-level distributed representations. In Association for Computational Linguistics (ACL), 2016b.
Alexander P. Dawid and Allan M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Stutistics, 28(1):20-28, 1979.
Paul Felt, Robbie Haertel, Eric K. Ringger, and Kevin D. Seppi. MOMRESP: Bayesian model for multi-annotator document labeling. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2014), Reykjavik, 2014.
Paul Felt, Kevin Black, Eric Ringger, Kevin Seppi, and Robbie Haertel. Early gains matter: A
for preferring generative over discriminative crowdsourcing models. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2015.
Eraldo R. Fernandes, Cícero N. dos Santos, and Ruy L. Milidiú. Latent trees for coreference res- olution. Computational Linguisties, 40(4):801-835, December 2014. ISSN 0891-2017. doi: N,.116//COLLaa00000 URL http:/pppp...a...oooo///........ 1ttp:////....111.1111111
00200
Ralph Grishman and Beth Sundheim. Design of the MUC-6 evaluation. In Proceedings of the 6th
Conference Message Understanding, MUC6 '95, pp. 1-11, Stroudsburg. PA. USA, 1995. As- sociation for Computational Linguistics. ISBN 1-55860-402-2. doi: 10:3115/10723999107720...

---

Under review as a conference paper at ICLR 2020
Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley, Stochastic variational infer- ence. The Journal of Machine Learning Research, 14(1):1303 (111:::1133--44,,MM May 2013. ISSN 1532-4435. BtRh://::/dd acm. org/ citation. wtw.::ccmmccc2202222222222200000
Veronique Hoste. The mention-pair model. In Anaphora Resolution: Algorithms, Resources and Applications, chapter 9, pp. 269- 269-282. Springer, 2016.
Dirk Hovy, Taylor Berg- Kirkpatrick, Ashish Vaswani, and Eduard Hovy. Learning whom to trust
with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp. 1120-1130, 2013.
Panagiotis G. Ipeirotis. Analyzing the amazon mechanical turk marketplace. XRDSS 17(2):16-1,, December 2010. ISSN 4528-4972. doi: 10.1145/186908.. 1869094, URL http::////do acm.
org/10. 1145/ 10..//88696686 1869094
Tommi S.
Jaakkola and Michael Jordan. Bayesian parameter estimation via variational meth- ods. Statistics and Computing, 10(1):25-37, Jan 2000. ISSN 1573-1375. doi: 10.1023/A:
1008932416310. CtRLhttps://Cet org. 1023:0000000011100001000: http:///......../.o.....0/111113
Ece Kamar, Ashish Kapoor, and Eric Horvitz. Identifying and accounting for task-dependent bias in crowdsourcing. In Third AAAI Conference on Human Computation and Crowdsourcing, 2015.
Hyun-Chul Kim and Zoubin Ghahramani. Bayesian classifier combination. In Neil D. Lawrence and Mark Girolami (eds.), Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pp. 619- 627, La Palma, Canary Islands, 21-23 Apr 2012. PMLR. URLhttt::/pprcceeeeee mir. press/ wttp://2//kk/eee htt./.mmm.... html]
Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Juraf- sky. Stanford's ' multi-pas, pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of the Fifftenthh Conference on Computational Natural Language Learning: Shared Task, CONLL Shared Task 11, pp. 28 34, Stroudsburg, PA, USA, 2011. Association for Compu- tational Linguistics. ISBN 9781937284084. URL htLp:tt::///// //lttp://d..... acm org/ ocm.orgg/ciiaaiiiionn citatiin.....ininnnnooo cfm?
id=2132936 2132938
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference res- olution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
Kenton Lee, Luheng He, and Luke Zettlemoyer. Higher-order coreference resolution with coarse- to-fine inference. In Proceedings the 2018 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2018
Sebastian Martschat and Michael Strube. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405-418, 2015.
Pablo G. Moreno, Antonio Artés- Rodriguez, Yee Whye Teh. and Fernando Perez- Bayesian nonparametric crowdsourcing. J. Mach. Learn. Res., 16(1):1007-1677, January 2015. ISSN 1532-4435. URL httt:://dd wac.m.lccmmmcmmmmmmmmmmmmmmmm... www.acc.rrg//ao www.or//ggtiiiiioo cttp::/.tatiiii.........mm.mmm wtt.aemioonccmmmmmmm77772770 28868011
An Thanh Nguyen, Byron Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. Aggre- gating and predicting sequence labels from crowd annotations. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 299-309. Association for Computational Linguistics, 2017. doi: 10.18653/v1/117--022 URL
http: http://wwwwee..rrr/tttttoo.oooe/ www.g/gggangggg/gooeoooooo
1028
Rebecca J. Passonneau and Bob Carpenter. The benefits of model of annotation. Transactions of the Association for Computational Linguistics, 2:311-3266 2014.
Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy. Udo Kruschwitz, and Massimo Poesio.
Comparing bayesian models of annotation. Transactions of the Association for Computational Linguistics, 6:571-585, 2018a. 8a. doi: 162//a// a), 00040. URL https: //ooi. orq/1C.
1162 2..////ac. 00040
10

---

Under review as a conference paper at ICLR 2020
Silviu Paun, Jon Chamberlain, Udo Kruschwitz, Juntao Yu, and Massimo Poesio. A probabilistic annotation model for crowdsourcing coreference. Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pp. 1926-1937, Brussels, Belgium, October- November 2018b. Association for Computational Linguistics. URL http: //www/ ac web.
org/antho www.antthhooo/ooo-oooge ogw/p1y-y12220
Massimo Poesio, Roland Stuckardt, and Yannick Versley. Anaphora Resolution: Algorithms, Re- sources and Applications. Springer, Berlin, 2016.
Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. Anaphora resolution with the ARRAU corpus. In Proceedings of the NAACL Workshop Computational Models of Reference, Anaphora and Coreference (CRAC), pp. 11-22, New Orleans, June 2018.
Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In TO APPEAR IN Proceedings of the 2019 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), Minneapolis, USA, 2019. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. CoNLL- 2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference Computational Natural Language Learning (CoNLL 2012), Jeju, Korea, 2012.
Sameer S. Pradhan, Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. Ontonotes: A unified relational semantic representation. In International Conference on Semantic Computing (ICSC 2007), pp. 517-526, Sept 2007. doi: 10. N0.1109/CCC.2007..0 883.
Nguyen Quoc Viet Hung, Nguyen Thanh Tam, Lam Ngoc Tran, and Karl Aberer. An evaluation of aggregation techniques crowdsourcing. In Web Information Systems Engineering WISE 2013, pp. 1-15, Berlin, Heidelberg. 2013. Springer Berlin Heidelberg.
Vikas C. Raykar and Shipeng Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J. Mach. Learn. Res., 13( 13(1):41--518 February 2012. ISSN 1532-4435. URL http://dl acm. org/ wttg/eittatiie
wtt 2188401
Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 11: 1297-1322, 2010.
Aashish Sheshadri and Matthew Lease. SQUARE: A benchmark for research on computing crowd consensus. In Proceedings of the Ist AAAI Conference on Human Computation (HCOMP), pp. 156-1666 164, 2013. URL htt.htttp:/... ischool. www.aa.aaa........... edu/ eebu/sqquere/Bocumentss: sshesaadi..pdf
Edwin Simpson, Stephen J Roberts, Arfon Smith, and Chris Lintott. Bayesian combination of multiple, imperfect classifiers. 2011.
Edwin Simpson, Stephen Roberts, Ioannis Psorakis, and Arfon Smith. Dynamic Bayesian Com- bination of Multiple Imperfect Classifiers, pp. 1-35. Springer Berlin Heidelberg, Berlin, Hei- delberg, 2013. ISBN 978-3-642-36406-8. doi: 10. 10.1007/978-3-6223360000 7/doi org/ 10. 1 1007 / 978 - 642-36406-88
URL https:
Padhraic Smyth, Usama M. Fayyad, Michael C. Burl, Pietro Perona, and Pierre Baldi. Inferring ground truth from subjective labelling of Venus images. In Advances in neural information pro- cexsing systems, pp. 1085-1092, 1995.
Rion Snow, Brendan O' Connor, Daniel Jurafsky, and Andrew Y. Ngg. Cheap and fast but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference Empirical Methods in Natural Language Processing, pp. 254-263, 2008
11

---

Under review as a conference paper at ICLR 2020
Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27( -444, December 2001. ISSN 0891-2017.
Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. Community- based bayesian aggregation models for crowdsoureing. In Proceedings of the 23rd International Conference on World Wide Web, WWW 14, pp. 155-164, New York, NY, USA, 2014. ACM. URL http:://ott acm. org/
ISBN 978 978-1-4503-274422 doi: 2611445/25664462267799999
10. N0.1145/2566486.. 2567988
Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. Ontonotes: large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. Springer, 2011.
Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22, pp. 2035-2043 Curran Associates, Inc., 2009.
A PARAMETER ESTIMATION DETAILS
In this appendix we first list all the formulas for the variational parameters that are required to fully reimplement the model. This is followed by section that provides additional details how to derive some of the more challenging formulas.
A.1 NOTES ON THE UPDATE FORMULAS OF THE VARIATIONAL PARAMETERS
The derivations for the formulas required compute the variational parameters follow a straight- forward procedure: 1) compute the complete conditionals of each parameter, 2) set the variational distributions same exponential family form as their corresponding complete conditionals, and 3) use the fact that the natural parameters of the variational distribution are equal to the expected value (under the variational distribution) the natural parameters of the complete conditionals.
A.1.1 MENTION-LEVEL PARAMETERS (AND PRIORS)
is straightforward to show the complete conditional for the class-level true label likelihood l..)...) a Beta distribution. This gives us g(TT Pn, mh) Beta 1hk,, where:
T.M;
\ ag +
Σ i,m
1(
(C 1)]
L.M,
Mh = a1 + Σ
1( = 1( 0)]
(2)
1.m
The conditional for the true label indicators p(C),mll..)) is Bernoulli distributed. This gives us (0 (i. = Bernoulli(@s.m)) where Si. 1 Dimm and:
Na
log Pi.m (log sm log (60.m..) + Egg Iemm,
0.5(E,/agggt.mmmn) S1.m.n) - A(5r.m. (E, (6 m. + consl.
(3)
12

---

Under review as a conference paper at ICLR 2020
Num
log Si -Eylog( Zz )
log (pi.m,n) Ey (3 ml. Iurmmmn
0.5(Eg133ft.mmn..z))) Pi.m,n) A( m.) pr.m,n) + const.
(4)
The solution for the variational parameters introduced by the Jaakkola & Jordan (0000 bound can easily be found by taking partial derivatives of the variational objective function (the evidence lower- bound):
m,,, Egla ft.
(5)
Pm =
(6)
A.1.2 ANNOTATOR-LEVEL PARAMETERS
The complete conditional of annotator's sensitivity -la.) is Gaussian distribution. There- fore, we have o(ay,ahy, P5,) = Normal(7g,. Hy,h), where:
[a = ,o,,,,,,,,
1=1
2
Σ 10ili, = [))m = 1)E = 1)]A(6s.m,n))
12, = -,.
AiA Mj,
(8)
1Gili, m, =))((zi, = Me,, ()) 0.5))
Similarly, the complete conditional of an annotator's specificity p(3),ll...) is also Gaussian distri- bution. We have (B,,, 0 J, (,n,) = Normal(O s,n), where:
Σ
E, 1,,,,,
G
(9)
2 Σ
IGili, m.li,,mn)) =))=)) - Mc,, 0))A(pimm,))
i,
8jh = Egll( 1)1
(10)
Σ i,m,n
10ili, m = ())(m.m = h))yg((cr.. 0))(0.5 V..m.))
The community indicator p(x, has its complete conditional categorically distributed. This gives us (1,)) Categorical where:
13

---

Under review as a conference paper at ICLR 2020
log H,,r log A,] + Eg [og -,])+
k Σ 0.55) Egglog or + h=1
(1I)
Fr.N91) +
(-0.5) (Eyllog + Eq 1 - -vra)9) + const.
A.1.3 COMMUNITY-LEVEL PARAMETERS (AND PRIORS)
The complete conditional of the class-level mean sensitivity of community p(Tr.ll...) is Gaussian. That means we have (Trr,)l 7/ TP, Normal (/M F. where:
TrA
dj
E r.
Σ j=l
E [2, =>)]]]]
(12)
J [a, r))E,
A = Tri
do/d E,
Σ ==1
(13)
The conditional for the class-level variance of the sensitivity of community p(o,,, (...)) is an Inverse- Gamma distribution. That leads to q(0,,110f,, o.,)) Inverse Gamma 7. 7.),, where:
i = co 0.5 Σ
Eg = =r]
(14)
1=1
of + 0.5
E = [())) -
(15)
A similar discussion applies to the specificity-related parameters of community. The conditional of the mean specificity p(vr,al-..) a Gaussian, leading to q(vr, I vT,,) = Normal (v/ V where:
rho r.
v9 = 1- E
Σ j=1
(2, = r)]
(16)
1 vith do//2 E, (2,
E, [2 = r))
(17)
The complete conditional for the variance of the specificity p(22rht...) is an Inverse-Gamma distri-
bution, giving us gS22r, 22 Inverse Gamma (D? 2/99 where:
2r = ug + 0.5
E = =r]
(18)
288 u1 + +.5 Σ
E, 11l 1(3,,,(3,,n -1,
(19)
14

---

Under review as a conference paper at ICLR 2020
Var.
Summary
Var.
Summary
((a.m)) = Bernoulli(@s.m)
((vr,)) = Normal (v4
r.h Ur
6,m
E = 1)] = Pi.m
Vr.h
Eg[Ur,h] U Egl = (v Vig
Eyl1(ci.m = 0)] = Se
(xzz) = Categorical
((n,)) = Inverse G Dp.h)) E 22, log S2 (
2r
E,[1(r, r)] Rej.r
Eg r)] Σ Exer+1 Mggu
E, S.
2
((ay,)) Normal Ni.ho P3,h)
(A,) Beta fri((frr))) 9r)
aj.h
Eglaj.h1 = 7ijhh Eg
A
log A = (((r)) (Tr + gr)
2
"j.n + Mi,h
E, A, (gr) V/ gr)
(BB,,))
Normal (O,.n SiN)
((TA)] (nn m.)
3gA
E
Th
E, nT. v/ ()a) (A,, + mh) logg(11 - nT)] = v (mm) T m))
03 Eii
Eg j.h.
(Tr,,)) = Normal (TT T
((b) = Gamma (b b²)
Tr.h
[T THe E (T ) #+++ T
b
[6
6
g
E, log bj (b²) log bd
((Tr,)) Inverse fo o. Eyllog d,,Al = log of (ofh)) E o
Jr.
Table 2: A summary of the variational distributions together with relevant expectations.
The conditional for the stick proportions p(A,|...) is Beta distribution. The corresponding varia- tional distribution has the same form q( (A Ir, gr) = Beta(fr, Ir), where:
fr =+
1+
E E12, r)]
(20)
gr b Σ Ezz > r)]
(21)
Finally, the conditional for the scale p(bl...) ... is a Gamma distribution, which means q(blba, ,b)) = Gamma(b", 68), where:
= s0 -C-1
(22)
C-1
6 = [lo([1 - A,)]
(23)
A.1.4 IMPLEMENTATION NOTES
The parameters are updated turn until the variational objective function (the evidence lowerbound) plateaus. The algorithm is often referred Coordinate Ascent (mean field) Variational Inference (CAVI) Blei et al. 2017). (ablee]ssmmariizss the variational distributions together with the relevant expectations required for the implementation.
For initialization we used the annotator profiles inferred by MPA together with random commu- nity assignments. MPA
with the default configuration: uniform priors, with the sensitivity
15

---

Under review as a conference paper at ICLR 2020
and specificity of the annotators receiving bit of extra mass initialization to address the label switching problem. This problem, specific to mixture models which are at the core of many models of annotation, is caused by the likelihood's invariance under the permutation of the labels, mak- ing the models nonidentifiable. Regarding the choice of byper-parameters, we used either uniform weakly informative priors that inform the scale of the results, but are not otherwise sensitive 0, eg = 5). The truncation level
(ao = 50 51 dj 1 = e1 was set to C 10 in all our experiments.
1, do = to
A.2 FURTHER DETAILS
This subsection provides additional parameter estimation details for the readers interested in deriving the update formulas presented above. As stated back in Section 2.2 we parameterized the model with conjugacy in mind, and where that was not directly possible, i.e., in those cases involving the standard logistic function, we addressed the problem using the bound from Jaakkola Jordan (2000). We include below additional guidance for these cases. Focusing on the sensitivity side of derivations (specificity analogous) we have:
= Bernoulli( (Tagsiiaa
Y ji m.n).. Mimm (1 ml " Bt.m..
(24)
mm Mi ni Wt- alliam,,
The bound from Jaakkola & Jordan (2000) gives us:
T(agslimma)
r m,, exp(a// - -m, m/2
(25)
(o
02 kmm,,))
where
A()....)) = 0.50 m,,, (T - 0.5)
Using the bound back in Equation 24 leads us to:
1i 2 (6) exp(a Vimm,
(26)
(Ajj(i.mm........ m,,,n A6 f...m.)))
Drawing annotations based on the sensitivity of the annotators comes up in two complete condi- tionals, related the sensitivity itself, and the other in the positive outcome of the true label indicators. The first conditional is:
I.MeNN..m
p(ag.l...) & play,lTZ, Rr)
II K,
P(u.m (ll.ma M
(27)
The first term can be expressed as:
20
plaghh Iry,h) x exp
-0.5 Σ
(αx = (a3, 20g.7rrl))
(28)
The second term is:
p(yi,,alas..) "j.hli T(-ag,))
2 T()mm,,)) exp faj.ndh.mm (ajj, + m1, A(Brm..))a.,.
Kayy, exp {as.tiiimm - 0.50g. A (85 )a3,,
(29)
16

---

Under review as a conference paper at ICLR 2020
Going back to Equation 27 we have:
06 Σ 1=
[ry r)o 20gntman
plaasal...)) x xxepp
-0.5
(iili,,,n) n. izi h))(cim
II
(exp (agghth..m. 0.50r,. m00
shrmm ....).....)
4,m.n
00
x eexpf 0.5 Σ 11
I,M,N.
+
Σ
Kr..lagaiiimmm- - 0.5ajj A )a
6,,,
expl- expf-0.5fa..a.
Σ (1, (ayy=)oap +2 E ((-)))()m
let's call this
22aj.
[rz = r)o n Tr.b Σ I(..))us..m. 0.5))]}
r=1
let's call this
})}
x exp
-0.5a
ai.1
-1
b -
exp
-0.5
a33h
NNormal(//a,
(30)
We have just showed the complete conditional of the annotator's sensitivity p(az.al..) is a Gaussian distribution. The corresponding variational distribution will take the same form, i.e. ((a ki )j.A) Normal(7,,h, M)..). Using the aforementioned connection between the natural parameters of the two distributions leads us the formulas presented back in Section A.1.2
The other complete conditional which involves the bounded logistic function is the true label indi- cator, Again, focusing on the sensitivity side of the dependencies, have:
N s.m.
p(c,, = 1l...) p(ci.
II
p(y..m P (ijssismm)))
(31)
Ni..
exp
log T Tecm log (:5.:.mm
After expanding p(c.m = 0l...) in similar manner, it can be seen the conditional for the true label indicators, p(C.,ml...). is Bernoulli distributed. This means the corresponding variational distribu- tion takes the same form, i.c., ((C..//o... Bernoulli(ó)mm) To get the update formula of the variational parameter we simply take the necessary expectations:
No
log di.m -ogglog ग T..
+
Σ
[log ((yi (itf:m/m const.
(32)
n=1
17

---

Under review as a conference paper at ICLR 2020
The first expectation is trivial, while for the second
need to apply the bound from Equation few further derivations should result in the update formulas presented back in Section A.
25
The Jaakkola & Jordan (0000) bound introduces additional variational parameter (see Equation
25
which can found by taking partial derivatives of the variational objective function (the ELBO).
For example, the terms dependent msmm, are:
-imm
log Y (ōi.,,,)) - 0.50 A()i,,,))
(Egla)))) (5tmm,) 82
(33)
Deriving the ELBO with respect to di,m,n leads to:
(CBL.m.a) -i ((.. Pl.m m,
(34)
Setting the above equation to and solving for 5,,,, gives the result stated in Section A.1.1
Computing the other complete conditionals and deriving the of the update formulas presented in the paper should be straightforward due to conjugacy; simply follow the three-steps procedure outlined in the heginning of this appendix and put in practice above. The stick breaking representa- tion of the Dirichlet process nicely complies with conjugacy as well; the notes from Blei & Jordan (2066) can be a useful read. Finally, following all of the steps up to this point, should also make straightforward to expand the variational objective function (ELBO) used to assess the convergence of the coordinate ascent algorithm.
B DETAILED EVALUATION RESULTS
This Appendix includes detailed view of the evaluation results presented in the paper. Specifically.
Table 3 shows more details for the inferred mention pairs evaluated back in Section 3.1 Table presents deeper view into the quality of the silver chains discussed in Section 3.2 while Tables and 6 show extended view of the performance obtained by state of the art coreference system when trained on silver chains as it was presented back in Scction 3.3
Property
Non referring
Discourse new
Discourse
#
Acc.
P
Fl
F1
F1
MPA
5
86.2
100.0
0.7
1.4
92.3
24.5
38.7
83.9
98.7
90.7
91.4
73.7
81.6
MPA
100.0
11.7
100.0
57.1
72.7
88.5
97.5
92.8
89.5
81.6
85.4
MPA
20
90.5
69.4
17.2
27.6
100.0
69.4
81.9
91.7
96.7
94.1
88.6
86.0
87.3
MPA
40
91.6
74.1
29.7
42.4
97.6
81.6
88.9
93.2
96.7
94.9
89.3
88.0
88.6
MPA
all
92.2
64.0
72.4
68.0
94.1
98.0
96.0
94.5
96.0
95.3
90.4
87.3
88.8
CMPA
5
88.6
76.7
15.9
26.3
97.1
69.4
81.0
87.1
98.1
92.3
91.7
78.9
84.8
CMPA
10
91.3
72.2
39.3
50.9
92.3
98.0
95.1
92.9
96.3
94.6
89.3
86.7
88.0
CMPA
91.7
64.8
46.9
54.4
94.1
98.0
96.0
93.9
96.1
95.0
89.3
87.6
88.4
CMPA
40
92.0
66.4
58.6
62.3
94.0
95.9
95.0
94.4
95.9
95.1
89.7
88.2
88,9
CMPA
all
92.2
64.0
72.4
68.0
94.1
98.0
96.0
94.5
96.0
95.3
90.6
87.4
89.0
MV
82.9
53.9
9.7
16.4
97.2
71.4
82.4
79.1
99.0
87.9
94.5
62.8
75.4
Table 3: Detailed evaluation results of the inferred mention pairs matched against expert annotations. The "#" column shows the maximum user workload where the "all" configuration uses the dataset as it is (does not alter the player workloads). CMPA is shorthand for COMMUNITYMPA and MV for majority voting.
18

---

Under review as a conference paper at ICLR 2020
MUC
BCUB
CEAFE
Avg.
Method
F1
F1
F1
MPA
93.3
76.3
84.0
94.0
83.9
88.7
81.9
95.2
88.1
86.9
MPA
90.8
83.8
87.2
93.1
86.9
89.9
86.5
94.5
90.3
MPA
20
90.4
87.2
88.8
93.0
89.9
91.4
89.6
94.2
91.9
90.7
MPA
40
90.9
88.8
89.8
93.5
91.5
92.5
91.2
94.3
92.7
91.7
MPA
all
91.6
82.3
86.7
94.8
88.3
91.4
92.3
93.6
93.0
90.4
Singletons included
CMPA
5
93.3
80.1
86.2
94.2
87.0
90.5
85.1
94.9
89.8
88.8
CMPA
10
90.8
88.1
89.4
93.4
89.8
91.6
90.8
93.7
92.2
91.1
CMPA
90.8
86.5
88.6
93.7
90.0
91.8
91.7
93.7
92.7
91.0
CMPA
40
90.9
84.8
87.7
94.1
89.2
91.6
92.2
93.5
92.8
90.7
CMPA
all
91.7
82.2
86.7
94.8
88.3
91.4
92.3
93.6
92.9
90.4
96.0
63.9
76.7
95.7
78.7
86.4
77.1
94.9
85.1
82.7
Stanford
65.4
62.4
63.8
78.9
76.1
77.5
78.4
85.2
81.7
74.3
MPA
93.6
76.0
83.9
90.1
57.9
70.5
71.3
59.7
65.0
73.1
MPA
87.4
87.0
66.9
75.7
73.4
70.8
72.1
78.4
MPA
90.8
88.4
89.6
86.2
75.8
80.7
76.7
77.7
77.2
82.5
MPA
40
91.3
90.1
90.7
87.0
80.1
83.4
79.4
80.9
80.1
84.7
MPA
all
92.2
89.2
90.7
88.1
77.8
82.6
79.5
80.2
79.8
84.4
Singletons excluded
CMPA
5
93.6
81.1
86.9
89.4
66.5
76.3
75.5
65.3
70.0
77.7
CMPA
10
91.1
88.8
89.9
85.9
76.2
80.8
78.2
77.7
78.0
82.9
CMPA
20
91.3
89.5
90.4
86.7
78.8
82.6
79.1
80.2
79.7
84.2
CMPA
91.6
90.1
90.9
87.3
78.5
82.6
80.0
80.1
80.1
84.5
CMPA
all
92.4
89.1
90.7
88.1
77.8
82.6
79.4
80.0
79.7
84.3
MV
96.1
64.8
77.4
93.8
45.0
60.8
66.3
48.5
56.1
64.8
Stanford
65.7
62.1
63.9
50.3
42.5
46.1
42.7
49.8
46.0
$2.0
Table 4: The quality of the silver chains evaluated using standard coreference metrics against gold chains. The "#" column shows the maximum user workload where the "all" configuration uses the dataset as it is (does not alter the player workloads). CMPA is shorthand for COMMUNIYMPA. MV for majority voting, while Stanford the deterministic coreference system developed by Lee et al. (2011)
19

---

Under review as a conference paper at ICLR 2020
MUC
BCUB
CEAFE
Avg.
Method
F1
P
R
P
R
P
R
F1
MPA
81.3
62.8
70.9
76.7
70.0
70.0
MPA
10
80.5
66.9
73.1
73.6
67.7
70.5
66.4
75.8
70.8
71.5
MPA
20
78.1
73.2
75.6
68.4
70.9
69.6
68.3
73.6
70.9
72.0
MPA
40
78.3
71.7
74.8
71.3
69.0
70.1
69.3
74.6
71.9
72.3
Singletons included
MPA
all
79.3
72.5
75.7
72.1
69.3
70.7
70.5
73.2
71.8
72.7
CMPA
5
80.2
67.1
73.0
72.8
66.7
69.6
67.5
75.6
71.3
71.3
CMPA
10
78.8
72.7
75.6
72.4
69.5
70.9
70.1
73.9
71.9
72.8
CMPA
20
76.6
73.9
75.2
70.9
69.3
70.1
70.7
73.3
72.0
72.4
CMPA
40
78.8
72.2
75.3
71.2
70.0
70.6
69.8
74.1
71.9
72.6
CMPA
all
78.9
71.0
74.7
71.8
68.0
69.9
69.6
73.6
71.6
72.0
MPA
5
81.3
62.8
70.9
60.4
37.3
46.1
52.7
37.4
43.7
53.6
MPA
10
80.5
66.9
73.1
62.2
44.9
52.2
54.3
43.4
48.2
57.8
MPA
20
78.1
73.2
75.6
51.5
54.4
$2.9
55.5
47.8
51.4
60.0
40
78.3
71.7
74.8
49.9
58.1
50.4
54.0
60.7
Singletons excluded
MPA
all
79.3
72.5
75.7
58.3
52.4
55.2
58.3
49.5
53.5
61.5
CMPA
80.2
73.0
42.9
49.5
$5.2
43.2
48.5
57.0
CMPA
10
78.8
72.7
75.6
58.5
51.5
54.8
56.2
49.4
52.6
61.0
CMPA
20
76.6
73.9
75.2
56.4
51.3
53.7
54.6
50.2
52.4
60.4
CMPA
40
78.8
72.2
75.3
56.2
52.5
54.3
55.9
49.5
52.5
60.7
CMPA
all
78.9
71.0
74.7
57.2
49.1
52.8
56.3
47.7
51.7
59.7
Table 5: Detailed evaluation results of a state of the art coreference system trained on silver chains.
The "#" column shows the maximum user workload where the "all" configuration uses the dataset as it is (does not alter the player workloads). CMPA is shorthand for COMMUNIYMMPA.
Max.
Method
F1
workload
MPA
73.3
5.1
9.6
MPA
10
76.6
16.7
27.5
MPA
71.4
30.2
42.5
MPA
40
67.2
37.2
47.9
MPA
all
55.2
54.0
54.6
COMMUNITYMPA
69.3
24.2
35.9
COMMUNITYMPP
10
66.9
45.1
53.9
COMMUNITYMPA
20
60.5
48.4
53.8
COMMUNITYMPA
40
56.4
51.2
53.7
COMMUNITYMPA
all
49.6
54.9
52.1
Table 6: Detailed non referring scores (expletives + predicative NPs) obtained by a state of the art coreference system trained on silver chains.
20


Reviews:
[{"review_id": "HJxjk86dcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an interesting crowdsourcing approach as an alternative to expert annotation. Different from the conventional crowdsourcing approaches that concentrate on classification tasks with predefined classes, the proposed Community MPA, as an extension of MPA can address anaphoric annotation well where coders relate markables to coreference chains whose number cannot be predefined. Authors demonstrate that on a recent large-scale crowdsourced anaphora dataset, the proposed Community MPA works better than the unpooled counterpart in conditions of sparsity, and on par when enough observations are available. It also generalizes well to other crowdsourcing tasks under different settings.\n\n* The paper is an incremental work to the existing one by Paun et al. (2018b). It considers the underlying communities of different workers, which may lead to better annotations when the data is scarce.\n* It is also partially similar to the Moreno et al. (2015) that models a latent number of communities through Gibbs sampling. The proposed method also shows comparable performance compared to Moreno et al. (2015) in standard annotation tasks.\n* Technically, the paper is written well (I appreciate detailed deduction in the appendix), but is hard to follow for people not from the crowdsourcing field. There are lots of field-specific terms without much explanation. It may take quite a while for readers with only machine learning background.\n* The complexity of the work is not explicitly explained. While the solution itself refers to (Blei et al., 2017), it would be better to show running time in experiments.\n* Authors claimed that “We let the number of communities grow with the data, a flexibility that we achieve using a Dirichlet process mixture,” but this is not clear in the modeling in section 2.1. Which parameters control such increases and how they were adjusted? More explanations are needed.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1l4l3C-qS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an improvement over the mention-pair model for anaphoric annotation by Paun et al. to mitigate the effects of sparsity that are inherently present in some crowd-sourcing environments. The extension involves hierarchic communities of annotators, where the number of communities grows with data via \"Dirichlet process mixture based on a stick-breaking representation of the underlying Dirichlet process\".\n\nThe paper is very well-written and the results are convincing. Experiments pack appropriate breadth of comparison and the discussion is relatively extensive. THe coverage of related work is extensive and the contribution placement apt. The paper does not state the language(s); it seems to be English-only, and it is not clear how the model would scale up/down with the number of languages or limited availability of resources.\n\nI see no major issues with accepting the paper.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byec-4kCYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper extends the unpooled mention pair model of annotation (MPA) (Paun et al., 2018b) with the hierarchical priors (e.g., mean and variance) on the ability of the annotators. The proposed method was evaluated on Phrase Detectives 2 corpus, which was annotated by players in a game with a purpose setting for coreference resolution. To control the sparsity of the dataset, the authors split annotations from larger player workloads into smaller batches, and assumed that each batch was produced by a different player. The experimental results show that, when the data is sparse, the proposed method (CommunityMPA) worked better than MPA on Phrase Detectives 2 corpus in terms of mention-pair accuracy, silver-chain quality, and the performance of the state-of-the-art method trained on the aggregated mention pairs. This paper also includes a discussion about the inferred community profiles. The comparison with the traditional approaches that consider communities showed that the proposed method is comparable to the traditional approaches.\n\nI am wondering of the connection between community and sparsity. This study assumes that knowledge of communities (spammers, adversarial, biased, average or high-quality players) would allow to regularize the ability of the annotators towards. In P2, this paper wrote, \"This partially pooled structure can prove effective in conditions of sparsity where there are not enough observations to accurately estimate the ability of the annotators in isolation.\" I have two questions here: why is it effective to consider community for reducing the problem of the sparsity? If the knowledge of communities is useful, why did the advantage of the proposed method disappear in Figures 1, 2, and 3 when we used all data?\n\nIn addition, I'm not convinced with the idea of \"breaking the larger player workloads into smaller batches\" for simulating the sparsity and communities. This treatment introduces quite a few shadow users whose capabilities are exactly the same, and deviates from the reality of the user community. Does this treatment favor the proposed method over MPA more than necessary? I am also wondering why evaluating portions of the dataset where annotations were made by 'sparse' users would not work to highlight the effectiveness of the proposed method for sparse users.\n\nThe impact of this paper would be greater if the experimental results could support the importance of modeling user community on the real data. The authors may justify the simulation procedure of the sparsity because 98.67% of Phrase Detectives 2 corpus was annotated by those who produced more than 40 annotations. However, I also think that it is important to show how the proposed method is effective on the real data with sparse annotators. Currently, Table 1 showed no improvement over the conventional methods.\n\nMinor comment\n\nIn Section 2.1: It was difficult to separate which part is the base model (MPA) and the novel proposal without reading Paun et al. (2018b).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgzkH-vqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to get a more qualified output from multiple crowdsourced mention pair annotations for coreference annotation. It builds on Paun et al. (2018b) who presented a probabilistic model able to aggregate crowdsourced data while also turning co-reference annotation into a task with four pre-determined classed, denoted the Mention Pair Model (MPA). This study extends MPA to CommunityMPA by including information about the annotators' hierarchical community profiles from Simpson et al. (2011, 2013): spammers, adversarial, biased, average, and high-quality players. CommunityMPA mention pairs are compared to MPA and silver chains derived from both mention pairs are compared to gold chains. The paper also compares a SOTA model trained on silver chains. The general trend for all experiments is that for lower annotator workloads, CommunityMPA is better, but performance for both models generally increase with higher annotator workloads. For all experiments, using higher annotator workload (40 or higher) is the better and MPA is then better or on par. MPA and CommunityMPA are also compared to results from Moreno et al. (2015) on other tasks and datasets showing on par or better performance. \n\nThe paper is well-written and the method is well-justified and generally well-described. The comparisons are meaningful. The work in interesting and much needed if SOTA indeed is from 2015, but the CommunityMPA results are not strong enough for me to recommend full accept. \n\nStrengths:\nThis seems like a simple, strong and sensible approach when annotations per annotator are sparse.\n\nWeaknesses:\nBut performance is not better than MPA with a higher workload. When designing the annotation process, it does not seem like a good idea to use CommunityMPA over asking for a minimum of 40 annotations per annotator and use MPA. \n\nQuestions:\nWhich state of the art coreference system do you use for results in Figure 3?\n\nSmall comments\nFigure 4 is an interesting insight into the four community profiles (but why not include adversaries?) but is kind of detached from the rest of the experiments, since these profiles are not discussed much in the remaining part of the paper. I suggest explaining community profiles more.\n\nTable 1: I would find it useful if you would briefly introduce the tasks from Moreno et al. (2015). Otherwise, I would assume the task to be the same as previous parts of the paper. The number of correctly labelled items seems irrelevant, whereas the size of each dataset is relevant to include (though not necessarily in the table)\np 7: seminar work -> seminal work?\nCaption of Table 1: (Moreno et al., 2015) > Moreno et al. (2015)\nTable 1: Boldface best result per dataset", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks to the reviewers and the authors for an interesting discussion. The reviewers are mixed, learning toward positive, but a few shortcomings were left unaddressed: (i) Turning the task into a mention-pair classification problem ignores the mention detection step, and synergies from joint modeling are lost. (ii) Lee et al. (2018) has been surpassed by some margin by BERT and spanBERT, models ignored in this paper. (iii) Several approaches to aggregating structured annotations have already been introduced, e.g., for sequence labelling tasks. [0] Overall, the limited novelty, the missing baselines, and the missing related work lead me to not favor acceptance at this point. 

[0] https://www.aclweb.org/anthology/P17-1028/
Decision: Reject