Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper presents an interesting crowdsourcing approach as an alternative to expert annotation. Different from the conventional crowdsourcing approaches that concentrate on classification tasks with predefined classes, the proposed Community MPA, as an extension of MPA can address anaphoric annotation well where coders relate markables to coreference chains whose number cannot be predefined. Authors demonstrate that on a recent large-scale crowdsourced anaphora dataset, the proposed Community MPA works better than the unpooled counterpart in conditions of sparsity, and on par when enough observations are available. It also generalizes well to other crowdsourcing tasks under different settings.

* The paper is an incremental work to the existing one by Paun et al. (2018b). It considers the underlying communities of different workers, which may lead to better annotations when the data is scarce.
* It is also partially similar to the Moreno et al. (2015) that models a latent number of communities through Gibbs sampling. The proposed method also shows comparable performance compared to Moreno et al. (2015) in standard annotation tasks.
* Technically, the paper is written well (I appreciate detailed deduction in the appendix), but is hard to follow for people not from the crowdsourcing field. There are lots of field-specific terms without much explanation. It may take quite a while for readers with only machine learning background.
* The complexity of the work is not explicitly explained. While the solution itself refers to (Blei et al., 2017), it would be better to show running time in experiments.
* Authors claimed that “We let the number of communities grow with the data, a flexibility that we achieve using a Dirichlet process mixture,” but this is not clear in the modeling in section 2.1. Which parameters control such increases and how they were adjusted? More explanations are needed.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper presents a method to get a more qualified output from multiple crowdsourced mention pair annotations for coreference annotation. It builds on Paun et al. (2018b) who presented a probabilistic model able to aggregate crowdsourced data while also turning co-reference annotation into a task with four pre-determined classed, denoted the Mention Pair Model (MPA). This study extends MPA to CommunityMPA by including information about the annotators' hierarchical community profiles from Simpson et al. (2011, 2013): spammers, adversarial, biased, average, and high-quality players. CommunityMPA mention pairs are compared to MPA and silver chains derived from both mention pairs are compared to gold chains. The paper also compares a SOTA model trained on silver chains. The general trend for all experiments is that for lower annotator workloads, CommunityMPA is better, but performance for both models generally increase with higher annotator workloads. For all experiments, using higher annotator workload (40 or higher) is the better and MPA is then better or on par. MPA and CommunityMPA are also compared to results from Moreno et al. (2015) on other tasks and datasets showing on par or better performance. 

The paper is well-written and the method is well-justified and generally well-described. The comparisons are meaningful. The work in interesting and much needed if SOTA indeed is from 2015, but the CommunityMPA results are not strong enough for me to recommend full accept. 

Strengths:
This seems like a simple, strong and sensible approach when annotations per annotator are sparse.

Weaknesses:
But performance is not better than MPA with a higher workload. When designing the annotation process, it does not seem like a good idea to use CommunityMPA over asking for a minimum of 40 annotations per annotator and use MPA. 

Questions:
Which state of the art coreference system do you use for results in Figure 3?

Small comments
Figure 4 is an interesting insight into the four community profiles (but why not include adversaries?) but is kind of detached from the rest of the experiments, since these profiles are not discussed much in the remaining part of the paper. I suggest explaining community profiles more.

Table 1: I would find it useful if you would briefly introduce the tasks from Moreno et al. (2015). Otherwise, I would assume the task to be the same as previous parts of the paper. The number of correctly labelled items seems irrelevant, whereas the size of each dataset is relevant to include (though not necessarily in the table)
p 7: seminar work -> seminal work?
Caption of Table 1: (Moreno et al., 2015) > Moreno et al. (2015)
Table 1: Boldface best result per dataset
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The paper proposes an improvement over the mention-pair model for anaphoric annotation by Paun et al. to mitigate the effects of sparsity that are inherently present in some crowd-sourcing environments. The extension involves hierarchic communities of annotators, where the number of communities grows with data via "Dirichlet process mixture based on a stick-breaking representation of the underlying Dirichlet process".

The paper is very well-written and the results are convincing. Experiments pack appropriate breadth of comparison and the discussion is relatively extensive. THe coverage of related work is extensive and the contribution placement apt. The paper does not state the language(s); it seems to be English-only, and it is not clear how the model would scale up/down with the number of languages or limited availability of resources.

I see no major issues with accepting the paper.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper extends the unpooled mention pair model of annotation (MPA) (Paun et al., 2018b) with the hierarchical priors (e.g., mean and variance) on the ability of the annotators. The proposed method was evaluated on Phrase Detectives 2 corpus, which was annotated by players in a game with a purpose setting for coreference resolution. To control the sparsity of the dataset, the authors split annotations from larger player workloads into smaller batches, and assumed that each batch was produced by a different player. The experimental results show that, when the data is sparse, the proposed method (CommunityMPA) worked better than MPA on Phrase Detectives 2 corpus in terms of mention-pair accuracy, silver-chain quality, and the performance of the state-of-the-art method trained on the aggregated mention pairs. This paper also includes a discussion about the inferred community profiles. The comparison with the traditional approaches that consider communities showed that the proposed method is comparable to the traditional approaches.

I am wondering of the connection between community and sparsity. This study assumes that knowledge of communities (spammers, adversarial, biased, average or high-quality players) would allow to regularize the ability of the annotators towards. In P2, this paper wrote, "This partially pooled structure can prove effective in conditions of sparsity where there are not enough observations to accurately estimate the ability of the annotators in isolation." I have two questions here: why is it effective to consider community for reducing the problem of the sparsity? If the knowledge of communities is useful, why did the advantage of the proposed method disappear in Figures 1, 2, and 3 when we used all data?

In addition, I'm not convinced with the idea of "breaking the larger player workloads into smaller batches" for simulating the sparsity and communities. This treatment introduces quite a few shadow users whose capabilities are exactly the same, and deviates from the reality of the user community. Does this treatment favor the proposed method over MPA more than necessary? I am also wondering why evaluating portions of the dataset where annotations were made by 'sparse' users would not work to highlight the effectiveness of the proposed method for sparse users.

The impact of this paper would be greater if the experimental results could support the importance of modeling user community on the real data. The authors may justify the simulation procedure of the sparsity because 98.67% of Phrase Detectives 2 corpus was annotated by those who produced more than 40 annotations. However, I also think that it is important to show how the proposed method is effective on the real data with sparse annotators. Currently, Table 1 showed no improvement over the conventional methods.

Minor comment

In Section 2.1: It was difficult to separate which part is the base model (MPA) and the novel proposal without reading Paun et al. (2018b).
review_title: Official Blind Review #2

Meta-review: Thanks to the reviewers and the authors for an interesting discussion. The reviewers are mixed, learning toward positive, but a few shortcomings were left unaddressed: (i) Turning the task into a mention-pair classification problem ignores the mention detection step, and synergies from joint modeling are lost. (ii) Lee et al. (2018) has been surpassed by some margin by BERT and spanBERT, models ignored in this paper. (iii) Several approaches to aggregating structured annotations have already been introduced, e.g., for sequence labelling tasks. [0] Overall, the limited novelty, the missing baselines, and the missing related work lead me to not favor acceptance at this point. 

[0] https://www.aclweb.org/anthology/P17-1028/
Decision: Reject