Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The submission addresses a problem with collecting ground truth: human annotations are noisy, a common approach is to collect many annotations and apply majority voting to elicit a single label. With this approach, the annotators' expertise and the difficulty of single data instances is ignored. What the authors propose is a framework which allows one to combine a direct graphical model of how human annotations are produced with model training. The graphical model introduces latent variables for the difficulty of an instance and the competence of an annotator as well as the (unobserved) true label. This way one can potentially benefit from the meta-information about the annotators (e.g., their demographics) and improve upon the majority-voting baseline of aggregating available annotations. Maybe most importantly, the proposed framework allows one to get the true label with fewer annotations (significantly reduces redundancy). 

The proposed model is intuitive, the learning of the hidden variables is done with EM, the presentation is easy to follow. The experimental part is done on five annotation tasks (image classification, NLP, bio-NLP) and compares the proposed model (LIA) with six prior approaches (e.g., majority vote, MMCE, Snorkel). The evaluation metric is accuracy (i.e., guessing the true label as obtained from experts). Overall, the new model achieves higher or comparable accuracy to that of MMCE which in turn outperforms all other methods. W.r.t. redundancy reduction, on some tasks LIA achieves much better accuracy with fewer annotations. For example, on the word similarity task the accuracy with only two annotations is higher than that of the majority baseline with ten. 

The submission is well-written and I enjoyed reading it. I am not an expert in this area, but as far as I am concerned the contributions are sufficient to accept it. I have not found any technical or methodological flaws. However, I have the following questions and concerns:

1. The analysis part is very short and while the accuracy numbers are impressive, I wonder if a different experiment is needed to fully demonstrate the claimed benefits of the model. For example, I am not sure which part shows empirically that annotators' features are indeed used and useful. 

2. Related to the above point, maybe one could create a synthetic dataset where the true labels and true noise are added as if coming from two additional annotators and show that the model can identify them as highly competent / incompetent? 

3. Section 3.1 mentions a fine-tuning procedure in the end but the experimental part does not specify how much of a gain it delivered. How does the model perform without this fine-tuning? 

4. I have not followed this topic much but it seems to me that there should be more related work on modelling annotators' competence and item difficulty for crowd-sourced annotations. Isn't, for example, work by Bachrach et al. 2012 [1] relevant? 

5. How stable are results along the redundancy dimension? For example, the word similarity task has only 30 word pairs with ten ratings per item. How much, if at all, is accuracy with redundancy@2 affected by using different samples of two? 

Minor typos:
 - "can be can be" in Sec. 1 on page 2.
 - "a models" in Sec. 3.2. on page 5.
 - "a sources" in Sec. 3.3 on page 6.
 - "LIA-E" (should be "LIA"?) on page 7.

[1] "How To Grade a Test Without Knowing the Answers — A Bayesian
Graphical Model for Adaptive Crowdsourcing and Aptitude Testing" ICML'12.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Update after author response: 
I would like to thank the authors for their thoughtful response, and for addressing some of the concerns raised by the reviewers. One of my main complaints arose from a misunderstanding that none of the baselines model worker competencies and task difficulty. The authors clarified that MMCE does model those, making it a competitive baseline. Given that and the other additions to the draft, I am changing my assessment from 3 to 6.
--------------------------- 

In this paper, the authors extend the classical probabilistic model of Dawid-Skene (DS) for predicting the final label of crowdsourced tasks. The extensions include explicit modeling of image difficulty and worker competence as a function of image and worker features respectively, as well as a more expressive formulation of learned functions in terms of neural networks (NNs). The authors show that the proposed approach outperforms several baselines on 5 different datasets.

The problem studied in the paper is relevant since more and more data is needed to train ever more complex models, and often crowdsourcing is the way to generate such data. The paper is clearly written and conveys its central idea concisely. I liked the paper but I believe the contribution to be incremental in its current state. Here are the main issues in my opinion:

1. The presentation suggests that somehow the proposed approach is novel in its end-to-end framework. However, the central idea is very similar to Dawid-Skene (1979), and its subsequent augmentations like the ones in Carpenter (2008) that model both image difficulty and worker competence. These previously proposed models are also end-to-end approaches so that they can infer worker competence and image difficulties while also outputting a final label. Therefore, I believe the novel contributions are then slightly different formulation of these variables, and using NNs to learn the function parameters. I see that the proposed model can be somewhat more general since it uses semantic image features (as opposed to indicator features) but then it learns worker embeddings starting from random initializations which will not generalize to new workers, necessitating a re-run of the whole EM loop.

2. If Q is a stochastic matrix, its rows should sum to 1. I don’t understand the summation on Q in eq. 4. I am also a little confused by eq. 5 since the joint likelihood should just be the prior times the conditional likelihood of the observed annotations. At least trying to do it in my head doesn’t lead to what’s presented in eq. 5. It may just be an issue of notations which can be simplified by using conditional distributions instead of expectations of indicator random variables (same quantity). 

3. As far as I understand, none of the baselines considered explicitly model the task difficulty and worker competence. Therefore, the proposed model enjoys this extra level of expressiveness making its superior performance relatively unsurprising. I skimmed Zhou et al. (2015) and simple DS was already competitive on some of the datasets so I would think inclusion of its successors that model image difficulty and worker competence could perform quite well. 

3. Minor issue but eq. 6 is not majority vote as stated just above. It’s the maximum likelihood estimate P(y_i = k | D) assuming each worker is an unbiased estimator of the true label. 

4. Minor formatting issues: “learn a models” (sec 3.2), “lantent variables” (sec 4) “and by merging” (sec 5). 

In summary, even though I like that the paper is clearly written and tackles an important problem, I am not convinced that the contribution is substantial or the experiments very insightful. I would recommend addressing these issues and resubmitting the paper.

Carpenter, B. (2008). Multilevel bayesian models of categorical data annotation. Unpublished manuscript, 17(122), 45-50.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method for dealing with noisy human annotated in training data. The idea is to unify the annotation aggregation and model training whilst modelling the sample annotation difficulty and annotator competency level. The experiments on five datasets show improvements over a number of baselines.

This is a solid piece work that deals with a very practical problem – training ML models with crowdsourced data with imperfect annotations. The proposed method is not completely new: many ideas have been adopted from previous works. However putting everything together seems to work as demonstrated by the experiments.

I have a number of concerns:

1, One of the main claimed novelties of the paper is an “end-to-end” approach that unifies ground truth label prediction and label aggregation. However there is no ablation study to show that this is indeed better than a two-stepped variant with everything else remaining the same. 

2, The authors made a claim on the ability to estimate the quality of the annotator. But the statement is the Introduction is misleading – it suggests that the attributes of the annotator such as age and gender will be used as input to the estimator, but later it is clear that no benchmarks contain that information hence it is not implemented. Also it is not clear how this vector embedding (Sec. 3.2) is implemented. Any evidence that this embedding is indeed a clustering index?

3, In the implementation, the strong BERT model was used for language but VGG was used for image representation. Stronger CNN feature extractors such as ResNet101 should be used. 

4, In general, the lack of any ablation study is a problem for analysing why the model works.  

review_title: Official Blind Review #1

Meta-review: The paper introduces a novel way of jointly modeling annotator competencies and learning from imperfect annotations. Reviewers were moderately positive. One reviewer mentioned Carpenter (2002) and subsequent work. One prominent example of this line of work, which the authors do not cite, is: https://www.isi.edu/publications/licensed-sw/mace/ - from 2013. I encourage the authors to cite this paper. In the discussion, the authors point out this type of work is not *end-to-end* in their sense. However, there's, to the best of my knowledge, a relatively big body of literature on end-to-end approaches that the authors completely ignore, e.g., [0-3]. In the absence of a discussion of this work, it is hard to accept the paper. 

[0] https://link.springer.com/article/10.1007/s10994-013-5411-2
[1] https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7405343
[2] http://www.cs.utexas.edu/~atn/nguyen-acl17.pdf
[3] https://arxiv.org/pdf/1803.04223.pdf
Decision: Reject