Learning from Imperfect Annotations: An End-to-End Approach
Authors: Emmanouil Antonios Platanios
Maruan Al-Shedivat
Eric Xing
Tom Mitchell
Venue:  (2020)

Abstract: Many machine learning systems today are trained on large amounts of human-annotated data. Annotation tasks that require a high level of competency make data acquisition  expensive, while the resulting labels are often subjective, inconsistent, and may contain a variety of human biases. To improve data quality, practitioners often need to collect multiple annotations per example and aggregate them before training models. Such a multi-stage approach results in redundant annotations and may often produce imperfect ``ground truth'' labels that limit the potential of training supervised machine learning models. We propose a new end-to-end framework that enables us to: (i) merge the aggregation step with model training, thus allowing deep learning systems to learn to predict ground truth estimates directly from the available data, and (ii) model difficulties of examples and learn representations of the annotators that allow us to estimate and take into account their competencies. Our approach is general and has many applications, including training more accurate models on crowdsourced data, ensemble learning, as well as classifier accuracy estimation from unlabeled data. We conduct an extensive experimental evaluation of our method on 5 crowdsourcing datasets of varied difficulty and show accuracy gains of up to 25% over the current state-of-the-art approaches for aggregating annotations, as well as significant reductions in the required annotation redundancy.

Under review as a conference paper at ICLR 2020
LEARNING FROM IMPERFECT ANNOTATIONS: AN END- -O--TO APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Many machine learning systems today are trained on large amounts of human- annotated data, Annotation tasks that require a high level of competency make data acquisition expensive, while the resulting labels are often subjective, inconsistent, and may contain a variety of human biases. To improve data quality, practitioners often need to collect multiple annotations per example and aggregate them before training models. Such multi-stage approach results in redundant annotations and may often produce imperfect "ground truth" labels that limit the potential of training supervised machine learning models. We propose new end-to-end framework that enables us to: (i) merge the aggregation step with model training, thus allowing deep learning systems to learn to predict ground truth estimates directly from the available data, and (ii) model difficulties of examples and learn representations of the annotators that allow us to estimate and take into account their competencies. Our approach is general and has many applications, including training more accurate models on crowdsourced data, ensemble learning, as well as classifier accuracy estimation from unlabeled data. We conduct an extensive experimental evaluation of our method on 5 crowdsoureing datasets of varied difficulty and show accuracy gains of up to 25% over the current state-of-the-ard approaches for aggregating annotations, as well as significant reductions in the required annotation redundancy. We further conduct an ablation study to evaluate the effect of both end-to-end learning and instance features and show that both contribute to the performance gains achieved by the proposed method.
INTRODUCTION
The rising popularity and recent success of deep learning has resulted in machine learning systems that rely on large amounts of annotated training data (LeCun et al.) 2015 Wu al.) [0166] Gulshan et al. 2016: Esteva et al.) 2017). The most common, scalable way to collect such large amounts of training data through crowdsoureing (Howe) 2006). Crowdsoureing works well in simple settings where annotation tasks do not require domain expertise- for example, in object detection and recognition tasks in natural images and videos de.g..Deng et al. 2009 Kovashka et al. 2016). However, annotation in specialized domains such as medical pathology requires a certain level of competency and expertise from the annotators which makes annotation expensive. Moreover, often times there is high rate of disagreement even between experts, which results in increasingly subjective and inconsistent labels (Elmore et al. 2015 Hutson et al. 2019).
A typical approach to dealing with subjectivity is to treat each annotation as simply noisy, collect multiple redundant labels per example (e.g., from different annotators), and then aggregate them using majority voting or other more advanced techniques (c.g., Dawid & Skene 1979 Carpenter 2008: Liu et al., 2012, Bachrach et al., 2012) Zhou et al.) 2hou & 2016) to obtain a single "ground truth" label. At the expense of redundancy, this results in better data quality and more accurate estimates of the ground truth. More recently, the emerging systems for data programming and weak supervision also internally rely on label aggregation techniques similar to methods used for solving the crowdsourcing problem. Snorkel (Ratner et al. 2017 Bach et al. 2019) is a popular such system and was designed for efficient and low-cost creation of large-scale labeled datasets using programmatically generated, so-called weak labels. However, as we show in our empirical evaluation none of these systems solve label aggregation effectively in the presence of high subjectivity. We argue that to become more effective, these methods need to make use of meta-data and other types of information that may be available about the data instances and the annotators labeling them.

---

Under review as a conference paper at ICLR 2020
Is there a bone fracture in this image?
No
No
Maybe
Yes
Initialization
Fine-Tuning
Model
Fetimate the ground truthh labels bascd on a majonity vote among the annotators.
Fine-bune the parameters by maximizing the cata marginal log likelinced.
Legend
Oliameed
Compured
Li
EM Optimization
D. dg(an)
C. = cp(ry)
Alternae between the following two steps until converge 1cc:
9 D, C2
yi TRUE LABEL
D
Categorical (sr))
E-Step
M-Step
DIFFICULTY
Categoricai
Compute the expected ground crutn labeis based the current ground and predictor quality modells.
Learn the ground truth and predictor quality models based on the expected ground
Predictors
ViJ
C COMPETENCE
OBSERVATION
QUALITY
PREDICTOR
Figure 1: Overview of the proposed algorithm and probabilistic model.
To this end, we propose novel approach that allows us to train accurate predictive models of the ground truth directly on the non-aggregated imperfectly labeled data. Our method merges the two steps of: (i) aggregating subjective, weak, noisy annotations, and (ii) training machine learning models. At training time, along with learning a model that predicts the ground truth, we also learn models of the difficulty each example and the competence of each annotator a generalizable manner (i.e., these models make predictions for previously unseen examples and annotators). Our approach can be effectively used for training on crowdsourced data as well as on weakly labeled data, and also be used within frameworks such Snorkel (Ratner et al. [207]] Bach et al. 2019) and significantly improve their performance. We propose a method that can:
1. Learn truth estimators: Learn functions representing the underlying ground truth, while impos- ing almost no constraints (as opposed to prior work). In fact, we are able to leverage the capacity of deep neural networks along with the interpretability provided by Bayesian models, in order to obtain highly expressive estimators of the underlying truth.
Learn quality estimators: Learn functions that estimate the quality of each annotator. When annotators can be described by some features (e.g., age, gender, location, etc. of an Amazon Mechanical Turk annotator, instead of just an ID), our quality estimators able to generalize to new, previously unseen, predictors. Previous work only considered estimating accuracies of a fixed set of predictors, without being able leverage any information we might have about them. Furthermore, in contrast to previous work, we are also able to predict the per-instance predictor comptencies (i.e., our method can determine whether a human annotator an expert for a subset of queries, instead of just estimating his/her overall accuracy), which is done by learning dependencies between the instances and the annotators. Finally, approach is able to distinguish between multiple different types of errors by estimating the full confusion matrix for each instance- instance-predictor pair.
3. Be easily extended:
The truth and quality estimators can take arbitrary functional forms and fully leverage the expressivity of deep neural networks.
Both human annotators and machine learning classifiers may sometimes be unable to make predictions about certain aspects of the ground truth (e.g.. human annotators may be unsure about what the correct answer to question). The proposed method is formulated in a way that allows it to be extended such that can also learn decision function estimators for the annotators (i.e., estimators that predict whether an annotator will be able to provide prediction for given data instance). These estimators can have significant implications for data annotation systems where the cost of querying annotators is high (e.g., when these annotators are highly qualified, such as doctors or other kinds of domain experts). This because it allows for better matching annotators instances, thus reducing the required amount of annotation redundancy. An overview of the proposed approach and model is shown in Figure I and detailed description is provided in Section
2

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
Research on label aggregation and crowdsourcing dates back to the early 1970s, when Dawid & Skene (1979) proposed probabilistic model to estimate ground truth labels using the expectation maximization (EM) algorithm. Since then, a variety of generalizations and improvements upon the original method have been proposed (Whitehill et al. 2009; Welinder et al., 2010 Liu al. 2012] Zhou et al.) 2015) Zhou & He (016). One of the central parts of the label aggregation algorithms is estimation of the accuracy of the annotators (or predictors) without having access to the ground truth. This problem has been of independent interest to the machine learning community, was termed as estimating accuracy from unlabeled data and studied by [oolins & Singer (1999). Dasgupta et al. (2011) Bengio & Chapados (2003). |adani et al. (2004), [Schuurmans et al. (2006), Balcan et al. (0113) and Parisi et al.) (2014), among others. Almost none of the previous approaches explicitly considered modeling the ground truth, but rather assumed either some form of independence or knowledge of the true label distribution.
Collins & Huynh (2014) reviewed many methods that were proposed for estimating the accuracy of medical tests in the absence of gold standard. PPlatanios et al. (0144) proposed formulating the problem as an optimization problem that uses agreement rates between multiple noisy annotators. Platanios et al. (2017) improved upon srreement-based accuracy estimation using logical constraints between the noisy labels. Tian & Zhu (2015) proposed a max-maaginn majority voting scheme applied to crowdsourcing. More recently, Khetan et al. (2017) proposed to use parametric function to model the ground truth and showed that the approach can sometimes be functional even in the limit of single noisy label per example. Among recent approaches, [Zhou et al. (2015) formulated the problem as form of regularized minimax conditional entropy and established one of the most competitive baselines on many public crowdsourcing datasets.
Our proposed method generalizes the approaches of Zhou et al. (0155), Platanios et al. (2016). and Khetan et al. (2017). Similar to Platanioo et al. (2016), we define a generative process for our observations. However, our model is able to handle categorical labels, opposed to just binary. Similar to Zhou et al. (2015), we define the confusion matrix for each sestance-predicton pair as a function of instance difficulty and predictor competence However, we explicitly learn the difficulty and competence as functions, which allows us to generalize to previously unseen instances and annotators. Interestingly, the inference algorithm for our generative probabilistic model has a similar form to that of Zhou al. (2015) (except for the explicit learning the ground truth, difficulty, and competence functions). Thus, we also show that the algorithm of Zhou et al. 2015) can be re-derived as an EM inference algorithm for generative model, simplifying the argument of the original paper. Finally, similar to Khetan et al. 2017). we use a parametric function to model the ground truth, and also go step further and propose to use parametric functions to model the instance difficulties and predictor competences. Thus, our approach enables estimation of which annotators are likely to perform better on which instances, potentially enabling more optimal allocation of annotators and thus annotation cost reductions.
In contrast to prior work, our method also allows for end-to-end learning. Prior methods do not allow for this as they (implicitly) separate ground truth inference (i.e., label aggregation) from model training. More specifically, previously one would have to train machine learning model in two stages: (i) infer the ground truth labels from the provided annotations, and (ii) train a machine learning model on the inferred labels. Our approach merges these two stages and allows us to train machine learning models directly on the imperfect annotations. In Section 4.3] we conduct an ablation study that showcases the performance gains obtained by employing end-to-end learning in this fashion.
PROPOSED METHOD
We denote the observed data by D (æ,, J. where }), (Mi, (irjbieM,) h. M; is the set of predictors that made predictions for instance æ;; and ÿt is the output of predictor f3 for instance æ;, Our goal is to learn functions representing the underlying ground truth and predictor qualities, given our observations D.
'The idea of modeling instance difficulties and annotator competencies has been studied before by Carpenter 2008) and Bachrach al. (2012) among others.
3

---

Our goal is to learn functions ho, dæ. and Ca.. given observations D. In order to do that, propose the following generative process for our observations. For i = 1...., N, we first sample the true label for 2; Y/ Categorical((o((z))) Then, for j E M,, we sample the predictor output Under review as a conference paper at ICLR 2020
Ground Truth. We define the ground truth as function ho(2;) that is parameterized by 0 and that approximates the true distribution of the label given z;- In our setting, ha TRCO and (0100(z))) I, where C' is number of values the label can take (i.e., assuming categorical labels). More specifically, The (z;)]ı- å P(y: k æ;), where we use square brackets and subscripts to denote indexing of vectors, matrices, and tensors. For example, he could be a deep neural network that would normally be trained in isolation using the cross-entrope loss function. In our method the algorithm, described in the next section.
network is trained using the ampectation-Mxxmmzaatn
Predictor Qualities. We define the predictor qualities as the confusion matrices Qiy CxC for 20
each instance æ; and predictor fj. where Qigja 1, for all k E (1...., C). [Qu]] kt represents the probability that predictor ʃj outputs label I given that the true label of instance æ; is k. We define these confusion matrix in way that generalizes the successful approach of Zhou et al. 2015
Q₂ = D; 3 Cj+
where ; represents an inner product along the 7th dimension of the two tensors, and:
D; = dg(x);) represents the difficulty tensor for instance I;, where d is a function parameterized by Φb D; E CRCxL and L is a latent dimension (it is hyperparameter of our model). [Di]ez Ja- is L-dimensional embedding representing the likelihood of confusing æ; as having label I instead of k, when k is its true label.
Cj (ry) represents the competence tensor for predictor fy. where is function parameter- ized by s/, T is some representation of fy (c.g., could be one-hot encoding of the predictor, in the simplest case), and C, E CC [C,]u- is an L-dimensional embedding representing the likelihood that predictor f, confuses label l for I, when & is the true label.
Using L>> 1 allows the instance difficulties and predictor competences to encode more information. An intuitive way to think about this is that we are embedding difficulties and competencies in common latent space, which can be thought of as jointly clustering them. This in fact very similar to how matrix factorization methods are used for collaborative filtering in recommender systems.
dij Categorical(( ((:ull -). where [Qijly:- represents the y, th row of Qij. In the next section, we propose an algorithm learning the parameters 0, Φ, and sb.
3.1 LEARNING
widespread approach for performing learning with probabilistic generative models, is to maximize the likelihood of the observed data with respect to the model parameters. Let y {3r J. The complete likelihood of a single observation, Vij, can be derived as follows:
D(D, 9) IP(w II JEM.
p(u 1t),
where P(0; 9:) depends on Q.y- There are two main approaches in which we can use the likelihood function of Equation for forr learning: (i) marginalize out the yı latent variables and then maximize with respect to 0, Φ. and W. or (ii) use the expectation maximization (EM) algorithm originally proposed by Dempster et al. (1977). It has previously been observed that the EM algorithm can perform much better than approach (i) for mixture models (Bishop) 2006), as the latter tends to get stuck in bad local optima. Since our model resembles a Bernoulli mixture model with the latent assignments being defined by the y, 's, we decided to use the EM algorithm The steps of the EM algorithm for our model are as follows:
We also perform a normalization step such that all elements of Qij are non-negative and such that each row sums to 1 (thus making each row a valid probability distribution). In fact, also experimented with the marginalization approach and consistently underperformed EM.
4

---

Under review as a conference paper at ICLR 2020
E-Step. We need to compute the expectation of ys given D and y (which denotes all of y except for y:), for = l,..., N, and we know that:
N P(V.))
p D. x y)
II p(ge) Vs). JEM.
(3)
Therefore, by removing all terms that do not depend on y/ and normalizing, we obtain the following expectation (which we compute during this step, while keeping 6, Ø, and & fixed)
A
1Qylaa
B [m p(u: k (.
where X = [ha(<;)]
II JEM,
Ei= Qiylinn
1=
(4)
For brevity, in what follows, we use the following notation // Ey D {1 (y.
M-Step. We maximize the following log- ligliiil function with respect to 6, ø, and eb, while using the values of iz computed in the last E-step:
C I1 p(u ú.))
II JEM.
p(us y: ūi)
(5)
log L
logp(ur ū.))
log p(uss 18 ūz)
i=1
i=1
JEML
N log L = Σ
N
of log(ho(z:))
+
log
E 1
=1
JEMM,
The training procedure for learning the parameters 0, i, and vú consists of iterating over the E-step and the M-step shown above, until convergence, where convergence can be measured by computing the change in the parameter values across learning iterations. It is important to note that EM finds local optima of the likelihood function, and so the starting point can play an important role. Also, as Platannos et al. (016) mention, there exists an inherent symmetry in our model that can be problematic. The likelihood of any observed data is the same we flip the true underlying labels and the predictor qualities (i.e., set y/;
llpped = 1 - Q3y). ). We would like to somehow [lipped 1 y; and Q encode the prior assumption that most of the predictors are correct most of the time. One way to do this is by choosing the starting point of the EM algorithm carefully.
Initialization. In the E-step shown in Equation underlying labels, yt- We can encode the assumption mentioned in the previous paragraph by replacing the first E-step with a majority vote among the predictors:
compute the expected values of the true
Enjo D (1 [Imm= ΣjEM. [(ou=N] [.M
where IM& denotes the size of set M,. We initialize the EM algorithm by replacing the first E-step with this majority vote approximation. As we show our experiments, this helps us avoid the aforementioned symmetry, and thus we refer to this initialization scheme as symmeetry-beeaking initialization. Note also that in the case where the predictors provide with F(gi = k). instead of a single categorical value, we can still use this initialization scheme by replacing [0,, with P00 = k)), in Equation g
Marginal Likelihood Fine-Tuning. In experiments found that maximizing the marginal likelihood function after EM converges tends to improve performance. We refer to this step as marginal likelihood fine-tuning. More specifically, after the values of the parameters 0, ø, and v, converge to fixed values across multiple EM steps, we solve the following maximization problem using these fixed values as the initial point:
C The (2: (xx)a
Quiskens
max Σ P(D.y)
max R.G..
Σ log
(9)
_ Qijltgss
JEM;
1=1
NNot that QUy depends and sb.
5

---

Under review as a conference paper at ICLR 2020
3.2 INSTANCE AND PREDICTOR REPRESENTATIONS
A major advantage of the proposed approach over prior work is that we learn models of the ground truth and the predictor qualities functions of some representations (i.e., representations of the data instances, T; and of the annotators, ry). It is thus important to define these representations. For many problems, the representations of the data instances can be defined in the same manner as was previously done when performing supervised learning (e.g., we can directly use raw pixel values representing images). However, predictor representations are introduced here for the first time simple approach would be to use one-hot encoding of the predictors. However, this would not allow for any amount of information sharing across predictor (e.g., what if two predictors are very similar). We know from prior work that modeling dependencies between the predictors can be very important (e.g., Platanios et al. 2016). One way to allow for that is to learn vector embedding representations for the predictors, which would be implicitly equivalent to clustering them. Ideally, one would want to use any available information about these predictors (e.g., Amazon Mechanical Turk annotators could be described by their age, location, etc.). Unfortunately, we could find any public datasets that provide such information about the predictors/annottttor,, and hence we used embedding representations in our empirical study.
3.3 DISCUSSION
The approach we have proposed in this section can be thought of as introducing a new loss function for training the model he using multiple imperfect labels per training instance, each coming from different sources. This new loss function introduces latent variables representing the ground truth labels, as well couple of auxiliary models that are learned, and which represent the instance difficulties and predictor competences. We also proposed an EM-based algorithm to minimize this new loss function as well as initialization scheme. Perhaps most interestingly, a key difference between this approach and previous work that we are able to explicitly learn functions that output the likelihood that a predictor will label specific instance correctly. This enables using this approach to perform crowdsoureing more actively by assigning annotators to instances they are more likely to label correctly, thus helping reduce redundancy and drive costs down.
3.4 EXTENDING TO MULTI-LABEL SETTINGS
Our method can easily be extended handle settings where we have multiple categorical labels that can be assigned to each instance. In that case, the model per label is defined in the same way as previously, except that now the functions he. dg. and cy also take as input a representation for the label (e.g.. a label embedding). This allows us to share information across labels and can be thought of generalization of the approach by Platanios et al. (2016), where information is shared by clustering the labels. Furthermore, it allows us to use the proposed method in extreme classification settings (e.g.. Prabhu & Varma 2014) or settings where the number of labels is not fixed and known a priori and can keep increasing (e.g.. face recognition; Weinberger & Saul 2009 Liu et al. (016). This is made possible by learning label representations and then letting the difficulty and competence functions also take as input a pairs of labels and return a vector instead of a three- dinee-dimensional tensor. In the next section, we show how learning label representations can significantly enhance the robustness and performance of our approach.
EXPERIMENTS
We evaluate the proposed approach on multiple datasets from the crowdsourcing domain, all of which have ground truth labels, (multiple) subjective annotations for each example, as well as information on who provided each annotation (i.e., the annotator ID):
1. Blue Birds (BB) WWelinder et al. 2010); : Bird photos labeled as Indigo Bunting or Blue Grosbeak.
2. Word Similarity (WS) (Snow et al. 2008): Pairs of words labeled as similar or dissimilar.
3. RTE (Snow et al.) 2008): Pairs of sentences labeled as whether the first entails the second.
4. Medical Causes (MC) (Dumitrache et al. 2018): Sentences that contain 2 medical terms labeled if one of the terms causes the other (e.g., pancreatic adenocarcinoma causes weight loss).
'We note that learning representations of the agent behaviors has been recently explored in the context of imitation and reinforcement learning (c.g., Grover al. 2018).

---

Under review as a conference paper at ICLR 2020
Table 1: Statistics for the datasets we used in experiments. "#Predictors" refers to the total number of predictors in the dataset, "Average Redundancy" " refers the average number of predictions provided for each instance, "Average Accuracy" refers to the average predictor accuracy, and "Random Accuracy" refers to the accuracy obtained a completely random predictor.
Average Redundancy
Average Accuracy (%)
Random
Dataset
#Instances
#Predictors
Accuracy (%
BLUE BIRDS (Welinder 2010
108
39
39
63.56
50.00
WORn SIMI ARITY (Snow 2008
30
10
10
81.33
50.00
RTE (Snow al. (000)
800
164
10
84.13
50.00
MEDICAL CAUSES Dumitrache or X018
3,984
408
15
32.40
7.00
MEDICAL TREATS Dumitrache ot
2018
3,984
15
38.88
7,00
5. Medical Treats (MT) (Dumitrache et al. 2018): The same sentences with medical terms labeled if one of the terms treats the other (e.g.. aspirin treats pain).
The last two datasets are in fact part of a single dataset on medical relations, and thus we are able to perform experiments both with the single task formulation of our algorithm and the multi-task formulation. As we discuss at the end of this section, this allows us to show how our approach can be used share information across labels and improve the quality of the learned models.
Statistics for these datasets are provided in Table Note that these datasets were provided without the associated features of the annotator identifiers and thus are unable to evaluate the usefulness of annotator features (we instead learn embeddings for the annotators). Unfortunately we were unable to obtain any crowdsourcing datasets with associated annotator meta-data. This is probably due to the fact that no prior method is able to make use of such information. However, we do make use of instance features for all of these datasets. In cases where such features were not readily available, we manually computed them by using pre-trained machine learning models. We are making all such features and annotator identification information publicly available in a standardized format. More details are provided in our code and data repository which is available hletp://ancnyoon/
4.1 EXPERIMENTAL SETUP
We perform experiments using the following two variants of our approach:
LIA: A version of our method which uses instance and predictor features specific to each dataset. When features are not available for the instances and/or the predictors, we learn embeddings of size 16 which are initialized randomly and optimized along with the other model parameters during the M-step (see Section 3.T.
LIA-ML: A multi-label variant of the aforementioned method. This method is only used with the medical relations datasets. In this case, we consider all medical relations included the dataset jointly and only evaluate on the two for which the ground truth is provided (i.e., "causes" and "treats"). We use this method variant in order to show how our approach can effectively share information across labels.
In both instances of LIA. he and dg are multi-layer perceptrons (MLPs) with 4 layers of 16 hidden units each, with the only exception being the medical relations dataset where we used 32 units for each layer. Gp is always modeled MLP sizes, we did not perform an extensive search to choose these values; we rather performed a small grid search and selected the number that resulted in the highest validation data likelihood. We compare against the following baselines for ground truth estimation:
linear function. Note that for both the embedding sizes and the
MAJ:
Simple majority voting. Note that we use soft majority voting whenever possible, i.e., we use soft labels (probabilities or confidence scores) whenever the predictors provide them, instead of always thresholding them to obtain discrete labels.
MMCE: Regularized minimax conditional entropy by /hou et al. (2015), which has been shown to outperform alternatives. We consider it the current state-of-thee -hhe-ar for crowdsourcing.
Snorkel: A method originally designed for aggregating annotations of programmatic weak predictors proposed by Ratner et al. (2017). which is part of a popular software package that allows for subsequent training of machine learning models on the aggregated data.
MeTaL: Successor to Snorkel, proposed by |atneer et al. (2018). For both this method and for Snorkel we use the original implementation provided by the authors
https: tww.. http:////gggthuub.ooo/aaaaaeeeee wtt.h//w...oooee.eeeeeeeeeecc//e https///////ttoooooooa/oaaeeeeee
7

---

Under review as a conference paper at ICLR 2020
Table 2: Accuracy across varying levels of redundancy, for all datasets we used in experiments. For each experiment, we report mean accuracy and standard error over 50 from random initializations. The best results are shown in red color (the best results within each sub-group are shown in bold). The methods marked with a "x" are used for the ablation study of Section
Accuracy (%)
MAJ
MAJ"-E
MAJ'
MMCE
MMCE
Sworkel
MeTal
LIA-E
LIA
LIA-ML
63,9=0.5
64.4
65.1
66.9
63.2
63.2
6388
63.0::07
65.1:2066
71.520.4
71.1:04
70.9:005
70.9
73.91:0:5
74.2::04
73.2
72.4:06
71,5106
73.0:0044
76.2:00.5
74.41:04
0.
75.4:005
76.5::0.3
76.9:04
37.0
76.01004
83.0.:03
78.12:0
83.1:002
76.4
76.2::0.3
76.1::022
78.5
77.74:04
77.71:04
76.01:03
87.0.10.1
77.2::0.3
90.0:10.3
75.9
0..4
78.4
79.61:0.1
78.8
78.8
76.01000
89.0::0.0
78.9:10.0
93.0:10.6
Datase & Rechundancy
82.8
87.2
87.7
80.1
79.3::04
80.0=004
76.2::07
76.0::1
87.7=2066
88.7:0077
0..6
01
91.4
87.0
84,4
84.0
76.3
85.1
87.7
92.7:0044
10
0.2
030
93.3
90.2:10.33
80.0:000
0..0
76.3
93.1::00
87.7:
96.3:10
72.8 0..2
72.8
74,5
75.3
77.3
03.
65.2:1:3
61.02:033
76.7:20.3
78.0:0033
RTTE
84.8 0.1
0..0
84.8 1011
88.5 0:03
88.9 0..2
85.
79.1
72.4
84.3:00.1
89.1:00.1
10
00.0:0011
89.9 0..1
1:01
92.7
87.1
90.0 1:00
78.0::0.0
91.6:10.00
93.1:10.11
26.8
24.2
26.5=20.1
29.1 +::02
29.3
22.0=
27.1
25.3+
25,0=t0.2
29.5
30.1
MC
24.1 1:011
23.6
24.2::0.1
24.5
24.6
21.3
24.0
21.02:0.1
24.0:20.1
30,9:003
36,4
10
24.1 -0011
23.6:20.1
24.1
24.41:0.1
24.6
20.1
24.0::00
20.0::00
23.6:20.1
30,5:00
34.14003
33.8 +..4
35.7 0..4
34.2:4002
35.3--0.3
38.6 60.3
34,2-40.4
33,3+-0.1
22.1+-0.3
34.0++0.3
38.640044
40.8-103
00.3
34, 1:0.
33.6
36.8+-0.2
37.0
35.0=004
34.0
21.0+:0.3
33.0:10.2
38.3+002
46.1
34.2
3..3
35.2::0.2
38.5
38.3
36.3
35.0+b0.1
03.1::0.1
33.8:b0.2
42.1=002
45.4:b04
Aside from these baselines, we also perform experiments using the following custom methods that we designed for the purposes of performing ablation study (the study is presented in Section 4
LIA-E: In order to evaluate the usefulness of instance features, we learn embeddings of size 16 for the instances instead of using their features.
-A:'-E: A two-step method that resembles how machine learning models are currently being trained when using crowdsourced data. First, we estimate ground truth using MAJ. Next, we train the he model used in LIA-E directly on the aggregated labels.
-MA':: Same as MAJ'-E, except that we use the model hig of LIA (i.e., making use of instance
features instead of learning instance embeddings).
.MMCEE- -E: Same as MAJ' -E, but with MMCE used for label aggregation.
.MMEE: Same as MAJ', but with MMCE used for label aggregation.
During each M-step we use the AMSGrad optimizer (Reddi et al., 2018) to maximize the log- likelihood function with the learning rate set to 0.001, and we perform 1,000 optimization iterations using a batch size of 1,024. Overall, we perform 10 EM iterations (all models did converge within that limit) with warm starting (i.e., the model parameters are always initialized to the values obtained during the previous M-step). When using LIA with image instances we use as image features the activations of the last layer of pre-trained ResNet- 101 Convolutional Neural Network (CNN). Similarly, for all text instances we use as text features the representations provided by a pre-trained BERT model (Devlin et al. 2018). More details on our setup and the model hyperparameters can be found in our code repository at httt::://nonmmmo//
We evaluate all methods by computing the accuracy of the predicted instance labels. This is common metric for evaluating crowdsourcing methods and it also implicitly measures the quality of the confusion matrices predicted by our model. This is because these confusion matrices heavily influence the supervision provided to the ground truth model, he, while training. Furthermore, instead of just computing accuracy for the full datasets, we also measure how performance varies as a function of redundancy- the maximum number of annotations provided per instance. In order to limit the redundancy for existing datasets we randomly sample subsets of the provided annotations. Performing well in low redundancy settings is very important because it can result in significantly reduced crowdsourcing costs.
4.2 RESULTS
Our results are presented in Table 2 certain cases (e.g., in Blue Birds) are able to boost accuracy over the best alternative method by 14%, thus establishing a new state-of-the-art for this dataset. In the multi-task setting, where we train the L.I--MI. model to jointly infer ground truth for both Medical Causes (MC) and Medical Treats (MT)
LIA methods consistently outperform alternative approaches. In
8

---

Under review as a conference paper at ICLR 2020
while sharing the representations of instance difficulties and annotator competencies. We observe that multi-task training boosts performance by more 8% absolute (or over 20% relative) over the single task counterpart, outperforning the baselines by over 25% relative. Finally, our approach can obtain the performance of the best alternative method using up to 4 times less redundancy, which can have significant implications for the cost of crowdsoureing. especially when annotation requires domain expertise (e-g., healthcare). We note that Snorkel and MeTal, tend to perform well overall, but sometimes fail entirely (often performing on par with or worse than majority voting) MeTaaL also suffers from calibration issues, as often achieves very low accuracy while having reasonable mean average precision. Data programming systems could thus benefit significantly by integrating our method in their pipeline, tying together the label aggregation and model training phases.
4.3 ABLATION STUDY
Our main contributions are: (i) end-to-end learning by fusing the label aggre- gation and model training phases, and (ii) allowing for instance and annotator features to inform label aggregation
Table Results for LIA without marginal likelihond fine-tuning.
[ In this section, we show how each one of these contributions is important on its own by performing experiments where we introduce each one on their own, while keeping everything else constant.
Accuracy (%)
End-to-Fane End Learning. The best way to test the effectiveness of end-to-end learning is to compare end-to-end approaches with two-stage approaches where: (i) we first aggregate labels, and (ii) we then train machine learning models using the aggregated labels. To this end, we introduced the baseline methods marked with a "x" in Table 2 The results indicate that the two-stage approach underperforms the base label aggregation method for both MAJ and MMCE. This is most likely due to the fact that in both these cases the model being learned cannot inform the label aggregation stage. In contrast, LIA is able to outperform all two-stage approaches because it allows for exactly that. Note that MMCE models instance difficulty and annotator competence similar to LIA, with the exception that does not use instance features and it does not allow for end-to- end learning of the ground truth predictor. Also note that LIA-E does not use instance features. but is still able to outperform MAJ' -E and MMCE* -E in many cases, indicating that end-to-end learning is, in fact, effective and accounts for at least part of the performance gains achieved by LIA.
71.9
74.9=2044
76.9
77.5:20.4
79.1:20.2
Dutased & Redundancy
88.7 78.77
ws
91.0-b006
10
93.3:t000
63.8.10..
RTF
67.5
10
69.6-+-.1
24.0:20.2
MC
23.9:1002
23.0+t0:2
38.1:1033
MT
38.5
39.2-40.11
Instance Features. The methods which use instance features are LIA, MAJ' and MMCE'-M To test for the usefulness of these features, we provide variants of these methods (labeled LIA-E, -A and MMCE" -E, respectively. in Table 2 that use indicator features instead and learn instance embeddings. We observe that for MAJ and MMCE the results are inconclusive (the seature-based methods outperform the alternative in around half of the experiments). However, in the context of end-to-end learning, we observe that LIA consistently outperforms LIA-E by a significant margin. This indicates that instance features are indeed useful, especially so in the context of end-to-end learning where they can inform the label aggregation phase.
It important to also mention that results pertinent to this ablation study for the Word Similarity (WS) dataset were a bit unstable with many models effectively failing to learn anything meaningful (specifically MMCE" -E, MMCE" -M, and LIA-E). Our best explanation for this is that the Word Similarity (WS) dataset is very small with only 30 instances and is thus highly prone to overfitting.
Finally, in order evaluate the effect of the marginal likelihood fine-tuning approach presented in Section 3.1 we also run experiments using LIA without this fine-tuning phase. The results are shown in Table Jand it is clear that marginal likelihood fine-tuning results in better performance. Also, in order to sanity check that LIA is able to predict the qualities of the predictors accurately, we also performed synthetic experiment. More specifically, we added an "always correct" and an "always wrong" oracle to all datasets used in our experiments. It turns out the predicted qualities for the two oracles are the highest and lowest among all predictors, respectively. This indicates that our model is indeed capable of uncovering such highly competent and incompetent predictors.
"This behavior has also been observed by others (e.g.. hnttps://githhh http://hhhbbb.omm/laaaaeeeeeaaa
As mentioned in the beginning of this section, we were unable obtain any crowdsourcing datasets with associated annotator meta-data and thus our experiments we only use instance features.

---

Under review as a conference paper at ICLR 2020
MISTAKES
FALSE POSITIVE RATE
FALSE NEGATIVE RATE
Blue Birds
Word Similarity
Cluster Cluster 02
High
Cluster Cluster
learned predictor embeddings. Each the figures represents a predictor projected on 2D plane using UMAP. One the left, the predictors were first clustered based on which instances they make mistakes on and then colored and shaped based on which cluster they belong to (in the embedding space, predictors that make similar mistakes tend to cluster together). In the middle. the predictors are colored based on their false positive rate. On the right, they colored based their false negative rate.
Figure Visualization
4.4 PREDICTOR EMBEDDINGS VISUALIZATION
To evaluate whether the learned predictor embeddings are meaningful in some way, we perform dimensionality reduction using UMAP (McInnes et al., 2018). plot them in Figure [ and color predictors in three different ways, which lets us understand the information captured by the manifold:
1. Mistakes Cluster: To cluster predictors, represent each with one-hot vector that indicates the instances made mistakes on, and then run agglomerative clustering with L₃ distance metric. On the plot, each cluster is associated with unique shape.
2. False Positive Rate: Each predictor is colored based on its false positive rate.
3. False Negative Rate: Each predictor is colored based on its false negative rate.
We have provided figures for Blue Birds and Word Similarity datasets which are the only ones for which all predictors annotated all instances (it is unclear how to properly compute the mistakes clustering distance metric when some or most annotations are missing). From these plots, clear that the learned predictor embeddings encode both the expertise of the corresponding predictor as well as the likelihood of making a false positive false negative mistake.
5 CONCLUSION
In this paper, we have introduced learning framework for: (i) training deep models directly on data with imperfect annotations, and (ii) modeling the processes that produced the labels. Our approach improves upon the classical and widely used two-stage setup (first aggregate and denoise the labels, then train the model) by merging the two stages. As a result, we are able to train models end-to-end using multiple noisy labels, while estimating the difficulties of the examples and learning accurate representations for the annotators that produced the labels. Experimental results multiple small and large scale publicly available crowdsoureing datasets indicate that our method results in significant gains in accuracy (up to 25% relative gain over the current state-of titee--f-the--ee the approaches for aggregating noisy labels). Moreover, it turns out that training the model to predict multiple related labels simultaneously improves the learned representations and results in further gains in predictive performance of the model. Finally, performed ablation study evaluate the effect of both end-to-end learning and instance features and showed that both contribute to the performance gains achieved by the proposed method.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, et al. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In Proceedings the the 2019 International Conference on Management of Data, pp. 362-375. ACM, 2019.
Yoram Bachrach, Thore Graepel, Tom Minka, and John Guiver. How to grade test without knowing the answers-a bayesian graphical model for adaptive crowdsourcing and aptitude testing. arXiv preprint vrXiv:/206.6386. 2012.
Maria-Florina Balcan, Avrim Blum, and Yishay Mansour. Exploiting Ontology Structures and Unlabeled Data for Learning. International Conference Machine Learning, pp. 1112-1120, 2013.
Yoshua Bengio and Nicolas Chapados. Extensions to Metric-Based Model Selection. Journal of Machine Learning Research, 3:1209-1227, 2003.
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Verlag, Berlin, Heidelberg, 2006. ISBN 0387310738.
Bob Carpenter. Multilevel bayesian models of categorical data annotation. Unpublished manuscript, 17(122):45-500 2008
John Collins and Minh Huynh. Estimation of Diagnostic Test Accuracy Without Full Verification: A Review Latent Class Methods. Statistics in Medicine, 33(24):4141 -1699, June 2014.
Michael Collins and Yoram Singer. Unsupervised Models for Named Entity Classification. In Joint Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, 1999.
Sanjoy Dasgupta, Michael L Littman, and David McAllester. PAC Generalization Bounds for Co-training. In Neural Information Processing Systems, pp. 375-382, 2001.
A P Dawid and A M Skene. Maximum Likelihood Estimation of Observer Pator-Rates Using the EM Algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics). 28(1):20-28, 1979.
Arthur Dempster, Nan Laird, and Donald Rubin. Maximum Likelihood from Incomplete Data via the EM Algorithm. Journal of the Royal Statistical Society. Series B (Methodological), pp. 1-38, 1977.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-244824 225.255.222 Icee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv preprint PrXiv:/810004800 2018.
Anca Dumitrache, Lora Aroyo, and Chris Welty. Crowdsourcing ground truth for medical relation extraction. ACM Transactions on Interactive Intelligent Systems (TiiS), 8(2):12, 2018.
Joann G Elmore, Gary M Longton, Patricia A Carney, Berta M Geller, Tracy Onega, Anna NA Tosteson, Heidi D Nelson, Margaret S Pepe, Kimberly H Allison, Stuart J Schnitt, et al. Diagnostic Concordance among Pathologists Interpreting Breast Biopsy Specimens. Journal of the American 2015.
Medical Association,
Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature, 15, 2017.
Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yura Burda, and Harrison Edwards. Learning policy representations in multiagent systems. arXiv preprint arXiv: / 886.064644 2018.
11

---

Under review as a conference paper at ICLR 2020
Varun Gulshan, Lily Peng, Mare Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al. Development and Validation of Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. Journal of the American Medical Association, 316(22):2402-244) 2016.
Jeff Howe. The rise of crowdsoureing. Wired magazine, 14(6):1-4, 2006.
Miles Hutson, Olga Kanzheleva, Caitlin Taggart, Bilson Campana, and Quang Anh Duong. Quality control challenges in crowdsoureing medical labelingand diagnosis. 2019.
Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from Noisy Singly- Data. arXiv preprint arXiv: 2017.
Adriana Kovashka, Olga Russakovsky, Li Fei-Fei, Kristen Grauman, et al. Crowdsourcing in computer vision. Foundations and Trends@ in computer graphics and Vision, 10(3):177-243, 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(5333) 2015.
Qiang Liu, Jian Peng, and Alexander T Ihler. Variational inference for crowdsourcing. In Advances in neural information processing systems, pp. 692-700, 2012.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-Margin Softmax Loss for Convolu- tional Neural Networks. In Proceedings of the 33rd International Conference on International Conference Machine Learning, volume 48, pp. 507-516, 2016.
Omid Madani, David Pennock, and Gary Flake Co-Validation: Using Model Disagreement Unlabeled Data to Validate Classification Algorithms. In Neural Information Processing Systems, 2004.
Leland McInnes, John Healy, and James Melville. Umap: projection for dimension reduction. arXiv preprint arXiv: 1802.03426, 2018.
Uniform manifold approximation and
Fabio Parisi, Francesco Strino, Boaz Nadler, and Yuval Kluger. Ranking and combining multiple predictors without labeled data. Proceedings of the National Academy of Sciences, 2014.
Emmanouil Platanios, Avrim Blum, and Tom M Mitchell. Estimating Accuracy from Unlabeled Data. In Conference on Uncertainty in Artificial Intelligence, pp. 1-10, 2014.
Emmanouil A Platanios, Avinava Dubey, and Tom M Mitchell. Estimating Accuracy from Unlabeled Data: A Bayesian Approach. In International Conference in Machine Learning, pp. 1416-1425, 2016.
Emmanouil A Platanios, Hoifung Poon, Eric Horvitz, and Tom M Mitchell. Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach. In Neural Information Processing Systems, 2017.
Yashoteja Prabhu and Manik Varma. FastXML: A Fast, Accurate and Stable Tree-Classifier for Extreme Multi-Label Learning. In Proceedings the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 263-272. ACM, 2014.
Alex Ratner, Braden Hancock, Jared Dunnmon, Roger Goldman, and Christopher Ré. Snorkel MeTaL: Weak Supervision for Multi-Task learning. Proceedings of the Second Workshop on Data Management for End-To-Erd Machine Learning, pp. 3, 2018.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid Training Data Creation with Weak Supervision. Proceedings of the VLDB Endow- ment, 11 (33):269-282) 2017.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the Convergence of Adam and Beyond. In International Conference on Learning Representations, 2018. URL htttps:://pppeeeeeeee
Dale Schuurmans, Finnegan Southey, Dana Wilkinson, and Yuhong Guo. Metric-Based Approaches for Semi-Sune -uperruper Regression and Classification. In Semi- Supervsuee Learning. 2006.
12

---

Under review as a conference paper at ICLR 2020
Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y Ng. Cheap and Fast But is Good? Evaluating Non-Expert Annotations for Natural Language Tasks. Proceedings of the Conference Empirical Methods in Natural Language Processing, pp. 254-263, Association for Computational Linguisties, 2008.
Tian Tian and Jun Zhu. Max-Margin Majority Voting for Learning from Crowds. Neural Information Processing Systems, 2015.
Kilian Q Weinberger and Lawrence K Saul. Distance Metric Learning for Large Margin Nearest Neighbor Classification. Journal of Machine Learming Research, 10(Feb):007-244, 2009.
Peter Welinder, Steve Branson, Pictro Perona, and Serge Belongie. The Multidimensional Wisdom of Crowds. In Advances in Neural Information Processing Systems, pp. 2424-2432, 2010.
Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R Movellan, and Paul L Ruvolo. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in neural information processing systems, pp. 2035-2043, 2009.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1e0..000944444 2016.
Dengyong Zhou, Qiang Liu, John C. Platt, Christopher Meek, and Nihar B. Shah. Regularized Minimax Conditional Entropy for Crowdsourcing. CoRR, 0bs//503300740,, 2015.
Yao Zhou and Jingrui He. Crowdsourcing via tensor augmentation and completion. In IJCAI, pp. 2435-2441, 2016.
13


Reviews:
[{"review_id": "rye3iuF1qH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update after author response: \nI would like to thank the authors for their thoughtful response, and for addressing some of the concerns raised by the reviewers. One of my main complaints arose from a misunderstanding that none of the baselines model worker competencies and task difficulty. The authors clarified that MMCE does model those, making it a competitive baseline. Given that and the other additions to the draft, I am changing my assessment from 3 to 6.\n--------------------------- \n\nIn this paper, the authors extend the classical probabilistic model of Dawid-Skene (DS) for predicting the final label of crowdsourced tasks. The extensions include explicit modeling of image difficulty and worker competence as a function of image and worker features respectively, as well as a more expressive formulation of learned functions in terms of neural networks (NNs). The authors show that the proposed approach outperforms several baselines on 5 different datasets.\n\nThe problem studied in the paper is relevant since more and more data is needed to train ever more complex models, and often crowdsourcing is the way to generate such data. The paper is clearly written and conveys its central idea concisely. I liked the paper but I believe the contribution to be incremental in its current state. Here are the main issues in my opinion:\n\n1. The presentation suggests that somehow the proposed approach is novel in its end-to-end framework. However, the central idea is very similar to Dawid-Skene (1979), and its subsequent augmentations like the ones in Carpenter (2008) that model both image difficulty and worker competence. These previously proposed models are also end-to-end approaches so that they can infer worker competence and image difficulties while also outputting a final label. Therefore, I believe the novel contributions are then slightly different formulation of these variables, and using NNs to learn the function parameters. I see that the proposed model can be somewhat more general since it uses semantic image features (as opposed to indicator features) but then it learns worker embeddings starting from random initializations which will not generalize to new workers, necessitating a re-run of the whole EM loop.\n\n2. If Q is a stochastic matrix, its rows should sum to 1. I don’t understand the summation on Q in eq. 4. I am also a little confused by eq. 5 since the joint likelihood should just be the prior times the conditional likelihood of the observed annotations. At least trying to do it in my head doesn’t lead to what’s presented in eq. 5. It may just be an issue of notations which can be simplified by using conditional distributions instead of expectations of indicator random variables (same quantity). \n\n3. As far as I understand, none of the baselines considered explicitly model the task difficulty and worker competence. Therefore, the proposed model enjoys this extra level of expressiveness making its superior performance relatively unsurprising. I skimmed Zhou et al. (2015) and simple DS was already competitive on some of the datasets so I would think inclusion of its successors that model image difficulty and worker competence could perform quite well. \n\n3. Minor issue but eq. 6 is not majority vote as stated just above. It’s the maximum likelihood estimate P(y_i = k | D) assuming each worker is an unbiased estimator of the true label. \n\n4. Minor formatting issues: “learn a models” (sec 3.2), “lantent variables” (sec 4) “and by merging” (sec 5). \n\nIn summary, even though I like that the paper is clearly written and tackles an important problem, I am not convinced that the contribution is substantial or the experiments very insightful. I would recommend addressing these issues and resubmitting the paper.\n\nCarpenter, B. (2008). Multilevel bayesian models of categorical data annotation. Unpublished manuscript, 17(122), 45-50.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SklG7TV3cr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission addresses a problem with collecting ground truth: human annotations are noisy, a common approach is to collect many annotations and apply majority voting to elicit a single label. With this approach, the annotators' expertise and the difficulty of single data instances is ignored. What the authors propose is a framework which allows one to combine a direct graphical model of how human annotations are produced with model training. The graphical model introduces latent variables for the difficulty of an instance and the competence of an annotator as well as the (unobserved) true label. This way one can potentially benefit from the meta-information about the annotators (e.g., their demographics) and improve upon the majority-voting baseline of aggregating available annotations. Maybe most importantly, the proposed framework allows one to get the true label with fewer annotations (significantly reduces redundancy). \n\nThe proposed model is intuitive, the learning of the hidden variables is done with EM, the presentation is easy to follow. The experimental part is done on five annotation tasks (image classification, NLP, bio-NLP) and compares the proposed model (LIA) with six prior approaches (e.g., majority vote, MMCE, Snorkel). The evaluation metric is accuracy (i.e., guessing the true label as obtained from experts). Overall, the new model achieves higher or comparable accuracy to that of MMCE which in turn outperforms all other methods. W.r.t. redundancy reduction, on some tasks LIA achieves much better accuracy with fewer annotations. For example, on the word similarity task the accuracy with only two annotations is higher than that of the majority baseline with ten. \n\nThe submission is well-written and I enjoyed reading it. I am not an expert in this area, but as far as I am concerned the contributions are sufficient to accept it. I have not found any technical or methodological flaws. However, I have the following questions and concerns:\n\n1. The analysis part is very short and while the accuracy numbers are impressive, I wonder if a different experiment is needed to fully demonstrate the claimed benefits of the model. For example, I am not sure which part shows empirically that annotators' features are indeed used and useful. \n\n2. Related to the above point, maybe one could create a synthetic dataset where the true labels and true noise are added as if coming from two additional annotators and show that the model can identify them as highly competent / incompetent? \n\n3. Section 3.1 mentions a fine-tuning procedure in the end but the experimental part does not specify how much of a gain it delivered. How does the model perform without this fine-tuning? \n\n4. I have not followed this topic much but it seems to me that there should be more related work on modelling annotators' competence and item difficulty for crowd-sourced annotations. Isn't, for example, work by Bachrach et al. 2012 [1] relevant? \n\n5. How stable are results along the redundancy dimension? For example, the word similarity task has only 30 word pairs with ten ratings per item. How much, if at all, is accuracy with redundancy@2 affected by using different samples of two? \n\nMinor typos:\n - \"can be can be\" in Sec. 1 on page 2.\n - \"a models\" in Sec. 3.2. on page 5.\n - \"a sources\" in Sec. 3.3 on page 6.\n - \"LIA-E\" (should be \"LIA\"?) on page 7.\n\n[1] \"How To Grade a Test Without Knowing the Answers — A Bayesian\nGraphical Model for Adaptive Crowdsourcing and Aptitude Testing\" ICML'12.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hkg9lOl6tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for dealing with noisy human annotated in training data. The idea is to unify the annotation aggregation and model training whilst modelling the sample annotation difficulty and annotator competency level. The experiments on five datasets show improvements over a number of baselines.\n\nThis is a solid piece work that deals with a very practical problem – training ML models with crowdsourced data with imperfect annotations. The proposed method is not completely new: many ideas have been adopted from previous works. However putting everything together seems to work as demonstrated by the experiments.\n\nI have a number of concerns:\n\n1, One of the main claimed novelties of the paper is an “end-to-end” approach that unifies ground truth label prediction and label aggregation. However there is no ablation study to show that this is indeed better than a two-stepped variant with everything else remaining the same. \n\n2, The authors made a claim on the ability to estimate the quality of the annotator. But the statement is the Introduction is misleading – it suggests that the attributes of the annotator such as age and gender will be used as input to the estimator, but later it is clear that no benchmarks contain that information hence it is not implemented. Also it is not clear how this vector embedding (Sec. 3.2) is implemented. Any evidence that this embedding is indeed a clustering index?\n\n3, In the implementation, the strong BERT model was used for language but VGG was used for image representation. Stronger CNN feature extractors such as ResNet101 should be used. \n\n4, In general, the lack of any ablation study is a problem for analysing why the model works.  \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel way of jointly modeling annotator competencies and learning from imperfect annotations. Reviewers were moderately positive. One reviewer mentioned Carpenter (2002) and subsequent work. One prominent example of this line of work, which the authors do not cite, is: https://www.isi.edu/publications/licensed-sw/mace/ - from 2013. I encourage the authors to cite this paper. In the discussion, the authors point out this type of work is not *end-to-end* in their sense. However, there's, to the best of my knowledge, a relatively big body of literature on end-to-end approaches that the authors completely ignore, e.g., [0-3]. In the absence of a discussion of this work, it is hard to accept the paper. 

[0] https://link.springer.com/article/10.1007/s10994-013-5411-2
[1] https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7405343
[2] http://www.cs.utexas.edu/~atn/nguyen-acl17.pdf
[3] https://arxiv.org/pdf/1803.04223.pdf
Decision: Reject