Venue: NAACL 2019

# Modelling Instance-Level Annotator Reliability for Natural Language Labelling Tasks

Maolin Li\({}^{1,2}\), Arvid Fahlstrom Myrman\({}^{2}\), Tingting Mu\({}^{2}\) and Sophia Ananiadou\({}^{1,2}\)

\({}^{1}\)National Centre for Text Mining, University of Manchester, United Kingdom

\({}^{2}\)School of Computer Science, University of Manchester, United Kingdom {maolin.li, arvidfm, tingting.mu, sophia.ananiadou}@manchester.ac.uk

###### Abstract

When constructing models that learn from noisy labels produced by multiple annotators, it is important to accurately estimate the reliability of annotators. Annotators may provide labels of inconsistent quality due to their varying expertise and reliability in a domain. Previous studies have mostly focused on estimating each annotator's overall reliability on the entire annotation task. However, in practice, the reliability of an annotator may depend on each specific instance. Only a limited number of studies have investigated modelling per-instance reliability and these only considered binary labels. In this paper, we propose an unsupervised model which can handle both binary and multi-class labels. It can automatically estimate the per-instance reliability of each annotator and the correct label for each instance. We specify our model as a probabilistic model which incorporates neural networks to model the dependency between latent variables and instances. For evaluation, the proposed method is applied to both synthetic and real data, including two labelling tasks: text classification and textual entailment. Experimental results demonstrate our novel method can not only accurately estimate the reliability of annotators across different instances, but also achieve superior performance in predicting the correct labels and detecting the least reliable annotators compared to state-of-the-art baselines.1

Footnote 1: Code is available at [https://github.com/createmomo/instance-level-reliability](https://github.com/createmomo/instance-level-reliability)

## 1 Introduction

In many natural language processing (NLP) applications, the performance of supervised machine learning models depends on the quality of the corpus used to train the model. Traditionally, labels are collected from multiple annotators/experts who are assumed to provide reliable labels. However, in reality, these experts may have varying levels of expertise depending on the domains, and thus may disagree on labelling in certain cases Aroyo and Welty (2013). A rapid and cost-effective alternative is to obtain labels through crowdsourcing Snow et al. (2008); Poesio et al. (2013, 2017). In crowdsourcing, each instance is presented to multiple expert or non-expert annotators for labelling. However, labels collected in this manner could be noisy, since some annotators could produce a significant number of incorrect labels. This may be due to differing levels of expertise, lack of financial incentive and interest Poesio et al. (2017), as well as the tedious and repetitive nature of the annotation task Raykar et al. (2010); Bonald and Combes (2017).

Thus, in order to ensure the accuracy of the labelling and the quality of the corpus, it is crucial to estimate the reliability of the annotators automatically without human intervention.

Previous studies have mostly focused on evaluating the annotators' overall reliability Gurevych and Kim (2013); Sheshadri and Lease (2013); Pessio et al. (2017). Measuring the reliability on a per-instance basis is however useful as we may expect certain annotators to have more expertise in one domain than another, and as a consequence certain annotation decisions will be more difficult than others. This resolves a potential issue of models that only assign an overall reliability to each annotator, where such a model would determine an annotator with expertise in a single domain to be unreliable for the model, even though the annotations are reliable within the annotator's domain of expertise.

Estimating per-instance reliability is also helpful for unreliable annotator detection and task allocation in crowdsourcing, where the cost of labelling data is reduced using proactive learn

ing strategies for pairing instances with the most cost-effective annotators Donmez and Carbonell (2008); Li et al. (2017). Although reliability estimation has been studied for a long time, only a limited number of studies have examined how to model the reliability of each annotator on a per-instance basis. Additionally, these in turn have only considered binary labels Yan et al. (2010); Wang and Bi (2017), and cannot be extended to multi-class classification in a straightforward manner.

In order to handle both binary and multi-class labels, our approach extends one of the most popular probabilistic models for label aggregation, proposed by Hovy et al. (2013). One challenge of extending the model is the definition of the label and reliability probability distributions on a per-instance basis. Our approach introduces a classifier which predicts the correct label of an instance, and a reliability estimator, providing the probability that an annotator will label a given instance correctly. The approach allows us to simultaneously estimate the per-instance reliability of the annotators and the correct labels, allowing the two processes to inform each other. Another challenge is to select appropriate training methods to learn a model with high and stable performance. We investigate training our model using the EM algorithm and cross entropy. For evaluation, we apply our method to six datasets including both synthetic and real-world datasets (see Section 4.1). In addition, we also investigate the effect on the performance when using different text representation methods and text classification models (see Section 4.2).

Our contributions are as follows: firstly, we propose a novel probabilistic model for the simultaneous estimation of per-instance annotator reliability and the correct labels for natural language labelling tasks. Secondly, our work is the first to propose a model for modelling per-instance reliability for both binary and multi-class classification tasks. Thirdly, we show experimentally how our method can be applied to different domains and tasks by evaluating it on both synthetic and real-world datasets. We demonstrate that our method is able to capture the reliability of each annotator on a per-instance basis, and that this in turn helps improve the performance when predicting the underlying label for each instance and detecting the least reliable annotators.

## 2 Related Work

### Modelling Annotator Reliability

Probabilistic graphical models have been widely used for inferring the overall reliability of annotators in the absence of ground truth labels. Approaches include modelling a single overall reliability score for each annotator Whitehill et al. (2009); Welinder et al. (2010); Karger et al. (2011); Liu et al. (2012); Demartini et al. (2012); Hovy et al. (2013); Rodrigues et al. (2014); Li et al. (2014a,b), estimating the reliability of each annotator on a per-category basis Dawid and Skene (1979); Zhou et al. (2012); Kim and Ghahramani (2012); Zhang et al. (2014), and estimating the sensitivity and specificity for each annotator in binary classification tasks Raykar et al. (2010).

Fewer attempts have been made to model the per-instance reliability of annotators, focusing mainly on medical image classification. One approach is that by Yan et al. (2010, 2014) who use logistic regression to predict the per-instance reliability of annotators. Wang and Bi (2017) used a modified support vector machine (SVM; Cortes and Vapnik 1995) loss, modelling the per-instance reliability as the distance from the given instance to a separation boundary.

### True Label Prediction in Crowdsourcing

True label prediction in crowdsourcing is the aggregation of labels produced by different annotators to infer the correct label of each instance. Majority voting assigns to each instance the most commonly occurring label among the annotators, which can result in a high agreement between the predicted label and the ground truth for some NLP tasks Snow et al. (2008). Dawid and Skene (1979), Whitehill et al. (2009), Raykar et al. (2010), Welinder et al. (2010), Liu et al. (2012), Zhou et al. (2012), Kim and Ghahramani (2012), Hovy et al. (2013), Yan et al. (2010, 2014), Li et al. (2014b) and Zhang et al. (2014) investigated binary or multi-class label prediction using probabilistic graphical models.

 



## 3 Methodology

### Model

In the description of our model we let \(N\) be the number of training instances, \(M\) the number of annotators, \({\bf x}_{i}\) the \(i\)th training instance, \(t_{i}\) its true underlying label, \(T\) the set of values \(t_{i}\) can take on, \(r_{ij}\) whether annotator \(j\) is reliable for the \(i\)th instance, and \(a_{ij}\) the label that annotator \(j\) gave the \(i\)th instance. Below we describe the components of the model in more detail.

Probabilistic Model:Our model is inspired by the method proposed by hovy2013efficient, and it shares the same graphical representation (see Figure 1). The distributions of the model, however, are defined differently, as can be seen in Figure 2, due to the inclusion of a classifier and a reliability estimator.

We assume that the underlying label \(t_{i}\) depends only on the corresponding instance, while the reliability \(r_{ij}\) depends on the instance and the identity of the annotator. If \(r_{ij}=0\), then the annotator \(j\) is unreliable for instance \({\bf x}_{i}\), and a label is chosen randomly from among the available categories. Otherwise, the annotation \(a_{ij}\) is set to be the correct label.

Classifier:The classifier \(f_{t}({\bf x}_{i})\) provides the predicted probabilities of an instance belonging to each category, \(p(t_{i}~{}|~{}{\bf x}_{i})\). \(t_{i}\) is the underlying label for instance \({\bf x}_{i}\), in \(i\)th instance, and takes a value in the set of categories \(T\). Note that there is no restriction on what classifier is used, other than that it can be trained using expectation maximisation. The inclusion of a classifier directly in the model means that it can be trained while taking into account the uncertainty of the data and predictions, as opposed to first making a hard assignment of a label for each instance and training the classifier post-hoc.

Reliability Estimator:The reliability estimator \(f_{r}({\bf x}_{i},j)\) predicts the probability of annotator \(j\) producing the correct label for instance \({\bf x}_{i}\), \(p(r_{ij}~{}|~{}{\bf x}_{i})\). \(r_{ij}\) is a binary variable, with \(1\) and \(0\) representing annotator \(j\) being reliable and unreliable for instance \({\bf x}_{i}\), respectively. The reliability estimator is modelled as a feed-forward neural network, where \(j\) is encoded as a one-hot vector. The exact representation of \({\bf x}_{i}\) depends on the model used for the classifier. If the classifier is a neural network, the output of the last hidden layer is used; otherwise, the original feature vector is used.

### Learning

Pre-trainingAs the number of parameters in our model is much larger than that of previous studies Yan et al. (2010, 2014); Wang and Bi (2017) due to the introduction of both a classifier and a reliability estimator, the model is much harder to train from scratch. Therefore, before we start training the model, we first pre-train the classifier using labels predicted by a simpler method as targets, using e.g. majority voting or the method proposed by dawid2019multi. Although these labels may be noisy, we have observed empirically that a better initialisation strategy does result in better performance (see Section 5). For the reliability estimator, for each instance \({\bf x}_{i}\) we compare each annotation \(a_{ij}\) to the labels predicted in the previous step. If \(a_{ij}\) is the same as the predicted label, we take the corresponding \(r_{ij}\) to be \(1\), and \(0\) otherwise. We then pre-train the reliability estimator \(f_{r}\) to predict these values for \({\bf r}\).

EM TrainingWe first consider training our model using expectation maximisation EM; Dempster et al. (1977).

Figure 2: Generative process for our method. \(f_{t}\) is the classifier, returning a probability distribution over predicted labels, and \(f_{r}\) is the reliability estimator, returning the probability that the annotator is accurate for the instance. \(\mathrm{Uniform}(T)\) is a uniform distribution over the categories in \(T\). \(\delta_{t_{i}}\) is the deterministic distribution that only takes on the value \(t_{i}\).



This involves maximising the expectation of the complete log likelihood of the model with respect to the posterior of the latent variables in the model. For the posterior of the model, we fix the parameters of the model and denote them \({\boldsymbol{\theta}^{(k)}}\) at iteration \(k\) of the algorithm. We only maximise the expectation with respect to the parameters \({\boldsymbol{\theta}}\) of the complete log likelihood.

The expectation is calculated as:

\[Q({\boldsymbol{\theta}}\mid{\boldsymbol{\theta}^{(k)}}) =\mathbb{E}[\log p({\bf a},{\bf t},{\bf r}\mid{\bf x},{\boldsymbol {\theta}})] \tag{1}\] \[=\sum_{i=1}^{N}\mathbb{E}\left[\log p(t_{i}\mid{\bf x}_{i},{ \boldsymbol{\theta}})\right]\] \[+\sum_{i=1}^{N}\sum_{j=1}^{M}\mathbb{E}\left[\log p(r_{ij}\mid{ \bf x}_{i},{\boldsymbol{\theta}})\right]\] \[+\sum_{i=1}^{N}\sum_{j=1}^{M}\mathbb{E}\left[\log p(a_{ij}\mid t_ {i},r_{ij},{\bf x}_{i},{\boldsymbol{\theta}})\right],\]

where each expectation is calculated with respect to the posterior \(p({\bf t},{\bf r}\mid{\bf a},{\bf x},{\boldsymbol{\theta}^{(k)}})\).

E Step:For the E step we compute the posterior with fixed parameters \({\boldsymbol{\theta}^{(k)}}\), \(\pi_{ij}(t,r)=p(t_{i}=t,r_{ij}=r\mid{\bf a}_{i},{\bf x}_{i})\), as:

\[\begin{split}&\pi_{ij}(t,r)=p(t_{i}=t,r_{ij}=r\mid{\bf a}_{ i},{\bf x}_{i})\\ &\quad\propto p(t_{i}=t\mid{\bf x}_{i})p(r_{ij}=r\mid{\bf x}_{i}) \\ &\quad\cdot p(a_{ij}\mid t_{i}=t,r_{ij}=r,{\bf x}_{i})\cdot\prod_{ j^{\prime}\neq j}\gamma_{ij^{\prime}}^{(k)}(t)\\ &\gamma_{ij}^{k}(t)=\sum_{rr\in\{0,1\}}\big{(}p(r_{ij}=r\mid{\bf x }_{i})\\ &\qquad\qquad\qquad\cdot p(a_{ij}\mid t_{i}=t,r_{ij}=rt,{\bf x}_{ i})\big{)},\end{split} \tag{2}\]

where we drop the dependency on \({\boldsymbol{\theta}^{(k)}}\) for brevity. Note that \(\pi_{ij}(t,r)=0\) when \(r=1\) and \(a_{ij}\neq t\).

We can then compute the marginalised posteriors, needed for Equation (1), as follows:

\[p(t_{i}=t\mid{\bf a}_{i},{\bf x}_{i})=\sum_{r\in\{0,1\}}\pi_{i1}(t,r) \tag{4}\]

\[p(r_{ij}=r\mid{\bf a}_{i},{\bf x}_{i})=\sum_{t\in T}\pi_{ij}(t,r), \tag{5}\]

where the posterior \(p(t_{i},r_{i1}\mid{\bf a}_{i},{\bf x}_{i})\) of the model is chosen arbitrarily to marginalise over to get the posterior for \(t_{i}\).

M Step:Using the posterior calculated in the E step we can compute the expectation of the complete log likelihood, \(Q({\boldsymbol{\theta}}\mid{\boldsymbol{\theta}^{(k)}})\), and calculate its gradient with respect to the parameters \({\boldsymbol{\theta}}\). We then use gradient ascent to update the classifier and reliability estimator jointly.

```
1:
2:a, the annotations
3:x, the instances
4:L, the number of inner iterations
5:m, the training mode: 0 for expectation maximisation, 1 for cross entropy (training identity), 2 for cross entropy (training identity)
6:l: Pre-train \({\boldsymbol{\theta}}=\{{\boldsymbol{\theta}}_{t},{\boldsymbol{\theta}}_{v}\}\) (Section 3.2)
7:k\(\gets 0\)
8:while stopping criteria not met do
9:k\(\leftarrow k+1\)
10:Calculate \(\pi_{ij}(t,r)\) (Equation (2)) \(\triangleright\) E step
11:for\(n=0\)then\(\triangleright\) M stop
12:for\(f\) = 1 to L
13:\({\boldsymbol{\theta}}\leftarrow{\boldsymbol{\theta}}+\alpha\nabla_{{ \boldsymbol{\theta}}}Q({\boldsymbol{\theta}}\mid{\boldsymbol{\theta}^{(k)}})\)
14:endfor
15:for\(l=16\)do
16:\({\boldsymbol{\theta}}_{r}\leftarrow{\boldsymbol{\theta}}_{r}-\alpha\nabla_{{ \boldsymbol{\theta}}_{r}}L({\boldsymbol{\theta}}_{r}\mid{\boldsymbol{\theta}^ {(k)}})\)
17:endfor
18:for\(l=1\)to L
19:\({\boldsymbol{\theta}}_{t}\leftarrow{\boldsymbol{\theta}}_{t}-\alpha\nabla_{{ \boldsymbol{\theta}}_{t}}L({\boldsymbol{\theta}}_{t}\mid{\boldsymbol{\theta}^ {(k)}})\)
20:endfor
21:if\(l=1\)to L
22:endif
23:endwhile
```

**Algorithm 1** Training procedure

### Cross Entropy Training

As an alternative training procedure, we also consider training the model using cross entropy. As with expectation maximisation, we first calculate the posterior \(\pi_{ij}(t,r)\) using the fixed parameters \({\boldsymbol{\theta}^{(k)}}\). The networks \(f_{t}\) and \(f_{r}\) are then trained to minimise the cross entropy between the priors \(p(t_{i}\mid{\bf x}_{i})\) and \(p(r_{ij}\mid{\bf x}_{i})\), and the corresponding posteriors \(p(t_{i}\mid{\bf a}_{i},{\bf x}_{i})\) and \(p(r_{ij}\mid{\bf a}_{i},{\bf x}_{i})\).

The networks can be trained in an alternating fashion, with \(f_{r}\) being trained while \(f_{t}\) is kept fixed, and the other way around. Denoting the parameters of \(f_{t}\) as \({\boldsymbol{\theta}}_{t}\) and \(f_{r}\) as \({\boldsymbol{\theta}}_{r}\), the loss functions for the respective networks then become

\[\begin{split} L({\boldsymbol{\theta}}_{t}\mid{\boldsymbol{\theta}^ {(k)}})&=-\frac{1}{N}\sum_{i,t,r}\pi_{i1}(t,r)\log p(t_{i}\mid{\bf x }_{i})\\ L({\boldsymbol{\theta}}_{r}\mid{\boldsymbol{\theta}^{(k)}})& =-\frac{1}{NM}\sum_{i,j,t,r}\pi_{ij}(t,r)\log p(r_{ij}\mid{\bf x }_{i})\end{split} \tag{6}\]

Alternatively, they can be trained jointly by minimising the total cross entropy.

\[L({\boldsymbol{\theta}}\mid{\boldsymbol{\theta}^{(k)}})=L({\boldsymbol{\theta}}_{ t}\mid{\boldsymbol{\theta}^{(k)}})+L({\boldsymbol{\theta}}_{r}\mid{\boldsymbol{ \theta}^{(k)}}) \tag{7}\]

The training algorithm is summarised in Algorithm 1. The algorithm is run until either a maximum number of iterations is reached, or the objective function stops improving 



## 4 Evaluation Settings

### Data

#### Simulated Annotators

2-Dimensional Datasets:In order to see whether our method can work well on simple cases, we create three 2-dimensional synthetic datasets, which we refer to as moon, circle and 3-class as shown in Figure 3.

Text Classification:For text classification we use the datasets Question Classification Li and Roth (2002), which contains short questions along with the type of answer expected, and Sentence Classification Chambers (2013), which consists of sentences selected from medical publications. Examples of instance/class pairs for the text classification datasets include "Where is the Orinoco?" (class: "location") for the Question Classification dataset, and "New types of potent force clamps are discovered." (class: "author's own work") for the Sentence Classification dataset.

For these datasets that do not include crowd annotations, we synthesise annotations by simulating different annotators as follows: 1) **Narrow Expert**: has expertise in a single domain (i.e. class). For the instances of this class, the annotator will always provide the correct label. For other classes, a correct label will be provided with a probability of 0.65; otherwise, a random label will be selected with uniform probability; 2) **Broad Expert**: has expertise in every domain and only makes mistakes with a probability of 0.05; 3) **Random Annotation**: selects labels at random; 4) **Adversarial Annotator**: deliberately provides incorrect labels with a probability of 0.8.

For each of the datasets, we generated annotations using one narrow expert per class, one broad expert, one random annotator and one adversarial annotator, for a total of \(|T|+3\) annotators, where \(|T|\) is the number of classes in the dataset.

In order to evaluate the generality of our model, we also apply it to another task in which we have 5 annotators with different overall reliabilities for the text classification tasks. They produce incorrect labels with probabilities 0.1, 0.3, 0.5, 0.7, 0.9 respectively.

#### Recognising Textual Entailment:

Finally, we evaluate our model on a real-world dataset for the recognising textual entailment (RTE) task Snow et al. (2008). Given a text pair, the annotator decides whether the hypothesis sentence can be inferred from the text fragment. The dataset includes both ground truth and crowdsourced labels from 164 annotators.

Table 1 shows the number of instances of each class2 in the above-mentioned datasets.

Footnote 2: The classes in the Sentence Classification dataset are defined as follows: AIMX—the goal of the paper; OWNX—the author’s own work; CONT—the comparison including contrast and critique of past work; BASE—the past research that provides the basis for the work; MISC—any other sentences.

### Experimental and Model Settings

Our model was implemented using the Chainer deep learning framework3 Tokui et al. (2015).

Footnote 3: [https://chainer.org/](https://chainer.org/)

Classifier:As shown in Table 2, in each experiment the output of the classifier is generated by a feed-forward neural network (FNN). Each FNN consists of an input layer, two hidden layers and a softmax output layer. The number of hidden units in each layer is listed in the third column of the table. The ReLU activation function Nair and

\begin{table}
\begin{tabular}{c c c} \hline
**Dataset** & **Class** & **\# Instances** \\ \hline moon & 0/1 & 500500 \\ \hline circle & 0/1 & 500500 \\ \hline
3-class & 0/1/2 & 3343/33333 \\ \hline  & DESCRIPT (DESC) & 1162 \\  & ENTITY (ENTY) & 1250 \\ Question & ABBREV (ABRB) & 86 \\ Classification & HUMAN (UHM) & 1223 \\  & NUMERIC (NUM) & 896 \\  & LOCATION (LOC) & 835 \\ \hline  & AMX & 94 \\ Sentence & OWNX & 427 \\ Classification & CONT & 104 \\  & BASE & 33 \\  & MSC & 852 \\ \hline RTE & 0/1 & 4009400 \\ \hline \end{tabular}
\end{table}
Table 1: Classes and per-class instance counts.

Figure 3: Three 2-dimensional datasets.



Hinton, 2010) was applied after each hidden layer. The output size of all the Long Short-Term Memory (LSTM; Hochreiter and Schmidhuber, 1997) layers in our experiments is 100.

For the 2-dimensional classification task, each instance is simply represented using its position in 2-dimensional space. For the text classification tasks, we investigated 3 methods of representing the sentences: bag-of-words (BoW) weighted by Term Frequency-Inverse Document Frequency (TFIDF), an average word embedding (Avg.) and the output at the last step of an LSTM layer (Embed.\(\rightarrow\)LSTM). For the embedding we use word2vec embeddings pre-trained on Google News Mikolov et al. (2013) for the question classification and RTE tasks, and a pre-trained embedding Pyysalo et al. (2013) trained on a combination of English Wikipedia, PubMed and PMC texts for the sentence classification task.

For the RTE task, we implemented two classifiers. For the first one, each instance (i.e. a sentence pair) was represented as a concatenation of the average word embedding for each sentence (Cat. Avg.). We also implemented Bowman et al. (2015), which runs each sentence through an LSTM, concatenates the outputs, and then feeds the concatenated output to an FNN with tanh activations.

Reliability Estimator:We model the reliability estimator as an FNN. Its structure is the same as the classifier, albeit with different sizes of the two hidden layers. For the experiments listed in Table 2, the number of units of each hidden layer in the FNN are 5, 100, 25, 25, 50, and 100 respectively. The input to the estimator is the concatenation of the instance \({\bf x}_{i}\) (i.e. its original feature vector or the output of the last hidden layer of the classifier) and a one-hot vector representing the annotator identity.

Learning Settings:For every experiment we use the Adam Kingma and Ba (2015) optimiser with a weight decay rate \(0.001\), a gradient clipping of \(5.0\), \(\alpha=0.001\), \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\). We pre-train the classifier and reliability estimator for 200 epochs, using both majority voting and the model proposed by Dawid and Skene (1979). The maximum number of outer iterations is set to 500 and 20 for EM training and cross entropy training respectively. The number of inner iterations is 50 in both cases.

True Label Prediction and Reliability Estimation:After training, for each instance \({\bf x}_{i}\) we take its underlying label to be the most probable label according to the posterior of \(t_{i}\) (see Equation (4)). We compared our predicted labels to the following state-of-the-art baselines: Majority Voting (MV), DS Dawid and Skene (1979), GLAD Whitehill et al. (2009), LFC Raykar et al. (2010), CUBAM Welinder et al. (2010), Yan et al. (2010), KOSKarger et al. (2011), VI Liu et al. (2012), BCC Kim and Ghahramani (2012), MINIMAX Zhou et al. (2012), MACE Hovy et al. (2013), CATD Li et al. (2014a), PM Li et al. (2014b) and EM-MV and Opt Zhang et al. (2014).

Note that CUBAM, Yan et al. (2010), KOS and VI are only suitable for aggregating binary labels, and Yan et al. (2010) is the state-of-the-art method that models per-instance reliability. We take the reliability of annotator \(j\) on instance \({\bf x}_{i}\) to be the posterior probability that \(r_{ij}\) is 1 (see Equation (5)).

## 5 Results and Analysis

We measure the inter-annotator agreement (IAA) of each dataset. Fleiss's kappa Fleiss et al. (2013), denoted by \(\kappa\), is measured for the 2-dimensional and text classification datasets, and Kippendorff's alpha Krippendorff (1970) is calculated for the RTE dataset4. We find that the IAA values indicate slight agreement among annotators for all datasets.

Footnote 4: Although there are 164 annotators in total in this dataset, each instance was labelled by only 10 of these annotators. Therefore we use Krippendorff’s alpha which is applicable to incomplete data to measure the inter-annotator agreement.

Our experiments using different settings are shown as follows: our model is denoted by \(O\), with \(M\) and \(D\) denoting the model pre-trained using MV and DS respectively. \(E\) denotes training using expectation maximisation, while \(C\) denotes cross entropy training. \(AL\) and \(JT\) denote cross entropy training done alternatingly and jointly, respectively.

In the rest of this section, Tables 3 to 7 and Tables 8 to 10 present the results on the synthetic datasets and RTE dataset respectively. For the synthetic datasets, in Tables 3 to 6, we first consider a scenario where we have multiple narrow experts (N), one broad expert (B), one random annotator (R) and one adversarial annotator (A). In Table 7, we further consider a scen 

each of differing reliability, as explained in Section 4.1.

Table 3 shows that our method performs well on the 2-dimensional datasets, obtaining higher label prediction F1 scores than the baselines. We omit the analysis of the true label prediction and reliability estimation results on these datasets as all models performed similarly, choosing instead to focus the discussion on the results for the NLP tasks.

### Classifier and Reliability Estimator

In order to explore the separate performance contribution of classifier and reliability estimator, we compare the performance of our model to a classifier pre-trained using DS labels, as well as a variant of our model without the reliability estimator, i.e. setting all the annotators have the same reliability on all the instances.

As shown in Tables 4, 7 and 8, the pre-trained classifier performed worse than some aggregation methods. This indicates that the noise in the labels predicted by DS has an adverse effect on the training of the classifier. The much lower performance of the model with the reliability estimator removes hints at the importance of modelling per-annotator reliability to ensure accurate predictions.

### Instance Representation for Reliability Estimator

For the representation of the instance \({\bf x}_{i}\) as it is fed to the reliability estimator, we compared the performance of using the original feature vector of \({\bf x}_{i}\) to using the last hidden layer output of the classifier (which we refer to as the "full model").

We found that using the hidden layer representation can not only improve the label prediction performance (see Tables 4, 7 and 8), but also sped up the training compared to using the feature vector directly. The hidden layer representation allows us to reduce the number of parameters in the model, by sharing parameters with the classifier.

### Full Model on Synthetic Datasets

Based on the results of the full model in Table 4, we can conclude that per-instance reliability modelling is beneficial to the label prediction task, and using the average pre-trained embedding can result in slightly better performance. It is worth noting that the method used to pre-train the model had a noticeable effect on its performance, with better F1 scores being obtained when using DS pretraining. In the following experiments we only consider models pre-trained using the DS algorithm.

In order to investigate whether our method can successfully capture per-instance annotator reliability, for each annotator, we counted the number of correctly labelled instances and calculated the

\begin{table}
\begin{tabular}{l c c c}  & **Question** & **Sentence** \\  & \multicolumn{2}{c}{**Classification**} & **Classification** \\  & \(\kappa=0.094\) & \(\kappa=0.0634\) \\ \hline MV & 78.8 & 78.8 & 71.3 & 71.3 \\ DS (David and Slee, 1979) & 98.3 & 98.3 & 97.3 & 97.3 \\ GLAD (Whitehill et al., 2009) & 87.1 & 87.1 & 79.9 & 79.9 \\ LFC (Raykar et al., 2010) & 98.2 & 98.2 & 97.0 & 97.0 \\ BC (Kim and Ghahramani, 2012) & 98.3 & 98.3 & 98.1 & 98.1 \\ MINMAX (Zhou et al., 2012) & 28.2 & 28.2 & 50.9 & 50.9 \\ MACE (Hovy et al., 2013) & 91.6 & 91.6 & 63.6 & 63.6 \\ CAD (Li et al., 2014a) & 91.1 & 91.1 & 92.2 & 92.2 \\ PM (Li et al., 2014b) & 91.1 & 91.2 & 92.2 & 92.2 \\ EM-MV (Zhang et al., 2014) & 88.3 & 88.3 & 66.9 & 66.9 \\ BL-OPG (Zhang et al., 2014) & 13.5 & 13.5 & 26.5 & 26.5 \\ \hline Classifier (pre-trained by DS labels) & 92.3 & 86.5 & 89.6 & 85.4 \\ O-DC (without reliability estimator) & 92.1 & 91.2 & 77.6 & 87.1 \\ O-DC-CL (without reliability estimator) & 90.5 & 93.2 & 84.0 & 88.2 \\ O-DCT (without reliability estimator) & 88.7 & 91.1 & 78.5 & 87.4 \\ \end{tabular}
\end{table}
Table 4: F1 scores of predicted labels on the text classification datasets.

\begin{table}
\begin{tabular}{l c c c} \hline  & \multicolumn{2}{c}{**2-Dimensional Datasets**} \\  & \multicolumn{2}{c}{**mono**} & **circle** & **y-class** \\  & \(\kappa=0.092\) & \(\kappa=0.029\) & \(\kappa=0.153\) \\ \hline MV & 84.6 & 84.6 & 89.0 \\ DS (Dawid and Skene, 1979) & 97.8 & 97.9 & 99.4 \\ GLAD (Whitehill et al., 2009) & 97.9 & 97.9 & 93.7 \\ LFC (Raykar et al., 2010) & 97.9 & 97.9 & 99.4 \\ CUBAM (Welinder et al., 2010) & 97.9 & 97.9 & - \\ Yan et al. (Yan et al., 2010) & 84.1 & 84.1 & - \\ KOS (Karger et al., 2011) & 96.7 & 96.7 & - \\ VI (Liu et al., 2012) & 97.9 & 97.9 & - \\ BCC (Kim and Ghahramani, 2012) & 97.9 & 97.9 & 99.4 \\ MINMAX (Zhou et al., 2012) & 97.9 & 97.9 & 99.2 \\ MACE (Howy et al., 2013) & 97.8 & 97.9 & 97.0 \\ CATD (Li et al., 2014a) & 94.5 & 94.5 & 94.3 \\ PM (Li et al., 2014b) & 94.5 & 94.5 & 94.3 \\ ER-MV (Zhang et al., 2014) & 97.9 & 97.9 & 96.3 \\ EM-Opt (Zhang et al., 2014) & 72.7 & 72.7 & 93.2 \\ \hline O-ME & 97.0 & 97.1 & 94.8 \\ O-MC & 97.3 & 98.9 & 99.3 \\ O-DE & 91.2 & 97.2 & 96.5 \\ O-DC-AL & 99.7 & **99.7** & **99.5** \\ O-DC-JT & **99.8** & 99.6 & **99.5** \\ \hline \end{tabular}
\end{table}
Table 3: F1 scores of predicted labels on the 2-dimensional datasets when using the output of the last hidden layer of the classifier to represent an instance for the reliability estimator.

 

average reliability for each class among the top 100 instances with the highest per-instance reliability as shown in Table 5 and 65. The cells with grey background colour indicate which domain, or class, the annotator has expertise in. It can be seen that all annotators obtain high accuracy on these instances. In general our method also captured the varying expertise of each narrow annotator, estimating their reliability on instances belonging to the corresponding classes as particularly high.

Footnote 5: We omit the results for the sentence classification task for lack of space, as we consider the results on the question classification dataset to be representative.

For these experiments in Table 7, we also investigated the performance when using two different classification models. As seen in this table, both of them outperformed all baselines significantly.

### Full Model on RTE Dataset

Table 8 presents the label prediction performance on the RTE dataset. As not every annotator has provided labels for every instance in this dataset, for both the EM and cross entropy training we simply omitted missing instance/annotator pairs when calculating the loss functions. As seen in the table, most of the baselines obtained high performance as the textual entailment recognition task is easy for non-expert annotators. However, our full model still achieved better prediction performance than all of the baseline methods.

We also investigated the effectiveness of our model for removing noisy labels. We compare our model to the five best-performing baselines (DS, LFC, CUBAM, VI and EM-MV in Table 8). Each of these models are trained on the RTE dataset, after which the least reliable annotation for each instance is removed. We use the per-instance reliability for our model, the global reliability score of each annotator for LFC, CUBAM and VI, and the per-category annotator reliability for DS and EM-MV as the measure of the reliability of each annotation. For each of these models, we then retrain the models in Table 8 using the denoised dataset; the difference in performance can be seen in Table 9. We can see that using per-instance reliability results in the largest improvement, while only considering the annotators' overall reliability may

\begin{table}
\begin{tabular}{c c c c c c c c} \hline  & \multicolumn{4}{c}{**Question Classification**} \\ \cline{3-8}  & \multicolumn{2}{c}{**DESC**} & **ENTY** & **ABER** & **HUM** & **NIM** & **LOC** & **Overall** \\ \hline
1 (N) & [0.0] & 24.4 & 25.3 & 21.4 & 26.8 & 22.2 & 36.8 \\
2 (N) & [2.2] & 26.6 & 25.2 & 24.3 & 26.7 & 37.3 \\
3 (N) & 24.7 & 24.9 & 25.0 & 23.5 & 24.7 & 23.6 & 29.9 \\
4 (N) & 25.7 & 26.3 & 23.1 & 28.9 & 24.2 & 23.7 & 37.9 \\
5 (N) & 25.6 & 21.9 & 25.2 & 25.7 & **0.8** & 26.0 & 36.4 \\
6 (N) & 26.4 & 25.0 & 24.1 & 22.1 & 27.5 & **0.8** & 36.6 \\
7 (R) & [0.0] & 28.3 & 29.0 & 28.1 & 29.4 & **0.8** & 39.4 \\
8 (R) & 56.0 & 6.3 & 8.0 & 4.80 & 6.20 & 5.90 & 6.20 \\
9 (A) & 9.10 & 8.90 & 12.0 & 8.20 & 7.90 & 8.30 & 9.10 \\ \hline \end{tabular}
\end{table}
Table 6: Average reliability of each annotator among the 100 instances with the highest per-instance reliability on the question classification dataset.

\begin{table}
\begin{tabular}{l l} \hline
**Method** & **F-measure** \\ \hline MV & 91.9 \\ DS (David and Stone, 1979) & 92.6 \\ GLAD (Whitehill et al., 2009) & 92.4 \\ LICR (Baykar et al., 2010) & 92.5 \\ CIMRM (Welinder et al., 2010) & 92.6 \\ Yan et al. (Yan et al., 2010) & 90.4 \\ KOS (Karger et al., 2011) & 63.2 \\ UI (Jai et al., 2012) & 92.5 \\ BCC (Kim and Ghahramani, 2012) & 92.3 \\ MMIANZ (Zhou et al., 2011) & 92.4 \\ MACI (Burov et al., 2013) & 92.4 \\ CATD (Li et al., 2014a) & 92.3 \\ PM Li et al. (2014b) & 92.0 \\ EA-MP (Zhang et al., 2014) & 92.5 \\ EM-Opt (Zhang et al., 2014) & 92.4 \\ Classifier type (fained by IDS labels) & 89.9 \\ D-CD-T (FNS) (without training estimator) & 90.2 \\ D-DC-T (FNS) (using feature vector) & 92.6 \\ \hline ODE (ESSIN null model) & 92.9 \\ O-DC-AL (FNS) (full model) & 92.8 \\ O-DC-DT (FNS) (full model) & **93.0** \\ O-DE (LSTH+FNS) (full model) & 92.7 \\ O-DC-AL (LSTM+FNS) (full model) & 92.8 \\ \hline \end{tabular}
\end{table}
Table 7: F1 scores on text classification tasks when only the reliability differs between the annotators.

\begin{table}
\begin{tabular}{c|c c c c c} \hline  & \multicolumn{4}{c}{**Questing Classification**} \\ \cline{2-5}  & \multicolumn{2}{c}{**DESC**} & **ENTY** & **ABER** & **HUM** & **NIM** & **LOC** & **Accuracy** \\ \hline
1 (N) & [0.0] & 24.4 & 25.3 & 21.4 & 26.8 & 22.2 & 36.8 \\
2 (N) & [2.2] & 26.6 & 25.2 & 24.3 & 26.7 & 37.3 \\
3 (N) & [2.2] & 24.9 & 25.0 & 23.5 & 24.7 & 23.6 & 29.9 \\
4 (N) & [2.5] & 25.6 & 21.9 & 25.2 & 25.7 & **0.8** & 26.0 \\
6 (N) & [2.6] & 26.4 & 25.0 & 24.1 & 22.1 & 27.5 & **0.8** & 36.6 \\
7 (R) & [0.0] & 28.3 & 28.1 & 29.4 & 28.9 & 39.4 \\
8 (R) & [0.0] & 28.0 & 8.0 & 4.80 & 6.20 & 5.90 & 6.20 \\
9 (A) & 9.10 & 8.90 & 12.0 & 8.20 & 7.90 & 8.30 & 9.10 \\ \hline \end{tabular}
\end{table}
Table 6: Average reliability of each annotator among the 100 instances with the highest per-instance reliability on the question classification dataset.

\begin{table}
\begin{tabular}{l r} \hline
**Method** & **F-measure** \\ \hline MV & 91.9 \\ DS (David and Stone, 1979) & 92.6 \\ GLAD (Whitehill et al., 2009) & 92.4 \\ LICR (Baykar et al., 2010) & 92.5 \\ CIMRM (Welinder et al., 2010) & 92.6 \\ Yau et al. (Yan et al., 2010) & 90.4 \\ KOs (Karger et al., 2011) & 63.2 \\ VI (Jai et al., 2012) & 92.5 \\ BCC (Kim and Ghahramani, 2012) & 92.3 \\ MIMIX (Zhou et al., 2011) & 92.4 \\ MATCI (Birov et al., 2013) & 92.4 \\ CAPD (Li et al., 2014a) & 92.3 \\ PIAI (Li et al., 2014b) & 92.0 \\ BA-MP (Zhang et al., 2014) & 92.5 \\ BI-Opt (Zhang et al., 2014) & 92.4 \\ \hline Classifier type (fained by IDS labels) & 89.9 \\ DO-CT (FNS) (without training estimator) & 90.2 \\ O-DC-TP (FNS) (tuning feature vector) & 92.6 \\ \hline ODE (ESSIN null model) & 92.9 \\ O-DC-AT (FNS) (full model) & 92.8 \\ O-DC-AT (FNS) (full model) & 92.8 \\ \hline \end{tabular}
\end{table}
Table 8: Performance of predicted labels on the RTE dataset (Krippendorff’s alpha \(=0.0995\)).

 

cause a reduction in performance.

In order to analyse the per-instance reliability of the human annotators, for each annotator we rank the instances according to the annotator's per-instance reliability. We look at the top 15 and bottom 15 instances, then count how many of them were correctly labelled (Cor. Labels) as well as the average reliability on these instances (Avg. Reliability). Table 10 shows the results of five annotators6. It can be seen that each annotator has considerably different reliabilities across instances.

Footnote 6: For lack of space, we only present the results for 5 of the 164 annotators.

### Training Stability

Pre-training:As discussed in Section 3.2, the predicted labels produced by a simpler method are used for pre-training. Although these labels are not perfect, we assume that our method can still learn some useful information from them for a better starting point than random parameter initialisation.

EM and Cross Entropy Training:From Tables 3, 4, 7 and 8, it can be seen that, in most cases, using cross entropy achieved much better and more stable performance than the models learned using EM training. We also noticed that the objective function would improve when using cross entropy training, and tended to converge faster in our experiments--generally within just a few epochs. Therefore, we recommend to use this training method in practice.

Early Stopping:When using both EM and cross entropy training, we found that even if the objective function improved between iterations, the label prediction performance would eventually start to decrease. It is worth to investigate the reason for this phenomenon. To counteract this issue we used early stopping, where training is halted when the objective function does not improve more than 0.001 between iterations. Another option is to reduce the maximum number of outer iterations, e.g. to 20.

## 6 Conclusion and Future Work

We propose a novel probabilistic model which learns from noisy labels produced by multiple annotators for NLP crowdsourcing tasks by incorporating a classifier and a reliability estimator. Our work constitutes the first effort to model the persistance reliability of annotators for both binary and multi-class NLP labelling tasks. We investigate two methods of training our model using the EM algorithm and cross entropy. Experimental results on 6 datasets including synthetic and real datasets demonstrate that our method can not only capture the per-instance reliability of each annotator, but also obtain better label prediction and the least reliable annotator detection performance compared to state-of-the-art baselines.

For future work, we plan to apply our model to other NLP tasks such as relation extraction and named entity recognition. We also plan to investigate the use of variational inference Jordan et al. (1999) as a means of training our model. Using variational inference might improve the stability and performance of our model.

## Acknowledgement

We would like to thank the anonymous reviewers and Paul Thompson for their valuable comments. Discussions with Austin J. Brockmeier have been insightful. The work is funded by _School of Computer Science Kilburn Overseas Fees Bursary_ from University of Manchester.

\begin{table}
\begin{tabular}{c c c c c c} \hline  & \multicolumn{3}{c}{**True Entainment**} \\  & **Amstatifier** & **fccx. Labels** & **Avg. Relibility** & **corc. Labelab** & **Avg. Relibility** & Acc. \\ \hline \multirow{3}{*}{Top} & 1 & 15 & 95.5 & - & - & 100 \\  & 2 & 15 & 92.6 & - & - & 100 \\  & 3 & 10 & 98.4 & 3 & 86.9 & 86.7 \\ References & 4 & 11 & 94.2 & 2 & 92.6 & 86.7 \\  & 5 & 8 & 71.9 & 2 & 23.7 & 68.7 \\ \hline \multirow{5}{*}{Bottom} & 1 & 4 & 0.1 & 2 & 0.1 & 49 \\  & 2 & 1 & 1.0 & 8 & 51.4 & 60 \\ \cline{1-1}  & 3 & 0 & 9.7 & 1 & 13.8 & 6.6 \\ \cline{1-1}  & 4 & 2 & 28.6 & 5 & 63.1 & 46.7 \\ \cline{1-1}  & 5 & 3 & 22.3 & 2 & 23.7 & 33.3 \\ \hline \end{tabular}
\end{table}
Table 10: Number of correct labels and average reliability for each annotator among the instances with highest and lowest per-instance reliability on the RTE dataset.

\begin{table}
\begin{tabular}{l l r r r r r} \hline
**Method** & **LFC** & **CUAM** & **VI** & **DS** & **SMW** & **Overs** \\ \hline \hline
**MV** & -0.2 & -0.9 & +0.6 & -0.2 & -0.2 & **+0.8** \\ DS David Senee, [19] & 0 & -0.2 & -0.1 & -0.0 & -0.0 & **+0.6** \\ GLAD Whitehill et al., [20] & **+0.2** & 0.0 & -0.0 & **-0.2** & -0.3 & **+0.2** \\ LFC Rugar et al., [20] & +0.1 & -0.1 & -0.3 & -0.1 & -0.1 & **+0.6** \\ CUBAMI Mihalizov et al., [20] & +0.3 & 0.4 & -0.3 & -0.2 & **+0.9** \\ Yau et al. (Yun et al., 2019) & +1.5 & +0.4 & -0.2 & -1.5 & -0.8 & **+0.2** \\ KOS Kagregor et al., [20] & +0.7 & +3.9 & +11.3 & -0.7 & +13.4 & **+17.3** \\ VI Liu et al., [20] & +0.1 & +0.1 & +0.1 & +0.1 & +0.2 & **+0.6** \\ BCC Kim and Dhraiman, [20] & +0.3 & -0.2 & -0.3 & -0.3 & -0.3 & **+0.4** \\ MINMM Zhou et al., [20] & +0.3 & -0.4 & **+0.7** & -0.3 & +0.2 & **+0.7** \\ MACER Hovy et al., [20] & +0.2 & -0.2 & -0.2 & -0.2 & **+0.4** \\ CATD Li et al., [20] & +0.3 & -0.8 & -0.2 & -0.3 & +0.2 & **+0.3** \\ PM Li et al., [20] & +0.9 & 0 & -0.3 & **+0.9** & +0.1 & **+0.9** \\ ERM-MW Zhang et al., [20] & +0.7 & +0.6 & -0.7 & -0.7 & -0.7 & **+1.1** \\ EM-Qip Zhang et al., [20] & +0.2 & +0.5 & +0.2 & +0.2 & +0.6 & **+0.7** \\ O-DCJ FTN (full mode) & +0.1 & 0 & -0.3 & -0.1 & -0.1 & **+0.5** \\ \hline \end{tabular}
\end{table}
Table 9: F1 score improvements after removing the labeled produced by the least reliable annotator by using the estimated overall reliability (LFC, CUBM, VI, DS, EM-MV) and per-instance reliability (Ours).

 



## References

* Aroyo and Welty (2013) Lora Aroyo and Chris Welty. 2013. Measuring crowd truth for medical relation extraction. In _AAAI 2013 Fall Symposium on Semantics for Big Data_, pages 1-9, Arlington, Virginia.
* Bonald and Combes (2017) Thomas Bonald and Richard Combes. 2017. A minimax optimal algorithm for crowdsourcing. In _Advances in Neural Information Processing Systems 30 (NIPS)_, pages 4352-4360. Curran Associates, Inc.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
* Chambers (2013) America Chambers. 2013. _Statistical Models for Text Classification and Clustering: Applications and Analysis_. Ph.D. thesis, University of California, Irvine.
* Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995. Supportvector networks. _Machine Learning_, 20(3):273-297.
* Dawid and Skene (1979) Alexander Philip Dawid and Allan M Skene. 1979. Maximum likelihood estimation of observer errorrates using the em algorithm. _Applied statistics_, pages 20-28.
* Demartini et al. (2012) Gianluca Demartini, Djelhel Eddine Drifallah, and Philippe Cudre-Mauroux. 2012. ZenCrowd: leveraging probabilistic reasoning and crowdsourcing techniques for large-scale entity linking. In _Proceedings of the 21st international conference on World Wide Web (WWW)_, WWW '12, pages 469-478, New York, NY, USA. Association for Computing Machinery.
* Dempster et al. (1977) Arthur P. Dempster, Nan Laird, and Donald B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. _Journal of the Royal Statistical Society_, B39:1-38.
* Donmez and Carbonell (2008) Pinar Donmez and Jaime G. Carbonell. 2008. Proactive learning: cost-sensitive active learning with multiple imperfect oracles. In _Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM)_, CIKM '08, pages 619-628, New York, NY, USA. Association for Computing Machinery.
* Fleiss et al. (2013) Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. 2013. _Statistical methods for rates and proportions_, 3 edition. John Wiley Sons.
* Gurevych and Kim (2013) Iryna Gurevych and Jungi Kim, editors. 2013. _The People's Web Meets NLP: Collaboratively Constructed Language Resources_. Springer-Verlag Berlin Heidelberg.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. _Neural Computation_, 9(8):1735-1780.
* Hovy et al. (2013) Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_, pages 1120-1130, Atlanta, Georgia. Association for Computational Linguistics.
* Jordan et al. (1999) Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. 1999. An introduction to variational methods for graphical models. _Machine Learning_, 37(2):183-233.
* Karger et al. (2011) David R. Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 24 (NIPS)_, pages 1953-1961. Curran Associates, Inc.
* Kim and Ghahramani (2012) Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In _Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics_, volume 22 of _Proceedings of Machine Learning Research_, pages 619-627, La Palma, Canary Islands. PMLR.
* Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of the International Conference on Learning Representations (ICLR)_, pages 1-13, San Diego, CA, USA.
* Kippendorff (1970) Klaus Krippendorff. 1970. Estimating the reliability, systematic error and random error of interval data. _Educational and Psychological Measurement_, 30(1):61-70.
* Li et al. (2017) Maolin Li, Nhung Nguyen, and Sophia Ananaidou. 2017. Proactive learning for named entity recognition. In _BioNLP 2017_, pages 117-125, Vancouver, Canada. Association for Computational Linguistics.
* Li et al. (2014a) Qi Li, Yaliang Li, Jing Gao, Lu Su, Bo Zhao, Murat Demirbas, Wei Fan, and Jiawei Han. 2014a. A confidence-aware approach for truth discovery on long-tail data. _Proceedings of the VLDB Endowment_, 8(4):425-436.
* Li et al. (2014b) Qi Li, Yaliang Li, Jing Gao, Bo Zhao, Wei Fan, and Jiawei Han. 2014b. Resolving conflicts in heterogeneous data by truth discovery and source reliability estimation. In _Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data_, SIGMOD '14, pages 1187-1198, New York, NY, USA. Association for Computing Machinery.
* Li and Roth (2002) Xin Li and Dan Roth. 2002. Learning question classifiers. In _Proceedings of the 19th International Conference on Computational Linguistics (COLING)_,

volume 1, pages 1-7, Taipei, Taiwan. Association for Computational Linguistics.
* Liu et al. (2012) Qiang Liu, Jian Peng, and Alexander T Ihler. 2012. Variational inference for crowdsourcing. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 25 (NIPS)_, pages 692-700. Curran Associates, Inc.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In _Advances in neural information processing systems 26 (NIPS)_, pages 3111-3119. Curran Associates, Inc.
* Nair and Hinton (2010) Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In _Proceedings of the 27th international conference on machine learning (ICML-10)_, pages 807-814.
* Nguyen et al. (2017) An Thanh Nguyen, Byron Wallace, Junyi Jessy Li, Ani Nenkova, and Matthew Lease. 2017. Aggregating and predicting sequence labels from crowd annotations. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL)_, pages 299-309, Vancouver, Canada. Association for Computational Linguistics.
* Poesio et al. (2017) Massimo Poesio, Jon Chamberlain, and Udo Kruschwitz. 2017. Crowdsourcing. In Nancy Ide and James Pustejovsky, editors, _Handbook of Linguistic Annotation_, pages 277-295. Springer Netherlands, Dordrecht.
* Poesio et al. (2013) Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation. _ACM Transactions on Interactive Intelligent Systems (TiiS)_, 3(1):3:1-3:44.
* Pyysalo et al. (2013) Sampo Pyysalo, Filip Ginter, Hans Moen, Tapio Salakoski, and Sophia Ananiadou. 2013. Distributional semantics resources for biomedical text processing. In _Proceedings of the Fifth International Symposium on Languages in Biology and Medicine (LBM)_, pages 39-44, Tokyo, Japan.
* Raykar et al. (2010) Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010. Learning from crowds. _The Journal of Machine Learning Research_, 11(Apr):1297-1322.
* Rodrigues et al. (2014) Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. 2014. Sequence labeling with multiple annotators. _Machine Learning_, 95(2):165-181.
* Sheshadri and Lease (2013) Aashish Sheshadri and Matthew Lease. 2013. SQUARE: a benchmark for research on computing crowd consensus. In _Proceedings of the 1st AAAI Conference on Human Computation (HCOMP)_, pages 156-164, Palm Springs, CA, USA. The AAAI Press.
* but is it good? evaluating non-expert annotations for natural language tasks. In _Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 254-263, Honolulu, Hawaii. Association for Computational Linguistics.
* Tokui et al. (2015) Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. 2015. Chainer: a next-generation open source framework for deep learning. In _Proceedings of Workshop on Machine Learning Systems (LearningSyss) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS)_, pages 1-6.
* Wang and Bi (2017) Xin Wang and Jinbo Bi. 2017. Bi-convex optimization to learn classifiers from multiple biomedical annotations. _IEEE/ACM Transactions on Computational Biology and Bioinformatics_, 14(3):564-575.
* Welinder et al. (2010) Peter Welinder, Steve Branson, Pietro Perona, and Serge J Belongie. 2010. The multidimensional wisdom of crowds. In _Advances in Neural Information Processing Systems 23 (NIPS)_, pages 2424-2432. Curran Associates, Inc.
* Whitehill et al. (2009) Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In _Advances in Neural Information Processing Systems 22 (NIPS)_, pages 2035-2043. Curran Associates, Inc.
* Yan et al. (2010) Yan Yan, Romer Rosales, Glenn Fung, Mark Schmidt, Gerardo Hermosillo, Luca Bogoni, Linda Moy, and Jennifer Dy. 2010. Modeling annotator expertise: Learning when everybody knows a bit of something. In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_, volume 9 of _Proceedings of Machine Learning Research_, pages 932-939, Chia Laguna Resort, Sardinia, Italy. PMLR.
* Yan et al. (2014) Yan Yan, Romer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. 2014. Learning from multiple annotators with varying expertise. _Machine Learning_, 95(3):291-327.
* Zhang et al. (2014) Yuchen Zhang, Xi Chen, Denny Zhou, and Michael I Jordan. 2014. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. In _Advances in Neural Information Processing Systems 27 (NIPS)_, pages 1260-1268. Curran Associates, Inc.
* Zhou et al. (2012) Denny Zhou, Sumit Basu, Yi Mao, and John C. Platt. 2012. Learning from the wisdom of crowds by minimax entropy. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 25 (NIPS)_, pages 2195-2203. Curran Associates, Inc.