Learning from Imperfect Annotations: An End-to-End Approach

1 INTRODUCTION . The rising popularity and recent success of deep learning has resulted in machine learning systems that rely on large amounts of annotated training data ( LeCun et al. , 2015 ; Wu et al. , 2016 ; Gulshan et al. , 2016 ; Esteva et al. , 2017 ) . The most common , scalable way to collect such large amounts of training data is through crowdsourcing ( Howe , 2006 ) . Crowdsourcing works well in simple settings where annotation tasks do not require domain expertise—for example , in object detection and recognition tasks in natural images and videos ( e.g. , Deng et al. , 2009 ; Kovashka et al. , 2016 ) . However , annotation in specialized domains such as medical pathology requires a certain level of competency and expertise from the annotators which makes annotation expensive . Moreover , often times there is high rate of disagreement even between experts , which results in increasingly subjective and inconsistent labels ( Elmore et al. , 2015 ; Hutson et al. , 2019 ) . A typical approach to dealing with subjectivity is to treat each annotation as simply noisy , collect multiple redundant labels per example ( e.g. , from different annotators ) , and then aggregate them using majority voting or other more advanced techniques ( e.g. , Dawid & Skene , 1979 ; Carpenter , 2008 ; Liu et al. , 2012 ; Bachrach et al. , 2012 ; Zhou et al. , 2015 ; Zhou & He , 2016 ) to obtain a single “ ground truth ” label . At the expense of redundancy , this results in better data quality and more accurate estimates of the ground truth . More recently , the emerging systems for data programming and weak supervision also internally rely on label aggregation techniques similar to methods used for solving the crowdsourcing problem . Snorkel ( Ratner et al. , 2017 ; Bach et al. , 2019 ) is a popular such system and was designed for efficient and low-cost creation of large-scale labeled datasets using programmatically generated , so-called weak labels . However , as we show in our empirical evaluation none of these systems solve label aggregation effectively in the presence of high subjectivity . We argue that to become more effective , these methods need to make use of meta-data and other types of information that may be available about the data instances and the annotators labeling them . To this end , we propose a novel approach that allows us to train accurate predictive models of the ground truth directly on the non-aggregated imperfectly labeled data . Our method merges the two steps of : ( i ) aggregating subjective , weak , or noisy annotations , and ( ii ) training machine learning models . At training time , along with learning a model that predicts the ground truth , we also learn models of the difficulty of each example and the competence of each annotator in a generalizable manner ( i.e. , these models can make predictions for previously unseen examples and annotators ) . Our approach can be effectively used for training on crowdsourced data as well as on weakly labeled data , and also be used within frameworks such as Snorkel ( Ratner et al. , 2017 ; Bach et al. , 2019 ) and significantly improve their performance . We propose a method that can : 1 . Learn truth estimators : Learn functions representing the underlying ground truth , while imposing almost no constraints ( as opposed to prior work ) . In fact , we are able to leverage the capacity of deep neural networks along with the interpretability provided by Bayesian models , in order to obtain highly expressive estimators of the underlying truth . 2 . Learn quality estimators : Learn functions that estimate the quality of each annotator . When annotators can be described by some features ( e.g. , age , gender , location , etc . of an Amazon Mechanical Turk annotator , instead of just an ID ) , our quality estimators are able to generalize to new , previously unseen , predictors . Previous work only considered estimating accuracies of a fixed set of predictors , without being able to leverage any information we might have about them . Furthermore , in contrast to previous work , we are also able to predict the per-instance predictor comptencies ( i.e. , our method can determine whether a human annotator is an expert for a subset of queries , instead of just estimating his/her overall accuracy ) , which is done by learning dependencies between the instances and the annotators . Finally , our approach is able to distinguish between multiple different types of errors by estimating the full confusion matrix for each instance-predictor pair . 3 . Be easily extended : The truth and quality estimators can take arbitrary functional forms and fully leverage the expressivity of deep neural networks . Both human annotators and machine learning classifiers may sometimes be unable to make predictions about certain aspects of the ground truth ( e.g. , human annotators may be unsure about what the correct answer to a question ) . The proposed method is formulated in a way that allows it to be extended such that it can also learn decision function estimators for the annotators ( i.e. , estimators that predict whether an annotator will be able to provide a prediction for a given data instance ) . These estimators can have significant implications for data annotation systems where the cost of querying annotators is high ( e.g. , when these annotators are highly qualified , such as doctors or other kinds of domain experts ) . This is because it allows for better matching annotators to instances , thus reducing the required amount of annotation redundancy . An overview of the proposed approach and model is shown in Figure 1 , and a detailed description is provided in Section 3 . 2 RELATED WORK . Research on label aggregation and crowdsourcing dates back to the early 1970s , when Dawid & Skene ( 1979 ) proposed a probabilistic model to estimate ground truth labels using the expectation maximization ( EM ) algorithm . Since then , a variety of generalizations and improvements upon the original method have been proposed ( Whitehill et al. , 2009 ; Welinder et al. , 2010 ; Liu et al. , 2012 ; Zhou et al. , 2015 ; Zhou & He , 2016 ) . One of the central parts of the label aggregation algorithms is estimation of the accuracy of the annotators ( or predictors ) without having access to the ground truth . This problem has been of independent interest to the machine learning community , was termed as estimating accuracy from unlabeled data and studied by Collins & Singer ( 1999 ) , Dasgupta et al . ( 2001 ) , Bengio & Chapados ( 2003 ) , Madani et al . ( 2004 ) , Schuurmans et al . ( 2006 ) , Balcan et al . ( 2013 ) , and Parisi et al . ( 2014 ) , among others . Almost none of the previous approaches explicitly considered modeling the ground truth , but rather assumed either some form of independence or knowledge of the true label distribution . Collins & Huynh ( 2014 ) reviewed many methods that were proposed for estimating the accuracy of medical tests in the absence of a gold standard . Platanios et al . ( 2014 ) proposed formulating the problem as an optimization problem that uses agreement rates between multiple noisy annotators . Platanios et al . ( 2017 ) improved upon agreement-based accuracy estimation using logical constraints between the noisy labels . Tian & Zhu ( 2015 ) proposed a max-margin majority voting scheme applied to crowdsourcing . More recently , Khetan et al . ( 2017 ) proposed to use a parametric function to model the ground truth and showed that the approach can sometimes be functional even in the limit of a single noisy label per example . Among recent approaches , Zhou et al . ( 2015 ) formulated the problem as a form of regularized minimax conditional entropy and established one of the most competitive baselines on many public crowdsourcing datasets . Our proposed method generalizes the approaches of Zhou et al . ( 2015 ) , Platanios et al . ( 2016 ) , and Khetan et al . ( 2017 ) . Similar to Platanios et al . ( 2016 ) , we define a generative process for our observations . However , our model is able to handle categorical labels , as opposed to just binary . Similar to Zhou et al . ( 2015 ) , we define the confusion matrix for each instance-predictor pair as a function of instance difficulty and predictor competence.1 However , we explicitly learn the difficulty and competence as functions , which allows us to generalize to previously unseen instances and annotators . Interestingly , the inference algorithm for our generative probabilistic model has a similar form to that of Zhou et al . ( 2015 ) ( except for the explicit learning of the ground truth , difficulty , and competence functions ) . Thus , we also show that the algorithm of Zhou et al . ( 2015 ) can be re-derived as an EM inference algorithm for a generative model , simplifying the argument of the original paper . Finally , similar to Khetan et al . ( 2017 ) , we use a parametric function to model the ground truth , and also go a step further and propose to use parametric functions to model the instance difficulties and predictor competences . Thus , our approach enables estimation of which annotators are likely to perform better on which instances , potentially enabling more optimal allocation of annotators and thus annotation cost reductions . In contrast to prior work , our method also allows for end-to-end learning . Prior methods do not allow for this as they ( implicitly ) separate ground truth inference ( i.e. , label aggregation ) from model training . More specifically , previously one would have to train a machine learning model in two stages : ( i ) infer the ground truth labels from the provided annotations , and ( ii ) train a machine learning model on the inferred labels . Our approach merges these two stages and allows us to train machine learning models directly on the imperfect annotations . In Section 4.3 , we conduct an ablation study that showcases the performance gains obtained by employing end-to-end learning in this fashion . 3 PROPOSED METHOD . We denote the observed data by D = { xi , Ŷi } Ni=1 , where Ŷi = { Mi , { ŷij } j∈Mi } , Mi is the set of predictors that made predictions for instance xi , and ŷij is the output of predictor f̂j for instance xi . Our goal is to learn functions representing the underlying ground truth and predictor qualities , given our observations D. 1The idea of modeling instance difficulties and annotator competencies has been studied before by Carpenter ( 2008 ) and Bachrach et al . ( 2012 ) , among others . Ground Truth . We define the ground truth as a function hθ ( xi ) that is parameterized by θ and that approximates the true distribution of the label given xi . In our setting , hθ ( xi ) ∈ RC≥0 and∑ j [ hθ ( xi ) ] j = 1 , where C is number of values the label can take ( i.e. , assuming categorical labels ) . More specifically , [ hθ ( xi ) ] k , P ( yi = k | xi ) , where we use square brackets and subscripts to denote indexing of vectors , matrices , and tensors . For example , hθ could be a deep neural network that would normally be trained in isolation using the cross-entropy loss function . In our method the network is trained using the Expectation-Maximization algorithm , as described in the next section . Predictor Qualities . We define the predictor qualities as the confusion matrices Qij ∈ RC×C≥0 , for each instance xi and predictor f̂j , where ∑ l [ Qij ] kl = 1 , for all k ∈ { 1 , . . . , C } . [ Qij ] kl represents the probability that predictor f̂j outputs label l given that the true label of instance xi is k. We define these confusion matrix in a way that generalizes the successful approach of Zhou et al . ( 2015 ) :2 Qij = Di •3 Cj , ( 1 ) where •i represents an inner product along the ith dimension of the two tensors , and : – Di = dφ ( xi ) represents the difficulty tensor for instance xi , where d is a function parameterized by φ , Di ∈ RC×C×L , and L is a latent dimension ( it is a hyperparameter of our model ) . [ Di ] kl− is an L-dimensional embedding representing the likelihood of confusing xi as having label l instead of k , when k is its true label . – Cj = cψ ( rj ) represents the competence tensor for predictor f̂j , where c is a function parameterized by ψ , rj is some representation of f̂j ( e.g. , could be a one-hot encoding of the predictor , in the simplest case ) , and Cj ∈ RC×C×L . [ Cj ] kl− is an L-dimensional embedding representing the likelihood that predictor f̂j confuses label k for l , when k is the true label . Using L > 1 allows the instance difficulties and predictor competences to encode more information . An intuitive way to think about this is that we are embedding difficulties and competencies in a common latent space , which can be thought of as jointly clustering them . This is in fact very similar to how matrix factorization methods are used for collaborative filtering in recommender systems . Our goal is to learn functions hθ , dφ , and cψ , given observations D. In order to do that , we propose the following generative process for our observations . For i = 1 , . . . , N , we first sample the true label for xi , yi ∼ Categorical ( hθ ( xi ) ) . Then , for j ∈ Mi , we sample the predictor output ŷij ∼ Categorical ( [ Qij ] yi− ) , where [ Qij ] yi− represents the yith row of Qij . In the next section , we propose an algorithm learning the parameters θ , φ , and ψ .

Summary: The submission addresses a problem with collecting ground truth: human annotations are noisy, a common approach is to collect many annotations and apply majority voting to elicit a single label. With this approach, the annotators' expertise and the difficulty of single data instances is ignored. What the authors propose is a framework which allows one to combine a direct graphical model of how human annotations are produced with model training. The graphical model introduces latent variables for the difficulty of an instance and the competence of an annotator as well as the (unobserved) true label. This way one can potentially benefit from the meta-information about the annotators (e.g., their demographics) and improve upon the majority-voting baseline of aggregating available annotations. Maybe most importantly, the proposed framework allows one to get the true label with fewer annotations (significantly reduces redundancy).