On the study of crowdsourced labeled data and annotators: Beyond Noisy Labels

In this PhD project, multi-class classification problems are considered. The standard framework is supervised classification: there is a dataset with instances (x) that belong each one to only one class c from a set of possible class labels ΩC , where |ΩC | > 2. The goal is to train a classifier φ that is as accurate as possible, based on the instances from a part of the dataset called training set. The examples in a training set are hand-labeled by means of expert knowledge. However, due to the rising volume of data and limitations regarding time, expert availability and/or features of the data, other techniques have emerged and their use has become more extended. Recently, and with the development of new technologies in the information era, the use of crowdsourcing has popularized through multiple web platforms (e.g. Amazon Mechanical Turk or CrowdFlower) that allow to process short tasks by using the workforce of thousands of workers of unknown expertise. In the last years, crowdsourcing has been widely used to solve different kinds of problems, such as text correction [1], text translation [2] or malaria diagnostics [3]. This new paradigm has been welcomed in the machine learning community as a means to collect labels for unlabeled instances in a fast way and at a low cost. In the machine learning context, the workers are referred to as annotators or labelers. Crowd labeling is the process of getting noisy labels for the instances in the training set from a set of various nonexpert annotators A. In this sense, an annotator a ∈ A can be seen as a classifier which provides labels with a certain amount of noise. As the annotators are not guaranteed to be experts, many labels are usually gathered for each example. In the traditional crowdsourcing scenario, every annotator is asked to select a single label for each instance. This crowd labeling approach is referred to as full labeling throughout this document. Crowd learning consists of learning a classifier from a dataset with crowdsourced labels. This learning task could be roughly separated into two stages: (i) label aggregation (to determine the ground truth label of each instance of the training set) and (ii) model inference (to learn a model using the aggregated labels and standard supervised classification techniques). If the collected labels fulfill certain conditions, crowd learning can be as reliable as learning from a single expert in a traditional supervised classification framework [4], [5]. Most of the approaches to crowd learning, mentioned below, focus on the first stage mentioned above (label aggregation). Probably the most popular label aggregation technique is majority voting (MV), which assigns to each instance the label that most annotators have selected for it. In weighted voting [6], the label selection of each annotator is weighted according to their reliability. As the expertise of the annotators is often unknown, their reliability has to be calculated based on the labels they have provided. Many aggregation methods that also model the reliability of annotators were derived from the expectation-maximization (EM) strategy [7]. This strategy was first implemented to learn with multiple (expert) annotators by Dawid and Skene [8], and has been widely used since then [9]–[14]. This method computes estimates for the ground truth and at the same time computes maximum likelihood estimates for the parameters that model the reliability of the annotators. It consists of two steps that are iterated until convergence: (i) Expectation (E-step), where the expected values of ground truth values are computed using the current parameter estimates and (ii) Maximization (M-step), where the parameters are updated with the new maximum likelihood estimates given the current expected data. There are some methods [9] that train a classifier throughout this process. In the model inference stage, the most common approach is to use the aggregated labels (which may be a single label or a probability distribution over the class labels for each instance [15], for example) to train a classifier. As can be seen, in the traditional crowd learning scenario, usually only the labels provided by the annotators are used to perform aggregation and model inference, disregarding the explanatory variables. An extensive review of different label aggregation and crowd learning techniques can be found in [16].