Doubly Active Learning : when Active Learning meets Active Crowdsourcing

In this paper we study an active learning setting where training samples are adaptively selected to be labeled, and the learner can only query a set of crowdsourcing workers with unknown expertise level for label information. We aim to perform active sample assignment and active worker selection jointly. Though the idea is straightforward, we face several technical challenges towards reaching our goal. First, without knowing worker’s expertise level (and hence the error rate of the solicited data), design of active learning procedures can be challenging. Secondly, due to the lack of groundtruth labels, we need a learning framework that helps differentiate workers. Finally, interleaving active sample selection and active worker selection requires non-trivial effort. We propose an Upper Confidence Bound (UCB)-alike algorithm Crowd UCB for selecting the best worker among a candidate pool, without knowledge of the ground-truth. Then we connect Crowd UCB with a parameter free active learning algorithm. We provide performance guarantees for the proposed algorithms. Introduction With increasing demand for data-efficient learning methods, active learning rises as a promising option for spending label budget more efficiently. The idea is fairly simple: by carefully collecting labels of data points near the decision boundary (possibly in a sequential manner), fewer samples are required to learn a model up to a certain precision. Past works have developed extensive results on improving classification performance via active learning (Balcan et al. 2006; Balcan et al. 2007; Balcan and Long 2013; Wang 2011), where labels for only a set of selected data samples are collected. When there is no oracle being available for querying labels, such labeling tasks are often outsourced to crowdsourcing workers (Abraham et al. 2013; Ho and Vaughan 2012; Liu and Liu 2015). Interestingly the notion of active selection has also been studied recently in the context of worker selection in crowdsourcing, where workers may contribute label information with different qualities. Similar to the idea of active learning setting where data points are carefully screened and selected for labeling, in active crowdsourcing Copyright c © 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. an algorithm selects different workers to label data samples. The hope is to collect more consistent labels by exploiting the heterogeneity of a pool of workers. In this paper, we consider performing active data sampling and worker selection jointly. In practice, actively crowdsourcing tasks within an active learning context helps save labeling budget, compared to an uniform data assignment mechanism (via avoiding labeling non-useful samples). We also provide an analytical framework to study active learning where label queries do not necessarily go to an oracle or a fixed noisy worker (with either known or unknown expertise levels). Though the idea is natural, its implementation is highly non-trivial. On one hand, the presence of several workers with diverse labeling capabilities significantly affects the active data selection procedure, as naive data selection methods do not take into account the heterogeneity of workers’ expertise levels. To make matters even worse, many existing active learning algorithms would fail without knowing properties of the noisy labeling oracles (Balcan et al. 2007; Balcan and Long 2013), which adds difficulties to this “doubly-active” selection problem. On the other hand, due to the lack of ground-truth (labels), how to explore or learn the workers’ expertise level is not clear either. Some ideas of using redundant assignments (Karger et al. 2013; Karger et al. 2011; Abraham et al. 2013; Liu and Liu 2015) were proposed and studied. But such redundancy will also degrade the learning performance when facing fixed budget. For example, when each sample is assigned twice, only half of the budget will be made effective in the final learning outcome. In this paper, we propose a novel framework for the doubly active learning problem. We consider the specific problem of learning a multi-dimensional homogeneous 1 binary linear classifier under Tsybakov noise conditions by querying a pool of workers (oracles) with varied, yet unknown levels of accuracy. We propose an Upper Confidence Bound (UCB)-alike algorithm Crowd UCB for selecting the best worker among a candidate pool, without knowledge of the ground-truth. Then we connect Crowd UCB with a parameter free margin based active learning algorithm. The proposed algorithm enjoys rigorous guarantees for its perforA binary linear classifier f is homogeneous if its decision hyperplane passes through the origin (Dasgupta 2005). mance. In addition, for a flipping error worker model, the performance of our proposed algorithm approximates the best active learning performance with the best worker in the worker pool. We believe our model is generally applicable to many other possible worker models. Throughout the paper, we assume no prior knowledge on the quality of each individual worker, which would instead be estimated on the fly. Proofs of some technical lemmas are placed in the supplementary material. Related works Active learning: As a fast progressing area, it is nearly impossible to survey all relevant works in active learning. In the classical active learning setting a learner receives a pool of unlabeled data points and decides, in a feedbackdriven manner, a small portion of data points whose labels are requested (Hsu 2010). In most of existing active learning work only one worker (labeling oracle) is present and the only worker answers all label requests made. One notable exception is (Zhang and Chaudhuri 2015) where both a strong labeler and a weak labeler are considered at the same time. In (Zhang and Chaudhuri 2015) the qualities of both labelers are known a priori and label requests directed to the weak labeler are unlimited, both are different from our models. Being aware of noise in imperfect labelers, a relabeling based active learning algorithm was proposed in (Lin and Weld 2016) to gradually remove uncertainties in labels. Most relevant to our work is (Yan et al. 2011), where the idea of active learning with crowd is well executed; yet no analytical framework nor theoretical guarantees was given. Instead we provide such a framework and sample complexity results. Active worker selection: The notion of active worker selection has been adopted primarily in a crowdsourcing setting where the worker’s expertise in labeling data differs from each other and is unknown a-priori. In (Abraham et al. 2013), an adaptive bandit survey problem is studied and a stopping time for task assignment when facing crowds of workers is determined. (Ho and Vaughan 2012) considered an online task assignment problem in crowdsourcing market. (Liu and Liu 2015) proposed an online algorithm that learns to select the best combination of workers using redundant assignments. Both (Tran-Thanh et al. 2014) and (Ho et al. 2013) studied active worker selection under different settings (a non-active classification setting and a budget constrained setting respectively), assuming access to feedbacks on workers’ qualities, following each selection. In line of game theory approaches, (Bhat et al. 2014) studies the issues of cost-quality balance when workers/labelers are strategic. In (Liu and Chen 2017), authors proposed a sequential peer prediction approach to incentivize high quality workers. As far as we know, none of the active worker selection works has considered an active learning setting, which turns out to be a technically challenging task as we will elaborate later. Proactive learning: The proactive learning setting is designed specifically to relax certain unrealistic assumptions (e.g., on noise in labels) made by most active learning research (Donmez 2010). One particularly interesting case is when multiple workers (labeling oracles) are available with varied unknown qualities. Prior research on proactive learning primarily focuses on heuristics that improve performance of practical systems (Donmez and Carbonell 2008; Donmez 2010), while theoretical analysis is few. (Yang and Carbonell 2009) derived theoretically sound algorithms for the setting where workers have different bounded labeling error. However, bounded noise is a rather strong assumption and is usually not satisfied in practice. In all, despite solid theoretical understandings on both of the active procedures separately, there is a lack of theoretical foundation for this doubly active learning question, which is one of our major motivations for carrying out this study.