Efficiency of active learning for the allocation of workers on crowdsourcing classification tasks ∗

Crowdsourcing has been successfully employed in the past as an effective and cheap way to execute classification tasks and has therefore attracted the attention of the research community. However, we still lack a theoretical understanding of how to collect the labels from the crowd in an optimal way. In this paper we focus on the problem of worker allocation and analyse two active learning policies proposed in the empirical literature. In the end we are able to show their equivalence and prove their superiority over an existing theoretical result by Karger et al. [2011]. Introduction and background Crowdsourcing has emerged as an effective way of completing a large number of simple and repetitive tasks by hiring workers from an online community in exchange for a small payment. Despite the unreliability of the single members, as a whole these crowds have been proved capable of producing valuable solutions at a fraction of the cost of traditional methods. A common way of interacting with the workers is to present them with a series of classification tasks and use their answers to infer the correct labels. This model has been successfully employed for various purposes including image labeling [Sorokin and Forsyth, 2008], ballot voting in complex workflows [Dai et al., 2011] and filtering [Parameswaran et al., 2012]. Even so, we still lack a principled approach towards the management of the crowd during the label collection process, and how to efficiently allocate the workers on the available tasks. One of the main obstacles to the realisation of such a goal is the heterogeneity in the quality of the answers collected from the crowd [Ipeirotis et al., 2010]. As a large number of them may be incorrect, the employer is forced to require multiple labels for each task in order to come up with a reliable conclusion. Over the past decades several statistical methods have been proposed to efficiently aggregate those labels and reduce the number of misclassified items [Dawid and Skene, 1979, Liu et al., 2012]. However, these methods are designed to be used after the labels have been collected from the crowd, and give no indication on how to optimise the collection process itself. This second question has recently received more attention in the literature [Slivkins and Vaughan, 2013]. On the theoretical side, Karger et al. [2011] analyse a scenario with homogeneous tasks and propose to assign the same number of workers on each task, showing that this approach is orderoptimal to the best allocation knowing the quality of the workers in advance. Conversely, when the tasks are heterogeneous, Ho et al. [2013] advocate the use of online primal-dual techniques to improve the efficiency of the allocation. However, the authors of these two papers fail to use the labels provided by the crowd as a potential source of information to improve the allocation process. In contrast, such piece of information has been extensively used in more empirical-oriented works on crowdsourcing [Kamar et al., 2012, Chen et al., 2013]. In particular Welinder and Perona [2010] propose an algorithm that computes the confidence in the classifications after every new label, and allocates more workers on the most uncertain tasks. Simpson and Roberts [2014] address the same problem from a different perspective and introduce a greedy algorithm to maximise the amount of information ∗This research is funded by the UK Research Council project ORCHID, grant EP/I011587/1 †University of Southampton ‡Imperial College, London