Dynamic Worker-Task Assignment for High-Quality Task Resultswith ML Workers
Yu Yamashita

Masaki Kobayashi

Kei Wakabayashi

Atsuyuki Morishima

University of Tsukubas1811547@s.tsukuba.ac.jp

University of Tsukuba
makky@klis.tsukuba.ac.jp

University of Tsukuba
kwakaba@slis.tsukuba.ac.jp

University of Tsukuba
mori@slis.tsukuba.ac.jp

Abstract
How to obtain higher quality task results with a fixed budget(labor cost) is one of the most important problems in practical crowdsourcing. We approach this problem by (1) dynamically choosing appropriate sets of tasks and workers during
the worker-task assignment process and (2) introducing anML worker that performs tasks as if it was a human worker.
Our approach, named Dynamic Worker and Task Assignment
with ML workers (DWTAML), chooses tasks and workers
during the worker-task assignment process, based on the estimation of the reliability of workers and the confidence ofthe task results at each moment. It extends the Dawid&Skene
model and IEThresh. The results of an extensive set of experiments showed that DWTAML is effective in most cases.

Introduction
Quality management of task results has remained one of thecritical concerns in crowdsourcing and many methods havebeen proposed in the literature. Some of them are compatible with practical scenarios and easy to implement on thecurrent crowdsourcing platforms but not necessarily costeffective. Others are more sophisticated, but not necessarily
easy to implement on the crowdsourcing platforms today.
This paper proposes a method, named Dynamic Workerand Task Assignment with ML workers (DWTAML), thatcan be used in the practical setting and also more costeffective than existing ones. The method assumes a practical scenario today, where the followings hold (1) requesters
want to obtain better results of a set of tasks for a fixed budget, (2) they do not want to do a batch assignment to a fixed
set of workers (Chen, Lin, and Zhou 2013), but want to do
an online task assignment where we assign the best task toa worker who requests a task in the situation where workers
join and leave while other tasks are processed.
DWTAML has three features: (1) Dynamic blacklistingthat dynamically estimates the reliability of workers and updates the blacklist to exclude workers with low reliability
from the worker pool by extending IEThresh (Donmez, Carbonell, and Schneider 2009). (2) Task prioritization that
Copyright c 2020, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

dynamically estimates the confidence of task results by applying the Dawid&Skene model (Dawid and Skene 1979),
and assigns a task with an uncertain result to the worker.
(3) ML worker participation that assigns an ML algorithm
as a ‘worker’ tasks so that we can apply the Dawid&Skene
model in the situation we have the answer from one human
worker only for each task.
We conducted some simulation experiments. The results
showed that DWTAML is effective in most cases and especially effective with a small budget.

Proposed Method: DWTAML
We have as input (1) a fixed budget (labor cost) C. For simplicity, we assume that the monetary cost for a worker toperform a task is one, i.e., we will have C task assignments.
We also have (2) the set I of tasks and (3) the set J of labelsthat will be assigned to data objects shown in the tasks. Theoutput is the estimated true label, denoted by Â = {aˆi }i∈I .
Avoid assigning the tasks to the workers with low reliability. Interval Estimation (IE) (Kaelbling 1990) (Mooreand Schneider 1996) estimates the upper bound of confidence interval U Ik,t of a worker k at the time when t tasks
were performed, based on the agreement ratio between k’s
answers and correct ones so far. In DWTAML, we used
Dawid&Skene model to estimate correct answers from thek∈K
workers’ answers ri∈I
where K is the set of workers who
performed the tasks. Each worker k starts with U Ik,t = 1.0(as long as we do not have enough number of (i.e., two) answers from him or her for obtaining t-distribution) and thevalues become more precise as the number of answers increases. We make a blacklist of workers whose upper bounds
are lower than a threshold θ. For example, we can use the average of the upper bound of all workers:
XBWt = {k|U Ik,t <U Ik0 ,t /|K|}
(1)
k0 ∈K

where BWt denotes a set of workers to be in the blacklist att. It is so difficult to set the good threshold, but as we showedlater, the average threshold worked well in our preliminary
experiments.
Assign tasks with uncertain results to workers.
Dawid&Skene model estimates the posterior distribution ofTable 1: The accuracy of EM, DWA, DWTA, DWTAML(LR), and DWTAML(RF)workerbudgetEM
DWA
DWTA
DWTAML(LR)DWTAML(RF)

6000
0.832
0.832
0.832
0.909
0.895

β(8, 2)9000 12000
0.914 0.944
0.930 0.962
0.955 0.972
0.955 0.974
0.940 0.963

β(3, 2)9000 12000
0.679 0.771
0.728 0.834
0.817 0.912
0.845 0.917
0.804 0.901

6000
0.576
0.576
0.576
0.708
0.712
kthe true labels eij = p(ai = j|rik1 , rik2 , . . . , ri |K| ). We takemaxj∈J eij as the confidence of task i. We assign the task i0
with the least confidence to the worker, where i0 is:
i0 = arg min(max eij )ij∈J(2)

Assign an ML worker as the second worker in eachtask. Dawid&Skene model requires more than one worker
for each task to estimate the confidence of task results. DWTAML employs an ML algorithm as if it was another humanworker so that it is always able to apply the David&Skenemodel. Therefore, we call such an algorithm to be used as aworker an ML worker. An ML worker learns the answers
estimated by Dawid&Skene model at t − 1 and outputs
the labels of all tasks in I. Note that our purpose of using
ML algorithms is to obtain another worker, not to obtain agood classifier. We also note that there are ML algorithms
whose outputs are prediction probabilities. We interpret theDawid&Skene model in a probabilistic setting in order todeal with probabilistic answers from the ML worker.
Algorithm 1 summarizes DWTAML. It first assigns every task in I to workers, assuming that BWt = φ for
0 ≤ t < |I| (Line 1). Then the ML worker learns the answers and outputs labels (Line 2). Next, DWTAML performs
David&Skene model to compute Â at that moment. At last,
update the blacklist and the assigned task (Line 4). After receiving one answer to every task in I, the following iteration
proceeds each time it receives the answer of a task from aworker; Each iteration updates the blacklist of workers andthe assigned task. If a worker (not in the blacklist) requests atask, the task which has the results with the least confidence
is assigned to the worker as long as t < C.

Experiments
We conducted some simulation experiments.
Tasks and Workers. We used two distributions of worker
accuracies: (1) Synthesized distribution (Beta distribution
model and Hammer-Spammer model) and (2) Real distribution taken from Amazon Mechanical Turk (AMT). In
Hammer-Spammer model, 60 percent of all workers arehammers (always respond to the correct labels) and 40 percent of workers are spammers (respond to the labels uniformly at random). In Beta distribution model, the worker’s
reliability distribution follows the defined beta distribution.
We use β(8, 2) (average accuracy 0.8), β(3, 2) (average accuracy 0.6) and β(2, 3) (average accuracy 0.4). For the task,
we use CIFAR-10 (three classifications of Dog, Horse, andDeer) and Fashion-MNIST (three classifications of Coat,

6000
0.363
0.363
0.363
0.4090.434

β(2, 3)9000 12000
0.414 0.433
0.432 0.476
0.468 0.561
0.442 0.479
0.459 0.482

Hammer-Spammer
6000 9000 12000
0.890 0.961 0.980
0.890 0.998
1.0
0.890
1.0
1.0
0.999 0.999 0.999
0.997 0.999 0.999

3000
0.502
0.502
0.502
0.572
0.562

Real4500
0.548
0.560
0.577
0.596
0.596

6000
0.579
0.610
0.607
0.612
0.616

Algorithm 1 DWTAML
Input: A fixed budget C, the set I of tasks, andthe set J of labels
Output: The estimated true labels Â1: Have everytask i ∈ I performed by workers; t ← |I|
2: Let ML worker learn the answers and outputs the labels (probabilities) for all tasks i ∈ I
3: Perform David&Skene; Set âi ← arg max(eij )j

4: Update the blacklist and the assigned task with Eq 1 and 2
5: for each time a task result is submitted; t + + do
6:
Let the ML worker learn Â and output the labels (probabilities) for all tasks i ∈ I
7:
Perform Dawid&Skene; Set âi ← arg max(eij )j8:Update the blacklist and the assigned task with Eq 1 and 2
9: end for

Pullover, and Shirt). CIFAR-10 was used for the synthesized
distribution and Fashion-MNIST was used for the real distribution. It has 3000 tasks (images) and 200 workers in thesynthesized distribution. It has 1500 tasks (images) and 280
workers in the real distribution.
Budget. In CIFAR-10, we set the budget to 6000 (redundancy 2), 9000 (redundancy 3), and 12000 (redundancy 4).
In Fashion-MNIST, we set the budget to 3000 (redundancy2), 4500 (redundancy 3), and 6000 (redundancy 4).
Comparative Experiments. (1) EM: Dawid&Skenemodel. (2) DWA: It omits steps 2, 6, 7 in DWTAML algorithm. The task i0 to be assigned is decided in a roundrobin manner in (1) EM and (2) DWA. (3) DWTA: It omits
steps 2, 7 in DWTAML algorithm. (4) DWTAML (LR): ML
algorithm is Logistic Regression. (5) DWTAML (RF): ML
algorithm is Random Forest. Logistic Regression and Random Forest implemented by scikit-learn with default hyperparameters.
Result. Table 1 shows the accuracy of EM, DWA,DWATA, DWTAML(LR), and DWTAML(RF) in some simulation experiments. We found that (1) dynamic blacklisting and (2) task prioritization can improve task results fromalmost DWTA > DWA > EM. In addition, we found that
(3) ML worker participation can improve task results witha small budget. At last, we found that DWTAML can improve task results in most cases and especially effective witha small budget.

Acknowledgments
This work was supported by JST CREST Grant Number JPMJCR16E3 including AIP challenge, Japan.

References
Chen, X.; Lin, Q.; and Zhou, D. 2013. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In International conference on machine learning,
64–72.
Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood
estimation of observer error-rates using the em algorithm.
Journal of the Royal Statistical Society: Series C (AppliedStatistics) 28(1):20–28.
Donmez, P.; Carbonell, J. G.; and Schneider, J. 2009. Efficiently learning the accuracy of labeling sources for selective sampling. In Proceedings of the 15th ACM SIGKDD
international conference on Knowledge discovery and datamining, 259–268.
Kaelbling, L. P. 1990. Learning in embedded systems.
Moore, A. W., and Schneider, J. G. 1996. Memory-based
stochastic optimization. In Touretzky, D. S.; Mozer, M. C.;
and Hasselmo, M. E., eds., Advances in Neural Information
Processing Systems 8. MIT Press. 1066–1072.