Journal of Information Processing Vol.27 51–60 (Jan. 2019)[DOI: 10.2197/ipsjjip.27.51]

Regular Paper

Additional Operations of Simple HITs on Microtask
Crowdsourcing for Worker Quality Prediction
Yu Suzuki1,a)Yoshitaka Matsuda1

Satoshi Nakamura1

Received: April 16, 2018, Accepted: October 2, 2018

Abstract: In microtask crowdsourcing, low-quality workers damage the quality of work results. Therefore, if a system automatically eliminates the low-quality workers, the requesters will obtain high-quality work results with low
wages. When we consider simple Human Intelligent Tasks (HITs), such as yes-no questions of a labeling task, therequesters have diﬃculty assessing the worker quality only from the work results. Therefore, we need a method toaccurately predict the worker quality automatically from the behaviors of workers, such as working time and the number of clicks. When we accurately predict the worker quality, we are able to prepare many features from the worker
behaviors. However, when we submit simple HITs, we can capture only a small number of behaviors of workers, thenthe accuracy of predicted worker quality will be low. To solve this issue, we propose a method to insert into the simple
task of obtaining many features of worker behaviors. We prepared a classification task of tweets as simple HITs. We
added a button to the work screen. The workers can browse the target tweets on the work screen during the time theworkers are pressing the button, but the workers cannot browse the target tweets when the workers have released the
button. Using this button, we can obtain six more kinds of features of worker behaviors. Using our method, we can
improve the recall ratio 12% of identifying low-quality workers. However, as the load of workers increases, then theprocessing time becomes longer, and the motivation of workers decreases. From this result, we also discovered that
there is a trade-oﬀ between the number of obtained behaviors and the load of workers.
Keywords: crowdsourcing, worker quality, machine learning, random forest, microtask, active intervention

1. Introduction
In microtask crowdsourcing, not all workers are able to produce results of the required quality. Moreover, some workers may
have malicious intent and insuﬃcient knowledge about the work
to be done [1]. In this paper, worker quality is quantified as theprobability of a worker doing the work correctly. This definition
is followed by Refs. [2] and [3]. By hiring high-quality workers
and by eliminating or instructing low-quality workers, requesters
can reduce the number of inaccurate work results. Accordingly,
assessments of worker quality should be viewed as essential for
improving the quality of task results.
Here, we consider simple human intelligence tasks (HITs),
such as answering yes-no questions as annotations. In such HITs,
requesters find it diﬃcult to determine the worker quality only
from their results. This is because the workers who correctly
select yes or no without browsing their work screens cannot be
distinguished from those who carefully select the options. Moreover, requesters cannot prepare correct answers if they have theintention that only the workers should decide the results and therequesters do not want to bias the results.
To solve this problem, researchers have proposed several methods that measure worker behaviors, such as working time andnumber of clicks, for predicting their quality [4], [5], [6], [7].
The systems embodying these approaches automatically obtainworker behaviors on their work screens and estimate worker qual1a)Nara Institute of Science and Technology, Ikoma, Nara 630–0192, Japanysuzuki@is.naist.jp

c 2019 Information Processing Society of Japan

ity by using machine learning. The research has mainly focusedon methods to analyze behaviors, rather than methods to analyze
the behaviors of workers. Moreover, the target tasks in the research tend to be relatively complex, because the systems should
gather many features from the workers. Therefore, these methods
are diﬃcult to apply to relatively simple HITs for which the system can obtain only a small number of behaviors from the work
screens. To predict worker quality accurately, we should extract
as many features from the behaviors as possible.
If we cannot capture many features from the behaviors duringsimple tasks, the accuracy of the predicted worker quality would
be low. To solve this problem, we propose a method that adds
operations on the work screens for the purpose of obtaining many
features from their behaviors. We prepared a classification task oftweets, which we consider to be a simple task, as a baseline task.
The purpose of this task is to classify which tweets are related to
Kyoto sightseeing. We added a button to the work screen, which
we call the “proposed task.” In the proposed task, workers canbrowse the target tweets only while they are pressing the button.
Whereas we can obtain five kinds of behavior from the baselinetask, and six more kinds of behavior from the proposed task.
In the proposed task, the recall ratio of low-quality workers improved by at least 12% from the baseline task. We also found thatbehaviors, such as browsing time and browsing count, obtainable
only with the proposed task play an important role in predicting
the worker quality. On the other hand, we noticed a trade-oﬀ relationship between processing time and worker dropout rate. The
findings in this research can be used as guidelines for designing

51

Journal of Information Processing Vol.27 51–60 (Jan. 2019)

crowdsourcing systems that strike a balance between prediction
accuracy of quality and workload of workers.
We survey related work in Section 2. We describe the baseline
HIT in Section 3. Then, we describe how we added operations
on the work screens to obtain more features in Section 4. We describe our empirical experiments in Section 5 and conclusions in
Section 6.

2. Related Work
There are three major approaches to predicting worker quality:
1) gold tasks, 2) analysis of work results, and 3) assessment ofworkers by analyzing their behavior.
The gold task approach mixes HITs that have answers (theseare prepared by the requesters) and HITs which do not have answers. By comparing the answers of the workers with those of
the requesters, one can automatically calculate the worker quality [8]. For individual tasks, it is necessary to create a gold taskthat correctly judges the quality of the worker, the result of whichis added labor and costs for the requester. In addition, it is necessary to pay the worker for performing the gold task.
The approach predicts the worker quality from their results [9],[10]. Redundancy-based methods assign multiple workers toeach HIT and aggregate their results. These techniques areused [11] for obtaining high-quality work results. For example, ina text labeling task, multiple workers are requested to add a label
to the same piece of text. Then, the system integrates the obtainedlabels by majority vote [12], [13] or by using an expectationmaximization (EM) algorithm [9], [14]. The requester does notneed to prepare correct answers when using this method. The basic idea of the redundancy-based method is that the majority of
the labels are the correct answers. Therefore, it does not work
well when low-quality workers are the majority. Also, for requesters, the wages and temporal costs increase in proportion to
the number of workers made redundant. By contrast, our method
reduces the number of workers assigned to each HIT. Moreover,it can be combined with the redundancy-based method to output
high-quality work results at a lower cost.
Approaches that predict worker quality from their behavior
have been proposed [5], [6], [7], [15]. They do not require thecreation of a gold task and do not assume redundancy; hence,
they make it possible to reduce costs. We decided to focus on thissort of approach in our research.
Rzeszotarski et al. [5] proposed a method that uses worker behavior for predicting the worker quality. They gathered feature
data such as the number of clicks, keyboard operations, and processing time from the workers. On the other hand, Hirth et al. [6]used the following features: the duration in which the worker
reads the target text, and the answer time in which the answer isconsidered as determined from scrolling of the page or the click
interval of the radio button. Moreover, they predicted the qualityof the workers by using a machine learning algorithm that took
the workers’ behaviors as input. They considered work events
such as mouse and keyboard operations to be worker behaviors.
Mok et al. [15] tackled the problem of extracting worker behavior from measurements in detail. They analyzed the mouse cursor
position by using submovement analysis [16] and grasped the be-

c 2019 Information Processing Society of Japanhavior of the worker not by the timing of the event but by the flow.
In addition, machine learning algorithms have been used for estimating the worker quality as well. Rzeszotarski et al. [7] assumed
that preparing data on fully trained workers in advance improves
the accuracy prediction for the worker quality. They used a machine learning model for predicting the quality of worker, where
the behavior of fully trained workers was used as training data.
In these studies, the target HITs were relatively complicatedtasks from which the systems could identify many kinds of behavior. For example, Rzeszotarski et al. imposed a word classification task; this single HIT can be divided into multiple HITs,
and high-quality results can be obtained by dividing the task as
finely as possible [1], [17].
On the other hand, there are many simple HITs, such as text
labeling tasks and image annotation tasks, submitted by many requesters. However, if the number of features the system obtains
is limited, the above methods do not work so well and cannot beapplied as is. On simple HITs, workers do not move the mouse
cursor and do not scroll page frequently [18]. Therefore, we cannot acquire enough behavior data for the machine learning modelto accurately predict the worker quality. Furthermore, with only
a few kinds of behavior, it is diﬃcult to judge the worker quality
even manually. As a solution to these problems, in this research,
we extracted more features from simple HITs with little expression of behavior by adding more operations to the worker screens.

3. Baseline
We describe the baseline task, in particular, the procedure followed by the workers and the method to predict worker quality
in the baseline task. We selected the features that we used in thisbaseline method by referring to Rzeszotarski et al. [7].
3.1 HITs
We devised a task for labeling twitter texts. Here, the requesterneeds to find texts which are relevant to sightseeing in Kyoto. We
prepared a set of tweets of which some were relevant to sightseeing in Kyoto. Then we had workers browse several tweets andclassify whether they are useful for tourists visiting Kyoto by following both of two evaluation criteria:• Tourists can (partially) understand the (current or past) situation of Kyoto.
• The situation is useful for sightseeing in Kyoto.
The workers were to browse a tweet, consider whether it satisfies these two criteria, and select “Yes,” “No,” or “Tweet is not
displayed” for each tweet. Figure 1 shows a work screen for thisbaseline task. For example, if there is a tweet “Kyoto station isFig. 1 Work screen for the baseline task.

52

Journal of Information Processing Vol.27 51–60 (Jan. 2019)

Table 1
ID
B1
B2
B3
B4
B5

Worker behaviors used for baseline method and proposed method (B1 , B2 , · · · , B5 ).

Behavior
Processing time
Processing time per character
Response time
Response time per character
Number of responses

Explanation
Total time from the display of the work page until pressing the send button (seconds)
Processing time divided by the number of characters of tweet (seconds)
Time from display of work page to selection of answer (seconds)
Response time divided by the number of characters of tweet (seconds)
Number of changes in answer (times)

very crowded” on the work screen, workers should select “Yes,”
because when the tourists browse this tweet, they can avoid goingto Kyoto station when it is crowded. On the other hand, if there
is a tweet “I am in Kyoto,” the workers should select “No,” because the tourist does not learn any useful information from thistweet. The option “Tweet is not displayed” is used if someone
has deleted the tweet. Our crowdsourcing platform cannot display tweets on the work screen if they have been deleted from theoriginal twitter site. When a worker pushes the submit button, thenext tweet is displayed.
This task is easy to answer for a human, but diﬃcult for machine learning. For example, if there is a tweet “Kyoto stationis crowded.,” it should be easy for both machine learning and ahuman to select the appropriate option “Yes,” because there aremany terms which are related to sightseeing in Kyoto, such as
Kyoto, station, and crowded. However, if there is a tweet “Thistrain is crowded. I would have liked to go sightseeing in Kyoto
unless it is crowded.,” it is diﬃcult to know whether this text
is useful or not, because there is no information about Kyoto.
When this tweet is processed automatically, the system may select “Yes,” because there are many keywords related to Kyoto
sightseeing in the text, such as Kyoto, train, and crowded.
3.2 Worker Quality Prediction
Here, we describe a method for predicting the worker quality
from the worker behaviors on the work screen. The extracted features used in the baseline method are almost the same as thoseused by Rzeszotarski et al. [7]. We used supervised learning for
estimating the worker quality. We assessed the workers who processed at least 100 HITs.
3.2.1 Data
3.2.1.1 Feature Vectors as an Input of the Classifier
First, we describe the features of the worker behaviors. We
monitored the work screens and obtained five kinds of worker behaviors B1 , B2 . · · · , B5 (Table 1). We acquired these behaviors
every time a worker processed one HIT. Then, we calculated theaverage value, median value, maximum value, minimum value,
standard deviation, and entropy of each behavior obtained foreach worker and used them as feature values for each worker.
The feature values of each behavior xwi (i = 1, 2, · · · , 5) whichcorrespond to B1 , B2 , · · · , B5 are defined as follows:
xwi = [ BiAve BiMed BiMin BiMax BSi td BiEnt ],(1)

where BiAve is the average value, BiMed is the median value, BiMin
is the minimum value, BiMax is the maximum value of Bi BSi td isthe standard deviation, and BiEnt is the entropy of Bi .
In addition, we used the number of processed HITs Nw and thetotal processing time T w as the features of worker behaviors. Accordingly, the feature vector xw of worker w is defined as follows:c 2019 Information Processing Society of Japan

xw = concat[ Nw T w x1w x2w x3w x4w x5w ]Sample5, 10, 60
0.05, 0.1, 1
5, 10, 60
0.05, 0.1, 1
1, 2, 5(2)

3.2.1.2 Worker Quality as an Output of the Classifier
The values which the classifier predicts are worker qualities.
We used the correct answer rate rw as follows:
rw =

Mw
Nw

(3)

where Mw is the number of processed HITs with correct answers.
We created the correct answers using majority voting. We assigned ten workers to each tweet and set the correct option as theone selected by six or more workers. We checked the data andfound that there is no tweet which the maximum number of votes
was not less than half of the total number of votes. Moreover,there are less than 1% of tweets in which the maximum numberof votes were six. Therefore, we believe that this dataset is adequate to be used.
Next, we calculated the worker quality Qw to determine which
workers were low-quality and which were high-quality as follows:⎧⎪⎪⎪⎨−1 if rw ≤ β
Qw = ⎪
(4)
⎪
⎪1
⎩
else
where Qw = −1 means that the quality of w is low, and Qw = 1
means that the quality of w is high. α is the ratio of the lowquality workers to the total number of workers. β is a threshold of
low-quality workers. We treat a worker as low-quality if his/hercorrect answer rate is less than β. For example, if α equals 0.1,
10% of the workers are low-quality and Qw = −1 and Qw of theother workers equals 1. If the highest correct answer rate of the
low-quality workers is 0.7, β is 0.7. If α is given, β is automatically calculated.
In our experiments, almost all workers were high-quality.
Therefore the numbers of high-quality and low-quality workers
were imbalanced in the training data. When we classified workers, almost all of them were classified into major classes incorrectly. Therefore, we used the SMOTE algorithm [19] which virtually increases the samples of a small number of classes so thatthe numbers of high-quality and low-quality workers would almost be the same.
3.2.2 Classifier Construction
The classifier f was as follows:
f : xw → Q̂w ∈ −1, 1(5)where Q̂w is the quality of worker w predicted by the classifier f .
To construct f , we prepared a set of workers with xw and Qw . We
used random forests [20] because the classifier should be resistantto outliers. There are many outliers in the input feature values.

53

Journal of Information Processing Vol.27 51–60 (Jan. 2019)

Table 2 Hyperparameters.
name# estimatermaximum # of features

candidates
2, 3, 4, 5, 6, 7, 8, 9, 10
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ‘auto’, NoneFig. 3Fig. 2 Example of processing times of three workers.

For example, suppose a worker performs an HIT while watching
TV. If he or she becomes very distracted, the maximum processing time per HIT could be more than one hour. Friedman etal. [21] discovered that the random forest is resistant to this kindof outlier. They also discovered that random forests could accurately predict if many features were not useful for prediction. We
used a grid search for setting appropriate hyperparameters in thecandidates shown in Table 2.

4. Proposed Method
Here, we describe the procedure of the HITs, the method topredict worker quality, and the diﬀerences between the baseline
and proposed method.
4.1 Basic Idea
First, let us consider the workflow in the tweet classification
task described in Section 3.1 and discuss the features related toprocessing time. Figure 2 shows several examples of worker behaviors in the HITs. The figure shows three types of workers;
A, B, and C. Worker A first browses a tweet for a long time before answering. Worker B works while watching TV; she does
not continuously browse tweets. Worker C first browses a tweet
and answers; then she browses the tweet again to re-check the answer. From these worker behaviors, we intuitively find that C is
the highest quality and B is the lowest quality.
Current crowdsourcing platforms do not monitor worker behaviors directly; the platform could not understand whether theworkers were browsing tweets, watching TV, or doing somethingelse. In this example, the processing times of browsing a tweet
and answering for each worker are actually diﬀerent, but the system treats these times as the same. Therefore, if we extract features listed at the baseline method in Table 1 from the behaviorsof the three workers, the feature values would be the same; theclassifier considers the three workers as having the same quality.
Based on the above, we believe that we can estimate the worker
quality more accurately if we could obtain more data on their behavior. For this reason, the proposed method obtains the behaviorof the browsing part of the task. In particular, we added operations on the work screens to obtain the browsing time and thenumber of browsings. We verified the eﬀectiveness of such behaviors which have not been dealt with in previous research on
predicting the worker quality.

c 2019 Information Processing Society of Japan
Work screen of intervening task.

4.2 HITs
In the proposed method, tweets are not simply displayed on
the work screen. Instead, a button, which we call the browsing
button, has to be pressed and held down to display the tweets.
We show the work screen in Fig. 3. The left side of this figure is
the work screen when the browsing button is not pressed, and theright side is the work screen when the worker presses the browsing button. When the worker releases the browsing button, thetweet disappears from the work screen, as indicated on the left
side of the figure.
The system can obtain the browsing time and frequency fromthe browsing button operations. For example, if a worker is doinga task while watching TV, the processing time B1 will likely belonger.
4.3 Feature Values
Using the browsing button described in Section 4.2, we constructed the feature values shown in Table 1 and Table 3. We
predicted the worker quality using random forests, as in the baseline method. The feature vectors xwp of worker w in the proposed
method are defined as follows:
xwp = concat[ Nw T w x1w x2w . . . x11
w ]

(6)

where xwi (i = 1, 2, · · · , 11) is the feature value which corresponds
to B1 , B2 , · · · , B11 defined by Eq. (1).

5. Experimental Evaluation
In this section, we describe our experimental evaluation to verify whether the features of the worker behaviors obtained by theoperations that were added to the work screens are eﬀective forpredicting worker quality. We also investigate the eﬀect of thebrowsing buttons on the relationship between the prediction accuracy of worker quality and workload of workers.
5.1 Experimental Setup
We constructed our crowdsourcing platform as a Web application using Ruby on Rails 5.1 and Oracle Database 12c. All
workers we hired performed the HITs on our platform. We used
jQuery *1 , a javascript library, to obtain the worker behaviors on
the work screens.
5.1.1 Worker
We hired workers through Crowdworks *2 , a major crowdsourcing platform in Japan. All workers who applied for our task
were hired without filtering; we did not select the workers to be*1*2

https://jquery.com/
https://www.crowdworks.jp/

54

Journal of Information Processing Vol.27 51–60 (Jan. 2019)

Worker behaviors used only in proposed method (B6 , B7 , · · · , B11 ).

Table 3
ID
B6
B7
B8
B9
B10
B11

Behavior
Browsing countBrowsing timeBrowsing time per character
Browsing time per countBrowsing time per count andcharacter
Browsing ratio

Explanation
Number of times tweet display button was pressed (times)
Total time while holding down tweet display button (seconds)
Browsing time divided by the number of characters of tweet (seconds)
Browsing time divided by the browsing count (seconds)
B7 divided by B6 and B8 (seconds)

Sample1, 2, 5
5, 10, 60
0.05, 0.1, 1
5, 10, 60
0.05, 0.1, 1

Percentage of browsing time out of total processing time

0.1, 0.5, 0.8

Table 4Baseline
ProposedSum# workers
439
486
793

Number of workers, HITs, and tweets.

# active workers
250
255
439

# early departed workers
189
231
354

hired in advance. Workers can earn 30 JPY (about 0.3 USD) as
wages for classifying 100 tweets. The average worker classified
about 450 tweets per hour and earned 135 JPY, which is almost
equal to the standard crowdsourcing wage (1.38 USD/hour) [22].
Workers could freely suspend or resume their work. As a result,some workers processed fewer than 100 tweets, while others processed more than 10,000 tweets.
5.1.2 Data
Table 4 shows the number of workers we hired and the total
number of HITs when using the proposed method and the baseline method. Here, “#workers” refers to the total number of workers who applied for our task and engaged in it more than once, and“#active workers” refers to the workers who processed more than
100 HITs. “#early departed workers” refers to the workers who
are not active workers. “#processed HITs” is the number of tasks
processed by all workers. “#tweets” is the number of tweets classified by more than one worker. In our system, we assigned ten
workers to each tweet. However, more than two workers reported
that the tweets had been deleted, so we canceled the assignments
to those tweets.
5.2 Worker Quality Prediction
Here, we compare the worker qualities predicted by the following three methods: the baseline method, the proposed method,
and the proposed method with limited features (proposed-LF).
The features of the baseline method are almost the same as thosedescribed in Rzeszotarski’s paper [7]. Moreover, those of the
proposed method with limited features, which we call proposedLF, used only the features of the baseline system. Thus, in the
proposed-LF, the prediction accuracy of the worker quality is thesame as in the baseline method, while the workload of the workers is the same as in the proposed method.
Furthermore, we discuss the eﬀectiveness of the behaviors obtained using the browsing button. We also analyze the propertiesof the correctly predicted workers and those of the workers who
were not correctly predicted.
5.2.1 Evaluation of Prediction Accuracy
We evaluated the classifiers by five-fold cross-validation. We
divided the workers into five groups; four groups were used forlearning data, and the remaining group was used for calculating
precisions and recalls. We repeated this process five times.
Table 5 shows the correlation between α and β, and Table 6

c 2019 Information Processing Society of Japan# processed HITs
132,594
108,836
241,430# tweets––
26,713

Table 5 Correlation between α and β.
α
0.05
0.1
0.15

Table 6
α
0.05

0.1

0.15

MethodBaseline
Proposed
Proposed-LF
Baseline
Proposed
Proposed-LF
Baseline
Proposed
Proposed-LF

MethodBaseline
Proposed
Baseline
Proposed
Baseline
Proposed

β
65.5%
62.8%
73.0%
72.6%
80.7%
78.7%

Clustering results.
Precision
0.17
0.21
0.16
0.21
0.23
0.20
0.22
0.32
0.27

Recall
0.55
0.75
0.58
0.72
0.84
0.64
0.62
0.79
0.71

F-value
0.26
0.33
0.25
0.32
0.36
0.30
0.33
0.45
0.39

shows the clustering result for each value of α. From this table,
we conclude that the accuracy of the proposed method is bettter than that of the baseline method and that of the proposed-LF
method.
We do not compare the value of α and β of the proposed
method and that of the baseline method. This is because theworkers of our proposed method and that of the baseline method
are not the same, we cannot compare the worker quality ofeach worker. Instead of comparing the baseline system with theproposed system, we compared the proposed method with theproposed-LF system, which simulates the baseline system withthe workers of the proposed method.
We do not set the value of α to a lower value than 0.05, because
the number of workers is too few. When the values of α are set
to 0.01, the number of low-quality workers is 2. If a classifier
can classify these workers as low quality, the precision ratio ofdetecting low-quality workers should be 100%, but we think that
this precision ratio is meaningless. We think that at least 10 workers should exist to obtain reliable results. Therefore, we assume
that the lower bound of α should be 0.05.
Also, we do not set the value of α to a higher value than 0.15.
If we set the value of α to 0.15, the workers whose correct answer
rates are 78% are considered as low-quality workers. Intuitively,
the workers who select appropriate options with 80% accuracy
should be considered as high-quality workers. Therefore, we did

55

Journal of Information Processing Vol.27 51–60 (Jan. 2019)not set the value of α or β to a higher value.
Here we consider the case where α is 0.1 as an example; thismeans that the workers with correct answer rates of lower than
10% are considered low quality, and the other workers are considered high quality. The correct answer rate β, which is the boundary between the low-quality workers and the high-quality workers, was 73.0% in the baseline method and 72.6% in the proposed
method.
The confusion matrix of the baseline method, proposed
method, and proposed-LF are shown in Table 7. It is more useful for requesters to identify low-quality workers accurately than
to identify high-quality workers, because they can then eliminate
or train low-quality workers who would otherwise degrade thequality of the work results. Therefore, the recall ratio of lowquality workers is important. The recall ratios of low-quality
workers were 0.72, 0.84, and 0.64 for the baseline, proposed, andproposed-LF methods, respectively. From these results, we discovered that the proposed method has the highest accuracy of thethree at predicting low-quality workers. However, it also predicted many high-quality workers to be low-quality. Therefore,
there is room for improving it.
5.2.2 Eﬀective Behaviors
The five-fold cross-validation calculated the importance degrees for each of five classifications. Figure 4 shows the top-5
Table 7(a) Baseline

Collect

L
H

Predict
L
H
18
7
69
156

important features in the proposed method. From these figures,
we discovered the features obtained using the browsing button(B6 , B7 , · · · , B11 ) were more important than those using the baseline method (B1 , B2 , · · · , B5 ).
From Fig. 4, we discovered that the minimum browsing time
per count (B9 -minimum) is selected as an important behavior at
every threshold α. Figure 5 shows the relationship between B9 minimum and the correct answer rate. Each point represents aworker. The workers with a small B9 -minimum have a high correct answer rate. Therefore, we can say that a worker has a highcorrect answer rate if B9 -minimum is high. However, the con-Fig. 5

B9 -minimum vs. correct answer rate.

Confusion matrix (L: low-quality, H: high-quality).
(b) Proposed

Collect

L
H

Predict
L
H
21
4
72
158

(c) Proposed-LF

Collect

L
H

Predict
L
H
16
9
65
165

Fig. 4 Top-5 important features.

c 2019 Information Processing Society of Japan

56

Journal of Information Processing Vol.27 51–60 (Jan. 2019)

Fig. 7

Fig. 6 Prediction results (α = 0.1).

verse is not necessarily true; we cannot say that a worker has a
low correct answer rate if his/her B9 -minimum is low. Therefore,
although the proposed method can detect low-quality workers, it
may also misclassify high-quality workers as low-quality ones.
This fact also appears in the confusion matrix of Table 7.
Also, from Fig. 4, we discovered that the features about thebrowsing count (B6 ) are important for classification. In theworker behavior records, many high-quality workers browsed thetweets many times, and some browsed even after selecting options.
5.2.3 Discussion of Worker Behaviors
We compared the proposed and the proposed-LF method todiscover which features are eﬀective for classification of theworkers. The workers in the proposed method and the proposedLF method are the same. Figure 6 shows classification results
when α = 0.1. Figure 6 (a) is the classification result of the
proposed-LF method, and Fig. 6 (b) is the classification result of
the proposed method. Each point represents a worker, and thehorizontal axis is the correct answer rate of the workers. In Fig. 6,
the workers represented in blue are predicted as low-quality, andthe workers represented in red are predicted as high-quality. In
an ideal classifier, blue circles are to the left of the boundary between low-quality and high-quality workers, and red circles areon the right side. Also, we discovered that high-quality workers
could classify workers with lower correct answer rates as betterclassifiers.
5.2.3.1 Case 1: Correctly Classified by Proposed but Misclassified by Proposed-LF
Here, we discuss the lowest-quality worker. This worker hada correct answer rate of only 28% and thus should be considered low-quality. This worker is shown at the left end in thetwo figures of Fig. 6. In Fig. 6 (a), this worker is represented inred; this means that the proposed-LP method incorrectly predicts
this worker as high-quality. On the other hand, in Fig. 6 (b), thisworker is represented in blue. The proposed method thus correctly predicts the worker as low-quality.
The average processing time of this worker was 6 seconds,
shorter than that of the average worker. However, as described in

c 2019 Information Processing Society of JapanCorrect answer rates of workers in baseline and proposed tasks.

Section 5.2.2, workers are not always low quality if their processing time is short. As a result, the proposed-LF method misclassifies this worker as a high-quality worker. However, the worker did
not browse more than 90% of the tweets, and the browsing time
was at most 2 seconds, much shorter than average. Incorporating
the browsing button into the task revealed this fact. Therefore, theproposed method correctly classified this worker as low-quality.
5.2.3.2 Case 2: Incorrectly Classified by Both Proposed andProposed-LF
Here, we discuss a low-quality worker who was correctlyclassified as low-quality by both proposed-LF and the proposed
method. The browsing count was 1.5 times more than average,
and the browsing time was two times longer than average. Although the correct answer rate of this worker is low, we feel that
this worker carefully read the tweets. We believe that this worker
did not have enough knowledge about Kyoto or sightseeing, butworked eagerly and attentively. When we use our method, thesystem will misclassify this kind of worker. To solve this problem, we should develop a method to predict worker quality using
not only worker behaviors but also work results and eﬀective features obtained by other methods.
5.3 Burden on Workers of Proposed Method
In the proposed method, workers must keep pressing the button in order to read a tweet. However, while the browsing button
is eﬀective for accurately predicting worker quality, it is not directly eﬀective in improving worker quality. It should actually
decrease the usability of work screens and increase the workloadof workers compared with the baseline system. In this section, we
discuss the eﬀect of the browsing button in the proposed methodfrom three points of view: task diﬃculty, processing time, andmotivation.
5.3.1 Task DiﬃcultyFirst, we determined whether the browsing button aﬀects
worker quality. We measured the diﬃculty level of the task fromthe diﬀerences and distributions of correct answer rates in thetasks of the proposed method and those of the baseline method.
We identified workers who were hired for both the baseline andproposed tasks. Then, we calculated their correct answer rates in
the proposed method and the baseline method. The results areshown in Fig. 7, where the horizontal axis is the correct answer
rate in the baseline method, the vertical axis is the correct answer
rate in the proposed method, and each point represents a worker.

57

Journal of Information Processing Vol.27 51–60 (Jan. 2019)

Fig. 8Distribution of correct answer rates of workers in baseline and proposed tasks.

The correlation coeﬃcient is 0.78; we thus know that there is astrong positive correlation between the correct answer rates forthe baseline and for the proposed method.
Next, we show the distribution of correct answer rates of workers in the two tasks in Fig. 8. The horizontal axis is the correct
answer rate, and the vertical axis is the number of workers withthe correct answer rate. We used the Wilcoxon signed-rank test to
determine any diﬀerence between the correct answer rates of the
proposed and baseline method, but no significant diﬀerence wasfound (p > 0.05). From this figure, we can conclude that there
is a possibility that the dificulties of the baseline and that of the
proposed method are the same.
There is a positive correlation between the correct answer rates
in the baseline method and the proposed method. Therefore, wecannot conclude that there is a diﬀerence in task diﬃculty, andthere is a diﬀerence between the correct answer rate of the baseline method and that of the proposed method. Therefore, we canconfirm that there is a possibility that the user interface does not
aﬀect worker quality.
5.3.2 Processing TimeIn the proposed method, workers have to push the browsing
button; they must move the mouse cursor to the browsing button
each time they read a tweet. Therefore, the processing time ofthe proposed method should be longer than that of the baseline
method.
Figure 9 shows a box plot of the distributions of processing
time for the baseline and proposed method. The vertical axis of
the graph is the processing time on a logarithmic scale. The median value is 6 seconds in the baseline method, and 9 seconds
in the proposed method. As we expected, the proposed method
increased the processing time by 3 seconds, which proved a timeconsuming burden on workers. Furthermore, it took more time
for all the tasks to be processed by the task requester as well.
Also, the processing times of some workers exceeded 1,000
seconds in both methods. The maximum number of characterson Twitter is currently 140, and it is hard to believe that a worker
could take so much time. As mentioned in Section 4, we cannot believe that workers are always concentrating on their tasks.
We imagine that these workers were processing HITs while doingother things or working on tasks across breaks.

c 2019 Information Processing Society of Japan

Fig. 9Processing time.

5.3.3 Motivation
For measuring worker motivation, we use the departing ratio of
HITs, which is the ratio of the number of early departed workers
to the number of all workers. It is diﬃcult to measure motivation
used in a psychological research field because the target workers
are an unspecified number of workers. In this experiment, we did
not measure the psychological motivation. On the other hand, in
this experiment, a worker can cancel the task freely at his/her convenience at any time. Therefore, assuming that the total numberof workers is large, it was assumed that motivation is high. We
defined the early departed workers as those who processed lessthan 100 HITs, because we only paid wages to those who processed more than 100 HITs. Table 4 shows the number of early
departed workers. From this table, we can see that in the baseline method, 43.1% of the workers (189) departed early, while in
the proposed method, 47.5% of the workers (231) departed early.
The proposed method increased the departing ratio by 4.4%.
Next, Fig. 10 shows the number of processed HITs versus thenumber of workers. The horizontal axis is the number of processed HITs, while the vertical axis is the number of workers.
Although the wages of both the baseline and the proposed system
are the same, the number of workers who process less than 20
tasks in the proposed method is larger than them in the baseline
method.
Figure 11 shows the distribution of task processing times of a58

Journal of Information Processing Vol.27 51–60 (Jan. 2019)

Fig. 10

Fig. 11

Distribution of number of workers who processed less than 100
HITs.

Distribution of number of workers who processed at least 100 HITs.

particular worker. It shows that there is no significant diﬀerencein the distribution of task processing times between the methods.
To summarize the above results, worker motivation decreased in
the proposed method when the workers processed less than 100
HITs. However, the motivation of workers who processed morethan 100 HITs was almost the same in the proposed and baseline
method.

6. Conclusion
We proposed a method of predicting worker quality fromworker behaviors in simple HITs by adding operations to theworker screens. While previous research analyzed worker behavior, we focused on obtaining more behaviors by installing abutton in the task. In particular, to obtain the browsing time moreaccurately, the proposed method forces the worker to press a button for displaying tweets.
We performed an empirical evaluation on detecting low-quality
workers in a yes-no type annotation task, which is a simple HIT.
The results confirm that compared with a baseline method withno button, our method improves the recall ratio and maintains theprecision ratio; this means our method can more accurately detect low-quality workers than the baseline method can. In particular, the recall rate was the highest, 0.84, when we defined a lowquality worker as one with a correct answer rate of the lower 10%.
In addition, we found that behaviors measurable with the button,such as the browsing time per visit and number of viewing times,
were more useful in classifying workers than those not obtainedwith the button. A worker with a short processing time could beeither high-quality or low-quality, but by looking at a breakdownof processing time into browsing time, etc., it becomes possible
to classify these workers. The disadvantage due to the installation of the buttons is that the processing time increases and thehurdles over which the worker has to go in their tasks go up.
Finally, we describe four issues as open problems. The first issue is misclassification of workers who diligently process HITs.
In order not to misjudge workers who can quickly perform appropriate work and those who do not understand the goal of thetask correctly, we should obtain more features not only on theirc 2019 Information Processing Society of Japanbehavior but also their work results.
The second issue is to verify the versatility of the proposed
method. In this paper, we conducted experiments on one type oftask. Although we did not consider translation tasks and article
writing tasks, we believe that it is possible to apply the proposed
method to tasks that give a simple evaluation by looking at certain
content, for example, image labeling and questionnaires. Also,
the task performed in this study was easy for many workers, witha correct answer rate of about 90%. It should also be verified fortasks in which the incorrect answer rate varies depending on theworker’s ability and tasks that normally have a low overall correct
answer rate.
The third issue is to consider adding operations to other tasks.
The proposed method uses the browsing button to obtain the features of worker behaviors. However, this method would not beappropriate for tasks other than annotation. In future, we willconsider a method to obtain more behaviors that puts less load on
workers.
The fourth issue is to use the prediction of worker quality forthe selected workers to earn better work results. To discover thisconsideration, further challenges such as how to select a worker
and how much the number of worker needs to be reduced are required.
Acknowledgments The research results have been achieved
by “Research and Development on Fundamental and Utilization
Technologies for Social Big Data,” the Commissioned Researchof National Institute of Information and Communications Technology (NICT), Japan.
References
[1]
[2]

[3]

[4]
[5]

[6]

[7]
[8]

[9]
[10]

Kittur, A., Chi, E.H. and Suh, B.: Crowdsourcing user studies with
Mechanical Turk, Proc. SIGCHI Conference on Human Factors in
Computing Systems, pp.453–456, ACM (2008).
Oleson, D., Sorokin, A., Laughlin, G., Hester, V., Le, J. and Biewald,
L.: Programmatic Gold: Targeted and Scalable Quality Assurance in
Crowdsourcing, Proc. 11th AAAI Conference on Human Computation, AAAIWS ’11-11, pp.43–48, AAAI Press (2011) (online), available from http://dl.acm.org/citation.cfm?id=2908698.2908706.
Vuurens, J., de Vries, A. and Eickhoﬀ, C.: How much spam can you
take? An analysis of crowdsourcing results to increase accuracy, Proc.
ACM SIGIR Workshop on Crowdsourcing for Information Retrieval(CIR ’11), pp.21–26 (2011).
Zhu, D. and Carterette, B.: An analysis of assessor behavior in crowdsourced preference judgments, SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pp.17–20 (2010).
Rzeszotarski, J.M. and Kittur, A.: Instrumenting the crowd: Using
implicit behavioral measures to predict task performance, Proc. 24th
Annual ACM Symposium on User Interface Software and Technology,
pp.13–22, ACM (2011).
Hirth, M., Scheuring, S., Hoßfeld, T., Schwartz, C. and Tran-Gia,
P.: Predicting result quality in crowdsourcing using application layer
monitoring, 2014 IEEE 5th International Conference on Communications and Electronics (ICCE), pp.510–515, IEEE (2014).
Kazai, G. and Zitouni, I.: Quality management in crowdsourcing using gold judges behavior, Proc. 9th ACM International Conference on
Web Search and Data Mining, pp.267–276, ACM (2016).
Kazai, G., Kamps, J., Koolen, M. and Milic-Frayling, N.: Crowdsourcing for book search evaluation: Impact of hit design on comparative system ranking, Proc. 34th International ACM SIGIR Conference
on Research and Development in Information Retrieval, pp.205–214,
ACM (2011).
Dawid, A.P. and Skene, A.M.: Maximum likelihood estimation of observer error-rates using the EM algorithm, Applied Statistics, pp.20–28(1979).
Raykar, V.C., Yu, S., Zhao, L.H., Jerebko, A., Florin, C., Valadez,
G.H., Bogoni, L. and Moy, L.: Supervised learning from multiple experts: Whom to trust when everyone lies a bit, Proc. 26th Annual59

Journal of Information Processing Vol.27 51–60 (Jan. 2019)

[11]
[12]

[13]

[14]
[15]
[16]
[17]
[18]
[19]
[20]
[21]
[22]

International Conference on Machine Learning, pp.889–896, ACM(2009).
Smyth, P., Fayyad, U.M., Burl, M.C., Perona, P. and Baldi, P.: Inferring ground truth from subjective labelling of venus images, Advancesin Neural Information Processing Systems, pp.1085–1092 (1995).
Sheng, V.S., Provost, F. and Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers, Proc.
14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.614–622, ACM (2008).
Snow, R., O’Connor, B., Jurafsky, D. and Ng, A.Y.: Cheap and fast—
but is it good?: Evaluating non-expert annotations for natural languagetasks, Proc. Conference on Empirical Methods in Natural Language
Processing, pp.254–263, Association for Computational Linguistics(2008).
Oyama, S., Baba, Y., Sakurai, Y. and Kashima, H.: Accurate Integration of Crowdsourced Labels Using Workers’ Self-reported Confidence Scores, IJCAI, pp.2554–2560 (2013).
Mok, R.K., Chang, R.K. and Li, W.: Detecting Low-Quality Workers in QoE Crowdtesting: A Worker Behavior-Based Approach, IEEE
Trans. Multimedia, Vol.19, No.3, pp.530–543 (2017).
Meyer, D.E., Abrams, R.A., Kornblum, S., Wright, C.E. and Smith,
J.K.: Optimality in human motor performance: Ideal control of rapid
aimed movements, Psychological Review, Vol.95, No.3, p.340 (1988).
Alonso, O.: Implementing crowdsourcing-based relevance experimentation: An industrial perspective, Information Retrieval, Vol.16,
No.2, pp.101–120 (2013).
Ling, W., Marujo, L., Dyer, C., Black, A. and Trancoso, I.: Crowdsourcing High-Quality Parallel Data Extraction from Twitter, ACL
2014, p.426 (2014).
Chawla, N.V., Bowyer, K.W., Hall, L.O. and Kegelmeyer, W.P.:
SMOTE: Synthetic minority over-sampling technique, Journal of Artificial Intelligence Research, Vol.16, pp.321–357 (2002).
Breiman, L.: Random forests, Machine Learning, Vol.45, No.1, pp.5–
32 (2001).
Friedman, J., Hastie, T. and Tibshirani, R.: The elements of statisticallearning, Vol.1, Springer (2001).
Horton, J.J. and Chilton, L.B.: The labor economics of paid crowdsourcing, Proc. 11th ACM Conference on Electronic Commerce,
pp.209–218, ACM (2010).

Satoshi Nakamura is a Professor of
Nara Institute of Science and Technology (NAIST), Japan, Project Leaderof Tourism Information Analytics Teamof RIKEN, Center for Advanced Intelligence Project AIP, Honorarprofessor of Karlsruhe Institute of Technology,
Germany, ATR Fellow, IEEE Fellow, IPSJ
Fellow. He received his Ph.D. from Kyoto University in 1992. He
was Associate Professor of NAIST in 1994–2000. He was Director of ATR Spoken Language Communication Research Laboratories in 2000–2008, and Director General of Keihanna Research Laboratories, National Institute of Information and Communications Technology, Japan in 2009–2010. He received the
IPSJ Yamashita Award, IPSJ Kiyasu Award and the Commendation for Science and Technology by the Minister of Education,
Science and Technology, and the Commendation for Science andTechnology by the Minister of Internal Aﬀair and Communications.

Yu Suzuki was born in 1977. He received his M.E. and Ph.D. degrees fromNara Institute of Science and Technology in 2001 and 2004, respectively.
He became an assistant professor at
Ritsumeikan University in 2004, a researcher at Kyoto University in 2009, andan assistant professor at Nagoya University in 2010. He is currently an associate professor at Nara Institute of Science and Technology. His current research interests
include Social Web analysis and data mining. He is a member of
IPSJ, IEICE, DBSJ, IEEE-CS, and ACM.

Yoshitaka Matsuda is a graduate student of Nara Institute of Science andTechnology. He received his M.E. degreefrom Nara Instutute of Science and Technology in 2018. His research interests are
crowdsourcing, human behavior, and machine learning. He is a member of DBSJ.

c 2019 Information Processing Society of Japan

60