Improving Multiclass Classification in
Crowdsourcing by Using Hierarchical Schemes
Xiaoni Duan, Keishi Tajima
School of Informatics, Kyoto University

In hierarchical classification, we can assign
different workers to different sub-tasks

Different workers may be good at
distinguishing different items

E.g.: Flat Classification of Canis Animals?

Worker A

AlaskanMalamute

Siberian
Husky

Siberian
Husky

AlaskanMalamute

GermanShepherd

SamoyedGray Wolf

CoyoteDhole

E.g.: Hierarchical classification scheme

Worker BGray Wolf

Coyote?

Hierarchical Scheme Selection :Dogs

1. Generate many candidate hierarchical schemes.
2. For each scheme, simulate the worker allocation process,and estimate the expected accuracy.
3. Choose the scheme with the best accuracy.

GermanShepherd

AlaskanMalamute

Wild Animals

SiberianHusky

SamoyedGray Wolf

CoyoteDhole

Worker Allocation Algorithm
Greedy algorithm focusing on variance of worker ability
Overview:
1. We publish a flat classification task as a qualification task.
2. We calculate accuracy of each worker for each subtask by using the ground truth.
3. We assign workers to subtasks by a greedy algorithm giving priority to workers whose accuracy largely changes depending on tasks.

Example of worker allocation:

Worker 3 has the largest variance.

Create three worker lists sorted by accuracy for subtask AB, A, and B.
WorkerAccuracy AB Jobs

WorkerAccuracy AJobs

WorkerAccuracy BJobs

Worker 1 1

60

Worker 2 120

Worker 3 0.98

80

Worker 2 0.98

20

Worker 1 0.95

60

Worker 2 0.95

20Worker 3 0.9580

Worker 3 0.90

80Worker 1 0.93

60

…

…

…

…

…

…

…

AB

…

B

A

…

Assign worker 3 to the subtask that he can do best – subtask B.
Remove task instances assigned to worker 3 from subtask B.
BAAB

Worker 3: task instances

Assign Worker 2, then Worker 1, in their priority order.
Calculate standard scores of workers’ accuracy in each subtask.
Sort workers by the variance of the standard scores in subtasks.
WorkerVarianceStandard Score AB

Standard Score A

Standard Score B

Worker 3
Worker 2
Worker 1

1.484540.34768
0.311-1.3140.146
1.119-1.2251.225
01.314-0.146-1.119Experiment 1 – Canis Animals• Data: 800+ photos of 7 categories• Qualification Task: a flat classificationwith 200 photos• Collect 6420 answers from 152 workers• Easily mistaken pair:• Alaskan Malamute & Husky

Scheme
A: Samoyed, Coyote, Wolf
B: others
A: Alaskan malamute, Coyote
B: others
A: wolf
B: others
Flat classification (majority vote)
Flat classification (EM)Total
Accuracy
0.875

ABA

Worker 1

Worker 2

Experiment 2 – Reptile and Amphibian Animals

B

Worker 3

Summary

Data: 1000+ photos of 10 categories of animals
Qualification Task: 200 photos & 307 workers
Generate hierarchical schemes with 3-6 sub-tasks
We double the number of workers in flat classificationin order to compare the accuracy at the same cost.

• Published 2 experiments on Amazon Mechanical Turk.
• Compare the accuracy of flat and hierarchical schemeswith majority voting and EM-based weighted voting.
• Conclusion: Hierarchical schemes improve the accuracy
if we choose an appropriate hierarchy by our algorithm.

0.9

0.9

0.85

0.85

•
•
•
•

0.8

0.885
0.86

0.855

0.89

0.863

0.845

0.86
0.83

0.82

0.8

0.590
0.75
0.765
0.833
0.767

Result of Experiment 1 on AMT

0.755

0.75
0.7

0.7Majority Voting by 3 (flat 6)
Flat Classification (Majority Voting)
Scheme 1

Scheme 202

Result of Experiment 2 on AMT

Majority Voting by 5 (flat 10)
Scheme 6422
Bottom Scheme