Worker Qualifications
for Image-Aesthetic-Assessment Tasks
in Crowdsourcing
Yudai Kato
Marie Katsturai
Doshisha University

Keishi Tajima
Kyoto University

HCOMP2022 Work-in-ProgressBackground• Automatic assessment of images'
aesthetic quality has been actively studied

Large number ofimages with aestheticscore• CNN have yielded significant performance
improvements over conventional visualfeatures• Supervised learning approaches require alarge amount of data.
CNN to predict aesthetic score

We should consider how to efficiently collectaesthetic scores that are carefully assessed.

Background
Crowdsourcing
🙆Good Point• Fast• Inexpensive• Can use large data

🙅Bad Point• Spammer• Diverse skilled worker

We need a way to stabilize the quality of the results

Method used in prior studies• postprocessing method• Outlier detection• Answer aggregation(e.g., majority voting)• preprocessing method••

Qualification test
Gold injection

Problems when applied to subjective tasks• Large variation in answers from worker to worker.
• Difficult to prepare gold standard.

We need a way to stabilize the quality of the results

System of MTurks• Features• Can set order qualifications.
• If rejected, no compensation will be paid.

• Example of qualifications• approved rate
• The number of approved tasks• region
• age
• sex
• etc…

System of MTurks• Features• Can set order qualifications.
• If rejected, no compensation will be paid.

• Example of qualifications• approved rate
• The number of approved tasks• region
• age
• sex
• etc…

Task design

dataset1. Extract images from an existing dataset.
2. Order tasks under various qualification
conditions.

imageextract

3. Compare the gold standard from theexisting dataset and labels obtained in
MTurk

compare
MTurk

image

What conditions can produce results close
to the gold standard?

label

label
annotated by worker

AVA dataset

Qualification conditions• Aesthetic Visual analysis dataset• Five qualification conditions were set.

• 255,000 images with an average of 210
annotations• Expected: condition 4 achieves the bestresults.

Condition

Approval rate

The number ofapprovedtasks

1

over 95%

over 100

common inprevious studies

2

under 95%

None

Bad worker

3

None

under 100

New worker

4over 98%

over 5,000

Very strict5None

None

No qualifications

Remarks

ResultAverage
Conditio
ncorrelation
MAE

Variance
correlation

MAE

Approval rate

The number ofapproved tasks

Remarks

10.32

1.06

-0.06

2.38

over 95%

over 100

common in previous
studies

20.29

1.39

0.12

2.46

under 95%

None

Bad worker

30.25

1.18

0.03

3.29

None

under 100

New worker

40.43

0.81

-0.02

3.17

over 98%

over 5,000

Very strict

50.29

1.55

0.12

2.39

None

None

No qualifications• As expected, the average scores produced by workers that satisfied
Condition 4 were the closest to the AVA dataset.
• Condition 1 was a relatively severe restriction;
but interestingly, it did not lead to any significant differences from theother conditions, except for Condition 4

Result
Average
Conditio
ncorrelation
MAE

Variance
correlation

MAE

Approval rate

The number ofapproved tasks

Remarks

10.32

1.06

-0.06

2.38

over 95%

over 100

common in previous
studies

20.29

1.39

0.12

2.46

under 95%

None

Bad worker

30.25

1.18

0.03

3.29

None

under 100

New worker

40.43

0.81

-0.02

3.17

over 98%

over 5,000

Very strict50.29

1.55

0.12

2.39

None

None

No qualifications• As expected, the average scores produced by workers that satisfied
Condition 4 were the closest to the AVA dataset.

Result
Average
Conditio
ncorrelation
MAE

Variance
correlation

MAE

Approval rate

The number ofapproved tasks

Remarks

10.32

1.06

-0.06

2.38

over 95%

over 100

common in previousstudies

20.29

1.39

0.12

2.46

under 95%

None

Bad worker

30.25

1.18

0.03

3.29

None

under 100

New worker

40.43

0.81

-0.02

3.17

over 98%

over 5,000

Very strict

50.291.55

0.122.39

None

None

No qualifications• Condition 1 was a relatively severe restriction;
but interestingly, it did not lead to any significant differences from theother conditions, except for Condition 4

Conclusion• It is effective limiting eligibility to only those workers who had
been approved for thousand tasks and had a high approval rateof over 98%• Standard criterion, which was often used in related studies, wasinsufficient for the target subjective task