1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Getting Started with Using Amazon 

Mechanical Turk to Label Data

Brandon Kopp

Office of Survey Methods Research @ BLS

Government Advances in Statistical Processing

23 September 2019
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Labeled Datasets

 One of the main chokepoints in machine learning is collecting 

high quality labeled data for training and testing

 Getting expertly trained coders in your organization may be too 

expensive and may be low priority if those coders have more 

pressing production concerns

 Crowdsourcing the data labeling task is one way to limit this 

expense if you can share the data that needs to be labeled and 

you can control the quality
Our Use Case

 Summary of Benefits and Coverage (SBC) are mandated by the 

Affordable Care Act (ACA) to be available for every healthcare plan

 The National Compensation Survey (NCS) sometimes receives these 

as part of data collection and they then need to be transcribed
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Our Use Case

 Copay

 Coinsurance

 After Deductible

 Per

 Not Covered

 No Charge

 Is it possible to automate the extraction of data used by the National 

Compensation Survey from the SBCs and code that information into tables?

 Sampled 3,000 SBC PDFs from publically available set on data.healthcare.gov* 

 Used mTurk to collect transcriptions of the data we were interested in

* PDFs downloaded from URLS in the 2017 & 2018 Plan Attribute Public Use Files
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Amazon Mechanical Turk (mTurk)

 A labor marketplace where those needing discrete, online tasks 

perforマed Iaミ fiミd workers aミd ruミ their task’s workflow

 Often used for work that is simple for humans but difficult for 

computers (image tagging, audio transcription, etc.)

 Other, similar services

• RapidWorkers • Figure-Eight

• microWorkers • CrowdSource

• ClickWorker
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

mTurk Terminology

 Requestor – The person requesting a service on mTurk

 Worker – The person performing a service on mTurk

 HIT (Human Intelligence Task) – A discrete task that a Requestor 

wants performed

Does this image contain a cat?

o Yes

o No

Labeling 1,000 Images = 1,000 HITs
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

mTurk Terminology

 Assignment – The number of Workers completing a HIT

 Reward – Compensation for the completion of a single HIT

 Batch – A grouping of HITs

 Qualification – Either custom or mTurk provided filter for 

Workers who can complete (or even see) your HIT
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Getting Started with 

Amazon Mechanical Turk
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Two Major Questions Before You Start

Are the data I want labeled releasable?

They will, by necessity, be viewed by people outside of your 

organization

Is my use of mTurk more like contracting a service or 

more like survey data collection?

Talk to your ageミIy’s OMB liaisoミ

Talk to your procurement folks
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Getting Used to mTurk

 Live Site

https://requester.mturk.com

https://worker.mturk.com 

 Sandbox Environment

The sandbox has the exact same functionality as the live site, but your 

aIIouミt isミ’t Iharged wheミ you さpostざ a HIT **He Iareful you’re oミ the right site**

https://requestersandbox.mturk.com

https://workersandbox.mturk.com 
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Project Setup

 Provide a title & description 

for Workers

 Specify number of 

assignments per HIT

 Specify how much you will pay 

per HIT

 Choose Worker qualifications

 They do not have a 

SurveyMonkey-style interface

 mTurk has templates to get 

you started, but you will 

almost certainly need to 

modify the one you select

 You have to know some HTML 

& CSS 

Choose Parameters of Project Design Your Collection Instrument
Input Data

HTML

Design Your Collection Instrument

How HIT 

variables are 

defined

Webpage Output Data
Frame with 

PDF

Data Collection 

Form

Collapsible 

Instructions

Popup 

Instructions
Publish Your Project

 You can then publish your 

project

 This is where you upload 

your input file

 You will see a printout of 

your costs.

 Your project will then be 

visible to Workers
X 10

X 20

Pilot Qualification

1 SBCs

9 Workers

Main Coding 3

rd Coder

1 HIT

9 Assignments

1 Batch

$2.00 Reward
X 10

X 20

Pilot Qualification

1 SBCs 3 SBCs

9 Workers 70 Workers

Main Coding 3

rd Coder

1 HIT

9 Assignments

1 Batch

1 HIT

70 Assignments

1 Batch

$2.00 Reward $2.00 Reward Applied Custom Qualification

(Workers > 90% Accuracy)

31 Workers
X 10

X 20

Pilot Qualification

1 SBCs 3 SBCs

9 Workers 70 Workers

Main Coding 3

rd Coder

3,000 HITs

2 Assignments per HIT

6 Batches

24 Workers

X 2

1 HIT

9 Assignments

1 Batch

1 HIT

70 Assignments

1 Batch

3,000 SBCs

$2.00 Reward $2.00 Reward $0.30 Reward
X 10

X 20

Pilot Qualification

1 SBCs 3 SBCs

9 Workers 70 Workers

Main Coding 3

rd Coder

3,000 HITs

2 Assignments per HIT

6 Batches

1,015 SBCs

24 Workers 18 Workers

X 2

Same HITs as Main Coding

Added 1 Assignment

1 Batch

1 HIT

9 Assignments

1 Batch

1 HIT

70 Assignments

1 Batch

3,000 SBCs

$2.00 Reward $2.00 Reward $0.30 Reward $0.30 Reward
X 10

X 20

Pilot Qualification

1 SBCs 3 SBCs

9 Workers 70 Workers

Main Coding 3

rd Coder

3,000 HITs

2 Assignments per HIT

6 Batches

1,015 SBCs

24 Workers 18 Workers

X 2

Same HITs as Main Coding

Added 1 Assignment

1 Batch

1 HIT

9 Assignments

1 Batch

1 HIT

70 Assignments

1 Batch

3,000 SBCs

$2.00 Reward $2.00 Reward $0.30 Reward $0.30 Reward

1d 4hr

$22

2d 6hr

$196

11d 1hr

$2,197

1d 3hr

$460
36 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

mTurk API

import boto3

client = boto3.client(

service_name= 'mturk',

region_name= 'us-east-1',

aws_access_key_id= [YOUR ACCESS KEY],

aws_secret_access_key= [YOUR SECRET KEY],

endpoint_url= [URL FOR API])

 Some tasks are not easy with 

the tools on mturk.com

 We used the Python version of 

the mTurk API to:

 Communicate with Workers

 Add assignments to HITs

 Extend expiration time for HIT

 Can test things out in Sandbox

 **Be careful, there are no 

guard rails**

 API Documentation

 Connecting to the mTurk API

 Assigning permission to use API
38 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Takeaways

 Start by talking to your OMB liaison and contracting folks

 You can get a lot of work done in a short amount of time

 You can control quality through qualifications, releasing in 

batches, and worker monitoring

 It helps to know HTML, CSS, and Python (or some other 

language that can access the API)

 The Sandbox environment helps alleviate some anxiety about 

getting started
Contact Information

39 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Brandon Kopp

Research Psychologist

OSMR/BLS

bls.gov/osmr

202-691-7514

kopp.brandon@bls.gov