Machine Learning and the Commodity Flow Survey

Christian Moscardi

FCSM

11/4/2021

Any views expressed are those of the author(s) and not necessarily those of the U.S Census Bureau. 

2

Overview

Commodity Flow Survey (CFS)

Sponsored by U.S. DOT Bureau of Transportation Statistics

Conducted every 5 years (2017, 2022)

Many uses, including infrastructure funding decision-making

Respondents provide sampling of shipments from each quarter

3

Overview

4

ML model – details and pilot

Training data: 6.4M shipment records from 2017 CFS

Data provided by respondents, cleaned and edited by analysts

Model output: 5-digit product code (SCTG)

Model input: free-text shipment description, establishment’s NAICS (industry) Code

Model form: Bag-of-words logistic regression

Initial uses: Correcting and imputing 2017 CFS response data

Proved concept on a small scale

Eliminating need for respondents to provide product code benefits:

Improved data quality for codes themselves

Ability to ask respondents for significantly more shipment data

4

5

ML model – in production

Upcoming uses (2022 CFS)

Eliminating the need for respondents to provide SCTG product codes

Benefits

Reducing respondent burden -> increased response rates

Reducing Census cost to process data

Improving data quality

Eliminating need for respondents to provide product code benefits:

Improved data quality for codes themselves

Ability to ask respondents for significantly more shipment data

5

6

Improving Model -- Crowdsourcing

To improve the model, we need improved training data

More coverage of product types

More variety in description text

To improve training data, need humans to label more records

Solution: Crowdsourcing new data labels

Model

Human

Labeled Data

Unlabeled Data

Train model

Predict Labels

Send least confident predictions for labelling

Update training dataset

7

Amazon MTurk

Overview

“Crowdsourcing marketplace”

Used for surveys (incl. pre-testing), data labelling for AI/ML

Anyone can offer a task

Anyone can complete a task

Workers (aka Turkers) paid per task completed

Alternatives we considered

In-house labelling prohibitively expensive

For AI/ML labelling, alternatives exist (e.g. CrowdFlower) but not as flexible and larger-scale

8

Crowdsourcing Task Setup

Product description data sources

NAICS manual

Publicly available online product catalogs

~250,000 descriptions total

Task details

“Which product category best matches the given product description?”

List of categories comes from current best model (Top 10 shown in random order + “None of the above”)

U.S. workers only

Calculated reasonable pay rate

9

Crowdsourcing Quality Control

How do we ensure quality?

Gateway/gold standard task

Label 50 “gold standard” records

Worker must be at least 60% correct on min. 5 records

Production task: “Quadruple-key entry”

Only qualifying workers

4 workers label each record

Work in batches of 1,000; update model

Continuous validation

Keep track of inter-rater reliability

Inject “gold standard” records during production task

Remove poorly performing workers after each batch of production task

10

Step 1: Onboard workers with “gold standard” task
| Task Type     | Records per batch | Purpose                                                                                                                                                                                             |
|---------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Gold Standard | 50                | Recruit workers. Compare worker quality to gold standard. Workers who correctly label >=60% of time qualify to Production task.                                                                  |
| Production    | 1000              | Label records. Have 4 workers label per task. Records with >50% agreement, or 2-1-1 vote split considered labelled. 2-2 records or 1-1-1-1 records considered difficult, require further review. |

Upload Gold Standard data to MTurk

Evaluate workers for agreement to gold standard

Qualify high performers for Production 

Workers label gold standard product descriptions

Crowdsourcing  Worker Onboarding

11

Step 2: Label new data in “production task”
| Task Type     | Records per batch | Purpose                                                                                                                                                                                             |
|---------------|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Gold Standard | 50                | Recruit workers. Compare worker quality to gold standard. Workers who agree with gold standard >=60% of time qualify to Production task.                                                         |
| Production    | 1000              | Label records. Have 4 workers label per task. Records with >50% agreement, or 2-1-1 vote split considered reliable. 2-2 records or 1-1-1-1 records considered difficult, require further review. |

Baseline model, predict top 10 codes for all product descriptions

Upload Production batch type to MTurk; 4 workers label each desc.

Download results; incorporate results with majority agreement into baseline model; worker QC

Sample 1,000 descriptions where model is not highly confident

Crowdsourcing Production Task

12

Results

To date

Workers labelled ~5,000 records

~25,000 tasks completed

82% classified with high reliability by majority vote

Among reliable classifications, estimated correct classification rate of 95%

16% improvement to model’s classification ability on public product descriptions (~40,000 more records)

Direct cost: ~$3,000

Model’s classification power before/after MTurk labels

Note: all results in figure derived from public product description data

13

Zooming out – Costs of Implementation

14

Thank you!

Email: Christian.L.Moscardi@census.gov

15

Crowdsourcing Best Practices

Provide clear, simple instruction and fair pay to crowd-workers. 

Perform preliminary tests to improve instructions, pre-qualify and recruit skilled workers.

Use a gold standard to onboard workers and continuously monitor workers’ quality; contact or replace workers if necessary.

Use batches to continuously update workers’ qualifications based on gold standard and agreement metrics.

Use a baseline model if possible, to help limit the choices that workers need to make and simplify the task.

* Thanks to Magdalena Asborno and Sarah Hernandez at University of Arkansas for collaborating to develop best practices.

# image2.png

# image3.png

# image4.jpeg

# image5.png

# image6.png

# image7.png

# image1.png