1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Rapid Implementation of Test Design

Using Python

David H. Oh

Economist

Office of Compensation and Working Conditions

GASP Workshop

September 17, 2019
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Background

 Machine learning (ML) prototype 

Developed in early 2019

Used Occupational Requirements Survey (ORS) and other

supplemental datasets to train the model

Predicts the top-five most likely Standard Occupational Classification 

(SOC) codes
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Pilot Test

 Computer-Assisted Review (CAR) Pilot

A small team formed in May, 2019

Test the feasibility of implementing the ML algorithm into the 

production cycle, specifically in the review process

Three weeks of testing in the actual ORS production environment

– From late July to early August

May June July August September

Team 

kickoff

Begin 

pilot test

End

pilot test
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Questions to Answer

 The effects of CAR on SOC code review

Its effects on the time spent reviewing the SOC code

Its effects on the number of questions being sent out

Its effects on the review resulting in a positive change

 The effects of CAR on reviewer bias

Do e┝posures to ML algorithマ’s outputs result iミ re┗ie┘ers fa┗oriミg ふor 

not favoring) specific SOC codes?
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Test Design

 Randomized, controlled crossover trial

Eight participants from the microdata review staff

Each participant expected to review approximately 150 incoming data

Every incoming data reviewed by a participant gets randomly assigned 

to a control/treatment group 
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Test Design

Incoming Data

ふさQuoteざぶ Control Group

Treatment Group

Treatment Group

#1

Treatment Group

#2

Treatment Group

#3

Treatment Group

#4

-No prompts 

-Match or no match

-Ordered top-5 list

-Probabilities

-Match or no match

-Ordered top-5 list

-Match or no match

-Random top-5 list

-Match or no match
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Challenges and Constraints

 Some information are readily captured by the existing 

production/review system

SOC codes

Questions sent

 Other information are not available

Time spent on reviewing a SOC code

Re┗ie┘er’s e┝pected “OC code
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Challenges and Constraints

 Random assignment

Test instrument must be able to perform random assignments

 Resource constraint

Minimal disturbance on the actual production

 Time constraint

A little over a month to develop and implement the test instrument

May June July August September

Team 

kickoff

Begin 

pilot test

End

pilot test

Develop and implement 

test instrument

Design 

pilot test
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Benefits of Using Python

 Same language as the one used to develop the ML algorithm

Can easily import the ML algorithm that has been Pickle-ed

 Can create a standalone application that is easily distributable

 Can take user inputs

 Can access database

 Can output varying prompts based on random assignments

 Can write to a centralized dataset
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Test Instrument

Retrieve information 

from the database

Show varying 

prompts based on 

random assignment
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Test Instrument – Treatment #1 Prompt
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Test Instrument – Treatment #2 Prompt
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Test Instrument – Control Prompt
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Test Instrument – Follow-up Questions

 Whether the participant suspects the entered SOC code to be 

incorrect

If yes, a follow-up question on what the correct SOC code would be

 Participaミt’s faマiliarit┞ ┘ith the eミtered “OC code

On a scale from 1 to 5

 Duration (in seconds) collected in the background

From the time the random assignment was made to the time 

participant moved on to the next quote
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Lessons for Future Iterations

 Create a more robust centralized database structure for

collecting information

Few instances of application crashing on the users due to multiple 

users writing to the central dataset at the same time

 Develop a web-based application to improve user experience
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Conclusion

 Pilot test ended in early August with almost 1,500 quotes 

reviewed using the test instrument

 The flexibility of Python language and its various applicability 

enabled the rapid implementation of a randomized, controlled 

crossover trial

 Results from the pilot test?

Currently being analyzed
Contact Information

17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

David H. Oh

Economist

Office of Compensation and Working Conditions

www.bls.gov/ect

202-691-5985

oh.david@bls.gov