1 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Electronically Mediated

Employment:

Designing new questions and 

assessing data quality

July 2019

Karen Kosanovich Kosanovich.Karen@bls.gov

Roxanna Edwards Edwards.Roxanna@bls.gov
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

 Background 

 Developing new questions: outreach & 

cognitive testing

 Problems with the questions

 Recoding the data

 Next steps

Outline
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Current Population Survey (CPS)

 Monthly household labor force survey

 Periodic contingent worker supplement (CWS)

February 1995, 1997, 1999, 2001, and 2005; with 

new update for May 2017

Data on contingent (temporary) work and 

alternative employment arrangements (main job)

 Capture emerging new work arrangement

Electronically Mediated Employment (EME)
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

New questions

 4 new questions added to May 2017 CWS

 Constraints:

Time & funding

 Developing new questions:

Outreach

Cognitive testing

Consulted outside experts

Incorporating feedback at each step
5 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

In-person electronically mediated 

employment: 2 questions

Q1. Some people find short, IN-PERSON tasks or jobs through 

companies that connect them directly with customers using a website 

or mobile app. These companies also coordinate payment for the 

service through the app or website.

For example, using your own car to drive people from one place to 

another, delivering something, or doing someone’s household tasks or 

errands.

Does this describe ANY work you did LAST WEEK?

Q1a. Was that for your main job, your second job or other additional 

work for pay?
6 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Online electronically mediated

employment: 2 questions

Q2. Some people select short, ONLINE tasks or projects through 

companies that maintain lists that are accessed through an app or a 

website. These tasks are done entirely online, and the companies 

coordinate payment for the work.

For example, data entry, translating text, web or software 

development, or graphic design.

Does this describe ANY work you did LAST WEEK?

Q2a. Was that for your main job, your second job or other additional 

work for pay?
7 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

New questions

 After developing new questions:

Training interviewers

Programming instrument

Testing instrument

Estimation

Editing
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Evaluating the data

 BLS performed extensive review of the data, 

including:

Monitoring taped interviews

Reviewing information available only on the 

confidential microdata file

 Determined the new questions did not work 

as intended 
9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Problem: false positives

 Most “yes” answers were clearly incorrect

Many people had jobs inconsistent with 

electronically mediated work

Many people said they had done their work in 

person and entirely online for the same job

 Some people said “yes” to these questions if 

they used a computer or mobile app in their 

job
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

False positives: examples

People said they had used a website or app to 

connect with customers and were paid through 

that website for the following jobs: 

Vice president of a major bank

Manager of a fast food restaurant

Local police officer

Surgeon at a large hospital 
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

False positives: examples

People said they had done electronically mediated 

work entirely online for the following jobs:

Medical assistant administering medication to patients

Hair stylist

Railroad engineer

Front desk clerk at a motel
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

What went wrong? 

 The questions may have been too complicated

Questions were long

Multiple concepts were included in a single 

question

Questions included examples that may have 

misled respondents
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Now what? 

 What can we do with the new data?

Figured out the extent of the problem

Manually recoded data using details available only 

on the confidential microdata file

– Teams of five, reading info and recoding, compared 

results at end

Evaluated new “recoded” estimates

– For transparency, BLS released both the recoded and 

as-collected data
14 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Impact of BLS recoding process

BLS does not recommend using the data as collected

Source: Current Population Survey (CPS), Bureau of Labor Statistics (BLS).

Electronically mediated workers

Recoded Collected

Total In 

person

Online Total In 

person

Online

Number of workers 

(in thousands)

1,609 990 701 5,057 3,021 2,969

Percent of total 

employed 1.0 0.6 0.5 3.3 2.0 1.9
15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Where are the data?

 Documentation and communication 

Data highlights

Excel tables

Frequently Asked Questions

Commissioner’s blog post

A paper describing the entire data recoding process, 

estimates, and lessons learned

 Transparency with stakeholders

www.bls.gov/cps/electronically-mediated-employment.htm
16 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Next steps

 Currently have no plans to collect data on 

electronically mediated work again 

 Questions would need to be revised 

 Working with the Committee on National 

Statistics (CNSTAT) for advise on what we 

should measure in the CWS if we field it again 
Contact Information

17 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Karen Kosanovich

Kosanovich.Karen@bls.gov

Roxanna Edwards

Edwards.Roxanna@bls.gov

www.bls.gov/cps

202-691-6378

CPSInfo@bls.gov