1 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Understanding Data Collection Through 

a Survey of Field Staff

Brandon Kopp, Lucilla Tan, & Adam Safir

International Field Directors & Technologies Conference

May, 2016
“We can reduce measurement error if 

we just _______.”

...implement mobile 

web data collection

...ask respondents 

for financial records

...ask everyone in a 

household over age 15 

to complete a diary
3 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Introduction

 Over years and decades, these ideas make their way into our 

surveys

Almost always they are cognitively/qualitatively tested

Sometimes they are field tested

 Interviewers are on the front lines in implementing survey 

changes/improvements

They can tell us what works and what doesn’t

Their buy-in is important for successful implementation
4 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Feedback From Interviewers

 The Consumer Expenditure (CE) Survey at BLS periodically 

conducts surveys with field staff

Prior surveys were conducted in 2009 and 2011

The focus of today’s talk is a field staff survey conducted in 2015

 We ask for feedback on training, specific issues relevant to data 

collection

 We use this feedback to improve training and data collection 

processes
5 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Today’s Talk

 What insights we (or at least I) gained from the 2015 survey

Who are our interviewers?

What do new CE interviewers think of the training they received?

How burdensome do interviewers feel the survey is for respondents?

 How we disseminate our findings

 Conclusions
6 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

The Consumer Expenditure Survey

 Both surveys involve personal visits from field staff

 Field staff at the Census Bureau collect the data

 Interview Survey (CEQ)

In-person interview

Focus is on large, infrequent 

expenditures

Covers a 3-month reference period

Average length: 56 minutes

 Diary Survey (CED)

Two one-week, paper diaries

Collects all expenditures

Focus is on small, frequent 

purchases (e.g., groceries)
The 2015 CE 

Field Staff Survey
8 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Design

 The U.S. Census Bureau collected the data and monitored data collection 

through Google Forms

 An email request was sent to 1,169 field staff email addresses. This 

includes some duplication for staff that work on both the interview and 

the diary survey

 Staff were allowed to charge 1-hour of work time for completing the 

survey

 The data were screened for PII and delivered to BLS
9 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

The Questionnaire

 43 Questions

15 Open-ended 

28 Close-ended

 Topics

Demographics

Interviewer Experience

Training

Respondent Burden

Contact History Instrument

Neighborhood Observation Instrument

Redesign Topics
10 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

0

100

200

300

400

500

600

700

800

Cumulative Response

Data Collection and Response

Email Reminder Email Reminder n =716
11 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Who were our respondents?

180

117 113

78

102

126

0

40

80

120

160

200

Atlanta Chicago Denver Los Angeles New York Philadelphia

Regional Office

522

194

0

100

200

300

400

500

600

Interviewer Supervisor

Staff Position
12 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Overall Interviewer Experience

19%

12%

69%

0%

20%

40%

60%

80%

Less Than 2 Years 2 or 3 Years 4 or More Years

Years of Interviewing Experience

25%

32%

18%

26%

0%

10%

20%

30%

40%

None 1 or 2 3 or 4 5 or More

# of Other Surveys Currently Working

 Interviewers have a wealth of experience outside of CE
13 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Experience With CE Surveys?

13%

4%

73%

10%

0%

20%

40%

60%

80%

CEQ Only CED Only CEQ and CED Neither Survey

Experience With CE Surveys

< 2 Years, 

33%

< 2 Years, 

36%

2-3 Years, 

17%

2-3 Years, 

18%

4+ Years, 

49%

4+ Years, 

46%

0%

20%

40%

60%

80%

100%

CEQ

(n=616)

CED

(n=553)

Years of CE Experience

1 to 3, 32%

1 to 3, 71%

4 to 6, 27%

4 to 6, 

7 to 10, 24%

24%

11 or 7 to 10, 4%

More, 17%

11 or More, 2%

0%

20%

40%

60%

80%

100%

CEQ

(n=616)

CED

(n=553)

Number of Monthly Cases

 73% of Field Staff have 

experience with both the CEQ 

and CED

 Nearly half have 4 or more 

years of experience, 30% have 

less than 1 year

 There are more CEQ cases 

worked per month
Training
15 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Training

 Most interviewers are satisfied with the training they receive

8%

17%

71%

4%

0%

20%

40%

60%

80%

Poorly Neither Well

Nor Poorly

Well DK

How Well Did Training Prepare You?

58%

9%

14%

20%

0%

20%

40%

60%

80%

2012 or

Earlier

2013 2014 2015

Year Completed Initial Training

N = 298

13% 15%

66%

7% 7%

18%

75%

0%

6%

19%

71%

4%

0%

20%

40%

60%

80%

Poorly Neither Well

Nor Poorly

Well DK

How Well Did Training Prepare You?

2013

2014

2015
16 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Paired Practice Interviews

 How well did paired-practice interviews prepare you for CE 

respondents? (n=119)

8%

34%

59%

0%

20%

40%

60%

80%

Poorly Neither Well Nor

Poorly

Well

 “Practice interviews were 

‘perfect case’ scenarios not 

real world situations”

 “...you can get all the jitters 

out about how you are going 

to get the interview and 

what to say to the 

respondent”
17 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Pre-Classroom Materials

The current pre-classroom self-study for CE initial training contains some 

material in computer-based training modules (CBTs) and some material in 

paper chapters. Which do you prefer? (n=140)

46%

18%

36%

0%

20%

40%

60%

Prefer CBTs Prefer Paper

Chapters

No Opinion/

No Difference
Respondent Burden
19 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Burden

 The subjective feeling of discomfort or 

unease that respondents have during 

the interview

 We have directly asked survey 

respondents about burden and have 

seen mixed findings

 Interviewers hear directly from 

respondents, they see non-verbal 

behaviors, they witness varying levels 

of effort

30% 31%

28%

11%

0%

20%

40%

Not at All A Little Somewhat Very Extremely

How burdensome was this survey to you?
20 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Burden

 How burdensome do you feel the CEQ survey is for 

respondents? (n=624)

1%

11%

35%

25%

18%

0%

20%

40%

Not at All A Little Somewhat Very Extremely

 “For those that are willing to 

be interviewed.... they are 

very cooperative.”

 “TOO MANY INTERVIEWS.”

 “too time consuming and 

wordy. Ask too many of the 

same questions.”

30% 31%

28%

11%

0%

20%

40%

Not at All A Little Somewhat Very Extremely
21 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Disseminating the Findings

 Report

 Google Forms Summary

 Memo for Field Staff

 Internal Symposium

 Conference Presentations

 Internal Data Application
22 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Conclusions

 Regular communication between survey administrators and field 

staff can lead to higher data quality through improvement of 

procedures and through interviewer buy-in

 Field staff can serve as an early warning system for potential issues

 Open-ended questions are necessary to capture those issues you 

don’t know to ask about

 It is important to show field staff when/where their 

recommendations are incorporated into data collection
Contact Information

23 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Brandon Kopp

Research Psychologist

Office of Survey Methods Research

www.bls.gov/osmr

202-691-7514

kopp.brandon@bls.gov