1 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Survey Redesign 

Recommendations from a Survey 

of Data Collection Field Staff 

Wallace Fraser, Tanisha Keller (Census Bureau)

Safia Abdirizak, Brandon Kopp, Adam Safir, Lucilla Tan (BLS)

Joint Statistical Meeting

July 31, 2016
2 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Introduction

 The CE Survey, like other large scale surveys, periodically 

conducts surveys with field staff

 Interviewers and supervisors are on the front lines in 

implementing survey changes/improvements

They can tell us what works and what doesn’t

Their buy-in is important for successful implementation

Their ideas and suggestions are considered as we improve the survey

– Followed by cognitive or field testing
3 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

2015 Field Staff Survey

 The focus of today’s talk is the 2015 field staff survey – including 

both interviewers and supervisors

 Prior surveys were conducted in 2009 and 2011, but with only 

interviewers and senior interviewers

 The 2015 survey included questions on training, respondent 

burden, and the Gemini Project to redesign the survey

 CE plans to use this feedback to improve training procedures, 

data collection processes, and the survey redesign protocol
4 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Presentation Outline

1. 2015 Field Staff Survey Overview

2. Gemini Project Overview

3. 2015 Field Staff Survey Results

4. Recommendations
5 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

1. 2015 Field Staff Survey 

Overview
6 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Protocol 

 The U.S. Census Bureau collected the data and monitored data collection 

through Google Forms

 An email request was sent to 1,169 field staff email addresses

 Staff were allotted 1-hour of work time for completing the survey

 The data were screened for PII before delivery to BLS
7 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Questionnaire

 43 Questions

15 Open-ended 

28 Close-ended

 7 Topics

Demographics

Interviewer Experience

Training

Respondent Burden

Contact History Instrument

Neighborhood Observation Instrument

Survey Redesign 
8 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

0

100

200

300

400

500

600

700

800

Cumulative Response

Data Collection and Response

Email Reminder Email Reminder n =716
9 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

2. Gemini Project Overview
10 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Gemini Project Overview

 In 2013, CE proposed a new design for the separate Interview Survey and 

Diary Survey, consisting of:

1. One survey (in place of what is now two surveys, Interview and Diary)

2. Online and paper diaries (in place of only paper)

3. Two collection periods, one year apart (in place of 4 quarterly waves)

4. Each collection period would include:

a. A first interview collecting some recalled expenses, plus diary placement, a diary for each 

household member over the age of 15

b. One week of diary keeping 

c. A second interview collecting expenses with records and diary pickup

d. Incentives to complete the survey and adhere to protocols (e.g., record usage)
11 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Gemini Project Related Questions

 The field staff survey questionnaire included questions about the Gemini 

Project to identify potential concerns and suggestions on how to improve 

the CE survey from a field staff perspective 

 Examples:

 Records interview – Feasibility and sensitivity

 Incentives – Helpfulness of offering a token gift in gaining respondent cooperation

 Spending report summary – Helpfulness of a personalized pamphlet summarizing household 

expenditures in gaining respondent cooperation 

 Technology – Respondent use of financial records via a computer, smartphone, or other electronic 

device during the interview 
12 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

3. 2015 Field Staff Survey 

Results
13 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Gemini Project: Record Use

 Impact of financial record use on interview difficulty, accuracy, and length

The majority of interviewers reported that using records made the interview 

process easier and more accurate but also more time consuming.

66%

34%

0%

20%

40%

60%

80%

100%

Easier More difficult

2%

98%

0%

20%

40%

60%

80%

100%

Less accurate More accurate

14%

86%

0%

20%

40%

60%

80%

100%

Less time

consuming

More time

consuming
14 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Gemini Project: Incentives

 How helpful would token incentives be in gaining respondent 

cooperation? 

20%

24% 25%

15% 16%

0%

20%

40%

Not at All A Little Somewhat Very Extremely

 “I would not offer a [token 

incentive, like a] pen, or 

magnet, they would laugh at 

that.”

 “Offer cash incentives and 

people would do [the 

survey]!”
15 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Gemini Project: Spending Report Summary

 How helpful would an informational pamphlet outlining a respondent’s reported 

response, compared to national averages, be in gaining respondent cooperation? 

16% 15%

30%

22%

16%

0%

20%

40%

Not at All A Little Somewhat Very Extremely
16 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Gemini Project: Concerns about the Design

17% 18%

23%

20%

23%

0%

20%

40%

Don't Know/Not

Available

No Concerns Minor Concerns Some Concerns Major Concerns
17 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Gemini Project: Participation in Field Tests

 Breakdown of feedback based on whether interviewer 

participated in redesign testing field tests

5%

21%

19%

27%

15%

19%

29%

19%

15%

22%

19%

15% 16%

24%

32%

0%

20%

40%

Yes No Don't know/Not available

Don't know/ Not available

No Concerns

Minor Concerns

Some Concerns

Major Concerns
18 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Gemini Project: Selected Verbatim Comments

 Interviewer feedback for why they had minor or no concerns: 

 "I am part of the test doing that POC survey and it is MUCH easier to gain participation"

 "I think combining the interview and diary into one survey may help interviewers see the point of the 

information they give.” 

 Interviewer feedback for why they had concerns with proposed plan: 

 "Picking up one diary is often difficult, picking up several in one household will greatly increase the time 

involved."

 “It makes for hard feelings when a family member doesn't complete the diary and then doesn't get the debit 

card like on the [Proof of Concept Test] POC."

 Interviewer suggestions for improvement: 

 “Three major factors influence participation:

– respondent selection - most people are skeptical of how they were selected to participate

– time to complete the survey - the diary takes the least time and has a better success rate for [FR] 

– information collected - find a way to explain in greater detail and reassure interviewers that data collected 

is not used to create a 'personal profile' of the respondent.” 
19 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

4. Recommendations
20 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Recommendations: General

 Review association between number of surveys worked and CE response 

rates, to gauge potential impact on CE sample performance measures

 Numerous suggestions for improving training, particularly in providing 

more time for:

Material covered 

Answering questions and reviewing situations

Topics related to converting reluctant respondents

Training in the field (as opposed to the classroom)
21 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Recommendations: Redesign

 Interviewer handling of respondent records

Interviewers reported a preference for remaining with respondents when 

collecting information from their financial records

But in general are not comfortable handling the records themselves

Follow up about reluctance to handle records, and alternative approaches 

 Charter a team to review verbatim responses (out of scope for this team)

Actionable items related to any of the survey topics

Specific concerns about the redesign, and recommendations for responding to 

those concerns, either in terms of redesign protocol changes, or as topics to focus 

on in interviewer training 
22 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Thank You