1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Monetary Incentives in the Consumer 

Expenditure Interview Survey: An 

Assessment of Data Quality

Ian Elkin, Barry Steinberg, and Brett McBride

Bureau of Labor Statistics

2017 Joint Statistical Meetings

O8/01/2017

Any opinions expressed in this presentation are those of the authors and do 

not constitute policy of the Bureau of Labor Statistics.
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Outline

I. Background

II. Study Design

III. Analysis

IV. Conclusions

V. Next Steps
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

I. Background
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Gemini Redesign Project

 “... is the multi-year redesign 

project for the purpose of 

researching, developing, and 

implementing an improved 

survey design to improve data 

quality through a verifiable 

reduction in measurement 

error.” 
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test - Background

 Study objectives included developing a plan for -

 Operationalizing & implementing incentives

 Researching & recommending incentive amounts

 Proposing incentive distribution procedures, including procedures to capture respondents that 

generally do not respond to classic incentives

 Analyze test data to make a recommendation regarding incentive implementation 

 Past CE incentives research -

 Gemini Incentive Structure Review: Summary of Incentive Experiences 

 CE incentives operational summaries

 CE Interview Incentives Test Report

 CE Diary Incentives Test Report 
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

II. Study Design
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test - Overview

 Test Information and Conditions:

 July 2016 – March 2017 

Experiment

 First Interviews

 Test Sample Size: 1,350 per 

treatment group

 Control Sample Size: 1,950

 Conditional and unconditional 

incentives
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test – Treatment & Control Groups

Treatment and Control Groups

$5 Token 

Incentive 

(unconditional)

$40 Survey 

Incentive 

(conditional)

$20 Records Use 

Incentive 

(conditional)

All $5 $40 $20

No Token None $40 $20

No Record $5 $40 None

Control (1st Interviews) None None None
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

III. Analysis
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Data Quality Indicators

 Response rate

 Total expenditures reported

 Number of expenditures

 Mode – collection level

 Records usage

 Number of contact attempts

 Doorstep concerns

 Converted refusal

 Length of interview
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Response Rate

Response Rates, Second Interview

Difference (Treatment Minus 

Control)

All 5.0 % points

No Token 3.2 % points

No Record 5.1 % points

Response Rates, First Interview

Difference (Treatment Minus 

Control)

All 4.6 % points

No Token 1.4 % points

No Record 5.0 % points
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – First Interview Total 

Expenditures Reported

Number of Expenditures, First Interview

Difference in Medians

(Control group baseline)

Difference in Means

(Control group baseline)

All -$124.00 -$97.32

No

Token -$67.00 +$1,305.29

No

Record +$86.00 +$48.24

No significant differences.
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Second Interview 

Total Expenditures Reported

Number of Expenditures, Second Interview

Difference in Medians

(Control group baseline)

Difference in Means

(Control group baseline)

All +$85.00 -$426.60

No

Token +$446.50 -$833.66

No

Record1 +$295.00 +$229.51

1. Indicates significant difference from Control Wilcoxon-Mann-Whitney; Z= 2.11, Pr>Z 0.035. 
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Number of First 

Interview Expenditures

Number of Expenditures, First Interview

Difference in Medians

(Control group baseline)

Difference in Means

(Control group baseline)

All +1.0 +1.1

No

Token +1.0 +1.2

No

Record +1.0 +1.0

No significant differences.
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Number of Second 

Interview Expenditures

Number of Expenditures, Second Interview

Difference in Medians

(Control group baseline)

Difference in Means

(Control group baseline)

All +1.0 +1.6

No

Token1 +1.5 +2.2

No

Record2 +1.5 +2.4

1. Indicates significant difference from Control Wilcoxon-Mann-Whitney; Z= 2.36, Pr>Z 0.018. 

2. Indicates significant difference from Control Wilcoxon-Mann-Whitney; Z= 2.66, Pr>Z 0.008. 
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Mode of Collection

Difference in Mode of Interview (Treatment Minus Control)

Telephone In-Person Mix

All -5.4 % points 4.5 % points 0.9 % points

No Token -4.4 % points 4.2 % points 0.2 % points

No Record -1.8 % points 1.7 % points 1.1 % points
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Records Usage

Records Usage

Difference (Treatment Minus 

Control)

All 29.6 % points

No Token 29.3 % points

No Record 14.3 % points
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – First Interview Contact 

Attempts

Median and Mean Number of Contact Attempts, First

Interview

Difference in Medians

(Control group baseline)

Difference in Means

(Control group baseline)

All 0.0 -0.2

No Token 0.0 -0.4

No Record 0.0 -0.3

No significant differences.
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Second Interview Contact 

Attempts

Median and Mean Number of Contact Attempts, Second

Interview

Difference in Medians

(Control group baseline)

Difference in Means

(Control group baseline)

All 0.0 0.2

No Token 0.0 -0.3

No Record 0.0 -0.3

No significant differences.
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Doorstep Concerns: 

All Responders

Doorstep Concerns (Treatment Minus Control)

Not 

Interested/Hostile Time Privacy Other

All -2.1 % points -5.9 % points -5.5 % points 0.6 % points

No Token 1.8 % points -5.2 % points -0.1 % points 1.0 % points

No Record -4.2 % points -3.0 % points -2.9 % points -1.2 % points
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Doorstep Concerns: NonResponders

Doorstep Concerns (Treatment Minus Control)

Not 

Interested/Hostile Time Privacy Other

All 1.1 % points -6.9 % points -1.2 % points -0.8 % points

No Token 5.0 % points -6.6 % points 2.5 % points 0.6 % points

No Record 2.2 % points -0.6 % points 1.2 % points -2.6 % points
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Converted Refusal

Converted Refusals

Difference (Treatment Minus Control)

All -3.3 % points

No Token -0.9 % points

No Record -3.4 % points
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Length of Interview

Total Survey Time 

Difference in 

Medians

(Control group baseline)

Difference in 

Means

(Control group baseline)

All1 2.6 minutes 2.9 minutes

No

Token2 5.5 minutes 3.0 minutes

No

Record 0.9 minutes 3.2 minutes

1. Indicates significant difference from Control Wilcoxon-Mann-Whitney; Z= 1.97, Pr>Z 0.049. 

2. Indicates significant difference from Control Wilcoxon-Mann-Whitney; Z= 2.12, Pr>Z 0.027. 
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

IV. Conclusions
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test - Conclusions

 Increase in response rates

 Increase in expenditures

 Decrease in contact 

attempts

 Increase in records use

 Decrease in converted 

refusals

 Increase in interview time

 What does this all mean?
26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

V. Next Steps
27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test – Next Steps

 Continue analysis of 

expenditure data focusing 

on imputed expenditures

 Determine impact on 

respondent burden

 Analyze cost effectiveness
28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test – Next Steps

 Are incentives right for CE?

 What amount is the right 

amount?

 Are token incentives cost 

effective?

 What is the most impactful 

way to distribute incentives?
Contact Information

29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Ian J. Elkin

Senior Economist

Division of the Consumer Expenditure Surveys

www.bls.gov/cex

202-691-6865

Elkin.Ian@bls.gov