1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Experimenting With Monetary Incentives

in the CE Survey

Barry Steinberg

Bureau of Labor Statistics

2019 CE Survey Methods Symposium

O7/16/2019

Special thanks to: Ian Elkin, Brett McBride, Yezzi Angi

Lee and Clayton Knappenberger

Any opinions expressed in this presentation are those of the authors and do not constitute 

policy of the Bureau of Labor Statistics.
2 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Outline

I. Background

II. Study Design

III. Analysis

IV. Conclusions

V. Next Steps
3 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

I. Background
4 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test - Background

Study objectives included developing a plan for: 

Operationalizing & implementing incentives

Researching & recommending incentive amounts

Proposing incentive distribution procedures
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

II. Study Design
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Interview Survey

 4 quarters, one interview every 3 months

 Personal visit (majority)

 Approximately 60-65 minutes per 

interview (median)

 6,000 – 6,500 usable interviews per 

quarter
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test - Overview

 Test Information and Conditions:

July 2016 – December 2016 

Experiment 

First Interviews (Wave 1 Only)

Treatments: 3 Groups

Control: 1 Group
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test – Treatment & Control Groups

Treatment and Control Groups

(1)

$5 “Token” Incentive 

(Unconditional)

(2)

$40 Survey Incentive 

(Conditional)

(3) 

$20 Records Use 

Incentive (Conditional)

All (n=1,350) $5 $40 $20

No Token (n=1,350) None $40 $20

No Record (n=1,350) $5 $40 None

Control (1st Interviews) None None None
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test – Operational Issues

 Many respondents did NOT receive debit card incentive which 

came with the Advance Letter

Incentive Receipt– Of Completed First 

Interviews

% Received 

All Incentives (766) 70.7%

No Token (747) 61.6%

No Record (781) 70.2% 
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test – Operational Issues

 Many respondents did NOT activate and use their debit card

Debit Card Activation Rate – Of Completed 

First Interviews

% Activated 

All Incentives (766) 59.9%

No Token (747) 52.4%

No Record (781) 55.4% 
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

III. Analysis
12 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

CE Data Quality Indicators

 Response rate

 Number of expenditures

 Number of contact attempts

 Mode – collection level

 Records usage

 Length of interview

 Doorstep Concerns

 Converted refusal
13 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Response Rate

1

st Interview 2nd Interview 3rd Interview

Treatment Response Rate 

(Treatment Minus Control)

Response Rate 

(Treatment Minus Control)

Response Rate 

(Treatment Minus Control)

All Incentives (1,2,3) 68.4% (+4.3%) 64.1% (+5.4%) 59.2% (+1.2%)

No Token (2,3) 64.7% (+0.6%) 62.7% (+4.0%) 60.4% (+2.4%)

No Record (1,2) 69.8% (+5.7%) 64.3% (+5.6%) 59.9% (+1.9%)

Control 64.1% 58.7% 58.0%
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Response Rates by Treatment Group
15 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Records Usage

Records Usage: Interview 1

By Incentive Group

% Using 

Records

Record-Use by Either 

Incentive Group (1,513 

CUs) 83 %

No Record-Use Incentive: 

Includes the Control 

(1,835 CUs) 60 %
16 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test Analysis – Length of Interview

Survey Time (All Sections Plus Front/Back)

Mean Survey Time

(Treatment minus Control)

Median Survey Time

(Treatment minus Control)

All Incentive (1,2,3) 84.3 minutes (+3.0) 78.5 minutes (+2.6)

No Token (2,3) 84.2 minutes (+2.9) 81.4 minutes (+5.5)

No Record (1,2) 84.5 minutes (+3.2) 76.8 minutes (+0.9)

Control 81.3 minutes 75.9 minutes
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

IV. Conclusions
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test - Conclusions

 Increase in response rates

 Increase in expenditures 

(Slight but not significant)

 Decrease in contact 

attempts (minimal)

 Increase in records use

 Decrease in converted 

refusals

 Increase in interview time
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Incentives Test – Supplemental Analysis

 Continue analysis of 

expenditure data focusing 

on imputed expenditures

 Determine impact on 

respondent burden

 Analyze cost effectiveness
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data to Analyze Burden (Wave 4)

 Incentives vs No Incentives

“How burdensome was this 

survey to you?” N=2,741

Not Burdened 1. Not at all 932

Burdened

2. A little 829

3. Somewhat 661

4. Very 205

5. Extremely 114

Incentive N=2,741

No Incentive Control group 850

Incentives

Up to $65 599

Up to $60 622

Up to $45 670

 Burdened vs Not Burdened
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Burden: Methodology

 First Method - Average Treatment Effect (ATE)*

ATE = E[Yi 1 − Yi 0 ]

 Second Method - Conditional Average Treatment (CATE)

CATE = E Yi 1 − Yi 0 Xi = x]

Treatment Group Burdened (%) Not Burdened (%) N

Incentives 63.6 36.4 1,891

No Incentives 71.3 27.7 850

Difference* -7.7

* Statistically significant with p < 0.05 
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Burden: Results

 Treatment Effect Estimate

Statistically significant: p < 0.05

 Respondents were less likely to report being burdened if incentives 

were offered

Treatment % difference SE

ATE 7.7% 0.019
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

V. Next Steps
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Future Incentive Programs

 Large Scale Feasibility (LSF) test for Diary Survey: Online Diaries

$5 unconditional cash incentive being sent with the advance letter to a 

random half of the sample (Priority Mail). 

 Consideration of an Unconditional Incentive and an Incentive for 

Record Usage: Interview Survey
Contact Information

25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Barry Steinberg

Mathematical Statistician

Division of the Consumer Expenditure Surveys

202-691-6885

Steinberg.Barry@bls.gov