The Effect of Large Monetary 

Incentives on Survey Completion: 

Evidence From a Randomized 

Experiment 

Joanne Hsu, Federal Reserve Board

Max Schmeiser, Amazon

Catherine Haggerty, NORC

Shannon Nelson, NORC

2016 CE Survey Methods Symposium

The analysis and conclusions set forth are those of the authors and do not indicate concurrence by other members of 

the research staff or the Board of Governors of the Federal Reserve System.
Objective

• Overall goal: determine optimal level of incentives for Survey of 

Consumer Finances 2016

• Would use of pre-paid incentives and/or larger post-paid incentives get more 

people to cooperate sooner?

• Would a quicker escalation strategy with escalated incentives get more 

people to participate sooner, thus shortening the field period?

• Experiment to analyze how response rates/interviewer effort are 

influenced by variation in:

• Pre-paid incentives ($5 vs none)

• Promised post-paid incentive levels ($50, $100, $150)

• Escalation of post-paid incentives

2
Context: Survey of Consumer Finances

• Triennial survey sponsored by Federal Reserve Board, data collected 

by NORC

• Focus on finances of American families

• Collects detailed information about assets, liabilities, income, employment 

and retirement benefits

• Administered by field interviewers either in-person or by telephone

• Dual frame sample: national area probability sample and list 

oversample of the wealthy

3
How do incentives influence completion?

• Main theoretic reasons for survey response: 

1. Altruistic (contribute to science/research),

2. Egoistic (such as monetary incentives), 

3. Reasons associated with survey itself (interest in the topic, trust in the 

organization behind the survey). 

• According to theories of economic and social exchange, incentives

• Appeal to norms of reciprocity

• Establish trust and legitimacy of organization

• Can complement efforts to appeal to non-egoistic motivation

• But do effects continue to increase when incentives are larger?

4
How do incentives influence completion? (2)

• Timing of incentives matter: prepaid typically works better than 

postpaid

• Amounts matter (but most studies use incentives much lower than 

the SCF)

• Effect on response of higher amounts may be non-linear? 

• Newer studies suggest non-linear response (Edwards 2005, Singer and Ye 

2013, Mercer et al 2015), but older studies did not (Singer et al 1999, Jobber 

et al 2004)

5
“Current” SCF incentive structure (2013)

• $50 base incentive

• Escalations in stages: $75 - $100 in week 16, $150 - $200 in week 28, 

$300 in week 42.

• Still, substantial difficulties securing interviews from certain 

subgroups (especially higher income areas)

• Question: do prepaid incentives or higher base incentives decrease 

overall cost of achieving target response rate?

6
Set-up of experiment 

• Miami, LA, NYC: 300 cases in each city

• Focus on census tracts with above average income (~80th percentile)

• Respondents approached for an “SCF 2014”

• Pre-notification letters in the mail

• Field Interviewer (FI) visits

• If R consents to interview, FI will ask a short interview on 

demographics/income/motivation for responding to interview

• Shortened field period (about one quarter)

7
Experimental groups

• Prepaid included with invitation letter: $5; no pre-paid

• Pre-paid group received a pre-invitation postcard

• Postpaid incentive listed on invitation: $50; $100; $150

• Phase 1: FI works case based on initial offer; phase ends when R makes soft 

refusal

• Phase 2: Half of cases eligible for escalated incentive, other half receives same 

initial incentive (will not discuss today)

• Interviewer does not know escalation eligibility until after the first soft refusal

8
PHASE 1 

FI attempts contact (808)

Live 

contact

made?

Z Out of scope (92) Initial mailing (900)

R says come back later/need 

more info

R soft refusal

no

yes

NORC sends more info

D: R hard refusal (49)

C: R accepts interview (236)

B: Phase 1 incomplete (321) 

(if no time)

AA: R accepts interview (22)

AB: R hard refusal (11)

R calls 

NORC 

(33)

9

PHASE 2 (not discussed today)

FI attempts contact (200)
Response rates

Phase 1 response rates, by incentive levels

0

0.1

0.2

0.3

0.4

50 100 150

no prepaid prepaid TOTAL

Overall results of phase 1

• 29% completed a survey

• 6% hard refusal

• 40% no live contact 

(phase 1 incomplete)

•  25% moved to phase 2 

(soft refusal 200 cases, 

but little time to 

complete)

10
LPM Regression results: Completion

Mean dependent variable 0.292

Prepaid (relative to no prepaid) 0.0611 **

(0.0289)

$100 (relative to $50) 0.0196

(0.0353)

$150 0.0697 **

(0.0352)

N 808

R2 0.202

• Prepaid and larger postpaid 

incentives both increased 

response

• No evidence of adverse effects 

at $150

• Effect of $150 more than triple the 

effect of $100 

• Similar results for probit, logits

11

* p < 0.1, ** p < 0.05, *** p < 0.01

Other controls: number of attempts made to contact the respondent, an indicator for respondents living in a locked 

building or gated community, indicators for FL and NY, the Census tract median income, and whether the respondent was 

ever reached by an interviewer
Interaction between prepaid and postpaid

Mean dependent variable 0.292

$100 alone 0.0494

(0.0499)

$150 alone 0.0491

(0.0498)

Prepaid + $50 0.0670

(0.0497)

Prepaid + $100 0.0569

(0.0498)

Prepaid + $150 0.1570 **

(0.0497)

N 808

R2 0.204

• Holding postpaid constant, 

prepaid still increases response

• Holding prepaid constant, large 

postpaid increases response

12
Examining intermediate outcomes

• Goal is not just to achieve completion, but to measure if surveys can 

be completed with less cost/interviewer effort

• R can pre-emptively call NORC and complete interview 

• Substantial savings of money and time

13
Regression results: R calls NORC to complete

Mean dependent variable 0.030

Prepaid (relative to no prepaid) 0.0313***

(0.0118)

$100 (relative to $50) 0.0068

(0.0145)

$150 0.0168

(0.0145)

N 808

R2 0.035

• Prepaid doubles the call-in rate

14

* p < 0.1, ** p < 0.05, *** p < 0.01

Other controls: number of attempts made to contact the respondent, an indicator for respondents living in a locked 

building or gated community, indicators for FL and NY, the Census tract median income, and whether the respondent was 

ever reached by an interviewer
Interaction between prepaid and postpaid: R 

calls NORC

Mean dependent variable 0.030

$100 alone 0.0077

(0.0204)

$150 alone 0.0095

(0.0204)

Prepaid + $50 0.0149

(0.0203)

Prepaid + $100 0.0351 *

(0.0204)

Prepaid + $150 0.0413 **

(0.0203)

N 808

R2 0.204

• Prepaid increases R calling to 

complete, particularly at higher 

levels of postpaid incentives

15
Other intermediate outcomes

• Analysis did not yield any measurable effect of larger incentives on:

• FI successfully making a “live contact” with R (sensitive to what we define as a 

“live contact”)

• No improvement in hard-to-reach areas, like locked/gated communities in Florida

• Conditional on live contact, R completing the survey

16
Conclusion

• Prepaid (plus postcard) provides a lot of “bang for the buck”

• No evidence of adverse effects at $150

• Potentially different effects of incentives for live contact, completion

• Larger incentives do not appear to improve contact with cases in locked/gated 

communities in Florida (implications for overall target response rates)

• Experiment targeted higher-income census tracts; effects likely a 

lower bound

• Average case takes approximately 16 hours, so higher incentives could 

be easily offset by even a small reduction in FI hours/case

17