.N"'C.S'£S National Center for Science and Engineering Statistics 

An investment in Goodwill or Encouraging 

Delays? Examining the Effects of 

Incentives in a Longitudinal Study

FCSM

January 2012

Karen Grigorian 

NORC at the University of Chicago

Lynn Milan

NCSES, National Science Foundation
 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

Disclaimer

This paper reports the general results of research 

undertaken by staff at the NORC at the University of 

Chicago and at the National Center for Science and

Engineering Statistics at the National Science

Foundation (NSF). The views expressed are attributable 

to the authors and do not necessarily reflect those of 

NORC or those of the sponsors: the NSF and the 

National Institutes of Health.

Do not quote without permission from the authors. 1
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

Acknowledgements

Stephen Cohen, NSF – for support and guidance

Daniel Foley, NSF – 2008 SDR Program Officer

Nirmala Kannankutty, NSF – for developing the 2008 

SDR incentive offer method

Eric Hedberg, NORC – for assistance in quality 

assurance, statistical testing, and analytical review

Do not quote without permission from the authors. 2
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

Background

• OMB standards for federal statistical surveys 

require high response rates

• One strategy to increase response rates is to offer

an incentive

• Project funds are limited

• Longitudinal surveys require a long-term view

Do not quote without permission from the authors. 3
e .N"'C.S'£S National Center for Science and Engineering Statistics 

Key Question 

What impact do incentives offered in one survey

round have on subsequent rounds?

1. Negative – incentives in a previous round cause

delayed response in the subsequent round and/or no 

response without an incentive

2. Neutral – no impact on the final response rate of 

subsequent rounds 

3. Positive – the response rate in subsequent rounds 

increases, regardless of whether incentives are offered 

again

Do not quote without permission from the authors. 4
 

 

 

 

 

 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

Analysis Overview

To answer the key question, the analysis compares

groups of cases that were and were not offered an 

incentive in a prior round on the following dimensions:

• Timely vs. Slow Response – where timely is defined as 

providing a final response before a late-stage incentive 

is offered

• Survey Response – as measured by the percent 

completing the survey

• Effort and Quality

– Level of Effort – defined as the number of contacts made by

mail, phone, or email

– Data Quality – measured by imputation score and verbatim 

response length

5 – Incentive Cost – average cost of the incentive checks cashed
e .N"'C.S'£S National Center for Science and Engineering Statistics 

Survey of Doctorate Recipients (SDR)

• Sponsors: NSF and NIH

• Design: Longitudinal survey

• Target population: U.S. residents under age 76 with 

U.S.-granted doctoral degrees in science, engineering, 

and health fields

• Periodicity: Conducted every 2 years

• Question topics: Demographic, education, and career 

history information

Do not quote without permission from the authors. 6
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

SDR Sample Design

• SDR is a panel survey

• Sample size is approximately 40,000

• Each survey cycle

– Some panel members become permanently 

ineligible

– Some panel members are randomly cut

– New doctorate earners are added to ensure 

population coverage

Do not quote without permission from the authors. 7
e .N"'C.S'£S National Center for Science and Engineering Statistics 

SDR Data Collection Protocol

• Multi-mode: mail, telephone, and Web

• All start modes follow a parallel contacting protocol

• Eventually all nonrespondents become eligible for the 

late-stage contacting protocol 

• In the 2003, 2006, and 2008 SDRs, the late-stage 

protocol included monetary incentives

Do not quote without permission from the authors. 8
 

 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Analysis Opportunity

To evaluate the effect of...

– Past incentives (2003/2006) only

– Incentives both in the past and in 2008

– No incentives ever

...on 2008 SDR behavior, including...

– Time to respond (timely vs. slow)

– Response rate

– Data quality

– Incentive check cashing 

Do not quote without permission from the authors. 9
e .N"'C.S'£S National Center for Science and Engineering Statistics 

2003 SDR Incentive Experiment 

• Late-stage incentive offered after 7 months of data 

collection

• Limited size random controlled experiment

– $30 pre-paid incentive (n=323) 

– $50 post-paid incentive (n=329) 

– Informational brochure 

– Control group

• Results: $30 pre-paid incentive was most effective 

with regard to cooperation, data quality, and cost

Do not quote without permission from the authors. 10
 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2006 SDR Incentive Experiment 

• Late-stage incentive random controlled experiment 

offered in two stages

– First stage: after 5.5 months of data collection 

(n=4,581)

– Second stage: after 6.5 months of data collection 

(n=2,390)

• $25 pre-paid incentive mailing

• Results

– Earlier was more cost-effective for new cohort cases

– Later yielded a higher response rate for past 

refusers Do not quote without permission from the authors. 11
 

 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Late-stage Incentive

• 11,163 nonrespondents after 5.5 months of data 

collection

• $30 pre-paid incentive; not an experiment

– Offered to all in low-responding strata (≤ 72.1% 

response rate)

– Offered to at least 20% in higher responding 

strata

• 7,499 selected for an incentive (4,717 not in locating) 

Do not quote without permission from the authors. 12
 

 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Analysis Sample

• Included in the 2003, 2006, and 2008 SDRs

• Not a locating problem and not part of an early 

incentive experiment in 2008

• 21,610 sample members

– No prior incentive (n=19,224)

o Not late-stage eligible in 2003 or 2006

(n=18,055)

o Late-stage eligible in 2003 or 2006 (n=1,169)

– Prior incentive in 2003, 2006, or both (n=2,386)

Do not quote without permission from the authors. 13
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Analysis Sample

Slow in the past (2003 and/or 2006) n = 3,555

No past incentive n = 1,169 Incentives in the past

n = 2,386

Timely in the past (2003 and 2006)

Never offered incentives in past n = 18,055

All analysis cases n = 21,610 83.5%

32.9%

16.5%

67.1%

Do not quote without permission from the authors. 14
e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results - Overall

Analysis group Sample

Complete 

surveys

Percent 

completing 

survey

Overall 21,610 20,488 94.8

Timely response 19,239 18,837 97.9

Slow response 2,371 1,651 69.6

No incentive 956 559 58.5

Incentive 1,415 1,092 77.2

Do not quote without permission from the authors. 15
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Analysis – Time to Respond 

Do not quote without permission from the authors. 16
 

e 

Timely in t he past (2003 and 2006~ 

N@l.i@r offer@d lncet"ltlY ~ In past 

n = 18,055 

.N"'C.S'£S National Center for Science and Engineering Statistics 

All a nalysis. caises 

n = 21~610 

Slow in the past (1003 and/o.- 1006) 

n = 3,555 

No past inc:en iYe 

n = 1,169 

I nc:entl~ 1n the 

past 

n = 2, 86 

2008 SDR Analysis Sample

17
 

 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Time to Respond

• Focus – “Slow” in the Past (2003/2006)

Analysis group Sample

Timely in 

'08 Slow in '08

Percent 

slow in '08

Overall 3,555 2,338 1,217 34.2

No $ in past 1,169 830 339 29.0

$ in past*** 2,386 1,508 878 36.8

• Results – Those offered $ in past were 43 percent more 

likely to be slow to respond in 2008, significant at p<0.001

Do not quote without permission from the authors. 18
 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

Key Question: Time to Respond 

What impact do incentives offered in one survey

round have on subsequent rounds?

1. Negative – incentives in a previous round cause

delayed response in the subsequent round and/or no 

response without an incentive

2. Neutral – no impact on the final response rate of 

subsequent rounds 

3. Positive – the response rate in subsequent rounds 

increases, regardless of whether incentives are offered 

again

Do not quote without permission from the authors. 19
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Analysis – Survey Response

Do not quote without permission from the authors. 20
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

51 in the 03 d/or 006) 

BO. ~ n 

t 

n= 

S .1 com 1ng 

2008 SDR Analysis Sample – Slow in Past 

21
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

Slow in the past (2003 and/o.- 1006) 

n : 3,555 

.-----~·2.9':i"i,-----~-------ci• 1.1':!t · .------, 

No past incentive 

n = 1,16-9 

Timely In 2.008~ 

No incentive in 1008 

n = 830 

Slowln2008 

n = ll9 

56.6% completing 

B9.5'6 ieomplet1ng 

I nc:~ntivie~ In theP s.t 

Timely in lO~; 

No incentive in 2008 

n -= 1,508 

n = 2,386 

Slcw inl008 

n : 878 

59. 1'6 ieomplet1ng 

94.0% com pl ng 

2008 SDR Analysis Sample – Slow in Past 

22
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Survey Response

• Focus – “Slow” in the Past (2003/2006); “Timely” in 2008

Analysis group Timely in '08

Complete 

surveys

Percent 

completing 

survey

Overall 2,338 2,160 92.4

No $ in past 830 743 89.5

$ in past 1,508 1,417 94.0

• Results – Those offered $ in past and responding in a 

timely way in 2008 were more likely to respond with a

complete survey (versus a refusal) in 2008, significant at 

p<0.01 Do not quote without permission from the authors. 23
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

Slow in the past (2003 and/or 2006) 

n = 3,555 

~---:32:,9·%.·.~-----..__-------h'.:7.1"%.'~--~ 

No past incentive 

n = 1,169 

71.0.,.,_--~--

Timely in 1008; 

No incentive in 2008 

n : 830 

89.5% comple ng 

n = 139 

36.cm completing 

lncet'ltlve In 2008 

n = 200 

71.0% completing 

1ncentl1Je~ In the 

p,a~t 

Timely in 1008; 

No incentive in 2008 

n = 1,500 

94.0% comple ng 

n = 2,386 

n = 327 

44.3% comple~ing 

n = 551 

67 .9% c::ompleting 

2008 SDR Analysis Sample – Slow in Past 

24
 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Survey Response

• Focus – “Slow” in the Past (2003/2006); “Slow” in 2008

Analysis group Slow in '08

Complete 

surveys

Percent 

completing 

survey

Overall 1217 711 58.4

No $ in '08 466 195 41.8

$ in '08 751 516 68.7

No $ in past 339 192 56.6

No $ in '08 139 50 36.0

$ in '08 200 142 71.0

$ in past 878 519 59.1

No $ in '08 327 145 44.3

$ in '08 551 374 67.9

• Results – Seems to be a “good will” effect for “slow” cases

offered past incentive only, but it is not significant 25
 

 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

Key Question: Survey Response

What impact do incentives offered in one survey

round have on subsequent rounds?

1. Negative – incentives in a previous round cause

delayed response in the subsequent round and/or no 

response without an incentive

2. Neutral – no impact on the final response rate of 

subsequent rounds 

3. Positive – the response rate in subsequent rounds 

increases, regardless of whether incentives are 

offered again 

Do not quote without permission from the authors. 26
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Analysis – Effort and Quality

Do not quote without permission from the authors. 27
 

 

 

 

 

~[lill~sl~ Grill.I~ I 

Timely in 2008; 

No incentive in 1008 

n = 16,901 

98.7% completing 

AJI analysis '-l~s 

n = 21,611) 

94.8% completing 

~------,.;3.5,,,. ~-----~------------~_.6,S"''·•----------~ 

i 

Timely In the past (2003 and 2006) 

Never offered lncentlYes In past 

n = 18,055 

97,6,£ rompleting 

Slow In the past (2003 or 2006) 

n = 3,555 

80,8% completing 

~-----~2.9,,.,,..=------~i~-----<l ~.i,7.lw,,,,__ __ ~ 

No pa~ lncentlYe 

n = 1,169 

80,0~ rompleting 

I ncenthle!. ,n the 

past 

n = l,386 

81.1% complet ,ng 

Slow In 2008 Analys~ Group H Slow In 2008 Analysis Group c Slowln2008 

n = 1,154 Timely In 2008; n= 339 Timely In 2008; n=87S 

No incentive in 2008 No incentive in 1008 

81.5% complellng n = 830 56.6% rompleting n = 1,508 59.1% completing 

89.5% oompletlng 94.03⁄4 compl~me 

Anal~sis GrouE! G Analysis Group E An,1tysls Group F Analysl~ Group D Analysis Group B Analysl~ Group A 

No incentive in 2008 lncentiYe in 2008 No Incentive In 2008 Incentive In 2008 No Incentive In 2008 lncentl11e In 2008 

n - 490 n = 664 n = 139 n = lOO n = 327 n = 551 

74.3% oompletlng 86.7% oompletlng 36.0lb completing 71.0lb complellng 44.3% complet ing 67.9% completing 

Color Key:

Survey Round Time to Respond

Orange = Past SDR (2003/2006) Pink = Timely

Purple = 2008 SDR Blue = Slow

Incentive Treatment

Yellow = No incentive

Green = Incentive 

28

 

e N"'C.SC:S National Center for Science and Engineering Statistics 

2008 SDR Analysis Sample
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Analysis Sample

Response

Incentive 

offered? Response

Incentive 

offered?

Timely NA Timely NA I 16,901

Timely NA Slow No G 490

Timely NA Slow Yes E 664

Slow No Timely NA H 830

Slow No Slow No F 139

Slow No Slow Yes D 200

Slow Yes Timely NA C 1,508

Slow Yes Slow No B 327

Slow Yes Slow Yes A 551

Past 2008

Analysis 

group Cases

NA = Not applicable; sample members who responded in a timely way did not have a chance to receive an incentive offer. Do not quote without permission from the authors. 29
e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Effort and Quality

Response

Incentive 

offered? Response

Incentive 

offered?

Timely NA Timely NA I

Past 2008

Analysis 

group

Results – The majority 

of the analysis sample 

(78.2%) is highly 

cooperative

Metric I

Percent of Analysis Sample 78.2

Percent Completing the Survey 98.7

Level of Effort 4.5

Data Quality

Imputation Score 0.5

Occupation Verbatim Length 103.4

Do not quote without permission from the authors. 30
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Effort and Quality

Response

Incentive 

offered? Response

Incentive 

offered?

Timely NA Slow No G

Timely NA Slow Yes E

Past 2008

Analysis 

group

Results – Group E 

trends in the desired

direction for all 

metrics, but at an 

average incentive cost

of $19 per complete 

survey

Metric G E

Percent of Analysis Sample 2.3 3.1

Percent Completing the Survey 74.3 86.7

Level of Effort 25.5 24.3

Data Quality

Imputation Score 3.8 3.5

Occupation Verbatim Length 77.0 89.0

Average Incentive $ per Survey NA $19

31
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Effort and Quality

Response

Incentive 

offered? Response

Incentive 

offered?

Slow No Slow No F

Slow No Slow Yes D

Past 2008

Analysis 

group

Results – Group D 

nearly doubles % 

completing the survey 

while improving data 

quality at an average 

incentive cost of $21 

per complete survey

Metric F D

Percent of Analysis Sample 0.6 0.9

Percent Completing the Survey 36.0 71.0

Level of Effort 23.4 23.4

Data Quality

Imputation Score 5.4 4.0

Occupation Verbatim Length 75.5 92.3

Average Incentive $ per Survey NA $21

32
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Effort and Quality

Response

Incentive 

offered? Response

Incentive 

offered?

Slow No Timely NA H

Slow Yes Timely NA C

Past 2008

Analysis 

group

Results – Group C 

shows a greater 

likelihood to respond 

with a complete 

survey; but all other 

differences are

nominal in 2008

Metric H C

Percent of Analysis Sample 3.8 7.0

Percent Completing the Survey 89.5 94.0

Level of Effort 6.3 7.0

Data Quality

Imputation Score 0.7 1.0

Occupation Verbatim Length 87.6 92.1

Do not quote without permission from the authors. 33
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Effort and Quality

Response

Incentive 

offered? Response

Incentive 

offered?

Slow No Slow No F

Slow Yes Slow No B

Past 2008

Analysis 

group

Results – Group B 

shows a greater 

likelihood to respond

with a complete 

survey

Metric F B

Percent of Analysis Sample 0.6 1.5

Percent Completing the Survey 36.0 44.3

Level of Effort 23.4 24.0

Data Quality

Imputation Score 5.4 5.9

Occupation Verbatim Length 75.5 88.4

Do not quote without permission from the authors. 34
 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Effort and Quality

Response

Incentive 

offered? Response

Incentive 

offered?

Slow Yes Slow No B

Slow Yes Slow Yes A

Past 2008

Analysis 

group

Metric B A

Percent of Analysis Sample 1.5 2.5

Percent Completing the Survey 44.3 67.9

Level of Effort 24.0 21.6

Data Quality

Imputation Score 5.9 3.5

Occupation Verbatim Length 88.4 87.6

Average Incentive $ per Survey NA $22

Results – Repeatedly 

offering an incentive to 

the slow sample shows

a greater likelihood to 

respond with a 

complete survey of 

higher data quality at 

an average incentive 

cost of $22 per survey

35
 

 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

2008 SDR Results – Effort and Quality

Response

Incentive 

offered? Response

Incentive 

offered?

Slow No Slow Yes D

Slow Yes Slow Yes A

Past 2008

Analysis 

group

Metric D A

Percent of Analysis Sample 0.9 2.5

Percent Completing the Survey 71.0 67.9

Level of Effort 23.4 21.6

Data Quality

Imputation Score 4.0 3.5

Occupation Verbatim Length 92.3 87.6

Average Incentive $ per Survey $21 $22

Results – For the

consistently slow 

sample, repeat offers

of the incentive appear

to have the same 

effect as an initial

incentive offer 

36
 

 

 

e .N"'C.S'£S National Center for Science and Engineering Statistics 

Answers to Key Question

What impact do incentives offered in one survey round

have on subsequent rounds?

1. Negative – incentives in a previous round cause 

delayed response in the subsequent round

2. Neutral – for consistently slow cases the percent of 

cases completing the survey is the same for those

offered the incentive repeatedly or for the first time

3. Positive – some “good will” effect of a past incentive

can be seen with a higher percent of cases completing 

the current survey when no incentive is offered; some 

data quality metrics improve for some incentivized

groups regardless of whether incentives are offered 

again Do not quote without permission from the authors. 37
e .N"'C.S'£S National Center for Science and Engineering Statistics 

2010 SDR Incentive Decision

• After 5 months of data collection, overall response rate 

was 65.4%

• Three options for continuing the data collection

1. Offer an incentive to attempt to achieve an 80%

response rate and finish data collection on time

2. Do not offer an incentive, finish on time with a 

response rate less than 80%, and conduct a 

nonresponse bias study

3. Do not offer an incentive and continue data 

collection until an 80% response rate is reached,

potentially delaying delivery of the resulting data

Do not quote without permission from the authors. 38
e .N"'C.S'£S National Center for Science and Engineering Statistics 

2010 SDR Incentive Design

• Late-stage incentive plan implemented similar to that 

used for the 2008 SDR

• $30 pre-paid incentive offered after 6 months of data 

collection; not an experiment

– Offered to all in low-responding strata

– Offered to at least 20% in higher responding strata

Do not quote without permission from the authors. 39
e .N"'C.S'£S National Center for Science and Engineering Statistics 

N&RC 

at the UNIVERSITY of CHICAGO 

Please direct questions and comments to:

Karen Grigorian, SDR Project Director 

Grigorian-Karen @ norc.org

Lynn Milan, SDR Program Officer

Thank you! 40

LMilan @ nsf.gov