What Has It Gotten Us? Examining Incentives 

Over Time in a Cross-sectional Study

Tracy Hunt-White, National Center for Education Statistics 

Jennifer Wine and Peter Siegel, RTI International 

 

January 10, 2012 

FCSM 

Washington, DC

This paper is intended to promote the exchange of ideas among researchers and policy 

makers. The views expressed in it are part of ongoing research and analysis and do 

not necessarily reflect the position of the U.S. Department of Education. 

1 
2 

Purpose of the Session 

• Background 

• Monetary Incentive Experiments 

–NPSAS:2000 

–NPSAS: 2004 

–NPSAS:2008 

–NPSAS:2012 

• Future directions 
3 

National Postsecondary Student Aid Study (NPSAS) 

• Purpose: Examine how students and their 

families pay for college. 

– Nationally representative sample of 

undergraduate and graduate students attending 

postsecondary institutions. 

Thus, we can... 

 Examine the characteristics of students 

enrolled in all levels of postsecondary 

education. 
4 

National Postsecondary Student Aid Study (NPSAS) 

• Legislatively mandated data collection to make 

sense of the federal government’s $100B per 

year investment in financing students’ 

postsecondary educations 

• Heavily used by policymakers, analysts, and 

researchers to understand programs like the Pell 

Grant and the Stafford Loan 
5 

NPSAS Sampling Design 

• Sampling occurs in two stages: 

Sample institutions - about 1/3 of all Title IV 

eligible postsecondary 

 Sample students within institutions, totaling 

about 138,000 students for NPSAS:08 

• Respondent data are weighted to represent 

national estimates 

- 21 million undergraduate and 3 million 

graduate students enrolled in about than 6,800 

institutions. 
6 

NPSAS Data Sources 

 Self-reported, web and telephone interview 

 Institutionally-reported data 

 Other administrative data sources 

-- Central Processing System (FAFSA) 

-- National Student Loan Data System 

 (Federal Loans, Pell Grants) 

-- National Student Clearinghouse 

-- SAT/ACT Data 

 -- Integrated Postsecondary Education Data System 

(IPEDS) 

 
7 

Unique Features of NPSAS 

• Large scale, repeated cross-sectional study, not 

longitudinal 

• Students sampled on a flow basis 

–Like multiple cohorts starting study at different 

times 

• Student interview just one source to define a 

respondent

• Abundance of administrative data sources 

• Increasingly self-administered 
8 

The Problem 

•Difficulty achieving response rates: 

Hard to reach students with all the screening 

devices and cell phones available 

Mobile students 

Unconventional schedules of students 

•Agency budget constraints 

•Tight schedules 
9 

Student Interview Response Rates – Full Scale 

0%

10%

20%

30%

40%

50%

60%

70%

80%

Student interview

response rate
10

The Solution 

A two-pronged approach: 

 Effective use of respondent incentives 

 Implement web-based data collection 

instrument 
11

NPSAS Field Test Incentive Experiments in a Nutshell
12

NPSAS:2000 Field Test Experiment 

Question: Will an incentive increase participation 

among certain nonresponse groups?

Target: Refusals, those with no telephone, and 

hard to reach 

Experiment: $0 vs. $20 ($5 prepaid cash with $15 

promised) to certain types of nonrespondents

Result: Overall, 55% of treatment group responded 

vs. 50% of the control group. Not significant. 

 
13

NPSAS:2000 Full-Scale Data Collection 

FS Incentive plan: 

Initial: $20 ($5 prepaid/$15 promised) to refusals 

and those with no valid telephone number 

Final: $20 promised to all nonrespondents

Result: 

Overall, student interview response rate was 70%. 

Of those offered incentives, 50% responded (about 

one-quarter of all student interviews). 
14

NPSAS:2004 Field Test Experiments 

Question 1: Will an incentive increase participation 

during the early response phase and promote a 

higher rate of online interview completion? 

Experiment: $0, $10 promised, $20 promised to all 

sample members during early response phase 

Results: Yes, incentive groups had higher response 

rates: 23% for treatment groups, 13% for control 

group. No difference between $10 and $20 groups. 

The treatment groups also responded online at a 

higher rate. 
15

NPSAS:2004 Field Test Experiments 

Question 2: Will an incentive increase participation 

during the nonresponse conversion phase? 

Experiment: $0 vs. $20 promised to certain types of 

nonrespondent (e.g., hard to reach, etc.)

Results: Yes, incentive increased participation: 33% of 

treatment group responded vs. 15% of the control 

group. 
16

NPSAS:2004 Full-Scale Data Collection 

FS Incentive plan: 

Early Response: $10 promised 

Nonresponse: 

 Initial: $20 promised to certain nonrespondents

 Final: $30 promised to all nonrespondents

Result: 

Overall, student interview response rate about 

63%, lowest ever in NPSAS history....Why? 
17

NPSAS:2008 Field Test Experiment 

Question: Is an incentive with a prepaid component 

more effective than a promised incentive among 

nonrespondents? 

Target: Refusals and hard to reach 

Experiment: $10 prepaid check with $20 promised 

vs. $30 promised 

Result: No difference; 34% response rate for each 

treatment group 
18

NPSAS:2008 Full-Scale Data Collection 

FS Incentive plan: 

 

Result: 

Early Response: $30 promised to all 

Nonresponse: 

Initial: $30 promised to certain types of nonrespondents 

Final: $30 promised to all nonrespondents 

Overall, student interview response rate about 72%: 34.5% 

of sample members responded during early response 

period, 7.5% during outbound call period with no incentive, 

29.7% during nonresponse conversion period 
19

Questions resulting from NPSAS:08 

• Is $30 the correct incentive level to yield the best 

results? 

• Is the outbound calling period with no incentive 

necessary. Only yielded 10% of the completed 

interviews. 
20

NPSAS:2012 Time For Something New 

• Should everyone get the same incentive amount 

regardless of their likelihood of responding? 

Or

 

• Should we use a person’s likelihood of response 

to target incentives – response propensity? 
21

NPSAS:2012 Time For Something New 

Why ask? 

• Possibly reduce nonresponse bias by targeting 

higher incentives at low propensity cases. 

• Possibly get higher propensity cases to respond 

at lower incentive levels. 
22

NPSAS:2012 Field Test Experiment 

Developed a propensity model based on 

NPSAS:04 data, applied model to NPSAS:12 field 

test sample members 

Experiment 

Treatment group Count Control Treatment 

High propensity 3,190 $30 $15 

Low propensity 1,400 $30 $45 
23

NPSAS:2012 Field Test Experiment 

Questions: 

1) Can the response propensity model distinguish 

between high and low response propensity cases? 

2) Will high propensity cases respond at same rate at a 

lower incentive amount? 

3) Will more low propensity cases response at a higher 

incentive amount? 

4) Will this model help reduce nonresponse bias? 
24

NPSAS:2012 Field Test Experiment 

Questions 2 and 3 results: High propensity treatment 

group had significantly lower response rate. Low propensity 

– no difference. 

Treatment group Response rate 

High propensity

Control ($30) 71.6% 

Treatment ($15) 64.6% * 

Low propensity

Control ($30) 57.2% 

Treatment ($45) 60.2% 

*p<0.0001 
25

NPSAS:2012 Field Test Experiment 

Other results 

Q1: Was the response propensity model able to 

distinguish between high and low propensity cases? 

Yes, did a good job of identifying cases. 

Q4: Did the response propensity model result in a 

reduction in nonresponse bias? 

No. Used low propensity cases to test for bias and since 

there was no significant difference in participation 

between the control and treatment groups, a reduction in 

bias would not be observed. 
26

NPSAS:2012 Full-Scale Plans 

• Offer $30 to all sample members throughout data 

collection 

• Don’t use response propensity – It did not reduce 

bias 

• Target nonmonetary interventions to students in 

certain types of institutions that historically have 

low response rates. How? 

• Make outbound calls immediately instead of 

allowing for a 3 week self-administration period 

• Start tracing activities earlier 
27

Future Directions – Some Thoughts 

• Increasing response rates do not mean reduced 

bias. 

• Can we identify a priori sample members that 

contribute most to bias? 

• What about using Responsive Design techniques to 

change strategies during data collection to target 

these folks and gain their participation? 

• Will try in our Baccalaureate and Beyond 

Longitudinal Study. 
28

Questions? 

 

 

Thank you! 

Tracy Hunt-White, Ph.D. 

NPSAS Project Officer 

National Center for Education Statistics 

Tracy.Hunt-White@ed.gov 

(202) 502-7438