Adaptive Design Experimentation in 

the High School Longitudinal Study of 

2009 Second Follow-up Field Test: 

Investigating Incentive Treatments

FCSM - December 2015

Elise Christopher
Introduction to adaptive design

• “Alteration of sampling and collection approaches during 

the course of a data collection using real time process 

and survey data to improve survey cost efficiency and to 

achieve more precise and less biased estimates.”

• Largely driven by declining response rates, costs for 

improving them, and concern about related data 

precision and biases

• Adaptive design experiments are numerous in the field 

and tend to focus on response rates

2
Introduction to NCES adaptive design

• This presentation highlights current 

research at NCES with longitudinal 

studies that: 

– Focuses on improving response rates

• Using response propensity to more efficiently use 

resources

– Simultaneously considers bias reduction

3
Targeting based on likelihood to introduce 

bias and response propensity

High bias and less 

likely to respond

High bias and 

highly likely to 

respond

Low bias and less 

likely to respond 

Low bias and 

highly likely to 

respond 

Assumption: 

Bringing in more 

individuals that 

introduce higher 

bias may be 

more effective 

way of reducing 

bias than just 

increasing 

response rate

Response propensity

Bias on 

key 

estimates
NCES has used adaptive design in 

multiple studies

• Baccalaureate and Beyond (B&B) 

Longitudinal Study

• Education Longitudinal Study of 

2002 (ELS:2002) 

• High School Longitudinal Study of 

2009 (HSLS:09) 

• Beginning Postsecondary Student 

Study (BPS) 

• National Household Education 

Study (NHES)

• 2012

• 2012

• 2013

• 2014

• 2016 (planned)

5
Adaptive design in HSLS:09

• Information presented here is drawn from recent 

work on one NCES longitudinal study

– High School Longitudinal Study of 2009 (HSLS:09) 

Second Follow-up Field Test

• Study and related adaptive design research 

conducted with RTI International

6
Previous HSLS:09 adaptive design work

• HSLS:09 follows a representative sample of 9th

graders from the Fall of 2009

• The 2013 update gathered information during 

transition into postsecondary education or work

– Over 23,000 students currently in HSLS:09 sample

• 2013 Update: Bias was successfully 

reduced on key estimates

– As adaptive phases progressed, the respondent 

algebra 1 coursetaking rate more closely 

approximated known 2009 rate

7
HSLS:09 Second Follow-up 

Field Test (2015)

• 2013 Update experience tells us the bias 

likelihood model works

– Individuals likely to introduce bias were targeted

• Effectiveness of the design still depends on 

success of treatments used to encourage 

response

– Monetary and non-monetary incentives needed testing

8
Experiments tested treatments

• Used monetary incentives but not just about 

testing incentive amounts

• Effect of active treatments such as prepaying 

for incentives, the act of boosting incentive 

amounts, or monetary incentive vs. time

• Most effective treatments to be used in main 

study adaptive design (2016)

• Sample assigned randomly across treatment 

groups (N=1,100) 9
Experiments tested 4 treatments

• Treatments included in field test experiments:

1. Baseline incentive offer ($15 or no baseline 

incentive)

2. Timing of prepay 

• Early prepay (sent with data collection announcement letter)

• Late prepay (6 weeks into data collection)

3. Incentive boost

• $0, $15, or $30

4. Second boost ($25) or abbreviated interview

10
11

Field test phases and treatments

Phase Group A Group B Group C Group D

Phase 1 (4/13): 

Web only, 

$5 prepay for 

selected cases

No baseline 

incentive

offer; no 

prepay

$15 

incentive 

offer; no 

prepay

No baseline 

incentive 

offer; $5 

prepay

$15 

incentive 

offer; $5 

prepay

Phase 2 (5/4): 

Telephone

interviewing

added 

Telephone 

added

Telephone 

added

Telephone 

added

Telephone 

added

Phase 3 (5/26): 

$5 prepay for 

selected cases $5 prepay $5 prepay

(Prepay at 

baseline)

(Prepay at 

baseline)

Phase 4 (6/8): 

Increased 

incentive for 

selected cases

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

Phase 5 (7/6): 

Increased 

incentive or 

abbreviated

$25 boost 

or 

abbreviated 

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview
12

1. Baseline incentive: B and D received $15 offer

Phase Group A Group B Group C Group D

Phase 1 (4/13): 

Web only, 

$5 prepay for 

selected cases

No baseline 

incentive

offer; no 

prepay

$15 

incentive 

offer; no 

prepay

No baseline 

incentive 

offer; $5 

prepay

$15 

incentive 

offer; $5 

prepay

Phase 2 (5/4): 

Telephone

interviewing

added 

Telephone 

added

Telephone 

added

Telephone 

added

Telephone 

added

Phase 3 (5/26): 

$5 prepay for 

selected cases $5 prepay $5 prepay

(Prepay at 

baseline)

(Prepay at 

baseline)

Phase 4 (6/8): 

Increased 

incentive for 

selected cases

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

Phase 5 (7/6): 

Increased 

incentive or 

abbreviated

$25 boost 

or 

abbreviated 

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview
• Baseline incentive was 

significantly effective overall

– No baseline offer (AC) vs. $15 

baseline offer (BD): 

Chi-square = 6.72, p = 0.009

Experiment group

Final 

response 

rate

AC: No baseline offer 46.5

BD: $15 baseline offer 54.4

1. Baseline incentive : AC vs. BD

13
14

2. Timing of $5 prepay: at baseline for C and D

Phase Group A Group B Group C Group D

Phase 1 (4/13): 

Web only, 

$5 prepay for 

selected cases

No baseline 

incentive

offer; no 

prepay

$15 

incentive 

offer; no 

prepay

No baseline 

incentive 

offer; $5 

prepay

$15 

incentive 

offer; $5 

prepay

Phase 2 (5/4): 

Telephone

interviewing

added 

Telephone 

added

Telephone 

added

Telephone 

added

Telephone 

added

Phase 3 (5/26): 

$5 prepay for 

selected cases $5 prepay $5 prepay

(Prepay at 

baseline)

(Prepay at 

baseline)

Phase 4 (6/8): 

Increased 

incentive for 

selected cases

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

Phase 5 (7/6): 

Increased 

incentive or 

abbreviated

$25 boost 

or 

abbreviated 

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview
Experiment group

Final 

response

rate

AB: Late prepay 48.9

CD: Early prepay 52.0

• Prepay timing had no effect

– Late prepay (AB) vs. Baseline 

prepay (CD)

Chi-square = 1.05, p = 0.31

2. Timing of prepay: AB vs. CD

15
16

3. Incentive boost offer

Phase Group A Group B Group C Group D

Phase 1 (4/13): 

Web only, 

$5 prepay for 

selected cases

No baseline 

incentive

offer; no 

prepay

$15 

incentive 

offer; no 

prepay

No baseline 

incentive 

offer; $5 

prepay

$15 

incentive 

offer; $5 

prepay

Phase 2 (5/4): 

Telephone

interviewing

added 

Telephone 

added

Telephone 

added

Telephone 

added

Telephone 

added

Phase 3 (5/26): 

$5 prepay for 

selected cases $5 prepay $5 prepay

(Prepay at 

baseline)

(Prepay at 

baseline)

Phase 4 (6/8): 

Increased 

incentive for 

selected cases

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

Phase 5 (7/6): 

Increased 

incentive or 

abbreviated

$25 boost 

or 

abbreviated 

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview
Not quite significant difference between no 

boost and $30 boost: 

Chi-square = 2.67, p = 0.10

Significant effect of boost to no boost: 

Chi-square = 6.90, p = 0.009

No significant difference between $15 

and $30 conditions: 

Chi-square = 2.09, p = 0.15

Significant effect of $15 boost to no boost: 

Chi-square = 9.22, p = 0.002

Group

Within phase 

response 

rate

No boost 11.9

Any boost 19.5

$15 boost 22.0

$30 boost 17.0

No boost 11.9

$15 boost 22.0

No boost 11.9

$30 boost 17.0

3. Incentive boost: comparison overall and by amount

 Incentive boost
18

4. Abbreviated interview vs. second incentive boost

Phase Group A Group B Group C Group D

Phase 1 (4/13): 

Web only, 

$5 prepay for 

selected cases

No baseline 

incentive

offer; no 

prepay

$15 

incentive 

offer; no 

prepay

No baseline 

incentive 

offer; $5 

prepay

$15 

incentive 

offer; $5 

prepay

Phase 2 (5/4): 

Telephone

interviewing

added 

Telephone 

added

Telephone 

added

Telephone 

added

Telephone 

added

Phase 3 (5/26): 

$5 prepay for 

selected cases $5 prepay $5 prepay

(Prepay at 

baseline)

(Prepay at 

baseline)

Phase 4 (6/8): 

Increased 

incentive for 

selected cases

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

$0 or 

$15 or 

$30 boost

Phase 5 (7/6): 

Increased 

incentive or 

abbreviated

$25 boost 

or 

abbreviated 

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview

$25 boost 

or 

abbreviated

interview
Group

Within phase 

response 

rate

Abbreviated 10.4

$25 boost 17.9

• Abbreviated or $25 boost

– Significant effect of $25 boost 

over abbreviated

Chi-square = 7.37, p = 0.007

4. Abbreviated interview vs. second incentive boost offer

19
Summary of Experiment Results

1. Baseline incentive was significantly effective

2. Prepay timing had no effect

3. Incentive boost was significantly effective, 

though no difference between $15 and $30 

levels

– More testing on best amounts is recommended

4. Final incentive boost more effective than 

abbreviated interview

20
Plans for 2016 Main Study

• Start with baseline incentive for targeted cases

• Use up to 2 incentive boosts for targeted cases

• Target sample members using bias likelihood 

model, adding measures of response propensity 

to effectively use resources

– Not targeting cases of highest response propensity

21
22 22

Example*: Plot of bias likelihood by 

response propensity score

HSLS F2 would 

target cases in 

green area

*Example distribution 

from BPS:12/14
Questions?

• For additional information, please contact

– Elise Christopher (Project Officer of HSLS:09) –

elise.christopher@ed.gov

– http://nces.ed.gov/surveys/hsls09

• Studies discussed here were done in close 

cooperation with experts at RTI International

– Thanks to Dan Pratt, David Wilson, Jeffrey Rosen

23
Thank you!

24