Calling all Early Birds: 

Testing a Deadline-Limited Incentive in a Sequential Mixed-Mode Survey

October 25, 2023

Federal Committee on Statistical Methods, College Park, MD 

Michelle McNamara

Study Director, National 

Household Education Surveys 

Program

National Center for Education 

Statistics

This presentation is intended to promote ideas. The views expressed are part of ongoing 

research and do not necessarily reflect the position of the U.S. Department of Education.

Rebecca Medway

Principal Researcher, American 

Institutes for Research

Danielle Battle

Principal Researcher, American 

Institutes for Research

Ai Rene Ong

Researcher, American Institutes 

for Research
National Center for 

Education Statistics

2

Overview

• National Household Education Survey 

• Background

• Methods

• Early Bird experiment 

• Results 

• Conclusions and Next Steps
National Center for 

Education Statistics

3

NHES:2023 - Background

• Household-based, cross-sectional, 

nationally representative study of children

• Collects information from parents about 

children’s early care and education 

experiences and family involvement in 

school

• Data collection every 3-4 years

• Data are collected by Census on behalf of 

NCES
National Center for 

Education Statistics

• Recruitment by mail

• Multimodal collection (web, paper, phone)

• Multiphase design

4

NHES:2023 - Methods Screen

er

Sa

mpl

ing

Topica

l 

Survey

Screener

National ABS 

Sampling Topical 

Survey

Within 

Household

Sampling
National Center for 

Education Statistics

Why use a timed incentive for NHES?

• Increase response rates

• Increase response rates through web survey

• Design of NHES lends itself well to an early bird incentive

6

Early Bird Experiment - Inspiration

Early Screener Mailings Late Screener Mailing - Web + Paper

Screener Recruitment Mailings

Web Push

Reminder 

Postcard

Fourth 

screener 

package

Third 

screener 

package

Second 

screener 

reminder 

letter

Pressuresealed 

envelope

Initial 

screener 

invite + 

$5 (cash)

Advance 

letter 
National Center for 

Education Statistics

Baseline Condition (control)

Early mailing contact envelopes asked for 

response by March 13, 2023

N=53,000

Early Bird Condition

Early mailing contact envelopes asked for 

response by March 13, letters in early 

mailings promised $20 (cash) for response by 

March 13, 2023

N=53,000

Intervention: Promise $20 (cash) for completing the screener in the early mailing phase 

of screener recruitment before participants receive paper copies of the screener

7

Early Bird Experiment - Methods
Envelope featuring respond-by date
Excerpt from early bird letter promising incentive:
National Center for 

Education Statistics

10

Early Bird Experiment - Research Questions

Questions Outcomes of interest

Does an early bird incentive increase response 

rates?

Final response rates

• Screener response rate 

• Topical response rate

• Overall response rate 

Does an early bird incentive affect screener 

response timing?

Screener response after early mailings and after all 

screener mailings

Does an early bird incentive affect screener 

response mode?

Mode of screener response

Does an early bird incentive affect the 

representativeness of NHES screener respondents

Characteristics of screener respondents
National Center for 

Education Statistics

11

Findings - Does an early bird incentive increase the 

final screener response rate?

52.2

30.8

49.6

26.4

0

25

50

75

100

All Cases Low Response Propensity Cases

Percentage

Final Screener Response Rate, by Condition and Response Propensity

*

*

*= significant difference from baseline

Early Bird Condition Baseline

Preliminary Results
National Center for 

Education Statistics

12

Findings - Does an early bird incentive increase the 

final topical response rate?

95.3 94.4 93.6 92.2

0

25

50

75

100

All Cases Low Response

Propensity Cases

Percentage

PFI

96.0 96.1 * 94.7 92.8

0

25

50

75

100

All Cases Low Response

Propensity Cases

Percentage

ECPP

* *

*= significant difference from baseline

Final Topical Response Rate by Topical Survey, Condition, and Response Propensity

Early Bird Condition Baseline

Preliminary Results
National Center for 

Education Statistics

13

Findings - Does an early bird incentive increase the 

final overall response rate?

Early Bird Condition Baseline

*= significant difference from baseline

49.8

29.0

46.9

24.4

0

15

30

45

60

All Cases Low Response Propensity Cases

Percentage

Final Overall Response Rate, by Condition and Response Propensity

*

*

Preliminary Results
National Center for 

Education Statistics

14

Findings - Does an early bird incentive affect 

response timing?

Final Screener Response Rate by Timing, Condition, and Response Propensity

Responses received during Early Mailing 

Responses received during Later Mailing 

42.8

33.8

9.4

15.8

0

15

30

45

60

Early Bird Baseline

Percentage

All Cases

24.1

16.8

6.6

9.6

0

15

30

45

60

Early Bird Baseline

Percentage

Low Response Propensity

*

*

*

*

*= significant difference from baseline

Preliminary Results
National Center for 

Education Statistics

25.6 20.2

3.9

4.8

1.3

1.4

0

15

30

45

60

Early bird Baseline

Percentage

Screener Response Mode

15

Findings - Does an early bird incentive affect screener 

response mode?

43.8 39.7

6.3

7.3

2.0 2.6

0

15

30

45

60

Early Bird Baseline

Percentage

All Cases

*

Final Screener Response by Mode, Condition, and Response Propensity

Web Paper Phone

*

*

*

*

*= significant difference from baseline

Preliminary Results
National Center for 

Education Statistics Early Bird 16

Findings - Does an early bird incentive affect the 

representativeness of NHES screener respondents?

23.5

30.5 32.4

0

15

30

45

60

75

Over 65

Percentage

Age

11.6 12.6 9.7

14.2

9.0

15.0

0

15

30

45

60

75

$0 to $24,999 $100,000-$149,999

Percentage

Income

53.2

16.6

62.4

12.2

64.4

11.2

0

15

30

45

60

75

Own Rent

Percentage

Home Tenure

Sample Baseline *= significant difference from baseline

*

*

*

*

*

Preliminary Results
National Center for 

Education Statistics

17

Conclusions and Next Steps

The Early Bird Incentive... 

... increased response rates

... encouraged earlier screener responses

...encouraged screener response by web, and 

... has respondents that are as or more so 

representative than the baseline condition

Verify findings with final weighted data

Compare Early Bird Incentive with other 

methodological experiment results

Compare cost of Early Bird Incentive condition 

to other NHES:2023 conditions

Next Steps
National Center for 

Education Statistics

18

Questions?

michelle.mcnamara@ed.gov

rmedway@air.org

dbattle@air.org

aong@air.org