Converting the National Household 

Education Survey from RDD/CATI to 

ABS/Mail 

Andrew Zukerberg 

Survey Redesign Seminar 

Presented July 17, 2012 

This discussion is intended to promote the exchange of ideas among researchers and policy makers. The views expressed during 

discussion are part of ongoing research and analysis and do not necessarily reflect the position of the U.S. Department of 

Education. 
NHES Overview 

• Conducted approximately every 2 years since 1991 

• RDD/CATI survey, representative of U.S. 

noninstitutionalized civilian population 

– Nationally representative samples of children from birth through 

grade 12 and adults. 

• Last conducted as RDD/CATI in 2007 

• Covers many topics that can not be measured through 

institution-based studies: 

– Homeschooling, adult education, after-school activities, 

school readiness, child care and more 
State of the NHES in 2007 

• Conducted a bias study in 2007 that indicated 

little evidence of bias and some potential 

coverage issues in our RDD design 

• Response rate on Adult Education survey 

was going to be too low to publish 
NHES screener response rate over time 

Year of 

survey 

Completed 

Screeners 

Response 

rate 

Nonresponse type (percentage 

distribution) 

Refusals 

Maximum 

calls Other 

1991 60,322 81 84 7 9 

1993 63,844 82 68 15 18

1995 45,465 73 84 9 7 

1996 55,838 70 83 10 7 

1999 55,929 74 76 17 7 

2001 48,385 68 74 18 8 

2003 32,049 62 76 16 8 

2005 58,140 64 77 15 8 

2007 54,034 53 86 10 4 
Motivation for NHES redesign 

• Improve response rates 

• Improve coverage 

• Contain cost 
NHES redesign 

• Contracted for redesign in 2008 

• 2009 feasibility test 

– Would two phase mail survey work? 

• 2011 large scale field test 

– Develop a packaged design that could be used in 

production 
2009 feasibility test 

• Could two phase study be conducted by 

mail? 

• Three independent samples: 

– Nationally representative sample: 10,200 cases 

– Linguistically isolated sample: 800 cases 

– Households with children: 800 cases 

• Data collection September – mid-December 

2009
2009 Results 

• Found approach could boost response rates 

and improve coverage 

• Raised a number of questions about best way 

to implement design 

• Best screener questionnaire? 

• Telephone non response follow up? 

• How to increase response in Spanish-speaking populations? 

• Incentives? 
Conducted large scale field test in 2011 

 60,000 household ABS study 

 Random national sample 

 Spanish-language targeted sample (Linguistically Isolated, 

Hispanic Surname) 

 Up to 3 screener mailings, screener phone 

nonresponse follow-up

 Up to 3 topical mailings 

 12 experiments 

 Conducted January to early June 2011 
2011 Field test key research areas 

• Incentives 

– Various cash levels and timing, magnet 

• Spanish language materials 

• Mailing strategies 

– Advance letter, Fedex vs. Priority mail, Priority 

mail vs. first class, different envelope designs 

• Screener form type, content and design 
Value of multiple mailings (2011) 

End of 1st

mailing

End of 2nd

mailing

End of 3rd

mailing

Response Rate 40.9 53.5 66.7

% of HH with 

kids

28.7 30.2 31.2

Response rate and percent of HH with kids increased 

with each mailing 
Impact of Nonresponse Follow Up by Phone 

on Screener Completion (2009) 

Assigned Follow up 

mode

Vendor phone 

available

1

st follow up 

completion 

rate

2

nd follow up 

completion rate

Mail No 42.8 27.1

Mail Yes 49.3 34.6

Phone (includes late 

mail completes)

Yes 34.4 21.8

Phone (completed by 

phone)

Yes 18.0 12.4

• Most phone assigned cases were completed by mail prior to a phone call in the first 

follow up. 

• Mail/FedEx outperformed phone at the first and second nonresponse follow up stages 

amongst cases with a vendor-provided phone number. 
Screener

Treatment Wave 1 Wave 2 Wave 3

Screener 

Overall

Prenotice letter improved

response

1.243 (p < .0001)

$5.00 improved 

response

1.309 (p < .0001)

improved 

response

1.107 (p = .0049)

improved 

response

1.230 (p< .0001)

Magnet

Requested name decreased 

response

0.918 (p = .0197)

Requested 

phone

decreased 

response

0.849 (p = .0083)

decreased 

response

0.840 (p< .0001)

Screenout improved 

response

1.047 (p=.044)

improved 

response

1.303 (p< .0001)*

*as 2nd/3rd mailing

FedEx N/A N/A improved 

response

1.363 (p < .0001)

N/A
2011 ‘Best’ and ‘Worst’ final response rate

Screener Topical Overall

2011 best 

overall*

74.0 79.3 58.7

2011 worst 

overall*

64.5 70.8 45.7

2009 field test* 61.8 72.3 44.7

2007 PFI 52.8 74.1 39.1

2005 ECPP 64.2 84.4 54.2

*unweighted, based on multiple treatment groups 
Non response bias analysis from 2011 field 

test 

• Some concern that mail mode would 

underrepresent very low income population

– Did not see evidence of this (mainly because we 

used parent’s highest educational level as raking 

dimension) 

• No evidence unit non response bias on the 

characteristics selected for evaluation 

• Item non response rates were low (below 

10% for most items) mitigating potential item 

non response bias 
Major changes from RDD/CATI to mail 

• CATI/RDD fielded multiple modules in HH 

– Now we only ask about one person 

– This impacts our sample size 

• CATI/RDD had complex skips and fills 

– We have simplified the items and ask less detail 

– e.g., collect fewer care arrangements 

– Created different questionnaire versions 

• Mail data collection period will likely be similar 

to CATI but topical fielded later 
2012 Screener design 

• Advance letter 

• Initial mailing: screener with $5 incentive, first class, 

cover letter 

• Spanish/English forms to LI and Hispanic surname 

households, expanding with each mailing 

• Reminder postcard after 1 week 

• 2nd questionnaire mailing after 3 weeks 

• FedEx to all non responding HH at second non 

response follow up, reminder phone call, after 6 weeks 

• 3rd non response mailing (4th mailing), after 9 weeks 

• TQA interview 
2012 Design continued 

• Returned screeners are clerically 

reviewed to determine if children reported 

• Questionnaires with children are scanned 

• Data is transmitted weekly 

• Within household sampling is performed 

and a topical input file is created 

• File is reviewed to check names 

• Topical questionnaires are labeled and 

mailed in groups 
2012 Topical design 

• Initial mailing first class, distinctive envelope, 

$5 cash incentive 

• Reminder postcard after 1 week 

• 2nd mailing, first class, distinctive envelope, 

after 3 

• 3rd mailing, FedEx, reminder phone call, after 

6 weeks 

• 4th mailing, first class distinctive envelope, 

after 9 weeks 
2012 Experiments 

Screener 

• Census vs. NCES/ED branding 

• No name screener 

Topical 

• $15 incentive at topical level for late screener 

responders 

• A topical ‘short form’ against fourth mailing of a full 

form 
Thank You 
Additional materials 
English screener 

questionnaire

$5 incentive English/Spanish screener 

questionnaire 

$5 incentive

Reminder 

Postcard

Returned questionnaire? 2 nd Mailing: cover letter, questionnaire, 

Spanish questionnaire to 

more HH, return envelope, first class

Returned Questionnaire? No 3 rd mailing: questionnaire, Spanish questionnaire, cover letter, return 

envelope, Fedex, telephone reminder call

Returned Questionnaire?

Stop No Eligible Child?

No

Yes

Yes

Yes

Sample 

Reference 

Child Returned Questionnaire? Yes

Begin 

Topical 

Cycle

No

4th Mailing: questionnaire, Spanish questionnaire, cover letter, return envelope, first class mail

No

Stop

No

Yes

Hispanic oversample 

or Spanish Surname

Advance 

Letter

No Yes 1st Mailing: Screener Data Collection



Spanish screener 

Sampled 

child’s 

information