National Household Education Survey 

2016 (NHES:2016)

Full-scale Data Collection

OMB# 1850-0768 v.11

Part B

April 2015

revised in June 2015
i

TABLE OF CONTENTS

Page

DESCRIPTION OF STATISTICAL METHODOLOGY ......................................................... 1

B.1 Respondent Universe and Statistical Design and Estimation .................................... 1

B.1.1 Sampling Households..................................................................................... 1

B.1.2 Within-Household Sampling.......................................................................... 2

B.1.3 Expected Yield............................................................................................... 4

B.1.4 Reliability....................................................................................................... 5

B.1.5 Estimation Procedures.................................................................................... 6

B.1.6 Nonresponse Bias Analysis............................................................................ 7

B.2 Statistical Procedures for Collection of Information ................................................. 9

B.3 Methods for Maximizing Response Rates................................................................. 17

B.4 Tests of Procedures and Methods.............................................................................. 17

B.5 Individuals Responsible for Study Design and Performance .................................... 18

List of Tables

Table Page

1 Expected percentage of households with eligible children or adults, by 

sampling domain........................................................................................................ 4

2 Expected numbers sampled and expected numbers of completed screeners 

and topical surveys .................................................................................................... 5

3 Numbers of completed topical interviews in previous NHES surveys administrations 5

4 Margins of error for screener..................................................................................... 5

5 Margins of error for topical surveys.......................................................................... 6

List of Figures

Figure

1 Mail Screener Data Collection Plan .......................................................................... 12

2 Web Screener Data Collection Plan .......................................................................... 13

3 Mail ECPP/PFI Data Collection Plan........................................................................ 14

4 Mail ATES Data Collection Plan .............................................................................. 15

3 Web Topical Data Collection Plan ............................................................................ 6
1

PART B. DESCRIPTION OF STATISTICAL METHODOLOGY

B.1 Respondent Universe and Statistical Design and Estimation

Historically, an important purpose of the National Household Education Surveys Program (NHES) has been to 

collect repeated measurements of the same phenomena at different points in time. Decreasing response rates during 

the past decade required NCES to redesign the NHES. This redesign involved changing the sampling frame from a 

list-assisted Random Digit Dial (RDD) to an Address-Based Sample (ABS) frame. The mode of data collection also 

changed from telephone, interviewer administered to mail, self administered.

The last NHES data collection prior to the redesign was conducted in 2007 and included the PFI and the School 

Readiness (SR) survey1. The NHES:2012, comprised of the PFI and ECPP topical surveys, was the first full-scale 

data collection using an addressed-based sample and a self-administered questionnaire. The overall screener plus 

topical response rate was approximately 58 percent for both the PFI and the ECPP in 2012, compared to the 2007 

overall response rate of 39-41% (depending on the survey). The results suggest that the new methodology has the 

ability to increase response rates and address coverage issues identified in the 2007 data collection.

The NHES:2016 will be an address-based sample covering the 50 states and the District of Columbia and will be 

conducted from January through August 2016. The household target population is all residential addresses 

(excluding P.O. Boxes that are not flagged by the United States Postal Service [USPS] as the only way to get mail)

and is estimated at 126,115,692 addresses.

Households will be randomly sampled as described in section B.1.1, and a screening questionnaire will be sent to 

each sampled household.2 Demographic information about household members provided on the screener will be 

used to determine whether anyone is eligible for the Early Childhood Program Participation (ECPP) survey, the 

Parent and Family Involvement in Education-Enrolled (PFI-E) survey, the Parent and Family InvolvementHomeschooled (PFI-H) survey, or the Adult Training and Education Survey (ATES). In addition, the study will 

include a seeded sample of 1,000 adults with educational certificates who will receive the ATES. This supplemental 

sample will not be included in the screener portion of NHES:2016.

The target population for the ECPP survey consists of children age 6 or younger who are not yet in kindergarten. 

The target population for the PFI-E survey includes children/youth ages 20 or younger who are enrolled in 

kindergarten through twelfth grade. The target population for the PFI-H survey includes children/youth ages 20 or 

younger who are homeschooled for kindergarten through twelfth grade. The target population for the ATES survey 

includes adults ages 16-65 who are not enrolled in grade 12 or below or homeschooled for equivalent grades. Ages 

will be calculated based on information provided in the screener.

B.1.1 Sampling Households

A nationally representative sample of 206,000 addresses will be used. The sample will be drawn in a single stage 

from a file of residential addresses maintained by a vendor, Marketing Systems Group (MSG), based on the USPS 

Computerized Delivery Sequence File (CDSF). MSG provided the sample for NHES:2012 and for the NHES 

Feasibility Study (NHES-FS) in 2014. As in past NHES surveys, the NHES:2016 will oversample black and 

Hispanic households using Census data. This oversampling is necessary to produce more reliable estimates for 

subdomains defined by race and ethnicity. NHES:2016 will use the stratification methodology that was used for the 

NHES:2012 administration. The sampling design concentrates on achieving a significant sample of black and 

Hispanic households. The universe consists of Census tracts likely to contain relatively high proportions of these 

subgroups. The three strata for the NHES:2016 are defined as households in:

1) Tracts with 25 percent or more Black persons

2) Tracts with 40 percent or more Hispanic persons (and not 25 percent or more Black persons)

3) All other tracts

The Hispanic stratum contains a high concentration of households in which at least one adult age 15 or older speaks 

Spanish and does not speak English very well and will be used for the assignment of bilingual screener mailing 

materials. The sample allocation to the three strata will be 20 percent to the black stratum, 15 percent to the

 

1 NHES:2007 also included an adult education module, but data collection was stopped due to low response rates.

2

In the web experiment, each household will receive a letter with a user identification code and URL address asking someone in the household to complete the 

survey by web rather than a letter with a screening questionnaire.
2

Hispanic stratum and 65 percent to the third stratum. This allocation will provide improvement in the precision of 

estimates by race/ethnicity compared to use of a uniform sampling rate across each stratum, and will protect against 

unknown factors that may affect the estimates for key subgroups, especially differential response rates.

In addition to stratifying by the race/ethnicity groups mentioned above, these three strata will be sorted by a poverty 

variable and the sample will be selected systematically from the sorted list in order to maintain the true poverty 

level proportions in the sample. The Census tracts will be used to define the two categories of poverty level as 

households in:

1) Tracts with 20% or more below the poverty line

2) Tracts with less than 20% below the poverty line

Households will be randomly assigned to the web experiment data collection treatment (web treatment) or the

standard mail data collection treatment (mail treatment). A total of 35,000 households out of the 206,000 sampled 

will be chosen for the web experiment.

B.1.2 Within-Household Sampling

Among households that return a completed screener and report members eligible for one or more topical surveys, a 

four-phase procedure will be used to select a single household member to receive a topical questionnaire. In order 

to minimize household burden, only one eligible member from each household will be sampled; therefore, each 

household will receive only one of the four topical surveys. The topical sampling procedure works by randomly 

assigning three pre-designations to each household. Depending on the composition of the household, some or all of 

these pre-designations are used to assign the household to one of the four topical surveys. If the household has more 

than one member eligible for the survey for which it is selected, the final phase of sampling randomly selects one of 

these members for that survey. The remainder of this section provides additional detail on each of the four phases, 

and provides examples of how this procedure would work for three hypothetical households.

First-Phase sampling: Homeschooling Status (PFI-H or Other Survey)

In the first phase of sampling, each address will be randomly pre-designated as either a “PFI-H household” or an 

“other household.” This pre-designation will be referred to as the homeschooling pre-designation. A household will 

be pre-designated as a “PFI-H household” with a .8 probability and as an “other household” with a .2 probability. 

This pre-designation will only be used when a household has at least one member eligible for the PFI-H and at least 

one member eligible for one of the other topical surveys, to determine which person in the household should be 

sampled. If a household only has a member or members eligible for the PFI-H, that household will receive the PFIH regardless of pre-designation. Likewise, if a household only has members eligible for one of the other surveys, 

that household will receive a non-homeschooling survey regardless of pre-designation. The purpose of sampling for 

the PFI-H in the first phase of topical sampling is to increase the number of homeschooled children for whom data 

are collected. NHES is the only nationally representative survey that collects information about homeschooled 

children in the United States. Because this is such a small sub-population, prior NHES administrations, which did 

not sample households separately for the PFI-H

3

, have typically collected homeschooling data from only a few 

hundred households. Based on preliminary analysis, sampling households for the PFI-H with high probability in the 

first phase of topical sampling is expected to approximately double the PFI-H sample size, relative to what would 

be obtained from the sampling design used in prior administrations. Due to the small number of households with 

homeschooling children, this is expected to have a minimal impact on sample sizes and design effects for the other 

topical surveys.

Second-Phase Sampling: Age Group (Child or Adult)

The second phase of sampling is at the age group level (child or adult). In addition to the homeschool predesignation, each household will be randomly pre-designated as either a “child household,” or an “adult 

household.” This pre-designation will be referred to as the age group pre-designation. A household will be predesignated as a “child household” with a .8 probability and an “adult household” with a .2 probability. Because 

 

3

In prior administrations, a household received the PFI-H if the household was sampled for the PFI, and the individual child sampled in the final phase of 

sampling was identified on the screener as a homeschooler.
3

eligible children comprise a smaller portion of the population compared to eligible adults, differential sampling is 

applied to ensure a sufficient sample size for the ECPP, PFI-E, and PFI-H surveys. The pre-designation will only be 

used when a household was not selected for the PFI-H in the first phase of sampling, and has both an eligible 

child/children and an eligible adult/adults, to determine which person in the household should be sampled. If a 

household only has an eligible child/children and no eligible adults, that household will receive a child topical 

survey, regardless of pre-designation. Likewise, if a household only has an eligible adult/adults and no eligible 

children, that household will receive an adult topical survey, regardless of pre-designation.

Third-Phase Sampling for Households Selected to Receive a Child Topical Survey: Domain (ECPP or PFI)

The third phase of sampling is at the topical survey level, or the domain level. Because households are sampled for 

the PFI-H in the first phase of topical sampling, there are two topical surveys that a child could possibly be eligible 

for in the third phase: the ECPP or the PFI-E survey. A child’s age, among other factors, will determine if the child 

is eligible for the ECPP or the PFI-E topical surveys. The eligibility criterion only allows a child to be eligible for 

one of the topical surveys, not both.

In addition to the homeschool and age group pre-designations, each household will be pre-designated as an “ECPP 

household” with a .7 probability or a “PFI-E household” with a .3 probability. This pre-designation will be referred 

to as the child topical survey pre-designation. Because children eligible for the ECPP survey comprise a smaller 

portion of the population compared to children eligible for the PFI-E survey, differential sampling is done to ensure 

a sufficient sample size of the ECPP group. This designation will only be used when a household was not selected 

for the PFI-H in the first phase, was selected for a child survey in the second phase, and has a child/children eligible 

for the ECPP topical survey and a child/children eligible for the PFI-E topical survey, to determine which person in 

the household should be sampled. If a child household only has a child/children eligible for the ECPP topical 

survey and not the PFI-E topical survey, that household will receive an ECPP topical survey, regardless of predesignation. Likewise, if a child household only has a child/children eligible for the PFI-E topical survey and not 

the ECPP topical survey, that household will receive a PFI-E topical survey, regardless of pre-designation.

Fourth-phase Sampling: Person Level

The fourth phase of sampling is at the person level. If any household has only one person that is eligible for the 

domain that was selected in the first three phases of sampling, that person is selected to receive that topical survey. 

If any household has two or more people eligible for the domain that was selected in the first three phases of 

sampling, one of those persons will be randomly selected (with equal probability) to receive the appropriate topical 

survey. Therefore, at the end of the four phases of sampling, one eligible child or one eligible adult within each

household that has a completed screener will be sampled for the ECPP/PFI topical survey or the ATES topical 

survey, respectively.

Topical Sampling Examples

The use of the topical pre-designations can be illustrated by considering hypothetical households with members 

eligible for various combinations of the topical surveys. First, consider a household with six people: one member 

eligible for the ECPP, two members eligible for the PFI-E, one member eligible for the PFI-H, and two members 

eligible for ATES. Suppose that the household is pre-designated as an “other household” in the first phase, a “child 

household” in the second phase, and a “PFI-E household” in the third phase. The first pre-designation indicates that 

the household should not receive the PFI-H; the second pre-designation indicates that the household should receive 

a child survey rather than an adult survey; and the third pre-designation indicates that the household should receive 

the PFI-E. Because the household has two children eligible for the PFI-E, one of these children is randomly 

selected for the PFI-E with .5 probability in the final phase.

Second, consider a household with two people: one member eligible for the PFI-H and one member eligible for 

ATES. Suppose that the household is pre-designated as a “PFI-H household” in the first phase, an “adult 

household” in the second phase, and an “ECPP household” in the third phase. Because the household has a member 

eligible for the PFI-H, the first pre-designation is used to assign the household to the PFI-H. Therefore, the other 

two pre-designations are not used. Because only one member is eligible for the PFI-H, this child is selected with 

certainty for the PFI-H in the final phase.

Third, consider a household with three people: one member eligible for the ECPP and two members eligible for 

ATES. Suppose that the household is pre-designated as a “PFI-H household” in the first phase, a “child household” 
4

in the second phase, and a “PFI-E household” in the third phase. Because the household has no members eligible 

for the PFI-H, the first pre-designation is not used. Because the household has both eligible adults and an eligible 

non-homeschooling child, the second pre-designation is used to assign the household to a child survey. However, 

because the household only has an ECPP-eligible child, the third pre-designation is not used, and the household is 

assigned to receive the ECPP. Because only one member is eligible for the ECPP, this child is selected with 

certainty for the ECPP in the final phase.

Finally, consider a household with three people: one member eligible for ATES, one member eligible for the PFI-E, 

and one member eligible for the ECPP. Suppose that the household is pre-designated as an “other household” in the 

first phase, an “adult household” in the second phase, and a “PFI-E household” in the third phase. Due to the lack 

of homeschoolers in the household, the first pre-designation is not used. The second pre-designation assigns the 

household to ATES, and the third pre-designation is therefore not used.

B.1.3 Expected Yield

As described above, the initial sample will consist of approximately 206,000 addresses. An expected screener 

response rate of approximately 64.021 percent and an address ineligibility4rate of approximately 9.375 percent are 

assumed, based on results from the 2014 NHES-FS and expected differences in screener eligibility and response 

rates between the sampling strata and experimental treatment groups. Under this assumption, the total number of 

expected screeners is 119,520.

The ECPP/PFI or the ATES topical surveys will be administered to households with completed screeners that have 

eligible children or adults, respectively. For the NHES:2016, we expect to achieve a total percentage of households 

with eligible children of approximately 31.6 percent, and for eligible adults a total percentage of approximately 

77.8 percent.5 Expected estimates of the percentage of households with eligible children or adults overall and in 

each sampling domain are given in Table 1, as well as the expected number of screened households in the 

nationally representative sample of 206,000, based on the distribution of household composition and assuming 

119,520 total completed screeners.

Table 1. Expected percentage of households with eligible children or adults, by sampling domain

Household composition Percent of 

households

Expected number 

of screened 

households

Total households with eligible adults 77.810 92,999

Total households with eligible children 31.649 37,827

Households with at least one PFI-E eligible child and no PFI-H or 

ECPP eligible children 19.139 22,875

Households with at least one PFI-H eligible child and no PFI-E or 

ECPP eligible children 0.474 566

Households with at least one ECPP eligible child and no PFI-E or 

PFI-H eligible children 5.751 6,874

Households with at least one PFI-E eligible child, at least one PFI-H 

eligible child, and no ECPP eligible children 0.163 195

Households with at least one PFI-E eligible child, at least one ECPP 

eligible child, and no PFI-H eligible children 5.857 7,000

Households with at least one PFI-H eligible child, at least one ECPP 

eligible child, and no PFI-E eligible children 0.182 218

Households with at least one PFI-E eligible child, at least one PFI-H 

eligible child, and at least one ECPP eligible child 0.082 98

NOTE: The distribution in this table assumes 119,520 screened households. Detail may not sum to totals because of rounding. Estimates are 

based on calculations from the 2014 NHES Feasibility Study (NHES-FS).

 

4

Ineligible addresses are those that are undeliverable. Screener mailings for an address where one or more mailings are returned as a postmaster return (PMR) 

and no mailings are returned complete or refused will lead to an address being coded as ineligible.

5

Percentages based on estimates from the 2014 NHES Feasibility Study (NHES-FS) screener.
5

Table 2 summarizes the expected numbers of completed interviews for the NHES:2016. These numbers take into 

account within-household sampling. Based on results from the NHES:2012, a topical response rate of 

approximately 79.245% is expected for the ECPP, approximately 79.242% for the PFI-E, and approximately 

79.171% for the PFI-H. A response rate ofapproximately 74.245 percent is assumed for ATES based on the 2014 

NHES-FS.6 Based on an initial sample of 206,000 addresses and an expected eligibility rate of 90.625 percent, the 

expected number of completed screener questionnaires is 119,520. Of these, we expect to have approximately 9,540

children selected for the ECPP, 20,224 children for the PFI-Enrolled, 869 children for the PFI-Homeschooled, and 

63,855 adults selected for the ATES questionnaire.

Table 2. Expected number of sampled and completed screeners and topical surveys for households

Survey Expected number sampled 

NHES:2016

Expected number of completed interviews 

NHES:2016

Household screeners 206,000 119,520

ECPP 9,540 7,560

PFI-Enrolled 20,224 16,026

PFI-Homeschooled 869 688

ATES 63,855 47,409

Table 3 shows the number of ECPP and PFI completed interviews from previous NHES surveys administrations.

Table 3. Numbers of completed topical interviews in previous NHES surveys administrations

Number of 

completed topical 

interviews

NHES surveys administration

1993 1995 1996 1999 2001 2003 2005 2007 2012

ECPP † 7,564 † 6,939 6,749 † 7,209 † 7,893

PFI 19,144 † 17,774 17,652 † 12,422 † 10,681 17,563

† Not applicable; children in this category were not eligible for these topical interviews.

B.1.4 Reliability

The reliability of the sample sizes is measured by the margins of error. The reliability of the screener sample size is 

shown in Table 4. The “value of p” represents a survey estimate proportion. So, if an estimated proportion is 70 

percent, the margin of error will be below .01 for the overall population, as well as within subgroups that constitute

50 percent, 20 percent, and 10 percent of the population. The same is true for survey estimate proportions of 60 or 

50 percent, with the exception of estimates within subgroups that constitute 10 percent of the population—for these, 

the margin of error will slightly exceed .01.

Table 4. Margins of error for screener†

Survey value of p 1.96 times standard 

error - overall

1.96 times standard error -

50% characteristic

1.96 times standard error 

- 20% characteristic

1.96 times standard error -

10% characteristic

Screener

0.7 0.0030 0.0042 0.0067 0.0094

0.6 0.0032 0.0045 0.0071 0.0101

0.5 0.0033 0.0046 0.0073 0.0103

†Based on preliminary analysis, a design effect of 1.317 was used for the calculations in this table. This represents the design 

effect due to unequal weighting at the screener level.

 

6 Expected topical response rates represent rounded weighted averages of the response rates expected within each experimental group.
6

The margins of error for the topical surveys are shown in Table 5.

Table 5. Margins of error for topical surveys†

Survey value 

of p

1.96 times standard 

error - overall

1.96 times standard 

error - 50% 

characteristic

1.96 times standard 

error - 20% 

characteristic

1.96 times standard 

error - 10% 

characteristic

ECPP

0.7 0.0142 0.0200 0.0317 0.0448

0.6 0.0151 0.0214 0.0339 0.0479

0.5 0.0155 0.0219 0.0346 0.0489

PFI-Enrolled

0.7 0.0108 0.0152 0.0241 0.0341

0.6 0.0115 0.0163 0.0257 0.0364

0.5 0.0117 0.0166 0.0263 0.0372

PFI-Homeschooled

0.7 0.0448 0.0634 0.1003 0.1418

0.6 0.0479 0.0678 0.1072 0.1516

0.5 0.0489 0.0692 0.1094 0.1547

ATES

0.7 0.0073 0.0103 0.0163 0.0231

0.6 0.0078 0.0110 0.0175 0.0247

0.5 0.0080 0.0113 0.0178 0.0252

†Based on preliminary analysis, the following design effects were used in the calculations for this table: 1.880 for the ECPP, 

2.304 for the PFI-E, 1.715 for the PFI-H, and 3.133 for ATES. These represent the design effects due to unequal weighting at the 

screener and topical levels.

B.1.5. Estimation Procedures

The data sets from the NHES:2016 will have weights assigned to facilitate estimation of nationally representative 

statistics. All households responding to the screener will be assigned weights based on their probability of selection 

and a non-response adjustment, making them representative of the household population. All individuals in the 

national sample responding to the topical questionnaires will have a record with a person weight designed such that 

the complete data set represents the target population. The 1,000 seeded certificate sample cases who receive the 

ATES survey will be excluded from the weighting procedures since they were not selected using a probabilistic 

sampling procedure.

The estimation weights for the NHES:2016 surveys will be formed in stages. The first stage is the creation of a base 

weight for the household, which is the inverse of the probability of selection of the address. The second stage is a 

screener nonresponse adjustment to be performed based on characteristics available on the frame and screener.

The household-level weights are the base weights for the person-level weights. For each completed topical 

questionnaire, the person-level weights also undergo a series of adjustments. The first stage is the adjustment of 

these weights for the probability of selecting the person within the household. The second stage is the adjustment of 

the weights for topical survey nonresponse to be performed based on characteristics available on the frame and 

screener. The third stage is the raking adjustment of the weights to Census Bureau estimates of the target 

population. The variables that may be used for raking at the person level include race and ethnicity of the sampled 

person, household income, home tenure (own/rent/other), region, age, grade of enrollment, gender, family structure 

(one parent or two parent), and highest educational attainment in household. These variables (e.g., family structure) 

have been shown to be associated with response rates. The final raked person-level weights include undercoverage 

adjustments as well as adjustments for nonresponse.

Standard errors of the estimates will be computed using a jackknife replication method. The replication process 

repeats each stage of estimation separately for each replicate. The replication method is especially useful for 

obtaining standard errors for the statistics that the NHES:2016 produces.
7

B.1.6 Nonresponse Bias Analysis

To the extent that those who respond to surveys and those who do not differ in important ways, there is a potential 

for nonresponse biases in estimates from survey data. The estimates from NHES:2016 are subject to bias because of 

unit nonresponse to both the screener and the extended topical surveys, as well as nonresponse to specific items. 

Per NCES statistical standards, a unit-level nonresponse bias analysis will be conducted if the NHES:2016 overall 

unit response rate (the screener response rate multiplied by the topical response rate) falls below 85 percent (as is 

expected). Additionally, any item with an item-level response rate below 85 percent will be subject to an 

examination of bias due to item nonresponse.

Unit nonresponse

To identify characteristics associated with unit nonresponse, a multivariate analysis will be conducted using a 

categorical search algorithm called Chi-Square Automatic Interaction Detection (CHAID). CHAID begins by 

identifying the characteristic of the data that is the best predictor of response. Then, within the levels of that 

characteristic, CHAID identifies the next best predictor(s) of response, and so forth, until a tree is formed with all 

of the response predictors that were identified at each step. The final result is a division of the entire data set into 

cells by attempting to determine sequentially the cells that have the greatest discrimination with respect to the unit 

response rates. In other words, it divides the data set into groups so that the unit response rate within cells is as 

constant as possible, and the unit response rate between cells is as different as possible. Since the variables 

considered for use as predictors of response must be available for both respondents and nonrespondents, 

demographic variables from the sampling frame provided by the vendor (including household education level, 

household race/ethnicity, household income, age of head of household, whether the household owns or rents the 

dwelling, and whether there is a surname and/or phone number present on the sampling frame) and data from the 

screener (including number of children and the number of adults in the household) will be included in the CHAID 

analysis. The results of the CHAID analysis will be used to statistically adjust estimates for nonresponse. 

Specifically, nonresponse-adjusted weights will be generated by multiplying each household’s unadjusted weight 

(the reciprocal of the probability of selection, reflecting all stages of selection) by the inverse response rate within 

its CHAID cell.

An alternative method for identifying characteristics associated with unit nonresponse is to use regression trees to 

model all the covariate relationships in the data rather than CHAID. Each covariate is then put into one regression 

model to predict unit nonresponse. We plan to calculate nonresponse adjustments using both the CHAID method 

and regression tree method in 2016 to see if differences in estimates would result. Depending on the results of this 

analysis, we could decide to provide either one or two sets of weights to data analysts on the 2016 file and we can 

evaluate whether to use CHAID or regression trees for nonresponse adjustments to future NHES collections.

The extent of potential unit nonresponse bias in the estimates will be analyzed in several ways. First, the percentage 

distribution of variables available on the sampling frame will be compared between the entire eligible sample and 

the subset of the sample that responded to the screener. While the frame variables are of unknown accuracy and 

may not be strongly correlated with key estimates, significant differences between the characteristics of the 

respondent pool and the characteristics of the eligible sample may indicate a risk of nonresponse bias. Respondent 

characteristics will be estimated using both unadjusted and nonresponse-adjusted weights (without the raking 

adjustment), in order to assess the potential reduction in bias attributable to statistical adjustment for nonresponse. 

A similar analysis will be used to analyze unit nonresponse bias at the topical level, using both sampling frame 

variables and information reported by the household on the screener.

In addition to the above, the magnitude of unit nonresponse bias and the likely effectiveness of statistical 

adjustments in reducing that bias will be examined by comparing estimates computed using adjusted weights to 

those computed using unadjusted weights. The unadjusted weight is the reciprocal of the probability of selection, 

reflecting all stages of selection. The adjusted weight is the extended interview weight adjusted for unit 

nonresponse (without the raking adjustment). In this analysis, the statistical significance of differences in estimates 

will be investigated only for key survey estimates including, but not limited to, the following:

All surveys

 Age/grade of child or age of adult

 Census region

 Race/ethnicity
8

 Parent 1 or adult’s employment status

 Parent 1 or adult’s home language

 Educational attainment of parent 1/adult

 Family type

 Household income

 Home ownership

Early Childhood Program Participation (ECPP)

 Child receiving relative care

 Child receiving non-relative care

 Child receiving center-based care

 Number of times child was read to in past week

 Someone in family taught child letters, words, or numbers

 Child recognizes letters of alphabet

 Child can write own name

 Child is developmentally delayed

 Child has health impairment

 Child has good choices for child care/early childhood programs

Parent and Family Involvement in Education (PFI)

 School type: public, private, homeschool

 Whether school assigned or chosen

 Contact from school about child’s behavior

 Contact from school about child’s school work

 Child’s overall grades

 Parents participate in 5 or more activities in the child’s school

 Parents report school provides information very well

About how child is doing in school

About how to help child with his/her homework

About why child is placed in particular groups or classes

About how to help child plan for college or vocational school

About the family's expected role at child’s school

 Parents attended a general school meeting (open house), back-to-school night, meeting of parent-teacher organization

 Parents went to a regularly scheduled parent-teacher conference with child’s teacher

 Parents attended a school or class event (e.g., play, sports event, science fair) because of child

 Parents acted as a volunteer at the school or served on a committee

 Parents check to see that child's homework gets done

Adult Training and Education Survey

 Certification or license

 Certificate

 Completed work experience program

 Labor force participation and employment status

The final component of the bias analysis will include comparisons between respondent characteristics and known 

population characteristics from extant sources including the Current Population Survey (CPS) and the American 

Community Survey (ACS). Additionally, for substantive variables, weighted estimates will be compared to prior 

NHES administrations if available. While differences between NHES:2016 estimates and those from external 

sources as well as prior NHES administrations could be attributable to factors other than bias, differences will be 

examined in order to confirm the reasonableness of the 2016 estimates.

Item nonresponse

In order to examine item nonresponse, all items with response rates below 85 percent will be listed. Alternative sets 

of imputed values will be generated by imposing extreme assumptions on the item nonrespondents. For most items, 

two new sets of imputed values—one based on a “low” assumption and one based on a “high” assumption—will be 

created. For most continuous variables, a “low” imputed value variable will be created by resetting imputed values 
9

to the value at the fifth percentile of the original distribution; a “high” imputed value variable will be created by 

resetting imputed values to the value at the 95th percentile of the original distribution. For dichotomous and most 

polytomous variables, a “low” imputed value variable will be created by resetting imputed values to the lowest 

value in the original distribution, and a “high” imputed value variable will be created by resetting imputed values to 

the highest value in the original distribution. Both the “low” imputed value variable distributions and the “high” 

imputed value variable distributions will be compared to the unimputed distributions. This analysis helps to place 

bounds on the potential for item nonresponse bias through the use of “worst case” scenarios.

B.2 Statistical Procedures for Collection of Information

This section describes the data collection procedures to be used in the NHES:2016. These procedures represent a 

combination of best practices to maximize response rates based on findings from the ATES 2010 Pilot Study, the 

NHES:2011 Field Test, NHES:2012, the 2013 National Adult Training and Education Survey (NATES), and the

2014 NHES-Feasibility Study (NHES-FS) within NCES’s cost constraints. The NHES is a two-phase selfadministered survey. In the first phase, households are screened to determine if they have eligible adults or 

children. In the second phase, a detailed topical questionnaire is sent to households with the eligible adults or 

children. The NHES employs multiple contacts with households to maximize response. These include an advance 

letter and up to four questionnaire mailings for both the screener and the topical surveys. In addition, households 

will receive one reminder postcard after the initial mailing of a screener or topical questionnaire. Respondent 

contact materials are provided in Appendix 1 and survey instruments in Appendix 2.

Screener Procedures

Advance letter. The NHES:2016 data collection will begin with the mailing of an advance notification letter in early 

January 2016. The NHES:2011 Field Test showed that using an advance letter was effective at raising response 

rates to the first questionnaire mailing which reduces the amount of followup required. Based on this finding,

NHES:2012, NATES, and NHES-FS also used an advance letter.

Screener package. Households will be randomly designated as either a mail treatment household or a web treatment 

household. As shown in Figure 1, mail treatment households will be mailed a questionnaire package one week after 

the advance letter. The packages will contain a cover letter, household screener, business reply envelope, and a cash

incentive. Figure 2 presents the data collection plan for the web treatment households. The initial mailing will 

include an advance letter by mail (which includes the web survey URL and log-in credentials) and a cash incentive.

Incentive. Many years of testing in the NHES have shown the effectiveness of a small cash incentive on increasing 

response. The NHES:2011 Field Test showed that a $5 incentive was more effective at increasing screener response 

than a $2 incentive. The NHES-FS confirmed the positive effects on response rates of a $5 incentive versus no 

incentive. NHES:2016 will include an experiment to further evaluate alternative incentive structures. Specifically, 

households selected for the targeted incentive experiment will receive either a $2, $5, or $7 incentive depending on 

their modeled response propensity. A smaller group will be selected to automatically receive $2 regardless of 

predicted response propensity to enable additional analysis of the sensitivity of different types of households to the 

incentive amount. Households not allocated to one of these treatment groups will receive $5 regardless of predicted 

response propensity. Web treatment households are not included in the targeted incentive experiment and will all 

receive the standard $5 incentive with the initial mailing. Part A, section A.1 of this submission provides a detailed 

description of the targeted incentive experiment.

Reminders and nonresponse follow-up. A thank you/reminder postcard will be sent to all sampled addresses 

approximately one week after the first mailing. Three nonresponse follow-up questionnaire mailings follow this 

postcard. All nonresponse follow-up questionnaire mailings for mail treatment households will contain a cover 

letter, replacement screener questionnaire, and business reply envelope. A second questionnaire mailing will be sent 

to nonresponding households approximately two weeks after the postcard. Approximately three weeks after the 

second mailing, a third questionnaire mailing will be sent to non-responding households using rush delivery (FedEx 

or USPS Priority Mail7). Results from the 2009 and 2011 tests indicated that households with eligible children tend 

to respond to later mailings. For this reason we added a mailing to our process and extended the data collection 

period for the NHES:2012 and NHES-FS. We will continue that protocol in the NHES:2016. The fourth 

 

7 Because FedEx cannot deliver to P.O. Boxes, P.O. Box-only addresses will be delivered using USPS Priority Mail.
10

questionnaire package will be sent to nonresponding households approximately three weeks after the third 

questionnaire mailing.

Web treatment households who do not respond to the first mailing will be sent a reminder postcard followed by 

another letter with their log-in credentials. Nonrespondent web treatment households will be merged into the mail

treatment data collection stream after the second mailing—they will receive a cover letter, paper questionnaire, and 

business reply envelope in the third (FedEx or USPS Priority Mail) and fourth mailings.

Language. The following criteria were used in NHES:2012 and are proposed again for NHES:2016 to determine 

which addresses receive a bilingual screener package:

• First mailing criteria: An address is in the Hispanic stratum, or a Hispanic surname is associated with the 

address, or the address is in a Census tract where 10 percent or more of households meet the criterion of 

being linguistically isolated Spanish-speaking.

• Second mailing criteria: An address is in the Hispanic stratum, or a Hispanic surname is associated with 

the address, or the address is in a Census tract where 3 percent or more of households meet the criterion of 

being linguistically isolated Spanish-speaking.

• Third and fourth mailing criteria: An address is in the Hispanic stratum, or a Hispanic surname is

associated with the address, or the address is in a Census tract where 2 percent or more of households meet 

the criterion of being linguistically isolated Spanish- speaking, or the address is in a Census tract where 2 

percent or more of the population spoke Spanish.

Topical Procedures

Topical survey mailings will follow procedures similar to the screener procedures. As described earlier, households 

with at least one eligible person will be assigned to one of four topical survey mailing groups:

(1) Households that have been selected to receive an ECPP questionnaire;

(2) Households that have been selected to receive a PFI-Enrolled questionnaire;

(3) Households that have been selected to receive a PFI-Homeschooled questionnaire; and

(4) Households that have been selected to receive an ATES questionnaire.

Within each group, questionnaires will be mailed in waves to minimize the time between the receipt of the screener 

and the mailing of the topical. Regardless of wave, households with a selected sample member will have the same 

topical contact strategy within the mail treatment group and the web treatment group. The initial topical mailing for 

sampled householdsin the mail treatment group will include a cover letter, questionnaire, business reply envelope, 

and a cash incentive. A reminder/thank you postcard is then sent to all topical sample members. A total of three 

nonresponse follow-up mailings follow. All nonresponse follow-up mailings will contain a cover letter, 

replacement questionnaire, and business reply envelope.

Web treatment cases that break off prior to completing any topical questionnaire items and those for which the 

screener respondent is not the sampled household member for the topical survey will receive a initial topical 

mailing (with the web survey URL and log-in credentials) and a $5 incentive. Web nonrespondents will receive a 

2

nd followup mailing with the URL and log-in credentials. In the 3rd and 4th mailings, web nonrespondents will be 

moved to the mail treatment group process, receiving a paper-and-pencil instrument and business reply envelope 

each time.

Incentive. The NHES:2011 Field Test, the NHES:2012, and the NHES-FS showed that a $5 incentive was effective 

with most respondents at the topical level. However, late screener respondents (those that responded after the third 

or fourth mailing) were more likely to respond to the $15 incentive amount. As a result, we will offer these late 

screener responders a $15 prepaid incentive with their first topical questionnaire mailing. All other respondents in 

the main sample will receive $5 with their initial mailing. Households with low response propensity in the targeted 

incentive experiment who received a $7 incentive for the screener will also receive a $7 incentive for the topical 

survey, unless they are a late screener responder in which case they will receive a $15 incentive. One week after the 

initial topical mailing, all households will receive a thank you/reminder postcard. Web treatment households that 

complete the topical instrument online immediately after completing the screener will receive no additional 

incentive. Web treatment households that break off before completing any topical items or for whom the screener 
11

respondent is not the sampled household member will receive a $5 topical incentive with the initial topical mailing. 

Late screener respondents in the web treatment will receive a $15 incentive comparable to the those in the mail 

treatment.

Language. Households in the mail treatment group will receive a topical package in English or Spanish depending 

on the language they used to complete the screener. Households that prefer to complete the topical questionnaire in 

a different language from that used to complete the screener will be able to contact the Census Bureau to request 

the appropriate questionnaire. Households in the web treatment group will be able to toggle between Spanish and 

English at any time in the web topical instruments.

Reminders and nonresponse follow-up. Nonresponding households in the mail treatment group will be mailed a 

second topical questionnaire approximately two weeks after the reminder postcard. If households that have been 

mailed a second topical questionnaire do not respond, a third package will be mailed by rush delivery (FedEx or 

USPS Priority Mail) approximately three weeks after the second mailing. Most households will be sent packages 

via FedEx delivery; only those households with a P.O. Box-only address will be sent USPS Priority Mail. If the 

household does not respond to the third mailing, a fourth mailing will be sent via USPS First Class mail. Figures 3 

and 4 show the topical data collection plan for mail treatment households for the ECPP/PFI and ATES, 

respectively. Nonresponding households in the web treatment groups will receive a second letter with the survey 

URL and log-in credentials. If they do not respond by web after the second mailing, they will be merged into the 

mail treatment data collection stream receiving a cover letter, paper questionnaire, and business reply envelope in 

the third (FedEx or USPS Priority Mail) and fourth mailings.Figure 5 shows the topical data collection plan for web 

treatment households.

Experiment-related Procedures

One objective of the web experiment is to assess the feasibility of using e-mail to follow-up with respondents to the 

web survey. For this reason, half of the web experiment sample will be asked to provide an email address for the 

topical respondent. From this embedded experiment, we will have data about the expected proportion of web 

respondents who will break off after being asked for someone’s email address and the expected proportion of web 

respondents who will skip the question item that asks for an email address. We would also like to know the 

expected proportion of email addresses that we collect that are bouncebacks or otherwise invalid and unuseable. To 

find this rate, we plan to send emails in monthly waves to the last set of emails collected from the web respondents. 

The emails will say, “Your household recently participated in the National Household Education Survey. Thank

you!” The email will also tell recipients that this is an automated message. We will track email bouncebacks from 

this procedure.
12

Figure 1: Mail Screener Data Collection Plan

Returned 

questionnaire?

2

nd Mailing:

cover letter, English 

questionnaire, Spanish 

questionnaire, return 

envelope, first class

Returned 

Questionnaire?

No

3

rd mailing:

English questionnaire, Spanish 

questionnaire, cover letter, return 

envelope, FedEx

Returned 

Questionnaire?

Is the 

questionnaire 

complete?

No

Yes

Yes

Yes

Returned

Questionnaire?

No

4th Mailing:

English questionnaire, Spanish 

questionnaire, cover letter, return 

envelope, first class mail

No

No Stop

Yes

Advance 

Letter

Impute 

Reminder 

Postcard

Hispanic oversample

or Spanish Surname?

1

st Mailing:

English cover letter, 

English Screener with 

$2/$5/$7 incentive, 

First Class

Bilingual Cover Letter, 

English and Spanish 

Screener with $2/$5/$7 

incentive, First Class

No Yes

Stop

Sample 

Reference Child

Begin 

ECPP/

PFI 

Topical 

Cycle

Is there an 

eligible Adult?

Sample 

Reference 

Adult

Begin 

ATES 

Topical 

Cycle

Which topical 

treatment group?

Is there an 

eligible child?

Yes

No

No

Yes

Is there an 

eligible adult?

Yes

No

Sample 

Reference 

Adult

Group RC

Group RA

Yes

Is the 

questionnaire 

complete?

Yes

No Stop
13

Figure 2: Web Screener Data Collection Plan

Web Screener 

complete?

No

2

nd Mailing:

English or Bilingual 

letter with login info, 

first class

Web Screener 

Complete?

Follows MO Screener Flow 

beginning at 

3

rd mailing

Eligible person in 

household?

No

Yes

Yes

Advance 

Letter

Hispanic oversample

or Spanish Surname?

1

st Mailing:

English Letter with 

login info, $5 

incentive, First Class

Bilingual Letter with 

login info,$5 

incentive, First Class

No Yes

Yes Yes

No

Yes

No

Stop

Follows MO 

ECPP/PFI/ATES 

Topical Flow

Household is sent a paper 

Screener questionnaire 

without cover letter. Will 

follow MO Screener flow 

beginning at 3

rd mailing.

No

Stop

Eligible person in 

household?

Screener 

conducted on 

phone?

No

Yes

Screener 

respondent 

also topical 

respondent?

No

Yes Begin Web 

Topical Flow

Begin topical 

questions

No

Did 

respondent 

answer at least 1 

question?

Yes

Household contacts 

Census to 

indicate no internet or 

request paper version?

Send to 

Data 

Processing

Reminder 

Postcard
14

Figure 3: Mail Topical Data Collection Plan – ECPP/PFI
15

Figure 4: Mail Topical Data Collection Plan – ATES

1

st Mailing: Cover letter, 

questionnaire, $5 incentive 

($15 for late screener 

returns), return envelope, 

First Class 

Reminder 

Postcard

Returned 

questionnaire?

2 nd Mailing

Cover letter, 

questionnaire, return 

envelope, First Class

Returned 

questionnaire?

Yes No

3 rd Mailing:

Cover Letter, questionnaire, 

return envelope, FedEx 

reminder phone call

Returned 

questionnaire?

No

Yes

Yes

4th Mailing:

questionnaire, cover letter, 

return envelope, First 

Class

No

Send to 

ATES Data 

Processing

Returned 

questionnaire?

Yes No Stop

1

st Mailing: Cover letter, 

questionnaire, $5/$7 incentive 

($15 for late screener 

returns), return envelope, 

First Class 

Reminder 

Postcard

Returned 

questionnaire?

2

nd Mailing

Cover letter, 

questionnaire, return 

envelope, First Class

Returned 

questionnaire?

Yes No

3

rd Mailing:

Cover Letter, questionnaire, 

return envelope, FedEx 

Returned 

questionnaire?

No

Yes

Yes

4th Mailing:

questionnaire, cover letter, 

return envelope, First 

Class

No

Begin 

ATES 

Topical 

Cycle

Send to 

ATES Data 

Processing

Yes No Stop

From screener or 

seeded sample?

Screener

Seeded Sample Advance Letter

1

st Mailing: Cover Letter, 

questionnaire, 

$5 incentive, return 

envelope, First Class

Figure 5: Web Topical Data Collection Plan
16

No and different 

respondent from Screener

Follows MO ECPP/PFI/ATES 

Topical Flow 

beginning at 

3

rd mailing

Yes

1st Web Topical Mailing for 

breakoff respondent who 

completed Screener: Cover letter 

with same user ID as screener, $5 

incentive ($15 for late screener 

returns), First Class

No

1st Web Topical Mailing for new 

respondent : Cover letter with new 

user ID, $5 incentive ($15 for late 

screener returns), First Class

Household contacts Census 

to indicate no internet or 

request paper version?

Yes

No

At least 1 web topical 

question answered?

Household is sent a paper 

Topical questionnaire 

without cover letter. Will 

follow MO ECPP/PFI/ATES 

Topical Flow beginning at 3

rd

mailing.

Yes Send to Data 

Processing

2nd Web Topical Mailing for 

new respondent:

Cover letter with new user 

ID, first class

2nd Web Topical Mailing for 

same respondent that 

completed Screener: 

Cover letter with same user 

ID as screener, first class

No and same 

respondent from Screener

At least 1 web topical 

question answered?

Reminder 

Postcard

Survey Monitoring
17

Mail survey returns will be processed upon receipt, and reports from the survey management system will be 

prepared at least weekly. The reports will be used to continually assess the progress of data collection.

B.3 Methods for Maximizing Response Rates

The NHES:2016 design incorporates a number of features to maximize response rates. This section discusses those 

features.

Total Design Method/Respondent-Friendly Design. Surveys that take advantage of respondent-friendly design 

have demonstrated increases in survey response (Dillman, Smyth, and Christian 2008; Dillman, Sinclair, and Clark 

1993). We have honed the design of the NHES forms through multiple iterations of cognitive interviewing and field 

testing. These efforts have focused on the design and content of all respondent contact materials. In recent months, 

we conducted cognitive testing of all Spanish-language materials, including letters and questionnaires. As noted 

previously, we will include a respondent incentive in the initial screener mailing. Respondent incentives will also 

be used in the initial topical mailing. Many years of testing in the NHES have shown the effectiveness of a small 

cash incentive on increasing response. The Census Bureau will maintain an email address and a toll-free 

questionnaire assistance (TQA) line to answer respondent questions or concerns. If a respondent chooses to provide 

their information to the TQA staff, staff will be able to collect the respondent’s information on an Internet-based 

screener questionnaire. Additionally, the questionnaires contain frequently asked questions (FAQs) and contact 

information for the Project Officer.

Engaging Respondent Interest and Cooperation. The content of respondent letters and FAQs is focused on 

communicating the legitimacy and importance of the study. Past experience has shown that the NHES child survey 

topics are salient to most parents. In the NHES:2012, Census Bureau “branding” was experimentally tested against 

Department of Education branding. Response rates to Census Bureau branded questionnaires were higher compared 

to Department of Education branded questionnaires. Based on this finding, we will use Census Bureau-branded 

materials in the data collection.

Nonresponse Follow-up. The data collection protocol includes several stages of nonresponse follow-up at each 

phase. In addition to the number of contacts, changes in method (USPS First Class mail, FedEx, and automated 

reminder phone calls) are designed to capture the attention of potential respondents.

B.4 Tests of Procedures and Methods

Web Experiment: As described in Part A, section A.1, of this submission, the NHES:2016 will include a selfresponse internet experiment to determine if asking respondents to complete the survey on the internet results in an 

acceptable response rate. If an acceptable response rate is achieved, self-response internet surveys will be 

considered as a main mode of data collection in the future. A total of 35,000 of the 206,000 screened households 

will be selected for the web experiment.

Targeted Incentive Experiment: As described in Part A, sections A.1 and A.9 of this submission and section B.2 

of this document, the NHES:2016 will also include an incentive experiment designed to examine the effectiveness 

of leveraging auxiliary frame data to target (a) lower screener incentives to households with a higher modeled 

propensity to respond and (b) higher screener incentives to households with a lower modeled propensity to respond.

A total of 45,000 households will be allocated to this experiment: 35,000 to a modeled incentive group for which 

the incentive amount will be determined based on the modeled response propensity, and 10,000 to receive $2 

regardless of response propensity. Only households not selected for the web experiment will be eligible for the 

targeted incentive experiment.

Seeded Sample Experiment: As described in Part A, section A.1, of this submission, the NHES:2016 will include 

an opportunity sample of 1,000 known holders of educational certificates to evaluate the characteristics of true 

certificates and to assess the number of false positive responses. The seeded sample is not included in the 206,000 

screener sample and none of the seeded sample cases will be included in the web experiment or the targeted 

incentive experiment.

B.5 Individuals Responsible for Study Design and Performance

The persons listed below participated in the study design and are responsible for the collection and analysis of the 

data:
18

Sarah Grady, NCES

Sharon Boivin, NCES

Lisa Hudson, NCES

Andrew Zukerberg, NCES

Carolyn Pickering, Census

REFERENCES

Dillman, D.A., Sinclair, M.D., and Clark, J.R. (1993). Effects of questionnaire length, respondent-friendly design, 

and difficult questions on response rates for occupant-addressed Census mail surveys. Public Opinion 

Quarterly, 57, 289-304.

Dillman, D.A., Smyth, J.D., and Christian, L.M. (2008). Internet, mail, and mixed mode surveys: The Tailored 

Design Method. New York: Wiley.