Appendix F

PIAAC Reducing Nonresponse Bias and 

Preliminary Nonresponse Bias Analysis
PIAAC

Reducing Nonresponse Bias and Preliminary Nonresponse 

Bias Analysis

PIAAC_NRB Paper_10March10
TABLE OF CONTENTS

1. INTRODUCTION AND OVERVIEW ........................................................................................4

2. PROBABILITY SAMPLES AND UNBIASED ESTIMATES ...................................................5

3. PIAAC DATA QUALITY GOAL REDUCE NRB BEFORE, DURING, AND AFTER 

DATA COLLECTION .................................................................................................................6

3.1 REDUCE NRB BEFORE DATA COLLECTION.......................................................6

3.2 REDUCE NRB DURING DATA COLLECTION ......................................................7

3.3 REDUCE NRB AFTER DATA COLLECTION .........................................................8

3.3.1 BASIC NONRESPONSE BIAS ANALYSIS ....................................................9

3.3.2 NRB ANALYSIS IF RESPONSE RATE IS LESS THAN 70% .....................10

4. SUMMARY................................................................................................................................24
LIST OF FIGURES

FIGURE 1. EXAMPLE RESPONSE RATES IN WEST REGION, USA, DURING DATA 

COLLECTION ................................................................................................................8

FIGURE 2. PRELIMINARY PLANS FOR PIAAC NRB ANALYSIS.............................................9

LIST OF TABLES

TABLE 1. SAMPLE DISTRIBUTION OF UNIT RESPONDENTS VS. ELIGIBLES FOR 

THE ALL BACKGROUND QUESTIONNAIRE, BY KEY

CHARACTERISTICS: 2003.........................................................................................12

TABLE 2. ESTIMATES OF UNIT NRB FOR THE ALL BACKGROUND

QUESTIONNAIRE, BY KEY CHARACTERISTICS: 2003 .......................................14

TABLE 3. WEIGHTED RESPONSE RATES FOR THE ALL BACKGROUND 

QUESTIONNAIRE, BY RESPONSE CELL: 2003......................................................16

TABLE 4. MULTIVARIATE ANALYSIS OF RESPONSE INDICATORS: 2003 ......................17

TABLE 5. PERCENTAGE DISTRIBUTION OF SAMPLE CASES AT EACH 

WEIGHTING STEP FOR THE ALL BACKGROUND QUESTIONNAIRE, BY 

KEY CHARACTERISTICS: 2003................................................................................17

TABLE 6. CORRELATIONS OF DEMOGRAPHIC VARIABLE WITH THE OUTCOME 

STATISTIC ...................................................................................................................19

TABLE 7. DISTRIBUTION OF BQ INCOMPLETES, BY DISPOSITION CODE 

CATEGORY..................................................................................................................19

TABLE 8. HOUSEHOLD RACE FOR SCREENER INCOMPLETES WITH NIR DATA, 

BY REGION..................................................................................................................21

TABLE 9. DIFFERENCES BETWEEN THE LAST 10% AND FIRST 90% OF 

RESPONDENTS, BY DOMAIN ..................................................................................22

TABLE 10. AVERAGE DIFFERENCES BETWEEN RESPONDENTS AND 

NONRESPONDENTS NEEDED TO ACHIEVE VARIOUS AMOUNTS OF 

BIAS IN OUTCOME STATISTICS .............................................................................23
4

PIAAC

REDUCING NONRESPONSE BIAS AND

PRELIMINARY NONRESPONSE BIAS ANALYSIS

MARCH 2010

1. INTRODUCTION AND OVERVIEW

This document addresses the issue of bias in PIAAC outcome statistics and contains an overview of the survey 

goals to reduce nonresponse bias (NRB) and preliminary plans for conducting NRB analysis. The Consortium will 

produce detailed guidelines for the NRB analysis after the field test is completed and comments are collected from 

countries on their lessons learned from the field test. 

Section 2 provides a brief background on why reducing NRB is critical in producing high-quality estimates for 

PIAAC. Section 3 contains a summary of plans and procedures that are critical to reducing bias in the outcome 

statistics and preliminary plans to conduct the NRB analysis, including examples using data from similar literacy 

surveys.

The PIAAC Technical Standards and Guidelines (TSG) includes plans and procedures that should be put in place 

before, during, and after main study data collection to improve response rates and help reduce the potential for 

NRB. More specifically, the TSG require each country to:

 Identify sources of NRB,

 Reduce potential for NRB before and during data collection,

 Evaluate NRB throughout the design and data collection process,

 Evaluate NRB after data collection, and

 Adjust for biases through nonresponse adjustment weighting steps.

Sections 3.1 and 3.2 contain brief summaries of procedures included in the TSG that countries should use to make 

plans to reduce nonresponse bias to the extent possible before and during data collection. At the conclusion of the 

data collection, countries are required to conduct a comprehensive NRB analysis to help guide the most effective 

NRB adjustment to the sampling weights. This will help reduce the NRB to the maximum extent possible. 

Countries with low response rates (see below) are then asked to evaluate the potential for remaining bias using final 

adjusted weights.

The NRB analyses conducted prior to weighting are referred to as the basic analysis and are described in Section 

3.3.1. Countries will be required to conduct a more extensive NRB analysis for each stage of data collection

(screener, BQ/JRA, and assessment) with less than an 80% response rate and if the overall response rate is less than 

70%. Section 3.3.2 provides preliminary plans for conducting the extended NRB analysis, including some examples 

using similar competency surveys.

The focus of this document is on unit response rate (not item response rate). Refer to the TSG for details on the 

item response rate goals and analyses.
5

2. PROBABILITY SAMPLES AND UNBIASED ESTIMATES

Probability sampling is essential for PIAAC for two main reasons. First, probability sampling contains a set of 

designs that leads to a variety of unbiased sampling approaches that allow analysts to generalise the results to the 

target population. Second, measures of precision related to survey estimates can be computed only under a 

probability design. The production of unbiased estimates assumes that every eligible adult in the target population 

would have a nonzero chance of selection (exclusion rate is equal to zero) in a national sample, and would be 

located, agree to participate in the study, and respond to all survey items (100% response rate). In practice, these 

circumstances are not realised in any survey population. Missing data occurs when; 1) subsets of the target 

population are excluded from the sampling frame, 2) some of the adults selected in the sample are not contacted or 

refuse to participate (referred to as unit nonresponse), 3) they fail to respond to a particular survey item (referred to 

as item nonresponse), or 4) because data collected from the sampled adults is contaminated (and thus not useful) or 

lost during or after the data collection phase. The statistical theory behind unbiased survey estimates and associated 

precision levels are not true anymore when data is missing for the units selected into the probability sample.

NRB (due to missing data for selected adults) can be substantial when two conditions hold: 1) the response rate is 

relatively low and 2) the difference between the competency levels of respondents and those of nonrespondents is 

relatively large. This is reflected in the following deterministic NRB formula:

Bias( ) (1 )( ) R R R NRy W   Y Y ,

where WR is the proportion of respondents, YRis the mean outcome for respondents, and YNR is the mean outcome 

for nonrespondents.

Similarly, undercoverage bias (due to exclusions) can be substantial if 1) undercoverage rate is high, and 2) the 

difference in competency levels between adults included in the sample and those excluded from the frame is 

relatively large. PIAAC TSG sets the maximum allowable exclusion rate at 5% to guard against high 

undercoverage bias in PIAAC estimates.

Given the relationships between bias and undercoverage and response rates, countries must keep the exclusion rates 

low and implement procedures to reduce the potential for NRB and attain high response rates.

An alternative model of nonresponse assumes each sampled person has a certain propensity to respond, and NRB in 

a characteristic is a function of the covariance between the response propensity and the characteristic: 

Bias( )

yp

R

y

p



 ,

where σyp is the covariance between the outcome variable and response propensity, and p is the mean response 

propensity. Based on this model, NRB is present if missingness is related to competency, as measured PIAAC.

It is well known that NRB can be reduced to some unknown extent through sample weighting if competency is 

correlated with auxiliary variables, and auxiliary variables are correlated with response propensity. Weighting 

assumes response probabilities are constant within every group created for weight adjustment, the competency 

score has zero variance within each group, and response propensity is uncorrelated with competency. It is known 

that these assumptions are not correct, and the impact of weight adjustments is limited to the number of variables 

(correlated with competency) available for nonresponse adjustment. Also, it is not possible to measure the exact 

departure from these assumptions since competency levels of nonrespondents are not known. It is, therefore,

necessary to evaluate NRB when response rates are low.
6

3. PIAAC DATA QUALITY GOAL

REDUCE NRB BEFORE, DURING, AND AFTER DATA COLLECTION

There are several ways to reduce the potential for NRB. First and foremost is to plan and implement field 

procedures that obtain a high level of cooperation. Section 3.1 provides a brief overview of steps that countries 

should take to plan and implement effective survey operation procedures to reduce the potential for bias.

Second, it is also critical to monitor the field when data collection is in progress and monitor the distribution of the 

sample during data collection to ensure steps are taken to reduce the potential for bias as much as possible. Section 

3.2 contains a brief summary of these steps.

Finally, countries should search for auxiliary data that can be used to evaluate the potential for bias and that could 

be used to reduce the impact of nonresponse on survey estimates. Countries should evaluate the potential for 

remaining bias after weighting adjustments are completed if response rate falls below 70%. Section 3.3 describes 

the procedures that need to be followed after the data collection is completed. 

3.1 REDUCE NRB BEFORE DATA COLLECTION

No single factor leads to obtaining high response rates in any survey, especially large-scale national surveys. As 

mentioned above, countries need to focus on a number of critical factors that are key to obtaining reliable data and 

highest response rates possible.

One of the most critical steps in improving response rates is to hire the most qualified staff for operating the survey. 

First, countries should attempt to hire interviewers with experience, good people skills, enthusiasm, perseverance,

and language skills necessary to conduct PIAAC interviews. Countries should hire some senior interviewers to 

travel to areas with high nonresponse if necessary. In addition, countries should hire supervisors with experience 

and knowledge to mentor, motivate, and monitor interviewers. There should be a sufficient number of interviewers 

and supervisors to conduct the field activities, taking into account the data collection period.

A related critical aspect is establishing effective plans for training the staff. The goal is to ensure that all field staff 

understand PIAAC so well that they can focus on obtaining quality data and high response rates, not just focusing 

on the mechanics of the work. Countries should provide thorough initial training, including importance of PIAAC, 

presentation of study to respondents, and techniques for convincing reluctant respondents to participate. In addition, 

countries should plan to provide ongoing training via telephone and email. Countries should train supervisors on 

initial case assignment and reassignment strategies for potential/actual nonresponse.

Community/respondent outreach is another critical component in increasing response rates. Countries should plan 

to develop an outreach plan, design attractive/cohesive set of materials (e.g., letters, brochures, etc.). Countries 

should make sure to produce sufficient amount of each material. They should obtain media coverage: 

newspaper/journal articles about PIAAC that interviewers can show/give to respondents. Various endorsements

should be obtained from well-known and well-respected authorities and agencies.

Interviewers/supervisors should be trained on effective use of materials. In addition, countries should design a 

website that could be used by respondents for reference and to legitimise the survey. Countries should establish 

toll-free telephone number/hotline—to answer questions and concerns raised by survey participants.

The timing of various activities is also very critical. Introductory materials should be mailed 7-10 days prior to 

respondent contact. Nonresponse conversion letters should be used for reluctant or hard-to-contact/reach 

respondents in a timely manner.

Each country should develop a non-interview report (NIR) form to collect information about nonrespondents (see 

Standard 10.3.7 in the TSG). The form can be paper and pencil or automated, and it should contain, at minimum, 
7

demographic data on the non-interviewed person, the strength of the refusal (if applicable), problems encountered, 

comments, and the likelihood of conversion. The NIR can aid in future interview attempts and possibly provide 

information for a NRB analysis. 

The National Sampling Managers (NSMs) should develop plans for monitoring the sample during data collection to 

ensure the resulting sample has the lowest NRB attainable. NSMs should monitor the sample and increase response 

rates for groups that have potential to increase NRB (i.e. identify pockets of nonresponse and use methods to bring

in initial nonrespondents that are not like respondents [see Section 3.2, Figure 1]). NSMs are encouraged to use the 

field test process to develop and test their monitoring programs for the main survey. 

A critical task that should be completed prior to the start of data collection is identifying and collecting correlated 

auxiliary variables that are known with a high degree of accuracy for both respondents and nonrespondents. 

Evaluating and reducing NRB requires auxiliary variables, available for all sampled units, that are correlated with 

competency and response propensity. The correlation with competency could be measured using past adult 

competency or similar studies; variables hypothesised to be correlated with literacy should also be included. A 

segmentation analysis (see section 3.2) could be used to determine correlation with response propensity. NSMs are 

encouraged to use the field test data to help identify a set of candidate auxiliary variables for evaluating and 

reducing NRB. More discussion can be found in the sampling Plan Part II forms, specifically form W-1.

The Cross-Cultural Survey Guidelines website (http://ccsg.isr.umich.edu/datacoll.cfm) has additional suggestions 

for improving response rates and reducing NRB. 

3.2 REDUCE NRB DURING DATA COLLECTION

National Project Managers (NPMs) should promote close and timely supervision/monitoring and conduct scheduled 

weekly telephone conference calls with interviewers. They should keep the pressure on performance and allow for 

timely feedback. Countries should use Consortium forms or country-specific automated reports that allow for 

regular/frequent monitoring of case completion. Reports must be based on daily interviewer transmission of 

interview/assessment data and current contact attempt and disposition code information.

It is important that countries seek and be able to implement new ideas or experiences used successfully by other 

survey organisations/countries prior to and during data collection. It is always helpful to encourage 

interviewers/supervisors to make suggestions for improving response rates throughout the data collection period.

Countries should use OTRS to ask for ad hoc advice and use quality control monitoring calls to discuss national 

issues, concerns, and solutions dealing with respondent cooperation. Countries are encouraged to share with the 

Consortium novel approaches for obtaining respondent cooperation that can be shared with other countries.

The NSMs should use the planned monitoring procedures (as described in Section 3.1) to monitor response rates 

and sample yield according to key demographics, known to be associated with competency. This will help countries 

identify potential shortfalls for specific subgroups that could result in biased estimates. The goal is to reach 

subgroups or areas with low response and use various approaches to improve response rates for these subgroups

(areas). The first step is to identify pockets of potential NRB through modelling, for example, segmentation 

modelling with local area-level auxiliary variables (smallest geography for which reliable auxiliary data is 

available, or use registry data when available). A typical segmentation modelling approach involves a classification 

tree algorithm that uses a chi-square test of independence to identify auxiliary variables that best define subgroups 

with differential response rates. The subgroups with low response rates, or areas containing these subgroups, can 

then be targeted for followup efforts to address the potential for NRB.

Figure 1 shows an example of a classification tree used for the West region during a U.S. adult competency 

household survey data collection period.
8

W e st

74%

% < 9th g rad e educatio n

Low

67%

M ed iu m

77%

H ig h

81%

MSA statu s % Poverty

% not speak English w e ll

Non-MSA

78%

M S A

66%

Low

88%

H ig h

80%

Low

67%

H ig h

60%

M inority concentratio n

Low

70%

H ig h

50%

Figure 1. Example response rates in West region, USA, during data collection

From Figure 1, it is evident that during data collection in the West there was a low response rate (50%) in segments 

with a high minority concentration, a high percentage of the population not speaking English well, located in 

metropolitan statistical areas (MSAs), and having a low percentage of the population with less than a ninth grade 

education. To the extent that these variables were related to literacy, the low response rate in these areas indicated

potential for NRB.

3.3 REDUCE NRB AFTER DATA COLLECTION 

As mentioned in Section 1, all countries are required to conduct the basic NRB analysis described in Section 3.3.1. 

Countries will be required to conduct a more extensive NRB analysis for all stages of data collection (screener, 

BQ/JRA, and assessment) with less than an 80% response rate and for the overall sample if the overall response 

rate is less than 70%. Figure 2 shows the overall plan for PIAAC NRB analysis. The remainder of this section 

describes the preliminary plans for various analyses mentioned in the figure.
9

Figure 2. Preliminary plans for PIAAC NRB analysis

3.3.1 Basic NRB Analysis 

The main goal is to conduct a comprehensive NRB analysis to help guide the most effective NRB adjustments to 

the weights. The basic NRB analysis uses auxiliary variable estimates for survey respondents and nonrespondents. 

The auxiliary variables may be available on the sampling frame or from a previous data collection stage (e.g. 

screener data for the BQ/JRA analysis), can come from an external source that can be matched to each sampled 

unit, or could be observational data on respondents and nonrespondents collected during data collection, assuming 

the data is of sufficient quality. The auxiliary variables must be available for all eligible units and should be related 

to competency. 

An advantage of the basic analysis is that they could be performed using data that are readily available for both 

respondents and nonrespondents. However, they are only beneficial if auxiliary variables are highly correlated with 

competency.

The following techniques can be used:

 Use a chi-square test to compare the distribution of auxiliary variables (correlated with competency) for

respondents and nonrespondents. Deviations between the auxiliary survey estimates and higher quality 

external source estimates may indicate a potential for bias in the survey estimates of competency levels, 

to the extent that the auxiliary variables are related to the key statistics;

 Compare response rates for different subgroups; 

 Use a classification tree algorithm to identify subgroups with low response rates; and/or 

 Use logistic regression to model the relationship between response status and the auxiliary variables. 

PIAAC Preliminary NRB Analysis

RR > 70% RR<70%

Basic Analysis

Descriptive Model-based

Use the results to compute 

non-response adjusted 

weights

Analysis using base 

weights and auxiliary

variables

Basic Analysis

Descriptive Model-based

Extended Analysis

Descriptive Model-based

Analysis using base 

weights, non-response 

adjusted weights, BQ/JRA, 

and auxiliary variables
10

Sample Weighting Adjustments

At the conclusion of the basic NRB analysis, countries will be asked to put together a list of variables found to be 

significantly related to competency and response status. The goal is to use these variables in developing NRB

adjustments. Such adjustments will reduce NRB to the extent that the auxiliary variables and the competency

measures are correlated, and the auxiliary variables and response propensity are correlated. 

3.3.2 NRB Analysis if Response Rate is Less Than 70%

The basic descriptive analysis is a good initial assessment of NRB and is essential in identifying effective weighting 

variables. However, it has its limitations. The analysis does not reflect the effect of weighting adjustments on NRB, 

and the extent of bias remaining after nonresponse adjustments are conducted.

Brief descriptions of some possible analyses are provided below. Note that multiple analyses to assess NRB are

necessary because each analysis has its own limitations. Together, they will provide an insight into the patterns and 

potential for bias.

Comparison of estimates before and after weight adjustments are done - The basic analysis described in section 

3.3.1 comparing the base-weighted estimates for respondents to the base-weighted full sample estimates can be 

extended to include a third stage – the respondent estimates using weights adjusted for nonresponse and calibrated 

to control totals. In other words, estimates from the full sample can be compared to estimates from the respondents 

before and after weighting adjustments. If the full sample estimates are closer to the final estimates than the base 

weighted estimates, this indicates bias in the auxiliary variables was reduced through the weighting process.

Comparison of weighted estimates to external totals - Another extended analysis is to compare estimates from 

PIAAC to estimates from external source. The PIAAC estimates should be produced using the final weights that 

have been adjusted for nonresponse and calibrated to control totals. Care should be taken to choose external source 

estimates that measure the same characteristic for a similar time period. The external source estimates will also be 

subject to error, and the variance of these estimates should be taken into account when making this comparison. 

Correlations between weighting adjustment variables and competency measures - The analyses described thus far 

rely on auxiliary variables and do not directly measure bias in the competency estimates. As mentioned earlier, bias 

in the auxiliary variables is indicative of bias in the competency estimates to the extent that the auxiliary variables 

and competency estimates are correlated. Thus, correlations between the auxiliary variables and competency data 

can be computed to evaluate this relationship. For variables used in the weighting adjustments, a low correlation 

with literacy implies that using the variable in the weighting adjustments did little to reduce NRB. On the other 

hand, a high correlation with literacy implies a potentially high reduction in NRB. For variables not used in the 

weighting adjustments, a high correlation with literacy indicates potential bias in the literacy estimates. This can be 

used to inform weighting decisions for future surveys. 

The disadvantage of using correlations to evaluate NRB is that the correlations are based on respondents only and 

the relationship between competency and the auxiliary variables might be different for nonrespondents. However, 

this is not as much a concern if the relationship can be confirmed using outside sources.

Comparison of estimates using alternative weighting adjustments - For this evaluation, an auxiliary variable is recalibrated to known totals, and estimates of the key statistics are compared before and after the re-weighting. Reweighting can be useful as an evaluation tool when:

 The variable was not used in weighting (because it was of a low quality);

 The variable is correlated with the outcome measure; and

 The variable is correlated with response propensity.
11

Any differences between estimates using the official survey weights and the re-weighted weights reflect 

noncoverage as well as NRB, but if there is not a large change in the estimates, this is further confirmation that 

NRB may not be a concern. 

Analysis of variables collected during data collection - BQ variables - The descriptive analyses described earlier 

assess the potential for bias in some statistics that may not necessarily be highly related to competency. Therefore, 

the next step of the analysis should be to evaluate bias on the key statistics or closely related variables. For PIAAC, 

the key statistics are scores measuring competency in the components of the assessments that will be available later. 

However, before literacy scores are available, an analysis can be performed with BQ/JRA variables believed to be 

closely related to literacy, such as questions on measuring habits (newspapers or magazines), for example. Analyses 

of such variables may show patterns of bias that cannot be effectively eliminated by the auxiliary variables used in 

weighting.

Analysis of variables collected during data collection - Disposition codes - Disposition codes contain information 

on reasons for nonresponse. For this analysis, distributions of sampled persons with known characteristics related to 

outcome (i.e. the language barrier cases) should be examined. For example, the demographic distribution of

language barrier cases can be compared to other eligible persons using auxiliary data, and interview data. 

Analysis of variables collected during data collection - NIR - The NIR forms identify observable demographic 

information and reasons for nonresponse that are not captured in the disposition codes. The NIR forms can 

potentially indicate whether the reasons for nonresponse are related to competency estimates and suggest ways to 

improve response rates for future surveys.

Comparison of late or “hard-to-contact” respondents to early respondents - This analysis is helpful for evaluating

the potential for NRB for differences that cannot be captured through adjusting for known demographics. 

Significant differences between the competency levels of early and late respondents imply that the bias could have 

been potentially reduced through focused efforts to obtain the late respondents. However, to the extent that 

differences between early and late respondents reflect differences between respondents and nonrespondents, the 

findings indicate that some level of NRB might still be present (which depends on the magnitude of nonresponse 

rate). The validity of this assumption, that late respondents are similar to nonrespondents, depends largely on the 

strategies used in the late data collection effort. For many surveys this may not be a valid assumption, but it seems 

reasonable for a household survey such as PIAAC where multiple contacts are usually required to gain cooperation.

Calculation of the range of potential bias - How can we evaluate the potential for bias remaining after weighting if 

nonrespondents are very different from respondents within the weighting classes? The range of potential bias can

be evaluated using a deterministic NRB formula that calculates how different respondent and nonrespondents 

competency estimates need to be to bias the overall competency estimate by varying degrees. For example, one can 

show that for a response rate of 70% (WR = 0.7), the mean of respondents and nonrespondents would have to differ 

by 50 points (YR -YNR = 50) to bias the overall estimate by 15 points (Bias( y

R

) = 15). 

Other analysis under review - The Consortium is currently reviewing other NRB analysis procedures, such as the 

R indicator (Schouten, Cobben, and Bethlehem, 20091), for possible inclusion in the NRB plan for PIAAC.

The following provides a summary of the steps included in the NRB analysis when response rate falls below 70%.

Step 1 of 3: Identify Variables

1.1 Identify auxiliary variables from the frame or external sources.

1.2 Identify informative BQ variables correlated with competency.

1.3 Identify other informative variables collected during data collection.

 

1

Schouten B., Cobben F., and Bethlehem J. Indicators for the representativeness of survey response, Survey Methodology, June 2009. 
12

Step 2 of 3: Basic NRB Analysis

2.1 Conduct basic descriptive and model-based analysis.

2.2 Finalise variables selected for weighting adjustments.

2.3 Conduct nonresponse adjustments and compute final sampling weights.

Step 3 of 3: Extended NRB Analysis

3.1 Conduct extended descriptive and model-based analysis using auxiliary data.

3.2 Conduct analysis of informative BQ variables.

3.3 Conduct analysis of disposition codes.

3.4 Conduct analysis of NIR (information not available in disposition codes).

Example: The following information and tables are summaries extracted from the NRB analysis conducted for the 

Adult Literacy and Lifeskills Survey (ALL) 2003: U.S. NRB Analysis http://nces.ed.gov/pubs2009/2009063.pdf, 

and other unpublished NRB analyses conducted for similar adult competency household surveys. 

The steps carried out in the following NRB analysis are summarized below.

Step 1 of 3: Identify variables from the frame or external sources

Step 1.1 The following variables were extracted from Census 2000 county-level data, and Census 2000 block 

group level data. 

 Census variables available for local areas (where respondents and nonrespondents resided)

 Urban/rural status

 Majority own/rent

 Average household size

 Percentage of persons 25 years or older with less than a high school diploma

 Percentage of persons speaking English not well or not at all

 Percentage below 150% poverty

Step 1.2 The following variables were extracted from the BQ.

 Percentage born in the United States

 Percentage with high school diploma

 Percentage never receiving remedial help in school

 Percentage agreeing with the statement: I enjoyed math in school

 Percentage speaking English most often at home

 Percentage employed

 Percentage reading letters, memos, or emails at least once a week as part of main job



Step 1.3 Disposition codes and NIR variables were collected after the end of data collection.

Step 2 of 3: Basic NRB analysis – Tables 1 through 4 provide examples of analysis conducted for ALL. 

Table 1 shows the results of chi-square tests conducted for ALL. The chi-square tests indicated a significant 

relationship of response status to region, MSA status, the percentage below 150% of poverty, age, sex, and 

race/ethnicity. These variables were included in the nonresponse adjustments to reduce the potential NRB to the 

extent that these auxiliary variables were related to literacy.
13

Table 1. Sample distribution of unit respondents vs. eligibles for the ALL background questionnaire, by 

key characteristics: 2003

Analysis variable

Respondents

percent

Standard

error

Eligibles

percent

Standard

error

Chi-square 

statistic p value

Region 

Northeast 15.5 2.50 18.3 2.36 31.95 0.000

Midwest 22.9 2.54 22.1 2.24 † †

South 35.0 1.66 34.9 1.50 † †

West 26.6 2.47 24.7 2.21 † †

Metropolitan Statistical Area status 

Non-Metropolitan Statistical Area 22.7 2.63 21.8 2.40 7.26 0.007

Metropolitan Statistical Area 77.3 2.63 78.2 2.40 † †

Locale1

Urban 64.6 2.67 65.3 2.45 3.78 0.12

Suburban 13.0 2.00 12.4 1.83 † †

Rural 22.4 2.11 22.3 2.02 † †

Majority own/rent 

Rent 25.7 2.48 25.2 2.40 1.14 0.287

Own 74.3 2.48 74.8 2.40 † †

Average household size 

less than or equal to 2.2 18.1 1.87 18.6 1.81 4.73 0.081

greater than 2.2 and less than or equal to 

2.8 

52.9 2.50 51.5 2.80 † †

greater than 2.8 29.1 2.92 29.9 3.10 † †

Percent less than high school 

less than or equal to 10% 33.1 2.71 33.4 2.73 2.7 0.257

greater than 10% and less than or equal to 

20% 

30.9 1.81 31.4 1.75 † †

greater than 20% 36.1 3.01 35.2 3.09 † †

Percent limited English proficient 

0% 29.9 2.68 29.9 2.51 0.06 0.963

greater than 0% and less than 2% 34.7 3.19 34.6 3.09 † †

greater than or equal to 2% 35.4 3.78 35.5 3.64 † †

Percent below 150% of poverty 

less than or equal to 10% 30.7 2.65 31.5 2.79 8.59 0.011

greater than 10% and less than or equal to 

30% 

44.9 3.27 45.4 3.27 † †

greater than 30% 24.4 2.95 23.1 2.76 † †
14

Table 1. Sample distribution of unit respondents vs. eligibles for the ALL background questionnaire, by 

key characteristics: 2003 (Continued)

Analysis variable

Respondents

percent

Standard

error

Eligibles

percent

Standard

error

Chi-square 

statistic p value

Age 

16–25 23.7 1.38 22.4 1.25 17.92 0

26–35 21.2 0.80 20.8 0.77 † †

36–45 22.2 0.84 22.9 0.79 † †

46–65 32.9 1.29 33.9 1.16 † †

Sex 

Male 48.1 1.08 49.6 0.99 22.77 0

Female 51.9 1.08 50.4 0.99 † †

Race/ethnicity2

Hispanic 14.3 1.79 13.3 1.61 19.11 0

Black, non-Hispanic 11.1 1.82 10.2 1.62 † †

Other, non-Hispanic 74.6 2.08 76.4 1.94 † †

† Not applicable.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

Table 2 contains estimates of bias and relative bias for the subgroup percentages presented in Table 1. Bias was 

calculated using the deterministic NRB formula in section 2. A t test was performed to determine whether the NRB

was significantly different from zero. Another measure, the relative bias, was computed as the bias divided by the 

estimate from the eligible sample. The relative bias is a measure of the size of the bias compared to the eligible 

sample estimate. 

The results of the analysis in Table 2 were consistent with the chi-square analysis in Table 1. It also showed that 

even though there were statistically significant results as described above, the relative magnitude of the differences 

was small. The relative bias estimates were all less than 10 percent, except in the Northeast, which was 15 percent.

Table 2. Estimates of unit NRB for the ALL background questionnaire, by key characteristics

2003 Analysis variable

Respondents

percent

Eligibles

percent

Bias 

Percent

Estimate relative bias

Standard

error p value

Region 

Northeast 15.5 18.3 -2.8 0.53 0.000 -15.3

Midwest 22.9 22.1 0.8 0.61 0.187 3.6

South 35.0 34.9 0.1 0.49 0.842 0.3

West 26.6 24.7 1.8 0.48 0.001 7.3

Metropolitan Statistical Area status 

Non-Metropolitan Statistical Area 22.7 21.8 0.9 0.34 0.010 4.1

Metropolitan Statistical Area 77.3 78.2 -0.9 0.34 0.010 -1.2

Locale1

Urban 64.6 65.3 -0.7 0.38 0.074 -1.1

Suburban 13.0 12.4 0.6 0.37 0.108 4.8

Rural 22.4 22.3 0.1 0.31 0.747 0.4
15

Table 2. Estimates of unit NRB for the ALL background questionnaire, by key characteristics

(Continued)

2003 Analysis variable

Respondents

percent

Eligibles

percent

Bias 

Percent

Estimate relative bias

Standard

error p value

Majority own/rent 

Rent 25.7 25.2 0.5 0.44 0.298 2.0

Own 74.3 74.8 -0.5 0.44 0.298 -0.7

Average household size 

less than or equal to2.2 18.1 18.6 -0.5 0.39 0.215 -2.7

greater than 2.2 and less than or equal to 2.8 52.9 51.5 1.4 0.65 0.044 2.7

greater than 2.8 29.1 29.9 -0.9 0.6 0.155 -3.0

Percent less than high school 

less than or equal to10% 33.1 33.4 -0.3 0.53 0.540 -0.9

greater than 10% and less than or equal to 

20% 30.9 31.4 -0.5 0.48 0.284 -1.6

greater than 20% 36.1 35.2 0.9 0.52 0.112 2.6

Percent limited English proficient 

0% 29.9 29.9 # 0.56 0.977 #

greater than 0% and less than 2% 34.7 34.6 0.1 0.69 0.841 0.3

greater than or equal to 2% 35.4 35.5 -0.1 0.55 0.826 -0.3

Percent below 150% of poverty 

less than or equal to10% 30.7 31.5 -0.8 0.52 0.129 -2.5

greater than 10% and less than or equal to 

30% 44.9 45.4 -0.5 0.58 0.379 -1.1

greater than 30% 24.4 23.1 1.3 0.38 0.001 5.6

Age 

16–25 23.7 22.4 1.3 0.35 0.001 5.8

26–35 21.2 20.8 0.4 0.25 0.106 1.9

36–45 22.2 22.9 -0.6 0.36 0.088 -2.6

46–65 32.9 33.9 -1 0.41 0.016 -2.9

Sex 

Male 48.1 49.6 -1.5 0.31 0.000 -3.0

Female 51.9 50.4 1.5 0.31 0.000 3.0

Race/ethnicity2

Hispanic 14.3 13.3 0.9 0.29 0.003 6.8

Black, non-Hispanic 11.1 10.2 0.9 0.31 0.009 8.8

Other, non-Hispanic 74.6 76.4 -1.8 0.32 0.000 -2.4

# Rounds to zero

1 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban 

clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.

2 All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other 

include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial.

NOTE: Detail may not sum to totals because of rounding.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

The analyses in Tables 1 and 2 were useful in explaining the relationship of response status to each auxiliary 

variable individually. A classification tree algorithm and a logistic regression model were used to evaluate the 

relationship between response status and multiple auxiliary variables. The results of the classification tree 

algorithm are shown in Table 3. The algorithm was similar to that described in section 2 and used chi-square tests 

to divide the sample into subgroups that best explain differential response rates. As Table 3 shows, 22 cells were 
16

formed with weighted response rates ranging from 58.7% to 97.4%. The subgroups defined by the classification 

tree algorithm could be used as nonresponse adjustment cells in the weighting adjustment.

Table 3. Weighted response rates for the ALL background questionnaire, by response cell: 2003

Response

cell Description

Weighted

response rate 

(%)

Overall 79.7

1 Northeast, Non–Metropolitan Statistical Area (MSA) 88.9

2 Northeast, MSA, Average household size less than or equal to 2.8 68.1

3 Northeast, MSA, Average household size greater than 2.8 58.7

4

Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size less than 

or equal to 2.2 79.8

5

Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size greater 

than 2.2 and less than or equal to 2.8 91.1

6

Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size greater 

than 2.8 78.0

7 Midwest and South, Hispanic or non-Hispanic Black, Female, Age 16-45 94.4

8 Midwest and South, Hispanic or non-Hispanic Black, Female, Age 46-65 82.0

9 Midwest and South; Other, non-Hispanic; Age 16-35; Male; Rent 70.0

10 Midwest and South; Other, non-Hispanic; Age 16-35; Male; Own 82.1

11 Midwest; Other, non-Hispanic; Female; Age 16-25 97.4

12 Midwest; Other, non-Hispanic; Female; Age 26-35 90.2

13 South; Other, non-Hispanic; Female; Percent limited English proficient less than 2% 86.0

14

South; Other, non-Hispanic; Female; Percent limited English proficient greater than or equal to 

2% 70.3

15

Midwest; Other, non-Hispanic; Female; Age 36-65; Percent below 150% of poverty less than or 

equal to 30% 77.3

16

South; Other, non-Hispanic; Female; Age 36-65; Percent below 150% of poverty less than or 

equal to 30% 71.5

17

Midwest and South; Other, non-Hispanic; Age 36-65; Percent below 150% of poverty greater 

than 30% 84.7

18 West, Rent 88.6

19 West, Own, Age 16-25 88.7

20 West, Own, Age 26-45 79.6

21 West, Own, Age 46-65, Urban or Suburban 90.5

22 West, Own, Age 46-65, Rural 78.5

NOTE: The response cells were formed using the Chi-square Automated Interaction Detector.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

Logistic regression models are also useful in identifying significant effects on response propensity. Response status 

was used as the binary dependent variable and the set of auxiliary variables in Table 1 as the predictors. An F test 

was performed for each auxiliary variable to determine if it was significantly related to response propensity, after 

accounting for the other auxiliary variables. Results are shown in Table 4. Average household size, age, 

race/ethnicity, sex, and region are shown to be potentially valuable variables to include in the nonresponse 

adjustments. While this analysis was limited to main effects, interaction terms can also be included.
17

Table 4. Multivariate analysis of response indicators: 2003

Test F statistic

Degrees of freedom

Numerator Denominator p value

Overall fit 3.89 21 10 0.016

Average household size 6.77 2 29 0.004

Majority own/rent 0.32 1 30 0.578

Percent less than high school 0.44 2 29 0.647

Percent limited English proficient 2.08 2 29 0.143

Locale1 0.12 2 29 0.887

Age 4.47 3 28 0.011

Race/ethnicity210.79 2 29 0.000

Sex 19.43 1 30 0.000

Metropolitan Statistical Area status 0.03 1 30 0.871

Percent below 150% of poverty 3.22 2 29 0.055

Region 6.65 3 28 0.002

1 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban 

clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.

2 All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other 

include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

Step 3 of 3: Extended analysis – Tables 5 through 10 provide examples of extended NRB analysis as 

conducted for ALL and other adult competency surveys

Table 5 shows an example of the comparison of estimates before and after weighting adjustments. The following t

test comparisons were made: 

 Comparison of percentage distributions from BQ base weights for the total eligible sample of persons 

with the BQ base weights for the BQ respondents only to check for differences due to nonresponse to 

the BQ

 Comparison of percentage distributions from BQ base weights for the total eligible sample of persons 

with that from the BQ nonresponse adjusted weights for respondents to check for differences even after 

the nonresponse adjustment process to the BQ

 Comparison of percentage distributions from BQ nonresponse adjusted weights for respondents with 

that from the BQ raked weights for respondents to check for differences that may have been introduced 

through the initial raking procedure

 This analysis showed that, through the nonresponse adjustment, the potential for bias was reduced (pvalues moved from significant to nonsignificant) in all domains shown below, with the exception of 

two domains relating to average household size. These two domains show significant bias created by 

the nonresponse adjustments. While the differences in the percentage distributions of these domains 

remain statistically significant even after the raking adjustment, they are not large enough to be 

meaningful.
18

Table 5. Percentage distribution of sample cases at each weighting step for the ALL background 

questionnaire, by key characteristics: 2003

Total sample 

%

BQ respondents 

Nonresponse

adjustment Initial raking 

% p value1 % p value1 % p value2

Total 100.0 100.0 † 100.0 † 100.0 †

Region 

Northeast 18.4 15.6 0.000 18.6 0.355 19.0 0.866

Midwest 22.1 22.9 0.180 21.9 0.355 23.0 0.728

South 34.9 35.0 0.842 34.9 0.825 35.0 0.716

West 24.7 26.6 0.001 24.7 0.895 23.0 0.413

Metropolitan Statistical Area status 

Non-Metropolitan Statistical Area 21.8 22.7 0.012 21.9 0.558 18.0 0.100

Metropolitan Statistical Area 78.2 77.3 0.012 78.1 0.558 82.0 0.100

Locale3

Urban 65.3 64.6 0.080 65.1 0.448 68.0 0.097

Suburban 12.4 13.0 0.103 12.4 0.849 11.0 0.198

Rural 22.3 22.4 0.749 22.5 0.566 20.0 0.075

Majority own/rent 

Rent 25.2 25.7 0.294 25.0 0.486 26.0 0.254

Own 74.8 74.3 0.294 75.0 0.486 74.0 0.254

Average household size 

less than or equal to 2.2 18.6 18.1 0.225 18.1 0.213 18.1 0.946

greater than 2.2 and less than or equal 

to 2.8 51.5 52.9 0.051 53.3 0.003 52.5 0.257

greater than 2.8 30.0 29.1 0.157 28.7 0.005 29.4 0.381

Percent less than high school 

less than or equal to 10% 33.4 33.1 0.542 33.1 0.569 32.7 0.431

greater than 10% and less than or 

equal to 20% 31.4 30.9 0.292 31.4 0.887 31.3 0.937

greater than 20% 35.2 36.1 0.123 35.5 0.349 36.0 0.524

Percent limited English proficient 

0% 29.9 29.9 0.977 30.0 0.953 28.8 0.088

greater than 0% and less than 2% 34.6 34.7 0.840 35.4 0.162 35.5 0.756

greater than or equal to 2% 35.5 35.4 0.825 34.7 0.052 35.8 0.314
19

Table 5. Percentage distribution of sample cases at each weighting step for the ALL background 

questionnaire, by key characteristics: 2003 (Continued)

Total sample 

%

BQ respondents 

Nonresponse

adjustment Initial raking 

% p value1 % p value1 % p value2

Percent below 150% of poverty 

less than or equal to 10% 31.5 30.7 0.137 31.6 0.912 32.0 0.629

greater than 10% and less than or 

equal to 30% 45.4 44.9 0.372 45.3 0.682 45.4 0.814

greater than 30% 23.1 24.4 0.002 23.2 0.452 22.7 0.585

1 The p value and SE are for tests of difference between estimated percentage of current step and total sample (BQ base weights). 

2 The p value and SE are for tests of difference between estimated percentage of current and previous weighting steps. 

3 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban 

clusters; ‘rural’ if the largest percentage was the rural population. The terms ‘urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms.

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003.

Table 6 gives an example of correlations between auxiliary variables and competency score measures for an adult 

competency survey. The percentage aged 25+ with less than a high school education was found to be highly 

correlated with literacy score. If this variable was also related to response propensity, using it in nonresponse 

adjustments should reduce the potential NRB of literacy estimates.

Table 6. Correlations of demographic variable with the outcome statistic

Variable

Absolute correlation with outcome statistic

<0.1 0.1-0.2 0.2-0.3 0.3-0.4 >0.4

Percent aged 25+ with less than high school 

education X

Percent of adults speaking Spanish at home and 

English not well or not at all X

Percent below 150% of poverty X

Median Income X

Percent who rent X

Average household size X

Age X

Percent minority X

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

Table 7 provides an example of the analysis of disposition codes to evaluate NRB in an adult competency survey. 

The distribution of cases with a BQ disposition code of language problem or mental disability was compared to the 

distribution of the other nonrespondents. Chi-square tests were processed to determine if there was a relationship 

between select demographic variables and the disposition code groups (literacy-related or other) for incompletes. 

Significant differences in the distribution by region, MSA status, and age were found between the two groups. 

These variables were used in the weighting adjustments, which are expected to reduce the potential bias from 

literacy-related nonresponse.
20

Table 7. Distribution of BQ incompletes, by disposition code category

Domain Literacy-related2other

Total incompletes 355 5,191

Domain

Incompletes1

Literacy- related Statistic p value 2 percent OTHER percent

Total incompletes 100 100

Region 10 0.006

Northeast 16 17

Midwest 18 22

South 21 38

West 45 23

MSA status 9 0.002

Non-MSA 9 18

MSA 91 82

Age 23 0.000

16-29 16 20

30-49 37 43

50-69 23 27

70+ 24 11

Gender 3 0.067

Male 47 53

Female 53 47

Race/ethnicity 2 0.244

Hispanic 17 11

NH Black only 6 9

Other 77 79

† Not applicable

1 Nonrespondents

2 Literacy-related reasons for nonresponse at the BQ stage consist of language problems and mental disabilities only.

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

An example of an analysis of NIR data for an adult competency survey is provided in Table 8. The table 

summarizes data collected on the NIR regarding the household race of screener incompletes, as observed or 

gathered from neighbour information. The base-weighted distribution is shown overall and by region. For 

households where race was determined, Hispanic households were less common among screener incompletes 

compared to the population proportion. This suggests a possible increase in response rates from accommodations 

made for Spanish speakers, such as bilingual interviewers.
21

Table 8. Household race for screener incompletes with NIR data, by region

Household race Overall

Region

Northeast Midwest South West

Total 4,276 1,327 819 1,571 559

Not ascertained 2,197 548 535 804 310

Percent Percent Percent Percent Percent

Total 100 100 100 100 100

White 73 73 84 70 67

Black 10 6 7 15 7

Hispanic 4 3 4 3 8

Asian 4 4 1 2 9

NH/PI10 # # # 1

AI/AN20 # # 0 #

Other 0 1 # 0 1

Cannot determine 10 14 5 10 8

SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

Table 9 shows an example of a comparison of literacy estimates for late versus early respondents. Calculations 

were done using marginal maximum likelihood (MML) means. The results indicate a small but significant bias in 

Statistic 2 if focused followup efforts (as describe in section 2) had not been successful in obtaining responses from 

respondents at the end of the data collection period (i.e. if data about respondents at the end of the data collection 

period had been missing). This suggests some small level of NRB may be present to the extent that nonrespondents 

are similar to respondents at the end of the data collection period.
22

Table 9. Differences between the last 10% and first 90% of respondents, by domain

Domain

Estimate of population proportion Statistic 1 Statistic 2 Statistic 3

Proportion

among last 

10%

Proportion

among first 

90% p value

Relative 

difference p value

Relative 

difference p value

Relative 

difference p value

Overall 1.00 1.00 † 0.00 0.92 -0.03 0.01* 0.00 0.80

Region

Northeast 0.20 0.19 0.82 0.00 0.89 -0.03 0.12 0.01 0.79

Midwest 0.18 0.23 0.03 0.03 0.22 0.01 0.58 0.03 0.08

South 0.36 0.36 0.95 0.01 0.64 -0.02 0.36 0.00 0.96

West 0.27 0.21 0.02 -0.03 0.13 -0.06 0.01* -0.01 0.61

MSA status

Non-MSA 0.13 0.19 0.04 0.00 0.95 -0.02 0.38 0.02 0.38

MSA 0.87 0.81 0.04 0.00 0.97 -0.03 0.01* 0.00 0.88

Gender

Male 0.53 0.48 0.01* 0.01 0.50 -0.02 0.22 0.00 0.94

Female 0.47 0.52 0.01* -0.01 0.70 -0.04 0.01* 0.00 0.84

Race/ethnicity

Hispanic 0.17 0.12 0.00* -0.02 0.35 -0.07 0.00* -0.01 0.55

NH Black only 0.11 0.11 0.51 0.01 0.57 0.02 0.41 0.06 0.03

Other 0.73 0.77 0.04 0.01 0.40 -0.01 0.20 0.01 0.51

Educational attainment

Less than high school 0.18 0.19 0.33 -0.05 0.01* -0.07 0.00* -0.02 0.33

High school or equivalent 0.28 0.31 0.10 -0.02 0.50 -0.04 0.02 0.00 0.82

More than high school 0.54 0.50 0.06 0.00 0.99 -0.03 0.01* -0.01 0.25

Age category

16-29 0.25 0.25 0.68 -0.01 0.75 -0.02 0.22 0.01 0.79

30-49 0.42 0.39 0.06 -0.01 0.35 -0.04 0.0* 0.00 0.87

50-69 0.25 0.25 0.92 0.02 0.25 0.00 0.85 0.00 0.96

70+ 0.09 0.11 0.07 -0.03 0.53 -0.10 0.01* -0.07 0.11

Country of birth

US 0.80 0.86 0.00 0.01 0.44 -0.01 0.32 0.01 0.29

Other 0.20 0.14 0.00 0.00 0.97 -0.05 0.04 0.00 0.97

† Not applicable

SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.
23

An example of the calculation of the range of potential bias in literacy estimates is shown in Table 10. The table 

shows the difference between respondent and nonrespondents literacy scores that would bias the population 

estimate by b = 3, 5, 10, or 15 points. The required difference between respondents and nonrespondents was 

computed as described earlier in section 3.3.2. For the overall sample, literacy scores of respondents and 

nonrespondents would have to differ by about 8 points on average to bias the national estimate by 3 points and 

would need to differ by 39 points to bias the national estimate by 15 points.

Table 10. Average differences between respondents and nonrespondents needed to achieve various 

amounts of bias in outcome statistics

Domain

Percent of population1

represented by 

nonrespondents2

Average point difference between 

respondents and nonrespondents needed to 

bias estimated scores by

3 5 10 15 

Total 38 8 13 26 39

Region

Northeast 43 7 12 23 35

Midwest 35 9 14 28 43

South 38 8 13 26 40

West 38 8 13 26 39

MSA status

Non-MSA 33 9 15 30 45

MSA 40 8 13 25 38

Average household size

≤ 2.42 39 8 13 26 39

(2.42, 2.8] 39 8 13 26 39

> 2.8 38 8 13 27 40

Percent less than high school

≤ 10.4% 43 7 12 23 35

(10.4%, 20.3%] 39 8 13 26 38

(20.3%, 32%] 35 8 14 28 42

> 32% 31 10 16 32 48

Percent speaking Spanish but not English

0% 39 8 13 26 38

(0%, 28%] 40 8 13 25 38

> 28% 34 9 15 29 44

Percent below 150% of poverty

≤ 10.7% 43 7 12 23 35

(10.7%, 20%] 40 8 13 25 38

(20%, 33.3%] 35 9 14 29 43

> 33.3% 29 10 17 34 52

Median income

≤ 28,400 28 11 18 36 53

(28,400, 37,850] 34 9 15 29 44

(37,850, 52,100] 40 8 13 25 38

> 52,100 44 7 11 23 34
24

Table 10. Average differences between respondents and nonrespondents needed to achieve various 

amounts of bias in outcome statistics (Continued)

Domain

Percent of population1

represented by 

nonrespondents2

Average point difference between 

respondents and nonrespondents needed to 

bias estimated scores by

3 5 10 15 

Percent rent

≤ 16% 42 7 12 24 36

(16%, 31%] 37 8 13 27 40

(31%, 59%] 37 8 14 27 41

> 59% 35 9 14 29 43

Low response rate CHAID cells

Cell 1359 5 8 17 25

Cell 2451 6 10 20 30

Cell 3550 6 10 20 30

1 The population excludes language problems and mental disabilities.

2 Nonrespondents to the Screener and BQ.

3 Median income greater than 52,100, average household size 2.42 or less, Northeast region

4 Median income 37,850 - 52,100, percent less than high school greater than 20.3, Northeast region

5 Median income greater than 52,100, average household size 2.42 - 2.8, South or West region

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003.

4. SUMMARY

This document presented the main goals of PIAAC for reducing NRB throughout the survey process. A series of 

procedures for implementation before and during the data collection were described to help increase the response 

rates with the goal of reducing NRB to the extent possible. Preliminary plans are provided to conduct basic NRB

analysis to allow an effective weighting adjustment plan that reduces the bias to the maximum extent possible. 

Then extended analyses are described with the goal of assessing the bias remaining in the estimates after 

nonresponse adjustment is completed. In addition to assessing the remaining bias in outcome statistics, the results 

of the NRB analysis can be used to inform analysts of data limitations, and can also help in future rounds of the 

survey in defining the target population and improving data collection methods, weighting, and estimation 

approaches.

The PIAAC Consortium is continuing its research and evaluation of various NRB analysis methods and welcomes 

comments and suggestions from participating countries on such analyses.