# Appendix F

PIAAC Reducing Nonresponse Bias and Preliminary Nonresponse Bias Analysis

## PIAAC

**Reducing Nonresponse Bias**  **a nd**  **Preliminary Nonresponse Bias Analysis** 

**PIAAC_NRB Paper_10March10 TABLE OF**  **CONTENTS**

1. INTRODUCTION AND OVERVIEW ................................ ................................ ........................ 4 2. PROBABILITY SAMPLES AND UNBIASED ESTIMATES ................................ ................... 5 3. PIAAC DATA QUALITY GOAL REDUCE NRB BEFORE, DURING, AND AFTER DATA COLLECTION ................................ ................................ ................................ ................. 6 3.1 REDUCE NRB BEFORE DATA COLLECTION ................................ ....................... 6 3.2 REDUCE NRB DURING DATA COLLECTION ................................ ...................... 7 3.3 REDUCE NRB AFTER DATA COLLECTION ................................ ......................... 8 3.3.1 BASIC NONRESPONSE BIAS ANALYSIS ................................ .................... 9 3.3.2 NRB ANALYSIS IF RESPONSE RATE IS LESS THAN 70% ..................... 10 4. SUMMARY ................................ ................................ ................................ ................................ 24**LIST OF FIGURES**

FIGURE 1 . EXAMPLE RESPONSE RATES IN WEST REGION, USA, DURING DATA

COLLECTION ................................ ................................ ................................ ................ 8 FIGURE 2. PRELIMINARY PLANS FOR PIAAC NRB ANALYSIS ................................ ............. 9 **LIST OF TABLES** TABLE 1. SAMPLE DISTRIBUTION OF UNIT RESPONDENTS VS. ELIGIBLES FOR

THE ALL BACKGROUND QUESTIONNAIRE, BY KEY CHARACTERISTICS: 200 3 ................................ ................................ ......................... 12 TABLE 2. ESTIMATES OF UNIT NRB FOR THE ALL BACKGROU ND QUESTIONNAIRE, BY KEY CHARACTERISTICS: 2 003 ................................ ....... 14 TABLE 3. WEIGHTED RESPONSE RATES FOR THE ALL BACKGROUND

QUESTIONNAIRE, BY RE SPONSE CELL: 2003 ................................ ...................... 16 TABLE 4. MULTIVARIATE ANALYSIS OF RESPONSE INDICATORS: 2003 ...................... 17 TABLE 5. PERCENTAGE DISTRIBUTION OF SAMPLE CASES AT EACH

WEIGHTING STEP FOR THE ALL BACKGROUND QUESTIONNAIRE, BY KEY CHARACTERISTIC S: 2003 ................................ ................................ ................ 17 TABLE 6. CORRELATIONS OF DEMOGRAPHIC VARIABLE WITH THE OUTCOME

STATISTIC ................................ ................................ ................................ ................... 19 TABLE 7. DISTRIBUTION OF BQ INCOMPLETES, BY DISPOSITION CODE

CATEGORY ................................ ................................ ................................ .................. 19 TABLE 8. HOUSEHOLD RACE FOR S CREENER INCOMPLETES WITH NIR DATA,

BY REGION ................................ ................................ ................................ .................. 21 TABLE 9. DIFFERENCES BETWEEN THE LAST 10% AND FIRST 90% OF

RESPONDENTS, BY DOMAIN ................................ ................................ .................. 22 TABLE 10. AVERAGE DIFFERENCES BETWEEN RESPONDENTS AND

NONRESPONDENTS NEEDED TO ACHIEVE VARIOUS AMOUNTS OF BIAS IN OUTCOME STATISTICS ................................ ................................ ............. 234 **PIAAC REDUCING**  **NONRESPONSE BIAS AND PRELIMINARY NONRESPO** **NSE BIAS ANALYSIS**

**MARCH 2010 1.**  **INTRODUCTION AND OVE RVIEW**  This document addresses the issue of bias in PIAAC outcome statistics and contains an overview of the survey goals to reduce nonresponse bias (NRB ) and preliminary plans for conducting NRB analysis. The Consortium will produce detailed guidelines for the NRB analysis after the field test is completed and comments are collected from countries on their lessons learned from the field test. Section 2 provides a brief background on why reducing NRB is critical in producing high-quality estimates for PIAAC. Section 3 contains a summary of plans and procedures that are critical to reducing bias in the outcome statistics and preliminary plans to conduct the NRB analysis , including examples using data from similar literacy surveys. The PIAAC Technical Standards and Guidelines (TSG) includes plans and procedures that should be put in place before, during, and after main study data collection to improve response rates and help reduce the potential for NRB . More specifically, the TSG require each country to:

 Identify sources of NRB,  Reduce potential for NRB before and during data collection,  Evaluate NRB throughout the design and data collection process ,  Evaluate NRB after data collection, and  Adjust for biases through nonresponse adjustment weighting steps . Sections 3.1 and 3.2 contain brief summaries of procedures included in the TSG that countries should use to make plans to reduce nonresponse bias to the extent possible before and during data collection. At the conclusion of the data collection, countries are required to conduct a comprehensive NRB analysis to help guide the most effective NRB adjustment to the sampling weights. This will help reduce the NRB to the maximum extent possible. Countries with low response rates (see below) are then asked to evaluate the potential for remaining bias using final adjusted weights. The NRB analyses conducted prior to weighting are referred to as the basic analysis and are described in Section 3.3.1 . Countries will be required to conduct a more extensive NRB analysis for each stage of data collection (screener, BQ/JRA, and assessment) with less than an 80% re sponse rate and if the overall response rate is less than 70%. Section 3.3.2 provides preliminary plans for conducting the extended NRB analysis , including some examples using similar competency surveys . The focus of this document is on unit response rate (not item response rate). Refer to the TSG for details on the item response rate goals and analyses .5 **2. PR** **OBABILITY SAMPLES AND UNBIASED ESTIMATES** Probability sampling is essential for PIAAC for two main reasons. First, probability sampling contains a set of designs that leads to a variety of unbiased sampling approaches that allow analysts to generalise the results to the target population. Second, measures of precision related to survey estimates can be computed only under a probability design. The production of unbiased estimates assumes that every eligible adult in the target population would have a nonzero chance of selection (exclusion rate is equal to zero) in a national sample, and would be located, agree to participate in the study, and respond to all survey items (100% response rate) . In practice, these circumstances are not realised in any survey population. Missing data occurs when; 1) subsets of the target population are excluded from the sampling frame, 2) some of the adults selected in the sample are not contacted or refuse to participate (referred to as unit nonresponse), 3) they fail to respond to a particular survey item (referred to as item nonresponse), or 4) because data collected from the sam pled adults is contaminated (and thus not useful) or lost during or after the data collection phase. The statistical theory behind unbiased survey estimates and associated precision levels are not true anymore when data is missing for the units selected into the probability sample. NRB (due to missing data for selected adults) can be substantial when two conditions hold: 1) the response rate is relatively low and 2) the difference between the competency levels of respondents and those of nonrespondents is relatively large. This is reflected in the following deterministic NRB formula: 

### Bias( ) (1 ) ( ) R R R N R 

### y W Y Y    , where W

R is the proportion of respondents, Y R is the mean outcome for respondents, and Y NR is the mean outcome for nonrespondents. Similarly, undercoverage bias (due to exclusions) can be substantial if 1) undercoverage rate is high, and 2) the difference in competency levels between adults included in the sample and those excluded from the frame is relatively large. PIAAC TSG sets the maximum allowable exclusion rate at 5 % to guard against high undercoverage bias in PIAAC estimates. Given the relationship s between bias and undercoverage and response rates, countries must keep the exclusion rates low and implement procedures to reduce the potential for NRB and attain high response rates . An alternative model of nonresponse assumes each sampled person has a certain propensity to respond, and NRB in a characteristic is a function of the covariance between the response propensity a nd the characteristic: 

### Bias( ) yp R 

### y p   , where σ

yp is the covariance between the outcome variable and response propensity, and p is the mean response propensity. Based on this model, NRB is present if missingness is related to competency, as measured PIAAC . Iti s well known that NRB can be reduced to some unknown extent through sample weighting if competency is correlated with auxiliary variables, and auxiliary variables are correlated with response propensity. Weighting assumes response probabilities are constant within every group created for weight adjustment, the competency score has zero variance within each group , and response propensity is uncorrelated with competency. Iti s known that these assumptions are not correct, and the impact of weight adjustments is limited to the number of variables (correlated with competency) available for nonresponse adjustment . Also, it is not possible to measure the exact departure from these assumptions since competency levels of nonrespondents are not known . Iti s , therefore , necessary to evaluate NRB when response rates are low .6 **3. PIAAC DATA QUALITY GOAL REDUCE**  **NRB BEFORE, DURING, AND AFTER DATA COLLECTION**  There are several ways to reduce the potential for NRB . First and foremost is to plan and implement field procedures that obtain a high level of cooperation. Section 3.1 provides a brief overview of steps that countries should take to plan and implement effective survey operation procedures to reduce the potential for bias. Second, it is also critical to monitor the field when data collection is in progress and monitor the distribution of the sample during data collection to ensure steps are taken to red uce the potential for bias as much as possible. Section 3.2 contains a brief summary of these steps. Finally, countries should search for auxiliary data that can be used to evaluate the potential for bias and that could be used to reduce the impact of nonresponse on survey estimates. Countries should evaluate the potential for remaining bias after weighting adjustments are completed if response rate falls below 70% . Section 3.3 de scribes the procedure s that need to be followed after the data collection is completed. **3.1 REDUCE**  **NRB BEFORE DATA COLLECTI** **ON** No single factor leads to obtaining high response rates in any survey, especially large-scale national surveys. As mentioned above, countries need to focus on a number of critical factors that are key to obtaining reliable data and highest response rates possible . One of the most critical steps in improving response rates is to hire the most qualified staff for operating the survey. First, countries should attempt to hire interviewers with experience, good people skills, enthusiasm, perseverance , and language skills necessary to conduct PIAAC interviews. Countries should hire some senior interviewers to travel to areas with high nonresponse if necessary. In addition, countries should hire supervisors with experience and knowledge to mentor, motivate , and monitor interviewers . There should be a sufficient number of interviewers and supervisors to conduct the field activities, taking into account the data collection period. A related critical aspect is establishing effective plans for training the staff. The goal is to ensure that all field staff understand PIAAC so well that they can focus on obtaining quality data and high response rates, not just focusing on the mechanics of the work. Countries should provide thorough initial training, including importance of PIAAC, presentation of study to respondents, and techniques for convincing reluctant respondents to participate . In addition, countries should plan to provide ongoing training via telephone and email . Countries should t rain supervisors on initial case assignment and reassignment strategies for potential/actual nonresponse . Community/respondent outreach is another critical component in increasing response rates. Countries should plan to develop an outreach plan, design attractive/cohesive set of materials (e.g., letters, brochures, etc.) . Countries should make sure to produce sufficient amount of each material . They should obtain media coverage: newspaper/journal articles about PIAAC that interviewers can show/give to respondents . Various endorsements should be obtained from well-known and well-respected authorities and agencies. Interviewers/supervisors should be trained on effective use of materials . In addition, countries should design a website that could be used by respondents for reference and to legitimise the survey. Countries should establish toll-free telephone number/hotline

—to answer questions and concerns raised by survey participants . The timing of various activities is also very critical . Introductory materials should be mailed 7 - 10 days prior to respondent contact . Nonresponse conversion letters should be used for reluctant or hard-to -contact/reach respondents in a timely manner. Each country should develop a non-interview report (NIR) form to collect information about nonrespondents (see Standard 10.3.7 in the TSG). The form can be paper and pencil or automated, and it should contain, at minimum,7

demographic data on the non-interviewed person, the strength of the refusal (if applicable), problems encountered, comments, and the likelihood of conversion. The NIR can aid in future interview attempts and possibly provide information for a NRB analysis. The National Sampling Managers (N SMs) should develop plans for monitoring the sample during data collection to ensure the resulting sample has the lowest NRB attainable. NSMs should monitor the sample and increase response rates for groups that have potential to increase NRB (i.e. identify pockets of nonresponse and use methods to bring in initial nonrespondents that are not like respondents [ see Section 3.2, Figure 1 ]). NSMs are encouraged to use the field test process to develop and test their monitoring programs for the main survey. A critical task that should be completed prior to the start of data collection is identifying and collecting correlated auxiliary variables that are known with a high degree of accuracy for both respondents and nonrespondents . Evaluating and reducing NRB requires auxiliary variables , available for all sampled units, that are correlated with competency and response propensity. The correlation with competency could be measured using past adult competency or similar studies; variables hypothesi sed to be correlated with literacy should also be included. A segmentation analysis (see section 3.2) could be used to determine correlation with response propensity. NSMs are encouraged to use the field test data to help identify a set of candidate auxiliary variables for evaluating and reducing NRB . More discussion can be found in the sampling Plan Part II forms , specifically form W- 1. The Cross -Cultural Survey Guidelines website (http://ccsg.isr.umich.edu/datacoll.cfm) has additional suggestions for improving response rates and reducing NRB . **3.2 REDUCE**  **NRB DURING DATA COLLECTI** **ON** National Project Managers (NPMs) should promote close and timely supervision/monitoring and conduct scheduled weekly telephone conference calls with interviewers . They should keep the pressure on performance and allow for timely feedback. Countries should use Consortium forms or country- specific automated reports that allow for regular/frequent monitoring of case completion . Reports must be based on daily interviewer transmission of interview/assessment data and current contact attempt and disposition code information . Iti s important that countries seek and be able to implement new ideas or experiences used successfully by other survey organi s ations/countries prior to and during data collection . Iti s always helpful to encourage interviewers/supervisors to make suggestions for improving response rates throughout the data collection period. Countries should use OTRS to ask for ad hoc advice and use quality control monitoring calls to discuss national issues, concerns, and solutions dealing with respondent cooperation . Countries are encouraged to share with the Consortium novel approaches for obtaining respondent cooperation that can be shared with other countries . The NSMs should use the planned monitoring procedures (as described in Section 3.1) to monitor response rates and sample yield according to key demographics, known to be associated with competency. This will help countries identify potential shortfalls for specific subgroups that could result in biased estimates. The goal is to reach subgroups or areas with low response and use various approaches to improve response rates for these subgroups (areas) . The first step i s to identify pockets of potential NRB through modelling, for example, segmentation modelling with local area-level auxiliary variables (smallest geography for which reliable auxiliary data is available , or use registry data when available) . A typical segmentation modelling approach involves a classification tree algorithm that uses a chi-square test of independence to identify auxiliary variables that best define subgroups with differential response rates. The subgroups with low response rates, or areas containing these subgroups, can then be targeted for followup efforts to address the potential for NRB . Figure 1 shows an example of a classification tree used for the West region during a U . S . adult competency household survey data collection period.8 **W e st**  **74% %**  **< 9th**  **g rad**  **e educat**  **io n**  **Low 67%**  **M e d i**  **u m 77%**  **H i g h**  **81% %**  **Pover ty**  **MSA s**  **tatu s** 

**% not**  **s peak**  **Engl is**  **h w e**  **ll Non-**  **MSA 78%**  **M S A**  **66% Low**  **88% H i**  **g h 80%** 

**Low 67%**  **H i g h**  **60% M i**  **nor ity**  **c oncent**  **rati o n**  **Low 70%**  **H i g h**  **50% Figure**  **1 .**  **Example response rates in West**  **region, USA, during data collection**  From Figure 1, it is evident that during data collection in the West there was a low response rate (50%) in segments with a high minority concentration, a high percentage of the population not speaking English well, located in metropolitan statistical areas (MSAs), and having a low percentage of the population with less than a ninth grade education. To the extent that these variables were related to literacy, the low response rate in these areas indicated potential for NRB . **3.3 REDUCE**  **NRB AFTER DATA COLLECTIO** **N** As mentioned in S ection 1 , all countries are required to conduct the basic NRB analysis described in Section 3.3.1 . Countries will be required to conduct a more extensive NRB analysis for all stages of data collection (screener, BQ/JRA, and assessment) with less than an 80% response rate and for the overall sample if the overall response rate is less than 70%. Figure 2 shows the overall plan for PIAAC NRB analysis. The remainder of this section describe s the preliminary plans for various analyses mentioned in the figure.9 **Figure 2. Preliminary plans for PIAAC NRB**  **analysis 3.3.1**  **Basic NRB**  **Analysis** The main goal is to conduct a comprehensive NRB analysis to help guide the most effective NRB adj ustments to the weights. The basic NRB analysis uses auxiliary variable estimates for survey respondents and nonrespondents. The auxiliary variables may be available on the sampling frame or from a previous data collection stage (e.g. screener data for the BQ/JRA analysis), can come from an external source that can be matched to each sampled unit, or could be observational data on respondents and nonrespondents collected during data collection, assuming the data is of sufficient quality. The auxiliary variables must be available for all eligible units and should be related to competency. An advantage of the basic analysis is that they could be performed using data that are readily available for both respondents and nonrespondents. However, they are only beneficial if auxiliary variables are highly correlated with competency. The following techniques can be used:

 Use a chi- square test to compare the distribution of auxiliary variables (correlated with competency) for respondents and nonrespondents. Deviations between the auxiliary survey estimates and higher quality external source estimates may indicate a potential for bias in the survey estimates of competency levels , to the extent that the auxiliary variables are related to the key statistics;  Compare response rates for different subgroups;  Use a classification tree algorithm to identify subgroups with low response rates; and/or  Use logistic regression to model the relationship between response status and the auxiliary variables. PIAAC Prelim inary

NRB Analysis RR >7 0% RR<70% Ba sic Analysis Descriptive Model-based Use the results to compute non-response adjusted we ights Analysis using base weights and auxiliary variables Ba sic Analysis Descriptive Model-based Extended Analysis Descriptive Model-based An alysis using base weights, non-response adjusted weights, BQ/JRA, and auxiliary

variables10

**Sample Weighting Adjustments** At the conclusion of the basic NRB analysis, countries will be asked to put together a list of variables found to be significantly related to competency and response status. The goal is to use these variables in developing NRB adjustments. Such adjustments will reduce NRB to the extent that the auxiliary variables and the competency measure s are correlated, and the auxiliary variables and response propensity are correlated. **3.3.2 NRB**  **Analysis if Response Rate is Less Than 70%** The basic descriptive analysis is a good initial assessment of NRB and is essential in identifying effective weighting variables. However, it has its limitations. The analysis does not reflect the effect of weighting adjustments on NRB , and the extent of bias remaining after nonresponse adjustments are conducted. Brief descriptions of some possible analyses are provided below. Note that m ultiple analyses to assess NRB are necessary because each analysis has its own limitations . Together, they will provide an insight into the patterns and potential for bias. **Com parison**  **of estimates**  **before and**  **after weight**  **adjustments are**  **done -**  The basic analysis described in section 3.3.1 comparing the base-weighted estimates for respondents to the base-weighted full sample estimates can be extended to include a third stage

– the respondent estimates using weights adjusted for nonresponse and calibrated to control totals. In other words, e stimates from the full sample can be com pared to estimates from the respondents before and after weighting adjustments. If the full sample estimates are closer to the final estimates than the base weighted estimates, this indicates bias in the auxiliary variables was reduced through the weighting process. **Com parison**  **of weighted**  **estimates to**  **external totals**  **-** Another extended analysis is to compare estimates from PIAAC to estimates from external source. The PIAAC estimates should be produced using the final weights that have been adjusted for nonresponse and calibrated to control totals. Care should be taken to choose external source estimates that measure the same characteristic for a similar time period. The external source estimates will also be subject to error, and the variance of these estimates should be taken into account when making this comparison. **Correlations between**  **weighting adjustment variables**  **and competency measures -**  The analyses described thus far rely on auxiliary variables and do not directly measure bias in the competency estimates. As mentioned earlier, bias in the auxiliary variables is indicative of bias in the competency estimates to the extent that the auxiliary variables and competency estimates are correlated. Thus, correlations between the auxiliary variables and competency data can be computed to evaluate this relationship. For variables used in the weighting adjustments, a low correlation with literacy implies that using the variable in the weighting adjustments did little to reduce NRB . On the other hand, a high correlation with literacy implies a potentially high reduction in NRB . For variables not used in the weighting adjustments, a high correlation with literacy indicates potential bias in the literacy estimates. This can be used to inform weighting decisions for future surveys. The disadvantage of using correlations to evaluate NRB is that the correlations are based on respondents only and the relationship between competency and the auxiliary variables might be different for nonrespondents. However, this is not as much a concern if the relationship can be confirmed using outside sources. **Com parison**  **of estimates using alternative weighting adjustments -**  For this evaluation, an auxiliary variable is re calibrated to known totals, and estimates of the key statistics are compared before and after the re-weighting. Re weighting can be useful as an evaluation tool when:

 The variable was not used in weighting (because it was of a low quality);  The variable is correlated with the outcome measure; and  The variable is correlated with response propensity.11

Any differences between estimates using the official survey weights and the re-weighted weights reflect noncoverage as well as NRB , but if there is not a large change in the estimates, this is further confirmation that NRB may not be a concern. **Analysis of**  **variables collected**  **during data**  **collection -**  **BQ variables**  **-** The de scriptive analyses described earlier assess the potential for bias in some statistics that may not necessarily be highly related to competency. Therefore, the next step of the analysis should be to evaluate bias on the key statistics or closely related variables. For PIAAC, the key statistics are scores measuring competency in the components of the assessments that will be available later. However, before literacy scores are available, an analysis can be performed with BQ/JRA variables believed to be closely related to literacy, such as questions on measuring habits (newspapers or magazines), for example. Analyses of such variables may show patterns of bias that cannot be effectively eliminated by the auxilia ry variables used in weighting. **Analysis of**  **variables collected**  **during data**  **collection -**  **D isposition**  **codes -**  Disposition codes contain information on reasons for nonresponse. For this analysis, distributions of sampled persons with known characteristics related to outcome (i.e. the language barrier cases) should be examined. For example, the demographic distribution of language barrier cases can be compared to other eligible persons using auxiliary data, and interview data. **Analysis of**  **variables collected**  **during data**  **collection -**  **NIR -**  The NIR forms identify observable demographic information and reasons for non response that are not captured in the disposition codes. The NIR forms can potentially indicate whether the reasons for nonresponse are related to competency estimates and suggest ways to improve response rates for future surveys. **Com parison of late or “h** **ard**

**- to** **- contact” respondents to early respondents**  **-** This analysis is helpful for evaluating the potential for NRB for differences that cannot be captured through adjusting for known demographics. Significant differences between the competency levels of early and late respondents imply that the bias could have been potentially reduced through focused efforts to obtain the late respondents. However, to the extent that differences between early and late respondents reflect differences between respondents and nonrespondents, the findings indicate that some level of NRB might still be present (which depends on the magnitude of nonresponse rate) . The validity of this assumption, that late respondents are similar to nonrespondents, depends largely on the strategies used in the late data collection effort. For many surveys this may not be a valid assumption, but it seems reasonable for a household survey such as PIAAC where multiple contacts are usually required to gain cooperation. **Calculation of the range of potential bias -**  How can we evaluate the potential for bias remaining after weighting if nonrespondents are very different from respondents within the weighting classes? The range of potential bias can be evaluated using a deterministic NRB formula that calculates how different respondent and nonrespondents competency estimates need to be to bias the overall competency estimate by varying degrees. For example, one can show that for a response rate of 70 % ( W R = 0.7), the mean of respondents and nonrespondents would have to differ by 50 points ( Y

R - Y NR = 50) to bias the overall estimate by 15 points ( Bias( y

R) = 15). **Other analysis**  **under review**  **-** The Consortium is currently reviewing other NRB analysis procedures, such as the R indicator ( Schouten, Cobben, and Bethlehem, 2009 1 ) , for possible inclusion in the NRB plan for PIAAC. The following provides a summary of the steps included in the NRB analysis when response rate falls below 70% . **Step 1 of 3:**  **Identify Variables** 1.1 Identify auxiliary variables from the frame or external sources . 1.2 Identify informative BQ variables correlated with competency. 1.3 Identify other informative variables collected during data collection.

1 Schouten B., Cobben F., and Bethlehem J. I ndicators for the representativeness of survey response, Survey Methodology, June 2009.12

**Step 2 of 3: Basic**  **NRB Analysis**  2.1 Conduct basic descriptive and model-based analysis . 2.2 Finalise variables selected for weighting adjustments . 2.3 Conduct nonresponse adjustments and compute final sampling weights . **Step 3 of 3: Extended**  **NRB Analysis**  3.1 Conduct extended descriptive and model-based analysis using auxiliary data . 3.2 Conduct analysis of informative BQ variables . 3.3 Conduct analysis of disposition codes . 3.4 Conduct analysis of NIR (information not available in disposition codes) . Example: The following information and tables are summaries extracted from the NRB analysis conducted for the Adult Literacy and Lifeskills Survey (ALL) 2003: U.S. NRB Analysis http://nces .ed.gov/pubs2009/2009063.pdf , and other unpublished NRB analyses conducted for similar adult competency household surveys. The steps carried out in the following NRB analysis are summarized below. **Step 1 of 3 : Identify**  **variables from the frame or external sources Step 1.1**  The following variables were extracted from Census 2000 county-level data, and Census 2000 block

group level data.  Census variables available for local areas (where respondents and nonrespondents resided)  Urban/rural status  Majority own/rent  Average hous ehold size  Percentage of persons 25 years or older with less than a high school diploma  Percentage of persons speaking English not well or not at all  Percentage below 150% poverty **Step 1.2** The following variables were extracted from the BQ.

 Percentage born in the United States  Percentage with high school diploma  Percentage never receiving remedial help in school  Percentage agreeing with the statement: I enjoyed math in school  Percentage speaking English most often at home  Percentage employed  Percentage reading letters, memos , or emails at least once a week as part of main job  **Step 1.3** Disposition codes and NIR variables were collected after the end of data collection. **Step 2 of 3: Basic NRB**  **analysis –**  **Tables 1 through 4 pr o** **vide examples of analysis conducted for ALL.**  Table 1 shows the results of chi-square tests conducted for ALL. The chi-square tests indicated a significant relationship of response status to region, MSA status, the percentage below 150% of poverty, age, sex, and race/ethnicity. These variables were included in the nonresponse adjustments to reduce the potential NRB to the extent that these auxiliary variables were related to literacy.13

**Table 1 .**  **Sample distribution of unit respondents vs. eligibles for the ALL background question naire, by** 

**key characteristics: 2003 Analysis variable**  **Respondents p**  **ercent Standard** 

**error Eligibles** 

**p ercent**  **Standard**

**error Chi** **- square** 

**s tatistic**  **p value**  **Region** Northeast 15.5 2.5 0 18.3 2.36 31.95 0 .000 Midwest 22.9 2.54 22.1 2.24 † † South 35 .0 1.66 34.9 1.5 0 † † West 26.6 2.47 24.7 2.21 † † **Me tropolitan Statistical Area status**  Non-Metropolitan Statistical Area 22.7 2.63 21.8 2.4 0 7.26 0.007 Metropolitan Statistical Area 77.3 2.63 78.2 2.4 0 † † **Locale 1**  Urban 64.6 2.67 65.3 2.45 3.78 0.12 Suburban 13 .0 2 .00 12.4 1.83 † † Rural 22.4 2.11 22.3 2.02 † † **Majority own/rent** Rent 25.7 2.48 25.2 2.4 0 1.14 0.287 Own 74.3 2.48 74.8 2.4 0 † † **Average household size** less than or equal to 2.2 18.1 1.87 18.6 1.81 4.73 0.081 greater than 2.2 and less than or equal to 2.8 52.9 2.5 0 51.5 2.8 0 † † greater than 2.8 29.1 2.92 29.9 3.1 0 † † **Percent less than high school** less than or equal to 10% 33.1 2.71 33.4 2.73 2.7 0.257 greater than 10% and less than or equal to 20% 30.9 1.81 31.4 1.75 † † greater than 20% 36.1 3.01 35.2 3.09 † † **Percent limited English proficient** 0% 29.9 2.68 29.9 2.51 0.06 0.963 greater than 0% and less than 2% 34.7 3.19 34.6 3.09 † † greater than or equal to 2% 35.4 3.78 35.5 3.64 † † **Percent below 150% of poverty** less than or equal to 10% 30.7 2.65 31.5 2.79 8.59 0.011 greater than 10% and less than or equal to 30% 44.9 3.27 45.4 3.27 † † greater than 30% 24.4 2.95 23.1 2.76 † †14

**Table 1 .**  **Sample distribution of unit respondents vs. eligibles for the ALL background questionnaire, by**

**key characteristics: 2003 (Continued)**  **Analysis variable Respondents**  **p ercent**  **Standard**

**error Eligibles** 

**p ercent**  **Standard**

**error Chi** **- square** 

**s tatistic**  **p value**  **Age** 16

–25 23.7 1.38 22.4 1.25 17.92 0 26

–35 21.2 0.8 0 20.8 0.77 † † 36

–45 22.2 0.84 22.9 0.79 † † 46

–65 32.9 1.29 33.9 1.16 † † **Sex** Male 48.1 1.08 49.6 0.99 22.77 0 Female 51.9 1.08 50.4 0.99 † † **Race/ethnicity 2**  Hispanic 14.3 1.79 13.3 1.61 19.11 0 Black, non-Hispanic 11.1 1.82 10.2 1.62 † † Other, non-Hispanic 74.6 2.08 76.4 1.94 † † † Not applicable. SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003. Table 2 contains estimates of bias and relative bias for the subgroup percentages presented in Table 1. B ias was calculated using the deterministic NRB formula in section 2. A t test was performed to determine whether the NRB was significantly different from zero. Another measure, the relative bias, was computed as the bias divided by the estimate from the eligible sample. The relative bias is a measure of the size of the bias compared to the eligible sample estimate. The results of the analysis in Table 2 were consistent with the chi- square analysis in Table 1. It also showed that even though there were statistically significant results as described above, the relative magnitude of the differences was small. The relative bias estimates were all less than 10 percent, except in the Northeast, which was 15 percent. **Table 2. Estimates of unit**  **NRB for the ALL background**  **qu esti** **onnaire, by key characteristics**

**2003 Analysis variable Respondents**  **percent Eligibles** 

**percent Bias**  **Percent relative bias**  **Estimate Standard** 

**error p**  **value Region**  Northeast 15.5 18.3 -2.8 0.53 0 .000 - 15.3 Midwest 22.9 22.1 0.8 0.61 0.187 3.6 South 35 .0 34.9 0.1 0.49 0.842 0.3 West 26.6 24.7 1.8 0.48 0.001 7.3 **Me tropolitan Statistical Area status**  Non-Metropolitan Statistical Area 22.7 21.8 0.9 0.34 0.01 0 4.1 Metropolitan Statistical Area 77.3 78.2 -0.9 0.34 0.01 0 - 1.2 **Locale 1**  Urban 64.6 65.3 -0.7 0.38 0.074 - 1.1 Suburban 13 .0 12.4 0.6 0.37 0.108 4.8 Rural 22.4 22.3 0.1 0.31 0.747 0.415

**Table 2. Estimates of unit**  **NRB for the ALL background**  **questionnaire, by key characteristics**

**(Continued) 2003 Analysis**  **variable Respondents**  **percent Eligibles** 

**percent Bias**  **Percent relative bias**  **Estimate Standard** 

**error p**  **value Majority own/rent**  Rent 25.7 25.2 0.5 0.44 0.298 2 .0 Own 74.3 74.8 -0.5 0.44 0.298 -0.7 **Average household size** less than or equal to2.2 18.1 18.6 -0.5 0.39 0.215 -2.7 greater than 2.2 and less than or equal to 2.8 52.9 51.5 1.4 0.65 0.044 2.7 greater than 2.8 29.1 29.9 -0.9 0.6 0.155 - 3 .0 **Percent less than high school** less than or equal to10% 33.1 33.4 -0.3 0.53 0.54 0 -0.9 greater than 10% and less than or equal to 20% 30.9 31.4 -0.5 0.48 0.284 - 1.6 greater than 20% 36.1 35.2 0.9 0.52 0.112 2.6 **Percent limited English proficient** 0% 29.9 29.9 # 0.56 0.977 # greater than 0% and less than 2% 34.7 34.6 0.1 0.69 0.841 0.3 greater than or equal to 2% 35.4 35.5 -0.1 0.55 0.826 -0.3 **Percent below 150% of poverty** less than or equal to10% 30.7 31.5 -0.8 0.52 0.129 -2.5 greater than 10% and less than or equal to 30% 44.9 45.4 -0.5 0.58 0.379 - 1.1 greater than 30% 24.4 23.1 1.3 0.38 0.001 5.6 **Age** 16

–25 23.7 22.4 1.3 0.35 0.001 5.8 26

–35 21.2 20.8 0.4 0.25 0.106 1.9 36

–45 22.2 22.9 -0.6 0.36 0.088 -2.6 46

–65 32.9 33.9 - 1 0.41 0.016 -2.9 **Sex** Male 48.1 49.6 - 1.5 0.31 0 .000 - 3 .0 Female 51.9 50.4 1.5 0.31 0 .000 3 .0 **Race**/ethnicity2 Hispanic 14.3 13.3 0.9 0.29 0.003 6.8 Black, non-Hispanic 11.1 10.2 0.9 0.31 0.009 8.8 Other, non-Hispanic 74.6 76.4 - 1.8 0.32 0 .000 -2.4 # Rounds to zero 1 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘ r

ural’ if the largest percentage was the rural population. The terms ‘ urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms. 2 All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003. The analyses in Tables 1 and 2 were useful in explaining the relationship of response status to each auxiliary variable individually. Aclassification tree algorithm and a logistic regression model were used to evaluate the relationship between response status and multiple auxiliary variables. The results of the classification tree algorithm are shown in Table 3. The algorithm was similar to that described in section 2 and used chi-square tests to divide the sample into subgroups that best explain differential response rates. As Table 3 shows, 22 cells were16

formed with weighted response rates ranging from 58.7 % to 97.4% . The subgroups defined by the classification tree algorithm could be used as nonresponse adjustment cells in the weighting adjustment. **Table 3 .**  **Weighted response rates for the ALL background questionnaire, by response cell: 2003 Response**  **cell Description**  **Weighted r** **esponse rate**  **(%)** Overall 79.7 1 Northeast, Non

–Metropolitan Statistical Area (MSA) 88.9 2 Northeast, MSA, Average household size less than or equal to 2.8 68.1 3 Northeast, MSA, Average household size greater than 2.8 58.7 4 Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size less than or equal to 2.2 79.8 5 Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size greater than 2.2 and less than or equal to 2.8 91.1 6 Midwest and South, Hispanic or non-Hispanic Black, Male, Average household size greater than 2.8 78 .0 7 Midwest and South, Hispanic or non-Hispanic Black, Female, Age 16 -45 94.4 8 Midwest and South, Hispanic or non-Hispanic Black, Female, Age 46 - 65 82 .0 9 Midwest and South; Other, non-Hispanic; Age 16 - 35; Male; Rent 70 .0 10 Midwest and South; Other, non-Hispanic; Age 16 - 35; Male; Own 82.1 11 Midwest; Other, non-Hispanic; Female; Age 16 -25 97.4 12 Midwest; Other, non-Hispanic; Female; Age 26 - 35 90.2 13 South; Other, non-Hispanic; Female; Percent limited English proficient less than 2% 86 .0 14 South; Other, non-Hispanic; Female; Percent limited English proficient greater than or equal to 2% 70.3 15 Midwest; Other, non-Hispanic; Female; Age 36 - 65; Percent below 150% of poverty less than or equal to 30% 77.3 16 South; Other, non-Hispanic; Female; Age 36 - 65; Percent below 150% of poverty less than or equal to 30% 71.5 17 Midwest and South; Other, non-Hispanic; Age 36 - 65; Percent below 150% of poverty greater than 30% 84.7 18 West, Rent 88.6 19 West, Own, Age 16 -25 88.7 20 West, Own, Age 26 -45 79.6 21 West, Own, Age 46 - 65, Urban or Suburban 90.5 22 West, Own, Age 46 - 65, Rural 78.5 NOTE: The response cells were formed using the Chi- square Automated I nteraction Detector. SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003. Logistic regression models are also useful in identifying significant effects on response propensity. Response status was used as the binary dependent variable and the set of auxiliary variables in Table 1 as the predictors. An F test was performed for each auxiliary variable to determine if it was significantly related to response propensity, after accounting for the other auxiliary variables. Results are shown in Table 4. Average household size, age, race/ethnicity, sex, and region are shown to be potentially valuable variables to include in the nonresponse adjustments. While this analysis was limited to main effects, interaction terms can also be included.17

**Table 4 .**  **Multivariate analysis of response indicators: 2003**

**Test F**  **statistic Degrees of**  **f reedom**  **p value**  **Numerator Denominator**  Overall fit 3.89 21 10 0.016 Average household size 6.77 2 29 0.004 Majority own/rent 0.32 1 30 0.578 Percent less than high school 0.44 2 29 0.647 Percent limited English proficient 2.08 2 29 0.143 Locale 1 0.12 2 29 0.887 Age 4.47 3 28 0.011 Race/ethnicity2 10.79 2 29 0 .000 Sex 19.43 1 30 0 .000 Metropolitan Statistical Area status 0.03 1 30 0.871 Percent below

150% of poverty 3.22 2 29 0.055 Region 6.65 3 28 0.002 1 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘ r

ural’ if the largest percentage was the rural population. The terms ‘ urbanized areas’, ‘urban clusters’, a nd ‘rural’ are Census Bureau-defined terms . 2 All adults of Hispanic origin are classified as Hispanic regardless of race. Those classified as Black are non-Hispanic Black only. Those classified as Other include non-Hispanics of all other races, including White, Asian, American Indian/Alaska Native, or multiracial. SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003. **Step 3 of 3: E** **xtended analysis –**  **Tables 5 through 10 provide ex amples of extended**  **NRB analysis**  **as**

**conducted fo** **r ALL and**  **other adult**  **compet e** **ncy surveys**  Table 5 shows an example of the comparison of estimates before and after weighting adjustments. The following t test comparisons were made:

 Comparison of percentage distributions from BQ base weights for the total eligible sample of persons with the BQ base weights for the BQ respondents only to check for differences due to nonresponse to the BQ  Comparison of percentage distributions from BQ base weights for the total eligible sample of persons with that from the BQ nonresponse adjusted weights for respondents to check for differences even after the nonresponse adjustment process to the BQ  Comparison of percentage distributions from BQ nonresponse adjusted weights for respondents with that from the BQ raked weights for respondents to check for differences that may have been introduced through the initial raking procedure  This analysis showed that, through the nonresponse adj ustment, the potential for bias was reduced ( pvalues moved from significant to nonsignificant) in all domains shown below, with the exception of two domains relating to average household size. These two domains show significant bias created by the nonresponse adjustments. While the differences in the percentage distributions of these domains remain statistically significant even after the raking adjustment, they are not large enough to be meaningful.18

**Table 5 .**  **Percent age**  **distribution of sample cases at each weighting step for the ALL background** 

**qu estionnaire, by key characteristics: 2003**  **Total sample %**  **BQ respondents**  **Nonresponse adjustment**  **Initial raking %**  **p value**  **1 %**  **p value**  **1 %**  **p value** **2** Total 100 .0 100 .0 † 100 .0 † 100 .0 † **Region** Northeast 18.4 15.6 0 .000 18.6 0.35 5 19 .0 0.866 Midwest 22.1 22.9 0.18 0 21.9 0.355 23 .0 0.728 South 34.9 35 .0 0.842 34.9 0.825 35 .0 0.716 West 24.7 26.6 0.001 24.7 0.895 23 .0 0.413 **Me tropolitan Statistical Area status**  Non-Metropolitan Statistical Area 21.8 22.7 0.01 2 21.9 0.558 18 .0 0.100 Metropolitan Statistical Area 78.2 77.3 0.012 78.1 0.558 82 .0 0.100 **Locale 3**  Urban 65.3 64.6 0.0 80 65.1 0.448 68 .0 0.097 Suburban 12.4 13 .0 0.10 3 12.4 0.849 11 .0 0.198 Rural 22.3 22.4 0.74 9 22.5 0.566 20 .0 0.075 **Majority own/rent** Rent 25.2 25.7 0.294 25 .0 0.486 26 .0 0.254 Own 74.8 74.3 0.294 75 .0 0.486 74 .0 0.254 **Average household size** less than or equal to 2.2 18.6 18.1 0.22 5 18.1 0.213 18.1 0.946 greater than 2.2 and less than or equal to 2.8 51.5 52.9 0.05 1 53.3 0.003 52.5 0.257 greater than 2.8 30 .0 29.1 0.157 28.7 0.005 29.4 0.381 **Percent less than high school** less than or equal to 10% 33.4 33.1 0.542 33.1 0.569 32.7 0.431 greater than 10% and less than or equal to 20% 31.4 30.9 0.292 31.4 0.887 31.3 0.937 greater than 20% 35.2 36.1 0.123 35.5 0.349 36 .0 0.524 **Percent limited English proficient** 0% 29.9 29.9 0.977 30 .0 0.953 28.8 0.088 greater than 0% and less than 2% 34.6 34.7 0.840 35.4 0.162 35.5 0.756 greater than or equal to 2% 35.5 35.4 0.825 34.7 0.052 35.8 0.31419

**Table 5. Percentage distribution of sample cases at each weighting step**  **for the ALL background**

**questionnaire, by key characteristics: 2003 (Continued)**  **Total sample %**  **BQ respondents**  **Nonresponse adjustment**  **Initial raking %**  **p value**  **1 %**  **p value**  **1 %**  **p value** **2 Percent below 150% of poverty**  less than or equal to 10% 31.5 30.7 0.137 31.6 0.912 32.0 0.629 greater than 10% and less than or equal to 30% 45.4 44.9 0.372 45.3 0.682 45.4 0.814 greater than 30% 23.1 24.4 0.002 23.2 0.452 22.7 0.585 1 The p value and SE are for tests of difference between estimated percentage of current step and total sample (BQ base weights). 2 The p value and SE are for tests of difference between estimated percentage of current and previous weighting steps. 3 This indicator was set to ‘urban’ if the largest percentage in the segment was inside urbanized areas; ‘suburban’ if the largest percentage was inside urban clusters; ‘ r

ural’ if the largest percentage was the rural population. The terms ‘ urbanized areas’, ‘urban clusters’, and ‘rural’ are Census Bureau-defined terms . SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Literacy and Lifeskills Survey (ALL), 2003. Table 6 gives an example of correlations between auxiliary variables and competency score measures for an adult competency survey. The percentage aged 25+ with less than a high school education was found to be highly correlated with literacy score. If this variable was also related to response propensity, using it in nonresponse adjustments should reduce the potential NRB of literacy estimates.

**Table 6 .**  **Correlations of demographic variable with the outcome statistic**

**Variable Absolute correlation with outcom** **e statistic <0.1**  **0.1 -** **0.2 0.2**  **- 0.3**  **0.3 -** **0.4 >0.4**  Percent aged 25+ with less than high school education X Percent of adults speaking Spanish at home and English not well or not at all X Percent below

150% of poverty X Median Income X Percent who rent X Average household size X Age X Percent minority X SOURCE: U.S. Department of Education, National C enter for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003 . Table 7 provides an example of the analysis of disposition codes to evaluate NRB in an adult competency survey. The distribution of cases with a BQ disposition code of language problem or mental disability was compared to the distribution of the other nonrespondents. Chi-square tests were processed to determine if there was a relationship between select demographic variables and the disposition code groups (literacy-related or other) for incompletes. Significant differences in the distribution by region, MSA status, and age were found between the two groups. These variables were used in the weighting adjustments, which are expected to reduce the potential bias from literacy-related nonresponse.20

**Table 7 .**  **Distribution of BQ**  **incompletes, by disposition code category**

**Dom ain**  **Literacy -** **related 2**  **other** Total incompletes 355 5,191

**Dom ain**  **Incompletes 1**  **Statistic p**  **value**

**Literacy -**  **related 2**  **percent OTHER percent**  **Total incompletes** 100 100 **Region** 10 0.006 Northeast 16 17 Midwest 18 22 South 21 38 West 45 23 **MSA status** 9 0.002 Non-MSA 9 18 MSA 91 82 **Age** 23 0 .000 16 -29 16 20 30 -49 37 43 50 - 69 23 27 70+ 24 11 **Gender** 3 0.067 Male 47 53 Female 53 47 **Race/ethnicity** 2 0.244 Hispanic 17 11 NH Black only 6 9 Other 77 79 † Not applicable 1 Nonrespondents 2 Literacy-related reasons for nonresponse at the BQ stage consist of language problems and mental disabilities only. SOURCE : U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003 . An example of an analysis of NIR data for an adult competency survey is provided in Table 8. The table summarizes data collected on the NIR regarding the household race of screener incompletes, as observed or gathered from neighbour information. The base-weighted distribution is shown overall and by region. F or households where race was determined, Hispanic households were less common among screener incompletes compared to the population proportion. This suggests a possible increase in response rates from accommodations made for Spanish speakers, such as bilingual interviewers.21

**Table 8 .**  **Household race for screener incompletes with NIR data, by region**

**Household race Overall**  **Region Northeast**  **Midwest South**  **West** Total 4,276 1,327 819 1,571 559 Not ascertained 2,197 548 535 804 310 **Percent Percent**  **Percent Percent**  **Percent Total**  100 100 100 100 100 White 73 73 84 70 67 Black 10 6 7 15 7 Hispanic 4 3 4 3 8 Asian 4 4 1 2 9 NH/PI 1 0 # # # 1 AI/AN2 0 # # 0 # Other 0 1 # 0 1 Cannot determine 10 14 5 10 8 SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003 . Table 9 shows an example of a comparison of literacy estimates for late versus early respondents. Calculations were done using marginal maximum likelihood (MML) means. The results indicate a small but significant bias in Statistic 2 if focused followup efforts (as describe in section 2) had not been successful in obtaining responses from respondents at the end of the data collection period (i.e. if data about respondents at the end of the data collection period had been missing). This suggests some small level of NRB may be present to the extent that nonrespondents are similar to respondents at the end of the data collection period.22

**Table 9 .**  **Differences between the last 10% and first 90% of respondents, by domain**

**Dom ain**  **Estimate of population proportion Statistic 1**  **Statistic 2 Statistic 3** 

**Proportion am** **ong last 10%**  **Proportion**

**am ong first**  **90% p**  **value Relative**  **difference p**  **value Relative**  **difference p**  **value Relative**  **difference p**  **value Overall**  1. 0 0 1. 0 0 † 0 . 0 0 0 .92 -0. 0 3 0. 0 1* 0 .0 0 0 . 8 0 **Region** Northeast 0 . 2 0 0 .1 9 0 .8 2 0 .0 0 0 .8 9 -0. 0 3 0 .1 2 0. 0 1 0 .7 9 Midwest 0 .1 8 0 .2 3 0 .0 3 0. 0 3 0 .22 0. 0 1 0 .5 8 0. 0 3 0. 0 8 South 0 .3 6 0 .3 6 0 .9 5 0. 0 1 0 .64 -0. 02 0 .3 6 0 .0 0 0 .9 6 West 0 .2 7 0 .2 1 0 .02 -0. 0 3 0 .1 3 -0. 0 6 0. 0 1* -0. 0 1 0 .6 1 **MSA status** Non-MSA 0 .1 3 0 .1 9 0 .04 0 .0 0 0 .9 5 -0. 02 0 .3 8 0. 02 0 .3 8 MSA 0 .8 7 0 .8 1 0 .04 0 .0 0 0 .9 7 -0. 0 3 0 .0 1* 0 .0 0 0 .8 8 **Gender** Male 0 .5 3 0 .4 8 0 .0 1* 0. 0 1 0 . 5 0 -0. 02 0 .22 0 .0 0 0 .94 Female 0 .4 7 0 .5 2 0 .0 1* -0. 0 1 0 . 7 0 -0. 04 0. 0 1* 0 .0 0 0 .8 4 **Race/ethnicity** Hispanic 0 .1 7 0 .1 2 0. 0 0* -0. 02 0 .3 5 -0. 0 7 0. 0 0* -0. 0 1 0 .5 5 NH Black only 0 .1 1 0 .1 1 0 .5 1 0. 0 1 0 .5 7 0. 02 0 .4 1 0. 0 6 0. 0 3 Other 0 .7 3 0 .7 7 0 .04 0. 0 1 0 . 4 0 -0. 0 1 0 . 2 0 0. 0 1 0 .5 1 **Educational attainm ent**  Less than high school 0 .1 8 0 .1 9 0 .3 3 -0. 0 5 0 .0 1* -0. 0 7 0. 0 0* -0. 02 0 .3 3 High school or equivalent 0 .2 8 0 .3 1 0 . 1 0 -0. 02 0 . 5 0 -0. 04 0. 02 0 .0 0 0 .8 2 More than high school 0 .5 4 0 . 5 0 0 .0 6 0 .0 0 0 .9 9 -0. 0 3 0. 0 1* -0. 0 1 0 .2 5 **Age category** 16 -29 0 .2 5 0 .2 5 0 .6 8 -0. 0 1 0 .7 5 -0. 02 0 .22 0. 0 1 0 .7 9 30 -49 0 .42 0 .3 9 0 .0 6 -0. 0 1 0 .3 5 -0. 04 0.0* 0 .0 0 0 .8 7 50 - 69 0 .2 5 0 .2 5 0 .92 0. 02 0 .2 5 0 . 0 0 0 .8 5 0 .0 0 0 .9 6 70+ 0 .0 9 0 .1 1 0 .0 7 -0. 0 3 0 .5 3 -0 . 1 0 0. 0 1* -0. 0 7 0 .1 1 **Country of birth** US 0 . 8 0 0 .8 6 0 . 0 0 0. 0 1 0 .44 -0. 0 1 0 .3 2 0. 0 1 0 .2 9 Other 0 . 2 0 0 .1 4 0 . 0 0 0 .0 0 0 .9 7 -0. 0 5 0 .04 0 .0 0 0 .9 7 † Not applicable SOURCE: Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003 .23

An example of the calculation of the range of potential bias in literacy estimates is shown in Table 10. The table shows the difference between respondent and nonrespondents literacy scores that would bias the population estimate by b = 3, 5, 10, or 15 points. The required difference between respondents and nonrespondents was computed as described earlier in section 3.3.2. For the overall sample, literacy scores of respondents and nonrespondents would have to differ by about 8 points on average to bias the national estimate by 3 points and would need to differ by 39 points to bias the national estimate by 15 points. **Table 10 .**  **Average differences between respondents and nonr espondents needed to achieve various** 

**amounts of bias in outcome statistics**

**Dom ain**  **Percent of population 1**  **represented by nonrespondents**  **2 Average**  **point difference between** 

**respondents and nonrespondents needed to**

**bias estimated scores by 3**  **5 10**  **15 Total**  38 8 13 26 39 **Region** Northeast 43 7 12 23 35 Midwest 35 9 14 28 43 South 38 8 13 26 40 West 38 8 13 26 39 **MSA status** Non-MSA 33 9 15 30 45 MSA 40 8 13 25 38 **Average household size** ≤ 2.42 39 8 13 26 39 (2.42, 2.8] 39 8 13 26 39 > 2.8 38 8 13 27 40 **Percent less than high school** ≤ 10.4% 43 7 12 23 35 (10.4%, 20.3%] 39 8 13 26 38 (20.3%, 32%] 35 8 14 28 42 > 32% 31 10 16 32 48 **Percent speaking Spanish but not English** 0% 39 8 13 26 38 (0%, 28%] 40 8 13 25 38 > 28% 34 9 15 29 44 **Percent below 150% of poverty** ≤ 10.7% 43 7 12 23 35 (10.7%, 20%] 40 8 13 25 38 (20%, 33.3%] 35 9 14 29 43 > 33.3% 29 10 17 34 52 **Median incom e**  ≤ 28,400 28 11 18 36 53 (28,400, 37,850] 34 9 15 29 44 (37,850, 52,100] 40 8 13 25 38 > 52,100 44 7 11 23 3424

**Table 10 .**  **Average differences between respondents and nonrespondents needed to achieve various**

**amounts of bias in outcome statistics (Continued)** 

**Dom ain**  **Percent of population 1**  **represented by nonrespondents**  **2 Average point difference between** 

**respondents and nonrespondents needed to**

**bias estimated scores by 3**  **5 10**  **15 Percent rent**  ≤ 16% 42 7 12 24 36 (16%, 31%] 37 8 13 27 40 (31%, 59%] 37 8 14 27 41 > 59% 35 9 14 29 43 **Low response rate CHAID cells**  Cell 1 3 59 5 8 17 25 Cell 2 4 51 6 10 20 30 Cell 3 5 50 6 10 20 30 1 The population excludes language problems and mental disabilities. 2 Nonrespondents to the Screener and BQ. 3 Median income greater than 52,100, average household size 2.42 or less, Northeast region 4 Median income 37,850 - 52,100, percent less than high school greater than 20.3, Northeast region 5 Median income greater than 52,100, average household size 2.42 - 2.8, South or West region SOURCE : U.S. Department of Education, National Center for Education Statistics, National Assessment of Adult Literacy (NAAL), 2003 .

**4. SUMMARY**  This document presented the main goals of PIAAC for reducing NRB throughout the survey process. A series of procedures for implementation before and during the data collection were described to help increase the response rates with the goal of reducing NRB to the extent possible. Preliminary plans are provided to conduct basic NRB analysis to allow an eff

ective weighting adjustment plan that reduces the bias to the maximum extent possible. Then extended analyses are described with the goal of assessing the bias remaining in the estimates after nonresponse adjustment is completed. In addition to assessing the remaining bias in outcome statistics, the results of the NRB analysis can be used to inform analysts of data limitations, and can also help in future rounds of the survey in defining the target population and improving data collection methods, weighting, and estimation approaches. The PIAAC Consortium is continuing its research and evaluation of various NRB analysis methods and welcomes comments and suggestions from participating countries on such analyse s .