1

APPENDIX D: NON-RESPONSE BIAS ANALYSIS PLAN
2

Outline of the PISA 2015 Non-response Bias Analyses

Westat is planning to conduct two nonresponse bias analyses. A preliminary NRBA will be conducted 

in the early spring, prior to the data adjudication by the International Consortium in April, 2016. These 

analyses are a subset of the full analysis and are outline after the full analysis below. The full (final) 

NRBA will be conducted for schools. Currently, it is assumed that the student response rate for PISA 

2015 is above 85 percent and thus, no NRBA at the student level will be required. 

Final PISA School Nonresponse Bias Analysis Outline

The final NRBA would be conducted in the summer of 2016 before the release of the PISA 2015 

national report in December of 2016. Aa summary of the findings would be included in the technical 

appendix of the U.S. national report. The full NRBA would be included in the U.S. technical report.

1. INTRODUCTION 

2. METHODOLOGY

The analysis will be conducted in three parts:

 First, the distribution of the participating original school sample will be compared with that 

of the total eligible original school sample. The original sample is the sample before 

substitution. In each sample, schools will be weighted by their school base weights, 

excluding any nonresponse adjustment factor. 

 Second, the distribution of the participating sample, which includes the participating 

replacements for schools from the original sample that did not participate, will be 

compared to the total eligible final sample. The final sample is the sample after 

substitution. Again, school base weights will be used for both the eligible sample and the 

participating schools. 

 Third, the same sets of schools will be compared as in the second analysis but, this time, 

when analyzing the participating schools alone, school nonresponse adjustments will be

applied to the weights. 

The following categorical variables will be available for all schools:

 School type—public or private;

 Locality—urban-centric locale code, i.e., city, suburb, town, rural;

 Census region; and

 Poverty level—for public schools, a high poverty school is defined as one in which 

50 percent or more of the students are eligible for participation in the National School 
3

Lunch Program (NSLP), and a low poverty school is defined as one in which less than 

50 percent are eligible; all private schools are treated as low poverty schools.

The following continuous variables will be available for all schools:

 Number of age-eligible students enrolled;

 Total number of students;

 Mean percentage of students by race/ethnicity (White, non-Hispanic, Black, non-Hispanic, 

Hispanic, Asian or Pacific Islander, American Indian or Alaska Native, and two or more 

races).

An additional continuous variable, the percentage of students eligible to participate in the NSLP, was 

available only for public schools.

Two forms of analysis will be undertaken:

 A test of the independence of each school characteristic and participation status, and

 A logistic regression in which the conditional independence of these school characteristics 

as predictors of participation will be examined.

For categorical variables, the distribution of frame characteristics for participants will be compared with 

the distribution for all eligible schools. The hypothesis of independence between the characteristic and 

participation status will be tested using a Rao-Scott modified Chi-square statistic at the 5 percent level 

(Rao and Thomas 2003). For continuous variables, summary means will be calculated and the difference 

between means will be tested using a t test. In addition to these tests, logistic regression models 

(including all characteristics) will be used to provide a multivariate analysis in which the conditional 

independence of these school characteristics as predictors of participation will be examined. 

3. RESULTS

For each categorical or continuous variable, a table will be shown giving the percentage (or mean) for 

the participating and eligible populations along with the bias, relative bias, and the p-value of the test. 

Text summaries of the results will also be provided. The logistic regression results will be shown giving 

the parameter estimate, standard error, t test and p-value. The results will be given for each analysis. 

 3.1 Original Respondent Sample

Categorical Variables 

Continuous Variables 

Logistic Regression Model 

3.2 Respondent Sample With Replacements (Final Sample)

Categorical Variables 

Continuous Variables 
4

Logistic Regression Model 

3.3 Final Sample, With Nonresponse Adjustments Applied

Categorical Variables 

Continuous Variables 

Logistic Regression Model 

4. CONCLUSIONS

A summary of the results will be presented along with a conclusion on the effect of substitutes and 

the nonresponse weighting adjustment. 

Preliminary PISA School Nonresponse Bias Analysis Outline

This NRBA would be conducted in the first quarter of the 2016 to be available to the IC by April. 

1. INTRODUCTION 

A preliminary analysis will be conducted just on the original respondent sample. The analysis will be 

identical to the final analysis except that the base weights and school statuses may not be final. 

2. METHODOLOGY

3. RESULTS

 Original Respondent Sample

 Categorical Variables 

 Continuous Variables 

 Logistic Regression Model 

4. CONCLUSIONS