i

PROGRAM FOR INTERNATIONAL STUDENT ASSESSMENT

(PISA 2015) MAIN STUDY DATA COLLECTION

REQUEST FOR OMB CLEARANCE

OMB# 1850-0755 v.16

SUPPORTING STATEMENT PART B

Prepared by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

Submitted:

January 2015

TABLE OF CONTENTS

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe............................................................................................................. 1

B.2 Statistical Methodology ......................................................................................................... 1

B.3 Maximizing Response Rates................................................................................................. 3

B.4 Tests of Procedures and Methods....................................................................................... 4

B.5 Individuals Consulted on Study Design.............................................................................. 4
1

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent Universe

PISA 2015 assesses students’ knowledge and skills in science, mathematics, and reading as they near the end 

of their compulsory school experience. For international comparability, this is defined as students who are 15 

years old, in grades 7 or higher. A range of exact birthdates is specified by the international coordinating 

committees based on the months in which the data will be collected: students must be between the ages of 15 

years and 3 completed months and 16 years and 2 completed months at the beginning of the test period. The 

universe for the selection of schools is all types of schools in all states of the United States and the District of 

Columbia. Within sampled schools, students will be selected for participation by drawing a random sample 

among the 15-year-old students.

B.2 Statistical Methodology

The Technical Standards for main study PISA 2015 established by the international governing board include 

the following:

Standard 1.8. The student sample size must be a minimum of 5,250 assessed students, or the National Defined 

Target Population.

Standard 1.9. The school sample size must be a minimum of 150 schools or all schools that have students in the 

National Defined Target Population.

Standard 1.10. The target cluster size is typically 35 PISA eligible students, which upon agreement can be increased 

or reduced to a number not less than 20.

Standard 1.11. School response rates must be above 85 percent of sampled schools. If a response rate is below 85 

percent then an acceptable response rate can still be achieved through agreed upon use of replacement schools. PISA 

establishes three response rate zones—acceptable, intermediate, and not acceptable. “Acceptable” refers to original 

school response rates above 85 percent and means that the country’s data will be included in all international 

comparisons. “Not Acceptable” refers to original response rates below 65 percent and means that the country’s data 

will be a candidate for not being reported in international comparisons unless considerable evidence is presented that 

nonresponse bias is minor. “Intermediate” refers to original school response rates of between 65 and 85 percent and 

means that a decision on whether or not to include the country’s data in comparisons must be made while taking into 

account a variety of factors, such as student response rates, quality control, etc. In addition, schools with less than 50 

percent participation of students are not considered participating schools and neither that school nor those students 

that did participate are considered in the calculation of response rates.

Standard 1.12. The overall student response rates must be above 80 percent of sampled students.

In addition, NCES has a standard in which student response rate should be at least 85 percent, and the 

sampling design described below is based on that rate.

Overview

The design for this study will be self-weighting, stratified, consist of two stages, and will use probability 

proportional to size (PPS). There will be no oversampling of schools or students. Schools will be selected in 

the first stage with PPS and students will be sampled in the second stage yielding overall equal probabilities of 

selection.

Target Populations

The PISA target population is 15-year-old students attending education institutions located within the United 

States in grades 7 and higher. The plan is to implement the survey in the fall of 2015 (October-November).
2

The specific definition of age eligibility that will be used in the survey is “...between 15 years and 3 

(completed) months to 16 years and 2 (completed) months at the beginning of the testing window.”

Sampling Frame of Schools

The population of schools for PISA 2015 is defined as all schools enrolling any 15-year-olds in grades 7 

through 12. As in previous PISA cycles, the school sampling frame will be developed from the most up-todate NCES Common Core of Data (CCD) and Private Schools Survey (PSS). For the PISA 2015 main study, 

we will use the school sampling frame prepared for the National Assessment of Educational Progress 

(NAEP) 2014 which used the 2012-2013 CCD and the 2011-2012 PSS school data.

The grade structure of the school is a key stratification variable designed to reduce sampling error, but this is 

especially so in PISA because data analyses have shown that achievement is highly related to grade. Other 

stratification variables include public/private, region of the country, location (urban/suburban/town/rural, 

etc.), and enrollment by race/ethnicity.

School, Teacher, and Student Samples

The international minimum number of completed assessments—for the core computer-based assessments in 

science, mathematics, reading, and collaborative problem solving—is 5,250 students in 150 schools. In PISA, 

the United States typically assesses between 5,600 and 5,900 students in 165 schools. Assuming a similar

response rate as PISA 2012, our initial target is a total sample of about 240 schools to yield about 165

participating schools. To achieve the target final school response rate, we will use replacement schools to 

complete the sample.

The sample of teachers will be based on an assumption of up to 25 teachers per school and an average of 24 

teachers per school for a total of 3,882 sampled teachers and 3,300 responding teachers.

The student-per-school target for the core assessment is at least 36 completed student assessments per 

school. Assuming a within-school response rate of 85 percent (rates were 85 percent in 2000, 82 percent in 

2003, 91 percent in 2006, 86 percent in 2009, and 89 percent in 2012), the original sample size of students 

within schools will be 42. The two states that are finalizing plans to participate in PISA 2015 will each have a 

sample of 50 schools and 2,100 students to yield 1,890 assessed students.1 Puerto Rico will have a sample size 

of 50 schools and 2,100 students.

Gathering Contact Information to Support a Future Methodological Study

In PISA 2012, NCES collected student contact information for a follow-up study to validate PISA by relating 

student performance on PISA to other cognitive, education, and employment outcomes (the new PISA 

Validation Study, OMB# 1850-0900). NCES is again considering conducting a follow-up with PISA 2015 

students. As in PISA 2012, PISA 2015 students will be asked to supply contact information so NCES can 

contact them in the future. The follow-up study, including any follow-up contact with students, would be 

carried out under a separate OMB clearance request. In this current request we are seeking approval only to 

gather student contact information as was done in PISA 2012. Appendix D contains items that will be used to 

ask students for their email, home address, and phone numbers, along with those of a relative or a close 

friend. The items are the same as those used to collect contact information from the PISA 2012 students 

(OMB# 1850-0755 v.13) with an additional question asking for the email of a relative or another contact.

 

1 This is based on an expected response rate in the state of 90 percent; response rate in the states has been historically slightly higher than the national sample.

This is attributable to the fact that, for state PISA samples, states recruit and manage the participation of the schools and the students.
3

Nonresponse Bias Analysis, Weighting, Sampling Errors

It is inevitable that nonresponse will occur at both levels of sampling: school and student. We will analyze the 

nonrespondents and provide information about whether and how they differ from the respondents along 

dimensions for which we have data for the nonresponding units, as required by NCES standards. After the 

international contractor calculates sample weights, sampling errors will be calculated for a selection of key 

indicators incorporating the full complexity of the design, that is, clustering and stratification (see the already 

approved Appendix C for more detail on plans for a nonresponse bias analysis).

B.3 Maximizing Response Rates

Historically PISA has struggled to achieve adequate school-level response rates. PISA is a long assessment 

requiring over 3 hours of student time. Schools are reluctant to participate because of the commitment of 

time and loss of instructional time for students. Because PISA is administered to an age-based sample, 

students are sampled from across grades and classes, which schools find disruptive. We have reviewed our 

materials and conducted focus groups (OMB# 1850-0803) in order to understand the attitudes that school 

administrators have toward PISA and what materials work best in describing PISA to schools. The following 

table provides the school and student response rates for each PISA round.

Table B. Historical PISA school and student response rates for the United States

Year Weighted Original school 

response rate

Weighted School response rate after 

replacement

Weighted Student 

response rate

2000 56 70 85

2003 65 68 83

2006 69 79 91

2009 68 78 87

2012 67 77 89

Our approach to maximizing school and student response rates in the main study includes the following:

 Use of a fall test administration, to avoid conflicts with state testing;

 Selecting and notifying schools during the academic year preceding PISA data collection;

 Communicating with state officials early in the process and applying a more proactive approach with 

states to gain assistance with sampled schools, including working with NAEP staff and state 

recruitment personnel to reach out to districts and schools;

 Assigning personal recruiters for specific schools;

 Incentives for schools, school coordinators, teachers, and students (see Section A9);

 Contact with schools and school coordinators at set intervals throughout the year preceding the 

assessment;

 A summer meeting several months before the data collection (June 2015) for representatives from 

sampled schools to inform them about PISA and keep them engaged in the study; and

 Provision of school-level results on PISA for schools that achieve a minimum student response rate 

of 85 percent.

These approaches are based on experience from previous PISA administrations and consultation with staff 

from other studies such as TIMSS and NAEP.
4

B.4 Tests of Procedures and Methods

The main study will collect data on selected students who will be assessed in science, mathematics, reading,

collaborative problem solving, and financial literacy. These data, or their subset, depending on the 

international data release plans, will be reported via a U.S. National Report that will be released approximately 

one year after data collection, in December 2016, to correspond with the international data release. This 

report will be tailored to a general audience and will provide preliminary results about the overall performance 

of U.S. students relative to student performance in other countries. Following the release of the national 

report, additional data will be made available to secondary users in the form of the International Data 

Explorer (IDE), an online tool on the NCES website, and a U.S. public-release dataset. Also, those schools 

that qualify (based on response rate and sample size requirements) will receive a school report with basic 

comparisons of the performance means of students in the school with overall means for the United States, 

OECD countries, and other similar schools.

The U.S. PISA field test assessed 72 schools and 2,288 students. Given that PISA 2015 uses computer-based 

assessments (CBA), the field test employed a mode effect study to determine the effect, if any, of moving 

trend items, previously administered in paper-based form (PBA), to a computer-based mode. The mode effect 

study examined the differences in student performance between the paper-based versions and the computerbased versions of the same items. Overall, the large majority of items performed similarly between both 

modes. While there were differences in student performance on some items across countries, the level of 

difficulty of an item stayed largely the same between PBA parameters and the new CBA-based parameters. 

Where mode effects do occur, an item can be adjusted using a statistical model-based method to maintain 

item comparability between modes and allow trend measurement to be maintained.

Overall, the International Consortium of contractors concluded that the CBA mode is an improvement over 

the PBA mode. The CBA allows more variations in the order in which assessment items are presented to 

students across the sample thereby providing better measurement of the subject domains. CBA also shows

less effect on the proportion correct and student non-response due to the order in which students see the 

items. Additionally, timing and process data obtained from computer system log files help evaluate and better 

understand student non-response, which was not possible with a paper-based administration of PISA.

NCES was concerned that the length of the core assessment with equipment set up and the need to hold a 

second session for financial literacy would require a second day in schools. This proved not to be the case in 

the field trial. We were able to assess both the core PISA assessment and financial literacy in a single day in all 

schools. PISA 2015 will administer school questionnaires and new teacher questionnaires online. These 

questionnaires were launched and administered successfully in the field test, achieving adequate response

rates. We believe that we can sustain these response rates in the main assessment.

B.5 Individuals Consulted on Study Design

Many people at the OECD, ETS, and other organizations around the world have been involved in the design

and development of PISA. Some of the lead people are listed in section 8 of Part A. Overall direction for 

PISA is provided by Drs. Dana Kelly and Patrick Gonzales, the PISA National Project Managers at the 

National Center for Education Statistics, U.S. Department of Education. Dave Kastberg and David Ferraro at 

Westat are responsible for the sample design, data collection, and reporting. Internationally, Irwin Kirsch and 

Claudia Tamassia at ETS have responsibility for overall project coordination.