TRENDS IN INTERNATIONAL MATHEMATICS AND 

SCIENCE STUDY (TIMSS 2015) FIELD TEST AND

RECRUITMENT FOR FIELD TEST AND MAIN STUDY

REQUEST FOR OMB CLEARANCE

OMB# 1850-0695 v.3

SUPPORTING STATEMENT PART B

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

June 2013
21

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe 

TIMSS. The respondent universe for the TIMSS field test is all students enrolled in grade 4 and grade 8 during the 

2013-2014 school year. The universe for the selection of schools is all types of schools in seven populous states. A 

sample of 40 schools will be selected for the field test, with the goal of obtaining participation from a minimum of 

35 schools at each grade. Within sampled schools, students will be selected for participation by drawing a random 

sample of two classes. At grade 8, students will be selected by drawing a sample of two intact mathematics 

classrooms (in which grade 8 students are enrolled) in each sampled school. All selected students will be asked to 

participate in a combined TIMSS mathematics and science assessment. Only students in intact classrooms will be 

assessed at each grade.

TIMSS Advanced. The respondent universe for the TIMSS Advanced field test is all students in the last year of 

secondary school who have taken or are taking advanced mathematics or physics courses as defined by the TIMSS 

2015 Advanced Framework. For advanced mathematics, the framework includes algebra, geometry, and calculus

content. In the United States, algebra and geometry are pre-requisites for calculus, so the focus will be on calculus.

The TIMSS Advanced Framework includes limits, derivatives, and integrals as topics; all of which are typically 

included in calculus courses in the United States. Using the NCES Secondary School Course Classification System:

School Codes for Exchange of Data (SCED) (http://nces.ed.gov/pubs2007/2007341.pdf ), the following courses, at 

a minimum, would be eligible for the advanced mathematics assessment:

02121 Calculus

02122 Multivariate Calculus

02123 Differential Calculus

02124 AP Calculus AB

02125 AP Calculus BC

02126 Particular Topics in Calculus (when it follows a survey calculus course)

02132 IB Mathematics

The physics framework includes mechanics and thermodynamics, electricity and magnetism, wave phenomena, and 

atomic/nuclear physics. In the United States, this content is most likely to be covered in second-year/advanced 

physics courses, and not in first-year survey physics courses. The following courses in the SCED, at a minimum,

would be eligible for the TIMSS Advanced physics assessment:

03152 Physics—Advanced Studies

03155 AP Physics B

03156 AP Physics C (either E&M or MECH)

03157 IB Physics

Note that course eligibility will continue to be reviewed as the sampling frame is developed for U.S. participation.

In particular, some students may have completed eligible coursework at a college level as a summer course, as a 

dual credit course, or through a distant learning program.

The advanced mathematics and physics student populations are separate, but overlapping populations. All selected 

students will be asked to participate in either an advanced mathematics assessment or a physics assessment, but not 

both.

The universe for the selection of schools is all types of eligible schools in the same seven populous states as in 

grades 4 and 8 field test, where “eligibility of schools” is determined by having courses in their course catalogue (or 

available to enrolled students through colleges or virtual schools) that meet the content specifications for advanced 
22

mathematics and physics in the TIMSS 2015 Advanced Frameworks. Note that eligibility can only be determined 

after the original sample of schools is selected. To maximize the likelihood that selected schools are in fact 

“eligible”, we will consider using the NAEP High School Transcript Study (HSTS) schools from 2009 as the 

sampling frame for selecting schools. We will sample 61 schools with the goal of obtaining participation from a 

minimum of 35 eligible schools. 

Using student transcripts and current course enrollment, two lists of eligible students (one for advanced 

mathematics and one for physics) will be created from which to sample students. Eligible students are those who 

have taken or are currently taking the relevant courses. Students who are on both lists (eligible for both advanced 

mathematics and physics) will be randomly assigned to one sample.

B.2 Statistical Methodology

Field Test Sampling Plan and Sample

TIMSS. The purpose of the TIMSS field test is to test out new assessment items and background questions, and to 

ensure that classroom and student sampling procedures proposed for the main study are successful. In selecting a 

school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize 

impact on these entities while also ensuring that the field test data are collected effectively.

As required by the International Study Center, the field test sample is to consist of 40 schools at grade 4 and 8, with

1,400 students assessed at each of these grades. The student samples will be obtained by selecting two classes from 

each school (two mathematics classes for grade 8). As the field test is designed only to test items, questions, and 

procedures, a probability sample of schools is not required. However, the sample must include a broad range of 

schools covering such features as public (including charter schools), private, large, small, urban, and rural schools, 

and schools from a variety of different states.

In addition, the selection of the field test sample must take into account the fact that the TIMSS field test will take 

place in the spring of 2014 after the NAEP 2014 assessment. NAEP 2014 will be a national sample and will involve 

several hundred schools at each of grades 4 and 8 (and not several thousand as in the main, state-sample NAEP in 

2013 and 2015). Because the TIMSS assessment follows the NAEP assessment, we will select the TIMSS field test 

sample so as not to include any schools that are sampled for NAEP 2014. The following steps describe the 

sampling approach for the TIMSS field test:

1. We will identify a set of seven states where we will conduct the field test, with the aim of gaining the 

cooperation of 7 schools per grade in each of the four largest states, and 4 schools per grade in each of the 

three smallest states. With such an allocation, the selection of states should be skewed somewhat towards 

larger states, but include some smaller states. The states will be spread across different regions. This will 

provide an appropriate geographic diversity for the selected schools.

2. After identifying the eligible schools in each state, using the most current NCES CCD (public schools) and 

PSS (private schools) files, we will subset these by eliminating schools that (a) are unlikely to have two 

classes per grade, and (b) have been included in the NAEP 2014 sample. We will also eliminate schools that 

have been selected for any other NCES studies involving students at grades 4 or 8 during the 2013-14 school 

year.

3. We will then select a stratified probability sample of schools, ensuring representation by school type 

(approximately 4 of the 40 schools selected will be private schools), urban/rural, and race/ethnic composition. 
23

We will use probability sampling not because it is required, but because it is an efficient way to select a 

sample of schools representing a variety of school characteristics.

4. For each sampled school we will identify two replacement schools, matched exactly to the original sample in 

terms of state and public/private status, and as closely as possible with regard to the other characteristics used 

to stratify the original sample. The replacement schools will be recruited in cases where the original sample 

school is unable to participate.

Once the field test sample has been selected, a summary of the distribution of the characteristics of the selected 

schools will be prepared, showing the comparison with the national population of schools. 

The student sampling procedures for the field test will correspond as closely as feasible to what is planned for the 

main study, so as to try out the operational procedures for student sample selection. The sample will be selected by 

selecting two classes per school. Each participating school will be asked to submit an exhaustive list of classes (that 

is, one that accounts for each student in the grade exactly once). At grade 8 this will be a list of mathematics 

classes. Because in some cases at grade 8 a student can be enrolled in more than one mathematics class, the list of 

classes, and the students to be assessed, must be handled in such a way as to ensure that each student has only one 

chance of selection. In cases where one or more class on the list has fewer than 10 students, smaller classes will be 

combined to form “pseudoclasses” for the purposes of sampling. Once the list of classes is submitted, we will use a 

sampling algorithm to select two classes (or pseudoclasses) with equal probability. The student sample will then 

consist of all students in the selected classes.

We plan to gather class and student lists from participating schools electronically using an adaptation of our secure 

E-filing process (as explained in Part A). E-filing was successfully used in PISA 2012, and provides advantageous 

features such as efficiency and data quality checks. Schools will access the E filing system through the MyTIMSS 

web site.

TIMSS Advanced. Like TIMSS at grades 4 and 8, the purpose of the TIMSS Advanced field test is to test new 

assessment items and background questions. Because of the nature of the TIMSS Advanced target population, the 

field test will also help ensure that school and student sampling procedures (including determining school and 

student eligibility) proposed for the main study are successful. The TIMSS Advanced field test sample is to consist 

of 35 high schools, and assess 1,400 students (700 for advanced mathematics and 700 for physics). As with the 

TIMSS field test at grade 4 and 8, the TIMSS Advanced field test is designed only to test items, questions, and 

procedures; so a probability sample of schools is not required.

The TIMSS Advanced field test will also take place in the spring of 2014 after the NAEP 2014 assessment. NAEP 

2014 will involve less than 100 schools at grade 12 (and not almost two thousand as in 2013 and 2015). The 

following steps describe the sampling approach for the TIMSS Advanced field test:

1. In the same 7 states that we are conducting the grades 4 and 8 field test, we will select 10 high schools in 

each of the four largest states, and 7 high schools in each of the three smallest states. Using the same 7 states 

will provide operational efficiency and appropriate geographic diversity for the selected schools.

2. After identifying the potentially eligible schools in each state, using the most current NCES CCD (public 

schools) and PSS (private schools) files, we will subset these by eliminating schools that have been included 

in the NAEP 2014 sample.

3. We will then select a stratified probability sample of 61 schools, ensuring representation by school type 

(approximately 10 percent of the schools selected will be private schools), urban/rural, and race/ethnic 

composition.
24

4. We will sample 61 schools to ensure that we have 35 eligible schools, where school eligibility is determined 

by offering at least one of the eligible courses for advanced mathematics and physics. We will obtain course 

catalogues from all 61 schools, and code the course catalogues according to the course criteria to determine 

eligible schools. Mathematics and science department chairs will also be asked to identify eligible courses 

from their catalogues. If more than 35 are eligible, we will select 35 for actual data collection.

5. For each sampled school we will identify two replacement schools, matched to the original sample in terms 

of state and public/private status, and matched, as closely as possible, with regard to the other characteristics 

used to stratify the original sample. The replacement schools will be recruited in cases where the original 

sample school is unable to participate.

Once the field test sample has been selected, a summary of the distribution of the characteristics of the selected 

schools will be prepared, showing the comparison with the national population of schools. 

The student sampling procedures for the field test will correspond as closely as feasible to what is planned for the 

main study, so as to try out the operational procedures for student sample selection. The student samples will be 

obtained by conducting the following steps:

1. Collect electronic (or paper) student transcripts as well as current class enrollment of all twelfth grade 

students in the selected schools. 

2. Determine eligible students for both advanced mathematics and physics; these are students in the target 

population in the school, who have taken or are currently taking eligible courses.

3. Draw random sample of 20 students for each assessment (advanced mathematics and physics) from those 

students eligible for each. If a student is in both populations, we will randomly assign the student to one 

subject.

We plan to gather class, transcript, and student lists from participating schools electronically, using an adaptation of 

our secure E-filing process. High schools will access the E filing system through the MyTIMSS web site.

Main Study Sampling Plan and Sample

TIMSS and TIMSS Advanced. The school sample design for the main study must be more rigorous than that for 

the field test. It must be a probability sample of schools (one sample for each of grades 4, 8, and 12) that fully 

represents the entire United States. At the same time, to ensure maximum participation it must be designed so as to 

minimize overlap with other NCES studies involving student assessment that will be conducted around the same 

time.

The main study of both TIMSS and TIMSS Advanced will take place in the spring of 2015, about two months after 

a very large NAEP assessment. NAEP will assess several thousand schools nationally, at grades 4, 8, and 12. The 

NAEP sample will be relatively heavy in smaller states, and in a number of these states all eligible schools will be 

included in NAEP, especially at grade 8 and 12 (for the 13 states included in grade 12 state NAEP). Thus to be 

fully representative, the TIMSS sample may include some schools that will have participated in NAEP at the same 

grade. However, this number can be kept to minimum using the overlap control procedures outlined below. 

While PISA 2015 will take place in the fall of the subsequent school year to TIMSS, it will overlap with school 

recruiting. PISA will include a small number of grade 8 schools in the sample, but very few grade 8 students will be 

assessed in PISA, since very few grade 8 students meet the PISA 15 years of age requirements. Most of the PISA 
25

selected schools will include grade 9 and therefore also grade 12. Thus overlap control between the TIMSS and 

PISA samples will be desirable, and should not be difficult to accomplish successfully. 

Overlap control procedures in studies such as this, where stratified probability proportional to size samples of 

schools are selected, can be implemented via a procedure that applies Bayes Theorem to modify the conditional 

probability of selection of a given school for one study, depending upon its selection probability for a second study, 

and whether or not it was selected for that study. This approach was first documented in a survey sampling 

application by Keyfitz (1951)1. The principles involved can be extended to more than two studies simultaneously, 

and a procedure for doing this is described by Chowdhury et al. (2000)2.

The sample size for the TIMSS main study will be 300 schools at each of grades 4 and 8. The sample size for the 

TIMSS Advanced main study will be 250 high schools (that contain grade 12). For each original sample school, 

two replacement schools will also be identified. The sampling frames of grade 4, 8, and 12 schools will be obtained 

from the most current versions of NCES’s Common Core of Data (CCD) and Private School Survey (PSS) files, 

restricted to schools having grade 4, 8, or 12 respectively, and eliminating schools in Puerto Rico, U.S. territories, 

and Department of Defense overseas schools.

The sample will be stratified according to school characteristics such as public/private, Census region, poverty 

status (as measured by the percentage of students in the school receiving free or reduced-price lunch in the National 

School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the 

selected sample of schools.

Determining school eligibility, student eligibility, and student sampling for TIMSS Advanced will be accomplished 

as described above for the field test. For TIMSS at grade 4 and 8 it will be accomplished as described below.

Schools will be selected with probability proportional to the number of estimated classes at the appropriate grade (4 

or 8), with schools expected to have either one or two classes being given the same selection probability. The use of 

a probability proportional to sample design ensures that all students have an approximately equal chance of 

selection, since two classes will be selected from each school regardless of the size of the school. Note that we will 

modify this equal probability design in the following way. So as to increase the available sample size of students in 

high poverty schools, we will double the probability of selection of each school with at least 50 percent of students 

eligible for free or reduced-price lunch under NSLP, relative to other schools of the same size.

Student sampling will be accomplished by selecting two classes per school. Each grade 4 school will be asked to 

prepare a list of classes that is comprehensive, and includes each grade 4 student in the school in one of the listed 

classes. As described above, schools will submit these classes and student lists via secure E-filing. Grade 8 schools 

will be asked to prepare such a list also, but in this case the students should be organized into mathematics classes. 

At either grade, any class with fewer than ten students will be combined with another class to form a ‘pseudoclass’ 

with at least ten students in it. We will then select two classes (or pseudoclasses) from each school, with equal 

probability, and all students in those classes/pseudoclasses will be included in the sample. If a school has only one 

or two classes, then all students in the grade will be included in the sample. At grade 8, mathematics classes are 

used for three reasons. First, this minimizes the burden on mathematics teachers, as only two mathematics teachers 

 

1

Keyfitz, N. (1951). Sampling With Probabilities Proportional to Size: Adjustment for Changes in Probabilities. Journal of the 

American Statistical Association, 46, 105-109.

2 Chowdhury, S., Chu, A., & Kaufman, S. (2000). Minimizing overlap in NCES surveys. Proceedings of the Survey Methods 

Research Section, American Statistical Association, 174-179. Retrieved from http://www.amstat.org/sections/srms/ 

proceedings/papers/2000_025.pdf.
26

need to fill out a teacher questionnaire (but typically more than two science teachers are required to do so, since the 

students in the two selected math classes often attend more than two different science classes). Second, it makes for 

sound data for conducting analyses of the extent to which classroom factors moderate the relationship of student 

factors to achievement (e.g., “Does having a well-qualified math teacher reduce the correlation between math 

achievement and parental education?”). Third, at grade 8, most students take one and only one mathematics class, 

and thus mathematics classes make for a foolproof partitioning of the eligible students. 

Determining school eligibility for the TIMSS Advanced assessment at grade 12, as well as student eligibility and 

student sampling will be accomplished as described above for the field test.

Nonresponse Bias Analysis, Weighting, and Sampling Errors

It is inevitable that nonresponse will occur at both levels: school and student. We will analyze the nonrespondents

and provide information about whether and how they differ from the respondents along dimensions for which we 

have data for the nonresponding units, as required by NCES standards. After the calculation of weights, sampling 

errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, 

clustering and stratification (see Appendix D for more detail).

B.3 Maximizing Response Rates

With the recent exception of TIMSS 2011, the most significant challenge in recruitment for TIMSS has been 

engaging the schools and gaining their cooperation. The circumstances that aided our success in 2011—the NAEPTIMSS Linking Study and the involvement of NAEP State Coordinators—may not recur in 2015. However, there 

are important lessons to be learned from that experience that can be used regardless of the overall TIMSS 2015 

program. Given that classrooms are selected, student participation is not as great of a challenge. Historically student 

participation rates have never fallen below 90 percent (see table 1). That said, it is important to U.S. TIMSS that 

students are engaged and try hard on the assessment, which could be an issue at grade 8 and 12.

Table 1. Historical TIMSS school and student participation rates 

Year Grade School Participation Rate Overall Student Participation Rate Before Replacement After Replacement

2011

4 79 84 95

8 87 87 94

2007

4 70 89 95

8 68 83 93

2003

4 70 82 95

8 71 78 94

1999 8 83 90 94

1995 4 86 NA 94

Our approach to school recruitment is to:

 Obtain endorsements about the value of TIMSS from relevant organizations;

 Inform Chief State Officers and test directors about the sample of schools in their state. Enclose a sample 

letter of endorsement they can send to schools; 

 Send letters and informational materials to schools and districts. These letters will be customized by type 

of school;
27

 Train experienced school recruiters about TIMSS;

 Implement strategies from NAEP’s Private School Recruiting Toolkit. This toolkit, developed for NAEP, 

includes well-honed techniques used to recruit a very challenging type of schools;

 Followup mailings with telephone calls to explain the study and schools involvement, including placing 

the TIMSS assessment date on school calendars;

 Maintain continued contact until schools have built a relationship with the recruiter and fully understand 

TIMSS; and

 Make in-person visits to some schools, as necessary.

B.4 Purpose of Field Test and Data Uses

TIMSS. The central goals are to test out new assessment items and background questions, and to ensure that 

classroom and student sampling procedures proposed for the main study are successful.

TIMSS Advanced. As with the field test at grades 4 and 8, the main focus of a field test for TIMSS Advanced is to 

collect enough assessment data to perform reliable tests of the items. In addition, and in particular for TIMSS 

Advanced, the field test will allow us to evaluate procedures for conducting the main study, including recruitment 

methods for obtaining school and student participation, and identifying and sampling eligible schools and students.

B.5 Individuals Consulted on Study Design

Overall direction for TIMSS is provided by Dr. Stephen Provasnik, National Research Coordinator, National Center 

for Education Statistics, U.S. Department of Education.

The following persons are responsible for the statistical design of TIMSS:

 Pierre Foy. TIMSS International Study Center, Boston College (617-552-6253); and

 Marc Joncas and Jean Dumais, Statistics Canada (613-951-0007).

Westat is the contractor responsible for sampling and data analysis:

 Chris Averett, Project Director, Westat (301-314-2492); and

 David Ferraro, Senior Statistician, Westat (301-251-4261).

Analysis and reporting will be performed by:

 TIMSS International Study Center, Boston College;

 American Institutes for Research, under contract to Westat; and

 National Center for Education Statistics, U.S. Department of Education.