TRENDS IN INTERNATIONAL MATHEMATICS AND 

SCIENCE STUDY 2015 (TIMSS 2015) MAIN STUDY DATA 

COLLECTION

REQUEST FOR OMB REVIEW

OMB# 1850-0695 v.5

SUPPORTING STATEMENT PART B

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

July 2014
1

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe 

TIMSS

The respondent universe for the TIMSS main study is all students enrolled in grades 4 and 8 during the 2014-15 

school year. National probability samples of schools containing a grade 4 or grade 8 were drawn in April 2014 and 

state probability samples for Florida were drawn in July 2014. All selected students will be asked to participate in a 

combined mathematics and science assessment.

At grades 4 and 8, a national sample of 300 schools each was selected with the goal of obtaining participation from 

a minimum of 255 at each grade level. The universe for the selection of schools was all types of schools containing 

a grade 4 or grade 8 in the 50 states and the District of Columbia, Department of Defense (DoD) domestic schools, 

and Bureau of Indian Education (BIE) schools. The overlap with the 2015 National Assessment of Educational 

Progress (NAEP) school sample will be minimized for TIMSS 2015. Up to two intact classes will be assessed in 

each school when possible, with an expected yield of at least 4,000 students for each grade.

A state sample of schools was drawn for Florida to produce a total state sample of 50 additional schools at each 

grade level, with the expectation that 99 of the 100 total Florida schools will participate. The sample design for the 

TIMSS Florida state samples at grades 4 and 8 will closely follow the TIMSS national design where possible. The 

school frame will be identical to the national frame of public schools in Florida. The school samples will be a twostage design—a stratified systematic sample of schools with sampling probabilities proportional to size (PPS) and 

then classes within sampled schools. Schools will be stratified by poverty level, locale, race/ethnicity status, and 

estimated grade enrollment. A sample of 55 schools will be drawn at each grade to attain the target sample size of 

50 eligible schools. Ideally, the objective for the Florida state samples is that they will not include the schools that 

were previously selected as part of the TIMSS national samples. By following the Keyfitz procedure outlined in 

table 2 of Chowdhury, Chu, and Kaufman (2000), the procedure allows us to minimize the overlap with the TIMSS 

national and NAEP public school samples. Designating substitute schools and classroom sampling will follow the 

national procedures.

TIMSS Advanced.

The respondent universe for the TIMSS Advanced is all students in the last year of secondary school who have 

taken or are taking advanced mathematics or physics courses as defined by the TIMSS 2015 Advanced Framework.

For advanced mathematics, the framework includes algebra, geometry, and calculus content. In the United States, 

algebra and geometry are pre-requisites for calculus, so the focus will be on calculus. The TIMSS Advanced 

Framework includes limits, derivatives, and integrals as topics; all of which are typically included in calculus 

courses in the United States. Using the NCES Secondary School Course Classification System: School Codes for 

Exchange of Data (SCED) (http://nces.ed.gov/pubs2007/2007341.pdf ), the following courses, at a minimum,

would be eligible for the advanced mathematics assessment:

02121 Calculus

02122 Multivariate Calculus

02123 Differential Calculus

02124 AP Calculus AB

02125 AP Calculus BC

02126 Particular Topics in Calculus (when it follows a survey calculus course)

02132 IB Mathematics
2

The physics framework includes mechanics and thermodynamics, electricity and magnetism, wave phenomena, and 

atomic/nuclear physics. In the United States, this content is most likely to be covered in second-year/advanced 

physics courses, and not in first-year survey physics courses. The following courses in the SCED, at a minimum,

would be eligible for the TIMSS Advanced physics assessment:

03152 Physics—Advanced Studies

03155 AP Physics B

03156 AP Physics C (either E&M or MECH)

03157 IB Physics

Note that course eligibility will continue to be reviewed as the sampling frame is developed for U.S. participation.

In particular, some students may have completed eligible coursework at a college level as a summer course, as a 

dual credit course, or through a distant learning program.

The advanced mathematics and physics student populations are separate, but overlapping populations. All selected 

students will be asked to participate in either an advanced mathematics assessment or a physics assessment, but not 

both.

The universe for the selection of schools is all types of eligible schools in the 50 states and the District of 

Columbia, DoD domestic schools, and BIE schools, where “eligibility of schools” is determined by having courses 

in their course catalogue (or available to enrolled students through colleges or virtual schools) that meet the content 

specifications for advanced mathematics and physics in the TIMSS 2015 Advanced Frameworks. Note that 

eligibility can only be determined after the original sample of schools is selected. To maximize the likelihood that

selected schools are in fact “eligible”, we will use a file provided by the College Board that identifies schools where 

AP calculus and/or AP physics exams were taken in 2014. We will sample 350 schools with the goal of obtaining 

participation from a minimum of 300 eligible schools. The overlap with NAEP 2015 school sample will be 

minimized for TIMSS Advanced.

Using student transcripts and current course enrollment, two lists of eligible students (one for advanced 

mathematics and one for physics) will be created from which to sample students. Eligible students are those who 

have taken or are currently taking the relevant courses. Students who are on both lists (eligible for both advanced 

mathematics and physics) will be randomly assigned to one sample. The TIMSS Advanced main study sample is to 

consist of high schools with a target of 4,000 students in advanced mathematics and physics for a total of 8,000 

students, and a goal of 3,600 assessed students per subject.

B.2 TIMSS and TIMSS Advanced Statistical Methodology

Main Study Sampling Plan and Sample

The school sample design for the main study is a probability sample of schools (one sample for each of grades 4, 8,

and 12) that fully represents the entire United States. At the same time, to ensure maximum participation it must be 

designed so as to minimize overlap with other NCES studies involving student assessment that will be conducted 

around the same time.

The main study of both TIMSS and TIMSS Advanced will take place in the spring of 2015, about two months after 

a very large NAEP Main 2015 assessment. NAEP will assess several thousand schools nationally, at grades 4, 8,

and 12. The NAEP sample will be relatively heavy in smaller states, and in a number of these states all eligible 

schools will be included in NAEP, especially at grade 8. Thus to be fully representative, the TIMSS sample may
3

include some schools that will have participated in NAEP at the same grade. However, this number can be kept to 

minimum using the overlap control procedures outlined below. 

While PISA 2015 will take place in the fall of the subsequent school year to TIMSS, it will overlap with school 

recruiting. PISA will include a small number of grade 8 schools in the sample, but very few grade 8 students will be 

assessed in PISA, since very few grade 8 students meet the PISA 15 years of age requirements. Most of the PISA 

selected schools will include grade 9 and therefore also grade 12. Thus overlap control between the TIMSS and 

PISA samples will be desirable, and should not be difficult to accomplish successfully. 

Overlap control procedures in studies such as this, where stratified probability proportional to size samples of 

schools are selected, can be implemented via a procedure that applies Bayes Theorem to modify the conditional 

probability of selection of a given school for one study, depending upon its selection probability for a second study, 

and whether or not it was selected for that study. This approach was first documented in a survey sampling 

application by Keyfitz (1951)1. The principals involved can be extended to more than two studies simultaneously, 

and a procedure for doing this is described by Chowdhury et al. (2000)2.

The sample size for the TIMSS main study will be 300 schools at each of grades 4 and 8. The sample size for the 

TIMSS Advanced main study will be 350 high schools (that contain grade 12), with the aim to obtain about 300 

eligible schools. For each original sample school we will identify two replacement schools, matched to the original 

sample in terms of state and public/private status, and matched, as closely as possible, with regard to the other 

characteristics used to stratify the original sample. The replacement schools will be recruited in cases where the 

original sample school is unable to participate. The sampling frames of grade 4, 8, and 12 schools will be obtained 

from the most current versions of NCES’s Common Core of Data (CCD) and Private School Survey (PSS) files, 

restricted to schools having grade 4, 8, or 12 respectively, and eliminating schools in Puerto Rico, U.S. territories, 

and Department of Defense overseas schools.

The sample will be stratified according to school characteristics such as public/private, Census region, poverty 

status (as measured by the percentage of students in the school receiving free or reduced-price lunch in the National 

School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the 

selected sample of schools.

Determining school eligibility, student eligibility, and student sampling for TIMSS Advanced will be accomplished 

as described above. For TIMSS at grade 4 and 8, it will be accomplished as described below.

Schools will be selected with probability proportional to the number of estimated classes at the appropriate grade (4 

or 8), with schools expected to have either one or two classes being given the same selection probability. The use of 

a probability proportional to sample design ensures that all students have an approximately equal chance of 

selection, since two classes will be selected from each school regardless of the size of the school. Note that we will 

modify this equal probability design in the following way. So as to increase the available sample size of students in 

high poverty schools, we will double the probability of selection of each school with at least 50 percent of students 

eligible for free or reduced-price lunch under NSLP, relative to other schools of the same size.

1

Keyfitz, N. (1951). Sampling with Probabilities Proportional to Size: Adjustment for Changes in Probabilities. Journal of the 

American Statistical Association, 46, 105-109.

2 Chowdhury, S., Chu, A., & Kaufman, S. (2000). Minimizing overlap in NCES surveys. Proceedings of the Survey Methods 

Research Section, American Statistical Association, 174-179. Retrieved from http://www.amstat.org/sections/srms/

proceedings/papers/2000_025.pdf.
4

Student sampling will be accomplished by selecting two classes per school. Each grade 4 school will be asked to 

prepare a list of classes that is comprehensive, and includes each grade 4 student in the school in one of the listed 

classes. As described above, schools will submit these classes and student lists via secure E-filing. Grade 8 schools 

will be asked to prepare such a list also, but in this case the students should be organized into mathematics classes. 

At either grade, any class with fewer than ten students will be combined with another class to form a ‘pseudoclass’ 

with at least ten students in it. We will then select two classes (or pseudoclasses) from each school, with equal 

probability, and all students in those classes/pseudoclasses will be included in the sample. If a school has only one 

or two classes, then all students in the grade will be included in the sample. At grade 8, mathematics classes are 

used for three reasons. First, this minimizes the burden on mathematics teachers, as only two mathematics teachers 

need to fill out a teacher questionnaire (but typically more than two science teachers are required to do so, since the 

students in the two selected math classes often attend more than two different science classes). Second, it makes for 

sound data for conducting analyses of the extent to which classroom factors moderate the relationship of student 

factors to achievement (e.g., “Does having a well-qualified math teacher reduce the correlation between math 

achievement and parental education?”). Third, at grade 8, most students take one and only one mathematics class, 

and thus mathematics classes make for a foolproof partitioning of the eligible students. 

School eligibility for the TIMSS Advanced assessment at grade 12 will be determined by:

1. Obtaining course catalogues from school and district websites;

2. Analyzing catalogues to determine courses that appear to be eligible for either advanced mathematics or 

physics;

3. Contacting schools in the fall of 2014 to finalize eligibility for the 2014-2015 school year.

As we did in the TIMSS Advanced field test, lists of students who have taken or are taking the eligible courses at

the school (or through the school virtually or in colleges) will be obtained for student sampling. Details of student 

sampling are described below:

1. Select all students eligible for both physics and math, up to a total of 40. Otherwise take a sample of size 

40.

2. Assign 1/5 of the doubly-eligible students to the math assessment and the rest to physics.

3. Select all students eligible for physics only, up to a total of 20. Otherwise take a sample of size 20. This 

may be modified to take all up to a total of 40.

4. Let the number of math-only students in the school be M. Take a sample of size

MIN(MIN(M,MAX(10, M/4)), 40).

That is, we will sample one-quarter of the math-only students, subject to a maximum of 40 and a minimum of 10 

(or all students if there are fewer than 10).

Based on data from 2009 HSTS, on average in a school with 500 twelfth-grade students we expect:

A. 17 students eligible for both math and physics. We will sample all of them, and assign 3 or 4 to math and 

13 or 14 to physics.

B. 6 students eligible for physics only. We will assess all in physics.

C. 67 students eligible for math only. We will sample 16 or 17 of them.
5

Therefore, we will sample 39 or 40 students from such a school, with about equal numbers assessed in math and 

physics.

The maximum sample size in any school will be 100 students (or 120 if we permit up to 40 physics-only students). 

This would be in a very unusual school, that had at least 40 double specialists, at least 20 physics-only specialists 

(or 40), and at least 160 math-only specialists; so at least 220 specialists (or 240), when specialists are 18 percent of 

the total U.S. student population.

Nonresponse Bias Analysis, Weighting, and Sampling Errors

It is inevitable that nonresponse will occur at both levels: school and student. We will analyze the nonrespondents 

and provide information about whether and how they differ from the respondents along dimensions for which we 

have data for the nonresponding units, as required by NCES standards. After the calculation of weights, sampling 

errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, 

clustering and stratification (see Appendix D for more detail).

B.3 Maximizing Response Rates

With the recent exception of TIMSS 2011, the most significant challenge in recruitment for TIMSS has been 

engaging the schools and gaining their cooperation. The circumstances that aided our success in 2011—the NAEPTIMSS Linking Study and the involvement of NAEP State Coordinators—may not recur in 2015. However, there 

are important lessons to be learned from that experience that can be used regardless of the overall TIMSS 2015 

program. Given that classrooms are selected, student participation is not as great of a challenge. Historically student 

participation rates have never fallen below 90 percent (see table 1). That said, it is important to U.S. TIMSS that 

students are engaged and try hard on the assessment, which could be an issue at grade 8 and 12.

Table 1. Historical TIMSS school and student participation rates 

Year Grade School Participation Rate Overall Student Participation Rate Before Replacement After Replacement

2011

4 79 84 95

8 87 87 94

2007

4 70 89 95

8 68 83 93

2003

4 70 82 95

8 71 78 94

1999 8 83 90 94

1995 4 86 NA 94

Our approach to school recruitment is to:

 Obtain endorsements about the value of TIMSS from relevant organizations;

 Inform Chief State Officers and test directors about the sample of schools in their state. Enclose a sample 

letter of endorsement they can send to schools; 

 Send letters and informational materials to schools and districts. These letters will be customized by type of 

school;

 Train experienced school recruiters about TIMSS;

 Involve NAEP State Coordinators in gaining and maintaining cooperation of schools selected for TIMSS and 

TIMSS Advanced. These Coordinators have been highly successful in gaining cooperation for NAEP.
6

 Implement strategies from NAEP’s Private School Recruiting Toolkit. This toolkit, developed for NAEP, 

includes well-honed techniques used to recruit a very challenging type of schools;

 Follow-up mailings with telephone calls to explain the study and schools involvement, including placing the 

TIMSS assessment date on school calendars;

 Maintain continued contact until schools have built a relationship with the recruiter and fully understand 

TIMSS; and

 Make in-person visits to some schools, as necessary.

Gaining cooperation of high school seniors is a challenge, so additional strategies will be used for TIMSS 

Advanced. We learned from NAEP assessments at grade 12 that students are responsive to the adults in the school, 

particularly their teachers and principal. Since most of the sampled students will be taking Advanced Placement 

(AP) courses, we will obtain letters of endorsement from the College Board (which overseas AP courses) for 

principals and teachers, asking them to encourage students to participate in the assessment. In order to illustrate the 

value of TIMSS and TIMSS Advanced, we will offer optional professional development web seminars throughout 

the school year on TIMSS and educational policy, as well as using released TIMSS questions in classroom 

instruction. We will also develop a short motivational video for students that talks about the value of TIMSS 

Advanced to understanding the U.S. international competitiveness in science, technology, engineering, and 

mathematics, that appeals to their own competiveness and desire to show how much they know and can do. As an 

additional incentive, participating students will receive a volunteer service certificate of 4 hours from the U.S. 

Department of Education. Assessments will be scheduled so as not to interfere with final preparations for AP 

exams and the exams themselves. NAEP State Coordinators and data collection staff will also share strategies for 

improving student participation with school staff, such as using the student video, announcements in parent 

meetings and student meetings, and offering pizza or other treats to students who participate.

B.4 Purpose of Field Test and Data Uses

During the TIMSS Advanced field test, we assessed approximately 1,000 12th grade calculus and physics students 

in 32 schools during March and April 2014. Seventy-eight public and private schools were contacted overall. We 

learned that gaining cooperation was more difficult than we anticipated, primarily due to (a) late notification of 

selection, (b) conflicts with state assessments and Common Core assessment pilots in the same assessment window, 

(c) difficulty with scheduling assessments the closer the TIMSS Advanced assessment date got to the AP testing 

window (the first two weeks in May), and (d) schools protecting instructional time due to many weather-related 

school closures in the winter of 2014. To improve school participation in the main study, we are contacting schools 

prior to the next school year (almost one year in advance) and providing them with assessment dates so they can put 

the TIMSS assessment date on their school calendar. The assessment window in the main study is much longer 

than in the field test, so we are able to schedule the grade-12 assessments early in April, or after the AP assessment 

window in the first two weeks of May 2015. We are continuing to be as flexible as possible in rescheduling 

assessment dates to accommodate school conflicts. We also collected information about known district and state 

spring breaks, teacher workdays, and local assessment dates in order to facilitate trying to schedule around them. 

In addition, during the field test we learned that advanced physics students are even rarer than we originally 

estimated, and we have increased our grade-12 school sample size to compensate for this fact. We learned that 

schools can provide lists of eligible students electronically, which will improve the efficiency and quality of student 

sampling procedures. We also were able to identify eligible courses for each school from their websites, and then 

confirm these courses in a short phone call with school administrators, likely reducing school burden. Lastly, more 

12th grade students than usual walked out of the field test assessments once they started, so we are enhancing our 

messaging to students about the value of participation and obtaining additional endorsement letters as described in 

section B.3. We will continue to refine all these processes in the main study. 
7

B.5 Individuals Consulted on Study Design

Overall direction for TIMSS is provided by Dr. Stephen Provasnik, National Research Coordinator, National Center 

for Education Statistics, U.S. Department of Education.

The following persons are responsible for the statistical design of TIMSS:

 Pierre Foy. TIMSS International Study Center, Boston College (617-552-6253); and

 Marc Joncas and Jean Dumais, Statistics Canada (613-951-0007).

Westat is the contractor responsible for sampling and data analysis:

 Chris Averett, Project Director, Westat (301-314-2492); and

 David Ferraro, Senior Statistician, Westat (301-251-4261).

Analysis and reporting will be performed by:

 TIMSS International Study Center, Boston College;

 American Institutes for Research, under contract to Westat; and

 National Center for Education Statistics, U.S. Department of Education.