## TRENDS IN INTERNATIO **NAL MATHEMATICS AND**

**SCIENCE STUDY (TIMSS 2015) FIELD TEST AND**  **RECRUITMENT FOR FIEL D TEST AND MAIN STUD**  **Y**

**REQUEST FOR OMB CLEA RANCE**  **OMB# 1850 -**  **0695 v**  **.3**

**SUPPORTING STATEMENT PART B** 

**Submitted by:** 
**National Center for Education Statistics U.S. Department of Education**  **Institute of Education Sciences Washington, DC** 
**June 2013** 21



### B. **COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION B**  **.1 Respondent Universe**  **TIMSS.** The respondent universe for the TIMSS field test is all students enrolled in grade 4 and grade 8 during the 2013
- 2014 school year. The universe for the selection of schools is all types of schools in seven populous states. A sample of 4 0 schools will be selected for the field test, with the goal of obtaining participation from a minimum of 35 schools at each grade. Within sampled schools, students will be selected for participation by drawing a random sample of two classes. At grade 8, students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. All selected students will be asked to participate in a combined TIMSS mathematics and science ass essment. Only students in intact classrooms will be assessed at each grade. **TIMSS Advanced.** The respondent universe for the TIMSS Advanced field test is all students in the last year of secondary school who have taken or are taking advanced mathematics or physics courses as defined by the TIMSS 2015 Advanced Framework. For advanced mathematics , the framework includes algebra, geometry, and calculus content
. In the United States, algebra and geometry are pre
-requisites for calculus, so the focus will be on c alculus. The TIMSS Advanced F ramework includes limits, derivatives, and integrals as topic s ; all of which are typically included in calculus courses in the United States. Using the NCES Secondary School Course Classification System: School Codes for Exchange of Data (SCED) (http://nces.ed.gov/pubs2007/2007341.pdf ), the following courses , at a minimum, would be eligible for the advanced mathematics assessment: 02121 Calculus 02122 Multivariate Calculus 02123 Differential Calculus 02124 AP Calculus AB 02125 AP Calculus BC 02126 Particular Topics in Calculus (when it follows a survey calculus course) 02132 IB Mathematics The physics framework includes mechanics and thermodynamics, electricity and magnetism, wave phenomena , and atomic/nuclear physics. In the United States, this content is most likely to be covered in second
- year/advanced physics courses, and not in first
-year survey physics courses. The following courses in the SCED , at a minimum, would be eligible for the TIMSS Advanced physics assessment: 03152 Physics

—

Advanced Studies 03155 AP Physics B 03156 AP Physics C (either E&M or MECH) 03157 IB Physics Note that course eligibility will continue to be reviewed as the sampling frame is developed for U.S. participation. In particular, some students may have completed eligible coursework at a college level as a summer course, as a dual credit course, or through a distant learning program. The advanced mathematics and physics student populations are separate, but overlapping populations. All selected students will be asked to participate in either an advanced mathematics assessment or a physics assessment , but not both
. The universe for the selection of schools is all types of eligible schools in the same seven populous states as in grades 4 and 8 field test, where “eligibility of schools” is determined by having courses in their course catalogue (or available to enrolled students through colleges or virtual schools) that meet the content specific ations for advanced22

mathematics and physics in the TIMSS 2015 Advanced Frameworks. Note that eligibility can only be determined after the original sample of schools is selected. To maximize the likelihood that selected schools are in fact “eligible”, we wi ll consider using the NAEP High School Transcript Study (HSTS) schools from 2009 as the sampling frame for selecting schools. We will sample 61 schools with the goal of obtaining participation from a minimum of 35 eligible schools. Using student transcrip ts and current course enrollment, two lists of eligible students (one for advanced mathematics and one for physics) will be created from which to sample students. Eligible students are those who have taken or are currently taking the relevant courses. Students who are on both lists (eligible for both advanced mathematics and physics) will be randomly assigned to one sample.


### B.2 **Statistical Methodology Field Test Sampling Plan and Sample**  **TIMSS.** The purpose of the TIMSS field t est is to test out new assessment items and background questions, and to ensure that classroom and student sampling procedures proposed for the main study are successful. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize impact on these entities while also ensuring that the field test data are collected effectively. As required by the International Study Center, the field test sample is to consist of 40 schools at grade 4 and 8, with 1,400 students assessed at each of these grades. The student samples will be obtained by selecting two classes from each school (two mathematics classes for grade 8). As the field test is designed only to test items, questions, and procedures, a p robability sample of schools is not required. However, the sample must include a broad range of schools covering such features as public (including charter schools), private, large , small, urban , and rural schools, and schools from a variety of different s tates. In addition, the selection of the field test sample must take into account the fact that t he TIMSS field test will take place in the spring of 2014 after the NAEP 2014 assessment. NAEP 2014 will be a national sample and will involve several hundred schools at each of grades 4 and 8 ( and not several thousand as in the main, state
- sample NAEP in 2013 and 2015). Because the TIMSS assessment follows the NAEP assessment, we will select the TIMSS field test sample so as not to include any schools that are sampled for NAEP 2014
. The following steps describe the sampling approach for the TIMSS field test: 1. We will i dentify a set of seven states where we will conduct the field test, with the aim of gaining the cooperation of 7 schools per grade in each of the f our largest states, and 4 schools per grade in each of the three smallest states . With such an allocation , the selection of states should be skewed somewhat towards larger states, but include some smaller states. The states will be spread across different regions. This will provide an appropriate geographic diversity for the selected schools. 2. After identifying the eligible schools in each state, using the most current NCES CCD (public schools) and PSS (private schools) files, we will subset these by eliminating schools that (a) are unlikely to have two classes per grade, and (b) have been included in the NAEP 2014 sample. We will also eliminate schools that have been selected for any other NCES studies involving students at grades 4 or 8 during the 2013
- 14 s chool year . 3. We will then select a stratified probability sample of schools, ensuring representation by school type (approximately 4 of the 40 schools selected will be private schools), urban/rural, and race/ethnic composition.23

We will use probability sampl ing not because it is required, but because it is an efficient way to select a sample of schools representing a variety of school characteristics. 4. For each sampled school we will identify two replacement schools, matched exactly to the original sample in t erms of state and public/private status, and as closely as possible with regard to the other characteristics used to stratify the original sample. The replacement schools will be recruited in cases where the original sample school is unable to participate. Once the field test sample has been selected, a summary of the distribution of the characteristics of the selected schools will be prepared, showing the comparison with the national population of schools. The student sampling procedures for the field test will correspond as closely as feasible to what is planned for the main study, so as to try out the operational procedures for student sample selection. The sample will be selected by selecting two classes per school. Each participating school will be asked to submit an exhaustive list of classes (that is, one that accounts for each student in the grade exactly once). At grade 8 this will be a list of mathematics classes. Because in some cases at grade 8 a student can be enrolled in more than one mathemati cs class, the list of classes, and the students to be assessed, must be handled in such a way as to ensure that each student has only one chance of selection. In cases where one or more class on the list has fewer than 10 students, smaller classes will be combined to form “pseudoclasses” for the purposes of sampling. Once the list of classes is submitted, we will use a sampling algorithm to select two classes (or pseudoclasses) with equal probability. The student sample will then consist of all students in the selected classes. We plan to gather class and student lists from participating schools electronically using an adaptation of our secure E
- filing process (as explained in Part A)
. E
- filing was successfully used in PISA 2012, and provides advantageous features such as efficiency and data quality checks. Schools will access the E filing sy stem through the MyTIMSS web site. **TIMSS Advanced .**  Like TIMSS at grades 4 and 8, the purpose o f the TIMSS Advanced field t est is to test new assessment items and background questions
. Because of the nature of the TIMSS Advanced target population, the field test will also help ensure that school and student sampling procedures (including determining school and student eligibility) proposed for the main study are successful. T he TIMSS Advanced field test sample is to consist of 35 high schools , and assess 1,400 students (700 for advanced mathematics and 700 for physics ) . As with the TIMSS field test at grade 4 and 8, the TIMSS Advanced field test is designed only to test items, questions, and procedures ; so a probability sample of schools is not required. The TIMSS Advanced field test will also take place in the spring of 2014 after the NAEP 2014 assessment. NAEP 2014 will involve less than 100 schools at grade 12 ( and not almost two thousand as in 2013 and 2015). The following steps describe the sampling approach for the TIMSS Advanced field test: 1. In the same 7 states that we are conducting the grades 4 and 8 field test , we will select 10 high schools in each of the four largest states, and 7 high schools in each of the three smallest states
. Using the same 7 states will provide operational efficiency and appropriate geographic diversity for the selected schools. 2. After identifying the potentially eligible schools in each state, using the most current NCES CCD (public schools) and PSS (private schools) files, we will subset th ese by eliminating schools that have been included in the NAEP 2014 sample. 3. We will then select a stratified probability sample of 6 1 schools, ensuring representation by school type (approximately 10 percent of the schools selected will be private schools), urban/rural, and race/ethnic composition
.24

4. We will sampl e 61 schools to ensure that we have 35 eligible schools, where school eligibility is determined by offering at least one of the eligible courses for advanced mathematics and physics. We will obtain course catalogues from all 6 1 schools, and code the course catalogues according to the course criteria to determine eligible schools. Mathematics and science department chairs will also be asked to identify eligible courses from their catalogues. If more than 35 are eligible, we will select 35 for actual data col lection. 5. For each sampled school we will identify two replacement schools, matched to the original sample in terms of state and public/private status, and matched, as closely as possible , with regard to the other characteristics used to stratify the original sample. The replacement schools will be recruited in cases where the original sample school is unable to participate. Once the field test sample has been selected, a summary of the distribution of the characteristics of the selected schools will be prep ared, showing the comparison with the national population of schools. The student sampling procedures for the field test will correspond as closely as feasible to what is planned for the main study, so as to try out the operational procedures for student sample selection. The student samples will be obtained by conducting the following steps: 1. Collect electronic (or paper) student transcripts as well as current class enrollment of all twelfth grade students in the selected schools
. 2. Determine eligible students for both advanced mathematics and physics; these are students in the target population in the school , who have taken or are currently taking eligible courses
. 3. Draw random sample of 20 students for each assessment (advanced mathematics and physics) from those students eligible for each. If a student is in both populations, we will randomly assign the student to one subject. We plan to gather class , transcript , and student lists from participating schools electronically , using an adaptation of our secure E
- filing process. High schools will access the E filing system through the MyTIMSS web site. **Main Study Sampling Plan and Sample TIMSS**  **and TIMSS Advanced .**  The school sample design for the main study must be more rigorous than that for the field test. It must be a probability sample of schools (one sample for each of grades 4 , 8 , and 12 ) that fully represents the entire United States. At the same time, to ensure maximum participation it must be designed so as to minimize overlap with other NCES studies involving student assessment that will be conducted around the same time. The main study of both TIMSS and TIMSS Advanced will take place in the spring of 2015, ab out two months after a very large NAEP assessment. NAEP will assess several thousand schools nationally, at grades 4 , 8 , and 12 . The NAEP sample will be relatively heavy in smaller states, and in a number of these states all eligible schools will be included in NAEP, especially at grade 8 and 12 (for the 13 states included in grade 12 state NAEP) . Thus to be fully representative , the TIMSS sample may include some schools that will have participated in NAEP at the same grade. However, this number can be kept to minimum using the overlap control procedures outlined below. While PISA 2015 will take place in the fall of the subsequent school year to TIMSS, it will overlap with school recruiting. PISA will include a small number of grade 8 schools in the sample , but very few grade 8 students will be assessed in PISA, since very few grade 8 students meet the PISA 15 years of age requirements. Most of the PISA25

selected schools will include grade 9 and therefore also grade 12. Thus overlap control between the TIMSS and PI SA samples will be desirable, and should not be difficult to accomplish successfully. Overlap control procedures in studies such as this , where stratified probability proportional to size samples of schools are selected , can be implemented via a procedure that applies Bayes Theorem to modify the conditional probability of selection of a given school for one study, depending upon its selection probability for a second study, and whether or not it was selected for that study. This approach was first documented in a survey sampling application by Keyfitz (1951) 1 . The principles involved can be extended to more than two studies simultaneously, and a procedure for doing this is described by Chowdhury et al. (2000) 2

. The sample size for the TIMSS main study will be 300 schools at each of grades 4 and 8. The sample size for the TIMSS Advanced main study will be 250 high schools (that contain grade 12)
. For each original sample school, two replacement schools will also be identified. The sampling frames of grade 4 , 8 , and 12 schools will be obtained from the most current versions of NCES’s Common Core of Data (CCD) and Private School Survey (PSS) files, restricted to schools having grade 4 , 8 , or 12 respectively, and eliminating schools in Puerto Rico, U.S. territories, and Department of Defense overseas schools. The sample will be stratified according to school characteristics such as public/private, Census region, poverty status (as measured by the percentage of students in the school receiving free or red uced
-price lunch in the National School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the selected sample of schools. Determining school eligibility, student eligibility , and student sampling for TIMSS Advanced will be accomplished as described above for the field test. For TIMSS at grade 4 and 8 it will be accomplished as described below. Schools will be selected with probability proportional to the number of estimated classes at the appropriate grade (4 or 8), with schools expected to have either one or two classes being given the same selection probability. The use of a probability proportional to sample design ensures that all students have an approximately equal chance of selection, since two classes will be selected from each school regardless of the size of the school. Note that we will modify this equal probability design in the following way. So as to increase the available sample size of students in high poverty schools, we will double the probabil ity of selection of each school with at least 50 percent of students eligible for free or reduced
-price lunch under NSLP, relative to other schools of the same size. Student sampling will be accomplished by selecting two classes per school. Each grade 4 s chool will be asked to prepare a list of classes that is comprehensive, and includes each grade 4 student in the school in one of the listed classes. As described above, schools will submit these classes and student lists via secure E
- filing. Grade 8 schoo ls will be asked to prepare such a list also, but in this case the students should be organized into mathematics classes. At either grade , any class with fewer than ten students will be combined with another class to form a ‘pseudoclass’ with at least ten students in it. We will then select two classes (or pseudoclasses) from each school, with equal probability, and all students in those classes/pseudoclasses will be included in the sample. I f a school has only one or two classes , then all students in the grade will be included in the sample. At grade 8 , mathematics classes are used for three reasons. First, this minimizes the burden on mathematics teachers, as only two mathematics teachers

1 Keyfitz, N. (1951). S ampling With Probabilities Proportional to Size: Adjustment for Changes in Probabilities
. Journal of the American Statistical Association , 46, 105
- 109. 2 Chowdhury, S., Chu, A., & Kaufman, S. (2000). Minimizing overlap in NCES surveys. Proceedings of the Survey Methods Research Section, American Statistical Association, 174
- 179. Retrieved from http://www.amstat.org/sections/srms/ proceedings/papers/2000_025.pdf
.26

need to fill out a teacher questionnaire (but typically more than two science teachers are required to do so, since the students in the two selected math classes often attend more than two different science classes). Second, it makes for sound data for conducting analyses of the extent to which classroom factors moderate the relationship of student factors to achievement ( e.g., “Does having a well
- qualified math teacher reduce the correlation between math achievement and parental education?”). Third, at grade 8 , most students take one and only one mathematics class, and thu s mathematics classes make for a foolproof partitioning of the eligible students. Determining school eligibility for the TIMSS Advanced assessment at grade 12, as well as student eligibility and student sampling will be accomplished as described above fo r the field test. **Nonresponse Bias Analysis, Weighting, and**  **Sampling Errors** Itis inevitable that nonresponse will occur at both levels: school and student. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the calculation of weights , sampling errors will be calculated for a s election of key indicators incorporating the full complexity of the design, that is, clustering and stratification (see Appendix D for more detail)
. 

### B.3 **Maximizing Response Rates** With the recent exception of TIMSS 2011, the most significant challenge in recruitment for TIMSS has been engaging the schools and gaining their cooperation. The circumstances that aided our success in 2011

—the NAEP
TIMSS Linking Study and the involvement of NAEP State Coordinators

— may not recur in 2015. However, there are important lessons to be learned from that experience that can be used regardless of the overall TIMSS 2015 program. Given that classrooms are selected, student participation is not as great of a challenge. H istorically student participation rates have never fallen below 90 percent (see table 1) . That said, it is important to U.S. TIMSS that students are engaged and try hard on the assessment, which could be an issue at grade 8 and 12
. 

### Table 1 **. Historical TIMSS school and student participation rates**  **Year Grade**  **School Participation Rate Overall Student Participation Rate**  **Before Replacement After Replacement**  **2011** 4 79 84 95

8 87 87 94 **2007** 4 70 89 95

8 68 83 93 **2003** 4 70 82 95

8 71 78 94 **1999** 8 83 90 94 **1995** 4 86 NA 94 Our approach to school recruitment is to:

 Obtain endorsements about the value of TIMSS from relevant organizations;

 Inform Chief State Officers and test directors about the sample of schools in their state. Enclose a sample letter of endorsement they can send to schools;

 Send letters and informational materials to schools and districts. These letters will be customized by type of school;27

 Train experienced school recruiters about TIMSS;

 Implement strategies from NAEP’s Private School Recruiting Toolkit. This toolkit, developed f or NAEP, includes well
-honed techniques used to recruit a very challenging type of school s ;

 Followup mailings with telephone calls to explain the study and schools involvement, including placing the TIMSS assessment date on school calendars;

 Maintain continued contact until schools have built a relationship with the recruiter and fully understand TIMSS; and

 Make in
-person visits to some schools , as necessary . 

### B.4 **Purpose of Field Test and Data Uses TIMSS.**  The central goal s are to test out new assessment items and background questions, and to ensure that classroom and student sampling procedures proposed for the main study are successful. **TIMSS Advanced.** As with the field test at grades 4 and 8, the main focus of a field test for TIMSS Advanced is to colle ct enough assessment data to perform reliable tests of the items. In addition, and in particular for TIMSS Advanced, the field test will allow us to evaluate procedures for conducting the main study, including recruitment methods for obtaining school and s tudent participation, and identifying and sampling eligible schools and students.


### B.5 **Individuals Consulted on Study**  **Design** Overall direction for TIMSS is provided by Dr. Stephen Provasnik, National Research Coordinator, National Center for Education Statistics, U.S. Department of Education
. The following persons are responsible for the statistical design of TIMSS:

 Pierre Foy
. TIMSS International Study Center, Boston College (617
- 552
- 6253) ; and

 Marc Joncas and Jean Dumais, Statistics Canada (613
- 951
- 0007)
. Westat is the contractor responsible for sampling and data analysis:

 Chris Averett , Project Director, Westat (301
- 314
- 2492 ) ; and

 David Ferraro, Senior Statistician, Westat (301
- 251
- 4261)
. Analysis and reporting will be performed by:

 TIMSS International Study Center, Boston College ;
 American Institutes for Research, under contract to Westat ; and

 National Center for Education Statistics, U.S. Department of Education
.