1

**THE IMPACT OF PROFESSIONAL DEVELOPMENT IN FRACTIONS FOR FOURTH GRADE TEACHERS ON STUDENT ACHIEVEMENT AND TEACHER KNOWLEDGE IN GEORGIA**  **AND SOUTH CAROLINA OMB Clearance Request**  **Supporting Statement Part B**  **November 2**  **6 , 2013**  Prepared for: NCEE Contracting Officer’s Representative: Sandra Garcia U.S. Department of Education Institute of Education Sciences 555 New Jersey Ave., NW, Rm. 308 Washington, DC 20208 (202) 208 - 7078 Submi tted by: Regional Educational Laboratory Southeast at the Florida Center for Reading Research

– Florida State University 2010 Levy Avenue, Suite 100 Tallahassee, FL 32310 (850) 644 - 9352 Lab Director: Barbara Foorman, Ph.D. Florida State University 2010 Levy Avenue Suite 100 Tallahassee, FL 32310 (850) 644 - 9352 rel - se@fsu.edu http://rel - se.fsu.edu Principal Investigator: Russell Gersten, Ph.D. Instructional Research Group 4821 Katella Avenue Suite 205 Los Alamitos, CA 90720 Phone: (714) 826 - 9600 Fax: (714) 826 - 9610 http://www.inresg.org2 **SUPPORTING STATEMENT Part B**  **FOR PAPERWORK REDUCTION ACT SUBMISSION**

**Contents** B. **Data Collection Procedures and Statistical Methods**  ................................ ................................ .... 4 B1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government u nits, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection ................................ ................................ ................................ ................. 4 B2. Describe the procedures for the collection of information ................................ ............................ 5 B3. Describe methods to maximize response rates and to deal with issues of non - response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not y ield "reliable" data that can be generalized to the universe studied ................................ ................................ ................................ ............................... 6 B4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information ................................ ............ 6 B5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency ................................ ........ 73 **Appendi ces Referenced in Part B**  Appendix F : Details for Question B1 Appendix G : Details for Question B24

**SECTION B: B1. Describe (including a numerical estimate) the potential respondent universe and any**  **sampling or other respondent selection methods to be used. Data on the number of entities**  **(e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in**  **tabular form for the universe as a whole a nd for each of the strata in the proposed sample.**  **Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.**  Full details of the re cruiting and sampling plan can be found in the attached design document included in Appendix F . The study design was approved by

NCEE/IES at the end of June 2013. This study

is not intended to formally generalize to any

population beyond the sample included in the study

itself. The study

focuses on grade 4 teachers and their students from Georgia and South Carolina . The sample consists of a non - representative group of rural, small - town and urban schools from 6 districts in these 2 states. S chools will be recruited from each of those three urbanicity

categories. It is estimate d that at least 8 0 schools , 240 teachers, and 4560 students will be enrolled in the study . ( Initial power estimates suggest than a minimum of 80 schools is necessary for an MDES of g = .12. However, 84 schools will be recruited to participate in the study to account for possible school - level attrition.) Eligible schools will be restricted to tho se below the state median in achievement as based on the prior- year school - wide state assessment results. By definition eligible schools represent 50% of the schools in each of the two states. Lo cal Education Agencies ( LEAs ) will be recruited based on meeting these very

basic criteria and on the perceived level of commitment of school administrators to the study . This commitment will be evidenced by the willingness of district and school administrators to sign MOUs (See Appendix D for District/School Memorandum of Understanding). These schools (and their teachers and students) represent a sample of convenience and are not intended to generalize beyond the sample itself. In schools which have agreed to participate, all grade 4 teachers will be invited in May

– June 2014 to participate in the study . Those returning a signed consent form will be enrolled in the study and be eligible for random assignment . For students, a waiver of informed consent was granted from the IRB, as the intervention is focused on teachers, the intervention represents a professional development activity typical of general LEA practice, and students will only participate in a short post - test at the end of the ye ar. All students in classrooms where teachers have consented will participate, unless parents opt - out. Information forms will be distributed to all parents at the beginning of the school - ye ar with the opportunity to opt out (See Appendix E for Parent/Guardian Information L etter and Opt - out). If parents return the signed form, their5

child will be excluded from the fractions assessment. The information and opt - out forms will be translated and made available in Spanish as well. **B2. Describe the procedures for the collection of information including:** 

・ **Statistical methodology for stratification and sample selection,**

・ **Estimation procedure,**

・ **Degree of accuracy needed for the purpose described in the justification,**

・ **Unusual problems requiring specialized sampling procedures, and**

・ **Any use of periodic (l**

**ess frequent than annual) data collection cycles to reduce burden.**  Full details of the sampling and anal ytic plan can be found in the attached design document included in Appendix G . Design ED intend s to conduct a multi - level analysis (confirmatory) of student outcomes, comparing students whose teachers participated in DMI, versus students whose teachers participated in business - as - usual professional development activities. Multi - level models will appropriatel y account for the natural clustering of students within classrooms and teachers within schools. Another multi - level analysis (exploratory) of teacher knowledge outcomes will be conducted to compare teachers who participated in DMI versus teachers who participated in business - as - usual professional development activities . This multi - level model will account for clustering of teachers within schools. M atched school -pairs will be created (i.e. blocking schools within pairs), then a school in each pair will be randomly assigned to the treatment conditio n. A What Works Clearinghouse - ty pe (WWC) literature review was conducted in the domain of mathematics professional development to determine an appropriate minimum detectable effect size (MDES) 1 to design the study to detect. Using other parameters from the research literature for R2 between pretest covariates and the outcome measure, and ICCs at different levels (i.e. teachers/schools), a power anal ys is was conducted and resulted in an estimate of approximately 80 schools required to achieve the desired M DES ( g ≈ . 12 ). However, at least 84 schools will be recruited to allow for some school - level attrition. Assuming 80 schools for the student outcome and three participating teachers per school (N=240 teachers), an MDES of g ≈ .33 was estimated for the teacher outcome.

1 Minimum detectable effect size (MDES) stands for the minimally detectable differences in means between two groups, expressed in standard deviation units .6

Full details (Appendix G) about the multi - level model, and assumption s made for the power anal ys is are included from the study proposal approved b y NCEE/IES. **B3. Describe methods to maximize response rates and to deal with issues of**  **non -** **response. The accuracy and reliability of information collected must be shown to be adequate for intended**  **uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can**  **be generalized to the universe studied.** Districts and schools will be enrolled in the study based on perceived commitment, and completion of MOUs which explicitly delineate LEA commitments. Teachers will sign consent forms and be informed that it represent s a contract, a commitment to participate in an important scientific activity . In our experience, this requirement helps retain teachers at rates of over 90%. The IRB for this study

has granted a waiver of active consent for students. Parents will be sent a letter informing them of the study , of its importance, and of the very small amount of time (30 - 40 minutes) their child will be asked to participate. Under such circumstances, in previous professional development ( PD ) research, the research team has experienced a parent refusal rate of < 5%. Study staff will carefull y monitor teacher engagement through the completion of monthly teacher logs. Methods to reduce attrition and nonresponse will include i nvitation emails prompting teachers to complete monthly logs, use of a monthly PD online survey

to maximize convenience, and gathering of students in group setting s to administer post - test . Follow - up e - mails and reminder calls will be sent to all non - respondents. Teachers in both the treatment and control group will receive a remuneration of $ 1 50 for completion of the consent/demographic form, teacher knowledge assessments and monthly PD survey s, which should help reduce attrition. **B4. Describe any tests of procedures or methods to be undertaken. Testing is**  **encouraged as an effective means of refining collections of information to minimize burden and improve utility.**  **Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submi**  **tted for approval separately or in combination with the main collection of information.**  A small feasibility

study was performed with five grade 4 teachers to evaluate both instructions and testing time for the t eacher measures . Teacher Measures: Five grade 4 teachers were recruited from Southern California school districts via a Craigs l ist posting. The teacher consent and demographic form and teacher PD log were administered in sequence in a paper format. Teachers who piloted the forms were paid approx imately $150 upon completion of the pilot.7

Teachers attended proctored test sessions individually . With the exception of requesting timing information for the demographic form, proctors followed the protocol intended for the large scale study. Teachers we re first given the demographic form to complete. Teachers wrote their own start and completion time on the form. After writing the completion time they were asked to write down any components they found confusing, or any

general comments. The demographic f orms were returned to the proctors. The teacher professional development (PD) survey form was distributed. Proctors noted the time of distribution. Teachers read the instructions on the form and completed them individually . The MKT teacher knowledge test was not piloted, as it has been used extensively in other studies and consistently

takes under an hour to complete. This information was used to estimate burden hours and future pay ments in A12 ( see Table A2). The burden on teacher s for the data col lection instruments is approximately .2 hours for the Teacher Consent Forms with Demographics, 4 hours for the Teacher Assessment (Pre & Post) , and 1.8 hours for the monthly PD survey . The total burden hours are 6 for each teacher, totaling approximately 1512 hours for the 252 teachers in the study . Student Measure (TUF): The student measure has not been piloted as the final TUF form will not be available from the Center for Improving Learning of Fractions (CILF) until June 2014. CILF has scheduled two iterative pilots (beginning Winter 201 4 ) for the development process where items, instructions , and timing will be repeatedly revised and refined. **B5. Provide the name and telephone number of individuals consulted on statistical aspects of**  **the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who**  **will actually collect and/or analyze the information for the agency.** The following individuals were consulted on the statistical aspects of this proposal: Dr. Sy billa Beckman , University of Georgia

– ph# 706 - 542 - 2548 Dr. John Deke, Mathematica Policy Research

– ph# 609 - 275 - 2230 Dr. Mike Garet, American Institutes for Research

– ph# 202 - 403 - 5000 Dr. Russell Gersten, Instructional Research Group

– ph# 714 - 826 - 9600 Dr. Nathan Jones, Boston University

– ph# 617 - 353 - 3295 Dr. Yaacov Petscher, Florida State University – ph# 850 - 644 - 0327 Dr. Eric Rolfhus, Instructional Research Group

– ph# 714 - 826 - 9600 Dr. Mengli Song, American Institutes for Research

– ph# 202 - 403 - 5000 The following individual designed the data collection: Dr. Russell Gersten, Instructional Research Group8

The following individual will oversee the data collection: Dr. Russell Gersten, Instructional Research Group The following individual(s) will anal yz e the data: Dr. Tran Key s , Instructional Research Group

– ph# 714 - 826 - 9600 Dr. Rebecca Newman - Gonchar, Instructional Research Group

– ph# 714 - 826 - 9600 Dr. Madhavi Jayanthi, Instructional Research Group

– ph# 714 - 826 - 9600 Ms. Kelly Hay mond, Instructional Research Group

– ph# 714 - 826 - 9600 Dr. Mengli Song, American Institutes for Research

– ph# 202 - 403 - 5000