1

**THE IMPACT OF PROFESSIONAL DEVELOPMENT IN FRACTIONS FOR FOURTH GRADE TEACHERS ON STUDENT ACHIEVEMENT AND TEACHER KNOWLEDGE IN GEORGIA**  **AND SOUTH CAROLINA OMB Clearance Request**  **Supporting Statement Part A**
**November 2**  **6 , 2013**  Prepared for: NCEE Contracting Officer’s Representative: Sandra Garcia U.S. Department of Education Institute of Education Sciences 555 New Jersey Ave., NW, Rm. 506C Washington, DC 20208 (202) 2 19 - 1597 Submitted by: Regional Educational Laboratory Southeast at the Florida Center for Reading Research

– Florida State University 2010 Levy Avenue, Suite 100 Tallahassee, FL 32310 (850) 644
- 9352 Lab Director: Barbara Foorman, Ph.D. Florida State University 2010 Levy Avenue Suite 100 Tallahassee, FL 32310 (850) 644
- 9352 rel
- se@fsu.edu http://rel
- se.fsu.edu Principal Investigator: Russell Gersten, Ph.D. Instructional Research Group 4821 Katella Avenue Suite 205 Los Alamitos, CA 90720 Phone: (714) 826
- 9600 Fax: (714) 826
- 9610 http://www.inresg.org2 **Contents** Supporting Statement for Paperwork Reduction Act Submission A. Justification
................................ ................................ ................................ ................................ .... A1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information
................................ ................................ ................................ .......... 5 A2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection
................................ ................................ ................................ ....... 7 A3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or any other technological collection techniques or forms of information technology, e.g. permitting electronic submission of responses, and the basis for the decision of adopting the means of collection. Also describe any consideration of using information technology to reduce burden
................................ ........... 1 2 A4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use of the purposes described in Item 2 a bove
................................ ................................ ................................ ................................ ........ 1 2 A5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83
- 1), describe any methods used to minimize burden
................................ ......... 1 3 A6. Describe the consequences to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or leg al obstacles to reducing burden
................................ ................................ ................................ ....................... 1 3 A7. Explain any special circumstances that would cause an information collection to be conducted in a manner inconsistent with Section 1320.5(d)(2) of the federal regulations
..... 1 4 A8. Federal Register Comments and Persons Consulted Outside of the Agency
........................... 1 5 A9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contrac tors or grantees ................................ ................................ ................. 1 6 A10. Describe any assurances of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy
................................ ................................ ... 1 63

A11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. The justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent
................................ ................................ ................................ .................. 1 7 A12. Provide estimates of the hour burden of the collection of information
................................ .... 1 7 A13. Describe any other costs to respondents or record keepers
................................ .................... 19 A14. Provide estimates of annualized cost to the Federal government. Also provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expenses that would not have been incurred without this collection information. Agencies may aggregate cost estimates from Items 12, 13, and 14 in a single table
................................ ................................ ................................ ................................ .......... 19 A15. Describe any changes in the burden from prior approvals
................................ ..................... 20 A16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of the report, publication dates, and other actions
................................ ................................ ................................ ................................ ...... 20 A17. Describe arrangements for displaying the number provided by OMB and its expiration date
................................ ................................ ................................ ................................ ........... 2 1 A18. Exceptions to Certification Statement
................................ ................................ ...................... 22

References
................................ ................................ ................................ ................................ ............ 2 34 **Appendices Referenced in Part A**  Appendix A: Approved Teacher Consent & Demographic Form Appendix B: MKT Appendix C: Teacher Professional Devel opment Survey Appendix D: District/School Memorandum of Understanding (MOU) Appendix E: Parent/Guardian Information Letter and Opt
- out Appendix H : Process for sele cting DMI from possible programs Appendix I : Placeholder for 60
- Day and 30
- Day

Federal Register Notices5 **SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION**  **A. Justification**  **A 1.**  **E xplain**  **the circumstances**  **that make**  **the collection**  **of information**  **necessary. I**  **dentify any**  **legal or**  **administrative requirements**  **that necessitate**  **the collection.**  **A ttach**  **a copy**  **of the**  **appropriate section**  **of each**  **statute and**  **regulation ma** **ndating or**  **authorizing the**  **colle ction**  **of information** 
. The dismal state of student mathematics achievement, especially in fractions, has been well documented in research literature (e.g., Ma, 1999; Yanik, Helding, & Baek, 2006); national reports (e.g., National Mathematics Advisory P anel [NMAP], 2008); the IES practice guide on fractions (Siegler et al., 2010); and the results of national assessments (National Assessment of Educational Progress [NAEP], 2011). The National Mathematics Advisory P anel (NMAP) reported that “difficulty

with fractions (including decimals and percent) is pervasive and is a major obstacle to further progress in mathematics, including algebra” (2008, p. xix). Inf act, Siegler and his colleagues (2012) recently found that knowledge of fractions at age 10 is predictive of algebra knowledge and overall mathematics achievement in high school, above and beyond the effects of general intellectual ability, other mathematical knowledge, and family background. 1 There is, in general, a consensus among researchers, educators, and policy makers that improving American students’ knowledge of fractions should be made a major educational priority
. In fact, the National Center of Educational Research at IES recently awarded a $10 million dollar grant to fund a five
- ye ar research and development center aimed at understanding difficulties students have with fractions and developing effective interventions for struggling learners in this area (Center for Improving Learning of Fractions, 2013). Unfortunately, research examining teacher knowledge in mathematics has repeatedly documented that elementary teachers often have limited mathematical knowledge, particularly in the area of fractions (Hill et al., 2008; Hill, Kapitula, & Umland, 2011; Hill, Rowan, & Ball, 2005; Ma, 1999). Clearly, providing professional development that focuses on addressing teachers’ mathematical content knowledge underlying the Common Core State Standards (CCSS) is a necessary

step for full and effective implementation of CCSS. With the roll out of

1 Fractions knowledge at age 10 - 12 was found to be uniquely associated with a 0.17 SD increase ( p < .05) in algebra knowledge and a 0.18 SD increase (

p < .01) in total math achievement at age 15 - 17. This is based on a U.S. sample of 5 99 children who were tested in 1997 as 10
- 12 year- olds and again in 2002 as 15
- 17 year- olds. Nearly identical results were found in a UK sample (Siegler et al., 2012).6

the CCSS in mathematics scheduled for 2014

–2015 in Georgia and South Carolina2 , understanding the mathematical ideas and concepts involved with fractions and decimals

–as well as computational algorithms

–is a critical professional development need in the region, as well as the nation. There is a small body

o f high quality experimental and quasi
- experimental research that has examined the impact of intensive mathematics professional development (PD) programs on student mathematics outcomes. The research team conducted a What Works Clearinghouse
-ty pe review of rigorous research evaluating mathematics PD approaches. A search of the research literature published between January 2006 and July 2012 (or rigorous studies that were identified in a previous review by Yoon , Duncan, Le e, Scarloss, & Shapley, 2007) was conducted. All studies that potentially

c ould provide causal evidence of the effectiveness of teacher PD on student achievement were analyzed
. Base d on the five studies that were of acceptable technical quality, two professional development approaches , namely Lesson Study and intensive math content courses yi elded either statistically s ignificant outcomes on some mathematics measures or indicated relatively

large impacts on mathematics outcomes (effect size greater than .25). There is clearly a need for additional research conducted under more ty pical conditions and with a clear topical focus. Thus, the purpose of the proposed study i s to assess the impact of a professional development program that focuses on developing teachers’ knowledge of the formal mathematics that underlies fractions as well as pedagogical knowledge of techniques that promote development of student understanding of the mathematical ideas as well as computational proficiency
. The study team, in consultation with experts in the field of mathematics, determined that fourth grade will be the focus of this study as it is a critical year for fractions instruction, especially as the foundation is laid for clearly understanding the meaning of fractions in later grades. For this project, the study team will use a randomized controlled trial design to study the Developing Mathematical Ideas (DMI; Schifter, Bastable, & Russell, 2010) program to determine if such an approach to professional development is effective in producing positive impacts on student achievement in mathematics. DMIw as selected after a comprehensive examination of professional development programs, as it has a robust infrastructure for implementing on a large scale
. DMIwas selected in partnership with a research alliance sponsored by REL Southeast. Members of the alliance helped review and select DMI from other candidate programs. This process is presented in more detail in Appendix H .

2 This is based on the information retrieved from the South Carolina State Department of Education and Georgia Department of Education websites (South Carolina State Department of Education, 2013; Georgia Department of Education, 2012).7

DMIw as created and field
-tested by The Education Development Center (EDC) through a grant funded by

the National Science Foundation. The primary objectives of the DMI program are for teachers to: a) spend time on the actual teaching of mathematical topics to students in their classes , and b) explore students’ thinking about mathematics, including the misconceptions and hazy conceptions of mathematical ideas that develop during early phases of instruction. Using a random assignment design for this study helps ensure that

– all else equal

– this study will yield the strongest, most reliable evidence possible on which to base policy and practice. The current authorization for the Regional Educational Laboratories program is under the Education Sciences Reform Act of 2002, Part D, Section 174, (20 U.S.C. 9564), administered by the Department of Education (ED), Institute of Education Scienc es (IES), National Center for Education Evaluation and Regional Assistance (NCEE). **A 2.**  **Indicate how,**  **by whom,**  **and for**  **what purpose**  **the information**  **is to**  **be used.**  **E xcept**  **for a**  **new collection,**  **indicate the**  **actual use**  **the agency**  **has made**  **of the**  **information received**  **from the current collection.**  **Research Questions and Overview of the Project Plan** This study

i s a large
- scale evaluation of a mathematics professional development program for grade 4 teachers. Specifically, it is an evaluation of Developing Mathematical Ideas (DMI). The goal of the proposed study i s to assess the impact of a professional development program that focuses on developing teachers’ knowledge of the formal mathematics that underlies fractions in order to promote development of student understanding of the mathematical ideas as well as computational proficiency. Information will be collected by the study

team at the In structional Research Group (IRG) and Florida State University, the prime contractor of the Regional Education Laboratory S outheast (REL
- SE) and will be analy zed and used to inform the research questions and build upon existing IE S initiatives. IES will review all results and release a report describing the study ’s findings. The study team will address the following research questions , the first confirmatory3 , and the second exploratory : RQ1. What is the impact of teacher participation in Developing Mathematical Ideas for one year on students’ proficiency in the area of fractions when compared with those students whose teachers receive existing professional development activities provided by schools and districts?

3 The term confirmatory refers to the primary research question for which the study was specifically designed and powered to address.8

RQ2. What is the impact of teacher participation in Developing Mathematical Ideas for one ye ar on teacher knowledge of mathematics relevant for teaching fractions to 4th graders when compared with teachers who only p articipate in existing professional development activities provided by schools and districts? To answer these questions, the study team has developed an experimental research design wherein data will be collected over the course of one ye ar using a sample of fourth grade teachers from at least 80 schools in diverse districts (city /town/rural) in Georgia and South Carolina. Only s chools with scores that fall below the respective state median will b e recr uited
. Schools will be randomly assigned to treatment and control conditions (Shadish, Cook, & Campbell, 2002) and all fourth grade teachers in each participating school will be asked to participate. School
- level random assignment will reduce the potential for contamination between intervention and control group teachers. Assuming there are 80 schools participating, this will yi eld 40 treatment and 40 control schools. The study team will need to collect new information from the participating districts and schools through student and teacher testing, analysis of audio recordings of DM I professional development sessions , and teacher surveys. Data routinely collected and compiled by s chool districts will also help answer these questions. The data collecti on measures for each research question are summarized in Table A1.9 TABLE A1 PROPOSED DATA COLLECTION PLAN **Data Collection Sources**  **Mode Timeline**  **Key Data**
Student Achievement

testing of all students Post-test only May

–June 2015 Student achievement in

mathematics (RQ1) School/District Student records July 2014 Student grade 3 state

assessment scores to use as a pretest covariate and student demographic information (RQ1) Teacher Consent Form May

–June 2014 Teacher demographic

information (RQ2) Teacher knowledge of mathematics testing of all teachers Pretest: Aug.

– Sept. 2014 Post-test: May

–June 2015 Teacher knowledge of

mathematics

– (RQ2) Monthly online surveys of DMI activities (all teachers) September 2014

– April 2015 Professional development

activities in treatment and control schools (RQ2)
PD Facilitator Procedural fidelity

checklists to use when analyzing audio tapes of DMI s essions (treatment PD only) August 2014

– November 2014 DM I fidelity (RQ1 & 2)10

**Student and Teacher Assessment**  **Student Assessment:**  For the student
- level confirmatory analys is, grade 4 students will be assessed on a test, Test for Understanding of Fractions (TUF ) , that is currently

b eing developed by the IES/ NCSER (National Center for Special Education Research) Center for Improving Learning of Fractions. TUF is being developed in conjunction with the PIs of the Center (Nancy Jordan, Lynn Fuchs, and Robert Siegler) and will include, in part, measures that have been field
tested in their current longitudinal and intervention research with fourth graders. These existing measures focus on both procedural and conceptual knowledge, and include, for example, items from NAEP, batteries used in the research of He cht, Close, and Santisi (2003) and measures specific ally developed by Jordan (e.g. Jordan et al. , 2013 ). In addition, the measure may include items from the fractions measure recently utilized by P erry and Lewis (2011). This measure was influenced by

the Japanese fractions curriculum, which stresses the linear measurement model of fractions, and thus overlaps with the mathematical ideas highlighted in the CCSS. Finally, publicly available measures may b e included from Smarter Balance d and PARCC consortia as they b ecome available. These measures are being constructed to directly align with CCSS. Smarter Balance d and PARCC consortia are exp erimenting with short answer items that probe depth of understanding by, for example, including subtle discriminations among distractors that provide students with answers that represent partial understanding of a concept or mathematical relationship. Students who choose these responses receive only p artial credit. Few of these items are currently available. However, the study team anticipates that items will be available by January of 2014, when the first field test version of the TUF will be finalized
. Insummary, items aligned with the fourth grade objectives for fractions in the CCSS will be selected or adapted. As items from various sources will be incorporated into the measure for use in this study, scales will be constructed and evaluated using both classical test theory and item
response theory
. **Teacher Assessment:** The impact of the DMI intervention on teacher knowledge of mathematics relevant for teaching fractions to 4th graders will be assessed using the Mathematical Knowledge for Teaching (MKT) (Ball & Hill, 2008)
. The IRT reliability for MKT at the elementary

level is .94. (Hill, Rowan, & Ball, 2005). The MKT consist s of a large pool of items and relevant items on fractions will be selected under the guidance of Kristin Umland and Jim Lewi s, two mathematician s . The measure is currently under development and is expected to take less than 1 hour to complete
.11

**Data Collection Methods** In each district, data will be collected from both treatment and control schools at the same time, by assessors blind to condition, to guard against bias entering the data collection process. **Collecting Teacher**  **Data:** Active consent of individual teachers for participation will be acquired, before assignment of individual schools. Prior to the start of the intervention (August 2014), data on teacher demographics will be collected (e.g., teacher experience and education) as well as for teacher knowledge of mathematics (MKT) 4

. The demographic survey and teacher knowledge measure are included in our calculations of respondent burden. Demographic information will be collected at the outset of the study as part of the teacher consent form. A copy o f this form has been included with this submission as Appendix A . Immediately after the end of the professional development intervention, teachers will complete the MKT measure (for RQ2). Two MKT released items have been included as Appendix B. A monthly online measure assessing the ty

pe and amount of professional development teachers participate in will be sent to all teachers using SurveyMonkeyTM
. This survey is included in our calculations of respondent burden. A cop y of the monthly teacher survey

is included with this document as Appendix C . **Collecting Student**  **Data:** Student prior ye ar pretest data on the third grade state assessment will be collected from school districts when it becomes available in 2014 . Student demographic data will be gathered from the school databases upon completion of the parent opt- out phase
. The study team has a waiver of active consent from the IRB, as direct engagement of students in the study i s limited strictly to the f ractions outcome assessment. A 30

–40 minute group assessment (very similar to the state assessments) appears to represent a very low risk to students. Post-test data will be collected in the spring after the professional development has ended, around 4 weeks prior to the end of the school year. Students will be given the Test for Understanding of Fractions (TUF) measure by trained evaluation staff in large group s during independent seatwork to maximize teacher instructional time during the school day (RQ1). The administration of an achievement test like the TUF is not considered for information collection burden under 5 CF R 1320.3(h) (7). **Collecting School**  **Data:** School level data (e.g., free and reduced lunch, school achievement scores) will be collected from the prior year school report cards for purposes of matching schools. As is the case with teacher demographic information, student data are pre - existing, already

4 Mathematical Knowledge for Teaching (MKT) (Ball and Hill, 2008 )12

collected by

s chools and districts, and are not included in our calculation of burden. Using existing school records data helps to minimize overall burden because it avoids the collection of these same data items from individual respondents at the school level. **Collecting Fidelity**  **of Implementation**  **Data** : DMIf acilitators at each site will be asked to audiotape each session they have with participating teachers
. Two research staff with training in observati on systems will listen to each tape and use the fidelity o f implementation instrument to measure adherence to critical core components of the DMIprogram using checklists developed in conjunction with the developers. Discrepancies will be resolved by s enior staff. It is our understanding that the collection of this audio data is not considered in calculations of paperwork burden. **A 3. Describe whether, and to what extent, the collection of information involves the use of**  **automated, electronic, mechanical, o r any other technological collection techniques or forms**  **of information technology, e.g. permitting electronic submission of responses, and the basis for the decision of adopting the means of collection. Also describe any consideration of using**  **information technology to reduce burden.**  Wherever possible the research team will use information technologies to maximize the efficiency and completeness of the information gathered for this evaluation and to minimize the burden on respondents. In particular, student demographic and prior year assessment data will be gathered from existing electronic school administrative records. Brief monthly teacher surveys regarding the typ e and amount of professional development teachers participate in will be sent to all teachers using the online application SurveyMonkeyTM. **A 4. Describe efforts to identify duplication. Show specifically why any similar information**  **already available cannot be used or modified for use of the purposes described in Item 2**  **above.** The data collection effort planned for this project will produce data that are unique and specifically target the research questions identified for this project. A rigorous literature review was conducted along the standards outlined by the What Works Clearinghouse that identified the need for the kind of experimental research that will be conducted in this study
. Furthermore, there is no current evidence from studies using randomized controlled designs available on the DMIprogram. To complete this research, the study team is planning to administer the TUF to all students in the study sample because the data collected with such an instrument are not available from any other source on a national basis; the state - administered assessments vary in what they

m easure, and how precisely they

c an measure student achievement in mathematics. Thus, administering our13

own assessment is necessary b ecause it provides site
-to - site consistency and assures a reliable and common mathematics achievement score. Data collection through the teacher assessment will provide a measure of teacher knowledge of mathematics relevant for teaching fractions to 4th graders as well as the treatment contrast between DMI (treatment) and non- DMI (control) schools. Surveying all of the teachers on a monthly b asis as to professional development activities will provide descriptive data to augment the discussion of treatment versus control schools, and taking audio recordings of DMI professional development sessions will provide useful descriptive data regarding fidelity o f implementation of the DMIprogram. In every case, these data do not currently exist. **A 5.**  **If the**  **collection of**  **information impacts**  **sm all**  **businesses or**  **other sm**  **all entities**  **(Item 5**  **of OMB Form 83** **- 1), describe any methods used to minimize burden.**  Not applicable. No small businesses will be burdened by this data collection. The focus of this study is on school districts and the attendant schools

– which includes the students, teachers, and administrative staff

– within these districts. The study team has reduced burden for respondents using a data collection plan that requests the minimum information needed to succ essfully execute this study
. To minimize burden on respondents, the study design requests information already collected by

s chools and districts, and any new collection instruments have been designed to ask questions that cannot be answered through any other available sources. In some cases, these data will be collected electronically. The study team will conduct all data collection activities. **A 6. Describe the consequences to Federal program or policy activities if the collection is not**  **conducted or is conducted less frequently, as well as any technical or legal obstacles to**  **reducing burden.** The systematic collection, analys is, and reporting of the data described in this submission are required to accomplish the goals of the research project approved by IES. Participation in all data collection activities is voluntary but the study

team will work to assure high participation rates. Multiple data collection strategies are planned and these data are necessary to measure impacts on academic achievement and teacher knowledge. The TUF will be administered to students at treatment and control schools once as an outcome measure at the end of the 2014

–2015 school year. This is the primary outcome measure for mathematics skills and if it is not collected the study would lack a common measure of students’ mathematical ability
. There is no other common measure across the two states and multiple districts
.14

The teacher measure will be administered to teachers at treatment and control schools prior to beginning the DMI program and after professional development at the end of the academic ye ar . This is the principal outcome measure for teacher knowledge and if it is not collected, the study would lack a common measure of teacher knowledge at baseline and after re ceipt of professional development. Pretesting teacher knowledge is necessary to boost the statistical power, and is assumed in the power analysis , because of the inherently smaller number of teachers in the study. If the pretest is not given, the study may not be as sensitive to effects on teacher knowledge that may be present
. During the course of the study year, treatment and control teachers will be surveyed monthly regarding any professional development in the field of mathematics they received. These surveys will provide the study

team with valuable descriptive information about the professional development activities going on in all of the study s chools. If the study

team surveys the teacher sample less frequently there is a risk that the teachers mi ght have less valid memories of the PD they received. All DMI professional development sessions with teachers will be audio recorded (8 sessions delivered per site) and these recordings will be analyz ed by two independent raters to ensure implementation f

idelity o f the DMIprogram. Without these recordings, the study team will not have means to make certain the program is being implemented correctly
. High response rates are anticipated given that states, districts, and even schools will be selected based on their willingness to fully p articipate in the project. The study team is confident that the use of multiple measures of data collection will ensure accurate analys es and results. Finally, if this study

i s not conducted, REL Southeast may not meet the requirements for conducting rigorous research that meets regional needs
. REL (NCEE) contracts are awarded contingent on a plan to partner in research alliances with educational policy makers from the region to identify

research needs and to conduct studies that address them. These researcher
practitioner alliances are mandated to conduct rigorous studies (typically RCTs) wherever possible. The proposed study meets this contractual requirement. **A 7.**  **Explain any**  **special circumstances**  **that would**  **cause an**  **information collection**  **to be**  **conducted in a manner inconsistent with Section 1 320.5(d)(2) of the federal regulations** **.** The proposed study

m eets all guidelines listed under A7 except for a requirement that respondents report information more than quarterly
. As explained in section A6, the study team will survey

all teachers about the ir professional development activities on a monthly b asis. This survey i s brief (< 10 minutes per month) and administered electronically. Not surveying the teachers as regularly

c ould result in less valid responses owing to potential inaccuracy o f15

memory over longer periods. **A 8. Federal Register Comments and Persons Consulted Outside of the Agency**  A notice to solicit public comments will be published in the Federal Register when this package is submitted in order to provide the opportunity for public comment. The following individuals were consulted on the statistical aspects of this proposal: Dr . Sy billa Beckman, University of Georgia Expertise: mathematician, mathematics education Email: sy

billa@math.uga.edu Phone: 706 - 542 - 2548 Dr. John Deke, Mathematica Policy Research Expertise: methods for large
- scale randomized trials Email: jdeke@mathematica
-mpr.com Phone: 609 - 275 - 2230 Dr. Mike Garet, American Institutes for Research Expertise: teacher professional development, research methods , large
- scale randomized trials Email: MGaret@air.org Phone: 202 - 403 - 5000 Dr. Nathan Jones, Boston University Expertise: measurement of teacher quality, teacher evaluation, special education Email: ndjones@bu.edu Phone: 617 - 353 - 3295 Dr. Yaacov Petscher, Florida State University Expertise: research design, measurement, and statistical methods Email: ypetscher@fcrr.org Phone: 850 - 644 - 0327 Dr. Mengli Song, American Institutes for Research Expertise: research design, advanced quantitative methods, and evaluations of educational programs and policy Email: MSong@air.org Phone: 202 - 403 - 500016

The first five individuals in the list above attended a full
- day m eeting in Washington, D.C. on October 12, 2012 to discuss the proposal. Of particular focus were the statistical assumptions informing the power analysis, advantages and disadvantages of the mathematics PD approaches under consideration at that time, covariates to be included in the HLM models and the proposed outcome measures. The sixth individual, Dr. Song , reviewed the power analys es and HLM models independently. **A 9.**  **E xplain**  **any decision**  **to provide**  **any payment**  **or gift**  **to respondents,**  **other than**  **remuneration of contractors or grantees.** All teachers in the PD and the control condition will be compensated for their time spent completing the measures for this study (see section A12

– Table A2). These measures include the teacher consent/demographics form, the pre - and post-test of te acher knowledge of mathematics relevant for teaching fractions to 4th graders , and the monthly professional development activity survey . Treatment teachers will not be compensated for DMIp articipation time. Instead teachers will be provided with one full
-release day f rom instruction for each DMI session. The release time will cover both the PD session and the time required to complete the homework. **A 10.**  **Describe any**  **assurances of**  **confidentiality provided**  **to respondents**  **and the**  **basis for**  **the assurance in statute, regulation, or agency policy.**  RELS outheast will be following the new policies and procedures required by the Educ ation Sciences Reform Act of 2002, Title I,P art E, Section 183 requires "All collection, maintenance, use, and wide dissemination of data by the In stitute" to "conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h)." These citations refer to the Privacy Act, the Family

E ducational Rights and Privacy Act, and the Protection of Pupil Rights Amendment. In addition for student information, "The Director shall ensure that all individually i dentifiable information about students, their academic achievements, their families, and information with respect to individual schools, shall remain confidential in accordance with section 552a of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act. Subsection (c) of section 183 referenced above re quires the Director of IES to "develop and enforce standards designed to protect the confidentiality

of persons in the collection, reporting, and publication of data".17

Subsection (d) of section 183 prohibits disclosure of individually identifiable information as well as making any the publishing or communicating of individually identifiable information by employees or staff a felony
. RELS outheast will protect the confidentiality of all information collected for the study and will use it for research purp oses only
. No information that identifies any study p articipant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Information on respondents will be linked to their institution but not to any individually

i dentifiable information. No individually i dentifiable information will be maintained by

the study

team. Staff working on the project have signed confidentiality p ledges and been screened for data security through the U.S. Department of Education e - QIP system. Only p ersons conducting this study and maintaining its records will have access to the records collected that contain individually identifying information.
**A 11. Provide additional justification for any questions of a**  **sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered**  **private. The justification should include the reasons why the agency considers the questions necessary, the specific uses to be made**  **of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their**  **consent.** None of the questions utilized to collect data for this study, including interviews, concern topics commonly

considered private or sensitive, such as religious beliefs or sexual practices. IRB approval has been granted
. Note that modifications to the original parent information forms have been requested, and will be resubmitted to the IRB for re
- approval
. **A 12. Provide estimates of the hour burden of the collection of information.**  The estimated burden on respondents for completing the activities included in the study ’s data collection activities is listed in Table A2. Respondents will include teachers at treatment and control schools. Table A2 summarizes reporting burden on respondents for the data collection instruments. The annual burden is estimated from the total number of minutes taken to complete the data collection. Thus, the total burden across all respondents is expected to be 2039 hours or $ 3 9 , 72 0 in monetary

cost.18 TABLE A2 ANNUAL HOUR BURDEN ESTIMATES **Data Collection Activity**  **Num ber of** 

**Respondents Per**  **School Num**  **ber**

**of Schools**  **Total**

**Respon dents**  **Average**

**Burden Hours per**  **Response Total** 

**Burden Hours**  **Planned**

**incentive Estimated** 

**Monetary Cost of**  **Burden** Teacher Consent Forms with Demographics 3 84 252 .2 50.4 $10 total $ 2,520 Teacher Assessment

– (Pre & Post) 3 84 252 4 1008 $50 total (2 * $ 25 ; pre & post) $ 12 , 6 00

Teacher Professional Development Survey

– Monthly 3 84 252 9 * .2 453.6 $ 90 total ( 9 * $10 ; monthly) $ 22,680
Student Study Participation Agreement Form 57 84 4788 .1 479 0 0
School records data collection: Student information 6 districts 6 8 48 $ 40/hour $ 1,920

**TOTAL 2039**  **$ 39,720** 19

**A 13. Describe**  **any other costs to respondents or record keepers.** Not applicable. The information collection activities do not place any capital cost or cost of maintaining capital requirements on respondents. **A 14. Provide estimates of annualized cost to the Federal**  **government. Also provide a description of the method used to estimate cost, which should include quantification of hours,**  **operational expenses (such as equipment, ove rhead, printing, and support staff), and any**  **other expenses that would not have been incur red without this collection information.**  **Agencies may aggregate cost estimates from Items 12, 13, and 14 in a single table.** The proposed data collection is being supported through Department of Education contract ED
IES - 12 - C - 0011. This project is considered Task 4.1 .5 of the NCEE RELSoutheast contract awarded to Florida State University
. The annualized cost of data collection for this project is calculated simply as the planned annual fixed price allocations for the study during the two calendar years (2014 and 2015) that this study is gearing up for data collection, actually being in the field, and wrapping up data collection. Total cost of data collection for this project is estimated to be: $2,469,612 . This represents a fixed price contract to c onduct these activities in the two calendar ye ars that involve data collection. 2014: Budget = $1,237,979  January

– April 2014: Approximate date
-range to receive OMB clearance  January

– June 2014: After OMB clearance : Recruiting of districts/schools/teachers

– sign district/school MOUs  January

– June 2014: Work with DMI developers to plan/refine program as needed  January

– June 2014: Collaborate with the Center for Improving Learning of Fractions on development of student outcome measure.  May

– June 2014: Distribute/acquire teacher consent forms  Ju ne – August 2014: Recruit & train teacher knowledge assessors  July 2014: Conduct random assignment  August

– September 2014: Conduct teacher knowledge pretests  August

– November 2014: Conduct eight DMI sessions with over 120 teachers in 6 districts  September

– Dec 2014: Collect monthly teacher PD logs 2015 : Budget = $1,231,633  January

– April 2015: Collect monthly teacher PD logsApril

– May 2015: Recruit/train assessors  May

– June 2015: Conduct teacher and student post
-tests  Ju ne 2015: Data entry & cleanup  July

– October 2015: Conduct data analysis  September

– Dec. 2015: Report Preparation20

 December 2015: Submit first draft of technical report to IES Funding includes paying for REL S outheast staff , as well as covering costs of subcontractors on this study who will assist in study design, data collection, and data analysis. Also included are costs associated with delivering the intervention to treatment schools, and collecting data from both treatment and control schools. There are no additional costs, other than staff collection time and duplication, for the student and teacher outcome measures. **A 15. Describe any changes in**  **the burden from prior approvals.** This submission to OMB is a new request for approval of data collection plans. **A 16. For collections of information whose results will be published, outline plans for**  **tabulation and publication. Address any complex analyt ical techniques that will be used.**  **Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of the report, publication dates, and other actions.**  The project schedule is as follows : 2014: Budget  January

– April 2014: Approximate date
-range to receive OMB clearance  January

– June 2014: After OMB clearance : Recruiting of districts/schools/teachers

– sign district/school MOUs  January

– June 2014: Work with DMI developers to plan/refine program as needed  January

– June 2014: Collaborate with the Center for Improving Learning of Fractions on development of student outcome measure.  May

– June 2014: Distribute/acquire teacher consent forms  Ju ne – August 2014: Recruit & train teacher knowledge assessors  July 2014: Conduct random assignment  August

– September 2014: Conduct teacher knowledge pretests  August

– September 2014: Conduct eight DMI sessions  September

– Dec 2014: Collect monthly teacher PD logs 2015 : Budget  January

– April 2015: Collect monthly teacher PD logs  April

– May 2015: Recruit/train assessors  May

– June 2015: Conduct teacher and student post
-tests  Ju ne 2015: Data entry & cleanup  July

– October 2015: Conduct data analysis  September

– Dec. 2015: Report Preparation  December 2015: Submit first draft of technical report to IES 2016:  January

– June 2016: Approximate date range to respond iteratively to IES reviews of draft s of report  September 2016: Estimated date of f inal report release by IES21

The design, data collection, analys is, and reporting for this study i s driven by

the primary (confirmatory) research question: What is the impact of teacher participation in Developing Mathematical Ideas on students’ proficiency in the area of fractions when compared with those students whose teachers receive existing professional development activities provided by schools and districts? A secondary

( exploratory) research question will also be addressed, although the study w as not specifically powered to do so: What is the impact of teacher participation in Developing Mathematical Ideas for one ye ar on teacher knowledge of mathematics relevant for teaching fractions to 4th graders when compared with teachers who only participate in existing professional development activities provided by schools and districts? Our approach to estimating the effects of DMI has the following core features: 1) A focus on student
- level impacts based directly on the experimental design. 2) Estimation of impacts in ways that account for clustering of students within classrooms, within s chools , and within districts
. The basic logic of our analysis strategy i s to compare the schools that are randomly assigned to receive the treatment to those that are not. As random assignment occurs at the school level, schools are the primary

unit of analys is. Schools will be paired within district based on overall demographic characteristics to increase the probability of baseline equivalence. The research team will use multi
- level models in the impact analysis to account for the clustering of students within classrooms, and teachers within schools. The analy tic methods are discussed in further detail in Part B 2 and Appendix G of this document
. All results for RELrigorous studies will be made available to the public through peer
- reviewed evaluation reports that are published by IES. The datasets from these rigorous studies will be turned over to the REL’s IES project officer. These data will b ecome IES restricted use datasets requiring a user’s license that is applied for through the same process as NCES restricted use datasets. Even the REL contractor would be required to obtain a restricted use license to conduct any work with the data beyond the original evaluation. **A 17. Describe arrangements for displaying the number provided by OMB and its expiration**  **date.** The approval number provided by OMB and its expiration date will appear in the heading on all instruments for this project.22

**A18. Exce**

**ptions to Certification Statement** No exceptions are necessary for this information collection
.23 **References** Ball, D. L., & Hill, H. C. (2008). Mathematical knowledge for teaching (MKT) measures: Mathematics released items 2008 . Ann Arbor, MI: University of Michigan. Retrieved from http://sitemaker.umich.edu/lmt/files/LMT_sample_items.pdf Center for Improving Learning of Fractions. (2013). Center for Improving Learning of Fractions. Retrieved from https://sites.google.com/a/udel.edu/fractions/ Georgia Department of Education. (2012). Common core Georgia performance standards
. Retrieved from http://www.doe.k12.ga.us/Curriculum
- In struction
- and
Assessment/Curriculum
- and
- Instruction/Page s/CCGPS.aspx Hecht, S. A., Close, L., & Santisi, M. (2003). Sources of individual differences in fraction skills. Journal of Experimental Child Psychology , 86(4), 277
- 302. Hill, H., Blunk, M., Charalambous, C., Lewis, J., Phelps, G., Sleep, L ., & Ball, D. L. (2008). Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory

study
. Cognition and Instruction , 26(4), 430
- 511. Hill, H., Kapitula, L. , & Umland, K. (2011). A validity argument approach to evaluating teacher value
- added scores. American Educational Research Journal, 48 (3), 794
- 831. Hill, H., Rowan, B., & Ball, D. (2005). Effects of teachers' mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42 (2), 371
- 406. Jordan, N. C., Hansen, N., Fuchs, L. S., Siegler, R. S., Gersten, R., & Micklos, D. (2013). Developmental predictors of fraction concepts and procedures. Journal of Experimental Child Psychology, 116 (1), 45
- 58. Ma, L. (1999). Knowing and teaching elementary math ematics: Teacher's understanding of fundamental mathematics in China and the United States . New York, NY : Routledge. National Assessment of Educational Progress (2011). The nation’s report card: Mathematics 2011. (NCES: 2012
- 458)
. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012458.pdf National Mathematics Advisory Panel. (2008). Foundations for success: The final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education. Retrieved from http://www2.ed.gov/about/bdscomm/list/mathpanel/report/final
report.pdf Perry, R. R., & Lewis, C. C. (2011). A randomized trial of lesson study with mathematical resources: Measuring the impact on fractions knowledge. Manuscript submitted for publication, School of Education, Mills College, Oakland, California. Schifter, D., Bastable, V. & Russell, S. J. (2010). Making meaning for operations in the domains of whole numbers and fractions: Facilitator’s guide . Boston, MA: Pearson Education, Inc.24

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi
- experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin. Siegler, R., Carpenter, T., Fennell, F., Geary, D., Lewis, J., Okamoto, Y., . . . Wray, J. (2010). Developing effective fractions instruction for kindergarten through 8th grade: A practice guide (NCEE #2010
- 4039). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http://whatworks.ed.gov/publications/practiceguides Siegler, R. S., Duncan, G. J., Davis
- Kean, P. E., Duckworth, K., Claessens, A., Engle, M., . . . Chen, M. (2012). Early predictors of high school mathematics achievement. Psychological Science, 23 (7), 691
- 697. doi: 10.1177/0956797612440101 South Carolina State Department of Education. (2013). Common Core State Standards. Retrieved from http://ed.sc.gov/agency /programs
- service s/190/ What Works Clearinghouse. (2013). What works clearinghouse: Procedures and standards handbook (Version 3.0). Retrieved from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_draft_standar ds_handbook.pdf Yanik, H. B., Helding , B., & Baek, J. M. (2006). Students’ difficulties in understanding fractions as measures. Problem Solving, 2 , 323 - 325. Yoon, K. S., Duncan, T., Lee, S. W. -Y., Scarloss, B., & Shapley, K. (2007). Reviewing the evidence on how teacher professional developm ent affects student achievement (Issues & Answers Report, REL 2007

– No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest. Retrieved from http://ies.ed.gov/ncee/edlabs