**Impact Evaluation of Data**  **- D**  **riven Instruction**  **Professional Development for Teachers**  **Supporting Statement Part A :**  **Justification for the Study** May 2 6 , 201 5Contract Number:

ED- IES- 12- C-0086 Mathematica Reference Number: 40166
. 7 1 0 Submitted to: Institute of Education Sciences IES/NCEE U.S. Department of Education 555 New Jersey Avenue, NW Washington, DC 20208 Project Officer: Erica Johnson Submitted by: Mathematica Policy Research 600 Alexander Park Suite 100 Princeton, NJ 08540 Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: Phil Gleason **Impact Evaluation of**  **Data -** **D riven**  **Instruction Professional**  **Development for Teachers Supporting Statement Part**  **A: Justification for the Study**  May 2 6 , 201 540166 OMB Supporting Statement: Part A Mathematica Policy Research

iii **CONTENTS** PART A: JU STIFICATIO N ................................ ................................ ................................ 1

1. Circumst ances Necessitating the Collection of Information ................ 1 2. Pu rposes and Us es of Data ................................ ................................ 8 3. Us e of Technology to Reduce Burden ................................ ................. 9 4. Efforts to Avoid Duplication of Effort ................................ ................. 9 5. Met hods to Minimize Burden on Small Entities ................................ . 10 6. Consequences of Not Collecting Data ................................ .............. 10 7. Special Circumstances ................................ ................................ ..... 10 8. Federal Register Announcement and Consultation ........................... 11 9. Payment s or Gifts ................................ ................................ ............ 12 10. Assurances of Confidentiality ................................ .......................... 12 11. Addit ional Justification for Sensitive Questions ................................ 13 12. Estim at es of Hours Burden ................................ .............................. 13 13. Estim at es of CostBurden to Respondents ................................ ........ 14 14. Estim at es of An nual Costs to the Federal Government ..................... 14 15. Reasons for Program Changes or Adjust ments ................................ 14 16. Plan for Tabulation and Publication of Results ................................ . 14 17. Approval Not to Display the OMB Expirat ion Dat e ............................ 21 18. Explanation of Exceptions ................................ ............................... 21 REFERENCES ................................ ................................ ................................ ............... 22
APPENDIX A : TEACHER ASSIGNMENT DATA REQUEST APPENDIX B : TEACHER SURVEY QUEST IONNAIRE AND ACCOMPA NYING LETTER APPENDIX C : TEACHER LOG AND ACCOMPANYING LETTER APPENDIX D : PRINCIPAL SURVEY QUESTIONNAIRE AN D ACCOMPANYING LETTER APPENDIX E : STUD ENT RECORDS REQU EST APPENDIX F : CONFIDENTIALITY PL EDGE40166 OMB Supporting Statement: Part A Mathematica Policy Research

iv **TA BLES**  1. Schedule of Major St udy Act iv it ies for Data
- Driven Instruction

Evaluat ion ................................ ................................ ................................ ...... 7 2. Research Questions and Data Collection Method ................................ ............ 8 3. Expert s Consulted ................................ ................................ ....................... 11 4. Estim at ed Response Time for Data Collection Activities ............................... 13 5. Data- relat ed Act iv it ies, by Source ................................ ................................ . 16 6. Teacher and Principal Intermediate Outcome Measures, by Source ............... 19OMB Supporting Statement: Part A Mathematica Policy Research

v **Figures** Figure 1: Logic Model for Planned DDI Intervention ................................ ...................... 4OMB Supporting Statement: Part A Mathematica Policy Research

1 **SUPPORTING STA**

**TEMENT FOR PA PERWORK**  **REDUCTION A CT SUBMISSION**  This OMB package requests clearance for data collection activities for a rigorous evaluation of data

-driven instruction (DDI) in 104 schools from 12 school districts that have been recruited for the study. Data

-driven instruction involves the use of student assessment data to help teachers adapt their instruction and, ultimately, improve student achievement. The study’s intervention plan will build school capacity for DDI by: (1) helping schools set up structures and processes that enable teachers and other school staff to efficiently carry out data

-driven instruction, and (2) training and coaching teachers in the skills needed to understand student data and implement improved instructional strategies to address student needs. By participating in professional development and ongoing DDI activities, teachers are expected to develop the knowledge and skills needed to help them adapt and improve their instructional strateg ies based on student data, and ultimately, improve student achievement. We plan to collect student records and teacher

-assignment data from participating districts and schools, and conduct a teacher survey, teacher logs, and a principal survey. The Institute of Education Sciences (IES) within the Department of Education (ED) has contracted with Mathematica Policy Research and its partners Abt Associates and Evidence

-Based Education Research & Evaluation to conduct the evaluation, and Public Consulting Group

–Focus on Results to provide technical assistance to schools implementing the DDI program. The evaluation’s main objectives are to understand how DDI is implemented and to rigorously estimate the impact of a comprehensive DDI program on student achievement and teacher and principal practices . The implementation component will use information collected from the technical assistance (TA) team, a teacher survey and logs, and a principal survey. For the impact evaluation, the experimental design involves randomly assigning schools within a district to either a treatment or control group. The treatment schools will implement a comprehensive DDI program, with technical assistance provided by an organization that works with schools to implement such instruction, and the control schools will not implement new DDI initiatives during the study years . Student outcomes will include students’ achievement on math and reading state assessments . Teacher and principal outcomes will include teachers’ and principals’ use of data, teachers’ instructional strategies, and the extent and nature of teacher collaboration. This OMB clearance request concentrates on materials that will be used to collect principal, teacher, and student data. Included in this request are the following: a teacher assignment data request (Appendix A

) , a teacher survey questionnaire and an accompanying letter (Appendix B) , a form and letter for the administration of teacher logs (Appendix C), a principal survey questionnaire and accompanying letter (Appendix D) , a student records request (Appendix E) , and a confidentiality pledge (Appendix F ) . **PART A: JUSTIFICATION**  **1. Circumstances**  **Necessitating the Collection of Information**  **a. Statement of Need for a**  **Rig orous**  **Evaluation of DDI**  The specific piece of legislation authorizing this evaluation is Title II, Part A of the Elementary and Secondary Education Act (ESEA ) , Section 2121

-2123 as amended by No Child Left Behind ( NCLB) (20 USC 6621

- 6623) . The Title II legislation provides grants to states and districts toOMB Supporting Statement: Part A Mathematica Policy Research

2 increase student academic achievement. State and local education agencies can use Title II funding to offer professional development activities that “provide training on how to understand and use data and assessments to improve classroom practice and student learning” (U.S. Department of Education, 2013). The federal government is also actively promoting DDI through other initiat ives, including Race to the Top and School Improvement Grants . A growing number of schools across the country have adopted DDI programs. Although there are no comprehensive national data on the current number of schools implementing DDI

— in part due to differing definitions of this intervention

—DDI providers report implementing DDI programs in hundreds of districts and tens of thousands of schools. 1 One provider reports that its DDI materials and training are used in 20 percent of K

–12 school districts nationwide (Cordray et al. 2012). Despite the growing interest in data use across the country, there is limited rigorous evidence on the effects of comprehensive DDI, especially for programs that feature intensive on- site support for teachers and other school staff . The few rigorous evaluations of DDI typically find no statistically significant impacts on student achievement. 2 Several of these experimental studies have had methodological limitations, however, and the DDI interventions they examined did not provide substantive supports at the school and teacher levels (Konstantopoulo s et al. 2013; Cordray et al. 2012; Carlson et al. 2011; Slavin et al. 2013). In general, the studies have focused on interventions that did not include teacher collaboration, an important best practice (Hamilton et al. 2009; Lachat and Smith 2005), or intensive professional development and on- site coaching and other supports specifically designed to help teachers fully understand student data and adapt their instruction accordingly. In fact, the studies found that the interventions they examined had little or no effect on instructional practices (Cordray et al. 2012) or that DDI teachers were actually less likely to use differentiated instruction, an important DDI teaching practice (Williams et al. 2013; Konstantopoulos et al. 2013). Thus, the existing literature is limited in the information it can provide on the potential impact of a comprehensive DDI intervention. 3 This study would provide important, new experimental evidence on the impact of a comprehensive DDI program on student achievement. We will also gather valuable information on how DDI is implemented in schools. **b. Study Design and**  **Research Questions** 

1 This information is based on conversations with representatives of nine large DDI providers: Achievement Network, Acuity, Curriculum Associates, Data Wise, Education for the Future, Northwest Evaluation Association, Public Consulting Group

-Focus on Results, TERC, and Wireless Generation. 2 There have been several nonexperimental examinations of DDI (Shaw and Wayman 2012; Henderson et al. 2007; Quint et al. 2008; Datnow et al. 2007), but because schools or teachers using DDI may systematically differ in other ways from schools or teachers not choosing to use DDI, these non-experimental results do not support rigorous causal inference. 3 In addition to the studies listed here, there is an ongoing evaluation of the Achievement Network DDI intervention. That intervention includes the administration of a particular set of interim assessments along with supports to participating schools to aide them in interpreting and using the assessment results. A report on this study is forthcoming in spring 201 5. Therefore, at this time, the detailed characteristics of the evaluation design and the findings are not known.OMB Supporting Statement: Part A Mathematica Policy Research

3 The study team will evaluate the effectiveness of the DDI program provided by Focus on Results using a random assignment design. Atotal of 104 schools from 12 school districts have been recruited for the study. Within study districts, schools have been randomly assigned to either a treatment group that is implementing the study provided DDI program or a control group that is not receiving the intervention and therefore not implementing the key DDI treatment features during the study. While the study will examine outcomes for the 2015

-2016 school year, training and setting up the infrastructure to support DDI activities will occur during the winter and spring of 2014

-2015 in the treatment schools. These activities occur in advance of the 2015

-2016 school year because it may take some time to set up the necessary school structures and to train staff on data

-driven instruction procedures. Treatment schools will learn how to set up the structures and implement the procedures appropriate to data

-driven instruction through ongoing professional development and technical assistance. Beginning these supports prior to the 2015

-2016 school year will allow treatment schools time to gradually receive training and gain practice implementing key data

-driven instruction activities to then be ready for full DDI implementation during the 2015

-2016 school year. The study implementation team will support and monitor implementation both during the initial infrastructure building phase as well as the full implementation phase. The implementation and impact evaluation study will address the following research questions:
 What is the impact of DDI professional development on students’ achievement on state assessments in math and reading?  What is the impact of DDI professional development on teachers’ and principals’ access to and use of data to inform instructional support, planning, and practice?  How does DDI professional development affect teachers’ use of instructional strategies, such as differentiated instruction?  How do DDI

-related activities differ in treatment versus control schools? Because the primary research question focuses on the impact of DDI on student achievement, the study will examine impacts on grades in which state assessments are administered, grades four and five. Targeting fourth and fifth grades will also focus intervention resources more intensively on a subset of participating schools’ teachers, while allowing us to examine the effects of DDI on achievement across two different grade levels.

**c. The Data Driven Instruction Program**  **Provided by Focus on Results** The Focus on Results plan for DDI is based on a comprehensive framework for training and supporting school leadership and teachers on using multiple sources of student data (such as formative and summative student assessment data, student classroom work, student attendance and behavior data, etc.) to inform instruction and improve student achievement. The DDI program is designed to be implemented in schools with regular interim assessments and a data system that produces reports on student achievement. Key DDI components and how they are expected to contribute to achieving intermediate and final outcomes are depicted in Figure 1 . The overall logic of the DDI program is that a core set of training and technical assistance inputs inform and guide various activities by school leaders and individual teachers . These inputs and activities lead to teachers increasing their use of student data to guide instruction and altering their instructional strategies, which in turn leads to improved student achievement. Specifically, the instructionalOMB Supporting Statement: Part A Mathematica Policy Research

4 leadership team (described below) explores data at the school and individual grade levels to diagnose core issues for school

-wide focus. Teacher collaboration teams (also described below) and individual teachers explore data in their grade and classroom to generate ideas for instructional improvement of greatest benefit and consistent with the school

-wide focus. Teacher collaboration teams and individual teachers adjust instruction and continuously monitor student progress. The school

-wide instructional leadership team also monitors progress at the school level. Guided by this ongoing review of student progress, adjustments in instruction and focus take place as needed in a continual feedback/adjustment process to accomplish school level goals. **Figure 1: Logic Model for Planned DDI Intervention Tr** **aining and Te chnical**  **Assistance Inputs**  **School Structures and Activities**  **Intermediate Outcomes**  **Final Outcome**  **s**

**Winter In**  **stitute** ( for principals and data coaches) Or ientation, training, and reso urce s for DDI im plem entation

**Instructional Leadership**  **Te am**  (principal, data coach, and grade/department chair s) Exam in e sch ool
-wi de data Se t instructional focus an d instructional

best practice s Se t school goals and monitor progress Commu nicate im portance of data
-dr iven

cu lture Use PD to t rain t eachers **Train ing**  **an d**  **Pr of** **es sional**  **Dev elopment**  **(P D)**  ( for inst ruc tional leadership

team ) Ongoing PD se ssion s Sum me r refresher training **Improv ed**  **Studen t**  **Ach ievem**  **ent i**  **n Engli**  **sh /**  **Languag e**  **Art s**  **and Ma**  **thematics Te**  **acher Collaborati**  **on Te**  **am s**  **( TCT** **)** De dicated me etings Structured activities to underst and st udent data Exa mine s tudent work Share effective practice s Set and mon itor goals **Te achers**  Rec eive training and coaching **Improv ed**  **Access to**  **and Use of Dat**  **a** Acce ss to student data an d work f rom mu ltiple sou rce s Exa mine student data and student work Use data to se t and monit or goals

**On going** **TA** ( fo r instructional leadership

team ) Monthly calls with coache s In - per son visits Ad hoc con tact s **Dat a**  **Co ach**  Strategizeswith principal An alyzesdata Reviews re se arch Co - leads in structional leader sh ip team

(with principal) Organizes teacher collaboration t eam s Facilitates TCT me etings Provides coaching Develops PD room De velops data walls **Pr incipa** **l** Commu nicates importance ofdata
-driven cu lture An alyzes student data Co - leads instructional le ad ership tea m Monito rs im plem entation

**Con tex** **tu al**  **Factors** Dist rict policies Data coach exper ience Te ach ere xper ience Principal e xper ie nce Access to student data ( interiman d summat ive assessm ent s) **Imp ro**  **ved Ins** **tr uct**  **ional Strategi**  **es** Different iate instructio n Adjust student gro ups Increase instructional tim e on low
perform ing are as Inco rporate evidence
ba se d pract ice s
Participating schools assigned to the treatment will set up new structures to support the implementation of DDI. School leaders and individual teachers will engage in activities designed to help staff learn to analyze data from interim assessments and other sources to identify areas for improvement, jointly formulate improvement goals and priorities, implement evidence

-based changes in instructional practices, and monitor progress toward achieving the agreed

-upon goals. The following people and teams of people will be involved in the intervention in treatment schools:
 **Data coach.**  Participating schools will hire a half

-time data coach to lead and monitor the implementation of DDI activities on a day

-to-day basis. Trained and supported by Focus on Results consultants, the data coach will analyze student data, conduct strategic planning with the principal and other school leaders, and work directly with teachers

— individually and in groups (teacher collaboration teams). These activities will help teachers learn to use availableOMB Supporting Statement: Part A Mathematica Policy Research

5 data effectively to formulate instructional improvement goals, identify relevant best practices, and implement such practices. The coach will create and maintain a DDI resource room with materials to help school leaders and teachers analyze data and research possible instructional practices. The coach also will develop a set of data displays, available in the resource room and throughout the school, to help school leaders and teachers monitor student progress.
 **Principal.** As a school’s main instructional leader, the principal plays a critical role in establishing a data

-driven culture in the school and building support and momentum for DDI implementation. After receiving initial training, the school’s principal and data coach together plan for DDI implementation; they also meet weekly to review progress, troubleshoot issues encountered, and adjust plans for DDI implementation as needed. Principals work to support the initiative by emphasizing the targeted instructional focus and practices during regular classroom visits, follow

-up feedback to teachers, and interactions with the instructional leadership team and data coach.
 **Instructional Leadership**  **Team.** Each part icipating school will establish an instructional leadership team, whose primary role is to lead the school’s DDI implementation effort. The team will include the school’s principal, data coach, grade

-level chairs (and/or department chairs, if instruction is departmentalized), and other formal or informal school leaders (for example, assistant principal, other teachers, or IEP specialist) . This team is charged with analyzing school

-wide data and, based on this analysis, identifying areas of instructional focus for school

-wide DDI improvement efforts (including specific instructional practices linked with these areas), and setting and monitoring goals for increased student achievement. Since DDI implementation relies on a train

-the

-trainer approach to build capacity within each school, this team also has the important responsibility to take what it learns from ongoing professional development sessions with Focus on Results and pass it on to teacher collaboration teams (described next), to guide their efforts. The instructional leadership team is also charged with addressing more concrete issues, such as ensuring that teacher collaboration teams have the time and space to meet on a regular basis.
 **Teacher Collaboration**  **Teams.** Teachers in each participating grade or department will form a professional learning community to work collaboratively, under the leadership of the data coach, to do much of the on-the

-ground work of DDI. Each team will hold regular meetings (at least every other week) to determine what the data (both formal interim assessments and informal assessments) suggest regarding student learning needs, collectively set grade/department

- specific goals and monitor progress toward goals, and identify instructional strategies that could better help them meet the needs of students scoring at the lowest levels.
 **Individual Teacher**  **Activities.** Drawing on their work within the teacher collaboration teams, as well as one-on-one assistance from the data coach, each teacher will implement agreed

-upon instructional strategies in support of both the school

-wide and grade

-level improvement goals. Each teacher will also consider her students’ proficiency levels with respect to different standards (based on assessment data and student work) and make decisions about how best to address the students’ identified learning needs. After implementing the instructional strategies designed to meet these learning needs, teachers will assess the success of their efforts based on feedback from the coach as well as evidence from subsequent student work and assessment data.OMB Supporting Statement: Part A Mathematica Policy Research

6 By collaborating on teams and participating in the activities described above, teachers are expected to develop the knowledge and skills needed to help them adapt and improve their instructional strategies based on student data. The relevant instructional strategies for a given teacher will depend on the specific learning needs of that teacher’s students, as indicated by the student data. DDI does not require that teachers implement any one specific instruct ional approach. Nonetheless, based on Focus on Results’ experiences and best

-practices identified by DDI experts (Hamilton et al. 2009), we expect teachers in treatment schools to be more likely than teachers in control schools to adopt certain strategies or use them more frequently. For example, DDI should lead to a greater use of differentiated instruction, more frequent review and adjustment of students’ small group assignments, and integrating evidence

-based practices into all aspects of their teaching. Schools selected to implement DDI for the study will receive resources, training, technical assistance, and ongoing support from consultants from Focus on Results to help them implement DDI. Capacity

-building inputs will include (1) an initial training for school principals and data coaches, (2) professional development training for instructional leadership teams, and (3) other technical assistance support for data coaches and instructional leadership teams during DDI implementation. The following training and technical assistance activities will take place in treatment schools:
 **Initial Training.**  A two-day orientation and training session ensure s that data coaches and principals are prepared to begin implementing DDI in treatment schools. Activities include both general information about DDI and its implementation and specific training tailored to the activities each group will be expected to perform at their schools. Participants also are provided an opportunity to network and exchange ideas with data coaches and principals at other study schools implementing DDI.
 **Ongoing professional**  **development for**  **instructional leadership**  **team s.**  Each school’s leadership team will receive a total of six pro fessional development sessions to support the team in guiding their school’s adoption of DDI and provide training and skills that can be passed on to teachers at the school. Each session focuses on a particular set of activities related to the implementation of DDI, such as identifying an instructional focus, looking at student data and setting goals, using best practices in teachers’ classrooms, and monitoring their progress . Members of these instructional leadership teams will then train teachers and other staff in their schools using the skills and knowledge they learn from the professional development sessions.
 **Other technical**  **assistance.** Instructional leadership teams will receive customized technical assistance to help address questions and needs that arise as schools implement data driven instruction. Each participating school will be visited a total of eight times by Focus on Results consultants to monitor DDI implementation and provide customized technical assistance. Technical assistance also will be delivered during monthly phone calls with the data coach, and additional phone calls or e

-mails, as needed. **d. Data Collection Plan**  To address the study research questions, the study team will collect and analyze data from teachers, principals, and district staff . A brief description of each data source and data collection activity is provided below.OMB Supporting Statement: Part A Mathematica Policy Research

7  **Teacher and**  **Principal Assignment**  **Data :**  We will collect teacher assignment data from participating schools in winter 2016. We will ask schools to identif y teachers currently teaching fourth and fifth grades, provide contact information for these teachers (such as school e-mail address) , and indicate if the teacher taught at the school in f all 2014 (Appendix A). 4 The study team will attempt to identify principals at study schools from public sources (such as school websites). If that information is not publicly available, we will request schools to provide principal assignment information.  **Teacher Survey:**  We will administer a web

-based survey in spring 2016 to 500 teachers (Appendix B). Teachers will also have the option of completing the survey by hard copy or by phone, if they prefer. The survey will provide information on professional development and training received by teachers (particularly related to DDI topic s), school data culture, instructional planning and collaboration, teachers’ access and use of data to guide instruction, teachers’ instructional strategies, and teachers’ demographic and background characteristics.  **Teacher Logs:**  We will administer web

-based logs with teachers during the 2015

-2016 school year (Appendix C) . The same set of 500 teachers who will be asked to complete the teacher survey will also be asked to complete these logs at two points during the year, and in each case report on their activities over two consecutive days. The logs will provide information on teachers’ day

-to-day activities, including planning activities (individually and in collaboration with other teachers) and instructional strategies in the classroom.  **Principal S** **urvey :**  We will administer a hard

-copy survey to all 104 study principals in spring 2016 (Appendix D) . The survey will focus on topics such as school

-wide leadership activities, school data culture, school

-wide access to and use of data, and principals ’ demographic and background characteristics. Principals will have the option to provide answers to a trained interviewer over the phone.  **Student Records**  **Collection:** In summer 2016, we will collect student outcome measures (for example, math and reading test scores from state assessments) for the year of full implementation (201 5

–2016) as well as for two prior years (2014

–2015 and 201 3

-2014 if available). We will also collect student demographic and background data, such as gender, race/ethnicity, and eligibility for free and reduced

-price lunch, and student attendance and disciplinary measures (if available) , and student enrollment data for fall 2014 and winter 2016 (Appendix E) . **e. Study Activities and Time Line**  In Table 1 , we show the timing of the major study activities. **Table 1. Schedule of Major Study A** **ctivities for**  **D ata**  **- Driven Instruction Evaluation**  Spring 2014 Summer

2014 Fall

2014 Dec

2014 Spring

2015 Fall/ Wi nter 2015 Wi nter/ Spring 2016 Summer/ Fall 2016 Summer

2017 Recruit Districts X X X Conduct Random X X X
4 Teacher assignment data will be collected following approval of the OMB clearance package.OMB Supporting Statement: Part A Mathematica Policy Research

8 Assignment Training and Infrastructure S etup X X Full Implementation X X Collect Teacher Assignment Data* X Conduct Teacher Survey X Conduct Teacher Logs X X Conduct Principal Survey X Collect Student Records Data X Prepare Report X *Schools will be asked to provide principal assignment data only if the information cannot be obtained from publicly availabl e sources. **2. Purposes and Uses of Data**  The main purpose of this evaluation is to estimate the impacts of DDI on student achievement and other outcomes, as well as to document implementation of DDI programs and activities . In Table 2, we list the study’s research questions and the data collection to support the answers. **Table 2. Research Questions and Data Collection Method** Research Questions Data Collection Method 1
. Wh at is the impact of DDI professional development on students’ achievement on state assessments in math and reading?  Student records data 2
. Wh at is the impact of DDI professional development on teachers’ and principals’ access to and use of data to inform instructional support, planning, and practice?  Teacher survey  Teacher logs  Principal survey 3. How does DDI professional development affect teachers’ use of instructional strategies, such as differentiated instruction?  Teacher survey  Teacher logs 4
. How do DDI - related activities differ in treatment versus control schools ?  Teacher survey  Principal survey  Technical assistance documents

 **Teacher and**  **Principal Assignment**  **Data:** Teacher assignment data will be used to randomly select approximately 500 fourth and fifth grade teachers to complete the teacher survey and logs . Principal assignment data will be used to identify the principals in the 104 study schools to determine the sample for the principal survey. Information on teacher and principal assignments in fall 2014 will be used to examine whether the DDI intervent

ion lead to a treatment-control difference in educators’ mobility between fall 2014 (following random assignment) and the 2015

-2016 school year.  **Teacher Survey and Logs**  **:** Teacher survey and log data will be used to analyze teacher intermediate outcomes and to measure important aspects of the treatment-control contrast. Intermediate outcomes that will be examined include the impact of DDI on teachers’ access to and use of data to guide instruction and teachers’ instructionalOMB Supporting Statement: Part A Mathematica Policy Research

9 strategies (such as the use of differentiated instruction ) . We will also examine treatment-control diff

erences in the training teachers receive and other activities surrounding the use of data in schools.  **Principal Survey** **:** The principal survey will enable a better understanding of the treatment-control contrast in schools’ training and activities surrounding the use of data, as well as shed light on some aspects of DDI implementation. It will examine topics such as whether principals have established school

-wide goals, their expectations for school

-wide data use, whether the school has an instructional leadership team and the activities it undertakes, and the degree to which school leaders review and discuss student data.  **Student Records Collection:** We will use existing district test score data to estimate the impact of DDI on student achievement, the key outcome of interest. Information on students’ demographic and socioeconomic characteristics and their achievement test scores prior to the study school year will be used to describe the students in the study and to develop more precise impact estimates. In the process of implementing the intervention and providing technical assistance to treatment schools, we will also obtain information on aspects of implementation from the evaluation team’s technical assistance provider (Focus on Results) , such as materials from their professional development sessions. We will present the study findings in a report that will include information from both the implementation and impact analyses. The data collected by this evaluation will also be available as restricted

-use files, serving as a valuable resource for researchers. **3. Use of Technology to Reduce Burden**  The data collection plans are designed to obtain reliable information in an efficient way that minimizes respondent burden. When feasible, we will gather information from existing data sources, using the most efficient methods available. A web

-based survey and web

-based logs will be the primary modes of data collection for teachers. Respondents will also have the option of completing a self

-administered hard copy questionnaire or providing answers to a trained interviewer over the phone. The survey and logs will enable respondents to complete them at a location and time of their choice, and its automatic editing system will reduce the number of response errors. Using email to follow up with nonrespondents will also offer an additional convenient option for respondents because email reminders will include a link to the survey website and a username-password combination, as well as an attached PDF of the survey if respondents choose to complete it in hard copy. The principal survey will be administered as a hard copy questionnaire . Principals will also have the option to provide answers to a trained interviewer over the phone. **4. Efforts to Avoid Duplication of Effort**  The study seeks to provide new evidence on the impact of a comprehensive DDI program, as well as factors and context that might affect DDI effectiveness. The experimental studies that have been conducted do not examine a comprehensive DDI program that incorporates intensive professional development and coaching or regular teacher collaboration. Moreover, much of theOMB Supporting Statement: Part A Mathematica Policy Research

10 existing work examines branded interventions using proprietary assessments, and cannot shed light on the effectiveness of a general DDI approach. **5. Methods to Minimize Burden on Small Entities**  No small businesses or entities will be involved as respondents. **6. Consequences of Not Collecting Data**  The data collection plan described in this submission are necessary for ED to conduct a rigorous national evaluation of DDI and to understand the effectiveness of this education reform strategy. The study represents an important step toward developing a systematic and rigorous evaluation agenda in the area of school reform. Without the data collected in this evaluation, states, districts, schools and policymakers will not know if their considerable investment in DDI is improving student learning, whether additional investment in this strategy is merited, and what challenges districts may encounter implementing a comprehensive DDI program and how to overcome those challenges . The consequences of not collecting specific data are outlined below:
 **Teacher and Principal**  **Assignment Data:**  Without teacher assignment data, the study will not be able to identify the pool of fourth and fifth grade teachers from which a random sample of teachers will be chosen to complete the teacher survey and logs . Without principal assignment data, the study will not be able to identify the principals who will be asked to complete the principal survey. 5
 **Teacher Survey**  **and Logs**  **:** Without the data from the teacher survey and logs, we will not be able to examine the impact of DDI on teacher

-level outcomes such as their review and use of student data, collaboration, and instructional strategies . In addition, we will not be able to examine treatment-control differences in the training they receive and other activities surrounding the use of data in schools.  **Principal Survey** **:** Without the principal survey data, we will not understand or be able to describe treatment and control differences in school contextual factors (such as the school culture pertaining to data use), or school

-wide data

-use activities.  **Student Records**  **Collection:** Without the study records, we will have to administer assessments to students in place of using their district test scores to measure student achievement. Without the data on student characteristics, we will not be able to fully describe the study sample and verify the effectiveness of the random assignment. **7. Special Circumstances**  There are no special circumstances associated with the school recruiting effort.

5 Schools will only be asked to provide principal assignment data if this information is not publicly available.OMB Supporting Statement: Part A Mathematica Policy Research

11 **8. Federal**  **Register Announcement and Consultation** 
**a. Federal Regis** 

**ter Announcement (PENDING)**  The 60

-day notice to solicit public comments was published in Volume XX, page XXX of the Federal Register on XX XX, 20XX. **b. Consultations Outside of the Agency**  In formulating the study design, DDI intervention, surveys, and teacher logs, the study team sought input from experts in DDI, evaluation methodology, and education policy. Table 3 lists the experts who were consulted. **c. Unresolved Issues**  None. **Table 3. Experts Consulted Name**  **Title and Affiliation Expertise**  Geoffrey Borman Professor of Education, University of Wi sconsin - Madison Evaluation methodology; education policy Russell Gersten Executive Director, Instructional Research Group Education policy; reading and math instruction Laura Hamilton Associate Director, Rand Data-driven instruction; evaluation methodology Ken Koedinger Professor of Human Computer Interaction and Psychology, Carnegie Mellon University Education policy; educational technologies; experimental methods Spyros Konstantopoulos Professor of Measurement and Quantitative Methods, Michigan State University Quantitative and experimental methods; education research; data-driven instruction Ellen Mandinach Senior Research Scientist, WestED Data-driven decision making Jon Supovitz Director, Consorti um for Policy Research in Education, University of Pennsylvania Data-driven decision making; teacher and principal professional development in using data Leslie Nabors Olah Managing Research Scientist, Educational Testing Service Data-driven decision making; teacher professional development in using interim assessment data Jeff Wayman President, Waym an Services, LLC Data-driven instruction and decision making; program evaluation Martin W est Associate Professor of Education, Har vard University Data-driven instruction; evaluation methodologyOMB Supporting Statement: Part A Mathematica Policy Research

12 **9. Payments or Gifts**  Incentives have been proposed for the teacher surveys and logs and the principal surveys to partially offset respondents’ time and effort in completing the surveys. We propose offering a $ 30 incentive to both teachers and principals after completion to compensate them for the 30 minutes required to complete the questionnaire. We also propose a $15 incentive to teachers in study schools for each completed teacher log in order to compensate them for the 15 minutes we expect each log to take to complete. These proposed amounts are within the incentive guidelines outlined in the March 22, 2005 memo, “Guidelines for Incentives for NCEE Evaluation Studies,” prepared for OMB. Incentives are also proposed because high response rates are needed to make the survey findings reliable, and we are aware that teachers are the targets of numerous requests to complete surveys on a wide variety of topics from state and district offices, independent researchers, and ED . Although some districts will have solicited buy

-in from teachers to participate in the evaluation, our recent experience with numerous teacher surveys supports our view that obtaining teacher buy

-in does not guarantee teachers will devote the time it takes to complete a survey or log, and monetary incentives increase the likelihood of cooperation of school staff. **10. Assurances of Confidentiality**  The data collection efforts, including student records, teacher survey, teacher logs, and principal survey, will be conducted in accordance with all relevant regulations and requirements. These include the Education Sciences Institute Reform Act of 2002, Title I, Part E, Section 183, that requires “[a]ll collection, maintenance, use, and wide dissemination of data by the Institute ... to conform with the requirements of section 552 of Title 5, United States Code, the confidentiality standards of subsections (c) of this section, and Sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the Privacy Act, the Family Education Rights and Privacy Act, and the Protection of Pupil Rights Amendment. In addition, for student information, the project director will ensure that all individually identifiable information about students and their academic achievement, as well as information regarding individual schools will remain confidential in accordance with Section 552a of Title 5, United States Code, the confidentiality standards subsection (c), and Sections 444 and 445 of the General Educations Provision Act. Subsection (c) of Section 183 , referenced above, requires the director of IES to “develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data.” The study will also adhere to requirements of subsection (d) of Section 183 prohibiting disclosure of individually identifiable information as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a felony. Mathematica and its subcontractors will protect the confidentiality of all information for the study and use it for research purposes only. When reporting results, data will be presented in aggregate form only, such that individuals and institutions will not be identified. A statement to this effect will be included with all requests for data. All members of the study team with access to the data will be trained and certified on the importance of confidentiality and data security. All data will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required.OMB Supporting Statement: Part A Mathematica Policy Research

13 The following safeguards are routinely employed by Mathematica to carry out confidentiality assurances during the study:
 All Mathematica employees sign a confidentiality pledge (Appendix H) emphasizing its importance and describing their obligation.  Identifying information is maintained on separate forms and files, which are linked only by sample identification number.  Access to hard

-copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.  Computer data files are protected with passwords and access is limited to specific users.  Especially sensitive data are maintained on removable storage devices that are kept physically secure when not in use. **11. Additional Justification for Sensitive Questions**  We do not anticipate that any of the data collected will contain items considered to be of a sensitive nature. **12. Estimates of Hours Burden**  Table 4 provides an estimate of time burden for the data collection activities . The total of 779 burden hours for data collection involves collecting student records from districts, teacher assignment data from schools, surveying principals, and surveying and obtaining logs from teachers . These estimates are based on our experience collecting data for other evaluation studies . The total of 779 burden hours covers all three years of data collection approval, and includes the following efforts: an annualized total of 32 hours (96/3) for each of the 12 districts to collect and assemble administrat

ive records on students participating in the evaluation; an annualized total of 9 hours (104*.25/3) for school staff to provide teacher assignment data (and principal assignment data if not publicly available) ; an annualized total of 15 hours (88*.5/3) to complete a 30 minute principal survey in the spring of 2016 for 88 principals (85 percent of the 1 04 principals in the sample) ; an annualized total of 71 hours (425*.5/3) to complete a 30 minute teacher survey in the spring of 2016 for 425 teachers (85 percent of the 500 teachers in the sample); and an annualized total of 133 hours (400/3) to complete 4 rounds of teachers logs of 15 minutes each. **Table 4**  **. Estimated Response Time**  **for Data Collection Activit i**  **es**
Respondent Instrument Number of Targeted Respondents Expected

Response Rate (%) Number

of Respondents Unit

Response Time (Hours) Total

Response Time (Hours) Total

annual hours (hrs/3) **Districts** Student records collection **a** 12 100 12 8 96 32 **Schools** Teacher assignment data b 104 100 104 0.25 26 9 **Principals** Surveys 104 85 8 8 0. 5 44 15OMB Supporting Statement: Part A Mathematica Policy Research

14 **Teachers** Surveys 500 85 425 0.5 21 3 71 Logs 500 80 400 1 c 400 133 **Overall Total 1029**  **77 9**  **2 60**  NOTE: Reporting on an annualized data collection over 3 years. Annual number of respondents is 343 [(12+ 104+ 88+425+400)/3] a In some cases, we might request this data from schools
. b If principal assignment data is not publicly available, schools will also be asked to provide this information for principals. c Reflects 4 completed logs for each respondent. **13. Estimates of Cost Burden to Respondents**  There are no additional respondent costs associated with this data collection beyond the burden estimated in section 12 above . **14. Estimates of Annual Costs to the Federal Government**  The total cost to the f

ederal government of carrying out this study is $9,697,224, to be expended over the study period of four years . The estimated average annual cost of the study is $2,424,306 . **15. Reasons for Program Changes or Adjustments**  This is a new data collection. **16. Plan for Tabulation and Publication of Results**  We discuss our plans below for tabulating data for the final report to address the research questions and publishing results. **a. Tabulation Plans**  Our tabulation plans include implementation and impact analyses aligned to the research questions. Nonexperimental analyses will describe implementation and the association of specific features of DDI programs with DDI impacts. Random assignment of schools within districts to a treatment group that will implement a comprehensive DDI program or to a control group that will not is an ideal design for assessing overall effectiveness of DDI . Our primary impact analysis will exploit this experimental design to provide rigorous estimates of the impact of DDI on student achievement and other student outcomes. Additional experimental analyses are designed to estimate the impact of a comprehensive DDI program on teacher outcomes, such as teachers’ access to and use of student data and teachers’ use of instructional strategies . **i. Implementation**  **a nalysis**  The implementation analyses have two main objectives . The first objective is to describe the DDI activities implemented in treatment schools and the extent to which they were implemented with fidelity. The second objective is to describe differences in treatment and control schools’ implementation of DDI

-related activities. The analyses will rely upon information from the dataOMB Supporting Statement: Part A Mathematica Policy Research

15 coach weekly activity logs and the Focus on Results consultant activity logs, as well as from the principal survey, teacher survey, and teacher logs. **F idelity**  **of DDI**  **implementation.** The implementation analysis will describe DDI activities undertaken by treatment schools and the extent to which they were implemented with fidelity to the intervention plan. As a part of this analysis, we will review log entries from January 2015 through spring 2016. Because it may take time for treatment schools to set up necessary school structures and roll out intervention activities, we expect some variance in implementation at different points in time during the intervention. Thus, we will examine fidelity of implementation at three points : (1) at the end of the 2014

-2015 school year, (2) in December 2015, and (3) at the end of the 2015

-2016 school year. To assess the extent to which treatment schools implemented DDI with fidelity, it will be important to quantify the data and summarize it unif ormly. The DDI intervention aims to build school capacity in two ways, by (1) helping schools set up structures and activities that enable school staff to carry out DDI, and (2) training and coaching teachers in the skills needed to use and interpret student data. Implementation fidelity measures will therefore examine the training and coaching that treatment schools receive and the degree to which treatment schools fully implement the anticipated DDI structures and activities. In describing the fidelity of implementation of a particular DDI component, we will focus on the activities we expect to have happened if the component has been implemented. We will measure both whether the activity occurred and, if appropriate, the level of participation in the activities. To measure the fidelity of implementation of the instructional leadership team meetings, for example, we will measure whether a school holds monthly instructional leadership team meetings among all key staff, attendance at these meetings among the staff, and the key activities taking place at these meetings. These activities would include that the team establishes school

-wide goals for performance, establishes areas of instructional focus, and examines student data. Our examination of implementation fidelity will be accompanied by descriptive text that summarizes our findings, describes changes in implementation over the course of the intervention, and provides a range of examples of how schools implemented DDI with fidelity. We will also describe treatment schools’ experiences they faced in implementing DDI. This information will be especially important if the DDI approach is found to be effective and other districts wish to replicate the program. We will examine school

-level factors that may influence DDI implementation, such as the level of engagement of the school principal. **Implementation of**  **DDI -** **related activities**  **in treatment**  **and co** **ntrol schools.**  As depicted in the logic model (Figure 1), the DDI intervention is expected to lead the principal, data coach, instructional leadership team, teacher collaboration teams, and individual teachers in treatment schools to engage in numerous data-focused activities. These include (1) activities undertaken by school leaders in directing data use, (2) school

-wide communications about data use, (3) professional development and support for the principal and teachers on data use, and (4) collaboration among teachers to review student data and share instructional strategies. We will use data from the principal survey, teacher survey, and teacher logs to examine whether implementation of the DDI intervention leads to a treatment

-control difference in these activities (Table 5) . **ii. Impact**  **a nalysis** OMB Supporting Statement: Part A Mathematica Policy Research

16 The impact analysis will rigorously assess the effectiveness of a comprehensive DDI program. Calculating the statistical significance of the impacts requires that the nested structure of the data

— with students clustered in schools

– is incorporated in the analysis. Due to clustering, the variance of the impact estimates is larger than it would have been if each individual student were randomly assigned to DDI. Below we describe our approach to calculating outcomes of student achievement and intermediate outcomes on teacher and principal practices. Studentfinal outcomes. Student achievement will be the study’s primary outcome, measured using spring 2016 math and reading scores on state standardized tests. Specifically, this main impact analysis will examine the effect of DDI on:
 Math achievement among fourth and fifth graders in a school that implemented data

driven instruction
 ELA achievement among fourth and fifth graders in a school that implemented datadriven instruction
**Table 5**  **. Data -** **related A** **ctivities, by S**  **ource** Principal Survey Teacher Survey Teacher Logs Activities undertaken by school leaders in directing data use Frequency of leadership team meetings X Leadership team m embers X Leadership team activities (e.g., setting achievement and priority learning goals , monitoring progress, planning professional development activities) X Degree to which school leaders ensure that teachers have the time and resources needed to analyze and interpret student data X X School -wide communications about data Frequency of communication on student achievement goals and results X X Frequency of communication on priority learning goals X X Frequency of communication on expectations for and actual use of data X X Use of data displays in classrooms and other public areas X X Professional development and support for the principal, teachers around data use Amount of professional development/training activities this school year X X Topics covered by trainings (analyzing student data, establishing priority learning goals for the school , individualizing student learning goals, tracking progress toward goals , using evidencebased instructional strategies ) X X Availability of on -site coaching/support for data use X X Data coach activities X Frequency of, and topics addressed during, individual coaching X X F requency of classroom observations and feedback X X Collaboration among teachers around data use and sharing instructional strategies Frequency of, and amount of time spent on, teacher collaboration X XOMB Supporting Statement: Part A Mathematica Policy Research

17 Principal Survey Teacher Survey Teacher Logs Teacher collaboration activities (analyzing student data , setting common learning goals for students, sharing effective instructional practices , j ointly modifying lesson plans, monitoring im plementation and results of instructional changes ) X X

To assess the impact of data

-driven instruction on student achievement, we will use a place

based impact estimat

ion strategy that compares outcomes for students in treatment schools to those of students in control schools using spring 2016 state test scores. This implies that the impacts could reflect either the impacts of data

-driven instruction on student achievement or impacts on student mobility. Given the nature of the intervention, we do not expect that the data

-driven instruction intervention would affect students’ mobility patterns during the study period. However, we will test this empirically through the analysis of impacts on student mobility based on the student sample in fall 2014 (following randomization) and spring 2016 (at the conclusion of the intervention). The main impact analysis based upon spring 2016 test scores will estimate an impact model in which the overall impact estimate is based on treatment-control differences in the outcome of interest within each stratum (matched school pair) . Although a simple treatment-control difference in mean outcomes will yield an unbiased estimate of the impact of DDI, the precision of estimates can be improved by controlling for baseline characteristics that may influence the outcomes of interest but are not related to the treatment itself. For all the student outcomes, we will control for baseline student and school covariates. If available from state or district records, specific student

level covariates will include prior years’ (spring 2015 and, if available, spring 2014 ) math and reading test scores (in z- score units) , prior years’ student attendance, prior years’ student suspensions, gender, race/ethnicity, eligibility for free or reduced

-price lunch, English language learner status, and special education status. 6 We also anticipate including school

-level aggregates of these variables and other relevant school characteristics in cases where individual student

-level data are not available . Accordingly, we will estimate student impacts using the following model: (1) = + + + +

6 Since the intervention begins in December 2014, students’ spring 2015 will not be measured during a true baseline period. However, we argue that it is unlikely that the intervention will have an effect on sample members’ spring 2015 scores, for two reasons. First, sample members in fourth grade during 2015

–2016 will typically have been in third grade in spring 2015, and the intervention does not cover this grade level. Second, intervention activities during the first several months of the intervention period (that is, through the early months of 2015) will focus on providing support to schools in setting up the structures necessary to implement comprehensive DDI and on providing training to teachers in the overall intervention approach and in the initial steps of understanding student data. Thus, we argue that these early activities would be unlikely to lead to instructional changes that affect the spring 2015 achievement levels of the study’s fifth grade cohort. However, we will address this possibility by examining the sensitivity of the results to a model that uses spring 2014 achievement levels for this cohort, as well as to a model that excludes baseline achievement altogether. We will also estimate a model in which the spring 2015 test score is the dependent variable. If our premise that the intervention was not sufficiently implemented to affect student achievement by this time, we would expect to see no significant treatment

-control difference in spring 2015 test scores.OMB Supporting Statement: Part A Mathematica Policy Research

18 where y

ijk is the outcome of individual student i in school j within stratum k; αk is a vector of stratum (matched pair) indicators (fixed effects) included to control for differences across strata in average student, teacher, principal and school characteristics; T jk is a treatment indicator that equals one if the school was assigned to DDI and zero otherwise; X ijk is a vector of baseline individual student characteristics; u jk is a school

- specific random error term; ε ijk is an individual

-level random error term; and 

, γ, and δ are parameters to be estimated. The estimate of  represents the overall impact of DDI on the student outcome of interest. We will estimate the model with ordinary least squares (OLS) using standard errors that account for school

-level clustering. We will consider student achievement in reading and math to be separate domains, across which the impact of data

-driven instruction might differ. Given that assessments differ across state, grade level, and subject area, we will standardize the raw achievement scale scores by converting them to z- scores. We will calculate the z- scores by subtracting the state mean score from the raw scale score and dividing by the standard deviation of the state scores. We will also estimate impacts on other student

-level outcomes, such as attendance and suspensions, if possible . We will request student

-level data on these outcomes from districts or states, but if the data are not available or are unreliable, we will estimate impacts on these outcomes measured at the school (or, if available, grade) level. Impacts on a school

-level outcome y jkd can be estimated using model (1) above but including school

-level averages of individual

-level covariates rather than the individual

-level covariates themselves. The estimation of overall impacts on student achievement may mask differences in impacts across subgroups of students. For example, DDI may prove more or less effective at boosting student achievement for students with different baseline characteristics; it may, for example, raise achievement among lower performing students to a greater degree than it does among higher performing students. We will provide a subgroup analysis of the impacts of DDI for student groups based on their level of baseline achievement, focusing on impacts for particularly low

-achieving students as well as for students at moderate to high baseline achievement levels. To estimate impacts for student subgroups, we will create a version of the model that interacts a subgroup indicator with the treatment indicator; the coefficient on the subgroup

-treatment indicator interaction term will represent the impact estimate for the subgroup. Teacher andprincipal intermediate outcomes . The core set of professional development and technical assistance inputs under DDI are intended to help teachers and principals use data to improve instruction, which in turn would lead students to realize higher achievement gains. We will use responses from the teacher survey, teacher logs, and the principal survey to examine these intermediate outcomes. Teachers and principals offer different vantage points from which to assess the extent to which schools engage in data use activities. For example, principals may be able to provide detailed information on school leadership activities, while teachers may be able to provide detailed information on the frequency and content of teacher collaboration activities. The surveys will provide useful information on the frequency of activities over an extended period of time (such as how often a teacher attended professional development during the school year), while teacher logs will provide a one-day snapshot of teacher activities that occur relatively frequently (such as lesson planning and collaboration based on analysis of data). Table 6 lists the measures and their sources that will be used to estimate impacts on intermediate outcomes. The first t wo sets of items listed in the table measure intermediateOMB Supporting Statement: Part A Mathematica Policy Research

19 outcomes related to teachers’ access to and use of data to guide instruction. The second two sets of items capture information on teachers’ instructional strategies. To assess the impact of DDI on teachers’ use of data and instructional strategies, we will compare outcomes for teachers in treatment schools to those of teachers in control schools. Our impact model for teacher outcomes will take a similar approach to that taken in the student outcomes model, calculating treatment-control differences among teachers . For all teacher outcomes, we will control for teacher covariates derived from the survey. Teacher covariates will include years of teaching experience, gender, race/ethnicity, teacher certification, and an indicator for a master’s degree. Similar to our approach in the student model, we will estimate the model with ordinary least squares (OLS) using standard errors that account for school

-level clustering, and we will compute the overall average impact of DDI by taking a weighted average of the coefficients on the treatment indicators . Each district

- specific impact will be weighted by the number of study schools in each district.
**Table 6**  **. Teacher and Principal Intermediate Outcome Measures, by**  **Source** Teacher Survey Teacher Logs Principal Survey **Ac cess**  **to student -** **level data**  Access to interim assessment results X X Access to summative assessment results X X Access to student background characteristics, attendance, and school behavior information X X Barriers to data use (usable format, technology, tools) X X **Use of s** **tudent -** **level d**  **ata** Frequency of use by type (summative assessment results, interim assessments, formative assessments, samples of student work, student characteristics) X X X Purposes of data use (understand student needs, set learning goals, monitor progress toward goals, differentiate instruction, revise lesson plans) X X Understanding of instructional changes to make based on data X X **Differentiated Instruction** Placing students in small groups based on student data X Providing small group instruction X X Providing individualized instruction X X Identifying and referring students in need of pull -out services or other intensive interventions X X Changing instructional group assignments of students based on student data X X **Whole -** **Class Instruction** Providing additional instruction in areas where students are struggling X X I dentifying evidence- based instructional changes to help address students’ needs X Using new instructional strategies to teach challenging concepts to students XOMB Supporting Statement: Part A Mathematica Policy Research

20 Different types of teachers may be differentially affected by the implementation of DDI. For example, less experienced teachers may be more (or less) at ease in using data to inform instruction than more experienced teachers . They also may be able to benefit to a greater extent from the information on student performance provided by student data, since they will be less able to rely on experience to understand their students’ needs. The study will also examine impacts on teacher outcomes separately for subgroups of teachers defined by their level of experience. Similar to the approach used in the student subgroup analysis, we will estimate subgroup impacts by creating a version of the model that interacts a subgroup indicator with the treatment indicator. Our approach to analyzing principal outcomes will similarly compare principal survey responses regarding the structures and professional development activities in treatment and control schools. However, while the teacher and student models will be estimated at the classroom and student levels, respectively, the principal model will be estimated at the school level. We will adapt the student

-level model presented in equation (1 ) to est imate impacts on principal

-reported measures. For all principal survey analyses, we will control for principal covariates, such as years of experience, gender, race/ethnicity, and an indicator for a master’s degree. **ii i** **. Nonexperimental**  **a nalysis**  **S chool**  **contextual factors.**  Contextual factors, such as student characteristics or principal and teacher background characteristics, may aid in the interpretation of the impact of DDI in treatment schools relative to control schools. We will therefore examine how school contextual factors are related to impacts. Examples of contextual factors include:
 **School characteristics.**  School math and ELA proficiency measured at baseline, the percentage of students eligible for free or reduced

-price meals, the percentage that are English language learners, the percentage in special education programs, and racial/ethnic composition of the school.  **Teacher and**  **principal characteristics**  . Education, experience, and background characteristics. The study will use descriptive analyses and regression analyses to examine how impacts are related to school contextual factors. We will use the descriptive analysis to identify conditions and practices that are candidates for regression analyses of impacts. **Correlational a** **nalyses of**  **impacts and**  **implementation fidelity** **.** The potential impact of DDI on student achievement may differ within treatment schools based on aspects of their fidelity of implementation of DDI. To explore how school characteristics and other contextual f actors may influence student impacts, we will conduct a correlational analysis that examines the relationship between estimated impacts of DDI and key features of treatment schools (or of matched pairs of treatment and control schools when data are available for the control schools). For example, we may examine whether the qualifications of the data coaches hired at each treatment school are related to impacts on student achievement. Because data coaches play a central role in supporting the DDI intervention at each school, the prior skills and knowledge they bring to the work may influence the degree to which teachers are supported in analyzing student data and using them to improve instruction, which may in turn influence student outcomes. UnderstandingOMB Supporting Statement: Part A Mathematica Policy Research

21 the degree to which the success of the DDI intervention hinges upon the qualifications of the data coach may prove helpful in developing effective DDI interventions in the future. At the time they were interviewed for the position in fall 2014, each data coach completed a data proficiency exercise developed by Focus on Results to assess their initial skills in analyzing student assessment data. As above, we will correlate the data proficiency score of each coach with the estimated impact of the coach’s school. Because schools cannot be randomly assigned to specific DDI contexts, this correlational analysis will be nonexperimental. We will stress that any significant relationships between impacts and DDI features or contextual factors might not be causal and might reflect the influence of other unobserved factors. **b. Publication Plans**  We will prepare one report that will present the results of the implementation and outcomes analysis. The projected release date of this report is Summer 2018 . Reports will be written in a style and format accessible to policymakers and research

- savvy practitioners, and will comply fully with the standards set by the National Center for Education Statistics. **17. Approval**  **Not to Display the OMB Expiration Date** The study will display the OMB expiration date. **18. Explanation of Exceptions**  No exceptions are being sought.OMB Supporting Statement: Part A Mathematica Policy Research

22 **RE FE**  **RENCES** Carlson, D., G. Borman, and M. Robinson. “A Multi

-State District

-Level Cluster Randomized Trial of the Impact of Data

-Driven Reform on Reading and Mathematics Achievement. Educational Evaluation and Policy Analysis, vol. 33, 2011, pp. 378

–398. Cordray, D., G. Pion, C. Brandt, A. Molefe, and M. Toby. “The Impact of the Measures of Academic Progress (MAP) Program on Student Reading Achievement. ” (NCEE 2013

-4000). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, 2012. Datnow, A., V. Park, and P. Wohlstetter. “Achieving with Data: How High

-Performing School Systems Use Data to Improve Instruction f or Elementary Students. ” Los Angeles: Center for Educational Governance, University of Southern California, 2007. Hamilton, L., R. Halverson, S. Jackson, E. Mandinach, J. Supovitz, and J. Wayman. “Using Student Achievement Data to Support Instructional Decision Making. ” NCEE 2009

-4067. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, 2009 . Available at http://ies.ed.gov/ncee/wwc/publications/practiceguides/
. Retrieved DATE. Henderson, S., A. Petrosino, S. Guckenburg, and S. Hamilton. “Measuring How Benchmark Assessments Affect Student Achievement” Issues and Answers Report, REL 2007

–No. 039. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northeast and Islands, 2007 . Konstantopoulos, S., S. Miller, A. van der Ploeg, C. Li, and A. Traynor. “The Impact of Indiana’s System of Benchmark Assessments on Mathematics Achievement. ” Educational Evaluation and Policy Analysis, vol. 35 , no. 4, 2013, pp. 481

–499. Lachat, M. , and S. Smith. “Practices That Support Data Use in Urban High Schools . ” Journal of Education for Students Placed at Risk, vol. 10, no. 3, 2005, pp. 333

–349. Quint, J., S. Sepanik, and J. Smith. “Using Student Data to Improve Teaching and Learning: Findings f

rom an Evaluation of the Formative Assessments of Student Thinking in Reading (FAST

-R ) Program in Boston Elementary Schools . ” New York, NY: MDRC, 2008 . Shaw, Shana, and Jeffrey C. Wayman. "Third

-Year Results From an Efficacy Study of the Acuity Data System." (2012). Slavin, R., A. Cheung, G. Holmes, N. Madden, and A. Chamberlain. “Effects of a Data

-Driven District Reform Model on State Assessment Outcomes. ” American Educational Research Journal , vol. 50, no. 2, 2013, pp. 371

–396. U.S. Department of Education. “Sec. 2121. Allocations to local education agencies.” Available at [http://www2.ed.gov/policy/elsec/leg/esea02/pg22.html/]. Accessed November 6, 2013. Williams, R., A. Swanlund, S. Miller, S . Konstantopoulos, and A. van der Ploeg. “Making Sense of Unanticipated Results: Instructional Differentiation and the Indiana Diagnostic Assessment Study. ” Society for Research on Educational Effectiveness (SREE) Spring Conference, 2013**www.mathematica -** **mpr.com**

Improving public well - being by conducting high -quality, objective research and surveys Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research