**Impact Evaluation of Data** **- Driven**  **Instruction Professional Development for Teachers**  **Supporting Statement Part B** 
May 26 , 201 5Contract Number:

ED- IES- 12-C-0086 Mathematica Reference Number: 40166-700 Submitted to: Institute of Education Sciences IES/NCEE U.S. Department of Education 555 New Jersey Avenue, NW Washington, DC 20208 Project Officer: Erica Johnson Submitted by: Mathematica Policy Research 600 Alexander Park Suite 100 Princeton, NJ 08540 Telephone: (609) 799-3535 Facsimile: (609) 799-0005 Project Director: Phil Gleason **Impact Evaluation for Data** **- Driven**  **Instruction Professional Development for Teachers**  **Supporting Statement Part B**  May 26 , 201 540166 OMB Supporting Statement: Part B Mathematica Policy Research

iii **CONTENTS** PART A: JU STIFICATION ................................ ................................ ................................ 1

1. Circumst ances Necessitating the Collection of Information ................ 1 2. Purposes and Us es of Data ................................ ................................ 8 PART B: COLLECTION O F INFORMATION EMPLOYING STATISTICAL METH ODS ................ 9

1. Respondent Universe and Sampling Methods ................................ ..... 9 2. Stat ist ical Methods for Sample Selecti on and Degree of Accuracy

Needed ................................ ................................ ............. 13 3. Met hods to Maximize Response Rates and Deal wit h Nonresponse ................................ ................................ ................... 21 4. Pilot Testing ................................ ................................ .................... 22 5. Individuals Consulted on Statistical Aspects of the Design ............... 22 REFERENCES ................................ ................................ ................................ ............... 24
APPENDIX A: TEACHER ASSIGNMENT DATA REQUEST APPENDIX B: TEACHER SURVEY QUESTIONNAIRE AND ACCOMPANYING LETTER APPENDIX C: TEACHER ACTIVITY LOG AND ACCOMPANYING LETTER APPENDIX D: PRINCIPAL SURVEY QUESTIONNAI RE AND ACCOMPANYING LETTER APPENDIX E: STUD ENT RECORDS REQUEST APPENDIX F: CONFI DENTIALITY PLEDGE40166 OMB Supporting Statement: Part B Mathematica Policy Research

iv **TABLES** Table 1. Data
- related Activities, by Source ................................ ................................ .. 14 Table 2. Teacher and Principal Intermediate Out come Measures, by Source ................ 17 Table 3. Minimum Detectable Effects by Sample Size ................................ .................. 1940166 OMB Supporting Statement: Part B Mathematica Policy Research

1 **SUPPORTING STATEMENT FOR PAPERWORK**

**RE DU**  **CTION ACT SUBMISSION** This OMB package requests clearance for data collection activities for a rigorous evaluation of data

-driven instruction (DDI) in 104 schools from 12 school districts that have been recruited for the study. Data

-driven instruction involves the use of student assessment data to help teachers adapt their instruction and, ultimately, improve student achievement. The study’s intervention plan will build school capacity for DDI by: (1) helping schools set up structures and processes that enable teachers and other school staff to efficiently carry out data

-driven instruction, and (2) training and coaching teachers in the skills needed to understand student data and implement improved instructional strategies to address student needs. By participating in professional development and ongoing DDI activities, teachers are expected to develop the knowledge and skills needed to help them adapt and improve their instructional strategies based on student data, and ultimately, improve student achievement. We plan to collect student records and teacher

-assignment data from participating districts and schools, and conduct a teacher survey, teacher logs, and a principal survey. The Institute of Education Sciences (IES) within the Department of Education (ED) has contracted with Mathematica Policy Research and its partners Abt Associates and Evidence

-Based Education Research & Evaluation to conduct the evaluation, and Public Consulting Group

–Focus on Results to provide technical assistance to schools implementing the DDI program. The evaluation’s main objectives are to understand how DDI is implemented and to rigorously estimate the impact of a comprehensive DDI program on student achievement and teacher and principal practices. The implementation component will use information collected from the technical assistance (TA) team, a teacher survey and logs, and a principal survey. For the impact evaluation, the experimental design involves randomly assigning schools within a district to either a treatment or control group. The treatment schools will implement a comprehensive DDI program, with technical assistance provided by an organization that works with schools to implement such instruction, and the control schools will not implement new DDI initiatives during the study years. Student outcomes will include students’ achievement on math and reading state assessments. Teacher and principal outcomes will include teachers’ and principals’ use of data, teachers’ instructional strategies, and the extent and nature of teacher collaboration. This OMB clearance request concentrates on materials that will be used to collect principal, teacher, and student data. Included in this request are the following: a teacher assignment data request (Appendix A

) , a teacher survey questionnaire and an accompanying letter (Appendix B) , a form and letter for the administration of teacher logs (Appendix C), a principal survey questionnaire and accompanying letter (Appendix D) , a student records request (Appendix E) , and a confidentiality pledge (Appendix F) . **PART A :**  **JUSTIFICATION 1.**  **Circum stances**  **Necessitating the Collection of Information**  **a. Statement of Need for a**  **Rigorous Evaluation of**  **DDI** The specific piece of legislation authorizing this evaluation is Title II, Part A of the Elementary and Secondary Education Act (ESEA ) , Section 2121

-2123 as amended by No Child Left Behind (NCLB) (20 USC 6621

-6623). The Title II legislation provides grants to states and districts to40166 OMB Supporting Statement: Part B Mathematica Policy Research

2 increase student academic achievement. State and local education agencies can use Title II funding to offer professional development activities that “provide training on how to understand and use data and assessments to improve classroom practice and student learning” (U.S. Department of Education, 2013). The federal government is also actively promoting DDI through other initiat ives, including Race to the Top and School Improvement Grants. A growing number of schools across the country have adopted DDI programs. Although there are no comprehensive national data on the current number of schools implementing DDI

— in part due to differing definitions of this intervention

—DDI providers report implementing DDI programs in hundreds of districts and tens of thousands of schools. 1 One provider reports that its DDI materials and training are used in 20 percent of K

–12 school districts nationwide (Cordray et al. 2012). Despite the growing interest in data use across the country, there is limited rigorous evidence on the effects of comprehensive DDI, especially for programs that feature intensive on-site support for teachers and other school staff. The few rigorous evaluations of DDI typically find no statistically significant impacts on student achievement. 2 Several of these experimental studies have had methodological limitations, however, and the DDI interventions they examined did not provide substantive supports at the school and teacher levels (Konstantopoulos et al. 2013; Cordray et al. 2012; Carlson et al. 2011; Slavin et al. 2013). In general, the studies have focused on interventions that did not include teacher collaboration, an important best practice (Hamilton et al. 2009; Lachat and Smith 2005), or intensive professional development and on-site coaching and other supports specifically designed to help teachers fully understand student data and adapt their instruction accordingly. In fact, the studies found that the interventions they examined had little or no effect on instructional practices (Cordray et al. 2012) or that DDI teachers were actually less likely to use differentiated instruction, an important DDI teaching practice (Williams et al. 2013; Konstantopoulos et al. 2013). Thus, the existing literature is limited in the information it can provide on the potential impact of a comprehensive DDI intervention. 3 This study would provide important, new experimental evidence on the impact of a comprehensive DDI program on student achievement. We will also gather valuable information on how DDI is implemented in schools. **b. Study Design and**  **Research Questions**  The study team will evaluate the effectiveness of the DDI program provided by Focus on Results using a random assignment design. Atotal of 104 schools from 12 school districts have
1 This information is based on conversations with representatives of nine large DDI providers: Achievement Network, Acuity, Curriculum Associates, Data Wise, Education for the Future, Northwest Evaluation Association, Public Consulting Group

-Focus on Results, TERC, and Wireless Generation. 2 There have been several nonexperimental examinations of DDI (Shaw and Wayman 2012; Henderson et al. 2007; Quint et al. 2008; Datnow et al. 2007), but because schools or teachers using DDI may systematically differ in other ways from schools or teachers not choosing to use DDI, these non-experimental results do not support rigorous causal inference. 3 In addition to the studies listed here, there is an ongoing evaluation of the Achievement Network DDI intervention. That intervention includes the administration of a particular set of interim assessments along with supports to participating schools to aide them in interpreting and using the assessment results. A report on this study is forthcoming in spring 2015. Therefore, at this time, the detailed characteristics of the evaluation design and the findings are not known.40166 OMB Supporting Statement: Part B Mathematica Policy Research

3 been recruited for the study. Within study districts, schools have been randomly assigned to either a treatment group that is implementing the study provided DDI program or a control group that is not receiving the intervention and therefore not implementing the key DDI treatment features during the study. While the study will examine outcomes for the 2015

-2016 school year, training and setting up the infrastructure to support DDI activities will occur during the winter and spring of 2014

-2015 in the treatment schools. These activities occur in advance of the 2015

-2016 school year because it may take some time to set up the necessary school structures and to train staff on data

-driven instruction procedures. Treatment schools will learn how to set up the structures and implement the procedures appropriate to data

-driven instruction through ongoing professional development and technical assistance. Beginning these supports prior to the 2015

-2016 school year will allow treatment schools time to gradually receive training and gain practice implementing key data

-driven instruction activities to then be ready for full DDI implementation during the 2015

-2016 school year. The study implementation team will support and monitor implementation both during the initial infrastructure building phase as well as the full implementation phase. The implementation and impact evaluation study will address the following research questions:
 What is the impact of DDI professional development on students’ achievement on state assessments in math and reading?  What is the impact of DDI professional development on teachers’ and principals’ access to and use of data to inform instructional support, planning, and practice?  How does DDI professional development affect teachers’ use of instructional strategies, such as differentiated instruction?  How do DDI

-related activities differ in treatment versus control schools? Because the primary research question focuses on the impact of DDI on student achievement, the study will examine impacts on grades in which state assessments are administered, grades four and five. Targeting fourth and fifth grades will also focus intervention resources more intensively on a subset of participating schools’ teachers, while allowing us to examine the effects of DDI on achievement across two different grade levels.

**c. The Data Driven Instruction P** **rogram Provided by Focus on Results**  The Focus on Results plan for DDI is based on a comprehensive framework for training and supporting school leadership and teachers on using multiple sources of student data (such as formative and summative student assessment data, student classroom work, student attendance and behavior data, etc.) to inform instruction and improve student achievement. The DDI program is designed to be implemented in schools with regular interim assessments and a data system that produces reports on student achievement. Key DDI components and how they are expected to contribute to achieving intermediate and final outcomes are depicted in Figure 1 . The overall logic of the DDI program is that a core set of training and technical assistance inputs inform and guide various activities by school leaders and individual teachers. These inputs and activities lead to teachers increasing their use of student data to guide instruction and altering their instructional strategies, which in turn leads to improved student achievement. Specifically, the instructional leadership team (described below) explores data at the school and individual grade levels to diagnose core issues for school

-wide focus. Teacher collaboration teams (also described below) and individual40166 OMB Supporting Statement: Part B Mathematica Policy Research

4 teachers explore data in their grade and classroom to generate ideas for instructional improvement of greatest benefit and consistent with the school

-wide focus. Teacher collaboration teams and individual teachers adjust instruction and continuously monitor student progress. The school

-wide instructional leadership team also monitors progress at the school level. Guided by this ongoing review of student progress, adjustments in instruction and focus take place as needed in a continual feedback/adjustment process to accomplish school level goals. **Figure 1: Logic Model for Planned DDI Intervention Tr** **aining and Te chnical**  **Assistance Inputs**  **School Structures and Activities**  **Intermediate Outcomes**  **Final Outcome** **s**

**Winter In**  **stitute** ( for principals and data coaches) Orientation, training, and resource s for DDI im plementation

**Instructional Leadership**  **Te am**  (principal, data coach, and grade/department chair s) Exam in e sch ool
-wi de data Se t instructional focus an d instructional

best practice s Se t school goals and monitor progress Commu nicate im portance of data
-driven

cu lture Use PD to t rain t eachers **Train ing**  **an d**  **Pr of** **es sional**  **Dev elopment**  **(P D)**  (for instructional leadership

team ) Ongoing PD se ssion s Sum me r refresher training **Improv ed**  **Studen t**  **Ach ievem** **ent i**  **n Engli**  **sh /**  **Languag e**  **Art s**  **and Ma** **thematics Te**  **acher Collaborati** **on Te**  **am s**  **( TCT** **)** De dicated me etings Structured activities to underst and st udent data Examine student work Share effective practice s Set and mon itor goals **Te achers**  Receive training and coaching **Improv ed**  **Access to**  **and Use of Dat**  **a** Acce ss to student data an d work f rom mu ltiple sou rce s Examine student data and student work Use data to se t and monit or goals

**On going** **TA** (for instructional leadership

team ) Monthly calls with coache s In - person visits Adhoc con tact s **Dat a**  **Co ach**  Strategizeswith principal An alyzesdata Reviews re se arch Co- leads in structional leadersh ip team

(with principal) Organizes teacher collaboration t eam s Facilitates TCT me etings Provides coaching Develops PD room De velops data walls **Pr incipa** **l** Commu nicates importance ofdata
-driven cu lture An alyzes student data Co- leads instructional le ad ership team Monito rs im plementation

**Con tex** **tu al**  **Factors** Dist rict policies Data coach experience Te ach ere xperience Principal e xperie nce Access to student data ( interiman d summat ive assessm ent s) **Imp ro** **ved Ins** **tr uct**  **ional Strategi** **es** Different iate instructio n Adjust student groups Increase instructional time on low
perform ing are as Inco rporate evidence
base d pract ice s
Participating schools assigned to the treatment will set up new structures to support the implementation of DDI. School leaders and individual teachers will engage in activities designed to help staff learn to analyze data from interim assessments and other sources to identify areas for improvement, jointly formulate improvement goals and priorities, implement evidence

-based changes in instructional practices, and monitor progress toward achieving the agreed

-upon goals. The following people and teams of people will be involved in the intervention in treatment schools:
 **Data coach.**  Participating schools will hire a half

-time data coach to lead and monitor the implementation of DDI activities on a day

-to-day basis. Trained and supported by Focus on Results consultants, the data coach will analyze student data, conduct strategic planning with the principal and other school leaders, and work directly with teachers

— individually and in groups (teacher collaboration teams). These activities will help teachers learn to use available data effectively to formulate instructional improvement goals, identify relevant best practices, and implement such practices. The coach will create and maintain a DDI resource40166 OMB Supporting Statement: Part B Mathematica Policy Research

5 room with materials to help school leaders and teachers analyze data and research possible instructional practices. The coach also will develop a set of data displays, available in the resource room and throughout the school, to help school leaders and teachers monitor student progress.
 **Principal.** As a school’s main instructional leader, the principal plays a critical role in establishing a data

-driven culture in the school and building support and momentum for DDI implementation. After receiving initial training, the school’s principal and data coach together plan for DDI implementation; they also meet weekly to review progress, troubleshoot issues encountered, and adjust plans for DDI implementation as needed. Principals work to support the initiative by emphasizing the targeted instructional focus and practices during regular classroom visits, follow

-up feedback to teachers, and interactions with the instructional leadership team and data coach.
 **Instructional Leadership**  **Team.** Each participating school will establish an instructional leadership team, whose primary role is to lead the school’s DDI implementation effort. The team will include the school’s principal, data coach, grade

-level chairs (and/or department chairs, if instruction is departmentalized), and other formal or informal school leaders (for example, assistant principal, other teachers, or IEP specialist). This team is charged with analyzing school

-wide data and, based on this analysis, identifying areas of instructional focus for school

-wide DDI improvement efforts (including specific instructional practices linked with these areas), and setting and monitoring goals for increased student achievement. Since DDI implementation relies on a train

-the

-trainer approach to build capacity within each school, this team also has the important responsibility to take what it learns from ongoing professional development sessions with Focus on Results and pass it on to teacher collaboration teams (described next), to guide their efforts. The instructional leadership team is also charged with addressing more concrete issues, such as ensuring that teacher collaboration teams have the time and space to meet on a regular basis.
 **Teacher Collaboration**  **Teams.** Teachers in each participating grade or department will form a professional learning community to work collaboratively, under the leadership of the data coach, to do much of the on-the

-ground work of DDI. Each team will hold regular meetings (at least every other week) to determine what the data (both formal interim assessments and informal assessments) suggest regarding student learning needs, collectively set grade/department

-specific goals and monitor progress toward goals, and identify instructional strategies that could better help them meet the needs of students scoring at the lowest levels.
 **Individual Teacher**  **Activiti es.**  Drawing on their work within the teacher collaboration teams, as well as one-on-one assistance from the data coach, each teacher will implement agreed

-upon instructional strategies in support of both the school

-wide and grade

-level improvement goals. Each teacher will also consider her students’ proficiency levels with respect to different standards (based on assessment data and student work) and make decisions about how best to address the students’ identified learning needs. After implementing the instructional strategies designed to meet these learning needs, teachers will assess the success of their efforts based on feedback from the coach as well as evidence from subsequent student work and assessment data.40166 OMB Supporting Statement: Part B Mathematica Policy Research

6 By collaborating on teams and participating in the activities described above, teachers are expected to develop the knowledge and skills needed to help them adapt and improve their instructional strategies based on student data. The relevant instructional strategies for a given teacher will depend on the specific learning needs of that teacher’s students, as indicated by the student data. DDI does not require that teachers implement any one specific instructional approach. Nonetheless, based on Focus on Results’ experiences and best

-practices ident ified by DDI experts (Hamilton et al. 2009), we expect teachers in treatment schools to be more likely than teachers in control schools to adopt certain strategies or use them more frequently. For example, DDI should lead to a greater use of differentiated instruction, more frequent review and adjustment of students’ small group assignments, and integrating evidence

-based practices into all aspects of their teaching. Schools selected to implement DDI for the study will receive resources, training, technical assistance, and ongoing support from consultants from Focus on Results to help them implement DDI. Capacity

-building inputs will include (1) an initial training for school principals and data coaches, (2) professional development training for instructional leadership teams, and (3) other technical assistance support for data coaches and instructional leadership teams during DDI implementation. The following training and technical assistance activities will take place in treatment schools:
 **Initial Traini** 

**ng.** A two-day orientation and training session ensures that data coaches and principals are prepared to begin implementing DDI in treatment schools. Activities include both general information about DDI and its implementation and specific training tailored to the activities each group will be expected to perform at their schools. Participants also are provided an opportunity to network and exchange ideas with data coaches and principals at other study schools implementing DDI.
 **Ongoing professional**  **developm ent**  **for instructional**  **leadership team**  **s.** Each school’s leadership team will receive a total of six professional development sessions to support the team in guiding their school’s adoption of DDI and provide training and skills that can be passed on to teachers at the school. Each session focuses on a particular set of activities related to the implementation of DDI, such as identifying an instructional focus, looking at student data and setting goals, using best practices in teachers’ classrooms, and monitor ing their progress. Members of these instructional leadership teams will then train teachers and other staff in their schools using the skills and knowledge they learn from the professional development sessions.
 **Other technical**  **assistance.** Instructional leadership teams will receive customized technical assistance to help address questions and needs that arise as schools implement data driven instruction. Each participating school will be visited a total of eight times by Focus on Results consultants to monitor DDI implementation and provide customized technical assistance. Technical assistance also will be delivered during monthly phone calls with the data coach, and additional phone calls or e

-mails, as needed. **d. Data Collection Plan**  To address the study research questions, the study team will collect and analyze data from teachers, principals, and district staff . A brief description of each data source and data collection activity is provided below.40166 OMB Supporting Statement: Part B Mathematica Policy Research

7  **Teacher and**  **Principal Assignment**  **Data :**  We will collect teacher assignment data from participating schools in winter 2016. We will ask schools to identify teachers currently teaching fourth and fifth grades, provide contact information for these teachers (such as school e-mail address), and indicate if the teacher taught at the school in fall 2014 (Appendix A). 4 The study team will attempt to identify principals at study schools from public sources (such as school websites). If that information is not publicly available, we will request schools to provide principal assignment information.  **Teacher Survey:**  We will administer a web

-based survey in spring 2016 to 500 teachers (Appendix B). Teachers will also have the option of completing the survey by hard copy or by phone, if they prefer. The survey will provide information on professional development and training received by teachers (particularly related to DDI topics), school data culture, instructional planning and collaboration, teachers’ access and use of data to guide instruction, teachers’ instructional strategies, and teachers’ demographic and background characteristics.  **Teacher Logs:**  We will administer web

-based logs with teachers during the 2015

-2016 school year (Appendix C). The same set of 500 teachers who will be asked to complete the teacher survey will also be asked to complete these logs at two points during the year, and in each case report on their activities over two consecutive days. The logs will provide information on teachers’ day

-to-day activities, including planning activit ies (individually and in collaboration with other teachers) and instructional strategies in the classroom.  **Principal Survey:**  We will administer a hard

-copy survey to all 104 study principals in spring 2016 (Appendix D). The survey will focus on topics such as school

-wide leadership activities, school data culture, school

-wide access to and use of data, and principals’ demographic and background characteristics. Principals will have the option to provide answers to a trained interviewer over the phone.  **Student Records**  **Collection:** In summer 2016, we will collect student outcome measures (for example, math and reading test scores from state assessments) for the year of full implementation (2015

–2016) as well as for two prior years (2014

–2015 and 2013

-2014 if available). We will also collect student demographic and background data, such as gender, race/ethnicity, and eligibility for free and reduced

-price lunch, and student attendance and disciplinary measures (if available), and student enrollment data for fall 2014 and winter 2016 (Appendix E) . **e. Study Activities and Time Line**  In Table 1 , we show the timing of the major study activities. **Table 1. Schedule of Major Study Activities for Data -**  **Driven Instruction Evaluation** Spring 2014 Summer

2014 Fall

2014 Dec

2014 Spring

2015 Fall/ Winter 2015 Winter/ Spring 2016 Summer/

Fall 2016 Summer

2017 Re cruit Districts X X X Conduct Random Assignment X X X

4 Teacher assignment data will be collected following approval of the OMB clearance package.40166 OMB Supporting Statement: Part B Mathematica Policy Research

8 Training and Infrastructure Setup X X Full Implementation X X Collect Teacher Assignment Data* X Conduct Teacher Survey X Conduct Teacher Logs X X Conduct Principal Survey X Collect Student Records Data X Prepare Report X *Schools will be asked to provide principal assignment data only if the information cannot be obtained from publicly available sources. **2. Purposes and Uses of Data**  The main purpose of this evaluation is to estimate the impacts of DDI on student achievement and other outcomes, as well as to document implementation of DDI programs and activities. In Table 2, we list the study’s research questions and the data collection to support the answers. **Table 2. Research Questions and Data Collection Method** Re search Questions Data Collection Method 1. What is the impact of DDI professional development on students’ achievement on state assessments in math and reading?  Student records data 2. What is the impact of DDI professional development on teachers’ and principals’ access to and use of data to inform instructional support, planning, and practice?  Teacher survey  Teacher logs  Principal survey 3. How does DDI professional development affect teachers’ use of instructional strategies,such as differentiated inst ruction?  Teacher survey  Teacher logs 4. How do DDI -related activities differ in treatment versus control schools?  Teacher survey  Principal survey  Technical assistance documents

 **Teacher and**  **Principal Assignment**  **Data:** Teacher assignment data will be used to randomly select approximately 500 fourth and fifth grade teachers to complete the teacher survey and logs. Principal assignment data will be used to identify the principals in the 104 study schools to determine the sample for the principal survey. Information on teacher and principal assignments in fall 2014 will be used to examine whether the DDI intervention lead to a treatment-control difference in educators’ mobility between fall 2014 (following random assignment) and the 2015

-2016 school year.  **Teacher Survey and Logs** **:** Teacher survey and log data will be used to analyze teacher intermediate outcomes and to measure important aspects of the treatment-control contrast. Intermediate outcomes that will be examined include the impact of DDI on teachers’ access to and use of data to guide instruction and teachers’ instructional strategies (such as the use of differentiated instruction ). We will also examine40166 OMB Supporting Statement: Part B Mathematica Policy Research

9 treatment-control differences in the training teachers receive and other activities surrounding the use of data in schools.  **Principal Survey** **:** The principal survey will enable a better understanding of the treatment-control contrast in schools’ training and activities surrounding the use of data, as well as shed light on some aspects of DDI implementation. It will examine topics such as whether principals have established school

-wide goals, their expectations for school

-wide data use, whether the school has an instructional leadership team and the activities it undertakes, and the degree to which school leaders review and discuss student data.  **Student Records Collection:** We will use existing district test score data to estimate the impact of DDI on student achievement, the key outcome of interest. Information on students’ demographic and socioeconomic characteristics and their achievement test scores prior to the study school year will be used to describe the students in the study and to develop more precise impact estimates. In the process of implementing the intervention and providing technical assistance to treatment schools, we will also obtain information on aspects of implementation from the evaluation team’s technical assistance provider (Focus on Results), such as materials from their professional development sessions. We will present the study findings in a report that will include information from both the implementation and impact analyses. The data collected by this evaluation will also be available as restricted

-use files, serving as a valuable resource for researchers.
**PART B: COLLE** **CTION OF INFORM ATION EMPLOYING STAT** **ISTICAL METHODS**  **1. Respondent Universe and Sampling Methods**  This study will not statistically sample districts and schools. It will, instead, rely on a convenience sample of 104 schools within 12 districts that volunteered to participate in the evaluation and meet the study requirements. The evaluation does not aim to make statements that generalize beyond the districts and schools under study. **Number of**  **Districts and**  **Schools .**  To be able to detect the impact of DDI on student achievement at the level requested by ED, we will include 104 schools that include grades 4 and 5 in the evaluation (see Table 3 below). Each district will include an average of 8 to 9 schools in the evaluation; overall, 12 districts will take part in the evaluation. **Randomly Assigning**  **Schools.** Our random assignment approach involved identify ing strata—or matched pairs

—of similar schools within districts and then randomly assigning schools within each stratum to the treatment or control group. The matched pairs of schools were formed prior to random assignment, and each stratum was primarily based on average student achievement in prior years, the characteristic most likely to be predictive of student outcomes during the study period. Other characteristics we considered in forming strata include: (1) the proportion of students receiving free or reduced

-price meals, (2) the racial/ethnic distribution of students, (3) the proportion of students classified as English language learners, (4) Title I eligibility, and (5) school40166 OMB Supporting Statement: Part B Mathematica Policy Research

10 size. We obtained information on these school characteristics from the Common Core of Data and state websites. Random assignment within strata will increase statistical precision by reducing random differences in the average baseline characteristics of treatment and control schools (Imai et al. 2009) . **Identifying and**  **Recruiting Districts**  **and Schools** **.** The districts and schools included in this study were not randomly sampled and so are not statistically representative of the broader group of all public schools serving fourth and fifth graders nationally. Participating districts and schools had to be both interested in and eligible to participate in the study. Interested districts had to be comfortable with implementing DDI; allowing the intervention to be randomly assigned to its participating schools; and willing to comply with data collection activities, including student data collection, a teacher survey, teacher logs, and a principal survey. 5 Eligible districts also met the following criteria:
 Summative and interim assessments. To effectively implement DDI, teachers and administrators in participating schools needed to monitor student achievement on a regular basis using student data from both summative assessments (cumulative end

-of

-year assessments) and interim assessments (periodic assessments intended to evaluate student learning progress during the year). Teachers in study schools will rely upon existing summative and interim assessments used within each district in order to monitor student achievement; the evaluation will not implement study

-specific assessments within participating schools. Both summative and interim assessments used by each participating district needed to be uniformly administered in 2014

-2015 and 2015

-2016. Interim assessments had to be highly aligned to the state summative assessment and, ideally, made by the same developer as the summative assessment. 6 Thus, recruitment targeted districts that used summative and interim assessments meeting both of these criteria (including, but not limited to, states that were part of the Smarter Balanced Assessment Consortium).  Districts serving high needs students. Recruitment efforts aimed to include districts with a large number of high needs schools that would most benefit from DDI, and thus targeted districts with a relatively high proportion of free or reduced price meal (FRP) students. Recruitment targeted districts serving disadvantaged student populations, as indicated by having rates of FRP eligibility higher than 40 percent; within those districts, we targeted the higher poverty schools.  District size. In order to meet our targeted sample size, the study recruited districts with at least eight elementary schools that contained fourth and fifth grades. The study targeted elementary schools for two reasons. Districts typically have a larger number of elementary schools than middle schools, so identifying and recruiting a sufficient number of elementary schools would potentially be easier. In addition, elementary schools are less likely than middle schools to be departmentalized, and tracking by ability level is also less common at the elementary school level. Thus, the intervention will be able to work with groups of

5 Originally all states were considered for inclusion in the study. However, during the recruitment process, we decided to exclude states that planned to use the PARCC assessments because it was unclear when those assessments would be available. One large state was also excluded due to concerns about the ability to obtain student achievement data without parental consent. 6 This criterion resulted in excluding states that developed their own summative assessments.40166 OMB Supporting Statement: Part B Mathematica Policy Research

11 teachers who are with their students all day and perhaps have more flexibility in refining their instructional practices.  No District

-wide DDI Initiatives. For the purposes of ensuring a sufficient contrast between treatment and control schools, the study targeted districts that had made minimal (or no) efforts to implement data

-driven instruction. We avoided districts that had implemented district

-wide DDI initiatives, as well as districts that planned to implement new DDI programs during the 2014

-15 or 2015

-16 school years. Similarly, the study avoided districts that had implemented district

-wide Response to Intervention (RtI) programs, given that RtI programs utilize a similar data

-driven approach. Eligible districts also expressed a willingness to cooperate with data collection activities and agreed to allow their schools to be randomly assigned to treatment or control conditions. Once we identified districts that met the criteria above, we identified a subset of eligible schools within those districts, based on the following eligibility criteria:  High needs schools. Within each district, recruitment efforts aimed to include high needs schools, and thus targeted schools with a proportion of free or reduced price meal (FRP) students that was high relative to other schools in the district.  No school history of orplans to implement DDI. Within each district, the study avoided schools that had a data coach in place, received regular professional development focused on data use, or held regular teacher collaboration group meetings focused on analyzing and making use of student data. We also excluded schools that planned to implement new DDI programs during the 2014

-15 or 2015

-16 school years and schools that had implemented RtI programs.  No Schools with School Improvement Grants (SIG). Schools that had received School Improvement Grants (SIG) were not eligible for the study, given that DDI was an important part of required SIG activities under some of the intervention models specified by ED.  Title I schools. The U.S. Department of Education has expressed a strong interest in studying effective approaches to help low

-income children within Title I schools meet state standards for academic achievement. A comprehensive approach to DDI is consistent with this interest. While the study preferred to include Title I schools, this was not a strict requirement for participating; as a result, some eligible schools did not have Title I status.  Excluding charter and magnet schools. Charter schools, which often utilize data

-driven approaches to instruction and that typically operate autonomously from other district schools, were excluded from the study. Magnet programs were included only when the district provided an even number and similar type of magnet schools. 7 Eligible schools also expressed a willingness to cooperate with data collection activities. Eligible schools agreed to provide the study with student

-level assessment and demographic data (if not provided by the district). In addition, their principals expressed willingness to complete a survey in spring 2016, and the schools agreed to encourage their teachers to complete a teacher survey in spring 2016.
7 Among the 104 schools included in the study, 11 schools were magnet programs. Districts contributed magnet schools in pairs that could be matched to one another, with one exception

—a case where a magnet school was matched to a non-magnet school that the district reported had similar programming and structures in place as the magnet school.40166 OMB Supporting Statement: Part B Mathematica Policy Research

12 A key goal of recruitment for the evaluation of DDI was, thus, to recruit a sufficient number of schools that both met the above eligibility requirements and that were interested in the study. As described below, the study estimated that 100 schools would be needed to generate sufficient statistical power to detect meaningful differences in student achievement between the treatment and control groups. Assuming that each district would be able to provide 8 to 9 schools, the study aimed to recruit 12 districts. During the recruitment process, we first used data from the Common Core of Data and other publicly available sources to identify districts meeting a basic set of eligibility criteria (including their plans for summative and interim assessments, percent FRP, district size, SIG status and Title I status). While we could not be entirely certain about state assessment plans in future years, we collected detailed information from state contacts and public sources to help us identify states that were fairly confident about their assessments plans in 2014

-2015 and 2015

-2016. Our team identified an initial pool of 337 districts that met these criteria, and began recruiting a subset of these districts. Out of the initial pool of 337 eligible districts, a total of 103 districts were targeted for recruitment and, among those targeted, 12 districts were included in the final sample. 8 Within the 12 study districts, 104 schools agreed to participate in the study. Districts contributed a total of 4 to 14 schools each, depending upon a number of factors, including the size of the district, overall eligibility and suitability of selected schools, and the remaining number of schools needed to reach the overall recruitment target. **2. Statistical Methods for Sample Selection and Degree of Accuracy Needed**  **a. Statistical Methods for Sample Selection**  As described in Section 1, the study is using a convenience sample of districts and schools that volunteered to participate in the evaluation and met the study requirements. To meet the study’s target effect size, we included 12 districts and 104 schools. We will conduct surveys with all 104 principals, and we will collect administrative records on all students in tested grades within the participating schools. Thus, the study will not statistically sample principals, schools, or students. We will sample teachers for the teacher survey as explained below. We will administer a web

-based survey to 500 teachers in the target grade levels in the spring of the 201 5

–201 6 school year, and administer web

-based logs in the winter and spring of that school year (Appendices B and C) . The teacher sample will consist of five teachers from each study school, on average, randomly selected from the pool of all fourth and fifth grade teachers in the study schools. Selecting only teachers who teach students in tested subjects and grades 4 and 5 has the advantage that it aligns the information we learn from teachers with the student achievement

8 Among the 91 districts that did not end up in the final sample, approximately half (44 districts) were dropped because they lacked interest in the study, and half (47 districts) were dropped because they did not meet all of the eligibility criteria. The most common reason that districts were found to be ineligible was because they did not meet study criteria for the use of summative and interim assessments. Other districts were dropped because they already had an existing DDI program in place or did not have a sufficient number of schools that met eligibility criteria.40166 OMB Supporting Statement: Part B Mathematica Policy Research

13 analyses. 9 The disadvantage is that we will not have as a complete picture of DDI implementation in the school as we would if we also included all teachers in the participating grades. We plan to identify teachers by collecting teacher assignment data from the study schools. We will use the teacher assignment data to identify fourth and fifth grade teachers in treatment and control schools. We will stratify teachers by grade and/or subject in order to obtain a similar number of teachers by grade and subject from treatment and control schools. From each stratum, we will randomly select teachers to receive the survey, for a total of 500 teachers. **b. Esti mation Procedures**  Random assignment of schools within districts to a treatment group that will implement the study provided DDI program or to a control group that will not is an ideal design for assessing overall effectiveness of DDI. Our primary impact analysis will exploit this experimental design to provide rigorous estimates of the impact of DDI on student achievement and other student outcomes. Additional experimental analyses are designed to estimate the impact of DDI on teacher outcomes, such as teachers’ access to and use of student data and teachers’ use of instructional strategies. **Implementation Analysis** . The implementation analyses have two main objectives. The first objective is to describe the DDI activities implemented in treatment schools and the extent to which they were implemented with fidelity. The second objective is to describe differences in treatment and control schools’ implementation of DDI

-related activities. The analyses will rely upon information from the data coach weekly activity logs and the Focus on Results consultant activity logs, as well as from the principal survey, teacher survey, and teacher logs. Fidelity of DDI implementation. The implementation analysis will describe DDI activities undertaken by treatment schools and the extent to which they were implemented with fidelity to the intervention plan. As a part of this analysis, we will review log entries from January 2015 through spring 2016. Because it may take time for treatment schools to set up necessary school structures and roll out intervention activities, we expect some variance in implementation at different points in time during the intervention. Thus, we will examine fidelity of implementation at three points: (1) at the end of the 2014

-2015 school year, (2) in December 2015, and (3) at the end of the 2015

-2016 school year. To assess the extent to which treatment schools implemented DDI with fidelity, it will be important to quantify the data and summarize it unif ormly. The DDI intervention aims to build school capacity in two ways, by (1) helping schools set up structures and activities that enable school staff to carry out DDI, and (2) training and coaching teachers in the skills needed to use and interpret student data. Implementation fidelity measures will therefore examine the training and coaching that treatment schools receive and the degree to which treatment schools fully implement the anticipated DDI structures and activities. In describing the fidelity of implementation of a particular DDI component, we will focus on the activities we expect to have happened if the component has been implemented. We will measure both whether the activity occurred and, if appropriate, the level of participation in the activities. To

9 Although we will collect student test scores on all students in tested grades in the study schools, our primary impact analyses will focus on fourth and fifth graders.40166 OMB Supporting Statement: Part B Mathematica Policy Research

14 measure the fidelity of implementation of the instructional leadership team meetings, for example, we will measure whether a school holds monthly instructional leadership team meetings among all key staff, attendance at these meetings among the staff, and the key activities taking place at these meetings. These activities would include that the team establishes school

-wide goals for performance, establishes areas of instructional focus, and examines student data. Our examination of implementation fidelity will be accompanied by descriptive text that summarizes our findings, describes changes in implementation over the course of the intervention, and provides a range of examples of how schools implemented DDI with fidelity. We will also describe treatment schools’ experiences in implementing DDI. This information will be especially important if the DDI approach is found to be effective and other districts wish to replicate the program. We will examine school

-level factors that may influence DDI implementation, such as the level of engagement of the school principal. Implementation of DDI

-related activities in treatment and control schools. The DDI intervention is expected to lead the principal, data coach, instructional leadership team, teacher collaboration teams, and individual teachers in treatment schools to engage in numerous data

-focused activities. These include (1) activities undertaken by school leaders in directing data use, (2) school

-wide communications about data use, (3) professional development and support for the principal and teachers on data use, and (4) collaboration among teachers to review student data and share instructional strategies. We will use data from the principal survey, teacher survey, and teacher logs to examine whether implementation of the DDI intervention leads to a treatment

-control difference in these activities (Table 1) . **Table 1**  **. Data** **- related Activities, by Source**  Principal Survey Teacher Survey Teacher Logs Activities undertaken by school leaders in directing data use Frequency of leadership team meetings X Leadership team m embers X Leadership team activities (e.g., setting achievement and priority learning goals , monitoring progress, planning professional development activities) X Degree to which school leaders ensure that teachers have the time and resources needed to analyze and interpret student data X X School-wide communications about data Frequency of communication on student achievement goals and results X X Frequency of communication on priority learning goals X X Frequency of communication on expectations for and actual use of data X X Use of data displays in classrooms and other public areas X X Professional development and support for the principal, teachers around data use Amount of professional development/training activities this school year X X Topics covered by trainings (analyzing student data, establishing priority learning goals for the school , individualizing student learning goals, tracki ng progress toward goals , using evidenceX X40166 OMB Supporting Statement: Part B Mathematica Policy Research

15 based instructional strategies) Availability of on-site coaching/support for data use X X Data coach activities X Frequency of, and topics addressed during, individual coaching X X Frequency of classroom observations and feedback X X Collaboration among teachers around data use and sharing instructional strategies Frequency of, and amount of time spent on, teacher collaboration X X Teacher collaboration activities (analyzing student data , setting common learning goals for students, sharing effective instructional practices , jointly modifying lesson plans, monitoring im plementation and results of instructional changes) X X **Impact analysis** **.** The impact analysis will rigorously assess the effectiveness of a comprehensive DDI program. Calculating the statistical significance of the impacts requires that the nested structure of the data

— with students clustered in schools

– is incorporated in the analysis. Due to clustering, the variance of the impact estimates is larger than it would have been if each individual student were randomly assigned to DDI. Below we describe our approach to calculating outcomes of student achievement and intermediate outcomes on teacher and principal practices. Studentfinal outcomes. Student achievement will be the study’s primary outcome, measured using spring 2016 math and reading scores on state standardized tests. Specifically, this main impact analysis will examine the effect of DDI on:
 Math achievement among fourth and fifth graders in a school that implemented data

driven instruction
 ELA achievement among fourth and fifth graders in a school that implemented data

driven instruction To assess the impact of data

-driven instruction on student achievement, we will use a place

based impact estimation strategy that compares outcomes for students in treatment schools to those of students in control schools using spring 2016 state test scores. This implies that the impacts could reflect either the impacts of data

-driven instruction on student achievement or impacts on student mobility. Given the nature of the intervention, we do not expect that the data

-driven instruction intervention would affect students’ mobility patterns during the study period. However, we will test this empirically through the analysis of impacts on student mobility based on the student sample in fall 2014 (following randomization) and spring 2016 (at the conclusion of the intervention). The main impact analysis based upon spring 2016 test scores will estimate an impact model in which the overall impact estimate is based on treatment-control differences in the outcome of interest within each stratum (matched school pair) . Although a simple treatment-control difference in mean outcomes will yield an unbiased estimate of the impact of DDI, the precision of estimates can be improved by controlling for baseline characteristics that may influence the outcomes of interest but are not related to the treatment itself. For all the student outcomes, we will control for baseline student and school covariates. If available from state or district records, specific student

level covariates will include prior years’ (spring 2015 and, if available, spring 2014 ) math and reading test scores (in z-score units) , prior years’ student attendance, prior years’ student suspensions,40166 OMB Supporting Statement: Part B Mathematica Policy Research

16 gender, race/ethnicity, eligibility for free or reduced

-price lunch, English language learner status, and special education status. 10 We also anticipate including school

-level aggregates of these variables and other relevant school characteristics in cases where individual student

-level data are not available. Accordingly, we will estimate student impacts using the following model: (1) = + + + + where y

ijk is the outcome of individual student i in schoolj within stratum k; αk is a vector of stratum (matched pair) indicators (fixed effects) included to control for differences across strata in average student, teacher, principal and school characteristics; T jk is a treatment indicator that equals one if the school was assigned to DDI and zero otherwise; X ijk is a vector of baseline individual student characteristics; u jk is a school

-specific random error term; ε ijk is an individual

-level random error term; and 

, γ, and δ are parameters to be estimated. The estimate of  represents the overall impact of DDI on the student outcome of interest. We will estimate the model with ordinary least squares (OLS) using standard errors that account for school

-level clustering. We will consider student achievement in reading and math to be separate domains, across which the impact of data

-driven instruction might differ. Given that assessments differ across state, grade level, and subject area, we will standardize the raw achievement scale scores by converting them to z-scores. We will calculate the z-scores by subtracting the state mean score from the raw scale score and dividing by the standard dev iation of the state scores. We will also estimate impacts on other student

-level outcomes, such as attendance and suspensions, if possible. We will request student

-level data on these outcomes from districts or states, but if the data are not available or are unreliable, we will estimate impacts on these outcomes measured at the school (or, if available, grade) level. Impacts on a school

-level outcome y jkd can be estimated using model (1) above but including school

-level averages of individual

-level covariates rather than the individual

-level covariates themselves. The estimation of overall impacts on student achievement may mask differences in impacts across subgroups of students. For example, DDI may prove more or less effective at boosting student achievement for students with different baseline characteristics; it may, for example, raise achievement among lower performing students to a greater degree than it does among higher

10 Since the intervention begins in December 2014, students’ spring 2015 will not be measured during a true baseline period. However, we argue that it is unlikely that the intervention will have an effect on sample members’ spring 2015 scores, for two reasons. First, sample members in fourth grade during 2015

–2016 will typically have been in third grade in spring 2015, and the intervention does not cover this grade level. Second, intervention activities during the first several months of the intervention period (that is, through the early months of 2015) will focus on providing support to schools in setting up the structures necessary to implement comprehensive DDI and on providing training to teachers in the overall intervention approach and in the initial steps of understanding student data. Thus, we argue that these early activities would be unlikely to lead to instructional changes that affect the spring 2015 achievement levels of the study’s fifth grade cohort. However, we will address this possibility by examining the sensitivity of the results to a model that uses spring 2014 achievement levels for this cohort, as well as to a model that excludes baseline achievement altogether. We will also estimate a model in which the spring 2015 test score is the dependent variable. If our premise that the intervention was not sufficiently implemented to affect student achievement by this time, we would expect to see no significant treatment

-control difference in spring 2015 test scores.40166 OMB Supporting Statement: Part B Mathematica Policy Research

17 performing students. We will provide a subgroup analysis of the impacts of DDI for student groups based on their level of baseline achievement, focusing on impacts for particularly low

-achieving students as well as for students at moderate to high baseline achievement levels. To estimate impacts for student subgroups, we will create a version of the model that interacts a subgroup indicator with the treatment indicator; the coefficient on the subgroup

-treatment indicator interaction term will represent the impact estimate for the subgroup. Teacher andprincipal intermediate outcomes. The core set of professional development and technical assistance inputs under DDI are intended to help teachers and principals use data to improve instruction, which in turn would lead students to realize higher achievement gains. We will use responses from the teacher survey, teacher logs, and the principal survey to examine these intermediate outcomes. Teachers and principals offer different vantage points from which to assess the extent to which schools engage in data use activities. For example, principals may be able to provide detailed information on school leadership activities, while teachers may be able to provide detailed information on the frequency and content of teacher collaboration activities. The surveys will provide useful information on the frequency of activities over an extended period of time (such as how often a teacher attended professional development during the school year), while teacher logs will provide a one-day snapshot of teacher activities that occur relatively frequently (such as lesson planning and collaboration based on analysis of data). Table 2 lists the measures and their sources that will be used to estimate impacts on intermediate outcomes. The first two sets of items listed in the table measure intermediate outcomes related to teachers’ access to and use of data to guide instruction. The second two sets of items capture information on teachers’ instructional strategies. **Table 2**  **. Teacher and Principal Intermediate Outcome Measures, by Source**  Teacher Survey Teacher Logs Principal Survey **Ac cess to student** **- level**  **data** Access to interim assessment results X X Access to summative assessment results X X Access to student background characteristics, attendance, and school behavior information X X Barriers to data use (usable format, technology, tools) X X **Use of s** **tudent -** **level d** **ata** Frequency of use by type (summative assessment results, interim assessments, formative assessments, samples of student work, student characteristics) X X X Purposes of data use (understand student needs, set learning goals, monitor progress toward goals, differentiate instruction, revise lesson plans) X X Understanding of instructional changes to make based on data X X **Differentiated Instruction** Placing students in small groups based on student data X Providing small group instruction X X Providing individualized instruction X X Identifying and referring students in need of pull-out services or other intensive interventions X X Changing instructional group assignments of students based on student data X X40166 OMB Supporting Statement: Part B Mathematica Policy Research

18 **Whole -** **Class Instruction** Providing additional instruction in areas where students are struggling X X I dentifying evidence-based instructional changes to help address students’ needs X Using new instructional strategies to teach challenging concepts to students X To assess the impact of DDI on teachers’ use of data and instructional strategies, we will compare outcomes for teachers in treatment schools to those of teachers in control schools. Our impact model for teacher outcomes will take a similar approach to that taken in the student outcomes model, calculating treatment-control differences among teachers. For all teacher outcomes, we will control for teacher covariates derived from the survey. Teacher covariates will include years of teaching experience, gender, race/ethnicity, teacher certification, and an indicator for a master’s degree. Similar to our approach in the student model, we will estimate the model with ordinary least squares (OLS) using standard errors that account for school

-level clustering, and we will compute the overall average impact of DDI by taking a weighted average of the coefficients on the treatment indicators. Each district

-specific impact will be weighted by the number of study schools in each district. Different types of teachers may be differentially affected by the implementation of DDI. For example, less experienced teachers may be more (or less) at ease in using data to inform instruction than more experienced teachers. They also may be able to benefit to a greater extent from the information on student performance provided by student data, since they will be less able to rely on experience to understand their students’ needs. The study will also examine impacts on teacher outcomes separately for subgroups of teachers defined by their level of experience. Similar to the approach used in the student subgroup analysis, we will estimate subgroup impacts by creating a version of the model that interacts a subgroup indicator with the treatment indicator. Our approach to analyzing principal outcomes will similarly compare principal survey responses regarding the structures and professional development activities in treatment and control schools. However, while the teacher and student models will be estimated at the classroom and student levels, respectively, the principal model will be estimated at the school level. We will adapt the student

-level model presented in equation (1) to estimate impacts on principal

-reported measures. For all principal survey analyses, we will control for principal covariates, such as years of experience, gender, race/ethnicity, and an indicator for a master’s degree. **Nonexperimental analysis .**  Contextual factors, such as student characteristics or principal and teacher background characteristics, may aid in the interpretation of the impact of DDI in treatment schools relative to control schools. We will therefore examine how school contextual factors are related to impacts. Examples of contextual factors include:
 **School characteristics.**  School math and ELA proficiency measured at baseline, the percentage of students eligible for free or reduced

-price meals, the percentage that are English language learners, the percentage in special education programs, and racial/ethnic composition of the school.40166 OMB Supporting Statement: Part B Mathematica Policy Research

19  **Teacher and**  **principal characteristics** . Education, experience, and background characteristics. The study will use descriptive analyses and regression analyses to examine how impacts are related to school contextual factors. We will use the descriptive analysis to identify conditions and practices that are candidates for regression analyses of impacts. **Correlational analyses**  **of impacts**  **and implementation**  **fidelity .**  The potential impact of DDI on student achievement may differ within treatment schools based on aspects of their fidelity of implementation of DDI. To explore how school characteristics and other contextual factors may influence student impacts, we will conduct a correlational analysis that examines the relationship between estimated impacts of DDI and key features of treatment schools (or of matched pairs of treatment and control schools when data are available for the control schools). For example, we may examine whether the qualifications of the data coaches hired at each treatment school are related to impacts on student achievement. Because data coaches play a central role in supporting the DDI intervention at each school, the prior skills and knowledge they bring to the work may influence the degree to which teachers are supported in analyzing student data and using them to improve instruction, which may in turn influence student outcomes. Understanding the degree to which the success of the DDI intervention hinges upon the qualifications of the data coach may prove helpful in developing effective DDI interventions in the future. At the time they were interviewed for the position in fall 2014, each data coach completed a data proficiency exercise developed by Focus on Results to assess their initial skills in analyzing student assessment data. As above, we will correlate the data proficiency score of each coach with the estimated impact of the coach’s school. Because schools cannot be randomly assigned to specific DDI contexts, this correlational analysis will be nonexperimental. We will stress that any significant relationships between impacts and DDI features or contextual factors might not be causal and might reflect the influence of other unobserved factors. **Degree of**  **Accuracy Needed** **.** In Table 3 we present the minimum detectable effect sizes (MDEs) on student achievement and teacher practices for the sample size of 104 schools (52 treatment and 52 control), 5 teachers per school, and 112 students in grades 4 and 5 per school. The MDEs incorporate conservative assumptions about design effects due to the clustering of students or teachers in schools and precision gains from regression adjustments and stratified random assignment. These assumptions (listed in the notes of Table 3) are based on estimates from recent large

-scale studies in education with school

-level random assignment. Under these assumptions, we will be able to detect an impact on student achievement of 0.1 2 standard deviations. With this sample size, we also will be able to detect an impact on teacher practices of 0.33 standard deviations. Minimum detectable effect sizes for student and school subgroups are also shown in the table. **Table 3**  **. M inimum Detectable E** **ffects by Sample Size**  Sample Size Minimum Detectable Effect (MDE) Number of Schools Number of

Students per School Number of

Teachers per School Number of

Principals per School Student

Outcome Teacher

Outcomes Principal

Outcomes **Balanced**40166 OMB Supporting Statement: Part B Mathematica Policy Research

20 **Design** 104 (52 T, 52 C) 112 5 1 0.12 0.33 0.59 **Student/teacher subgroup**  104 (52 T, 52 C) 56 3 1 0.14 0.39 -- **School Subgroup**  52 (26 T, 26 C) 112 5 1 0.18 0.48 0.90 Note: The MDEs were calculated assuming: (1) a stratified random assignment design; (2) a twotailed test; (3) a 5 percent significance leve l and 80 percent level of power; (4) a school
- level intraclass correlation of 0.15; (5) a response rate to the teacher survey of 85 pe rcent; and (6) a reduction in variance of 40 percent at th e student level , 70 percent at the school level from the inclusion of covariates in the student outcome models and 10 percent reduction in variance at the teacher and principal level s, and 10 percent at the school level from the inclusion of covariates in the teacher outcome models. T= treatment; C= control. **3. Methods to Maximize Response Rates and Deal with Nonresponse**  There are multiple strategies to maximize response while minimizing burden on respondents and the following techniques are major contributions to a high completion rate: establishing positive relationships with respondents and school and district staff; sending letters prior to the surveys; and establishing efficient and flexible scheduling. We will include a statement on confidentiality and data collection requirements (Education Sciences Reform Act of 2002, Title I, Part E, Section 183) in all letters, data collection instruments, and study information sheets. In cases when the data collection activity is voluntary (the teacher survey, teacher logs, and principal surveys), we will include a statement indicating that participation is voluntary, yet emphasize the importance of their response for the study findings and highlight the incentive for completion. **Teacher and**  **Principal Assignment**  **Data.** We will collect teacher assignment information using a form that requires minimal effort to complete (Appendix A ). The study team will attempt to identify principals at study schools from public sources (such as school websites). If that information is not publicly available, we will request schools to provide principal assignment information. Schools can choose to provide this information by posting an electronic file to our secure website or by sending a hard copy via a prepaid Federal Express packet provided. We will work with school staff if they prefer to provide the information by another method (such as by phone). We will be courteous but persistent in our follow

-up with participants who do not respond quickly to our attempts to reach them. Based on our experience colleting this type of information, we expect a 100 percent response rate. **Teacher Survey**  **and Teacher**  **Logs .**  Based on Mathematica’s experience survey ing teachers on other studies, we expect at least an 85 percent response rate for the teacher survey, and an 80 percent response rate for the teacher logs. To ensure a high response rate, we will send teachers a letter that will describe the study and provide instructions to complete the survey online at their convenience (Appendix B). In the case of the teacher logs, we will request that teachers complete them within a more limited time period so that they will be able to accurately recall the day’s activities. We will send out reminder emails and make reminder telephone calls to teachers who do not respond within two to three weeks of recei ving the survey. For the logs, we will send the reminder emails and make the reminder calls during the shorter time period in which we hope to get a response. If necessary, Mathematica staff will follow up with nonrespondents and administer the survey over the telephone at the teachers’ convenience, or provide teachers with a hard

-copy survey40166 OMB Supporting Statement: Part B Mathematica Policy Research

21 they can complete. Experienced interviewers will be recruited and extensively trained on data collection procedures, including methods for promoting cooperation among school staff. Interviewers especially skilled at encouraging cooperation will be available to persuade reluctant teachers to participate. To compensate teachers for their time to complete the survey and to increase the response rate, we propose offering teachers $30 for each completed survey. For each completed log, we propose offering teachers $15. **Principal Survey** . To ensure a high response rate to the principal survey (Appendix D) , we will draw on our collegial relationship established with principals during the recruitment effort. We will provide reminder materials about the study that explain the importance of the survey as an opportunity for principals to provide their perspective on DDI practices in their school. As with the teacher survey, we will send out reminder emails and make reminder calls to principals who do not respond within two to three weeks of receiving the survey. Principals will also be given the option to complete the survey by telephone, if desired. To compensate principals for their time to complete the survey and to increase the response rate, we also propose offer ing principals $30 for each completed survey. **Student Records**  **Data.** To minimize burden on the district and maximize the likelihood of obtaining the data, during the initial phases of recruiting we asked each district how administrative records data are stored, how we can obtain permission for collecting this information, and the contact person we should work with to obtain the data (Appendix E). We will accept electronic data file or hard copy lists. Federal rules permit ED and its designated agents to collect student demographic and existing achievement data from schools and districts without prior parental or student consent (Family Educational and Rights and Privacy Act (20 U.S.C. 1232g; 34 CFR Part 99)). To maximize the response rate and minimize burden on schools and parents, we will follow these federal rules, and we plan to compensate districts for the burden of their time spent providing student records. In addition, we will be courteous but persistent in our follow

-up with participants who do not respond quickly to our attempts to reach them. We assume that we will be able to obtain records for 100 percent of the participating students. **4. Pilot Testing**  We will pilot the survey instruments and logs for features such as clarity, accuracy, length, flow, and wording. Trained quality control staff will check responses for completeness and reasonableness as soon as hard

-copy principal surveys are received and they will follow up with respondents if problems are identified. The web

-based teacher surveys and logs will not allow respondents to enter out-of

-range or inconsistent responses, and data entry programs also will check for inconsistencies. We will pilot the teacher survey and teacher log with nine or fewer respondents who are either teaching or have taught fourth or f ifth grade math or reading. We also will pilot the principal survey with nine or fewer respondents who are or were school administrators. We will monitor the time it takes to complete the surveys and logs, to ensure the final versions can be completed within the allotted time. We will ask respondents to identify any questions that they found problematic and to provide general feedback about their experience. **5. Individuals Consulted on Statistical Aspects of the Design**  The following individuals were consulted on the statistical aspects of the study:40166 OMB Supporting Statement: Part B Mathematica Policy Research

22 Name Title Telephone Number Phil Gleason Senior Fellow, Mathematica 202 - 264 - 3443 Alison Wellington Senior Researcher,M athematica 202 -484 -4696 Allison McKie Senior Researcher,M athematica 202 -484 -4681 Hanley Chiang Senior Researcher,M athematica 617 - 674 - 8374 Elias Walsh Re searcher,Mathematica 202 - 554 - 7516 The following individuals will be responsible for the data collection and analysis: Name Title Telephone Number Phil Gleason Senior Fellow, Mathematica 202 - 264 - 3443 Sheila Heaviside Associate Director of Survey Research, M athematica 202 -484 - 3096 Irma Perez -

Johnson Senior Researcher,M athematica 609 - 275 - 2339 Tim Bruuresema Survey Researcher,M athematica 202 -484 - 3097 Fran O’Reilly Principal Researcher,EBERE 617 - 792 - 0422 Kim Dadisman Associate,Abt Associates 617 - 520- 304040166 OMB Supporting Statement: Part B Mathematica Policy Research

23 **RE FE**  **RENCES** Imai, K., G. King, and C. Nall. “The Essential Role of Pair Matching in Cluster Randomized Experiments, with Application to the Mexican Universal Health Insurance Evaluation.” Statistical Science, vol. 24, no. 1, 2009, pp. 29

–53.**www.mathematica -** **mpr.com**

Improving public well
- being by conducting high
-quality, objective research and surveys Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Was hington, DC

Mathematica® is a registered trademark of Mathematica Policy Research