Impact Evaluation of Data-Driven 

Instruction Professional 

Development for Teachers

Supporting Statement Part B

May 26, 2015
Contract Number:

ED-IES-12-C-0086

Mathematica Reference Number:

40166-700

Submitted to:

Institute of Education Sciences

IES/NCEE

U.S. Department of Education

555 New Jersey Avenue, NW

Washington, DC 20208

Project Officer: Erica Johnson

Submitted by:

Mathematica Policy Research

600 Alexander Park

Suite 100

Princeton, NJ 08540

Telephone: (609) 799-3535

Facsimile: (609) 799-0005

Project Director: Phil Gleason

Impact Evaluation for Data-Driven 

Instruction Professional 

Development for Teachers

Supporting Statement Part B

May 26, 2015
40166 OMB Supporting Statement: Part B Mathematica Policy Research

iii

CONTENTS

PART A: JUSTIFICATION ................................................................................................1

1. Circumstances Necessitating the Collection of Information................1

2. Purposes and Uses of Data................................................................8

PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS................9

1. Respondent Universe and Sampling Methods.....................................9

2. Statistical Methods for Sample Selection and Degree of 

Accuracy Needed.............................................................................13

3. Methods to Maximize Response Rates and Deal with 

Nonresponse...................................................................................21

4. Pilot Testing....................................................................................22

5. Individuals Consulted on Statistical Aspects of the Design...............22

REFERENCES ...............................................................................................................24

APPENDIX A: TEACHER ASSIGNMENT DATA REQUEST

APPENDIX B: TEACHER SURVEY QUESTIONNAIRE AND ACCOMPANYING LETTER 

APPENDIX C: TEACHER ACTIVITY LOG AND ACCOMPANYING LETTER

APPENDIX D: PRINCIPAL SURVEY QUESTIONNAIRE AND ACCOMPANYING LETTER

APPENDIX E: STUDENT RECORDS REQUEST

APPENDIX F: CONFIDENTIALITY PLEDGE
40166 OMB Supporting Statement: Part B Mathematica Policy Research

iv

TABLES

Table 1. Data-related Activities, by Source..................................................................14

Table 2. Teacher and Principal Intermediate Outcome Measures, by Source................17

Table 3. Minimum Detectable Effects by Sample Size..................................................19
40166 OMB Supporting Statement: Part B Mathematica Policy Research

1

SUPPORTING STATEMENT FOR PAPERWORK 

REDUCTION ACT SUBMISSION

This OMB package requests clearance for data collection activities for a rigorous evaluation of 

data-driven instruction (DDI) in 104 schools from 12 school districts that have been recruited for 

the study. Data-driven instruction involves the use of student assessment data to help teachers adapt 

their instruction and, ultimately, improve student achievement. The study’s intervention plan will 

build school capacity for DDI by: (1) helping schools set up structures and processes that enable 

teachers and other school staff to efficiently carry out data-driven instruction, and (2) training and 

coaching teachers in the skills needed to understand student data and implement improved 

instructional strategies to address student needs. By participating in professional development and 

ongoing DDI activities, teachers are expected to develop the knowledge and skills needed to help 

them adapt and improve their instructional strategies based on student data, and ultimately, improve 

student achievement.

We plan to collect student records and teacher-assignment data from participating districts and 

schools, and conduct a teacher survey, teacher logs, and a principal survey. The Institute of 

Education Sciences (IES) within the Department of Education (ED) has contracted with 

Mathematica Policy Research and its partners Abt Associates and Evidence-Based Education 

Research & Evaluation to conduct the evaluation, and Public Consulting Group–Focus on Results 

to provide technical assistance to schools implementing the DDI program.

The evaluation’s main objectives are to understand how DDI is implemented and to rigorously 

estimate the impact of a comprehensive DDI program on student achievement and teacher and 

principal practices. The implementation component will use information collected from the 

technical assistance (TA) team, a teacher survey and logs, and a principal survey. For the impact 

evaluation, the experimental design involves randomly assigning schools within a district to either a 

treatment or control group. The treatment schools will implement a comprehensive DDI program, 

with technical assistance provided by an organization that works with schools to implement such 

instruction, and the control schools will not implement new DDI initiatives during the study years. 

Student outcomes will include students’ achievement on math and reading state assessments. 

Teacher and principal outcomes will include teachers’ and principals’ use of data, teachers’ 

instructional strategies, and the extent and nature of teacher collaboration. 

This OMB clearance request concentrates on materials that will be used to collect principal, 

teacher, and student data. Included in this request are the following: a teacher assignment data 

request (Appendix A), a teacher survey questionnaire and an accompanying letter (Appendix B), a 

form and letter for the administration of teacher logs (Appendix C), a principal survey questionnaire 

and accompanying letter (Appendix D), a student records request (Appendix E), and a 

confidentiality pledge (Appendix F).

PART A: JUSTIFICATION

1. Circumstances Necessitating the Collection of Information

a. Statement of Need for a Rigorous Evaluation of DDI

The specific piece of legislation authorizing this evaluation is Title II, Part A of the Elementary 

and Secondary Education Act (ESEA), Section 2121-2123 as amended by No Child Left Behind 

(NCLB) (20 USC 6621-6623). The Title II legislation provides grants to states and districts to 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

2

increase student academic achievement. State and local education agencies can use Title II funding 

to offer professional development activities that “provide training on how to understand and use 

data and assessments to improve classroom practice and student learning” (U.S. Department of 

Education, 2013). The federal government is also actively promoting DDI through other initiatives, 

including Race to the Top and School Improvement Grants.

A growing number of schools across the country have adopted DDI programs. Although there 

are no comprehensive national data on the current number of schools implementing DDI—in part 

due to differing definitions of this intervention—DDI providers report implementing DDI 

programs in hundreds of districts and tens of thousands of schools.1 One provider reports that its 

DDI materials and training are used in 20 percent of K–12 school districts nationwide (Cordray et 

al. 2012). 

Despite the growing interest in data use across the country, there is limited rigorous evidence 

on the effects of comprehensive DDI, especially for programs that feature intensive on-site support 

for teachers and other school staff. The few rigorous evaluations of DDI typically find no 

statistically significant impacts on student achievement.

2

Several of these experimental studies have 

had methodological limitations, however, and the DDI interventions they examined did not provide

substantive supports at the school and teacher levels (Konstantopoulos et al. 2013; Cordray et al. 

2012; Carlson et al. 2011; Slavin et al. 2013). In general, the studies have focused on interventions 

that did not include teacher collaboration, an important best practice (Hamilton et al. 2009; Lachat

and Smith 2005), or intensive professional development and on-site coaching and other supports 

specifically designed to help teachers fully understand student data and adapt their instruction

accordingly. In fact, the studies found that the interventions they examined had little or no effect on 

instructional practices (Cordray et al. 2012) or that DDI teachers were actually less likely to use 

differentiated instruction, an important DDI teaching practice (Williams et al. 2013; 

Konstantopoulos et al. 2013). Thus, the existing literature is limited in the information it can provide 

on the potential impact of a comprehensive DDI intervention.

3

This study would provide important, new experimental evidence on the impact of a 

comprehensive DDI program on student achievement. We will also gather valuable information on

how DDI is implemented in schools.

b. Study Design and Research Questions

The study team will evaluate the effectiveness of the DDI program provided by Focus on 

Results using a random assignment design. A total of 104 schools from 12 school districts have 

 

1 This information is based on conversations with representatives of nine large DDI providers: Achievement 

Network, Acuity, Curriculum Associates, Data Wise, Education for the Future, Northwest Evaluation Association, 

Public Consulting Group-Focus on Results, TERC, and Wireless Generation.

2 There have been several nonexperimental examinations of DDI (Shaw and Wayman 2012; Henderson et al. 2007; 

Quint et al. 2008; Datnow et al. 2007), but because schools or teachers using DDI may systematically differ in other 

ways from schools or teachers not choosing to use DDI, these non-experimental results do not support rigorous causal 

inference.

3 In addition to the studies listed here, there is an ongoing evaluation of the Achievement Network DDI

intervention. That intervention includes the administration of a particular set of interim assessments along with supports 

to participating schools to aide them in interpreting and using the assessment results. A report on this study is 

forthcoming in spring 2015. Therefore, at this time, the detailed characteristics of the evaluation design and the findings 

are not known.
40166 OMB Supporting Statement: Part B Mathematica Policy Research

3

been recruited for the study. Within study districts, schools have been randomly assigned to either a 

treatment group that is implementing the study provided DDI program or a control group that is 

not receiving the intervention and therefore not implementing the key DDI treatment features 

during the study. 

While the study will examine outcomes for the 2015-2016 school year, training and setting up 

the infrastructure to support DDI activities will occur during the winter and spring of 2014-2015 in 

the treatment schools. These activities occur in advance of the 2015-2016 school year because it may 

take some time to set up the necessary school structures and to train staff on data-driven instruction 

procedures. Treatment schools will learn how to set up the structures and implement the procedures 

appropriate to data-driven instruction through ongoing professional development and technical 

assistance. Beginning these supports prior to the 2015-2016 school year will allow treatment schools 

time to gradually receive training and gain practice implementing key data-driven instruction 

activities to then be ready for full DDI implementation during the 2015-2016 school year. The study 

implementation team will support and monitor implementation both during the initial infrastructure 

building phase as well as the full implementation phase.

The implementation and impact evaluation study will address the following research questions: 

 What is the impact of DDI professional development on students’ achievement on state 

assessments in math and reading?

 What is the impact of DDI professional development on teachers’ and principals’ access 

to and use of data to inform instructional support, planning, and practice?

 How does DDI professional development affect teachers’ use of instructional strategies, 

such as differentiated instruction? 

 How do DDI-related activities differ in treatment versus control schools?

Because the primary research question focuses on the impact of DDI on student 

achievement, the study will examine impacts on grades in which state assessments are 

administered, grades four and five. Targeting fourth and fifth grades will also focus intervention 

resources more intensively on a subset of participating schools’ teachers, while allowing us to 

examine the effects of DDI on achievement across two different grade levels.

c. The Data Driven Instruction Program Provided by Focus on Results

The Focus on Results plan for DDI is based on a comprehensive framework for training and 

supporting school leadership and teachers on using multiple sources of student data (such as 

formative and summative student assessment data, student classroom work, student attendance and 

behavior data, etc.) to inform instruction and improve student achievement. The DDI program is 

designed to be implemented in schools with regular interim assessments and a data system that 

produces reports on student achievement. Key DDI components and how they are expected to 

contribute to achieving intermediate and final outcomes are depicted in Figure 1. The overall logic 

of the DDI program is that a core set of training and technical assistance inputs inform and guide 

various activities by school leaders and individual teachers. These inputs and activities lead to 

teachers increasing their use of student data to guide instruction and altering their instructional 

strategies, which in turn leads to improved student achievement. Specifically, the instructional 

leadership team (described below) explores data at the school and individual grade levels to diagnose 

core issues for school-wide focus. Teacher collaboration teams (also described below) and individual 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

4

teachers explore data in their grade and classroom to generate ideas for instructional improvement 

of greatest benefit and consistent with the school-wide focus. Teacher collaboration teams and 

individual teachers adjust instruction and continuously monitor student progress. The school-wide 

instructional leadership team also monitors progress at the school level. Guided by this ongoing 

review of student progress, adjustments in instruction and focus take place as needed in a continual 

feedback/adjustment process to accomplish school level goals. 

Figure 1: Logic Model for Planned DDI Intervention

Training and Technical 

Assistance Inputs School Structures and Activities Intermediate

Outcomes

Final 

Outcomes

Winter Institute 

(for principals and

data coaches)

Orientation, training, and 

resources for DDI

implementation

Instructional Leadership Team (principal, 

data coach, and grade/department chairs)

Examine school-wide data

Set instructional focus and instructional 

best practices

Set school goals and monitor progress

Communicate importance of data-driven 

culture

Use PD to train teachers

Training and Professional 

Development (PD) 

(for instructional leadership 

team)

Ongoing PD sessions

Summer refresher training

Improved 

Student 

Achievement in 

English/

Language Arts 

and 

Mathematics

Teacher Collaboration 

Teams (TCT)

Dedicated meetings

Structured activities to 

understand student data

Examine student work

Share effective practices

Set and monitor goals

Teachers 

Receive training and 

coaching

Improved Access to 

and Use of Data

Access to student 

data and work from 

multiple sources

Examine student data 

and student work

Use data to set and 

monitor goals

Ongoing TA

(for instructional leadership 

team)

Monthly calls with coaches

In-person visits

Ad hoc contacts

Data Coach

Strategizes with principal

Analyzes data

Reviews research

Co-leads instructional leadership team

(with principal)

Organizes teacher collaboration teams 

Facilitates TCT meetings

Provides coaching

Develops PD room

Develops data walls

Principal

Communicates importance 

of data-driven culture

Analyzes student data

Co-leads instructional

leadership team

Monitors implementation

Contextual Factors

District policies Data coach experience Teacher experience Principal experience Access to student data (interim and summative assessments) 

Improved 

Instructional

Strategies

Differentiate 

instruction

Adjust student 

groups

Increase instructional 

time on lowperforming areas

Incorporate evidencebased practices

Participating schools assigned to the treatment will set up new structures to support the 

implementation of DDI. School leaders and individual teachers will engage in activities designed to 

help staff learn to analyze data from interim assessments and other sources to identify areas for 

improvement, jointly formulate improvement goals and priorities, implement evidence-based 

changes in instructional practices, and monitor progress toward achieving the agreed-upon goals. 

The following people and teams of people will be involved in the intervention in treatment schools: 

 Data coach. Participating schools will hire a half-time data coach to lead and monitor the 

implementation of DDI activities on a day-to-day basis. Trained and supported by Focus on 

Results consultants, the data coach will analyze student data, conduct strategic planning with 

the principal and other school leaders, and work directly with teachers—individually and in 

groups (teacher collaboration teams). These activities will help teachers learn to use available 

data effectively to formulate instructional improvement goals, identify relevant best 

practices, and implement such practices. The coach will create and maintain a DDI resource 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

5

room with materials to help school leaders and teachers analyze data and research possible 

instructional practices. The coach also will develop a set of data displays, available in the 

resource room and throughout the school, to help school leaders and teachers monitor 

student progress.

 Principal. As a school’s main instructional leader, the principal plays a critical role in 

establishing a data-driven culture in the school and building support and momentum for 

DDI implementation. After receiving initial training, the school’s principal and data coach 

together plan for DDI implementation; they also meet weekly to review progress, 

troubleshoot issues encountered, and adjust plans for DDI implementation as needed. 

Principals work to support the initiative by emphasizing the targeted instructional focus and 

practices during regular classroom visits, follow-up feedback to teachers, and interactions 

with the instructional leadership team and data coach.

 Instructional Leadership Team. Each participating school will establish an instructional 

leadership team, whose primary role is to lead the school’s DDI implementation effort. The 

team will include the school’s principal, data coach, grade-level chairs (and/or department 

chairs, if instruction is departmentalized), and other formal or informal school leaders (for 

example, assistant principal, other teachers, or IEP specialist). This team is charged with 

analyzing school-wide data and, based on this analysis, identifying areas of instructional 

focus for school-wide DDI improvement efforts (including specific instructional practices 

linked with these areas), and setting and monitoring goals for increased student achievement. 

Since DDI implementation relies on a train-the-trainer approach to build capacity within 

each school, this team also has the important responsibility to take what it learns from 

ongoing professional development sessions with Focus on Results and pass it on to teacher 

collaboration teams (described next), to guide their efforts. The instructional leadership team 

is also charged with addressing more concrete issues, such as ensuring that teacher 

collaboration teams have the time and space to meet on a regular basis.

 Teacher Collaboration Teams. Teachers in each participating grade or department will 

form a professional learning community to work collaboratively, under the leadership of the 

data coach, to do much of the on-the-ground work of DDI. Each team will hold regular 

meetings (at least every other week) to determine what the data (both formal interim 

assessments and informal assessments) suggest regarding student learning needs, collectively 

set grade/department-specific goals and monitor progress toward goals, and identify 

instructional strategies that could better help them meet the needs of students scoring at the 

lowest levels. 

 Individual Teacher Activities. Drawing on their work within the teacher collaboration 

teams, as well as one-on-one assistance from the data coach, each teacher will implement 

agreed-upon instructional strategies in support of both the school-wide and grade-level 

improvement goals. Each teacher will also consider her students’ proficiency levels with 

respect to different standards (based on assessment data and student work) and make 

decisions about how best to address the students’ identified learning needs. After 

implementing the instructional strategies designed to meet these learning needs, teachers will 

assess the success of their efforts based on feedback from the coach as well as evidence from 

subsequent student work and assessment data.
40166 OMB Supporting Statement: Part B Mathematica Policy Research

6

By collaborating on teams and participating in the activities described above, teachers are 

expected to develop the knowledge and skills needed to help them adapt and improve their 

instructional strategies based on student data. The relevant instructional strategies for a given teacher 

will depend on the specific learning needs of that teacher’s students, as indicated by the student data. 

DDI does not require that teachers implement any one specific instructional approach. Nonetheless, 

based on Focus on Results’ experiences and best-practices identified by DDI experts (Hamilton et 

al. 2009), we expect teachers in treatment schools to be more likely than teachers in control schools 

to adopt certain strategies or use them more frequently. For example, DDI should lead to a greater 

use of differentiated instruction, more frequent review and adjustment of students’ small group 

assignments, and integrating evidence-based practices into all aspects of their teaching. 

Schools selected to implement DDI for the study will receive resources, training, technical 

assistance, and ongoing support from consultants from Focus on Results to help them implement 

DDI. Capacity-building inputs will include (1) an initial training for school principals and data 

coaches, (2) professional development training for instructional leadership teams, and (3) other 

technical assistance support for data coaches and instructional leadership teams during DDI 

implementation. The following training and technical assistance activities will take place in treatment 

schools: 

 Initial Training. A two-day orientation and training session ensures that data coaches and 

principals are prepared to begin implementing DDI in treatment schools. Activities include 

both general information about DDI and its implementation and specific training tailored to 

the activities each group will be expected to perform at their schools. Participants also are 

provided an opportunity to network and exchange ideas with data coaches and principals at 

other study schools implementing DDI. 

 Ongoing professional development for instructional leadership teams. Each school’s 

leadership team will receive a total of six professional development sessions to support the 

team in guiding their school’s adoption of DDI and provide training and skills that can be 

passed on to teachers at the school. Each session focuses on a particular set of activities 

related to the implementation of DDI, such as identifying an instructional focus, looking at 

student data and setting goals, using best practices in teachers’ classrooms, and monitoring 

their progress. Members of these instructional leadership teams will then train teachers and 

other staff in their schools using the skills and knowledge they learn from the professional 

development sessions.

 Other technical assistance. Instructional leadership teams will receive customized 

technical assistance to help address questions and needs that arise as schools implement data 

driven instruction. Each participating school will be visited a total of eight times by Focus on 

Results consultants to monitor DDI implementation and provide customized technical 

assistance. Technical assistance also will be delivered during monthly phone calls with the 

data coach, and additional phone calls or e-mails, as needed.

d. Data Collection Plan

To address the study research questions, the study team will collect and analyze data from

teachers, principals, and district staff. A brief description of each data source and data collection 

activity is provided below.
40166 OMB Supporting Statement: Part B Mathematica Policy Research

7

 Teacher and Principal Assignment Data: We will collect teacher assignment data 

from participating schools in winter 2016. We will ask schools to identify teachers 

currently teaching fourth and fifth grades, provide contact information for these 

teachers (such as school e-mail address), and indicate if the teacher taught at the school 

in fall 2014 (Appendix A).4 The study team will attempt to identify principals at study 

schools from public sources (such as school websites). If that information is not 

publicly available, we will request schools to provide principal assignment information.

 Teacher Survey: We will administer a web-based survey in spring 2016 to 500 teachers 

(Appendix B). Teachers will also have the option of completing the survey by hard copy 

or by phone, if they prefer. The survey will provide information on professional 

development and training received by teachers (particularly related to DDI topics), 

school data culture, instructional planning and collaboration, teachers’ access and use of 

data to guide instruction, teachers’ instructional strategies, and teachers’ demographic 

and background characteristics. 

 Teacher Logs: We will administer web-based logs with teachers during the 2015-2016 

school year (Appendix C). The same set of 500 teachers who will be asked to complete 

the teacher survey will also be asked to complete these logs at two points during the 

year, and in each case report on their activities over two consecutive days. The logs will 

provide information on teachers’ day-to-day activities, including planning activities 

(individually and in collaboration with other teachers) and instructional strategies in the 

classroom.

 Principal Survey: We will administer a hard-copy survey to all 104 study principals in 

spring 2016 (Appendix D). The survey will focus on topics such as school-wide 

leadership activities, school data culture, school-wide access to and use of data, and 

principals’ demographic and background characteristics. Principals will have the option 

to provide answers to a trained interviewer over the phone.

 Student Records Collection: In summer 2016, we will collect student outcome 

measures (for example, math and reading test scores from state assessments) for the 

year of full implementation (2015–2016) as well as for two prior years (2014–2015 and 

2013-2014 if available). We will also collect student demographic and background data, 

such as gender, race/ethnicity, and eligibility for free and reduced-price lunch, and 

student attendance and disciplinary measures (if available), and student enrollment data 

for fall 2014 and winter 2016 (Appendix E).

e. Study Activities and Time Line

In Table 1, we show the timing of the major study activities. 

Table 1. Schedule of Major Study Activities for Data-Driven Instruction Evaluation

Spring 

2014

Summer 

2014

Fall 

2014

Dec

2014

Spring 

2015

Fall/

Winter

2015

Winter/

Spring 

2016

Summer/

Fall 

2016

Summer 

2017

Recruit Districts X X X

Conduct Random 

Assignment

X X X

 

4 Teacher assignment data will be collected following approval of the OMB clearance package. 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

8

Training and 

Infrastructure Setup

X X

Full Implementation X X

Collect Teacher

Assignment Data*

X

Conduct Teacher Survey X

Conduct Teacher Logs X X

Conduct Principal Survey X

Collect Student Records 

Data

X

Prepare Report X

*Schools will be asked to provide principal assignment data only if the information cannot be obtained from publicly available 

sources. 

2. Purposes and Uses of Data

The main purpose of this evaluation is to estimate the impacts of DDI on student achievement 

and other outcomes, as well as to document implementation of DDI programs and activities. In 

Table 2, we list the study’s research questions and the data collection to support the answers.

Table 2. Research Questions and Data Collection Method

Research Questions Data Collection Method

1. What is the impact of DDI professional development on 

students’ achievement on state assessments in math and 

reading?

 Student records data

2. What is the impact of DDI professional development on 

teachers’ and principals’ access to and use of data to 

inform instructional support, planning, and practice?

 Teacher survey

 Teacher logs

 Principal survey

3. How does DDI professional development affect teachers’ 

use of instructional strategies, such as differentiated 

instruction? 

 Teacher survey

 Teacher logs

4. How do DDI-related activities differ in treatment versus 

control schools?

 Teacher survey

 Principal survey

 Technical assistance 

documents

 Teacher and Principal Assignment Data: Teacher assignment data will be used to 

randomly select approximately 500 fourth and fifth grade teachers to complete the 

teacher survey and logs. Principal assignment data will be used to identify the principals 

in the 104 study schools to determine the sample for the principal survey. Information 

on teacher and principal assignments in fall 2014 will be used to examine whether the 

DDI intervention lead to a treatment-control difference in educators’ mobility between 

fall 2014 (following random assignment) and the 2015-2016 school year.

 Teacher Survey and Logs: Teacher survey and log data will be used to analyze teacher 

intermediate outcomes and to measure important aspects of the treatment-control 

contrast. Intermediate outcomes that will be examined include the impact of DDI on 

teachers’ access to and use of data to guide instruction and teachers’ instructional 

strategies (such as the use of differentiated instruction). We will also examine 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

9

treatment-control differences in the training teachers receive and other activities 

surrounding the use of data in schools. 

 Principal Survey: The principal survey will enable a better understanding of the

treatment-control contrast in schools’ training and activities surrounding the use of data, 

as well as shed light on some aspects of DDI implementation. It will examine topics 

such as whether principals have established school-wide goals, their expectations for 

school-wide data use, whether the school has an instructional leadership team and the 

activities it undertakes, and the degree to which school leaders review and discuss 

student data. 

 Student Records Collection: We will use existing district test score data to estimate the 

impact of DDI on student achievement, the key outcome of interest. Information on 

students’ demographic and socioeconomic characteristics and their achievement test 

scores prior to the study school year will be used to describe the students in the study 

and to develop more precise impact estimates. 

In the process of implementing the intervention and providing technical assistance to treatment 

schools, we will also obtain information on aspects of implementation from the evaluation team’s 

technical assistance provider (Focus on Results), such as materials from their professional 

development sessions.

We will present the study findings in a report that will include information from both the 

implementation and impact analyses. The data collected by this evaluation will also be available as 

restricted-use files, serving as a valuable resource for researchers.

PART B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL 

METHODS

1. Respondent Universe and Sampling Methods

This study will not statistically sample districts and schools. It will, instead, rely on a 

convenience sample of 104 schools within 12 districts that volunteered to participate in the 

evaluation and meet the study requirements. The evaluation does not aim to make statements that 

generalize beyond the districts and schools under study.

Number of Districts and Schools. To be able to detect the impact of DDI on student 

achievement at the level requested by ED, we will include 104 schools that include grades 4 and 5 in 

the evaluation (see Table 3 below). Each district will include an average of 8 to 9 schools in the 

evaluation; overall, 12 districts will take part in the evaluation. 

Randomly Assigning Schools. Our random assignment approach involved identifying

strata—or matched pairs—of similar schools within districts and then randomly assigning schools 

within each stratum to the treatment or control group. The matched pairs of schools were formed 

prior to random assignment, and each stratum was primarily based on average student achievement 

in prior years, the characteristic most likely to be predictive of student outcomes during the study 

period. Other characteristics we considered in forming strata include: (1) the proportion of students 

receiving free or reduced-price meals, (2) the racial/ethnic distribution of students, (3) the 

proportion of students classified as English language learners, (4) Title I eligibility, and (5) school 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

10

size. We obtained information on these school characteristics from the Common Core of Data and 

state websites. Random assignment within strata will increase statistical precision by reducing 

random differences in the average baseline characteristics of treatment and control schools (Imai et 

al. 2009). 

Identifying and Recruiting Districts and Schools. The districts and schools included in this 

study were not randomly sampled and so are not statistically representative of the broader group of 

all public schools serving fourth and fifth graders nationally. Participating districts and schools had 

to be both interested in and eligible to participate in the study. Interested districts had to be 

comfortable with implementing DDI; allowing the intervention to be randomly assigned to its 

participating schools; and willing to comply with data collection activities, including student data 

collection, a teacher survey, teacher logs, and a principal survey.5 Eligible districts also met the 

following criteria:

 Summative and interim assessments. To effectively implement DDI, teachers and administrators 

in participating schools needed to monitor student achievement on a regular basis using 

student data from both summative assessments (cumulative end-of-year assessments) and 

interim assessments (periodic assessments intended to evaluate student learning progress 

during the year). Teachers in study schools will rely upon existing summative and interim 

assessments used within each district in order to monitor student achievement; the 

evaluation will not implement study-specific assessments within participating schools. Both 

summative and interim assessments used by each participating district needed to be 

uniformly administered in 2014-2015 and 2015-2016. Interim assessments had to be highly 

aligned to the state summative assessment and, ideally, made by the same developer as the 

summative assessment.6 Thus, recruitment targeted districts that used summative and 

interim assessments meeting both of these criteria (including, but not limited to, states that 

were part of the Smarter Balanced Assessment Consortium). 

 Districts serving high needs students. Recruitment efforts aimed to include districts with a large 

number of high needs schools that would most benefit from DDI, and thus targeted 

districts with a relatively high proportion of free or reduced price meal (FRP) students. 

Recruitment targeted districts serving disadvantaged student populations, as indicated by 

having rates of FRP eligibility higher than 40 percent; within those districts, we targeted the 

higher poverty schools.

 District size. In order to meet our targeted sample size, the study recruited districts with at 

least eight elementary schools that contained fourth and fifth grades. The study targeted 

elementary schools for two reasons. Districts typically have a larger number of elementary 

schools than middle schools, so identifying and recruiting a sufficient number of elementary 

schools would potentially be easier. In addition, elementary schools are less likely than 

middle schools to be departmentalized, and tracking by ability level is also less common at 

the elementary school level. Thus, the intervention will be able to work with groups of 

 

5 Originally all states were considered for inclusion in the study. However, during the recruitment process, we 

decided to exclude states that planned to use the PARCC assessments because it was unclear when those assessments 

would be available. One large state was also excluded due to concerns about the ability to obtain student achievement 

data without parental consent. 

6 This criterion resulted in excluding states that developed their own summative assessments.
40166 OMB Supporting Statement: Part B Mathematica Policy Research

11

teachers who are with their students all day and perhaps have more flexibility in refining 

their instructional practices.

 No District-wide DDI Initiatives. For the purposes of ensuring a sufficient contrast between 

treatment and control schools, the study targeted districts that had made minimal (or no) 

efforts to implement data-driven instruction. We avoided districts that had implemented 

district-wide DDI initiatives, as well as districts that planned to implement new DDI 

programs during the 2014-15 or 2015-16 school years. Similarly, the study avoided districts 

that had implemented district-wide Response to Intervention (RtI) programs, given that RtI 

programs utilize a similar data-driven approach. 

Eligible districts also expressed a willingness to cooperate with data collection activities and 

agreed to allow their schools to be randomly assigned to treatment or control conditions. Once we 

identified districts that met the criteria above, we identified a subset of eligible schools within those 

districts, based on the following eligibility criteria:

 High needs schools. Within each district, recruitment efforts aimed to include high needs 

schools, and thus targeted schools with a proportion of free or reduced price meal (FRP) 

students that was high relative to other schools in the district. 

 No school history of or plans to implement DDI. Within each district, the study avoided schools 

that had a data coach in place, received regular professional development focused on data 

use, or held regular teacher collaboration group meetings focused on analyzing and making 

use of student data. We also excluded schools that planned to implement new DDI 

programs during the 2014-15 or 2015-16 school years and schools that had implemented 

RtI programs. 

 No Schools with School Improvement Grants (SIG). Schools that had received School 

Improvement Grants (SIG) were not eligible for the study, given that DDI was an 

important part of required SIG activities under some of the intervention models specified 

by ED.

 Title I schools. The U.S. Department of Education has expressed a strong interest in studying 

effective approaches to help low-income children within Title I schools meet state standards 

for academic achievement. A comprehensive approach to DDI is consistent with this 

interest. While the study preferred to include Title I schools, this was not a strict 

requirement for participating; as a result, some eligible schools did not have Title I status. 

 Excluding charter and magnet schools. Charter schools, which often utilize data-driven

approaches to instruction and that typically operate autonomously from other district 

schools, were excluded from the study. Magnet programs were included only when the 

district provided an even number and similar type of magnet schools. 7

Eligible schools also expressed a willingness to cooperate with data collection activities. Eligible 

schools agreed to provide the study with student-level assessment and demographic data (if not 

provided by the district). In addition, their principals expressed willingness to complete a survey in 

spring 2016, and the schools agreed to encourage their teachers to complete a teacher survey in 

spring 2016.

 

7 Among the 104 schools included in the study, 11 schools were magnet programs. Districts contributed magnet 

schools in pairs that could be matched to one another, with one exception—a case where a magnet school was matched 

to a non-magnet school that the district reported had similar programming and structures in place as the magnet school.
40166 OMB Supporting Statement: Part B Mathematica Policy Research

12

A key goal of recruitment for the evaluation of DDI was, thus, to recruit a sufficient number of 

schools that both met the above eligibility requirements and that were interested in the study. As 

described below, the study estimated that 100 schools would be needed to generate sufficient 

statistical power to detect meaningful differences in student achievement between the treatment and 

control groups. Assuming that each district would be able to provide 8 to 9 schools, the study aimed 

to recruit 12 districts. 

During the recruitment process, we first used data from the Common Core of Data and other 

publicly available sources to identify districts meeting a basic set of eligibility criteria (including their 

plans for summative and interim assessments, percent FRP, district size, SIG status and Title I 

status). While we could not be entirely certain about state assessment plans in future years, we 

collected detailed information from state contacts and public sources to help us identify states that 

were fairly confident about their assessments plans in 2014-2015 and 2015-2016. Our team 

identified an initial pool of 337 districts that met these criteria, and began recruiting a subset of these 

districts. 

Out of the initial pool of 337 eligible districts, a total of 103 districts were targeted for 

recruitment and, among those targeted, 12 districts were included in the final sample.8 Within the 12 

study districts, 104 schools agreed to participate in the study. Districts contributed a total of 4 to 14 

schools each, depending upon a number of factors, including the size of the district, overall 

eligibility and suitability of selected schools, and the remaining number of schools needed to reach 

the overall recruitment target. 

2. Statistical Methods for Sample Selection and Degree of Accuracy Needed

a. Statistical Methods for Sample Selection

As described in Section 1, the study is using a convenience sample of districts and schools that

volunteered to participate in the evaluation and met the study requirements. To meet the study’s 

target effect size, we included 12 districts and 104 schools. We will conduct surveys with all 104

principals, and we will collect administrative records on all students in tested grades within the 

participating schools. Thus, the study will not statistically sample principals, schools, or students. 

We will sample teachers for the teacher survey as explained below.

We will administer a web-based survey to 500 teachers in the target grade levels in the spring of 

the 2015–2016 school year, and administer web-based logs in the winter and spring of that school 

year (Appendices B and C). The teacher sample will consist of five teachers from each study school, 

on average, randomly selected from the pool of all fourth and fifth grade teachers in the study 

schools. Selecting only teachers who teach students in tested subjects and grades 4 and 5 has the 

advantage that it aligns the information we learn from teachers with the student achievement 

 

8 Among the 91 districts that did not end up in the final sample, approximately half (44 districts) were dropped 

because they lacked interest in the study, and half (47 districts) were dropped because they did not meet all of the 

eligibility criteria. The most common reason that districts were found to be ineligible was because they did not meet 

study criteria for the use of summative and interim assessments. Other districts were dropped because they already had 

an existing DDI program in place or did not have a sufficient number of schools that met eligibility criteria. 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

13

analyses.9 The disadvantage is that we will not have as a complete picture of DDI implementation in 

the school as we would if we also included all teachers in the participating grades. 

We plan to identify teachers by collecting teacher assignment data from the study schools. We 

will use the teacher assignment data to identify fourth and fifth grade teachers in treatment and 

control schools. We will stratify teachers by grade and/or subject in order to obtain a similar number 

of teachers by grade and subject from treatment and control schools. From each stratum, we will 

randomly select teachers to receive the survey, for a total of 500 teachers. 

b. Estimation Procedures

Random assignment of schools within districts to a treatment group that will implement the 

study provided DDI program or to a control group that will not is an ideal design for assessing 

overall effectiveness of DDI. Our primary impact analysis will exploit this experimental design to 

provide rigorous estimates of the impact of DDI on student achievement and other student 

outcomes. Additional experimental analyses are designed to estimate the impact of DDI on teacher 

outcomes, such as teachers’ access to and use of student data and teachers’ use of instructional 

strategies. 

Implementation Analysis. The implementation analyses have two main objectives. The first 

objective is to describe the DDI activities implemented in treatment schools and the extent to which 

they were implemented with fidelity. The second objective is to describe differences in treatment 

and control schools’ implementation of DDI-related activities. The analyses will rely upon 

information from the data coach weekly activity logs and the Focus on Results consultant activity 

logs, as well as from the principal survey, teacher survey, and teacher logs.

Fidelity of DDI implementation. The implementation analysis will describe DDI activities 

undertaken by treatment schools and the extent to which they were implemented with fidelity to the 

intervention plan. As a part of this analysis, we will review log entries from January 2015 through 

spring 2016. Because it may take time for treatment schools to set up necessary school structures 

and roll out intervention activities, we expect some variance in implementation at different points in 

time during the intervention. Thus, we will examine fidelity of implementation at three points: (1) at 

the end of the 2014-2015 school year, (2) in December 2015, and (3) at the end of the 2015-2016 

school year. 

To assess the extent to which treatment schools implemented DDI with fidelity, it will be 

important to quantify the data and summarize it uniformly. The DDI intervention aims to build

school capacity in two ways, by (1) helping schools set up structures and activities that enable school 

staff to carry out DDI, and (2) training and coaching teachers in the skills needed to use and 

interpret student data. Implementation fidelity measures will therefore examine the training and 

coaching that treatment schools receive and the degree to which treatment schools fully implement 

the anticipated DDI structures and activities.

In describing the fidelity of implementation of a particular DDI component, we will focus on 

the activities we expect to have happened if the component has been implemented. We will measure 

both whether the activity occurred and, if appropriate, the level of participation in the activities. To 

 

9 Although we will collect student test scores on all students in tested grades in the study schools, our primary 

impact analyses will focus on fourth and fifth graders.
40166 OMB Supporting Statement: Part B Mathematica Policy Research

14

measure the fidelity of implementation of the instructional leadership team meetings, for example, 

we will measure whether a school holds monthly instructional leadership team meetings among all 

key staff, attendance at these meetings among the staff, and the key activities taking place at these 

meetings. These activities would include that the team establishes school-wide goals for 

performance, establishes areas of instructional focus, and examines student data. 

Our examination of implementation fidelity will be accompanied by descriptive text that 

summarizes our findings, describes changes in implementation over the course of the intervention, 

and provides a range of examples of how schools implemented DDI with fidelity. 

We will also describe treatment schools’ experiences in implementing DDI. This information 

will be especially important if the DDI approach is found to be effective and other districts wish to 

replicate the program. We will examine school-level factors that may influence DDI implementation, 

such as the level of engagement of the school principal. 

Implementation of DDI-related activities in treatment and control schools. The DDI intervention is 

expected to lead the principal, data coach, instructional leadership team, teacher collaboration teams, 

and individual teachers in treatment schools to engage in numerous data-focused activities. These

include (1) activities undertaken by school leaders in directing data use, (2) school-wide 

communications about data use, (3) professional development and support for the principal and 

teachers on data use, and (4) collaboration among teachers to review student data and share

instructional strategies. We will use data from the principal survey, teacher survey, and teacher logs 

to examine whether implementation of the DDI intervention leads to a treatment-control difference 

in these activities (Table 1).

Table 1. Data-related Activities, by Source

Principal Survey Teacher Survey Teacher Logs

Activities undertaken by school leaders in directing data use

Frequency of leadership team meetings X

Leadership team members X

Leadership team activities (e.g., setting achievement and 

priority learning goals, monitoring progress, planning 

professional development activities)

X

Degree to which school leaders ensure that teachers have the

time and resources needed to analyze and interpret student 

data

X X

School-wide communications about data

Frequency of communication on student achievement goals 

and results X X

Frequency of communication on priority learning goals X X

Frequency of communication on expectations for and actual 

use of data X X

Use of data displays in classrooms and other public areas X X

Professional development and support for the principal, teachers around data use

Amount of professional development/training activities this school 

year

X X

Topics covered by trainings (analyzing student data, establishing 

priority learning goals for the school, individualizing student 

learning goals, tracking progress toward goals, using evidenceX X
40166 OMB Supporting Statement: Part B Mathematica Policy Research

15

based instructional strategies)

Availability of on-site coaching/support for data use X X

Data coach activities X

Frequency of, and topics addressed during, individual coaching X X

Frequency of classroom observations and feedback X X

Collaboration among teachers around data use and sharing instructional strategies

Frequency of, and amount of time spent on, teacher collaboration X X

Teacher collaboration activities (analyzing student data, setting 

common learning goals for students, sharing effective instructional 

practices, jointly modifying lesson plans, monitoring 

implementation and results of instructional changes)

X X

Impact analysis. The impact analysis will rigorously assess the effectiveness of a 

comprehensive DDI program. Calculating the statistical significance of the impacts requires that the 

nested structure of the data—with students clustered in schools–is incorporated in the analysis. Due 

to clustering, the variance of the impact estimates is larger than it would have been if each individual 

student were randomly assigned to DDI. Below we describe our approach to calculating outcomes 

of student achievement and intermediate outcomes on teacher and principal practices. 

Student final outcomes. Student achievement will be the study’s primary outcome, measured using 

spring 2016 math and reading scores on state standardized tests. Specifically, this main impact 

analysis will examine the effect of DDI on:

 Math achievement among fourth and fifth graders in a school that implemented datadriven instruction 

 ELA achievement among fourth and fifth graders in a school that implemented datadriven instruction 

To assess the impact of data-driven instruction on student achievement, we will use a placebased impact estimation strategy that compares outcomes for students in treatment schools to those 

of students in control schools using spring 2016 state test scores. This implies that the impacts could 

reflect either the impacts of data-driven instruction on student achievement or impacts on student 

mobility. Given the nature of the intervention, we do not expect that the data-driven instruction 

intervention would affect students’ mobility patterns during the study period. However, we will test 

this empirically through the analysis of impacts on student mobility based on the student sample in 

fall 2014 (following randomization) and spring 2016 (at the conclusion of the intervention).

The main impact analysis based upon spring 2016 test scores will estimate an impact model in 

which the overall impact estimate is based on treatment-control differences in the outcome of 

interest within each stratum (matched school pair). Although a simple treatment-control difference

in mean outcomes will yield an unbiased estimate of the impact of DDI, the precision of estimates 

can be improved by controlling for baseline characteristics that may influence the outcomes of 

interest but are not related to the treatment itself. For all the student outcomes, we will control for 

baseline student and school covariates. If available from state or district records, specific studentlevel covariates will include prior years’ (spring 2015 and, if available, spring 2014) math and reading 

test scores (in z-score units), prior years’ student attendance, prior years’ student suspensions, 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

16

gender, race/ethnicity, eligibility for free or reduced-price lunch, English language learner status, and 

special education status.10 We also anticipate including school-level aggregates of these variables and 

other relevant school characteristics in cases where individual student-level data are not available.

Accordingly, we will estimate student impacts using the following model:

(1)  =  +  + 



 +  + 

where yijk is the outcome of individual student i in school j within stratum k; αkis a vector of 

stratum (matched pair) indicators (fixed effects) included to control for differences across strata in 

average student, teacher, principal and school characteristics; Tjk is a treatment indicator that equals 

one if the school was assigned to DDI and zero otherwise; Xijk is a vector of baseline individual

student characteristics; ujk is a school-specific random error term; εijk is an individual-level random 

error term; and , γ, and δ are parameters to be estimated.

The estimate of  represents the overall impact of DDI on the student outcome of interest. We 

will estimate the model with ordinary least squares (OLS) using standard errors that account for 

school-level clustering.

We will consider student achievement in reading and math to be separate domains, across 

which the impact of data-driven instruction might differ. Given that assessments differ across state, 

grade level, and subject area, we will standardize the raw achievement scale scores by converting 

them to z-scores. We will calculate the z-scores by subtracting the state mean score from the raw 

scale score and dividing by the standard deviation of the state scores.

We will also estimate impacts on other student-level outcomes, such as attendance and 

suspensions, if possible. We will request student-level data on these outcomes from districts or 

states, but if the data are not available or are unreliable, we will estimate impacts on these outcomes 

measured at the school (or, if available, grade) level. Impacts on a school-level outcome yjkdcan be 

estimated using model (1) above but including school-level averages of individual-level covariates 

rather than the individual-level covariates themselves.

The estimation of overall impacts on student achievement may mask differences in impacts 

across subgroups of students. For example, DDI may prove more or less effective at boosting 

student achievement for students with different baseline characteristics; it may, for example, raise 

achievement among lower performing students to a greater degree than it does among higher 

 

10 Since the intervention begins in December 2014, students’ spring 2015 will not be measured during a true 

baseline period. However, we argue that it is unlikely that the intervention will have an effect on sample members’ spring 

2015 scores, for two reasons. First, sample members in fourth grade during 2015–2016 will typically have been in third 

grade in spring 2015, and the intervention does not cover this grade level. Second, intervention activities during the first 

several months of the intervention period (that is, through the early months of 2015) will focus on providing support to 

schools in setting up the structures necessary to implement comprehensive DDI and on providing training to teachers in 

the overall intervention approach and in the initial steps of understanding student data. Thus, we argue that these early 

activities would be unlikely to lead to instructional changes that affect the spring 2015 achievement levels of the study’s 

fifth grade cohort. However, we will address this possibility by examining the sensitivity of the results to a model that 

uses spring 2014 achievement levels for this cohort, as well as to a model that excludes baseline achievement altogether. 

We will also estimate a model in which the spring 2015 test score is the dependent variable. If our premise that the 

intervention was not sufficiently implemented to affect student achievement by this time, we would expect to see no 

significant treatment-control difference in spring 2015 test scores.
40166 OMB Supporting Statement: Part B Mathematica Policy Research

17

performing students. We will provide a subgroup analysis of the impacts of DDI for student groups 

based on their level of baseline achievement, focusing on impacts for particularly low-achieving 

students as well as for students at moderate to high baseline achievement levels. To estimate impacts 

for student subgroups, we will create a version of the model that interacts a subgroup indicator with 

the treatment indicator; the coefficient on the subgroup-treatment indicator interaction term will 

represent the impact estimate for the subgroup. 

Teacher and principal intermediate outcomes. The core set of professional development and technical 

assistance inputs under DDI are intended to help teachers and principals use data to improve 

instruction, which in turn would lead students to realize higher achievement gains. We will use 

responses from the teacher survey, teacher logs, and the principal survey to examine these 

intermediate outcomes. Teachers and principals offer different vantage points from which to assess 

the extent to which schools engage in data use activities. For example, principals may be able to 

provide detailed information on school leadership activities, while teachers may be able to provide 

detailed information on the frequency and content of teacher collaboration activities. The surveys 

will provide useful information on the frequency of activities over an extended period of time (such 

as how often a teacher attended professional development during the school year), while teacher 

logs will provide a one-day snapshot of teacher activities that occur relatively frequently (such as 

lesson planning and collaboration based on analysis of data). 

Table 2 lists the measures and their sources that will be used to estimate impacts on 

intermediate outcomes. The first two sets of items listed in the table measure intermediate 

outcomes related to teachers’ access to and use of data to guide instruction. The second two sets of 

items capture information on teachers’ instructional strategies. 

Table 2. Teacher and Principal Intermediate Outcome Measures, by Source

Teacher Survey Teacher Logs Principal Survey

Access to student-level data

Access to interim assessment results X X

Access to summative assessment results X X

Access to student background characteristics, attendance, and 

school behavior information X X

Barriers to data use (usable format, technology, tools) X X

Use of student-level data

Frequency of use by type (summative assessment results, interim 

assessments, formative assessments, samples of student work, 

student characteristics)

X X X

Purposes of data use (understand student needs, set learning 

goals, monitor progress toward goals, differentiate instruction, 

revise lesson plans)

X X

Understanding of instructional changes to make based on data X X

Differentiated Instruction

Placing students in small groups based on student data X

Providing small group instruction X X

Providing individualized instruction X X

Identifying and referring students in need of pull-out services or 

other intensive interventions X X

Changing instructional group assignments of students based on 

student data X X
40166 OMB Supporting Statement: Part B Mathematica Policy Research

18

Whole-Class Instruction

Providing additional instruction in areas where students are 

struggling X X

Identifying evidence-based instructional changes to help address 

students’ needs X

Using new instructional strategies to teach challenging concepts 

to students X

To assess the impact of DDI on teachers’ use of data and instructional strategies, we will 

compare outcomes for teachers in treatment schools to those of teachers in control schools. Our

impact model for teacher outcomes will take a similar approach to that taken in the student 

outcomes model, calculating treatment-control differences among teachers. For all teacher 

outcomes, we will control for teacher covariates derived from the survey. Teacher covariates will 

include years of teaching experience, gender, race/ethnicity, teacher certification, and an indicator 

for a master’s degree. 

Similar to our approach in the student model, we will estimate the model with ordinary least 

squares (OLS) using standard errors that account for school-level clustering, and we will compute 

the overall average impact of DDI by taking a weighted average of the coefficients on the treatment 

indicators. Each district-specific impact will be weighted by the number of study schools in each 

district. 

Different types of teachers may be differentially affected by the implementation of DDI. For 

example, less experienced teachers may be more (or less) at ease in using data to inform instruction 

than more experienced teachers. They also may be able to benefit to a greater extent from the 

information on student performance provided by student data, since they will be less able to rely on 

experience to understand their students’ needs. The study will also examine impacts on teacher 

outcomes separately for subgroups of teachers defined by their level of experience. Similar to the 

approach used in the student subgroup analysis, we will estimate subgroup impacts by creating a 

version of the model that interacts a subgroup indicator with the treatment indicator. 

Our approach to analyzing principal outcomes will similarly compare principal survey responses 

regarding the structures and professional development activities in treatment and control schools. 

However, while the teacher and student models will be estimated at the classroom and student 

levels, respectively, the principal model will be estimated at the school level. We will adapt the 

student-level model presented in equation (1) to estimate impacts on principal-reported measures. 

For all principal survey analyses, we will control for principal covariates, such as years of experience, 

gender, race/ethnicity, and an indicator for a master’s degree. 

Nonexperimental analysis. Contextual factors, such as student characteristics or principal and 

teacher background characteristics, may aid in the interpretation of the impact of DDI in treatment 

schools relative to control schools. We will therefore examine how school contextual factors are 

related to impacts. Examples of contextual factors include:

 School characteristics. School math and ELA proficiency measured at baseline, the 

percentage of students eligible for free or reduced-price meals, the percentage that are 

English language learners, the percentage in special education programs, and racial/ethnic 

composition of the school. 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

19

 Teacher and principal characteristics. Education, experience, and background 

characteristics. 

The study will use descriptive analyses and regression analyses to examine how impacts are 

related to school contextual factors. We will use the descriptive analysis to identify conditions and 

practices that are candidates for regression analyses of impacts.

Correlational analyses of impacts and implementation fidelity. The potential impact of 

DDI on student achievement may differ within treatment schools based on aspects of their fidelity 

of implementation of DDI. To explore how school characteristics and other contextual factors may 

influence student impacts, we will conduct a correlational analysis that examines the relationship 

between estimated impacts of DDI and key features of treatment schools (or of matched pairs of 

treatment and control schools when data are available for the control schools).

For example, we may examine whether the qualifications of the data coaches hired at each 

treatment school are related to impacts on student achievement. Because data coaches play a central 

role in supporting the DDI intervention at each school, the prior skills and knowledge they bring to 

the work may influence the degree to which teachers are supported in analyzing student data and 

using them to improve instruction, which may in turn influence student outcomes. Understanding 

the degree to which the success of the DDI intervention hinges upon the qualifications of the data 

coach may prove helpful in developing effective DDI interventions in the future. At the time they 

were interviewed for the position in fall 2014, each data coach completed a data proficiency exercise 

developed by Focus on Results to assess their initial skills in analyzing student assessment data. As 

above, we will correlate the data proficiency score of each coach with the estimated impact of the 

coach’s school.

Because schools cannot be randomly assigned to specific DDI contexts, this correlational 

analysis will be nonexperimental. We will stress that any significant relationships between impacts 

and DDI features or contextual factors might not be causal and might reflect the influence of other 

unobserved factors.

Degree of Accuracy Needed. In Table 3 we present the minimum detectable effect sizes 

(MDEs) on student achievement and teacher practices for the sample size of 104 schools (52

treatment and 52 control), 5 teachers per school, and 112 students in grades 4 and 5 per school. The 

MDEs incorporate conservative assumptions about design effects due to the clustering of students 

or teachers in schools and precision gains from regression adjustments and stratified random 

assignment. These assumptions (listed in the notes of Table 3) are based on estimates from recent 

large-scale studies in education with school-level random assignment. Under these assumptions, we 

will be able to detect an impact on student achievement of 0.12 standard deviations. With this 

sample size, we also will be able to detect an impact on teacher practices of 0.33 standard deviations. 

Minimum detectable effect sizes for student and school subgroups are also shown in the table.

Table 3. Minimum Detectable Effects by Sample Size

Sample Size Minimum Detectable Effect (MDE)

Number of 

Schools

Number of 

Students per 

School

Number of 

Teachers per 

School

Number of 

Principals 

per School

Student 

Outcome

Teacher 

Outcomes

Principal 

Outcomes

Balanced 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

20

Design

104 (52 T, 52 C) 112 5 1 0.12 0.33 0.59

Student/teacher 

subgroup 

104 (52 T, 52 C) 56 3 1 0.14 0.39 --

School 

Subgroup

52 (26 T, 26 C) 112 5 1 0.18 0.48 0.90

Note: The MDEs were calculated assuming: (1) a stratified random assignment design; (2) a twotailed test; (3) a 5 percent significance level and 80 percent level of power; (4) a school-level 

intraclass correlation of 0.15; (5) a response rate to the teacher survey of 85 percent; and (6) 

a reduction in variance of 40 percent at the student level, 70 percent at the school level from 

the inclusion of covariates in the student outcome models and 10 percent reduction in 

variance at the teacher and principal levels, and 10 percent at the school level from the 

inclusion of covariates in the teacher outcome models. T= treatment; C= control.

3. Methods to Maximize Response Rates and Deal with Nonresponse

There are multiple strategies to maximize response while minimizing burden on respondents

and the following techniques are major contributions to a high completion rate: establishing positive 

relationships with respondents and school and district staff; sending letters prior to the surveys; and 

establishing efficient and flexible scheduling. We will include a statement on confidentiality and data 

collection requirements (Education Sciences Reform Act of 2002, Title I, Part E, Section 183) in all 

letters, data collection instruments, and study information sheets. In cases when the data collection 

activity is voluntary (the teacher survey, teacher logs, and principal surveys), we will include a 

statement indicating that participation is voluntary, yet emphasize the importance of their response 

for the study findings and highlight the incentive for completion.

Teacher and Principal Assignment Data. We will collect teacher assignment information 

using a form that requires minimal effort to complete (Appendix A). The study team will attempt to 

identify principals at study schools from public sources (such as school websites). If that information 

is not publicly available, we will request schools to provide principal assignment information. 

Schools can choose to provide this information by posting an electronic file to our secure website or 

by sending a hard copy via a prepaid Federal Express packet provided. We will work with school 

staff if they prefer to provide the information by another method (such as by phone). We will be 

courteous but persistent in our follow-up with participants who do not respond quickly to our 

attempts to reach them. Based on our experience colleting this type of information, we expect a 100 

percent response rate. 

Teacher Survey and Teacher Logs. Based on Mathematica’s experience surveying teachers 

on other studies, we expect at least an 85 percent response rate for the teacher survey, and an 80 

percent response rate for the teacher logs. To ensure a high response rate, we will send teachers a 

letter that will describe the study and provide instructions to complete the survey online at their 

convenience (Appendix B). In the case of the teacher logs, we will request that teachers complete 

them within a more limited time period so that they will be able to accurately recall the day’s 

activities. We will send out reminder emails and make reminder telephone calls to teachers who do 

not respond within two to three weeks of receiving the survey. For the logs, we will send the 

reminder emails and make the reminder calls during the shorter time period in which we hope to get 

a response. If necessary, Mathematica staff will follow up with nonrespondents and administer the 

survey over the telephone at the teachers’ convenience, or provide teachers with a hard-copy survey 
40166 OMB Supporting Statement: Part B Mathematica Policy Research

21

they can complete. Experienced interviewers will be recruited and extensively trained on data 

collection procedures, including methods for promoting cooperation among school staff. 

Interviewers especially skilled at encouraging cooperation will be available to persuade reluctant 

teachers to participate. To compensate teachers for their time to complete the survey and to increase 

the response rate, we propose offering teachers $30 for each completed survey. For each completed 

log, we propose offering teachers $15.

Principal Survey. To ensure a high response rate to the principal survey (Appendix D), we will 

draw on our collegial relationship established with principals during the recruitment effort. We will 

provide reminder materials about the study that explain the importance of the survey as an 

opportunity for principals to provide their perspective on DDI practices in their school. As with the 

teacher survey, we will send out reminder emails and make reminder calls to principals who do not 

respond within two to three weeks of receiving the survey. Principals will also be given the option to 

complete the survey by telephone, if desired. To compensate principals for their time to complete 

the survey and to increase the response rate, we also propose offering principals $30 for each 

completed survey.

Student Records Data. To minimize burden on the district and maximize the likelihood of 

obtaining the data, during the initial phases of recruiting we asked each district how administrative 

records data are stored, how we can obtain permission for collecting this information, and the 

contact person we should work with to obtain the data (Appendix E). We will accept electronic data 

file or hard copy lists. Federal rules permit ED and its designated agents to collect student 

demographic and existing achievement data from schools and districts without prior parental or 

student consent (Family Educational and Rights and Privacy Act (20 U.S.C. 1232g; 34 CFR Part 

99)). To maximize the response rate and minimize burden on schools and parents, we will follow 

these federal rules, and we plan to compensate districts for the burden of their time spent providing 

student records. In addition, we will be courteous but persistent in our follow-up with participants 

who do not respond quickly to our attempts to reach them. We assume that we will be able to 

obtain records for 100 percent of the participating students.

4. Pilot Testing

We will pilot the survey instruments and logs for features such as clarity, accuracy, length, flow, 

and wording. Trained quality control staff will check responses for completeness and reasonableness 

as soon as hard-copy principal surveys are received and they will follow up with respondents if 

problems are identified. The web-based teacher surveys and logs will not allow respondents to enter 

out-of-range or inconsistent responses, and data entry programs also will check for inconsistencies.

We will pilot the teacher survey and teacher log with nine or fewer respondents who are either 

teaching or have taught fourth or fifth grade math or reading. We also will pilot the principal survey 

with nine or fewer respondents who are or were school administrators. We will monitor the time it 

takes to complete the surveys and logs, to ensure the final versions can be completed within the 

allotted time. We will ask respondents to identify any questions that they found problematic and to 

provide general feedback about their experience.

5. Individuals Consulted on Statistical Aspects of the Design

The following individuals were consulted on the statistical aspects of the study:
40166 OMB Supporting Statement: Part B Mathematica Policy Research

22

Name Title Telephone Number

Phil Gleason Senior Fellow, Mathematica 202-264-3443

Alison Wellington Senior Researcher, Mathematica 202-484-4696

Allison McKie Senior Researcher, Mathematica 202-484-4681

Hanley Chiang Senior Researcher, Mathematica 617-674-8374

Elias Walsh Researcher, Mathematica 202-554-7516

The following individuals will be responsible for the data collection and analysis:

Name Title Telephone Number

Phil Gleason Senior Fellow, Mathematica 202-264-3443

Sheila Heaviside Associate Director of Survey Research,

Mathematica

202-484-3096

Irma Perez-Johnson Senior Researcher, Mathematica 609-275-2339

Tim Bruuresema Survey Researcher, Mathematica 202-484-3097

Fran O’Reilly Principal Researcher, EBERE 617-792-0422

Kim Dadisman Associate, Abt Associates 617-520-3040
40166 OMB Supporting Statement: Part B Mathematica Policy Research

23

REFERENCES

Imai, K., G. King, and C. Nall. “The Essential Role of Pair Matching in Cluster Randomized 

Experiments, with Application to the Mexican Universal Health Insurance Evaluation.” Statistical 

Science, vol. 24, no. 1, 2009, pp. 29–53. 
www.mathematica-mpr.com

Improving public well-being by conducting high-quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research