ADDRESSING 

EARLY WARNING 

INDICATORS 

Interim Impact Findings from 

the Investing in Innovation (i3) 

Evaluation of DIPLOMAS NOW

William Corrin

Susan Sepanik

Rachel Rosen

Andrea Shane

June 2016

Addressing Early Warning Indicators

Interim Impact Findings from the Investing in Innovation (i3) 

Evaluation of Diplomas Now

William Corrin

Susan Sepanik

Rachel Rosen

Andrea Shane

June 2016
Funding for this report came from the U.S. Department of Education under its Investing in 

Innovation (i3) initiative. The i3 grant called for Diplomas Now to expand its whole-school 

reform initiative and for MDRC and ICF International to conduct an independent evaluation of the 

implementation and impacts of the reform. Matching grants were also provided by the PepsiCo 

Foundation and Ballmer Family Giving.

Dissemination of MDRC publications is supported by the following funders that help finance 

MDRC’s public policy outreach and expanding efforts to communicate the results and implications of our work to policymakers, practitioners, and others: The Annie E. Casey Foundation, 

Charles and Lynn Schusterman Family Foundation, The Edna McConnell Clark Foundation, Ford 

Foundation, The George Gund Foundation, Daniel and Corinne Goldman, The Harry and Jeanette 

Weinberg Foundation, Inc., The JBP Foundation, The Joyce Foundation, The Kresge Foundation, 

Laura and John Arnold Foundation, Sandler Foundation, and The Starr Foundation.

In addition, earnings from the MDRC Endowment help sustain our dissemination efforts. Contributors to the MDRC Endowment include Alcoa Foundation, The Ambrose Monell Foundation, 

Anheuser-Busch Foundation, Bristol-Myers Squibb Foundation, Charles Stewart Mott Foundation, Ford Foundation, The George Gund Foundation, The Grable Foundation, The Lizabeth and 

Frank Newman Charitable Foundation, The New York Times Company Foundation, Jan Nicholson, Paul H. O’Neill Charitable Foundation, John S. Reed, Sandler Foundation, and The Stupski 

Family Fund, as well as other individual contributors.

The findings and conclusions in this report do not necessarily represent the official positions or 

policies of the funders. 

For information about MDRC and copies of our publications, see our website: www.mdrc.org. 

Copyright © 2016 by MDRC®. All rights reserved.
iii

Overview

Too many students in high-poverty urban communities drop out of high school, and too few graduate prepared for college and careers. Three national organizations — Talent Development Secondary, City Year, and Communities In Schools — partnered to form Diplomas Now in an effort to turn

those numbers around. Supported by funds from a U.S. Department of Education 2010 Investing in 

Innovation (i3) validation grant and private sources, Diplomas Now teams have been implementing 

their data-driven, tiered intervention model in urban secondary schools across the nation. The model 

combines a comprehensive school reform strategy, intended to transform the academic experience of 

all students, with more targeted interventions for students who have “early warning indicators” related to attendance, behavior, and course performance. By identifying students at risk of dropping out 

and providing individual support, Diplomas Now attempts to get struggling students back on a stable 

trajectory toward their diplomas. 

MDRC and ICF International are conducting an independent, experimental evaluation of the impact 

and implementation of Diplomas Now. Sixty-two secondary schools in 11 school districts agreed to 

participate in this study between 2011 and 2013. Thirty-two of these schools were randomly assigned to implement the Diplomas Now model while the other 30 schools were assigned to a control 

group, continuing their existing school programs or implementing other reform strategies of their 

choosing. Two prior evaluation reports focused on the first two years of Diplomas Now implementation. This third report shares interim impact findings for those years, paying particular attention to 

attendance, behavior, and course performance outcomes of students in sixth or ninth grade, their 

transition year into middle or high school — the first-year impacts of a multiyear program. 

•

•

•

•

•

The Diplomas Now model produced a positive and statistically significant impact on the percentage of students with no early warning indicators — students with better than 85 percent attendance, fewer than three days suspended or expelled, and passing grades in both English/language arts and math. Helping students maintain or reach these thresholds is an explicit 

target of Diplomas Now school teams.

Diplomas Now did not have a statistically significant impact on the percentage of students meeting a more stringent threshold suggestive of a more stable educational trajectory: better than 90 

percent attendance, no suspensions or expulsions, and passing all four core subject areas of English/language arts, math, social studies, and science.

Diplomas Now did not produce a significant impact on average attendance, discipline, and 

course passing rates in sixth and ninth grades compared with rates at schools that did not implement the model. These outcomes improved from baseline in both Diplomas Now and comparison schools. 

There were more promising impacts for middle schools than for high schools. In middle 

schools, Diplomas Now had a positive, statistically significant impact on the percentage of 

sixth-graders with no early warning indicators. There were no significant impacts, positive or 

negative, on the attendance, behavior, and course performance outcomes of ninth-graders.

Students at Diplomas Now schools reported participating in more academically focused afterschool activities, and more reported having a positive relationship with an adult at school who is 

not a teacher, than their peers in the comparison schools. Students in both groups of schools reported similar perceptions of school safety and climate, and the Diplomas Now model did not 

have an effect on students’ self-perceptions or school behaviors.

Contents

Overview iii

List of Exhibits vii

Preface ix

Acknowledgments xi 

Executive Summary ES-1 

Chapter

1 1 

1 

1 

2 

6 

8 

12 

2 13 

13 

15 

17 

19 

22 

24 

3 27 

28 

30 

32 

4 

35 

37 

37 

42 

48 

51 

Introduction

Background 

The Diplomas Now Partnership

The Diplomas Now Model 

The National i3 Evaluation of Diplomas Now 

Summary of Key Findings from the First Two Reports 

Orientation to This Report

Study Design and Characteristics of Study Schools

The Random Assignment Design 

Characteristics of Schools and Students in the Study 

Data Sources

Measures

Analytic Sample

Analysis

Early School and Student Outcomes

Findings on Early School Outcomes 

Findings on Early Student Outcomes 

Conclusion 

The Impact of Diplomas Now on Intermediate Outcomes of Student 

Attendance, Behavior, and Course Performance

Impacts on Continuous Measures of the ABC Outcomes 

Impacts on Stability and Early Warning Indicator Thresholds for Attendance, 

Behavior, and Course Performance 

Impacts for More- and Less-Prepared Students

Impacts on ABC Outcomes During the First Year of Implementation

Variation in Impacts Across Schools 

Conclusion 

v 

53 
vi

Appendix

A Samples, Analytic Methods, and Early Outcome Measures 57 

B Supplemental ABC Outcome Findings 81 

References 97

Earlier MDRC Publications on Diplomas Now 99 
vii

List of Exhibits

Table

ES.1 Impacts on Continuous Measures of Attendance, Behavior, and Course 

Performance, Full Sample, Cohort 2 ES-8 

2.1 Baseline Comparison of DN and Non-DN Schools on Attendance, Behavior, 

and Course Performance 16

2.2 Baseline Comparison of DN and Non-DN Schools, Characteristics of Students 

in Cohort 2 Analysis Sample 18

2.3 Description of ABC Outcome Measures 20

3.1 Early School Outcomes, Teacher and Administrator Responses, 

Implementation Year 2 29

3.2 Early School Outcomes, Sixth- and Ninth-Grade Student Responses, 

Implementation Year 1 31

3.3 Early Student Outcomes, Sixth- and Ninth-Grade Student Responses, 

Implementation Year 1 33

4.1 Impacts on Continuous Measures of Attendance, Behavior, and Course 

Performance, Cohort 2 38

4.2 Impacts on Threshold Measures of Attendance, Behavior, and Course 

Performance, Cohort 2 40

4.3 Impacts on Threshold Measures of Attendance, Behavior, and Course 

Performance, Middle Schools, Cohort 2 42

4.4 Impacts on Threshold Measures of Attendance, Behavior, and Course 

Performance, High Schools, Cohort 2 44

4.5 Impacts on Threshold Measures of Attendance, Behavior, and Course 

Performance, Students Above and Below Stability Threshold in Previous Year, 

High Schools, Cohort 2 46

4.6 Impacts on Threshold Measures of Attendance, Behavior, and Course 

Performance, Cohort 1 49

A.1 Number and Percentage of Randomly Assigned Schools in Each Analysis 

Sample 60

A.2 Individual-Level Response Rates for Teacher, Administrator, and Student 

Surveys 63
viii

Table

A.3 Minimum Detectable Effect and Effect Size for Impacts on ABC Outcomes, 

Cohort 2 67

B.1 Impacts on Threshold Measures of Attendance, Behavior, and Course 

Performance, Prior-Year Math and English Proficiency Subgroups, Middle 

Schools, Cohort 2 84

B.2 Impacts on Continuous and Threshold Measures of Attendance, Behavior, and 

Course Performance, Full-Year Sample, Cohort 2 87

B.3 Impacts on Threshold Measures of Attendance, Behavior, and Course 

Performance, Middle Schools and High Schools, Cohort 1 89

B.4 Comparison of Cohort 1 and Cohort 2 Impacts by Random Assignment Block, 

Percentage of Students Above Stability Threshold 92

B.5 Comparison of Cohort 1 and Cohort 2 Impacts by Random Assignment Block, 

Percentage of Students with No Early Warning Indicators 93

B.6 Impacts on Continuous and Threshold Measures of Attendance, Behavior, and 

Course Performance, Accounting for Variation in School Grade Configuration, 

Middle Schools, Cohort 2 95

Figure

ES.1 Percentage of Students At or Above Threshold Measures, by DN and Non-DN 

Schools, Cohort 2 ES-7 

ES.2 Diplomas Now Impacts on ABC Composite Measures, Middle Schools and 

High Schools, Cohort 2 ES-10

1.1 Diplomas Now Logic Model 4 

4.1 Impacts on Percentage of Students Above Stability Threshold, by Random 

Assignment Block, Cohort 2 52

4.2 Impacts on Percentage of Students with No Early Warning Indicators, by 

Random Assignment Block, Cohort 2 54

A.1 Construction of the Sample of Administrators for the Analyses of Early School 

Outcomes 64

A.2 Construction of the Sample of Teachers for the Analyses of Early School Outcomes 65

Box

1.1 The Diplomas Now Partners and Their Roles 3 

2.1 Understanding the Impact Tables in This Report 25
ix

Preface

American education has entered its next phase with the signing of the Every Student Succeeds 

Act (ESSA) by President Obama in December 2015. In a change from the prior education law 

(No Child Left Behind), states rather than the federal government now play the lead role in the 

identification of struggling schools and the kinds of interventions that can help them improve. 

Stressing accountability, ESSA encourages states to bring evidence-based practices to bear in 

this endeavor. The federal Investing in Innovation (i3) evaluation of Diplomas Now is a notable 

effort to collect such evidence.

Diplomas Now is a secondary school reform model that aims to change school 

structures and practices in ways that will affect students’ engagement and persistence, starting 

with their transition year into middle school (sixth grade) or high school (ninth grade). The first 

two reports from this evaluation focused on the implementation of the Diplomas Now model. 

This third report provides a valuable interim look at the reform model’s impact on students’ 

attendance, behavior, and course performance — predictors of whether students graduate or 

drop out — as they begin their middle school or high school journeys. A later report will discuss 

the quality of model implementation across four years and its impact on two key four-year 

outcomes: ninth-grade completion for middle school students and high school graduation. 

The opportunity for rigorous study of a whole school reform, nationally scaled and 

implemented for four or more years, is rare, and the signing of ESSA only increases its 

relevance. The information generated from the full scope of this evaluation will contribute 

meaningfully to the existing knowledge base of how best to support underserved students and 

communities, and can help states and their local education agencies make more thoroughly 

informed decisions about how they can improve the secondary schools that are most in need. 

Gordon L. Berlin 

President

xi

Acknowledgments

This report could not have been accomplished without the efforts of a great many people. The 

study has benefited especially from the time, energy, and commitment put forth by the staff at

the participating school districts who provided administrative records data; the school staff and 

the Diplomas Now staff working within some of the schools who helped organize survey 

administration; and the school administrators and teachers who took time out of their busy 

schedules to participate in surveys. 

The assistance and cooperation of the Diplomas Now staff at the national level has also 

been invaluable. The Diplomas Now Implementation Support Team informed the creation of 

the survey protocols. The Diplomas Now National Executive Team has answered our many 

requests for program information throughout the project and provided feedback on earlier drafts 

of this report.

Mike Puma provided technical assistance to the evaluation team and ensured that the 

team understood and met the standards set for Investing in Innovation (i3) evaluations.

Several ICF International and MDRC staff members and consultants worked directly 

with schools and school districts participating in the study. Rob Ivry, Stephanie Safran, and 

Jacklyn Willard worked with Diplomas Now staff members to recruit districts and schools to 

participate in this project. Many dedicated colleagues supported survey data collection efforts, 

including Aracelis Gray, Felix Fernandez, Allison Alexander, Ashley Briggs, Nicole Dutch, 

Elyse Goldenberg, Lisa Luo, Caitlin Murphy, Allison Nebbergall, Jackie Rhodes, Kathleen 

Wang, Eleanor Leahy, Rachel Pedraza, Stephanie Safran, Kelly Walton, and Seth Muzzy. 

Many MDRC staff also supported the data processing, analysis, and report writing. 

Kristin Porter helped design the analysis plan, while Marie-Andrée Somers and Pei Zhu offered 

technical advice throughout the process. Leslyn Hall supported the survey design. Nicole 

Clabaugh, Katie Lashko, David Roy, Cathy Corbin, Rebecca Coven, Micah DeLaurentis, Zeest 

Haider, Heilyn Paulino, and Sara Staszak provided programming and analysis support. Katie 

Lashko also worked closely with school districts to collect school records data. Nick Commins 

supported the creation of some of the report exhibits. Larissa Saco and Nicole Clabaugh 

coordinated the report production and moved the report to completion. Rekha Balu, Fred 

Doolittle, Rob Ivry, Joshua Malbin, and Marie-Andrée Somers carefully reviewed earlier drafts 

of the report and offered helpful critiques throughout the writing process. Jennie Kaufman 

edited the full report, and Stephanie Cowell and Carolyn Thomas prepared the report for 

publication. 

The Authors

ES-1 

Executive Summary

Although the national high school graduation rate has increased over the past decade, one in 

five students still do not complete high school in four years.1 Among low-income students, almost 30 percent fail to graduate on time.2 Compared with high school graduates, dropouts are 

more likely to earn less money, live in poverty, suffer from poor health, be incarcerated, or be 

dependent on social services.3

Diplomas Now is a partnership of three national organizations — Talent Development 

Secondary, City Year, and Communities In Schools — collaborating in an effort to transform 

urban secondary schools so that fewer students drop out and more graduate ready for postsecondary education and work. The Diplomas Now model is a comprehensive multiyear approach 

to whole-school reform that includes structural change, instructional materials and curricula, 

teacher and administrator coaching and support, and an early warning indicator and intervention 

system to identify and support students falling off track for graduation. The program brings additional human resources into the school both to bolster implementation of the model and to 

provide direct assistance to students. With the goal of a continuous system of support through 

secondary school, the model seeks to help more students graduate by improving their attendance, behavior, and course performance, particularly in English/language arts and math, during 

the middle grades and high school.

Acting as a representative for the partnership, Johns Hopkins University, home to Talent Development Secondary, was awarded an Investing in Innovation (i3) validation grant by 

the U.S. Department of Education in 2010 to support the expansion of Diplomas Now from a 

few schools to more than 30 middle and high schools in more than 10 school districts. The grant 

funds also support a rigorous random assignment evaluation of the Diplomas Now model, led 

by MDRC. 

This report discusses the early impacts of the Diplomas Now model on student and 

school outcomes at the end of the first and second years of model implementation. It focuses in 

particular on students during sixth and ninth grades, critical transition years into middle and

high school. Accordingly, this report presents the first-year impacts of a multiyear program. 

 1

Richard J. Murnane, “U.S. High School Graduation Rates: Patterns and Explanations,” Journal of Economic Literature 51, 2 (2013): 370-422. 2

Robert Balfanz, John M. Bridgeland, Joanna Hornig Fox, Jennifer L. DePaoli, Erin S. Ingram, and Mary 

Maushard, Building a Grad Nation: Progress and Challenge in Ending the High School Dropout Epidemic 

(Washington, DC: Civic Enterprises, 2014). 3

Kristin Anderson Moore, Making the Grades: Assessing the Evidence for Integrated Student Supports

(Bethesda, MD: Child Trends, 2014).
ES-2 

The Diplomas Now Model

The Diplomas Now partnership works with schools to ensure that students are getting the support they need to (1) get to school and to class, (2) behave in ways that facilitate learning, and 

(3) keep up with the lessons being taught. In other words, the pathway to student success in 

schools using the model is linked to attendance, behavior, and course performance: the “ABCs” 

that predict whether students graduate or drop out.4 The Diplomas Now model is a multidimensional system of organizational and instructional reforms and targeted student support services 

organized under the following Four Pillars.5

Pillar I. Teacher Teams and Small Learning Communities 

Diplomas Now collaborates with school leaders to reorganize schools so that small 

groups of teachers work consistently with the same population of students. Not only does this 

allow teams of teachers to work together, the better to teach and support their students; it also 

creates a sense of community among the students. These teacher teams and small learning 

communities function best when teachers have a chance to collaborate within the daily schedule 

and when classes are long enough to cover material in depth and keep up the pace of instruction.

Pillar II. Curriculum and Instruction with Professional Development

This pillar is focused on teaching and learning, and on giving teachers the training and 

resources they need to deliver strong lessons. Through professional development that includes 

an intensive peer coaching system for math and English/language arts teachers, teachers have an 

opportunity to sharpen their pedagogy. Diplomas Now also offers curricular materials aligned 

with college- and career-ready standards and ensures that schools offer accelerated remediation 

courses for struggling students so that all students can meet their potential.

Pillar III. Tiered Student Supports 

The Diplomas Now partners collaborate to help schools provide the right services to the 

right students at the right time and at the right level of intensity.6 To do so, they use early warn-

 4

Robert Balfanz, Liza Herzog, and Douglas J. Mac Iver, “Preventing Student Disengagement and Keeping 

Students on the Graduation Path in Urban Middle-Grade Schools: Early Identification and Effective Interventions,” Educational Psychologist 42, 4 (2007), 223-235. 5

For more information on the Diplomas Now model, see William Corrin, Susan Sepanik, Aracelis Gray, 

Felix Fernandez, Ashley Briggs, and Kathleen K. Wang, Laying Tracks to Graduation: The First Year of Implementing Diplomas Now (New York: MDRC, 2014). 6

“Tiered Student Supports” refers to different levels of support offered across the school or to individual 

students based on need. Tier I interventions support the entire school. Tier II interventions are individual support services offered to students identified as falling off track. Many of these interventions are provided by City 

Year AmeriCorps members who serve as mentors, tutors, and role models. Tier III interventions are for stu-
ES-3 

ing indicators to identify students who require different types of support. Teachers, administrators, and Diplomas Now staff members meet regularly to review students’ attendance rates, disciplinary referrals or suspensions, and course performance and to plan interventions for students

in need of support. 

Pillar IV. Can-Do Culture and Climate 

School reform is difficult, and school staff members often have much to do when they 

are asked to effect change. Diplomas Now brings at least a dozen staff members to a school to 

help coordinate the transformation, introduce new practices and structures, provide training and 

support to school staff members, provide additional services to students, and engage with families and community organizations. Providing resources to assist the school’s staff helps foster a 

culture and climate in which it feels possible to improve the school.

The National i3 Evaluation of Diplomas Now 

In total, 62 schools (33 middle schools and 29 high schools) from 11 large urban school districts 

across the country were recruited to participate in the study starting in either the 2011-2012 or 

the 2012-2013 school year. 7 By design, Diplomas Now works in high-needs schools. The participating schools, all eligible for Title I funds, 8 serve large populations of low-income and minority students. Thirty-two of the participating secondary schools were randomly assigned to

implement the Diplomas Now model (DN schools), and 30 were assigned to continue with 

“business as usual” (non-DN schools), either maintaining their existing practices and structures 

or pursuing other types of school reform.9 This random assignment design is often referred to

as the “gold standard” in evaluation because the schools are all similar at the beginning of the 

study, and the decision about which schools will implement the program is not related to any 

preexisting characteristics of the schools. Therefore, any differences between the DN and nonDN schools that emerge after random assignment can be attributed to the program rather than to 

school characteristics; that is, Diplomas Now caused the observed differences. 

 

dents at the highest risk of dropping out and are generally coordinated by a case manager from Communities In 

Schools. 7

Five of the school districts are among the 20 largest in the country, and all but 1 are among the 100 largest. Chris Plotts and Jennifer Sable, Characteristics of the 100 Largest Public Elementary and Secondary 

School Districts in the United States: 2007-08, NCES 2010-349 (Washington, DC: National Center for Education Statistics, U.S. Department of Education, 2010). 8

Title I funds from the U.S. Department of Education go to schools with high numbers or high percentages 

ofstudents from low-income families. 9

Two middle schools are not included in the analyses in either the first or second year due to issues with 

grade configuration. Two more middle schools are not included in the second-year analyses because one 

school closed and one school stopped serving sixth grade.
ES-4 

The study’s experimental design makes it possible for the evaluation to assess the impact of Diplomas Now. This third report focuses on the early impacts of Diplomas Now on 

students’ attendance, behavior, and course performance measures (the ABC outcomes), separately and in combination, during their first year in middle school or high school over the 

course of the first two years that the model was implemented in participating schools.

10 Does

the implementation of Diplomas Now have an impact on how many students are on a path to 

high school graduation by the end of their first year of middle school or high school? During 

that first year, what difference does Diplomas Now make for attendance rates, suspensions and 

expulsions, and successful course completion? This report also discusses the impact of Diplomas Now on possible precursors to the ABC outcomes, such as the climate of the 

school, support from parents and the community, and students’ attitudes and relationships. 

Early School and Student Outcomes

The Diplomas Now model is hypothesized to achieve its intermediate goals of improving attendance, behavior, and course performance, especially in English/language arts and math, 

through several mediating pathways. These early precursor outcomes include measures of positive school climate, the addition of academic after-school activities, and increased parent and 

community support, along with measures of student attitudes and behaviors, including selfconfidence, engagement and effort in school, study habits, and relationships with adults and 

peers. Administrator, teacher, and student survey items were used to measure these outcomes. 

Analyses of Diplomas Now’s impact on these outcomes found the following:

•

•

The Diplomas Now model had positive and statistically significant impacts 

on teachers’ perceptions of school climate during the second year of implementation. There were no other statistically significant impacts on early 

school outcomes as reported by administrators and teachers, but the findings 

tend to point in a positive direction.

Students at DN schools reported participating more in academically focused 

after-school activities than their peers at non-DN schools. Students at both 

groups of schools reported similar perceptions of safety, climate, and behavioral issues at school during the first year of implementation.11 

 10The primary analyses for this report focus on the second year of implementation, when the model was 

more mature. 11Sixth- and ninth-grade students were surveyed in the spring of the first year of implementation, so all 

early school and student outcomes coming from student survey items represent the first rather than the second 

year of implementation.
ES-5 

•

•

The Diplomas Now model did not have an effect on students’ selfperceptions and school behaviors as measured by the student survey during 

the first year of implementation. 

Students at DN schools were more likely to report a positive relationship 

with an adult at school who was not a teacher, but the Diplomas Now model 

had no impact on student perceptions of relationships with teachers, administrators, and other students. 

The areas in which Diplomas Now is having some positive effects — teachers’ perceptions of the climate of the school, and students’ participation in after-school activities and

relationships with adults at school — align with specific reforms implemented as part of the 

Diplomas Now model. 

ABC Outcomes

The long-term goal of the Diplomas Now model is to increase student graduation rates and college readiness by improving students’ success on the indicators that past research has suggested 

are connected to graduation: the ABC outcomes of attendance, behavior, and course performance.12 Thus far, the study team has been able to explore early impacts for sixth- and ninthgrade students making the transition into Diplomas Now secondary schools during the first two 

years of model implementation in those schools. The reported impacts of the model on student 

outcomes are based on the first year of students’ experience with this multiyear intervention. 

The Diplomas Now model includes structures and practices intended to help students 

stay above or move above two ABC outcome thresholds: The first serves as an intervention 

standard, below which the model targets students for additional support, and the second indicates a more secure course to graduation. Specifically, the first identifies students with “early 

warning indicators” of being at risk of not progressing successfully to high school graduation: 

daily attendance of 85 percent or less, suspensions or expulsions for a total of three or more 

days, and failing grades in English/language arts or math classes. Diplomas Now staff members 

work with school staff members in DN schools in an effort to increase the number of students 

without early warning indicators. Over time, Diplomas Now implementation ideally will increase the number of students who meet the second threshold, indicating a more stable educational trajectory: having better than 90 percent daily attendance, no suspensions or expulsions, 

 12Elaine Marie Allensworth and John Q. Easton, The On-Track Indicator as a Predictor of High School 

Graduation (Chicago: University of Chicago Consortium on Chicago School Research, 2005); Balfanz et al. 

(2007); James J. Kemple, Micha D. Segeritz, and Nickisha Stephenson, “Building On-Track Indicators for 

High School Graduation and College Readiness: Evidence from New York City,” Journal of Education for 

Students Placed at Risk (JESPAR) 18, 1 (2013): 7-28.
ES-6 

and passing grades in all core courses (that is, English/language arts, math, social studies, and 

science). These higher “stability” thresholds represent normative expectations of secondary 

school students: to go to school regularly, to stay out of serious trouble, and to pass their classes.

For students at this level, whole-school programming and instruction are deemed adequate support. Students in between the two levels are monitored by Diplomas Now and school staff 

members and may receive additional attention as needed. 

Impacts for the Full Sample of Schools

The evaluation team analyzed whether implementation of Diplomas Now had an impact on the percentages of students at both stability and early warning thresholds for each of the 

ABC outcomes, as well as the percentages of students at these threshold levels for all three of 

the outcomes combined. The results of these analyses are summarized in Figure ES.1.

•

•

For the combined sample of sixth- and ninth-graders in the second year of 

implementation (that is, the second cohort of students), there was a statistically significant 3.6 percentage point impact of Diplomas Now on the percentage of students with no early warning indicators. That is, DN schools were 

more successful than non-DN schools in helping students stay above or move 

above the early warning thresholds for all three ABC outcomes in combination. The Diplomas Now model did not produce a statistically significant impact on the percentage of students meeting the stability threshold across all 

three ABC outcomes. (See the ABC composite section of Figure ES.1.)

The Diplomas Now model did not produce a statistically significant impact 

on the percentage of students above either the stability or early warning 

thresholds for any of the separate ABC outcome measures.13 (See the attendance, behavior, and course performance sections of Figure ES.1.)

Table ES.1 displays the impacts of the implementation of the Diplomas Now model on continuous measures of attendance, behavior, and course performance outcomes — for example, the 

average percentage of days attended by students rather than the percentage of students meeting 

an attendance threshold — for the combined sample of sixth- and ninth-grade students. These 

analyses provide insight into whether the implementation of the Diplomas Now model, with its 

targeted interventions for struggling students and broader instructional and structural reforms, 

had an impact, on average, across the sixth and ninth grades. 

 13A student with only one early warning indicator will fall below the composite threshold; therefore, although impacts may not be significant when outcomes are measured separately, it is possible that these impacts 

in combination translate to lifting enough students over the composite threshold to be significant cumulatively.
ES-7 

Figure ES.1

 Percentage of Students At or Above Threshold Measures,

by DN and Non-DN Schools, Cohort 2

b

67.5

79.3

21.3

15.2

72.8

82.1

86.4

48.1

59.5

0.8

0.4

2.5

1.7

0.7

2.3

1.0

2.5

3.6

0 10 20 30 40 50 60 70 80 90 100

Above 90 percent

Above 85 percent

Ever suspended

Suspended 3 or more days

Core classes

Math

English

Stability indicators

No early warning indicators

Behaviora

Course

Performanceb

ABC

Composite

Outcome for non-DN schools: Stability indicator

Outcome for non-DN schools: No early warning indicator Estimated impact for DN schools

Attendance

c

d **

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 a

Behavior outcomes include suspensions and expulsions.

 b

Measure indicates the percentage of students who passed all core courses (math, English/language arts, science, 

and social studies courses), all core math courses, and all core English/language arts courses, respectively.

 c

Students with stability indicators attended over 90 percent of days enrolled in the district, were never suspended or 

expelled, and did not fail any core courses (math, English/language arts, science, or social studies) attempted during 

the school year.

 d

Students with no early warning indicators attended over 85 percent of the days enrolled in the district, were 

suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during the 

school year. 
ES-8 

Table ES.1

Impacts on Continuous Measures of

Attendance, Behavior, and Course Performance, Full Sample, Cohort 2 

•

P-Value for

 DN Non-DN Estimated Effect Estimated

Outcome Schools Schools Impact Size Impact

Attendance

Percentage of enrolled days attended 89.7 89.3 0.4 0.03 0.602

Behavior

Percentage of enrolled days suspended or expelled 1.5 1.2 0.3 0.05 0.177

Course performance

Percentage of core courses passeda 86.6 86.6 0.1 0.00 0.924

Sample size 29 29

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 58 study schools, 14,950 nonrepeating sixth- and ninth-grade students are included in the 

analyses. Among the sample, 6,997 students attended Diplomas Now (DN) schools and 7,953 students 

attended non-DN schools. Some students are not included in the analyses of course performance measures 

because data were not available on their grades for specific courses, and an entire DN middle school was 

dropped from these analyses because there were no baseline course data for the students attending that school. 

There are no more than 8 percent missing DN school students and 8 percent missing non-DN school students 

for any of the course performance measures.

 Estimated impacts are based on a two-level model with students nested within schools controlling for 

random assignment block and school- and student-level covariates. The “DN Schools” and “Non-DN Schools” 

columns display regression-adjusted mean outcomes for each group, using the mean covariate values for 

students in the “DN Schools” column as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures.

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

The denominator includes all core courses (math, English/language arts, science, or social studies) each 

student attempted during the school year.

Diplomas Now did not produce a statistically significant impact on students’ 

attendance rates, percentage of days suspended or expelled, or percentage of 

core courses passed.

Thus, after two years of model implementation, the Diplomas Now model has neither increased 

average attendance and course passing rates nor decreased disciplinary days beyond the levels 
ES-9 

achieved in the non-DN schools; both groups of schools experienced similar levels of improvement on these measures. 

Impacts for Middle Schools and High Schools

Middle schools and high schools typically vary in size and structure and serve students 

at different places in their developmental and educational pathways. Furthermore, ninth-graders 

have had more time to accumulate early warning indicators through middle school and may 

reach the ninth grade further off track.14 Given these differences, some aspects of the Diplomas 

Now model vary for middle schools and high schools. Therefore, impacts on student outcomes 

were analyzed for these two groups of schools separately as well as together. 

•

•

The Diplomas Now model had a positive and statistically significant impact 

on the percentage of students with no early warning indicators in middle 

school. (See Figure ES.2.)

The Diplomas Now model had a positive and statistically significant impact 

on the percentage of middle school students who attended over 90 percent of 

enrolled days. 

Although there are no other statistically significant impacts for middle schools, the middle 

school impact estimates across multiple analyses were more often positive than were those for 

high schools. 

• On average, Diplomas Now did not produce statistically significant impacts 

at the high school level compared with the outcomes at non-DN schools.

Overall, results of the evaluation analyses suggest that Diplomas Now may have had a 

more positive early impact in middle schools than in high schools. While DN high schools did 

see improvements across ninth-grade attendance, behavior, course performance, and composite 

measures from the first to the second year of implementation, there were similar gains in the 

non-DN high schools as well.

Impacts for More- and Less-Prepared Students

The Diplomas Now model seeks to help struggling students overcome early warning 

indicators and get on a more stable pathway to graduation as well as to prevent students who are 

on a more stable educational trajectory from slipping off that path. Thus, the evaluation team 

analyzed the impact of Diplomas Now on the outcomes of students entering high school or

 14Ruth Curran Neild, Robert Balfanz, and Liza Herzog, “An Early Warning System,” Educational Leadership 65, 2 (2007): 28-33.
ES-10

 

Figure ES.2

Diplomas Now Impacts on ABC Composite Measures,

Middle and High Schools, Cohort 2

4.8

0.7

0

1

2

3

4

5

6

Stability indicators No early warning indicators

Impact (percentage points) 

Middle school High school

5.5*

2.6

a b

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 aStudents with stability indicators attended over 90 percent of days enrolled in the district, were never suspended 

or expelled, and did not fail any core courses (math, English/language arts, science, or social studies) attempted 

during the school year.

 b

Students with no early warning indicators attended over 85 percent of the days enrolled in the district, were 

suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 

the school year.

middle school according to how prepared they were for the transition, based on whether entering high school students were above or below the stability threshold at the end of eighth grade, 

and on middle school students’ levels of academic proficiency at the end of fifth grade.

15

• Although Diplomas Now had no statistically significant positive impacts at 

the high school level on average, implementation of the model had more success keeping the stable students above the stability threshold than moving the 

less stable students above that threshold. 

15Because elementary school data across participating study districts did not consistently include the disciplinary and course performance metrics used to create the composite ABC indicators, the evaluation team used 

standardized state assessment scores to represent students’ preparedness for middle school. 
ES-11

• The Diplomas Now model did have a statistically significant positive impact 

on the percentage of more-prepared students who passed math in ninth grade.

The students entering high school below the stability threshold — those who had been 

absent at least 10 percent of the time, been suspended or expelled, or failed a core course in 

eighth grade — posed a challenge for both the DN and non-DN schools. For example, in both 

groups of schools only about 30 percent of less-prepared students had no early warning indicators in ninth grade (compared with more than 70 percent of the more-prepared students), and 

only about 20 percent were above the stability threshold (compared with about 60 percent of the 

more-prepared students). This finding reinforces how valuable intervening successfully with atrisk students during the middle grades might be.

• In general, the patterns of impacts were similar for sixth-grade students 

whether or not they were proficient on state English/language arts and math 

assessments at the end of elementary school. 

Next Steps

After two years, implementation of the Diplomas Now secondary school reform model produced a statistically significant, positive impact on the percentage of students with no early 

warning indicators, suggesting that a lower percentage of students were notably off track on the 

pathway to high school graduation at DN schools compared with non-DN schools. The focus of 

the Diplomas Now model, particularly with its early warning system and tiered support, is to 

reduce the percentage of students with early warning indicators, and this finding suggests those 

efforts are starting to make a difference after two years. The model’s effect on the percentage of 

students meeting the “higher bar” indicating a stable pathway to graduation is not statistically 

significant, but it shows progress: The 2.5 percentage point impact for the second cohort of 

sixth- and ninth-grade students was about 5 percentage points higher than the impact for the first 

cohort of students (-2.6 percentage points).16 The increased impact for the second cohort aligns 

with two-year implementation findings from this evaluation. From the first to the second year of 

implementation, DN schools became more different from non-DN schools in terms of their use 

of reform-oriented practices and structures — DN schools showed sustained levels of reform 

implementation, while the levels declined in the non-DN schools.17

 16The impact on the percentage of students with no early warning indicators increased 3.4 percentage 

points from the first cohort to the second cohort. This increase was not statistically significant. 17Susan Sepanik, William Corrin, David Roy, Aracelis Gray, Felix Fernandez, Ashley Briggs, and Kathleen K. Wang, Moving Down the Track: Changing School Practices During the Second Year of Diplomas Now

(New York: MDRC, 2015). 
ES-12

Overall, the early impacts of the model during the second year of implementation were 

stronger for sixth-graders than for ninth-graders. In particular, higher percentages of sixthgraders in DN schools had better than 90 percent attendance and no early warning indicators 

than their peers in non-DN schools. Although ninth-graders in DN high schools did not have 

better outcomes compared with their peers at non-DN high schools, ninth-grade outcomes improved from the first to the second year of implementation at both groups of schools. 

Johns Hopkins University was granted funding from the Office of Innovation and Improvement of the U.S. Department of Education to support an extension of this evaluation that

will make it possible to analyze the impact of Diplomas Now on the longer-term student outcomes that represent the model’s primary target: What is the impact of Diplomas Now on high 

school graduation rates and on the ninth-grade success of students from Diplomas Now middle 

schools? This will allow the study team to see whether the increase in impacts from the first to

the second year continues over a longer period of implementation, and how well implementation is maintained over that period. Furthermore, given how much better the ninth-grade outcomes were for students entering high school above the stability threshold, the promising impact findings for middle school students are worth continued attention. If these impacts are 

maintained, following these students into high school will indicate whether Diplomas Now’s 

encouraging intervention with students in the middle grades yields positive high school outcomes. The study team will also be able to explore the ABC outcomes, including the composite 

measures, of sixth- and ninth-graders in the fourth year of implementation, when the model has 

had even more time to mature in the schools. 

As of the publication of this report, the Diplomas Now partners had managed to maintain model implementation for at least four years in almost all the DN schools. Since wholeschool reform efforts implemented for several years have been shown to have greater impacts 

on student outcomes than those implemented for only a couple of years, analyzing the longerterm effects of this model is important.18 Furthermore, the extension will also allow for some 

exploration of the variation in implementation and outcomes found across schools and students, 

which may support a better understanding of best practices and generate hypotheses that could 

inform practice moving forward.

 18Geoffrey D. Borman, Gina M. Hewes, Laura T. Overman, and Shelly Brown, “Comprehensive School 

Reform and Achievement: A Meta-Analysis,” Review of Educational Research 73, 2 (2003): 125-230.
1 

Chapter 1

Introduction

Background

Although the national high school graduation rate has increased over the past decade, one in 

five students still do not complete high school in four years.1 Among low-income students, almost 30 percent fail to graduate on time.2 Compared with high school graduates, dropouts are 

more likely to live in poverty, earn less money, suffer from poor health, be incarcerated, or be 

dependent on social services.3 Students who face the most serious barriers to earning their diplomas are in the greatest need of intensive academic, social, and other interventions to make it 

through high school, and most dropouts are concentrated in urban high schools in low-income 

communities.

Research has shown that it is possible to predict a student’s likelihood of dropping out 

of high school using indicators of poor attendance, poor behavior, and course failure in mathematics and English/language arts measured as early as middle school.4 These findings suggest 

that programs may have greater success getting more students to graduation if they intervene 

with students who are off track as early as middle school. Moreover, ninth grade is a critical 

year, and researchers have shown that improving student course performance in the ninth grade 

can lead to substantial improvements in graduation rates.5

Among those students who do graduate from high school, many do not graduate ready 

for college and need to take remedial (developmental education) courses: Over 30 percent of 

college undergraduates and about 40 percent of community college students enroll in such 

courses.6 Beyond supporting struggling students to graduation, school improvement efforts need 

to ensure that all students participate and succeed in rigorous curricula that prepare them for 

college and careers.

The Diplomas Now Partnership

The Diplomas Now partnership — created by three national organizations, Talent Development 

Secondary, City Year, and Communities In Schools — works with schools to ensure that stu-

 1

Murnane (2013). 2

Balfanz et al. (2014). 3

Moore (2014). 4

Balfanz, Herzog, and Mac Iver (2007). 5

Roderick, Kelley-Kemple, Johnson, and Beechum (2014). 6

National Center for Education Statistics (2014).
2 

dents are getting the support they need to (1) get to school and to class, (2) behave in ways that 

facilitate learning, and (3) keep up with the lessons being taught. In other words, the pathway to 

student success in Diplomas Now (DN) schools is linked to attendance, behavior, and course 

performance in mathematics and English/language arts: the “ABCs” that predict whether students graduate or drop out. The initiative targets underfunded urban secondary schools with 

many students who are not performing well academically, in communities struggling with poverty. The Diplomas Now partners collaborate to help schools provide the right services to the 

right students at the right time and at the right level of intensity. They do so by offering varying 

levels of support for students with different needs: whole-school restructuring and instructional 

reform to strengthen the educational experience of all students, individual support for students 

showing early signs of falling off track, and case management for students in need of deeper 

interventions. To determine which students need extra support, the model relies on regular monitoring of students’ attendance, behavior, and course performance in mathematics and English/language arts. See Box 1.1 for more on what each of the Diplomas Now partners contributes to the overall model.7

The Diplomas Now Model 

The Diplomas Now model is a multiyear, multidimensional system of organizational and instructional reforms and targeted student support services. The elements of the model are classified as nine “inputs,” some of which represent substantial interventions on their own, such as 

implementing a rigorous curriculum or setting up a tiered intervention system to identify at-risk 

students and tailor interventions to their specific needs. Diplomas Now integrates these interventions into a cohesive model focused on ensuring that all students have a clear and achievable 

path to graduation. Eight of these inputs are implemented in collaboration with school staff 

members and align with the Four Pillars of Diplomas Now, a characterization of the model used 

by Diplomas Now staff members to help them organize their work. The Four Pillars and their 

associated inputs are presented in Figure 1.1. The ninth input (shown on the left-hand side of 

Figure 1.1) consists of professional development for the staff members of the Diplomas Now 

organizations, intended to give them the knowledge and skills they need to help schools implement the Four Pillars.

Pillar I: Teacher Teams and Small Learning Communities

The Diplomas Now partners collaborate with school leaders to reorganize schools so

that small groups of teachers work consistently with the same population of students. These small

 7

For a more detailed description of the partners and their roles, see Corrin et al. (2014), pp. 3-4, and

Sepanik et al. (2015), pp. 2-3.
3 

Box 1.1

The Diplomas Now Partners and Their Roles 

Talent Development Secondary

Talent Development Secondary, based at Johns Hopkins University, provides organizational, instructional, curricular, and data support to schools intended to help all students 

achieve at high levels and prevent them from falling off track. This school-wide effort includes reorganizing students and teachers into small learning communities, providing professional development and coaching to strengthen teacher pedagogy, and supplying college and career preparatory course content. Talent Development Secondary employs a 

school transformation facilitator who works with school leaders to develop a systematic school transformation plan, creates and manages an Early Warning Indicator data tool, 

organizes frequent multidisciplinary teacher-team data-response meetings that use the tool 

to guide a multitiered student support process, and works closely with local or regional 

instructional facilitators to oversee instructional and curricular reforms. School-based 

instructional coaches in English/language arts and math support teachers’ delivery of

course content. For many students, whole-school organizational and instructional reforms,

referred to as Tier I interventions, are enough to keep them on track. However, for some 

students, the Early Warning Indicator data indicate that additional and more intensive services are necessary. City Year and Communities In Schools play leading roles in providing these additional services.

City Year

City Year is an AmeriCorps program through which young adults ages 18 to 24 participate in a year of national service. A team of about a dozen City Year AmeriCorps 

members is assigned to a school, increasing the number of adults in a building paying attention to students and working with them both in- and outside of classrooms. The team is 

led by a City Year program manager and team leaders (typically second-year AmeriCorps members), and its members are trained to provide a variety of academic and behavioral interventions — referred to as Tier II support — intended to help students stay on

track to graduate. These “near peers” (given their proximity in age to the students) serve 

as tutors, mentors, and role models, personalizing the school experience of the students. In 

addition, the AmeriCorps members provide after-school programs and help teachers by 

working with students during class time.

Communities In Schools

Through a school-based site coordinator, Communities In Schools, a national dropoutprevention organization, draws on school and community resources to organize services 

— referred to as Tier III support — intended to move the students at the highest risk of

dropping out back on track to graduation. The site coordinator assesses the needs of a student, develops an individual case plan to address those needs, and connects the student to 

services aligned with the case plan. Examples include professional counseling on anger

management for a student with behavioral issues or long-term tutoring with a subject-area

expert for a student far behind in class. A site coordinator will also provide direct service

— for example, leading student discussion groups on topics like conflict resolution or the

transition to adulthood.
4 

Figure 1.1

Diplomas Now Logic Model

Pillar I.

Teacher Teams and Small Learning Communities

Strong Learning Environments

(extended class periods, interdisciplinary teams with 

common planning time)

Pillar II.

Curriculum and Instruction with Professional 

Development

Professional Development and Peer Coaching

(math and English/language arts instructional coaching, 

principal leadership network)

Curriculum for College Readiness

(academic reform curriculum, transitional support 

classes)

Pillar III. 

Tiered Student Supports

Tiered Intervention Model

(early warning system and response meetings)

Student Supports

(tutoring, mentoring, near-peer role modeling, afterschool programs)

Student Case Management

(needs assessment, counseling, specialized 

interventions) 

Pillar IV.

Can-Do Culture and Climate

Integrated On-Site Support 

(transformation facilitator, site coordinator, City Year 

AmeriCorps members, instructional coaches)

Family and Community Involvement

(school events, parent and community outreach, 

community service)

Program Staff Training and Development

(Diplomas Now Summer Institute for school-based staff members, national Diplomas Now Implementation Support Team)

School 

Ø More positive school 

climate

Ø Additional after-school 

activities

Ø Increased stakeholder 

communication

Ø Increased parent/

community support

Student

Ø Greater confidence and 

sense of self-worth

Ø Increased engagement in 

and commitment to 

school learning

Ø Increased effort and 

persistence

Ø Improved study habits 

and strategies

Ø More positive 

relationships with adults 

and peers

More students on track to 

graduate

(“ABC” indicators)

Ø Higher attendance 

Ø Better behavior

Ø Improved course 

performance

More students graduate 

high school and are collegeready/job-competitive

INPUTS OUTCOMES

Early

Long-term

Intermediate
5 

learning communities create opportunities for personalization where teams of teachers know the 

same students and can work together to best teach and support them. Students also share the 

same classes and become known to one another. These teacher teams and small learning communities function best when there are opportunities for teachers to collaborate within the daily 

schedule and when they have classes long enough to cover material in depth and keep up the 

pace of instruction. This means that Diplomas Now staff members work with schools to reorganize master schedules to accommodate block scheduling and specific teacher collaboration 

time.8

 

Pillar II: Curriculum and Instruction with Professional Development

This pillar is focused on teaching and learning, and on giving teachers the training and 

resources they need to deliver strong lessons. Through professional development that includes 

an intensive peer coaching system for math and English/language arts teachers, teachers have an 

opportunity to sharpen their pedagogy. For students, Diplomas Now offers curricular materials 

aligned with college- and career-ready standards; it also supports the provision of accelerated 

remediation courses for struggling students, so that all students can meet their potential.

Pillar III: Tiered Student Supports

Providing more intensive support for students with greater needs is the core idea of this 

pillar. The tiered intervention model involves implementing an early warning system that draws 

on data on the ABC indicators to identify students who are either off track or at risk of going off 

track. Students may be identified by a low attendance rate or disciplinary referrals or suspensions, or because they are failing or at risk of failing their mathematics or English/language arts 

courses — or a combination of these factors. Teachers, administrators, and Diplomas Now staff 

members participate in regular meetings to review data on these indicators and plan interventions for these students. The intensity of the interventions depends on student need and the degree to which the student is off track.9

 8

“Block scheduling” refers to a school scheduling model in which daily class periods are longer than in

traditional models (for example, 80 to 90 minutes rather than 45 to 60 minutes). This means that fewer class

periods are scheduled per day in block-scheduled schools (four extended class periods, for example) than in

traditional schools, where a daily schedule with six or more class periods is more common. 9

The model has three tiers: Tier I interventions support the entire school. Tier II interventions are individually tailored supports offered to students identified as falling off track. Many of these interventions are provided by City Year AmeriCorps members, who serve as mentors, tutors, and role models. Tier III interventions are for students at the highest risk of dropping out and are generally coordinated by a case manager

from Communities In Schools.
6 

Pillar IV: Can-Do Culture and Climate

School reform is difficult, and school staff members often have much to do when they 

are asked to effect change. Diplomas Now brings at least a dozen staff members to a school to 

help coordinate school transformation, introduce new practices and structures, provide training 

and support, provide additional services to students, and engage with families and community 

organizations. All of these staff members are trained by Diplomas Now before and throughout 

the school year. Providing and organizing resources to assist the school’s staff helps foster a 

culture and climate where it feels possible to improve the school and support students better. 

Implementing these four pillars is hypothesized to affect a series of school outcomes 

(such as the school’s climate and communication among stakeholders) and student outcomes 

(like study habits and engagement with school), leading to improvements in students’ attendance, behavior, and course performance in mathematics and English/language arts, which 

should in turn lead to increased high school graduation rates.

The National i3 Evaluation of Diplomas Now

In total, 62 schools (33 middle schools and 29 high schools) from 11 large urban school districts 

across the country were recruited to participate in the study starting in either the 2011-2012 or 

the 2012-2013 school year.10 By design, Diplomas Now works in high-needs schools. The 

schools in the study come from high-poverty urban areas where students struggle academically 

and drop out at high rates. The participating schools, all eligible for Title I funds,11 serve large 

populations of low-income and minority students (81 percent eligible for free or reduced-price 

lunch; 83 percent black or Hispanic). Furthermore, the high schools participating in the study 

have weak promoting power (56 percent), suggesting that they struggle to move students from 

ninth through twelfth grade on time.12 Thirty-two of the participating secondary schools were 

randomly assigned to implement the Diplomas Now model (DN schools) and 30 were assigned 

to continue with “business as usual” (non-DN schools), either maintaining their existing practices and structures or pursuing other types of school reform. This random assignment design is 

often referred to as the “gold standard” in evaluation because the schools are all similar at the 

beginning of the study, and the decision about which schools will implement the program is 

random and not related to any preexisting characteristics of schools. The implication of this design means that any differences between the DN and non-DN schools that emerge after random 

 10Five of the school districts are among the 20 largest in the country, and all but 1 are among the 100

largest (Plotts and Sable, 2010). 11Title I funds from the U.S. Department of Education go to schools with high numbers or high percentages of students from low-income families. 12Corrin et al. (2014). “Promoting power” is calculated as the ratio of twelfth-graders to ninth-graders

three years earlier.
7 

assignment can be attributed to the program rather than to specific school attributes; in short, 

Diplomas Now caused the observed differences.

The study’s experimental design makes it possible for the evaluation to assess the impact of Diplomas Now. This third report of the evaluation focuses on the impact of Diplomas 

Now on students’ ABC outcomes after one year of Diplomas Now intervention in the sixth and 

ninth grades.13 For students who attend a Diplomas Now school in sixth or ninth grade, what 

difference does the Diplomas Now model make on attendance rates, suspensions and expulsions, and successful course completion? Essentially, does the implementation of Diplomas 

Now keep more students on track to high school graduation by the end of their middle school or 

high school transition years? 

This report will also explore the impact of Diplomas Now on outcomes that may act as 

precursors to the ABCs. As shown in Figure 1.1, the theory of change for the Diplomas Now 

model posits that the multiyear implementation of the Four Pillars will affect early school and 

student outcomes (displayed in the first and second boxes on the top right-hand side of the figure). The early school outcomes (shown in the top box) focus on changes the Diplomas Now 

partners would expect to see throughout a school as the model is implemented. The second box 

includes changes to student attitudes and behaviors that would be expected to occur as the model is implemented and begins to “take hold” at a school. There are two possible scenarios for 

change depicted in the logic model. First, implementation of school reforms could lead to 

changes in early school outcomes that will subsequently affect student attitudes and behaviors. 

Second, implementation of services that support individual students could lead directly to 

changes in early student outcomes. In both these cases, the theory of change implies that positive growth in student attitudes and behaviors should lead to better student attendance, fewer 

issues with behavior, and fewer course failures, and ultimately bring more students to graduate 

with the skills needed to be successful after high school.

Johns Hopkins University, as a representative of the Diplomas Now partnership, was 

granted funding from the Office of Innovation and Improvement of the U.S. Department of Education to support an extension of the evaluation. This extension will allow for a future report 

exploring variation in implementation and outcomes across schools that may generate hypotheses that could inform best practices, and a final report that will make it possible to analyze the 

impact of Diplomas Now on student outcomes after four years, answering the question: What is 

the impact of Diplomas Now on high school graduation rates and on the ninth-grade success of 

 13Two middle schools are not included in the analyses in either the first or second year due to issues with

grade configuration. Two more middle schools are not included in the second-year analyses because one

school closed and one school stopped serving sixth grade.
8 

students from Diplomas Now middle schools? As the ultimate focus of the Diplomas Now 

model, this will offer the most definitive test of whether the model met its outcome goals. 

Summary of Key Findings from the First Two Reports

The evaluation has produced two prior reports on the implementation of the Diplomas Now 

model, documenting how this complex, multicomponent reform intended to transform secondary schools is implemented by multiple partners. The implementation research explores what it 

takes to put the model into practice, what factors promote or hinder implementation, and the 

nature of the collaboration among multiple actors from the Diplomas Now organizations and 

the schools. The first report from the evaluation, released in 2014, focused on program start-up 

and first-year implementation fidelity, as well as exploring qualitative data collected in a subset 

of DN schools on the collaborative processes taking place. The second report, released in 2015, 

continued the story, including findings about fidelity in the second year of model implementation. It also examined school structure and staff practice at DN and non-DN schools to see 

whether Diplomas Now had created differences (that is, a “service contrast”) between the two 

groups of schools. Finally, it presented analyses of qualitative data, again collected in a subset 

of DN schools, exploring the context in which implementation was occurring, the integration of 

the model at schools, and the importance of staff stability. The following provides a summary of 

prior report findings.

Fidelity of Implementation

To create a measure of fidelity of implementation, each of the Diplomas Now partner 

organizations detailed the components it felt were needed for full implementation of the model. 

Over 100 components were identified, and each component was measured on a 0-1 scale with 0 

equaling no or low implementation and 1 equaling high fidelity to the model as originally intended.14 The overall average fidelity score for all the components across all the DN schools 

was 0.61 during the first year and 0.62 during the second year. These scores suggest that on average, schools were implementing a majority of the components with high fidelity to the model 

each year, but they still had room for growth to reach ideal implementation. The similarity in 

scores suggests stability in program implementation over the two years. Primary findings regarding fidelity of implementation include:

• DN schools were most successful in obtaining, retaining, and training the 

auxiliary staff members needed to implement the model effectively (Pillar 

 14The Diplomas Now national team and individual school teams are constantly working to improve the

model and revise it to better support specific school needs. For this reason, some digression from the original

model is expected.
9 

IV); using data to identify at-risk students; and collaborating to plan and provide individual and small-group interventions for those students (Pillar III).

Implementation fidelity grew from Year 1 to Year 2 in staffing and providing training 

to the auxiliary Diplomas Now staff, and by Year 2, almost all schools had all the necessary 

staff members in place to implement the model. A large majority of schools were also successful during both years in establishing a data system to track students’ attendance, behavior, and 

course performance in mathematics and English/language arts and ensuring collaboration 

among staff members to plan and provide interventions for students falling off track. City Year 

AmeriCorps members (see Box 1.1) were successful in providing extra and complementary 

services to students across all schools, but some schools were less successful in ensuring that 

enough City Year AmeriCorps members were available to meet the needs of all the students. 

Schools were also moderately successful in establishing small learning communities of students 

who shared the same classes and teachers (Pillar I), but many schools struggled to hold frequent 

meetings of these communities’ interdisciplinary teams of teachers.

• DN schools were least successful at offering teachers the peer coaching 

needed to strengthen practice, implementing curricular additions to ensure 

college and career readiness for all students (Pillar II), and involving parents 

and community members in school activities and decisions (Pillar IV).

In both the first and second years of implementation, it was hard for schools to meet 

implementation goals in some areas where it was necessary to change school policy or structure 

or to get the school’s staff on board. This may have been due, in part, to the short timeline available for recruitment of a large number of schools, which did not allow for the ideal amount of 

communication with school leaders ahead of implementation. Most schools struggled to achieve 

consistent coaching for math and English/language arts teachers at the level desired by Diplomas Now, and most schools did not provide the prescribed academic foundations and accelerated remediation courses for struggling students. Finally, on average, schools were not meeting 

the ideal levels of implementation for involving parents and community members.

Service Contrast

DN schools were more likely to implement the types of activities found under several 

of the Diplomas Now pillars than non-DN schools, suggesting that Diplomas Now is making 

DN schools different from non-DN schools. Unlike the fidelity of implementation score, there 

was quite a bit of growth from Year 1 to Year 2 in service contrast. Key findings include:

• There was service contrast between DN and non-DN schools for the pillars 

that showed strong and moderate implementation, including Pillar I (Teacher 

Teams and Small Learning Communities), Pillar III (Tiered Student Sup-
10

ports), and the part of Pillar IV (Can-Do Culture and Climate) focused on 

hiring and retaining the staff needed to implement the model.

Teachers at DN schools were more likely than teachers in non-DN schools to report 

teaching in extended class periods and collaborating with interdisciplinary teams that shared the 

same students; using data to identify at-risk students and meeting with other school staff members to plan interventions for them; and seeing that students receive academic, behavioral, or 

emotional support.

• Even though the implementation of teacher professional development and 

coaching did not fully meet the model’s goals, the service-contrast findings 

suggest that teachers at DN schools received more coaching than teachers at 

non-DN schools. Yet teachers at DN schools and teachers at non-DN schools 

reported similar usage of college readiness curricula. 

Math and English/language arts teachers at DN schools reported receiving more coaching than teachers at non-DN schools. Both groups of teachers reported relatively high levels of 

adoption of academic reform curricula, incorporation of transitional support classes for struggling students, and use of student-centered and college and career readiness-focused strategies 

in the classroom.

• The contrast between DN and non-DN schools increased from Year 1 to 

Year 2 in several areas, including collaboration of teachers within interdisciplinary teams (Pillar I), professional development of teachers (Pillar II), and 

the use of data to identify struggling students (Pillar III).

From Year 1 to Year 2 practices and structures aligned with the Diplomas Now model 

were maintained or slightly improved in DN schools, while similar structures and practices 

were less evident in non-DN schools in Year 1 and became even less evident in Year 2, suggesting that Diplomas Now helped stabilize resources and programs at the DN schools.

Challenges and Emerging Successes in Implementation During

the First Two Years

Effective collaboration is at the heart of the Diplomas Now school reform model, which 

deploys staff members from Talent Development Secondary, City Year, and Communities In 

Schools to work in concert with each other and with school staff members to implement the 

tiered intervention model. As part of its research during the first year, the evaluation team investigated the collaboration of the Diplomas Now and school staff members at a subset of DN 

schools. 
11

• Across the Diplomas Now and school staff members, two aspects of collaboration were most often reported to be important for successful implementation: investment and role clarity.

Administrators and teachers are key stakeholders whose engagement in implementing 

the Diplomas Now model inputs is indispensable. In order to become actively engaged, it is important that they understand the model through activities such as information sessions and meeting school staff members at other DN schools. Continual communication, including regular 

meetings and informal check-ins, helps build the trust and acceptance necessary for the collaborative work of model implementation. In addition, it is crucial that Diplomas Now school-based 

staff members establish their purposes and roles with teachers and administrators as well as 

among themselves. The Talent Development Secondary, City Year, and Communities In 

Schools staff members provided schools with the human resources necessary for the implementation of the Diplomas Now model. However, increased clarity regarding the roles and responsibilities of these Diplomas Now staff members would have further improved model implementation in the first year.

During the second year, the research team focused on examining how school context affected implementation, exploring both the successes and struggles of implementing the program. Findings include:

•

•

•

Various factors external to the program (such as school closures, principal 

turnover, and budget cuts) influenced the implementation of Diplomas Now. 

DN and non-DN schools were probably equally affected by these issues, but 

in some cases Diplomas Now was able to offer schools consistency and additional support in the face of the challenges.

Program staff members can foster stronger Diplomas Now implementation at 

a school in two main ways: (1) aligning program goals with school priorities 

and (2) securing administrator and teacher support for the Diplomas Now 

model. Qualitative data about Year 2 implementation indicates that Diplomas 

Now staff members understood how to do those two things, and as a result 

were able to make Diplomas Now “part of the school.”

Despite gains from Year 1 to Year 2 in hiring and training program staff 

members, in some schools program staff turnover may have undermined implementation and increased the burdens on the remaining staff members. On 

the other hand, maintaining at least some consistency in Diplomas Now 

leadership probably preserved useful institutional knowledge about the successes and challenges of the first year of Diplomas Now implementation, and 
12

the collaborative relationships among partner organizations allowed for some 

flexibility to support consistent implementation.

Orientation to This Report

The ultimate goal of the Diplomas Now model is to lead more students to high school graduation and college and career readiness. This report explores the impact of the Diplomas Now 

model during the first and second years of implementation on early and intermediate school and 

student outcomes, which are theorized to lead to the future goals of successful ninth-grade completion for middle school students and high school graduation for high school students. Chapter 

2 discusses the methods used in analyzing the data for this report. It describes the outcome 

measures analyzed and the sample of students included in the analyses. It also compares baseline characteristics of DN and non-DN schools at random assignment to show whether the two 

sets of schools were equivalent before program implementation. Chapter 3 describes the early 

school and student outcomes as measured through administrator, teacher, and student survey 

data. Chapter 4 first discusses the impact of Diplomas Now on student ABC outcomes (attendance, behavior, and course performance) in the second year of implementation for all incoming 

sixth- and ninth-grade students. It also describes the findings for middle and high school students separately and looks at the impacts for subgroups of students based on their prior (that is, 

baseline) academic achievement and behavior. It then further explores the effectiveness of the 

model by reviewing impact findings for the sixth- and ninth-grade students during the first year 

of implementation and looking at the variation in impacts across schools in the study.
13

Chapter 2

Study Design and Characteristics of Study Schools

This chapter describes the design of the study, including the random assignment of schools; the 

research questions being answered and outcome measures reported on; and the samples of students, teachers, and administrators included in the analyses. Also discussed are the characteristics of schools and students in the study, along with the comparability of the schools implementing the Diplomas Now model (DN schools) and the schools not implementing the model (nonDN schools) at the time of random assignment. A brief description of how the results were obtained is also included, along with information for understanding impact findings. 

Key points from this chapter are as follows: 

•

•

Random assignment resulted in two groups of schools (DN and non-DN 

schools) that were comparable on measures of attendance, student behavior, 

and course performance at the start of the Diplomas Now model implementation. In general, the students in the analytic sample who attended DN schools 

were similar to the students who attended non-DN schools. Baseline tests of 

characteristics of students in both sets of schools revealed no systematic differences, as expected in a randomized design.

The comparability of (1) the DN and non-DN schools at the start of the study 

and (2) the students entering DN schools and those entering non-DN schools 

bolsters confidence that the random assignment of schools to implement Diplomas Now or not resulted in two equivalent groups, allowing for any difference in outcomes between the DN and non-DN schools to be attributed to 

the implementation of the Diplomas Now model.

The Random Assignment Design

The core of the national evaluation of Diplomas Now is a school-level random assignment 

study design, presented in Chapter 1, in which some schools were randomly assigned to implement the program (the Diplomas Now school reform model), and others were randomly assigned not to implement the program but instead to continue with “business as usual” programming or other reforms. It is important to note that “business as usual” does not mean that these 

schools experienced no other reforms or interventions, but rather that they experienced whatever else a given school or district may have invested in during the study period, which was not

Diplomas Now. This kind of study is considered the “gold standard” for evaluation because the

random process creates conditions under which any differences in outcomes between imple-
14

menting and nonimplementing schools that are found after a program has been put in place can 

be causally attributed to that program as implemented. In other words, this method allows researchers to know with certainty that the program caused specific outcomes, and that impacts 

are not related to other school or student characteristics. 

The Diplomas Now evaluation recruited 62 schools to participate; 32 received the Diplomas Now program, and 30 represent the “business as usual” comparison condition. Schools 

were recruited from 11 urban districts around the country. Within each district, schools were 

clustered into middle school or high school “blocks” and randomly assigned to implement Diplomas Now (DN schools) or not (non-DN schools) within those blocks. In addition, because 

schools were recruited in two waves, they were also randomized by year. Overall, the block design increases precision by ensuring that schools that are alike are compared with each other, 

and that similar numbers of middle and high schools within districts are randomized into each 

condition. Furthermore, assigning blocks that contain only schools recruited at the same time 

ensures that any differences in the temporal nature of recruitment and implementation do not 

influence outcomes. However, not all of the recruited schools are included in the main impact 

analyses. Two schools were dropped from the analysis because one closed after the first year of 

implementation and one stopped serving sixth grade, and the main analyses are focused on the 

impacts of Diplomas Now in the second year of implementation. Also, one middle school block 

was dropped because the DN school and the non-DN school in that block had different grade 

configurations, in which students in one school started in sixth grade and those in the other 

started in seventh grade.1 Since these analyses compare students entering middle school (or high 

school) at the same grade level, students could not be compared across these schools. Thus, 

there are a total of 29 high schools (15 DN and 14 non-DN) and 29 middle schools (14 DN and 

15 non-DN) across 21 random assignment blocks in the analysis sample.

Analyses for this report focus on the transition year for students (that is, the year students first enter middle school or high school) because the Diplomas Now model is designed to 

provide extra support to students at critical transition points, where they tend to falter and need 

additional support. Although findings are presented for two cohorts of students (that is, students 

who enter and travel through a grade together), the main analysis focuses on the second cohort 

of students to enroll in study schools after implementation, because schools may need at least a 

year to fully understand and adapt to all the new program elements. 

 1

This difference in grade configurations was discovered after random assignment had been conducted,

but it was a preexisting situation and not related to assignment of the schools to implement DN or proceed

with “business as usual.”
15

Characteristics of Schools and Students in the Study 

By design, the Diplomas Now program aims to work with schools facing many challenges, typically schools whose students underperform academically and are at risk of eventually dropping 

out. These are the kinds of schools participating in this evaluation. In addition, the schools in 

this study are all located in urban districts and primarily serve black and Hispanic populations. 

They also have high numbers of students who qualify for free or reduced-price lunch, suggesting that the majority of the students in these schools come from low-income families. Although 

the random assignment design should create groups of schools that are comparable, a check of 

the equivalency of school characteristics before implementation was done in order to ensure that 

random assignment resulted in a similar distribution of characteristics across the DN and nonDN schools.2

Table 2.1 compares DN and non-DN schools during the year before implementation on 

the ABC indicators (attendance, behavior, and course performance) that the study uses as intermediate outcome measures. Across the full sample, no statistically significant differences were 

observed on any measures between the DN and non-DN schools, indicating that before the 

rollout of Diplomas Now, the schools were equivalent on these outcomes, and that any differences that are observed between the two groups of schools post-random assignment were 

caused by the implementation of the Diplomas Now model, and not by preexisting differences 

between schools.3

 

Table 2.2 shows baseline comparisons for students in the analytic sample used for the 

impact analyses. Since the primary impacts in this report were calculated for the second cohort

of students to be enrolled in DN schools after implementation, these tables present information 

only on students in that cohort. Because Table 2.2 compares students rather than schools at

baseline, figures are calculated from student records that provide data about students before

their entry into a study school. For example, the test score measures are calculated from students’ performance on state standardized tests in the year before implementation, in fifth and 

eighth grades, effectively providing a measure of performance that is as close to the beginning 

of the school year as possible. 

Table 2.2 indicates that there were no statistically significant differences between students enrolled in DN and non-DN schools, with the exception of Hispanic students, of which 

there were significantly more in DN schools. In addition, a statistical test of all race/ethnicity

 2

For additional details about the characteristics of the study schools at the time of random assignment,

see the first report from this evaluation (Corrin et al., 2014). 3

For this report, a finding is described as statistically significant if there is a 10 percent probability or less

that it was the result of chance.
16

Table 2.1

Baseline Comparison of DN and Non-DN Schools

on Attendance, Behavior, and Course Performance

 DN

Schools

Non-DN

Schools

Estimated

Difference P-Value

Attendance

Percentage of enrolled days attended

Percentage of students who attended over 

90 percent of enrolled days

Percentage of students who attended over 

85 percent of enrolled days

Behavior

Percentage of enrolled days suspended or expelled

Percentage of students who were ever

 suspended or expelled during year

Percentage of students who were suspended or

expelled for 3 or more days

a Course performance

Percentage of core courses passed

Percentage of students who had no core 

course failures during year

Percentage of students who had no math course

failures during year

Percentage of students who had no English/

language arts course failures during year

b ABC composite measures

Percentage of students above stability threshold

Percentage of students with no early warning

indicators

88.1

65.6

77.3

2.0

30.9

22.2

82.3

66.9

79.0

82.6

43.0

55.7

88.2

67.5

77.7

1.9

27.2

19.6

82.0

65.1

76.2

80.7

42.5

54.4

0.0

-1.9

-0.3

0.2

3.7

2.6

0.3

1.8

2.8

1.9

0.5

1.4

0.959

0.558

0.897

0.675

0.213

0.386

0.800

0.381

0.101

0.335

0.873

0.700

Sample size 29 29

(continued)
17

variables combined, as well as a combined test of all variables in the table, did not find significant differences.4 Thus, aside from one difference in ethnic composition between DN and nonDN schools, the two groups of schools and students were similar before the intervention. 

Data Sources 

All data used for impact analyses were obtained from individual administrative student records 

collected from each of the 11 participating school districts. In addition to demographic information about each student, including race, gender, and free or reduced-price lunch eligibility,5

these records included information on student days of enrollment, attendance, and school suspensions and expulsions. Furthermore, all student records included all courses students were 

enrolled in and their final grades. Finally, for years in which students were enrolled in tested 

grades, the data included both scaled scores and proficiency levels on state-administered English language arts (ELA) and math exams.

 4

A chi-square test was used. A test of all variables is used because conducting many individual tests increases the risk of a false positive. 5

After DN implementation began, one district began a program in some schools in which all students received free lunch, and parents no longer had to turn in paperwork establishing eligibility. Beginning with this

cohort year, all students in those schools are marked as free-lunch eligible, so the measure is not usable as a

proxy measure for income status. There are four schools in this district. 

SOURCE: MDRC calculations based on sixth- and ninth-grade student records obtained from school districts 

for the year prior to the implementation of Diplomas Now (2010-2011 for wave 1 schools and 2011-2012 for 

wave 2 schools). 

NOTES: One DN school was dropped from the course performance analysis because baseline course data were 

not available for that school.

 Difference estimates are regression adjusted, controlling for blocking of random assignment. A two-tailed ttest is used for all statistical tests presented in this table. Statistical significance levels are indicated as follows: 

*** = 1 percent; ** = 5 percent; * = 10 percent.

 Rounding may cause slight discrepancies in calculating sums and differences.

 a

Core courses include all math, English/language arts, science, or social studies courses each student 

attempted during the school year.

 b

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. 

The “no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the 

district, was suspended or expelled for fewer than three days, and did not fail any math or English/language arts 

courses during the school year. If a student has missing course data but meets the other two criteria for an ABC 

composite measure, the student is assigned a value of 1. 

Table 2.1 (continued)
18

Table 2.2

Baseline Comparison of DN and Non-DN Schools,

Characteristics of Students in Cohort 2 Analysis Sample 

Characteristic (%)

DN

Schools

Non-DN

Schools

Estimated

Difference P-Value

Eligible for free/reduced-price lunch

Race/ethnicitya

Black

Hispanic

Asian

White

Other

Gender 

Male

English language learners

Special education status

Overage for assigned grade

Proficient on state tests in prior year 

English/language arts

Math

89.9

54.0

36.7

3.7

4.2

1.3

54.7

18.0

19.2

32.0

36.3

35.0

89.1

61.7

27.6

4.9

4.6

1.2

54.3

17.3

17.1

31.9

38.9

38.0

0.8

-7.6

9.1 **

-1.3

-0.3

0.2

0.4

0.7

2.1

0.1

-2.6

-3.0

0.525

0.195

0.029

0.399

0.863

0.811

0.691

0.737

0.119

0.973

0.269

0.158

Sample size 29 29

SOURCE: MDRC calculations based on student records obtained from school districts of students attending sixth 

and ninth grades during the second year of implementation (2012-2013 for wave 1 schools and 2013-2014 for 

wave 2 schools). 

NOTES: Difference estimates are regression adjusted, controlling for blocking of random assignment. A twotailed t-test was used for all statistical tests presented in this table. Statistical significance levels are indicated as 

follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 An omnibus test of all baseline characteristics included in this table did not find a statistically significant 

overall difference between DN and non-DN schools.

 a

A chi-square test of all race/ethnicity categories combined did not find a statistically significant overall 

difference between DN and non-DN schools.
19

In addition to collecting administrative student records, the research team fielded administrator, teacher, and student surveys in all study schools in the springs of 2012, 2013, and 

2014. The surveys were designed to collect student and staff views on issues such as the school 

environment, the climate of teaching and learning, levels of bullying and other behavioral matters, and community relations. 

Measures

ABC Outcomes

Central to the Diplomas Now theory of action is the development and regular monitoring in schools of a set of early warning indicators that alert school leaders and staff members to 

students who are off track or at risk of falling off track on ABC outcomes (attendance, behavior, 

and course performance). The ABC outcomes are considered to be intermediate outcomes that 

are critical to the long-term program goals of more students graduating from high school who 

are college and career ready. Therefore the focus is on these three outcomes, as well as an overall composite measure of students being on track to graduate, as indicative of whether Diplomas 

Now is having a positive impact on students’ likelihood of achieving the longer-term goals of 

the model. 

The measures of attendance, behavior, and course performance used in the impact 

analyses for this report are described in more detail in Table 2.3.

6 These outcomes all capture 

the average change across all students. The evaluation’s primary research questions are about 

the impact of Diplomas Now implementation relative to schools’ typical school improvement 

efforts:

•

•

•

What is the impact of Diplomas Now on students’ attendance rates (proportion of enrolled days attended) during the sixth- or ninth-grade school year?

What is the impact of Diplomas Now on the proportion of enrolled days suspended (in or out of school) or expelled during students’ sixth- or ninth-grade 

year?

What is the impact of Diplomas Now on the proportion of attempted core 

courses passed by the end of students’ sixth- or ninth-grade year?

 6

The primary measures were identified before the start of the study as likely to be affected by the Diplomas Now theory of action. Preidentification of these measures is one of the strategies used to prioritize outcomes for determining the effectiveness of the Diplomas Now model and help protect against spurious findings that might arise from conducting multiple hypothesis tests.
20

 

Outcome Description

Primary

Outcome?

Attendance

Percentage of enrolled days attendeda

Percentage of students who attended 

b over 90 percent of enrolled days

Percentage of students who attended 

over 85 percent of enrolled days

Proportion of the days that a student was enrolled in

the school district on which that student was present

at school

For each student, the value is 1 if the student attended

over 90 percent of enrolled days and 0 otherwise.

For each student, the value is 1 if the student attended

over 85 percent of enrolled days and 0 otherwise.

Yes

No

No

Behavior

Percentage of enrolled days

suspended or expelleda

Percentage of students who were ever

suspended or expelled during year

Percentage of students who were 

suspended or expelled for 3 or

more days

Proportion of the days that a student was enrolled in

the school district for which that student was

suspended or expelled

For each student, the value is 1 if the student was ever

suspended or expelled during the school year and 0

otherwise.

For each student, the value is 1 if the student was

suspended or expelled for 3 or more days and

0 otherwise.

Yes

No 

No 

Course performance

Percentage of core courses passedc

Percentage of students who had no 

core course failures during year

Percentage of students who had no

math course failures during year

Percentage of students who had no 

English/language arts course

failures during year 

Proportion of the core English/language arts, math,

science, and social studies classes a student took

during the year that the student passed

For each student, the value is 1 if the student did not

fail any core courses during the year and 0 otherwise.

For each student, the value is 1 if the student did not

fail any math courses during the year and 0 otherwise.

For each student, the value is 1 if the student did

not fail any English/language arts courses

during the year and 0 otherwise.

Yes

No 

No 

No 

ABC composite measures

Percentage of students above stability

threshold

Percentage of students with no early 

warning indicators

For each student, the value is 1 if the student attended over 

90 percent of enrolled days, was never suspended or

expelled, and passed all core courses.

For each student, the value is 1 if the student attended over

85 percent of enrolled days, had fewer than 3 days of 

suspensions or expulsions, and passed all math and

English/language arts courses.

No

No

NOTES: aEnrolled days are used as the denominator rather than total days in the school year because the analysis 

sample includes students who did not attend school in the district for the entire school year. There is no statistically 

significant difference in enrolled days between DN and non-DN schools.

 bStudents who attend school for less than 90 percent of days are typically labeled “chronic absentees.”

 c

Focus was placed on passing all core courses because these are the courses that students generally need to 

complete to be promoted to the next grade and to graduate.

Description of ABC Outcome Measures

Table 2.3
21

In addition to these primary outcomes, which are measured on a continuous scale, several threshold indicators of student success are used in exploratory impact analyses. These 

measures capture two levels of the ABC outcomes, a more stringent “stability” threshold and a 

threshold tied to whether students have any early warning indicators. The stability threshold 

measures are based on whether students attended school more than 90 percent of the time, were

not suspended or expelled during the study year, or passed all of their core classes.7 These 

thresholds also represent normative expectations for students not to be chronically absent (typically defined as missing 10 percent or more days of school), not to get in serious trouble, and 

not to fail classes. A composite of all three of these threshold measures of success indicates 

whether a student is defined in this report as being on a stable trajectory to graduation. 

The second level of thresholds on the ABC outcomes measures whether Diplomas Now 

is making incremental progress with students by reducing the number with any early warning 

indicators, which are suggestive of being off the track to graduation. These thresholds align with 

levels used by Diplomas Now and school staff members in their day-to-day practice for assessing student need for additional intervention. These early warning indicators are defined as 

85 percent or lower attendance, more than one suspension or disciplinary action,8 or English/language arts or math course failure — the two subject areas for which Diplomas Now provides additional support to teachers and students. Analyses of the percentages of students above 

and below these early warning thresholds allow for the assessment of the impacts of Diplomas 

Now on the number of students who are in early warning status in terms of each of the ABC 

outcomes as well as overall across all three. All of these outcomes are described in Table 2.3.

Early Outcomes (Survey Measures)

As illustrated in Figure 1.1, the early school and student outcomes described in this section are predicted to be affected by Diplomas Now program activities. Effects on these outcomes are hypothesized to lead to effects on the ABC outcomes. Chapter 3 presents analyses on 

data collected from surveys of administrators, teachers, and students about these outcomes. 

Early School Outcomes

Through their survey responses, teachers reported on their perception of the school climate, including whether the school environment was conducive to teaching and learning and 

 7

Diplomas Now provides specific support for teachers and students in English/language arts and math,

but graduation from high school is dependent on earning credits across core course areas that typically include science and social studies (or history) as well. 8

Because data on the number of disciplinary actions were not provided across all districts, analyses conducted for this report use three or more total days suspended or expelled as a proxy for more than one disciplinary action.
22

whether there were consistently enforced rules for student behavior. Students reported whether 

they felt safe and happy at school and whether the school had problems with serious student 

behavior issues (such as bullying, fighting, physical or verbal abuse of teachers by students, 

drug and alcohol abuse by students, and gang activity). Teachers and administrators both reported on parent and community involvement, including whether parents and community members volunteered at the school or were involved in school activities and whether there was effective communication between the school and parents and community members. Students also 

reported on their participation in after-school activities, especially those that included academic 

support.9

Early Student Outcomes

Students were also asked specific questions about their attitudes and behaviors as

measures of their confidence, engagement and commitment to school, effort and persistence in 

school, study habits, and relationships with adults and peers.10

Analytic Sample

Sample Used for Impact Analysis

The main analytic sample includes first-time sixth- and ninth-grade students attending 

the study schools during the second year of implementation after random assignment (Cohort 

2).11 The key assumption behind choosing this group as the primary analysis sample is that the 

Diplomas Now model is a multiyear and multifaceted program. Therefore, it may take schools 

more than a year to adjust to the program and figure out how implementation works best within 

their individual sites. First-time sixth- and ninth-graders are chosen because the model includes 

intensive supports for students during their first year attending middle or high school, and some 

of those supports lessen in intensity as students enter higher grade levels.12 Thus measuring impacts at the end of the year in which the second cohort of students made the transition into the 

 9

The surveys did not measure the effects of Diplomas Now on the outcome of increased stakeholder

communication. Given the importance of the Diplomas Now staff members as stakeholders at the DN

schools, it was hard to find comparable measures of communication between the DN and non-DN schools. 10See Appendix A for a description of the construction of all early school and student outcomes that are

made up of more than one survey item. 11For one random assignment block with three schools, the data in the main analysis is from sixth-grade

students who attended the schools in the first year of implementation, because two of the schools did not have

sixth-grade students in the second implementation year. 12All high schools are ninth through twelfth grade. There are a few middle schools that are kindergarten

through eighth or fifth through eighth grade. In these cases, Diplomas Now still focused efforts on sixthgraders.
23

school may present a truer picture of the impact of Diplomas Now on student outcomes than 

would analyses at the end of the first year of implementation. That said, examining only the 

second year of implementation may not be enough to show the full effects of a program that is 

designed to create change for students in multiple grades across multiple years. 

While many of the schools in the analysis sample have large portions of repeating students, particularly students repeating ninth grade, the decision was made to focus on first-time 

students because repeaters really belong to the prior cohort of entering students. Regarding the 

primary analysis sample, students who are repeaters in the second implementation year are 

members of the first cohort of study students, and thus already have their sixth-grade or ninthgrade outcomes analyzed with those of their cohort peers; their full trajectory will be included in 

the longer-term analyses to be discussed in subsequent reports. The analysis of first-time-ingrade students therefore addresses how well the Diplomas Now model keeps students on track 

as they move through secondary school. Longitudinal analysis that follows students to graduation or successful high school transition will better capture the impacts of the full intent of Diplomas Now to influence students’ longer-term school success. 

Findings are presented for all students in Cohort 2, as well as separately for students in 

middle and high schools. As mentioned, also presented in this report are outcomes for the first 

cohort of sixth- and ninth-grade students to enroll in study schools after random assignment.13

In addition, subgroup analyses are included for high school students who were or were not on a 

stable trajectory to graduation in eighth grade; middle school students who scored above and 

below proficient on state standardized tests at the end of elementary school; and students who 

remained enrolled in study schools for the entire school year and therefore fully experienced the 

programming offered at their school, whether it was the Diplomas Now model in DN schools or 

other practices or reforms being implemented in the non-DN schools. 

It is important to note that the main analytic sample (that is, first-time sixth- and ninthgraders in Cohort 2) represents the policy-relevant sample because it includes students who 

were enrolled in the school at any time during the year, regardless of how long they attended. 

This sample presents a truer picture of the kinds of students who enroll in study schools that 

serve mobile populations and represents impacts for a program designed to serve all students in 

a school, regardless of whether they remain for the entire year. 

 13Cohort 1 includes three schools in one random assignment block that served only seventh and eighth

grades. For these three schools, seventh-grade student outcomes are included in the analyses because seventh

grade was the transition year into middle school. Of those three schools, two began offering sixth grade in the

second year of implementation, and one closed and is not included in the Cohort 2 analyses.
24

Sample Used for Analysis of Early Outcomes 

The samples of teachers and administrators included in the analyses of the early school 

outcomes were chosen to match the analyses of the primary student ABC outcomes. With that 

goal in mind, the staff surveys were fielded in the spring of the second year after Diplomas Now 

implementation began, and only teachers who taught sixth- and ninth-grade students are included in the analyses. The findings provide context for understanding how the faculty and administration viewed initial changes in their schools that occurred as a result of the implementation 

of the Diplomas Now model.

The early school and student outcomes measured using the student survey come from 

the survey of sixth- and ninth-grade students fielded during the spring of the first year after implementation, so these outcome measures reflect the opinions and feelings of the students in 

Cohort 1.14

Analysis

A two-level statistical model was used to estimate the impacts of the Diplomas Now model, 

generating estimates at both the block level (since schools were randomized within blocks) and 

overall for the full sample of schools. Block-level estimates essentially provide the level of difference for a given outcome between DN and non-DN schools within a block. The estimated 

impact for the overall sample is an average of the effects for each block, weighted by the number of DN schools within the block. The impact thus represents the average outcome for the 

average student enrolled in the average DN school. However, it is important to note that the difference presented in the impact tables represents the estimated impact rather than the true impact, because although schools were randomly assigned to implement Diplomas Now, there is 

still a possibility that differences could be observed by chance. Further technical details of the 

analysis are available in Appendix A, including school- and student-level characteristics controlled for in the analysis and the hierarchical linear regression model. Results from the impact 

analysis on early outcomes based on survey data are in Chapter 3. Results from the impact analysis on intermediate ABC outcomes are presented in Chapter 4. Box 2.1 explains how to interpret the findings reported in the tables in both of these chapters.

 14The second cohort of students, sixth- and ninth-grade students attending the study schools during the

second year of implementation, was not surveyed.
25

Box 2.1

Understanding the Impact Tables in This Report

The tables in Chapters 3 and 4 display the impacts of the Diplomas Now model on school and 

student outcomes. These tables also present the average outcomes for students, teachers, and 

administrators in the study schools, to provide context for interpreting the magnitude of the impact findings. The values presented in these tables are derived as follows: 

Estimated difference or estimated impact. This column shows the estimated difference between DN and non-DN schools with respect to the attendance, behavior, and course performance outcomes of their students or the survey responses of their students, teachers, and administrators. The values in this column are obtained by estimating the difference in outcomes for 

each random assignment block, and then adjusting, or weighting, each of these estimates by the 

number of DN schools in the block, so that each school contributes equally. Because of this 

weighting procedure, the average estimate presented in the table represents the difference (or 

impact) for the average school in the DN group. The statistical significance of the estimated difference is indicated when the p-value is less than or equal to 10 percent (*** = 1 percent; ** = 5 

percent; * = 10 percent). A p-value of 10 percent means that if there truly were no difference between DN schools and non-DN schools on a particular outcome, the probability of finding the 

estimated difference would be quite small — no more than 10 percent. Therefore, probability 

suggests that there is an actual difference, and the number in the impact column is an estimate of 

what that difference is.

DN schools. This column shows the observed mean outcome for schools in the DN group. As 

with the differences above, these mean outcome levels are obtained by weighting the mean outcome level in each random assignment block by the number of DN schools in that block. Thus 

the mean outcome represents the mean outcome for the average school in the DN group. 

Non-DN schools. This column provides an estimate of what the mean outcome for the average 

DN school would have been had it not been randomly assigned to implement the DN model. 

The values in this column are the mean outcomes for schools in the non-DN group, adjusted to 

reflect the observed distribution of DN schools across random assignment blocks. 

Effect size. This column shows the estimated impact translated into an effect size. The effect 

size is the estimated impact divided by the standard deviation of the outcome for the non-DN 

group. For example, an effect size of 0.20 represents 20 percent of the standard deviation for 

that outcome. This measure allows comparison between outcome measures that are on different 

scales.

27

Chapter 3

Early School and Student Outcomes

As illustrated in the logic model in Chapter 1, Diplomas Now is hypothesized to achieve its intermediate goals of improving attendance, behavior, and course performance (ABC outcomes) 

through several mediating pathways. These pathways, or early outcomes, include positive 

school climate, the addition of academic after-school activities, and increased parent and community support, along with student attitudes and behaviors, including self-confidence, engagement and effort in school, study habits, and relationships with adults and peers.1 As noted in 

Chapter 2, the analyses of the early school and student outcomes are based on data collected 

from administrator, teacher, and student surveys. Survey responses at schools randomly assigned to implement Diplomas Now (DN schools) were compared with the responses from the 

same respondent groups in the schools assigned not to implement the program (non-DN 

schools). The responses from administrators and teachers come from surveys administered at 

the end of the second year of implementation. Administrators include school principals and assistant principals. Teacher respondents include sixth- and ninth-grade teachers at the schools.2

Since sixth- and ninth-grade students are the focus of the analyses in this report and students in 

sixth and ninth grades were surveyed only in the spring of the first year of implementation, the 

early school and student outcomes reported using student survey responses come from the first 

year of implementation.3

 

The following are key points from the early school and student outcome findings:

•

•

The Diplomas Now model had a positive and statistically significant impact 

on teachers’ perceptions of school climate. Although there were no other statistically significant impacts on early school outcomes as reported by administrators and teachers, the findings across all the climate- and communityoriented outcomes tend to be positive.

Students at DN schools reported participating in more academically focused 

after-school activities than students at non-DN schools. Student perceptions 

of other outcomes, including school safety and climate and behavioral issues 

 1

One of the early school outcomes listed in Figure 1.1, increased stakeholder communication, is not represented in these analyses because the Diplomas Now school-level staff members represent a set of stakeholders, 

and comparable stakeholders do not necessarily exist in non-DN schools. 2

Seventh-grade teachers’ responses are included for schools that did not have sixth grade in the second 

year of implementation. 3

Seventh-grade students’ responses are included for schools that did not have a sixth grade in the first year 

of implementation.
28

at school, were similar for students at DN and non-DN schools during the 

first year of implementation. 

•

•

The Diplomas Now model did not have an effect on students’ selfperceptions and school behaviors as measured by the student survey during 

the first year of implementation. 

Students at DN schools were more likely to report a positive relationship 

with an adult at school who was not a teacher, but there was no difference in 

student perceptions of relationships with teachers, administrators, and other 

students at DN schools compared with non-DN schools. 

Findings on Early School Outcomes

Table 3.1 focuses on teachers and administrators’ perceptions of school climate and parent and 

community involvement at the school. (See Box 2.1 for an explanation of how to read the impact tables.) The Diplomas Now model had a positive and statistically significant impact on 

how strongly teachers agreed that the climate at their school was conducive to teaching and 

learning, suggesting that by the second year of implementation, the Diplomas Now model was 

positively affecting teachers’ perceptions of their school environment.

Although there was only one statistically significant finding on early school outcomes 

reported by administrators and teachers, there is a general positive pattern in the findings; all but 

one of the estimated differences in Table 3.1 is positive. This is encouraging especially for the 

outcomes related to increased parent and community involvement, considering that there was 

little difference in service during the second year between DN and non-DN schools on efforts 

by schools to plan for and offer more opportunities for parents and community members to participate.4 The one negative but not statistically significant outcome in this area is on the average

number of times per month parents volunteered in the classroom, according to administrator 

reports. Given the influx of student support from AmeriCorps members in the City Year program, it may not be surprising if DN schools are less likely to incorporate parent volunteers as 

well — especially during the early years of implementation, when schools are still figuring out 

how to employ the AmeriCorps volunteers in classrooms most efficiently and effectively. 

Table 3.2 displays the impact of the Diplomas Now model on sixth- and ninth-grade 

students’ perceptions of school climate and their participation in after-school activities at the 

end of the first year of implementation. There are no statistically significant impacts on students’ feelings of safety at school, their enjoyment of school, or their reports of serious student

 4

See Sepanik et al. (2015), pp. 38-41.
29

(continued)

Outcome

DN

Schools

Non-DN

Schools

Estimated

Difference

Effect

Size P-Value

More positive school climate

Teachers reported the school environment was 

conducive to teaching and learning. 

(0 = strongly disagree, 10 = strongly agree)

Teachers reported rules for student behavior were

consistently reinforced by administrators and

other teachers. (0 = never, 5 = sometimes, 

10 = always, as needed)

Increased parent and community involvement

Teachers reported their school had effective

communication with and participation from students'

families and the local community. (0 = not offered, 5 = fair,

10 = excellent)a

Teachers reported parents were involved in school

activities and supported school reform efforts.

(0 = strongly disagree, 10 = strongly agree)

Teachers reported community members were involved in

school activities and supported school reform efforts.

(0 = strongly disagree, 10 = strongly agree)

Administrators reported parents were involved in school

activities and supported school reform efforts.

(0 = strongly disagree, 10 = strongly agree)

Administrators reported community members were

involved in school activities and supported school reform

efforts. (0 = strongly disagree, 10 = strongly agree)

Teachers reported parents worked as volunteers in

classes. (average times per month)b

Teachers reported community members worked as

volunteers in classes. (average times per month)b

Administrators reported parents worked as volunteers in

classes. (average times per month)b

Administrators reported community members worked as

volunteers in classes. (average times per month)b

6.47

6.75

6.39

4.87

5.42

6.51

6.70

1.25

1.24

3.89

4.39

6.07

6.65

6.20

4.59

5.17

5.88

6.53

0.83

1.01

4.47

4.00

0.40

0.10

0.19

0.28

0.26

0.63

0.17

0.42

0.23

-0.58

0.39

0.14 *

0.04

0.08

0.11

0.11

0.25

0.07

0.13

0.08

-0.09

0.07

0.096

0.647

0.370

0.129

0.213

0.137

0.663

0.274

0.548

0.701

0.772

Sample size 31 30

Table 3.1

Early School Outcomes, Teacher and Administrator Responses, 

Implementation Year 2
30

Table 3.1 (continued)

SOURCE: Follow-up surveys of teachers and administrators (principals and assistant principals) administered 

during the school years of 2012-2013 and 2013-2014. Respondents included middle school teachers who taught 

sixth grade (or seventh grade for schools not serving sixth grade) and high school teachers who taught ninth grade.

NOTES: Across 60 study schools, 1,339 teachers participated in the follow-up survey. One study school stopped 

serving sixth grade after the first year of implementation and is not included in this analysis. For each of the 

outcome measures, data are missing for no more than 5.5 percent of the teachers. The difference in the percentage 

of missing data between DN and non-DN schools is no more than 2.4 percentage points for any of the outcome 

measures. 

 Across 61 study schools, 170 administrators participated in the follow-up survey. For each of the above 

measures, data are missing for no more than 7.1 percent of the administrators. The difference in the percentage of 

missing data between DN and non-DN schools is no more than 2.2 percentage points for any of the above 

measures. 

 Effect sizes were computed using the standard deviations of all non-DN school respondents for the respective 

measures. 

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Markow and Pieters (2012).

 b

This outcome is calculated by weighting the frequency per month and then summing the weighted survey items 

in order to get a measure of total times per month. For example, 0 = never, 1 = at least once a month, 2 = more than 

once a month but not weekly, 5 = once a week, 12 = more than once a week but not daily, and 20 = daily.

behavior issues such as bullying, fighting, physical and verbal abuse, drug and alcohol abuse, or

gang activity at school. This suggests that the Diplomas Now model had little effect on student 

perceptions of school climate during the first year of implementation. It is possible that it could 

take more than one year for the Diplomas Now model to take hold at a school in a way that 

changes the overall climate of the school and the level of behavioral problems, as well as students’ feelings of safety and ability to enjoy school. As noted above, second-year surveys revealed an impact on teachers’ perceptions of school climate.

Students at DN schools did report participating in more after-school activities, including 

those that offered homework help, test preparation, and tutoring, than students at non-DN 

schools. One of the services under the third pillar of the Diplomas Now model, Tiered Student 

Supports, is an after-school program run by the City Year AmeriCorps members, open to all 

students but targeted to students struggling academically. (See Figure 1.1.) In this way, the Diplomas Now model is set up to directly affect this outcome, and it succeeded in this goal during 

the first year. 

Findings on Early Student Outcomes

Table 3.3 displays the early student outcomes during the first year of implementation. The direction of the effects is inconsistent and there is only one statistically significant impact, 
31

(continued)

Table 3.2

Early School Outcomes, Sixth- and Ninth-Grade Student Responses,

Implementation Year 1

Outcome

DN Non-DN

Schools Schools

Estimated

Difference

Effect

Size P-Value

More positive school climate

Students felt safe at school. (0 = strongly disagree,

3 = strongly agree)

Students enjoyed coming to school.

(0 = strongly disagree, 3 = strongly agree)

Students would switch to a different school if they

were given the choice. (0 = strongly disagree,

3 = strongly agree)

The school had problems with bullying/cyber

bullying, students fighting, students cutting class,

and teachers not being able to control the 

classroom.

(0 = not a problem, 3 = big problem)

The school had problems with students physically

or verbally abusing teachers or other adults.

(0 = not a problem, 3 = big problem)

The school had problems with destruction of 

school property, students bringing weapons to 

school, students abusing drugs or alcohol, 

and gang activity.

(0 = not a problem, 3 = big problem)

After-school activities

Students participated in school-sponsored 

after-school activities. (%)

Students participated in after-school activities such

as homework help/assignment completion, test 

preparation, or tutoring. (%)

1.73

1.77

1.55

1.81

1.15

1.36

56.7

25.1

1.75

1.80

1.53

1.85

1.12

1.40

51.2

17.3

-0.01

-0.03

0.02

-0.04

0.03

-0.04

5.4

7.8

-0.02

-0.04

0.02

-0.05

0.03

-0.04

0.11 **

0.20 ***

0.795

0.355

0.652

0.424

0.560

0.487

0.027

<0.001

Sample size 32 30

suggesting that the Diplomas Now model had little effect during the first year on sixth- and 

ninth-grade students’ attitudes and behaviors related to school. It may be that model components needed to be implemented at a higher level than they were during the first year in order to 

elicit effects on these outcomes. Another possibility is that students’ attitudes and behaviors in
32

SOURCE: Follow-up surveys of students administered during the school years of 2011-2012 and 2012-

2013. 

NOTES: Across 62 study schools, 9,356 students participated in the follow-up survey. For each outcome 

measure, data are missing for no more than 5.3 percent of the students, except for the “students reported 

participating in school-sponsored after-school activities” item and the “students reported participating in 

after-school activities such as homework help/assignment completion, test preparation, and tutoring” item, 

where data for 11.5 percent of students are missing. The difference in the percentage of missing data 

between DN and non-DN schools is no more than 1 percentage point for any of the outcome measures. 

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures.

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 

are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences.

Table 3.2 (continued)

school may be particularly tough to change and may require persistent effort over a longer period of time. 

Although students at DN schools did not report stronger relationships with teachers, 

administrators, or other students than did students at non-DN schools, they were more likely 

than their counterparts at non-DN schools to report that they had a positive relationship with an 

adult at the school who was not a teacher. This finding aligns with the implementation of the 

Diplomas Now model, which creates more opportunities for students to interact with and be 

supported by adults who are not teachers. Specifically, City Year brings a group of 8 to 15 

AmeriCorps members into the schools to work directly with students identified as struggling on 

one or more of the ABC outcomes. In addition, Communities In Schools case managers assess

the needs of students who are struggling the most and connect them with specialized interventions, such as group or individual counseling and tutoring, which may be provided by adults 

other than teachers. 

Conclusion

Although the Diplomas Now model has shown few statistically significant impacts on early 

school outcomes, the results suggest that the model is having an effect on teachers’ perceptions 

of school climate by the second year of implementation. It is important to note that changes in 

teachers’ perceptions are likely to be an important step toward changing overall school climate

and students’ perceptions, which may take longer. The model also had an impact on some early 

school and student outcomes, including students’ participation in after-school activities and relationships with adults at the school who were not teachers, that can be traced to Diplomas Now

model components. Many of the student outcomes for which Diplomas Now did not produce
33

(continued)

 

Table 3.3

Early Student Outcomes, Sixth- and Ninth-Grade Student Responses,

Implementation Year 1

DN Non-DN Estimated Effect

Outcome Schools Schools Difference Size P-Value

Greater confidence and self-worth

Students believed if they tried hard, did not give up,

and had enough time, they could do their schoolwork

well. (0 = strongly disagree, 3 = strongly agree) 2.39 2.38 0.01 0.02 0.541

Students worried about projects, tests, poor grades,

and schoolwork in general. 

(0 = strongly disagree, 3 = strongly agree) 1.77 1.77 0.00 0.00 0.962

Increased engagement and commitment

Students thought their work was interesting and

liked what they learned in class. 1.53 1.57 -0.04 -0.05 0.323

(0 = never, 3 = all the time)

Students tried less at school over time, not caring

about school, and not participating in class.

(0 = strongly disagree, 3 = strongly agree) 0.81 0.82 -0.01 -0.02 0.612

Students believed learning and working hard in

class was important for their future. 

(0 = strongly disagree, 3 = strongly agree) 2.25 2.26 -0.01 -0.02 0.678

Increased effort and persistence

Students did not give up easily when they did not

understand or when homework was difficult.

(0 = strongly disagree, 3 = strongly agree) 2.04 2.03 0.01 0.01 0.758

Students paid attention, stayed on task in class,

and completed all their schoolwork.

(0 = never, 3 = all the time) 1.84 1.84 0.00 0.00 0.982

Improved study habits and strategies

Students used study strategies such as note taking,

graphic organizers, and formulas to help remember

information. (0 = never, 3 = all the time) 1.67 1.70 -0.02 -0.04 0.467

Students used social skills and conflict resolution

strategies for controlling anger. 

(0 = never, 3 = all the time) 1.49 1.49 0.00 0.00 0.969
34

Outcome

DN

Schools

Non-DN

Schools

Estimated

Difference

Effect

Size P-Value

Table 3.3 (continued)

Positive relationships with adults and peers

Students had positive relationships with teachers.a

(0 = strongly disagree, 3 = strongly agree)

Administrators were respectful of students and

teachers and were fair when enforcing rules.

(0 = strongly disagree, 3 = strongly agree)

Students had a positive relationship with at least

one adult in the school other than a teacher.b

(0 = strongly disagree, 3 = strongly agree)

Students felt they fit in at school and that other

students at school accepted them for who they

were. (0 = strongly disagree, 3 = strongly agree)

1.65

1.90

1.85

1.91

1.69

1.86

1.77

1.91

-0.04

0.04

0.08

0.00

-0.06

0.05

0.11 **

-0.01

0.316

0.373

0.011

0.866

Sample size 32 30

 Effect sizes were computed using the standard deviations of all non-DN school students for the 

respective measures.

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels 

are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

This composite measure includes the following items: teachers got along with students, met with 

students to talk about schoolwork and problems outside of the classroom, really listened to students, set a 

positive example, praised students for good work, and tried to be fair. 

 b

This composite measure includes the following items: at least one adult other than a teacher really cared 

about students, checked in with students, encouraged students to do their best, and could help solve student 

problems at school or at home.

SOURCE: Follow-up surveys of students administered during the school years of 2011-2012 and 2012-

2013. 

NOTES: Across 62 study schools, 9,356 students participated in the follow-up survey. For each outcome 

measure, data are missing for no more than 5.7 percent of the students. The difference in the percentage of 

missing data between DN and non-DN schools is no more than 1 percentage point for any of the outcome 

measures. 

statistically significant impacts during the first year involve changing attitudes, behaviors, and 

relationships. Although these outcomes are described as “early” in the logic model, it may take 

more than a year for these attitudes and relationships to form and build. These early school and 

student outcomes were theorized to lead to impacts on the ABC outcomes and, ultimately, 

graduation rates. The findings from analyses of the impacts of Diplomas Now implementation 

on the intermediate outcomes are discussed in the next chapter.
35

Chapter 4

The Impact of Diplomas Now on Intermediate

Outcomes of Student Attendance, Behavior,

and Course Performance

As described in Chapter 1, the Diplomas Now partnership works with challenged schools in 

an effort to improve student attendance, to address and reduce behaviors that lead to disciplinary action, and to improve student course performance, particularly successful completion of 

English/language arts and math courses. Attendance, behavior, and course performance are

the “ABCs” that help predict whether students graduate or drop out. Prior reports from this 

national evaluation of Diplomas Now focused on the implementation of the whole-school reform model over the first two years, and the previous chapter focused on how that implementation affected early school and student outcomes using staff and student survey data. This 

chapter extends the story by looking at impacts on students’ ABC outcomes during their first 

year of middle or high school. 

The chapter first discusses the impacts on sixth- and ninth-grade students’ ABC outcomes for the full sample of schools in the study as well as looking at the high school and middle school samples separately. The chapter then turns to related findings, including the impact 

of the Diplomas Now model on the percentages of students who reach specific thresholds on 

each individual ABC outcome, as well as a composite measure; the impacts on subgroups of 

students based on their preparedness before entering middle or high school; the differences in 

the impacts on sixth- and ninth-grade student outcomes after the first and second years of implementation; and the variation in impacts across the randomly assigned blocks of schools. Together, these findings offer a comprehensive view of the effects of the Diplomas Now model 

during the first two years of implementation. 

Ultimately, the goal of Diplomas Now is to get more students to graduate from high 

school. (See the long-term outcomes in Figure 1.1.) A future report from this evaluation will 

discuss whether implementation of the Diplomas Now model had an impact on ninth-graders’ 

high school graduation, as well as sixth-graders’ successful completion of their first year of high 

school. So while the focus of this report is on interim findings about the effect of Diplomas 

Now on students’ trajectory toward graduation, more follow-up is required to fully assess the 

model’s impact.

Key points from this chapter include the following:

• Implementation of Diplomas Now resulted in a positive, statistically significant impact on the percentage of sixth- and ninth-graders without any early 
36

warning indicators — that is, students who had maintained an 85 percent attendance rate or better, were suspended fewer than three days, and passed

English/language arts and math, the subject areas explicitly targeted for support as part of the Diplomas Now model. There was also a small, nonsignificant impact estimated on the percentage of students achieving a more stringent threshold of a 90 percent attendance rate, no suspensions or expulsions,

and passing grades in four core content classes: English/language arts, math, 

history or social studies, and science. 

•

•

•

•

The impacts on both these ABC composite measures increased from the first 

year of implementation to the second year of implementation, and by a statistically significant amount for the higher threshold indicator. This increase 

from Year 1 to Year 2 aligns with the increased service differentiation that 

occurred across the same time period (discussed in the review of implementation findings in Chapter 1). 

Continuous measures of the individual ABC outcomes — overall average attendance, behavior as measured by average number of days disciplined, and 

percentage of core courses passed — provide a snapshot of whether there are 

impacts on average across all students regardless of their position relative to 

the upper and lower thresholds. There were no statistically significant impacts on these measures.

On the threshold measures of the individual ABC outcomes, there were no 

statistically significant impacts on percentages of students achieving the attendance or course performance thresholds, although the impact estimates 

were positive in direction. Neither were there significant impacts on the percentages of students disciplined at either threshold level, although estimates 

indicate that slightly higher percentages of students were disciplined in the 

DN schools. 

Diplomas Now may have had a stronger impact in middle schools than in 

high schools. In middle schools, the impact of Diplomas Now implementation resulted in statistically significant, positive impacts on the percentage of 

students with better than 90 percent attendance, the threshold below which 

students are considered “chronically absent,” and on the percentage of students achieving the lower composite threshold. There were small, positive, 

nonsignificant impacts estimated for the other attendance and composite 

measures and the course performance measures. The effects were consistently more promising than those at the high school level. On average, Diplomas 
37

Now did not produce statistically significant impacts at the high school level

since outcomes improved similarly at DN and non-DN schools. 

• At the high school level, Diplomas Now was more successful at keeping 

students who had better than 90 percent attendance, no suspensions or expulsions, and no core course failures in eighth grade above those thresholds 

in ninth grade than getting other students across them. 

Impacts on Continuous Measures of the ABC Outcomes

Table 4.1 presents the impacts of the Diplomas Now model on continuous measures of attendance, behavior, and course performance for the second cohort of sixth- and ninth-grade students 

in the full analysis sample and separately for the middle school and high school samples.1 Overall, as shown in Panel A, Diplomas Now did not have a statistically significant positive or negative impact on any of the three measures presented, and the impact estimates are small in size. 

Although Diplomas Now is designed to help both middle school and high school students stay, get back, or get on track, middle schools and high schools typically vary in size and 

structure and serve students at different places in their developmental and educational paths. In 

fact, some aspects of the Diplomas Now model differ between middle schools and high schools. 

Thus, impacts on student outcomes were also analyzed for these two groups of schools separately, as presented in Panel B and Panel C of Table 4.1. There were no statistically significant impacts on these three ABC outcome measures at either level. However, the impact estimates at 

the middle school level appear more promising than those at the high school level. 

Impacts on Stability and Early Warning Indicator Thresholds for 

Attendance, Behavior, and Course Performance

While continuous measures of the ABC outcomes provide information about whether Diplomas 

Now had an impact on average across all sixth- and ninth-grade students, Diplomas Now programming uses early warning indicators defined by attendance, behavior, and course performance thresholds to inform interventions for students and adjustments to practices within 

schools. As discussed in Chapter 2, students are defined as being on a stable trajectory or having 

no early warning indicators based on these thresholds. This section discusses the results of analyses of the percentages of students meeting stability thresholds on these measures (students on a

 1

Please see Chapter 2 for a discussion of the analysis samples of schools and students, as well as descriptions of the outcome measures. Box 2.1 explains how to read the impact tables. 
38

(continued)

Table 4.1

Impacts on Continuous Measures of

Attendance, Behavior, and Course Performance, Cohort 2 

Outcome

DN 

Schools

Non-DN Estimated

Impact

Effect

Schools Size

P-Value for

Estimated

Impact

Panel A: Full sample

Attendance

Percentage of enrolled days attended 89.7 89.3 0.4 0.03 0.602

Behavior

Percentage of enrolled days suspended or expelled 1.5 1.2 0.3 0.05 0.177

Course performance

Percentage of core courses passeda 86.6 86.6 0.1 0.00 0.924

Sample size 29 29

Panel B: Middle school sample

Attendance

Percentage of enrolled days attended 92.7 91.4 1.3 0.12 0.186

Behavior

Percentage of enrolled days suspended or expelled 1.2 1.2 -0.1 -0.01 0.842

Course performance

Percentage of core courses passeda 94.5 92.8 1.7 0.10 0.147

Sample size 14 15

Panel C: High school sample

Attendance

Percentage of enrolled days attended 86.9 87.3 -0.4 -0.03 0.704

Behavior

Percentage of enrolled days suspended or expelled 1.8 1.5 0.3 0.05 0.403

Course performance

Percentage of core courses passeda 79.8 79.2 0.7 0.02 0.433

Sample size 15 14
39

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 58 study schools, 14,950 nonrepeating sixth- and ninth-grade students are included in the 

analyses. Among the sample, 6,997 students attended DN schools and 7,953 students attended non-DN schools. 

Some students are not included in the analyses of course performance measures because data were not available on 

their grades for specific courses, and an entire DN middle school was dropped from these analyses because there 

are no baseline course data for the students attending that school. There are no more than 8 percent missing DN 

school students and 8 percent missing non-DN school students for any of the course performance measures.

 Estimated impacts are based on a two-level model with students nested within schools, controlling for random 

assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 

observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 

column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 

covariate values for students in DN schools as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures.

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

The denominator includes all core courses (math, English/language arts, science, or social studies) each student 

attempted during the school year.

Table 4.1 (continued)

promising path to graduation) as well as those with no early warning indicators (students who 

are not primary targets for intervention to get them on track to graduation). As is shown in Table 4.2, there is one measure on which Diplomas Now had a statistically significant impact: the 

percentage of students with no early warning indicators. On average, in the DN schools, just 

over 63 percent of the sixth- and ninth-grade students met the three conditions: better than an 85 

percent attendance rate, fewer than three days of disciplinary action that kept them out of classes, and no failed English/language arts or math courses. At the non-DN schools, 59.5 percent 

of their peers met these thresholds, resulting in an impact of 3.6 percentage points.

Table 4.3 displays the impact findings specific to middle schools. Diplomas Now had 

statistically significant, positive impacts on the percentage of students with an attendance rate 

greater than 90 percent and also on the percentage of students with no early warning indicators. 

The impact estimates for these outcomes were 4.1 percentage points and 5.5 percentage points, 

respectively. On average, there were 193 sixth-grade students per school in the analysis sample; 

therefore, these impact estimates suggest that about 8 more students had better than 90 percent 

attendance and more than 10 students did not have early warning indicators, on average, in DN 

schools compared with non-DN schools. Supplemental analyses that accounted specifically for 

the grade configurations of middle schools in the sample also found positive impacts on attendance, course performance, and composite on-track outcomes, further supporting the impression 

that Diplomas Now is making a difference in middle schools. In particular, these analyses
40

(continued)

 

Table 4.2

Impacts on Threshold Measures of

Attendance, Behavior, and Course Performance, Cohort 2

P-Value for

DN Non-DN Estimated Effect Estimated

Outcome Schools Schools Impact Size Impact

Attendance

Percentage of students who attended over 

90 percent of enrolled days 68.3 67.5 0.8 0.02 0.601

Percentage of students who attended over 

85 percent of enrolled days 79.7 79.3 0.4 0.01 0.787

Behavior

Percentage of students who were ever suspended 

or expelled during year 23.8 21.3 2.5 0.06 0.415

Percentage of students who were suspended or

expelled for 3 or more days 16.8 15.2 1.7 0.05 0.472

Course performance

Percentage of students who had no core course 

failures during yeara 73.5 72.8 0.7 0.02 0.675

Percentage of students who had no math course 

failures during year 84.5 82.1 2.3 0.06 0.233

Percentage of students who had no English/

language arts course failures during year 87.5 86.4 1.0 0.03 0.382

ABC composite measuresb

Percentage of students above stability threshold 50.6 48.1 2.5 0.05 0.210

Percentage of students with no early warning

indicators 63.2 59.5 3.6 0.07 ** 0.039

Sample size 29 29

accounted for whether entry into sixth grade represented a true transition experience, because 

the school’s earliest grade was sixth, or whether it represented more of a continuation, as in the 

few schools with grades 5-8 or grades K-8 configurations. (See Appendix Table B.6.)2

2

Supplemental analyses (or “sensitivity tests”) accounting for other school-level characteristics (for example, percentages of teachers with different certification levels) were conducted and did not reveal different patterns of outcomes from what is presented in this chapter for the main analyses. Various supplemental analyses 

are discussed in Appendix B.
41

The high school findings presented in Table 4.4 are not as promising as those for the 

middle schools. None of the impact estimates presented in this table are statistically significant, 

suggesting that the implementation of Diplomas Now did not result in better outcomes for 

ninth-graders than what was happening in the non-DN schools. There are more positive estimates across the nine threshold measures for middle schools than for high schools, suggesting

that Diplomas Now may be more effective at the middle school level than at the high school 

level. However, for only one measure — the percentage of students with better than 90 percent 

attendance — do the impacts at the middle school and high school levels differ to a statistically 

significant degree.

3 That is, Diplomas Now is having a greater impact on keeping students from 

being chronically absent in middle school than in high school.

 3

The “dagger” to the right of the p-values for the middle school and high school impact estimates on that 

measure indicates that the two impact estimates are statistically different from each other (p ≤ 0.10). This symbol has the same meaning when used in other tables in this report.

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 58 study schools, 14,950 nonrepeating sixth- and ninth-grade students are included in the 

analyses. Among the sample, 6,997 students attended DN schools and 7,953 students attended non-DN schools. 

Some students are not included in the analyses of course performance measures because data were not available 

on their grades for specific courses, and an entire DN middle school was dropped from these analyses because 

there are no baseline course data for the students attending that school. There are no more than 14 percent 

missing DN school students and 9 percent missing non-DN school students for any of the course performance 

measures. The large percentage of missing DN school students is due to one school with a large amount of 

missing math course data. Without this school included, there are no more than 9 percent missing students on 

any of the measures. A sensitivity test removing the random assignment block that includes this school from the 

analyses resulted in similar findings.

 Estimated impacts are based on a two-level model with students nested within schools, controlling for 

random assignment block and school- and student-level covariates. The values in the column labeled “DN 

Schools” are the observed mean outcomes weighted by the number of schools in a random assignment block, 

and the values in the column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN 

schools using the mean covariate values for students in DN schools as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures.

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Core courses include math, English/language arts, science, or social studies courses.

 b

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. 

The “no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the 

district, was suspended or expelled for fewer than three days, and did not fail any math or English/language arts 

courses during the school year. If a student has missing course data but meets the other two criteria for an ABC 

composite measure, the student is assigned a value of 1.

Table 4.2 (continued)
42

(continued)

Outcome

DN Non-DN Estimated Effect

Schools Schools Impact Size

P-Value for

Estimated

Impact

Attendance

Percentage of students who attended over 

90 percent of enrolled days

Percentage of students who attended over 

85 percent of enrolled days

Behavior

Percentage of students who were ever suspended 

or expelled during year

Percentage of students who were suspended or

expelled for 3 or more days

Course performance

Percentage of students who had no core course a failures during year

Percentage of students who had no math course 

failures during year

Percentage of students who had no English/

language arts course failures during year

b ABC composite measure

Percentage of students above stability threshold

Percentage of students with no early warning

indicators

77.2

86.8

22.1

15.6

87.0

94.3

95.5

62.5

74.5

73.0

84.2

21.6

16.1

85.7

93.6

93.5

57.6

69.0

4.1

2.5

0.5

-0.4

1.3

0.7

2.1

4.8

5.5

0.09 *

0.07

0.01

-0.01

0.04

0.02

0.10

0.10

0.12 *

0.073 †

0.300

0.932

0.916

0.631

0.753

0.101

0.181

0.076

Sample size 14 15

Table 4.3

Impacts on Threshold Measures of Attendance, 

Behavior, and Course Performance, Middle Schools, Cohort 2

Impacts for More- and Less-Prepared Students

The Diplomas Now partnership, through implementation of its reform model, intends both to 

get students who are on less stable paths to graduation onto more stable paths and to keep stable

students on track. The evaluation team analyzed the impact of Diplomas Now on the outcomes 

of students entering high school based on whether they met the stability thresholds in eighth 

grade (that is, whether they exceeded the more stringent threshold for the ABC outcomes in
43

SOURCE: MDRC calculations based on student records obtained from school districts. 

 Estimated impacts are based on a two-level model with students nested within schools, controlling for random 

assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are 

the observed mean outcomes weighted by the number of schools in a random assignment block, and the values in 

the column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the 

mean covariate values for students in DN schools as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures. 

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Statistically significant differences in impacts between the middle school and high school groups are indicated 

by † when the p-value is less than or equal to 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Core courses include math, English/language arts, science, or social studies courses.

 b

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 

“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, 

was suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses 

during the school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

NOTES: Across 29 study middle schools, 5,606 nonrepeating sixth-grade students are included in the analyses. 

Among the sample, 2,513 students attended DN schools and 3,093 students attended non-DN schools. Some 

students are not included in the analyses of course performance measures because data were not available on their 

grades for specific courses, and an entire DN middle school was dropped from these analyses because there are no 

baseline course data for the students attending that school. There are no more than 9 percent missing DN school 

students and 12 percent missing non-DN school students for any of the course performance measures.

Table 4.3 (continued)

their last year before entering high school). Because elementary school data across participating 

study districts did not consistently include the disciplinary and course performance metrics used 

for the stability indicators, the evaluation team could not do the same analysis for students entering middle school. Instead, standardized state assessment scores were used to represent students’ preparedness for middle school as a proxy for being on a more stable trajectory or not, 

and the impact of Diplomas Now was analyzed according to whether students entering middle 

school had achieved academic proficiency by the end of elementary school. How did the impact

of the Diplomas Now model differ for students entering high school or middle school based on

their readiness for the next level of school?

Table 4.5 presents the impacts on student outcomes for ninth-graders who were or were 

not above the stability threshold level at the end of eighth grade (see Panel A and Panel B, respectively). The table displays one statistically significant impact. Among the students who 

were above the stability threshold in eighth grade, there is an estimated 8 percentage point impact on the percentage of students who did not fail math in ninth grade. This is equivalent to
44

(continued)

Table 4.4

Impacts on Threshold Measures of Attendance,

Behavior, and Course Performance, High Schools, Cohort 2

Outcome

DN

Schools

Non-DN Estimated

Schools Impact

Effect

Size

P-Value for

Estimated

Impact

Attendance

Percentage of students who attended over 

90 percent of enrolled days

Percentage of students who attended over 

85 percent of enrolled days

Behavior

Percentage of students who were ever suspended 

or expelled during year

Percentage of students who were suspended or

expelled for 3 or more days

Course performance

Percentage of students who had no core course a failures during year

Percentage of students who had no math course 

failures during year

Percentage of students who had no English/

language arts course failures during year

b ABC composite measure

Percentage of students above stability threshold

Percentage of students with no early warning

indicators

60.1

73.1

25.4

18.0

61.8

75.9

80.5

39.6

52.6

61.9

74.5

21.1

15.4

60.8

74.1

78.9

38.9

50.0

-1.8

-1.4

4.3

2.6

1.0

1.8

1.6

0.7

2.6

-0.04

-0.04

0.12

0.08

0.02

0.04

0.04

0.01

0.05

0.472 †

0.383

0.167

0.368

0.531

0.603

0.486

0.726

0.257

Sample size 15 14

about 26 more stable entering ninth-graders not failing math in DN schools compared with nonDN schools. 

Overall, the pattern of results looks different for the two subgroups of students, with the 

results appearing more positive in general for the stable entering ninth-grade students. On one 

measure — the percentage of students at stability levels in ninth grade — the difference in impacts between the two groups of students is statistically significant: Among the students who
45

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 29 study high schools, 9,344 nonrepeating ninth-grade students are included in the analyses. 

Among the sample, 4,484 students attended DN schools and 4,860 students attended non-DN schools. Some 

students are not included in the analyses of course performance measures because data were not available on their 

grades for specific courses. There are no more than 16 percent missing DN school students and 9 percent missing 

non-DN school students for any of the course performance measures. The large percentage of missing DN school 

students is due to one school with a large amount of missing math course data. Without this school included, there 

are no more than 9 percent missing students on any of the measures. A sensitivity test removing the random 

assignment block that includes this school from the analyses resulted in similar findings.

 Estimated impacts are based on a two-level model with students nested within schools, controlling for random 

assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are 

the observed mean outcomes weighted by the number of schools in a random assignment block, and the values in 

the column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the 

mean covariate values for students in DN schools as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures. 

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Statistically significant differences in impacts between the middle school and high school groups are indicated 

by † when the p-value is less than or equal to 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Core courses include math, English/language arts, science, or social studies courses.

 b

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 

“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, 

was suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses 

during the school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

Table 4.4 (continued)

came into high school on a stable path, Diplomas Now had a 5 percentage point impact on the 

percentage who were still maintaining stability levels at the end of the year, compared with a

negative 1.8 percentage point impact among those who had not been on a more stable path in 

eighth grade. This indicates that, in high school, the Diplomas Now model was more effective 

at keeping students on a stable path than getting students on a stable path. 

The evaluation team also analyzed the impact of Diplomas Now for two subgroups of 

middle school students: one group of students who were rated proficient or better on both their 

math and English/language arts state assessments in the prior year, and another group of students rated less than proficient on at least one of those exams. In general, the pattern of impacts 

is similar across both groups, and none of the impacts differ by a statistically significant amount 

between the groups. Of note is that there are relatively large positive impacts for both groups for
46

(continued)

Table 4.5

Impacts on Threshold Measures of

Students Above and Below Stability Threshold in Previous Year,

Attendance, Behavior, and Course Performance,

High Schools, Cohort 2

P-Value for

Estimated

Impact

DN

Schools

Non-DN

Schools

Estimated

Impact

Effect

Outcome Size

Panel A: Students above stability threshold in

previous year

Attendance

Percentage of students who attended over 

90 percent of enrolled days 80.3 79.3 1.0 0.03 0.731

Percentage of students who attended over 

85 percent of enrolled days 90.7 90.4 0.4 0.02 0.898

Behavior

Percentage of students who were ever suspended 

or expelled during year 13.7 13.3 0.4 0.01 0.893

Percentage of students who were suspended or

expelled for 3 or more days 8.6 5.9 2.6 0.13 0.302

Course performance

Percentage of students who had no core course a failures during year 80.4 76.7 3.7 0.08 0.172

Percentage of students who had no math course 

failures during year 90.2 81.9 8.2 0.22 ** 0.018

Percentage of students who had no English/

language arts course failures during year 92.1 90.0 2.1 0.07 0.327

b ABC composite measure

Percentage of students above stability threshold 62.7 57.7 5.0 0.10 0.148 †

Percentage of students with no early warning

indicators 76.4 71.7 4.7 0.11 0.193

Sample size 15 14
47

(continued)

P-Value for

DN Non-DN Estimated Effect Estimated

Outcome Schools Schools Impact Size Impact

Table 4.5 (continued)

Panel B: Students below stability threshold in

previous year

Attendance

Percentage of students who attended over 

90 percent of enrolled days 41.3 45.0 -3.6 -0.07 0.212

Percentage of students who attended over 

85 percent of enrolled days 57.9 59.4 -1.6 -0.03 0.575

Behavior

Percentage of students who were ever suspended 

or expelled during year 38.8 30.9 7.9 0.17 0.144

Percentage of students who were suspended or

expelled for 3 or more days 28.9 24.6 4.3 0.10 0.359

Course performance

Percentage of students who had no core course a failures during year 44.3 43.6 0.7 0.01 0.760

Percentage of students who had no math course 

failures during year 62.3 62.2 0.1 0.00 0.986

Percentage of students who had no English/

language arts course failures during year 69.3 64.9 4.4 0.09 0.219

b ABC composite measure

Percentage of students above stability threshold 18.4 20.2 -1.8 -0.04 0.320 †

Percentage of students with no early warning

indicators 32.1 29.5 2.6 0.06 0.291

Sample size 15 14

the stability and early warning threshold measures of attendance and being on track. For the 

nonproficient group, the 4.6 percentage point impact estimate on the 90 percent attendance stability threshold and the 7 percentage point impact on having no early warning indicators were 

statistically significant. These findings are presented in Appendix Table B.1.
48

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 29 study high schools, 7,500 nonrepeating ninth-grade students are included in the analyses. 

Among the sample, 3,577 students attended DN schools and 3,923 students attended non-DN schools. The abovethreshold subgroup includes 4,328 students and the below-threshold subgroup includes 3,172 students. Some 

students are not included in the analyses of course performance measures because data were not available on their 

grades for specific courses. There are no more than 20 percent missing DN school students and 11 percent 

missing non-DN school students for any of the course performance measures. The large percentage of missing 

DN school students is due to one school with a large amount of missing math course data. Without this school 

included, there are no more than 11 percent missing students on any of the measures. A sensitivity test removing 

the random assignment block that includes this school from the analyses resulted in similar findings.

 Estimated impacts are based on a two-level model with students nested within schools, controlling for random 

assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are 

the observed mean outcomes weighted by the number of schools in a random assignment block, and the values in 

the column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the 

mean covariate values for students in DN schools as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures. 

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Statistically significant differences in impacts between the groups above and below the stability threshold in 

the previous year are indicated by † when the p-value is less than or equal to 10 percent.

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Core courses include math, English/language arts, science, or social studies courses.

 b

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 

“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, 

was suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses 

during the school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

Table 4.5 (continued)

Impacts on ABC Outcomes During the First Year of 

Implementation

Were impacts different for the first cohorts of sixth- and ninth-grade students? Table 4.6 displays the impacts of the Diplomas Now model during the first year of implementation on students’ ABC outcomes — the impacts for the first cohort in each school. As with the second cohort of students, there was one statistically significant impact on the threshold outcomes during 

the first year of implementation. Diplomas Now has an almost 4 percentage point impact on the 

percentage of students who did not fail math in sixth or ninth grade. The estimated impact on 

the percentage of first-cohort students above the ABC composite stability threshold at the end of 

their sixth- and ninth-grade years is negative but not statistically significant. Still, there is a statistically significant difference between the first and second years of implementation on this
49

measure, as represented by the dagger in Table 4.6. Although the difference between DN and 

non-DN schools in the second year was not statistically significant, the effect the Diplomas 

Now model had on this measure had improved by a statistically significant amount from the 

prior year. This may have been driven predominantly by sixth-graders, as there was a similar 

statistically significant improvement in the percentage of students on track in the middle 

schools. (See Panel A of Appendix Table B.3.)

Outcome

DN

Schools

Non-DN Estimated

Schools Impact

Effect

Size

P-Value for

Estimated

Impact

Attendance

Percentage of students who attended over 

90 percent of enrolled days

Percentage of students who attended over 

85 percent of enrolled days

Behavior

Percentage of students who were ever suspended 

or expelled during year

Percentage of students who were suspended or

expelled for 3 or more days

Course performance

Percentage of students who had no core course a failures during year

Percentage of students who had no math course 

failures during year

Percentage of students who had no English/ 

language arts course failures during year

b ABC composite measure

Percentage of students above stability threshold

Percentage of students with no early warning

indicators

66.8

78.2

25.7

18.0

73.2

86.0

86.4

47.3

60.9

68.2

78.9

22.3

16.4

70.9

82.1

86.1

49.9

60.7

-1.4

-0.8

3.4

1.7

2.3

3.9

0.4

-2.6

0.2

-0.03

-0.02

0.09

0.05

0.05

0.10 **

0.01

-0.05

0.00

0.418

0.650

0.276

0.501

0.318

0.029

0.850

0.164 †

0.894

Sample size 31 29

(continued)

Table 4.6

Attendance, Behavior, and Course Performance, Cohort 1

Impacts on Threshold Measures of
50

 

SOURCE: MDRC calculations based on student records obtained from school districts.

NOTES: Across 60 study schools, 15,280 nonrepeating sixth- and ninth-grade students are included in the analyses. 

Among the sample, 7,322 students attended DN schools and 7,958 students attended non-DN schools. Some 

students are not included in the analyses of course performance measures because data were not available on their 

grades for specific courses, and an entire DN middle school was dropped from these analyses because there are no 

baseline course data for the students attending that school. There are no more than 9 percent missing DN school 

students and 10 percent missing non-DN school students for any of the course performance measures.

 Estimated impacts are based on a two-level model with students nested within schools, controlling for random 

assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 

observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 

column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 

covariate values for students in DN schools as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures. 

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Statistically significant differences in impacts between the Cohort 1 and Cohort 2 groups are indicated by † 

when the p-value is less than or equal to 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Core courses include math, English/language arts, science, or social studies courses.

 b

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 

“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 

suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 

the school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

Table 4.6 (continued)

There was a small increase in the average percentage of students on a stable path at DN 

schools between the first and second year. As shown in Table 4.6, on average at DN schools, 

47.3 percent of the first-cohort students were achieving the composite stability threshold at the 

end of the first year of implementation, while in the second year of implementation, 50.6 percent of the second-cohort students achieved that threshold (see Table 4.2). At the non-DN 

schools, there was a slight decline in the percentage of stable students from Year 1 to Year 2, 

from 49.9 percent to 48.1 percent.4 This increase in DN schools and decrease in non-DN 

schools mirrors some of the service contrast findings that grew between the first and second 

years, owing, in part, to a decline in the non-DN schools’ average service levels, as discussed in 

4

Although the impact estimates for the first and second cohorts of students on the lower threshold on-track 

indicator (that is, the percentage of students with no early warning indicators) do not differ to a statistically 

significant degree, there is a similar pattern, in which the percentage of students exceeding the lower threshold 

increases in the DN schools and decreases in the non-DN schools.
51

Chapter 1.5 Although there are only two time points, this growth in the percentage of students 

on track from the first year to the second year offers some promise for future growth in impacts 

as the DN school staff members build their understanding and implementation of the model. 

Variation in Impacts Across Schools

Table 4.2 shows the average impact of the Diplomas Now model on the percentage of students 

on a stable path during the second year of implementation. As discussed earlier in this chapter, 

on average across all the blocks of DN and non-DN schools in the model, the Diplomas Now 

model does not show a statistically significant impact on the percentage of students above the 

composite stability threshold. Still, it is possible that for some DN schools the model could be 

generating more of an impact than in others. Table 4.2 also shows a statistically significant impact on the percentage of students with no early warning indicators, the less stringent threshold 

for the ABC composite measure. It is possible that the overall impact on this measure is driven 

more by implementation in some DN schools compared with others. 

To investigate this possibility, Figure 4.1 and Figure 4.2 display the impact estimates 

for each of the random assignment blocks in the study for the upper and lower threshold ontrack indicators. As discussed in Chapter 2, random assignment blocks are a set of like 

schools within a school district that were randomly assigned at the same time to implement 

Diplomas Now or not to implement Diplomas Now. The blocks range from two to six 

schools, and each block has both DN and non-DN schools, but only middle schools or only 

high schools. There are 21 blocks in the second-year analytic sample. In Figure 4.1, statistically significant differences in the percentage of stable students in DN schools compared with 

non-DN schools are indicated by vertical bars that do not cross the 0 line. As the figure indicates, only two of the random assignment blocks show statistically significant, positive mean 

differences, even though over half the blocks had positive mean differences (shown by the 

circle or square being above the 0 line). In Figure 4.2, three of the random assignment blocks 

show statistically significant, positive mean differences on the percentage of students with no 

early warning indicators while over half the blocks had positive mean differences. Although 

there is some variation in the impacts across the blocks in each of the figures, a test across all 

blocks indicates that this variation across schools is not statistically significant. This suggests 

that there was not a lot of variation in DN schools’ ability to effect change, and while more 

blocks show a positive difference for DN schools compared with their non-DN counterparts, 

this variation is small and may be due to chance.

 5

For in-depth comparison of service contrast between Year 1 and Year 2, see Sepanik et al. (2015), pp.

48-54.
52

Figure 4.1

Impacts on Percentage of Students Above Stability Threshold,

by Random Assignment Block, Cohort 2

-50-40-30-20-10

0

1020304050

MMHi

Estimated impact (percentage points)
53

Figure 4.1 (continued) 

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 58 study schools, there are 21 random assignment (RA) blocks. 

 These fixed-effects impact estimates are based on a two-level model with students nested within 

schools controlling for random assignment block and school- and student-level covariates. The vertical 

error bars represent a 90 percent confidence interval for each data point.

 A composite F-test was used to assess whether the variation in impacts across random assignment 

blocks is larger than would be expected due to chance. The result is not statistically significant.

 The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in 

the district, was never suspended or expelled, and did not fail any core courses attempted during the 

school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

Conclusion 

In the second year of its implementation under i3, the Diplomas Now model did not have statistically significant impacts on sixth- and ninth-grade students’ average attendance rates, days 

seriously disciplined, or course passing rates. In terms of the composite measures of attendance, 

behavior, and course performance there was no statistically significant impact on the percentage 

of stable students, but there was a small statistically significant impact on the percentage of students with no early warning indicators. However, the impact story becomes more nuanced when 

one looks specifically at middle schools or high schools. There are signs that implementation of 

the Diplomas Now model may have led to some improvement on the ABC outcomes of middle 

school students, particularly regarding attendance and having no early warning indicators, and 

the effect was similar for students regardless of their academic proficiency level coming into 

middle school. At the high school level, the Diplomas Now model appears not to have much 

impact compared with what was happening in the non-DN schools. Within high schools, however, there appears to be some differentiation in how well Diplomas Now may be meeting the 

needs of students coming into high school who are on more or less stable paths to graduation. It 

does appear that the implementation of the Diplomas Now model has helped stable students 

stay on that trajectory more than it has helped other students get onto that trajectory. 

Although the Diplomas Now model did not have an overall impact on most of the ABC 

outcomes measured by this study during the second year of implementation, the program is 

complex and could take more than two years to fully take hold in a school in a way that leads to
54

Figure 4.2

Impacts on Percentage of Students with No Early Warning Indicators,

by Random Assignment Block, Cohort 2

-50-40-30-20-10

0

1020304050

MeMiHig

Estimated impact (percentage points)
55

Figure 4.2 (continued) 

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 58 study schools, there are 21 random assignment (RA) blocks.

 These fixed-effects impact estimates are based on a two-level model with students nested within 

schools controlling for random assignment block and school- and student-level covariates. The vertical 

error bars represent a 90 percent confidence interval for each data point.

 A composite F-test was used to assess whether the variation in impacts across random assignment 

blocks is larger than would be expected due to chance. The result is not statistically significant.

 The “no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled 

in the district, was suspended or expelled for fewer than three days, and did not fail any math or

English/language arts courses during the school year. If a student has missing course data but meets

the other two criteria for an ABC composite measure, the student is assigned a value of 1.

impacts on student outcomes.6 The impact on the percentage of students with no early warning 

indicators (see Table 4.2) suggests that Diplomas Now is positively affecting the percentage of 

students meeting incremental thresholds that align with outcome levels targeted by the Diplomas Now staff members. Still, the model did not show a statistically significant impact on students meeting any of the separate early warning indicator thresholds specific to attendance, behavior, or course performance. Few of the early school and student outcomes measured show a 

difference between DN and non-DN schools after the first year, but it is possible that these hypothesized “early” outcomes could take longer to be fully realized since changing school climate and student attitudes and behaviors is not generally a short-term endeavor. The one early 

student outcome with a positive impact is directly related to the model components that involved the placement of Diplomas Now staff in schools — increasing the number of adults in 

the building — and that involved some of the specific activities these staff members supported. 

This outcome suggests that the model is causing some change in students’ experiences even 

after just the first year. Furthermore, the impact of the Diplomas Now model on the percentage

of students on a stable path to graduation was stronger in the second than in the first year, suggesting that although not yet statistically significant after two years, the program effects may be

growing. But based on data for only two years, it is still too early to know for sure.

The ultimate goal of the Diplomas Now model is to help more high school students 

graduate and more middle school students successfully complete ninth grade on time. Other 

than the estimated impact on the percentage of students with no early warning indicators, there 

are no statistically significant impacts on the intermediate ABC outcomes hypothesized to lead 

to these long-term goals, but the exploratory analyses in this chapter offer some tentative signs 

 6

For example, Borman, Hewes, Overman, and Brown (2003) conducted a meta-analysis of whole-school 

reform models and found that those implemented for five years or more had greater impacts than those implemented for fewer years.
56

that impacts on longer-term outcomes could emerge, particularly for the middle school students. 

If affecting students around the early warning indicator level is most important, then having a 

positive impact on the lower threshold ABC composite measure may bode well for keeping or 

getting more students on track through graduation and successful ninth-grade completion.

7

Whereas if having impacts on specific, individual ABC outcomes ends up being a more important predictor of later success, impacts on longer-term outcomes may be less likely.8 It is also 

possible that the model takes more than two years to fully develop the capability to cause positive change at schools, and the growth between the first and second cohorts, if part of a trend, 

could lead to more positive ABC impacts in future years.

The next report will investigate relationships between implementation and outcome

data. A final report will present the same ABC outcomes as this report but for the fourth cohort 

of sixth- and ninth-grade students, answering the question of whether Diplomas Now could be 

more effective for entering students after two more years of implementation in the schools. It 

will also look at the four-year impacts of Diplomas Now on Cohort 1 and Cohort 2 high school 

students’ graduation and middle school students’ completion of ninth grade.

 7

Frazelle and Nagel (2015, pp. 7-8) give examples of multi-indicator Early Warning Systems from the

National High School Center and Johns Hopkins University. 8

For example, on-track indicators used in Chicago and New York City are based on course and test performance and do not include attendance and disciplinary outcomes in their definitions. Allensworth and Easton 

(2005) from the Consortium on Chicago School Research define ninth-graders as being on track to graduation 

based solely on course-related outcomes: earning at least five course credits and no more than one semester F 

in a core course. Kemple, Segeritz, and Stephenson (2013) from the Research Alliance for New York City 

Schools discuss the New York City Department of Education’s on-track measure, which is similar to Chicago’s and uses credit earning to define being on track; they also discuss how passing at least one Regents exam 

(a subject area assessment administered in New York State) in ninth grade improves the predictability of 

whether a New York City ninth-grader will earn a Regents high school diploma. 
Appendix A 

Samples, Analytic Methods, and

Early Outcome Measures

59

Chapter 2 summarizes the data, measures, samples, and analyses used in this report. This

appendix offers additional information in each of these areas, including data sources, response

rates, and sample construction for the ABC outcomes discussed in Chapter 4 and the early

school and student outcomes discussed in Chapter 3; the analytic model used for all the analyses; the approximate statistical power for detecting effects on the ABC outcomes; and a 

description of the construction of the early school and student outcome measures using administrator, teacher, and student survey items. 

Data Sources, Response Rates, and Samples for ABC Outcomes

As discussed in Chapter 2, the Diplomas Now intervention uses a school-level random assignment design, but student-level data are used in the analyses of the ABC outcomes. All data used

for the impact analyses discussed in Chapter 4 and Appendix B were obtained from individual

administrative student records collected from each of the 11 participating school districts, as

described in Chapter 2. Student administrative records data were collected for students who

were enrolled in grades six and nine in the study schools during the first and second years of

Diplomas Now implementation (2011-2012 and 2012-2013 for the schools recruited in the first 

wave and 2012-2013 and 2013-2014 for schools recruited in the second wave). For these two

cohorts of students, data were also collected from the year before student enrollment in study

schools (fifth-grade data for middle school students and eighth-grade data for high school

students) for use in the baseline analyses of state test proficiency found in Table 2.2 and as

covariates in the impact analyses. In addition, administrative student records data were collected

on sixth- and ninth-grade students who attended the study schools the year before implementation (2010-2011 for wave 1 schools and 2011-2012 for wave 2 schools). These data were used

to compare DN and non-DN schools at baseline on the outcome measures to confirm that

random assignment produced comparable groups. The findings from this analysis are shown in

Table 2.1. Aggregate measures of the ABC outcomes at baseline were also created from these

data and used as covariates in the impact estimation models. 

Sixty-two schools, including 29 high schools and 33 middle schools, were randomly assigned within blocks either to implement the Diplomas Now model (DN schools) or not to 

implement the Diplomas Now model (non-DN schools). The schools not assigned to implement

the model could continue their programming or implement any other reforms. Over the first two

years of program implementation, a few schools were dropped from the study because of school

closures or differences in grade configurations between DN and non-DN schools in the same

random assignment block. Appendix Table A.1 presents the number of schools in each ABC

outcome analysis and indicates what percentage of schools originally randomly assigned that

number represents. Panel A presents these school-level response rates for the main analysis
60

sample, representing the second cohort of sixth- and ninth-grade students to attend the study

schools, while Panel B presents the school-level response rates for the first cohort. 

Specifically, two schools were dropped from all analyses, two were dropped from the 

second year of analyses, and one school was excluded from all course analyses, while one 

random assignment block used first-year data: 

DN Schools Non-DN Schools All Schools

Analysis Percentage Analysis Percentage Analysis Percentage

Sample of RA Sample of RA Sample of RA

Outcome Measure a Size Schools a Size Schools a Size Schools

Panel A: Cohort 2

Attendance, behavior, and stability

threshold

All schools 29 90.6 29 96.7 58 93.5

Middle schools 14 82.4 15 93.8 29 87.9

High schools 15 100.0 14 100.0 29 100.0

Course performance

All schools 28 87.5 29 96.7 57 91.9

Middle schools 13 76.5 15 93.8 28 84.8

High schools 15 100.0 14 100.0 29 100.0

Panel B: Cohort 1

Attendance, behavior, and stability

threshold

All schools 31 96.9 29 96.7 60 96.8

Course performance

All schools 30 93.8 29 96.7 59 95.2

Number and Percentage of Randomly Assigned Schools in Each Analysis Sample

Appendix Table A.1

SOURCE: MDRC calculations based on records obtained from school districts (2012, 2013, and 2014). 

NOTES: One DN middle school and one non-DN middle school were removed from both years of analyses due to 

differences in grade configurations within the same random assignment block. One DN middle school closed prior 

to the second year of implementation, and one DN middle school was not included in the second-year analyses 

because it no longer served sixth grade. In addition, one DN middle school was not included in the course 

analyses for either year because the school did not provide baseline course data. aValues in column represent the percentage of randomly assigned (RA) schools used in the analysis sample for 

a given outcome measure.
61

•

•

•

•

Two middle schools from the same random assignment block, one DN school and

one non-DN school, were not included in any analyses because one school served

only seventh and eighth grade while the other school served sixth through eighth

grade. The study focuses on the transition year (the first grade level that students attend the school), and in these two schools, the transition years were not comparable. 

Two middle schools were not included in the analyses of the second cohort: One

DN school closed before the second year of implementation, and another DN

school stopped serving sixth grade before the second year of implementation, so it

was no longer comparable to the other schools in the random assignment block that

kept serving sixth grade. 

In both the first and second cohorts, one DN middle school was dropped from all

course performance analyses because baseline course performance data, which

were used to create a school-level covariate included in the course performance

analyses, were not available for that school. 

In one random assignment block with two DN schools and one non-DN school, the

two DN schools stopped serving sixth grade after the first year of implementation.

In order to keep the block in the analyses, the Cohort 1 data for all three schools are

used in the impact analyses for Cohort 2.

The data received from the districts and used in the analyses represent the study team’s

best knowledge of the full sample of students present at the schools. Only nonrepeating sixthand ninth-grade students are included in the analyses because the focus of the Diplomas Now

model is on incoming students during their transition year. There are 14,950 nonrepeating sixthand ninth-grade students in Cohort 2. Among the sample, 6,997 students attended DN schools

and 7,953 students attended non-DN schools. Students were included in the study regardless of 

how long they attended the study school or other schools in the district, but some students did

not have course performance data because they did not attend the school long enough to have

received course grades. A total of 652 DN school students and 621 non-DN school students in

Cohort 2 were dropped from the course performance analyses because of missing course

performance data (including the students in the aforementioned school that was dropped from

the course performance analyses due to missing baseline data).

Data Sources, Response Rates, and Samples for

Early School and Student Outcomes

As discussed in Chapter 2, data from administrator, teacher, and student surveys were used in

the analyses of the early school and student outcome measures. Teacher and administrator
62

surveys were administered online or in pencil-and-paper format during the spring of the second

year of implementation (2013 for wave 1 schools and 2014 for wave 2 schools). Student

surveys were administrated using pencil-and-paper format during the spring of the first year of

implementation (2012 for wave 1 schools and 2013 for wave 2 schools).1 Copies of the teacher,

administrator, and student surveys can be found in the Survey Instrument Supplement to this

report.2 All schools participated in the student survey conducted in the spring of the first year of

implementation and are included in the analyses of student survey data. In schools that did not

have sixth grade, seventh-grade student data were collected and included in the analyses. As 

noted previously, by the second year of implementation, one DN school had closed, so it is not

included in the analyses of teacher and administrator early school outcomes, and one school was

dropped from the teacher survey analyses because it no longer included a sixth grade in the

second year of implementation.

Appendix Table A.2 describes the individual-level response rates and sample sizes for

the teacher, administrator, and student surveys used in analyses of early school and student

outcomes. The sample sizes represent the total number of teachers, administrators, and sixthand ninth-grade students at the schools, and the response rates are the percentages of those

groups that participated in the surveys. 

The construction of the analytic samples for the early outcome analyses from the pool

of eligible administrators and teachers is presented in Appendix Figures A.1 and A.2, respectively. Appendix Figure A.1 illustrates that 217 eligible administrators (principals and assistant

principals) across the study schools were identified during the second year of implementation.

Of these eligible administrators, 182 (84 percent) responded to the survey questionnaire.

Although the study team tried to identify only principals and assistant principals to participate in

the survey, 12 respondents listed positions other than principal or assistant principal or similar

administrative positions. These 12 respondents were dropped from the analysis. 

Appendix Figure A.2 shows that 3,128 eligible teachers were identified during the second year of implementation. Of these eligible teachers, 2,418 (77 percent) responded to the

survey questionnaire. Again, the study team tried to identify only teachers to participate in the

survey, but 130 respondents listed their position as something other than teacher (for example,

counselor or academic adviser). These respondents were dropped from the analyses. Since the

focus of this report is on the impact during the transition year, the primary analysis sample is

 1

Survey data files and administrative records data are not linked at the individual student level because

research approval policies in many participating districts regarding parent and student consent for survey

participation and data collection led the evaluation team to collect anonymous survey data. 2

Corrin, Sepanik, Rosen, and Shane (2016), available on MDRC’s website (www.mdrc.org). 
63

 

DN

Schools

Non-DN

Schools

All

Schools

Panel A: Teachers

Year 2 response rate (%) 75.6 78.8 77.3

Sample size 1,476 1,652 3,128

Panel B: Administrators

Year 2 response rate (%) 84.5 83.2 83.9

Sample size 110 107 217

Panel C: Students

Year 1 response rate (%) 68.7 63.0 65.6

Sample size 5,935 7,168 13,103

Appendix Table A.2

Individual-Level Response Rates for

Teacher, Administrator, and Student Surveys

SOURCE: Survey administration documentation (2012, 2013, 2014). 

NOTES: One DN middle school closed prior to the second year of implementation and is not included 

in the analyses of teacher and administrator surveys. One DN middle school was not included in the 

analyses of teacher surveys because it no longer served sixth grade during the second year of 

implementation. All schools participated in the student survey during the first year of implementation. 

composed of the 1,339 teachers who taught sixth or ninth grade. (As noted previously, in 

schools where there was no sixth grade, seventh-grade teachers were included.) Teachers who

did not report teaching these grades were not included in the analyses. 

Analytic Models

Impact estimates for the ABC outcomes were obtained by using a two-level fixed-effects

model, which accounts for the nesting of students in schools and districts. Level 1 specifies the

relationship between student-level characteristics and outcomes for students within schools.

This model is given by: 

YYiiii = ππ0ii + ∑ ππ1ssXXssiiii + ∑λλssMMssiiii + eeiiii, (1)

where
64

Expected number of administrator 

survey respondents 

(n = 217)

Eligibility

Pool

Respondent 

Sample

Number of survey respondents 

(n = 182)

Nonadministrator 

respondents 

dropped from 

sample

(n = 12) 

Analysis sample:

high school and middle school 

administrators

Analysis

Sample

(n = 170) 

YYiiii is an outcome for student i in school j; 

XXiiii is a set of S student-level covariates for student i in school j, measured before

students’ first exposure to the Diplomas Now program; 

MMss is a set of S missing indicators for each of the student-level characteristics,

coded 1 for students missing data on that characteristic and 0 otherwise; and 

eeiiii is the error term for student i from school j, assumed to be independently and 

identically distributed across students within schools. 

Therefore,

ππ0 is the average student outcome Y at school j in the sample.

Level 2 focuses on the variation in outcomes between schools. Because random assignment occurred at the school level, the treatment indicator is a level 2 variable that estimates

the treatment effect (that is, the impact of Diplomas Now) at the school level. Level 2 also

includes block-level fixed effects that serve to control for the variation in estimates between

schools and school levels (middle school or high school) in the same district. The school-level

equation is given as:

Appendix Figure A.1

Construction of the Sample of Administrators 

for the Analyses of Early School Outcomes
65

Appendix Figure A.2 

Construction of the Sample of Teachers for the Analyses of Early School Outcomes 

Expected number of teacher 

survey respondents 

(n = 3,128)

Eligibility

Pool

Respondent 

Sample

Teacher

Sample

Number of survey respondents 

(n = 2,418) 

Nonteacher

respondents 

dropped from 

sample

(n = 130) 

Teacher respondent sample

(n = 2,288) 

Analysis

Sample

Analysis sample: 

6th- and 9th-grade teachers

(n = 1,339)

ππ0ii = ∑ δδkkDDkk + ∑ ββ1kkTTiiDDkk + ∑ ββ2mmWWmmii + uu0ii KK KK MM , (2)

where

TT is equal to 1 if school j was randomly assigned to implement the Diplomas Now

program and 0 otherwise; 

DDkk denotes random assignment block indicators equal to 1 if school j is in random 

assignment block k and 0 otherwise; 

WWmm is a set of M school-level covariates for school j measured before the first year

of program implementation; and 

uu0ii is a random error term for school j, assumed to be independently and identically

distributed across schools. 

Therefore, 

ββ1kk is the difference between the school-level average of outcome Y in the DN

schools and the non-DN schools in block/district k, 

and the two-level model can be estimated by substituting equation (2) into equation (1), and

then fitting equation (3):

YYiiii = ∑KK δδkkDDkk + ∑KK ββ1kkTTiiDDkk + ∑MM ββ2mmWWmmii + ∑SS ππ1ssXXssiiii + uu0ii + eeiiii. (3)
66

With these models, the average impact of the Diplomas Now program across random

assignment blocks (ββ1 ���) is obtained by weighting the block-level impacts (ββii ) by the number of

DN schools in the block. Thus, ββ1 ��� is a fixed-effects estimate of the impact of the Diplomas Now

program for the average DN school in the study sample. Therefore, the average estimate cannot

be used to make statistical inferences about the impact of the program in some larger population

of schools. This fixed-effects approach to obtaining a pooled impact estimate is used because

the school districts in the study were selected purposefully and are not a random sample of

districts from a larger target population. In addition, the model includes student-level covariates

(Xi j) to reduce both within- and between-school variation in the outcome measure, which

increases the precision of the impact estimates. The student-level covariates include race,

gender, whether a student is overage for grade, special education status, English language

learner status, previous year math and English state standardized test proficiency level, and a 

previous year version of the outcome measure when it was available.3 For each outcome, one

school-level covariate (Wm j) — a baseline measure of the outcome — was also included. It

reduces between-school variation in outcomes and increases the precision of the impact

estimates. 

Similar two-level fixed-effects models were also used to obtain the difference estimates

for the early school and student outcomes reported in Chapter 3, accounting for nesting of

students, teachers, or administrators within schools and districts. Student, teacher, and administrator characteristics were included as covariates at the individual level. These covariates were

all created from survey items, and they include students’ race and gender and whether a student

is overage for grade; teachers’ degree level, certification status, and years of experience; and

administrators’ years of experience. No school-level covariates were included in the analyses of

early school and student outcomes. 

Statistical Power for Analyses of ABC Outcomes

The goal of this evaluation was to be able to detect impacts on student ABC outcomes that are

policy relevant. To do this the study team had to ensure that a minimum number of schools

were enrolled in the study and randomly assigned. The original goal was to recruit a total of 80

schools, 40 middle schools and 40 high schools. In the end, the program was able to recruit 62

schools; 58 of those schools are included in the Cohort 2 analyses (29 middle schools and 29

high schools). Appendix Table A.3 reports the approximate minimum detectable effect (MDE)

 3

Fifth-grade data do not generally include suspension/expulsion data or course data. Those measures are

not included as covariates in the main analyses or in the middle school subgroup analyses but are included

as covariates in the high school subgroup analyses.
67

(continued)

Appendix Table A.3

Minimum Detectable Effect and Effect Size for Impacts on

ABC Outcomes, Cohort 2

Number of

Outcome Schools MDE MDES

Panel A: All schools

Attendance

Percentage of enrolled days attended 58 1.76 0.13

Percentage of students who attended over

90 percent of enrolled days 58 4.03 0.09

Percentage of students who attended over 

85 percent of enrolled days 58 3.40 0.09

Behavior

Percentage of enrolled days suspended or expelled 58 0.55 0.10

Percentage of students who were ever suspended 

or expelled during year 58 7.81 0.20

Percentage of students who were suspended or

expelled for 3 or more days 58 5.95 0.18

a Course performance

Percentage of core courses passed 57 2.20 0.09

Percentage of students who had no core course 

failures during year 57 4.44 0.10

Percentage of students who had no math course 

failures during year 57 4.91 0.13

Percentage of students who had no English/

language arts course failures during year 57 3.04 0.09

b ABC composite measure

Percentage of students above stability threshold 58 5.02 0.10

Percentage of students with no early warning

indicators 58 4.20 0.09
68

and minimum detectable effect size (MDES) for the ABC outcome analyses for Cohort 2 across

all schools and for middle schools and high schools separately. Intuitively, the MDE is the

smallest program impact that could be estimated with confidence, given random sampling and

estimation error.4 As can be seen in Appendix Table A.3, the study is able to detect an impact of

 4

An MDE is defined as the smallest true program impact that would have an 80 percent chance of being 

detected (have 80 percent power) using a two-tailed hypothesis test at the 5 percent level of statistical 

significance.

Number of

Outcome Schools MDE MDES

Panel B: Middle schools

Attendance

Percentage of enrolled days attended 29 2.55 0.22

Percentage of students who attended over

90 percent of enrolled days 29 5.48 0.13

Percentage of students who attended over 

85 percent of enrolled days 29 6.27 0.18

Behavior

Percentage of enrolled days suspended or expelled 29 0.70 0.13

Percentage of students who were ever suspended 

or expelled during year 29 15.08 0.35

Percentage of students who were suspended or

expelled for 3 or more days 29 10.40 0.28

Course performancea

Percentage of core courses passed 28 2.95 0.17

Percentage of students who had no core course 

failures during year 28 7.17 0.21

Percentage of students who had no math course 

failures during year 28 5.61 0.21

Percentage of students who had no English/

language arts course failures during year 28 3.04 0.14

ABC composite measureb

Percentage of students above stability threshold 29 9.06 0.18

Percentage of students with no early warning

indicators 29 7.44 0.16

(continued)

Appendix Table A.3 (continued)
69

(continued)

Number of

Outcome Schools MDE MDES

Panel C: High schools

Attendance

Percentage of enrolled days attended 29 3.12 0.22

Percentage of students who attended over

90 percent of enrolled days 29 6.74 0.15

Percentage of students who attended over 

85 percent of enrolled days 29 4.28 0.11

Behavior

Percentage of enrolled days suspended or expelled 29 0.81 0.15

Percentage of students who were ever suspended 

or expelled during year 29 7.84 0.21

Percentage of students who were suspended or

expelled for 3 or more days 29 7.58 0.24

a Course performance

Percentage of core courses passed 29 2.24 0.08

Percentage of students who had no core course 

failures during year 29 4.38 0.09

Percentage of students who had no math course 

failures during year 29 9.60 0.22

Percentage of students who had no English/

language arts course failures during year 29 6.05 0.15

b ABC composite measure

Percentage of students above stability threshold 29 5.42 0.11

Percentage of students with no early warning

indicators 29 5.85 0.12

Appendix Table A.3 (continued)

the Diplomas Now model on the percentage of enrolled days attended of 1.76 percentage points.

This suggests that if DN schools had an attendance rate at least 1.76 percentage points higher

than the non-DN schools’ attendance rate, the impact would be statistically significant. Since

only half the schools are middle schools, the DN schools would need to show an impact of at

least 2.55 percentage points in the middle school analyses (see Panel B) for the outcome to be

statistically significant. These MDEs and MDESs represent the best estimation, but they are
70

 

Appendix Table A.3 (continued)

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: The minimum detectable effect (MDE) and minimum detectable effect size (MDES) in 

this table are calculated based on the standard error of the impact estimate (adjusted for random 

assignment blocks, school-level baseline measure of the outcome, and student baseline 

characteristics) and the number of schools in the sample. A statistical significance level of 10 

percent is assumed. The MDES is calculated by dividing the MDE by the standard deviation of the 

outcome measure for non-DN schools. 

Even though the MDE for a given outcome is x, an estimated impact smaller than x can still be 

found to be statistically significant. This is because the calculation of the MDE incorporates not 

only the probability of making a Type I error (that is, concluding that there is an impact when in 

fact there is not) but also the probability of making a Type II error (that is, concluding that there is 

no impact when in fact the program was effective). a

Core courses include math, English/language arts, science, or social studies courses.

bThe “above stability threshold” measure is 1 if a student attended over 90 percent of days 

enrolled in the district, was never suspended or expelled, and did not fail any core courses attempted 

during the school year. The “no early warning indicators” measure is 1 if a student attended over 85 

percent of days enrolled in the district, was suspended or expelled for fewer than three days, and did 

not fail any math or English/language arts courses during the school year. If a student has missing 

course data but meets the other two criteria for an ABC composite measure, the student is assigned 

a value of 1.

approximate, and it is possible that an impact smaller than the one listed in this table could be

found to be statistically significant.5

 

Measure Creation and Factor Analysis of Early School and

Student Outcomes 

The tables in Chapter 3 compare the early school and student outcomes between DN and nonDN schools. These exhibits are based on surveys of teachers, administrators, and students. In

some cases, single survey items were used in the analyses, but in other cases, survey items were

combined to create a single measure. The following discussion describes the measure creation

and factor analyses for measures that were constructed out of more than one survey item.

Items from Table 3.1

Under “more positive school climate,” Table 3.1 includes the teacher survey measure:

“Teachers reported rules for student behavior were consistently reinforced by administrators

and other teachers.” The value for the construct was calculated by taking an average of the

5

A smaller impact could be statistically significant because the calculation of the MDE incorporates not

only the probability of making a Type I error (that is, concluding that there is an impact when in fact there is

not) but also the probability of making a Type II error (that is, concluding that there is no impact when in

fact the program was effective).
71

following survey item responses and then rescaling to a 0-10 scale. (Two items, Cronbach’s

alpha = 0.79.)

To the best of your ability, please indicate how frequently the following activities

occurred at your school during the [current] school year.

͟ Rules for student behavior were consistently reinforced by administrators.

͟ Rules for student behavior were consistently reinforced by other

teachers.

(Scale: 1 = Never, 2 = Rarely, 3 = Sometimes, 4 = Often, 5 = Always, as needed) 

Under “increased parent and community involvement,” Table 3.1 includes the teacher survey

measure: “Teachers reported their school had effective communication with and participation

from students’ families and the local community.” The value for the construct was calculated by

taking an average of the following survey item responses and then rescaling to a 0-10 scale.6

Responses that stated “Not offered at the school this year” were coded as 0. (Four items,

Cronbach’s alpha = 0.91.)

For the [current] school year, how would you rate your school on each of the

following?

͟ Having effective communications from school-to-home and from hometo-school about school programs and student progress.

͟ Providing a wide range of volunteer opportunities for parents and community members to support the school.

͟ Involving families with their children on homework and other curriculum-related activities and decisions, such as academic goals or plans for

work or college. 

͟ Coordinating resources and services both from the community and to the

community.

(Scale: 0 = Not offered at the school this year, 1 = Poor, 2 = Fair, 3 = Good, 4 = 

Excellent)

Table 3.1 also includes the teacher survey measure: “Teachers reported parents were involved

in school activities and supported school reform efforts.” The value for the construct was

 6

These survey items come from Markow and Pieters (2012).
72

calculated by taking an average of the following survey item responses and then rescaling to a 

0-10 scale. (Two items, Cronbach’s alpha = 0.83.)

To what extent would you disagree or agree with each of the following statements

about your experiences at this school during the [current] school year?

͟ Parents/guardians were involved in school activities (e.g., health fairs,

fundraisers, sporting events).

͟ Parents/guardians supported school reform efforts, both those planned

and implemented (e.g., scheduling models, curriculum selection). 

(Scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly agree)

Table 3.1 also includes the teacher survey measure: “Teachers reported community members

were involved in school activities and supported school reform efforts.” The value for the

construct was calculated by taking an average of the following survey item responses and then

rescaling to a 0-10 scale. (Two items, Cronbach’s alpha = 0.83.)

To what extent would you disagree or agree with each of the following statements

about your experiences at this school during the [current] school year?

͟ Community Members were involved in school activities (e.g., health

fairs, fundraisers, sporting events).

͟ Community Members supported school reform efforts, both those

planned and implemented (e.g., scheduling models, curriculum selection). 

(Scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly agree)

Table 3.1 includes the administrator survey measure: “Administrators reported parents were

involved in school activities and supported school reform efforts.” The value for the construct

was calculated by taking an average of the following survey item responses and then rescaling

to a 0-10 scale. (Two items, Cronbach’s alpha = 0.79.)

To what extent would you disagree or agree with each of the following statements

about your school during the [current] school year?

͟ Parents/guardians were involved in school activities (e.g., health fairs,

fundraisers, sporting events).
73

͟ Parents/guardians supported school reform efforts, both those planned

and implemented (e.g., scheduling models, curriculum selection). 

(Scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly agree)

Table 3.1 also includes the administrator survey measure: “Administrators reported community

members were involved in school activities and supported school reform efforts.” The value for

the construct was calculated by taking an average of the following survey item responses and

then rescaling to a 0-10 scale. (Two items, Cronbach’s alpha = 0.80.)

To what extent would you disagree or agree with each of the following statements

about your school during the [current] school year?

͟ Community Members were involved in school activities (e.g., health

fairs, fundraisers, sporting events).

͟ Community Members supported school reform efforts, both those

planned and implemented (e.g., scheduling models, curriculum selection). 

(Scale: 1 = Strongly disagree, 2 = Disagree, 3 = Neither agree nor disagree, 4 = 

Agree, 5 = Strongly agree)

Items from Table 3.2

Under “more positive school climate,” Table 3.2 includes the student survey measure:

“The school had problems with bullying/cyber bullying, students fighting, students cutting

class, and teachers not being able to control the classroom.” The value for the construct was

calculated by taking an average of the following survey item responses. (Four items,

Cronbach’s alpha = 0.78.)

How much of a problem are the following...

͟ Bullying/cyber bullying (students repeatedly teasing other students in

person or online)

͟ Students fighting

͟ Students cutting classes

͟ Teachers not being able to control the classroom

(Scale: 0 = Not a problem, 1 = A small problem, 2 = A medium problem, 3 = A 

big problem) 
74

Table 3.2 also includes the student survey measure: “The school had problems with students

physically or verbally abusing teachers or other adults.” The value for the construct was

calculated by taking an average of the following survey item responses. (Two items, Cronbach’s

alpha = 0.81.)

How much of a problem are the following...

͟ Physical abuse of teachers or other adults in the school by students

͟ Verbal abuse of teachers or other adults in the school by students

(Scale: 0 = Not a problem, 1 = A small problem, 2 = A medium problem, 3 = A 

big problem) 

Table 3.2 also includes the student survey measure: “The school had problems with destruction

of school property, students bringing weapons to school, students abusing drugs or alcohol, and

gang activity.” The value for the construct was calculated by taking an average of the following

survey item responses. (Four items, Cronbach’s alpha = 0.86.)

How much of a problem are the following...

͟ Vandalism/destruction of school property

͟ Students bringing weapons like knives and guns to school

͟ Students using/abusing drugs and alcohol in school

͟ Students involved in gangs or gangs being on school property

(Scale: 0 = Not a problem, 1 = A small problem, 2 = A medium problem, 3 = A 

big problem) 

Items from Table 3.3

Under “greater confidence and self-worth,” Table 3.3 includes the student survey measure: “Students believed if they tried hard, did not give up, and had enough time, they could do

their schoolwork well.” The value for the construct was calculated by taking an average of the

following survey item responses. (Three items, Cronbach’s alpha = 0.75.)

How strongly do you agree or disagree...

͟ If I try hard, I believe I can do my schoolwork well.

͟ If I don’t give up, I believe I can do schoolwork that is hard.

͟ If I have enough time, I believe I can do well in my schoolwork. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 
75

Table 3.3 also includes the student survey measure: “Students worried about projects, tests, poor

grades, and schoolwork in general.” The value for the construct was calculated by taking an

average of the following survey item responses. (Four items, Cronbach’s alpha = 0.65.)

How strongly do you agree or disagree....

͟ When I have a project to do, I worry about it a lot.

͟ I worry about getting bad grades on tests and projects.

͟ When I take tests, I don’t feel very good.

͟ I worry about school and schoolwork. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Under “increased engagement and commitment,” Table 3.3 includes the student survey measure: “Students thought their work was interesting and liked what they learned in class.” The

value for the construct was calculated by taking an average of the following survey item

responses. (Two items, Cronbach’s alpha = 0.79.)

How often do you agree with the following statements about yourself?

͟ I think that what we are learning in my classes is interesting.

͟ I like what I am learning in my classes.

(Scale: 0 = Never, 1 = Sometimes, 2 = Very often, 3 = All the time) 

Table 3.3 also includes the student survey measure: “Students tried less at school over time,

not caring about school, and not participating in class.” The value for the construct was calculated by taking an average of the following survey item responses. (Four items, Cronbach’s

alpha = 0.79.)

How strongly do you agree or disagree...

͟ Each week I try less and less at school.

͟ I don’t really care about school.

͟ I’m not involved in things like class activities and class discussions at

school.

͟ I’ve given up on being interested in school.

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 
76

Table 3.3 also includes the student survey measure: “Students believed learning and working

hard in class was important for their future.” The value for the construct was calculated by

taking an average of the following survey item responses. (Five items, Cronbach’s alpha = 

0.83.)

How strongly do you agree or disagree...

͟ My classes prepare me for what I plan to do in life.

͟ What I learn in my classes is important for my future.

͟ Working hard in my classes will matter for my future success in a career.

͟ Learning at school is important to me.

͟ It’s important to me to understand what I’m taught at school.

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Under “increased effort and persistence,” Table 3.3 includes the student survey measure:

“Students did not give up easily when they did not understand or when homework was difficult.” The value for the construct was calculated by taking an average of the following survey

item responses. (Four items, Cronbach’s alpha = 0.74.)

How strongly do you agree or disagree....

͟ I don’t give up easily.

͟ If I don’t understand my schoolwork, I keep trying until I do.

͟ If my homework is difficult I keep trying to work on it trying to figure it

out.

͟ When I am taught something that doesn’t make sense, I spend time trying to understand it.

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Table 3.3 also includes the student survey measure: “Students paid attention, stayed on task in

class, and completed all their schoolwork.” The value for the construct was calculated by taking

an average of the following survey item responses. (Three items, Cronbach’s alpha = 0.80.) 

How often do you agree with the following statements about yourself?

͟ I pay attention in my classes.

͟ I stay on task in my classes.

͟ I complete all my schoolwork.
77

(Scale: 0 = Never, 1 = Sometimes, 2 = Very often, 3 = All the time) 

Under “improved study habits and strategies,” Table 3.3 includes the student survey measure:

“Students used study strategies such as note taking, graphic organizers, and formulas to help

remember information.” The value for the construct was calculated by taking an average of the

following survey item responses. (Four items, Cronbach’s alpha = 0.71.)

How often do you use the following skills THIS YEAR...

͟ Note taking

͟ Mind mapping/graphic organization of information (for example, KWL

charts and Venn diagrams)

͟ Formulas or rhymes used to help you remember things (mnemonic 

devices)

͟ Test preparation/test taking skills

(Scale: 0 = Never, 1 = Sometimes, 2 = Very often, 3 = All the time) 

Table 3.3 also includes the student survey measure: “Students used social skills and conflict

resolution strategies for controlling anger.” The value for the construct was calculated by taking

an average of the following survey item responses. (Two items, Cronbach’s alpha = 0.63.) 

How often do you use the following skills THIS YEAR...

͟ Conflict resolution/strategies for controlling anger

͟ Social skills (cooperation, active listening, compromise, etc.)

(Scale: 0 = Never, 1 = Sometimes, 2 = Very often, 3 = All the time) 

Under “positive relationships with adults and peers,” Table 3.3 includes the student survey

measure: “Students had positive relationships with teachers.” The value for the construct was

calculated by taking an average of the following survey item responses. (Seven items,

Cronbach’s alpha = 0.86.)

How strongly do you agree or disagree...

͟ Students at my school get along well with teachers.

͟ My teachers meet with me to talk about schoolwork and give me extra

help if I need it.

͟ My teachers really listen to what I have to say.
78

͟ Teachers at this school set a positive example for students with their actions.

͟ My teachers notice when I am doing a good job and let me know about it.

͟ Teachers at my school try to be fair.

͟ If I had a problem outside of the classroom, I felt I could talk to a teacher

at my school. (high school students only)

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Table 3.3 also includes the student survey measure: “Administrators were respectful of students

and teachers and were fair when enforcing rules.” The value for the construct was calculated by

taking an average of the following survey item responses. (Three items, Cronbach’s alpha = 

0.75.) 

How strongly do you agree or disagree...

͟ Principals, Assistant Principals, and other adults in this school are respectful of students.

͟ Principals, Assistant Principals, and other adults in this school are respectful of teachers.

͟ Principals, Assistant Principals, and other adults in my school are fair

when enforcing rules.

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Table 3.3 also includes the student survey measure: “Students had a positive relationship with at

least one adult in the school other than a teacher.” The value for the construct was calculated by

taking an average of the following survey item responses. (Four items, Cronbach’s alpha = 

0.79.) 

How strongly do you agree or disagree...

͟ At least one adult at this school other than my teacher(s) really cares

about me.

͟ At least one adult at this school other than my teacher(s) checks in with

me almost every day.

͟ At least one adult at this school other than my teacher(s) encourages me

to do my best.

͟ I could go to at least one adult at this school other than a teacher to get

help with solving problems at home or school.
79

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Table 3.3 also includes the student survey measure: “Students felt they fit in at school and that

other students at school accepted them for who they were.” The value for the construct was

calculated by taking an average of the following survey item responses. (Two items, Cronbach’s

alpha = 0.68.) 

How strongly do you agree or disagree...

͟ Overall, other students at this school accept me for who I am.

͟ I feel that I “fit in” at this school. 

(Scale: 0 = Strongly disagree, 1 = Disagree, 2 = Agree, 3 = Strongly agree) 

Appendix B 

Supplemental ABC Outcome Findings

83

This appendix offers additional analyses related to the findings discussed in Chapter 4. It 

includes impacts on the attendance, behavior, and course performance (ABC) outcomes for 

subgroups of middle school students who were and were not proficient before participating in 

the study, for the sample of students who attended the study school for the entire implementation year, and for Cohort 1 middle school and high school students separately. It also includes 

comparisons of the impacts on the percentage of students with stability indicators and the 

percentage of students with no early warning indicators by random assignment block for the 

first and second cohorts of students. Finally, it includes the findings from a sensitivity test of the 

analysis at the middle school level controlling for differences in grade configuration across the 

middle schools. The results of these subgroup and sensitivity analyses help to explain whether 

the main findings presented in Chapter 4 are sensitive to changes in the analytic sample or to 

differences in school grade configuration, and whether the growth in impacts from the first to 

the second cohort vary across the schools in the study.

Impacts for Middle School Students by Prior-Year Proficiency 

Appendix Table B.1 presents the ABC outcomes for subgroups of middle school students 

designated by whether they scored at or above proficiency on math and English state standardized tests in the previous year. The middle school students in Panel A received a proficient 

rating on both their English/language arts and math assessments in the previous school year. 

Panel B presents the findings for middle school students who were not proficient on one or both 

assessments in the prior year. Some students were not included in these analyses because their 

test scores for the previous school year were not available. Twenty-eight percent, or 1,261, of 

the students are in the proficient subgroup, while 72 percent, or 3,185 students, are in the 

nonproficient subgroup. 

In general, findings for the proficient and nonproficient subgroups are similar, and no statistically significant differences between these two groups were found, suggesting that the 

Diplomas Now model supported both these groups to a similar degree. There were two statistically significant impacts for the nonproficient subgroup. First, the impact of Diplomas Now on the 

proportion of students with at least 90 percent attendance is positive. While the impact on this 

measure for the students who were proficient on both tests in the previous year is not statistically 

significant, the effect size of this outcome is larger for the proficient students than for the nonproficient students, and the difference between these two impacts is not statistically significant.1

 1

Since there are fewer students in the proficient subgroup, the model for that subgroup has less statistical 

power in comparison with the nonproficient subgroup and cannot detect statistically significant impacts at the 

same level.
84

(continued)

Appendix Table B.1

Attendance, Behavior, and Course Performance,

Prior-Year Math and English Proficiency Subgroups, Middle Schools, Cohort 2

Impacts on Threshold Measures of

P-Value for

DN

Schools

Non-DN

Schools

Estimated

Impact

Effect

Size

Estimated

Outcome Impact

Panel A: Proficient on both tests

Attendance

Percentage of students who attended over 

90 percent of enrolled days 82.3 77.6 4.7 0.14 0.213

Percentage of students who attended over 

85 percent of enrolled days 91.2 87.2 4.0 0.14 0.426

Behavior

Percentage of students who were ever suspended 

or expelled during year 15.9 16.9 -0.9 -0.02 0.873

Percentage of students who were suspended or

expelled for 3 or more days 11.8 10.1 1.7 0.05 0.698

Course performance

Percentage of students who had no core course a failures during year 93.8 93.9 -0.1 -0.01 0.959

Percentage of students who had no math course 

failures during year 95.2 96.3 -1.0 -0.08 0.520

Percentage of students who had no English/ 

b language arts course failures during year 99.2 97.6 1.6 0.12 0.294

c ABC composite measure

Percentage of students above stability threshold 72.8 64.7 8.1 0.17 0.145

Percentage of students with no early warning

indicators 80.9 76.6 4.3 0.10 0.423

Sample size 14 15

Second, there was a statistically significant impact on the percentage of students who had no early 

warning indicators in the nonproficient subgroup. The proficient subgroup also had a positive 

though not statistically significant impact on that outcome, and again the differences between the 

two findings are not statistically significant.
85

(continued)

P-Value for

DN Non-DN Estimated Effect Estimated

Outcome Schools Schools Impact Size Impact

Panel B: Not proficient on one or both tests 

Attendance

Percentage of students who attended over 

90 percent of enrolled days 77.1 72.5 4.6 0.10 * 0.069

Percentage of students who attended over 

85 percent of enrolled days 87.3 84.2 3.1 0.08 0.234

Behavior

Percentage of students who were ever suspended 

or expelled during year 25.8 25.4 0.4 0.01 0.947

Percentage of students who were suspended or

expelled for 3 or more days 18.6 19.6 -1.0 -0.03 0.821

Course performance

Percentage of students who had no core course a failures during year 83.9 82.7 1.2 0.03 0.744

Percentage of students who had no math course 

failures during year 92.7 93.1 -0.4 -0.01 0.869

Percentage of students who had no English/ 

language arts course failures during year 94.7 92.6 2.2 0.09 0.217

c ABC composite measure

Percentage of students above stability threshold 59.5 54.1 5.4 0.11 0.182

Percentage of students with no early warning

indicators 72.2 65.2 7.0 0.15 ** 0.036

Sample size 14 15

Appendix Table B.1 (continued)

Impacts for the Full-Year Sample of Students 

In general, students who attended the study school for the full duration of the school year

represent a “stable sample” of students. More specifically, the criteria for including students 

in the full-year sample subgroup are as follows: (1) the student entered the study school 

within 14 days of the first day of school, (2) the student did not exit the study school more 

than 14 days before the end of the school year, (3) the student did not have a lapse in attendance of greater than 14 days, and (4) the student did not attend both a Diplomas Now (DN)
86

Appendix Table B.1 (continued)

SOURCE: MDRC calculations based on student records, inclusive of state assessment data, obtained from school 

districts. 

NOTES: Across 29 study middle schools, 4,446 nonrepeating sixth-grade students are included in the analyses. 

Among the sample, 2,030 students attended DN schools and 2,416 students attended non-DN schools. The 

proficient subgroup includes 1,261 students and the nonproficient subgroup includes 3,185 students. Some students 

are not included in the analyses of course performance measures because data were not available on their grades for 

specific courses. An entire DN middle school was dropped from these analyses because there are no baseline 

course data for the students attending that school. There are no more than 8 percent missing DN school students 

and 14 percent missing non-DN school students for any of the course performance measures.

 Estimated impacts are based on a two-level model with students nested within schools, controlling for random 

assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 

observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 

column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 

covariate values for students in DN schools as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures. 

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Statistically significant differences in impacts between the proficient and nonproficient groups are indicated by † 

when the p-value is less than or equal to 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Core courses include math, English/language arts, science, or social studies courses. 

 b

One random assignment block, which includes one DN school and one non-DN school, was removed from this 

analysis because lack of variation in that random assignment block was impeding the analysis from running 

properly. 

 c

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 

“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 

suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 

the school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

and a non-Diplomas Now (non-DN) study school during the school year. The full-year

sample subgroup comprises 10,948 students, which is approximately 73 percent of the full 

study sample. Appendix Table B.2 presents the impact results of the full-year sample subgroup analysis for Cohort 2. Similar to the main impact findings presented in Table 4.2, the 

full-year sample subgroup analyses yielded one statistically significant impact of the Diplomas Now model on the percentage of students with no early warning indicators. 
87

(continued)

Appendix Table B.2

Impacts on Continuous and Threshold Measures of

Attendance, Behavior, and Course Performance, Full-Year Sample, Cohort 2

Outcome

DN

Schools

Non-DN

Schools

Estimated

Impact

Effect

Size

P-Value for

Estimated

Impact

Attendance

Percentage of enrolled days attended 

Percentage of students who attended over 

90 percent of enrolled days

Percentage of students who attended over 

85 percent of enrolled days

Behavior

Percentage of enrolled days suspended or expelled

Percentage of students who were ever suspended 

or expelled during year

Percentage of students who were suspended or

expelled for 3 or more days

a Course performance

Percentage of core courses passed

Percentage of students who had no core course 

failures during year

Percentage of students who had no math course 

failures during year

Percentage of students who had no English/

language arts course failures during year

b ABC composite measure

Percentage of students above stability threshold

Percentage of students with no early warning

indicators

91.8

73.8

84.9

1.1

23.8

16.2

88.9

76.7

86.8

90.1

54.4

67.7

91.7

74.0

85.0

0.8

22.0

14.8

88.9

76.4

85.1

88.8

52.3

63.7

0.1

-0.3

-0.1

0.3

1.7

1.4

0.0

0.3

1.8

1.2

2.0

4.0

0.01

-0.01

0.00

0.11

0.04

0.04

0.00

0.01

0.05

0.04

0.04

0.08 **

0.863

0.877

0.963

0.161

0.631

0.633

0.977

0.873

0.430

0.357

0.334

0.033

Sample size 29 29
88

SOURCE: MDRC calculations based on student records obtained from school districts.

NOTES: Students are included in the full-year sample if they meet the following criteria: (1) entered the study 

school within 14 days of the first day of school, (2) did not exit the study school more than 14 days before the end 

of the school year, (3) did not have a lapse of attendance data greater than 14 days, and (4) did not attend both a 

DN and non-DN study school during the school year.

 Across 58 study schools, 10,948 nonrepeating sixth- and ninth-grade students are included in the full-year 

sample and analyses. Among the sample, 5,069 students attended DN schools and 5,879 students attended non-DN 

schools. Some students are not included in the analyses of course performance measures because data were not 

available on their grades for specific courses. An entire DN middle school was dropped from these analyses 

because there are no baseline course data for the students attending that school. There are no more than 7 percent 

missing DN school students and 2 percent missing non-DN school students for any of the course performance 

measures. The large percentage of missing DN school students is due to one school with a large amount of missing 

math course data. Without this school included, there are no more than 2 percent missing students on any of the 

measures. A sensitivity test was done removing the random assignment block that includes this school from the 

analyses and findings were similar.

 Estimated impacts are based on a two-level model with students nested within schools, controlling for random 

assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 

observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 

column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 

covariate values for students in DN schools as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures. 

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent.

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Core courses include all math, English/language arts, science, or social studies courses.

 bThe “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 

“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 

suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 

the school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

Appendix Table B.2 (continued)

Impacts for Cohort 1 Middle School and High School Subgroups

Appendix Table B.3 presents the impacts of the Diplomas Now model for the sixth- and ninthgrade students who attended the study schools during the first year of implementation (Cohort 

1), separated into middle school and high school subgroups. These subgroup analyses yielded

only one statistically significant impact of the Diplomas Now model, on the percentage of sixthgrade students who had no math course failures. This is consistent with the findings presented in 

Table 4.6 for the full Cohort 1 study sample (middle and high schools combined), where there 

was a statistically significant impact on this same measure and no others.
89

(continued)

Appendix Table B.3

Impacts on Threshold Measures of Attendance, Behavior, 

and Course Performance, Middle Schools and High Schools, Cohort 1

P-Value for

DN

Schools

Non-DN

Schools

Estimated

Impact

Effect Estimated

Outcome Size Impact

Panel A: Middle schools

Attendance

Percentage of students who attended over 

90 percent of enrolled days 76.1 76.5 -0.4 -0.01 0.871

Percentage of students who attended over 

85 percent of enrolled days 86.7 86.9 -0.2 -0.01 0.837

Behavior

Percentage of students who were ever suspended 

or expelled during year 23.0 19.1 3.9 0.10 0.490

Percentage of students who were suspended or

expelled for 3 or more days 15.4 12.7 2.8 0.08 0.518

Course performance

Percentage of students who had no core course a failures during year 88.5 84.8 3.7 0.10 0.320

Percentage of students who had no math course 

failures during year 96.5 91.2 5.3 0.18 ** 0.029

Percentage of students who had no English/ 

language arts course failures during year 95.2 94.2 1.0 0.04 0.629

b ABC composite measure

Percentage of students above stability threshold 59.4 62.9 -3.6 -0.07 0.306 †

Percentage of students with no early warning

indicators 73.2 73.7 -0.6 -0.01 0.859

Sample size 16 15

There was a statistically significant difference between the impact estimates for Cohort 

1 and Cohort 2 middle school students on the percentage of students with stability indicators, as 

represented by the dagger on the far right side of the table. Although not statistically significant 

for either cohort, the impact estimate was negative for Cohort 1 and positive for Cohort 2. The 

dagger suggests the Diplomas Now model grew more successful in supporting middle school
90

(continued)

P-Value for

DN Non-DN Estimated Effect Estimated

Outcome Schools Schools Impact Size Impact

Panel B: High schools

Attendance

Percentage of students who attended over 

90 percent of enrolled days 56.9 58.9 -2.0 -0.04 0.471

Percentage of students who attended over 

85 percent of enrolled days 69.0 70.2 -1.1 -0.03 0.768

Behavior

Percentage of students who were ever suspended 

or expelled during year 28.5 25.7 2.8 0.07 0.134

Percentage of students who were suspended or

expelled for 3 or more days 20.9 20.2 0.7 0.02 0.798

Course performance

Percentage of students who had no core course a failures during year 57.9 55.1 2.8 0.06 0.351

Percentage of students who had no math course 

failures during year 75.6 73.2 2.3 0.05 0.448

Percentage of students who had no English/ 

language arts course failures during year 77.6 76.4 1.2 0.03 0.727

b ABC composite measure

Percentage of students above stability threshold 34.4 35.5 -1.1 -0.02 0.506

Percentage of students with no early warning

indicators 47.9 46.6 1.3 0.03 0.560

Sample size 15 14

Appendix Table B.3 (continued)

students to get or stay on a stable path to graduation between the first and second years of 

implementation. 

Comparison of Random Assignment Block-Level Impacts

by Cohort

Appendix Tables B.4 and B.5 present random assignment block-level impact estimates for the 

percentage of students who had stability indicators and the percentage of students who had no
91

SOURCE: MDRC calculations based on student records obtained from school districts.

NOTES: Across 60 study schools, 5,729 nonrepeating sixth-grade students and 9,551 nonrepeating ninth-grade 

students are included in the analyses. Among the middle school sample, 2,671 students attended DN schools and 

3,058 students attended non-DN schools. Among the high school sample, 4,651 students attended DN schools and 

4,900 students attended non-DN schools. Some students are not included in the analyses of course performance 

measures because data were not available on their grades for specific courses. An entire DN middle school was 

dropped from these analyses because there are no baseline course data for the students attending that school. There 

are no more than 10 percent missing DN school students and 14 percent missing non-DN school students for any of 

the course performance measures.

 Estimated impacts are based on a two-level model with students nested within schools, controlling for random 

assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 

observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 

column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 

covariate values for students in DN schools as the basis for the adjustment.

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures. 

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Statistically significant differences in impacts between the impact estimates for Cohort 1 and Cohort 2 are 

indicated by † when the p-value is less than or equal to 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Core courses include math, English/language arts, science, or social studies courses.

 b

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 

“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 

suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 

the school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

Appendix Table B.3 (continued)

early warning indicators, respectively. The first set of columns in each of these tables lists the 

impacts for each random assignment block for the first cohort of sixth- and ninth-grade students, and the second set of columns presents the same information for the second cohort. The 

tables also present the differences between the Cohort 2 and Cohort 1 impacts. Appendix Table 

B.4 shows that 11 of the random assignment blocks had an increase in the impact on the 

percentage of students with stability indicators between Cohort 1 and Cohort 2, while 10 blocks 

experienced a decrease in the impact by the second year. The table also shows that in two 

random assignment blocks, there were statistically significant positive differences in impacts for 

Cohort 2 compared with Cohort 1. Appendix Table B.5 shows that 12 of the random assignment blocks experienced an increase in the impact on the percentage of students with no early 

warning indicators between Cohort 1 and Cohort 2, while 9 blocks experienced a decrease in 

the impact. One random assignment block had a statistically significant positive difference in 
92

Appendix Table B.4

Comparison of Cohort 1 and Cohort 2 Impacts

by Random Assignment Block,

Percentage of Students Above Stability Threshold

Random

Assignment Cohort 1 Cohort 2 Difference

Block Impact P-Value Impact P-Value (C2-C1) P-Value

109

112

118

101

107

108

104

122

120

113

119

115

111

102

117

116

103

106

110

105

114

-20.2 *

-17.8 ***

-1.6

0.4

-13.8

-15.0

-14.9

-5.6

-1.5

8.6

1.1

-4.5

10.1

-9.0

-11.0

12.0

8.8

13.2

5.2

29.2 **

13.4 *

0.052

0.003

0.876

0.964

0.136

0.173

0.248

0.370

0.867

0.343

0.889

0.476

0.290

0.309

0.251

0.197

0.258

0.180

0.565

0.012

0.052

6.6

8.4

21.3 *

15.4 *

-1.9

-3.6

-6.7

0.7

4.8

13.7

2.3

-5.5

9.1

-11.1

-14.3

7.1

3.0

6.9

-5.3

17.9

-1.3

0.527

0.171

0.059

0.097

0.847

0.752

0.624

0.915

0.615

0.173

0.786

0.422

0.376

0.275

0.174

0.476

0.716

0.504

0.586

0.116

0.856

26.9 *

26.2 ***

22.9

15.0

11.9

11.4

8.2

6.3

6.2

5.2

1.2

-1.0

-1.0

-2.1

-3.3

-5.0

-5.8

-6.2

-10.4

-11.3

-14.7

0.066

0.002

0.129

0.214

0.363

0.464

0.658

0.486

0.626

0.695

0.916

0.914

0.941

0.870

0.810

0.708

0.602

0.656

0.425

0.455

0.130

Sample size 60 58

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 60 study schools in Cohort 1 and 58 study schools in Cohort 2, 15,280 Cohort 1 

students and 14,950 Cohort 2 students are included in the analyses. 

 The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled 

in the district, was never suspended or expelled, and did not fail any core courses attempted during the 

school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance 

levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences.
93

Appendix Table B.5

Comparison of Cohort 1 and Cohort 2 Impacts

by Random Assignment Block,

Percentage of Students with No Early Warning Indicators

Random

Assignment Cohort 1 Cohort 2 Difference

Block Impact P-Value Impact P-Value (C2-C1) P-Value

118

109

112

104

108

107

101

113

102

119

122

120

117

111

110

103

115

106

114

116

105

-4.4

-6.1

-7.8

-25.0 **

-5.7

-11.5

2.6

9.9

-7.0

1.9

0.4

6.3

-5.7

10.7

-2.9

7.0

-0.8

14.0

7.8

14.1

31.3 ***

0.664

0.524

0.128

0.033

0.573

0.192

0.739

0.261

0.418

0.802

0.952

0.457

0.530

0.251

0.734

0.353

0.894

0.153

0.230

0.117

0.006

16.4 *

13.6

9.6 *

-8.8

7.4

0.2

13.5 *

12.3

-5.5

3.3

1.5

6.8

-6.4

10.0

-4.7

4.3

-4.3

7.6

0.9

2.7

5.9

0.085

0.137

0.063

0.408

0.426

0.980

0.075

0.140

0.535

0.639

0.783

0.378

0.446

0.252

0.548

0.528

0.449

0.396

0.877

0.736

0.528

20.7

19.7

17.3 **

16.2

13.1

11.7

10.9

2.3

1.6

1.4

1.1

0.5

-0.7

-0.7

-1.8

-2.7

-3.5

-6.4

-6.9

-11.5

-25.4 *

0.129

0.133

0.016

0.287

0.336

0.312

0.304

0.842

0.898

0.894

0.887

0.963

0.953

0.952

0.877

0.783

0.671

0.619

0.423

0.329

0.070

Sample size 60 58

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 60 study schools in Cohort 1 and 58 study schools in Cohort 2, 15,280 Cohort 1 

students and 14,950 Cohort 2 students are included in the analyses. 

 The “no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled 

in the district, was suspended or expelled for fewer than three days, and did not fail any math or 

English/language arts courses during the school year. If a student has missing course data but meets the 

other two criteria for an ABC composite measure, the student is assigned a value of 1.

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance 

levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences.
94

the impact for Cohort 2 compared with Cohort 1, and one random assignment block had a 

statistically significant decrease in the impact between the first and second cohorts. 

Middle School Impacts Controlling for Grade Configuration 

Appendix Table B.6 provides the results of a sensitivity analysis that was conducted to verify 

that the estimates of program impacts presented in Chapter 4 of this report are unbiased and that 

they can be interpreted as the effect of the Diplomas Now intervention and are not related to 

other measurable school characteristics. For this sensitivity analysis, an additional school-level 

covariate indicating middle school configuration was added to the analytic model described in 

Appendix A. This decision was made based on prior research suggesting that middle school 

configuration can affect student outcomes and, in particular, that transitioning to a new school 

can have an effect on a student’s outcomes during that year.2

The grade configuration covariate has a value of one if the school the student attended 

has a grade configuration of fifth through eighth grade or kindergarten through eighth grade. 

Because sixth grade is not the first grade level offered at these schools, the sixth-grade students 

in the analysis were not generally transitioning to a new school during this year. If the school a 

student attended serves only sixth- through eighth-graders or seventh- and eighth-graders, the 

covariate has a value of zero.

3 Six of the 29 study middle schools (21 percent) received a value 

of one, while the remaining 23 received a value of zero. 

The results of this sensitivity analysis show several statistically significant impacts. 

Similar to Table 4.3, which displays the impacts for the middle school subgroup without the 

added grade configuration covariate, there were positive and statistically significant impacts on 

the percentage of students who had a 90 percent attendance rate or better and on the percentage 

of students who have no early warning indicators at the end of the school year. In both cases the 

analysis including the school configuration covariate results in a lower p-value. The other two 

outcomes that had statistically significant impacts when the model controlled for middle school 

grade configuration are the percentage of core courses passed and the percentage of students 

who had no English/language arts course failures during the year. In both cases these findings 

are positive but not statistically significant in Table 4.1 (Panel B) and Table 4.3, although in the 

case of the percentage of students who had no English/language arts course failures the p-value 

is very near significant at 0.101.

 2

Cook, MacCoun, Muschkin, and Vigdor (2008); Rockoff and Lockwood (2010); Schwartz, Stiefel, 

Rubenstein, and Zabel (2011); Schwerdt and West (2013). 3

In schools with only seventh- and eighth-graders, seventh-graders were included in the analyses because

seventh grade was the transition year. 
95

(continued)

Appendix Table B.6

Impacts on Continuous and Threshold Measures of

Accounting for Variation in School Grade Configuration,

Middle Schools, Cohort 2

Attendance, Behavior, and Course Performance,

Outcome

DN 

Schools

Non-DN

Schools

Estimated

Impact

Effect

Size

P-Value for

Estimated

Impact

Attendance

Percentage of enrolled days attended

Percentage of students who attended over 

90 percent of enrolled days

Percentage of students who attended over 

85 percent of enrolled days

Behavior

Percentage of enrolled days suspended or expelled

Percentage of students who were ever suspended 

or expelled during year

Percentage of students who were suspended or

expelled for 3 or more days

a Course performance

Percentage of core courses passed

Percentage of students who had no core course 

failures during year

Percentage of students who had no math course 

failures during year

Percentage of students who had no English/

language arts course failures during year

b ABC composite measure

Percentage of students above stability threshold

Percentage of students with no early warning

indicators

92.7

77.2

86.8

1.2

22.1

15.6

94.5

87.0

94.3

95.5

62.5

74.5

91.1

72.2

83.8

1.2

21.9

16.4

92.4

84.8

93.6

93.0

56.5

68.1

1.6

5.0

3.0

-0.1

0.2

-0.8

2.1

2.2

0.7

2.5

5.9

6.4

0.14

0.11 **

0.08

-0.01

0.00

-0.02

0.12 *

0.06

0.03

0.12 **

0.12

0.14 *

0.161

0.046

0.285

0.785

0.976

0.864

0.068

0.425

0.757

0.021

0.123

0.059

Sample size 14 15
96

SOURCE: MDRC calculations based on student records obtained from school districts. 

NOTES: Across 29 study schools, 5,606 nonrepeating sixth-graders are included in the analyses. Among the 

sample, 2,513 students attended DN schools and 3,093 students attended non-DN schools. Some students are not 

included in the analyses of course performance measures because data were not available on their grades for 

specific courses. An entire DN middle school was dropped from these analyses because there were no baseline 

course data for the students attending that school. There are no more than 9 percent missing DN school students 

and 12 percent missing non-DN school students for any of the course performance measures.

 Estimated impacts are based on a two-level model with students nested within schools, controlling for random 

assignment block and school- and student-level covariates. The values in the column labeled “DN Schools” are the 

observed mean outcomes weighted by the number of schools in a random assignment block, and the values in the 

column labeled “Non-DN Schools” are regression-adjusted mean outcomes for the non-DN schools using the mean 

covariate values for students in DN schools as the basis for the adjustment. 

 Variation in school grade configuration was accounted for by including a covariate in the analysis that was 

equal to 1 if the school was kindergarten through eighth grade or fifth through eighth grade. 

 Effect sizes were computed using the standard deviations of all non-DN school students for the respective 

measures.

 A two-tailed t-test was used for all statistical tests presented in this table. Statistical significance levels are 

indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

 Rounding may cause slight discrepancies in calculating sums and differences. 

 a

Core courses include math, English/language arts, science, or social studies courses.

 b

The “above stability threshold” measure is 1 if a student attended over 90 percent of days enrolled in the 

district, was never suspended or expelled, and did not fail any core courses attempted during the school year. The 

“no early warning indicators” measure is 1 if a student attended over 85 percent of days enrolled in the district, was 

suspended or expelled for fewer than three days, and did not fail any math or English/language arts courses during 

the school year. If a student has missing course data but meets the other two criteria for an ABC composite 

measure, the student is assigned a value of 1.

Appendix Table B.6 (continued)
97

References 

Allensworth, Elaine Marie, and John Q. Easton. 2005. The On-Track Indicator as a Predictor

of High School Graduation. Chicago: University of Chicago Consortium on Chicago School 

Research. 

Balfanz, Robert, John M. Bridgeland, Joanna Hornig Fox, Jennifer L. DePaoli, Erin S. Ingram, and 

Mary Maushard. 2014. Building a Grad Nation: Progress and Challenge in Ending the High 

School Dropout Epidemic. Washington, DC: Civic Enterprises. 

Balfanz, Robert, Liza Herzog, and Douglas J. Mac Iver. 2007. “Preventing Student Disengagement 

and Keeping Students on the Graduation Path in Urban Middle-Grades Schools: Early Identification and Effective Interventions.” Educational Psychologist 42, 4: 223-235.

Borman, Geoffrey D., Gina M. Hewes, Laura T. Overman, and Shelly Brown. 2003. “Comprehensive School Reform and Achievement: A Meta-Analysis.” Review of Educational Research 73, 

2: 125-230. 

Cook, Phillip J., Robert MacCoun, Clara Muschkin, and Jacob Vigdor. 2008. “The Negative Impacts of Starting Middle School in Sixth Grade.” Journal of Policy Analysis and Management

27, 1: 104-121. 

Corrin, William, Susan Sepanik, Aracelis Gray, Felix Fernandez, Ashley Briggs, and Kathleen K. 

Wang. 2014. Laying Tracks to Graduation: The First Year of Implementing Diplomas Now. 

New York: MDRC.

Corrin, William, Susan Sepanik, Rachel Rosen, and Andrea Shane. 2016. Survey Instrument Supplement to Addressing Early Warning Indicators: Interim Impact Findings from the Investing 

in Innovation (i3) Evaluation of Diplomas Now. New York: MDRC.

Frazelle, Sarah, and Aisling Nagel. 2015. A Practitioner’s Guide to Implementing Early Warning 

Systems (REL 2015–056). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional 

Educational Laboratory Northwest.

Kemple, James J., Micha D. Segeritz, and Nickisha Stephenson. 2013. “Building On-Track Indicators for High School Graduation and College Readiness: Evidence from New York City.” 

Journal of Education for Students Placed at Risk (JESPAR) 18, 1: 7-28.

Markow, Dana, and Andrea Pieters. 2012. The MetLife Survey of the American Teacher: Teachers, 

Parents, and the Economy. New York: MetLife, Inc.

Moore, Kristin Anderson. 2014. Making the Grades: Assessing the Evidence for Integrated Student 

Supports. Bethesda, MD: Child Trends.

Murnane, Richard J. 2013. “U.S. High School Graduation Rates: Patterns and Explanations.” Journal of Economic Literature 51, 2: 370-422. 
98

National Center for Education Statistics. 2014. Web Tables — Profile of Undergraduate Students: 

2011-12 (NCES 2015-167). Washington, DC: National Center for Education Statistics, 

Institute of Education Sciences, U.S. Department of Education. 

Plotts, Chris, and Jennifer Sable. 2010. Characteristics of the 100 Largest Public Elementary and 

Secondary School Districts in the United States: 2007–08 (NCES 2010-349). Washington, DC: 

U.S. Department of Education, National Center for Education Statistics.

Rockoff, Jonah E., and Benjamin B. Lockwood. 2010. “Stuck in the Middle: Impacts of Grade Configuration in Public Schools.” Journal of Public Economics 94, 11: 1051-1061. 

Roderick, Melissa, Thomas Kelley-Kemple, David W. Johnson, and Nicole O. Beechum. 2014. 

Preventable Failure: Improvements in Long-Term Outcomes When High Schools Focused

on the Ninth Grade Year. Chicago: University of Chicago Consortium on Chicago School

Research. 

Schwartz, Amy Ellen, Leanna Stiefel, Ross Rubenstein, and Jeffrey Zabel. 2011. “The Path Not 

Taken: How Does School Organization Affect Eighth-Grade Achievement?” Educational 

Evaluation and Policy Analysis 33, 3: 293-317. 

Schwerdt, Guido, and Martin R. West. 2013. “The Impact of Alternative Grade Configurations on 

Student Outcomes Through Middle and High School.” Journal of Public Economics 97: 308-

326. 

Sepanik, Susan, William Corrin, David Roy, Aracelis Gray, Felix Fernandez, Ashley Briggs, and 

Kathleen K. Wang. 2015. Moving Down the Track: Changing School Practices During the 

Second Year of Diplomas Now. New York: MDRC.
99

Earlier MDRC Publications on Diplomas Now

Moving Down the Track

Changing School Practices During the Second Year of Diplomas Now

2015. Susan Sepanik, William Corrin, David Roy, Aracelis Gray, Felix Fernandez,

Ashley Briggs, Kathleen K. Wang

Laying Tracks to Graduation

The First Year of Implementing Diplomas Now

2014. William Corrin, Susan Sepanik, Aracelis Gray, Felix Fernandez, Ashley Briggs,

Kathleen K. Wang

Data Collection Instrument Supplement to Laying Tracks to Graduation

The First Year of Implementing Diplomas Now

2014. William Corrin, Susan Sepanik, Aracelis Gray, Felix Fernandez, Ashley Briggs,

Kathleen K. Wang

____________________________ 

NOTE: All the publications listed above are available for free download at www.mdrc.org.
About MDRC

MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 

to learning what works to improve the well-being of low-income people. Through its research 

and the active communication of its findings, MDRC seeks to enhance the effectiveness of 

social and education policies and programs.

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 

for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 

Its projects are a mix of demonstrations (field tests of promising new program approaches) 

and evaluations of ongoing government and community initiatives. MDRC’s staff bring an 

unusual combination of research and organizational experience to their work, providing 

expertise on the latest in qualitative and quantitative methods and on program design, 

development, implementation, and management. MDRC seeks to learn not just whether a 

program is effective but also how and why the program’s effects occur. In addition, it tries to 

place each project’s findings in the broader context of related research — in order to build 

knowledge about what works across the social and education policy fields. MDRC’s findings, 

lessons, and best practices are proactively shared with a broad audience in the policy and 

practitioner community as well as with the general public and the media.

Over the years, MDRC has brought its unique approach to an ever-growing range of policy 

areas and target populations. Once known primarily for evaluations of state welfare-to-work 

programs, today MDRC is also studying public school reforms, employment programs for exoffenders and people with disabilities, and programs to help low-income students succeed in 

college. MDRC’s projects are organized into five areas:

 • Promoting Family Well-Being and Children’s Development

Improving Public Education

Raising Academic Achievement and Persistence in College

Supporting Low-Wage Workers and Communities

Overcoming Barriers to Employment

Working in almost every state, all of the nation’s largest cities, and Canada and the United 

Kingdom, MDRC conducts its projects in partnership with national, state, and local 

governments, public school systems, community organizations, and numerous private 

philanthropies.