Evaluation of the 

Comprehensive School Reform Program

Implementation and Outcomes 

Fifth-Year Report

Martin Orland, WestEd

Amanda Hoffman, WestEd

E. Sidney Vaughn, III, Virginia Beach, Virginia

Prepared for

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

2010
This report was prepared for the U.S. Department of Education under Contract Number ED01CO0129. Menahem 

Herman served as the contracting officer’s representative. The content of this report does not necessarily reflect the 

views or policies of the U.S. Department of Education nor does the mention of commercial products or 

organizations imply endorsements by the U.S. government. The inclusion of such information is for the reader’s 

convenience and is not intended to endorse either any views expressed or any products, programs, models or 

services offered.

U.S. Department of Education

Arne Duncan

Secretary

Office of Planning, Evaluation and Policy Development

Carmel Martin

Assistant Secretary

Policy and Program Studies Service

Alan Ginsburg

Director

January 2010

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to 

reprint this publication is not necessary, the suggested citation is: U.S. Department of Education, Office of Planning, 

Evaluation and Policy Development, Policy and Program Studies Service, Evaluation of the Comprehensive School 

Reform Program Implementation and Outcomes: Fifth-Year Report, Washington, D.C., 2010.

This report is available on the Department’s Web site at 

www.ed.gov/about/offices/list/opepd/ppss/reports.html#title. 

On request, this publication is available in alternate formats such as Braille, large print, or computer diskette. For 

more information, please contact the Department’s Alternate Format Center at 202-260-0852 or 202-260-0818.
iii

CONTENTS

List of Exhibits.........................................................................................................v

Preface.......................................................................................................................................... vii

Acknowledgments........................................................................................................................ ix

Executive Summary..................................................................................................................... xi

Background................................................................................................................................ xi

Study Limitations..................................................................................................................... xiv

Main Findings........................................................................................................................... xv

Overall Study Conclusions ...................................................................................................... xix

Recommendations for Federal Policy and Research ............................................................... xxi

I. Introduction ............................................................................................................................... 1

II. Data Sources............................................................................................................................. 5

School-level Achievement Measures.......................................................................................... 6

Survey of School Reform Activities........................................................................................... 9

Case Studies of School Reform Activities................................................................................ 13

Study Limitations...................................................................................................................... 14

III. Overall Relationship Between CSR Award and Achievement......................................... 17

Methodology Used to Assess the Relationship Between Receipt of a CSR Award and 

Achievement ............................................................................................................................. 17

Findings .................................................................................................................................... 21

Discussion................................................................................................................................. 22

IV. The Comprehensiveness of CSR Implementation............................................................. 25

Measuring Implementation of the CSR Components............................................................... 25

Findings .................................................................................................................................... 45

V. Overall Study Conclusions.................................................................................................... 49

Summary of Findings................................................................................................................ 49

Future Research Needed for Improving Chronically Low-Performing Schools...................... 57
iv

References.................................................................................................................................... 61

Appendix A. Standard Error Tables for Analyses .................................................................. 65

Standard Errors for Report Exhibits ......................................................................................... 65

Appendix B. Data Collection Instruments................................................................................ 69
v

EXHIBITS

Exhibit E.1 Eleven Components of Comprehensive School Reform Described in the No 

Child Left Behind Act...................................................................................... xii

Exhibit E.2 Average Gains in Percent Proficient or Higher From 2003 to 2007 for CSR 

and Matched Non-CSR Schools .................................................................... xvi

Exhibit E. 3 Average Number of CSR Components Implemented by CSR and Non-CSR 

Schools in 2003 and 2007............................................................................. xvii

Exhibit E.4 Average Gains in Percent Proficient or Higher From 2002 to 2005, by Subject 

Area and Strength of Scientific Research Base ............................................. xix

Exhibit 1 Eleven Components of Comprehensive School Reform Described in the No 

Child Left Behind Act........................................................................................ 2

Exhibit 2 Standard Deviations of Percent Proficient or Higher for 2007 Elementary and 

Middle School Mathematics and Reading Achievement, by State................... 8

Exhibit 3 Average Baseline Standardized Achievement Measures of All 2002 CSR 

Schools and of the CSR and Non-CSR Schools in the ECSRIO Sample....... 11

Exhibit 4 Average Demographic Measures in 2007 of All 2002 CSR Schools and of the 

CSR and Non-CSR Schools in the ECSRIO Sample ..................................... 12

Exhibit 5 Differences Between the 2002 Cohort of Title I CSR Schools and Title I NonCSR Schools in Achievement, School Size, and Demographics in 2001–02. 18

Exhibit 6 Differences Between the 2002 Cohort of Title I CSR Schools and Non-CSR 

Title I Schools Chosen by Propensity Scoring: Schools With Elementary 

Mathematics and Reading Achievement ........................................................ 20

Exhibit 7 Differences Between the 2002 Cohort of Title I CSR Schools and Non-CSR 

Title I Schools Chosen by Propensity Scoring: Schools with Middle School 

Mathematics and Reading Achievement ........................................................ 21

Exhibit 8 Changes in Standardized Assessment Scores in CSR and Matched Non-CSR 

Schools from 2002–03 to 2006–07................................................................. 22

Exhibit 9 Means and Standard Deviations of Survey Items Used to Construct ResearchBased Design and Evidence-Based Practice Measure .................................... 28

Exhibit 10 Means and Standard Deviations of Survey Items Used to Construct 

Comprehensive Planning-Classroom Measure, by Year ................................ 29

Exhibit 11 Means and Standard Deviations of Survey Items Used to Construct 

Comprehensive Planning-School Measure, by Year ...................................... 30

Exhibit 12 Means and Standard Deviations of the Survey Item Used to Construct the 

Professional Development Measure, by Year................................................. 31

Exhibit 13 Means and Standard Deviations of Survey Items Used to Construct the Goals 

and Benchmarks Measure, by Year ................................................................ 33
vi

Exhibit 14 Means and Standard Deviations of Survey Items Used to Construct Staff 

Participation Measure, by Year....................................................................... 35

Exhibit 15 Means and Standard Deviations of Survey Items Used to Construct the 

District Support Measure, by Year ................................................................. 37

Exhibit 16 Means and Standard Deviations of Survey Items Used to Construct the 

Parental Involvement Measure, by Year......................................................... 39

Exhibit 17 Means and Standard Deviations of Survey Items Used to Construct the 

External Assistance Measure, by Year ........................................................... 41

Exhibit 18 Means and Standard Deviations of Survey Items Used to Construct the 

Evaluation Measures, by Year........................................................................ 43

Exhibit 19 Means and Standard Deviations of Survey Items Used to Construct 

Coordination of Resources Measure, by Year................................................ 44

Exhibit 20 Average Number of CSR Components Implemented by CSR and Non-CSR 

Schools in 2003 and 2007............................................................................... 46

Exhibit 21 Number of Components Implemented in CSR and Non-CSR Elementary 

Schools in 2007............................................................................................... 47

Exhibit 22 Number of Components Implemented in CSR and Non-CSR Middle Schools 

in 2007 ............................................................................................................ 48

Exhibit A.1 Standard Errors for Differences Between the 2002 Cohort of Title I CSR 

Schools and Title I Non-CSR Schools............................................................ 65

Exhibit A.2 Standard Errors for Differences Between the 2002 Cohort of Title I CSR 

Schools and Non-CSR Title I Schools Chosen by Propensity Scoring: Schools 

With Elementary Mathematics and Reading Achievement............................ 66

Exhibit A.3 Standard Errors for Differences Between the 2002 Cohort of Title I CSR 

Schools and Non-CSR Title I Schools Chosen by Propensity Scoring: Schools 

With Middle School Mathematics and Reading Achievement....................... 66

Exhibit A.4 Standard Errors for Changes in Standardized Assessment Scores in CSR and 

Non-CSR Schools From 2002–03 to 2006–07 ............................................... 67

Exhibit A.5 Standard Errors for Average Number of CSR Components Implemented by 

CSR and Non-CSR Elementary Schools in 2003 and 2007 ........................... 68

Exhibit A.6 Standard Errors for Average Number of CSR Components Implemented by 

CSR and Non-CSR Middle Schools in 2003 and 2007 .................................. 68
Preface vii

PREFACE

This Fifth-Year Report from the Evaluation of the Comprehensive School Reform Program 

Implementation and Outcomes (ECSRIO) is the final report on the outcomes of the federally 

funded Comprehensive School Reform (CSR) program. It presents findings about the 

relationship between participation in the U.S. Department of Education’s Comprehensive School 

Reform program in 2002 and subsequent student achievement five years later. This report 

follows the study’s Third-Year Report which presented findings as of 2005 and was published in 

2008.1 The Fifth-Year Report examines the CSR program throughout the country and its 

relationship with gains in student achievement as of 2007. 

The CSR program was established as a demonstration program in 1998 and authorized as a full 

program in 2002 as part of the No Child Left Behind Act (NCLB). It is one approach to help lowperforming K–12 public schools meet state performance standards. The CSR program 

emphasizes two major concepts. First, the approach mandates that school reform should be 

comprehensive in nature, strengthening all aspects of school operations—curriculum, instruction, 

professional development, parental involvement and school organization. Second, the CSR 

approach should involve the use of scientifically based research models—that is, models with 

evidence of effectiveness in multiple settings. 

In 2001, the U.S. Department of Education contracted with WestEd and COSMOS Corporation 

to conduct a five-year study of the CSR program; ECSRIO, which involved a survey of 500 CSR 

schools and 500 comparison schools; case studies of 30 sites; and analyses of student 

achievement in all schools receiving CSR funding in 2002. This report presents the fifth-year 

findings of this study. 

 

1. The Second-Year and Fourth-Year reports are internal documents to the U.S. Department of Education. The 

Second-Year report is titled “Longitudinal Assessment of Comprehensive School Reform Program 

Implementation and Outcomes” and was compiled October 2004 by the U.S. Department of Education Office 

of the Under Secretary, Policy and Program Studies Service. The Fourth-Year report is titled “Evaluation of 

the Comprehensive School Reform Program Implementation and Outcomes (ECSRIO): Interim Internal 

Report Draft” and was compiled November 2007 by the U.S. Department of Education, Office of Planning, 

Evaluation, and Policy Development/Policy and Program Studies Service.

Acknowledgments ix

ACKNOWLEDGMENTS

We wish to thank many individuals who contributed to this report. We are especially grateful to 

the state, district and school survey respondents as well as to all of the school or district staff 

members who planned or participated in site visits. We express our deep gratitude for your role 

in the study and the data you provided, without which this study would not have been possible. 

The information in this report was provided through the congressionally mandated Evaluation of 

the Comprehensive School Reform Program Implementation and Outcomes (ECSRIO), which 

was conducted by WestEd and COSMOS Corporation under contract to the U.S. Department of 

Education. Naida Tushnet of WestEd led this study from its inception in 2002 through the middle 

of 2007 while Robert Yin and, later, Emily Warner led the work of COSMOS Corporation on 

ECSRIO and Susan Cragle from DUERR Associates coordinated the survey data collection.

Several individuals at the U.S. Department of Education provided report guidance and direction. 

We would like to acknowledge the assistance of Alan Ginsburg, director, Policy and Program 

Studies Service (PPSS), and David Goodwin, past director, Program and Analytic Studies 

Division, PPSS. In addition, Diane Crow, Menahem Herman, Sabrina Jones, Calvin Marshall, 

Hina Mohammed and Aaron Neumann in PPSS contributed to the direction of the report.

We would like to acknowledge the contributions and guidance of the members of our Technical 

Working Group, including Carolyn Temple Adger, Geoffrey Borman, H. J. Green, Bryan Hassel, 

Elsie Leak, Valerie Lee, Paul Ruiz, Jean Rutherford, Malik Stewart, Sam Stringfield and Ken 

Wong.

Although we appreciate the assistance and support of all of these individuals, any errors in 

judgment or fact are the responsibility of the authors.

Executive Summary xi

EXECUTIVE SUMMARY

The Comprehensive School Reform (CSR) program was first established as a demonstration 

program in 1998. The program was subsequently authorized under the Elementary and 

Secondary Education Act of 1965, as amended by the No Child Left Behind Act of 2001 (NCLB).

The program emphasizes two major concepts. First, the approach mandates that school reform 

should be comprehensive in nature, strengthening all aspects of school operations—curriculum, 

instruction, professional development, parental involvement and school organization. Second, 

the CSR approach involves the use of scientifically based research models—that is, models with 

evidence of effectiveness in multiple settings. 

In 2001, the U.S. Department of Education contracted with WestEd and COSMOS Corporation 

to conduct the Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO). This fifth-year final report of the study first updates the Third-Year Report

(U.S. Department of Education, 2008) by addressing whether schools that received CSR awards 

had higher achievement gains and/or made greater progress implementing the 11 components 

than did non-CSR schools. Specifically, we address the following questions:

 Was receipt of a CSR award associated with improvements in school-level 

mathematics and reading achievement five years later?

 Were schools that received CSR awards more likely five years later to implement 

the legislatively specified components of CSR than other schools?

Next, this report’s final chapter describes and discusses what we learned over the course of the 

entire study about school reform by summarizing key findings across the major ECSRIO reports, 

including the recently completed examination of dramatic school improvement. This chapter also 

includes recommendations for future federal policy and research.

BACKGROUND

Originally funded as the Comprehensive School Reform Demonstration Program in 1998 with 

$145 million, the program became part of NCLB in 2002 with funding of $310 million. In FY 

2003 and FY 2004, Congress allocated $308 million for the CSR program, and in FY 2005, $205 

million. Between 1998 and 2006, nearly 7,000 schools nationwide received three-year awards to 

implement CSR models. In FY 2006, funding was appropriated only for a clearinghouse to 

support comprehensive school reform and not for school-based activities. 

NCLB defines the CSR program as containing 11 components, which are assumed to work 

together as schools undergo reform (Exhibit E.1).
Executive Summary xii

Exhibit E.1

Eleven Components of Comprehensive School Reform

Described in the No Child Left Behind Act

1. Proven methods and strategies for student learning, teaching, and school management that are based on 

scientifically based research and effective practices and that have been replicated successfully in schools with 

diverse characteristics.

2. Comprehensive design for effective school functioning, integrating instruction, assessment, classroom 

management, and professional development and aligning these functions into a schoolwide reform plan 

designed to enable all students to meet challenging state content and performance standards and to address 

needs identified through a school needs assessment.

3. Professional development, including high-quality and continuous teacher and staff professional development 

and training.

4. Measurable goals for student performance and benchmarks for meeting those goals.

5. Support from staff members, including school faculty, administrators and other staff members.

6. Support for staff members, including school faculty, administrators and other staff members (added in 2001).

7. Parent and community involvement, including meaningful involvement of parents and the local community in 

planning and implementing school improvement activities.

8. External assistance, including high-quality external support and assistance from a comprehensive school 

reform entity (which may be a university) with experience in schoolwide reform and improvement.

9. Evaluation, Including a plan to evaluate the implementation of school reforms and the student results achieved.

10. Coordination of resources, which involves identifying how other available resources (federal, state, local and 

private) help the school coordinate services to support and sustain the school reform.

11. Scientifically based research designed to significantly improve the academic achievement of students 

participating in such programs as compared with students in schools who have not participated in such programs 

or to provide strong evidence that such programs will significantly improve the academic achievement of 

participating children (added in 2001).

Source: Title I, Part F, Section 1606 of the Elementary and Secondary Education Act of 1965, as amended by the No Child Left 

Behind Act of 2001..

ECSRIO, mandated by Section 1607 of the Elementary and Secondary Education Act, as 

amended by the No Child Left Behind Act, addresses several broad questions related to the CSR 

program. 

 How were CSR funds targeted? 

 How was reform implemented in CSR and non-CSR schools? 

 How was reform related to achievement outcomes?

ECSRIO examines the implementation and outcomes of the cohort of CSR awardees that 

received their initial awards in 2002. Awards spanned three years, although some awardees 

carried over funds for an additional year. 

The series of studies conducted for the project view the CSR program through numerous lenses, 

including achievement analyses, surveys of school reform implementation, and multiple case 

studies. 

Three distinct methodological approaches were used:
Executive Summary xiii

 Multivariate statistical analyses comparing all Title I CSR with Title I non-CSR 

matched comparison schools to determine whether receiving a CSR award was 

related to school-level achievement increases three and five years later2

 Quantitative descriptive analyses of CSR reform implementation from a survey of 

principals and teachers in a random sample of 436 CSR and 382 matched nonCSR comparison schools to determine the comprehensiveness of implementation

 Qualitative case study analyses designed to study CSR reform implementation 

and illustrate the ways CSR components were implemented3 as well as to 

understand the process by which chronically low-performing schools turned 

themselves around and sustained achievement gains4

Only a small number of high schools received CSR awards in 2002. Consequently, the sample 

included too few high schools for analysis, so this report does not focus on them.5 This study 

frequently found differences in outcomes between elementary and middle schools, including 

differences in the relationships between achievement and award receipt. Such differences are 

possibly related to the different structures in the two types of schools, with middle schools 

having subject-area departments and elementary schools mainly teaching students in selfcontained classrooms. These structural differences may affect not only the nature of the reform 

approach each level of school adopts but also how well reforms are implemented across all 

grades, subjects and teachers. Consequently, the report includes separate findings for elementary 

and middle schools.

It is important to note that ESCRIO is not a study of the implementation of individual CSR 

models and their relationships with achievement outcomes. Many other researchers have focused 

on individual models (see Borman, Hewes, Overman, & Brown, 2002, for a meta-analysis of 

many of these studies). Instead, ECSRIO is a study of average achievement gains in schools that 

receive CSR grants, irrespective of the models these schools chose.

 

2. The achievement analyses rely on (a) standardized school-level achievement scores from the National 

Longitudinal School-Level State Assessment Score Database (NLSLSASD) maintained by the American 

Institutes for Research (AIR), (b) data supplied by the Education Data Exchange Network (EDEN) housed at 

the U.S. Department of Education, (c) the Council of Chief State School Officers (CCSSO), and (d) states’ 

Web sites.

3. These case studies involved the study of CSR reform implementation in (a) 15 pairs of schools (15 CSR 

schools and 15 matched comparison schools) originally selected to be part of this study to illustrate the ways 

CSR components were implemented in both CSR and non-CSR schools and (b) case studies in 13 schools (9 

improving and 4 comparison schools) selected to extend the previous qualitative case study analyses of CSR 

and non-CSR schools.

4. Results from the study of how underperforming schools were able to turn around their performance appear 

in the Achieving Dramatic School Improvement: An Exploratory Study report (U.S. Department of Education, 

2010). 

5. For more information on the number of high schools versus elementary and middle schools that received 

CSR awards, see Exhibit B-2 on page 57 of Longitudinal Assessment of Comprehensive School Reform 

Program Implementation and Outcomes: First-Year Report (U.S. Department of Education, 2004).
Executive Summary xiv

STUDY LIMITATIONS

The multi-method approach to this evaluation, like most studies of practice (Lipsey, 2003), has 

limitations.

First, all achievement analyses rely on standardized school-level achievement scores from 

NLSLSASD, CCSSO, EDEN, and state Web sites. The use of school-level outcome measures 

presents methodological and psychometric challenges. 

• School-level scores are aggregate measures of individual student performance within a 

school and, as such, do not account for variation in academic performance among 

students within a group. 

• These school level analyses are not sensitive to the fact that performance may be 

contingent on both individual factors and factors related to group membership such as 

membership in different classrooms and school environments. The multileveled structure 

of data is not adequately addressed in aggregate analyses, which limits the precision of 

the estimates produced. 

Despite these weaknesses, designating the school as the unit of analysis and using aggregate 

outcome measures is deemed appropriate given the goals of the evaluation and the nature of the 

CSR program. The analyses rely on aggregate outcome measures that are both meaningful to 

schools and relevant to the current policy environment. School-level proficiency is the 

performance measure for which all schools are held accountable. Although the use of aggregate 

measures may compromise precision, its relevance for school officials and policy-makers offsets 

its limitations. Moreover, the CSR program was designed to stimulate comprehensive change 

through a set of coordinated reform actions that influence school operations across the whole 

school. In this context, it is appropriate to use the school as the unit of analysis and school-level 

achievement measures to examine how the CSR program influences achievement. 

A second limitation of this study is shared with other efforts to assess program effectiveness 

across states over time. Such evaluations must overcome the difficulty of using the existing state 

assessments, which are designed to provide information about students’ progress toward 

mastering the content established in each state’s standards. Consequently, assessments differ. In 

fact, neither the content nor the criteria for determining proficiency are the same from state to 

state. Also, standards, assessments and proficiency criteria often change, making scores within 

states difficult to compare over time. Finally, because of the structure of state assessments 

(especially before NCLB), we were unable to track cohorts of students because such data were 

rare before 2004–05. During the time before NCLB, most states assessed students once during 

the elementary school years (typically fourth grade) and once during the middle school years 

(typically eighth grade). We chose these grades to represent school-level achievement measures 

during the baseline years wherever possible because they were the only data available. 

Throughout the study, we continued to focus on these grades as surrogate measures for schoollevel achievement to ensure consistency in our measures; that is, we sought to ensure that 

students at similar levels of age-specific development were being compared with each other 

rather than attempt to average students across different stages of development in later years 
Executive Summary xv

(when more assessment data were available) and compare them with the original group of fourthand eighth-grade students.

School-level assessment scores were standardized within states for each year to enable us to 

analyze outcomes across states and assessment instruments.6 The standardization provides a 

common metric for all achievement outcomes; however, it does not account for all sources of 

heterogeneity among states resulting from differences in assessments, the stringency of state 

proficiency standards, and the fact that the analyses rely on different groups of students in each 

year. 

Third, this study also may be limited by nonrandom attrition of a large number of schools from 

some analyses because of survey nonresponse, the fact that several states did not report all data 

to the Common Core of Data, and the removal of non-CSR schools that received CSR awards 

after 2002. The unrepresentative nature of missing data in the Year 5 sample and the 

inadvisability of imputing such data from Year 3 responses were factors precluding analyses in 

this report on the relationship between implementation and achievement.

MAIN FINDINGS

In this section, we address the questions of whether schools that received their initial CSR 

awards in 2002 realized achievement gains in mathematics and reading five years later, whether 

CSR schools made gains in implementing the federally mandated components of the CSR 

program, and whether CSR schools adopting curriculum models with a scientific research base 

were more likely to experience subsequent achievement gains. 

Was receipt of a CSR award associated with improvements in school-level mathematics 

and reading achievement?

 Five years after initially receiving their CSR awards, schools receiving awards 

did not demonstrate larger achievement growth than matched comparison 

schools not receiving CSR grants. In fact, achievement gains were nonexistent 

in CSR elementary schools, were marginally lower than their comparison 

schools in middle school mathematics, and were no different from their 

comparison schools in middle school reading.

Achievement in CSR schools was largely unchanged in the five years after receiving a CSR 

award (Exhibit E.2). Comparison schools had higher gains in middle school mathematics and 

reading, although the differences between CSR and comparison schools were only marginally 

significant for mathematics achievement and insignificant for reading achievement. These 

findings are largely consistent with findings from this study’s Third Year Report (U.S. 

Department of Education, 2008).

 

6. See Exhibit 2 on pages 9–16 in the Third Year Report (U.S. Department of Education, 2008) for a list of the 

measures used for each state for the years 1999–00 to 2004–05 of this study. This report updates those data 

with percent proficient in 2006–07; they were the only outcome measures available.
Executive Summary xvi

Exhibit E.2

Average Gains in Percent Proficient or Higher From 2003 to 2007 for CSR and Matched 

Non-CSR Schools

Average Gain in Percent Proficient

N CSR Schools Non-CSR 

Schools Difference

Elementary 

Mathematics 493 n.s. n.s. n.s.

Elementary 

Reading 563 n.s. n.s. n.s.

Middle School 

Mathematics 291 0.2* 2.8** -2.6+

Middle School 

Reading 294 n.s. 2.5* n.s.

Exhibit highlights: Comprehensive School Reform (CSR) schools had a statistically significant 

increase between 2002–03 and 2006–07 in middle school mathematics of 0.2 percentage points. 

However, matched non-CSR schools had larger statistically significant increases in middle school 

mathematics and reading of 2.8 and 2.5 percentage points, respectively. Nevertheless, only middle 

school mathematics had a marginally statistically significant difference between CSR schools and 

their matched non-CSR schools.

Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD), Education Data Exchange Network 

(EDEN), Council of Chief State School Officers (CCSSO), and state Web sites.

Note: “n.s.” indicates not statistically significant; + p < .10; * p < .05; ** p < .01. Tests for the 

statistical significance of achievement gains for CSR and non-CSR schools are t-tests to assess 

whether the value is different from zero. The differences between CSR and non-CSR schools are 

assessed through paired t-tests. Values for percent proficient or higher are computed by 

multiplying the average standard deviation of the achievement measures by the statistically 

significant estimates for the relationships between scientifically based research model adoption 

and school-level achievement.

Were schools that received CSR awards more likely to implement the legislatively 

specified components of CSR than other schools?

 No, consistent with earlier study findings, both CSR and non-CSR schools 

implemented a similar number of legislatively specified components in both 

2003 and 2007. The number of components rose slightly for CSR and nonCSR schools during this period at both the elementary and middle school 

levels.

CSR awards were three-year grants intended to stimulate the implementation of the 11 

components identified in NCLB. However, by 2007, CSR schools, on average, implemented 

fewer than five components five years after receiving their CSR awards (Exhibit E.3). 

Furthermore, implementation of the number of components at the elementary school level was 

nearly identical in both CSR and non-CSR schools. In 2003, CSR elementary schools reported 

implementing slightly more components than non-CSR schools; this difference was statistically 
Executive Summary xvii

significant. However, this difference was no longer statistically significant by 2007. No 

significant differences existed in either 2003 or 2007 in the number of reform components 

implemented by CSR and non-CSR middle schools. Both CSR and non-CSR schools increased 

the number of reform components implemented from 2003 to 2007. 

Exhibit E. 3

Average Number of CSR Components Implemented by CSR

and Non-CSR Schools in 2003 and 2007

Average Number of Components Implemented

Elementary School Middle School

School Type 2003 2007

Change 

from 2003 

to 2007 2003 2007

Change 

from 2003 

to 2007

CSR Schools 3.9 4.6 0.7** 3.4 4.1 0.6*

Non-CSR Schools 3.5 4.4 0.8** 3.2 4.2 1.0**

Difference Between CSR 

Schools and Non-CSR Schools 0.4+ 0.2 0.1 0.2 -0.1 0.3

Exhibit highlights: In 2003, Comprehensive School Reform (CSR) elementary schools 

reported implementing a somewhat higher average number of components than non-CSR 

schools. No other statistically significant differences between CSR and non-CSR schools 

exist. Both CSR and non-CSR schools reported similar increases between 2003 and 2007 in 

the number of components implemented.

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) surveys.

Note: + p < .10; * p < .05; ** p < .01. N = 150 for CSR elementary schools; N = 124 for nonCSR elementary schools; N = 112 for CSR and non-CSR middle schools.

Adoption of Scientifically Based Research Models and Achievement Outcomes

In the Third-Year Report, we reported on whether schools that received CSR awards adopted 

models with a scientific research base and whether that adoption was associated with 

achievement gains after three years.7 Those ratings derived from the Comprehensive School 

Reform Quality (CSRQ) Center ratings8 of the scientific research base in commonly adopted 

reform models to assess the extent to which the use of CSR approaches with such a base was 

associated with higher achievement. We found that only one-third of CSR schools adopted 

models that later received ratings from CSRQ that indicated they had a scientific research base.

Most schools chose models that were not rated by CSRQ.

 

7. See U.S. Department of Education (2008).

8. The CSRQ Center ratings provide a scale for the breadth and quality of the research base for the 31 most 

widely adopted CSR models in elementary, middle and high schools. The CSRQ Center conducted a total of 

40 ratings because some models were rated at both the elementary and secondary levels. The categories are 

Very Strong, Moderately Strong, Moderate, Limited, Zero and Negative. The CSRQ Center was funded under 

the U.S. Department of Education’s Comprehensive School Reform Quality Initiatives program.
Executive Summary xviii

In general, schools that adopted comprehensive school reform models with scientific research 

bases had higher mathematics achievement gains compared with reading gains after three years. 

Mathematics achievement improved in low-performing elementary schools9 and in all middle 

schools (including low-performing schools) that adopted models with scientific research bases. 

Mathematics achievement improved in low-performing CSR elementary schools that adopted 

models with scientific research bases, but it did not improve in non-low-performing CSR 

elementary schools (Exhibit E.4).10 Low-performing CSR elementary schools gained about 2 

percentage points in the percentage of students scoring at the proficient level or higher in 

mathematics if they adopted a model with a limited scientific research base and 4 percentage 

points if they adopted a model with a moderate or moderately strong scientific research base. 

There was no relationship between changes in elementary reading achievement and the selection 

of a model with a scientific research base. 

At the middle school level, all CSR schools (including low-performing schools) that adopted 

models with limited scientific research bases realized gains after three years in the percent 

proficient or higher levels (about 6 percentage points in mathematics and 3 percentage points in 

reading), although the relationship with reading achievement is only weakly significant.11

Furthermore, low-performing CSR middle schools that adopted a model with a moderate or 

higher research base had gains of about 4 percentage points in the percent proficient or higher 

levels in mathematics, although this relationship, too, is also only weakly significant.

One possible explanation for these inconsistent findings lies with the way the CSRQ Center 

determined ratings. The CSRQ Center determined “strength of the research base” by the nature 

of the evaluation design, with more weight given to more rigorous designs and the number of 

models evaluated that use such designs. It is possible that some potent strategies had not been 

evaluated rigorously, so the ratings underplay their potential for effectiveness. It is also possible 

that implementation in schools in this study may have been less comprehensive than it was in 

schools included in the studies that the CSRQ Center used. Note that we did not update these 

results for this Fifth-Year Report because we had no way of knowing whether these schools 

continued to implement the models they chose five years earlier. 

 

9. Low-performing schools refer to those CSR schools that were in the lowest 25 percent of achievement of all 

CSR schools in 2001–02. 

10. The CSRQ rating scale includes Very Strong, Moderately Strong, Moderate, Limited, Zero and Negative 

categories. No model received a “Negative” or “Very Strong” rating. The analyses that examine the 

relationship between strength of the research base and achievement treat the scientific research base variable as 

a series of dummy coded variables for Limited and Moderate/Moderately Strong CSRQ ratings. 

11. Throughout this report, findings that are reported as weakly significant are statistically significant at p < 

.10.
Executive Summary xix

Exhibit E.4

Average Gains in Percent Proficient or Higher From 2002 to 2005, by Subject Area and

Strength of Scientific Research Base

Average Gains in Percent Proficient or Higher 

Elementary School Middle School

Mathematics Reading Mathematics Reading

All CSR Schools

Limited Scientific Research Base n.s. n.s. 5.92** 3.36+

Moderate/Moderately Strong 

Research Base n.s. n.s. n.s. n.s.

Low-Performing CSR Schools

Limited Scientific Research Base 1.92** n.s. 5.92** 3.36+

Moderate/Moderately Strong 

Research Base 4.00* n.s. 4.16+ n.s.

Exhibit highlights: At the elementary level, low-performing Comprehensive School Reform 

(CSR) schools that adopted models with limited or moderate to moderately strong scientific 

research bases had higher mathematics achievement; there was no relationship between reading 

achievement and models with a scientific research base. At the middle school level, CSR 

schools that adopted models with limited scientific research bases had higher mathematics and 

reading achievement gains in the percent proficient or higher levels than those that did not 

adopt such models; low-performing CSR schools that adopted models with moderate to 

moderately strong research bases had higher mathematics achievement than those that chose 

other models.

Source: From “Comprehensive School Reform Quality (CSRQ) Center Report on Elementary 

School Comprehensive School Reform Models,” by CSRQ Center, 2005, Washington, DC: 

American Institutes for Research; “Comprehensive School Reform Quality (CSRQ) Center 

Report on Middle and High School Comprehensive School Reform Models,” by CSRQ Center, 

2006, Washington, DC: American Institutes for Research; CSR Awards Database; Common 

Core of Data (CCD); National Longitudinal School-Level State Assessment Score Database 

(NLSLSASD).

Note: n.s. indicates not statistically significant; + p < .10; * p < .05; ** p < .01. Values for 

percent proficient or higher are computed by multiplying the average standard deviation of the 

achievement measures by the statistically significant estimates for the relationships between 

scientifically based research model adoption and school-level achievement.

OVERALL STUDY CONCLUSIONS 

In this section, we present the key overall findings from the multiple studies forming the 

ESCRIO project as well as the major implications for future federal education policy-making and 

for conducting high-quality research and evaluation studies in this area.
Executive Summary xx

Effectiveness of the Federal CSR Program

The federal CSR program did not yield comprehensively reformed schools. Although states 

largely succeeded in providing CSR funds to those schools most in need, schools receiving CSR 

awards made little progress in implementing more than just a few of the legislatively mandated 

components and were largely indistinguishable from non-CSR schools that were similar in 

baseline achievement and demographics. 

Furthermore, the federal CSR program was not associated with widespread achievement gains. 

Although CSR schools did realize improvements in mathematics and reading achievement after 

three years of their initial awards, they improved at quite similar rates as those of comparable, 

non-CSR Title I schools. After five years, any gains that CSR schools made had evaporated so 

that mathematics and reading achievement were similar to baseline levels at the time they 

received their awards. 

There were some instances where we did observe achievement gains, however. These gains were 

somewhat more likely in mathematics in schools that selected models identified as having a 

scientific research base. However, only one-third of the schools receiving CSR awards selected 

reform models identified as having a scientific research basis.

Examples of School Improvement

Despite the finding that the federal CSR program as a whole was not associated with 

improvements in achievement or school operations, we did find instances where targeted, 

sustained efforts appeared to lead to achievement gains.12 Results from the Achieving Dramatic 

School Improvement: An Exploratory Study report (U.S. Department of Education, 2010) not

only highlighted instances of noteworthy achievement gains among initially low-performing 

schools but also expanded the knowledge base about the challenges and intricacies of turning 

around achievement in these schools. In this exploratory set of case studies of rapidimprovement schools (i.e., initially low-performing schools that made substantial achievement 

gains in a short time period) and schools improving at a slower and steadier pace, we identified 

leadership, school climate, instruction and data use, and external support as important factors in 

the schools visited. At the same time, we observed that these factors were “put together” by 

schools in many different ways. Some, for example, placed greater emphasis on distributed 

leadership over transparent use of student-level data while others did the reverse. Further, the 

reform strategies interacted with each other and with their school’s preexisting context in 

multiple ways. We believe these cases add to a growing body of literature associating specific 

school-level policies and practices with school improvement while also demonstrating that there 

is no “one way” to implement an effective school improvement program because different 

factors play out in different ways in different settings. 

 

12. A substudy was undertaken of CSR schools that made dramatic improvements in achievement, focusing on 

factors such as the role of leadership, use of data, school climate, and external support in helping to turn around poor 

performance. While results are reported separately in the Achieving Dramatic School Improvement (U.S. 

Department of Education, 2010), we provide a summary on pages 51–56 of this report.
Executive Summary xxi

RECOMMENDATIONS FOR FEDERAL POLICY AND RESEARCH

Given the limited effects of the CSR program, how can the federal government foster scalable or 

systemwide reform that resembles the successes observed in the case study sites? We would 

posit that with no “one way” to implement a successful school improvement program, federally 

supported improvement will require highly contextualized and resource intensive support. The 

U.S. Department of Education would need to begin to provide substantial amounts of direct and 

individually tailored technical assistance to state departments of education and possibly also to 

local school districts. The assistance the Department would provide to states could mirror the 

situationally dependent assistance some states in the case study sites provided to districts and 

schools.

Future research for improving chronically low-performing schools could play a significant role 

in shaping the nature of not only subsequent federal efforts but also the efforts of other service 

providers so they are targeted on interventions most likely to result in positive effects. To do so, 

we recommend that a future research agenda address three critical needs. First, there is a need for 

a robust national database of school information—including achievement measures, school 

characteristics (e.g., demographics and staffing), budgets, and relevant reform activities—to be 

made publicly available. This recommendation supports the second recommendation—the study 

of school improvement in “real-time” instead of retrospective efforts. Third, the effective study 

of school reform relies on the effective measurement of reform activities. Well-developed school 

reform implementation measures should be both specific enough to effectively characterize the 

school reform effort and broad enough to allow for comparison across school contexts; research, 

then, can effectively identify that set of practices a school facing a particular set of circumstances 

can adopt and feel reasonably confident that those practices have been shown in previous

research to yield positive reform. 
Executive Summary xxii
Chapter I 1

I. INTRODUCTION

The Comprehensive School Reform (CSR) program was first established as a demonstration 

program in 1998. The program was subsequently authorized under the Elementary and 

Secondary Education Act of 1965, as amended by the No Child Left Behind Act of 2001 (NCLB).

It is one approach to help low-performing K–12 public schools meet state performance 

standards. The CSR program emphasizes two major concepts. First, the approach mandates that 

school reform should be comprehensive in nature, strengthening all aspects of school 

operations—curriculum, instruction, professional development, parental involvement and school 

organization. Second, the CSR approach involves the use of scientifically based research 

models—that is, models with evidence of effectiveness in multiple settings. 

Originally funded as the Comprehensive School Reform Demonstration (CSRD) Program in 

1998 with $145 million, the program became part of NCLB in 2002 with funding of $310 

million. In FY 2003 and FY 2004, Congress allocated $308 million for the CSR program, and in 

FY 2005, $205 million. Between 1998 and 2006, nearly 7,000 schools nationwide received 

three-year awards to implement CSR models. In FY 2006, funding was appropriated only for a 

clearinghouse to support comprehensive school reform and not for school-based activities. Since 

then, there have been no federal funds dedicated to schools under the CSR program.13

NCLB defines CSR as containing 11 components, which are assumed to work together as schools 

undergo reform (Exhibit 1). For example, the “support from staff members” component of the 

CSR program dictates that the entire school should adopt the reform model, while the “support 

for staff members” component focuses on helping teachers and other staff members learn to use 

the reform strategy. This report studies these components and their implementation. 

 

13. Title I schoolwide schools may use Title I funds to fund the adoption of CSR models. Furthermore, the 

clearinghouse has continued to receive federal funds.
Chapter I 2

Exhibit 1

Eleven Components of Comprehensive School Reform

Described in the No Child Left Behind Act

1. Proven methods and strategies for student learning, teaching, and school management that are based on 

scientifically based research and effective practices and that have been replicated successfully in schools with 

diverse characteristics.

2. Comprehensive design for effective school functioning, integrating instruction, assessment, classroom 

management, and professional development and aligning these functions into a schoolwide reform plan 

designed to enable all students to meet challenging state content and performance standards and to address 

needs identified through a school needs assessment.

3. Professional development, including high-quality and continuous teacher and staff professional development 

and training.

4. Measurable goals for student performance and benchmarks for meeting those goals.

5. Support from staff members, including school faculty, administrators and other staff members.

6. Support for staff members, including school faculty, administrators and other staff members (added in 2001).

7. Parent and community involvement, including meaningful involvement of parents and the local community in 

planning and implementing school improvement activities.

8. External assistance, including high-quality external support and assistance from a comprehensive school 

reform entity (which may be a university) with experience in schoolwide reform and improvement.

9. Evaluation, Including a plan to evaluate the implementation of school reforms and the student results achieved.

10. Coordination of resources, which involves identifying how other available resources (federal, state, local and 

private) help the school coordinate services to support and sustain the school reform.

11. Scientifically based research designed to significantly improve the academic achievement of students 

participating in such programs as compared with students in schools who have not participated in such programs 

or to provide strong evidence that such programs will significantly improve the academic achievement of 

participating children (added in 2001).

Source: Title I, Part F, Section 1606 of the Elementary and Secondary Education Act of 1965, as amended by the No Child Left 

Behind Act of 2001.

In 2001, the U.S. Department of Education contracted with WestEd to conduct a study of the 

Federal CSR program. The Evaluation of the Comprehensive School Reform Program 

Implementation and Outcomes (ECSRIO), mandated by Section 1607 of the Elementary and 

Secondary Education Act, as amended by the No Child Left Behind Act, addresses four broad 

questions related to the CSR program. 

 How were CSR funds targeted? 

 How was reform implemented in CSR and non-CSR schools? 

 How did state and district conditions influence reform implementation?

 How was reform related to achievement outcomes?
Chapter I 3

ECSRIO examines the implementation and outcomes of a cohort of CSR awardees that received 

their initial awards in 2002. Early findings in this study illustrated that states targeted CSR funds 

to low-performing, high-need schools. The Third-Year Report (U.S. Department of Education, 

2008) found that neither the receipt of CSR awards nor the implementation of CSR components 

were associated with achievement gains in mathematics or reading achievement. There were, 

however, mixed relationships between achievement and the adoption of models with scientific 

research bases. 

As policy-makers grappled with the increasing numbers of schools not making Adequate Yearly 

Progress (AYP) under NCLB, the U.S. Department of Education asked WestEd and the 

American Institutes for Research (AIR) to identify those CSR schools that made rapid 

improvement along with some making “slow and steady progress” during the study period and to 

“unpack” the efforts of those schools to understand what actions led to their improvements. In 

the 2007-08 school year, teams of researchers from WestEd and AIR visited 11 such schools and 

interviewed current and former school and district staff members, parents and community 

members. The results from these case studies can be found in the report Achieving Dramatic 

School Improvement: An Exploratory Study (U.S. Department of Education, 2010).

This fifth-year, final report from the ECSRIO project serves two purposes. First, we update the 

third-year report by addressing whether schools that received CSR awards had higher 

achievement gains and/or made greater progress implementing the 11 components than did nonCSR schools. Specifically, we address the following questions:

 Was receipt of a CSR award associated with improvements in school-level 

mathematics and reading achievement five years later?

 Were schools that received CSR awards more likely five years later to implement 

the legislatively specified components of CSR than other schools?

The next chapter describes the data sources used for the Year 5 analyses. The subsequent two 

chapters report on the overall relationship between CSR award and achievement (Chapter III) 

and the comprehensiveness of CSR implementation (Chapter IV). These two chapters include 

discussions of methodological approaches because each question relies on different 

methodologies and on samples constructed in different manners. The report appendixes include 

the standard error tables (Appendix A) and data collection instruments (Appendix B).

The second purpose of this report is to describe and discuss what we learned over the course of 

the entire study about school reform. The final chapter presents a summary of key findings from 

the entire ECSRIO study, focusing primarily on the lack of any relationship between the federal 

CSR program and achievement gains and specifically contrasting that finding with those from 

this project’s intensive case studies of dramatic school improvement and from the National 

Longitudinal Evaluation of Comprehensive School Reform (Aladjem et al., 2006), both of which 

point to instances where school reform efforts can be effective. This chapter also presents 

recommendations for improved federal policy and research support for school reform and 

improvement.
Chapter I 4
Chapter II 5

II. DATA SOURCES

This study compares implementation and school-level achievement between schools that 

received CSR awards and similar schools that did not receive such awards. Comparisons were 

done by using multiple research methods for collecting and organizing data and by incorporating 

information from teachers and principals about reform in their schools. Analyses included the 

following:

 Multivariate statistical analyses comparing all Title I CSR with Title I non-CSR 

matched comparison schools to determine whether receiving a CSR award was 

related to school-level achievement increases over time14

 Quantitative descriptive analyses of CSR reform implementation from a survey of 

principals and teachers in a random sample of 436 CSR and 382 matched nonCSR comparison schools15

 Qualitative case study analyses of CSR reform implementation in 15 pairs of 

schools (15 CSR schools and 15 matched comparison schools) originally selected 

to be part of this study to illustrate the ways CSR components were implemented 

in both CSR and non-CSR schools and in 13 schools (9 improving and 4 

comparison schools) chosen to extend the previous qualitative case study analyses 

of CSR and non-CSR schools

Using multiple methods increases the validity of key measures and results. In addition, this study 

is an evaluation of a program as it operates in the field. Such an evaluation cannot control the 

conditions under which activities take place. Consequently, the study cannot use an experimental 

design, and no causal claims can be made. The evaluation uses a quasi-experimental design, 

matching schools that received CSR funds with schools that did not receive funding, in both the 

quantitative and qualitative aspects of data collection and analysis. 

This section describes the samples and the following three data sources for ECSRIO: 

 School-level achievement measures

 Surveys of school reform activities

 Case studies of school reform activities

Because each evaluation question relies on different methods and samples constructed in 

different ways, discussions of the specific methods used are included with the findings for each 

evaluation question. The last part of this section presents study limitations.

 

14. The achievement analyses rely on standardized school-level achievement scores from the National 

Longitudinal School-Level State Assessment Score Database (NLSLSASD) maintained by AIR, data supplied 

by the Education Data Exchange Network (EDEN) housed at the U.S. Department of Education, the Council of 

Chief State School Officers (CCSSO), and states’ Websites.

15. Most CSR awards were given to elementary and middle schools. Consequently, the sample included too 

few high schools for analysis.
Chapter II 6

SCHOOL-LEVEL ACHIEVEMENT MEASURES

The achievement analyses rely on standardized school-level achievement scores from the

National Longitudinal School-Level State Assessment Score Database (NLSLSASD) maintained 

by AIR, EDEN, CCSSO, and state Web sites. The NLSLSASD includes school-level measures 

of achievement for nearly all public schools. This database includes seven years of data, 

spanning 1998–99 through 2004–05; however, many states did not have or provide assessment 

data for the first year or two. Also, before 2003–04, nearly all states tested only a few grades. 

Therefore, for this study, one grade (typically fourth) was chosen to represent elementary 

achievement, and one grade (typically eighth) was chosen to represent middle school 

achievement in the school. School-level achievement data for 2006–07 were obtained from 

EDEN at the U.S. Department of Education and were supplemented with data from School Data 

Direct and from individual states’ Web sites.16 During the time before NCLB, most states 

assessed students once during the elementary school years (typically fourth grade) and once 

during the middle school years (typically eighth grade). We chose these grades to represent 

school-level achievement measures during the baseline years wherever possible because they 

were the only data available. Throughout the study, we continued to focus on these grades as 

surrogate measures for school-level achievement to ensure consistency in our measures; that is, 

we sought to ensure that students at similar levels of age-specific development were being 

compared with each other,, rather than attempt to average students across different stages of 

development in later years (when more assessment data were available) and compare them with 

the original group of fourth- and eighth-grade students.

In most instances, the average scale score or percent proficient or higher was used to calculate a 

standardized z-score within each state for each year.17 In several instances, however, other 

measures such as percentile ranks were the only available outcome measures, and in those 

situations, standardization was done in the following manner:

1. In each year, appropriate assessment measures for mathematics and reading/language arts 

in elementary and middle grades were selected. 

2. In many cases, fourth-grade scores were used for elementary schools and eighth-grade 

scores for middle schools; however, where those scores were not available, other 

proximate grades were used.

3. The schools in each state in each year were ranked according to their achievement 

measure. 

4. Percentile ranks were computed from these rankings.

 

16. School Data Direct is compiled and supported by CCSSO. It can be accessed online at 

http://www.schooldatadirect.org/.

17. The z-score is calculated by subtracting the population mean from an individual score and dividing it by the 

population standard deviation.
Chapter II 7

5. Percentile ranks were converted to normal curve equivalent scores.

6. Normal curve equivalent scores were converted to z-scores, so each state was standardized 

with a 0 mean and a standard deviation of 1.

Where no assessment scores were available for a state in an academic year, the standardized 

score was estimated by averaging the standardized score from the prior year and that of the year 

after. For instance, there were no assessment measures available for Idaho in 2004. The 

elementary mathematics score for each school in Idaho in 2004 was computed by averaging the 

2003 and 2005 standardized scores in elementary school mathematics for each school. 

Because the standardized scores are standard deviations, coefficients are comparable to effect 

sizes; that is, coefficients are interpreted as changes in standard deviations of the outcome 

measure. Coefficients that result from analyses can also be converted to more interpretable 

measures such as percent proficient or higher (Gill et al., 2005). For example, if the CSR 

program accounts for an increase in achievement scores of 0.5 standard deviations, this value can 

be multiplied by the average standard deviation of the percentage of students who achieve at the 

proficient level or higher across states to develop a rough estimate of the increase in achievement 

levels associated with the CSR approach. Exhibit 2 presents the standard deviations of 

elementary and middle school mathematics and reading achievement calculated from the 45 

states in which achievement data were expressed as a percentage of those achieving proficient or 

higher levels in 2007. The average standard deviation for each of the achievement measures in 

elementary school mathematics and reading is approximately 20 percent and, for middle school 

mathematics and reading, 24.6 percent. Therefore, a finding that the CSR program accounts for 

an increase in achievement scores of 0.5 standard deviations corresponds to an increase of about 

10 percent in elementary schools and 12.3 percent in middle schools. 
Chapter II 8

Exhibit 2

Standard Deviations of Percent Proficient or Higher for 2007 Elementary and

Middle School Mathematics and Reading Achievement, by State

Elementary School Middle School 

State Mathematics Reading Mathematics Reading

AK 23.7 24.7 25.4 24.8

AL 14.0 10.8 17.3 17.3

AR 17.4 14.8 16.6 16.6

AZ 17.3 19.1 22.8 22.5

CA 19.0 20.4 21.3 21.3

CO 9.3 13.0 17.1 17.1

CT 17.6 21.4 18.2 18.2

DE 17.2 16.7 32.0 31.1

FL 17.7 16.4 23.7 23.2

GA 13.0 10.8 14.9 15.0

HI 15.7 15.9 12.3 12.3

ID 11.4 10.8 14.4 14.4

IL 13.6 17.6 15.1 15.1

IN 13.3 13.0 16.2 16.2

KY 16.6 13.0 15.0 15.1

LA 21.1 18.6 23.5 23.5

MA 12.7 11.3 19.9 19.9

MD 12.5 11.9 24.4 24.4

ME 16.9 14.7 16.7 16.7

MI 15.0 13.3 22.9 22.9

MN 18.0 17.9 19.5 19.4

MO 18.4 16.0 18.6 18.6

MS 13.6 7.4 15.3 15.3

MT 16.2 12.6 19.6 19.6

NC 16.3 10.7 20.7 20.7

ND 15.1 14.3 19.3 18.7

NE 9.1 10.0 11.0 11.0

NH 18.3 18.3 19.9 19.9

NJ 14.0 16.1 21.9 19.5

NV 14.8 16.1 18.8 19.2

NY 15.2 18.4 23.4 21.4

OH 19.7 16.9 21.8 21.6

OR 14.1 11.6 15.4 14.9

PA 16.8 18.0 20.1 20.1

SC 15.0 15.5 13.9 13.4

[continued on next page]
Chapter II 9

[continued from previous page]

SD 12.7 9.4 14.4 14.4

TN 11.4 11.3 12.8 12.8

TX 12.3 11.3 16.8 16.3

UT 13.9 12.9 20.1 20.1

VA 11.1 8.4 30.2 18.0

VT 15.3 13.8 15.1 15.0

WA 17.1 13.0 17.9 18.0

WI 16.2 12.7 21.3 21.0

WV 12.2 9.2 9.6 9.6

WY 10.5 15.5 21.1 21.1

Average 19.9 20.1 24.6 24.6

Exhibit highlights: The average standard deviation of all of the states was between 19.9 and 

24.6 across school level and mathematics and reading subject areas.

Source: Education Data Exchange Network (EDEN), Council of Chief State School Officers 

(CCSSO), and state Web sites.

SURVEY OF SCHOOL REFORM ACTIVITIES

The implementation analyses rely on survey data from a random sample of CSR schools and 

matched comparison non-CSR schools developed at the outset of this study. The survey provides 

descriptive quantitative data on reform implementation—that is, the extent that schools 

implemented the 11 elements of the CSR program included in NCLB and the extent that schools 

engaged in other non-CSR reform-related activities. The survey also examines other factors that 

research has associated with successful reform such as school organization (e.g. instructors teach 

all subjects versus the same subject to most classes; common planning periods). 

The evaluation team ensured that the sample of schools in the ECSRIO survey represented the 

population of schools by randomly selecting a large enough number of CSR awardees for the 

survey sample. When the sample for the study was drawn at the beginning of this study, the 

initial CSR universe for calendar year 2002 comprised 38 states and 1,096 schools. The sample 

of 400 schools was about 36 percent of this universe, representing 35 states. The evaluation team 

then updated the sample based on new data, increasing the sample by 100 schools from the 15 

remaining states (further explained below). 

To select the sample, researchers obtained the most complete list of CSR schools from the U.S. 

Department of Education’s CSR Awards Database. The Department of Education required states 

to report their awardees to the CSR Awards Database. However, the database did not include all 

states from the year 2002 for two reasons. First, although data collection had been ongoing, at 

any given time the data may be incomplete because of delinquent reporting by state education 

agencies. Second, many states award CSR funds to schools biennially or irregularly.
Chapter II 10

Of the initial 15 states in the 2002 CSR universe that were missing from the original sample, 

three states were left out as a by-product of the random selection process. Five additional states 

reported data after the initial sample selection. The remaining seven states did not make any 

awards in 2002 but reported 2003 data. During the first year of the study, the Department of 

Education allocated additional funds to gather data from these 15 states.

Researchers purposefully selected an additional 100 schools from a larger random sample of 

these missing states to ensure that the sample represented all states and school levels. Also, to 

measure the value added by CSR to Title I schoolwide grants, a larger comparison group of 

schools with non-CSR Title I schoolwide grants was required. Thus, in the second sample, the 

choice of comparison schools was limited to those classified as having Title I schoolwide grants 

in 2002. The resulting combined sample of 500 CSR schools makes up 37 percent of the 

eventual (now relatively stable) universe of 1,340 awardees of the 2002 cohort. Survey data from 

the original sample have been combined with the follow-up sample for analyses in this report. 

Only a small number of high schools received CSR awards in 2002. Consequently, the high 

school survey sample was too small for statistical analyses, so this report focuses on elementary 

and middle schools, but not high schools.

Researchers used a two-step process to select potential matches for comparison with CSR 

schools. First, they created a school equivalency index for all schools in each state (where data 

were available). Second, they calculated a proximity score between each pair of schools within a 

state. Matching schools were selected that had the closest proximity on the index to CSR schools 

within the same district.

A regression-based approach to weighting and combining background characteristics was used to 

construct the school equivalency index for each state. This method is a simplified version of the 

California School Characteristics Index (which was used by the Technical Design Group of the 

Advisory Committee for the Public Schools Accountability Act of 1999). Using the NLSLSASD, 

the team regressed measures of academic performance on measures of schoolwide participation 

in federally subsidized free or reduced-price lunch programs and on schoolwide counts of 

student ethnicity. The estimated coefficients led to a composite of background characteristics for 

each school. In short, each background characteristic was weighted by the amount it contributed 

to student performance.

The team selected potential matches for each first-year CSR school based on a minimum 

distance criterion. This method was used in the majority of cases. However, alternative methods 

were used in two circumstances. First, in some districts (or states), either demographic or 

performance data were not available. Second, a suitable comparison non-CSR school was not 

available within the same district because the district was too small or all other comparable 

schools had previously participated in the CSR program. Each of these contingencies is 

discussed below.

In some cases, not enough data were available to construct a school equivalency index. Where 

states or districts did not report free or reduced-price lunch or ethnicity, schools were ranked 

within districts using only achievement scores. Examples of states where these data were not 

available are Tennessee and Washington. Achievement data were missing for some schools or 

districts. The proximity scores in these cases were based on an unweighted composite of free or 
Chapter II 11

reduced-price lunch and ethnicity. Finally, in cases where neither student performance nor 

demographic data were available, non-CSR comparison schools were matched by school grade 

span, size, and locale.

In districts where a non-CSR comparison school could not be selected, the team searched for a 

suitable comparison in an adjacent district or similar locale. Because the school equivalency 

index included all public schools in the state, the proximity of any school within the state could 

be calculated. The same criteria were used for selection across districts when data were available. 

In cases where data were not available, the team used the same procedures that applied to 

selecting comparison schools within districts.

Among the schools selected at the outset of this study to chart implementation over the life of the 

CSR awards and beyond, non-CSR schools in the sample were higher-performing than the CSR 

schools in the sample and among all newly-awarded CSR schools in 2002. The baseline 

achievement in the ECSRIO sample of elementary CSR schools was slightly more than 0.3 

standard deviations lower than their matched non-CSR comparisons while the differences were 

approximately 0.4 for middle schools (Exhibit 3). This difference may be related to the 

procedures used to select CSR award recipients. The baseline achievement for the sample of 

CSR schools was higher than the baseline achievement for the universe of CSR schools. This 

difference may be because of the purposeful selection of the additional 100 schools.

Exhibit 3

Average Baseline Standardized Achievement Measures of All 2002 CSR

Schools and of the CSR and Non-CSR Schools in the ECSRIO Sample

Standardized Achievement Measures (z-scores)

CSR Universe(SD)

ECSRIO CSR 

Sample 

Schools(SD)

ECSRIO Non-CSR 

Sample 

Schools(SD)

Elementary School 

Mathematics -0.95 -0.82 -0.50

Elementary School Reading -0.95 -0.86 -0.54

Middle School Mathematics -0.90 -0.87 -0.47

Middle School Reading -0.91 -0.87 -0.46

Exhibit highlights: The average baseline standardized achievement measures of the 

Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 

(ECSRIO) Comprehensive School Reform (CSR) schools was slightly more than 0.3 standard 

deviations lower than the average baseline standardized achievement measures of the 

ECSRIO non-CSR schools.

Source: CSR Awards Database; National Longitudinal School-Level State Assessment Score 

Database (NLSLSASD).

In their demographic compositions in 2007, the samples of ECSRIO CSR and non-CSR schools 

were similar to each other and to the universe of schools that received their CSR awards in 2002 

(Exhibit 4). The universe of CSR schools had a higher percentage of students who were eligible 

for free or reduced-price lunch than did the ECSRIO CSR or non-CSR schools. The universe of 
Chapter II 12

CSR schools had a higher percentage of black students than either the ECSRIO CSR schools or 

the non-CSR schools. The percentage of Hispanic students was similar across all categories. 

Exhibit 4

Average Demographic Measures in 2007 of All 2002 CSR Schools

and of the CSR and Non-CSR Schools in the ECSRIO Sample

Demographic Measures CSR Universe

ECSRIO CSR 

Sample Schools

ECSRIO Non-CSR 

Sample Schools

Free or Reduced-Price Lunch (%) 71.24 61.10 57.43

Black (%) 43.05 36.15 31.92

Hispanic (%) 19.09 18.80 19.70

Type of School (%)

Elementary School (includes K–8) 60.96 66.60 69.00

Middle School 20.74 16.08 13.28

High School 14.83 15.24 14.69

Other Configuration 3.47 2.09 3.27

Exhibit highlights: The average percentages of students receiving free or reduced-price lunch, 

average percentage of black students, and average percentage of Hispanic students in the 

Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 

(ECSRIO) Comprehensive School Reform (CSR) schools in 2007 were about the same as in 

the ECSRIO non-CSR schools.

Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal 

School-Level State Assessment Score Database (NLSLSASD).

From the 1,000 schools originally selected for the survey sample, 961 agreed to participate in the 

study and were surveyed in 2003 and again in 2005. Principals and up to three teachers in each 

school were surveyed on various aspects of their respective school’s reform activities (see 

Appendix B for the survey instruments). The response rate for the first-year data collection 

(2003) was 90 percent for the 479 CSR schools and 82 percent for the 482 non-CSR schools. The 

response rate for the third-year data collection (2005) was 88 percent for CSR schools and 78 

percent for non-CSR schools. The response rate for surveyed staff members was 77 percent and 

64 percent for principals and teachers, respectively, in 2003 and 75 percent and 71 percent, 

respectively, in 2005. 

In spring 2007, the third round of ECSRIO surveys was administered to principals and teachers. 

Several districts declined to participate in this round of surveys, reducing the initial sample from 

961 to 879 CSR and matched non-CSR schools. In addition, 61 non-CSR schools that received 

CSR awards after 2002 were removed from the sample so that the comparison would be between 

schools that received a 2002 CSR award and those that had not received a CSR award at any 

time, further reducing the total sample size to 818 schools (436 CSR and 382 non-CSR schools). 

Principals and up to 15 teachers in each school were surveyed on various aspects of their 

respective school’s reform activities. The principal response rate was 65 percent and 66 percent 

for CSR and non-CSR schools, respectively. Because principals were instructed to provide up to 

15 teachers with surveys and the number of possible teacher responses from school to school 

varied considerably, a conventional response rate for teachers cannot be calculated. 
Chapter II 13

Administration of the teacher survey yielded at least one teacher response from 87 percent of 

CSR schools and 83 percent of non-CSR schools.

The survey instrument includes items that measure behaviors rather than attitudes and 

expectations. For example, respondents were asked about procedures that monitor student 

achievement during implementation rather than whether the reform strategy meets student needs. 

Although this approach is likely to increase the objectivity of the responses, self-reported 

responses may still have limitations such as the accuracy of respondent recall and the tendency 

for respondents to give socially desirable responses (Flower, 1995; Tourangeau, Rips, & Rasinki, 

2000). Survey findings were analyzed in conjunction with the results of the case studies to 

validate information on implementation from both sources. 

CASE STUDIES OF SCHOOL REFORM ACTIVITIES

The case studies included in this evaluation provide in-depth insight into comprehensive school 

reform implementation processes and outcomes. They also provide clues for interpreting the 

survey data to determine whether self-reported data reflect inflated or conservative responses.

Of the 1,000 schools selected for the survey sample, a subsample of 15 pairs of schools from 15 

districts across 14 states—with each pair containing a CSR school and a non-CSR school—was 

originally selected to provide case studies that supplement the quantitative analyses with 

qualitative understanding of implementation dynamics. For example, although almost all schools 

reported having a comprehensive school plan, the case studies revealed differences in how the 

plan was used, with some schools developing a plan as a ritual activity and others making it into 

a living document. With such a small number of case study schools, this sample does not 

represent either the geographic distribution or the distribution of school levels in the universe of 

CSR schools. However, the selection ensured that the case study sample included enough 

geographic and school-level diversity to reflect a variety of state policy environments, including 

schools in states with a long tradition of local control and others in states that exercise more 

centralized control over schools.

Researchers selected CSR and comparison pairs that resided in the same district for the fieldbased study because it was the only way to observe the differential effect of district policies. 

Also, visiting a single district for each pair minimized the data collection burden for both the 

evaluators and respondents.

The case studies included two visits to each site, with a “site” defined as a combination of four 

entities:

1. a CSR-funded school,

2. a demographically matched non-CSR school that did not receive any federal CSR funds 

before 2002 and is located in the same district as the CSR-funded school,

3. the district within which the two schools are located, and
Chapter II 14

4. the state education agency in the state within which the district is located.

The first phase of case study data collection occurred from spring 2003 to spring 2004. A second 

phase of site visits occurred during the 2004–05 school year. During visits to the 15 pairs of 

schools, the evaluation team used a formal observation instrument to observe classrooms ; 

reviewed school documents and materials; and interviewed state and district officials, school 

administrators, teachers, reform developers, and parents. 

A new round of case studies was conducted in spring 2007 through fall 2007 to extend the 

analyses of the earlier case studies. This study, known as the Improving Schools Study, involved 

13 schools identified by WestEd. Of this group of 13 schools, 9 were considered “improving 

schools,” and the other 4 were “comparison schools.” Ten of the schools in this sample were new 

to the case study analysis, while 3 were from the original group of 30 case study schools. 

Furthermore, 7 of the improving schools were CSR schools, and 2 were non-CSR schools. One 

of the comparison schools was a CSR school, and 3 were not. Site visitors used the same data 

collection protocols that were used in the 2004–05 round of site visits to the original 15 case 

study sites.

The case studies include information related to reform implementation as well as its relationship 

to district and state policies, including those policies that do not focus directly on comprehensive 

school reform. The case studies follow a formal protocol that calls for integrating information 

from documents, interviews, focus groups, observations, and quantitative data (see Appendix B).

STUDY LIMITATIONS

The multimethod approach to this evaluation, like most studies of practice (Lipsey, 2003), has 

limitations.

All achievement analyses presented in this report rely on standardized school-level achievement 

scores from NLSLSASD, CCSSO, EDEN and state Web sites. The use of school-level outcome 

measures presents methodological and psychometric challenges. School-level scores are 

aggregate measures of individual student performance within a school and, as such, do not 

account for variation in academic performance among students within a group. In addition, such 

analyses are not sensitive to the fact that performance may be contingent on both individual 

factors and factors related to group membership such as membership in different classrooms and 

school environments. The multileveled structure of data is not adequately addressed in aggregate 

analyses, which limits the precision of the estimates produced. Three major areas have been 

widely acknowledged as contributing to imprecision in aggregate analyses: aggregation bias, 

misestimated standard errors, and heterogeneity of regression (Burstein, 1978; Burstein & Miller, 

1981; Haney, 1980; Raudenbush & Bryk, 2002):

1. Aggregation bias exists because variables at different organizational levels have different 

contextual meanings and may show varying relationships across organizational levels. 

Analyses based on aggregate outcomes do not account for differences in variables across 

organizational levels, leading to biased estimates of effects. Raudenbush and Bryk (2002) 

use socioeconomic status to explicate aggregation bias. They note that both individual 
Chapter II 15

social class and socioeconomic status of the school influence student achievement. 

School-level socioeconomic measures may influence achievement beyond the effects 

estimated at the individual level. 

2. Misestimation of standard errors occurs in single-level analyses because such analyses 

ignore the similarities based on common experiences among responses from individuals 

within a group. For example, all students in a classroom are likely to be exposed to the 

same curriculum. Multilevel analyses calculate standard errors in a way that accounts for 

the clustering of individuals and adjusts for the dependence of responses from a particular 

group. 

3. Heterogeneity of regression is an issue because, despite the common experiences, 

differences exist within all groups. Consequently, the relationships between individual 

characteristics and outcomes may vary among individuals. This heterogeneity is masked 

in school-level analyses that assume a single linear relationship between characteristics 

and outcomes for all individuals within a group. In contrast, multilevel models provide a 

mechanism to explore the heterogeneity of relationships and potential moderating factors 

that may account for differences in the relationships between outcomes and explanatory 

variables. For example, the amount of school resources or level of teacher professional 

development in schools may influence the relationship between CSR and student 

achievement.

Despite these weaknesses, designating the school as the unit of analysis and using aggregate 

outcome measures is deemed appropriate given the goals of the evaluation and the nature of the 

CSR program. The analyses rely on aggregate outcome measures that are both meaningful to 

schools and relevant to the current policy environment. School-level proficiency is the 

performance measure for which all schools are held accountable. Although the use of aggregate 

measures may compromise precision, its relevance for school officials and policy-makers offsets 

its limitations. Moreover, the CSR program was designed to stimulate comprehensive change 

through a set of coordinated reform actions that influence school operations across the whole 

school. Thus, it is appropriate to use the school as the unit of analysis and school-level 

achievement measures to examine how the CSR program influences achievement. 

Furthermore, although a multilevel model, in which students are nested within schools, would 

provide more precise estimates of the relationships between CSR participation and achievement 

outcomes than the models used in this evaluation, collecting individual student records over an 

extended time period from a large nationally representative sample of CSR and comparison 

schools places a heavy burden on schools. 

A second limitation of this study is shared with other efforts to assess program effectiveness 

across states over time. Such evaluations must overcome the difficulty of using the existing state 

assessments, which are designed to provide information about students’ progress toward 

mastering the content established in each state’s standards. Consequently, assessments differ. In 

fact, neither the content nor the criteria for determining proficiency are the same from state to 

state. Also, standards, assessments, and proficiency criteria often change, making scores within 

states difficult to compare over time. Finally, because of the structure of state assessments 
Chapter II 16

(especially before NCLB), we were unable to track cohorts of students because such data were 

rare before 2004–05. School-level assessment scores were standardized within states for each 

year to analyze outcomes across states and assessment instruments.18 The standardization 

provides a common metric for all achievement outcomes; however, it does not account for all 

sources of heterogeneity among states resulting from differences in assessments, the stringency 

of state proficiency standards, and the fact that the analyses rely on different groups of students 

in each year. A more precise analytic approach would use a multilevel model of schools nested 

within states to account for the variation across states. Insufficient within-state samples, 

stemming from both missing data and the manner in which 2002 CSR grants were awarded 

across states, precluded the use of a multilevel model to explore the relationships between CSR 

participation and student academic achievement.

Finally, this study may also be limited by nonrandom attrition of a large number of schools from 

some analyses because of survey nonresponse, the fact that several states did not report all data 

to the Common Core of Data, and the removal of non-CSR schools that received CSR awards 

after 2002. The unrepresentative nature of missing data in the Year 5 sample, and the 

inadvisability of imputing such data from Year 3 responses, were factors precluding analyses in 

this report on the relationship between implementation and achievement. Thus, the failure to find 

a program effect of CSR participation may be a result of limitations in the design of this study as 

well as the lack of a relationship between the CSR program, its implementation, and achievement 

outcomes.

 

18. See Exhibit 2 on pages 9–16 in the Third Year Report (U.S. Department of Education, 2008) for a list of 

the measures used for each state for the years 1999–2000 to 2004–05 of this study. This report updates those 

data with percent proficient in 2006–07; they were the only outcome measures available.
Chapter III 17

III. OVERALL RELATIONSHIP BETWEEN CSR AWARD AND 

ACHIEVEMENT

METHODOLOGY USED TO ASSESS THE RELATIONSHIP BETWEEN RECEIPT OF A CSR

AWARD AND ACHIEVEMENT

Analyses of the relationship between CSR award and growth in achievement were restricted to 

Title I schools that received CSR program funding. The evaluation team selected Title I schools 

using demographic information from the Common Core of Data, the U.S. Department of 

Education’s national database of school demographics, and the standardized achievement scores 

developed for this evaluation. Selection of the comparison group was based on data from 2001–

02, the year before the CSR schools began implementing their awards.

States targeted CSR awards to low-performing Title I schools, resulting in 96 percent of CSR 

awards going to such schools. Therefore, analyzing Title I schools allows for inference regarding 

the population of greatest interest. However, Title I encompasses a wide range of schools; in 

2003–04, more than 54,000 schools received Title I assistance as either a Schoolwide Title I or 

Targeted Assistance School (U.S. Department of Education, 2007). Given the emphasis most 

states placed on making CSR awards to the lowest performing and highest poverty schools, it 

comes as no surprise that the population of all Title I non-CSR schools is higher performing, is 

smaller, and contains lower percentages of students from traditionally underserved minority 

groups and those eligible for free and reduced-price lunch than Title I CSR schools (Exhibit 5). 

Key Finding

Was receipt of a CSR award associated with improvements in school-level 

mathematics and reading achievement?

 Five years after initially receiving their CSR awards, schools 

receiving awards did not demonstrate larger achievement growth than 

matched comparison schools not receiving CSR grants. In fact, 

achievement gains were nonexistent in CSR elementary schools and 

were lower than their comparison schools at the middle school level.
Chapter III 18

Exhibit 5

Differences Between the 2002 Cohort of Title I CSR Schools and Title I Non-CSR 

Schools in Achievement, School Size, and Demographics in 2001–02 

Non-CSR CSR

N Average N Average Difference

Standardized Assessment Scores

Elementary Mathematics (SD) 31,960 -0.20 273 -0.87 0.67**

Elementary Reading (SD) 32,117 -0.22 274 -0.90 0.68**

Middle School Mathematics (SD) 12,044 -0.16 105 -0.94 0.78**

Middle School Reading (SD) 12,222 -0.17 105 -0.95 0.78**

Demographic and School Characteristics

Membership (N) 53,758 479 402 544 -65**

Minority (%) 52,016 35.2 386 54.5 -19.3**

FRL (%) 48,872 51.0 362 70.5 -19.5**

Exhibit highlights: Achievement in Title I Comprehensive School Reform (CSR) schools in the 

2001–02 school year was approximately three-fourths of a standard deviation lower than in 

Title I non-CSR schools. The percentage of minority students in Title I CSR schools was about 

19 percent higher, and the percentage of students eligible for free and reduced-price lunch was 

about 21 percent higher than in Title I non-CSR schools.

Source: Common Core of Data (CCD).

Note: ** p < .01. “FRL” refers to students eligible for free or reduced-price lunches. All data 

are for the 2001–02 school year to demonstrate baseline achievement and demographics. These 

data are restricted to those schools for which achievement data were available in both 2002 and 

2007 (see Exhibit 2 for a list of these states). Standard errors are included in Exhibit A.1.

Because of this disparity, the evaluation team developed a comparison group of non-CSR 

schools for the analysis of the relationship between receiving a CSR award and achievement, 

using Mahalanobis propensity scoring (Rosenbaum & Rubin, 1983, 1985; Rubin, 1980) based on 

mathematics and reading achievement, percentage of free and reduced-price lunch, and 

percentage of minority.19 In this instance, CSR schools are counted as treated schools and nonCSR schools are counted as untreated schools. For each CSR school, Mahalanobis propensity 

scoring chooses a comparison school from the pool of untreated schools that most closely 

matches the CSR school based on the achievement and demographic measures used.20

Because some states did not report percentage of free and reduced-price lunch, and because 

sometimes CSR schools were located in small districts, the propensity score matching was run 

several times, first starting with the most restrictive matching procedure and then gradually 

 

19. Note that the comparison group developed for the analysis of the relationship between receiving an award 

and achievement gains is different from the comparison group developed at the outset of this study. The 

comparison group used in this section is based only on Title I schools; furthermore, it relies on updated CSR 

award, Common Core of Data, and NLSLSASD data files not available at the outset of this study.

20. The Mahalanobis propensity score matching was completed using the -mahapick- module in Stata (Kantor, 

2006).
Chapter III 19

relaxing the matching conditions. All matches are based on, at the minimum, mathematics and 

reading achievement and percentage of minority. Matches were restricted to being located in the 

same state and having the same school configuration (elementary, K–8, or 6–8 middle school) as 

the CSR school. The conditions that were used are listed from most restrictive to least restrictive. 

The comparison school was chosen from the most restrictive set of conditions where a successful 

comparison school was found.

1. The first set of matches is based on percentage of free and reduced-price lunch and being 

located in the same school district. Comparison schools selected in this group most 

closely matched their CSR school on mathematics and reading achievement, percentage 

of minority, and percentage of free and reduced-price lunch; were located in the same 

school district as the CSR school; and had the same school configuration as the CSR 

school.

2. The second set of matches removes free and reduced-price lunch from the restrictions 

used for the first set.

3. The third set of matches is similar to the first, but instead of restricting the matching to 

the same district, restricts the matching to the same type of locality (e.g., large city, small 

city, suburban, or rural) in the state. This approach expands the pool of potential matches 

for schools in small school districts that may not have many potential matches. 

4. The fourth set of matches removes free and reduced-price lunch from the third set of 

matches.

5. The fifth set of matches includes free and reduced-price lunch and removes any locality 

restriction (choosing any school in the state).

6. The last set of matches removes free and reduced-price lunch from the restrictions used 

for the fifth set of matches.

The next two exhibits provide data for CSR schools and their comparison schools selected by the 

propensity-score matching process described above.21 Exhibit 6 presents average school-level 

achievement scores and demographics for schools with elementary mathematics and reading 

scores, while Exhibit 7 presents these measures for schools with middle school mathematics and 

reading scores. The CSR and comparison schools are very closely matched on baseline 

achievement and demographics.

 

21. Note that this group of schools chosen for the achievement analysis is different from the group of schools 

chosen at the outset of the study to document implementation (see pages 12–16). The group of schools chosen 

for the achievement analysis encompasses all Title I CSR schools and matched comparison Title I schools; the 

selection process was done toward the end of this study and is based on the most recent data available for CSR 

schools, which had been updated as recently as spring 2007. The group of CSR schools and their matched 

comparison schools chosen at the outset of this study to document implementation is based on data that were 

available only at the beginning of this study; furthermore, it is a sample drawn from the population of CSR 

schools.
Chapter III 20

Exhibit 6

Differences Between the 2002 Cohort of Title I CSR Schools and Non-CSR Title I 

Schools Chosen by Propensity Scoring: Schools With Elementary Mathematics and 

Reading Achievement

Non-CSR CSR

N Average N Average Difference

Standardized Assessment Scores

Elementary Mathematics 478 -0.79 234 -0.77 -0.02

Elementary Reading 478 -0.77 235 -0.81 0.04

Demographic and School Characteristics

Membership (N) 478 530 244 497 33

Minority (%) 478 56.2 244 53.1 3.1

FRL (%) 461 68.8 233 71.1 -2.3

Exhibit highlights: There are no statistically significant differences between matched 

Comprehensive School Reform (CSR) and non-CSR Title I schools in the baseline (2001–02) 

school year.

Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD).

Note: “FRL” refers to students eligible for free or reduced-price lunches. These data are 

restricted to those schools for which achievement data were available in both 2002 and 2007 

(see Exhibit 2 for a list of these states). Standard errors are included in Exhibit A.2.
Chapter III 21

Exhibit 7

Differences Between the 2002 Cohort of Title I CSR Schools and Non-CSR Title I 

Schools Chosen by Propensity Scoring: Schools with Middle School Mathematics and 

Reading Achievement

Non-CSR CSR

N Average N Average Difference

Standardized Assessment Scores

Middle School Mathematics 248 -0.66 95 -0.74 0.08

Middle School Reading 248 -0.63 95 -0.63 0.06

Demographic and School Characteristics

Membership (N) 248 675 109 615 59

Minority(%) 248 55.9 109 51.1 4.8

FRL (%) 243 65.1 103 65.3 -0.2

Exhibit highlights: There are no statistically significant differences between matched 

Comprehensive School Reform (CSR) and non-CSR Title I schools in the baseline (2001–02) 

school year.

Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD).

Note: “FRL” refers to students eligible for free or reduced-price lunches. These data are 

restricted to those schools for which achievement data were available in both 2002 and 2007 

(see Exhibit 2 for a list of these states). Standard errors are included in Exhibit A.3.

After the selection of the comparison schools, paired t-tests were used to assess whether the 

change in achievement from 2002–03 to 2004–05 is different between CSR schools and their 

comparisons (Zhao, Li, Gao, & Tierney, 2001; Fraas, Newman, & Pool, 2007).22

FINDINGS

Was receipt of a CSR award associated with improvements in school-level mathematics 

and reading achievement?

 Five years after initially receiving their CSR awards, schools receiving awards did 

not demonstrate larger achievement growth than matched comparison schools not 

receiving CSR grants. In fact, achievement gains were nonexistent in CSR 

elementary schools, were marginally lower than their comparison schools in 

middle school mathematics, and were no different from their comparison schools 

in middle school reading.

At the elementary school level, changes in mathematics and reading achievement are not 

statistically significant for both the CSR and matched non-CSR schools (Exhibit 8). At the 

 

22. T-tests are used to test the null hypothesis that the means of two groups (in this instance, CSR and 

comparison schools) are the same. A t-statistic is calculated from the two groups’ means and standard 

deviations, which is then used to determine the probability of correctly rejecting the null hypothesis of no 

relationship by comparing it with students’ t-distribution.
Chapter III 22

middle school level, changes in mathematics achievement are statistically significant for CSR 

schools but not for non-CSR schools. Differences in achievement change between CSR and nonCSR schools are statistically significant for mathematics achievement, although only at the .10 

level, while differences in changes in reading achievement are statistically insignificant. Thus, 

after five years, schools receiving three-year CSR grants in 2002 were no better at improving 

achievement than Title I schools that were similar in demographics and achievement in the 

baseline year. 

Exhibit 8

Changes in Standardized Assessment Scores in CSR and Matched Non-CSR Schools 

from 2002–03 to 2006–07

Changes in Standardized Assessment Scores 

(SD)

N CSR Schools

Non-CSR 

Schools Difference

Elementary Mathematics 493 0.04 0.00 0.04

Elementary Reading 563 0.02 0.04 -0.02

Middle School Mathematics 291 0.01* 0.11** -0.10+

Middle School Reading 294 0.04 0.10* -0.06

Exhibit highlights: Community School Reform (CSR) schools had statistically significant 

increases between 2002–03 and 2006–07 in middle school mathematics of 0.01 standard 

deviations. However, matched non-CSR schools had larger statistically significant increases in 

middle school mathematics and reading of 0.11 and 0.10 standard deviations, respectively, 

although only middle school mathematics had a marginally statistically significant difference 

between CSR schools and their matched non-CSR schools. 

Source: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD); Education Data Exchange Network 

(EDEN); Council of Chief State School Officers (CCSSO); and state Web sites.

Note: + p < .10; * p < .05; ** p < .01. Tests for the statistical significance of achievement gains for 

CSR and non-CSR schools are t-tests to assess whether the value is different from zero. The 

differences between CSR and non-CSR schools are assessed through paired t-tests. Standard errors 

are included in Exhibit A.4.

DISCUSSION

Unlike these results, earlier CSR research shows an increase in mathematics and reading 

achievement over multiple years of implementation (Borman, Hewes, & Overman, 2003). Two 

possible reasons for the lack of findings here are suggested by the initial case study of 15 pairs of 

schools. First, a number of schools ended implementation of an original CSR strategy, replacing 

it with a portfolio of programs. Second, some schools added reform programs, creating 

competition for teachers’ time for professional development. 

School W, for example, chose the Comer School Development model after award receipt in 

2002. However, a new principal arrived in 2003 and instead chose the Effective Schools model. 

Teachers indicated they were not involved in this decision and had already scheduled 
Chapter III 23

professional development for the original model for the summer. Staff members then engaged in 

what they termed a “whirlwind” of professional development on the Effective Schools model. 

However, key consultants for the Effective Schools model became ill, compounding staff 

indifference to the program. Ultimately, neither the Effective Schools model nor the Comer 

School Development model was implemented. 

Another school, School E, was encouraged by the district’s then superintendent to adopt and 

implement Accelerated Schools. The external assistance provided by the model developer helped 

the school arrange common planning time for teachers and offered more than 20 days of 

professional development in 2002–03. However, in 2003, district leadership changed, and the 

new superintendent focused efforts on supplementary programs. Teachers were required to 

attend professional development sessions for the district’s priority programs, creating scheduling 

conflicts. By 2004, Accelerated Schools was no longer present in the school.

Thus, implementation (or lack thereof) may explain the overall lack of a relationship between 

receiving a CSR award and achievement gains. The next section provides data results of the 

survey as well as case studies on the implementation of the 11 components and reports on the 

comprehensiveness of CSR implementation, as measured by the number of components a school 

implements.
Chapter III 24
Chapter IV 25

IV. THE COMPREHENSIVENESS OF CSR IMPLEMENTATION

This report includes analyses of changes in the extent to which both CSR and non-CSR schools 

implemented components of the CSR program over a five-year period. This chapter focuses on 

the implementation of CSR components, comparing schools receiving three-year CSR grants in 

2002 with comparison schools that did not receive such grants. It also includes the findings from 

the case studies of CSR and non-CSR schools.

MEASURING IMPLEMENTATION OF THE CSR COMPONENTS

The analyses of the CSR components rely on measures of the CSR components developed from 

the surveys administered in 2003, 2005 and 2007 to both CSR and non-CSR schools. The 

surveys ask principals and teachers about the implementation of activities consistent with the 11 

components identified in NCLB. For most questions, respondents were presented with a forced 

choice of “0” for not having implemented the activity and “1” for implementing the activity.23

Within each component, respondents’ answers were averaged to develop a component score for 

each school.

The component scores were converted into measures of whether a school is counted as 

implementing a component. That is, schools were assigned a 0 if their component score or their 

responses to survey questions fell below a certain threshold and a 1 if they exceeded the 

 

23. This evaluation measures implementation differently from Aladjem et al. (2006), who measured 

implementation as the difference between what the model developers consider to be full implementation and 

what the school actually does. To do so, they surveyed each of the model developers and asked them to 

respond as if they were a school that implemented their model. The researchers compared each school’s 

responses to this measure of implementation to develop a measure of fidelity. To measure implementation in 

the comparison schools, Aladjem et al. predicted what CSR model the comparison school would have chosen 

based on its school characteristics and those of the CSR schools. The researchers then compared the school’s 

responses with those of the model developer of the predicted CSR model. This measure of implementation is 

consistent with the focus on models as compared with this study’s focus on fidelity with the 11 components 

identified in NCLB.

Key Finding

Were schools that received CSR awards more likely to implement the 

legislatively specified components of CSR than other schools?

 No, consistent with earlier study findings, both CSR and non-CSR 

schools implemented a similar number of legislative components in 

both 2003 and 2007. The number of components rose slightly for 

CSR and non-CSR schools during this period at both the elementary 

and middle school levels.
Chapter IV 26

threshold. For most of the components, the school had to report that it was implementing all of 

the activities to be counted as implementing the component. In some instances, however, this 

requirement would have resulted in no schools implementing the component, so the threshold 

was relaxed somewhat. Also, for parental involvement, the measures used were not dichotomies 

but, instead, were estimates of the percentage of parents engaging in various activities in the 

school; another threshold was developed based on the distribution of the estimates. The metric 

used to measure whether a school implemented a component is included in the discussions of 

how each component is measured. This conversion process was done to measure whether a 

school implemented a component and was used to calculate the number of components 

implemented.

The surveys were modified between the three administrations as a result of findings from the 

2003 survey concerning the existence of multiple reforms in schools, the need to develop more 

refined measures of professional development, and the fact that some of the questions in the 

2003 survey were no longer applicable in 2005 and 2007. For instance, in the 2003 survey, 

schools often reported implementing more than one reform method. The 2005 survey was 

modified to capture information on up to four reforms the school was undertaking (see Appendix 

B). As a result of these changes, some of the components were not measured consistently across 

the two time periods; these modifications also required changes in assessing whether a 

component was implemented. All such changes are documented below in the discussion of each 

component. As a result, comparisons on some components across time were limited or 

unavailable. 

In addition to the survey results, the results of the case studies are integrated in the sections on 

components below. Two different sets of case studies were conducted during this longitudinal 

study. The initial case studies consisted of 15 pairs of schools from 15 districts across 14 states—

with each pair containing a CSR school and a non-CSR school—and were selected to 

supplement the quantitative analyses with qualitative understanding of implementation 

dynamics. With such a small number of case study schools, this sample does not represent either 

the geographic distribution or the distribution of school levels in the universe of CSR schools. 

However, the selection ensured that the case study sample includes enough geographic and 

school-level diversity to reflect a variety of state policy environments, including schools in states 

with a long tradition of local control and others in states that exercise more centralized control 

over schools. Site visitors paid three visits to these schools, in 2003, 2005 and 2007.

The COSMOS Corporation conducted an additional round of data collection beyond the original 

30 case study schools. This round of case studies, known as the Improving Schools Study, 

involved 13 schools identified by WestEd. Of this group, 9 were considered “improving 

schools,” and the other 4 were “comparison schools.” Ten of the schools in this sample were new 

to the case study analysis, while 3 were from the original group of 30 case study schools. 

Furthermore, 7 of the improving schools were CSR schools, and 2 were non-CSR schools. One 

of the comparison schools was a CSR school, and 3 were not. The 13 schools in the Improving 

Schools Study were visited in spring through fall 2007. Site visitors relied on the same site visit 

protocols as were used in the original 15 case study sites during the 2004–05 site visits.
Chapter IV 27

Components 1 and 11—Research-Based Design and Evidence-Based Practice

Given the substantial similarity between Research-Based Design and Evidence-Based Practice

components, the decision was made to combine them into one measure. As defined in NCLB, 

evidence-based practice focuses on proven methods and strategies for student learning, teaching 

and school management that reflect scientifically based research and effective practices 

replicated successfully in schools with diverse characteristics. Added in NCLB, the researchbased design component calls for the adoption of programs that use scientifically based research 

to document either significant improvement in the academic achievement of students 

participating in such programs as compared with students in schools that have not participated in 

such programs or the existence of strong evidence that such programs will significantly improve 

the academic achievement of participating children.

The 2003 measure consists of a principal’s responses to three questions, each of which was a yes 

or no question on the survey: whether there was evidence based on independent research 

supporting the model (33 percent indicating yes), whether there was evidence based on research 

conducted by the reform developer (34 percent indicating yes), and whether there was evidence 

that relied on the use of control or comparison groups (21 percent indicating yes). The 2003 

measure was calculated as the average of these three items (Exhibit 9). A school was counted as 

implementing this component if it reported that at least two of the three types of evidence were 

present.

The 2005 measure consists of similar items as the 2003 measure: whether the reform model has 

evidence that it improves student achievement (89 percent indicating yes), relies on a comparison 

or control group (60 percent indicating yes), and was shown to improve student achievement at 

other schools (91 percent indicating yes). Each of these items was also a yes or no question. 

Nonmissing responses were totaled, and the school was counted as implementing this component 

if the average of the nonmissing responses was 1. Thus, if a school reported on only two of the 

measures and reported that it implemented both of them, then that school was counted as 

implementing a research-based design. 

The 2007 measure relies on principals’ identification of three sources of evidence that contribute 

to the selection of reform. Fifty-nine percent of principals indicated that the theoretical or 

research foundation of the reform model was important in its selection; 55 percent stated that 

successful implementation at other schools was important; and 21 percent indicated that the best 

published results were important. The measure for implementation was constructed in the same 

manner as in previous years.
Chapter IV 28

Exhibit 9

Means and Standard Deviations of Survey Items Used to Construct Research-Based 

Design and Evidence-Based Practice Measure

Year Evidence Items N Mean

Standard 

Deviation

Independent research 716 0.33 0.47

Research by reform designer 716 0.34 0.47

2003

Comparison or control groups 716 0.21 0.41

Improves achievement 654 0.89 0.28

Comparison or control groups 632 0.60 0.45

2005

Student achievement at other schools 628 0.91 0.26

Theoretical or research foundation 536 0.59 0.49

Successful implementation at other schools 536 0.55 0.50

2007

Best published results 536 0.21 0.41

Exhibit highlights: The survey items used to construct the research-based design and evidencebased practice measure changed from year to year. The reader may compute the means into a 

percentage by multiplying the mean value by 100. For example, the 2003 mean score of 0.21 for 

comparison or control groups indicates that 21 percent of respondents indicated that the reform 

method they chose had evidence based on comparison or control groups.

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Case Study Findings

Nearly all of the 15 matched-pair original case study schools reported the use of scientifically 

based research. In three instances, there was no information or evidence of use in CSR schools, 

and in another CSR school, the reform effort had been dropped. Two of the comparison schools 

were reported to be having “issues with implementation” of scientifically based research to 

support reform efforts.

Component 2: Comprehensive Planning

Questions on the survey forms measure two aspects of comprehensive planning—classroombased planning and school-based planning. Surveys asked respondents about both aspects 

consistently in all survey years. On all questions, respondents were asked whether their school 

improvement plans contained nine components of planning; they could answer yes or no to each. 

The classroom-based measures include curriculum and instruction, student assessment, 

classroom management, and professional development. Results show that there was an increase 

of schools identifying the items in their school improvement plans between 2003 and 2005, 

followed by a decrease in the number of schools in each item in 2007 (Exhibit 10). The schoolbased planning measure includes measurable goals for reform, periodic evaluation, parental 

involvement, professional development, participation in school management, and integration of 

new technology. As with the previous exhibit, the same trend of reporting appears with an 
Chapter IV 29

increase in items between 2003 and 2005 followed by a decrease in 2007 (Exhibit 11). Note that 

professional development occurs in both school- and classroom-level planning because 

professional development opportunities may be around subject-specific topics or school-reform 

related topics. In all years, a school was counted as implementing these components if it reported 

that the school improvement plan contained all of the items. 

Exhibit 10

Means and Standard Deviations of Survey Items Used to Construct Comprehensive 

Planning-Classroom Measure, by Year

Aspects of Reform 2003 2005 2007

Covered by School 

Improvement Plan Mean

Standard 

Deviation Mean

Standard 

Deviation Mean

Standard 

Deviation

Curriculum and instruction 0.88 0.32 0.95 0.22 0.81 0.39

Student assessment 0.50 0.50 0.64 0.48 0.36 0.48

Classroom management 0.43 0.49 0.57 0.49 0.31 0.46

Professional development 0.93 0.26 0.96 0.20 0.87 0.34

Exhibit highlights: The survey items used to construct the comprehensive planning-classroom 

measure were consistent across years. The reader may compute the means into a percentage by 

multiplying the mean value by 100. For example, the 2003 mean curriculum and instruction 

score of 0.88 indicates that 88 percent of respondents indicated that the school improvement 

plan covered curriculum and instruction.

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Note: There are 720 observations in 2003, 673 observations in 2005, and 536 observations in 

2007.
Chapter IV 30

Exhibit 11

Means and Standard Deviations of Survey Items Used to Construct Comprehensive 

Planning-School Measure, by Year

Aspects of Reform 2003 2005 2007

Covered by School 

Improvement Plan Mean

Standard 

Deviation Mean

Standard 

Deviation Mean

Standard 

Deviation

Measurable goals 0.97 0.16 0.98 0.15 0.95 0.22

Periodic evaluation 0.84 0.37 0.91 0.28 0.80 0.40

Parental involvement 0.80 0.40 0.85 0.36 0.79 0.41

Professional development 0.93 0.27 0.96 0.20 0.87 0.34

Participation in school 

management 0.53 0.50 0.63 0.48 0.41 0.49

New technology 0.72 0.45 0.78 0.42 0.62 0.49

Exhibit highlights: The survey items used to construct the comprehensive planning-school 

measure were consistent across years. The reader may compute the means into a percentage by 

multiplying the mean value by 100. For example, the 2003 mean parental involvement score of 

0.80 indicates that 80 percent of respondents indicated that their school improvement plan 

covered parental involvement. 

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Note: There are 720 observations in 2003, 673 observations in 2005, and 536 observations in 

2007.

Case Study Findings

In spite of the fact that coherent strategies imply planned and systematic approaches, there was 

little documented evidence in any of the original 15 case study sites that such comprehensive 

planning was used to address the reform strategies. Planning was primarily focused on 

instruction and use of assessment data. While data were widely used, it was not clear how these 

data were used to implement or modify the CSR program or other initiatives. Some schools 

found what formative evaluations were done to be helpful, but no explanations were offered as to 

how these evaluations resulted in plan modifications or adjustments.

The 13 improving schools case study sites used complementary research-based supporting 

strategies that seemed to indicate a well-planned approach to reform. For example, nearly half of 

the schools used differentiated learning as a supporting strategy for the primary approach to 

literacy instruction. In another example, 9 of 13 schools implemented whole-school literacy 

reform across all grades, suggesting a coherent approach.

Component 3: Professional Development

The professional development component relies on the reported number of days of professional 

development; however, it was measured differently across survey years and any changes should 

be interpreted cautiously. On the 2003 survey, respondents were asked whether their school 
Chapter IV 31

provided at least 10 days of professional development. Principals could respond yes or no, and 

the school was counted as fully implementing the measure if they responded yes. On the 2005 

and 2007 surveys, teachers were asked to report the number of hours of professional 

development they received in the last year. For analysis and reporting, the teacher’s response was 

converted to days by taking the number of hours and dividing by six (approximately the length of 

an average school day). Schools were counted as implementing professional development if the 

average number of days of professional development received was 10 or more. The result of the 

survey shows that in 2005, 65 percent of schools provided 10 or more days of professional 

development as reported by teachers, which was an increase from 2003 results provided by the 

principals (48 percent). There was a decrease in the number of schools that provided 10 or more 

professional development days, as reported by teachers, in 2007 compared with 2005 (Exhibit 

12).

Exhibit 12

Means and Standard Deviations of the Survey Item Used to Construct the Professional 

Development Measure, by Year

2003 2005 2007

Professional 

Development 

Item N Mean

Standard 

Deviation N Mean

Standard 

Deviation N Mean

Standard 

Deviation

Professional 

Development 695 0.48 0.50 714 0.65 0.48 686 0.53 0.50

Exhibit highlights: The reader may compute the means into a percentage by multiplying the 

mean value by 100. For example, the 2003 mean professional development score of 0.48 

indicates that 48 percent of respondents indicated that the school offered at least 10 days of 

professional development to teachers. 

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Case Study Findings

Within the case study schools, two different findings emerged. First, while documentation of 

professional development hours did not occur for all case study schools, professional 

development activities seemed to be more sustained at some schools. At one school for example, 

a database was used to document teacher professional development activities that were provided 

to each staff member and how these activities were linked to both the school and individual 

professional development plans. Stipends were available to teachers who exceeded 24 hours of 

professional development. Teachers averaged between 24 and 34 hours per year in Year 2 of the 

study and 20 to 40 hours in Year 3. 

Second, all but 2 of the 13 schools in the improving schools case study used literacy coaches to 

support teachers in implementing specific strategies. Numbers of coaches per school varied from 

one or two who focused on both mathematics and literacy to one high-performing school with 

one coach and five reading specialists. Coaches not only provided professional development but 

also assisted teachers one-on-one, modeled lessons within classrooms, and often led grade-level 
Chapter IV 32

and vertical planning sessions. Coaches were also responsible for keeping track of student 

performance. One school reported providing more than 1,700 hours of coaching support to 

teachers throughout the school year. 

Component 4: Goals and Benchmarks

The goals and benchmark measures rely on a somewhat different set of items across survey years 

and any change should be interpreted cautiously. Four questions from the 2003 surveys were 

used to construct the measure for goals and benchmarks (see Exhibit 13): whether the school 

improvement plan includes student assessment rubrics (50 percent indicating yes), whether the 

school improvement plan includes measurable goals and objectives (97 percent), whether the 

school has end-of-year student achievement goals (88 percent), and whether the school has 

interim student achievement goals (70 percent). Nonmissing responses were totaled, and the 

school was counted as implementing this component if the average of the nonmissing responses 

was 1. Thus, if a school reported on two of the measures and indicated those were in place, the 

school was counted as implementing goals and benchmarks. 

The 2005 goals and benchmark measure relies on the same two school improvement plan items 

as the 2003 measure, that is, student assessment rubric (64 percent) and measurable goals and 

objectives (98 percent). However, the two questions about student goals were not asked in the 

2005 surveys. Instead, respondents were asked whether the reform strategies were accompanied 

by implementation benchmarks (75 percent indicating yes). One of the earlier findings in this 

study is that schools were often implementing more than one reform strategy simultaneously. As 

a result, reform-specific questions were asked four times for each respondent. In some cases, the 

respondent indicated that there was only one reform going on in the school; in other cases, 

respondents listed up to four reform strategies. The responses were averaged to derive a schoollevel measure for this item that ranged between 0 and 1. The 2005 goals and benchmarks 

measure was created by averaging the nonmissing responses; schools with a score of 1 were 

counted as implementing goals and benchmarks.

The 2007 goals and benchmark measure relies on the same two school improvement plan items 

as the measures from previous years, that is, student assessment rubric (36 percent) and 

measurable goals and objectives (95 percent). In addition, the 2007 measure includes an item on 

whether the school improvement plan included a mechanism for periodic evaluation of goals (80 

percent indicating yes). The 2007 goals and benchmarks measure was created by averaging the 

nonmissing responses; schools with a score of 1 were counted as implementing goals and 

benchmarks.
Chapter IV 33

Exhibit 13

Means and Standard Deviations of Survey Items Used to Construct the Goals and 

Benchmarks Measure, by Year

Year Goals and Benchmark Items N Mean

Standard 

Deviation

Does the SIP include student assessment rubrics? 720 0.50 0.50

Does the SIP include measurable goals or objectives? 720 0.97 0.16

Does the school have student goals at the end of the 

year? 526 0.88 0.33

2003

Does the school have intermediate student goals? 571 0.70 0.46

Does the SIP include student assessment rubrics? 673 0.64 0.48

Does the SIP include measurable goals or objectives? 673 0.98 0.15

2005

Strategies accompanied by implementation 

benchmarks? 651 0.75 0.38

Does the SIP include student assessment rubrics? 536 0.36 0.48

Does the SIP include measurable goals or objectives? 536 0.95 0.22

2007

Does the SIP include periodic evaluation of goals? 536 0.80 0.48

Exhibit highlights: The goals and benchmarks measures rely on a somewhat different set of 

items across survey years and any change should be interpreted cautiously. The reader may 

compute the means into a percentage by multiplying the mean value by 100. For example, the 

2003 mean score of 0.50 for whether the school improvement plan includes student assessment 

rubrics indicates that 50 percent of respondents indicated that the school improvement plan 

includes student assessment rubrics.

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Note: “SIP” refers to school improvement plan.

Case Study Findings

The use of data-driven decision making was evident in case study schools. However, in almost 

all cases, use of data for the specific objective of benchmarking and goal setting was not clearly 

identified. Case study schools approached this task in a variety of ways, including test analysis 

by grade level, student performance across grades (involving vertical planning), use of data 

coaches, and use of technology to support and facilitate analysis. Emphasis on the use of data 

seemed to be a function of the degree of district focus and support. In instances where use of data 

was the strongest, usually part of weekly team or planning meetings, similar findings were noted 

for both CSR schools and their matched comparison schools. In one instance, use of data was 

cited as part of the “school culture” for a CSR school and its comparison school. 

Use of data for instructional decision making was evident in case study schools. Use of 

assessments other than those mandated by the state or district or use of formative assessment 

strategies was less evident. In two instances both CSR schools and their comparison schools 

were noted for using a wide array of assessments. For one pair of schools, these assessments 

included state results from criterion-referenced and norm-referenced tests as well as a phonics 
Chapter IV 34

assessment, reading assessments for all grades, district assessment data for each nine-week 

period, and alternative assessments for special education students. Only one school was reported 

to have used student work samples as part of its ongoing analysis of data. 

Case study schools were more likely to use data to assess and modify instruction than to evaluate 

school initiatives and programs. In one example, a school used results from the Iowa Tests of 

Basic Skills not only to examine performance of subgroups of students that missed AYP targets 

but also to determine what specific professional development was needed to address the needs of 

these students. Another school combined results from a state-provided “gap analysis” that 

identified misalignment between students’ scores and state standards, benchmarks from previous 

school improvement plans, and information from the formative evaluation process to revise the 

comprehensive school improvement plan.

In the case study sites, training on data use was more likely to be provided by the district than by 

the state. In one district, principals received monthly training on how to make data-driven 

decisions and were expected to share training with their schools’ staff members. The district also 

organized a course for teachers at a local college on using data to inform decisions. 

As noted in Component 3 results from the improving schools case study, coaches tended to be 

the data keepers and provided the faculty with performance results by student and by classroom 

that could then be used to inform instruction. Most coaches kept data books with detailed 

performance findings. At two schools, coaches prepared “data rooms” where teachers met and 

planned lesson content. At one of these schools, the library wall had a color-coded card for each 

student with the student’s performance on each assessment as well as the prescribed intervention. 

Coaches updated the wall after each assessment, and during one site visit, clusters of teachers 

were observed at the wall discussing overall classroom and individual student performance.

Component 5: Staff Participation

The 2003 and 2005 measures for staff participation rely on almost the same survey items; the 

2003 measure includes one additional item that was not asked on the 2005 survey (Exhibit 14). 

Both measures rely on yes or no questions about all teachers participating in reform and about 

factors that limit participation (i.e., funding and subject-specific reform). The 2003 measure 

includes estimation by the principal of the percent of teachers who participate in reform. The 

2007 measure relies on principal survey items that asked whether administrators and teachers 

participated in a vote to implement the current reform (43 percent and 45 percent, respectively), 

whether there were no limitations on teacher participation in the reform (63 percent), and 

whether the reform has support from staff members (56 percent). The staff participation was an 

aggregate measure, which was developed by averaging all the nonmissing responses within a 

school; schools with a score of 1 were counted as implementing.
Chapter IV 35

Exhibit 14

Means and Standard Deviations of Survey Items Used to Construct Staff Participation 

Measure, by Year

Year Staff Participation Items N Mean

Standard 

Deviation

Percentage of teachers who participate in reform 655 0.88 0.33

All teachers participate in reform (yes or no) 715 0.80 0.40

Participation is not limited by subject-specific reform 715 0.79 0.41

2003

Participation is not limited because of funding 716 0.93 0.25

All teachers participate in reform (yes or no) 674 0.78 0.42

Participation is not limited by subject-specific reform 674 0.90 0.30

2005

Participation is not limited because of funding 674 0.88 0.33

Administrators voted for the current reforms 536 0.43 0.49

Teachers voted for the current reforms 536 0.45 0.50

No limits on teacher participation 536 0.63 0.48

2007

Support from staff 508 0.56 0.50

Exhibit highlights: Survey items used to measure staff participation changed each year of the 

survey. The reader may compute the means into a percentage by multiplying the mean value by 

100. For example, the 2003 mean score of 0.80 for whether all teachers participate in reform 

indicates that 80 percent of respondents indicated that all teachers participate in reform.

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Case Study Findings

The case studies pointed to several examples of staff support for school reform. In one school, 

charts and posters on walls throughout the school indicated participation in, and support of, the 

reform method by both teachers and students. The charts and posters reflected all of the methods 

and strategies being used. At another school, the faculty was initially unanimously behind the 

reform method but then voted to discontinue it after two years. This faculty remained involved in 

implementation of school improvement strategies through faculty meetings, grade-level

planning, and curriculum content meetings. Finally, there were instances where faculty support, 

or lack thereof, was dependent on the principal. For example, one school had strong support for 

the reform method, yet when the program ended and a new principal took over, the new principal 

emphasized an individual approach to teaching in which teachers used their own methods to 

prepare students for the state assessment.

Teachers in the Improving Schools Study were generally supportive of the reforms. In the case of 

K–8 schools, teachers at the lower grades were more supportive than teachers at the upper 

grades. In 7 of the 13 schools in the Improving Schools Study, staff members selected the 

research-based strategy. In one case where the principal selected the strategy, it was no longer in 

use in 2006–07. As noted earlier, coaches played a major role in providing teacher support for 

specific reforms strategies.
Chapter IV 36

Component 6: District Support

For 2003 and 2005, the district support measure is a combination of the same six items 

pertaining to the types of district support: needs assessment, additional staff, selecting a reform 

model, writing grant proposals, professional development, and release time for teachers. Each of 

these items is formatted as a yes or no question. All six items saw an increase in district support 

between 2003 and 2005, except writing grant proposals, which remained relatively consistent 

(Exhibit 15). For 2007, the district support measure is a combination of five different items 

pertaining to types of district support: developing school improvement plan (71 percent), 

reviewing school improvement plan (70 percent), providing data for school improvement plan 

review (81 percent), providing training on school improvement plan development or review (76 

percent), and establishing school goals or benchmarks (63 percent). In each year, the nonmissing 

scores were averaged, and a school with an average score of 0.8 or higher was counted as 

implementing this measure.
Chapter IV 37

Exhibit 15

Means and Standard Deviations of Survey Items Used to Construct the District Support 

Measure, by Year

Year District Support Items N Mean

Standard 

Deviation

Needs assessment 722 0.45 0.50

Additional staff 722 0.41 0.49

Selecting a reform model 722 0.37 0.48

Writing grant proposals 722 0.58 0.49

Professional development 722 0.76 0.43

2003

Release time for teachers 722 0.61 0.49

Needs assessment 671 0.54 0.50

Additional staff 671 0.45 0.50

Selecting a reform model 671 0.43 0.49

Writing grant proposals 671 0.56 0.50

Professional development 671 0.82 0.39

2005

Release time for teachers 671 0.63 0.48

Developing school improvement plan 536 0.71 0.45

Reviewing school improvement plan 536 0.70 0.46

Providing data for school improvement plan review 536 0.81 0.39

Providing training on school improvement plan 

development or review 536 0.76 0.43

2007

Establishing school goals or benchmarks 536 0.63 0.43

Exhibit highlights: Although survey items used to measure district support were the same in 

2003 and 2005, they changed in 2007. The reader may compute the means into a percentage by 

multiplying the mean value by 100. For example, the 2003 mean needs assessment score of 0.45 

indicates that 45 percent of respondents indicated that the district assisted the school in 

conducting a needs assessment. 

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Case Study Findings

The most widespread strategy in case study schools was for the district to provide coaches to 

assist in schools. Peer observations and reviews were the most frequently mentioned use of 

coaches. Coaches were also reported to mentor new teachers in several schools while another 

school specifically noted lesson modeling provided by a coach.

Several schools in the improving schools case study used bonuses as a teacher retention strategy. 

At one school, bonuses based on achievement of growth targets varied from $3,700 to $8,000 per 

teacher. Although the source of these funds was not mentioned, the amount suggests district 

support at the very least. At another school, teachers received extended pay for working in an 

inner-city school. 
Chapter IV 38

Component 7: Parental Involvement

For 2003 and 2005, the parental involvement measures were calculated from the same five items 

(Exhibit 16). Teachers were asked to estimate the percentage of parents who participated in their 

schools through parent-teacher conferences (64 percent and 58 percent for 2003 and 2005, 

respectively), by demanding frequent reports (36 percent and 30 percent, respectively), by 

actively volunteering (16 percent and 12 percent, respectively), by observing classrooms (11 

percent and 10 percent, respectively), and by being active in PTA or PTO organizations (16 

percent and 14 percent, respectively). The 2007 parental involvement measure included four 

different items relating to parental involvement: school improvement plan involved parental 

involvement (79 percent), parents involved in selecting school reform (25 percent), school 

encourages parents to be involved in mission and goals (59 percent), and parents and community 

preferences were influential in classroom teaching (37 percent). In each year, nonmissing 

responses were averaged, and a school was credited as implementing the parental involvement 

measure if the average was equal to or greater than 0.4. 
Chapter IV 39

Exhibit 16

Means and Standard Deviations of Survey Items Used to Construct the Parental 

Involvement Measure, by Year

Year Parental Involvement Items N Mean

Standard 

Deviation

Parent-teacher conferences 706 0.64 0.26

Demanding frequent reports 656 0.36 0.32

Actively volunteering 693 0.16 0.17

Observing classrooms 661 0.11 0.17

2003

Actively participate in PTO 666 0.16 0.20

Parent-teacher conferences 731 0.58 0.28

Demanding frequent reports 732 0.30 0.23

Actively volunteering 729 0.12 0.12

Observing classrooms 725 0.10 0.12

2005

Actively participate in PTO 721 0.14 0.15

School improvement plan covers parental involvement 536 0.79 0.41

Parents involved in selecting school reform 536 0.25 0.43

School encourages parents to be involved in mission 

and goals

536 0.59 0.49

2007

Parent and community preferences are influential in 

classroom teaching

536 0.37 0.48

Exhibit highlights: Although survey items used to measure parental involvement were the same 

in 2003 and 2005, they changed in 2007. The reader may compute the means into a percentage 

by multiplying the mean value by 100. For example, the 2003 mean score of 0.64 indicates that 

respondents estimated that 64 percent of parents were active in their schools through parentteacher conferences. 

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Case Study Findings

Several types of parental involvement were evident from the case studies. Schools reported 

parent sign-off on homework as the most frequently used parent involvement strategy. They also 

reported use of parent resource centers and use of parent liaisons or parent coordinators. Several 

schools reported parent involvement in school advisory councils or committees. Instances of 

parents assisting in classes were reported only in two schools in the original group of 15 case 

study sites (encompassing 30 schools). 

A specific example of effective parent involvement involved a school in which parents helped 

with fund-raising events, ran an after-school program, and served as advocates at the district 

office for facilities and maintenance. Support was gained by use of a Communities in Schools 

program that linked families with services such as a migrant program, ESL classes, housing 

assistance, family violence prevention, and health services. In addition, the school sought 

bilingual staff members, parent representatives were elected annually to the school council, and 

parents had a voice in the school. Although a decline in parent attendance was noted during the 

second site visit, increasing participation was reported during the third visit. The increase in the 
Chapter IV 40

third year was reported to be the result of leadership exerted by a core group of parents. Parent 

volunteers were also reported to have provided adult ESL and computer training. 

All of the schools in the improving schools case study had some form of parent involvement 

initiatives. Seven had a parent resource center and a range of strategies for involving parents in 

their students’ learning, including homework sign-off, family nights, and family literacy. Two 

schools had full-time parent coordinators and an additional two schools had part-time 

coordinators. Finally, one school had the staff members visit all students’ homes.

Component 8: External Assistance

The external assistance measures constructed for 2003 and 2007 rely on different items than the 

2005 measure (Exhibit 17). Consequently, any year-to-year comparisons should be made with 

caution. In 2003 and 2007, the external assistance measure is constructed from six yes or no 

questions on the types of external assistance available. Nonmissing responses were averaged, and 

a school was credited as implementing the external assistance measure if the school reported that 

at least 80 percent of the nonmissing items were reported implemented.

The 2005 survey includes four items that make up the external assistance measure. First, teachers 

were asked how many hours of training on reform they had received. The highest number 

reported was 959 hours. Some respondents may have interpreted this question as meaning over 

the life of the reform program, while others may have interpreted the question as the number of 

hours in the last year. Also, because there is often high teacher turnover in lower-performing 

schools, some respondents may have been in their school for only a portion of the CSR award 

implementation period. However, to scale this number to vary between 0 and 1, it was divided by 

1,000 and thus varies between 0 and 0.96. The next three questions (strategies include 

curriculum materials; all teachers received training on all strategies; and ongoing support is 

available) were asked up to four times, once for each reform identified in the school. The 

responses for each question were averaged to derive a school-level measure for each item that 

varied between 0 and 1. The external assistance measure was constructed by taking the average 

of the nonmissing four items, and schools with an average of at least 0.7 were counted as having 

implemented this item.
Chapter IV 41

Exhibit 17

Means and Standard Deviations of Survey Items Used to Construct the External 

Assistance Measure, by Year

Year External Assistance Items N Mean

Standard 

Deviation

Onsite consulting 725 0.70 0.46

Professional development 725 0.89 0.31

Networking 725 0.56 0.50

Written materials for students 725 0.43 0.50

Written materials for teachers 725 0.67 0.47

2003

Software or technology 725 0.45 0.50

Hours (1,000) of training on reform strategy 736 0.06 0.09

Strategies include curriculum materials 667 0.68 0.40

All teachers received training on all strategies 653 0.73 0.39

2005

Ongoing support is available 652 0.77 0.37

Onsite consulting 536 0.53 0.50

Professional development 536 0.79 0.41

Networking 536 0.45 0.50

Written material for students 536 0.32 0.47

Written material for teachers 536 0.54 0.50

2007

Software or technology 536 0.34 0.48

Exhibit highlights: The external assistance measures constructed for 2003 and 2007 rely on 

different items than the 2005 measure. The reader may compute the means into a percentage by 

multiplying the mean value by 100. For example, the 2003 mean onsite consulting score of 0.70 

indicates that 70 percent of respondents indicated that onsite consulting from external entities 

was available.

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Case Study Findings

External consultants served as one source of assistance for schools. At one school, the reform 

model developer was reported not only to have provided classroom demonstration but also to 

have worked with another external facilitator to help teachers incorporate method strategies into 

action plans, assist with curriculum mapping, and help teachers recognize the link between the 

school improvement plan and classroom activities. In another example, a school received a 

National Science Foundation grant and used the money to fund one mathematics facilitator and 

one science facilitator to focus on closing the achievement gap for traditionally underachieving 

groups of students. These facilitators provided professional development throughout the course 

of the study. Other types of external support received by schools included funding (National 

Science Foundation and 21st Century Fund), assistance from regional labs, a business partner, 

outside experts, and support from universities.
Chapter IV 42

Component 9: Evaluation

The 2003 and 2005 measures for evaluation are nearly the same, but one item in the 2003 

measure (whether there is a formal written plan to evaluate progress) was not asked of 

respondents in 2005 (Exhibit 18). All of the questions asked in both years were yes or no 

questions. For each year, nonmissing responses were averaged, and a school was credited as 

implementing the evaluation component if the average score was 1.0 in 2003 and at least 0.75 in 

2005. This distinction was made because of the one item not present in the 2005 survey and the 

large drop in several of the items from 2003 to 2005.

For 2007, the evaluation measure relies on only one item in which respondents were asked 

whether their school improvement plan included periodic evaluation goals. Eighty percent of the 

principals who responded indicated that their plan does have periodic evaluation goals.

Case study results showed that no school reported that it evaluated whole-school reform. 

Therefore, no case study discussion is included for this component.
Chapter IV 43

Exhibit 18

Means and Standard Deviations of Survey Items Used to Construct the Evaluation 

Measures, by Year

2003 2005 2007

Evaluation Items N Mean

Standard 

Deviation N Mean

Standard 

Deviation N Mean

Standard 

Deviation

Does the school 

improvement plan have 

a mechanism for 

periodic evaluation

720 0.84 0.37 673 0.91 0.28 537 0.80 0.40

Evaluation Plan Topics

Student performance 711 0.96 0.20 678 1.00 0.07 NA NA NA

Program implementation 711 0.74 0.44 678 0.86 0.35 NA NA NA

Parental participation 711 0.63 0.48 678 0.62 0.49 NA NA NA

Staff development 711 0.86 0.35 678 0.92 0.28 NA NA NA

Utility of external 

assistance 711 0.34 0.48 678 0.45 0.50 NA NA NA

Sources of financial 

support 711 0.43 0.50 678 0.35 0.48 NA NA NA

Is there a formal written 

plan to evaluate 

progress

712 0.85 0.36 NA NA NA NA NA NA

Exhibit highlights: Although survey items used to construct the evaluation measure were largely 

the same in 2003 and 2005, they changed in 2007. The reader may compute the means into a 

percentage by multiplying the mean value by 100. For example, the 2003 mean score of 0.84 for 

whether the school improvement plan has a mechanism for periodic evaluation indicates that 84 

percent of respondents indicated that the plan did have such a mechanism.

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Component 10: Coordination of Resources

The 2003 and 2005 measures for the coordination of resources component are constructed from 

different measures (Exhibit 19), and any year-to-year comparisons should be made cautiously. 

All of the questions asked in both years were yes or no questions. For each year, nonmissing 

responses were averaged, and a school was credited as implementing the coordination of 

resources component if the average score was 1.0 in 2003 and at least 0.7 in 2005. This 

distinction was made because of the increase in the types of funds that were asked about in 2005. 

That is, some schools may not have received foundation grants or local donations and may have 

reported on their survey that they did not have control over these funds.

In 2007, the coordination of resources measure relies on an item in which principals were asked 

whether their school had funding flexibility needed to implement the major elements of the 

reform at their school. Eighteen percent of principals responded that they definitely did have that 

flexibility.
Chapter IV 44

Exhibit 19

Means and Standard Deviations of Survey Items Used to Construct Coordination of 

Resources Measure, by Year

Year Items N Mean

Standard 

Deviation

The school has control over budgetary issues 711 0.77 0.42

The school has control over personnel decisions 708 0.78 0.41

How have existing resources been 

coordinated

Align district professional development 711 0.86 0.35

Align Title I activities 711 0.69 0.46

Align other funds 711 0.31 0.46

2003

Reallocate staff 711 0.40 0.49

The school has control over the following 

resources

Federal CSR funds 662 0.52 0.50

Title I funds 662 0.68 0.47

Other federal funds 662 0.33 0.47

Special state grants 662 0.40 0.49

Discretionary district funds 662 0.59 0.49

Foundation grants 662 0.25 0.43

2005

Local community or business donations 662 0.62 0.49

2007 Funding flexibility 536 0.18 0.38

Exhibit highlights: The coordination of resources measure was constructed on different survey 

items each year. The reader may compute the means into a percentage by multiplying the mean 

value by 100. For example, the 2003 mean score of 0.77 for the school has control over 

budgetary issues indicates that 77 percent of respondents indicated that the school does have 

such control.

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) Surveys.

Case Study Findings

There was evidence of district support and coordination in a majority of schools visited from the 

original 15 case study sites. In addition, a majority of schools qualified for support under Title I, 

although detail on how Title I services were coordinated with other aspects of the reform efforts 

was generally lacking. State support and coordination was less evident, however.

There were a few examples where coordination across all levels was in place. In one instance, a 

school facilitator worked with the method developer to refine curriculum mapping while the state 

provided a review and approval of the reform model evaluation plan. District support and 

coordination was evident also during the second visit to this school. Finally, findings from the 

third site visit revealed that the school had used Title I funding to hire a consultant to work with
Chapter IV 45

the school and coordinate with the district coaches to ensure that aspects of the reform model 

continued to be emphasized across the school.

There were also examples of an absence of integration and coordination between the district and 

the state. In one case, the state was the driver not only of the CSR effort but also of reform 

efforts in non-CSR schools. The district, in contrast, used its own program improvement 

initiatives, and the CSR program was a low priority. Although there was no information as to 

how, or whether, this conflict was resolved, the ultimate reliance on state assessments and 

benchmarks for the accountability measure may have been the deciding factor. In another 

instance, the state tried to use the reform model as the umbrella for other reforms. In this district, 

the model did not match the district curricula. In the final year, the model was discontinued 

because the district reforms were considered robust.

District decisions could also have a negative effect. In one instance, a district designated an 

improving school to serve as a “lighthouse” school that received students from other schools as 

part of the district’s choice plan. The district also added additional grades to the school, with 

little time for preparation, and the improving school subsequently lost its status because of low 

student performance. This school then became a sending school rather than a receiving school 

and may illustrate the fragile nature of some school improvement efforts. 

State frameworks in the form of school improvement plan expectations, grade-level content 

expectations, and core curriculum and professional development requirements supported reform 

and had an effect in at least two of the improving schools. In one instance, the state’s format for 

school improvement planning guided the school’s reform efforts for more than five years, with 

some adjustments.

FINDINGS

Were schools that received CSR awards more likely to implement the legislatively 

specified components of CSR than other schools?

 No, consistent with earlier study findings, both CSR and non-CSR schools 

implemented a similar number of legislatively specified components in both 2003 

and 2007. The number of components rose slightly for CSR and non-CSR schools 

during this period at both the elementary and middle school levels. 

Both CSR and non-CSR schools implemented an average of four or fewer components in 2003 

and fewer than five in 2007 at both the elementary and middle school levels (Exhibit 20). Both 

CSR and non-CSR elementary schools reported making similar gains in implementing 

components in that the number of components implemented increased by 0.7 in CSR schools and 

0.8 in non-CSR schools. At the middle school level, CSR and non-CSR schools in both 2003 and 

2007 also reported similar increases in the number of components implemented. 
Chapter IV 46

Exhibit 20

Average Number of CSR Components Implemented by CSR

and Non-CSR Schools in 2003 and 2007

Elementary Schools Middle Schools

School Type 2003 2007

Change 

from 2003 

to 2007 2003 2007

Change 

from 2003 

to 2007

CSR Schools 3.9 4.6 0.7** 3.4 4.1 0.6*

Non-CSR Schools 3.5 4.4 0.8** 3.2 4.2 1.0**

Difference Between CSR 

and Non-CSR Schools 0.4+ 0.1 0.1 0.2 -0.1 0.3

Exhibit highlights: In 2003, CSR elementary schools reported implementing a somewhat 

higher average number of components than non-CSR schools. No other significant differences 

between CSR and non-CSR schools were found. Both CSR and non-CSR schools reported 

increases between 2003 and 2007 in the number of components implemented.

Source: Evaluation of the Comprehensive School Reform Program Implementation and 

Outcomes (ECSRIO) surveys.

Note: + p < .10; * p < .05; ** p < .01; N = 150 for CSR elementary schools; N = 124 for nonCSR elementary schools; N = 112 for CSR and non-CSR middle schools. Standard errors for 

elementary schools are included in Exhibit A.5; standard errors for middle schools are in 

Exhibit A.6. 

Although the average number of components reported implemented in CSR and non-CSR 

schools was similar, the distribution of components implemented indicates minor differences in 

comprehensiveness between CSR and non-CSR schools. Nearly similar percentages of CSR and 

non-CSR elementary schools implemented more than seven components in 2007 (Exhibit 21). At 

the middle school level, a higher percentage of non-CSR schools than CSR schools implemented 

between four and six components (Exhibit 22). 
Chapter IV 47

Exhibit 21

Number of Components Implemented in CSR and Non-CSR

Elementary Schools in 2007

Exhibit highlights: A slightly higher percentage of Comprehensive School Reform (CSR) elementary 

schools reported implementing seven or more components in 2007 than did non-CSR schools. A 

slightly higher percentage of non-CSR schools than CSR schools implemented four to six 

components. The same percentage of CSR and non-CSR schools implemented zero to three 

components.

Source: Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 

(ECSRIO) surveys.
Chapter IV 48

Exhibit 22

Number of Components Implemented in CSR and Non-CSR

Middle Schools in 2007

Exhibit highlights: A slightly higher percentage of Comprehensive School Reform (CSR) middle 

schools implemented seven or more components in 2007 than did non-CSR schools. A slightly higher 

percentage of non-CSR middle schools reported implementing at least four components. The 

percentage of CSR schools implementing zero to three components was 4 percentage points higher 

than that of non-CSR middle schools. 

Source: Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 

(ECSRIO) surveys.

Three reasons may explain the similar implementation gains in both CSR and non-CSR school 

reform models. First, as shown in three of the case study sites, a school’s receipt of a CSR award 

can allow its school district to shift resources away from that school and toward other non-CSR 

schools also in need of improvement. Second, NCLB was in its first full year in 2002–03; newly 

available assessment data may have prodded school districts to assist schools identified as lowperforming. Third, as school districts gained experience with implementing the CSR program, 

they may have taken the lessons learned from those schools and applied them to non-CSR 

schools. This behavior was evident in two of the case study sites.
Chapter V 49

V. OVERALL STUDY CONCLUSIONS 

This study views the CSR program through numerous lenses, including achievement analyses, 

surveys of school reform implementation, and multiple case studies. This chapter summarizes the 

key findings deriving from this project and details what we see as the major implications for 

future federal education policy-making as well as for conducting high-quality research and 

evaluation studies in this area.

SUMMARY OF FINDINGS

In this summary of findings, we first lay out what we found about the implementation of the 

federal CSR program and related achievement outcomes. Specifically, we describe how states 

allocated their federal CSR program dollars and supported schools. We then detail the progress 

schools made in implementing their CSR awards by focusing on the implementation of the 

legislatively mandated components and the adoption of models with a scientific research basis. 

Finally, we describe achievement outcomes associated with the federal CSR program and its 

implementation. 

Our findings indicate that, overall, the federal CSR program did not lead to much comprehensive 

school reform or achievement gains. However, we also found that achievement gains were 

somewhat more likely in schools selecting models subsequently identified as having a scientific 

research base. The importance of adopting such models was generally more important in 

mathematics than in reading.

Overall Results of the Federal Comprehensive School Reform Program

The federal Comprehensive School Reform program did not yield comprehensively reformed 

schools nor was it associated with widespread achievement gains. There were some instances 

where we did observe achievement gains, however. These instances were somewhat more likely 

in mathematics in schools that selected models identified as having a scientific research base.

In addition, through intensive case study investigations, we did find some instances where 

schools undertaking comprehensive school reform substantially improved their operations and

subsequent achievement outcomes. After the discussion of the federal CSR program as a whole, 

we will turn to identifying the central conclusions from these case studies. We found that there 

was not a single route to improving low-performing schools; rather, success relied on the 

substantial overlap and interconnectedness among the many activities schools undertook as well 

as the alignment of those activities to specific contextual factors within the school.

Implementation of the Federal Comprehensive School Reform Program

CSR funds were strongly directed to high-poverty schools, i.e. those with poverty rates of at least 

75 percent, and those with high concentrations of minority students. Almost half (45 percent) of 

the CSR schools were high-poverty schools, nearly three times greater than the percentage of all 
Chapter V 50

high-poverty schools (16 percent) and more than one and a half times the percentage of Title I 

schools (26 percent).

Many states and districts gave priority for CSR funding to such schools.24 One state, for 

example, listed all the schools identified in need of improvement and provided CSR funds to the 

most “in need,” working through the list until the funds ran out. Slightly more than half of the 

states awarded CSR funds based on a combination of merit and academic need. Significantly, 

when states evaluated CSR applications based solely on merit, fewer high-poverty schools 

received CSR awards. High-poverty schools may lack capacity to develop proposals, which was 

mitigated by including need as a criterion. 

Although states largely succeeded in providing CSR funds to those schools most in need, schools 

receiving CSR awards made little progress in implementing more than just a few of the 

legislatively mandated components and were largely indistinguishable from non-CSR schools 

that were similar in baseline achievement and demographics. By the third year of their grants, 

CSR schools reported implementing 4 of the 11 mandated components on average; two years 

later, they had not made much progress beyond that. These levels of implementation are quite 

similar to a group of comparison non-CSR schools that participated in this study, indicating lowperforming, high-need schools (both CSR and non-CSR) were reforming at similar rates.

Furthermore, only one-third of the schools receiving CSR awards selected reform models that 

would eventually be identified as having a scientific research basis. Schools primarily adopted 

externally developed models that lacked such a research base, or “home-grown,” locally 

developed models.25

This finding is similar to that of another major national study of comprehensive school reform, 

the National Longitudinal Examination of Comprehensive School Reform (NLECSR) (Aladjem 

et al., 2006), which studied implementation of seven of the most commonly adopted, externally 

developed CSR models. The NLECSR study also found that CSR schools had implemented the 

components of their CSR models at levels similar to non-CSR comparison schools and that both 

groups of schools had similar increases in fidelity of implementation over time. However, 

whereas the NLECSR found that implementation increased between the first and third years of 

model implementation, such increases were not evident in ECSRIO. The NLECSR also found 

that the degree of implementation was related to the model being implemented: those models that 

are considered highly prescriptive, for example, Success for All, tended to be implemented with 

greater fidelity than models that were not as prescriptive. ECSRIO did not focus on differences 

 

24. Findings on how states allocated their CSR funds to districts and schools derive from the Second-Year 

report, “Longitudinal Assessment of Comprehensive School Reform Program Implementation and Outcomes,” 

which is an internal document to the U.S. Department of Education, compiled by the Office of Planning, 

Evaluation, and Policy Development/ Policy and Program Studies Service, November 2007.

25. Reports from the CSRQ Center, which formed the foundation for our measures of the research base of CSR 

models (see U.S. Department of Education, 2008, for a complete discussion of how research bases of CSR 

models were measured), were not available at the time that 2002 CSR awardees made their model selections; 

however, schools may have used data from two earlier reviews of CSR models: Herman (1999) and Borman, 

Hewes, Overman, & Brown (2002). The CSRQ reports drew heavily from the data used in previous reviews of 

CSR models. Of the 40 ratings conducted, 25 models were rated as having some evidence of effectiveness. The 

CSRQ Center relied on the number and quality of research studies of the models to develop its ratings.
Chapter V 51

between models and, as a result, may have missed this potentially important covariate of 

implementation. 

Three reasons may potentially explain the similar implementation gains in both CSR and nonCSR school reform models in our study. First, as shown in three of the case study sites, a 

school’s receipt of a CSR award can allow its school district to shift resources away from that 

school and toward other non-CSR schools also in need of improvement. Second, NCLB was in 

its first full year in 2002–03; newly available assessment data may have prodded school districts 

to assist all schools (CSR and non-CSR) identified as low-performing. Finally, as school districts 

gained experience with implementing CSR, they may have taken the lessons learned from those 

schools and applied them to non-CSR schools. This behavior was evident in two of the case 

study sites and was cited as a potential explanation in NLECSR. 

If these explanations are valid, then the federal CSR program may be responsible for more 

reform than is being given credit here. That is, it may have provided states and districts with the 

resources and information to assist a large number of schools identified as being in need of 

improvement, whether they were CSR schools or not. Conversely, other factors might explain 

the absence of difference in implementation of reforms between CSR and non-CSR schools in 

this study. The pressing need of assisting large numbers of schools may have prevented states 

and districts from supporting CSR schools that needed in-depth assistance. Further, with the 

accountability requirements in NCLB, many schools may have pursued strategies targeting 

assistance to those students nearest the state’s proficiency cut point as well as to specific 

population groups (such as English language learners). Such strategies might well have diverted 

attention from more comprehensive approaches to school reform. 

Achievement Outcomes Associated With the Federal Comprehensive School Reform 

Program

The federal CSR program did not result in achievement gains. During the time that schools were 

implementing their awards, the 2002 cohort of CSR schools made small statistically significant 

gains in elementary mathematics and reading achievement. However, a group of comparison 

schools made similar gains, indicating that the CSR program was no better than comparable Title 

I schools at improving achievement after three years. Similarly, five years after initially 

receiving their CSR awards, schools receiving awards did not demonstrate larger achievement 

growth than matched comparison schools not receiving CSR grants. In fact, achievement gains 

were nonexistent in CSR elementary schools and lower than their comparison schools at the 

middle school level.

When we examined the relationship between implementation and achievement, we found in the 

ECSRIO study that the number of components a school implemented after three years did not 

predict achievement gains. In contrast, the NLECSR study found a relationship between 

implementation and achievement. In that study, CSR schools that implemented their models with 

fidelity, especially between implementation years three and five, had higher achievement gains 

than other schools. The NLECSR also found model-specific achievement effects. Schools that 

adopted Success for All tended to implement it faithfully and thus realized achievement gains.
Chapter V 52

One potential explanation for the differences in study findings is the different foci of the two 

studies: specifically, NLECSR focused on fidelity to specific curriculum models while ECSRIO 

focused on fidelity to the principles in the comprehensive school reform law. The focus of 

ECSRIO on fidelity to the law, though appropriate for a program evaluation of a federal effort, 

may not capture those more nuanced aspects of implementation related to achievement.26

One instance where ECSRIO found some evidence of improved achievement is when CSR 

schools adopted models with a scientific research base. In general, the results demonstrated 

stronger relationships between the presence of models with scientific research bases and 

improved achievement in mathematics compared with those models and reading. Specifically, 

schools implementing reform models with some scientific research backing of effectiveness were 

more likely than others to experience achievement gains in middle school mathematics. In 

addition, low-performing schools that adopted CSR models with scientific evidence of 

effectiveness had a better chance of improving their elementary mathematics scores than schools 

adopting models without evidence of effectiveness. There was also some evidence, albeit 

weaker, that schools adopting models with some scientific evidence of effectiveness were more 

likely to experience achievement gains in middle school reading than schools adopting other 

models. 

Case Studies of School Improvement 

Despite the finding that the federal CSR program as a whole did not lead to improvements in 

achievement or school operations, we did find instances where targeted, sustained efforts 

resulted in achievement gains. Results reported in the Achieving Dramatic School Improvement: 

An Exploratory Study report (U.S. Department of Education, 2010) not only highlighted 

instances of achievement gains but also expanded the knowledge base about the challenges and 

intricacies of comprehensive school reform. That study of rapid improvement schools (i.e., those 

that made substantial achievement gains in a short time period) and of schools improving at a 

slower and steadier pace specifically identified leadership, school climate, instruction, and data 

use as important factors in the improvement process.

School Leadership

One of the most frequently cited practices for improving schools is changing the school leader. 

The arrival of a new principal sends a clear message that change is necessary. In line with the 

research literature, this study found that both the principal and distributed leadership were 

important in stimulating, implementing, and, in some cases, sustaining reforms to improve 

student achievement. That is, although we found instances where strong principals arrived in a 

low-performing school and almost single-handedly began the turnaround process, evidence 

emerged in many of our case studies that improving schools possess varying degrees of 

distributed leadership. This distributed leadership strategy appeared to be especially important in 

providing continuity when schools experienced turnover of a principal. 

 

26. The NLECSR study examined schools that were receiving and not receiving CSR funding.
Chapter V 53

Principals created conditions and structures to distribute leadership by mobilizing staff members 

and building a professional community—some with selected participants, others requiring 

universal participation. In all cases, principals tied those structures to school improvement 

efforts. For instance, several principals in the case study schools placed a major emphasis on 

enhancing the roles and responsibilities of school staff members for school improvement by 

recruiting more experienced staff members to participate in school decision making, opening up 

the decision-making process to the broader staff community (particularly when the previous 

involvement had been limited to a few members), and/or instituting committees or research 

teams that zeroed in on specific topics to support school improvement. 

Reported benefits of distributed leadership were interrelated and included shared responsibility, 

greater staff buy-in, more effective implementation of new practices, continuity of leadership, 

and enhanced collaboration. For example, by drawing on teachers from across grade levels, the 

organization of committees in one of the schools enhanced both horizontal (within grade) and 

vertical (across grades) collaboration. Teachers at another school believed that they were 

creating their own reforms, even though they were receiving considerable external help, because 

their own senior teachers were rolling out the strategies. 

Another benefit of distributed leadership is that it can be critical to maintaining continuity of 

leadership and thereby sustaining reforms despite leadership changes. At one school, for 

example, the staff members selected by the principal to serve on the management team continued 

to serve in pivotal leadership roles at the school five years later. When the principal left a school 

after the 2005–06 school year, its teacher leaders continued to lead and run the school with 

seemingly little guidance from the new principal. This case study information leads us to 

hypothesize that these schools’ continued achievement growth across two school leaders might 

be partially attributed to the continuity provided by a distributed leadership model.27

School Climate

The literature on school improvement and turnaround schools suggests that improving the 

climate in troubled schools can accompany—and facilitate—improved achievement (Herman et 

al., 2008). Changes to the climate that make the school safer and more orderly can serve a dual 

purpose: they demonstrate that it is possible to make quick and dramatic changes to the school, 

motivating staff members and students to support successful turnaround efforts (Herman et al., 

2008), and they eliminate a set of pressing, nonacademic needs to allow staff members to focus 

on academics (Kowal & Hassel, 2005).

All of the schools visited in our study faced school climate challenges at the beginning of the 

improvement period, for example, a disruptive student learning environment or uninvolved 

parents, and each school adopted specific strategies intended to improve this condition. Nearly 

all of the schools considered changes to improve school climate as important, and in about half 

of the schools, new principals considered addressing school climate concerns as a necessary first 

step in laying the groundwork for instructional improvement efforts. Common approaches to 

 27. For additional information, please see Chapter 3: Leadership from Achieving Dramatic School Improvement: An 

Exploratory Study (U.S. Department of Education, 2010).
Chapter V 54

improving school climate included enhancing behavior management efforts, involving parents 

meaningfully in school governance, and soliciting resources from the community.28

Instruction

The literature on turning around low-performing schools suggests that a relentless focus on 

improving student achievement is a common success factor. All of the case study schools that 

made dramatic improvement reported making instructional improvements to raise student 

achievement. Common strategies to enhance instruction, in line with the research literature, 

included aligning the curriculum to district or state standards and assessments; adopting a new 

curriculum; and increasing learning time through an extended school year, after-school 

programs, or block scheduling within the school day. 

To support instructional improvement efforts, almost all schools reported sharing and 

systematically using data to guide instructional changes. One school, for example, prominently 

displayed data boards throughout the school and, instead of using them for punitive monitoring, 

established a shared responsibility for results, with one teacher noting “we all sink, or we all 

swim.” At another school, the notion of data display and transparency was extended to students, 

who regularly tracked their own progress against an “aim line” that served as a means of 

comparing their current achievement with established goals. 

Professional development from coaches and external consultants, often hired through district or 

state support, also appeared to be key in improving instruction. This strategy not only provided 

immediate instructional assistance but also laid the groundwork for schools becoming less reliant 

on funding and external assistance. Involvement in extensive professional development helped 

identify and develop teacher leaders who helped to sustain and institutionalize reforms.29

Federal and State Support

In addition to actions at the school level, districts, states and the federal government can play an 

important role in stimulating both rapid improvement and more incremental achievement gains. 

We found that accountability requirements can push schools to change, and additional resources 

can support these efforts. Respondents in about half of the schools that were visited in the 

dramatic school improvement study cited district support, guidance and assistance as being 

instrumental to their success. Furthermore, four of these schools received substantial state 

support. While respondents were somewhat less likely to mention states and the federal 

government as specifically and directly affecting local reform, the overall accountability context 

appeared to have an important motivating effect at the local level. Respondents noted that the 

consequences for chronic low performance, as determined by federal and state accountability 

measures, had stimulated change. 

 

28. For additional information, please see Chapter 4: School Climate from Achieving Dramatic School 

Improvement: An Exploratory Study (U.S. Department of Education, 2010).

29. For additional information, please see Chapter 5: Instructional Improvement Strategies from Achieving 

Dramatic School Improvement: An Exploratory Study (U.S. Department of Education, 2010).
Chapter V 55

Although the primary form of external support was financial, some respondents provided 

examples of in-kind assistance, predominantly from the districts. In-kind assistance was 

generally in the form of consultation and professional development for instructional coaches and 

teachers. In some cases, schools and districts used funding to purchase assistance directly from 

state and private agencies. Other forms of district support included assigning experienced 

principals with the explicit purpose of turning the school around or guiding reform efforts by 

establishing consistent expectations.30

Sustaining Improvement

Each of the schools visited in the study faced challenges in sustaining their improvement 

strategies and achievement gains, including challenges caused by lost resources and staff 

turnover. Some respondents perceived their schools’ achievement gains as being related to the 

actions of, and resources provided by, districts, states, and, in some cases, the federal 

government. These respondents were concerned about the potential loss of external support. 

Some of these schools found ways to replace lost resources, and some have succeeded in 

institutionalizing improved instructional practices. 

Most schools developed strategies to alleviate resource fluctuations, for example, by seeking 

supplemental fiscal resources to replace lost funds and building expertise among staff members 

to reduce reliance on external support. Schools generally had an influx of resources over several 

years at a time. As grant funding streams ended, 8 of the 11 schools studied were able to rely on 

new funding streams. 

Several schools made themselves less reliant on funding and external assistance by providing 

extensive professional development for their teachers and by building teacher leaders through 

distributed leadership opportunities. Five schools used their funds and other resources to focus 

on enhancing the knowledge and skills of their staff members through professional development 

or to purchase or develop materials or instructional programs that would remain in place once 

funding ended. One school’s principal focused on securing grants that would build staff skills to 

outlive the life of the grant. Regional consultants at another school trained a selected group of 

teacher leaders on instructional practices and on training other teachers in those strategies. Near 

the end of our study, that school lost its school improvement specialist because of lack of 

funding. However, much of the implementation of the instructional practices had become 

institutionalized. 

Respondents also identified staff turnover, both voluntary and involuntary, as an impediment to 

sustaining academic achievement. One charter school consistently saw high staff turnover as 

teachers sought to move from that school to regular, higher paying, tenure-track positions in the 

local public school district. Another school reported nearly half of the teaching staff being new to 

the school in 2004–05, largely because of a new school opening up in the district. To combat the 

problems associated with staff turnover, a few schools strengthened the orientation of new staff 

members. At two schools, teachers participated in a new teacher academy. Yet another instituted 

 

30. For additional information, please see Chapter 6: External Support from Achieving Dramatic School 

Improvement: An Exploratory Study (U.S. Department of Education, 2010).
Chapter V 56

a mentor system that assigned new teachers to an experienced member of the staff to help 

integrate and acculturate the new teachers to the school.

Closing Discussion: No One Way to Improve Low-Performing Schools 

One of the key findings from the Achieving Dramatic School Improvement: An Exploratory 

Study (U.S. Department of Education, 2010) report is that although there are several overarching 

categories of activity on which improving schools generally rely, the paths toward school 

improvement varied based on local needs and conditions. Some placed greater emphasis on one 

factor (e.g., distributed leadership) than another (e.g., transparent use of student-level data); 

others chose a different order of factors or they combined features of reform into unique ways to 

establish a coherent whole-school approach. Furthermore, reform strategies interacted in multiple 

ways, suggesting that the same reforms may be more or less successful depending on differences 

in leadership, staff capacity, community support and other factors. There is no “one way” to 

implement an effective school improvement program since different factors play out in different 

ways in different settings. 

This report also highlighted the multiple ways by which schools can improve quickly and 

dramatically. The NLECSR findings similarly illustrated the interdependence of CSR model 

components. Neither this study nor NLECSR identified a single path to improvement or a unique 

bundle of components or activities associated with school improvement. On the contrary, both 

studies demonstrated that sustained improvement implies substantial overlap and

interconnectedness among the many activities that schools undertake. These suggested multiple 

connections lead directly to the question of how the federal government can foster scalable or 

systemwide reform that resemble the successes observed in our case study sites and in NLECSR.

We would posit that systemwide reform or scaled-up improvement of this type requires highly 

contextualized and resource-intensive support. The U.S. Department of Education would need to 

begin to provide substantial amounts of direct and individually tailored technical assistance to 

state departments of education and possibly also to local school districts. The assistance the 

Department would provide to states would mirror the situationally dependent assistance some 

states in the case studies provided districts and schools.

The Department could expand the scope of work of its network of regional Comprehensive 

Centers to provide this kind of support to states and districts attempting to turn around lowperforming schools. Currently, the Comprehensive Centers are tasked with providing assistance 

primarily to state education agencies to implement NCLB; to use scientifically valid teaching 

methods and assessment tools; and to facilitate communication between education experts, 

school officials and parents. A new expanded mission for these Centers could take the form of 

developing and providing training to assist states, districts and schools in identifying the 

organizational approach or approaches (e.g., leadership changes, accountability systems, and/or 

the implementation of research-based school reform models with fidelity) needed to improve 

quality instruction in schools. 
Chapter V 57

FUTURE RESEARCH NEEDED FOR IMPROVING CHRONICALLY LOW-PERFORMING 

SCHOOLS

Future research for improving chronically low-performing schools could play a significant role 

in shaping the nature of subsequent federal efforts and of other service providers so they are 

targeted on interventions most likely to result in positive effects. To do so, we recommend that 

such an agenda address the following critical needs. 

1. The need for a more robust national database. In our national study of the federal CSR 

program, CSR award databases provided limited information about the CSR model chosen 

beyond the initial year of funding. These databases also lacked information about the total 

amount of funding provided beyond the first year of the award.

Similarly, one challenge in selecting sites for the case studies of dramatic school improvement 

was the absence of up-to-date national data on student achievement. Reliable estimates of 

school-level performance should ideally be derived from longitudinal, student-level data to 

ameliorate problems inherent in “percent proficient” school-level calculations through the use of 

robust student growth curve models. This issue is of particular importance in schools that 

experience high levels of student mobility, which is the case in many low-performing schools.

We were also constrained in our work by having a data set that not only was out of date but also 

provided achievement information only at the school level. Because the school-level data 

available from the NLSLSASD were not updated, we needed to gather data from individual 

states that varied in their completeness and in how recent they were. 

Although one might reasonably expect future updates to national school-level data sets such as 

the NLSLSASD, it seems unlikely that a comparable national database of student-level data will 

be compiled in the near future. Nevertheless, the U.S. Department of Education could work with 

the states to enhance the EdFacts database by including relevant activities within the schools. 

Critical data elements should address principal and faculty stability, measures of school climate, 

reform models) in use, award amounts from federal funds and other major sources, use of 

extended hours for instruction, use of data, and instructional support such as teacher mentors and 

internal and/or external coaches. Furthermore, the Department should make these data available 

to the research community in real time so that timely, relevant analyses can be conducted. 

2. The need for real-time investigations and the avoidance of retrospective analyses. 

Researchers, policy-makers, and practitioners in recent years have paid much attention to the 

scientific search for “what works” to improve schools and turn them around. Our study’s 

findings underscore the challenges inherent in such efforts by drawing attention to the fact that 

comprehensive school reform is not just about adopting a set of effective or promising practices. 

It is also about navigating and adapting to a constantly changing landscape that needs to be 

monitored on an ongoing basis. Real-time feedback and analysis is needed to accomplish this 

type of reform. 

With a more robust database as described above, researchers could identify potential lowperforming schools that turned around as soon as dramatic improvements in achievement were 

observed. As schools are identified as potential case study sites, researchers could identify 
Chapter V 58

matched comparison schools. Next, with potential treatment and comparisons identified, 

researchers could quickly begin more in-depth, qualitative work to examine and track practices 

close in time to each school’s achievement improvement. Such an approach not only would 

minimize the error induced by retrospective data collection but also would provide critically 

important counterfactual data from the identified comparison sites.

Finally, after tracking these matched pairs of successful turnaround and comparison schools for 

several years, enough should have been learned about the process of dramatic school 

improvement to design one or several interventions that would be the basis for experimental or 

quasi-experimental studies of particular attributes of successful school turnaround identified in 

the first two stages of this study.

This ability to design interventions could also suggest an alternative, expanded role for the 

Regional Educational Laboratories, whose goal is to provide “access to high-quality, 

scientifically valid education research through applied research and development projects, 

studies, and other related technical assistance activities.”31 The Regional Educational 

Laboratories could be charged with identifying and conducting real-time investigations of lowperforming schools that dramatically improved in their regions that could be used to develop 

interventions with descriptions of the conditions under which those interventions seem to work 

best.

3. The need for well-developed implementation measures. One of the key differences between 

the NLECSR and ECSRIO is in the definition of implementation. By focusing on fidelity to the 

law, ECSRIO did not capture differences in models and essentially treated all models as having 

the same components. However, the findings from the dramatically improving schools case 

studies and from the NLECSR highlight the fact that school reform is context-specific. That is, 

while certain overarching themes should be addressed (e.g., leadership, school climate and 

instruction), the specifics of addressing these critical areas rely on the particular circumstances of 

the school.

Thus, what is needed are well-developed school reform implementation measures serving two 

purposes that are at risk of potentially conflicting with each other. First, implementation 

measures need to be specific enough to effectively characterize the school reform effort. 

Effective leadership occurs not just through the presence of an effective leader but, rather, 

evolves through the correct style and distribution of leadership in a school to match that school’s 

context. Second, implementation measures must be broad enough to allow for comparison across 

school contexts so that research can effectively identify the set of practices that a school can 

adopt in a particular set of circumstances and feel reasonably confident that those practices have 

been shown in previous research to yield positive reform.

Once these measures are developed, it would not be unreasonable to expect states, districts and 

schools receiving funding for school reform to provide periodic robust information about 

implementation. The systematic collection and analysis of such data would be a major step 

 

31. See the welcome statement of the U.S. Department of Education’s Institute of Education Sciences, 

Regional Education Laboratory Program, which is found on the program’s home page at 

http://ies.ed.gov/ncee/edlabs/. 
Chapter V 59

toward a continuously more refined and comprehensive understanding of the implementation 

phenomena and how they relate to improved teaching and learning in classrooms. This in-depth 

understanding would in turn help policy-makers design better interventions over time, leading to 

more effective reforms for low-achieving schools and heightened student achievement.

References 61

REFERENCES 

Aladjem, D. K., LeFloch, K. C., Zhang, Y., Kurki, A., Boyle, A., Taylor, J. E., Herrmann, S., 

Uekawa, K., Thomsen, K., & Fashola, O. (2006). Models matter—The final report of the 

National Longitudinal Evaluation of Comprehensive School Reform. Washington, DC: 

American Institutes for Research. Retrieved October 6, 2009, from 

http://www.air.org/publications/documents/NLECSR_2006.pdf.

Borman, G. D., Hewes, G. M., & Overman, L. T. (2003). Comprehensive school reform and 

achievement: A meta-analysis. Review of Educational Research, 73(2), 125–230.

Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. A. (2002). Comprehensive 

school reform and student achievement: A meta-analysis (CRESPAR Technical Report 

No. 59). Baltimore, MD: Center for Research on the Education of Students Placed at Risk, 

Johns Hopkins University.

Burstein, L. (1978). Assessing differences between grouped and individual-level regression 

coefficients: Alternative approaches. Sociological Methods and Research, 7, 5–28.

Burstein, L., & Miller, M. D. (1981). Regression-based analyses of multi-level educational 

data (Center for the Study of Evaluation Report No. 175). Los Angeles: University of 

California, Los Angeles, Graduate School of Education. 

Comprehensive School Reform Quality Center. (2005). CSRQ Center report on elementary 

school comprehensive school reform models. Washington DC: American Institutes for 

Research.

Comprehensive School Reform Quality Center. (2006). CSRQ Center report on middle and 

high school comprehensive school reform models. Washington D.C.: American Institutes 

for Research.

Elementary and Secondary Education Act of 1965, Pub. L. 89-313, 20 U.S.C. 2701 et seq.

Flower, F. (1995). Improving survey questions: Design and evaluation. Thousand Oaks, CA: 

Sage.

Fraas, J. W., Newman, I., & Pool, S. (2007). The use of propensity score analysis to address 

issues associated with the use of adjusted means produced by analysis of covariance. 

Multiple Linear Regression Viewpoints, 33(1), 23–31.

Gill, B. P., Hamilton, L. S., Lockwood, J. R., Marsh, J. A., Zimmer, R. W., Hill, D., & 

Pribesh, S. (2005). Inspiration, perspiration, and time: Operations and achievement in 

Edison schools. Santa Monica, CA: RAND. Retrieved October 6, 2009, from 

http://www.rand.org/pubs/monographs/MG351/.

Haney, W. (1980). Units and level of analysis in large-scale evaluation. New Directions for 

Methodology and Social and Behavioral Science, 6, 1–15.
References 62

Herman, R. (1999). An educator’s guide to schoolwide reform. Arlington, VA: Educational 

Research Service.

Herman, R., Dawson, P., Thomas, D., Greene, J., Maynard, R., Redding, S., & Darwin, M. 

(2008). Turning around chronically low-performing schools (NCEE No. 2008-4020).

Washington, D.C.: National Center for Education Evaluation and Regional Assistance, 

Institute of Education Sciences, U.S. Department of Education. Retrieved October 6, 

2009, from http://www.ies.ed.gov/ncee/wwc/pdf/practiceguides/Turnaround-pg04181.pdf.

Kantor, D. (2006). MAHAPICK: Stata module to select matching observations based on a 

Mahalanobis distance measure. Retrieved October 6, 2009, from 

http://ideas.repec.org/c/boc/bocode/s456703.html.

Kowal, J. M., & Hassel, E. A. (2005). Turnarounds with new leaders and staff. Washington, 

DC: The Center for Comprehensive School Reform and Improvement.

Lipsey, M. (2003). Assessing systematic evidence in crime and methodological concerns and 

empirical outcomes. The Annals of the American Academy of Political and Social 

Sciences, 587, 69–81.

No Child Left Behind Act of 2001, Public Law 110, 107th Congress, 1st Session (Jan. 8, 

2002).

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data 

analysis methods. 2nd ed. Thousand Oaks, CA: Sage.

Rosenbaum, P. R., & Rubin, D. B., (1983). The central role of the propensity score in 

observational studies for causal effects. Biometrika, 70, 41–55.

Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control group using multivariate 

matched sampling methods that incorporate the propensity score. The American 

Statistician, 39(1), 33–38.

Rubin, D. B. (1980). Bias reduction using Mahalanobis-metric matching. Biometrics, 36, 293–

298.

Tourangeau, R., Rips, L. J., & Rasinki, K. (2000). The psychology of survey responses. New 

York: Cambridge University Press.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, and 

Office of Elementary and Secondary Education. (2004). Longitudinal assessment of 

comprehensive school reform program implementation and outcomes: First-year report.

Washington, DC: Author.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, and 

Office of Elementary and Secondary Education. (2007). State ESEA Title I participation 

information for 2003–04. Washington, DC: Author.
References 63

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, and 

Office of Elementary and Secondary Education (2010). Achieving dramatic school 

improvement: An exploratory study. Washington, DC: Author.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, 

Policy and Program Studies Service. (2008). Evaluation of the comprehensive school 

reform program implementation and outcomes: Third-year report. Retrieved October 6, 

2009, from http://www.ed.gov/rschstat/eval/other/csrd-outcomes/year3-report.doc.

Zhao, X. H., Li, C. M., Gao, S. J., & Tierney, W. M. (2001). Methods for testing equality of 

means of health care costs in a paired design study. Statistics in Medicine, 11(20), 1703–

1720.
Appendix A 64
Appendix A 65

APPENDIX A.

STANDARD ERROR TABLES FOR ANALYSES

This appendix includes the standard error tables for the analyses presented throughout the report.

STANDARD ERRORS FOR REPORT EXHIBITS

Exhibit A.1

Standard Errors for Differences Between the 2002 Cohort of Title I CSR Schools and 

Title I Non-CSR Schools

Non-CSR CSR

N Standard 

Error N Standard Error

Standardized Assessment Scores

Elementary Mathematics 31,960 0.005 273 0.056

Elementary Reading 32,117 0.005 274 0.054

Middle School Mathematics 12,044 0.009 105 0.081

Middle School Reading 12,222 0.009 105 0.090

Membership 53,758 1.552 402 19.365

Percentage Minority 52,016 0.156 386 1.839

Percentage FRL 48,872 0.122 362 1.096

Exhibit highlights: Standard errors for estimates presented in Exhibit 5 in the main body of the 

report.

Source: Common Core of Data (CCD).

Note: “FRL” refers to students eligible for free or reduced-price lunches.
Appendix A 66

Exhibit A.2

Standard Errors for Differences Between the 2002 Cohort of Title I CSR Schools and 

Non-CSR Title I Schools Chosen by Propensity Scoring: Schools With Elementary 

Mathematics and Reading Achievement

Non-CSR CSR

N Standard 

Error N Standard Error

Standardized Assessment Scores

Elementary Mathematics 478 0.042 234 0.063

Elementary Reading 478 0.041 235 0.059

Membership 478 11.266 244 17.670

Percentage Minority 478 1.599 244 2.324

Percentage FRL 461 1.002 233 1.338

Exhibit highlights: Standard errors for estimates presented in Exhibit 6 in the main body of the 

report.

Source: Comprehensive School Reform (CSR) Awards Database; Common Core of Data 

(CCD); National Longitudinal School-Level State Assessment Score Database (NLSLSASD).

Note: “FRL” refers to students eligible for free or reduced-price lunches.

Exhibit A.3

Standard Errors for Differences Between the 2002 Cohort of Title I CSR Schools and NonCSR Title I Schools Chosen by Propensity Scoring: Schools With Middle School 

Mathematics and Reading Achievement

Non-CSR CSR

N Standard 

Error N Standard Error

Standardized Assessment Scores

Middle School Mathematics 248 0.053 95 0.082

Middle School Reading 248 0.048 95 0.093

Membership 248 26.716 109 43.026

Percentage Minority 248 2.283 109 3.516

Percentage FRL 243 1.442 103 2.287

Exhibit highlights: Standard errors for estimates presented in Exhibit 7 in the main body of the 

report.

Source: Comprehensive School Reform (CSR) Awards Database; Common Core of Data (CCD); 

National Longitudinal School-Level State Assessment Score Database (NLSLSASD).

Note: “FRL” refers to students eligible for free or reduced-price lunches.
Appendix A 67

Exhibit A.4

Standard Errors for Changes in Standardized Assessment Scores in CSR and Non-CSR 

Schools From 2002–03 to 2006–07

N CSR 

Schools

Non-CSR 

Schools

Elementary Mathematics 560 0.043 0.041

Elementary Reading 563 0.038 0.040

Middle School Mathematics 291 0.040 0.041

Middle School Reading 294 0.045 0.045

Exhibit highlights: Standard errors for estimates presented in Exhibit 8 in the main body of the 

report.

Source: Comprehensive School Reform (CSR) Awards Database; Common Core of Data (CCD); 

National Longitudinal School-Level State Assessment Score Database (NLSLSASD).
Appendix A 68

Exhibit A.5

Standard Errors for Average Number of CSR Components Implemented by

CSR and Non-CSR Elementary Schools in 2003 and 2007

School Type 2003 2005

CSR Schools (N = 153) 0.136 0.156

Non-CSR Schools (N = 128) 0.143 0.162

Exhibit highlights: Standard errors for estimates presented in Exhibit 20 in the main body of the 

report.

Source: Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 

(ECSRIO) surveys.

Exhibit A.6

Standard Errors for Average Number of CSR Components Implemented by CSR and

Non-CSR Middle Schools in 2003 and 2007

School Type 2003 2005

CSR Schools (N = 69) 0.170 0. 230

Non-CSR Schools (N = 55) 0.262 0.259

Exhibit highlights: Standard errors for estimates presented in Exhibit 20 in the main body of the 

report.

Source: Evaluation of the Comprehensive School Reform Program Implementation and Outcomes 

(ECSRIO) surveys.
Appendix B 69

APPENDIX B.

DATA COLLECTION INSTRUMENTS

Principal Inventory Year 1 ........................................................................................................ 71

Principal Survey Year 2 ............................................................................................................. 87

Principal Survey Year 5 ............................................................................................................. 97

Teacher Inventory Year 1 ........................................................................................................ 113

Teacher Survey Year 2 ............................................................................................................. 127

Teacher Survey Year 5 ............................................................................................................. 137

Field Study Protocol ................................................................................................................. 157
Appendix B 70
Appendix B 71 Principal Inventory Year 1

Principal Inventory Year 1
Appendix B 72 Principal Inventory Year 1
Appendix B 73 Principal Inventory Year 1
Appendix B 74 Principal Inventory Year 1
Appendix B 75 Principal Inventory Year 1
Appendix B 76 Principal Inventory Year 1
Appendix B 77 Principal Inventory Year 1
Appendix B 78 Principal Inventory Year 1
Appendix B 79 Principal Inventory Year 1
Appendix B 80 Principal Inventory Year 1
Appendix B 81 Principal Inventory Year 1
Appendix B 82 Principal Inventory Year 1
Appendix B 83 Principal Inventory Year 1
Appendix B 84 Principal Inventory Year 1
Appendix B 85 Principal Inventory Year 1
Appendix B 86 Principal Inventory Year 1
Appendix B 87 Principal Survey Year 2

Principal Survey Year 2
Appendix B 88 Principal Survey Year 2
Appendix B 89 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix B 90 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix B 91 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix B 92 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix B 93 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix B 94 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix B 95 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix B 96 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix B 97 Principal Survey Year 5

Principal Survey Year 5

Longitudinal Assessment of 

Comprehensive School 

Reform 

Principal Survey 

2006-07 School Year

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless 

such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1875-

0222. The time required to complete this information collection is estimated to average 40 minutes per response, including the 

time to review instructions, search existing data resources, gather the data needed, and complete and review the information 

collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, 

please write to: U.S. Department of Education, Washington, D.C. 20202-4651. If you have comments or concerns about the 

contents of this questionnaire, write directly to: 

WestEd, attn: John Flaherty, Project Coordinator, 4665 Lampson Avenue, Los Alamitos, CA 90720
Appendix B 98 Principal Survey Year 5
Appendix B 99 Principal Survey Year 5

Dear Principal,

WHO IS CONDUCTING THIS SURVEY?

WestEd and Cosmos Corporation, educational research organizations under contract to the U.S. 

Department of Education, request your participation in this survey. WestEd and the Department 

of Education are conducting this survey by the authority of Section 1607 of the No Child Left 

Behind Act of 2001 (P.L. 107-110).

WHAT IS THE PURPOSE OF THIS SURVEY?

This survey is designed to understand how comprehensive reform at your school changes the 

ways that you and your staff approach teaching and learning. This survey focuses on changes 

that can affect every facet of school operations including grade-level planning, curriculum and 

instruction, data driven decision making, professional development and alignment with state and 

district reform efforts and priorities.

HOW WILL THE RESULTS BE USED?

The data from this survey will be used by the U.S. Department of Education and Congress to 

evaluate the implementation and effectiveness of federal school reform legislation. Responses to 

this data collection will be used only for statistical purposes. The reports prepared for this study 

will summarize findings across the sample and will not associate responses with a specific 

district or individual. We will not provide information that identifies you or your district to 

anyone outside the study team, except as may be required by law.

WHY SHOULD YOU PARTICIPATE IN THIS SURVEY?

We are conducting this survey with only a sample of schools. Therefore, the value of your 

individual contribution is greatly increased because it represents many other schools. We 

encourage you to participate in this voluntary survey.

WHERE SHOULD YOU MAIL YOUR COMPLETED QUESTIONNAIRE?

Please return your completed questionnaire in the enclosed envelope. If you do not have the 

return envelope, you can mail the survey to the address below, or contact the study team via 

email or using our toll-free number provided below. 

Longitudinal Assessment of Comprehensive School Reform

Attn: Susan Cragle

55 Hanover Lane 

Chico, CA 95973

1-866-880-2773

scragle@duerrevaluation.com

THANK YOU FOR YOUR COOPERATION IN THIS IMPORTANT EFFORT!
Appendix B 100 Principal Survey Year 5

Grade-Level and Content Planning

1. On average during the last 12 months, how many hours per week 

have you spent with others in common planning time? (Write in the 

number of hours) hours per week

2. Who participates in these planning activities? (Circle one number in each row)

Never

Once or a 

few times 

a year

Once or 

twice a 

month

Once or 

twice a 

week

Daily or 

almost 

daily

a. Classroom teachers 1 2 3 4 5

b. Teachers on special assignment 1 2 3 4 5

c. Curriculum coaches 1 2 3 4 5

d. School administrators 1 2 3 4 5

e. Teachers from other schools 1 2 3 4 5

f. District officials 1 2 3 4 5

g. State officials 1 2 3 4 5

h. Reform program developers 1 2 3 4 5

i. Other (specify) ________________________ 1 2 3 4 5

Coordinated Curriculum and Instruction

3. How much does each of the following influence what is taught in the classroom? (Circle one 

number in each row)

Strong 

negative 

influence

Somewhat 

negative 

influence

Little or 

no 

influence

Somewhat 

positive 

influence

Strong 

positive 

influence

a. State curriculum frameworks or content 

standards 1 2 3 4 5

b. District curriculum frameworks or content 

standards 1 2 3 4 5

c. Textbook/instructional materials 1 2 3 4 5

d. State testing requirements 1 2 3 4 5

e. District testing requirements 1 2 3 4 5

f. Students’ special needs or individual 

instructional plans 1 2 3 4 5

g. Parent or community preferences 1 2 3 4 5

h. Professional development experiences 1 2 3 4 5

i. Diagnostic or classroom assessment results 1 2 3 4 5
Appendix B 101 Principal Survey Year 5

4. Which entities – state, district or school officials – are 

involved in decision-making for each of the following? (mark all that apply)

State District School

a. Selecting textbooks and other instructional materials

b. Selecting content, topics, skills to be taught

c. Selecting teaching techniques

d. Creating student ability groups for instruction

e. Allocating instructional time for each academic subject

f. Determining the content of in-service professional development 

in which teachers participate

g. Hiring new full-time teachers

h. Planning school budgets

i. Determining professional and teaching assignments

j. Establishing the school curriculum

k. Other (specify) _____________________________

5. Has your school received any of the following designed to align 

curriculum and instruction with state and/or district content 

standards? (mark all that apply)

a. Detailed standards-based curriculum guides

b. Detailed pacing schedules

c. Classroom-embedded assessments with a standard scoring rubric to monitor 

progress toward district or state standards

d. Curriculum map aligning the required curricula and instructional programs to 

standards

e. Curriculum map aligning the required curricula and instructional programs to 

assessments

f. Model lesson plans based on the standards

g. Analytic reports summarizing student achievement data to identify specific 

strengths and weaknesses related to the attainment of standards

h. Information on listservs related to standards

i. Professional development designed to ensure that curriculum and instruction are 

consistent with state and/or district content standards

j. Professional development designed to ensure that curriculum and instruction are 

consistent with state and/or district assessments

k. Other (specify) _____________________________
Appendix B 102 Principal Survey Year 5

Data Driven Decision Making

6. Does your school receive reports or a summary of assessment results 

from state or district tests? (mark one response)

a. Yes

b. No

7. Does your school receive any of the following to support the use of 

assessment results? (mark all that apply)

a. Test results disaggregated by classroom

b. Test results disaggregated for special populations of students

c. Test results disaggregated by grade

d. Test results disaggregated by sub-test

e. Item-by-item review of test results

f. School reports showing trends over three years or more

g. Training and consultation for school staff to allow school-based analysis of 

student achievement data

8. Did you receive state or district assessment results summarized by 

any of the following categories of students? (mark all that apply)

a. Race or ethnicity

b. Gender

c. Title I participation

d. Migrant status

e. Poverty status

f. Limited English Proficiency (LEP) status

g. Students Individualized Education Programs (IEPs)

9. Who at your school receives such reports or summaries of 

assessment results? (mark all that apply)

a. Principals/administrators

b. Teachers

c. Parents

d. Students
Appendix B 103 Principal Survey Year 5

10. Have you received any assistance or training on how to interpret or 

analyze state or district assessment results? (mark all that apply)

a. Yes, from the state

b. Yes, from the district

c. Yes, from another external entity

d. No assistance or training on interpreting or analyzing assessment results

11. Which data sources are available to you and your school for student 

assessment or planning purposes? (mark all that apply)

a. “Raw” student performance data on state or district tests

b. Reports summarizing student performance on state or district tests

c. Reports that analyze or interpret student performance on state or district tests 

based upon demographic characteristics (such as race or gender) or educational 

program (such as results for students with special needs)

d. Reviews or evaluations of program implementation by external entity

e. Diagnostic tests to measure student readiness

f. Classroom observations by peers or administrators

g. Reviews of student projects or portfolios

h. Surveys of students, parents, teachers or staff regarding the school climate and 

satisfaction

i. Research reports from external resources on the efficacy of improvement 

strategies

j. Research conducted at the school (e.g., teacher or administrator action research 

projects; staff analysis of school data) on the efficacy of current improvement 

strategies

k. Tests of student knowledge or skills at the end of instructional units

l. Tests of student knowledge and skills at regularly scheduled times (e.g., 

quarterly or every six weeks)
Appendix B 104 Principal Survey Year 5

12. To what extent have state or district assessment results 

been used for the following purposes? (circle one number per row)

Little or no 

use

Moderate 

use

Great use

a. Measuring student progress toward meeting state or district 

standards or benchmarks 1 2 3

b. Identifying students for intervention or support services 1 2 3

c. Placement of students into remedial or advanced courses 1 2 3

d. Identifying areas where more professional development is 

needed 1 2 3

e. Assessing teacher performance 1 2 3

f. Helping teachers align their instruction to the school curriculum 1 2 3

g. Establishing priorities for future instruction (e.g., more 

emphasis on subjects or topics where students scored low) 1 2 3

h. Revising the school improvement plan (SIP) 1 2 3

i. Evaluating school initiatives/programs 1 2 3

School Improvement Plans

13. How formal is the school improvement plan at your school?

(mark one response)

a. Comprehensive written plan

b. Outline of a plan

c. Written mission statement only

d. No formal plan but teachers generally share the same ideas

e. Multiple plans

f. No plan (Skip to question 18)

14. Does your school improvement plan cover the following?

(mark all that apply)

a. Measurable goals or objectives

b. Mechanism for periodic evaluation of goals

c. Curriculum and instruction content

d. Student assessment rubrics

e. Classroom management guidelines

f. Professional development activities

g. Parental involvement plan

h. Framework for participation in school management

i. Integration of new technology

j. Other (specify) _____________________________
Appendix B 105 Principal Survey Year 5

15. What factors influenced the content of your formal school 

improvement plan? (mark all that apply)

a. State or district content standards

b. State or district performance standards

c. Needs identified through a school needs assessment

d. School performance standards

e. Specifications of adopted/adapted reform design

f. Assigned by district/state

g. Other (specify) _____________________________

16. Which entities – state, district or school officials – are 

involved in decision-making for each of the following? (mark all that apply)

State District School

a. Developing the school improvement plan

b. Reexamining the school improvement plan

c. Providing data to review for the school improvement plan (e.g., 

standardized test data)

d. Provide training on developing and/or reexamining the school 

improvement plan

e. Establishing goals and benchmarks for the school

17. How often does your school review or monitor progress toward the 

school improvement plan’s goals? (mark one response)

a. Never

b. Once every two years

c. Annually

d. Two or three times a year

e. Quarterly

f. Monthly

g. Other (specify) _____________________________
Appendix B 106 Principal Survey Year 5

Professional Development

The question in this section refers to all activities intended to help teachers develop and improve their 

content knowledge and classroom instruction. Examples of professional development activities include 

mentoring programs and coaching in addition to more traditional activities such as internships, 

workshops, conferences, institutes, and college courses. These may be conducted within the school or 

outside the school setting.

18. How is professional development organized at your school? Include 

only workshop, coursework, and conferences sponsored by your 

school, district, or state during the last 12 months. (mark all that apply)

a. Teachers select activities sponsored by the district or other organizations

b. The district identifies common professional development themes for all teachers

c. The school reform plan includes professional development activities for all 

teachers

d. All teachers participate in the same professional development activities

e. Each teacher has an individual professional development plan

Adoption of Scientifically Based Methods

19. Is your school using a school reform strategy or any combination of 

school reform strategies? (mark one response)

a. Yes

b. No

c. Not sure

20.

Please name the most important reform strategy or strategies in use at your school. 

21. In what year did your school begin implementing the most 

important reform model or strategy? year
Appendix B 107 Principal Survey Year 5

22.

At which grade levels in your school is reform mainly focused?

K 1 2 3 4 5 6 7 8 9 10 11 12 (mark all that apply)

23.

How is reform at your school designed? (mark one response)

a. Totally designed at this school

b. Adapted with modifications from external source

c. Adapted selected parts from multiple external sources

d. Adopted unmodified from external source

24.

Indicate the primary designer for the reform efforts at your school: (mark one response)

a. Locally developed

b. School district

c. University

d. Private developer or publisher

e. Other (specify) _____________________________

25.

Which categories best describe the focus of reform efforts at your 

school?

(mark all that apply)

a. Whole-school

b. Reading/language arts

c. Mathematics

d. Science

e. Social studies

f. Arts

g. Other (specify) _____________________________
Appendix B 108 Principal Survey Year 5

26.

Who is responsible for selecting the school reform in which your 

school participates?

(mark all that apply)

a. School board

b. District central office

c. School administrators

d. School improvement team

e. Teachers

f. Parents

g. Other (specify) _____________________________

27.

Which of the following factors contribute to the selection of 

reform at your school?

(mark all that apply)

a. Cost of reform

b. Theoretical or research foundation

c. Successful implementation at other schools

d. “Fit” with school needs

e. State or district mandate

f. Compatibility with assessment tools

g. Best published results

h. Don’t know

i. Other (specify) _____________________________
Appendix B 109 Principal Survey Year 5

Capacity and Support for Reform

The following questions refer to the most important reform strategy or strategies in your school. Such 

reform strategies are those intended to improve school operations and student outcomes.

28.

Which of these external entities is primarily responsible for 

supporting reform efforts at your school?

(mark one response)

a. University

b. Regional Education Laboratory

c. State agency

d. School district

e. Reform program developer

f. Teachers from other schools

g. Non-affiliated consultants

h. State-designated assistance providers

i. Other (specify) _____________________________

j. None, our school does not use external support

29.

What kind of assistance does this primary external entity provide 

to support reform efforts at your school? (mark all that apply)

a. Onsite consulting

b. Professional development

c. Networking opportunities

d. Written materials for students

e. Written materials for teachers

f. Software or technological assistance

g. Other (specify) _____________________________

h. None, our school does not use external support

30.

Was there a vote at your school to adopt or implement the current 

reform at your school? If so, who participated in the voting? (mark all that apply)

a. Administrators 

b. Teachers 

c. Students 

d. Parents 

e. None, there was no vote on the reform
Appendix B 110 Principal Survey Year 5

31.

Do you willingly participate in reform-related activities at your 

school?

(mark one response)

a. Yes 

b. No 

32. To what extent do you believe reform at your school: (Circle one number in each row)

Not at all

To a 

small 

extent

To some 

extent

To a 

fairly 

large 

extent

To a

great 

extent

a. Helps improve student learning 1 2 3 4 5

b. Helps teachers to teach more effectively 1 2 3 4 5

c. Detracts from more important efforts 1 2 3 4 5

d. Improves communication among teachers 1 2 3 4 5

e. Helps to improve student behavior 1 2 3 4 5

33.

What factors limit teacher participation in reform at your school? (mark all that apply)

a. No limitation – all teachers participate at this school

b. Our reform is subject-specific – only teachers in a specific subject participate

c. Our funding limits the number of teachers who can participate

d. Our school reform is being phased in over time

e. Other reason (specify) _____________________________
Appendix B 111 Principal Survey Year 5

34. To what extent do you disagree or agree with the following statements about the academic 

program at your school? (Circle one number in each row)

Strongly 

disagree Disagree To some extent Agree Strongly agree

a. At this school, we have a common

understanding of the objectives we’re trying to 

achieve with students

1 2 3 4 5

b. Goals and priorities for the school are clear 1 2 3 4 5

c. Most teachers at this school have values and 

philosophies of education similar to my own 1 2 3 4 5

d. Most of my colleagues share my beliefs and 

values about what the central mission of the 

school should be

1 2 3 4 5

e. Most staff in this school share a focused 

vision for student learning 1 2 3 4 5

f. Teachers are committed to developing 

partnerships with parent(s)/guardian(s) for 

student learning

1 2 3 4 5

g. The school encourages parents to be involved 

in defining the school mission and goals 1 2 3 4 5

35. To what extent does your school have the resources 

needed to implement the major elements of the reform 

at your school? (circle one number per row)

Not 

available

Somewhat 

available

Definitely 

available

a. Design team materials to support instruction 1 2 3

b. Availability of design team materials to further implement the 

design in your school 1 2 3

c. Professional development for teachers 1 2 3

d. Time for planning, collaboration, and development 1 2 3

e. Staff or consultants to mentor, advise, and provide ongoing 

support

1 2 3

f. Technology and connectivity 1 2 3

g. Funds 1 2 3

h. Funding flexibility 1 2 3

i. Support from staff 1 2 3
Appendix B 112 Principal Survey Year 5

Background

36. How many years have you been a principal? years

37. How many years have you been a principal at this school? years

THANK YOU FOR PARTICIPATING!

Please return your completed questionnaire in the enclosed envelope. If you do

not have the return envelope, you can mail the survey to the address below, or contact the study 

team via email or using our toll-free number provided below. 

Longitudinal Assessment of Comprehensive School Reform

Attn: Susan Cragle

55 Hanover Lane 

Chico, CA 95973

1-866-880-2773

csr@duerrevaluation.com
Appendix B 113 Teacher Inventory Year 1

Teacher Inventory Year 1
Appendix B 114 Teacher Inventory Year 1
Appendix B 115 Teacher Inventory Year 1
Appendix B 116 Teacher Inventory Year 1
Appendix B 117 Teacher Inventory Year 1
Appendix B 118 Teacher Inventory Year 1
Appendix B 119 Teacher Inventory Year 1
Appendix B 120 Teacher Inventory Year 1
Appendix B 121 Teacher Inventory Year 1
Appendix B 122 Teacher Inventory Year 1
Appendix B 123 Teacher Inventory Year 1
Appendix B 124 Teacher Inventory Year 1
Appendix B 125 Teacher Inventory Year 1
Appendix B 126 Teacher Inventory Year 1
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 127 Teacher Survey Year 2

Teacher Survey Year 2
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 128 Teacher Survey Year 2
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 129 Teacher Survey Year 2
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 130 Teacher Survey Year 2
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 131 Teacher Survey Year 2
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 132 Teacher Survey Year 2
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 133 Teacher Survey Year 2
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 134 Teacher Survey Year 2
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 135 Teacher Survey Year 2
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 136 Teacher Survey Year 2
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 137 Teacher Survey Year 5

Teacher Survey Year 5

Longitudinal Assessment of 

Comprehensive School 

Reform 

TEACHER SURVEY 

2006-07 School Year

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless 

such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1875-

0222. The time required to complete this information collection is estimated to average 40 minutes per response, including the 

time to review instructions, search existing data resources, gather the data needed, and complete and review the information 

collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, 

please write to: U.S. Department of Education, Washington, D.C. 20202-4651. If you have comments or concerns about the 

contents of this questionnaire, write directly to: 

WestEd, attn: John Flaherty, Project Coordinator, 4665 Lampson Avenue, Los Alamitos, CA 90720
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 138 Teacher Survey Year 5
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 139 Teacher Survey Year 5

Dear Teacher,

WHO IS CONDUCTING THIS SURVEY?

WestEd and Cosmos Corporation, educational research organizations under contract to the U.S. 

Department of Education, request your participation in this survey. WestEd and the Department 

of Education are conducting this survey by the authority of Section 1607 of the No Child Left 

Behind Act of 2001 (P.L. 107-110).

WHAT IS THE PURPOSE OF THIS SURVEY?

This survey is designed to understand how comprehensive reform at your school changes the 

ways that you approach teaching and learning. This survey focuses on changes that can affect 

every facet of school operations including grade-level planning, curriculum and instruction, data 

driven decision making, professional development and alignment with state and district reform 

efforts and priorities.

HOW WILL THE RESULTS BE USED?

The data from this survey will be used by the U.S. Department of Education and Congress to 

evaluate the implementation and effectiveness of federal school reform legislation. Responses to 

this data collection will be used only for statistical purposes. The reports prepared for this study 

will summarize findings across the sample and will not associate responses with a specific 

district or individual. We will not provide information that identifies you or your district to 

anyone outside the study team, except as may be required by law.

WHY SHOULD YOU PARTICIPATE IN THIS SURVEY?

We are conducting this survey with only a sample of schools. Therefore, the value of your 

individual contribution is greatly increased because it represents many other schools. We 

encourage you to participate in this voluntary survey.

WHERE SHOULD YOU MAIL YOUR COMPLETED QUESTIONNAIRE?

Please return your completed questionnaire in the enclosed envelope. If you do not have the 

return envelope, you can mail the survey to the address below, or contact the study team via 

email or using our toll-free number provided below. 

Longitudinal Assessment of Comprehensive School Reform

Attn: Susan Cragle

55 Hanover Lane 

Chico, CA 95973

1-866-880-2773

scragle@duerrevaluation.com

THANK YOU FOR YOUR COOPERATION IN THIS IMPORTANT EFFORT!
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 140 Teacher Survey Year 5

Grade-Level and Content Planning

1. On average during the current school year, how many hours per 

week have you spent with others in common planning time? (Write 

in the number of hours) hours per week

2. During the last 12 months, how frequently have you met formally or informally with others 

at your school to conduct the following activities related to planning and teaching? (Circle one 

number in each row)

Never

Once or a 

few times 

a year

Once or 

twice a 

month

Once or 

twice a 

week

Daily or 

almost 

daily

a. Participate in common planning period for 

teachers grouped by grade level 1 2 3 4 5

b. Participate in common period for teachers 

grouped by content area 1 2 3 4 5

c. Receive training or guidance on the role or 

purpose of common planning periods 1 2 3 4 5

d. Consult with other teachers about individual 

students (e.g., discussing specific students and 

arranging appropriate help)

1 2 3 4 5

e. Exchange feedback with other teachers based 

on classroom observations (e.g., a teacher’s 

observation of your class, your observation of 

another teacher’s class, or observation of a 

class via video)

1 2 3 4 5

f. Exchange feedback with other teachers based 

on student work 1 2 3 4 5

g. Act as a formal or informal coach or mentor to 

other teachers or staff 1 2 3 4 5

h. Receive formal or informal coaching or 

mentoring from other teachers or staff 1 2 3 4 5

i. Be observed/evaluated by the school principal 

or other staff (e.g., department chair, master 

teacher)

1 2 3 4 5

j. Participate in a learning community (e.g., 

teacher collaborative, network, or study 

group)

1 2 3 4 5

k. Participate in a district or school committee 

focused on curriculum, instruction, or student 

assessment

1 2 3 4 5

l. Visit other schools to observe classroom 

teaching and learning 1 2 3 4 5
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 141 Teacher Survey Year 5

3. Who participates in these planning activities? (Circle one number in each row)

Never

Once or a 

few times 

a year

Once or 

twice a 

month

Once or 

twice a 

week

Daily or 

almost 

daily

a. Classroom teachers 1 2 3 4 5

b. Teachers on special assignment 1 2 3 4 5

c. Curriculum coaches 1 2 3 4 5

d. School administrators 1 2 3 4 5

e. Teachers from other schools 1 2 3 4 5

f. District officials 1 2 3 4 5

g. State officials 1 2 3 4 5

h. Reform program developers 1 2 3 4 5

i. Other (specify) ________________________ 1 2 3 4 5

Coordinated Curriculum and Instruction

4. How much does each of the following influence what is taught in the classroom? (Circle one 

number in each row)

Strong 

negative 

influence

Somewhat 

negative 

influence

Little or 

no 

influence

Somewhat 

positive 

influence

Strong 

positive 

influence

a. State curriculum frameworks or content 

standards 1 2 3 4 5

b. District curriculum frameworks or content 1 2 3 4 5

c. Textbook/instructional materials 1 2 3 4 5

d. State testing requirements 1 2 3 4 5

e. District testing requirements 1 2 3 4 5

f. Students’ special needs or individual 

instructional plans 1 2 3 4 5

g. Parent or community preferences 1 2 3 4 5

h. Professional development experiences 1 2 3 4 5

i. Diagnostic or classroom assessment results 1 2 3 4 5
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 142 Teacher Survey Year 5

5. Which entities – state, district or school officials – are 

involved in decision-making for each of the following? (mark all that apply)

State District School Don’t 

know

a. Selecting textbooks and other instructional materials

b. Selecting content, topics, skills to be taught

c. Selecting teaching techniques

d. Creating student ability groups for instruction

e. Allocating instructional time for each academic subject

f. Determining the content of in-service professional development 

in which teachers participate

g. Hiring new full-time teachers

h. Planning school budgets

i. Determining professional and teaching assignments

j. Establishing the school curriculum

k. Other (specify) _____________________________

6. Have you received any of the following designed to align curriculum 

and instruction with state and/or district content standards? (mark all that apply)

a. Detailed standards-based curriculum guides

b. Detailed pacing schedules

c. Classroom-embedded assessments with a standard scoring rubric to monitor 

progress toward district or state standards

d. Curriculum map aligning the required curricula and instructional programs to 

standards

e. Curriculum map aligning the required curricula and instructional programs to 

assessments

f. Model lesson plans based on the standards

g. Analytic reports summarizing student achievement data to identify specific 

strengths and weaknesses related to the attainment of standards

h. Information on listservs related to standards

i. Professional development designed to ensure that curriculum and instruction are 

consistent with state and/or district content standards

j. Professional development designed to ensure that curriculum and instruction are 

consistent with state and/or district assessments

k. Other (specify) _____________________________

Data Driven Decision Making

7. Do you receive reports or a summary of assessment results from 

state or district tests? (mark one response)

a. Yes

b. No
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 143 Teacher Survey Year 5

8. Do you receive any of the following to support the use of assessment 

results? (mark all that apply)

a. Test results disaggregated by classroom

b. Test results disaggregated for special populations of students

c. Test results disaggregated by grade

d. Test results disaggregated by sub-test

e. Item-by-item review of test results

f. School reports showing trends over three years or more

g. Training and consultation for school staff to allow school-based analysis of 

student achievement data

9. Did you receive state or district assessment results summarized by 

any of the following categories of students? (mark all that apply)

a. Race or ethnicity

b. Gender

c. Title I participation

d. Migrant status

e. Poverty status

f. Limited English Proficiency (LEP) status

g. Students Individualized Education Programs (IEPs)

10. Who at your school receives such reports or summaries of 

assessment results? (mark all that apply)

a. Principals/administrators

b. Teachers

c. Parents

d. Students

e. Don’t know

11. Have you received any assistance or training on how to interpret or 

analyze state or district assessment results? (mark all that apply)

a. Yes, from the state

b. Yes, from the district

c. Yes, from another external entity

d. No assistance or training on interpreting or analyzing assessment results
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 144 Teacher Survey Year 5

12. Which data sources are available to you and your school for student 

assessment or planning purposes? (mark all that apply)

a. “Raw” student performance data on state or district tests

b. Reports summarizing student performance on state or district tests

c. Reports that analyze or interpret student performance on state or district tests 

based upon demographic characteristics (such as race or gender) or educational 

program (such as results for students with special needs)

d. Reviews or evaluations of program implementation by external entity

e. Diagnostic tests to measure student readiness

f. Classroom observations by peers or administrators

g. Reviews of student projects or portfolios

h. Surveys of students, parents, teachers or staff regarding the school climate and 

satisfaction

i. Research reports from external resources on the efficacy of improvement 

strategies

j. Research conducted at the school (e.g., teacher or administrator action research 

projects; staff analysis of school data) on the efficacy of current improvement 

strategies

k. Tests of student knowledge or skills at the end of instructional units

l. Tests of student knowledge and skills at regularly scheduled times (e.g., 

quarterly or every six weeks)

(mark one response per row)

13. To what extent have state or district assessment results 

been used for the following purposes? Little 

or no 

use

Moderat

e use

Great 

use

Don’t 

know

a. Measuring student progress toward meeting state or district 

standards or benchmarks

b. Identifying students for intervention or support services

c. Placement of students into remedial or advanced courses 

d. Identifying areas where more professional development is 

needed

e. Assessing teacher performance

f. Helping teachers align their instruction to the school curriculum

g. Establishing priorities for future instruction (e.g., more 

emphasis on subjects or topics where students scored low)

h. Revising the school improvement plan (SIP)

i. Evaluating school initiatives/programs
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 145 Teacher Survey Year 5

School Improvement Plans

14. How formal is the school improvement plan at your school?

(mark all that apply)

a. Comprehensive written plan

b. Outline of a plan

c. Written mission statement only

d. No formal plan but teachers generally share the same ideas

e. Multiple plans

f. No plan (Skip to question 19)

15. Does your school improvement plan cover the following?

(mark all that apply)

a. Measurable goals or objectives

b. Mechanism for periodic evaluation of goals

c. Curriculum and instruction content

d. Student assessment rubrics

e. Classroom management guidelines

f. Professional development activities

g. Parental involvement plan

h. Framework for participation in school management

i. Integration of new technology

j. Other (specify) _____________________________

16. What factors influenced the content of your formal school 

improvement plan? (mark all that apply)

a. State or district content standards

b. State or district performance standards

c. Needs identified through a school needs assessment

d. School performance standards

e. Specifications of adopted/adapted reform design

f. Assigned by district/state

g. Other (specify) _____________________________

h. Don’t know
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 146 Teacher Survey Year 5

17. Which entities – state, district or school officials – are 

involved in decision-making for each of the following? (mark all that apply)

State District School Don’t 

know

a. Developing the school improvement plan

b. Reexamining the school improvement plan

c. Providing data to review for the school improvement plan (e.g., 

standardized test data)

d. Provide training on developing and/or reexamining the school 

improvement plan

e. Establishing goals and benchmarks for the school

18. How often does your school review or monitor progress toward the 

school improvement plan’s goals? (mark one response)

a. Never

b. Once every two years

c. Annually

d. Two or three times a year

e. Quarterly

f. Monthly

g. Other (specify) _____________________________

h. Don’t know

Professional Development

The questions in this section refer to all activities intended to help teachers develop and improve their 

content knowledge and classroom instruction. Examples of professional development activities include 

mentoring programs and coaching in addition to more traditional activities such as internships, 

workshops, conferences, institutes, and college courses. These may be conducted within the school or 

outside the school setting.

19. How is professional development organized at your school? Include 

only workshop, coursework, and conferences sponsored by your 

school, district, or state during the last 12 months. (mark all that apply)

a. Teachers select activities sponsored by the district or other organizations

b. The district identifies common professional development themes for all teachers

c. The school reform plan includes professional development activities for all 

teachers

d. All teachers participate in the same professional development activities

e. Each teacher has an individual professional development plan
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 147 Teacher Survey Year 5

20. During the last 12 months, how many of the following types of professional development 

activities did you participate in? (Circle one number in each row)

Type of Professional Development Activity None 1- 2 

activities

3 - 5 

activities

6 - 10 

activities

11 or 

more 

activities

a. Conferences lasting two days or longer 1 2 3 4 5

b. Institutes (i.e., an intensive course of 

instruction on a particular topic or set of 

topics) lasting two days or longer

1 2 3 4 5

c. A series of connected workshops lasting two 

days or longer 1 2 3 4 5

d. Workshops lasting one day or less 1 2 3 4 5

e. Courses for college credit 1 2 3 4 5

f. Internships 1 2 3 4 5

21.

Indicate the number of hours allocated for teacher professional development from each of 

the following sources during the last 12 months. Include only workshops, coursework, and 

conferences sponsored by your school, district, or state. (Circle one number in each row)

Source of Professional Development Time None 1- 5 

hours

6 - 24 

hours

25 – 40 

hours

41 - 80 

hours

More 

than 80 

hours

a. Pupil-free days 1 2 3 4 5 6

b. Release time 1 2 3 4 5 6

c. Faculty meeting time 1 2 3 4 5 6

d. Personal time (evenings and weekends) 1 2 3 4 5 6

e. Holidays (including summer vacation) 1 2 3 4 5 6
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 148 Teacher Survey Year 5

22.

During the last 12 months, about how many hours of professional development did you 

receive in each of the following areas? (Circle one number in each row)

Area of Professional Development None 1- 5 

hours

6 - 24 

hours

25 – 40 

hours

41 - 80 

hours

More 

than 80 

hours

a. Instructional strategies for teaching 

Reading/Language Arts/English 1 2 3 4 5 6

b. In-depth study of topics in Reading/Language 

Arts/English 1 2 3 4 5 6

c. Instructional strategies for teaching 

Mathematics 1 2 3 4 5 6

d. In-depth study of topics in Mathematics 1 2 3 4 5 6

e. Instructional strategies or in-depth study of 

topics in other academic subject (e.g., Science, 

Social Studies, Foreign language, etc.)

1 2 3 4 5 6

f. Instructional strategies for students with 

Limited English Proficiency (LEP) 1 2 3 4 5 6

g. Instructional strategies for students with 

Individualized Education Programs (IEPs) 1 2 3 4 5 6

h. Preparing students to take the annual state 

assessment 1 2 3 4 5 6

i. Analyzing and interpreting student 

achievement data 1 2 3 4 5 6

j. Classroom and behavior management 1 2 3 4 5 6

k. Use of technology to improve classroom 

instruction 1 2 3 4 5 6

l. Use of appropriate assessment 

accommodations 1 2 3 4 5 6

m. Family/community involvement 1 2 3 4 5 6

n. Ensuring that curriculum and instruction are 

consistent with state and/or district content 

standards

1 2 3 4 5 6

o. Ensuring that curriculum and instruction are 

consistent with state and/or district 

assessments

1 2 3 4 5 6

p. Implementation of a school reform model 1 2 3 4 5 6

q. Monitoring individual students’ progress 

toward learning goals 1 2 3 4 5 6

r. Interpreting reports of student achievement 

data 1 2 3 4 5 6
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 149 Teacher Survey Year 5

23. Thinking of all of the different professional development activities 

that you participated in during the last 12 months, and reported in 

this section, approximately how many total hours were spent in 

professional development? (Write in the number of hours) hours

Adoption of Scientifically Based Methods

24. Is your school using a school reform strategy or any combination of 

school reform strategies? (mark one response)

a. Yes

b. No

c. Not sure

25.

Please name the most important reform strategy or strategies in use at your school. 

26. In what year did your school begin implementing the most 

important reform model or strategy? year

27.

At which grade levels in your school is reform mainly focused?

K 1 2 3 4 5 6 7 8 9 10 11 12 (mark all that apply)

28.

How is reform at your school designed? (mark one response)

a. Totally designed at this school

b. Adapted with modifications from external source

c. Adapted selected parts from multiple external sources

d. Adopted unmodified from external source
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 150 Teacher Survey Year 5

29.

Indicate the primary designer for the reform efforts at your school: (mark one response)

a. Locally developed

b. School district

c. University

d. Private developer or publisher

e. Other (specify) _____________________________

30.

Which categories best describe the focus of reform efforts at your 

school?

(mark all that apply)

a. Whole-school

b. Reading/Language Arts

c. Mathematics

d. Science

e. Social studies

f. Arts

g. Other (specify) _____________________________

31.

Who is responsible for selecting the school reform in which your 

school participates?

(mark all that apply)

a. School board

b. District central office

c. School administrators

d. School improvement team

e. Teachers

f. Parents

g. Other (specify) _____________________________

h. Don’t know
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 151 Teacher Survey Year 5

32.

Which of the following factors contribute to the selection of 

reform at your school?

(mark all that apply)

a. Cost of reform

b. Theoretical or research foundation

c. Successful implementation at other schools

d. “Fit” with school needs

e. State or district mandate

f. Compatibility with assessment tools

g. Best published results

h. Don’t know

i. Other (specify) _____________________________

Capacity and Support for Reform

The following questions refer to the most important reform strategy or strategies in your school. Such 

reform strategies are those intended to improve school operations and student outcomes.

33.

Which of these external entities is primarily responsible for 

supporting reform efforts at your school?

(mark one response)

a. University

b. Regional Education Laboratory

c. State agency

d. School district

e. Reform program developer

f. Teachers from other schools

g. Non-affiliated consultants

h. State-designated assistance providers

i. Other (specify) _____________________________

j. None, our school dos not use external support
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 152 Teacher Survey Year 5

34.

What kind of assistance does this primary external entity provide 

to support reform efforts at your school? (mark all that apply)

a. Onsite consulting

b. Professional development

c. Networking opportunities

d. Written materials for students

e. Written materials for teachers

f. Software or technological assistance

g. Other (specify) _____________________________

h. None, our school does not use external support

i. Don’t know

35.

Was there a vote at your school to adopt or implement the current 

reform at your school? If so, who participated in the voting? (mark all that apply)

a. Administrators 

b. Teachers 

c. Students 

d. Parents 

e. None, there was no vote on the reform

36.

Do you willingly participate in reform-related activities at your 

school?

(mark one response)

a. Yes 

b. No 

37. To what extent do you believe reform at your school: (Circle one number in each row)

Not at all

To a 

small 

extent

To some 

extent

To a 

fairly 

large 

extent

To a 

great 

extent

a. Helps improve student learning 1 2 3 4 5

b. Helps teachers to teach more effectively 1 2 3 4 5

c. Detracts from more important efforts 1 2 3 4 5

d. Improves communication among teachers 1 2 3 4 5

e. Helps to improve student behavior 1 2 3 4 5
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 153 Teacher Survey Year 5

38.

What factors limit teacher participation in reform at your school? (mark all that apply)

a. No limitations – all teachers participate at this school

b. Our reform is subject-specific – only teachers in a specific subject participate

c. Our funding limits the number of teachers who can participate

d. Our school reform is being phased in over time

e. Other reason (specify) _____________________________

39. To what extent do you disagree or agree with the following statements about the academic 

program at your school? (Circle one number in each row)

Strongly 

disagree Disagree To some extent Agree Strongly agree

a. At this school, we have a common 

understanding of the objectives we’re trying to 

achieve with students

1 2 3 4 5

b. Goals and priorities for the school are clear 1 2 3 4 5

c. Most teachers at this school have values and 

philosophies of education similar to my own 1 2 3 4 5

d. Most of my colleagues share my beliefs and 

values about what the central mission of the 

school should be

1 2 3 4 5

e. Most staff in this school share a focused 

vision for student learning 1 2 3 4 5

f. The principal communicates a clear academic 

vision for our school 1 2 3 4 5

g. Teachers are committed to developing 

partnerships with parent(s)/guardian(s) for 

student learning

1 2 3 4 5

h. The school encourages parents to be involved 

in defining the school mission and goals 1 2 3 4 5
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 154 Teacher Survey Year 5

40. To what extent does your school have the resources 

needed to implement the major elements of the reform 

at your school? (mark one response per row)

Not 

available

Somewhat 

available

Definitely 

available

a. Design team materials to support instruction

b. Availability of design team materials to further implement the 

design in your school

c. Professional development for teachers

d. Time for planning, collaboration, and development

e. Staff or consultants to mentor, advise, and provide ongoing 

support

f. Technology and connectivity

g. Funds

h. Funding flexibility

i. Support from staff

Background

41. How many years have you been a teacher? years

42. How many years have you been a teacher at this school? years

43.

What type of credential(s) do you hold? (mark all that apply)

a. Professional 

b. Preliminary

c. Emergency

d. Substitute

e. Other 
OMB No. 1875-0222 Approval Expires 09/30/2007

Appendix B 155 Teacher Survey Year 5

44.

Which best describes your MAIN teaching assignment? (mark one response)

a. Self-contained classroom teacher

b. Specialist teacher (mark below your primary subject area assignment 

for this year)

English as a second language Science

Fine Arts Special Education

English/Language Arts Social Studies, History, 

Government

Mathematics Speech, Communication

Physical Education Writing Specialist

Reading Specialist Other 

________________________

c. Grade level (mark below your grade level assignment for this year)

K 1 2 3 4 5 6 7 8 9 10 11 12 (mark all that apply)

THANK YOU FOR PARTICIPATING!

Please return your completed questionnaire in the enclosed envelope. If you do

not have the return envelope, you can mail the survey to the address below, or contact the study 

team via email or using our toll-free number provided below. 

Longitudinal Assessment of Comprehensive School Reform

Attn: Susan Cragle

55 Hanover Lane 

Chico, CA 95973

1-866-880-2773

csr@duerrevaluation.com
OMB No. 1875

-0222 Approval Expires 09/30/2007

Appendix B 156 Teacher Survey Year 5
Appendix B 157 Field Study Protocol

Field Study Protocol
Appendix B 158 Field Study Protocol
Appendix B 159 Field Study Protocol
Appendix B 160 Field Study Protocol
Appendix B 161 Field Study Protocol
Appendix B 162 Field Study Protocol
Appendix B 163 Field Study Protocol
Appendix B 164 Field Study Protocol
Appendix B 165 Field Study Protocol
Appendix B 166 Field Study Protocol
Appendix B 167 Field Study Protocol
Appendix B 168 Field Study Protocol
Appendix B 169 Field Study Protocol
Appendix B 170 Field Study Protocol
Appendix B 171 Field Study Protocol
Appendix B 172 Field Study Protocol
Appendix B 173 Field Study Protocol
Appendix B 174 Field Study Protocol
Appendix B 175 Field Study Protocol
Appendix B 176 Field Study Protocol
Appendix B 177 Field Study Protocol
Appendix B 178 Field Study Protocol
Appendix B 179 Field Study Protocol

SCHOOL-LEVEL LOOSE ENDS 

FOR FIELD STUDY TEAMS FOR 2004–05

Topics

Protocol 

Item 

to be 

Referenced

1. Was the CSR grant scaled-up or was it implemented all at once at the beginning of the grant 

period? Variability in start-ups, Timing for CSR.

1.2

2. Where are the funds and who controls the funds? (Survey) That data could be collected during site 

visits in the principal and district interviews. “Where are the funds?” “Who makes decisions about 

how the CSR and other funds are used? [this could be addressed on the one-page principal process 

influence document or in principal’s interview]

10.1

3. Burkett mentioned Schoolwide Title I (in Fed Register on July 2, 2004) Field staff may need to 

know about it before going to the field.

2.1

4. Scope of Change (Maggie)—How demanding is the intervention, whether comprehensive or 

subject-based? Could be included in Component 1 (Research-based Method)

1.1

5. If principal change, does principal deviate from adopted method or facilitate CSR grant? (Principal 

behavior, not whether a change occurred or not. Burkett)

B

6. How to define PD? (Stephanie concerned about survey) SV protocol is complete. Might add across 

and among grade planning and training periods and structural PD, such as ongoing small learning 

communities to protocol. Expand definition of professional development (PD): The 

inventory/protocol addresses a wide variety of types of PD–school level. Site visitors should be 

reminded of the variety of topics that could be considered in PD, such as the number of hours spent in 

vertical and horizontal teacher planning time.

3.2.3

7. Effects of Voluntary Public School Choice (VPSC): Describe student mobility in your school. 

(Probe: Has VPSC been a factor in student mobility?)

ADD AS 

RIVAL

8. Burkett, “Ask teachers if they participate in decisions.” [Do in the Leadership Team meeting in the 

prototype field study agenda.]

5

9. How are CSR funds being spent? D

10. Has this school been identified as low-performing? If so, has it been sanctioned because of low 

performance?

A

11. Did the faculty formally vote to adopt the current reform in your school? 5.1

12. What opportunities do teachers have for making management decisions? B

13. Do teachers have a common planning period in the day? 3.5

14. Do any of the following occur in ;your school? (Probe: Looping, grouping by houses or families; 

student groups stay with teachers for more than one year; interdisciplinary teaching, and paired or 

teacher teaming)

School 

Operations

15. If a secondary school, what kind of student groups exist to achieve particular instructional goals? School 

Operations

16. Who organizes the scope and sequence of the curriculum at this school? B
Appendix B 180 Field Study Protocol
Appendix B 181 Field Study Protocol

TOPICS FOR DISTRICT AND STATE OFFICIALS

FOR FIELD STUDY TEAMS, 2004–05

District CSR Contact: 

1. Administration of CSR Program as a Whole: Define and update district involvement in 

CSR program (targeting and selection process, number of awardees, funding process, and goals 

for how whole-school improvement will be achieved). 

2. Technical Assistance and Implementation Support: Describe state technical assistance 

and other support provided to districts and schools that might be related to CSR. 

3. Relationships between CSR and District Initiatives. Describe the relationships, if any, 

between CSR and:

a) district reform initiatives (e.g., standards, curriculum requirements, use of 

research-based methods, new report card forms)

b) district administration of Title I and schoolwide Title I 

c) district requirements for PD

4. Relationships between CSR and NCLB and Other Federal and State Initiatives.

Describe the relationships, if any, between CSR and other initiatives, such as NCLB and public 

school choice (federal) and state accountability and reform (state)—e.g., any relationships 

between CSR and state standards and assessments, and the implications for CSR implementation. 

5. Monitoring and Evaluation: Describe whether or how the district monitors and 

evaluates CSR implementation and outcomes—if possible focusing specifically on the two site 

visit schools. 

State CSR Contact: 

1. Administration of CSR Program as a Whole: Define and update state involvement in 

CSR program (targeting and selection process, number of awardees, funding process, and goals 

for how whole-school improvement will be achieved). Describe state priorities for CSR.

2. Technical Assistance and Implementation Support: Describe state technical assistance 

and other support provided to districts and schools that might be related to CSR. 

3. Relationships between CSR and State Initiatives. Describe the relationships, if any, 

between CSR and:

a) state reform initiatives (e.g., standards, curriculum requirements, use of 

research-based methods)

b) state assessments and accountability initiatives 

c) state requirements for PD, teacher certification, or related policies

4. Relationships between CSR and NCLB and Other Federal Initiatives. Describe the 

relationships, if any, between CSR and other federal initiatives, such as NCLB and public school 

choice. 

5. Monitoring and Evaluation: Describe whether o how the state monitors and evaluates 

CSR implementation and outcomes—if possible focusing specifically on the two site visit 

schools. Obtain a copy of the state CSR evaluation report (there may be several such reports, 

each completed annually).
Appendix B 182 Field Study Protocol
Appendix B 183 Field Study Protocol
Appendix B 184 Field Study Protocol
Appendix B 185 Field Study Protocol
Appendix B 186 Field Study Protocol
Appendix B 187 Field Study Protocol