U . S . D E P A R T M E N T O F E D U C A T I O N

Evaluation of the Comprehensive School Reform 

Program Implementation and Outcomes

Third-Year Report

Evaluation of the

Comprehensive School Reform Program

Implementation and Outcomes

Third-Year Report

Martin Orland, WestEd

Brooke Connolly, WestEd

Tony Fong, WestEd

Lauren Davis Sosenko, WestEd

Naida C. Tushnet, WestEd

Robert K. Yin, COSMOS Corp.

Janeula M. Burt, COSMOS Corp.

Emily Warner, COSMOS Corp.

Prepared for:

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

2008
This report was prepared for the U.S. Department of Education under Contract Number ED01CO0129. Menahem

Herman served as the contracting officer’s representative. The content of this report does not necessarily reflect the

views or policies of the U.S. Department of Education, nor does the mention of commercial products or

organizations imply endorsements by the U.S. government. The inclusion of such information is for the reader’s

convenience and is not intended to endorse any views expressed, or products, programs, models or services offered.

U.S. Department of Education

Margaret Spellings

Secretary

Office of Planning, Evaluation and Policy Development

Bill Evers

Assistant Secretary

Policy and Program Studies Service

Alan Ginsburg

Director

Program and Analytic Studies Division

David Goodwin

Director

December 2008

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to

reprint this publication is not necessary, the suggested citation is: U.S. Department of Education, Office of Planning,

Evaluation and Policy Development, Policy and Program Studies Service, Evaluation of the Comprehensive School

Reform Program Implementation and Outcomes: Third-Year Report, Washington, D.C., 2008.

This report is available on the Department’s Web site at:

www.ed.gov/about/offices/list/opepd/ppss/reports.html#title.

On request, this publication is available in alternate formats, such as Braille, large print, or computer diskette. For

more information, please contact the Department’s Alternate Format Center at 202-260-0852 or 202-260-0818.
iii

CONTENTS

List of Exhibits..........................................................................................................................v

Preface......................................................................................................................................ix

Acknowledgments....................................................................................................................xi

Executive Summary ...............................................................................................................xiii

Background..........................................................................................................................xiv

Evaluation Design................................................................................................................xvi

Findings..............................................................................................................................xvii

Summary ............................................................................................................................xxii

I. Introduction...........................................................................................................................1

Relevant Past Research ...........................................................................................................3

Organization of the Third-year Report.....................................................................................5

II. Data Sources.........................................................................................................................7

School-level Achievement Measures.......................................................................................8

Survey of School Reform Activities......................................................................................18

Case Studies of School Reform Activities.............................................................................23

Study Limitations..................................................................................................................24

Evaluation Questions ............................................................................................................26

III. Overall Relationship Between CSR Award and Achievement .......................................27

Methodology Used to Assess the Relationship Between Receipt of a CSR Award and

Achievement.........................................................................................................................27

Findings................................................................................................................................31

Discussion ............................................................................................................................32

IV. The Comprehensiveness of CSR Implementation and Its

Relationship to Achievement ......................................................................................35

Measuring Implementation of the CSR Components.............................................................35

Findings................................................................................................................................47

Discussion ............................................................................................................................55
iv

V. CSR Schools’ Adoption of a Scientifically Based Research Model and

Its Relationship to Achievement .................................................................................57

Measures of the Scientific Research Base of CSR Models.....................................................58

Findings................................................................................................................................63

Discussion ............................................................................................................................69

VI. Conclusions at the End of Three Years............................................................................71

Summary of Findings............................................................................................................71

References ...............................................................................................................................73

Appendix A. CSRQ Center Quality Review Tool.................................................................77

Appendix B. Standard Error Tables for Analyses................................................................99

Standard Errors for Report Exhibits ......................................................................................99

Appendix C. Data Collection Instruments..........................................................................105

Appendixes References.........................................................................................................177
v

EXHIBITS

Exhibit E.1 Eleven Components of Comprehensive School Reform Described in the No

Child Left Behind Act............................................................................................xv

Exhibit E.2 Average Gains in Percent Proficient or Higher from 2003 to 2005 for CSR and

Non-CSR Schools..............................................................................................xviii

Exhibit E.3 Average Number of CSR Components Implemented by CSR and Non-CSR

Schools in 2003 and 2005 ....................................................................................xix

Exhibit E.4 Average Gains in Percent Proficient or Higher from 2002 to 2005, by Subject

Area and Strength of Scientific Research Base....................................................xxii

Exhibit 1 Eleven Components of Comprehensive School Reform Described in the No

Child Left Behind Act..............................................................................................2

Exhibit 2 Grade-level Tested and Achievement Measures Used to Compute Standardized

Achievement Scores for Elementary and Middle Schools in Mathematics and

Reading, by State and Year.....................................................................................9

Exhibit 3 Standard Deviations of Percent Proficient or Higher for 2005 Elementary and

Middle School Achievement, by State ..................................................................17

Exhibit 4 Average Baseline Standardized Achievement Measures of All 2002 CSR

Schools and of the CSR and Non-CSR Schools in the ECSRIO Sample................21

Exhibit 5 Average Demographic Measures in 2005 of All 2002 CSR Schools and of the

CSR and Non-CSR Schools in the ECSRIO Sample .............................................22

Exhibit 6 Differences Between the 2002 Cohort of Title I CSR Schools and Title I NonCSR Schools.........................................................................................................28

Exhibit 7 Differences Between the 2002 Cohort of Title I CSR Schools and Non-CSR

Title I Schools Chosen by Propensity Scoring: Schools with Elementary

Mathematics and Reading Achievement ...............................................................30

Exhibit 8 Differences Between the 2002 Cohort of Title I CSR Schools and Non-CSR

Title I Schools Chosen by Propensity Scoring: Schools with Middle School

Mathematics and Reading Achievement ...............................................................30

Exhibit 9 Changes in Standardized Assessment Scores in CSR and Non-CSR Schools

from 2002–03 to 2004–05.....................................................................................32

Exhibit 10 Means and Standard Deviations of Survey Items Used to Construct

Research-based Design and Evidence-based Practice Measure, 2003 and 2005.....37

Exhibit 11 Means and Standard Deviations of Survey Items Used to Construct

Comprehensive Planning-Classroom Measure, 2003 and 2005..............................38

Exhibit 12 Means and Standard Deviations of Survey Items Used to Construct

Comprehensive Planning-School Measure, 2003 and 2005 ...................................39

Exhibit 13 Means and Standard Deviations of Professional Development Item, 2003 and

2005 .....................................................................................................................40
vi

Exhibit 14 Means and Standard Deviations of Survey Items Used to Construct the Goals

and Benchmarks Measure, 2003 and 2005 ............................................................41

Exhibit 15 Means and Standard Deviations of Survey Items Used to Construct Faculty

Participation Measure, 2003 and 2005 ..................................................................42

Exhibit 16 Means and Standard Deviations of Survey Items Used to Construct the District

Support Measure, 2003 and 2005..........................................................................43

Exhibit 17 Means and Standard Deviations of Survey Items Used to Construct the Parental

Involvement Measure, 2003 and 2005...................................................................44

Exhibit 18 Means and Standard Deviations of Survey Items Used to Construct the External

Assistance Measure, 2003 and 2005......................................................................45

Exhibit 19 Means and Standard Deviations of Survey Items Used to Construct the

Evaluation Measure, 2003 and 2005 .....................................................................46

Exhibit 20 Means and Standard Deviations of Survey Items Used to Construct

Coordination of Resources Measure, 2003 and 2005.............................................47

Exhibit 21 Average Number of CSR Components Implemented by CSR and Non-CSR

Schools in 2003 and 2005 .....................................................................................48

Exhibit 22 Number of Components Implemented in CSR and Non-CSR Elementary

Schools in 2005 ....................................................................................................49

Exhibit 23 Number of Components Implemented in CSR and Non-CSR Middle Schools in

2005 .....................................................................................................................50

Exhibit 24 Differences in Achievement and Demographic Characteristics Between Schools

Included and Excluded in Analyses of the Relationships Between

Implementation and Achievement, 2002 and 2005................................................52

Exhibit 25 Differences in Standardized Mathematics and Reading Achievement Between

CSR and Non-CSR Elementary Schools Included and Excluded in Analyses of

the Relationships Between Implementation and Achievement, 2005 .....................53

Exhibit 26 Differences in Standardized Mathematics and Reading Achievement Between

CSR and Non-CSR Middle Schools Included and Excluded in Analyses of the

Relationships Between Implementation and Achievement, 2005...........................54

Exhibit 27 Relationships Between Number of CSR Components Implemented and

Achievement Gains from 2002 to 2005.................................................................55

Exhibit 28 Number and Percentage of Schools Using CSR Models Included in CSRQ

Reports.................................................................................................................60

Exhibit 29 Description of CSRQ Rating Scale for Strength of Research Base ........................62

Exhibit 30 Distribution of 2002 CSR Awardees Across CSRQ Evidence Ratings ..................63

Exhibit 31 Average Baseline Standardized Achievement Scores for CSR Awardees Used in

Scientifically Based Research Analyses by Evidence Category, 2002 ...................64
vii

Exhibit 32 Regression Coefficients Demonstrating the Relationships Between Scientific

Research Base and Achievement Gains from 2002 to 2005...................................68

Exhibit A.1 CSRQ Center Quality Review Tool ......................................................................79

Exhibit A.2 CSRQ Overall Causal Validity Mapping Rubrics..................................................95

Exhibit A.3 CSRQ Evidence of Positive Effects on Student Achievement ...............................97

Exhibit B.1 Standard Errors for Differences Between the 2002 Cohort of Title I CSR

Schools and Title I Non-CSR Schools...................................................................99

Exhibit B.2 Standard Errors for Differences Between the 2002 Cohort of Title I CSR

Schools and Non-CSR Title I Schools Chosen by Propensity Scoring: Schools

with Elementary Mathematics and Reading Achievement.....................................99

Exhibit B.3 Standard Errors for Differences Between the 2002 Cohort of Title I CSR

Schools and Non-CSR Title I Schools Chosen by Propensity Scoring: Schools

with Middle School Mathematics and Reading Achievement.............................. 100

Exhibit B.4 Standard Errors for Changes in Standardized Assessment Scores in CSR and

Non-CSR Schools from 2002–03 to 2004–05 ..................................................... 100

Exhibit B.5 Standard Errors for Average Number of CSR Components Implemented by

CSR and Non-CSR Elementary Schools in 2003 and 2005 ................................. 101

Exhibit B.6 Standard Errors for Average Number of CSR Components Implemented by

CSR and Non-CSR Middle Schools in 2003 and 2005 ........................................ 101

Exhibit B.7 Standard Errors for Differences in Achievement and Demographic

Characteristics Between Schools Included and Excluded in Analyses of the

Relationships Between Implementation and Achievement, 2002 and 2005.......... 101

Exhibit B.8 Standard Errors for Differences in Mathematics and Reading Achievement

Between CSR and Non-CSR Elementary Schools Included and Excluded

in Analyses of the Relationships Between Implementation and Achievement,

2005 ................................................................................................................... 102

Exhibit B.9 Standard Errors for Differences in Mathematics and Reading Achievement

Between CSR and Non-CSR Middle Schools Included and Excluded in

Analyses of the Relationships Between Implementation and Achievement,

2005 ................................................................................................................... 102

Exhibit B.10 Standard Errors for Relationship Between Number of Components Implemented

and Achievement Gains from 2002 to 2005 ........................................................ 103

Exhibit B.11 Standard Errors for the Relationships Between Scientific Research Base and

Achievement Gains from 2002 to 2005............................................................... 104
viii
Preface ix

PREFACE

This Third-Year Report from the Evaluation of the Comprehensive School Reform Program

Implementation and Outcomes (ECSRIO) presents findings about the relationship between

participation in the U.S. Department of Education’s Comprehensive School Reform (CSR)

Program and student achievement. It follows the study’s First-Year Report published in 2004.1

The Third-Year Report examines the CSR program throughout the country and its relationship

with gains in student achievement.

The CSR program was established as a demonstration program in 1998 and authorized as a full

program in 2002 as part of the No Child Left Behind Act (NCLB). It is one approach to help lowperforming K–12 public schools meet state performance standards. CSR emphasizes two major

concepts. First, the approach mandates that school reform should be comprehensive in nature,

strengthening all aspects of school operations—curriculum, instruction, professional

development, parental involvement, and school organization. Second, CSR should involve the

use of scientifically based research models—that is, models with evidence of effectiveness in

multiple settings.

In 2002, the U.S. Department of Education contracted with WestEd and COSMOS Corp. to

conduct a five-year study of the CSR program, ECSRIO, which involved a survey of 500 CSR

schools and 500 comparison schools, case studies of 30 sites, and analyses of student

achievement in all schools receiving CSR funding in 2002. This report presents the third-year

findings of this study.

 1 The Second-Year Report was an internal document to the U.S. Department of Education.

Acknowledgements xi

ACKNOWLEDGMENTS

We wish to thank many individuals who contributed to this report. We are especially grateful to

the state, district, and school survey respondents, as well as to all of the school or district staff

who planned or participated in site visits. We express our deep gratitude for your role in the

study and the data you provided, without which this study would not have been possible.

The information in this report was provided through the congressionally mandated Evaluation of

the Comprehensive School Reform Program Implementation and Outcomes (ECSRIO), which

was conducted by WestEd and COSMOS Corp. under contract to the U.S. Department of

Education. Naida Tushnet of WestEd and Robert Yin of COSMOS Corp. led ECSRIO, and

Susan Cragle from DUERR Associates coordinated the survey data collection.

Several individuals at the U.S. Department of Education provided report guidance and direction.

We would like to acknowledge the assistance of Alan Ginsburg, director, Policy and Program

Studies Service (PPSS), and David Goodwin, director, Program and Analytic Studies Division,

PPSS. In addition, Carol Chelemer, Menahem Herman, Aaron Neumann, and Dante Randazzo in

PPSS contributed to the direction of the report.

We would like to acknowledge the contributions and guidance of the members of our Technical

Working Group, including Carolyn Temple Adger, Geoffrey Borman, H. J. Green, Bryan Hassel,

Elsie Leak, Valerie Lee, Paul Ruiz, Jean Rutherford, Malik Stewart, Sam Stringfield, and Ken

Wong.

We also would like to thank Noel White, WestEd senior editor, and Colleen Montoya, WestEd

senior communicator, for their editorial assistance.

While we appreciate the assistance and support of all of these individuals, any errors in judgment

or fact are the responsibility of the authors.
Acknowledgements xii
Executive Summary xiii

EXECUTIVE SUMMARY

The Comprehensive School Reform (CSR) Program was established as a demonstration program

in 1998 and authorized as a full program in 2002 as part of the No Child Left Behind Act (NCLB).

It is one approach to help low-performing K–12 public schools meet state performance

standards. CSR emphasizes two major concepts. First, the approach mandates that school reform

should be comprehensive in nature, strengthening all aspects of school operations—curriculum,

instruction, professional development, parental involvement, and school organization. Second,

CSR should involve the use of scientifically based research models—that is, models with

evidence of effectiveness in multiple settings.

In 2002, the U.S. Department of Education contracted with WestEd and COSMOS Corp. to

conduct a five-year study of the CSR program, the Evaluation of the Comprehensive School

Reform Program Implementation and Outcomes (ECSRIO), which involved a survey of 500

CSR schools and 500 comparison schools, case studies of 30 sites, and analyses of student

achievement in all schools receiving CSR funding in 2002.

This report presents the third-year findings of this study and focuses on the relationship between

CSR award receipt and growth in student achievement and whether aspects of program

implementation are associated with higher CSR achievement outcomes. Overall, the findings

indicate that schools receiving CSR awards did no better in mathematics and reading

achievement than comparable schools not receiving CSR funding.

The report’s major findings, organized by research question, are:

Was receipt of a CSR award associated with improvements in school-level

mathematics and reading achievement?

 Receipt of a CSR award was not associated with achievement gains in

mathematics or reading achievement through the first three years of

award.2

Were schools that received CSR awards more likely to implement the legislatively

specified components of CSR than other schools?

 No, both CSR and non-CSR schools implemented an average of fewer

than four components in 2003 and fewer than five in 2005 at both the

elementary and middle school levels.

 2 Note that, in this report, all references to achievement gains refer to gains in school-level achievement

measures, such as the percent proficient or higher or the school’s mean scaled score. Readers are cautioned not

to infer that any gains on school-level metrics refer to absolute improvements in students’ achievement. See

Robinson (1950) for a discussion of ecological fallacies, in which inferences about individuals are mistakenly

made based on aggregate statistics.
Executive Summary xiv

Was fidelity of CSR implementation associated with gains in school-level

mathematics and reading achievement?

Two analyses address this question:

Comprehensiveness of implementation

 The comprehensiveness of implementation, as measured by the

number of CSR components implemented, was not related to

mathematics and reading achievement gains in CSR schools.

Adoption of models with a recognized scientific research base

 Only one-third of 2002 CSR awardees chose reform approaches with

recognized scientific research bases.

 Low-performing elementary schools that adopted models with stronger

evidence of effectiveness had gains in mathematics achievement that

were not found in higher-performing schools.

 Adoption of a CSR model independently determined to have had

limited scientific evidence of effectiveness was associated with higher

gains in middle school mathematics achievement in all CSR schools,

whether they were low-performing or not. There is also weaker

evidence that CSR middle schools that adopted models with limited

scientific evidence may have experienced gains in middle school

reading achievement relative to schools that adopted other models.

 There was weaker evidence that low-performing middle schools that

adopted models with moderate or higher bases of evidence showed

improvement in mathematics compared with schools using other

models.

 In no other instances was adoption of models with a scientific research

base related to achievement gains.

BACKGROUND

Originally funded as the Comprehensive School Reform Demonstration (CSRD) Program in

1998 with $145 million, the CSR program became part of NCLB in 2002 with funding of $310

million. In FY 2003 and FY 2004, Congress allocated $308 million for CSR, and in FY 2005,

$205 million. In FY 2006, funding was appropriated only for a clearinghouse to support

comprehensive school reform and not for school-based activities. Between 1998 and 2006,

nearly 7,000 schools nationwide received three-year awards to implement CSR models.

NCLB defines CSR as containing 11 components, which are assumed to work together as schools

undergo reform (Exhibit E.1).
Executive Summary xv

Exhibit E.1

Eleven Components of Comprehensive School Reform

Described in the No Child Left Behind Act

1. Proven methods and strategies for student learning, teaching, and school management that are based on

scientifically based research and effective practices, and have been replicated successfully in schools with

diverse characteristics.

2. Comprehensive design for effective school functioning, integrating instruction, assessment, classroom

management, and professional development and aligning these functions into a schoolwide reform plan

designed to enable all students to meet challenging state content and performance standards and address

needs identified through a school needs assessment.

3. Professional development. High-quality and continuous teacher and staff professional development and

training.

4. Measurable goals for student performance and benchmarks for meeting those goals.

5. Support from staff. Support from school faculty, administrators, and staff.

6. Support for staff. Support for school faculty, administrators, and staff (added in 2001).

7. Parent and community involvement. Meaningful involvement of parents and the local community in planning

and implementing school improvement activities.

8. External assistance. High-quality external support and assistance from a comprehensive school reform entity

(which may be a university) with experience in schoolwide reform and improvement.

9. Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved.

10. Coordination of resources. Identification of how other available resources (federal, state, local, and private)

help the school coordinate services to support and sustain the school reform.

11. Scientifically based research. Scientifically based research to significantly improve the academic achievement

of students participating in such programs as compared with students in schools who have not participated in

such programs; or strong evidence that such programs will significantly improve the academic achievement of

participating children (added in 2001).

Source: No Child Left Behind Act, Title I, Part F, Section 1606.

This evaluation, mandated by Section 1606 of NCLB, addresses four broad questions related to

the CSR program.

 How were CSR funds targeted?

 How was reform implemented in CSR and non-CSR schools?

 How did state and district conditions influence reform

implementation?

 How was reform related to student achievement outcomes?

ECSRIO examines implementation and outcomes of a cohort of CSR awardees that received

their initial awards in 2002. The First-Year Report (2004) addressed the first three questions. It

concluded that CSR funds primarily targeted low-performing, high-poverty elementary and

middle schools in rural and urban areas. Although both CSR and non-CSR schools were engaged

in reform, reform in CSR schools was more likely to include adoption of models and other

activities closely associated with externally developed models. Furthermore, states that

integrated CSR with their standards and testing programs provided more CSR implementation

support than those states that did not integrate CSR. Building from these findings, this ThirdYear Report answers three questions directly related to two of the mandated questions
Executive Summary xvi

above—how reform relates to student achievement outcomes and how reform is implemented in

CSR and non-CSR schools:

 Was receipt of a CSR award associated with improvements in schoollevel mathematics and reading achievement?

 Were schools that received CSR awards more likely to implement the

legislatively specified components of CSR than other schools?

 Was fidelity of CSR implementation associated with gains in schoollevel mathematics and reading achievement?

EVALUATION DESIGN

This study focuses on schools nationwide that received their initial CSR awards in 2002, the year

most of them began implementing CSR. Awards spanned three years, although some awardees

carried over funds for an additional year.

The evaluation includes four methodological approaches.

 Multivariate statistical analyses comparing all Title I CSR with Title I

non-CSR matched comparison schools to determine whether receiving

a CSR award was related to school-level achievement increases over

time.3

 Qualitative case study analyses of CSR reform implementation in 15

pairs of schools (15 CSR schools and 15 matched comparison schools)

to illustrate the ways CSR components were implemented in both CSR

and non-CSR schools.

 Quantitative descriptive analyses of CSR reform implementation from

a survey of principals and teachers in a random sample of 500 CSR

and 500 matched non-CSR comparison schools to relate the

comprehensiveness of implementation to achievement.4

 Multivariate statistical analyses of the universe of 2002 CSR awardees

to examine the relationship between scientific research-based model

adoption and achievement.

 3 The achievement analyses rely on standardized school-level achievement scores from the National

Longitudinal School-Level State Assessment Score Database (NLSLSASD) maintained by the American

Institutes for Research (AIR). The NLSLSASD includes school-level measures of student achievement for

nearly all public schools. (For more information, see http://www.schooldata.org/.)

4 Most CSR awards were given to elementary and middle schools. Consequently, the sample included too few

high schools for analysis.
Executive Summary xvii

Only a small number of high schools received CSR awards in 2002. Consequently, the sample

included too few high schools for analysis, so this report does not focus on them.5 This study

frequently found differences in CSR implementation and outcomes between elementary and

middle schools. These include differences in the relationships between achievement and award

receipt and component implementation. Such differences are possibly related to the different

structures in the two types of schools, with middle schools having subject-area departments and

elementary schools mainly teaching students in self-contained classrooms. These structural

differences may affect the nature of the reform approach each level of school adopts, as well as

how well reforms are implemented across all grades, subjects, and teachers. Consequently, the

report includes separate findings for elementary and middle schools.

Frequently, this study examined the differences between low-performing schools and the

remaining schools in the study. Low-performing schools, defined as those in bottom 25 percent

of achievement in 2001–02 (the baseline year in this study), are of particular interest to

policymakers given the recent focus on schools identified for restructuring under No Child Left

Behind.

FINDINGS

Was Receipt of a CSR Award Associated With Improvements in School-level

Mathematics and Reading Achievement?

Receipt of a CSR award was not associated with achievement gains in mathematics

or reading achievement through the first three years of award.

While CSR schools made small but statistically significant gains in elementary mathematics and

reading achievement during the time they were implementing their award, these changes were

not statistically different from achievement changes in comparison schools (Exhibit E.2). Over

the course of their three-year awards, CSR schools were no better at improving achievement than

Title I schools that were similar in demographics and achievement in the baseline year.

 5 For more information on the number of high schools versus elementary and middle schools that received

CSR awards, see Exhibit B-2 on page 57 of Longitudinal Assessment of Comprehensive School Reform

Program Implementation and Outcomes: First-Year Report (Tushnet, Flaherty, and Smith, 2004).
Executive Summary xviii

Exhibit E.2

Average Gains in Percent Proficient or Higher from 2003 to 2005 for CSR and Non-CSR

Schools

Average Gain in Percent Proficient

N CSR Schools non-CSR Schools Difference

Elementary

Mathematics 634 0.96+ 1.44** n.s.

Elementary

Reading 638 1.12* 1.12* n.s.

Middle School

Mathematics 318 n.s. 1.44** n.s.

Middle School

Reading 320 n.s. n.s. n.s.

Exhibit highlights: While CSR schools made small but statistically significant gains in

elementary mathematics and reading achievement during the time they were implementing their

award, these changes were not statistically different from achievement changes in non-CSR

schools.

Note: “n.s.” indicates not statistically significant; + p<.10; * p<.05; ** p<.01. Tests for the

statistical significance of achievement gains for CSR and non-CSR schools are t-tests to assess

whether the value is different from zero. The differences between CSR and non-CSR schools are

assessed through paired t-tests. Values for percent proficient or higher are computed by

multiplying the average standard deviation of the achievement measures by the statistically

significant estimates for the relationships between scientifically based research model adoption

and school-level achievement.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD).

Were Schools That Received CSR Awards More Likely to Implement the

Legislatively Specified Components of CSR Than Other Schools?

No, both CSR and non-CSR schools implemented an average of fewer than four

components in 2003 and fewer than five in 2005 at both the elementary and middle

school levels.

CSR awards were intended to stimulate the implementation of the 11 components identified in

NCLB. However, as shown in Exhibit E.3, by 2005, CSR schools, on average, implemented

fewer than five components after three years of their CSR awards. Furthermore, implementation

of these components at the elementary school level was nearly identical in both CSR and nonCSR schools. In 2003, CSR elementary schools reported implementing slightly more

components than non-CSR schools; this difference was statistically significant. However, this

difference was no longer statistically significant by 2005. No significant differences existed in

either 2003 or 2005 in the number of reform components implemented by CSR and non-CSR

middle schools. Both CSR and non-CSR schools increased the number of reform components

implemented from 2003 to 2005.
Executive Summary xix

Exhibit E.3

Average Number of CSR Components Implemented by CSR

and Non-CSR Schools in 2003 and 2005

Average Number of Components Implemented

Elementary School Middle School

School Type 2003 2005

Change

from 2003

to 2005 2003 2005

Change

from 2003

to 2005

CSR Schools 3.8 4.7 0.9** 2.7 4.0 1.3**

Non-CSR Schools 3.3 4.4 1.1** 3.1 4.4 1.3**

Difference Between CSR

Schools and Non-CSR Schools 0.5* 0.3 -0.2 0.4 0.4 0.0

Exhibit highlights: In 2003, CSR elementary schools reported implementing a somewhat

higher average number of components than non-CSR schools. No other significant differences

between CSR and non-CSR schools exist. Both CSR and non-CSR schools reported similar

increases between 2003 and 2005 in the number of components implemented.

Note: + p<.10; * p<.05; ** p<.01. N = 292 for CSR elementary schools; N = 304 for non-CSR

elementary schools; N = 128 for CSR and non-CSR middle schools.

Source: ECSRIO surveys.

Was Fidelity of CSR Implementation Associated With Gains in School-level

Mathematics and Reading Achievement?

The comprehensiveness of implementation was not related to mathematics and

reading achievement gains between 2002 and 2005.

While only one-third of schools implemented reform models with limited scientific

research backing of effectiveness, those doing so were likely to experience

achievement gains in middle school mathematics. There was also some evidence,

albeit weaker, that schools adopting models with limited scientific evidence may

have experienced achievement gains in middle school reading.

Additionally, low-performing schools that adopted CSR models with scientific

evidence of effectiveness did improve in elementary mathematics.

The CSR program is based on two key implementation concepts: Schools implementing reforms

that encompass the 11 components indicated in NCLB will have higher achievement than schools

that do not; and schools implementing reform models with a scientific research base will have

higher achievement than schools implementing models that lack a scientific research base.

Two different analyses, included in the next two sections, assess the relationship between

implementation and achievement. The first analyzes the relationship between the

comprehensiveness of CSR implementation, as measured by the number of components a school
Executive Summary xx

implemented, and achievement outcomes. The second analyzes the relationship between the

adoption of a scientifically based research model and achievement.

Comprehensiveness of CSR Reform Implementation and Achievement Outcomes

The comprehensiveness of CSR reform implementation, as measured by the number of CSR

components implemented, was not related to achievement gains in mathematics and reading at

the elementary or middle school levels. There were no statistically significant relationships

between the number of components implemented and achievement gains from 2002 to 2005.

This finding could be due to the overall low levels of implementation of the components; that is,

since most schools implemented relatively few of the components, it is difficult to detect

statistically a consistent relationship between component implementation and achievement gains.

Adoption of Scientifically Based Research Models and Achievement Outcomes

This evaluation used information from the Comprehensive School Reform Quality Center

(CSRQ) ratings6 of the scientific research base in commonly adopted reform models to assess the

extent to which the use of CSR approaches with such a base was associated with higher

achievement. The CSRQ reports were not available at the time that 2002 CSR awardees made

their model selections; however, schools may have used data from two earlier reviews of CSR

models: Herman (1999), and Borman, Hewes, Overman, and Brown (2003). The CSRQ reports

draw heavily from the data used in previous reviews of CSR models. Of the 40 ratings

conducted, 25 models were rated as having some evidence of effectiveness. CSRQ relies on the

number and quality of research studies of the models to develop its ratings. However, only onethird of CSR schools adopted models that later received ratings from CSRQ indicating that they

had a scientific research base. Most schools chose models that were not rated by CSRQ.

In general, the results demonstrate stronger relationships between the scientific research bases of

comprehensive school reform models and mathematics achievement gains compared to reading

achievement gains. Mathematics achievement improved in low-performing elementary schools,7

as well as in all middle schools (including low-performing schools), that adopted models with

scientific research bases. Mathematics achievement improved in low-performing CSR

elementary schools that adopted models with scientific research bases, while it did not improve

in non-low-performing CSR elementary schools (Exhibit E.4).8 Low-performing CSR

elementary schools gained about 2 percentage points in the percent of students proficient or

 6 The CSRQ ratings provide a scale for the breadth and quality of the research base for the 31 most widely

adopted CSR models in elementary, middle, and high schools. The CSRQ Center conducted a total of 40

ratings because some models were rated at both the elementary and secondary levels. The categories are Very

Strong, Moderately Strong, Moderate, Limited, Zero, and Negative. The CSRQ Center was funded under the

U.S. Department of Education’s Comprehensive School Reform Quality Initiatives program.

7 Low-performing schools refer to those CSR schools that were in the lowest 25 percent of achievement of all

CSR schools in 2001-02.

8 The CSRQ rating scale includes Very Strong, Moderately Strong, Moderate, Limited, Zero, and Negative

categories. No model received a “Negative” or “Very Strong” rating. The analyses that examine the

relationship between strength of the research base and achievement treat scientific research base as a series of

dummy coded variables for Limited and Moderate/Moderately Strong CSRQ ratings.
Executive Summary xxi

higher in mathematics if they adopted a model with a limited scientific research base and 4

percentage points if they adopted a model with a moderate or moderately strong scientific

research base. There was no relationship between changes in elementary reading achievement

and the selection of a model with a scientific research base.

At the middle school level, all CSR schools (including low-performing schools) that adopted

models with limited scientific research bases realized gains in the percent proficient or higher of

about 6 percentage points in mathematics and 3 percentage points in reading, although the

relationship with reading achievement is only weakly significant.9 Furthermore, low-performing

CSR middle schools that adopted a model with a moderate or higher research base had gains of

about 4 percentage points in the percent proficient or higher in mathematics, although this

relationship is also only weakly significant.

 9 Throughout this report, findings that are reported as weakly significant are statistically significant at p<.10.
Executive Summary xxii

Exhibit E.4

Average Gains in Percent Proficient or Higher from 2002 to 2005, by Subject Area and

Strength of Scientific Research Base

Average Gains in Percent Proficient or Higher

Elementary School Middle School

Mathematics Reading Mathematics Reading

All CSR Schools

Limited Scientific Research Base n.s. n.s. 5.92** 3.36+

Moderate/Moderately Strong

Research Base n.s. n.s. n.s. n.s.

Low-performing Schools

Limited Scientific Research Base 1.92** n.s. 5.92** 3.36+

Moderate/Moderately Strong

Research Base 4.00* n.s. 4.16+ n.s.

Exhibit highlights: At the elementary level, low-performing CSR schools that adopted models

with limited or moderate to moderately strong scientific research bases had higher mathematics

achievement; there was no relationship between reading achievement and models with a

scientific research base. At the middle school level, CSR schools that adopted models with

limited scientific research bases had higher mathematics and reading achievement gains in the

percent proficient or higher than those that did not; low-performing CSR schools that adopted

models with moderate to moderately strong research bases had higher mathematics

achievement than those that chose other models.

Note: n.s. indicates not statistically significant; + p<.10; * p<.05; ** p<.01. Values for percent

proficient or higher are computed by multiplying the average standard deviation of the

achievement measures by the statistically significant estimates for the relationships between

scientifically based research model adoption and school-level achievement.

Sources: CSRQ Center Report on Elementary CSR Models; CSRQ Center Report on Middle

and High School CSR Models; CSR Awards Database; Common Core of Data (CCD);

National Longitudinal School-Level State Assessment Score Database (NLSLSASD).

One possible explanation for these inconsistent findings lies with the way CSRQ determined

ratings. CSRQ determined “strength of the research base” by the nature of the evaluation design,

with more weight given to more rigorous designs and the number of models evaluated that use

such designs. It is possible that some highly potent strategies had not been evaluated rigorously,

so the ratings underplay their potential for effectiveness. It is also possible that implementation

in schools in this study may have been less comprehensive than it was in schools included in the

studies that CSRQ used.

SUMMARY

These third-year evaluation findings demonstrate both the complexity of implementing CSR and

a weak and uneven relationship between implementation and achievement. The relationship
Executive Summary xxiii

varies by content area (mathematics or reading) and by school level (elementary or middle

school). The findings indicate that as a reform strategy, comprehensive school reform grants

were not systematically associated with gains in either student reading or mathematics

achievement. It also indicates that relatively few schools implemented CSR as intended by the

legislation and that by the third year of implementation (2005), CSR reform components were

about equally likely to be present in non-CSR as in CSR schools.

CSR provided funds for schools to implement comprehensive reform using scientifically based

research strategies. However, relatively few of the schools adopted reforms backed by

scientifically based research. When they did, the findings provide preliminary evidence of the

advantage of adopting scientifically research-based models, although the pattern of findings was

not consistent across all school levels and subject areas.
Executive Summary xxiv
Chapter I 1

I. INTRODUCTION

The Comprehensive School Reform (CSR) program was established as a demonstration program

in 1998 and authorized as a full program in 2002 as part of the No Child Left Behind Act (NCLB).

It is one approach to help low-performing K–12 public schools meet state performance

standards. CSR emphasizes two major concepts. First, the approach mandates that school reform

should be comprehensive in nature, strengthening all aspects of school operations—curriculum,

instruction, professional development, parental involvement, and school organization. Second,

CSR involves the use of scientifically based research models—that is, models with evidence of

effectiveness in multiple settings.

Originally funded as the Comprehensive School Reform Demonstration (CSRD) Program in

1998 with $145 million, the program became part of NCLB in 2002 with funding of $310

million. In FY 2003 and FY 2004, Congress allocated $308 million for CSR, and in FY 2005,

$205 million. Between 1998 and 2006, nearly 7,000 schools nationwide received three-year

awards to implement CSR models. In FY 2006, funding was appropriated only for a

clearinghouse to support comprehensive school reform and not for school-based activities.

NCLB defines CSR as containing 11 components, which are assumed to work together as schools

undergo reform (Exhibit 1). For example, the “support from staff” component of CSR dictates

that the entire school should adopt the reform model, while the “support for staff” component

focuses on helping teachers and other staff members learn to use the reform strategy. This report

studies these components, their implementation, and their relationship to achievement.

In addition to implementing the 11 components, CSR schools are expected to use reform models

with a strong scientific research base. According to Hale (2000), the unique aspect of CSR is its

expectation that schools will collaborate with expert partners to implement research-based,

whole-school reform methods with a successful replication record. This report examines the

relationships between schools adopting models with strong scientific research bases and

achievement.
Chapter I 2

Exhibit 1

Eleven Components of Comprehensive School Reform

Described in the No Child Left Behind Act

1. Proven methods and strategies for student learning, teaching, and school management that are based on

scientifically based research and effective practices, and have been replicated successfully in schools with

diverse characteristics.

2. Comprehensive design for effective school functioning, integrating instruction, assessment, classroom

management, and professional development and aligning these functions into a schoolwide reform plan

designed to enable all students to meet challenging state content and performance standards and address

needs identified through a school needs assessment.

3. Professional development. High-quality and continuous teacher and staff professional development and

training.

4. Measurable goals for student performance and benchmarks for meeting those goals.

5. Support from staff. Support from school faculty, administrators, and staff.

6. Support for staff. Support for school faculty, administrators, and staff (added in 2001).

7. Parent and community involvement. Meaningful involvement of parents and the local community in planning

and implementing school improvement activities.

8. External assistance. High-quality external support and assistance from a comprehensive school reform entity

(which may be a university) with experience in schoolwide reform and improvement.

9. Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved.

10. Coordination of resources. Identification of how other available resources (federal, state, local, and private)

help the school coordinate services to support and sustain the school reform.

11. Scientifically based research. Scientifically based research to significantly improve the academic achievement

of students participating in such programs as compared with students in schools who have not participated in

such programs; or strong evidence that such programs will significantly improve the academic achievement of

participating children (added in 2001).

Source: No Child Left Behind Act, Title I, Part F, Section 1606.

In 2002, the U.S. Department of Education contracted with WestEd and COSMOS Corp. to

conduct a five-year study of the CSR program. The Evaluation of the Comprehensive School

Reform Program Implementation and Outcomes (ECSRIO), mandated by Section 1606 of

NCLB, addresses four broad questions related to the CSR program.

 How were CSR funds targeted?

 How was reform implemented in CSR and non-CSR schools?

 How did state and district conditions influence reform

implementation?

 How was reform related to achievement outcomes?

ECSRIO examines the implementation and outcomes of a cohort of CSR awardees that received

their initial awards in 2002. The First-Year Report10 (2004) addressed the first three questions. It

concluded CSR funds primarily targeted low-performing, high-poverty elementary and middle

schools in rural and urban areas. Although both CSR and non-CSR schools were engaged in

reform, reform in CSR schools included adoption of models and other activities closely

 10 The Second-Year Report was an internal document to the U.S. Department of Education.
Chapter I 3

associated with externally developed models. Furthermore, states that integrated CSR with their

standards and testing programs provided more support for CSR implementation than those

without this integration. Building from these findings, this Third-Year Report answers questions

directly related to two of the mandated questions—how reform related to achievement outcomes

and how reform was implemented in CSR and non-CSR schools:

 Was receipt of a CSR award associated with improvements in schoollevel mathematics and reading achievement?

 Were schools that receive CSR awards more likely to implement the

legislatively specified components of CSR than other schools?

 Was fidelity of CSR implementation associated with gains in schoollevel mathematics and reading achievement?

RELEVANT PAST RESEARCH

The remainder of this introduction reviews the findings from the first two years of the evaluation

and other related research efforts.

Review of Findings From the First and Second Years of the Study

The first- and second-year reports addressed the first three study questions listed on page 2. The

major findings, organized by study question, are:

How were CSR funds targeted?

States targeted CSR funds largely to high-poverty and low-performing urban and

rural elementary and middle schools.11 Most states helped these schools identify

appropriate research-based methods and with applications for grant awards.

How was reform implemented in CSR and non-CSR schools?

Both CSR and non-CSR schools exhibited many aspects of comprehensive reform.

However, CSR schools were more likely to adopt externally developed models than

non-CSR schools.

How did state and district conditions influence reform implementation?

In nearly three-fourths of the states (72 percent), state CSR coordinators saw the

program as a way to help schools meet state standards and succeed in state testing

programs.

 11 Because only a small number of high schools received CSR awards, these reports focus only on elementary

and middle schools.
Chapter I 4

States that aligned CSR with state standards and testing programs provided more

support for implementation than states without such alignment. When states used

CSR to advance statewide reform priorities, they more often targeted CSR funds to

specific sites, provided support for schools, and maintained higher accountability

standards. In 26 percent of states, coordinators indicated the absence of a state

reform agenda, thereby relying on federal standards or guidelines. Such states

provided less support and were less likely to monitor or evaluate CSR efforts than

other states.

Other Research Efforts and Relationship to ECSRIO

There are numerous studies of individual CSR models, such as Success for All (Borman et al.,

2005a, 2005b) and America’s Choice (May and Supovitz, 2006). In addition, Borman, Hewes,

and Overman (2003) conducted a meta-analysis of studies of 29 individual CSR models. Most

recently, the American Institutes for Research’s (AIR’s) National Longitudinal Evaluation of

Comprehensive School Reform (NLECSR) examined implementation and outcomes of eight

models over a six-year period. The following summarizes the NLECSR findings.

Aladjem et al. (2006) reported on a six-year mixed-method study of eight CSR models in 650

elementary and middle schools. NLECSR addressed questions about the relationship between the

characteristics of specific CSR models and individual student and school-level achievement.12

The 650 schools were a sample of schools that received a CSR award between 1999 and 2001

and implemented one of the eight most popular models. The NLECSR included survey data from

district administrators, principals, and teachers; achievement analyses from both CSR schools

and matched non-CSR comparison schools that did not receive a CSR award; and case studies of

34 schools.

Aladjem et al. (2006) measured implementation as the difference between what the model

developers consider to be full implementation and what the school actually does. To do so, they

surveyed each of the model developers and asked them to respond as if they were a school

implementing their model. The researchers compared each school’s responses to this measure of

implementation to develop a measure of fidelity. In order to measure implementation in the

comparison schools, Aladjem et al. predicted what CSR model the comparison school would

have chosen based on its school characteristics and those of the CSR schools. The researchers

then compared the school’s responses against those of the model developer of the predicted CSR

model. This measure of implementation is consistent with the focus on models as compared with

this study’s focus on the 11 components identified in NCLB.

CSR schools and their matched comparisons in NLECSR differed little in the level of

implementation. Furthermore, implementation was infrequently comprehensive in CSR schools,

with only about 20 percent of the schools in 2002 (and 10 percent in 2004) implementing a CSR

model comprehensively. However, CSR schools were more likely to implement CSR

comprehensively than their comparison schools. One predictor of the level of implementation

was the CSR model chosen, and this relationship did not change over time. That is,

 12 Student-level achievement data were available for a subset of five school districts during the study period.
Chapter I 5

implementation in schools with highly prescriptive models (e.g., Success for All) was higher

than in schools with other models.

In the Aladjem et al. (2006) study, CSR schools with higher levels of implementation had higher

levels of achievement gains than the matched comparison schools. This is particularly apparent

when implementation was high three to five years after initially receiving a CSR award, and

when high implementation was comprehensive (as opposed to focused on a few specific aspects

of the model). Furthermore, models matter in that schools that selected Success for All typically

had higher levels of implementation and achievement than schools that selected other models.

However, achievement results of several other models (Accelerated Schools Project, ATLAS

Communities, and Co-Nect) indicate that comprehensive implementation of these models

consistent with the model developers’ visions may result in higher achievement in the long term.

Zhang, Fashola, Shkolnik, and Boyle (2006) reanalyzed the NLECSR implementation and

achievement data and largely came to the same conclusions in the NLECSR final report. Because

some schools in the NLECSR study switched CSR models, and a number of comparison schools

adopted CSR models, Zhang et al. used a smaller sample of 115 schools. They analyzed the

relationship between implementation and achievement, using school-level mathematics and

reading scores. Achievement gains (relative to comparison schools) were largest in CSR schools

between years 3 and 5 in implementation. Zhang et al. also found the level of implementation is

marginally associated with gains in achievement and confirmed the NLECSR finding that

models matter, with similar patterns of results as those reported by AIR.

In contrast to these studies, ECSRIO focuses on how funds allocated through the CSR program

were used. Consequently, the schools in this report did not necessarily adopt a single model, and

the measure of implementation is different, focusing on fidelity to the 11 legislatively identified

components of reform rather than to a particular model. Similarly, the focus on how awards were

used by a representative sample of recipients allows policymakers to view the program as it was

actually implemented. As such, this study provides insight into the extent to which

comprehensive school reform prompts changes in school operations and student outcomes. This

focus on the program in operation limits any causal claims. However, it provides meaningful and

policy-relevant insights regarding the value and limitations of a broad program design in

stimulating school reform nationally.

ORGANIZATION OF THE THIRD-YEAR REPORT

The next chapter describes the data sources used in the ECSRIO. The subsequent three chapters

report on: the overall relationship between CSR award and achievement (Chapter III), the

comprehensiveness of CSR implementation and its relationship to achievement (Chapter IV),

and CSR schools’ adoption of scientifically based research models and its relationship to

achievement (Chapter V). These three chapters that report findings include discussions of

methodological approaches because each question relies on different methodologies and on

samples constructed in different manners. The final chapter presents a brief summary of key

findings. The report appendixes include the CSRQ Center Quality Review Tool (Appendix A),

standard error tables (Appendix B), and data collection instruments (Appendix C).
Chapter I 6
Chapter II 7

II. DATA SOURCES

This study compares implementation and school-level achievement between schools that

received CSR awards and similar schools that did not receive such awards. Comparisons were

done by employing multiple research methods for collecting and organizing data, and

incorporating information from teachers and principals about reform in their schools.

 Multivariate statistical analyses comparing all Title I CSR with Title I

non-CSR matched comparison schools to determine whether receiving

a CSR award was related to school-level achievement increases over

time.13

 Qualitative case study analyses of CSR reform implementation in 15

pairs of schools (15 CSR schools and 15 matched non-CSR

comparison schools) to illustrate the ways CSR components were

implemented in both CSR and non-CSR schools.

 Quantitative descriptive analyses of CSR reform implementation from

a survey of principals and teachers in a random sample of 500 CSR

and 500 matched non-CSR comparison schools to relate the

comprehensiveness of implementation to achievement.14

 Multivariate statistical analyses of the universe of 2002 CSR awardees

to examine the relationship between scientific research-based model

adoption and achievement.

Using multiple methods increases the validity of key measures and results. In addition, this is an

evaluation of a program as it operates in the field. Such an evaluation cannot control the

conditions under which activities take place. Consequently, the study cannot use an experimental

design, and no causal claims can be made. The evaluation uses a quasi-experimental design,

matching schools that received CSR funds with schools that did not receive funding, in both the

quantitative and qualitative aspects of data collection and analysis.

This section describes the samples and the three following data sources for ECSRIO.

• School-level achievement measures

• Surveys of school reform activities

• Case studies of school reform activities

 13 The achievement analyses rely on standardized school-level achievement scores from the National

Longitudinal School-Level State Assessment Score Database (NLSLSASD) maintained by the American

Institutes for Research (AIR). The NLSLSASD includes school-level measures of student achievement for

nearly all public schools. (For more information, see http://www.schooldata.org/.)

14 Most CSR awards were given to elementary and middle schools. Consequently, the sample included too few

high schools for analysis.
Chapter II 8

Because each evaluation question relies on different methods and samples constructed in

different ways, discussions of the specific methods used are included with the findings for each

evaluation question. The last part of this section presents study limitations.

SCHOOL-LEVEL ACHIEVEMENT MEASURES

The achievement analyses rely on standardized school-level achievement scores from the

National Longitudinal School-Level State Assessment Score Database (NLSLSASD) maintained

by the American Institutes for Research (AIR). The NLSLSASD includes school-level measures

of achievement for nearly all public schools. (For more information, see

http://www.schooldata.org/.) This database includes seven years of data, spanning 1998–99

through 2004–05; however, many states did not have or provide assessment data for the first year

or two. Also, prior to 2003–04, nearly all states only tested a few grades. Therefore, for this

study, one grade (typically fourth) was chosen to represent elementary achievement, and one

grade (typically eighth) was chosen to represent middle school achievement in the school.

Exhibit 2 details each state’s assessment measures used to compute the standardized achievement

scores by year.

In most instances, the average scale score or percent proficient or higher was used to calculate a

standardized z-score15 within each state for each year. In several instances, however, other

measures, such as percentile ranks, were the only available outcome measures and, as such,

standardization was done in the following manner:

1. In each year, appropriate assessment measures for mathematics and reading or language

arts in elementary and middle grades were selected.

2. In many cases, fourth-grade scores were used for elementary schools and eighth-grade

scores for middle schools; however, when those scores were not available, other

proximate grades were used.

3. The schools in each state in each year were ranked according to their achievement

measure.

4. Percentile ranks were computed from these rankings.

5. Percentile ranks were converted to normal curve equivalent (NCE) scores.

6. NCE scores were converted to z-scores, so each state was standardized with 0 mean and a

standard deviation of 1.

Where no assessment scores were available for a state in an academic year, the standardized

score was estimated by averaging the standardized score from the year prior and the year after.

For instance, there were no assessment measures available for Idaho in 2004. The elementary

 15 The z-score is calculated by subtracting the population mean from an individual score and dividing it by the

population standard deviation.
Chapter II 9

mathematics score for each school in Idaho in 2004 was computed by averaging the 2003 and

2005 standardized scores in elementary school mathematics for each school.

Exhibit 2

Grade-level Tested and Achievement Measures Used to Compute Standardized

Achievement Scores for Elementary and Middle Schools in Mathematics and

Reading, by State and Year16

Mathematics Reading

State Year Elem Middle Elem Middle Measure

2005 4 8 4 8 Percent proficient or above

2004 3 8 3 8 Percent proficient or above

2003 4 8 4 8 Percent proficient or above

2002 4 8 4 8 Percent proficient or above

2001 4 8 4 8 Percent proficient or above

Alaska

2000 4 8 4 8 Percent proficient or above

2005 4 6 4 8 Percent meets academic standards

2004 4 6 4 8 Percent meets academic standards

2003 4 8 4 8 Percentile rank

2002 4 8 4 8 Percentile rank

2001 4 8 4 8 Percentile rank

2000 4 8 4 8 Percentile rank

Alabama

1999 4 8 4 8 Percentile rank

2005 4 8 4 8 Percent proficient or above

2004 4 8 4 8 Percent proficient or above

2003 4 8 4 8 Percent proficient or above

2002 4 8 4 8 Percent proficient or above

2001 4 8 4 8 Percent proficient or above

Arkansas

2000 4 8 4 8 Percent proficient or above

2005 3 8 3 8 Mean Scale Score

2004 3 8 3 8 Mean Scale Score

2003 3 8 3 8 Mean Scale Score

2002 3 8 3 8 Mean Scale Score

2001 3 8 3 8 Mean Scale Score

Arizona

2000 3 8 3 8 Mean Scale Score

2005 4 8 4 8 Mean Scale Score

2004 4 8 4 8 Mean Scale Score

2003 4 8 4 8 Mean Scale Score

2002 4 8 4 8 Mean Scale Score

2001 4 8 4 8 Mean Scale Score

2000 4 8 4 8 Mean Scale Score

1999 4 8 4 8 Mean Scale Score

California

1998 4 8 4 8 Mean Scale Score

[continued on next page]

 16 Note that many states did not have or provide school-level assessment data prior to 2000.
Chapter II 10

[continued from previous page]

2005 4 8 4 8 Percent proficient or above

2004 5 8 4 8 Percent proficient or above

2003 5 8 4 8 Percent proficient or above

2002 5 8 4 8 Percent proficient or above

2001 5 8 4 8 Percent proficient or above

Colorado

2000 NA 8 4 7 Percent proficient or above

2005 4 8 4 8 Percent at goal (level 4)

2004 4 8 4 8 Percent at goal (level 4)

2003 4 8 4 8 Percent at goal (level 4)

2002 4 8 4 8 Percent at goal (level 4)

2001 4 8 4 8 Percent at goal (level 4)

Connecticut

2000 4 8 4 8 Percent at goal (level 4)

2005 3 8 3 8 Percent proficient or higher

2004 4 8 4 8 Percent proficient or higher

2003 4 8 4 8 Mean NCE (SAT-9)

2002 4 8 4 8 Mean NCE (SAT-9)

2001 4 8 4 8 Mean NCE (SAT-9)

District of

Columbia

2000 4 8 4 8 Mean NCE (SAT-9)

2005 3 8 3 8 Mean Scale Score

2004 3 8 3 8 Mean Scale Score

2003 3 8 3 8 Mean Scale Score

2002 3 8 3 8 Mean Scale Score

2001 3 8 3 8 Mean Scale Score

2000 3 8 3 8 Mean Scale Score

Delaware

1999 3 8 3 8 Mean Scale Score

2005 4 8 4 8 Percent at or above level 3

2004 4 8 4 8 Percent at or above level 3

2003 4 8 4 8 Percent at or above level 3

2002 4 8 4 8 Percent at or above level 3

2001 5 8 4 8 Percent at or above level 3

2000 5 8 4 8 Percent at or above level 3

Florida

1999 5 8 4 8 Percent at or above level 3

2005 4 8 4 8 Mean Scale Score

2004 4 8 4 8 Mean Scale Score

2003 4 8 4 8

Percent meeting/exceeding the

standard

2002 4 8 4 8 Percent meeting the standard

2001 4 8 4 8 Percent meeting the standard

Georgia

2000 4 8 4 8 Percent meeting the standard

2005 3 8 3 8 Percent at or above proficiency

2004 3 8 3 8 Percent at or above proficiency

2003 3 8 3 8 Percent at or above proficiency

2002 3 8 3 8 Percent at or above proficiency

2001 3 8 3 8 Average of 2000 and 2002

Hawaii

2000 3 8 3 8 Percent at or above proficiency

[continued on next page]
Chapter II 11

[continued from previous page]

2005 4 8 4 8 Percent at or above proficiency

2004 4 8 4 8 Percent at or above proficiency

Iowa

2003 4 8 4 8 Percent at or above proficiency

2005 3 8 3 8 Percent at or above proficiency

2004 3 8 3 8 Percent at or above proficiency

2003 3 8 3 8 Percent at or above proficiency

2002 3 8 3 8 Percent at or above proficiency

2001 3 8 3 8 Percent at or above proficiency

2000 3 8 3 8 Percent at or above proficiency

Illinois

1999 3 8 3 8 Percent at or above proficiency

2005 4 8 4 8 Percent at or above proficiency

2004 3 8 3 8 Percent at or above proficiency

2003 3 8 3 8 Percent at or above proficiency

2002 3 8 3 8 Percent at or above proficiency

2001 3 8 3 8 Percent at or above proficiency

Indiana

2000 3 8 3 8 Percent at or above proficiency

2005 4 8 4 8 Percent at or above proficiency

2004 (1) (1) (1) (1)

2003 4 8 4 8 Percent at or above proficiency

2002 4 8 4 8 Percentile rank

2001 4 8 4 8 Percentile rank

Idaho

2000 4 8 4 8 Percentile rank

2005 4 7 5 8 Percent at or above proficiency

2004 4 7 5 8 Percent at or above proficiency

2003 4 7 5 8 Percent at or above proficiency

2002 4 7 5 8 Percent at or above proficiency

2001 4 7 5 8 Percent at or above proficiency

Kansas

2000 4 7 5 8 Percent at or above proficiency

2005 5 8 4 7 Mean Scale Score

2004 5 8 4 7 Mean Scale Score

2003 5 8 4 7 Mean Scale Score

2002 5 8 4 7 Mean Scale Score

2001 5 8 4 7 Mean Scale Score

2000 5 8 4 7 Mean Scale Score

Kentucky

1999 5 8 4 7 Mean Scale Score

2005 4 8 4 8 Mean Scale Score

2004 4 8 4 8 Mean Scale Score

2003 4 8 4 8 Mean Scale Score

Louisiana

2002 4 8 4 8 Mean Scale Score

[continued on next page]
Chapter II 12

[continued from previous page]

2005 4 8 4 8 Percent at or above proficiency

2004 4 8 4 8 Percent at or above proficiency

2003 4 8 4 8 Percent at or above proficiency

2002 4 8 4 8 Mean Scale Score

2001 4 8 4 8 Mean Scale Score

2000 4 8 4 8 Mean Scale Score

1999 4 8 4 8 Mean Scale Score

Massachusetts

1998 4 8 4 8 Mean Scale Score

2005 4 8 4 8 Percent at or above proficiency

2004 4 8 4 8 Percent at or above proficiency

2003 3 8 3 8 Percent at or above proficiency

2002 3 8 3 8 Percent at or above proficiency

2001 3 8 3 8 Percent Satisfactory

2000 3 8 3 8 Percent Satisfactory

1999 3 8 3 8 Percent Satisfactory

Maryland

1998 3 8 3 8 Percent Satisfactory

2005 4 8 4 8 Mean Scale Score

2004 4 8 4 8 Mean Scale Score

2003 4 8 4 8 Mean Scale Score

2002 4 8 4 8 Mean Scale Score

2001 4 8 4 8 Mean Scale Score

2000 4 8 4 8 Mean Scale Score

Maine

1999 4 8 4 8 Mean Scale Score

2005 4 8 4 7 Percent at or above proficiency

2004 4 8 4 7 Percent at or above proficiency

2003 4 8 4 7 Percent at or above proficiency

2002 4 8 4 7 Percent at or above proficiency

2001 4 (1) 4 7 Percent at or above proficiency

Michigan

2000 4 7 4 7 Percent at or above proficiency

2005 3 7 3 7 Mean Scale Score

2004 3 7 3 7 Mean Scale Score

2003 3 (1) 3 (1) Mean Scale Score

2002 3 8 3 8 Mean Scale Score

2001 3 8 3 8 Mean Scale Score

2000 3 8 3 8 Mean Scale Score

Minnesota

1999 3 8 3 8 Mean Scale Score

2005 4 8 3 7 Mean Scale Score

2004 4 8 3 7 Mean Scale Score

2003 4 8 3 7 Mean Scale Score

2002 4 8 3 7 Achievement level

2001 4 8 3 7 Mean Scale Score

2000 4 8 3 7 Mean Scale Score

Missouri

1999 4 8 3 7 Mean Scale Score

[continued on next page]
Chapter II 13

[continued from previous page]

[continued on next page]

2005 4 8 4 8 Percent at or above proficiency

2004 4 8 4 8 Percent at or above proficiency

2003 4 8 4 8 Percent at or above proficiency

2002 4 8 4 8 Percent at or above proficiency

Mississippi

2001 4 8 4 8 Percent at or above proficiency

2005 4 8 4 8 Percent at or above proficiency

2004 4 8 4 8 Percent at or above proficiency

2003 4 8 4 8 Percent at or above proficiency

2002 4 8 4 8 Percent at or above proficiency

Montana

2001 4 8 4 8 Percent at or above proficiency

2005 4 8 4 8 Mean Scale Score

2004 4 8 4 8 Mean Scale Score

2003 4 8 4 8 Mean Scale Score

2002 4 8 4 8 Mean Scale Score

2001 4 8 4 8 Mean Scale Score

2000 4 8 4 8 Mean Scale Score

North Carolina

1999 4 8 4 8 Mean Scale Score

2005 4 8 4 8 Percent meeting the standard

2004 4 8 4 8 Percent meeting the standard

North Dakota

2003 4 8 4 8 Percent meeting the standard

Nebraska

Omitted because Nebraska only tested in a few of the years and, in 2005, used a

local assessment.

2005 NA NA NA NA

2004 3 6 3 6 Mean Scale Score

2003 3 6 3 6 Mean Scale Score

2002 3 6 3 6 Mean Scale Score

2001 3 6 3 6 Mean Scale Score

2000 3 6 3 6 Mean Scale Score

New

Hampshire

1999 3 6 3 6 Mean Scale Score

2005 4 8 4 8

Percent proficient or above (4th

grade mathematics is percent

advance proficient)

2004 4 8 4 8 Percent proficient or above

2003 4 8 4 8 Percent proficient or above

New Jersey

2002 4 8 4 8 Percent proficient or above

2005 4 8 4 8 Percent proficient or above

2004 4 8 4 8 Percent proficient or above

New Mexico

2003 4 8 4 8 Mean Scale Score
Chapter II 14

[continued from previous page]

2005 3 8 3 8

Achievement level (percent at or

above level 3)

2004 3 8 3 8

Achievement level (percent at or

above level 3)

2003 3 8 3 8 Mean Scale Score

2002 3 (1) 3 (1) Elementary: Mean Scale Score

2001 4 8 4 8 Mean Scale Score

Nevada

2000 4 8 4 8 Mean Scale Score

2005 4 8 4 8 Mean Scale Score

2004 4 8 4 8 Mean Scale Score

2003 4 8 4 8 Mean Scale Score

2002 4 8 4 8 Mean Scale Score

2001 4 8 4 8 Mean Scale Score

2000 4 8 4 8 Mean Scale Score

New York

1999 4 8 4 8 Mean Scale Score

2005 4 6 4 6 Percent proficient or above

2004 4 6 4 6 Percent proficient or above

2003 4 6 4 6 Percent proficient or above

2002 4 6 4 6 Percent proficient or above

2001 4 6 4 6 Percent proficient or above

2000 4 6 4 6 Percent proficient or above

Ohio

1999 4 8 4 8 Percent proficient or above

2005 4 8 4 8 Percent proficient or above

2004 NA NA NA NA

2003 5 8 5 8 Percent proficient or above

2002 NA NA NA NA

2001 5 8 5 8 Percent proficient or above

Oklahoma

2000 5 8 5 8 Percent proficient or above

2005 3 8 3 8

Percent proficient or above

(Elementary: Percent advanced)

2004 3 8 3 8 Percent proficient or above

2003 3 8 3 8 Percent proficient or above

2002 3 8 3 8 Percent proficient or above

2001 3 8 3 8 Percent proficient or above

2000 3 8 3 8 Percent proficient or above

Oregon

1999 3 8 3 8 Percent proficient or above

2005 5 8 5 8 Percent proficient or above

2004 5 8 5 8 Percent proficient or above

2003 5 8 5 8 Percent proficient or above

2002 5 8 5 8 Percent proficient or above

Pennsylvania

2001 5 8 5 8 Percent proficient or above

[continued on next page]
Chapter II 15

[continued from previous page]

2005 NA NA NA NA

2004 4 8 4 8 Percent proficient or above

2003 4 8 4 8 Percent proficient or above

2002 4 8 4 8 Percent proficient or above

2001 4 8 4 8 Percent proficient or above

2000 4 8 4 8 Percent proficient or above

Rhode Island

1999 4 8 4 8 Percent proficient or above

2005 4 8 4 8 Mean Scale Score

2004 4 8 4 8 Mean Scale Score

2003 4 8 4 8 Mean Scale Score

2002 4 8 4 8 Percent proficient or above

2001 4 8 4 8 Percent proficient or above

2000 4 8 4 8 Percent proficient or above

South Carolina

1999 4 8 4 8 Percent proficient or above

2005 NA NA 4 NA

2004 4 8 4 8 Percent proficient or above

2003 4 8 4 8 Percent proficient or above

2002 4 8 4 8 Percentile rank

South Dakota

2001 4 8 4 8 Percentile rank

2005 4 8 4 8 Percent proficient or above

2004 (1) (1) (1) (1)

2003 4 8 4 8 Percentile rank

2002 4 8 4 8 Percentile rank

2001 4 8 4 8 Percentile rank

2000 4 8 4 8 Percentile rank

Tennessee

1999 4 8 4 8 Percentile rank

2005 4 8 4 8 Mean Scale Score

2004 4 8 4 8 Mean Scale Score

2003 4 8 4 8 Mean Scale Score

2002 4 8 4 8 Percent mastering

2001 4 8 4 8 Percent passing

2000 4 8 4 8 Percent passing

Texas

1999 4 8 4 8 Percent passing

2005 3 8 3 8 No variable labels available

2004 3 8 3 8 No variable labels available

2003 3 8 3 8 Percent proficient or above

2002 3 8 3 8 Percent proficient or above

2001 3 8 3 8 Percent proficient or above

2000 3 8 3 8 Percent proficient or above

Virginia

1999 3 8 3 8 Percent proficient or above

[continued on next page]
Chapter II 16

[continued from previous page]

2005 NA NA NA NA

2004 4 8 4 8 Percent achieving standard

2003 4 8 4 8 Percent achieving standard

2002 4 8 4 8 Percent proficient or above

2001 4 8 4 8 Percent proficient or above

2000 4 8 4 8 Percent proficient or above

Vermont

1999 4 8 4 8 Percent proficient or above

2005 4 7 4 7

Achievement Level (Pct. at or

above level 3 or 4)

2004 4 7 4 7

Achievement level (percent at or

above level 3)

2003 4 7 4 7 Index score

2002 4 7 4 7 Index score

2001 4 7 4 7 Index score

2000 4 7 4 7 Index score

Washington

1999 4 7 4 7 Index score

2005 4 8 4 8 Percent proficient or above

2004 4 8 4 8 Percent proficient or above

2003 4 8 4 8 Percent proficient or above

2002 4 8 4 8 Percent proficient or above

2001 4 8 4 8 Percent proficient or above

2000 4 8 4 8 Percent proficient or above

Wisconsin

1999 4 8 4 8 Percent proficient or above

2005 4 8 4 8 Percent proficient or above

2004 4 8 4 8 Percent proficient or above

2003 (1) (1) (1) (1)

West Virginia

2002 Elem Middle Elem Middle

Percent above 3rd quartile; WV only

reported scores as elementary or

middle grades

2005 4 8 4 8 Percent proficient or above

2004 4 8 4 8 Percent proficient or above

2003 4 8 4 8 Percent proficient or above

2002 4 8 4 8 Percent proficient or above

2001 4 8 4 8 Percent proficient or above

2000 4 8 4 8 Percent proficient or above

Wyoming

1999 4 8 4 8 Percent proficient or above

Exhibit highlights: The achievement measures used in the analyses varied across state and

year. As a result, achievement measures were standardized.

Note: “NA” indicates scores not available that year and not imputed from other data. “(1)”

indicates scores not available that year, imputed by averaging the standardized scores from

the year prior and the year after.

Source: National Longitudinal School-Level State Assessment Score Database (NLSLSASD).

Because the standardized scores are standard deviations, coefficients are comparable to effect

sizes; that is, coefficients are interpreted as changes in standard deviations of the outcome

measure. Coefficients that result from analyses can also be converted to more interpretable
Chapter II 17

measures, such as percent proficient or higher (Gill et al., 2005). For example, if CSR accounts

for an increase in achievement scores of 0.5 standard deviations, this value can be multiplied by

the average standard deviation of the percentage of students who achieve at the proficient level

or higher across states to develop a rough estimate of the increase in achievement levels

associated with CSR. Exhibit 3 presents the standard deviations of elementary and middle school

mathematics and reading achievement calculated from the 46 states in which achievement data

were expressed as percent proficient or higher in 2005. The average standard deviation for each

of the four achievement measures (elementary and middle school mathematics and reading) is

approximately 16 percent. Therefore, a finding that CSR is associated with an increase in

achievement scores of 0.5 standard deviations corresponds to an increase of about 8 percentage

points.

Exhibit 3

Standard Deviations of Percent Proficient or Higher for 2005 Elementary and

Middle School Achievement, by State

Elementary School Middle School

State Mathematics Reading Mathematics Reading

AK 17.9 19.3 17.7 21.4

AL 15.5 10.8 16.9 15.0

AR 19.5 17.3 14.5 15.1

AZ 18.6 23.2 20.2 22.7

CA 19.7 19.9 21.2 20.3

CO 19.6 19.5 20.6 19.7

CT 20.2 21.4 22.0 20.0

DC 19.6 21.6 21.0 23.9

DE 14.6 26.8 12.1 18.0

FL 17.2 14.8 21.4 18.6

GA 13.9 17.2 9.2 13.2

HI 14.8 16.2 9.9 14.4

IA 11.1 11.1 10.8 11.1

ID 8.0 9.0 13.4 10.6

IL 18.0 19.7 22.7 15.9

IN 14.1 16.4 14.0 14.6

KS 13.6 13.6 16.6 12.8

KY 16.8 14.8 15.1 14.0

LA 19.2 27.5 18.1 24.1

MA 20.1 20.5 20.4 20.7

MD 17.0 13.9 22.8 20.1

ME 16.9 14.3 16.3 19.0

MI 19.2 14.5 20.9 17.6

MN 16.5 18.1 15.5 16.5

[continued on next page]
Chapter II 18

[continued from previous page]

MO 16.9 10.0 15.6 14.2

MS 14.4 8.2 16.8 15.1

MT 14.1 13.7 16.7 16.9

NC 7.3 15.8 10.7 12.3

ND 13.1 13.9 16.1 13.7

NJ 16.0 14.3 22.5 20.0

NM 17.7 18.0 13.4 15.2

NV 16.1 8.5 15.9 17.6

NY 19.7 8.9 14.3 8.5

OH 19.0 17.1 22.7 15.8

OK 15.0 16.1 14.8 13.3

OR 17.2 17.4 17.0 16.9

PA 19.1 19.8 19.6 18.5

SC 16.8 15.7 12.2 13.0

SD NA NA 15.2 NA

TN 17.5 15.9 17.3 15.7

TX 12.3 19.3 11.8 11.8

VA 8.8 11.6 22.9 20.2

WA 17.7 14.6 17.1 15.5

WI 17.1 13.3 18.5 13.9

WV 11.9 9.9 9.7 8.9

WY 17.8 16.3 15.9 14.3

Average 16.2 16.0 16.7 16.2

Exhibit highlights: The average standard deviation of all of the states was between 16.0 and

16.7 percent across school level and mathematics and reading subject areas.

Note: NA indicates not available.

Source: National Longitudinal School-Level State Assessment Score Database (NLSLSASD).

SURVEY OF SCHOOL REFORM ACTIVITIES

The implementation analyses and the analyses of the relationship between reform

implementation and outcomes rely on survey data from a random sample of CSR and matched

comparison non-CSR schools developed at the outset of this study. The survey provides

descriptive quantitative data on reform implementation—that is, the extent that schools

implemented the 11 elements of CSR included in NCLB as well as the extent that schools

engaged in other non-CSR reform-related activities. The survey also examines other factors that

research associates with successful reform, such as school organization (e.g., instructors teach all

subjects versus the same subject to most classes; common planning periods).
Chapter II 19

The evaluation team ensured the sample of schools in the ECSRIO survey represented the

population of schools by randomly selecting a large enough number of CSR awardees for the

survey sample. When the sample for the study was drawn, the initial CSR universe for calendar

year 2002 comprised 38 states and 1,096 schools. The sample of 400 schools was about 36

percent of this universe, representing 35 states. The evaluation team then updated the sample

based on new data, increasing the sample by 100 schools.

In order to select the sample, researchers obtained the most complete list of CSR schools from

the U.S. Department of Education’s CSR Awards Database. The Department of Education

requires states to report their awardees to the CSR Awards Database. However, the database did

not include all states from the year 2002 for two reasons. First, although data collection is

ongoing, at any given time the data may be incomplete due to delinquent reporting by state

education agencies. Second, many states award CSR funds to schools biennially or irregularly.

Of the initial 15 states in the 2002 CSR universe missing from the original sample, three states

were left out as a by-product of the random selection process. Five additional states reported data

after the initial sample selection. The remaining seven states did not make any awards in 2002

but reported 2003 data. During the first year of the study, the Department of Education allocated

additional funds to gather data from these 15 states.

Researchers purposefully selected an additional 100 schools from a larger random sample of

these missing states to ensure the sample represented all states and school levels. Also, to

measure the value added by CSR over Title I schoolwide grants, a larger comparison group of

schools with non-CSR Title I schoolwide grants was required. Thus, in the second sample, the

choice of comparison schools was limited to those classified as having Title I schoolwide grants

in 2002. The resulting combined sample of 500 CSR schools makes up 37 percent of the

eventual (now relatively stable) universe of 1,340 awardees of the 2002 cohort. Survey data from

the original sample have been combined with the follow-up sample for analyses in this report.

Only a small number of high schools received CSR awards in 2002. Consequently, the high

school survey sample was too small for statistical analyses, so this report focuses on elementary

and middle schools, but not high schools.

Researchers used a two-step process to select potential matches for comparison with CSR

schools. First, they created a school equivalency index for all schools in each state (where data

were available). Second, they calculated a proximity score between each pair of schools within a

state. Matching schools were selected that had the closest proximity on the index to CSR schools

within the same district.

A regression-based approach to weighting and combining background characteristics was used to

construct the index of school similarity for each state. This method is a simplified version of the

California School Characteristics Index (Technical Design Group of the Advisory Committee for

the Public Schools Accountability Act of 1999). Using the National Longitudinal School-Level

State Assessment Score Database (NLSLSASD), the team regressed measures of academic

performance on measures of schoolwide participation in federally subsidized free or reducedprice lunch programs and on schoolwide counts of student ethnicity. The estimated coefficients

led to a composite of background characteristics for each school. In short, each background

characteristic was weighted by the amount it contributed to student performance.
Chapter II 20

The team selected potential matches for each first-year CSR school based upon a minimum

distance criterion. This method was used in the majority of cases. However, alternative methods

were used in two circumstances. First, in some districts (or states), either demographic or

performance data were not available. Second, a suitable comparison non-CSR school was not

available within the same district because the district was too small or all other comparable

schools had previously participated in CSR. Each of these contingencies is discussed below.

In some cases, not enough data were available to construct a school equivalency index. Where

states or districts did not report free or reduced-price lunch or ethnicity, schools were ranked

within districts using only achievement scores. Examples of states where these data were not

available are Tennessee and Washington. Achievement data were missing for some schools or

districts. This was often the case in high schools where the SAT takes the place of districtadministered standardized tests. The proximity scores in these cases were based on an

unweighted composite of free or reduced-price lunch and ethnicity. Finally, in cases in which

neither student performance nor demographic data were available, non-CSR comparison schools

were matched by school grade span, size, and locale.

In districts where a non-CSR comparison school could not be selected, the team searched for a

suitable comparison in an adjacent district or similar locale. Because the school equivalency

index included all public schools in the state, the proximity of any school within the state could

be calculated. The same criteria were used for selection across districts when data were available.

In cases where data were not available, the team used the same procedures that applied to

selecting comparison schools within districts.

Non-CSR schools in the sample were higher-performing than the CSR schools in the sample and

among all newly awarded CSR schools in 2002. The baseline achievement in the ECSRIO

sample of CSR schools was about 0.3 standard deviations lower than their matched non-CSR

comparisons (Exhibit 4). This difference may be related to the procedures used to select CSR

award recipients. The baseline achievement for the sample of CSR schools was also lower than

the baseline achievement for the universe of CSR schools. This difference may be due to the

purposeful selection of the additional 100 schools.
Chapter II 21

Exhibit 4

Average Baseline Standardized Achievement Measures of All 2002 CSR

Schools and of the CSR and Non-CSR Schools in the ECSRIO Sample

Standardized Achievement Measures (Z-scores)

CSR Universe

ECSRIO CSR

Sample Schools

ECSRIO non-CSR

Sample Schools

Elementary School

Mathematics -0.96 -0.82 -0.54

Elementary School Reading -0.97 -0.86 -0.57

Middle School Mathematics -0.90 -0.87 -0.55

Middle School Reading -0.92 -0.87 -0.54

Exhibit highlights: The average baseline standardized achievement measures of the ECSRIO

CSR schools was about 0.3 standard deviations lower than the average baseline standardized

achievement measures of the ECSRIO non-CSR schools.

Sources: CSR Awards Database; National Longitudinal School-Level State Assessment Score

Database (NLSLSASD).

The sample of ECSRIO CSR and non-CSR schools were similar to each other and to the

universe of schools that received their CSR awards in 2002 in their demographic compositions in

2005 (Exhibit 5). The non-CSR schools had a slightly lower percentage of free or reduced-price

lunch eligible students than the CSR universe or the CSR schools. The universe of CSR schools

had a somewhat higher percentage of black students than either the ECSRIO CSR schools or the

ECSRIO non-CSR schools.
Chapter II 22

Exhibit 5

Average Demographic Measures in 2005 of All 2002 CSR Schools

and of the CSR and Non-CSR Schools in the ECSRIO Sample

Demographic Measures CSR Universe

ECSRIO CSR

Sample Schools

ECSRIO non-CSR

Sample Schools

Free or Reduced-Price Lunch (%) 71.24 69.23 64.91

Black (%) 43.05 35.75 31.48

Hispanic (%) 19.09 20.09 21.48

Type of School (%)

Elementary School (includes K–8) 61.13 65.88 67.09

Middle School 19.68 16.20 14.35

High School 15.24 15.78 15.40

Other Configuration 3.95 2.13 3.16

Exhibit highlights: The average percentages of students receiving free or reduced-price

lunch, average percentage of black students, and average percentage of Hispanic students in

the ECSRIO CSR schools in 2005 were about the same as in the ECSRIO non-CSR schools.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal

School-Level State Assessment Score Database (NLSLSASD).

From the 1,000 schools originally selected for the survey sample, 961 agreed to participate in the

study and were surveyed in 2003 and again in 2005. Principals and up to three teachers in each

school were surveyed on various aspects of their respective school’s reform activities (see

Appendix C for the survey instruments). The response rate for the first-year data collection

(2003) was 90 percent for the 479 CSR schools and 82 percent for the 482 non-CSR schools. The

response rate for the third-year data collection (2005) was 88 percent for CSR schools and 78

percent for non-CSR schools. The response rate for surveyed staff was 77 and 64 percent for

principals and teachers respectively in 2003 and 75 and 71 percent in 2005.

The survey instrument includes items that measure behaviors rather than attitudes and

expectations. For example, rather than ask whether the reform strategy meets student needs,

respondents were asked about procedures that monitor student achievement during

implementation. Although this approach is likely to increase the objectivity of the responses,

self-reported responses may still have limitations such as the accuracy of respondent recall and

the tendency for respondents to give socially desirable responses (Flower, 1995; Tourangeau,

Rips, and Rasinki, 2000). Survey findings were analyzed in conjunction with the results of the

case studies to validate information on implementation from both sources. In general, both

methods yield similar findings and, when differences exist, the case studies found greater reform

implementation than did the surveys, indicating that survey respondents do not inflate their

estimates of reform implementation.
Chapter II 23

CASE STUDIES OF SCHOOL REFORM ACTIVITIES

The case studies included in this evaluation provide in-depth insight into comprehensive school

reform implementation processes and outcomes. They also provided clues for interpreting the

survey data to determine if self-reported data reflect inflated or conservative responses.

Of the 1,000 schools selected for the survey sample, a subsample of 15 pairs of schools from 15

districts across 14 states—with each pair containing a CSR school and a non-CSR school—was

selected to provide case studies that supplement the quantitative analyses with qualitative

understanding of implementation dynamics. For example, although almost all schools reported

having a comprehensive school plan, the case studies revealed differences in how the plan was

used, with some schools developing a plan as a ritual activity and others making it into a living

document. With such a small number of case study schools, this sample does not represent either

the geographic distribution or the distribution of school levels in the universe of CSR schools.

However, the selection ensured the case study sample included enough geographic and schoollevel diversity to reflect a variety of state policy environments, including schools in states with a

long tradition of local control and others in states that exercise more centralized control over

schools.

Researchers selected CSR and comparison pairs that resided in the same district for the fieldbased study because it was the only way to observe the differential effect of district policies.

Also, visiting a single district for each pair minimized the data collection burden for both the

evaluators and respondents.

The case studies included two visits to each site, with a “site” defined as a combination of four

entities:

1. A CSR-funded school;

2. A demographically matched non-CSR school that did not receive any federal CSR funds

prior to 2002 and is located in the same district as the CSR-funded school;

3. The district within which the two schools are located; and

4. The state education agency in the state within which the district is located.

The first phase of case study data collection occurred from spring 2003 to spring 2004. A second

phase of site visits occurred during the 2004–05 school year. During visits to the 15 pairs of

schools, the evaluation team observed classrooms using a formal observation instrument;

reviewed school documents and materials; and interviewed state and district officials, school

administrators, teachers, reform developers, and parents.

The case studies include information related to reform implementation as well as its relationship

to district and state policies, including those that do not focus directly on CSR. The case studies

follow a formal protocol that calls for integrating information from documents, interviews, focus

groups, observations, and quantitative data (see Appendix C).
Chapter II 24

STUDY LIMITATIONS

The multi-method approach to this evaluation, like most studies of practice (Lipsey, 2003), has

limitations.

All achievement analyses presented in this report rely on standardized school-level achievement

scores from NLSLSASD. The use of school-level outcome measures presents methodological

and psychometric challenges. School-level scores are aggregate measures of individual student

performance within a school and, as such, do not account for variation in academic performance

among students within a group. In addition, such analyses are not sensitive to the fact that

performance may be contingent on both individual factors and factors related to group

membership, such as membership in different classrooms and school environments. The multileveled structure of data is not adequately addressed in aggregate analyses, which limits the

precision of the estimates produced. Three major areas have been widely acknowledged as

contributing to imprecision in aggregate analyses: aggregation bias, misestimated standard

errors, and heterogeneity of regression (Burstein, 1978; Burstein and Miller, 1981; Haney, 1980;

Raudenbush and Bryk, 2002). Discussion of these issues follows:

1. Aggregation bias exists because variables at different organizational levels have different

contextual meanings and may show varying relationships across organizational levels.

Analyses based on aggregate outcomes do not account for differences in variables across

organizational levels leading to biased estimates of effects. Raudenbush and Bryk (2002)

use socioeconomic status to explicate aggregation bias. They note that both individual

social class and socioeconomic status of the school influence student achievement.

School-level socioeconomic measures may influence achievement beyond the effects

estimated at the individual level.

2. Misestimation of standard errors occurs in single-level analyses because such analyses

ignore the similarities based on common experiences among responses from individuals

within a group. For example, all students in a classroom are likely to be exposed to the

same curriculum. Multi-level analyses calculate standard errors in a way that accounts for

the clustering of individuals and adjusts for the dependence of responses from a particular

group.

3. Despite the common experiences, differences exist within all groups. Consequently, the

relationships between individual characteristics and outcomes may vary among

individuals. This heterogeneity is masked in school-level analyses that assume a single

linear relationship between characteristics and outcomes for all individuals within a

group. In contrast, multi-level models provide a mechanism to explore the heterogeneity

of relationships and potential moderating factors that may account for differences in the

relationships between outcomes and explanatory variables. For example, the amount of

school resources or level of teacher professional development in schools may influence

the relationship between CSR and student achievement.

Despite these weaknesses, designating the school as the unit of analysis and using aggregate

outcome measures is deemed appropriate given the aims of the evaluation and the nature of CSR.

The analyses rely on aggregate outcome measures that are both meaningful to schools and
Chapter II 25

relevant to the current policy environment. School-level proficiency is the performance measure

upon which all schools are held accountable. Although the use of aggregate measures may

compromise precision, its relevance for school officials and policymakers offsets its limitations.

Moreover, the CSR program was designed to stimulate comprehensive change through a set of

coordinated reform actions that influence school operations across the whole school. As such, it

is appropriate to use the school as the unit of analysis and school-level achievement measures to

examine how CSR influences achievement.

Furthermore, although a multi-level model, in which students are nested within schools, would

provide more precise estimates of the relationships between CSR and achievement outcomes

than the models employed in this evaluation, collecting individual student records over an

extended time period from a large nationally representative sample of CSR and comparison

schools places a heavy burden on schools.

A second limitation of this study is shared with other efforts to assess program effectiveness

across states. Such evaluations must overcome the difficulty of using the existing state

assessments, which are designed to provide information about students’ progress toward

mastering the content established in each state’s standards. Consequently, assessments differ. In

fact, neither the content nor the criteria for determining proficiency are the same from state to

state. Also, standards, assessments, and proficiency criteria often change, making scores within

states difficult to compare over time. To analyze outcomes across states and assessment

instruments, school-level assessment scores were standardized within states for each year. The

standardization provides a common metric for all achievement outcomes; however, it does not

account for all sources of heterogeneity among states resulting from differences in assessments

and the stringency of state proficiency standards. A more precise analytic approach would

employ a multi-level model of schools nested within states to account for the variation across

states. Insufficient within-state samples, stemming from both missing data and the manner in

which 2002 CSR grants were awarded across states, precluded the use of a multi-level model to

explore the relationships between CSR and student academic achievement.

The study is further limited by the loss of a large number of schools from some analyses due to

survey nonresponse in 2005 and the fact that several states did not report all data to the Common

Core of Data. For example, nearly one-half of the elementary and middle schools in the original

ECSRIO sample are not included in the analyses of the relationships between comprehensiveness

of CSR implementation and achievement growth. If the survey had captured information about

implementation from the excluded schools and found they were implementing fewer CSR

components while still experiencing comparable achievement gains as the included schools (see

Exhibits 24, 25, and 26 in Chapter IV for details), then a weaker relationship between CSR

implementation and achievement may exist than reported from this study. This possibility seems

plausible if one assumes that schools implementing more CSR components are more likely to

complete the survey than those implementing fewer components.
Chapter II 26

EVALUATION QUESTIONS

The next three chapters provide findings related to the year-three evaluation questions:

 Was receipt of a CSR award associated with improvements in schoollevel mathematics and reading achievement?

 Were schools that received CSR awards more likely to implement the

legislatively specified components of CSR than other schools?

 Was fidelity of CSR implementation associated with gains in schoollevel mathematics and reading achievement?
Chapter III 27

III. OVERALL RELATIONSHIP BETWEEN CSR AWARD AND

ACHIEVEMENT

METHODOLOGY USED TO ASSESS THE RELATIONSHIP BETWEEN RECEIPT OF A CSR

AWARD AND ACHIEVEMENT

Analyses of the relationship between CSR award and growth in achievement were restricted to

Title I schools that received CSR program funding. The evaluation team selected Title I schools

using demographic information from the Common Core of Data (CCD), the U.S. Department of

Education’s national database of school demographics, and the standardized achievement scores

developed for this evaluation. Selection of the comparison group was based on data from

2001–02, the year before the CSR schools began implementing their awards.

States targeted CSR awards to low-performing Title I schools, resulting in 96 percent of CSR

awards going to such schools. Therefore, analyzing Title I schools allows for inference regarding

the population of greatest interest. However, Title I encompasses a wide range of schools; in

2003–04, over 54,000 schools received Title I assistance as either a Schoolwide Title I or

Targeted Assistance School (U.S. Department of Education, 2007). Given the emphasis most

states placed on making CSR awards to the lowest-performing and highest-poverty schools, it

comes as no surprise that the population of Title I non-CSR schools are higher-performing,

smaller, and contain lower percentages of students from traditionally underserved minority

groups and those eligible for free and reduced-price lunch than Title I CSR schools (Exhibit 6).

Key Finding

Was receipt of a CSR award associated with improvements in school-level

mathematics and reading achievement?

 Receipt of a CSR award was not associated with achievement gains in

mathematics and reading achievement through the first three years of award.
Chapter III 28

Exhibit 6

Differences Between the 2002 Cohort of Title I CSR Schools and Title I Non-CSR

Schools

Non-CSR CSR

N Average N Average Difference

Standardized Assessment Scores

Elementary Math 28,366 -0.22 649 -0.99 0.77**

Elementary Reading 28,512 -0.25 654 -0.97 0.73**

Middle School Math 10,135 -0.19 310 -0.95 0.76**

Middle School Reading 10,282 -0.19 318 -0.97 0.78**

Demographic and School Characteristics

Membership (N) 46,667 475 973 579 -104**

Percent Minority 45,635 52.0 940 72.1 -20.1**

Percent FRL 46,632 36.2 973 59.9 -23.8**

Exhibit highlights: Achievement in Title I CSR schools in the 2001–02 school year was

approximately three-fourths of a standard deviation lower than in Title I non-CSR schools. The

percentage of minority students in Title I CSR schools was about 20 percent higher, and the

percentage of students eligible for free and reduced-price lunch was about 24 percent higher,

than in Title I non-CSR schools.

Note: + p<.10; * p<.05; ** p<.01. “FRL” refers to students eligible for free or reduced-price

lunches. All data are for the 2001–02 school year to demonstrate baseline achievement and

demographics. Standard errors are included in Exhibit B.1.

Source: Common Core of Data (CCD).

Because of this disparity, the evaluation team developed a comparison group of non-CSR

schools for the analysis of the relationship between receiving a CSR award and achievement

through Mahalanobis propensity scoring (Rosenbaum and Rubin, 1983; Rosenbaum and Rubin,

1985; Rubin, 1980) based on mathematics and reading achievement, percent free and reducedprice lunch, and percent minority.17 In this instance, CSR schools are counted as treated schools

and non-CSR schools are counted as untreated schools. For each CSR school, Mahalanobis

propensity scoring chooses a comparison school from the pool of untreated schools that most

closely matches the CSR school based on the achievement and demographic measures used.18

Because some states did not report percent free and reduced-price lunch, and because sometimes

CSR schools were located in small districts, the propensity score matching was run several times,

first starting with the most restrictive matching procedure and then gradually relaxing the

matching conditions. All matches are based on, at the minimum, mathematics and reading

achievement and percent minority. Matches were restricted to being located in the same state and

 17 Note that the comparison group developed for the analysis of the relationship between receiving an award

and achievement gains is different from the comparison group developed at the outset of this study. The

comparison group used in this section is based only on Title I schools; furthermore, it relies on updated CSR

award, Common Core of Data, and NLSLSASD data files not available at the outset of this study.

18 The Mahalanobis propensity score matching was completed using the -mahapick- module in Stata (Kantor,

2006).
Chapter III 29

having the same school configuration (elementary, K–8, or 6–8 middle school) as the CSR

school. The conditions that were used are listed from most restrictive to least restrictive. The

comparison school was chosen from the most restrictive set of conditions where a successful

comparison school was found.

1. The first set of matches is based on percent free and reduced-price lunch and being

located in the same school district. Comparison schools selected in this group most

closely matched their CSR school on mathematics and reading achievement, percent

minority and percent free and reduced-price lunch; were located in the same school

district as the CSR school; and had the same school configuration as the CSR school.

2. The second set of matches removes free and reduced-price lunch from the restrictions

used for the first set.

3. The third set of matches is similar to the first, but instead of restricting the matching to

the same district, restricts the matching to the same type of locality (e.g., large city, small

city, suburban, or rural) in the state. This expands the pool of potential matches for

schools in small school districts that may not have many potential matches.

4. The fourth set of matches removes free and reduced-price lunch from the third set of

matches.

5. The fifth set of matches includes free and reduced-price lunch, and removes any locality

restriction (choosing any school in the state).

6. The last set of matches removes free and reduced-price lunch from the restrictions used

for the fifth set of matches.

Exhibit 7 presents average school-level achievement scores and demographics for schools with

elementary mathematics and reading scores, while Exhibit 8 presents these measures for schools

with middle school mathematics and reading scores. The CSR and comparison schools are very

closely matched on baseline achievement and demographics.
Chapter III 30

Exhibit 7

Differences Between the 2002 Cohort of Title I CSR Schools and Non-CSR Title I

Schools Chosen by Propensity Scoring: Schools with Elementary Mathematics and

Reading Achievement

Non-CSR CSR

N Average N Average Difference

Standardized Assessment Scores

Elementary Math 550 -0.90 645 -0.99 0.09

Elementary Reading 550 -0.89 645 -0.97 0.08

Demographic and School Characteristics

Membership (N) 550 514 645 509 5

Percent Minority 533 72.4 618 74.1 -1.7

Percent FRL 550 58.8 645 61.2 1.4

Exhibit highlights: There are no statistically significant differences between matched CSR and

non-CSR Title I schools in the baseline (2001–02) school year.

Note: + p<.10; * p<.05; ** p<.01. “FRL” refers to students eligible for free or reduced-price

lunches. Standard errors are included in Exhibit B.2.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD).

Exhibit 8

Differences Between the 2002 Cohort of Title I CSR Schools and Non-CSR Title I

Schools Chosen by Propensity Scoring: Schools with Middle School Mathematics and

Reading Achievement

Non-CSR CSR

N Average N Average Difference

Standardized Assessment Scores

Middle School Math 248 -0.86 309 -0.96 0.10

Middle School Reading 248 -0.83 309 -0.94 0.11

Demographic and School Characteristics

Membership (N) 248 664 309 663 1

Percent Minority 245 71.3 301 73.5 -2.1

Percent FRL 248 63.7 309 65.2 -1.5

Exhibit highlights: There are no statistically significant differences between matched CSR and

non-CSR Title I schools in the baseline (2001–02) school year.

Note: + p<.10; * p<.05; ** p<.01. “FRL” refers to students eligible for free or reduced-price

lunches. Standard errors are included in Exhibit B.3.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD).
Chapter III 31

After the selection of the comparison schools, paired t-tests were used to assess whether the

change in achievement from 2002–03 to 2004–05 is different between CSR schools and their

comparisons (Zhao, Li, Gao, and Tierney, 2001; Fraas, Newman, and Pool, 2007).19

FINDINGS

Was receipt of a CSR award associated with improvements in school-level mathematics

and reading achievement?

 Receipt of a CSR award was not associated with achievement gains in mathematics and

reading achievement through the first three years of award.

CSR schools made small but statistically significant gains in elementary mathematics and

reading achievement during the time they were implementing their award (effect sizes of 0.06

and 0.07, respectively) (Exhibit 9). The comparison schools, however, made similar gains,

indicating that CSR was no better than comparable Title I schools at improving reading

achievement. At the middle school level, changes in mathematics and reading achievement are

not statistically significant, except for non-CSR schools in mathematics. Again, differences in

achievement change between CSR and non-CSR schools are not statistically significant. Over

the course of their three-year awards, CSR schools were no better at improving achievement than

Title I schools that were similar in demographics and achievement in the baseline year.

 19 T-tests are used to test the null hypothesis that the means of two groups (in this instance, CSR and

comparison schools) are the same. A t-statistic is calculated from the two groups’ means and standard

deviations, which is then used to determine the probability of correctly rejecting the null hypothesis by

comparing it to a table of students’ t-distribution.
Chapter III 32

Exhibit 9

Changes in Standardized Assessment Scores in CSR and Non-CSR Schools from

2002–03 to 2004–05

Changes in Standardized Assessment Scores

N CSR Schools

Non-CSR

Schools Difference

Elementary Mathematics 634 0.06+ 0.09** -0.03

Elementary Reading 638 0.07* 0.07* 0.00

Middle School Mathematics 318 0.03 0.09** -0.06

Middle School Reading 320 0.02 0.03 -0.01

Exhibit highlights: CSR schools had statistically significant increases between 2002–03 and

2004–05 in elementary mathematics and reading achievement of 0.06 and 0.07 standard

deviations, respectively. There were, however, no statistically significant differences between

CSR schools and their matched non-CSR schools.

Note: + p<.10; * p<.05; ** p<.01. Tests for the statistical significance of achievement gains for

CSR and non-CSR schools are t-tests to assess whether the value is different from zero. The

differences between CSR and non-CSR schools are assessed through paired t-tests. Standard errors

are included in Exhibit B.4.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD).

DISCUSSION

Unlike these results, earlier CSR research shows an increase in mathematics and reading

achievement over multiple years of implementation (Borman et al., 2003). Two possible reasons

for the lack of findings here are suggested by the case studies. First, a number of schools ended

implementation of an original CSR strategy, replacing it with a portfolio of programs. Second,

some schools added reform programs, creating competition for teachers’ time for professional

development.

School W, for example, chose the Comer School Development model following award receipt in

2002. A new principal arrived in 2003 and instead chose the Effective Schools model. Teachers

indicated they were not involved in this decision and had already scheduled professional

development for the original model for the summer. Staff then engaged in what they termed a

“whirlwind” of professional development on Effective Schools. However, key consultants for

Effective Schools became ill, compounding staff indifference to the program. Ultimately, neither

Effective Schools nor Comer School Development was implemented.

Another school, School E, was encouraged by the district’s then superintendent to adopt and

implement Accelerated Schools. The external assistance provided by the model developer helped

the school arrange common planning time for teachers and offered over 20 days of professional

development in 2002–03. However, in 2003, district leadership changed, and the new

superintendent focused efforts on supplementary programs. Teachers were required to attend

professional development sessions for the district’s priority programs, creating scheduling

conflicts. By 2004, Accelerated Schools was no longer present in the school.
Chapter III 33

Thus, implementation (or lack thereof) may explain the overall lack of a relationship between

receiving a CSR award and achievement gains. The CSR program is based on two key

implementation concepts: Schools implementing reforms that encompass the 11 components

indicated in NCLB will have higher achievement than schools that do not; and schools that

implement reform models with a scientific research base will have higher achievement than

those schools that implement models that lack a scientific research base. Two different analyses,

included in the next two sections, assess the implementation of CSR and its relationship to

achievement. The first analyzes the relationship between the comprehensiveness of CSR

implementation, as measured by the number of components a school implements, and

achievement outcomes. The second analyzes the relationship between the adoption of a

scientifically based research model and achievement.
Chapter III 34
Chapter IV 35

IV. THE COMPREHENSIVENESS OF CSR IMPLEMENTATION AND

ITS RELATIONSHIP TO ACHIEVEMENT

This report includes analyses of changes in the extent of component implementation for both

CSR and non-CSR schools over the three-year award period. The first section of this chapter

focuses on the implementation of CSR, comparing CSR and non-CSR schools; the second

section of this chapter focuses on the relationship between the implementation of the CSR

components and achievement gains.

MEASURING IMPLEMENTATION OF THE CSR COMPONENTS

The analyses of the relationships between the CSR components and school-level achievement

rely on measures of the CSR components developed from the surveys administered in 2003 and

2005 to both CSR and non-CSR schools. The surveys ask principals and teachers about the

implementation of activities consistent with the 11 components identified in NCLB. For most

questions, respondents were presented with a forced choice of “0” for not having implemented

Key Findings

Were schools that received CSR awards more likely to implement the

legislatively specified components of CSR than other schools?

 No, both CSR and non-CSR schools implemented an average of fewer than

four components in 2003 and fewer than five in 2005 at both the elementary

and middle school levels.

Was fidelity of CSR implementation associated with gains in school-level

mathematics and reading achievement?

 The comprehensiveness of implementation, as measured by the number of

CSR components implemented, was not related to mathematics and reading

achievement gains in CSR schools.
Chapter IV 36

the activity and “1” for implementing the activity.20 Within each component, respondents’

answers were averaged to develop a component score for each school.

The component scores were converted into measures of whether a school is counted as

implementing a component. That is, schools were assigned a 0 if their component score or their

responses to survey questions fell below a certain threshold and a 1 if they exceeded the

threshold. For most of the components, the school had to report that it was implementing all of

the activities to be counted as implementing the component. In some instances, however, this

would have resulted in no schools implementing the component, and so the threshold was

relaxed somewhat. Also, for parental involvement, the measures used were not dichotomies, but

instead were estimates of the percentage of parents engaging in various activities in the school;

and another threshold was developed based on the distribution of the estimates. The metric used

to measure whether a school implemented a component is included in the discussions of how

each component is measured. This was done to measure whether a school implemented a

component and was used to calculate the number of components implemented. The number of

components implemented was also used in regression analyses to measure the relationship

between number of components implemented and achievement.

The surveys were modified between the two administrations as a result of findings from the 2003

survey concerning the existence of multiple reforms in schools, the need to develop more refined

measures of professional development, and the fact that some of the questions in the 2003 survey

were no longer applicable in 2005. For instance, in the 2003 survey, schools often reported

implementing more than one reform method. The 2005 survey was modified to capture

information on up to four reforms the school was undertaking (see Appendix C). As a result of

these changes, some of the components were not measured consistently across the two time

periods; these modifications also required changes in assessing whether a component was

implemented. All such changes are documented below in the discussion of each component.

Components 1 and 11—Research-Based Design and Evidence-Based Practice

Given the substantial similarity between Component 1: Research-Based Design and Component

11: Evidence-Based Practice, the decision was made to combine them into one measure. As

defined in the No Child Left Behind Act (NCLB) of 2001, evidence-based practice focuses on

proven methods and strategies for student learning, teaching, and school management that have

been replicated successfully in schools with diverse characteristics. Added in NCLB, the

research-based design component calls for the adoption of programs that use scientifically based

 20 This evaluation measures implementation differently from Aladjem et al. (2006), who measured

implementation as the difference between what the model developers consider to be full implementation and

what the school actually does. To do so, they surveyed each of the model developers and asked them to

respond as if they were a school that implemented their model. The researchers compared each school’s

responses to this measure of implementation to develop a measure of fidelity. In order to measure

implementation in the comparison schools, Aladjem et al. predicted what CSR model the comparison school

would have chosen based on its school characteristics and those of the CSR schools. The researchers then

compared the school’s responses with those of the model developer of the predicted CSR model. This measure

of implementation is consistent with the focus on models as compared with this study’s focus on fidelity with

the 11 components identified in NCLB.
Chapter IV 37

research to document significant improvement in the academic achievement of students

participating in such programs.

The 2003 measure consisted of a principal’s responses to three questions, each of which was a

yes or no question on the survey: whether there was evidence based on independent research

supporting the model (33 percent indicating yes), whether there was evidence based on research

conducted by the reform developer (also 33 percent), and whether there was evidence that relied

on the use of control or comparison groups (21 percent of respondents). A school was counted as

implementing this component if it reported that at least two of the three types of evidence were

present in 2003 (Exhibit 10).

Two of the items for the 2005 component measure were different from items used for the 2003

measure: whether the reform model had evidence that it improved student achievement and had

been shown to improve student achievement at other schools. One item, whether evidence was

based on research that relied on comparison or control groups, was consistent across the two

years. Each of these was also a yes or no question. Schools were asked to report on up to four

reform methods in their school at the time of the survey; these responses were averaged to derive

a school-level score, which ranged from 0 to 1, for each of the three items. The school was

counted as implementing this component if the score for each of the non-missing items was one.

Thus, if a school only reported on two of the items and reported that they were both present, then

that school was counted as implementing a research-based design. This was done to make the

measure for year three relatively consistent with the measure for year one, which required that

the school indicate that two types of evidence were present.

Exhibit 10

Means and Standard Deviations of Survey Items Used to Construct

Research-based Design and Evidence-based Practice Measure, 2003 and 2005

Year Evidence Items N Mean

Standard

Deviation

Independent Research 759 0.33 0.47

2003 Research by Reform Designer 759 0.33 0.47

Comparison/Control Groups 759 0.21 0.41

Improves Achievement 695 0.89 0.28

2005 Comparison/Control Groups 674 0.60 0.45

Student Achievement at Other Schools 670 0.91 0.26

Exhibit highlights: The type of evidence to construct the research-based design and

evidence-based practice measure differed from 2003 to 2005. The only consistent measure

was comparison or control groups. The reader may convert any of the means into a percent by

multiplying the mean value by 100. For example, the mean score of 0.21 for comparison or

control groups in 2003 indicates that 21 percent of respondents in 2003 indicated the reform

method they chose had evidence based on comparison or control groups; in 2005, 60 percent

of respondents said this was true.

Note: Means and standard deviations are computed from respondents’ answers to questions

about the existence of the various evidence items (0=No; 1=Yes).

Source: ECSRIO surveys.
Chapter IV 38

Component 2: Comprehensive Planning

Questions on the survey form measured two aspects of comprehensive planning—classroombased planning and school-based planning. Both were asked of respondents consistently in 2003

and 2005. On all questions, respondents were asked if their school improvement plans contained

nine components of planning; they could answer yes or no to each. The classroom-based

measures included curriculum and instruction, student assessment, classroom management, and

professional development (Exhibit 11). The school-based planning measure included measurable

goals for reform, periodic evaluation, parental involvement, professional development,

participation in school management, and integration of new technology (Exhibit 12). Note that

professional development occurred in both school- and classroom-level planning because

professional development opportunities may be around subject-specific topics or school-reform

related topics. In both years, a school was counted as implementing these components if it

reported that the school improvement plan contained all of the items.

Exhibit 11

Means and Standard Deviations of Survey Items Used to Construct

Comprehensive Planning-Classroom Measure, 2003 and 2005

2003 2005

Comprehensive Planning Items Mean

Standard

Deviation Mean

Standard

Deviation

Curriculum and Instruction 0.88 0.32 0.95 0.21

Student Assessment 0.50 0.50 0.65 0.48

Classroom Management 0.43 0.49 0.58 0.49

Professional Development 0.93 0.26 0.96 0.20

Exhibit highlights: Four survey items were used to measure comprehensive planning in the

classroom. The reader may convert any of the means into a percent by multiplying the mean

value by 100. For example, in 2003 the mean curriculum and instruction score of 0.88

indicates that 88 percent of respondents indicated that the school improvement plan covered

curriculum and instruction.

Note: Means and standard deviations are computed from respondents’ answers to questions

about the existence of the various classroom planning items (0=No; 1=Yes). There are 764

observations in 2003 and 717 observations in 2005.

Source: ECSRIO surveys.
Chapter IV 39

Exhibit 12

Means and Standard Deviations of Survey Items Used to Construct

Comprehensive Planning-School Measure, 2003 and 2005

2003 2005

Comprehensive Planning Items Mean

Standard

Deviation Mean

Standard

Deviation

Measurable Goals 0.97 0.16 0.98 0.15

Periodic Evaluation 0.85 0.36 0.91 0.28

Parental Involvement 0.81 0.40 0.85 0.36

Professional Development 0.93 0.26 0.96 0.20

Participation in School Management 0.53 0.50 0.63 0.48

New Technology 0.72 0.45 0.78 0.42

Exhibit highlights: Six survey items were used to measure comprehensive planning in the

classroom. The reader may convert any of the means into a percent by multiplying the mean

value by 100. For example, in 2003 the mean parental involvement score of 0.81 indicates

that 81 percent of respondents indicated that the school improvement plan covered parental

involvement.

Note: Means and standard deviations are computed from respondents’ answers to questions

about the existence of the various school planning items (0=No; 1=Yes). There are 764

observations in 2003 and 717 observations in 2005.

Source: ECSRIO surveys.

Component 3: Professional Development

The professional development component relied on the reported number of days of professional

development; however, it was measured differently in the two years, and any changes should be

interpreted cautiously. On the 2003 survey, respondents were asked if their school provided at

least 10 days of professional development. Principals could respond yes or no, and the school

was counted as implementing the measure if they responded yes. On the 2005 survey, teachers

were asked to report the number of hours of professional development they received in the last

year. The research team divided the number of hours by six to convert their responses to days

(because that is approximately the length of an average school day). Schools were counted as

implementing professional development if the average number of days of professional

development received was 10 or more (Exhibit 13).
Chapter IV 40

Exhibit 13

Means and Standard Deviations of Professional Development Item,

2003 and 2005

2003 2005

Professional

Development Item N Mean

Standard

Deviation N Mean

Standard

Deviation

Professional Development 736 0.48 0.50 718 0.61 0.49

Exhibit highlights: The reader may convert any of the means into a percent by multiplying

the mean value by 100. For example, on the 2003 survey, principals responded 48 percent of

the time that at least 10 days of professional development were available to teachers. On the

2005 survey, 61 percent of teachers reported that they received at least 10 days of professional

development during the last year.

Note: Means and standard deviations are computed from respondents’ answers to questions

pertaining to the professional development opportunities exceeding 10 days (0=No; 1=Yes).

This question was asked differently in 2003 and 2005; any changes should be interpreted

cautiously.

Source: ECSRIO surveys.

Component 4: Goals and Benchmarks

The 2003 and 2005 goals and benchmarks measures relied on a somewhat different set of items,

and any change statistics should be interpreted cautiously. Four questions from the 2003 surveys

were used to construct the measure for goals and benchmarks—whether the school improvement

process (SIP) included student assessment rubrics and measurable goals and objectives as well as

whether the school had end-of-year and interim student achievement goals (Exhibit 14). Nonmissing responses were summed, and the school was counted as implementing this component if

the average of the non-missing responses was one. Thus, if a school reported on two of the

measures and indicated those were in place, the school was counted as implementing goals and

benchmarks.

The 2005 goals and benchmarks measure relies on the same two SIP items as the 2003 measure.

However, the two questions about student goals were not asked in the 2005 surveys. Instead,

respondents were asked if the reform strategies were accompanied by implementation

benchmarks. One of the earlier findings in this study is that schools were often implementing

more than one reform strategy simultaneously. As a result, reform-specific questions were asked

four times for each respondent. In some cases, the respondent indicated that his or her school was

only undertaking one reform strategy; in other cases, respondents listed up to four reform

strategies. The responses were averaged to derive a school-level measure for this item that varied

between 0 and 1. The 2005 goals and benchmarks measure was created by averaging the nonmissing responses. Schools with a score of one were counted as implementing goals and

benchmarks.
Chapter IV 41

Exhibit 14

Means and Standard Deviations of Survey Items Used to Construct

the Goals and Benchmarks Measure, 2003 and 2005

Year Goals and Benchmarks Items N Mean

Standard

Deviation

SIP Includes Student Assessment Rubrics 764 0.50 0.50

SIP Includes Measurable Goals or Objectives 764 0.97 0.16

School Has Student Goals at the End of the Year 561 0.88 0.33

2003

School Has Intermediate Student Goals 607 0.70 0.46

SIP Includes Student Assessment Rubrics 717 0.65 0.48

2005 SIP Includes Measurable Goals or Objectives 717 0.98 0.15

Strategies Accompanied by Implementation Benchmarks 694 0.75 0.38

Exhibit highlights: The survey items used to measure goals and benchmarks differed slightly

from 2003 to 2005. The reader may convert any of the means into a percent by multiplying

the mean value by 100. For example, the mean score of 0.50 for whether the school

improvement plan includes student assessment rubrics in 2003 indicates that one-half of

respondents said that the school improvement plan includes student assessment rubrics.

Note: SIP refers to school improvement process. Means and standard deviations are computed

from respondents’ answers to questions about the existence of the various items measuring

goals and benchmarks (0=No; 1=Yes).

Source: ECSRIO surveys.

Component 5: Faculty Participation

The 2003 and 2005 measures for faculty participation relied on almost the same survey items,

but the 2003 measure included one additional item that was not asked on the 2005 survey

(Exhibit 15). Both measures relied on yes or no questions about all teachers participating in

reform and factors that limit participation. The 2003 measure included the estimate by the

principal of the percentage of teachers who participate in reform. In both years, the faculty

participation measure was created by averaging the non-missing responses. Schools with a score

of one were counted as implementing the faculty participation component.
Chapter IV 42

Exhibit 15

Means and Standard Deviations of Survey Items Used to Construct

Faculty Participation Measure, 2003 and 2005

Year Faculty Participation Items N Mean

Standard

Deviation

Percentage of teachers who participate in reform 694 0.89 0.21

All teachers participate in reform (yes/no) 759 0.80 0.40

Participation is not limited by subject-specific reform 759 0.79 0.41

2003

Participation is not limited because of funding 760 0.93 0.25

All teachers participate in reform (yes/no) 718 0.78 0.42

2005 Participation is not limited by subject-specific reform 718 0.90 0.30

Participation is not limited because of funding 718 0.88 0.33

Exhibit highlights: The survey items used to measure faculty participation differed slightly

from 2003 to 2005. The reader may convert any of the means into a percent by multiplying

the mean value by 100. For example, the mean score of 0.80 for whether all teachers

participate in reform in 2003 indicates that 80 percent of respondents indicated that all

teachers participate in reform.

Note: Means and standard deviations are computed from respondents’ answers to questions

about the existence of the faculty participation items (0=No; 1=Yes).

Source: ECSRIO surveys.

Component 6: District Support

In both years, the district support measure was a combination of six items pertaining to the types

of district support (Exhibit 16): conducting a needs assessment, providing additional staff to

support reform efforts, selecting a reform strategy, writing grant proposals, providing

professional development, and providing release time for teachers to support the reform. Each of

these items was a yes or no question. In each year, the non-missing scores were averaged, and a

school with an average score of 0.8 or higher was counted as implementing this component.
Chapter IV 43

Exhibit 16

Means and Standard Deviations of Survey Items Used to Construct

the District Support Measure, 2003 and 2005

2003 2005

District Support Items Mean

Standard

Deviation Mean

Standard

Deviation

Needs Assessment 0.45 0.50 0.54 0.50

Additional Staff 0.40 0.49 0.45 0.50

Selecting a Reform Model 0.37 0.48 0.43 0.49

Writing Grant Proposals 0.58 0.49 0.57 0.49

Professional Development 0.77 0.42 0.81 0.39

Release Time for Teachers 0.61 0.49 0.63 0.48

Exhibit highlights: The survey items used to measure district support in 2003 and 2005 were

the same. All district item means increased except for writing grant proposals from 2003 to

2005. The reader may convert any of the means into a percent by multiplying the mean value

by 100. For example, the 2003 needs assessment score of 0.45 indicates that 45 percent of

respondents reported they received district assistance for their needs assessment.

Note: Means and standard deviations are computed from respondents’ answers to questions

about the existence of the various district support items (0=No; 1=Yes). There are 766

observations in 2003 and 715 observations in 2005.

Source: ECSRIO surveys.

Component 7: Parental Involvement

In both years, the parental involvement measures were calculated from the same five items.

Principals were asked to estimate the percentage of parents who participated in their schools

through parent-teacher conferences, by demanding frequent reports, actively volunteering,

observing classrooms, and who were active in Parent-Teacher Associations or Parent-Teacher

Organizations (Exhibit 17). For each year, non-missing responses were averaged, and a school

was credited as implementing the parental involvement measure if the average was equal to or

greater than 0.4.
Chapter IV 44

Exhibit 17

Means and Standard Deviations of Survey Items Used to Construct

the Parental Involvement Measure, 2003 and 2005

2003 2005

Parental Involvement Items N Mean

Standard

Deviation N Mean

Standard

Deviation

Parent-Teacher Conferences 750 0.64 0.26 781 0.57 0.28

Demanding Frequent Reports 699 0.36 0.32 782 0.30 0.22

Actively Volunteering 736 0.16 0.17 779 0.12 0.11

Observing Classrooms 704 0.11 0.16 775 0.09 0.12

Active in PTA/PTO 707 0.16 0.19 769 0.14 0.14

Exhibit highlights: The survey items used to measure parental involvement in 2003 and 2005

were the same. The mean of all parental involvement items decreased from 2003 to 2005. The

reader may convert any of the means into a percent by multiplying the mean value by 100.

For example, the 2003 parent-teacher conferences mean of 0.64 indicates that principals

reported, on average, that 64 percent of parents attended parent-teacher conferences.

Note: Means and standard deviations are computed from respondents’ estimates of the

percentage of parents who participated in school activities.

Source: ECSRIO surveys.

Component 8: External Assistance

The 2003 and 2005 measures of external assistance were constructed from different survey

questions (Exhibit 18), and any year-to-year comparisons should be made with caution. The

2003 measure was constructed from six yes or no answers on the types of external assistance

available. Non-missing responses were averaged, and a school was credited as implementing the

external assistance measure if at least 80 percent of the non-missing items were reported as

implemented.

The 2005 survey included an external assistance measure comprising four items on the number

of reform training hours received, if strategies are included in the curriculum, if all teachers

receive the training, and if support is ongoing. First, teachers were asked how many hours of

training on reform they had received. The highest number reported was 959 hours. Some

respondents may have interpreted this question as meaning over the life of the reform program,

while others may have interpreted this as the number of hours in the last year. Also, because

there is often high teacher turnover in lower-performing schools, some respondents may have

only been in their school for a portion of the CSR award implementation period. However, to

scale this number to vary between 0 and 1, it was divided by 1,000 and thus varies between 0 and

0.96. The next three questions were asked up to four times, once for each reform identified in the

school. The responses for each question were averaged to derive a school-level measure for each

item that varied between 0 and 1. The external assistance measure was constructed by taking the

average of the non-missing four items, and schools with an average of at least 0.7 were counted

as having implemented this item.
Chapter IV 45

Exhibit 18

Means and Standard Deviations of Survey Items Used to Construct

the External Assistance Measure, 2003 and 2005

Year External Assistance Items N Mean

Standard

Deviation

On-site Consulting 769 0.71 0.46

Professional Development 769 0.90 0.31

Networking 769 0.56 0.50

Written Materials for Students 769 0.44 0.50

Written Materials for Teachers 769 0.67 0.47

2003

Software or Technology 769 0.46 0.50

Hours (1,000) of Training on Reform Strategy 959 0.05 0.09

Strategies Include Curriculum Materials 696 0.65 0.42

All Teachers Received Training on all Strategies 695 0.72 0.39

2005

Ongoing Support is Available 694 0.77 0.37

Exhibit highlights: The survey items used to construct the external assistance measure were

different in 2003 and 2005. The reader may convert any of the means into a percent by

multiplying the mean value by 100. For example, the mean score of 0.71 for whether onsite

consulting was available in 2003 means that 71 percent of respondents indicated that on-site

consulting was available.

Note: Means and standard deviations are computed from respondents’ answers to questions

about the existence of the various external assistance items (0=No; 1=Yes).

Source: ECSRIO surveys.

Component 9: Evaluation

The 2003 and 2005 measures for evaluation were nearly the same, but there was one item in the

2003 measure that was not asked of respondents in 2005 (Exhibit 19): whether there is a formal

written plan to evaluate progress. All of the questions asked in both years were yes or no

questions. In each year, non-missing responses were averaged and a school was credited as

implementing the evaluation component if the average score was 1.0 in 2003 and at least 0.75 in

2005. This distinction was made due to the one item not present in the 2005 survey and the large

drop in several of the items from 2003 to 2005.
Chapter IV 46

Exhibit 19

Means and Standard Deviations of Survey Items Used to Construct

the Evaluation Measure, 2003 and 2005

2003 2005

Evaluation Items N Mean

Standard

Deviation N Mean

Standard

Deviation

School Improvement Process has

Mechanism for Periodic Evaluation 764 0.85 0.36 717 0.91 0.28

Evaluation Plan Topics

Student Performance 752 0.96 0.19 722 0.86 0.35

Program Implementation 752 0.74 0.44 722 0.62 0.48

Parental Participation 752 0.63 0.48 722 0.92 0.28

Staff Development 752 0.86 0.35 722 0.46 0.50

Utility of External Assistance 752 0.34 0.48 722 0.35 0.48

Sources of Financial Support 752 0.44 0.50 722 0.06 0.23

Formal Written Plan to Evaluate

Progress 756 0.85 0.36 NA NA NA

Exhibit highlights: The survey items used to construct the evaluation items were the same in

2003 and 2005, except in 2005, the survey did not ask about a formal written plan to evaluate

progress. The reader may convert any of the means into a percent by multiplying the mean

value by 100. For example, the mean score of 0.85 for whether the school improvement

process (SIP) has a mechanism for periodic evaluation in 2003 indicates that 85 percent of

respondents said that the SIP did have such a mechanism.

Note: Means and standard deviations are computed from respondents’ answers to questions

about the existence of the various evaluation items (0=No; 1=Yes).

Source: ECSRIO surveys.

Component 10: Coordination of Resources

The 2003 and 2005 measures for the coordination of resources component were constructed from

different survey responses (Exhibit 20), and any year-to-year comparisons should be made

cautiously. All of the questions asked in both years were yes or no questions. In each year, nonmissing responses were averaged, and a school was credited as implementing the evaluation

component if the average score was 1.0 in 2003 and at least 0.7 in 2005. This distinction was

made due to the increase in the types of funds that were asked about in 2005. That is, some

schools may not have received foundation grants or local donations and may have reported on

their survey that they did not have control over these funds.
Chapter IV 47

Exhibit 20

Means and Standard Deviations of Survey Items Used to Construct

Coordination of Resources Measure, 2003 and 2005

Year Coordination of Resources Items N Mean

Standard

Deviation

School Has Control Over Budgetary Issues 754 0.77 0.42

School Has Control Over Personnel Decisions 751 0.78 0.42

How Existing Resources Have Been

Coordinated

Align District Professional Development 754 0.86 0.35

Align Title I Activities 754 0.69 0.46

Align Other Funds 754 0.31 0.46

2003

Reallocate Staff 754 0.40 0.49

School Has Control Over the Following

Resources

Federal CSR Funds 704 0.52 0.50

Title I Funds 704 0.68 0.47

Other Federal Funds 704 0.33 0.47

Special State Grants 704 0.40 0.49

Discretionary District Funds 704 0.59 0.49

Foundation Grants 704 0.25 0.43

2005

Local Community or Business Donations 704 0.62 0.49

Exhibit highlights: The survey items used to construct the coordination of resources measure

were different in 2003 and 2005. The reader may convert any of the means into a percent by

multiplying the mean value by 100. For example, the mean score of 0.77 for the item

indicating that the school has control over budgetary issues in 2003 indicates that 77 percent

of respondents said that the school did have such control.

Note: Means and standard deviations are computed from respondents’ answers to questions

about the existence of the various items used to measure the coordination of resources (0=No;

1=Yes).

Source: ECSRIO surveys.

FINDINGS

Were schools that received CSR awards more likely to implement the legislatively

specified components of CSR than other schools?

 No, both CSR and non-CSR schools implemented an average of fewer than four

components in 2003 and fewer than five in 2005 at both the elementary and middle

school levels.

Both CSR and non-CSR schools implemented an average of fewer than four components in 2003

and fewer than five in 2005 at both the elementary and middle school levels (Exhibit 21). At the

elementary school level, CSR schools reported slightly greater implementation (the difference is
Chapter IV 48

statistically significant but substantively small) than did non-CSR schools in 2003. Three years

after initial funding, both CSR and non-CSR schools reported implementing a more

comprehensive reform (i.e., one that uses more components) in that the number of components

implemented increased by 0.9 in CSR schools and 1.1 in non-CSR schools.

At the middle school level, CSR and non-CSR schools in both 2003 and 2005 reported

implementing about the same number of components (Exhibit 21). Both reported statistically

significant similar increases in the number of components implemented between these periods.

Exhibit 21

Average Number of CSR Components Implemented by CSR

and Non-CSR Schools in 2003 and 2005

Elementary Schools Middle Schools

School Type 2003 2005

Change

from 2003

to 2005 2003 2005

Change

from 2003

to 2005

CSR Schools 3.8 4.7 0.9** 2.7 4.0 1.3**

Non-CSR Schools 3.3 4.4 1.1** 3.1 4.4 1.3**

Difference Between CSR

and Non-CSR Schools 0.5* 0.3 0.2 0.4 0.4 0.0

Exhibit highlights: In 2003, CSR elementary schools reported implementing a somewhat

higher average number of components than non-CSR schools. No other significant differences

between CSR and non-CSR schools were found. Both CSR and non-CSR schools reported

similar increases between 2003 and 2005 in the number of components implemented.

Note: + p<.10; * p<.05; ** p<.01. N = 292 for CSR Elementary Schools; N = 304 for nonCSR Elementary Schools; N = 128 for CSR and non-CSR Middle Schools. Standard errors

for elementary schools are included in Exhibit B.5; standard errors for middle schools are in

Exhibit B.6.

Source: ECSRIO surveys.

While the average number of components reported implemented in CSR and non-CSR schools

was similar, the distribution of components implemented indicates some modest difference in

comprehensiveness between CSR and non-CSR schools. Somewhat more non-CSR than CSR

elementary schools implemented between zero and three components in 2005 (Exhibit 22), while

the proportion of CSR elementary schools that implemented seven components is higher than

non-CSR elementary schools. At the middle school level, CSR schools outnumber non-CSR

schools at all levels above four components implemented (Exhibit 23).
Chapter IV 49

Exhibit 22

Number of Components Implemented in CSR and Non-CSR

Elementary Schools in 2005

37%

44%

19%

33%

44%

23%

0% 10% 20% 30% 40% 50%

0 to 3

4 to 6

7 and higher

Number of Components 

Implemented

Percentage of Schools

CSR

Non-CSR

Exhibit highlights: A slightly higher percentage of CSR elementary schools reported

implementing seven or more components in 2005 than did non-CSR schools. The same percentage

of CSR and non-CSR schools implemented four to six components. A slightly higher percentage

of non-CSR schools than CSR schools implemented zero to three components.

Source: ECSRIO surveys.
Chapter IV 50

Exhibit 23

Number of Components Implemented in CSR and Non-CSR

Middle Schools in 2005

43%

41%

16%

38%

44%

18%

0% 10% 20% 30% 40% 50%

0 to 3

4 to 6

7 and higher

Number of Components 

Implemented

Percentage of Schools

CSR

Non-CSR

Exhibit highlights: A slightly higher percentage of CSR middle schools implemented seven or

more components in 2005 than did non-CSR schools. A slightly higher percentage of CSR middle

schools reported implementing at least four components. The percentage of non-CSR schools

implementing zero to three components was 5 percentage points higher than that of CSR middle

schools.

Source: ECSRIO surveys.

Three reasons may explain the similar implementation gains in both CSR and non-CSR school

reform models. First, as shown in three of the case study sites, a school’s receipt of a CSR award

can allow its school district to shift resources away from that school and toward other non-CSR

schools also in need of improvement. Second, NCLB was in its first full year in 2002–03; newly

available assessment data may have prodded school districts to assist schools identified as lowperforming. Third, as school districts gained experience with implementing CSR, they may have

taken the lessons learned from those schools and applied them to non-CSR schools. This

behavior was evident in two of the case study sites.

Was fidelity of CSR implementation associated with gains in school-level mathematics

and reading achievement?

 The comprehensiveness of implementation, as measured by the number of CSR

components implemented, was not related to mathematics and reading achievement gains

in CSR schools.

The implementation analyses focus on assessing how a school’s overall commitment to CSR is

related to school-level achievement outcomes, rather than on the effects of individual CSR
Chapter IV 51

components or particular component combinations on achievement. Regression analyses were

used to examine the relationship between the number of components survey respondents reported

as being implemented in their school as a measure of comprehensiveness and achievement

outcomes and how the receipt of a CSR award may have moderated this relationship. These

analyses relied on the ECSRIO sample of schools from the 2002 cohort of CSR recipients and

the matched non-CSR comparison schools. Ordinary Least Squares (OLS) regression analyses

predicted achievement in 2005 from the number of components implemented as reported on the

2005 surveys, controlling for achievement in 2002 and demographics in 2005.21 These regression

analyses also used the interaction between the receipt of a CSR award and the number of

components implemented to test whether implementation comprehensiveness had different

effects on achievement for CSR and non-CSR schools. These analyses, then, captured whether

differences existed between CSR and non-CSR schools in the relationship between component

implementation and achievement, controlling for school demographics. Equation 1 presents the

model specifications for the analyses of CSR reform implementation comprehensiveness.

Equation 1

Y y FRL Black HISP CSR COMP (CSR *COMP) 2005 = α + β1 2002 + β 2 + β3 + β 4 + β5 + β6 + β7

where

Y2005 is the school’s standardized achievement score in 2005;

y2002 is the school’s standardized achievement score in 2002;

FRL is the percent free or reduced-price lunch eligible in 2005;

Black is the percent black in 2005;

HISP is the percent Hispanic in 2005;

CSR is a 0/1 variable that indicates whether the school received a CSR award in 2002;

COMP is the number of components a school reported implementing in 2005;

CSR*COMP is the interaction between a school receiving a CSR award in 2002 and the number

of components the school implemented in 2005; and

β1, β2, ..., β7 are the unstandardized regression coefficients for each of the variables.

Differences Between Included and Excluded Schools

Before introducing the regression results, it is important to note that there were several

significant differences between schools included and those excluded from these analyses

(Exhibit 24). These differences warrant additional caution when interpreting the results.

 21 The regression analyses relied on robust standard errors clustered at the state level.
Chapter IV 52

Specifically, elementary schools included in the analyses had significantly higher achievement in

both 2002 and 2005 and somewhat lower percentages of students eligible for free or reducedprice lunches (FRL) than those schools that were excluded from the analyses because of missing

data. In addition, CSR elementary schools excluded from the analyses had significantly lower

mathematics and reading achievement in 2005 than the CSR schools included in the analyses

(Exhibit 25). CSR middle schools excluded from the analyses had slightly lower mathematics

achievement than CSR schools included in the analyses. No other significant differences existed

at the middle school level (Exhibit 26).

Exhibit 24

Differences in Achievement and Demographic Characteristics Between

Schools Included and Excluded in Analyses of the Relationships

Between Implementation and Achievement, 2002 and 2005

Elementary School Middle School

Excluded Included Difference Excluded Included Difference

Standardized Achievement Score

Mathematics Achievement,

2002 -0.84 -0.56 -0.28** -0.77 -0.67 -0.10

Mathematics Achievement,

2005 -0.76 -0.45 -0.31** -0.75 -0.62 -0.13

Reading Achievement, 2002 -0.90 -0.58 -0.32** -0.82 -0.61 -0.21

Reading Achievement, 2005 -0.77 -0.46 -0.31** -0.77 -0.63 -0.14

Demographic Characteristics

Percent FRL, 2005 0.61 0.69 -0.08** 0.64 0.66 0.02

Percent Black, 2005 0.35 0.33 0.02 0.34 0.32 0.02

Percent Hispanic, 2005 0.20 0.20 0.00 0.19 0.23 -0.04

Exhibit highlights: Elementary schools included in the analyses had higher 2002 and 2005

mathematics and reading achievement than those schools excluded from the analyses.

Furthermore, schools excluded from the analyses had somewhat lower percent free or

reduced-price lunch (FRL) than the included schools. No other differences existed between

excluded and included schools in either school level.

Note: + p<.10; * p<.05; ** p<.01. Standard errors are included in Exhibit B.7.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal

School-Level State Assessment Score Database (NLSLSASD).
Chapter IV 53

Exhibit 25

Differences in Standardized Mathematics and Reading Achievement Between CSR

and Non-CSR Elementary Schools Included and Excluded in Analyses of the

Relationships Between Implementation and Achievement, 2005

Subject and School Type Excluded Included Difference

Mathematics

CSR Schools -0.98 -0.49 -0.44**

Non-CSR Schools -0.57 -0.41 -0.15

Difference -0.41** -0.08

Reading

CSR Schools -0.96 -0.54 -0.42**

Non-CSR Schools -0.61 -0.38 -0.23*

Difference -0.34* -0.16+

Exhibit highlights: CSR elementary schools excluded from the analyses had substantially

lower mathematics and reading achievement than CSR schools included in the analyses as

well as non-CSR schools excluded from the analyses.

Note: + p<.10; * p<.05; ** p<.01. Measures in table are standardized achievement scores.

Standard errors are included in Exhibit B.8.

Sources: ECSRIO surveys; CSR Awards Database; Common Core of Data (CCD); National

Longitudinal School-Level State Assessment Score Database (NLSLSASD).
Chapter IV 54

Exhibit 26

Differences in Standardized Mathematics and Reading Achievement Between CSR

and Non-CSR Middle Schools Included and Excluded in Analyses of the

Relationships Between Implementation and Achievement, 2005

Subject and School Type Excluded Included Difference

Mathematics

CSR Schools -0.91 -0.63 -0.28+

Non-CSR Schools -0.61 -0.63 0.02

Difference -0.30 0.00

Reading

CSR Schools -0.80 -0.72 -0.08

Non-CSR Schools -0.75 -0.54 -0.21

Difference -0.05 -0.18

Exhibit highlights: CSR middle schools excluded from the analyses had slightly lower

mathematics achievement than CSR schools included in the analyses. All other differences

were not statistically significant.

Note: + p<.10; * p<.05; ** p<.01. Measures in table are standardized achievement scores.

Standard errors are included in Exhibit B.9.

Sources: ECSRIO surveys; CSR Awards Database; Common Core of Data (CCD); National

Longitudinal School-Level State Assessment Score Database (NLSLSASD).

Comprehensiveness of CSR Reform Implementation and Achievement Outcomes

Comprehensiveness, as measured by the number of CSR components a school implemented, was

not related to achievement gains in elementary or middle schools (Exhibit 27). CSR posits that

schools that more comprehensively implement CSR should have higher achievement growth

than schools that implement a less comprehensive approach. More comprehensive CSR

implementation implies that the CSR award affects school operations in multiple areas, such as

professional development, parental involvement, school planning, and external support. The

synergies that arise from this comprehensive implementation should result in greater

opportunities for learning compared with a piecemeal implementation across a few areas of

school operations. However, when prior achievement and demographics were controlled for, the

number of components implemented in 2005 was not associated with achievement gains in CSR

schools. Our case studies also showed varied patterns in the relationship between component

implementation and achievement. Of the 15 case study CSR schools, five that had implemented

over four components improved in both mathematics and reading, two improved in either

reading or mathematics, while eight failed to improve in either subject.
Chapter IV 55

Exhibit 27

Relationships Between Number of CSR Components Implemented

and Achievement Gains from 2002 to 2005

Elementary School Middle School

Effect Mathematics Reading Mathematics Reading

Constant 0.42* 0.59** 0.23 0.12

Achievement (2002) 0.45** 0.47** 0.79** 0.80**

Percent FRL (2005) -0.76** -0.90** -0.44 -0.39

Percent Black (2005) -0.07 -0.08 -0.31 -0.01

Percent Hispanic (2005) -0.02 -0.28+ -0.05 0.25

CSR Award (0=No, 1=Yes) -0.04 -0.15 0.07 -0.14

Number of Components,

CSR and Non-CSR Schools -0.02 -0.02 0.01 -0.02

Number of Components,

CSR Schools Only 0.03 0.04 0.01 0.04

R-Squared 0.34 0.39 0.62 0.61

N 349 353 129 131

Exhibit highlights: There is no relationship between the number of CSR components

implemented and mathematics and reading achievement gains at the elementary or middle

school levels, controlling for school demographics and baseline achievement.

Note: + p<.10; * p<.05; ** p<.01. Percent FRL, percent black, and percent Hispanic have

been rescaled to a ratio varying between 0 and 10. The “Number of Components, CSR and

Non-CSR Schools” row presents the change in achievement for implementing one more

component among both CSR and non-CSR schools. The “Number of Components, CSR

Schools Only” row presents the change in achievement among CSR schools when

implementing one more component over and above the effects seen among all schools in

previous first row. Standard errors are included in Exhibit B.10.

Sources: ECSRIO surveys; CSR Awards Database; Common Core of Data (CCD); National

Longitudinal School-Level State Assessment Score Database (NLSLSASD).

DISCUSSION

What might explain the overall weak relationship between implementation and achievement in

this study, especially given stronger relationships found in previous work? One likely

explanation is that prior research (e.g., Borman et al., 2003; Aladjem and Borman, 2006;

Aladjem et al., 2006) focused on the adoption and implementation of clearly defined researchbased reform models. The difference in the fairly strong findings from these works and the

weaker conclusions in this report may lie in the large number and representativeness of schools

included in this study and the relatively small percentage of these schools that implemented all,

or even a majority of, the CSR components. That is, those studies focused on models with

somewhat high fidelity of implementation. In this study, the average number of components
Chapter IV 56

implemented after three years did not exceed five, and less than one in five schools implemented

more than seven components. The relative lack of schools implementing a majority of the

components may result in the analysis not being sensitive enough to identify any relationship.

One high-implementing site observed in our case studies illustrates how the components can

work together to improve achievement. School I is a rural, pre-K school located in the Southeast.

The state, as do other states, mandated a school improvement process (SIP) as a comprehensive

strategy for increasing achievement (Component 2). School I faculty and staff were active

participants in the SIP process (Component 5). As a result, the SIP was considered to be a “living

document,” as indicated by the handwritten comments included in monthly revisions.

Furthermore, staff members used student assessment outcomes (Component 4), input from

parents (Component 7), and student surveys and interviews to measure the needs,

accomplishments, and perceptions of stakeholders. From 2002–05, School I’s teachers

participated in formal professional development associated with the reform strategies adopted by

the school (Component 3). Teachers also participated in grade-level teacher meetings, crossgrade planning meetings, and school improvement committee meetings to adjust the instructional

methods used. In 2003–04, the teachers began participating in a district-designed professional

development strategy, which required them to observe instructional strategies implemented at

other schools (Component 6). By 2003–04, the school moved out of corrective action status by

making Adequate Yearly Progress (AYP) for two years. As illustrated by the survey results,

however, this case is the exception rather than the rule.

Another explanation may be in the length of time needed for the achievement impacts of

comprehensive school reform to become manifest. Other studies (as documented by Borman et

al., 2003) have typically found the strongest effects for CSR in the fifth year and beyond, while

the time period covered in this study is only three years.22

 22 The lack of a relationship between implementation of CSR after three years and achievement may also

partly be explained by a reduction in the sample size due to nonresponse and missing survey data.
Chapter V 57

V. CSR SCHOOLS’ ADOPTION OF A SCIENTIFICALLY BASED

RESEARCH MODEL AND ITS RELATIONSHIP TO ACHIEVEMENT

NCLB emphasizes the adoption of reform methods with strong research-based evidence of

improving achievement. Specifically, two components described in NCLB emphasize the use of

methods and strategies that are supported by scientifically based research:

• Proven methods and strategies for student learning, teaching, and school

management that are based on scientifically based research and effective practices

and have been replicated successfully in schools with diverse characteristics.

• Scientifically based research to significantly improve the academic achievement

of students participating in such programs as compared with students in schools

who have not participated in such programs; or strong evidence that such programs

will significantly improve the academic achievement of participating children.

Key Findings

What was the distribution of CSR schools across CSRQ scientific research

base ratings?

 Only one-third of 2002 CSR awardees chose reform approaches with

recognized scientific research bases.

Was the use of a scientifically based research model related to positive

achievement outcomes? Did low-performing schools that adopted models

with stronger scientific research bases have different achievement gains

than other schools?

 Low-performing elementary schools that adopted models with stronger

evidence of effectiveness had gains in mathematics achievement that were

not found in higher-performing schools.

 Adoption of a CSR model independently determined to have had limited

scientific evidence of effectiveness was associated with higher gains in

middle school mathematics achievement in all CSR schools, whether they

were low-performing or not. There also was weaker evidence that CSR

middle schools that adopted models with limited scientific evidence may

have experienced gains in middle school reading achievement relative to

schools that adopted other models.

 There was weaker evidence that low-performing middle schools that

adopted models with moderate or higher bases of evidence showed

improvement in mathematics compared with schools using other models.

 In no other instances was adoption of models with a scientific research

base related to achievement gains.
Chapter V 58

Within the 11 CSR components, two focus on the use of programs with a scientific research

base. Focusing on the universe of 2002 CSR awardees, this evaluation examines the extent to

which CSR schools adopted scientifically based research models and whether such models

affected achievement outcomes.

The CSR Awards Database identifies 1,340 schools that received a CSR award in 2002. As with

the implementation analyses that examine the relationship between comprehensiveness and

achievement, the outcome measures for these analyses rely on standardized school-level

achievement scores from NLSLSASD. Schools that implemented CSR models for fewer than

three years were eliminated from the analyses (N = 60). The analyses are restricted to schools

with full data, the number of which is provided in the results tables by school level and subject

area for each analysis.

MEASURES OF THE SCIENTIFIC RESEARCH BASE OF CSR MODELS

The measure of any given reform method’s scientific strength was derived by using rating scales

from two Comprehensive School Reform Quality Center (CSRQ) reports on CSR reform models

(CSRQ, 2006a; 2006b).23 The CSRQ ratings provided a scale for the breadth and quality of the

research base for the 31 most widely adopted CSR models.24 The final CSRQ sample of

reviewed models was based on three factors: 1) total number of schools implementing the model

nationally; 2) replicability of the CSR model; and 3) comprehensiveness of model design. CSRQ

ratings for the evidence of positive effects on student achievement were based on a rigorous,

systematic review of the current literature on CSR models. The ratings were based on the number

of published research studies meeting standards for rigorous research; the number of studies that

provide conclusive versus suggestive findings for achievement; and the percentage of findings in

either category that demonstrate a positive effect on achievement. (The CSRQ Center Quality

Review Tool is presented in Appendix A.)

CSRQ recently reviewed the evidence of positive effects on achievement for 31 of the most

widely adopted CSR models for elementary and secondary levels (Exhibit 28). The

Comprehensive School Reform Quality Center (CSRQ) reports were not available at the time

that 2002 CSR awardees made their model selections; however, schools may have used data

from two earlier reviews of CSR models: Herman (1999), and Borman et al. (2003). The CSRQ

reports draw heavily from the data used in previous reviews of CSR models.

These models addressed multiple aspects of school practices. Note, though, that most of the 2002

cohort of CSR schools chose models that were not subsequently rated in the CSRQ analyses.

Among elementary schools, one-fourth chose models that received a moderate or moderately

strong rating; most chose School Renaissance, Success for All, or America’s Choice. Eleven

percent of elementary schools chose models that later received limited ratings; Pearson

Achievement Solutions (earlier known as Co-Nect) was the most frequently adopted of these.

Among middle schools, just over 10 percent chose models that received a moderate or higher

 23 The CSRQ Center is funded under the U.S. Department of Education’s Comprehensive School Reform

Quality Initiatives program.

24 The CSRQ Center conducted a total of 40 ratings because some models were rated at both the elementary

and secondary levels.
Chapter V 59

rating (primarily America’s Choice, Success for All, and School Development Program), while

less than 5 percent chose models that received a limited rating (with most choosing Middle

Start).
Chapter V 60

Exhibit 28

Number and Percentage of Schools Using CSR Models Included in CSRQ Reports

Elementary School Middle School

Model Rating School Level N % N %

Moderately Strong

Direct Instruction (Full Immersion Model) Elementary 30 3.5%

Success for All Elementary 54 6.2%

Moderate

Accelerated Schools PLUS Elementary 12 1.4%

America’s Choice Both 46 5.3% 20 4.1%

Core Knowledge Elementary 13 1.5%

First Things First Secondary 0 0.0%

Literacy Collaborative Elementary 13 1.5%

National Writing Project Elementary 4 0.5%

School Development Program Both 14 1.6% 12 2.5%

School Renaissance Elementary 54 6.2%

Success for All Secondary 18 3.7%

Talent Development High Schools Secondary 2 0.4%

Total Moderate and Moderately Strong 240 27.6% 52 10.7%

Limited

ATLAS Learning Communities Elementary 5 0.6%

Different Ways of Knowing Elementary 7 0.8%

Expeditionary Learning Secondary 5 1.0%

Integrated Thematic Instruction Elementary 13 1.5%

Knowledge Is Power Program (KIPP) Secondary 0 0.0%

Middle Start Secondary 16 3.3%

Modern Red School House Elementary 17 2.0%

More Effective Schools Secondary 0 0.0%

Pearson Achievement Solutions (formerly Co-Nect) Elementary 37 4.3%

Project GRAD Secondary 0 0.0%

Ventures Initiative and Focus System Elementary 17 2.0%

Total Limited 96 11.0% 21 4.3%

Zero

Accelerated Schools PLUS Secondary 3 0.6%

ATLAS Learning Communities Secondary 7 1.4%

Breakthrough to Literacy Elementary 11 1.3%

Coalition of Essential Schools Both 13 1.5% 5 1.0%

Community for Learning Elementary 1 0.1%

Comprehensive Early Literacy Elementary 1 0.1%

Expeditionary Learning Elementary 9 1.0%

First Steps Elementary 2 0.2%

High Schools That Work Secondary 3 0.6%

Making Middle Grades Work Secondary 4 0.8%

Modern Red School House Secondary 3 0.6%

Onward to Excellence II Both 6 0.7% 8 1.7%

Turning Points Secondary 30 6.2%

Total Zero 43 4.9% 63 13.0%

Total Rated Models 379 43.6% 136 28.1%

Other (Unrated) Models 490 56.4% 348 71.9%

Exhibit highlights: More than one-fourth of elementary schools, and just over 10 percent of middle schools, chose

CSR models with a moderate or moderately strong research base. Well over half of elementary schools, and nearly

three-quarters of middle schools, chose CSR models that were not subsequently rated in CSRQ Center reports.

Sources: CSRQ Center Report on Elementary CSR Models; CSRQ Center Report on Middle and High School CSR

Models; CSR Awards Database.
Chapter V 61

The CSRQ builds from work by Borman, Hewes, Overman, and Brown (2002) and Herman

(1999) “to quantitatively evaluate CSR models as well as to provide qualitatively a narrative

description of each reviewed model” (CSRQ, 2006a, p. 23). Furthermore, Kidron and Darwin

(2007) state the CSRQ systematic review approach is aligned with standards articulated in a

2003 Institute of Education Sciences’ technical report (Coalition for Evidence-Based Policy,

2003).

CSRQ authors conducted a systematic review (CSRQ, 2006a) of the research base on

comprehensive school reform models consisting of five steps. The first step was a literature

search. In order to be considered for the review, the full text publication needed to be released

between 1980 and April 2005, must have examined a CSR model, and must have used

quantitative methods. The study team then completed a Study Description Outcome Form, which

coded studies based on the research design, outcome variables, and demographic information. At

this stage of the analysis, researchers considered the use of quantitative data and the strength of

the study design in determining the inclusion of studies for further review. “Research designs

that passed this stage included experimental designs and quasi-experimental research designs

with both pre- and posttests that evaluated the CSR model with a control group...and

longitudinal and cohort designs with multiple testing periods” (CSRQ, 2006a, p. 27). Acceptable

quasi-experimental designs needed to appropriately account for non-equivalence of comparison

groups through statistical matching or employing statistical controls during data analysis.

The next step included the use of a Quality Indicators Form (QLIF — see Appendix A.1). Two

independent researchers used this form to assess the quality of the research based on factors such

as internal and external validity, and face and psychometric validity of the outcome measures.

The researchers also used this form to gather statistical information, including effect sizes and

other raw statistical information. The absence of valid equivalent comparison groups, failure to

assess the degree and quality of implementation, or invalid assessment measures were all

considered to be critical threats to validity and precluded a study’s inclusion in the later stages of

the review. The research team then reconciled any disagreement between the QLIF coders, and

rated the study’s overall causal validity as inconclusive, suggestive, or conclusive. Suggestive

and conclusive studies had zero critical threats to validity and fewer than two noncritical threats

to validity.25 Conclusive studies included experimental and quasi-experimental designs, while the

suggestive category included less rigorous studies (e.g., longitudinal and cohort research

designs). Only suggestive and conclusive studies remained in the analysis.

Individual study ratings were then synthesized for each comprehensive reform model to

determine the overall strength and nature of the scientific evidence regarding model effects on

student achievement. The following categories were used in rating the evidence behind the

models: Very Strong, Moderately Strong, Moderate, Limited, Zero, Negative, and No Rating

(see Exhibit 29). No model reviewed by CSRQ met the criteria for very strong evidence of

positive effects on student achievement, although several models had a large enough body of

rigorous research to be categorized as having moderate or moderately strong effects.

 25 Examples of noncritical threats to validity included differential attrition, changes in instrumentation,

disruptive or novelty events, maturation, and mismatches between level of assignment and unit of analysis

(Crowley and Hauser, 2007).
Chapter V 62

Exhibit 29

Description of CSRQ Rating Scale for Strength of Research Base

Very Strong: Minimum of 10 studies meeting CSRQ standard for rigor of research design, with at

least 50 percent presenting conclusive results, and 75 percent showing positive effect sizes of +.25 or

higher.

Moderately Strong: Five to nine studies meeting CSRQ standard for rigor of research design, with a

minimum of 50 percent presenting conclusive results, and 51–75 percent showing positive effect sizes

of +.20 to +.24 or higher.

Moderate: Two to four studies meeting CSRQ standard for rigor of research design, with minimum of

50 percent presenting conclusive results, and 26–50 percent showing positive effect sizes of +.15 to

+.19 or higher.

Limited: One study meeting CSRQ standard for rigor of research design, and 1–25 percent showing

positive effect sizes at p< .05.

Zero: No studies meeting CSRQ standard for rigor of research design, or 0 percent showing

statistically significant effects for achievement outcomes.

Negative: Minimum of 10 studies meeting CSRQ standard for rigor of research design, with at least

50 percent presenting conclusive results, and 75 percent showing negative effect at p<.05.

Exhibit highlights: CSRQ rating scale for the research base demonstrating evidence of

positive student achievement effects.

Sources: CSRQ Center Report on Elementary CSR Models; CSRQ Center Report on Middle

and High School CSR Models.

As with all analyses that rely on secondary data sources, the analyses examining the relationship

between scientifically based research model adoption and achievement are contingent on the

quality of data collection efforts and analyses conducted by external entities. As a widely

distributed CSR model guidance tool for schools, the CSRQ reports aim to provide evidence

from the research field in a consumer-friendly format, but the reports are not without their

limitations, as acknowledged by the researchers and the broader education research community.

For example, models that have shown promising effects over time are likely to be studied more

often than other models in an effort to examine replicability of positive interventions across

different populations. Given that the CSRQ ratings are determined in part by the number of highquality studies conducted, the ratings may be biased toward those models that have demonstrated

a more positive pattern of findings.
Chapter V 63

FINDINGS

What was the distribution of CSR schools across CSRQ scientific research base ratings?

 Only one-third of 2002 CSR awardees chose reform approaches with recognized

scientific research bases.26

This finding is in line with prior research on the use of scientifically based research models that

indicates a relatively low proportion of schools choose these evidence-based methods (Center on

Education Policy, 2004). Among the CSR schools that use models with at least some evidence of

effectiveness (i.e., limited to moderately strong evidence), about one-half selected approaches

with moderate evidence of effectiveness in improving mathematics achievement, and two-thirds

selected approaches with moderate to moderately strong evidence of effectiveness in improving

reading achievement. The CSRQ reports were not available at the time that 2002 CSR awardees

made their model selections; however, schools may have used data from two earlier reviews of

CSR models: Herman (1999), and Borman, Hewes, Overman, and Brown (2003). The CSRQ

reports draw heavily from the data used in previous reviews of CSR models.

Exhibit 30

Distribution of 2002 CSR Awardees Across CSRQ Evidence Ratings

Category Mathematics Reading

N Percent N Percent

Zero 777 68 841 67

Limited Scientific Research Base 169 15 123 10

Moderate to Moderately Strong Scientific Research Base 200 17 296 23

Total 1,146 100 1260 100

Exhibit highlights: Two-thirds of schools did not adopt a scientifically based research reform

model that meets CSRQ criteria.

Sources: CSRQ Center Report on Elementary CSR Models; CSRQ Center Report on Middle

and High School CSR Models; CSR Awards Database.

The average baseline scores for schools adopting moderate to moderately strong research models

were lower than schools adopting models with more limited evidence of effectiveness (Exhibit

31).

 26 The CSRQ Center conducted a total of 40 ratings because some models were rated at both the elementary

and secondary levels. Of the 40 ratings conducted, 25 models were rated as having some evidence of

effectiveness. CSRQ relies on the number and quality of research studies of the models to develop its ratings.

Further detail on CSRQ methodology is provided in Chapter II of this report.
Chapter V 64

Exhibit 31

Average Baseline Standardized Achievement Scores for CSR Awardees

Used in Scientifically Based Research Analyses by Evidence Category, 2002

Average Baseline Standardized Achievement Scores

Elementary School Middle School

Mathematics Reading Mathematics Reading

Evidence Category N Mean N Mean N Mean N Mean

Zero 312 -0.94 377 -0.95 264 -0.81 274 -0.84

Limited Scientific Research Base 129 -0.75 81 -0.88 24 -0.89 30 -0.69

Moderate to Moderately Strong

Scientific Research Base 132 -1.13 203 -0.91 88 -1.27 51 -1.30

Exhibit highlights: The standardized achievement scores of CSR schools that adopted

models with moderate or moderately strong research bases were over one-half a standard

deviation lower in middle school reading and one-third of a standard deviation lower in

middle and elementary school mathematics than schools that adopted models with limited

evidence of effectiveness. There were no differences in baseline reading achievement by

evidence category at the elementary school level.

Sources: CSRQ Center Report on Elementary CSR Models; CSRQ Center Report on Middle

and High School CSR Models; CSR Awards Database; Common Core of Data (CCD);

National Longitudinal School-Level State Assessment Score Database (NLSLSASD).

Was the use of a scientifically based research model related to positive achievement

outcomes? Did low-performing schools that adopted models with stronger scientific

research bases have different achievement gains than other schools?

 Low-performing elementary schools that adopted models with moderate or moderately

strong evidence of effectiveness had gains in mathematics achievement that were not

found in higher-performing schools.

 Adoption of a CSR model with limited scientific evidence of effectiveness was associated

with higher gains in middle school mathematics achievement in all CSR schools, whether

they were low-performing or not. There also was weaker evidence that CSR middle

schools that adopted models with limited scientific evidence may have experienced gains

in middle school reading achievement relative to schools that adopted other models.

 There was weaker evidence that low-performing middle schools that adopted models

with moderate or moderately strong bases of evidence showed improvement in

mathematics compared with schools using other models.

 In no other instances was adoption of models with a scientific research base related to

achievement gains.

To examine the relationship between adoption of scientifically based research models and

growth in achievement, the CSRQ ratings for each of the models included in the elementary and

middle school reports were converted into a four-point scale, with 0 indicating the model did not
Chapter V 65

meet the standards for the initial CSRQ review or received a CSRQ rating of 0 for having an

insufficient number of quality research studies demonstrating effectiveness. A 3 indicated the

model received a “Moderate” or “Moderately Strong” rating.27 Of the 40 ratings conducted, 25

models were rated as having some evidence of effectiveness. No model received a “Negative” or

“Very Strong” rating. Separate codes were created for each subject area because several CSRQ

rated models, such as Literacy Collaborative and Comprehensive Early Literacy Learning, were

designed to address learning in a specific subject area.

Additional analyses were conducted to test the appropriateness of including both nonrated

models and models that received 0 ratings in the referent group. The same pattern of results

emerged. Therefore, the larger sample size was retained in the analyses to promote the

generalizability of the findings.28 Eleven percent of schools reported implementing multiple CSR

models simultaneously (N = 143). In those cases where schools adopted multiple models with

different CSRQ ratings (N = 4), schools were assigned the higher evidence rating.

The regression analyses used to examine relationships between scientifically based research

strategy adoption and school-level achievement included the rating of the scientific research base

for models as a series of dummy coded variables, with models with no research base serving as

the comparison group (Equation 2).29

Because of the differences in baseline achievement in schools that adopted models with zero,

limited, and moderate to moderately strong research bases, the analyses include a measure for

low-performing schools and interactions between the low-performing schools measure and the

measure for the scientific research base of the model chosen. This allowed for the determination

of differential achievement gains for low-performing schools that adopted models with higher

levels of scientific research bases. Low-performing schools were those in the lowest quartile of

mathematics or reading achievement in the baseline year (2001–02).30

 27 Due to the small number of middle schools in the moderately strong category with non-missing data, the

moderate and moderately strong CSRQ ratings were collapsed into a single category in the analyses.

28 The coefficients for “scientifically based research strategy” adoption were slightly smaller in the restricted

sample analyses for both middle school mathematics and reading (0.22 versus 0.26 and 0.27 versus 0.29,

respectively). In the analyses that examine the strength of the research base, the coefficients for the limited

research base category are slightly smaller for both middle school mathematics and reading (0.43 versus 0.47

and 0.40 versus 0.42, respectively) in the restricted sample analyses.

29 The regression analyses in this section relied on robust standard errors clustered at the state level.

30 The incorporation of the interaction term results in some shared variance in the estimates of the relationships

between scientifically based research and achievement. For instance, there are two estimates for limited

scientifically based research—one for all schools and one for low-performing schools that represents the

additional change in achievement over and above the estimate for all schools. Because the group of all schools

overlaps with the group of low-performing schools, the two groups share some variance in their relationship

with achievement. Substantively, this makes sense, in that there are characteristics of all CSR schools that are

shared with or similar to characteristics found in low-performing schools. This overlap is not included in any

of the regression coefficients because there is no way to tell how much of the variance in the overlap to

attribute to all schools and how much to attribute to low-performing schools.
Chapter V 66

Equation 2

Y y FRL Black HISP Lim Mod LP (LP* Lim) (LP*Mod ) 2005 =α + β1 2002 + β2 + β3 + β4 + β5 + β6 + β7 + β8 + β9

where

Y2005 is the school’s standardized achievement score in 2005;

y2002 is the school’s standardized achievement score in 2002;

FRL is the percent free or reduced-price lunch eligible in 2005;

Black is the percent black in 2005;

HISP is the percent Hispanic in 2005;

Lim and Mod are a series of dummy coded variables indicating strength of the scientifically

based research model (limited and moderate to moderately strong, respectively);

LP is a dummy coded variable for low-performing schools (those in the lowest quartile of

achievement); and,

(LP*Lim) and (LP*Mod) are the interaction terms between low-performing schools and the

scientific base rating.

The results when examining the relationship between the strength of the scientific research base

of the adopted model and achievement are mixed (Exhibit 32). In general, the results

demonstrate stronger relationships between the scientific research bases of comprehensive school

reform models and mathematics achievement gains compared to reading achievement gains.

Note that Wald tests indicate that in elementary mathematics and in both middle school

mathematics and reading, the models with the interaction terms (denoted as (2) in the exhibit)

should be used. In elementary reading, the Wald test was not significant, so the model without

the interaction terms (model (1)) should be used.

Low-performing schools that adopted models with stronger evidence of effectiveness had gains

in elementary mathematics achievement that were not found in higher-performing schools. In

elementary mathematics, low-performing schools choosing models with moderate or moderately

strong evidence had a net gain of 0.25 standard deviations, or a 4.0 percentage point gain in

percent proficient or higher.31 There was less of a gain for schools that chose a model with

 31 The net gain of 0.25 standard deviations was calculated by adding the decrease of 0.44 standard deviations

that low-performing elementary schools experience in mathematics relative to higher-performing elementary

schools to the increase of 0.69 standard deviations that low-performing schools that adopt a model with

moderate or moderately strong scientifically based research experience relative to low-performing schools that

do not adopt such models.
Chapter V 67

limited evidence, which had net increase of 0.12 standard deviations, or about a 1.9 percentage

point increase in the proportion of students scoring proficient or higher.32

In elementary reading, the Wald test indicates that model (1) should be used. In it, there were no

statistically significant relationships between the scientific research base of the model chosen and

achievement.

In middle school mathematics, CSR schools overall that chose a model with limited evidence

had a gain of 0.37 standard deviations, or about a 5.9 percentage point gain in proficiency over

schools choosing other models. Low-performing schools that chose models with moderate or

higher evidence had an average gain of 0.26 standard deviations, or about 4.2 percentage points

more than low-performing schools choosing other models. Note that this relationship is only

statistically significant at p<.10 and should be interpreted with caution.

In middle school reading, CSR schools that chose a model with limited evidence experienced a

gain of 0.21 standard deviations, or about a 3.4 percentage point increase in proficiency over

schools choosing other models. This, too, was statistically significant only at p<.10, warranting

caution in placing too much emphasis on the results.

 32 The net gain of 0.12 standard deviations was calculated by adding the decrease of 0.44 standard deviations

that low-performing elementary schools experience in mathematics relative to higher-performing elementary

schools to the increase of 0.56 standard deviations that low-performing schools that adopt a model with limited

scientifically based research experience relative to low-performing schools that do not adopt such models.
Chapter V 68

Exhibit 32

Regression Coefficients Demonstrating the Relationships Between

Scientific Research Base and Achievement Gains from 2002 to 2005

Elementary Math Elementary

Reading

Middle School

Math

Middle School

Reading

Model (1) (2) (1) (2) (1) (2) (1) (2)

Constant 0.20 0.24+ 0.20 0.22 0.15 0.16 0.13 0.16

Achievement (2002) 0.52** 0.47** 0.50** 0.47** 0.71** 0.74** 0.60** 0.66**

Percent FRL (2005) -0.03+ -0.03 -0.04* -0.04+ -0.02 -0.02 -0.02 -0.02

Percent Black (2005) -0.05** -0.05** -0.04* -0.04* -0.04** -0.04** -0.06** -0.06**

Percent Hispanic (2005) -0.04* -0.04* -0.05** -0.05** -0.03+ -0.03+ -0.05* -0.05*

Limited Scientific Research Base 0.16 0.03 0.11 -0.01 0.38** 0.37** 0.29+ 0.21+

Moderate/Moderately Strong Research Base 0.11 -0.09 0.02 -0.04 0.10 0.00 0.20+ 0.06

Low-performing Schools -0.44** -0.21+ 0.06 0.05

Limited Scientific Research Base in Low-performing Schools 0.56** 0.45+ 0.00 0.32

Moderate/Moderately Strong Research Base in Low-performing Schools 0.69* 0.23 0.26+ 0.38

r2 0.40 0.42 0.39 0.40 0.63 0.63 0.56 0.56

N 593 593 684 684 332 332 360 360

Wald Test (3 degrees of freedom) 3.43* 1.56 2.40+ 2.89+

Exhibit highlights: At the elementary level, low-performing CSR schools that adopted models with limited or moderate to moderately strong scientific research

bases had higher mathematics achievement; there was no relationship between reading achievement and models with a scientific research base. At the middle

school level, CSR schools that adopted models with limited scientific research bases had higher mathematics and reading achievement than those that did not; lowperforming CSR schools that adopted models with moderate to moderately strong research bases had higher mathematics achievement than those that chose other

models.

Note: + p<.10; * p<.05; ** p<.01. Percent FRL, percent black, and percent Hispanic have been rescaled to a ratio varying between 0 and 10. Regression coefficients

presented in the table represent the change in standard deviations resulting from a one-unit change in the explanatory variable and may thus be thought of as effect

sizes. Model 1 refers the basic model, which does not include the term for low-performing schools and the interaction terms between low-performing schools and

the strength of the research base. Model 2 refers to the full model, which includes those terms. The incorporation of interaction terms results in shared variance not

reflected in the regression estimates but included in the r2. See footnote 30 above. The Wald Test tests the hypothesis that the regression estimates that do not

include the term for low-performing schools and the interaction terms between being low-performing and adopting a model with a scientific research base (labeled

1) are different from the regression estimates that do include them (labeled 2). Each Wald Test is similar to an F-test with three degrees of freedom. A statistically

significant result implies that the analysis should include the terms for low-performing and the interaction terms; a result that is not statistically significant implies

that the analysis should not include these terms. In the four models above, the one for elementary math and the two middle school models should include the terms

for low-performing schools and the interaction terms, while the model for elementary reading should not. Robust standard errors were computed by clustering by

state. Standard errors are included in Exhibit B.11.

Sources: CSRQ Center Report on Elementary CSR Models; CSRQ Center Report on Middle and High School CSR Models; CSR Awards Database; Common Core of

Data (CCD); National Longitudinal School-Level State Assessment Score Database (NLSLSASD).
Chapter V 69

DISCUSSION

These findings overall do demonstrate some statistical association between the adoption by CSR

grantees of “evidence-based” models and achievement improvements. However, they are also

inconsistent, being more strongly manifest in mathematics than reading, and in the lowestperforming schools. Furthermore, they do not offer compelling evidence that the strength of the

evidence base is strongly and consistently associated with achievement improvements.

Why does it appear that models with a scientific research base are consistently more strongly

related to achievement gains in mathematics than reading? One possible explanation lies with the

way CSRQ determined ratings. CSRQ determined “strength of the research base” by the nature

of the evaluation design, with more weight given to more rigorous designs and the number of

models evaluated that use such designs. It is possible that some highly potent strategies had not

been evaluated rigorously, so the ratings underplay their potential for effectiveness. It is also

possible that implementation in schools in this study may have been less comprehensive than it

was in schools included in the studies that CSRQ used. Implementation problems could well

have outweighed the research base for the strategy.

In addition, the measure of rigor itself may be an issue, because the CSRQ scales were originally

developed to guide school decision-making in selecting a program model with strong scientific

evidence of effectiveness. The most rigorous research, according to the CSRQ rating system, was

that with randomized treatment (randomized controlled trials). Such research may have shown

highly positive outcomes for the groups with which the model was tested. However, the adopting

sites may have had very different populations, and the replicability in such sites could have been

limited.

As Lipsey (2003) noted, evaluations of research and demonstration programs, such as the

individual models, and what he calls “practice” programs (programs “administered by a service

agency in an ongoing fashion,” p. 75) yield consistently different results. The research and

demonstration programs show larger effects, within each type of evaluation design, than practice

programs. The results reported in this study, then, provide additional evidence underscoring the

complex association between evidence of effectiveness from evaluations of educational

interventions in demonstration programs and their adoption on a broader scale.
Chapter V 70
Chapter VI 71

VI. CONCLUSIONS AT THE END OF THREE YEARS

SUMMARY OF FINDINGS

A summary of study findings, categorized by key evaluation questions, follows:

Was Receipt of a CSR Award Associated With Improvements in School-level

Mathematics and Reading Achievement?

 Receipt of a CSR award was not associated with achievement gains in

mathematics or reading achievement through the first three years of

award.

Were Schools That Received CSR Awards More Likely to Implement the

Legislatively Specified Components of CSR Than Other Schools?

 No, both CSR and non-CSR schools implemented an average of fewer

than four components in 2003 and fewer than five in 2005 at both the

elementary and middle school levels.

Was Fidelity of CSR Implementation Associated With Gains in School-level

Mathematics and Reading Achievement?

Two analyses address this question:

Comprehensiveness of implementation

 The comprehensiveness of implementation, as measured by the

number of CSR components implemented, was not related to

mathematics and reading achievement gains in CSR schools.

Adoption of models with a recognized scientific research base

 Only one-third of 2002 CSR awardees chose reform approaches with

recognized scientific research bases.

 Low-performing elementary schools that adopted models with stronger

evidence of effectiveness had gains in mathematics achievement that

were not found in higher-performing schools.

 Adoption of a CSR model with limited scientific evidence of

effectiveness was associated with higher gains in middle school

mathematics achievement in all CSR schools, whether they were lowperforming or not. There also was weaker evidence that CSR middle
Chapter VI 72

schools that adopted models with limited scientific evidence may have

experienced gains in middle school reading achievement relative to

schools that adopted other models.

 There was weaker evidence that low-performing middle schools that

adopted models with moderate or higher bases of evidence showed

improvement in mathematics compared with schools using other

models.

 In no other instances was adoption of models with a scientific research

base related to achievement gains.

What do these results suggest about the theory underlying comprehensive school reform? The

CSR program awards schools relatively small increases in funds to supplement their existing

resources. The funds are intended to stimulate holistic, systemic approaches to reform. However,

few schools implemented even a majority of the components identified in NCLB or adopted

models based on scientific research. Furthermore, the list of components is based on an image

that they will work in a synergistic, coherent manner to bring about change. The data from this

evaluation provide little evidence of such synergy and coherence in actual practice.

Consequently, the findings really do not address whether the existence of comprehensive and

coherent approaches to school reform and improvement would help students perform at

dramatically higher levels, only that achieving this objective is much more difficult than the

designers of this program probably had envisioned.
References 73

REFERENCES

Aladjem, D., and Borman, K. M. (eds.). (2006). Examining Comprehensive School Reform.

Washington, D.C.: Urban Institute Press.

Aladjem, D. K., LeFloch, K. C., Zhang, Y., Kurki, A., Boyle, A., Taylor, J. E., Herrman, S.,

Uekawa, K., Thomsen, K., and Fashola, O. (2006). Models Matter—The Final Report of

the National Longitudinal Evaluation of Comprehensive School Reform. Washington,

D.C.: American Institutes for Research. Retrieved May 31, 2007, from

http://www.air.org/publications/documents/NLECSR_2006.pdf.

Borman, G. D., Hewes, G. M., Overman, L. T., and Brown, S. A. (2002). Comprehensive

School Reform and Student Achievement: A Meta-Analysis. CRESPAR Technical Report,

No. 59. Baltimore, Md.: Center for Research on the Education of Students Placed at Risk,

Johns Hopkins University.

Borman, G. D., Hewes, G. M., and Overman, L. T. (2003). “Comprehensive School Reform

and Achievement: A Meta-Analysis.” Review of Educational Research, 73(2), 125–230.

Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A. M., Madden, N. A., and

Chambers, B. (2005a). “Success for All: First Year Results from the National Randomized

Field Trial.” Educational Evaluation and Policy Analysis, 27(1), 1–22.

Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A. M., Madden, N. A., and

Chambers, B. (2005b). “The National Randomized Field Trial of Success for All: SecondYear Outcomes.” American Education Research Journal, 42(4), 673–696.

Burstein, L. (1978). “Assessing Differences Between Grouped and Individual-Level

Regression Coefficients: Alternative Approaches.” Sociological Methods and Research, 7,

5–28.

Burstein, L., and Miller, M. D. (1981). “Regression-Based Analyses of Multi-Level

Educational Data.” In R. F. Boruch, P. M. Wortman, D. S. Cordray, and Associates (eds.),

Reanalyzing Program Evaluations. San Francisco, Calif.: Jossey-Bass.

Center on Education Policy (2004). From the Capital to the Classroom: Year 2 of the No

Child Left Behind Act. Washington, D.C.

Coalition for Evidence-Based Policy (2003, December). Identifying and Implementing

Education Practices Supported by Rigorous Evidence: A User Friendly Guide. Retrieved

October 15, 2006, from

http://www.ed.gov/rschstat/research/pubs/rigorousevid/index.html.

Comprehensive School Reform Quality Center (2006a). CSRQ Center Report on Elementary

School Comprehensive School Reform Models. Washington D.C.: American Institutes for

Research.
References 74

Comprehensive School Reform Quality Center (2006b). CSRQ Center Report on Middle and

High School Comprehensive School Reform Models. Washington D.C.: American

Institutes for Research.

Crowley, J., and Hauser, A. (2007). “Evaluating Whole School Improvement Models:

Creating Meaningful and Reasonable Standards of Review.” Journal of Education for

Students Placed at Risk, 12(1), 37–58.

Flower, F. (1995). Improving Survey Questions: Design and Evaluation. Thousand Oaks,

Calif.: Sage Publications.

Fraas, J. W., Newman, I., and Pool, S. (2007). “The Use of Propensity Score Analysis to

Address Issues Associated with the Use of Adjusted Means Produced by Analysis of

Covariance.” Multiple Linear Regression Viewpoints, 33(1).

Gill, B. P., Hamilton, L. S., Lockwood, J. R., Marsh, J. A., Zimmer, R. W., Hill, D., and

Pribesh, S. (2005). Inspiration, Perspiration, and Time: Operations and Achievement in

Edison Schools. Santa Monica, Calif.: RAND Corporation. Retrieved March 10, 2007,

from http://www.rand.org/pubs/monographs/MG351/.

Hale, S. (2000). Comprehensive School Reform: Research-Based Strategies to Achieve High

Standards. San Francisco, Calif.: Comprehensive Assistance Center, Region XI, WestEd.

Haney, W. (1980). “Units and Level of Analysis in Large-Scale Evaluation.” New Directions

for Methodology and Social and Behavioral Science, 6, 1–15.

Hausman, J. A. (1978). “Specification Tests in Econometrics.” Econometrica, 46(6),

1251–1271.

Herman, R. (1999). An Educators’ Guide to Schoolwide Reform. Arlington, Va.: Educational

Research Service.

Kantor, D. (2006). “MAHAPICK: Module to Select Matching Observations Based on a

Mahalanobis Scoring.” Retrieved from http://ideas.repec.org/c/boc/bocode/s456703.html.

Kidron, Y., and Darwin, M. (2007). “A Systematic Review of Whole School Improvement

Models.” Journal of Education for Students Placed at Risk, 12(1), 9–35.

Lipsey, M. (2003). “Assessing Systematic Evidence in Crime and Methodological Concerns

and Empirical Outcomes.” The Annals of the American Academy of Political and Social

Sciences, 587, 69–81.

May, H., and Supovitz, J. A. (2006). “Capturing the Cumulative Effects of School Reform:

An 11-Year Study of the Impacts of America’s Choice on Student Achievement.”

Educational Evaluation and Policy Analysis, 28(3), 231–257.

No Child Left Behind Act of 2001, Public Law 110, 107th Congress, 1st Session (Jan. 8,

2002).
References 75

Raudenbush, S. W., and Bryk, A. S. (2002). Hierarchical Linear Models: Applications and

Data Analysis Methods, 2nd ed. Thousand Oaks, Calif.: Sage Publications.

Robinson, W. S. (1950). “Ecological Correlations and the Behavior of Individuals.” American

Sociological Review, 15, 351–357.

Rosenbaum, P. R., and Rubin, D. B., (1983). “The Central Role of the Propensity Score in

Observational Studies for Causal Effects.” Biometrika, 70, 41–55.

Rosenbaum, P. R., and Rubin, D. B. (1985). “Constructing a Control Group Using

Multivariate Matched Sampling Methods that Incorporate the Propensity Score.” The

American Statistician, 39(1), 33–38.

Rubin, D. B. (1980). “Bias Reduction Using Mahalanobis-Metric Matching.” Biometrics, 36,

293–298.

Tushnet, N., Flaherty, J., and Smith, A. (2004). Longitudinal Assessment of Comprehensive

School Reform Program Implementation and Outcomes: First-Year Report. Los Alamitos,

Calif.: WestEd.

Tourangeau, R., Rips, L. J., and Rasinki, K. (2000). The Psychology of Survey Responses.

New York, N.Y.: Cambridge University Press.

U.S. Department of Education. (2002). Elementary and Secondary Education: Part F —

Comprehensive School Reform. Washington, D.C. Retrieved July 26, 2004, from

http://www.ed.gov/policy/elsec/leg/esea02/pg13.html.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, and

Office of Elementary and Secondary Education. (2007). State ESEA Title I Participation

Information for 2003–04. Washington, D.C.

U.S. Department of Education, Institute of Education Sciences, National Center for Education

Statistics, Common Core of Data: School Years 1998–99 through 2004–05,

[http://nces.ed.gov/ccd/pubschuniv.asp], cited August 2008.

Wooldridge, J. M. (2002). Econometric Analysis of Cross Section and Panel Data.

Cambridge, Mass.: MIT Press.

Zhang, Y., Fashola, O., Shkolnik, J., and Boyle, A. (2006). “Implementation of

Comprehensive School Reform and Its Impact on Increases in Student Achievement.”

Journal of Education for Students Placed at Risk, 11(3–4), 309–329.

Zhao, X. H., Li, C. M., Gao, S. J., and Tierney, W. M. (2001). “Methods for Testing Equality

of Means of Health Care Costs in a Paired Design Study.” Statistics in Medicine, 11(20),

1703–20.
References 76
Appendix A 77

APPENDIX A.

CSRQ CENTER QUALITY REVIEW TOOL
Appendix A 78
Appendix A 79

The following form, from the American Institutes for Research, is reprinted here with permission.

Exhibit A.1

CSRQ Center Quality Review Tool
Exhibit A.1, Cont.

Appendix A 80
Exhibit A.1, Cont.

Appendix A 81
Exhibit A.1, Cont.

Appendix A 82
Exhibit A.1, Cont.

Appendix A 83
Exhibit A.1, Cont.

Appendix A 84
Exhibit A.1, Cont.

Appendix A 85
Exhibit A.1, Cont.

Appendix A 86
Exhibit A.1, Cont.

Appendix A 87
Exhibit A.1, Cont.

Appendix A 88
Exhibit A.1, Cont.

Appendix A 89
Exhibit A.1, Cont.

Appendix A 90
Exhibit A.1, Cont.

Appendix A 91
Exhibit A.1, Cont.

Appendix A 92

Note: Item replicated for the following additional subgroups: Socioeconomic status (Low); Special

needs: Learning disabled; Special Needs: other; Limited English proficient; Gender: female;

Gender: male; Achievement: high; Achievement: low; Ethnicity: African American; Ethnicity:

Hispanic; Ethnicity: Asian; Ethnicity: Native American; Grade: Grade 1; Grade: Grade 2; Grade:

Grade 3; Grade: Grade 4; Grade: Grade 5; Grade: Grade 6; Other subgroup.

Note: Item replicated for the following additional settings/setting characteristics: Geographical

region: South Central; Geographical region: Midwest; Geographical region: Mountain;

Geographical region: Pacific; Geographical region: Outside of U.S.; Urbanicity: urban;

Urbanicity: rural; School- or District-level SES: low; School- or District-level SES: high; Other

settings/setting characteristics.
Exhibit A.1, Cont.

Appendix A 93
Exhibit A.1, Cont.

Appendix A 94
Appendix A 95

Exhibit A.2

CSRQ Overall Causal Validity Mapping Rubrics
Appendix A 96
Appendix A 97

Exhibit A.3

CSRQ Evidence of Positive Effects on Student Achievement

1 This table repeats for the following sub-groups: 1) High-poverty students; 2) Minority students (African

American, Hispanic, Asian, Native American, Other Race/Ethnicity); 3) ELL students; 4) Learning disabled

students; 5) Other special needs students; 6) Urban; and 7) Rural.

2 We will only report information from the MODERATELY STRONG and VERY STRONG categories. For

efficiency in conducting QRT 3, we will therefore conduct Quality Control on these two categories only.

3 This table repeats for the following subject areas: 1) Reading; 2) Writing; 3) Mathematics; 4) Science; 5)

Social Studies; and 6) Other subjects.

4 We will only report information from the MODERATELY STRONG and VERY STRONG categories. For

efficiency in conducting QRT 3, we will therefore conduct Quality Control on these two categories only.
Appendix A 98
Appendix B 99

APPENDIX B.

STANDARD ERROR TABLES FOR ANALYSES

This appendix includes the standard error tables for the analyses presented throughout the report.

STANDARD ERRORS FOR REPORT EXHIBITS

Exhibit B.1

Standard Errors for Differences Between the 2002 Cohort of Title I CSR Schools and

Title I Non-CSR Schools

Non-CSR CSR

N Standard

Error N StandardError

Standardized Assessment Scores

Elementary Math 28,366 0.006 649 0.038

Elementary Reading 28,512 0.006 654 0.036

Middle School Math 10,135 0.010 310 0.047

Middle School Reading 10,282 0.010 318 0.051

Membership 46,667 1.631 973 14.972

Percent Minority 45,635 0.125 940 0.645

Percent Free or Reduced-price Lunch 46,632 0.166 973 1.131

Exhibit highlights: Standard errors for estimates presented in Exhibit 6 in the main body of the

report.

Source: Common Core of Data (CCD).

Exhibit B.2

Standard Errors for Differences Between the 2002 Cohort of Title I CSR Schools and

Non-CSR Title I Schools Chosen by Propensity Scoring: Schools with Elementary

Mathematics and Reading Achievement

Non-CSR CSR

N Standard

Error N StandardError

Standardized Assessment Scores

Elementary Math 550 0.038 645 0.038

Elementary Reading 550 0.037 645 0.037

Membership 550 11.396 645 10.028

Percent Minority 533 0.868 618 0.789

Percent Free or Reduced-price Lunch 550 1.514 645 1.392

Exhibit highlights: Standard errors for estimates presented in Exhibit 7 in the main body of the

report.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD).
Appendix B 100

Exhibit B.3

Standard Errors for Differences Between the 2002 Cohort of Title I CSR Schools and NonCSR Title I Schools Chosen by Propensity Scoring: Schools with Middle School

Mathematics and Reading Achievement

Non-CSR CSR

N Standard

Error N StandardError

Standardized Assessment Scores

Middle School Math 248 0.051 309 0.047

Middle School Reading 248 0.050 309 0.051

Membership 248 26.536 309 23.495

Percent Minority 245 1.339 301 1.127

Percent Free or Reduced-price Lunch 248 2.346 309 2.038

Exhibit highlights: Standard errors for estimates presented in Exhibit 8 in the main body of the

report.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD).

Exhibit B.4

Standard Errors for Changes in Standardized Assessment Scores in CSR and Non-CSR

Schools from 2002–03 to 2004–05

N CSR

Schools

Non-CSR

Schools

Elementary Mathematics 634 0.032 0.031

Elementary Reading 638 0.070 0.068

Middle School Mathematics 318 0.032 0.029

Middle School Reading 320 0.033 0.031

Exhibit highlights: Standard errors for estimates presented in Exhibit 9 in the main body of the

report.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal SchoolLevel State Assessment Score Database (NLSLSASD).
Appendix B 101

Exhibit B.5

Standard Errors for Average Number of CSR Components Implemented by

CSR and Non-CSR Elementary Schools in 2003 and 2005

School Type 2003 2005

CSR Schools (N=292) 0.137 0.155

Non-CSR Schools (N=304) 0.137 0.146

Exhibit highlights: Standard errors for estimates presented in Exhibit 21 in the main body of

the report.

Source: ECSRIO surveys.

Exhibit B.6

Standard Errors for Average Number of CSR Components Implemented by CSR and

Non-CSR Middle Schools in 2003 and 2005

School Type 2003 2005

CSR Schools (N=128) 0.187 0.217

Non-CSR Schools (N=128) 0.208 0.220

Exhibit highlights: Standard errors for estimates presented in Exhibit 21 in the main body of

the report.

Source: ECSRIO surveys.

Exhibit B.7

Standard Errors for Differences in Achievement and Demographic Characteristics

Between Schools Included and Excluded in Analyses of the Relationships

Between Implementation and Achievement, 2002 and 2005

Achievement or Demographic Elementary School Middle School

Characteristics Excluded Included Excluded Included

Mathematics Achievement, 2002 0.062 0.050 0.068 0.091

Mathematics Achievement, 2005 0.062 0.047 0.091 0.077

Reading Achievement, 2002 0.060 0.047 0.095 0.076

Reading Achievement, 2005 0.063 0.047 0.096 0.082

Percent Free or Reduced-price Lunch, 2005 0.012 0.012 0.010 0.020

Percent Black, 2005 0.015 0.017 0.012 0.028

Percent Hispanic, 2005 0.012 0.014 0.009 0.026

Exhibit highlights: Standard errors for estimates presented in Exhibit 24 in the main body of

the report.

Sources: CSR Awards Database; Common Core of Data (CCD); National Longitudinal

School-Level State Assessment Score Database (NLSLSASD).
Appendix B 102

Exhibit B.8

Standard Errors for Differences in Mathematics and Reading Achievement Between

CSR and Non-CSR Elementary Schools Included and Excluded in Analyses of the

Relationships Between Implementation and Achievement, 2005

Subject and School Type Excluded Included

Mathematics

CSR Schools 0.091 0.064

Non-CSR Schools 0.083 0.069

Reading

CSR Schools 0.092 0.065

Non-CSR Schools 0.084 0.067

Exhibit highlights: Standard errors for estimates presented in Exhibit 25 in the main body of

the report.

Sources: ECSRIO surveys; CSR Awards Database; Common Core of Data (CCD); National

Longitudinal School-Level State Assessment Score Database (NLSLSASD).

Exhibit B.9

Standard Errors for Differences in Mathematics and Reading Achievement Between

CSR and Non-CSR Middle Schools Included and Excluded in Analyses of the

Relationships Between Implementation and Achievement, 2005

Subject and School Type Excluded Included

Mathematics

CSR Schools 0.120 0.114

Non-CSR Schools 0.135 0.104

Reading

CSR Schools 0.134 0.128

Non-CSR Schools 0.140 0.099

Exhibit highlights: Standard errors for estimates presented in Exhibit 26 in the main body of

the report.

Sources: ECSRIO surveys; CSR Awards Database; Common Core of Data (CCD); National

Longitudinal School-Level State Assessment Score Database (NLSLSASD).
Appendix B 103

Exhibit B.10

Standard Errors for Relationship Between Number of Components

Implemented and Achievement Gains from 2002 to 2005

Elementary School Middle School

Effect Mathematics Reading Mathematics Reading

Constant 0.169 0.161 0.237 0.178

Achievement (2002) 0.055 0.054 0.087 0.087

Percent Free or Reduced-price

Lunch (2005) 0.195 0.191 0.368 0.302

Percent Black (2005) 0.154 0.165 0.188 0.244

Percent Hispanic (2005) 0.171 0.160 0.204 0.236

CSR Award (0=No, 1=Yes) 0.181 0.167 0.233 0.254

Number of Components 0.025 0.023 0.030 0.027

Number of Components by CSR

Status 0.038 0.034 0.054 0.053

N 349 353 129 131

Exhibit highlights: Standard errors for the estimates presented in Exhibit 27.

Sources: ECSRIO Surveys; CSR Awards Database; Common Core of Data (CCD); National

Longitudinal School-Level State Assessment Score Database (NLSLSASD).
Appendix B 104

Exhibit B.11

Standard Errors for the Relationships Between Scientific Research Base and Achievement Gains from 2002 to 2005

Elementary Math Elementary Reading Middle School Math Middle School

Reading

Model (1) (2) (1) (2) (1) (2) (1) (2)

Constant 0.124 0.131 0.125 0.131 0.153 0.154 0.118 0.114

Achievement (2002) 0.058 0.057 0.041 0.044 0.056 0.056 0.097 0.075

Percent Free or Reduced-price Lunch (2005) 0.018 0.019 0.019 0.020 0.023 0.023 0.023 0.022

Percent Black (2005) 0.019 0.017 0.016 0.015 0.010 0.009 0.017 0.017

Percent Hispanic (2005) 0.018 0.019 0.019 0.019 0.016 0.016 0.021 0.019

Limited Scientific Research Base 0.112 0.087 0.104 0.110 0.119 0.111 0.151 0.118

Moderate to Moderately Strong Research Base 0.147 0.083 0.100 0.081 0.100 0.106 0.102 0.141

Low-performing Schools 0.155 0.111 0.082 0.171

Limited Scientific Research Base in

Low-performing Schools 0.185 0.238 0.339 0.386

Moderate to Moderately Strong Research Base in

Low-performing Schools 0.257 0.230 0.149 0.228

Exhibit highlights: Standard errors for Exhibit 32 in the main body of the report.

Sources: CSRQ Center Report on Elementary CSR Models; CSRQ Center Report on Middle and High School CSR Models; CSR Awards

Database; Common Core of Data (CCD); National Longitudinal School-Level State Assessment Score Database (NLSLSASD).
Appendix C 105

APPENDIX C.

DATA COLLECTION INSTRUMENTS

PRINCIPAL INVENTORY YEAR 1.......................................................................................107

PRINCIPAL SURVEY YEAR 2 ...........................................................................................123

TEACHER INVENTORY YEAR 1 ........................................................................................133

TEACHER SURVEY YEAR 2.............................................................................................147

FIELD STUDY PROTOCOL...............................................................................................157
Appendix C 106
Appendix C 107 Principal Inventory Year 1
Appendix C 108 Principal Inventory Year 1
Appendix C 109 Principal Inventory Year 1
Appendix C 110 Principal Inventory Year 1
Appendix C 111 Principal Inventory Year 1
Appendix C 112 Principal Inventory Year 1
Appendix C 113 Principal Inventory Year 1
Appendix C 114 Principal Inventory Year 1
Appendix C 115 Principal Inventory Year 1
Appendix C 116 Principal Inventory Year 1
Appendix C 117 Principal Inventory Year 1
Appendix C 118 Principal Inventory Year 1
Appendix C 119 Principal Inventory Year 1
Appendix C 120 Principal Inventory Year 1
Appendix C 121 Principal Inventory Year 1
Appendix C 122 Principal Inventory Year 1
Appendix C 123 Principal Survey Year 2
Appendix C 124 Principal Survey Year 2
Appendix C 125 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix C 126 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix C 127 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix C 128 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix C 129 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix C 130 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix C 131 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix C 132 Principal Survey Year 2

OMB No. 1875-0222 Approval Expires 01/31/2006
Appendix C 133 Teacher Inventory Year 1
Appendix C 134 Teacher Inventory Year 1
Appendix C 135 Teacher Inventory Year 1
Appendix C 136 Teacher Inventory Year 1
Appendix C 137 Teacher Inventory Year 1
Appendix C 138 Teacher Inventory Year 1
Appendix C 139 Teacher Inventory Year 1
Appendix C 140 Teacher Inventory Year 1
Appendix C 141 Teacher Inventory Year 1
Appendix C 142 Teacher Inventory Year 1
Appendix C 143 Teacher Inventory Year 1
Appendix C 144 Teacher Inventory Year 1
Appendix C 145 Teacher Inventory Year 1
Appendix C 146 Teacher Inventory Year 1
Appendix C 147 Teacher Survey Year 2
Appendix C 148 Teacher Survey Year 2
Appendix C 149 Teacher Survey Year 2
Appendix C 150 Teacher Survey Year 2
Appendix C 151 Teacher Survey Year 2
Appendix C 152 Teacher Survey Year 2
Appendix C 153 Teacher Survey Year 2
Appendix C 154 Teacher Survey Year 2
Appendix C 155 Teacher Survey Year 2
Appendix C 156 Teacher Survey Year 2
Appendix C 157 Field Study Protocol
Appendix C 158 Field Study Protocol
Appendix C 159 Field Study Protocol
Appendix C 160 Field Study Protocol
Appendix C 161 Field Study Protocol
Appendix C 162 Field Study Protocol
Appendix C 163 Field Study Protocol
Appendix C 164 Field Study Protocol
Appendix C 165 Field Study Protocol
Appendix C 166 Field Study Protocol
Appendix C 167 Field Study Protocol
Appendix C 168 Field Study Protocol
Appendix C 169 Field Study Protocol
Appendix C 170 Field Study Protocol
Appendix C 171 Field Study Protocol
Appendix C 172 Field Study Protocol
Appendix C 173 Field Study Protocol
Appendix C 174 Field Study Protocol
Appendix C 175 Field Study Protocol

SCHOOL-LEVEL LOOSE ENDS

FOR FIELD STUDY TEAMS FOR 2004–05

Topics

Protocol

Item

to be

Referenced

1. Was the CSR grant scaled-up or was it implemented all at once at the beginning of the grant

period? Variability in start-ups, Timing for CSR.

1.2

2. Where are the funds and who controls the funds? (Survey) That data could be collected during site

visits in the principal and district interviews. “Where are the funds?” “Who makes decisions about

how the CSR and other funds are used? [this could be addressed on the one-page principal process

influence document or in principal’s interview]

10.1

3. Burkett mentioned Schoolwide Title I (in Fed Register on July 2, 2004) Field staff may need to

know about it before going to the field.

2.1

4. Scope of Change (Maggie)—How demanding is the intervention, whether comprehensive or

subject-based? Could be included in Component 1 (Research-based Method)

1.1

5. If principal change, does principal deviate from adopted method or facilitate CSR grant? (Principal

behavior, not whether a change occurred or not. Burkett)

B

6. How to define PD? (Stephanie concerned about survey) SV protocol is complete. Might add across

and among grade planning and training periods and structural PD, such as ongoing small learning

communities to protocol. Expand definition of professional development (PD): The

inventory/protocol addresses a wide variety of types of PD–school level. Site visitors should be

reminded of the variety of topics that could be considered in PD, such as the number of hours spent in

vertical and horizontal teacher planning time.

3.2.3

7. Effects of Voluntary Public School Choice (VPSC): Describe student mobility in your school.

(Probe: Has VPSC been a factor in student mobility?)

ADD AS

RIVAL

8. Burkett, “Ask teachers if they participate in decisions.” [Do in the Leadership Team meeting in the

prototype field study agenda.]

5

9. How are CSR funds being spent? D

10. Has this school been identified as low-performing? If so, has it been sanctioned because of low

performance?

A

11. Did the faculty formally vote to adopt the current reform in your school? 5.1

12. What opportunities do teachers have for making management decisions? B

13. Do teachers have a common planning period in the day? 3.5

14. Do any of the following occur in ;your school? (Probe: Looping, grouping by houses or families;

student groups stay with teachers for more than one year; interdisciplinary teaching, and paired or

teacher teaming)

School

Operations

15. If a secondary school, what kind of student groups exist to achieve particular instructional goals? School

Operations

16. Who organizes the scope and sequence of the curriculum at this school? B
Appendix C 176 Field Study Protocol

TOPICS FOR DISTRICT AND STATE OFFICIALS

FOR FIELD STUDY TEAMS, 2004–05

District CSR Contact:

1. Administration of CSR Program as a Whole: Define and update district involvement in

CSR program (targeting and selection process, number of awardees, funding process, and goals

for how whole-school improvement will be achieved).

2. Technical Assistance and Implementation Support: Describe state technical assistance

and other support provided to districts and schools that might be related to CSR.

3. Relationships between CSR and District Initiatives. Describe the relationships, if any,

between CSR and:

a) district reform initiatives (e.g., standards, curriculum requirements, use of

research-based methods, new report card forms)

b) district administration of Title I and schoolwide Title I

c) district requirements for PD

4. Relationships between CSR and NCLB and Other Federal and State Initiatives.

Describe the relationships, if any, between CSR and other initiatives, such as NCLB and public

school choice (federal) and state accountability and reform (state)—e.g., any relationships

between CSR and state standards and assessments, and the implications for CSR implementation.

5. Monitoring and Evaluation: Describe whether or how the district monitors and

evaluates CSR implementation and outcomes—if possible focusing specifically on the two site

visit schools.

State CSR Contact:

1. Administration of CSR Program as a Whole: Define and update state involvement in

CSR program (targeting and selection process, number of awardees, funding process, and goals

for how whole-school improvement will be achieved). Describe state priorities for CSR.

2. Technical Assistance and Implementation Support: Describe state technical assistance

and other support provided to districts and schools that might be related to CSR.

3. Relationships between CSR and State Initiatives. Describe the relationships, if any,

between CSR and:

a) state reform initiatives (e.g., standards, curriculum requirements, use of

research-based methods)

b) state assessments and accountability initiatives

c) state requirements for PD, teacher certification, or related policies

4. Relationships between CSR and NCLB and Other Federal Initiatives. Describe the

relationships, if any, between CSR and other federal initiatives, such as NCLB and public school

choice.

5. Monitoring and Evaluation: Describe whether o how the state monitors and evaluates

CSR implementation and outcomes—if possible focusing specifically on the two site visit

schools. Obtain a copy of the state CSR evaluation report (there may be several such reports,

each completed annually).
Appendixes References 177

APPENDIXES REFERENCES

Coalition for Evidence-Based Policy (December 2003). Identifying and Implementing Education

Practices Supported by Rigorous Evidence: A User Friendly Guide. Retrieved October 15,

2006, from http://www.ed.gov/rschstat/research/pubs/rigorousevid/index.html.

Comprehensive School Reform Quality Center (2006a). CSRQ Center Report on Elementary

School Comprehensive School Reform Models. Washington D.C.: American Institutes for

Research.

Comprehensive School Reform Quality Center (2006b). CSRQ Center Report on Middle and

High School Comprehensive School Reform Models. Washington D.C.: American

Institutes for Research.

Gill, B. P., Hamilton, L. S., Lockwood, J. R., Marsh, J. A., Zimmer, R. W., Hill, D., and Pribesh,

S. (2005). Inspiration, Perspiration, and Time: Operations and Achievement in Edison

Schools. Santa Monica, Calif.: RAND Corporation. Retrieved March 10, 2007, from

http://www.rand.org/pubs/monographs/MG351/.

Hausman, J. A. (1978). “Specification Tests in Econometrics.” Econometrica, 46(6), 1251–1271.

Wooldridge, J. M. (2002). Econometric Analysis of Cross Section and Panel Data. Cambridge,

Mass.: MIT Press.
Appendixes References 178