POLICY AND PROGRAM STUDIES SERVICE

Implementation and Early Outcomes 

of the Comprehensive School Reform 

Demonstration (CSRD) Program 

2004 

U.S. DEPARTMENT OF EDUCATION

DOC # 2004-15

OFFICE OF THE UNDER

SECRETARY
Implementation and Early Outcomes 

of the Comprehensive School Reform 

Demonstration (CSRD) Program

Prepared by: 

U.S. Department of Education

Office of the Under Secretary

Policy and Program Studies Service 

2004 
U.S. Department of Education 

Rod Paige 

Secretary 

May 2004

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While 

permission to reprint this publication is not necessary, the citation should be: U.S. Department of

Education, Office of the Under Secretary, Implementation and Early Outcomes of the Comprehensive 

School Reform Demonstration (CSRD) Program, Washington, D.C., 2004. 

To order copies of this report, write:

ED Pubs 

Education Publications Center

U. S. Department of Education 

P.O. Box 1398 

Jessup, MD 20794-1398 

Or via electronic mail, send your request to: edpubs@inet.ed.gov. 

You may also call toll-free: 1-877-433-7827 (1-877-4-ED-PUBS). If 877 service is not yet available in 

your area, call 1-800-872-5327 (1-800-USA-LEARN). Those who use a telecommunications device for the 

deaf (TDD) or a teletypewriter (TTY), should call 1-800-437-0833. 

To order online, point your Internet browser to: www.edpubs.org. 

This report is also available on the Department’s Web site at: 

www.ed.gov/about/offices/list/ous/ppss/reports.html. 

On request, this publication is available in alternative formats, such as Braille, large print, audiotape, or 

computer diskette. For more information, please contact the Department’s Alternate Format Center (202) 

205-8113. 
Contents

List of Exhibits.................................................................................................................................. iv 

Executive Summary............................................................................................................................v 

I. The Federal Role in Comprehensive School Reform.....................................................................1 

II. Implementation of the CSRD Program.........................................................................................7 

III. Achievement Trends for CSRD Schools ...................................................................................19 

Endnotes............................................................................................................................................27 

Acknowledgments.............................................................................................................................29 

Appendix A: Methodology for Achievement Analyses................................................................. A-1 

iii
Exhibits 

E-1. Eleven Components of School Reform Described in the No Child Left Behind Act.............. vi 

E-2. Demographic Characteristics of CSRD Schools..................................................................... vii 

E-3. Implementation of Comprehensive School Reform in CSRD and non-CSRD Schools with 

Schoolwide Programs ........................................................................................................ ix 

E-4. Summary of State-Level Changes in CSRD Schools from Baseline to One to Two Years 

after Receipt of CSRD Funds ............................................................................................ xi 

E-5. States with Significant Growth in Achievement for CSRD Elementary Schools................... xii 

E-6. Summary of Change in Within-State Rankings for CSRD Schools....................................... xii 

I-1. Eleven Components of School Reform Described in the No Child Left Behind Act.................2 

I-2. CSRD and CSR Appropriations, Number of Grantees, and Average Grant Size.......................3 

II-1. Demographic Characteristics of CSRD Schools Compared with All Schools 

and CSRD Schoolwides Compared with Non-CSRD Schoolwides....................................9 

II-2. Nine Components of the CSRD Program................................................................................10 

II-3. Ten Most Common Methods Implemented by CSRD Schools...............................................11 

II-4. Professional Development in CSRD and Non-CSRD Schoolwide Schools............................14 

II-5. Implementation of Selected CSRD Components at CSRD and Non-CSRD 

Schoolwide Schools...........................................................................................................16

III-1. Metrics Used by States in Reporting School-Level Test Scores............................................20 

III-2. Summary of State-Level Changes in CSRD Schools from Baseline to One to Two Years 

after Receipt of CSRD Funds ............................................................................................22 

III-3. Summary of State-Level Changes in CSRD Schools from Baseline to Two Years after 

Receipt of CSRD Funds.....................................................................................................23 

III-4. States with Significant Growth in Achievement for CSRD Elementary Schools..................24 

III-5. Summary of Change in Within-State Rankings for CSRD Schools.......................................25 

iv
Implementation and Early Outcomes of 

the Comprehensive School Reform Demonstration Program

Executive Summary 

Created in 1998 under Public Law 105-78, and authorized under the No Child Left 

Behind (NCLB) Act of 2001, the Comprehensive School Reform (CSR) program provides 

financial assistance to help schools develop and implement systematic approaches to schoolwide 

improvement that are grounded in scientifically based research and effective practices. The goal 

of the program is to enable all children to meet challenging state academic content and 

achievement standards. The annual grants of at least $50,000 per school support the initial 

implementation costs of adopting a research based reform strategy, over a three-year period. 

Since the program’s inception in 1998, federal appropriations totaling nearly $1.4 billion have 

supported grants to over 5,000 recipients. 

The federal CSR program builds on the research on effective schools and expands the 

concept of the Title I “schoolwide program,” first introduced in the 1988 Hawkins-Stafford 

amendments to the Elementary and Secondary Education Act (ESEA). Before 1988, federal 

funding for low-income and low-performing schools was to provide targeted services to students 

on an individualized basis. Schoolwide programs allow high-poverty schools to use federal 

resources in a comprehensive, integrated way to reform the entire school to meet the educational 

needs of all students in the school. The CSR program, targeted to schools serving the same highpoverty student populations, provides additional resources to help schools implement a cohesive 

reform plan. 

Under the provisions of the federal CSR program, schools may implement any reform

strategy that is based on rigorous research, within a plan that addresses 11 components of 

comprehensive school reform detailed in the No Child Left Behind Act (Exhibit E-1). While the 

original legislation included nine components of reform, the 2001 legislation added the 

importance of support for school staff and stressed the need for scientifically based research to 

improve academic outcomes of students in participating schools; NCLB also removed the term

“demonstration” from the program’s name. Throughout this report, descriptions of the 

evaluation findings, which relate to the program as created in 1998, will refer to the “CSRD” 

program, while descriptions of the current program authorized in the 2001 legislation will refer 

to the “CSR” program. 

National Evaluation of the Federal CSRD Program 

The 1998 legislation also mandated national evaluation activities, requiring the 

Department of Education “to assess results achieved by the implementation of comprehensive 

school reform in Title I schools.” The Department released its first report in 2000, describing 

early implementation of the CSRD program. This report provides updated implementation data 

and analyzes preliminary data on achievement outcomes for CSRD schools. 

v 
Exhibit E-1 

Eleven Components of Comprehensive School Reform 

Described in the No Child Left Behind Act

 Proven methods and strategies for student learning, teaching, and school management that are based on 

scientifically based research and effective practices and that have been replicated successfully in schools with 

diverse characteristics. 

 Comprehensive design for effective school functioning, integrating instruction, assessment, classroom

management, and professional development and aligning these functions into a schoolwide reform plan 

designed to enable all students to meet challenging state content and performance standards and address needs 

identified through a school needs assessment. 

 Professional development. High-quality and continuous teacher and staff professional development and 

training. 

 Measurable goals for student performance and benchmarks for meeting those goals. 

 Support from staff. Support from school faculty, administrators, and staff. 

 Support for staff. Support for school faculty, administrators, and staff. (Added in 2001)

 Parent and community involvement. Meaningful involvement of parents and the local community in

planning and implementing school improvement activities. 

 External assistance. High-quality external support and assistance from a comprehensive school reform entity 

(which may be a university) with experience in schoolwide reform and improvement. 

 Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved. 

 Coordination of resources. Identification of how other available resources (federal, state, local, or private) 

will help the school coordinate services to support and sustain the school reform.

 Scientifically based research. Scientifically based research to significantly improve the academic achievement 

of students participating in such programs as compared with students in schools who have not participated in

such programs. This requirement may also be met by strong evidence that such programs will significantly 

improve the academic achievement of participating children. (Added in 2001) 

Source: Elementary and Secondary Education Act as amended by the No Child Left Behind Act, Title I, Part F, 

Section 1606.

Guided by the components included in the authorizing legislation, the National 

Evaluation of the CSRD Program addresses the following research questions: 

● Are CSRD funds well targeted to schools with the greatest need? 

● Have schools implemented the nine components of CSRD?

● Has student achievement improved in funded schools? 

vi
Data to address these three research questions were obtained from four sources: the 

National Longitudinal Survey of Schools (NLSS), the National School-Level State Assessment 

Score Database, the Field-Focused Study of the CSRD Program, and the Southwest Educational 

Development Laboratory (SEDL) database of CSRD grantee information. Data presented in this 

report are for various years between 1998-99 and 2001-02. 

Key Findings 

CSRD funds are well targeted. 

CSRD funds are more likely to be received by schools with a high concentration of 

students who are low income, minority, and limited English proficient (Exhibit E-2). Nearly half

(46 percent) of CSRD schools were in the highest poverty category (defined as having more than 

75 percent of their students eligible for free or reduced-price lunches), compared with only 17 

percent of all schools in the United States. 

CSRD schools are also more likely to be identified as in need of improvement under 

Title I (37 percent) compared with Title I schools (17 percent) or all schools (9 percent). CSRD 

grants are concentrated in urban settings and elementary schools. 

Exhibit E-2 

Demographic Characteristics of CSRD Schools, 1999-2000 

CSRD 

Schools

All 

Schools 

Percent of students eligible for free or reduced-price lunches 

 75-100% 46%* 17% 

 50-74.9% 31%* 19% 

 35-49.9% 9%* 16% 

 0-34.9% 14%* 47% 

Percent of schools identified as in need of improvement 

under Title I 

37%* 9% 

Percent of minority students 

 75-100% 46%* 16% 

 50-74.9% 18%* 12% 

 25-49.9% 14%.. 16%

 0-24.9% 22%* 55% 

Percent of students with limited English proficiency (LEP) 

 25-100% 15%* 7% 

 10-24.9% 14%* 8% 

 0.1-9.9% 27%* 39% 

 0% 44%.. 46%

Exhibit reads: The highest-poverty group of schools, where 75 percent or more of 

the students are eligible for free or reduced-price lunches, accounted for 46 percent of 

CSRD schools and 17 percent of all schools. 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 

level. 

Source: NLSS Principal Survey, 1999-2000, and Schools and Staffing Survey, 1999-2000.

vii
CSRD schools are more likely to adopt external reform methods, and staff in CSRD schools 

showed greater support for the school’s chosen reform method. In a number of other 

areas, CSRD schools and non-CSRD schools did not differ significantly, although this was 

often because non-CSRD schools as well as CSRD schools were highly likely to report 

practices associated with comprehensive school reform. 

In 2000-01, every CSRD school reported having adopted an externally developed reform

method, compared with 71 percent of non-CSRD schoolwide schools (Exhibit E-3). CSRD 

schools showed greater support from school staff for the school’s reform method. In particular, 

teachers were more likely to report that the school’s reform method had improved teaching, 

professional growth, and students’ engagement in learning “to a great extent.” 

Teachers in CSRD and non-CSRD schoolwides were about equally likely to report 

receiving professional development (four-fifths of teachers in both groups), but teachers in 

CSRD schools reported fewer total hours of professional development than teachers in nonCSRD schoolwide schools. Findings varied by school level; elementary school teachers in 

CSRD schools were more likely to obtain professional development in reading and math than 

their counterparts in non-CSRD schools, but no significant difference was found for secondary 

school teachers. Principals in CSRD schools were more likely to report that professional 

development activities in their school were influenced by school plans, student assessment data, 

and state or local content standards. 

On most other measures—comprehensive planning, measurable goals, parent and 

community involvement, and evaluation—the NLSS surveys did not find significant differences 

between CSRD and non-CSRD schoolwide programs. Nearly all schools—both CSRD and nonCSRD schoolwides—reported having a written comprehensive or strategic plan, quantifiable 

annual goals for student achievement, and a student assessment component to their primary 

reform method. Teacher reports on parent involvement were similar in both groups of schools. 

No significant difference was found between CSRD and non-CSRD schools on combining 

federal funds with other funding sources, although CSRD principals were less likely to report 

uncertainty about what is allowed in this area. 

Case studies in 18 sites indicate that implementation of the nine CSRD components was 

mixed. 

While half of the 18 CSRD schools were fully implementing their chosen method, 

implementation was uneven or faltering in one-third of the schools, and at three of the schools, 

implementation was minimal or had stalled completely. In addition, fidelity to the original 

method, as judged by the model developer, was considered “high” in less than half of the 

schools. 

Nearly all of the case study schools were engaged in moderate to intensive professional 

development, and most of this professional development was provided through an external 

technical support team. The professional development was largely focused on implementation of 

the model program, not necessarily comprehensive reform more broadly. 

viii
Exhibit E-3 

Implementation of Comprehensive School Reform 

in CSRD and Non-CSRD Schools with Schoolwide Programs 

CSRD 

Schools 

Non-CSRD 

Schools 

Innovative strategies and methods

Adoption of externally developed reform method 100%* 71% 

Percent of teachers who participated in 

professional development in the past year

Professional development in their content area 

Average number of PD hours in content areas 

82% 

28 hours* 

79% 

34 hours

Important influences on professional development, as reported by principals 

School plans 87%* 72% 

Student assessment data 78%* 65% 

Implementation of content standards 74%* 62% 

Comprehensive plan for school reform

Overall written annual or strategic plan in SY 2000-01 99% 99% 

Measurable goals and benchmarks

Quantifiable annual goals for student achievement 96% 89% 

Adopted reform methods with student assessments, goals,

and benchmarks 

90% 93%

Support from school staff

Teachers report supporting reform method “to a great extent” 61% 55% 

Teachers report that reform method improved teaching, professional 

growth, and students engagement in learning “to a great extent” **

 40% - 50%* 33% - 41% 

Parent involvement

Teachers report using school-parent compacts 64% 64% 

Teachers discussed how teachers and parents could help 

the student achieve these goals 

98% 99%

Evaluation

Primary reform method included a student assessment component 95% 98% 

Coordination of resources

Combined federal funds with other funding sources 

“to a great extent” 

Principals uncertain about what is allowed

52% 

 31%* 

44% 

41% 

Exhibit reads: All CSRD schoolwide schools reported adopting an externally developed reform method, 

compared with 71 percent of non-CSRD schoolwides. 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 level. 

** This item combines responses from three questions: 1) impact on teaching, 2) professional growth, and

3) students’ engagement. The range of percentages reflects the answers across all three questions. 

Source: NLSS Teacher and Principal Surveys, 1998-99, 1999-2000, and 2000-01. 

ix
Less than half of the case study schools had in place reforms that were judged to be 

comprehensive. Some of the schools had comprehensive plans, but staff were unaware of or 

were detached from any comprehensive vision associated with these plans; in other schools, 

there was neither documentation nor staff understanding that demonstrated that the school had a 

comprehensive plan for improving the school. Most model programs themselves are by design 

not comprehensive because they are limited to certain grades or certain academic subjects.

All but one of the schools had measurable goals and benchmarks for measuring progress 

in student achievement. Schools typically did not measure their progress in implementing the 

nine CSRD components. 

High turnover among staff and students at some schools resulted in a transient level of 

staff and parent support for the reform effort. Few schools had defined strategies to produce 

broad and sustained parental and community involvement. 

Most of the case study schools were coordinating funds to support the reform effort, 

although some schools were coordinating only funds from federal sources because they did not 

have the access to local funds needed to redirect them. Nearly all of the schools were unclear 

about the amount or source of new funds that might be used to implement the research-based 

method after the CSRD award has ended. 

Early evidence suggests that some CSRD schools made gains on state assessments, but no 

relationship was found between CSRD funding and improved student achievement. 

However, these findings should be viewed with caution because the achievement data 

available for this report covered a very short period of implementation. 

Although this report includes an analysis of student achievement data, findings regarding 

student outcomes are preliminary due to the limited amount of data available for this report. 

Trends in state assessment results were examined over a three-year period, from 1998-99 through 

2000-01, in 38 states that had this data available; however, for about two-thirds of these states, 

only two years of data were available. (Data showing whether absolute gains occurred were 

available in only 28 states, while data showing relative gains were available for 38 states.) 

Not only is this period of time very short for making reasonable judgments about 

achievement effects, the amount of time that schools had to implement their CSRD grants may 

have been even shorter due to delays in some states awarding CSRD grants. For these reasons, 

the analyses presented here have limited value for judging the effects of the CSRD program on 

student achievement. 

x 
Overall, CSRD schools made gains on state assessments in reading and mathematics in 

about one-quarter of the states. Among elementary schools, CSRD schools made gains in 

reading in 10 out of 28 states; they made gains in mathematics in 9 out of 27 states (Exhibit E-4). 

Achievement gains were less common at the middle school and high school levels. 

Exhibit E-4 

Summary of State-Level Changes in CSRD Schools 

From Baseline to One to Two Years after Receipt of CSRD Funds 

Number 

of States Positive No Change* Negative 

 Elementary Reading 28 10 14 4 

 Elementary Math 27 9 18 0 

 Middle School Reading 23 4 18 1

 Middle School Math 23 4 18 1

High School Reading 14 1 11 2 

High School Math 14 3 11 0

 Total 129 31 90 8 

 % of Cases 24% 70% 6% 

* Includes states where change was not statistically significant. Statistical significance was 

evaluated at the .05 level of confidence using a two-sample t-test. 

Exhibit reads: State reading assessments for elementary schools showed positive growth for 

CSRD schools in 10 out of 28 states; negative changes occurred in 4 states. 

However, states with significant improvement in student achievement for CSRD schools 

also had significant growth in achievement for non-CSRD schools (Exhibit E-5). Looking at 

change in the within-state rankings of CSRD schools compared with all other schools in the 

state, more states show positive average gains than declines, particularly at the elementary level 

(Exhibit E-6). These gains are slightly more prevalent in states that have assessment trend data 

available for a longer period of time (three years instead of the usual two years), raising hope for 

further analyses when additional years of data are available. However, nearly all of the gains 

(and losses) were not statistically significant. 

This analysis of student outcomes should be considered preliminary because the 

time frame covered here is too short to expect large measurable effects of the CSRD 

program. Most schools had been receiving CSRD funding for less than one or two years, and 

the state assessment data that was available provided only one or two years of achievement 

change data. Additional time would allow for deeper implementation of reforms. Additional 

time would also provide increased data points, allowing for a more robust analysis. A later 

report on student achievement in the first cohort of CSRD grantees would offer a better 

understanding of the progress of schools participating in this program. 

xi
Exhibit E-5 

States with Significant Growth in Achievement for CSRD Elementary Schools 

Reading Mathematics

CSRD 

Schools 

Non-CSRD 

Schools 

CSRD 

Schools 

Non-CSRD 

Schools 

States with One-Year Change Data (1998-99 to 1999-2000)

Colorado

Minnesota

South Carolina

+ 

+ 

+ 

+ 

+ 

+ 

NS 

+ 

+ 

NS 

+ 

+ 

States with Two-Year Change Data (1998-99 to 2000-01)

California 

Florida 

Kentucky

Massachusetts 

New York

Texas 

Virginia 

Washington

+ 

+ 

+ 

+ 

+ 

+ 

NS 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

NS 

+ 

+ 

+ 

+ 

NS 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

NS 

+ 

+ 

+ 

+ 

* Statistical significance was evaluated at the .05 level of confidence using a two-sample t-test. 

“NS” indicates that the change in achievement was not statistically significant. 

Exhibit reads: In Colorado, CSRD schools made significant gains on state reading assessments

from 1998-99 to 1999-2000, but non-CSRD schools also had significant achievement growth. 

Exhibit E-6 

Summary of Change in Within-State Rankings for CSRD Schools 

All States States with Two Years of Change Data 

Number 

of States Positive 

No

Change Negative

Number 

of States Positive 

No

Change Negative

 Elementary Reading 38 21 1 16 9 6 0 3 

 Elementary Math 37 22 2 13 9 6 1 2 

 Middle School Reading 29 15 0 14 8 3 0 5

 Middle School Math 29 15 0 14 8 4 0 4

High School Reading 15 7 0 8 6 3 0 3 

High School Math 15 9 1 5 6 4 2 1

 Total 163 89 4 70 46 26 2 18 

 % of Cases 55% 43% 57% 39%

Exhibit reads: Across all states, subjects, and grade levels, CSRD schools made gains relative to non-CSRD

schools in 55 percent of the states, on average, and relative declines in 43 percent of the states; however, 

very few of these changes were statistically significant. 

xii
Section I: The Federal Role in Comprehensive School Reform 

Highlights 

Public Law 105-78. The Comprehensive School Reform Demonstration (CSRD) program was initially

created in 1998 under Public Law 105-78. It was authorized as the Comprehensive School Reform

(CSR) program in the Elementary and Secondary Education Act as amended by the No Child Left Behind 

Act of 2001. 

Varied federal role. The U.S. Department of Education supports comprehensive school reform through 

a state formula grant program, technical assistance, and research and development efforts to support the 

development of research-based reform methods. 

Funding. Between fiscal 1998 and fiscal 2003, nearly $1.4 billion has been appropriated for the 

program. Funds have supported over 5,000 awards. The federal program requires a minimum annual 

award of $50,000; in fiscal 2003, awards averaged $98,353 annually per school. 

Congressionally mandated report. This report fulfills the evaluation requirements of the 1998 

legislation. It is a follow-up to the report that the Department released in 2000.

Rationale for the Comprehensive School Reform Program 

Research suggests that aligning organizational structure and practices with common 

goals produces an effective organization or school.1

 Evidence from a national study of Chapter 1 

(now Title I) indicated that schools with well-integrated, coordinated approaches to teaching and 

learning had higher student achievement than schools using more targeted services.2

 Some

evidence suggests that a schoolwide approach to school reform may be more effective in 

producing gains in student achievement than separate initiatives that target individual aspects of 

a school.3

 Findings such as these were used to support the concept of comprehensive school 

reform. 

In the 1980s, researchers and practitioners began developing best-practice models for a 

school seeking to implement schoolwide improvements.4

 Throughout the 1990s, thousands of 

schools across the nation adopted various reform methods. To assist schools with the cost of 

implementing these reform strategies, Congress created the Comprehensive School Reform

Demonstration (CSRD) program in 1998 through P.L. 105-78. This legislation required all 

schools that receive CSRD grants to use the funding to adopt or develop research-based 

comprehensive school reform approaches. 

The 1998 legislation laid out nine components of comprehensive school reform to guide 

this reform process (Exhibit I-1). These components echo both the recommendations of the 

effective schools literature as well as the guidelines for Title I schoolwide programs. The 

overarching goal of the federal CSRD program was to help schools transform their operations so 

that each student will meet challenging state content and student performance standards. 

1
 

Exhibit I-1 

Eleven Components of Comprehensive School Reform 

Described in the No Child Left Behind Act

 Proven methods and strategies for student learning, teaching, and school management that are based on 

scientifically based research and effective practices and that have been replicated successfully in schools with 

diverse characteristics. 

 Comprehensive design for effective school functioning, integrating instruction, assessment, classroom

management, and professional development and aligning these functions into a schoolwide reform plan 

designed to enable all students to meet challenging state content and performance standards and address needs 

identified through a school needs assessment. 

 Professional development. High-quality and continuous teacher and staff professional development and 

training. 

 Measurable goals for student performance and benchmarks for meeting those goals. 

 Support from staff. Support from school faculty, administrators, and staff. 

 Support for staff. Support for school faculty, administrators, and staff. (Added in 2001)

 Parent and community involvement. Meaningful involvement of parents and the local community in

planning and implementing school improvement activities. 

 External assistance. High-quality external support and assistance from a comprehensive school reform entity 

(which may be a university) with experience in schoolwide reform and improvement. 

 Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved. 

 Coordination of resources. Identification of how other available resources (federal, state, local, or private) 

will help the school coordinate services to support and sustain the school reform.

 Scientifically based research. Scientifically based research to significantly improve the academic achievement 

of students participating in such programs as compared with students in schools who have not participated in

such programs. This requirement may also be met by strong evidence that such programs will significantly 

improve the academic achievement of participating children. (Added in 2001) 

Source: Elementary and Secondary Education Act as amended by the No Child Left Behind Act of 2001, Title I, 

Part F, Section 1606.

The original nine components guided program implementation from 1998 through 2001. 

The 2001 reauthorization of ESEA changed the program’s name to the Comprehensive School 

Reform program (removing the word “demonstration”) and added two additional components 

(support for school staff and use of scientifically based research). Throughout this report, 

descriptions of the evaluation findings, which relate to the program as created in 1998, will refer 

to the “CSRD” program, while descriptions of the current program authorized in the 2001 

legislation will refer to the “CSR” program. The findings and discussions in this report are 

2
framed by the original nine components; future implementation and research will be guided by 

the 11 components included in the No Child Left Behind Act of 2001. 

Between fiscal 1998 and fiscal 2003, appropriations for the program totaled nearly 

$1.4 billion (Exhibit I-2). The majority of the money is committed to Title I eligible schools, but 

a portion of the funds are available to all public schools under the Fund for the Improvement of 

Education (FIE) program. Between 1998 and 2003, three-year grants have been provided to over 

5,000 recipients to support the initial implementation costs of adopting a research-based reform

strategy. There is a minimum annual award of $50,000, but actual grants often exceed that 

amount; for fiscal 2003, states reported making grants that averaged $98,353 annually per 

school. 

Exhibit I-2 

CSRD and CSR Appropriations, Number of Grantees, and Average Grant Size 

Appropriations ($ in millions) Number of Grantees

Title I 

Fund for the 

Improvement of

Education (FIE) 

Total 

New

Awards 

Continuation

Grants 

Average

Grant Size 

FY 1998 $120 $25 $145 445 0 $66,669 

FY 1999 $120 $25 $145 1,389 470 $70,983 

FY 2000 $170 $50 $220 283 1,869 $70,339 

FY 2001 $210 $50 $260 1,184 1,525 $90,351 

FY 2002 $235 $75 $310 1,334 1,476 $101,322

FY 2003 $233 $75 $308 471* 1,422* $98,353*

Total $1,088 $300 $1,388 5,106

Sources: U.S. Department of Education, Budget Service (appropriations figures); Southwest Educational

Development Laboratory (SEDL) Database, May 2004 (number of grantees and average grant size). 

Notes: SEDL data on continuation grants do not match exactly the number of new awards made in previous years;

SEDL compiled these data based on state reports, which may contain errors. Award data for 2003 are incomplete. 

Challenges to Whole School Reform 

Implementation difficulties. While a school reform model can be a vehicle for organizing 

comprehensive school reform, studies have shown that whole school change can be difficult to 

implement. Even with the external support of a model developer, various studies have shown 

that it can take several years for the reform efforts to improve student outcomes.5

 Initial studies 

in the early 1990s indicated that full implementation of some reform methods was associated 

with improved student achievement but that few schools have achieved full implementation.6

 

In addition, the level and quality of implementation appears to vary widely—both within schools 

and between schools. Studies show that implementation may take hold in a few classrooms but 

not spread to all facets of a school.7

 A recent external evaluation of New American Schools 

(NAS) models found that schools with two or more challenges (e.g., internal tensions, leadership 

turnover, forced adoptions of designs, poor understanding of designs) consistently ranked low on 

the outside evaluator’s implementation scale.8

 

3
Misalignment of model programs and reform goals. Emerging research indicates that 

successful implementation of reform intervention does not guarantee improved student 

achievement.9

 Several factors may contribute to the lack of significant increases in test scores. 

Few of these interventions are comprehensive (e.g., they do not actually reform all aspects of a 

school). In addition, nationally available model programs may not be well-aligned with the 

reforming school’s district or state accountability system. Also, a number of these models 

support general school management and organizational improvements and do not focus on 

strengthening content and pedagogy. While a best-practice model can be a powerful tool for 

school improvement, the model is but one piece of a reform effort that should help a school 

remain accountable to district and state requirements. 

Need for external assistance. External assistance may be crucial to the reform process by 

providing schools with effective strategies and resources to improve their teaching and learning. 

Presumably, schools that are persistently low-performing would have turned around if they knew 

how. Assistance may come from a model developer, a local university, the school district, the 

state, or from federal resources. 

Department of Education Activities 

Supported with funds appropriated under both Title I and the Fund for the Improvement 

of Education (FIE), the federal CSR program is both an extension of the Title I program and a 

strategy to encourage the implementation of research-based, innovative practices in schools 

throughout the country. Accordingly, the CSR initiative includes several parts: grants to 

schools, technical assistance, and research and development. 

 

The largest portion is the grant program under which funding is allocated to states on a 

formula basis, with states making competitive grants to districts on behalf of individual schools. 

The federal program office administers the grant program and provides technical assistance to 

state coordinators to help them design and implement their competitive grant programs. 

To support reforms at the school level, the U.S. Department of Education has funded 

the National Clearinghouse on Comprehensive School Reform (NCCSR), which collects and 

disseminates information on implementation and evaluation of CSR.10 The Department has also 

contracted with regional education laboratories so that each lab can provide technical assistance 

for comprehensive school reform efforts in their region. Finally, the Department has a contract 

with the Southwest Educational Development Laboratory (SEDL) to collect data from states on 

their CSR grantees. The database is publicly available online11 and provides the sampling frame 

for the national evaluation of the CSRD program. 

The Department’s Institute of Education Sciences (formerly the Office of Educational 

Research and Improvement) funds research and development of externally designed reform

models through grants to study the effectiveness of comprehensive school reform and grants to 

model developers to expand their programs or make them more comprehensive. 

4
National Evaluation of the Federal CSRD Program 

The 1998 legislation also mandated national evaluation activities, requiring the 

Department of Education “to assess results achieved by the implementation of comprehensive 

school reform in Title I schools.” The Department released its first report in 2000, describing the 

early implementation of the CSRD program.

12 While the 2000 report to Congress was too early 

to provide data on student achievement outcomes, this evaluation report includes an assessment 

of CSRD at the state, district, and school levels, and provides preliminary information on how 

student achievement has changed in schools receiving CSRD funds. 

Guided by the components included in the authorizing legislation, the National 

Evaluation of the CSRD Program examines the following research questions: 

● Are CSRD funds well targeted to schools with the greatest need? 

● Have schools implemented the nine components of CSRD?

● Has student achievement improved in funded schools? 

Four primary data sources contribute to this final report: 1) the National Longitudinal 

Survey of Schools (NLSS); 2) the National School-Level State Assessment Score Database; 

3) the Field-Focused Study of CSRD Schools; and 4) the CSRD grantee database compiled by 

the Southwest Educational Development Laboratory (SEDL database). Data presented in this 

report are for various years between 1998-99 and 2001-02. 

National Longitudinal Survey of Schools. The NLSS survey, conducted by Westat with 

assistance from RAND, included longitudinal surveys of principals and teachers in nationally 

representative samples of Title I schools and CSRD schools. The surveys asked questions about 

the implementation of content and performance standards and student assessments; adequate 

yearly progress and schools identified as in need of improvement; school and family 

partnerships; teacher professional development; and the use of model programs. 

Data were collected during three consecutive school years: 1998-99, 1999-2000, and 

2000-01. In the first year of data collection, the sample included 311 schools from the group of 

early recipients of CSRD awards, but the sample was expanded for the second and third years of 

the study to include an additional 721 grantees. Most NLSS data presented in this report are for 

1999-2000 and 2000-01 and are based on the full sample of 1,032 CSRD schools, but some data 

are for 1998-99 and are based on the original sample of 311 schools. The principal and up to six 

teachers in each school were surveyed using computer-assisted telephone interviewing (CATI). 

Response rates were above 80 percent for both principals and teachers in all three years. 

National School-Level State Assessment Score Database. A database constructed by the 

American Institutes of Research (AIR), under a contract with Policy and Program Studies 

Service of the U.S. Department of Education, provides data on aggregate grade-level test scores 

for each school within a state for which school-level test-score data were reported. These data 

allow comparisons of the improvement in student outcomes in CSRD and non-CSRD schools 

5
within each state. This analysis includes 37 states plus the District of Columbia, which all had 

consistent school-level assessment data for at least two or three consecutive years. For nine 

states, there were three years of state assessment data, from 1998-99 to 2000-01; for 29 states, 

there were only two years of data, from 1998-99 to 1999-2000.13

Field-Focused Study of the CSRD Program. The Field-Focused Study, conducted by 

COSMOS Corporation with assistance from the McKenzie Group, included case studies of 18 

CSRD schools in nine districts using site visits; classroom observations; interviews with 

principals, teachers, and district CSRD coordinators; and analysis of school documents. The 

study drew a purposive sample to represent various geographic regions and the most commonly 

implemented reform interventions. Four visits were made to each school, two in the fall and 

spring of the 2000-01 school year and the other two in the fall and spring of 2001-02. 

SEDL database. In order to track the amount of CSRD funds awarded to individual 

schools, the U.S. Department of Education contracted with the Southwest Educational 

Development Laboratory (SEDL) to compile a database of grantees. As states award their 

grants, they are expected to provide the following details to SEDL: name of grantee school, 

contact information, award amount, method(s) implemented, the school’s poverty rate, whether 

the grantee is a Title I school, whether the school is a Title I schoolwide or a targeted assistance 

program, and whether the school has been identified as in need of improvement. 

6
Section II: Implementation of the CSRD Program 

Highlights 

CSRD resources are targeted to high-need schools. The CSRD program is meeting the legislative 

mandate to serve high-need schools in diverse settings. Nearly half (46 percent) of all CSRD schools 

have at least three-fourths of their students eligible for free or reduced-price lunches, compared with 17 

percent of all schools. CSRD schools are also more likely to have high concentrations of minority and 

limited English proficient students. Compared with non-CSRD schools, CSRD schools are four times as 

likely to have been identified as in need of improvement under Title I. 

CSRD schools were twice as likely to adopt a reform method as Title I schoolwide programs. In the 

1999-2000 school year, every CSRD school had adopted a reform method. In contrast, only 71 percent of 

non-CSRD schools with schoolwide programs had adopted an externally-developed method. 

Staff in CSRD schools showed greater support for the school’s chosen reform method, and principals in 

CSRD schools were more likely to report that professional development activities in their school were 

influenced by school plans, student assessment data, and state or local content standards. 

In a number of other areas, CSRD schools and non-CSRD schools did not differ significantly, 

although this was often because non-CSRD schools as well as CSRD schools were highly likely to 

report practices associated with comprehensive school reform.

Targeting of CSRD Grants 

 

The CSRD legislation encouraged states to award grants to schools and districts to 

support high-poverty, under-performing schools in diverse settings. Programs like CSRD and 

Title I offer additional federal support to high-need schools to help increase the educational 

opportunities for high-need students. CSRD schools are more likely to have high concentrations 

of minority students and students from low-income families. CSRD schools are also more likely 

to be identified as in need of improvement under Title I. 

School poverty level. Nearly half (46 percent) of CSRD schools were in the highest 

poverty category (defined as having more than 75 percent of their students eligible for free or 

reduced-price lunches), compared with only 17 percent of all schools in the U.S. 

Even among schools with Title I schoolwide programs, which usually have poverty rates 

above 50 percent (unless a waiver has been granted), CSRD schoolwides were much more likely 

to be in the highest poverty group (56 percent) than were non-CSRD schoolwides (35 percent). 

7
Schools in need of improvement. States tend to award CSRD grants to schools that 

have been identified under Title I as in need of improvement: 37 percent of CSRD schools have 

been identified as in need of improvement, compared with 17 percent of Title I schools. Title I 

schools that have been identified for improvement account for approximately 9 percent of all 

schools. 

Minority students. CSRD schools were much more likely to serve high proportions of 

students from racial or ethnic minorities. Almost half (46 percent) of all CSRD schools, and 

53 percent of CSRD schoolwides, served student populations that were over 75 percent minority. 

This degree of racial isolation was present in only 28 percent of Title I schoolwide schools and 

16 percent of all schools nationwide. 

 

LEP and migrant students. CSRD schools also tended to serve higher concentrations 

of limited English proficient (LEP) students and migrant students. More than one-fourth (29 

percent) of CSRD schools serve student populations that are over 10 percent LEP, compared 

with only 15 percent of all schools. Similarly, 26 percent of CSRD schools serve migrant 

students, compared with 18 percent of all schools. 

Urbanicity. CSRD schools were almost twice as likely as the average school to be 

located in an urban area (44 percent of CSRD schools, compared with 24 percent of all schools). 

CSRD schools were much less likely to be located in suburban areas and large towns (30 percent 

vs. 45 percent) and slightly less likely to be located in rural areas (26 percent vs. 31 percent). 

Grade level. Elementary schools were much more likely to receive CSRD grants than 

middle schools or high schools. In 2000-01, 71 percent of all CSRD schools were elementary 

schools, compared with 59 percent of all schools nationally. High schools accounted for 11 

percent of CSRD schools and 22 percent of all schools, while middle and junior high schools 

were equally represented among CSRD schools and all schools (15 percent). 

8
Exhibit II-1 

Demographic Characteristics of CSRD Schools Compared with All Schools 

and CSRD Schoolwides Compared with Non-CSRD Schoolwides 

CSRD 

Schools

All 

Schools

CSRD 

Schoolwide 

Schools 

Non-CSRD 

Schoolwide 

Schools

Percent of students eligible for free/reduced price lunch 

 75-100% 46%* 17% 56%* 35% 

 50-74.9% 31%* 19% 32% 41% 

 35-49.9% 9%* 16% 5%* 18% 

 0-34.9% 14%* 47% 7% 6% 

Percent of schools identified as in need of improvement 

under Title I 

37%* 9% 41%* 20%

Percent of minority students 

 75-100% 46%* 16% 53%* 28% 

 50-74.9% 18%* 12% 19% 16% 

 25-49.9% 14% 16% 13%* 23% 

 0-24.9% 22%* 55% 15%* 33% 

Percent of students with limited English proficiency (LEP) 

 25-100% 15%* 7% 17% 16% 

 10-24.9% 14%* 8% 14%* 9% 

 0.1-9.9% 27%* 39% 25% 26% 

 0% 44% 46% 43% 48% 

Number of migrant students

 50 or more migrant students 6% 3% 6% 6% 

 1-49 migrant students 20%* 15% 21% 29% 

 None 74%* 82% 74% 65% 

Urbanicity

 Urban 44%* 24% 48%* 31% 

 Suburban/Large Town 30%* 45% 28% 28% 

 Rural/Small Town 26%* 31% 25%* 41% 

School grade level

 Elementary schools 71%* 59% 79% 84% 

 Middle/junior high schools 15% 15% 13% 9% 

 High schools 11%* 22% 5% 5% 

 Other 3% 6% 3% 3% 

Exhibit reads: The highest-poverty group of schools, in which 75 percent or more of the students are eligible 

for free or reduced-price lunches, accounted for 46 percent of all CSRD schools, 17 percent of all schools, 

56 percent of CSRD schools with schoolwide programs, and 35 percent of non-CSRD Title I schoolwides. 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 level. 

Sources: NLSS Principal Survey, 1999-2000, and Schools and Staffing Survey, 1999-2000. Sample sizes for 

responding principals in the NLSS were 837 CSRD schools, 652 CSRD schoolwides, and 588 non-CSRD

schoolwides. The sample size for the estimates for all schools from the SASS were 9,302 schools. 

Note: The percent of all schools identified as in need of improvement under Title I was calculated by dividing the 

total number of identified schools reported in State Consolidated Performance Reports (8,505) by the total number 

of schools reported by the Common Core of Data (92,012) in 1999-2000. 

9
Implementing the Components of the Federal Program 

During the time period covered by the studies included in this report, grantees were 

expected to implement the nine components of the 1998 CSRD legislation (Exhibit II-2). 

Exhibit II-2 

Nine Components of the CSRD Program

 Innovative strategies and proven methods that are based on reliable research and effective practices and that 

have been replicated in schools with diverse characteristics. 

 Comprehensive design for effective school functioning, integrating all facets of teaching and learning into a 

schoolwide reform plan to enable all students to meet challenging state content and performance. 

 High-quality and continuous teacher and staff professional development and training. 

 Measurable goals for student performance and benchmarks for meeting those goals. 

 Support from school faculty, administrators, and staff. 

 Involvement of parents and the local community in school improvement activities. 

 External support and assistance from an organization with experience in schoolwide reform and improvement. 

 Plan to evaluate the implementation of school reforms and the student results achieved. 

 Coordination of resources and services (federal, state, local, or private) to support and sustain the school

reform. 

Source: Public Law 105-78, the Fiscal Year 1998 Appropriations Act for the U.S. Department of Education.

The main focus of the CSRD program is on implementing innovative strategies and 

proven methods as part of a plan for whole-school change. These strategies are generally 

introduced through professional development, often delivered through external technical 

assistance. Examining the implementation of these three components provides the best sense of 

how the CSRD program has been implemented. For that reason, this section will focus on these 

components. There is less information on other components, but they also provide insight into 

the extent and the quality of program implementation. 

Innovative strategies and proven methods. The CSRD program was intended to 

promote innovative strategies and proven methods for student learning and teaching, which often 

has often been interpreted as implementing an externally developed reform method or model. 

The reform can be externally developed and available nationally or regionally, or the school can 

adapt its own program drawing from the research literature. According to the NLSS data, by 

2000-01, every CSRD school had adopted an externally developed reform method, compared 

with 71 percent of non-CSRD schoolwide schools. Findings from the Field-Focused Study 

indicate that the implementation process is similar regardless of whether the reform method is 

national or locally developed: in both cases, schools receive external technical assistance and 

make similar progress toward implementing the program components.

10
As of February 2004, CSRD and CSR schools had implemented 875 different reform

methods. More than one-third (35 percent) of grantees had chosen a method listed in the original 

CSRD legislation.*

 More than half (51 percent) had chosen one listed in the Northwest Regional 

Educational Lab catalog (NWREL).14 The most commonly implemented method was Success 

for All, selected by 9 percent of grantees. The 10 most common methods account for 38 percent 

of all grantees. 

Exhibit II-3 

Ten Most Common Methods Implemented by CSR Schools 

Number of

CSR Schools 

Percent of 

CSR Schools 

Success for All 446 9% 

Lightspan 243 5%

Accelerated Schools 230 5% 

America's Choice 184 4% 

Co-NECT 165 3%

Coalition of Essential Schools 162 3%

High Schools that Work 138 3% 

Direct Instruction 137 3% 

Effective Schools 135 3% 

School Development Program 112 2% 

 Total 1,952 38%

Source: SEDL Database, February 2004. 

Note: Schools in this table include both those that received grants under the

original CSRD program as well as the subsequent CSR program.

However, case study findings from the Field Focused Study indicate that implementation 

of the chosen research-based method was uneven across the 18 sites. Half of the CSRD schools 

were fully implementing their chosen method; elements of the method were observed in all or 

nearly all classrooms in these nine schools, and faculty appeared to be comfortable with using 

the method. Another one-third of the schools were implementing some components of the 

method, but this implementation was uneven or faltering. In some of these schools, use of the 

method was inconsistent across teachers or implemented at only one grade level. Other schools 

were implementing portions of the method schoolwide but had not yet implemented all 

components of the method. Finally, at three of the schools, implementation was minimal or had 

stalled completely. 

 

*

 The models listed in the 1998 legislation included: Accelerated Schools, ATLAS Communities, Audrey Cohen

College, Coalition of Essential Schools, Community for Learning, Comer School Development Program, Co-NECT, 

Direct Instruction, Expeditionary Learning Outward Bound, High Schools that Work, Modern Red Schoolhouse, 

Paideia, Roots and Wings, Success for All, Talent Development High School, and Urban Learning Centers. 

11
In addition, fidelity to the original method was also uneven, as judged by the model 

developers. In the 14 schools for which the study was able to obtain judgments about fidelity of 

implementation from the model developer, the developer judged fidelity to be high in six of the 

schools, moderate in six schools, and low in two of the schools. 

How Schools Chose Reform Methods. School decisions to adopt a particular reform

method are often not completely voluntary decisions. According to the NLSS, in 2000-01, 

almost two-thirds (63 percent) of CSRD schoolwide schools had adopted their primary reform

method either as a result of being identified as in need of improvement or at the direction of the 

district or the community, compared with 53 percent of non-CSRD schoolwide schools. Being 

identified for improvement was the reason for adopting a reform method in 44 percent of CSRD 

schoolwides but only 16 percent of non-CSRD schoolwides. 

Virtually all CSRD schools in the NLSS sample reported that they participated in a 

careful selection process, the first step of which generally consisted of a thorough needs 

assessment (this was also true for non-CSRD schoolwides). In making decisions to adopt a 

method or methods, 96 percent of CSRD principals said that research evidence was an important 

factor in their selection; 95 percent of principals of non-CSRD schoolwides also gave this 

response. The overwhelming majority of CSRD schoolwide principals reported that they had 

sufficient access to information on the various methods, sufficient information about the needs of 

their own schools, and the time needed to explore various options. (This was also true for nonCSRD schoolwides.) 

Despite these findings, only 40 percent of CSR grantees have used a method rated as 

having “strong” or “promising” evidence of effectiveness by the Educators’ Guide to Schoolwide 

Reform; 26 percent have used a method rated as “strong.” 15 Although principals appear eager to 

make informed choices about reform programs, the fact remains that strong evidence on the 

effectiveness of many models is often not available. 

Schools may adopt more than one externally developed reform method, and the NLSS 

data show that 21 percent of CSRD grantees had adopted two methods. In contrast, 43 percent 

of non-CSRD schoolwides had adopted one reform method and 31 percent had adopted two. The 

Field-Focused Study also found that some reform methods are being deliberately coupled with 

other methods. Since research has shown few of the nationally available methods are 

comprehensive, adopting more than one method may be a strategy to reach all aspects of a 

school. 

Implementation Strategies. Analysis of implementation data from the Field-Focused 

Study suggests three pathways that lead to reform. First, a school can explicitly implement each 

of the nine components. Second, a school can adopt a research-based method that requires 

comprehensive changes in school operations. A comprehensive method can help a school 

implement the other eight CSRD components, without necessarily addressing each component 

individually. Third, a school can be part of a school system that fosters whole school change 

because state, district, and school policies and practices are aligned toward school reform.

12
While each path can lead to sustainable reforms, the implementation process looks 

different in each type of school. Some schools write a school reform plan that explicitly 

incorporates the program components. This written plan provides a blueprint for school reform. 

By implementing the program components, within a plan for change, schools meet the program

requirements. Other schools implement a comprehensive, externally designed reform method. 

The method, coupled with external technical assistance, can lead CSRD grantees to whole-school 

change. 

In other cases, a strong, well-aligned accountability system can provide continuing 

support for reforms. Because the effort is led by the district or state, it incorporates local and 

state expectations. Unfortunately, if the district or state modifies policy and adopts a different 

reform method than the one included in the CSRD plan, schools can be forced to radically 

change strategies and may lose momentum. In other instances, schools are implementing 

comprehensive school reform because they have been identified by the district or state as 

underperforming or in need of improvement. This identification may result in the school 

receiving additional assistance and financial support from the state or district. If these resources 

are aligned with the CSRD-funded program, the combined effort could lead to more sustainable 

implementation of the reforms. 

Barriers to Implementation. The majority of principals believe that they are able to 

implement their reform methods without much difficulty. Few principals (3 percent of CSRD 

schoolwides and 2 percent of non-CSRD schoolwides) reported that they were finding it difficult 

“to a great extent” to implement the reform. Slightly less than one-quarter of principals in CSRD 

schoolwide schools and non-CSRD schoolwide schools with reform methods reported that they 

were finding it moderately difficult to implement the method. This relatively sanguine view 

contrasts with the findings of the Field-Focused Study that only half of the 18 CSRD schools 

visited had fully implemented their chosen method and that the fidelity of implementation, as 

judged by the model developers, was considered “high” in less than half of these schools. 

Among those principals experiencing implementation difficulties, the two greatest 

problems reported were insufficient planning time and staff turnover. Inadequate funding and 

resources were reported to be more of a barrier for non-CSRD schoolwide schools with reform

methods (23 percent) than for CSRD schoolwide schools (14 percent), suggesting that CSRD 

grants may help reduce financial barriers to implementing reform. 

Professional development. Professional development is an important component of the 

CSRD program, providing assistance to enable educators to improve teaching and learning for 

students in their schools. Teachers in CSRD and non-CSRD schoolwides were about equally 

likely to report receiving professional development (four-fifths of teachers in both groups), but 

teachers in CSRD schools reported fewer total hours of professional development than teachers 

in non-CSRD schoolwide schools. Findings varied by school level; elementary school teachers 

in CSRD schools were more likely to obtain professional development in reading and math than 

their counterparts in non-CSRD schools, but no significant difference was found for secondary 

school teachers. Principals in CSRD schools were more likely to report that professional 

development activities in their school were influenced by school plans, student assessment data, 

and state or local content standards. 

13
Exhibit II-4 

Professional Development in CSRD and Non-CSRD Schoolwide Schools 

CSRD 

Schools 

Non-CSRD 

Schools 

Percent of teachers who participated in professional development in the past year

Professional development in their content area

Average number of PD hours in content area

 82% 

28 hours* 

79% 

34 hours

Professional development in reading

Elementary teachers 

Secondary teachers 

 88%*

80% 

82% 

80% 

Professional development in math

Elementary teachers 

Secondary teachers 

 68%*

85% 

61% 

87% 

Other types of professional development

Common planning time 

Formal mentoring relationships 

Networking with outside teachers 

97% 

 57%*

66% 

98% 

53% 

67% 

Important influences on professional development, as reported by principals 

School plans 87%* 72% 

Student assessment data 78%* 65% 

Implementation of content standards 74%* 62% 

Exhibit reads: Although teachers in both CSRD and non-CSRD schoolwides were equally likely to report 

receiving professional development in their content area, teachers in CSRD schoolwides reported receiving 

fewer hours (28) of such professional development than teachers in non-CSRD schoolwides (34). 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 level. 

Source: NLSS Teacher and Principal Surveys, SY 1999-2000 and 2000-01. The sample size for responding 

principals ranged from 651 to 676 in CSRD schoolwides and from 588 to 611 in non-CSRD schoolwides, 

depending on the year of the survey. For teachers, the sample size ranged from 3,571 to 3,577 in CSRD

schoolwides and from 3,183 to 3,301 in non-CSRD schoolwides

The Field-Focused Study found that nearly all of the case study schools were engaged in 

moderate to intensive professional development, and most of this professional development was 

provided through an external technical support team. The professional development was largely 

focused on implementation of a model program, not necessarily comprehensive reform more 

broadly. 

External technical assistance. According to the NLSS data, 97 percent of CSRD schoolwide 

schools that were beyond the initial selection and planning phase had received assistance in

implementing the method during the past year. (Comparable information for non-CSRD schools is 

not available.) The Field-Focused Study found no differences in the amount or type of technical 

assistance received by schools using nationally or locally developed methods, because the locally 

developed methods usually had a regional or university-based developer that assisted the school.

Much of the technical assistance was provided by the developer of the school’s reform model. 

Fifty-six percent of principals in CSRD schoolwide schools reported that the model developer was the 

primary provider of professional development or technical assistance in implementing the model. 

14
This technical assistance focused on implementing the external model and not necessarily on reforms 

aligned to the school’s accountability requirements. 

CSRD schools identified as in need of improvement were more likely than other CSRD 

schools to receive additional technical assistance from a variety of sources. These schools receive 

more assistance from school districts, state departments of education, school support teams, university 

staffs, and independent consultants. 

Comprehensive plan for school reform. NLSS data indicate that almost all schoolwide 

schools (both CSRD and non-CSRD) had an overall written annual or strategic plan in 2000-01. 

However, the Field-Focused Study found that less than half of the case study schools (eight out 

of 18) had in place reforms that were truly comprehensive. Six of the 18 schools had 

comprehensive plans or documentation, such as school improvement plans, but staff were 

unaware of or were detached from any comprehensive vision associated with these plans. 

Among most of the remaining schools, there was neither documentation nor staff understanding 

that demonstrated that the school had a comprehensive plan for improving the school. 

Technical assistance typically did not address the comprehensiveness of a school’s 

reform plan. The Field-Focused Study found that external technical assistance largely focused 

on a specific reform method. However, this assistance did not address the comprehensive, 

whole-school design of the reform unless the method itself was comprehensive. Many model 

programs are by design not comprehensive because they are limited to certain grades or certain 

academic subjects.

Measurable goals and benchmarks. The NLSS study examined several indicators of 

whether schools had measurable goals and benchmarks and found no differences between CSRD 

and comparable non-CSRD schools. Nearly all CSRD schoolwide schools reported having 

quantifiable goals for how student achievement should improve every year (96 percent in 

2000-01) and had adopted reform methods that included a student assessment component as 

well as goals and benchmarks for students (90 percent in 1999-2000). The Field-Focused Study 

noted that goals and benchmarks were usually focused on student achievement and typically did 

not measure progress in implementing the nine CSRD components.

Support from school staff. Teachers in CSRD schoolwide schools were more positive 

about the effects of the method on their teaching and their students, with between 40 to 50 

percent reporting that it had improved their teaching, professional growth, and students’ 

engagement in learning “to a great extent,” compared with 33 to 41 percent of teachers in nonCSRD schoolwide schools. However, there was no significant difference between CSRD and 

comparable schools in the percent of teachers reporting that they supported the reform method 

“to a great extent.” The Field-Focused Study found that high turnover among staff at some

schools can result in a transient level of support for the reform effort. In particular, turnover 

among principals can sway support or involvement when new principals have agendas and 

priorities that differ from their predecessors’ agendas, since strong principal leadership can be a 

critical ingredient in the success of a reform effort. 

15
Exhibit II-5 

Implementation of Selected CSRD Program Components 

at CSRD and Non-CSRD Schoolwide Schools 

CSRD 

Schools 

Non-CSRD 

Schools 

Innovative strategies and methods

Adoption of externally developed reform method 100%* 71% 

Comprehensive plan for school reform

Overall written annual or strategic plan in SY 2000-01 99% 99% 

Measurable goals and benchmarks

Quantifiable annual goals for student achievement 96% 89% 

Adopted reform methods with student assessments, goals,

and benchmarks 

90% 93%

Written plans to improve student achievement (reading) 96% 87% 

Written plans to improve student achievement (math) 85% 84% 

Support from school staff 

Teachers report supporting reform method to “great extent” 61% 55% 

Teachers report that reform method improved their teaching “to a 

great extent” 

 40%* 33% 

Teachers report that reform method improved their professional 

growth “to a great extent” 

 44%* 36% 

Teachers report that reform method improved students’ engagement 

in learning “to a great extent” 

 50%* 41% 

Parent involvement 

Principals report use of school-parent compacts 90% 84% 

Teachers report use of school-parent compacts 64% 64% 

Teachers discussed performance goals with parents 82% 81% 

Teachers discussed how teachers and parents could help 

the student achieve these goals 

98% 99%

Evaluation 

Primary reform method included a student assessment component 95% 98% 

Coordination of Resources 

Combined their federal funds with other funding sources 

“to a great extent” 

52% 44%

Principals uncertain about what is allowed 31%* 41% 

Exhibit reads: Nearly all (96 percent) CSRD schoolwide schools reported having quantifiable annual 

goals for student achievement. 

* Difference between CSRD and non-CSRD schoolwides is statistically significant at the .05 level. 

Source: NLSS Teacher and Principal Surveys, 1998-99, 1999-2000, and 2000-01. The sample size for 

responding principals ranged from 245 to 676 in CSRD schoolwides and from 588 to 611 in non-CSRD

schoolwides, depending on the year of the survey. For teachers, the sample size ranged from 1,069 to 3,577 

in CSRD schoolwides and from 2,822 to 3,301 in non-CSRD schoolwides

16
Parent and community involvement. No significant differences were found between 

CSRD and comparable schools in principal and teacher reports on parent involvement. Almost 

all CSRD schools reported that parents played a role in developing the strategic plan. Most 

principals reported using school-parent compacts that spell out the responsibilities of the school 

and the parents (90 percent of CSRD schoolwides). Although teachers in these schools were less 

likely than their principals to report the use of compacts (64 percent), they usually reported 

talking with parents about student performance goals (82 percent) and about how the teacher and 

parents could help the student achieve these goals (98 percent). 

The Field-Focused Study found that the extent of parent involvement can be transient, 

particularly where there is high student turnover, and high involvement during the kickoff year 

of a reform effort may not have been sustained the following year. The study found that few 

schools had defined strategies to produce broad and sustained parental and community 

involvement; in most cases the observed support and involvement was based on the use of CSRD 

funds to hire a parent coordinator or isolated milestones such as the convening of a parents’ 

night. 

Evaluation. The Field-Focused Study found that schools do not necessarily have explicit 

evaluation plans. However, evidence from the NLSS indicates that CSRD grantees may be 

relying on their externally developed reform method’s evaluation process. In schools adopting 

reform methods—both CSRD and non-CSRD—well over 90 percent reported that their primary 

reform method included a student assessment component. 

Coordination of Resources. Because the CSRD grant is provided for only three years, 

it is important for grantees to coordinate and leverage other funding sources in order to continue 

the reforms beyond the duration of the federal grant. In SY 1998-99, half (52 percent) of the 

principals of CSRD schoolwide schools reported combining their federal funds with other 

funding sources “to a great extent,” compared with 44 percent of principals of non-CSRD 

schoolwides. Among CSRD grantees, elementary schools were significantly more likely to 

integrate funds than were secondary schools. Both CSRD and non-CSRD schoolwide principals 

reported that district and state control over the use of funds was a challenge for about half the 

CSRD schoolwide principals in the NLSS sample. However, CSRD schoolwide principals were 

less likely to report uncertainty about what is allowed as a barrier to the coordination of 

resources (31 percent) than principals in non-CSRD schoolwide schools (41 percent). 

The Field-Focused Study found that while most of the case study schools were 

coordinating funds to support the reform effort, some schools were coordinating only funds from

federal sources because they did not have the access to local funds needed to redirect them. The 

study also found that nearly all of the schools were unclear about the amount or source of new 

funds that might be used to implement the research-based method after the CSRD award has 

ended.

17
18
Section III: Achievement Trends for CSRD Schools

Highlights 

This analysis of student outcomes should be considered preliminary because the time frame covered

here is too short to expect large measurable effects of the CSRD program 

Most schools included in this analysis had been receiving CSRD funding for one or two years, and the 

state assessment data that was available provided only one or two years of achievement change data. 

This early evidence found that some CSRD schools made gains on state assessments, but no

relationship was found between CSRD funding and student achievement. 

CSRD schools made gains in reading and mathematics in about one quarter of the states; however, 

non-CSRD schools also improved on state assessments. 

Looking at changes in the achievement ranking of CSRD schools compared with other schools in the 

state, states were more likely to show positive average gains than declines for CSRD schools, particularly

at the elementary level. These gains were more prevalent in states that had assessment trend data 

available for three years instead of the usual two years. However, few of these relative achievement gains 

were statistically significant.

The intent of the CSRD program is to raise student achievement by promoting 

schoolwide reform and improvement strategies. If the program is effective, we would expect to 

see achievement growth trends in CSRD schools that outpace achievement growth in other 

schools with similar demographics. This section examines trends in student achievement on 

state assessments. However, it is important to note that insufficient data are currently available 

for a meaningful examination of achievement trends in CSRD schools. According to the 

literature on school change, schools may require more than five years to accomplish significant 

schoolwide reforms, but this report is only able to examine one- or two-year trends from the 

baseline year to the first or second year after receipt of CSRD funds. 

Therefore, this report provides only a preliminary examination of achievement trends in 

CSRD schools, and further analysis is needed as additional years of state assessment data 

become available in order to understand the impact of the CSRD program on student 

achievement. 

This achievement analysis focuses on two evaluation questions: 

 Have CSRD schools made statistically significant gains from the baseline year 

(1998–99) to the first or second year after receipt of CSRD funds? 

19
 Have CSRD schools made relative gains compared with non-CSRD schools within 

the state from the baseline year (1998-99) to the first or second year after receipt of

CSRD funds?

Methodology 

This analysis uses data from the National School-Level State Assessment Score 

Database,16 which provides data on aggregate school-level test scores for each grade tested on 

state assessments. The database also includes data on Title I status and school poverty levels (as 

measured by the percent of students eligible for the free and reduced-price lunch program). To 

determine which schools in this dataset received CSRD funds, it was merged with data compiled 

by the Southwest Educational Development Laboratory (SEDL).17 

State assessments provide information on student progress toward meeting state academic 

content and academic achievement standards. However, these data cannot be aggregated across 

states to examine national trends or used to make comparisons among states. Because each state 

has developed its own standards and assessments, there is little comparability across states in the 

content and rigor of these standards and assessments. Therefore, this analysis is based on 

separate state-by-state comparisons of CSRD and non-CSRD schools within each state. 

This analysis includes 37 states plus the District of Columbia. Five states were excluded 

because they did not have consistent school-level assessment data available (Iowa, Montana, 

Nebraska, New Mexico, and North Dakota). Seven additional states were excluded because they 

had only one year of state assessment data available (Alaska, Hawaii, Kansas, Louisiana, 

Michigan, Missouri, and Nevada). Mississippi was also excluded because their state assessment 

changed during the period covered by this analysis. 

Exhibit III-1 

Metrics Used by States in Reporting School-Level Test Scores

Test Score Metric # of States States 

Scale score or normal curve 

equivalent (NCE) score 

15 CA, DE, FL, IL, KY, MA, ME, MN, MS, NC, NH, NJ, NY, 

PA, SC 

Percent passing level 21 AK, CO, CT, DC, HI, IN, KS, LA, MD, MI, MO, OH, OK, 

OR, RI, TX, VA, VT, WA, WI, WY 

Percentile 11 AL, AR, AZ, GA, ID, MS, NV, SD, TN, UT, WV 

No uniform, school-level

test scores 

5 IA, MT, ND, NE, NM 

20
The available assessment data at the time of this analysis span a different period of time

in each state. For nine states, there were three years of state assessment data, from 1998-99 to 

2000-01 (these states are California, Florida, Kentucky, Massachusetts, New York, Texas, Utah, 

Virginia, and Washington). For 29 states, there were only two years of data, from 1998-99 to 

1999-2000. 

Not only is this period of time very short for making reasonable judgments about 

achievement effects, but the amount of time that schools had to implement their CSRD grants 

may have been even shorter due to delays in some states awarding CSRD grants. For these 

reasons, the analyses presented here have limited value for judging the effects of the CSRD 

program on student achievement. Future studies may be able to use achievement data covering a 

longer period of time, allowing for more time to fully implement CSRD reforms. In the 

meantime, the following analyses provide some preliminary information about how achievement 

trends in CSRD schools compare with those in non-CSRD schools. 

CSRD schools were compared with all non-CSRD schools in the state. Because the 

program emphasizes selecting schools identified in need of improvement, CSRD schools are 

likely to be both higher-poverty and lower-achieving, on average, than other schools in the state, 

and it might be preferable to compare CSRD schools with a comparison group of non-CSRD 

schools with similar characteristics in terms of poverty level and prior achievement. Early 

analyses conducted for this report included selection of a comparison group of non-CSRD 

schools, but there were concerns about whether the methods used to select these comparison 

schools actually resulted in a truly comparable comparison group. Because this analysis 

examines whether CSRD schools moved up or down in their state ranking, it was not necessary 

to restrict the non-CSRD samples to a comparable group of schools. Indeed, the results 

comparing CSRD schools with non-CSRD schools were slightly more positive than the 

comparison with the selected comparison group, and the analysis using all non-CSRD schools is 

the one presented in this report. 

The findings that follow are based on the average change in test scores for CSRD schools 

from the baseline year to one or two years after receipt of CSRD funds, to see whether these 

schools improved over time. For states that report percentile ranks, the analyses are more limited 

because percentile ranks cannot be averaged across schools as with test scores; thus, these states 

are not included in the first analysis, which relies on averaging test scores across schools. More 

detailed information on the methods used in these analyses is provided in Appendix A. 

It should be noted that comparisons of grade-level scores over successive years (e.g., 

comparing the fourth-grade test scores in 1999 with the fourth-grade test scores in 2001) contrast 

the achievement of different cohorts of students. These scores are likely to vary from year to 

year because different students are taking the test. In addition, average test scores in small 

schools are likely to fluctuate widely from year to year because of the small number of students 

taking the test. A more rigorous research design for examining the impact of the CSRD program 

on student achievement would follow students longitudinally to examine whether they make 

greater gains than a comparison group of students in non-CSRD schools. Nevertheless, schoollevel achievement trends on state assessments provide one indicator of program impact that 

should continue to be monitored over a longer period of time. 

21
Findings

CSRD schools made gains in reading and mathematics in about one-quarter of the states; 

most gains were not statistically significant. 

Exhibit III-2 summarizes whether average test scores for CSRD schools increased or 

decreased from the baseline year of 1998–99 to one or two years after receipt of CSRD funds. 

This analysis includes 28 states that reported test scores as scale scores, mean normal curve 

equivalent (NCE) scores, or percentage passing. States that reported test results only in terms of 

percentiles are not included in this analysis. 

Overall, CSRD schools made statistically significant gains in reading and mathematics in 

about one-quarter of the states. In most states (70 percent, on average), there was no significant 

change in student achievement in CSRD schools, while significant declines occurred in 6 percent 

of the cases. 

Significant gains were most common at the elementary level (about one-third of the 

states). In reading, CSRD elementary schools made gains in reading in 10 out of 28 states, were 

unchanged in 14 states, and declined in four states. In mathematics, CSRD elementary schools 

made gains in nine out of 27 states and did not change in 18 states; no state showed a decline in 

elementary math achievement for CSRD schools. Middle schools and high schools were less 

likely to show positive gains. At both levels, average change in student achievement for CSRD 

schools was not statistically significant for about three-quarters of the states. Among middle 

schools, CSRD showed significant gains in four out of 23 states. High school results were 

available for only 14 states. 

Exhibit III-2 

Summary of State-Level Changes in CSRD Schools 

From Baseline to One to Two Years after Receipt of CSRD Funds 

Number 

of States Positive No Change* Negative 

 Elementary Reading 28 10 14 4 

 Elementary Math 27 9 18 0 

 Middle School Reading 23 4 18 1

 Middle School Math 23 4 18 1

High School Reading 14 1 11 2 

High School Math 14 3 11 0

 Total 129 31 90 8 

 % of Cases 24% 70% 6% 

* Includes states where change was not statistically significant. Statistical significance was 

evaluated at the .05 level of confidence using a two-sample t-test. 

Exhibit reads: State reading assessments for elementary schools showed positive growth 

for CSRD schools in 10 out of 28 states; negative changes occurred in four states, and 

there was no significant change in 14 states. 

22
The above results are dominated by states with only one year of change data, and 

comprehensive school reforms may take a longer period of time to be implemented and begin to 

have an impact on student achievement. According to the literature on school change, schools 

may require more than five years to accomplish meaningful schoolwide reforms. 

Exhibit III-3 shows the same analysis looking only at those states with two years of 

change data (that is, a baseline year plus two years after the receipt of CSRD funds). The twoyear change data show much more positive results for CSRD schools. At the elementary level, 

CSRD schools made gains in reading and mathematics in seven out of eight states. Middle 

school and high school results were less striking but were still more likely to be positive than the 

results including states with only one year of change data. Overall, CSRD schools showed 

achievement gains in 48 percent of the cases. 

Exhibit III-3 

Summary of State-Level Changes in CSRD Schools 

From Baseline to Two Years after Receipt of CSRD Funds 

All States 

Number 

of States Positive No Change* Negative 

 Elementary Reading 8 7 1 0 

 Elementary Math 8 7 1 0 

 Middle School Reading 8 2 5 1

 Middle School Math 8 3 5 0

High School Reading 8 1 6 1 

High School Math 8 3 5 0 

 Total 48 23 23 2 

 % of Cases 48% 48% 4% 

* Includes states where change was not statistically significant. Statistical significance was 

evaluated at the .05 level of confidence using a two-sample t-test. 

Exhibit reads: Looking only at states with three years of assessment data, state reading 

assessments for elementary schools showed positive growth for CSRD schools in seven 

out of eight states. 

23
Achievement improved for both CSRD and non-CSRD schools. 

However, it may be the case that the CSRD program itself was not the cause of the 

significant growth in student achievement for CSRD schools in these states. Exhibit III-4 shows 

that those states with significant growth in achievement for CSRD schools at the elementary 

level also had significant growth in achievement for non-CSRD schools as well (both the 

comparison schools and all non-CSRD schools). Data for middle schools and high schools show 

similar patterns. 

Exhibit III-4 

States with Significant Growth in Achievement for CSRD Elementary Schools 

Reading Mathematics

CSRD 

Schools 

Non-CSRD 

Schools 

CSRD 

Schools 

Non-CSRD 

Schools 

States with Two Years of Assessment Data (1998-99 to 1999-2000)

 Colorado + + NS NS

 Minnesota + + + + 

 South Carolina + + + + 

States with Three Years of Assessment Data (1998-99 to 2000-01)

 California + + + + 

 Florida + + + + 

 Kentucky + + + + 

 Massachusetts + + NS NS 

 New York + + + + 

 Texas + + + + 

 Virginia NS NS + + 

 Washington + + + + 

* Statistical significance was evaluated at the .05 level of confidence using a two-sample 

t-test. “NS” indicates that the change in achievement was not statistically significant. 

Exhibit reads: In Colorado, CSRD schools made significant gains on state reading 

assessments from 1998-99 to 1999-2000, but comparison schools and all non-CSRD

schools also had significant achievement growth. 

24
CSRD schools improved relative to non-CSRD schools in slightly more than half of the 

states, but few of these relative achievement gains were statistically significant.

Exhibit III-5 summarizes state-level findings on changes in performance made by CSRD 

schools relative to all non-CSRD schools in their states. This analysis examines change in the 

relative ranking of CSRD schools within each state to determine whether CSRD schools, on 

average, moved up in this ranking. Very few states showed a statistically significant change in 

the ranking of CSRD schools relative to non-CSRD schools. Looking at all changes, regardless 

of statistical significance, states showed improvement in the ranking of CSRD schools in 55 

percent of the cases and declines in 43 percent of the cases. 

As before, looking only at states with two years of change data shows somewhat more 

positive results for CSRD schools. Among these states, CSRD schools showed relative 

achievement gains in 57 percent of the cases, and they lost ground relative to comparison schools 

in 39 percent of the cases. 

Relative gains for CSRD schools were more common at the elementary school level. At 

the middle school level, states were about equally likely to show relative gains and relative losses 

in the ranking of CSRD schools versus other schools. High school results in reading mirrored 

the middle school patterns, but math results were more likely to show gains for CSRD schools. 

Exhibit III-5 

Summary of Change in Within-State Rankings for CSRD Schools 

All States States with Two Years of Change Data 

Number 

of States Positive 

No

Change Negative

Number 

of States Positive 

No

Change Negative

 Elementary Reading 38 21 1 16 9 6 0 3 

 Elementary Math 37 22 2 13 9 6 1 2 

 Middle School Reading 29 15 0 14 8 3 0 5

 Middle School Math 29 15 0 14 8 4 0 4

High School Reading 15 7 0 8 6 3 0 3 

High School Math 15 9 1 5 6 4 2 1 

 Total 163 89 4 70 46 26 2 18 

 % of Cases 55% 2% 43% 57% 4% 39% 

Exhibit reads: Across all states, subjects, and grade levels, CSRD schools made gains relative to non-CSRD

schools in 55 percent of the states, on average, and relative declines in 43 percent of the states. 

25
This report provides a preliminary analysis covering a time period during which CSRD 

schools had had limited time to implement their programs. 

Overall, these data show little evidence of a positive impact of the CSRD program, at this 

early stage, on student achievement on state assessments. In states where CSRD schools made 

significant gains, so too did non-CSRD schools. There are some positive indications in the data, 

including the findings that states are more likely to show gains than declines in the within-state 

rankings of their CSRD schools, although these changes were not statistically significant. The 

gains were more prevalent in states that have assessment trend data available for a longer period 

of time (three years instead of the usual two years), raising hope for future analyses when 

additional years of data are available. 

Once again, it is important to keep in mind that this is a very preliminary analysis, 

because the time frame covered in this report is too short to expect large effects of the CSRD 

program. At the time recorded by the test data points, most schools had been receiving CSRD 

funding for less than one or two full school years. 

Further analysis is needed as additional years of state assessment data become available 

in order to understand the impact of the CSRD program on student achievement. Additional 

time would allow for deeper implementation of reforms. Additional time would also provide 

increased data points, allowing for a more robust analysis. A later report on student achievement 

in the first cohort of CSRD grantees would offer a better understanding of the progress of 

schools participating in this program. 

26
Endnotes 

 

1

 A. Odden, with E. Odden (1995), Educational Leadership for America’s Schools, New York: McGraw-Hill, Inc.; 

T. Peters and R. Waterman (1982), In Search of Excellence, New York: Harper and Row; T. Peters (1992), 

Liberation Management, New York: Alfred Knopf; J. Galbraith, E. Lawler, et al. (1993), Organizing for the Future, 

San Francisco: Jossey Bass; A. Mohrman, S.A. Mohrman, G. Ledford, et al. (1989), Large-Scale Organizational

Change, San Francisco: Jossey Bass; P. Berman and D. Chambliss (2000), Readiness of Low-Performing Schools

for Comprehensive Reform, RPP International: Emeryville, Calif.; R. Edmonds (1979), “Effective schools for the 

urban poor,” Educational Leadership, 37(1); M. Fullan and M.B. Miles (1992), “Getting reform right: What works 

and what doesn’t,” Phi Delta Kappan, 73(10); U.S. Department of Education, Office of the Under Secretary and 

Office of Elementary and Secondary Education (January 2001), School Improvement Report: Executive Order on

Actions for Turning Around Low-Performing Schools, Washington. 2

 J.V. D’Agostino, G.D. Borman, L.V. Hedges, and K.K. Wong (1998), “Longitudinal achievement and Chapter 1

coordination in high poverty schools: A multi-level analysis of the Prospects data,” Journal of Education for

Students Placed at Risk, 3(4). 3

 U.S. Department of Education, Policy and Program Studies Service (1997), Special Strategies Study: Final Report, 

Washington. 4

 L. Desimone (2000), Making Comprehensive School Reform Work, New York: ERIC Clearinghouse on Urban 

Education. 5 T.R. Sizer (1992), Horace’s School: Redesigning the American High School, New York: Houghton Mifflin; 

H. Levin (1991), Learning from Accelerated Schools, Philadelphia: Pew Charitable Trust; L. Darling-Hammond 

(1995), “Policy for restructuring,” in The Work of Restructuring Schools: Building from the Ground Up, edited by

A. Lieberman, New York: Teachers College Press; L. Darling-Hammond (1997), The Right to Learn: A Blueprint

for Creating Schools that Work, San Francisco: Jossey-Bass; A.G. Hess, Jr. (1995), Restructuring Urban Schools: 

A Chicago Perspective, New York: Teachers College Press. 6

 Desimone (2000). 7

 S.J. Bodilly (1998), Lessons from New American Schools’ scale-up phase: Prospects for bringing designs to

multiple schools, Santa Monica, Calif.: RAND (MR–1777–NAS). 8

 M. Berends, S., Bodilly, and S.N. Kirby (2002), Facing the challenges of whole-school reform: New American

Schools after a decade, Santa Monica, Calif.: RAND (MR–1498–NAS). 9

 M. Berends (2000), “Teacher reported effects of New American Schools design: Exploring relationships to

teacher background and school content,” Educational Evaluation and Policy Analysis, 22(1), p 65-82. 10 For more information, see www.goodschools.gwu.edu. 11 For more information, see www.sedl.org/csrd/awards.html. 12 U.S. Department of Education, Office of the Under Secretary, Planning and Evaluation Service (2000), Early 

Implementation of the Comprehensive School Reform Demonstration Program. Washington. 13 D. McLaughlin, V. Bandeira de Mello, S. Cole, and C. Blankenship, National Longitudinal School-Level State 

Assessment Score Database. The database is available on the Web at http://208.253.216.16/assessment. 14 The NWREL catalog is available at http://www.nwrel.org/scpd/catalog/guide/csr.shtml. 15 R. Herman, D. Aladjem, et al. (1999), An Educators Guide to Schoolwide Reform, Arlington, Va.: Education 

Research Services. 16 McLaughlin et al. 17 For more information, see www.sedl.org/csrd/awards.html. 

27
 

28
Acknowledgments 

Many people, at the U.S. Department of Education and at outside research organizations, 

facilitated the production of this report. In the Policy and Program Studies Service at the U.S. 

Department of Education, Michelle LaPointe and Stephanie Stullich were the primary authors of 

this report. Alan Ginsburg, Daphne Hardcastle, and Beth Franklin provided helpful comments 

and suggestions, and Andrew Yarrow edited this document. In the Office of Elementary and 

Secondary Education, Hugh Burkett and Fran Walters were particularly generous with their time: 

they helped shape the studies whose data forms the basis of this report and reviewed early drafts. 

This report relied on the research and analysis conducted under several contracts, by 

various researchers. We would like to thank the research teams at Westat, led by Babette 

Gutmann; at the RAND Corporation, led by Sheila Kirby and Mark Berends; and at the Cosmos 

Corporation, led by Robert Yin. 

We also wish to thank the schools, districts, and states across the country who 

participated in the studies that informed this report. 

29
30
Appendix A 

Methodology for Achievement Analyses 

This appendix outlines the methodology the RAND Corporation used to conduct the achievement 

analysis. In order to understand the evaluation questions outlined above, separate state-by-state analyses 

of CSRD and non-CSRD schools were performed. Although the detailed analyses are not reported here, 

the state-specific analyses encompassed the following tasks: 

• First, a profile of each state’s accountability system was developed in order to document the data 

contained in the National Longitudinal School-Level State Assessment Score Database as of 

January 2002 and understand the testing regime over the time period covered by the study

(SY 1998–99 through SY 2000–01).1

• Second, a comparison group of non-CSRD schools that were similar to the CSRD schools in 

terms of poverty status was defined. Because the CSRD program emphasized selecting schools 

identified in need of improvement, these schools are likely to be both higher poverty and lower 

achieving than schools in the rest of the state. Therefore, to understand the gains made by these 

schools, it is important to compare them with the performance and the gains made by schools that 

match them in terms of poverty, where possible. Because analysts may disagree with our choice 

of comparison schools, the analyses were bound by comparing the CSRD schools to a set of 

comparison schools as well as the to entire set of non-CSRD schools in the state. 

• Third, the average level of performance in CSRD schools was examined to see whether and how 

it differed from that of non-CSRD comparison schools and all non-CSRD schools. Changes in the 

average level of performance among CSRD schools from the baseline year to one or two years 

after receipt of CSRD funds were compared to see whether these schools improved over time. 

Changes in the average level of performance of comparison schools and all non-CSRD schools in 

the state also were calculated. 

• Fourth, CSRD schools were ranked in two distributions: one of comparison non-CSRD schools 

and the other of all non-CSRD schools in the state, and gains made by CSRD schools relative to 

these two distributions were examined. 

• Fifth, the gains in all schools and all states in the average level of performance in CSRD schools 

and in the rankings of CSRD schools relative to comparison non-CSRD schools and all nonCSRD schools were summarized. 

These tasks are further detailed below. 

Profile Each State’s Testing Regime 

Analysts at RAND developed a profile to examine the test used for accountability purposes, the grades 

and subjects tested and the metric in which they reported scores. The profile included detailed notes 

regarding changes in tests or test metrics over time, the nature and extent of accommodations, and a link 

to the state’s Web site. This allowed us to examine whether the reported scores were comparable over 

time. 

 

1 This information was not present in the database. 

A-1
Two examples illustrate why this task was an important and necessary part of the analysis. Mississippi 

adopted a new criterion-referenced assessment system, the Mississippi Curriculum Tests, in 2000-01. 

Thus change scores could not be computed for Mississippi schools (Exhibit A-1). 

Exhibit A-1. State Profile of Testing Regime: Mississippi 

____________________________________________________________________________________ 

Test on AIR Database 

TerraNova, norm-referenced exam

Writing Assessment

Iowa Test of Basic Skills (ITBS), norm-referenced exam

Year Tests First Administered 

1999-2000 – TerraNova 

1994-95 – ITBS

Time of Test Administration 

Spring – TerraNova 

Fall – ITBS 

Test Scores on AIR Database 

Grades Test Subjects Years Reported Test Metric 

3rd - 8th TerraNova Reading; language arts;

mathematics; total score 

1999-2000 Percentile

4th, 7th Writing

Assessment 

Writing 1999-2000 Percent at levels 

4th-9th ITBS Reading; language arts;

mathematics 

1994-95; 1995-96; 

1996-99; 1997-98; 

1998-99

Mean NCE 

Data Notes 

• Between the fall of 1994 and 1998, Mississippi administered the ITBS in grades four through eight and the 

Tests of Achievement and Proficiency in grade nine. Subject area tests and the functional Literacy Examination

were used to determine high school level proficiency. 

• In the summer of 1999, Mississippi began the design and approval process for the new assessment system that

includes the Terra Nova and criterion-referenced assessments. The criterion-referenced assessments 

(Mississippi Curriculum Tests) were first administered in 2000-01. Mississippi warns against calculating 

change scores across the two tests. 

• For students with disabilities, the IEP committee determines whether the student will participate in the 

assessment. Scores will be excluded from school-level reporting for students who receive accommodations that 

exceed those allowed for a specific test, students who are provided instruction utilizing an alternate or parallel 

curriculum in any of the assessed areas, or students whose IEPs indicate that due to educational delay they are 

not expected to meet the basic curriculum objectives for that year. 

• Limited English proficient (LEP) students are expected to participate in the assessments, but may be exempted

for no more than two years based on a language assessment designed for LEP students.

• The scores of LEP students who participate in the assessment are included in the overall school score unless the

student met the requirements to be exempted but elected to participate in the assessment or if the appropriate 

accommodations for the assessment were insufficient or inappropriate to meet a student’s needs. 

State Accountability Web Site: www.mde.k12.ms.us/acad/osa/testdata.html. 

____________________________________________________________________________________ 

A-2
In Utah, although the test had not changed, the Salt Lake City School District changed their practices for 

exempting limited English proficient students in 2000-01 (Exhibit A-2). Thus, the 2000-01 results could 

not be compared with those of earlier years. For this reason, we excluded schools in this district from our 

analyses of Utah schools. 

Exhibit A-2. State Profile of Testing Regime: Utah 

____________________________________________________________________________________ 

Test on AIR Database 

Stanford-9, norm-referenced exam

Year Test First Administered 

1997-98

Time of Test Administration 

Fall 

Test Scores on AIR Database 

Grades Subjects School Years Reported Test Metric 

3rd Reading; mathematics; language; 

science; social studies 

2000-01 Percentile

5th, 8th, 11th Reading; mathematics; language;

science; social studies; total battery 

1997-98;

1998-99;

1999-2000;

2000-01

Percentile 

Data Notes 

• Utah developed a basic skills test to be administered as an exit exam to 10th-grade students starting in 2003. 

In addition, districts administer criterion-referenced subject tests to all grades as part of the Core Assessment

Program, but these scores are not included in the AIR database. 

• Students with disabilities may be exempted from testing if their IEP team determines that the student cannot

demonstrate his or her knowledge without an accommodation. 

• Limited English proficient (LEP) students may be exempted from testing only if they have received less than

three years of instruction primarily in English and cannot demonstrate their knowledge in English.

• In 2000-01, the Salt Lake City School District changed their practices for exempting LEP students in a way that 

was inconsistent with the state requirements. In many cases, students were exempted who had not reached the

“fluent/competent” level on the IPT Language Proficiency Tests, regardless of how long the student had

received in English instruction. This resulted in the exemption, in some schools, of up to 75 percent of the 

students. The exemption of additional students from Salt Lake School District alters the data trend that has 

been established and makes the 2000-01 results noncomparable. 

State Accountability Web Site: www.usoe.k12.ut.us/eval/. 

_____________________________________________________________________________________

A-3
Analyzing the Level of Performance in CSRD Schools 

As mentioned earlier, states used different metrics for reporting test scores, some of which lend 

themselves to aggregation across sets of schools, while others do not. In addition, some states had no 

school-level test score information. Exhibit A-3 shows the states that fell into each category.

Exhibit A-3. Metrics Used by States in Reporting School-Level Test Scores

Test Score Metric States 

Scale score or normal curve 

equivalent (NCE) score 

CA, DE, FL, IL, KY, MA, ME, MN, MS, NC, NH, NJ, NY, PA, SC 

Percent passing level AK, CO, CT, DC, HI, IN, KS, LA, MD, MI, MO, OH, OK, OR, RI, TX, VA, VT, 

WA, WI, WY 

Percentile AL, AR, AZ, GA, ID, MS (post-baseline year), NV, SD, TN, UT, WV 

No uniform, school-level test

scores

IA, MT, ND, NE, NM 

If the test metric allowed aggregation across sets of schools (scale scores/NCEs, percent passing)

When the data allowed us to aggregate across schools, data were summarized in a table that provides a 

description of the average performance levels in the original metric (for example, percent passing) to 

allow the reader to examine the average performance levels for CSRD schools, a comparison group of 

non-CSRD schools, and all non-CSRD schools in the entire state. The table includes sample sizes and 

provides an indication of how reliable the estimate was. 

Simple two-sample t-tests were conducted to determine whether the average level of performance of 

CSRD schools differed from that of comparison schools, and that of all non-CSRD schools in the state for 

each school year, using a p-value of 0.05 to determine statistical significance. The test for levels is useful 

to determine if there are significant differences between the two groups at a point in time. 

In the tables accompanying each state summary, only results for data aggregated across grades for each 

school level are shown. Thus, although the detailed analyses examined means and changes in means for 

all grades for which information was available, the summary tables show only the results for elementary, 

middle, and high schools. When data were available, for elementary schools, third- through fifth-grade 

scores were averaged; for middle schools, seventh- and eighth-grade scores were averaged; and for high 

schools, ninth- through 11th-grade scores were averaged. In some states only a particular grade was tested, 

in which case that score was reported for the school as a whole. Each table shows the grades on which the 

reported score was based. 

Exhibit A-4, which shows data for California schools, reported data as scale scores for four years, 

SY 1997-98 to SY 2000-01. There were 47-49 CSRD elementary schools (depending on the year), 13 

middle schools, and 10 high schools. As mentioned above, third- through fifth-grade scores were 

averaged to get a score for each elementary school, seventh- and eighth-grade scores to get a score for 

each middle school; and ninth- through 11th-grade scores to get a score for each high school. School 

scores were then averaged to get a mean for CSRD schools, a mean for the comparison group, and a mean 

for all non-CSRD schools in the state (as well as all schools). 

Looking at the baseline year, SY 1998-99, the mean for CSRD elementary schools in reading was 603 

compared with 613 for the comparison group and 629 for all non-CSRD schools. By 2000-01, the means 

for each group had increased to 609 (CSRD schools), 619 (comparison group), and 635 (all non-CSRD 

schools). Data for middle and high schools can be interpreted in a similar fashion. 

A-4
Exhibit A-4. School-Level Achievement in California Elementary Schools: 

SY 1997–98 through SY 2000–01, Average Scale Scores 

CSRD

Schools 

High-Poverty

Non-CSRD Schools 

All 

Non-CSRD Schools 

1997–

1998

1998–

1999

1999–

2000

2000–

2001

1997–

1998

1998–

1999

1999– 

2000

2000– 

2001

1997–

1998

1998–

1999

1999– 

2000

2000–

2001

Average Scale Score 

Elementary Schools (3rd, 4th, & 5th Grades) 

Reading 598 603 607 609 609* 613* 616* 619* 626* 629* 632* 635* 

(Number of schools) (n=47) (n=48) (n=47) (n=49) (n=2405) (n=2457) (n=2438) (n=2477) (n=4522) (n=4619) (n=4578) (n=4734) 

Mathematics 593 601 608 612 603* 608* 616* 620* 616* 622* 629* 633* 

(Number of schools) (n=47) (n=48) (n=47) (n=49) (n=2411) (n=2460) (n=2439) (n=2479) (n=4535) (n=4622) (n=4578) (n=4741) 

Middle Schools (7th & 8th Grades) 

Reading 657 662 664 665 666* 667 669 670 679* 681* 682* 683* 

(Number of schools) (n=13) (n=13) (n=13) (n=13) (n=730) (n=753) (n=745) (n=761) (n=1665) (n=1703) (n=1663) (n=1777) 

Mathematics 655 661 663 662 661 664 666 668 673* 675* 678* 680* 

(Number of schools) (n=13) (n=13) (n=13) (n=13) (n=732) (n=753) (n=745) (n=761) (n=1665) (n=1704) (n=1659) (n=1780) 

High Schools(9th, 10th, & 11th Grades)

Reading 671 673 675 673 675* 675 677 676 688* 688* 689* 688* 

(Number of schools) (n=10) (n=10) (n=10) (n=10) (n=236) (n=241) (n=242) (n=242) (n=1120) (n=1137) (n=1110) (n=1186) 

Mathematics 677 680 685 684 680 682 683 683 690* 691* 693* 693* 

(Number of schools) (n=10) (n=10) (n=10) (n=10) (n=238) (n=241) (n=243) (n=244) (n=1118) (n=1143) (n=1112) (n=1188) 

Note: * indicates a statistically significant difference between the mean score for the particular group (i.e., high-poverty non-CSRD sch

schools) and mean score for the CSRD schools (p-value<0.05). 

A-5
To determine how the average performance of CSRD schools compared with that of similar non-CSRD 

comparison schools and all other non-CSRD schools in the state, simple two sample t-tests were 

conducted, using a p-value of .05 to judge statistical significance. Statistically significant results are 

indicated by an asterisk. In California: 

• At the elementary level (grades 3-5 combined), CSRD schools scored significantly below 

comparison schools (high-poverty non-CSRD schools in the state) in both subjects across the four 

years (SY 1997-98 through SY 2000-01). 

• At the middle and high school levels, CSRD schools scored significantly below comparison 

schools in SY 1997-98, but there was no difference in the average performance of CSRD and 

comparison schools from SY 1998-99 to SY 2000-01. 

• At the elementary, middle, and high school levels, CSRD schools scored significantly lower than 

all non-CSRD schools in both reading and mathematics and across all four years (SY 1997-98

through SY 2000-01). 

If the test metric did not allow aggregation across sets of schools (percentile ranks)

The most common method of ensuring comparability of scores across different tests or aggregating across 

schools is to calculate a z-score for each school that tells us how far a raw score is from the mean of the 

distribution of the state schools, the distance being expressed in standard deviation units. However, the zscore assumes that the underlying distribution is normal or nearly so. In some states, given the metric 

used and the test, this assumption is not likely to be met. In some states, the small sample size of the 

CSRD schools may make the mean subject to considerable measurement error. Thus, a method was 

employed that has the advantages of being distribution-free. 

First, all schools within a given distribution were ranked. All schools within the overall state distribution 

were ranked to examine the level of performance in CSRD and non-CSRD schools. Essentially, the 

procedure ranked values (i.e., percentile rank) from lowest to highest, assigning the rank 1 to the school 

with the lowest score, 2 to the next higher, and so on up to rank n, the number of schools in the 

distribution. Tied values were given averaged ranks. 

Next, the rank ri for each school i, was divided by (n+1), to get values in the range 0 to 1; these values 

estimate the cumulative distribution function. An inverse Gaussian cumulative distribution function was 

then applied to these fractional ranks to obtain standardized scores. In particular, a value was estimated 

for each school—call it a standardized unit, ui to distinguish it from a z-score—as follows: 

u i= Φ-1 (ri/(n+1)) 

where Φ-1 is the inverse cumulative normal (e.g., probit) function, ri is the rank of the ith school, and n is 

the total number of schools in the distribution. It has the property that it gives somewhat greater weight 

to increases in the tails of the distribution than to increases in the middle, which is important when 

examining CSRD schools because so many of them are low-performing. 

The values of the uis range from approximately –3 to +3 and are analogous to z-scores in terms of 

showing how far each school is from the center of the distribution. However, they avoid the assumption 

that the underlying distribution of scores is normal. If the underlying distribution is normal, they will 

essentially be equal to z-scores. The uis can be averaged across schools, even when the original metric 

(such as percentile rank) does not lend itself to averaging across a set of schools. 

A-6
The uis were averaged, yielding a mean score for CSRD schools, comparison non-CSRD schools, and all 

non-CSRD schools in the state. Simple two-sample t-tests were conducted to determine whether the 

average performance in these schools differed from each other. 

These data are summarized in tables that are part of the state-specific summaries. The tables, as 

mentioned earlier, show data at the school level, aggregated across grades. 

Exhibit A-5, which shows a state that reports test scores in percentile ranks, presents standardized scores 

for Alabama for SY 1998-99 and SY 1999-2000 (the only two years for which data were available on the 

AIR database). Thus, instead of reporting an average for the CSRD schools in terms of percentile ranks, it 

reports an average in terms of standardized units. CSRD schools are nested within the distribution of all 

schools in the state. The mean score for all schools is by definition equal to zero. The mean score for all 

non-CSRD schools is very close to zero because they represent a large proportion of all the schools in the 

state; thus the mean on a standardized scale for these schools must be close to the overall mean of the full 

distribution. The mean score for CSRD schools represents their standing relative to all schools in the 

state; similarly the mean score for the comparison schools represents their standing relative to all schools 

in the state. Exhibit A-5 shows that: 

• Elementary CSRD schools scored significantly lower than both the highest-poverty non-CSRD 

schools and all non-CSRD schools in both subjects and across both years. For example, in 1998-

99, CSRD schools scored 1.4 standardized units below all other schools in the state in reading, 

significantly below the average for the comparison group (1 standardized unit below all other 

schools in the state), and significantly below the 0.0 mean for all non-CSRD schools in the state. 

While the highest-poverty schools also rank lower in the overall distribution, they tend to rank

higher than CSRD schools. 

• At the middle school level, both the comparison and CSRD schools ranked about 1.0 standardized 

units below the overall mean for the state in both subjects. CSRD schools scored significantly

below non-CSRD schools in both subjects and in both years but are similar to the comparison

schools. 

• At the high school level, the CSRD schools are between 0.9 – 1.4 standardized units below the 

overall mean for the state, while comparison schools are between 1.1 – 1.6 standardized units 

below. CSRD schools scored significantly below non-CSRD schools in both subjects and in both 

years but are similar to the comparison schools. 

A-7
Exhibit A-5. School-Level Achievement in Alabama Schools: 

SY 1998–99 through SY 1999–2000, Standardized Scores 

CSRD 

Schools 

Highest-Poverty 

Non-CSRD Schools 

All 

Non-CSRD Schools 

All 

Schools 

1998–

1999

1999–

2000

1998–

1999

1999–

2000

1998–

1999

1999–

2000

1998–

1999

1999–

2000

Standardized Scores 

Elementary Schools 

(3rd, 4th, & 5th Grades) 

(Number of schools) (n=24) (n=24) (n=136) (n=138) (n=646) (n=644) (n=670) (n=668) 

Reading -1.4 -1.4 -1.0* -1.0* 0.0* 0.0* 0.0 0.0

(Number of schools) (n=25) (n=24) (n=137) (n=138) (n=648) (n=644) (n=673) (n=668) 

Mathematics -1.3 -1.2 -0.7* -0.8* 0.0* 0.0* 0.0 0.0

Middle Schools 

(7th & 8th Grades) 

(Number of schools) (n=14) (n=15) (n=57) (n=62) (n=463) (n=463) (n=477) (n=478) 

Reading -1.1 -1.2 -1.2 -1.2 0.0* 0.0* 0.0 0.0

(Number of schools) (n=14) (n=15) (n=58) (n=62) (n=464) (n=463) (n=478) (n=478) 

Mathematics -0.9 -1.0 -1.0 -1.0 0.0* 0.0* 0.0 0.0

High Schools 

(9th, 10th, & 11th 

Grades)

(Number of schools) (n=17) (n=17) (n=23) (n=27) (n=343) (n=354) (n=360) (n=371) 

Reading -1.4 -1.2 -1.6 -1.5 0.1* 0.0* 0.0 0.0

(Number of schools) (n=17) (n=17) (n=24) (n=27) (n=349) (n=354) (n=366) (n=371) 

Mathematics -1.1 -0.9 -1.1 -1.2 0.1* 0.0* 0.0 0.0

Note: * indicates a statistically significant difference between the mean score for the particular group (i.e., highestpoverty non-CSRD schools or all non-CSRD schools) and mean score for the CSRD schools (p-value<0.05). 

Analyzing the Change in Level of Performance over Time in CSRD Schools 

Relative to Comparison Group and All Non-CSRD Schools 

Change in performance over time was analyzed in three ways: 

A. For states in which data were reported in original units that can be aggregated (average scale 

score, mean NCEs, or percent passing) 2

For each CSRD school, comparison school, and non-CSRD school in the state, a simple change score was 

computed from baseline to one to two years after receipt of CSRD funds. The key questions were whether 

the CSRD schools made any gains (or losses) over time, whether these changes were statistically

significant, and whether comparison schools and all non-CSRD schools experienced similar changes in 

test scores. The small sample sizes for CSRD schools in most states lead to two problems: the average 

scores are themselves subject to measurement error, and it is harder to find statistically significant gains 

for CSRD schools. 

 

2

In states that reported percentile ranks, gain scores were not computed using the ranks. This analysis was focused

on the gain scores of schools. For example, did the average scale score of CSRD schools increase over time? In the 

next section, the transformed metric was used to examine the question of whether the CSRD schools improved their 

relative standing compared with similar schools. 

A-8
There are three points that should be noted about the gain scores. First, for CSRD schools, the gain score 

represents different “post-CSRD” periods depending on the month the schools received their funds. 

Second, in some states, gains are measured over a one-year period, from baseline to one year after receipt 

of CSRD funds (SY 1998–99 to SY 1999–2000) and in others, over a two-year period (SY 1998–99 to 

SY 2000–01). Third, even if differences in gains were evident across the maximum of two years, it 

would be difficult to attribute those gains to the CSRD program. Ideally, one would want to control for 

the length of time for which these schools received CSRD funding and also for the length of time actually

implementing the components of reform called for by CSRD. For example, some schools received CSRD 

awards as late as November 1999; for these schools, the spring 2000 test score does not even represent a 

full year after receipt of funding. Thus these schools have had very little time to implement reform, much

less see the effect of those reforms. All that can be measured in this analysis is whether CSRD schools 

made gains over this particular one- or two-year period and whether comparison schools and all nonCSRD schools made similar gains. 

Returning to the California example, shown in Exhibit A-4, the significance of the change in scale scores 

was tested separately for the CSRD, non-CSRD comparison schools, and all non-CSRD schools that had 

scores in SY 1998-99 and SY 2000-01. 

• Over this two-year period, the CSRD elementary schools made statistically significant increases 

of six points in reading, and these gains were similar to those made by the comparison group and 

all non-CSRD elementary schools. Similarly, in mathematics, California elementary schools 

made statistically significant gains of 11 points, which were similar to the 11-12 point gains made 

by comparison and all non-CSRD schools in the state. 

• The CSRD middle schools made gains of three points in reading and one point in mathematics, 

and these were equal to or smaller than those made by non-CSRD comparison schools (three 

points in reading and four points in mathematics). All non-CSRD middle schools made gains of 

two points and five points in reading and mathematics respectively. None of these changes was 

statistically significant. 

• The CSRD high school scores remained the same in reading, as did the scores of comparison and 

all non-CSRD high schools. CSRD high school scores increased by four points in mathematics; 

this was larger than the one- to two-point increase experienced by comparison and all non-CSRD 

schools. None of the changes was statistically significant. 

B. In order to test whether CSRD schools had improved relative to comparison schools, all states 

for which change scores could be calculated: 

CSRD schools were ranked in the distribution of non-CSRD comparison schools, using the score for the 

individual school (either in the original metric or the transformed metric) and calculated uis as explained 

above. Because of the variability in the data across grades, the focus was only on the aggregated test score 

for the school (i.e., averaged scores across grades). Gain scores in the ui metric, δi, were then calculated 

for each school in the two groups (CSRD schools and comparison non-CSRD schools). The change was 

calculated from the baseline year, SY 1998–99, to either SY 1999–00 or SY 2000–01, depending on

availability of data. A one-tailed t-test was used to examine the hypothesis that the relative gains of the 

CSRD schools (relative to the comparison group) were significantly greater than zero. In other words, did

the CSRD schools move up in the distribution of comparison and CSRD schools. 

A-9
The null and alternative hypotheses follow the following notational form: 

H0: δi ≤ 0 

HA: δi > 0 

The null hypothesis is one of no difference (δ) between the standardized mean scores against a one-sided 

alternative that CSRD schools gained relative to the comparison group.3

 Significant differences (pvalues<.05) are noted with an asterisk in the tables. 

C. In order to test whether CSRD schools had improved relative to all non-CSRD schools, all states 

for which change scores could be calculated: 

Because the choice of a comparison group potentially could bias results in favor of finding an effect, our 

estimates were bounded by ranking CSRD schools in the distribution of all non-CSRD schools and 

determining whether CSRD schools had improved relative to these schools. The procedure was exactly

the same as the one outlined in the above section. Once the changes were calculated, a similar one-tailed 

test was conducted to see whether the gains of the CSRD schools (relative to all non-CSRD schools) were 

significantly greater than zero. Obviously, the same caveats listed above apply to this analysis. 

Example 

The examples of California and Alabama illustrate the analyses described in sections B and C above. 

Exhibits A-6 and A-7 show the change scores for California and Alabama CSRD schools relative to the 

comparison schools (high-poverty non-CSRD schools in California and highest-poverty non-CSRD 

schools in Alabama) and all non-CSRD schools. Note that the change score for California schools 

represents change over a two-year time period (SY 1998-99 to SY 2000-01) while the change score for 

Alabama represents a one-year change (SY 1998-99 to SY 1999-2000). 

California (Exhibit A-6): 

Relative to comparison schools (high-poverty non-CSRD schools): 

• The average elementary CSRD school score declined by 0.09 standardized units in reading and 

by 0.04 standardized units in mathematics. 

• At the middle school level, the average CSRD school score declined 0.05 standardized units in

reading and 0.17 standardized units in mathematics. 

• At the high school level, the average CSRD high school score remained essentially unchanged in 

reading but increased by 0.19 units in mathematics. 

• However, none of these changes was statistically significant from zero; i.e., CSRD schools did 

not make significant gains relative to comparison schools in California. In statistical terms, this 

result fails to reject the null hypothesis that the average change in CSRD schools relative to the 

comparison non-CSRD schools over this time period was essentially zero. 

 

3 When there was only one CSRD school at a grade level in a state, this test could not be run. A single value could 

not be tested for statistical significance against zero. 

A-10
Exhibit A-6. School-Level Achievement Changes in CSRD Schools 

Relative to Comparison Non-CSRD Schools and Relative to All Non-CSRD Schools in California, 

SY 1998–99 to SY 2000–01 

Standardized Scores 

Relative to Comparison 

Non-CSRD Schools 

Relative to All NonCSRD Schools 

Elementary Schools 

(Grades 3–5 Combined)

(Number of schools) (n=48) (n=48) 

Reading -0.09 -0.06

Mathematics -0.04 -0.02

Middle Schools 

(Grades 7–8 Combined) 

(Number of schools) (n=13) (n=13) 

Reading -0.05 -0.03

Mathematics -0.17 -0.12

High Schools

(Grades 9–11 Combined) 

(Number of schools) (n=10) (n=10) 

Reading -0.03 0.01

Mathematics 0.19 0.11

Exhibit reads: On average, relative to high-poverty non-CSRD schools, CSRD elementary

schools’ reading scores decreased by 0.09 standardized units from SY 1998–99 to SY 2000–01. 

Relative to all non-CSRD schools: 

• The average elementary CSRD school score declined by 0.06 standardized units in reading and 

remained essentially unchanged in mathematics. 

• At the middle school level, the average CSRD school score declined 0.03 standardized units in

reading and 0.12 standardized units in mathematics. 

• At the high school level, the average CSRD high school score remained essentially unchanged in 

reading but increased by 0.11 units in mathematics. 

Again, none of these changes was statistically significant from zero; i.e., CSRD schools did not make 

significant gains relative to all non-CSRD schools in California. 

Alabama (Exhibit A-7) 

Relative to comparison schools (highest-poverty non-CSRD schools): 

• At the elementary level, the average score in CSRD schools remained essentially the same in 

reading and increased by 0.17 standardized units in mathematics over this one-year time period. 

• CSRD middle schools did somewhat worse compared with comparison schools, with a decline of 

0.34 standardized units in reading and 0.21 standardized units in mathematics. 

A-11
• CSRD high schools remained at the same level relative to comparison schools over this time 

period. 

However, none of these changes were statistically significant; i.e., CSRD schools did not make significant 

gains relative to comparison schools in Alabama. That is, they fail to reject the null hypothesis that the 

average change in CSRD schools relative to the comparison schools over this time period was zero. 

Relative to all non-CSRD schools: 

• At the elementary level, the average score in CSRD elementary schools remained essentially the 

same in reading and increased by 0.17 standardized units in mathematics over this one-year time

period. 

• CSRD middle schools did somewhat worse compared with all non-CSRD schools, with a decline 

of 0.16 standardized units in reading and 0.18 standardized units in mathematics. 

• CSRD high school scores increased by 0.08 standardized units in reading and 0.12 standardized 

units in mathematics. 

• However, none of these changes was statistically significant from zero; i.e., CSRD schools did 

not make significant gains relative to all non-CSRD schools in Alabama. 

Exhibit A-7. School-Level Achievement Changes in CSRD Schools 

Relative to Comparison Non-CSRD Schools and Relative to All Non-CSRD Schools in Alabama, 

SY 1998–99 to SY 1999–2000 

Standardized Scores 

Relative to Comparison 

Non-CSRD Schools 

Relative to All

Non-CSRD Schools 

Elementary Schools 

(3rd, 4th, & 5th Grades) 

(Number of schools) (n=22) (n=22) 

Reading -0.03 -0.03

Mathematics 0.17 0.17

Middle Schools 

(7th & 8th Grades) 

(Number of schools) (n=14) (n=14) 

Reading -0.34 -0.16

Mathematics -0.21 -0.18

High Schools 

(10th Grade)

(Number of schools) (n=17) (n=17) 

Reading -0.03 0.08

Mathematics 0.00 0.12

Exhibit reads: On average, relative to the highest-poverty non-CSRD schools, elementary CSRD

schools’ reading scores decreased by 0.03 standardized units from SY 1998–99 to SY 1999–2000.

A-12