U.S. Department of Education DOC # 2004-20

Office of the Deputy Secretary

PPSS P O L I C Y A N D P R O G R A M S T U D I E S S E R V I C E

Longitudinal Assessment of

Comprehensive 

School Reform Program

Implementation and Outcomes

F I R S T- Y E A R R E P O R T
Longitudinal Assessment of 

Comprehensive School Reform Program Implementation 

and Outcomes 

First-Year Report

By: 

Naida C. Tushnet 

John Flaherty Jr. 

Andrew Smith 

WestEd 

Los Alamitos, Calif. 

Prepared for: 

U.S. Department of Education 

Office of the Deputy Secretary 

Policy and Program Studies Service 

2004 
This report was prepared for the U.S. Department of Education under Contract Number ED01CO0129. The views 

expressed herein do not necessarily represent the positions or policies of the Department of Education. No official 

endorsement by the U.S. Department of Education is intended or should be inferred. 

U.S. Department of Education 

Rod Paige 

Secretary 

Office of the Deputy Secretary 

Eugene Hickok 

Deputy Secretary 

Policy and Program Studies Service 

Alan Ginsburg 

Director 

Policy and Analytic Studies Division 

David Goodwin 

Director 

December 2004 

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to 

reprint this publication is not necessary, the suggested citation is: U.S. Department of Education, Office of the 

Deputy Secretary, Policy and Program Studies Service, Longitudinal Assessment of Comprehensive School Reform 

Program Implementation and Outcomes: First-Year Report, Washington, D.C., 2004. 

To order copies of this report, write: 

ED Pubs 

Education Publications Center 

U. S. Department of Education 

P. O. Box 1398 

Jessup, MD 20794-1398; 

via fax, dial (301) 470-1244; 

or via electronic mail, send your request to: edpubs@inet.ed.gov. 

You may also call toll-free: 1-877-433-7827 (1-877-4-ED-PUBS). If 877 service is not yet available in your area, 

call 1-800-872-5327 (1-800-USA-LEARN); Those who use a telecommunications device for the deaf (TDD) or a 

teletypewriter (TTY) should call 1-800-437-0833. 

To order online, point your Internet browser to: www.edpubs.org. 

This report is also available on the Department’s Web site at http://www.ed.gov/about/offices/list/ods/ppss. 

 

On request, this publication is available in alternate formats, such as Braille, large print, audiotape, or computer 

diskette. For more information, please contact the Department’s Alternate Format Center at (202) 260-9895 or (202) 

205-8113. 
iii 

Contents 

Exhibits.........................................................................................................................................................................v

Executive Summary.................................................................................................................................................. vii

Study Purpose ....................................................................................................................................................... viii

First-Year Findings..................................................................................................................................................ix

I. Introduction..............................................................................................................................................................1

Background...............................................................................................................................................................1

Study Purpose ...........................................................................................................................................................6

Focus and Organization of the Report ......................................................................................................................8

II. Evaluation Design...................................................................................................................................................9

Study Data Collection and Analysis .........................................................................................................................9

Methods Used in the First-Year Report ..................................................................................................................10

III. First-Year Findings.............................................................................................................................................13

Targeting of CSR Funds .........................................................................................................................................13

Implementation of School Reform Activities in CSR Schools and Other Schools.................................................18

The Influence of State and District Policies on the Implementation of Comprehensive Reform Programs ...........37

IV. Summary..............................................................................................................................................................41

Summary of Findings..............................................................................................................................................41

Implications ............................................................................................................................................................42

Appendix A: Selected Data Tabulations..................................................................................................................43

Appendix B: Technical Appendix ............................................................................................................................53

Sample and Comparison School Selection for LACIO...........................................................................................53

Analysis Methods—Year One ................................................................................................................................62

Bibliography...............................................................................................................................................................65

v 

Exhibits 

Exhibit E-1 Eleven Components of Comprehensive School Reform Described in the No Child Left Behind Act... viii

Exhibit E-2 Entities Involved in Selecting a Reform Model or Approach at the School..............................................xi

Exhibit E-3 Status of Professional Development in School Reform .......................................................................... xii

Exhibit E-4 Types of Professional Development in Which CSR Teachers Participated During the First Year of 

Reform Implementation (2002-03) Compared with the Previous Year ................................................. xiii

Exhibit E-5 Entity Primarily Responsible for Supporting Reform Efforts at the School ...........................................xiv

Exhibit E-6 Sources of Funding that Contribute to Implementation and Operation of School Reform ......................xv

Exhibit E-7 Distribution of CSR Schools and Other Schools by School Poverty Rate ............................................ xvii

Exhibit 1 Eleven Components of Comprehensive School Reform Described in the No Child Left Behind Act ..........5

Exhibit 2 Distribution of CSR Schools and Other Schools by School Poverty Rate...................................................14

Exhibit 3 Distribution of CSR Schools and Other Schools by Percentage of Minority Students................................15

Exhibit 4 Distribution of CSR Schools and Other Schools by Urbanicity...................................................................16

Exhibit 5 Schools Reported Being Identified as Low-Performing or Sanctioned Because of Low Performance 

According to State Criteria ......................................................................................................................17

Exhibit 6 Average School-Level Z-Scores for Math and Reading at CSR Schools (2002 Cohort), Title I Schools, 

and All Schools........................................................................................................................................18

Exhibit 7 Aspects of Reform Covered by School Improvement Plans........................................................................20

Exhibit 8 Extent of Faculty or Teacher Participation in Reform at the School............................................................21

Exhibit 9 Factors Influencing the Creation of Performance Goals at the School ........................................................23

Exhibit 10 Source of Evidence to Link School Reform to Student Achievement .......................................................24

Exhibit 11 Status of Professional Development in School Reform .............................................................................25

Exhibit 12 Entity Primarily Responsible for Supporting Reform Efforts at the School ..............................................27

Exhibit 13 Sources of Funding That Contribute to Implementation and Operation of School Reform.......................28

Exhibit 14 Primary Organizer of the Scope and Sequence of Curriculum at the School.............................................29

Exhibit 15 Methods of Formal Communication from Schools to Parents ...................................................................30

Exhibit 16 Ways Parents Are Engaged and Involved at School ..................................................................................31

Exhibit 17 Schools, by Type, with Primary Reform Efforts Identified by Name.......................................................32

Exhibit 18 Number and Type of Named Primary Reform Efforts in Use at Schools ..................................................32

Exhibit 19 Teacher Enhancement Opportunities Available before and after Reform Implementation .......................34

Exhibit 20 Types of Professional Development in Which Teachers Participated in during the First Year of 

Reform Implementation (2002-03) Compared with the Previous Year ...................................................35

Exhibit 21 Influence of State and Local Policies on School Improvement Plans........................................................38

Exhibit 22 Control of Budget and Personnel in CSR and Non-CSR Schools.............................................................39

Exhibit 23 Entities Involved in Selecting a Reform Model or Approach at the School ..............................................40

Exhibit A-1 Baseline Student Achievement Scores.....................................................................................................43

Exhibit A-2 CSR and Non-CSR Schools Reporting Formal Comprehensive School Improvement Plans .................44

Exhibit A-3 Factors Influencing the Content of the School Improvement Plan ..........................................................45

Exhibit A-4 Source of Reform Plans at CSR and Non-CSR Schools..........................................................................46

Exhibit A-5 The Primary Designer for the Reform at Your School ............................................................................47

Exhibit A-6 Academic Subjects Included in School Goals or Benchmarks ................................................................48

Exhibit A-7 Types of School Performance Goals for Students at CSR and Non-CSR Schools ..................................49

Exhibit A-8 Factors Influencing the Evaluation of School Performance Goals ..........................................................50

Exhibit A-9 Professional Development Opportunities Available to Teachers in the First Year of CSR Reform 

(2002-03) Compared with the Previous Year ..........................................................................................51

Exhibit A-10 Type of Support Available through the District for Reform Efforts......................................................51

Exhibit A-11 Type of Support Available through the State for Reform Efforts..........................................................52

Exhibit B-1 Summary of Performance Indicators by State..........................................................................................55

Exhibit B-2 Type of School within All 2002 CSR Schools, CSR Sample, and Non-CSR Comparison Sample.........57

Exhibit B-3 Location of All 2002 CSR Schools, CSR Sample, and Non-CSR Comparison Sample..........................57

Exhibit B-4 Percentage of Students in Poverty at 2002 CSR Schools, CSR Sample and Non-CSR Comparison 

Sample .....................................................................................................................................................58

Exhibit B-5 Average School-Level Z-Scores (Math and Reading) .............................................................................58
vi 
vii 

Executive Summary 

The Comprehensive School Reform (CSR) program is one response to the persistent failure 

of some schools to provide students with educational opportunities to meet high standards for 

learning. The program was formed in an atmosphere of increased focus on school accountability 

and provides both a framework and the funding to enable schools to change their organization 

and practices so all students can achieve high standards. 

In 1998, Congress appropriated $145 million for the Comprehensive School Reform 

Demonstration (CSRD) program. It was designed not as an add-on to be placed on top of already 

existing programs and efforts but as a way to encourage schools to integrate local, state, and 

federal resources into a comprehensive effort that would better meet student learning needs. Like 

schoolwide Title I programs, CSRD was intended to help schools use multiple sources of funds 

and integrate programs while allowing flexibility and enhancing accountability for student 

learning. Its unique aspect was the expectation that schools would collaborate with expert 

partners to implement whole-school reform models that had a strong research base and a 

successful replication record. 

With the passage of the No Child Left Behind (NCLB) Act in 2002, CSR became a fully 

authorized program and is no longer a demonstration program. Further, NCLB described 11 

components of comprehensive school reform (Exhibit E-1), and, some argue, focused less on 

models than on the underlying processes that facilitate the kinds of changes needed in order for 

schools to ensure that all students learn. 
viii 

Exhibit E-1 

Eleven Components of Comprehensive School Reform 

Described in the No Child Left Behind Act

 Proven methods and strategies for student learning, teaching, and school management that are based 

on scientifically based research and effective practices and have been replicated successfully in 

schools with diverse characteristics. 

 Comprehensive design for effective school functioning, integrating instruction, assessment, 

classroom management, and professional development and aligning these functions into a schoolwide 

reform plan designed to enable all students to meet challenging state content and performance 

standards and address needs identified through a school needs assessment. 

 Professional development. High-quality and continuous teacher and staff professional development 

and training. 

 Measurable goals for student performance and benchmarks for meeting those goals. 

 Support from staff. Support from school faculty, administrators, and staff. 

 Support for staff. Support for school faculty, administrators, and staff. (Added in 2001) 

 Parent and community involvement. Meaningful involvement of parents and the local community 

in planning and implementing school improvement activities. 

 External assistance. High-quality external support and assistance from a comprehensive school 

reform entity (which may be a university) with experience in schoolwide reform and improvement. 

 Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved. 

 Coordination of resources. Identification of how other available resources (federal, state, local, or 

private) will help the school coordinate services to support and sustain the school reform.

 Scientifically based research. Scientifically based research to significantly improve the academic 

achievement of students participating in such programs as compared with students in schools who 

have not participated in such programs or strong evidence that such programs will significantly 

improve the academic achievement of participating children. (Added in 2001) 

Source: No Child Left Behind Act, Title I, Part F, Section 1606. 

Study Purpose 

The Longitudinal Assessment of Comprehensive School Reform Implementation and 

Outcomes (LACIO) responds to the NCLB Act’s requirement for an evaluation of the federal 

Comprehensive School Reform (CSR) program. The legislation stipulates two broad goals for 

the evaluation: first, to evaluate the implementation and outcomes achieved by schools after 
ix 

three years of implementing comprehensive school reforms and, second, to assess the 

effectiveness of comprehensive school reform in schools with diverse characteristics. In order to 

address these requirements, the study focused on four evaluation questions: 

1. How are CSR funds being targeted? 

2. How is comprehensive school reform implemented in schools receiving CSR funds, 

in schools receiving Title I funds and in other schools? 

3. What is the relationship between CSR implementation and student achievement 

outcomes? 

4. What conditions (at the state and district level) influence the implementation of 

comprehensive reform programs? 

This report presents data collected from a random sample of 400 CSR schools that received 

funding in 2002 and 400 non-CSR schools with similar demographic and achievement 

characteristics.1 It draws from three data sources—school-level surveys of principals and 

teachers, the National School-Level State Assessment Score Database and the National Center 

for Educational Statistics (NCES) Common Core of Data (CCD). The data were collected in 

spring 2003 at the end of the first year of CSR implementation (2002-03). The emphasis in this 

report is on the first two evaluation questions that focus on school reform activities and the 

targeting of CSR program funds. 

First-Year Findings 

The first year of the evaluation has yielded information with implications for federal 

policy. The implications relate to two key findings: 

• Although both CSR and non-CSR schools are engaged in reform, reform in 

CSR schools is more likely to include adoption of models and other activities 

closely associated with research-based models. 

• CSR funds are strongly targeted to high-poverty schools and low-performing 

schools, and schools receiving CSR funds are lower performing than are other 

schools with similar demographic characteristics at the time they receive 

awards. 

 

1 The sample of 400 represents 36 percent of the approximately 1,100 schools reported to receive CSR funds for the calendar 

year 2002. As a random sample, it does not mirror the universe on all characteristics. The distribution of the sample across locale 

and school level were comparable to the distributions of the universe, while reading and mathematics scores were slightly higher 

for the CSR sample. Similarly, the non-CSR schools, which were required to be in the same districts as the sample schools and 

with no current or past CSR funding, had slightly higher baseline achievement levels. These comparison schools represent the 

best available matches given these requirements. Further, analyses of achievement outcomes will control for variables such as 

achievement and poverty level, among others. 
x 

Implementation of School Reform Activities in CSR Schools and Other Schools 

Both CSR and non-CSR schools reported they were implementing specific activities that 

prior research indicates are associated with reform. However, as discussed below, the CSR 

schools differed from the non-CSR schools in their implementation of components directly 

related to selecting, implementing and evaluating models for reform. Further, CSR funding 

seems to contribute to building capacity for ongoing reform, with schools reporting more school 

activity that reflects coherence and cohesiveness during the first year of implementation (2002-

03) compared with the previous year. 

SCHOOL IMPROVEMENT PLANS

Nearly all schools in the sample (CSR and non-CSR schools) reported they had formal 

comprehensive plans for school reform. Principals at both CSR and non-CSR schools indicated 

these plans included components similar to the 11 CSR components, although CSR schools were 

more likely to report seeking research evidence about a proposed reform and adopting a reform 

created outside of the school. 

Teachers and school administrators were involved in selecting the reform model or 

approach being implemented, both at CSR schools and non-CSR schools. However, the 

school board and the district central office played a more significant role in selecting 

reform at non-CSR schools than at CSR schools, indicating more “top-down” requirements 

for changes in practice in non-CSR schools. One third of CSR schools reported that the district 

central office was one of several entities responsible for selecting the reform, compared with 57 

percent of non-CSR schools, which reported this method. Non-CSR schools reported that school 

board members were involved in the decision at a higher rate than did CSR schools (24 percent 

for non-CSR schools compared with 15 percent of CSR schools) (Exhibit E-2). Further, state or 

district mandates were more likely to contribute to the selection of reform at non-CSR schools 

(60 percent) than at CSR schools (31 percent). 
xi 

Exhibit E-2 

Entities Involved in Selecting a Reform Model or Approach at the School 

72%

66%

57%

34% 32%

15%

76%

63% 63%

57%

36%

24%

0%

20%

40%

60%

80%

100%

School

improvement

team

Teachers School

administrators

District central

office*

Parents School Board*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: School staff were involved in selecting a reform model or approach at both 

CSR and non-CSR schools. However, the district central office (57 percent) and school 

boards (24 percent) had a greater role at non-CSR schools than at CSR schools (34 

percent; 15 percent). 

A higher percentage of CSR principals reported their schools had a comprehensive 

written plan in the first year of CSR implementation, 2002-03, (93 percent) as compared 

with the previous year (75 percent), indicating some influence of CSR. In contrast, principals 

in non-CSR schools reported little change (89 percent had comprehensive written plans in 2002-

03, compared with 86 percent in 2001-02). In addition, CSR schools were significantly more 

likely to report engaging in whole school reform in 2002-03 (76 percent) than the prior year 

(55 percent). 

PROFESSIONAL DEVELOPMENT 

Professional development for all teachers was included in the school reform plan 

more frequently in CSR schools than in non-CSR schools. A greater number of CSR 

schools than non-CSR schools provided more than 10 days for professional development 

and received on-site assistance from external supporters. 

Ninety percent of CSR schools included professional development for all teachers in their 

school reform plan compared with 73 percent of non-CSR schools. CSR schools also provided 

more than 10 days for professional development more often than did non-CSR schools (56 
xii 

percent as compared with 39 percent). External assistance providers supported reform efforts onsite in significantly more CSR schools (86 percent) than non-CSR schools (57 percent). Finally, 

formal evaluation plans in CSR schools were more likely to include assessment of the utility of 

external assistance than such plans in non-CSR schools (41 percent vs. 30 percent) (Exhibit E-3). 

Exhibit E-3 

Status of Professional Development in School Reform 

*Difference is statistically significant at the .01 level. 

Exhibit reads: In CSR schools as compared with non-CSR schools, professional 

development more often was included in the school reform plan (90 percent vs. 73 

percent), offered for over 10 days (56 percent vs. 39 percent) and took the form of on-site 

assistance from external sources (86 percent vs. 57 percent). 

Teacher participation in grade-level or content area teams increased significantly in CSR 

schools in 2002-03, compared with 2001-02. In addition, CSR teachers reported receiving more 

days of professional development in 2002-03 than in the prior year, and the training was more 

focused on issues related to reform (Exhibit E-4). 

CSR schools Non-CSR 

schools 

Professional development for all teachers is 

included in school improvement plan 90% 73%* 

School provides 10 or more days for 

professional development 56% 39%* 

School receives on site support for reform 

efforts from external providers 86% 57%* 

School evaluation plan includes assessment of 

the utility of external assistance 41% 30%* 
xiii 

Exhibit E-4 

Types of Professional Development in Which CSR Teachers Participated During the 

First Year of Reform Implementation (2002-03) Compared with the Previous Year 

84%

46%

73%

63% 67%

54%

50%

61%

94%

85% 83% 83% 79%

71% 70% 69%

0%

20%

40%

60%

80%

100%

Reading/lang.

arts

Model

implementation

Mathematics Intrepreting

reports

Consistency 

with content

Instructional

strategies

Monitoring

students'

progress

Consistency with

assessments

2001-02 2002-03

Difference is statistically significant at the .01 level for all items. 

Exhibit reads: CSR teachers were much more likely to receive training on model 

implementation (85 percent), monitoring students’ progress (70 percent) and interpreting 

reports (83 percent) in 2002-03, compared with 2001-02 (the year prior to CSR 

implementation). 

SUPPORT FOR SCHOOL REFORM

Significant differences existed between CSR and non-CSR schools in the type of support 

they receive. As might be predicted, CSR schools were far more likely to receive support 

from a model developer than non-CSR schools (31 percent vs. 6 percent). However, non-CSR 

schools were more likely to report receiving support for school reform efforts from the 

district than were CSR schools (72 percent of non-CSR schools vs. 34 percent of CSR schools) 

(Exhibit E-5). 
xiv 

Exhibit E-5 

Entity Primarily Responsible for Supporting Reform Efforts at the School 

34% 31%

72%

6% 0%

20%

40%

60%

80%

100%

School district* Reform program developer*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: CSR schools were far more likely to identify model developers as the 

primary supporters of reform at their school (31 percent for CSR schools vs. 6 percent for 

non-CSR schools). Conversely, non-CSR schools (72 percent) reported more district 

support for school reform efforts than did CSR schools (34 percent). 

States and districts were more likely to provide funds for reform to non-CSR schools 

than to CSR schools. Discretionary district funds went to 58 percent of non-CSR schools 

compared with 44 percent of CSR schools. Special state grants were awarded to 53 percent of 

non-CSR schools compared with 44 percent of CSR schools (Exhibit E-6). Further, in 2002-03, 

districts supported different kinds of activities in CSR schools as compared with the previous 

year. Districts were more likely to help CSR schools select a school reform model in 2002-03 

(45 percent) than in 2001-02 (32 percent) but were less likely to provide CSR schools with 

professional development for school reform in 2002-03 (72 percent) than in 2001-02 (86 

percent). 
xv 

Exhibit E-6 

Sources of Funding that Contribute to Implementation 

and Operation of School Reform 

44% 44%

58%

53%

0%

20%

40%

60%

80%

100%

Discretionary district funds* Special state grants*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: Special state grants and district discretionary funds were more commonly 

used for reform at non-CSR schools (58 percent for district funds; 53 percent for state 

funds) than at CSR schools (44 percent for each source of funds). 

SUMMARY OF CSR IMPLEMENTATION 

CSR comprises 11 components whose interaction may improve schools. Respondents to 

the survey indicated that both CSR and non-CSR schools were implementing a number of the 

components. However, CSR schools were more likely than non-CSR schools to implement 

components most associated with adopting a model. Consequently, the presences of some similar 

components in CSR and non-CSR schools may not indicate equal progress toward reform nor 

lead to equal outcomes for students. The differences in the components that are implemented in 

CSR and non-CSR schools may well encompass different interactions, which, in turn affect the 

extent to which schools are coherent and cohesive, enabling them to provide students with 

focused and challenging opportunities to learn to high standards. 

CSR schools, as compared with non-CSR schools, were more likely to implement the 

following components: 

• Adopt externally developed strategies that have been replicated. They did so 

by: 
xvi 

- Identifying a specific reform model (85 percent compared with 49 

percent). 

- Using evidence from research that the reform model chosen improves 

student achievement (42 percent compared with 26 percent). 

• Provide more continuous professional development. They did so by: 

- Including professional development activities for all teachers (90 percent 

compared with 73 percent). 

- Allocating over 10 days to teacher professional development (56 percent 

compared with 39 percent). 

• Include measurable goals for student performance associated with the reform 

model (57 percent compared with 41 percent). 

• Reflect support from staff by including a formal vote by teachers for the 

reform model (82 percent compared with 55 percent). 

• Provide support for staff by receiving on-site consulting relevant to the reform 

(85 percent compared with 57 percent). 

• Evaluate the reform. They did so by: 

- Including the requirements of the reform model in the scope and content 

of evaluation (66 percent compared with 42 percent). 

- Assessing the utility of external assistance (41 percent compared with 30 

percent). 

In sum, both CSR and non-CSR schools exhibited many aspects of comprehensive reform. 

However, CSR schools were more likely to adopt externally developed models. Other 

differences between the two types of schools were related to model adoption. 

Targeting of CSR Funds 

The legislation intends for CSR funds to be targeted to low-performing schools that serve 

high-need students. 

CSR funds were strongly targeted to high-poverty schools and those with high 

concentrations of minority students. Almost half (45 percent) of CSR schools had poverty 

rates of at least 75 percent, nearly three times greater than the percentage of all schools in this 

highest-poverty group (16 percent) and close to double the percentage of Title I schools (26 

percent) (Exhibit E-7). Similarly, schools with high concentrations of minority students (75 
xvii 

percent or higher) accounted for nearly half (47 percent) of CSR schools, compared with 30 

percent of Title I schools and 21 percent of all schools. CSR schools were much more likely to 

be located in urban areas (46 percent of CSR schools) than were Title I schools (26 percent) or 

all schools (25 percent). Rural schools were equally represented among CSR schools, Title I 

schools and all schools (13 percent of each group). CSR schools were less likely than Title I and 

all schools to be located in suburbs and towns. 

The distribution of CSR schools by poverty and minority status was similar to the 

distribution of Title I schoolwide programs—not a surprising finding, because both 

programs are targeted to high-need schools. For example, the highest-poverty schools 

accounted for 45 percent of CSR schools and 42 percent of Title I schoolwides. However, CSR 

schools were more likely to be located in urban areas (46 percent) than were Title I schoolwides 

(37 percent). The proportion of CSR schools that were operating Title I schoolwide programs 

was 56 percent, compared with 25 percent of all schools operating Title I schoolwide programs. 

Exhibit E-7 

Distribution of CSR Schools and Other Schools by School Poverty Rate 

18%

34%

18% 14%

28%

29%

33% 37%

28%

21%

45% 42%

26%

16%

0% 4% 7%

20%

40%

60%

80%

100%

CSR Schools Title I Schoolwide

Programs

All Title I Schools All Schools

75-100%

50-74%

25-49%

0-24%

Exhibit reads: Almost half (45 percent) of CSR schools had poverty rates of at least 75 

percent, nearly three times greater than the percentage of all schools in this high poverty 

group (16 percent) and close to double the percentage of Title I schools (26 percent). The 

distribution of high poverty CSR schools was similar to the distribution of Title I 

schoolwide programs (42 percent). 

At the time of funding, CSR schools were significantly more likely to report that they 

were identified as a low-performing school according to the criteria used in their state (46 

percent) at the time of award than were the non-CSR schools (28 percent). When they received 

CSR funding they were also more likely to have received state sanctions due to low performance 
xviii 

(11 percent as compared with 3 percent). These are additional indicators that CSR funds are 

being targeted to schools in need of improvement. 

CSR schools had lower baseline achievement scores than did Title I schoolwides in 

reading and mathematics at most grade levels (elementary, middle, and high school) at the 

time the awards were made. For example, in elementary and middle grades, students in CSR 

schools scored an average of .4 standard deviations lower in reading and math achievement than 

students in Title I schoolwides. Further, the difference was true regardless of school locale or 

poverty level, a further indication of targeting.

Conclusion 

Taken together, the findings about CSR implementation and targeting raise interesting 

questions. One such question is whether the use of CSR funds accelerates reform in the lowest 

performing schools. States and districts seem to have targeted CSR funds to those schools that 

have the greatest need to change practices in order to support high achievement for all students. 

With CSR funds, the schools were more likely to adopt models, focus professional development, 

and track student performance than were non-CSR schools. Both CSR and non-CSR schools 

were engaged in other reform activities. In subsequent years, the evaluation will provide 

information about whether CSR schools implement more reform components more thoroughly 

than do non-CSR schools. If they do, CSR can be seen as adding value to improvement by 

providing a mechanism that focuses efforts and enables school staff to organize themselves in 

ways that offer greater educational opportunities for students. Perhaps CSR helps schools jumpstart improvement. 

Second, data from the first year of this evaluation indicate that all schools in the sample are 

engaged in many aspects of what the legislation defines as “comprehensive school reform.” 

Consequently, the study carries implications about the nature of reform in general. Most lowperforming schools in the non-CSR group are making efforts to improve. Questions then arise as 

to whether the efforts are associated with improved outcomes: Do schools succeed in reform 

without models to organize them? Are models only important in the lowest performing schools? 
1 

I. Introduction 

This report of the Longitudinal Assessment of Comprehensive School Reform 

Implementation and Outcomes (LACIO) responds to the No Child Left Behind (NCLB) Act’s 

requirement that an evaluation of the federal Comprehensive School Reform (CSR) program be 

completed. The legislation stipulates two broad goals for the evaluation: 

• To evaluate the implementation and results achieved by schools after three 

years of implementing comprehensive school reforms. 

• To assess the effectiveness of comprehensive school reform in schools with 

diverse characteristics. 

The federal CSR program provides funds to states, which, in turn make grants to schools to 

support comprehensive reform. The intention is that the vast majority of these schools will be 

Title I schools “in need of substantially improving” their student achievement levels. Further, the 

CSR program delineates 11 components of “comprehensive school reform,” which are supported 

by research and evaluation. 

This first-year report begins with an overview of the context for the CSR program, a 

description of its history, and a description of the 11 components of comprehensive school 

reform included in NCLB. It then addresses the goals of NCLB by presenting preliminary 

findings related to implementation of CSR and an examination of the types of schools receiving 

CSR program funding. The report includes information drawn from surveys sent to 400 CSR 

program schools (“CSR schools”) and 400 matched non-CSR program schools (“non-CSR 

schools”) that examine the presence and characteristics of the 11 components of CSR, as well as 

other elements (e.g., school organization) that prior research has shown to be associated with 

successful program implementation. 

Background 

Comprehensive school reform was a response to the persistent failure of some schools to 

provide students with educational opportunities to meet high standards. As state and federal 

governments have increased emphasis on academic standards, they have also increasingly held 

schools accountable for ensuring that students meet those standards. However, they have also 

provided guidance and funds to help schools change so students can successfully achieve high 

standards. CSR constitutes one mechanism for providing such assistance. 

This section provides background on CSR, including the relationship of the program to 

ongoing efforts to hold schools accountable for results and the history of CSR as a program

funded through the U.S. Department of Education (ED). It then moves to a discussion of the 

purpose of this evaluation, including how the study addresses emerging issues in CSR and the 

evaluation questions from NCLB. The section concludes with an overview of the report. 
2 

Assessment, Accountability, and Schoolwide Reform 

Since 1965, the federal government has authorized formula grants to states and local 

education agencies (LEAs) for the education of elementary and secondary students with low 

academic achievement who are enrolled in schools serving low-income areas. These grants, 

known as Title I, were designed to accomplish four primary goals: 

• Provide supplemental education to students eligible for services. 

• Provide additional funding to schools and LEAs serving high concentrations 

of children from low-income families. 

• Focus educators on the needs of special student populations. 

• Improve the academic achievement of eligible students, reduce performance 

gaps between advantaged and disadvantaged students, and assist eligible 

students in meeting high academic standards. 

In 1994, Congress changed the focus of Title I programs in the Improving America’s 

Schools Act (IASA). This act, which reauthorized the Elementary and Secondary Education Act 

(ESEA), included Title I provisions calling for schools that receive Title I funds to set high 

standards for all students, to assess all students relative to these standards, to report results to the 

public, and to make instructional and structural changes to ensure that all students have the 

opportunity to meet these standards (Quenemoen et al. 2001). This movement, part of standardsbased reform, marked a shift away from providing disadvantaged students with basic skills and 

toward more advanced content and performance standards for all students. 

The authorization signaled a new focus on schoolwide reform for Title I schools serving 

high concentrations of low-performing, high-poverty students (U.S. Department of Education 

2001). This focus came as evaluations suggested that targeted, “pull-out” education programs for 

students, the previous use of Title I funds, showed no clear positive effect on student 

achievement in high poverty schools. In fact, studies showed that pulling students out of their 

regular classes for special programs disrupted the classroom, stigmatized the students, reduced 

time spent in the regular class with their peers, and yielded uneven instruction (U.S. Department 

of Education 1997). In contrast, studies supported the notion that schoolwide reform would 

benefit even the most low achieving students by raising standards, implementing a challenging 

curriculum, and assessing learning (U.S. Department of Education 1996a; U.S. Department of 

Education 1996b; U.S Department of Education 1993). 

As a result, schools with high concentrations of high-poverty students were allowed to pool 

resources and encouraged to use these resources to leverage additional funds, as well as to 

integrate programs related to curriculum, parent involvement, professional development, and 

drug prevention (U.S. Department of Education 1998). The U.S. Department of Education 

identified several characteristics of schoolwide programs including: 
3 

• A comprehensive approach that integrates the whole school (students, faculty, 

parents, and the community), uses data to assess students’ needs, and then ties 

instructional and assessment practices in all curricular areas to this 

understanding. 

• A focus on examining and reforming the curriculum in multiple subject areas, 

not simply one or a few. 

• Collaboration between the school and district to implement reform, where the 

school receives autonomy in areas such as management, budget, and program 

development, while also getting district support and funding. 

• Strong leadership from the principal to shape a common vision. 

• Qualified professionals who receive professional development, small classes, 

and the right materials and equipment to facilitate excellent teaching. 

• An environment where everyone believes in the ability of students to achieve 

high standards—with no exceptions. 

• Accountability measures that monitor student progress, use data to 

continuously improve teaching and learning, and provide the necessary 

support for success (U.S. Department of Education 1998). 

Of course, change or reform is a complicated, demanding process. For change to take hold, 

consistent leadership is needed, as is support from the district (Finnan 2000; U.S. Department of 

Education 2000a; Stringfield et al. 1997). School reform is a political process requiring buy-in 

from a broad range of constituencies, including teachers, parents, and the larger community; 

developing this buy-in takes time. These challenges to reform are further complicated by 

changing expectations for schools, the growth of new programs, and the dismantling of old 

ones—all of which take staff time and can create a culture of cynicism about change (Sarason 

1996). 

NCLB raises the stakes for schools, particularly low-performing schools. As with earlier 

Title I authorizations, schools are required to make “adequate yearly progress” (AYP) toward 

state achievement standards. In addition, NCLB tightens requirements about how such progress 

is shown by mandating annual testing and reporting outcomes of key subgroups of students. 

Further, Title I schools “in need of substantially improving” student achievement levels can be 

subject to sanctions. Such schools also must make supplemental services available to students 

and can, if the need for improvement persists, be reconstituted. Students can also receive 

opportunities to attend different schools. 

The stakes, therefore, are high. Schools must focus on student achievement and change 

curriculum, instruction, organization, and other elements to meet students' needs better. CSR is a 

source of both ideas and funds to bring about school reform designed to facilitate schools’ 

abilities to meet accountability requirements. 
4 

The Comprehensive School Reform (CSR) Program 

In 1998, Congress appropriated $145 million for the Comprehensive School Reform 

Demonstration program (CSRD). It was designed as a way to encourage schools to engage in a 

comprehensive effort that would better meet student learning needs (U.S. Department of 

Education 2000a). CSRD was not to be an add-on placed on top of already existing programs 

and efforts but a way to encourage schools to integrate local, state, and federal resources to bring 

about improved student learning (U.S. Department of Education 1999). 

Like schoolwide Title I programs, CSRD was intended to help schools leverage funds from 

both public and private sources and integrate programs while giving them flexibility and 

enhancing accountability for student learning. Its unique aspect, relative to other Title I programs 

and the IASA legislation, was the expectation that schools collaborate with expert partners to 

implement whole-school reform programs that had a strong research base and a successful 

replication record (Hale 2000). 

As its cornerstone, CSRD had nine criteria that the reform programs used by funded 

schools had to meet (U.S. Department of Education 1999; U.S. Congress 105th Session). The 

legislation offered 17 programs as examples of the models schools might choose to employ but 

enabled schools to choose other models, combine models, or create their own reform programs, 

provided they met the nine criteria. The models include some developed for a school’s entire 

curriculum, some models focused on specific content areas, and process models that guide 

schools through the development of their own vision and corresponding materials and practices. 

In addition, schools had the option of crafting their own models (Hale 2000). A quick review of 

the models adopted by schools in the first cycle of CSRD funding indicates that subject-specific 

(mainly reading) models were among the "top 30" models adopted by schools 

(http://www.sedl.org/csrd/awards.html). Further, schools implemented different configurations 

of the nine components, with some schools focusing on fewer than all nine. 

The CSRD appropriation spurred a dramatic growth in school reform. Even prior to CSRD, 

more than 2,100 schools were affiliated with one of three schoolwide reform programs (Success 

for All, School Development Project, or Accelerated Schools) (Consortium for Policy Research 

in Education 1998). The CSRD initiative was expected to more than double the number of 

schools embarking on such reform efforts (Consortium for Policy Research in Education 1998). 

As of September 2000, 1,800 schools had received CSRD funds (U.S. Department of Education 

2000b). About 2,000 schools were funded as a result of the next round of applications for CSRD 

funds. Several states have adopted initiatives similar to CSRD. States such as Colorado, Hawaii, 

Wisconsin, and North Carolina have used the CSRD model to restructure their efforts at reform, 

providing similar grants to districts in their states; other states (Oregon, Tennessee, and West 

Virginia) used the CSRD model to guide how they distribute Title I and state school 

improvement funds (U.S. Department of Education 2000b). 

With the passage of the No Child Left Behind (NCLB) Act in 2001, CSR became a fully 

authorized program and is no longer considered a demonstration program. Further, NCLB 

described 11 components of comprehensive school reform (Exhibit 1) and did not include a list 

of models. 
5 

Exhibit 1 

Eleven Components of Comprehensive School Reform 

Described in the No Child Left Behind Act

 Proven methods and strategies for student learning, teaching, and school management that are based 

on scientifically based research and effective practices, and have been replicated successfully in 

schools with diverse characteristics. 

 Comprehensive design for effective school functioning, integrating instruction, assessment, 

classroom management, and professional development and aligning these functions into a schoolwide 

reform plan designed to enable all students to meet challenging state content and performance 

standards and address needs identified through a school needs assessment. 

 Professional development. High-quality and continuous teacher and staff professional development 

and training. 

 Measurable goals for student performance and benchmarks for meeting those goals. 

 Support from staff. Support from school faculty, administrators, and staff. 

 Support for staff. Support for school faculty, administrators, and staff. (Added in 2001) 

 Parent and community involvement. Meaningful involvement of parents and the local community 

in planning and implementing school improvement activities. 

 External assistance. High-quality external support and assistance from a comprehensive school 

reform entity (which may be a university) with experience in schoolwide reform and improvement. 

 Evaluation. Plan to evaluate the implementation of school reforms and the student results achieved. 

 Coordination of resources. Identification of how other available resources (federal, state, local, or 

private) will help the school coordinate services to support and sustain the school reform.

 Scientifically based research. Scientifically based research to significantly improve the academic 

achievement of students participating in such programs as compared with students in schools who 

have not participated in such programs; or strong evidence that such programs will significantly 

improve the academic achievement of participating children. (Added in 2001) 

Source: No Child Left Behind Act, Title I, Part F, Section 1606. 

The schools included in this evaluation received funding starting in 2002. States applied the 

NCLB definition of comprehensive reform to the schools they funded as described in the 

Department's guidance for the program, despite the fact that some funding came from earlier 

appropriations. The 11 components, then, frame the study. The underlying questions related to 

implementation are: 
6 

• To what extent do schools receiving CSR funding implement the 11 

components? 

• How is implementation of reform in CSR schools different in schools, 

particularly Title I schoolwides, that do not receive program funding? 

This report focuses on the implementation question in the first year of funding. Later 

reports will assess progress on implementing the components and will relate implementation to 

outcomes in both CSR and non-CSR schools. 

Study Purpose 

The CSR program has evolved along with the changing context for education. It is one 

approach to improving opportunities for students in low-performing schools. NCLB includes 

other approaches, some of which focus on students (e.g., access to supplemental services; the 

option to transfer students to higher performing schools) and some of which focus on improving 

the schools (e.g., schoolwide Title I; CSR; professional development). Further, federal, state, and 

district policies create increased pressure on low-performing schools because accountability for 

results has increased. Such pressure could potentially increase “CSR-like” activities in non-CSR 

schools. As a result, the study, while focused on CSR, has implications for how all schools in 

need of improvement may serve their students better. This section focuses first on the potential 

implications of the evaluation and then moves to the questions that guided the evaluation, 

placing them within the broader context as well. 

Emerging Issues in CSR 

CSR as a program facilitates access to models with scientifically based evidence of their 

effectiveness. It also articulates a set of principles (in the form of components) that are designed 

to reform low-performing schools so students in such schools can meet high standards. The 

current study is evaluating the extent to which CSR achieves its objectives. As such, the findings 

of the study will have broad implications for both policy and practice. This section points to a 

few such implications and places this evaluation within the framework of earlier CSR-related 

research and evaluation related. 

This study represents an important change in evaluating CSR. Whereas earlier evaluations 

focused strongly on models, this evaluation focuses on comprehensive school reform as 

manifested in the 11 components and interactions among them. Examples of work focused on 

models abound. Borman, Hewes, Overman, and Brown used meta-analysis to review “research 

on the achievement effects of comprehensive school reform (CSR) and summarizes the specific 

effects of 29 widely implemented models” (Borman et al. 2003). Similarly, Desimone addressed 

the question “Can comprehensive school reform models be successfully implemented?” 

(Desimone 2002). Both articles focus attention on the use of “proven strategies and proven 

methods for student learning, teaching and school management” that have “been found to have 

strong evidence that such programs will significantly improve the academic achievement of 

participating children” (components 1 and 11). They also include some discussion of a 

“comprehensive design for effective school functioning,” “high quality and continuous teacher 
7 

and staff professional development,” and support by “teachers, principals and administrators” 

(components 2, 3, and 5). 

As NCLB makes clear, the use of scientifically based research is not confined to adopting 

models. The principle of basing practice on scientific research runs throughout the law. 

However, CSR focuses on principles beyond a single subject or service to students. CSR 

presumes that low-performing schools, as institutions, should (and can) change to serve students 

more effectively. The institutional focus of CSR differentiates it from, for example, programs 

that focus on improving reading instruction or any other curriculum area. Further, CSR assumes 

that comprehensiveness itself is a spur to reform. Although NCLB includes 11 components of 

comprehensiveness, their nature is such that interactions among them (and with the context in 

which the school exists) are expected to lead to greater impact than each of them alone or even a 

subset of them. In short, the CSR components are not a checklist for comprehensiveness, but 

rather are indicators of coherence and cohesiveness of school structures and processes 

(Newmann et al. 2001). This evaluation provides a vehicle for addressing such views 

empirically. 

As a result of the evaluation, ED will have information about the extent to which CSR 

helps low-performing schools change practices and improve outcomes with appropriate support, 

including support in implementing research-based practices and organizational structures. 

Evaluation Questions 

In NCLB, Congress required a national evaluation to: (1) evaluate the implementation and 

results achieved by schools after three years of implementing comprehensive school reforms; and 

(2) assess the effectiveness of comprehensive school reforms in schools with diverse 

characteristics (No Child Left Behind Act of 2001, P.L., 107-110). In order to address these 

requirements, this study is focused on four broad questions: 

1. How are CSR funds being targeted? 

- Are states targeting CSR funds to low-performing and under-performing 

schools in both urban and rural areas? Does the funding reach students at 

all grade levels? 

2. How is comprehensive school reform implemented in schools receiving CSR funds, 

in schools receiving Title I funds, and in other schools? 

- How well have schools implemented the eleven components of 

comprehensive school reform identified in the NCLB Act of 2001? 

- What types of school reform models or strategies are schools 

implementing? 

- Do CSR resources help schools build their capacity for reform? How do 

schools sustain the reform process, given natural attrition in staff, changes 

in district priorities, and shifts in funding? 
8 

3. What is the relationship between CSR implementation and student achievement 

outcomes? 

- To what extent have CSR schools made progress on state assessments, in 

comparison to other schools in their state with similar characteristics? 

- Were schools previously identified as in need of improvement able to 

make sufficient progress to move out of “school improvement” status? 

- How do achievement trends vary across different types of CSR schools 

(e.g., high poverty, high minority)? 

4. What conditions (at the state and district level) influence the implementation of 

comprehensive reform programs? 

- To what extent have state and district policies supported the 

implementation of comprehensive school reform? Have state and district 

policies or support helped develop the capacity to begin the reform 

process? How have states, districts, and schools planned to sustain 

reforms after the federal funding ends?

Focus and Organization of the Report 

In this report, the primary emphasis is on the first two evaluation questions that focus on 

school reform activities and the targeting of CSR program funds. In the report, this chapter is 

followed by a description of the evaluation design, including data collection and analysis 

approaches. The third chapter presents the findings from the study, including: 

• A look at which schools received CSR program funds in 2002, including 

comparisons of baseline achievement levels of CSR schools and non-CSR 

schools prior to implementation. 

• The status of various aspects of reform in the CSR and non-CSR schools. 

• The school-based practitioners' perceptions of state and local policy influence 

in implementing school reform. 

The findings section concludes with a discussion of the nature of comprehensive reform as 

it currently exists in the sample, including how components relate to one another and the value of 

CSR funds in advancing reform and helping schools build their capacity for reform. The final 

chapter summarizes the findings and points to policy implications. 
9 

II. Evaluation Design 

The five-year evaluation will employ a quasi-experimental design to analyze student data 

and school reform at multiple points in time, comparing CSR program and non-CSR program 

schools. The evaluation will include multiple methods of data collection and analysis in order to 

increase the robustness of the study. The study will include three approaches, each contributing 

to answering the questions posed by the legislation. The approaches comprise a complementary 

set of inquiries, going beyond the capability of any single method. The approaches are: 

• An analysis of student achievement in all CSR schools that received funding 

in 2002, compared with student achievement in a sample of similar schools 

that have not received CSR funding as well as with a sample of Title I 

schoolwides. 

• An analysis of survey and interview data from a large sample of schools, a 

smaller sample of districts and each state. 

• A field-based inquiry of a small sample of CSR program and nonprogram 

schools in the district and state context. 

The section below briefly outlines the data collection and analysis plan for the five-year 

evaluation, as well as the evaluation activities used in preparing this first-year report. 

Study Data Collection and Analysis 

Analysis of Student Achievement 

For all schools in the evaluation, a quantitative analysis of changes in student achievement

will be conducted. The analyses will cover the universe of CSR schools funded in 2002 

compared with a sample of 400 non-CSR schools selected to match the CSR schools in the 

sample for the surveys (see below). In addition, student achievement will be compared with a 

sample of Title I schoolwides. 

Survey of School Reform Activities 

The study includes a survey of a sample of 800 schools (400 CSR schools and 400 

comparable schools) that will complete survey forms asking them to describe the various reform 

activities occurring at the schools. The CSR schools in the evaluation represent a random sample 

of all schools receiving CSR funds in 2002. 

The survey instrument will be distinguished from a typical survey form in that it will 

include items that are evidence-based and measure behaviors, rather than measuring attitudes and 

expectations (see Appendix B for a complete discussion of the use of the survey forms). The 

survey will inquire into the extent to which the 11 elements of comprehensive school reform 
10 

included in NCLB are present and other elements (e.g., school organization) that prior research 

shows to be associated with successful program implementation. 

In addition to the mailed surveys, the evaluation will collect data about CSR activities and 

state and district policies from a smaller sample of districts and each state. These telephone 

interviews will provide a more complete picture of CSR implementation, as well as data needed 

to analyze the effects of state and district policies on CSR implementation and outcomes. Such 

effects will be addressed using multilevel modeling techniques (Bryk and Raudenbush 1992). 

Subsequent reports will include those analyses. 

Field-Based Study of Reform Activities 

Out of the larger sample of 800 schools, a subsample of 15 pairs of schools (half of each 

pair is CSR program school and half, non-CSR) have been selected to participate in the 

Field-Based Study of Reform Activities at CSR Program and non-CSR Program Schools. 

The field-based study will include two visits to each site, occurring during the second and 

third years of a school’s CSR award. Each “site” consists of four entities: 

1. A CSR-funded school. 

2. A demographically matched non-CSR school (a school that has not received any 

federal CSR funds) located in the same district as the CSR-funded school. 

3. The district within which the two schools are located. 

4. The state within which the district is located. 

By covering these four entities, the field-based component will address all four evaluation 

questions and also produce an understanding of the dynamic of the actual relationships among 

school, district, and state actions, policies, and practices. 

During the site visits in the field-based component, the evaluation team will complete 

organizational inventories covering relevant events at the “site.” Data for the inventories will 

come from classroom observations, using a formal observation instrument, as well as other direct 

field observations, reviews of relevant school documents and materials, and discussions with 

school staff and parents. The field-based study is underway and will be concluded by early 

spring 2005. Data from the study will be included in later reports. 

Methods Used in the First-Year Report 

The First-Year Report presents data collected from a random sample of 400 CSR schools 

that received funding in 2002 and 400 non-CSR schools with similar demographic and 
11 

achievement characteristics.2 It draws from three data sources—the mailed school-level surveys 

of principals and teachers, the National State Assessment Score Database and the National 

Center for Educational Statistics (NCES) Common Core of Data (CCD). 

Mailed School-level Survey 

Survey data were collected from a random sample of CSR schools that received funding in 

2002 and the comparison schools in spring 2003 at the end of the first year of implementation. 

Of the schools selected for the survey, 367 CSR and 356 non-CSR schools agreed to participate 

in the evaluation, with 350 of them representing matched pairs. In addition to schools that 

refused to participate, some late in the school year, the sample included five schools that had 

closed. They were replaced with a second random sample and mailed surveys in fall 2003 

(Appendix B). Five additional comparison schools were eliminated from the sample because 

they were later found to be CSR schools. These comparisons will not be replaced. 

The analyses presented in this year-one report are based on responses from 89 percent of 

participating target schools and 89 percent of participating non-CSR schools. Matching 

individual CSR and non-CSR schools is not crucial for the descriptive analyses in this report. It 

will become important when implementation and outcomes are related in subsequent years. 

Currently, the study data includes matched responses from 88 percent of schools. 

National State Assessment Score Database 

The National State Assessment Score Database, maintained by the American Institutes for 

Research (AIR), includes student achievement data from all states. The data will be used to 

compare outcomes in CSR and similar non-CSR schools. In the first-year report, the data were 

used to establish the baseline for later comparisons. 

Common Core of Data 

The CCD, which provides information about demographic and other school characteristics, 

served two roles in the first-year report. First, it was an important source of data for matching 

CSR and non-CSR schools. Second, CCD provided the data to analyze the extent to which CSR 

funds were targeted as intended by NCLB. 

 

2 The sample of 400 represents 36 percent of the approximately 1,100 schools reported to receive CSR funds for the calendar 

year 2002. As a random sample, it does not mirror the universe on all characteristics. The distribution of the sample across locale 

and school level were comparable to the distributions of the universe, while reading and mathematics scores were slightly higher 

for the CSR sample. Similarly, the non-CSR schools, which were required to be in the same districts as the sample schools and 

with no current or past CSR funding, had slightly higher baseline achievement levels. These comparison schools represent the 

best available matches given these requirements. Further, analyses of achievement outcomes will control for variables such as 

achievement and poverty level, among others. 

13 

III. First-Year Findings 

Comprehensive school reform is both a concept and a program. As a concept, it implies 

systematic, whole school, coherent reform in how a school is organized, delivers instruction, 

uses curriculum, provides professional development, and integrates resources. As a program, 

comprehensive school reform is defined by 11 components, which reflect research and practice 

in school reform. This evaluation is designed to provide information about both the concept of 

comprehensive school reform and the program, and the analyses will relate outcomes to school 

reform, not to program funding. Consequently, the data collection instruments included questions 

about the extent to which CSR and non-CSR schools were engaged in reform, whether the 

characteristics of reform differed in CSR and non-CSR schools, and the outcomes of reform 

activities in both types of schools. To that end, and because CSR is frequently associated with 

implementation of “models,” the surveys avoided the use of the terms “comprehensive school 

reform” and “models.” 

Overall, both CSR and non-CSR schools were engaged in activities associated with most of 

the 11 components, probably because both were responding to similar state and federal 

accountability requirements. However, implementation of reform at the CSR schools differed 

from implementation in the non-CSR schools on the components that were most closely 

associated with externally developed models. 

This section contains the findings from the first year of the evaluation. The findings are 

drawn from the National School-Level State Assessment Score Database, NCES Common Core 

of Data, and the school-level surveys developed for the evaluation. Only selected exhibits are in 

this section; a complete set of tabulations appears in Appendix A. 

Targeting of CSR Funds 

Funding for schools in the 2002 cohort varied. CSR schools received an average of 

$100,565 for CSR activities per year, which represents an increase of funding from $66,175 in 

1998. However, the amount varied by type of school, with high schools receiving more money 

than elementary and middle schools. Although high schools received more money, the allocation 

per student was smaller at $122 than in middle schools ($162 per student) or elementary schools 

($189 per student). Further, schools in urban districts received more funding than schools in 

suburban districts and small towns, but rural schools received more money than small town 

schools. Funding also varied by state, with California granting $387,403 as an average award and 

Maine granting $50,000. 

The CSR legislation intentionally targets CSR funds to low-performing schools that serve 

high-need students. 

CSR funds were strongly targeted to high-poverty schools and those with high 

concentrations of minority students, as well as to urban schools. Almost half (45 percent) of 

CSR schools had poverty rates of at least 75 percent, nearly three times greater than the 

percentage of all schools in this highest-poverty group (16 percent) and close to double the 
14 

percentage of Title I schools (26 percent) (Exhibit 2). Similarly, schools with high concentrations 

of minority students (75 percent or higher) accounted for nearly half (47 percent) of CSR 

schools, compared with 30 percent of Title I schools and 21 percent of all schools (Exhibit 3). 

CSR schools were much more likely to be located in urban areas (46 percent of CSR schools) 

than were Title I schools (26 percent) or all schools (25 percent). Rural schools were equally 

represented among CSR schools, Title I schools, and all schools (13 percent of each group). CSR 

schools were less likely to be located in suburbs and towns than in rural or urban communities 

(Exhibit 4). 

The distribution of CSR schools by poverty and minority status was similar to the 

distribution of Title I schoolwide programs—not a surprising finding, because both 

programs are targeted to high-need schools. For example, the highest-poverty schools 

accounted for 45 percent of CSR schools and 42 percent of Title I schoolwides. However, CSR 

schools were more likely to be located in urban areas (46 percent) than were Title I schoolwides 

(37 percent). The proportion of CSR schools that were operating Title I schoolwide programs 

was 56 percent, compared with 25 percent of all schools operating Title I schoolwide programs. 

Exhibit 2 

Distribution of CSR Schools and Other Schools by School Poverty Rate 

18%

34%

18% 14%

28%

29%

33% 37%

28%

21%

45% 42%

26%

16%

0% 4% 7%

20%

40%

60%

80%

100%

CSR Schools Title I Schoolwide

Programs

All Title I Schools All Schools

75-100%

50-74%

25-49%

0-24%

Exhibit reads: Almost half (45 percent) of CSR schools had poverty rates of at least 75 

percent, nearly three times greater than the percentage of all schools in this high poverty 

group (16 percent) and close to double the percentage of Title I schools (26 percent). The 

distribution of CSR schools by poverty status was similar to the distribution of Title I 

schoolwide programs (42 percent). 
15 

Exhibit 3 

Distribution of CSR Schools and Other Schools by Percentage of Minority Students 

21% 24%

44%

14% 51%

14%

14%

17%

18% 15%

12%

11% 47% 47%

30%

21%

0%

20%

40%

60%

80%

100%

CSR Schools Title I Schoolwide

Programs

All Title I Schools All Schools

75-100%

50-74%

25-49%

0-24%

Exhibit reads: Schools with high concentrations of minority students (75 percent or 

higher) accounted for nearly half of CSR schools and Title I schoolwide programs (47 

percent each), compared with 30 percent of Title I schools and 21 percent of all schools. 
16 

Exhibit 4 

Distribution of CSR Schools and Other Schools by Urbanicity 

19% 26% 33% 30%

13%

13%

13% 13%

22%

24%

28% 32%

46%

37%

26% 25%

0%

20%

40%

60%

80%

100%

CSR Schools Title I Schoolwide

Programs

All Title I Schools All Schools

Urban

Suburban

Rural

Town

Exhibit reads: CSR schools were much more likely to be located in urban areas (46 

percent of CSR schools) than were Title I schoolwides (37 percent), Title I schools (26 

percent), or all schools (25 percent). Rural schools were equally represented among CSR 

schools, Title I schools, and all schools (13 percent of each group). 

Perhaps more important, school survey data show that CSR schools were significantly 

more likely to report that they were identified as a low-performing school according to the 

criteria used in their state at the time of award (46 percent) than were non-CSR schools (28 

percent). When they received funding CSR schools were also more likely to have received state 

sanctions due to low performance (11 percent as compared with 3 percent). These are additional 

indicators that CSR funds are being targeted to schools in need of improvement (Exhibit 5). 
17 

Exhibit 5 

Schools Reported Being Identified as Low-Performing or Sanctioned Because of 

Low Performance According to State Criteria 

46%

11%

28%

3%

0%

20%

40%

60%

80%

100%

Low-performing* Sanctions due to low performance*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: CSR schools were significantly more likely to report that they were 

identified as a low-performing school and had received sanctions due to low 

performance. 

Perhaps the most important characteristic of CSR schools is the level of student 

achievement prior to receiving funds. Because student achievement is measured and reported 

differently across states, available scores cannot be directly compared. In order to summarize 

data across states in the simplest manner, scores within each state were first transformed into zscores. Z-scores are centered on state means and are scaled in units of standard deviations. Thus, 

the average z-score for all schools, across all states is, by definition, zero. In this case, a negative 

average z-score indicates that the performance of a group is below state averages, across all 

states. Reports in subsequent years will use meta-analytic techniques to aggregate data across 

states. (For more information about this year's calculation of student achievement z-scores, see 

Appendix B.) 

To compare baseline student achievement scores across funding groups, average schoollevel z-scores were calculated in reading and mathematics for each school level. CSR schools 

had lower baseline achievement scores than did Title I schoolwides in reading and 

mathematics at most grade levels (elementary, middle and high school in math; elementary and 

middle in reading) at the time awards were made (Exhibit 6). Further, the difference was true 

regardless of school locale or poverty level (Exhibit A-1). 
18 

Exhibit 6 

Average School-Level Z-Scores for Math and Reading 

at CSR Schools (2002 Cohort), Title I Schools, and All Schools 

Exhibit reads: CSR schools had lower baseline achievement scores in math and reading 

than did Title I schoolwides, Title I schools, or all schools. (A negative average z-score 

indicates that the performance of a group is below state averages). 

The differences between 2002 CSR schools and the population of Title I schoolwides 

points to an important policy issue. CSR schools’ lower initial scores may reflect additional 

targeting to schools with the greatest need for improvement. Interviews with state and district 

officials will explore the rationale for allocating funds. 

CSR was designed to provide support for school reform in low-performing, high-need 

schools. Funding is provided through states, which then select schools within districts for 

funding. Clearly, states are selecting schools that meet the intent of the legislation. 

Implementation of School Reform Activities in CSR Schools and Other Schools 

Both CSR and non-CSR schools reported they were implementing specific activities that 

prior research indicates are associated with reform. However, as discussed below, the CSR 

schools differed from the non-CSR schools in their implementation of components directly 

related to selecting, implementing, and evaluating models for reform. Further, CSR funding 

seems to contribute to building capacity for ongoing reform, with schools reporting more school 

activity that reflects coherence and cohesiveness during the first year of implementation (2002-

03) compared with the previous year. 

This section begins with data related to the extent to which the 11 components of CSR 

were present in the CSR and comparison schools. It concludes with a discussion of the nature of 

comprehensive school reform. Because this is the first year of data collection and analysis, the 

final section is fairly speculative, pointing to areas for greater concentration in future years. (All 

Math 

CSR 

Schools 

Title I 

Schoolwides

All Title I 

Schools All Schools 

Elementary -0.95 -0.50 -0.24 0.0 

Middle -0.86 -0.46 -0.19 0.0 

High -0.68 -0.66 -0.24 0.0 

 

Reading 

Elementary -0.92 -0.54 -0.26 0.0 

Middle -0.89 -0.49 -0.21 0.0 

High -0.69 -0.72 -0.25 0.0 
19 

data in this section are drawn from the mailed surveys, and significance is reported at the p< .01 

level.). Because of the large association between the teacher and principal responses, only the 

principal responses are reported here as the measure of school reform. However, on a small 

number of items minor disagreements between principals and teachers existed. On those items, 

the responses for both teachers and principals are noted. These differences will be explored in 

subsequent reports. 

School Improvement Plans 

Nearly all schools in the sample (CSR and non-CSR schools) reported they had formal 

comprehensive plans for school reform (93 percent for CSR schools; 89 percent for non-CSR 

schools) (Exhibit A-2). Principals at both CSR and non-CSR schools indicated that these plans 

included components similar to the 11 CSR components, although CSR schools were more likely 

than non-CSR schools to report seeking research evidence about a proposed reform and adopting 

a reform created outside of the school. 

According to respondents, nearly all improvement plans for both CSR and non-CSR 

schools included measurable goals and objectives (98 percent in both groups) and professional 

development activities (94 percent in both groups). Both were highly likely to include curriculum 

and instruction (90 percent and 89 percent), a mechanism for periodic evaluation of goals (88 

percent and 84 percent), and a plan for parental involvement (83 percent and 81 percent). Both 

were less likely to include classroom management guidelines (48 percent and 39 percent) or 

student assessment rubrics in the plan (52 percent and 49 percent) (Exhibit 7). It may be that 

state and federal regulations for underperforming schools influence all schools. As data are 

gathered from interviews with state and district policymakers, the study will explore that 

possibility. 
20 

Exhibit 7 

Aspects of Reform Covered by School Improvement Plans 

CSR Schools 

Non-CSR 

Schools 

Measurable goals and objectives 98% 98% 

Professional development activities 94% 94% 

Curriculum and instruction 90% 89% 

Periodic evaluation of goals 88% 84% 

Parental involvement 83% 81% 

Student assessment rubrics 52% 49% 

Management guidelines 48% 39% 

Exhibit reads: Nearly all school improvement plans included measurable goals and 

objectives (98 percent for CSR schools and non-CSR schools), professional development 

activities (94 percent for both groups) and plans for curriculum and instruction (90 

percent and 89 percent). 

Few differences were found in the factors that influenced the content of the school 

improvement plan. Both CSR and non-CSR schools cited state and district content standards as 

the major influence on the plan (90 percent compared with 91 percent), and, although non-CSR 

schools selected “state or district performance standards” more frequently (91 percent) than did 

the CSR schools (87 percent), the difference was not significant. However, CSR schools were 

significantly more likely than non-CSR schools to cite the specifications of a reform design 

as influencing their school improvement plan (63 percent as compared with 34 percent) 

(Exhibit A-3). This indicates that reform models are more widespread in CSR schools than in 

non-CSR schools. 

Characteristics of Reform 

Although CSR schools and non-CSR schools were equally likely to state the reform was 

“adapted with modifications from an external source” (42 percent and 41 percent) (Exhibit A-4), 

they indicated different primary designers. Recipients of CSR funds reported they were more 

likely to adopt a reform that was created outside the school than were non-CSR schools (32 

percent compared with 6 percent) (Exhibit A-5). Further, CSR schools were influenced by design 

specifications of the reform significantly more than were non-CSR schools (63 percent and 34 

percent, as mentioned above and cited earlier in Exhibit A-3). CSR schools also were more 

likely than non-CSR schools to identify a specific reform model (85 percent vs. 49 percent) 

and indicate they were implementing only one reform (67 percent vs. 57 percent). 

However, both groups were equally likely to claim participation in reform by all grades and 

the “whole school.” Further, if they were not involved in whole-school reform, both CSR and 
21 

non-CSR schools focused primarily on reading and language arts, followed by mathematics. 

Relatively few respondents from either group indicated that the reform involved science, social 

studies, or the arts. These similarities will be explored further in the field-based inquiry, 

interviews with district officials and a second round of survey data collection. Later analyses will 

also focus on the extent of implementation over time. 

Faculty Role 

The presence of CSR funding is associated with differences in the faculty role in reform. 

Teachers in CSR schools were significantly more likely to have voted to adopt the reform

than in non-CSR schools (82 percent as compared with 55 percent). Further, although the 

greatest number of CSR respondents indicated that all teachers participate in the reform (81 

percent), this was not significantly different from the response from the non-CSR schools (78 

percent). However, CSR schools were more likely to be phasing in the reform (31 percent) and 

involving additional teachers over time than were non-CSR schools (21 percent), according to 

respondents (Exhibit 8). 

Exhibit 8 

Extent of Faculty or Teacher Participation in Reform at the School 

92%

82% 81%

31%

86%

55%

78%

21%

0%

20%

40%

60%

80%

100%

Teachers participating

in reform

Teachers voted to

adopt reform*

Schools with ALL

teachers participating

in reform

Phasing in the reform*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: Teachers at CSR schools were significantly more likely to have voted to 

adopt the reform effort at their school (82 percent compared with 55 percent at non-CSR 

schools). CSR schools also were more likely to be phasing in the reform (31 percent) than 

were non-CSR schools (21 percent). 
22 

Most model developers require a faculty vote before they work with a school, so the 

association of votes with CSR is also an association with the use of a model. In addition, the 

phasing in of reform may also be associated with working closely with model developers. As 

will be seen, developers provide on-site assistance, which gives them knowledge of the school 

and faculty and may lead to their advising phasing in reform. As the evaluation continues, it will 

explore more closely whether developers influence such strategies as phasing in reform. Further, 

the evaluation will assess the extent to which actions such as faculty votes, phasing in reform, 

and receiving on-site assistance influence implementation and outcomes. 

School Performance Goals 

Almost all respondents in both groups reported having performance goals in reading and 

language arts (98 percent and 100 percent) and mathematics (94 percent and 96 percent) 

(Exhibit A-6), and less than half of either group have annual performance goals by either grade 

level (48 percent and 44 percent) or other content area (32 percent and 29 percent) (Exhibit A-7). 

(The new requirements of NCLB and the influence of state performance goals are likely to 

change such findings in the next round of data collection). In both groups, more teachers than 

principals reported that their schools had annual performance goals by grade level and content 

area (80 percent of teachers compared with 48 percent of principals in CSR schools; 78 percent 

of teachers compared with 44 percent of principals in non-CSR schools). 

However, differences existed about the factors that influence the performance goals and 

how the goals and the reform were evaluated. Each of these is discussed in turn. 

State tests were the major determinant of school performance goals for both CSR and 

non-CSR schools (87 percent and 89 percent). Both groups were also greatly influenced by state 

content standards (83 percent and 81 percent). Few in either group are influenced by parent 

concerns (28 percent and 32 percent). However, schools participating in CSR report that the 

reform effort influences their performance goals more often than do the non-CSR schools

(57 percent vs. 41 percent) (Exhibit 9). This difference is significant. 
23 

Exhibit 9 

Factors Influencing the Creation of Performance Goals at the School 

87% 83%

57%

28%

89%

81%

41%

32%

0%

20%

40%

60%

80%

100%

State tests State content standards Reform effort* Parent concerns

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: State tests and state content standards were the major determinants of 

school performance goals for both CSR and non-CSR schools. However, 57 percent of 

CSR schools reported that the reform effort being implemented influenced their 

performance goals, compared with 41 percent of non-CSR schools. 

CSR schools differed from the non-CSR schools in how they evaluated their performance 

and the evidence they used to show the progress of reform. CSR schools were significantly 

more likely to have a formal plan to evaluate their progress (91 percent compared with 82 

percent). However, the greatest influence on the questions addressed in evaluation comes from 

state requirements for both CSR and non-CSR schools. Nonetheless, CSR schools were more 

likely to attend to the requirements of reform (66 percent), than were non-CSR schools (42 

percent) (Exhibit A-8). 

Equally important, CSR and non-CSR schools use different evidence for evaluation. 

Non-CSR schools use “results from students at this school” significantly more than do CSR 

schools (82 percent for the non-CSR schools and 68 percent for the CSR schools). (Because the 

information comes from the surveys, it is not possible to tell whether the results being used are 

formal evaluations or informal assessments. The field-based study will pursue this question 

further.) In contrast, the CSR schools, used research conducted by the reform designer (47 

percent compared with 26 percent for non-CSR schools) and independent research (42 percent 

vs. 26 percent for non-CSR schools). The difference in sources of evidence seems related to the 

adoption of a reform model. It may also be an indication that CSR schools’ capacity for reform is 

increasing (Exhibit 10). 
24 

Exhibit 10 

Source of Evidence to Link School Reform to Student Achievement 

68%

47%

42%

82%

26% 26%

0%

20%

40%

60%

80%

100%

Results from students at this

school*

Research conducted by reform

designer*

Independent research*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: When evaluating school reform efforts, non-CSR schools use “results from 

students at this school” (82 percent) significantly more often than CSR schools (68 

percent). CSR schools, on the other hand, were more likely to rely on research conducted 

by the reform designer (47 percent) and independent research (42 percent). 

Professional Development 

Professional development is an essential component of school reform because reform 

requires teachers to learn new practices, either for classroom application or engagement in 

different forms of school organization. The surveys showed differences in professional 

development practices in CSR and non-CSR schools, in the locus of activity, the number of days 

provided, and the content. CSR schools noted changes in the kinds of professional development 

activities from last year to this year. 

Professional development was included in the school reform plan more frequently in 

CSR schools than in non-CSR schools. In addition, a greater number of CSR schools than 

non-CSR schools provided more than 10 days for professional development and received 

on-site assistance from external supporters. Ninety percent of CSR schools included 

professional development for all teachers in their school reform plan compared with 73 percent 

of non-CSR schools. CSR schools also provided more than 10 days for professional development 

more often than did non-CSR schools (56 percent as compared with 39 percent). External 

assistance providers supported reform efforts on-site in significantly more CSR schools (85 

percent) than non-CSR schools (57 percent). Finally, formal evaluation plans in CSR schools 
25 

were more likely to include assessment of the utility of external assistance than such plans in 

non-CSR schools (41 percent vs. 30 percent) (Exhibit 11). 

Exhibit 11 

Status of Professional Development in School Reform 

*Difference is statistically significant at the .01 level. 

Exhibit reads: In CSR schools as compared with non-CSR schools, professional 

development more often was included in the school reform plan (90 percent vs. 73 

percent), offered for 10 days or more (56 percent vs. 39 percent), and took the form of 

on-site assistance from external sources (85 percent vs. 57 percent) 

The following professional development opportunities increased significantly in CSR 

schools from 2001-02 to 2002-03 (Exhibit A-9): 

• Reading or language arts instruction. 

• Mathematics instruction. 

• Instructional strategies for low-achieving, limited English proficient, special 

education, or migrant students. 

• Ensuring that curriculum and instruction are consistent with state and district 

content standards. 

• Ensuring that curriculum and instruction are consistent with state and district 

assessments. 

• Implementation of a school reform model. 

• Monitoring individual students’ progress toward learning goals. 

CSR schools Non-CSR 

schools 

Professional development for all teachers is 

included in school improvement plan 90% 73%* 

School provides 10 or more days for 

professional development 56% 39%* 

School receives on site support for reform 

efforts from external providers 85% 57%* 

School evaluation plan includes assessment of 

the utility of external assistance 41% 30%* 
26 

• Interpreting reports of student achievement data. 

CSR teachers reported slightly fewer professional development opportunities than 

principals related to instructional strategies for low-achieving, limited English proficient, special 

education or migrant students (71 percent of teachers as compared with 60 percent for 

principals). CSR teachers also reported slightly fewer professional development opportunities 

than principals related to monitoring student progress (70 percent of teachers compared with 59 

percent of principals) and interpreting reports of student data (83 percent of teachers compared 

with 69 percent of principals). Despite these differences, the rates of change reported by CSR 

teachers for these three types of opportunities from 2001-02 to 2002-03 were similar to those 

reported by CSR principals. 

In addition, teachers and principals in CSR schools reported the availability of different 

types of professional development opportunities. CSR principals were more likely than CSR 

teachers to report that teachers coached other teachers (71 percent of principals versus 59 percent 

of teachers) and made management decisions (62 percent versus 46 percent). 

For the most part, differences in professional development opportunities are differences 

associated with the formal CSR model requirements. In one area, scheduling for common 

planning time, non-CSR schools reported significantly less opportunities than did CSR schools 

(31 percent of non-CSR schools principals responded that no common planning time is 

scheduled vs. 21 percent of CSR schools). The reason for this difference will be explored further 

in the field-based study. 

Support for School Reform 

Significant differences existed between CSR and non-CSR schools in the type of support 

they received. As might be predicted, CSR schools were far more likely to receive support 

from a model developer than non-CSR schools (31 percent vs. 6 percent). However, non-CSR 

schools were more likely to report receiving support for school reform efforts from the 

district than were CSR schools (72 percent of non-CSR schools and 34 percent of CSR 

schools) (Exhibit 12). This difference will be explored further in the telephone interviews and 

case studies. 
27 

Exhibit 12 

Entity Primarily Responsible for Supporting Reform Efforts at the School 

34% 31%

72%

6% 0%

20%

40%

60%

80%

100%

School district* Reform program developer*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: CSR schools were far more likely to identify model developers as the 

primary supporters of reform at their school (31 percent for CSR schools compared with 

6 percent for non-CSR schools). Conversely, non-CSR schools (72 percent) reported 

more district support for school reform efforts than CSR schools (34 percent). 

States and districts were more likely to provide funds for reform to non-CSR schools 

than to CSR schools. Discretionary district funds went to 58 percent of non-CSR schools 

compared with 44 percent of CSR schools. Special state grants were awarded to 53 percent of 

non-CSR schools compared with 44 percent of CSR schools (Exhibit 13). 
28 

Exhibit 13 

Sources of Funding That Contribute to Implementation and Operation of School Reform 

44% 44%

58%

53%

0%

20%

40%

60%

80%

100%

Discretionary district funds* Special state grants*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: Schools reported that special state grants and district discretionary funds 

were more commonly used for reform at non-CSR schools (58 percent for district funds; 

53 percent for state funds) than at CSR schools (44 percent for each source of funds). 

Further, districts supported different kinds of activities in CSR schools than non-CSR 

schools in the first year of reform implementation. Districts were more likely to help CSR 

schools select a school reform model (45 percent in CSR schools compared with 32 percent in 

non-CSR schools) but were less likely to provide CSR schools with professional development 

for school reform (72 percent in CSR schools and 86 percent in non-CSR schools) 

(Exhibit A-10). States also were more likely to help CSR schools select a school reform model 

(25 percent) than they were to help non-CSR schools select a model (16 percent) (Exhibit A-11). 

Both CSR and non-CSR schools reported coordinating funds from a variety of sources to 

support professional development and align Title I activities, but neither aligned other funds 

(e.g., bilingual education) or reallocated staff positions.3

Instructional Practice 

CSR and non-CSR school respondents indicated little difference in instructional 

practice. Both groups reported that curriculum scope and sequence was determined at the district 

 

3 Although 22 percent of the non-CSR schools reported receiving federal CSR funds, later follow-up indicated that only five 

schools were actually recipients of CSR funds. Those schools have been removed from the analysis. 
29 

level (78 percent CSR; 83 percent non-CSR). Other loci for determining scope and sequence 

varied, but not significantly. For example more non-CSR schools reported that the state 

organized curriculum (52 percent) than did CSR schools (46 percent), and more CSR schools 

reported individual teachers (34 percent) or schools (51 percent) organized curriculum than did 

non-CSR schools (27 percent; 47 percent) (Exhibit 14). 

Exhibit 14 

Primary Organizer of the Scope and Sequence of Curriculum at the School 

78%

51%

46%

34%

83%

47%

52%

27%

0%

20%

40%

60%

80%

100%

District School State Individual teacher

CSR Schools Non-CSR Schools

Exhibit reads: The district was primarily responsible for determining curriculum scope 

and sequence at CSR schools (78 percent) and non-CSR schools (83 percent). More 

non-CSR schools reported that the state organized curriculum while CSR schools were 

slightly more likely to cite individual teachers or schools as controlling curriculum and 

instruction. 

Further, about the same percentages of CSR and non-CSR schools reported participating in 

grade-level teams (86 percent of CSR respondents vs. 84 percent of non-CSR school 

respondents) or in content area teams across grades (71 percent, CSR vs. 75 percent non-CSR). 

As implementation continues, changes may be observed with regard to scope and sequence, 

but little change in responses related to teaming is expected, given the high percentages that 

currently exist. Additional information will come from the field-based studies. 
30 

The Role of Parents 

CSR and non-CSR schools did not differ in how parents were involved in reform.

Both communicated with parents and encouraged parent involvement in similar ways. They were 

also similar in their assessment of the ways parents were engaged with the school. 

Both groups communicated with parents most frequently through telephone calls (95 

percent for CSR schools; 94 percent for non-CSR schools), regular newsletters (83 percent CSR 

and 86 percent non-CSR) and in the language other than English spoken at home (45 percent, 

CSR; 46 percent, non-CSR). However more CSR schools reported having a parent coordinator 

(43 percent) than did non-CSR schools (36 percent) although the differences were not 

significant. (Exhibit 15). 

Exhibit 15 

Methods of Formal Communication from Schools to Parents 

95%

83%

45% 43%

94%

86%

46%

36%

0%

20%

40%

60%

80%

100%

Telephone calls Regular newsletters Language spoken at

home (not English)

Designated parent

coordinator

CSR Schools Non-CSR Schools

Exhibit reads: CSR and non-CSR schools communicated with parents most frequently 

through telephone calls and newsletters. Slightly more CSR schools reported having a 

parent coordinator (43 percent) than non-CSR schools (36 percent). 

The schools encouraged parental involvement in governance with regard to fundraising (79 

percent for CSR schools vs. 86 percent for non-CSR schools), defining the school mission and 

goals (75 percent CSR schools vs. 77 percent, non-CSR schools) and evaluating school 

performance (67 percent CSR schools vs. 68 percent non-CSR schools) (Exhibit 16). Parents 

were not involved in choosing instructional materials, hiring teachers and staff, or developing the 

calendar in either type of school. 
31 

Exhibit 16 

Ways Parents Are Engaged and Involved at School 

79%

75%

67%

86%

77%

68%

0%

20%

40%

60%

80%

100%

Fundraising Defining school mission and

goals

Evaluating school peformance

CSR Schools Non-CSR Schools

Exhibit reads: Most commonly, schools reported parent involvement in fundraising 

activities. Three quarters of the schools solicited parent input when defining the school 

mission. Fewer schools reported parent involvement in evaluating school performance. 

Types of School Reform Models or Strategies Being Implemented 

The distribution of reform models in the CSR and non-CSR samples was determined by 

examining the responses to the survey item, “Does the primary reform effort at your school have 

a name? If so, write the name here.” The higher number of named reforms in CSR sample 

schools suggests that CSR schools are adopting externally developed models more than 

non-CSR schools (Exhibit 17). Of the 85 percent of CSR schools that responded their primary 

reform was named, 80 percent of CSR principals named models that were focused on the entire 

school. In contrast, of the 49 percent of non-CSR schools that had a named reform, 78 percent of 

principals indicated a whole-school model. The remainder in each group was subject specific, 

such as literacy reforms. 
32 

Exhibit 17 

 Schools, by Type, with Primary Reform Efforts Identified by Name 

Exhibit reads: On average, 85 percent of CSR schools identified their primary reform 

effort with a specific name, suggesting the presence of an externally developed model, 

compared with half of the non-CSR schools. CSR schools at each grade level were 

equally likely to identify their reform effort with a specific name. 

CSR schools implemented a greater diversity of reforms than did non-CSR schools. 

CSR schools named 38 different reforms and non-CSR schools 28 (excluding generic names 

such as “school improvement program”). In CSR schools, the most frequently named models 

were the products of private developers. However, in non-CSR schools, other than Success for 

All, regional school accreditation and state programs were the most frequently mentioned 

(Exhibit 18). 

Exhibit 18 

Number and Type of Named Primary Reform Efforts in Use at Schools 

Exhibit reads: CSR schools identified a more diverse list of reforms in use at 

their schools. Further, the most frequently named models in CSR schools were 

the products of private developers, as compared with non-CSR schools. 

 CSR Schools Non-CSR Schools 

Elementary 87% 52% 

Middle 81% 45% 

High 87% 42% 

Weighted Average 85% 49% 

 CSR Schools Non-CSR Schools 

Number of 

Whole School 

Reforms Named 

38 28 

Success for All 

Renaissance 

Co-nect 

America’s Choice 

Most Frequently 

Named Models 

Direct Instruction 

Success for All 

State 

Regional 
33 

CSR and Developing Capacity for Reform 

Data that focus on differences between the first year of CSR (2002-03) and the prior year 

(2001-02) assess whether CSR increases schools’ capacity for reform. As noted earlier, CSR 

schools in the sample showed lower student achievement and higher poverty than non-CSR 

schools. They were also more likely to have been identified or sanctioned as low performing 

according to the criteria used in their state. Consequently, although changes in practice related to 

capacity for reform are limited, they are important because CSR schools must show greater 

improvement than non-CSR schools to provide students with opportunity to learn high standards. 

Newmann et al. (2001) state that the three following conditions are important in schools' 

capacity to ensure the coherence and cohesiveness of reform. 

• A common framework to guide curriculum, instruction, assessment, and the 

learning climate. 

• Staff working conditions to support implementation of the framework, 

including professional development for staff. 

• School allocation of resources to support implementation. 

Most CSR schools are working toward a common framework. The framework is shaped by 

the model adopted as well as state and district performance standards. The existence of a 

framework in 2002-03 represents a major change from the previous year. A higher percentage 

of CSR principals reported their schools had a comprehensive written plan in 2002-03 (93 

percent) as compared with the previous year (75 percent), indicating some influence of CSR. 

In contrast, principals in non-CSR schools reported little change (89 percent had comprehensive 

written plans in 2002-03, compared with 86 percent in 2001-02). In addition, CSR schools in the 

sample were significantly more likely to report engaging in whole school reform in 2002-03 

(76 percent) than in 2001-02 (55 percent). As indicated above, they were more likely to report 

implementing a single reform effort, increasing coherence. 

Although CSR schools reported little difference in how the instructional staff was 

organized and fewer cross-subject area or within-grade teams than the non-CSR schools, 

teachers in CSR schools did report more opportunities for professional development. Teacher 

participation in grade-level or content area teams increased significantly in 2002-03, compared 

with the previous year (Exhibit 19). In addition, CSR teachers reported receiving more days of 

professional development in 2002-03 than the year prior, and the training was more focused on 

issues related to reform (Exhibit 20). 
34 

Exhibit 19 

Teacher Enhancement Opportunities Available 

before and after Reform Implementation 

73%

58%

86%

71%

0%

20%

40%

60%

80%

100%

Grade-level teams* Content area teams*

2001-02 2002-03

*Difference is statistically significant at the .01 level. 

Exhibit reads: Compared with pre-implementation levels, teachers in CSR schools 

reported more opportunities to work on grade-level teams (86 percent vs. 73 percent the 

previous year) and content area teams (71 percent vs. 58 percent) during the first year of 

CSR. 
35 

Exhibit 20 

Types of Professional Development in Which Teachers Participated in during the First 

Year of Reform Implementation (2002-03) Compared with the Previous Year 

84%

46%

73%

63% 67%

54%

50%

61%

94%

85% 83% 83% 79%

71% 70% 69%

0%

20%

40%

60%

80%

100%

Reading/lang.

arts

Model

implementation

Mathematics Intrepreting

reports

Consistency 

with content

Instructional

strategies

Monitoring

students'

progress

Consistency with

assessments

2001-02 2002-03

*Difference is statistically significant at the .01 level for all items. 

Exhibit reads: CSR teachers were much more likely to receive training on model 

implementation (85 percent), monitoring students’ progress (70 percent), and interpreting 

reports (83 percent) in 2002-03, compared with 2001-02 (the year prior to CSR 

implementation). 

On the other hand, the CSR schools were not coordinating resources to any greater 

extent than non-CSR schools in support of reform efforts at the school (28 percent 

compared with 35 percent). CSR schools were using the federal CSR funds to support the 

reform, which indicates that capacity to continue reform may end with the end of the CSR grant. 

The importance of coordinated resources will be assessed as the evaluation continues 

At the end of a year of implementation, CSR has increased participating schools’ capacity 

to reform to some extent on two key dimensions, developing comprehensive written plans and 

increasing team meetings. It has yet to produce changes in how schools coordinate resources. 

The Nature of Comprehensive School Reform 

This report addresses the question: "What does comprehensive school reform look like at 

the end of the first year of implementation?" This section summarizes findings in response to that 

question. In addition, CSR is intended to increase schools’ capacities to reform. This section also 

summarizes findings relevant to reform capacity. 
36 

SUMMARY OF CSR IMPLEMENTATION 

CSR comprises 11 components whose existence and interaction may improve schools. 

Respondents to the survey indicated that both CSR and non-CSR schools were implementing a 

number of the components. However, CSR schools were more likely than non-CSR schools to 

implement components most associated with adopting a model. Consequently, the presence of 

some similar components in CSR and non-CSR schools may not indicate equal progress toward 

reform nor lead to equal outcomes for students. The differences in the components that are 

implemented in CSR and non-CSR schools may well encompass different interactions, which, in 

turn affect the extent to which schools are coherent and cohesive, enabling them to provide 

students with focused and challenging opportunities to learn to high standards. 

CSR schools, as compared with non-CSR school, were more likely to implement the 

following components: 

• Adopt externally developed methods and strategies that have been replicated. 

They did so by: 

- Identifying a specific reform model (85 percent compared with 49 

percent). 

- Using evidence from research that the reform model chosen improves 

student achievement (42 percent compared with 26 percent). 

• Provide more continuous professional development. They did so by: 

- Including professional development activities for all teachers (90 percent 

compared with 73 percent). 

- Allocating over 10 days to teacher professional development (56 percent 

compared with 39 percent). 

• Include measurable goals for student performance associated with the reform 

model (57 percent compared with 41 percent). 

• Reflect support from staff by including a formal vote by teachers for the 

reform model (82 percent compared with 55 percent). 

• Provide support for staff by receiving on-site consulting relevant to the reform 

(85 percent compared with 57 percent). 

• Evaluate the reform. They did so by: 

- Including the requirements of the reform model in the scope and content 

of evaluation (66 percent compared with 42 percent). 
37 

- Assessing the utility of external assistance (41 percent compared with 30 

percent). 

To some extent, the differences are associated with the adoption of models by the CSRfunded schools. At this point of early implementation, the evaluation cannot assess whether some 

models lead to greater implementation and outcomes than others. However, the act of selecting a 

model has some immediate outcomes. For example, most models require faculty votes as 

indications of buy-in. The vote itself may create the conditions for coherent implementation, and 

when accompanied by on-site assistance (provided by the model developer), strengthen the 

probability of full implementation. Further, models frequently include performance goals, so the 

CSR schools are more likely to have reform-specific performance goals. Model developers also 

require professional development, and even when they do not require it, offer model-focused 

opportunities, so the higher number of days for professional development can also be associated 

with adoption of a model. 

In sum, both CSR and non-CSR schools exhibited many aspects of comprehensive 

reform. However, CSR schools were more likely to adopt externally developed models. 

Other differences between the two types of schools were related to model adoption. 

The Influence of State and District Policies on the Implementation of 

Comprehensive Reform Programs 

State and local policies influence CSR and non-CSR schools in similar ways, except 

with regard to selecting the reform model. Reform plans in both types of schools are highly 

influenced by state or district content and performance standards (90 percent of CSR schools and 

91 percent of non-CSR schools on content standards; 87 percent of CSR schools and 91 percent 

of non-CSR schools on performance standards) (Exhibit 21). Similarly, state content standards 

and testing requirements have the greatest influence on performance goals at both CSR and 

non-CSR schools. Further, districts, rather than states or individual teachers, have the most 

influence on curriculum scope and sequence in both CSR and non-CSR schools (51 percent in 

CSR schools and 47 percent in non-CSR schools). 
38 

Exhibit 21 

Influence of State and Local Policies on School Improvement Plans 

90%

87%

91% 91%

0%

20%

40%

60%

80%

100%

Content standards Performance standards

CSR Schools Non-CSR Schools

Exhibit reads: Most principals in CSR and non-CSR schools report that content and 

performance standards influence school improvement plans. 

Both groups of schools report control over budgets and personnel decisions similarly (82 

percent for CSR and 75 percent for non-CSR with budgetary control; 78 percent for CSR and 80 

percent for non-CSR regarding personnel decisions) (Exhibit 22). 
39 

Exhibit 22 

 Control of Budget and Personnel in CSR and Non-CSR Schools 

82% 78% 75%

80%

0%

20%

40%

60%

80%

100%

Bugetary control Personnel decisions

CSR Schools Non-CSR Schools

Exhibit reads: The majority of CSR and non-CSR schools report that budgetary and 

personnel decisions are controlled at the school level. 

Teachers and school administrators were involved in selecting the reform model or 

approach being implemented, both at CSR schools and non-CSR schools. However, the 

school board and the district central office played a more significant role in selecting 

reform at non-CSR schools than at CSR schools, indicating more “top-down” requirements 

for changes in practice. One third of CSR schools reported that the district central office was 

one of several entities responsible for selecting the reform, compared with 57 percent of 

non-CSR schools. Non-CSR schools also reported that school board members were involved in 

the decision at a higher rate than did CSR schools (24 percent for non-CSR schools compared 

with 15 percent of CSR schools) (Exhibit 23). Further, state or district mandates were more 

likely to contribute to the selection of reform at non-CSR schools (60 percent) than at CSR 

schools (31 percent). 
40 

Exhibit 23 

Entities Involved in Selecting a Reform Model or Approach at the School 

72%

66%

57%

34% 32%

15%

76%

63% 63%

57%

36%

24%

0%

20%

40%

60%

80%

100%

School

improvement

team

Teachers School

administrators

District central

office*

Parents School Board*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: School staff were involved in selecting a reform model or approach at both 

CSR and non-CSR schools. However, the district central office (57 percent) and school 

boards (24 percent) had a greater role at non-CSR schools than at CSR schools (34 

percent; 15 percent). 
41 

IV. Summary 

This final chapter summarizes findings related to school reform implementation. The 

following section presents policy implications. 

Summary of Findings 

CSR funds have been targeted to high-need schools as intended. Participating schools tend 

to have high poverty rates and high concentrations of minority students, and have frequently 

been identified as low-performing according to state criteria. 

Both CSR and non-CSR school personnel view themselves as engaged in activities 

associated with school reform. Further, activities include the majority of the 11 components 

identified in NCLB. However, CSR schools are more intensively implementing components 

most associated with adopting a research-based model. They were more likely to: 

• Adopt externally developed methods and strategies that have been replicated. 

• Provide more continuous professional development. 

• Include measurable goals for student performance associated with the reform 

model. 

• Reflect support from staff by including a formal vote by teachers for the 

reform model. 

• Provide support for staff by receiving on-site consulting relevant to the 

reform. 

• Evaluate the reform. 

Consequently, the Comprehensive School Reform program is, at the early stage of 

implementation in the schools in the sample, affecting how they address low performance. While 

CSR and non-CSR schools are implementing many components of reform, CSR influences how 

schools approach reform. CSR schools are likely to adopt a single, externally developed reform 

model. However, they are not more likely to integrate budgets than are other schools, which may 

have longer-term consequences for the reform effort. The evaluation will seek to determine 

whether the differences between CSR and non-CSR school reform affects implementation and 

outcomes over time. 
42 

Implications 

The first year of the evaluation has yielded information with implications for federal 

policy. The implications relate to two key findings: 

• Although both CSR and non-CSR schools are engaged in reform, reform in 

CSR schools includes adoption of models and other activities closely 

associated with research-based models. 

• CSR funds are strongly targeted to high-poverty schools and low-performing 

schools, and schools receiving CSR funds are lower performing than are other 

schools with similar demographic characteristics. 

Taken together, the findings raise interesting questions. The first is whether the use of CSR 

funds accelerates reform in the lowest performing schools. States and districts seem to have 

targeted CSR funds to those schools that have the greatest need to change practices in order to 

support high achievement for all students. With CSR funds, the schools were more likely to 

adopt models, focus professional development on reform, and track student performance than 

were non-CSR schools. Both CSR and non-CSR schools were engaged in other reform activities. 

In subsequent years, the evaluation will provide information about whether CSR schools 

implement more reform components more thoroughly than do the non-CSR schools. If they do, 

CSR can be seen as adding value to improvement by providing a mechanism that focuses efforts 

and enables school staff members to organize themselves in ways that offer greater educational 

opportunities for students. Perhaps CSR helps schools jump-start improvement. 

Second, data from the first year of this evaluation indicate that all schools in the sample are 

engaged in many aspects of what the legislation defines as “comprehensive school reform.” 

Consequently, the study has implications about the nature of reform in general. Most lowperforming schools in the non-CSR group are making efforts to improve. Questions then arise as 

to whether the efforts are associated with improved outcomes: Do schools succeed in reform 

without models to organize them? Are models only important in the lowest performing schools? 
43 

Appendix A: Selected Data Tabulations 

Exhibit A-1 

Baseline Student Achievement Scores 

 

Averaged Baseline Student Achievement Z-Scores** 

 

MATH READING 

 Elem. Middle High Elem. Middle High

2002 CSR 

Cohort -0.95 -0.86 -0.68 

2002 CSR 

Cohort -0.92 -0.89 -0.69 

SW Title I -0.50 -0.46 -0.66 SW Title I -0.54 -0.49 -0.72 

Title I -0.24 -0.19 -0.24 Title I -0.26 -0.21 -0.25 

All schools 0.00 0.00 0.00 All schools 0.00 0.00 0.00 

 

Distribution of 2002 CSR Cohort Average Z-Scores by Poverty Level 

 

MATH READING 

 Elem. Middle High Elem. Middle High

75-100% -1.30 -1.22 -1.10 75-100% -1.30 -1.23 -1.05 

50-74 -0.59 -0.60 -0.88 50-74 -0.54 -0.71 -0.99 

25-49 -0.27 -0.33 -0.51 25-49 -0.22 -0.29 -0.41 

0-24 -0.62 -0.32 -0.25 0-24 -0.67 -0.07 -0.38 

 

Distribution of 2002 CSR Cohort Average Z-Scores by Locale 

 

MATH READING 

 Elem. Middle High Elem. Middle High

Urban (1&2) -1.31 -1.22 -1.09 Urban (1&2) -1.29 -1.22 -1.11 

Suburb. (3&4) -0.83 -0.84 -0.49 Suburb. (3&4) -0.84 -0.88 -0.57 

Town (5&6) -0.37 -0.36 -0.81 Town (5&6) -0.38 -0.36 -0.58 

Rural (7&8) -0.28 -0.27 -0.25 Rural (7&8) -0.34 -0.32 -0.24 

 

**Data from some states were not available from the 2002 school year. See achievement indicators table for details. 

 

Exhibit reads: CSR schools had lower achievement scores in math and reading than did 

Title I schoolwides, Title I schools, or all schools. Within the 2002 CSR cohort, schools 

with high poverty rates and urban locales scored lowest on reading and math achievement 

tests 
44 

Exhibit A-2 

CSR and Non-CSR Schools Reporting Formal Comprehensive School Improvement Plans 

93%

75%

89% 86%

0%

20%

40%

60%

80%

100%

2002-03 2001-02

CSR Schools Non-CSR Schools

Exhibit reads: A higher percentage of CSR schools reported having formal school 

improvement plans in 2002-03 (the first year of CSR implementation) as compared with 

the year prior to CSR implementation (93 percent compared with 75 percent). The 

number of non-CSR schools reporting formal school improvement plans over that time 

increased marginally (89 percent compared with 86 percent). 
45 

Exhibit A-3 

Factors Influencing the Content of the School Improvement Plan 

90% 87%

82%

74%

63%

37%

91% 91%

83%

77%

34%

46%

0%

20%

40%

60%

80%

100%

State or district

content

standards

State or district

performance

standards

Needs identified

through a school

needs

assessment

School

performance

standards

Specifications of

adopted/adapted

reform design*

Assignment by

district/state

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: CSR and non-CSR schools were equally likely to cite state and district 

content standards as the major influence on their school improvement plan (90 percent; 

91 percent). CSR schools were significantly more likely to cite the specifications of a 

reform design as influencing their school improvement plan (63 percent as compared 

with 34 percent). 
46 

Exhibit A-4 

Source of Reform Plans at CSR and Non-CSR Schools 

42%

32%

16%

10%

41% 40%

16%

3%

0%

20%

40%

60%

80%

100%

Adapted with

modifications from

external source

Totally designed at the

school

Adapted selected parts

from external source

Adapted unmodified

from external source

CSR Schools Non-CSR Schools

Exhibit reads: CSR schools and non-CSR schools were equally likely to report that their 

reform efforts were “adapted with modifications from an external source” (42 percent and 

41 percent). Non-CSR schools were slightly more likely than CSR schools to be 

implementing a reform “totally designed at the school.” 
47 

Exhibit A-5 

The Primary Designer for the Reform at Your School 

37%

32%

14% 10%

51%

37%

6% 0%

0% 6% 5%

20%

40%

60%

80%

100%

Locally

developed*

Private developer

or publisher*

School district* University* Other*

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level for all items. 

Exhibit reads: Schools receiving CSR funds were more likely that non-CSR schools to 

adopt a reform designed by a private developer or publisher (32 percent compared with 6 

percent). Nearly 90 percent of non-CSR schools reported implementing reforms that were 

designed locally (51 percent) or by the district (37 percent), compared with just over half 

of CSR schools reporting the same. 
48 

Exhibit A-6 

Academic Subjects Included in School Goals or Benchmarks 

98%

94%

58%

51%

19%

100% 96%

62%

55%

19%

0%

20%

40%

60%

80%

100%

Reading/language

arts

Mathematics Science Social Studies Arts

CSR Schools Non-CSR Schools

Exhibit reads: Almost all CSR and Non-CSR schools report performance goals in reading 

or language arts and mathematics. 
49 

Exhibit A-7 

Types of School Performance Goals for Students at CSR and Non-CSR Schools 

48%

32%

44%

29%

0%

20%

40%

60%

80%

100%

By grade level By content area

CSR Schools Non-CSR Schools

Exhibit reads: Less than half of CSR or non-CSR schools have annual performance goals 

by either grade level or content area. 
50 

Exhibit A-8 

Factors Influencing the Evaluation of School Performance Goals 

86%

74%

67% 66% 63%

89%

82%

73%

42%

62%

0%

20%

40%

60%

80%

100%

State requirements District

requirements

School-developed

requirements

Requirements of

reform*

Federal Title I

requirements

CSR Schools Non-CSR Schools

*Difference is statistically significant at the .01 level. 

Exhibit reads: A significant difference existed in the consideration of the requirements of 

reform, with 66 percent of CSR schools reporting that they include such requirements in 

their evaluation as compared with 42 percent of non-CSR schools. 
51 

Exhibit A-9 

Professional Development Opportunities Available to Teachers in 

the First Year of CSR Reform (2002-03) Compared with the Previous Year 

2001-02 2002-03 

Reading/language arts instruction 84% 94% 

Mathematics instruction 73% 83% 

Instructional strategies for low-achieving, limited English 

proficient, special education, and/or migrant students 

54% 71% 

Ensuring that curriculum and instruction are consistent with 

state and/or district content standards 

67% 79% 

Ensuring that curriculum and instruction are consistent with 

state and/or district assessments 

61% 69% 

Implementation of a school reform model 46% 85% 

Monitoring individual students’ progress toward learning 

goals 

50% 70% 

Interpreting reports of student achievement data 63% 83% 

Exhibit reads: Professional development opportunities increased significantly at CSR 

schools during the first year of CSR implementation. Professional development around 

reading or language arts and mathematics instruction was most frequent. 

Exhibit A-10 

Type of Support Available through the District for Reform Efforts 

CSR non-CSR 

Administering a needs assessment 45% 47% 

Providing additional school staff to support school reform 41% 43% 

Selecting a school reform model 45% 32%* 

Writing grants to support school reform 64% 56% 

Providing professional development for school reform 72% 86%* 

Release time for teachers 57% 66% 

None, the district does not supply additional support 6% 2%* 

*Difference is statistically significant at the .01 level. 

Exhibit reads: School districts were more likely to assist CSR schools in selecting school 

reform models than non-CSR schools (45 percent compared with 32 percent). In 

non-CSR schools, districts were more likely to provide professional development for 

school reform (86 percent vs. 72 percent in CSR schools). 
52 

Exhibit A-11 

Type of Support Available through the State for Reform Efforts 

CSR non-CSR 

Administering a needs assessment 22% 21% 

Selecting a school reform model 25% 16%* 

Writing grants to support school reform 27% 35% 

Providing professional development for school reform 51% 49% 

Release time for teachers 13% 18% 

None, the state does not supply additional support 26% 27% 

*Difference is statistically significant at the .01 level. 

Exhibit reads: Support from states was significantly higher for selecting a reform model 

in CSR schools (25 percent) vs. non-CSR schools (16 percent) otherwise the level of state 

support for CSR and non-CSR schools is not significantly different. Note, nearly half the 

schools reported that states also provided professional development to support reform in 

both CSR and non-CSR schools. 
53 

Appendix B: Technical Appendix 

Sample and Comparison School Selection for LACIO 

The evaluation team sampled Comprehensive School Reform (CSR) schools in two stages. 

In the first stage, researchers selected a large number of schools randomly from the universe of 

schools that received CSR funding in 2002 and selected matched comparison schools for each 

sample school. Schools for the field-based study were selected in the second stage. From a 

random sub-sample of schools researchers deliberately selected schools to ensure that the sample 

was distributed evenly geographically and by school level. 

Selection of School Survey Sample 

The evaluation team ensured that the diversity in the population of schools was maintained 

in the sample by randomly selecting a large enough number of CSR schools to complete surveys. 

The sample of 400 schools is about 36 percent of the universe of approximately 1,100 schools 

reported to receive CSR funds for calendar year 2002. 

Researchers obtained the most complete list of CSR schools from the Southwest 

Educational Development Laboratory (SEDL). SEDL maintains this database under contract to 

the U.S. Department of Education. The Department of Education requires states to report their 

awardees to SEDL; however, the database did not include all states in the year 2002 for several 

reasons. Although SEDL continuously collects data, at any given time the data may be 

incomplete due to delinquent reporting by state education agencies. In addition, many states only 

award CSR funds to schools biennially or irregularly. As a result, the universe for calendar year 

2002 comprised 38 states and 1,096 schools out of an estimated potential number of 2,000 

awards. 

Using random sampling only, researchers chose schools that were representative of the 

population without stratification for the survey sample. The study does not meet the requirements 

for stratification because there are no subpopulations (schools) that are either domains of the 

study or require different study procedures (Kish 1995). However, in order to represent the 

diversity of policy environments where CSR is being applied, researchers took some school 

characteristics into account to check the representativeness of the sample. As shown later in this 

section, the distributions of the study sample across locale and school level were comparable to 

the distributions of the CSR universe. 

During the preparation of this report, the Department of Education allocated additional 

funds to study 15 states that were not included in the original random sample. Of the initial 38 

states in the 2002 CSR universe, three states were left out as a by-product of the random 

selection process. Five additional states reported data to SEDL after the initial sample selection. 

The remaining seven states did not make any awards in 2002 but have begun to report 2003 data 

to SEDL. Researchers selected an additional 100 schools from these missing states for two 

reasons. First, in order to be representative of all states, the sample should include schools from 

all states. Also, in order to measure the value added by CSR over Title I schoolwides, a larger 
54 

comparison group of Title I schoolwides was required. Thus in the second sample, the choice of 

comparison schools was limited to those classified as Title I schoolwides in 2002. Survey data 

from this sample will be included in the next annual report. 

Selection of Comparison Schools 

Many studies show that school-level student performance is influenced by school 

background characteristics such as student socioeconomic status and ethnicity. In fact, within 

each state, by regressing only the percentage of students receiving free and reduced-price lunches 

and the percentage of nonwhite students on a composite of reading and mathematics 

performance, 15-80 percent of the variation in student performance can be explained. In order to 

make a reasonable comparison between the CSR schools and other schools, researchers selected 

comparison schools to have matching background characteristics. 

Researchers used a two-step process to select potential matches for comparison with CSR 

schools in this study. First, the team created a school equivalency index for all schools in each 

state (where data were available) and then calculated a proximity score between each pair of 

schools within a state. Matching schools were selected that had the closest proximity on the 

index to CSR schools within the same district. 

A regression-based approach to weighting and combining background characteristics was 

used to construct the index of school similarity for each state. This method is a simplified version 

of the California School Characteristics Index (Technical Design Group of the Advisory 

Committee for the Public Schools Accountability Act of 1999). Using the National School-Level 

State Assessment Score Database developed by the American Institutes of Research, the team 

regressed measures of student academic performance on measures of schoolwide participation in 

federally subsidized free or reduced-price lunch programs and on schoolwide counts of student 

ethnicity. The estimated coefficients led to a composite of background characteristics for each 

school. In short, each background characteristic was weighted by the amount that it contributed 

to student performance. 

The variety of reporting formats used by states necessitated construction of a separate 

index for each state. For example, some states reported percentile rank while others reported 

percent above cut-points (usually quartiles or proficiency standards). The index reflected 

whatever measure was available, limiting comparisons among schools to within states. States 

also varied in the grades and subjects tested. The constructed indices for elementary, middle, and 

high school levels used scores from the third, seventh, and eleventh grade preferentially. If 

scores for these grades were not available, the closest available grade was used. The most 

consistently reported subject scores across states were results of reading assessments. With few 

exceptions, the reading scores were used to construct indices. The most current set of scores was 

from the 1999-2000 or 2000-01 school years in most states. Exhibit B-1 summarizes the 

performance indicators used for each state, as well as the strength of the association between the 

predictor variables and performance. 
55 

Exhibit B-1 

Summary of Performance Indicators by State 

State Year 

tested 

Subject Type of 

measure* 

Elementary 

R2 

Middle 

R2

High 

R2

Arizona 2000 Reading PR 0.54 0.55 0.42 (10th) 

Arkansas 2000 Total PR 0.49 (5th) - - 

California 2001 Reading PR 0.76 0.53 0.37 

Colorado 2000 Reading CT 0.72 0.82 - 

Delaware 2000 Reading PR - - - 

Florida 2001 Reading CT 0.67 (4th) 0.36 (8th) 0.24 (10th) 

Georgia 2000 Reading PR 0.56 0.41 (8th) - 

Hawaii 1999 Reading CT 0.44 - - 

Iowa - - - - - - 

Idaho 2001 Reading PR - 0.21 0.18 (10th) 

Indiana 2000 Reading PR 0.47 - - 

Kentucky - - - - - - 

Louisiana 2000 Total PR 0.63 0.56 0.55 (9th) 

Massachusetts 2001 Reading CT 0.10 (4th) - - 

Maryland 2000 Reading CT 0.50 0.49 (8th) - 

Maine - - - - - - 

Michigan 2000 Reading CT 0.21 (4th) 0.20 - 

Minnesota 2000 Reading CT 0.28 - - 

Mississippi 2000 Reading PR 0.56 0.58 - 

Missouri 2000 Language CT 0.34 0.34 - 

North Carolina 2000 Reading PR 0.48 0.48 - 

North Dakota - 

New Hampshire 2000 Reading CT 0.09 - - 

Nevada 1998 Reading CT 0.48 (4th) - - 

New York 2001 Reading CT 0.67 (4th) 0.52 (8th) - 

Ohio 2000 Reading CT 0.61 (4th) 0.62 (6th) - 

Oregon - - - - - - 

Pennsylvania 1999 Reading CT 0.69 (5th) - - 

South Carolina 2000 Reading CT 0.57 - - 

Tennessee - - - - - - 

Utah 2000 Reading PR 0.54 0.48 (8th) 0.50 

Virginia 2001 Reading PR 0.58 (4th) - 0.45 (9th) 

Washington 2001 Reading PR - - - 

Wisconsin 2000 Reading PR 0.46 (4th) 0.42 (8th) - 

 *PR—percentile rank, CT—cut 

** R2—for 3rd, 7th and 11th grade scores regressed on percent of students receiving free or reduced-price lunches 

and minority status unless otherwise noted 

After eliminating schools that previously participated in CSR, the team selected potential 

matches for each first-year CSR school based upon a minimum distance criterion. This method 

was used in the majority of cases. However alternative methods were used in two circumstances. 

First, in some districts (or states) either demographic or performance data were not available. 

Second, a suitable comparison school was not available within the same district because the 
56 

district was too small or all other comparable schools had previously participated in CSR. Each 

of these contingencies is outlined below. 

In some cases, not enough data were available to construct a school equivalency index. 

Where states or districts do not report participation in the free or reduced-price lunch program or 

ethnicity, schools were ranked within districts using only achievement scores. Examples of states 

where these data are not available are Tennessee and Washington. Achievement data were 

missing for some schools or districts. This was often the case in high schools where the SAT 

takes the place of district-administered standardized tests. The proximity scores in these cases 

were based on an unweighted composite of the number of students qualifying for free or 

reduced-price lunches and ethnicity. Finally, in cases where neither achievement nor 

demographic data were available, comparison schools were matched by school grade span, size, 

and locale. 

In districts where a comparison school could not be selected, the team searched for a 

suitable comparison in an adjacent district of similar locale. Because the school equivalency 

index included all public schools in the state, the proximity of any school within the state could 

be calculated. The same criteria were used for selection across districts where data were 

available. In cases where data were not available, the team used the same procedures that applied 

to selecting comparison schools within districts. Of the 400 sample schools, 319 comparison 

schools were within the same district and 81 were from a different district. 

Composition of the CSR Sample and Non-CSR Sample 

LACIO focuses on questions of implementation and outcomes, linking school reform 

activities in diverse settings to student achievement. To achieve the goals of the study, 

researchers compared achievement in the universe of CSR schools with a random sample of 

schools receiving CSR funding and a comparable group of non-CSR schools. The baseline 

establishes the representativeness of the sample and the validity of the comparison group. In 

brief, findings indicate the sample was similar to all CSR schools and the comparison schools on 

school level and locale. And, although the universe and sample were similar in level of poverty, 

the comparison schools were less likely to serve the highest poverty students. The universe of 

CSR schools had lower initial achievement than the sample schools, which had lower 

achievement than the comparison schools. This section presents the baseline data and then offers 

some possible explanations for the differences. 

Both CSR and comparison schools included in the sample were similar to the universe of 

CSR schools on level of school and locale. For example, 62 percent of the universe, 67 percent 

of the sample, and 67 percent of comparison schools are elementary schools (Exhibit B-2). 

Further, 46 percent of the universe, 46 percent of the sample, and 45 percent of the comparison 

schools are urban, with similar comparability for rural schools (19 percent, 22 percent, and 22 

percent) (Exhibit B-3). 
57 

Exhibit B-2 

Type of School within All 2002 CSR Schools, CSR Sample, 

and Non-CSR Comparison Sample 

Exhibit reads: Type of school in the CSR sample is representative of 2002 CSR universe 

and type of school in the comparison group is exactly matched to the sample. 

Exhibit B-3 

Location of All 2002 CSR Schools, CSR Sample, and Non-CSR Comparison Sample 

Exhibit reads: Distribution of CSR sample and comparison school by locale are 

representative of CSR universe. 

Differences exist between the CSR universe, the sample schools, and the comparison 

schools with regard to the poverty levels of the students they serve. Although the universe and 

sample of CSR schools serve similar percentages of high-poverty students (78 percent of the 

universe and 75 percent of the sample are schools with over 50 percent of students in poverty), 

the comparison schools are less likely to be high poverty (66 percent have over 50 percent of 

students in poverty) (Exhibit B-4). 

 CSR Universe CSR Sample Comparison 

Elementary 62% 67% 67% 

Middle 20% 15% 15% 

High 14% 15% 15% 

Other 4% 3% 3% 

Total 100% 100% 100% 

 CSR Universe CSR Sample Comparison 

Urban 46% 46% 45% 

Suburban 23% 20% 23% 

Town 13% 12% 10% 

Rural 19% 22% 22% 

Total 100% 100% 100% 
58 

Exhibit B-4 

Percentage of Students in Poverty at 2002 CSR Schools, CSR Sample 

and Non-CSR Comparison Sample 

Exhibit reads: While distributions of the samples and CSR universe are similar, 

comparison schools were less likely to be in the highest poverty quartile. 

In order to determine baseline student performance, the team calculated z-scores for each 

CSR and comparison school. Exhibit B-5 indicates performance at baseline for the universe of 

CSR schools, the sample schools, and the comparisons schools. The CSR sample performed 

better than the universe of CSR schools, and the comparison group performed better than the 

sample at all grade levels in both reading and mathematics. 

Exhibit B-5 

Average School-Level Z-Scores (Math and Reading) 

Exhibit reads: CSR universe schools had lower average baseline performance levels than 

sample and comparison schools in math and reading. 

The characteristics of the comparison schools chosen for the study are similar, but not 

exactly like the sample of CSR schools. Overall, the school level and locale of comparison 

Math CSR Universe CSR Sample Comparison 

Elementary -0.95 -0.72 -0.39 

Middle -0.86 -0.82 -0.38 

High -0.68 -0.51 -0.35 

 

Reading 

Elementary -0.92 -0.77 -0.47 

Middle -0.89 -0.84 -0.40 

High -0.69 -0.52 -0.36 

 CSR Universe CSR Sample Comparison 

75–100% 45 41 37 

50–74% 33 34 29 

24–49% 18 22 26 

0–24% 4 3 8 

Total 100 100 100 
59 

schools were matched closely; however, the preference for choosing comparison schools within 

the districts of the sample schools created pairs that were not ideally matched. In some districts, 

the availability of comparison schools was limited because all other CSR-eligible schools had 

already received funding, removing them from the selection pool, or the district was too small to 

have more than one school at the appropriate level. 

The comparison group for LACIO is composed of schools that have never received CSR 

funding. Their higher initial scores may reflect targeting of CSR funds to schools with greater 

need. Although, matched on prior achievement as well as demographic variables, the pool of 

available comparison schools was limited because over time the pool of potential comparison 

sites has become smaller, leaving schools that exhibit higher (although not very high) 

achievement levels. Interviews of state and district officials will explore the validity of that 

explanation. 

Replacement of Survey Sample and Comparison Schools 

Not every school in the initial sample of 400 CSR and 400 comparison schools agreed to 

participate in the evaluation. When school principals (or their districts) would not consent to 

receive surveys, they were defined as “refusing to participate.” Note that this definition excludes 

schools that received surveys yet failed to return the forms. Members of the second group were 

not dropped from the sample and were treated as nonresponders (see response rate section 

below). 

Two methods were used to replace nonparticipating schools. When comparison schools 

refused to participate, another comparison for the sample school was chosen based on the same 

criteria as the initial selection process. However, if a CSR sample school refused to participate, 

both the CSR school and matching school were removed. The new pair was chosen from the 

same locale, school level, and state to avoid biasing the sample in any systematic manner. 

Selection of Schools for Field-based Study 

With such a small number of schools in the field-based study, the sample is representative 

neither of the geographic distribution nor distribution of school levels in the universe of CSR 

schools. From the larger sample of 400 CSR and comparison school pairs, the team randomly 

selected 25 potential pairs, and narrowed the subsample further by deliberately selecting 15 

schools within the initial 25. The selection was necessary to ensure that the subsample would 

include enough geographic diversity and diversity of school levels to maximize the variety of 

policy environments available for study within the constraints of the design. The remaining list 

of 10 randomly selected schools was retained in the event that a field-based study school or 

district did not agree to be part of the evaluation. 

Researchers took an additional precaution by selecting field-based study CSR and 

comparison pairs that resided in the same district. This precaution was necessary for two reasons. 

Unlike the large-scale sample, where policy comparisons occur over the entire sample, in the 

field-based study the only way to observe the differential effect of district policies is to make 
60 

comparisons within a single district. Also, visiting a single district for each pair minimizes the 

data collection burden for both the evaluators and respondents. 

Selection of District Interview Sample 

The evaluation also includes school districts as part of the large-scale and field-based 

studies. The team will conduct telephone interviews with each of the 15 field-based study 

districts as well as 50 additional districts across the country. From the current sample covering 

34 states, at least one district was selected to participate per state. Districts with CSR schools 

included in the large-scale school survey were pooled within each state. Then random selections 

were made state-by-state from the districts in the pool. 

Administration of Principal and Teacher Surveys 

The survey instrument uses items that are evidence-based and measure behaviors, rather 

than attitudes and expectations. The use of these types of items in the survey represents our 

desire to measure respondents’ actual behaviors rather than attitudes or beliefs. 

Items asking about schools’ actual behaviors and conditions maximize the validity of 

survey data. Such items may be compared with the traditional design of survey items, which 

emphasize subjective ratings and responses from individual respondents’ and are vulnerable to 

merely soliciting “socially desirable” responses. When activity or behavioral items are used—

e.g., “Which academic subjects are covered by goals or benchmarks for student achievement?” 

—in principle, the item is more amenable to external corroboration, and respondents are less 

likely to make a response other than by giving their most accurate one. Although the entire 

survey still consists of self-reported data, the use of such behavioral items increases the quality 

of the data (Fowler 1993). 

The research questions central to the evaluation drove the selection of survey items. 

Because the research questions focus on identifiable actions of schools and their impact as the 

result of CSR program implementation, the survey administered to participants in this evaluation 

is focused strictly on behaviors. In essence, the survey tracks behaviors that result from the 

implementation of CSR rather than attitudes about its implementation. 

Expert Review and Field Testing 

All data collection instruments were reviewed before being used in the field. Instruments 

and procedures were shared with members of the Technical Working Group, who brought their 

expertise as researchers and practitioners to their review of the design of the items, the burden on 

respondents, and the implications for data analysis. In addition to such review, the evaluation 

team pilot tested all data collection instruments, including all interview and observation 

protocols and surveys. During these tests, which were administered to no more than nine 

respondents, the team assessed item comprehension, the effectiveness of the proposed strategies 

for gaining cooperation, and the length of time for respondents to answer questions in the 

instruments. 
61 

Survey Data Collection 

Data collection for the principal and teacher surveys took place in three stages: obtaining 

consent, distributing and collecting surveys, and follow-up. Data collection began with 

contacting the districts to inform them of the evaluation and solicit support. Most districts 

decided to administer the surveys through the superintendent's office or CSR contact person. 

This had both a positive and a negative effect. On the positive side, the district contact lent 

support to the research effort (in some cases, districts attached approval forms or letters of 

encouragement). However, it also increased the time between survey administration and 

response. 

Over the course of the data collection, schools from several districts were eliminated from 

the sample. Some districts refused to participate because of previous commitments to other 

projects. In other districts, target schools were in the process of shutting down or restructuring. 

Schools in these categories (9 sample and 16 comparison) were replaced during the study. 

Surveys were distributed to schools in three waves as consent was obtained from districts 

and schools. Follow-up began one week after mailing surveys in the form of a reminder postcard. 

After two weeks and every week thereafter, schools were contacted by phone until survey forms 

were received. Schools required an average of three to four phone call reminders before 

returning survey forms. 

In some cases, the last wave of mailing coincided with a variety of end-of-school activities. 

In other cases, districts delayed data collection until the start of school in fall 2003. For example, 

New York City officials initially agreed to participate in spring 2002 but then reversed their 

decision because a statewide research effort was underway. District officials agreed to allow us 

to administer the surveys in September. Administration and follow-up with New York City and 

other nonresponding districts (24 sample and 24 comparison schools) were concentrated in 

September. Duerr Research has continued follow-up, and groups of completed surveys continue 

to arrive at WestEd. 

Response Rate 

The data in this report are drawn from the survey forms returned to WestEd before July 30, 

2003. The 239 pairs of schools that have responded so far represent 60 percent of the original 

target of 400 pairs of CSR and comparison schools. Survey forms have come in from 318 of 367 

CSR schools surveyed so far (87 percent response rate), and 279 of 360 non-CSR comparison 

schools (78 percent response rate). However, the response rate is likely to increase as follow-up 

efforts continue, especially in the delayed and replacement schools. To ensure that we receive the 

necessary number of surveys, follow-up efforts will intensify. The evaluators are confident that 

the 88 percent response rate required will be achieved in order to get adequate power for the 

more complex statistical models that will be included in the second annual report of both sample 

and comparison schools (0.88*0.88*400=310). 

The preliminary analyses included in the report will be supplemented by a final analysis 

when data from all possible cases have been entered and processed. Further, the evaluators will 
62 

(once all data collection has been completed) conduct an analysis of nonrespondents to 

determine the extent to which the final sample is biased. This analysis will also appear in the 

second annual report. 

Analysis Methods—Year One 

The analysis methods used in this report are mainly descriptive with simple bivariate 

associations used when necessary. A more complex analysis including an estimation of student 

achievement outcomes will appear in the second annual report. The reason for this delay is the 

lag in availability of student achievement data corresponding to the first year of implementation 

for the 2002 cohort. 

Four sources of data were used to produce the descriptions in this report. The first source is 

the CSR Awards Database maintained by the Southwest Regional Educational Development 

Laboratory (SEDL). Lists of CSR schools are aggregated here from all U.S. states and territories. 

Second, the National Center for Education Statistics Common Core of Data was the source for 

school characteristics for CSR and comparison schools. The third source is the National SchoolLevel Assessment Database maintained by the American Institutes for Research. Results of 

statewide standardized achievement tests are aggregated here. The final source of data for this 

report is the results from the administration of the LACIO Principal and Teacher Surveys. 

Summaries of school characteristics using the SEDL and NCES databases consisted of 

straightforward cross-tabulations. Schools were included in this analysis only if the NCES 

database indicated that the school was operational. 

Student Achievement Data 

Summaries of student achievement data across states were calculated in three steps. First, a 

single grade level was selected to represent the performance of a school. Often this selection was 

made simpler by the availability of only one grade level. Elementary schools were preferentially 

represented by grade 3, but if this grade was not available, grade 4 or grade 5 was selected. 

Grade 7 was used preferentially for middle schools, followed by grade 8. Grade 11 was preferred 

over grade 10 for high school when possible. In some cases, only an aggregate of all grades in a 

school was available to use. 

Second, because scores were reported in three different formats across states, the most 

desirable format for each state was selected. Scaled scores were used, when available, because of 

their ratio properties. Normal curve equivalent (NCE) scores, or percentile ranks converted to 

NCE, were used when scaled scores were not available because they have some interval 

properties. However, about half of the states only reported the percentage above cut scores or 

quartiles. In these cases, the distribution of each cut (often three were reported) was examined in 

order to choose the one that was closest to a normal distribution. The mean value had to be at 

least two standard deviations from the upper or lower limit of the scale. 

In the third step, the values for the selected grade level or scale were converted to z-scores 

and aggregated across states by category. This method was used in the first-year report in the 
63 

interests of simplicity. However, in future reports, student achievement from different tests and 

different reporting formats will be aggregated using meta-analytic techniques. 

Survey Data 

Survey data from teachers and principals was first tabulated for all responses on the forms. 

Because each school was mailed three teacher surveys, an intervening step was performed. 

Before tabulation, the modal response for teachers within each school was determined. This 

aggregate response was used in further analyses. 

Items on both forms were associated to check for validity. Almost all items were 

determined to be statistically significantly at the .01 level using the chi-square test, and phi was 

calculated for each item. In items with polytomous response variables, gamma was calculated to 

determine the magnitude of association. For items with continuous response variables, simple 

correlations were used. The size of the associations ranged from medium to large. Because of the 

large association between the teacher and principal responses, only the principal responses are 

reported here as the measure of school reform. However, on a small number of items minor 

disagreements between principals and teachers existed. On those items, the responses for both 

teachers and principals are both noted. These differences will be explored in subsequent reports. 

Two types of comparisons were performed on survey items. When the responses of CSR 

principals are compared with non-CSR principals, a simple chi-squared test was used. However, 

when the responses of CSR principals from items asking information about last year are 

compared with responses about this year, a McNemar chi-square was used to test the symmetry 

of rows. 
64 
65 

Bibliography 

Borman, G.D., Hewes, G. M., Overman,, L.T., and Brown, S. (2003). “Comprehensive School 

Reform and achievement: A meta-analysis.” Review of Educational Research, 73(2), 125-

230. 

Bryk, A. and Raudenbush, S. (1992). Hierarchical linear models: Applications and data analysis 

method. Thousand Oaks, Calif.: Sage. 

Consortium for Policy Research in Education (1998). CPRE policy briefs. Philadelphia: 

University of Pennsylvania Graduate School of Education. 

Desimone, L. (2002). How can comprehensive school reform models be successfully 

implemented? Review of Educational Research, 72(3), 433-479. 

Finnan, C. (2000). Implementing school-reform models: Why is it so hard for some schools and 

easy for others? Paper presented at the American Educational Research Association Annual 

Meeting, New Orleans. 

Hale, S. (2000). Comprehensive School Reform: Research-based strategies to achieve high 

standards. San Francisco, Calif.: Comprehensive Assistance Center, Region XI, WestEd. 

Kish L. (1995), Survey sampling. New York: John Wiley and Sons. pp.75-77. 

Lee, V., Smith, J., and Croninger, R. (1995) Another look at high school restructuring. In Issues 

in restructuring schools. Madison, Wis.: Center on Organizations and Restructuring of 

Schools. 

Newmann, F., Smith, B.A., Allensworth, E. and Bryk. A.S. (2001). “Instructional program 

coherence: What it is and why it should guide school improvement policy.” Educational 

Evaluation and Policy Analysis, 23(4), 297-321. 

Quenemoen, R.F., Lehr, C.A., Thurlow, M.L., and Massanari, C.B. (2001). “Students with 

disabilities in standards-based assessment and accountability systems: Emerging issues, 

strategies, and recommendations” (Synthesis Report 37). Minneapolis: University of 

Minnesota: National Center on Educational Outcomes. Retrieved June 20, 2001, from the 

World Wide Web: http://education.umn.edu/NCEO/OnlinePubs/Synthesis37.html. 

Sarason, S. (1996). Revisiting the culture of the school and the problem of change. New York: 

Teachers College Press. 

Stringfield, S., Millsap M.A., and Herman, R. (1997). Urban and suburban/rural special 

strategies for educating disadvantaged children: Findings and policy implications of a 

longitudinal study. Excerpted from full text in: CC-VI Forum. Eileen Kaiser, Ed. Volume 2, 

2 Spring/Summer, 1997. 
66 

Stringfield, S., Millsap, M., and Herman, R. (1998). “Using ‘promising programs’ to improve 

educational processes and student outcomes.” In A. Hargreaves, A. Lieberman, M. Fullan, 

and D. Hopkins (eds.), International handbook of educational change (pp. 1314-1338). 

Dordrecht, The Netherlands: Kluwer. 

U.S. Congress (105th Session). Section II: Excerpts from the joint explanatory statement of the 

committee of conference (H.R. No. 390, 105th Congress 1st Session, 32, 38, 96-99, 106-109)

Washington, D.C.: Author. 

U.S Department of Education (1993). Chapter 1 schoolwide project study: Final report. 

Washington, D.C.: author. 

U.S. Department of Education (1996a). Improving America’s schools: A newsletter on issues in 

school reform. October 1996. Washington, D.C.: Author. 

U.S. Department of Education (1996b). Mapping out the national assessment of Title I: The 

interim report. Washington, D.C.: Author. 

U.S. Department of Education (1997). Prospects: Final report on student outcomes. by M. 

Puma, N. Karweit, C. Price, A. Ricciuti, W. Thompson, and M. Vaden-Kiernan. Washington, 

D.C.: U.S. Department of Education. 

U.S. Department of Education (1998). Implementing schoolwide reform programs: An idea 

book on planning. Retrieved June 20, 2001, from the World Wide Web: 

http://www.ed.gov/pubs/Idea_Planning/index.html. 

U.S. Department of Education (1999). Guidance on the Comprehensive School Reform 

Demonstration Program. Washington, D.C.: Author. 

U.S. Department of Education (2000a). CSRD in the field: Final update. Washington, D.C.: 

Author. 

U.S. Department of Education (2000b). Early implementation of the Comprehensive School 

Reform Demonstration Program: Summary report. Washington, D.C.: Author. 

U.S. Department of Education (2001). High standards for all students: A report from the 

national assessment of Title I on progress and challenges since the 1994 reauthorization. 

Washington, D.C.: Author. 

Wong, K. and Meyer, S. (1998). “Title I schoolwide programs: A synthesis of findings from 

recent evaluations.” Educational Evaluation and Policy Analysis, 20, 115-136.