This is some 

POLICY AND PROGRAM STUDIES SERVICE

FINDINGS FROM THE 

FIELD-FOCUSED STUDY OF THE

COMPREHENSIVE SCHOOL REFORM 

DEMONSTRATION PROGRAM

Volume I: FINAL REPORT 

2003 

U.S. DEPARTMENT OF EDUCATION

DOC # 

OFFICE OF THE UNDER SECRETARY
Findings from the Field-Focused Study of the 

Comprehensive School Reform Demonstration Program 

Volume I: Final Report 

2003 

Submitted to: 

Policy and Program Studies Service 

U.S. Department of Education 

400 Maryland Avenue, SW

Room 6W207 

Washington, DC 20202-0498 

Task Order #6 

Contract No. 282-98-0027 

Submitted by: 

COSMOS Corporation 

3 Bethesda Metro Center 

Suite 950 

Bethesda, MD 20814 
Preface

This Final Report presents the findings from the Field-Focused Study, one of several 

components in the National Evaluation of the Comprehensive School Reform

Demonstration (CSRD) Program. The U.S. Department of Education (ED) sponsored the 

study under a task order and contract administered by the U.S. Department of Health and 

Human Services’ Program Support Center (Task Order #6, Contract No. 282-98-0027), to 

COSMOS Corporation. The McKenzie Group, Inc. (TMG), served as a subcontractor to 

COSMOS. In this function, TMG conducted and completed a review of all CSRD-related 

studies supported by the U.S. Department of Education (COSMOS/TMG, 2001). TMG 

staff also comprised about one-third of the site visit teams. 

The Field-Focused Study and the entire task order were directed by Robert K. Yin, 

Ph.D., of COSMOS. Margaret K. Gwaltney, M.B.A., of COSMOS served as the deputy 

project director, and Dawn Kim served as project coordinator. Many other staff members 

from COSMOS and TMG collaborated in the study, which involved four rounds of site 

visits to 18 CSRD schools and the collection and analysis of student achievement data from

these schools. Michelle LaPointe, Ph.D., of the Policy and Program Studies Service, U.S. 

Department of Education, served as the ED project officer for most of the evaluation 

(Kathryn Doherty, Ph.D., was ED’s first project officer). 

The report was mainly prepared by Robert Yin and Dawn Kim and is presented in two 

volumes. Volume I contains the main text. Volume II contains six appendices to Volume

I, including short summaries of the 18 schools that were studied. The summaries, as well 

as data presented in Volume I, are based on 18 in-depth and detailed cases reviewed by 

each school and available separately from COSMOS (see www. cosmoscorp.com). 

Throughout, all schools are identified anonymously. 

ii
iii

EXECUTIVE SUMMARY

The Comprehensive School Reform Demonstration Program. Comprehensive 

school reform calls for a “whole-school” and coordinated approach to improve schools. 

The strategy differs from piecemeal and fragmented efforts that also in the past have 

seemed only to lead to short-lived changes. 

 To stimulate whole-school reform across the country, Congress appropriated funds in 

FY1998 for the U.S. Department of Education (ED) to start the Comprehensive School 

Reform Demonstration (CSRD) Program. ED allocated the funds on a formula basis to 

states, who made awards to support 1,840 mostly Title I schools “in need of substantially 

improving” their performance. Subsequent rounds of annual awards to support additional 

schools have continued through FY2003. 

 In applying for and accepting these funds, schools were expected to implement 9 

components, one of which is an effective, research-based method or strategy. Together, the 

9 components comprise the comprehensive reform aimed at improving student 

achievement: 

CSRD Components

No. Component 

1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 

Effective, research-based methods and strategies 

Comprehensive design with aligned components 

Professional development 

Measurable goals and benchmarks 

Support within the school 

Parental and community involvement 

External technical support and assistance 

Evaluation strategies 

Coordination of resources 

 The awards are for three years and had to be a minimum of $50,000 per year. Schools 

can only receive a single CSRD award. Given these conditions, federal funds therefore are 

intended to serve as “seed money” for whole-school reform. Beyond this period of time, 

schools are to continue reform with their own resources. 
iv

The Field-Focused Study of CSRD. ED commissioned the Field-Focused Study of 

CSRD as one part of its national evaluation of the CSRD program. The study began by 

randomly selecting 18 schools from the first two annual rounds of CSRD awards (1998-99 

to 2000-01, and 1999-00 to 2001-02). Field data were collected through four site visits to 

each school, conducted during 2000-02. 

 The objective of the Field-Focused Study was to provide initial feedback about the 

implementation of the CSRD program. The topics covered by the study and in this report 

include: 

 1) The progress in implementing the 9 CSRD components by the 

 18 schools; 

 2) The role of district and state influences in implementing the program; 

 3) The apparent pathways or strategies for implementing comprehensive 

 school reform; and 

 4) The early signs regarding the potential sustainability of the program at 

 the school level. 

 

Conclusions on CSRD Implementation: A Mixed Pattern 

 The main priority of the Field-Focused Study was to determine the extent and nature 

of CSRD implementation at the 18 schools. Overall, the schools’ implementation of CSRD 

at the 18 schools was uneven. Based on a 47-point instrument reflecting the 9 components 

and devised specifically for the Field-Focused Study, nine schools garnered 80 percent of 

the 47 points and could be labeled as “nearly fully” implementing CSRD. Another six 

schools garnered 65 percent of the 47 points and could be labeled as “partially” 

implementing CSRD. The remaining three schools all scored lower and were judged to be 

“poorly” implementing CSRD. 

 Implementation of the 9 CSRD Components. Implementation also was uneven 

across the 9 components (to streamline the discussion, several related components have 

been paired): 

 Component 1 (see Section 2.1 of the main report for more details): 

 The bulk of the schools’ CSRD resources and attention has been 

 devoted to the implementation of a research-based method. By 

 the time of the final site visit, implementation of such a method 

 was proceeding fully at 9 of the 18 schools, with partial or minimal 

 implementation at the other 9; 
v

 Component 2 (see Section 2.2): 

 Less than half of the 18 schools had reforms with a compre-

 hensive design, reflected by a comprehensive plan combined 

 with staff awareness that CSRD should extend to a school’s 

 entire way of doing business and all its operations—and not 

 simply adding a new function or project or activity; 

 Components 3 and 7 (see Section 2.3.1): 

 Both professional development and external technical support 

 and assistance had largely been devoted to the implementation 

 of the research-based method, not necessarily comprehensive 

 reform more broadly; 

 Components 4 and 8 (see Section 2.3.2): 

 Both the measurable goals and benchmarks and evaluation 

 strategies had been devoted to tracking student performance, 

 not necessarily implementation progress; 

 Components 5 and 6 (see Section 2.3.3): 

 High turnover among staff and students had resulted in transient 

 levels of either support within the school or parental and com- 

 unity involvement; 

 Component 9 (see Section 2.3.4): 

 Most schools were in a position to coordinate or converge re- 

 sources, but resources for sustainability were still uncertain. 

 

District and State Influences. The Field-Focused Study also collected data about 

district and state actions potentially affecting the schools’ CSRD implementation. Some of 

these actions were part of the CSRD administrative procedures, because states implement 

CSRD by having districts apply competitively on behalf of some or all of their schools. In 

the process, both states and districts can support or monitor the schools’ CSRD efforts. 

The study found varying degrees of such support (see Section 3.1.1). 

 More important than these procedures related directly to the administration of CSRD, 

the study uncovered other important state and district policies, not directly related to 

administering CSRD, that nevertheless influenced CSRD implementation (see Section 

3.1.2). Some conditions, such as extremely limited professional resources, had a negative 

influence on CSRD. Other conditions, such as the direct alignment of CSRD designs with 

district improvement plans and state standards, had an extremely positive influence. Other 

conditions reflected the ongoing dynamics of school systems—e.g., districts reducing 

financial support for all external research-based methods, a district allowing a CSRD 
vi

school to become a charter school, and a district deciding to merge two schools that 

happened to be CSRD schools. 

 Strong district or state influence, creating a “vertical” alignment to the school level, 

led alternatively to either complementarity or conflict with CSRD. As examples on the 

complementary side, CSRD provided two schools with resources and a compatible reform 

agenda to respond to their designation as underperforming schools in the state’s 

accountability system. As examples on the conflicting side, the shifting content of state 

assessments led districts to use resources for alternative curricula and professional 

development that were contrary to those involved in a CSRD school’s original plans or 

implementation. In general, these external state and district conditions appear to be highly 

relevant to CSRD implementation. 

Conditions Associated with Successful Implementation and the Role of State and 

District Influences: Three Pathways to Reform 

 The study identified three different sets of conditions, or “pathways,” that appeared to 

be associated with the successful implementation of CSRD (see Section 3.4). The first 

pathway is a component-driven pathway, whereby a school uses the 9 CSRD components 

to guide the development and implementation of a comprehensive reform. The second is a 

method-driven pathway, whereby the school adopts and implements a comprehensive 

research-based method that affects virtually all school operations and whose successful 

implementation substitutes for the need for any independent articulation of the 9 CSRD 

components. (However, many research-based methods focus on specific curricula and are 

not comprehensive.) The third is a vertical-driven pathway, whereby a school articulates 

and pursues the needed comprehensive strategies as a result of state and district 

requirements involving: the setting of standards, use of appropriate assessment tools, and 

required alignment of district- and school-based strategic planning and improvement plans 

to meet state performance standards. 

 No single pathway was considered the “best” or preferred pathway, and no pathway 

was necessarily more immune than the others to such disruptive conditions as: high 

principal turnover rates, limited professional development resources, or planned or 

unplanned school restructuring. 

Sustainability of “Whole-School” Reform: Still Questionable Given Current Fiscal 

Climate

 As a final topic, the study examined the prospects for sustaining school reform beyond 

the final year of CSRD funding (see Section 3.5). 
vii

 Neither the original legislation nor ED defined the exact nature of a school’s changes 

to be associated with sustaining a comprehensively reforming school beyond the three-year 

CSRD award period. As a result, the Field-Focused Study examined two different views of 

sustainability and judged the 18 schools according to both. 

 The first view, based mainly on the experiences of the New American Schools 

initiative, holds that the central changes to be sustained should be the practices associated 

with the originally-supported research-based method. The second view is that 

comprehensive reform, though embracing a research-based method, also transcends it. By 

this second view, successful sustainability would not necessarily be associated with the 

continued use of any particular method but could involve transitions from one researchbased method to another, over time. The transitions would have to reflect a progression 

toward continued school and student improvement rather than the “churning” of innovative 

practices. 

 Using the most lenient benchmark and accepting either of these two views as a 

criterion for assessing sustainability, 14 of the 18 schools were exhibiting a promising level 

of sustainability by the time their CSRD awards were ending. Accepting only the second 

view reduces the number to 11, and accepting the first view reduces it to 10. 

 The main barrier to sustainability continues to be the limited availability of sufficient 

resources, especially in light of states’ and districts’ revenue shortfalls in recent years. To 

sustain a reforming process, even when existing resources have been coordinated and 

targeted to reform, still requires discretionary funds to support such essential activities as: 

adequate professional development (including support for teacher substitutes), especially in 

situations of high teacher turnover; time for common planning periods or teachers’ work on 

school leadership teams; and support for external technical assistance. Though such needs 

can be served with modest levels of funds, serving the needs is still a discretionary activity 

that may have to be ignored if core school operations are underbudgeted. 

 
viii

Contents 

Page 

Volume I 

Preface.................................................................................................................................... ii 

Executive Summary .............................................................................................................. iii 

Sections 

 1. The Comprehensive School Reform Demonstration Program 

and the Field-Focused Study...................................................................... 1-1 

 1.1 Comprehensive School Reform................................................................... 1-1 

 1.2 The Field-Focused Study............................................................................. 1-8 

 1.3 Sample of Schools Being Studied ............................................................. 1-13 

 1.4 Study Activities and a General Caveat...................................................... 1-14 

 2. CSRD Implementation: The 9 Components ............................................................ 2-1 

 2.1 Implementing Research-Based Methods..................................................... 2-1 

 2.2 Comprehensiveness of Designs for School Reform.................................... 2-9 

 2.3 Progress in Implementing the 7 Other CSRD Components...................... 2-13 

 2.4 Summary ................................................................................................... 2-30 

 3. CSRD Implementation and Sustainability: The Integrated Findings......................... 3-1 

 3.1 District and State Influences on CSRD ....................................................... 3-1 

 3.2 Schools Implementing Comprehensive School Reform ............................. 3-5 

3.3 Three Examples of Comprehensive School Reform 

 under the CSRD Program..................................................................... 3-11 

3.4 Three Pathways for Implementing Comprehensive School Reform......... 3-12 

3.5 Sustaining Comprehensive School Reform............................................... 3-14 

 4. Conclusions from the Field-Focused Study ............................................................... 4-1 

 5. References................................................................................................................. 5-1 
ix

Exhibits

1-1 Description of Nine Components of Reform in 

 Congressional Provisions for CSRD.............................................................. 1-2 

1-2 Summary of CSRD Awards................................................................................ 1-5 

1-3 Theoretical Framework for a CSRD School..................................................... 1-10 

1-4 Beginning to Customize the Original Framework: 

 Two Different CSRD Models ...................................................................... 1-11 

1-5 Location of the Final Sample of CSRD Schools for 

 Inclusion in the Field-Focused Study........................................................... 1-14 

2-1 Research-Based Methods Adopted by 18 CSRD Schools and 

 Fidelity Assessments Obtained from Developers .......................................... 2-2 

2-2 Implementation Status of Research-Based Methods (Component 1)................. 2-4 

2-3 Three CSRD Schools with Minimal or Stalled Implementation of 

 the Originally-Adopted Research-Based Method ........................................... 2-5 

2-4 Four Locally Developed Models ........................................................................ 2-8 

2-5 Evidence of Comprehensive Design (Component 2) ....................................... 2-11 

2-6 Professional Development Activities (Component 3) ...................................... 2-14 

2-7 External Technical Support and Assistance (Component 7) ............................ 2-18 

2-8 Goals and Benchmarks (Component 4) and 

 Evaluation Strategies (Component 8) .......................................................... 2-22 

2-9 Status of Coordination of Resources (Component 9) and Sustainability ......... 2-27 

3-1 School, District, and State Conditions................................................................ 3-2 

3-2 Scores for CSRD Implementation Strength at 18 Schools, 

 Using 47-Point Instrument .............................................................................. 3-7 

3-3 Comparison of Rankings for 18 Schools, Based on 

 Two Independent Assessments of Strength of CSRD Implementation ....... 3-10 

3-4 Sustainability of CSRD at the 18 Schools ........................................................ 3-16 
x

Volume II – Appendices 

A. Evaluation Methodology .................................................................................... A-1 

B. Promising Reform Strategies............................................................................... B-1 

C. Descriptions of the Research-Based Methods or Strategies Being Used............ C-1 

D. Short Summaries of 18 CSRD Schools............................................................... D-1 

E. 47-Point Instrument for Assessing Strength of 

 CSRD Implementation at 18 Schools ............................................................E-1 

F. Corroboratory Assessment of CSRD Implementation for 

 18 Schools in Field-Focused Study................................................................F-1
1-1

 

1. THE COMPREHENSIVE SCHOOL REFORM 

DEMONSTRATION PROGRAM1 AND 

THE FIELD-FOCUSED STUDY 

1.1 COMPREHENSIVE SCHOOL REFORM

The Comprehensive School Reform Demonstration Program (CSRD) was designed 

to stimulate schoolwide change. The Comprehensive School Reform Demonstration 

(CSRD) Program provided formula grants to states. The states in turn invited districts to 

compete for subgrants, on behalf of schools, to implement comprehensive school reform. 

The program’s goal was to improve student achievement and enable all children to “meet 

challenging state content standards and performance goals” through reform strategies (U.S. 

Congress, 1998).2

 Participating schools were expected to: 

• Implement CSRD’s nine components3

 (see Exhibit 1-1), one of 

which was the implementation of an effective, research-based 

method or strategy;4

• Stimulate schoolwide change covering virtually all aspects of 

school operations, rather than implement a piecemeal, 

fragmented approach to reform; and 

1

The Comprehensive School Reform Demonstration (CSRD) Program is the name of the education 

reform grant program as it was created in 1998. The program was subsequently renamed the Comprehensive 

School Reform (CSR) Program in P.L. 107-110, the No Child Left Behind Act of 2001 signed by the President 

in January 2002. Because the Field-Focused Study was commissioned to study the first two cohorts of CSRD 

schools (1998-1999 and 1999-2000 grantees), this final report will refer to the program as CSRD.

2

The program was initially established through appropriations language, not authorizing legislation 

(Appropriations Act for the U.S. Department of Education, P.L. 105-78). In subsequent years, this 

appropriations language continued to be used to authorize the program, until the enactment of P.L. 107-110.

3

The CSRD program called for the implementation of nine reform components (referred to throughout 

this report). The redesigned CSR program under P.L. 107-110 now calls for eleven components. The two 

new components cover “support to the school’s staff” and the use of a research-based method (based on 

comparison group designs or on other strong evidence of improved student achievement).

4

The original congressional legislation listed 17 specific methods as examples, although schools were 

not limited to adopting these methods.
1-2

Exhibit 1-1

DESCRIPTION OF NINE COMPONENTS OF REFORM 

IN CONGRESSIONAL PROVISIONS FOR CSRD*

1. Effective, research-based methods and strategies: Employ innovative strategies and 

proven methods for student learning, teaching, and school management that are based on reliable 

research and effective practices, and have been replicated successfully in schools with diverse 

characteristics. 

 2. Comprehensive design with aligned components: Have a comprehensive design for 

effective school functioning, including instruction, assessment, classroom management, 

professional development, parental involvement, and management—that aligns the school’s 

curriculum, technology, professional development, etc. into a schoolwide reform plan designed to 

enable all students [including children from low-income families, children with limited English 

proficiency, and children with disabilities]** to meet challenging state content and performance 

standards and addresses needs identified through a school needs assessment. 

 3. Professional development: Provide high-quality and continuous teacher and staff 

professional development and training. 

 4. Measurable goals and benchmarks: Have measurable goals for student performance 

[tied to the State’s challenging content and student performance standards, as those standards are 

implemented]** and benchmarks for meeting those goals. 

 5. Support within the school: Are supported by school faculty, administrators, and staff. 

 6. Parental and community involvement: Provide for the meaningful involvement of 

parents and the local community in planning and implementing school improvement activities. 

 7. External technical support and assistance: Utilize high-quality external technical 

support and assistance from a comprehensive school reform entity (which may be a university) with 

experience or expertise in school-wide reform and improvement. 

 8. Evaluation strategies: Include a plan for the evaluation of the implementation of school 

reform and the student results achieved. 

 9. Coordination of resources: Identify how other resources (federal/state/local/private) 

available to the school will be utilized to coordinate services to support and sustain the school 

reform effort. 

 *Conferees’ original language in House Report No. 105-309, accompanying P.L. 105-78. 

**Language in brackets was added to the conferees’ language for the original “Guidance on the 

Comprehensive School Reform Demonstration Program,” issued by the US Department of Education on 

March 13, 1998; the language is present in the subsequent update dated August 11, 2000. 
1-3

 

• Assess the impact from implementing comprehensive 

 school reform in terms of changes in student achievement. 

For most of the CSRD funds, the eligible K-12 schools were Title I eligible schools, 

particularly those schools in need of substantially improving their student achievement 

levels. A small portion of the funds was available to all schools in need of such 

improvement. As a result, Title I schools represented the great bulk (over 85 percent) of 

CSRD schools. 

CSRD funding was disbursed to nearly 4,000 schools by FY 2001. CSRD funds first 

became available in FY1998, when $145 million in federal funds were allocated to state 

education agencies (SEAs).5

 The SEAs then made competitive three-year subgrants to 

their districts, eventually supporting 1,840 schools with the first year’s funds.6

 The awards 

(a minimum of $50,000 per year) were intended to stimulate or provide financial incentives 

for reform, but not support the entirety of reform.7

 In FY1999, $145 million in funds were again allocated to the states, supporting the 

continuation of the initial awards and also the startups of a small number of new awards. In 

FY2000, the program awarded $220 million to the states, anticipating the continuation of 

the original schools into their third year but also providing the opportunity to make awards 

to a new set of schools. Similarly, $260 million was provided in FY2001, which permitted 

the funding of an entirely new set of schools starting in 2001–02, as well as the 

continuation of those schools begun with the FY2000 increase. The total number of 

schools funded under the CSRD program reached 3,875 by the 2001–2002 academic year 

(SEDL, 2002). For 2002-03, the appropriation increased funding for the newly defined 

5

Funds appropriated under the Title I program are allocated to States based on each State’s share of Title I 

Basic Grants in the previous fiscal year and may be used to support schools eligible to receive funds under Part 

A of Title I. Funds under FIE are allocated to States based on each State’s share of school-aged children, and 

may be used to support any school, including those not eligible to receive funds under Part A of Title I. Under 

both the Title I and FIE CSRD programs, ED was able to reserve up to 1 percent of the total amount 

appropriated for grants to outlying areas and the BIA, and up to 1 percent for national evaluation activities.

6

A study of 40 responding states (representing $103 million of the original $145 million in federal 

funds awards in FY1998) revealed that 3,001 school applications had been received from 1,088 school 

districts and that the average CSRD award had been $68,000 per school per year (New American Schools, 

1999). 

7

The states do not necessarily make all of their subawards in the same academic year, and states also 

are permitted to set aside up to five percent of the funds for state administration, evaluation, and technical 

assistance. States also have the option to set aside award money for use by districts, and this may or may not 

be reflected in calculations of the average school award.
1-4

 

CSR program to $310 million, enough to support grant awards to an estimated 2,000 

additional schools (see Exhibit 1-2). 

 CSRD called for research-based methods and strategies. The CSRD program drew 

its rationale and design from a longstanding and broad research and policy literature.8

 For 

instance, from a research perspective, two contemporary analysts (Desimone, 2000, p. 36; 

and Doherty, 2000, p. 3) noted that the theoretical underpinnings for comprehensive school 

reform were similar to and extended as far back as research on effective schools in the 

1970s and 1980s (e.g., Edmonds, 1979; and Cohen, 1982), among other educational 

strands. 

 From a policy perspective, the practical underpinnings for CSRD emanated from the 

events leading to the redesign of Title I of the Elementary and Secondary School Act as 

amended by the Improving America’s Schools Act (IASA) of 1994, which shifted the 

federal government’s compensatory education investments away from supplemental and 

remedial (pull-out) programs toward standards-based, whole school reform—as reflected 

by a renewed appreciation for schoolwide programs and local flexibility (Shelley, 1997; 

Berends and Kirby, 1999; Puma and Drury, 2000; and Borman, 2000). As one result of the 

policy shift, the number of schoolwide programs within Title I increased from 1,200 in 

1991 to 9,000 in 1997-98 and represented 50 percent of all Title I schools (Desimone, 

2000).9

 Coinciding with these research and policy developments, a critical catalytic influence 

was the ongoing experience of the New American Schools (NAS) initiative (e.g., Bodilly, 

1998; Berends, Bodilly, and Kirby, 2002). The New American Schools Corporation was 

formed in 1991 to design innovative, ‘break-the-mold’ schools. NAS’s initial wave 

supported eight research-based methods,10 which were then promoted by over 1,500 

schools in 11 partner communities (New American Schools, 1999). The initiative came at 

a time when the supply of research-based methods converged with the demand for 

continued school reform among low-performing schools (Bodilly, 2001). Early lessons 

from the initiative included a shift away from an emphasis on method development-designonly to include external technical assistance and support to schools (NAS, 1999), as well as 

8

The present report, however, is not the occasion for a comprehensive review of this literature.

9

Throughout these policy transitions, an important undercurrent continued to be that “teachers matter” 

(Puma and Drury, 2000). Operationally, the assumption has been that changed vision and school-based 

planning will lead to changed classroom practices. All these practices will then produce the desired change 

(and improvement) in student performance. That “teachers matter” therefore suggests that policymakers need 

to focus on classrooms—the “operational core of education systems” (Puma and Drury, 2000, p. 33).

10Six of these eight methods were listed among the CSRD legislation’s original 17 methods. For a 

crosswalk between both groups and the top 25 methods or practices adopted by the first cohort of CSRD 

schools, see COSMOS, May 2000, Exhibit 1.
1-5

Exhibit 1-2 

SUMMARY OF CSRD AWARDS

 

Federal Fiscal Year 

Appropriation 

Amount Allocated 

(In Millions)

Cohort: Number of Schoolsa School Year First 

Implemented

1998 $145b Cohort 1: 1,840 1998-1999 

1999 $145b Cohort 1: 1,840 

Cohort 2: 146 

1999-2000 

2000 $220c Cohort 1: 1,856 

Cohort 2: 146 

Cohort 3: 796 

2000-2001 

2001 $260c Cohort 2: 146 

Cohort 3: 796 

Cohort 4: 1093 

2001-2002 

2002 

(No Child Left Behind)

$310d Cohort 3: 796 

Cohort 4: 1093 

Cohort 5: 2,000 (est.) 

2002-2003 

a

 Amounts reported by States for each fiscal year do not always add to the amount allocated to States in 

that fiscal year (do to inconsistencies in State reporting). 

 b

 Includes $25 million provided under the Fund for the Improvement of Education (FIE). 

 s

 Includes $50 million provided under the Fund for the Improvement of Education (FIE). 

 d These are the appropriated funds (including $75 million in FIE funds) and the estimated number of 

schools the funds will support. 

Sources: Comprehensive School Reform Program: State Allocations for all Fiscal Years, U.S. Department of 

 Education, June 25, 2002, www.ed.gov/offices/OESE/compreform/allyears.html. 

 CSR Awards Database, Southwest Educational Development Laboratory, August 2002 (Note: The 

number of schools receiving awards was taken from the SEDL database, which was undergoing 

revisions at the time of this final report; the final number of schools awarded funds may have 

changed).

an increased appreciation for the role of the district in supporting or facilitating 

comprehensive school reform (NAS, 1999). 

 Emerging studies of comprehensive school reform have shown mixed outcomes. 

Some early studies suggested that the outcomes can be promising, as reflected by syntheses 

of these studies (Desimone, 2000; and Datnow and Stringfield, 2000). However, other 

studies have found no significant effects of comprehensive school reform on student 

achievement (e.g., Berends et al., 2002). The completed research includes a variety of 
1-6

 

evaluations of specific research-based methods (e.g., Slavin and Madden, 2000),11 analyses 

of student achievement outcomes in entire school systems (e.g., Ross et al., 1999), and 

policy studies of strategies for educating disadvantaged children (e.g., Stringfield et al., 

1997). In addition, the U.S. Department of Education funded 46 studies,12 whose progress 

was tracked and documented to encourage coordination among the studies (COSMOS/ 

TMG, 2001), including the several components of the National Evaluation of CSRD. Yet 

other studies and related activities are being tracked by the National Clearinghouse for 

Comprehensive School Reform (NCCSR). 

 Earlier research also includes individual case studies of CSRD-like comprehensive 

school reform, such as five case studies commissioned by the Consortium for Policy 

Research in Education—CPRE (Odden, Archibald, and Tychsen, 1999a and 1999b; 

Archibald and Odden, 2000; Odden and Archibald, 2000; and Fermanich, Odden, and 

Archibald, 2000). Another multiple-case study covered nine elementary schools engaged 

in comprehensive efforts similar to the CSRD program (The Dana Center, 1999). Among 

other findings, the cross-case analysis suggested that reform was more likely to succeed 

when efforts focused on boosting the schools’ instructional leadership (e.g., the capacity of 

principals and the channeling of resources, including technical assistance, to enhance 

instructional leadership), and when these efforts were combined with clear, measurable, 

and rigorous school accountability provisions. 

 Overall, the available and ongoing research demonstrates that the CSRD program 

emerged against a solid backdrop of prior conceptual work and empirical evidence. At the 

same time, the experiences have raised at least two ongoing issues worth mentioning.13 

First, “CSRD” is not synonymous with “comprehensive school reform.” As used in the 

present report, CSRD has been defined specifically to represent the congressionally 

legislated program that started in 1998—its major distinction being the identification of 

nine specific components to be implemented as part of comprehensive reform. In contrast, 

11One criticism is that very few of the research-based methods “...have been the subject of high quality 

evaluations that justify confidence in their effectiveness,” and that in addition, “...the few models that do have 

a substantial amount of research...are models for which evaluations were conducted mainly by the design 

team, not objective third-party evaluators” (Desimone, 2000, p. 37).

12Most of the studies are independent studies, totally unrelated to the National Evaluation of the CSRD 

Program. Further, many of the studies are on “comprehensive school reform,” which may have been put into 

place quite independent of the CSRD program. In this sense, the bulk of the studies should not be considered 

evaluations of the CSRD program. For instance, eight of the studies are evaluations of individual researchbased methods. Many of the studies are being conducted by the U.S. Department of Education’s (ED) 

Regional Educational Laboratories and supported by ED’s Office of Educational Research and Improvement 

(OERI) or Policy and Program Studies Service. 

13Unfortunately, neither of these issues has been given sufficient attention in the existing literature (e.g., 

Datnow and Stringfield, 2000).
1-7

“comprehensive school reform” does not carry the specific connotation that there are nine 

components—although the principles of comprehensive school reform are compatible with 

and may even be comparable in some instances to CSRD’s nine components. 

 Second, the reform heritage just traced has been heavily dominated by the existence of 

research-based “models.” A resulting confusion is that the desired reform (and especially 

its comprehensiveness) has been sometimes assumed to be no more than the adoption and 

implementation of an externally-developed, research-based “model.” However, in reality 

the required reform effort must go beyond the “model,” because most of the “models” are 

not comprehensive. They focus on either a specific subject matter (e.g., reading) or a 

particular phase of instruction (e.g., homework), or both; but a school needs to implement a 

reform that embraces the entirety of its operations, covering all grades, all subjects, and all 

students in the school. To this extent, the breadth of successful reform will go well beyond 

the adoption and implementation of most of the available research-based “models.” 

 A final clarification of comprehensive school reform also derives from the ongoing 

NAS experience. Originally, NAS considered “break the mold designs” and “design 

teams”—and ultimately “externally developed reform design”—to be the desired agents of 

change. The original concept of “design,” as used by NAS, may indeed have been 

sufficient to capture the breadth of comprehensive reform. “Design” implies an overall 

plan for the comprehensive reform of an entire school, incorporating all aspects of the 

curriculum, school operation, and instruction. However, many of the research-based 

“models” that later emerged were then assumed to be comprehensive, especially because 

many had the characteristic of being externally developed (e.g., CPRE, 1998). In fact, 

many of the “models” were limited to specific grades or academic subjects and alone could 

not support comprehensive reform unless they were integrated into a comprehensive design 

or plan. 

 

 As a result, the present report considers a research-based “model” to be a way of 

dealing with the first of the nine components of the comprehensive school reform 

demonstration (CSRD) program— implementing research-based methods and strategies. 

Further, to avoid perpetuating the use of the word “model” as necessarily implying 

comprehensive reform, this report uses the language of the legislation, or research-based 

“method” or “strategies,” including when referring to such “models” that are externallydeveloped (such as Success for All or Accelerated Schools). Thus, externally-developed 

methods can lead to comprehensive reform, but only if the method requires a school to 

attend thoroughly the other eight CSRD components. However, if the method does not 

lead in a comprehensive direction, a CSRD school must undertake additional initiatives to 

cover the other eight components necessary for the school to meet its CSRD mandate. 
1-8

 

1.2 THE FIELD-FOCUSED STUDY

 The present report covers the Field-Focused Study. It is one part of the National 

Evaluation of the CSRD Program, which involved multiple related efforts being conducted 

separately by different research organizations (Doherty, 2000).14 The study called for data 

collection from a stratified, random sample of 18 schools from the first two cohorts of 

CSRD awardees. The data collection was based on intensive, field-based efforts: two 

visits per school year, for each of two years. Appendix A contains the detailed design and 

methodology for the study. 

 The Field-Focused Study was mainly aimed at describing the links among school 

reform processes, instruction, and other educational activities. The study involved in-depth 

examination of these linkages, thereby limiting the number of schools that could be the 

subject of inquiry. In addition, the schools in the Field-Focused Study also were examined 

as a source of promising reform strategies (i.e., practices, programs, and technical 

assistance or support strategies) to share with other schools (see Chapter 2). The topics 

covered by the study and in this report include: 

 1) The progress in implementing the 9 CSRD components by the 

 18 schools; 

 2) The role of district and state influences in implementing the program; 

 3) The apparent pathways or strategies for implementing comprehensive 

school reform; and 

 4) The early signs regarding the potential sustainability of the program at 

the school level. 

 Because the Field-Focused Study directly investigated the complexity of events 

(including contextual conditions) involved with the nine CSRD components as they might 

occur at an individual school, the study used the case study method as the primary research 

14For instance, the Southwest Educational Development Laboratory (SEDL) compiled baseline 

information on all CSRD schools. SEDL’s database provided the sampling frame for selecting the sample of 

schools for the Field-Focused Study, to assure that the selected schools represented the entire universe. A 

second project was a large-scale sample survey of CSRD schools conducted in 1998-99, 1999-00, and 2000-

01 by Westat. The survey was included as part of the broader National Longitudinal Survey of Schools 

(NLSS), also designed by Westat, that followed a longitudinal sample of Title I schools over that same time 

period. Principals and up to six teachers were surveyed in the CSRD sample of schools as well as the NLSS 

sample, and comparable data were collected from both sets of schools. The 1998-99 sample of CSRD schools 

consisted of 318 schools. In 1999-00, the CSRD sample was augmented by 725 schools. The data on the 

CSRD and NLSS schools were analyzed by RAND.
1-9

 

method (Yin, 2003).15 To tighten the analytic process, the application of the method to the 

Field-Focused Study included four important features not always found in other case study 

research, as follows. 

A theoretical framework guided the study. The first feature was the use of a 

theoretical framework—i.e., “logic model”—whose development began prior to data 

collection. The evaluation team developed a logic model to depict the working of CSRD at 

any given school (see Exhibit 1-3). The model portrays the 9 components of CSRD as 

boxes in the exhibit, with CSRD Component 4 (“Measurable Goals and Benchmarks”) 

being split into two boxes (4a and 4b). The generic model also contains two important 

concepts not explicitly part of the original 9 CSRD components but that are necessarily 

involved in attempts to improve student performance: school organization (located as part 

of box 4a) and instruction and learning (see the unnumbered box).16 To preserve space, 

the model does not show the contextual conditions—e.g., district and state policies 

affecting the school—although these conditions also have been part of the Field-Focused 

Study. 

 The model, as with all logic models, shows all of the boxes, connected by arrows, that 

hypothesizes how comprehensive school reform might produce change in student 

performance (see Box 4b). The dotted arrows are hypothetically related to the researchbased method defined as CSRD Component 1. The solid arrows are hypothetical but 

nevertheless serve as logical links suggesting how other school actions can lead to effects, 

which in turn become actions for other effects—all part of a chain of events. Key to 

interpreting and using the overall model, therefore, is that data and findings are needed 

about the “arrows,”17 and not just the “boxes” in the model—that is, “how” and “why” 

15Investigators assessing a directly comparable initiative—the scale-up phase of the New American 

Schools (NAS)—arrived at the same methodological conclusion, even though NAS funding levels were 

higher than CSRD funding. The investigators argued that the interventions and research questions called for 

“...a replicated case-study approach, with the unit of analysis being the implemented school” (Bodilly, 1998, 

p. 25). [Emphasis added.]

16Any reform undertaking should be proceeded by a needs assessment; and, in fact, some of the 

externally developed methods called for the conduct of a needs assessment prior to adoption. Without a 

needs assessment, it is unlikely that a school would be able to properly design a comprehensive reform, or 

know which method to select (Datnow and Stringfield, 2000). The Field-Focused Study did not address the 

topic of needs assessments because it was not one of the 9 CSRD components. However, the evaluation team 

subscribes to the research that needs assessments are a necessary step in reform efforts, and believes this topic 

should be addressed in future research.

17For illustrative purposes, the arrows have been shown in a linear sequence that does not appreciate the 

full dynamics of events in real life. However, such dynamics were not ignored in the data collection for the 

Field-Focused Study.
1-10

Exhibit 1-3 

THEORETICAL FRAMEWORK FOR A CSRD SCHOOL 

Instruction and learning

* = The numbered boxes refer to the nine CSRD components.

2. Comprehensive and 

aligned design

6. Parental and 

community involvement

7. External support and 

assistance

3. Professional development

9. Coordination of 

resources

5. Support by 

faculty, etc.

8. Evaluation 

School Organization: 

4a. Measurable goals and 

benchmarks

4b. Student 

performance

1.* Research -

based method 

or strategy

Instruction and learning

* = The numbered boxes refer to the nine CSRD components.

2. Comprehensive and 

aligned design

6. Parental and 

community involvement

7. External support and 

assistance

3. Professional development

9. Coordination of 

resources

5. Support by 

faculty, etc.

8. Evaluation 

School Organization: 

4a. Measurable goals and 

benchmarks

4b. Student 

performance

1.* Research -

based method 

or strategy

events in one box affected those in another, not just the identity and presence or absence of 

the events within each box. 

In the Field-Focused Study, data collection teams were asked to customize the generic 

logic model into a specific but tentative model for their school(s) prior to actual data 

collection. The details of the logic model were likely to vary, depending upon the nature of 

the research-based method that the school had adopted to satisfy CSRD Component 1, as 

well as other conditions that might have been revealed during a site-screening phase that 

led to the selection of the school for inclusion in the Field-Focused Study in the first place. 

The stipulated logic model then became the occasion for field inquiry—appreciating that 

“discovery” of unexpected events and relationships also could occur as part of the data 

collection procedure. 

The evaluation team could then customize the generic logic model, depending upon 

the CSRD Component 1 research-based method or practice that had been adopted (Success 

for All and America’s Choice in these examples) (see Exhibit 1-4). The labels in the boxes 

reflect the activities emphasized by each method, and the arrows suggest the connections 

implicit in each program. Important in these two logic models are therefore the boxes with 
1-11

Exhibit 1-4 

BEGINNING TO CUSTOMIZE THE ORIGINAL FRAMEWORK: 

TWO DIFFERENT CSRD MODELS 

1-4a: Success for All

* = The numbered boxes refer to the nine CSRD components.

9. Local, state, and federal

funds need to replace

reading program

2. Comprehensive and

aligned design

7. SFA provides external advisor 

who visits school regularly

6. Family Support Team:

Parents engaged in reading

for 20 minutes each day;

advisory committee

Prescribed

instructional 

materials

3. Ongoing 

coaching support

for teachers

8. Existing state and

local assessments

evaluate progress

4a. Redistribute time and

schedule so each child

has 90-minute block for

reading instruction

4b. Student

performance

in reading

1.* Success 

for All:

Reading

1-4b: America’s Choice

9. Requires coordination

of resources

not addressed specifically

2. Comprehensive and

aligned design

7. National center and regional

coordinators provide training and support

6. Parental and

community

involvement

Core model is

standards-driven

instruction and

learning

3. Extensive attention

to professional

development

5. Substantial

majority of faculty

must support model

8. Uses new standards 

reference examinations

and state assessment

results

4a. May require some

adaptation, but not a key

part of the model

4b. Improved

student

performance is

core

1.* America’s 

Choice:

Comprehensive in 

areas of instruction

5. 80% of faculty vote

by secret ballot to

adopt the model

1-4a: Success for All

* = The numbered boxes refer to the nine CSRD components.

9. Local, state, and federal

funds need to replace

reading program

2. Comprehensive and

aligned design

7. SFA provides external advisor 

who visits school regularly

6. Family Support Team:

Parents engaged in reading

for 20 minutes each day;

advisory committee

6. Family Support Team:

Parents engaged in reading

for 20 minutes each day;

advisory committee

Prescribed

instructional 

materials

Prescribed

instructional 

materials

3. Ongoing 

coaching support

for teachers

3. Ongoing 

coaching support

for teachers

8. Existing state and

local assessments

evaluate progress

4a. Redistribute time and

schedule so each child

has 90-minute block for

reading instruction

4b. Student

performance

in reading

4b. Student

performance

in reading

1.* Success 

for All:

Reading

1.* Success 

for All:

Reading

1-4b: America’s Choice

9. Requires coordination

of resources

not addressed specifically

2. Comprehensive and

aligned design

7. National center and regional

coordinators provide training and support

6. Parental and

community

involvement

Core model is

standards-driven

instruction and

learning

3. Extensive attention

to professional

development

3. Extensive attention

to professional

development

5. Substantial

majority of faculty

must support model

5. Substantial

majority of faculty

must support model

8. Uses new standards 

reference examinations

and state assessment

results

4a. May require some

adaptation, but not a key

part of the model

4b. Improved

student

performance is

core

1.* America’s 

Choice:

Comprehensive in 

areas of instruction

5. 80% of faculty vote

by secret ballot to

adopt the model

5. 80% of faculty vote

by secret ballot to

adopt the model
1-12

 

no connecting arrows—implying that the adopted method does not, on the surface, cover 

all of the CSRD components. To achieve comprehensive school reform, a school must 

therefore find another way of addressing the non-integrated component(s). For example, 

Success for All’s defining feature is that it only covers reading—even though it covers 

reading comprehensively; thus, the program in and of itself does not connect with CSRD 

Component 2 (“Comprehensive and Aligned Design”); America’s Choice’s attention to 

instruction omits any explicit connection to CSRD Component 6 (“Parental and 

Community Involvement”) and Component 9 (“Coordination of Resources”). 

 Overall, careful use of logic models strongly enhances the precision and logic of case 

study research, especially in evaluation studies. 

Rival explanations were investigated. A second feature was the explicit identification 

and then collection of data about rival explanations (Yin, 2000). In the case of the FieldFocused Study, the rivals were alternative explanations for any of the observed outcomes—

e.g., whether conditions other than the CSRD funding and mission could account for the 

observed student performance outcomes (positive, negative, or neutral) or even for a 

school’s reform-oriented implementation actions.18 The use of rivals is aimed at reducing 

the uncertainty about potential relationships, even though the reduction will not nearly be 

as large as that occurring in a “true experiment.” Collecting data about possible rival 

explanations was therefore part of the case studies of the CSRD schools. Such data 

collection strengthened the case studies by broadening the data to be collected but keeping 

the entire effort well-targeted. 

Chronological timelines traced key events. A third feature was the explicit use of 

chronological timelines to trace the key events in the schools being studied. The temporal 

sequence of events was examined for clues regarding the strength of CSRD or rival 

explanations. The Field-Focused Study therefore attempted to collect data faithfully on the 

timing of various events. 

Classrooms were observed. A fourth and final feature involved data collection aimed 

at revealing possible differences in implementation patterns across different classrooms 

within each school. If reform actions were incrementally implemented or affected one 

curriculum topic but not another, the targeted classrooms should have shown earlier 

evidence of implementation than the nontargeted classrooms. To put this feature into 

place, the field teams used a formal classroom observation instrument that was part of the 

case study protocol, described next. 

18A common misinterpretation in using the rivals method is that the rival is perceived to be that “no 

positive outcome occurred.” Key to avoiding this misinterpretation is that the method calls for defining rival 

explanations (or hypotheses), not rival outcomes. 
1-13

 

Site visit teams used a case study protocol. The main instrument for the FieldFocused Study was a case study protocol, contained as an attachment to Appendix A, 

which also gives fuller details on the methods used in the entire study. This protocol was 

the subject of ED and OMB review, final clearance for which was received in October 

2000.19

 The protocol was designed as an instrument aimed at the investigator, not at any 

group of respondents. The instrument represented a field team’s agenda for collecting 

evidence from a variety of sources—including but not limited to open-ended interviews. 

Using the collected evidence, the protocol’s topics were then addressed by the team by 

creating a formal and documented database that included quantitative as well as qualitative 

data. The focus of the protocol was on tracking and documenting actual behaviors and 

events through converging evidence. This objective may be contrasted with surveys, where 

the information is based on self-reports by a survey’s respondents. 

1.3 SAMPLE OF SCHOOLS BEING STUDIED

 Though small in size, the role of the 18 schools in the Field-Focused Study was 

nevertheless to represent all of the CSRD schools. A two-stage sampling procedure, 

described fully in Appendix A, also attended to stratifications by research-based method 

and by school district. The operational goal was to have two schools with the same 

research-based method (in different districts) and two schools in the same district (but with 

different methods). The stratification provided the basis for a minimal replication design, 

whereby implementation experiences could more readily be attributed either to the method 

or to the district conditions or to neither—thereby helping to arrive at a broader set of 

conclusions than attainable through a set of individualized or idiosyncratic case studies 

alone.20

19The clearance came just in time to schedule and conduct the first round of site visits in the fall and 

winter of 2000-01, which also included a final training session for the teams held in early November 2001. 

Thus, the teams had to scramble during these few months to cover all the designated schools.

20This methodology proved to be extremely effective as the Field-Focused Study progressed. Site visit 

teams were able to crosswalk information and data gathered at schools within the same district, and 

investigate reasons for missing information. For example, staff from one school frequently referenced 

another districtwide research-based method or strategy they credited with allowing them to continue reform 

efforts once CSRD funding ended. When no reference to that initiative was made at the second school in that 

district, the team had enough contextual information to ask probing questions and learn that staff at the 

second school were struggling with internal support issues, as well as potential reorganization, and had not 

bought into the new method.
1-14

The selected schools were promised anonymity for participating in the evaluation. As 

a result, Exhibit 1-5 only shows the locations of the schools, by state. However, more 

detail about the schools’ characteristics can be found in Exhibit 7 of Appendix A. 

1.4 STUDY ACTIVITIES AND A GENERAL CAVEAT

The Field-Focused Study was a multi-year effort whose main data collection occurred 

over a two-year period (2000-01 and 2001-02). Findings from the first two rounds of site 

visits, which took place during each of the two semesters in the academic year 2000-01, 

were reported in an annual report to ED in December of 2001 (COSMOS, 2001). The 

current report updates and expands on those findings (Section 2), and draws conclusions 

about the implementation of CSRD (Section 3). As a general caveat in interpreting the 

findings, note that the Field-Focused Study was not designed to determine the causal 

effects of the CSRD program or of the specific research-based methods employed by the 18 

schools. 

Exhibit 1-5 

LOCATION OF THE FINAL SAMPLE OF CSRD SCHOOLS 

FOR INCLUSION IN THE FIELD-FOCUSED STUDY 
2. CSRD IMPLEMENTATION: 

THE 9 CSRD COMPONENTS 

The initial two rounds of site visits during 2000-01 provided preliminary information 

about the status of CSRD implementation at the 18 schools. These findings were updated 

and expanded upon following the final two rounds of site visits in 2001-02. During the 

final year of data collection, five schools had officially completed CSRD and were to be 

operating without CSRD funding21 (their CSRD activities began in 1998-99), and 13 were 

in their third and final year of funding (CSRD activities began in 1999-00). The site visits 

collected data about: the progress being made on the 9 CSRD components; salient district 

and state conditions; the likelihood that reform would be sustained; and promising reform

strategies practices emerging from the sites.22

This section discusses observations regarding each of the 9 CSRD components, 

including the use of the research-based method at each school (CSRD Component 1) and 

the breadth and comprehensiveness of the schools’ reform designs (CSRD Component 2). 

Throughout this section are references to promising reform strategies (found in Appendix 

B) investigated during site visits to the 18 CSRD schools. These strategies are provided as 

examples of innovative or successful ways in which schools addressed one or more of the 9 

CSRD components. 

Section 3 of this report then covers related data collection about contextual conditions, 

such as district and state policies possibly affecting comprehensive reform at the schools. 

The section summarizes the overall strength of CSRD implementation, also examining the 

conditions or pathways associated with implementation. The section ends with an analysis 

of the likely sustainability of comprehensive school reform following the end of CSRD 

funding. 

2.1 IMPLEMENTING RESEARCH-BASED METHODS

(CSRD Component 1)

All of the schools initially adopted either a nationally or locally developed researchbased method (see Exhibit 2-1). Among the “national” methods, 3 schools were 

implementing Success for All (SFA) only; 2 each were implementing High Schools that 

Work (HSTW), Accelerated Schools, Lightspan, and Coalition of Essential Schools (CES); 

1 each was implementing Success in the Making (SIM) and Co-NECT and the dual method 

2-1

 

21Three of the five schools had received no-cost extensions allowing them to carry on some CSRD 

activities for another semester or year.

22 Appendix D contains short summaries of these reports.
2-2

Exhibit 2-1 

RESEARCH-BASED METHODS ADOPTED BY 18 CSRD SCHOOLS AND 

FIDELITY ASSESSMENTS OBTAINED FROM DEVELOPERS 

School Name/Method* 

CSRD 

Award 

Status in 

2001-02 Developer Fidelity Assessments Obtained

Fidelity 

Rating1

School A 

Accelerated Schools

PostAward Accelerated Schools 

University of Connecticut 

Assessment made by local 

representative of developer 

High 

School B 

High Schools That Work

Year 3 Southern Regional Ed. Bd.; 

Atlanta, GA 

On-site reports obtained from the 

developer 

Medium

School C 

Co-NECT

Year 3 BBN Corporation 

Cambridge, MA 

Assessment made by local 

representative of developer 

Medium

School D 

Success for All/Roots & Wings

Year 3 SFA 

Baltimore, MD 

SFA semi-annual implementation 

review reports 

High 

School E 

Coalition of Essential Schools

PostAward CES 

Oakland, CA 

Developer had no contact with 

school; school not implementing 

Low 

School F 

Coalition of Essential Schools

Year 3 CES 

Oakland, CA 

Assessment made by local 

representative of developer 

Medium

School G 

“Special Literacy Model”*

Year 3 Local university professor Assessment made by local 

representative of developer 

High 

School H 

Success for All

PostAward SFA 

Baltimore, MD 

Assessment made by local 

representative of developer 

Low 

School I 

Lightspan

Year 3 Lightspan Partnership 

San Diego, CA 

Self-assessment by school staff (no 

contact with developer) 

Low 

School J 

“Performance Tasks Model”*

Year 3 Local superintendent of 

schools 

Assessment made by local 

representative of developer 

Medium

School K 

Success for All

Year 3 SFA 

Baltimore, MD 

SFA semi-annual implementation 

review reports 

High 

School L 

“Behavior Mod. Model”*

Year 3 Local consultant and school 

leadership 

Assessment made by local 

representative of developer 

Medium 

School M 

Lightspan

Year 3 Lightspan Partnership 

San Diego, CA 

Assessment made by local 

representative of developer 

Low 

School N 

Success for All

PostAward SFA 

Baltimore, MD 

SFA semi-annual implementation 

review reports 

Medium

School O2

Success-in-the-Making

Year 3 Computer Curriculum Corp.; 

Sunnyvale, CA 

Assessment made by school 

coordinator; developer made no 

fidelity judgements 

Medium

School P 

High Schools that Work

Year 3 Southern Regional Ed. Bd.; 

Atlanta, GA 

On-site reports obtained from the 

developer 

High 

School Q 

Accelerated Schools

Year 3 Accelerated Schools 

University of Connecticut 

Assessment made by local 

representative of developer 

High 

School R 

“Comp. Reform Model”*

PostAward 

School leadership Developer makes no fidelity 

judgments 

Low 

 *Locally developed methods are in quotes and have been given pseudonyms. 

 1Ratings are based on developer enumeration of presence of method components at each school, as well as 

judgment about school’s understanding of the method’s essence. Developers articulated judgments with some degree 

of detail about each school. 

 2School O originally identified Co-NECT as its reform method in its CSRD application; it decided instead to 

expand the use of the SIM method.
of SFA and Roots & Wings; and 4 schools were implementing “locally developed” 

methods.23 Appendix C contains descriptions of the methods in the final sample. 

2.1.1 Implementation of the research-based method is fully proceeding at 9 of the 

18 schools, with partial or minimal implementation at the other 9. Using a five-point 

scale (not implementing, planning, piloting, implementing, and fulfilling or sustaining),24

15 of the 18 schools were judged to be implementing the method at some level. However, 

the teams’ judgments distinguished between two subgroups: 9 schools (2 of the 9 had 

completed their CSRD grant period) that appeared to be in the implementing or fulfilling or 

sustaining category, and 6 schools (all in their third year of the CSRD grant) still only 

partially implementing (i.e., planning or piloting) their method (see Exhibit 2-2). [As a 

promising reform strategy, see “Stringent Implementation of the Research-Based Method 

Believed to Result in Improved Reading Score,” in Appendix B.] 

Of the 6 schools judged to be only partially implementing (i.e., in the planning or 

piloting phase) two schools (Schools I and M) adopted Lightspan as their research-based 

method—a method requiring extensive home-parent participation. Both schools had failed 

to achieve such participation, and by the time of the final site visit, the method was in use at 

only one grade level at each school. A third school (School B) was a school whose 

implementation of a method only began to expand in the second year of CSRD funding, but 

by its final year appeared to be diminishing again. The other three schools (Schools C, L, 

and O) were implementing portions of the method schoolwide but had not yet adopted all 

of the components associated with the method. 

Implementation appeared to be minimal or had stalled completely at three schools 

(Schools E, H, and R—all of which had completed their CSRD grant period), as explained 

in Exhibit 2-3. 

2.1.2 All but four schools are implementing the originally proposed method. With 

the exception of 4 of the 18 schools, all had kept to the method named in their original 

CSRD application. Schools E and H were no longer implementing the method they 

proposed for the reasons described above. School R was only using remnants of its locally2-3

 

23Because the CSRD legislation called for schools to employ research-based methods and strategies that 

have been successfully replicated, most schools in the first and second cohort adopted externally developed

methods. While locally developed methods could have been entirely developed within the schools, the

likelihood (confirmed during the Field-Focused Study) was that local methods were developed externally.

24The scale is derived from Bodilly (1998). To arrive at a score, field teams were instructed to 

triangulate data from three sources: interviews of school staff; direct observations of classroom and school

behavior; and consultations with the originator or “developer” of the method (who was available for most of 

the schools). The five-point scale does not fully reflect the complexity of method implementation: methods 

can be implemented 1) fully in some classrooms but not at all in others; and 2) partially in some or all 

classrooms. Further, different components of a method can be implemented or not implemented (and in some

classrooms but not others).
2-4

Exhibit 2-2 

IMPLEMENTATION STATUS OF RESEARCH-BASED METHODS 

(Component 1)

Level of 

Implementation School/Method* Implementation Status

School A 

Accelerated Schools

Elements of the method were observed in all classrooms; teachers 

were actively engaged in accountability and decisionmaking at the 

school, using expertise to affect change. 

School D 

Success for All

Method was in use in all classrooms in reading and math; teachers and 

students were familiar and comfortable with the curriculum. 

School F 

Coalition of Essential Schools

Nearly all faculty were involved in collaboration groups, school 

established three school-to-work academies (with additional academies 

planned for future years). 

School G 

“Special Literacy Model”*

Materials and concepts were in use in every classroom and in all facets 

of the schools curriculum. 

School J 

“Performance Tasks Model”*

Aspects of the method were in use in all classrooms; parts of the 

method (i.e., portfolios) were more evident. 

School K 

Success for All

Method was in use in all classrooms, for all students, every day, though 

use in the upper grades was faltering. 

School N 

Success for All

All classrooms were using the method during the designated time, with 

varying levels of expertise. 

School P 

High Schools That Work

Use of method strategies was evident in nearly all classrooms; class 

periods scheduled in HSTW-recommended blocks. 

Appears to be 

 “implementing” 

 or “fulfilling and 

 sustaining” the 

 method (n=9)

School Q 

Accelerated Schools

Most teachers and administrators supported use of the method, and 

aspects were observed in all classrooms. 

School B 

High Schools That Work 

School began implementation efforts, but support faltered during 2000-

2001 when staff voted against structural changes. 

School C 

Co-NECT

Teachers were organized into small communities; students worked in 

interdisciplinary project groups; technology use was improved. 

However, the method was not used equally by all staff. 

School I 

Lightspan

No consistent use of the method observed during 2000–01; changes in 

implementation plan resulted for 2001–02 and the method was used in 

all 1st grade classrooms (and 80 percent of 1st grade homes). 

School L 

“Behavior Modification Model”*

Implementation of behavioral aspects of the method varied by teacher; 

implementation of instructional aspects was minimal.

School M 

Lightspan

Support for the method was limited throughout CSRD period; despite 

improvements in use in 2000-01, only one class level and an 

afterschool program used the method in 2001-02, and home 

implementation stopped. 

Appears to be 

 partially 

 implementing 

 the method (i.e., 

 “planning” or 

 piloting”) (n=6)

School O 

Success-in-the-Making

Teachers used method inconsistently, with preparation for state 

assessment taking precedence; implementation became less consistent

in 2001-2002 when state-intervention ended. 

School E 

Coalition of Essential Schools

By 2001-2002, implementation had stopped, as the school focused on 

organizational and behavioral problems; some aspects of the method 

remained as part of school philosophy. 

School H 

Success for All

School fully implemented the method throughout CSRD award period; 

voted to shift to district reading curriculum and implementation stopped 

in 2001-2002. 

Is not implementing, 

 or ceased 

 implementing 

 the method 

 (n=3)

School R 

“Comprehensive Reform 

Model”*

Aspects of the method were initiated, but diminished (almost entirely) 

after first year of funding; in 2001-2002, some elements were being 

revived, but with no attention to any CSRD design. 

 *Locally developed methods are in quotes and have been given pseudonyms.
2-5

Exhibit 2-3 

THREE CSRD SCHOOLS WITH MINIMAL OR STALLED IMPLEMENTATION 

OF THE ORIGINALLY-ADOPTED RESEARCH-BASED METHOD

School/Method* Reason for “Not Implementing” Score

School E 

Coalition of Essential Schools

School E underwent significant organizational changes (splitting from a 

middle school and becoming a separate charter high school), and the 

attention and energy of staff shifted between year 2 and year 3 from 

reform to these structural changes. In addition, following the move to a 

new location, the school staff were faced with further school management 

problems— including disciplinary issues with students, as well as the 

general administration of the school. 

School H 

Success for All

During the first two rounds of site visits in 2000-01, the site visit team 

identified School H as fully implementing the method. The loss of CSRD 

funding was accompanied by a shift in district support for the method, a 

greater emphasis in the district and state on student achievement scores, 

and a faculty vote at School H to stop implementing the method during 

the 2001-02 school year. While the school staff continue to use some of 

the method’s practices, the school’s implementation status nevertheless 

shifted to the “not implementing” category. 

School R 

“Comprehensive Reform Model”*

School R had started implementing a locally developed, schoolwide 

reform method in its first CSRD year (1998-1999), following several years 

of planning by the school’s top officials and faculty. However, the 

principal who had led the reform planning left School R just before the 

start of the first CSRD year, and the school then had three different 

principals during the three-year period coinciding with its CSRD award. 

During this period, although implementation of the method had begun in 

one grade in 1998-1999, commitment to the method diminished rapidly 

with the succession of principals, and as of 2000-2001 none of the 

method’s features were in place. The school obtained a one-year, nocost extension for their CSRD award1

 and in 2001–02, and began to 

implement remnants of the original reform method, but without any 

understanding of the relation of those elements to a CSRD reform 

design. The school therefore was still judged to be “not implementing.” 

 * Locally developed methods are in quotes and have been given pseudonyms. 

 1 The school spent only a fraction of its CSRD funds by the end of year 3. The newest principal 

(serving his first year during 2000-01) was committed to revisiting the CSRD activities and the state 

subsequently approved the request. 
2-6

 

developed method that aligned with district policy and the changing needs of the school. 

School O changed its method from the one originally proposed in its CSRD application 

because, following the submission of the CSRD application, the school received a poor 

performance rating in the state’s accountability system. Because of the poor rating, the 

district encouraged School O to expand its use of an alternative method, which was being 

implemented at many other district schools. With state approval, School O therefore 

switched from its originally proposed Co-NECT method to Success-in-the-Making (SIM).25

 A fifth school (School N) was implementing the originally proposed method, but the 

field team failed to identify it properly during the screening procedure. School N had been 

expected to adopt Roots & Wings, a method developed by the Success for All Foundation 

to cover mathematics, science, and social studies, in addition to reading. However, 

although School N was later found to have only implemented Success for All (and hence 

was only covering reading), the school was using both the “roots” level and the “wings” 

level of SFA (but in reading only). During the screening, the school therefore responded 

that it was using Roots & Wings when in fact no method was being used outside of 

reading.26

2.1.3 Developers gave mixed fidelity ratings to the schools. The field teams were 

able to obtain information about fidelity from developers for 14 of the 18 schools 

(including locally developed methods).27 In 6 of these 14 cases, the developers judged the 

fidelity of implementation to be high; in 1 of the 2 judged to be low, implementation also 

was low as judged by the site visit team (the other school was rated as having partial 

implementation). The teams were unable to obtain fidelity ratings from the method 

developers for four schools: two schools were implementing a national method, but had no 

contact with the developer; one school was implementing a locally-developed method and 

had no external consultant; and one school’s developer was not willing to make a 

judgement about fidelity. In all but the last case, the school’s own assessment of their 

fidelity was “low” (the school with the developer unwilling to make an assessment rated 

their own fidelity as “medium”). Exhibit 2-1 also contains the ratings made about the 

fidelity in implementing each method. 

25The evaluation team became aware of the switch too late to change the final sample of schools to be 

studied. As a result, the final sample, originally to include a pair of Co-NECT schools, had one Co-NECT

and one SIM. 

26Contrary to School N’s usage, schools adopting only the reading component generally report that 

they have adopted Success for All. As a result, the final sample—intending to have one pair of Success for 

All (only) and one pair of dual SFA-Roots & Wings combined methods—then had three Success for All (only) 

schools and one school using a combination of the Success for All and Roots & Wings methods.

27Most often, the information came as a result of a face-to-face meeting with the developer’s field 

representative who was providing external technical assistance to the school. However, in other cases the 

team either contacted the developer by telephone or reviewed a formal report submitted by the developer to 

the school.
2-7

 

2.1.4 Experiences between locally developed and national methods have not 

markedly differed. Of particular interest regarding these methods was the collection of 

locally developed methods, for two reasons. First, at the national level, less is known about 

them than about the “national” methods. Second, one possibility entertained at the outset 

of the Field-Focused Study was that a CSRD school might have shared in the development 

of a locally developed method (e.g., serving as a co-developer), thereby qualitatively 

altering its expected CSRD implementation experience. 

 Of the 4 locally developed methods, 3 were similar to national methods in having: a 

formal developer; prior evidence of effectiveness; and ongoing use at two or more schools. 

Proportionately, the implementation experiences by the schools with locally developed 

methods did not differ markedly, as shown previously in Exhibit 2-2, from those of the 

national methods. Further, with the exception noted below, the schools did not have any 

role in developing the methods. Thus the main difference between the national and locally 

developed methods appears to be that the latter were only known and being used within a 

limited regional area—i.e., either a single school district or multiple regions within a single 

state. 

 As one result, the field teams were able to locate the local method developers and to 

collect data from them similar to the data from the national method developers. For 

instance, the local developers were providing schools with external technical assistance, 

similar to the assistance being provided by their counterparts with national methods. The 

single exception again had to do with School R, whose locally developed method was a 

truly “home-grown” combination of several practices, none of which itself had any method 

that had a developer.28 

 Two of the methods (adopted by Schools G and J) had both instructional and 

curriculum content (see Exhibit 2-4). For instance, School J’s method involved the 

development and maintenance of a bank of substantive student tasks (“authentic” tasks) 

covering each academic subject and enabling students to develop portfolios of their work. 

The tasks also promoted reasoning and thinking as part of the learning process. In 

addition, local teachers were encouraged to use their classroom experiences to devise and 

nominate new tasks to be reviewed for inclusion in the bank. [As a promising reform 

strategy, see “Research-Based Method Guiding Scope and Sequence of Lessons in All 

28School R’s CSRD application had indicated that, of the component practices comprising its whole 

“model,” the adherence to state standards represented the research-based component of the entire “model.” 

The state standards go beyond a mere listing of standards and specify a profile of learning—defining what 

students should know, understand, and be able to demonstrate to a high level of achievement. Students must 

then satisfactorily complete a state-prescribed classroom assessment for each standard, to show that they have 

met the standard. However, schools and teachers are to define how students are taught and the specific 

assignments they are required to complete. In this sense, a case can be made that the state’s standards do 

provide guidance similar to that of a research-based method.
2-8

Exhibit 2-4 

FOUR LOCALLY DEVELOPED METHODS 

School Method 

Pseudonym 

Description 

School G “Special Literacy 

Model’”

The method has three goals: 1) to implement effective practices for K-3 literacy learning and 

teaching; 2) to develop teacher leadership teams within a school and its district; and 3) to 

develop mentor teachers to support the preservice education at the university. The method 

is centered around a “special topics” university graduate course. Teachers receive credit for 

their method-related participation. An initial two-week summer institute, attended by 

representatives from the school, helps to plan the activities over the ensuing academic year, 

such as the establishment of teacher teams, parent literacy nights, and parent mathematics 

nights. The method claims to be highly effective in building good instructional practice, 

teacher cooperation, and the use of a specific array of instructional materials. The method is 

based on academic research on early literacy. 

School J “Performance 

Tasks Model”

The method is composed of discrete performance tasks, accessible via a Web site, that 

teachers may select and use to deliver instruction. Tasks are available for all content areas 

(though more are available for reading/language arts than for any other area). Tasks are 

designed to be examples of real-life experiences (“authentic” tasks) enabling students to 

acquire content knowledge, good work habits, and thinking and communication skills. The 

method promotes reasoning and thinking as part of the learning process. Examples include 

organizing a healthy luncheon, guiding classmates on a trip, corresponding with a scientific 

author, and composing a song. The method also incorporates student self-assessments for 

judging the quality of a product and overall student performance. Students develop a 

portfolio of their work, and these, along with students’ self-assessments, are used to 

demonstrate and evaluate process and achievement.

School L “Behavior 

Modification 

Model”

The goal of the method is to achieve schoolwide change enabling “greater academic 

achievement, productivity, and prosocial competency.” The method consists of a behavioral 

protocol, climate-setting strategies, and instructional approaches to improving student 

achievement. Teachers use techniques such as the good behavior game (used during 

regular instruction time to reward good behavior and cite inappropriate behavior) to teach 

students what behaviors to exhibit and what behaviors to inhibit. Other behavior 

modification techniques also are prescribed by the method, including organizing students in 

the classroom into teams, giving students “jobs” within the classroom, and student 

monitoring. In addition, students are taught to walk in “model lines” (or one behind the other, 

with their hands behind their backs) and to use “model voices” (volume levels appropriate to 

the situation). Other elements of the method have included having students graph their own 

performance in spelling and math, so that they learn how to do self-assessments and can 

see their progress, and engage in peer tutoring. 

School R “Comprehensive 

Reform Model”

The original method included the implementation of state standards, which the school 

identified as among the first at the state level to embrace educational quality reform and 

“significantly different from other state and national models in that they were more projectand performance-oriented.” Other components of the comprehensive method included a 

Freshman Foundation course, intended to help freshman to get prepared for high school; a 

flexible block schedule; an interdisciplinary curriculum, whereby teachers of different 

subjects would have common planning time, hold daily team meetings, and coordinate their 

instructional schedule; the integration of school-to-work (school-to-career) topics; the 

integrated use of technology into the curriculum, especially taking advantage of individuallyassigned laptop computers; and a vision of the school as having a “global” magnet, 

transcending the four separate magnet programs at the school. In the final year of 

implementation, the method had been revised: 1) implementation of state standards 

continued, 2) the Freshman Foundation course was replaced with Freshman Teams and the 

use of “My Action Plans (MAPs)” by individual students to guide progress through high 

school and toward future goals; and 3) the laptop project ended; and 4) the concept of a 

“global” magnet was replaced with district required small learning communities. 
2-9

 

Subject Areas,” in Appendix B.] The methods at the two other schools (Schools L and R) 

more closely resembled “process” methods, focusing either on behavior and climate-setting 

strategies (School L) or on the use of technology in the curriculum and Freshman transition 

initiatives (School R). [As a promising reform strategy, see “Teacher Teams Provide 

Students with Cross-Curricular Authentic Learning Opportunities,” in Appendix B.] 

2.1.5 Classroom observations are consistent about ongoing experiences with the 

methods. Direct observations of classrooms in all of the 18 schools were an integral part 

of assessing the extent and nature of implementation.29 In only a few cases was the 

observed implementation different from that expected as a result of the interviews of the 

schools’ officials. For instance, Schools J and B had individual classrooms where 

implementation was much lower than expected based on the officials’ interviews. In the 

opposite direction, School F’s classrooms showed a high degree of implementation even 

though the interviews had described implementation as still “in progress.” 

 More difficult to assess through classroom observation was any potential role of the 

other CSRD components. Brief interviews with the teachers whose classrooms had been 

observed proved more informative. The teachers reported about: their professional 

development experiences, their perceptions of staff support for reform; and apparent parent 

and community support in conjunction with their classroom activities. 

2.2 COMPREHENSIVENESS OF DESIGNS FOR SCHOOL REFORM 

 (CSRD Component 2)

 The field teams assessed the extent and quality of comprehensiveness by attending to 

the presence of the eight remaining CSRD components and their apparent linkage. Key 

among these other 8 components is Component 2 (evidence of a “comprehensive design”), 

which the teams assumed needed to exist if implementation was later to achieve actual 

comprehensiveness. 

 To assess progress on this component, the field teams first tried to obtain any 

documentation of such designs—e.g., the breadth and scope of any school improvement 

plan, or even the apparent comprehensiveness and convergence of a school budget that 

integrated CSRD funding within it.30 The teams then also tried to determine the school 

29Each classroom observation involved the use of a formal classroom observation instrument (included 

in the case study protocol—an attachment to Appendix A).

30Not considered sufficient evidence of a comprehensive design was sheer existence of the school’s 

original CSRD application alone. All of the applications address the nine components because states required 

their districts and schools to do so. However, addressing these components did not necessarily mean that 

they had been adequately operationalized (and, in some cases the application had been totally prepared by the 

district and therefore may not have become a salient part of the school’s understanding of CSRD). To be 
2-10

 

staff’s level of awareness and understanding of reform, independent of whether such 

documentation existed or not. On both fronts, the teams’ inquiry was aimed at determining 

whether the school understood that CSRD meant reforming the entire school—i.e., 

potentially expanding to its entire way of doing business and all its operations—and not 

simply adding a new function or project or activity; and that in carrying out such reform, a 

school would necessarily have to attend to the alignment or connectivity among all of its 

operations. 

2.2.1 Less than half of the 18 schools had reforms with a comprehensive design. 

Eight of the 18 schools (Schools A, B, D, F, G, J, L, and P) satisfied both conditions: 

a) school reform plans or documentation that were comprehensive, and 

b) acknowledgments or understandings of comprehensiveness by the school staff who had 

been interviewed (see Exhibit 2-5). Six other schools (Schools C, H, K, N, O, and Q) had 

comprehensive plans or documentation, such as school improvement plans, but staff were 

unaware of or detached from any comprehensive vision associated with such 

documentation. Where understandings of comprehensiveness fell short, the general reason 

was that CSRD breadth or implementation strongly resembled a separate project or even 

“add-on” activity at the school. Among most of the remaining schools, neither 

comprehensive documentation nor requisite staff understanding was present. 

2.2.2 Comprehensiveness of CSRD can emerge from either working with 

comprehensive research-based methods or having a comprehensive school improvement 

plan. Of the 8 schools that had satisfied both conditions, the comprehensive designs for 5 

of them (Schools A, B, F, J, and P) were associated with the comprehensiveness of the 

research-based methods that had been adopted, but not necessarily because the school had 

developed any separate comprehensive reform vision or plan; for the other 3 schools 

(Schools D, G, and L), the comprehensiveness was associated with a required school 

improvement plan that was comprehensive even if the research-based method was not. [As 

a promising reform strategy, see “School Improvement Planning Process Aligns Reform 

Initiatives,” in Appendix B.] 

 Overall, 10 of the 18 schools may not yet have fully embraced the principles of 

reform and steps needed to achieve comprehensive reform. Compared to the proportion of 

schools implementing research-based methods (CSRD Component 1), the proportion of 

schools with comprehensive designs (CSRD Component 2) was substantially lower. 

credited with pursuing a comprehensive design, schools had to show that they were actually using their 

CSRD applications in planning and implementing reform, and further that the staff interviewed could express 

some awareness and understanding of the comprehensive nature of the activities. 
2-11

Exhibit 2-5 

EVIDENCE OF COMPREHENSIVE DESIGN 

(Component 2) 

School/Method* Evidence of Comprehensive Design

School A 

Accelerated Schools

Interviews, classroom observations and documents all made clear that the staff 

 understood and strongly supported the reform process, including some 

 familiarity with the nine CSRD Components. 

School B 

High Schools That Work

The reform strategies were integrated into the school improvement plan. The 

method and plan appeared to be well understood by staff, students, and 

parents, though support for some elements was declining. 

School C 

Co-NECT

The school had a single plan, required by the district, linking all activities to goals, 

objectives, benchmarks, and strategies. The plan served as funding 

application for CSRD and was aligned with state standards. However, staff 

were reportedly disappointed with the plan, which they believed failed to 

capture all activities and funding streams. Co-NECT was recognized staff as 

the only comprehensive reform strategy. 

School D 

Success for All/Roots & Wings

The district required the school to develop a comprehensive plan aligned to the 

district’s vision for improving student performance. Much of the district plan 

paralleled the CSRD reform components. Many of the strategies in the plan 

were specifically tied to the reform methods. Evidence that the staff supported 

the plan were visible throughout the school, particularly in display materials 

posted on walls in classrooms and gathering areas. 

School E 

Coalition of Essential Schools

By the final year of funding, the staff had no vision for comprehensive school 

reform, probably due to its pre-occupation with its own survival and 

restructuring. While many staff shared a common philosophy about reform, the 

methods or means for accomplishing this vision were a source of conflict. 

School F 

Coalition of Essential Schools

The CSRD reform plan encompassed all 9 CSRD components, though the school 

improvement plan focused solely on goals to increase student performance. 

The majority of staff clearly understood and supported the reform plan. 

School G 

“Special Literacy Model”*

The district-required school improvement planning process played a central role 

in establishing the comprehensiveness of the reform effort. Every faculty 

member signs the plan to indicate an understanding of his or her role in 

implementing the plan. 

School H 

Success for All

The school improvement plan had a comprehensive set of goals and strategies, 

generally aligned with the CSRD components, by driven by the state 

assessment. The comprehensiveness of the reform effort was widely 

understood, though by 2001-02, the staff felt their reform efforts were in conflict 

with the districtwide reform agenda and plan. 

School I 

Lightspan

The school had two plans, one covering achievement goals, and another 

summarizing a variety of school initiatives. Neither offered a comprehensive 

picture or plan for the nine CSRD components. Teachers described the school 

as having a split focus and a mission that was way to broad. 

School J 

“Performance Tasks Model”*

The school’s improvement plan did not address the CSRD reform components, 

but did reflect the school’s goals and strategies for improving student 

achievement. The method itself included all reform components, and staff 

appeared to understand and support both the reform effort and the method. 

*Locally developed methods are in quotes and have been given “pseudonyms.” 

(Continued on next page)
Exhibit 2-5 (Continued)

2-12

School/Method* Evidence of Comprehensive Design

School K 

Success for All

The school improvement plan addressed each component in relation to reading, 

but little emphasis was placed on other subjects. A steering committee 

managed implementation and integration of reform components. Staff 

appeared to understand reform effort.

School L 

“Behavior Modification Model”*

The district required the school to develop a comprehensive plan aligned to the 

district’s vision for improving student performance, which paralleled the CSRD 

reform components. Staff appeared to understand the district reform plan, 

which was comprehensive (even though the research-based method did not 

appear to be well-integrated into this plan).

School M

Lightspan

The school’s original improvement plan included Lightspan, but didn’t cover the 

other reform components. A revised plan was finalized in June 2001 and was 

intended to be aligned with state and district standards, however, a new school 

administration the following year did not think the plan addressed the school’s 

needs sufficiently. Some staff believed the plan was being ignored and others 

argued it was not a practical document. 

School N 

Success for All

All 9 elements of the reform were present in the school. A new plan was adopted

in 2001–2002 that aligned the plan to the state assessment, and set strategies 

for reaching improvement goals. However, it did not appear to be a working 

document at the school; programmatic and other decisions were made without 

reference to the overall design. 

School O 

Success-in-the-Making

The school’s improvement plan listed its objective as improving student 

performance on state assessments; the plan contained references to a number 

of reform components. The breath of the planning process and SIM appeared 

to provide a unifying plan of action throughout the school. 

School P

High Schools That Work

The reform strategies were integrated into the school improvement plan. Staff 

and the core leadership team understood and supported the method and the 

reform plan. 

School Q 

Accelerated Schools

The school addressed the nine reform components in their proposal, with varying 

levels of detail. Some staff, particularly on the leadership committee supported 

the plan, but high turnover among staff resulted little understanding or support 

for the comprehensive reform design by a large number of newer staff. 

School R 

“Comprehensive Reform Model”* 

The method initially reflected comprehensive whole school reform; but was not 

used after the first year of CSRD funding. During the school’s final CSRD

year, the reform effort was revived, but the lack of a clear and comprehensive 

plan (understood by all staff) was limiting the impact of the changes. 

*Locally developed methods are in quotes and have been given “pseudonyms.”
2-13

 

2.3 PROGRESS IN IMPLEMENTING THE 7 OTHER CSRD COMPONENTS

 As with the first two components, the site visit protocol called for covering each of 

the other seven CSRD components individually. However, the findings from the site visits 

suggested that these seven components could be dealt with in several combinations, in the 

interest of streamlining the discussion: 

• Professional development (Component 3) and external technical 

 support and assistance (Component 7); 

• Measurable goals and benchmarks (Component 4) and evaluation 

 strategies (Component 8); 

• Support within the school (Component 5) and parent and 

 community involvement (Component 6); and 

• Coordination of resources (Component 9) and sustainability. 

To be noted is that the last combination joins one of the seven components (Component 9) 

with another topic of high interest but that has not been regarded as a separate CSRD 

component—sustainability. Inherent to comprehensive school reform is that efforts cannot 

be short-lived as in a specially funded “project.” Thus the Field-Focused Study has 

devoted explicit attention to the issue of sustainability (Section 3.5 addresses the issue of 

sustainability in greater detail). 

2.3.1 Both professional development (Component 3) and external technical support 

and assistance (Component 7) have largely been devoted to the implementation of the 

research-based method, not necessarily comprehensive reform more broadly. The 

rationale for pairing these two CSRD components is as follows: Most schools were 

engaged in moderate to intensive professional development in relation to implementing 

CSRD, but most if not all of this professional development was being provided by an 

external technical support team (see Exhibit 2-6).31

 Exhibit 2-6 summarizes the professional development reported in the site visits. The 

exhibit distinguishes between method-related professional development and other, reformrelated professional development. The last column in the exhibit also comments on any 

unusual barriers to professional development identified through the site visits. 

31By design and instruction from the U.S. Department of Education, and consistent with CSRD 

program guidance, the Field-Focused Study deliberately defined “external” support and assistance to exclude 

assistance by the district.
2-14

Exhibit 2-6 

PROFESSIONAL DEVELOPMENT ACTIVITIES 

(Component 3) 

School Name/

Method*

Types and Frequency of 

Method-Related Professional 

Development (PD) Activities

Other Professional Development 

Activities Barriers

School A 

Accelerated Schools

Extensive training in AS philosophy and 

“powerful learning” strategies for all 

staff; weekly cadres meetings allowed 

staff to identify issues of concern for 

entire school and develop hypotheses 

to address them. 

Staff received training in 

successful reading strategies 

for LEP students from local 

university consultants, as well 

as test preparation and 

leadership skills. 

District-wide budget cuts 

 would limit formal PD 

 activities in 2002-03. 

School B 

High Schools That 

Work

CSRD increased PD activities by 12 

times compared to year prior to grant; 

all activities were focused on HSTW 

and included peer-to-peer 

presentations. 

Implementation of Making 

Schools Work at the middle 

schools involved some training 

for high school teachers, 

related to facilitating the 

transition from middle to high 

school. 

Nearly one-third of the 

teachers quit or retired; 

support for HSTW was 

fading, though upcoming 

merger with county high 

school may revive the 

reform effort. 

School C 

Co-NECT

Professional development focused on 

comprehensive planning in relation to 

the SIP process, as well as use of 

student achievement data for 

planning, technology integration, 

project-based learning, and 

assessment. 

Teachers had a common 

 planning period to facilitate 

cooperative learning. 

Professional development 

requirements were not 

tied to teacher recertification, giving 

teachers less incentive 

to participate. 

School D 

Success for All/Roots 

& Wings

Virtually all professional development 

was linked to the reform method; the 

principal estimated that 40 hours of 

professional development activities 

were offered per teacher, and that 

about 70 percent of staff participated 

at this level. 

Administrators and staff meet bimonthly to work on curricular 

strategies; receive training 

from district on use of data. 

All other schools in the 

district used districtpromoted reading and 

math curricula, and 

received districtsponsored professional 

development; isolating 

school staff. 

School E 

Coalition of Essential 

Schools

Formal professional development related 

to the method was primarily available 

to the high school teachers during the 

first year of the CSRD grant; little 

occurred in later years. 

Beginning in 2001-02,the school 

 closed early two days a month 

to allow staff to address 

schoolwide concerns. 

PD sessions and staff 

planning time were being 

used to address 

administrative issues, 

such as discipline. 

School F 

Coalition of Essential 

Schools

Collaborative groups and action teams 

were the main PD-delivery method, 

and nearly all staff participated by 

2001-02; staff took part in a week-long 

“CES change process” course and in a 

local HSTW network. 

Training by district consultants 

has included: using electronic 

portfolios, using independent 

thinking, distance learning, 

and state standards/ testing. 

Lack of funding for 

substitute teachers in 

2001-02 may mean 

groups can no longer 

meet during the school 

day. 

School G 

“Special Literacy 

Model”*

Nearly all staff participate in methodrelated PD that is related to 

comprehensive reform and is resultsbased. 

Twice monthly PD related to 

math is presented by a math 

specialist. 

*Locally developed methods are in quotes and have been given pseudonyms. 

(Continued on next page)
2-15

School Name/

Method*

Types and Frequency of 

Method-Related Professional 

Development (PD) Activities

Other Professional Development 

Activities Barriers

Exhibit 2-6 (Continued)

School H 

Success for All

Until 2001-02, school staff received

method-related PD twice per year 

from developer, and on an ongoing 

basis from campus instructional 

coordinator (through modeling, 

observations, and grade-level 

planning). 

The district provided training-oftrainer PD on content 

standards, access to 

resources, and student 

expectations. When the 

school stopped implementing

SFA in 2001-02, the district 

also provided training on the 

reading curriculum. 

School I 

Lightspan

Earlier training in the method was limited, 

but in 2001-02, training by in-house 

staff and accessibility to materials 

better enabled 1st grade teachers to 

begin implementation. 

School provided staff with 

collaborative time during 2001-

02, allowing staff to meet by

grade level, examine student 

achievement scores, develop

ideas and share plans. 

District restricts early

school dismissal and 

teacher planning days.

School J 

“Performance Tasks 

Model”*

All but three teachers received PD 

related to the method, including 

training sessions on active learning, 

self and peer assessment, 

cooperative planning, and alignment 

with state standards. 

Teachers had to complete 2 halfday training sessions to meet 

state’s continuing education 

requirement; district provided

PD on understanding and 

using student achievement 

data. 

School K 

Success for All

Extensive PD offered around method, 

including five-day introductory

workshop, as well as opportunities for 

training of tutors and the Family

Support Team. 

District required teachers to 

complete 2 half-day training 

sessions to meet state’s 

continuing education 

requirement; district provided

PD related to understanding 

and using student 

achievement data. 

School L 

“Behavior 

Modification

Model”*

Staff received training in the method 

through in-service days at the start of 

the year, annual retreats, and ongoing 

modeling and observations by the 

lead teacher. 

Teachers met bi-weekly (once on 

staff issues; once on curricular 

topics); district required 18

hours of PD activities, and 

provided PD on the math and

reading curricula. 

Lack of implementation of 

instructional portions of 

the method indicated 

inadequate PD on this 

component. 

School M

Lightspan

Activities were primarily focused on 

grade 3–5 teachers who used

Lightspan; one presentation was 

made to all teachers in the school. 

Four Lightspan training events were

provided during 2000–01; no training 

was provided in 2001-02. 

The district provided between 

three and four PD days a year

through a catalog of staff 

development opportunities; 

there was no coordination to 

ensure staff selected PD 

sessions related to the 

school’s reform efforts. 

*Locally developed methods are in quotes and have been given pseudonyms. 

(Continued on next page)
2-16

School Name/

Method*

Types and Frequency of 

Method-Related Professional 

Development (PD) Activities

Other Professional Development 

Activities Barriers

Exhibit 2-6 (Continued)

School N 

Success for All

Method-related PD received twice per 

year from developer, and on an

ongoing basis from campus 

instructional coordinator (through 

modeling, observations, and gradelevel planning)

The district provided training-oftrainer PD on content 

standards, access to 

resources, and student 

expectations. 

District provided PD 

related to the districtsupported reading 

curriculum, which 

School N was not using.

School O 

Success-in-theMaking

Teachers participated in monthly

activities related to the method; CSRD 

funding allowed inclusion of 

kindergarten and Grade 5 teachers; 

training included how to incorporate 

the method into the classroom and 

how to use data. 

District provided PD on 

curriculum goals, co-teaching

methods, using the SFA 

curricula, instructional 

strategies in reading and 

writing, and magnet school 

curriculum. 

School P

High Schools that 

Work

All teachers had a minimum of 6 PD 

days from HSTW framework; 75% of 

teachers opted to participate in more 

activities. 

Staff participated in PD related to 

vocational education, as well 

as PD related to the transition 

students from middle to high

school through Making 

Schools Work (supported by a

2000 CSRD grant to the 

district middle schools. 

School Q 

Accelerated Schools

Each staff member received an average 

of 6 PD days; 90% was reform-related. 

School used a training-of-trainers 

approach; topics covered the specific 

instructional and pedagogical 

practices consistent with AS. 

The school received training on

teaching children of poverty. 

Incentives for faculty

participation in PD included 

tuition/expense reimbursement

and continuing education units. 

District placed restrictions 

on early dismissals and 

teacher work/planning 

days. 

School R 

“Comprehensive 

Reform Model”*

CSRD funds were used in the Year-1 for 

method-related PD, including funds for 

technology-focused training; funds 

were devoted to PD activities in 2001-

02 for both the development of small 

learning communities and freshman 

teams. 

The school’s focus was on 

aligning teachers’ individual 

PD plans with school and 

district plans; the district had a 

catalog of course offerings 

available to school staff. 

Budget cutbacks have 

greatly reduced the 

amount of professional 

opportunities that the 

district can make 

available to schools. 

*Locally developed methods are in quotes and have been given pseudonyms. 
2-17

 

 The first column of Exhibit 2-6 shows that, except in the case of three schools 

(Schools E, M, and R), the schools all have been participating in extensive professional 

development, mainly led by the external method’s developer. To their credit, the 

developers have been able to provide a broad and intense variety of assistance, including: 

1) participation in national conferences and workshops; 2) convening of local workshops 

for the school’s faculty; 3) organizing of annual retreats for the faculty; 4) convening of 

all-staff meetings; 5) specific training to individual teachers; and 6) periodic and in some 

cases frequent on-site support and problem-solving. [As a promising reform strategy, see 

“Professional Development Activities and Instructional Practice Driven by Student 

Assessment Measures,” in Appendix B.) 

 The second column of Exhibit 2-6 shows that, although the field teams were to 

inquire about professional development not related to the research-based method, not a lot 

of such professional development was occurring. In 15 of the 18 cases, the broader 

professional development was: 1) being provided by the district, 2) embedded in the job 

structure, or 3) informal. [As a promising reform strategy, “Staff Utilizes Common 

Planning Period as Professional Development Opportunity” in Appendix B.] In 2 of those 

15 cases, as well as in the remaining 3 schools, staff members participated in professional 

development related to the schools’ adoption of a second research-based method. 

 Directly corresponding to these observations about professional development, nearly 

all of the 18 schools: a) had external technical support or assistance, and b) derived such 

assistance from the developer of the research-based method (see Exhibit 2-7). 

 The intensity and breadth of the assistance in part depended upon the amount of 

CSRD funds allocated for such assistance by the school. Although precise budget data 

were not always available, the site visit team observed that in most cases the bulk of 

schools’ CSRD funds were used to support external assistance from the developer.32 As a 

result, schools that adopted research-based methods requiring ongoing annual outlays 

(compared to outlays for startup years only) were especially likely to need to identify new 

sources of funds, in order to continue comprehensive school reform activities beyond the 

period of CSRD funding. 

 Exhibit 2-7 also shows the diversity of the external assistance and support activities, 

ranging from participation in national meetings to teacher workshops to individualized 

(within classroom) on-site support. In reviewing these data, it is important to note that the 

site visit protocol was not limited to external assistance by the method’s developer. Rather, 

to cover CSRD Component 7, the protocol guided the site visit teams to inquire about any

type of external assistance. As a result, in many cases the teams did find such assistance 

32This does not necessarily mean that the funds went directly to the developer. For instance, funds 

might have been used to pay for teacher substitutes so that the teachers could participate in the developer’s 

workshops.
2-18

Exhibit 2-7 

EXTERNAL TECHNICAL SUPPORT AND ASSISTANCE 

(Component 7) 

School Name/ 

Method* 

Types and Frequency of Method-Related 

External Support and Assistance

Types and Frequency of 

Other Reform-Related 

External Support and Assistance

School A 

Accelerated Schools

School received extensive on-site support from 

the developer since formal adoption of the 

method; by Spring 2002, the principal was 

concerned that her staff had training needs 

beyond the capacity of the national center. 

Consultants provided support in reading 

and writing instruction and classroom 

and time management. 

School B 

High Schools that Work

Developer conducted two site visits per year, 

as well as onsite workshops, sponsored 

visits by schools staff to high achieving 

schools, hosted an annual conference, and 

conducted monthly telephone conferences 

with mentor site leaders. 

During the final year of CSRD funding, 

staff received training in Making 

Schools Work, an initiative to better 

prepare students for the transition to 

high school. 

School C 

Co-NECT

Onsite Co-NECT consultant serviced School C 

and five others, was available on-site almost 

weekly, taught classes (modeling) and made 

presentation to all-school assemblies. 

Teams from other Co-NECT schools 

conducted annual site visits. Developersupported “Exchange” website as resource 

for school staff. 

The school was implementing a number 

of other initiatives (particularly 

reading programs), for which staff 

received training. 

School D 

Success for All/Roots & 

Wings

SFA Foundation provided extensive training for 

both the reading and math curricula in the 

first year of implementation. Refresher 

training sessions took place annually, along 

with visits to assess the level and fidelity of 

implementation (twice per year). 

External assistance provided by a local 

consultant hired by the district to 

support the district’s CSRD schools. 

She helped the school identify 

technical assistance providers and 

develop classroom planning guides, 

and was available during monthly 

phone-calls with the principal. 

School E 

Coalition of Essential 

Schools

After the first year of funding, the school had 

limited contact with the method developer 

(following 1998-99, the school was not an 

official CES site and did not pay annual fees 

to any CES organization. 

No other external support identified. 

School F 

Coalition of Essential 

Schools

Week-long training by CES designed to 

perform a year-long activity for visualizing 

and implementing plans for change. 

Extensive training from Annenberg and 

NSF on using collaborative inquiry 

and action research; HSTW network 

meetings every two months to share 

ideas. 

School G 

“Special Literacy 

Model”*

The developer conducted a summer institute 

and graduate course work, and was on-site 

twice per year for in-service sessions. A 

regional project liaison for the developer was 

on site twice per month, working with 

teachers and providing in-class modeling. 

The developer also worked with staff to 

design and develop professional 

development workshops based on a 

local needs assessment process. 

*Locally developed methods are in quotes and have been given pseudonyms.

(Continued on next page)
2-19

School Name/ 

Method* 

Types and Frequency of Method-Related

External Support and Assistance

Types and Frequency of 

Other Reform-Related 

External Support and Assistance

Exhibit 2-7 (Continued)

School H 

Success for All

Until 2000-02, the method developer visited 

twice a year and provided training and TA on

SFA implementation. The school did not 

complete a contract with SFA for 2001-02 

when staff voted to shift to the districtendorsed reading curriculum.

A local university provided student 

teachers who supported reform by

tutoring students.

School I 

Lightspan

During the first two years of CSRD funding, the 

developer conducted one annual site visit to 

observe classes and provide demonstrations 

of how Lightspan could be used in the 

classroom. In 2001-02, the school 

discontinued its relationship with the 

developer and relied on an internal staff 

person to conduct staff training. 

Due to state budget cuts, the district 

sought assistance from its regional 

service center.

School J 

“Performance Tasks 

Model”*

The school worked with the developer during 

Year 1 who provided initial training. In years2 and -3, the school worked with an external

consultant for the developer, who provided 

training focused on the development of new

tasks. 

School received assistance from 

reading consultants, and district 

assessment staff, as well as “critical 

friend” mentoring support (sponsored 

by the state for schools rated as inneed-of-improvement). 

School K 

SFA

The developer provided training in the method 

for all staff, as well as for members of family 

support team. Staff complained that the 

developer did not respond to concerns about 

appropriateness SFA for students in 

intermediate grades, which may impact 

sustainability of the method at this school. 

No additional training from external

providers was identified. 

School L 

“Behavior Modification

Model”*

Developer provided consistent and strong 

assistance since the inception of the 

program, including helping the school select 

and customize the reform method, 

conducting introductory and follow-up

training for the method, visiting classrooms, 

serving as facilitator at staff retreats, and 

generally assisting as requested. The 

developer also hosted bi-annual networking 

dinners for local principals. 

No additional training from external

providers was identified. 

School M

Lightspan

In 1999–00, the developer provided support at 

the beginning of the school year. In 2000–

01, the school expanded the contract to 

include monthly visits to assist in 

implementation. In 2001-02, the new

administrators at the school decided not to 

renew its contract with the external

developer. The school relied on internal staff 

to train new staff members. 

No additional training from external

providers was identified. 

*Locally developed methods are in quotes and have been given pseudonyms. 

(Continued on next page)
2-20

School Name/ 

Method* 

Types and Frequency of Method-Related

External Support and Assistance

Types and Frequency of 

Other Reform-Related 

External Support and Assistance

Exhibit 2-7 (Continued)

School N 

SFA

The developer conducted implementation visits 

to the schools twice annually. On-site 

support included training in use of SFA, 

conducting classroom observations, and 

customizing training to meet district- and 

school-specific needs (such as the alignment 

between state standards, the state 

assessment, and the method). The school 

reduced the number of visits in 2001-02, as 

CSRD funding dwindled. 

School also received training in use of 

other reading initiatives (Accelerated 

Reader and state-created program). 

School O 

SIM

The developer provided support (ranging from

0.5 to 3 days each month) on technology

planning and grade level meetings, made 

presentations on the method during 

education conferences, and assisted in data 

collection review and analysis. 

No additional training from external

providers was identified. 

School P

HSTW

The developer provided two technical review

visits per year, site specific development 

activities (around the method), visits to high 

achieving schools, monthly teleconferences 

with site leaders, and assessment and 

analysis of student achievement data. 

During the final year of CSRD funding, 

staff received training in Making 

Schools Work, an initiative to better 

prepare students for the transition to 

high school.

School Q 

Accelerated Schools

The developer provided training to school 

leadership and staff on-site, conducted 

technical assistance and on-site monitoring 

every six weeks, and developed an 

evaluation plan. In 2001-02 (the school’s 

final year of CSRD funding), the developer 

spent less time at the school, and 

encouraged staff to develop internal capacity

to support the method. Staff reported more 

assistance in Year 3 was needed, 

particularly for new teachers. 

No additional training from external

providers was identified. 

School R 

“Comprehensive Reform

Model”*

The method was so customized that no single 

“developer” or external technical assistance 

team was relevant. 

No other external support identified. 

*Locally developed methods are in quotes and have been given pseudonyms. 
2-21

 

being provided by parties other than the developer (Schools A, B, C, D, F, G, H, J, N, and P). 

However, within this subset, only rarely did the assistance address comprehensive school reform 

independent of the method. 

 The field teams concluded that perhaps the previously observed lower proportion of 

awareness and design for comprehensive reform—independent of the research-based method that 

a school has adopted—was in part because few consultants were available to provide this type of 

assistance,33 and further, because schools have not understood the need to seek such assistance. 

2.3.2 Both measurable goals and benchmarks (Component 4) and evaluation strategies 

(Component 8) have been devoted to tracking student performance, not necessarily 

implementation progress. Having measurable goals and benchmarks (Component 4) enables a 

school to monitor the progress of a school’s CSRD initiative. The actualization of any 

monitoring or evaluation activity, however, requires evaluation strategies (Component 8) and 

planning. For this reason the two topics are covered together. 

 For measurable “goals,” all but one of the 18 schools (School E, which became a new 

charter school in 2001-02) identified detailed student performance goals (see Exhibit 2-8).34 

However, rather than considering “benchmarks” as markers of implementation progress or other 

hypothetical steps (e.g., increased enrollment, hours spent, or time on task) needed to achieve the 

goals, many of the schools simply defined “benchmarks” by providing yearly breakdowns of the 

incremental performance gains that would extrapolate to meeting the overall performance goals. 

[As a promising reform strategy, see “Method-Recommended Curriculum Team Aligning Scope 

and Sequence to Provide Greater Opportunities for Interdisciplinary Projects,” in Appendix B.] 

The second column of Exhibit 2-8 shows a small group of schools (Schools B, C, G, J, O, and Q) 

that had defined and were actually monitoring implementation benchmarks, apart from any 

monitoring being done by the external developer. (As a promising reform strategy, see “External 

‘Critical Friends’ Group Provides Objective Assessment of Progress Towards Goals and 

Benchmarks,” in Appendix B.] For instance, the original language of Component 4 does focus 

on student performance and is readily reinforced by the attention that has been devoted to student 

performance in judging school performance. 

 

33For instance, no references during the site visits were made to assistance from the U.S. Department of 

Education’s regional educational laboratories, which might have been in a position to deal with comprehensive 

school reform and not just the implementation of a specific research-based method.

34Exhibit 2-8 merely indicates the existence of these goals and benchmarks, but does not list them. The field 

teams collected the actual lists, which are contained in the teams’ archival databases. The complete lists were too 

lengthy to include in this report, as most of them address target scores or percentage improvements for each grade 

level, for different academic subjects, and, in some cases, for different annual periods of time.
2-22

Exhibit 2-8 

GOALS AND BENCHMARKS (Component 4) AND 

EVALUATION STRATEGIES (Component 8) 

Benchmarks Evaluation School Name/ 

Method* Student Achievement Implementation Plan Activities

School A 

Accelerated 

Schools

Starting in 2001-02, 

school developed 

benchmarks to meet 

state proficiency 

levels by grade and 

subject 

Accelerated Schools 

Assessment Toolkit allowed 

school to develop 

benchmarks; process had 

not begun 

No formal plan; school 

used AS school 

evaluation 

questionnaire; other 

plans developed by 

individual cadres 

No written findings from 

cadres, who analyzed 

student achievement data, 

and developed plans to 

address deficiencies 

School B 

High Schools 

That Work

Included extensive 

coverage of student 

performance on 

assessments and 

toward graduation 

requirements 

Leadership team developed 

process benchmarks and 

indicators and reviewed 

progress annually 

State conducting a 

formal evaluation of 

CSRD 

implementation 

Formal instrument used to 

evaluate instruction during 

classroom observations 

School C 

Co-NECT

Benchmarking plans 

pointed to 

performance on state 

mandated tests 

Co-NECT benchmarks 

included shared 

accountability, projectbased learning, 

comprehensive 

assessment, team-based 

school organization, and 

use of technology 

No formal external 

evaluation; selfassessment for SIP 

process; and CONECT processes to 

measure 

implementation 

Conduct of parent, teacher, 

principal, and student 

surveys; review of 

achievement scores; 

conduct of focus groups; 

reports from external 

technical assistance 

providers 

School D 

Success for All

School expected to 

reach benchmarks 

established by the 

district targeting 

student performance 

(reading readiness, 

attendance, reading 

and math proficiency) 

Implementation and fidelity 

reviews conducted by 

external developer 

District contracted 

with a consultant to 

conduct evaluation, 

but turned 

responsibility over to 

schools in 2001-02 

End-of-year report 

contained findings 

mandated by district; SFA 

made bi-annual visits to 

the school to monitor 

implementation; no 

evidence of formal 

evaluation report 

School E 

Coalition of 

Essential Schools

In 2001-02, the 

benchmarks and 

goals were unclear, 

after the school 

restructured as a 

charter school 

No implementation 

benchmarks 

No recent contact with 

external evaluator 

Prior activities of an 

external evaluator were 

lost during the school’s 

transition; no activities 

were planned for 2001-02 

School F 

Coalition of 

Essential Schools

State establishes 

benchmarks for 

student achievement 

scores incorporated 

annually in SIP 

School met first set of 

implementation 

benchmarks, related to 

participation in collaborative 

groups and development of 

academies, but did not 

establish new benchmarks 

pertaining to full reform 

District intends to 

conduct an impact 

assessment (using 

survey and student 

achievement data); 

no evidence of 

written plan 

School completed surveys; 

district analysis of data in 

progress; no written finding 

available

*Locally developed methods are in quotes and have been given pseudonyms. 

(Continued on next page)
2-23

Benchmarks Evaluation School Name/ 

Method* Student Achievement Implementation Plan Activities

Exhibit 2-8 (Continued)

School G 

“Special Literacy 

Model”*

Benchmarks follow

state guidelines for 

student performance 

School established process 

benchmarks, relating to 

faculty participation, parent 

participation and student 

absences 

No external 

evaluation; selfassessment as part 

if year-round SIP 

processes 

Conduct of parent, teacher, 

principal, and student 

surveys; review of 

achievement scores; 

conduct of focus groups; 

reports from external 

technical assistance 

providers 

School H 

Success for All

SIP describes 

benchmarks for 

student performance, 

as well as goals for 

attendance and

school ranking

No implementation 

benchmarks, other than 

SFA implementation 

monitoring 

No formal activities or 

plans 

Evaluation of student 

performance based on

state assessment; 

implementation monitored 

by SFA; no written 

findings available 

School I 

Lightspan

No clear benchmarks in 

place; did have goal 

of increasing scores 

on state assessment 

No clear benchmarks in 

place; did have goal of 

increased parent 

participation 

No clear evaluation 

plan

School frequently reviews 

student achievement 

data; developer

previously used

questionnaires and 

surveys to measure 

implementation; nor 

written evaluation 

School J 

“Performance 

Tasks Model”*

Goals/benchmarks set 

as part of SIP 

process 

School set implementation 

benchmarks annually (e.g., 

teacher must prepare one 

performance task to be 

accepted by Performance 

Task model database) 

External evaluator 

focused on 

standards, 

performance 

measures and 

alternative 

assessments (no 

formal plan) 

Schools reviewed and 

discussed student 

achievement data, 

compared current data 

with data from earlier 

years, other schools and 

district and state level 

data 

School K 

Success for All

Goals/benchmarks set 

as part of SIP 

process and covered 

limited subjects 

No implementation 

benchmarks 

School did not have a 

clear evaluation

plan

Schools reviewed and 

discussed student 

achievement data, 

compared current data 

with data from earlier 

years, other schools and 

district and state level 

data, and utilized SFA 

eight-week assessments 

School L 

“Behavioral 

Modification

Model”*

School expected to 

reach benchmarks 

established by the 

district targeting 

student performance 

(reading readiness, 

attendance, reading 

and math proficiency) 

Implementation benchmarks 

were stated in the CSRD

application, and monitored 

by the external developer 

District contracted 

with a consultant to 

conduct evaluation, 

but turned 

responsibility over to 

schools in 2001-02 

End-of-year report 

contained district findings; 

final evaluation of CSRD 

delayed

*Locally developed methods are in quotes and have been given pseudonyms. 

(Continued on next page)
2-24

Benchmarks Evaluation School Name/ 

Method* Student Achievement Implementation Plan Activities

Exhibit 2-8 (Continued)

School M

Lightspan

District established 

benchmarks for 

student performance, 

covering all grades 

and subjects 

No clear implementation 

benchmarks 

Evaluation activities 

suggested by

Lightspan (no 

written plan) 

Findings indicated that 

Lightspan students 

outperformed comparison 

students in the state 

assessment (report 

available after the 2001-

02 SIP developed) 

School N 

Success for All

SIP set benchmarks for 

performance, as well 

as goals around

student attendance 

and school ranking 

School had goals for parental

involvement and conflict 

resolution, but no 

information on how goals 

would be measured or 

benchmarks established 

No formal activities or 

plans 

Evaluation of student 

performance based on

state assessment; 

implementation monitored 

by SFA

School O 

Success-in-theMaking

District-mandated 

benchmark tested as 

part of 

comprehensive

reading program

receiving greatest 

priority. 

Scale-up benchmarks had 

been met; method in use in

all classrooms

District intends to 

conduct an impact 

assessment (using 

survey and student 

achievement data); 

no evidence of 

written plan 

School completed surveys; 

district analysis of data in 

progress; no written 

finding available

School P

HSTW

Included extensive 

coverage of student 

performance on

assessments and 

toward graduation 

requirements 

Listed as addressing

organization and 

curriculum, and instruction 

State conducting 

formal evaluation of 

CSRD 

implementation

Formal instrument used to 

evaluate instruction 

during classroom 

observations 

School Q 

Accelerated

Schools

District established 

benchmarks based 

on student scores on 

state assessment 

Benchmarks in form of 

implementation rubric, 

including unity of purpose, 

empowerment, building on 

strengths, forging a vision, 

taking stock, setting 

priorities, establishing 

governance, embedding

inquiry, and creating a 

collaborative culture 

No external evaluator; 

district provided

assistance to 

schools in analyzing 

student 

achievement 

School reviewed student 

achievement data; cadres 

made recommendations 

for curricular or

instructional reform 

School R 

“Comprehensive 

Reform Model”*

Three-year 

benchmarks in SIP 

for reading, writing, 

and mathematics 

Three-year benchmarks in

SIP for use of technology

and cooperation 

No written evaluation 

plan (except for plan 

for technology 

component, 

conducted as a

state requirement) 

No formal reporting of 

evaluation findings

*Locally developed methods are in quotes and have been given pseudonyms. 
2-25

 At the same time, the original CSRD language for the evaluation strategies 

(Component 8) does indeed address the implementation of reform, but the existing 

evaluation activities also have largely focused on assessing student performance (third and 

fourth columns in Exhibit 2-8.) [As a promising reform strategy, see “Clear Scope and 

Sequence and Quarterly Assessments Provide Frequent Measures of Student Growth,” in 

Appendix B.] 

2.3.3 High turnover among staff and students has resulted in transient levels of 

support within the school (Component 5) or parental and community involvement 

(Component 6). The field teams found it difficult to assess these two CSRD components 

because schools do not routinely monitor or maintain precise records regarding either type 

of support or involvement. In some cases, faculties did hold votes during the formative 

period when a research-based method was first being adopted. However, documentation 

for any subsequent faculty support was absent thereafter. Similarly, schools might have 

kept records of parent attendance at one or two key events during the year (e.g., the first 

open-school night), but they did not formally monitor or document other parental or 

community involvements. 

 Despite the lack of records, the field teams made the following observations about the 

two components. First, high turnover among staff (e.g., Schools B, E, I, M, and R) or 

students (e.g., Schools C, E, L, and P) has meant that the extent of staff support or parental 

involvement can be transient: Any judgment regarding support or involvement therefore 

depends upon the timing of the data collection “snapshot.” Unfortunately, several of the 

schools had staff who were highly supportive of the reform effort but who then left the 

school (e.g., Schools C, E, M, and R); assessing staff support earlier would have shown 

strong internal support for the method and the reform effort, but such support likely 

dissipated greatly by the time of the site visit. At other schools, support among the staff 

increased over time as individual teachers became more familiar with the reform efforts. In 

a likewise manner, student turnover meant that the relevant body of parents also changed, 

and high involvement during the kickoff year of a reform effort may not have been 

sustained the following year. 

 Second, strong principal leadership in initiating and supporting reform was a critical 

ingredient at many of the schools. [As a reform strategy, see “School Leadership Committed 

to Institutionalizing Reform Efforts,” in Appendix B.] However, turnover among principals 

also swayed support or involvement if, as is usually the case, the new principals had agendas 

and priorities different from their predecessors (e.g., Schools L, M, and R). 

 Interestingly, any difficulties in assessing support and involvement also may be 

considered substantive findings, not just methodological barriers. Whereas time-limited 

innovations may call for staff support or parental involvement over shorter periods of time, 

comprehensive reform calls for broad support and involvement. The team found that 

schools had defined few if any strategies to produce such broad efforts. Other than a few 
2-26

 

cases where either staff support for the reform effort (e.g., Schools A and F) or community 

involvement (e.g., Schools O and K) were strong and persistent, the observed support and 

involvement only involved isolated milestones (e.g., the original vote by a faculty to adopt 

a research-based method, the convening of a parents’ night, or the use of CSRD funds to 

hire a parent coordinator) but not sustained trends or patterns. [As promising reform 

strategy, see “Internal Collaborative Meetings Foster Inter-curricular Teaching and Teacher 

Buy-in to Schoolwide Issues,” and “Parents and Community Members Participate in 

School Governance and Goal Setting,” in Appendix B.] 

 For other reasons related to the comprehensiveness of the reform effort, support or 

involvement also can be spotty. For instance, schools whose reforms were limited to single 

academic subjects would not necessarily have involved staff support or parental 

involvement outside of these subjects. Similarly, methods being scaled-up incrementally 

by grade level would not necessarily have called for universal support or involvement 

during the scale-up process. 

2.3.4 Most schools are in a position to coordinate or converge resources 

(Component 9), but resources for sustainability are still uncertain. The topics of 

coordination and sustainability (i.e., extending reform efforts beyond the period of CSRD 

funding) were joined because schools’ prospects for sustainability have been heavily driven 

by the need to identify new sources of funds.35 Similarly, the resources to be coordinated 

under Component 9 have largely been considered budgetary resources by the schools. 

 The significance of Component 9 also is related to the underlying spirit of reform. By 

definition, comprehensive reform should not be implemented as an “add-on” or incremental 

activity, but should affect all school operations and hence the entire set of a school’s 

resources—core building budget as well as funds from external sources not legally needing 

to be dedicated to a specific activity (e.g., Title I, Title II, Eisenhower, and other federal 

and state funds). Any actual and ongoing comprehensive reform would therefore logically 

require that the school’s resources be applied in a coordinated fashion. 

 To collect data on these two topics of coordination and resources for sustainability, 

the field teams did focus on budget data but also collected other information, mainly from 

the principal, regarding the school’s funding intentions and priorities during the coming 

year(s). At 10 of the 18 schools, the field teams found evidence that both external and local 

funds were being coordinated in some manner to support the reform effort at the school. In 

most cases, the majority of that coordination involved Title I funding (see Exhibit 2-9). 

 An initial observation is that coordination can be greatly facilitated if the planning for 

all of a school’s funds is represented within a single school improvement plan that 

35Section 3 discusses sustainability more broadly.
2-27

Exhibit 2-9 

STATUS OF COORDINATION OF RESOURCES 

(Component 9) 

AND SUSTAINABILITY 

School Name/ 

Method* Types of Resource Coordination Evidence of Sustainability

School A 

Accelerated 

Schools

School already used Title I, other 

external funds, and central budget— 

along with CSRD funds—to support 

reform. Funds appeared to be used in 

a coordinated manner in covering 

various reform activities. 

The principal was committed to providing the needed 

resources for the foreseeable future. ED and the 

state recognized the school for its reform efforts. The 

maintenance support needed by the research-based 

method is assumed to be modest. 

School B 

High Schools that 

Work

Nearly all CSRD funds were directed at 

professional development (with the 

addition of some Title II and other 

funds). The coordinator position was 

paid from the core budget. 

No funding streams appear to be available to continue 

school’s relationship with HSTW following the end of 

CSRD. The merger of this high school with a county 

school adds further uncertainty about sustainability of 

reform efforts, though the site visit team believes the 

merger will actually further reform efforts. 

School C 

Co-NECT

The SIP embraces the use of funds from 

all sources as part of an integrated 

allocation plan. 

The principal hopes to use Title I funds to continue 

after CSRD has ended. In addition, if the school can 

become a “demonstration” school, the developer of 

the research-based method also would then provide 

its own support. 

School D 

Success for 

All/Roots & Wings

In 2001, district started requiring schools 

to have single plan to converge 

resources, including CSRD funds, with 

common objectives (previously, each 

funding source had its own plan). 

Convergence included external funding 

sources, but school reports no 

additional local funds available due to 

fiscal crisis. 

The principal did not think the school would continue to 

use SFA after CSRD funding ended for two reasons: 

1) a district move to site-based budgeting meant that 

the school would have to identify funds to fill gap left 

by CSRD (when large state cutbacks were looming), 

and 2) the SFA reading curriculum was not submitted 

for state review, and, therefore, the school could not 

use state textbook funds to purchase SFA reading 

materials. 

School E 

Coalition of 

Essential Schools

A previous pattern of coordinating funds 

cannot be applied to the new charter 

school. 

The school has begun to be interested in adopting an 

entirely new research-based method, which may 

overlap with CES; most teachers show no interest in 

formally continuing with the CES method. 

School F 

Coalition of 

Essential Schools

School has been coordinating funds 

from multiple external and internal 

sources to support reform effort in a 

converging manner. 

The school is searching for additional funds to replace 

CSRD funding. However, main reform initiatives, 

some of which involved changes in school scheduling 

and operations and not an outlay of funds, are likely 

continue independent of any funding success. 

School G 

“Special Literacy 

Model”*

CSRD funds were used almost entirely 

for professional development, along 

with Title I and other local funds. 

Principal was confidant that new grant monies, along 

with Title I, would continue to support future reform 

activities.

*Locally developed methods are in quotes and have been given pseudonyms. 

(Continued on next page)
2-28

School Name/ 

Method* Types of Resource Coordination Evidence of Sustainability

Exhibit 2-9 (Continued) 

School H 

Success for All

Convergence of resources from many

different external sources appeared to 

have taken place because the school 

and district coordinate school’s funds in 

relation to the school improvement 

plan. 

The school decided to transition to the district 

curriculum and end support for the external SFA 

developer beginning in the 2001-02 school year. 

Nevertheless, teacher indicated that they intended to 

continue using SFA-related classroom management 

techniques and questioning strategies. 

School I 

Lightspan

School has funds from many external

sources and prides itself in its 

entrepreneurship. However, multiple

sources may have been more often 

associated with fragmentation rather 

than coordination of efforts. 

Although School I’s reform efforts could be described 

as unfocused, teachers and administrators expressed 

hope in Spring 2002 that things were beginning to 

improve. The 1st grade teachers were

enthusiastically using Lightspan in the absence of 

external support. 

School J 

“Performance 

Tasks Model”*

Some local funds were used in concert 

with CSRD to support the 

implementation of the research-based 

method. 

The school believes it can support the reform effort 

with Title I funds. However, the assignment of a new

principal to the school may affect that plan; the 

associate principal being considered for the position

stated he would support a modified use of the 

method, with only some components remaining. 

School K 

SFA

School has used multiple sources, 

including Title I and state grants, to 

support reform. Degree of 

convergence is unclear. 

Title I funds are expected to be used to continue SFA 

support after CSRD funding ends. However, 

continuation also depends on improved student 

performance, as well as SFA’s response to teacher’s 

concerns. 

School L 

“Behavior 

Modification

Model”*

In 2001, district started requiring schools 

to have single plan to converge 

resources, including CSRD, with

common objectives. Process has 

brought all dollars into a single reform 

effort. However, school reports no 

additional local funds available due to 

fiscal crisis. 

Funds to replace CSRD were still being sought, but a 

new administration and waning staff support 

appeared to be bigger threats to sustainability. By

Spring 2002, teacher implementation of some

components of the method were already beginning to 

lapse. 

School M

Lightspan

A former principal appeared to have 

converged resources around 

Lightspan, its coordinator, computer 

equipment, and the other resources 

needed for implementation; by 2001-

02, no funds were used to support 

implementation.

The school’s investment in purchasing the components 

of the method (the PlayStations, software, and 

television monitors), is evidence that some aspect of 

the method will remain. By the end of the third year, 

however, the method was used sparingly in one 

grade level. The school also learned that it would be 

merged with another “underperforming” elementary

school in 2002-03, adding to uncertainty about future 

use of Lightspan. 

School N 

SFA

School and district coordinated Title I 

funds with CSRD funds as part of the 

school improvement plan. However, 

school has had few other discretionary

sources. 

Withdrawal of district support for the research-based 

method jeopardized sustainability of reform effort 

beyond CSRD funding. The principal was skeptical

about whether the school would continue to 

implement CSRD without new funding, and the staff 

indicated that the school would not be implementing 

SFA in 2002-03, even if funding were available.

*Locally developed methods are in quotes and have been given pseudonyms. 

(Continued on next page)
2-29

School Name/ 

Method* Types of Resource Coordination Evidence of Sustainability

Exhibit 2-9 (Continued) 

School O 

SIM

School successfully coordinated funds 

from multiple sources (e.g., Title I and 

Title II, as well as local funding), in 

supporting reform activities. 

Because reform had been in progress prior to CSRD, 

school anticipated continuing reform after CSRD. 

External costs associated with research-based 

method were likely to diminish, and Title I and other 

funds would continue to be used, but the school also 

was exploring the availability of new sources of 

funding. 

School P

HSTW

School converged federal and local 

funds, primarily for professional 

development activities related to the 

reform effort. 

School and district staff strongly support continuation 

of the reform effort. Extent of need for funds to cover 

external costs associated with the research-based

method were unclear. School staff were also unclear 

how the merger with the city high school would affect 

reform efforts. 

School Q 

Accelerated

Schools

School had started using other external 

funds, in a converging manner to 

support variety of reform activities prior 

to CSRD funding, and such 

convergence has continued along with

CSRD funding. 

Main threat to sustainability is continued high rate of 

staff turnover as well as threat of principal turnover. 

The principal hoped that the formation of a leadership

team would make the reform efforts less dependent 

on any single individual and therefore serve as a 

mechanism for sustaining reform. 

School R 

“Comprehensive 

Reform Model”*

School had not used a large portion of its 

CSRD funds in its final year of funding

and obtained an extension on the grant 

(and, consequently, had not converged 

or coordinated funds). 

Some isolated initiative would be sustained, nut not as 

part of a overall reform design; greater attention was 

placed on the district-required “small learning

communities. 

*Locally developed methods are in quotes and have been given pseudonyms. 

converges on common goals and objectives (see the coordination column of Exhibit 2-9). 

Of the 10 schools exhibiting converging funding sources, 3 had such a condition (Schools 

C, G, and H). Even in the absence of such site-based management conditions, 7 other 

schools reported coordination or convergence of resources to the field teams (Schools A, B, 

F, I, K, O, and P), with 3 of them mentioning integration with the building’s budget. Thus 

a total of 10 of the 18 schools appear to have been in a position to coordinate or converge 

their resources. Another 5 schools (D, J, L, N, and Q) reported coordination of federal 

funding sources to support reform, but did not have the necessary access to local funds in 

order to redirect them (in the case of School D and L, the principals reported that they had 

the ability to access local funds, but severe budget cuts meant that after obligatory 

allocations—i.e., teacher salaries—no funds remained). (See “Move to Site-Based 

Management Allows School to Utilize Slack Resources for Reform Initiatives,” in 

Appendix B.) Reports of lack of coordination came from only 3 schools: School E, which 

had ceased implementation; School M, which had made the capital expenditures associated 

with Lightspan and was now implementing the method at no cost; and School R, which had 

not really implemented CSRD, as previously discussed. 
2-30

 At the same time, the field teams found that the ability to coordinate funds did not 

necessarily coincide with the extent of a school’s discretionary resources. Yet, the latter 

may be critical for sustainability (see the second column of Exhibit 2-9). 

 One reason for the need for discretionary resources is that a major CSRD 

component—e.g., the use of a research-based method—usually requires an outlay of 

additional funds, for such items as: a) on-site assistance, b) release time for professional 

development, c) salary support for on-site coordinators, d) the purchase of new types of 

supplies, and in some cases, e) hardware and software purchases. The availability of the 

CSRD funds has served well the needs of the startup periods in implementing these 

methods. Although the extent of outlays for maintaining a method after startup are 

reported to diminish, nearly all of the schools are unclear about the amount or source of any 

new funds needed after the CSRD award has ended. Title I funds are mentioned frequently 

as one source of such funds, but whether they will be sufficient also is unclear. 

 The likelihood of sustainability also will be affected by non-funding conditions— 

mainly the possible lack of support for reform on the part of the school staff or the threat of 

staff or principal turnover. Of the 18 schools, 5 have reported such jeopardy (Schools B, D, 

L, M, and Q), in addition to School H, which already stopped implementing the researchbased method. Section 3.5 examines this issue in greater detail. 

2.4 SUMMARY

 Across the 9 CSRD components, schools have devoted their greatest implementation 

efforts to the research-based method (Component 1); professional development 

(Component 3)—usually linked with the method; and external technical assistance and 

support (Component 7)—also usually linked with the method. Schools have established 

clear student performance benchmarks (Component 4). All of the other components have 

been less well developed.
3-1

3. CSRD IMPLEMENTATION AND SUSTAINABILITY: 

THE INTEGRATED FINDINGS

 This section first discusses district and state influences on the CSRD schools. The 

section then provides a summary assessment of CSRD implementation, using the district 

and state influences, along with other conditions, to identify three potentially different 

pathways to implementation. Finally, the section discusses the prospects for CSRD 

sustainability at the 18 schools. 

3.1 DISTRICT AND STATE INFLUENCES ON CSRD

 Although CSRD focuses on school activities, district and state conditions can be 

extremely important to the administration of CSRD. The federal CSRD awards are to state 

education agencies, who in turn make subawards to districts on behalf of specific schools. 

As a result, any number of CSRD processes and outcomes may be influenced by the district 

and state policies and practices. 

 To examine district and state conditions, the field teams conducted interviews, 

collected documents, and reviewed other data from the 18 schools’ district and state 

agencies. For the districts, site visits were made (in addition, district officials were 

occasionally interviewed while present at the schools at the time of the school site visit). 

Information about the state agencies more frequently came from short telephone 

conversations with state officials, searches of the agencies’ Web sites, and copies of state 

documents and reports provided by the district offices. 

 Exhibit 3-1 shows the various district and state conditions that appeared relevant to 

the CSRD activities in the 18 schools. The exhibit pairs the schools, with each pair being a 

part of the same district and state, reflecting the original sample design. For each pair, the 

exhibit then enumerates salient conditions that might affect the work of CSRD, dividing the 

salient conditions into two categories: assistance or support in administering CSRD; and 

other policies or conditions potentially affecting CSRD implementation though not a direct 

part of administering CSRD. Each category is discussed in turn. 

 

 3.1.1 Administering CSRD: Districts and states provide varying degrees of support. 

Both states and districts have explicit responsibilities in administering the funds for CSRD. 

However, beyond these core responsibilities, both districts and states varied widely in 

providing other forms of CSRD support or assistance. At one extreme (e.g., Schools M and 

R), the state education officials engaged schools in the CSRD application process; made 

site visits to monitor the initial progress of the schools; and convened annual meetings of 

the CSRD schools in the state, to facilitate the exchange of information on 
Exhibit 3-1 

SCHOOL, DISTRICT, AND STATE CONDITIONS 

Schools State Conditions District Conditions

School A 

Accelerated 

Schools 

and 

School E 

Coalition of

Essential Schools

Non-CSRD related:

! State sets benchmarks for 

improvement based on state 

assessment. Schools that fail to 

make progress, over time, may be 

closed. School E has been in lowest 

category and School A has been “in 

need of improvement,” in the state 

system. 

Non-CSRD related:

! The district essentially only provides 

assistance on an as-needed basis. 

Professional development funds are 

distributed to schools; schools decide how

to use them. Other assistance only 

provided when requested by the schools. 

! District sets benchmarks for improvement on

state assessments. 

! District has vacillated on future of the School

E, now has decided to continue the school 

as a charter school.

School B 

HSTW 

and 

School P 

HSTW

Non-CSRD related:

! No state assessment has yet appeared

related to reform. 

CSRD related:

! District has provided staff support and 

professional development in relation CSRD.

Non-CSRD related: 

! The district’s superintendent resigned effective 

the end of 2000-01. 

! Board votes to merge Schools B and P into a

single high school, starting in 2003-04.

School C 

Co-NECT 

and 

School G 

“Special Literacy 

Model”*

Non-CSRD related:

! Performance on state assessment is 

used to set goals. The state also 

provides a rubric for the 

development of district plans that is 

tied to state standards, requiring 

school plans to link funded activities 

with goals, benchmarks, and 

strategies. 

CSRD related:

! The district provided workshops in plan 

development to ensure alignment. 

Non-CSRD related:

! The district is suffering the loss of teachers due 

to low teacher salaries and teacher 

shortages. Nearby districts and states are 

offering bonuses to attract teachers. 

! The district has had a serious fiscal shortfall. 

School D 

Success for All 

Roots & Wings

and 

School L 

“Behavior

Modification

Model”* 

CSRD related:

! Because of state timing in notifying 

schools of eligibility and application 

deadline, applications were written 

over the summer. At School L, 

many staff were not present to 

participate in selection of the CSRD 

method. 

Non-CSRD related:

! State has put into place new high

stakes examinations. 

! State revising assessments to make 

them standards-based. 

CSRD related:

! District decided which schools should apply;

hired consultant to write CSRD proposals 

for the schools.

Non-CSRD related:

! District’s strategic plan adopted under new

superintendent in 1998, calls for a new

emphasis on high standards of academic 

performance and has been a driving force 

behind school reform efforts.

*Locally developed methods are in quotes and have been given pseudonyms. 

(Continued on next page)

3-2
Exhibit 3-1 (Continued) 

Schools State Conditions District Conditions

School F 

Coalition of

Essential Schools 

and

School O 

Success-in-theMaking (SIM)

CSRD related:

! State conducted monitoring visits to 

each site during the past year, and 

provided assistance. 

Non-CSRD related:

! New state accountability system holds 

teachers and schools accountable 

for student progress. School O 

received an “F” rating which hurt its 

magnet program (and which they

believed was unfair). 

CSRD related:

! District office provides extensive support to 

schools at the start of CSRD implementation

and conducts a mid-year checkup. 

Non-CSRD related:

! School O originally intended to implement CoNECT, but was “encouraged” by the district 

to expand existing SIM method (after 

receiving an “F” rating). 

! District implementing a reading program that 

will phase out SFA.

School H 

Success for All 

and 

School N 

Success for All

CSRD related:

! The curriculum match between the 

CSRD method and the state 

assessment is not perfect, causing 

School N to add a reading program 

specifically on skills addressed by

the state assessment. 

Non-CSRD related:

! The state assessment and state 

accountability system is driving 

instruction in the schools. 

CSRD related:

! The previous superintendent strongly 

supported the use of external methods for 

reform and provided local funds to pay for 

them. 

! A new superintendent has ceased that 

practice, and only schools with CSRD funds 

currently have external methods. Schools H 

stopped using its external method in 2001-

02, and School N may have difficulty finding 

budget support for external methods after 

CSRD; other schools have adopted a local 

reading model aligned with the state 

assessment. 

Non-CSRD related:

! The district also has started interim and highstakes assessments, aligned with state 

standards.

School I 

Lightspan 

and 

School Q 

Accelerated 

Schools

CSRD related:

! State education staff conduct site 

visits to CSRD schools, also 

providing assistance.

Non-CSRD related:

! Simultaneous with the CSRD award, 

state budget cuts resulted in a 

several million dollar cut in district 

funding, in part resulting in the loss 

of the district’s P.D. department and 

cuts to program evaluation. 

CSRD-related:

! The district identified schools that were ready

for CSRD and notified only those schools 

about the availability of CSRD funding. 

Non-CSRD related:

! District has helped schools prepare for state 

assessment and to use data to improve 

instruction. 

! District has now gone two years with no 

professional development department.

School J 

“Performance

Tasks Model”* 

and 

School K 

Success for All

Non-CSRD related:

! State designated School K as

“needing improvement,” requiring it 

to select a research-based method 

using CSRD funds. The state also 

designated School J as a “priority

school,” receiving extra funds to 

support improvement efforts and 

being assigned a “critical friend.” 

! The state assessment is standards 

based. 

CSRD related:

! CSRD is coordinated by the district’s director 

of federal and state programs, the assistant 

superintendent for instruction, and the 

director of research and evaluation. 

Non-CSRD related:

! District hired a new superintendent in 2000-

01. Superintendent supports reform but is 

focused on reorganizing the district’s central 

office. 

*Locally developed methods are in quotes and have been given pseudonyms. 

(Continued on next page)

3-3
3-4

Schools State Conditions District Conditions

Exhibit 3-1 (Continued) 

School M 

Lightspan 

and 

School R 

“Comprehensive 

Reform Model”* 

CSRD related:

! State convenes CSRD staffs annually

to reinforce and support schools’ 

efforts. 

! State makes site visits to CSRD 

schools, also providing assistance. 

Non-CSRD related:

! School M was in state’s lowest

category, reducing staff morale, and 

esteem. 

! State’s restructuring of school finance, 

and budget cutbacks, disrupt district 

and schools. 

CSRD related:

! CSRD administered by Title I coordinator. 

! District helped School R develop its CSRD 

application. 

Non-CSRD related:

! District assessments are standards-based and 

aligned with state. 

! However, School M received district quality

performance award in same year as state 

placed it in lowest underperforming 

category.

*Locally developed methods are in quotes and have been given pseudonyms. 

CSRD experiences. In a similar manner (e.g., School L), a district completed the 

applications and selected the research-based method on behalf of the school. At the other 

extreme (e.g., School E) the state played little or no role beyond making the CSRD awards, 

and the district also offered little assistance to these schools in administering CSRD 

(School A). Exhibit 3-1 shows how the other states or districts assumed different postures 

in the CSRD administrative process. 

Among all the variants, an intermediate state and district posture would seem to be the 

best in serving comprehensive school reform. Such reform requires strong internal staff 

support within the school, but such reform is aimed at improving student achievement— 

which is assessed through district or state instruments (or both). In the most active role, if a 

district selects the candidate schools to be in the CSRD program, writes their proposals, and 

defines the research-based methods to be adopted, or if the state requires the school to 

participate in CSRD because of low achievement scores, the school may then have a poor 

understanding of its CSRD mission and may not become sufficiently empowered to 

undertake serious reform. In the most passive role, if districts and states leave schools 

entirely alone to deal with CSRD, the schools may receive insufficient guidance to 

understand the most desired curriculum and instructional practices relevant to the district or 

state’s student assessment instruments. The hypothesized intermediate role would be 

where districts and states: a) encourage and then support schools to make their own 

decisions about method selection and reform design—explicitly defining the alignment 

between these choices and the content of the district or state assessments, b) engage schools 

in developing their CSRD applications from their outset, and finally c) provide technical or 

other assistance to assure the continued alignment between the school’s reform initiative 

and the student outcome measures against which the success of the initiative will be 

judged. 

3.1.2 District and state policies not directly related to administering CSRD 

nevertheless can influence CSRD implementation. Exhibit 3-1 also enumerates many 
3-5

district and state policies not directly related to administering CSRD. These policies reflect 

ongoing district and state priorities independent of the CSRD program, and an important 

benefit of CSRD was its complementarity with these ongoing priorities. 

 The original field protocol did not consider most of these conditions external to the 

administration of CSRD to be relevant topics of data collection. However, the field teams 

quickly encountered a host of external events that appeared to affect CSRD implementation 

directly. The conditions found included: 1) extremely limited professional development 

resources (see Schools I and Q); 2) the direct alignment of schools’ CSRD designs with 

district plans and state standards (Schools C and G); 3) districts reducing financial support 

for all external methods (Schools H and N); 4) decisions to allow a CSRD school to 

become a charter school (School E); 5) decisions to merge two CSRD schools (Schools B 

and P). Many states and districts also were suffering severe revenue shortfalls during this 

period of time, creating an uncertain if not disruptive environment for school reform. 

 These non-CSRD-related district and state policies were found to be sufficiently 

important that Exhibit 3-1 divides the schools into two categories—one in which there 

appeared to be strong “vertical” influence (from state to district to school) and the other in 

which such influence appeared to be neutral. Strong influence could lead to either 

complementarity or conflict with CSRD. On the complementary side, CSRD provided 

Schools J and K, for instance, with resources and a compatible reform agenda to respond to 

their designation as underperforming schools in the state’s accountability system. On the 

conflicting side, the shifting content of state assessments could lead districts to use 

resources for alternative curricula and professional development that were contrary to those 

involved in the schools’ original CSRD plans or implementation (e.g., Schools H, N, and 

O). Likewise, a report suggesting that schools with block schedules performed more 

poorly on state assessments led another CSRD school (School B) to reconsider its ongoing 

implementation of a research-based method requiring the use of such schedules. 

 In general, these external state and district conditions appear to be highly relevant to 

CSRD implementation. In fact, any tendency toward a “vertical” alignment among state, 

district, and school policies and practices may not only impact CSRD implementation but 

may very well be critical to the long-run sustainability of CSRD, discussed later in this 

section. 

3.2 SCHOOLS IMPLEMENTING COMPREHENSIVE SCHOOL REFORM

 Given the preceding discussion of district and state conditions, the analysis now turns 

to a summary assessment of the extent to which the 18 schools have been implementing 

CSRD. In this assessment, the more that a school was implementing the nine CSRD 

components, the greater was judged its strength of CSRD implementation. 
3-6

 

 Implementation Instrument and Measurement. Assessing the strength of 

implementation of a complex school intervention, such as comprehensive school reform, 

has neither a standard methodology nor a standardized instrument. As a result, the FieldFocused Study developed and used its own assessment instrument, shown in Appendix E. 

This assessment instrument is but the start of a tool with which to measure reform and 

requires much further work to formalize. Nevertheless, the instrument should provide a 

solid framework upon which others can improve. 

 The instrument covers the 9 components, but deliberately weights the components 

differently. The weights for the different components reflected the Field-Focused Study 

team’s view of the importance of the 9 components, based on observations during the early 

rounds of site visits. The research-based method (9 points) and the comprehensiveness of 

design (11 points) have the highest proportion of the total points; professional development 

emerged as the third most impactful component; and the other components followed. This 

weighting system should be refined during future evaluations. 

 The instrument asked the site visit teams to use their case reports and provide 

“objective” information for somewhat less than half of the items and to make “subjective” 

ratings for the remainder of the items.36 The teams had sought accurate and precise data on 

objective items, but such data were not always available or retrievable, and in many cases 

the “objective” response was based on the team’s best estimate. The scores for each school 

were tallied, and the results are shown in Exhibit 3-2. 

 The 47-point instrument’s strength is that is operationalizes CSRD’s nine 

components. With no standardized instrument available to measure CSRD, the instrument 

allowed the evaluation team to break global concepts of reform into smaller operational 

components. If a school successfully implemented all aspects of all 9 components, 

therefore, it could achieve a maximum score of 47 points (thus the instrument has been 

referred to as the “47-point instrument”). The greatest threat to measuring reform in this 

manner is that it risks losing site of the “whole picture.” As a result, the final tallies were 

checked against a global judgment by the teams. 

A weakness of the instrument is that the Field-Focused Study did not have sufficient 

time or resources to establish the validity or reliability of the instrument to a level that 

36 For objective information, examples in the instrument included (see Appendix E): 2.1) the existence 

of a written design or plan for school reform; 2.2) the contents of the plan; 3.2) the range of professional 

development days required or taken by the average teacher per year; and 4.1) the number of academic 

subjects covered by measurable goals and benchmarks. For subjective information, examples of items 

included (also see Appendix E): 1.1) the implementation score for the research-based component; 3.1) 

whether professional development had a strong content focus; 5.3) interviewees voicing strong support or 

enthusiasm for reform; and 6.2) the level of parental involvement. 
COSMOS Corp

oration, 2003 3-7

Exhibit 3-2 

SCORES FOR CSRD IMPLEMENTATION STRENGTH AT 18 SCHOOLS, 

USING 47-POINT INSTRUMENT* 

CSRD Component CSRD School

A B C D E F G H I J K L M N 

1. Research-Based Method (9 points)

Bodilly Score 5 3 3 5 1 4 5 1 3 4 4 2 3 5 

% Classrooms Using 1 .75 .75 1 0 .75 1 0 1 .9 .9 .8 .1 

Fidelity Rating 3 2 2 3 1 2 3 1 1 2 3 2 1 2 

2. Comprehensive Design (11 points)

Written Design or Plan 1 1 1 1 1 1 1 1 1 1 1 1 1 

Contents of Plan 7 6 5 7 4 7 6 6 4 7 5 7 2 4 

Breadth of Plan 3 3 3 3 2 3 3 2 2 3 3 3 2 2 

3. Professional Development (7 points)

Strong Content Focus 1 1 0 1 0 1 0 1 1 1 1 1 0 

Range of PD Days 3 2 2 2 2 3 3 2 3 2 3 2 2 2 

Collective Participation 1 1 1 1 1 1 1 1 1 1 1 1 1 

Classroom-Based PD 1 1 1 1 0 1 1 1 1 1 1 0 0 0 

Alignment of PD 1 1 1 1 0 1 1 1 1 1 1 1 1 

4. Measurable Goals and Benchmarks (4 points)

Number of Academic Subjects Covered 2 3 2 2 0 2 2 3 2 2 1 2 3 2 Number of Grades Covered 1 1 1 1 0 1 1 1 .8 1 1 1 1 

5. Support within the School (4 points)

Formal Vote on Method 1 1 1 1 0 1 1 1 0 1 1 0 0 Support Exceeded 75 percent 1 0 1 1 0 1 1 1 0 1 1 0 0 

Support Voiced for Support 1 0 1 1 1 1 1 1 1 1 0 0 1 0 No Dissent Voiced 1 0 0 1 0 1 1 0 1 1 0 0 0 

* See Appendix E for a copy of the instrument

(C
CSRD Component CSRD School

A B C D E F G H I J K L M N 

Exhibit 3-2 (Continued)

COSMOS Corporation, 2003 3-8

6. Parent and Community Involvement (4 points)

New Forms of Parent Involvement 1 1 0 1 0 1 1 1 1 1 1 1 1 

Level of Involvement 2 1 0 1 0 1 1 1 1 0 2 0 2 2 

At Least One Community Org. 1 1 1 1 1 1 0 1 1 1 1 1 1 

7. External Technical Support and Assistance (4 points)

Level of Developer Support 3 3 3 3 1 2 3 2 1 3 3 3 2 3 Other External Support 1 1 1 1 0 1 1 1 1 1 0 0 0 

8. Evaluation Strategies (2 points)

Written Evaluation Plan 0 1 0 1 0 0 0 0 0 0 1 1 0 0 Evidence of Written Findings 0 1 1 0 0 0 1 0 0 1 0 0 1 0 

9. Coordination of Resources (2 points)

Coordination of External Funds 1 1 1 1 0 1 1 1 1 0 1 1 0 Coordination of External and Local Funds 1 1 1 0 0 1 1 1 1 1 1 0 0 

Total (maximum = 47) 44 37.75 33.75 42 15 39.75 41 32 30.8 38.9 37.9 30.8 25.1 3

Rank 1 9 12 2 18 5 3 13 14 6 8 14 16 1
would satisfy psychometric standards. The needed tool would be a “policy” tool, because 

of the instability of the policy environment—e.g., the 9 CSRD components became 11 

components a few years later under P.L. 110-103. Psychometric tools require a longer 

“shelf-life,” because the time and effort to establish their validity and reliability (and their 

generalizability) assumes that a set of substantive priorities will remain stable over a much 

longer period of time. As a result, “policy” tools seldom have sufficient evidence to 

establish their validity or reliability to a degree that will satisfy psychometric standards. 

The team did corroborate its use of the instrument, however, through an independent 

review of the cases by an analyst “blind” to the rating generated by the instrument (see 

Appendix F).37 Exhibit 3-3 compares the analyst’s rankings of the 18 schools with the 

rankings from the scores on the 47-point instrument. The results show a strong correlation 

between the two sets of rankings, with the rankings differing to a modest degree for a few 

schools (Schools O, Q, and L), but with the overall pattern achieving statistical significance 

(correlation coefficient = .734, n = 18, p < .001). 

Implementation: A solid majority of the schools implemented CSRD. Given 

support by the corroboratory procedure, the scores in Exhibit 3-2 were accepted as 

assessing the strength of CSRD implementation at the 18 schools. Nine schools (Schools 

A, B, D, F, G, J, K, O, and P) garnered 80 percent of the 47 points, with scores higher than 

37.6 points. These schools were labeled as “nearly fully” implementing CSRD. Another 

six schools (Schools C, H, I, L, N, and Q) garnered 65 percent of the 47 points, with scores 

between 30.6 and 37.6. These schools were labeled “partially” implementing CSRD. The 

remaining three schools were judged to be “poorly” implementing CSRD.38

Other observers may differ with regard to the cutoff points chosen for these labels, or 

even the use of the labels themselves. However, the Field-Focused Study interpreted the 

overall set of scores as indicating that a solid majority of the 18 schools were implementing 

CSRD. As a result, student achievement scores at these schools could be expected to be 

affected, assuming some relationship between comprehensive school reform and student 

performance. 

 37 An analyst independently read all of the case reports and ranked the 18 schools according to their 

level of implementation. In arriving at these judgments, the analyst deliberately avoided tallying the nine

components, instead assuming a holistic perspective. The analyst was “blind” to the ratings that had been

made by using the 47-point instrument. The analyst’s findings, together with the rationale for the holistic

judgments, is found in Appendix F.

38The 9-6-3 breakdown happens to be the same as that found for implementing the research-based

method (see Sect. 2.1.1). However, the specific schools in each grouping were not synonymous and the 

measure for CSRD implementation is not synonymous with the implementation of the research-based 

method. 

3-9
Exhibit 3-3 

COMPARISON OF RANKINGS FOR 18 SCHOOLS, 

BASED ON TWO INDEPENDENT ASSESSMENTS OF 

STRENGTH OF CSRD IMPLEMENTATION 

R2 = 0.5384

0

2

4

6

8

10

12

14

16

18

2 0

02468 10 12 14 16 18 2 0

A

D

G

P

F

J

O

K

B

Q

N

C

L

H

I

M

R

E

Ranking Based on Use of 47-Point Instrument

R

anking B

a

sed on H

olistic A

s

se

s

s

m

ent

in Appendi

x F

R2 = 0.5384

0

2

4

6

8

10

12

14

16

18

2 0

02468 10 12 14 16 18 2 0

A

D

G

P

F

J

O

K

B

Q

N

C

L

H

I

M

R R E

2

 = 0.5384

0

2

4

6

8

10

12

14

16

18

2 0

02468 10 12 14 16 18 2 0

A

D

G

P

F

J

O

K

B

Q

N

C

L

H

I

M

R

E

Ranking Based on Use of 47-Point Instrument

R

anking B

a

sed on H

olistic A

s

se

s

s

m

ent

in Appendi

x F

.734 18 .0005

N p

Correlation

Coefficient

.734 18 .0005

N p

Correlation

Coefficient

3-10
3.3 THREE EXAMPLES OF COMPREHENSIVE SCHOOL REFORM 

UNDER THE CSRD PROGRAM

So far, the analysis has examined the 9 CSRD components, pertinent district and state 

conditions, and the overall extent of implementation based on progress on the 9 

components. The analysis has purposely been fragmented, to assure readers that the 9 

components and external conditions have been systematically examined. However, how all 

of these components and conditions might work together and become integrated within a 

specific school may not be readily apparent. 

To gain a more coherent picture, three schools were selected because they seem to be 

making good progress in implementing CSRD but appear to show a slightly different 

pattern of events in doing so. Appendix D contains brief summaries for each of the 18 

schools, and the descriptions for the three schools have been extracted from these reports.39

First Example. The first school (School F) began implementing Coalition of 

Essential Schools prior to its CSRD award (Component 1). Under CSRD, the school has 

expanded its use of the method and also now uses it in combination with other reform 

methods, including High Schools That Work. Together, the methods cover the school’s 

entire range of curriculum and instructional practices. The school has developed and 

shared a comprehensive vision for reform (Component 2), including a written plan that is 

well understood and supported by the entire staff (Component 5). The school is making 

progress on all CSRD components. For instance, large numbers of the faculty participate in 

learning groups and related professional development activities (Component 3), the school 

has successfully engaged parents and community members to serve on key committees 

(Component 6), and the school coordinates multiple external and internal resources in a 

converging manner (Component 9). However, the school’s clear goals and benchmarks 

(Component 4) are not matched by a clear evaluation plan or activity (Component 8). 

An apparent key to the first school’s progress has been the strong leadership of the 

principal and the low turnover of the teaching staff. The district also uses a performance 

and appraisal process that emphasizes professional development and is therefore 

compatible with the needs of comprehensive reform. 

Second Example. The second school (School P) only started its reform activities with 

its CSRD award and is implementing High Schools That Work as its main reform method 

(Component 1). The method’s comprehensive coverage provides a framework for all 

school improvement activities at the school (Component 2), and the staff and a core 

 

39In turn, the brief summaries are based on 18 in-depth and detailed cases reviewed by each school and 

available separately from COSMOS (see www.cosmoscorp.com).

3-11
leadership team fully support the research-based method and the reform plan (Component 

5). The school also is making progress on all CSRD components including the 

coordination of resources (Component 9). For instance, the state is conducting an external 

evaluation covering all 9 components, and the school itself uses a formal instrument to 

observe and evaluate classroom instruction (Component 8). The school-to-work aspect of 

the research-based method attracts a good deal of business and community support 

(Component 6). 

In support of reform at this second school, the district provides substantial staff 

support for CSRD activities and has compensated for any CSRD funding shortfalls in 

professional development. With strong external support from the developer of HSTW 

(Component 7), School P is now engaged in implementing a complementary researchbased method that aligns its work with that of middle school feeders. 

Third Example. The third school (School G) also began its reform activities with the 

first year of CSRD funding, adopting a locally developed Special Literacy Model

(Component 1). Although the method mainly pertains to grades K-3 and reading, the 

school’s reform activities also have embraced the other grades (4-6) and academic subjects, 

as driven by a comprehensive and converging school improvement plan (Component 2). 

The district requires the plan to be aligned with its own strategic goals and plan. Both must 

reflect, according to a state mandate, the state’s standards and assessment tools (the state 

also mandates that every school have an aligned plan). School G has therefore made 

progress on all CSRD components. For instance, the school’s staff strongly support the 

reform (Component 5), and the school closely monitors its progress on relevant goals and 

benchmarks (Component 4) that is key to the year-round school improvement planning 

process. School G’s small size, collaborative management style, and history of reform 

have all contributed strongly to the CSRD efforts. 

3.4 THREE PATHWAYS FOR IMPLEMENTING 

 COMPREHENSIVE SCHOOL REFORM

The essence of CSRD is its focus on improving student performance through 

comprehensive school reform. The three schools just described are among several that 

appear to be making solid progress along these lines but also may provide insight into the 

conditions under which schools are successful in implementing

40 schoolwide reform. 

 

40 The Field-Focused Study deliberately sought to identify implementation conditions substantively 

related to comprehensive school reform. The study did not attempt to examine more general implementation

conditions, such as principal leadership, sufficiency of resources, and absence of turnover or other disruptive 

3-12
Simply put, the three schools may represent three different sets of conditions—each 

characterized as a different pathway to comprehensive school reform. 

The first pathway, represented by School F, is implicit in the formal design of the 

CSRD program and is thus a component-driven pathway: A school envisions and 

implements comprehensiveness in terms of the 9 CSRD components, with many of the 

components compatible with the principles espoused by a “process”-oriented researchbased method (Coalition of Essential Schools). The second pathway, represented by 

School P, is totally compatible with the formal design of the CSRD program but may be 

characterized as a method-driven pathway: A school implements a broad and 

comprehensive research-based method that affects virtually all school operations, including 

curriculum and instruction, and whose successful implementation substitutes for the need 

for any independent articulation of the 9 CSRD components.41 The third, represented by 

School G and also fully compatible with the formal design of the CSRD program, is a 

vertical-driven pathway: State or district strategic planning drives schools toward the nine 

components, including the research-based methods or strategies (Component 1), without 

specifically enumerating them. 

Whether the latter two pathways should be given as much weight as the first pathway, 

which best suits the language of the legislation, is unclear. Furthermore, no pathway is 

necessarily more immune than the others to such disruptive conditions as: high principal 

turnover rates, limited professional development resources, or planned or unplanned school 

restructuring. And, as previously described under state and district conditions, compared to 

the first two pathways, the third pathway does not necessarily make reform any more 

immune from shifts in state or district priorities, either. 

The potential insight presented by identifying these three pathways may be magnified, 

not by comparing the three pathways, but by hypothesizing the conditions when all three 

may be absent. A few of the 18 schools did not attend explicitly to the nine CSRD 

components, did not adopt a comprehensive method, and were not driven to strategic and 

performance-related planning by state or district policies. These schools therefore pursued 

none of the three pathways, and implementation of reform lagged or was even absent. 

 effects. These general conditions remain important, but highlighting them would not have contributed to

understanding CSRD, which was taken as the main objective of the current study.

41The distinction between the importance of method vs. program has continued to evolve over the 

CSRD program period. This study was designed to address the first two cohorts of CSRD schools, when such

a distinction was not as evident.

3-13
3.5 SUSTAINING COMPREHENSIVE SCHOOL REFORM

The experiences of the 18 schools in the Field-Focused Study also provide some clues 

regarding the sustainability of reform beyond the period of CSRD funding. To be recalled 

is that the CSRD program consists of three-year awards, of modest size, and are intended to 

serve (only) as a catalytic force in stimulating comprehensive reform at a school. Using 

these awards to implement short-lived changes—i.e., as in supporting a special “project”—

does not fulfill the basic CSRD mission. Rather, schools must demonstrate a more 

sustained set of changes. However, neither the original legislation nor ED define the exact 

nature of the changes to be associated with sustaining a comprehensively reforming school. 

One implicit view, based mainly on the experiences of the New American Schools 

initiative, holds that the central changes to be sustained should be the practices associated 

with the research-based method supported by CSRD (Component 1). An alternative view 

is that comprehensive reform, though embracing a research-based method, also transcends 

it. Successful sustainability would therefore not be associated with the use of any 

particular method or combination of methods. A successfully reforming school may 

implement one or a combination of methods for a period of time and then later transition to 

other methods—but the transitions should reflect a progression toward continued school 

and student improvement rather than the “churning” of innovative practices. These two 

views led to two different interpretations of sustainability. 

To succeed according to the first view, a school must identify a continuing source of 

funds for a specific method, because the use of the method usually requires some annual 

expenditure, whether: a) to support the work of the external developer, or b) to provide 

professional development opportunities for training new staff and refreshing existing staff 

about the method. Although external developers reduce these annual costs as a school 

gains experience with their method over time, none of the 18 schools had unequivocally 

identified such ongoing sources of support for these activities. Title I funds were the most 

common candidate, but most schools could not be confident about making discretionary 

commitments in a context dominated by district or state budget cutbacks. At the same

time, 10 of the 18 schools (Schools A, C, F, G, I, J, K, O, P, and Q) did indicate their 

intention to seek such support from external developers for the coming school year (2002-

03).42 Most of these schools, however, were not likely to support the method (or the 

 

42One of these schools (School K) may only have been using the CSRD-supported method because of a

state requirement to adopt the method—because the school had been rated as a school “in need of

improvement.” Whether the school will continue using the method if the school sufficiently improves and the 

state requirements are lifted remains to be seen. Alternatively, if the school fails to improve sufficiently, 

other sanctions may be applied and the continued use of the method also would be unclear.

3-14
3-15

 

needed professional development) at the same level as under CSRD, so the quality of the 

continued method may be diluted.43

 To succeed according to the second view, a school substituting for its original CSRDsupported method should evidence a clear rationale related to school or student 

performance. Such was the case with four schools (Schools B, D, H, and N). For example, 

the faculty at one school (School B) that was otherwise doing well in implementing the 

CSRD-supported method decided not to proceed to the next phase, which required adoption 

of a block schedule—because of reports that schools with block schedules had done less 

well on state assessments than had schools without block schedules. A second school 

(School H) had already started using a district-developed curriculum aligned with the state 

assessment,44 instead of the original CSRD method. At the other two schools (Schools D 

and N), the districts had developed their own curricula that were believed to be more 

aligned with state assessments than the CSRD-supported method. Either school was free to 

continue using the original method, but the districts’ resources were devoted to supporting 

their own curricula and related professional development, and a strong possibility is that the 

two schools will transition to the district curricula. 

 Pursuing the second view a step further raises the question of whether any of the 

original 10 schools continuing the CSRD-supported method might have been doing so in 

the face of an emerging and contrary rationale. Three of the 10 schools (Schools I, J, and 

P) may be vulnerable to this jeopardy. Two (Schools I and J) were continuing despite the 

lack of centrality of their method to their districts’ priorities; continuation may depend on 

the identity of a new principal, yet to be appointed at either school. The third (School P) 

was due to be merged with another school in 2003-2004, raising uncertainties because the 

other school’s principal already has been announced as the principal of the merged entity.45

 Exhibit 3-4 places the 18 schools into a sustainability framework that accommodates 

both views. The most lenient condition occurs by accepting both views. Under this 

condition, 14 of the 18 Field-Focused Study schools were exhibiting some level of 

43Of the 10 schools, two schools (Schools I and O) were in CSRD Cohort 1 (whose award period was 

1998-99 to 2000-01), so they already had shown an ability to continue the use of the method in their first 

“post-CSRD” year. Of the other three schools in Cohort 1, two (Schools E and R) had been rated as being the 

lowest in implementing CSRD in the first place (making the issue of sustainability moot). The status of the 

one remaining school in the cohort (School N) is discussed elsewhere in the text.

44Whether district-developed curricula can be considered a “research-based” method was beyond the 

scope of the Field-Focused Study. The districts’ main claim is the alignment between the substance of the 

curricula and that of the state assessments.

45The other school happens also to be one of the 18 in the Field-Focused Study (School B). 
3-16

Exhibit 3-4 

SUSTAINABILITY OF CSRD AT THE 18 SCHOOLS

Likely to Continue

Research-Based

Method Supported 

by CSRD

Likely to Sustain

Some Method in

Relation to

Student Performance

Both Neither

I, J, P

A, C,

F, G, K,

O, Q

B, D, H, N E, L, M, R

Likely to Continue

Research-Based

Method Supported 

by CSRD

Likely to Sustain

Some Method in

Relation to

Student Performance

Both Neither

I, J, P

A, C,

F, G, K,

O, Q

B, D, H, N E, L, M, R

sustainability by the time their CSRD awards were ending. Accepting only the second 

view reduces the number to 11, and accepting only the first view reduces it to 10. 

 

The main barriers to sustainability continue to be the availability of slack resources, 

especially in light of states’ and districts’ revenue shortfalls in recent years. To sustain a 

reforming process, even when existing resources have been sufficiently coordinated and 

targeted to reform, still requires discretionary funds to support such essential activities as: 

adequate professional development (including support for teacher substitutes), especially in 

situations of high teacher turnover; time for common planning periods or teachers’ work on 

school leadership teams; and support for external technical assistance. Though such needs 

can be served with modest levels of funds, serving the needs is still a discretionary activity 

that may have to be ignored if the core school operations are underbudgeted. 
4-1

4. CONCLUSIONS FROM THE FIELD-FOCUSED STUDY 

 The Field-Focused Study arrived at the following conclusions in relation to the main 

research questions posed at the outset of the evaluation. 

Overall Implementation of CSRD. Using a 47-point instrument, the Field-Focused 

Study determined that 9 of the 18 schools were nearly fully implementing CSRD and its 9 

components, another 6 were partially implementing CSRD, and the remaining 3 were 

judged to be poorly implementing CSRD. On the whole, CSRD implementation was 

therefore uneven. 

Three Conditions (or “Pathways”) for Successful Implementation, Including 

District and State Influences. The study identified three different sets of conditions, or 

“pathways,” that appeared to be associated with the successful implementation of CSRD. 

The first pathway is a component-driven pathway, whereby a school uses the 9 CSRD 

components to guide the development and implementation of a comprehensive reform. 

The second is a method-driven pathway, whereby the school adopts and implements a 

comprehensive research-based method that affects virtually all school operations and 

whose successful implementation substitutes for the need for any independent articulation 

of the 9 CSRD components. The third is a vertical-driven pathway, whereby a school 

articulates and pursues the needed comprehensive strategies as a result of state and district 

requirements involving the setting of standards, use of appropriate assessment tools, and 

required and aligned district- and school-based strategic planning and improvement plans 

to meet performance standards. 

 No single pathway was considered the “best” or preferred pathway, and no pathway 

was necessarily more immune than the others to such disruptive conditions as: high 

principal turnover rates, limited professional development resources, or planned or 

unplanned school restructuring. 

Sustainability of Comprehensive School Reform. Neither the original legislation nor 

ED define the exact nature of a school’s changes to be associated with sustaining a 

comprehensively reforming school beyond the three-year CSRD award period. As a result, 

the Field-Focused Study examined two different views of sustainability and judged the 18 

schools according to both. 

 The first implicit view, based mainly on the experiences of the New American 

Schools initiative, holds that the central changes to be sustained should be the practices 

associated with the originally-supported research-based method. An alternative view is that 

comprehensive reform, though embracing a research-based method, also transcends it. By 

this second view, successful sustainability would not necessarily be associated with the 
4-2

continued use of any particular method but could involve transitions from one researchbased method to another, over time. The transitions would have to reflect a progression 

toward continued school and student improvement rather than the “churning” of innovative 

practices. 

 Using the most lenient benchmark and accepting either of these two views as a 

criterion for assessing sustainability, 14 of the 18 schools were exhibiting a promising level 

of sustainability by the time their CSRD awards were ending. Accepting the 

“comprehensive reform” view alone reduces the number to 11; accepting the “researchbased method” view alone reduces the number to 10. 

 The main barriers to sustainability continued to be the availability of slack resources, 

especially in light of states’ and districts’ revenue shortfalls in recent years. To sustain a 

reforming process, even when existing resources have been sufficiently coordinated and 

targeted to reform, still requires discretionary funds to support such essential activities as: 

adequate professional development (including support for teacher substitutes), especially in 

situations of high teacher turnover; time for common planning periods or teachers’ work on 

school leadership teams; and support for external technical assistance. Though such needs 

can be served with modest levels of funds, serving the needs is still a discretionary activity 

that may have to be ignored if the core school operations are underbudgeted. Thus, the 

sustainability of comprehensive school reform is still questionable, given the current fiscal 

climate. 
5-1

5. REFERENCES

Archibald, Sarah, and Allan Odden, A Case Study of Resource Reallocation to Implement 

a Whole School Reform Model and Boost Student Achievement: Parnell 

Elementary School, Consortium for Policy Research in Education, Philadelphia, PA, 

March 2000. 

Berends, Mark, Susan Bodilly, and Sheila Nataraj Kirby, Reforming Whole Schools: 

Challenges and Complexities, 2002. 

Berends, Mark, Joan Chun, Gina Schuyler, Sue Stockly, and R.J. Briggs, Challenges of 

Conflicting School Reforms: Effects of New American Schools in a High-Poverty 

District, RAND, 2002. 

Berends, Mark, and Sheila Nataraj Kirby, Monitoring the Progress of Title I and CSRD 

Schools: RAND’s Analysis Plan, proposal submitted to Westat and the U.S. 

Department of Education, August 1999. 

Bodilly, Susan, “The Evolution and Future of Comprehensive School Reform,” transcript 

from Education Policy Forum Luncheon Series, National Clearinghouse for 

Comprehensive School Reform, Washington, DC, February 9, 2001. 

Bodilly, Susan, Lessons from New American Schools’ Scale-Up Phase: Prospects for 

Bringing Designs to Multiple Schools, The Rand Corporation, Santa Monica, CA, 

1998. 

Borman, Geoffrey D., “Title I: The Evolving Research Base,” Journal of Education for 

Students Placed at Risk, 2000, 5(1&2):27-45. 

Cohen, Michael, “Effective School: Accumulating Research Findings,” American 

Education, Jan./Feb. 1982, 18:13-16. 

Consortium for Policy Research in Education, “States and Districts and Comprehensive 

School Reform,” CPRE Policy Briefs, May 1998, RB-24. 

COSMOS Corporation, First-Year Observations from the Field-Focused Study of the 

Comprehensive School Reform Demonstration Program, submitted to Planning and 

Evaluation Service, Office of the Undersecretary, U.S. Department of Education, 

December 2001. 
5-2

COSMOS Corporation and The McKenzie Group, Inc., Funded by the U.S. Department of 

Education Studies of Comprehensive School Reform and CSRD, Bethesda, MD, 

September 2001. 

COSMOS Corporation, Evaluation Design for Component I, Bethesda, MD, May 2000. 

Datnow, Amanda, and Sam Stringfield, “ Working Together for Reliable School Reform,” 

Journal of Education for Students Placed at Risk, 2000, 5(1&2):183-204. 

Desimone, Laura, Making Comprehensive School Reform Work, ERIC Clearinghouse on 

Urban Education and the Institute for Urban and Minority Education, New York, NY, 

May 2000. 

Doherty, Katherine M., Early Implementation of the Comprehensive School Reform 

Demonstration (CSRD) Program, Planning and Evaluation Service, U.S. Department 

of Education, Washington, DC, Sept. 2000. 

Edmonds, Ronald, “Effective Schools for the Urban Poor,” Educational Leadership, 

October 1979, 37:15-24. 

Fermanich, Mark, Allan Odden, and Sarah Archibald, A Case Study of District 

Decentralization and Site-Based Budgeting: Cordell Place School District, 

Consortium for Policy Research in Education, Philadelphia, PA, March 2000. 

Loveless, Tom, How Well Are American Students Learning? With Sections on 

Arithmetic, High School Culture, and Charter Schools, Brown Center on Education 

Policy, The Brookings Institution, Washington, DC, September 2002, Vol. 1, No. 3. 

New American Schools, Working Toward Excellence: Examining the Effectiveness of 

New American Schools Designs, no place of publication, February 1999. 

Odden, Allan, and Sarah Archibald, A Case Study of Resource Reallocation to Reduce 

Class Size, Enhance Teacher Planning Time, and Strengthen Literacy: Clayton 

Elementary School, Consortium for Policy Research in Education, Philadelphia, PA, 

March 2000. 

Odden, Allan, Sarah Archibald, and Anita Tychsen, Hollister Elementary School: A Case 

Study of Resource Reallocation, Consortium for Policy Research in Education, 

Philadelphia, PA, 1999a. 
5-3

Odden, Allan, Sarah Archibald, and Anita Tychsen, Farnham Elementary School: A Case 

Study of Resource Allocation, Consortium for Policy Research in Education, 

Philadelphia, PA, 1999b. 

Puma, Michael J., and Darrel W. Drury, Exploring New Directions: Title I in the Year 

2000, National School Boards Association, Alexandria, VA, 2000. 

Ross, Steven M., et al., Two-and Three-Year Achievement Results on the Tennessee 

Value-Added Assessment System for Restructuring Schools in Memphis, University 

of Memphis, Memphis, TN, June 1999. 

Shelley, Fred M., “Education Policy and the104th Congress,” in Lynn A. Staeheli, Janet E. 

Kodras, and Colin Flint (eds.), State Devolution in America: Implications for a 

Diverse Society, Sage Publications, Thousand Oaks, CA, 1997, pp. 221-232. 

Slavin, Robert E., and Nancy A. Madden, “Roots & Wings: Effects of Whole-School 

Reform on Student Achievement,” Journal of Education for Students Placed at 

Risk, 2000, 5(1&2):109-136. 

Southwest Educational Development Laboratory, CSRD Awards Database, 2002. 

Stringfield, Sam, et al., Urban and Suburban/Rural Special Strategies for Educating 

Disadvantaged Children: Final Report, U.S. Department of Education, Washington, 

DC, 1997. 

The Charles A. Dana Center, Hope for Urban Education: A Study of Nine HighPerforming, High-Poverty, Urban Elementary Schools, The University of Texas at 

Austin, 1999. 

U.S. Congress, Fiscal Year 1998 Appropriations Act for the U.S. Department of 

Education, P.L. 105-78, 1998. 

U.S. Congress, Conferees’ House Report No. 105-309, leading to the enactment of P.L. 

105-78. 

U.S. Department of Education, Update to Guidance on the Comprehensive School Reform 

Demonstration Program, Washington, DC, August 11, 2000. 

U.S. Department of Education, Guidance on the Comprehensive School Reform 

Demonstration Program, Washington, DC, March 13, 1998. 
5-4

Yin, Robert K., “Rival Explanations as An Alternative to Reforms as ‘Experiments,’” in 

Leonard Bickman (ed.), Validity and Social Experimentation: Donald Campbell’s 

Legacy, Sage Publications, Thousand Oaks, CA, 2000, pp. 241-268. 

Yin, Robert K., Case Study Research: Design and Methods, Sage Publications, Thousand 

Oaks, CA, 2003, third edition.