**This is some**

+------------------+---------------------------------------------------+
|                  | ######## Policy and Program Studies Service       |
+------------------+---------------------------------------------------+

+-----------------------------------------------------------------------+
| ### [Findings from the]{.smallcaps}                                   |
|                                                                       |
| **[Field-Focused Study of the]{.smallcaps}**                          |
|                                                                       |
| Comprehensive School Reform Demonstration Program                     |
|                                                                       |
| ##### Volume I: FINAL REPORT                                          |
|                                                                       |
| #### 2003                                                             |
+-----------------------------------------------------------------------+

+---------------------------+---------------------------+--------------+
|                           |                           | ![]          |
|                           |                           | (media/image |
|                           |                           | 1.jpeg){widt |
|                           |                           | h="1.5222222 |
|                           |                           | 222222221in" |
|                           |                           | height       |
|                           |                           | ="1.52222222 |
|                           |                           | 22222221in"} |
+---------------------------+---------------------------+--------------+
| **[U.S. Department of     | [Office of the Under      |              |
| Education]{.smallcaps}**  | Secretary]{.smallcaps}    |              |
|                           |                           |              |
| [Doc \#]{.smallcaps}      |                           |              |
+---------------------------+---------------------------+--------------+

**Findings from the Field-Focused Study of the**

**Comprehensive School Reform Demonstration Program**

# Volume I: Final Report

# 2003

**Preface**

This Final Report presents the findings from the Field-Focused Study,
one of several components in the National Evaluation of the
Comprehensive School Reform Demonstration (CSRD) Program. The U.S.
Department of Education (ED) sponsored the study under a task order and
contract administered by the U.S. Department of Health and Human
Services' Program Support Center (Task Order #6, Contract No.
282-98-0027), to COSMOS Corporation. The McKenzie Group, Inc. (TMG),
served as a subcontractor to COSMOS. In this function, TMG conducted and
completed a review of all CSRD-related studies supported by the U.S.
Department of Education (COSMOS/TMG, 2001). TMG staff also comprised
about one-third of the site visit teams.

The Field-Focused Study and the entire task order were directed by
Robert K. Yin, Ph.D., of COSMOS. Margaret K. Gwaltney, M.B.A., of COSMOS
served as the deputy project director, and Dawn Kim served as project
coordinator. Many other staff members from COSMOS and TMG collaborated
in the study, which involved four rounds of site visits to 18 CSRD
schools and the collection and analysis of student achievement data from
these schools. Michelle LaPointe, Ph.D., of the Policy and Program
Studies Service, U.S. Department of Education, served as the ED project
officer for most of the evaluation (Kathryn Doherty, Ph.D., was ED's
first project officer).

The report was mainly prepared by Robert Yin and Dawn Kim and is
presented in two volumes. Volume I contains the main text. Volume II
contains six appendices to Volume I, including short summaries of the 18
schools that were studied. The summaries, as well as data presented in
Volume I, are based on 18 in-depth and detailed cases reviewed by each
school and available separately from COSMOS (see www. cosmoscorp.com).
Throughout, all schools are identified anonymously.

**EXECUTIVE SUMMARY**

***The Comprehensive School Reform Demonstration Program***.
Comprehensive school reform calls for a "whole-school" and coordinated
approach to improve schools. The strategy differs from piecemeal and
fragmented efforts that also in the past have seemed only to lead to
short-lived changes.

To stimulate whole-school reform across the country, Congress
appropriated funds in FY1998 for the U.S. Department of Education (ED)
to start the Comprehensive School Reform Demonstration (CSRD) Program.
ED allocated the funds on a formula basis to states, who made awards to
support 1,840 mostly Title I schools "in need of substantially
improving" their performance. Subsequent rounds of annual awards to
support additional schools have continued through FY2003.

In applying for and accepting these funds, schools were expected to
implement 9 components, one of which is an effective, research-based
method or strategy. Together, the 9 components comprise the
comprehensive reform aimed at improving student achievement:

**CSRD Components**

+---+------+---------------------------------------------------------+---+
|   |      |                                                         |   |
+---+------+---------------------------------------------------------+---+
|   | No.  | Component                                               |   |
+---+------+---------------------------------------------------------+---+
|   | 1\.  | Effective, research-based methods and strategies        |   |
|   |      |                                                         |   |
|   | 2\.  | Comprehensive design with aligned components            |   |
|   |      |                                                         |   |
|   | 3\.  | Professional development                                |   |
|   |      |                                                         |   |
|   | 4\.  | Measurable goals and benchmarks                         |   |
|   |      |                                                         |   |
|   | 5\.  | Support within the school                               |   |
|   |      |                                                         |   |
|   | 6\.  | Parental and community involvement                      |   |
|   |      |                                                         |   |
|   | 7\.  | External technical support and assistance               |   |
|   |      |                                                         |   |
|   | 8\.  | Evaluation strategies                                   |   |
|   |      |                                                         |   |
|   | 9\.  | Coordination of resources                               |   |
+---+------+---------------------------------------------------------+---+
|   |      |                                                         |   |
+---+------+---------------------------------------------------------+---+

The awards are for three years and had to be a minimum of \$50,000 per
year. Schools can only receive a single CSRD award. Given these
conditions, federal funds therefore are intended to serve as "seed
money" for whole-school reform. Beyond this period of time, schools are
to continue reform with their own resources.

***The Field-Focused Study of CSRD***. ED commissioned the Field-Focused
Study of CSRD as one part of its national evaluation of the CSRD
program. The study began by randomly selecting 18 schools from the first
two annual rounds of CSRD awards (1998-99 to 2000-01, and 1999-00 to
2001-02). Field data were collected through four site visits to each
school, conducted during 2000-02.

The objective of the Field-Focused Study was to provide initial feedback
about the implementation of the CSRD program. The topics covered by the
study and in this report include:

1\) The progress in implementing the 9 CSRD components by the

18 schools;

2\) The role of district and state influences in implementing the
program;

3\) The apparent pathways or strategies for implementing comprehensive

school reform; and

4\) The early signs regarding the potential sustainability of the
program at

the school level.

**Conclusions on CSRD Implementation: A Mixed Pattern**

The main priority of the Field-Focused Study was to determine the extent
and nature of CSRD implementation at the 18 schools. Overall, the
schools' implementation of CSRD at the 18 schools was uneven. Based on a
47-point instrument reflecting the 9 components and devised specifically
for the Field-Focused Study, nine schools garnered 80 percent of the 47
points and could be labeled as "nearly fully" implementing CSRD. Another
six schools garnered 65 percent of the 47 points and could be labeled as
"partially" implementing CSRD. The remaining three schools all scored
lower and were judged to be "poorly" implementing CSRD.

***Implementation of the 9 CSRD Components***. Implementation also was
uneven across the 9 components (to streamline the discussion, several
related components have been paired):

Component 1 (see Section 2.1 of the main report for more details):

The bulk of the schools' CSRD resources and attention has been

devoted to the implementation of a *research-based method*. By

the time of the final site visit, implementation of such a method

was proceeding fully at 9 of the 18 schools, with partial or minimal

implementation at the other 9;

Component 2 (see Section 2.2):

Less than half of the 18 schools had reforms with a *compre-*

*hensive design*, reflected by a comprehensive plan combined

with staff awareness that CSRD should extend to a school's

entire way of doing business and all its operations---and not

simply adding a new function or project or activity;

Components 3 and 7 (see Section 2.3.1):

Both *professional development* and *external technical support*

*and assistance* had largely been devoted to the implementation

of the research-based method, not necessarily comprehensive

reform more broadly;

Components 4 and 8 (see Section 2.3.2):

Both the *measurable goals and benchmarks* and *evaluation*

*strategies* had been devoted to tracking student performance,

not necessarily implementation progress;

Components 5 and 6 (see Section 2.3.3):

High turnover among staff and students had resulted in transient

levels of either *support within the school* or *parental and com-*

*unity* *involvement*;

Component 9 (see Section 2.3.4):

Most schools were in a position to *coordinate or converge re*-

*sources*, but resources for sustainability were still uncertain.

***District and State Influences***. The Field-Focused Study also
collected data about district and state actions potentially affecting
the schools' CSRD implementation. Some of these actions were part of the
CSRD administrative procedures, because states implement CSRD by having
districts apply competitively on behalf of some or all of their schools.
In the process, both states and districts can support or monitor the
schools' CSRD efforts. The study found varying degrees of such support
(see Section 3.1.1).

More important than these procedures related directly to the
administration of CSRD, the study uncovered other important state and
district policies, not directly related to administering CSRD, that
nevertheless influenced CSRD implementation (see Section 3.1.2). Some
conditions, such as extremely limited professional resources, had a
negative influence on CSRD. Other conditions, such as the direct
alignment of CSRD designs with district improvement plans and state
standards, had an extremely positive influence. Other conditions
reflected the ongoing dynamics of school systems---e.g., districts
reducing financial support for all external research-based methods, a
district allowing a CSRD school to become a charter school, and a
district deciding to merge two schools that happened to be CSRD schools.

Strong district or state influence, creating a "vertical" alignment to
the school level, led alternatively to either complementarity or
conflict with CSRD. As examples on the complementary side, CSRD provided
two schools with resources and a compatible reform agenda to respond to
their designation as underperforming schools in the state's
accountability system. As examples on the conflicting side, the shifting
content of state assessments led districts to use resources for
alternative curricula and professional development that were contrary to
those involved in a CSRD school's original plans or implementation. In
general, these external state and district conditions appear to be
highly relevant to CSRD implementation.

**Conditions Associated with Successful Implementation and the Role of
State and District Influences: Three Pathways to Reform**

The study identified three different sets of conditions, or "pathways,"
that appeared to be associated with the successful implementation of
CSRD (see Section 3.4). The first pathway is a *component-driven
pathway*, whereby a school uses the 9 CSRD components to guide the
development and implementation of a comprehensive reform. The second is
a *method-driven pathway*, whereby the school adopts and implements a
comprehensive research-based method that affects virtually all school
operations and whose successful implementation substitutes for the need
for any independent articulation of the 9 CSRD components. (However,
many research-based methods focus on specific curricula and are not
comprehensive.) The third is a *vertical-driven pathway*, whereby a
school articulates and pursues the needed comprehensive strategies as a
result of state and district requirements involving: the setting of
standards, use of appropriate assessment tools, and required alignment
of district- and school-based strategic planning and improvement plans
to meet state performance standards.

No single pathway was considered the "best" or preferred pathway, and no
pathway was necessarily more immune than the others to such disruptive
conditions as: high principal turnover rates, limited professional
development resources, or planned or unplanned school restructuring.

**Sustainability of "Whole-School" Reform: Still Questionable Given
Current Fiscal Climate**

As a final topic, the study examined the prospects for sustaining school
reform beyond the final year of CSRD funding (see Section 3.5).

Neither the original legislation nor ED defined the exact nature of a
school's changes to be associated with sustaining a comprehensively
reforming school beyond the three-year CSRD award period. As a result,
the Field-Focused Study examined two different views of sustainability
and judged the 18 schools according to both.

The first view, based mainly on the experiences of the New American
Schools initiative, holds that the central changes to be sustained
should be the practices associated with the originally-supported
research-based method. The second view is that comprehensive reform,
though embracing a research-based method, also transcends it. By this
second view, successful sustainability would not necessarily be
associated with the continued use of any particular method but could
involve transitions from one research-based method to another, over
time. The transitions would have to reflect a progression toward
continued school and student improvement rather than the "churning" of
innovative practices.

Using the most lenient benchmark and accepting either of these two views
as a criterion for assessing sustainability, 14 of the 18 schools were
exhibiting a promising level of sustainability by the time their CSRD
awards were ending. Accepting only the second view reduces the number to
11, and accepting the first view reduces it to 10.

The main barrier to sustainability continues to be the limited
availability of sufficient resources, especially in light of states' and
districts' revenue shortfalls in recent years. To sustain a reforming
process, even when existing resources have been coordinated and targeted
to reform, still requires discretionary funds to support such essential
activities as: adequate professional development (including support for
teacher substitutes), especially in situations of high teacher turnover;
time for common planning periods or teachers' work on school leadership
teams; and support for external technical assistance. Though such needs
can be served with modest levels of funds, serving the needs is still a
discretionary activity that may have to be ignored if core school
operations are underbudgeted.

# Contents

Page

###### Volume I

Preface ii

Executive Summary iii

###### Sections

1\. The Comprehensive School Reform Demonstration Program

> and the Field-Focused Study 1-1

1.1 Comprehensive School Reform 1-1

1.2 The Field-Focused Study 1-8

1.3 Sample of Schools Being Studied 1-13

1.4 Study Activities and a General Caveat 1-14

2\. CSRD Implementation: The 9 Components 2-1

2.1 Implementing Research-Based Methods 2-1

2.2 Comprehensiveness of Designs for School Reform 2-9

2.3 Progress in Implementing the 7 Other CSRD Components 2-13

2.4 Summary 2-30

3\. CSRD Implementation and Sustainability: The Integrated Findings 3-1

3.1 District and State Influences on CSRD 3-1

3.2 Schools Implementing Comprehensive School Reform 3-5

3.  Three Examples of Comprehensive School Reform

> under the CSRD Program 3-11
>
> 3.4 Three Pathways for Implementing Comprehensive School Reform 3-12
>
> 3.5 Sustaining Comprehensive School Reform 3-14

4\. Conclusions from the Field-Focused Study 4-1

5\. References 5-1

**Exhibits**

> 1-1 Description of Nine Components of Reform in

Congressional Provisions for CSRD 1-2

> 1-2 Summary of CSRD Awards 1-5
>
> 1-3 Theoretical Framework for a CSRD School 1-10
>
> 1-4 Beginning to Customize the Original Framework:
>
> Two Different CSRD Models 1-11

5.  Location of the Final Sample of CSRD Schools for

> Inclusion in the Field-Focused Study 1-14

1.  Research-Based Methods Adopted by 18 CSRD Schools and

> Fidelity Assessments Obtained from Developers 2-2
>
> 2-2 Implementation Status of Research-Based Methods (Component 1) 2-4
>
> 2-3 Three CSRD Schools with Minimal or Stalled Implementation of
>
> the Originally-Adopted Research-Based Method 2-5
>
> 2-4 Four Locally Developed Models 2-8
>
> 2-5 Evidence of Comprehensive Design (Component 2) 2-11
>
> 2-6 Professional Development Activities (Component 3) 2-14
>
> 2-7 External Technical Support and Assistance (Component 7) 2-18
>
> 2-8 Goals and Benchmarks (Component 4) and
>
> Evaluation Strategies (Component 8) 2-22
>
> 2-9 Status of Coordination of Resources (Component 9) and
> Sustainability 2-27
>
> 3-1 School, District, and State Conditions 3-2
>
> 3-2 Scores for CSRD Implementation Strength at 18 Schools,
>
> Using 47-Point Instrument 3-7

3.  Comparison of Rankings for 18 Schools, Based on

> Two Independent Assessments of Strength of CSRD Implementation 3-10
>
> 3-4 Sustainability of CSRD at the 18 Schools 3-16

# Volume II -- Appendices

> A. Evaluation Methodology A-1
>
> B. Promising Reform Strategies B-1
>
> C. Descriptions of the Research-Based Methods or Strategies Being Used
> C-1
>
> D. Short Summaries of 18 CSRD Schools D-1

E.  47-Point Instrument for Assessing Strength of

> CSRD Implementation at 18 Schools E-1
>
> F. Corroboratory Assessment of CSRD Implementation for
>
> 18 Schools in Field-Focused Study F-1

**1. THE COMPREHENSIVE SCHOOL REFORM**

**DEMONSTRATION PROGRAM**[^1] **AND**

# THE FIELD-FOCUSED STUDY

**1.1 COMPREHENSIVE SCHOOL REFORM**

***The Comprehensive School Reform Demonstration Program (CSRD) was
designed to stimulate schoolwide change***. The Comprehensive School
Reform Demonstration (CSRD) Program provided formula grants to states.
The states in turn invited districts to compete for subgrants, on behalf
of schools, to implement comprehensive school reform. The program's goal
was to improve student achievement and enable all children to "meet
challenging state content standards and performance goals" through
reform strategies (U.S. Congress, 1998).[^2]

Participating schools were expected to:

-   Implement CSRD's nine components[^3] (see Exhibit 1-1), one of which
    was the implementation of an effective, research-based method or
    strategy;[^4]

-   Stimulate schoolwide change covering virtually all aspects of school
    operations, rather than implement a piecemeal, fragmented approach
    to reform; and

**Exhibit 1-1**

**DESCRIPTION OF NINE COMPONENTS OF REFORM**

**IN CONGRESSIONAL PROVISIONS FOR CSRD**^\*^

1\. **Effective, research-based methods and strategies:** Employ
innovative strategies and proven methods for student learning, teaching,
and school management that are based on reliable research and effective
practices, and have been replicated successfully in schools with diverse
characteristics.

2\. **Comprehensive design with aligned components:** Have a
comprehensive design for effective school functioning, including
instruction, assessment, classroom management, professional development,
parental involvement, and management---that aligns the school's
curriculum, technology, professional development, etc. into a schoolwide
reform plan designed to enable all students \[including children from
low-income families, children with limited English proficiency, and
children with disabilities\]^\*\*^ to meet challenging state content and
performance standards and addresses needs identified through a school
needs assessment.

3\. **Professional development:** Provide high-quality and continuous
teacher and staff professional development and training.

4\. **Measurable goals and benchmarks:** Have measurable goals for
student performance \[tied to the State's challenging content and
student performance standards, as those standards are
implemented\]^\*\*^ and benchmarks for meeting those goals.

5\. **Support within the school:** Are supported by school faculty,
administrators, and staff.

6\. **Parental and community involvement:** Provide for the meaningful
involvement of parents and the local community in planning and
implementing school improvement activities.

7\. **External technical support and assistance:** Utilize high-quality
external technical support and assistance from a comprehensive school
reform entity (which may be a university) with experience or expertise
in school-wide reform and improvement.

8\. **Evaluation strategies:** Include a plan for the evaluation of the
implementation of school reform and the student results achieved.

9\. **Coordination of resources:** Identify how other resources
(federal/state/local/private) available to the school will be utilized
to coordinate services to support and sustain the school reform effort.

^\*^Conferees' original language in House Report No. 105-309,
accompanying P.L. 105-78.

^\*\*^Language in brackets was added to the conferees' language for the
original "Guidance on the Comprehensive School Reform Demonstration
Program," issued by the US Department of Education on March 13, 1998;
the language is present in the subsequent update dated August 11, 2000.

-   Assess the impact from implementing comprehensive

> school reform in terms of changes in student achievement.

For most of the CSRD funds, the eligible K-12 schools were Title I
eligible schools, particularly those schools in need of substantially
improving their student achievement levels. A small portion of the funds
was available to all schools in need of such improvement. As a result,
Title I schools represented the great bulk (over 85 percent) of CSRD
schools.

***CSRD funding was disbursed to nearly 4,000 schools by FY 2001*.**
CSRD funds first became available in FY1998, when \$145 million in
federal funds were allocated to state education agencies (SEAs).[^5] The
SEAs then made competitive three-year subgrants to their districts,
eventually supporting 1,840 schools with the first year's funds.[^6] The
awards (a minimum of \$50,000 per year) were intended to stimulate or
provide financial incentives for reform, but not support the entirety of
reform.[^7]

In FY1999, \$145 million in funds were again allocated to the states,
supporting the continuation of the initial awards and also the startups
of a small number of new awards. In FY2000, the program awarded \$220
million to the states, anticipating the continuation of the original
schools into their third year but also providing the opportunity to make
awards to a new set of schools. Similarly, \$260 million was provided in
FY2001, which permitted the funding of an entirely new set of schools
starting in 2001--02, as well as the continuation of those schools begun
with the FY2000 increase. The total number of schools funded under the
CSRD program reached 3,875 by the 2001--2002 academic year (SEDL, 2002).
For 2002-03, the appropriation increased funding for the newly defined
CSR program to \$310 million, enough to support grant awards to an
estimated 2,000 additional schools (see Exhibit 1-2).

***CSRD called for research-based methods and strategies***. The CSRD
program drew its rationale and design from a longstanding and broad
research and policy literature.[^8] For instance, from a research
perspective, two contemporary analysts (Desimone, 2000, p. 36; and
Doherty, 2000, p. 3) noted that the theoretical underpinnings for
comprehensive school reform were similar to and extended as far back as
research on *effective schools* in the 1970s and 1980s (e.g., Edmonds,
1979; and Cohen, 1982), among other educational strands.

From a policy perspective, the practical underpinnings for CSRD emanated
from the events leading to the redesign of Title I of the Elementary and
Secondary School Act as amended by the Improving America's Schools Act
(IASA) of 1994, which shifted the federal government's compensatory
education investments away from supplemental and remedial (pull-out)
programs toward standards-based, whole school reform---as reflected by a
renewed appreciation for schoolwide programs and local flexibility
(Shelley, 1997; Berends and Kirby, 1999; Puma and Drury, 2000; and
Borman, 2000). As one result of the policy shift, the number of
schoolwide programs within Title I increased from 1,200 in 1991 to 9,000
in 1997-98 and represented 50 percent of all Title I schools (Desimone,
2000).[^9]

Coinciding with these research and policy developments, a critical
catalytic influence was the ongoing experience of the *New American
Schools* (NAS) initiative (e.g., Bodilly, 1998; Berends, Bodilly, and
Kirby, 2002). The New American Schools Corporation was formed in 1991 to
design innovative, 'break-the-mold' schools. NAS's initial wave
supported eight research-based methods,[^10] which were then promoted by
over 1,500 schools in 11 partner communities (New American Schools,
1999). The initiative came at a time when the supply of research-based
methods converged with the demand for continued school reform among
low-performing schools (Bodilly, 2001). Early lessons from the
initiative included a shift away from an emphasis on method
development-design-only to include external technical assistance and
support to schools (NAS, 1999), as well as

**Exhibit 1-2**

**SUMMARY OF CSRD AWARDS**

+-----------------+--------------+----------------------+--------------+
| **Federal       | **Amount     | **Cohort: Number of  | **School     |
| Fiscal Year     | Allocated**  | Schools**^a^         | Year First   |
| Appropriation** |              |                      | I            |
|                 | **(In        |                      | mplemented** |
|                 | Millions)**  |                      |              |
+-----------------+--------------+----------------------+--------------+
| **1998**        | \$145^b^     | Cohort 1: 1,840      | 1998-1999    |
+-----------------+--------------+----------------------+--------------+
| **1999**        | \$145^b^     | Cohort 1: 1,840      | 1999-2000    |
|                 |              |                      |              |
|                 |              | Cohort 2: 146        |              |
+-----------------+--------------+----------------------+--------------+
| **2000**        | \$220^c^     | Cohort 1: 1,856      | 2000-2001    |
|                 |              |                      |              |
|                 |              | Cohort 2: 146        |              |
|                 |              |                      |              |
|                 |              | Cohort 3: 796        |              |
+-----------------+--------------+----------------------+--------------+
| **2001**        | \$260^c^     | Cohort 2: 146        | 2001-2002    |
|                 |              |                      |              |
|                 |              | Cohort 3: 796        |              |
|                 |              |                      |              |
|                 |              | Cohort 4: 1093       |              |
+-----------------+--------------+----------------------+--------------+
| **2002**        | \$310^d^     | Cohort 3: 796        | 2002-2003    |
|                 |              |                      |              |
| **(No Child     |              | Cohort 4: 1093       |              |
| Left Behind)**  |              |                      |              |
|                 |              | Cohort 5: 2,000      |              |
|                 |              | (est.)               |              |
+-----------------+--------------+----------------------+--------------+
|                 |              |                      |              |
+-----------------+--------------+----------------------+--------------+

^a^ Amounts reported by States for each fiscal year do not always add to
the amount allocated to States in that fiscal year (do to
inconsistencies in State reporting).

^b^ Includes \$25 million provided under the Fund for the Improvement of
Education (FIE).

^s^ Includes \$50 million provided under the Fund for the Improvement of
Education (FIE).

^d^ These are the appropriated funds (including \$75 million in FIE
funds) and the estimated number of schools the funds will support.

Sources: Comprehensive School Reform Program: State Allocations for all
Fiscal Years, U.S. Department of Education, June 25, 2002,
www.ed.gov/offices/OESE/compreform/allyears.html.

> CSR Awards Database, Southwest Educational Development Laboratory,
> August 2002 (Note: The number of schools receiving awards was taken
> from the SEDL database, which was undergoing revisions at the time of
> this final report; the final number of schools awarded funds may have
> changed).

an increased appreciation for the role of the district in supporting or
facilitating comprehensive school reform (NAS, 1999).

Emerging studies of comprehensive school reform have shown mixed
outcomes. Some early studies suggested that the outcomes can be
promising, as reflected by syntheses of these studies (Desimone, 2000;
and Datnow and Stringfield, 2000). However, other studies have found no
significant effects of comprehensive school reform on student
achievement (e.g., Berends et al., 2002). The completed research
includes a variety of evaluations of specific research-based methods
(e.g., Slavin and Madden, 2000),[^11] analyses of student achievement
outcomes in entire school systems (e.g., Ross et al., 1999), and policy
studies of strategies for educating disadvantaged children (e.g.,
Stringfield et al., 1997). In addition, the U.S. Department of Education
funded 46 studies,[^12] whose progress was tracked and documented to
encourage coordination among the studies (COSMOS/ TMG, 2001), including
the several components of the National Evaluation of CSRD. Yet other
studies and related activities are being tracked by the National
Clearinghouse for Comprehensive School Reform (NCCSR).

Earlier research also includes individual case studies of CSRD-like
comprehensive school reform, such as five case studies commissioned by
the Consortium for Policy Research in Education---CPRE (Odden,
Archibald, and Tychsen, 1999a and 1999b; Archibald and Odden, 2000;
Odden and Archibald, 2000; and Fermanich, Odden, and Archibald, 2000).
Another multiple-case study covered nine elementary schools engaged in
comprehensive efforts similar to the CSRD program (The Dana Center,
1999). Among other findings, the cross-case analysis suggested that
reform was more likely to succeed when efforts focused on boosting the
schools' instructional leadership (e.g., the capacity of principals and
the channeling of resources, including technical assistance, to enhance
instructional leadership), and when these efforts were combined with
clear, measurable, and rigorous school accountability provisions.

Overall, the available and ongoing research demonstrates that the CSRD
program emerged against a solid backdrop of prior conceptual work and
empirical evidence. At the same time, the experiences have raised at
least two ongoing issues worth mentioning.[^13] First, "CSRD" is not
synonymous with "comprehensive school reform." As used in the present
report, CSRD has been defined specifically to represent the
congressionally legislated program that started in 1998---its major
distinction being the identification of nine specific components to be
implemented as part of comprehensive reform. In contrast, "comprehensive
school reform" does not carry the specific connotation that there are
nine components---although the principles of comprehensive school reform
are compatible with and may even be comparable in some instances to
CSRD's nine components.

Second, the reform heritage just traced has been heavily dominated by
the existence of research-based "models." A resulting confusion is that
the desired reform (and especially its comprehensiveness) has been
sometimes assumed to be no more than the adoption and implementation of
an externally-developed, research-based "model." However, in reality the
required reform effort must go beyond the "model," because most of the
"models" are not comprehensive. They focus on either a specific subject
matter (e.g., reading) or a particular phase of instruction (e.g.,
homework), or both; but a school needs to implement a reform that
embraces the entirety of its operations, covering all grades, all
subjects, and all students in the school. To this extent, the breadth of
successful reform will go well beyond the adoption and implementation of
most of the available research-based "models."

A final clarification of comprehensive school reform also derives from
the ongoing NAS experience. Originally, NAS considered "break the mold
designs" and "design teams"---and ultimately "externally developed
reform design"---to be the desired agents of change. The original
concept of "design," as used by NAS, may indeed have been sufficient to
capture the breadth of comprehensive reform. "Design" implies an overall
plan for the comprehensive reform of an entire school, incorporating all
aspects of the curriculum, school operation, and instruction. However,
many of the research-based "models" that later emerged were then assumed
to be comprehensive, especially because many had the characteristic of
being externally developed (e.g., CPRE, 1998). In fact, many of the
"models" were limited to specific grades or academic subjects and alone
could not support *comprehensive* reform unless they were integrated
into a comprehensive design or plan.

As a result, the present report considers a research-based "model" to be
a way of dealing with the first of the nine components of the
comprehensive school reform demonstration (CSRD) program--- implementing
research-based methods and strategies. Further, to avoid perpetuating
the use of the word "model" as necessarily implying comprehensive
reform, this report uses the language of the legislation, or
research-based "method" or "strategies," including when referring to
such "models" that are externally-developed (such as *Success for All*
or *Accelerated Schools*). Thus, externally-developed methods can lead
to comprehensive reform, but only if the method requires a school to
attend thoroughly the other eight CSRD components. However, if the
method does not lead in a comprehensive direction, a CSRD school must
undertake additional initiatives to cover the other eight components
necessary for the school to meet its CSRD mandate.

**1.2 THE FIELD-FOCUSED STUDY**

The present report covers the Field-Focused Study. It is one part of the
National Evaluation of the CSRD Program, which involved multiple related
efforts being conducted separately by different research organizations
(Doherty, 2000).[^14] The study called for data collection from a
stratified, random sample of 18 schools from the first two cohorts of
CSRD awardees. The data collection was based on intensive, field-based
efforts: two visits per school year, for each of two years. Appendix A
contains the detailed design and methodology for the study.

The Field-Focused Study was mainly aimed at describing the links among
school reform processes, instruction, and other educational activities.
The study involved in-depth examination of these linkages, thereby
limiting the number of schools that could be the subject of inquiry. In
addition, the schools in the Field-Focused Study also were examined as a
source of promising reform strategies (i.e., practices, programs, and
technical assistance or support strategies) to share with other schools
(see Chapter 2). Number range CHAPTERThe topics covered by the study and
in this report include:

1\) The progress in implementing the 9 CSRD components by the

18 schools;

2\) The role of district and state influences in implementing the
program;

3\) The apparent pathways or strategies for implementing comprehensive

school reform; and

4\) The early signs regarding the potential sustainability of the
program at

the school level.

Because the Field-Focused Study directly investigated the complexity of
events (including contextual conditions) involved with the nine CSRD
components as they might occur at an individual school, the study used
the case study method as the primary research method (Yin, 2003).[^15]
To tighten the analytic process, the application of the method to the
Field-Focused Study included four important features not always found in
other case study research, as follows.

***A theoretical framework guided the study***. The first feature was
the use of a theoretical framework---i.e., "*logic model*"---whose
development began prior to data collection. The evaluation team
developed a logic model to depict the working of CSRD at any given
school (see Exhibit 1-3). The model portrays the 9 components of CSRD as
boxes in the exhibit, with CSRD Component 4 ("Measurable Goals and
Benchmarks") being split into two boxes (4a and 4b). The generic model
also contains two important concepts not explicitly part of the original
9 CSRD components but that are necessarily involved in attempts to
improve student performance: *school organization* (located as part of
box 4a) and *instruction and learning* (see the unnumbered box).[^16] To
preserve space, the model does not show the contextual
conditions---e.g., district and state policies affecting the
school---although these conditions also have been part of the
Field-Focused Study.

The model, as with all logic models, shows all of the boxes, connected
by arrows, that hypothesizes how comprehensive school reform might
produce change in *student performance* (see Box 4b). The dotted arrows
are hypothetically related to the research-based method defined as CSRD
Component 1. The solid arrows are hypothetical but nevertheless serve as
logical links suggesting how other school actions can lead to effects,
which in turn become actions for other effects---all part of a chain of
events. Key to interpreting and using the overall model, therefore, is
that data and findings are needed about the "arrows,"[^17] and not just
the "boxes" in the model---that is, "how" and "why"

**Exhibit 1-3**

**THEORETICAL FRAMEWORK FOR A CSRD SCHOOL**

![](media/image2.wmf){width="5.802777777777778in" height="3.625in"}

events in one box affected those in another, not just the identity and
presence or absence of the events within each box.

In the Field-Focused Study, data collection teams were asked to
customize the generic logic model into a specific but tentative model
for their school(s) *prior to* actual data collection. The details of
the logic model were likely to vary, depending upon the nature of the
research-based method that the school had adopted to satisfy CSRD
Component 1, as well as other conditions that might have been revealed
during a site-screening phase that led to the selection of the school
for inclusion in the Field-Focused Study in the first place. The
stipulated logic model then became the occasion for field
inquiry---appreciating that "discovery" of unexpected events and
relationships also could occur as part of the data collection procedure.

The evaluation team could then customize the generic logic model,
depending upon the CSRD Component 1 research-based method or practice
that had been adopted (*Success for All* and *America's Choice* in these
examples) (see Exhibit 1-4). The labels in the boxes reflect the
activities emphasized by each method, and the arrows suggest the
connections implicit in each program. Important in these two logic
models are therefore the boxes with

**Exhibit 1-4**

**BEGINNING TO CUSTOMIZE THE ORIGINAL FRAMEWORK:**

**TWO DIFFERENT CSRD MODELS**

![](media/image3.wmf){width="6.0777777777777775in"
height="7.030555555555556in"}

no connecting arrows---implying that the adopted method does not, on the
surface, cover all of the CSRD components. To achieve comprehensive
school reform, a school must therefore find another way of addressing
the non-integrated component(s). For example, *Success for All*'s
defining feature is that it only covers reading---even though it covers
reading comprehensively; thus, the program in and of itself does not
connect with CSRD Component 2 ("Comprehensive and Aligned Design");
*America's Choice*'s attention to instruction omits any explicit
connection to CSRD Component 6 ("Parental and Community Involvement")
and Component 9 ("Coordination of Resources").

Overall, careful use of logic models strongly enhances the precision and
logic of case study research, especially in evaluation studies.

***Rival explanations were investigated***. A second feature was the
explicit identification and then collection of data about *rival
explanations* (Yin, 2000). In the case of the Field-Focused Study, the
rivals were alternative explanations for any of the observed
outcomes---e.g., whether conditions *other than* the CSRD funding and
mission could account for the observed student performance outcomes
(positive, negative, or neutral) or even for a school's reform-oriented
implementation actions.[^18] The use of rivals is aimed at reducing the
uncertainty about potential relationships, even though the reduction
will not nearly be as large as that occurring in a "true experiment."
Collecting data about possible rival explanations was therefore part of
the case studies of the CSRD schools. Such data collection strengthened
the case studies by broadening the data to be collected but keeping the
entire effort well-targeted.

***Chronological timelines traced key events***. A third feature was the
explicit use of chronological timelines to trace the key events in the
schools being studied. The temporal sequence of events was examined for
clues regarding the strength of CSRD or rival explanations. The
Field-Focused Study therefore attempted to collect data faithfully on
the timing of various events.

***Classrooms were observed***. A fourth and final feature involved data
collection aimed at revealing possible differences in implementation
patterns across different classrooms within each school. If reform
actions were incrementally implemented or affected one curriculum topic
but not another, the targeted classrooms should have shown earlier
evidence of implementation than the nontargeted classrooms. To put this
feature into place, the field teams used a formal classroom observation
instrument that was part of the case study protocol, described next.

***Site visit teams used a case study protocol***. The main instrument
for the Field-Focused Study was a case study protocol, contained as an
attachment to Appendix A, which also gives fuller details on the methods
used in the entire study. This protocol was the subject of ED and OMB
review, final clearance for which was received in October 2000.[^19]

The protocol was designed as an instrument aimed at the investigator,
not at any group of respondents. The instrument represented a field
team's agenda for collecting evidence from a variety of
sources---including but not limited to open-ended interviews. Using the
collected evidence, the protocol's topics were then addressed by the
team by creating a formal and documented database that included
quantitative as well as qualitative data. The focus of the protocol was
on tracking and documenting actual behaviors and events through
converging evidence. This objective may be contrasted with surveys,
where the information is based on self-reports by a survey's
respondents.

**1.3 SAMPLE OF SCHOOLS BEING STUDIED**

Though small in size, the role of the 18 schools in the Field-Focused
Study was nevertheless to represent all of the CSRD schools. A two-stage
sampling procedure, described fully in Appendix A, also attended to
stratifications by research-based method and by school district. The
operational goal was to have two schools with the same research-based
method (in different districts) and two schools in the same district
(but with different methods). The stratification provided the basis for
a minimal *replication* design, whereby implementation experiences could
more readily be attributed either to the method or to the district
conditions or to neither---thereby helping to arrive at a broader set of
conclusions than attainable through a set of individualized or
idiosyncratic case studies alone.[^20]

The selected schools were promised anonymity for participating in the
evaluation. As a result, Exhibit 1-5 only shows the locations of the
schools, by state. However, more detail about the schools'
characteristics can be found in Exhibit 7 of Appendix A.

**1.4 STUDY ACTIVITIES AND A GENERAL CAVEAT**

The Field-Focused Study was a multi-year effort whose main data
collection occurred over a two-year period (2000-01 and 2001-02).
Findings from the first two rounds of site visits, which took place
during each of the two semesters in the academic year 2000-01, were
reported in an annual report to ED in December of 2001 (COSMOS, 2001).
The current report updates and expands on those findings (Section 2),
and draws conclusions about the implementation of CSRD (Section 3). As a
general caveat in interpreting the findings, note that the Field-Focused
Study was not designed to determine the causal effects of the CSRD
program or of the specific research-based methods employed by the 18
schools.

**Exhibit 1-5**

**LOCATION OF THE FINAL SAMPLE OF CSRD SCHOOLS**

**FOR INCLUSION IN THE FIELD-FOCUSED STUDY**

![](media/image4.wmf){width="4.949305555555555in"
height="2.7930555555555556in"}

**2. CSRD IMPLEMENTATION:**

**THE 9 CSRD COMPONENTS**

The initial two rounds of site visits during 2000-01 provided
preliminary information about the status of CSRD implementation at the
18 schools. These findings were updated and expanded upon following the
final two rounds of site visits in 2001-02. During the final year of
data collection, five schools had officially completed CSRD and were to
be operating without CSRD funding[^21] (their CSRD activities began in
1998-99), and 13 were in their third and final year of funding (CSRD
activities began in 1999-00). The site visits collected data about: the
progress being made on the 9 CSRD components; salient district and state
conditions; the likelihood that reform would be sustained; and promising
reform strategies practices emerging from the sites.[^22]

This section discusses observations regarding each of the 9 CSRD
components, including the use of the research-based method at each
school (CSRD Component 1) and the breadth and comprehensiveness of the
schools' reform designs (CSRD Component 2). Throughout this section are
references to ***promising reform strategies*** (found in Appendix B)
investigated during site visits to the 18 CSRD schools. These strategies
are provided as examples of innovative or successful ways in which
schools addressed one or more of the 9 CSRD components.

Section 3 of this report then covers related data collection about
contextual conditions, such as district and state policies possibly
affecting comprehensive reform at the schools. The section summarizes
the overall strength of CSRD implementation, also examining the
conditions or pathways associated with implementation. The section ends
with an analysis of the likely sustainability of comprehensive school
reform following the end of CSRD funding.

**2.1 IMPLEMENTING RESEARCH-BASED METHODS**

**(CSRD Component 1)**

All of the schools initially adopted either a nationally or locally
developed research-based method (see Exhibit 2-1). Among the "national"
methods, 3 schools were implementing *Success for All (SFA)* only; 2
each were implementing *High Schools that*

*Work (HSTW*), *Accelerated Schools*, *Lightspan*, and *Coalition of
Essential Schools (CES)*; 1 each was implementing *Success in the Making
(SIM*) and *Co-NECT* and the dual method

# Exhibit 2-1

RESEARCH-BASED METHODS ADOPTED BY 18 CSRD SCHOOLS AND FIDELITY
ASSESSMENTS OBTAINED FROM DEVELOPERS

+-----------------+-----+-----------------+---------------------+----+
| **School        | **C | **Developer**   | **Fidelity          | ** |
| Na              | SRD |                 | Assessments         | Fi |
| me/*Method***\* | Aw  |                 | Obtained**          | de |
|                 | ard |                 |                     | li |
|                 | Sta |                 |                     | ty |
|                 | tus |                 |                     | R  |
|                 | in  |                 |                     | at |
|                 | 200 |                 |                     | in |
|                 | 1-0 |                 |                     | g* |
|                 | 2** |                 |                     | *^ |
|                 |     |                 |                     | 1^ |
+=================+=====+=================+=====================+====+
| **School A**    | P   | Accelerated     | > Assessment made   | Hi |
|                 | ost | Schools         | > by local          | gh |
| ***Accelerated  | -Aw |                 | > representative of |    |
| Schools***      | ard | University of   | > developer         |    |
|                 |     | Connecticut     |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School B**    | Y   | Southern        | > On-site reports   | Me |
|                 | ear | Regional Ed.    | > obtained from the | di |
| ***High Schools | 3   | Bd.;            | > developer         | um |
| That Work***    |     |                 |                     |    |
|                 |     | Atlanta, GA     |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School C**    | Y   | BBN Corporation | > Assessment made   | Me |
|                 | ear |                 | > by local          | di |
| ***Co-NECT***   | 3   | Cambridge, MA   | > representative of | um |
|                 |     |                 | > developer         |    |
+-----------------+-----+-----------------+---------------------+----+
| **School D**    | Y   | SFA             | > SFA semi-annual   | Hi |
|                 | ear |                 | > implementation    | gh |
| ***Success for  | 3   | Baltimore, MD   | > review reports    |    |
| All/Roots &     |     |                 |                     |    |
| Wings***        |     |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School E**    | P   | CES             | > Developer had no  | L  |
|                 | ost |                 | > contact with      | ow |
| ***Coalition of | -Aw | Oakland, CA     | > school; school    |    |
| Essential       | ard |                 | > not implementing  |    |
| Schools***      |     |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School F**    | Y   | CES             | > Assessment made   | Me |
|                 | ear |                 | > by local          | di |
| ***Coalition of | 3   | Oakland, CA     | > representative of | um |
| Essential       |     |                 | > developer         |    |
| Schools***      |     |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School G**    | Y   | Local           | > Assessment made   | Hi |
|                 | ear | university      | > by local          | gh |
| ***"Special     | 3   | professor       | > representative of |    |
| Literacy        |     |                 | > developer         |    |
| Model"***\*     |     |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School H**    | P   | SFA             | > Assessment made   | L  |
|                 | ost |                 | > by local          | ow |
| ***Success for  | -Aw | Baltimore, MD   | > representative of |    |
| All***          | ard |                 | > developer         |    |
+-----------------+-----+-----------------+---------------------+----+
| **School I**    | Y   | Lightspan       | > Self-assessment   | L  |
|                 | ear | Partnership     | > by school staff   | ow |
| ***Lightspan*** | 3   |                 | > (no contact with  |    |
|                 |     | San Diego, CA   | > developer)        |    |
+-----------------+-----+-----------------+---------------------+----+
| **School J**    | Y   | Local           | > Assessment made   | Me |
|                 | ear | superintendent  | > by local          | di |
| ***"Performance | 3   | of schools      | > representative of | um |
| Tasks           |     |                 | > developer         |    |
| Model"***\*     |     |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School K**    | Y   | SFA             | > SFA semi-annual   | Hi |
|                 | ear |                 | > implementation    | gh |
| ***Success for  | 3   | Baltimore, MD   | > review reports    |    |
| All***          |     |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School L**    | Y   | Local           | > Assessment made   | Me |
|                 | ear | consultant and  | > by local          | di |
| ***"Behavior    | 3   | school          | > representative of | um |
| Mod.            |     | leadership      | > developer         |    |
| Model"***\*     |     |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School M**    | Y   | Lightspan       | > Assessment made   | L  |
|                 | ear | Partnership     | > by local          | ow |
| ***Lightspan*** | 3   |                 | > representative of |    |
|                 |     | San Diego, CA   | > developer         |    |
+-----------------+-----+-----------------+---------------------+----+
| **School N**    | P   | SFA             | > SFA semi-annual   | Me |
|                 | ost |                 | > implementation    | di |
| ***Success for  | -Aw | Baltimore, MD   | > review reports    | um |
| All***          | ard |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School O**^2^ | Y   | Computer        | > Assessment made   | Me |
|                 | ear | Curriculum      | > by school         | di |
| ***Success-i    | 3   | Corp.;          | > coordinator;      | um |
| n-the-Making*** |     |                 | > developer made no |    |
|                 |     | Sunnyvale, CA   | > fidelity          |    |
|                 |     |                 | > judgements        |    |
+-----------------+-----+-----------------+---------------------+----+
| **School P**    | Y   | Southern        | > On-site reports   | Hi |
|                 | ear | Regional Ed.    | > obtained from the | gh |
| ***High Schools | 3   | Bd.; Atlanta,   | > developer         |    |
| that Work***    |     | GA              |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School Q**    | Y   | Accelerated     | > Assessment made   | Hi |
|                 | ear | Schools         | > by local          | gh |
| ***Accelerated  | 3   |                 | > representative of |    |
| Schools***      |     | University of   | > developer         |    |
|                 |     | Connecticut     |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| **School R**    | P   | School          | > Developer makes   | L  |
|                 | ost | leadership      | > no fidelity       | ow |
| ***"Comp.       | -Aw |                 | > judgments         |    |
| Reform          | ard |                 |                     |    |
| Model"***\*     |     |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+
| \*Locally       |     |                 |                     |    |
| developed       |     |                 |                     |    |
| methods are in  |     |                 |                     |    |
| quotes and have |     |                 |                     |    |
| been given      |     |                 |                     |    |
| pseudonyms.     |     |                 |                     |    |
|                 |     |                 |                     |    |
| ^1^Ratings are  |     |                 |                     |    |
| based on        |     |                 |                     |    |
| developer       |     |                 |                     |    |
| enumeration of  |     |                 |                     |    |
| presence of     |     |                 |                     |    |
| method          |     |                 |                     |    |
| components at   |     |                 |                     |    |
| each school, as |     |                 |                     |    |
| well as         |     |                 |                     |    |
| judgment about  |     |                 |                     |    |
| school's        |     |                 |                     |    |
| understanding   |     |                 |                     |    |
| of the method's |     |                 |                     |    |
| essence.        |     |                 |                     |    |
| Developers      |     |                 |                     |    |
| articulated     |     |                 |                     |    |
| judgments with  |     |                 |                     |    |
| some degree of  |     |                 |                     |    |
| detail about    |     |                 |                     |    |
| each school.    |     |                 |                     |    |
|                 |     |                 |                     |    |
| ^2^School O     |     |                 |                     |    |
| originally      |     |                 |                     |    |
| identified      |     |                 |                     |    |
| Co-NECT as its  |     |                 |                     |    |
| reform method   |     |                 |                     |    |
| in its CSRD     |     |                 |                     |    |
| application; it |     |                 |                     |    |
| decided instead |     |                 |                     |    |
| to expand the   |     |                 |                     |    |
| use of the SIM  |     |                 |                     |    |
| method.         |     |                 |                     |    |
+-----------------+-----+-----------------+---------------------+----+

of *SFA* and *Roots & Wings*; and 4 schools were implementing "locally
developed" methods.[^23] Appendix C contains descriptions of the methods
in the final sample.

***2.1.1 Implementation of the research-based method is fully proceeding
at 9 of the 18 schools, with partial or minimal implementation at the
other 9***. Using a five-point scale (*not implementing, planning*,
*piloting*, *implementing*, and *fulfilling or sustaining*),[^24] 15 of
the 18 schools were judged to be implementing the method at some level.
However, the teams' judgments distinguished between two subgroups: 9
schools (2 of the 9 had completed their CSRD grant period) that appeared
to be in the *implementing* or *fulfilling or sustaining* category, and
6 schools (all in their third year of the CSRD grant) still only
partially implementing (i.e., *planning* or *piloting*) their method
(see Exhibit 2-2). \[As a promising reform strategy, see "Stringent
Implementation of the Research-Based Method Believed to Result in
Improved Reading Score," in Appendix B.\]

Of the 6 schools judged to be only partially implementing (i.e., in the
planning or piloting phase) two schools (Schools I and M) adopted
*Lightspan* as their research-based method---a method requiring
extensive home-parent participation. Both schools had failed to achieve
such participation, and by the time of the final site visit, the method
was in use at only one grade level at each school. A third school
(School B) was a school whose implementation of a method only began to
expand in the second year of CSRD funding, but by its final year
appeared to be diminishing again. The other three schools (Schools C, L,
and O) were implementing portions of the method schoolwide but had not
yet adopted all of the components associated with the method.

Implementation appeared to be minimal or had stalled completely at three
schools (Schools E, H, and R---all of which had completed their CSRD
grant period), as explained in Exhibit 2-3.

***2.1.2 All but four schools are implementing the originally proposed
method***. With the exception of 4 of the 18 schools, all had kept to
the method named in their original CSRD application. Schools E and H
were no longer implementing the method they proposed for the reasons
described above. School R was only using remnants of its locally-

#   {#section-2 .unnumbered}

# Exhibit 2-2

**IMPLEMENTATION STATUS OF RESEARCH-BASED METHODS**

**(Component 1)**

+----------+-----------------+----------------------------------------+
| **Level  | **Scho          | **Implementation Status**              |
| of       | ol/*Method***\* |                                        |
| Implemen |                 |                                        |
| tation** |                 |                                        |
+----------+-----------------+----------------------------------------+
| *        | School A        | Elements of the method were observed   |
| *Appears |                 | in *all* classrooms; teachers were     |
| to be    | *Accelerated    | actively engaged in accountability and |
| "imple   | Schools*        | decisionmaking at the school, using    |
| menting" |                 | expertise to affect change.            |
| or       |                 |                                        |
| "fu      |                 |                                        |
| lfilling |                 |                                        |
| and**    |                 |                                        |
|          |                 |                                        |
| **sus    |                 |                                        |
| taining" |                 |                                        |
| the      |                 |                                        |
| method   |                 |                                        |
| (n=9)**  |                 |                                        |
+----------+-----------------+----------------------------------------+
|          | School D        | Method was in use in *all* classrooms  |
|          |                 | in reading and math; teachers and      |
|          | *Success for    | students were familiar and comfortable |
|          | All*            | with the curriculum.                   |
+----------+-----------------+----------------------------------------+
|          | School F        | *Nearly all* faculty were involved in  |
|          |                 | collaboration groups, school           |
|          | *Coalition of   | established three school-to-work       |
|          | Essential       | academies (with additional academies   |
|          | Schools*        | planned for future years).             |
+----------+-----------------+----------------------------------------+
|          | School G        | Materials and concepts were in use in  |
|          |                 | *every* classroom and in *all* facets  |
|          | *"Special       | of the schools curriculum.             |
|          | Literacy        |                                        |
|          | Model"*\*       |                                        |
+----------+-----------------+----------------------------------------+
|          | School J        | Aspects of the method were in use in   |
|          |                 | *all* classrooms; parts of the method  |
|          | *"Performance   | (i.e., portfolios) were more evident.  |
|          | Tasks Model"*\* |                                        |
+----------+-----------------+----------------------------------------+
|          | School K        | Method was in use in *all* classrooms, |
|          |                 | for *all* students, *every* day,       |
|          | *Success for    | though use in the upper grades was     |
|          | All*            | faltering.                             |
+----------+-----------------+----------------------------------------+
|          | School N        | *All* classrooms were using the method |
|          |                 | during the designated time, with       |
|          | *Success for    | varying levels of expertise.           |
|          | All*            |                                        |
+----------+-----------------+----------------------------------------+
|          | School P        | Use of method strategies was evident   |
|          |                 | in *nearly all* classrooms; class      |
|          | *High Schools   | periods scheduled in HSTW-recommended  |
|          | That Work*      | blocks.                                |
+----------+-----------------+----------------------------------------+
|          | School Q        | *Most* teachers and administrators     |
|          |                 | supported use of the method, and       |
|          | *Accelerated    | aspects were observed in *all*         |
|          | Schools*        | classrooms.                            |
+----------+-----------------+----------------------------------------+
| *        | School B        | School began implementation efforts,   |
| *Appears |                 | but *support faltered* during          |
| to be    | *High Schools   | 2000-2001 when staff *voted against    |
| p        | That Work*      | structural changes*.                   |
| artially |                 |                                        |
| impl     |                 |                                        |
| ementing |                 |                                        |
| the      |                 |                                        |
| method   |                 |                                        |
| (i.e.,   |                 |                                        |
| "p       |                 |                                        |
| lanning" |                 |                                        |
| or       |                 |                                        |
| pi       |                 |                                        |
| loting") |                 |                                        |
| (n=6)**  |                 |                                        |
+----------+-----------------+----------------------------------------+
|          | School C        | Teachers were organized into small     |
|          |                 | communities; students worked in        |
|          | *Co-NECT*       | interdisciplinary project groups;      |
|          |                 | technology use was improved. However,  |
|          |                 | the method was *not used equally by    |
|          |                 | all* staff.                            |
+----------+-----------------+----------------------------------------+
|          | School I        | No consistent use of the method        |
|          |                 | observed during 2000--01; changes in   |
|          | *Lightspan*     | implementation plan resulted for       |
|          |                 | 2001--02 and the method was used in    |
|          |                 | *all* 1st grade classrooms (and *80    |
|          |                 | percent* of 1st grade homes).          |
+----------+-----------------+----------------------------------------+
|          | School L        | Implementation of behavioral aspects   |
|          |                 | of the method *varied* by teacher;     |
|          | *"Behavior      | implementation of instructional        |
|          | Modification    | aspects was *minimal.*                 |
|          | Model"*\*       |                                        |
+----------+-----------------+----------------------------------------+
|          | School M        | Support for the method was limited     |
|          |                 | throughout CSRD period; despite        |
|          | *Lightspan*     | improvements in use in 2000-01, *only  |
|          |                 | one class level* and an afterschool    |
|          |                 | program used the method in 2001-02,    |
|          |                 | and *home implementation stopped*.     |
+----------+-----------------+----------------------------------------+
|          | School O        | Teachers used method *inconsistently,* |
|          |                 | with preparation for state assessment  |
|          | *Success        | taking precedence; implementation      |
|          | -in-the-Making* | became *less consistent in 2001-2002*  |
|          |                 | when state-intervention ended.         |
+----------+-----------------+----------------------------------------+
| **Is not | School E        | By 2001-2002, *implementation had      |
| imple    |                 | stopped*, as the school focused on     |
| menting, | *Coalition of   | organizational and behavioral          |
| or       | Essential       | problems; some aspects of the method   |
| ceased   | Schools*        | remained as part of school philosophy. |
| impl     |                 |                                        |
| ementing |                 |                                        |
| the      |                 |                                        |
| method   |                 |                                        |
| (n=3)**  |                 |                                        |
+----------+-----------------+----------------------------------------+
|          | School H        | School fully implemented the method    |
|          |                 | throughout CSRD award period; voted to |
|          | *Success for    | shift to district reading curriculum   |
|          | All*            | and *implementation stopped* in        |
|          |                 | 2001-2002.                             |
+----------+-----------------+----------------------------------------+
|          | School R        | Aspects of the method were initiated,  |
|          |                 | but *diminished (almost entirely)*     |
|          | *"Comprehensive | after first year of funding; in        |
|          | Reform          | 2001-2002, some elements were being    |
|          | Model"*\*       | *revived*, but with no attention to    |
|          |                 | any CSRD design.                       |
+----------+-----------------+----------------------------------------+
| \        |                 |                                        |
| *Locally |                 |                                        |
| d        |                 |                                        |
| eveloped |                 |                                        |
| methods  |                 |                                        |
| are in   |                 |                                        |
| quotes   |                 |                                        |
| and have |                 |                                        |
| been     |                 |                                        |
| given    |                 |                                        |
| pse      |                 |                                        |
| udonyms. |                 |                                        |
+----------+-----------------+----------------------------------------+

**Exhibit 2-3**

**THREE CSRD SCHOOLS WITH MINIMAL OR STALLED IMPLEMENTATION OF THE
ORIGINALLY-ADOPTED RESEARCH-BASED METHOD**

+---------------------+------------------------------------------------+
| **                  | **Reason for "Not Implementing" Score**        |
| School/*Method\**** |                                                |
+---------------------+------------------------------------------------+
| **School E**        | School E underwent significant organizational  |
|                     | changes (splitting from a middle school and    |
| ***Coalition of     | becoming a separate charter high school), and  |
| Essential           | the attention and energy of staff shifted      |
| Schools***          | between year 2 and year 3 from reform to these |
|                     | structural changes. In addition, following the |
|                     | move to a new location, the school staff were  |
|                     | faced with further school management           |
|                     | problems--- including disciplinary issues with |
|                     | students, as well as the general               |
|                     | administration of the school.                  |
+---------------------+------------------------------------------------+
| **School H**        | During the first two rounds of site visits in  |
|                     | 2000-01, the site visit team identified School |
| ***Success for      | H as fully implementing the method. The loss   |
| All***              | of CSRD funding was accompanied by a shift in  |
|                     | district support for the method, a greater     |
|                     | emphasis in the district and state on student  |
|                     | achievement scores, and a faculty vote at      |
|                     | School H to stop implementing the method       |
|                     | during the 2001-02 school year. While the      |
|                     | school staff continue to use some of the       |
|                     | method's practices, the school's               |
|                     | implementation status nevertheless shifted to  |
|                     | the "not implementing" category.               |
+---------------------+------------------------------------------------+
| **School R**        | School R had started implementing a locally    |
|                     | developed, schoolwide reform method in its     |
| ***"Comprehensive   | first CSRD year (1998-1999), following several |
| Reform Model"\****  | years of planning by the school's top          |
|                     | officials and faculty. However, the principal  |
|                     | who had led the reform planning left School R  |
|                     | just before the start of the first CSRD year,  |
|                     | and the school then had three different        |
|                     | principals during the three-year period        |
|                     | coinciding with its CSRD award. During this    |
|                     | period, although implementation of the method  |
|                     | had begun in one grade in 1998-1999,           |
|                     | commitment to the method diminished rapidly    |
|                     | with the succession of principals, and as of   |
|                     | 2000-2001 none of the method's features were   |
|                     | in place. The school obtained a one-year,      |
|                     | no-cost extension for their CSRD award^1^ and  |
|                     | in 2001--02, and began to implement remnants   |
|                     | of the original reform method, but without any |
|                     | understanding of the relation of those         |
|                     | elements to a CSRD reform design. The school   |
|                     | therefore was still judged to be "not          |
|                     | implementing."                                 |
+---------------------+------------------------------------------------+

\* Locally developed methods are in quotes and have been given
pseudonyms.

^1^ The school spent only a fraction of its CSRD funds by the end of
year 3. The newest principal (serving his first year during 2000-01) was
committed to revisiting the CSRD activities and the state subsequently
approved the request.

developed method that aligned with district policy and the changing
needs of the school. School O changed its method from the one originally
proposed in its CSRD application because, following the submission of
the CSRD application, the school received a poor performance rating in
the state's accountability system. Because of the poor rating, the
district encouraged School O to expand its use of an alternative method,
which was being implemented at many other district schools. With state
approval, School O therefore switched from its originally proposed
*Co-NECT* method to *Success-in-the-Making* (*SIM*).[^25]

A fifth school (School N) was implementing the originally proposed
method, but the field team failed to identify it properly during the
screening procedure. School N had been expected to adopt *Roots &
Wings*, a method developed by the Success for All Foundation to cover
mathematics, science, and social studies, in addition to reading.
However, although School N was later found to have only implemented
*Success for All* (and hence was only covering reading), the school was
using both the "roots" level and the "wings" level of *SFA* (but in
reading only). During the screening, the school therefore responded that
it was using *Roots & Wings* when in fact no method was being used
outside of reading.[^26]

***2.1.3 Developers gave mixed fidelity ratings to the schools***. The
field teams were able to obtain information about fidelity from
developers for 14 of the 18 schools (including locally developed
methods).[^27] In 6 of these 14 cases, the developers judged the
fidelity of implementation to be high; in 1 of the 2 judged to be low,
implementation also was low as judged by the site visit team (the other
school was rated as having partial implementation). The teams were
unable to obtain fidelity ratings from the method developers for four
schools: two schools were implementing a national method, but had no
contact with the developer; one school was implementing a
locally-developed method and had no external consultant; and one
school's developer was not willing to make a judgement about fidelity.
In all but the last case, the school's own assessment of their fidelity
was "low" (the school with the developer unwilling to make an assessment
rated their own fidelity as "medium"). Exhibit 2-1 also contains the
ratings made about the fidelity in implementing each method.

***2.1.4 Experiences between locally developed and national methods have
not markedly differed.*** Of particular interest regarding these methods
was the collection of *locally developed* methods, for two reasons.
First, at the national level, less is known about them than about the
"national" methods. Second, one possibility entertained at the outset of
the Field-Focused Study was that a CSRD school might have shared in the
development of a locally developed method (e.g., serving as a
co-developer), thereby qualitatively altering its expected CSRD
implementation experience.

Of the 4 locally developed methods, 3 were similar to national methods
in having: a formal developer; prior evidence of effectiveness; and
ongoing use at two or more schools. Proportionately, the implementation
experiences by the schools with locally developed methods did not differ
markedly, as shown previously in Exhibit 2-2, from those of the national
methods. Further, with the exception noted below, the schools did not
have any role in developing the methods. Thus the main difference
between the national and locally developed methods appears to be that
the latter were only known and being used within a limited regional
area---i.e., either a single school district or multiple regions within
a single state.

As one result, the field teams were able to locate the local method
developers and to collect data from them similar to the data from the
national method developers. For instance, the local developers were
providing schools with external technical assistance, similar to the
assistance being provided by their counterparts with national methods.
The single exception again had to do with School R, whose locally
developed method was a truly "home-grown" combination of several
practices, none of which itself had any method that had a
developer.[^28]

Two of the methods (adopted by Schools G and J) had both instructional
and curriculum content (see Exhibit 2-4). For instance, School J's
method involved the development and maintenance of a bank of substantive
student tasks ("authentic" tasks) covering each academic subject and
enabling students to develop portfolios of their work. The tasks also
promoted reasoning and thinking as part of the learning process. In
addition, local teachers were encouraged to use their classroom
experiences to devise and nominate new tasks to be reviewed for
inclusion in the bank. \[As a promising reform strategy, see
"Research-Based Method Guiding Scope and Sequence of Lessons in All

**Exhibit 2-4**

**FOUR LOCALLY DEVELOPED METHODS**

+------+----------+----------------------------------------------------+
| ##   | **Method | **Description**                                    |
| # Sc | Pse      |                                                    |
| hool | udonym** |                                                    |
+------+----------+----------------------------------------------------+
| **Sc | *        | The method has three goals: 1) to implement        |
| hool | "Special | effective practices for K-3 literacy learning and  |
| G**  | Literacy | teaching; 2) to develop teacher leadership teams   |
|      | Model'"* | within a school and its district; and 3) to        |
|      |          | develop mentor teachers to support the preservice  |
|      |          | education at the university. The method is         |
|      |          | centered around a "special topics" university      |
|      |          | graduate course. Teachers receive credit for their |
|      |          | method-related participation. An initial two-week  |
|      |          | summer institute, attended by representatives from |
|      |          | the school, helps to plan the activities over the  |
|      |          | ensuing academic year, such as the establishment   |
|      |          | of teacher teams, parent literacy nights, and      |
|      |          | parent mathematics nights. The method claims to be |
|      |          | highly effective in building good instructional    |
|      |          | practice, teacher cooperation, and the use of a    |
|      |          | specific array of instructional materials. The     |
|      |          | method is based on academic research on early      |
|      |          | literacy.                                          |
+------+----------+----------------------------------------------------+
| **Sc | *"Per    | The method is composed of discrete performance     |
| hool | formance | tasks, accessible via a Web site, that teachers    |
| J**  | Tasks    | may select and use to deliver instruction. Tasks   |
|      | Model"*  | are available for all content areas (though more   |
|      |          | are available for reading/language arts than for   |
|      |          | any other area). Tasks are designed to be examples |
|      |          | of real-life experiences ("authentic" tasks)       |
|      |          | enabling students to acquire content knowledge,    |
|      |          | good work habits, and thinking and communication   |
|      |          | skills. The method promotes reasoning and thinking |
|      |          | as part of the learning process. Examples include  |
|      |          | organizing a healthy luncheon, guiding classmates  |
|      |          | on a trip, corresponding with a scientific author, |
|      |          | and composing a song. The method also incorporates |
|      |          | student self-assessments for judging the quality   |
|      |          | of a product and overall student performance.      |
|      |          | Students develop a portfolio of their work, and    |
|      |          | these, along with students' self-assessments, are  |
|      |          | used to demonstrate and evaluate process and       |
|      |          | achievement.                                       |
+------+----------+----------------------------------------------------+
| **Sc | *"       | The goal of the method is to achieve schoolwide    |
| hool | Behavior | change enabling "greater academic achievement,     |
| L**  | Modi     | productivity, and prosocial competency." The       |
|      | fication | method consists of a behavioral protocol,          |
|      | Model"*  | climate-setting strategies, and instructional      |
|      |          | approaches to improving student achievement.       |
|      |          | Teachers use techniques such as the good behavior  |
|      |          | game (used during regular instruction time to      |
|      |          | reward good behavior and cite inappropriate        |
|      |          | behavior) to teach students what behaviors to      |
|      |          | exhibit and what behaviors to inhibit. Other       |
|      |          | behavior modification techniques also are          |
|      |          | prescribed by the method, including organizing     |
|      |          | students in the classroom into teams, giving       |
|      |          | students "jobs" within the classroom, and student  |
|      |          | monitoring. In addition, students are taught to    |
|      |          | walk in "model lines" (or one behind the other,    |
|      |          | with their hands behind their backs) and to use    |
|      |          | "model voices" (volume levels appropriate to the   |
|      |          | situation). Other elements of the method have      |
|      |          | included having students graph their own           |
|      |          | performance in spelling and math, so that they     |
|      |          | learn how to do self-assessments and can see their |
|      |          | progress, and engage in peer tutoring.             |
+------+----------+----------------------------------------------------+
| **Sc | *"Compr  | The original method included the implementation of |
| hool | ehensive | state standards, which the school identified as    |
| R**  | Reform   | among the first at the state level to embrace      |
|      | Model"*  | educational quality reform and "significantly      |
|      |          | different from other state and national models in  |
|      |          | that they were more project- and                   |
|      |          | performance-oriented." Other components of the     |
|      |          | comprehensive method included a *Freshman          |
|      |          | Foundation* course, intended to help freshman to   |
|      |          | get prepared for high school; a flexible *block    |
|      |          | schedule*; an *interdisciplinary curriculum*,      |
|      |          | whereby teachers of different subjects would have  |
|      |          | common planning time, hold daily team meetings,    |
|      |          | and coordinate their instructional schedule; the   |
|      |          | integration of *school-to-work (school-to-career)  |
|      |          | topics*; the integrated use of *technology* into   |
|      |          | the curriculum, especially taking advantage of     |
|      |          | individually-assigned laptop computers; and a      |
|      |          | vision of the school as having a *"global"         |
|      |          | magnet*, transcending the four separate magnet     |
|      |          | programs at the school. In the final year of       |
|      |          | implementation, the method had been revised: 1)    |
|      |          | implementation of state standards continued, 2)    |
|      |          | the Freshman Foundation course was replaced with   |
|      |          | Freshman Teams and the use of "My Action Plans     |
|      |          | (MAPs)" by individual students to guide progress   |
|      |          | through high school and toward future goals; and   |
|      |          | 3) the laptop project ended; and 4) the concept of |
|      |          | a "global" magnet was replaced with district       |
|      |          | required small learning communities.               |
+------+----------+----------------------------------------------------+

Subject Areas," in Appendix B.\] The methods at the two other schools
(Schools L and R) more closely resembled "process" methods, focusing
either on behavior and climate-setting strategies (School L) or on the
use of technology in the curriculum and Freshman transition initiatives
(School R). \[As a promising reform strategy, see "Teacher Teams Provide
Students with Cross-Curricular Authentic Learning Opportunities," in
Appendix B.\]

***2.1.5 Classroom observations are consistent about ongoing experiences
with the methods***. Direct observations of classrooms in all of the 18
schools were an integral part of assessing the extent and nature of
implementation.[^29] In only a few cases was the observed implementation
different from that expected as a result of the interviews of the
schools' officials. For instance, Schools J and B had individual
classrooms where implementation was much lower than expected based on
the officials' interviews. In the opposite direction, School F's
classrooms showed a high degree of implementation even though the
interviews had described implementation as still "in progress."

More difficult to assess through classroom observation was any potential
role of the other CSRD components. Brief interviews with the teachers
whose classrooms had been observed proved more informative. The teachers
reported about: their professional development experiences, their
perceptions of staff support for reform; and apparent parent and
community support in conjunction with their classroom activities.

**2.2 COMPREHENSIVENESS OF DESIGNS FOR SCHOOL REFORM**

**(CSRD Component 2)**

The field teams assessed the extent and quality of comprehensiveness by
attending to the presence of the eight remaining CSRD components and
their apparent linkage. Key among these other 8 components is Component
2 (evidence of a "comprehensive design"), which the teams assumed needed
to exist if implementation was later to achieve actual
comprehensiveness.

To assess progress on this component, the field teams first tried to
obtain any documentation of such designs---e.g., the breadth and scope
of any school improvement plan, or even the apparent comprehensiveness
and convergence of a school budget that integrated CSRD funding within
it.[^30] The teams then also tried to determine the school staff's level
of awareness and understanding of reform, independent of whether such
documentation existed or not. On both fronts, the teams' inquiry was
aimed at determining whether the school understood that CSRD meant
reforming the entire school---i.e., potentially expanding to its entire
way of doing business and all its operations---and not simply adding a
new function or project or activity; and that in carrying out such
reform, a school would necessarily have to attend to the alignment or
connectivity among all of its operations.

***2.2.1 Less than half of the 18 schools had reforms with a
comprehensive design***. Eight of the 18 schools (Schools A, B, D, F, G,
J, L, and P) satisfied both conditions: a) school reform plans or
documentation that were comprehensive, and b) acknowledgments or
understandings of comprehensiveness by the school staff who had been
interviewed (see Exhibit 2-5). Six other schools (Schools C, H, K, N, O,
and Q) had comprehensive plans or documentation, such as school
improvement plans, but staff were unaware of or detached from any
comprehensive vision associated with such documentation. Where
understandings of comprehensiveness fell short, the general reason was
that CSRD breadth or implementation strongly resembled a separate
project or even "add-on" activity at the school. Among most of the
remaining schools, neither comprehensive documentation nor requisite
staff understanding was present.

***2.2.2 Comprehensiveness of CSRD can emerge from either working with
comprehensive research-based methods or having a comprehensive school
improvement plan***. Of the 8 schools that had satisfied both
conditions, the comprehensive designs for 5 of them (Schools A, B, F, J,
and P) were associated with the comprehensiveness of the research-based
methods that had been adopted, but not necessarily because the school
had developed any separate comprehensive reform vision or plan; for the
other 3 schools (Schools D, G, and L), the comprehensiveness was
associated with a required school improvement plan that was
comprehensive even if the research-based method was not. \[As a
promising reform strategy, see "School Improvement Planning Process
Aligns Reform Initiatives," in Appendix B.\]

Overall, 10 of the 18 schools may not yet have fully embraced the
principles of reform and steps needed to achieve *comprehensive* reform.
Compared to the proportion of schools implementing research-based
methods (CSRD Component 1), the proportion of schools with comprehensive
designs (CSRD Component 2) was substantially lower.

# Exhibit 2-5

**EVIDENCE OF COMPREHENSIVE DESIGN**

**(Component 2)**

+-------------------+--------------------------------------------------+
| **Sc              | **Evidence of Comprehensive Design**             |
| hool/*Method***\* |                                                  |
+===================+==================================================+
| **School A**      | Interviews, classroom observations and documents |
|                   | all made clear that the staff understood and     |
| ***Accelerated    | strongly supported the reform process, including |
| Schools***        | some familiarity with the nine CSRD Components.  |
+-------------------+--------------------------------------------------+
| **School B**      | > The reform strategies were integrated into the |
|                   | > school improvement plan. The method and plan   |
| ***High Schools   | > appeared to be well understood by staff,       |
| That Work***      | > students, and parents, though support for some |
|                   | > elements was declining.                        |
+-------------------+--------------------------------------------------+
| **School C**      | > The school had a single plan, required by the  |
|                   | > district, linking all activities to goals,     |
| ***Co-NECT***     | > objectives, benchmarks, and strategies. The    |
|                   | > plan served as funding application for CSRD    |
|                   | > and was aligned with state standards. However, |
|                   | > staff were reportedly disappointed with the    |
|                   | > plan, which they believed failed to capture    |
|                   | > all activities and funding streams. Co-NECT    |
|                   | > was recognized staff as the only comprehensive |
|                   | > reform strategy.                               |
+-------------------+--------------------------------------------------+
| **School D**      | > The district required the school to develop a  |
|                   | > comprehensive plan aligned to the district's   |
| ***Success for    | > vision for improving student performance. Much |
| All/Roots &       | > of the district plan paralleled the CSRD       |
| Wings***          | > reform components. Many of the strategies in   |
|                   | > the plan were specifically tied to the reform  |
|                   | > methods. Evidence that the staff supported the |
|                   | > plan were visible throughout the school,       |
|                   | > particularly in display materials posted on    |
|                   | > walls in classrooms and gathering areas.       |
+-------------------+--------------------------------------------------+
| **School E**      | > By the final year of funding, the staff had no |
|                   | > vision for comprehensive school reform,        |
| ***Coalition of   | > probably due to its pre-occupation with its    |
| Essential         | > own survival and restructuring. While many     |
| Schools***        | > staff shared a common philosophy about reform, |
|                   | > the methods or means for accomplishing this    |
|                   | > vision were a source of conflict.              |
+-------------------+--------------------------------------------------+
| **School F**      | > The CSRD reform plan encompassed all 9 CSRD    |
|                   | > components, though the school improvement plan |
| ***Coalition of   | > focused solely on goals to increase student    |
| Essential         | > performance. The majority of staff clearly     |
| Schools***        | > understood and supported the reform plan.      |
+-------------------+--------------------------------------------------+
| **School G**      | > The district-required school improvement       |
|                   | > planning process played a central role in      |
| ***"Special       | > establishing the comprehensiveness of the      |
| Literacy          | > reform effort. Every faculty member signs the  |
| Model"***\*       | > plan to indicate an understanding of his or    |
|                   | > her role in implementing the plan.             |
+-------------------+--------------------------------------------------+
| **School H**      | > The school improvement plan had a              |
|                   | > comprehensive set of goals and strategies,     |
| ***Success for    | > generally aligned with the CSRD components, by |
| All***            | > driven by the state assessment. The            |
|                   | > comprehensiveness of the reform effort was     |
|                   | > widely understood, though by 2001-02, the      |
|                   | > staff felt their reform efforts were in        |
|                   | > conflict with the districtwide reform agenda   |
|                   | > and plan.                                      |
+-------------------+--------------------------------------------------+
| **School I**      | > The school had two plans, one covering         |
|                   | > achievement goals, and another summarizing a   |
| ***Lightspan***   | > variety of school initiatives. Neither offered |
|                   | > a comprehensive picture or plan for the nine   |
|                   | > CSRD components. Teachers described the school |
|                   | > as having a split focus and a mission that was |
|                   | > way to broad.                                  |
+-------------------+--------------------------------------------------+
| **School J**      | > The school's improvement plan did not address  |
|                   | > the CSRD reform components, but did reflect    |
| ***"Performance   | > the school's goals and strategies for          |
| Tasks Model"***\* | > improving student achievement. The method      |
|                   | > itself included all reform components, and     |
|                   | > staff appeared to understand and support both  |
|                   | > the reform effort and the method.              |
+-------------------+--------------------------------------------------+
| \*Locally         |                                                  |
| developed methods |                                                  |
| are in quotes and |                                                  |
| have been given   |                                                  |
| "pseudonyms."     |                                                  |
|                   |                                                  |
| **(Continued on   |                                                  |
| next page)**      |                                                  |
+-------------------+--------------------------------------------------+
| **School K**      | > The school improvement plan addressed each     |
|                   | > component in relation to reading, but little   |
| ***Success for    | > emphasis was placed on other subjects. A       |
| All***            | > steering committee managed implementation and  |
|                   | > integration of reform components. Staff        |
|                   | > appeared to understand reform effort.          |
+-------------------+--------------------------------------------------+
| **School L**      | > The district required the school to develop a  |
|                   | > comprehensive plan aligned to the district's   |
| ***"Behavior      | > vision for improving student performance,      |
| Modification      | > which paralleled the CSRD reform components.   |
| Model"***\*       | > Staff appeared to understand the district      |
|                   | > reform plan, which was comprehensive (even     |
|                   | > though the research-based method did not       |
|                   | > appear to be well-integrated into this plan).  |
+-------------------+--------------------------------------------------+
| **School M**      | > The school's original improvement plan         |
|                   | > included Lightspan, but didn't cover the other |
| ***Lightspan***   | > reform components. A revised plan was          |
|                   | > finalized in June 2001 and was intended to be  |
|                   | > aligned with state and district standards,     |
|                   | > however, a new school administration the       |
|                   | > following year did not think the plan          |
|                   | > addressed the school's needs sufficiently.     |
|                   | > Some staff believed the plan was being ignored |
|                   | > and others argued it was not a practical       |
|                   | > document.                                      |
+-------------------+--------------------------------------------------+
| **School N**      | > All 9 elements of the reform were present in   |
|                   | > the school. A new plan was adopted in          |
| ***Success for    | > 2001--2002 that aligned the plan to the state  |
| All***            | > assessment, and set strategies for reaching    |
|                   | > improvement goals. However, it did not appear  |
|                   | > to be a working document at the school;        |
|                   | > programmatic and other decisions were made     |
|                   | > without reference to the overall design.       |
+-------------------+--------------------------------------------------+
| **School O**      | > The school's improvement plan listed its       |
|                   | > objective as improving student performance on  |
| ***Success        | > state assessments; the plan contained          |
| -in-the-Making*** | > references to a number of reform components.   |
|                   | > The breath of the planning process and SIM     |
|                   | > appeared to provide a unifying plan of action  |
|                   | > throughout the school.                         |
+-------------------+--------------------------------------------------+
| **School P**      | > The reform strategies were integrated into the |
|                   | > school improvement plan. Staff and the core    |
| ***High Schools   | > leadership team understood and supported the   |
| That Work***      | > method and the reform plan.                    |
+-------------------+--------------------------------------------------+
| **School Q**      | > The school addressed the nine reform           |
|                   | > components in their proposal, with varying     |
| ***Accelerated    | > levels of detail. Some staff, particularly on  |
| Schools***        | > the leadership committee supported the plan,   |
|                   | > but high turnover among staff resulted little  |
|                   | > understanding or support for the comprehensive |
|                   | > reform design by a large number of newer       |
|                   | > staff.                                         |
+-------------------+--------------------------------------------------+
| **School R**      | > The method initially reflected comprehensive   |
|                   | > whole school reform; but was not used after    |
| ***"Comprehensive | > the first year of CSRD funding. During the     |
| Reform            | > school's final CSRD year, the reform effort    |
| Model"***\*       | > was revived, but the lack of a clear and       |
|                   | > comprehensive plan (understood by all staff)   |
|                   | > was limiting the impact of the changes.        |
+-------------------+--------------------------------------------------+
| \*Locally         |                                                  |
| developed methods |                                                  |
| are in quotes and |                                                  |
| have been given   |                                                  |
| "pseudonyms."     |                                                  |
+-------------------+--------------------------------------------------+

**2.3 PROGRESS IN IMPLEMENTING THE 7 OTHER CSRD COMPONENTS**

As with the first two components, the site visit protocol called for
covering each of the other seven CSRD components individually. However,
the findings from the site visits suggested that these seven components
could be dealt with in several combinations, in the interest of
streamlining the discussion:

-   Professional development (Component 3) and external technical

support and assistance (Component 7);

-   Measurable goals and benchmarks (Component 4) and evaluation

strategies (Component 8);

-   Support within the school (Component 5) and parent and

community involvement (Component 6); and

-   Coordination of resources (Component 9) and sustainability.

To be noted is that the last combination joins one of the seven
components (Component 9) with another topic of high interest but that
has not been regarded as a separate CSRD component---*sustainability*.
Inherent to comprehensive school reform is that efforts cannot be
short-lived as in a specially funded "project." Thus the Field-Focused
Study has devoted explicit attention to the issue of sustainability
(Section 3.5 addresses the issue of sustainability in greater detail).

***2.3.1 Both professional development (Component 3) and external
technical support and assistance (Component 7) have largely been devoted
to the implementation of the research-based method, not necessarily
comprehensive reform more broadly***. The rationale for pairing these
two CSRD components is as follows: Most schools were engaged in moderate
to intensive professional development in relation to implementing CSRD,
but most if not all of this professional development was being provided
by an external technical support team (see Exhibit 2-6).[^31]

Exhibit 2-6 summarizes the professional development reported in the site
visits. The exhibit distinguishes between method-related professional
development and other, reform-related professional development. The last
column in the exhibit also comments on any unusual barriers to
professional development identified through the site visits.

**Exhibit 2-6**

# PROFESSIONAL DEVELOPMENT ACTIVITIES 

**(Component 3)**

+-----------+-----------------------+---+------------------+--------------+
| **School  | **Types and Frequency | * |                  | **Barriers** |
| Name/**   | of**                  | * |                  |              |
|           |                       | O |                  |              |
| ***Me     | **Method-Related      | t |                  |              |
| thod*\*** | Professional          | h |                  |              |
|           | Development (PD)      | e |                  |              |
|           | Activities**          | r |                  |              |
|           |                       | P |                  |              |
|           |                       | r |                  |              |
|           |                       | o |                  |              |
|           |                       | f |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | s |                  |              |
|           |                       | i |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | a |                  |              |
|           |                       | l |                  |              |
|           |                       | D |                  |              |
|           |                       | e |                  |              |
|           |                       | v |                  |              |
|           |                       | e |                  |              |
|           |                       | l |                  |              |
|           |                       | o |                  |              |
|           |                       | p |                  |              |
|           |                       | m |                  |              |
|           |                       | e |                  |              |
|           |                       | n |                  |              |
|           |                       | t |                  |              |
|           |                       | A |                  |              |
|           |                       | c |                  |              |
|           |                       | t |                  |              |
|           |                       | i |                  |              |
|           |                       | v |                  |              |
|           |                       | i |                  |              |
|           |                       | t |                  |              |
|           |                       | i |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | * |                  |              |
|           |                       | * |                  |              |
+===========+=======================+===+==================+==============+
| **School  | Extensive training in | S |                  | D            |
| A**       | AS philosophy and     | t |                  | istrict-wide |
|           | "powerful learning"   | a |                  | budget cuts  |
| ***Ac     | strategies for all    | f |                  | would limit  |
| celerated | staff; weekly cadres  | f |                  | formal PD    |
| S         | meetings allowed      | r |                  | activities   |
| chools*** | staff to identify     | e |                  | in 2002-03.  |
|           | issues of concern for | c |                  |              |
|           | entire school and     | e |                  |              |
|           | develop hypotheses to | i |                  |              |
|           | address them.         | v |                  |              |
|           |                       | e |                  |              |
|           |                       | d |                  |              |
|           |                       | t |                  |              |
|           |                       | r |                  |              |
|           |                       | a |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | s |                  |              |
|           |                       | u |                  |              |
|           |                       | c |                  |              |
|           |                       | c |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | s |                  |              |
|           |                       | f |                  |              |
|           |                       | u |                  |              |
|           |                       | l |                  |              |
|           |                       | r |                  |              |
|           |                       | e |                  |              |
|           |                       | a |                  |              |
|           |                       | d |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | r |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | g |                  |              |
|           |                       | i |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | f |                  |              |
|           |                       | o |                  |              |
|           |                       | r |                  |              |
|           |                       | L |                  |              |
|           |                       | E |                  |              |
|           |                       | P |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | u |                  |              |
|           |                       | d |                  |              |
|           |                       | e |                  |              |
|           |                       | n |                  |              |
|           |                       | t |                  |              |
|           |                       | s |                  |              |
|           |                       | f |                  |              |
|           |                       | r |                  |              |
|           |                       | o |                  |              |
|           |                       | m |                  |              |
|           |                       | l |                  |              |
|           |                       | o |                  |              |
|           |                       | c |                  |              |
|           |                       | a |                  |              |
|           |                       | l |                  |              |
|           |                       | u |                  |              |
|           |                       | n |                  |              |
|           |                       | i |                  |              |
|           |                       | v |                  |              |
|           |                       | e |                  |              |
|           |                       | r |                  |              |
|           |                       | s |                  |              |
|           |                       | i |                  |              |
|           |                       | t |                  |              |
|           |                       | y |                  |              |
|           |                       | c |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | s |                  |              |
|           |                       | u |                  |              |
|           |                       | l |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | n |                  |              |
|           |                       | t |                  |              |
|           |                       | s |                  |              |
|           |                       | , |                  |              |
|           |                       | a |                  |              |
|           |                       | s |                  |              |
|           |                       | w |                  |              |
|           |                       | e |                  |              |
|           |                       | l |                  |              |
|           |                       | l |                  |              |
|           |                       | a |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | p |                  |              |
|           |                       | r |                  |              |
|           |                       | e |                  |              |
|           |                       | p |                  |              |
|           |                       | a |                  |              |
|           |                       | r |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | i |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | a |                  |              |
|           |                       | n |                  |              |
|           |                       | d |                  |              |
|           |                       | l |                  |              |
|           |                       | e |                  |              |
|           |                       | a |                  |              |
|           |                       | d |                  |              |
|           |                       | e |                  |              |
|           |                       | r |                  |              |
|           |                       | s |                  |              |
|           |                       | h |                  |              |
|           |                       | i |                  |              |
|           |                       | p |                  |              |
|           |                       | s |                  |              |
|           |                       | k |                  |              |
|           |                       | i |                  |              |
|           |                       | l |                  |              |
|           |                       | l |                  |              |
|           |                       | s |                  |              |
|           |                       | . |                  |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | CSRD increased PD     | I |                  | Nearly       |
| B**       | activities by 12      | m |                  | one-third of |
|           | times compared to     | p |                  | the teachers |
| ***High   | year prior to grant;  | l |                  | quit or      |
| Schools   | all activities were   | e |                  | retired;     |
| That      | focused on HSTW and   | m |                  | support for  |
| Work***   | included peer-to-peer | e |                  | HSTW was     |
|           | presentations.        | n |                  | fading,      |
|           |                       | t |                  | though       |
|           |                       | a |                  | upcoming     |
|           |                       | t |                  | merger with  |
|           |                       | i |                  | county high  |
|           |                       | o |                  | school may   |
|           |                       | n |                  | revive the   |
|           |                       | o |                  | reform       |
|           |                       | f |                  | effort.      |
|           |                       | * |                  |              |
|           |                       | M |                  |              |
|           |                       | a |                  |              |
|           |                       | k |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | S |                  |              |
|           |                       | c |                  |              |
|           |                       | h |                  |              |
|           |                       | o |                  |              |
|           |                       | o |                  |              |
|           |                       | l |                  |              |
|           |                       | s |                  |              |
|           |                       | W |                  |              |
|           |                       | o |                  |              |
|           |                       | r |                  |              |
|           |                       | k |                  |              |
|           |                       | * |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | t |                  |              |
|           |                       | h |                  |              |
|           |                       | e |                  |              |
|           |                       | m |                  |              |
|           |                       | i |                  |              |
|           |                       | d |                  |              |
|           |                       | d |                  |              |
|           |                       | l |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | c |                  |              |
|           |                       | h |                  |              |
|           |                       | o |                  |              |
|           |                       | o |                  |              |
|           |                       | l |                  |              |
|           |                       | s |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | v |                  |              |
|           |                       | o |                  |              |
|           |                       | l |                  |              |
|           |                       | v |                  |              |
|           |                       | e |                  |              |
|           |                       | d |                  |              |
|           |                       | s |                  |              |
|           |                       | o |                  |              |
|           |                       | m |                  |              |
|           |                       | e |                  |              |
|           |                       | t |                  |              |
|           |                       | r |                  |              |
|           |                       | a |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | f |                  |              |
|           |                       | o |                  |              |
|           |                       | r |                  |              |
|           |                       | h |                  |              |
|           |                       | i |                  |              |
|           |                       | g |                  |              |
|           |                       | h |                  |              |
|           |                       | s |                  |              |
|           |                       | c |                  |              |
|           |                       | h |                  |              |
|           |                       | o |                  |              |
|           |                       | o |                  |              |
|           |                       | l |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | a |                  |              |
|           |                       | c |                  |              |
|           |                       | h |                  |              |
|           |                       | e |                  |              |
|           |                       | r |                  |              |
|           |                       | s |                  |              |
|           |                       | , |                  |              |
|           |                       | r |                  |              |
|           |                       | e |                  |              |
|           |                       | l |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | d |                  |              |
|           |                       | t |                  |              |
|           |                       | o |                  |              |
|           |                       | f |                  |              |
|           |                       | a |                  |              |
|           |                       | c |                  |              |
|           |                       | i |                  |              |
|           |                       | l |                  |              |
|           |                       | i |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | t |                  |              |
|           |                       | h |                  |              |
|           |                       | e |                  |              |
|           |                       | t |                  |              |
|           |                       | r |                  |              |
|           |                       | a |                  |              |
|           |                       | n |                  |              |
|           |                       | s |                  |              |
|           |                       | i |                  |              |
|           |                       | t |                  |              |
|           |                       | i |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | f |                  |              |
|           |                       | r |                  |              |
|           |                       | o |                  |              |
|           |                       | m |                  |              |
|           |                       | m |                  |              |
|           |                       | i |                  |              |
|           |                       | d |                  |              |
|           |                       | d |                  |              |
|           |                       | l |                  |              |
|           |                       | e |                  |              |
|           |                       | t |                  |              |
|           |                       | o |                  |              |
|           |                       | h |                  |              |
|           |                       | i |                  |              |
|           |                       | g |                  |              |
|           |                       | h |                  |              |
|           |                       | s |                  |              |
|           |                       | c |                  |              |
|           |                       | h |                  |              |
|           |                       | o |                  |              |
|           |                       | o |                  |              |
|           |                       | l |                  |              |
|           |                       | . |                  |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Professional          | T |                  | Professional |
| C**       | development focused   | e |                  | development  |
|           | on comprehensive      | a |                  | requirements |
| ***C      | planning in relation  | c |                  | were not     |
| o-NECT*** | to the SIP process,   | h |                  | tied to      |
|           | as well as use of     | e |                  | teacher      |
|           | student achievement   | r |                  | re-ce        |
|           | data for planning,    | s |                  | rtification, |
|           | technology            | h |                  | giving       |
|           | integration,          | a |                  | teachers     |
|           | project-based         | d |                  | less         |
|           | learning, and         | a |                  | incentive to |
|           | assessment.           | c |                  | participate. |
|           |                       | o |                  |              |
|           |                       | m |                  |              |
|           |                       | m |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | p |                  |              |
|           |                       | l |                  |              |
|           |                       | a |                  |              |
|           |                       | n |                  |              |
|           |                       | n |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | p |                  |              |
|           |                       | e |                  |              |
|           |                       | r |                  |              |
|           |                       | i |                  |              |
|           |                       | o |                  |              |
|           |                       | d |                  |              |
|           |                       | t |                  |              |
|           |                       | o |                  |              |
|           |                       | f |                  |              |
|           |                       | a |                  |              |
|           |                       | c |                  |              |
|           |                       | i |                  |              |
|           |                       | l |                  |              |
|           |                       | i |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | c |                  |              |
|           |                       | o |                  |              |
|           |                       | o |                  |              |
|           |                       | p |                  |              |
|           |                       | e |                  |              |
|           |                       | r |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | i |                  |              |
|           |                       | v |                  |              |
|           |                       | e |                  |              |
|           |                       | l |                  |              |
|           |                       | e |                  |              |
|           |                       | a |                  |              |
|           |                       | r |                  |              |
|           |                       | n |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | . |                  |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Virtually all         | A |                  | All other    |
| D**       | professional          | d |                  | schools in   |
|           | development was       | m |                  | the district |
| *         | linked to the reform  | i |                  | used         |
| **Success | method; the principal | n |                  | district-    |
| for       | estimated that 40     | i |                  | promoted     |
| All/Roots | hours of professional | s |                  | reading and  |
| &         | development           | t |                  | math         |
| Wings***  | activities were       | r |                  | curricula,   |
|           | offered per teacher,  | a |                  | and received |
|           | and that about 70     | t |                  | distri       |
|           | percent of staff      | o |                  | ct-sponsored |
|           | participated at this  | r |                  | professional |
|           | level.                | s |                  | development; |
|           |                       | a |                  | isolating    |
|           |                       | n |                  | school       |
|           |                       | d |                  | staff.       |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | f |                  |              |
|           |                       | f |                  |              |
|           |                       | m |                  |              |
|           |                       | e |                  |              |
|           |                       | e |                  |              |
|           |                       | t |                  |              |
|           |                       | b |                  |              |
|           |                       | i |                  |              |
|           |                       | - |                  |              |
|           |                       | m |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | t |                  |              |
|           |                       | h |                  |              |
|           |                       | l |                  |              |
|           |                       | y |                  |              |
|           |                       | t |                  |              |
|           |                       | o |                  |              |
|           |                       | w |                  |              |
|           |                       | o |                  |              |
|           |                       | r |                  |              |
|           |                       | k |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | c |                  |              |
|           |                       | u |                  |              |
|           |                       | r |                  |              |
|           |                       | r |                  |              |
|           |                       | i |                  |              |
|           |                       | c |                  |              |
|           |                       | u |                  |              |
|           |                       | l |                  |              |
|           |                       | a |                  |              |
|           |                       | r |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | r |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | g |                  |              |
|           |                       | i |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | ; |                  |              |
|           |                       | r |                  |              |
|           |                       | e |                  |              |
|           |                       | c |                  |              |
|           |                       | e |                  |              |
|           |                       | i |                  |              |
|           |                       | v |                  |              |
|           |                       | e |                  |              |
|           |                       | t |                  |              |
|           |                       | r |                  |              |
|           |                       | a |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | f |                  |              |
|           |                       | r |                  |              |
|           |                       | o |                  |              |
|           |                       | m |                  |              |
|           |                       | d |                  |              |
|           |                       | i |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | r |                  |              |
|           |                       | i |                  |              |
|           |                       | c |                  |              |
|           |                       | t |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | u |                  |              |
|           |                       | s |                  |              |
|           |                       | e |                  |              |
|           |                       | o |                  |              |
|           |                       | f |                  |              |
|           |                       | d |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | . |                  |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Formal professional   | B |                  | PD sessions  |
| E**       | development related   | e |                  | and staff    |
|           | to the method was     | g |                  | planning     |
| ***       | primarily available   | i |                  | time were    |
| Coalition | to the high school    | n |                  | being used   |
| of        | teachers during the   | n |                  | to address   |
| Essential | first year of the     | i |                  | ad           |
| S         | CSRD grant; little    | n |                  | ministrative |
| chools*** | occurred in later     | g |                  | issues, such |
|           | years.                | i |                  | as           |
|           |                       | n |                  | discipline.  |
|           |                       | 2 |                  |              |
|           |                       | 0 |                  |              |
|           |                       | 0 |                  |              |
|           |                       | 1 |                  |              |
|           |                       | - |                  |              |
|           |                       | 0 |                  |              |
|           |                       | 2 |                  |              |
|           |                       | , |                  |              |
|           |                       | t |                  |              |
|           |                       | h |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | c |                  |              |
|           |                       | h |                  |              |
|           |                       | o |                  |              |
|           |                       | o |                  |              |
|           |                       | l |                  |              |
|           |                       | c |                  |              |
|           |                       | l |                  |              |
|           |                       | o |                  |              |
|           |                       | s |                  |              |
|           |                       | e |                  |              |
|           |                       | d |                  |              |
|           |                       | e |                  |              |
|           |                       | a |                  |              |
|           |                       | r |                  |              |
|           |                       | l |                  |              |
|           |                       | y |                  |              |
|           |                       | t |                  |              |
|           |                       | w |                  |              |
|           |                       | o |                  |              |
|           |                       | d |                  |              |
|           |                       | a |                  |              |
|           |                       | y |                  |              |
|           |                       | s |                  |              |
|           |                       | a |                  |              |
|           |                       | m |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | t |                  |              |
|           |                       | h |                  |              |
|           |                       | t |                  |              |
|           |                       | o |                  |              |
|           |                       | a |                  |              |
|           |                       | l |                  |              |
|           |                       | l |                  |              |
|           |                       | o |                  |              |
|           |                       | w |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | f |                  |              |
|           |                       | f |                  |              |
|           |                       | t |                  |              |
|           |                       | o |                  |              |
|           |                       | a |                  |              |
|           |                       | d |                  |              |
|           |                       | d |                  |              |
|           |                       | r |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | s |                  |              |
|           |                       | s |                  |              |
|           |                       | c |                  |              |
|           |                       | h |                  |              |
|           |                       | o |                  |              |
|           |                       | o |                  |              |
|           |                       | l |                  |              |
|           |                       | w |                  |              |
|           |                       | i |                  |              |
|           |                       | d |                  |              |
|           |                       | e |                  |              |
|           |                       | c |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | c |                  |              |
|           |                       | e |                  |              |
|           |                       | r |                  |              |
|           |                       | n |                  |              |
|           |                       | s |                  |              |
|           |                       | . |                  |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Collaborative groups  | T |                  | Lack of      |
| F**       | and action teams were | r |                  | funding for  |
|           | the main PD-delivery  | a |                  | substitute   |
| ***       | method, and nearly    | i |                  | teachers in  |
| Coalition | all staff             | n |                  | 2001-02 may  |
| of        | participated by       | i |                  | mean groups  |
| Essential | 2001-02; staff took   | n |                  | can no       |
| S         | part in a week-long   | g |                  | longer meet  |
| chools*** | "CES change process"  | b |                  | during the   |
|           | course and in a local | y |                  | school day.  |
|           | HSTW network.         | d |                  |              |
|           |                       | i |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | r |                  |              |
|           |                       | i |                  |              |
|           |                       | c |                  |              |
|           |                       | t |                  |              |
|           |                       | c |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | s |                  |              |
|           |                       | u |                  |              |
|           |                       | l |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | n |                  |              |
|           |                       | t |                  |              |
|           |                       | s |                  |              |
|           |                       | h |                  |              |
|           |                       | a |                  |              |
|           |                       | s |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | c |                  |              |
|           |                       | l |                  |              |
|           |                       | u |                  |              |
|           |                       | d |                  |              |
|           |                       | e |                  |              |
|           |                       | d |                  |              |
|           |                       | : |                  |              |
|           |                       | u |                  |              |
|           |                       | s |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | e |                  |              |
|           |                       | l |                  |              |
|           |                       | e |                  |              |
|           |                       | c |                  |              |
|           |                       | t |                  |              |
|           |                       | r |                  |              |
|           |                       | o |                  |              |
|           |                       | n |                  |              |
|           |                       | i |                  |              |
|           |                       | c |                  |              |
|           |                       | p |                  |              |
|           |                       | o |                  |              |
|           |                       | r |                  |              |
|           |                       | t |                  |              |
|           |                       | f |                  |              |
|           |                       | o |                  |              |
|           |                       | l |                  |              |
|           |                       | i |                  |              |
|           |                       | o |                  |              |
|           |                       | s |                  |              |
|           |                       | , |                  |              |
|           |                       | u |                  |              |
|           |                       | s |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | d |                  |              |
|           |                       | e |                  |              |
|           |                       | p |                  |              |
|           |                       | e |                  |              |
|           |                       | n |                  |              |
|           |                       | d |                  |              |
|           |                       | e |                  |              |
|           |                       | n |                  |              |
|           |                       | t |                  |              |
|           |                       | t |                  |              |
|           |                       | h |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | k |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | , |                  |              |
|           |                       | d |                  |              |
|           |                       | i |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | n |                  |              |
|           |                       | c |                  |              |
|           |                       | e |                  |              |
|           |                       | l |                  |              |
|           |                       | e |                  |              |
|           |                       | a |                  |              |
|           |                       | r |                  |              |
|           |                       | n |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | , |                  |              |
|           |                       | a |                  |              |
|           |                       | n |                  |              |
|           |                       | d |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | a |                  |              |
|           |                       | n |                  |              |
|           |                       | d |                  |              |
|           |                       | a |                  |              |
|           |                       | r |                  |              |
|           |                       | d |                  |              |
|           |                       | s |                  |              |
|           |                       | / |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | i |                  |              |
|           |                       | n |                  |              |
|           |                       | g |                  |              |
|           |                       | . |                  |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Nearly all staff      | T |                  |              |
| G**       | participate in        | w |                  |              |
|           | method-related PD     | i |                  |              |
| **        | that is related to    | c |                  |              |
| *"Special | comprehensive reform  | e |                  |              |
| Literacy  | and is results-based. | m |                  |              |
| Mo        |                       | o |                  |              |
| del"**\** |                       | n |                  |              |
|           |                       | t |                  |              |
|           |                       | h |                  |              |
|           |                       | l |                  |              |
|           |                       | y |                  |              |
|           |                       | P |                  |              |
|           |                       | D |                  |              |
|           |                       | r |                  |              |
|           |                       | e |                  |              |
|           |                       | l |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | d |                  |              |
|           |                       | t |                  |              |
|           |                       | o |                  |              |
|           |                       | m |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | h |                  |              |
|           |                       | i |                  |              |
|           |                       | s |                  |              |
|           |                       | p |                  |              |
|           |                       | r |                  |              |
|           |                       | e |                  |              |
|           |                       | s |                  |              |
|           |                       | e |                  |              |
|           |                       | n |                  |              |
|           |                       | t |                  |              |
|           |                       | e |                  |              |
|           |                       | d |                  |              |
|           |                       | b |                  |              |
|           |                       | y |                  |              |
|           |                       | a |                  |              |
|           |                       | m |                  |              |
|           |                       | a |                  |              |
|           |                       | t |                  |              |
|           |                       | h |                  |              |
|           |                       | s |                  |              |
|           |                       | p |                  |              |
|           |                       | e |                  |              |
|           |                       | c |                  |              |
|           |                       | i |                  |              |
|           |                       | a |                  |              |
|           |                       | l |                  |              |
|           |                       | i |                  |              |
|           |                       | s |                  |              |
|           |                       | t |                  |              |
|           |                       | . |                  |              |
+-----------+-----------------------+---+------------------+--------------+
| **\*      |                       |   |                  |              |
| **Locally |                       |   |                  |              |
| developed |                       |   |                  |              |
| methods   |                       |   |                  |              |
| are in    |                       |   |                  |              |
| quotes    |                       |   |                  |              |
| and have  |                       |   |                  |              |
| been      |                       |   |                  |              |
| given     |                       |   |                  |              |
| ps        |                       |   |                  |              |
| eudonyms. |                       |   |                  |              |
|           |                       |   |                  |              |
| **(       |                       |   |                  |              |
| Continued |                       |   |                  |              |
| on next   |                       |   |                  |              |
| page)**   |                       |   |                  |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Until 2001-02, school |   | The district     |              |
| H**       | staff received        |   | provided         |              |
|           | method-related PD     |   | tra              |              |
| *         | twice per year from   |   | ining-of-trainer |              |
| **Success | developer, and on an  |   | PD on content    |              |
| for       | ongoing basis from    |   | standards,       |              |
| All***    | campus instructional  |   | access to        |              |
|           | coordinator (through  |   | resources, and   |              |
|           | modeling,             |   | student          |              |
|           | observations, and     |   | expectations.    |              |
|           | grade-level           |   | When the school  |              |
|           | planning).            |   | stopped          |              |
|           |                       |   | implementing SFA |              |
|           |                       |   | in 2001-02, the  |              |
|           |                       |   | district also    |              |
|           |                       |   | provided         |              |
|           |                       |   | training on the  |              |
|           |                       |   | reading          |              |
|           |                       |   | curriculum.      |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Earlier training in   |   | School provided  | District     |
| I**       | the method was        |   | staff with       | restricts    |
|           | limited, but in       |   | collaborative    | early school |
| ***Lig    | 2001-02, training by  |   | time during      | dismissal    |
| htspan*** | in-house staff and    |   | 2001-02,         | and teacher  |
|           | accessibility to      |   | allowing staff   | planning     |
|           | materials better      |   | to meet by grade | days.        |
|           | enabled 1st grade     |   | level, examine   |              |
|           | teachers to begin     |   | student          |              |
|           | implementation.       |   | achievement      |              |
|           |                       |   | scores, develop  |              |
|           |                       |   | ideas and share  |              |
|           |                       |   | plans.           |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | All but three         |   | Teachers had to  |              |
| J**       | teachers received PD  |   | complete 2       |              |
|           | related to the        |   | half-day         |              |
| ***"Pe    | method, including     |   | training         |              |
| rformance | training sessions on  |   | sessions to meet |              |
| Tasks     | active learning, self |   | state's          |              |
| Mo        | and peer assessment,  |   | continuing       |              |
| del"**\** | cooperative planning, |   | education        |              |
|           | and alignment with    |   | requirement;     |              |
|           | state standards.      |   | district         |              |
|           |                       |   | provided PD on   |              |
|           |                       |   | understanding    |              |
|           |                       |   | and using        |              |
|           |                       |   | student          |              |
|           |                       |   | achievement      |              |
|           |                       |   | data.            |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Extensive PD offered  |   | District         |              |
| K**       | around method,        |   | required         |              |
|           | including five-day    |   | teachers to      |              |
| *         | introductory          |   | complete 2       |              |
| **Success | workshop, as well as  |   | half-day         |              |
| for       | opportunities for     |   | training         |              |
| All***    | training of tutors    |   | sessions to meet |              |
|           | and the Family        |   | state's          |              |
|           | Support Team.         |   | continuing       |              |
|           |                       |   | education        |              |
|           |                       |   | requirement;     |              |
|           |                       |   | district         |              |
|           |                       |   | provided PD      |              |
|           |                       |   | related to       |              |
|           |                       |   | understanding    |              |
|           |                       |   | and using        |              |
|           |                       |   | student          |              |
|           |                       |   | achievement      |              |
|           |                       |   | data.            |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Staff received        |   | Teachers met     | Lack of      |
| L**       | training in the       |   | bi-weekly (once  | im           |
|           | method through        |   | on staff issues; | plementation |
| ***       | in-service days at    |   | once on          | of           |
| "Behavior | the start of the      |   | curricular       | i            |
| Mod       | year, annual          |   | topics);         | nstructional |
| ification | retreats, and ongoing |   | district         | portions of  |
| Mo        | modeling and          |   | required 18      | the method   |
| del"**\** | observations by the   |   | hours of PD      | indicated    |
|           | lead teacher.         |   | activities, and  | inadequate   |
|           |                       |   | provided PD on   | PD on this   |
|           |                       |   | the math and     | component.   |
|           |                       |   | reading          |              |
|           |                       |   | curricula.       |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Activities were       |   | The district     |              |
| M**       | primarily focused on  |   | provided between |              |
|           | grade 3--5 teachers   |   | three and four   |              |
| ***Lig    | who used Lightspan;   |   | PD days a year   |              |
| htspan*** | one presentation was  |   | through a        |              |
|           | made to all teachers  |   | catalog of staff |              |
|           | in the school. Four   |   | development      |              |
|           | Lightspan training    |   | opportunities;   |              |
|           | events were provided  |   | there was no     |              |
|           | during 2000--01; no   |   | coordination to  |              |
|           | training was provided |   | ensure staff     |              |
|           | in 2001-02.           |   | selected PD      |              |
|           |                       |   | sessions related |              |
|           |                       |   | to the school's  |              |
|           |                       |   | reform efforts.  |              |
+-----------+-----------------------+---+------------------+--------------+
| **\*      |                       |   |                  |              |
| **Locally |                       |   |                  |              |
| developed |                       |   |                  |              |
| methods   |                       |   |                  |              |
| are in    |                       |   |                  |              |
| quotes    |                       |   |                  |              |
| and have  |                       |   |                  |              |
| been      |                       |   |                  |              |
| given     |                       |   |                  |              |
| ps        |                       |   |                  |              |
| eudonyms. |                       |   |                  |              |
|           |                       |   |                  |              |
| **(       |                       |   |                  |              |
| Continued |                       |   |                  |              |
| on next   |                       |   |                  |              |
| page)**   |                       |   |                  |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Method-related PD     |   | The district     | District     |
| N**       | received twice per    |   | provided         | provided PD  |
|           | year from developer,  |   | tra              | related to   |
| *         | and on an ongoing     |   | ining-of-trainer | the          |
| **Success | basis from campus     |   | PD on content    | distri       |
| for       | instructional         |   | standards,       | ct-supported |
| All***    | coordinator (through  |   | access to        | reading      |
|           | modeling,             |   | resources, and   | curriculum,  |
|           | observations, and     |   | student          | which School |
|           | grade-level planning) |   | expectations.    | N was not    |
|           |                       |   |                  | using.       |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Teachers participated |   | District         |              |
| O**       | in monthly activities |   | provided PD on   |              |
|           | related to the        |   | curriculum       |              |
| ***Succes | method; CSRD funding  |   | goals,           |              |
| s-in-the- | allowed inclusion of  |   | co-teaching      |              |
| Making*** | kindergarten and      |   | methods, using   |              |
|           | Grade 5 teachers;     |   | the SFA          |              |
|           | training included how |   | curricula,       |              |
|           | to incorporate the    |   | instructional    |              |
|           | method into the       |   | strategies in    |              |
|           | classroom and how to  |   | reading and      |              |
|           | use data.             |   | writing, and     |              |
|           |                       |   | magnet school    |              |
|           |                       |   | curriculum.      |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | All teachers had a    |   | Staff            |              |
| P**       | minimum of 6 PD days  |   | participated in  |              |
|           | from HSTW framework;  |   | PD related to    |              |
| ***High   | 75% of teachers opted |   | vocational       |              |
| Schools   | to participate in     |   | education, as    |              |
| that      | more activities.      |   | well as PD       |              |
| Work***   |                       |   | related to the   |              |
|           |                       |   | transition       |              |
|           |                       |   | students from    |              |
|           |                       |   | middle to high   |              |
|           |                       |   | school through   |              |
|           |                       |   | *Making Schools  |              |
|           |                       |   | Work* (supported |              |
|           |                       |   | by a 2000 CSRD   |              |
|           |                       |   | grant to the     |              |
|           |                       |   | district middle  |              |
|           |                       |   | schools.         |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | Each staff member     |   | The school       | District     |
| Q**       | received an average   |   | received         | placed       |
|           | of 6 PD days; 90% was |   | training on      | restrictions |
| ***Ac     | reform-related.       |   | teaching         | on early     |
| celerated | School used a         |   | children of      | dismissals   |
| S         | training-of-trainers  |   | poverty.         | and teacher  |
| chools*** | approach; topics      |   | Incentives for   | w            |
|           | covered the specific  |   | faculty          | ork/planning |
|           | instructional and     |   | participation in | days.        |
|           | pedagogical practices |   | PD included      |              |
|           | consistent with AS.   |   | tuition/expense  |              |
|           |                       |   | reimbursement    |              |
|           |                       |   | and continuing   |              |
|           |                       |   | education units. |              |
+-----------+-----------------------+---+------------------+--------------+
| **School  | CSRD funds were used  |   | The school's     | Budget       |
| R**       | in the Year-1 for     |   | focus was on     | cutbacks     |
|           | method-related PD,    |   | aligning         | have greatly |
| ***"Comp  | including funds for   |   | teachers'        | reduced the  |
| rehensive | technology-focused    |   | individual PD    | amount of    |
| Reform    | training; funds were  |   | plans with       | professional |
| Mo        | devoted to PD         |   | school and       | o            |
| del"**\** | activities in 2001-02 |   | district plans;  | pportunities |
|           | for both the          |   | the district had | that the     |
|           | development of small  |   | a catalog of     | district can |
|           | learning communities  |   | course offerings | make         |
|           | and freshman teams.   |   | available to     | available to |
|           |                       |   | school staff.    | schools.     |
+-----------+-----------------------+---+------------------+--------------+
| \*Locally |                       |   |                  |              |
| developed |                       |   |                  |              |
| methods   |                       |   |                  |              |
| are in    |                       |   |                  |              |
| quotes    |                       |   |                  |              |
| and have  |                       |   |                  |              |
| been      |                       |   |                  |              |
| given     |                       |   |                  |              |
| ps        |                       |   |                  |              |
| eudonyms. |                       |   |                  |              |
+-----------+-----------------------+---+------------------+--------------+

The first column of Exhibit 2-6 shows that, except in the case of three
schools (Schools E, M, and R), the schools all have been participating
in extensive professional development, mainly led by the external
method's developer. To their credit, the developers have been able to
provide a broad and intense variety of assistance, including: 1)
participation in national conferences and workshops; 2) convening of
local workshops for the school's faculty; 3) organizing of annual
retreats for the faculty; 4) convening of all-staff meetings; 5)
specific training to individual teachers; and 6) periodic and in some
cases frequent on-site support and problem-solving. \[As a promising
reform strategy, see "Professional Development Activities and
Instructional Practice Driven by Student Assessment Measures," in
Appendix B.)

The second column of Exhibit 2-6 shows that, although the field teams
were to inquire about professional development not related to the
research-based method, not a lot of such professional development was
occurring. In 15 of the 18 cases, the broader professional development
was: 1) being provided by the district, 2) embedded in the job
structure, or 3) informal. \[As a promising reform strategy, "Staff
Utilizes Common Planning Period as Professional Development Opportunity"
in Appendix B.\] In 2 of those 15 cases, as well as in the remaining 3
schools, staff members participated in professional development related
to the schools' adoption of a second research-based method.

Directly corresponding to these observations about professional
development, nearly all of the 18 schools: a) had external technical
support or assistance, and b) derived such assistance from the developer
of the research-based method (see Exhibit 2-7).

The intensity and breadth of the assistance in part depended upon the
amount of CSRD funds allocated for such assistance by the school.
Although precise budget data were not always available, the site visit
team observed that in most cases the bulk of schools' CSRD funds were
used to support external assistance from the developer.[^32] As a
result, schools that adopted research-based methods requiring ongoing
annual outlays (compared to outlays for startup years only) were
especially likely to need to identify new sources of funds, in order to
continue comprehensive school reform activities beyond the period of
CSRD funding.

Exhibit 2-7 also shows the diversity of the external assistance and
support activities, ranging from participation in national meetings to
teacher workshops to individualized (within classroom) on-site support.
In reviewing these data, it is important to note that the site visit
protocol was not limited to external assistance by the method's
developer. Rather, to cover CSRD Component 7, the protocol guided the
site visit teams to inquire about *any* *type* of external assistance.
As a result, in many cases the teams did find such assistance

**Exhibit 2-7**

**EXTERNAL TECHNICAL SUPPORT AND ASSISTANCE**

**(Component 7)**

+--------------+-----------------------------+------------------------+
| **School     | **Types and Frequency of    | **Types and Frequency  |
| Name/**      | Method-Related External     | of**                   |
|              | Support and Assistance**    |                        |
| **           |                             | **Other                |
| *Method***\* |                             | Reform-Related**       |
|              |                             |                        |
|              |                             | **External Support and |
|              |                             | Assistance**           |
+==============+=============================+========================+
| **School A** | School received extensive   | Consultants provided   |
|              | on-site support from the    | support in reading and |
| **           | developer since formal      | writing instruction    |
| *Accelerated | adoption of the method; by  | and classroom and time |
| Schools***   | Spring 2002, the principal  | management.            |
|              | was concerned that her      |                        |
|              | staff had training needs    |                        |
|              | beyond the capacity of the  |                        |
|              | national center.            |                        |
+--------------+-----------------------------+------------------------+
| **School B** | Developer conducted two     | During the final year  |
|              | site visits per year, as    | of CSRD funding, staff |
| ***High      | well as onsite workshops,   | received training in   |
| Schools that | sponsored visits by schools | Making Schools Work,   |
| Work***      | staff to high achieving     | an initiative to       |
|              | schools, hosted an annual   | better prepare         |
|              | conference, and conducted   | students for the       |
|              | monthly telephone           | transition to high     |
|              | conferences with mentor     | school.                |
|              | site leaders.               |                        |
+--------------+-----------------------------+------------------------+
| **School C** | Onsite Co-NECT consultant   | The school was         |
|              | serviced School C and five  | implementing a number  |
| *            | others, was available       | of other initiatives   |
| **Co-NECT*** | on-site almost weekly,      | (particularly reading  |
|              | taught classes (modeling)   | programs), for which   |
|              | and made presentation to    | staff received         |
|              | all-school assemblies.      | training.              |
|              | Teams from other Co-NECT    |                        |
|              | schools conducted annual    |                        |
|              | site visits. Developer-     |                        |
|              | supported "Exchange"        |                        |
|              | website as resource for     |                        |
|              | school staff.               |                        |
+--------------+-----------------------------+------------------------+
| **School D** | SFA Foundation provided     | External assistance    |
|              | extensive training for both | provided by a local    |
| ***Success   | the reading and math        | consultant hired by    |
| for          | curricula in the first year | the district to        |
| All/Roots &  | of implementation.          | support the district's |
| Wings***     | Refresher training sessions | CSRD schools. She      |
|              | took place annually, along  | helped the school      |
|              | with visits to assess the   | identify technical     |
|              | level and fidelity of       | assistance providers   |
|              | implementation (twice per   | and develop classroom  |
|              | year).                      | planning guides, and   |
|              |                             | was available during   |
|              |                             | monthly phone-calls    |
|              |                             | with the principal.    |
+--------------+-----------------------------+------------------------+
| **School E** | After the first year of     | No other external      |
|              | funding, the school had     | support identified.    |
| ***Coalition | limited contact with the    |                        |
| of Essential | method developer (following |                        |
| Schools***   | 1998-99, the school was not |                        |
|              | an official CES site and    |                        |
|              | did not pay annual fees to  |                        |
|              | any CES organization.       |                        |
+--------------+-----------------------------+------------------------+
| **School F** | Week-long training by CES   | Extensive training     |
|              | designed to perform a       | from Annenberg and NSF |
| ***Coalition | year-long activity for      | on using collaborative |
| of Essential | visualizing and             | inquiry and action     |
| Schools***   | implementing plans for      | research; HSTW network |
|              | change.                     | meetings every two     |
|              |                             | months to share ideas. |
+--------------+-----------------------------+------------------------+
| **School G** | The developer conducted a   | The developer also     |
|              | summer institute and        | worked with staff to   |
| ***"Special  | graduate course work, and   | design and develop     |
| Literacy     | was on-site twice per year  | professional           |
| Model"**\**  | for in-service sessions. A  | development workshops  |
|              | regional project liaison    | based on a local needs |
|              | for the developer was on    | assessment process.    |
|              | site twice per month,       |                        |
|              | working with teachers and   |                        |
|              | providing in-class          |                        |
|              | modeling.                   |                        |
+--------------+-----------------------------+------------------------+
| \*Locally    |                             |                        |
| developed    |                             |                        |
| methods are  |                             |                        |
| in quotes    |                             |                        |
| and have     |                             |                        |
| been given   |                             |                        |
| pseudonyms.  |                             |                        |
|              |                             |                        |
| **(Continued |                             |                        |
| on next      |                             |                        |
| page)**      |                             |                        |
+--------------+-----------------------------+------------------------+
| **School H** | Until 2000-02, the method   | A local university     |
|              | developer visited twice a   | provided student       |
| ***Success   | year and provided training  | teachers who supported |
| for All***   | and TA on SFA               | reform by tutoring     |
|              | implementation. The school  | students.              |
|              | did not complete a contract |                        |
|              | with SFA for 2001-02 when   |                        |
|              | staff voted to shift to the |                        |
|              | district-endorsed reading   |                        |
|              | curriculum.                 |                        |
+--------------+-----------------------------+------------------------+
| **School I** | During the first two years  | Due to state budget    |
|              | of CSRD funding, the        | cuts, the district     |
| ***          | developer conducted one     | sought assistance from |
| Lightspan*** | annual site visit to        | its regional service   |
|              | observe classes and provide | center.                |
|              | demonstrations of how       |                        |
|              | Lightspan could be used in  |                        |
|              | the classroom. In 2001-02,  |                        |
|              | the school discontinued its |                        |
|              | relationship with the       |                        |
|              | developer and relied on an  |                        |
|              | internal staff person to    |                        |
|              | conduct staff training.     |                        |
+--------------+-----------------------------+------------------------+
| **School J** | The school worked with the  | School received        |
|              | developer during Year 1 who | assistance from        |
| ***          | provided initial training.  | reading consultants,   |
| "Performance | In years-2 and -3, the      | and district           |
| Tasks        | school worked with an       | assessment staff, as   |
| Model"**\**  | external consultant for the | well as "critical      |
|              | developer, who provided     | friend" mentoring      |
|              | training focused on the     | support (sponsored by  |
|              | development of new tasks.   | the state for schools  |
|              |                             | rated as               |
|              |                             | in                     |
|              |                             | -need-of-improvement). |
+--------------+-----------------------------+------------------------+
| **School K** | The developer provided      | No additional training |
|              | training in the method for  | from external          |
| ***SFA***    | all staff, as well as for   | providers was          |
|              | members of family support   | identified.            |
|              | team. Staff complained that |                        |
|              | the developer did not       |                        |
|              | respond to concerns about   |                        |
|              | appropriateness SFA for     |                        |
|              | students in intermediate    |                        |
|              | grades, which may impact    |                        |
|              | sustainability of the       |                        |
|              | method at this school.      |                        |
+--------------+-----------------------------+------------------------+
| **School L** | Developer provided          | No additional training |
|              | consistent and strong       | from external          |
| ***"Behavior | assistance since the        | providers was          |
| Modification | inception of the program,   | identified.            |
| Model"**\**  | including helping the       |                        |
|              | school select and customize |                        |
|              | the reform method,          |                        |
|              | conducting introductory and |                        |
|              | follow-up training for the  |                        |
|              | method, visiting            |                        |
|              | classrooms, serving as      |                        |
|              | facilitator at staff        |                        |
|              | retreats, and generally     |                        |
|              | assisting as requested. The |                        |
|              | developer also hosted       |                        |
|              | bi-annual networking        |                        |
|              | dinners for local           |                        |
|              | principals.                 |                        |
+--------------+-----------------------------+------------------------+
| **School M** | In 1999--00, the developer  | No additional training |
|              | provided support at the     | from external          |
| ***          | beginning of the school     | providers was          |
| Lightspan*** | year. In 2000--01, the      | identified.            |
|              | school expanded the         |                        |
|              | contract to include monthly |                        |
|              | visits to assist in         |                        |
|              | implementation. In 2001-02, |                        |
|              | the new administrators at   |                        |
|              | the school decided not to   |                        |
|              | renew its contract with the |                        |
|              | external developer. The     |                        |
|              | school relied on internal   |                        |
|              | staff to train new staff    |                        |
|              | members.                    |                        |
+--------------+-----------------------------+------------------------+
| \*Locally    |                             |                        |
| developed    |                             |                        |
| methods are  |                             |                        |
| in quotes    |                             |                        |
| and have     |                             |                        |
| been given   |                             |                        |
| pseudonyms.  |                             |                        |
|              |                             |                        |
| **(Continued |                             |                        |
| on next      |                             |                        |
| page)**      |                             |                        |
+--------------+-----------------------------+------------------------+
| **School N** | The developer conducted     | School also received   |
|              | implementation visits to    | training in use of     |
| ***SFA***    | the schools twice annually. | other reading          |
|              | On-site support included    | initiatives            |
|              | training in use of SFA,     | (Accelerated Reader    |
|              | conducting classroom        | and state-created      |
|              | observations, and           | program).              |
|              | customizing training to     |                        |
|              | meet district- and          |                        |
|              | school-specific needs (such |                        |
|              | as the alignment between    |                        |
|              | state standards, the state  |                        |
|              | assessment, and the         |                        |
|              | method). The school reduced |                        |
|              | the number of visits in     |                        |
|              | 2001-02, as CSRD funding    |                        |
|              | dwindled.                   |                        |
+--------------+-----------------------------+------------------------+
| **School O** | The developer provided      | No additional training |
|              | support (ranging from 0.5   | from external          |
| ***SIM***    | to 3 days each month) on    | providers was          |
|              | technology planning and     | identified.            |
|              | grade level meetings, made  |                        |
|              | presentations on the method |                        |
|              | during education            |                        |
|              | conferences, and assisted   |                        |
|              | in data collection review   |                        |
|              | and analysis.               |                        |
+--------------+-----------------------------+------------------------+
| **School P** | The developer provided two  | During the final year  |
|              | technical review visits per | of CSRD funding, staff |
| ***HSTW***   | year, site specific         | received training in   |
|              | development activities      | Making Schools Work,   |
|              | (around the method), visits | an initiative to       |
|              | to high achieving schools,  | better prepare         |
|              | monthly teleconferences     | students for the       |
|              | with site leaders, and      | transition to high     |
|              | assessment and analysis of  | school.                |
|              | student achievement data.   |                        |
+--------------+-----------------------------+------------------------+
| **School Q** | The developer provided      | No additional training |
|              | training to school          | from external          |
| **           | leadership and staff        | providers was          |
| *Accelerated | on-site, conducted          | identified.            |
| Schools***   | technical assistance and    |                        |
|              | on-site monitoring every    |                        |
|              | six weeks, and developed an |                        |
|              | evaluation plan. In 2001-02 |                        |
|              | (the school's final year of |                        |
|              | CSRD funding), the          |                        |
|              | developer spent less time   |                        |
|              | at the school, and          |                        |
|              | encouraged staff to develop |                        |
|              | internal capacity to        |                        |
|              | support the method. Staff   |                        |
|              | reported more assistance in |                        |
|              | Year 3 was needed,          |                        |
|              | particularly for new        |                        |
|              | teachers.                   |                        |
+--------------+-----------------------------+------------------------+
| **School R** | The method was so           | No other external      |
|              | customized that no single   | support identified.    |
| ***"C        | "developer" or external     |                        |
| omprehensive | technical assistance team   |                        |
| Reform       | was relevant.               |                        |
| Model"**\**  |                             |                        |
+--------------+-----------------------------+------------------------+
| \*Locally    |                             |                        |
| developed    |                             |                        |
| methods are  |                             |                        |
| in quotes    |                             |                        |
| and have     |                             |                        |
| been given   |                             |                        |
| pseudonyms.  |                             |                        |
+--------------+-----------------------------+------------------------+

being provided by parties other than the developer (Schools A, B, C, D,
F, G, H, J, N, and P). However, within this subset, only rarely did the
assistance address comprehensive school reform independent of the
method.

The field teams concluded that perhaps the previously observed lower
proportion of awareness and design for comprehensive
reform---independent of the research-based method that a school has
adopted---was in part because few consultants were available to provide
this type of assistance,[^33] and further, because schools have not
understood the need to seek such assistance.

***2.3.2 Both measurable goals and benchmarks (Component 4) and
evaluation strategies (Component 8) have been devoted to tracking
student performance, not necessarily implementation progress***. Having
measurable goals and benchmarks (Component 4) enables a school to
monitor the progress of a school's CSRD initiative. The actualization of
any monitoring or evaluation activity, however, requires evaluation
strategies (Component 8) and planning. For this reason the two topics
are covered together.

For measurable "goals," all but one of the 18 schools (School E, which
became a new charter school in 2001-02) identified detailed student
performance goals (see Exhibit 2-8).[^34] However, rather than
considering "benchmarks" as markers of implementation progress or other
hypothetical steps (e.g., increased enrollment, hours spent, or time on
task) needed to achieve the goals, many of the schools simply defined
"benchmarks" by providing yearly breakdowns of the incremental
performance gains that would extrapolate to meeting the overall
performance goals. \[As a promising reform strategy, see
"Method-Recommended Curriculum Team Aligning Scope and Sequence to
Provide Greater Opportunities for Interdisciplinary Projects," in
Appendix B.\] The second column of Exhibit 2-8 shows a small group of
schools (Schools B, C, G, J, O, and Q) that had defined and were
actually monitoring implementation benchmarks, apart from any monitoring
being done by the external developer. (As a promising reform strategy,
see "External 'Critical Friends' Group Provides Objective Assessment of
Progress Towards Goals and Benchmarks," in Appendix B.\] For instance,
the original language of Component 4 does focus on student performance
and is readily reinforced by the attention that has been devoted to
student performance in judging school performance.

**Exhibit 2-8**

**GOALS AND BENCHMARKS (Component 4) AND**

**EVALUATION STRATEGIES (Component 8)**

+---------+-------------+----------------+------------+---------------+
| *       | **B         |                | **Ev       |               |
| *School | enchmarks** |                | aluation** |               |
| Name/** |             |                |            |               |
|         |             |                |            |               |
| ***Meth |             |                |            |               |
| od*\*** |             |                |            |               |
+=========+=============+================+============+===============+
|         | **Student   | **Im           | **Plan**   | *             |
|         | Ac          | plementation** |            | *Activities** |
|         | hievement** |                |            |               |
+---------+-------------+----------------+------------+---------------+
| *       | Starting in | Accelerated    | No formal  | No written    |
| *School | 2001-02,    | Schools        | plan;      | findings from |
| A**     | school      | Assessment     | school     | cadres, who   |
|         | developed   | Toolkit        | used AS    | analyzed      |
| ***Acce | benchmarks  | allowed school | school     | student       |
| lerated | to meet     | to develop     | evaluation | achievement   |
| Sch     | state       | benchmarks;    | ques       | data, and     |
| ools*** | proficiency | process had    | tionnaire; | developed     |
|         | levels by   | not begun      | other      | plans to      |
|         | grade and   |                | plans      | address       |
|         | subject     |                | developed  | deficiencies  |
|         |             |                | by         |               |
|         |             |                | individual |               |
|         |             |                | cadres     |               |
+---------+-------------+----------------+------------+---------------+
| *       | Included    | Leadership     | State      | Formal        |
| *School | extensive   | team developed | conducting | instrument    |
| B**     | coverage of | process        | a formal   | used to       |
|         | student     | benchmarks and | evaluation | evaluate      |
| ***High | performance | indicators and | of CSRD    | instruction   |
| Schools | on          | reviewed       | impl       | during        |
| That    | assessments | progress       | ementation | classroom     |
| Work*** | and toward  | annually       |            | observations  |
|         | graduation  |                |            |               |
|         | r           |                |            |               |
|         | equirements |                |            |               |
+---------+-------------+----------------+------------+---------------+
| *       | B           | Co-NECT        | No formal  | Conduct of    |
| *School | enchmarking | benchmarks     | external   | parent,       |
| C**     | plans       | included       | e          | teacher,      |
|         | pointed to  | shared         | valuation; | principal,    |
| ***Co-  | performance | a              | self-      | and student   |
| NECT*** | on state    | ccountability, | assessment | surveys;      |
|         | mandated    | project-based  | for SIP    | review of     |
|         | tests       | learning,      | process;   | achievement   |
|         |             | comprehensive  | and        | scores;       |
|         |             | assessment,    | CO-NECT    | conduct of    |
|         |             | team-based     | processes  | focus groups; |
|         |             | school         | to measure | reports from  |
|         |             | organization,  | impl       | external      |
|         |             | and use of     | ementation | technical     |
|         |             | technology     |            | assistance    |
|         |             |                |            | providers     |
+---------+-------------+----------------+------------+---------------+
| *       | School      | Implementation | District   | End-of-year   |
| *School | expected to | and fidelity   | contracted | report        |
| D**     | reach       | reviews        | with a     | contained     |
|         | benchmarks  | conducted by   | consultant | findings      |
| ***     | established | external       | to conduct | mandated by   |
| Success | by the      | developer      | e          | district; SFA |
| for     | district    |                | valuation, | made          |
| All***  | targeting   |                | but turned | bi-annual     |
|         | student     |                | resp       | visits to the |
|         | performance |                | onsibility | school to     |
|         | (reading    |                | over to    | monitor       |
|         | readiness,  |                | schools in | im            |
|         | attendance, |                | 2001-02    | plementation; |
|         | reading and |                |            | no evidence   |
|         | math        |                |            | of formal     |
|         | p           |                |            | evaluation    |
|         | roficiency) |                |            | report        |
+---------+-------------+----------------+------------+---------------+
| *       | In 2001-02, | No             | No recent  | Prior         |
| *School | the         | implementation | contact    | activities of |
| E**     | benchmarks  | benchmarks     | with       | an external   |
|         | and goals   |                | external   | evaluator     |
| ***Co   | were        |                | evaluator  | were lost     |
| alition | unclear,    |                |            | during the    |
| of      | after the   |                |            | school's      |
| Es      | school      |                |            | transition;   |
| sential | r           |                |            | no activities |
| Sch     | estructured |                |            | were planned  |
| ools*** | as a        |                |            | for 2001-02   |
|         | charter     |                |            |               |
|         | school      |                |            |               |
+---------+-------------+----------------+------------+---------------+
| *       | State       | School met     | District   | School        |
| *School | establishes | first set of   | intends to | completed     |
| F**     | benchmarks  | implementation | conduct an | surveys;      |
|         | for student | benchmarks,    | impact     | district      |
| ***Co   | achievement | related to     | assessment | analysis of   |
| alition | scores      | participation  | (using     | data in       |
| of      | i           | in             | survey and | progress; no  |
| Es      | ncorporated | collaborative  | student    | written       |
| sential | annually in | groups and     | a          | finding       |
| Sch     | SIP         | development of | chievement | available     |
| ools*** |             | academies, but | data); no  |               |
|         |             | did not        | evidence   |               |
|         |             | establish new  | of written |               |
|         |             | benchmarks     | plan       |               |
|         |             | pertaining to  |            |               |
|         |             | full reform    |            |               |
+---------+-------------+----------------+------------+---------------+
| \*      |             |                |            |               |
| Locally |             |                |            |               |
| de      |             |                |            |               |
| veloped |             |                |            |               |
| methods |             |                |            |               |
| are in  |             |                |            |               |
| quotes  |             |                |            |               |
| and     |             |                |            |               |
| have    |             |                |            |               |
| been    |             |                |            |               |
| given   |             |                |            |               |
| pseu    |             |                |            |               |
| donyms. |             |                |            |               |
|         |             |                |            |               |
| **(Co   |             |                |            |               |
| ntinued |             |                |            |               |
| on next |             |                |            |               |
| page)** |             |                |            |               |
+---------+-------------+----------------+------------+---------------+
| *       | Benchmarks  | School         | No         | Conduct of    |
| *School | follow      | established    | external   | parent,       |
| G**     | state       | process        | e          | teacher,      |
|         | guidelines  | benchmarks,    | valuation; | principal,    |
| ***"    | for student | relating to    | self-      | and student   |
| Special | performance | faculty        | assessment | surveys;      |
| L       |             | participation, | as part if | review of     |
| iteracy |             | parent         | year-round | achievement   |
| Mode    |             | participation  | SIP        | scores;       |
| l*"\*** |             | and student    | processes  | conduct of    |
|         |             | absences       |            | focus groups; |
|         |             |                |            | reports from  |
|         |             |                |            | external      |
|         |             |                |            | technical     |
|         |             |                |            | assistance    |
|         |             |                |            | providers     |
+---------+-------------+----------------+------------+---------------+
| *       | SIP         | No             | No formal  | Evaluation of |
| *School | describes   | implementation | activities | student       |
| H**     | benchmarks  | benchmarks,    | or plans   | performance   |
|         | for student | other than SFA |            | based on      |
| ***     | p           | implementation |            | state         |
| Success | erformance, | monitoring     |            | assessment;   |
| for     | as well as  |                |            | i             |
| All***  | goals for   |                |            | mplementation |
|         | attendance  |                |            | monitored by  |
|         | and school  |                |            | SFA; no       |
|         | ranking     |                |            | written       |
|         |             |                |            | findings      |
|         |             |                |            | available     |
+---------+-------------+----------------+------------+---------------+
| *       | No clear    | No clear       | No clear   | School        |
| *School | benchmarks  | benchmarks in  | evaluation | frequently    |
| I**     | in place;   | place; did     | plan       | reviews       |
|         | did have    | have goal of   |            | student       |
| *       | goal of     | increased      |            | achievement   |
| **Light | increasing  | parent         |            | data;         |
| span*** | scores on   | participation  |            | developer     |
|         | state       |                |            | previously    |
|         | assessment  |                |            | used          |
|         |             |                |            | q             |
|         |             |                |            | uestionnaires |
|         |             |                |            | and surveys   |
|         |             |                |            | to measure    |
|         |             |                |            | im            |
|         |             |                |            | plementation; |
|         |             |                |            | nor written   |
|         |             |                |            | evaluation    |
+---------+-------------+----------------+------------+---------------+
| *       | Goals       | School set     | External   | Schools       |
| *School | /benchmarks | implementation | evaluator  | reviewed and  |
| J**     | set as part | benchmarks     | focused on | discussed     |
|         | of SIP      | annually       | standards, | student       |
| *       | process     | (e.g., teacher | p          | achievement   |
| **"Perf |             | must prepare   | erformance | data,         |
| ormance |             | one            | measures   | compared      |
| Tasks   |             | performance    | and        | current data  |
| Mode    |             | task to be     | a          | with data     |
| l"\**** |             | accepted by    | lternative | from earlier  |
|         |             | Performance    | a          | years, other  |
|         |             | Task model     | ssessments | schools and   |
|         |             | database)      | (no formal | district and  |
|         |             |                | plan)      | state level   |
|         |             |                |            | data          |
+---------+-------------+----------------+------------+---------------+
| *       | Goals       | No             | School did | Schools       |
| *School | /benchmarks | implementation | not have a | reviewed and  |
| K**     | set as part | benchmarks     | clear      | discussed     |
|         | of SIP      |                | evaluation | student       |
| ***     | process and |                | plan       | achievement   |
| Success | covered     |                |            | data,         |
| for     | limited     |                |            | compared      |
| All***  | subjects    |                |            | current data  |
|         |             |                |            | with data     |
|         |             |                |            | from earlier  |
|         |             |                |            | years, other  |
|         |             |                |            | schools and   |
|         |             |                |            | district and  |
|         |             |                |            | state level   |
|         |             |                |            | data, and     |
|         |             |                |            | utilized SFA  |
|         |             |                |            | eight-week    |
|         |             |                |            | assessments   |
+---------+-------------+----------------+------------+---------------+
| *       | School      | Implementation | District   | End-of-year   |
| *School | expected to | benchmarks     | contracted | report        |
| L**     | reach       | were stated in | with a     | contained     |
|         | benchmarks  | the CSRD       | consultant | district      |
| ***"Beh | established | application,   | to conduct | findings;     |
| avioral | by the      | and monitored  | e          | final         |
| Modif   | district    | by the         | valuation, | evaluation of |
| ication | targeting   | external       | but turned | CSRD delayed  |
| Mode    | student     | developer      | resp       |               |
| l"\**** | performance |                | onsibility |               |
|         | (reading    |                | over to    |               |
|         | readiness,  |                | schools in |               |
|         | attendance, |                | 2001-02    |               |
|         | reading and |                |            |               |
|         | math        |                |            |               |
|         | p           |                |            |               |
|         | roficiency) |                |            |               |
+---------+-------------+----------------+------------+---------------+
| \*      |             |                |            |               |
| Locally |             |                |            |               |
| de      |             |                |            |               |
| veloped |             |                |            |               |
| methods |             |                |            |               |
| are in  |             |                |            |               |
| quotes  |             |                |            |               |
| and     |             |                |            |               |
| have    |             |                |            |               |
| been    |             |                |            |               |
| given   |             |                |            |               |
| pseu    |             |                |            |               |
| donyms. |             |                |            |               |
|         |             |                |            |               |
| **(Co   |             |                |            |               |
| ntinued |             |                |            |               |
| on next |             |                |            |               |
| page)** |             |                |            |               |
+---------+-------------+----------------+------------+---------------+
| *       | District    | No clear       | Evaluation | Findings      |
| *School | established | implementation | activities | indicated     |
| M**     | benchmarks  | benchmarks     | suggested  | that          |
|         | for student |                | by         | Lightspan     |
| *       | p           |                | Lightspan  | students      |
| **Light | erformance, |                | (no        | outperformed  |
| span*** | covering    |                | written    | comparison    |
|         | all grades  |                | plan)      | students in   |
|         | and         |                |            | the state     |
|         | subjects    |                |            | assessment    |
|         |             |                |            | (report       |
|         |             |                |            | available     |
|         |             |                |            | after the     |
|         |             |                |            | 2001-02 SIP   |
|         |             |                |            | developed)    |
+---------+-------------+----------------+------------+---------------+
| *       | SIP set     | School had     | No formal  | Evaluation of |
| *School | benchmarks  | goals for      | activities | student       |
| N**     | for         | parental       | or plans   | performance   |
|         | p           | involvement    |            | based on      |
| ***     | erformance, | and conflict   |            | state         |
| Success | as well as  | resolution,    |            | assessment;   |
| for     | goals       | but no         |            | i             |
| All***  | around      | information on |            | mplementation |
|         | student     | how goals      |            | monitored by  |
|         | attendance  | would be       |            | SFA           |
|         | and school  | measured or    |            |               |
|         | ranking     | benchmarks     |            |               |
|         |             | established    |            |               |
+---------+-------------+----------------+------------+---------------+
| *       | Distri      | Scale-up       | District   | School        |
| *School | ct-mandated | benchmarks had | intends to | completed     |
| O**     | benchmark   | been met;      | conduct an | surveys;      |
|         | tested as   | method in use  | impact     | district      |
| ***Suc  | part of     | in all         | assessment | analysis of   |
| cess-in | co          | classrooms     | (using     | data in       |
| -the-Ma | mprehensive |                | survey and | progress; no  |
| king*** | reading     |                | student    | written       |
|         | program     |                | a          | finding       |
|         | receiving   |                | chievement | available     |
|         | greatest    |                | data); no  |               |
|         | priority.   |                | evidence   |               |
|         |             |                | of written |               |
|         |             |                | plan       |               |
+---------+-------------+----------------+------------+---------------+
| *       | Included    | Listed as      | State      | Formal        |
| *School | extensive   | addressing     | conducting | instrument    |
| P**     | coverage of | organization   | formal     | used to       |
|         | student     | and            | evaluation | evaluate      |
| ***     | performance | curriculum,    | of CSRD    | instruction   |
| HSTW*** | on          | and            | impl       | during        |
|         | assessments | instruction    | ementation | classroom     |
|         | and toward  |                |            | observations  |
|         | graduation  |                |            |               |
|         | r           |                |            |               |
|         | equirements |                |            |               |
+---------+-------------+----------------+------------+---------------+
| *       | District    | Benchmarks in  | No         | School        |
| *School | established | form of        | external   | reviewed      |
| Q**     | benchmarks  | implementation | evaluator; | student       |
|         | based on    | rubric,        | district   | achievement   |
| ***Acce | student     | including      | provided   | data; cadres  |
| lerated | scores on   | unity of       | assistance | made          |
| Sch     | state       | purpose,       | to schools | re            |
| ools*** | assessment  | empowerment,   | in         | commendations |
|         |             | building on    | analyzing  | for           |
|         |             | strengths,     | student    | curricular or |
|         |             | forging a      | a          | instructional |
|         |             | vision, taking | chievement | reform        |
|         |             | stock, setting |            |               |
|         |             | priorities,    |            |               |
|         |             | establishing   |            |               |
|         |             | governance,    |            |               |
|         |             | embedding      |            |               |
|         |             | inquiry, and   |            |               |
|         |             | creating a     |            |               |
|         |             | collaborative  |            |               |
|         |             | culture        |            |               |
+---------+-------------+----------------+------------+---------------+
| *       | Three-year  | Three-year     | No written | No formal     |
| *School | benchmarks  | benchmarks in  | evaluation | reporting of  |
| R**     | in SIP for  | SIP for use of | plan       | evaluation    |
|         | reading,    | technology and | (except    | findings      |
| ***     | writing,    | cooperation    | for plan   |               |
| "Compre | and         |                | for        |               |
| hensive | mathematics |                | technology |               |
| Reform  |             |                | component, |               |
| Mode    |             |                | conducted  |               |
| l"\**** |             |                | as a state |               |
|         |             |                | re         |               |
|         |             |                | quirement) |               |
+---------+-------------+----------------+------------+---------------+
| \*      |             |                |            |               |
| Locally |             |                |            |               |
| de      |             |                |            |               |
| veloped |             |                |            |               |
| methods |             |                |            |               |
| are in  |             |                |            |               |
| quotes  |             |                |            |               |
| and     |             |                |            |               |
| have    |             |                |            |               |
| been    |             |                |            |               |
| given   |             |                |            |               |
| pseu    |             |                |            |               |
| donyms. |             |                |            |               |
+---------+-------------+----------------+------------+---------------+

At the same time, the original CSRD language for the evaluation
strategies (Component 8) does indeed address the implementation of
reform, but the existing evaluation activities also have largely focused
on assessing student performance (third and fourth columns in Exhibit
2-8.) \[As a promising reform strategy, see "Clear Scope and Sequence
and Quarterly Assessments Provide Frequent Measures of Student Growth,"
in Appendix B.\]

***2.3.3 High turnover among staff and students has resulted in
transient levels of support within the school (Component 5) or parental
and community involvement (Component 6).*** The field teams found it
difficult to assess these two CSRD components because schools do not
routinely monitor or maintain precise records regarding either type of
support or involvement. In some cases, faculties did hold votes during
the formative period when a research-based method was first being
adopted. However, documentation for any subsequent faculty support was
absent thereafter. Similarly, schools might have kept records of parent
attendance at one or two key events during the year (e.g., the first
open-school night), but they did not formally monitor or document other
parental or community involvements.

Despite the lack of records, the field teams made the following
observations about the two components. First, high turnover among staff
(e.g., Schools B, E, I, M, and R) or students (e.g., Schools C, E, L,
and P) has meant that the extent of staff support or parental
involvement can be transient: Any judgment regarding support or
involvement therefore depends upon the timing of the data collection
"snapshot." Unfortunately, several of the schools had staff who were
highly supportive of the reform effort but who then left the school
(e.g., Schools C, E, M, and R); assessing staff support earlier would
have shown strong internal support for the method and the reform effort,
but such support likely dissipated greatly by the time of the site
visit. At other schools, support among the staff increased over time as
individual teachers became more familiar with the reform efforts. In a
likewise manner, student turnover meant that the relevant body of
parents also changed, and high involvement during the kickoff year of a
reform effort may not have been sustained the following year.

Second, strong principal leadership in initiating and supporting reform
was a critical ingredient at many of the schools. \[As a reform
strategy, see "School Leadership Committed to Institutionalizing Reform
Efforts," in Appendix B.\] However, turnover among principals also
swayed support or involvement if, as is usually the case, the new
principals had agendas and priorities different from their predecessors
(e.g., Schools L, M, and R).

Interestingly, any difficulties in assessing support and involvement
also may be considered substantive findings, not just methodological
barriers. Whereas time-limited innovations may call for staff support or
parental involvement over shorter periods of time, comprehensive reform
calls for broad support and involvement. The team found that schools had
defined few if any strategies to produce such broad efforts. Other than
a few cases where either staff support for the reform effort (e.g.,
Schools A and F) or community involvement (e.g., Schools O and K) were
strong and persistent, the observed support and involvement only
involved isolated milestones (e.g., the original vote by a faculty to
adopt a research-based method, the convening of a parents' night, or the
use of CSRD funds to hire a parent coordinator) but not sustained trends
or patterns. \[As promising reform strategy, see "Internal Collaborative
Meetings Foster Inter-curricular Teaching and Teacher Buy-in to
Schoolwide Issues," and "Parents and Community Members Participate in
School Governance and Goal Setting," in Appendix B.\]

For other reasons related to the comprehensiveness of the reform effort,
support or involvement also can be spotty. For instance, schools whose
reforms were limited to single academic subjects would not necessarily
have involved staff support or parental involvement outside of these
subjects. Similarly, methods being scaled-up incrementally by grade
level would not necessarily have called for universal support or
involvement during the scale-up process.

***2.3.4 Most schools are in a position to coordinate or converge
resources (Component 9), but resources for sustainability are still
uncertain***. The topics of coordination and sustainability (i.e.,
extending reform efforts beyond the period of CSRD funding) were joined
because schools' prospects for sustainability have been heavily driven
by the need to identify new sources of funds.[^35] Similarly, the
resources to be coordinated under Component 9 have largely been
considered budgetary resources by the schools.

The significance of Component 9 also is related to the underlying spirit
of reform. By definition, comprehensive reform should not be implemented
as an "add-on" or incremental activity, but should affect all school
operations and hence the entire set of a school's resources---core
building budget as well as funds from external sources not legally
needing to be dedicated to a specific activity (e.g., Title I, Title II,
Eisenhower, and other federal and state funds). Any actual and ongoing
comprehensive reform would therefore logically require that the school's
resources be applied in a coordinated fashion.

To collect data on these two topics of coordination and resources for
sustainability, the field teams did focus on budget data but also
collected other information, mainly from the principal, regarding the
school's funding intentions and priorities during the coming year(s). At
10 of the 18 schools, the field teams found evidence that both external
and local funds were being coordinated in some manner to support the
reform effort at the school. In most cases, the majority of that
coordination involved Title I funding (see Exhibit 2-9).

An initial observation is that coordination can be greatly facilitated
if the planning for all of a school's funds is represented within a
single *school improvement plan* that

**Exhibit 2-9**

**STATUS OF COORDINATION OF RESOURCES**

**(Component 9)**

**AND SUSTAINABILITY**

+----------+------------------------+---------------------------------+
| **School | **Types of Resource    | **Evidence of Sustainability**  |
| Name/**  | Coordination**         |                                 |
|          |                        |                                 |
| ***Met   |                        |                                 |
| hod*\*** |                        |                                 |
+==========+========================+=================================+
| **School | School already used    | The principal was committed to  |
| A**      | Title I, other         | providing the needed resources  |
|          | external funds, and    | for the foreseeable future. ED  |
| ***Acc   | central budget---      | and the state recognized the    |
| elerated | along with CSRD        | school for its reform efforts.  |
| Sc       | funds---to support     | The maintenance support needed  |
| hools*** | reform. Funds appeared | by the research-based method is |
|          | to be used in a        | assumed to be modest.           |
|          | coordinated manner in  |                                 |
|          | covering various       |                                 |
|          | reform activities.     |                                 |
+----------+------------------------+---------------------------------+
| **School | Nearly all CSRD funds  | No funding streams appear to be |
| B**      | were directed at       | available to continue school's  |
|          | professional           | relationship with HSTW          |
| ***High  | development (with the  | following the end of CSRD. The  |
| Schools  | addition of some Title | merger of this high school with |
| that     | II and other funds).   | a county school adds further    |
| Work***  | The coordinator        | uncertainty about               |
|          | position was paid from | sustainability of reform        |
|          | the core budget.       | efforts, though the site visit  |
|          |                        | team believes the merger will   |
|          |                        | actually further reform         |
|          |                        | efforts.                        |
+----------+------------------------+---------------------------------+
| **School | The SIP embraces the   | The principal hopes to use      |
| C**      | use of funds from all  | Title I funds to continue after |
|          | sources as part of an  | CSRD has ended. In addition, if |
| ***Co    | integrated allocation  | the school can become a         |
| -NECT*** | plan.                  | "demonstration" school, the     |
|          |                        | developer of the research-based |
|          |                        | method also would then provide  |
|          |                        | its own support.                |
+----------+------------------------+---------------------------------+
| **School | In 2001, district      | The principal did not think the |
| D**      | started requiring      | school would continue to use    |
|          | schools to have single | SFA after CSRD funding ended    |
| **       | plan to converge       | for two reasons: 1) a district  |
| *Success | resources, including   | move to site-based budgeting    |
| for      | CSRD funds, with       | meant that the school would     |
| A        | common objectives      | have to identify funds to fill  |
| ll/Roots | (previously, each      | gap left by CSRD (when large    |
| &        | funding source had its | state cutbacks were looming),   |
| Wings*** | own plan). Convergence | and 2) the SFA reading          |
|          | included external      | curriculum was not submitted    |
|          | funding sources, but   | for state review, and,          |
|          | school reports no      | therefore, the school could not |
|          | additional local funds | use state textbook funds to     |
|          | available due to       | purchase SFA reading materials. |
|          | fiscal crisis.         |                                 |
+----------+------------------------+---------------------------------+
| **School | A previous pattern of  | The school has begun to be      |
| E**      | coordinating funds     | interested in adopting an       |
|          | cannot be applied to   | entirely new research-based     |
| ***C     | the new charter        | method, which may overlap with  |
| oalition | school.                | CES; most teachers show no      |
| of       |                        | interest in formally continuing |
| E        |                        | with the CES method.            |
| ssential |                        |                                 |
| Sc       |                        |                                 |
| hools*** |                        |                                 |
+----------+------------------------+---------------------------------+
| **School | School has been        | The school is searching for     |
| F**      | coordinating funds     | additional funds to replace     |
|          | from multiple external | CSRD funding. However, main     |
| ***C     | and internal sources   | reform initiatives, some of     |
| oalition | to support reform      | which involved changes in       |
| of       | effort in a converging | school scheduling and           |
| E        | manner.                | operations and not an outlay of |
| ssential |                        | funds, are likely continue      |
| Sc       |                        | independent of any funding      |
| hools*** |                        | success.                        |
+----------+------------------------+---------------------------------+
| **School | CSRD funds were used   | Principal was confidant that    |
| G**      | almost entirely for    | new grant monies, along with    |
|          | professional           | Title I, would continue to      |
| ***      | development, along     | support future reform           |
| "Special | with Title I and other | activities.                     |
| Literacy | local funds.           |                                 |
| Mod      |                        |                                 |
| el"**\** |                        |                                 |
+----------+------------------------+---------------------------------+
| \        |                        |                                 |
| *Locally |                        |                                 |
| d        |                        |                                 |
| eveloped |                        |                                 |
| methods  |                        |                                 |
| are in   |                        |                                 |
| quotes   |                        |                                 |
| and have |                        |                                 |
| been     |                        |                                 |
| given    |                        |                                 |
| pse      |                        |                                 |
| udonyms. |                        |                                 |
|          |                        |                                 |
| **(C     |                        |                                 |
| ontinued |                        |                                 |
| on next  |                        |                                 |
| page)**  |                        |                                 |
+----------+------------------------+---------------------------------+
| **School | Convergence of         | The school decided to           |
| H**      | resources from many    | transition to the district      |
|          | different external     | curriculum and end support for  |
| **       | sources appeared to    | the external SFA developer      |
| *Success | have taken place       | beginning in the 2001-02 school |
| for      | because the school and | year. Nevertheless, teacher     |
| All***   | district coordinate    | indicated that they intended to |
|          | school's funds in      | continue using SFA-related      |
|          | relation to the school | classroom management techniques |
|          | improvement plan.      | and questioning strategies.     |
+----------+------------------------+---------------------------------+
| **School | School has funds from  | Although School I's reform      |
| I**      | many external sources  | efforts could be described as   |
|          | and prides itself in   | unfocused, teachers and         |
| ***Ligh  | its entrepreneurship.  | administrators expressed hope   |
| tspan*** | However, multiple      | in Spring 2002 that things were |
|          | sources may have been  | beginning to improve. The 1st   |
|          | more often associated  | grade teachers were             |
|          | with fragmentation     | enthusiastically using          |
|          | rather than            | Lightspan in the absence of     |
|          | coordination of        | external support.               |
|          | efforts.               |                                 |
+----------+------------------------+---------------------------------+
| **School | Some local funds were  | The school believes it can      |
| J**      | used in concert with   | support the reform effort with  |
|          | CSRD to support the    | Title I funds. However, the     |
| ***"Per  | implementation of the  | assignment of a new principal   |
| formance | research-based method. | to the school may affect that   |
| Tasks    |                        | plan; the associate principal   |
| Mod      |                        | being considered for the        |
| el"**\** |                        | position stated he would        |
|          |                        | support a modified use of the   |
|          |                        | method, with only some          |
|          |                        | components remaining.           |
+----------+------------------------+---------------------------------+
| **School | School has used        | Title I funds are expected to   |
| K**      | multiple sources,      | be used to continue SFA support |
|          | including Title I and  | after CSRD funding ends.        |
| *        | state grants, to       | However, continuation also      |
| **SFA*** | support reform. Degree | depends on improved student     |
|          | of convergence is      | performance, as well as SFA's   |
|          | unclear.               | response to teacher's concerns. |
+----------+------------------------+---------------------------------+
| **School | In 2001, district      | Funds to replace CSRD were      |
| L**      | started requiring      | still being sought, but a new   |
|          | schools to have single | administration and waning staff |
| ***"     | plan to converge       | support appeared to be bigger   |
| Behavior | resources, including   | threats to sustainability. By   |
| Modi     | CSRD, with common      | Spring 2002, teacher            |
| fication | objectives. Process    | implementation of some          |
| Mod      | has brought all        | components of the method were   |
| el"**\** | dollars into a single  | already beginning to lapse.     |
|          | reform effort.         |                                 |
|          | However, school        |                                 |
|          | reports no additional  |                                 |
|          | local funds available  |                                 |
|          | due to fiscal crisis.  |                                 |
+----------+------------------------+---------------------------------+
| **School | A former principal     | The school's investment in      |
| M**      | appeared to have       | purchasing the components of    |
|          | converged resources    | the method (the PlayStations,   |
| ***Ligh  | around Lightspan, its  | software, and television        |
| tspan*** | coordinator, computer  | monitors), is evidence that     |
|          | equipment, and the     | some aspect of the method will  |
|          | other resources needed | remain. By the end of the third |
|          | for implementation; by | year, however, the method was   |
|          | 2001-02, no funds were | used sparingly in one grade     |
|          | used to support        | level. The school also learned  |
|          | implementation.        | that it would be merged with    |
|          |                        | another "underperforming"       |
|          |                        | elementary school in 2002-03,   |
|          |                        | adding to uncertainty about     |
|          |                        | future use of Lightspan.        |
+----------+------------------------+---------------------------------+
| **School | School and district    | Withdrawal of district support  |
| N**      | coordinated Title I    | for the research-based method   |
|          | funds with CSRD funds  | jeopardized sustainability of   |
| *        | as part of the school  | reform effort beyond CSRD       |
| **SFA*** | improvement plan.      | funding. The principal was      |
|          | However, school has    | skeptical about whether the     |
|          | had few other          | school would continue to        |
|          | discretionary sources. | implement CSRD without new      |
|          |                        | funding, and the staff          |
|          |                        | indicated that the school would |
|          |                        | not be implementing SFA in      |
|          |                        | 2002-03, even if funding were   |
|          |                        | available.                      |
+----------+------------------------+---------------------------------+
| \        |                        |                                 |
| *Locally |                        |                                 |
| d        |                        |                                 |
| eveloped |                        |                                 |
| methods  |                        |                                 |
| are in   |                        |                                 |
| quotes   |                        |                                 |
| and have |                        |                                 |
| been     |                        |                                 |
| given    |                        |                                 |
| pse      |                        |                                 |
| udonyms. |                        |                                 |
|          |                        |                                 |
| **(C     |                        |                                 |
| ontinued |                        |                                 |
| on next  |                        |                                 |
| page)**  |                        |                                 |
+----------+------------------------+---------------------------------+
| **School | School successfully    | Because reform had been in      |
| O**      | coordinated funds from | progress prior to CSRD, school  |
|          | multiple sources       | anticipated continuing reform   |
| *        | (e.g., Title I and     | after CSRD. External costs      |
| **SIM*** | Title II, as well as   | associated with research-based  |
|          | local funding), in     | method were likely to diminish, |
|          | supporting reform      | and Title I and other funds     |
|          | activities.            | would continue to be used, but  |
|          |                        | the school also was exploring   |
|          |                        | the availability of new sources |
|          |                        | of funding.                     |
+----------+------------------------+---------------------------------+
| **School | School converged       | School and district staff       |
| P**      | federal and local      | strongly support continuation   |
|          | funds, primarily for   | of the reform effort. Extent of |
| **       | professional           | need for funds to cover         |
| *HSTW*** | development activities | external costs associated with  |
|          | related to the reform  | the research-based method were  |
|          | effort.                | unclear. School staff were also |
|          |                        | unclear how the merger with the |
|          |                        | city high school would affect   |
|          |                        | reform efforts.                 |
+----------+------------------------+---------------------------------+
| **School | School had started     | Main threat to sustainability   |
| Q**      | using other external   | is continued high rate of staff |
|          | funds, in a converging | turnover as well as threat of   |
| ***Acc   | manner to support      | principal turnover. The         |
| elerated | variety of reform      | principal hoped that the        |
| Sc       | activities prior to    | formation of a leadership team  |
| hools*** | CSRD funding, and such | would make the reform efforts   |
|          | convergence has        | less dependent on any single    |
|          | continued along with   | individual and therefore serve  |
|          | CSRD funding.          | as a mechanism for sustaining   |
|          |                        | reform.                         |
+----------+------------------------+---------------------------------+
| **School | School had not used a  | Some isolated initiative would  |
| R**      | large portion of its   | be sustained, nut not as part   |
|          | CSRD funds in its      | of a overall reform design;     |
| *        | final year of funding  | greater attention was placed on |
| **"Compr | and obtained an        | the district-required "small    |
| ehensive | extension on the grant | learning communities.           |
| Reform   | (and, consequently,    |                                 |
| Mod      | had not converged or   |                                 |
| el"**\** | coordinated funds).    |                                 |
+----------+------------------------+---------------------------------+
| \        |                        |                                 |
| *Locally |                        |                                 |
| d        |                        |                                 |
| eveloped |                        |                                 |
| methods  |                        |                                 |
| are in   |                        |                                 |
| quotes   |                        |                                 |
| and have |                        |                                 |
| been     |                        |                                 |
| given    |                        |                                 |
| pse      |                        |                                 |
| udonyms. |                        |                                 |
+----------+------------------------+---------------------------------+

converges on common goals and objectives (see the coordination column of
Exhibit 2-9). Of the 10 schools exhibiting converging funding sources, 3
had such a condition (Schools C, G, and H). Even in the absence of such
site-based management conditions, 7 other schools reported coordination
or convergence of resources to the field teams (Schools A, B, F, I, K,
O, and P), with 3 of them mentioning integration with the building's
budget. Thus a total of 10 of the 18 schools appear to have been in a
position to coordinate or converge their resources. Another 5 schools
(D, J, L, N, and Q) reported coordination of federal funding sources to
support reform, but did not have the necessary access to local funds in
order to redirect them (in the case of School D and L, the principals
reported that they had the ability to access local funds, but severe
budget cuts meant that after obligatory allocations---i.e., teacher
salaries---no funds remained). (See "Move to Site-Based Management
Allows School to Utilize Slack Resources for Reform Initiatives," in
Appendix B.) Reports of lack of coordination came from only 3 schools:
School E, which had ceased implementation; School M, which had made the
capital expenditures associated with Lightspan and was now implementing
the method at no cost; and School R, which had not really implemented
CSRD, as previously discussed.

At the same time, the field teams found that the ability to coordinate
funds did not necessarily coincide with the extent of a school's
discretionary resources. Yet, the latter may be critical for
sustainability (see the second column of Exhibit 2-9).

One reason for the need for discretionary resources is that a major CSRD
component---e.g., the use of a research-based method---usually requires
an outlay of additional funds, for such items as: a) on-site assistance,
b) release time for professional development, c) salary support for
on-site coordinators, d) the purchase of new types of supplies, and in
some cases, e) hardware and software purchases. The availability of the
CSRD funds has served well the needs of the startup periods in
implementing these methods. Although the extent of outlays for
maintaining a method after startup are reported to diminish, nearly all
of the schools are unclear about the amount or source of any new funds
needed after the CSRD award has ended. Title I funds are mentioned
frequently as one source of such funds, but whether they will be
sufficient also is unclear.

The likelihood of sustainability also will be affected by non-funding
conditions--- mainly the possible lack of support for reform on the part
of the school staff or the threat of staff or principal turnover. Of the
18 schools, 5 have reported such jeopardy (Schools B, D, L, M, and Q),
in addition to School H, which already stopped implementing the
research-based method. Section 3.5 examines this issue in greater
detail.

**2.4 SUMMARY**

Across the 9 CSRD components, schools have devoted their greatest
implementation efforts to the research-based method (Component 1);
professional development (Component 3)---usually linked with the method;
and external technical assistance and support (Component 7)---also
usually linked with the method. Schools have established clear student
performance benchmarks (Component 4). All of the other components have
been less well developed.

**3. CSRD IMPLEMENTATION AND SUSTAINABILITY:**

**THE INTEGRATED FINDINGS**

This section first discusses district and state influences on the CSRD
schools. The section then provides a summary assessment of CSRD
implementation, using the district and state influences, along with
other conditions, to identify three potentially different pathways to
implementation. Finally, the section discusses the prospects for CSRD
sustainability at the 18 schools.

**3.1 DISTRICT AND STATE INFLUENCES ON CSRD**

Although CSRD focuses on school activities, district and state
conditions can be extremely important to the administration of CSRD. The
federal CSRD awards are to state education agencies, who in turn make
subawards to districts on behalf of specific schools. As a result, any
number of CSRD processes and outcomes may be influenced by the district
and state policies and practices.

To examine district and state conditions, the field teams conducted
interviews, collected documents, and reviewed other data from the 18
schools' district and state agencies. For the districts, site visits
were made (in addition, district officials were occasionally interviewed
while present at the schools at the time of the school site visit).
Information about the state agencies more frequently came from short
telephone conversations with state officials, searches of the agencies'
Web sites, and copies of state documents and reports provided by the
district offices.

Exhibit 3-1 shows the various district and state conditions that
appeared relevant to the CSRD activities in the 18 schools. The exhibit
pairs the schools, with each pair being a part of the same district and
state, reflecting the original sample design. For each pair, the exhibit
then enumerates salient conditions that might affect the work of CSRD,
dividing the salient conditions into two categories: assistance or
support in administering CSRD; and other policies or conditions
potentially affecting CSRD implementation though not a direct part of
administering CSRD. Each category is discussed in turn.

***3.1.1 Administering CSRD: Districts and states provide varying
degrees of support***. Both states and districts have explicit
responsibilities in administering the funds for CSRD. However, beyond
these core responsibilities, both districts and states varied widely in
providing other forms of CSRD support or assistance. At one extreme
(e.g., Schools M and R), the state education officials engaged schools
in the CSRD application process; made site visits to monitor the initial
progress of the schools; and convened annual meetings of the CSRD
schools in the state, to facilitate the exchange of information on

**Exhibit 3-1**

**SCHOOL, DISTRICT, AND STATE CONDITIONS**

+-----------+--------------------------+-------------------------------+
| **        | **State Conditions**     | **District Conditions**       |
| Schools** |                          |                               |
+===========+==========================+===============================+
| **School  | **Non-CSRD related:**    | **Non-CSRD related:**         |
| A**       |                          |                               |
|           | ! State sets benchmarks  | ! The district essentially    |
| *Ac       | for improvement based on | only provides assistance on   |
| celerated | state assessment.        | an as-needed basis.           |
| Schools*  | Schools that fail to     | Professional development      |
|           | make progress, over      | funds are distributed to      |
| and       | time, may be closed.     | schools; schools decide how   |
|           | School E has been in     | to use them. Other assistance |
| **School  | lowest category and      | only provided when requested  |
| E**       | School A has been "in    | by the schools.               |
|           | need of improvement," in |                               |
| *         | the state system.        | ! District sets benchmarks    |
| Coalition |                          | for improvement on state      |
| of        |                          | assessments.                  |
| Essential |                          |                               |
| Schools*  |                          | ! District has vacillated on  |
|           |                          | future of the School E, now   |
|           |                          | has decided to continue the   |
|           |                          | school as a charter school.   |
+-----------+--------------------------+-------------------------------+
| **School  | **Non-CSRD related:**    | **CSRD related:**             |
| B**       |                          |                               |
|           | ! No state assessment    | ! District has provided staff |
| *HSTW*    | has yet appeared related | support and professional      |
|           | to reform.               | development in relation CSRD. |
| and       |                          |                               |
|           |                          | **Non-CSRD related:**         |
| **School  |                          |                               |
| P**       |                          | ! The district's              |
|           |                          | superintendent resigned       |
| *HSTW*    |                          | effective the end of 2000-01. |
|           |                          |                               |
|           |                          | ! Board votes to merge        |
|           |                          | Schools B and P into a single |
|           |                          | high school, starting in      |
|           |                          | 2003-04.                      |
+-----------+--------------------------+-------------------------------+
| **School  | **Non-CSRD related:**    | **CSRD related:**             |
| C**       |                          |                               |
|           | ! Performance on state   | ! The district provided       |
| *Co-NECT* | assessment is used to    | workshops in plan development |
|           | set goals. The state     | to ensure alignment.          |
| and       | also provides a rubric   |                               |
|           | for the development of   | **Non-CSRD related:**         |
| **School  | district plans that is   |                               |
| G**       | tied to state standards, | ! The district is suffering   |
|           | requiring school plans   | the loss of teachers due to   |
| *"Special | to link funded           | low teacher salaries and      |
| Literacy  | activities with goals,   | teacher shortages. Nearby     |
| Model"\** | benchmarks, and          | districts and states are      |
|           | strategies.              | offering bonuses to attract   |
|           |                          | teachers.                     |
|           |                          |                               |
|           |                          | 1.  The district has had a    |
|           |                          |     serious fiscal shortfall. |
+-----------+--------------------------+-------------------------------+
| **School  | **CSRD related:**        | **CSRD related:**             |
| D**       |                          |                               |
|           | ! Because of state       | ! District decided which      |
| *Success  | timing in notifying      | schools should apply; hired   |
| for All*  | schools of eligibility   | consultant to write CSRD      |
|           | and application          | proposals for the schools.    |
| *Roots &  | deadline, applications   |                               |
| Wings*    | were written over the    | **Non-CSRD related:**         |
|           | summer. At School L,     |                               |
| and       | many staff were not      | ! District's strategic plan   |
|           | present to participate   | adopted under new             |
| **School  | in selection of the CSRD | superintendent in 1998, calls |
| L**       | method.                  | for a new emphasis on high    |
|           |                          | standards of academic         |
| *         | **Non-CSRD related:**    | performance and has been a    |
| "Behavior |                          | driving force behind school   |
| Mod       | ! State has put into     | reform efforts.               |
| ification | place new high stakes    |                               |
| Model"\** | examinations.            |                               |
|           |                          |                               |
|           | ! State revising         |                               |
|           | assessments to make them |                               |
|           | standards-based.         |                               |
+-----------+--------------------------+-------------------------------+
| \*Locally |                          |                               |
| developed |                          |                               |
| methods   |                          |                               |
| are in    |                          |                               |
| quotes    |                          |                               |
| and have  |                          |                               |
| been      |                          |                               |
| given     |                          |                               |
| ps        |                          |                               |
| eudonyms. |                          |                               |
|           |                          |                               |
| **(       |                          |                               |
| Continued |                          |                               |
| on next   |                          |                               |
| page)**   |                          |                               |
+-----------+--------------------------+-------------------------------+
| **School  | **CSRD related:**        | **CSRD related:**             |
| F**       |                          |                               |
|           | ! State conducted        | ! District office provides    |
| *         | monitoring visits to     | extensive support to schools  |
| Coalition | each site during the     | at the start of CSRD          |
| of        | past year, and provided  | implementation and conducts a |
| Essential | assistance.              | mid-year checkup.             |
| Schools*  |                          |                               |
|           | **Non-CSRD related:**    | **Non-CSRD related:**         |
| and       |                          |                               |
|           | ! New state              | ! School O originally         |
| **School  | accountability system    | intended to implement         |
| O**       | holds teachers and       | Co-NECT, but was "encouraged" |
|           | schools accountable for  | by the district to expand     |
| *Suc      | student progress. School | existing SIM method (after    |
| cess-in-t | O received an "F" rating | receiving an "F" rating).     |
| he-Making | which hurt its magnet    |                               |
| (SIM)*    | program (and which they  | ! District implementing a     |
|           | believed was unfair).    | reading program that will     |
|           |                          | phase out SFA.                |
+-----------+--------------------------+-------------------------------+
| **School  | **CSRD related:**        | **CSRD related:**             |
| H**       |                          |                               |
|           | ! The curriculum match   | ! The previous superintendent |
| *Success  | between the CSRD method  | strongly supported the use of |
| for All*  | and the state assessment | external methods for reform   |
|           | is not perfect, causing  | and provided local funds to   |
| and       | School N to add a        | pay for them.                 |
|           | reading program          |                               |
| **School  | specifically on skills   | ! A new superintendent has    |
| N**       | addressed by the state   | ceased that practice, and     |
|           | assessment.              | only schools with CSRD funds  |
| *Success  |                          | currently have external       |
| for All*  | **Non-CSRD related:**    | methods. Schools H stopped    |
|           |                          | using its external method in  |
|           | ! The state assessment   | 2001-02, and School N may     |
|           | and state accountability | have difficulty finding       |
|           | system is driving        | budget support for external   |
|           | instruction in the       | methods after CSRD; other     |
|           | schools.                 | schools have adopted a local  |
|           |                          | reading model aligned with    |
|           |                          | the state assessment.         |
|           |                          |                               |
|           |                          | **Non-CSRD related:**         |
|           |                          |                               |
|           |                          | ! The district also has       |
|           |                          | started interim and           |
|           |                          | high-stakes assessments,      |
|           |                          | aligned with state standards. |
+-----------+--------------------------+-------------------------------+
| **School  | **CSRD related:**        | **CSRD-related:**             |
| I**       |                          |                               |
|           | ! State education staff  | ! The district identified     |
| *L        | conduct site visits to   | schools that were ready for   |
| ightspan* | CSRD schools, also       | CSRD and notified only those  |
|           | providing assistance.    | schools about the             |
| and       |                          | availability of CSRD funding. |
|           | **Non-CSRD related:**    |                               |
| **School  |                          | **Non-CSRD related:**         |
| Q**       | ! Simultaneous with the  |                               |
|           | CSRD award, state budget | ! District has helped schools |
| *Ac       | cuts resulted in a       | prepare for state assessment  |
| celerated | several million dollar   | and to use data to improve    |
| Schools*  | cut in district funding, | instruction.                  |
|           | in part resulting in the |                               |
|           | loss of the district's   | ! District has now gone two   |
|           | P.D. department and cuts | years with no professional    |
|           | to program evaluation.   | development department.       |
+-----------+--------------------------+-------------------------------+
| **School  | **Non-CSRD related:**    | **CSRD related:**             |
| J**       |                          |                               |
|           | ! State designated       | ! CSRD is coordinated by the  |
| *"Pe      | School K as "needing     | district's director of        |
| rformance | improvement," requiring  | federal and state programs,   |
| Tasks     | it to select a           | the assistant superintendent  |
| Model"\** | research-based method    | for instruction, and the      |
|           | using CSRD funds. The    | director of research and      |
| and       | state also designated    | evaluation.                   |
|           | School J as a "priority  |                               |
| **School  | school," receiving extra | **Non-CSRD related:**         |
| K**       | funds to support         |                               |
|           | improvement efforts and  | ! District hired a new        |
| *Success  | being assigned a         | superintendent in 2000-01.    |
| for All*  | "critical friend."       | Superintendent supports       |
|           |                          | reform but is focused on      |
|           | ! The state assessment   | reorganizing the district's   |
|           | is standards based.      | central office.               |
+-----------+--------------------------+-------------------------------+
| \*Locally |                          |                               |
| developed |                          |                               |
| methods   |                          |                               |
| are in    |                          |                               |
| quotes    |                          |                               |
| and have  |                          |                               |
| been      |                          |                               |
| given     |                          |                               |
| ps        |                          |                               |
| eudonyms. |                          |                               |
|           |                          |                               |
| **(       |                          |                               |
| Continued |                          |                               |
| on next   |                          |                               |
| page)**   |                          |                               |
+-----------+--------------------------+-------------------------------+
| **School  | **CSRD related:**        | **CSRD related:**             |
| M**       |                          |                               |
|           | ! State convenes CSRD    | ! CSRD administered by Title  |
| *L        | staffs annually to       | I coordinator.                |
| ightspan* | reinforce and support    |                               |
|           | schools' efforts.        | ! District helped School R    |
| and       |                          | develop its CSRD application. |
|           | ! State makes site       |                               |
| **School  | visits to CSRD schools,  | **Non-CSRD related:**         |
| R**       | also providing           |                               |
|           | assistance.              | ! District assessments are    |
| *"Comp    |                          | standards-based and aligned   |
| rehensive | **Non-CSRD related:**    | with state.                   |
| Reform    |                          |                               |
| Model"\** | ! School M was in        | ! However, School M received  |
|           | state's lowest category, | district quality performance  |
|           | reducing staff morale,   | award in same year as state   |
|           | and esteem.              | placed it in lowest           |
|           |                          | underperforming category.     |
|           | ! State's restructuring  |                               |
|           | of school finance, and   |                               |
|           | budget cutbacks, disrupt |                               |
|           | district and schools.    |                               |
+-----------+--------------------------+-------------------------------+
| \*Locally |                          |                               |
| developed |                          |                               |
| methods   |                          |                               |
| are in    |                          |                               |
| quotes    |                          |                               |
| and have  |                          |                               |
| been      |                          |                               |
| given     |                          |                               |
| ps        |                          |                               |
| eudonyms. |                          |                               |
+-----------+--------------------------+-------------------------------+

CSRD experiences. In a similar manner (e.g., School L), a district
completed the applications and selected the research-based method on
behalf of the school. At the other extreme (e.g., School E) the state
played little or no role beyond making the CSRD awards, and the district
also offered little assistance to these schools in administering CSRD
(School A). Exhibit 3-1 shows how the other states or districts assumed
different postures in the CSRD administrative process.

Among all the variants, an intermediate state and district posture would
seem to be the best in serving comprehensive school reform. Such reform
requires strong internal staff support within the school, but such
reform is aimed at improving student achievement--- which is assessed
through district or state instruments (or both). In the most active
role, if a district selects the candidate schools to be in the CSRD
program, writes their proposals, and defines the research-based methods
to be adopted, or if the state requires the school to participate in
CSRD because of low achievement scores, the school may then have a poor
understanding of its CSRD mission and may not become sufficiently
empowered to undertake serious reform. In the most passive role, if
districts and states leave schools entirely alone to deal with CSRD, the
schools may receive insufficient guidance to understand the most desired
curriculum and instructional practices relevant to the district or
state's student assessment instruments. The hypothesized intermediate
role would be where districts and states: a) encourage and then support
schools to make their own decisions about method selection and reform
design---explicitly defining the alignment between these choices and the
content of the district or state assessments, b) engage schools in
developing their CSRD applications from their outset, and finally c)
provide technical or other assistance to assure the continued alignment
between the school's reform initiative and the student outcome measures
against which the success of the initiative will be judged.

***3.1.2 District and state policies not directly related to
administering CSRD nevertheless can influence CSRD implementation***.
Exhibit 3-1 also enumerates many district and state policies not
directly related to administering CSRD. These policies reflect ongoing
district and state priorities independent of the CSRD program, and an
important benefit of CSRD was its complementarity with these ongoing
priorities.

The original field protocol did not consider most of these conditions
external to the administration of CSRD to be relevant topics of data
collection. However, the field teams quickly encountered a host of
external events that appeared to affect CSRD implementation directly.
The conditions found included: 1) extremely limited professional
development resources (see Schools I and Q); 2) the direct alignment of
schools' CSRD designs with district plans and state standards (Schools C
and G); 3) districts reducing financial support for all external methods
(Schools H and N); 4) decisions to allow a CSRD school to become a
charter school (School E); 5) decisions to merge two CSRD schools
(Schools B and P). Many states and districts also were suffering severe
revenue shortfalls during this period of time, creating an uncertain if
not disruptive environment for school reform.

These non-CSRD-related district and state policies were found to be
sufficiently important that Exhibit 3-1 divides the schools into two
categories---one in which there appeared to be strong "vertical"
influence (from state to district to school) and the other in which such
influence appeared to be neutral. Strong influence could lead to either
complementarity or conflict with CSRD. On the complementary side, CSRD
provided Schools J and K, for instance, with resources and a compatible
reform agenda to respond to their designation as underperforming schools
in the state's accountability system. On the conflicting side, the
shifting content of state assessments could lead districts to use
resources for alternative curricula and professional development that
were contrary to those involved in the schools' original CSRD plans or
implementation (e.g., Schools H, N, and O). Likewise, a report
suggesting that schools with block schedules performed more poorly on
state assessments led another CSRD school (School B) to reconsider its
ongoing implementation of a research-based method requiring the use of
such schedules.

In general, these external state and district conditions appear to be
highly relevant to CSRD implementation. In fact, any tendency toward a
"vertical" alignment among state, district, and school policies and
practices may not only impact CSRD implementation but may very well be
critical to the long-run sustainability of CSRD, discussed later in this
section.

**3.2 SCHOOLS IMPLEMENTING COMPREHENSIVE SCHOOL REFORM**

Given the preceding discussion of district and state conditions, the
analysis now turns to a summary assessment of the extent to which the 18
schools have been implementing CSRD. In this assessment, the more that a
school was implementing the nine CSRD components, the greater was judged
its *strength of CSRD implementation*.

***Implementation Instrument and Measurement***. Assessing the strength
of implementation of a complex school intervention, such as
comprehensive school reform, has neither a standard methodology nor a
standardized instrument. As a result, the Field-Focused Study developed
and used its own assessment instrument, shown in Appendix E. This
assessment instrument is but the start of a tool with which to measure
reform and requires much further work to formalize. Nevertheless, the
instrument should provide a solid framework upon which others can
improve.

The instrument covers the 9 components, but deliberately weights the
components differently. The weights for the different components
reflected the Field-Focused Study team's view of the importance of the 9
components, based on observations during the early rounds of site
visits. The research-based method (9 points) and the comprehensiveness
of design (11 points) have the highest proportion of the total points;
professional development emerged as the third most impactful component;
and the other components followed. This weighting system should be
refined during future evaluations.

The instrument asked the site visit teams to use their case reports and
provide "objective" information for somewhat less than half of the items
and to make "subjective" ratings for the remainder of the items.[^36]
The teams had sought accurate and precise data on objective items, but
such data were not always available or retrievable, and in many cases
the "objective" response was based on the team's best estimate. The
scores for each school were tallied, and the results are shown in
Exhibit 3-2.

The 47-point instrument's strength is that is operationalizes CSRD's
nine components. With no standardized instrument available to measure
CSRD, the instrument allowed the evaluation team to break global
concepts of reform into smaller operational components. If a school
successfully implemented all aspects of all 9 components, therefore, it
could achieve a maximum score of 47 points (thus the instrument has been
referred to as the "47-point instrument"). The greatest threat to
measuring reform in this manner is that it risks losing site of the
"whole picture." As a result, the final tallies were checked against a
global judgment by the teams.

A weakness of the instrument is that the Field-Focused Study did not
have sufficient time or resources to establish the validity or
reliability of the instrument to a level that

**Exhibit 3-2**

**SCORES FOR CSRD IMPLEMENTATION STRENGTH AT 18 SCHOOLS,**

# USING 47-POINT INSTRUMENT\*

  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  **CSRD Component**   **CSRD                                                                                                                                                                            
                       School**                                                                                                                                                                          
  -------------------- ---------- ----------- ----------- -------- -------- ----------- -------- -------- ---------- ---------- ---------- ---------- ---------- -------- -------- ---------- ---------- --------
                       **A**      **B**       **C**       **D**    **E**    **F**       **G**    **H**    **I**      **J**      **K**      **L**      **M**      **N**    **O**    **P**      **Q**      **R**

  **1. Research-Based                                                                                                                                                                                    
  Method (9 points)**                                                                                                                                                                                    

  Bodilly Score        5          3           3           5        1        4           5        1        3          4          4          2          3          5        3        4          4          1

  \% Classrooms Using  1          .75         .75         1        0        .75         1        0        1          .9         .9         .8         .1         1        1        .9         .8         0

  Fidelity Rating      3          2           2           3        1        2           3        1        1          2          3          2          1          2        2        3          3          1

  **2. Comprehensive                                                                                                                                                                                     
  Design (11 points)**                                                                                                                                                                                   

  Written Design or    1          1           1           1        1        1           1        1        1          1          1          1          1          1        1        1          1          1
  Plan                                                                                                                                                                                                   

  Contents of Plan     7          6           5           7        4        7           6        6        4          7          5          7          2          4        6        6          5          2

  Breadth of Plan      3          3           3           3        2        3           3        2        2          3          3          3          2          2        2        3          2          1

  **3. Professional                                                                                                                                                                                      
  Development (7                                                                                                                                                                                         
  points)**                                                                                                                                                                                              

  Strong Content Focus 1          1           0           1        0        1           0        1        1          1          1          1          0          1        1        1          1          0

  Range of PD Days     3          2           2           2        2        3           3        2        3          2          3          2          2          2        3        3          2          2

  Collective           1          1           1           1        1        1           1        1        1          1          1          1          1          1        1        1          1          0
  Participation                                                                                                                                                                                          

  Classroom-Based PD   1          1           1           1        0        1           1        1        1          1          1          0          0          0        1        1          0          1

  Alignment of PD      1          1           1           1        0        1           1        1        1          1          1          1          1          1        1        0          1          1

  **4. Measurable                                                                                                                                                                                        
  Goals and Benchmarks                                                                                                                                                                                   
  (4 points)**                                                                                                                                                                                           

  Number of Academic   2          3           2           2        0        2           2        3        2          2          1          2          3          2        2        3          2          3
  Subjects Covered                                                                                                                                                                                       

  Number of Grades     1          1           1           1        0        1           1        1        .8         1          1          1          1          1        1        1          1          1
  Covered                                                                                                                                                                                                

  **5. Support within                                                                                                                                                                                    
  the School** **(4                                                                                                                                                                                      
  points)**                                                                                                                                                                                              

  Formal Vote on       1          1           1           1        0        1           1        1        0          1          1          0          0          1        1        1          1          0
  Method                                                                                                                                                                                                 

  Support Exceeded 75  1          0           1           1        0        1           1        1        0          1          1          0          0          1        1        1          1          0
  percent                                                                                                                                                                                                

  Support Voiced for   1          0           1           1        1        1           1        1        1          1          0          0          1          0        1        1          1          0
  Support                                                                                                                                                                                                

  No Dissent Voiced    1          0           0           1        0        1           1        0        1          1          0          0          0          0        1        1          1          0

  \* See Appendix E                                                                                                                                                                                      
  for a copy of the                                                                                                                                                                                      
  instrument                                                                                                                                                                                             

  **(Continued on next                                                                                                                                                                                   
  page)**                                                                                                                                                                                                

                                                                                                                                                                                                         

  New Forms of Parent  1          1           0           1        0        1           1        1        1          1          1          1          1          1        1        0          1          0
  Involvement                                                                                                                                                                                            

  Level of Involvement 2          1           0           1        0        1           1        1        1          0          2          0          2          2        2        0          2          0

  At Least One         1          1           1           1        1        1           0        1        1          1          1          1          1          1        1        1          1          1
  Community Org.                                                                                                                                                                                         

  **7. External                                                                                                                                                                                          
  Technical Support                                                                                                                                                                                      
  and Assistance (4                                                                                                                                                                                      
  points)**                                                                                                                                                                                              

  Level of Developer   3          3           3           3        1        2           3        2        1          3          3          3          2          3        3        3          2          1
  Support                                                                                                                                                                                                

  Other External       1          1           1           1        0        1           1        1        1          1          0          0          0          1        0        1          0          0
  Support                                                                                                                                                                                                

  **8. Evaluation                                                                                                                                                                                        
  Strategies (2                                                                                                                                                                                          
  points)**                                                                                                                                                                                              

  Written Evaluation   0          1           0           1        0        0           0        0        0          0          1          1          0          0        0        1          0          0
  Plan                                                                                                                                                                                                   

  Evidence of Written  0          1           1           0        0        0           1        0        0          1          0          0          1          0        0        1          1          0
  Findings                                                                                                                                                                                               

  **9. Coordination of                                                                                                                                                                                   
  Resources (2                                                                                                                                                                                           
  points)**                                                                                                                                                                                              

  Coordination of      1          1           1           1        0        1           1        1        1          0          1          1          0          1        1        1          1          0
  External Funds                                                                                                                                                                                         

  Coordination of      1          1           1           0        0        1           1        1        1          1          1          0          0          0        1        1          0          0
  External and Local                                                                                                                                                                                     
  Funds                                                                                                                                                                                                  

  **Total (maximum =   **44**     **37.75**   **33.75**   **42**   **15**   **39.75**   **41**   **32**   **30.8**   **38.9**   **37.9**   **30.8**   **25.1**   **34**   **38**   **40.9**   **35.8**   **16**
  47)**                                                                                                                                                                                                  

  **Rank**             **1**      **9**       **12**      **2**    **18**   **5**       **3**    **13**   **14**     **6**      **8**      **14**     **16**     **11**   **7**    **4**      **10**     **17**
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

would satisfy psychometric standards. The needed tool would be a
"policy" tool, because of the instability of the policy
environment---e.g., the 9 CSRD components became 11 components a few
years later under P.L. 110-103. Psychometric tools require a longer
"shelf-life," because the time and effort to establish their validity
and reliability (and their generalizability) assumes that a set of
substantive priorities will remain stable over a much longer period of
time. As a result, "policy" tools seldom have sufficient evidence to
establish their validity or reliability to a degree that will satisfy
psychometric standards.

The team did corroborate its use of the instrument, however, through an
independent review of the cases by an analyst "blind" to the rating
generated by the instrument (see Appendix F).[^37] Exhibit 3-3 compares
the analyst's rankings of the 18 schools with the rankings from the
scores on the 47-point instrument. The results show a strong correlation
between the two sets of rankings, with the rankings differing to a
modest degree for a few schools (Schools O, Q, and L), but with the
overall pattern achieving statistical significance (correlation
coefficient = .734, n = 18, p \< .001).

***Implementation: A solid majority of the schools implemented CSRD***.
Given support by the corroboratory procedure, the scores in Exhibit 3-2
were accepted as assessing the strength of CSRD implementation at the 18
schools. Nine schools (Schools A, B, D, F, G, J, K, O, and P) garnered
80 percent of the 47 points, with scores higher than 37.6 points. These
schools were labeled as "nearly fully" implementing CSRD. Another six
schools (Schools C, H, I, L, N, and Q) garnered 65 percent of the 47
points, with scores between 30.6 and 37.6. These schools were labeled
"partially" implementing CSRD. The remaining three schools were judged
to be "poorly" implementing CSRD.[^38]

Other observers may differ with regard to the cutoff points chosen for
these labels, or even the use of the labels themselves. However, the
Field-Focused Study interpreted the overall set of scores as indicating
that a solid majority of the 18 schools were implementing CSRD. As a
result, student achievement scores at these schools could be expected to
be affected, assuming some relationship between comprehensive school
reform and student performance.

**Exhibit 3-3**

**COMPARISON OF RANKINGS FOR 18 SCHOOLS,**

**BASED ON TWO INDEPENDENT ASSESSMENTS OF**

![](media/image5.wmf){width="6.094444444444444in"
height="4.975694444444445in"}**STRENGTH OF CSRD IMPLEMENTATION**

**3.3 THREE EXAMPLES OF COMPREHENSIVE SCHOOL REFORM**

**UNDER THE CSRD PROGRAM**

So far, the analysis has examined the 9 CSRD components, pertinent
district and state conditions, and the overall extent of implementation
based on progress on the 9 components. The analysis has purposely been
fragmented, to assure readers that the 9 components and external
conditions have been systematically examined. However, how all of these
components and conditions might work together and become integrated
within a specific school may not be readily apparent.

To gain a more coherent picture, three schools were selected because
they seem to be making good progress in implementing CSRD but appear to
show a slightly different pattern of events in doing so. Appendix D
contains brief summaries for each of the 18 schools, and the
descriptions for the three schools have been extracted from these
reports.[^39]

***First Example***. The first school (School F) began implementing
*Coalition of Essential Schools* prior to its CSRD award (Component 1).
Under CSRD, the school has expanded its use of the method and also now
uses it in combination with other reform methods, including *High
Schools That Work*. Together, the methods cover the school's entire
range of curriculum and instructional practices. The school has
developed and shared a comprehensive vision for reform (Component 2),
including a written plan that is well understood and supported by the
entire staff (Component 5). The school is making progress on all CSRD
components. For instance, large numbers of the faculty participate in
learning groups and related professional development activities
(Component 3), the school has successfully engaged parents and community
members to serve on key committees (Component 6), and the school
coordinates multiple external and internal resources in a converging
manner (Component 9). However, the school's clear goals and benchmarks
(Component 4) are not matched by a clear evaluation plan or activity
(Component 8).

An apparent key to the first school's progress has been the strong
leadership of the principal and the low turnover of the teaching staff.
The district also uses a performance and appraisal process that
emphasizes professional development and is therefore compatible with the
needs of comprehensive reform.

***Second Example***. The second school (School P) only started its
reform activities with its CSRD award and is implementing *High Schools
That Work* as its main reform method (Component 1). The method's
comprehensive coverage provides a framework for all school improvement
activities at the school (Component 2), and the staff and a core
leadership team fully support the research-based method and the reform
plan (Component 5). The school also is making progress on all CSRD
components including the coordination of resources (Component 9). For
instance, the state is conducting an external evaluation covering all 9
components, and the school itself uses a formal instrument to observe
and evaluate classroom instruction (Component 8). The school-to-work
aspect of the research-based method attracts a good deal of business and
community support (Component 6).

In support of reform at this second school, the district provides
substantial staff support for CSRD activities and has compensated for
any CSRD funding shortfalls in professional development. With strong
external support from the developer of HSTW (Component 7), School P is
now engaged in implementing a complementary research-based method that
aligns its work with that of middle school feeders.

***Third Example***. The third school (School G) also began its reform
activities with the first year of CSRD funding, adopting a locally
developed *Special Literacy Model* (Component 1). Although the method
mainly pertains to grades K-3 and reading, the school's reform
activities also have embraced the other grades (4-6) and academic
subjects, as driven by a comprehensive and converging school improvement
plan (Component 2). The district requires the plan to be aligned with
its own strategic goals and plan. Both must reflect, according to a
state mandate, the state's standards and assessment tools (the state
also mandates that every school have an aligned plan). School G has
therefore made progress on all CSRD components. For instance, the
school's staff strongly support the reform (Component 5), and the school
closely monitors its progress on relevant goals and benchmarks
(Component 4) that is key to the year-round school improvement planning
process. School G's small size, collaborative management style, and
history of reform have all contributed strongly to the CSRD efforts.

**3.4 THREE PATHWAYS FOR IMPLEMENTING**

**COMPREHENSIVE SCHOOL REFORM**

The essence of CSRD is its focus on improving student performance
through comprehensive school reform. The three schools just described
are among several that appear to be making solid progress along these
lines but also may provide insight into the conditions under which
schools are successful in *implementing*[^40] schoolwide reform. Simply
put, the three schools may represent three different sets of
conditions---each characterized as a different pathway to comprehensive
school reform.

The first pathway, represented by School F, is implicit in the formal
design of the CSRD program and is thus a *component-driven* pathway: A
school envisions and implements comprehensiveness in terms of the 9 CSRD
components, with many of the components compatible with the principles
espoused by a "process"-oriented research-based method (*Coalition of
Essential Schools*). The second pathway, represented by School P, is
totally compatible with the formal design of the CSRD program but may be
characterized as a *method-driven* pathway: A school implements a broad
and comprehensive research-based method that affects virtually all
school operations, including curriculum and instruction, and whose
successful implementation substitutes for the need for any independent
articulation of the 9 CSRD components.[^41] The third, represented by
School G and also fully compatible with the formal design of the CSRD
program, is a *vertical-driven* pathway: State or district strategic
planning drives schools toward the nine components, including the
research-based methods or strategies (Component 1), without specifically
enumerating them.

Whether the latter two pathways should be given as much weight as the
first pathway, which best suits the language of the legislation, is
unclear. Furthermore, no pathway is necessarily more immune than the
others to such disruptive conditions as: high principal turnover rates,
limited professional development resources, or planned or unplanned
school restructuring. And, as previously described under state and
district conditions, compared to the first two pathways, the third
pathway does not necessarily make reform any more immune from shifts in
state or district priorities, either.

The potential insight presented by identifying these three pathways may
be magnified, not by comparing the three pathways, but by hypothesizing
the conditions when all three may be absent. A few of the 18 schools did
not attend explicitly to the nine CSRD components, did not adopt a
comprehensive method, and were not driven to strategic and
performance-related planning by state or district policies. These
schools therefore pursued none of the three pathways, and implementation
of reform lagged or was even absent.

**3.5 SUSTAINING COMPREHENSIVE SCHOOL REFORM**

The experiences of the 18 schools in the Field-Focused Study also
provide some clues regarding the sustainability of reform beyond the
period of CSRD funding. To be recalled is that the CSRD program consists
of three-year awards, of modest size, and are intended to serve (only)
as a catalytic force in stimulating comprehensive reform at a school.
Using these awards to implement short-lived changes---i.e., as in
supporting a special "project"---does not fulfill the basic CSRD
mission. Rather, schools must demonstrate a more sustained set of
changes. However, neither the original legislation nor ED define the
exact nature of the changes to be associated with sustaining a
comprehensively reforming school.

One implicit view, based mainly on the experiences of the New American
Schools initiative, holds that the central changes to be sustained
should be the practices associated with the research-based method
supported by CSRD (Component 1). An alternative view is that
comprehensive reform, though embracing a research-based method, also
transcends it. Successful sustainability would therefore not be
associated with the use of any particular method or combination of
methods. A successfully reforming school may implement one or a
combination of methods for a period of time and then later transition to
other methods---but the transitions should reflect a progression toward
continued school and student improvement rather than the "churning" of
innovative practices. These two views led to two different
interpretations of sustainability.

To succeed according to the first view, a school must identify a
continuing source of funds for a specific method, because the use of the
method usually requires some annual expenditure, whether: a) to support
the work of the external developer, or b) to provide professional
development opportunities for training new staff and refreshing existing
staff about the method. Although external developers reduce these annual
costs as a school gains experience with their method over time, none of
the 18 schools had unequivocally identified such ongoing sources of
support for these activities. Title I funds were the most common
candidate, but most schools could not be confident about making
discretionary commitments in a context dominated by district or state
budget cutbacks. At the same time, 10 of the 18 schools (Schools A, C,
F, G, I, J, K, O, P, and Q) did indicate their intention to seek such
support from external developers for the coming school year
(2002-03).[^42] Most of these schools, however, were not likely to
support the method (or the needed professional development) at the same
level as under CSRD, so the quality of the continued method may be
diluted.[^43]

To succeed according to the second view, a school substituting for its
original CSRD-supported method should evidence a clear rationale related
to school or student performance. Such was the case with four schools
(Schools B, D, H, and N). For example, the faculty at one school (School
B) that was otherwise doing well in implementing the CSRD-supported
method decided not to proceed to the next phase, which required adoption
of a block schedule---because of reports that schools with block
schedules had done less well on state assessments than had schools
without block schedules. A second school (School H) had already started
using a district-developed curriculum aligned with the state
assessment,[^44] instead of the original CSRD method. At the other two
schools (Schools D and N), the districts had developed their own
curricula that were believed to be more aligned with state assessments
than the CSRD-supported method. Either school was free to continue using
the original method, but the districts' resources were devoted to
supporting their own curricula and related professional development, and
a strong possibility is that the two schools will transition to the
district curricula.

Pursuing the second view a step further raises the question of whether
any of the original 10 schools continuing the CSRD-supported method
might have been doing so in the face of an emerging and contrary
rationale. Three of the 10 schools (Schools I, J, and P) may be
vulnerable to this jeopardy. Two (Schools I and J) were continuing
despite the lack of centrality of their method to their districts'
priorities; continuation may depend on the identity of a new principal,
yet to be appointed at either school. The third (School P) was due to be
merged with another school in 2003-2004, raising uncertainties because
the other school's principal already has been announced as the principal
of the merged entity.[^45]

Exhibit 3-4 places the 18 schools into a sustainability framework that
accommodates both views. The most lenient condition occurs by accepting
both views. Under this condition, 14 of the 18 Field-Focused Study
schools were exhibiting some level of

**Exhibit 3-4**

**SUSTAINABILITY OF CSRD AT THE 18 SCHOOLS**

![](media/image6.wmf){width="5.822916666666667in"
height="3.365972222222222in"}

sustainability by the time their CSRD awards were ending. Accepting only
the second view reduces the number to 11, and accepting only the first
view reduces it to 10.

The main barriers to sustainability continue to be the availability of
slack resources, especially in light of states' and districts' revenue
shortfalls in recent years. To sustain a reforming process, even when
existing resources have been sufficiently coordinated and targeted to
reform, still requires discretionary funds to support such essential
activities as: adequate professional development (including support for
teacher substitutes), especially in situations of high teacher turnover;
time for common planning periods or teachers' work on school leadership
teams; and support for external technical assistance. Though such needs
can be served with modest levels of funds, serving the needs is still a
discretionary activity that may have to be ignored if the core school
operations are underbudgeted.

**4. CONCLUSIONS FROM THE FIELD-FOCUSED STUDY**

The Field-Focused Study arrived at the following conclusions in relation
to the main research questions posed at the outset of the evaluation.

***Overall Implementation of CSRD***. Using a 47-point instrument, the
Field-Focused Study determined that 9 of the 18 schools were nearly
fully implementing CSRD and its 9 components, another 6 were partially
implementing CSRD, and the remaining 3 were judged to be poorly
implementing CSRD. On the whole, *CSRD implementation was therefore
uneven*.

***Three Conditions (or "Pathways") for Successful Implementation,
Including District and State Influences.*** The study identified three
different sets of conditions, or "pathways," that appeared to be
associated with the successful implementation of CSRD. The first pathway
is a *component-driven pathway*, whereby a school uses the 9 CSRD
components to guide the development and implementation of a
comprehensive reform. The second is a *method-driven pathway*, whereby
the school adopts and implements a comprehensive research-based method
that affects virtually all school operations and whose successful
implementation substitutes for the need for any independent articulation
of the 9 CSRD components. The third is a *vertical-driven pathway*,
whereby a school articulates and pursues the needed comprehensive
strategies as a result of state and district requirements involving the
setting of standards, use of appropriate assessment tools, and required
and aligned district- and school-based strategic planning and
improvement plans to meet performance standards.

No single pathway was considered the "best" or preferred pathway, and no
pathway was necessarily more immune than the others to such disruptive
conditions as: high principal turnover rates, limited professional
development resources, or planned or unplanned school restructuring.

***Sustainability of Comprehensive School Reform***. Neither the
original legislation nor ED define the exact nature of a school's
changes to be associated with sustaining a comprehensively reforming
school beyond the three-year CSRD award period. As a result, the
Field-Focused Study examined two different views of sustainability and
judged the 18 schools according to both.

The first implicit view, based mainly on the experiences of the New
American Schools initiative, holds that the central changes to be
sustained should be the practices associated with the
originally-supported research-based method. An alternative view is that
comprehensive reform, though embracing a research-based method, also
transcends it. By this second view, successful sustainability would not
necessarily be associated with the continued use of any particular
method but could involve transitions from one research-based method to
another, over time. The transitions would have to reflect a progression
toward continued school and student improvement rather than the
"churning" of innovative practices.

Using the most lenient benchmark and accepting either of these two views
as a criterion for assessing sustainability, 14 of the 18 schools were
exhibiting a promising level of sustainability by the time their CSRD
awards were ending. Accepting the "comprehensive reform" view alone
reduces the number to 11; accepting the "research-based method" view
alone reduces the number to 10.

The main barriers to sustainability continued to be the availability of
slack resources, especially in light of states' and districts' revenue
shortfalls in recent years. To sustain a reforming process, even when
existing resources have been sufficiently coordinated and targeted to
reform, still requires discretionary funds to support such essential
activities as: adequate professional development (including support for
teacher substitutes), especially in situations of high teacher turnover;
time for common planning periods or teachers' work on school leadership
teams; and support for external technical assistance. Though such needs
can be served with modest levels of funds, serving the needs is still a
discretionary activity that may have to be ignored if the core school
operations are underbudgeted. Thus, the sustainability of comprehensive
school reform is still questionable, given the current fiscal climate.

**5. REFERENCES**

Archibald, Sarah, and Allan Odden, ***A Case Study of Resource
Reallocation to Implement a Whole School Reform Model and Boost Student
Achievement: Parnell Elementary School***, Consortium for Policy
Research in Education, Philadelphia, PA, March 2000.

Berends, Mark, Susan Bodilly, and Sheila Nataraj Kirby, ***Reforming
Whole Schools: Challenges and Complexities***, 2002.

Berends, Mark, Joan Chun, Gina Schuyler, Sue Stockly, and R.J. Briggs,
***Challenges of Conflicting School Reforms: Effects of New American
Schools in a High-Poverty District***, RAND, 2002.

Berends, Mark, and Sheila Nataraj Kirby, ***Monitoring the Progress of
Title I and CSRD Schools: RAND's Analysis Plan***, proposal submitted to
Westat and the U.S. Department of Education, August 1999.

Bodilly, Susan, "The Evolution and Future of Comprehensive School
Reform," transcript from Education Policy Forum Luncheon Series,
National Clearinghouse for Comprehensive School Reform, Washington, DC,
February 9, 2001.

Bodilly, Susan, ***Lessons from New American Schools' Scale-Up Phase:
Prospects for Bringing Designs to Multiple Schools***, The Rand
Corporation, Santa Monica, CA, 1998.

Borman, Geoffrey D., "Title I: The Evolving Research Base," ***Journal
of Education for Students Placed at Risk**,* 2000, 5(1&2):27-45.

Cohen, Michael, "Effective School: Accumulating Research Findings,"
***American Education***, Jan./Feb. 1982, 18:13-16.

Consortium for Policy Research in Education, "States and Districts and
Comprehensive School Reform," ***CPRE Policy Briefs***, May 1998, RB-24.

COSMOS Corporation, ***First-Year Observations from the Field-Focused
Study of the Comprehensive School Reform Demonstration Program***,
submitted to Planning and Evaluation Service, Office of the
Undersecretary, U.S. Department of Education, December 2001.

COSMOS Corporation and The McKenzie Group, Inc., ***Funded by the U.S.
Department of Education Studies of Comprehensive School Reform and
CSRD***, Bethesda, MD, September 2001.

COSMOS Corporation, ***Evaluation Design for Component I***, Bethesda,
MD, May 2000.

Datnow, Amanda, and Sam Stringfield, " Working Together for Reliable
School Reform," ***Journal of Education for Students Placed at Risk***,
2000, 5(1&2):183-204.

Desimone, Laura, ***Making Comprehensive School Reform Work***, ERIC
Clearinghouse on Urban Education and the Institute for Urban and
Minority Education, New York, NY, May 2000.

Doherty, Katherine M., ***Early Implementation of the Comprehensive
School Reform Demonstration (CSRD) Program***, Planning and Evaluation
Service, U.S. Department of Education, Washington, DC, Sept. 2000.

Edmonds, Ronald, "Effective Schools for the Urban Poor," ***Educational
Leadership***, October 1979, 37:15-24.

Fermanich, Mark, Allan Odden, and Sarah Archibald, ***A Case Study of
District Decentralization and Site-Based Budgeting: Cordell Place School
District***, Consortium for Policy Research in Education, Philadelphia,
PA, March 2000.

Loveless, Tom, ***How Well Are American Students Learning? With Sections
on Arithmetic, High School Culture, and Charter Schools***, Brown Center
on Education Policy, The Brookings Institution, Washington, DC,
September 2002, Vol. 1, No. 3.

New American Schools, ***Working Toward Excellence: Examining the
Effectiveness of New American Schools Designs***, no place of
publication, February 1999.

Odden, Allan, and Sarah Archibald, ***A Case Study of Resource
Reallocation to Reduce Class Size, Enhance Teacher Planning Time, and
Strengthen Literacy: Clayton Elementary School***, Consortium for Policy
Research in Education, Philadelphia, PA, March 2000.

Odden, Allan, Sarah Archibald, and Anita Tychsen, ***Hollister
Elementary School: A Case Study of Resource Reallocation***, Consortium
for Policy Research in Education, Philadelphia, PA, 1999a.

Odden, Allan, Sarah Archibald, and Anita Tychsen, ***Farnham Elementary
School: A Case Study of Resource Allocation***, Consortium for Policy
Research in Education, Philadelphia, PA, 1999b.

Puma, Michael J., and Darrel W. Drury, ***Exploring New Directions:
Title I in the Year 2000***, National School Boards Association,
Alexandria, VA, 2000.

Ross, Steven M., et al., ***Two-and Three-Year Achievement Results on
the Tennessee Value-Added Assessment System for Restructuring Schools in
Memphis***, University of Memphis, Memphis, TN, June 1999.

Shelley, Fred M., "Education Policy and the104th Congress," in Lynn A.
Staeheli, Janet E. Kodras, and Colin Flint (eds.), ***State Devolution
in America: Implications for a Diverse Society**,* Sage Publications,
Thousand Oaks, CA, 1997, pp. 221-232.

Slavin, Robert E., and Nancy A. Madden, "Roots & Wings: Effects of
Whole-School Reform on Student Achievement," ***Journal of Education for
Students Placed at Risk***, 2000, 5(1&2):109-136.

Southwest Educational Development Laboratory, ***CSRD Awards
Database***, 2002.

Stringfield, Sam, et al., ***Urban and Suburban/Rural Special Strategies
for Educating Disadvantaged Children: Final Report***, U.S. Department
of Education, Washington, DC, 1997.

The Charles A. Dana Center, ***Hope for Urban Education:*** ***A Study
of Nine High-Performing, High-Poverty, Urban Elementary Schools*,** The
University of Texas at Austin, 1999.

U.S. Congress, ***Fiscal Year 1998 Appropriations Act for the U.S.
Department of Education, P.L. 105-78***, 1998.

U.S. Congress, ***Conferees' House Report No. 105-309,*** leading to the
enactment of P.L. 105-78.

U.S. Department of Education, ***Update to Guidance on the Comprehensive
School Reform Demonstration Program***, Washington, DC, August 11, 2000.

U.S. Department of Education, ***Guidance on the Comprehensive School
Reform Demonstration Program***, Washington, DC, March 13, 1998.

Yin, Robert K., "Rival Explanations as An Alternative to Reforms as
'Experiments,'" in Leonard Bickman (ed.), ***Validity and Social
Experimentation: Donald Campbell's Legacy***, Sage Publications,
Thousand Oaks, CA, 2000, pp. 241-268.

Yin, Robert K., ***Case Study Research: Design and Methods***, Sage
Publications, Thousand Oaks, CA, 2003, third edition.

[^1]: ^?^The Comprehensive School Reform Demonstration (CSRD) Program is
    the name of the education reform grant program as it was created in
    1998. The program was subsequently renamed the Comprehensive School
    Reform (CSR) Program in P.L. 107-110, the *No Child Left Behind Act
    of 2001* signed by the President in January 2002. Because the
    Field-Focused Study was commissioned to study the first two cohorts
    of CSRD schools (1998-1999 and 1999-2000 grantees), this final
    report will refer to the program as CSRD.

[^2]: ^?^The program was initially established through appropriations
    language, not authorizing legislation (Appropriations Act for the
    U.S. Department of Education, P.L. 105-78). In subsequent years,
    this appropriations language continued to be used to authorize the
    program, until the enactment of P.L. 107-110.

[^3]: ^?^The CSRD program called for the implementation of nine reform
    components (referred to throughout this report). The redesigned CSR
    program under P.L. 107-110 now calls for eleven components. The two
    new components cover "support to the school's staff" and the use of
    a research-based method (based on comparison group designs or on
    other strong evidence of improved student achievement).

[^4]: ^?^The original congressional legislation listed 17 specific
    methods as examples, although schools were not limited to adopting
    these methods.

[^5]: ^?^Funds appropriated under the Title I program are allocated to
    States based on each State's share of Title I Basic Grants in the
    previous fiscal year and may be used to support schools eligible to
    receive funds under Part A of Title I. Funds under FIE are allocated
    to States based on each State's share of school-aged children, and
    may be used to support any school, including those not eligible to
    receive funds under Part A of Title I. Under both the Title I and
    FIE CSRD programs, ED was able to reserve up to 1 percent of the
    total amount appropriated for grants to outlying areas and the BIA,
    and up to 1 percent for national evaluation activities.

[^6]: ^?^A study of 40 responding states (representing \$103 million of
    the original \$145 million in federal funds awards in FY1998)
    revealed that 3,001 school applications had been received from 1,088
    school districts and that the average CSRD award had been \$68,000
    per school per year (New American Schools, 1999).

[^7]: ^?^The states do not necessarily make all of their subawards in
    the same academic year, and states also are permitted to set aside
    up to five percent of the funds for state administration,
    evaluation, and technical assistance. States also have the option to
    set aside award money for use by districts, and this may or may not
    be reflected in calculations of the average school award.

[^8]: ^?^The present report, however, is not the occasion for a
    comprehensive review of this literature.

[^9]: ^?^Throughout these policy transitions, an important undercurrent
    continued to be that "teachers matter" (Puma and Drury, 2000).
    Operationally, the assumption has been that changed vision and
    school-based planning will lead to changed classroom practices. All
    these practices will then produce the desired change (and
    improvement) in student performance. That "teachers matter"
    therefore suggests that policymakers need to focus on
    classrooms---the "operational core of education systems" (Puma and
    Drury, 2000, p. 33).

[^10]: ^?^Six of these eight methods were listed among the CSRD
    legislation's original 17 methods. For a crosswalk between both
    groups and the top 25 methods or practices adopted by the first
    cohort of CSRD schools, see COSMOS, May 2000, Exhibit 1.

[^11]: ^?^One criticism is that very few of the research-based methods
    "\...have been the subject of high quality evaluations that justify
    confidence in their effectiveness," and that in addition, "\...the
    few models that do have a substantial amount of research\...are
    models for which evaluations were conducted mainly by the design
    team, not objective third-party evaluators" (Desimone, 2000, p. 37).

[^12]: ^?^Most of the studies are independent studies, totally unrelated
    to the National Evaluation of the CSRD Program. Further, many of the
    studies are on "comprehensive school reform," which may have been
    put into place quite independent of the CSRD program. In this sense,
    the bulk of the studies should not be considered evaluations of the
    CSRD program. For instance, eight of the studies are evaluations of
    individual research-based methods. Many of the studies are being
    conducted by the U.S. Department of Education's (ED) Regional
    Educational Laboratories and supported by ED's Office of Educational
    Research and Improvement (OERI) or Policy and Program Studies
    Service.

[^13]: ^?^Unfortunately, neither of these issues has been given
    sufficient attention in the existing literature (e.g., Datnow and
    Stringfield, 2000).

[^14]: ^?^For instance, the Southwest Educational Development Laboratory
    (SEDL) compiled baseline information on all CSRD schools. SEDL's
    database provided the sampling frame for selecting the sample of
    schools for the Field-Focused Study, to assure that the selected
    schools represented the entire universe. A second project was a
    large-scale sample survey of CSRD schools conducted in 1998-99,
    1999-00, and 2000-01 by Westat. The survey was included as part of
    the broader National Longitudinal Survey of Schools (NLSS), also
    designed by Westat, that followed a longitudinal sample of Title I
    schools over that same time period. Principals and up to six
    teachers were surveyed in the CSRD sample of schools as well as the
    NLSS sample, and comparable data were collected from both sets of
    schools. The 1998-99 sample of CSRD schools consisted of 318
    schools. In 1999-00, the CSRD sample was augmented by 725 schools.
    The data on the CSRD and NLSS schools were analyzed by RAND.

[^15]: ^?^Investigators assessing a directly comparable initiative---the
    scale-up phase of the New American Schools (NAS)---arrived at the
    same methodological conclusion, even though NAS funding levels were
    higher than CSRD funding. The investigators argued that the
    interventions and research questions called for "\...a *replicated
    case-study* approach, with the unit of analysis being the
    implemented school" (Bodilly, 1998, p. 25). \[Emphasis added.\]

[^16]: ^?^Any reform undertaking should be proceeded by a needs
    assessment; and, in fact, some of the externally developed methods
    called for the conduct of a needs assessment prior to adoption.
    Without a needs assessment, it is unlikely that a school would be
    able to properly design a comprehensive reform, or know which method
    to select (Datnow and Stringfield, 2000). The Field-Focused Study
    did not address the topic of needs assessments because it was not
    one of the 9 CSRD components. However, the evaluation team
    subscribes to the research that needs assessments are a necessary
    step in reform efforts, and believes this topic should be addressed
    in future research.

[^17]: ^?^For illustrative purposes, the arrows have been shown in a
    linear sequence that does not appreciate the full dynamics of events
    in real life. However, such dynamics were not ignored in the data
    collection for the Field-Focused Study.

[^18]: ^?^A common misinterpretation in using the rivals method is that
    the rival is perceived to be that "no positive outcome occurred."
    Key to avoiding this misinterpretation is that the method calls for
    defining rival *explanations* (or hypotheses), not rival outcomes.

[^19]: ^?^The clearance came just in time to schedule and conduct the
    first round of site visits in the fall and winter of 2000-01, which
    also included a final training session for the teams held in early
    November 2001. Thus, the teams had to scramble during these few
    months to cover all the designated schools.

[^20]: ^?^This methodology proved to be extremely effective as the
    Field-Focused Study progressed. Site visit teams were able to
    crosswalk information and data gathered at schools within the same
    district, and investigate reasons for missing information. For
    example, staff from one school frequently referenced another
    districtwide research-based method or strategy they credited with
    allowing them to continue reform efforts once CSRD funding ended.
    When no reference to that initiative was made at the second school
    in that district, the team had enough contextual information to ask
    probing questions and learn that staff at the second school were
    struggling with internal support issues, as well as potential
    reorganization, and had not bought into the new method.

[^21]: ^?^Three of the five schools had received no-cost extensions
    allowing them to carry on some CSRD activities for another semester
    or year.

[^22]: ^?^ Appendix D contains short summaries of these reports.

[^23]: ^?^Because the CSRD legislation called for schools to employ
    research-based methods and strategies that have been successfully
    replicated, most schools in the first and second cohort adopted
    externally developed methods. While locally developed methods could
    have been entirely developed within the schools, the likelihood
    (confirmed during the Field-Focused Study) was that local methods
    were developed externally.

[^24]: ^?^The scale is derived from Bodilly (1998). To arrive at a
    score, field teams were instructed to triangulate data from three
    sources: interviews of school staff; direct observations of
    classroom and school behavior; and consultations with the originator
    or "developer" of the method (who was available for most of the
    schools). The five-point scale does not fully reflect the complexity
    of method implementation: methods can be implemented 1) fully in
    some classrooms but not at all in others; and 2) partially in some
    or all classrooms. Further, different components of a method can be
    implemented or not implemented (and in some classrooms but not
    others).

[^25]: ^?^The evaluation team became aware of the switch too late to
    change the final sample of schools to be studied. As a result, the
    final sample, originally to include a pair of *Co-NECT* schools, had
    one *Co-NECT* and one *SIM*.

[^26]: ^?^Contrary to School N's usage, schools adopting only the
    reading component generally report that they have adopted *Success
    for All.* As a result, the final sample---intending to have one pair
    of *Success for All* (only) and one pair of dual *SFA-Roots & Wings*
    combined methods---then had three *Success for All* (only) schools
    and one school using a combination of the *Success for All* and
    *Roots & Wings* methods.

[^27]: ^?^Most often, the information came as a result of a face-to-face
    meeting with the developer's field representative who was providing
    external technical assistance to the school. However, in other cases
    the team either contacted the developer by telephone or reviewed a
    formal report submitted by the developer to the school.

[^28]: ^?^School R's CSRD application had indicated that, of the
    component practices comprising its whole "model," the adherence to
    state standards represented the research-based component of the
    entire "model." The state standards go beyond a mere listing of
    standards and specify a profile of learning---defining what students
    should know, understand, and be able to demonstrate to a high level
    of achievement. Students must then satisfactorily complete a
    state-prescribed classroom assessment for each standard, to show
    that they have met the standard. However, schools and teachers are
    to define how students are taught and the specific assignments they
    are required to complete. In this sense, a case can be made that the
    state's standards do provide guidance similar to that of a
    research-based method.

[^29]: ^?^Each classroom observation involved the use of a formal
    classroom observation instrument (included in the case study
    protocol---an attachment to Appendix A).

[^30]: ^?^Not considered sufficient evidence of a comprehensive design
    was sheer existence of the school's original CSRD application alone.
    All of the applications address the nine components because states
    required their districts and schools to do so. However, addressing
    these components did not necessarily mean that they had been
    adequately operationalized (and, in some cases the application had
    been totally prepared by the district and therefore may not have
    become a salient part of the school's understanding of CSRD). To be
    credited with pursuing a comprehensive design, schools had to show
    that they were actually using their CSRD applications in planning
    and implementing reform, and further that the staff interviewed
    could express some awareness and understanding of the comprehensive
    nature of the activities.

[^31]: ^?^By design and instruction from the U.S. Department of
    Education, and consistent with CSRD program guidance, the
    Field-Focused Study deliberately defined "external" support and
    assistance to exclude assistance by the district.

[^32]: ^?^This does not necessarily mean that the funds went directly to
    the developer. For instance, funds might have been used to pay for
    teacher substitutes so that the teachers could participate in the
    developer's workshops.

[^33]: ^?^For instance, no references during the site visits were made
    to assistance from the U.S. Department of Education's regional
    educational laboratories, which might have been in a position to
    deal with comprehensive school reform and not just the
    implementation of a specific research-based method.

[^34]: ^?^Exhibit 2-8 merely indicates the existence of these goals and
    benchmarks, but does not list them. The field teams collected the
    actual lists, which are contained in the teams' archival databases.
    The complete lists were too lengthy to include in this report, as
    most of them address target scores or percentage improvements for
    each grade level, for different academic subjects, and, in some
    cases, for different annual periods of time.

[^35]: ^?^Section 3 discusses sustainability more broadly.

[^36]: For objective information, examples in the instrument included
    (see Appendix E): 2.1) the existence of a written design or plan for
    school reform; 2.2) the contents of the plan; 3.2) the range of
    professional development days required or taken by the average
    teacher per year; and 4.1) the number of academic subjects covered
    by measurable goals and benchmarks. For subjective information,
    examples of items included (also see Appendix E): 1.1) the
    implementation score for the research-based component; 3.1) whether
    professional development had a strong content focus; 5.3)
    interviewees voicing strong support or enthusiasm for reform; and
    6.2) the level of parental involvement.

[^37]: An analyst independently read all of the case reports and ranked
    the 18 schools according to their level of implementation. In
    arriving at these judgments, the analyst deliberately avoided
    tallying the nine components, instead assuming a holistic
    perspective. The analyst was "blind" to the ratings that had been
    made by using the 47-point instrument. The analyst's findings,
    together with the rationale for the holistic judgments, is found in
    Appendix F.

[^38]: ^?^The 9-6-3 breakdown happens to be the same as that found for
    implementing the research-based method (see Sect. 2.1.1). However,
    the specific schools in each grouping were not synonymous and the
    measure for CSRD implementation is not synonymous with the
    implementation of the research-based method.

[^39]: ^?^In turn, the brief summaries are based on 18 in-depth and
    detailed cases reviewed by each school and available separately from
    COSMOS (see www.cosmoscorp.com).

[^40]: ^?^ The Field-Focused Study deliberately sought to identify
    implementation conditions substantively related to comprehensive
    school reform. The study did not attempt to examine more general
    implementation conditions, such as principal leadership, sufficiency
    of resources, and absence of turnover or other disruptive effects.
    These general conditions remain important, but highlighting them
    would not have contributed to understanding CSRD, which was taken as
    the main objective of the current study.

[^41]: ^?^The distinction between the importance of method vs. program
    has continued to evolve over the CSRD program period. This study was
    designed to address the first two cohorts of CSRD schools, when such
    a distinction was not as evident.

[^42]: ^?^One of these schools (School K) may only have been using the
    CSRD-supported method because of a state requirement to adopt the
    method---because the school had been rated as a school "in need of
    improvement." Whether the school will continue using the method if
    the school sufficiently improves and the state requirements are
    lifted remains to be seen. Alternatively, if the school fails to
    improve sufficiently, other sanctions may be applied and the
    continued use of the method also would be unclear.

[^43]: ^?^Of the 10 schools, two schools (Schools I and O) were in CSRD
    Cohort 1 (whose award period was 1998-99 to 2000-01), so they
    already had shown an ability to continue the use of the method in
    their first "post-CSRD" year. Of the other three schools in Cohort
    1, two (Schools E and R) had been rated as being the lowest in
    implementing CSRD in the first place (making the issue of
    sustainability moot). The status of the one remaining school in the
    cohort (School N) is discussed elsewhere in the text.

[^44]: ^?^Whether district-developed curricula can be considered a
    "research-based" method was beyond the scope of the Field-Focused
    Study. The districts' main claim is the alignment between the
    substance of the curricula and that of the state assessments.

[^45]: ^?^The other school happens also to be one of the 18 in the
    Field-Focused Study (School B).