Archived Information

  -----------------------------------------------------------------------
  [Planning and Evaluation Service]{.smallcaps}

  -----------------------------------------------------------------------

![](media/image1.jpeg)

**[Comprehensive Regional Assistance Centers Program:]{.smallcaps}**

**[Final Report on the Evaluation]{.smallcaps}**

**[Volume I]{.smallcaps}**

[2000]{.smallcaps}

![](media/image2.wmf){width="1.5152777777777777in"
height="1.632638888888889in"}

+----------------------------------+-----------------------------------+
| **[U.S. Department of            | **[Office of the Under            |
| Education]{.smallcaps}**         | Secretary]{.smallcaps}**          |
|                                  |                                   |
| **[Doc. #2000-19]{.smallcaps}**  |                                   |
+----------------------------------+-----------------------------------+

COMPREHENSIVE REGIONAL ASSISTANCE CENTERS PROGRAM:

FINAL REPORT ON THE EVALUATION

Volume I

Katrina G. Laguarda

Jeanine L. Hildreth

Catherine T. Kelliher

Derek L. Riley

Karen P. Walking Eagle

Ellen M. Pechman

Policy Studies Associates

Washington, D.C.

Prepared for:

U.S. Department of Education

Office of the Under Secretary

2000

This report was prepared for the U.S. Department of Education under
Contract Number EA94053010. The views expressed herein are those of the
contractor. No official endorsement by the U.S. Department of Education
is intended or should be inferred.

**U.S. Department of Education**

Richard W. Riley

*Secretary*

**Office of the Under Secretary**

Judith A. Winston

*Under Secretary(A)*

**Planning and Evaluation Service**

Alan L. Ginsburg

*Director*

**Elementary and Secondary Education Division**

Ricky T. Takai

*Director*

September 2000

This report is in the public domain. Authorization to reproduce it in
whole or in part is granted. While permission to reprint this
publication is not necessary, the citation should be: U.S. Department of
Education, Office of the Under Secretary, Planning and Evaluation
Service, Elementary and Secondary Education Division, *Comprehensive
Regional Assistance Centers Program: Final Report on the Evaluation.
Volume I*, Washington, D.C., 20202.

To order copies of this report, write:

ED Pubs

Education Publications Center

U. S. Department of Education

P. O. Box 1398

Jessup, MD 20794-1398;

via fax, dial (301) 470-1244;

or via electronic mail, send your request to: <edpubs@inet.ed.gov>.

You may also call toll-free: 1-877-433-7827 (1-877-4-ED-PUBS). If 877
service is not yet available in your area, call 1-800-872-5327
(1-800-USA-LEARN); Those who use a telecommunications device for the
deaf (TDD) or a teletypewriter (TTY), should call 1-800-437-0833.

To order online, point your Internet browser to:
[www.ed.gov/pubs/edpubs.html](http://www.ed.gov/pubs/edpubs.html).

This report is also available on the Department's Web site at:
www.ed.gov/offices/OUS/eval/.

On request, this publication is available in alternative formats, such
as Braille, large print, audiotape, or computer diskette. For more
information, please contact the Department's Alternate Format Center
(202) 260-9895 or (202) 205-8113.

ACKNOWLEDGMENTS

Many people contributed to the planning, development, and review of this
report. We would like to express our appreciation to all of the
Comprehensive Center customers and potential customers who participated
in interviews and who responded to surveys. Without their cooperation,
this evaluation would not have been possible.

We would also like to acknowledge the 15 Comprehensive Center directors
and their staffs, who assisted in all phases of the data collection by
providing Policy Studies Associates (PSA) with customer lists and
activity registration lists, by helping PSA site visitors gain access to
case study sites, and by hosting site visit teams at their centers.

We would like to thank the members of the technical working group for
the evaluation, who reviewed preliminary findings from early phases of
the project, as well as an early version of the final report, and made
valuable suggestions for further analysis and reporting:

> Joyce Benjamin, Oregon Department of Education
>
> Ronn Friend, Maryland Department of Education
>
> Patricia Gonzales, U.S. Department of Education
>
> Mary Jean Haberman Lopez, New Mexico Department of Education
>
> Trudy Hensley, Region XIV Comprehensive Center
>
> Cheryl Kane, U.S. Department of Education
>
> Diana Lam, Providence Public Schools
>
> William Morrill, Caliber Associates
>
> Mary Ramirez, School District of Philadelphia
>
> Maris Vinovskis, University of Michigan

At the U.S. Department of Education, we would like to thank the study
team who provided advice, critique, and practical help of various kinds
through all phases of the evaluation, as well as a careful review of
several drafts of this report. Susan Sanchez of the Planning and
Evaluation Service served as project officer for the evaluation. Other
present and former members of this team include: Patricia Gore, Milagros
Lanauze, Joyce Murphy, and Sharon Saez of the Goals 2000 Office; Larry
Cohen of the Budget Service; Mark Smith of the Office of General
Counsel; and Art Cole and Edith Harvey of the School Improvement
Programs Office.

Finally, we would like to thank our colleagues at PSA whose hard work
and skill made this study possible. Brenda Turnbull served as an advisor
to the evaluation through all stages of the project and reviewed the
final report. In addition to the authors of this report, members of the
study team at PSA included: Anne Castle, Lara Fabiano, Bruce Haslam,
Mary Leighton, Michael Rubenstein, and Jessica Wodatch. They were
assisted by Amy Hightower, David Kauffman, Dan Odenwald, Mari Parker,
and Catherine Zudack. Dan Odenwald, Elizabeth Shinkman, and Dwayne Smith
assisted with all phases of survey administration, follow-up, and
coding. Leslie Anderson, Joelle Gruber, Jennifer Suh, and Megan Welsh
assisted in the production of exhibits. Keon Ferguson, Benjamin
Lagueruela, and Kimberly Thomas provided production support.

**CONTENTS**

### Page

ACKNOWLEDGMENTS iii

CONTENTS v

EXHIBITS vii

THE COMPREHENSIVE REGIONAL ASSISTANCE CENTER PROGRAM AT A GLANCE xi

EXECUTIVE SUMMARY 1

I. INTRODUCTION [1](#refheading___toc469391849)3

> Program Mission and Goals [13](#__RefHeading___Toc469391850)
>
> Overview of the Comprehensive Center Program
> [15](#__RefHeading___Toc469391851)
>
> Evaluation Purposes and Methods [18](#__RefHeading___Toc469391852)
>
> Overview of the Report 20

II\. COMPREHENSIVE CENTER CUSTOMERS 23

> Who Are the Comprehensive Centers' Customers?
> [23](#__RefHeading___Toc469391857)
>
> Are Centers Serving Their Highest-Priority School and District
> Customers? [29](#__RefHeading___Toc469391860)
>
> Needs and Purposes: Why Do Customers Seek Assistance?
> [38](#refheading___toc469391862)
>
> Summary 46

III\. PORTFOLIO OF SERVICES [49](#__RefHeading___Toc469391865)

> What Are the Centers Major Strategies for Organizing and Delivering
> Assistance? [49](#__RefHeading___Toc469391866)
>
> What Services Do Customers Receive? [56](#__RefHeading___Toc469391872)
>
> How Do Services to State and Local Customers Differ?
> [66](#__RefHeading___Toc469391874)
>
> What Services Do the Centers Provide to the U.S. Department of
> Education? 71
>
> Summary 72

IV\. QUALITY OF COMPREHENSIVE CENTER SERVICES
[74](#__RefHeading___Toc469391880)

> Are Services Comprehensive? [75](#__RefHeading___Toc469391881)
>
> Is the Content of the Assistance Reported to Be Accurate and Grounded
> in Research? 81
>
> Are Centers Responsive to Customer Needs and Interests? 85
>
> Are the Scope and Intensity of Services Adequate? 89
>
> Do Center Staff Members Have Adequate Expertise? 90
>
> Are Center Customers Satisfied with Comprehensive Center Services? 92
>
> Summary 105

**CONTENTS (Continued)**

**[Page]{.underline}**

V. EFFECTS 107

> Effects on the Work of Individuals 107
>
> Effects on the Work of Organizations 112
>
> Assisting Customers to Respond to the Requirements of ESEA 129
>
> Enhancing State Capacity to Deliver Technical Assistance in Support of
> ESEA 131
>
> Effects on Teaching and Learning 131

Center Strategies for Assessing the Effects of Their Services 134

> Summary 137

VI\. COLLABORATION WITH COMPREHENSIVE CENTERS AND OTHER TECHNICAL

ASSISTANCE PROVIDERS 139

> Collaboration within the Comprehensive Center Network 139

Collaboration with Other Technical Assistance Providers 141

> Conditions that Encourage Collaboration 144

Conditions that Impede Collaboration 145

Summary 147

VII\. CONCLUSIONS 149

> Recommendations for Future Oversight of the Program 152

APPENDIX A: Comprehensive Center Mission Statements A-1

APPENDIX B: Comprehensive Centers, Grantee Organizations, and Partner
Organizations B-1

APPENDIX C: Survey Methods C-1

APPENDIX D: Survey Instruments D-1

APPENDIX E: Matrix of Case Study Sites E-1

APPENDIX F: Semi-Annual Progress Reports: Aggregated Data Tables

(December-January, 1999) F-1

EXHIBITS

I.1 Timeline of Key Events 15

I.2 Budget Allocations and Demographic Data for the 15 Comprehensive
Center Regions 17

II.1 Percent of Potential District Customers Nationwide Reporting That
They Had Heard

Of the Comprehensive Center Serving their Region and That They Had
Received

Services 25

II.2 Number of Organizations Receiving On-Site Professional Development
or

Consultation in 1998, and Number of Gatekeepers at those Organizations,

as Reported by the Comprehensive Centers 26

II.3 Professional Responsibilities of Gatekeepers 27

II.4 ESEA Programs Administered by SEA, District, and School Gatekeepers
28

II.5 Professional Responsibilities of Participants in Center Activities
29

II.6 Percent of Students Eligible for Free and Reduced-Price Lunches in
Districts

Served by Comprehensive Centers, as Reported by Gatekeepers 30

II.7 Percent of Target Student Populations in Districts Served by
Comprehensive Centers,

as Reported by Gatekeepers 31

II.8 Percent of Potential District Customers Reporting that They Have
Heard of the

Comprehensive Center Serving Their Region, by District-level
Demographics 33

II.9 Percent of Potential District Customers Reporting that Someone in
Their District

Had Received Technical Assistance from a Comprehensive Center Serving
Their

Region, by District-level Demographics 34

II.10 Percent of Potential District Customers Reporting on the Types of
Contact They Have

Had with a Comprehensive Center, by District Size 35

II.11 Percent of Students Eligible for Free and Reduced-Price Lunches in
Schools Served

by Comprehensive Centers, as Reported by Gatekeepers 36

II.12 Percent of Target Student Populations in Schools Served by
Comprehensive Centers,

as Reported by Gatekeepers 37

II.13 Percent of Participants Reporting Their Purposes in Participating
in a Center-

sponsored Activity, by Type of Organization 39

II.14 Topics for Which Customers Have the Greatest Technical Assistance
Needs 40

II.15 Reasons to Begin Using Comprehensive Center Products and Services
44

Exhibits (Continued)

II.16 Reasons Why Districts Have Not Made Use of Comprehensive Center
Products

and Services 45

II.17 Percent of Gatekeepers Who Reported Turning to Various Sources of
Assistance

"Always" or "Sometimes", by Type of Organization 47

III.1 Percent of Gatekeepers Reporting That the Center's Products and
Services Addressed

Certain Topics 58

III.2 Percent of Gatekeepers Reporting That the Center's Products and
Services Addressed

Certain Categorical Topics 59

III.3 Percent of Gatekeepers Reporting on the Number of Times That Their
Organization

Received Various Products or Services from a Comprehensive Center in a
Year,

by Type of Product or Service 62

III.4 Percent of All Participants Reporting That They Received Center
Follow-up

Assistance Once and More than Once 63

III.5 Other Sources of Assistance to Which Gatekeepers Were Referred by
the Center,

by Type of Organization 64

III.6 Percent of Gatekeepers Who Received Various Center Products or
Services Five or

More Times in a Year, by Type of Organization 68

III.7 Percent of Gatekeepers Who Were Very Satisfied with Their Access
to Center

Products or Services, by Type of Organization 69

IV.1 Gatekeepers' Ratings of the Quality and Utility of Comprehensive
Center Products

and Services 83

IV.2 Participants' Ratings of the Quality and Utility of
Center-Sponsored Activities 84

IV.3 Percent of Gatekeepers Reporting That They Contributed to the
Design or Planning

of Center Services Provided to Their Organization, by Type of
Organization 88

IV.4 Comprehensive Center Staffing Patterns: Number of Regular
Professional Staff and

Areas of Expertise 93

IV.5 Percent of Participants Rating the Quality and Utility of Center
Assistance as

"Excellent," by Type of Organization 97

IV.6 Percent of Gatekeepers Rating the Quality and Utility of Center
Products and

Services as "Excellent," by Volume of Services 99

Exhibits (Continued)

IV.7 Percent of Participants Rating the Quality and Utility of Center
Activities as

"Excellent," by Frequency of Telephone Follow-up 100

IV.8 Percent of Participants Rating the Quality and Utility of Center
Activities as "Fair" or

"Poor," by Frequency of Workplace Visits 102

IV.9 Bivariate Regression Analysis of the Relationship between
Gatekeepers Ratings of

Center Services and Products 104

IV.10 Bivariate Regression Analysis of the Relationship between the
Percentage of

Customers Receiving High-end Services and Size of Center Budget 105

V.1 Percent of Participants and Gatekeepers Reporting on Effects of
Center Assistance

on Their Work 108

V.2 Percent of Participants Reporting Effects of Center Assistance on
Their Work, by

Type of Organization 110

V.3 Percent of Participants Reporting That a Center-sponsored Activity
Had an Effect

on Their Work, by Frequency of Follow-up with Additional Workshops,
Training,

and Meetings 111

V.4 Percent of Participants Reporting That a Center-sponsored activity
Had an Effect

on Their Work, by Whether They Were Required to Participate or
Volunteered in

That Activity 113

V.5 Percent of Participants Reporting Effects on Their Own Work, by
Number of

Purposes for Participation 114

V.6 Percent of Gatekeepers and Participants Reporting on Effects of
Center Assistance on

the Way Their Organization Does Its Work 115

V.7 Percent of Gatekeepers Reporting the Types of Effects Center
Assistance Had on

Their Organization's Work, by Type of Organization 117

V.8 Percent of Participants Reporting the Types of Effects Center
Assistance Had on

Their Organization's Work, by Type of Organization 118

V.9 Percent of Gatekeepers Reporting Organizational Effects of Center
Assistance by

Volume of Services 120

V.10 Percent of Participants Reporting That a Center-sponsored Activity
Had an Effect on

Their Organization's Work, by Frequency of Follow-Up with Additional
Workshops,

Training, and Meetings 121

**Exhibits (continued)**

V.11 Percent of All Respondents Reporting Organizational Effects of
Center Assistance,

by the Number of Topics on Which They Received Assistance 122

V.12 Percent of Participants Reporting That a Center-sponsored Activity
Had an Effect on

Their Organization's Work, by Whether They Participated in That Activity
as a

Member of a Team or Group 123

V.13 Percent of Gatekeepers Reporting That Center Assistance Had an
Effect on Their

Organization's Work by Whether They Were Involved in the Design or
Planning of

Center Services 125

V.14 Percent of Participants Reporting That a Center-sponsored Activity
Had an Effect on

Their Organization's Work, by Whether They Contributed to the Design of
the

Activity 126

V.15 Percent of Participants Reporting Effects of the Work of Their
Organizations, by

Number of Purposes 127

V.16 Percent of Participants Reporting That a Center-sponsored Activity
Had an Effect on

Their Organization's Work, by Their Purposes in Participating 128

Insert Map

**The Comprehensive Regional Assistance Centers Program at a Glance**

The Comprehensive Regional Assistance Centers help states, school
districts, and schools meet the educational needs of children served
under ESEA. As of September 2000, 15 regional centers are being operated
under cooperative agreements by the following grantees. Telephone
numbers and Web addresses follow their names and locations.

**Region I (Connecticut, Maine, Massachusetts, New Hampshire, Rhode
Island, and Vermont):** New England Comprehensive Assistance Center,
Education Development Center, Inc., Newton, Mass., (800) 332-0226,
www.edc.org/NECAC/.

**Region II (New York):** New York Technical Assistance Center, the
Metropolitan Center for Urban Education at New York University, New
York, N.Y., (800) 4NYU-224 or (212) 998-5100,
www.nyu.edu/education/metrocenter/nytac/nytac.html.

**Region III (Delaware, District of Columbia, Maryland, New Jersey,
Ohio, and Pennsylvania):** Region III Comprehensive Center at the George
Washington University, Center for Equity and Excellence in Education,
George Washington University, Arlington, Va., (800) 925-3223 or (703)
528-3588, r3cc.ceee.gwu.edu.

**Region IV (Kentucky, North Carolina, South Carolina, Tennessee,
Virginia, and West Virginia):** Region IV Comprehensive Center at AEL,
Appalachia Educational Laboratory, Inc., Arlington, Va., (800) 755-3277,
www.ael.org/cac/.

**Region V (Alabama, Arkansas, Georgia, Louisiana, and Mississippi):**
Southeast Comprehensive Assistance Center, Southwest Educational
Development Laboratory, Metairie, La., (504) 838-6861 or (800) 644-8671,
www.sedl.org/secac/welcome.html.

**Region VI (Iowa, Michigan, Minnesota, North Dakota, South Dakota, and
Wisconsin):** Region VI Comprehensive Assistance Center, University of
Wisconsin-Madison, Madison, Wis., (888) 862-7763 or (608) 263-4220,
www.wcer.wisc.edu/ccvi/.

**Region VII (Illinois, Indiana, Kansas, Missouri, Nebraska, and
Oklahoma):** Region VII Comprehensive Center, University of Oklahoma,
Norman, Okla., (800) 228-1766 or (405) 325-1729, region7.ou.edu.

**Region VIII (Texas):** STAR (Support for Texas Academic Renewal)
Center, Intercultural Development Research Association, San Antonio,
Tex., (888) FYI-STAR, www.starcenter.org/.

**Region IX (Arizona, Colorado, Nevada, New Mexico, and Utah):**
Southwest Comprehensive Regional Assistance Center, New Mexico Highlands
University, Rio Rancho, N.Mex., (505) 891-6111,
www.cesdp.nmhu.edu/swcc/.

**Region X (Idaho, Montana, Oregon, Washington, and Wyoming):**
Northwest Regional Educational Laboratory's Comprehensive Center, Region
X, Northwest Regional Educational Laboratory, Portland, Ore., (503)
275-9500, www.nwrac.org.

**Region XI (Northern California):** Region XI Comprehensive Assistance
Center, WestEd, Oakland, Calif., (800) 645-3276, www.wested.org/cc/.

**Region XII (Southern California):** Southern California Comprehensive
Assistance Center, Los Angeles County Office of Education, Downey,
Calif., (562) 922-6343, sccac.lacoe.edu.

**Region XIII (Alaska):** Alaska Comprehensive Regional Assistance
Center, Southeast Regional Resource Center, Juneau, Alaska, (888)
43-AKRAC or (907) 586-6806, www.akrac.k12.ak.us.

**Region XIV (Florida, Puerto Rico, Virgin Islands):** Region XIV
Comprehensive Center, Educational Testing Service, Tampa, Fla., (800)
756-9003, www.ets.org/ccxiv.

**Region XV (Hawaii, American Samoa, Federated States of Micronesia,
Guam, Republic of the Marshall Islands, Commonwealth of the Northern
Mariana Islands, and Republic of Palau):** Pacific Comprehensive
Regional Assistance Center, Pacific Resources for Education and
Learning, Honolulu, Hawaii, (808) 441-1300, www.prel.org.

EXECUTIVE SUMMARY

Introduction

The Improving America's Schools Act (IASA) of 1994, which amended the
Elementary and Secondary Education Act (ESEA) of 1965, represents a
departure in federal education policy. The categorical programs funded
under the reauthorized ESEA are designed to work together to enable all
students, especially those who live in areas of poverty and have special
educational needs, to reach high and challenging academic standards.
Instead of operating separate, categorical services to support students'
"regular" educational program, the U.S. Department of Education
encourages states and districts to use ESEA programs to reinforce state
and community reform efforts geared to challenging state standards.

To support this shift in the implementation of federal education
programs, Congress replaced the U.S. Department of Education\'s existing
network of 48 categorically based ESEA technical assistance centers with
15 Comprehensive Regional Assistance Centers. The new Comprehensive
Centers were to pioneer a new form of comprehensive, cross-program
assistance to support standards-based reform as envisioned in other
sections of the law. This new vision of comprehensive assistance
emphasizes the dissemination of new ideas and practices that strengthen
teaching and learning for all children served under ESEA, over a more
narrow focus on helping grantees respond to the administrative or
programmatic requirements of a single categorical program.

As authorized under Title XIII of the ESEA, the Comprehensive Centers
provide intensive professional development, training and technical
assistance to states, local education agencies (LEAs), schools, American
Indian tribes, community-based organizations, and other ESEA grantees in
a long list of areas. They include: (1) improving the quality of
instruction, curricula, assessments, and other aspects of school reform;
(2) implementing effective schoolwide programs; and (3) meeting the
needs of children, especially children in high-poverty areas, migrant
children, immigrant children, limited English proficient (LEP) children,
neglected or delinquent children, homeless children, American Indian
children, and children with disabilities. Comprehensive Centers are
required to assign highest priority to serving: (1) Title I schoolwide
programs and (2) LEAs and Bureau of Indian Affairs (BIA)-funded schools
with the highest percentage or numbers of poor children. This specific
focus on high-poverty schools and districts is unique to the
Comprehensive Centers among ED's various technical assistance programs.

The U.S. Department of Education set regional boundaries and allocated
funds for individual Comprehensive Centers based on the concentrations
of Title I-eligible, limited English proficient, migrant, and American
Indian children in the area. In creating the Comprehensive Centers
program, Congress greatly expanded the range and scope of the centers\'
responsibilities relative to the work of the 48 categorically-organized
technical assistance centers that had served ESEA grantees before 1994.
At the same time, Congress drastically reduced the resources available
to support the new program by funding the Comprehensive Centers at
approximately half the level of the previous programs.

The evaluation of the Comprehensive Centers addressed two basic
questions: (1) How are the Comprehensive Centers fulfilling their
statutory mission, as embodied in Title XIII of the ESEA?; and (2) How
does the centers' work contribute to educational change and improvement
in the states, districts, and schools that they serve? To ensure the
quality of the Comprehensive Center program, Title XIII of ESEA requires
the U.S. Department of Education to conduct regular surveys of customers
to determine if they are satisfied with their access to and the quality
of Comprehensive Center services. The evaluation was also designed to
satisfy this requirement.

Data collection for the evaluation, conducted by Policy Studies
Associates (PSA) of Washington D.C., was carried out in three phases:

-   Phase 1 consisted of site visits to all 15 Comprehensive Centers,
    beginning in November 1998, interviews with professional staff at
    all of the organizations that contribute to the centers' work, and a
    review of key center documents.

-   Phase 2 consisted of three different surveys of customers and
    potential customers of the Comprehensive Centers, administered in
    spring and summer 1999. The first customer survey was directed to a
    random sample of 1,086 *gatekeepers* in states, districts, schools,
    and other organizations that received services from the
    Comprehensive Centers in 1998, drawn from customer lists compiled by
    the centers. (Gatekeepers are customers of the Comprehensive Centers
    who request or negotiate for services on behalf of the organizations
    they represent.) The second customer survey was directed to 1,123
    *participants* in a sample of 30 major Comprehensive Center
    activities, such as intensive professional development and model
    schools initiatives. (These customers are the end-users of
    Comprehensive Center assistance.) The third survey was directed to
    *potential customers* of the Comprehensive Centers in a nationally
    representative sample of 1,122 districts with poverty rates above
    the national median. This survey was sent to the district-level
    staff member most directly responsible for the administration of
    ESEA programs.

-   Phase 3 consisted of 15 case studies, conducted in spring and summer
    1999, of center services to states and local sites (both districts
    and schools), and eight case studies of collaboration among
    Comprehensive Centers and between the centers and other technical
    assistance providers. We selected case study sites where the effects
    of the technical assistance provided could reasonably be observed by
    spring or summer 1999.

The final evaluation report consists of three volumes. *Volume I:
Comprehensive Regional Assistance Centers Program: Final Report on the
Evaluation* synthesizes the information gathered by the evaluation.
*Volume II: Case Studies of Center Services to State and Local Sites and
Case Studies of Collaborative Activities* contains case studies of
center services to states, districts, and schools, and profiles of
collaboration among centers and other technical assistance providers.
*Volume III: Comprehensive Center Profiles* describes the mission and
goals, organization and staffing, needs assessment strategies, portfolio
of services, and collaboration activities with other technical
assistance providers, for each of the 15 centers.

Comprehensive Center Customers

**The Comprehensive Centers have reached a number of their potential
customers at the school, district, and state levels.** A review of the
entire portfolio of Comprehensive Center activities shows that the
centers, as a network, allocate resources in roughly equal proportions
to the state, district, and school levels.

-   **Most end-users of Comprehensive Center training and technical
    assistance are based in schools.** Two-thirds of all participants in
    a sample of 30 major Comprehensive Center initiatives were based in
    schools. Seventy percent of all school-based participants
    (accounting for half of all participants) reported that they were
    teachers.

-   **Sixteen percent of all districts with poverty rates above the
    national median received services from a Comprehensive Center in
    1998-99,** based on a nationally representative survey of 1,122
    potential district customers. An additional 26 percent reported that
    they had heard of the Comprehensive Centers but had not received
    services or were not sure if they had received services.

-   **The Comprehensive Centers provide services to SEAs in all 50
    states, the District of Columbia, the U.S. Virgin Islands, Puerto
    Rico, and nine Pacific entities.** In 1998, two-thirds of all
    state-level federal program managers reported that they had received
    services from the Comprehensive Centers.

The Comprehensive Centers are more likely to serve their
highest-priority school and district customers, targeting both
high-poverty districts and schools, and districts and schools serving
large numbers of English language learners, migrant students, and
American Indian students:

-   Nationwide, districts with high rates of poverty and districts with
    significant enrollments of LEP students, American Indian students,
    and migrant students were more likely to report that they had
    received services from the Comprehensive Center serving their
    region, compared with other potential district customers. For
    example, 40 percent of districts with LEP enrollments greater than
    25 percent reported that they had received services, compared with
    16 percent of all potential district customers.

-   Almost two-thirds of gatekeepers who represent districts that
    received services and three-quarters of gatekeepers who represent
    schools that received services reported that the majority of their
    students were eligible for free- or reduced-price lunches. (These
    gatekeepers are customers of the Comprehensive Centers who negotiate
    for services on behalf of the organizations they represent.) Almost
    a third of district-level gatekeepers and 38 percent of school-level
    gatekeepers reported very high concentrations of poverty, with 75
    percent or more of their students eligible.

```{=html}
<!-- -->
```
-   Sixty-nine percent of school-level gatekeepers reported that their
    school operated a Title I schoolwide program.

-   Almost half of gatekeepers in schools served by the centers reported
    that their school enrolled a significant proportion of limited
    English proficient students (that is, more than 10 percent of their
    total enrollment). Similarly, almost a third of gatekeepers reported
    that their schools had a significant proportion of American Indian
    students (more than 10 percent of their total enrollment).

**Gatekeepers seek out Comprehensive Center services because the
services are free, and because they have reason to believe that they
will be useful.** The reasons most commonly cited for seeking out
Comprehensive Center products and services were: (1) products and
services are free of charge (with 87 percent of gatekeepers reporting
that this was a \"very important\" or \"moderately important\" reason
for seeking assistance); (2) the centers have a reputation for providing
high-quality assistance (86 percent); and (3) products and services are
easily accessible (85 percent).

Survey data from potential district customers suggest that increased
marketing and outreach efforts would create additional demand for center
services. Among high-poverty districts that knew about the centers but
had not received services, the reasons most often cited for not making
use of center products and services were: (1) not receiving information
about center products or services (with 44 percent of districts
reporting that this was a reason for not using center services); (2) not
having time to learn about center products and services (36 percent);
and (3) not knowing whom to contact at the center (31 percent). Few
potential customers reported that they had avoided the centers because
they did not need their assistance or because they had a negative
impression of the centers\' work. Only one percent of potential
customers reported that they had requested assistance, but that the
center had not been able to provide it. These findings suggest that many
more districts would be interested in receiving center services, given
additional outreach efforts.

Portfolio of Services

**The Comprehensive Centers organize most of their work around a small
group of key initiatives that engage the centers in long-term
relationships with customers.** They can be classified into three broad
categories: (1) professional development initiatives, (2) model schools
and partnership sites, and (3) continuing consultation with states,
districts, and schools. These initiatives command most of the centers\'
staff time and other resources. In addition, most Comprehensive Centers
operate a "dual track" of services that includes a set of much
lower-intensity activities designed to reach a wider audience. These
lower-intensity activities include single, \"one-shot\" workshops and
dissemination of written materials.

**The Comprehensive Centers are fulfilling the intent of their
authorizing legislation to provide \"comprehensive\" assistance.** As
the reauthorized ESEA has sought to shift emphasis from the
implementation of parallel programs for special populations of students
to more comprehensive reforms, the aims of technical assistance have
shifted from strengthening the performance of single categorical
programs to supporting the work of entire schools or school support
systems. On surveys, gatekeepers most commonly reported that their
organization received assistance on topics usually associated with
comprehensive, standards-based reform as envisioned in ESEA. According
to gatekeepers who represent organizations that received services, the
topics most commonly addressed by those services were: (1) student
assessment (with 49 percent of all gatekeepers reporting that their
organization received assistance on this topic); (2) improving
curriculum and instruction in reading and language arts (49 percent);
(3) implementing schoolwide programs (48 percent); (4) challenging
standards and accountability (45 percent); and (5) analyzing student
achievement data and interpreting the results (43 percent).

## 

## Categorical assistance---including help in carrying out the provisions of specific ESEA programs and addressing the needs of specific student populations---is also a major focus of center services. Almost half of gatekeepers reported that their organizations received assistance in serving special student populations, with almost one-quarter receiving assistance for LEP students. About one-third of gatekeepers reported that their organization received assistance in responding to the provisions of at least one ESEA program. 

## 

Customer Satisfaction with Comprehensive Center Services

**Most Comprehensive Center customers gave the centers high ratings for
the accessibility, quality, and utility of their services.** This
finding is true for both gatekeepers, who are responsible for requesting
services from the Comprehensive Centers on behalf of their
organizations, and participants in center activities, who are the
end-users of Comprehensive Center assistance.

-   Eighty-five percent of gatekeepers reported that they were "very
    satisfied" or "moderately satisfied" with the accessibility of
    center products and services. Gatekeepers gave similar ratings for
    all types of services (satisfaction was lower with regard to the
    accessibility of center Web pages and electronic products).

-   Eighty-six percent of gatekeepers reported that they were "very
    satisfied" or "moderately satisfied" with the overall quality of the
    assistance they received. Gatekeeper ratings of quality were similar
    for all topics of assistance.

-   More than 75 percent of gatekeepers and participants gave the
    centers ratings of "excellent" or "good" on various dimensions of
    quality and utility.

Gatekeepers in high-poverty schools and districts are as likely to be
satisfied with center services as gatekeepers in lower-poverty schools
and districts.

**Some customers were less satisfied with the utility of Comprehensive
Center services, as related to their specific needs.** Both gatekeepers
and participants gave Comprehensive Center services the lowest ratings
for the extent to which their services responded to specific local
conditions or their own needs and interests:

-   24 percent of participants and 16 percent of gatekeepers rated
    center services \"fair\" or \"poor\" on the extent to which they
    responded to local conditions

-   16 percent of participants and 14 percent of gatekeepers rated
    center services \"fair\" or \"poor\" on the extent to which they
    addressed their needs and interests

-   16 percent of gatekeepers and 13 percent of participants rated
    services \"fair\" or \"poor\" on their ability to respond in depth
    to all of their questions and interests

A small proportion of respondents in our case study sites expressed
dissatisfaction that they were not receiving the same volume of
services, nor the same degree of specialized expertise, as they had
received from the categorical technical assistance centers operated by
ED before 1994.

**After their state education agency, gatekeepers report that they are
more likely to consult the Comprehensive Centers than any other source
of assistance.** Approximately two-thirds of gatekeepers report that
they \"always\" or \"sometimes\" turn to the Comprehensive Center for
help in areas that are important to them.

**On average, Comprehensive Centers with larger budgets generated higher
levels of customer satisfaction.** Customer satisfaction varied by
individual Comprehensive Center, and in most cases these variations were
quite pronounced. The study tested many possible hypotheses to explain
this variation. Only one variable appears to be consistently related to
various measures of customer satisfaction: the size of the center's
budget. Although it is not true that all large centers receive high
ratings of customer satisfaction and all small centers receive low
ratings, the relationship between size and satisfaction across the
entire network of 15 centers is strong and statistically significant.
Specifically, every \$100,000 increase in center budget corresponds with
a 2 percentage point increase in the number of gatekeepers reporting
that they are "very satisfied" with the overall quality and
accessibility of center services.

The explanation for this finding appears to be that larger centers, on
average, provide a higher volume of higher-intensity, more expensive
services to their customers. Centers with larger budgets tend to have
more resources available for each of the customers they serve, and
customers of larger centers do, in fact, report that they receive more
services. It appears that customers of larger centers are more satisfied
with the quality and accessibility of the services as a result.

Effects

# 

**Most Comprehensive Center customers report that the centers'
assistance has had an effect on their own work**. This finding holds
true for both gatekeepers, who are responsible for negotiating services
with the Comprehensive Centers on behalf of their organizations, and
participants, who are the end-users of Comprehensive Center assistance.

-   Eighty percent of participants and 77 percent of gatekeepers
    reported that they had gained new information as a result of center
    assistance.

```{=html}
<!-- -->
```
-   Seventy-two percent of participants and 61 percent of gatekeepers
    reported that they had incorporated something they had learned from
    the center into their work.

> **Individual effects varied among different groups of customers:**

-   Participants in center activities were more likely to report that
    the activity had an effect on the way they do their job if they
    received follow-up services from the center.

-   Participants who cited multiple purposes for participating in an
    activity were more likely to report that the activity had an effect
    on the way they do their job.

**Comprehensive Center customers were more likely to report that
assistance had generated awareness of new information within their
organizations than they were to report that assistance had prompted
changes in practice:**

-   More than three-quarters of participants and gatekeepers reported
    that center assistance generated awareness of new information within
    the organization.

-   Almost half of participants and gatekeepers reported that center
    services helped their organization carry out a major, planned
    improvement effort.

-   About a third of participants and gatekeepers reported that center
    services prompted their organization to initiate a new practice.

**Organizational effects varied significantly by the characteristics of
the service and the type of organization served:**

-   Gatekeepers whose organizations received a higher volume of services
    were more likely than those who received a lower volume of services
    to report all types of organizational effects.

```{=html}
<!-- -->
```
-   Schools were more likely than districts or states to report that
    Comprehensive Center services had changed teachers' classroom
    practices, improved teaching and learning for all students, and
    increased student achievement. This finding is explained by the fact
    that services to schools are more likely to be designed to directly
    affect classroom practice.

-   States were more likely than districts or schools to report that the
    centers' assistance had enhanced their organization's capacity to
    provide technical assistance in support of ESEA programs.

**Among our case study sites, federal program administrators maintained
that the Comprehensive Centers played a key role in assisting them to
respond to the requirements of the reauthorized ESEA.** In several
states respondents explained that center staff members had helped to
extend their capacity by filling in gaps left by the their own limited
staffing and resources\--by conducting training sessions on the state\'s
behalf or by helping with grant applications. At the local level,
Comprehensive Center assistance has supported schools throughout the
schoolwide planning process and in the development of written plans;
however, the assistance has had limited effect on the development of
local capacity in schools for planning on their own.

**Just over one-half of all center customers reported that the centers'
work had helped to increase teacher knowledge and skills, and just over
one-third reported that the assistance had resulted in a change in
classroom practices.** End-user participants in activities directly
targeted to

instruction were more likely to report effects on teaching and learning.
Changes in teaching and learning were a specific goal of center services
in only four of our local case study sites. In these sites, teachers
reported making changes in their classroom practices, though these
changes were limited in scope.

**Increases in student achievement were the least commonly reported
effect of Comprehensive Center services,** with about one-quarter of
both participants and gatekeepers reporting that the centers' work had
helped to improve student achievement. This finding is not surprising
when we consider that technical assistance activities have, at best, an
indirect effect on student learning. Indeed, participants in activities
specifically designed to improve teachers\' knowledge and skills were
more likely to report effects on student achievement as a result of
Comprehensive Center assistance. Among the four local case study sites
where services were specifically targeted at schools and classrooms, one
had data showing improvements in student outcomes. However, the
Comprehensive Center network is currently conducting an extensive
evaluation of the Reading Success Network (an intensive professional
development initiative focused on the diagnosis of and interventions for
reading difficulties in young children) across the 15 Comprehensive
Center regions that includes an examination of student outcomes.

Conclusions

By the spring of 1999, the end of their third full year of operation,
the Comprehensive Centers had achieved at least two important
milestones. First, they had developed a fairly coherent portfolio of
services that responded to the broad charge set out in their authorizing
legislation. Second, the centers had succeeded, in a relatively short
period of time, in cultivating a base of satisfied customers. A close
examination of the effects of Comprehensive Center assistance, as they
are reported by customers, confirms much of what we already know about
high-quality technical assistance and professional development:

-   Comprehensive Center technical assistance is more useful to
    customers if it is intensive and if it extends over time.

-   Comprehensive Center technical assistance is more useful to
    customers if it is tailored to address their needs and interests.

-   The effects of Comprehensive Center services are most apparent when
    services are intensive and targeted directly at the individuals and
    organizations expected to change.

The Comprehensive Centers face continuing challenges, however. The
centers' broad mandate and limited resources have forced them to make
choices about whom they will serve and the breadth

and depth of services they will provide. Comprehensive Center services,
at their present level of intensity, do not produce radical changes in
the behavior of organizations and individuals in any but a few
instances. Changes in teaching and learning are particularly limited.
This dilemma is not unique to the Comprehensive Centers program.
Instead, it reflects the great level of effort required to achieve ED's
goals for its technical assistance systems.

The key findings of this evaluation suggest that the operations of the
Comprehensive Centers could be modified to build on their early work:

-   **The Comprehensive Centers could focus attention on developing
    their capacity to respond to customers\' particular local conditions
    and customer needs and interests.** Although customers are generally
    satisfied with the content and format of Comprehensive Center
    assistance, survey results suggest that services could be even more
    closely tailored to address their particular questions, needs, and
    interests and local conditions. The Comprehensive Centers could
    place special emphasis on learning even more about particular
    customers\' needs and interests and fine-tuning their services to
    supply the support and expertise their customers require.

-   **The Comprehensive Centers could consider targeting their services
    even more carefully on organizations that can benefit most from
    their assistance.** Surveys of potential customers suggest that
    there is a sizable market for Comprehensive Center services. By
    stepping up their marketing activities, the centers may be able to
    locate schools and districts that are best able to benefit from
    their particular expertise, achieving the best fit between their
    particular portfolio of services and expertise and the needs and
    interests of the customers they serve. This strategy would require
    that the centers turn down requests for assistance in some cases.

-   **The Comprehensive Centers could develop further strategies for
    capacity building in states and districts.** Most Comprehensive
    Center services to states and districts aim to build their capacity
    to improve education programs in high-priority schools. In many
    cases, however, this capacity building consists of supplying
    additional staff to extend the state\'s or district\'s reach among
    schools, rather than adding knowledge or expertise not readily
    available within the organization. The Comprehensive Centers also
    need to consider ways that they can increase the knowledge and
    skills of SEA and LEA staff to enable them to serve schools more
    effectively.

-   **The Comprehensive Centers could build on opportunities to
    collaborate across the network on key initiatives like the Reading
    Success Network.** When collaboration among technical assistance
    providers is successful, Comprehensive Center staff members report
    that it enhances the quality of their services to the field. The RSN
    is a good example of a case where collaboration on a joint effort
    has allowed the entire network to benefit from products and
    expertise developed by individual centers. The Comprehensive Centers
    should continue to pursue similar opportunities in the future,
    especially with regard to current efforts to develop strategies for
    assisting low-performing schools.

ED can also take steps to strengthen its oversight of the Comprehensive
Center program:

-   **ED should continue to explore ways to build its capacity to
    provide effective assistance and support to the Comprehensive
    Centers.** In the past year, ED managers of the Comprehensive Center
    program have taken steps to improve communications between the
    centers and various ED program offices by hosting meetings at ED on
    various high-priority topics. ED should continue to seek ways to
    provide the centers with the information and access to program
    managers that they need to be responsive to ED\'s needs and
    purposes.

-   **ED should continue to work with the Comprehensive Centers to
    develop standard annual procedures for monitoring their work and for
    evaluating its effects.** The adoption of some standard annual
    evaluation procedures would allow the Comprehensive Centers to
    collect much better and more systematic data on the effects of their
    services on customers. ED should continue to work with the centers
    on refining and improving the uniform reporting format that they use
    in their semi-annual progress reports to ED. In addition, ED could
    work with the centers to develop standard instruments and data
    collection procedures for each of the 15 centers to use in assessing
    the impact of services on customers.

Although there are steps that the Comprehensive Centers and ED can both
take to strengthen existing services, the findings of this evaluation
indicate that, to have major effects on education change and
improvement, the Comprehensive Centers would have to work on a much more
intensive and costly scale. For the sake of refining the goals and
objectives of the Comprehensive Centers program so that the centers can
organize their work around objectives that are both important *and*
feasible, we make the following recommendations:

-   **ED and the Comprehensive Centers should continue to work together
    to clarify and refine the mission and goals of the program,
    especially with regard to the content and purposes of services
    provided.** As a new entity, each center has developed its own set
    of priorities and strategies for organizing services to its region.
    Now may be a good time for Comprehensive Center staff and ED
    managers to review those strategies with the aim of clarifying the
    nature and goals of "comprehensive" assistance.

-   **The Comprehensive Centers and ED should discuss goals for
    extending the centers' reach among district and school customers, as
    well as reasonable goals for the effects expected as a result of
    this effort.** There is a clear trade-off between the intensity and
    effectiveness of services and the extent of the centers' reach.

-   ED and the Comprehensive Centers should consider even stronger
    targeting of high-priority customers as a way of addressing the
    problem of reach. Now that the centers have succeeded in
    establishing themselves as a valuable source of assistance in the
    regions they serve, it may be time to consider targeting services
    even more closely on their highest priority customers\--schoolwide
    programs, high-poverty schools and districts, or low-performing
    schools and districts.

[]{#refheading___toc469391849 .anchor}I. INTRODUCTION

In 1994, as part of its reauthorization of the Elementary and Secondary
Education Act (ESEA), Congress replaced the U.S. Department of
Education's (ED's) existing network of 48 categorically based technical
assistance centers with 15 Comprehensive Regional Assistance
Centers.[^1] The Comprehensive Centers, as authorized under Title XIII
of ESEA, are charged with delivering a new brand of "comprehensive"
assistance to support standards-based reform as envisioned in other
sections of the law. As envisioned in Title XIII, the Comprehensive
Centers would become the cornerstone of a "national technical assistance
and dissemination system" that would support states, districts, American
Indian tribes, schools, and other ESEA grantees in administering and
implementing ESEA programs and in implementing school reforms that
improve teaching and learning for all students. This report examines the
ways in which the Comprehensive Centers have responded to their
statutory mission, and the ways in which they have contributed to
educational reform in the states, districts, and schools they serve.

[]{#__RefHeading___Toc469391850 .anchor}Program Mission and Goals

The Improving America's Schools Act (IASA) of 1994, which amended ESEA,
represents a departure in federal education policy. Instead of fostering
separate, categorical programs in parallel to students' "regular"
instruction, ESEA provisions now encourage states and districts to use
federal funds to reinforce state and local efforts geared to challenging
state standards. In its reauthorization of ESEA, Congress found that:

> Current technical assistance and dissemination efforts \[those
> operating before 1994\] are fragmented and categorical in nature, and
> thus fail to address adequately . . . integrating \[various ESEA
> programs and state and local reforms\] into a coherent strategy for
> improving teaching and learning \[P.L.103-382, Section 13001(4)\].

Rather than assisting ESEA grantees to implement the provisions of a
single, narrowly-focused categorical program, the Comprehensive
Centers[^2] were charged with pioneering a new form of comprehensive,
cross-program assistance that would help states, schools, districts, and
American Indian tribes help all students, particularly those who are
poor, limited English proficient, migrant, or American Indian, to attain
challenging new academic standards. More specifically, the Comprehensive
Centers were charged with providing training and technical assistance
in:

-   Improving the quality of instruction, curricula, assessments, and
    other aspects of school reform, supported by funds under Title I.

-   Implementing effective schoolwide programs.

-   Meeting the needs of children, especially children in high-poverty
    areas, migrant children, immigrant children, limited English
    proficient (LEP) children, neglected or delinquent children,
    homeless children, American Indian children, and children with
    disabilities.

-   Implementing high-quality professional development.

-   Improving the quality of bilingual education.

-   Creating safe and drug-free environments.

-   Implementing educational applications of technology.

-   Coordinating education services and programs to ensure the full
    participation of all students.

-   Increasing parent involvement in education.

-   Reforming schools and school systems, and the governance and
    management of schools.

-   Evaluating programs.

-   Meeting the special needs of students living in urban and rural
    areas \[P.L.103-382, Section 13102 (a)(1)(A-L)\].

In creating the Comprehensive Centers program, Congress greatly expanded
the range and scope of the centers' responsibilities relative to the
work of the five technical assistance programs that had existed before
1994. At the same time, Congress drastically reduced the resources
available to support the new program by funding the Comprehensive
Centers at approximately half the level of the five previous categorical
technical assistance programs. This combination of expanded mission and
reduced funding has forced the Comprehensive Centers to make strategic
choices about whom they will serve and how.

In the face of this ambitious legislative mandate, each Comprehensive
Center has taken steps to formulate its own mission and core strategies
for delivering assistance that will address the needs and interests of
customers in its region. (A table displaying mission statements drawn
from Comprehensive Center annual reports is attached in Appendix A.)
Although each center\'s mission and goals varies slightly from the rest
in content and focus, all reflect the charge to serve high-poverty
schools and districts and to build the capacity of the SEAs and LEAs in
their regions.

The Comprehensive Centers differ from other federally funded technical
assistance programs in two ways. First, the Comprehensive Centers are
required by their authorizing legislation to assign highest priority to
serving: (1) schoolwide programs, and (2) LEAs and Bureau of Indian
Affairs-(BIA-) funded schools with the highest percentage or numbers of
poor children. This specific focus on high-poverty schools and districts
is unique to the Comprehensive Centers program. Second, the
Comprehensive Centers are the only technical assistance program charged
with assisting states, schools, and districts in the implementation of
ESEA programs.

[]{#__RefHeading___Toc469391851 .anchor}Overview of the Comprehensive
Centers Program

The 15 Comprehensive Centers signed cooperative agreements with ED in
October 1995 and began operation in April 1996 after a six-month startup
period. At the time of data collection for this evaluation, the
Comprehensive Centers had been in operation for about three years.

Exhibit I.1

Timeline of Key Events

The \"Comprehensive Centers at a Glance\" exhibit at the beginning of
this volume shows the 15 Comprehensive Center regions. ED set regional
boundaries and allocated funds to individual Comprehensive Centers based
on the concentrations of Title I-eligible children, limited English
proficient children, migrant children, and American Indian children in
each state. Although some of the organizations that operate regional
educational laboratories also operate Comprehensive Centers, the
boundaries of these regions do not match the boundaries of the regional
educational laboratory program or any other ED technical assistance
programs.

The Comprehensive Centers vary in size, with the budget of the largest
center almost three times as large as the budget of the smallest center.
Exhibit I.2 shows the size of these allocations and the numbers of
states, districts, Title I schools, and eligible students served by each
center. Each Comprehensive Center employs an average of 18 regular
professional staff in 12 full-time-equivalent (FTE) positions, although
staffing levels vary a great deal from center to center. With only one
or two exceptions, all the Comprehensive centers employ at least some
staff who worked for one of the technical assistance programs operated
by ED before 1994.

Most Comprehensive Centers are made up of a primary grantee and several
partner organizations (a list of grantees and partner organizations is
included in Appendix B). Five of the 15 institutions that received
grants to operate Comprehensive Centers also operate regional
educational laboratories, five are universities, and the remaining five
are educational service organizations of various kinds. Ten of the 15
Comprehensive centers are made up of one or more organizations that
operated a grant or a contract under the old system of
categorically-based technical assistance.

In most cases, partner organizations hold subcontracts to provide
services on behalf of a Comprehensive Center and contribute some
specialized expertise to the overall mix of center services. In most
cases, primary grantees and their partner organizations are located in
different cities or even states. Working relationships between primary
grantees and partners vary from center to center, ranging from close
collaboration to relatively independent activity.

Because the Comprehensive centers replaced technical assistance programs
that were operated by both the Office of Elementary and Secondary
Education (OESE) and the Office of Bilingual Education and Minority
Languages Affairs (OBEMLA), senior staff from both of these offices
jointly oversee the Comprehensive Centers program. Day-to-day management
of the Comprehensive Centers program comes from the Goals 2000 Office
within OESE.

Exhibit I.2

**Budget Allocations and Demographic Data for the 15 Comprehensive
Center Regions**

<table style="width:100%;">
<colgroup>
<col style="width: 6%" />
<col style="width: 12%" />
<col style="width: 10%" />
<col style="width: 9%" />
<col style="width: 7%" />
<col style="width: 9%" />
<col style="width: 11%" />
<col style="width: 11%" />
<col style="width: 11%" />
<col style="width: 8%" />
</colgroup>
<tbody>
<tr class="odd">
<td rowspan="2"><h1 id="region">Region</h1></td>
<td rowspan="2"><p><strong>Comprehensive</strong></p>
<p><strong>Center</strong></p>
<p><strong>FY99 Budget</strong></p></td>
<td rowspan="2"><strong>Total Professional FTEs</strong></td>
<td rowspan="2"><p><strong>Number of</strong></p>
<p><strong>State</strong></p>
<p><strong>Education</strong></p>
<p><strong>Agencies</strong></p></td>
<td rowspan="2"><strong>Number of Districts</strong></td>
<td rowspan="2"><p><strong>Number of</strong></p>
<p><strong>Title I Schools</strong></p></td>
<td colspan="4"><p><strong>Number of Students</strong></p>
<p><strong>(rounded to the nearest thousand)</strong></p></td>
</tr>
<tr class="even">
<td><p><strong>Title I</strong></p>
<p><strong>Eligible</strong></p></td>
<td><strong>Migrant</strong></td>
<td><strong>LEP</strong></td>
<td><strong>Indian</strong></td>
</tr>
<tr class="odd">
<td>I</td>
<td>$1,260,615</td>
<td>4.55</td>
<td>6</td>
<td>1,302</td>
<td>2,256</td>
<td>380,000</td>
<td>17,000</td>
<td>79,000</td>
<td>2,000</td>
</tr>
<tr class="even">
<td>II</td>
<td>$1,974,681</td>
<td>17.38</td>
<td>1</td>
<td>706</td>
<td>2,593</td>
<td>841,000</td>
<td>11,000</td>
<td>247,000</td>
<td>5,000</td>
</tr>
<tr class="odd">
<td>III</td>
<td>$2,320,022</td>
<td>17.28</td>
<td>7</td>
<td>1,814</td>
<td>5,175</td>
<td>1,183,000</td>
<td>19,000</td>
<td>85,000</td>
<td>1,000</td>
</tr>
<tr class="even">
<td>IV</td>
<td>$2,145,861</td>
<td>16</td>
<td>6</td>
<td>726</td>
<td>3,715</td>
<td>1,059,000</td>
<td>32,000</td>
<td>39,000</td>
<td>18,000</td>
</tr>
<tr class="odd">
<td>V</td>
<td>$2,156,149</td>
<td>13</td>
<td>5</td>
<td>837</td>
<td>4,086</td>
<td>1,068,000</td>
<td>34,000</td>
<td>35,000</td>
<td>17,000</td>
</tr>
<tr class="even">
<td>VI</td>
<td>$1,881,624</td>
<td>13.65</td>
<td>6</td>
<td>2,268</td>
<td>5,977</td>
<td>801,000</td>
<td>27,000</td>
<td>103,000</td>
<td>70,000</td>
</tr>
<tr class="odd">
<td>VII</td>
<td>$2,323,837</td>
<td>18.7</td>
<td>6</td>
<td>3,253</td>
<td>5,380</td>
<td>1,069,000</td>
<td>43,000</td>
<td>186,000</td>
<td>110,000</td>
</tr>
<tr class="even">
<td>VIII</td>
<td>$2,385,538</td>
<td>17.63</td>
<td>1</td>
<td>1,043</td>
<td>3,923</td>
<td>979,000</td>
<td>97,000</td>
<td>514,000</td>
<td>1,000</td>
</tr>
<tr class="odd">
<td>IX</td>
<td>$1,618,371</td>
<td>13.1</td>
<td>6</td>
<td>624</td>
<td>2,160</td>
<td>496,000</td>
<td>30,000</td>
<td>260,000</td>
<td>121,000</td>
</tr>
<tr class="even">
<td>X</td>
<td>$1,264,964</td>
<td>9.2</td>
<td>5</td>
<td>1,148</td>
<td>2,669</td>
<td>324,000</td>
<td>56,000</td>
<td>113,000</td>
<td>49,000</td>
</tr>
<tr class="odd">
<td>XI</td>
<td>$1,773,799</td>
<td>8.1</td>
<td>1</td>
<td>823</td>
<td>N/A</td>
<td>535,000</td>
<td>76,000</td>
<td>469,000</td>
<td>25,000</td>
</tr>
<tr class="even">
<td>XII</td>
<td>$2,593,355</td>
<td>7.5</td>
<td>1</td>
<td>232</td>
<td>N/A</td>
<td>918,000</td>
<td>131,000</td>
<td>913,000</td>
<td>8,000</td>
</tr>
<tr class="odd">
<td>XIII</td>
<td>$849,294</td>
<td>5.5</td>
<td>1</td>
<td>53</td>
<td>205</td>
<td>18,000</td>
<td>12,000</td>
<td>35,000</td>
<td>31,000</td>
</tr>
<tr class="even">
<td>XIV</td>
<td>$2,375,262</td>
<td>11.75</td>
<td>3</td>
<td>69</td>
<td>2,274</td>
<td>1,104,000</td>
<td>57,000</td>
<td>290,000</td>
<td>0</td>
</tr>
<tr class="odd">
<td>XV</td>
<td>$861,628</td>
<td>4.7</td>
<td>7</td>
<td>7</td>
<td>117</td>
<td>44,000</td>
<td>0</td>
<td>42,000</td>
<td>0</td>
</tr>
</tbody>
</table>

Sources: Budget Service, Office of the Under Secretary, U.S. Department
of Education; Comprehensive Centers.

[]{#__RefHeading___Toc469391852 .anchor}Evaluation Purposes and Methods

In September 1998, ED awarded a task order to Policy Studies Associates
(PSA) of Washington, D.C., to conduct an evaluation of the Comprehensive
Centers. The evaluation was designed to address two basic questions:

(1) How are the Comprehensive Centers fulfilling their statutory
    mission, as embodied in Title XIII of ESEA?

(2) How does the Comprehensive Centers' work contribute to educational
    change and improvement in the states, districts, and schools they
    serve?

Data collection for the evaluation was carried out in three phases.
Beginning in November 1998, PSA visited all 15 Comprehensive Centers and
interviewed professional staff at all of the organizations that
contribute to the centers' work. PSA reviewed each center's annual plan
of operation or strategic plan, quarterly and annual progress reports to
ED, a budget for the most recent fiscal year, and reports on any
evaluation activities conducted by the centers. Profiles of each
Comprehensive Center, based on information collected during these site
visits, are collected in Volume III of this report.

In the second phase of data collection, PSA administered three different
surveys to customers and potential customers of the Comprehensive
Centers in spring and summer 1999:

-   The ***Survey of Center Customers: State and Local Contacts*** was
    mailed to a random sample of 1,086 representatives of the states,
    districts, schools, and other organizations served by the
    Comprehensive Centers in 1998. Survey respondents were key contacts
    who had worked closely with the centers on the delivery of technical
    assistance to their organizations. To draw the sample, PSA worked
    from customer lists compiled by the Comprehensive Centers according
    to standard sampling criteria.

-   The ***Survey of Center Customers: Participants in Center
    Activities*** was administered to participants in a sample of 30
    Comprehensive Center initiatives. The activities included in the
    sample for this survey were relatively long-term and intensive, and
    represented a variety of major center initiatives (for example,
    model schools initiatives, teacher networks, or long-term
    professional development efforts). Working from registration or
    participant lists provided by the centers, PSA administered surveys
    to 1,123 teachers and administrators.

-   The ***Survey of Potential Customers*** was administered to a
    nationally representative sample of 1,122 school districts. Surveys
    were sent to the administrator at each district office directly
    responsible for the implementation of federal programs.

In addition, two items in the ***Follow-Up Survey of State
Implementation of the Elementary and Secondary Education Act***, funded
by ED under another contract, and fielded in the summer of 1998, asked
federal programs administrators in all 50 state education agencies about
their experiences with the Comprehensive Centers.

The purpose of these surveys was to collect information from customers
about their interaction with the centers, to assess their satisfaction
with the quality and utility of services, and to learn why some
potential customers have not worked with the centers.

The third and final phase of data collection consisted of 15 case
studies of center services to states and local sites (both districts and
schools), and case studies of collaboration among Comprehensive Centers
and between the centers and other technical assistance providers. The
purpose of these case studies was to take a much more fine-grained look
at the effects of Comprehensive Center services on individuals and
organizations.

In consultation with Comprehensive Center staff, PSA identified state
and local case study sites where the Comprehensive Centers had worked
intensively and where effects could reasonably be observed by spring or
summer 1999. Because PSA was interested in studying the effects of
Comprehensive Center services on individuals and on organizations, PSA
limited its sample to those cases where the centers had been working
intensively enough and for long enough to produce observable effects.
From a group of 34 nominations, PSA selected a sample of seven states
and eight local sites, with attention given to variation in geographic
and demographic characteristics, the content of the assistance, the
kinds of assistance delivered, and the types of customers served. PSA
chose one case study site from each Comprehensive Center region (see the
table in Appendix C for a brief description of the case study sites).
The results of these case studies are written up in Volume II of this
report.

PSA also consulted with Comprehensive Center staff to identify eight
examples of collaboration among the Comprehensive Centers or between
Comprehensive Centers and other federally funded technical assistance
providers for more in-depth study. In choosing subjects for these
profiles of collaboration, we tried to identify the most extensive
examples of collaboration in the Comprehensive Center network. The
resulting profiles are also included in Volume II.

One final note on PSA\'s data collection methods: This evaluation was
not designed to collect data on the experiences of the entire universe
of center customers. Instead, we sampled customers---

both gatekeepers (those who arrange for services to their organizations)
and participants---whose organizations had received professional
development or consultation services directly from the centers. We did
not sample participants in regional events; participants in one-time,
short-term training

activities; or customers whose contact with the center was limited to
receipt of written materials or information over the telephone. We did
not attempt to collect data on these customers for two reasons: (1) the
centers do not keep adequate records of all customers they reach through
short-term training and other low-intensity activities, and it would not
be reasonable to expect them to do so (for example, they do not collect
names and contact information for every individual who participates in
every one of their workshops or calls the center on the telephone); and
(2) customers who have received a limited amount of service from the
centers are unlikely to remember the service well enough to comment on
it in a survey or in an interview.

**Overview of the Report**

Since 1996, when they began full operation, the Comprehensive Centers
have succeeded in establishing themselves in the regions and developing
an established customer base. As we will show in the following chapter,
the Comprehensive Centers have reached a number of their potential
customers at the state, district, and school levels, and they have
targeted their services in response to the priorities established in
their authorizing legislation. The Comprehensive Centers spend the
majority of their staff time and other resources on a class of major
initiatives and activities that they describe as high-intensity,
high-impact assistance---professional development and consultation
services designed to change the behavior of organizations and
individuals. A review of the centers' major initiatives indicates that
most Comprehensive Center activities are indeed "comprehensive," meaning
that they support the vision of reform contained in other sections of
ESEA. For these reasons, we would conclude that the Comprehensive
Centers are responding in a credible way to their statutory mission, as
embodied in Title XIII of ESEA.

Comprehensive Center customers give center services high ratings for
their quality and usefulness. In fact, one of the most important
accomplishments of the Comprehensive Centers program is that in a
relatively short period of time (three years at the time the surveys
were administered), the centers have been able to assemble a relatively
large base of satisfied customers. Our review of the quality of center
services along various dimensions suggests that, at least for their
highest profile, highest intensity activities, the centers do deliver
services of good quality.

The Comprehensive Centers program faces challenges, however. The
centers' broad mandate and limited resources have forced them to make
choices about whom they will serve and the breadth and depth of services
they will provide. Because the Comprehensive Centers have been at times
unable to provide the volume of services that the antecedent centers had
provided, some customers have been dissatisfied with the level and kinds
of assistance they have received.

In addition, the Comprehensive Centers do not have the resources
necessary to support the depth and breadth of change envisioned in Title
XIII of ESEA in any but a handful of schools and districts. The preamble
to Title XIII of ESEA describes the purpose of ED's technical assistance
as: (1) administering ESEA programs, (2) implementing school reforms to
improve teaching and learning for all students, (3) coordinating those
reforms with other federal, state, and local efforts to help all
students meet challenging state standards, and (4) adopting, adapting,
and implementing promising and proven practices for improving teaching
and learning. Comprehensive Center customers report that their own work
and the work of their organizations has changed as a result of
Comprehensive Center services. Our observations of these effects in the
case study sites reveal, however, that the centers have been most
successful in disseminating new ideas related to school improvement, not
in promoting the kind of whole-cloth reform that might plausibly lead to
changes in teaching and learning for all students.

As a result of center services, states are running programs more
effectively and providing new technical assistance services to the
field, and school and district administrators have acquired new skills
in program planning and needs assessment. However, Comprehensive Center
services, at their present level of intensity, do not produce radical
changes in the behavior of organizations and individuals in any but a
few instances. This dilemma is not unique to the Comprehensive Centers
program. Instead, it reflects the great level of effort required to
achieve ED's goals for its technical assistance systems. This report
will explore some of these themes in greater depth.

The text that follows is divided into six sections. In Chapter II, we
describe who the Comprehensive Centers' customers are and their purposes
in seeking out technical assistance. Chapter III describes the centers'
portfolio of services and the volume and content of services received by
the Comprehensive Centers' core customers. In Chapter IV, we assess the
quality of Comprehensive Center services and examine how customer
satisfaction with the quality and utility varies by the characteristics
of the service as well as by center. In Chapter V we report on the
effects of Comprehensive Center services, both as reported by customers
on surveys and as observed in our case study sites. Chapter VI examines
collaboration within the Comprehensive Center network and between the
Comprehensive Centers and other federally funded technical assistance
providers. Chapter VII presents our conclusions and some recommendations
for the Comprehensive Centers and ED to consider in planning future
technical assistance services. Appendices to this volume include the
study's customer surveys and information on survey sampling methods,
brief descriptions of the case study sites, and a summary of the
Comprehensive Centers' progress report data.

Two additional volumes to this report contain additional descriptions of
Comprehensive Center services and activities. Volume II contains case
studies of Comprehensive Center services to states and

local sites, as well as profiles of Comprehensive Center collaborations
with other technical assistance providers. Volume III contains profiles
of each of the 15 Comprehensive Centers, which describe key features of
the centers' operations in 1998 and summarize each center's portfolio of
services at the time of PSA's site visit.

II\. COMPREHENSIVE CENTER CUSTOMERS

Title XIII of ESEA requires that the Comprehensive Centers provide
training and assistance to "State educational agencies, tribal divisions
of education, local educational agencies, schools, and other grant
recipients under this Act" \[P.L. 103-382, Section 13102 (a)(1)\]. In
addition, Title XIII directs the Comprehensive Centers to give priority
to: (1) Title I schoolwide programs and (2) local educational agencies
and Bureau of Indian Affairs (BIA) schools with the highest percentages
or numbers of children in poverty. This evaluation examined the breadth
of the Centers\' reach among their potential customer base, assessed the
Centers\' success in targeting their services to high-priority
customers, and described customers\' purposes in seeking technical
assistance.

[]{#__RefHeading___Toc469391857 .anchor}Who Are the Comprehensive
Centers' Customers?

This evaluation drew on surveys of customers identified in the
Comprehensive Centers\' own records, as well as nationally
representative surveys of state and district administrators who are
potential users of center assistance, to describe the centers\' existing
customer base.

**[States, Districts, Schools, and Other Organizations
Served]{.underline}**

***State Education Agencies.*** The Comprehensive Centers provide
services to SEAs in all 50 states, the District of Columbia, the U.S.
Virgin Islands, Puerto Rico, and nine Pacific entities. Within SEAs, the
centers' potential customers include managers of federal programs and
staff responsible for school improvement programs. In 1998, 68 percent
of all SEA staff responsible for the administration of nine different
federal programs reported that they had received services from the
Comprehensive Centers.

***Districts.*** Not all districts are potential customers of the
Comprehensive Centers. Because the Comprehensive Centers are required to
assign priority to high-poverty districts, we considered all districts
with poverty levels above the national median to be potential center
customers of the centers.

Of this group of districts, 16 percent reported that they had received
services from a Comprehensive Center in 1998-99 (Exhibit II.1). An
additional 26 percent of potential district customers reported that they
had heard of the Comprehensive Centers but had not received services or
were not sure if they had received services. The remaining 58 percent of
districts had not heard of the Comprehensive Center serving their region
or didn't remember if they had heard. As we discuss below, the figures
are different for particular kinds of districts, such as those with very
high poverty.

In 1998, the Comprehensive Centers' own customer lists show that they
served approximately 387 districts directly through professional
development or consultations provided on-site (Exhibit II.2). This group
of 387 districts represents the Comprehensive Centers' "core" group of
customers at the district level, a group that generally received a more
intensive set of services than other district customers. It is a subset
of the 16 percent of districts that reported that they had received
services---any kind of services---in Exhibit II.1.

***Schools.*** At the school level, the Comprehensive Centers define
their potential customer base as all schoolwide program schools and all
schools with poverty rates of 50 percent or more.

This evaluation did not conduct a survey of schools to determine, for
example, the percentage of schoolwide program schools that have received
services from the Comprehensive Centers. In 1998, the centers reported
that they served 680 schools by providing professional development or
consultation directly on-site, or by serving teams of teachers from the
school in relatively long-term professional development activities
offsite (Exhibit II.2).

Although the Comprehensive Centers work with gatekeepers at all levels
of the educational system to gain entrée to sites and to plan services,
the end users of training and technical assistance are based
predominantly in schools. For example, two-thirds of all participants in
our sample of Comprehensive Center activities were based in schools.

*Other agencies*. In addition to SEAs, districts, and schools, the
Comprehensive Centers also serve a number of intermediate education
agencies and other organizations. Intermediate education agencies serve
groups of schools and districts in regions within states. Sometimes they
are organized and funded by SEAs; sometimes they are cooperatives
supported by member districts. Several Comprehensive Centers, especially
those that operate in California, New York, and Texas, work closely with
intermediate education agencies to deliver services to their region. In
1998, the Comprehensive Centers reported that they provided services to
124 intermediate education agencies

(Exhibit II.2).

![](media/image3.wmf){width="9.3in" height="7.59375in"}

Exhibit II.2

Number of Organizations Receiving On-Site Professional Development

or Consultation in 1998, and Number of Gatekeepers at Those
Organizations,

as Reported by the Comprehensive Centers

  ----------------------------- ------------------------ ----------------------------
  **[Type of                    **Number of              **[Number of
  Organization]{.underline}**   Organizations [Served    Gatekeepers]{.underline}**
                                On-Site in               
                                1998^1^]{.underline}**   

  State Education Agencies      62^2^                    392

  Districts                     387                      592

  Schools                       680                      845

  Intermediate Education        124                      187
  Agencies                                               

  Other Educational             211                      240
  Organizations                                          

  Total                         1,464                    2,256
  ----------------------------- ------------------------ ----------------------------

Exhibit reads: In 1998, 62 State Education Agencies received on-site
professional development or consultation; 392 gatekeepers were located
at the State Education Agencies.

1.  Includes organizations that received one or more days of
    professional development off-site for a team of representatives from
    the organization.

2.  Includes education agencies in the 50 states, the District of
    Columbia, Puerto Rico, the U.S. Virgin Islands, and entities in the
    Pacific region.

In 1998, the centers also reported that they served 211 other
organizations, including American Indian tribes, colleges and
universities, Title I Parent Advisory Councils, and various professional
organizations.

**[Roles and Responsibilities of Individuals Served]{.underline}**

***Gatekeepers.*** Within organizations, there are two kinds of center
customers. The first, whom we call the gatekeeper, is the Comprehensive
Center's primary point of contact with the organization being served.
The gatekeeper (and, occasionally, his or her supervisors) makes the
initial decision to engage the center's services. The gatekeeper helps
to plan activities and negotiates the terms of the center's work on
site. In large organizations, particularly SEAs and large school
districts, Comprehensive Centers may work with multiple gatekeepers. As
a group, gatekeepers are of particular interest to this evaluation
because they decide how and when their organizations will seek out and
use technical assistance resources.

In 1998, Comprehensive Center records show that they worked with nearly
400 gatekeepers to plan and deliver services to SEAs (Exhibit II.2).
Exhibit II.3 shows that three-quarters of these SEA gatekeepers report
that they administer a federal program of some kind. In addition, most
gatekeepers (81 percent) are responsible for providing training,
professional development, or technical assistance themselves.

**Exhibit II.3**

Professional Responsibilities of Gatekeepers

+-------------------------+--------+------+-----+---------+----------+
| **Which of the          |        |      |     |         |          |
| following activities    |        |      |     |         |          |
| are major parts of your |        |      |     |         |          |
| professional            |        |      |     |         |          |
| responsibilities?**     |        |      |     |         |          |
+-------------------------+--------+------+-----+---------+----------+
|                         | #      |      |     |         |          |
+-------------------------+--------+------+-----+---------+----------+
|                         | # Per  |      |     |         |          |
|                         | cent o |      |     |         |          |
|                         | f Gate |      |     |         |          |
|                         | keeper |      |     |         |          |
|                         | s Repo |      |     |         |          |
|                         | rting, |      |     |         |          |
|                         |        |      |     |         |          |
|                         | # [    |      |     |         |          |
|                         | by Typ |      |     |         |          |
|                         | e of O |      |     |         |          |
|                         | rganiz |      |     |         |          |
|                         | ation] |      |     |         |          |
|                         | {.unde |      |     |         |          |
|                         | rline} |      |     |         |          |
+-------------------------+--------+------+-----+---------+----------+
|                         |        |      |     |         |          |
+-------------------------+--------+------+-----+---------+----------+
| ###                     | *      | **Sc | *   | **Inter | **Other  |
| ## [Professional Respon | *State | hool | *Sc | mediate | Organi   |
| sibilities]{.underline} | Edu    | Dist | hoo | Ed      | zation** |
|                         | cation | rict | l** | ucation |          |
|                         | Ag     | Cen  |     | A       | **[(n=73 |
|                         | ency** | tral | *   | gency** | )]{.unde |
|                         |        | Offi | *[( |         | rline}** |
|                         | *      | ce** | n=2 | **[     |          |
|                         | *[(n=1 |      | 67) | (n=59)] |          |
|                         | 29)]{. | *    | ]{. | {.under |          |
|                         | underl | *[(n | und | line}** |          |
|                         | ine}** | =192 | erl |         |          |
|                         |        | )]{. | ine |         |          |
|                         |        | unde | }** |         |          |
|                         |        | rlin |     |         |          |
|                         |        | e}** |     |         |          |
+-------------------------+--------+------+-----+---------+----------+
| Administering an        | 27     | 46   | 77  | 43      | 27       |
| educational             |        |      |     |         |          |
| organization            |        |      |     |         |          |
+-------------------------+--------+------+-----+---------+----------+
| Administering a         | 74     | 68   | 42  | 55      | 42       |
| federally funded        |        |      |     |         |          |
| program                 |        |      |     |         |          |
+-------------------------+--------+------+-----+---------+----------+
| Providing training,     | 81     | 71   | 45  | 98      | 61       |
| professional            |        |      |     |         |          |
| development, or         |        |      |     |         |          |
| technical assistance    |        |      |     |         |          |
+-------------------------+--------+------+-----+---------+----------+

Exhibit reads: Twenty-seven percent of state education agency (SEA)
gatekeepers administer an educational organization as part of their
professional responsibilities.

Of all gatekeepers at the state level, about half help to administer
Title I. About a quarter help to administer each of seven other
programs, including Title I, Part C (Education of Migratory Children),
Safe and Drug-Free Schools and Communities, and Title VII (Bilingual
Education, Language Enhancement, and Language Acquisition) (Exhibit
II.4).

**Exhibit II.4**

######## ESEA Programs Administered by SEA, District, and School Gatekeepers

+---------------------------------+------------+------------+---------+
| **Which of the following        |            |            |         |
| federal education programs, if  |            |            |         |
| any, do you help to             |            |            |         |
| administer?**                   |            |            |         |
+---------------------------------+------------+------------+---------+
|                                 | #          |            |         |
+---------------------------------+------------+------------+---------+
|                                 | # Per      |            |         |
|                                 | cent of Ga |            |         |
|                                 | tekeepers  |            |         |
|                                 | Reporting, |            |         |
|                                 |            |            |         |
|                                 | # [by Typ  |            |         |
|                                 | e of Organ |            |         |
|                                 | ization]{. |            |         |
|                                 | underline} |            |         |
+---------------------------------+------------+------------+---------+
|                                 |            |            |         |
+---------------------------------+------------+------------+---------+
| ##                              | **State    | **School   | > **S   |
| ### [ESEA Programs]{.underline} | Education  | District   | chool** |
|                                 | Agency**   | Central    |         |
|                                 |            | Office**   | **[(    |
|                                 | **[(n      |            | n=258)] |
|                                 | =126)]{.un | **[(n      | {.under |
|                                 | derline}** | =189)]{.un | line}** |
|                                 |            | derline}** |         |
+---------------------------------+------------+------------+---------+
| Title I, Part A                 | 48         | 65         | 63      |
+---------------------------------+------------+------------+---------+
| Title I, Part B (Even Start)    | 22         | 11         | 5       |
+---------------------------------+------------+------------+---------+
| Title I, Part C (Education of   | 29         | 26         | 16      |
| Migratory Children)             |            |            |         |
+---------------------------------+------------+------------+---------+
| Title II, Eisenhower            | 24         | 48         | 28      |
| Professional Development        |            |            |         |
+---------------------------------+------------+------------+---------+
| Title IV, Safe and Drug-Free    | 23         | 38         | 44      |
| Schools and Communities         |            |            |         |
+---------------------------------+------------+------------+---------+
| Title VI, Innovative Education  | 28         | 39         | 17      |
| Program Strategies              |            |            |         |
+---------------------------------+------------+------------+---------+
| Title VII, Bilingual Education, | 24         | 28         | 39      |
| Language Enhancement, and       |            |            |         |
| Language Acquisition            |            |            |         |
+---------------------------------+------------+------------+---------+
| Title IX, Indian, Native        | 9          | 26         | 12      |
| Hawaiian, and Alaska Native     |            |            |         |
| Education                       |            |            |         |
+---------------------------------+------------+------------+---------+
| Goals 2000                      | 28         | 34         | 23      |
+---------------------------------+------------+------------+---------+

Exhibit reads: Forty-eight percent of SEA gatekeepers help to administer
Title I, Part A.

In 1998, the Comprehensive Centers worked with nearly 600 district-level
gatekeepers. Almost two-thirds of these gatekeepers administer Title I
programs, a little more than a third administer Safe and Drug-Free
Schools and Communities, and roughly a quarter administer Title I, Part
C (Education of Migratory Children), Title VII (Bilingual Education),
and Title IX (Indian Education). Seventy-one percent also have
responsibility for providing training, professional development, or
technical assistance themselves.

In 1998, the Comprehensive Centers worked with almost 850 gatekeepers at
the school level. More than three-quarters of school-level gatekeepers
report that they administer an educational organization; the centers'
records tell us that most of these are principals.

***Participants.*** The second kind of customer is the participant in
center-sponsored training and technical assistance activities.
Participants are the end-users of technical assistance services; they
are expected to use the information they acquire from the centers in
their daily work. They may or may not have been involved in the planning
of services, and they usually do not help to decide whether their
organizations will continue to use the Comprehensive Centers as a
resource.

About two-thirds of all participants in our sample of 30 Comprehensive
Center activities were based in schools. Exhibit II.5 shows that of all
school-based participants in center activities, 70 percent (accounting
for half of all participants in Comprehensive Center activities)
reported that they teach elementary or secondary students as a major
part of their professional responsibilities.

**Exhibit II.5**

Professional Responsibilities of Participants in Center Activities

+-----------------------------------+---------+-----------+-----------+
| **Which of the following are      |         |           |           |
| major parts of your professional  |         |           |           |
| responsibilities?**               |         |           |           |
+-----------------------------------+---------+-----------+-----------+
|                                   | #       |           |           |
+-----------------------------------+---------+-----------+-----------+
|                                   | #       |           |           |
|                                   |  Percen |           |           |
|                                   | t of Pa |           |           |
|                                   | rticipa |           |           |
|                                   | nts Rep |           |           |
|                                   | orting, |           |           |
|                                   |         |           |           |
|                                   | **[by   |           |           |
|                                   | Type of |           |           |
|                                   | Organi  |           |           |
|                                   | zation] |           |           |
|                                   | {.under |           |           |
|                                   | line}** |           |           |
+-----------------------------------+---------+-----------+-----------+
|                                   |         |           |           |
+-----------------------------------+---------+-----------+-----------+
| #####                             | **S     | **School  | **State   |
|                                   | chool** | D         | Education |
| #####                             |         | istrict** | Agency**  |
|                                   | **[(    |           |           |
| ##### [Professio                  | n=513)] | ####### * | **[(n=    |
| nal Responsibilities]{.underline} | {.under | *Central  | 50)]{.und |
|                                   | line}** | Office**  | erline}** |
|                                   |         |           |           |
|                                   |         | **[(n=1   |           |
|                                   |         | 32)]{.und |           |
|                                   |         | erline}** |           |
+-----------------------------------+---------+-----------+-----------+
| Teaching elementary or secondary  | 70      | 5         | 2         |
| students                          |         |           |           |
+-----------------------------------+---------+-----------+-----------+
| Providing training, professional  | 24      | 64        | 74        |
| development, or technical         |         |           |           |
| assistance                        |         |           |           |
+-----------------------------------+---------+-----------+-----------+
| Administering an educational      | 23      | 29        | 6         |
| organization                      |         |           |           |
+-----------------------------------+---------+-----------+-----------+
| Administering a federally funded  | 10      | 56        | 50        |
| education program                 |         |           |           |
+-----------------------------------+---------+-----------+-----------+
| Parent or community               | 5       | 6         | 0         |
| representative                    |         |           |           |
+-----------------------------------+---------+-----------+-----------+

Exhibit reads: Seventy percent of school-based participants in Center
activities report that teaching elementary

> or secondary students is a major part of their professional
> responsibilities.

[]{#__RefHeading___Toc469391860 .anchor}Are Centers Serving Their
Highest-Priority School and District Customers?

The Comprehensive Centers are targeting their highest-priority
customers, at both the school and district levels, and this targeting
appears to be reasonably effective. The centers are targeting both
high-poverty districts and schools and districts and schools serving
large numbers of students in the target populations named in the
centers' authorizing legislation, including limited English proficient
students, migrant students, and Indian students.

High-Priority Districts

The majority of the centers' existing district customers may be
considered "high-priority" according to the criteria set in the centers'
authorizing legislation:

-   Almost two-thirds of district-level gatekeepers report that more
    than 50 percent of their students were eligible for free or
    reduced-price lunches. (Exhibit II.6)

**Exhibit II.6**

Percent of Students Eligible for Free and Reduced-Price Lunches in
Districts Served by Comprehensive Centers, as Reported by Gatekeepers

**(n=139)**

  ----------------------------------- -----------------------------------
  **What percentage of students in    
  your district are eligible to       
  receive free or reduced-price       
  lunches?**                          

                                      

  **[Percent of                       **Percent of Gatekeepers Reporting
  Students]{.underline}**             on the Percent of Students in Their
                                      District Who Are [Eligible for Free
                                      and Reduced-Price
                                      Lunches]{.underline}**

  75 -- 100%                          29

  51 -- 75%                           33

  26 -- 50%                           29

  0 -- 25%                            9
  ----------------------------------- -----------------------------------

Exhibit reads: Twenty-nine percent of district-level gatekeepers
reported that between 75 and 100 percent of the students in their
district are eligible to receive free or reduced-price lunches.

-   Almost a third of district-level customers reported very high
    concentrations of poverty, with 75 percent or more of their students
    eligible.

-   More than half of district-level customers report that their
    districts enroll a significant proportion of limited English
    proficient students (that is, more than 10 percent of the total
    student population). (Exhibit II.7)

-   About a fifth of district-level customers report that their
    districts enroll a significant proportion of Indian students.

**Exhibit II.7**

Percent of Target Student Populations in Districts Served by
Comprehensive Centers,

as Reported by Gatekeepers

(n=137)

+------------------------+----------+-------+-------+-------+-------+
| ## What percentage of  |          |       |       |       |       |
|  children in your dist |          |       |       |       |       |
| rict belong to each of |          |       |       |       |       |
|  the following groups? |          |       |       |       |       |
+------------------------+----------+-------+-------+-------+-------+
|                        |          |       |       |       |       |
+------------------------+----------+-------+-------+-------+-------+
|                        | *        |       |       |       |       |
|                        | *Percent |       |       |       |       |
|                        | of       |       |       |       |       |
|                        | Gat      |       |       |       |       |
|                        | ekeepers |       |       |       |       |
|                        | R        |       |       |       |       |
|                        | eporting |       |       |       |       |
|                        | on the   |       |       |       |       |
|                        | Range of |       |       |       |       |
|                        | Students |       |       |       |       |
|                        | in       |       |       |       |       |
|                        | [Their   |       |       |       |       |
|                        | District |       |       |       |       |
|                        | Who      |       |       |       |       |
|                        | Belong   |       |       |       |       |
|                        | to       |       |       |       |       |
|                        | Target   |       |       |       |       |
|                        | Po       |       |       |       |       |
|                        | pulation |       |       |       |       |
|                        | s]{.unde |       |       |       |       |
|                        | rline}** |       |       |       |       |
+------------------------+----------+-------+-------+-------+-------+
|                        |          |       |       |       |       |
+------------------------+----------+-------+-------+-------+-------+
| **[Target Student      | **[\<1   | **[   | **[1  | **[2  | **    |
| Popu                   | %]{.unde | 2-10% | 1-25% | 6-50% | [Over |
| lations]{.underline}** | rline}** | ]{.un | ]{.un | ]{.un | 50%   |
|                        |          | derli | derli | derli | ]{.un |
|                        |          | ne}** | ne}** | ne}** | derli |
|                        |          |       |       |       | ne}** |
+------------------------+----------+-------+-------+-------+-------+
| Children with limited  | 12       | 30    | 22    | 22    | 14    |
| English proficiency    |          |       |       |       |       |
+------------------------+----------+-------+-------+-------+-------+
| Indian, Native         | 52       | 27    | 8     | 3     | 8     |
| Hawaiian, or Alaska    |          |       |       |       |       |
| Native children        |          |       |       |       |       |
+------------------------+----------+-------+-------+-------+-------+
| Migratory children     | 54       | 32    | 7     | 4     | 1     |
+------------------------+----------+-------+-------+-------+-------+
| Immigrant children     | 41       | 41    | 11    | 5     | 1     |
+------------------------+----------+-------+-------+-------+-------+
| Neglected or           | 54       | 28    | 10    | 4     | 1     |
| delinquent children    |          |       |       |       |       |
+------------------------+----------+-------+-------+-------+-------+
| Homeless children      | 70       | 22    | 3     | 2     | 3     |
+------------------------+----------+-------+-------+-------+-------+
| Children with          | 5        | 39    | 44    | 7     | 2     |
| disabilities           |          |       |       |       |       |
+------------------------+----------+-------+-------+-------+-------+

Exhibit reads: Twelve percent of district-level gatekeepers reported
that less than 1 percent of students in their district are of limited
English proficiency.

Note: Rows may not sum to 100 percent because some respondents reported
that they did not know.

Among our nationally representative sample of potential district
customers, high-priority districts are more likely to report that they
have heard of the centers and that they have received services than
other kinds of districts. For example, Exhibit II.8 shows that districts
with high rates of poverty and districts with high enrollments of LEP
students, Indian students, and migrant students were more likely to
report that they had heard of the Comprehensive Center serving their
region than were other districts in the sample. Among the 50 districts
in the country with the largest numbers of children in poverty, 90
percent had heard about the Comprehensive Center serving their region.

High-priority districts are also more likely to report that they have
actually received services from the centers. For example, Exhibit II.9
shows that 40 percent of districts with significant enrollments of
limited English proficient students, 36 percent of districts with
significant enrollments of Indian students, and 25 percent of districts
with significant enrollments of migrant students reported that they had
received services from a center, compared with just 16 percent of all
districts with poverty rates over the median. Districts with the highest
concentrations of poor children and districts with the largest numbers
of poor children were also much more likely to report that they had
received services from the Comprehensive Centers.

One final set of findings provides additional evidence that the
Comprehensive Centers are effectively targeting high-priority customers
at the district level. High-poverty districts and districts with
significant enrollments of certain student populations are more likely
to report that they receive high levels of center services than are
other district customers. This is especially true for the centers'
higher-intensity, more expensive forms of service---on-site consultation
and help convening special events. For example, high-poverty districts
are more likely to report that they receive on-site consultation five or
more times a year than are other districts; the same finding holds true
for districts that enroll migrant students, Indian students, and limited
English proficient students. Although the differences are not
statistically significant for all center services and for all categories
of high-priority districts, these findings do suggest that, even within
their established customer base, the centers target higher levels of
resources to higher-priority customers.

Large Districts

Large districts were more likely to report contact with the
Comprehensive Centers than smaller districts. For example, Exhibit II.10
shows that 53 percent of districts with enrollments over 25,000 reported
that someone in the district had received services from a center,
compared with just 9 percent of districts with enrollments under 2,500.
These varying rates of contact suggest that larger districts are more
active consumers of technical assistance. However, they also suggest
that the centers are targeting their services to reach large numbers of
students efficiently.

![](media/image4.wmf){width="9.3in" height="7.633333333333334in"}

![](media/image5.wmf){width="9.279861111111112in"
height="7.298611111111111in"}

![](media/image6.wmf){width="9.3in" height="7.6in"}

## Schools

The Comprehensive Centers are also targeting high-priority schools, as
defined by the centers\' authorizing legislation:

-   Sixty-nine percent of school-level gatekeepers report that their
    school operates a Title I schoolwide program.

-   Almost three-quarters of school-level gatekeepers report that 50
    percent or more of their students are eligible for free or
    reduced-price lunches. (Exhibit II.11)

-   More than a third of school-level gatekeepers represent very
    high-poverty schools, where 75 percent or more of students are
    eligible.

-   Almost half of the schools served by the centers reported that they
    had a significant proportion of limited English proficient students
    (that is, more than 10 percent of their total enrollment), with 13
    percent reporting that *half or more* of the students they served
    were limited English proficient. (Exhibit II.12)

-   Almost a third of schools served reported that they enroll a
    significant proportion of Indian students (more than 10 percent of
    their total enrollment), with 14 percent of schools reporting that
    *half or more* of their students were Indian.

**Exhibit II.11**

Percent of Students Eligible for Free and Reduced-Price Lunches in
Schools Served by Comprehensive Centers, as Reported by Gatekeepers

(n=190)

  ----------------------------------- -----------------------------------
  **What percentage of students in    
  your school are eligible to receive 
  free or reduced-price lunches?**    

                                      

  **[Percent of                       **Percent of Gatekeepers Reporting
  Students]{.underline}**             on the Percent of Students in Their
                                      School Who Are [Eligible for Free
                                      and Reduced Price
                                      Lunches]{.underline}**

  75 -- 100%                          38

  51 -- 75%                           36

  26 -- 50%                           14

  0 -- 25%                            12
  ----------------------------------- -----------------------------------

Exhibit reads: Thirty-eight percent of school-level gatekeepers reported
that between 75 and 100 percent of the students in their school are
eligible for free or reduced-price lunches.

**Exhibit II.12**

Percent of Target Student Populations in Schools Served by Comprehensive
Centers,

as Reported by Gatekeepers

(n=193)

  ---------------------------- ---------------------------- ------------------------- -------------------------- -------------------------- --------------------
  **What percentage of                                                                                                                      
  children in your school                                                                                                                   
  belong to each of the                                                                                                                     
  following groups?**                                                                                                                       

                                                                                                                                            

                               **Percent of Gatekeepers                                                                                     
                               Reporting on the Range of                                                                                    
                               Students in [Their Schools                                                                                   
                               Who Belong to Target                                                                                         
                               Populations]{.underline}**                                                                                   

                                                                                                                                            

  **[Target Student            **[\<1%]{.underline}**       **[2-10%]{.underline}**   **[11-25%]{.underline}**   **[26-50%]{.underline}**   **[Over
  Populations]{.underline}**                                                                                                                50%]{.underline}**

  Children with limited        30                           22                        19                         16                         13
  English proficiency                                                                                                                       

  Indian, Native Hawaiian, or  49                           19                        9                          8                          14
  Alaska Native children                                                                                                                    

  Migratory children           56                           26                        7                          7                          3

  Immigrant children           58                           21                        12                         5                          3

  Neglected or delinquent      27                           40                        17                         9                          5
  children                                                                                                                                  

  Homeless children            68                           24                        4                          1                          2

  Children with disabilities   13                           42                        35                         6                          2
  ---------------------------- ---------------------------- ------------------------- -------------------------- -------------------------- --------------------

Exhibit reads: Thirty percent of school-level gatekeepers reported that
less than 1 percent of the students in their school are of limited
English proficiency.

Note: Rows may not sum to 100 percent because some respondents reported
that they did not know.

Finally, the same pattern of providing higher levels of service to
high-priority customers holds true for schools as well. High-poverty
schools and schools with significant enrollments of migrant students,
Indian students, and limited English proficient students were more
likely to report that they got certain kinds of service five or more
times a year than were other schools. Again, this pattern held true for
the centers' higher-intensity and more expensive kinds of services,
although not every difference was statistically significant for every
category of high-priority customers. These findings suggest that within
the centers' established customer base, high-priority schools are more
likely to receive services than other kinds of schools.

[]{#refheading___toc469391862 .anchor}Needs and Purposes: Why Do
Customers Seek Assistance?

States, schools, and districts seek assistance for a variety of needs
and purposes having to do with the specific local conditions they face,
the overall policy context, and the availability of a resource that they
view as likely to be useful. Despite the variety of purposes that spur
customers to seek the centers\' assistance, the centers have established
themselves as a useful source of assistance among their existing
customers.

**[Personal and Organizational Goals]{.underline}**

On surveys, large numbers of participants in Comprehensive Center
activities reported that they hoped to obtain new ideas and contacts for
advancing their own work and the work of their organizations. About
three-quarters of all respondents identified "keeping current
professionally" and "obtaining ideas or contacts that would strengthen
my work" as reasons for their participation in center-sponsored
activities. Exhibit II.13 shows that school-based participants were
slightly more likely to report that they hoped to benefit personally
from center activities, suggesting that they were slightly more likely
to see the Comprehensive Center as a source of new ideas for their own
work.

About two-thirds of all participants in center activities reported that
they hoped to obtain ideas or contacts that would strengthen their
organization's existing work or help their organization carry out an
improvement effort. Exhibit II.13 shows that district-based participants
were slightly more likely than others to report that they hoped to
advance their organization's work by participating in a center-sponsored
activity. State-based participants were slightly more likely than others
to report that they hoped to obtain new ideas that would help them be a
resource to others, probably because they are most heavily engaged in
providing training, professional development, and technical assistance
themselves.

**[Comprehensive Reform and School Improvement]{.underline}**

On surveys, gatekeepers were most likely to identify "core" issues
associated with school improvement as areas where their organization had
the greatest need for technical assistance services***.*** Exhibit II.14
shows that the topics most commonly identified by Comprehensive Center
customers were: (1) improving curriculum and instruction (29 percent);
(2) analyzing student achievement data and interpreting results (28
percent); (3) expanding parent/family involvement (25 percent); and

\(4\) designing program evaluations (23 percent) (Exhibit II.14).
Customers' responses to this survey item suggest that they share an
interest in comprehensive reform and data use that is central to the
agenda of the Comprehensive Centers and to many ED programs.

![](media/image7.wmf){width="9.920833333333333in" height="7.8in"}

**Exhibit II.14**

Topics for Which Customers Have the Greatest Technical Assistance Needs

(n=529)

+---------------------------------------+------------------------------+
| ## For which top                      |                              |
| ics does your organization have the g |                              |
| reatest need of technical assistance? |                              |
+---------------------------------------+------------------------------+
| ### Topic                             | ### Perc                     |
|                                       | ent of Gatekeepers Reporting |
+---------------------------------------+------------------------------+
| Improving curriculum and instruction  | 29                           |
+---------------------------------------+------------------------------+
| Analyzing student achievement data    | 28                           |
| and interpreting the results          |                              |
+---------------------------------------+------------------------------+
| Expanding parent/family involvement   | 25                           |
+---------------------------------------+------------------------------+
| Program evaluation                    | 23                           |
+---------------------------------------+------------------------------+
| Challenging standards; accountability | 21                           |
+---------------------------------------+------------------------------+
| Student assessment                    | 21                           |
+---------------------------------------+------------------------------+
| Improving the quality of bilingual    | 20                           |
| education programs                    |                              |
+---------------------------------------+------------------------------+
| Addressing the needs of special       | 16                           |
| populations                           |                              |
+---------------------------------------+------------------------------+
| Implementing schoolwide programs      | 15                           |
+---------------------------------------+------------------------------+
| Adopting model programs or            | 15                           |
| comprehensive school reforms          |                              |
+---------------------------------------+------------------------------+
| Implementing educational applications | 11                           |
| of technology                         |                              |
+---------------------------------------+------------------------------+
| Creating safe and drug-free learning  | 8                            |
| environments                          |                              |
+---------------------------------------+------------------------------+
| Consolidating or integrating          | 7                            |
| federally funded programs             |                              |
+---------------------------------------+------------------------------+
| Implementing ESEA programs            | 5                            |
+---------------------------------------+------------------------------+
| Issues identified by ED Integrated    | 2                            |
| Review Teams                          |                              |
+---------------------------------------+------------------------------+

Exhibit reads: Twenty-nine percent of gatekeepers reported that
improving curriculum and instruction is the topic for which they have
the greatest technical assistance need.

Comprehensive Center customers were less likely to identify topics
associated with specific ESEA provisions as areas where they had need of
assistance. Topics identified least often included:

\(1\) addressing issues identified by ED Integrated Review Teams (2
percent); (2) implementing ESEA programs (5 percent); (3) consolidating
or integrating federally funded education programs (7 percent); (4)
creating safe and drug-free learning environments (8 percent); and (5)
implementing the educational applications of technology (11 percent).
Center customers in state departments of education were more likely to
identify topics related to ESEA program administration as areas of need,
although the percentage was still low compared with those identifying
more "comprehensive" concerns. Other responses did not vary
significantly among state, district, and school customers, nor did they
vary between districts that were established customers of the centers
and districts that had not yet received services.

These findings suggest that customers of the Comprehensive Centers are
responding to the emphasis in the current policy context on
accountability systems, using data to drive decision making, and
comprehensive, rather than categorical, approaches to reform. Our
interviews with Comprehensive Center staff also suggested that federal
priorities and programs can drive demand for certain types of services.
In the year after the Comprehensive Centers began operation, for
example, many of their activities focused on helping schools that were
newly eligible to run schoolwide programs under the 1994 law to develop
schoolwide program plans. As most of the eligible schools have come on
line, however, Comprehensive Center staff members report that calls for
assistance with schoolwide program planning have been dwindling. By the
time we fielded our survey, in spring 1999, "implementing schoolwide
programs" ranked in the middle of the pack as a technical assistance
need. On the other hand, "adopting model programs or comprehensive
school reforms," a topic related to the Comprehensive School Reform
Demonstration (CSRD) Program, a newer federal initiative, ranked much
higher as an expressed need than specific ESEA programs.

## Responding to Specific Local Challenges

Our collection of 15 case studies suggests that states and local sites
are likely to seek out or accept assistance for reasons that are quite
specific to a site and to a particular task at hand. Customer purposes
that were common across several case studies included:

***Extending state or local capacity***. At both the state and local
level, some customers saw the centers as a way of extending the capacity
of their own organizations to provide services to teachers and schools.
In both Texas and Mississippi, for example, state administrators
explained that lack of staff and legislative mandates prevented them
from working with as many schools as they would like; they saw the
Comprehensive Centers as a way to serve a larger number of schools on
the state's behalf. Several districts in California joined the Reading
Success Network (RSN), a professional development initiative sponsored
by the Southern California Comprehensive Assistance Center, in order to
secure scarce professional development resources for teachers. In
addition, several principals reported that an important benefit of the
RSN was that it helped them to support a large number of new and
provisionally certified teachers in their schools, who had been hired as
a result of California's class size reduction initiative.

***Accountability pressures.*** In a few sites, accountability pressures
created an important incentive for seeking out help or accepting help
when offered. In North Carolina, for example, a new state requirement
that LEP students be tested in English two years after arriving in the
state prompted several districts to seek help with their English as a
Second Language (ESL) programs. In the spring of 1999, the middle school
at Mountainview[^3] was identified for Title I program improvement, and
school staff turned to the Comprehensive Center for help with school
improvement planning and related activity throughout the 1999-2000
school year.

***Complying with ESEA and other program provisions.*** In several
sites, state and local administrators requested help from the
Comprehensive Centers in schoolwide program planning. For example,
teachers from the Mountainview School began work with their
Comprehensive Center because they were concerned about completing their
schoolwide plan on time; the second year of their partnership with the
Mountainview School focused on assembling data and preparing a report
for their state accreditation review. At the state level, the New York
State Department of Education turned to the Comprehensive Center for
help in developing a statewide consolidated plan. Several other states,
including New York and Texas, used the Comprehensive Centers early on to
help train their Title I school support teams.

***Responding to the needs of a particular student population.***
Several states and districts in our case study sample, including the
North Carolina ESL clusters, Hawaii, and Riverside City Schools, sought
help from the Comprehensive Centers because of recent increases in the
numbers of LEP, migrant, and immigrant students in their schools. In all
of these cases, state and district administrators wanted help in
training teachers to address the special educational needs of these new
students. Our case study site in Alaska decided to participate in Alaska
Onward to Excellence because of a long-standing interest in including
native culture and knowledge in the curriculum and a commitment to
building a school culture that effectively engaged the native community.

**[Availability of Useful, Free Services]{.underline}**

In addition to satisfying their own needs and purposes, customers seek
out or accept Comprehensive Center services because the services are
free, and because customers have reason to believe that they will be
useful. For example, Exhibit II.15 shows that 87 percent of center
customers reported that the fact that center services were free of
charge was a "very important" or "moderately important" reason for using
center services and products. In interviews at all of our case study
sites, respondents commented that the fact that Comprehensive Center
services were free was an important condition allowing them to avail
themselves of those services.

Almost 90 percent of survey respondents identified the centers'
reputation for providing high-quality technical assistance as "very
important" or "moderately important" in prompting their organization to
begin using the services and products offered by the centers. This
perception of quality appears to be shared by customers and noncustomers
alike. Only 2 percent of districts that had not received services
reported that they had avoided the Comprehensive Centers because they
believed that they had a reputation for poor service.

**[An Untapped Market for Services]{.underline}**

Survey data from a nationally representative survey of districts suggest
that increased marketing and outreach efforts would create additional
demand for center services. Potential district customers who had not
made use of center products and services attributed their lack of
contact with the centers to the fact that they did not have enough
information about what the centers do and how they operate. For example,
among high-poverty districts that knew about the centers but had not
received services, the reasons most often cited for not making use of
center products and services were: (1) not receiving information about
center products or services (44 percent); (2) not having time to learn
about center products and services (36 percent); (3) not knowing whom to
contact at the center (31 percent); and

\(4\) not knowing if the center can address the district's particular
needs and interests (29 percent) (Exhibit II.16).

Few districts reported that they had avoided the centers because they
did not need their assistance. Among districts that had heard of the
centers but had not received services, only 27 percent said that they
receive adequate assistance from other sources. In addition, very few
districts reported that they had avoided the centers because they had a
negative impression of the centers' work. For example, very few reported
that: (1) they have not been able to develop productive working
relationship with center staff (9 percent); (2) center products and
services do not address their needs

(2 percent); (3) the center does not have a reputation for providing
high-quality products and services

(2 percent); or (4) they had requested assistance, but the center was
unable to provide it (1 percent).

**Exhibit II.15**

Reasons to Begin Using Comprehensive Center Products and Services

**(n= 554)**

+-----------------------------------+-------------+----------+--------+
| **How important were the          |             |          |        |
| following factors in prompting    |             |          |        |
| you or your organization to begin |             |          |        |
| using the products or services    |             |          |        |
| offered by the Comprehensive      |             |          |        |
| Center?**                         |             |          |        |
+-----------------------------------+-------------+----------+--------+
|                                   | #           |          |        |
+-----------------------------------+-------------+----------+--------+
|                                   | # Percen    |          |        |
|                                   | t of Gateke |          |        |
|                                   | epers Repor |          |        |
|                                   | ting on the |          |        |
|                                   |             |          |        |
|                                   | *           |          |        |
|                                   | *Importance |          |        |
|                                   | of Various  |          |        |
|                                   | Factors in  |          |        |
|                                   | Prompting   |          |        |
|                                   | [Them to    |          |        |
|                                   | Use Center  |          |        |
|                                   | Products or |          |        |
|                                   | S           |          |        |
|                                   | ervices]{.u |          |        |
|                                   | nderline}** |          |        |
+-----------------------------------+-------------+----------+--------+
|                                   |             |          |        |
+-----------------------------------+-------------+----------+--------+
| #####                             | **Very      | **       | **Not  |
|                                   | I           | Somewhat | at All |
| ##### Reasons to Begin Using      | mportant/** | [        | [      |
|                                   |             | Importan | Import |
| ##### [Comprehensive Center P     | *           | t]{.unde | ant]{. |
| roducts and Services]{.underline} | *Moderately | rline}** | underl |
|                                   | [Im         |          | ine}** |
|                                   | portant]{.u |          |        |
|                                   | nderline}** |          |        |
+-----------------------------------+-------------+----------+--------+
| Center products and services were | 87          | 11       | 3      |
| free of charge                    |             |          |        |
+-----------------------------------+-------------+----------+--------+
| The Center had a reputation for   | 86          | 11       | 3      |
| providing high-quality technical  |             |          |        |
| assistance                        |             |          |        |
+-----------------------------------+-------------+----------+--------+
| Center products and services were | 85          | 13       | 3      |
| easily accessible                 |             |          |        |
+-----------------------------------+-------------+----------+--------+
| Center products and services met  | 78          | 10       | 2      |
| the needs of our organization     |             |          |        |
+-----------------------------------+-------------+----------+--------+
| The Center provided products or   | 77          | 17       | 7      |
| services that were not available  |             |          |        |
| elsewhere                         |             |          |        |
+-----------------------------------+-------------+----------+--------+
| Center services built on          | 76          | 13       | 11     |
| technical assistance we had       |             |          |        |
| received in the past              |             |          |        |
+-----------------------------------+-------------+----------+--------+

Exhibit reads: Eighty-seven percent of gatekeepers thought that it was
very important or moderately important to begin using Comprehensive
Center products and services because they were free of charge.

Note: Rows may not sum to 100 because of rounding.

##### 

**Exhibit II.16**

Reasons Why Districts Have Not Made Use of

Comprehensive Center Products and Services

(n=213)

+---------------------------------------+------------------------------+
| **If your district has not made use   |                              |
| of the products or services offered   |                              |
| by the Comprehensive Center, what are |                              |
| the reasons?**                        |                              |
+---------------------------------------+------------------------------+
|                                       |                              |
+---------------------------------------+------------------------------+
| **[Reasons for Not Using Center       | **Percent of Potential       |
| Products or Services]{.underline}**   | District**                   |
|                                       |                              |
|                                       | ####### **[Custom            |
|                                       | ers Reporting]{.underline}** |
+---------------------------------------+------------------------------+
| We have not received any information  | 44                           |
| about Center products or services     |                              |
+---------------------------------------+------------------------------+
| We have not had time to learn about   | 36                           |
| Center products or services           |                              |
+---------------------------------------+------------------------------+
| We don't know whom to contact at the  | 31                           |
| Center                                |                              |
+---------------------------------------+------------------------------+
| We don't know if the Center can       | 29                           |
| address our particular needs and      |                              |
| interests                             |                              |
+---------------------------------------+------------------------------+
| We receive adequate assistance from   | 27                           |
| other sources                         |                              |
+---------------------------------------+------------------------------+
| We have not been able to develop      | 9                            |
| productive working relationships with |                              |
| Center staff                          |                              |
+---------------------------------------+------------------------------+
| Center products and services do not   | 2                            |
| address our needs                     |                              |
+---------------------------------------+------------------------------+
| The Center does not have a reputation | 2                            |
| for providing high-quality products   |                              |
| and services                          |                              |
+---------------------------------------+------------------------------+
| We have requested assistance, but the | 1                            |
| Center was unable to provide it       |                              |
+---------------------------------------+------------------------------+

Exhibit reads: Forty-four percent of potential district customers
reported that one reason their district has not made use of Center
products or services is because they have not received any information
about Center products or services.

These findings suggest that many more districts would be interested in
receiving center services, given additional outreach efforts.

**[The Comprehensive Centers as a Regular Source of
Assistance]{.underline}**

Among their established customers, the Comprehensive Centers have
managed to carve out a niche for themselves as a valuable source of
technical assistance. Approximately two-thirds of all gatekeepers report
that they "always" or "sometimes" turn to their Comprehensive Center for
help in areas that are important to them. Established customers report
that they are slightly more likely to consult the Comprehensive Centers
than any other source of assistance, with the exception of their state
education agency. These findings vary somewhat by type of customer, with
school-level gatekeepers slightly less likely to report that they would
consult the Comprehensive Centers than gatekeepers in districts or
states (see Exhibit II.17). Among all types of customers, however, the
Comprehensive Centers have established themselves as a credible source
of assistance to which gatekeepers can turn on a regular basis.

Summary

Evidence from nationally representative surveys of school districts and
state administrators of federal programs shows that the Comprehensive
Centers have reached a number of their potential customers with some
level of service. In addition, the centers are targeting their services
to their highest priority customers, as defined in their authorizing
legislation. Comprehensive Center customers engage in work with the
Comprehensive Centers in the expectation that the centers' assistance
will be of high quality and will address their needs and purposes. In
the next chapter, we describe the major categories of center services
and describe the content and format of services received by individual
customers.

![](media/image8.wmf){width="9.3in" height="7.564583333333333in"}

[]{#__RefHeading___Toc469391865 .anchor}III. PORTFOLIO OF SERVICES

Title XIII of ESEA charges the Comprehensive Centers with providing
training and technical assistance to customers on a long list of
possible topics, from improving the quality of instruction, to
implementing effective schoolwide programs, to meeting the needs of
special populations of students. A central purpose of this evaluation is
to describe the ways in which the centers have responded to their
statutory mission, and in particular, the ways in which they have
organized their portfolio of services. In this chapter we examine both
center strategies for organizing and delivering assistance, and the
content and format of services received by customers and the
organizations they represent.

[]{#__RefHeading___Toc469391866 .anchor}What Are the Centers' Major
Strategies for Organizing and Delivering Assistance?

Most center services can be classified into one of two categories: (1)
"major" initiatives that consist of multiple contacts with an
organization over a period of time, and (2) information dissemination
activities and short-term training events that do not, on their own,
involve a relationship with customers. In this section we describe both
types of activities.

Major Center Initiatives

Many of the Comprehensive Centers organize a substantial portion of
their activities around a small group of key initiatives. These
initiatives generally engage the centers in long-term relationships with
customers and are intended to create some kind of systematic change in
the organizations served. In interviews, Comprehensive Center staff made
it clear that they believe that these kinds of long-term services are
their most valuable contribution to the regions they serve. One comment
summarized the observations of many other center staff members:

> One of the things we decided a while back is that one-time
> interventions don't really have potential of creating systemic,
> comprehensive change. We shy away from this. Even in cases when we're
> going into a district that's in a crisis, we try to use it as an
> entree for sustained support.

The centers' major initiatives, which represent their highest-intensity,
most expensive set of services, can be classified into three broad
categories: (1) professional development initiatives;

\(2\) services to model schools and partnership sites; and (3)
consultation with states, districts, and schools.

***Professional development initiatives.*** Most of the centers'
services to states and local sites involve some form of training and
professional development. Among the centers' major initiatives, training
and other professional development provided is relatively intensive
(meaning that the total package of training delivered amounts to five
days or more) and is usually provided during a period of weeks or
months. Typically, centers follow up the training with additional
opportunities for networking, information dissemination, and
over-the-shoulder consultation, although individual participation in
these follow-up activities may vary. School and district administrators
and teachers usually participate in center-sponsored professional
development as members of a team representing their organization.
Examples of major professional development initiatives in the
Comprehensive Center network include the following:

-   **Reading Success Network (RSN):** The goal of the RSN is to improve
    student achievement in reading in grades K-3 by training a cadre of
    school-level coaches to support classroom teachers' efforts to
    improve instruction. The RSN was developed by the Southern
    California Comprehensive Assistance Center and has since been
    adopted by all 15 Comprehensive Centers as one of the network's
    flagship professional development initiatives. The RSN provides
    professional development to classroom teachers and school-based
    literacy coordinators in the assessment of reading and the diagnosis
    of reading difficulties, the use of data to assess reading
    performance, intervention strategies, and peer coaching and
    cognitive coaching as strategies for working with colleagues.
    Coaches work with classroom teachers to incorporate new strategies
    in the diagnosis of reading difficulties and in early intervention
    into their existing repertoire of skills. The Comprehensive Center
    sponsors the initial training series and works with the county
    offices of education to provide follow-up training as needed.

```{=html}
<!-- -->
```
-   **Cognitively Guided Instruction (CGI):** The Region VI
    Comprehensive Center sponsors training in Cognitively Guided
    Instruction, a math program based on problem solving for students in
    kindergarten through third grades. Developed by researchers at the
    University of Wisconsin-Madison from a decade of research on the way
    that children reason mathematically, CGI helps teachers to observe
    student cognitive processes in order to tailor and individualize
    instruction. Two teachers and one resource specialist/trainer from
    each school attend a week-long summer institute sponsored by the
    Comprehensive Center that introduces them to the principles and
    practices of CGI. Teachers may return the following summer for an
    advanced CGI institute. The center expects that teachers who receive
    training will disseminate what they have learned in some way at
    their schools.

-   **Learning Facilitators' Academy:** The Region VII Comprehensive
    Center developed the Learning Facilitators' Academy to help school
    staff in a large urban district think strategically about using
    categorical resources to help all children, especially limited
    English proficient children and those eligible for Title I, to reach
    local standards for student performance. The Learning Facilitators'
    Academy provided training to teams from 12 schools, consisting of
    administrators; Title I, bilingual, and content area teachers; and
    guidance counselors. Five day-long Saturday training sessions
    focused on the change process, team building, student learning
    styles, curriculum alignment, and instructional strategies.
    Participants concluded each session with a service delivery plan
    that they tried in schools and classrooms between sessions and
    reported on at the following session.

-   **North Carolina ESL Clusters:** The Region IV Comprehensive Center
    at the Appalachia Regional Educational Laboratory provides
    professional development and training to districts that are
    experiencing a sharp rise in the numbers of migrant students
    enrolled in their schools. The center sponsors a five-day summer
    "lab school," where mainstream teachers learn both the theory of
    second language acquisition and the practice of developing lesson
    plans using effective strategies for limited English proficient
    students. Participating teachers observe other classes and pilot
    their lessons in a summer school setting. Center staff provided
    additional workshops as requested during the school year and
    conducted follow-up visits to teachers' classrooms to provide
    feedback on the implementation of the strategies they learned over
    the summer.

```{=html}
<!-- -->
```
-   **Schoolwide Programs Network.** The New England Comprehensive
    Assistance Center sponsors a Schoolwide Programs Network for
    elementary schools that have a Title I schoolwide program or are
    planning one. The goal of the network is to develop schools'
    capacity to plan and implement comprehensive school reforms.
    Participation in the Network has four components: (1) professional
    development opportunities, including an annual Schoolwide Congress,
    special topic workshops, Spring "Retooling" Meetings, and a
    Leadership Academy for principals and coaches; (2) a coach,
    generally an SEA staff member trained by the center, who provides
    follow-up, on-site consultation to assist with planning and
    implementation; (3) guided visits to schools that have raised
    student achievement; and (4) monthly mailings and on-line list serve
    discussions.

```{=html}
<!-- -->
```
-   **Training for Schoolwide Program Planning Teams.** The Region XI
    Comprehensive Assistance Center provides training to teams of
    administrators, principals, parents, and teachers between four and
    eight times over the course of a year. These full-day or half-day
    sessions address needs assessment, team building, data collection,
    data disaggregation, analysis, plan writing, and obstacles to
    implementation. Comprehensive Center staff members follow up with
    consultation to individual schools by telephone and in person when
    feasible.

***Model schools and partnership sites.*** Of all the services provided
to schools and districts by the Comprehensive Center network, services
to model schools and partnership sites are by far the most intensive.
Three Comprehensive Centers have made "model schools" initiatives a
centerpiece of their portfolio of services. Typically, Comprehensive
Center staff develop an agenda for service in consultation with
individual schools. Although all model schools or partnership sites may
receive some standardized services (attending a summer institute
provided to all schools, for example), they also receive a fair number
of services tailored to their specific needs. Model schools initiatives
are among the few center-sponsored activities that include regular,
on-site over-the-shoulder technical assistance in individual schools.

-   **Region XIV Model Schools.** The Region XIV Comprehensive Center
    has provided services to eight schools with Title I schoolwide
    programs since 1996. These schools all share high rates of poverty,
    mobility, and disciplinary suspensions; high concentrations of
    targeted populations; and low achievement. Model schools set goals
    and priorities, develop and implement school improvement plans to
    meet these goals, evaluate results, periodically revise goals to
    incorporate challenges, and share experiences and expertise with
    other schools. Center staff, who function as coaches for
    school-level teams, help model schools plan their programs, provide
    information on research-based programs and professional development
    strategies, facilitate teacher collaboration, provide assistance
    with specific federal program provisions, assist with program
    evaluation, and document and disseminate the schools' successes.
    Model school institutes, sponsored quarterly by the centers, provide
    a forum for staff from the Model Schools to share ideas and
    expertise. Each model school works with one or two primary contacts
    at the center.

```{=html}
<!-- -->
```
-   **Region V Initiative Schools.** Since early 1998, the Southeast
    Comprehensive Assistance Center (SECAC) has provided intensive,
    needs-based, sustained, comprehensive services in one or two schools
    in each of the states in its region (a total of 8 schools). In
    general, the state team leader has the primary relationship with the
    schools, although he or she typically brokers resources for the
    school, such as recruiting other center staff to fulfill special
    roles. Typical center services to initiative schools include a needs
    assessment, creation of an improvement plan, and assistance in
    implementing the plan. Professional development is often a
    substantial aspect of the plan and its implementation. In the first
    few months of their work with the initiative schools, state team
    leaders visited the school in person twice a month. After the
    initial planning period, center staff visits to the school range
    from one to several times a month, although telephone and electronic
    communications are more frequent.

```{=html}
<!-- -->
```
-   **Region X Partnership Sites.** The Northwest Regional Educational
    Laboratory's (NWREL's) Comprehensive Center in Region X has
    recruited one district in each of three states in its region to
    become a partnership site. These partnerships are intended to build
    capacity in one school and to develop approaches to comprehensive
    reform that can be applied in other schools. The center negotiates a
    Memorandum of Agreement with each site that specifies how the center
    will provide services to the site, and how the site will contribute
    to advancing the center's work. Technical assistance is customized
    and includes such activities as developing culturally specific
    instruction; designing responsive programs; building
    interdisciplinary teams; and training staff to use data-based
    "profiling" to summarize, disaggregate, and analyze student
    assessment data for instruction. Partner sites have also
    participated in jointly sponsored center/NWREL conferences on
    integrating programs, gaining parent and community support,
    responding to cultural diversity, and using computers to profile and
    track schools' achievements. State liaisons at the Comprehensive
    Center work with each of the partnership sites; center staff members
    visit each school an average of once a month.

***Continuing consultation with states, districts, and schools.*** Many
Comprehensive Centers provide continuing consultation and
over-the-shoulder technical assistance to each of the states in their
region. Some centers divide their staff into state-based teams; staff
members spend all, or almost all, of their time providing services in
their assigned states. The Southeast Comprehensive Assistance Center
houses one staff member in each of the states in its region. Some
centers also provide consultation and technical assistance services to
groups of schools and districts in its region. Typically these
consultation services extend over a period of years. At the state level,
the assistance generally focuses on the implementation of ESEA and other
federal programs; at the local level, the assistance usually focuses on
needs assessment and schoolwide planning.

-   **Support to the Maryland State Department of Education:** The
    Region III Comprehensive Center has provided various kinds of
    support to federal programs of the Maryland State Department of
    Education (MSDE). The center assisted in the development of a new
    Even Start application process, helped develop a manual for projects
    making the transition from federal to local funding support, and
    documented meetings of Even Start program directors. The center
    worked closely with Maryland's Migrant Education Director to
    evaluate the effectiveness of the academic component of its migrant
    program, and to create a self-assessment tool for principals and
    teachers. Center staff attend MSDE's Title I administrators'
    meetings, and have conducted presentations on schoolwide programs
    and school reform, developed a scoring rubric for the Title I
    distinguished schools program, and assisted with consolidated
    planning efforts. Finally, the center helped Title IV staff to draft
    the program's RFP, delivered a presentation on the principles of
    effectiveness and conducting needs assessments, and assisted in
    identifying Safe and Drug-Free Schools program experts, consultants,
    and materials in preparation for a state conference.

```{=html}
<!-- -->
```
-   **Implementation of the Comprehensive School Reform Demonstration
    (CSRD) program in New York:** The New York Technical Assistance
    Center (NYTAC) has assisted the New York State Education Department
    with the implementation of CSRD by helping to develop and refine its
    application for CSRD subgrants and reviewing incoming proposals for
    grants. During the planning stage, the center worked with SEA staff
    to ensure that the program would address the needs of migrant
    students; similarly, the center reviewed the approved CSRD models to
    determine how well each addresses the educational needs of LEP
    students. Recognizing the need for continuing and collaborative
    assistance to make the program a success, the center formed the CSRD
    Think Tank, made up of staff members from the center, the SEA, the
    Effective Schools Consortium Network, the Northeast and Islands
    Laboratory at Brown University, and the New York City Board of
    Education. The think tank meets quarterly to discuss how to
    coordinate technical assistance among these service providers.
    Currently, the group is revising a scoring rubric for CSRD
    applications and designing an evaluation of CSRD schools across the
    state.

```{=html}
<!-- -->
```
-   **Support to critically low-performing schools in New Mexico:** The
    Southwest Comprehensive Center provides technical assistance to 11
    schools identified by the state's accountability system as
    "critically low performing." Center staff members meet with
    individual school teams to develop state-mandated Educational Plans
    for Student Success (EPSS), and provide training in leadership and
    comprehensive planning. With assistance from the center, the schools
    also convene meetings with external groups and community partners.

```{=html}
<!-- -->
```
-   **Alaska Onward to Excellence (AOTE):** Originally developed by the
    Northwest Regional Educational Laboratory (NWREL), AOTE trains
    leadership cadres made up of school and community members who set
    goals for student outcomes, gather data to assess progress toward
    those outcomes, and guide the development and adoption of
    educational programs. With the help of NWREL, the Alaska
    Comprehensive Regional Assistance Center (AKRAC) collaborates
    intensively over a period of at least two years with schools and
    districts selected because of their need for stable, effective
    support for educational improvement. During this period, staff help
    the communities set up broadly representative advisory groups that
    tackle the hard questions about what students should know and be
    able to do at the end of their elementary and secondary schooling
    and how to measure student progress. In the Snowbank School
    District, for example, certified teachers meet with village elders
    for several weeks in the summer, recording their ideas about
    curriculum goals and assessment strategies. AKRAC staff ran the
    initial training sessions for AOTE leadership cadres, and continues
    to provide consultation and support to local AOTE coordinators.

-   **Ebeye Public School Partnership.** The Ebeye Public School
    Partnership is a comprehensive school improvement initiative at the
    Ebeye Public School in the Marshall Islands. Originally, the local
    government asked the Pacific Center, the organization that houses
    the Pacific Comprehensive Regional Assistance Center, to take over
    and administer the school; instead, the Pacific Center chose to work
    with school staff to develop capacity and work with the community to
    develop a sense of ownership and responsibility. Pacific Center
    staff, including staff from the Comprehensive Center, have helped
    convene community meetings to create a school plan; helped structure
    the school into clusters; and organized and facilitated a four-week
    summer academic improvement program for 120 students that also
    served as a teacher training institute.

Information Dissemination and Short-Term Training

Most Comprehensive Centers report that they operate a "dual track" of
services---a group of high-intensity activities aimed at a relatively
small group of schools and districts, and a set of much lower-intensity
services that reach a wider audience. Comprehensive Centers engage in
short-term training and information dissemination activities in order to
establish a presence in the region and in order to reach a reasonable
proportion of the schools and districts that are eligible to receive
their services. As one center staff member described it:

> We operate with dual streams---one that works more closely with a
> smaller number of schools and another that tries to work with as many
> schools in the region as possible. Reaching out to the large numbers
> involves conferences and workshops. The work in the individual schools
> involves more fine-grained work.
>
> Many Comprehensive Center staff members reported that they feel a
> strong obligation to assist every customer who approaches them for
> help. As one center staff member put it, \"We never turn anybody away,
> ever.\" Short-term activities allow the centers to be responsive to
> requests from the field and to market more intensive services.

Examples of "one-shot" workshops and information dissemination
activities include the following:

-   In the summer of 1998, SECAC provided the Regional Institute on
    Curriculum Integration, which focused on integrating math, science,
    reading, and technology curriculum. The institute was a two-day
    training for school teams of three---including a principal, a math
    or science teacher, and a reading or technology teacher.

```{=html}
<!-- -->
```
-   Working closely with two Education Service Centers in Texas, the
    STAR Center in Region VIII co-sponsored two schoolwide institutes
    for school improvement teams from approximately 25 school districts.
    The STAR Center's *A Toolkit for Assessing and Revising the
    Integrated Campus Improvement and Title I Schoolwide Plan* was
    widely disseminated to help teams align their campus improvement and
    schoolwide plans.

```{=html}
<!-- -->
```
-   The Region VII Comprehensive Center has sponsored several planning
    meetings for administrators of Indian education programs, including
    a two-day training at the National Indian School Board (NISB)
    meeting. About 35 BIA representatives from inside and outside the
    region participated in the NISB training.

```{=html}
<!-- -->
```
-   The Northwest Regional Educational Laboratory's Comprehensive
    Center, Region X, maintains a resource library of materials for
    teachers and administrators in the region. Educators can contact the
    center's full-time information specialist by e-mail, Web site or
    over the telephone, and borrow resource materials for 21 days.
    Comprehensive Center staff members are available to answer
    questions. The center also distributes a monthly newsletter.

```{=html}
<!-- -->
```
-   The Region XI Comprehensive Assistance Center produces publications
    on numerous reform subjects, including standards-based
    accountability, parent education and multi-lingual communities, the
    special education needs of linguistically diverse students and
    communities, serving the region's Indian populations, and schoolwide
    comprehensive reform programs. The center disseminates these
    publications throughout its region.

Allocating Resources among High-Intensity and Low-Intensity Activities

The Comprehensive Centers maintain that they assign highest priority in
their work to their major initiatives and continuing work with
customers. In order to estimate how much of the centers\' work revolves
around their major, continuing initiatives, we asked center directors to
estimate how much of their staff time was allocated to this broad
category of service. Because staff time is by far the largest expensive
category for the Comprehensive Centers, estimating how centers allocate
staff time provides us with a reasonable estimate of how the centers
allocate their resources overall. In response, two-thirds of the centers
reported that 75-90 percent of their staff time was spent on
high-intensity assistance or major initiatives. The remaining third
estimated that they invested 50-75 percent of their staff time in
high-intensity service. Although it is possible that center directors\'
commitment to high-intensity service may have lead them to overestimate
the amount of time their staffs spend on these activities, these
estimates suggest that the centers are investing the majority of their
resources, at the very least, in high-intensity activities that involve
continuing contact with customers.

[]{#__RefHeading___Toc469391872 .anchor}What Services Do Customers
Receive?

Our survey and case study samples consist almost entirely of customers
of the center's major, high-intensity initiatives. Our survey data are
based on a representative sample of gatekeepers whose organizations
received a minimum level of service in 1998-99, and from a randomly
selected sample of participants in 30 major professional development and
consultation initiatives. Their experiences, coupled with our case study
data, provide some insight into the content and format of the services
received by the Comprehensive Centers' most important customers.

Common Content Strands

#### Multiple topics. Most gatekeepers reported that the Comprehensive Centers provided their organizations with assistance on a range of different topics. The average gatekeeper reported that his or her organization had received assistance on six different topics over the course of the last year. As a result, the percentage of gatekeepers reporting that their organization received assistance on a topic is high for almost every topic we listed in the survey (despite the fact that we included instructions to check the topic only if the topic was a [primary]{.underline} focus of the assistance received). Almost no Comprehensive Center gatekeepers reported that the Comprehensive Centers provided them with services on one topic alone. 

#### "Comprehensive" topics. On surveys, gatekeepers most commonly report that their organization received assistance on topics usually associated with comprehensive, standards-based reform. According to gatekeepers, the topics most commonly addressed by Comprehensive Center assistance are: (1) student assessment (with 49 percent of all customers reporting that their organizations received assistance in this area); (2) improving curriculum and instruction in reading and language arts (49 percent); (3) implementing schoolwide programs (48 percent); (4) challenging standards and accountability (45 percent); and (5) analyzing student achievement data and interpreting results (43 percent) (Exhibit III.1). Other topics, such as using educational technology, creating safe and drug-free learning environments, and addressing issues raised by ED Integrated Review Teams, were much less common. The large number of gatekeepers reporting that their organization received assistance on topics related to the "core issues" of schooling, such as improving curriculum and instruction and student assessment, indicates that many of the centers' services have indeed been comprehensive. 

***Categorical topics.*** Categorical assistance---which includes help
in carrying out the provisions of specific ESEA programs and addressing
the needs of specific student populations---is also a major focus of
center services. Exhibit III.2 show that help in addressing the needs of
special student populations was one of the most commonly reported topics
of assistance, with almost half of the Comprehensive Centers' customers
reporting that their organizations received assistance in this area.
Customers reported that they received help in serving LEP students more
often than any other student population, with almost one-quarter of
gatekeepers reporting that their organization received assistance with
LEP students.

Examples of assistance designed to address the needs of special
populations include the following:

-   In both North Carolina and the Riverside City Schools, the Region IV
    Comprehensive Center at AEL and the Region VII Comprehensive Center
    delivered training to mainstream teachers coping for the first time
    with large numbers of limited English proficient students in their
    classrooms. In both regions, the training introduced teachers to
    research on language acquisition and provided them with techniques
    for integrating limited English proficient students into their
    regular classroom lessons.

-   For the Hawaii Department of Education, the Pacific Comprehensive
    Regional Assistance Center has provided numerous services addressing
    the needs of limited English proficient children and their parents.
    The center has helped translate materials for non-English speaking
    parents; coordinated training for district and school staff in the
    state's Language Assessment Scales, which determine English language
    proficiency as spoken and heard; hosted a teleconference for the
    state's ESL teachers, specialists, resource teachers, and
    administrators; and connected the SEA with contacts throughout the
    Pacific who can facilitate the transition of immigrant children from
    the Pacific into Hawaiian schools.

Although one half of all the centers' customers reported that they
received help with serving special populations of students, this help
was not necessarily targeted at districts and schools with high
enrollments of LEP, Indian, or migrant students. In other words,
districts that enroll high proportions of LEP, Indian, or migrant
students were no more likely than other districts to report that they
received assistance with serving special populations of students. In
keeping with their mission, the centers do not appear to be seeking out
districts and schools with large numbers of LEP, Indian, or migrant
students simply to provide assistance with those populations.

**Exhibit III.1**

Percent of Gatekeepers Reporting That the Center's Products and Services

Addressed Certain Topics

(n=577)

  ------------------------------------- ---------------------------------
  **Since the beginning of 1998, has    
  your organization received any        
  products or services from the         
  Comprehensive Center that addressed   
  the following topics? (Check "yes"    
  only if the topic was a               
  [primary]{.underline} focus of the    
  products or services you received.)** 

  **[Topics]{.underline}**              **Percent of Gatekeepers
                                        Reporting That the [Centers'
                                        Assistance Addressed Certain
                                        Topics]{.underline}**

  Improving curriculum and instruction  49
  in reading/language arts              

  Student assessment                    49

  Implementing schoolwide programs      48

  Addressing the needs of special       47
  student populations                   

  Challenging standards; accountability 45

  Analyzing student achievement data    43
  and interpreting results              

  Program evaluation                    40

  Adopting and implementing particular  39
  model programs                        

  Expanding parent/family involvement   38

  Implementing ESEA programs            35

  Improving the quality of bilingual    33
  education and ESL programs            

  Consolidating or integrating          27
  federally funded education programs   

  Improving curriculum and instruction  25
  in mathematics                        

  Implementing educational applications 21
  of technology                         

  Improving curriculum and instruction  20
  in other core academic areas          

  Creating safe and drug-free learning  17
  environments                          

  Issues identified by ED Integrated    6
  Review Teams                          
  ------------------------------------- ---------------------------------

Exhibit reads: Forty-nine percent of gatekeepers reported that their
organization received Center products or services that addressed
improving curriculum and instruction in reading/language arts.

**Exhibit III.2**

Percent of Gatekeepers Reporting That the Center's Products and Services
Addressed Certain Categorical Topics

(n=577)

+-------------------------------------+--------------------------------+
| **Since the beginning of 1998, has  |                                |
| your organization received any      |                                |
| products or services from the       |                                |
| Comprehensive Center that addressed |                                |
| the following topics? (Check "yes"  |                                |
| only if the topic was a             |                                |
| [primary]{.underline} focus of the  |                                |
| products and services you           |                                |
| received.)**                        |                                |
+-------------------------------------+--------------------------------+
| #####                               | ###                            |
|                                     | ###  {#section-12 .unnumbered} |
| ##### [Topics]{.underline}          |                                |
|                                     | ######                         |
|                                     | Percent of Gatekeepers Reporti |
|                                     | ng That {#percent-of-gatekeepe |
|                                     | rs-reporting-that .unnumbered} |
|                                     |                                |
|                                     | ###### the Centers' Assistan   |
|                                     | ce Addressed {#the-centers-ass |
|                                     | istance-addressed .unnumbered} |
|                                     |                                |
|                                     | ###### [Certain Categorical T  |
|                                     | opics]{.underline} {#certain-c |
|                                     | ategorical-topics .unnumbered} |
+-------------------------------------+--------------------------------+
| Addressing the needs of special     | 47                             |
| student populations                 |                                |
+-------------------------------------+--------------------------------+
| Limited English Proficient          | 18                             |
+-------------------------------------+--------------------------------+
| Migratory                           | 8                              |
+-------------------------------------+--------------------------------+
| Indian                              | 7                              |
+-------------------------------------+--------------------------------+
| High-poverty                        | 1                              |
+-------------------------------------+--------------------------------+
| Special needs                       | 1                              |
+-------------------------------------+--------------------------------+
| Other                               | 1                              |
+-------------------------------------+--------------------------------+
| Unspecified                         | 19                             |
+-------------------------------------+--------------------------------+
| Implementing ESEA programs          | 35                             |
+-------------------------------------+--------------------------------+
| Title I, Part A                     | 12                             |
+-------------------------------------+--------------------------------+
| Title I, Part B (Even Start)        | 1                              |
+-------------------------------------+--------------------------------+
| Title I, Part C (Education of       | 1                              |
| Migratory Children)                 |                                |
+-------------------------------------+--------------------------------+
| Title II                            | 1                              |
+-------------------------------------+--------------------------------+
| Title IV                            | 1                              |
+-------------------------------------+--------------------------------+
| Title VI                            | 2                              |
+-------------------------------------+--------------------------------+
| Title VII                           | 3                              |
+-------------------------------------+--------------------------------+
| Title IX                            | 1                              |
+-------------------------------------+--------------------------------+
| Other programs                      | 1                              |
+-------------------------------------+--------------------------------+
| Unspecified ESEA                    | 18                             |
+-------------------------------------+--------------------------------+

Exhibit reads: Forty-seven percent of gatekeepers report that they
received assistance that addressed the needs of special populations.

About a third of gatekeepers reported receiving assistance in
implementing at least one ESEA program. Respondents most frequently
identified Title I as the program for which they received assistance.
About a third of gatekeepers also reported that they received assistance
in improving the quality of bilingual education and ESL programs.

##### Many centers rely on staff from their partner organizations---many of which served as technical assistance centers under ED's previous system---to provide categorical, program-specific services; staff from these partner organizations tend to specialize in programs for migrant, LEP, and American Indian children. Examples of categorical assistance include:

-   The Southwest Comprehensive Center developed and disseminates *A
    Guide for Planning Indian Education Programs: Meeting Federal
    Program Requirements.* It has also successfully advocated for the
    establishment in one state's education agency of a BIA staff
    position and the re-establishment of one state's Indian Education
    Association.

```{=html}
<!-- -->
```
-   The Region IV Comprehensive Center at the Appalachia Education
    Laboratory (AEL) helped the state's Title I office establish its
    measure of adequate yearly progress as required by the Title I
    legislation. It worked with state program staff to streamline the
    program's process for evaluating progress made by Title I schools;
    subsequently, the center trained LEA Title I coordinators to use
    data to evaluate their progress and identify problem areas in their
    schools.

-   The Region IV Center also developed a series of training sessions to
    provide local Safe and Drug-Free Schools subgrantees with guidance
    on implementing the program's new principles of effectiveness.

#### 

## Volume of Services

## The Comprehensive Centers' core customers---the states, districts, and schools that they identified on their customer lists---receive a relatively high volume of services. For example, Exhibit III.3 shows that 87 percent of gatekeepers reported that their organization had received professional development services in the last year; 22 percent of gatekeepers reported that they received professional development services five or more times. Almost two-thirds of gatekeepers reported that their organizations had received on-site consultations in the last year; 20 percent reported that they had received visits from center staff five or more times. 

Most participants in center-sponsored activities also report that they
receive some form of follow-up to the services they receive from the
centers. Participants in center-sponsored activities reported, for
example, that they received additional materials from the centers (67
percent); additional workshops, training, and meetings, beyond what was
originally scheduled (62 percent); follow-up by telephone (45 percent);
and visits on-site (42 percent) (Exhibit III.4). Because most
participants in center-sponsored activities are based in schools, this
finding is interesting because it suggests that the centers are
investing a significant amount of effort in visiting individual schools
and classrooms.

In addition to the centers' own follow-up efforts, more than half of the
participants reported that they received assistance on a topic related
to the activity from another source. Participant reports on follow-up,
combined with gatekeeper reports on the quantity of services received
and the variety of topics addressed, demonstrate that the one-shot
"parachute drop" is a relatively rare occurrence among established
customers.

There were almost no statistically significant differences among state,
district, and school participants in the amount of follow-up they
received from the centers. The single exception was follow-up by
telephone; not surprisingly, school participants, most of whom are
teachers, were less likely to report that they received follow-up
assistance in this way.

In addition to services they provide themselves, 59 percent of
gatekeepers reported that the centers referred them to other sources of
assistance as well. Gatekeepers reported that centers referred them most
often to SEAs (with 51 percent of those who received referrals reporting
that they were referred to an SEA); regional educational laboratories
were another common object of referrals (45 percent). Exhibit III.5
shows that centers were more likely to refer state-level gatekeepers to
regional laboratories than customers in other types of organizations;
local-level customers were more likely to be referred to SEAs or
intermediate education agencies than state-level customers.

Role of Technology in Delivery of Services

In general, the Comprehensive Centers do not yet rely heavily on
advanced telecommunications or other forms of technology to deliver
services. All 15 of the Comprehensive Centers maintain Web sites, some
of which provide access to newsletters and center-developed products.
Most centers also rely on e-mail to communicate with customers; 52
percent of state-, district-, and school-level gatekeepers reported that
they had received technical assistance services via e-mail. Other modes
of communication, however, were less common:

![](media/image9.wmf){width="9.3in" height="7.14375in"}

![](media/image10.wmf){width="9.3in" height="7.9in"}

![](media/image11.wmf){width="9.3in" height="8.015972222222222in"}

-   About a quarter of center customers reported that they visited their
    center's Web page or received other electronic products from the
    center. Only 7 percent of customers reported that they visited the
    Web page five or more times over the course of the year. The small
    number of repeat visitors suggests that the center Web pages are not
    yet used as a mechanism for communicating with established customers
    or for delivering services.

-   Nineteen percent of center customers reported that their
    organizations have received services from the Comprehensive Center
    via teleconference. Five percent reported that their organizations
    have received services via videoconference.

-   Nineteen percent reported that their organizations have received
    services via audio or videotaped presentations

Some examples of the ways in which the centers use technology to provide
services to customers include the following:

-   Under a Goals 2000 grant, the Southwest Comprehensive Center and its
    host organization have involved 20 New Mexico teachers in a
    "Standards Online Group." Each teacher in the group received a
    laptop computer so that they can communicate with each other about
    the state's standards and their implications for instruction.

-   The New England Comprehensive Assistance Center maintains five
    listservs. They include a listserv for members of the centers'
    schoolwide network, members of the Data Strategies Initiative, Safe
    and Drug-Free Schools Coordinators in the region, principals in the
    schoolwide network, and members of the Reading Success Network, in
    Region I and nationwide.

-   The Region VII Comprehensive Center pilots several online products,
    including Profiler, a database software that school technology
    specialists can use to assess and prioritize internal technology
    needs and to build technology capacity among teachers. Another
    product is a language instruction CD-ROM that teaches the Choctaw
    language.

Of those customers who had received services via some form of advanced
telecommunication, 60 percent said that the technology allowed the
center to deliver the information more quickly and effectively. Some
also said that the use of the technology enhanced the quality of the
service by making new information easier to understand (48 percent), or
by making presentations more interesting

(44 percent). Of all center products and services, customers were least
likely to be satisfied with their access to center Web pages or other
electronic products. Twenty-nine percent of center customers reported
that they were "Somewhat satisfied" or "Not at all satisfied" with the
accessibility of those services.

These examples suggest that the Comprehensive Centers have begun to
explore some possible applications of technology for the purposes of
disseminating information. Such examples are relatively rare across the
Comprehensive Center network, however, and this fact, coupled with
customers' reporting of the level of technology used by the centers in
delivering services, suggests that the centers are not likely to adopt
wide-scale use of technology and advanced telecommunications as a
substitute for face-to-face contact with customers any time in the near
future.

[]{#__RefHeading___Toc469391874 .anchor}How Do Services to State and
Local Customers Differ?

Centers cultivate different kinds of customer relationships at the state
level than they do at the local level. As a result, their portfolio of
services to states differs from their portfolio of services to districts
and schools.

Targeting

A key difference among state and local customers is that local contacts
are more likely to be screened by the centers for their capacity to
implement reform. Because the Comprehensive Centers are limited in their
ability to serve all districts and schools within their regions, centers
conduct both formal and informal screenings to determine districts' and
schools' capacities to use technical assistance and to make changes. For
example:

-   Schools wishing to participate in the Region XII Comprehensive
    Center's Reading Success Network (RSN) must submit an application.
    The application requires each school to (1) identify an RSN school
    team; (2) complete a site process data sheet describing the school's
    schoolwide and classroom intervention strategies, assessment
    indicators, grade levels, and expectations of RSN; (3) submit a copy
    of the school's accountability report and reading/literacy component
    plan; and (4) demonstrate school and district commitment to RSN.

```{=html}
<!-- -->
```
-   Similarly, to target its services in North Carolina, the Region IV
    Comprehensive Center asked districts to identify "focus schools,"
    schools with extensive needs for technical assistance that were
    prepared to receive intensive help. According to a center staff
    member, "Even if other schools are more needy, if they're not ready,
    it's a waste of our time. . . . Being ready for change is not a
    straight line; it's a critical mass."

In comparison, the expectation that the centers will serve every state
in their region requires that the centers adapt to each state's
readiness for reform, rather than using readiness as a criterion for
targeting services. A state's capacity to use technical assistance,
however, affects the level and types of services delivered by the
center.

Volume of Services

State-level gatekeepers tend to have more frequent contact with the
Comprehensive Centers than their local counterparts do. A major thrust
of center services to states includes consultations with state liaisons,
over-the-shoulder technical assistance, and other activities designed to
build the capacity of SEA staff members, and which result in a fairly
frequent contact with SEA staff. Many Comprehensive Centers assign
individual staff members to a particular state to serve as liaison with
the center and to broker services in that state.

Our survey data show that state gatekeepers are more likely to report
that their agencies receive various types of assistance five or more
times in a year than are gatekeepers at the district or local levels.
For example, Exhibit III.6 shows that SEA gatekeepers were more likely
to report that they received on-site consultations, consultations by
e-mail or telephone, collaborating or networking with other programs,
and help convening special events.

On a surprising note, however, state-level gatekeepers were somewhat
*less* satisfied with their access to center products and services than
were district-level gatekeepers. Exhibit III.7 shows that across all
types of services, district-level gatekeepers were more likely to report
that they were "very satisfied" with their access to the center's
products or services than were state-level gatekeepers or school-level
gatekeepers. Although SEA gatekeepers receive higher levels of service,
on average, they may have higher expectations for a particular level of
services as well, because of their status as state-level customers and
the way the centers market their services to states. District-level
gatekeepers may experience a better fit between the services they do
receive and their expectations of the centers.

Gatekeepers at various levels of the educational system reported
relatively few differences in the topics of the assistance they
received. Among the few significant differences in survey responses,
state-level gatekeepers were more likely to report that they received
assistance with the implementation of ESEA programs, while district- and
school-level contacts are more likely to report that they received
technical assistance related to comprehensive school improvement.
Fifty-four percent of state gatekeepers, for example, reported that they
received assistance with the implementation of ESEA programs, compared
with 34 percent of district gatekeepers and 26 percent of school
gatekeepers. By contrast, half or more of school- and district-level
gatekeepers reported receiving assistance on improving curriculum and
instruction in reading/language arts, compared with 36 percent of
state-level gatekeepers.

![](media/image12.wmf){width="9.3in" height="7.4in"}

![](media/image13.wmf){width="9.3in" height="7.5in"}

Content of Services

States are more likely than locals to set the direction of center
services; in contrast, locals rely on centers to take the lead in
providing services, often relying on centers' expertise in specific
content areas and with established reform models and strategies. In most
places, SEA and center staff members spend considerable time developing
workplans detailing the center's portfolio of services to the state and
establishing priorities for the center. This process takes place both
formally and informally. For example, the Region V Comprehensive Center
and the Mississippi Department of Education established a coordinating
committee that meets several times each year to identify the state's
technical assistance needs and develop a service delivery plan for the
state. Staff members from the various federal programs sit on the
committee. A respondent from the NYSED described the Region II
Comprehensive Center's responsiveness to his state's needs:

> They \[center staff members\] were especially effective because they
> did not have a work plan when they came in. They didn't say we had to
> work with standards and testing . . . the mandate. They asked us,
> 'What are your needs?' and we told them. They came back six weeks
> later with a plan. It's an annual work plan, which they reviewed with
> us prior to it being established.

Unlike their state counterparts, local clients are more likely to seek
the center's assistance in content areas, reform models, and strategies
in which the center has demonstrated expertise. Such activities enable
centers to build on lessons learned elsewhere. For example:

-   The Region XI Comprehensive Center tapped into its expertise in
    schoolwide programs to sponsor multi-session institutes on
    schoolwide program planning for school teams from the Monterey
    Peninsula Unified School District. The center had used a similar
    training strategy to host schoolwide institutes in other districts
    in northern California and had published a guide to schoolwide
    implementation, *Schoolwide Reform: A New Outlook*.

```{=html}
<!-- -->
```
-   In its services to three clusters of districts in North Carolina,
    the Region IV Comprehensive Center adapted its Lab School model,
    which (1) models the use of specific strategies; (2) provides
    opportunities for application and experimentation; and (3) offers
    follow-up support. The services focused on the special needs of LEP
    students and involved consciousness-raising activities and
    discussion related to theories of language acquisition and
    appropriate instructional activities for LEP students. The center
    worked with one cluster to deviate slightly from the model so that
    the districts were able to offer the Lab School in conjunction with
    a regular summer academy for ESL teachers.

Other examples of established programs adapted by centers to meet local
needs include Alaska Onward to Excellence, Cognitively Guided
Instruction, the Reading Success Network, and Continuous Progress/Model
Schools.

**[Allocating Resources Among States, Districts, and
Schools]{.underline}**

Across the entire center network, Comprehensive Centers allocate their
resources approximately equally among states, districts, and schools. As
we have noted, most major center initiatives, which include both
professional development activities and consultation, are designed for
and targeted to school-level staff. In addition, the Comprehensive
Centers work with larger numbers of gatekeepers in schools and districts
than they do in states. On the other hand, our survey data and our
observations in case study sites indicate that state-level gatekeepers
have more frequent contact with center staff and command a slightly
higher volume of services. Although it is impossible to make a precise
estimate of the amount of staff time and other resources centers devote
to serving each level of the system, the centers' own descriptions of
their work, our survey data, and our observations in case study sites
suggest that all three major groups of customers---states, districts,
and schools---command a significant portion of the centers' time and
resources.

Allocations of time and center resources among states, districts, and
schools do vary by center, with some centers adopting a more
state-intensive approach to serving their regions than others. However,
these differences are not generally reflected in the customer lists that
the centers compiled, or in the reports of center customers. In other
words, SEA customers in regions where the center has adopted a strong
state-focused approach to serving the region are no more likely to
report frequent contact with the Comprehensive Center and a high volume
of services than are SEA customers in regions where the center has not
adopted a state-focused approach.

What Services Do the Centers Provide to the U.S. Department of
Education?

Although they are not a central focus of this evaluation, the services
provided by the 15 Comprehensive Centers to the U.S. Department of
Education occupy a prominent place in their portfolio of activities. In
fall 1998, for example, a memorandum from ED to center directors set out
the expectation that the centers would devote up to 10 percent of their
resources to carrying out ED projects. The centers are also available to
take on supplemental contracts for additional projects from ED.

One of the most important examples of the service that the centers
provide to ED is their role in planning and supporting three Improving
America's Schools (IAS) conferences each year. Each year one to three
centers are responsible for some of the conferences; centers receive a
supplement of \$25,000 from ESEA Title XIII funds to cover the costs of
staff time devoted to planning the event. By all accounts, however, this
stipend does not cover all of the staff time spent on IAS-related
activity. In addition to planning and supporting IAS conferences, some
examples of the work that the centers do for ED include the following:

-   The Comprehensive Centers work with the regional educational
    laboratories on the implementation of the Comprehensive School
    Reform Demonstration program (CSRD). To expand their work in
    supporting the implementation of CSRD in their regions, the centers
    receive an additional supplement of \$50,000 from CSRD funds. The
    centers work with states on setting up an RFP process, organizing
    showcases of CSRD models, assessing districts' capacity to support
    CSRD schools, and helping schools review data to select the most
    appropriate and effective reform model.

-   The Southern California Comprehensive Assistance Center received a
    supplemental grant award of \$60,000 from Safe and Drug-Free Schools
    funds to organize the School Security Chiefs forum. The three-day
    forum was attended by the heads of school security of large urban
    and suburban school districts across the United States and was also
    attended by representatives from the Departments of Justice and
    Education.

Summary

Although they provide one-shot services from time to time, the
Comprehensive Centers generally concentrate their time and resources on
providing long-term technical assistance to states, districts, and
schools. In fact, two-thirds of centers reported that 75-90 percent of
their staff time was spent on high-intensity services. The centers'
major initiatives are in the areas of professional development, services
to model schools and partnership sites, and consultations with clients.
Several centers targeted schools for intensive services based on narrow
demographic or performance criteria, such as the list of low-performing
schools identified by a state. Most gatekeepers reported that the
centers provided organizations with assistance on a range of topics,
including those associated with comprehensive, standards-based reform as
well as those concerning categorical programs. State gatekeepers are
more likely than their local counterparts to have more frequent contact
with the centers and set the direction of center services. However,
state-level customers were generally less satisfied with their access to
center products and services than were local customers. centers may
provide lower-intensity services, such as information dissemination and
sort-term training, to establish a presence in a particular region or to
reach a wider audience. In general, the Comprehensive Centers do not
rely heavily on advanced telecommunications to deliver services,
although they do maintain Web sites and use e-mail to communicate with
customers.

[]{#__RefHeading___Toc469391880 .anchor}IV. QUALITY OF COMPREHENSIVE
CENTER SERVICES

In Title XIII of ESEA, Congress finds that "high-quality technical
assistance can enhance the improvements in teaching and learning
achieved through the implementation of programs under \[ESEA\]" and
charges the Comprehensive Centers with providing technical assistance
using "the highest quality strategies available" \[P.L. 103-382, Section
13001(1), and Section 13102(3)\]. This chapter of the report examines
certain dimensions of quality in the work of the Comprehensive Centers.
These dimensions of quality, which we have developed from the literature
on professional development and technical assistance, from ED's own
expectations of the program, and from the experience of customers in the
field, provide a framework for assessing Comprehensive Center services.
They include:

-   **Comprehensiveness.** The Comprehensive Centers' particular mission
    as a federal technical assistance program is to provide assistance
    to states, districts, and schools in using federal programs to
    support comprehensive, standards-based reform. Consistent with the
    vision of standards-based reform contained in other parts of the
    law, Comprehensive Center assistance is intended to support the
    integration of categorical programs in the service of state and
    local goals for student achievement.

-   **Accurate content grounded in research.** Information or assistance
    that is inaccurate or out of date is not helpful to customers. In
    addition, technical assistance services should be based on the best
    research and practitioner knowledge available on how students learn
    and how organizations change.

-   **Responsiveness to customer needs and interests.** The literatures
    on both professional development and technical assistance maintain
    that individuals are much more likely to benefit from services that
    address a clearly defined local need or interest.

-   **Adequate scope and intensity.** Changing the behavior of
    individuals and organizations is not a simple undertaking. Unless
    the goal of the assistance is to convey relatively simple
    information (like the timetable for implementing particular
    provisions of a new law), assistance must be relatively intensive
    and continuing.

-   **Adequate staff expertise.** Individual technical assistance
    providers may have more or less relevant expertise, as well as
    interpersonal skills that enable them to work more or less
    effectively with particular customers. These variations in
    individual expertise make a difference to the quality of service.

Our findings on the quality of Comprehensive Center services are based
on three sources of data: (1) our observations of the design and
implementation of center services in the case study sites; (2)
customers' ratings of the quality and utility of the services they
received; and (3) interviews with Comprehensive Center staff members.

[]{#__RefHeading___Toc469391881 .anchor}Are Services Comprehensive?

As the reauthorized ESEA seeks to shift the emphasis from the
implementation of parallel programs for target populations to
comprehensive reforms that support improved teaching and learning for
all students, especially those served under ESEA, the focus of technical
assistance in support of ESEA has shifted from the performance of a
single program to the work of the entire school or the system that
supports the school.

In interviews, center staff maintained that the assistance provided the
Comprehensive Center program is substantially different from the work
done in the past under the Department's older categorical system. This
was especially true for staff members who had worked for one of the
antecedent centers: they argued that the change in focus from
categorical to comprehensive assistance has had a significant impact on
their work. Center staff members described comprehensive assistance as
training and consultation geared to whole school reform and systems
reform, with special attention to the integration of programs and
funding streams. One Comprehensive Center staff member summed up the
transition to a new kind of comprehensive assistance in this way:

####### It's been really quite dramatic to go from a categorical, narrow, "in-our-box" approach to very systemic, broad-based, whole school approach. This has resulted in a better approach for impacting schools, \[but\] it has also been a process, and a challenge, to have different federal program coordinators sit down together and work to realize that whether a child is Title I, LEP, or migrant, he or she is the same child.

In practice, the Comprehensive Centers have responded to this charge to
provide comprehensive assistance in a variety of ways. From their entire
portfolio of activities, the categories of services that most clearly
advance "comprehensive" goals include the following: (1) assisting
schools to develop Title I schoolwide programs; (2) assisting schools
and districts to conduct needs assessments and use data to drive
decision making; and (3) assisting states and districts to develop
consolidated plans for ESEA programs.

## Schoolwide Program Planning as a Comprehensive Strategy

Assisting schools in the planning and implementation of schoolwide
programs is a natural fit for the Comprehensive Centers. The planning
process, when implemented successfully, helps school staff to think
comprehensively about the strengths and weaknesses of their existing
programs and to improve learning for special populations of students by
drawing on the resources of the entire school, not just categorical
programs. In addition, many schools became eligible to operate
schoolwide programs for the first time in 1994, thus creating a ready
market for Comprehensive Center services.

All the Comprehensive Centers have provided some training and technical
assistance to schools and districts planning schoolwide programs. Most
of the centers' activity has focused on raising awareness of the
schoolwide program option, disseminating information about planning
requirements, and supporting schools as they plan. Supporting the
planning and implementation of schoolwide programs is a central focus of
activities, consuming a significant proportion of staff time, for at
least a third of the centers. Examples of this category of assistance
include:

-   The New England Comprehensive Assistance Center (NECAC) facilitates
    a network of Title I schoolwide program schools. The keystone event
    for the network is an annual Congress for teams from member schools.
    Teams come to this Congress with a school action or improvement plan
    on which to work, and NECAC provides facilitators to help them with
    the planning process and to provide follow-up coaching on
    implementation during the school year.

```{=html}
<!-- -->
```
-   The STAR Center co-sponsored two schoolwide institutes with the
    state Education Service Centers and developed *A Toolkit for
    Assessing and Revising the Integrated Campus Improvement and Title I
    Schoolwide Plan,* which assists schools in aligning their schoolwide
    program plans with the campus improvement plans required by the
    state of Texas. Additionally, the center worked closely with one
    large city district in the planning and implementation of schoolwide
    programs and in applying the lessons learned from center research on
    successful schoolwide programs in Texas.

```{=html}
<!-- -->
```
-   The Region XI Comprehensive Assistance Center has sponsored
    schoolwide institutes in which more than 250 schools have
    participated. The institutes are organized around a center product
    called *Schoolwide Reform: A New Outlook.* This product has also
    been sent to 1,600 individuals who have requested it. To improve
    parent involvement in schoolwide planning, the center has developed
    information for parents in five languages.

```{=html}
<!-- -->
```
-   The Southern California Comprehensive Assistance Center collaborates
    with the state's own network for supporting schoolwide programs,
    called the Statewide System of School Support, to provide a program
    of coaching and technical assistance to the county's schoolwide
    programs. The assistance focuses on standards and assessments,
    comprehensive school planning, governance, and student-centered
    decision making.

## Needs Assessment, Strategic Planning, and Using Data

Comprehensive reform requires that districts and schools think more
strategically about how they will use federal resources to improve the
regular education of children served under ESEA. This emphasis on the
integration of federal resources and whole school reform requires more
sophisticated planning on the part of schools and districts. As a
result, school-level and district-level needs assessment and planning
are central components of many Comprehensive Center activities. One
Comprehensive Center staff member summed up this approach to assistance
in the following terms:

> Comprehensiveness is about noticing what's going on and not going on
> in terms of students' achievement, focusing on resources inside and
> outside, and paying attention to the external environment.

Coupled with this new emphasis on strategic planning and needs
assessment is a strong emphasis on analyzing and using student
achievement and demographic data to drive decision making. Comprehensive
Center staff point out that disaggregating data becomes especially
important as a way to ensure that the students who are meant to be
served under ESEA---poor children, LEP children, or American Indians,
for example---benefit under whole school reform efforts. As a result,
many Comprehensive Centers have incorporated training in the analysis
and use of data into their repertoire of services.

Examples of training in needs assessment, data use, and setting
benchmarks for tracking progress include the following:

-   Alaska Onward to Excellence, sponsored by the Alaska Comprehensive
    Regional Assistance Center, trains leadership cadres of community
    members and school staff to evaluate student performance data, set
    goals for improvement, and monitor progress over time.

```{=html}
<!-- -->
```
-   The Northwest Regional Educational Laboratory's Comprehensive
    Center, Region X, sponsored a summer institute for its partnership
    sites on the use of data to drive school decision making. School and
    district staff learned to use Excel spreadsheets and SPSS to
    summarize, disaggregate, and analyze student achievement data.
    During the school year, Comprehensive Center staff consulted with
    school team members on assembling elaborate "school profiles," which
    one partnership site used in its state accreditation process. The
    center has modified school "profiling" tools, created by the
    regional educational laboratory's School Improvement Program, to
    help schools and districts initiate schoolwide program needs
    assessment and implementation evaluation.

## Consolidated Planning at the State and Local Levels

At the state level, a few centers worked with states in their region in
the years immediately following reauthorization to help consolidate
their federal programs. In these few states, the centers' understanding
of the reauthorized ESEA combined with their knowledge of school change
provided states with "a guiding vision of what comprehensive" meant and
of how comprehensive services could work. According to an SEA program
director in a state where the Comprehensive Center had been particularly
active in helping to implement the reauthorized ESEA:

> The center served a pivotal role during the transition to
> comprehensive services. It was a dramatic change in all the subtitles.
> \[The center's\] role was to bring information about this change to
> us. Instead of us taking our time and resources to learn all the
> information, he was here physically when we needed him. Technically,
> we could have done it for ourselves, but his involvement delivered the
> information in a more expedient and efficient way.

Several Comprehensive Centers convene meetings of state-level federal
program administrators on a regular basis. For example, the New England
Comprehensive Assistance Center convenes federal program directors from
the six states in its region on a quarterly basis for a
one-and-a-half-day meeting. The first day of the meeting is devoted to
discussion of cross-program coordination; on the second day,
participants split into role-alike groups.

At the local level, a few centers have also developed tools and training
to assist in local consolidated planning:

-   The Region III Comprehensive Center worked with the Delaware
    Department of Public Instruction to implement the quality review
    component of the local consolidated application and school
    improvement process. The center assisted in the design of questions
    for the SEA to use as part of the reviews, observed SEA staff
    interacting with LEA staff during pilot reviews, and gave feedback
    and recommendations to the SEA.

```{=html}
<!-- -->
```
-   The Region IV Comprehensive Center at the Appalachia Regional
    Educational Laboratory provided assistance to the Tennessee
    Department of Education in putting together a consolidated plan for
    Title I, Title II, and Title VI. The center and the SEA piloted the
    redesigned plan in five districts, with center staff providing
    training to district officials in developing effective consolidated
    applications. With assistance from the center, each site collected
    and analyzed data to identify district needs.

```{=html}
<!-- -->
```
-   The STAR Center developed the *Resource Guide for Coordinating
    Funding* and provided training to the state's Education Service
    Center staff, district administrators, and school administrators.
    Also, in partnership with one of the Education Service Centers in
    its region, it created a simulation game, called *Show Me the
    Money,* that it uses in many of its training activities.

## Limitations of "Comprehensive" Assistance 

##### 

##### In interviews, several Comprehensive Center directors reported that their commitment to providing "comprehensive" services has led them to make concrete trade-offs in their capacity to provide categorical services as well. As we reported in Chapter III, Comprehensive Centers continue to provide, and their customers continue to request, assistance related to the implementation of specific categorical program provisions and to the needs of special populations of students. At least some of the Comprehensive Centers have found that it is impossible to maintain the specialized program expertise necessary to satisfy all of these requests for assistance. One experienced provider described this tension between categorical and comprehensive assistance as a "conundrum:"

##### 

> Can you deal with these two pieces simultaneously? The problem is,
> when you are talking about categorical programs, you're talking about
> taking an advocacy and protector role. For 20 years we have
> conditioned bureaucracies to advocate and protect . . . but
> comprehensive reform has to have a forward thrust, \[and an
> assurance\] that the system is in fact doing the right things *vis 
> vis* those designated groups.

These choices have not always been received well among the centers'
potential customers. Center staff members, especially those who work
with schools serving Indian children, migrant children, and limited
English proficient children, report that in the field they encounter
some resistance to comprehensive approaches to assistance. Some
customers strongly object to blurring the lines across traditional
categorical programs, fearing that "their" students will fall through
the cracks. In addition, as we will see later in this chapter, some
customers have been severely critical of the limitations of the
comprehensive approach and of the services they have received as a
result. According to one center staff member:

> We run into the barrier of categorical assistance. . . . Everyone's
> open until you step on someone's categorical toes. We run into the
> fear that somehow, if we do schoolwide program planning, \[that\]
> "These children won't get serviced," but it's also saying, "My piece
> of the pie will disappear."

In an attempt to balance their charge to provide comprehensive
assistance with the demand for categorical services that continues to
exist, most Comprehensive Centers have allocated at least a portion of
their resources to meeting purely program-specific or
population-specific needs.

Our review of the centers' portfolio of services indicates, however,
that the centers have made a credible effort to organize their services
to the field around "comprehensive" initiatives---those that promote
strategic planning, data-based needs assessment, and integration of
program funds to support standards-based goals for student learning.
Most services include some component designed to promote the goals of
comprehensive, standards-based reform. As we have seen, most
Comprehensive Center customers report receiving assistance on topics
closely related to comprehensive reform as envisioned in ESEA, and to
the activities that the centers themselves describe as
comprehensive---student assessment, implementing schoolwide programs,
challenging standards and accountability, and analyzing student
achievement data and interpreting the results. For these reasons, we
would argue that the Comprehensive Centers have responded effectively to
the mandate contained in their authorizing legislation.

**Is the Content of the Assistance Reported to Be Accurate and Grounded
in Research?**

This evaluation drew on two sources of data to examine the quality of
the content of center services: (1) the centers' own descriptions of the
research base that underlies many of their products and services; and
(2) customers' ratings of the accuracy and quality of the products and
services they receive from centers, compared with other products and
services.

The Reported Research Base for Center Products and Services

Comprehensive Center staff are generally well-read and familiar with the
research on their particular areas of expertise, and they can readily
describe the research foundation for the services they deliver. The
extent to which the Comprehensive Centers' major initiatives are
specifically grounded in research or in a well-developed theoretical
framework varies considerably from activity to activity, however. A
handful of Comprehensive Center activities are, in fact, the products of
research and development efforts. In these cases, links to a research
base are specific, and a key element of the product or service delivered
to customers. The most prominent of center initiatives is Cognitively
Guided Instruction (CGI), which is based on the work of a group of
researchers at the University of Wisconsin on children's cognitive
development in mathematics. These researchers have developed a set of
teacher materials and institutes that each Comprehensive Center then
provides to its customers. The Region VI Comprehensive Center also
sponsors a school leadership institute based on the work of another
researcher at the University of Wisconsin. Other examples of research
and development efforts adopted by the center network include Alaska
Onward to Excellence, which is an adaptation of Onward to Excellence, a
model of school reform based on effective schools research that was
developed and piloted by the Northwest Regional Educational Laboratory.
In the Houston Independent School District, the STAR Center has provided
training to principals in the attributes of high-achieving, high-poverty
schoolwides, based on research conducted by the Charles A. Dana Center
at the University of Texas-Austin (a partner in the STAR Center).

For other Comprehensive Center activities, the theoretical framework
supporting the activity is often based on more than one body of
research, and the links to research are often less direct and less
specific. Training and assistance provided by the Reading Success
Network, for example, are based on research on both literature-based and
phonetic approaches to reading instruction, as well as research on
cognitive coaching and peer coaching. In describing the research
foundation for the Learning Facilitator's Academy in Riverside, the
Comprehensive Center staff member who developed the activity cited
specific pieces of literature in the effective schools research,
research on second language acquisition and cross-cultural learning, and
research on professional development.

For still other activities, center staff members report no specific
links to published research. Instead, center staff rely on a
well-developed "craft knowledge" of their areas of specialization, some
of which is based on their own or their colleagues' action research. In
many areas, such as schoolwide program planning, or conducting needs
assessments using student achievement data, this practitioner expertise
is far more important than any related published research. As one center
director expressed it, this strand of center activity is driven more by
client needs and circumstances than by a particular set of research
findings. As this director commented, the center is committed to
providing "help of a practical nature. It is grounded in theory and
research, and it is rich in content . . . \[but\] it is client driven."

One final note on the links between center services and the research
base: As part of their information dissemination activities,
Comprehensive Center staff often provide summaries of research on topics
of interest to schools, districts, and (most often) states. For example,
the deputy commissioner in one of our case study states recalled that
when his state faced potential challenges to its bilingual programs, the
center provided summaries of research showing their effectiveness:
"\[The center\] did the work researching it; they contacted national
researchers; and we were able to avoid a 'California crisis' \[as a
result of their efforts\]." Comprehensive Center staff prepared research
summaries on school uniforms and block scheduling for the Mountainview
School as it considered its options for the development of a Title I
school improvement plan. The Comprehensive Centers regularly provide
clients with bibliographies, information about researchers and product
developers, and supplementary evidence for their work through
newsletters, on their Web sites, and in project materials. Each center
also disseminates the available research evidence for the CSRD models
that are available for their customers; two centers distribute
literature reviews on CSRD models through Web sites or written products.

## Customer Ratings of Accuracy and Quality of Content

Customers give the Comprehensive Centers high marks for the accuracy and
the quality of the information they provide. For example, Exhibit IV.1
shows that half of all gatekeepers rated the accuracy of the content and
the extent to which it reflected sound research or the most current
thinking in the field "excellent," compared with similar products and
services available from other sources; an additional third rated it
"good." Exhibit IV.2 shows that more than 90 percent of participants in
center activities rated the centers "excellent" or "good" on these
dimensions as well.

**Exhibit IV.1**

Gatekeepers' Ratings of the Quality and Utility of Comprehensive Center

Products and Services

**(n=542)**

+------------------------------+-------+------+-------+-------+-------+
| **Compared with similar      |       |      |       |       |       |
| products and services        |       |      |       |       |       |
| available from other         |       |      |       |       |       |
| sources, how would you rate  |       |      |       |       |       |
| the quality of the products  |       |      |       |       |       |
| and services you received    |       |      |       |       |       |
| from the Comprehensive       |       |      |       |       |       |
| Center on each of the        |       |      |       |       |       |
| following dimensions?**      |       |      |       |       |       |
|                              |       |      |       |       |       |
| **Compared with similar      |       |      |       |       |       |
| products and services        |       |      |       |       |       |
| available from other         |       |      |       |       |       |
| sources, how would you rate  |       |      |       |       |       |
| the overall USEFULNESS of    |       |      |       |       |       |
| the products and services    |       |      |       |       |       |
| you received from the        |       |      |       |       |       |
| Comprehensive Center?**      |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
|                              | #     |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
|                              | # [   |      |       |       |       |
|                              | Perce |      |       |       |       |
|                              | nt of |      |       |       |       |
|                              |  Gate |      |       |       |       |
|                              | keepe |      |       |       |       |
|                              | rs Re |      |       |       |       |
|                              | porti |      |       |       |       |
|                              | ng]{. |      |       |       |       |
|                              | under |      |       |       |       |
|                              | line} |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| ##### [Dimensions of Qual    | ##    | **   | **    | **    | **Not |
| ity and Utility]{.underline} |  Exce | [Goo | [Fair | [Poor | able  |
|                              | llent | d]{. | ]{.un | ]{.un | [to   |
|                              |       | unde | derli | derli | judge |
|                              |       | rlin | ne}** | ne}** | ]{.un |
|                              |       | e}** |       |       | derli |
|                              |       |      |       |       | ne}** |
+------------------------------+-------+------+-------+-------+-------+
| Addressed our organization's | 52    | 29   | 11    | 3     | 6     |
| needs and interests          |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Accuracy of content          | 51    | 31   | 5     | 2     | 12    |
+------------------------------+-------+------+-------+-------+-------+
| Timely                       | 50    | 32   | 8     | 5     | 6     |
+------------------------------+-------+------+-------+-------+-------+
| Reflects sound research or   | 50    | 29   | 6     | 2     | 12    |
| the most current thinking in |       |      |       |       |       |
| the field                    |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Useful and accessible format | 49    | 30   | 11    | 4     | 6     |
+------------------------------+-------+------+-------+-------+-------+
| Useful for guiding           | 48    | 31   | 10    | 4     | 7     |
| improvement efforts          |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Responsive to local          | 48    | 29   | 12    | 4     | 7     |
| conditions                   |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Presentation and format      | 42    | 35   | 8     | 2     | 12    |
+------------------------------+-------+------+-------+-------+-------+
| Extent to which the Center   | 42    | 34   | 9     | 2     | 13    |
| provided knowledge and       |       |      |       |       |       |
| expertise not available      |       |      |       |       |       |
| within your organization     |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Ability to respond in depth  | 42    | 33   | 10    | 3     | 12    |
| to all of your questions and |       |      |       |       |       |
| interests                    |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+

Exhibit reads: Fifty-two percent of gatekeepers rate Center services
"excellent" for the extent to which they addressed their organization's
needs and interests.

Note: Rows may not sum to 100 because of rounding.

**Exhibit IV.2**

Participants' Ratings of the Quality and Utility of Center-Sponsored
Activities

**(n = 675)**

+------------------------------+-------+------+-------+-------+-------+
| **Compared with similar      |       |      |       |       |       |
| products and services        |       |      |       |       |       |
| available from other         |       |      |       |       |       |
| sources, how would you rate  |       |      |       |       |       |
| the quality of the Center's  |       |      |       |       |       |
| assistance related to \[the  |       |      |       |       |       |
| activity\] in each of the    |       |      |       |       |       |
| following areas?**           |       |      |       |       |       |
|                              |       |      |       |       |       |
| **Compared with similar      |       |      |       |       |       |
| products and services        |       |      |       |       |       |
| available from other         |       |      |       |       |       |
| sources, how would you rate  |       |      |       |       |       |
| the overall USEFULNESS of    |       |      |       |       |       |
| \[the activity\] on each of  |       |      |       |       |       |
| the following dimensions?**  |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
|                              | #     |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
|                              | # [P  |      |       |       |       |
|                              | ercen |      |       |       |       |
|                              | t of  |      |       |       |       |
|                              | Parti |      |       |       |       |
|                              | cipan |      |       |       |       |
|                              | ts Re |      |       |       |       |
|                              | porti |      |       |       |       |
|                              | ng]{. |      |       |       |       |
|                              | under |      |       |       |       |
|                              | line} |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| ##### [Dimensions of Qual    | ##    | **   | **    | **    | **Not |
| ity and Utility]{.underline} |  Exce | [Goo | [Fair | [Poor | able  |
|                              | llent | d]{. | ]{.un | ]{.un | [to   |
|                              |       | unde | derli | derli | judge |
|                              |       | rlin | ne}** | ne}** | ]{.un |
|                              |       | e}** |       |       | derli |
|                              |       |      |       |       | ne}** |
+------------------------------+-------+------+-------+-------+-------+
| Reflects sound research or   | 54    | 37   | 6     | 1     | 3     |
| the most current thinking in |       |      |       |       |       |
| the field                    |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Accuracy of content          | 51    | 41   | 5     | 1     | 2     |
+------------------------------+-------+------+-------+-------+-------+
| Presentation and format      | 44    | 41   | 12    | 2     | 1     |
+------------------------------+-------+------+-------+-------+-------+
| Accessibility to             | 44    | 41   | 12    | 2     | 1     |
| participants                 |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Timely                       | 43    | 39   | 12    | 3     | 3     |
+------------------------------+-------+------+-------+-------+-------+
| Useful for guiding           | 43    | 39   | 12    | 3     | 3     |
| improvement efforts          |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Opportunities for networking | 41    | 38   | 13    | 5     | 3     |
| among participants           |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Addressed my needs and       | 38    | 44   | 13    | 3     | 2     |
| interests                    |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Ability to respond in depth  | 36    | 46   | 14    | 2     | 2     |
| to all of your questions and |       |      |       |       |       |
| interests                    |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+
| Responsive to local          | 30    | 42   | 18    | 6     | 4     |
| conditions                   |       |      |       |       |       |
+------------------------------+-------+------+-------+-------+-------+

Exhibit reads: Fifty-four percent of participants in Center-sponsored
activities rated as "excellent" the extent to which the activity
reflected sound research or the most current thinking in the field.

Note: Rows may not sum to 100 because of rounding.

Customers in our case study sites gave similarly high ratings to the
quality and accuracy of the information they had received from the
Comprehensive Centers. Most gatekeepers in our case study sites reported
that the accuracy and quality of the information they had received from
the centers compared well with information they had received from other
sources. As one customer noted, "There are other

free sources \[of technical assistance\] that we do not use because they
do not do solid work. I would prefer to use the center more."

These assessments of the quality of the centers' assistance are
subjective, of course. Although both case study respondents and survey
respondents reported that the centers compared well with other sources
of assistance, we have little information about the basis on which they
made those comparisons. Some of the gatekeepers in our case study sites
are sophisticated consumers of information and assistance who have
worked in their areas of interest for many years and know the research
literature well themselves. Others have had rather limited access to
technical assistance resources and have had relatively little experience
with other providers. It is even more difficult to assess the ratings of
our survey respondents because we know little about their relevant
experience.

###### Are Centers Responsive to Customer Needs and Interests? {#are-centers-responsive-to-customer-needs-and-interests .unnumbered}

This evaluation of Comprehensive Center services examined both the
extent to which centers involved their customers in planning and
designing services, and the extent to which customers said that the
centers' services addressed their purposes.

**[Customers' Role in Planning Services]{.underline}**

Our case study and survey data, taken together, suggest that the
customer's role in planning services can vary considerably. In most of
our case study sites, where services were relatively intensive or
sustained, Comprehensive Center staff worked closely with state and
local gatekeepers to design services adapted to the particular needs and
purposes of their customers. In the most successful working
relationships, there was a symbiosis between the customer's role in
defining the services they desired and the center's. The center would
first elicit information about the customer's needs and suggest several
options. Together, center staff and customers would decide among the
options.

In the Kingston Unified School District, for example, the Comprehensive
Center staff met with principals and district staff for a full day to
plan the center's work in the district. They worked from a planning
matrix that identified the center's products and training activities
related to schoolwide program planning and agreed upon ways these could
be adapted to suit the purposes of the district. This pattern was
typical among many of our local case study sites. Among states, SEA and
center staff met regularly to identify needs and to discuss the state's
requests for service. Typically, SEA and center staff worked together to
plan activities that address the state's identified needs.

Among our local case study sites there were two examples of activities
that had been designed with relatively little input from school or
district contacts: Cognitively Guided Instruction (CGI), a professional
development initiative developed by researchers at the University of
Wisconsin, and the Reading Success Network (RSN), which was planned by
staff from the eight county offices of education in Southern California.
In both cases, schools and districts are invited to participate in the
activities as conceived by the center. Including such "off the shelf,"
research-based activities in their repertoire of services helps to
conserve center resources by limiting the time needed for new product
and service development.

Customers in case study sites generally praised the centers for their
responsiveness during planning. The following is a typical comment from
a satisfied customer:

> They try to find out what our needs are. If it's within their capacity
> to assist us, they always try to do so. They do have some packaged
> ideas---things that they feel will be helpful. . . . If we were to ask
> for specific things, I think they would try to address them as best
> they could.

Although most respondents in case study sites praised the centers for
designing services that were useful and relevant to their daily work,
some clients complained that center staff began work with their
organization with a pre-determined agenda, and others complained that
the centers were not sufficiently proactive in their outreach. These
criticisms were relatively rare among all respondents, however.

Across all gatekeepers in our survey sample, about two-thirds reported
that they take an active role in the design and planning of services.
This percentage is not especially high, given that gatekeepers are
generally responsible for working with center staff to plan services to
their organizations. Organizations whose gatekeepers report that they
played no role in designing or planning services probably have very
little input into the content or format of the services they receive. Of
the gatekeepers who did participate in planning or designing services,
about half reported that they help to select and frame the content, half
reported that they help to identify participants, and about 40 percent
reported that they help to plan logistics.

Gatekeepers whose organizations received a high volume of services were
almost twice as likely to report that they had been involved in
selecting and framing the content of the activities and in designing the
format of the activities than were gatekeepers whose organizations
received a low volume of services. In addition, state-level gatekeepers
were more likely to report that they were involved in designing
activities than were local gatekeepers. For example, Exhibit IV.3 shows
that 66 percent of state-level gatekeepers reported that they
participated in selecting and framing the content of activities,
compared with 51 percent of district-level gatekeepers and 33 percent of
school-level gatekeepers. These differences probably reflect the
variation in the types of relationships that centers develop with state
staff and with gatekeepers in districts and schools; as we have noted,
centers have much more frequent contact with SEA gatekeepers and tend to
customize services for states more often. Center initiatives like the
RSN and CGI, which are designed as a package of services by the centers,
tend to be targeted at districts and schools. As a group, then, we would
expect local sites to take a less active role in planning services.

As would be expected, end-user participants in center activities were
less likely to report that they contributed to activity design than were
gatekeepers. Among participants in center activities, 17 percent said
they contributed to selecting and framing the content, 16 percent
reported that they coordinated the activity with other assistance they
receive, and 40 percent said they provided feedback to the provider or
presenter. Among participants surveyed, more than half (53 percent) said
they did not contribute at all to designing the activity. As we noted in
an earlier chapter, most participants in center activities are based in
schools, and most of those are teachers. Teachers who participate in
center--sponsored professional development activities and other kinds of
activities, then, report that they have relatively little opportunity to
influence the content or format of those activities.

## Customer Ratings of Responsiveness 

End-user participants in Comprehensive Center activities reported that
the activities generally served their purposes. Thirty percent reported
that the activities "served my purposes completely," and an additional
57 percent reported that the activity "was a good start." Only 13
percent of participants reported that the activity did not serve their
purposes or that it did not provide sufficient information or guidance
to allow the participant to follow up.

Although they were generally satisfied with the extent to which
center-sponsored activities addressed their purposes, end-user
participants gave the Comprehensive Centers slightly lower ratings for
the responsiveness and the utility of their services than for other
dimensions of quality, such as accuracy of content. For example, Exhibit
IV.2 shows that while more than half of participants rated the centers
"excellent" for the extent to which their assistance reflected sound
research, only about a third rated them "excellent" for their ability to
respond in depth to all of their questions or for their responsiveness
to local needs.

Centers also received their highest negative ratings from participants
on items related to responsiveness. Almost a quarter of participants
rated center services "fair" or "poor" for the extent to which they were
responsive to local conditions, 16 percent rated them "fair" or "poor"
for the extent to which they addressed participant's needs and
interests, and 16 percent rated them "fair" or "poor" for their ability
to respond in depth to all the participant's questions.

![](media/image14.wmf){width="9.3in" height="7.9in"}

It is not surprising that both our case study and survey data yield
somewhat mixed messages about customers' satisfaction with the
responsiveness of Comprehensive Center services, compared with their
overall quality or their basis in research. Responsiveness is a much
more difficult standard to achieve than basic accuracy of content; it
requires that technical assistance providers understand specific local
conditions and customize their services for each organization and
individual served. In addition, dimensions of responsiveness that are
related to the ability to provide in-depth answers to all questions and
to address all needs and interests require a much deeper knowledge of
the subject matter at hand. Customers may rate a superficial
presentation of new ideas high for the accuracy of content, but they
would almost surely rate such a presentation low for the extent to which
it responded in depth to all of their questions and interests.

**Are the Scope and Intensity of Services Adequate?**

Survey data from participants show that Comprehensive Center services
are at least designed to supply individual customers with assistance
that is of adequate scope and intensity to produce intended effects. As
we reported in Chapter III, most participants in center activities
report receiving some form of follow-up assistance from the centers, in
the form of additional materials; additional workshops, training, and
meetings; and telephone follow-up. In addition, most participants report
receiving some form of assistance on the same topic from another source.

Among our local case study sites, however, we observed a great deal of
variation in the scope and intensity of center services, both in the way
that services were designed and in the participation levels of
individual participants. In schools that were part of intensive model
schools initiatives or partnership sites, including the Mountainview
School and Richmond Elementary, center services were very intensive, by
almost any measure. Center staff members were in the school regularly,
as often as once a month, and school staff members called on the center
for help frequently in between visits. At both these schools,
school-based customers saw their center contact as an adjunct member of
the school community. In other cases, local contact with center staff
members was much more limited.

In several sites where centers had provided services to schools through
an initiative organized at the district level, for example, we found
that school-level staff members were hesitant to follow through on
offers from the center staff to provide additional assistance. Center
staff members routinely provided information about reaching them for
follow-up purposes, but some school-based customers reported that they
"wouldn't feel comfortable" calling them because they had first made
contact with the center through a district-level gatekeeper and didn't
feel comfortable taking on that role; others reported they "just didn't
feel the need." Despite one center's repeated willingness to offer
follow-up services, some principals still didn't understand the
parameters for continued consultation. One principal explained, "What we
don't know is if there is a time line, and if it stops now." As a group,
they were generally skeptical about the possibility of asking for more
assistance on their own behalf.

In other cases, the services were not designed to be especially
intensive over the long term. In the case of the CGI, for example,
participants attended one five-day summer institute, followed by a
second institute a year later. Although participants praised the initial
institute for the quality of the training provided and believed that it
had provided them with a very solid introduction to the theory and
practice of CGI, many reported during the second summer institute that
lack of follow-up and support during the school year had been a problem.
As one participant noted, "I don't want to wait a whole year to find out
if I'm doing it right."

Finally, in some cases we observed that the participation of individuals
across a collection of center activities in a particular site can be
uneven. For example, the Reading Success Network (RSN) sponsors regular
network meetings and support activities for RSN coaches, but none of the
coaches we interviewed as part of our case study of the RSN had actually
participated in any of those meetings. In these cases, uneven
participation rates dilute the scope and intensity of services for
participants.

Comprehensive Center services are generally designed to provide adequate
support and follow-up to individual participants. The Comprehensive
Centers have organized their work in line with their commitment to
provide intensive, continuing assistance that has a reasonable chance of
producing change in the states, districts, and schools served. There are
clear limitations to what the centers are able to do, however. Even
among the centers' most intensively served customers, moreover, we find
that individual participation levels in center activities vary. Our
evidence suggests that even the best intentions and the best-designed
activities may not provide enough support to meet all needs.

Do Center Staff Members Have Adequate Expertise?

In our case study sites, customers praised individual center staff
members for their knowledge and skill in delivering technical
assistance. Customers praised center staff members, many of whom had
worked in the more categorical centers under ED's previous system, for
their broad expertise, skill as facilitators, and flexibility. At the
state level, customers appreciated center staff members' range of
talents---from planning a conference to assisting with a grant
application to sponsoring schoolwide institutes---as well as their
expertise in specific areas. Typical comments in praise of center staff
members included:

\[Our assigned center staff member\] is very knowledgeable. Everyone
here knew her. . . . She is competent and everyone trusts her. She never
contradicts us as a state. She really works well with people, enhances
everyone's opinions, and is an outstanding facilitator.

> We chose \[to work with the center\] because we value their expertise
> in terms of being able to do the work and in developing plans. They've
> had so much experience in school improvement. I felt that their
> contributions were very valuable.
>
> At the leadership level what you experience with the center is more of
> an intangible . . . just knowing there is such a strong body of
> knowledge and expertise that you can rely on. I have confidence in the
> center . . . there\'s no question that whatever support they\'re
> \[giving\] is going to be appropriate. . . It\'s peace of mind,
> knowing that I trust these people.

In rare cases, customers described occasions when they were not pleased
with the quality of center staff members. In one instance, an SEA staff
member complained to us that his contact at the center knew less about
analyzing data than he did. In another local site, turnover in center
staff affected the center's relationship with an individual school and
therefore its progress in implementing reform. In this school, teachers
were quite vocal about their dissatisfaction with the knowledge base and
presentation skills of this particular center staff member. Another
district worked for two years with the same two talented center staff
members. When both consultants became unavailable during the final
several months of their project in the district, some schools lost
interest in doing further work with the center. In all cases when we
heard about problems with individual center staff members, however, they
had been replaced by the center leadership as soon as possible.

In interviews Comprehensive Center directors pointed out the importance
of recruiting staff members who are both generalists and specialists. As
one director explained, "In recruiting staff, the dilemma is: Do you
hire specialists or generalists? In a sense the background needs to be
generic and, having said that, we need someone who knows a lot about all
these specific areas." One strategy for responding to this challenge is
to create cross-disciplinary teams that draw on the knowledge and skills
of several staff members. Sharing staff with host institutions and
partner organizations can enhance the knowledge and expertise available
to individual centers. Comprehensive Centers often construct staffing
plans in a puzzle-like fashion, drawing on specific areas of expertise
in partner organizations to create a unified whole. These arrangements
vary tremendously from host organization to host organization, however.

As Exhibit IV.4 shows, the centers rely heavily on part-time staff to
carry out their work. These part-time staff usually divide their time
between Comprehensive Center work and other projects\--often other
federally funded technical assistance contracts\--at the host
institution. In this way they can maximize the pool of expertise
available to them. The number of full-time equivalent staff employed by
a center correlates roughly with the size of a center's budget (and
probably would correlate more closely if we made adjustments for the
cost of living and average salaries in the various regions). Centers
with larger budgets were able to recruit more staff members, diversify
the expertise available, and therefore better meet the needs of the
customers in their regions.

Are Center Customers Satisfied with Comprehensive Center Services?

To ensure the quality of the Comprehensive Centers program, Title XIII
of ESEA requires that the secretary of education conduct surveys of
populations to be served under the act every two years to determine if
they are satisfied with their access to and the quality of Comprehensive
Center services \[P.L.103-382, Section 13101(c)(2)\]. This evaluation
was designed, in part, to satisfy this requirement.

High Overall Customer Satisfaction Ratings

Most Comprehensive Center customers gave the centers high ratings for
the accessibility, quality, and utility of their services. For example,
85 percent of gatekeepers reported that they were "very satisfied" or
"moderately satisfied" with the accessibility of center products and
services. Customers gave similar ratings for all types of center
services, including on-site consultations, professional development, and
help convening special events (satisfaction was slightly lower for
accessibility of center Web pages and electronic products, as we have
already noted). Eighty-six percent of gatekeepers reported that they
were "very satisfied" or "moderately satisfied" with the overall quality
of the assistance they received. Customer ratings of quality were
similar for all topics of assistance. Finally, as we have already seen
in Exhibits IV.1 and IV.2, both gatekeepers and participants also gave
the centers high ratings for the quality and utility of their services,
especially the extent to which they reflect sound research and accurate
content.

Dissatisfaction with the Utility of Comprehensive Center Services

Although most customers are generally satisfied with the services they
receive from the Comprehensive Centers, some are not. Fifteen 15 percent
of gatekeepers reported that they were "somewhat satisfied" or "not at
all satisfied" with the overall accessibility of center services, and 14
percent reported the same for the overall quality of services. Up to a
quarter of respondents rated the centers "fair" or "poor" on some
dimensions of quality and utility, with participants generally giving
center services lower ratings than gatekeepers. For example, Exhibits
IV.1 and IV.2 show that:

# Exhibit IV.4

# 

# Comprehensive Center Staffing Patterns: Number of Regular

# Professional Staff and Areas of Expertise

<table>
<colgroup>
<col style="width: 7%" />
<col style="width: 12%" />
<col style="width: 13%" />
<col style="width: 10%" />
<col style="width: 8%" />
<col style="width: 7%" />
<col style="width: 5%" />
<col style="width: 8%" />
<col style="width: 15%" />
<col style="width: 9%" />
</colgroup>
<tbody>
<tr class="odd">
<td></td>
<td colspan="2"><h1
id="total-number-of-regular-professional-staff">Total Number Of Regular
Professional Staff</h1></td>
<td colspan="7"><h1 id="section-19"></h1>
<h1
id="number-of-regular-professional-staff-with-expertise-in-the-following-areas">Number
Of Regular Professional Staff with Expertise in the Following
Areas</h1></td>
</tr>
<tr class="even">
<td><strong>Region</strong></td>
<td><p><strong>Main</strong></p>
<p><strong>Organization</strong></p></td>
<td><p><strong>Partner</strong></p>
<p><strong>Organization(s)</strong></p></td>
<td><strong>Curriculum and Instruction</strong></td>
<td><strong>School Reform</strong></td>
<td><h2 id="section-20"></h2>
<h2 id="section-21"></h2>
<h2 id="title-i">Title I</h2></td>
<td><h2 id="section-22"></h2>
<h2 id="section-23"></h2>
<h2 id="ell">ELL</h2></td>
<td><h2 id="section-24"></h2>
<h2 id="section-25"></h2>
<h2 id="migrant">Migrant</h2></td>
<td><h2 id="section-26"></h2>
<h2 id="section-27"></h2>
<h2 id="readingliteracy">Reading/Literacy</h2></td>
<td><h2 id="section-28"></h2>
<h2 id="section-29"></h2>
<h2 id="other">Other</h2></td>
</tr>
<tr class="odd">
<td>I</td>
<td>7</td>
<td>2</td>
<td>2</td>
<td>3</td>
<td>4</td>
<td>1</td>
<td>1</td>
<td>1</td>
<td>6</td>
</tr>
<tr class="even">
<td>II</td>
<td>14</td>
<td>15</td>
<td>7</td>
<td>7</td>
<td>3</td>
<td>5</td>
<td>2</td>
<td>3</td>
<td>10</td>
</tr>
<tr class="odd">
<td>III</td>
<td>13</td>
<td>16</td>
<td>17</td>
<td>13</td>
<td>16</td>
<td>17</td>
<td>17</td>
<td>16</td>
<td>6</td>
</tr>
<tr class="even">
<td>IV</td>
<td>15</td>
<td>9</td>
<td>2</td>
<td>3</td>
<td>1</td>
<td>2</td>
<td>2</td>
<td>5</td>
<td>12</td>
</tr>
<tr class="odd">
<td>V</td>
<td>15</td>
<td>2</td>
<td>7</td>
<td>8</td>
<td>5</td>
<td>3</td>
<td>2</td>
<td>2</td>
<td>2</td>
</tr>
<tr class="even">
<td>VI</td>
<td>12</td>
<td>5</td>
<td>10</td>
<td>8</td>
<td>11</td>
<td>7</td>
<td>5</td>
<td>5</td>
<td>17</td>
</tr>
<tr class="odd">
<td>VII</td>
<td>18</td>
<td>6</td>
<td>1</td>
<td>9</td>
<td>0</td>
<td>4</td>
<td>1</td>
<td>3</td>
<td>16</td>
</tr>
<tr class="even">
<td>VIII</td>
<td>7</td>
<td>17</td>
<td>23</td>
<td>21</td>
<td>21</td>
<td>7</td>
<td>4</td>
<td>16</td>
<td>2</td>
</tr>
<tr class="odd">
<td>IX</td>
<td>10</td>
<td>9</td>
<td>17</td>
<td>17</td>
<td>14</td>
<td>16</td>
<td>5</td>
<td>13</td>
<td>8</td>
</tr>
<tr class="even">
<td>X</td>
<td>10</td>
<td>2</td>
<td>9</td>
<td>9</td>
<td>6</td>
<td>4</td>
<td>1</td>
<td>5</td>
<td>9</td>
</tr>
<tr class="odd">
<td>XI</td>
<td>12</td>
<td>0</td>
<td>7</td>
<td>10</td>
<td>6</td>
<td>5</td>
<td>2</td>
<td>0</td>
<td>10</td>
</tr>
<tr class="even">
<td>XII</td>
<td>6</td>
<td>8</td>
<td>5</td>
<td>4</td>
<td>10</td>
<td>2</td>
<td>1</td>
<td>4</td>
<td>11</td>
</tr>
<tr class="odd">
<td>XIII</td>
<td>8</td>
<td>3</td>
<td>2</td>
<td>4</td>
<td>4</td>
<td>2</td>
<td>2</td>
<td>1</td>
<td>11</td>
</tr>
<tr class="even">
<td>XIV</td>
<td>9</td>
<td>6</td>
<td>8</td>
<td>13</td>
<td>7</td>
<td>6</td>
<td>3</td>
<td>7</td>
<td>15</td>
</tr>
<tr class="odd">
<td>XV</td>
<td>10</td>
<td>5</td>
<td>6</td>
<td>10</td>
<td>3</td>
<td>3</td>
<td>0</td>
<td>2</td>
<td>9</td>
</tr>
</tbody>
</table>

> Source: Comprehensive Center Progress Reports

-   Twenty-four percent of participants and 16 percent of gatekeepers
    rated center services \"fair\" or \"poor\" for their responsiveness
    to local conditions

-   Sixteen percent of participants and 13 percent of gatekeepers rated
    center services \"fair\" or \"poor\" for their ability to respond in
    depth to all their questions

-   Sixteen percent of participants and 14 percent of gatekeepers rated
    center services \"fair\" or \"poor\" for the extent to which they
    addressed their needs and interests

-   Eighteen percent of participants rated center services \"fair\" or
    \"poor\" for the opportunities they provided for networking among
    participants

Some respondents in our case study sites also expressed dissatisfaction
with center services, compared with the services they had received in
the past from the categorically-based technical assistance centers that
operated before 1994. Most of the customers we interviewed who had had
some experience with the antecedent technical assistance centers
maintained that the current Comprehensive Centers are unable to provide
anything close to the level of service they had received in the past.
Some maintained that the quality of assistance they received from the
Comprehensive Centers was comparable to what they had received in the
past, although they did not benefit from nearly the same quantity of
service. Others maintained that both the quality and quantity suffered
in comparison with what the antecedent centers had provided to them.
Some typical comments included:

####### Through the current \[comprehensive\] method, we are not getting the same level or quality of services. There isn't the same reaction time or time line for service delivery. There is nowhere near what we got when you had specific technical assistance centers. . . . With the current system, I have no idea who's in charge and where they operate from.

> They're all knowledgeable but not for the particular things we need.
> There's a big learning curve. I just don't see the centers functioning
> the same way the \[old\] technical assistance centers did.

Respondents also pointed to shortcomings among center staff related to
their areas of expertise. Because center staff are often called upon to
be both generalists and specialists, many customers thought that the
centers lacked the capacity to address special population needs as well
as the antecedent centers did. In some states staff members said that
they taught Comprehensive Center staff the nuts and bolts of specific
ESEA programs provisions, rather than center staff teaching SEA staff.
As one SEA staff member put it:

It's been a constant struggle. . . . Those who should be receiving help
from the center are actually giving the technical assistance. . . . They
just haven't hired the expertise. . . . We want our old centers back
because we want that expertise. . . .

These comments come from a small number of respondents, almost all of
whom have special responsibility for the administration of a particular
categorical program, and almost all of whom were customers of the
antecedent centers. Their dissatisfaction is explained, in part, by the
expectations set by the work of the antecedent centers. Because the
Comprehensive Centers operate under a different mission and with a much
lower level of resources, it is not surprising that some customers would
find they suffer by comparison with the older system of categorical
centers.

Satisfaction with center services varies significantly by respondent, by
the characteristics of the service, and by Comprehensive Center. In the
remainder of this chapter we explore those variations.

**[Variation by Respondent]{.underline}**

As we have noted, end-user participants gave the centers somewhat lower
ratings for the utility of their assistance than did gatekeepers (see
Exhibits IV.1 and IV.2). For example, 30 percent of participants rated
center-sponsored activities "excellent" on the extent to which they were
responsive to local conditions, compared with 48 percent of gatekeepers.
Forty-three percent of participants reported that the assistance
addressed the needs of their organization, compared with 52 percent of
gatekeepers. It is not surprising that ratings of utility might be
somewhat lower for participants than for gatekeepers. Center services
are designed to benefit participants, and they are the ones who are
expected to use center assistance in their daily work. As a result,
participants would be inclined to set a higher standard for the utility
of a service. In addition, gatekeepers are much more likely to be
involved in the design of activities than are participants. As a result,
they are probably disposed to believe that the activity is useful.

Among state, district, and school customers, there were some significant
differences in customer satisfaction with the quality of services.
District-level gatekeepers were most likely to report that they were
"very satisfied" with the overall quality of center products and
services. For example, 70 percent of district-level gatekeepers reported
that they were "very satisfied" with the overall quality of services,
compared with 63 percent of school gatekeepers and 56 percent of
state-level gatekeepers. In an earlier chapter we observed that
district-level respondents were also most likely to report that they
were "very satisfied" with their access to Comprehensive Center
services. As with our finding regarding access, we speculate that this
pattern (higher satisfaction for districts than for either states or
schools) reflects a better fit between district-level expectations and
level of services provided. States have higher expectations and are more
difficult to satisfy; schools receive fewer services and are somewhat
less satisfied for that reason.

State-level gatekeepers also gave the Comprehensive Centers somewhat
lower ratings for their utility of assistance in guiding improvements.
Twenty-five percent of state gatekeepers rated the centers "fair" or
"poor," compared with 10 percent of district-level gatekeepers and 15
percent of school-level gatekeepers.

Among participants, there were a few significant differences in ratings
by state, district, and school staff, with district-level participants
giving the centers the highest ratings for the quality and utility of
their services. For example, Exhibit IV.5 shows that district
participants were more likely than state and school participants to rate
the centers "excellent" for the presentation and format of their
activities, for opportunities for networking, and addressing their needs
and interests.

Variation by Characteristics of the Service

***By topic of assistance.*** Comprehensive Center customers report that
they are "very satisfied" with the quality of center services on all of
the topics we addressed in survey items, with no significant variations
in customer satisfaction across the topics of activity. However, several
other characteristics of center services and activities are associated
with customer ratings of quality and usefulness.

***By volume of services.*** Gatekeepers whose organizations receive a
higher volume of services are more likely to give the centers high
ratings on the quality and utility of their services. To measure the
volume of services received by organizations, we asked gatekeepers to
report on the number of times their organization had received the
following services: (1) written materials; (2) on-site consultations;
(3) consultations by telephone or e-mail; (4) professional development
activities;

\(5\) collaborating or networking with other programs or districts; (6)
help convening special events;

\(7\) center homepage or other electronic products. We combined
responses on each of these items to create a composite score across all
delivery methods. We then sorted respondents' scores, ranging them from
highest to lowest in terms of the number of times their organization had
received all of the services contained in the survey item. In the
following analyses, we compared gatekeepers with composite scores in the
top quartile (whom we describe as "high-volume" customers) with
gatekeepers whose composite scores were in the bottom quartile (whom we
describe as "low-volume" customers).

Seventy-four percent of high-volume customers reported that they were
"very satisfied" with the overall quality of the services and products
they received, compared with 48 percent of low-volume customers. In
addition, 36 percent of low-volume customers were "somewhat satisfied"
or "not at all satisfied" with the quality of assistance; compared with
7 percent of those with more frequent contact.

![](media/image15.wmf){width="9.3in" height="7.715277777777778in"}

When comparing the assistance provided by the centers with assistance
available from other sources, gatekeepers whose organizations received a
higher volume of services also tended to give the centers higher
ratings. For example, Exhibit IV.6 shows that 63 percent of high-volume
customers rated the centers "excellent" in guiding improvement efforts,
compared with 39 percent of respondents with less frequent contact.
Sixty-one percent of high-volume customers rated the responsiveness of
the assistance to local conditions "excellent," compared with 40 percent
of those with less frequent contact. Sixty-three percent of high-volume
customers rated center services "excellent" for addressing their needs
and interests, compared with 46 percent of low-volume customers.

Respondents who receive services frequently are more likely to (1)
receive greater depth of information; (2) have more opportunities to
work with centers to tailor services to meet their needs; and (3) have
time to develop positive relationships with center staff. Such customers
are more likely to have positive experiences and to perceive the quality
and utility of service as better than those who receive services less
frequently.

***By level of follow-up.*** As we have noted in earlier sections of
this chapter, high-quality technical assistance is intensive and is
delivered over an extended period of time. Persons receiving more
frequent services have more opportunities to receive in-depth
information and are more likely to receive assistance that more directly
meets their needs. They are also more likely to rate the quality of
assistance higher. Data from our participant surveys further illustrate
this relationship.

In general, customers who received follow-up assistance were more likely
to rate the quality and utility of center activities "excellent" and
less likely to rate them "fair" or "poor." This finding holds true for
most types of follow-up. As the frequency of additional assistance
increases, from "never" to "once" and from "once" to "more than once,"
positive ratings increase and negative ratings decrease. For example,
Exhibit IV.7 shows that 61 percent of participants who received
telephone follow-up more than once rated the centers' activities
"excellent" for the extent to which they proved useful for guiding
improvement efforts, compared with 32 percent of participants who
received no telephone follow-up. Similarly, 38 percent of participants
who received no telephone follow-up rated the centers "fair" or "poor"
for their responsiveness to local conditions, while only 17 percent of
participants who received telephone follow-up more than once rated them
"fair" or "poor." These findings hold true for other types of follow-up,
including additional materials and additional workshops and training,
although Exhibits IV.7 reports on telephone follow-up only.

![](media/image16.wmf){width="9.3in" height="7.388194444444444in"}

![](media/image17.wmf){width="9.3in" height="7.966666666666667in"}

Response patterns do vary slightly for follow-up that includes visits to
the workplace. In this category of follow-up, the percentage of
participants giving the centers low ratings does not consistently
decrease as the frequency of additional assistance increases. For
example, Exhibit IV.8 shows that 35 percent of respondents who received
no workplace visits gave the activities low ratings for the extent to
which they were responsive to local conditions, compared with 11 percent
of participants who received one visit to their workplace. The
percentage of participants assigning center activities a negative rating
*increases* for participants who received more than one workplace visit,
to 19 percent. This pattern is consistent across many, though not all,
dimensions of quality and utility addressed by our surveys.

Participants receiving frequent workplace visits may begin to perceive
the centers as becoming overly intrusive in their professional lives and
may consequently develop a more negative view of quality and usefulness
of the assistance. This finding further demonstrates the importance of
an continuing, collaborative relationship between providers and
customers. Providers and customers should continually work together to
assess whether or not the assistance continues to have desirable
outcomes and continues to meet the needs of those served.

Variation by Center

Customer satisfaction with the accessibility, quality, and utility of
center services varies by individual Comprehensive Center. On most of
the customer satisfaction items on our survey of gatekeepers, these
variations in satisfaction levels were quite pronounced. For example,
the percent of gatekeepers reporting that they are very satisfied with
the overall accessibility of Comprehensive Center products and services
ranges from a high of 90 percent at one center to a low of 31 percent at
another. The percent of gatekeepers reporting that they are very
satisfied with the overall quality of center services ranges from a high
of 89 percent at one center to a low of 35 percent at another. The
percent of gatekeepers who rated the Comprehensive Centers "excellent"
for the extent to which their assistance addressed their organizations'
needs and interests ranged from a high of 72 to a low of 31.

We tested many possible hypotheses to explain this variation in customer
satisfaction by center. We were unable to detect any statistically
significant relationship between Customer satisfaction and the following
center characteristics:

-   Type of host institution (that is, whether the center is based in a
    regional educational laboratory, a university, or some other type of
    organization)

-   Number of customers served (from customer lists submitted by the
    centers)

![](media/image18.wmf){width="9.3in" height="7.923611111111111in"}

-   Primary strategy for delivering assistance to the region (for
    example, whether centers with strong state-focused strategies for
    serving their regions received higher satisfaction ratings from
    their state-based customers than centers without strong
    state-focused strategies)

-   Coherence of mission (we identified a small group of centers that,
    in our judgment, had adopted a particularly coherent mission and set
    of strategies for serving their region, and tested whether this
    group of centers received higher satisfaction ratings)

In other words, whether a center is based in a certain kind of host
institution, whether it provides services in a certain kind of way,
whether it has adopted a particularly coherent set of strategies for
serving its region, or the number of customers it serves appears to have
no relationship to customer satisfaction with the accessibility,
quality, and utility of center services.

Only one variable is consistently and significantly related to various
measures of customer satisfaction, and that is the size of the center's
budget. The size of center budgets varies widely among the 15
Comprehensive Centers, depending on the numbers of Title I-eligible,
LEP, migrant, and Indian students in each region. The smallest center
has an annual budget of \$850,000, while the largest center has an
annual budget of more than three times that size, at \$2.6 million. We
examined the relationship between size of center budget and various
measures of satisfaction and found that there is indeed a strong
relationship between the two, with large centers receiving, on average,
higher ratings of customer satisfaction than smaller centers.

Although it is not true---and the reader should not infer---that all
large centers receive high ratings of customer satisfaction and all
small centers receive low ratings, the relationship between budget and
customer satisfaction across the entire network of 15 centers is strong
and statistically significant. Exhibit IV.9 shows, for example, that
every \$100,000 increase in center budget corresponds with a 2
percentage point increase in the number of gatekeepers reporting that
they are "very satisfied" with the overall quality and accessibility of
center services. Variation in the size of center budgets accounts for 84
percent of the variability in customers' ratings of the accessibility of
center services, and 77 percent of the variability in customers' ratings
of the quality of center services. Relationships of similar magnitude
and strength also exist between the size of center budgets and the
percentage of center gatekeepers rating the quality and utility of
center services "excellent."

The explanation for this finding appears to be that larger centers, *on
average*, provide a higher volume of higher-intensity, more expensive
services to their customers. Centers with larger budgets do not
necessarily serve larger *numbers* of clients; size of a center\'s
budget has no relationship with the

**Exhibit IV.9**

**Bivariate Regression Analysis of the Relationship between Gatekeeper
Ratings of Center Services and Products**

  ------------------------------- ----------- ------- ------------- ------ --------------
                                  Y intercept Slope   T value       R^2^   Number of
                                                                           Observations

  Percent of gatekeepers          32.86       1.96    (8.2)\*\*\*   .84    15
  reporting that they are "very                                            
  satisfied" with access to                                                
  Center services by the size of                                           
  budget (in hundred thousands)                                            

  Percent of gatekeepers          31.61       1.93    (6.6)\*\*\*   .77    15
  reporting that they are "very                                            
  satisfied" with the quality of                                           
  Center services by the size of                                           
  budget (in hundred thousands)                                            

  Percent of gatekeepers          1.43        2.46    (6.1)\*\*\*   .74    15
  reporting that the quality of                                            
  Center services is "excellent'                                           
  when compared with assistance                                            
  provided by other sources (in                                            
  hundred thousands)                                                       

  Percent of gatekeepers          15.09       1.97    (6.1)\*\*\*   .74    15
  reporting that the utility of                                            
  Center services is "excellent"                                           
  when compared with assistance                                            
  provided by other sources (in                                            
  hundred thousands)                                                       
  ------------------------------- ----------- ------- ------------- ------ --------------

\*\*\* p\<.001

Exhibit reads: For every \$100,000 difference in the budget, there is a
1.96 percent difference in the proportion of customers who are "very
satisfied" with their access to Center services and products. Center
budget explains 84 percent of the variance in high levels of
satisfaction.

number of customers it served in 1998, as recorded on the centers\' own
customer lists. Instead, centers with larger budgets tend to have more
resources available for each of the customers they do serve. In fact,
customers of larger centers do report that they receive a higher volume
of services, although the relationship is only significant for those
services that are particularly resource intensive and expensive to
provide---professional development, on-site consultations, and help
convening special events (see Exhibit IV.10). Rather than using
additional resources to reach a larger number of customers, then, it
appears that larger centers use their resources to serve roughly the
same number of customers with a higher volume of services.

As we discussed earlier in this chapter, volume of services and
frequency of follow-up are associated with customer assessments of
quality. Centers with larger budgets provide a larger volume of
high-intensity, relatively expensive services, and customers appear to
be more satisfied with services as a result.

**Exhibit IV.10**

**Bivariate Regression Analysis of the Relationship between the
Percentage of Customers Receiving High-end Services and Size of Center
Budget**

  ------------------------------- ----------- ------- --------- ------ --------------
                                  Y intercept Slope   T value   R^2^   Number of
                                                                       Observations

  Percentage of gatekeepers       .61         9.76    (2.9)\*   .39    15
  receiving professional                                               
  development, on-site                                                 
  consultations, and convening                                         
  assistance five or more times                                        
  (in hundred thousands)                                               
  ------------------------------- ----------- ------- --------- ------ --------------

\* p\<.05

Exhibit reads: For every \$100,000 difference in the budget, there is a
9.76 percent difference in the proportion of customers receiving
professional development, on-site consultations, and convening
assistance on five or more occasions. Center budget explains 39 percent
of the variance, by Center, in percentage of customers receiving such
assistance.

Summary

#  {#section-30}

The Comprehensive Centers have made a credible effort to respond to
their charge in Title XIII of ESEA to provide training and technical
assistance "using the highest-quality . . . strategies possible"
\[P.L.103-382, Section 13102 (a)(3)\]. The majority of Comprehensive
Center activities are, in fact, comprehensive; most of the information
and other content conveyed by the centers are reported to be accurate
and grounded in research; most center activities are responsive to
customer needs; most are sustained and intensive; and most customers are
satisfied with the expertise of individual center staff. Nonetheless,
not all activities meet all our criteria for quality equally well. Some
activities have been planned without the active involvement of
customers, some receive low ratings for responsiveness, and many are not
intensive. However, most Comprehensive Center activities satisfy one or
more of these criteria for high-quality assistance. In addition,
Comprehensive Center customers give the centers high ratings for the
quality and utility of their services. As one possible indicator of
quality, customer satisfaction data indicate that the centers are
providing a valuable set of services to their regions.

#   {#section-31}

#   {#section-32 .unnumbered}

# V. EFFECTS

#  {#section-33}

This evaluation examined the effects of Comprehensive Center services in
terms of the expectations set forth in their authorizing legislation. We
asked customers about the effects of center services in both surveys and
in visits to case study sites. We begin the chapter with customers'
reports of the general effects of the centers' services on their own
work and on the work of their organizations, and examine some of the
factors associated with those effects. Next, we examine the extent to
which center services have strengthened the implementation of ESEA
provisions, enhanced states' and districts' capacity to provide
technical assistance in support of ESEA, and improved teaching and
learning in participating schools and classrooms. For data on these
specific kinds of effects, we turn primarily to our case study sites.

# Effects on the Work of Individuals

Comprehensive Center customers participated in a variety of technical
assistance activities designed to enhance their knowledge and skills and
build their capacity, both to deliver technical assistance and fulfill
their job responsibilities. According to our survey and case study data,
the centers were more likely to influence state, district, and local
staff members' individual work than the work of their organizations.

Participants and gatekeepers most frequently reported that they had
gained new information as a result of center assistance (see Exhibit
V.1). More than three-quarters of both groups of respondents reported
this outcome. Respondents also frequently indicated that center
assistance directly affected the way they did their jobs. Sixty-one
percent of gatekeepers and 72 percent of participants reported that they
incorporated something they had learned from the center into their work.

Survey data also show that center services and products frequently
prompt participants and gatekeepers to share, formally and informally,
what they have learned with others. Forty percent of gatekeepers
reported that they have provided technical assistance in support of
ESEA. Additionally, three-quarters of participants reported that they
have informally shared new ideas with others, and 50 percent of
participants reported formally providing training or technical
assistance.

Exhibit V.1

**Percent of Participants and Gatekeepers Reporting on Effects of Center
Assistance**

**on Their Work**

+---------------------------------+-----------------+-----------------+
| ### Overall, what effect, if    |                 |                 |
| any, has Comprehensive Center a |                 |                 |
| ssistance had on your own work? |                 |                 |
+---------------------------------+-----------------+-----------------+
| Effect                          | Percent of      | ###             |
|                                 | Participants    |   {#section-34} |
|                                 |                 |                 |
|                                 | **[(n=669)      | ### Percent     |
|                                 | ]{.underline}** |  of Gatekeepers |
|                                 |                 |                 |
|                                 |                 | **[(n=573)      |
|                                 |                 | ]{.underline}** |
+---------------------------------+-----------------+-----------------+
| I have gained some new          | 80              | 77              |
| information                     |                 |                 |
+---------------------------------+-----------------+-----------------+
| I have informally shared ideas  | 74              | \*\*            |
| with colleagues                 |                 |                 |
+---------------------------------+-----------------+-----------------+
| I have incorporated into my job | 72              | 61              |
| something I learned from the    |                 |                 |
| Center                          |                 |                 |
+---------------------------------+-----------------+-----------------+
| It confirmed what I was already | 60              | 49              |
| doing                           |                 |                 |
+---------------------------------+-----------------+-----------------+
| I have communicated the ideas   | 50              | \*\*            |
| to colleagues or others through |                 |                 |
| training, technical assistance, |                 |                 |
| or similar activities           |                 |                 |
+---------------------------------+-----------------+-----------------+
| I have provided technical       | \*\*            | 40              |
| assistance to others in support |                 |                 |
| of federal programs             |                 |                 |
+---------------------------------+-----------------+-----------------+
| It has had little or no effect  | 9               | 9               |
| on how I do my job              |                 |                 |
+---------------------------------+-----------------+-----------------+
| Other                           | 3               | 4               |
+---------------------------------+-----------------+-----------------+

\*\* Item not included in the survey instrument for this group of
respondents.

Exhibit reads: Eighty percent of participants reported that Center
assistance allowed them to gain new information and additional
perspectives.

Variation by Type of Respondent

There is variation in the proportion of state, district, and school
gatekeepers and participants reporting individual effects. School-level
participants were more likely than state and district participants to
report gaining new information and incorporating something learned into
their job. Exhibit V.2 shows that 86 percent of school participants, 66
percent of district participants, and 55 percent of state participants
reported gaining new information about a program or instructional
practice. Seventy-six percent of school participants reported that they
had incorporated something into their job, compared with 56 percent of
state participants and 67 percent of district participants. There was,
however, no such variation among gatekeepers along these dimensions.
This is not surprising if we consider that gatekeepers typically work
with centers to plan services for their organizations. Gatekeepers may
or may not be end-user participants in the services. As a result, they
may be less disposed to carry away lessons from the services that they
can apply in their own work---the services may not be tailored to their
particular purposes, or they may not be thinking of themselves as
learners in their work with the center.

Among gatekeepers, district and state respondents were more likely than
school respondents to report that they had provided technical assistance
in support of federal programs. Overall, 57 percent of state gatekeepers
and 46 percent of district gatekeepers reported this effect, compared
with 23 percent of school gatekeepers. This difference is also likely
attributable to the professional responsibilities of each respondent.
District and state personnel have more direct responsibility for
administering federal programs and are also responsible for providing
assistance to schools. (Eighty-one percent of state gatekeepers and 74
percent of district gatekeepers identified providing technical
assistance as a major professional responsibility, compared with 46
percent of school gatekeepers.)

Overall, 9 percent of gatekeepers reported that the assistance had
little effect on how they did their jobs. State gatekeepers (16 percent)
were more likely than both school (6 percent) and district gatekeepers
(7 percent) to report that center services and products had little
effect on how they did their jobs.

Variation by Level of Follow-Up

End-user participants in center activities were more likely to report
that the activity had an effect on the way they do their job if they
received follow-up services from the center. This pattern holds for all
types of follow-up, including follow-up by telephone, additional
training, written materials, and visits to the workplace. For example,
Exhibit V.3 shows that 64 percent of participants who received
additional training more than once, beyond what was originally
scheduled, reported that they communicated the ideas learned to others
via technical assistance activities, compared with 43 percent of those
who participated in additional training only once, and 38 percent of
those who never participated in additional training. Similarly, the
percentage of participants reporting that they incorporated something
new into their job increases from 63 percent to 83 percent as the
frequency of telephone follow-up increases.

![](media/image19.wmf){width="9.3in" height="8.170833333333333in"}

![](media/image20.wmf){width="9.8in" height="8.1in"}

Variation by Participant Motivation

***By whether participants were required to participate in the
activity.*** Participants in center-sponsored activities who were
[required]{.underline} to participate are less likely to report that the
activity had an effect on their work than participants who volunteered.
For example, 80 percent of volunteers reported that they had informally
shared ideas with colleagues as a result of the activity, compared with
62 percent of those who were required to participate (Exhibit V.4). This
pattern repeats itself across all the individual effects identified in
our survey item.

***By participants' purposes for participating.*** Participants' goals
and purposes are also important factors that affect the impact of a
service or activity. Data from the participant survey indicate that
persons with multiple reasons for participating in an activity are more
likely to report that the activity had an effect on the way they do
their job. Persons who have more interests in and goals for
participation are more likely to be invested in the activity and are
thus more likely to reap benefits from the activity. Exhibit V.5 shows
that 53 percent of participants who identified just one of the purposes
listed in our survey item as the reason for their participation reported
that they had incorporated something they learned from the activity into
their job. In contrast, 81 percent of participants identifying all five
purposes listed in our survey item reported making changes in their
jobs. This pattern holds true for all types of individual effects.

In addition to the general level of enthusiasm or engagement displayed
by end-user participants, survey data show that specific purposes also
have an impact on individual outcomes. For example, participants who
reported that their purposes included "obtaining ideas or contacts that
would strengthen my work" were more likely to report that they included
something they had learned from the center into their job, compared with
participants who did not identify that purpose as a reason for
participating in the activity. This finding holds true for all kinds of
individual effects identified in our survey item.

# Effects on the Work of Organizations

Exhibit V.6 shows that for both participants and gatekeepers, the most
commonly reported organizational effect was that center assistance
helped generate awareness of new information within the organization.
Both participants and gatekeepers were more likely to report that center
assistance affected the existing work of their organization than it
prompted their organization to undertake a new effort. Approximately
half of both groups of customers reported that center assistance
supported an established practice of their organization. Thirty-five
percent of gatekeepers and 45 percent of participants indicated that
center services prompted their organization to initiate a new practice.

![](media/image21.wmf){width="9.4in" height="8.0in"}

![](media/image22.wmf){width="9.3in" height="7.9in"}

**Exhibit V.6**

**Percent of Gatekeepers and Participants Reporting on Effects of Center
Assistance**

**on the Way Their Organization Does Its Work**

+----------------------------------+-----------------+-----------------+
| Effect                           | Percent of      | Percent of      |
|                                  | Participants    | Gatekeepers     |
|                                  |                 |                 |
|                                  | **[(n=668)      | **[(n=552)      |
|                                  | ]{.underline}** | ]{.underline}** |
+----------------------------------+-----------------+-----------------+
| Helped generate awareness of new | 77              | 80              |
| information                      |                 |                 |
+----------------------------------+-----------------+-----------------+
| Increased teachers' knowledge    | 56              | 57              |
| and skills                       |                 |                 |
+----------------------------------+-----------------+-----------------+
| Supported an ongoing program,    | 52              | 55              |
| policy, or practice of the       |                 |                 |
| organization                     |                 |                 |
+----------------------------------+-----------------+-----------------+
| Helped the organization          | 35              | 45              |
| establish or                     |                 |                 |
|                                  |                 |                 |
| initiate a new program, policy,  |                 |                 |
| or practice                      |                 |                 |
+----------------------------------+-----------------+-----------------+
| Helped the organization take the | 46              | 45              |
| next step in a reform effort     |                 |                 |
+----------------------------------+-----------------+-----------------+
| Improved teaching and learning   | 38              | 42              |
| for a specific population of     |                 |                 |
| students                         |                 |                 |
+----------------------------------+-----------------+-----------------+
| Put the organization in touch    | 29              | 38              |
| with other organizations engaged |                 |                 |
| in similar tasks                 |                 |                 |
+----------------------------------+-----------------+-----------------+
| Improved teaching and learning   | \*\*            | 36              |
| for all students                 |                 |                 |
+----------------------------------+-----------------+-----------------+
| Enhanced the organization's      | \*\*            | 36              |
| capacity to provide technical    |                 |                 |
| assistance in support of ESEA    |                 |                 |
| programs                         |                 |                 |
+----------------------------------+-----------------+-----------------+
| Changed teachers' classroom      | 35              | 35              |
| practices                        |                 |                 |
+----------------------------------+-----------------+-----------------+
| Increased student achievement    | 28              | 24              |
+----------------------------------+-----------------+-----------------+
| It has had little or no effect   | 11              | \*\*            |
| on my organization's work        |                 |                 |
+----------------------------------+-----------------+-----------------+
| Other                            | 2               | 4               |
+----------------------------------+-----------------+-----------------+

\*\* Item not included in survey instrument for this group of
respondents.

Exhibit reads: Seventy-seven percent of participants reported that
Center assistance generated awareness of new information within the
organization.

Slightly less than half of participants (46 percent) and gatekeepers (45
percent) reported that center services helped their organization move
forward in reform efforts. This finding makes sense, given that it is
generally easier to make modifications to current activities than it is
to enact new policies and procedures.

**[Variation by Organization Served]{.underline}**

There were significant differences in the numbers of state, district,
and school gatekeepers reporting effects for some types of
organizational change. For example, Exhibit V.7 shows that 85 percent of
school-level gatekeepers reported that the center's services had helped
generate awareness of new information, compared with 77 percent of
districts and 71 percent of states. Predictably, schools were more
likely than districts or states to report that Comprehensive Center
services had changed teachers' classroom practices, improved teaching
and learning for all students, and increased student achievement. This
finding is explained by the fact that services to schools are more
likely to be designed to directly affect classroom practice.

Among participants in center activities, school-based respondents were
more likely to report changes in teachers' knowledge and skills, and
improvements in teaching and learning. District-level end-users of
assistance were more likely than school or state participants to report
that the activity helped their district initiate a new practice or take
the next step in a reform effort (Exhibit V.8).

Variation by Characteristics of the Assistance

***By volume of services.*** Gatekeepers whose organizations received a
higher volume of services were more likely than those who received a
lower volume of services to report that the service affected the work of
their organizations. This finding holds true for all types of services
and kinds of organizational effects. To conduct the analyses we
described below, we added up the number of times gatekeepers reported
receiving all types of center services---written materials, professional
development, on-site consultations, etc.---to produce a composite score
reflecting the overall levels of service. We then divided gatekeepers
into quartiles based on these composite scores and compared

![](media/image23.wmf){width="9.3in" height="7.8875in"}

![](media/image24.wmf){width="9.3in" height="7.6in"}

gatekeepers in the top quartile (who received what we call "high-volume"
services) with gatekeepers in the bottom quartile (who received what we
call "low-volume" services).

Exhibit V.9 shows that 94 percent of gatekeepers who received a high
volume of services reported that the assistance generated awareness of
new information within the organization, compared with 56 percent of
those who received a lower volume of services. Fifty-four percent of
those receiving a high volume of services noted a change in classroom
practices, compared with 16 percent of those receiving a low volume of
services. There are large differences for other kinds of organizational
effects as well. Exhibit V.10 shows that end-user participants in center
activities who received a high volume of follow-up services were more
likely to report all types of organizational effects.

As discussed in the previous chapter, customers receiving services over
an extended period of time have more opportunities to receive in-depth
information and work with the centers to tailor services to meet their
needs. Consequently, customers receiving high levels of service are more
likely to develop the capacity to implement new ideas and practices.

***Variation by number of topics addressed***. The number of topics of
assistance also affects gatekeeper reports of the impact of center
services and products. Respondents receiving assistance on a greater
number of topics were more likely to report that the assistance affected
their work and the work of their organization (Exhibit V.11). Sixty-one
percent of gatekeepers receiving assistance on six or more topics
reported that center services helped their organization take the next
step in a reform effort, compared with 33 percent of those receiving
assistance for fewer topics. Significant relationships exist for all
types of organizational effects.

Variation by Participation Patterns

***By membership on a team.*** Participants in center-sponsored
activities who came as members of a team representing their organization
were more likely to report that the activity had an effect on their
organization's work. More than a third of respondents who participated
as a member of a team representing their organization reported a change
in teacher classroom practices and improved teaching and learning for a
specific population of students (Exhibit V.12). In contrast, only a
quarter of those who participated in activities as individuals reported
these outcomes. Team members were also more likely to report that the
center assistance had helped their organization take the next steps in a
reform effort.

![](media/image25.wmf){width="9.3in" height="7.456944444444445in"}

![](media/image26.wmf){width="9.3in" height="7.336111111111111in"}

![](media/image27.wmf){width="9.3in" height="8.188194444444445in"}

![](media/image28.wmf){width="9.3in" height="6.785416666666666in"}

These results suggest that organizations that send teams of participants
to an activity may be demonstrating greater levels of support for the
activity, which translate into greater levels of support for
implementing new ideas and skills. Participants themselves are more
likely to perceive the activity as important to the overall goals of the
organization. Both individual and organizational support are critical
elements in the change process. Additionally, if a team of
representatives from an organization receives training, there will
simply be a greater number of trained individuals within the
organization. Organizations with a "critical mass" of people with new
knowledge and skills are more likely to produce change.

There were almost no significant differences between volunteers and
those who were required to participate in center activities with regard
to organizational effects.

***By involvement in planning services.*** Both gatekeepers and
participants who were involved in planning activities were more likely
to report that the assistance had an effect on their organizations. Such
involvement affords them the opportunity to actively work with the
centers to tailor assistance to meet the needs of their organization and
increases the likelihood that they will make changes as a result of the
assistance. Exhibits V.13 and V.14 show that about two-thirds of
participants and gatekeepers who were involved in activity design
reported increased teacher knowledge and skills as a result of the
center activity. In contrast, slightly less than half of participants
and gatekeepers who were not involved reported the same outcome. There
are similar differences in the proportions of respondents reporting
effects for most organizational effects.

Variation by Purposes for Participation

As with individual effects, participants who had clear purposes for
participating in an activity were more likely to report effects on their
organizations as a result of center services. For example, Exhibit V.15
shows that 50 percent of respondents who identified all five purposes in
our survey item as reasons for participating in an activity reported
that classroom practices had changed as a result of the activity,
compared with 18 percent of respondents who identified just one purpose.
This finding holds for all types of organizational effects included in
our survey item.

Individual purposes were also related to organizational effects, and
some purposes are associated with greater organizational effects than
others. For example, Exhibit V.16 shows that participants who reported
that their purposes for participating in an activity included "obtaining
ideas or contacts that would help my organization carry out a major,
planned improvement," as a purpose were more likely to report that the
assistance helped the organization initiate changes or make progress in
reform efforts than those who did not identify this purpose. By
contrast, participants who identified

![](media/image29.wmf){width="9.3in" height="7.895833333333333in"}

![](media/image30.wmf){width="9.3in" height="6.785416666666666in"}

![](media/image31.wmf){width="9.3in" height="8.019444444444444in"}

![](media/image32.wmf){width="9.3in" height="8.027777777777779in"}

"keeping current professionally" as a purpose were no more likely to
report that their organization initiated new changes or took the next
step in its reform process than persons who did not identify this
purpose (in other words, the difference in reporting by these two groups
of participants was not statistically significant). Participants with
goals that focus mostly on their own work appear to be less likely to
make the connections between the assistance and the work of their
organization.

Many of the characteristics we ascribe to high-quality assistance are in
fact associated with desirable effects. Comprehensive Center services
are more effective if they are intensive, if they are planned in
consultation with the recipients of services, if they are responsive to
customer needs, and if customers are motivated to participate in and
benefit from the assistance. Our survey data confirm these findings on
the general effects of center services on the work of individuals and
organizations. We turn now to an examination of the specific effects of
Comprehensive Center services.

# Assisting Customers to Respond to the Requirements of ESEA

Among our state case study sites, federal program administrators
maintained that the Comprehensive Centers played a key role in assisting
them to respond to the provisions of the reauthorized ESEA. In a few of
our case study states, SEA staff credited the centers with helping them
through the early stages of planning new programs in response to the
requirements of the reauthorized ESEA. In a closely related set of
activities, several centers also worked closely with states to carry out
the Comprehensive School Reform and Demonstration (CSRD) Program. One
SEA administrator described the centers' contribution this way:

\[They\] are always out in the front when a new initiative comes in.
They set the parameters under which the SEA should be able to think
about these things. When CSRD was introduced, for example, we engaged in
many conversations about what it meant. What's a program? What do the
nine components mean? We were trying to make sense of the time line.
They really helped us make critical decisions about how we'd approach
the issue of getting out to the schools and districts. They helped
provide the structure to make this thing happen at the beginning.

Other states we visited praised the centers' willingness and ability to
perform a variety of tasks---from editing reports to conducting training
to helping with grant applications. Staff members credited the centers
with filling in gaps left by limited staff, time, and resources. One SEA
staff member explained, "\[With their help, I can do\] way more than I
could have done by myself." Similarly, staff members at another SEA
maintain that their center "has made a difference in what we can offer
districts. We personally and financially would have to do the job they
are doing, which we just can't do."

There are many examples of the ways in which the Comprehensive Centers
have extended states' capacities to develop and carry out new ESEA
programs. A few examples include:

-   According to state staff, NYTAC was a vital collaborator and leader
    in New York's efforts to write their ESEA consolidated plan and
    bring the entire department on board. One said, "As the ESEA plan
    was written and finalized, we knew \[center staff\] could help with
    implementation." The center also assisted in providing training
    related to Title I School Support Teams and schoolwide planning and
    implementation.

```{=html}
<!-- -->
```
-   Beginning in 1996, the STAR center worked with one Education Service
    Center (ESC) to build the capacity of the ESC system to assist
    districts and schools in coordinating funding and programs. Together
    they sponsored a statewide summit, followed up with participants,
    and developed the *Resource Guide for Coordinating Funding and
    Programs*. A year later, the center and ESC held a training of
    trainers for the Texas Education Agency and other ESC staff. A state
    staffer said, "Even though I can't \[always provide assistance in
    fund coordination\], I know it's getting done \[through the
    center\]."

```{=html}
<!-- -->
```
-   In Region XV, one of the center's partners, RMC Research
    Corporation, was particularly helpful in building Hawaii's capacity
    to support and administer Title I schoolwide schools. A state staff
    member explained, \"We had no schoolwides five years ago. At that
    point, we needed help. Our schoolwide planning model is based on
    RMC's model. Then two years ago we needed help with School Support
    Teams, so Pacific Resources for Education and Learning (PREL) put us
    in touch with RMC in Denver." Through its partners, the
    Comprehensive Center played a key role in building state capacity to
    implement these ESEA provisions.

```{=html}
<!-- -->
```
-   According to Maryland staff, the Region III Comprehensive Center
    contributed to state, district, and school efforts related to Title
    I and consolidated planning. The center has presented on schoolwide
    programs, developed a scoring rubric for the Title I Distinguished
    Schools Program, and helped the Baltimore City School District
    develop its consolidated application.

At the local level, Comprehensive Center assistance has supported
schools throughout the schoolwide planning process and in the
development of written plans; however, the assistance has had limited
effect on the development of local capacity in schools for planning on
their own. Approximately one-third of district-level gatekeepers
reported that the centers' assistance had enhanced their organization's
capacity to provide technical assistance in support of ESEA programs. In
two of our local case study sites, support for Title I schoolwide
program planning was the central focus of Comprehensive Center
assistance. In each site, schools successfully completed their
schoolwide plans; in interviews, staff members referred to them with
pride (and relief) when they described the results of the centers' work.
In these same sites, however, school staff had mixed reactions when
asked if they could now engage in a similar planning process on their
own. Some respondents reported that they had learned skills in data
analysis and needs assessment that they could now apply on their own;
others insisted that they could not develop another plan without the
help of the center.

Enhancing State Capacity to Deliver Technical Assistance in Support of
ESEA

Overall, 40 percent of gatekeepers reported that center services
improved the capacity of their organization to provide technical
assistance in support of ESEA. There was, however, significant variation
by type of respondent. State gatekeepers were most likely and school
gatekeepers were least likely to report this effect. Sixty-four percent
of state gatekeepers, 37 percent of district gatekeepers, and 15 percent
of school gatekeepers reported that the centers' assistance had
increased their capacity to provide technical assistance in support of
ESEA.

At the state level, center assistance has helped build the capacity of
federal program administrators to deliver technical assistance related
to ESEA programs. Almost two-thirds of state-level gatekeepers reported
that the centers' assistance had enhanced their organization's capacity
to provide technical assistance in support of ESEA programs. Some SEA
staff reported that they "learn by watching" center staff during
workshops, presentations, and consultations and often use the approaches
to technical assistance the center uses with them. A couple of
respondents---based in the Hawaii and Mississippi Departments of
Education---noted that their own capacity for providing technical
assistance has increased since they first began working with the
centers. For example, a staff member from the Hawaii Department of
Education explained:

In the past my needs were more obvious because I had less knowledge of
schoolwide implementation. We had no schoolwides five years ago. . . .
At that point I really needed help. PREL was helpful then. . . . Now I
feel like I have greater expertise in schoolwides and managing Title I.
The substance of efforts is located at the SEA, and \[PREL\] is more a
supporter and helper. . . . Since then \[two years ago\], we took over
and don't rely on PREL in the same way.

Effects on Teaching and Learning

Changes in Teaching Practices

Survey data indicate that changes in teaching and learning are the least
common effects of Comprehensive Center services. Respondents most
frequently cited an increase in teacher knowledge and skills, with over
half of all respondents reporting this outcome. It should come as no
surprise that those closest to the classroom, school-based gatekeepers
and participants, were more likely than district and state gatekeepers
and participants to report such changes. Differences by respondent type
were significant, with 71 percent of school gatekeepers reporting this
effect, compared with 54 percent of district gatekeepers and 39 percent
of state gatekeepers. A similar response pattern also exists for
participants.

Fewer respondents reported that the increased knowledge of teachers
actually resulted in a change in classroom practices. Overall, 35
percent of both participants and gatekeepers reported this effect.
Again, school-based participants and gatekeepers were more likely than
respondents based in other organizations to report this effect.
Approximately 40 percent of school-based gatekeepers and participants
reported a change in teacher practices. In contrast, approximately
one-third of district and state gatekeepers and one-quarter of district
and state participants cited the same change.

Respondents were least likely to report increased student achievement as
a result of center

services, with one-quarter of all gatekeepers and participants
identifying this impact. There were also significant differences by
respondent for this effect, with one-third of school-based participants
and gatekeepers and less than one-quarter of district and state
participants and gatekeepers reporting this effect.

The relatively low percentage of gatekeepers and participants reporting
effects on teaching and learning is most likely affected by the focus of
center services. To conduct the analysis we describe here, we sorted the
30 activities in our survey sample into activities directly targeted to
instruction and other activities. Participants who participated in
activities directly targeted to instruction were more likely to report
effects on teaching and learning. For example, approximately
three-quarters of participants in instruction-focused activities
reported a change in teachers' skills and knowledge. By contrast, 53
percent of participants in noninstruction-focused activities reported
this change.

Producing changes in teaching and learning is not often an specific goal
of Comprehensive Center services. Among our seven state case study
sites, SEA and center staff argued that changes in teaching and learning
may not be appropriate outcomes for the centers' work at the state
level. Among our eight local case study sites, changes in teaching and
learning were an specific goal of center services in only four of the
sites.

Center assistance was more likely to increase teachers' knowledge and
skills than to result in actual changes in teaching practices. About 55
percent of survey respondents reported that Comprehensive Center
services increased teachers' knowledge and skills, and a little more
than a third reported that Comprehensive Center assistance had changed
teachers' classroom practices. Respondents based in schools were more
likely to report these findings than respondents based in districts and
states. In the four local case study sites where center assistance had
been designed to influence teaching and learning, teachers reported
making changes in their practices, though these changes were limited.
Examples of ways the centers increased teachers' knowledge and skills
and affected their practices are described below.

***Cognitively Guided Instruction.*** Many of the teachers we
interviewed reported that they had worked during the school year to
align their instruction with the "CGI philosophy." Teachers reported
that they had incorporated the use of mathematics manipulatives into
their teaching as well as group discussions of approaches to solving
complex mathematical problems that are the hallmark of CGI. Among the
classrooms we observed during mathematics instruction, teachers had
implemented some of the new approaches to instruction to a greater or
lesser degree, depending on their level of comfort with the new
techniques and the particular circumstances of their classroom.

***Reading Success Network.*** The Comprehensive Center that designed
the RSN describes the initiative as a professional development activity,
not a specific reading intervention. From the coaches, principals, and
teachers we interviewed, we heard that the RSN coaching strategies have
been effective at prompting teachers, particularly new or young
teachers, to become more engaged in problem solving and trying new ideas
in their classrooms. Teachers commented that the RSN "gives them tools,"
"polishes existing skills," and "serves as an impetus for
institutionalizing analysis, reflection, and change." These comments all
point to changes in teachers---in their attitudes and perspectives---and
to a lesser degree suggest examples of changes in teaching.

***Training for teachers of limited English proficient (LEP)
students.*** Teachers in both North Carolina and the Riverside City
Schools reported that the training offered by their Comprehensive Center
had provided them with a much better understanding of how language is
acquired and had taught them some useful techniques for working with the
LEP students in their classrooms. Several teachers in Riverside noted
that they now assign limited English proficient students "Reading
Buddies," and teachers in North Carolina described how they had adopted
more physical approaches to communication in the classroom.

**[Effects on Student Learning]{.underline}**

Increases in student achievement were the least commonly reported effect
of Comprehensive Center services, with only 28 percent of participants
and 24 percent of gatekeepers reporting this as an effect of services.
Not surprisingly, school-level gatekeepers and school-based participants
(most of whom are teachers) were much more likely to report this effect
than other types of center customers. Thirty-two percent of school-level
gatekeepers reported that center services had increased student
achievement, compared with 24 percent of district-level gatekeepers and
17 percent of state-level gatekeepers. Also, participants in center
activities specifically targeted at instruction were more likely to
report increased student achievement as an effect of the activity than
were participants in other activities (33 percent of those in
instruction-focused activities, compared with 11 percent of those in
other activities).

Even among those participants receiving assistance specifically focused
on instruction, the proportion of participants reporting increased
student achievement as an effect of the assistance is relatively low.
Similarly, there is little evidence that the Comprehensive Center
assistance has improved student outcomes on a wide scale among our case
study sites. Only one local case study site had assembled evidence of a
center's impact on student outcomes. In Alaska, staff at the Snowbank
Village School reported that student absenteeism and tardiness decreased
significantly since the school began the AOTE process; these changes are
accompanied by much higher levels of parent involvement and
collaboration between the village and the school. The Comprehensive
Centers that sponsor CGI and RSN are currently collecting data on
student outcomes as part of their evaluation of those programs; however,
the results of those evaluations are not yet available. There are other
sites that have outcome data that were not included in our case study
sample. For example, the Comprehensive Center network is currently
conducting an extensive evaluation of the Reading Success Network across
the 15 Comprehensive Center regions. This evaluation includes an
examination of student outcomes and should produce useful findings.

## Center Strategies for Assessing the Effects of Their Services

The Comprehensive Centers invest large amounts of time and effort in
compiling information on their activities and customers for reporting
back to ED. However, the centers collect relatively little systematic
information about the effects of their services (although the evaluation
of the RSN described above is one notable exception).

**[Collecting Data on Activities and Customers]{.underline}**

Every six months the Comprehensive Centers compile data on their
activities and on the customers they have served and submit those data
as part of semi-annual progress reports to ED. ED uses the data to
report to inform its own oversight of the program, and to measure the
Comprehensive Centers' progress toward the following performance
indicators developed under the Government Performance and Results Act
(GPRA):

-   ***1.1 Addressing legislative priorities:*** An increasing
    percentage of Comprehensive Center customers will be schoolwide
    programs, high-poverty schools, and Bureau of Indian Affairs-funded
    schools.

-   ***1.2 Integrating technical assistance:*** An increasing percentage
    of Comprehensive Center activities will provide integrated,
    noncategorical technical assistance (such as focusing on standards,
    assessment of special populations, reading, other challenging
    curricula, and whole-school reform).

To ensure that the data can be aggregated in a standardized way across
Comprehensive Centers, ED and the centers developed a uniform reporting
format in early 1998. Each center has developed a fairly sophisticated
database for recording basic information about each of its activities,
including information about the content and format of each of its
activities and about the customers and organizations served. A summary
of progress report data for 1999 is attached in Appendix F. Examples of
findings in this summary include the following:

-   In 1999, the Comprehensive Centers reported 3,737 professional
    development events, 1,181 consultation events lasting one day or
    more, and 2,679 events that they convened or facilitated. These
    counts represent slight increases over activity in FY1998.

-   The centers reported providing a larger proportion of high-intensity
    activities than in FY 1998. In 1999, for example, the centers
    reported that 58 percent of the professional development events they
    sponsored lasted for more than one day (up from 44 percent in FY
    1998).

-   The Comprehensive Centers had more contacts with teachers and
    schools than with other types of individuals or organizations.

-   Almost 90 percent of all contacts with schools were with
    high-priority schools (Title I schoolwide programs, high-poverty
    schools, or BIA schools).

**[Collecting Data on the Effects of Comprehensive Center
Activities]{.underline}**

Although the Comprehensive Centers collect large amounts of data on the
number of services they provide, the topics they address, the duration
of their activities, and the customers they serve, most centers collect
relatively little systematic information on the quality,
cost-effectiveness, and impact of their services.

Most Comprehensive Centers rely primarily on informal feedback loops to
assess their effectiveness. All centers pay close attention to verbal
feedback from SEA and LEA customers. State liaisons frequently rely on
their close relationships with SEA staff members to solicit feedback on
center services, and center staff members observe district and school
customers on follow-up visits to assess whether their work has had an
impact. The Comprehensive Centers also consider requests for follow-up
services and repeat business to be an important indicator of their
effectiveness. As one staff member described it:

> I am not doing a formal evaluation but always kind of scanning. . . .
> If we don't meet the needs of the clients in the region, we won't be
> refunded. This is the ultimate test. It is market-driven. Would they
> want us there again?

In addition to anecdotal information collected by the centers in various
ways, most centers administer surveys to clients at the end of each
professional development activity. These methods of self-evaluation are
usually only useful in helping the centers weed out activities that have
been seriously flawed or ineffective, however.

A few centers conduct random surveys of their customers some time after
they have provided a service. For example, NWREL\'s Comprehensive Center
administers a survey to a random sample of its clients that asks whether
the customer has learned new skills and whether those skills have led to
changes in practice. The New England Comprehensive Assistance Center
commissions an external evaluation of its work each year that relies
heavily on customer surveys.

In interviews, many center staff members discussed the need for more
systematic and formal processes for evaluating the effects of their
activities. As one center staff member noted, "We have happy customers,
but we do not have hard data on this." In addition, center staff members
reported that it is difficult or impossible to assess the
cost-effectiveness of particular center services, primarily because the
Comprehensive Centers do not track staff members' time or other expenses
by activity or by category of service.

Two of the Comprehensive Centers GPRA indicators require some
standardized data collection across centers on the effects of their
services:

-   ***1.3 Addressing customer needs:*** An increasing percentage of
    state and local administrators served by the CCs will report
    satisfaction with the usefulness of technical assistance provided.

-   ***1.4 Showing impact with customers:*** Participants in center
    activities report that they have incorporated information or skills
    they have learned from the centers into their work.

The customer survey data generated by this evaluation was used to report
on these two indicators in ED's 1999 GPRA Performance Report.

Summary

Customer survey data collected in this evaluation show clear effects of
Comprehensive Center services, especially as they are associated with
various characteristics of high-quality assistance. Thus, we see that
customers who receive a higher volume of services report a greater
number of effects, customers who are involved in planning the service
report a greater number of effects, and customers who are motivated to
participate in center activities report a greater number of effects.
When we examine the specific effects of Comprehensive Center services in
greater detail in our case study sites, however, especially in light of
the expectations set out in the centers' authorizing legislation, we
find that they are comparatively limited in scope. For example, the
centers most often enhance states' capacity to carry out ESEA programs
by providing extra hands to accomplish the work involved; much more
rarely do they enhance state capacity by increasing the knowledge and
skills of regular SEA staff. The Comprehensive Centers have assisted
many schools to develop Title I schoolwide program plans; after the plan
is complete, however, staff in those schools remain unsure of their
ability to undertake a similar needs assessment and strategic planning
process on their own. Improvements in teaching practice that can be
clearly attributed to the effects of Comprehensive Center assistance are
limited to those activities that are targeted directly at classroom
teachers.

Much of the centers' work has produced desirable effects on the work of
states, districts, and schools. It is not surprising that those effects
are sometimes limited in scope and almost always limited to the centers'
highest intensity, most carefully targeted activities. Many studies have
shown that the investment required to produce dramatic changes in the
operation of schools and classrooms is considerable. In most cases it is
beyond the capacity of the Comprehensive Centers to provide the required
levels of support. As a result, it is important that both the
Comprehensive Centers and their managers in the Department of Education
set realistic goals for the centers' work and set expectations for
outcomes appropriately.

VI\. COLLABORATION WITH COMPREHENSIVE CENTERS AND OTHER TECHNICAL
ASSISTANCE PROVIDERS

Title XIII of ESEA requires the Comprehensive Centers to "coordinate
services, work cooperatively, and regularly share information with, the
regional educational laboratories, the Eisenhower regional consortia . .
. research and development centers, State literacy centers . . . and
other entities engaged in research, development, dissemination, and
technical assistance activities which are supported by the \[U.S.
Department of Education\] as part of a Federal technical assistance

system. . . ." \[P.L.103-382, Section 13102 (a)(4)\]. In response to
this mandate to provide "coordinated assistance," the Comprehensive
Centers have engaged in a variety of collaborative activities with one
another, and with other federal, state, and local organizations. These
activities have varied not only in content and focus, but also in terms
of the depth and intensity of inter-agency interaction that is required.
As a group, they illustrate the extent to which the term "collaboration"
has been broadly and diversely interpreted.

Collaboration within the Comprehensive Center Network

Since the centers' inception, the 15 Comprehensive Center directors and
staff from each of the centers have met quarterly for three days to
discuss issues of common concern and to coordinate their work. These
quarterly meetings have become important opportunities for the centers
to showcase their work for one another, exchange ideas and resources,
and to plan together on joint projects. In addition, as the
Comprehensive Centers have matured, they have developed other mechanisms
for collaborating within the network.

[Reading Success Network]{.underline}

The most visible example of collaboration among the 15 Comprehensive
Centers is the Reading Success Network (RSN), a professional development
initiative designed to improve reading achievement among students in
grades 1-3. The RSN trains coaches who support classroom teachers'
efforts to improve their reading instruction by strengthening their
skills in the diagnosis of reading difficulties and early intervention
efforts. The Southern California Regional Assistance Center developed
the RSN in 1996 as one of its major professional development
initiatives. In 1998, all 15 centers adopted the RSN in response to some
prodding from ED to take on a project that would increase the centers'
visibility as a program and advance ED goals. Reading, one of ED's top
priorities, emerged as the topic of choice. The RSN initiative was
described by one center director as "the most significant" collaboration
among the centers because of the depth of their involvement.

Each center sends staff to network-sponsored RSN training sessions; at
these meetings centers share information, materials, staff, and
expertise to build their capacity to implement the RSN in their own
regions. In addition to helping center staff improve their own knowledge
and skills in this area, participation in RSN has also facilitated
center efforts to share expertise and resources while providing direct
services to clients in their regions who seek to introduce RSN in their
schools and districts. In 1999, the Comprehensive Centers began work on
a network-wide evaluation of the RSN and its effects on teaching and
learning in the schools where it is being implemented. A committee of
center staff members met several times to design the evaluation, and the
network has hired two evaluators to oversee their data collection
efforts and to analyze and report on the data.

**[Sharing Staff]{.underline}**

We heard of a few instances where Comprehensive Centers "borrow" staff
with specialized expertise from other centers. Staff at the Southwest
Comprehensive Regional Assistance Center, for example, have special
expertise in Title VII evaluation (because of their previous work at
ED's Title VII Evaluation Assistance Centers). These staff members have
worked with other Comprehensive Centers to deliver training on program
evaluation in bilingual education. In a similar vein, the Region VI
Comprehensive Assistance Center has made the summer institutes of its
major professional development initiative, Cognitively Guided
Instruction, open to school teams from other Comprehensive Center
regions. The Alaska Comprehensive Regional Assistance Center paid to
send teams from some schools in its region to the CGI institute in
summer 1999.

Some Comprehensive Center staff members see collaboration among
Comprehensive Centers as an important strategy for extending the
expertise available within their own center, acknowledging that some
centers are better equipped than others to respond to certain client
needs.

**[Coordinating Council and Other Special Interest Groups]{.underline}**

The Comprehensive Center network has developed several mechanisms for
sharing information and pooling resources. A "Coordinating Council" of
five center directors facilitates communication between ED and the
center network. The Coordinating Council meets with ED staff four times
a year, in the months between center directors' meetings, and
disseminates information and decisions from those meetings to the rest
of the network. The Coordinating Council has allowed the centers to
remain in closer contact with their own program officers at ED and with
OESE and OBEMLA senior managers. A publicity committee made up of center
directors works on developing marketing materials that all of the
centers can use. Finally, the centers have some resource dissemination
groups, such as a school climate specialty group that was convened by
the Region XI Comprehensive Assistance Center to study how to integrate
school climate issues into the schoolwide program planning process.

Collaboration with Other Technical Assistance Providers

The Comprehensive Centers collaborate with organizations outside their
network to accomplish a range of objectives that ultimately shape the
duration, nature, and content of the collaborative relationships, and
help to account for wide variations in the interactions that take place
among and between staff from different collaborating organizations. The
collaborations that we studied can be divided into two groups: (1) those
with a single objective or focus, and (2) those that address multiple
topics, issues, and concerns.

[Collaborations with a Single Objective]{.underline}

In these initiatives, the centers and their partners work closely
together over several months or years to provide direct assistance to
clients on a specific topic or narrow range of issues. Often the
Comprehensive Centers and their partners share resources, and jointly
provide services such as on-site consultations or hosting training
events. The close working relationship sometimes creates the impression
among service recipients of a single technical assistance entity.

Examples of activities in this first category include the following:

-   Comprehensive School Reform Demonstration program (CSRD): Enacted in
    1997 under the Department of Education Appropriations Act, CSRD
    provides funding for schools to implement comprehensive,
    research-based school reform programs. Since 1996 all of the centers
    have provided technical assistance related to the implementation of
    CSRD in their respective regions. PSA examined the CSRD-related
    activities of four centers: (1) the New York Technical Assistance
    Center, (2) the Southeast Comprehensive Assistance Center, (3) the
    Northwest Regional Educational Laboratory's Comprehensive Center,
    and (4) the Region XIV Comprehensive Center. These centers all
    collaborate with regional educational laboratories to: (1) host
    model showcases to introduce the CSRD program and reform models to
    LEAs; (2) sponsor conferences that address CSRD implementation and
    evaluation issues; (3) assist SEA personnel in completing their
    applications for CSRD funding; (5) review CSRD subgrantee
    applications and help SEA staff to identify and select the most
    promising applicants; and (4) provide materials and training to CSRD
    subgrantees.

-   Data Strategies Initiative (DSI*)*: Since the early 1990s, federally
    funded technical assistance providers in New England have met
    regularly to share expertise, resources, and information, and to
    discuss projects and activities of mutual interest. The New England
    Technical Assistance Collaborative, as it is known, includes the
    regional educational laboratory, the Comprehensive Center, and the
    Eisenhower regional mathematics and science consortium serving New
    England. One of the most recent initiatives to come out of these
    regular meetings is the Data Strategies Initiative, a joint project
    of the New England Comprehensive Assistance Center; the Eisenhower
    Regional Alliance for Mathematics, Science, and Technology Education
    Reform (the Regional Alliance); and the Northeast Regional Resource
    Center (NERRC). The DSI involves a group of states and 11 districts
    in an effort to enhance the capacity of program administrators to
    use data to drive decision making and to upgrade state and local
    accountability systems.

-   Civil Rights Project: The Region IV Comprehensive Center at AEL,
    ED's Office for Civil Rights (OCR), and the Southeastern Equity
    Center, an equity assistance center based in Miami, Florida, are
    collaborating to help school districts in North Carolina comply with
    civil rights laws regarding the education of limited English
    proficient students. OCR initiated the collaboration in an effort to
    bring the resources of all three organizations to bear on the
    districts' legal and educational problems. The Comprehensive Center
    and the Southeastern Equity Center combine their knowledge and
    expertise in designing and developing educational programs for LEP
    students, and in providing training and professional development to
    help teachers meet these students' academic needs more effectively.
    This support, coupled with legal guidance from OCR, has been
    invaluable to the resource-strapped, noncompliant school districts.

-   Self-Assessment Guide for Excellent Services (SAGES) for Children
    and Youth in Homeless Situations: The New York Technical Assistance
    Center, Support for Texas Academic Renewal (STAR) Center, Alaska
    Comprehensive Regional Assistance Center, and the Region XIV
    Comprehensive Center worked together on a collaborative research
    project with the Texas Office for the Education of Homeless Children
    and Youth. The goal of the project was to identify factors that
    contribute to academic success for homeless students and to develop
    a tool for assessing the impact of academic programs on homeless
    students. In order to broaden the scope of the project and increase
    its national significance, approximately two staff members from each
    of the participating organizations interviewed classroom teachers,
    homeless liaisons, school and district administrators, and homeless
    services providers in 25 school districts around the nation. The
    project team then analyzed the data and developed a self-assessment
    tool for educators and service providers.

-   Superintendents Policy Institute: The Region III Comprehensive
    Center, together with Southeastern Regional Vision for Education
    (SERVE)---a regional education laboratory based in Georgia---and
    superintendents from two school districts, organized and led a
    working group in examining issues pertaining to accountability,
    language minority students, education reform, and school
    improvement. The goals of the Institute, which brought together
    approximately 30 superintendents from school districts with high
    concentrations of poor students, were to: (1) generate cutting edge
    thinking about education reform and school improvement, and (2)
    disseminate new knowledge developed through collaborative inquiry
    among the participants. In addition, as part of the collaboration,
    one of the superintendents has allowed staff from the center to
    field-test in his school district some of the theories and ideas
    that emerged from the Institute.

[Open-Ended Collaborations]{.underline}

Other collaborations consist of partnerships between technical
assistance providers to deliver services to a common customer base. In
both cases, the collaborating partners are housed in the same host
organization, and their collaboration takes the form of an institutional
partnership. Examples of these collaborations include:

-   Services to rural areas in Comprehensive Center Region IX: The
    Southwest Comprehensive Regional Assistance Center (SWCC) and the
    Utah, Colorado, Arizona, and New Mexico Rural Systemic Initiative
    (UCAN-RSI) are both located at New Mexico Highlands University.
    UCAN-RSI is a National Science Foundation initiative that is
    designed to involve community members in efforts to improve science,
    mathematics, and technology education in selected rural school
    districts in the four-corner states. In partnership with UCAN-RSI,
    SWCC has endeavored to expand its services to American Indian
    communities, help reform education through professional development,
    and improve science, mathematics, and technology education in rural
    communities.

-   Services to the Pacific region: The Pacific Comprehensive Regional
    Assistance Center is one of several organizations located within
    Pacific Resources for Education and Learning (PREL), a nonprofit
    corporation that serves Hawaii and other entities throughout the
    Pacific. In addition to operating the Comprehensive Center, PREL
    operates the regional educational laboratory and the Eisenhower
    mathematics and science consortium that serve the Pacific region.
    Since the early 1980s, PREL (in both its current and previous
    capacities) has hosted the Pacific Education Conference, a three-day
    meeting for teachers, administrators, and other stakeholders in the
    Pacific region. Within PREL, the Comprehensive Center collaborates
    with the regional educational laboratory, the STAR Program, and the
    Pacific Math and Science Regional Consortium to host the conference.
    In addition, the Comprehensive Center and other technical assistance
    organizations within PREL have sponsored a leadership academy for
    school and district administrators, and provided intensive technical
    assistance to a local public school on a small Pacific island that
    enrolls 1,200 students in the first through eighth grades.
    Collaboration between the center and other PREL organizations is
    facilitated by the fact that PREL leadership and staff members share
    a common mission and service region.

Conditions that Encourage Collaboration

Participants in the collaborative activities we studied in depth
identified a common set of conditions that help to ensure the success of
a collaboration.

[**Clear Benefits for Both Partners**]{.underline}

Our respondents agreed that collaboration will only occur if the parties
involved are able to identify how they and their customers will benefit.
When collaboration works well, collaborating organizations are able to
extend their reach and influence in their service areas, and customers
receive better services that draw upon a wider range of resources and
expertise. Staff from the Region IV Comprehensive Center at AEL and the
Southeastern Equity Center observed that their collaborative efforts
facilitated the delivery of a "double dose of expertise \[because\]
instead of one resource, \[clients\] have two. . . ." The two providers
maintained that they were able to reach more customers than would have
been possible otherwise. Collaborating partners in the Data Strategies
Initiative (DSI) expressed similar sentiments, saying that the
collaborative effort "enabled us to do something that we could not have
afforded to do on our own, and to do it the way we really wanted to do
it."

Participants in the SAGES research project argued that their
collaboration resulted in a "more comprehensive, meaningful, and
validated product" than would have been developed without the input and
perspectives of staff from several Comprehensive Centers around the
country. Similarly, Comprehensive Center staff members credited the
UCAN-RSI collaboration with improving their work in Indian education,
and encouraging them to look beyond the elementary grades in the course
of their work.

Staff in collaborating organizations also reported that the
collaboration had enhanced their own knowledge, skills, and
capabilities. For example, Comprehensive Center staff members described
the SAGES project as an "enlightening process" that increased awareness
among the participants of the expertise that exists in other
Comprehensive Centers. Staff members from collaborating organizations
reported that they had also enhanced their personal and organizational
knowledge and capacity by: (1) reviewing materials developed by others,
such as surveys, training manuals, and research instruments; (2)
learning how to create a Web site and use other technology to improve
services to clients; and (3) listening to their partners discuss
unfamiliar topics such as schoolwide change, education for the homeless,
and mathematics and science education.

**[Mutual Trust and Respect]{.underline}**

Collaborating partners report that they are much more likely to form
partnerships when they know about and respect the knowledge and
expertise of their potential partners. In several of the collaborations
that we studied in detail, center staff had prior working relationships
with individuals in their partner organizations, or had first-hand
knowledge of relevant work they had done. For example, a staff member
from the Office for Civil Rights (OCR) reported that she had invited the
Region IV Comprehensive Center and the Southeastern Equity Assistance
Center to work with her office on a project because she was aware of the
Comprehensive Center director's knowledge and capabilities, and knew
that staff in both organizations had complementary expertise.

Mutual trust is also a key ingredient in a successful collaboration. A
staff member from one of the regional educational laboratories noted
that her collaboration with the Comprehensive Center in her region had
worked well because her counterpart at the Comprehensive Center did not
"jockey for position" with her. Rather, they shared responsibilities and
were supportive of each other to the extent that clients were not aware
that they were employed by separate institutions.

[**Economic Efficiency**]{.underline}

Center staff members reported that working together can sometimes save
money. Several interviewees from the centers and the regional
educational laboratories pointed out that it was much more efficient for
the two organizations to host a single CSRD model showcase together than
for them to sponsor separate events on their own. Staff members at PREL
explained that collaboration among organizations that serve the Pacific
region is motivated by a desire to reduce travel costs. When one staff
member travels to a distant island and provides technical assistance
services on behalf of the Comprehensive Center and other technical
assistance organizations housed at PREL, all of the partners save time
and money.

Conditions That Impede Collaboration

Nearly everyone with whom we spoke identified the following obstacles to
collaboration: (1) time and other costs associated with collaboration;
(2) competition among technical assistance providers; and (3)
differences among technical assistance providers in terms of their
mission, agenda, philosophy, and geographical boundaries. Several
Comprehensive Center staff members reported that, without the
legislative requirement to do so, they would probably spend less time
and fewer resources collaborating with other organizations.

**[Costs of Collaboration]{.underline}**

Comprehensive Center staff members and their counterparts in other
organizations also maintained that collaboration carries significant
costs that sometimes outweigh the benefits to customers and to
collaborating organizations.

Nearly everyone we interviewed commented on how time consuming and
expensive collaboration can be. Collaboration requires that partners
invest time in planning projects that will advance the interests of both
organizations, dividing responsibilities in a way that makes the most
sense for the work at hand, and maintaining close communication between
partners. The fact that the Comprehensive Centers operate on a very
limited budget has led to frustration and the feeling among some that
the true "cost" of collaboration will be a reduction in the quantity or
quality of services to clients. One respondent effectively summarized
the sentiments of several people by stating: "We are under-funded and
over-promised. We have a press to have impact, satisfy clients, and get
new work. \[This\] doesn't make a lot of time to collaborate and
leverage resources." Another complained that "Collaboration often looks
like we are taking money away from direct services."

Several respondents who are involved in the CSRD-related collaboration
questioned the wisdom of ED's decision to fund two different technical
assistance programs to do the same work with essentially the same
clientele. They assert that it might have been more cost effective if
all of the funding had gone to either the regional educational
laboratories or the Comprehensive Centers, rather than to both. This
would have eliminated the significant amount of time and other resources
that were spent by both organizations in communicating, coordinating,
and planning with partners that were not always close by.

**[Competition Among Collaborators]{.underline}**

Comprehensive Center staff and their potential partners acknowledge that
mistrust, suspicion, and the desire to protect their organizations from
competition have impeded collaborative efforts. Several respondents
attributed concerns about protecting "turf" to the fact that the
Comprehensive Centers and other federally funded technical assistance
providers know that organizations that are partners now might become
competitors in future rounds of competition for ED contracts and grants.
In at least three Comprehensive Center regions, Comprehensive Centers
are now collaborating with organizations that had competed against them
for the original Comprehensive Center grant in 1995, and that they have
every reason to believe will compete against them for ED grants in the
future. One Comprehensive Center director explained that the fact that
the regional educational laboratory in her region had competed for, and
lost, the Comprehensive Center grant hindered collaboration in the first
year or two of the Comprehensive Center's existence.

**[Differences in Mission, Philosophy, or Service Area]{.underline}**

Finally, several interviewees noted differences in the organizations'
mission, agenda, philosophy, and or geographic boundaries that affect
their ability to collaborate. Respondents involved in several different
projects observed that when regional borders do not overlap completely,
as is the case with the Comprehensive Centers and the regional
educational laboratories, it is difficult to assign responsibilities
equitably. Furthermore, the amount of resources that each organization
will be able to commit to a specific state or sub-area will vary based
on the total size of their respective service areas.

Several respondents noted that collaboration can be difficult because
technical assistance providers often have different missions and
concepts of technical assistance. One Comprehensive Center staff member
argued that because the laboratories have "a research agenda," and the
centers have "a service agenda," the two are often involved in very
different activities and this makes collaboration difficult. Another
Comprehensive Center staff member explained that because individuals who
are employed in organizations that provide technical assistance are
trained differently from those in research and development and those in
advocacy organizations, they emerge with different "values." These
values in turn shape goals, objectives, and activities.

Summary

In response to a mandate in their authorizing legislation to collaborate
with one another and with other technical assistance providers as part
of a \"federal technical assistance system,\" the Comprehensive Centers
collaborate among themselves and with others on a somewhat limited
basis. Their experience shows that collaboration can have important
benefits and allow the Comprehensive Centers to extend their reach and
their influence within their regions. Because collaboration gives
centers access to a wider range of expertise, information, and other
resources for serving customers, it can also improve the quality and
effectiveness of their services. However, nearly every respondent
interviewed\--both Comprehensive Center staff and staff of collaborating
organizations\--also reported that collaboration is both time consuming
and expensive. In addition, differences in mission and goals among
potential partners, as well as past and future competition for the same
customers and funding sources, make collaboration more difficult. As a
result, the Comprehensive Centers maintain that collaboration is only
worthwhile if it advances the interests of each of the partners
involved, and if the potential benefits of the collaboration outweigh
the costs. The experience of the Comprehensive Centers to date suggests
that these conditions are met in a relatively limited number of cases.

VII\. CONCLUSIONS

This evaluation of the Comprehensive Centers program was designed to
address two basic questions:

(1) How are the Comprehensive Centers fulfilling their statutory
    mission, as embodied in Title XIII of the Elementary and Secondary
    Education Act (ESEA)?

(2) How does the Comprehensive Centers' work contribute to educational
    change and improvement in the states, districts, and schools they
    serve?

Because the Comprehensive Centers are a new program, and because the
work they do represents a significant departure from ED's past programs
of technical assistance in support of ESEA, this evaluation set out to
describe in some detail the portfolio of services provided by the
centers, as well as the effect of those services on the individuals and
organizations served.

As we have described, the Comprehensive Centers' mission and goals were
far broader than those of the five categorical technical assistance
programs they had replaced. For the first time, technical assistance in
support of ESEA programs was meant to support comprehensive, schoolwide
reform, integrating federal programs and funding streams to support the
attainment of ambitious goals for student performance. In comparison
with the work of the antecedent technical assistance programs, a
substantial portion of which had focused on helping ESEA grantees with
the nuts and bolts issues of administering programs---such as developing
applications, conducting evaluations, coordinating with other
programs---the Comprehensive Centers' aims of technical assistance had
now become significantly more ambitious, much closer to changing the
core of schooling itself.

By the end of their third full year of operation, the Comprehensive
Centers had achieved at least two important milestones. First, they
developed a fairly coherent portfolio of services that responded to the
broad charge set out in their authorizing legislation. The Comprehensive
Centers spend the majority of their staff time and other resources on a
class of major initiatives and activities that they describe as
high-intensity, high-impact assistance---professional development and
consultation services designed to change the behavior of organizations
and individuals. A review of the centers' major initiatives and of the
way center staff describe their work suggests that most Comprehensive
Center activities are indeed "comprehensive," meaning that they support
the vision of reform contained in other sections of ESEA.

Second, the centers have succeeded, in a relatively short period of
time, in cultivating a relatively large base of satisfied customers. In
nationally representative surveys of state and district federal programs
administrators, we found that the centers had provided services to a
significant proportion of their potential customers at each of these
levels. The centers had targeted their services appropriately in
response to the priorities established in their authorizing legislation.
As a result, the centers' reach has been considerably broader among
their highest priority customers.

In general, the Comprehensive Centers' customers are satisfied with the
services they receive and find them useful. Most established customers
value Comprehensive Center services enough to return for additional
assistance. Among potential district customers who have heard of the
centers but have not received services, the reasons given for not
seeking out assistance include not having either enough information
about the centers or enough time to get that information, rather than
reservations about the quality of Comprehensive Center services. These
findings suggest that the centers have been successful in establishing a
reputation in their regions as a useful source of technical assistance.

A close examination of the effects of Comprehensive Center assistance,
as they are reported by customers, demonstrates anew the importance of
intensity, duration, and customer focus in high-quality technical
assistance and professional development. If the centers are to
contribute to educational change and improvement, their services require
more funding.

***The Comprehensive Centers' technical assistance is more useful to
customers when it is intensive and when it extends over time.*** The
volume and intensity of center services are good predictors of both
customer satisfaction and the reported effects of center services.
Gatekeepers are more likely to report that center services result in
improvements in their organization's work if their organizations receive
a large volume of different kinds of services (professional development,
on-site consultation, help convening special events) and if their
organizations receive services on multiple topics. Participants are more
likely to report effects if they receive follow-up services. In
addition, participant satisfaction with center services increases as the
number of follow-up activities increases.

The fact that larger centers, on average, receive higher ratings for the
quality and usefulness of their services also illustrates the importance
of volume and intensity of services as predictors of customer
satisfaction. On average, larger centers provide a larger volume of
services to their customers than smaller centers. Larger centers do not
serve proportionately larger numbers of customers than smaller centers;
given the choice, larger Comprehensive Centers appear to allocate
additional resources to serving individual customers more intensively,
rather than to extend their reach. Our findings suggest that serving a
fixed number of customers more intensively with additional resources may
be a reasonable choice.

***The Comprehensive Centers' technical assistance is more useful to
customers when it is tailored to address their needs and interests.***
Both gatekeepers and participants who participate in the design and
planning of Comprehensive Center services are more likely to report that
those services had an effect on their own work or the work of their
organizations. End-user participants who report that an activity served
their purposes are more likely to report that it also had an effect on
their work. Those who are committed to benefiting from Comprehensive
Center services are more likely to do so; thus we find that teachers and
administrators who volunteer to participate in center activities are
more likely to benefit from those activities than those who are required
to attend, and those who bring a clear sense of purpose to the activity
are also more likely to benefit. These patterns are not surprising.
However, it is probably also true that centers that seek out motivated
customers and tailor their services to address their particular needs
and interests are likely to produce greater effects on the individuals
and organizations served.

***The effects of Comprehensive Center services are most apparent when
services are intensive and targeted directly at the individuals and
organizations expected to change.*** Gatekeepers based in schools are
much more likely to report changes in teaching practices and student
learning as a result of Comprehensive Center services than state-level
or district-level gatekeepers. For these gatekeepers, the link between
center services and the intended effects (in this case, improvements in
teaching and learning) is direct, and it is relatively easy to trace the
effects back to the influence of the centers' work. Similarly, end-user
participants in center activities are much more likely to report changes
in teaching practices and student learning if the activities in which
they participated are designed to address instruction.

Among our case study sites, we found that changes in the ways schools
organize to serve students and changes in teaching and learning were
easiest to detect in local sites where customers had received intensive
assistance specifically designed to produce intended effects.
Cognitively Guided Instruction teachers reported experimenting with new
teaching strategies in their mathematics classrooms, for example, and
schoolwide planning teams completed new plans in schools that had
received coaching on the schoolwide planning process. In our case study
sites at the state level, Comprehensive Center staff members maintained
that their goal was to build state capacity to support change in schools
and changes in teaching and learning. Not surprisingly, in these cases
it was much more difficult to track the influence of the centers' work
through to its intended effects on schools and classrooms. This was
especially true with effects on teaching and learning. In fact, most of
our state-level respondents argued that it was inappropriate to expect
Comprehensive Center services to produce measurable effects in schools
and classrooms.

The centers' goals for their own work have always been ambitious. A
quick reading of mission statements prepared by the 15 Comprehensive
Centers in Appendix A shows that the centers are firmly committed to
bringing about fundamental change in the way schools operate and to
improving teaching and learning for the students served under ESEA. The
centers did not create these lofty goals in a vacuum. The preamble to
Title XIII of ESEA describes the purpose of ED's technical assistance
as: (1) administering ESEA programs; (2) implementing school reforms to
improve teaching and learning for all students; (3) coordinating those
reforms with other federal, state, and local efforts to help all
students meet challenging state standards; and (4) adopting, adapting,
and implementing promising and proven practices for improving teaching
and learning. ED has reinforced the notion that the Comprehensive
Centers' work should produce change in classrooms and schools. ED's
encouragement and support of the Reading Success Network as a
programwide initiative is a good illustration of its belief that the
centers should be able to demonstrate some positive effects on teaching
and learning.

In the face of these ambitious goals, the Comprehensive Centers program
faces continuing challenges. The centers' broad mandate and limited
resources have forced them to make choices about whom they will serve
and the breadth and depth of services that they will provide. Because
the Comprehensive Centers have been at times unable to provide the
volume of services that the antecedent centers had provided, some
customers have been dissatisfied with the level and kinds of assistance
they have received.

Comprehensive Center customers report that their own work and the work
of their organizations has changed as a result of Comprehensive Center
services. Our observations of these effects in our case study sites
suggest, however, that the centers have been most successful in
disseminating new ideas related to school improvement, not in promoting
the kind of whole cloth reform that might plausibly lead to changes in
teaching and learning for all students. As a result of center services,
states are running programs more effectively and providing new technical
assistance services to the field, and school and district administrators
have acquired new skills in program planning and needs assessment.
However, Comprehensive Center services, at their present level of
intensity, do not produce radical changes in the behavior of
organizations and individuals in any but a few instances. This dilemma
is not unique to the Comprehensive Centers program. Instead, it reflects
the great level of effort required to achieve ED's goals for its
technical assistance systems.

Recommendations for Future Oversight of the Program

The vision for the Comprehensive Centers program set forth in Title XIII
of ESEA is an appealing one. Title XIII charges the Comprehensive
Centers with encouraging and supporting education reform and improvement
that will benefit those students who most need improved education
opportunities. In response to this charge, the centers have engaged in
work designed to produce a host of important outcomes. In response to
this charge, the centers have engaged in work designed to produce a host
of important outcomes. The key findings of this evaluation suggest that
the operations of the Comprehensive Centers could be modified to build
on their early work:

***The Comprehensive Centers could focus attention on developing their
capacity to respond to customers\' particular local conditions and
customer needs and interests.*** Customers are generally satisfied with
the content and the format of Comprehensive Center assistance they
receive. The dissatisfaction that exists hinges on customers\'
perception that the assistance could be even more closely tailored to
address their particular questions, needs, and interests and the local
conditions they face. In response, the Comprehensive Centers could place
special emphasis on learning even more about particular customers\'
needs and interests and fine-tuning their services to supply the support
and expertise their customers require. This approach would probably
require even more intensive services for most customers, however, so it
would have serious implications for the breadth and reach of the
Comprehensive Centers in the future.

***The Comprehensive Centers could consider targeting their services
even more carefully on organizations that can benefit most from their
assistance.*** Surveys of potential customers suggest that there is a
sizable market for Comprehensive Center services. The Comprehensive
Centers have been cautious to date about marketing their services in a
way that would create more demand than they can satisfy. However, the
centers could consider stepping up their marketing activities in order
to locate schools and districts that are best able to benefit from their
particular expertise. In this way they could achieve the best fit
between their particular portfolio of services and expertise and the
needs and interests of the customers they serve. This strategy would
require that the centers turn down requests for assistance in some
cases. In these cases, they would need ED\'s support in responding to
potential customers who are unhappy about not receiving services.

***The Comprehensive Centers could develop further strategies for
capacity building in states and districts.*** Most Comprehensive Center
services to states and districts aim to build their capacity to improve
education programs in high-priority schools. In many cases, however,
this capacity building consists of supplying additional staff to extend
the state\'s or district\'s reach among schools, rather than adding
knowledge or expertise not readily available within the organization.
The Comprehensive Centers also need to consider ways that they can
increase the knowledge and skills of SEA and LEA staff to enable them to
serve schools more effectively.

***The Comprehensive Centers could build on opportunities to collaborate
across the network on key initiatives like the Reading Success
Network.*** When collaboration among technical assistance providers is
successful, Comprehensive Center staff members report that it enhances
the quality of their services to the field. The RSN is a good example of
a case where collaboration on a joint effort has allowed the entire
network to benefit from products and expertise developed by individual
centers. The Comprehensive Centers should continue to pursue similar
opportunities in the future, especially with regard to current efforts
to develop strategies for assisting low-performing schools.

ED can also take steps in the next phase of the centers\' operations to
strengthen its oversight of the program:

***ED should continue to explore ways to build its capacity to provide
effective assistance and support to the Comprehensive Centers.*** In the
past year, ED program managers of the Comprehensive Center program have
taken steps to improve communications between the centers and various ED
program offices by hosting meetings at ED on various high-priority
topics. ED should continue to seek ways to provide centers with the
information and access to program managers that they need to be
responsive to ED's needs and purposes.

***ED should continue to work with the Comprehensive Centers to develop
standard procedures for monitoring their own work and for evaluating its
effects on an annual basis.*** The adoption of some standard annual
evaluation procedures would allow the Comprehensive Centers to collect
much better and more systematic data on the effects of their services on
customers. Because the uniform reporting format used by the
Comprehensive Centers in their semi-annual progress reports to ED is the
primary source of data for reporting on the program's Government
Performance Results Act (GPRA) indicators, ED should continue to work
with the centers on refining and improving that reporting format. In
addition, ED could work with the centers on developing standard
instruments and data collection procedures for each of the 15 centers to
use in assessing the effect of their services on customers. ED may also
want to consider commissioning regular independent evaluations of the
centers' highest profile activities. The centers' evaluation of the
Reading Success Network (which was designed by a group of center staff
members from across the 15 centers and is being conducted by an outside
evaluator) may be a useful model for future efforts in this regard.

Although there are steps that the Comprehensive Centers and ED can both
take to strengthen existing services, the findings of this evaluation
indicate that, to have major effects on education change the
Comprehensive Centers would have to work on a more intensive and costly
scale. Their statutory mission is far broader than their resources can
support, and this suggests that ED and Congress should revisit the whole
subject of the program's purpose, aspirations, and scope. For the sake
of refining the goals and objectives of the Comprehensive Center program
so that the centers can organize their work around objectives that are
both important *and* feasible, we make the following recommendations:

***The Comprehensive Centers and ED should continue to work together to
clarify and refine the mission and goals of the program, especially with
regard to the content and purposes of services provided***. In
developing new forms of "comprehensive" assistance to respond to the
mandate in their authorizing legislation, and in the absence of clear
guidance from ED about what "comprehensive" might mean, each center
developed its own set of priorities and strategies for organizing its
portfolio of services to its region. Now may be a good time for
Comprehensive Center staff and ED managers to review those strategies
with the aim of clarifying the nature and goals of "comprehensive"
assistance. The Comprehensive Centers have focused a great deal of their
time and attention on providing support to the schoolwide planning
process, and they have also worked on improving needs assessment and
strategic planning among schools and districts. Are these still the best
strategies for promoting comprehensive school reform, as envisioned by
ESEA? Are there other ways through which the Comprehensive Centers might
support the implementation of strong comprehensive programs that promote
student learning? After three years of operation, the Comprehensive
Centers have made a good start at developing a portfolio of services
that could legitimately be characterized as comprehensive. Now may be a
good time to consider whether those strategies can be refined.

In addition, we have seen that the Comprehensive Centers provide a
considerable amount of support to customers specifically designed to
address the needs of special populations, and that customers find the
Comprehensive Centers a particularly useful source of assistance in this
area. Providing technical assistance on the needs of special
populations, including limited English proficient students, American
Indian students including Native Alaskan and Hawaiian children, and
migrant students, is clearly a central component of the centers'
mission. Comprehensive Center staff and ED managers might review the
current level of effort expended on addressing customers' needs related
to special populations, and consider whether the balance between these
services and more comprehensive services is appropriate.

***The Comprehensive Centers and ED should discuss goals for extending
the Centers' reach among district and school customers, as well as
reasonable goals for the effects expected as a result of this effort.***
The centers have succeeded in reaching a number of their potential
customers at the state, district, and school levels. As we have seen,
however, there is a clear trade-off between the intensity and
effectiveness of services and the extent of the centers' reach.
Comprehensive Center directors and ED managers who oversee the program
should discuss these trade-offs between depth and breadth of services.

This evaluation also found that, across the Comprehensive Center
network, the centers allocate their resources roughly equally among
states, districts, and schools. Working with and through states is an
important strategy for strengthening ESEA programs and building capacity
to use ESEA to serve target populations more effectively. As we have
seen, however, it is difficult to trace the effects of services to
states on districts and schools. Working directly with districts and
schools produces desirable effects but consumes a large proportion of
center resources. ED should also attempt to clarify priorities for
allocating technical assistance resources to various levels of the
education system, and modify expectations for effects accordingly.

*The Comprehensive Centers and ED should consider even stronger
targeting of high-priority customers as a way of addressing the problem
of reach.* The Comprehensive Centers have been responsive to their
charge to target their services on high-priority school and district
customers, including schoolwide program schools and high-poverty
districts and Bureau of Indian Affairs (BIA) schools. As a new program,
however, the centers needed to develop a customer base quickly, and they
served almost everyone who asked for help. Now that the centers have
succeeded in establishing themselves as a valuable source of assistance
in the regions they serve, it may be time to consider targeting services
even more closely on their highest priority customers. Targeting
services even more closely---perhaps, for example, by serving *only*
schoolwide program schools, schools in need of improvement, or very
high-poverty schools---would have the effect of limiting the program's
potential customer base, and hence some of the pressures related to
extending the reach of Comprehensive Center services.

**APPENDICES**

APPENDIX A

**Comprehensive Center Mission Statements**

+-----------------------------------------------------------------------+
| # Region I---New England Comprehensive Assistance Center              |
+-----------------------------------------------------------------------+
| To create sustainable structures that promote achievement of all      |
| students to high standards; to enhance the capacity of the region's   |
| six state education agencies as technical assistance providers to     |
| help children achieve high standards; and to enhance the capacity of  |
| school districts' administration, parents, and teachers to help       |
| children to achieve high standards.                                   |
+-----------------------------------------------------------------------+
| # Region II---New York Technical Assistance Center                    |
+-----------------------------------------------------------------------+
| To promote an understanding of, and provide services and assistance   |
| to, underserved populations and the education, government, and        |
| community agencies that serve them, which will result in ensuring     |
| equity and achieving excellence in the educational experiences of     |
| children and youth.                                                   |
+-----------------------------------------------------------------------+
| # Region III Comprehensive Center at the George Washington University |
+-----------------------------------------------------------------------+
| To provide high-quality technical assistance and service to state and |
| local education agencies and schools to facilitate the success of     |
| comprehensive education reform and school improvement initiatives.    |
| The Center collaborates with their clients and partners to focus on   |
| improving teaching and learning and ensuring equitable opportunities  |
| for all students to achieve challenging content and performance       |
| standards. The Center is committed to building strong working         |
| partnerships with other service providers and educators to bring      |
| coordinated assistance to educational systems throughout the region.  |
+-----------------------------------------------------------------------+
| # Region IV Com                                                       |
| prehensive Center at Appalachia Regional Educational Laboratory (AEL) |
+-----------------------------------------------------------------------+
| To help schools realize the goal of the 1994 Improving America\'s     |
| Schools Act---that all children achieve high standards, particularly  |
| those students with special needs.                                    |
+-----------------------------------------------------------------------+
| # Region V---Southeast Comprehensive Assistance Center                |
+-----------------------------------------------------------------------+
| To provide comprehensive training and technical assistance related to |
| the administration and implementation of programs under the Improving |
| America's School Act of 1994.                                         |
+-----------------------------------------------------------------------+
| # Region VI Comprehensive Assistance Center                           |
+-----------------------------------------------------------------------+
| To promote excellence in education through provision of quality       |
| services based on research and proven practices and that are targeted |
| to the needs of clients to improve instruction and learning for all   |
| children.                                                             |
+-----------------------------------------------------------------------+
| # Region VII Comprehensive Center                                     |
+-----------------------------------------------------------------------+
| The Center provides technical assistance to local education agencies, |
| schools, tribes, state education agencies, and community-based        |
| organizations that build capacity, improve teaching, promote school   |
| reform, assist with the development of state performance and content  |
| standards, and facilitate the federal education priorities in the     |
| Improving America\'s Schools Act to benefit all schools. The Center   |
| also ensures that the students most at risk of failure are supported  |
| through networking, coordination, and collaboration with other        |
| entities that share this mission.                                     |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
| **Region VIII---Support for Texas Academic Renewal (STAR) Center**    |
+-----------------------------------------------------------------------+
| To focus energy, expertise, and commitment to prepare our children    |
| for the 21st century by providing support and technical assistance    |
| services to the Texas Education Agency, Education Service Centers,    |
| and local school districts that implement state and local education   |
| reform efforts funded under the Improving America\'s Schools Act.     |
+-----------------------------------------------------------------------+
| **Region IX---Southwest Comprehensive Regional Assistance Center**    |
+-----------------------------------------------------------------------+
| To provide training and technical assistance to programs funded under |
| the Improving America\'s Schools Act.                                 |
+-----------------------------------------------------------------------+
| # North                                                               |
| west Regional Educational Laboratory's Comprehensive Center, Region X |
+-----------------------------------------------------------------------+
| To help ensure that all students are provided opportunities to meet   |
| challenging state content and performance challenges.                 |
+-----------------------------------------------------------------------+
| # Region XI Comprehensive Assistance Center                           |
+-----------------------------------------------------------------------+
| To support schools as they engage in comprehensive school change and  |
| efforts to continuously improve teaching, learning, and student       |
| achievement.                                                          |
+-----------------------------------------------------------------------+
| # Region XII---Southern California Comprehensive Assistance Center    |
+-----------------------------------------------------------------------+
| To improve teaching and learning for all children including those who |
| live in poverty; are migratory; are immigrants; have limited English  |
| proficiency; are neglected, delinquent, or homeless; or have          |
| disabilities.                                                         |
+-----------------------------------------------------------------------+
| # Region XIII---Alaska Comprehensive Regional Assistance Center       |
+-----------------------------------------------------------------------+
| To provide comprehensive training and technical assistance to         |
| teachers and school administrators, to improve the teaching and       |
| learning of all students in Alaska, and to help schools meet the      |
| goals of Alaska 2000.                                                 |
+-----------------------------------------------------------------------+
| # Region XIV Comprehensive Center                                     |
+-----------------------------------------------------------------------+
| To develop comprehensive schoolwide programs to meet the needs of all |
| children and to                                                       |
|                                                                       |
| deliver effective instructional strategies to local educational       |
| agencies and Bureau of Indian Affairs schools that have the highest   |
| percentages or numbers of children in poverty.                        |
+-----------------------------------------------------------------------+
| # Region XV---Pacific Comprehensive Regional Assistance Center        |
+-----------------------------------------------------------------------+
| To provide ongoing training and assistance to educators in the region |
| and to collaborate with technical assistance providers to deliver     |
| coordinated services.                                                 |
+-----------------------------------------------------------------------+

Appendix B

Comprehensive Centers, Grantee Organizations, and Partner Organizations

+-----+-----------------+-----------------+----------------------------+
| *   | **Name of**     | **Grantee**     | **Partner Organizations**  |
| *Re |                 |                 |                            |
| gio | **Comprehensive |                 |                            |
| n** | Center**        |                 |                            |
+=====+=================+=================+============================+
| I   | New England     | Education       | The Council of Chief State |
|     | Comprehensive   | Development     | School Officers            |
|     | Assistance      | Center, Inc.    |                            |
|     | Center          |                 | Learning Innovations       |
|     |                 |                 |                            |
|     |                 |                 | The Urban Institute        |
+-----+-----------------+-----------------+----------------------------+
| II  | New York        | Metropolitan    | ESCORT of the State        |
|     | Technical       | Center for      | University of New York at  |
|     | Assistance      | Urban Education | Oneonta                    |
|     | Center (NYTAC)  | School of       |                            |
|     |                 | Education, New  | Hunter College of the City |
|     |                 | York University | University of New York     |
|     |                 |                 |                            |
|     |                 |                 | ORBIS Associates of        |
|     |                 |                 | Washington, D.C.           |
|     |                 |                 |                            |
|     |                 |                 | RMC Research Corporation   |
+-----+-----------------+-----------------+----------------------------+
| III | Region III      | The George      | RMC Research Corporation   |
|     | Comprehensive   | Washington      |                            |
|     | Center at The   | University      | Research for Better        |
|     | George          |                 | Schools                    |
|     | Washington      |                 |                            |
|     | University      |                 | ESCORT of the State        |
|     |                 |                 | University of New York at  |
|     |                 |                 | Oneonta                    |
+-----+-----------------+-----------------+----------------------------+
| IV  | Region IV       | Appalachia      | ESCORT of the State        |
|     | Comprehensive   | Educational     | University of New York at  |
|     | Center at       | Laboratory,     | Oneonta                    |
|     | Appalachia      | Inc.            |                            |
|     | Educational     |                 | ORBIS Associates of        |
|     | Laboratory      |                 | Washington, D.C.           |
|     | (AEL)           |                 |                            |
|     |                 |                 | MaiTran Associates         |
|     |                 |                 |                            |
|     |                 |                 | SERVE Regional Education   |
|     |                 |                 | Laboratory                 |
+-----+-----------------+-----------------+----------------------------+
| V   | Southeast       | Southwest       | Northwestern State         |
|     | Comprehensive   | Educational     | University of Louisiana    |
|     | Assistance      | Development     |                            |
|     | Center (SECAC)  | Laboratory      | Texas A&M                  |
|     |                 |                 | University-Kingsville      |
|     |                 |                 |                            |
|     |                 |                 | American Indian Research   |
|     |                 |                 | and Development, Inc.      |
+-----+-----------------+-----------------+----------------------------+
| VI  | Comprehensive   | University of   | Eastern Michigan           |
|     | Center at The   | Wi              | University                 |
|     | University of   | sconsin-Madison |                            |
|     | Wi              |                 | United Tribes Technical    |
|     | sconsin-Madison |                 | College                    |
|     |                 |                 |                            |
|     |                 |                 | Hamline University         |
+-----+-----------------+-----------------+----------------------------+
| VII | Region VII      | University of   | PRC, Inc./Educational and  |
|     | Comprehensive   | Oklahoma        | Evaluation Services        |
|     | Center          |                 |                            |
|     |                 |                 | Texas A&M                  |
|     |                 |                 | University-Kingsville      |
|     |                 |                 |                            |
|     |                 |                 | Phi Delta Kappa            |
|     |                 |                 | International              |
|     |                 |                 |                            |
|     |                 |                 | Inter-Tribal Associates,   |
|     |                 |                 | Inc.                       |
+-----+-----------------+-----------------+----------------------------+
| V   | Star Center     | Intercultural   | Charles A. Dana Center at  |
| III | (Support for    | Development     | the University of Texas at |
|     | Texas Academic  | Research        | Austin                     |
|     | Renewal)        | Association     |                            |
|     |                 | (IDRA)          | RMC Research Corporation   |
+-----+-----------------+-----------------+----------------------------+
| IX  | Southwest       | New Mexico      | Midcontinent Regional      |
|     | Comprehensive   | Highlands       | Educational Laboratory     |
|     | Regional        | University      |                            |
|     | Assistance      |                 | WestEd                     |
|     | Center          |                 |                            |
|     |                 |                 | Indigenous Institute for   |
|     |                 |                 | Educational Excellence     |
+-----+-----------------+-----------------+----------------------------+
| X   | Northwest       | Northwest       | Affiliated Tribes of       |
|     | Regional        | Regional        | Northwest Indians          |
|     | Educational     | Educational     |                            |
|     | Laboratorys    | Laboratory      | Salish Kootenai College    |
|     | Comprehensive   |                 |                            |
|     | Center, Region  |                 |                            |
|     | X               |                 |                            |
+-----+-----------------+-----------------+----------------------------+

+-----+-----------------+-----------------+----------------------------+
| XI  | Region XI       | WestEd          | Chicago State              |
|     | Comprehensive   |                 | University/Education for   |
|     | Assistance      |                 | the Future                 |
|     | Center          |                 |                            |
|     |                 |                 | San Jose State University, |
|     |                 |                 | Division of Teacher        |
|     |                 |                 | Education, Special         |
|     |                 |                 | Education-Rehabilitative   |
|     |                 |                 | Services                   |
|     |                 |                 |                            |
|     |                 |                 | MaiTran Associates         |
|     |                 |                 |                            |
|     |                 |                 | Win Visions                |
+-----+-----------------+-----------------+----------------------------+
| XII | Southern        | Los Angeles     | Imperial County Office of  |
|     | California      | County Office   | Education                  |
|     | Comprehensive   | of Education    |                            |
|     | Assistance      |                 | Inyo County Office of      |
|     | Center          |                 | Education                  |
|     |                 |                 |                            |
|     |                 |                 | Los Angeles County Office  |
|     |                 |                 | of Education               |
|     |                 |                 |                            |
|     |                 |                 | Office of the Mono County  |
|     |                 |                 | Superintendent of Schools  |
|     |                 |                 |                            |
|     |                 |                 | Orange County Office of    |
|     |                 |                 | Education                  |
|     |                 |                 |                            |
|     |                 |                 | Riverside County Office of |
|     |                 |                 | Education                  |
|     |                 |                 |                            |
|     |                 |                 | San Diego County           |
|     |                 |                 | Department of Education    |
|     |                 |                 |                            |
|     |                 |                 | San Bernardino County      |
|     |                 |                 | Superintendent of Schools  |
+-----+-----------------+-----------------+----------------------------+
| X   | Alaska          | South East      | Alaska State Department of |
| III | Comprehensive   | Regional        | Education                  |
|     | Regional        | Resource Center |                            |
|     | Assistance      |                 | Northwest Regional         |
|     | Center (AKRAC)  |                 | Education Laboratory       |
+-----+-----------------+-----------------+----------------------------+
| XIV | Region XIV      | Educational     | Center for Applied         |
|     | Comprehensive   | Testing Service | Linguistics, Sunbelt       |
|     | Center          |                 | Office                     |
|     |                 |                 |                            |
|     |                 |                 | David Anchin Center at     |
|     |                 |                 | University of South        |
|     |                 |                 | Florida                    |
|     |                 |                 |                            |
|     |                 |                 | Developing Resources for   |
|     |                 |                 | Education in America, Inc. |
|     |                 |                 |                            |
|     |                 |                 | ESCORT of the State        |
|     |                 |                 | University of New York at  |
|     |                 |                 | Oneonta                    |
|     |                 |                 |                            |
|     |                 |                 | Litton/PRC, Inc.           |
+-----+-----------------+-----------------+----------------------------+
| XV  | Region XV       | Pacific         | University of              |
|     | Pacific         | Resources for   | Guam-Multicultural         |
|     | Comprehensive   | Education and   | Education Resource Center  |
|     | Assistance      | Learning        | (MERC)                     |
|     | Center          |                 |                            |
|     |                 |                 | WestEd                     |
|     |                 |                 |                            |
|     |                 |                 | RMC Research Corporation   |
|     |                 |                 | of Portland, Oregon        |
+-----+-----------------+-----------------+----------------------------+
| To  |                 | 15              | 53 (41 unduplicated count) |
| tal |                 |                 |                            |
+-----+-----------------+-----------------+----------------------------+

APPENDIX C

Survey Procedures

# Survey of Center Customers: State and Local Contacts

Survey Administration and Response Rate

In January 1999, the Centers submitted lists of all customers served
during the 1998 calendar year. Customers were defined as key contacts
who had worked closely with the Centers on the design and delivery of
technical assistance. Since the focus of analysis was the organization
and not the individual, we categorized key contacts according to the
type of organization they represented. The organization served as the
basis for sampling.

A random sample of key contacts representing states, districts, schools,
and other organizations was drawn using the proportional sample
technique. This technique ensured that the proportion of sampled
customers would represent the proportion of all customers served by each
Center.

We administered surveys between April and July 1999. We sent surveys to
1,086 key contacts; 762 respondents completed and returned the survey
for an overall response rate of 70 percent. To adjust for differential
response patterns by Center, we assigned weights to again reflect the
proportion of all customers served by each Center.

Follow-up Procedures

We mailed reminder postcards to all non-respondents three weeks after
the initial mailing to encourage them to complete and return the survey
as soon as possible. We mailed or faxed additional surveys to
respondents who had not received the survey or had misplaced it.
Additionally, approximately six weeks after the initial mailing we
contacted non-respondents by telephone and mailed additional surveys on
two other occasions. We also updated the center\'s list to replace
contacts who were no longer with the organization.

## Survey of Center Customers: Participants in Center Activities

**[Survey Administration and Response Rate]{.underline}**

During the late winter and early spring of 1999 the Comprehensive
Centers submitted registration or participant lists for a sample of 30
Center-sponsored activities. The activities included in the sample were
relatively long-term and intensive and represented a variety of major
Center initiatives. These lists served as the basis for sampling. We
administered surveys to all participants for activities with less than
50 participants. We administered surveys to a random sample of
participants for activities with more than 50 participants.

We administered surveys between April and July 1999, on a staggered
schedule based on the availability of participant lists. We tailored the
survey for each activity, identifying the name of the activity; the
dates the activity occurred; the Center and, where applicable, any
collaborating organizations sponsoring the activity. In all, we sent
surveys to 1,132 teachers and administrators; 795 participants completed
and returned the survey for an overall response rate of 70 percent.

Follow-up Procedures

We mailed reminder postcards to non-respondents approximately two weeks
after the initial mailing and conducted a second mailing four weeks
after the initial mailing. We mailed a final round of surveys to
non-responders in July.

## Survey of Potential Customers

Survey Administration and Response Rate

Using the *Common Core of Data -- 1995*, we calculated the median
percentage of students living below the poverty line for all districts,
14.1 percent. We selected all districts with higher proportion of poor
students than the median, yielding 6,295 districts. We then drew a
random sample using the proportional sampling technique for the
following for strata: (1) 0-2,500 students; (2) 2,501-10,000 students;
(3) 10,001-25,000 students; and (4) greater than 25,000 students. This
technique ensured that the proportion of sampled public school districts
within each of the strata would represent the proportion of districts
within the entire population of poor school districts. In addition, we
sent surveys to the 50 school districts serving the largest numbers of
poor students and to the 50 districts serving the largest concentration
of poor students.

We administered surveys between June and August 1999. In all, we sent
surveys to 1,122 federal program directors; 838 respondents completed
and returned the survey, yielding a response rate of 76 percent.
However, 13 surveys were returned without the identifying information
that would have allowed us to link the survey with a particular
district. Therefore, these surveys were not included in any of the
analyses. We weighted the data to adjust for oversampling large school
districts and differing response rates among the different strata.

Follow-up Procedures

We mailed reminder postcards to non-respondents approximately three
weeks after the initial mailing. Approximately four weeks after the
first mailing we contacted non-responding districts by telephone to ask
for a specific person to whom we could send additional correspondence.
The initial survey was directed to the *Director of Federal Programs.*
We sent a second mailing of surveys approximately five weeks after the
first mailing and a third mailing in late July.

**Exhibit C.1**

**Survey of Center Customers: State and Local Contacts from
Organizations Served in 1998**

# *Survey Sample*

  ----------------------------- ------------------------------- ----------------------------- ------------------------ ------------------------ ------------------------
  **[Type of                    **Number of                     **Number of Primary           **[Sample                **Number of Completed    **Number of Completed
  Organization]{.underline}**   [Organizations]{.underline}**   [Contacts]{.underline}^1^**   Size]{.underline}^2^**   Surveys - Unweighted     Surveys -- Weighted
                                                                                                                       [Sample]{.underline}**   [Sample]{.underline}**

  State Education Agencies      62                              392                           188                      154                      102

  Districts                     387                             592                           284                      211                      156

  Schools                       680                             845                           409                      234                      223

  Intermediate Agencies         124                             187                           90                       76                       47

  Other Organizations           211                             240                           115                      87                       53

  Total                         1464                            2256                          1086                     762^3^                   581^4^
  ----------------------------- ------------------------------- ----------------------------- ------------------------ ------------------------ ------------------------

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

^1^ Primary contacts are defined as key persons with whom the centers
have worked in designing and delivering services and products.

^2^ The sample is stratified by type of respondent and is designed to be
representative at the state, district, and school levels.

^3^ Includes those reporting that they did not receive services from a
center.

^4^ Includes only those reporting that they received services from a
center.

> Exhibit C.2

**Description of Activities Included in *Survey of Participants in
Center Activities***

  ---------------------------------------------------------------------------------------------------------------------
         **Region**   **Name of        **Description**      **Participant        **Number of      **Intensity**
                      Activity**                            Characteristics**    Participants**   
  ------ ------------ ---------------- -------------------- -------------------- ---------------- ---------------------
  1\.    I            Data Strategies  Training for         Open to all high-    46               Two three-day
                      Initiative       districts and        poverty districts;                    institutes; follow-up
                                       technical assistance seven currently                       varies by district.
                                       providers (SEA       participating.                        Last institute fall
                                       staff) to learn to                                         1998.
                                       interpret student                                          
                                       achievement data and                                       
                                       use it in decision                                         
                                       making.                                                    

  2\.    I            Schoolwide       Network of           Nine school-level    94               Annual two-day
                      Network (SWN):   elementary schools   teams of teachers                     schoolwide congress;
                      Core Member      planning or          and principals.                       optional follow-up
                      Schools          operating schoolwide                                       workshops; spring
                                       programs. Provides                                         follow-up meeting;
                                       ongoing professional                                       follow-up by a coach;
                                       development and                                            a list-serve; and
                                       support to school                                          PrincipalsAcademy.
                                       improvement teams.                                         Last Congress October
                                                                                                  1998.

  3\.    II           ACT: Aligning    Training for teams   Interdisciplinary    206              A two-day workshop
                      Curriculum with  of teachers from     teams of up to five                   held in May 1998.
                      Technology       schools and          teachers.                             Additional activities
                                       districts in such                                          with the district
                                       areas as digital                                           included planning for
                                       libraries, digital                                         a second conference
                                       learning, and local                                        in May 1999 and
                                       area networks.                                             widening the focus to
                                                                                                  include smaller
                                                                                                  schools.

  4\.    II           Parent Training  Introduce parents to Parents              75               A series of two-day
                      in               the district's                                             workshops held
                      Low-Performing   system for                                                 throughout the
                      Schools          identifying                                                district over the
                                       low-performing                                             course of a year;
                                       schools and propose                                        follow-up activities
                                       strategies for                                             are ongoing.
                                       increasing parental                                        
                                       involvement.                                               

  5\.    III          Retreat for a    Assist district      District staff       23               One-day workshop in
                      District's       staff in shifting                                          August 1998 and four
                      Compensatory     from compliance                                            days of planning and
                      Education Staff  monitoring to                                              follow-up.
                                       providing technical                                        
                                       assistance and                                             
                                       support to schools.                                        

  6\.    III          Pennsylvania Big Assisted SEA with    SEA and district     19               Varies according to
                      Cities           the planning and     staff                                 the needs of schools
                      Consortium       implementation of                                          and districts; most
                                       school improvement                                         ongoing work is done
                                       activities for                                             at the state level
                                       member schools and                                         (few sustained
                                       districts. Topics                                          activities with
                                       included assessment,                                       district reps).
                                       school improvement,                                        
                                       student diversity                                          
                                       and learning, and                                          
                                       improving                                                  
                                       mathematics                                                
                                       performance.                                               

  7\.    IV           North Carolina   A train-the-trainer  LEA and SEA          33               Activities vary by
                      ESL Clusters     model for three      administrators,                       cluster; On average,
                                       clusters of schools  principals,                           six days of training
                                       and districts to     teachers, ESL                         during the academic
                                       help mainstream      coordinators                          year and five days in
                                       teachers work with                                         the summer for
                                       ESL students. There                                        teachers and
                                       are four categories                                        administrators.
                                       of services:                                               
                                       training in                                                
                                       assessment;                                                
                                       regional/statewide                                         
                                       training; model                                            
                                       schools; and lab                                           
                                       schools.                                                   

  8\.    IV           Tennessee Local  Develop a process    Staff from five      34               Two one-day meetings
                      Consolidated     for local districts  LEAs, SEA staff,                      with SEA staff and
                      Planning         to complete          administrators,                       six days of training
                                       consolidated         teachers, and                         for school
                                       applications for     parents                               representatives
                                       Titles I, II, and                                          (February-June 1998).
                                       IV.                                                        

  9\.    V            Initiative       Assist schools in    Staff in eight       Exact number     Two face-to-face
                      Schools          conducting a needs   schools              unknown          contacts per month in
                                       assessment and                                             the beginning; then
                                       creating and                                               at least one visit
                                       implementing an                                            per month (more if
                                       improvement plan.                                          needed); ongoing
                                                                                                  telephone calls;
                                                                                                  follow-up based on
                                                                                                  needs of the school

  10\.   V            Regional         "Foundations of      School teams         148              Two-day institute in
                      Institute        Teaching and         consisting of                         summer 1998.
                                       Learning:            principal, math or                    
                                       Integration of       science teacher, and                  
                                       Reading, Math, and   reading or                            
                                       Technology."         technology teacher                    
                                       Training focused on                                        
                                       integrating math,                                          
                                       science, reading,                                          
                                       and technology in                                          
                                       the curriculum.                                            

  11\.   VI           Cognitively      A problem-solving    Teams of two         42               Five-day training in
                      Guided           math program for K-3 teachers and one                      summer 1998 and
                      Instruction      teachers based on    resource                              telephone/survey
                                       developmental stages specialist/trainer                    follow-up throughout
                                       and individualized                                         the year.
                                       assessment.                                                

  12\.   VI           Principal        A workshop for       Principals and some  30               2 days in summer
                      Leadership       school               district                              1998 with limited
                      Institute        administrators on    administrators                        follow-up in areas
                                       how to build a                                             with a concentration
                                       positive school                                            of participants and
                                       culture.                                                   some e-mail
                                                                                                  follow-up. Follow-up
                                                                                                  planned for summer
                                                                                                  2000 in all three
                                                                                                  institutes.

  13\.   VII          Learning         Preparing school     Staff from eight     77               Five weekend sessions
                      Facilitators    staff to develop     elementary and four                   in a three-month
                      Academy          consolidated plans   middle schools                        period (six hours per
                                       for federal grants                                         session) in spring
                                       by training teams in                                       1998 and a six-hour
                                       the change process,                                        follow-up session in
                                       team building,                                             fall 1998; on-site
                                       student learning                                           observations.
                                       styles, curriculum                                         
                                       alignment, and                                             
                                       instructional                                              
                                       strategies.                                                

  14\.   VII          Schoolwide       An initial three-day SEA federal program  18               Three-day meeting
                      Showcase         schoolwide showcase  directors from four                   (early 1998) with
                                       with SEA staff       states                                follow-up telephone
                                       demonstrated what                                          consultations for
                                       the Center could do                                        those interested;
                                       to help support                                            activities with three
                                       state and local                                            states are ongoing.
                                       implementation of                                          
                                       schoolwide programs.                                       
                                       This initial meeting                                       
                                       resulted in                                                
                                       follow-up assistance                                       
                                       to Kansas, Nebraska,                                       
                                       and Missouri.                                              

  15\.   VIII         Excellence and   Assisting campus     Campus improvement   62               Two-day institutes
                      Equity Through   improvement teams to teams from 32                         with additional
                      Technology       develop and          schools in 23                         assistance from the
                      Network          implement technology districts                             Center including a
                      Institutes       plans. Focused on                                          two-day follow-up
                      (EETNet)         conducting on-line                                         institute within a
                                       needs assessments                                          year. Initial
                                       and engaging in                                            institute in Fall
                                       long-range                                                 1997; follow-up in
                                       technology planning.                                       early 1998.

  16\.   VIII         Leadership       Topics include       Bilingual directors, 84               Three-day institute
                      Institute for    research on          principals, and                       in fall 1998.
                      Bilingual        bilingual education, Title I staff                         
                      Directors        leadership skills,                                         
                                       and the Title VII                                          
                                       evaluation.                                                
                                       Institute designed                                         
                                       to allow                                                   
                                       participants to                                            
                                       share successes and                                        
                                       work collaboratively                                       
                                       to overcome barriers                                       
                                       to success.                                                

  17\.   IX           Reading Success  Training coaches who Teachers and key     17               Participants receive
                      Network in One   assist teachers in   administrators from                   coaching, materials,
                      District         diagnosing           8 schools                             and attend workshops;
                                       students\' reading                                         began fall 1998.
                                       problems in K-3 and                                        
                                       in supplying                                               
                                       appropriate                                                
                                       interventions.                                             

  18\.   IX           Outstanding      Worked with a state  School support team  24               Ongoing activities as
                      Nevada Educators Title I school       members                               negotiated with NDE,
                      Cadre School     support team on                                            level of assistance
                      Support Team     assisting troubled                                         varies.
                                       schools. Designed a                                        
                                       plan for needs                                             
                                       assessment, Terra                                          
                                       Nova administration,                                       
                                       and training.                                              

  19\.   X            Partner Sites    Center works with    School-based teams   19               Summer institutes and
                                       three partner sites  from three partner                    on-site consultations
                                       in such areas as     sites                                 every other month;
                                       analyzing,                                                 intensity varies by
                                       disaggregating, and                                        site.
                                       using student                                              
                                       performance data in                                        
                                       decisionmaking.                                            

  20\.   X            Support for      Training and         Distinguished        53               Intensity varies by
                      State Level      technical assistance educators from each                   state with a common
                      Distinguished    to the regions      of five states in                     symposium held each
                      Educators        distinguished        the region                            year; a total of two
                                       educators.                                                 to three activities
                                                                                                  each year. The common
                                                                                                  symposium was held in
                                                                                                  March 1998.
  ---------------------------------------------------------------------------------------------------------------------

  ----------------------------------------------------------------------------------------------------------------
         **Region**   **Name of       **Description**   **Participant       **Number of      **Intensity**
                      Activity**                        Characteristics**   Participants**   
  ------ ------------ --------------- ----------------- ------------------- ---------------- ---------------------
  21\.   XI           Principals     Supplements       Principals and      34               Half-day sessions;
                      Institute       ongoing           administrators from                  number of activities
                                      assistance to     50-80 schools in                     varies by site;
                                      school teams by   two regions                          ongoing with
                                      providing                                              activities beginning
                                      intensive                                              as a pilot in 1997-98
                                      professional                                           and other schools
                                      development                                            added in 1998-99.
                                      support to                                             
                                      principals whose                                       
                                      schools are                                            
                                      beginning the                                          
                                      schoolwide                                             
                                      process.                                               

  22\.   XI           Proposition 227 Collaborative     SEA staff and       26               Varies by activity;
                      Collaborative   efforts with the  informal                             efforts are long-term
                      Initiative      SEA, regional     consortiums of                       and ongoing.
                                      networks, and     schools and                          
                                      schools and       districts                            
                                      districts to help                                      
                                      meet the                                               
                                      requirements of                                        
                                      Proposition 227                                        
                                      legislation.                                           
                                      Activities                                             
                                      include                                                
                                      involvement in                                         
                                      the state                                              
                                      taskforce,                                             
                                      hosting and                                            
                                      presenting at                                          
                                      statewide                                              
                                      networks,                                              
                                      assisting schools                                      
                                      and districts,                                         
                                      and                                                    
                                      field-initiated                                        
                                      research.                                              

  23\.   XII          Reading Success Network of Title  School coaches      383              Five-day training
                      Network         I schoolwide and                                       (two counties provide
                                      targeted                                               additional days of
                                      assistance                                             training);  day
                                      schools that have                                      follow-up; activities
                                      identified                                             are ongoing.
                                      reading as a                                           
                                      focus for school                                       
                                      reform. The                                            
                                      Center trains                                          
                                      coaches who                                            
                                      assist teachers                                        
                                      in diagnosing                                          
                                      reading problems                                       
                                      in K-3 students                                        
                                      and in supplying                                       
                                      appropriate                                            
                                      interventions.                                         
  ----------------------------------------------------------------------------------------------------------------

  --------------------------------------------------------------------------------------------------------------
         **Region**   **Name of     **Description**   **Participant       **Number of      **Intensity**
                      Activity**                      Characteristics**   Participants**   
  ------ ------------ ------------- ----------------- ------------------- ---------------- ---------------------
  24\.   XII          Regional      "Creating a       District trainers   48               Three-day training in
                      Institute     Standards-based                                        spring 1998.
                                    Assessment                                             
                                    Program: Making                                        
                                    the Connection                                         
                                    Between                                                
                                    Standards,                                             
                                    Assessment, and                                        
                                    Student                                                
                                    Learning."                                             
                                    Assisted nine                                          
                                    districts in                                           
                                    developing an                                          
                                    assessment                                             
                                    program that                                           
                                    meets state and                                        
                                    federal mandates;                                      
                                    provides an                                            
                                    accurate picture                                       
                                    of student                                             
                                    achievement; and                                       
                                    is used to make                                        
                                    informed                                               
                                    instructional                                          
                                    decisions.                                             

  25\.   XIII         Alaska Onward An Effective      Teachers,           154              Eight days of
                      to Excellence Schools           administrators,                      training for team
                      (AOTE)        development       parents, and                         leaders and four
                                    program in which  community members                    meetings per year for
                                    the community is                                       faculty; activities
                                    extensively                                            ongoing. Most recent
                                    involved in                                            activity in fall
                                    setting                                                1998.
                                    performance and                                        
                                    content                                                
                                    standards,                                             
                                    assessing                                              
                                    progress, and                                          
                                    nurturing the                                          
                                    school's mission.                                      

  26\.   XIII         Integrating   Assists teachers  Teachers and        82               Faculty received two
                      the Internet  in meeting the    students                             days of training;
                      into the      Alaska Technology                                      students one-day with
                      Curriculum    Content Standards                                      telephone and other
                                    by providing                                           distance follow-up.
                                    training on using                                      Last session fall
                                    the Internet as                                        1998.
                                    an educational                                         
                                    research tool and                                      
                                    providing a                                            
                                    guided practice                                        
                                    approach to                                            
                                    teaching                                               
                                    technology.                                            
  --------------------------------------------------------------------------------------------------------------

  -------------------------------------------------------------------------------------------------------------
         **Region**   **Name of    **Description**   **Participant       **Number of      **Intensity**
                      Activity**                     Characteristics**   Participants**   
  ------ ------------ ------------ ----------------- ------------------- ---------------- ---------------------
  27\.   XIV          Reading      Training coaches  K-3 teachers from   31               Three days of initial
                      Success      who assist        six schools in                       training in fall 1998
                      Network      teachers in       Hillsborough                         with follow-up at
                                   diagnosing        County, FL                           monthly RSN meetings.
                                   reading problems                                       
                                   in K-3 students                                        
                                   and in supplying                                       
                                   appropriate                                            
                                   interventions.                                         

  28\.   XIV          Model        Assists eight     Staff in eight      27               Three-year project
                      Schools      low-performing,   schools                              beginning in 1996.
                      Project      high-poverty                                           
                                   schools in                                             
                                   developing school                                      
                                   improvement                                            
                                   plans, and                                             
                                   evaluating                                             
                                   results.                                               

  29\.   XV           Pohnpei      Provides          State staff         14               A year-long activity
                      Five-Year    assistance to                                          with plans to provide
                      Plan for     Pohnpei in                                             technical assistance
                      Education    developing its                                         for implementation.
                                   first five-year                                        
                                   plan.                                                  

  30\.   XV           Expanding    Using a toolkit   Educators from all  63               Initial three-day
                      Classroom    on alternative    areas in the                         institute in May 1998
                      Assessment   assessment        regions                              with additional
                      Literacy     developed by the                                       institutes to be
                                   Regional                                               conducted yearly;
                                   Educational                                            list-serve follow-up.
                                   Laboratories to                                        
                                   train a statewide                                      
                                   cadre of                                               
                                   assessment                                             
                                   trainers. Topics                                       
                                   include embedding                                      
                                   standards-based                                        
                                   assessment into                                        
                                   classroom                                              
                                   instruction and                                        
                                   assessing                                              
                                   Adequate Yearly                                        
                                   Progress in                                            
                                   schools.                                               
  -------------------------------------------------------------------------------------------------------------

**Exhibit C.3**

**Survey of Potential Customers**

**Survey Sample**

##  {#section-35}

+------+----------+----------+---------------+-----------------------+
| **   | **Number | ##       | **Number of   | **Number of Completed |
| Dist | of       | #  {#sec | Completed     | Surveys** **Weighted  |
| rict | D        | tion-36} | Surveys       | [                     |
| Si   | istricts |          | [Unweighted   | Sample]{.underline}** |
| ze** | Above    | ###      | Sample]{      |                       |
|      | Median   | Number i | .underline}** |                       |
|      | [P       | n Sample |               |                       |
|      | overty]{ |          |               |                       |
|      | .underli |          |               |                       |
|      | ne}^1^** |          |               |                       |
+------+----------+----------+---------------+-----------------------+
| 0-   | 4634     | 638      | 451           | 598                   |
| 2500 |          |          |               |                       |
+------+----------+----------+---------------+-----------------------+
| 25   | 1263     | 250      | 198           | 163                   |
| 01-1 |          |          |               |                       |
| 0000 |          |          |               |                       |
+------+----------+----------+---------------+-----------------------+
| 100  | 263      | 114      | 83            | 34                    |
| 01-2 |          |          |               |                       |
| 5000 |          |          |               |                       |
+------+----------+----------+---------------+-----------------------+
| \>2  | 135      | 120      | 80            | 17                    |
| 5000 |          |          |               |                       |
+------+----------+----------+---------------+-----------------------+
| T    | 6295     | 1122     | 825           | 812^2^                |
| otal |          |          |               |                       |
+------+----------+----------+---------------+-----------------------+

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

^1^ Using data from the Common Core of Data---1995, the median poverty
level for all school districts is 14.1 percent.

^2^ Thirteen surveys are not included in the weighted sample because
they did not include information necessary for identifying the district
size.

**APPENDIX D**

**Survey Instruments**

+-----------------------------------------------------------------------+
| **EVALUATION OF THE COMPREHENSIVE CENTERS**                           |
|                                                                       |
| **SURVEY OF CENTER CUSTOMERS: STATE AND LOCAL CONTACTS**              |
+-----------------------------------------------------------------------+

The official name of the Comprehensive Center that serves your region is
the:

\[NAME OF CENTER\]

This Comprehensive Center is operated by

\[NAME OF PRIMARY GRANTEE\]

in partnership with

**\[NAME OF PARTNERS\]**

\[Center\] is directed by \[Name of Director\] and is located in
\[City\].

**Please answer the following questions based on your experiences with
the \[Center\], whether you worked primarily with \[Center\] or its
partner organization.**

. Please indicate the type of agency for which you work. (CIRCLE ONE)

a\. School 1

b\. School district central office 2

c\. Intermediate education agency or BOCES 3

d\. State education agency 4

e\. Other (Specify):
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
5

. Which of the following activities are [major]{.underline} parts of
your professional responsibilities? (CIRCLE ALL THAT APPLY)

a\. Administering an educational organization 1

b\. Administering a federally funded education program 1

c\. Providing training, professional development, or technical
assistance 1

d\. Other (Specify):
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
1

. Which of the following [federal]{.underline} education programs, if
any, do you help to administer? (CIRCLE ALL THAT APPLY)

> a\. Title I, Part A 1
>
> b\. Title I, Part B (Even Start) 1
>
> c\. Title I, Part C (Education of Migratory Children) 1
>
> d\. Title II, Eisenhower Professional Development Program 1
>
> e\. Title IV, Safe and Drug-Free Schools and Communities 1

f\. Title VI, Innovative Education Program Strategies 1

> g\. Title VII, Bilingual Education, Language Enhancement, and
>
> Language Acquisition Programs 1
>
> h\. Title IX, Indian, Native Hawaiian, and Alaska Native Education 1

i\. Goals 2000 1

j\. Other (SPECIFY) 1

> k\. I do not help to administer federal programs 1

. Since the beginning of 1998, have you communicated with Comprehensive
Center staff (i.e., by telephone, mail, e-mail, or through a survey or
other means) to request help or to discuss Center services to your
organization?

> a\. Yes 1
>
> b\. No 2
>
> c\. Not sure 3

. Since the beginning of 1998, have you or anyone in your organization
received training, technical assistance (i.e., by telephone, fax,
e-mail, through Center-sponsored conferences, workshops, etc.), or
written products from the Comprehensive Center?

> a\. Yes 1
>
> b\. No (SKIP TO END) 2
>
> c\. Not sure (SKIP TO END) 3

. How important were the following factors in prompting you or your
organization to begin using the products or services offered by the
Comprehensive Center?

  ------------------------------------------------------------------- ----------- ------------ ----------- -----------
                                                                      Very        Moderately   Somewhat    Not at all
                                                                      important   important    important   important

  a\. Center products or services met the needs of our organization   1           2            3           4

  b\. The Center provided products or services that were not          1           2            3           4
  available elsewhere                                                                                      

  c\. Center products and services were easily accessible             1           2            3           4

  d\. Center products and services were free of charge                1           2            3           4

  e\. Center services built on the technical assistance we had        1           2            3           4
  received in the past                                                                                     

  f\. The Center had a reputation for providing high-quality          1           2            3           4
  technical assistance                                                                                     

  g\. Other                                                           1           2            3           4
  (Specify)\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_                                        
  ------------------------------------------------------------------- ----------- ------------ ----------- -----------

. If you have been involved in designing or planning the services that
the Comprehensive Center provides to your organization, in what ways do
you usually contribute? (CIRCLE ALL THAT APPLY)

 I rarely contribute to the design or planning of services (SKIP TO
QUESTION 8)

> a\. Selecting and framing the content 1
>
> b\. Designing the format 1
>
> c\. Selecting the provider or presenter(s) 1
>
> d\. Coordinating the services with other assistance we receive 1
>
> e\. Identifying participants 1
>
> f\. Providing feedback to the provider or presenter(s) 1
>
> g\. Planning logistics 1
>
> h\. Other (Specify):
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> 1

. Since the beginning of 1998, how many times have you or your
organization received the following products or services from the
Comprehensive Center? How satisfied are you with the ACCESSIBILITY of
the Centers products and services (i.e., how easily can you obtain the
following products and services from the Center)?

  ---------------------------------------------- -------- ----- ----- ---- -------------- ------------ ----------- -----------
                                                 A. How                    B.                                      
                                                 many                      Satisfaction                            
                                                 times                     with your                               
                                                 since                     ACCESS to the                           
                                                 the end                   Centers                                
                                                 of last                   products or                             
                                                 school                    services?                               
                                                 year?                                                             

                                                 0        1-2   3-5   5+   Very satisfied Moderately   Somewhat    Not at all
                                                                                          satisfied    satisfied   satisfied

  a\. Written materials                          0        1     2     3    4              3            2           1

  b\. On-site consultations                      0        1     2     3    4              3            2           1

  c\. Consultations by e-mail or telephone       0        1     2     3    4              3            2           1

  d\. Professional development (e.g., workshops, 0        1     2     3    4              3            2           1
  institutes)                                                                                                      

  e\. Collaborating/networking with other        0        1     2     3    4              3            2           1
  programs or districts                                                                                            

  f\. Help convening special events; conferences 0        1     2     3    4              3            2           1

  g\. Center home page or other electronic       0        1     2     3    4              3            2           1
  products                                                                                                         

  h\. Other (Specify)                            0        1     2     3    4              3            2           1
  \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_                                                                     

  i\. Your satisfaction with the OVERALL                                   4              3            2           1
  accessibility of Center products and services?                                                                   
  ---------------------------------------------- -------- ----- ----- ---- -------------- ------------ ----------- -----------

. What kind of technology, if any, has the Comprehensive Center used to
provide your organization with technical assistance? (CIRCLE ALL THAT
APPLY)

>  The Center has not used any of the technologies listed below to
> provide our organization with technical assistance (SKIP TO QUESTION
> 11)

a\. E-mail 1

> b\. The Centers own Website 1
>
> c\. Some other Website 1
>
> d\. Teleconferencing 1
>
> e\. Videoconferencing 1
>
> f\. Audio or videotaped presentations 1
>
> g\. Other (Specify):
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> 1

. What effect, if any, did the Comprehensive Centers use of technology
have on the services your organization received? (CIRCLE ALL THAT APPLY)

a\. The volume of information provided was more than expected 1

b\. Information/training was delivered more quickly and efficiently 1

c\. The use of technology enhanced the ability of service recipients to
understand what was

presented (e.g., new concepts, materials, skills) 1

d\. The use of technology made the presentation/training more
interesting 1

e\. The use of technology had no effect on the quality or efficiency of
Center services 1

f\.
Other\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
1

. Since the beginning of 1998, has your organization received any
products or services from the Comprehensive Center that addressed the
following topics? (Check Yes only if the topic was a
[primary]{.underline} focus of the products or services you received).
How satisfied are you with the QUALITY of the products and services you
have received from the Comprehensive Center? (CIRCLE YES OR NO IN
EACH ROW IN COLUMN A; CIRCLE ONE RESPONSE IN COLUMN B [ONLY
IF]{.underline} YOU MARKED YES IN COLUMN A).

+--------------------------+---------+-------+-------+-------+-------+
|                          | A.      | B. If |       |       |       |
|                          | R       | you   |       |       |       |
|                          | eceived | rec   |       |       |       |
|                          | assi    | eived |       |       |       |
|                          | stance? | a     |       |       |       |
|                          |         | ssist |       |       |       |
|                          | YES /   | ance, |       |       |       |
|                          | NO      | sa    |       |       |       |
|                          |         | tisfa |       |       |       |
|                          |         | ction |       |       |       |
|                          |         | with  |       |       |       |
|                          |         | the   |       |       |       |
|                          |         | QU    |       |       |       |
|                          |         | ALITY |       |       |       |
|                          |         | of    |       |       |       |
|                          |         | C     |       |       |       |
|                          |         | enter |       |       |       |
|                          |         | pro   |       |       |       |
|                          |         | ducts |       |       |       |
|                          |         | and   |       |       |       |
|                          |         | serv  |       |       |       |
|                          |         | ices? |       |       |       |
+==========================+=========+=======+=======+=======+=======+
|                          |         | Very  | Moder | Som   | Not   |
|                          |         | sati  | ately | ewhat | at    |
|                          |         | sfied | sati  | sati  | all   |
|                          |         |       | sfied | sfied | sati  |
|                          |         |       |       |       | sfied |
+--------------------------+---------+-------+-------+-------+-------+
| a\. Implementing         | Y / N   | 4     | 3     | 2     | 1     |
| schoolwide programs      |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| b\. Implementing other   | Y / N   | 4     | 3     | 2     | 1     |
| ESEA programs (e.g.,     |         |       |       |       |       |
| Title I, Title VII)      |         |       |       |       |       |
|                          |         |       |       |       |       |
| > (Specify ESEA program) |         |       |       |       |       |
| >                        |         |       |       |       |       |
| > \_\_                   |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| > (Specify ESEA program) | Y / N   | 4     | 3     | 2     | 1     |
| >                        |         |       |       |       |       |
| > \_\_                   |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| c\. Consolidating or     | Y / N   | 4     | 3     | 2     | 1     |
| integrating federally    |         |       |       |       |       |
| funded education         |         |       |       |       |       |
| programs                 |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| d\. Addressing the needs | Y / N   | 4     | 3     | 2     | 1     |
| of special student       |         |       |       |       |       |
| populations (e.g., LEP,  |         |       |       |       |       |
| migratory, Indian)       |         |       |       |       |       |
|                          |         |       |       |       |       |
| > (Specify population)   |         |       |       |       |       |
| >                        |         |       |       |       |       |
| > \_\_                   |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| > (Specify population)   | Y / N   | 4     | 3     | 2     | 1     |
| >                        |         |       |       |       |       |
| > \_\_                   |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| e\. Improving curriculum | Y / N   | 4     | 3     | 2     | 1     |
| and instruction in:      |         |       |       |       |       |
|                          |         |       |       |       |       |
| > Reading/language arts  |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| > Mathematics            | Y / N   | 4     | 3     | 2     | 1     |
+--------------------------+---------+-------+-------+-------+-------+
| > Other core academic    | Y / N   | 4     | 3     | 2     | 1     |
| > areas                  |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| f\. Challenging          | Y / N   | 4     | 3     | 2     | 1     |
| standards;               |         |       |       |       |       |
| accountability           |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| g\. Student assessment   | Y / N   | 4     | 3     | 2     | 1     |
+--------------------------+---------+-------+-------+-------+-------+
| h\. Analyzing student    | Y / N   | 4     | 3     | 2     | 1     |
| achievement data and     |         |       |       |       |       |
| interpreting results     |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| i\. Improving the        | Y / N   | 4     | 3     | 2     | 1     |
| quality of bilingual     |         |       |       |       |       |
| education and ESL        |         |       |       |       |       |
| programs                 |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| j\. Creating safe and    | Y / N   | 4     | 3     | 2     | 1     |
| drug-free learning       |         |       |       |       |       |
| environments             |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| k\. Implementing         | Y / N   | 4     | 3     | 2     | 1     |
| educational applications |         |       |       |       |       |
| of technology            |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| l\. Expanding            | Y / N   | 4     | 3     | 2     | 1     |
| parent/family            |         |       |       |       |       |
| involvement              |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| m\. Program evaluation   | Y / N   | 4     | 3     | 2     | 1     |
+--------------------------+---------+-------+-------+-------+-------+
| n\. Adopting and         | Y / N   | 4     | 3     | 2     | 1     |
| implementing particular  |         |       |       |       |       |
| model programs or        |         |       |       |       |       |
| comprehensive school     |         |       |       |       |       |
| reforms                  |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| o\. Issues identified by | Y / N   | 4     | 3     | 2     | 1     |
| ED Integrated Review     |         |       |       |       |       |
| Teams                    |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| p\. Other (SPECIFY)      | Y / N   | 4     | 3     | 2     | 1     |
|                          |         |       |       |       |       |
| > \_\_                   |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
| \_\_\_\_\_\_\_\_\_\_\_\_ |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+
| q\. Your satisfaction    |         | 4     | 3     | 2     | 1     |
| with the OVERALL quality |         |       |       |       |       |
| of the Centers products |         |       |       |       |       |
| and services             |         |       |       |       |       |
+--------------------------+---------+-------+-------+-------+-------+

. Of the topics listed in question 11 (rows a-p), for which three does
your organization have the *greatest* need of technical assistance?
(WRITE UP TO THREE TOPICS ON THE LINES BELOW)

a\. \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

b\. \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

c\. \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

. When you are seeking assistance on each of the topics listed in
question 12, how often do you usually turn to each of the following
sources?

+-----------------------------------+------+--------+------+-----+-----+
|                                   | Al   | Som    | Se   | Ne  | N/A |
|                                   | ways | etimes | ldom | ver |     |
+-----------------------------------+------+--------+------+-----+-----+
| a\. Topic                         |      |        |      |     |     |
| (a)\_\_\_\_\                      |      |        |      |     |     |
| _\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ |      |        |      |     |     |
+-----------------------------------+------+--------+------+-----+-----+
| > U.S. Department of Education    | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > State education agency          | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > Intermediate education agency   | 4    | 3      | 2    | 1   | 0   |
| > (e.g, regional service center,  |      |        |      |     |     |
| > county office of education,     |      |        |      |     |     |
| > area educational agency)        |      |        |      |     |     |
+-----------------------------------+------+--------+------+-----+-----+
| > Local college/university        | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > Regional Educational Laboratory | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > Comprehensive Center            | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+

+-----------------------------------+------+--------+------+-----+-----+
|                                   | Al   | Som    | Se   | Ne  | N/A |
|                                   | ways | etimes | ldom | ver |     |
+-----------------------------------+------+--------+------+-----+-----+
| b\. Topic                         |      |        |      |     |     |
| (b)\_\_\_\_\                      |      |        |      |     |     |
| _\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ |      |        |      |     |     |
+-----------------------------------+------+--------+------+-----+-----+
| > U.S. Department of Education    | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > State education agency          | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > Intermediate education agency   | 4    | 3      | 2    | 1   | 0   |
| > (e.g, regional service center,  |      |        |      |     |     |
| > county office of education,     |      |        |      |     |     |
| > area educational agency)        |      |        |      |     |     |
+-----------------------------------+------+--------+------+-----+-----+
| > Local college/university        | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > Regional Educational Laboratory | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > Comprehensive Center            | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+

+-----------------------------------+------+--------+------+-----+-----+
|                                   | Al   | Som    | Se   | Ne  | N/A |
|                                   | ways | etimes | ldom | ver |     |
+-----------------------------------+------+--------+------+-----+-----+
| c\. Topic                         |      |        |      |     |     |
| (c)\_\_\_\_\                      |      |        |      |     |     |
| _\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ |      |        |      |     |     |
+-----------------------------------+------+--------+------+-----+-----+
| > U.S. Department of Education    | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > State education agency          | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > Intermediate education agency   | 4    | 3      | 2    | 1   | 0   |
| > (e.g, regional service center,  |      |        |      |     |     |
| > county office of education,     |      |        |      |     |     |
| > area educational agency)        |      |        |      |     |     |
+-----------------------------------+------+--------+------+-----+-----+
| > Local college/university        | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > Regional Educational Laboratory | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+
| > Comprehensive Center            | 4    | 3      | 2    | 1   | 0   |
+-----------------------------------+------+--------+------+-----+-----+

. Compared with similar products and services available from other
sources, how would you rate the quality of the products and services you
received from the Comprehensive Center on each of the following
dimensions?

> **a. Implementing schoolwide programs**

 We did not receive any assistance on this topic (SKIP TO QUESTION 14b)

+----------------------------------+---------+-----+-----+------+-----+
|                                  | Ex      | G   | F   | Poor | Not |
|                                  | cellent | ood | air |      | a   |
|                                  |         |     |     |      | ble |
|                                  |         |     |     |      | to  |
|                                  |         |     |     |      | ju  |
|                                  |         |     |     |      | dge |
+----------------------------------+---------+-----+-----+------+-----+
| > Accuracy of content            | 4       | 3   | 2   | 1    | 0   |
+----------------------------------+---------+-----+-----+------+-----+
| > Ability to respond in depth to | 4       | 3   | 2   | 1    | 0   |
| > all of your questions and      |         |     |     |      |     |
| > interests                      |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Extent to which the assistance | 4       | 3   | 2   | 1    | 0   |
| > reflected sound research or    |         |     |     |      |     |
| > the most current thinking in   |         |     |     |      |     |
| > the field                      |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Extent to which the Center     | 4       | 3   | 2   | 1    | 0   |
| > provided knowledge and         |         |     |     |      |     |
| > expertise not available within |         |     |     |      |     |
| > your organization              |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Presentation and format        | 4       | 3   | 2   | 1    | 0   |
+----------------------------------+---------+-----+-----+------+-----+

> **b. Improving curriculum and instruction in reading/language arts,
> mathematics, or other core academic areas**
>
>  We did not receive any products or services on this topic (SKIP TO
> QUESTION 14c)

+----------------------------------+---------+-----+-----+------+-----+
|                                  | Ex      | G   | F   | Poor | Not |
|                                  | cellent | ood | air |      | a   |
|                                  |         |     |     |      | ble |
|                                  |         |     |     |      | to  |
|                                  |         |     |     |      | ju  |
|                                  |         |     |     |      | dge |
+----------------------------------+---------+-----+-----+------+-----+
| > Accuracy of content            | 4       | 3   | 2   | 1    | 0   |
+----------------------------------+---------+-----+-----+------+-----+
| > Ability to respond in depth to | 4       | 3   | 2   | 1    | 0   |
| > all of your questions and      |         |     |     |      |     |
| > interests                      |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Extent to which the assistance | 4       | 3   | 2   | 1    | 0   |
| > reflected sound research or    |         |     |     |      |     |
| > the most current thinking in   |         |     |     |      |     |
| > the field                      |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Extent to which the Center     | 4       | 3   | 2   | 1    | 0   |
| > provided knowledge and         |         |     |     |      |     |
| > expertise not available within |         |     |     |      |     |
| > your organization              |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Presentation and format        | 4       | 3   | 2   | 1    | 0   |
+----------------------------------+---------+-----+-----+------+-----+

> **c. Products and services on [all other]{.underline}** topic areas

+----------------------------------+---------+-----+-----+------+-----+
|                                  | Ex      | G   | F   | Poor | Not |
|                                  | cellent | ood | air |      | a   |
|                                  |         |     |     |      | ble |
|                                  |         |     |     |      | to  |
|                                  |         |     |     |      | ju  |
|                                  |         |     |     |      | dge |
+----------------------------------+---------+-----+-----+------+-----+
| > Accuracy of content            | 4       | 3   | 2   | 1    | 0   |
+----------------------------------+---------+-----+-----+------+-----+
| > Ability to respond in depth to | 4       | 3   | 2   | 1    | 0   |
| > all of your questions and      |         |     |     |      |     |
| > interests                      |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Extent to which the assistance | 4       | 3   | 2   | 1    | 0   |
| > reflected sound research or    |         |     |     |      |     |
| > the most current thinking in   |         |     |     |      |     |
| > the field                      |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Extent to which the Center     | 4       | 3   | 2   | 1    | 0   |
| > provided knowledge and         |         |     |     |      |     |
| > expertise not available within |         |     |     |      |     |
| > your organization              |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Presentation and format        | 4       | 3   | 2   | 1    | 0   |
+----------------------------------+---------+-----+-----+------+-----+

. Compared with similar products and services available from other
sources, how would you rate the overall USEFULNESS of the products and
services you received from the Comprehensive Center? (CIRCLE ONE IN EACH
ROW)

+----------------------------------+---------+-----+-----+------+-----+
|                                  | Ex      | G   | F   | Poor | Not |
|                                  | cellent | ood | air |      | a   |
|                                  |         |     |     |      | ble |
|                                  |         |     |     |      | to  |
|                                  |         |     |     |      | ju  |
|                                  |         |     |     |      | dge |
+----------------------------------+---------+-----+-----+------+-----+
| > Addressed our organizations   | 4       | 3   | 2   | 1    | 0   |
| > needs and interests            |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Responsive to specific local   | 4       | 3   | 2   | 1    | 0   |
| > conditions our organization    |         |     |     |      |     |
| > faces                          |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Timely                         | 4       | 3   | 2   | 1    | 0   |
+----------------------------------+---------+-----+-----+------+-----+
| > Useful for guiding improvement | 4       | 3   | 2   | 1    | 0   |
| > efforts                        |         |     |     |      |     |
+----------------------------------+---------+-----+-----+------+-----+
| > Useful and accessible format   | 4       | 3   | 2   | 1    | 0   |
+----------------------------------+---------+-----+-----+------+-----+

. Has your organization contributed any of its own resources (e.g.,
release time, materials, building space, etc.) in order to expand or
otherwise enhance Comprehensive Center services?

> a\. Yes 1
>
> b\. No 2

. Has the Comprehensive Center ever referred you to other sources for
assistance, or has your contact with the Center ever introduced you to
other sources of technical assistance?  (CIRCLE ONE)

> a\. Yes 1
>
> b\. No (SKIP TO QUESTION 19) 2

. If so, to whom? (CIRCLE ALL THAT APPLY)

> a\. Regional Educational Laboratories 1
>
> b\. Eisenhower Mathematics and Science Education Regional Consortia 1
>
> c\. Regional Technology in Education Consortia 1
>
> d\. Local colleges/universities 1
>
> e\. Intermediate education agencies (e.g., county offices of
> education, BOCES,

regional service centers) 1

f\. State education agency 1

g\. Other Comprehensive Center(s) 1

> h\. Professional associations 1
>
> i\. Other (Specify):
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> 1

. Overall, what effect, if any, has Comprehensive Center assistance had
on *your own* work? (CIRCLE ALL THAT APPLY)

 Not applicable; I did not benefit directly from the services

a\. It confirmed what I was already doing 1

b\. I have gained some new information and additional perspectives 1

c\. I have incorporated into my job something I learned from the Center
1

d\. I have provided technical assistance to others in support of federal
programs 1

e\. It has had little or no effect on how I do my job 1

g\. Other (Specify):
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
1

. What, if anything, do *you* do differently in your job as a result of
your experience with the Comprehensive Center?

 Not applicable; I did not benefit directly from the services

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

. What effect, if any, has Comprehensive Center assistance had on the
way *your organization* does its work? (CIRCLE ALL THAT APPLY)

a\. Helped generate awareness of new information 1

> b\. Increased teachers knowledge and skills 1
>
> c\. Changed teachers classroom practices 1
>
> d\. Improved teaching and learning for a specific population of
> students 1
>
> e\. Improved teaching and learning for all students 1

f\. Increased student achievement 1

g\. Supported an ongoing program, policy, or practice of the
organization 1

> h\. Enhanced the organizations capacity to provide technical
> assistance in support of

ESEA programs 1

i\. Helped the organization initiate or establish a new program, policy,
or practice 1

j\. Helped the organization take the next step in a reform effort 1

k\. Put the organization in touch with other organizations engaged in
similar tasks 1

l\. Other (Specify):
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
1

. What, if anything, is different about the way *your organization*
operates as a result of its contact with Comprehensive Center?

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

**Note: The following items should be completed by school or district
staff only. All other respondents may skip to the end of the survey.**

. What percentage of students in your *school* or *district* are
eligible to receive free- or reduced-price lunches? (CIRCLE ONE)

 Does not apply; I am not based in a school or district

a\. 0-25% 1

b\. 26-50% 2

c\. 51-75% 3

d\. 76-100% 4

. What percentage of students in your *school* or *district* belong each
of the following groups? (CIRCLE ONE IN EACH ROW)

 Does not apply; I am not based in a school or district

+----------------------------------+-----+-------+------+------+-----+
|                                  | \   | 2-10% | 11   | 26   | O   |
|                                  | <1% |       | -25% | -50% | ver |
|                                  |     |       |      |      | 50% |
+----------------------------------+-----+-------+------+------+-----+
| > a\. Children with limited      | 5   | 4     | 3    | 2    | 1   |
| > English proficiency            |     |       |      |      |     |
+----------------------------------+-----+-------+------+------+-----+
| > b\. Indian, Native Hawaiian,   | 5   | 4     | 3    | 2    | 1   |
| > or Alaska Native children      |     |       |      |      |     |
+----------------------------------+-----+-------+------+------+-----+
| > c\. Migratory children         | 5   | 4     | 3    | 2    | 1   |
+----------------------------------+-----+-------+------+------+-----+
| > d\. Immigrant children         | 5   | 4     | 3    | 2    | 1   |
+----------------------------------+-----+-------+------+------+-----+
| > e\. Neglected or delinquent    | 5   | 4     | 3    | 2    | 1   |
| > children                       |     |       |      |      |     |
+----------------------------------+-----+-------+------+------+-----+
| > f\. Homeless children          | 5   | 4     | 3    | 2    | 1   |
+----------------------------------+-----+-------+------+------+-----+
| > g\. Children with disabilities | 5   | 4     | 3    | 2    | 1   |
+----------------------------------+-----+-------+------+------+-----+
| > h\. Other (SPECIFY)            | 5   | 4     | 3    | 2    | 1   |
| > \_\_\_\_                       |     |       |      |      |     |
| \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ |     |       |      |      |     |
+----------------------------------+-----+-------+------+------+-----+

. Does your *school* operate a Title I schoolwide program?

 Does not apply; I am not based in a school

a\. Yes 1

b\. No 2

Thank you!

+-----------------------------------------------------------------------+
| **EVALUATION OF THE COMPREHENSIVE CENTERS**                           |
|                                                                       |
| **SURVEY OF CENTER CUSTOMERS: PARTICIPANTS IN CENTER ACTIVITIES**     |
+-----------------------------------------------------------------------+

This survey is designed to gather data regarding the activity named

**\[NAME OF ACTIVITY\]**

which is an ongoing activity that began in **\[DATE\].**

This activity was provided by your regions Comprehensive Center, which
is officially named

**\[NAME OF CENTER\]**

and operated by the **\[NAME OF PRIMARY GRANTEE\].**

. Please indicate the type of agency for which you work: (CIRCLE ONE)

a\. School 1

> b\. School district central office 2
>
> c\. Intermediate education agency or BOCES 3

d\. State education agency 4

e\. Institutes of higher education 5

f\. Other (Specify):
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ 6

. Which of the following activities are [major]{.underline} parts of
your professional responsibilities?

(CIRCLE ALL THAT APPLY)

> a\. Teaching elementary or secondary students 1
>
> b\. Administering an educational organization 1
>
> c\. Administering a federally funded education program 1
>
> d\. Providing training, professional development, or technical
> assistance 1
>
> e\. Other (Specify):
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ 1
>
> f\. I am a parent or a representative of a community organization 1

. Did you participate in \[Center initiative\] provided by the
Comprehensive Center?

a\. Yes 1

b\. No (STOP HERE) 2

c\. Unsure/Dont remember (STOP HERE) 3

. What was the extent of your involvement in the activity? (CIRCLE ONE)

a\. Attended everything that was offered 1

> b\. Attended more than half of what was offered 2

c\. Attended less than half of what was offered 3

> d\. Attended very little of the activity 4

. Were you required to participate in this activity?

> a\. Yes 1
>
> b\. No 2
>
> If you answered Yes, by whom?
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

. Did you receive a stipend, professional development credit, or some
other payment for participating in this activity?

> a\. Yes 1
>
> b\. No 2

. Did you participate in this activity as a member of a team or group
representing your agency, organization, or program?

a\. Yes 1

> b\. No 2

. Were you involved at all in designing the format or the content of
this activity? In what ways?

(CIRCLE ALL THAT APPLY)

> a\. Selecting and framing the content 1
>
> b\. Designing the format 1
>
> c\. Selecting the provider or presenter(s) 1
>
> d\. Coordinating this activity with other assistance we receive 1
>
> e\. Identifying participants 1
>
> f\. Providing feedback to the provider or presenter(s) 1
>
> g\. Planning logistics 1
>
> h\. Other (Specify):
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> 1
>
> i\. I did not contribute at all to the design of this activity 1

. What were your purposes in participating in this activity? (CIRCLE ALL
THAT APPLY)

a\. Keeping current professionally 1

> b\. Obtaining ideas or contacts that would strengthen my work 1

c\. Obtaining ideas or contacts that would strengthen my organizations

existing work 1

> d\. Obtaining ideas or contacts that would help my organization carry
> out a major,

planned improvement effort 1

> e\. Being a resource to others 1
>
> f\. Other (Specify):
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> 1

. How fully did the activity serve your purposes? (CIRCLE ONE)

> a\. It served my purposes completely 1
>
> b\. It was a good start 2
>
> c\. It was a start, but it did not provide sufficient information or
> guidance to enable

me to follow up 3

> d\. It did not serve my purposes 4

. To what extent, if at all, did staff from the Comprehensive Center
provide you with the following kinds of assistance directly related to
this activity? (CIRCLE ONE RESPONSE IN EACH ROW)

+-------------------------------------------+-----------+-----+------+
|                                           | More than | O   | N    |
|                                           |           | nce | ever |
|                                           | once      |     |      |
+-------------------------------------------+-----------+-----+------+
| a\. Followed up by phone                  | 3         | 2   | 1    |
+-------------------------------------------+-----------+-----+------+
| b\. Followed up with workshops, training, | 3         | 2   | 1    |
| meetings beyond what was originally       |           |     |      |
| scheduled                                 |           |     |      |
+-------------------------------------------+-----------+-----+------+
| c\. Sent additional materials             | 3         | 2   | 1    |
+-------------------------------------------+-----------+-----+------+
| d\. Visited my workplace                  | 3         | 2   | 1    |
+-------------------------------------------+-----------+-----+------+
| e\. Other (Specify):                      | 3         | 2   | 1    |
| \_\_\_\_\_\_\_\_\_\_\_\_\_\               |           |     |      |
| _\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ |           |     |      |
+-------------------------------------------+-----------+-----+------+

. Have you received additional assistance on a topic related to this
activity from another source (e.g., school district office, state
education agency)?

a\. Yes 1

b\. No (SKIP TO Q 14) 2

c\. Not sure (SKIP TO Q 14) 3

. What assistance did you receive? (CIRCLE ALL THAT APPLY)

> a\. Additional workshops, training, meetings 1
>
> b\. Additional materials 1
>
> c\. Financial or other material support for implementation 1
>
> d\. Individual consultation 1

e\. Networking opportunities 1

> f\. Other (Specify):
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ 1

. Compared with other similar activities available from other sources,
how would you rate the quality of \[ACTIVITY\] in each of the following
areas? (CIRCLE ONE RESPONSE IN EACH ROW)

  ---------------------------------- ----------- ------ ------ ------- -------
                                     Excellent   Good   Fair   Poor    Not
                                                                       able to
                                                                       judge

  a\. Accuracy of content            4           3      2      1       0

  b\. Ability to respond in depth to 4           3      2      1       0
  all of your questions and                                            
  interests                                                            

  c\. Extent to which the assistance 4           3      2      1       0
  reflected sound research or the                                      
  most current thinking in the field                                   

  d\. Presentation and format        4           3      2      1       0

  e\. Accessibility to participants  4           3      2      1       0

  f\. Opportunities for networking   4           3      2      1       0
  among participants                                                   
  ---------------------------------- ----------- ------ ------ ------- -------

. Compared with similar products and services available from other
sources, how would you rate the overall USEFULNESS of the products and
services you received from the Comprehensive Center on each of the
following dimensions? (CIRCLE ONE RESPONSE IN EACH ROW)

  ---------------------------------- ----------- ------ ------ ------- -------
                                     Excellent   Good   Fair   Poor    Not
                                                                       able to
                                                                       judge

  a\. Fit with your needs and        4           3      2      1       0
  interests                                                            

  b\. Responsiveness to the specific 4           3      2      1       0
  local conditions you face                                            

  c\. Timeliness in serving your     4           3      2      1       0
  purposes                                                             

  d\. Extent to which you were able  4           3      2      1       0
  to use the assistance to guide                                       
  improvement efforts                                                  
  ---------------------------------- ----------- ------ ------ ------- -------

. Overall, what effect, if any, has your participation in \[ACTIVITY\]
had on *your* *work*?

(CIRCLE ALL THAT APPLY)

> a\. It confirmed what I was already doing 1
>
> b\. I have some new information about a program or an instructional
> practice 1
>
> c\. I have incorporated into my job something I learned from this
> activity 1
>
> d\. I have informally shared the ideas with a colleague(s) 1
>
> e\. I have communicated the ideas to colleagues or others through
> training,

technical assistance, or similar activities 1

> f\. I have begun communicating with other people engaged in work
> similar to mine 1
>
> g\. Other (Specify):
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> 1

h\. It has had little or no effect on how I do my job 1

. What, if anything, do you do differently in your job as a result of
*your* *experience* with the Comprehensive Center?

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

. What effect, if any, has this \[ACTIVITY\] had on the way *your
organization* does its work?

(CIRCLE ALL THAT APPLY)

a\. Helped generate awareness of new information 1

> b\. Supported an ongoing program, policy, or practice of the
> organization 1
>
> c\. Helped the organization initiate or establish a new program,
> policy, or practice 1
>
> d\. Helped the organization take the next step(s) in a reform effort 1
>
> e\. Put the organization in touch with other organizations engaged in
> similar tasks 1

f\. Other (Specify):
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
1

g\. It has had little or no effect on my organizations work 1

. What, if anything, is different about the way *your organization*
operates as a result of its experience with the Comprehensive Center?

> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

. Which of the following kinds of additional assistance, if any, you
have sought from the Comprehensive Center as a result of your
participation in this activity? (CIRCLE ALL THAT APPLY)

a\. Additional materials on the topic covered in the activity 1

> b\. Materials on other topics 1
>
> c\. Advice on other resources that might be available to you and your
> colleagues 1
>
> d\. Answers to questions about the topics covered in the activity 1
>
> e\. Answers to questions about other topics 1

f\. Additional training and/or technical assistance in testing or

implementing ideas presented in the activity 1

> g\. Additional information available on the Centers Web site 1
>
> h\. Other (Specify):
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> 1
>
> i\. I have not sought any additional assistance 1

**Thank you!**

+-----------------------------------------------------------------------+
| **EVALUATION OF THE COMPREHENSIVE REGIONAL ASSISTANCE CENTERS**       |
|                                                                       |
| SURVEY OF POTENTIAL CUSTOMERS                                         |
+-----------------------------------------------------------------------+

The official name of the Comprehensive Center that serves your region is
the:

\[NAME OF CENTER\]

This Center is operated by

\[NAME OF PRIMARY GRANTEE\]

in partnership with

**\[NAME OF PARTNERS\]**

\[Center\] is directed by \[NAME OF CENTER DIRECTOR\] and located in
\[CITY\].

**[Directions]{.underline}**: Please answer the following questions
based on your experiences with the \[Center\]. As you complete this
survey, please consult with other federal programs administrators or
district staff who may have had contact with the Center.

. What is your title?
\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_

. Have you heard of the \[Center\] or services the Center offers?

> a\. Yes 1
>
> b\. No (SKIP TO Q6) 2
>
> c\. Not sure (SKIP TO Q6) 3

. Since the end of last school year, have you or anyone in your district
communicated with \[Center\] staff (i.e., by telephone, mail, e-mail, or
through a survey or other means) to request help or to discuss Center
services in your district?

> a\. Yes (SPECIFY)
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ 1
>
> b\. No 2
>
> c\. Not sure 3

. Since the end of last school year, have you or anyone in your district
received technical assistance (i.e., by telephone, fax, e-mail, through
Center-sponsored conferences, workshops, etc.) from the Center?

> a\. Yes (SKIP TO Q6) 1
>
> b\. No 2
>
> c\. Not sure 3

. If your district has not made use of the products or services offered
by the Comprehensive Center, what are the reasons? (CIRCLE ALL THAT
APPLY)

> a\. We have not received any information about Center products

or services 1

> b\. We receive adequate assistance from other sources

(SPECIFY) \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ 1

> c\. We dont know if the Center can address our particular needs

and interests 1

> d\. We dont know whom to contact at the Center 1
>
> e\. We have not been able to develop productive working relationships

with Center staff 1

> f\. We have not had the time to learn about Center products and
> services 1
>
> g\. Center products and services do not address our needs 1
>
> h\. The Center does not have a reputation for providing high-quality

products and services 1

> i\. We have requested assistance, but the Center was unable to provide
> it 1
>
> j\. Other
> (SPECIFY)\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
>
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> 1
>
> k\. Dont know 1

. For which THREE of the topics listed below would your district be most
likely to seek technical assistance or other help from sources outside
the district? (CHECK UP TO THREE)

> a\. Implementing schoolwide programs \_\_\_\_\_\_
>
> b\. Implementing other ESEA programs

(e.g., Title I, Title VII)

> (SPECIFY)\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_\_
>
> (SPECIFY)\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_\_

c\. Integrating federally funded education programs \_\_\_\_\_\_

> d\. Addressing the needs of special student
>
> populations (e.g., LEP, migratory, Indian)
>
> (SPECIFY)\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_\_

(SPECIFY)\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_\_

> e\. Improving curriculum and instruction in:
>
> Reading/language arts \_\_\_\_\_\_
>
> Mathematics \_\_\_\_\_\_
>
> Other core academic areas \_\_\_\_\_\_
>
> f\. Challenging standards; accountability \_\_\_\_\_\_
>
> g\. Student assessment \_\_\_\_\_\_
>
> h\. Analyzing student achievement data and

interpreting results \_\_\_\_\_\_

> i\. Improving the quality of bilingual

education and ESL programs \_\_\_\_\_\_

> j\. Creating safe and drug-free learning

environments \_\_\_\_\_\_

> k\. Implementing educational applications of

technology \_\_\_\_\_\_

> l\. Expanding parent/family involvement \_\_\_\_\_\_
>
> m\. Program evaluation \_\_\_\_\_\_
>
> n\. Adopting and implementing particular model

programs or comprehensive school reforms \_\_\_\_\_\_

> o\. Issues identified by ED Integrated Review Teams \_\_\_\_\_\_
>
> p\. Other (SPECIFY)\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_
> \_\_\_\_\_\_

(SPECIFY)\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ \_\_\_\_\_\_

. For each of the topics you checked in question 6, how often do you
usually turn to each of the following sources for information and
assistance?

+-------------------------------------+--------+--------+------+------+
|                                     | Always | Som    | Se   | N    |
|                                     |        | etimes | ldom | ever |
+-------------------------------------+--------+--------+------+------+
| a\. Topic                           |        |        |      |      |
| (a)\_\_\_\                          |        |        |      |      |
| _\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ |        |        |      |      |
+-------------------------------------+--------+--------+------+------+
| > U.S. Department of Education      | 4      | 3      | 2    | 1    |
+-------------------------------------+--------+--------+------+------+
| > State education agency            | 4      | 3      | 2    | 1    |
+-------------------------------------+--------+--------+------+------+
| > Intermediate education agency     | 4      | 3      | 2    | 1    |
| > (e.g, regional service center,    |        |        |      |      |
| > county office of education, area  |        |        |      |      |
| > educational agency)               |        |        |      |      |
+-------------------------------------+--------+--------+------+------+
| > Local college/university          | 4      | 3      | 2    | 1    |
+-------------------------------------+--------+--------+------+------+
| > Regional Educational Laboratory   | 4      | 3      | 2    | 1    |
+-------------------------------------+--------+--------+------+------+
| > Comprehensive Center              | 4      | 3      | 2    | 1    |
+-------------------------------------+--------+--------+------+------+

+-------------------------------------+--------+--------+------+-----+
|                                     | Always | Som    | Se   | Ne  |
|                                     |        | etimes | ldom | ver |
+-------------------------------------+--------+--------+------+-----+
| b\. Topic                           |        |        |      |     |
| (b)\_\_\_\                          |        |        |      |     |
| _\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ |        |        |      |     |
+-------------------------------------+--------+--------+------+-----+
| > U.S. Department of Education      | 4      | 3      | 2    | 1   |
+-------------------------------------+--------+--------+------+-----+
| > State education agency            | 4      | 3      | 2    | 1   |
+-------------------------------------+--------+--------+------+-----+
| > Intermediate education agency     | 4      | 3      | 2    | 1   |
| > (e.g, regional service center,    |        |        |      |     |
| > county office of education, area  |        |        |      |     |
| > educational agency)               |        |        |      |     |
+-------------------------------------+--------+--------+------+-----+
| > Local college/university          | 4      | 3      | 2    | 1   |
+-------------------------------------+--------+--------+------+-----+
| > Regional Educational Laboratory   | 4      | 3      | 2    | 1   |
+-------------------------------------+--------+--------+------+-----+
| > Comprehensive Center              | 4      | 3      | 2    | 1   |
+-------------------------------------+--------+--------+------+-----+

+-------------------------------------+--------+---------+------+-----+
|                                     | Always | So      | Se   | Ne  |
|                                     |        | metimes | ldom | ver |
+-------------------------------------+--------+---------+------+-----+
| c\. Topic                           |        |         |      |     |
| (c)\_\_\_\                          |        |         |      |     |
| _\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ |        |         |      |     |
+-------------------------------------+--------+---------+------+-----+
| > U.S. Department of Education      | 4      | 3       | 2    | 1   |
+-------------------------------------+--------+---------+------+-----+
| > State education agency            | 4      | 3       | 2    | 1   |
+-------------------------------------+--------+---------+------+-----+
| > Intermediate education agency     | 4      | 3       | 2    | 1   |
| > (e.g, regional service center,    |        |         |      |     |
| > county office of education, area  |        |         |      |     |
| > educational agency)               |        |         |      |     |
+-------------------------------------+--------+---------+------+-----+
| > Local college/university          | 4      | 3       | 2    | 1   |
+-------------------------------------+--------+---------+------+-----+
| > Regional Educational Laboratory   | 4      | 3       | 2    | 1   |
+-------------------------------------+--------+---------+------+-----+
| > Comprehensive Center              | 4      | 3       | 2    | 1   |
+-------------------------------------+--------+---------+------+-----+

. What percentage of students in your district are eligible to receive
free- or reduced-price lunches? (CIRCLE ONE)

a\. 0-25% 1

b\. 26-50% 2

c\. 51-75% 3

d\. 76-100% 4

. What percentage of students in your district belong each of the
following groups? (CIRCLE ONE IN EACH ROW)

+----------------------------------+-----+-------+------+------+-----+
|                                  | \   | 2-10% | 11   | 26   | O   |
|                                  | <1% |       | -25% | -50% | ver |
|                                  |     |       |      |      | 50% |
+----------------------------------+-----+-------+------+------+-----+
| > a\. Children with limited      | 5   | 4     | 3    | 2    | 1   |
| > English proficiency            |     |       |      |      |     |
+----------------------------------+-----+-------+------+------+-----+
| > b\. Indian, Native Hawaiian,   | 5   | 4     | 3    | 2    | 1   |
| > or Alaska Native children      |     |       |      |      |     |
+----------------------------------+-----+-------+------+------+-----+
| > c\. Migratory children         | 5   | 4     | 3    | 2    | 1   |
+----------------------------------+-----+-------+------+------+-----+
| > d\. Immigrant children         | 5   | 4     | 3    | 2    | 1   |
+----------------------------------+-----+-------+------+------+-----+
| > e\. Neglected or delinquent    | 5   | 4     | 3    | 2    | 1   |
| > children                       |     |       |      |      |     |
+----------------------------------+-----+-------+------+------+-----+
| > f\. Homeless children          | 5   | 4     | 3    | 2    | 1   |
+----------------------------------+-----+-------+------+------+-----+
| > g\. Children with disabilities | 5   | 4     | 3    | 2    | 1   |
+----------------------------------+-----+-------+------+------+-----+
| > h\. Other (SPECIFY)            | 5   | 4     | 3    | 2    | 1   |
| > \_\_\_\_\_\_\_\_\_\_\_\_       |     |       |      |      |     |
+----------------------------------+-----+-------+------+------+-----+

. From which of the following [federal]{.underline} education programs,
if any, does your district receive funding?

 We do not receive any federal funding

> a\. Title I, Part A 1

b\. Title I, Part B (Even Start) 1

> c\. Title I, Part C (Education of Migratory Children) 1
>
> d\. Title II, Eisenhower Professional Development Program 1
>
> e\. Title IV, Safe and Drug-Free Schools and Communities 1

f\. Title VI, Innovative Education Program Strategies 1

> g\. Title VII, Bilingual Education, Language Enhancement, and
>
> Language Acquisition Programs 1
>
> h\. Title IX, Indian, Native Hawaiian, and Alaska Native Education 1

i\. Goals 2000 1

**Thank you!**

APPENDIX E

**Case Studies of Center Services to States**

+-----+-------------+-----------+-------------------------------------+
| *   | **State**   | **SEA     | **Activities**                      |
| *Re |             | Offices** |                                     |
| gio |             |           |                                     |
| n** |             |           |                                     |
+=====+=============+===========+=====================================+
| I   | # Ma        | Regional  | Staff from the Massachusetts SEA    |
|     | ssachusetts | service   | help to facilitate the New England  |
|     |             | p         | Comprehensive Assistance Center\'s  |
|     |             | roviders; | Schoolwide Network and its Data     |
|     |             | Early     | Strategies Initiative.              |
|     |             | Literacy  | Massachusetts is a focus state for  |
|     |             | Office    | the Centers Reading Success        |
|     |             |           | Network activities, and SEA staff   |
|     |             |           | have participated in the            |
|     |             |           | network-wide training activities.   |
|     |             |           | The Center works closely with the   |
|     |             |           | states regional service providers  |
|     |             |           | in planning and implementing        |
|     |             |           | technical assistance activities in  |
|     |             |           | the state. The Center also provides |
|     |             |           | technical assistance related to     |
|     |             |           | Title I and Title IV to the         |
|     |             |           | Massachusetts SEA.                  |
+-----+-------------+-----------+-------------------------------------+
| II  | **New       | Office of | The New York Technical Assistance   |
|     | York**      | the       | Center (NYTAC) assisted the New     |
|     |             | Deputy    | York State Education Department in  |
|     |             | Comm      | the development of its consolidated |
|     |             | issioner; | plan, provided training to its      |
|     |             | Office of | Title I School Support Teams, and   |
|     |             | C         | works extensively with the state's  |
|     |             | urriculum | regional technical assistance       |
|     |             | and       | network, including the BOCES, the   |
|     |             | Ins       | Effective Schools Consortia         |
|     |             | truction; | Network, and the state's regional   |
|     |             |           | bilingual education and migrant     |
|     |             | Office of | education centers. In addition,     |
|     |             | Federal   | NYTAC created a CSRD think tank to  |
|     |             | Programs  | support implementation of that      |
|     |             |           | program. In addition to advising    |
|     |             |           | the state, the think tank helped    |
|     |             |           | design the application process,     |
|     |             |           | review applications, and hold       |
|     |             |           | workshops and model showcases.      |
+-----+-------------+-----------+-------------------------------------+
| III | *           | Office of | The Center worked with the SEA to   |
|     | *Maryland** | Comp      | develop a network of exemplary      |
|     |             | rehensive | schools that can share best         |
|     |             | Planning  | practices with low-performing Title |
|     |             | and       | I schools. The state partnered 10   |
|     |             | School    | schools from the Maryland Blue      |
|     |             | Support   | Ribbon Schools Consortium with 10   |
|     |             |           | Title I schools. The Center also    |
|     |             |           | assisted the SEA with its CSRD      |
|     |             |           | application and helped it develop a |
|     |             |           | rubric for evaluating local         |
|     |             |           | applications. It is working with    |
|     |             |           | the Maryland Even Start Think Tank  |
|     |             |           | to develop a manual to assist       |
|     |             |           | programs become more                |
|     |             |           | self-sufficient at the end of their |
|     |             |           | funding cycle.                      |
+-----+-------------+-----------+-------------------------------------+
| V   | **Mi        | Regional  | Through its state liaison, SECAC    |
|     | ssissippi** | Service   | has built an onoing relationship    |
|     |             | Centers;  | with the Mississippi Department of  |
|     |             | Office of | Education and the five Regional     |
|     |             | I         | Service Centers (RSCs) it created   |
|     |             | nnovative | as a technical assistance arm in    |
|     |             | Services; | 1997. SECAC has provided training   |
|     |             | Bureau of | to the RSCs in needs assessment and |
|     |             | Inst      | service planning, facilitated       |
|     |             | ructional | communication among them, and       |
|     |             | De        | helped them become operational in   |
|     |             | velopment | their beginning stages. The Center  |
|     |             |           | has contributed to the development  |
|     |             |           | of the state reading initiative and |
|     |             |           | assisted in training pilot          |
|     |             |           | districts, and it has partnered to  |
|     |             |           | provide a series of events to       |
|     |             |           | assist Mississippi schools choose,  |
|     |             |           | implement, and evaluate CSR models. |
+-----+-------------+-----------+-------------------------------------+
| V   | **Texas**   | Education | The STAR Center provides ongoing    |
| III |             | Service   | training for staff from the state's |
|     |             | Centers;  | Education Service Centers (ESCs),   |
|     |             | T         | the technical assistance arm of the |
|     |             | echnology | Texas Educational Agency. The       |
|     |             | Office;   | Center also works with and through  |
|     |             | Office    | the ESCs to assist schools and      |
|     |             | for the   | districts in aligning schoolwide    |
|     |             | Education | plans with campus improvement       |
|     |             | of        | plans, integrating technology, and  |
|     |             | Special   | coordinating funding. The Center    |
|     |             | Po        | collaborates directly with the ESCs |
|     |             | pulations | and also models for the ESCs,     |
|     |             |           | building their capacity to provide  |
|     |             |           | high-quality technical assistance.  |
+-----+-------------+-----------+-------------------------------------+
| IX  | **New       | Division  | The Southwest Comprehensive         |
|     | Mexico**    | of        | Regional Assistance Center has      |
|     |             | Learning  | helped the New Mexico SEA develop   |
|     |             | Services; | long-range goals and has worked     |
|     |             | M         | with other technical assistance     |
|     |             | anagement | providers in the state to determine |
|     |             | Support   | how to support those goals. The     |
|     |             | and       | Center's own technical assistance   |
|     |             | Int       | services are related to the goals,  |
|     |             | ervention | including work with low-performing  |
|     |             |           | schools, training for School        |
|     |             |           | Support Teams, and assistance with  |
|     |             |           | standards implementation.           |
+-----+-------------+-----------+-------------------------------------+
| XV  | **Hawaii**  | Office of | The Pacific Comprehensive           |
|     |             | Accou     | Assistance Center works with the    |
|     |             | ntability | Hawaii SEA to provide assistance    |
|     |             | and       | regarding Title I issues. Pacific   |
|     |             | School    | Center staff maintain an ongoing    |
|     |             | In        | relationship with the state's Title |
|     |             | struction | I linkers and have trained and      |
|     |             | Support,  | assisted them in assessment,        |
|     |             | School    | workshop planning and presentation, |
|     |             | Renewal   | supporting schools, and school      |
|     |             | Group,    | monitoring. It has also assisted    |
|     |             | Special   | the state in training school        |
|     |             | Programs  | support teams and developing the    |
|     |             | M         | state's Title I schoolwide planning |
|     |             | anagement | system. The Pacific Center uses its |
|     |             | Section   | expertise in Pacific cultures and   |
|     |             |           | languages to provide consultation   |
|     |             |           | and create useful products for      |
|     |             |           | Hawaii. Through teleconferences and |
|     |             |           | training, the Pacific Center has    |
|     |             |           | helped build the knowledge and      |
|     |             |           | skills of ESL and special education |
|     |             |           | practitioners. Lastly, it has       |
|     |             |           | assisted the state in developing an |
|     |             |           | evaluation system for the Even      |
|     |             |           | Start program                       |
+-----+-------------+-----------+-------------------------------------+

**Case Studies of Center Services to Local Sites**

+-----+---------+------+-----------------------------+---------------+
| *   | *       | **F  | **Activities**              | **Demographic |
| *Re | *Name** | ocus |                             | Char          |
| gio |         | of   |                             | acteristics** |
| n** |         | Case |                             |               |
|     |         | Stu  |                             |               |
|     |         | dy** |                             |               |
+=====+=========+======+=============================+===============+
|     |         |      |                             |               |
+-----+---------+------+-----------------------------+---------------+
| IV  | Tyson   | T    | Ongoing, intensive          | Rural; high   |
|     | and     | hree | assistance to three         | poverty; high |
|     | Gardner | d    | clusters of approximately   | c             |
|     | C       | istr | seven districts each, which | oncentrations |
|     | ounties | icts | are experiencing rapid      | of ESL and    |
|     |         | and  | growth in the number of LEP | migrant       |
|     |         | six  | students. Assistance        | populations   |
|     |         | sch  | includes district-level     |               |
|     |         | ools | help in planning services   |               |
|     |         |      | for LEP students and        |               |
|     |         |      | professional development    |               |
|     |         |      | for ESL and classroom       |               |
|     |         |      | teachers. The focus of the  |               |
|     |         |      | professional development    |               |
|     |         |      | has been on assessing the   |               |
|     |         |      | educational needs of LEP    |               |
|     |         |      | students, using effective   |               |
|     |         |      | pedagogical strategies with |               |
|     |         |      | them, and involving LEP     |               |
|     |         |      | parents in school           |               |
|     |         |      | activities and affairs.     |               |
|     |         |      | Week-long "lab" schools     |               |
|     |         |      | during the summer and       |               |
|     |         |      | shorter training sessions   |               |
|     |         |      | during the school year are  |               |
|     |         |      | used.                       |               |
+-----+---------+------+-----------------------------+---------------+
| VI  | Cogn    | Two  | *Cognitively Guided         | Industrial    |
|     | itively | T    | Instruction* is a           | and           |
|     | Guided  | itle | problem-solving math        | agricultural  |
|     | Inst    | I    | program for students in     | communities;  |
|     | ruction | scho | K-3. Teachers observe the   | high poverty; |
|     |         | ols; | cognitive process to tailor | c             |
|     | (CGI)   | CGI  | instruction to the child\'s | oncentrations |
|     |         | In   | ability. The program is     | of ESL        |
|     |         | stit | based on developmental      | students in   |
|     |         | utes | stages and individualized   | some          |
|     |         | atte | assessments and is grounded | districts and |
|     |         | nded | in research. The Center     | schools;      |
|     |         | by   | holds five-day initial and  | American      |
|     |         | sc   | advanced institutes during  | Indian        |
|     |         | hool | the summer, which are       | populations   |
|     |         | t    | attended by teams from      | in some       |
|     |         | eams | schools. Program            | districts and |
|     |         | from | participants receive        | schools       |
|     |         | six  | surveys and telephone call  |               |
|     |         | d    | follow-ups.                 |               |
|     |         | istr |                             |               |
|     |         | icts |                             |               |
|     |         | (    |                             |               |
|     |         | both |                             |               |
|     |         | wi   |                             |               |
|     |         | thin |                             |               |
|     |         | and  |                             |               |
|     |         | out  |                             |               |
|     |         | side |                             |               |
|     |         | the  |                             |               |
|     |         | Cent |                             |               |
|     |         | er's |                             |               |
|     |         | reg  |                             |               |
|     |         | ion) |                             |               |
+-----+---------+------+-----------------------------+---------------+
| VII | Ri      | D    | *Riverside City Schools'    | Large,        |
|     | verside | istr | Learning Facilitator's      | Midwestern,   |
|     | Public  | ict; | Academy---*The Region VII   | urban         |
|     | Schools | four | Center, working with the    | district;     |
|     |         | sch  | schools, designed the       | high poverty  |
|     |         | ools | academy to increase the use |               |
|     |         |      | of consolidated planning    |               |
|     |         |      | for all federal programs,   |               |
|     |         |      | including Gifted and        |               |
|     |         |      | Talented, Bilingual, and    |               |
|     |         |      | ESEA programs.              |               |
|     |         |      | Participants---grouped in   |               |
|     |         |      | six-person,                 |               |
|     |         |      | cross-disciplinary          |               |
|     |         |      | teams---met for five        |               |
|     |         |      | weekend sessions over three |               |
|     |         |      | months. Session topics      |               |
|     |         |      | included: the change        |               |
|     |         |      | process, team building,     |               |
|     |         |      | student learning styles,    |               |
|     |         |      | curriculum alignment, and   |               |
|     |         |      | instructional strategies.   |               |
|     |         |      | The district is now         |               |
|     |         |      | preparing to join the       |               |
|     |         |      | Reading Success Network.    |               |
+-----+---------+------+-----------------------------+---------------+
| X   | Mount   | D    | One of the Northwest        | Rural; high   |
|     | ainview | istr | Center's three intensive    | poverty; 98   |
|     | School  | ict; | partnership sites. The      | percent       |
|     | D       | One  | Center provides technical   | American      |
|     | istrict | sc   | assistance to               | Indian        |
|     |         | hool | administrators, staff, and  |               |
|     |         |      | parents. Activities include |               |
|     |         |      | work on a data-based needs  |               |
|     |         |      | assessment/profile system,  |               |
|     |         |      | schoolwide program          |               |
|     |         |      | planning, and community     |               |
|     |         |      | involvement.                |               |
+-----+---------+------+-----------------------------+---------------+
| XI  | K       | D    | The Region XI Comprehensive | Mid-size      |
|     | ingston | istr | Assistance Center provides  | urban         |
|     | Unified | ict; | the Kingston Unified School | district;     |
|     | School  | s    | District continuing         | high poverty; |
|     | D       | even | technical assistance in the | ESL           |
|     | istrict | sch  | implementation of           | populations   |
|     |         | ools | schoolwide programs. The    |               |
|     |         |      | Center designed and         |               |
|     |         |      | conducted the districts    |               |
|     |         |      | planning for the transition |               |
|     |         |      | to schoolwides and          |               |
|     |         |      | continues to offer services |               |
|     |         |      | to eight school teams and   |               |
|     |         |      | principals as they          |               |
|     |         |      | implement their plans. The  |               |
|     |         |      | collaboration involves      |               |
|     |         |      | teachers and other district |               |
|     |         |      | personnel in 8-12 days of   |               |
|     |         |      | on-site assistance annually |               |
|     |         |      | and through various         |               |
|     |         |      | Center-sponsored            |               |
|     |         |      | activities.                 |               |
+-----+---------+------+-----------------------------+---------------+
| XII | Reading | Five | The RSN is made up of Title | Suburban;     |
|     | Success | sch  | I schoolwide and targeted   | high poverty  |
|     | Network | ools | assistance schools that     |               |
|     | (RSN)   | in   | have identified reading as  |               |
|     | in      | four | a focus for school reform.  |               |
|     | Orange  | d    | The Center trains coaches   |               |
|     | and San | istr | and others who assist       |               |
|     | Ber     | icts | teachers in diagnosing      |               |
|     | nardino | in   | reading problems in K-3     |               |
|     | Co      | two  | students and in choosing    |               |
|     | unties, | coun | appropriate interventions.  |               |
|     | CA      | ties | At present, RSN includes    |               |
|     |         |      | 211 schools and 329 coaches |               |
|     |         |      | as active participants in   |               |
|     |         |      | the network, which is being |               |
|     |         |      | implemented in all eight    |               |
|     |         |      | counties in the Center's    |               |
|     |         |      | region.                     |               |
+-----+---------+------+-----------------------------+---------------+
| X   | S       | Two  | An example of AKRAC\'s      | Rural; high   |
| III | nowbank | sch  | innovative Alaska Onward to | poverty; low  |
|     | School  | ools | Excellence program, where   | achievement;  |
|     | D       |      | members of the community    | high          |
|     | istrict |      | come together to set the    | concentration |
|     |         |      | school system\'s goals and  | of ESL        |
|     |         |      | establish benchmarks and    | populations   |
|     |         |      | standards within and beyond |               |
|     |         |      | the state's standards.      |               |
|     |         |      | Because of high turnover in |               |
|     |         |      | school staff, non-transient |               |
|     |         |      | staff develop and maintain  |               |
|     |         |      | school programs. The        |               |
|     |         |      | district's approach to      |               |
|     |         |      | school reform relies on     |               |
|     |         |      | building a strong community |               |
|     |         |      | investment in defining      |               |
|     |         |      | success and evaluating      |               |
|     |         |      | whether or not it is        |               |
|     |         |      | achieved.                   |               |
+-----+---------+------+-----------------------------+---------------+
|     |         |      |                             |               |
+-----+---------+------+-----------------------------+---------------+
|     |         |      |                             |               |
+-----+---------+------+-----------------------------+---------------+
|     |         |      |                             |               |
+-----+---------+------+-----------------------------+---------------+
|     |         |      |                             |               |
+-----+---------+------+-----------------------------+---------------+
|     |         |      |                             |               |
+-----+---------+------+-----------------------------+---------------+
|     |         |      |                             |               |
+-----+---------+------+-----------------------------+---------------+
|     |         |      |                             |               |
+-----+---------+------+-----------------------------+---------------+
| XIV | R       | One  | Richmond participates in    | Rural; high   |
|     | ichmond | Sc   | two initiatives of the      | poverty; high |
|     | Ele     | hool | Region XIV Comprehensive    | concentration |
|     | mentary |      | Center: the Model           | of ESL        |
|     | School  |      | Schools/Continuous Progress | populations;  |
|     |         |      | Program and the Reading     | low           |
|     |         |      | Success Network. The Center | achievement;  |
|     |         |      | has included the school in  | high mobility |
|     |         |      | a series of activities and  |               |
|     |         |      | events designed to promote  |               |
|     |         |      | an effective and            |               |
|     |         |      | comprehensive whole school  |               |
|     |         |      | program. Richmond also      |               |
|     |         |      | seeks assistance from the   |               |
|     |         |      | Center to help it target    |               |
|     |         |      | services to migrant         |               |
|     |         |      | students, involve parents,  |               |
|     |         |      | train ESL/bilingual         |               |
|     |         |      | teachers, and organize      |               |
|     |         |      | America Reads tutoring      |               |
|     |         |      | services.                   |               |
+-----+---------+------+-----------------------------+---------------+

APPENDIX F

**Semi-Annual Progress Reports From the Comprehensive Centers:
Aggregated Data Tables**

**January -- December, 1999**

As part of its evaluation of the Comprehensive Regional Assistance
Centers (Comprehensive Centers), the U.S. Department of Education (ED)
has asked Policy Studies Associates (PSA) to aggregate and report on
data elements submitted by the Comprehensive Centers in their
semi-annual progress reports. This report contains aggregated data
tables for the year 1999, beginning January 1 and ending December 31. It
follows a similar report, completed in March 1999, that aggregated data
elements in Center progress reports for the 1998 fiscal year (October 1,
1997 to September 30,1998). Findings from an analysis of these
aggregated semi-annual reports are presented below. Unless otherwise
noted, findings are based on data for the entire year 1999.[^4]

Beginning in January 1999, ED made changes to Comprehensive Center
reporting. Since then, Comprehensive Centers are required to submit
semi-annual reports instead of the quarterly reports they submitted in
prior years. Whereas Center reporting was previously based on a fiscal
year reporting schedule, it is now based on the calendar year. The
progress reports are due to ED following the end of each semi-annual
period---January 1 through June 30 and July 1 through December 31. For
this reason, analysis and aggregation of 1999 Center data must also
follow the calendar year, rather than the fiscal year as in 1998.

**[Number, Type, and Duration of Technical Assistance
Events]{.underline} (Tables 1 and 2)**

**The Comprehensive Centers reported high levels of activity in 1999,
with slight increases over activity reported for FY 1998.**

-   In 1999, the Comprehensive Centers reported 3,737 professional
    development events for the year, an average of 21 events per month
    per Center (up from 18 events in FY 98).

-   The Centers convened or facilitated 2,679 events, an average of 15
    events per month per Center (up from 14 events in FY 98).

```{=html}
<!-- -->
```
-   The Centers reported 1,181 consultation events that lasted one day
    or more, an average of 7 events per month per Center (up from 4
    events in FY 98).

**Centers continue to report providing a large proportion of
high-intensity activities, and do so to a greater degree than in FY
1998.**

-   In 1999, Centers reported that 58 percent of the professional
    development events they sponsored lasted for more than one day (up
    from 44 percent in FY 1998).

-   Centers reported that 17 percent of their consultation events lasted
    for more than one day (up from 12 percent in FY 1998). Although we
    have no data about where these consultations took place, it is
    likely that many took place at the customers' sites.

-   From July to December 1999, Centers reported that 64 percent of
    their professional development events and 54 percent of their
    consultation events took place as part of a series. This suggests
    that the majority of Center services are provided in the context of
    an established relationship with customers in which there was
    ongoing or follow-up assistance.

> **Note:** Information about continuity (that is, whether professional
> development and consultation events took place as stand-alone events
> or as part of a series) is only available for the July to December
> 1999 reporting period.

**Compared with FY 1998 reports, information dissemination figured less
prominently in the Centers' portfolio of services, though it still
accounts for a substantial bulk of all services.**

-   In 1999, information dissemination made up 27 percent of all
    reported technical assistance events (down from 57 percent in FY
    1998).

> **Note:** The decrease in the percent of information dissemination
> events can be accounted for in part by a change in reporting
> instructions from ED. Beginning in July 1999, Centers were asked to
> report mass mailings as a single event, rather than counting each
> mailed document as a separate event.

-   For the period July to December 1999, Centers reported that 81
    percent of all their information dissemination services were
    customized responses to individual requests for information; the
    remainder were mass mailings and broadcasts.

**[Topics Addressed by Technical Assistance]{.underline} (Table 3)**

**Most often, Comprehensive Center services addressed topics that were
"integrated" or crosscutting, rather than topics that focused on
specific ESEA program categories.** For instance, the most prominent
topics of Center services were reading, Title I schoolwide programs,
standards and accountability, and well-prepared teachers.

-   The greatest number of Center events addressed reading (27 percent
    of the total number of events). Other topics commonly addressed by
    the Centers included: implementing Title I schoolwide programs (23
    percent of the total number of events), standards and accountability
    (21 percent), well-prepared teachers (19 percent), curriculum and
    instruction (18 percent), and assessment (18 percent). Findings from
    the customer surveys administered by PSA as part of the
    Comprehensive Center evaluation confirm the Center's own data.

```{=html}
<!-- -->
```
-   Centers reported 4,095 events, or 21 percent of the total number of
    events, in the category "integrated, non-categorical technical
    assistance."

> **Note:** The number of events in this category is underreported
> because many Centers did not use this line in the table shells to
> report their activities. Between January and June 1999, 6 of the 15
> Centers reported a figure on this line, and between July and December
> 1999, 13 Centers reported a figure.

-   In contrast, Center services focused less often on ESEA
    program-specific topics, such as Title VII (12 percent of the total
    number of events), Title IX (10 percent), and Title IV (7 percent).

```{=html}
<!-- -->
```
-   In their professional development services, Centers focused most on
    reading (34 percent of the total number of professional development
    events), followed by assessment (25 percent), standards and
    accountability (25 percent), and curriculum and instruction (24
    percent).

-   In their consultation services, Centers focused most on reading (26
    percent of the total number of consultation events) and Title I
    schoolwide programs (25 percent).

> **Note:** The numbers reported in Table 3 are all duplicated counts.
> Most Centers reported that single events addressed more than one
> topic. For instance, it is plausible that a single professional
> development event could address the needs of limited English
> proficiency students, reading, standards, and assessment. Thus, the
> column totals in this table are far greater than the total number of
> events reported in Table 1a. To calculate the percent of all events
> that addressed a specific topic, we used the unduplicated total
> reported in Table 1a as the denominator.
>
> **Note:** The sum across each row does not always equal the reported
> count in the "Total Number of Events" column. This is a result of
> inaccurate reporting by individual Centers. For the purpose of
> aggregating across Centers, we consistently followed the rule of
> entering counts as the Centers reported them.

**[Recipients of Technical Assistance Services]{.underline} (Tables 5
and 6)**

**The Comprehensive Centers reported a large volume of contacts with
individual customers during 1999.**

-   During 1999, the Centers reported 519,051 contacts with educators
    and administrators at all levels of the education system (up from
    279,821 in FY 98). Additionally, they reported 88,817 contacts with
    parents, community members, and other customers. Each Center
    reported an average of 3,377 service contacts with individual
    customers per month.

**Contacts with first-time customers are a small percentage of all
Comprehensive Center contacts with individual customers, suggesting that
Centers establish ongoing relationships with customers that consist of
multiple contacts.**

-   From January to June 1999, Centers reported 5 percent (9,366) of all
    contacts were with first-time customers.

-   From July to December 1999, Centers reported that they gave
    assistance for the first time to 10 percent of all institutions they
    served.

> **Note:** Due to ED's mid-year change in reporting instructions,
> Centers reported first-time assistance to individual customers in the
> January to June 1999 report and first-time assistance to institutions
> in the July to December 1999 report.

**The Comprehensive Centers had more contacts with teachers than with
any other type of service recipient. Paraprofessionals, LEA
administrators, and principals had the next greatest number of contacts
with Centers, though they had substantially fewer than did teachers.**

-   The Centers reported that 48 percent of all Center service contacts
    were with teachers. Specifically, teachers had about half of all
    Center consultation (53 percent), information dissemination (52
    percent), and professional development (49 percent) contacts. They
    accounted for about a third of contacts in activities that the
    Center convened or facilitated (37 percent).

> **Note:** It is not surprising that Comprehensive Centers reach more
> teachers, because there are more teachers to serve than any other
> customer type and teachers are more likely to participate in
> large-group activities.

-   The Centers reported that they had a smaller number of service
    contacts with paraprofessionals (12 percent), LEA administrators (11
    percent), and principals (11 percent).

**The Centers had more contact with schools than with any other type of
institution. Most school-level contacts were with high-priority schools,
as defined in the Centers' authorizing legislation.**

-   From January to June 1999, 54 percent of all Center contacts with
    institutions were with schools. From July to December 1999, 45
    percent of all institutions receiving Center services were schools.

-   From January to June 1999, 88 percent of all Center contacts with
    schools were with high-priority schools---48 percent with high
    poverty, schoolwide program schools, 36 percent with high poverty,
    non-schoolwide program schools, and 4 percent with Bureau of Indian
    Affair (BIA) schools. Eleven percent of contacts with schools were
    with schools that were not high-priority schools, and 9 percent
    contacts with schools were unaccounted for in Center reports.

```{=html}
<!-- -->
```
-   From July to December 1999, 89 percent of all schools receiving
    services were high-priority schools---58 percent were high poverty,
    schoolwide program schools, 29 percent were high poverty,
    non-schoolwide program schools, and 2 percent were BIA schools. Nine
    percent of schools receiving services were not high-priority
    schools, and 2 percent were unaccounted for in Center reports.

-   For ED's reporting under the Government Performance Results Act
    (GPRA), PSA reported that an average of 77 percent of Center
    contacts with schools in 1999 were with schoolwide, high-poverty, or
    BIA schools. In order to prevent one Center's data (which are
    currently being retabulated) from skewing aggregate data, we
    calculated the percent of contacts with high-priority schools for
    each Center and averaged that percent across Centers. This method
    was used to calculate the percent of contacts with all types of
    institutions in 1999 reported under Indicator 1.1 in ED's GPRA
    report.

**Comprehensive Centers also had many contacts with districts, and to a
lesser extent, with SEAs and other institutions.**

-   From January to June 1999, 26 percent of all Center contacts with
    institutions were with LEAs. In the second half of the year, July to
    December 1999, the Centers reported that LEAs made up 31 percent of
    all institutions served.

-   After schools and districts, SEAs received the next greatest number
    of Center contacts. From January to June 1999, 8 percent of all
    Center contacts with institutions were with SEAs, and from July to
    December 1999, 11 percent of all contacts were with them.

-   From January to June 1999, 11 percent of all Center contacts with
    institutions were with institutions categorized as "other," and from
    July to December 1999, 10 percent of contacts were with "other"
    institutions. For both reporting periods, less than 2 percent of
    Center contacts were with either community-based organizations or
    intermediate units.

> **Note:** The Centers received a new set of reporting instruction for
> Table 6 in June 1999. From January to June 1999, the Centers reported
> duplicated counts of contacts with institutions. Thus, an institution
> that received more than one service would be counted more than once.
> For the July through December 1999 reporting period, the Centers were
> directed to report unduplicated counts of institutions. In other
> words, an institution that receives Center services would only be
> counted once. The fact that Center reports of total contacts with
> institutions dropped by two-thirds over the two periods suggests that
> they are increasingly reporting unduplicated counts. Nevertheless,
> based on counts of SEAs that received Center services, it appears that
> only 4 of the 15 Centers reported unduplicated counts of institutions
> (see Tables 6a and 6c).

**[Collaboration with Other Agencies]{.underline} (Tables 1 and 4)**

**Compared with FY 1998, Comprehensive Centers reported an increase in
their collaborations with other agencies to provide technical assistance
in 1999.**

-   In Table 1a, Centers reported that 18 percent of all technical
    assistance activities involved collaboration with a partner (up from
    8 percent in FY 98).

```{=html}
<!-- -->
```
-   Of the total number of events that involved collaboration in 1999,
    68 percent involved collaboration with non-federally funded
    technical assistance providers, which consisted mostly of SEAs (see
    next bullet and note). Also, 34 percent involved "other" providers.

-   For the July through December 1999 period, of the total number of
    events that involved collaboration, 61 percent involved
    collaboration with state education agencies,

-   26 percent with "other" organizations, 21 percent with institutions
    of higher education, 21 percent with intermediate units, and 10
    percent with community based organizations.

> **Note:** Beginning with the July through December 1999 period, the
> Center reports were reformatted to clarify categories of collaborative
> agencies (contrast Tables 4a and 4b). Most importantly, the category
> "non-federally funded technical assistance provider" was replaced with
> specific categories, such as "institutions of higher education,"
> "state education agencies," and "intermediate units." As a result,
> counts for the new categories are only available for the second half
> of 1999.

-   In many instances, Comprehensive Centers collaborated with other
    federally funded technical assistance providers. Of the total number
    of events that involved collaboration in 1999, 38 percent involved
    collaboration with Regional Educational Laboratories, 14 percent
    with another Comprehensive Center, 6 percent with Eisenhower
    Consortia, and 24 percent with other federally funded providers.

> **Note:** Figures in Table 4a may be duplicated in both FY 1998 and
> 1999. A single event may be counted more than once if it involved
> collaboration with more than one type of collaborating agency. For
> instance, one event that involved collaboration between a regional
> educational laboratory, second Comprehensive Center, and state
> department of education could be reported six times in this table
> (assuming both centers reported the event). This would inflate the
> total number of events involving collaboration as reported in Table
> 4a, as well as explain why the Table 4a total is much larger than the
> Table 1a collaboration total. Above, to avoid duplication of counts,
> events by type of collaborating agency reported in Table 4 are divided
> by the collaboration total reported in Table 1a.

**[Web Site Hits]{.underline} (Table 7)**

**Compared with FY 1998, Comprehensive Centers are receiving an
increasing number of Web site hits, although there continues to be great
variation among Centers.**

-   Comprehensive Centers received a total of 690,690 Web site hits, an
    average of 3,837 per month per Center. This is a substantial
    increase from FY 1998's total of 218,001.

-   The number of Web site hits continues to vary greatly from Center to
    Center, ranging from 2,580 to 323,384. The Region III Comprehensive
    Center and the Northwest Regional Educational Laboratory's
    Comprehensive Center, whose Web sites are hosted by a university and
    a regional educational laboratory, account for 65 percent of the
    total hits.

[^1]: The Comprehensive Regional Assistance Centers program replaced
    five technical assistance programs that had previously served
    grantees of the Bilingual Education, Chapter 1, Drug-Free Schools
    and Communities, Indian Education, and Migrant Education programs.

[^2]: Title XIII, Part A of the Elementary and Secondary Education Act,
    refers to this network as the \"Comprehensive Regional Assistance
    Centers.\" We use the name "Comprehensive Centers," which is
    commonly in use among the Comprehensive Centers themselves and
    within ED, for the remainder of this report.

[^3]: The names of this school and all the other schools and districts
    in our case study sample have been changed.

[^4]: Because the format of the tables was revised mid-year, some data
    from January 1 to June 30 could not be combined with data from July
    1 to December 31. This is true in the case of Table 6, where counts
    were duplicated for the first half of the year but not the second
    half. In these cases, our tables aggregate data for each half of the
    year separately. Also, some of the revised tables gained and lost
    row and column categories (Tables 1, 3, and 4). In these cases, we
    provide an aggregate table that reports the data that could be
    combined for the entire year 1999, as well as an aggregate table
    that reports only on July to December.