NCEE 2010-4034 U.S. DEPARTMENT OF EDUCATION

National Evaluation of the 

Comprehensive Technical 

Assistance Centers

Interim Report

Executive Summary

National Evaluation of the 

Comprehensive Technical Assistance 

Centers 

Interim Report 

Executive Summary 

JULY 2010

Brenda J. Turnbull 

Richard N. White 

Elizabeth Sinclair 

Derek Riley 

Policy Studies Associates, Inc. 

Cynthia L. Sipe 

Branch Associates, Inc. 

Carol Pistorino

Decision Information Resources, Inc. 

Yumiko Sekino 

Project Officer 

Institute of Education Sciences 

NCEE 2010-4034

U.S. DEPARTMENT OF EDUCATION
U.S. Department of Education

Arne Duncan 

Secretary

Institute of Education Sciences 

John Q. Easton 

Director

July 2010 

This report was prepared for the Institute of Education Sciences under Contract No. ED-04-CO-0028/0001. The project 

officer is Yumiko Sekino in the National Center for Education Evaluation and Regional Assistance. 

IES evaluation reports present objective information on the conditions of implementation and impacts of the programs being 

evaluated. IES evaluation reports do not include conclusion or recommendations or views with regard to actions 

policymakers or practitioners should take in light of the findings in the reports. 

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to reprint 

this publication is not necessary, the citation should be: Turnbull, B.J., White, R.N., Sinclair, E., Riley, D., Sipe, C.L., and 

Pistorino, C. (2010, July). National Evaluation of the Comprehensive Technical Assistance Centers: Interim Report. 

Executive Summary (NCEE 2010-4034). U.S. Department of Education, National Center for Education Statistics. 

Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. 

Department of Education. 

To order copies of this report, 

 Write to ED Pubs, Education Publications Center, U.S. Department of Education, P.O. Box 22207, Alexandria, VA 

22304. 

 Call in your request toll free to 1-877-4ED-Pubs. If 877 service is not yet available in your area, call 800-872-5327. 

Those who use a telecommunications device for the deaf (TDD) or a teletypewriter (TTY) should call 800-437-

0833. 

 Fax your request to 703-605-6794 or order online at www.edpubs.gov. 

This report also is available on the IES website at http://ncee.ed.gov. 

Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer diskette. For 

more information, please contact the Department„s Alternate Format Center at 202-205-8113. 

 
iii

Acknowledgments 

 This report benefited from the work of a colleague whose loss we have felt keenly: 

Cynthia Sipe passed away, much too young, in March 2010. Her professionalism and good 

humor were priceless assets to Branch Associates, Inc., and to this study. 

 We very gratefully acknowledge the cooperation of the Comprehensive Centers as well 

as the many other respondents to our data collection. Center leaders and staff graciously hosted 

site visits, compiled inventories of their projects, and assembled hefty collections of their 

materials for expert review. Senior managers in state education agencies were kind enough to 

respond to the study‟s survey regarding their experiences with Comprehensive Center technical 

assistance, as were the many technical assistance participants housed in state, intermediate, and 

local agencies and other organizations. Expert panel members brought knowledge and care to 

their task of reviewing and rating project materials.

A Technical Working Group offered useful insights and feedback in two meetings as well 

as informal consultations: the group‟s members were Rolf Blank, Margaret Goertz, Joseph 

Johnson, Michael Petrilli, Gary Ritter, Larry Rudner, and Colleen Seremet. 

In addition to the authors, other members of the participating organizations played 

important roles in study design, data collection, and analysis. Alvia Branch of Branch 

Associates, Inc. (BAI), provided steady leadership for the study team. Lynda Bell, Matthew Coll, 

Barbara Fink, Jennifer Gross, Ceane Rabada, Virginia Smith, and Jennifer Thompson were 

valued team members at BAI. Russell Jackson of Decision Information Resources, Inc. (DIR), 

was a source of wise counsel. Also at DIR, Nancy Dawson, Val Sheppard, Stella Weathersby, 

Frenetta Tate, and Pia White were stalwart contributors. At Policy Studies Associates (PSA), 

team members who made valuable contributions in data collection or coding included Erikson 

Arcaira, Tiffany Miller, Tandra Turner, and Yvonne Woods. Ben Lagueruela provided essential 

help in document production. Katrina Laguarda, formerly of PSA, made tremendous 

contributions to the study design and early activities. 

The study and this report reflect the careful attention of many professionals in the 

Institute of Education Sciences of the U.S. Department of Education. 

v

Disclosure of Potential Conflicts of Interest 

The research team for this evaluation consists of a prime contractor, Branch Associates, 

Inc. and two subcontractors, Policy Studies Associates, Inc. (PSA), and Decision Information 

Resources, Inc. (DIR). None of these organizations or their key staff members has a financial 

interest that could be affected by findings from the evaluation of the Comprehensive Center 

program considered in this report. Additionally, no one on the Technical Working Group, 

convened by the research team to provide advice and guidance, has financial interests that could 

be affected by findings from the evaluation. 

vii 

Contents 

Page 

Acknowledgments..........................................................................................................................iii

Disclosure of Potential Conflicts of Interest ................................................................................... v

Contents ............................................................................................................................. vii

Executive Summary ....................................................................................................................... ix

The Comprehensive Centers Program ..................................................................................... xi

Evaluation Topics and Methods ............................................................................................... xi

Operation of Centers .............................................................................................................. xiii

Ratings of Center Assistance ................................................................................................. xvi

State Capacity Building and the Use of Different Sources of Technical Assistance ............. xxi

Summary and Next Steps ....................................................................................................... xxi

ix

Executive Summary 

 This first of two reports presents early findings from the National Evaluation of the 

Comprehensive Technical Assistance Centers (Comprehensive Centers), a federally funded 

program that provides technical assistance to states in connection with the No Child Left Behind 

(NCLB) Act of 2001. The law authorizing the Comprehensive Centers, the Educational 

Technical Assistance Act of 2002, mandated that a national evaluation of the program be 

conducted by the Institute of Education Sciences (IES). The legislation indicated that the 

evaluation should “include an analysis of the services provided...[and] the extent to which each 

of the comprehensive centers meets the objectives of its respective plan, and whether such 

services meet the educational needs of State educational agencies, local educational agencies, 

and schools in the region.” The program evaluation is conducted by Branch Associates, Inc., 

Decision Information Resources, Inc., and Policy Studies Associates, Inc. 

This report addresses the first of the evaluation‟s three rounds of data collection, 

pertaining to the Centers‟ work of July 2006 through June 2007. It describes the program design 

and, drawing upon data provided by the Centers and their clients, program operations. It also 

describes assessments of Center activities and resources, reporting on quality as judged by panels 

of subject-matter experts, and on relevance, usefulness, and contributions to capacity as judged 

by practitioners (namely, state-level managers and also clients who participated directly in 

Center activities or received Center products). A final report will provide parallel findings for 

2007-08 and 2008-09. In addition, it will present findings from case studies of capacity building 

at the state level and any changes in findings over time. 

The main findings from the evaluation so far are: 

■ The Comprehensive Centers reported planning their work in coordination and 

consultation with their clients with the work evolving during the year. All 16 

Regional Comprehensive Centers (RCCs) reported obtaining state input into their 

initial plans and engaging states in refinements to the plans through ongoing 

interaction and negotiation. Similarly, all five Content Centers (CCs) reported 

forming their work plans incorporating RCC input acquired through either RCC 

staff surveys or direct communication. In addition, all five CCs described working 

with the U.S. Department of Education (ED) to learn of specific topics and tasks 

needed to advance ED priorities. A review of projects conducted indicates that both 

RCCs and CCs adjusted their work plans during the year. 

■ The technical assistance activities were varied and consisted of ongoing 

consultation, research syntheses, planning of technical assistance with 

participants, training events, conferences, and support for task forces or for 

development of formal plans. Consistent with the mission of “front-line” 

assistance, the majority of sampled RCC projects involved ongoing consultation and 

follow-up (82 percent). CC assistance most often focused on the delivery of research 

information, consistent with the CCs‟ prescribed focus on synthesizing, translating, 

and delivering knowledge on a particular topic. The delivery of research collections 

and syntheses occurred in 74 percent of the CC sampled projects. 
x

■ The Comprehensive Centers program delivered technical assistance that, 

according to state managers: (1) served state education agencies’ (SEAs’) 

purposes in seeking technical assistance, (2) was aligned with SEAs’ 

priorities for NCLB-related technical assistance, and (3) was perceived to 

expand SEA capacity. Eighty-eight percent of state managers rated the technical 

assistance they received from Centers as at least “a good start” in serving their 

purposes, and 36 percent overall reported that it “served the state‟s purposes 

completely.” For each of the four areas of NCLB implementation most widely 

identified as state priorities for technical assistance, at least 90 percent of those 

state managers who had identified the area as a priority had received assistance 

with it from the Centers. Overall, more than two-thirds of state managers (68 

percent) reported a perception that assistance from the Comprehensive Centers 

had greatly expanded their state‟s capacity to carry out its responsibilities in at 

least one NCLB area. 

■ Center projects in the evaluation sample were judged by clients to be on 

average in the “moderate” to “high” range of relevance and usefulness; 

panels of experts judged their quality to be in the “moderate” range on 

average. On a scale of 1 to 5 with a 3 representing “moderate” and a 4 

representing “high,” the programwide average ratings for the sampled projects 

were 3.34 for technical quality (scored by panels of content experts), and 3.94 for 

relevance and 3.70 for usefulness (scored by participants).1

The average quality 

rating was higher among CCs than RCCs by more than one-half of a standard 

deviation; the average relevance rating was higher among RCCs than CCs by at 

least one-half of a standard deviation; usefulness ratings were similar between the 

two Center types (i.e., did not differ by at least one-half of a standard deviation).2

 

1

 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 

to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 

and each project sampled from a Center contributed equally to the Center mean. 

2

 Using Cohen (1988) as a conceptual framework, we estimated Cohen's d (an estimate of the effect size defined as 

the difference in means divided by the pooled standard deviation) and adopted the logic of Cohen for what is a 

moderate difference. Specifically, we adopted a difference in the means of one-half of one standard deviation 

(analogous to an effect size of .5) as our minimum threshold for highlighting differences. The “pooled standard 

deviation” for each computation varied with the unit of analysis. For analyses conducted at the Center level, the 

pooled standard deviation was computed as the standard deviation of the variable of interest (e.g., relevance) 

computed at the Center level. For analyses using the project as the unit of analysis, the pooled standard deviation 

was computed at the project level. 
xi 

The Comprehensive Centers Program 

 In its authorization, the Comprehensive Centers program was given an overall charge of 

supporting state and local NCLB implementation. ED, using discretion provided in the 

legislation, established two major program features that differed from past Comprehensive 

Centers programs:3

■ First, the primary focus would be on assisting states to expand and strengthen 

states‟ capacity to deliver assistance to schools and districts; ED specified that 

Centers could only work directly with districts or schools under special 

circumstances. 

■ Second, awards would be made in two tiers, to 16 RCCs and 5 CCs. They were 

instructed to work as follows: 

■ Each RCC was charged with providing “frontline assistance” either to one 

large state or to a group of two to eight states and other jurisdictions.4The 

RCCs were also expected to deliver technical assistance to their assigned 

states, addressing the needs and building capacity of the states to assist 

their districts and schools. 

■ Meanwhile, each CC would work on a nationwide basis within a particular 

substantive area: Assessment and Accountability, Instruction, Teacher 

Quality, Innovation and Improvement, or High Schools. CCs would 

facilitate access to, and use of, existing research and practices. 

■ The absolute priorities for the two types of Centers indicated that they 

should work together: Regional Centers should draw information and 

resources from Content Centers as well as other sources; and Content 

Centers should both supply knowledge to Regional Centers and “work 

closely with Regional Centers to provide technical assistance to States.”

Evaluation Topics and Methods 

The research priorities for the evaluation were primarily driven by the statute and focused 

on the following key research questions: 

■ What are the objectives of the Comprehensive Center network and of each Center? 

 

3

 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 

32583-94. 

4

 The nonstate jurisdictions that the Centers were to serve were the following: the District of Columbia, Puerto Rico, 

the Virgin Islands, American Samoa, Commonwealth of the Northern Mariana Islands, Federated States of 

Micronesia [Chuuk, Kosrae, Pohnpei, and Yap], Guam, Republic of the Marshall Islands, and Republic of Palau. 

Throughout this report, the term “state” will be defined to include the 50 states as well as these other jurisdictions. 
xii 

■ What kinds of products and services are provided by the Comprehensive Center 

network and by each Center? 

■ How do Centers develop, refine, and carry out their plans for technical assistance? 

How do they define their clients‟ educational needs and priorities? How do Center 

clients (states or Regional Centers) define their needs and priorities? 

■ To what extent is the work of each Comprehensive Center of high quality, high 

relevance, and high usefulness?

■ To what extent do states report that Center projects have expanded state capacity 

to address underlying needs and priorities and meet the goals of NCLB? 

■ To what extent have states relied on other sources of technical assistance besides 

the Comprehensive Centers? What other sources? How does the usefulness of 

Center projects compare with the usefulness of projects from other sources? 

 The evaluation gathered annual information from six data sources in order to address the 

research questions above. Data collection included: 

■ Management plans. The evaluation reviewed these as a data source for each 

Center‟s intended focus at the beginning of the year, drawing from the plans a list 

of topics as foci of Center objectives. 

■ Center staff interviews. Using structured response categories, Center staff were 

asked about how they planned their programs of work; how their plans evolved 

during the program year; and what they offered to clients with respect to the 

topics addressed, the delivery modes used, and their sources for content expertise. 

(See appendix C for the protocols and other structured response materials used 

during the interviews). 

■ Survey of senior state managers. SEA managers were surveyed about their 

state‟s technical-assistance needs and what the Centers (including their RCC and 

the CCs) had provided. 

■ Project inventory forms. The evaluation team assisted each Center in grouping 

related activities and deliverables into “projects,” with the project defined as a 

group of closely related activities and/or deliverables designed to achieve a 

specific outcome for a specific audience. Projects were in turn classified by the 

Centers into major, moderate, and minor projects on the basis of the relative level 

of effort they reflected. The Centers and the evaluation team also classified the 

projects, according to the topics addressed, into 22 topical categories.5

 

5

The 22 topics were: components of effective systems of support for states, districts, and schools; data use or datadriven decision making; formative assessment; reading; adolescent literacy; mathematics; dropout prevention; high 

school redesign or reform; transition to high school; special education curriculum, instruction and professional 

development; special education assessment; English language learners;” highly qualified teacher” provisions of 

NCLB; teacher preparation and induction; teacher professional development; supplemental educational services; 

Response to Intervention; migrant education; Indian or Native American education; data management and 
xiii 

■ Survey of project participants. A representative sample of clients who had 

participated directly in the evaluation‟s purposive sample of major and moderate 

Center projects furnished descriptive information, through surveys, on the 

technical-assistance needs of their offices and on the activities and resources that 

the project had delivered to them. These clients included individuals working at 

the state level who had participated in RCC or CC projects and RCC employees 

who were among the clients of CC projects. They also rated the relevance and 

usefulness of the sampled projects. 

■ Expert panel review. The same sample of major and moderate projects was

reviewed for quality by a panel of experts. Content experts were recruited and 

trained to use standard criteria to rate the technical quality of the sampled Center 

projects on the basis of a review of all project materials. 

Operation of Centers 

Before the beginning of the 2006-07 program year, each Center was required to submit a 

management plan, setting out objectives and planned activities for the coming year, for ED 

review and approval. Almost all Centers (20 of 21) reported that client input was used in writing 

the plan. The same number reported conducting needs assessment through meetings or other 

communication with clients. 

In addition to initial needs assessments, to ultimately meet client needs Centers used 

planning and ongoing interactions with each other and with the SEAs to refine their needs 

assessments as indicated in exhibit ES.1. Fifteen of 16 RCCs reported forming work groups 

within state organizations that brought together staff from multiple departments to discuss service 

needs and delivery; the remaining RCC was 1 of 14 that reported working directly with the chief 

state school officer in their initial planning. Half of the RCCs (8) formed cross-agency work 

groups to discuss SEA service needs and delivery. All five CCs identified needs of their client 

RCCs primarily by conducting conference calls with designated RCC representatives, and three of 

the five CCs maintained communication about needs by forming workgroups that included RCC 

representatives. In addition, all CCs reported providing either large-group events or support of 

existing RCC programs and projects as additional ways to learn about and meet client needs. 

 

compliance; assessment design; and parent involvement. In addition, projects that addressed none of these 22 topics 

were categorized as “other.”
xiv 

Exhibit ES.1. Center strategies for planning and ongoing interactions with 

clients

Center strategy

RCCs

(N=16)

CCs

(N=5)

All

(N=21)

Assess RCC needs through meetings or other communication 

with RCC staff 5 -

Included SEA input when writing annual management plans 16 - -

Sponsor large events to make contact with many clients 13 4 17

Form work groups within client organizations that bring together 

staff from multiple departments/divisions to discuss service 

needs and delivery

15 1 16

Offer service to support existing client programs/ projects/ 

policies 9 4 13

Form work groups across client organizations to discuss service 

needs and delivery (e.g., different SEAs and/or RCCs) 8 3 11

EXHIBIT READS: All five CCs assessed RCC needs through meetings or other communication with RCC 

staff. 

SOURCE: Verbatim summaries of Center interviews conducted during 2007 site visits, coded by evaluation 

team, with Center review of coding results 

Although Centers reported providing technical assistance in a majority (80 percent) of the 

instances where identified topic areas were included in their management plans, Centers 

commonly adjusted the topic areas in which they conducted work, adding work in some areas 

and shifting away from work in other areas. Based on a review of each Center‟s inventory of 

projects against the topics that had been included in that Center‟s management-plan objectives, 

there were instances of the Center carrying out work on a topic not initially cited in its 

objectives. That is, in 19 of 22 topic areas Centers reported delivering technical assistance that 

was not in their original management plan. The most common shift toward topics were in the 

areas of Response to Intervention (seven Centers conducted work in this area that was not 

planned), English language learner issues (five Centers), highly qualified teacher provisions of 

NCLB (five Centers) and supplemental educational services (five Centers). By the same token, 

Centers appeared to delete work in particular topic areas such as special education curriculum, 

instruction, and professional development (five Centers appeared to delete planned work); and 

data use or data-driven decisionmaking (four Centers appeared to delete planned work). Centers 

that set an objective in a topic did not report a project on the topic in their inventory for 10 of the 

22 topic areas. The most common topic area for projects was that of statewide systems of support 

for educational improvement. 

Regardless of whether they were originally specified in work plans or added later, an 

analysis of the projects that the evaluation team sampled for closer study across all Centers 

provides more in-depth information about the nature of Center technical assistance activities or 

resources. Although the sample of projects is not statistically representative of the Centers‟ work, 

the process of sample selection favored each Center‟s most dominant projects and included over 

half (56 percent) of that year‟s designated major or moderate projects. Most projects (84 percent) 

used more than one mode of delivery from a list that included conferences, training, delivery of 

research collections or syntheses, support for a task force, support for development of a plan or 
xv

policy, engagement of clients in project planning, and ongoing consultation and follow-up. 

Across all the sampled projects and also across the Regional Centers‟ sampled projects, the most 

frequent modes of delivery as shown in exhibit ES.2 were ongoing consultation and follow-up 

(84 of 122, or 69 percent of all projects and 79 of 96, or 82 percent of Regional Center projects) 

or delivery of a research collection or synthesis (71 of 122, or 58 percent of all projects and 52 of 

96, or 54 percent of Regional Center projects). The Content Centers‟ projects most often 

included delivery of a research collection or synthesis (20 of 27, or 74 percent of their projects) 

or a conference (17 of 27, or 63 percent). 

While some projects were worked on by both the RCCs and the CCs, coordination 

between CCs and RCCs when it did occur was asymmetrical. In providing assistance to states, 

RCCs used CC input more than CCs used substantive RCC input. Almost half of the sampled 

RCC projects had a substantive CC contribution (such as a product or a presentation by a CC 

staff member); in contrast, in 11 percent of the sampled CC projects an RCC contributed content 

or delivered assistance. More often (in 37 percent of the sampled CC projects), the CC enlisted 

the help of one or more RCCs to identify and recruit participants. 

Exhibit ES.2. Sampled Center projects by types of participant activities 

and products 

Activities and products

(with clarifying definitions used by coders)

RCC 

projects

(n=96)

CC 

projects

(n=27)

All 

projects

(n=122)

Ongoing consultation and follow-up (multiple contacts to same 

participants, that were part of a coherent and purposeful whole) 79 6 84

Research collections and syntheses 52 20 71

Engagement of participants in project planning (more than needs 

assessment or identifying participants) 43 8 50

Training events (focused on implementing a specific program or 

strategy) 41 10 50

Task force meetings and work (focused on addressing a specific 

problem, program, or policy) 48 2 50

Conferences (symposium, forum, institute; highlights a range of 

perspectives, strategies, or programs) 26 17 43

Support development of a formal plan to implement a program or 

policy 18 2 20

EXHIBIT READS: Seventy-nine RCC projects included ongoing consultation and follow-up. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials; cover sheets 

coded by evaluation team. The total number of projects was 122. One project collaboratively conducted by an 

RCC and a CC was counted among both RCC projects and CC projects but was only counted once among the 

projects of all Centers. 
xvi 

The types of work emphasized in the RCC and CC sampled projects were consistent with 

their different charges. RCC assistance more often incorporated sustained interaction with 

participants: again, the majority of RCC projects in the sample involved ongoing consultation 

and follow-up (82 percent), whereas this was less so for the work of the CCs (22 percent of 

projects in the sample). Thus, this pattern of RCC activities was consistent with the mission of 

“front-line” assistance that would take clients‟ purposes and circumstances into account and 

provide ongoing support for their implementation of NCLB. For the CCs, the assistance more 

often focused on the delivery of research information, consistent with the CCs‟ prescribed focus 

on synthesizing, translating, and delivering knowledge on a particular topic. The delivery of 

research collections and syntheses occurred in 74 percent of the CC projects but 54 percent of the 

RCC projects. The sampled CC projects more often delivered technical assistance through 

conferences (63 percent of the CC projects but 27 percent of the RCC projects). 

Ratings of Center Assistance 

The sampled projects, all identified by the Centers as “major” and “moderate,” were rated 

in order to assess the services provided by the Comprehensive Centers program. Each project 

was evaluated for relevance and usefulness by a sample of participants—state staff, intermediate 

agency staff, local educators working on behalf of the state, and RCC staff—who were the 

intended beneficiaries of the project and who had received at least some of the technical 

assistance it provided. Ratings of project quality were gathered from panels of experts with 

strong knowledge of the content or substantive focus of the specific projects they reviewed. 

Relevance was assessed with eight survey items and usefulness with 11 items; quality was 

judged on three items called dimensions (exhibit ES.3). Each overall measure (relevance, 

usefulness, or quality) was calculated as the mean of ratings assigned to each item. The itemlevel ratings themselves were based on 5-point rating scales.6

 

6

 Efforts were made to develop parallel wording and rubrics that would result in similar gradations between rating 

levels (e.g., very high vs. high vs. moderate) across the three measures. However, given the different content of each 

set of items within the three measures and the different contexts for the ratings (experts who underwent training for 

the rating process and reviewed identical packages of materials vs. survey respondents who typically participated in 

different subsets of project activities), the ratings across the three measures are not directly comparable. 
xvii 

Exhibit ES.3. Relevance, usefulness, and quality items 

From expert panel scoring From project participant surveys

Technical quality Relevance Usefulness

Reviewers were directed to 

assign a score to each 

dimension and to include the 

basis for their ratings on the 

rating form, including the specific 

artifacts on which their score was 

based. The three dimensions 

are:

a. Demonstrated use of the 

appropriate documented

knowledge base – to include 

an accurate portrayal of the 

current state of information 

with prominence to those 

with the most 

accurate/rigorous evidence 

b. Fidelity of application of the 

knowledge base to the 

products and services 

provided – materials are 

consistent with the 

best/accurate information 

available and the

presentation adequately 

conveys the confidence of 

the information

c. Clear and effective delivery –

information is well organized 

and written and accessible to 

the intended audience for 

easy use 

Based on your experience, to 

what degree was this set of 

activities and resources relevant

to your work, in each of the 

following respects?

a. Addressed a need or problem 

that my organization faces

b. Addressed an important 

priority of my organization

c. Addressed a challenge that 

my organization faces related 

to the implementation of NCLB

d. Provided information, advice, 

and/or resources that could be 

directly applied to my 

organization’s work

e. Addressed our particular state 

context

f. Addressed my organization’s 

specific challenges (e.g., 

policy environment, leadership 

capacity, budget pressures, 

local politics) 

g. Provided information, advice, 

and/or resources that could be 

used to guide decisions about 

policies, programs, or 

practices

h. Highlighted the implications of 

research findings (or 

information about best 

practice) for policies, 

programs, or practices

Based on your experience, to 

what degree was this set of 

activities and resources useful to 

you, in each of the following 

respects?

a. Provided resources that were 

easy to understand and easy 

to use

b. Employed an appropriate 

format (e.g., a work group, a 

conference, individual 

consultation, written products)

c. Provided adequate opportunity 

to learn from colleagues in 

other states

d. Included adequate follow-up to 

support the use of new 

information and resources

e. Were timely

f. Helped my organization solve 

a problem

g. Helped my organization 

maintain or change a policy or 

practice

h. Helped my organization take 

the next step in a longer-term 

improvement effort

i. Provided my organization with 

information or resources that 

we will use again

j. Helped my organization 

develop a shared expertise or 

knowledge-base

k. Helped individuals in my 

organization to develop skills 

that they will use again

Based on the ratings, Center technical assistance was judged to be in the “moderate” to 

“high” range of quality, relevance, and usefulness. On a scale of 1 to 5 with a 3 representing 

“moderate” and a 4 representing “high,” the programwide average ratings for the sampled 

projects were 3.34 for technical quality (scored by panels of content experts), and 3.94 for 

relevance and 3.70 for usefulness (scored by participants) as indicated in exhibit ES.4.

7

 

 

7

 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 

to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 

and each project sampled from a Center contributed equally to the Center mean. 
xviii 

Exhibit ES.4. Center Level Mean ratings of technical quality, relevance, and 

usefulness 

Technical 

quality Relevance Usefulness

All Comprehensive 

Centers (N=21) 3.34 3.94 3.70

All RCCs (N=16) 3.21 3.99 3.71

All CCs (N=5) 3.73 3.78 3.65

Difference of RCC and 

CC means -0.52†

0.21† 0.06

Pooled standard deviation

(all Comprehensive 

Centers)

0.41 0.34 0.34

Ratio of difference in means 

to pooled standard deviation -1.28 0.60 0.18

NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” rating is the mean of the 

ratings for the three quality dimensions. A notation of † indicates that the difference in the mean ratings between the 

CCs and RCCs is at least one-half of one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.34. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 

ratings; each project contributed equally to Center ratings; and each Center contributed equally to cross-Center 

ratings. 

 

Given that the RCC and CC roles and activity emphasis differed, the evaluation looked at 

variation across Center types and projects that might provide information for program 

improvement. The average quality rating was higher among CCs than RCCs by more than onehalf of a standard deviation8

; the average relevance rating was higher among RCCs than CCs by 

at least one-half of a standard deviation; usefulness ratings were similar between the two Center 

types (differing by less than one-half of a standard deviation). The Content Centers received 

Center-level mean scores for technical quality that averaged 3.73, compared with 3.21 for the 

Regional Centers; the difference of 0.52 points exceeded one-half of one pooled standard 

deviation. The mean scores for relevance were 3.99 for the Regional Centers and 3.78 for the 

Content Centers. On usefulness, the mean score of 3.71 for the RCCs and 3.65 for the CCs were 

within one-half of a standard deviation of each other. 

There was variation in the ratings across and within individual Centers. On each measure, 

at least 11 Centers had a mean rating that was at least one-half of a standard deviation above or 

below the overall mean for its type of Center (RCC or CC) for that measure9(i.e., 11 of 21 

Centers were this far above or below the mean for quality, 11 for relevance, and 14 for 

 

8

 For analyses conducted at the Center level, the pooled standard deviation was computed as the standard deviation 

of the variable of interest (e.g., relevance) computed at the Center level. 

9

 For analyses using the project as the unit of analysis, the pooled standard deviation was computed at the project 

level. 
xix 

usefulness). One RCC was rated higher than others by at least one-half of a standard deviation on 

all three measures, and one CC and one RCC were rated lower than others on all three measures. 

Aside from these Centers, the other 18 Centers‟ ratings were not consistently higher or lower 

than the mean but varied across measures (Exhibit ES-5). 

Exhibit ES.5. Mean ratings of technical quality, relevance, and usefulness, by 

Center 

Center type

Technical 

quality Relevance Usefulness

RCCs (N=16) 3.78 3.78 3.42

3.63 3.22 3.00

3.46 4.18 3.94

3.44 3.90 3.63

3.36 3.97 3.63

3.35 3.97 3.57

3.35 3.93 3.51

3.31 4.15 3.69

3.21 4.08 3.82

3.17 4.31 4.05

3.15 4.12 3.93

3.11 4.70 4.46

2.98 4.07 3.92

2.74 3.20 3.05

2.74 4.01 3.54

2.63 4.18 4.17

Average RCC rating 3.21 3.99 3.71

Pooled standard 

deviation (RCCs) 0.32 0.37 0.38

CCs (N=5) 4.24 3.76 3.54

3.94 3.90 3.86

3.88 3.99 3.84

3.44 3.58 3.44

3.14 3.68 3.56

Average CC rating 3.73 3.78 3.65

Pooled standard 

deviation (CCs) 0.43 0.16 0.19

NOTE: The arrow pointing upward indicates the accompanying value is at least one-half of one standard 

deviation above the group mean (e.g., 3.78 is at least one-half of one standard deviation above the mean 

for the RCCs). The arrow pointing downward indicates the accompanying value is at least one-half of 

one standard deviation below the group mean. 

EXHIBIT READS: One of the RCCs had a mean rating for technical quality of 3.78, a mean rating for 

relevance of 3.78, and a mean rating for usefulness of 3.42, across the projects sampled from that 

Center. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project 

participants for relevance and usefulness. Responses weighted so that each panelist or participant 

contributed equally to project ratings, and each project contributed equally to Center ratings 
xx

The evaluation also looked at the relationship between the three measures: quality, 

relevance, and usefulness. It was reasoned that the content experts rating quality and the 

participants rating relevance and usefulness might value and be better able to judge different 

qualities in a Center project, which is why we did not have content experts evaluate the projects 

for their utility or the participants assess the technical quality. An examination of the associations 

among the three dimensions was conducted by calculating correlation coefficients.10 Such a 

statistic indicates the strength and direction of a linear relationship between two factors. A 

correlation coefficient can vary from positive one (indicating a perfect positive relationship), 

through zero (indicating the absence of a relationship), to negative one (indicating a perfect 

negative relationship). If the correlation is statistically significant (p <.05), we can have strong 

(95 percent) confidence that what we calculated is not due to chance. 

Ratings of quality were unrelated to ratings of relevance and usefulness, although 

relevance and usefulness ratings were highly correlated with each other. The correlation 

coefficient for relevance and usefulness was +0.84, while the coefficient of relevance with 

quality was -0.12, and the coefficient of usefulness and quality was -0.04. In other words, the 

extent to which a project faithfully reflected the knowledge base on a topic and provided 

appropriate caveats about the quality of its evidence was unrelated to the extent to which 

participants deemed that project relevant or useful to their agency. 

Given the variation in ratings across projects, additional analyses of project 

characteristics were conducted to explore whether there were any consistent patterns between 

ratings and the particular features of the projects. Such information may provide suggestions for 

possible program improvement. Specifically, if there is a consistent relationship between scale of 

the undertaking and the ratings, perhaps signaling more ambitious projects or projects that allow 

a greater focus of Center resources on the effort, then this might be suggestive of productive uses 

of Center resources for future emphasis. In fact, projects identified by the Centers as “major”

were rated higher by at least one-half a standard deviation on the measures of relevance and 

usefulness but not on the measure of quality. 

Projects with particular types of activities may be easier to carry out, may play to Center 

strengths, or may be seen as more productive to the ultimate clients. In addition, those RCC 

projects that included CC contributions might be expected to have higher quality ratings than 

other RCC projects, given the expected content and research focus of the CCs. Thus, the 

evaluation compared ratings of subgroups of projects (e.g., those with and without particular 

activities such as conferences, training, or research syntheses; RCC projects with and without CC 

involvement) to see if there were any consistent relationships between the ratings and particular 

Center activities or the incorporation of CC work. Across five of the seven project activities 

identified, comparisons of projects with and without the activities showed no differences in 

quality, relevance, or usefulness greater than one-half of a standard deviation. Ratings of RCC 

projects with CC contributions did not differ by more than one-half of a standard deviation on 

any measures, compared with those without CC contributions. Thus, these analyses do not 

suggest differences in ratings related to CC contributions or particular activities. 

 

10 For this analysis, the evaluation team used Spearman‟s rank order correlation, as this non-parametric rating is the 

appropriate statistical function to describe correlations between two variables where the values of the variables are 

not normally distributed and are on a scale (such as ratings). 
xxi 

It is also possible that Center technical assistance is viewed as more beneficial by some 

types of participants or that the extent to which participants are engaged in a project affects how 

they regard it. Therefore, the ratings of subgroups of individuals were examined for the relevance 

and usefulness measures. Involvement with the project design and time spent in project activities 

were both associated with statistically higher relevance and usefulness ratings. There was also 

evidence of statistically higher ratings among respondents whose job had a focus on NCLBrelated responsibilities (defined as respondents who spent at least 25 percent of their time on the 

job on NCLB). This suggests that those who worked on the projects and were most likely to 

benefit from the work rated the projects higher.

State Capacity Building and the Use of Different Sources of 

Technical Assistance 

 Capacity building was prominent as a goal for the Comprehensive Centers program. The 

first priority for all Centers, articulated by ED in the Notice Inviting Applications, included 

“helping states build the capacity to help school districts and schools implement NCLB 

provisions and programs.”11

 Fifty-three percent of state managers reported that technical assistance from the program, 

including both their Regional Center and any Content Centers with which they had experience, 

had expanded state capacity to a “great extent” or “very great extent” for building or managing a 

statewide system of support. This was the area of NCLB responsibility in which extensive 

capacity building was most widely reported. In addition, the Centers were the top source used for 

help “to plan the initial steps in solving a problem,” reported by 66 percent of state managers, 

and “to develop the skills of SEA or intermediate education agency staff,” reported by 61 percent 

of state managers. A case study component of the evaluation will examine further the 

contribution of the Comprehensive Centers to building state capacity. 

The Comprehensive Centers were one resource among several available to, and used by, 

state managers. On average, state managers ranked the Centers as one of the top three sources of 

technical assistance that they relied upon, along with professional associations and the EDfunded Regional Educational Laboratories. The Centers were not the resource used most widely 

for “working with districts and schools,” a purpose that ED de-emphasized in the Centers‟ 

charge: colleges and universities were used for this purpose by 37 percent of state managers, and 

consulting firms by the same percentage; the Centers were used for this purpose by 22 percent of 

state managers. 

Summary and Next Steps 

 This evaluation addresses questions about the technical assistance provided by the two 

types of Comprehensive Centers; how the Centers work with their clients; the match between 

client purposes and assistance delivered; and assessments of the quality, relevance, and 

usefulness of a sample of technical assistance projects. This interim report presents findings from 

 

11 Notice Inviting Applications, 32585. 
xxii 

2006-07, the Centers‟ second year of operation under a new design that ED established in 2005. 

The findings suggest that Centers attempted to strike a balance between adhering to their 

management plans for the year and accommodating client requests, and that state clients 

generally viewed Center technical assistance as serving state purposes. In addition, the study 

found that RCCs and CCs worked in the different ways that had been mandated in the design of 

the two types of Centers. 

 In this first round of project ratings, mean ratings across sampled projects and all Centers 

fell in the “moderate” to “high” range for quality (rated by expert panels) and relevance and 

usefulness (rated by participating clients). The CCs had higher mean ratings of technical quality 

for their sampled projects than did RCCs, while the RCCs had higher mean ratings of relevance 

than did CCs. There was no statistically significant relationship between ratings of quality on the 

one hand and relevance or usefulness on the other. These findings suggest that at least in the 

2006-07 program year, achieving high technical quality was unrelated to delivering assistance 

that clients found highly relevant or useful. 

 The evaluation team is continuing to study Center operations, outputs, and outcomes for 

the 2007-08 and 2008-09 program years. By repeating the processes of expert panel reviews and 

surveys, the evaluation team will be able to report on changes over time in the quality, relevance, 

and usefulness of Center projects as well as on trends in state managers‟ perspectives on Center 

technical assistance and contributions to capacity building in SEAs.