NCEE 2010-4033 U.S. DEPARTMENT OF EDUCATION

National Evaluation of the 

Comprehensive Technical 

Assistance Centers

Interim Report
National Evaluation of the 

Comprehensive Technical 

Assistance Centers 

Interim Report 

JULY 2010

Brenda J. Turnbull 

Richard N. White 

Elizabeth Sinclair 

Derek Riley 

Policy Studies Associates, Inc. 

Cynthia L. Sipe 

Branch Associates, Inc. 

Carol Pistorino

Decision Information Resources, Inc. 

Yumiko Sekino 

Project Officer 

Institute of Education Sciences 

NCES 2010-4033

U.S. DEPARTMENT OF EDUCATION

NCEE 2010-4033

U.S. DEPARTMENT OF EDUCATION
U.S. Department of Education

Arne Duncan 

Secretary

Institute of Education Sciences 

John Q. Easton 

Director

July 2010 

This report was prepared for the Institute of Education Sciences under Contract No. ED-04-CO-0028/0001. The 

project officer is Yumiko Sekino in the National Center for Education Evaluation and Regional Assistance. 

IES evaluation reports present objective information on the conditions of implementation and impacts of the 

programs being evaluated. IES evaluation reports do not include conclusion or recommendations or views with 

regard to actions policymakers or practitioners should take in light of the findings in the reports. 

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to 

reprint this publication is not necessary, the citation should be: Turnbull, B.J., White, R.N., Sinclair, E., Riley, D., 

Sipe, C.L., and Pistorino, C. (2010, July). National Evaluation of the Comprehensive Technical Assistance Centers: 

Interim Report (NCEE 2010-4033). U.S. Department of Education, National Center for Education Statistics. 

Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 

Sciences, U.S. Department of Education. 

To order copies of this report, 

 Write to ED Pubs, Education Publications Center, U.S. Department of Education, P.O. Box 22207, 

Alexandria, VA 22304. 

 Call in your request toll free to 1-877-4ED-Pubs. If 877 service is not yet available in your area, call 800-

872-5327. Those who use a telecommunications device for the deaf (TDD) or a teletypewriter (TTY) 

should call 800-437-0833. 

 Fax your request to 703-605-6794 or order online at www.edpubs.gov. 

This report also is available on the IES website at http://ncee.ed.gov. 

Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or computer 

diskette. For more information, please contact the Department ̳s Alternate Format Center at 202-205-8113. 
iii

Acknowledgments

 This report benefited from the work of a colleague whose loss we have felt keenly: 

Cynthia Sipe passed away, much too young, in March 2010. Her professionalism and good 

humor were priceless assets to Branch Associates, Inc., and to this study. 

 We very gratefully acknowledge the cooperation of the Comprehensive Centers as well 

as the many other respondents to our data collection. Center leaders and staff graciously hosted 

site visits, compiled inventories of their projects, and assembled hefty collections of their 

materials for expert review. Senior managers in state education agencies were kind enough to 

respond to the study‘s survey regarding their experiences with Comprehensive Center technical 

assistance, as were the many technical assistance participants housed in state, intermediate, and 

local agencies and other organizations. Expert panel members brought knowledge and care to 

their task of reviewing and rating project materials. 

A Technical Working Group offered useful insights and feedback in two meetings as well 

as informal consultations: the group‘s members were Rolf Blank, Margaret Goertz, Joseph 

Johnson, Michael Petrilli, Gary Ritter, Larry Rudner, and Colleen Seremet. 

In addition to the authors, other members of the participating organizations played 

important roles in study design, data collection, and analysis. Alvia Branch of Branch 

Associates, Inc. (BAI), provided steady leadership for the study team. Lynda Bell, Matthew Coll, 

Barbara Fink, Jennifer Gross, Ceane Rabada, Virginia Smith, and Jennifer Thompson were 

valued team members at BAI. Russell Jackson of Decision Information Resources, Inc. (DIR), 

was a source of wise counsel. Also at DIR, Nancy Dawson, Val Sheppard, Stella Weathersby, 

Frenetta Tate, and Pia White were stalwart contributors. At Policy Studies Associates (PSA), 

team members who made valuable contributions in data collection or coding included Erikson 

Arcaira, Tiffany Miller, Tandra Turner, and Yvonne Woods. Ben Lagueruela provided essential 

help in document production. Katrina Laguarda, formerly of PSA, made tremendous 

contributions to the study design and early activities. 

The study and this report reflect the careful attention of many professionals in the 

Institute of Education Sciences of the U.S. Department of Education. 

v

Disclosure of Potential Conflicts of Interest 

The research team for this evaluation consists of a prime contractor, Branch Associates, 

Inc. and two subcontractors, Policy Studies Associates, Inc. (PSA), and Decision Information 

Resources, Inc. (DIR). None of these organizations or their key staff members has a financial 

interest that could be affected by findings from the evaluation of the Comprehensive Center 

program considered in this report. Additionally, no one on the Technical Working Group, 

convened by the research team to provide advice and guidance, has financial interests that could 

be affected by findings from the evaluation. 

vii 

Contents 

Page 

Executive Summary ...................................................................................................................... vii 

1. Introduction and Background ..................................................................................................... 1

The Center Program .................................................................................................................. 1

Structure of the Comprehensive Center System ....................................................................... 2

Background on the State Role in NCLB ................................................................................... 7

Research Questions Addressed in This Report ......................................................................... 7

Questions for Future Reports .................................................................................................... 8

Organization of This Report ..................................................................................................... 8 

2. Study Design ............................................................................................................................... 9

Identification of a Sample of Center Work for Expert Review and Participant Surveys ....... 11

Describing Center Operations ................................................................................................. 16

Survey of State Managers ....................................................................................................... 18

Assessment of Quality by Expert Panels ................................................................................ 20

Survey of Project Participants ................................................................................................. 24

Units of Analysis for Quality, Relevance, and Usefulness Data in this Report ...................... 28 

3. How the Centers Operate .......................................................................................................... 31

Identifying State Needs and Priorities .................................................................................... 32

Planned and Conducted Center Activities .............................................................................. 36

Knowledge Base Used by Centers .......................................................................................... 44 

4. State Managers‘ Assessment of the Centers‘ Technical Assistance ......................................... 51

Extent to Which Center Assistance Served State Purposes .................................................... 52

Reported Assistance Delivery and Capacity Building for State NCLB Responsibilities ....... 54

State Uses and Perceptions of the Centers Compared with Other Sources of Technical 

Assistance ............................................................................................................................... 57

Overall Rating of Center Assistance ....................................................................................... 63 

5. Variation in the Reported Quality, Relevance, and Usefulness of the Centers‘ Technical 

Assistance ............................................................................................................ 65

Overall Ratings of Center Projects by Participants and Experts ............................................. 69

Variation by Type of Project ................................................................................................... 72

Relationships between the Relevance, Usefulness, and Quality of Center Projects ............... 78

Variation by Participant Experiences and Roles ..................................................................... 79 

Appendix A: Comprehensive Center Lead Grantee and Subgrantee Organizations 

Appendix B: Study Sample 

Appendix C: Describing Center Operations 

Appendix D: Survey of Senior State Managers 

Appendix E: Ratings of Quality, Relevance, and Usefulness

ix

List of Exhibits 

Page 

Exhibit ES.1. Center strategies for planning and ongoing interactions with clients .............. xviii

Exhibit ES.2. Sampled Center projects by types of participant activities and products .......... xix

Exhibit ES.3. Relevance, usefulness, and quality items ........................................................... xxi

Exhibit ES.4. Center Level Mean ratings of technical quality, relevance, and usefulness ...... xxii 

Exhibit ES.5. Mean ratings of technical quality, relevance, and usefulness, by Center ......... xxiii

Exhibit 1.1. Center network design ........................................................................................... 3

Exhibit 1.2. Comprehensive Centers funding in FY 2006 ........................................................ 6

Exhibit 2.1. Data sources for the research questions .............................................................. 10

Exhibit 2.2. Sample size by budget ......................................................................................... 13

Exhibit 2.3. Project sampling process ..................................................................................... 14

Exhibit 2.4. Distribution of all major and moderate projects and projects in the evaluation 

sample, by topic ................................................................................................... 15

Exhibit 2.5. Number of responses to the state manager survey .............................................. 19

Exhibit 2.6. Distribution of discrepancies and results of the discrepancy call process ......... 23

Exhibit 2.7. Survey of project participants sampling and survey administration summary.... 26

Exhibit 2.9. Project sample, by type of rating and unit of analysis ......................................... 29

Exhibit 3.1. RCC strategies for assessing state needs and planning work .............................. 33

Exhibit 3.2. CC strategies for assessing RCC and state needs and planning work ................. 34

Exhibit 3.3. Center strategies for ongoing interactions and negotiations with clients ........... 35

Exhibit 3.4. Formal connections with state clients through RCC staffing .............................. 35

Exhibit 3.5. Center strategies for adapting or adhering to the plan of work ........................... 37

Exhibit 3.6. State priorities for technical assistance and receipt of Center technical assistance, 

by area of NCLB responsibility ........................................................................... 38

Exhibit 3.7. Topics on which Centers stated objectives and/or delivered projects ................. 40

Exhibit 3.8. Topics on which RCCs and CCs stated objectives and/or delivered projects ..... 41

Exhibit 3.9. Sampled Center projects by number of activity or resource types provided ....... 43

Exhibit 3.10. Sampled Center projects by types of participant activities and products ............ 44

Exhibit 3.11. Sources of content expertise used in developing Center products and services . 46

Exhibit 3.12. Sampled RCC projects by CC contribution ........................................................ 47

Exhibit 3.13. Sampled CC projects by RCC contribution ........................................................ 48

Exhibit 3.14. Sources for vetting Center products and services prior to delivery..................... 49

Exhibit 4.1. Extent to which technical assistance from the Centers served state purposes, as 

judged by state managers .................................................................................... 53

Exhibit 4.2. Reasons why Center assistance served some states‘ purposes less 

than completely ................................................................................................... 54
x

List of Exhibits (continued) 

Page 

Exhibit 4.3. State priorities for technical assistance and assistance received from 

Comprehensive Centers ....................................................................................... 56

Exhibit 4.4 Extent to which Comprehensive Center Assistance expanded state 

capacity for NCLB implementation, as judged by senior state managers ........... 59 

Exhibit 4.5. States‘ use of external sources of technical assistance ........................................ 59

Exhibit 4.6. Purposes for which states used external sources of technical assistance ............ 61

Exhibit 4.7. Usefulness of Comprehensive Center assistance compared with assistance 

from other sources ............................................................................................... 62

Exhibit 4.8. Relevance and usefulness items from the state manager surveys ....................... 63

Exhibit 4.9. Mean ratings of the relevance and usefulness of Center assistance, 

as judged by state managers ................................................................................ 64

Exhibit 4.10. Breakout of ratings of the relevance and usefulness of Center assistance, 

as judged by state managers ................................................................................ 64

Exhibit 5.1. Quality, relevance, and usefulness items ............................................................. 66

Exhibit 5.2. Scales for rating of quality by expert panels and relevance 

and usefulness by participants ............................................................................. 67

Exhibit 5.3. Center Level Mean ratings of technical quality, relevance, and usefulness ........ 70

Exhibit 5.4. Mean ratings of technical quality, relevance, and usefulness, by Center ........... 71

Exhibit 5.5. Distribution of technical quality, relevance, and usefulness 

ratings – percent of projects ................................................................................ 72

Exhibit 5.6. Technical quality, relevance, and usefulness ratings, by projects 

rated ―high‖ and projects rated ―low‖ and Center type ....................................... 73

Exhibit 5.7. Technical quality, relevance, and usefulness ratings, by project size ................. 74

Exhibit 5.8. Technical quality, relevance, and usefulness ratings of projects, 

by type of activity offered in the projects ............................................................ 76

Exhibit 5.9. Technical quality, relevance, and usefulness ratings by Content Center 

contribution to RCC projects ............................................................................... 77

Exhibit 5.10. Correlations between project-level technical quality, relevance, 

and usefulness ratings .......................................................................................... 79

Exhibit 5.11. Participants‘ involvement in determining project goals or design, 

by type of Center ................................................................................................. 81

Exhibit 5.12. Participants‘ involvement in determining project goals or design, 

by type of participant organization ...................................................................... 81

Exhibit 5.13. Mean relevance and usefulness ratings, by respondent involvement in 

determining project goals or design .................................................................... 82

Exhibit 5.14. Mean relevance and usefulness ratings, by time respondents spent 

in each activity ..................................................................................................... 84

Exhibit 5.15. Where participants in RCC and CC projects were employed in 2006-07 .......... 85
xi 

List of Exhibits (continued) 

Page 

Exhibit 5.16. Mean relevance and usefulness ratings for RCC projects, by agency 

where respondent worked .................................................................................... 86

Exhibit 5.17. Relevance and usefulness ratings for CC projects, 

RCC staff vs. state-level staff .............................................................................. 87

Exhibit 5.18. Mean relevance and usefulness ratings for CC projects, 

by RCC staff role of respondent .......................................................................... 88

Exhibit 5.19. Mean relevance and usefulness ratings, by percent of time 

respondent spent on NCLB-related tasks in his/her job ...................................... 89

xiii 

Executive Summary 

 This first of two reports presents early findings from the National Evaluation of the 

Comprehensive Technical Assistance Centers (Comprehensive Centers), a federally funded 

program that provides technical assistance to states in connection with the No Child Left Behind 

(NCLB) Act of 2001. The law authorizing the Comprehensive Centers, the Educational 

Technical Assistance Act of 2002, mandated that a national evaluation of the program be 

conducted by the Institute of Education Sciences (IES). The legislation indicated that the 

evaluation should ―include an analysis of the services provided...[and] the extent to which each 

of the comprehensive centers meets the objectives of its respective plan, and whether such 

services meet the educational needs of State educational agencies, local educational agencies, 

and schools in the region.‖ The program evaluation is conducted by Branch Associates, Inc., 

Decision Information Resources, Inc., and Policy Studies Associates, Inc. 

This report addresses the first of the evaluation‘s three rounds of data collection, 

pertaining to the Centers‘ work of July 2006 through June 2007. It describes the program design 

and, drawing upon data provided by the Centers and their clients, program operations. It also 

describes assessments of Center activities and resources, reporting on quality as judged by panels 

of subject-matter experts, and on relevance, usefulness, and contributions to capacity as judged 

by practitioners (namely, state-level managers and also clients who participated directly in 

Center activities or received Center products). A final report will provide parallel findings for 

2007-08 and 2008-09. In addition, it will present findings from case studies of capacity building 

at the state level and any changes in findings over time. 

The main findings from the evaluation so far are: 

■ The Comprehensive Centers reported planning their work in coordination and 

consultation with their clients with the work evolving during the year. All 16 

Regional Comprehensive Centers (RCCs) reported obtaining state input into their 

initial plans and engaging states in refinements to the plans through ongoing 

interaction and negotiation. Similarly, all five Content Centers (CCs) reported 

forming their work plans incorporating RCC input acquired through either RCC 

staff surveys or direct communication. In addition, all five CCs described working 

with the U.S. Department of Education (ED) to learn of specific topics and tasks 

needed to advance ED priorities. A review of projects conducted indicates that both 

RCCs and CCs adjusted their work plans during the year. 

■ The technical assistance activities were varied and consisted of ongoing 

consultation, research syntheses, planning of technical assistance with 

participants, training events, conferences, and support for task forces or for 

development of formal plans. Consistent with the mission of ―front-line‖ 

assistance, the majority of sampled RCC projects involved ongoing consultation and 

follow-up (82 percent). CC assistance most often focused on the delivery of research 

information, consistent with the CCs‘ prescribed focus on synthesizing, translating, 

and delivering knowledge on a particular topic. The delivery of research collections 

and syntheses occurred in 74 percent of the CC sampled projects. 
xiv 

■ The Comprehensive Centers program delivered technical assistance that, 

according to state managers: (1) served state education agencies’ (SEAs’) 

purposes in seeking technical assistance, (2) was aligned with SEAs’ 

priorities for NCLB-related technical assistance, and (3) was perceived to 

expand SEA capacity. Eighty-eight percent of state managers rated the technical 

assistance they received from Centers as at least ―a good start‖ in serving their 

purposes, and 36 percent overall reported that it ―served the state‘s purposes 

completely.‖ For each of the four areas of NCLB implementation most widely 

identified as state priorities for technical assistance, at least 90 percent of those 

state managers who had identified the area as a priority had received assistance 

with it from the Centers. Overall, more than two-thirds of state managers (68 

percent) reported a perception that assistance from the Comprehensive Centers 

had greatly expanded their state‘s capacity to carry out its responsibilities in at 

least one NCLB area. 

■ Center projects in the evaluation sample were judged by clients to be on 

average in the “moderate” to “high” range of relevance and usefulness; 

panels of experts judged their quality to be in the “moderate” range on 

average. On a scale of 1 to 5 with a 3 representing ―moderate‖ and a 4 

representing ―high,‖ the programwide average ratings for the sampled projects 

were 3.34 for technical quality (scored by panels of content experts), and 3.94 for 

relevance and 3.70 for usefulness (scored by participants).1

The average quality 

rating was higher among CCs than RCCs by more than one-half of a standard 

deviation; the average relevance rating was higher among RCCs than CCs by at 

least one-half of a standard deviation; usefulness ratings were similar between the 

two Center types (i.e., did not differ by at least one-half of a standard deviation).2

 

1

 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 

to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 

and each project sampled from a Center contributed equally to the Center mean. 

2

 Using Cohen (1988) as a conceptual framework, we estimated Cohen's d (an estimate of the effect size defined as 

the difference in means divided by the pooled standard deviation) and adopted the logic of Cohen for what is a 

moderate difference. Specifically, we adopted a difference in the means of one-half of one standard deviation 

(analogous to an effect size of .5) as our minimum threshold for highlighting differences. The ―pooled standard 

deviation‖ for each computation varied with the unit of analysis. For analyses conducted at the Center level, the 

pooled standard deviation was computed as the standard deviation of the variable of interest (e.g., relevance) 

computed at the Center level. For analyses using the project as the unit of analysis, the pooled standard deviation 

was computed at the project level. 
xv

The Comprehensive Centers Program 

 In its authorization, the Comprehensive Centers program was given an overall charge of 

supporting state and local NCLB implementation. ED, using discretion provided in the 

legislation, established two major program features that differed from past Comprehensive 

Centers programs:3

■ First, the primary focus would be on assisting states to expand and strengthen 

states‘ capacity to deliver assistance to schools and districts; ED specified that 

Centers could only work directly with districts or schools under special 

circumstances. 

■ Second, awards would be made in two tiers, to 16 RCCs and 5 CCs. They were 

instructed to work as follows: 

■ Each RCC was charged with providing ―frontline assistance‖ either to one 

large state or to a group of two to eight states and other jurisdictions.4The 

RCCs were also expected to deliver technical assistance to their assigned 

states, addressing the needs and building capacity of the states to assist 

their districts and schools. 

■ Meanwhile, each CC would work on a nationwide basis within a particular 

substantive area: Assessment and Accountability, Instruction, Teacher 

Quality, Innovation and Improvement, or High Schools. CCs would 

facilitate access to, and use of, existing research and practices. 

■ The absolute priorities for the two types of Centers indicated that they 

should work together: Regional Centers should draw information and 

resources from Content Centers as well as other sources; and Content 

Centers should both supply knowledge to Regional Centers and ―work 

closely with Regional Centers to provide technical assistance to States.‖

Evaluation Topics and Methods 

The research priorities for the evaluation were primarily driven by the statute and focused 

on the following key research questions: 

■ What are the objectives of the Comprehensive Center network and of each Center? 

 

3

 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 

32583-94. 

4

 The nonstate jurisdictions that the Centers were to serve were the following: the District of Columbia, Puerto Rico, 

the Virgin Islands, American Samoa, Commonwealth of the Northern Mariana Islands, Federated States of 

Micronesia [Chuuk, Kosrae, Pohnpei, and Yap], Guam, Republic of the Marshall Islands, and Republic of Palau. 

Throughout this report, the term ―state‖ will be defined to include the 50 states as well as these other jurisdictions. 
xvi 

■ What kinds of products and services are provided by the Comprehensive Center 

network and by each Center? 

■ How do Centers develop, refine, and carry out their plans for technical assistance? 

How do they define their clients‘ educational needs and priorities? How do Center 

clients (states or Regional Centers) define their needs and priorities? 

■ To what extent is the work of each Comprehensive Center of high quality, high 

relevance, and high usefulness?

■ To what extent do states report that Center projects have expanded state capacity 

to address underlying needs and priorities and meet the goals of NCLB? 

■ To what extent have states relied on other sources of technical assistance besides 

the Comprehensive Centers? What other sources? How does the usefulness of 

Center projects compare with the usefulness of projects from other sources? 

 The evaluation gathered annual information from six data sources in order to address the 

research questions above. Data collection included: 

■ Management plans. The evaluation reviewed these as a data source for each 

Center‘s intended focus at the beginning of the year, drawing from the plans a list 

of topics as foci of Center objectives. 

■ Center staff interviews. Using structured response categories, Center staff were 

asked about how they planned their programs of work; how their plans evolved 

during the program year; and what they offered to clients with respect to the 

topics addressed, the delivery modes used, and their sources for content expertise. 

(See appendix C for the protocols and other structured response materials used 

during the interviews). 

■ Survey of senior state managers. SEA managers were surveyed about their 

state‘s technical-assistance needs and what the Centers (including their RCC and 

the CCs) had provided. 

■ Project inventory forms. The evaluation team assisted each Center in grouping 

related activities and deliverables into ―projects,‖ with the project defined as a 

group of closely related activities and/or deliverables designed to achieve a 

specific outcome for a specific audience. Projects were in turn classified by the 

Centers into major, moderate, and minor projects on the basis of the relative level 

of effort they reflected. The Centers and the evaluation team also classified the 

projects, according to the topics addressed, into 22 topical categories.5

 

5

The 22 topics were: components of effective systems of support for states, districts, and schools; data use or datadriven decision making; formative assessment; reading; adolescent literacy; mathematics; dropout prevention; high 

school redesign or reform; transition to high school; special education curriculum, instruction and professional 

development; special education assessment; English language learners;‖ highly qualified teacher‖ provisions of 

NCLB; teacher preparation and induction; teacher professional development; supplemental educational services; 

Response to Intervention; migrant education; Indian or Native American education; data management and 
xvii 

■ Survey of project participants. A representative sample of clients who had 

participated directly in the evaluation‘s purposive sample of major and moderate 

Center projects furnished descriptive information, through surveys, on the 

technical-assistance needs of their offices and on the activities and resources that 

the project had delivered to them. These clients included individuals working at 

the state level who had participated in RCC or CC projects and RCC employees 

who were among the clients of CC projects. They also rated the relevance and 

usefulness of the sampled projects. 

■ Expert panel review. The same sample of major and moderate projects was

reviewed for quality by a panel of experts. Content experts were recruited and 

trained to use standard criteria to rate the technical quality of the sampled Center 

projects on the basis of a review of all project materials. 

Operation of Centers 

Before the beginning of the 2006-07 program year, each Center was required to submit a 

management plan, setting out objectives and planned activities for the coming year, for ED 

review and approval. Almost all Centers (20 of 21) reported that client input was used in writing 

the plan. The same number reported conducting needs assessment through meetings or other 

communication with clients. 

 In addition to initial needs assessments, to ultimately meet client needs Centers used 

planning and ongoing interactions with each other and with the SEAs to refine their needs 

assessments as indicated in exhibit ES.1. Fifteen of 16 RCCs reported forming work groups 

within state organizations that brought together staff from multiple departments to discuss service 

needs and delivery; the remaining RCC was 1 of 14 that reported working directly with the chief 

state school officer in their initial planning. Half of the RCCs (8) formed cross-agency work 

groups to discuss SEA service needs and delivery. All five CCs identified needs of their client 

RCCs primarily by conducting conference calls with designated RCC representatives, and three of 

the five CCs maintained communication about needs by forming workgroups that included RCC 

representatives. In addition, all CCs reported providing either large-group events or support of 

existing RCC programs and projects as additional ways to learn about and meet client needs. 

 

compliance; assessment design; and parent involvement. In addition, projects that addressed none of these 22 topics 

were categorized as ―other.‖
xviii 

Exhibit ES.1. Center strategies for planning and ongoing interactions with 

clients

Center strategy

RCCs

(N=16)

CCs

(N=5)

All

(N=21)

Assess RCC needs through meetings or other communication 

with RCC staff 5 -

Included SEA input when writing annual management plans 16 - -

Sponsor large events to make contact with many clients 13 4 17

Form work groups within client organizations that bring together 

staff from multiple departments/divisions to discuss service 

needs and delivery

15 1 16

Offer service to support existing client programs/ projects/ 

policies 9 4 13

Form work groups across client organizations to discuss service 

needs and delivery (e.g., different SEAs and/or RCCs) 8 3 11

EXHIBIT READS: All five CCs assessed RCC needs through meetings or other communication with RCC 

staff. 

SOURCE: Verbatim summaries of Center interviews conducted during 2007 site visits, coded by evaluation 

team, with Center review of coding results 

Although Centers reported providing technical assistance in a majority (80 percent) of the 

instances where identified topic areas were included in their management plans, Centers 

commonly adjusted the topic areas in which they conducted work, adding work in some areas 

and shifting away from work in other areas. Based on a review of each Center‘s inventory of 

projects against the topics that had been included in that Center‘s management-plan objectives, 

there were instances of the Center carrying out work on a topic not initially cited in its 

objectives. That is, in 19 of 22 topic areas Centers reported delivering technical assistance that 

was not in their original management plan. The most common shift toward topics were in the 

areas of Response to Intervention (seven Centers conducted work in this area that was not 

planned), English language learner issues (five Centers), highly qualified teacher provisions of 

NCLB (five Centers) and supplemental educational services (five Centers). By the same token, 

Centers appeared to delete work in particular topic areas such as special education curriculum, 

instruction, and professional development (five Centers appeared to delete planned work); and 

data use or data-driven decisionmaking (four Centers appeared to delete planned work). Centers 

that set an objective in a topic did not report a project on the topic in their inventory for 10 of the 

22 topic areas. The most common topic area for projects was that of statewide systems of support 

for educational improvement. 

Regardless of whether they were originally specified in work plans or added later, an 

analysis of the projects that the evaluation team sampled for closer study across all Centers 

provides more in-depth information about the nature of Center technical assistance activities or 

resources. Although the sample of projects is not statistically representative of the Centers‘ work, 

the process of sample selection favored each Center‘s most dominant projects and included over 

half (56 percent) of that year‘s designated major or moderate projects. Most projects (84 percent) 

used more than one mode of delivery from a list that included conferences, training, delivery of 

research collections or syntheses, support for a task force, support for development of a plan or 
xix 

policy, engagement of clients in project planning, and ongoing consultation and follow-up. 

Across all the sampled projects and also across the Regional Centers‘ sampled projects, the most 

frequent modes of delivery as shown in exhibit ES.2 were ongoing consultation and follow-up 

(84 of 122, or 69 percent of all projects and 79 of 96, or 82 percent of Regional Center projects) 

or delivery of a research collection or synthesis (71 of 122, or 58 percent of all projects and 52 of 

96, or 54 percent of Regional Center projects). The Content Centers‘ projects most often 

included delivery of a research collection or synthesis (20 of 27, or 74 percent of their projects) 

or a conference (17 of 27, or 63 percent). 

While some projects were worked on by both the RCCs and the CCs, coordination 

between CCs and RCCs when it did occur was asymmetrical. In providing assistance to states, 

RCCs used CC input more than CCs used substantive RCC input. Almost half of the sampled 

RCC projects had a substantive CC contribution (such as a product or a presentation by a CC 

staff member); in contrast, in 11 percent of the sampled CC projects an RCC contributed content 

or delivered assistance. More often (in 37 percent of the sampled CC projects), the CC enlisted 

the help of one or more RCCs to identify and recruit participants. 

Exhibit ES.2. Sampled Center projects by types of participant activities 

and products 

Activities and products

(with clarifying definitions used by coders)

RCC 

projects

(n=96)

CC 

projects

(n=27)

All 

projects

(n=122)

Ongoing consultation and follow-up (multiple contacts to same 

participants, that were part of a coherent and purposeful whole) 79 6 84

Research collections and syntheses 52 20 71

Engagement of participants in project planning (more than needs 

assessment or identifying participants) 43 8 50

Training events (focused on implementing a specific program or 

strategy) 41 10 50

Task force meetings and work (focused on addressing a specific 

problem, program, or policy) 48 2 50

Conferences (symposium, forum, institute; highlights a range of 

perspectives, strategies, or programs) 26 17 43

Support development of a formal plan to implement a program or 

policy 18 2 20

EXHIBIT READS: Seventy-nine RCC projects included ongoing consultation and follow-up. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials; cover sheets 

coded by evaluation team. The total number of projects was 122. One project collaboratively conducted by an 

RCC and a CC was counted among both RCC projects and CC projects but was only counted once among the 

projects of all Centers. 
xx

The types of work emphasized in the RCC and CC sampled projects were consistent with 

their different charges. RCC assistance more often incorporated sustained interaction with 

participants: again, the majority of RCC projects in the sample involved ongoing consultation 

and follow-up (82 percent), whereas this was less so for the work of the CCs (22 percent of

projects in the sample). Thus, this pattern of RCC activities was consistent with the mission of 

―front-line‖ assistance that would take clients‘ purposes and circumstances into account and 

provide ongoing support for their implementation of NCLB. For the CCs, the assistance more 

often focused on the delivery of research information, consistent with the CCs‘ prescribed focus 

on synthesizing, translating, and delivering knowledge on a particular topic. The delivery of 

research collections and syntheses occurred in 74 percent of the CC projects but 54 percent of the 

RCC projects. The sampled CC projects more often delivered technical assistance through 

conferences (63 percent of the CC projects but 27 percent of the RCC projects). 

Ratings of Center Assistance 

The sampled projects, all identified by the Centers as ―major‖ and ―moderate,‖ were rated 

in order to assess the services provided by the Comprehensive Centers program. Each project 

was evaluated for relevance and usefulness by a sample of participants—state staff, intermediate 

agency staff, local educators working on behalf of the state, and RCC staff—who were the 

intended beneficiaries of the project and who had received at least some of the technical 

assistance it provided. Ratings of project quality were gathered from panels of experts with 

strong knowledge of the content or substantive focus of the specific projects they reviewed. 

Relevance was assessed with eight survey items and usefulness with 11 items; quality was 

judged on three items called dimensions (exhibit ES.3). Each overall measure (relevance, 

usefulness, or quality) was calculated as the mean of ratings assigned to each item. The itemlevel ratings themselves were based on 5-point rating scales.6

 

6

 Efforts were made to develop parallel wording and rubrics that would result in similar gradations between rating 

levels (e.g., very high vs. high vs. moderate) across the three measures. However, given the different content of each 

set of items within the three measures and the different contexts for the ratings (experts who underwent training for 

the rating process and reviewed identical packages of materials vs. survey respondents who typically participated in 

different subsets of project activities), the ratings across the three measures are not directly comparable. 
xxi 

Exhibit ES.3. Relevance, usefulness, and quality items 

From expert panel scoring From project participant surveys

Technical quality Relevance Usefulness

Reviewers were directed to 

assign a score to each 

dimension and to include the 

basis for their ratings on the 

rating form, including the specific 

artifacts on which their score was 

based. The three dimensions 

are:

a. Demonstrated use of the 

appropriate documented 

knowledge base – to include 

an accurate portrayal of the 

current state of information 

with prominence to those 

with the most 

accurate/rigorous evidence 

b. Fidelity of application of the 

knowledge base to the 

products and services 

provided – materials are 

consistent with the 

best/accurate information 

available and the

presentation adequately 

conveys the confidence of 

the information

c. Clear and effective delivery –

information is well organized 

and written and accessible to 

the intended audience for 

easy use 

Based on your experience, to 

what degree was this set of 

activities and resources relevant

to your work, in each of the 

following respects?

a. Addressed a need or problem 

that my organization faces

b. Addressed an important 

priority of my organization

c. Addressed a challenge that 

my organization faces related 

to the implementation of NCLB

d. Provided information, advice, 

and/or resources that could be 

directly applied to my 

organization’s work

e. Addressed our particular state 

context

f. Addressed my organization’s 

specific challenges (e.g., 

policy environment, leadership 

capacity, budget pressures, 

local politics) 

g. Provided information, advice, 

and/or resources that could be 

used to guide decisions about 

policies, programs, or 

practices

h. Highlighted the implications of 

research findings (or 

information about best 

practice) for policies, 

programs, or practices

Based on your experience, to 

what degree was this set of 

activities and resources useful to 

you, in each of the following 

respects?

a. Provided resources that were 

easy to understand and easy 

to use

b. Employed an appropriate 

format (e.g., a work group, a 

conference, individual 

consultation, written products)

c. Provided adequate opportunity 

to learn from colleagues in 

other states

d. Included adequate follow-up to 

support the use of new 

information and resources

e. Were timely

f. Helped my organization solve 

a problem

g. Helped my organization 

maintain or change a policy or 

practice

h. Helped my organization take 

the next step in a longer-term 

improvement effort

i. Provided my organization with 

information or resources that 

we will use again

j. Helped my organization 

develop a shared expertise or 

knowledge-base

k. Helped individuals in my 

organization to develop skills 

that they will use again

Based on the ratings, Center technical assistance was judged to be in the ―moderate‖ to 

―high‖ range of quality, relevance, and usefulness. On a scale of 1 to 5 with a 3 representing 

―moderate‖ and a 4 representing ―high,‖ the programwide average ratings for the sampled 

projects were 3.34 for technical quality (scored by panels of content experts), and 3.94 for 

relevance and 3.70 for usefulness (scored by participants) as indicated in exhibit ES.4.7 

 

7

 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 

to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 

and each project sampled from a Center contributed equally to the Center mean. 
xxii 

Exhibit ES.4. Center Level Mean ratings of technical quality, relevance, and 

usefulness 

Technical 

quality Relevance Usefulness

All Comprehensive 

Centers (N=21) 3.34 3.94 3.70

All RCCs (N=16) 3.21 3.99 3.71

All CCs (N=5) 3.73 3.78 3.65

Difference of RCC and 

CC means -0.52†

0.21† 0.06

Pooled standard deviation

(all Comprehensive 

Centers)

0.41 0.34 0.34

Ratio of difference in means 

to pooled standard deviation -1.28 0.60 0.18

NOTE: All ratings were on a 5-point scale, with 5 as the high value. The ―technical quality‖ rating is the mean of the 

ratings for the three quality dimensions. A notation of † indicates that the difference in the mean ratings between the 

CCs and RCCs is at least one-half of one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.34. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 

ratings; each project contributed equally to Center ratings; and each Center contributed equally to cross-Center 

ratings. 

 

Given that the RCC and CC roles and activity emphasis differed, the evaluation looked at 

variation across Center types and projects that might provide information for program 

improvement. The average quality rating was higher among CCs than RCCs by more than onehalf of a standard deviation8

; the average relevance rating was higher among RCCs than CCs by 

at least one-half of a standard deviation; usefulness ratings were similar between the two Center 

types (differing by less than one-half of a standard deviation). The Content Centers received 

Center-level mean scores for technical quality that averaged 3.73, compared with 3.21 for the 

Regional Centers; the difference of 0.52 points exceeded one-half of one pooled standard 

deviation. The mean scores for relevance were 3.99 for the Regional Centers and 3.78 for the 

Content Centers. On usefulness, the mean score of 3.71 for the RCCs and 3.65 for the CCs were 

within one-half of a standard deviation of each other. 

There was variation in the ratings across and within individual Centers. On each measure, 

at least 11 Centers had a mean rating that was at least one-half of a standard deviation above or 

below the overall mean for its type of Center (RCC or CC) for that measure9(i.e., 11 of 21 

 

8

 For analyses conducted at the Center level, the pooled standard deviation was computed as the standard deviation 

of the variable of interest (e.g., relevance) computed at the Center level. 

9

 For analyses using the project as the unit of analysis, the pooled standard deviation was computed at the project 

level. 
xxiii 

Centers were this far above or below the mean for quality, 11 for relevance, and 14 for 

usefulness). One RCC was rated higher than others by at least one-half of a standard deviation on 

all three measures, and one CC and one RCC were rated lower than others on all three measures. 

Aside from these Centers, the other 18 Centers‘ ratings were not consistently higher or lower 

than the mean but varied across measures (Exhibit ES-5). 

Exhibit ES.5. Mean ratings of technical quality, relevance, and usefulness, by

Center 

Center type

Technical 

quality Relevance Usefulness

RCCs (N=16) 3.78 3.78 3.42

3.63 3.22 3.00

3.46 4.18 3.94

3.44 3.90 3.63

3.36 3.97 3.63

3.35 3.97 3.57

3.35 3.93 3.51

3.31 4.15 3.69

3.21 4.08 3.82

3.17 4.31 4.05

3.15 4.12 3.93

3.11 4.70 4.46

2.98 4.07 3.92

2.74 3.20 3.05

2.74 4.01 3.54

2.63 4.18 4.17

Average RCC rating 3.21 3.99 3.71

Pooled standard 

deviation (RCCs) 0.32 0.37 0.38

CCs (N=5) 4.24 3.76 3.54

3.94 3.90 3.86

3.88 3.99 3.84

3.44 3.58 3.44

3.14 3.68 3.56

Average CC rating 3.73 3.78 3.65

Pooled standard 

deviation (CCs) 0.43 0.16 0.19

NOTE: The arrow pointing upward indicates the accompanying value is at least one-half of one standard 

deviation above the group mean (e.g., 3.78 is at least one-half of one standard deviation above the mean 

for the RCCs). The arrow pointing downward indicates the accompanying value is at least one-half of 

one standard deviation below the group mean. 

EXHIBIT READS: One of the RCCs had a mean rating for technical quality of 3.78, a mean rating for 

relevance of 3.78, and a mean rating for usefulness of 3.42, across the projects sampled from that 

Center. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project 

participants for relevance and usefulness. Responses weighted so that each panelist or participant 

contributed equally to project ratings, and each project contributed equally to Center ratings 
xxiv 

The evaluation also looked at the relationship between the three measures: quality, 

relevance, and usefulness. It was reasoned that the content experts rating quality and the 

participants rating relevance and usefulness might value and be better able to judge different 

qualities in a Center project, which is why we did not have content experts evaluate the projects 

for their utility or the participants assess the technical quality. An examination of the associations 

among the three dimensions was conducted by calculating correlation coefficients.10 Such a 

statistic indicates the strength and direction of a linear relationship between two factors. A 

correlation coefficient can vary from positive one (indicating a perfect positive relationship), 

through zero (indicating the absence of a relationship), to negative one (indicating a perfect 

negative relationship). If the correlation is statistically significant (p <.05), we can have strong 

(95 percent) confidence that what we calculated is not due to chance. 

Ratings of quality were unrelated to ratings of relevance and usefulness, although 

relevance and usefulness ratings were highly correlated with each other. The correlation 

coefficient for relevance and usefulness was +0.84, while the coefficient of relevance with 

quality was -0.12, and the coefficient of usefulness and quality was -0.04. In other words, the 

extent to which a project faithfully reflected the knowledge base on a topic and provided 

appropriate caveats about the quality of its evidence was unrelated to the extent to which 

participants deemed that project relevant or useful to their agency. 

Given the variation in ratings across projects, additional analyses of project 

characteristics were conducted to explore whether there were any consistent patterns between 

ratings and the particular features of the projects. Such information may provide suggestions for 

possible program improvement. Specifically, if there is a consistent relationship between scale of 

the undertaking and the ratings, perhaps signaling more ambitious projects or projects that allow 

a greater focus of Center resources on the effort, then this might be suggestive of productive uses 

of Center resources for future emphasis. In fact, projects identified by the Centers as ―major‖

were rated higher by at least one-half a standard deviation on the measures of relevance and 

usefulness but not on the measure of quality. 

Projects with particular types of activities may be easier to carry out, may play to Center 

strengths, or may be seen as more productive to the ultimate clients. In addition, those RCC 

projects that included CC contributions might be expected to have higher quality ratings than 

other RCC projects, given the expected content and research focus of the CCs. Thus, the 

evaluation compared ratings of subgroups of projects (e.g., those with and without particular 

activities such as conferences, training, or research syntheses; RCC projects with and without CC 

involvement) to see if there were any consistent relationships between the ratings and particular 

Center activities or the incorporation of CC work. Across five of the seven project activities 

identified, comparisons of projects with and without the activities showed no differences in 

quality, relevance, or usefulness greater than one-half of a standard deviation. Ratings of RCC 

projects with CC contributions did not differ by more than one-half of a standard deviation on 

any measures, compared with those without CC contributions. Thus, these analyses do not 

suggest differences in ratings related to CC contributions or particular activities. 

 

10 For this analysis, the evaluation team used Spearman‘s rank order correlation, as this non-parametric rating is the 

appropriate statistical function to describe correlations between two variables where the values of the variables are 

not normally distributed and are on a scale (such as ratings). 
xxv

It is also possible that Center technical assistance is viewed as more beneficial by some 

types of participants or that the extent to which participants are engaged in a project affects how 

they regard it. Therefore, the ratings of subgroups of individuals were examined for the relevance 

and usefulness measures. Involvement with the project design and time spent in project activities 

were both associated with statistically higher relevance and usefulness ratings. There was also 

evidence of statistically higher ratings among respondents whose job had a focus on NCLBrelated responsibilities (defined as respondents who spent at least 25 percent of their time on the 

job on NCLB). This suggests that those who worked on the projects and were most likely to 

benefit from the work rated the projects higher.

State Capacity Building and the Use of Different Sources of 

Technical Assistance 

 Capacity building was prominent as a goal for the Comprehensive Centers program. The 

first priority for all Centers, articulated by ED in the Notice Inviting Applications, included 

―helping states build the capacity to help school districts and schools implement NCLB 

provisions and programs.‖11

 Fifty-three percent of state managers reported that technical assistance from the program, 

including both their Regional Center and any Content Centers with which they had experience, 

had expanded state capacity to a ―great extent‖ or ―very great extent‖ for building or managing a 

statewide system of support. This was the area of NCLB responsibility in which extensive 

capacity building was most widely reported. In addition, the Centers were the top source used for 

help ―to plan the initial steps in solving a problem,‖ reported by 66 percent of state managers, 

and ―to develop the skills of SEA or intermediate education agency staff,‖ reported by 61 percent 

of state managers. A case study component of the evaluation will examine further the 

contribution of the Comprehensive Centers to building state capacity. 

 The Comprehensive Centers were one resource among several available to, and used by, 

state managers. On average, state managers ranked the Centers as one of the top three sources of 

technical assistance that they relied upon, along with professional associations and the EDfunded Regional Educational Laboratories. The Centers were not the resource used most widely 

for ―working with districts and schools,‖ a purpose that ED de-emphasized in the Centers‘ 

charge: colleges and universities were used for this purpose by 37 percent of state managers, and 

consulting firms by the same percentage; the Centers were used for this purpose by 22 percent of 

state managers. 

Summary and Next Steps 

 This evaluation addresses questions about the technical assistance provided by the two 

types of Comprehensive Centers; how the Centers work with their clients; the match between 

client purposes and assistance delivered; and assessments of the quality, relevance, and 

usefulness of a sample of technical assistance projects. This interim report presents findings from 

 

11 Notice Inviting Applications, 32585. 
xxvi 

2006-07, the Centers‘ second year of operation under a new design that ED established in 2005. 

The findings suggest that Centers attempted to strike a balance between adhering to their 

management plans for the year and accommodating client requests, and that state clients 

generally viewed Center technical assistance as serving state purposes. In addition, the study 

found that RCCs and CCs worked in the different ways that had been mandated in the design of 

the two types of Centers. 

 In this first round of project ratings, mean ratings across sampled projects and all Centers 

fell in the ―moderate‖ to ―high‖ range for quality (rated by expert panels) and relevance and 

usefulness (rated by participating clients). The CCs had higher mean ratings of technical quality 

for their sampled projects than did RCCs, while the RCCs had higher mean ratings of relevance 

than did CCs. There was no statistically significant relationship between ratings of quality on the 

one hand and relevance or usefulness on the other. These findings suggest that at least in the 

2006-07 program year, achieving high technical quality was unrelated to delivering assistance 

that clients found highly relevant or useful. 

 The evaluation team is continuing to study Center operations, outputs, and outcomes for 

the 2007-08 and 2008-09 program years. By repeating the processes of expert panel reviews and 

surveys, the evaluation team will be able to report on changes over time in the quality, relevance, 

and usefulness of Center projects as well as on trends in state managers‘ perspectives on Center 

technical assistance and contributions to capacity building in SEAs. 
1

1. Introduction and Background 

The Educational Technical Assistance Act of 2002 authorized the Secretary of Education 

to award ―not less than 20 grants to local entities, or consortia of such entities, with demonstrated 

expertise in providing technical assistance and professional development in reading, 

mathematics, science, and technology‖ (Section 203). The law mandated that these entities, 

called Comprehensive Technical Assistance Centers, undergo independent evaluation under the 

direction of the National Center for Education Evaluation and Regional Assistance (NCEE) in 

the Institute of Education Sciences of the U.S. Department of Education (ED). 

This is the first interim report of a multi-year evaluation conducted for NCEE by Branch 

Associates, Inc., Decision Information Resources, Inc., and Policy Studies Associates, Inc. The 

report addresses the second year of operation of the Center system, July 2006-June 2007. This 

introductory chapter provides background information on ED‘s purposes and design for the 

Center system. It also describes key state responsibilities included in the No Child Left Behind 

(NCLB) Act with which the Centers were expected to provide assistance. The chapter concludes 

with a description of the major evaluation questions. The study methods are described in the 

second chapter of this report, followed by the findings in successive chapters. 

The Center Program 

The current Centers replaced 15 existing Comprehensive Regional Assistance Centers 

established under the Elementary and Secondary Education Act of 1994 (ESEA). The current 

system represented a departure from that previous system with respect to the targeting of services 

and the responsibilities of Centers. 

The 1994 reauthorization of ESEA had charged the previous set of 15 Comprehensive 

Centers with delivering assistance to support standards-based reform as envisioned in other 

sections of the 1994 law. They were to provide training and technical assistance to states, local 

education agencies (LEAs), schools, tribes, community-based organizations, and other ESEA 

grantees related to several areas of local responsibility. These included: (1) improving the quality 

of instruction, curricula, assessments, and other aspects of school reform; (2) implementing 

effective schoolwide programs; and (3) meeting the needs of children, especially children in 

high-poverty areas, migrant children, immigrant children, limited-English-proficient children 

(LEP), neglected or delinquent children, homeless children, Indian children, and children with 

disabilities [Section 13102 (a)(1)(A-L)]. In short, the previous Centers‘ mandate included a focus 

on a number of aspects of local educational practice. 

A previous evaluation of the Comprehensive Centers,1based on surveys of clients 

conducted in 1999, found that the majority of direct participants in major Center training and 

 

1

 U.S. Department of Education, Office of the Under Secretary, Planning and Evaluation Service, Elementary and 

Secondary Education Division. (2000). Comprehensive Regional Assistance Centers Program: Final Report on the 

Evaluation(Volume I). Washington, DC: Author. 
2

technical assistance activities were school staff (either teachers or principals), and that the majority 

of clients who arranged for services from the Comprehensive Centers were also based in school 

districts or schools. At the state level, among those state education agency (SEA) staff members 

who had received assistance, 64 percent reported that the Centers had improved the ability of their 

SEA to provide assistance to districts and schools. The report noted that higher ratings for Centers 

were associated with more intensive technical assistance, based upon survey findings. 

The Educational Technical Assistance Act of 2002 authorized a new group of 

Comprehensive Centers to provide technical assistance for NCLB implementation at the state, 

district, substate region, and school levels for the purposes of ―improving academic achievement, 

closing achievement gaps, and encouraging and sustaining school improvement‖ (Section 203). 

It also gave ED discretion to ―establish priorities‖ for the Centers (Section 207). 

The current design of the Comprehensive Centers program reflects changes from the 

design of the predecessor program. In making its design choices, ED set up an advisory process 

to identify priorities for the new Centers: in 2004 the Secretary of Education appointed 10 

Regional Advisory Committees that would conduct needs assessments in their regions and make 

recommendations regarding technical assistance. The committees said SEAs needed help making 

better use of scientifically based research in decisionmaking, and that strengthening SEAs‘ 

capacity to serve local school districts was critical to the success of NCLB reforms, according to 

a synthesis of their recommendations.2

Following this effort, ED identified states as the Centers‘ primary client base, although 

the program had in the past served local clients, as described above. ED charged the Centers to 

work in new ways to expand and strengthen states‘ capacity to deliver assistance to schools and 

districts. The new program would also consolidate and expand responsibilities for 

Comprehensive Center technical assistance in that it would replace the Regional Technology in 

Education Consortia, the Eisenhower National Clearinghouse for Mathematics and Science 

Education, and the Regional Mathematics and Science Education Consortia. 

Structure of the Comprehensive Center System 

 To implement the provisions of the Educational Technical Assistance Act of 2002, ED 

created a new, two-tiered Center system. The Notice Inviting Applications detailed the design of 

that system. Under the new system, the Secretary of Education would award grants to 21 

Centers, each tasked with ―provid[ing] technical assistance to States as States work to help 

districts and schools to close achievement gaps in core content areas and raise student 

achievement in schools. To accomplish this goal, ED stipulated that applicants had to ―propose a 

plan of technical assistance specifically focused on helping States implement the provisions of 

NCLB applicable to States, and helping States build the capacity to help school districts and 

schools implement NCLB provisions and programs.‖

3

 

2

 Sheekey, A., Cymrot, D.J., and Fauntleroy, C. (2005, March). A Report to the U.S. Department of Education: 

Overview and Synthesis of the Regional Advisory Committee Reports on Educational Challenges and Technical 

Assistance Needs. Alexandria, VA: The CNA Corporation. 

3

 Ibid, 32585. 
3

While the overall goal of assisting states with NCLB implementation was common to all 

21 Centers, the two-tiered system designed by ED created distinct roles for the two types of 

Centers. These are described next. 

Distinct RCC and CC Roles 

Within the 21 Centers, ED constituted 16 Regional Comprehensive Centers (RCCs) and 

five Content Centers (CCs). By design, RCCs and CCs were given different roles and functions 

in a system of NCLB technical assistance (exhibit 1.1). RCCs, embedded within distinct 

geographic regions across the United States and territories, would deliver technical assistance to 

the states and territories in their region, addressing their needs and building their capacity to 

assist their districts and schools. Meanwhile, each of the CCs would take responsibility for 

synthesizing knowledge from the research and promising practices within a particular 

substantive area. The CCs would build the capacity of the RCCs by providing research-based 

information, products, guidance, and knowledge on key topics. The CCs would also work with 

RCCs to provide technical assistance to states. In turn, each state would help its districts and 

schools meet NCLB requirements. 

Exhibit 1.1. Center network design 

 

RCC CC

State

Other sources

of technical

assistance

LEAs and

schools

Elaborating on the functions of RCCs, ED required them to work directly with states to “provide 

frontline assistance.”

1

ED mandated that RCCs provide states with ongoing assistance and 

training that would draw from a range of knowledge sources, including but not limited to CCs; 

provide CCs with information about promising practices; convene states for collaboration; 

1

 Ibid., 32585. 
4

and deliver information based on both research and best practice. The specific list of RCC 

responsibilities included the following:5

■ ―Working closely with each State in its region on an ongoing basis‖

■ ―Linking States with the resources of Content Centers, Department staff, Regional 

Educational Laboratories, The What Works Clearinghouse, and other entities‖

■ ―Suggesting sources of appropriate service providers or assistance for State 

activities that are not within the core mission of the centers‖

■ ―Assisting State efforts to build statewide systems of support for districts and 

schools in need of improvement‖

■ ―Working to identify, broker, leverage, and deliver information, resources and 

services from the Content Centers and other sources‖

■ ―Convening in partnership with Content Centers and others, as appropriate, States 

and districts to receive training and information on best practices and researchbased improvement strategies‖

■ ―Providing guidance and training on implementation of requirements under 

NCLB and other related Federal programs‖

■ ―Facilitating collaboration at the State level to align Federal, State, and district 

school improvement programs‖ 

■ ―Helping Content Centers to identify, document, and disseminate emerging 

promising practices‖ 

The Notice Inviting Applications portrayed CCs as a central source of readily accessible 

knowledge, resources, and tools. Each CC was designed to consolidate in-depth knowledge in 

one of five key content areas: Assessment and Accountability, Instruction, Teacher Quality, 

Innovation and Improvement, or High Schools. The types of knowledge specifically mentioned 

included research, scientifically valid practices, and promising practices. The degree of emphasis 

on research and scientifically valid practice was heightened in this redesign of the 

Comprehensive Centers program. This was consistent with NCLB, which stated that 

scientifically based research must inform local practice. 

Within their content areas ED tasked CCs to:6

■ ―Identify, organize, select and translate existing key research knowledge...and 

communicate the information in ways that that are highly relevant and highly 

useful to State and local level policy makers and practitioners‖

■ ―Benchmark State and district practices for implementing NCLB provisions and 

school improvement interventions...and identify promising approaches that can 

be shared with States and districts‖

■ ―Convene States and districts, researchers and other experts to learn from each 

other about practical strategies for implementing NCLB provisions and programs‖

■ ―Train Regional Center staff on what is known about scientifically valid practices 

and programs‖

 

5

 Ibid., 32585-6.

6

 Ibid., 32586-7.
5

■ ―Collaborate with Regional Centers to address specific State requests for 

assistance‖

■ ―Communicate to the field...Department guidance related to the center‘s content 

focus‖

■ ―Design needs assessment and data analysis tools that States and districts can use 

to benchmark their programs and progress‖

Awards to the Centers 

 At the conclusion of the competition, new Regional Comprehensive Centers were located 

in 16 regions of the United States, covering all U.S. states and territories. Of the 16 RCCs, there 

were four that served only their respective state: New York, Texas, California, and Alaska. The 

remaining 12 Centers served from two to seven states and other jurisdictions. The non-state 

jurisdictions that the Centers were to serve were the following: the District of Columbia, Puerto 

Rico, the Virgin Islands, American Samoa, Commonwealth of the Northern Mariana Islands, 

Federated States of Micronesia [Chuuk, Kosrae, Pohnpei, and Yap], Guam, Republic of the 

Marshall Islands, and Republic of Palau. Throughout this report, the term ―state‖ will be defined 

to include the 50 states as well as these other jurisdictions. A full list of the grantees and 

subgrantees appears in appendix A of this report. 

Comprehensive Center funding for FY 2006 totaled $56.3 million (exhibit 1.2). 

Individual Centers‘ funding ranged from a low of $850,000 for the Alaska and Pacific 

Comprehensive Centers, to a high of $5,912,997 for the California Comprehensive Center. 

Funding for each RCC was driven by a formula based on the region‘s total population and its 

number of poor children ages 5-17. Average funding across all RCCs was $2,876,640. 

Content Center funding for FY 2006 ranged from $1,466,096 for the Assessment and 

Accountability Center and the Center on Innovation and Improvement, to $2,446,096 for the 

Centers on Instruction, Teacher Quality, and High Schools. The latter group of Centers was cofunded with an allocation of $1 million each from the Special Education Technical Assistance 

and Dissemination Act, authorized under the Individuals with Disabilities Education Act. The 

average amount of funding across CCs was $2,054,096. 
6

Exhibit 1.2. Comprehensive Centers funding in FY 2006 

Centers States

FY 2006 

funding

Total $56,296,713

Regional Centers

Alaska Comprehensive Center AK $850,000

Appalachia Regional Comprehensive 

Center KY, NC, TN, VA, WV $3,829,927

California Comprehensive Center CA $5,912,997

Florida and Islands Regional 

Comprehensive Center

FL, Puerto Rico, Virgin 

Islands $3,788,289

Great Lakes East Comprehensive 

Center IN, MI, OH $3,592,771

Great Lakes West Region 

Comprehensive Center IL, WI $3,702,196

Mid-Atlantic Comprehensive Center DE, MD, NJ, PA, DC $3,388,147

Mid-Continent Comprehensive Center AR, KS, MO, OK $2,111,226

New England Comprehensive Center CT, MA, ME, NH, RI, VT $1,644,795

New York Comprehensive Center NY $2,886,970

North Central Comprehensive Center IA, MN, ND, SD, NE $1,286,458

Northwest Regional Comprehensive 

Center ID, MT, WY, WA, OR $1,630,818

Pacific Comprehensive Center HI, American Samoa, 

Northern Mariana Islands, 

Federated States of 

Micronesia, Guam, Marshall 

Islands, Palau

$850,000

Southeast Comprehensive Center AL, GA, SC, LA, MS $4,120,988

Southwest Comprehensive Center AZ, UT, CO, NV, NM $2,491,327

Texas Comprehensive Center TX $3,939,324

Content Centers

Assessment and Accountability Comprehensive Center $1,466,096

Center on Innovation and Improvement $1,466,096

Center on Instruction $2,446,096

National Comprehensive Center for Teacher Quality $2,446,096

National High School Center $2,446,096

EXHIBIT READS: The Alaska Comprehensive Center received funding in the amount of 

$850,000 for FY 2006. 

SOURCE: U.S. Department of Education 
7

Background on the State Role in NCLB 

A basic premise of the Comprehensive Centers program as designed by ED was that 

NCLB assigned many tasks to states. The background section of the Notice itemized the 

following NCLB requirements for states: 

...set standards for student performance, implement statewide testing and accountability 

systems to measure school and student performance toward achieving those standards, 

adopt research-based instructional and program improvements related to teaching and 

learning in the classroom, ensure that all teachers in core subject areas are highly qualified, 

and improve or ultimately restructure schools that are consistently low-performing.7

 With respect to consistently low-performing schools and also low-performing districts, 

NCLB mandated that all states establish and sustain statewide systems of support and 

improvement for school districts and schools identified for improvement under NCLB. Districts 

and schools identified for improvement must receive assistance from support teams, institutions 

of higher education, and regional service centers in the state. This ―statewide system of support‖

(SSOS) must include individuals who were knowledgeable about research and practice on 

teaching and learning and who could develop and implement comprehensive improvement 

strategies. ―State support teams‖ (SSTs) were required to help schools plan for improvement and 

to evaluate the effectiveness of school personnel. The NCLB legislation also provided that these 

support teams should receive technical assistance from Comprehensive Centers and others.8

Research Questions Addressed in This Report 

The current evaluation takes a global look at the Center program as designed by ED, 

tracking the ways in which the Comprehensive Centers interacted with clients (both states and 

other Centers) over three program years. This report presents data describing implementation and 

outcomes in response to the following research questions: 

■ What are the objectives of the Comprehensive Center network and of each 

Center?

■ What kinds of products and services are provided by the Comprehensive Center 

network and by each Center? 

■ How do Centers develop, refine, and carry out their plans for technical assistance? 

How do they define their clients‘ educational needs and priorities? How do Center 

clients (states or Regional Centers) define their needs and priorities? 

■ To what extent is the work of each Comprehensive Center of high quality, high 

relevance and high usefulness? 

 

7

 Ibid., 32584. 

8

 No Child Left Behind, Title I, Part A, § 1117 (a) (1). 
8

■ To what extent have states relied on other sources of technical assistance besides 

the Comprehensive Centers? What other sources? How does the usefulness of 

Center projects compare with the usefulness of projects from other sources? 

■ To what extent do states report that Center projects have expanded state capacity 

to address underlying needs and priorities and meet the goals of NCLB? 

Questions for Future Reports 

Two additional research questions examining change over time are to be addressed in a 

final report; this report, based on data collection regarding the Centers‘ work in 2007-08 and 

2008-09, will address the questions: 

■ Has the performance of the Centers in addressing the needs and priorities changed 

over the period of time studied? 

■ Has the quality, relevance, or usefulness of each Center‘s projects changed over 

the period of time studied? 

The evaluation will also report on case studies of capacity building at the state level. 

These findings will provide additional insight related to the research question regarding the ways 

in which Center projects have expanded state capacity for NCLB implementation. 

Organization of This Report 

 Following this introductory chapter, chapter 2 describes the study‘s methods. Chapter 3 

describes the ways in which the Centers planned and conducted technical assistance for the 

2006-07 program year and it presents findings on the types of services and products they 

provided. Chapter 4 addresses the technical assistance needs and priorities reported at the state 

level and the ways in which Centers were reported to have addressed these needs and priorities, 

and expanded state capacity. Chapter 5 reports on the quality, relevance, and usefulness of 

selected Center projects as determined through expert review (for quality) and participant 

surveys (for relevance and usefulness). 
9

2. Study Design 

This chapter describes the data sources and analytic procedures used to address the 

report‘s research questions. Data collection for this interim report was conducted over a period 

from May 2007 through July 2008, and addressed the activities of Centers and their clients 

during the period from July 1, 2006, through June 30, 2007. A final report will include two 

additional rounds of data collection in 2008-09 and 2009-10 conducted to address activities in 

subsequent years, from July 2007 through June 2009. 

The evaluation team used six data sources to address the research questions: documents 

produced by the Centers with assistance from the evaluation team, Center management plans, 

site visits to Centers, state manager surveys, expert panel reviews, and participant surveys 

(exhibit 2.1). 

For the first three questions, which pertained to the operations of the Centers, the data 

were drawn from the Centers themselves. The team used Center management plans as a data 

source regarding Center objectives. In summer 2007 and summer 2008, the team conducted site 

visits to Centers; closed-ended prompts were used in face-to-face interviews to gather self-report 

data systematically on Center objectives, procedures for needs assessment and planning, and 

interactions with their clients. Two documentary sources were used as sources for descriptions of 

Center technical assistance: the project inventory forms and project cover sheets completed by 

the Centers with review and feedback from the evaluation team. The definition of a project and 

the procedures for gathering and reviewing these data are discussed in detail in this chapter. 

A survey of state managers was the source for data on the states‘ technical assistance 

needs and priorities, on other sources of technical assistance used, ratings of the overall technical 

assistance received, and on perceived capacity change at the state level. The technical quality of 

Center technical assistance was assessed by a panel of experts on the topic of each technical 

assistance project. Finally, participants answered survey questions pertinent to the relevance and 

usefulness of Center technical assistance. 

Each of these data sources is described in this chapter, and the analytic procedures 

specific to each source are discussed. The data collection instruments and further details 

regarding data sources and procedures can be found in appendices referenced throughout the 

chapter. The chapter concludes with a brief explanation of units of analysis for the reporting of 

quality, relevance, and usefulness ratings. 
10

Exhibit 2.1. Data sources for the research questions 

Research question

Project 

inventory 

forms 

and 

project 

cover 

sheets 

Center 

manage

-ment 

plans

Site 

visits to 

Centers

State 

manager 

survey

Expert 

review 

panels

Participant 

surveys

What are the objectives of the 

Comprehensive Center 

network and of each Center?

 

What kinds of products and 

services are provided by the 

Comprehensive Center 

network and by each Center?

 

How do Centers develop, 

refine, and carry out their 

plans for technical 

assistance? How do they 

define their clients’ 

educational needs and 

priorities? 



How do Center clients (states 

or Regional Centers) define 

their needs and priorities?

 

To what extent have states 

relied on other sources of 

technical assistance besides 

the Comprehensive Centers? 

What other sources? How 

does the usefulness of Center 

projects compare with the 

usefulness of projects from 

other sources?



To what extent do states 

report that Center projects 

have expanded state capacity 

to address underlying needs 

and priorities and meet the 

goals of NCLB?



To what extent is the work of 

each Comprehensive Center 

of high quality, high relevance

and high usefulness?



(overall 

relevance 

and 

usefulness)



(quality)



(projectlevel 

relevance 

and 

usefulness)
11

Identification of a Sample of Center Work for Expert Review 

and Participant Surveys 

A critical component of the evaluation was to rate the quality, relevance, and usefulness 

of Center products and services. Given available resources, it was not possible for the evaluation 

team to submit all of a Center‘s products and services to an independent review panel to rate 

quality. Nor was it feasible to survey all individuals who used Center products or participated in 

Center activities in the designated time period regarding relevance and usefulness. Therefore, the 

evaluation team developed and applied a strategy to select a sample of work from each Center 

for expert panel review and participant ratings. 

The following sections describe: (1) the unit of analysis, (2) the sample frame, (3) the 

evaluation sample, and (4) materials obtained from Centers. 

The Unit of Analysis: The Project 

The evaluation team initially reviewed the Centers‘ 2006-07 management plans to 

understand the nature of the work the Centers were conducting and determine if the management 

plans might serve as an appropriate sampling frame for the evaluation. Based on this effort, the 

team determined that the sampling procedures could not be based on the management plans for 

two reasons. First, the Centers‘ plans and work continued to evolve over time. Consequently, the 

plans, which were prepared before the program year, did not comprehensively reflect the work 

actually being done by the Centers months later. Second, the Centers used different approaches 

to organizing and aggregating their work. For example, some presented their work by state while 

others organized it by topic area. For the purposes of this evaluation, the team identified 

―projects‖ as a common level of aggregation that would constitute units large enough for review 

and rating, but focused enough for coherence. A ―project‖ was defined as a group of closely 

related activities and/or deliverables designed to achieve a specific outcome for a specific 

audience. To ensure that projects would constitute units that were large enough for review and 

rating, but focused enough for coherence, the study team provided the following criteria: 

■ Complete and coherent whole. Because each project should be able to stand on 

its own in an expert panel review, it should include all related activities and 

products. 

■ Common intended outcome. Where a cluster of activities and deliverables was 

designed by the Center to lead to the same outcome for the same audience, those 

activities and deliverables should be grouped as one project. 

■ Topic area focus. With few exceptions, a project addressed just one topic area 

(for example, statewide systems of support, adolescent literacy, assessment of 

English language learners) around which there was a body of research or 

professional wisdom. 
12

Since the project was a unit developed for this evaluation and was not necessarily how 

Centers divided up their work for programmatic or cost-tracking purposes, the evaluation team 

took several steps to ensure that the project concept was clear and used consistently across 

Centers. The evaluation team developed a standard Project Inventory Form (PIF) that Centers 

used to create an inventory of their work for each of the program years in this evaluation (see 

appendix B for a copy of the PIF) as well as written guidance in the instructions for completing 

the PIF (described in appendix B) and a sample inventory to serve as an example of the kinds of 

projects that should be listed by the Centers. The sample inventory included examples of 

activities or resources for defining projects at each level of effort—major, moderate, or minor. 

Members of the study team also invited Center staff to attend training sessions by conference call 

on completing the PIF, including defining projects. When draft inventories were received, the 

study team reviewed them to make sure the entries listed met the three criteria used to identify 

projects. They also reviewed the Center‘s reporting of the project level of effort for 

correspondence with the listed activities and resources and listing of the projects under each 

topic area for correspondence with the topic definitions provided in the written guidance. The 

study team provided technical assistance to Centers as needed to ensure that all projects 

conformed to the standards. 

The Sampling Frame

The evaluation was designed to assess the quality, relevance, and usefulness of a sample 

of Center work. The PIFs served as the basis for identifying the sampling frame from which the 

study team drew the sample of projects that became the subject of expert panel reviews and 

participant surveys. For the 2006-07 data collection cycle, the evaluation team asked each 

Comprehensive Center to use the PIF to prepare an inventory of all the projects active during the 

grant period from July 1, 2006 through June 30, 2007. A total of 364 projects (110 major, 106 

moderate, and 148 minor) were identified across the 21 PIFs.20 The number of projects included 

on the 2006-07 PIFs ranged from a minimum of 6 to a maximum of 32 projects. 

To be eligible for the sampling frame, projects first needed to represent a reasonable 

amount of effort (i.e., classified as ―major‖ or ―moderate‖ by the Center) and have a sufficient 

amount of material to give reviewers enough information to judge the quality of the work. 

Projects included in the sampling frame also needed to have identifiable participants since the 

evaluation design called for collecting relevance and usefulness ratings through surveys of 

project participants. All 148 minor projects were excluded from the sampling frame, along with 

16 major or moderate projects that had produced few or no reviewable materials or did not have 

identifiable participants in 2006-07. A total of 200 projects met both eligibility criteria and were 

included in the sampling frame. 

 

20 Centers designated on their PIFs whether each project was ―major,‖ ―moderate,‖ or ―minor‖ in terms of the level 

of effort and/or resources the Center devoted to it, relative to other projects in the same Center. 
13

The Sample of Projects 

To ensure that the final sample of projects reviewed reflected a range of each Center‘s 

work across topics and states and captured work that represented the largest investment of 

resources, the team implemented a sampling strategy that included a combination of Centernominated projects and a stratified set of purposively-selected projects. The desired sample size 

at each Center was a function of that Center‘s budget amount. For 2006-07, the number of 

projects sampled for each Center ranged from three to eight depending on the size of the Center‘s 

annual budget (exhibit 2.2). Although the sample was not statistically representative in scientific 

terms, it was designed to include a high percentage of the major projects of each Center as well 

as projects that Centers thought best represented their work. 

Exhibit 2.2. Sample size by budget 

Center funding level—

annual federal contract

(number of Centers)

Total 

projects to 

be selected 

for review, 

per Center

Number of 

projects to be 

nominated by 

Centers, per 

Center

Number of 

projects to 

be selected 

purposively, 

per Center

Number of 

projects 

selected 

across 

Centers

Total 122

Less than $1 million

(N=2) 4 1 3 7**

$1 to 1.9 million

(N=5) 5 1 4 24*

$2 to 2.9 million

(N=6) 6 2 4 35**

$3 to 3.9 million

(N=6) 7 2 5 40**

$4 million or more

(N=2) 8 2 6 16

* Does not include a collaborative project conducted jointly by an RCC and a CC that is accounted for in 

another category. 

** In four cases, Centers did not have enough eligible projects in their sampling frame to meet the desired 

sample size. 

EXHIBIT READS: There were four Centers whose annual funding level was less than $1 million. Three 

projects were selected for review for each of these centers, one of which was nominated by the Center 

for selection and two of which were selected for inclusion by the study team. 

When completing their inventory forms, Centers were given an opportunity to nominate 

the one or two projects they felt best represented their work.21 These Center-nominated projects 

 

21 The number of projects a Center was able to nominate depended on the size of its annual budget, as shown in 

exhibit 2.2. 
14

were selected first for the sample, followed by major projects. For Centers with insufficient 

major projects to meet the predetermined sample size, moderate projects were selected. The 

study team used an iterative sampling process to randomly select projects while controlling for 

topic area and state. 

Using these methods for 2006-07, the evaluation team selected a total sample of 122 

projects1

–93 major and 29 moderate (see exhibit 2.3). Overall, 61 percent of the projects that 

were eligible for the sampling frame (122 of 200) were selected. The sample predominantly 

included the most major work of the Centers (90 percent of the major projects) although the 

sample was not statistically representative in scientific terms, and covered 34 percent of all 

projects reported on the PIFs. 

Exhibit 2.3. Project sampling process 

EXHIBIT READS: Of the 364 projects listed on the PIFs, 164 were excluded from the sample 

frame, leaving 200 eligible projects from which 122 were sampled.

 

Further detail on the sampled projects by topic in relation to all major and moderate 

Center projects appears in exhibit 2.4. For 17 of the 232topics, 50 percent or more of all major 

and moderate projects addressing that topic were sampled. 

 

1

 This total includes one collaborative project conducted by an RCC and a CC together that was classified as a major 

project by both Centers. At the end of this chapter, in the section detailing the study’s analytic procedures, the 

handling of this project in analysis and reporting is described. 

2

 Exhibit 2.4 includes the topic area of “other” as the 23rd topic area, as a small number of the sampled projects were 

included in that area. Subsequent discussions in the report use the 22 specific substantive topic areas. 

Major

projects

(N=103)

Moderate

projects

(N=97)

Moderate projects

in sample (n=29)

Center-nominated

projects=9 

Major projects

in sample (n=93)

Center-nominated

projects=26

Projects without adequate 

reviewable materials or 

project participants

(N=16, 7 major, 9 

moderate)

Projects excluded

from sample

frame (N=164)

Study sample

(n=122)

Minor

projects

(N=148)

Sample frame

(N=200)

All projects on

Project Inventory

Forms (PIFS)

(N=364)
15

Exhibit 2.4. Distribution of all major and moderate projects and projects 

in the evaluation sample, by topic 

Project topic 

Number of 

major and 

moderate 

projects on 

project 

inventories

Number of 

projects in the 

sample

(all major or 

moderate)

Percent of all 

major and 

moderate 

projects in 

evaluation 

sample

Total 216 122 56%

Components of effective systems of 

support—state, district, school 54 36 67

Data use/data-driven decisionmaking 5 2 40

Formative assessment 5 1 20

Reading 17 7 41

Adolescent literacy 11 5 45

Mathematics 11 6 55

Dropout prevention 4 2 50

High school redesign/reform 10 5 50

Transition to high school 4 3 75

Special education—curriculum, instruction,

and professional development 5 3 60

Special education—assessment 1 1 100

English language learners 17 11 65

Highly qualified teacher provisions of NCLB 8 5 63

Teacher preparation and induction 6 4 67

Teacher professional development 8 7 88

Supplemental educational services (SES) 9 5 56

Response to Intervention (RtI) 7 3 43

Migrant education 4 2 50

Indian/native American education 2 1 50

Data management compliance 8 5 63

Assessment design 2 2 100

Parent involvement 6 3 50

Other 12 3 25

EXHIBIT READS: There were 54 major and moderate projects on the project inventories that focused on 

components of effective systems of support. Of these, 36 (or 67 percent) were in the project sample. 

SOURCE: Project inventory forms submitted by the Centers. 
16

Materials from Centers 

The evaluation team notified the 21 Center Directors about projects selected for review 

and sent a standard Request for Materials for Expert Panel Review (see appendix B for a copy of 

the transmittal memo and request form). Centers were asked to assemble and submit a 

comprehensive set of pre-existing materials associated with each project (meeting agendas, 

briefing books, meeting summaries, training materials, white papers, web resources, etc.) that 

would fully describe the project and provide reviewers with a sufficient basis for rating the 

technical quality of the work. Centers were also asked to include a participant list and a standard 

cover sheet, using a format developed by the evaluation team, for each project. The participant 

lists were used to draw the sample for participant surveys. 

When the project materials were received, they were reviewed by evaluation team 

members for completeness. If materials were missing or inaccessible (e.g., electronic files didn‘t 

open) or the cover sheet was incomplete, a follow-up memo was sent to the Center detailing the 

issued identified. Evaluation team members then worked with the Center to obtain the missing 

information and finalize the review package (see appendix B for a copy of the follow-up memo). 

Describing Center Operations 

The purpose of collecting data from the Centers was to describe Center operations and to 

address research questions regarding the objectives of the Centers, the kinds of products and 

services provided, how the Centers defined their clients‘ needs and priorities, and the extent to 

which the Centers met the objectives of their own plans. The description of Center operations in 

chapter 3 of this report is drawn from three sources of data: interview responses gathered in site 

visits to the Centers, PIFs, and the project cover sheets submitted for the sampled projects. The 

procedures used in gathering and analyzing data from these sources are described in the 

following pages. 

Management Plans 

Centers stated their objectives for the year in their annual program management plans. 

For the 2006-07 program year, Centers had developed their management plans in spring 2006, 

prior to the program year. The evaluation team relied on these management plans to gather 

information from statements of the Center‘s intended substantive focus for the year‘s technical 

assistance. The team coded each Center‘s stated objectives in their management plan by subject 

area, using the same list of 22 topic areas used to code the PIFs. Appendix C provides a 

description of the process used to code the management plans as well as an analysis of the 

intercoder reliability. 
17

Site Visits to Centers 

Site visits were conducted at two time points, first in summer 2007, followed by a visit in 

summer 2008 to capture descriptions of Center operations during the 2006-07 program year. As 

described earlier, the team reviewed Center management plans in 2006-07 to understand the 

nature of the work the Centers were conducting; this review provided a way of preliminarily 

identifying objectives, approaches, and procedures found in Center work. Structured site visit 

interviews were conducted with Center management teams in summer 2007, posing open-ended 

questions regarding the following topics (see appendix C for the protocols): 

■ Center organization (lead organization, subgrantees, ways of dividing 

responsibilities among staff) 

■ Major areas of focus 

■ Communication with client organizations (states in the case of RCCs, or RCCs in 

the case of CCs) regarding needs and assistance to be provided 

■ Modes of delivering technical assistance 

■ Sources of knowledge used

■ Approaches taken in quality assurance 

■ Working relationships within the Center network (between RCCs and CCs, with 

other Centers of the same type, and networkwide) 

■ Barriers or challenges encountered 

The team‘s review of the 2007 interview transcripts informed the development of closedended prompts that were subsequently administered to all Center leaders in interviews in the 

following summer. These second-round structured interviews included binary questions to 

follow up about Center objectives for 2006-07 and activities that had occurred during the 

program year (see appendix C for the interview prompts). It is the responses to these prompts 

that are reported, across Centers, in chapter 3. 

Project Inventory Forms (PIFs) 

 While the main purpose of the PIFs was to build the sample frame (described earlier in 

this chapter), the evaluation team also used the PIFs to gain Center-level data on the work the 

Centers undertook in 2006-07. The projects sampled for quality, relevance, and usefulness 

ratings were classified by topic area (see appendix C). The evaluation team coded the topics of 

the remaining projects using the same 22 topic areas to provide a full picture of the topics 

covered by each Center‘s 2006-07 PIF. Appendix C provides a description of the process used to 

code the projects as well as an analysis of the intercoder reliability. 
18

Project Cover Sheets 

For projects included in the sample, the standard cover sheets provided by the Centers 

described project activities and cross-Center collaboration in the project. The project cover 

sheets were primarily collected from each Center to help orient expert panels to the purpose and 

content of the materials to be reviewed (see appendix B for a copy of the Request for Materials 

for Expert Panel Review transmittal memo and cover sheet). 

The evaluation team used the cover sheets as a data source for an overall description of 

activities and collaboration in the sampled projects (reported in chapter 3) and for use in 

classifying projects into subgroups for which the ratings of relevance and usefulness could be 

analyzed (reported in chapter 5, as described below). The cover sheets provided descriptive 

information for each project, including the activities and deliverables associated with the project, 

and the contributions of other Centers to the project. Categories of Center activities and 

resources were drawn from review of the Center management plans and site visit interviews. 

These coding categories, thus, permitted a yes/no judgment of whether each project offered each 

of the following activities or resources to participants: 

■ Ongoing consultation and follow-up 

■ Research collections or syntheses 

■ Engagement of participants in project planning 

■ Training events 

■ Task force meetings and work 

■ Conferences 

■ Support development of a formal plan to implement a program or policy 

The team also coded the type of contribution, if any, of any CC to each RCC project, and 

of any RCC to each CC project. All these elements of the cover sheets were coded by members 

of the evaluation team using procedures described in appendix C; the appendix also provides 

results of the analysis of intercoder reliability. 

Survey of State Managers 

The purposes of the survey of state managers were to obtain information on state 

priorities in terms of state responsibilities related to the implementation of NCLB, obtain the 

state perspective on the relevance and usefulness of Center assistance, and obtain a comparative 

judgment of Center assistance in relation to assistance available through other sources such as 
19

professional associations. The survey instrument used for the survey of state managers appears in 

appendix D of this report. 

In order to identify appropriate respondents for the survey of state managers, the 

evaluation team collected the names of each RCC‘s main points of contact in each SEA during 

the summer 2007 site visits to the RCCs. For 2006-07, the resulting list of state managers 

included a total of 132 respondents across the 62 states included in the study.24

To be fully reflective of all SEAs in the analysis of these data, it was critical that the 

study team receive completed surveys from state managers in each state. The final response rate 

for the state manager survey was 100 percent for the 50 states and 50 percent for the 12 outlying 

areas, for an overall response rate of 90 percent. 

For 20 states, the evaluation team received completed responses from more than one state 

manager (exhibit 2.5). The state was the primary unit of analysis in analyzing data from the state 

manager survey for this report. The state managers‘ responses were weighted to ensure that each 

state was equally represented in all summary statistics while taking into account the variation in 

responses within each state. The weighting procedure, where each response was weighted by the

inverse of the number of managers responding from that state, ensured that each state was 

equally represented when the evaluation team aggregated responses across states to describe the 

distribution of responses. 

Exhibit 2.5. Number of responses to the state manager survey 

Number of responses 

from the states

Number of 

states

Total 56

1 36

2 15

3 4

4 1

EXHIBIT READS: In 36 states, the survey of state managers was 

completed by a single respondent. There was one state where the state 

manager survey was completed by four separate respondents. 

Across states, the offices, divisions, or departments represented by state managers were 

federal programs (in 68 percent of states), school improvement (64 percent), curriculum and 

instruction (53 percent), assessment and accountability (40 percent), and special education (30 

 

24 The Comprehensive Centers were expected to serve 62 jurisdictions including the 50 states and 12 other 

jurisdictions: the District of Columbia, Puerto Rico, the Virgin Islands, American Samoa, Commonwealth of 

Northern Marianas, Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap), Guam, Republic of the 

Marshall Islands, and Republic of Palau. For purposes of this report, the term ―state‖ refers to the 50 states and the 

outlying territories listed here.
20

percent). In relation to managing technical assistance, state managers were most frequently 

responsible for identifying needs and priorities (84 percent of states) and managing others in 

their use of technical assistance services (80 percent). Other frequent job responsibilities 

included serving as point of contact or manager for specific technical assistance projects (66 

percent), seeking out technical assistance providers (65 percent), and negotiating a scope of work 

with technical assistance providers (59 percent). 

Assessment of Quality by Expert Panels 

Given the Centers‘ charge to focus on scientifically based research and evidence-based 

practices to improve student achievement and close achievement gaps, one goal of the 

Comprehensive Centers evaluation is to assess the technical quality of work across the Center 

network using an independent panel of expert reviewers. Each sampled project was 

independently rated by a panel of three experts.25 Reviewers assigned a score to each of three 

quality dimensions discussed below, using a 5-point rating scale. 

In an effort to maximize interrater reliability, the evaluation team: (1) defined quality and 

developed a detailed scoring rubric that could be applied across all Centers and a range of 

projects; (2) recruited, trained and assigned highly qualified expert panelists; and 

(3) implemented a process for reviewers to discuss their findings with one another when scores 

were discrepant for particular projects. The following sections describe each of these steps, 

followed by a brief discussion of how the final technical quality ratings were calculated. 

Define Technical Quality and Develop Scoring Rubric 

Developing the definition of technical quality and the rubric used to measure it 

incorporated information from many sources, including federal legislation and ED specifications 

on what constituted scientifically based evidence and an example from another federal agency.26

When defining technical quality and developing the scoring rubric, the evaluation team sought to 

ensure that the definition was relevant to the range of projects Centers would provide–

recognizing that some projects might have a substantial research knowledge base and others 

might be guided more by promising practices (those that were supported by evidence but not yet 

rigorously studied) or legislative or regulatory requirements. Also, the definition of quality and 

the associated rubric had to be applicable to projects at varying points of development and 

implementation, from early-stage needs assessment and design work to fully-developed products 

and services. The evaluation team developed a quality scoring rubric, included in appendix E, to 

assess quality along the following three dimensions: 

■ Dimension 1: Demonstrated use of the appropriate documented knowledge base 

 

25 Two panelists were not able to complete their reviews in a timely fashion, resulting in nine projects having been 

rated by only two panelists. 

26 Review of Instructional Materials for Middle School Science. (1997, February). National Science Foundation, 

Retrieved from http://www.nsf.gov/pubs/1997/nsf9754/nsf9754.htm?org=NSF. 
21

■ Dimension 2: Fidelity of application of the knowledge base to the products and 

services provided 

■ Dimension 3: Clear and effective delivery 

Reviewers assigned a score to each dimension, using a 5-point rating scale (where 1 

meant ―very low quality‖ and 5 meant ―very high quality‖), according to the indicators defined 

for each dimension and examples in the scoring booklet. 

Recruit, Train and Assign Expert Panelists 

To meet the selection criteria for this evaluation, expert panelists had to have current, 

rigorous work in the particular topic of interest (for example, publications in peer-reviewed, 

scholarly journals; presentations at relevant professional organization meetings; recent 

membership on advisory panels or task forces) and be free of conflicts of interest.27 Nominations 

for panelists were made by staff in ED‘s Institute of Education Sciences (IES), the Technical 

Work Group for this evaluation, members of the evaluation team, and Center staff. Based on the 

selection criteria, the evaluation team selected a total of 70 expert panelists, 94 percent with a 

doctorate degree and two-thirds (67 percent) with university affiliations, to review the Centers‘ 

2006-07 sampled projects.28

The evaluation team sought to maximize interrater agreement in scoring of quality by 

training expert reviewers to systematically use a standard rubric. During the two-day training, 

experts worked in small groups to discuss how each of the three quality dimensions, and their 

corresponding indicators, applied to sample project descriptions that were provided by the study 

team. Reviewers independently scored one of two projects overnight and submitted their 

dimension-level scores at the beginning of the second day. Scores were posted and analyzed in 

terms of rater agreement at the dimension level. The panelists again worked in small groups to 

discuss their scores and identify possible reasons for any discrepancies in the results. During the 

small group discussions, evaluation staff circulated among the groups to assess whether 

reviewers had adhered to the standards of evidence discussed on Day 1.29

During the actual review process, expert reviewers were asked to score four to eight 

projects in their area(s) of expertise, with no more than three projects from any given Center. 

Since judgments about the state of the available evidence on a given topic and its applicability to 

 

27 Current Comprehensive Center staff, as well as individuals employed by organizations that had an ongoing 

financial relationship (for example, a contract or cooperative agreement) with a Comprehensive Center and who 

worked on a Center project, were not eligible to serve as reviewers. 

28 The final expert panels were made up of 67 expert reviewers. One of the initial trainees was removed due to a 

conflict of interest, and two reviewers were later dropped from the review process when they failed to complete their 

reviews in a timely fashion. 

29 During training, expert reviewers scored a sample project and discussed their scores in small groups, paying 

particular attention to areas where scores were divergent. This approach allowed the evaluation team to identify 

dimensions or indicators within the scoring rubric that seemed to be problematic (resulting in discrepant scores) or 

particular reviewers who needed additional training or appeared to be inappropriate to use. The goal was to have 

panelists leave the training with a common understanding of how to apply the quality rubric; the evaluation team did 

not intend to attempt to establish a specific interrater reliability criterion at training. 
22

the project being rated relied heavily upon the knowledge of the expert reviewers, it was 

important that reviewers were assigned projects that matched their area(s) of expertise. The 

evaluation team was also careful when assigning projects to avoid known conflicts of interest.30

When needed, the evaluation team reassigned projects when reviewers were unable to complete 

their reviews in a timely fashion, identified unforeseen conflicts of interest, or did not feel they 

had the requisite expertise to review the assigned projects. 

Overall, 113 of the 122 projects sampled were reviewed by three panelists. The 

remaining nine projects were reviewed by only two panelists because the third assigned reviewer 

did not complete the review within a reasonable timeframe.31

Address Interrater Reliability

In addition to the detailed scoring rubric and training provided, a resolution process was 

used to help achieve a high degree of interrater reliability in scoring. If the reviewers‘ projectlevel scores (defined as the simple average of their three dimension-level scores) were found to 

differ by 2 or more points from each other for any given project, the evaluation team convened 

the panel by telephone to discuss the ratings.32 The goal of the discrepancy conference calls was 

to give panelists an opportunity to understand the rationale behind their colleagues‘ scores and 

consider whether, on the basis of that discussion, any scoring revisions were warranted. The 

evaluation team emphasized to the reviewers that these discussions were not intended to achieve 

consensus among them. 

Overall, 24 percent (29 of 122) of the projects had discrepancies in the project-level 

scores (exhibit 2.6). In 7 of the 29 discrepant cases, there were discrepancies in two of the three 

dimensions, while the remaining 22 projects had discrepancies in all three dimensions. After a 

discrepancy conference call, panelists were given the opportunity to submit a revised scoring 

form and narrative report of the project‘s strengths and weaknesses. Although panels were not 

required to reach consensus, the discrepancy was resolved in all but two cases, increasing the 

number of projects in agreement from 76 percent to 98 percent. 

 To estimate interrater agreement, we calculated the rWG(J) index.33 Before the 

discrepancy calls, the average rWG(J) coefficient was 0.75.34 After the calls were conducted, the 

value increased to 0.87. 

 

30 For the purpose of assignments, the term ―conflict of interest‖ meant any financial or other interest that appeared 

to conflict with or significantly compromise the service of the individual reviewer because it could significantly 

impair the individual‘s objectivity.

31 In consultation with IES, the evaluation team decided to forgo the third review on these particular projects since 

there was interrater agreement (i.e., a difference less than 2 points) between the two scores that had been submitted 

by the other panelists assigned to these projects. 

32 The 2-point difference threshold was selected in an effort to identify projects with widely divergent scores that cut 

across qualitative categories (i.e., ―low‖ quality versus ―high‖ quality), while at the same time minimizing burden on 

expert panelists. 

33 James, L.R, Demaree R.G., and Wolf G. (1984). Estimating within-group interrater reliability with and without 

response bias. Journal of Applied Psychology, 69(1), 85-98.
23

Exhibit 2.6. Distribution of discrepancies and results of the discrepancy 

call process 

Description

N

(percent)

Total number of projects reviewed 122

Number of projects without a projectlevel discrepancy93

(76 percent of 122)

Number of project-level discrepancies 

identified

29

(24 percent of 122)

Number of projects with 

discrepancies in 2 dimensions

7

(24 percent of 29)

Number of projects with 

discrepancies in 3 dimensions

22

(76 percent of 29)

Number of project-level discrepancies 

resolved* by panel calls

27

(93 of 29 calls conducted)

Number of project-level discrepancies 

remaining after panel calls

2

(7 of 29 calls conducted)

* Discrepancies were considered resolved when the project-level scores differed by less than 2 points. 

EXHIBIT READS: Of the 122 projects reviewed, 93 (76 percent) had no discrepancies in the project level 

scores. 

Calculate Aggregate and Dimension-Level Measures of Technical Quality 

In order to analyze the technical quality of sampled projects, a series of steps was taken 

to combine individual expert ratings into aggregate scores. The first step was to combine the 

individual dimension-level scores into a single rating from each panelist. For each project 

reviewed, a simple average of the three dimension-level scores was computed to generate a 

project-level score for each reviewer. In the second step, the three reviewers‘ scores were 

averaged to determine the overall quality score for each project. In the third step, the overall 

project scores across the set of sample projects were averaged within each Center to calculate a 

Center-level quality score. 

 

34 We estimated interrater agreement using the rWG(J) index. Treating each dimension as an ―item,‖ we first 

calculated the score variance between raters on each dimension and then averaged the three variances. We assumed 

the random error variance to be 2.0 for a 5-point scale, as suggested by James et al. (1984). An rWG value between 

0.71 and 0.90 is generally considered an indicator of ―strong agreement‖ (LeBreton, J.M., and Senter, J.L. (2008, 

October). Answers to 20 questions about interrelated reliability and interrater agreement. Organizational Research 

Methods, 11, 815-852. Retrieved from http://orm.sagepub.com/cgi/content/abstract/11/4/815). 
24

Center-level scores were aggregated to calculate the average quality rating across 

sampled projects for the Center program as a whole, as well as for two subgroups of interest, the 

16 RCCs and the 5 CCs. Each Center was given equal weight in computing the overall 

programwide rating for technical quality, as well as the mean ratings for the RCCs and the CCs 

respectively, for the sampled projects; within Centers, each project was given equal weight. 

Survey of Project Participants 

The purpose of the participant survey was to obtain client views of technical assistance 

from the Comprehensive Centers, particularly in the areas of relevance and usefulness. As the 

primary role of the RCCs is to provide technical assistance to the states and territories in their 

region, the clients for the RCCs included state-level staff. The clients for the CCs included both 

RCC staff and state-level staff corresponding with the primary role of the CCs to build the 

capacity of the RCCs as well as to work with RCCs in providing technical assistance to states. 

The evaluation team developed two parallel survey forms for project participants: one for 

state-level staff who participated in any Center project, and one for RCC staff who participated 

in a CC project. In sampling participants to respond to surveys focused on specific projects, the 

goal was to identify a sizable number of participants, drawn from complete lists of all 

participants, so that their responses would collectively provide a picture of all participants‘ views 

regarding the sampled projects. The evaluation team drew samples of participants in the 122 

projects that were selected for expert panel review. In this way, expert panel ratings of quality 

and participant ratings of relevance and usefulness were gathered for the same set of projects. 

The survey instruments used for state-level and RCC staff are located in appendix E of this 

report.35 A brief description of the process used to calculate the measure of relevance and 

usefulness is provided at the end of this section. 

Centers were asked to furnish full lists of all participants and their contact information for 

each sampled project. These lists included individuals who had participated in projects in 

numerous ways including: (1) serving on task forces, school support teams, and work groups 

associated with the project; (2) attending conferences, technical assistance retreats, and other 

meetings held as a part of the project; or (3) receiving written materials or other disseminated 

resources. State-level participants included staff who were employed by SEAs as well as 

employees of intermediate agencies, LEAs, schools, or other agencies who had responsibilities 

for state-level implementation of NCLB and were participants in both RCC and CC projects. 

RCC staff, as clients of the CCs, were participants solely in CC projects. A total of 3,904 

participants who met the participation criteria were eligible for the sampling frame. 

After identifying the sampling frame, the team implemented a sampling strategy that 

combined random sampling from each project and a replacement strategy to minimize 

 

35 Since the survey asked respondents for their experiences with the Comprehensive Center system in relation to a 

specific project, the survey team provided each respondent with a list of the activities included in that project, based 

on the content of the project cover sheets provided by the Centers. For the on-line survey, this information was 

displayed on the first screen of the survey. For the paper version of the survey, the project-specific information was 

printed on yellow paper and inserted into the front of the booklet. 
25

respondent burden. A sampling strategy was developed for selecting a sufficient number of 

participants to obtain fair representation of client views for each project, and across projects, 

while balancing respondent burden and data collection costs. The team drew a simple random 

sample of participants within each sampled project using the following sampling rules, based on 

the number of participants in the project: 

 All participants in projects with 12 or fewer participants were sampled. 

 A random sample of 12 participants was selected for projects with 13 to 25 

participants. 

 A range of 12 to 48 participants were randomly selected to represent 48 percent of 

participants for projects with 26 to 100 participants. 

 For each project with more than 100 participants, a random sample of 48 participants 

was selected. 

After drawing a random sample for each project, replacements were made for 

respondents who had been identified as a participant across multiple projects. To help reduce 

respondent burden, 13 percent of state-level staff and 14 percent of the RCC staff were replaced 

by alternates. These replacements resulted in 65 percent of the sample being requested to 

complete a single survey and 35 percent requested to complete more than one survey. 

Exhibit 2.7 provides an overview of the participant survey sampling and administration. 

Of the 3,904 participants in the sampling frame, the resulting sample for the participant survey 

consisted of 1,658 participants across the 122 projects. 

The evaluation team initially administered the participant survey to the sampled 

respondents online. Nonrespondents were contacted by phone, sent reminder cards, and sent 

paper survey forms, all in an effort to obtain completed responses. The final response rate for the 

participant survey was 73 percent across the 122 sampled projects. For two of these projects, no 

usable survey responses were received. Nonrespondents included participants who did not return 

a completed survey because they no longer worked for the state organization or RCC, as well as 

participants whose contact information was incorrect. 
26

Exhibit 2.7. Survey of project participants sampling and survey administration 

summary 

EXHIBIT READS: The sample frame of 3,904 project participants for survey administration included 2,689 

participants in RCC projects and 1,215 participants in CC projects. 

Calculate Measures of Relevance and Usefulness 

For the relevance and usefulness questions in the participant survey, respondents were 

asked to rate each aspect of relevance and usefulness (exhibit 2.8) using a 5-point scale ranging 

from 5 (a very high degree) to 1 (a very low degree).36

A preliminary step in the analysis of responses was to assess the properties of the items 

included in the surveys for constructing indices of relevance and usefulness. Construction of 

these indices was based on item response from the 1,208 completed surveys. Principal 

components analysis with no rotation was conducted on each set of items in the relevance and 

usefulness scales to determine the underlying dimensions represented by these items. Among the 

items comprising the relevance scale, one factor was extracted with an eigenvalue of 5.4 and 

explained 67.6 percent of the variance (see exhibit 2.8 for factor loadings). Similarly, among the 

usefulness items, one factor was extracted with an eigenvalue of 7.6 and explained 69.2 percent 

of the variance (see exhibit 2.8 for factor loadings). The reliability of each of the scales was 

 

36 The response category, ―Not able to judge,‖ on the survey instruments indicated those respondents who were not 

appropriate for addressing the particular item and were, therefore, not included in analyses. None of the respondents 

had indicated ―not able to judge‖ for the eight items of the relevance scale. A range of 3 to 4 percent of respondents 

across the 11 items of the usefulness scale had indicated ―not able to judge.‖

Participants in 

RCC projects

(N=2,689)

Participants in CC 

projects

(N=1,215)

Sampled participants:

•State-level (n=188)

•RCC staff (n=304)

Sampled participants:

•State-level (n=1,166)

Sample frame

(N=3,904)

Study sample

(n=1,658)

Survey 

administration

Completed surveys 1,208

Refusals 90

No response 360
27

evaluated using Cronbach‘s Alpha. For the eight items in the relevance scale, the reliability 

coefficient is 0.93, and for usefulness the coefficient is 0.95 across the 11 items. 

Exhibit 2.8. Relevance and usefulness items from the project participant surveys 

Relevance

Loading 

on factor 

1

Based on your experience, to what degree 

was this set of activities and resources 

relevant to your work, in each of the 

following respects?

a. Addressed a need or problem that my 

organization faces

0.83

b. Addressed an important priority of my 

organization

0.82

c. Addressed a challenge that my 

organization faces related to the 

implementation of NCLB

0.81

d. Provided information, advice, and/or 

resources that could be directly 

applied to my organization’s work

0.86

e. Addressed our particular state context 0.80

f. Addressed my organization’s specific 

challenges (e.g., policy environment, 

leadership capacity, budget 

pressures, local politics) 

0.83

g. Provided information, advice, and/or 

resources that could be used to guide 

decisions about policies, programs, or 

practices

0.85

h. Highlighted the implications of 

research findings (or information 

about best practice) for policies, 

programs, or practices

0.78

Usefulness

Loading 

on factor 

1

Based on your experience, to what degree 

was this set of activities and resources 

useful to you, in each of the following 

respects?

a. Provided resources that were easy to 

understand and easy to use

0.82

b. Employed an appropriate format (e.g., 

a work group, a conference, individual 

consultation, written products)

0.81

c. Provided adequate opportunity to learn 

from colleagues in other states

0.64

d. Included adequate follow-up to support 

the use of new information and 

resources

0.82

e. Were timely 0.84

f. Helped my organization solve a 

problem

0.87

g. Helped my organization maintain or 

change a policy or practice

0.83

h. Helped my organization take the next 

step in a longer-term improvement 

effort

0.85

i. Provided my organization with 

information or resources that we will 

use again

0.88

j. Helped my organization develop a 

shared expertise or knowledge base

0.88

k. Helped individuals in my organization 

to develop skills that they will use again

0.87

Mean ratings at the respondent level were averaged so that each respondent for a given 

project contributed equally to a project-level rating. Thus, the relevance or usefulness rating at 

the project level was a mean of the ratings provided by sampled participants in that project 

(ranging in number from 1 to 48) who returned surveys.37 Next, the rating of each sampled 

 

37 Across the eight items on the relevance scale, the rate of item nonresponse ranged from 0.3 percent to 1.1 percent. 

The rate of missing data among the 11 usefulness scale items was slightly higher with the percent of item-level 

nonresponse ranging from 1.1 to 2.0 percent. For both relevance and usefulness calculations, respondents with 

missing data on more than two of the items were excluded from the calculation, resulting in a small number of 
28

project contributed equally to the computation of the mean rating across projects for each Center. 

Finally, the team calculated an overall mean across the 21 Centers and overall means for the 16 

RCCs and the 5 CCs, respectively, for the relevance and usefulness ratings of the sampled 

projects. In calculating these overall means, each Center‘s mean rating was weighted equally.38

 

Units of Analysis for Quality, Relevance, and Usefulness 

Data in this Report 

 The quality, relevance, and usefulness ratings of the sampled projects were used in 

analyzing and reporting data for several different units of analysis: the projects sampled from the 

program as a whole (all 21 Centers), the projects sampled for the 16 RCCs and the 5 CCs 

respectively, subgroups of the 122 projects, and (for relevance and usefulness) subgroups of the 

3,652 sample-eligible participants. For a Center‘s mean rating on quality, relevance, or 

usefulness, the team weighted each sampled project from that Center equally; for a mean across 

the entire program or a set of Centers, the team weighted each Center‘s mean equally. 

 Several analyses were conducted for subgroups of projects across Centers—e.g., those 

projects that included a research synthesis among their products, or included training among their 

services. Each rater (an expert or survey respondent) had equal weight in the individual project‘s 

rating, and each project had equal weight in the subgroup mean rating. 

Finally, some analyses were conducted for subgroups of participants across projects. For 

example, the ratings of relevance provided by participants who spent at least three days in a 

project activity were compared with the ratings provided by participants who spent less time. In 

these analyses, the unit was the participant, not the project, and the weighting was designed to 

permit generalization to all the sample-eligible participants across all the sampled projects. Thus 

the weight for each participant‘s response was the inverse proportion of the number of 

respondents compared with the total number of participants in the sample-eligible population for 

that project. 

Number of Projects Varying with the Unit of Analysis 

Readers will notice slight variations in the total number of projects shown in the exhibits 

and analyses throughout this report. These figures differ for two reasons, which are explained in 

detail here (exhibit 2.9).

First, there were two RCC projects for which the evaluation team did not receive any 

completed surveys. This meant that for any analysis incorporating participant data, the maximum 

 

respondents being excluded (6 respondents [0.5 percent] for relevance and 18 respondents [1.6 percent] for 

usefulness). 

38 The evaluation team reviewed the differences in relevance and usefulness ratings between RCC staff and statelevel staff (in ratings of those CC projects that served both types of staff) to address concerns of potential bias. No 

statistically significant differences were found in the relevance and usefulness ratings provided by RCC and statelevel staff (see additional details presented in chapter 5, exhibit 5.17). 
29

number of projects in the analysis was 94 RCCs projects and 120 total projects. For analyses that 

did not rely on participant data (e.g., the quality ratings), the team used 96 RCC projects and 122 

total projects. 

Second, as described earlier in this chapter, one project included in the sample was 

sponsored jointly by an RCC and a CC. This ―combined project‖ was one of the projects counted 

in analyses specific to RCC projects, and also one of the projects counted in analyses specific to 

CC projects. For analyses across all projects or all participants, however, the joint RCC/CC 

project or its participants were counted only once in the analysis. 

Exhibit 2.9. Project sample, by type of rating and unit of analysis 

NOTE: Of the 95 RCC projects, relevance and usefulness ratings were not received from participants 

for two RCC projects. Thus, only 93 RCC projects were included in the analyses for the relevance and 

usefulness ratings. 

EXHIBIT READS: Among the 122 Comprehensive Center projects in the sample, 95 were RCC 

projects, 26 were CC projects, and one project was sponsored jointly by a RCC and a CC. 

Original sample:

122 projects

RCC

projects

(n=95)

Combined

RCC/CC project

(n=1)

CC

projects

(n=26)

Quality ratings

•122 unique projects with quality 

ratings

•95 RCC, 26 CC, 1 combined

Relevance and Usefulness ratings

120 unique projects with R and U 

ratings

93 RCC, 26 CC, 1 Combined

Center

n=123

96 RCC

27 CC

Project

n=122

Unit of analysis n’s:

Center

n=121

94 RCC

27 CC

Project

n=120

Unit of analysis n’s:

31

3. How the Centers Operate 

Understanding how Comprehensive Centers operate is an important part of assessing 

their performance. As described in chapter 1, ED established certain structures and expectations 

for the effective functioning of the Centers, including the split of responsibilities between the 

two types of Centers, the emphasis on using and applying scientifically based research to build 

state capacity to carry out NCLB, and the flow of communication between and among the 

Regional Comprehensive Centers (RCCs), the Content Centers (CCs), and state agencies. 

This chapter examines how the Comprehensive Centers reported working with their 

client organizations to assess technical assistance needs and to formulate their annual work. It 

provides information about the range of technical assistance topic areas, products, and services 

delivered by the Centers and the processes by which these activities were determined. The 

analysis draws on information gathered from the Centers‘ submitted Project Inventory Forms 

(PIFs), Center annual management plans, and interviews with Center staff conducted by the 

evaluation team.39

The key findings include: 

■ The Centers reported planning their work in coordination with their clients. All 

16 RCCs reported obtaining state input into the initial plans and engaging states in 

refinements to the plans through ongoing interaction and negotiation. Similarly, 

all five of the CCs reported forming their work plans incorporating RCC input 

acquired through either RCC staff surveys or direct communication. In addition, 

all five CCs described working with ED to learn of specific topics and tasks 

needed to advance federal priorities. 

■ Centers conducted work in the topic areas that they identified among the 

objectives in their initial management plans (80 percent of the cases), although 

their activities evolved through the year to address client requests. For example, 

14 of 21 Centers said they conducted work outside their initially planned scope of 

work. Refinements to the work plans included additional work in some topic 

areas, with response to intervention, English language learners, and supplemental 

education services the most common additions. Centers reported that added work 

was in response to emerging needs that were recognized in the process of carrying 

out planned work, and in response to requests from states. Survey data gathered 

from state managers also indicated a strong relationship (.95 correlation) between 

their reported priorities for Center technical assistance and the actual support they 

received. 

■ The kinds of support provided by the RCCs and CCs differed in ways that were 

consistent with the model of technical assistance envisioned by ED. The 

 

39 See chapter 2 for more information about the data sources and the procedures for gathering, coding, and analyzing 

the data reported in this chapter. 
32

majority of RCC projects involved front-line assistance activities such as ongoing 

consultation and follow-up (82 percent), whereas this was less so for the work of 

the CCs (22 percent). In contrast, the delivery of research collections and 

syntheses was a central part of the work of the CCs, occurring in 74 percent of 

their projects but 54 percent of the RCC projects. 

■ RCCs were more likely to draw on the work of the CCs than the other way 

around. All 16 RCCs (100 percent) reported drawing on CC expertise, and 48 

percent of the sampled RCC projects had a CC contribution. In comparison, two 

of the five CCs (40 percent) reported that they used RCCs as a source in their 

work. Thirty-seven percent of the sampled CC projects reportedly had some input 

from an RCC; in all cases this included RCC help in identifying participants, and 

in 30 percent of these cases (11 percent of sampled CC projects) it also included 

assistance provision by RCC staff. To some extent, the RCCs and CCs relied on 

their respective networks, with 69 percent of RCCs using the expertise of other 

RCCs and 60 percent of CCs drawing on the knowledge base of other CCs. 

■ External review of Center work was more prevalent among the CCs than the 

RCCs. When asked how they reviewed the content of their technical assistance 

prior to delivery—i.e., what quality assurance process they used— all five CCs 

(100 percent) reported that they submitted all major publications to ED for review 

prior to public release and four of the CCs (80 percent) reported retaining outside 

experts to review draft products. In addition, two CCs described relying on other 

CCs for feedback on their own products as part of the five CCs developing a 

common vetting tool. In contrast, two (12 percent) of the RCCs said they used 

outside experts as a source for vetting. 

Identifying State Needs and Priorities 

A key expectation of the Centers was that they would organize their technical assistance 

work plans around the priorities and needs of the states. In the design of the Center program,40

RCCs had the role of providing ―front-line‖ assistance to a set of state clients and therefore were 

expected to communicate directly with those state agencies. The CCs, on the other hand, were 

charged with translating research and policy for a nationwide clientele and so were expected to 

primarily depend on the RCCs to provide input on aggregated state priorities. The CCs also 

received guidance from ED in order to advance federal priorities. For the 2006-07 year, which is 

the subject of this report, the Department approved Centers‘ management plans in late spring 

2006. According to the Centers, they gathered information to develop and refine these plans in a 

variety of ways. 

■ RCCs conducted initial needs assessment in consultation with states (exhibit 

3.1); CCs assessed state needs primarily through the RCCs (exhibit 3.2). All 16 

 

40 Notice Inviting Applications for New Awards for Fiscal Year 2005. (2005, June 3). Federal Register, 70(106), 

32584. 
33

RCCs reported including state input when writing their annual management plans. 

Fifteen of these Centers assessed needs through state client meetings or 

communications, with almost all of them (14 of the 15) communicating directly 

with the chief state school officers. Fewer RCCs conducted surveys of state 

agency staff (10), analyzed state performance data or accountability plans (9), or 

examined state and federal policy environments to identify priorities for technical 

assistance (7). All RCCs were able to conduct needs assessment by building on 

prior contacts with states, either from earlier existing relationships (13) or the 

development of the Center‘s grant proposal (11). All of the five CCs reported 

forming their work plans incorporating RCC input acquired through either RCC 

staff surveys or direct communication. In addition, all five CCs described ED as 

an influential client with specific requests to focus on certain topics and tasks to 

advance federal priorities such as communicating practical findings from key 

research on ELL or communicating promising practices in high school 

restructuring. Four of the five CCs also reported defining their initial work 

priorities by examining state or federal policy. 

Exhibit 3.1. RCC strategies for assessing state needs and planning work 

Strategy

RCCs

(N=16)

Included SEA input when writing annual management plans 16

Assess needs through meetings or other communication with SEA staff 15

Communicate directly with the chief state school officer to plan the 

Center’s work 14

Drew on SEA relationships or input as part of initial plans for work 16

Build on relationships with SEAs that existed prior to the 

Center’s grant award 13

Included SEA input when writing the original grant proposal or 

cooperative agreement with ED 11

Assess needs through survey of SEA staff 10

Assess needs through analysis of state performance data or 

accountability plans 9

Assess needs through analysis of state and federal policy context 7

EXHIBIT READS: Sixteen RCCs included SEA input when writing annual management plans. 

SOURCE: Center responses to standard response categories during site visits. 
34

Exhibit 3.2. CC strategies for assessing RCC and state needs and planning work 

Strategy

CCs

(N=5)

Assess RCC needs through meetings or other communication with RCC 

staffs 5

Included RCC input when writing annual management plans 4

Communicate directly with the RCC Director to plan the 

Center’s work 4

Assess needs through survey of RCC staff 4

Assess needs through analysis of state and federal policy context 4

Draw on RCC relationships or input as part of initial plans for work 2

Build on relationships with RCC staff members that existed prior 

to the Center’s grant award 2

Included RCC input when writing the original grant proposal or 

cooperative agreement with ED 1

Assess needs through analysis of state performance data or 

accountability plans 0

EXHIBIT READS: Five CCs assessed RCC needs through meetings or other communication. 

SOURCE: Center responses to standard response categories during site visits. 

■ Centers continued to interact with clients to refine their needs assessments and 

negotiate the implementation of their plans (exhibit 3.3). Fifteen of 16 RCCs 

reported forming work groups within state organizations that brought together staff 

from multiple departments to discuss service needs and delivery; the remaining RCC 

reported having worked directly with individual managers such as the chief state 

school officer at the planning stage. Half of the RCCs (8) formed cross-agency work 

groups to discuss common SEA service needs and delivery. All five CCs identified 

needs of their client RCCs primarily by conducting conference calls with designated 

RCC representatives, and three of the five CCs maintained communication about 

needs by forming workgroups that included RCC representatives. In addition, all CCs 

reported providing either large group events or support of existing RCC programs and 

projects as additional ways to learn about and meet both RCC and SEA needs. The 

five CCs maintained ongoing communication with ED regarding federal priorities and 

revisions to their work plans. 
35

Exhibit 3.3. Center strategies for ongoing interactions and negotiations 

with clients 

Strategy

RCCs

(N=16)

CCs

(N=5)

All

(N=21)

Sponsor large events to make contact with many clients 13 4 17

Form work groups within client organizations that bring together 

staff from multiple departments/divisions to discuss service 

needs and delivery

15 1 16

Offer service to support existing client programs/projects/ 

policies 9 4 13

Form work groups across client organizations to discuss service 

needs and delivery (e.g., different SEAs and/or RCCs) 8 3 11

EXHIBIT READS: Thirteen RCCs sponsored large events to make contact with many clients. 

SOURCE: Center responses to standard response categories during site visits. 

■ RCCs had direct state interactionsto gathering information and keep in touch 

with states’ needs (exhibit 3.4). In interviews, all 16 RCCs identified staff whose 

responsibilities included talking with top- or mid-level SEA managers. For the 12 

multi-state RCCs, staff positions (termed state liaisons) were designated for each 

of the states they served. Of these 12, half hired former SEA employees (one of 

whom was hired in consultation with the state) and 4 embedded the state liaisons 

(two of whom were former SEA employees) within the SEA office space. Among 

the single-state RCCs, one hired a former SEA employee. 

Exhibit 3.4. Formal connections with state clients through RCC staffing 

Type of interaction

Multistate 

RCCs

(N=12)

Single 

state 

RCCs 

(N=4)

All 

RCCs

(N=16)

Has staff whose responsibilities include communication with 

state managers 12 4 16

Has state-specific staff positions designated for serving as a 

liaison to a state client organization 12 1 13

Hires former employees of state client organizations 6 1 7

Has staff positions designated for serving as an in-house 

liaison to a state client organization, maintaining office space 

within the client’s building

4 0 4

Involves clients in making hiring decisions about Center staff 2 0 2

EXHIBIT READS: Twelve multi-state RCCs had staff positions designated for serving as a liaison to a 

state client organization. 

SOURCE: Center responses to standard response categories during site visits. 
36

Planned and Conducted Center Activities 

 To be most helpful to the states, the Centers had to not only plan their technical 

assistance around the identified priorities but also provide services that addressed state needs that 

might shift during the year. Annual management plans written well in advance might in some 

ways be an asset to the Centers, enabling them to mount serious, coordinated, sustained efforts 

around their intended programs of work. At the same time, however, the plans might impose 

rigidity on Center technical assistance, impeding a flexible response to changing circumstances 

in particular states or nationwide. Whether the Centers delivered the technical assistance states 

asked for and what it looked like can be analyzed from reported Center activities and state 

agency managers‘ views. 

Adapting or Adhering to the Management Plan Overall 

At various points in the program year, Centers were confronted with new client service 

requests (up to100 new requests as indicated on their quarterly request logs) and learned about 

emerging client needs. As a result, Centers reported ongoing refinements to the general plan of 

work that they had laid out in the management plan. 

■ Most Centers reported providing services and products outside their original 

plan (exhibit 3.5). In retrospective interviews, 14 of the 21 Centers said that they 

conducted services that were outside their initially planned scope of work for the 

year. In fact, nine Centers indicated that they had never declined a client request 

during the year. RCCs indicated that they added work in response to emerging 

needs that they recognized in the process of carrying out planned work, and in 

response to requests from states (commonly new requests for quick-turn-around 

information) that the Centers viewed as consistent with or as an entrée toward 

their mission. Added work reported by the CCs was in response to requests from 

ED or the RCCs. Specific requests from ED included work in emerging topic 

areas such as work on response to intervention, high school restructuring, and 

alternate assessments. A total of 6 of the 21 Centers also reported having worked 

with primary service recipients who had not been identified in their planned 

scopes of work; providing assistance to different recipients was more common 

among CCs (four of five, or 80 percent) than among RCCs (2 of 16, or 12 

percent). 

■ More than half (57 percent) of the Centers reported declining some client 

requests (exhibit 3.5). Twelve of the 21 Centers reported having declined to 

provide particular technical assistance services. Among the 12 Centers that 

declined work, 8 indicated that they, at some point, had discussed with their 

clients (i.e., RCCs with states and CCs with RCCs) what they were and were not 

authorized to do, based on the legislation and the original grant Notice Inviting 

Applications (e.g., Center declined request to work with districts directly). 

Eight Centers reported having declined requests because they judged that the 

work wasn‘t on target with the best use of their resources, particularly as it related 
37

to building capacity (e.g., Center declined request to provide quick-fix help rather 

than longer-term capacity building); eight indicated they had declined requests 

because the work fell outside the parameters of their cooperative agreement with 

ED (e.g., Center declined request to work on topics that were narrow and/or 

unrelated to NCLB); and four Centers cited both of these reasons for having 

declined requests. 

Exhibit 3.5. Center strategies for adapting or adhering to the plan of work 

RCCs

(N=16)

CCs

(N=5)

All

(N=21)

Refer to parameters set by NCLB or the Center’s authorizing 

legislation when negotiating with clients to plan its work 13 2 15

Provide service outside of initial planned scope of work 10 4 14

Work on topics outside of planned scope of work 10 3 13

Work with primary service recipients outside planned 

scope of work 2 4 6

Decline to provide a service 9 3 12

Center has a staff member and process for assuring that 

Center work is within its management plan 9 2 11

EXHIBIT READS: Thirteen RCCs referred to parameters set by NCLB or the Centers’ authorizing legislation 

when negotiating with clients to plan their work. 

SOURCE: Center responses to standard response categories during site visits. 

Technical Assistance Delivered, in Comparison with Initial Plans and with 

Reported State Priorities 

One way to measure the stability and flexibility in the Centers‘ work is to assess the 

extent to which Centers followed or adjusted their initial plans with respect to the topical 

coverage of the technical assistance. We examined Center perspectives to indicate the extent to 

which initially planned technical assistance matched the technical assistance the Centers 

delivered, by topic. The states‘ reports on their priorities and their receipt of Center technical 

assistance indicated the extent to which the technical assistance actually delivered was 

responsive to state priorities. 

■ State survey data revealed a strong association between their priorities and their 

receipt of technical assistance from the Centers, although for each priority area 

there were states that wanted technical assistance but did not receive it. Through 

a survey, managers in all the SEAs were asked to identify their technical 

assistance priorities from a list of state responsibilities in the No Child Left 

Behind (NCLB) legislation. The bar chart in exhibit 3.6 compares the percentage 

of all state managers who expressed a priority for technical assistance on a 
38

particular NCLB responsibility with the percentage reporting that Centers 

provided them with major or moderate technical assistance on those 

responsibilities. For example, 98 percent of state managers reported that building 

or managing statewide systems of support was a major or moderate priority for 

Center technical assistance, and 72 percent said that they received a major or 

moderate amount of Center technical assistance in this area.41 An analysis of the 

similarity of the distributions of the two rankings of the 11 areas shows a 

statistically significant relationship, with a correlation coefficient of +0.95 (using 

Spearman‘s rho).42

Exhibit 3.6. State priorities for technical assistance and receipt of Center 

technical assistance, by area of NCLB responsibility 

EXHIBIT READS: Ninety-eight percent of states reported that SSOS was a major or moderate priority for technical 

assistance. Seventy-two percent of states reported that they received a major or moderate amount of technical 

assistance in that area.

 

41 Among the subgroup of state managers who reported a major or moderate priority for any particular state 

responsibility, at least 69 percent reported receiving some technical assistance with that responsibility from the 

Centers (a finding discussed in chapter 4; see exhibit 4.5). 

42 The Spearman‘s rho correlation statistic measures the degree of monotonic relationship between two variables 

that are arranged in rank order on ordinal scales. It differs from Pearson's correlation only in that the computations 

are done after the numbers are converted to ranks. It is a nonparametric statistic, and does not require that the 

distribution of the individual measures meet assumptions of normalcy or be interval data.

13

28

33

36

57

49

51

42

60

63

72

40

53

55

74

74

76

77

67

81

85

98

0 20 40 60 80 100

Communicating with the public about NCLB 

requirements or report cards

Monitoring compliance with NCLB requirements 

in districts and schools

Aligning state accountability systems with NCLB 

accountability systems

Providing training and professional development 

to local educators in academic subjects

Disseminating information on scientifically-based 

research to districts and schools

Supporting use of assessment data by schools and 

districts

Formulating or refining state policies to respond 

to NCLB requirements

Designing or implementing state assessment and 

accountability systems

Identifying and/or developing programs or 

models that address district and/or school needs

Training or managing the state-level staff or 

school support teams

Building or managing a statewide system of 

support

Percent

Major or

Moderate

Priority

Major or

Moderate

Assistance
39

SOURCE: Survey of State Managers. Responses weighted so that each state was represented by a composite in 

cases where more than one manager from that state responded 

■ Centers reported providing technical assistance in a majority (80 percent) of the 

instances where identified topic areas were included in their management plans 

(exhibit 3.7). Comparing the contents of the management plans with the annual 

PIFs) where Centers reported their delivered work to the evaluation team, of the 

133 topic-related objectives in the plans, the Centers provided services and 

products on 107 of them. The fact that some planned cases43 (133-107=26) were 

not listed on the PIFs indicates that 19 percent of the cases (26 of the planned 

133) were not conducted. Looking at the work actually conducted, in 38 percent 

of cases ([172-107]/172) the Center provided technical assistance on a topic not 

initially cited in its planned objectives. 

■ The work of the CCs was spread across the topic areas (exhibit 3.8). Only in the 

area of English language learners did more than one CC (2) plan and deliver work 

products or services. This is consistent with the intent of ED in defining specific 

areas of expertise for each of the five CCs. 

■ Centers commonly expanded the topic areas their technical assistance covered 

by adding work on topics they did not initially include in their plans. Looking at 

exhibit 3.7 and comparing columns 3 and 2, there are 19 of the 22 topics in which 

technical assistance was not included in a Center‘s management plan but was 

delivered by that Center. The most common shift toward topics were in the areas 

of response to intervention (seven Centers conducted work in this area that was 

not planned), English language learner issues (five Centers), highly qualified 

teacher provisions of NCLB (five Centers) and Supplemental Educational 

Services (five Centers). Based on exhibit 3.8, addition of work was more common 

among RCCs than CCs (in 19 of 22 topics, RCCs that did not have an objective in 

a topic conducted work on that topic; in comparison, in 7 of 22 topics CCs 

conducted work on a topic that that Center had not anticipated in its annual 

management plan).

 

43 Using each Center‘s management plan, the evaluation team coded for each Center whether it had specifically 

stated an intention/objective to address work in each of the 22 topics (i.e., the first column of numbers in exhibit 3.7 

indicates the number of Centers that stated an objective of conducting any work in each of the 22 topics, beginning 

with the topic of state systems of support). Using the PIFs, the evaluation team coded project topics to determine the 

number of Centers that actually did work in each of the 22 topics (i.e., the last column of numbers, by topic, in 

exhibit 3.7). 
40

Exhibit 3.7. Topics on which Centers stated objectives and/or delivered projects 

Topic

Number of Centers

(N=21)

Objective on 

topic was 

stated in 

management 

plan

(1)

Objective on topic was 

stated in management plan

AND at least one

project on topic was 

reported on PIF

(2)

At least one 

project on 

topic was 

reported on 

PIF

(3)

Total cases of a Center setting an 

objective and/or reporting a project 133 107 172

Components of effective systems of 

support—state, district, school 15 15 17

English language learners 10 10 15

Highly qualified teacher provisions of NCLB 8 5 10

Teacher professional development 6 5 8

Teacher preparation and induction 5 3 5

Data use/data-driven decisionmaking 8 4 7

Assessment design 6 5 5

Formative assessment 5 4 7

Special education—curriculum, instruction,

and professional development 8 3 6

Response to Intervention (RtI) 3 3 10

Special education—assessment 3 3 3

High school redesign/reform 8 7 10

Transition to high school 2 2 5

Dropout prevention 2 2 2

Mathematics 7 4 8

Adolescent literacy 6 6 8

Reading 6 5 8

Supplemental educational services (SES) 7 7 12

Parent involvement 7 7 11

Migrant education 5 3 6

Data management compliance 5 3 5

Indian/Native American education 1 1 4

NOTE: Column (1) - column (2) =the number of Centers that deleted work in the topic area as indicated by 

number of Centers that had a topic-related objective in their management plan minus the number of Centers that 

had the topic area on their PIF and their management plan; column (3)- column (2) =the number of Centers that 

added work in the topic area as indicated by number of Centers that reported work on their PIF minus the 

number of Centers that had the topic area on their PIF and their management plan.

EXHIBIT READS: For the topic ―Components of effective systems of support—state, district, school,‖ 15 Centers 

had a related objective in their respective annual management plans; of these, all 15 reported projects on the 

topic in their PIF; and a total of 17 Centers had projects on the topic, whether or not they had stated an objective 

related to it in their management plan. 

SOURCES: Center management plans for 2006-07; PIFs prepared by Centers in consultation with evaluation 

team. 
41

Exhibit 3.8. Topics on which RCCs and CCs stated objectives and/or delivered 

projects

Number of RCCs (N=16) Number of CCs (N=5)

Objective Objective on 

on topic in topic in plan AND 

manage- project(s) on 

ment plan topic on PIF

(1) (2)

Project(s) 

on topic on 

PIF

(3)

Objective 

on topic in 

management plan

(4)

Objective on 

topic in plan AND 

project(s) on 

topic on PIF

(5)

Project(s) 

on topic on 

PIF

(6)

Total cases 112 89 146 21 18 26

Components of effective 

systems of support—state, 

district, school

English language learners

Highly qualified teacher 

provisions of NCLB

Teacher professional 

development

Teacher preparation and 

induction

Data use/data-driven 

decisionmaking

Assessment design

Formative assessment

Special education –

curriculum, instruction, 

professional development

Response to Intervention 

(RtI)

Special education–

assessment

High school 

redesign/reform

Transition to high school

Dropout prevention

Mathematics

Adolescent literacy

Reading

Supplemental educational 

services (SES)

Parent involvement

Migrant education

Data management

compliance

Indian/Native American 

education

14

8

7

4

4

7

5

4

6

3

2

7

1

1

6

5

5

6

6

5

5

1

14

8

4

4

2

3

4

3

3

3

2

6

1

1

3

5

4

6

6

3

3

1

15

12

9

7

4

5

4

5

5

8

2

9

4

1

7

6

7

11

10

6

5

4

1

2

1

2

1

1

1

1

2

0

1

1

1

1

1

1

1

1

1

0

0

0

1

2

1

1

1

1

1

1

0

0

1

1

1

1

1

1

1

1

1

0

0

0

2

3

1

1

1

2

1

2

1

2

1

1

1

1

1

2

1

1

1

0

0

0

NOTE: column (1)-column (2) =the number of RCCs that deleted work in the topic area as indicated by number of Centers that 

reported work on their management plan minus the number of Centers that had the topic area on their PIF and their management 

plan ; column (3)- column (2) =the number of RCCs that added work in the topic area as indicated by number of Centers that 

reported work on their PIF minus the number of Centers that had the topic area on their PIF and their management plan. These 

calculations are analogous for the CCs.

EXHIBIT READS: For the topic ―Components of Effective Systems of Support—State, District, School,‖ 14 RCCs reported a related 

objective in their management plans; of these, all 14 reported projects on the topic; and a total of 15 RCCs had projects on the topic, 

whether or not they had originally stated an objective related to it. 

SOURCES: Center management plans for 2006-07; PIFs prepared by Centers in consultation with evaluation team. 
42

■ Centers also shifted away from providing work on originally planned topics. A 

comparison of columns 2 versus 1 in exhibit 3.7 indicates that in 10 of the 22 

topics, Centers that set an objective in a topic did not deliver a project related to 

the topic—in other words, there was a shift away from initially stated objectives 

on these topics. The two topics in which the largest number of Centers planned to 

do work but did not deliver a project were special education curriculum, 

instruction, and professional development (five Centers); and data use or datadriven decision making (four Centers). As seen in exhibit 3.8, deletion of work 

was more common among RCCs than CCs (in 11 of 22 topics, one or more RCCs 

that reported having an objective in a topic in its management plan did not report 

a project on the topic in its PIF; in comparison, in 2 of the 22 topic areas, CCs that 

had planned work did not deliver a project in that topic). 

 It is not possible to ascertain from the data whether these shifts from initial objectives to 

projects delivered took place at the request of clients. Based on interviews with the Centers, 8 of 

the 16 RCCs said they changed the work they planned to deliver during the year because of 

changes in SEA priorities or leadership. One CC similarly said that, as a result of turnover of a 

key client in the state, it had to put on hold collaborative work it had with an RCC that was to be 

delivered to the SEA. In addition, the ratings of relevance and usefulness that state managers 

give to the Centers‘ work (see chapter 4) provide an indication that the changes in work plans 

may have been in response to evolving state needs. 

Activities and Products Provided by the Centers 

 Regardless of whether they were originally specified in work plans or added later, what 

types of technical assistance did the Centers provide? As described above, the most common 

topics on which Centers reported delivering assistance were systems of support, English 

language learners, high school redesign/reform, supplemental education services, and parent 

involvement (exhibit 3.7). 

An analysis of 122 sampled projects (described in more detail in chapter 2) provides 

more in-depth information about the nature of Center technical assistance activities or resources. 

Although the sample of projects is not statistically representative of the Centers‘ work, recall that 

project sample selection for review of each Center‘s work favored their most dominant projects 

and included over half (56 percent) of that year‘s designated major or moderate projects. A 

project is defined for this evaluation as ―a group of closely related activities and/or deliverables 

designed to achieve a specific outcome for a specific audience.‖

■ Most (102 of 122 = 84 percent) projects included the delivery of more than one 

type of activity or resource (exhibit 3.9). Projects with a single coded type of 

activity/resource were more common among the CC than the RCC sampled 

projects (9 of 27 = 33 percent and 11 of 96 = 11 percent, respectively), illustrating 

the differing nature of their work. The evaluation team coded the following seven 

types of activities and resources that Center projects could provide: ongoing 

consultation and follow-up, research collections and syntheses, engagement of 
43

participants in project planning, training events, task force meetings and work, 

conferences, and support for development of a formal plan to implement a 

program or policy. 

Exhibit 3.9. Sampled Center projects by number of activity or resource types 

provided 

Number of activity or 

resource types:

Total projects

One

RCC 

projects

(n=96)

96

11

CC 

projects

(n=27)

27

9

All 

projects

(n=122)

122

20

Two 22 8 30

Three 26 3 29

Four 16 4 19

Five 16 3 19

Six 5 0 5

EXHIBIT READS: Eleven RCC projects provided a single type of activity or resource for participants. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. The 

total number of projects was 122. One project collaboratively conducted by an RCC and a CC was 

counted among both RCC projects and CC projects but was only counted once among the projects of 

all Centers. 

■ The kinds of support provided by the RCCs and CCs differed in ways that are 

consistent with the model of technical assistance envisioned by ED 

(exhibit 3.10). The guidance given by ED through the Center grant competition 

and afterwards laid out a particular structure for the Centers‘ work: CCs would 

specialize in activities that require content specialization while the RCCs would 

specialize in state client interactions. The majority of RCC projects involved 

front-line assistance activities such as ongoing consultation and follow-up (79 of 

96 projects, or 82 percent), whereas this was less so for the work of the CCs (6 of 

27 projects, or 22 percent). In contrast, research collections and syntheses were a 

central part of the work of the CCs, occurring in 74 percent of their projects but 

54 percent of the RCC projects. The second most common project activity for 

CCs (63 percent) was conferences, an activity that also was consistent with the 

content specialization model promoted by ED. 
44

Exhibit 3.10. Sampled Center projects by types of participant activities and 

products 

Activities and products (with clarifying definitions used by 

coders)

RCC 

projects

(n=96)

CC 

projects

(n=27)

All 

projects

(n=122)

Ongoing consultation and follow-up (multiple contacts to same 

participants, that were part of a coherent and purposeful whole) 79 6 84

Research collections and syntheses 52 20 71

Engagement of participants in project planning (more than needs 

assessment or identifying participants) 43 8 50

Training events (focused on implementing a specific program or 

strategy) 41 10 50

Task force meetings and work (focused on addressing a specific 

problem, program, or policy) 48 2 50

Conferences (symposium, forum, institute; highlights a range of 

perspectives, strategies, or programs) 26 17 43

Support development of a formal plan to implement a program or 

policy 18 2 20

EXHIBIT READS: Seventy-nine RCC projects included ongoing consultation and follow-up. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. The total number 

of projects was 122. One project collaboratively conducted by an RCC and a CC was counted among both RCC 

projects and CC projects but was only counted once among the projects of all Centers. 

Knowledge Base Used by Centers 

An important part of Centers‘ operations is how and to what extent they draw on 

expertise to provide a solid foundation for recommended strategies and practices. We asked 

Centers about the sources they drew on in developing or reviewing their technical assistance, 

including their reliance on other Centers as sources. 

Sources of Content Expertise 

The analysis investigated the mix of in-house sources, scholarly sources, federally supported 

resources of various kinds, and state-level practitioners that were sources for each type of Center. A 

particular focus was the extent to which the RCCs used CC expertise, since that was a key 

component of how ED thought the Center system would operate. Of additional interest was the 

extent to which the CCs enlisted the RCCs or states as a source of knowledge since the Centers 

were charged with identifying and disseminating promising practices (exhibit 3.11). 

■ All 16 RCCs reported that CC expertise contributed to the knowledge base for 

their products and services, a finding consistent with the program design. To 

communicate with CCs and stay abreast of their activities, based on interviews 
45

with the Centers, all 16 RCCs reported using the CCs in developing their products 

and services. In fact, the RCCs reported designating staff to serve as liaisons to 

the CCs, and all five CCs described activities that involved these liaisons. This 

organizational strategy may have contributed to RCC use of CC expertise. 

■ The CCs were less likely to draw on the RCCs as a source of expertise 

(exhibit 3.10). Two of the five CCs (40 percent) reported that they relied on RCCs 

in the process of developing their materials or services. 

■ Every RCC and every CC reported using in-house expertise from its own staff. 

For all Comprehensive Centers, the grantees and/or subgrantees were affiliated 

with larger organizations that also included personnel who were not officially part 

of the Center (all grantees and subgrantees are listed in appendix A). Of the 16 

RCCs, 15 used the expertise of non-Center staff in their lead or subgrantee 

organizations, 13 used staff in their lead organization and 10 in their subgrantee 

organizations. Of the five CCs, three reported using non-Center staff from lead 

and subgrantee organizations. 

■ After drawing on their own organizations, the RCCs and CCs relied on different 

sources. Other than using the CCs, the RCCs most often reported turning to 

professional associations and other RCCs as sources of expertise (each type of 

source was reported by 11 of the 16 RCCs). The source most frequently cited by 

CCs, after internal Center staff, was consulting firms or other private contractors 

(four of the five CCs). 

■ RCCs were more likely than CCs to draw on state agency expertise. Because 

Centers were charged with identifying and disseminating promising practices 

from practitioner sources, especially states, the evaluation assessed the extent to 

which knowledge flowed to the Centers from their state clients. Ten of the 16 

RCCs (62 percent), those charged with working directly with the states, identified 

SEAs as a source of expertise. In contrast, one of five CCs (20 percent) identified 

SEAs as a source of expertise. Given the Center structure of the CCs working 

through the RCCs, the CCs could learn about state practices through the RCCs; 

however, three of the five CCs did not indicate RCCs as a source. 

■ Regional Educational Laboratories, with which the Centers were expected to 

cooperate, were reported as a source of expertise by 10 of the 16 RCCs and 1 of 

the 5 CCs. 
46

Exhibit 3.11. Sources of content expertise used in developing Center products 

and services 

Source

RCCs

(N=16)

CCs

(N=5)

All

(N=21)

Internal Center staff (lead organization and/or subgrantees) 16 5 21

Content Center(s) (CCs) 16 3 19

Other staff from lead and subgrantee organizations, who are not 

formally employed by the Center 15 3 18

Other staff in the lead organization 13 2 15

Other staff in subgrantee organization(s) 10 2 12

Professional associations (e.g., AERA, CCSSO) 11 3 14

Consulting firms or private contractors 9 4 13

Regional Center(s) (RCCs) 11 2 13

Other federally funded technical assistance providers (not part of the 

Center) 8 3 11

Regional labs (not part of the Center) 10 1 11

State education agencies (SEAs) 10 1 11

Universities 7 3 10

What Works Clearinghouse 7 1 8

EXHIBIT READS: All sixteen RCCs relied on internal Center staff as sources of content expertise used in 

Center products and services. 

SOURCE: Center responses to standard response categories during site visits. 

Extent of RCC-CC Coordination 

 As part of the assessment of the way the Centers functioned as a system, the evaluation 

also looked at the extent to which the two types of Centers coordinated with or informed each 

other, both at the Center level (as described above, in the description of Centers‘ reliance on one 

another as sources of expertise) and also at the project level. Thus, in the interviews Centers were 

asked how they informed each other of their work. The evaluation team also identified the 

sampled RCC projects in which a CC was a source of materials, advice, or in-person assistance 

to clients, and the CC projects in which an RCC participated by recruiting participants, brokering 

service, or delivering assistance to clients. 

■ Periodic conference calls and specific RCC requests to CCs were the primary 

mechanism for coordination between CCs and RCCs. All five CCs offered 

regular conference calls for their designated liaisons at the RCCs in order to 
47

convey their work to the RCCs. In addition, all CCs used these conference calls, 

sometimes in combination with webinars, to conduct RCC staff training and to 

identify staff needs. All RCCs reported making direct requests for CC assistance 

when they felt it would be useful. Finally, ED sponsored bi-annual Directors‘ 

meetings that provided opportunities for RCCs and CCs to coordinate.

■ Some informal coordination took place among RCCs. Although ED has few 

formal mechanisms for RCCs to share information about their work with each 

other, all RCCs reported in interviews that they communicated with other RCCs, 

most often those with which they had formal organizational relationships (i.e., the 

same parent organization, or a subgrant with the other RCC‘s parent 

organization). Reports of viewing other RCC websites, registering for webinars, 

and direct phone conversations were additional methods staff mentioned as 

informal ways to learn about other RCC work. 

■ Half (48 percent) of the RCC projects had a CC contribution of one or more 

kinds (exhibit 3.12). Forty-six of the 96 projects were found to have had input 

from a CC. The most frequent type of CC contribution to RCC projects was in 

material support (42 projects), including practice briefs, research syntheses, and 

implementation manuals. Seventeen RCC projects included CC staff providing 

technical assistance directly to project participants, such as to state workgroups or 

at conference events. 

Exhibit 3.12. Sampled RCC projects by CC contribution 

CC contribution

RCC 

projects

(n=96)

RCC project had no CC contribution 50

RCC project had a CC contribution 46

CC provided materials used in this RCC project 42

CC provided advice to the RCC on this project 16

CC delivered technical assistance to project participants 17

EXHIBIT READS: In 50 RCC projects, a CC did NOT contribute. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. 

■ A smaller proportion of CC projects had an RCC contribution of one or more 

kinds (exhibit 3.13). Ten of 27 sampled CC projects were documented to have 

input from the RCCs. In each of these 10 projects, RCCs recruited participants or 

brokered CC services. In 3 of them, or 11 percent of the sample of 27 CC 

projects, RCCs also delivered assistance to project participants (e.g., presented at 
48

a conference, contributed content for a product, delivered follow-up assistance 

that continued work initiated by a CC). 

Exhibit 3.13. Sampled CC projects by RCC contribution

RCC contribution

CC 

projects

(n=27)

CC project had no RCC contribution 17

CC project had an RCC contribution 10

RCC recruited participants or brokered service 10

RCC delivered technical assistance to project participants 3

EXHIBIT READS: In 17 CC projects, an RCC did NOT contribute. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. 

Product Quality Control Procedures 

 The evaluation inquired about the procedures and types of individuals brought to bear in 

the process of reviewing products before release. Given that project quality is assessed as part of 

the evaluation, it is useful to understand the processes that the Centers themselves reported using 

to promote quality of the products used in their technical assistance. 

■ Most RCCs and all CCs relied, in part, on internal staff for quality control 

(exhibit 3.14). When asked about how they reviewed the content of their technical 

assistance prior to delivery—i.e., what quality assurance process they used—14 of 

the 16 RCCs and all five of the CCs described processes that included internal 

staff review of their work. Of the two that did not describe a vetting process that 

included internal staff, one relied on an external source and another did not clearly 

identify a process. 

■ CCs were more likely than RCCs to report having external sources review their 

products (exhibit 3.14). All five CCs (100 percent) reported that they submitted 

all major publications to ED for review prior to public release, as they were 

required to do. Also, all five CCs (100 percent) reported using other external 

sources for review, including four CCs (80 percent) that reported retaining outside 

experts to review draft products and two CCs (40 percent) that described relying 

on other CCs for feedback on their own products as part of the five CCs 

developing a common vetting tool. In contrast, two (12 percent) of the RCCs said 

they used outside experts as a source for vetting. As important context, the nature 

of the products and services conducted by the RCCs and the CCs differs, with the 

RCC work consisting of fewer formal products than that of the CCs. 
49

Exhibit 3.14. Sources for vetting Center products and services prior to delivery 

Source

RCCs

(N=16)

CCs

(N=5)

All

(N=21)

Internal Center staff (lead organization and/or subgrantees) 14 5 19

Other staff from lead and subgrantee organizations, not formally 

employed by the Center 6 3 9

Other staff in the lead organization 5 3 8

Other staff in subgrantee organization(s) 5 2 7

External sources 10 5 15

U.S. Department of Education 6 5 11

Content Center(s) 6 2 8

Outside experts retained to review drafts 2 4 6

EXHIBIT READS: Fourteen RCCs relied on internal Center staff as sources for vetting Center products prior to 

release. 

SOURCE: Center responses to standard response categories during site visits. 

51

4. State Managers’ Assessment of the Centers’

Technical Assistance 

In any evaluation, performance is judged by whether the program meets its goals and 

objectives. The Comprehensive Centers were expected to provide technical assistance that would 

build states‘ capacity to carry out their responsibilities under NCLB, supporting district and 

school efforts to close achievement gaps and raise student achievement. Given the diffuse and 

indirect relationship between state-level technical assistance and student outcomes, it is 

impossible to attribute changes in test scores to the work of the Centers. It is possible, however, 

to determine whether state agency managers felt the technical assistance addressed their 

agencies‘ purposes and expanded agency capacity. 

 

This chapter examines how the work of the Comprehensive Centers was regarded by key 

clients, specifically senior managers in state education agencies. Although the assistance might 

vary in quality or utility across Centers or projects (see chapter 5), as a first step it is important to 

understand how states viewed the output of the Comprehensive Centers program in a broad 

sense. We asked state managers whether the technical assistance provided by their RCC and the 

Content Centers served their agency‘s purposes, whether it had addressed each of several 

specific areas of state responsibility under NCLB, to what extent it had expanded state capacity, 

and how it compared with technical assistance from other sources. 

The key findings include: 

■ Eighty-eight percent of state managers (weighted)44 reported that the Centers’ 

technical assistance was at least “a good start” toward serving state purposes, 

with 36 percent overall indicating that the Centers’ assistance “served the 

state’s purposes completely.” Managers who said the state‘s purposes were not 

completely served reported a variety of different issues. For 33 percent of them 

(or 21 percent of all state managers), the Centers were too slow in getting 

assistance up and going; 33 percent (21 percent overall) reported that the state‘s 

most important priorities for technical assistance were outside the scope of the 

Centers‘ work; and 31 percent (20 percent overall) reported that the state secured 

most of its technical assistance from sources other than the Centers. 

■ For each of the four areas of NCLB implementation most widely identified as 

state priorities for technical assistance (see chapter 3), at least 90 percent of the 

state managers who had identified that area as a priority had received 

 

44 For 20 states, the evaluation team received completed responses from more than one state manager. As the state 

was the primary unit of analysis in analyzing data from the state manager survey for this report, the state managers‘ 

responses were weighted to ensure that each state was equally represented in all summary statistics while taking into 

account the variation in responses within each state. The weighting procedure, where each response was weighted 

by the inverse of the number of managers responding from that state, ensured that each state was equally represented 

when the team aggregated responses across states to describe the distribution of responses. See chapter 2 for a 

detailed description of the sampling and analysis procedures. 
52

assistance with it from the Centers. In the area of building or managing a 

statewide system of support, 98 percent of the state managers who identified it as 

a priority said the Centers provided technical assistance with it; the corresponding 

figures were 95 percent for identifying programs or models for districts or 

schools, 90 percent for training or managing state support teams, and 97 percent 

for disseminating information on scientifically based research. 

■ Most (68 percent) of the state managers reported that Center assistance had 

“greatly” expanded their agency’s capacity in at least one of 11areas of NCLB 

responsibility about which they were surveyed. 

■ State managers reported that the Comprehensive Center network was one of 

multiple sources that they used for technical assistance. On average, state 

managers ranked the Centers as one of the top three sources of technical 

assistance that they relied upon, along with professional associations and the EDfunded Regional Educational Laboratories. The purposes for which states used the 

Centers more than other sources were ―to plan the initial steps in solving a 

problem‖ (reported as a purpose for Center technical assistance by 66 percent of 

state managers) and ―to develop the skills of SEA or intermediate education 

agency staff‖ (61 percent of state managers). 

■ Overall, on a scale of 1(very low degree) to 5 (very high degree), state managers 

gave the Centers’ early technical assistance close to a 4. The average score for 

 ̳relevance‖ was 3.94 and the average score for the ―usefulness‖ of the Center‘s 

work was 3.86. These scores fell just below the 4.0 threshold defined as ―to a high 

degree‖ on the 5-point scale that state managers used to determine their responses. 

Extent to Which Center Assistance Served State Purposes 

 ED placed a significant emphasis on having the Centers, both RCCs and CCs, deliver 

technical assistance that would advance state efforts to implement NCLB. In developing their 

work plans and delivering technical assistance, the Centers were expected to target their work on 

state concerns and priorities, as discussed in chapter 3. States‘ views of how well the assistance 

met their own purposes provided one perspective on the success of the program. 

■ More than one third (36 percent) of state managers reported that Center 

technical assistance “served the state’s purposes completely,” with another 52 

percent reporting that “it was a good start.” Six percent said the technical 

assistance ―was a start, but some important priorities were not addressed‖; the 

same percentage said that, ―for the most part, it did not serve the state‘s purposes‖

(exhibit 4.1).
53

Exhibit 4.1. Extent to which technical assistance from the Centers served state 

purposes, as judged by state managers 

Extent

Percent of state 

managers (weighted)

(n=56)

It served the state’s purposes completely 36

It was a good start 52

It was a start, but some important priorities were 

not addressed 6

For the most part, it did not serve the state’s 

purposes

6

EXHIBIT READS: Thirty-six percent of state managers (weighted) reported that Center 

technical assistance served the state’s purposes completely. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally 

represented in instances where more than one manager from a state responded. 

■ The two most common reasons cited for the Centers not fully serving state 

purposes were the length of time it took to get projects underway and the 

identification of state priorities that were outside the Center’s line of work. The 

survey posed a series of follow-up questions asking why, to the respondents who 

did not report that Center assistance ―served the state‘s purposes completely‖

(exhibit 4.2). These state managers were asked to check off as many as eight 

possible reasons that the assistance was less helpful to their state than it might be. 

Among these 30 state managers, more than half (56 percent) identified a single 

reason, a quarter (27 percent) identified two reasons, and 15 percent three or more 

reasons. Thirty-three percent (21 percent of all state respondents) cited (1) the 

process of negotiating a work scope and organizing projects took too long, and (2) 

the state‘s most important priorities for technical assistance fell outside the 

Center‘s scope of work. Out-of-scope requests for technical assistance would 

include, for example, working directly with local districts and schools. 

■ The next most common reasons reflected states’ use of, or desire to use, sources 

other than the Comprehensive Centers for technical assistance. In 31 percent of 

states whose purposes were not fully served (20 percent overall), managers 

reported that the state filled most of its technical assistance requirements from 

other sources (exhibit 4.2). Asked whether the state would prefer to contract 

directly with experts and consultants rather than receiving technical assistance 

from the Comprehensive Centers, 19 percent of these respondents (12 percent of 

state managers overall) agreed. The survey responses of these managers did not 

suggest a high level of dissatisfaction with the Center assistance or a heavy 

reliance on other sources, however. Among those who had given either or both of 

these explanations for less than complete satisfaction with Center assistance, 50 

percent nevertheless reported that they used the Centers ―to a great extent.‖ There 
54

was no other source of assistance that this group of managers reported using more 

than they used the Centers, and no source that they reported using more than the 

other state managers did.

Exhibit 4.2. Reasons why Center assistance served some states’ purposes 

less than completely 

Reasons

Percent of those state managers 

(weighted) who reported Center 

assistance had not served their 

purposes completely (n=36)

Yes No

The process of negotiating a work scope and 

organizing projects takes too long 33 67

The state’s most important priorities for assistance 

fall outside the Comprehensive Center’s scope of 

work

33 67

The state secures most of the technical assistance it 

needs from other sources 31 69

The state would prefer to locate and contract directly 

with the experts or consultants from whom it needs 

assistance, rather than working through the 

Comprehensive Centers

19 81

Comprehensive Center staff are not able to spend as 

much time working with the state as we would like 17 83

The Comprehensive Center does not have the 

expertise the state needs 15 85

A policy or priority shift at the state level caused the 

Center’s assistance to be less helpful than it might 7 93

The state has been unable to develop a productive 

working relationship with the Center 3 97

EXHIBIT READS: Of the 36 state managers (weighted) reporting Center assistance served their state’s 

purposes less than completely, 33 percent said a reason was that the process of negotiating a work scope 

and organizing projects takes too long. 

SOURCE: Survey of State Managers. Respondents limited to those who did not answer ―It served our 

purposes completely‖ to the preceding question. Responses weighted so that each state was equally 

represented in instances where more than one manager from a state responded. 

Reported Assistance Delivery and Capacity Building for State 

NCLB Responsibilities 

 The primary aim of the Center program was to deliver technical assistance that would 

both address state priorities in implementing NCLB and build state capacity for NCLB 
55

implementation. Senior state managers reported their perceptions of the extent to which the 

Centers had delivered assistance and built capacity in the areas of NCLB responsibility that were 

technical assistance priorities for their state. 

■ For each of the four areas of NCLB implementation most widely identified as 

state priorities for technical assistance, at least 90 percent of the state managers 

who identified that area as a priority had received assistance with it from the 

Centers. The corresponding percentage reporting at least a moderate level of 

assistance was at least 74 percent (exhibit 4.3) for each of four areas (building 

statewide systems of support, identifying programs, training statewide support 

teams, and disseminating information on scientifically based research). In the area 

of building or managing a statewide system of support, 98 percent of the state 

managers who identified it as a priority said the Centers provided technical 

assistance with it. 

■ There were four areas in which the Centers reportedly had not delivered 

assistance to 20 percent or more of the state managers for whom the area was a 

priority. These areas were communicating with the public (an area in which 31 

percent of the 16 states identifying it as a priority had not received Center 

assistance), monitoring compliance with requirements (29 percent of 24 states), 

providing training to local educators in academic subjects (21 percent of 29 

states), and designing or implementing state assessment or accountability systems 

(20 percent of 25 states) (exhibit 4.3). These four areas were also among the least 

widespread priorities for technical assistance among the states: the only one that 

was a moderate or greater priority for more than half of the states was training for 

local educators in academic subjects. 
56

Exhibit 4.3. State priorities for technical assistance and assistance 

received from Comprehensive Centers 

Priority area of state responsibility under NCLB

Among state managers reporting 

the responsibility as a major or 

moderate priority, the percent 

reporting they received:

Any assistance 

from the Centers 

related to the 

responsibility

Major or 

moderate

assistance from 

the Centers 

related to the 

responsibility

Building or managing a statewide system of support for 

districts and schools identified for improvement under NCLB 

(n=46)

98 80

Training or managing the state-level staff or school support 

teams who provide support to districts and schools identified 

for improvement under NCLB (n=41)

90 78

Identifying and/or developing programs or models that 

address district and/or school needs (n=38) 95 79

Disseminating information on scientifically based research to 

districts and schools (n=36) 97 74

Supporting use of assessment data by schools and districts 

(n=36) 86 66

Formulating or refining state policies to respond to NCLB 

requirements (n=34) 94 74

Providing training and other professional development to local 

educators in academic subjects (reading language arts, 

mathematics, science) (n=29)

79 59

Designing or implementing state assessment and 

accountability systems (n=25) 80 65

Monitoring compliance with NCLB requirements in districts 

and schools (n=24) 71 50

Aligning state accountability systems with NCLB accountability 

systems (n=21) 81 62

Communicating with the public about NCLB requirements or 

report cards (n=16) 69 33

EXHIBIT READS: Of the 46 state managers (weighted) that reported a major or moderate priority for technical 

assistance with building or managing a statewide system of support, 98 percent reported receiving technical 

assistance from the Centers related to that responsibility, and 80 percent reported that the technical assistance was 

major or moderate in scope. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 
57

 The intent of the Comprehensive Centers program was that technical assistance from the 

Centers would expand state capacity to carry out important responsibilities under NCLB. 

Although future data collection will look at this in more depth, the survey provided perceptions 

from managers on the extent to which state capacity had been expanded in each area of state 

NCLB responsibility. 

■ Overall, more than two-thirds of state managers (68 percent) reported that 

assistance from the Comprehensive Centers had increased their state’s 

capacity “to a great extent” to carry out its responsibilities in at least one 

NCLB area. On average, state managers reported an improvement in capacity to 

―very great‖ or ―great‖ extent in 3.6 out of 11 areas of NCLB responsibility 

listed on the survey instrument.

The evaluation looked not just at overall reports of state capacity expansion, but also at 

the capacity building reported in particular areas of NCLB responsibility by those states for 

which the area was a priority. 

■ For each of the four top-ranked areas of priority in NCLB (as described in 

chapter 3), at least two-thirds of state managers for whom that area was a 

technical-assistance priority perceived that Center assistance had expanded 

state capacity to a great or moderate extent. This level of capacity building was 

reported by 67 percent of the managers in building or managing a statewide 

system of support, 71 percent in identifying programs or models for districts or 

schools, 71 percent in training or managing state support teams, and 77 percent 

in disseminating information on scientifically based research. And in at least 49 

percent of the states where any particular NCLB area was a major or moderate 

priority, managers reported great or moderate levels of capacity expansion for 

that area (exhibit 4.4). 

State Uses and Perceptions of the Centers Compared with 

Other Sources of Technical Assistance 

With many other sources available to states, the Comprehensive Centers were designed to 

fill particular niches—in particular, that of building capacity for state implementation of NCLB 

requirements—rather than addressing every purpose for which states might seek technical 

assistance. The responses of state managers confirmed that the states relied upon multiple 

sources of technical assistance in their practice. 
58

Exhibit 4.4. Extent to which Comprehensive Center Assistance expanded state 

capacity for NCLB implementation, as judged by senior state managers 

Area of state responsibility

under NCLB

Percent of the state managers (weighted) who rated the area 

as a major or moderate technical assistance priority

Reporting capacity building in this area NA, state has 

not sought 

assistance 

for this 

purpose

To a very 

great or 

great extent

To a 

moderate 

extent

To a small 

or very 

small extent

Too soon 

to tell

Building or managing a statewide system 

of support for districts and schools 

identified for improvement under NCLB 

(n=53)

53 14 18 8 7

Training or managing the state-level staff 

or school support teams who provide 

support to districts and schools identified 

for improvement under NCLB (n=46)

51 20 12 7 10

Identifying and/or developing programs or 

models that address district and/or school 

needs (n=42)

47 24 8 11 10

Formulating or refining state policies to 

respond to NCLB requirements (n=40) 43 14 14 11 18

Supporting use of assessment data by 

schools and districts (n=40) 42 16 11 2 29

Disseminating information on scientifically

based research to districts and schools 

(n=38)

39 38 10 5 8

Providing training and other professional 

development to local educators in 

academic subjects (reading language arts, 

mathematics, science) (n=38)

33 17 11 9 30

Designing or implementing state 

assessment and accountability systems 

(n=35)

33 25 5 4 33

Aligning state accountability systems with 

NCLB accountability systems (n=28) 34 24 6 9 27

Monitoring compliance with NCLB 

requirements in districts and schools 

(n=25)

34 22 21 8 15

Communicating with the public about 

NCLB requirements or report cards (n=19)

26 23 7 10 34

EXHIBIT READS: Among the 53 state managers (weighted) who reported that technical assistance in building or 

managing a statewide system of support was a major or moderate priority for their state, 53 percent reported that 

technical assistance received from the Comprehensive Centers expanded the state’s capacity in this area to a great 

or very great extent. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 
59

■ Centers were ranked among the top sources of technical assistance overall. 

Thirty-five percent of senior state managers identified the Comprehensive Centers 

as ―one of the state‘s most important resources,‖ second to professional 

associations (37 percent) and above the Regional Educational Laboratories 

(33 percent) (exhibit 4.5). 

Exhibit 4.5. States’ use of external sources of technical assistance 

External source

Percent of state managers (weighted) reporting:

One of the 

state's most 

important 

resources

To a great 

extent, but 

not one of the 

state's most 

important 

resources

To a moderate 

extent Minimally No contact

Professional associations (e.g., 

CCSSO, ASCD) (n=54) 37 24 28 9 2

Comprehensive Center network 

(n=54) 35 37 14 12 2

Regional Educational 

Laboratory (n=53) 33 29 21 12 5

U.S. Department of Education 

(n=52) 29 16 35 19 1

Colleges and universities 

(n=54) 18 15 34 30 3

Consulting firms or private 

contractors (n=54) 13 29 33 19 6

Senior managers in other SEAs 

(n=54) 9 29 37 23 2

EXHIBIT READS: Thirty-seven percent of state managers (weighted) said that professional associations were one of 

the state’s most important technical-assistance resources. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

State managers‘ reports of the purposes for which they used each source of technical 

assistance shed light on the niches occupied by the Center program—the purposes for which the 

Centers were the most commonly used source. These responses also revealed some purposes for 

which other sources were the most widely used. 

■ The purposes for which the Centers were the source most often named by the 

state managers were consistent with the Center program’s purpose, including 

help with start-up and capacity building for state tasks. Using Center 

assistance ―to plan the initial steps in solving a problem‖ was reported as a 

purpose by 66 percent of state managers; using Centers ―to develop the skills 
60

of SEA or intermediate education agency staff‖ was reported by 61 percent of 

state managers (exhibit 4.6). Centers were tied with Regional Educational 

Laboratories as a resource ―to complete tasks for which the state lacks 

expertise,‖ with each of these two sources reported by 54 percent of states as a 

resource for this purpose. 

■ State managers did not cite Centers as a leading resource for three purposes 

that were peripheral to their charge: to gather information or keep current 

with new ideas, to complete tasks that the state could do itself if it had more 

resources, and to work directly with districts and schools. To help states ―keep 

current with new ideas,‖ 96 percent of state managers reported using 

professional associations, 88 percent reported using managers in other SEAs, 

and 87 percent reported using the Centers. For ―completing tasks the state could 

do itself if it had more staff or resources,‖ 63 percent reported using consulting 

firms or private contractors, and 59 percent reported using the Centers. For 

working with districts and schools, colleges and universities were reported to be 

used by 37 percent of state managers and consulting firms by the same 

percentage; 22 percent reported using the Centers. 
61

Exhibit 4.6. Purposes for which states used external sources of technical 

assistance 

EXHIBIT READS: Ninety-six percent of state managers (weighted) said that they turned to professional associations 

for the purpose of gathering information or keeping current with new ideas. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

 State managers also provided a rating of the usefulness of Center technical assistance, in 

comparison with other sources, with respect to particular areas of NCLB state responsibility 

(exhibit 4.7).

■ With regard to the top four areas of NCLB implementation identified by the 

state managers as priorities for technical assistance (see chapter 3), at least half 

of state managers reported that the Centers’ technical assistance was “much 

more” or “somewhat more” useful than that of other providers. When asked to 

compare the usefulness of Comprehensive Center assistance with that of 

assistance from other sources, at least half of state managers reported the Centers 

were more useful in providing assistance in the following areas: building and 

managing statewide systems of support (58 percent of state respondents); 

identifying or developing programs or models for districts or schools (57 percent); 

training or managing the state-level staff or school support teams (55 percent); 

and disseminating information on scientifically based research to districts and 

schools (50 percent). 

Purpose in se e king te chnica l a ssista nce

(pe rce nt of sta te ma na ge rs)

External Source To gather information or to keep current with new ideas To plan the initial steps in solving a problem To complete tasks that the state could do itself if it had more staff or resources To complete tasks for which the state lacks expertise To develop the skills of SEA or intermediate education agency staff To work directly with districts and schools Our state doesn’t seek assistance from this source

Professional associations (e.g., 

CCSSO, ASCD) (n=52) 96 51 43 36 57 14 5

Comprehensive Center network 

(n=54) 87 66 59 54 61 22 4

Regional Educational Laboratory 

(n=55) 84 57 45 54 47 20 4

U.S. Department of Education 

(n=52) 85 40 22 20 21 12 7

Colleges and universities (n=54) 43 31 48 40 30 37 12

Consulting firms or private 

contractors (n=54) 30 35 63 47 30 37 15

Senior managers in other SEAs 

(n=54) 88 54 10 12 30 3 8
62

Exhibit 4.7. Usefulness of Comprehensive Center assistance compared with 

assistance from other sources 

Areas of state responsibility under NCLB

Percent of state managers (weighted) rating 

Center technical assistance

Much more or 

somewhat 

more useful

About the 

same

Much less or 

somewhat 

less useful

Not able to 

judge

Building or managing a statewide system of 

support for districts and schools identified for 

improvement under NCLB (n=54)

58 16 14 12

Identifying and/or developing programs or models 

that address district and/or school needs (n=48) 57 15 14 14

Training or managing the state-level staff or school 

support teams who provide support to districts and 

schools identified for improvement under NCLB 

(n=49)

55 17 11 17

Disseminating information on scientifically based 

research to districts and schools (n=51) 50 23 12 15

Providing training and other professional 

development to local educators in academic 

subjects (reading language arts, mathematics, 

science) (n=40)

45 19 16 20

Formulating or refining state policies to respond to 

NCLB requirements (n=49) 44 20 12 24

Supporting use of assessment data by schools 

and districts (n=46) 37 21 21 21

Monitoring compliance with NCLB requirements in 

districts and schools (n=39) 33 22 18 27

Designing or implementing state assessment and 

accountability systems (n=39) 29 22 22 27

Aligning state accountability systems with NCLB 

accountability systems (n=39) 27 27 17 29

Communicating with the public about NCLB 

requirements or report cards (n=32)

22 26 15 37

NOTE: States that chose the response, ―not applicable, state has not sought assistance for this purpose,‖ were 

excluded from the analysis. 

EXHIBIT READS: Fifty-eight percent of state managers (weighted) reported that Comprehensive Centers were much 

more or somewhat more useful than other sources of technical assistance for the state responsibility of building or 

managing a statewide system of support. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

■ For each area of state responsibility in NCLB implementation, there were fewer 

than one-fourth of state managers who reportedly considered the Centers’ work 
63

less useful than that of other providers. The areas of NCLB responsibility in 

which the largest percentage of state managers reported that the Centers were less 

useful than other sources were the following: (1) designing or implementing state 

accountability systems (22 percent of respondents); and (2) supporting use of 

assessment data by schools and districts (21 percent). 

Overall Rating of Center Assistance 

As senior officials in their agencies, the state managers were in a position to provide 

overall judgments of the Center technical assistance their agency had received. The evaluation 

team gathered data on the degree to which Center technical assistance was seen as relevant and 

useful with respect to each of 9 indicators of relevance and 10 indicators of usefulness 

(exhibit 4.8). Ratings were based on a 1 to 5 scale,45 and the scores for the individual items 

within relevance and within usefulness were averaged for each manager, and for multiple 

managers within a state, as described in chapter 2. 

Exhibit 4.8. Relevance and usefulness items from the state manager surveys 

Relevance Usefulness

a. Addressed a need or problem that the state 

faces

b. Addressed an important state priority

c. Addressed a challenge that the state faces 

related to the implementation of NCLB

d. Responded flexibly to our state’s changing 

needs 

e. Provided information, advice, and/or resources 

that could be applied to the state’s work

f. Addressed the particular context in which our 

state operates

g. Addressed the state’s specific challenges (e.g., 

policy environment, leadership capacity, 

budget pressures, local politics) 

h. Provided information, advice, and/or resources 

that could be used to guide decisions about 

policies, programs, and practices

i. Highlighted the implications of research 

findings (or information about best practice) for 

state-level clients’ policies, programs, or 

practices

a. Provided state staff with resources that were 

easy to understand and easy to use

b. Employed appropriate formats (e.g., work 

groups, conferences, individual consultation, 

written products)

c. Provided adequate opportunity to learn from 

colleagues in other states

d. Included adequate follow-up to support the use 

of new information and resources

e. Were timely

f. Helped the state to solve a problem

g. Helped the state to maintain or change a policy 

or practice

h. Helped the state take the next step in a longerterm improvement effort

i. Provided state staff with information or 

resources that they will use again

j. Helped state staff to develop skills that they will 

be able to exercise again

 

45 The response options were: 5 (to a very high degree), 4 (to a high degree), 3 (to a moderate degree), 2 (to a low 

degree), and 1 (to a very low degree). 
64

■ On a scale of 1 (very low degree) to 5 (very high degree), on average state 

managers gave the Centers’ technical assistance about a 4. The average 

relevance score was 3.94 and the average usefulness score was 3.86 (exhibit 4.9). 

On both measures, in other words, the mean response was near, but below, the 4.0 

threshold defined as ―to a high degree‖ on the 5-point scale. Fifty-seven percent 

of the states‘ ratings fell at or above 4.0 for relevance. Fifty percent fell at or 

above 4.0 for usefulness (exhibit 4.10). Twelve percent of state managers rated 

the relevance and usefulness of the Centers‘ work as below moderate.

Exhibit 4.9. Mean ratings of the relevance and usefulness of Center assistance, 

as judged by state managers 

Mean 

relevance 

rating

(n=54)

Mean 

usefulness 

rating

(n=53)

Comprehensive Center 

program

3.94 3.86

EXHIBIT READS: Fifty-four state managers (weighted) gave the Center program a 

mean relevance rating of 3.94. 

SOURCE: Survey of State Managers. Responses weighted so that each state was 

equally represented in instances where more than one manager from a state 

responded. 

Exhibit 4.10: Breakout of ratings of the relevance and usefulness of Center 

assistance, as judged by state managers 

Relevance Usefulness

Percent of state managers (weighted) 

giving overall ratings in the following 

ranges:

Percent of state managers (weighted) 

giving overall ratings in the following 

ranges:

Very 

low to 

low

(1.0-1.9)

Low to 

moderate

(2.0-2.9)

Moderate 

to high

(3.0-3.9)

High to 

very 

high

(4.0-5.0)

Very low 

to low

(1.0-1.9)

Low to 

moderate

(2.0-2.9)

Moderate 

to high

(3.0-3.9)

High to 

very 

high

(4.0-5.0)

Comprehensive 

Center program 3 9 31 57 4 8 38 50

EXHIBIT READS: Three percent of state managers (weighted) gave the Center program an overall relevance rating 

in the very low to low range (1.0-1.9). 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 
65

5. Variation in the Reported Quality, Relevance, and 

Usefulness of the Centers’ Technical Assistance 

While state managers‘ reports provide important input to an assessment of the overall 

work of the Comprehensive Centers program (chapter 4), additional perspectives are helpful in 

assessing the quality of the technical assistance projects and their relevance and usefulness to the 

offices and the teams of professionals directly served. A deeper look at variation across Centers 

and projects might also provide information for program improvement: additional support or 

oversight might be provided if quality, relevance, or usefulness appeared substantially weaker in 

particular groups of Centers or projects, or for particular types of participants. Thus, variation 

was investigated by features of project design (project scope and the type of activities offered)

and by participant experiences in the project as well as participant background. 

 This chapter analyzes the ratings of relevance, usefulness, and quality given to the 122 

Center projects sampled for the evaluation.46 Each project was evaluated for relevance and 

usefulness by a sample of participants—state staff, intermediate agency staff, local educators 

working on behalf of the state, and RCC staff—who were the intended beneficiaries of the 

project and had received at least some of the technical assistance it provided.47 Ratings of project 

quality were gathered from panels of experts with strong knowledge of the content or substantive 

focus of the specific projects they reviewed.48 Relevance was assessed with eight survey items 

and usefulness with 11 items; quality was judged on three items called dimensions (exhibit 5.1). 

Each overall measure (relevance, usefulness, or quality) was calculated as the mean of ratings 

assigned to each item. The item-level ratings themselves were based on 5-point rating scales 

(exhibit 5.2).49 The chapter concludes with a brief discussion of the variation in ratings by 

project characteristics and by the participant experiences in projects. 

 

48 See chapter 2 for more information about the data sources and procedures for gathering, coding, and analyzing the data 

reported in this chapter, including discussions of the methods used for expert ratings and participant surveys. 

49 Efforts were made to develop parallel wording and rubrics that would result in similar gradations between rating 

levels (e.g., very high vs. high vs. moderate) across the three measures. However, given the different content of each 

set of items within the three measures and the different contexts for the ratings (experts who underwent training for 

the rating process and reviewed identical packages of materials vs. survey respondents who typically participated in 

different subsets of project activities), the ratings across the three measures are not directly comparable. 
66

Exhibit 5.1. Quality, relevance, and usefulness items 

From expert panel scoring From project participant surveys

Technical quality Relevance Usefulness

Reviewers were directed to 

assign a score to each 

dimension and to include the 

basis for their ratings on the 

rating form, including the specific 

artifacts on which their score was 

based. The three dimensions 

are:

a. Demonstrated use of the 

appropriate documented 

knowledge base–to include 

an accurate portrayal of the 

current state of information 

with prominence to those with 

the most accurate/rigorous 

evidence 

b. Fidelity of application of the 

knowledge base to the 

products and services 

provided–materials are 

consistent with the 

best/accurate information 

available and the presentation 

adequately conveys the 

confidence of the information

c. Clear and effective delivery–

information is well organized 

and written and accessible to 

the intended audience for 

easy use 

Based on your experience, to 

what degree was this set of 

activities and resources relevant

to your work, in each of the 

following respects?

a. Addressed a need or problem 

that my organization faces

b. Addressed an important 

priority of my organization

c. Addressed a challenge that 

my organization faces related 

to the implementation of NCLB

d. Provided information, advice, 

and/or resources that could be 

directly applied to my 

organization’s work

e. Addressed our particular state 

context

f. Addressed my organization’s 

specific challenges (e.g., 

policy environment, leadership 

capacity, budget pressures, 

local politics) 

g. Provided information, advice, 

and/or resources that could be 

used to guide decisions about 

policies, programs, or 

practices

h. Highlighted the implications of 

research findings (or 

information about best 

practice) for policies, 

programs, or practices

Based on your experience, to 

what degree was this set of 

activities and resources useful to 

you, in each of the following 

respects?

a. Provided resources that were 

easy to understand and easy 

to use

b. Employed an appropriate 

format (e.g., a work group, a 

conference, individual 

consultation, written products)

c. Provided adequate opportunity 

to learn from colleagues in 

other states

d. Included adequate follow-up to 

support the use of new 

information and resources

e. Were timely

f. Helped my organization solve 

a problem

g. Helped my organization 

maintain or change a policy or 

practice

h. Helped my organization take 

the next step in a longer-term 

improvement effort

i. Provided my organization with 

information or resources that 

we will use again

j. Helped my organization 

develop a shared expertise or 

knowledge base

k. Helped individuals in my 

organization to develop skills 

that they will use again
67

Exhibit 5.2. Scales for rating of quality by expert panels and relevance 

and usefulness by participants 

Rating

Quality ratings by 

content area experts

Relevance or usefulness 

ratings by project 

participants

5

Very high—All or almost 

all of the project meets the 

given indicators for a 

dimension

To a very high degree

4

High—Most of the project 

meets the given indicators 

for a dimension

To a high degree

3

Moderate—Some of the 

project meets the given 

indicators for a dimension

To a moderate degree

2

Low—Limited parts of the 

project meet the given 

indicators for a dimension

To a low degree

1

Very low—None or 

almost none of the project 

meets the given indicators 

for a dimension

To a very low degree

 Based on these ratings, the key findings suggest: 

■ Center technical assistance was judged in the “moderate” to “high” range of 

quality, relevance, and usefulness. On a scale of 1 to 5 with a 3 representing 

―moderate‖ and a 4 representing ―high,‖ the programwide average ratings for the 

sampled projects were 3.34 for technical quality (scored by panels of content 

experts), and 3.94 for relevance and 3.70 for usefulness (scored by participants).50

 

■ The average quality rating was higher among CCs than RCCs; relevance and 

usefulness ratings were similar between the two Center types.

51 The Content 

Centers received Center-level mean scores for technical quality that averaged 

3.73, compared with 3.21 for the Regional Centers; the difference of 0.52 points 

exceeded one-half of one pooled standard deviation. The scores for relevance and 

 

50 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 

to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 

and each project sampled from a Center contributed equally to the Center mean. 

51 Using Cohen (1988) as a conceptual framework, we estimated Cohen's d (an estimate of the effect size defined as 

the differences in means divided by the pooled standard deviation) and adopted the logic of Cohen for what is a 

moderate difference. Specifically, we adopted a difference in the means of one-half of one standard deviation 

(analogous to an effect size of .5) as our minimum threshold for highlighting differences. The ―pooled standard 

deviation‖ for each computation varied with the unit of analysis. For analyses conducted at the Center level, the 

pooled standard deviation was computed as the standard deviation of the variable of interest (e.g., relevance) 

computed at the Center level. For analyses using the project as the unit of analysis, the pooled standard deviation 

was computed at the project level. 
68

usefulness were 3.99 and 3.71 for the Regional Centers and 3.78 and 3.65 for the 

Content Centers; on these measures the two types of Centers scored within onehalf of a standard deviation of each other. 

■ Mean ratings of quality, relevance, and usefulness varied across and within 

Centers. On each measure, at least 10 Centers had a mean rating that was at least 

one-half of a standard deviation above or below the overall mean for its type of 

Center (RCC or CC) for that measure (i.e., 11 of 21 Centers were this far above or 

below the mean for quality, 10 for relevance, and 14 for usefulness). However, 

few Centers showed consistently high or low ratings across measures: one RCC 

had ratings of quality, relevance, and usefulness for its sampled projects that 

exceeded the RCC means for each measure by at least one-half of a standard 

deviation; one RCC and one CC had ratings at least one-half of a standard 

deviation lower than those of all RCCs and CCs, respectively, on all three 

measures. Ratings varied across projects within Centers: at least one project was 

rated ―high‖ (4.0 or above) for quality in 11 of the 16 RCCs and 3 of the 5 CCs; 

the same was true for relevance in 15 of the 16 RCCs and all 5 CCs; and for 

usefulness in 11 of the 16 RCCs and all 5 CCs. 

■ Across subgroups of projects, average project ratings differed by project scope 

but generally did not differ by the presence of particular types of activities.

Projects that Centers had designated as large in scope (―major‖ projects), relative 

to the rest of that Center‘s own work, had relevance and usefulness ratings at least 

one-half of a standard deviation higher than those for the moderate projects 

sampled (exhibit 5.7). With respect to project activities, the differences in ratings 

for projects that either did or did not offer each of five types of Center activities or 

resources (research syntheses, training, task force participation, engagement of 

participants in planning, or development of a formal implementation plan) were 

less than one-half of a standard deviation (see exhibit 5.8). Those projects that 

offered ongoing consultation or support among their activities had lower quality 

ratings than did other projects, but higher relevance ratings; quality was rated 

higher for projects that included conferences among their activities, compared 

with projects that did not offer conferences. Each of the differences highlighted 

was at least one-half of a standard deviation in size. 

■ Ratings of quality were not statistically related to ratings of relevance and 

usefulness. The correlations between the scores for quality and relevance (-0.12) and 

between the scores for quality and usefulness (-0.04) are not statistically significant, 

meaning we cannot be sure that they are different from zero (no relationship). This 

indicates that the expert rating of technical quality was not related to the extent to 

which participants deemed the projects to be relevant or useful. 

■ Participants who reported more active and extensive experience with a project 

gave higher ratings for relevance and usefulness than did other participants. 

Because participant experiences could and did vary within projects as well as 

across projects, the descriptive comparisons here are at the individual participant 
69

level. Higher ratings were found among participants who were involved upfront in 

project design (compared with those who were not), who spent at least three days 

in any project activity (compared with those who participated for fewer days in 

that activity), and who spent at least three days in each of a higher number of 

project activities (compared with those who spent that much time in fewer 

activities). These differences were statistically significant at the .05 level. These 

descriptive analyses merely show differences among participants, and should not 

be interpreted as implying that active or extensive experience with a project led to 

higher ratings. 

Overall Ratings of Center Projects by Participants and Experts 

 The data gathered on individual projects were compiled to produce mean ratings of 

quality, relevance, and usefulness for the sampled projects across the program as a whole. In 

these averages, each Center contributed equally to the overall mean for the program. Given the 

different tasks that the RCCs and CCs were charged with, it is appropriate to examine whether 

there were differences in ratings between the two types of Centers.52 In addition, the overall 

means for the two Center types potentially mask variation in the quality, relevance, and 

usefulness across individual Centers that would be useful to understand. 

■ Averaged across the Centers, the quality, relevance, and usefulness of sampled 

projects were judged in the “moderate” to “high” range. Overall, the mean 

ratings were 3.34 for technical quality (scored by the panels of content experts), 

and 3.94 for relevance and 3.70 for usefulness (as scored by participants in the 

sampled projects) (exhibit 5.3). The rating scales established a 3 as indicating a 

moderate degree of quality, relevance, and usefulness, while a score of 4 

represented a high degree of relevance and usefulness (see exhibit 5.2). The 

average of the ratings for each measure fell between a score of 3 and 4. 

■ Compared with Content Centers, Regional Centers received a lower score from 

reviewers on technical quality, but a higher score from participants on 

relevance, while the usefulness score was similar between the two Center types. 

There was a gap of more than one-half of a standard deviation53 between the two 

types of Centers on technical quality of -0.52, with the CCs receiving an average 

score of 3.73 and the RCCs receiving an average score of 3.21. The difference in 

the relevance score (0.21) exceeded one-half standard deviation, while there was 

no substantive difference in usefulness ratings. 

 

52 ED structured the Comprehensive Centers program to consist of 5 Content Centers charged with specializing in 

activities related to specific content areas and 16 Regional Centers charged with specializing in interactions with the 

states in a region. 

53 For analyses conducted at the Center level, the pooled standard deviation was computed as the standard deviation 

of the variable of interest (e.g., relevance) computed at the Center level. 
70

Exhibit 5.3. Center Level Mean ratings of technical quality, relevance, and 

usefulness 

Quality dimensions:

Technical 

quality

Dimension 1

(use of 

knowledge 

base)

Dimension 2

(fidelity of 

application)

Dimension 3

(clear and 

effective 

delivery) Relevance Usefulness

All 

Comprehensive 

Centers (N=21)

3.34 3.22 3.20 3.60 3.94 3.70

All RCCs 

(N=16) 3.21 3.05 3.07 3.52 3.99 3.71

All CCs (N=5) 3.73 3.75 3.59 3.85 3.78 3.65

Difference of 

RCC and CC 

means 

-0.52†-0.70†-0.52†-0.33†0.21† 0.06

Pooled standard 

deviation(all 

Comprehensive 

Centers)

0.41 0.49 0.44 0.35 0.34 0.34

Ratio of difference 

in means to 

pooled standard

deviation

-1.28 -1.42 -1.18 -0.95 0.60 0.18

NOTE: All ratings were on a 5-point scale, with 5 as the high value. The ―technical quality‖ rating is the mean of the 

ratings for the three quality dimensions. A notation of † indicates that the difference in the mean ratings between the 

CCs and RCCs is at least one-half of one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.34. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for relevance 

and usefulness. Responses weighted so that each panelist or participant contributed equally to project ratings; each 

project contributed equally to Center ratings; and each Center contributed equally to cross-Center ratings. 

■ There was variation in the ratings across and within individual Centers. On 

each measure, at least 11 Centers had a mean rating that was at least one-half of a 

standard deviation above or below the overall mean for its type of Center54 (RCC 

or CC) for that measure (i.e., 11 of 21 Centers were this far above or below the 

mean for quality, 11 for relevance, and 14 for usefulness). One RCC was rated 

higher than others by at least one-half of a standard deviation on all three 

measures, and one CC and one RCC were rated lower than others on all three 

measures. Aside from these Centers, the other 18 Centers‘ ratings were not 

consistently higher or lower than the mean but varied across measures (exhibit 

5.4). 

 

54 For analyses using the project as the unit of analysis, the pooled standard deviation was computed at the project 

level. 
71

Exhibit 5.4. Mean ratings of technical quality, relevance, and usefulness, 

by Center 

Center type

Technical 

quality Relevance Usefulness

RCCs (N=16) 3.78 3.78 3.42

3.63 3.22 3.00

3.46 4.18 3.94

3.44 3.90 3.63

3.36 3.97 3.63

3.35 3.97 3.57

3.35 3.93 3.51

3.31 4.15 3.69

3.21 4.08 3.82

3.17 4.31 4.05

3.15 4.12 3.93

3.11 4.70 4.46

2.98 4.07 3.92

2.74 3.20 3.05

2.74 4.01 3.54

2.63 4.18 4.17

Average RCC rating 3.21 3.99 3.71

Pooled standard 

deviation (RCCs) 0.32 0.37 0.38

CCs (N=5) 4.24 3.76 3.54

3.94 3.90 3.86

3.88 3.99 3.84

3.44 3.58 3.44

3.14 3.68 3.56

Average CC rating 3.73 3.78 3.65

Pooled standard 

deviation (CCs) 0.43 0.16 0.19

NOTE: The arrow pointing upward indicates the accompanying value is at least one-half of one standard 

deviation above the group mean (e.g., 3.78 is at least one-half of one standard deviation above the mean for 

the RCCs). The arrow pointing downward indicates the accompanying value was at least one-half of one 

standard deviation below the group mean. 

EXHIBIT READS: One of the RCCs had a mean rating for technical quality of 3.78, a mean rating for 

relevance of 3.78, and a mean rating for usefulness of 3.42, across the projects sampled from that Center. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to 

project ratings, and each project contributed equally to Center ratings
72 4

29

50

17

4 4

52

41

0

20

40

60

80

100

Very low to low 

(1.0-1.9)

Low to moderate 

(2.0-2.9)

Moderate to high 

(3.0-3.9)

High to very high 

(4.0-5.0)

Technical quality

1

3

36

57

0

11

52

37

0

20

40

60

80

100

Very low to low 

(1.0-1.9)

Low to moderate 

(2.0-2.9)

Moderate to high 

(3.0-3.9)

High to very high 

(4.0-5.0)

Relevance

1

6

67

26

0

19

52

30

0

20

40

60

80

100

Very low to low 

(1.0-1.9)

Low to moderate 

(2.0-2.9)

Moderate to high 

(3.0-3.9)

High to very high 

(4.0-5.0)

Usefulness

Key: RCCs CCs

Percent of projects

Percent of projects Percent of projects

Variation by Type of Project 

 Perhaps the Centers were more successful, at least from the perspective of the reviewers, 

at some types of projects compared with others. Thus, the evaluation looked at different groups 

of projects, across Centers, that might be expected to show higher quality, relevance, or 

usefulness than other projects. For example, projects that were larger in scope than others in the 

Center might have garnered higher ratings from the experts or the participants. Projects that 

offered particular types of activities or resources might also stand out for higher ratings. 

 These descriptive subgroup analyses were possible because there was variation in ratings 

among the individual projects within the sample (exhibit 5.5). On technical quality, 4 percent of 

the RCC and CC projects were judged ―very low to low,‖ 29 percent and 4 percent of RCC and 

CC projects respectively were scored in the ―low to moderate‖ range, 50 and 52 percent scored 

in the ―moderate to high‖ range, and 17 percent and 41 percent scored in the ―high to very high‖

range. The ratings for relevance and usefulness also varied across projects. 

Exhibit 5.5. Distribution of technical quality, relevance, and usefulness 

ratings – percent of projects 

EXHIBIT READS: For technical quality, 4 percent of the projects sampled from RCCs had a mean quality rating of 

very low to low (between 1.0 and 1.9 on a 5-point scale). 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating. 
73

 These distributions of ratings across projects were wider than the distributions across 

Centers, which on all three measures were clustered in the ―moderate to high‖ range as described 

above. At least one project was rated ―high‖ (at least 4.0) for quality in 11 of the 16 RCCs and 3 

of the 5 CCs; the same was true for relevance in 15 of the 16 RCCs and all 5 CCs; and for 

usefulness in 11 of the 16 RCCs and all 5 CCs. Among RCCs, four or fewer had at least one 

project rated ―low‖ (2.0 or lower) on quality, relevance or usefulness. One CC had at least one 

project rated ―low‖ for quality, but none of the CC projects received a ―low‖ rating for relevance 

or usefulness (exhibit 5.6). 

Exhibit 5.6. Technical quality, relevance, and usefulness ratings, by projects 

rated ―high‖ and projects rated ―low‖ and Center type 

Technical 

quality Relevance Usefulness

Regional Comprehensive Centers (RCCs) (N=16)

RCCs with at least one project

rated ―high‖ (at least 4.0) 11 15 11

RCCs with at least one project 

rated ―low‖ (2.0 or less) 4 3 3

Content Centers (CCs) (N=5)

CCs with at least one projected 

rated ―high‖ (at least 4.0) 3 5 5

CCs with at least one project 

rated ―low‖ (2.0 or less) 1 0 0

EXHIBIT READS: For technical quality, 11 of the 16 RCCs had at least one project that received a ―high‖

rating of at least 4.0. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants 

for relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to 

the project rating. 

Ratings by Project Size or Level of Effort 

The quality or utility of projects might be related to the scale of the undertaking. For 

example, more ambitious projects that extended over a longer period of time or included a larger 

number of participants might have been more challenging to carry out but might have been 

conceived to address a great need. Larger projects might allow greater focus on the effort and 

enjoy relatively higher levels of Center resources. Alternatively, the tighter focus possible in a 

smaller-scale project might have been conducive to higher quality, relevance, or usefulness. 
74

The Centers were asked to consider the scope of each of the 122 sampled projects (i.e., its 

cost or level of effort) in relation to other projects of that Center and to designate each project as 

either ―major‖ or ―moderate.‖ 

■ Projects identified by the Centers as “major” efforts were rated higher on the 

measures of relevance and usefulness. Participants gave the major projects a 

mean rating of 4.02 for relevance, which was one-half of a standard deviation 

higher than the mean rating of 3.76 that they gave the moderate projects 

(exhibit 5.7). The mean usefulness rating for the major projects was 3.77, 

exceeding the corresponding rating for the moderate projects (3.47) by more than 

one-half of a standard deviation. 

■ The quality ratings for major and moderate projects were within one-half of a 

standard deviation of each other, however. The means were 3.37 and 3.25 

respectively, both close to the ―moderate quality‖ rating of 3. 

Exhibit 5.7. Technical quality, relevance, and usefulness ratings, by project size 

Project size

Technical 

quality Relevance Usefulness

Major projects 3.37 4.02 3.77

Moderate projects 3.25 3.76 3.47

Difference in means 0.12 0.26†

0.30†

Pooled standard deviation 0.69 0.52 0.54

Ratio of difference in means 

to pooled standard deviation 0.18 0.50 0.55

NOTE: For technical quality the overall number of projects was 122 with 93 major projects and 29 moderate projects. 

For relevance and usefulness the overall number of projects was 120, with 91 major projects and 29 moderate 

projects. 

EXHIBIT READS: The mean technical quality rating for major projects in the sample was 3.37 while the mean 

technical quality rating for moderate projects was 3.25. The † indicates that the difference in the mean ratings 

between the major and moderate projects exceeds one-half of the pooled standard deviation in the rating. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 

Ratings by Project Activities 

The technical assistance provided by the Centers varied in delivery methods, including the 

types of activities offered. We examined whether the inclusion of certain types of technical 

assistance activities resulted in more favorable project ratings either by experts knowledgeable 

about the topic or by participants in the project (exhibit 5.8). The types of activities implemented by 

the Centers and coded by the evaluation team (first introduced in chapter 3) overlapped, in that 
75

several could be included in a single project.55 By comparing the ratings of projects that offered 

each activity with ratings of projects that did not, it may be possible to identify hypotheses for 

further exploration about strengths and weaknesses of particular modes of Center work. 

■ Across five of the seven project activities listed here, comparisons of projects 

with and without the activities showed no differences in quality, relevance, or 

usefulness ratings greater than one-half of a standard deviation (exhibit 5.8). 

Those projects that offered ongoing consultation among their activities had lower 

quality ratings than other projects, but higher relevance ratings; quality was rated 

higher for projects that included conferences among their activities, compared 

with projects that did not offer conferences. Each of the differences highlighted 

was at least one-half of a standard deviation in size. All other differences in 

exhibit 5.8 were within one-half of a standard deviation.

 

55 For each sampled project, the Center furnished the team with a cover sheet intended to inform the expert 

reviewers about the project context, purpose, activities, and products. As described in chapter 2 and appendix C, the 

team coded the content of each project cover sheet to describe the activities and resources offered as part of the 

project. When a cover sheet indicated that the project had multiple components (which was the case for 84 percent 

of the projects), multiple codes were assigned. 
76

Exhibit 5.8. Technical quality, relevance, and usefulness ratings of projects, by 

type of activity offered in the projects 

Type of assistance activity offered Technical quality Relevance Usefulness

Ongoing 

consultation or 

support

Yes 3.25

(n=84)

4.03

(n=82)

3.74

(n=82)

No 3.59

(n=38)

3.79

(n=38)

3.60

(n=38)

Diff SD Ratio -0.34†

0.50 -0.68 0.24†0.46 -0.52 0.14 0.54 0.26

Research 

collections and 

syntheses

Yes 3.50

(n=71)

3.94

(n=70)

3.67

(n=70)

No 3.17

(n=51)

3.98

(n=50)

3.73

(n=50)

Diff SD Ratio 0.33 0.67 0.49 -0.04 0.52 -0.08 -0.06 0.54 -0.11

Engagement of 

participants in 

project planning

Yes 3.24

(n=50)

4.00

(n=48)

3.73

(n=48)

No 3.44

(n=72)

3.93

(n=72)

3.67

(n=72)

Diff SD Ratio -0.20 0.67 -0.30 0.07 0.52 0.14 0.06 0.54 0.11

Training events Yes 3.34

(n=50)

4.09

(n=49)

3.83

(n=49)

No 3.37

(n=72)

3.86

(n=71)

3.60

(n=71)

Diff SD Ratio -0.03 0.67 -0.05 0.23 0.52 0.44 0.23 0.54 0.43

Task force 

meetings and work

Yes 3.23

(n=50)

4.05

(n=48)

3.71

(n=48)

No 3.45

(n=72)

3.89

(n=72)

3.68

(n=72)

Diff SD Ratio -0.22 0.52 -0.42 0.16 0.54 0.30 0.03 0.67 0.04

Conference Yes 3.60

(n=43)

3.83

(n=43)

3.61

(n=43)

No 3.21

(n=79)

4.03

(n=77)

3.74

(n=77)

Diff SD Ratio 0.39†

0.67 0.58 -0.20 0.52 -0.40 -0.13 0.54 -0.24

Support 

development of a 

formal plan to 

implement a 

program or policy

Yes 3.38

(n=20)

4.02

(n=20)

3.63

(n=20)

No 3.35

(n=102)

3.94

(n=100)

3.71

(n=100)

Diff SD Ratio 0.03 0.67 0.04 0.08 0.54 0.15 -0.08 0.54 -0.15

NOTE: For each type of assistance, the first two rows show the mean ratings and the number of projects in the 

sample for Yes (the activity or resource was offered) and No (not offered). The third row shows the difference in the 

means, the pooled standard deviation, and the calculated ratio of the difference in means to the pooled standard 

deviation. A notation of † indicates that the difference in the mean ratings is at least one-half of one pooled standard 

deviation. 

EXHIBIT READS: The mean quality rating of projects that offered ongoing consultation or support was 3.25, while the 

mean quality rating of projects that did not offer this was 3.59. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 
77

Ratings for RCC Projects by Whether They Drew on CC Contributions 

Cooperation between CCs and RCCs was one element of the design of the 

Comprehensive Centers system, as discussed throughout this report. Thus, among the types of 

resources that RCC projects might offer to their participants were materials developed by CCs, 

activities developed with advice from CCs, or technical assistance from CC staff members. As 

indicated in chapter 3, 46 of the 96 sampled RCC projects had one or more of these types of CC 

contributions; the other 50 did not. One hypothesis for the evaluation was that the RCC projects 

incorporating CC contributions would be rated higher for technical quality than RCC projects 

that were developed without CC input, since the CCs were specifically charged with synthesizing 

the knowledge base in their areas of focus, and since the projects sampled from CCs had 

received a higher mean rating of technical quality than those sampled from RCCs. 

■ Among the RCC projects sampled, the ratings of those with CC contributions 

did not differ by more than one-half of a standard deviation on any measure, 

compared with those without CC contributions (exhibit 5.9). For example, the 

RCC projects with CC contributions had a mean quality rating of 3.40, which was 

just under one-half of a standard deviation (0.47) of the rating of those that did not 

include CC contributions (3.09). 

Exhibit 5.9. Technical quality, relevance, and usefulness ratings by Content 

Center contribution to RCC projects 

RCC projects with...

Technical 

quality Relevance Usefulness

CC contribution (n=46) 3.40 3.92 3.63

No CC contribution (n=50) 3.09 4.09 3.78

Difference in means (with CC 

contribution vs. without) 0.31 -0.17 -0.15

Pooled standard deviation 0.67 0.53 0.56

Ratio of difference in means to pooled 

standard deviation 0.47 -0.32 0.27

NOTE: For the technical quality ratings, there were 46 RCC projects reporting a contribution from one or more CC 

while 50 RCC projects reported no CC contribution. For the relevance and usefulness ratings there were 45 RCC 

projects with a CC contribution and 49 projects with no CC contribution. 

EXHIBIT READS: The mean technical quality rating for RCC projects with a CC contribution was 3.40 while the mean 

technical quality rating for RCC projects with no CC contribution was 3.09. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 
78

Relationships between the Relevance, Usefulness, and 

Quality of Center Projects 

In our analysis of the relevance, usefulness, and quality of Center technical assistance, we 

observed that some groups of projects that received relatively high ratings for quality received 

relatively low ratings for relevance and usefulness, and vice versa. We hypothesized that 

differences in the rating and rankings (the rating relative to other groups of projects) could be 

due to the differences in the types of individuals who reviewed and scored projects for quality—

content experts—and those who scored the relevance and usefulness—project participants. It was 

expected that these two groups might value and be better able to judge different qualities in a 

Center project, which is why we did not have content experts evaluate the projects for their 

utility or the participants assess the technical quality. 

We examined the associations among the three dimensions more systematically by 

calculating correlation coefficients.56 These statistics indicate the strength and direction of a linear 

relationship between two factors. A correlation coefficient can vary from positive one (indicating a 

perfect positive relationship), through zero (indicating the absence of a relationship), to negative one 

(indicating a perfect negative relationship). If the correlation is statistically significant (p <.05), we 

can have strong (95 percent) confidence that what we calculated is not due to chance. 

 

■ Participants’ ratings of relevance and usefulness were closely related 

(exhibit 5.10). The correlation coefficient is +.84. This indicates that the extent to 

which participants rated the projects as relevant was associated with how they 

deemed the project to be useful to their agency. 

■ There was no statistically significant relationship between the ratings for the 

quality of Center projects and for their relevance or usefulness (exhibit 5.10). 

The results indicate correlations of -0.12 between quality and relevance, and -0.04 

between quality and usefulness. Although the coefficients are negative, because 

they are not statistically significant we cannot be sure that they are different from 

zero (no relationship). Still, the correlations suggest that the expert rating of 

technical quality was not related to the extent to which participants deemed the 

projects to be relevant or useful. This was also the case for the dimension-level 

ratings within the overall quality rating: none was related to the rating of relevance 

or usefulness. This latter finding will be explored further in the final evaluation 

report. 

 

56 For this analysis, the evaluation team used Spearman‘s rank order correlation because this non-parametric rating is 

the appropriate statistical function to describe correlations between two variables where the values of the variables 

are not normally distributed and are on a scale (such as ratings). 
79

Exhibit 5.10. Correlations between project-level technical quality, relevance, 

and usefulness ratings 

Combination Spearman’s rho

Quality and relevance -0.12

Quality and usefulness -0.04

Relevance and usefulness +0.84

EXHIBIT READS: The correlation between project-level ratings of technical 

quality and relevance ratings was a negative 0.12. 

SOURCE: Expert panel ratings of sampled projects for technical quality and 

surveys of project participants for relevance and usefulness. Responses 

weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 

Variation by Participant Experiences and Roles 

It is possible that Center technical assistance is viewed as more beneficial by some types 

of participants or that the extent to which participants are engaged in the projects affects how 

they regard it. These questions can be analyzed by examining whether the rating of relevance or 

usefulness given by participants was related to a participant‘s experience in the project or to his 

or her job responsibilities.57 These analyses provided an opportunity to follow up on findings 

from the prior evaluation conducted on the Comprehensive Regional Assistance Centers 

established under the 1994 ESEA reauthorization (described in chapter 1 of this report).58 The 

specific participant characteristics examined included: 

■ Whether the participant contributed to the design of the project 

■ How much time the participant spent with project activities or resources 

■ The participant‘s role in his or her job, and the extent to which the job focused on 

NCLB 

In the following discussion, the unit of analysis is the participant, not the project. The 

mean ratings are those provided by a group of participants who had a particular type of 

experience or background; within the same project there could be and were participants with 

different experiences and backgrounds. In addition, participants could and did participate in quite 

different subsets of all activities offered in a project. A typical project comprised multiple types 

 

57 Within this section, the counts in the tables represent the weighted number of project participants who responded 

to that item. Because respondents were sampled at random from complete lists of participants in each sampled 

project, it was possible to draw inferences from their responses to the universe of participants in the 120 projects 

rated for relevance and usefulness. Accordingly, this section reports the results of tests of statistical significance. 

58 U.S. Department of Education, Office of the Under Secretary, Planning and Evaluation Service, Elementary and 

Secondary Education Division. (2000). Comprehensive Regional Assistance Centers Program: Final Report on the 

Evaluation (Volume I). Washington, DC: Author. 
80

of activities. The median number of activities was 4.0, and ranged from 1 to 7. Thirty-two 

percent of participants took part in three or more activities, suggesting that it was unusual for the 

same individuals to participate in all of a project‘s activities. Just over half of project participants 

(52 percent) reported contributing to the design of the project. In the following analysis, we 

examine whether different patterns of participation in a project were associated with different 

ratings of the project‘s relevance and usefulness. 

Ratings by Participants’ Involvement in Project Design 

An aspect of participation that was of particular interest, given the Centers‘ charge to work 

closely with their clients, was the extent to which participants were involved in the projects‘ design 

stages and, then, whether perceptions of relevance and usefulness might vary with a participant‘s 

involvement. The evaluation of the earlier program of Comprehensive Regional Assistance 

Centers, based on survey results from a sample of participants, concluded: ―Comprehensive Center 

technical assistance is more useful to customers if it is tailored to address their needs and 

interests.‖

59 For the current evaluation, the team analyzed participant responses to a question 

asking respondents whether they were personally involved in determining the goals or designing 

the content or format of the project. 

■ A higher proportion of participants in the sampled RCC projects reported being 

involved in some aspect of determining project goals or designing the project 

than did participants in the CC projects. The figures were 57 percent for 

participants in RCC projects and 42 percent for those in CC projects 

(exhibit 5.11). A chi square test found this difference to be statistically significant 

(p<.01).60

■ Among participants in the sampled Content Center projects, RCC staff were 

more likely to be involved in some aspect of determining project goals or 

designing the project than were state-level staff (exhibit 5.12). Forty-five percent 

of RCC staff who participated in technical assistance provided by the CCs were 

involved in the project‘s design. The corresponding figure was 36 percent among 

―state-level‖ staff (a category that included not only SEA employees but also local 

educators and others working with a state team or task force (as described in 

chapter 2). A chi square test found this difference in proportions to be statistically 

significant (p<.01).61

 

59 U.S. Department of Education, 2000, op cit. 

60 Unlike the analyses at the project level, which are based on a purposive sample of projects and thus could not use 

the tools of inferential statistics, these analyses at the participant level are based on random sampling of participants; 

thus they can be generalized to all participants in the sampled projects using conventional statistical tests. 

61 No parallel analysis could be conducted for CC participants in RCC projects because there were none; the RCCs 

did not provide technical assistance to CC staff. 
81

Exhibit 5.11. Participants’ involvement in determining project goals or design, 

by type of Center 

Involved in determining 

the goals or design of

the project

Percent of participants

RCC projects

(n=2,355)

CC projects

(n=1,116)

Total 100 100

Yes 57 42

No 43 58

NOTE: Difference in the proportion of participants involved in design by type of Center is 

statistically significant (p<.01, Chi Square). 

EXHIBIT READS: Fifty-seven percent of participants in projects conducted by RCCs reported 

they were involved in determining the goals or design of the project. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented 

all sample-eligible participants for the project. 

Exhibit 5.12. Participants’ involvement in determining project goals or design, 

by type of participant organization 

Participant involvement

Percent of 

state-level 

staff

(n=455)

Percent of 

RCC staff

(n=661)

Total 100 100

Involved in determining the goals or 

designing the content of the CC project 36 45

Not involved in determining the goals or 

designing the content of the CC project 64 55

NOTE: Difference in the proportion of participants involved in design by type of participant 

organization is statistically significant (p=.01, Chi Square). 

EXHIBIT READS: Thirty-six percent of the state-level staff who participated in Content Center 

projects reported involvement in determining the project’s goals or design. 

Source: Surveys of project participants. Responses weighted so that respondents represented all 

sample-eligible participants for the project. 

■ Participants who reported being involved in determining the goals or designing 

the content of the project provided higher relevance and usefulness ratings than 

did other participants. The average ratings in projects sampled for the current 

study corroborate the survey findings from the prior evaluation of the Centers; 

participants who reported involvement in project design gave the projects a 4.06 

average score for relevance and a 3.83 score for usefulness (exhibit 5.13). In 

contrast, average scores for participants not involved in project design were 3.81 

and 3.56, respectively. These differences were statistically significant (with p<.01 

for differences in both relevance and usefulness). This finding stands in contrast 
82

to the one displayed in exhibit 5.8, which showed no difference in ratings among 

projects that offered some opportunity for involvement compared with projects 

that did not do so. This suggests that involving some participants in design work 

may make a difference in the ratings provided by those particular participants, 

although not in the ratings provided by all participants in the project. 

Exhibit 5.13. Mean relevance and usefulness ratings, by respondent involvement 

in determining project goals or design 

Respondent 

involvement Relevance Usefulness

Involved in project 

design (n=1,802) 4.06 3.84

Not involved in project 

design (n=1,638) 3.81 3.56

Difference

Significance

0.25*

p<.01

0.28*

p<.01

* Difference statistically significant at p<.05 using a one-way ANOVA. 

EXHIBIT READS: Participants who indicated that they were involved in determining the 

goals or design of the project provided a relevance rating of 4.06, which was statistically 

significantly higher than the relevance rating of 3.81 provided by participants who indicated 

that they were not involved in determining the goals or design of the project. 

SOURCE: Surveys of project participants. Responses weighted so that respondents 

represented all sample-eligible participants for the project. 

Ratings by Time Spent in Project Activities 

The literature on professional development suggests that the duration of training is 

associated with better outcomes.62 Similarly, the previous evaluation of the Comprehensive 

Centers program, based on survey data from participants, concluded: ―Comprehensive Center 

technical assistance is more useful to customers if it is intensive and if it extends over time.‖

63

Therefore, this study explored the possibility of a relationship between the time an individual 

spent in one or more project activities and the ratings he or she provided, assessing whether the 

same relationship persisted. 64

 

62 Desimone, L., Porter, A.C., Garet, M., Yoon, K.S., and Birman, B. (2002). ―Does Professional Development 

Change Teachers‘ Instruction? Results from a Three-Year Study.‖ Educational Evaluation and Policy Analysis,

24(2), 81-112; Garet, M., Porter, A., Desimone, L., Birman, B., and Yoon, K.S. (2001). ―What Makes Professional 

Development Effective? Results from a National Sample of Teachers.‖ American Educational Research Journal,

38(4), 915-945. 

63U.S. Department of Education, 2000, op cit., p. 9. 

64 At the participant level, the variable measured in the survey was the amount of time an individual respondent 

reported devoting to each of eight activities in which a participant might engage. For each activity, respondents 

indicated the amount of time spent by selecting from the following five options: ―More than 5 days,‖ ―3-5 days,‖ ―1-

2 days,‖ ―Less than 1 day,‖ and ―Not applicable.‖ The participant activities included counterparts of the seven 

activities and resources that were used to classify what the project offered (see, for example, Exhibit 5.13). In 
83

■ For each of eight project-related activities addressed in the survey, participants 

who reported spending at least three days in the activity rated the project higher 

on both relevance and usefulness than did participants who spent less time in that 

activity (exhibit 5.14).65 For example, those who participated for at least three 

days in conferences associated with a project gave an average rating of 4.15 for 

relevance and 3.87 for usefulness. In contrast, participants who spent less than 

three days in conferences gave average ratings of 3.82 and 3.63 for relevance and 

usefulness, respectively. All of the differences were statistically significant at the 

p<.05 level using a one-way ANOVA. 

■ Participating for at least three days in a larger number of activities was 

associated with higher relevance and usefulness ratings. There was a 

statistically significant relationship between the number of activities in which the 

participant spent three or more days and the rating of project relevance. The 

correlation between the number of different project activities in which the 

respondent participated for three or more days and the rating of relevance was 

+0.41, while the correlation with the rating of usefulness was +0.34. These 

correlation coefficients were statistically significant at p<.05. 

 

addition, participants were asked about time spent using ―tools and other resources‖ provided by the project. This 

was a suitable item for differentiating among participants, since there was variation in the amount of time individual 

participants spent using tangible items that the project had provided. However, a corresponding item could not be 

used to differentiate among projects because the coding process revealed that all projects offered one or more 

tangible tools or resources to participants. 

65 Participant activities referred to in the text and presented in exhibit 5.14 are based on responses to the survey 

question on how much time individual participants had spent engaged in their own activities related to the project. 

These activity categories are not precisely aligned with the different set of codes used to categorize activities and 

resources that the Comprehensive Center offered, as reported on the project cover sheets. 
84

Exhibit 5.14. Mean relevance and usefulness ratings, by time respondents spent 

in each activity 

Type of activity or resource Time spent Relevance Usefulness

Ongoing consultation on

this topic

3+ days (n=1,156) 4.35 4.06

<3 days (n=1,115) 3.81 3.59

Difference 

Significance

0.54* 

p<.01 

0.47* 

p<.01

Reviewing general or 

background materials 

provided by the 

Comprehensive Center

3+ days (n=1,105) 4.32 4.07

<3 days (n=1,534) 3.78 3.53

Difference

Significance

0.54* 

p<.01 

0.54* 

p<.01

Advance planning 3+ days (n=731) 4.37 4.08

<3 days (n=1,311) 3.91 3.65

Difference

Significance

0.46* 

p<.01 

0.40* 

p<.01

Training 3+ days (n=1,136) 4.13 3.93

<3 days (n=1,219) 3.94 3.72

Difference

Significance

0.19*

p<.01 

0.21*

p<.01

Task force meetings 3+ days (n=581) 4.29 3.94

<3 days (n=1,061) 3.86 3.61

Difference

Significance

0.43* 

p<.01

0.33* 

p<.01

Conferences 3+ days (n=1,219) 4.15 3.87

<3 days (n=1,556) 3.82 3.63

Difference

Significance

0.33*

p<.01 

0.24*

p<.01

Follow-up and action plans 3+ days (n=1,138) 4.38 4.04

<3 days (n=1,307) 3.81 3.57

Difference

Significance

0.57* 

p<.01 

0.47* 

p<.01

Using tools and other 

resources provided by the 

Center

3+ days (n=1,450) 4.34 4.10

<3 days (n=1,305) 3.68 3.42

Difference

Significance

0.66* 

p<.01 

0.68* 

p<.01

NOTE: The types of activities and resources shown in this table are based on the types of activities presented 

in the participant survey and do not match categories presented in chapter 3 exhibits and exhibit 5.8, which 

were coded categories from the project cover sheets. 

* Difference statistically significant at p<.05 using a one-way ANOVA. 

EXHIBIT READS: The relevance ratings among project participants who indicated that they spent 3 or more 

days receiving ongoing consultation or support was 4.35 and statistically significantly higher than the ratings 

from project participants who spent less time receiving ongoing consultation or support (3.81). 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all sampleeligible participants for the project. 
85

Ratings by Participant Roles 

Were the sampled projects rated differently by participants who occupied different roles 

in their day-to-day jobs? The team investigated the possibility of a relationship between the 

participant‘s place of employment or job responsibilities and the ratings he or she provided for a 

project‘s relevance or usefulness. 

Given the intended emphasis of the Center technical assistance on building capacity at 

the state level, it is important to understand the extent to which the views of participants from 

state agencies might have differed from those of other participants. Of the sampled participants 

in RCC projects, 41 percent were employed by SEAs, 20 percent by schools, 15 percent by local 

education agencies (LEAs), 14 percent by intermediate education agencies, and 10 percent by 

other agencies or institutions.66 Among the participants in the entire sample of CC projects, 59 

percent were employed by RCCs, 38 percent by SEAs, and 3 percent by schools or LEAs 

(exhibit 5.15). 

Exhibit 5.15. Where participants in RCC and CC projects were employed 

in 2006-07

Employed by:

Percent of participants

RCC projects CC projects

Total 100 100

RCCs N/A 59

State education agency (SEA) 41 38

Intermediate education agency 14 *

Local education agency (LEA) 15 1

School 20 2

Other 10 *

* Less than 0.5 percent. 

EXHIBIT READS: Among the surveys completed by participants in RCC projects, 41 percent 

were completed by respondents who were employed in a state education agency. 

SOURCE: Surveys of project participants. Responses weighted so that respondents 

represented all sample-eligible participants for the project. 

■ Participants employed in SEAs did not give significantly different ratings for 

relevance or usefulness of either RCC or CC projects, compared with other 

participants (exhibits 5.16 and 5.17). The RCCs and CCs targeted assistance to 

 

66 Participants who were employed by agencies other than the SEA were considered ―state-level‖ participants 

because they carried out state-level responsibilities such as membership in a school support team or state-level task 

force (see chapter 2 for more discussion of the definition of ―state-level‖ participants). For this analysis, however, 

we differentiate among participants by their place of employment. 
86

different clients corresponding with their respective roles: the RCCs provided 

technical assistance to the states and territories in their regions, whereas the CCs 

targeted RCCs for capacity building as well as working with RCCs in providing 

technical assistance to states. Ratings of relevance and usefulness did not differ 

between state-level staff and other client groups. For example, state-level and 

RCC staff did not rate the relevance and usefulness of CC projects differently. 

Exhibit 5.16. Mean relevance and usefulness ratings for RCC projects, by agency 

where respondent worked 

Agency Relevance Usefulness

State education agency

(SEA)

Yes (n=977) 3.99 3.68

No (n=1,409) 3.94 3.68

Difference

Significance

0.05

p=.13

0.00

p=.93

Intermediate education 

agency

Yes (n=345) 4.03 3.67

No (n=2,045) 3.95 3.68

Difference

Significance

0.08

p=.07

-.01

p=.90

Local education 

agency (LEA)

Yes (n=365) 3.87 3.72

No (n=2,022) 3.97 3.67

Difference

Significance

-0.10*

p=.03

0.05

p=.30

School Yes (n=469) 3.88 3.65

No (n=1,918) 3.98 3.69

Difference

Significance

-0.10*

p=.02

-0.04

p=.44

Other Yes (n=188) 4.01 3.67

No (n=2,201) 3.95 3.68

Difference

Significance

0.06

p=.25

-0.01

p=.90

* Difference statistically significant at p<.05 using a one-way ANOVA. 

NOTE: The category other included employees of institutions of higher education, nonprofits working with the 

state, and other RCCs. No CC staff members participated in RCC projects as recipients of technical assistance. 

EXHIBIT READS: Across RCC project participants, the mean relevance ratings provided by respondents who 

worked at state education agencies was 3.99. This was not statistically significantly different from the mean 

relevance rating of 3.94 provided by respondents who worked in other types of agencies. 

SOURCE: Surveys of project participants. Numbers reflect surveys returned, not individuals. Responses 

weighted so that respondents represented all sample-eligible participants for the project. 
87

Exhibit 5.17. Relevance and usefulness ratings for CC projects, 

RCC staff vs. state-level staff 

Participant role Relevance Usefulness

RCC staff (n=477) 3.91 3.73

State-level staff 

(n=669) 3.92 3.81

Difference

Significance

-0.02

p=.74

-0.08

p=.14

EXHIBIT READS: Among CC project participants, the mean relevance ratings provided by 

respondents who worked in RCCs was 3.91. This was not statistically significantly 

different from the mean relevance rating of 3.92 provided by state-level respondents. 

SOURCE: Surveys of project participants. Numbers reflect surveys returned, not 

individuals. Responses weighted so that respondents represented all sample-eligible 

participants for the project. 

■ Participants who were employed in LEAs or schools (although they participated in 

Center assistance because they were working with the state) gave lower ratings for 

relevance to RCC projects, compared with other participants. In each case, the 

rating given was 0.10 points lower for relevance (exhibit 5.16) than was the rating 

given by other participants, and the difference was statistically significant at the 

p<.05 level using a one-way ANOVA. There were no other differences in ratings 

given to RCC projects by participants from different agencies. 

■ Comparing participants in CC projects who held different roles within RCCs, 

content specialists gave higher ratings for relevance and usefulness than did 

other RCC staff, while directors and assistant directors gave lower ratings 

(exhibit 5.18). The staff members who specialized in a content or topic area rated 

the CC projects 4.12 for relevance and 3.97 for usefulness, on average, compared 

with average ratings of 3.80 and 3.72 from other RCC staff. One possible 

explanation for this difference is that the CC emphasis on content areas was a 

good fit for the content-focused responsibilities of such staff members. The 

directors and assistant directors gave ratings of 3.54 for relevance and 3.52 for 

usefulness, compared with 4.04 and 3.90 ratings from other RCC staff. All of the 

differences were statistically significant at the p<.05 level using one-way 

ANOVA. 
88

Exhibit 5.18. Mean relevance and usefulness ratings for CC projects, 

by RCC staff role of respondent 

Participant role Relevance Usefulness

State liaison Yes (n=315) 3.91 3.78

No (n=346) 3.93 3.84

Difference

Significance

-0.02

p=.80

-0.06 

p=.32

Content or topic area 

specialist

Yes (n=254) 4.12 3.97

No (n=415) 3.80 3.72

Difference

Significance

0.32*

p<.01

0.26*

p<.01

Center director or 

assistant director

Yes (n=152) 3.54 3.52

No (n=517) 4.04 3.90

Difference

Significance

-0.50*

p<.01

-0.38*

p<.01

Other RCC staff Yes (n=187) 3.75 3.58

No (n=482) 3.99 3.90

Difference

Significance

-0.24

p=.07

-0.32*

p<.01 

* Difference statistically significant at p<.05 using a one-way ANOVA. 

EXHIBIT READS: Among participants in CC projects who were employed by RCCs, the mean relevance 

rating provided by respondents who identified themselves as state liaisons was 3.91. 

SOURCE: Surveys of project participants. Numbers reflect surveys returned, not individuals. Responses 

weighted so that respondents represented all sample-eligible participants for the project. 

■ Participants who, in their regular jobs, spent more than 25 percent of their 

hours on NCLB-related state responsibilities gave higher ratings to Center 

projects than did those whose jobs focused less on NCLB (exhibit 5.19). Because 

the Centers‘ charge focused on assistance related to NCLB, the team hypothesized 

that participants whose regular job responsibilities focused most heavily on that 

law might give relatively higher ratings for relevance and usefulness of the 

sampled projects.67 We found that those who spent at least one quarter of their 

work time on NCLB gave significantly higher relevance and usefulness ratings to 

the projects, compared with those who spent 25 percent of their time or less 

(p<.01 for differences in both relevance and usefulness using ANOVA). 

 

67 Participants were asked in the survey to indicate what percentage of their hours on the job they spent working on 

any of the specific state-level responsibilities related to NCLB. They were given a choice of four percentage ranges: 

0-25 percent, 26-50 percent, 51-75 percent, and 76-100 percent. 
89

Exhibit 5.19. Mean relevance and usefulness ratings, by percent of time 

respondent spent on NCLB-related tasks in his/her job 

Percent of time spent on 

NCLB-related tasks Relevance Usefulness

More than 75 percent (n=908) 4.05 3.87

51 to 75 percent (n=541) 3.98 3.78

26 to 50 percent (n=493) 3.94 3.72

25 percent or less (n=476) 3.72 3.47

Range

Significance

0.33*

p<.01 

0.40*

p<.01

* Difference statistically significant at p<.05 using a one-way ANOVA. 

EXHIBIT READS: The mean relevance rating of 4.05 provided by project participants whose job 

responsibilities included NCLB-related tasks more than 75 percent of the time was statistically 

significantly than the mean relevance ratings provided by project participants whose NCLB-related 

job responsibilities took up less of their time. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all 

sample-eligible participants for the project. 
Appendix A 

Comprehensive Center Lead Grantee 

and Subgrantee Organizations 
A-1

Comprehensive Center Lead Grantee 

and Subgrantee Organizations 

Center name Lead grantee and subgrantees

Alaska Comprehensive 

Center

■ Southeast Regional Resource Center (SERRC of Alaska)

o Northwest Regional Educational Laboratory (NWREL)

Appalachia Regional 

Comprehensive Center

■ Edvantia

o George Washington University - Center for Equity and Excellence 

in Education (CEEE)

o Eastern Stream Center on Resources and Training (ESCORT)

o National Center for Family Literacy (NCFL)

o University of North Carolina at Greensboro - SERVE Center

o Southern Regional Education Board (SREB)

California 

Comprehensive Center

■ WestEd

o American Institutes for Research (AIR)

o School Services of California

Florida and Islands 

Comprehensive Center

■ Educational Testing Service (ETS)

o University of South Florida - David C. Anchin Center 

o Edvantia

o Eastern Stream Center on Resources and Training (ESCORT)

o Florida Association of School Administrators (FASA)

o International Center for Leadership in Education (ICLE)

o JLM Professional Education Services

o Robin Wheeler, LLC

Great Lakes East 

Comprehensive Center

■ Learning Point Associates 

o RMC Research Corporation

o American Institutes for Research (AIR)

o Center for Applied Linguistics (CAL)

o University of Michigan - Consortium for Policy Research in 

Education (CPRE)

Great Lakes West 

Comprehensive Center

■ Learning Point Associates 

o American Institutes for Research (AIR)

o Mid-continent Research for Education and Learning (McREL)

o University of Wisconsin - Wisconsin Center for Educational 

Research (WCER)

o University of Michigan - Consortium for Policy Research in 

Education (CPRE)

Mid-Atlantic 

Comprehensive Center

■ George Washington University - Center for Equity and Excellence in 

Education (CEEE) 

o Edvantia

o Eastern Stream Center on Resources and Training (ESCORT)

o Group Jazz

o Southern Regional Education Board (SREB)

Mid-Continent 

Comprehensive Center

■ University of Oklahoma

o Northrop Grumman Information Tech

o Accion Social Comunitaria

o Mountain Plains Regional Resource Center (MPRRC)

o North Central Regional Resource Center (NCRRC)

o Southeast Regional Resource Center (SERRC of Alabama)

New England 

Comprehensive Center

■ RMC Research Corporation

o Education Development Center (EDC)

o WestEd

o Education Alliance at Brown University
A-2

Comprehensive Center Lead Grantee 

and Subgrantee Organizations (continued) 

Center name Lead grantee and subgrantees

New York 

Comprehensive Center

■ RMC Research Corporation

o Education Development Center (EDC)

o WestEd 

o Education Alliance at Brown University

o United Federation of Teachers Teacher Center (UFTTC)

North Central 

Comprehensive Center

■ Mid-continent Research for Education and Learning (McREL)

Northwest Regional 

Comprehensive Center

■ Northwest Regional Educational Laboratory (NWREL)

o RMC Research Corporation

Pacific Comprehensive 

Center 

■ Pacific Resources for Education and Learning (PREL)

Southeast 

Comprehensive Center

■ Southwest Educational Development Laboratory (SEDL)

o Center for the Education and Study of Diverse Populations 

(CESDP)

Southwest 

Comprehensive Center

■ WestED

o RMC Research Corporation

o American Institutes for Research (AIR)

Texas Comprehensive 

Center

■ Southwest Educational Development Laboratory (SEDL)

o Center for the Education and Study of Diverse Populations 

(CESDP)

Assessment and 

Accountability Content 

Center

■ WestEd

o National Center for Research on Evaluation Standards, and Student 

Testing (CRESST)

Center for Innovation 

and Improvement 

■ Academic Development Institute (ADI)

o Temple University - Center for Research in Human Development 

and Education (CRHDE)

o Little Planet Learning 

Center on Instruction

■ RMC Research Corporation

o Florida Center for Reading Research (FCRR)

o RG Research Group

o Horizon Research

o Texas Institute for Measurement, Evaluation and Statistics (TIMES)

o University of Texas at Austin - Vaughn Gross Center for Reading 

and Language Arts

National Comprehensive 

Center on Teacher 

Quality 

■ Learning Point Associates

o Education Commission of the States (ECS)

o Educational Testing Service (ETS)

o Vanderbilt University

National High School 

Center

■ American Institutes for Research (AIR)

o Learning Point Associates

o MDRC

o National Center for Educational Accountability (NCEA)

o WestEd

EXHIBIT READS: The Alaska Comprehensive Center was operated by the lead grantee organization named the 

Southeast Regional Resource Center (of Alaska) with support from the subgrantee organization named the 

Northwest Regional Educational Laboratory. 

Source: U.S. Department of Education; individual RCC and CC websites. 
Appendix B 

Study Sample 
B-1

Study Sample 

This appendix section presents the Project Inventory Forms that were collected to 

identify the sample of projects to be included in the study as well as the documents sent 

to Centers requesting materials for expert panel review of sampled projects. 

Project Inventory Form 

The Project Inventory Forms (PIF) was developed by the study team as a standard 

way for Centers to provide an inventory of their work for each program year. When asked 

to complete the inventory forms, Comprehensive Center Directors were informed about 

the specific project year for which information was being gathered (July 1, 2006 – June 

30, 2007) as well as the purpose of this information to be used in the selection of a 

sample of projects for review of quality by expert panels to be conducted for the national 

evaluation. When completing the PIF, each Center was asked to nominate several 

projects that they believed best represented the work undertaken by that Center. 

Additional projects were purposefully sampled from the remaining projects on each 

center‘s inventory to reflect each center‘s overall portfolio of work, as well as the work of 

all regional or content centers in key topic areas. A sample PIF was provided including 

examples of the kinds of projects that should be listed by the centers and examples of 

activities and resources defining each project level of effort – major, moderate, minor. 

Examples of some projects, activities, and deliverables that should not be included on the 

inventory form were also provided, including the following: 

■ Training or professional development for Comprehensive Center staff 

■ Work on coordinating committees within the Comprehensive Center 

network 

■ Annual needs assessment activity or negotiations with states, unrelated to 

specific projects 

■ Other internal working meetings or documents 

When completing the PIF, each center provided a list of each project under the 

appropriate topic heading (see list of topic areas and definitions included in PIFs, on p. B11). For each listed project, Centers followed instructions to provide information 

describing the projects and listing the activities and deliverables associated with each 

project (see section of PIF, Instructions for Completing an Inventory of Projects, p. B-4 –

B-8). 

Letters Requesting Materials for Expert Panel Review from Centers 

Each Center received a notification letter from the evaluation team that listed the 

projects from their PIF that had been selected for expert panel review followed by 

guidance in assembling the materials for panel review, compiling lists of project 

participants for the participant survey, and completion of the cover sheet for each 
B-2

sampled project. The initial request for materials letter and supporting information starts 

on page B-19. The evaluation team reviewed the materials submitted by the Centers for 

accuracy and completeness and in cases where there were missing or incomplete 

materials or where further explanation was needed, followed up with Centers using the 

follow-up memo provided later in this appendix starting on page B-26. 
Project Inventory Form

B-3

Comprehensive Center Project Inventory Form 

Dear Center Director,

Enclosed are a Project Inventory Form and instructions to be used to compile a complete 

list of your Center‘s projects during the current project year (July 1, 2006–June 30, 2007).

The national evaluation will use the completed project inventories to select a sample of 

projects for review by expert panels. Please designate several projects that you believe best 

represent the work of your Center; these projects will be included among the sample of projects 

to be reviewed by expert panels. These panels will rate nominated and sampled projects for their 

quality. 

Please complete a draft inventory to review with a member of the national evaluation 

team during the site visit for the evaluation, scheduled to take place between April and June 

2006 when [EVALUATION STAFF MEMBER] visits your Center. A final version of the 

project inventory should be completed as soon as possible after the site visit, but no later than 

June 30, 2007. Please return the completed inventory form to your evaluation liaison, [name] at 

_________@___________.

If you have any questions about the inventory or the instructions for completing it, please 

contact _______________ at 1-xxx-xxxx or by email at _____@_________.

Thank you for your continued support of the national evaluation of the comprehensive 

center program.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless it displays a valid OMB control number. The valid OMB control number for this data collection 

instrument is 1850-8023. The time required to complete these worksheets is estimated to average 16 hours per 

response, including the time to review instructions, search existing data sources, gather the data needed, and fill in 

the form. Send comments regarding this burden estimate or any other aspect of this collection of information, 

including suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651. 
Project Inventory Form

B-4

Instructions for Completing an Inventory of Projects 

Each Comprehensive Center will prepare an inventory of all of the projects that were 

active during the current grant period. The first inventory will cover the period from July 1, 2006 

through June 30, 2007. The second inventory will cover materials from July 1, 2007 through 

June 30, 2008, and the third inventory will cover July 1, 2008 through June 30, 2009. 

 Projects that began in the current project year but will not be complete by June 30 should 

be entered in the inventory and noted as ―ongoing.‖

Purpose

This inventory will serve as the sampling frame for the expert panel reviews to be 

conducted for the national evaluation. A sample of projects will be selected in two ways. Each 

Center will nominate several projects that they believe best represents the work undertaken by 

that Center. The remaining projects will be purposefully sampled from the remaining projects on 

each Center‘s inventory. Samples will be drawn to reflect each Center‘s overall portfolio of 

work, as well as the work of all Regional or Content Centers in key topic areas. Items selected 

for review will be rated by an expert panel for their quality. 

If a project is nominated or sampled for expert panel review and rating, the evaluation 

team will ask your Center to collect and transmit all of the documents and other artifacts 

associated with that project (meeting agendas, briefing books, meeting summaries, training 

materials, white papers, web resources, etc.) for the expert panel to review. 

Identifying Projects for Inclusion in the Inventory

The unit of analysis for the expert review panels will be the project. For the purposes of 

this inventory, a ―project‖ will ordinarily comprise a group of closely related activities and/or 

deliverables designed to achieve a specific outcome for a specific audience. The Content 

Centers have some projects that consist of the development of a single product or deliverable, 

rather than a group of deliverables, but a project that comprises only a single product or 

deliverable will be the exception rather than the rule. The inventory should include all of the 

projects developed or delivered by the Center, including those developed or delivered in 

collaboration with other Centers, during the reporting period. 

Because each project listed in the inventory could potentially be sampled for expert panel 

review, each project shown as an entry (or row) in the inventory form should be a relatively 

complete project that can be understood and rated on its own by expert panel members who may 

not know anything about other aspects of the Center‘s work. Although a single project may 

include a number of deliverables and activities, it will be designed to achieve a specific outcome 

and address (in almost all cases) a single topic. Where a group of activities and deliverables can 

be divided up into separate projects, each constituting a complete and coherent whole, the Center 

should list these as separate projects in the inventory. 

The following criteria should guide the Centers as they identify projects (and their 

associated activities and deliverables) for recording in the inventory form. They have been 

developed to ensure that projects constitute units that are large enough for review and rating, but 
Project Inventory Form

B-5

focused enough for coherence. Each project entered in the inventory should satisfy all three of 

the following criteria: 

■ Complete and coherent whole. Each project listed in the inventory should be 

able to stand on its own in an expert panel review. Avoid listing activities and 

deliverables as separate projects in the inventory if they cannot be understood or 

evaluated without information about related activities and deliverables. For 

example, a training event may require extensive planning (e.g., needs assessment, 

materials development) and follow-up activities (e.g., evaluation of the training, 

consultation on action plans). These planning and follow-up activities would be 

very difficult for a panel to rate in the absence of information about the event 

itself. Therefore the Center should list these planning and follow-up activities and 

the event itself as a single project (one row) on the inventory form. Each phase of 

the project—planning, the event itself, and follow-up—will be described briefly 

in the ―activities and deliverables‖ column. Similarly, ongoing work with a statelevel task force should be listed as a single project rather than each meeting of the 

task force being listed as a project.

■ Common intended outcome. Where a cluster of activities and deliverables is 

designed by the Center to lead to the same outcome for the same audience(s), 

those activities and deliverables should be grouped as one project in the inventory 

form. On the other hand, where a set of activities is intended to produce more than 

one distinct outcome—for example, helping the state to develop a strategic plan 

for improving reading instruction, and helping the same state recruit and train 

literacy coaches—those two activities should be listed as separate projects. Where 

the Center replicates the same set of activities in each of several different states, 

that set of activities should be listed as a single project if the intended outcomes 

and processes do not differ materially from one state to another. Where the 

intended outcomes do differ substantially from state to state, the work in each 

state should be listed as a separate project. 

■ Topic area focus. With few exceptions, a project addresses just one of the 14 topic 

areas described in Exhibit A at the end of this document (e.g., state systems of 

support, reading/language arts). Where it is possible to divide a group of related 

activities into two different projects according to the topic area addressed, Centers 

should do so. For example, a regional forum on interventions for low-performing 

students in reading and mathematics that offers separate strands of sessions in each 

subject area should be listed as two different projects, one under reading/language 

arts and one under mathematics. In this case, all of the sessions on reading 

interventions would stand on their own as a complete and coherent whole for rating 

by an expert panel and should be listed as a separate project; the same would be true 

for the sessions on mathematics. (If a project cuts across topic areas and the 

activities and deliverables which it comprises cannot be divided up by topic area 

into complete and coherent units that would make sense to a review panel, the 

project should be listed under the most relevant topic area with a note crossreferencing other topics, following the instructions for column IV below.) 
Project Inventory Form

B-6

The sample inventory form at the end of this packet includes examples of the kinds of 

projects that should be listed by the Centers. 

 Some projects, activities, and deliverables should not be included on the inventory form 

at all. These include: 

■ Training or professional development for Comprehensive Center staff 

■ Work on coordinating committees within the Comprehensive Center network 

■ Annual needs assessment activity or negotiations with states, unrelated to specific 

projects 

■ Other internal working meetings or documents 

Completing the Inventory Form 

Centers should use the attached form to complete their inventory of projects. Sample 

projects and examples of entries can be found in Exhibit B. 

 Centers might find it useful to review their annual project plans, technical assistance 

plans, management plans or technical assistance logs as a starting point since those documents 

typically provide an overview of the various projects and activities that were planned for or 

conducted during the year. 

 Once the inventory form is complete, Centers should designate which projects they would 

like to nominate for inclusion in the sample of projects reviewed by the expert panels, by inserting 

** after the name of the project in Column II. Centers should designate several projects. 

 List each project under the appropriate topic heading. A list of topic headings, with 

definitions, is attached at the end of this document in Exhibit A. 

Directions for completing each column are as follows: 
Project Inventory Form

B-7

Topic Area Headings List items under the appropriate topic area heading. Where a project 

fits under more than one topic heading, list it once, under the topic 

heading that is most relevant. Note the project‘s relevance to other 

topics in Column IV. Add rows to the table as needed. Leave rows 

blank if your Center does no work in a particular topic area. 

I. Project number Assign consecutive numbers to each item listed in the inventory. 

(Centers may want to complete this column as a final step, after all of 

the items have been entered in the inventory.)

II. Name Enter the name of the project. Projects nominated by Center staff for 

review should be followed by ** in this column.

III. Description Provide a concise description of the project. (See examples for 

appropriate level of detail.)

IV. Additional topics 

addressed?

If the project addresses more than one topic area, note that here. Use 

the list of topic areas provided in the appendix (this list corresponds 

with the row headings in the inventory). Entering the appropriate 

number from the list will save space.

V. Activities and 

Deliverables

List all of the activities and deliverables associated with the project. 

(For products, include exact title, if applicable. For services, include 

location and type of participant.) Specific activities and deliverables 

may include:

 Meetings/conferences (includes items such as workshops, 

conferences, institutes, forums, webinars)

 Expert consultation/technical assistance (includes items such as 

assistance completing reports or applications, review of state plans, 

needs assessments, audits)

 Facilitation/support of working groups or teams (includes items 

such as planning meetings, participation in meetings, drafting 

summary documents)

 Guidance/information resources (includes items such as policy or 

issue briefs, fact sheets, congressional testimony, resource guides, 

planning tools, field guides, benchmarking rubrics, handbooks, 

exemplars, literature reviews/summaries, annotated bibliographies, 

case studies, websites)

 Training (includes items such as professional development 

materials/services, software, training materials)

 Other, specify ______________________



VI. Start Date Enter the start date for the project, including month and year.

VII. End Date Enter the end date for the project, including month and year.

If the project is currently ongoing, enter the note ―ongoing.‖
Project Inventory Form

B-8

VIII: Major, 

Moderate or Minor 

Project 

Indicate whether you consider this a major, moderate or minor project 

in terms of the relative level of effort and/or resources devoted to it.

IX: Target state(s), 

region(s), or regional 

center (s)

Regional Comprehensive Centers serving multiple states should note 

which state(s) participated in each project. Regional Centers serving a 

single state (Alaska, California, New York, and Texas) should note 

which region(s) within the state participated (e.g., New York City vs. 

rest of state). Content Centers should specify which Regional Centers 

participated in the project.

X: Collaborations and 

Sources: Other CCs

If the Center used materials developed by one of the Content Centers 

in the course of designing or delivering its own services, list this 

Content Center as a source.

If the Center collaborated with another Regional Center or Content 

Center on the design, development and/or delivery of products and 

services, list this Center as a collaborator. 

Note collaborations with other Comprehensive Centers only in this 

column.

XI: Collaborations 

and Sources: Other 

TA Providers

Note collaborations with other technical assistance providers, such as 

Regional Educational Laboratories or universities, in this column.

If the Center used materials developed by another TA provider in the 

course of designing or delivering its own services, list this provider as 

a source.

If the Center collaborated with another TA provider on the design, 

development and/or delivery of products and services, list this 

provider as a collaborator.
B-9

National Evaluation of the Comprehensive Technical Assistance Centers 

Inventory of Projects 

Center Name: ________________________________ 

I. No. II. Name III. Description IV. Additional 

topics addressed? 

(From list in 

Exhibit A—enter 

appropriate 

number)

V. Activities and 

Deliverables

VI. Start Date VII. End 

Date (enter 

―ongoing‖

if project is 

not 

complete)

VIII. 

Major, 

moderate,

or minor 

project

IX. Target 

state(s), 

region(s) 

within a 

state, or 

Regional 

Centers(s)

X. 

Collaborations 

and 

Sources:

Other 

CCs 

XI. 

Collaborations 

and 

Sources: 

Other TA 

Providers 

1. State Systems of Support for Schools Identified for Improvement

2. Building District/Local Capacity to Support School Improvement, or Address Corrective Action and Restructuring

3. Assessment (excluding assessment of special education students or ELL students)

4. Reading/Language Arts Curriculum, Instruction, and/or Professional Development (includes Adolescent Literacy)

5. Mathematics Curriculum, Instruction, and/or Professional Development

6. Other Content Area Curriculum, Instruction, Professional Development, Standards-Based and Research-Based Instructional Frameworks

7. High School Reform

8. Special Education

9. English Language Learners
B-10

I. No. II. Name III. Description IV. Additional 

topics addressed? 

(From list in 

Exhibit A—enter 

appropriate 

number)

V. Activities and 

Deliverables

VI. Start Date VII. End 

Date (enter 

―ongoing‖ if 

project is 

not 

complete)

VIII. 

Major, 

moderate,

or minor 

project

IX. Target 

state(s), 

region(s) 

within a 

state, or 

Regional 

Centers(s)

X. 

Collaborations 

and 

Sources:

Other 

CCs 

XI. 

Collaborations 

and 

Sources: 

Other TA 

Providers 

10. Highly Qualified Teachers

11. Parent and Community Involvement

12. Supplemental Educational Services

13. Other NCLB-Related TA

14. Other

**Projects nominated by the Center for review by expert panels 
Project Inventory Form

B-11

Exhibit A 

Topic Areas and Definitions 

Topic Area Definition

1. State Systems of 

Support for Schools 

and Districts 

Identified for 

Improvement

Work that supports an SEA and/or its service delivery system carry out 

state-level responsibilities related to supporting district and schools 

identified for improvement. Service may target School Support Teams 

(SSTs), Distinguished Educators (DEs), or regional education service 

centers (RSCs, ESCs).

2. Building 

District/Local 

Capacity to Support 

School

Improvement, or 

Address Corrective 

Action and 

Restructuring

Includes any work designed to build district- and school-level capacity 

to carry out school improvement and to make AYP, as required under 

NCLB, including remedies taken for schools or districts in ―corrective 

action‖ or ―restructuring.‖

3. Assessment

Design and implementation of local assessment systems, such as 

benchmark assessments aligned to state assessments; classroom 

assessments designed to support instructional change; or support for 

analysis and use of data to drive instructional change. Includes 

consultation on the design and implementation of state assessment 

systems. Does NOT include alternate assessments for SPED or ELL.

4. Language Arts 

Curriculum, 

Instruction, and 

Professional 

Development

Includes all work at all levels that addresses ELA, reading, and literacy. 

Includes projects on adolescent literacy. May include work related to 

curriculum alignment, research-based models or programs, standards 

and policy reviews, and teacher professional development. 

5. Mathematics

Curriculum, 

Instruction, and 

Professional 

Development

Includes all work at all levels that addresses mathematics and 

numeracy. May include work related to curriculum alignment, 

research-based models or programs, standards and policy reviews, and 

teacher professional development. 

6. Other Content Area 

Curriculum, 

Instruction, and 

Professional 

Development; 

Standards-Based and 

Research-Based 

Instructional 

Frameworks

Includes all work at all levels that addresses specific content areas other 

than language arts or math, such as science and social studies. May 

include work related to curriculum alignment, research-based models or 

programs, standards and policy reviews, and teacher professional 

development. Includes work on curriculum alignment and policies 

related to academic standards that are NOT content area-specific. 
Project Inventory Form

B-12

Topic Area Definition

7. High School Reform Includes work related to smaller learning communities, ninth grade or 

high school transition, dropout prevention, high school policies.

8. Special Education

Includes all work related to special education, such as alternate 

assessments for the ―1 percent and 2 percent students‖ or 

instructional/assessment strategies that target needs of students with 

IEPs.

9. English Language 

Learners

Includes all work related to ELL (i.e. LEP) students, such as work on 

Annual Measurable Achievement Objectives (AMAOs) for ELL 

students and instruction/assessment of ELL students.

10. Highly Qualified 

Teachers

All work addressing the NCLB requirement that states have 100 

percent of teachers licensed to teach in their subject area and 100

percent of paraprofessionals with 2-years of college.

11.

12. Supplemental 

Educational Services

Includes work related to helping SEAs evaluate and monitor 

Supplemental Education Services, as well as setting up systems for 

notification and outreach to districts, schools, and parents. 

13. Other NCLB-related 

TA

Completing miscellaneous federal applications or reports, convening 

SEA staff to address miscellaneous NCLB topics; ongoing consultation 

on NCLB topics, including migrant education.

14. Other

Includes projects that do not fit under any other topic area, including 

websites, membership on SEA committees, and ongoing phone support. 

Includes projects building general skills and knowledge of leaders at all 

levels. Includes projects that assist clients with the review, collection, 

and translation of research, where there is no specific topic. If the 

research addresses a specific topic area, put the project in the 

appropriate row. Does not include internal or network meetings—these 

types of activities should not be included in the project inventory.
B-13

Exhibit B 

Sample Project Inventory 

Center Name: ______________________________ 

Note: All examples in the table were taken from Year 2 baseline management plans. 

I. No. II. Name III. Description IV. 

Additional 

topics 

addressed? 

V. Activities and Deliverables VI. 

Start 

Date

VII. 

End 

Date 

VIII. 

Major, 

moderate 

or minor 

project

IX. 

Target 

state(s), 

region(s), 

or 

centers(s)

X. 

Collaborations 

and 

Sources:

Other 

CCs 

XI. 

Collaborations 

and 

Sources: 

Other TA 

Providers 

1. State Systems of Support for Schools Identified for Improvement

1. State TA 

system for PI 

districts and 

schools

Designed an 

integrated SEA TA 

system that reaches 

―program 

improvement‖

districts and schools

 Convened coordinating council;

 Created inventory of existing TA 

efforts;

 Helped create TA plan;

 Gave PD and TA to school support 

teams

7/2006 Ongoing Moderate

2. Statewide 

system of 

support**

Supported 

development of 

Regional School 

Service Centers as a 

system of support for 

addressing NCLB

 Helped create a plan to guide the 

Regional Network Strategy for the 

next five to seven years;

 Developed the next RFP for the 

RSSCs;

 Supported RSSC implementation;

 Developed and helped deliver PD;

 Developed a protocol for collecting 

information on implementation 

7/2006 6/2007 Major State A

3. ―Significant 

Change in 

School 

Improvement 

and 

Restructuring‖

Developed a modular 

handbook and 

workshop on 

implementation of 

fast-paced significant 

school improvement, 

including 

restructuring

2 (Local 

capacity)

 Drafted 8 modules for handbook to 

be used with SEA systems of 

support;

 Presented at institute;

 Revised and added workshop 

materials;

 Consulted at CC A regional 

meeting

Major All 

Regional 

CCs

Regional 

CC A
B-14

4. Support to 

School 

Improvement 

Team

Helped the SEA 

School Improvement 

Team to identify 

resource and design 

training for schools in 

―monitored‖ and 

―priority‖ statuses

 Attended meetings organized by 

SIP team and provided reflective 

syntheses;

 Brokered resources and services 

available through RRC and CC 

networks

7/2006 6/2007 Minor State A

2. Building District/Local Capacity to Support School improvement, or Address Corrective Action and Restructuring

5. District tools 

for monitoring 

schools

Helped the SEA 

develop processes and 

tools for districts to 

use in monitoring the 

implementation of 

school restructuring 

plans

 Prepared draft processes and tools 

for monitoring;

 Met with SEA school improvement 

staff to review/revise tools

7/2006 10/2006 Minor State A

6. Leadership 

Institute on 

Helping 

Districts Assist 

LowPerforming 

Schools

Designed and 

implemented a PD 

plan to support the 

Local Education 

Agency Assistance 

Program

 Convened planning team to design 

the Leadership Institute;

 Helped conduct institute;

 Provided ongoing support to 

participants as they work with 

districts and schools

7/2006 6/2007 Moderate

7. Guidance to 

districts on 

restructuring 

schools

Assisted SEA in 

developing guidance 

for districts with 

schools in 

restructuring

 Assisted state in writing guidance 

protocol;

 Piloted guidance;

 Collected and prepared research;

 Provided initial training to districts 

with schools in restructuring

7/2006 12/2006 Moderate State A

3. Assessment (excluding assessment of special education students or ELL students)

8. Growth 

models

Built SEA 

understanding of 

assessment to help in 

decision making about 

accountability growth 

models

 Identified state needs and create 

plan for growth model;

 Assisted states accepted as pilot 

states;

 Assisted ongoing development for 

other states

7/2006 Ongoing Major All 

states in 

region
B-15

9. Assessment 

data 

analysis**

Built SEA and other 

staff capacity by 

helping them train 

districts and schools 

in analyzing state 

assessment results and 

using them for 

instructional planning

 Met with SEA staff to plan PD;

 Helped host Statewide Assessment 

Conference;

 Continued to plan and offered 

training in assessment analysis;

 Established a cadre of educators to 

provide PD to districts

5/2006 6/2007 Major State A Content 

CC A

4. Reading/Language Arts Curriculum, Instruction, and/or Professional Development (includes Adolescent Literacy)

10. Adolescent 

Literacy 

Cadres

Organized and 

delivered training of 

adolescent literacy 

cadres made up of 

SEA staff

 Convened SEA work teams to 

analyze research;

 Reviewed literacy diagnostic tools;

 Customized a literacy assessment;

 Conducted PD for SEA work teams 

on customized assessments

10/200

6 

6/2007 Moderate All 

states in 

region

Content 

CC A; 

Content 

CC B; 

Content 

CC C

11. Effective 

Leadership in 

Literacy

Built SEA capacity to 

support ―effective 

leadership in literacy 

for grades 6-12‖

 Convened work group to discuss 

research; 

 Provided SEA with PD on Strategic 

Instruction Model;

 Developed state literacy plan

7/2006 6/2007 Major State A; 

State B

Content 

CC A

5. Mathematics Curriculum, Instruction, and/or Professional Development

12. K-2 

Mathematics 

Inventory

Developed and 

provided PD to a 

cadre of educators to 

help schools 

implement the state 

K-2 Mathematics 

Inventory

 Helped SEA design PD;

 Helped design team pilot 

workshop;

 Revised PD;

 Helped deliver regional trainings 

prioritized for schools in 

improvement;

 Reviewed feedback to redesign PD

7/2006 6/2007 Major State A Content 

CC A
B-16

7. High School Reform

13. Implementing 

High School 

Task Force 

Report

Developed capacity of 

SEA to implement 

recommendations of 

HS Task Force Report 

related to high school 

redesign

 Work group and Steering 

Committee met to consider task 

force recommendations, research, 

best practice, and review of prior 

initiatives;

 Identified ―knowledge 

management‖ components needed 

to support state implementation;

 Identified policies and practices 

needed to promote redesign;

 Developed multi-year 

implementation plan.

9/2006 4/2007 Moderate

8. Special Education

14. Assessing 

special needs 

students

Supported SEAs in 

establishing 

assessment and 

accountability systems 

that include and 

support special 

education, ELL, and 

low-income students

10 (ELL)  Collected information from Content 

CC A on research- based practices;

 Conducted SEA Needs 

Assessment;

 Provided regional training(s) on 

Scientifically Based Research on 

assessment and accountability 

systems for special needs students

7/2006 Ongoing Major All 

states in 

region

Content 

CC A

15. Developing an 

Alternate 

Assessment

Assisted SEAs in 

developing Alternate 

Academic 

Assessments for the 

―1‖ of students who 

have significant 

cognitive disabilities 

3 

(Assessmen

t)

 Identified needs and broker 

assistance in developing 

assessment;

 Provided TA on development of 

assessment;

 Provided ongoing TA on 

implementation

7/2006 6/2007 Moderate State A;

State B

Content 

CC A

16. Research 

Synthesis on 

Special Needs 

Students

Published Research 

Synthesis 1 focusing 

on teaching specialneeds students 

12 (HQT)  Identified and evaluated existing 

research and syntheses;

 Conducted gap analysis to identify 

research synthesis needs;

 Wrote synthesis (50-60 pages);

 Posted on on-line resource forum.

7/2006 9/2007 Major All 

Regional 

CCs
B-17

9. English Language learners

17. District 

guidance on 

the education 

of ELL 

students

Helped develop a 

comprehensive 

framework for 

educating English 

Language Learners to 

guide district work, 

including guidance on 

the use of formative 

assessments to 

improve instruction 

and on family and 

community 

engagement

13 (Parents)  Facilitated meetings of the SEA 

task force responsible for 

developing and disseminating a 

comprehensive framework for the 

education of ELLs;

 Assisted in integrating effective 

formative assessment practices for 

ELL students;

 Assisted in formulating guidance 

on ELL family and community 

engagement strategies;

 Provided consultation and 

resources from Content CC A

7/2006 6/2007 Major State A Content 

CC A

18. Addressing 

AMAOs

Assisted SEA in 

aligning Annual 

Measurable 

Achievement 

Objectives, standards, 

and assessments for 

ELL students

 Conducted a study on alignment of 

English Language Proficiency 

standards for grades K-12 with the 

state assessment for ELL students

 Helped state reconfigure AMAOs 

to align with state test

8/2006 12/2006 Minor State A;

State B;

State C

10. Highly Qualified Teachers

19. ―TQ Source 

Project‖

Developed online 

resource 

clearinghouse that 

provides clients with 

policy and research 

data specifically 

related to teacher 

quality, as addressed

in NCLB

 Developed ―interactive data tool‖

based primarily on NCES data;

 Updated ―policy database‖ for

state-by-state policy areas, 

including policies related to PD and 

teacher prep, recruitment, retention, 

certification, etc.;

 Published next issue of ―Tips and 

Tools: Emerging Strategies to 

Enhance Teacher Quality‖;

 Enhanced ―Publications database‖

focusing on teacher quality 

research

7/2006 Ongoing Major All 

Regional 

CCs
B-18

20. Teacher 

Preparation 

and Licensure 

in literacy

Assisted SEA in 

reviewing teacher 

preparation and 

licensure requirements 

related to literacy

4 

(Language 

arts)

 Assisted the Task Force on 

Licensure/Professional 

Development in revisiting 

preparation and licensure 

requirements related to literacy 

instruction for school leaders and 

teachers;

 Discussed findings and possible 

next steps related to engagements 

with representatives from institutes 

of higher education

10/200

6 

2/2007 Minor State A

12. Supplemental Educational Services

21. ―State 

Evaluation of 

Supplemental 

Educational 

Services‖

Manual

Updated ―State 

Evaluation of SES‖

manual and assist with 

its use by Regional 

CCs 

 Drafted updates to SES evaluation 

manual

 Presented at September 2006 

Institute

 Revised based on evaluations and 

usage

 Submitted evaluation report to ED

9/2006 4/2007 Major All 

Regional 

CCs

Regional 

CC A

13. Other NCLB-related TA

22. Enhancing 

Schoolwide 

Planning 

Materials

Assisted SEA in 

updating resources 

and training for 

district Title I 

directors on the topics 

of schoolwide 

planning, plan 

implementation, and 

district monitoring of 

school plans

 Helped update T1 schoolwide 

application and evaluation rubric;

 Ensured that North Central 

Association Commission on 

Accreditation and School 

Improvement school improvement 

processes include schoolwide plan 

components;

 Reviewed need assessment results 

from district/schools;

 Updated materials based on needs

7/2006 6/2007 Major State A

**Projects nominated by the Center for review by expert panels
Request for Materials for Expert Panel Reviews 

B-19

REQUEST FOR MATERIALS FOR EXPERT PANEL REVIEWS 

 

September 10, 2007

Dear [Center Director]: 

I am happy to report that we have completed the sampling process for your site and have 

selected the following (n) projects for review: 

Project Name Project Description

These projects will be rated for technical quality by panels of expert reviewers, and for 

relevance and usefulness through surveys of project participants. 

Assembling Panel Review Materials 

Attached you will find a Request for Materials form that includes some basic guidance 

for assembling materials for the panel review process. This form also includes the project 

cover sheet, which should be completed for each project. This cover sheet will be 

provided to the review panelists and is an important source of background and contextual 

information about the project. 

To facilitate the review process, we ask that you do the following: 

 Submit 1 hard copy and 1 electronic version of all the materials that comprise the 

review packet for each project. The hard copy version will allow us to assess whether 

the volume of material is appropriate, and will serve as a quick reference source 

should questions arise about the review packet. 
Request for Materials for Expert Panel Reviews 

B-20

 If an item is not available in electronic format or if you would prefer the reviewers to 

have a hard copy version of a particular item (for example, a training binder with 

multiple tabs and color handouts included), please provide 3 additional hard copies of 

that item. 

 Each review packet should include a cover sheet (using the attached template) that 

lists all of the materials included in the packet. Electronic versions of the materials 

should be filed together in a project folder that is clearly labeled with the project 

name. Feel free to use subfolders within the project folder if you think it will help 

reviewers organize the materials (for example, background/research base, meeting 

notes, training materials, etc.) in a more understandable way. You might also want to 

include the file name of each document referenced in question #5 of the cover sheet 

so that reviewers can easily sort through and match the documents on the cover sheet 

with the files they receive. 

 Please send your hard copy materials (bundled by project with a cover sheet on each), 

along with a CD containing separate folders for each of the sampled projects to: 

DIR 

Attn: _______________

[address] 

Materials are due no later than Oct 19, 2007.

I have attached a copy of the quality scoring rubric so that you can see what the panelists 

will be looking for in the review process. As you begin to assemble the materials for 

panel review, we offer the following guidance: 

 With the exception of the cover sheet and the participant list, all materials provided in 

the review package should be materials that already exist. We do not expect you to 

create any new materials for the review process. 

 Materials included in the packet should focus on work conducted between summer 

2006 and summer 2007. Materials developed before summer 2006 can be included if 

they provide relevant contextual or background information. Materials developed 

after summer 2007 will be considered in the next round of panel reviews (for the 

2007-2008 program year), if that particular project is sampled again for review. 

 Make sure you have documented the ―knowledge base‖ upon which the project was 

developed. As you can see in the attached rubric, this is a very important factor in the 

quality rating. Depending on the topic area and nature of the project, the knowledge 
Request for Materials for Expert Panel Reviews 

B-21

base might include empirical research, laws and regulations, or professional 

wisdom68. 

 Provide enough materials to allow reviewers to understand the project substantively, 

with a particular focus on the work conducted in the 2006-2007 program year. This 

does not mean sending everything ever done or developed in connection with the 

project. You want to provide reviewers with a sufficient basis for their ratings but at 

the same time, not overwhelm them with materials. 

 Focus more on the substantive materials than on those dealing with process or 

administrative issues alone. You do not need to include every e-mail message or set 

of meeting notes on a project unless it provides critical contextual information or 

speaks directly to the technical quality of the project. For example, if you are 

assembling materials about an annual forum or institute, you should not include 

meeting notes about conference logistics or attendee evaluation results, but you 

should include notes relating to the development of the agenda and selection of 

presenters. 

If you have any questions about what materials to include in the review packet or how to 

complete the cover sheet, please contact your site visitor. 

Compiling Lists of Project Participants 

For each project selected for the evaluation, a sample of participants will receive a short, 

web-based survey from the national evaluation. This is the reason we are asking each 

Regional Comprehensive Center (RCC) and each Content Center to compile a list of all

state-level staff who have participated in each project sampled for the evaluation. Content 

Centers should also include all RCC-based staff, as well as all state-level staff, on their 

lists of project participants. Compile a separate list of participants for each project 

sampled, allowing duplication of names across lists. 

Defining “participants.” ―Project participants‖ include all those who have served on task 

forces and work groups associated with the project; state-level or RCC staff who have 

attended conferences, technical assistance retreats, consultations, and other meetings that 

are part of the project; and state-level or RCC staff who have received written materials 

or other resources disseminated under the project. 

If the Comprehensive Center collaborated with state-level staff to provide a service to 

others, and that collaboration was intended to build the capacity of the state in some way, 

then all of the state-level staff who collaborated with the Center on the project should be 

 

68 Professional wisdom is the judgment that individuals acquire through experience and is reflected through 

consensus views on decisions on implementation. Grover J. Whitehurst. Evidence-Based Education, 

2002.
Request for Materials for Expert Panel Reviews 

B-22

included on the list of project participants. These collaborators will be included in the 

pool of potential survey respondents. 

The same logic applies to Content Center projects that involve collaborations with RCC 

staff. Again, if the collaboration was intended to build the capacity of the RCC in some 

way, then all RCC staff who collaborated on the project with the Content Center should 

be included on the list of project participants and will be included in the pool of potential 

survey respondents. 

Defining “state-level.” For the purposes of this evaluation, ―state-level‖ participants in 

Regional Center or Content Center projects may include the following: 

 State education agency (SEA) employees and consultants 

 Employees of other state agencies, such as governors‘ offices 

 Employees of intermediate education agencies69 who assist schools on behalf of 

the state 

 Individuals who serve on school support teams as part of a statewide system of 

school support under NCLB, including local educators (school district 

administrators, principals, and teachers)

 Individuals who serve on state-level work groups or task forces, including local 

educators 

Local educators who are not serving on school support teams and are not members of a 

state-level work group or task force should not be included in project participant lists. 

RCC staff. Content Centers should include all RCC staff who have participated in each 

project, in addition to state-level project participants. Participant lists for some Content 

Center projects may include no RCC staff, some may include both RCC and state-level 

staff, and some may be made up entirely of RCC staff, depending on the nature of the 

project. 

Contact information. For each project participant, please provide a name, title (if 

available), affiliation, email address, telephone number, and regular mail address. 

Because surveys will be administered via the web, with follow-up by regular mail and by 

telephone, all of this contact information is critical. Send a separate participant list for 

 

69 Intermediate education agencies are usually established by state statute, but their governance structures 

and funding sources vary from state to state. Depending on the state, they are known as Area Education 

Agencies (AEA), Boards of Cooperative Educational Services (BOCES), Cooperative Education Service 

Agencies (CESA), County Offices of Education (COE), Education Service Centers/Cooperatives (ESC), 

Education Service Districts (ESD), Regional Education Service Agency (RESA), or Regional Education 

Service Centers (RESC). Association of Educational Service Agencies, ―Questions Asked About 

Educational Service Agencies,‖ downloaded from http://www.aesa.us/Q&ABro04.pdf on July 6, 2007. 
Request for Materials for Expert Panel Reviews 

B-23

each sampled project, following the template shown below (using either Word or Excel to 

create the file). 

Project name: Really Important Project

Last 

name

First 

name

Title (if 

available)

Affiliation/ 

Organization Email address

Telephone 

number

Street 

address City State ZIP

Doe John Title I 

Director

State Department of 

Public Instruction

jdoe@sea.k12.us (101) 555-1234 99 State 

Street, 

Room 100

State 

Capital

ST 10001

Etc.

Participant lists for each project should be included with your sample review materials 

and sent to DIR no later than Oct 19, 2007.

For additional information. If you have any questions about the participant lists, or if 

there are special circumstances pertaining to a project or to a group of participants that 

you would like to discuss, please contact _______________ at Policy Studies 

Associates. _______________ can be reached at (xxx) xxx-xxxx or by email at 

_____@_________. 

Thank you for your continued cooperation. We look forward to receiving your materials. 

Sincerely, 

Vice President 

Decision Information Resources 
Request for Materials for Expert Panel Reviews 

B-24

National Evaluation of the Comprehensive Technical Assistance Centers 

Request for Materials for Expert Panel Review 

The projects below have been sampled from the Project Inventory Form that your Center completed for the most 

recent project year. These projects will be rated for quality by expert review panelists for the national evaluation 

of the Comprehensive Technical Assistance Centers. Surveys of participants in these projects will collect client 

ratings of the projects‘ relevance and usefulness. 

Project Activities and Deliverables

In order to facilitate this process, for each project please provide us with: 

(1) Copies of the materials that together fairly represent each of these items. This should include: documents 

(electronically, wherever possible) leading up to, or resulting from, these efforts–including plans, agendas, 

meeting notes, handouts, presentations, follow-up memos, resolutions, instruments, assessments, summaries, 

syntheses, papers, reports and memoranda of agreement. In other words, we are asking for materials that fully 

describe these products and services for the reviewers and give them a sufficient basis for rating their quality. 

(2) A cover sheet (one per project) that provides background and contextual information about the project. (See 

attached page for specific questions to be answered on the cover sheet.) 

(3) A list of the state-level participants in each project, including names, affiliations, and contact information. 

Questions about what materials to provide and how to complete the cover sheet should be directed to your 

evaluation team liaison. 

These materials will be the basis upon which the reviewers will make their ratings. 

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless it displays a valid OMB control number. The valid OMB control number for this data 

collection instrument is 1850-0823. The time required to complete these worksheets is estimated to average four

hours per response, including the time to review instructions, search existing data sources, gather the data needed, 

and respond to the questions. Send comments regarding this burden estimate or any other aspect of this collection 

of information, including suggestions for reducing the burden, to the U.S. Department of Education, Washington, 

DC 20202-4651. 
Request for Materials for Expert Panel Reviews 

B-25

Cover Sheet to Accompany Materials Sent to Review Panel for Sampled Projects 

Name of Comprehensive Center ________________________________ 

Name of person completing this form ____________________________ 

Telephone ________________ Email ____________________ 

Information about Sampled Projects: 

Project Name ______________________________________ 

Project Inventory # _______ 

1. Describe the purpose of this project and its intended outcome(s). 

2. Who is the customer/client? 

3. Describe the circumstances that led to the provision of this project. In this description, 

be sure to indicate whether it was initiated by the Center or requested by the client. 

4. List the activities and deliverables associated with this project, and describe how they 

relate to the overall project. 

5. Please provide a list of the materials and documents you are sending for the review 

panel. For each, describe the Comprehensive Center‘s role and contribution to the 

materials associated with this project (i.e., the materials sent to reviewers for rating 

the quality and relevance of this project.) Please let us know if you used materials 

developed by another source and, if so, provide information about that source.

6. What is the research basis for the product or service? Where appropriate, please 

provide a citation for the research or practice literature upon which it is based. 
Sample Follow-Up Memo 

B-26

Sample Follow-Up Memo 

Dear [Center Director]: 

Members of the evaluation team met in Houston last week to look over the 

review materials submitted by the Centers. In our cursory review of the PCC 

projects, we had trouble locating certain items that were either described on the 

project cover sheet(s) or that one would expect to see in the project materials. 

Project Name Issue Identified

If these items were included in your review package, please let us know where in 

the materials we can locate them so that we can make sure that the peer review 

panelists will be able to find them. 

If you didn’t intend to include them in the package, please let us know. 

If you would like to send new copies of these items or additional materials to 

address the issue(s) identified, please forward copies electronically to 

_____@_________ for hard copies to [name], DIR, [address] no later than 

November 8, 2007. 

If you have any questions, please feel free to contact me. 

Best, 

[name] 
Appendix C 

Describing Center Operations 
C-1

Describing Center Operations

 This appendix contains a description of the process used to code the objectives within 

Center management plans, the PIFs, and the project cover sheets. This appendix also includes a 

description of the site visits, the open-ended interview protocols used for the 2007 site visits to 

CCs and RCCs, and the closed-ended prompts administered to the Centers in summer 2008. 
C-2

Center Management Plan, Project Inventory, and Cover Sheet 

Coding and Intercoder Reliability 

 Data from the Center management plans, project inventory forms (PIFs), and the project 

cover sheets were coded by the evaluation team to build the sample frame and describe Center 

operations. Below is an explanation of the set of four procedures used in the coding process–(1) 

establish the code sets, (2) train coders, (3) code management plans, PIFs, and cover sheets, (4) 

resolve coding disagreements–as well as a summary of the level of intercoder reliability achieved 

through this process. 

Establish the Code Sets 

 Based on initial review of the project cover sheets, the PIFs, and the other materials from 

the Centers (such as the Center management plans), the evaluation team established code sets to 

use during the coding process. One set of codes was used to code the objectives stated in the 

Center management plans and the topics of the projects on the PIFs. Another set of codes was 

used to code the project cover sheets. 

Center Management Plan and PIF Code Set 

The team developed codes to summarize the primary topics in which Centers were doing 

their work. Review of Center materials resulted in the identification of categories of distinct 

domains or topics. These 22 specific topic codes are described below in terms of the substantive 

knowledge base the evaluation team determined would be covered by that topic.70 In the coding 

process, this set of possible codes was used to code the topic areas for the Center objectives as 

stated in their management plans as well as to code the topic area of each project that Centers 

submitted on their PIFs. 

■ Components of effective systems of support–state, district, school. This category 

included: (1) Design of state systems to meet NCLB requirements for statewide 

systems of support for districts and schools in corrective action or identified as in 

need of improvement [Sec. 1117(a)]; (2) Interventions for schools or districts in 

need of improvement or corrective action; (3) For schools, Centers' projects 

addressed a range of options for school leadership and organization, professional 

development and classroom practice, and development and implementation of 

school improvement plans. For districts, Centers' projects typically addressed

district leadership and development as well as implementation of district 

improvement plans.

 

70 Discussion in the report refers primarily to the 22 specific substantive topics. However, as a small number of 

sampled projects fell into the category of ―other‖ topic area, description of all the projects in the study sample by 

topic as displayed in Exhibit 2.4 includes the topic area of ―other‖ as the 23rd category. 
C-3

■ Data use/Data-driven decisionmaking. Use of data for educational program 

improvement. 

■ Formative assessment. Guidelines for implementing formative or benchmark 

assessment, comprehensive formative assessment systems.

■ Reading. Alignment of curriculum, instruction, and assessments with state 

standards in reading; teacher professional development in reading (not specific to 

adolescent literacy).

■ Adolescent literacy. Policy and practices relevant to literacy at middle school and 

high school levels. 

■ Mathematics. Alignment of curriculum, instruction, and assessments with state 

standards in mathematics as well as teacher professional development in 

mathematics. 

■ Dropout prevention. Policies and practices specifically designed to keep high 

school students in school through graduation. 

■ High school redesign/reform. High school renewal, school organization, 

curriculum, and instructional practice.

■ Transition to high school. Policies and practices designed to improve the 

preparation of middle-school students for success in high school. 

■ Special education–curriculum, instruction, and professional development. 

Curriculum, instruction, and professional development for service to students with 

disabilities. 

■ Special education–assessment. Implementation and interpretation of assessments 

for students with disabilities, including alternate assessments.

■ English Language Learners. Curriculum, instruction, and professional 

development for service to ELL students; Title III improvement plans. This 

category also includes determination of ELLs‘ achievement of AYP through 

assessment of Annual Measurable Achievement Objectives; implementation and 

interpretation of ELL assessments. 

■ Highly Qualified Teacher provisions of NCLB. State plans and policies required 

in connection with the HQT provisions of NCLB, including HOUSSE and 

equitable distribution.

■ Teacher preparation and induction. State policies and practices for pre-service 

teacher preparation and induction of novice teachers as well as activities targeting 

teacher retention.

■ Teacher professional development. State policies and local practices for the 

planning and delivery of professional development for teachers (not specific to 

reading, mathematics, high schools, ELLs, or special education). Activities in the 

area of effective teaching are included in this category.

■ Supplemental Educational Services (SES). Application process to select vendors, 

monitoring services, and evaluating performance of providers.

■ Response to Intervention (RtI). RtI implementation, moving RtI into general 

education, RtI as a strategy for school improvement. 

■ Migrant education. Statutorily required state Comprehensive Needs Assessment 

(CNA) process for migrant education. 
C-4

■ Indian/Native American education. Policies and practices to support the 

educational needs of American Indians, Alaska Natives, and Native Hawaiians; 

includes support of and compliance with ESEA Title VII.

■ Data management compliance. Title I compliance, EDEN/EDFacts reporting 

requirements. 

■ Assessment design. Developing new assessments for use by clients; includes 

classroom, benchmark, and state grade-level assessments for federal 

accountability; activities may include design, item development, validation, 

reliability testing, alignment studies.

■ Parent involvement. Title I and other parent involvement activities. 

Cover Sheet Code Sets 

The team developed codes to describe project activities and resources, contributions CCs 

made to RCC projects, and contributions RCCs made to CC projects. The development of these 

codes was informed by the team‘s review of the Center management plans, the transcripts of the 

interviews conducted in 2007, and a preliminary review of all project cover sheets. The code sets 

used to code the project cover sheets are shown below along with the criteria used to assign that 

code. 

Types of Project Activities and Resources

■ Engagement of participants in project planning was coded for projects that 

offered some or all of their participants the opportunity to take part in designing 

the project. This involvement went beyond merely expressing a need for technical 

assistance; rather, it included participant involvement in shaping the specific 

assistance that was delivered (who did what, when, and why in technical 

assistance) at various stages of technical assistance delivery). 

Project planning was different from ―ongoing consultation and follow-up,‖ though 

the same project may have had both components. ―Ongoing consultation and 

follow-up‖ occurred when the Center provided service to clients on multiple 

occasions, whereas ―engagement of participants in project planning‖ was specific 

to the Center working with the client to plan later service. 

■ Research collections or syntheses were research-based materials distributed to 

participants in a project. In some instances these products were developed by the 

Center conducting the project, but in others they were existing research materials 

that the Center collected and distributed. 

■ Training events were designed to impart skills to participants, equipping them to 

carry out a particular program or strategy. The content of trainings was discrete 

and specialized, and participants attended trainings expecting to learn something 

specific and be able to do something differently afterward. Participants and 

trainers assumed that the primary purpose of the training was for the participant to 
C-5

learn something from the trainer. Projects with multiple trainings to the same 

participants over time were also coded as ―ongoing consultation and follow-up.‖

■ Conferences featured presentations of information to audiences from multiple 

speakers or discussants representing a range of perspectives. They focused on 

informing the participants, not training them, and they were single events rather 

than series of meetings. The conference scope was broader than trainings, in that 

presentations or discussions in a single conference could have addressed multiple 

issues, programs, or solutions; whereas a training focused on a single strategy or 

program. Conference participants varied in their type, interest, and expectation for 

what they wanted to learn at the conference. Conferences also included symposia 

and forums in which participants came at common questions with varying 

perspectives. 

■ Support for development of a formal plan to implement a program or policy 

referred to instances in which Center technical assistance supported work by 

participants that led to a plan for implementing a program or policy. A formal 

state plan codified activities that were expected to occur in the state in order to 

serve state purposes with the intention to guide individuals and units toward a 

common goal. Formal plans may have been required by the federal or state 

government, or may have been initiated within the SEA to address a specific 

problem. This activity did not include service plans. 

■ Task force meetings and work were engagements over multiple interactions in 

which a Center provided technical assistance to a work group who were 

themselves constituted as a task force to address a state problem. Center support 

of a task force meant that the Center was a regular participant in task force 

meetings, and Center support ranged from setting the agenda and facilitating 

meetings to providing input upon request. A project was coded as ―task force 

meetings and work,‖ when there was a task force that was trying to accomplish 

some purpose—not just to plan or follow up on Center technical assistance, but to 

do something for its state (or, occasionally, a group of states). Center participation 

in a task force was also coded ―ongoing consultation and follow-up‖ if the Center 

provided assistance on an ongoing basis over time. If the Center was working 

with the task force to plan service, the project was also coded ―engagement of 

participants in project planning.‖ 

■ Ongoing consultation and follow-up was either a series of consultations (i.e., 

fulfilling repeated requests from participants for technical assistance) or sustained 

follow-up with individuals who had participated in some other kind of assistance 

(e.g., a conference or training). Ongoing consultation occurred when there were 

multiple service contacts over time, all related to the project goal or topic. The 

interactions focused on a client question or need. This code did not include 

projects with one-time contacts, such as projects that provided a large conference 

with no follow-up. It also did not include the production of publications that are 

not accompanied with follow-up assistance directly to the client. 
C-6

RCC and CC Contributions to RCC and CC Projects 

■ RCC projects with CC contribution were projects in which the CC explicitly 

contributed a service or resource used by the RCC or its clients as part of RCC 

project activities. A CC could contribute to an RCC project in any of three ways: 

■ CC provided materials used in this RCC project. The RCC must have 

used CC materials in specific delivery activities or in planning. 

■ CC provided advice to the RCC on this project. The RCC consulted or 

collaborated with the CC when making decisions about what or how to 

deliver assistance to project participants. Participants may or may not have 

had direct interaction with the CC or CC materials. 

■ CC delivered technical assistance to RCC project participants. The CC 

provided service directly to the participants, whether face-to-face, by 

phone, or virtually, regardless of what role RCC staff had in this RCC

project activity. 

■ CC projects with RCC contribution were projects in which the RCC explicitly 

contributed assistance used by the CC or its clients as part of CC project 

activities. An RCC could contribute to a CC project in either of two ways: 

■ RCC recruited participants or brokered service. The RCC connected the 

CC and its clients for the purpose of participation in the project. The RCC 

may have been driven by the CC‘s need for participants who could benefit 

from a specific project, or by a client‘s need for specific services or 

products that a CC could deliver. 

■ RCC delivered technical assistance to CC project participants. The RCC 

provided service directly to participants, whether face to face, by phone, or 

virtually, regardless of what role CC staff had in this CC project activity. 

Train Coders 

All coders were senior members of the evaluation team and were trained to use the 

established code sets. As part of the training, coders discussed each of the codes in the code set 

to ensure all coders had a shared understanding of their meaning. They also coded sample items 

and reviewed, coded, and discussed the final codes using representative sample text. The 

evaluation team also used the discussions in the coder training to verify that the established 

codes were clear and well understood. 
C-7

Code Center Management Plans, PIFs, and Cover Sheets 

All of the objectives in the Center management plans, the PIFs, and cover sheets were 

independently coded by two coders. For the Center management plans, each coder reviewed the 

objectives section of the plan and determined the topic(s) in which the Center indicated they 

planned to work. For the PIFs, each coder reviewed the project description and assigned the 

project subject code for each of the 364 projects. The cover sheets were reviewed by each coder 

to categorize the project information submitted by Centers (i.e., project purpose, customer/client, 

project activities and deliverables, roles and contributions of Centers, research basis for products 

and services) and to assign codes according to the coding criteria described earlier (see p. C-5) 

for identifying the various types of project activities and resources and the ways Centers 

contributed to projects. 

Resolve Coding Disagreements 

Members of the evaluation team compared the coding worksheets from each coder to 

determine the final coded responses for each item. In cases where the coders agreed, the final 

response was clear. In cases where the first and second coders did not arrive at the same code, 

they met to review evidence for the codes and choose a final code. If the first and second coder

were unable to reconcile their codes, a third senior study team member worked with the coders to 

determine the final code for that item. The product of this step was the final codes for the topic 

areas included in the Center management plan objectives, the topic areas for the 364 projects on 

the PIFs, and the three areas described above for the project cover sheets. 

Analysis of Intercoder Reliability 

The evaluation team calculated the percent agreement between the first and second coder 

as well as Cohen‘s Kappa analysis. The percent agreement calculation used the number of 

agreements divided by total number of possible responses. The Cohen‘s Kappa calculation used 

the number observed to agree minus the number expected to agree by chance divided by the 

number of items minus the number expected to agree. 

For the topic area assignment of the Center objectives stated on their management plans, 

the first and second coder agreed on 452 of 462 possible topic area assignments (98 percent 

agreement across the 22 topic areas for each of 21 Centers). The overall Cohen‘s Kappa between 

the two coders was .96 for this coding activity. 

For the PIFs, the first and second coder agreed 87 percent of the time (on 318 of the 364 

subject codes) and the overall Cohen‘s Kappa between the two coders was .92. Chapter 3 of the 

report used this information at an aggregated level–work done by subject, by Center–showing 

whether or not each Center had at least one project listed on its PIF for each of the 22 subject 

areas. To determine the intercoder reliability at this aggregate level, the evaluation team 

examined the final 462 cell matrix (22 topic areas for each of 21 Centers) showing the subject 
C-8

areas in which each Center reported projects on its PIF against the similar matrices for the first 

and second coder. When viewed at this level, the intercoder reliability was even higher. Of the 

462 possible yes/no designations, 98 percent were the same for the first and second coder, with a 

Cohen‘s Kappa of .95.

For the project cover sheets, the first and second coder agreed on 1,572 of 1,830 possible 

codes (122 projects with a possible 15 codes for each project) or 86 percent of the time. The 

overall Cohen‘s Kappa between the two coders was .89.

Site Visit Interviews 

Two rounds of site visits were conducted with the Center Director and other senior 

Center staff to gather information about the Center operations during the 2006-07 program year 

through structured interviews. The first round of interviews in summer 2007 were based on 

open-ended questions posed to RCC and CC staff about Center organization, major areas of 

focus, communication with client organizations, sources of knowledge used, approaches taken in 

quality assurance, modes of delivering technical assistance, and barriers they encountered (see 

page C-9 for the CC protocol and page C-12 for the RCC staff interview protocol). The protocol 

for the second interviews included closed-ended, binary questions to follow up specifically about

activities that had occurred in 2006-07, topics that had been addressed, and work requests that 

had fallen outside their planned scope of work. Additional questions targeted the strategies that 

Centers had used for planning work and engaging clients, the sources for content expertise, and 

sources they used for vetting their products and services (see protocol on page C-15). Responses 

to the second interview prompts are reported in chapter 3 as part of the description of Center 

operations. 
CC Staff Site Visit Interview Protocol 

C-9

National Evaluation of the 

Comprehensive Technical Assistance Centers 

Content Center (CC) Staff 

Site Visit Interview Protocol 

1. Tell me about your professional background and areas of expertise. What is your primary 

role within this Center? Describe the tasks you might perform or decisions you might

make during a typical one- or two-week period? 

2. Give us an overview of your Center‘s internal structure and operations:

--How are staff and consultants assigned, and work organized (e.g., by Regional Center, 

functional specialization)? 

--How is communication initiated and supported between the Center and its clients? 

--How are decisions made regarding the provision of services (e.g., the nature and 

duration of services)? 

3. Describe your Center‘s relationship with your larger host organization. What are the 

advantages and disadvantages to the Center of being housed in the host organization? To 

what extent can you access expertise and other resources from your host organization to 

support the work of the Content Center? 

4. Describe your Center‘s relationship with the Regional Centers (e.g., the frequency, 

methods, and content of communication). Do you interact with some Regional Centers 

more than others? 

5. Describe your Center‘s relationship with other Content Centers (e.g., the frequency, 

methods, and content of communication). Do you interact with some Content Centers 

more than others? How do you work together with other Content Centers to define roles 

and responsibilities when there is overlap in your areas of expertise or organizational 

capabilities? 

6. Can you give us some specific examples of ways in which other Content Centers or 

Regional Centers have helped or supported your Center to achieve a goal or objective? 

7. Describe the major focus areas or initiatives your Center has addressed or engaged in 

during the past year. How did you determine these specific priorities? To what extent do 

you find it necessary to conduct formal needs assessments? [If applicable] How are these 

done? 

8. Which of your current initiatives do you consider to be the largest or most significant 

(e.g., in terms of importance, amount of resources allocated, etc.)? 

Prompts: 
CC Staff Site Visit Interview Protocol 

C-10

--How and why did it begin? 

--What products and services have you provided? 

--What are the goals and objectives; how much progress have you made to date? 

--What obstacles and challenges have you encountered? 

9. To what extent do you plan and initiate the delivery of products and services to the 

Regional Centers versus respond to specific Regional Center requests? 

10. To what extent have Regional Centers expressed needs that are different from your 

Center‘s priorities?

11. How and to what extent do Regional Centers vary in the nature (e.g., topics); frequency, 

and type (e.g., seminars, requests for materials, etc.) of assistance they request? 

12. Other than the Regional Centers, whom do you serve? What needs and priorities have 

these other clients expressed? To what extent do you work directly with states? [If 

applicable] What products and services do you typically provide to states, and which 

offices do you work with? To what extent do you work with the U.S. Department of 

Education? 

13. To what extent do needs and priorities overlap among different clients? Describe any 

strategies you may have developed for maximizing your Center‘s efficiency in 

responding to similar needs across multiple clients? 

14. To what extent do your clients request products or services that address special 

education-related needs or questions? 

15. What technical assistance methods do you rely on the most as you try to meet priority 

needs? Are there unique drawbacks or benefits associated with employing specific 

techniques? 

Prompts: 

--facilitating inter-organizational cooperation and collaboration (with whom?) 

--identifying and disseminating documents and other products 

--developing new products and plans 

--assisting clients to implement new plans or systems 

--hosting workshops, seminars, and other training events 

--identifying and using scientifically based evidence 

16. Where and how do you get the substantive knowledge that you use in your products and 

services? How do you decide what to use and what not to use in your work with clients? 

What quality standards or guidelines, if any, do you apply? 

17. Where does your Center turn for help when, for example: (1) you experience challenges 

or encounter obstacles that limit your effectiveness, or (2) you need to increase 
CC Staff Site Visit Interview Protocol 

C-11

institutional capacity (e.g., develop new expertise or sharpen your skills in a particular 

area)? [If applicable]: 

--Give specific examples that illustrate how others have helped your Center. 

--Indicate if you seek help on an as-needed basis, or whether there are regularly 

scheduled workshops and trainings? 

18. What does your Center do best? What kinds of needs are you best equipped to address? 

As you consider your Center‘s many objectives, which ones do you believe you have met 

most satisfactorily? 

19. In what ways, if any, has your Center changed in recent years (e.g., over the past one to 

two years)? Describe, for example, new developments in how you organize and operate, 

hire and deploy staff and consultants, use technology, allocate resources, or in how you 

otherwise plan, design, or deliver technical assistance. What motivated these changes and 

what kinds of effects or outcomes have resulted? 

20. In what ways, if any, has the context within which you work changed in recent years 

(e.g., over the past one to two years)? Describe, for example, shifts in needs and 

priorities, new developments in your field, or changes in rules or regulations that affect 

your Center and the work that you do. What impact has this had on your Center, and on 

the nature and effectiveness of the work that you do? 

21. With which Regional Centers do you work most frequently? How does your organization 

assess whether it is helping to build capacity within these Centers? 

22. What factors and conditions obstruct or hinder your work in significant ways? How have 

you tried to address them and with what level of success? 

23. The quality and effectiveness of technical assistance is dependent on many factors. Based 

on your experiences, what recommendations or suggestions would you make to the U.S. 

Department of Education, state policymakers, your clients, or others, regarding ways to 

improve or support the work of the current Content Centers? 
RCC Staff Site Visit Interview Protocol

C-12

National Evaluation of the 

Comprehensive Technical Assistance Centers 

Regional Center (RCC) Staff 

Site Visit Interview Protocol 

1. Tell me about your professional background and areas of expertise. What is your primary 

role within this organization? Describe the tasks you might perform or decisions you might 

make during a typical one- or two-week period? 

2. Give us an overview of your Center‘s internal structure and operations:

--How are staff and consultants assigned? 

--How is communication initiated and supported between the Center and its clients? 

--How are decisions made regarding the provision of services (e.g., the nature and 

duration of services)? 

--How do you organize your work (e.g., by state, functional specialization)? 

3. Describe your Center‘s relationship with your larger host organization. What are the 

advantages and disadvantages to the Center of being housed in the host organization? To 

what extent can you access expertise and other resources from your host organization to 

support the work of the Regional Center? 

4. Describe the major focus areas that your Center has addressed or engaged in during the past 

year. How did you determine these specific priorities? To what extent do you find it 

necessary to conduct formal needs assessments? [If applicable] How are these done? 

5. Which focus area do you consider to be the largest or most significant (in terms of 

importance, amount of resources allocated, etc.)? Tell us about it. 

Prompts: 

--How and why did it begin? 

--What products and services have you provided? 

--What are the goals and objectives, and how much progress have you made to date? 

--What obstacles and challenges have you encountered? 

6. In your work with states, what aspects of NCLB is your Center most often called upon to 

address within your region? Is there much variation among SEAs in this regard? Describe the 

specific kinds of help that are most often requested. 

7. To what extent have states expressed different or additional needs to those described above? 

8. Describe your relationships with each of the states in your region (e.g., the frequency, 

duration, and purpose of your interactions with them). To what extent do these interactions 

help to shape your Center‘s priorities and policies?
RCC Staff Site Visit Interview Protocol

C-13

9. With which offices or departments within states do you work most frequently? Does this vary 

among the states in your region? Is there a coordinating body within each state or among the 

states that facilitates communication with your Center? Please name the senior managers 

with whom you coordinate your work in each state. 

10. Other than states, whom do you serve? What priorities have emerged among these other 

clients? How often do you serve these nonstate clients? Are you able to be as responsive to 

them as you would like? 

11. Describe your relationships with each of the Content Centers (e.g., the frequency, duration, 

and purpose of your interactions with them). To what extent do these interactions help to 

shape your Center‘s priorities and policies?

12. Consider the full range of needs and priorities that this Center, the states, and other clients 

within the region have identified. To what extent do needs and priorities overlap among 

different clients? Describe any strategies you may have developed for maximizing your 

organization‘s efficiency in responding to similar needs across multiple clients. 

13. What technical assistance methods do you rely on the most as you try to meet priority needs? 

Are there particular drawbacks or benefits associated with employing specific techniques? 

Prompts: 

--facilitating inter-organizational cooperation and collaboration (with whom?) 

--identifying and disseminating documents and other products 

--developing new products and plans 

--assisting clients to implement new plans or systems 

--hosting workshops, seminars, and other training events 

--identifying and using scientifically based evidence 

 

14. Where and how do you get the substantive knowledge that you use in your products and 

services? How do you decide what to use and what not to use in your work with clients? 

What quality standards or guidelines, if any, do you apply? 

15. Where does your Center turn for help when, for example: (1) you experience challenges or 

encounter obstacles that limit your effectiveness, or (2) you need to increase institutional 

capacity (e.g., develop new expertise or sharpen your skills in a particular area)? [If 

applicable]: 

--Give specific examples that illustrate how the Content Centers and other 

organizations have helped your Center. 

--Indicate if you seek help on an as-needed basis, or whether there are regularly 

scheduled workshops and trainings. 

16. Describe your organization‘s relationship with the Content Centers and other Regional 

Centers (e.g., the frequency, methods, and content of communication between organizations). 
RCC Staff Site Visit Interview Protocol

C-14

Do you interact with some Content Centers or Regional Centers more than others? Can you 

give us some specific examples of ways in which other Centers have helped to increase your 

capacity or effectiveness, or ways your organization has supported the work of other 

Centers? 

17. What does your Center do best? What kinds of needs are you best equipped to address? As 

you consider your organization‘s many objectives, which ones do you believe you have met 

most satisfactorily? 

18. In what ways, if any, has your organization changed in recent years (e.g., over the past one 

to two years)? Describe, for example, new developments in how you organize and operate, 

hire and deploy staff and consultants, use technology, allocate resources, or in how you 

otherwise plan, design, or deliver technical assistance. What motivated these changes and 

what kinds of effects or outcomes have resulted? 

19. In what ways, if any, has all or parts of the region in which you work changed in the past two 

years? Describe, for example, any political, social, institutional, legal, economic, or 

demographic changes that have affected the education system in significant ways. What 

impact has this had on your Center, and on the nature and effectiveness of the work that you 

do? 

20. To what extent have the states in your region changed (e.g., structurally or operationally) as a 

direct result of your Center‘s efforts? [If applicable] Give specific examples of changes that 

have occurred. 

21. How does your Center assess whether it is helping to build capacity within the states in your 

region? If change has been difficult to achieve, why do you think that is the case? 

22. What factors and conditions obstruct or hinder your work in significant ways? How have you 

tried to address them and with what level of success? 

23. The quality and effectiveness of technical assistance is dependent on many factors. What 

recommendations or suggestions would you make to the U.S. Department of Education, state 

policymakers, your clients, or others, regarding ways to improve or support the work of the 

current Regional Comprehensive Centers? 
Interview Prompts 

C-15

National Evaluation of the Comprehensive Centers 

Interview Prompts 

On-site, the site visitor asked the Center to respond to the items shown in the following 

tables, with respect to Center objectives and operations in 2006-07. 

Table 1. Management Plan Objectives That Address Particular Topics 

Topics

In management 

plan

(Y/N)

a. Components of Effective Systems of Support --State, District, School

b. Data Use / Data-driven decision making

c. Formative Assessment

d. Reading

e. Adolescent Literacy

f. Mathematics

g. Dropout Prevention

h. High School Redesign/Reform

i. Transition to High School

j. Special Education – curriculum, instruction and professional 

development

k. Special Education – assessment

l. English Language Learners

m. Highly Qualified Teacher provisions of NCLB

n. Teacher preparation and induction

o. Teacher professional development 

p. Supplemental Educational Services (SES)

q. Response to Intervention (RtI)

r. Migrant education

s. Indian/Native American education

t. Data Management Compliance

u. Assessment Design

v. Parent Involvement

w. Other
Interview Prompts 

C-16

Table 2. Center Strategies for Planning Work 

Center strategies: (Y/N)

a. Needs assessment through survey of client staff

b. Needs assessment through analysis of state performance data or 

accountability plans

c. Needs assessment through analysis of state and federal policy 

context

d. Needs assessment through meetings or other communication with 

clients

e. Center included client input when writing the original grant proposal 

or cooperative agreement with ED

f. Center includes client input when writing annual management/work 

plan

g. Center communicates directly with the director of client 

organizations to plan the Center’s work

h. Center builds on relationships with clients that existed prior to the 

Center’s grant award.

i. Center refers to parameters set by NCLB or the Center’s

authorizing legislation when negotiating with clients to plan its work

Table 3. Center Strategies for Building Client Engagement and Buy-In

Center strategies: (Y/N)

a. Work on topics outside planned scope of work

b. Work on tasks outside planned scope of work

c. Work with primary service recipients outside planned scope of work

d. Sponsor large events to make contact with many clients

e. Form work groups within client organizations that bring together staff 

from multiple departments/divisions to discuss service needs and 

delivery

f. Form work groups across client organizations to discuss service 

needs and delivery

g. Offer service to support existing client programs/projects/policies

h. Hires former employees of client organizations (SEAs or RCCs)

i. Involves clients in making hiring decisions about Center staff

j. Has staff positions designated for serving as an in-house liaison to a 

specific client organization (SEA or RCC), maintaining office space 

within the client’s building
Interview Prompts 

C-17

Table 4. Center Strategies for Managing Work Requests Falling Outside Their 

Planned Scope 

Center strategies: (Y/N)

a. Center preempted requests for work outside its planned 

scope by having introductory meetings with clients in 

order to define the parameters of work it would do

b. Center refers to parameters set by NCLB or the Center’s

authorizing legislation when negotiating with clients

c. Center agrees to work outside its planned scope in order 

to build a working relationship with clients

d. Center has a staff member and process for ensuring that 

Center work is within its work plan

e. Center submits a revised work plan for ED approval before 

responding to requests outside its plan

f. Center has declined to provide a service because it was 

outside of its planned scope

Table 5. Sources of Content Expertise 

Source (Y/N)

a. Center staff themselves (lead organization and/or subcontractors)

b. Other staff in the lead organization, not formally employed by the 

Center 

c. Other staff in subcontractor organization(s) 

d. Content Center(s) 

e. Universities

f. State education agencies

g. What Works Clearinghouse

h. Professional associations (e.g., AERA, CCSSO)

i. Consulting firms or private contractors

j. Other regional Center(s)

k. Other federally funded technical assistance providers (not part of the 

Center)

l. Regional labs
Interview Prompts 

C-18

Table 6. Sources for Vetting for Products and Services Prior to Delivery 

Source (Y/N)

a. Internal quality control by Center staff (lead organization and/or 

subcontractors)

b. Other staff in the lead organization, not formally employed by the 

Center 

c. Other staff in subcontractor organization(s) 

d. U.S. Department of Education

e. Outside experts retained to review drafts

f. Content Center(s) 

Table 6. Center Strategies for Managing Work Requests Falling Outside Their 

Planned Scope 

Center strategies:

From year 1 

site visit

(Y/N)

a. Center refers to parameters set by NCLB or the Center’s authorizing 

legislation when negotiating with clients

b. Center agrees to work outside its planned scope in order to build a 

working relationship with clients

c. Center has a staff member and process for ensuring that Center work 

is within its work plan

d. Works with primary service recipients outside planned scope of work

e. Center has declined to provide a service because it was outside of its 

planned scope

 
Appendix D 

Survey of Senior State Managers 
D-1

Survey of Senior State Managers 

This appendix section presents the survey that was administered to Senior State Managers 

serving as RCC‘s main points of contacts (see page D-2). The survey was developed to obtain 

information from state managers about state-level needs and priorities for technical assistance, 

state perspectives of Center technical assistance, and comparative judgment of Center assistance 

in relation to assistance available through other sources. 
State Manager Survey 

D-2

Comprehensive Center Evaluation 

Senior State Manager Survey 

This survey is designed to gather data on technical assistance received by your state from 

a variety of external sources, including federally funded technical assistance centers, colleges 

and universities, professional associations, colleagues in other states, consulting firms, and 

private contractors. 

 In particular, the survey asks about assistance your state has received from the federally 

supported network of Comprehensive Technical Assistance Centers, which includes in your 

region the following Regional Comprehensive Center: 

This Regional Center often collaborates with one or more Content Centers, which may also work 

directly with states: 

 The Assessment and Accountability Comprehensive Center (AACC), which is 

directed by Dr. Stanley Rabinowitz at WestEd in San Francisco, California. 

 The National High School Center (NHSC), which is directed by Dr. Joe Harris 

at the American Institutes of Research in Washington, DC. 

 The Center on Innovation and Improvement (CII), which is directed by Dr. 

Sam Redding at the Academic Development Institute, in Lincoln, Illinois. 

 The Center on Instruction (COI), which is directed by Ms. Angela Penfold at 

RMC Research Corporation in Portsmouth, NH. 

 The National Comprehensive Center for Teacher Quality (NCCTQ), which is 

directed by Dr. Sabrina Laine at Learning Point Associates in Naperville, Illinois. 

 Questions about the Comprehensive Center network in this survey refer to all of the 

Comprehensive Centers that have served your state, including the Regional Center and any 

Content Centers with which your state has worked. 

 The term ―technical assistance,‖ as used in this survey, includes print and electronic 

information and other resources, consultation, advice, facilitation, and training. 
State Manager Survey 

D-3

 Per the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to 

this data collection will be used only for statistical purposes. The reports prepared for this study 

will summarize findings across the sample and will not associate responses with a specific 

organization or individual. We will not provide information that identifies you or your 

organization to anyone outside the study team, except as required by law. 

Please return this survey using the self-addressed, stamped envelope enclosed 

to [name] of Policy Studies Associates (PSA) at: 

[address] 

_____@_________

Thank you for your feedback! 

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless it displays a valid OMB control number. The valid OMB control number for this survey is 1850-

0823. The time required to complete this survey is estimated to average 20 minutes per response, including the time 

to review instructions, search existing data sources, gather the data needed, and respond to the survey questions. 

Send comments regarding this burden estimate or any other aspect of this collection of information, including 

suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651.
State Manager Survey 

D-4

Priorities for Technical Assistance 

1. When your state requests technical assistance from outside sources, what are its 

priorities? To what extent is each of the following state responsibilities related to NCLB 

implementation a priority for the technical assistance the state requests? (Select one 

response in each row.)

Note: Please consider the technical assistance that your state has sought from all 

sources, including federally funded technical assistance centers, colleges and 

universities, professional associations, colleagues in other states, consulting 

firms, and private contractors. 

State Responsibilities

Major 

priority

Moderate 

priority

Minor 

priority

Not at all a 

priority

a. Formulating or refining state policies to 

respond to NCLB requirements 4 3 2 1

b. Building or managing a statewide system 

of support for districts and schools 

identified for improvement under NCLB

4 3 2 1

c. Training or managing the state-level staff 

or school support teams who provide 

support to districts and schools identified 

for improvement under NCLB

4 3 2 1

d. Designing or implementing state 

assessment and accountability systems 4 3 2 1

e. Aligning state accountability systems with 

NCLB accountability systems 4 3 2 1

f. Supporting use of assessment data by 

schools and districts 4 3 2 1

g. Disseminating information on 

scientifically based research to districts 

and schools

4 3 2 1

h. Identifying and/or developing programs 

or models that address district and/or 

school needs

4 3 2 1

i. Providing training and other professional 

development to local educators in 

academic subjects (reading language arts, 

mathematics, science)

4 3 2 1

j. Monitoring compliance with NCLB 

requirements in districts and schools 4 3 2 1

k. Communicating with the public about 

NCLB requirements or report cards 4 3 2 1

l. Other (Specify: ________________) 4 3 2 1
State Manager Survey 

D-5

Sources of Technical Assistance 

2. During the 2006-07 school year (beginning July 2006 and ending June 2007), to what 

extent has your state relied on each of the following sources of technical assistance? 

(Select one response in each row.)

Sources of Technical Assistance

One of the 

state’s 

most 

important 

resources

To a great 

extent, but 

not one of 

the state’s 

most 

important 

resources

To a 

moderate 

extent Minimally No contact

a. U.S. Department of Education 

(Specify office: 

_________________________)

4 3 2 1 0

b. Professional associations (e.g., 

CCSSO, ASCD) 4 3 2 1 0

c. Colleges and universities 4 3 2 1 0

d. Consulting firms or private 

contractors

4 3 2 1 0

e. Your counterparts in other SEAs 4 3 2 1 0

f. Comprehensive Center network 4 3 2 1 0

g. Regional Educational Laboratory 4 3 2 1 0

h. Other federally funded technical 

assistance providers (Specify: 

________________________)

4 3 2 1 0

i. Other (Specify:

______________) 4 3 2 1 0
State Manager Survey 

D-6

3. For what purpose(s) does your state seek assistance from each of the following sources? 

(Select all that apply for each source.)

a. U.S. Department of Education (Specify office(s): _________________________) 

 To gather information or to keep current with new ideas 

 To plan the initial steps in solving a problem 

 To complete tasks that the state could do itself if it had more staff or resources 

 To complete tasks for which the state lacks expertise 

 To develop the skills of SEA or intermediate education agency staff 

 To work directly with districts and schools 

 Not applicable; our state doesn’t seek assistance from this source

b. Professional associations (e.g., CCSSO, ASCD) 

 To gather information or to keep current with new ideas 

 To plan the initial steps in solving a problem 

 To complete tasks that the state could do itself if it had more staff or resources 

 To complete tasks for which the state lacks expertise 

 To develop the skills of SEA or intermediate education agency staff 

 To work directly with districts and schools 

 Not applicable; our state doesn’t seek assistance from this source

c. Colleges and universities 

 To gather information or to keep current with new ideas 

 To plan the initial steps in solving a problem 

 To complete tasks that the state could do itself if it had more staff or resources 

 To complete tasks for which the state lacks expertise 

 To develop the skills of SEA or intermediate education agency staff 

 To work directly with districts and schools 

 Not applicable; our state doesn’t seek assistance from this source

d. Consulting firms or private contractors 

 To gather information or to keep current with new ideas 

 To plan the initial steps in solving a problem 

 To complete tasks that the state could do itself if it had more staff or resources 

 To complete tasks for which the state lacks expertise 

 To develop the skills of SEA or intermediate education agency staff 

 To work directly with districts and schools 

 Not applicable; our state doesn’t seek assistance from this source
State Manager Survey 

D-7

e. Counterparts in other SEAs 

 To gather information or to keep current with new ideas 

 To plan the initial steps in solving a problem 

 To complete tasks that the state could do itself if it had more staff or resources 

 To complete tasks for which the state lacks expertise 

 To develop the skills of SEA or intermediate education agency staff 

 To work directly with districts and schools 

 Not applicable; our state doesn’t seek assistance from this source

f. Comprehensive Center network 

 To gather information or to keep current with new ideas 

 To plan the initial steps in solving a problem 

 To complete tasks that the state could do itself if it had more staff or resources 

 To complete tasks for which the state lacks expertise 

 To develop the skills of SEA or intermediate education agency staff 

 To work directly with districts and schools 

 Not applicable; our state doesn’t seek assistance from this source

g. Regional Educational Laboratory 

 To gather information or to keep current with new ideas 

 To plan the initial steps in solving a problem 

 To complete tasks that the state could do itself if it had more staff or resources 

 To complete tasks for which the state lacks expertise 

 To develop the skills of SEA or intermediate education agency staff 

 To work directly with districts and schools 

 Not applicable; our state doesn’t seek assistance from this source

h. Other federally funded technical assistance providers (Specify: 

____________________) 

 To gather information or to keep current with new ideas 

 To plan the initial steps in solving a problem 

 To complete tasks that the state could do itself if it had more staff or resources 

 To complete tasks for which the state lacks expertise 

 To develop the skills of SEA or intermediate education agency staff 

 To work directly with districts and schools 

 Not applicable; our state doesn’t seek assistance from this source

i. Other (Specify: ____________) 

 To gather information or to keep current with new ideas 

 To plan the initial steps in solving a problem 

 To complete tasks that the state could do itself if it had more staff or resources 

 To complete tasks for which the state lacks expertise 

 To develop the skills of SEA or intermediate education agency staff 

 To work directly with districts and schools 

 Not applicable; our state doesn’t seek assistance from this source
State Manager Survey 

D-8

4. Are there other purposes for which your state seeks assistance from outside sources, not 

mentioned in Question 3? To which sources do you turn for these purposes? Please 

describe briefly below. 

 _______________________________________________________________________ 

 _______________________________________________________________________ 

 _______________________________________________________________________ 

 _______________________________________________________________________ 

 _______________________________________________________________________ 

 _______________________________________________________________________ 

 _______________________________________________________________________
State Manager Survey 

D-9

Technical Assistance Received from the Comprehensive Center Network 

5. During the 2006-07 school year (beginning July 2006 and ending June 2007), with 

which responsibilities related to NCLB implementation did your state receive assistance 

from the Comprehensive Center network (your Regional Center and the Content Centers 

with whom your state has worked)? (Select one response in each row.) 

Our state received assistance 

from the Comprehensive 

Center network with the 

following responsibilities:

Major 

assistance

Moderate 

assistance

Minimal 

assistance

No 

assistance 

at all

NA, 

technical 

assistance 

not sought

by the 

SEA

NA, this 

task is not 

relevant to 

my unit’s 

work

a. Formulating or refining 

state policies to respond to 

NCLB requirements

4 3 2 1 95 98

b. Building or managing a 

statewide system of 

support for districts and 

schools identified for 

improvement under NCLB

4 3 2 1 95 98

c. Training or managing the 

state-level staff or school 

support teams who provide 

support to districts and 

schools identified for 

improvement under NCLB

4 3 2 1 95 98

d. Designing or 

implementing state 

assessment and 

accountability systems

4 3 2 1 95 98

e. Aligning state 

accountability systems 

with NCLB accountability 

systems

4 3 2 1 95 98

f. Supporting use of 

assessment data by schools 

and districts

4 3 2 1 95 98

g. Disseminating information 

on scientifically based 

research to districts and 

schools

4 3 2 1 95 98

h. Identifying and/or 

developing programs or 

models that address 

district and/or school 

needs

4 3 2 1 95 98
State Manager Survey 

D-10

Our state received assistance 

from the Comprehensive 

Center network with the 

following responsibilities:

Major 

assistance

Moderate 

assistance

Minimal 

assistance

No 

assistance 

at all

NA, 

technical 

assistance 

not sought

by the 

SEA

NA, this 

task is not 

relevant to 

my unit’s 

work

i. Providing training and 

other professional 

development to local 

educators in academic 

subjects (reading language 

arts, mathematics, science)

4 3 2 1 95 98

j. Conducting, designing, or 

overseeing parent outreach

4 3 2 1 95 98

k. Monitoring compliance 

with NCLB requirements 

in districts and schools

4 3 2 1 95 98

l. Communicating with the 

public about NCLB 

requirements or report 

cards 

4 3 2 1 95 98

m. Other (Specify: 

________________)

4 3 2 1 95 98
State Manager Survey 

D-11

6. During the 2006-07 school year (beginning July 2006 and ending June 2007), to what 

extent did your state receive the following types of assistance from the Comprehensive 

Center network (your Regional Center and the Content Centers with whom your state 

has worked)? (Circle one response in each row.)

Our state received the following 

types of assistance from the 

Comprehensive Center network:

Major 

assistance

Moderate 

assistance

Minimal 

assistance

No 

assistance 

at all

NA, 

technical 

assistance 

not sought

by the SEA

a. Reviewing state plans and 

policies

4 3 2 1 95

b. Designing, delivering, or 

convening professional 

development and conferences for 

local educators

4 3 2 1 95

c. Synthesizing and disseminating 

research findings

4 3 2 1 95

d. Collecting and disseminating 

information about policies and 

practices in other states 

4 3 2 1 95

e. Analyzing data or conducting 

needs assessments

4 3 2 1 95

f. Responding to federal planning 

and reporting requirements

4 3 2 1 95

g. Developing tools for monitoring

programs

4 3 2 1 95

h. Facilitating work groups or 

committees

4 3 2 1 95

i. Completing routine tasks more 

efficiently

4 3 2 1 95

j. Other (Specify:_____________) 4 3 2 1 95
State Manager Survey 

D-12

Relevance and Usefulness 

7. Please consider all of the technical assistance that your state has received from the 

Comprehensive Center network during the 2006-07 school year (beginning July 2006 

and ending June 2007). Considering just this set of products and services, to what degree 

was this set of activities and resources relevant to your state, in each of the following 

respects? (Circle one response in each row.) 

Comprehensive Center 

technical assistance:

To a 

very 

high 

degree

To a high 

degree

To a 

moderate 

degree

To a low 

degree

To a very 

low degree

Not able 

to judge

a. Addressed a need or problem 

that the state faces 5 4 3 2 1 95

b. Addressed an important state 

priority 5 4 3 2 1 95

c. Addressed a challenge that the 

state faces related to the 

implementation of NCLB

5 4 3 2 1 95

d. Responded flexibly to our 

state‘s changing needs 5 4 3 2 1 95

e. Provided information, advice, 

and/or resources that could be

applied to the state‘s work 

5 4 3 2 1 95

f. Addressed the particular 

context in which our state 

operates

5 4 3 2 1 95

g. Addressed the state‘s specific 

challenges (e.g., policy 

environment, leadership 

capacity, budget pressures, 

local politics) 

5 4 3 2 1 95

h. Provided information, advice, 

and/or resources that could be 

used to guide decisions about 

policies, programs, and 

practices

5 4 3 2 1 95

i. Highlighted the implications of 

research findings (or 

information about best practice) 

for policies, programs, or 

practices

5 4 3 2 1 95
State Manager Survey 

D-13

8. Please consider all of the technical assistance that your state has received from the 

Comprehensive Center network during the 2006-07 school year (beginning July 2006 

and ending June 2007). Considering just this set of products and services, to what degree 

was this set of activities and resources useful to your state, in each of the following 

respects? (Circle one response in each row.) 

Comprehensive Center

technical assistance:

To a 

very 

high 

degree

To a high 

degree

To a 

moderate 

degree

To a low 

degree

To a very 

low 

degree

Not able 

to judge

a. Provided state staff with 

resources that were easy to 

understand and easy to use 

5 4 3 2 1 95

b. Employed appropriate formats

(e.g., work groups, conferences, 

individual consultation, written 

products) 

5 4 3 2 1 95

c. Provided adequate opportunity 

to learn from colleagues in 

other states

5 4 3 2 1 95

d. Included adequate follow-up to 

support the use of new 

information and resources 

5 4 3 2 1 95

e. Was timely 5 4 3 2 1 95

f. Helped the state to solve a 

problem 5 4 3 2 1 95

g. Helped the state to maintain or 

change a policy or practice 5 4 3 2 1 95

h. Helped the state take the next 

step in a longer-term 

improvement effort

5 4 3 2 1 95

i. Provided state staff with 

information or resources that 

they will use again 

5 4 3 2 1 95

j. Helped state staff to develop 

skills that they will be able to 

exercise again 

5 4 3 2 1 95
State Manager Survey 

D-14

9. How fully has the technical assistance your state has received from the Comprehensive 

Center network during the 2006-07 school year served the state‘s purposes for technical 

assistance? (Circle one.) 

a. It served the state‘s purposes completely (Skip to question 11)

b. It was a good start 

c. It was a start, but some important priorities were not addressed 

d. For the most part, it did not serve the state‘s purposes

10. If the Comprehensive Center network assistance during the 2006-07 school year has been 

less helpful than it might be, what were the reasons? (Circle all that apply.) 

a. The state‘s most important priorities for assistance fall outside the Comprehensive 

Center‘s scope of work

b. The Comprehensive Center does not have the expertise the state needs 

c. Comprehensive Center staff are not able to spend as much time working with the state 

as we would like 

d. The process of negotiating a work scope and organizing projects takes too long 

e. The state has been unable to develop a productive working relationship with the 

Center 

f. The state secures most of the technical assistance it needs from other sources 

g. The state would prefer to locate and contract directly with the experts or consultants 

from whom it needs assistance, rather than working through the Comprehensive 

Center network 

h. A policy or priority shift at the state level caused the Center‘s assistance to be less 

helpful than it might 

i. Other (Specify _____________________________________________) 

11. How could the technical assistance your state receives from the Comprehensive Center 

network be improved or made more useful to your state? 

 ________________________________________________________________________ 

 ________________________________________________________________________ 

 ________________________________________________________________________ 

 ________________________________________________________________________ 
State Manager Survey 

D-15

Comprehensive Center Assistance Compared with Assistance from Other 

Sources 

12. Compared with the technical assistance your state has received from other sources

(see list of sources in Question 2), how would you rate the usefulness of the technical 

assistance your state has received from the Comprehensive Center network for each of 

the following purposes? (Circle one response in each row.) 

Much 

more 

useful 

than 

assistance 

from other 

sources

Somewhat 

more 

useful

About the 

same

Somewhat 

less useful

Much less 

useful 

than 

assistance 

from other 

sources

Not able to 

judge

NA, the 

state has 

not sought 

assistance 

for this 

purpose

a. Formulating or 

refining state policies 

to respond to NCLB 

requirements

5 4 3 2 1 95 98

b. Building or managing 

a statewide system of 

support for districts 

and schools identified 

for improvement 

under NCLB

5 4 3 2 1 95 98

c. Training or managing 

the state-level staff or 

school support teams 

who provide support 

to districts and 

schools identified for 

improvement under 

NCLB

5 4 3 2 1 95 98

d. Designing or 

implementing state 

assessment and 

accountability 

systems

5 4 3 2 1 95 98

e. Aligning state 

accountability 

systems with NCLB 

accountability 

systems

5 4 3 2 1 95 98

f. Supporting use of 

assessment data by 

schools and districts

5 4 3 2 1 95 98
State Manager Survey 

D-16

Much 

more 

useful 

than 

assistance 

from other 

sources

Somewhat 

more 

useful

About the 

same

Somewhat 

less useful

Much less 

useful 

than 

assistance 

from other 

sources

Not able to 

judge

NA, the 

state has 

not sought 

assistance 

for this 

purpose

g. Disseminating 

information on 

scientifically based 

research to districts 

and schools

5 4 3 2 1 95 98

h. Identifying and/or 

developing programs 

or models that address 

district and/or school 

needs

5 4 3 2 1 95 98

i. Providing training and 

other professional 

development to local 

educators in academic 

subjects (reading 

language arts, 

mathematics, science)

5 4 3 2 1 95 98

j. Conducting, 

designing, or 

overseeing parent

outreach

5 4 3 2 1 95 98

k. Monitoring 

compliance with 

NCLB requirements 

in districts and 

schools

5 4 3 2 1 95 98

l. Communicating with 

the public about 

NCLB requirements 

or report cards 

5 4 3 2 1 95 98

m. Other (Specify: 

____________)

5 4 3 2 1 95 98

 
State Manager Survey 

D-17

State Capacity to Carry Out Responsibilities Related to NCLB 

13. To what extent has assistance from the Comprehensive Center network (your Regional 

Center and the Content Centers with whom your state has worked) expanded your state‘s 

capacity to carry out state responsibilities related to NCLB? (Circle one response in 

each row.) 

State Responsibilities

To a very 

great 

extent

To a 

great 

extent

To a 

moderate 

extent

To a 

small 

extent

To a 

very 

small 

extent

Too soon to 

tell 

NA, state 

has not 

sought 

assistance 

for this 

purpose

a. Formulating or refining state 

policies to respond to NCLB 

requirements

5 4 3 2 1 95 98

b. Building or managing a 

statewide system of support 

for districts and schools 

identified for improvement 

under NCLB

5 4 3 2 1 95 98

c. Training or managing the 

state-level staff or school 

support teams who provide 

support to districts and 

schools identified for 

improvement under NCLB

5 4 3 2 1 95 98

d. Designing or implementing 

state assessment and 

accountability systems

5 4 3 2 1 95 98

e. Aligning state accountability 

systems with NCLB 

accountability systems

5 4 3 2 1 95 98

f. Supporting use of 

assessment data by schools 

and districts

5 4 3 2 1 95 98

g. Disseminating information 

on scientifically based 

research to districts and 

schools

5 4 3 2 1 95 98

h. Identifying and/or 

developing programs or 

models that address district 

and/or school needs

5 4 3 2 1 95 98

i. Providing training and other 

professional development to 

local educators in academic 

subjects (reading language 

arts, mathematics, science)

5 4 3 2 1 95 98
State Manager Survey 

D-18

State Responsibilities

To a very 

great 

extent

To a 

great 

extent

To a 

moderate 

extent

To a 

small 

extent

To a 

very 

small 

extent

Too soon to 

tell 

NA, state 

has not 

sought 

assistance 

for this 

purpose

j. Conducting, designing, or 

overseeing parent outreach 5 4 3 2 1 95 98

k. Monitoring compliance with 

NCLB requirements in 

districts and schools

5 4 3 2 1 95 98

l. Communicating with the 

public about NCLB 

requirements or report cards 

5 4 3 2 1 95 98

m. Other (Specify: 

____________) 5 4 3 2 1 95 98

Your Job Responsibilities 

14. Please describe the functions of the office/division/department that you direct. (Select all 

that apply.) 

a. School Improvement 

b. Curriculum and Instruction 

c. Assessment and Accountability 

d. Special education 

e. Federal programs (Specify: ____________________) 

f. Other (Specify: _____________________________) 

15. Which of the following statements describe your job responsibilities related to managing 

technical assistance resources from outside your organization? (Circle all that apply.)

a. Seeking out technical assistance providers for my organization 

b. Identifying needs and priorities that will be addressed by technical assistance 

c. Negotiating a scope of work with technical assistance providers 

d. Managing others within my office/division/unit in their use of technical assistance 

services 

e. Serving as point of contact or manager for specific technical assistance projects 

f. Participating in specific technical assistance projects 

g. Other (Specify: _________________) 

h. None, I am not responsible for managing technical assistance resources 
State Manager Survey 

D-19

16. Did you collaborate with any of your colleagues on your response to this survey? (Circle 

one.)

(We will refrain from following up with your colleagues if you collaborated on this 

response.) 

a. Yes, I collaborated on this response with colleagues 

b. No, I completed this survey without consulting colleagues 

Thank you! 

Please return this survey using the self-addressed, stamped envelope enclosed 

to [name] of Policy Studies Associates at: 

[address] 

_____@_________
Appendix E 

Ratings of Quality, Relevance, and Usefulness 
Ratings of Quality, Relevance, and Usefulness 

This appendix section presents the expert panel review rating materials and the 

participant surveys that were collected to determine ratings of technical quality, relevance, and 

usefulness. 

Expert Panel Reviewer Scoring Booklet Used for Quality Ratings 

This appendix section presents the Scoring Booklet that was used by the expert panel 

reviewers. The goal in developing the scoring rubric was to provide uniform, objective criteria 

for rating technical quality. Because the evaluation of each project was based on the professional 

judgment of three panelists, differences among raters were inevitable. Providing a welldeveloped scoring rubric and training the panelists on the use of the scoring criteria was intended 

to maximize interrater agreement and reduce bias. 

For the purposes of this evaluation, technical quality of Comprehensive Center projects 

was demonstrated through the comprehensive and balanced use of scientifically based research 

and the applicable nonresearch knowledge base. The best available research or knowledge base 

was expected to inform the delivery of technical assistance. Where rigorous and consistent 

evidence was lacking, there had to be an acknowledgment in the technical assistance provided of 

the lack of conclusive evidence and advice, and recommendations had to be appropriately 

tempered. Professional wisdom was expected to be integrated with the best available empirical 

evidence in planning for and delivering products and services. To be rated high quality, the 

materials were expected to be accurate, complete, and clear and should support use and 

implementation of the content. 

Relevance and Usefulness Ratings from RCC and State-Level Participant Surveys 

This appendix section presents the two forms of the Participant Surveys (starting on page 

E-11 of this appendix) that were administered to RCC and state-level staff to obtain client views 

of technical assistance from the Comprehensive Centers, particularly in the areas of the relevance 

and usefulness of Center products. 

The evaluation team developed the two parallel survey forms included in this appendix. 

One version of the survey was written using text appropriate for state-level staff who participated 

in any Center‘s project. The second, similar version of the survey was written with wording 

appropriate for RCC staff who participated in a CC project. Both surveys asked questions in each 

of the following areas: 

 Project Participation; 

 Relevance and Usefulness; 

 Priorities for Technical Assistance; 

 Capacity to Carry out Responsibilities Related to NCLB, and 

 Job Responsibilities.
E-2

National Evaluation of the Comprehensive Centers 

Scoring Booklet for Expert Reviewers 

Fall 2007 

Policy Studies Associates
E-3

Dimension 1: Demonstrated Use of Appropriate Knowledge Base 

Indicators Definitions for Indicators Evidence and Applicability to Project

1A. The knowledge base 

used is appropriately 

comprehensive, given the 

project’s purpose.

 The most important research and 

knowledge are used when 

applicable and appropriate for 

the project.

 Appropriate legal and regulatory 

guidelines are used when 

applicable for the project.

1B. The knowledge base 

used is accurately 

described.

 The applicable knowledge base 

is accurately described in the 

project’s materials.

 Research findings, regulatory 

guidelines, or professional 

wisdom used in the project 

materials are accurately 

described. 

1C. The knowledge base 

used for the project is 

relevant.

 The research/knowledge base 

used is relevant to the topic.

 Legal and regulatory guidelines 

that are presented are relevant to 

the topic.
Scoring Booklet for Quality Ratings 

E-4

Dimension 1: Demonstrated Use of Appropriate Knowledge Base (Continued) 

Indicators Definitions for Indicators Evidence and Applicability to Project

1D. The knowledge base 

represented in the project 

is balanced, when 

appropriate.

 More than one point of view is 

presented in the knowledge base 

used, if it is applicable and 

important.

 Where the level of evidence 

warrants, the knowledge base 

used reflects the variety of valid 

perspectives appropriate for 

consideration in the field.

1E. Prominence in 

findings and 

recommendations is given 

to current and best 

available scientifically 

based research, 

knowledge base, and 

professional wisdom 

 Where available, prominence is 

given to research that is 

scientifically based.

 Research and knowledge base 

used and reflected in the project 

represent the latest generation of 

findings.

 Research studies that used 

appropriate designs, 

methodologies, and measures 

are given prominence over those 

with weaker designs, when 

available.

1F. Limitations in the 

knowledge base are 

acknowledged in the 

project.

 Materials clearly note the 

strength (or limitations) of the 

knowledge or research base.

 Findings or recommendations 

based on best practice only 

(rather than more empirical 

evidence) are accurately 

identified as such in the 

materials.
E-5

Criteria for Scoring—Demonstrated Use of Appropriate Knowledge Base

Directions: Circle the number that best describes the demonstrated use of the appropriate knowledge base in the project, considering the project‘s performance on the various indicators identified.

Very High Quality High Quality Moderate Quality Low Quality Very Low Quality

5 4 3 2 1

1A The research/knowledge base 

used or referenced in project 

materials includes an appropriate 

set of important sources in this 

field, pertinent to the project’s 

purpose.

There is a research/knowledge base used or 

referenced in project materials, pertinent to the 

project’s purpose, but some important sources are not 

included.

There is no research/knowledge base 

used or referenced in project 

materials, or the research/knowledge 

base used or referenced in project 

materials includes none of the 

important source materials in the field 

pertinent to the project’s purpose.

1B All of the research findings, legal 

and regulatory guidelines, or 

professional wisdom used in the 

project’s materials is accurately 

described. 

Some inaccuracies exist in the description of the 

research/knowledge base in the project’s materials.

Inaccuracies exist throughout the 

project in describing the 

research/knowledge base.

1C All of the information presented 

about the research/ knowledge 

base in project materials is 

relevant to the topic.

Some of the research and/or knowledge base 

reflected in the materials is relevant; however, some 

of the material presented is not relevant.

Most of the information presented for 

the research/knowledge base is not 

relevant to the topic.

1D The research/

knowledge base used or 

referenced contains a balanced 

variety of valid perspectives in 

this field.

Some balance is provided in presenting the variety of 

valid perspectives in the field; however, some

imbalance is evident.

No balance is provided in presenting 

the various valid perspectives in the 

field.

1E The most current and rigorous 

research and knowledge 

available is given the most 

prominence in project materials.

The most current and rigorous research/knowledge 

base available is only partially reflected and given 

prominence in the project.

The most current and rigorous 

research/knowledge base available is 

minimally reflected and given little or 

no prominence in the project.

1F Limitations in the available 

research/knowledge base are 

clearly described and 

acknowledged in project 

materials.

Limitations in the available research/knowledge base 

are only partially described and acknowledged in the 

project materials, although they exist.

No limitations in the available 

research/knowledge base are 

acknowledged in the project 

materials, although they exist.
E-6

Dimension 2: Fidelity of Application of the Knowledge Base to the Products and Services Provided 

Indicators Definitions for Indicators Evidence and Applicability to Project

2A. The application of the 

research/ knowledge 

base is clear and 

accurate.

 Accurate application of the core 

findings from the available 

knowledge base is evident in the 

project’s products, assistance, or

advice.

 The project’s products, 

assistance, and advice clearly

interpret and apply the research 

and/or knowledge base.

2B. There is consistency 

between the strength of 

the research/knowledge 

base and its proposed 

application. 

 The certainty and strength of 

recommendations in the project’s 

products, assistance, and advice 

are consistent with and 

appropriate for the level of rigor 

and certainty in the available 

research/knowledge base.

2C. Appropriate 

emphasis is given in 

application of the most 

rigorous and consistent

research and knowledge 

base.

 Prominence is given to products, 

assistance, and advice derived 

from the most rigorous research 

and knowledge base.

 Prominence is given to products, 

services, and advice derived 

from the most consistent

research and knowledge base.

2D. Application of 

nonempirical research 

and professional wisdom 

only are appropriately 

tempered. 

 Products, assistance, and 

advice based on a weak 

research base, limited legal or 

regulatory guidance, or primarily 

nonempirical professional 

wisdom are appropriately 

tempered.
E-7

Criteria for Scoring—Fidelity of Application of the Knowledge Base to the Products and Services Provided

Directions: Circle the number that best describes the fidelity of application of the knowledge base to the products and services provided, considering the project‘s performance on the various 

indicators identified.

Very High Quality High Quality Moderate Quality Low Quality Very Low Quality

5 4 3 2 1

2A The products, assistance and 

advice provided through this 

project are consistently accurate 

and clear in interpreting and 

applying the available research 

and knowledge base used in the 

project.

The products, assistance, and advice provided 

through this project are accurate and clear in 

interpreting and applying the available research 

and knowledge base in the project used in 

some respects but not others.

The products, assistance and 

advice provided through this 

project are consistently 

inaccurate and unclear in 

interpreting and applying the 

available research and/or 

knowledge base used in the 

project.

2B The products, advice, and 

assistance provided through this 

project are fully consistent with 

the available research and 

knowledge base. 

The products, advice, and assistance provided 

through this project are partially consistent with 

the available research and knowledge.

The products, advice, and 

assistance provided through this 

project are not consistent and for 

the most part conflicts with the 

available research and 

knowledge base. 

2C The products, assistance, and 

advice provided through this 

project completely and 

consistently emphasize the 

application of findings derived 

from rigorous and consistent 

research and knowledge over 

the application of findings from 

less rigorous or consistent 

research or knowledge.

Some of the products, assistance, and advice 

provided through this project emphasize the 

application of findings derived from rigorous and 

consistent research and knowledge over the 

application of findings from less rigorous or less 

consistent research or knowledge in some 

instances, but some products, assistance, and 

advice provided emphasize less rigorous or 

consistent research and knowledge.

The products, assistance and 

advice provided through this 

project consistently emphasize 

the application of findings 

derived from less rigorous and 

less consistent research and 

knowledge over the application 

of findings from more rigorous or 

consistent research or 

knowledge.

2D Products, assistance, and 

advice provided through this 

project that are based on weak 

research or knowledge base are 

consistently acknowledged and 

appropriately tempered.

Products, assistance, and advice provided 

through this project that are based on weak 

research or knowledge base are partially 

acknowledged and are tempered only in part.

Products, assistance, and advice 

provided through this project that 

are based on weak research or 

knowledge base are never 

acknowledged as such and are 

rarely if ever tempered.
E-8

Dimension 3: Clear and Effective Delivery 

Indicators Definitions for Indicators Evidence and Applicability to Project

3A. Information 

communicated in 

the project 

products and 

services meets the 

project’s purpose.

 Information contained in the project 

materials is clear in meeting the 

purpose of the project.

 Information contained in the project 

materials is complete and applicable in 

meeting the purpose of the project.

3B. The project 

contains 

meaningful 

learning 

experiences, 

appropriate for the 

intended audience. 

 Where appropriate, products and 

services are designed to engage 

participants in effective learning 

experiences.

 In the opinion of the reviewer, the 

audience should be interested in the 

information and recommendations 

because of the way in which they are 

delivered. 

3C. The products 

and services in the 

project are 

appropriate for the 

intended audience.

 The products and services as presented 

are relevant and well-suited for the 

intended audience.

 The products and services as presented 

appear to be useful for the intended 

audience.

3D. Ideas are 

effectively

communicated.

 Clear and accessible language is used

in project materials.

 Project materials are well-written.

 Information conveyed will be 

understood by intended audience.
E-9

Criteria for Scoring—Clear and Effective Delivery 

Directions: Circle the number that best describes clear and effective delivery considering the project‘s performance on the various indicators identified.

Very High Quality High Quality Moderate Quality Low Quality Very Low Quality

5 4 3 2 1

3A Project materials and the 

information presented are

complete and applicable for 

meeting the project’s 

purpose. 

Project materials and 

information are partially 

incomplete or not applicable for 

meeting the project’s purpose.

Project materials and 

information is incomplete or 

not at all applicable to the 

purpose of the project.

3B The project materials will 

engage participants or users 

in meaningful learning 

experiences or will be of 

interest to them because of 

the way in which materials 

are delivered or packaged.

Some of the project materials 

will engage participants in 

learning experiences or will be 

of interest to them, but some of 

the materials will not because of 

the way they are delivered or 

packaged.

The project materials lack 

meaningful learning 

experiences and are very 

unlikely to engage or interest 

participants or users because 

of the way in which they are 

delivered or packaged.

3C The products and services 

produced for this project are 

very appropriate for the 

intended audience.

A portion of the products and 

services produced for this 

project are not appropriate or 

relevant for the intended 

audience.

The products and services 

produced for this project are 

not well-suited (neither 

appropriate nor relevant) for 

the intended audience.

3D Ideas and information are 

effectively communicated 

throughout all of the 

materials in the project.

The ideas and information in the 

project materials are not 

effectively communicated and 

are confusing in some 

instances.

The ideas and information 

are not effectively 

communicated throughout 

the project materials.
Scoring Booklet for Quality Ratings 

E-10

National Evaluation of Comprehensive Centers 

2007-08 

Score Reporting Form for Expert Reviews 

Project Title:_____________________________________________________________ 

________________________________________________________________________ 

Comprehensive Center 

Name:___________________________________________________________ 

Please list your scores below: 

Dimension 1 Score: ________________________

Dimension 2 Score: ________________________

Dimension 3 Score: ________________________

I attest that the scores listed on this scoring sheet are based on my thorough review and 

objective assessment of the project cover sheet and project artifacts based on the scoring 

guidance and rubric provided by the evaluation contractor. I attest that I have not 

discussed these materials or this review with anyone other than the designated evaluation 

team contact prior to submission. 

I have also provided a bulleted summary of this project’s strengths and weaknesses.

I further attest that I do not have any undisclosed conflict of interest for this particular 

project. 

Your Name 

(Print):__________________________________________________________________ 

Your Signature:_______________________________________________________________ 

Date:_____________________________
RCC-Level Participant Survey 

E-11

Comprehensive Center Evaluation Client Survey – RCC 

Participants 

This survey is designed to gather your feedback on a set of technical assistance activities 

and resources. Your name was included in a list of participants in one or more of the activities 

associated with the project in the box below. The specific activities are described on the yellow 

sheet in the front of this booklet. 

 Per the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to 

this data collection will be used only for statistical purposes. The reports prepared for this study 

will summarize findings across the sample and will not associate responses with a specific 

organization or individual. We will not provide information that identifies you or your 

organization to anyone outside the study team, except as required by law. 

Please return this survey using the self-addressed, stamped envelope enclosed 

to [name] of Policy Studies Associates (PSA) at: 

[address] 

_____@_________

Thank you for your feedback!

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless it displays a valid OMB control number. The valid OMB control number for this survey is 1850-

0823. The time required to complete this survey is estimated to average 20 minutes per response, including the time 

to review instructions, search existing data sources, gather the data needed, and respond to the survey questions. 

Send comments regarding this burden estimate or any other aspect of this collection of information, including 

suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651.
RCC-Level Participant Survey 

E-12

Participation 

1. Did you participate in any of the activities described on the previous page? (Circle one.)

a. Yes

b. No (STOP HERE – RETURN SURVEY TO [name] AT PSA) 

c. Unsure/Don‘t remember (STOP HERE – RETURN SURVEY TO [name] AT PSA) 

2. Did you work for a Comprehensive Center during the period from July 2006 through June 2007? 

(Circle one.)

a. Yes

b. No (STOP HERE. Call [name] at -xxx-xxx-xxxx for alternate survey form.) 

3. Of this set of activities and resources (described at the beginning of the survey), how 

much time did you spend participating in each of the following types of activities or 

making use of each of the following types of resources? (Circle one response in each 

row.) 

More than 

5 days 3-5 days 1-2 days

Less than 1 

day

Not 

applicable; 

not part of 

this set of 

activities 

and 

resources

a. Conferences 4 3 2 1 98

b. Training 4 3 2 1 98

c. Task force meetings 4 3 2 1 98

d. Reviewing general or 

background information 

provided by the Content 

Center

4 3 2 1 98

e. Using tools and other 

resources provided by 

the Content Center

4 3 2 1 98

f. Advance planning 4 3 2 1 98

g. Ongoing consultation 

on this topic 4 3 2 1 98

h. Follow-up and action 

plans 4 3 2 1 98
RCC-Level Participant Survey 

E-13

4. Were you personally involved in determining the goals or designing the content or 

format of the activities or resources described at the beginning of this survey? In what 

ways? (Circle all that apply.)

a. Identifying the problem or need to be addressed 

b. Selecting or framing the content 

c. Providing data or other background information during the planning phase 

d. Identifying or recruiting project participants 

e. Identifying or recruiting presenters or resources 

f. Designing activities 

g. Planning for or leading dissemination of new ideas and information 

h. Coordinating this project with other work that my organization does 

i. Planning logistics 

j. Other (Specify: ______________________________________) 

k. I did not contribute at all to the design of this set of activities and resources 
RCC-Level Participant Survey 

E-14

Relevance and Usefulness 

5. Based on your experience, to what degree was this set of activities and resources 

(described at the beginning of this survey) relevant to your work, in each of the following 

respects? (Circle one response in each row.) 

This set of activities and 

resources:

To a 

very 

high 

degree

To a high 

degree

To a 

moderate 

degree

To a low 

degree

To a very 

low degree

Not able 

to judge

a. Addressed a need or problem 

that my Center‘s state-level 

clients face

5 4 3 2 1 95

b. Addressed an important priority 

of my Center‘s state-level 

clients

5 4 3 2 1 95

c. Addressed a challenge that my 

Center‘s state-level clients face 

related to the implementation of 

NCLB

5 4 3 2 1 95

d. Provided information, advice, 

and/or resources that could be 

applied to my Center‘s work or 

our state-level clients‘ work

5 4 3 2 1 95

e. Addressed the particular state 

context in which our state-level 

clients operate

5 4 3 2 1 95

f. Addressed our state-level 

clients‘ specific challenges (e.g., 

policy environment, leadership 

capacity, budget pressures, local 

politics) 

5 4 3 2 1 95

g. Provided information, advice, 

and/or resources that could be 

used by state-level clients to 

guide decisions about policies, 

programs, and practices

5 4 3 2 1 95

h. Highlighted the implications of 

research findings (or 

information about best practice) 

for state-level clients‘ policies, 

programs, or practices

5 4 3 2 1 95
RCC-Level Participant Survey 

E-15

6. Based on your experience, to what degree was this set of activities and resources 

(described at the beginning of this survey) useful to you, in each of the following 

respects? (Circle one response in each row.) 

This set of activities and 

resources:

To a

very 

high 

degree

To a high 

degree

To a 

moderate 

degree

To a low 

degree

To a very 

low degree

Not able 

to judge

a. Provided resources that were

easy to understand and easy to 

use 

5 4 3 2 1 95

b. Employed an appropriate format 

(e.g., a work group, a 

conference, individual 

consultation, written products) 

5 4 3 2 1 95

c. Provided adequate opportunity 

to learn from other states 5 4 3 2 1 95

d. Included adequate follow-up to 

support the use of new 

information and resources 

5 4 3 2 1 95

e. Were timely 5 4 3 2 1 95

f. Helped my Center to help our

state-level clients solve a 

problem 

5 4 3 2 1 95

g. Helped my Center to help our

state-level clients maintain or 

change a policy or practice

5 4 3 2 1 95

h. Helped my Center to help our

state-level clients take the next 

step in a longer-term 

improvement effort

5 4 3 2 1 95

i. Provided my Center with 

information or resources that we 

will use again

5 4 3 2 1 95

j. Helped my Center to develop a 

shared expertise or knowledgebase5 4 3 2 1 95

k. Helped individuals in my Center 

to develop skills that they will 

use again

5 4 3 2 1 95
RCC-Level Participant Survey 

E-16

7. How could this set of activities and resources (described on the yellow sheet inserted into this 

survey) have been made more relevant or more useful for your Center?

______________________________________________________________________________ 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

Priorities for Technical Assistance 

8. Please consider your Comprehensive Center’s priorities for the technical assistance that it 

receives from outside sources. With which of the following tasks does your Center have the 

greatest need for technical assistance? (Please circle your Center’s three highest priorities 

from the list below.) 

Our center has the greatest need for technical assistance with: 

a. Helping state leaders formulate or refine policies that respond to NCLB requirements 

b. Helping states build or manage a state-level system to support districts and schools identified 

for improvement under NCLB 

c. Training or helping to train or manage the state-level staff who provide support to districts 

and schools identified for improvement under NCLB 

d. Helping state-level staff work directly with low-performing districts and schools on school 

improvement activities 

e. Helping state-level staff design or implement statewide assessment and accountability 

systems 

f. Helping state-level staff to align state accountability systems with NCLB accountability 

systems 

g. Supporting the use of assessment data by schools and districts 

h. Disseminating information on scientifically based research to states, or helping state-level 

staff disseminate information on scientifically based research to districts and schools 

i. Identifying and/or developing programs or models that address state, district, or school needs 

j. Helping state-level staff provide training and other professional development to local 

educators in academic subjects (reading language arts, mathematics, science) 

k. Helping state-level staff monitor compliance with NCLB requirements in districts and 

schools 

l. Helping state-level staff communicate with the public about NCLB requirements or report 

cards 

m. Other priorities (Specify: ______________________________) 
RCC-Level Participant Survey 

E-17

9. Did the set of activities and resources (described at the beginning of this survey) address any of 

the following tasks related to NCLB implementation? (Circle “yes” or “no” for all rows.) 

This set of activities and resources addressed: Yes No

a. Helping state leaders formulate or refine policies that respond 

to NCLB requirements 1 0

b. Helping states build on or manage a state-level system to 

support districts and schools identified for improvement under 

NCLB

1 0

c. Training or helping to train or manage the state-level staff who 

provide support to districts and schools identified for 

improvement under NCLB

1 0

d. Helping state-level staff work directly with low-performing 

districts and schools on school improvement activities 1 0

e. Helping state-level staff design or implement statewide 

assessment and accountability systems 1 0

f. Helping state-level staff to align state accountability systems 

with NCLB accountability systems 1 0

g. Supporting the use of assessment data by schools and districts 1 0

h. Disseminating information on scientifically based research to 

states, or helping state-level staff disseminate information on 

scientifically based research to districts and schools

1 0

i. Identifying and/or developing programs or models that address 

state, district, or school needs 1 0

j. Helping state-level staff provide training and other professional 

development to local educators in academic subjects (reading,

language arts, mathematics, science)

1 0

k. Helping state-level staff monitor compliance with NCLB 

requirements in districts and schools 1 0

l. Helping state-level staff communicate with the public about 

NCLB requirements or report cards 1 0

m. Other priorities (Specify: _____________________________) 1 0
RCC-Level Participant Survey 

E-18

Capacity to Carry Out Responsibilities Related to NCLB 

10. To what extent has the set of activities and resources (described at the beginning of the survey) 

expanded the capacity of your Comprehensive Center, including your own capacity, to help 

states carry out their responsibilities related to NCLB? (Circle one response in each row.)

This set of activities and 

resources:

To a 

great 

extent

To a 

moderate 

extent

To a 

small 

extent

Not at 

all

Too 

early to 

tell

Does not 

apply or 

unable 

to judge

a. Confirmed what my Center was 

already doing to help states

4 3 2 1 95 98

b. Helped my Center to raise 

awareness of new developments 

in fields important to states

4 3 2 1 95 98

c. Helped my Center to help states 

define or understand a problem in 

new ways

4 3 2 1 95 98

d. Helped my Center to help states 

accomplish a goal or solve a 

problem

4 3 2 1 95 98

e. Helped my Center to help states 

improve or support an ongoing 

state program, policy, or practice

4 3 2 1 95 98

f. Helped my Center enhance states‘ 

ability to address NCLB 

requirements

4 3 2 1 95 98

g. Helped my Center improve states‘

ability to work with districts 4 3 2 1 95 98

h. Helped my Center improve states‘

ability to work with schools 

identified for improvement under 

NCLB

4 3 2 1 95 98

i. Put my Center in touch with other 

organizations engaged in similar 

tasks

4 3 2 1 95 98

j. Enabled my Center to carry out 

its work more effectively 4 3 2 1 95 98

k. Helped my Center to help states 

complete NCLB-related 

applications, plans, and reports

4 3 2 1 95 98

l. Helped my Center to help states 

make a tangible change to a 

policy or practice

4 3 2 1 95 98

m. Other (Specify:_____________) 4 3 2 1 95 98
RCC-Level Participant Survey 

E-19

11. How could this set of activities and resources (described at the beginning of the survey) have 

better helped to expand your organization‘s capacity to help states carry out responsibilities 

related to NCLB? 

_____________________________________________________________________________ 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

Your Job Responsibilities at the Comprehensive Center 

Please consider all of your job responsibilities when responding to the questions in this section, not just 

those most closely related to the set of activities and resources described on the first page of this survey. 

12. Please describe your role at the Comprehensive Center during the period from July 2006 through 

June 2007: (Circle all that apply.)

a. State liaison 

b. Content or topic area specialist (Specify: ______________________________) 

c. Project-specific manager or specialist 

d. Center director or assistant director 

e. Support staff (technology, research, logistics, publishing, financial) 

f. Consultant 

g. Other (Specify: _____________________________) 
RCC-Level Participant Survey 

E-20

13. During the period from July 2006 through June 2007, which of the following statements best 

describes your job responsibilities related to NCLB implementation? (Circle all that apply.)

a. Helping state leaders formulate or refine policies that respond to NCLB 

requirements

b. Helping states build or manage a state-level system to support districts and schools 

identified for improvement under NCLB

c. Training or helping to train or manage the state-level staff who provide support to 

districts and schools identified for improvement under NCLB

d. Helping state-level staff work directly with low-performing districts and schools on 

school improvement activities

e. Helping state-level staff design or implement statewide assessment and 

accountability systems

f. Helping state-level staff to align state accountability systems with NCLB 

accountability systems

g. Supporting the use of assessment data by schools and districts

h. Disseminating information on scientifically based research to states, or helping 

state-level staff disseminate information on scientifically based research to districts 

and schools

i. Identifying and/or developing programs or models that address state, district, or 

school needs

j. Helping state-level staff monitor compliance with NCLB requirements in districts 

and schools

k. Helping state-level staff communicate with the public about NCLB requirements or

report cards

l. Providing technical assistance to SEA or other state-level staff

m. Other responsibilities (Specify: 

_________________________________________________)
RCC-Level Participant Survey 

E-21

14. During the period from July 2006 through June 2007, what percent of your time was spent on the 

tasks you selected in Question 13 above, combined? (Circle one.)

a. 0-25 percent 

b. 26-50 percent 

c. 51-75 percent 

d. 76-100 percent 

15. Have you participated in any other technical assistance provided by this Content Center, other 

than the set of activities and resources described at the beginning of the survey? (Circle one.)

a. Yes

b. No 

c. Unsure/Don‘t remember

Thank you! 

Please return this survey using the self-addressed, stamped envelope enclosed 

to [name] of Policy Studies Associates at: 

[address] 

_____@_________
State-Level Participant Survey 

E-22

Comprehensive Center Evaluation Client Survey – StateLevel Participants 

This survey is designed to gather your feedback on a set of technical assistance activities 

and resources. Your name was included in a list of participants in one or more of the activities 

associated with the project in the box below. The specific activities are described on the yellow 

sheet in the front of this booklet. 

 Per the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to 

this data collection will be used only for statistical purposes. The reports prepared for this study 

will summarize findings across the sample and will not associate responses with a specific 

organization or individual. We will not provide information that identifies you or your 

organization to anyone outside the study team, except as required by law. 

Please return this survey using the self-addressed, stamped envelope enclosed 

to [name] of Policy Studies Associates (PSA) at: 

[address] 

_____@_________

Thank you for your feedback!

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless it displays a valid OMB control number. The valid OMB control number for this survey is 1850-

0823. The time required to complete this survey is estimated to average 20 minutes per response, including the time 

to review instructions, search existing data sources, gather the data needed, and respond to the survey questions. 

Send comments regarding this burden estimate or any other aspect of this collection of information, including 

suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651.
State-Level Participant Survey 

E-23

Participation 

1. Did you participate in any of the activities described on the previous page? (Circle one.)

a. Yes

b. No (STOP HERE – RETURN SURVEY TO [name] AT PSA) 

c. Unsure/Don‘t remember (STOP HERE – RETURN SURVEY TO [name] AT 

PSA) 

2. In your experience with this set of activities and resources (described at the beginning of 

this survey), how much time did you spend participating in each of the following types of 

activities or making use of each of the following types of resources? (Circle one 

response in each row.) 

Type of activity or 

resource

More than 

5 days 3-5 days 1-2 days

Less than 1 

day

Not 

applicable; 

not part of 

this set of 

activities 

and 

resources

a. Conferences 4 3 2 1 98

b. Training 4 3 2 1 98

c. Task force meetings 4 3 2 1 98

d. Reviewing general or 

background information 

provided by the Content 

Center

4 3 2 1 98

e. Using tools and other 

resources provided by 

the Content Center

4 3 2 1 98

f. Advance planning 4 3 2 1 98

g. Ongoing consultation 

on this topic 4 3 2 1 98

h. Follow-up and action 

plans 4 3 2 1 98
State-Level Participant Survey 

E-24

3. Were you personally involved in determining the goals or designing the content or 

format of this set of activities and resources (described at the beginning of this survey)? 

In what ways? (Circle all that apply.)

a. Identifying the problem or need to be addressed 

b. Selecting or framing the content 

c. Providing data or other background information during the planning phase 

d. Identifying or recruiting project participants 

e. Identifying or recruiting presenters or resources 

f. Designing activities 

g. Planning for or leading dissemination of new ideas and information 

h. Coordinating this set of activities with other work that my organization does 

i. Planning logistics 

j. Other (Specify: ______________________________________) 

k. I did not contribute at all to the design of this set of activities and resources 
State-Level Participant Survey 

E-25

Relevance and Usefulness 

4. Based on your experience, to what degree was this set of activities and resources 

(described at the beginning of this survey) relevant to your work, in each of the following 

respects? (Circle one response in each row.) 

This set of activities and 

resources:

To a 

very 

high 

degree

To a high 

degree

To a 

moderate 

degree

To a low 

degree

To a very 

low degree

Not able 

to judge

a. Addressed a need or 

problem that my 

organization faces

5 4 3 2 1 95

b. Addressed an important 

priority of my organization 5 4 3 2 1 95

c. Addressed a challenge that 

my organization faces 

related to the 

implementation of NCLB

5 4 3 2 1 95

d. Provided information, 

advice, and/or resources 

that could be applied to 

my organization‘s work

5 4 3 2 1 95

e. Addressed our particular 

state context 5 4 3 2 1 95

f. Addressed my 

organization‘s specific 

challenges (e.g., policy 

environment, leadership 

capacity, budget pressures, 

local politics) 

5 4 3 2 1 95

g. Provided information, 

advice, and/or resources 

that could be used to guide 

decisions about policies, 

programs, and practices

5 4 3 2 1 95

h. Highlighted the 

implications of research 

findings (or information 

about best practice) for 

policies, programs, or 

practices

5 4 3 2 1 95
State-Level Participant Survey 

E-26

5. Based on your experience, to what degree was this set of activities and resources 

(described at the beginning of this survey) useful to you, in each of the following 

respects? (Circle one response in each row.) 

The activities and resources:

To a 

very 

high 

degree

To a high 

degree

To a 

moderate 

degree

To a low 

degree

To a very 

low degree

Not able 

to judge

a. Provided resources that were 

easy to understand and easy to 

use 

5 4 3 2 1 95

b. Employed an appropriate format 

(e.g., a work group, a 

conference, individual 

consultation, written products) 

5 4 3 2 1 95

c. Provided adequate opportunity 

to learn from colleagues in other 

states

5 4 3 2 1 95

d. Included adequate follow-up to 

support the use of new 

information and resources 

5 4 3 2 1 95

e. Were timely 5 4 3 2 1 95

f. Helped my organization to solve 

a problem 5 4 3 2 1 95

g. Helped my organization to 

maintain or change a policy or 

practice

5 4 3 2 1 95

h. Helped my organization take the 

next step in a longer-term 

improvement effort

5 4 3 2 1 95

i. Provided my organization with 

information or resources that we 

will use again

5 4 3 2 1 95

j. Helped my organization to

develop a shared expertise or 

knowledge base

5 4 3 2 1 95

k. Helped individuals in my 

organization to develop skills 

that they will use again

5 4 3 2 1 95
State-Level Participant Survey 

E-27

6. How could this set of activities and resources (described on the yellow sheet inserted into this 

survey) have been made more relevant or more useful for your organization? 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

 

Priorities for Technical Assistance 

7. Please consider your organization’s priorities for the technical assistance that it receives from 

outside sources. With which of the following tasks related to NCLB implementation does your 

organization have the greatest need for technical assistance? (Please circle your three highest 

priorities from the list below.)

a. Formulating or refining state policies to respond to NCLB requirements 

b. Building or managing a statewide system of support for districts and schools identified 

for improvement under NCLB 

c. Training or managing the state-level staff or school support teams who provide support to 

districts and schools identified for improvement under NCLB 

d. Working directly with low-performing schools or districts on school improvement 

activities 

e. Designing or implementing state assessment and accountability systems 

f. Aligning state accountability systems with NCLB accountability system 

g. Supporting use of assessment data by schools and districts 

h. Disseminating information on scientifically based research to districts and schools 

i. Identifying and/or developing programs or models that address district and/or school 

needs 

j. Providing training and other professional development to local educators in academic 

subjects (reading, language arts, mathematics, science) 

k. Monitoring compliance with NCLB requirements in districts and schools 

l. Communicating with the public about NCLB requirements or report cards 

m. Other priorities (Specify: _____________________________) 
State-Level Participant Survey 

E-28

8. Did the set of activities and resources (described at the beginning of this survey) address any of 

the following tasks related to NCLB implementation? (Select “yes” or “no” for all rows.) 

This set of activities and resources addressed: Yes No

a. Reformulating or refining state policies to respond to NCLB 

requirements 1 0

b. Building or managing a statewide system of support for districts 

and schools identified for improvement under NCLB 1 0

c. Training or managing the state-level staff or school support 

teams who provide support to districts and schools identified 

for improvement under NCLB

1 0

d. Working directly with low-performing schools or districts on 

school improvement activities 1 0

e. Designing or implementing state assessment and accountability 

systems

1 0

f. Aligning state accountability systems with NCLB 

accountability systems 1 0

g. Supporting use of assessment data by schools and districts 1 0

h. Disseminating information on scientifically based research to 

districts and schools 1 0

i. Identifying and/or developing programs or models that address 

district and/or school needs 1 0

j. Providing training and other professional development to local 

educators in academic subjects (reading, language arts, 

mathematics, science)

1 0

k. Monitoring compliance with NCLB requirements in districts 

and schools 1 0

l. Communicating with the public about NCLB requirements or 

report cards 1 0

m. Other priorities (Specify: ____________________________) 1 0
State-Level Participant Survey 

E-29

Capacity to Carry out Responsibilities Related to NCLB 

9. To what extent has the set of activities and resources described in the box on the first page 

expanded the capacity of your team, office, division, or unit to carry out its responsibilities 

related to NCLB? (Circle one response in each row.)

This set of activities and 

resources:

To a 

great 

extent

To a 

moderate 

extent

To a 

small 

extent

Not at 

all

Too 

early to 

tell

Does not 

apply or 

unable 

to judge

a. Confirmed what we were already 

doing 4 3 2 1 95 98

b. Raised awareness of new 

developments in fields important to 

my organization

4 3 2 1 95 98

c. Helped my organization define or 

understand a problem in new ways 4 3 2 1 95 98

d. Helped my organization take steps 

toward accomplishing a goal or 

solving a problem

4 3 2 1 95 98

e. Helped my organization maintain or 

improve an ongoing program, 

policy, or practice

4 3 2 1 95 98

f. Helped my organization plan or 

initiate a new program, policy, or 

practice

4 3 2 1 95 98

g. Enhanced my organization‘s ability 

to address NCLB requirements 4 3 2 1 95 98

h. Improved my organization‘s ability 

to work with districts 4 3 2 1 95 98

i. Improved my organization‘s ability 

to work with schools identified for 

improvement under NCLB

4 3 2 1 95 98

j. Put my organization in touch with 

other organizations engaged in 

similar tasks

4 3 2 1 95 98

k. Enabled my organization to carry 

out its work more effectively 4 3 2 1 95 98

l. Helped my organization complete 

NCLB-related applications, plans, 

and reports

4 3 2 1 95 98

m. Helped my organization make a 

tangible change to a policy or 

practice 

4 3 2 1 95 98

n. Other (Specify:

__________________________

4 3 2 1 95 98
State-Level Participant Survey 

E-30

10. How could this set of activities and resources have better helped to expand the capacity of your

team, office, division, or unit to carry out responsibilities related to NCLB? 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

 ______________________________________________________________________________ 

Your Job Responsibilities 

11. Please indicate the type of agency for which you worked during the period from July 2006 

through June 2007. (Circle one.)

a. State education agency (SEA) 

b. Governor‘s office or other state agency (Specify: ____________________)

c. Intermediate education agency (a regional resource center or area education agency serving a 

region within the state) 

d. Institution of higher education 

e. Regional Comprehensive Center 

f. Local education agency 

g. School 

h. Other (Specify: _____________________________) 

12. Did you participate in this set of activities and resources (described at the beginning of this 

survey) because you are a member of a state-sponsored school support team or a state-level task 

force? (Circle one.) 

a. Yes

b. No

c. Don‘t know/Not sure (Explain: ___________________________) 
State-Level Participant Survey 

E-31

13. Which of the following statements best describes your job responsibilities related to NCLB 

implementation, during the period from July 2006 through June 2007? (Circle all that apply.)

a. Formulating or refining state policies to respond to NCLB requirements

b. Building or managing a statewide system of support for districts and schools 

identified for improvement under NCLB

c. Training or managing the state-level staff or school support teams who provide 

support to districts and schools identified for improvement under NCLB

d. Working directly with low-performing schools or districts on school improvement 

activities

e. Designing or implementing state assessment and accountability systems

f. Aligning state accountability systems with NCLB accountability systems

g. Supporting use of assessment data by schools and districts

h. Disseminating information on scientifically based research to districts and schools

i. Identifying and/or developing programs or models that address district and/or 

school needs

j. Providing training and other professional development to local educators in 

academic subjects (reading, language arts, mathematics, science)

k. Monitoring compliance with NCLB requirements in districts and schools

l. Communicating with the public about NCLB requirements or report cards 

m. Other (Specify: _____________________________)

14. During the period from July 2006 through June 2007, what percent of your time was spent on all 

the tasks you selected in Question 13 above, combined? (Circle one.)

a. 0-25 percent 

b. 26-50 percent 

c. 51-75 percent 

d. 76-100 percent 

15. Have you participated in any other technical assistance provided by this Content Center, other 

than the set of activities and resources described at the beginning of the survey? (Circle one.)

a. Yes

b. No 

c. Unsure/Don‘t know

Thank you! 

Please return this survey using the self-addressed, stamped envelope enclosed 

to [name] of Policy Studies Associates at: 

[address] 

_____@_________