NCEE 2011-4031 U. S. D E PA R T M E N T OF E DUC ATIO N

National Evaluation of the 

Comprehensive Technical 

Assistance Centers

Final Report

National Evaluation of the 

Comprehensive Technical Assistance 

Centers 

Final Report 

AUGUST 2011 

Brenda J. Turnbull 

Richard N. White 

Elizabeth Sinclair 

Derek L. Riley 

Policy Studies Associates, Inc. 

Carol Pistorino 

Decision Information Resources, Inc. 

Yumiko Sekino 

Project Officer 

Institute of Education Sciences 

NCEE 2011-4031 

U.S. DEPARTMENT OF EDUCATION 
 

 

U.S. Department of Education 

Arne Duncan 

Secretary 

Institute of Education Sciences 

John Q. Easton 

Director 

National Center for Education Evaluation and Regional Assistance 

Rebecca Maynard 

Commissioner 

August 2011 

This report was prepared for the Institute of Education Sciences under Contract No. ED-04-CO-0028/0001. The 

project officer is Yumiko Sekino in the National Center for Education Evaluation and Regional Assistance. 

IES evaluation reports present objective information on the conditions of implementation and impacts of the 

programs being evaluated. IES evaluation reports do not include conclusions or recommendations or views with 

regard to actions policymakers or practitioners should take in light of the findings in the reports. 

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While permission to 

reprint this publication is not necessary, the citation should be: Turnbull, B.J., White, R.N., Sinclair, E., Riley, D.L., 

and Pistorino, C. (2011, August). National Evaluation of the Comprehensive Technical Assistance Centers: Final 

Report (NCEE 2011-4031). Washington, DC: National Center for Education Evaluation and Regional Assistance, 

Institute of Education Sciences, U.S. Department of Education. 

This report also is available on the IES website at http://ncee.ed.gov. 

Upon request, this report is available in alternate formats, such as Braille, large print, audiotape, or computer 

diskette. For more information, please contact the Department‘s Alternate Format Center at 202-205-8113. 
Acknowledgments 

We very gratefully acknowledge the cooperation of the Comprehensive Centers and the 

many other respondents to our three annual rounds of data collection. Center leaders and staff 

graciously hosted site visits, compiled inventories of their projects, and assembled hefty 

collections of their materials for expert review. Senior managers in state education agencies were 

kind enough to respond to the study’s survey regarding their experiences with Comprehensive 

Center technical assistance, as were the many technical assistance participants housed in state, 

intermediate, and local agencies and other organizations. Expert panel members brought 

knowledge and care to their task of reviewing and rating project materials. 

A Technical Working Group offered useful insights and feedback in two meetings as well 

as informal consultations: the group’s members were Rolf Blank, Margaret Goertz, Joseph 

Johnson, Michael Petrilli, Gary Ritter, Larry Rudner, and Colleen Seremet. 

In addition to the authors, other members of the participating organizations played 

important roles in study design, data collection, and analysis. Alvia Branch of Branch 

Associates, Inc. (BAI), provided steady leadership for the study team. We continue to miss 

Cynthia Sipe, a key member of the BAI leadership, who passed away in 2010. Lynda Bell, Tia 

Burroughs, Matthew Coll, Siobhan Cooney, Barbara Fink, Jennifer Gross, Ceane Rabada, Susan 

Robbins, Virginia Smith, and Jennifer Thompson were valued team members at BAI. Russell 

Jackson of Decision Information Resources, Inc. (DIR), was a source of wise counsel throughout 

the study. Also at DIR, Nancy Dawson, Val Sheppard, Stella Weathersby, Frenetta Tate, Pia 

White, and Jing Yu were stalwart contributors. At Policy Studies Associates (PSA), team 

members who made valuable contributions in data collection or coding included Erikson Arcaira, 

Alisha Butler, Stephen Coleman, Tiffany Miller, Andrea Palmiter, Tandra Turner, and Yvonne 

Woods. Ben Lagueruela provided essential help in document production. Katrina Laguarda, 

formerly of PSA, made tremendous contributions to the study design and early activities. 

iii 

Disclosure of Potential Conflicts of Interest 

The research team for this evaluation consists of a prime contractor, Branch Associates, 

Inc. and two subcontractors, Policy Studies Associates, Inc. (PSA), and Decision Information 

Resources, Inc. (DIR). None of these organizations or their key staff members has a financial 

interest that could be affected by findings from the evaluation of the Comprehensive Center 

program considered in this report. Additionally, no one on the Technical Working Group, 

convened by the research team to provide advice and guidance, has financial interests that could 

be affected by findings from the evaluation. 

v 

Contents 

Page 

Acknowledgments.......................................................................................................................... iii
 

Disclosure of Potential Conflicts of Interest ....................................................................................v
 

List of Exhibits............................................................................................................................... ix
 

Executive Summary ..................................................................................................................... xiii
 

1. Introduction and Background ......................................................................................................1
 

The Center Program........................................................................................................................ 1
 

Structure of the Center Program ..................................................................................................... 3
 

Background on the State Role in NCLB......................................................................................... 6
 

Research Questions Addressed in This Report ............................................................................... 8
 

Organization of This Report ........................................................................................................... 9
 

2. Study Design..............................................................................................................................11
 

Identification of a Sample of Center Work for Expert Review and Participant Surveys ............. 13
 

Describing Center Operations....................................................................................................... 19
 

Survey of State Managers ............................................................................................................. 21
 

Assessment of Quality by Expert Panels ...................................................................................... 23
 

Survey of Project Participants....................................................................................................... 28
 

Units of Analysis and Use of Statistical Tests in this Report ....................................................... 30
 

3. How the Centers Operated.........................................................................................................33
 

Developing Plans: Identifying Client Needs and Priorities .......................................................... 34
 

Adjustments Made in Ongoing Work ........................................................................................... 37
 

The Nature of Center Work .......................................................................................................... 41
 

Working Relationships in Technical Assistance Delivery ............................................................ 48
 

Quality Assurance Procedures ...................................................................................................... 55
 

4. State Managers’ Assessment of Center Technical Assistance ...................................................59
 

Extent to Which Center Assistance Served State Purposes .......................................................... 59
 

Reported Assistance Delivery and Capacity Building for State NCLB Responsibilities ............. 63
 

State Uses and Perceptions of the Centers Compared with Other Sources 

of Technical Assistance.......................................................................................... 67
 

SEA Overall Rating of Center Assistance .................................................................................... 71
 

5. Variation in the Reported Quality, Relevance, and Usefulness 

of the Centers’ Technical Assistance ......................................................................73
 

Average Ratings by Experts and Participants............................................................................... 76
 

Variation by Type of Project......................................................................................................... 78
 

Relationships between the Relevance, Usefulness, and Quality of Center Projects..................... 85
 

Variation by Participant Experiences and Roles ........................................................................... 87
 

vii 
Contents (continued) 

Page 

Appendix A: Center Lead Grantee and Subgrantee Organizations ............................................ A-1
 

Appendix B: Study Sample: Center Projects and Project Participants ........................................B-1
 

Appendix C: Describing Center Operations ............................................................................... C-1 

Appendix D: Supplemental Tables – Center Operations ............................................................ D-1 

Appendix E: Survey of State Managers and Case Study Visits to 10 States ...............................E-1 

Appendix F: Supplemental Tables – State Manager Survey Results ..........................................F-1 

Appendix G: Ratings of Quality, Relevance, and Usefulness .................................................... G-2
 

Appendix H: Supplemental Tables – Quality, Relevance, and Usefulness Ratings....................H-1 

viii 
List of Exhibits 

Page 

Exhibit ES.1. Sampled RCC and CC projects by types of activities and resources, by year .......xx 

Exhibit ES.2. Extent to which technical assistance from the Centers served state purposes, 

as judged by state managers, by year .................................................................. xxii 

Exhibit ES.3. Extent to which Center assistance expanded state capacity in priority areas, as 

judged by state managers who rated the area as a major or moderate technical 

assistance priority, by year ................................................................................. xxiii 

Exhibit ES.4. Quality, relevance, and usefulness items .............................................................xxv 

Exhibit ES.5. Mean ratings of technical quality, relevance, and usefulness, by center type 

and by year ......................................................................................................... xxvi
 

Exhibit 1.1. Center program design .............................................................................................4
 

Exhibit 1.2. Center funding, by year............................................................................................7
 

Exhibit 2.1. Data sources for the research questions, 2006-07, 2007-08, and 2008-09 ............12
 

Exhibit 2.2. Number of projects on the project inventory forms (PIFs), by project size and 

 by year ....................................................................................................................15
 

Exhibit 2.3. Sample size by budget ...........................................................................................17
 

Exhibit 2.4. Number of projects in the study sample, by project size and by year ..................17
 

Exhibit 2.5. Distribution of all major and moderate projects and projects in the evaluation 

sample, by topic in 2008-09 ...................................................................................18
 

Exhibit 2.6. Number of responses to the state manager survey, by year ...................................22
 

Exhibit 2.7. Office division or department directed by state manager survey respondents, 

 by year ....................................................................................................................23
 

Exhibit 2.8. Distribution of discrepancies and results of the discrepancy call process, 

 by year ....................................................................................................................27
 

Exhibit 2.9. Interrater agreement before and after discrepancy calls (original versus revised 

scores), as measured by the rWG(J) index, by year ...............................................27
 

Exhibit 2.10. Survey of project participants sampling and survey administration summary ......29
 

Exhibit 3.1. RCC and CC interaction with clients in planning work, by year ...........................35
 

Exhibit 3.2. CC assessment of state needs, 2008-09 .................................................................36
 

Exhibit 3.3. Reasons reported for declining technical assistance requests, 2008-09 ...............37
 

Exhibit 3.4. Topics on which Centers stated objectives and/or delivered projects, 2008-09 ....39
 

Exhibit 3.5. Topics on which RCCs and CCs stated objectives and/or delivered projects, 

2008-09 ..................................................................................................................40
 

Exhibit 3.6. Reported investment of Center resources in long-term vs. short-term projects, 

2008-09 ..................................................................................................................42
 

Exhibit 3.7. RCC and CC activities and resources by reported client demand, investment, and 

importance to objectives, 2008-09 .........................................................................43
 

Exhibit 3.8. State managers’ reports of assistance from Centers, by type of assistance, 

 by year ....................................................................................................................44
 

Exhibit 3.9. Percent of projects on PIFs, by topic and by year..................................................45
 

ix 
List of Exhibits (continued) 

Page 

Exhibit 3.10. Sampled RCC and CC projects by types of activities and resources, by year .......46
 

Exhibit 3.11. RCC-CC coordination reported by Center directors, 2008-09 ..............................50
 

Exhibit 3.12. Sampled RCC projects by CC contribution, by year .............................................51
 

Exhibit 3.13. Sampled CC projects by RCC contribution, by year .............................................51
 

Exhibit 3.14. Centers’ perception of barriers to achieving objectives in delivering technical 

assistance to states, 2008-09 ..................................................................................53
 

Exhibit 3.15. Barriers to CCs’ assistance to RCCs, 2008-09, perceived by RCCs and CCs .......55
 

Exhibit 3.16. Sources of expertise used in quality assurance, by Center type, 2008-09 ............56
 

Exhibit 4.1. Extent to which technical assistance from the Centers served state purposes, 

as judged by state managers, by year .....................................................................60
 

Exhibit 4.2. Reasons why Center assistance served some states’ purposes less than 

completely, by year, as reported by state managers, and barriers reported by 

Center directors to achieving Center aims, 2008-09 ..............................................62
 

Exhibit 4.3. State priorities for technical assistance and assistance received from Centers, 

2007-08 and 2008-09 .............................................................................................65
 

Exhibit 4.4. Extent to which Center assistance expanded state capacity in priority areas, as 

judged by state managers who rated the area as a major or moderate technical 

assistance priority, by year.....................................................................................66
 

Exhibit 4.5. States’ use of external sources of technical assistance, by year ............................68
 

Exhibit 4.6. Purposes for which states used external sources of technical assistance,

 2008-09 ..................................................................................................................69
 

Exhibit 4.7. Usefulness of Center assistance compared with assistance from other sources, 

2008-09 ..................................................................................................................70
 

Exhibit 4.8. Mean ratings of the relevance and usefulness of Center assistance 

by component, as judged by state managers, 2008-09 ...........................................72
 

Exhibit 5.1. Quality, relevance, and usefulness items ...............................................................75
 

Exhibit 5.2. Scales for rating of quality by expert panels and relevance and usefulness by 

participants.............................................................................................................76
 

Exhibit 5.3. Mean ratings of technical quality, relevance, and usefulness, by center type and 

by year....................................................................................................................77
 

Exhibit 5.4. Percent of projects by rating for technical quality, relevance, and usefulness, 

 by year ....................................................................................................................79
 

Exhibit 5.5. Technical quality, relevance, and usefulness ratings of SOS projects 

vs. all other projects in the study sample, by year .................................................82
 

Exhibit 5.6. Technical quality, relevance, and usefulness ratings of RCC projects, 

by presence or absence of CC contributions, by year ............................................83
 

Exhibit 5.7. Technical quality, relevance, and usefulness ratings of CC projects, 

by presence or absence of RCC contribution, by year ...........................................84
 

x 
List of Exhibits (continued) 

Page 

Exhibit 5.8. Technical quality, relevance, and usefulness ratings by source of quality 

assurance, 2008-09.................................................................................................85
 

Exhibit 5.9. Correlations between project-level technical quality, relevance, and usefulness 

ratings, by year.......................................................................................................86
 

Exhibit 5.10. Participants’ involvement in determining project goals or design, by type of 

Center and by year .................................................................................................89
 

Exhibit 5.11. Mean relevance and usefulness ratings, by respondent involvement in 

 determining project goals or design, by year .........................................................90
 

Exhibit 5.12. Relevance and usefulness ratings, by amount of time participants spent on 

 project activities, by year .......................................................................................92
 

Exhibit 5.13. Where participants in RCC projects were employed, by year ...............................93
 

Exhibit 5.14. Mean relevance and usefulness ratings for RCC projects, by agency where 

respondent worked, by year ...................................................................................94
 

Exhibit 5.15. Where participants in CC projects were employed, by year ..................................95
 

Exhibit 5.16. Mean relevance and usefulness ratings for CC projects, RCC staff vs. state-level 

staff, by year...........................................................................................................95
 

Exhibit 5.17. Mean relevance and usefulness ratings, by percent of time respondent spent on 

NCLB-related tasks in his/ her job and by year....................................................96
 

xi 

 

 

 

Executive Summary 

This final report presents findings from a multi-year evaluation of the Comprehensive 

Technical Assistance Centers, a federally funded program that provides technical assistance to 

states in connection with the Elementary and Secondary Education Act, as reauthorized by the 

No Child Left Behind (NCLB) Act of 2001. The law authorizing the Centers, the Educational 

Technical Assistance Act of 2002, mandated that a national evaluation of the program be 

conducted by the Institute of Education Sciences (IES). The legislation indicated that the 

evaluation should “include an analysis of the services provided...[and] the extent to which each 

of the comprehensive centers meets the objectives of its respective plan, and whether such 

services meet the educational needs of State educational agencies, local educational agencies, 

and schools in the region.” The program evaluation was conducted by Branch Associates, Inc., 

Decision Information Resources, Inc., and Policy Studies Associates, Inc. 

With the redesign of the Center program, the primary focus of technical assistance was 

directed to states. In order to build states’ capacity for carrying out NCLB responsibilities, which 

include assistance to struggling school districts and schools as well as other areas of NCLB 

program administration, the Center program was designed to supply ongoing technical assistance 

in using research knowledge and promising practices. There are two types of Centers: 

■	 Sixteen Regional Comprehensive Centers (RCCs) are responsible for providing 

ongoing technical assistance to states assigned to their region, working with a 

range of one to eight states per Center 

■	 Five Content Centers (CCs) are expected to supply knowledge to RCCs and work 

with RCCs to assist states in the CC’s specialty area: Assessment and 

Accountability, Instruction, Teacher Quality, Innovation and Improvement, or 

High Schools 

Given this program design, the evaluation provides a description of Center operations. It also 

reports on assistance delivery and contributions to state capacity as judged by managers in state 

education agencies (SEAs), on quality as judged by panels of subject-matter experts, and on 

relevance and usefulness as judged by practitioners who participated in Center activities or 

received Center products. The evaluation data, collected annually, pertain to the Center program 

years 2006-07, 2007-08, and 2008-09, covering three of the five program years starting with the 

second year of program funding.1 

■	 The operations of the RCCs and CCs were consistent with the Center program 

design. RCCs and CCs assessed client needs annually to determine their technical 

assistance plans, with informal communications as the mode most commonly 

reported for 2008-09. The most common activity found in sampled RCC projects2 

1 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106),
 

32583-94. The awards were subsequently extended.

2

 For the purposes of this evaluation, the team identified “projects” as a common level of aggregation of Center
 

activities that would constitute units large enough for review and rating, but focused enough for coherence. A 

xiii 
 

was “ongoing consultation and follow up” (82, 93, and 91 percent of the sampled 

RCC projects in years 2006-07, 2007-08, and 2008-09, respectively), consistent 

with the charge to provide frontline assistance on an ongoing basis to states. In 

CC projects the most common activity was “research collections and synthesis” 

(more than 70 percent of sampled projects in each year), consistent with the CCs’ 

prescribed focus on synthesizing, translating, and delivering knowledge to RCCs 

and states. Across the three years studied, both RCCs and CCs were more 

involved in each other’s projects. Among sampled RCC projects, the percentage 

that included direct assistance from CC staff was 18 percent in 2006-07, 22 

percent in 2007-08 and 30 percent in 2008-09. The percentage of CC projects that 

included RCC direct assistance was 11 percent in 2006-07, 12 percent in 2007-08, 

and 38 percent in 2008-09. In addition, by 2008-09 all 16 RCCs reported 

receiving knowledge resources from CCs and all 5 CCs reported providing 

knowledge resources to RCCs. 

■	 Centers addressed the most frequently cited state priority of “statewide systems 

of support,” and an increasing number of state managers reported each year 

that Center assistance served their purposes. “Systems of support” consists of an 

infrastructure for the delivery of onsite assistance, and strategies and materials 

designed to help struggling schools and districts improve student performance. 

The most widespread NCLB-related priority for state managers was “statewide 

systems of support or school support teams,” which was identified as a major or 

moderate priority for technical assistance by more than 90 percent of managers, 

weighted, in each year. Of this group of state managers, more than 90 percent 

reported each year that the Centers delivered assistance related to this 

responsibility. “Systems of support” was not only the most widely reported state 

priority but also the topic addressed in more Center projects in each year than any 

other topic, according to the inventories compiled by the Centers (19 percent of 

all projects in 2006-07, 25 percent in 2007-08, and 21 percent in 2008-09, 

compared with 10 percent or fewer projects addressing any other topic). With 

each state weighted equally in the analysis, the proportion of state agency 

managers reporting that assistance from the Centers had “served the state’s 

purposes completely” rose from about one-third (36 percent) in 2006-07 to more 

than half (56 percent) in 2008-09. 

■	 Center assistance was reported by state managers as having expanded state 

capacity in “statewide systems of support,” which has been a predominant focus 

of Center assistance. Among state managers who reported statewide systems of 

support or school support teams as a state priority for technical assistance in 

2008-09, 82 percent credited Center assistance with a “great” or “moderate” 

expansion of state capacity in this area. In other areas of state responsibility 

identified by state managers to be a priority for technical assistance, the 

percentage reporting a great or moderate expansion of state capacity in 2008-09 

ranged from 77 percent (for research-based curriculum, instruction, or 

“project” was defined as a group of closely related activities and/or deliverables designed to achieve a specific 

outcome for a specific audience. 

xiv 
 

 

 

professional development in academic subjects) to 39 percent (for NCLB’s 

provisions on supplemental educational services and choice). 

■	 On average across each of the three years, expert panels rated sampled project 

materials between “moderate” and “high” for quality, and project participants 

rated the sampled projects “high” for relevance and usefulness. Program-wide 

average ratings, on a 5-point scale with 5 at the high end, were 3.34 in 2006-07, 

3.51 in 2007-08, and 3.57 in 2008-09 for technical quality; 3.94, 4.08, and 4.15, 

respectively, for relevance; and 3.69, 3.95, 3.96, respectively, for usefulness.3

 In 

addition, the average quality rating was consistently higher among CC projects 

than RCC projects by more than one-half of a standard deviation while RCC 

ratings went up each year.4

 The average ratings of relevance were higher for RCC 

than CC projects in 2006-07 and 2007-08 although CC ratings went up each year; 

there were no consistent differences in the usefulness ratings between RCCs and 

CCs. 

The Comprehensive Centers Program 

In its authorization under the Educational Technical Assistance Act of 2002, the Center 

program was given an overall charge of supporting state and local NCLB implementation. The 

U.S. Department of Education (ED), using discretion provided in the legislation, established two 

major program features that differed from the design of Comprehensive Center programs under 

prior legislation:5  

■	 First, the primary focus would be on assisting states to carry out NCLB 

responsibilities and helping build state capacity to deliver assistance to schools 

and districts; ED specified that Centers could only work directly with districts or 

schools under special circumstances. 

■	 Second, awards would be made to 21 Centers to establish two-tiers of technical 

assistance with 16 RCCs and 5 CCs. They were instructed to work as follows: 

■	 Each RCC was charged with providing ongoing assistance directly to 

states in its region (“frontline assistance”), serving the needs of either one 

3

 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 

to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 

and each project sampled from a Center contributed equally to the Center mean. 4

 All project-level differences described in this report (e.g., more, higher) reflect a difference of one-half of one 

pooled standard deviation between groups of projects. Using a metric derived from Cohen (1988), the evaluation 

team estimated Cohen’s d (an estimate of the effect size defined as the difference in means divided by the pooled 

standard deviation) and adopted the logic of Cohen for what would be considered a moderate difference. For this 

study, inferential tests of statistical significance were not conducted to examine project-level differences in these 

non-probability samples. All participant-level differences described in this report reflect statistical test of 

significance with a criterion value of p<.05. 

5 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 

32583-94. 

xv 
 

large state or a group of two to eight states and other jurisdictions.6 The 

RCCs were also expected to deliver technical assistance to their assigned 

states, addressing the needs and building capacity of the states to assist 

their districts and schools. 

■	 Meanwhile, each CC would work on a nationwide basis to provide indepth knowledge of the content and research within a particular 

substantive area: Assessment and Accountability, Instruction, Teacher 

Quality, Innovation and Improvement, or High Schools. CCs would 

facilitate access to, and use of, existing research and practices. 

■	 The absolute priorities for the two types of Centers indicated that they 

should work together: Regional Centers should draw information and 

resources from Content Centers as well as other sources; and Content 

Centers should both supply knowledge to Regional Centers and “work 

closely with Regional Centers to provide technical assistance to States.” 

Research Questions and Methods 

The research priorities for the evaluation were primarily driven by the statute and focused 

on the following key research questions: 

1.	 How did the Regional Comprehensive Centers and Content Centers operate as part of the 

Comprehensive Technical Assistance Center program? 

■	 How did Centers develop, refine, and carry out their plans for technical 

assistance? How did they define their clients’ educational needs and priorities? 

■	 What were the objectives of the technical assistance the Centers offered? What 

kinds of products and services were provided by the Centers? 

■	 How did the Regional Comprehensive Centers and Content Centers coordinate 

their work? 

2.	 What was the performance of the Comprehensive Centers in addressing state needs and 

priorities? How did their performance change over the period of time studied? 

■	 How did the Centers’ state clients define their needs and priorities? 

■	 To what extent, as reported by states, did Center assistance expand state capacity 

to address underlying needs and priorities and meet the goals of NCLB? 

6

 The nonstate jurisdictions that the Centers were to serve were the following: the District of Columbia, Puerto Rico, 

the Virgin Islands, American Samoa, Commonwealth of the Northern Mariana Islands, Federated States of 

Micronesia (Chuuk, Kosrae, Pohnpei, and Yap), Guam, Republic of the Marshall Islands, and Republic of Palau. 

Throughout this report, the term “state” will be defined to include the 50 states as well as these other jurisdictions. 

xvi 
■	 To what extent did states rely on other sources of technical assistance besides the 

Centers? What were other sources of technical assistance that states used? How 

did the usefulness of Center assistance compare with the usefulness of assistance 

from other sources? 

3.	 To what extent was the assistance provided by the Centers of high quality, high relevance, 

and high usefulness? 

■	 Did the quality, relevance, or usefulness of Center assistance change over the 

period of time studied? 

■	 What was the variation in the quality, relevance, and usefulness of Center 

assistance across types of projects and participants? 

The evaluation gathered information annually on the Center program for the years 2006-

07, 2007-08, and 2008-09 from six data sources in order to address the research questions above. 

Data sources included: 

■	 Management plans. The evaluation reviewed these plans as a data source for 

each Center’s intended focus at the beginning of the year, drawing from the plans 

a list of topics as foci of Center objectives. 

■	 Project inventory forms and cover sheets. Each Center completed an inventory 

of its work that grouped related activities and deliverables into “projects,” with 

the project defined as a group of closely related activities and/or deliverables 

designed to achieve a specific outcome for a specific audience. Projects were in 

turn classified by the Centers into major, moderate, and minor projects on the 

basis of the relative level of effort they reflected. The Centers also classified the 

projects, according to the topics addressed, into 22 topical categories.7

 At each 

stage, the evaluation team provided written guidance and training for inventory 

development, reviewed the Centers’ drafts, and clarified definitions as needed. 

For projects sampled for the evaluation, the Centers prepared “cover sheets” 

providing brief descriptions and contexts for the activities and resources included 

in the project. The evaluation team used the cover sheets as a data source for 

coding project activities and resources. 

 

 

 

7

 The 22 topics were: components of effective systems of support for states, districts, and schools; data use or datadriven decision making; formative assessment; reading; adolescent literacy; mathematics; dropout prevention; high 

school redesign or reform; transition to high school; special education curriculum, instruction and professional 

development; special education assessment; English language learners;” highly qualified teacher” provisions of 

NCLB; teacher preparation and induction; teacher professional development; supplemental educational services; 

Response to Intervention; migrant education; Indian or Native American education; data management and 

compliance; assessment design; and parent involvement. In addition, projects that addressed none of these 22 topics 

were categorized as “other.” 

xvii 
■	 Center staff interviews. Using structured response categories, Center staff were 

asked about how they planned their programs of work; how their plans evolved 

during the program year; and what they offered to clients with respect to the 

topics addressed, the delivery modes used, and their sources for content expertise. 

■	 Survey of senior state managers. SEA managers were surveyed about their 

state’s technical-assistance needs and what was provided by the Centers 

(including their RCC and any CCs with whom their state had worked). 

■	 Expert panel review. The same sample of major and moderate projects was 

reviewed for quality by a panel of experts. Content experts were recruited and 

trained to use standard criteria to rate the technical quality of the sampled Center 

projects on the basis of a review of all project materials. 

■	 Survey of project participants. A representative sample of clients who had 

participated directly in the evaluation’s purposive sample of major and moderate 

Center projects furnished descriptive information, through surveys, on the 

activities and resources that the project had delivered to them. These clients 

included individuals working at the state level who had participated in RCC or 

CC projects, and RCC employees who were among the clients of CC projects. 

They rated the relevance and usefulness of the sampled projects. 

Center Operations 

In designing the Center program, ED established structures and expectations for the 

functioning of the Centers. Key features of the design, emphasized in ED’s Notice Inviting 

Applications, were a requirement for needs assessment in consultation with clients, a focus on 

technical assistance with state responsibilities in school and district improvement, and the 

specialized roles of RCCs and CCs. The Centers’ work from 2006-07 to 2008-09 conformed to 

the program’s requirements in each of these respects. The barriers to technical assistance that 

Centers most often reported were staff turnover in state agencies and limitations on the CCs’ 

scope of work. 

A key expectation of the Centers was to organize their plans around the priorities and 

needs of client organizations. At the start of each program year, the Centers were required to 

deliver a management plan to ED outlining the program of technical assistance they planned to 

provide. Across years, Centers used a range of methods to assess needs and plan technical 

assistance with their clients. Among RCCs, there was a shift away from conducting surveys for 

needs assessment while maintaining frequent interaction with states as a means of learning about 

needs: all 16 RCCs reported assessing needs for 2008-09 through ongoing communication with 

state staff (an increase from 15 RCCs in 2006-07); 15 had a designated liaison to the SEA on 

staff in 2008-09 (up from 13 in 2006-07); 8 of the 16 conducted surveys (down from 11). Across 

years, all 5 of the CCs reported forming their work plans with RCC input acquired through 

ongoing communication; in 2008-09 all 5 CCs reported surveying RCCs (up from 4 of the 5 in 

xviii 
2006-07). In addition, all 5 CCs reported learning about state needs for 2008-09 through ongoing 

interaction with states as well as through communication with RCCs. 

Centers were expected to show responsiveness to needs and requests for technical 

assistance but might not be in a position to respond to every client request. In each year, more 

than half of the Centers reported that they had turned down a client request for assistance, a 

situation that was handled differently by RCCs and by CCs. The number of Centers that reported 

having turned down one or more requests was 12 of the 21 in 2006-07, 13 of 21 in 2007-08, and 

14 of 21 in 2008-09. Among Centers that declined any client request for assistance in 2008-09, 

RCCs most often reported substituting a different type of assistance (7 of the 10 RCCs that 

turned down work used this strategy), but none of the CCs reported doing so. The reason most 

commonly reported by RCCs was that a request fell outside their legitimate scope of work (5 

RCCs vs. 1 CC). CCs reported more concerns with the requests fitting the Center’s priorities (2 

of the 4 CCs that turned down work vs. 2 of the 10 RCCs) or the Center’s capacity (2 of the 4 

CCs that turned down work vs. 2 of the 10 RCCs). 

The Centers placed a priority on assistance with the state role in supporting improvement 

in struggling schools and districts. In every year of the evaluation, on the inventories completed 

by Centers that grouped their technical assistance activities into projects and categorized projects 

into 23 topics, the most common topic for all Center projects was “components of effective 

systems of support—state, district, school,” a topic that included but was not limited to statewide 

systems of support and school support teams. Among all projects on the Center’s inventories, 19 

percent in 2006-07, 25 percent in 2007-08, and 21 percent in 2008-09 addressed the topic of 

systems of support, which in each year was more than twice as many as any other topic. 

Although the two types of Centers each retained a focus on activities distinctly associated 

with the original program design, their ways of working became more similar over the years. The 

guidance given by ED through the Center grant competition and afterwards laid out a particular 

structure for the Centers’ work: RCCs would specialize in interactions with state clients while 

CCs would specialize in activities that required a content focus. The most common activity 

found in sampled RCC projects was “ongoing consultation and follow up” (82 percent in 2006-

07; 93 percent in 2007-08; 91 percent in 2008-09); in CC projects, it was “research collections 

and synthesis” (74 percent in 2006-07, 85 percent in 2007-08, and 77 percent in 2008-09), while 

fewer RCC projects included this activity (53 or 54 percent in each year) (exhibit ES.1). In 2008-

09, in a departure from past CC practice, a majority of sampled CC projects (62 percent) 

included ongoing consultation and follow-up. 

xix 
 

 

 

 

 

 

 

Exhibit ES.1. Sampled RCC and CC projects by types of activities and 

resources, by year 

Percent of sampled RCC 

projects 

Percent of sampled CC 

projects 

Activities and resources 

2006-07 

(n=96) 

2007-08 

(n=96) 

2008-09 

(n=93) 

2006-07 

(n=27) 

2007-08 

(n=26) 

2008-09 

(n=26) 

Ongoing consultation and followup 

82% 93% 91% 22% 15% 62% 

Research collections and 

syntheses 54 53 53 74 85 77 

Engagement of participants in 

project planning 45 63 59 30 31 27 

Training events 43 55 59 37 50 50 

Task force meetings and work 50 58 56 7 8 8 

Conferences 27 35 40 63 42 38 

Support development of a formal 

plan to implement a program or 

policy 

19 32 31 7 15 23 

EXHIBIT READS: For the 2006-07 program year, 82 percent of sampled Center projects included ongoing 

consultation and follow-up. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. In addition to serving 

as resource material for the expert reviewers, these cover sheets were coded by the evaluation team. 

The delivery of technical assistance depended on the Centers working effectively with 

their clients. Both RCCs and CCs described the barriers they perceived as having impeded their 

assistance to states. Turnover in staff within state offices or intermediary units was reported by 

both types of Centers as a barrier to achieving their objectives in assisting states (10 of 16 RCCs 

and 3 of 5 CCs). Turnover at the leadership level was a reported barrier for 8 RCCs and 3 CCs. 

Three of the 5 CCs reported “a state’s most important priorities for assistance fell outside the 

Center’s scope of work,” as a barrier; they indicated that some states wanted help with topics that 

went beyond their own assigned substantive focus. 

Under the two-tiered Center program design, RCCs and CCs were expected to work 

together to serve state clients. Among RCC projects, 48 percent had a CC contribution (of 

materials, in-person assistance, or advice) in 2006-07, 32 percent in 2007-08, and 47 percent in 

2008-09. Among CC projects, the percent incorporating some RCC contribution was 37 percent, 

38 percent, and 42 percent across the years. The extent to which RCCs and CCs drew on the 

other as substantive partners in delivering assistance increased in 2008-09: the percent of 

sampled RCC projects in which CCs delivered technical assistance went up from 18 percent in 

2006-07 to 30 percent in 2008-09, and the percent of sampled CC projects in which the RCCs 

delivered technical assistance rose from 11 percent in 2006-07 to 38 percent in 2008-09. 

xx 
 

With 16 RCCs and 5 CCs all charged with working with the other type of Center, 

coordination varied across the different pairs of an RCC and a CC. For example, while 15 of the 

16 RCCs reported teaming up with at least one CC to provide technical assistance to states, 14 of 

them reported teaming up with one of the CCs but 7 of them reported doing so with another of 

the CCs. In addition, CCs were expected to provide assistance to RCCs, and the barrier most 

often reported by both types of centers to have impeded CCs’ achievement of their technical 

assistance objectives with RCCs was that “RCCs’ most important priorities for assistance fell 

outside the CC’s scope of work” (reported as a barrier by 7 of 16 RCCs and 4 of 5 CCs). 

Extent to Which Centers Addressed State Priorities 

The perceptions of senior managers in state education agencies, who are involved in 

identifying state needs and priorities for technical assistance, provide a relevant perspective on 

the outputs of the program. Because the Centers had a mandated focus on the states, the extent to 

which state managers perceived that Center technical assistance served state purposes is one way 

of gauging the program’s attainment of its objectives. Capacity building for states is also a focus 

of this evaluation, because it was prominent as a goal for the Comprehensive Centers program. 

The first priority for all Centers, articulated by ED in the Notice Inviting Applications, included 

“helping states build the capacity to help school districts and schools implement NCLB 

provisions and programs.”8 

An increasing percentage of state managers (weighted) over three years reported that the 

Centers’ technical assistance “served the state’s purposes completely” (exhibit ES.2). Thirty-six 

percent of the state managers, weighted, chose this response for 2006-07, 47 percent for 2007-08, 

and 56 percent for 2008-09. 

Among the managers who said their state’s purposes were not completely served, a larger 

proportion in each year reported that they wanted more interaction with the Centers. The percent 

of weighted state managers saying, “Center staff are not able to spend as much time working 

with the state as we would like” was 17 percent of those who did not say the state’s purposes 

were “completely” served in 2006-07. The corresponding figures for subsequent years were 27 

percent in 2007-08 and 43 percent in 2008-09.9 (These respondents were 9 percent, 10 percent, 

and 16 percent, respectively, of all state managers, weighted.) 

8 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106),
 

32585. 

9

 The percentage of state managers reporting that their state’s purposes were not completely served varied by year. 

Thus, for the follow-up question (“reasons the technical assistance has been less helpful than it might be”), 

comparison of percentages from year to year may include variation in responses over time as well as changes in
 

respondents addressing the question.
 

xxi 
 

 

Exhibit ES.2. Extent to which technical assistance from the Centers served 

state purposes, as judged by state managers, by year 

 

 

 

It did not serve the state’s Percent of state managers (weighted) 

purposes
 

100
 

36

47

56

52

40

36

6 7 

6 6 6 3 It was a good start, but some 

important priorities were 

not addressed 80 

It was a good start 

60 

40 

It served the state’s 

purposes completely 

20 

0 

2006-07 2007-08 2008-09 

(n=56) (n=57) (n=54) 

EXHIBIT READS: In 2006-07, 36 percent of state managers, weighted, reported that Center 

technical assistance served the state’s purposes completely. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally 

represented in instances where more than one manager from a state responded. 

State managers in a high proportion of states reported that Centers delivered assistance on 

“statewide systems of support or school support teams,” which was the most widespread priority 

among areas of technical assistance for state managers. In 2007-08 and 2008-09, more than 

90 percent of state managers, weighted, identified this area of state responsibility as a major or 

moderate priority for technical assistance (95 percent in 2007-08, 94 percent in 2008-09). Of this 

group of state managers that reported this priority10, more than 90 percent reported that the 

Centers delivered assistance related to this responsibility (94 percent in 2007-08, 91 percent in 

2008-09). 

Looking at state reported capacity building across areas of major or moderate state 

priority for technical assistance, the highest percentage of state managers reported Center 

assistance to have expanded their agency’s capacity to a “great” or “moderate” extent in 

statewide systems of support or school support teams11 (72 percent in of those who rated the area 

10 The subgroup of state managers who identified a particular area of state responsibility to be a major or moderate 

priority for technical assistance varied by year. Thus, for the follow-up question about the priority areas in which 

states received technical assistance from Centers, a comparison of the percentages from year to year may include 

variation in responses over time as well as changes in respondents addressing the question. 11 Percentages are based on the state manager respondents who rated each area as major or moderate technical 

assistance priority, which varied by year. Thus, for the question about state capacity building, comparison of 

xxii 
 

as a state priority in 2007-08 and 82 percent in 2008-09) (exhibit ES.3). The next-highest in both 

2007-08 and 2008-09 was “development or dissemination of research-based curriculum, 

instruction, or professional development programs in academic subject(s)” (64 percent in 2007-

08 and 77 percent in 2008-09). In both years, the lowest was “administration of supplemental 

educational services and choice provisions” (44 percent and 39 percent, respectively, of those 

who rated the area as a priority), which was an area rated as a priority by the fewest state 

managers (49 percent and 48 percent, respectively). 

Exhibit ES.3. Extent to which Center assistance expanded state capacity in 

priority areas, as judged by state managers who rated the area as 

a major or moderate technical assistance priority, by year 

Percent reporting 

capacity expanded by a 

great or moderate extent 

Area of state responsibility under NCLB 2007-08 2008-09 

Statewide systems of support or school support teams 

(n=56, n=50)
 72% 82%

Policies and practices for English language learners 

(n=43, n=40) 59 73

State assessment and accountability systems 

(n=42, n=39) 57 59

Development or dissemination of research-based 

curriculum, instruction, or professional development 

programs in academic subject(s) 

(n=41, n=39) 

64 77

Assistance with educators’ use of assessment data
 

(n=37, n=36) 62 61

Monitoring compliance with NCLB requirements 

(n=35, n=30) 57 57

Administering supplemental educational services (SES) 

and choice provisions (n=25, n=26) 44 39

Communication with parents or the public 

(n=25, n=26) 48 50

NOTE: Percentages are based on the state manager respondents who rated each area a major or moderate 

technical assistance priority, which varied by year. State managers who chose the response, “Does not 

apply, or not able to judge,” were included in the denominator of the percent calculation. 

EXHIBIT READS: For the 2007-08 program year, among the 56 state managers (weighted) who reported 

that technical assistance in statewide systems of support or school support teams was a major or moderate 

priority for their state, 72 percent reported that technical assistance received from the Centers expanded the 

state’s capacity in this area to a great or moderate extent. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in 

instances where more than one manager from a state responded. 

percentages from year to year may include variation in responses over time as well as changes in respondents 

addressing the question. 

xxiii 
 

 

 

State managers reported that the Centers were one of multiple sources that they used for 

technical assistance, but over time they turned to the Centers for more purposes. The purposes 

for which states used the Centers more than other sources in each of the three study years were 

“to plan the initial steps in solving a problem” (reported as a purpose for Center technical 

assistance by at least 60 percent of state managers in each year) and “to develop the skills of 

SEA or intermediate education agency staff” (at least 55 percent of state managers in each year). 

In 2007-08 and 2008-09, Centers were reported by state managers to be the top source for two 

additional purposes: to help states complete tasks where they lacked resources (58 percent and 64 

percent, respectively) or expertise (49 percent and 53 percent, respectively). 

Ratings of Center Assistance 

To assess the technical assistance provided by the Center program, quality, relevance, 

and usefulness of a sample of Center projects were rated. All sampled projects were identified by 

the Centers as “major” or “moderate” in their level of effort, relative to other projects in the same 

Center. The projects were rated for technical quality by panels of experts with strong knowledge 

of the content or substantive focus of the specific projects they reviewed. Projects’ relevance and 

usefulness were rated by a sample of participants—state staff, intermediate agency staff, local 

educators working on behalf of the state, and RCC staff—who were the intended beneficiaries of 

the project and who had received at least some of the technical assistance the project provided. 

Quality was judged on three dimensions; relevance was assessed with eight survey items and 

usefulness with 11 survey items (exhibit ES.4). Each overall measure (relevance, usefulness, or 

quality) was calculated as the mean of ratings assigned to each item. The item-level ratings 

themselves were based on 5-point rating scales.12 

Based on the ratings, Center technical assistance was rated higher on each measure in 

each successive year, with program-wide average ratings in 2008-09 falling in a range between 

“moderate” and “high” for quality, and around “high” for relevance and usefulness (exhibit 

ES.4). On a scale of 1 to 5 with a 3 representing “moderate” and a 4 representing “high,” the 

program-wide average ratings for the sampled projects were 3.34 in 2006-07, 3.51 in 2007-08, 

and 3.57 in 2008-09 for technical quality, scored by panels of content experts. Program-wide 

average ratings for relevance, scored by participants, were 3.94 in 2006-07, 4.08 in 2007-08, and 

4.15 in 2008-09. Average usefulness ratings for the program were 3.69 in 2006-07, 3.95 in 2007-

08, and 3.96 in 2008-09, also scored by participants.13 

12 Efforts were made to develop parallel wording and rubrics that would result in similar gradations between rating 

levels (e.g., very high vs. high vs. moderate) across the three measures. However, given the different content of each 

set of items within the three measures and the different contexts for the ratings (experts who underwent training for 

the rating process and reviewed identical packages of materials vs. survey respondents who typically participated in 

different subsets of project activities), the ratings across the three measures are not directly comparable. 13 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 

to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 

and each project sampled from a Center contributed equally to the Center mean. 

xxiv 
Exhibit ES.4. Quality, relevance, and usefulness items 

xxv 

From expert panel scoring From project participant surveys 

Technical quality Relevance Usefulness 

Reviewers were directed to 

assign a score to each 

dimension and to include the 

basis for their ratings on the 

rating form, including the specific 

artifacts on which their score was 

based. The three dimensions 

are: 

a. Demonstrated use of the 

appropriate documented 

knowledge base – to include 

an accurate portrayal of the 

current state of information 

with prominence to those 

with the most 

accurate/rigorous evidence 

b. Fidelity of application of the 

knowledge base to the 

products and services 

provided – materials are 

consistent with the 

best/accurate information 

available and the 

presentation adequately 

conveys the confidence of 

the information 

c. Clear and effective delivery – 

information is well organized 

and written and accessible to 

the intended audience for 

easy use 

Based on your experience, to 

what degree was this set of 

activities and resources relevant 

to your work, in each of the 

following respects? 

a. Addressed a need or problem 

that my organization faces 

b. Addressed an important 

priority of my organization 

c. Addressed a challenge that 

my organization faces related 

to the implementation of NCLB 

d. Provided information, advice, 

and/or resources that could be 

directly applied to my 

organization’s work 

e. Addressed our particular state 

context 

f. Addressed my organization’s 

specific challenges (e.g., 

policy environment, leadership 

capacity, budget pressures, 

local politics) 

g. Provided information, advice, 

and/or resources that could be 

used to guide decisions about 

policies, programs, or 

practices 

h. Highlighted the implications of 

research findings (or 

information about best 

practice) for policies, 

programs, or practices 

Based on your experience, to 

what degree was this set of 

activities and resources useful to 

you, in each of the following 

respects? 

a. Provided resources that were 

easy to understand and easy 

to use 

b. Employed an appropriate 

format (e.g., a work group, a 

conference, individual 

consultation, written products) 

c. Provided adequate opportunity 

to learn from colleagues in 

other states 

d. Included adequate follow-up to 

support the use of new 

information and resources 

e. Were timely 

f. Helped my organization solve 

a problem 

g. Helped my organization 

maintain or change a policy or 

practice 

h. Helped my organization take 

the next step in a longer-term 

improvement effort 

i. Provided my organization with 

information or resources that 

we will use again 

j. Helped my organization 

develop a shared expertise or 

knowledge-base 

k. Helped individuals in my 

organization to develop skills 

that they will use again 
 

Exhibit ES.5. Mean ratings of technical quality, relevance, and usefulness, 

by center type and by year 

 

 

 

 

 

Technical quality Relevance Usefulness 

2006-

07

2007-

08 

2008- 2006- 2007- 2008- 2006- 2007- 2008-

09 07 08 09 07 08 09 

All Centers (N=21) 3.34 3.51 3.57 3.94 4.08 4.15 3.69 3.95 3.96 

All RCCs (N=16) 3.21 3.41 3.52 3.99 4.18 4.15 3.71 3.99 3.94

All CCs (N=5) 3.73 3.86 3.72 3.78 3.96 4.17 3.65 3.84 4.01 

Difference of RCC 

and CC means -0.52†

 -0.45† -0.20† 0.21† 0.22† -0.02 0.06 0.15† -0.07

Pooled standard 0.41 0.41 0.37 0.34 0.22 0.24 0.34 0.23 0.27 deviation (all Centers) 

Ratio of difference in 

means to pooled -1.28 -1.09 -0.55
 0.62 1.00 -0.08 0.18 0.64 -0.26 

standard deviation 

NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” rating is the mean of the 

ratings for the three quality dimensions. A notation of †

 indicates that the difference in the mean ratings between the 

CCs and RCCs within that year is at least one-half of one pooled standard deviation in the rating. 

EXHIBIT READS: In 2006-07 among the 21 Centers, the mean technical quality rating was 3.34. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 

ratings; each project contributed equally to Center ratings; and each Center contributed equally to cross-Center 

ratings. 

Given that the RCC and CC roles and activity emphases differed, the evaluation looked at 

variation across Center types. The mean ratings for types of Centers, based on their sampled 

projects, showed the CCs with higher mean ratings than RCCs for the quality of their sampled 

projects in all three years although RCCs’ average quality ratings were higher in each successive 

year (exhibit ES.5). The RCCs had higher mean ratings than CCs for the relevance of their 

sampled projects in 2006-07 and 2007-08 although the average ratings of relevance for CCs went 

up each year. There were no consistent differences in mean ratings of usefulness across types of 

Centers. 

The evaluation also looked at the relationships between the three measures: quality, 

relevance, and usefulness. It was reasoned that the content experts rating quality and the 

participants rating relevance and usefulness might be better able to judge different aspects of a 

Center project. On this rationale, content experts rated the projects for their technical quality, and 

participants rated the projects for relevance and usefulness. An examination of the associations 

among the three dimensions was conducted by calculating correlation coefficients.14 Such a 

statistic indicates the strength and direction of a linear relationship between two factors. A 

correlation coefficient can vary from positive 1.00 (indicating a perfect positive relationship), 

14 For this analysis, the evaluation team used Spearman’s rank order correlation, as this non-parametric rating is the 

appropriate statistical function to describe correlations between two variables where the values of the variables are 

not normally distributed and are on a scale (such as ratings). 

xxvi 
through zero (indicating the absence of a relationship), to negative 1.00 (indicating a perfect 

negative relationship). If the correlation is statistically significant (p <.05), we can have strong 

(95 percent) confidence that what we calculated is not due to chance. 

In every year, ratings of quality were unrelated to ratings of relevance and usefulness, 

although relevance and usefulness ratings were highly correlated with each other within each of 

the three data collection years. The correlation coefficient for relevance and usefulness was 

+0.84 for 2006-07, +.79 for 2007-08, and +.83 for 2008-09. This indicates that the extent to 

which participants rated the projects as relevant was associated with how they deemed the 

project to be useful to their agency. These coefficients were all statistically significant at p<.05. 

On the other hand, the results indicated correlations ranging from -0.12 to +0.04 between quality 

and relevance, and from -0.09 to +0.07 between quality and usefulness. Because these 

coefficients are not statistically significant we cannot be sure that they are different from zero 

(no relationship). In other words, the extent to which a project faithfully reflected the knowledge 

base on a topic and provided appropriate caveats about the quality of its evidence was unrelated 

to the extent to which participants deemed that project relevant or useful to their agency. 

Given the variation in ratings across Centers, additional analyses were conducted to 

explore whether there were consistent patterns between ratings and the particular features of the 

projects. Such information could provide suggestions for possible program improvement if there 

were consistent relationships. Quality ratings in 2008-09 were higher for RCC projects that 

included CC contributions of materials or in-person help than projects that the RCCs completed 

without CC contributions (3.72 vs. 3.39), although this was not the case in earlier years. In 

addition, quality ratings were higher in 2008-09 for projects that had been reviewed by CCs 

(3.83 vs. 3.46) and by outside experts (3.73 vs. 3.42) for quality assurance as opposed to projects 

that had not been reviewed in each of these ways (a project-level feature that was studied only in 

that year of the evaluation). In other analyses of project-level variation, projects that differed 

from each other in the activities they encompassed or the topics they addressed did not show 

differences in ratings of quality, relevance, or usefulness that were consistent across the three 

years. 

On the other hand, more consistent differences were found in ratings of relevance and 

usefulness awarded to projects by different types of participants. Higher ratings were awarded by 

those participants who had been involved in determining the project goals or design than by 

participants not involved in this way, and by those who had spent more time in project activities 

(i.e., 6 or more days) as compared to participants who had spent five days or less (these 

differences were statistically significant, with p <.01 for both relevance and usefulness). For 

2007-08 and 2008-09, also, each type of Center targeted its assistance more successfully to 

participants who worked in one type of agency, compared with participants who worked in other 

types of agencies: specifically, RCC projects were rated higher by participants from SEAs than 

participants from intermediate or local education agencies or schools; CC projects were rated 

higher by RCC staff than by SEA staff (statistically significant differences, with p<.05 for both 

relevance and usefulness). 

xxvii 

1. Introduction and Background 

The Comprehensive Technical Assistance Centers are authorized under the Educational 

Technical Assistance Act of 2002 to provide technical assistance for implementation of the No 

Child Left Behind (NCLB) Act at the state, district, substate region, and school levels for the 

purposes of “improving academic achievement, closing achievement gaps, and encouraging and 

sustaining school improvement” (Section 203). The law authorized the Secretary of Education to 

award “not less than 20 grants to local entities, or consortia of such entities, with demonstrated 

expertise in providing technical assistance and professional development in reading, 

mathematics, science, and technology” (Section 203). Grants were awarded to 21 Centers to 

serve different geographic regions across the United States from FY2006 through FY2010. 

The same law authorizing these technical assistance centers also mandated that the 

Comprehensive Technical Assistance Centers undergo independent evaluation under the 

direction of the National Center for Education Evaluation and Regional Assistance (NCEE) in 

the Institute of Education Sciences of the U.S. Department of Education (ED). The legislation 

indicated that the evaluation should “include an analysis of the services provided...[and] the 

extent to which each of the comprehensive centers meets the objectives of its respective plan, 

and whether such services meet the educational needs of State educational agencies, local 

educational agencies, and schools in the region.” 

This is the final report of a multi-year evaluation conducted for NCEE by Branch 

Associates, Inc., Decision Information Resources, Inc., and Policy Studies Associates, Inc. This 

final report presents an overall description of how the system of technical assistance centers has 

operated from July 2006 to June 2009, covering the period from the second to the fourth program 

year (out of the five years), as part of the evaluation to inform the current grants as well as the 

grant recompetition. 

This introductory chapter provides background information on ED’s purposes and design 

for the Center system. It also describes key state responsibilities included in the No Child Left 

Behind (NCLB) Act with which the Centers were expected to provide assistance. The chapter 

concludes with a description of the major evaluation questions. 

The Center Program 

The current Center program represented a departure from the previous program with a 

new design for targeting services replacing the 15 existing Comprehensive Regional Assistance 

Centers established under the Elementary and Secondary Education Act of 1994 (ESEA). The 

1994 reauthorization of ESEA had charged the previous set of 15 Centers with delivering 

assistance to support standards-based reform as envisioned in other sections of the 1994 law. 

They were to provide training and technical assistance to states, local education agencies 

(LEAs), schools, tribes, community-based organizations, and other ESEA grantees related to 

several areas of local responsibility. These included: (1) improving the quality of instruction, 

curricula, assessments, and other aspects of school reform; (2) implementing effective 

1 
 

 

 

 

 

 

schoolwide programs; and (3) meeting the needs of children, especially children in high-poverty 

areas, migrant children, immigrant children, limited-English-proficient children (LEP), neglected 

or delinquent children, homeless children, Indian children, and children with disabilities [Section 

13102 (a)(1)(A-L)]. In short, the previous Centers’ mandate included a focus on a number of 

aspects of local educational practice. 

A previous evaluation15 found the majority of the technical assistance was targeted to 

school districts and schools. The study, based on surveys of clients conducted in 1999, found that 

the majority of direct participants in major Center training and technical assistance activities 

were school staff (either teachers or principals), and that the majority of clients who arranged for 

services from the Centers were also based in school districts or schools. At the state level, among 

those state education agency (SEA) staff members who had received assistance, 64 percent 

reported that the Centers had improved the ability of their SEA to provide assistance to districts 

and schools. The report noted that higher ratings for Centers were associated with more intensive 

technical assistance, based upon survey findings. 

The current design of the Center program reflects changes from the design of the 

predecessor program. The Educational Technical Assistance Act of 2002 authorized a new group 

of Centers to provide technical assistance for NCLB implementation at the state, district, substate 

region, and school levels for the purposes of “improving academic achievement, closing 

achievement gaps, and encouraging and sustaining school improvement” (Section 203). It also 

gave ED discretion to “establish priorities” for the Centers (Section 207). 

In making its design choices, ED set up an advisory process to identify priorities for the 

new Centers: in 2004 the Secretary of Education appointed 10 Regional Advisory Committees 

that would conduct needs assessments in their regions and make recommendations regarding 

technical assistance. The committees said SEAs needed help making better use of scientifically 

based research in decisionmaking, and that strengthening SEAs’ capacity to serve local school 

districts was critical to the success of NCLB reforms, according to a synthesis of their 

recommendations.16 

Following this effort, ED identified states as the Centers’ primary client base, although 

the program had in the past served local clients, as described above. ED charged the Centers to 

work in new ways to expand and strengthen states’ capacity to deliver assistance to schools and 

districts. The new program would also consolidate and expand responsibilities for Center 

assistance in that it would replace the Regional Technology in Education Consortia, the 

Eisenhower National Clearinghouse for Mathematics and Science Education, and the Regional 

Mathematics and Science Education Consortia. 

15 U.S. Department of Education, Office of the Under Secretary, Planning and Evaluation Service, Elementary and 

Secondary Education Division. (2000). Comprehensive Regional Assistance Centers Program: Final Report on the 

Evaluation(Volume I). Washington, DC: Author.
 16 Sheekey, A., Cymrot, D.J., and Fauntleroy, C. (2005, March). A Report to the U.S. Department of Education:
 

Overview and Synthesis of the Regional Advisory Committee Reports on Educational Challenges and Technical
 

Assistance Needs. Alexandria, VA: The CNA Corporation.
 

2 
 

 

Structure of the Center Program 

To implement the provisions of the Educational Technical Assistance Act of 2002, ED 

created a new Center structure with two tiers of technical assistance. The Notice Inviting 

Applications detailed the design. Under the new design, the Secretary of Education would award 

grants to 21 Centers, each tasked with “provid[ing] technical assistance to States as States work 

to help districts and schools to close achievement gaps in core content areas and raise student 

achievement in schools. To accomplish this goal, ED stipulated that applicants had to “propose a 

plan of technical assistance specifically focused on helping States implement the provisions of 

NCLB applicable to States, and helping States build the capacity to help school districts and 

schools implement NCLB provisions and programs.”17 

While the overall goal of assisting states with NCLB implementation was common to all 

21 Centers, the two-tiered technical assistance designed by ED created distinct roles for the two 

types of Centers. These are described next. 

Distinct RCC and CC Roles 

Within the 21 Centers, ED constituted 16 Regional Comprehensive Centers (RCCs) and 

five Content Centers (CCs). By design, RCCs and CCs were given different roles and functions 

in a system of technical assistance (exhibit 1.1). RCCs, embedded within distinct geographic 

regions across the United States and territories, would deliver technical assistance to the states 

and territories in their region, addressing their needs and building their capacity to assist their 

districts and schools. Meanwhile, each CC would take responsibility for synthesizing knowledge 

from the research and promising practices within a particular substantive area. The CCs would 

build the capacity of the RCCs by providing research-based information, products, guidance, and 

knowledge on key topics. The CCs would also work with RCCs to provide technical assistance 

to states. In turn, each state would help its districts and schools meet NCLB requirements. 

17 Notice Inviting Applications for New Awards for Fiscal Year 2005. Federal Register. (2005, June 3). 70(106), 

32585. 

3 
Exhibit 1.1. Center program design 

 

RCC CC 

State 

Other sources 

of technical 

assistance 

LEAs and 

schools 

Elaborating on the functions of RCCs, ED required them to work directly with states to 

“provide frontline assistance.”18 ED mandated that RCCs provide states with ongoing assistance 

and training that would draw from a range of knowledge sources, including but not limited to 

CCs; provide CCs with information about promising practices; convene states for collaboration; 

and deliver information based on both research and best practice. The specific list of RCC 

responsibilities included the following:19 

■	 “Working closely with each State in its region on an ongoing basis” 

■	 “Linking States with the resources of Content Centers, Department staff, Regional 

Educational Laboratories, The What Works Clearinghouse, and other entities” 

■ “Suggesting sources of appropriate service providers or assistance for State 

activities that are not within the core mission of the centers” 

■	 “Assisting State efforts to build statewide systems of support for districts and 

schools in need of improvement” 

■	 “Working to identify, broker, leverage, and deliver information, resources and 

services from the Content Centers and other sources” 

■	 “Convening in partnership with Content Centers and others, as appropriate, States 

and districts to receive training and information on best practices and researchbased improvement strategies” 

■	 “Providing guidance and training on implementation of requirements under 

NCLB and other related Federal programs” 

■	 “Facilitating collaboration at the State level to align Federal, State, and district 

school improvement programs” 

18 Ibid., 32585. 

19 Ibid., 32585-6. 

4 
■	 “Helping Content Centers to identify, document, and disseminate emerging 

promising practices” 

The Notice Inviting Applications portrayed CCs as a central source of readily accessible 

knowledge, resources, and tools. Each CC was designed to consolidate in-depth knowledge in 

one of five key content areas: Assessment and Accountability, Instruction, Teacher Quality, 

Innovation and Improvement, or High Schools. The types of knowledge specifically mentioned 

included research, scientifically valid practices, and promising practices. The degree of emphasis 

on research and scientifically valid practice was heightened in this redesign of the Center 

program. This was consistent with NCLB, which stated that scientifically based research must 

inform local practice. 

Within their content areas ED tasked CCs to:20 

■	 “Identify, organize, select and translate existing key research knowledge...and 

communicate the information in ways that that are highly relevant and highly 

useful to State and local level policy makers and practitioners” 

■	 “Benchmark State and district practices for implementing NCLB provisions and 

school improvement interventions...and identify promising approaches that can 

be shared with States and districts” 

■	 “Convene States and districts, researchers and other experts to learn from each 

other about practical strategies for implementing NCLB provisions and programs” 

■	 “Train Regional Center staff on what is known about scientifically valid practices 

and programs” 

■	 “Collaborate with Regional Centers to address specific State requests for 

assistance” 

■	 “Communicate to the field...Department guidance related to the center’s content 

focus” 

■	 “Design needs assessment and data analysis tools that States and districts can use 

to benchmark their programs and progress” 

Awards to Centers 

At the conclusion of the competition, new Regional Comprehensive Centers were located 

in 16 regions of the United States, covering all U.S. states and territories, and the five new 

Content Centers had also entered into cooperative agreements with ED (exhibit 1.2). Of the 16 

RCCs, there were four that served only their respective state: New York, Texas, California, and 

Alaska. The remaining 12 Centers served from two to seven states and other jurisdictions. The 

non-state jurisdictions that the Centers were to serve were the following: the District of 

Columbia, Puerto Rico, the Virgin Islands, American Samoa, Commonwealth of the Northern 

Mariana Islands, Federated States of Micronesia [Chuuk, Kosrae, Pohnpei, and Yap], Guam, 

Republic of the Marshall Islands, and Republic of Palau. Throughout this report, the term “state” 

will be defined to include the 50 states as well as these other jurisdictions. A full list of the 

grantees and subgrantees appears in appendix A of this report. 

 

20 Ibid., 32586-7. 

5 
 

 

Center funding for each of the three years that are the subjects of this evaluation totaled 

$56.3 million, with variation across Centers (exhibit 1.2). In 2008-09 (FY 2008) individual 

Regional Centers’ funding ranged from a low of $860,000 in Alaska and Pacific to a high of 

$6,039,909 for the California Comprehensive Center. The funding for each RCC was driven by a 

formula based on the region’s total population and its number of poor children ages 5-17. 

Average funding across all RCCs was $2,854, 047. 

Content Center funding for 2008-09 ranged from $1,518,400 for the Assessment and 

Accountability Center and Center on Innovation and Improvement, to $2,518,400 for the Centers 

on Instruction, Teacher Quality, and High Schools. The latter group of Centers was co-funded in 

each year with an allocation of $1 million each from the Special Education Technical Assistance 

and Dissemination Act, authorized under the Individuals with Disabilities Education Act. The 

average funding across CCs was $2,046,096. 

Background on the State Role in NCLB 

A basic premise of the Center program as designed by ED was that NCLB assigned many 

tasks to states. The background section of the Notice itemized the following NCLB requirements 

for states: 

...set standards for student performance, implement statewide testing and accountability 

systems to measure school and student performance toward achieving those standards, 

adopt research-based instructional and program improvements related to teaching and 

learning in the classroom, ensure that all teachers in core subject areas are highly qualified, 

and improve or ultimately restructure schools that are consistently low-performing.21 

With respect to consistently low-performing schools and also low-performing districts, 

NCLB mandated that all states establish and sustain statewide systems of support and 

improvement for school districts and schools identified for improvement under NCLB. Districts 

and schools identified for improvement must receive assistance from support teams, institutions 

of higher education, and regional service centers in the state. This “statewide system of support” 

(SSOS) must include individuals who were knowledgeable about research and practice on 

teaching and learning and who could develop and implement comprehensive improvement 

strategies. “State support teams” (SSTs) were required to help schools plan for improvement and 

to evaluate the effectiveness of school personnel. The NCLB legislation also provided that these 

support teams should receive technical assistance from Comprehensive Centers and others.22 

21 Ibid., 32584. 22 No Child Left Behind, Title I, Part A, § 1117 (a) (1). 

6 
Exhibit 1.2. Center funding, by year 

Centers States FY 2006 FY 2007 FY 2008 

Total $56,256,713 $56,256,750 $56,256,750 

Regional Centers 

Alaska Comprehensive 

Center 

Appalachia Regional 

Comprehensive Center 

California Comprehensive 

Center 

Florida and Islands Regional 

Comprehensive Center 

Great Lakes East 

Comprehensive Center 

Great Lakes West Region 

Comprehensive Center 

Mid-Atlantic Comprehensive 

Center 

Mid-Continent 

Comprehensive Center 

New England Comprehensive 

Center 

CT, MA, ME, NH, RI, VT 

New York Comprehensive 

Center 

North Central Comprehensive 

Center 

Northwest Regional 

Comprehensive Center 

Pacific Comprehensive 

Center 

Southeast Comprehensive 

Center 

Southwest Comprehensive 

Center 

Texas Comprehensive Center 

AK 

KY, NC, TN, VA, WV 

CA 

FL, Puerto Rico, Virgin 

Islands 

IN, MI, OH 

IL, WI 

DE, MD, NJ, PA, DC 

AR, KS, MO, OK 

NY 

IA, MN, ND, SD, NE 

ID, MT, WY, WA, OR 

HI, American Samoa, 

Northern Mariana Islands, 

Federated States of 

Micronesia, Guam, 

Marshall Islands, Palau 

AL, GA, SC, LA, MS 

AZ, UT, CO, NV, NM 

TX 

 

$850,000

3,829,927

5,912,997

3,788,289

3,592,771

3,702,196

3,388,147

2,111,226

1,644,795

2,886,970

1,286,458

1,630,818

850,000

4,120,988

2,491,327

3,939,324

$860,000 

3,912,131 

6,039,909 

3,869,599 

3,669,885 

2,448,739 

3,460,868 

2,156,541 

1,680,099 

2,948,935 

1,314,108 

1,665,821 

860,000 

4,209,438 

2,544,800 

4,023,877 

$860,000 

3,912,131 

6,039,909 

3,869,599 

3,669,885 

2,448,739 

3,460,868 

2,156,541 

1,680,099 

2,948,935 

1,314,108 

1,665,821 

860,000 

4,209,438 

2,544,800 

4,023,877 

Content Centers 

Assessment and Accountability Comprehensive Center 

Center on Innovation and Improvement 

Center on Instruction 

National Comprehensive Center for Teacher Quality 

National High School Center 

 

$1,446,096 

1,446,096 

2,446,096 

2,446,096

2,446,096 

$1,518,400 

1,518,400 

2,518,400 

2,518,400 

2,518,400 

$1,518,400 

1,518,400 

2,518,400 

2,518,400 

2,518,400 

NOTE: The figure shown in this table for Great Lakes West in FY 2006 is the sum of the six-month "start up" award 

($1,243,322) and the FY 2006 award ($2,458,844) as both were awarded in FY 2006. 

SOURCE: U.S. Department of Education 

7 
Research Questions Addressed in This Report 

The current evaluation takes a global look at the Center program as designed by ED, 

tracking the ways in which the Centers interacted with clients (both states and other Centers) 

over three program years. The first year of data collection was July 2006 through June 2007,the 

second year of program funding. The evaluation was designed to complete its data collection in 

2008-09, covering three of the five program years. 

The priorities for this study focused on the statute’s charge for the evaluation to provide 

“an analysis of the services provided...[and] the extent to which each of the comprehensive 

centers meets the objectives of its respective plan, and whether such services meet the 

educational needs ... in the region.” Among the following key research questions, the first 

focuses on an analysis of services provided and adherence to objectives; the second addresses the 

Centers’ performance in meeting state needs; and the third calls for more detailed assessment of 

the quality, relevance, and usefulness of Center technical assistance: 

1.	 How did the Regional Comprehensive Centers and Content Centers operate as part of the 

Comprehensive Technical Assistance Center program? 

■	 How did Centers develop, refine, and carry out their plans for technical 

assistance? How did they define their clients’ educational needs and priorities? 

■	 What were the objectives of the technical assistance the Centers offered? What 

kinds of products and services were provided by the Centers? 

■	 How did the Regional Comprehensive Centers and Content Centers coordinate 

their work? 

2.	 What was the performance of the Comprehensive Centers in addressing state needs and 

priorities? How did their performance change over the period of time studied? 

■	 How did the Centers’ state clients define their needs and priorities? 

■	 To what extent, as reported by states, did Center assistance expand state capacity 

to address underlying needs and priorities and meet the goals of NCLB? 

■	 To what extent did states rely on other sources of technical assistance besides the 

Centers? What were other sources of technical assistance that states used? How 

did the usefulness of Center assistance compare with the usefulness of assistance 

from other sources? 

3.	 To what extent was the assistance provided by the Centers of high quality, high relevance, 

and high usefulness? 

■	 Did the quality, relevance, or usefulness of Center assistance change over the 

period of time studied? 

8 
■	 What was the variation in the quality, relevance, and usefulness of Center 

assistance across types of projects and participants? 

Findings reported here address the Center program as a whole and also address each tier 

of Centers, RCCs and CCs, separately. Data gathered on individual projects were compiled and 

aggregated to the program level and by center type. Program-level aggregated findings reflect an 

average across all sampled projects, and center-type aggregated findings reflect the average 

across all RCC sampled projects or all CC sampled projects. Additional details are provided in 

chapter 2. 

Organization of This Report 

Following this introductory chapter, chapter 2 describes the study’s methods. Chapter 3 

addresses the first research question, describing Center technical assistance, procedures for needs 

assessment, and the functioning of the two-tiers of technical assistance. Chapter 4 addresses the 

second research question, describing the technical assistance priorities reported at the state level 

and the ways in which Centers were reported to have addressed these priorities and expanded 

state capacity. Chapter 5 addresses the third research question, reporting on the quality, 

relevance, and usefulness of selected Center projects as determined through expert review (for 

quality) and participant surveys (for relevance and usefulness). The report concludes with several 

appendices that provide additional technical notes, the materials used to collect data (surveys, 

protocols for Center interviews, and requests for materials from the Centers), and historical 

exhibits. 

9 

2. Study Design 

The evaluation team used six data sources in each of the three rounds of data collection to 

address the report’s research questions: documents produced by the Centers with assistance from 

the evaluation team, Center management plans, interviews with Centers, state manager surveys, 

expert panel reviews, and participant surveys (exhibit 2.1). Data were collected from all sources 

for the 2006-07, 2007-08, and 2008-09 program years to describe changes over time in the 

Center operations and assessments of Center’s technical assistance. This chapter describes those 

data sources and analytic procedures used in the analysis. 

For the first three questions, which pertained to the operations of the Centers, the data 

were drawn from the Centers themselves. The team used Center management plans as a data 

source regarding Center objectives. In the summers of 2007 and 2008 as well as spring 2010, the 

team conducted interviews with the Centers; closed-ended prompts were used in face-to-face 

interviews to gather self-report data systematically on Center objectives, procedures for needs 

assessment and planning, and interactions with their clients. The final interviews (2008-09) 

included structured questions to gather Center perspectives on changes in technical assistance 

work across program years and on quality assurance procedures. Two documentary sources were 

used as sources for descriptions of Center technical assistance: the project inventory forms and 

project cover sheets completed by the Centers with review and feedback from the evaluation 

team. The definition of a project and the procedures for gathering and reviewing these data are 

discussed in detail in this chapter. 

A survey of state managers was the source for data on the states’ technical assistance 

needs and priorities, on other sources of technical assistance used, ratings of the overall technical 

assistance received, and on perceived capacity change at the state level. The technical quality of 

Center technical assistance was assessed by a panel of experts on the topic of each technical 

assistance project. Finally, participants answered survey questions pertinent to the relevance and 

usefulness of Center technical assistance. 

Each of these data sources is described in this chapter, and the analytic procedures 

specific to each source are discussed. The data collection instruments and further details 

regarding data sources and procedures can be found in appendices referenced throughout the 

chapter. The chapter concludes with a brief explanation of units of analysis and use of statistical 

tests. 

11 
 Research question 

Project 

inventory forms 

and project 

cover sheets 

Center 

management plans 

Interviews with 

Centers 

State 

manager 

survey 

How did the Regional Comprehensive Centers and Content Centers operate as part of the Comprehensive Technical Assistanc

How did Centers develop, refine, and carry out their plans for technical assistance? How did they define their clients’ educational needs and priorities? What were the objectives of the technical assistance the 

Centers offered? What kinds of products and services were 

provided by the Centers? How did the Regional Comprehensive Centers and Content Centers coordinate their work? 



 



 

What was the performance of the Comprehensive Centers in addressing state needs and priorities? How did their performance 

of time studied? 

How did the Centers’ state clients define their needs and priorities? To what extent, as reported by states, did Center assistance 

expand state capacity to address underlying needs and priorities and meet the goals of NCLB? 

To what extent did states rely on other sources of technical assistance besides the Centers? What were other sources of technical assistance that states used? How did the 

usefulness of Center assistance compare with the usefulness of assistance from other sources? 



To what extent was the assistance provided by the Centers of high quality, high relevance, and high usefulness 

Did the quality, relevance, or usefulness of Center assistance change over the period of time studied? 

What was the variation in the quality, relevance, and usefulness of Center assistance across types of projects and participants? 



(overall 

relevance and 

usefulness) 



(overall 

relevance and 

usefulness) 

Exhibit 2.1. Data sources for the research questions, 2006-07, 2007-08, and 2008-09 

12
 
13 

Identification of a Sample of Center Work for Expert Review 

and Participant Surveys 

A critical component of the evaluation was to rate the quality, relevance, and usefulness 

of Center products and services. Given available resources, it was not possible for the evaluation 

team to submit all of a Center’s products and services to an independent review panel to rate 

quality. Nor was it feasible to survey all individuals who used Center products or participated in 

Center activities in the designated time period regarding relevance and usefulness. Therefore, the 

evaluation team developed and applied a strategy to select a sample of work from each Center 

for expert panel review and participant ratings. 

The following sections describe: (1) the unit of analysis, (2) the sample frame, (3) the 

evaluation sample, and (4) materials obtained from Centers. 

The Unit of Analysis: The Project 

The evaluation team initially reviewed the Centers’ 2006-07 management plans to 

understand the nature of the work the Centers were conducting and determine if the management 

plans might serve as an appropriate sampling frame for the evaluation. Based on this effort, the 

team determined that the sampling procedures could not be based on the management plans for 

two reasons. First, the Centers’ plans and work continued to evolve over time. Consequently, the 

plans, which were prepared before the program year, did not comprehensively reflect the work 

actually being done by the Centers months later. Second, the Centers used different approaches 

to organizing and aggregating their work. For example, some presented their work by state while 

others organized it by topic area. For the purposes of this evaluation, the team identified 

“projects” as a common level of aggregation that would constitute units large enough for review 

and rating, but focused enough for coherence. A “project” was defined as a group of closely 

related activities and/or deliverables designed to achieve a specific outcome for a specific 

audience. To ensure that projects would constitute units that were large enough for review and 

rating, but focused enough for coherence, the study team provided the following criteria: 

■ Complete and coherent whole. Because each project should be able to stand on 

its own in an expert panel review, it should include all related activities and 

products. 

■ Common intended outcome. Where a cluster of activities and deliverables was 

designed by the Center to lead to the same outcome for the same audience, those 

activities and deliverables should be grouped as one project. 

■ Topical focus. With few exceptions, a project addressed just one topic (for 

example, effective systems of support, adolescent literacy, assessment of English 

language learners) around which there was a body of research or professional 

wisdom. 
 

 

 

Since the project was a unit developed for this evaluation and was not necessarily how 

Centers divided up their work for programmatic or cost-tracking purposes, the evaluation team 

took several steps to ensure that the project concept was clear and used consistently across 

Centers. The evaluation team developed a standard Project Inventory Form (PIF) that Centers 

used to create an inventory of their work for each of the program years in this evaluation (see 

appendix B for a copy of the PIF) as well as written guidance in the instructions for completing 

the PIF (described in appendix B) and a sample inventory to serve as an example of the kinds of 

projects that should be listed by the Centers. The sample inventory included examples of 

activities or resources for defining projects at each level of effort—major, moderate, or minor. 

Members of the study team also invited Center staff to attend training sessions by conference call 

on completing the PIF, including defining projects. When draft inventories were received, the 

study team reviewed them to make sure the entries listed met the three criteria used to identify 

projects. They also reviewed the Center’s reporting of the project level of effort for 

correspondence with the listed activities and resources, and listing of the projects under each 

topic for correspondence with the topic definitions provided in the written guidance. The study 

team provided technical assistance to Centers as needed to ensure that all projects conformed to 

the standards. 

The Sampling Frame 

The evaluation was designed to assess the quality, relevance, and usefulness of a sample 

of Center work. The PIFs served as the basis for identifying the sampling frame from which the 

study team drew the sample of projects that became the subject of expert panel reviews and 

participant surveys. For each data collection cycle, the evaluation team asked each Center to use 

the PIF to prepare an inventory of all the projects active during the appropriate grant period.23 

Exhibit 2.2 shows the total number of projects on the PIFs by project size for each of the data 

collection cycles.24 Projects that spanned multiple years were counted each year they appeared 

on the PIF; therefore, the total number of distinct projects is less than the sum total of individual 

projects listed in Exhibit 2.2. 

23 In the instructions provided to Centers for completing the PIFs, “projects” excluded the following activities: (1) 

training or professional development of Center staff, (2) work on coordinating committees within the 

Comprehensive Center network, (3) annual needs assessments or negotiations with states, unrelated to specific 

projects, and (4) other internal meetings or documents.

24 Centers designated on their PIFs whether each project was “major,” “moderate,” or “minor” in terms of the level 

of effort and/or resources the Center devoted to it, relative to other projects in the same Center. Examples of 

activities or resources defining each project level of effort were provided in the instructions for the PIFs. After draft 

inventories were received, the evaluation team also reviewed the Center’s reporting of the project level of effort for 

correspondence with the listed activities and resources and followed up with the Centers where there were questions. 

Although instructions were provided with examples and checks were conducted, these designations were not 

standardized across Centers and as a result a project considered major by one center might be viewed as moderate or 

minor by another Center. 

14 
 

Exhibit 2.2. Number of projects on the project inventory forms (PIFs), by 

project size and by year 

Project size 2006-07 2007-08 2008-09 

Total 364 346 331 

Major 

Moderate 

110 1

106

11

111

108

110

Minor 148 124 113 

EXHIBIT READS: There were 364 projects included on the PIFs for the 2006-07 

program year. Of these, 110 were classified as major projects, 106 were 

moderate projects and 148 were minor projects. 

SOURCE: Project inventory forms. 

To be eligible for the sampling frame, projects first needed to represent a reasonable 

amount of effort (i.e., classified as “major” or “moderate” by the Center) and have a sufficient 

amount of material to give reviewers enough information to judge the quality of the work. 

Projects included in the sampling frame also needed to have identifiable participants since the 

evaluation design called for collecting relevance and usefulness ratings through surveys of 

project participants. All minor projects (approximately one-third of the Center projects in each 

year) were excluded from the sampling frame, as few of them had identifiable participants or 

sufficient materials for panel review. Thus, the sampling frame represents the portion of each 

Center’s work that they identified as using a major or moderate amount of their efforts or 

resources. 

The Sample of Projects 

To ensure that the final sample of projects reviewed each year reflected a range of each 

Center’s work across topics and states and captured work that represented the largest investment 

of resources, the team implemented a sampling strategy that included a combination of Centernominated projects and a stratified set of purposively-selected projects. 25 The desired sample 

size at each Center was a function of that Center’s budget amount. For each of the three data 

collection years, the number of projects sampled for each Center ranged from three to eight 

depending on the size of the Center’s annual budget. Centers with smaller budgets (less than 

$2 million) were asked to nominate one project for inclusion in the study sample while those 

with budgets of $2 million or more were asked to nominate two projects. Exhibit 2.3 shows the 

target and actual sample distribution by budget size across study years. In some cases, the 

Centers did not have enough eligible projects in the sample frame to meet their desired sample 

size so the total number of projects sampled was lower than expected. Although the sample was 

not statistically representative in scientific terms, it was designed to include a high percentage of 

the major projects of each Center as well as projects that Centers thought best represented their 

work. 

25 The full list of decision rules, applied across all Centers sampling projects within center are provided in appendix 

B. 

15 
 

 

 

 

To increase buy-in from the Centers and to allow them to showcase a project of their 

choice, Centers were given an opportunity to nominate the one or two projects they felt best 

represented their work.26 These Center-nominated projects were selected first for the sample. 

Selection of the remainder of the sample was then completed by the evaluation team using an 

iterative sampling process to randomly select projects while controlling for topic and state. Major 

projects (across topics and states) were sampled first and then moderate projects (if needed). 

When choosing between projects of similar categorical size, the evaluation team selected 

projects from different topics before sampling multiple projects within a given topic. When 

choosing between projects of similar categorical size and topic, the evaluation team selected 

projects from different states before sampling multiple projects within a given state.27 

Using these methods, the evaluation team selected a total sample of 122 projects in 2006-

07 and 2007-08 and 118 projects in 2008-09. Exhibit 2.4 shows the number of projects in the 

sample by project size and by year. In each of the years, the sample predominantly included the 

most major work of the Centers although the sample was not statistically representative in 

scientific terms, and covered 33 to 36 percent of all projects reported on the PIFs. 

26 The number of projects a Center was able to nominate depended on the size of its annual budget, as shown in 

exhibit 2.3. 

27 Overall, the procedures used to select the sample of projects to be included in the study each year were intended to 

provide a “fair” representation of Center work. In addition to the bias introduced by Center nominations of projects 

they considered to best represent their work, there are two dimensions for which there is the potential for selection 

bias in the sample of projects. Projects were selected to ensure inclusion in the study sample of projects covering a 

variety of content areas and serving all geographic areas. However, this strategy produced a non-probability sample. 

In addition, this strategy ensured that topic areas addressed by the largest number of projects are underrepresented in 

each year’s project sample, and topic areas addressed by relatively few projects are overrepresented. It is also 

possible that topic areas addressed by relatively few Centers would be underrepresented in the sample. The other 

dimension on which the sample is likely to be biased is project size. The evaluation team deliberately selected major 

projects over moderate projects whenever possible. To the degree that the major projects are unrepresentative of the 

Centers’ work conducted under moderate or minor projects, then the study sample is biased. 

16 
 

 

 

 

 

 

 

 

 

Exhibit 2.3. Sample size by budget 

Center funding 

level—annual federal 

contract 

(number of Centers) 

Target 

number of 

projects to be 

selected for 

review, per 

Center 

Number of 

projects 

nominated/ 

Number of 

projects selected 

purposively, per 

Center 

Number of projects selected 

across Centers in: 

2006-07 2007-08 2008-09 

Total 122* 122 118 

Less than $1 million 

(N=2) 

$1 to 1.9 million 

(N=5) 

$2 to 2.9 million 

(N=7) 

$3 to 3.9 million 

(N=5) 7 

$4 million or more 

(N=2) 

4 

5 

6 

8 

1/3 

1/4

2/4

2/5

2/6

7 6 7 

25 23 25 

41 43** 40 

34 34 31 

16 16 16 

* This total includes a collaborative project conducted jointly by an RCC and a CC that is accounted for once in the 

total sample but included once for each relevant Center in the table rows. 

 ** The total number of projects exceeded the expected target by one due to a program that was misclassified in a 

budget category with a larger target sample size. 

EXHIBIT READS: There were two Centers whose annual funding level was less than $1 million. Four projects were
 

selected for review for each of these centers, one of which was nominated by the Center for selection and three of 

which were selected for inclusion by the study team. 

SOURCE: U.S. Department of Education. 

Exhibit 2.4.	 Number of projects in the study sample, by project size and 

by year 

Project size 2006-07 2007-08 2008-09 

Total 122 122 118 

Major 93 88 92 

Moderate 29 34 26 

EXHIBIT READS: There were 122 projects in the project sample for the 2006-07 program year. Of these, 93 

were classified as major projects and 29 were classified as moderate projects. 

SOURCE: Project inventory forms (PIFs) submitted by the Centers.
 

Further detail on the sampled projects by topic in relation to all major and moderate 

Center projects for the 2008-09 data collection cycle appears in exhibit 2.5. Refer to appendix B, 

exhibits B.2 and B.3 for similar exhibits showing the sampled projects by topic for 2006-07 and 

2007-08. In 2008-09, 50 percent or more of all major or moderate projects were selected for the 

17 
 

 

 

 

project sample in 16 of the 2328 topics. This figure was 17 of 23 for 2006-07 and 15 of 23 for 

2007-08. 

Exhibit 2.5.	 Distribution of all major and moderate projects and projects 

in the evaluation sample, by topic in 2008-09 

Number of 

major and 

moderate 

projects on 

project 

inventories 

Percent of 

Project topic 

Number of 

projects in the 

sample 

(all major or 

moderate) 

all major and 

moderate 

projects in 

evaluation 

sample 

Total 218 118 54% 

Components of effective systems of 

support—state, district, school 54 33 61 

Data use/data-driven decisionmaking 6 3 50 

Formative assessment 4 2 50 

Reading 6 3 50 

Adolescent literacy 10 6 60 

Mathematics 16 8 50 

Dropout prevention 4 2 50 

High school redesign/reform 7 4 57 

Transition to high school 2 2 100 

Special education—curriculum, instruction, 

and professional development 1 0 0 

Special education—assessment 5 1 20 

English language learners 26 12 46 

Highly qualified teacher provisions of 

NCLB 11 4 36 

Teacher preparation and induction 5 2 40 

Teacher professional development 4 4 100 

Supplemental educational services (SES) 2 2 100 

Response to Intervention (RtI) 14 9 64 

Migrant education 7 3 43 

Indian/native American education 1 1 100 

Data management compliance 9 5 56 

Assessment design 3 2 67 

Parent involvement 6 2 33 

Other 15 8 53 

EXHIBIT READS: For the 2008-09 program year, there were 54 major and moderate projects on the project 

inventories that focused on components of effective systems of support. Of these, 33 (or 61 percent) were in the 

project sample. 

SOURCE: Project inventory forms submitted by the Centers. 

28 In all years, the topic area of “other” is included as the 23rd topic area, as a small number of the sampled projects 

were included in that area. Subsequent discussions in the report use the 22 specific substantive topic areas. 

18 
Materials from Centers 

The evaluation team notified the 21 Center Directors about projects selected for review 

and sent a standard Request for Materials for Expert Panel Review (see appendix B for a copy of 

the transmittal memo and request form). Centers were asked to assemble and submit a 

comprehensive set of pre-existing materials associated with each project (meeting agendas, 

briefing books, meeting summaries, training materials, white papers, web resources, etc.) that 

would fully describe the project and provide reviewers with a sufficient basis for rating the 

technical quality of the work. Centers were also asked to include a participant list and a standard 

cover sheet, using a format developed by the evaluation team, for each project. The participant 

lists were used to draw the sample for participant surveys. 

When the project materials were received, they were reviewed by evaluation team 

members for completeness. If materials were missing or inaccessible (e.g., electronic files didn’t 

open) or the cover sheet was incomplete, a follow-up memo was sent to the Center describing 

any issues. Evaluation team members then worked with the Center to obtain the missing 

information and finalize the review package (see appendix B for a copy of the follow-up memo). 

Describing Center Operations 

The purpose of collecting data from the Centers was to describe Center operations and to 

address research questions regarding the objectives of the Centers, the kinds of products and 

services provided, how the Centers defined their clients’ needs and priorities, and the extent to 

which the Centers met the objectives of their own plans. The description of Center operations in 

chapter 3 of this report is drawn from three sources of data: interview responses gathered from 

the Centers, PIFs, and the cover sheets submitted for the sampled projects. The procedures used 

in gathering and analyzing data from these sources in all three years are described in the 

following pages. 

Management Plans 

Centers stated their objectives for each year in their annual program management plans. 

For example, for the 2006-07 program year, Centers had developed their management plans in 

spring 2006, prior to the program year. The evaluation team relied on these management plans to 

gather information from statements of the Center’s intended substantive focus for the year’s 

technical assistance. The team coded each Center’s stated objectives in their management plan 

by subject area, using the same list of 22 topics used to code the PIFs. Appendix C provides a 

description of the process used to code the management plans as well as an analysis of the 

intercoder reliability. 

19 
Interviews with Centers 

The evaluation team visited each of the Centers in person in the summers of 2007 and 

2008 and conducted the third interview by phone in spring 2010. The primary purpose of these 

interviews was to capture descriptions of Center operations during each of the program years. 

Interviews conducted with Center directors and other key leadership staff posed a combination of 

structured open-ended and binary questions regarding the following topics (see appendix C for 

the protocols): 

■	 Center organization (lead organization, subgrantees, ways of dividing 

responsibilities among staff) 

■	 Major areas of focus 

■	 Communication with client organizations (states in the case of RCCs, or RCCs 

and states in the case of CCs) regarding needs and assistance to be provided 

■	 Modes of delivering technical assistance 

■	 Reasons for not carrying out technical assistance requests 

■	 Barriers to delivering technical assistance 

■	 Approaches taken in quality assurance 

■	 Working relationships within the Center network 

For the third round of Center interviews, the evaluation team added probes on topics of 

emerging policy interest such as the ways in which Centers viewed their work to have evolved 

over the years, the Centers’ views on barriers to providing technical assistance, and additional 

description of the process for quality assurance that Centers established. 

Project Inventory Forms (PIFs) 

While the main purpose of the PIFs was to build the sample frame (described earlier in 

this chapter), the evaluation team also used the PIFs to gain data on the work the Centers 

undertook in each of the study years. The projects sampled for quality, relevance, and usefulness 

ratings were classified by topic (see appendix B). 

Project Cover Sheets 

For projects included in the sample each year, the standard cover sheets provided by the 

Centers described project activities and cross-Center collaboration in the project. The project 

cover sheets were primarily collected from each Center to help orient expert panels to the 

20 
purpose and content of the materials to be reviewed (see appendix B for a copy of the Request 

for Materials for Expert Panel Review transmittal memo and cover sheet). 

The evaluation team used the cover sheets as a data source for an overall description of 

activities and collaboration in the sampled projects (reported in chapter 3) and for use in 

classifying projects into subgroups for which the ratings of relevance and usefulness could be 

analyzed (reported in chapter 5, as described below). The cover sheets provided descriptive 

information for each project, including the activities and deliverables associated with the project 

and the contributions of other Centers to the project. Categories of Center activities and 

resources were drawn from review of the Center management plans and site visit interviews. 

These coding categories, thus, permitted a yes/no judgment of whether each project offered each 

of the following activities or resources to participants: 

■ Ongoing consultation and follow-up 

■ Research collections or syntheses 

■ Engagement of participants in project planning 

■ Training events 

■ Task force meetings and work 

■ Conferences 

■ Support for development of a formal plan to implement a program or policy 

The team also coded the type of contribution, if any, of any CC to each RCC project, and 

of any RCC to each CC project. 

Additionally, for the 2008-09 data collection cycle, the evaluation team used the project 

cover sheets to provide additional detail regarding the 33 projects in the study sample included 

under the topic of effective systems of support. For these projects, the evaluation team coded 

activities and resources at a more fine-grained level. 

All these elements of the cover sheets were coded by members of the evaluation team 

using procedures described in appendix C; the appendix also provides a detailed description of 

all the codes used as well as the results of the analysis of intercoder reliability. 

Survey of State Managers 

The purposes of the survey of state managers were to obtain information on state 

priorities in terms of state responsibilities related to the implementation of NCLB, obtain the 

state perspective from SEA administrators working with the Centers on the relevance and 

usefulness of Center assistance, and obtain a comparative judgment of Center assistance in 

21 
 

 

relation to assistance available through other sources such as professional associations. The 

survey instrument used for the survey of state managers appears in appendix E of this report. 

In order to identify appropriate respondents for the survey of state managers, each year 

the evaluation team collected the names of each RCC’s main point(s) of contact in each SEA 

from the RCCs. For each of the data collection years, this resulted in a total of over 120 

respondents across the 62 states included in the study.29 

To be fully reflective of all SEAs in the analysis of these data, it was critical that the 

study team receive completed surveys from state managers in each state. In 2006-07 there was at 

least one response from each of the 50 states and 6 of 12 outlying areas, in and 2007-08 there 

was at least one response from each of the 50 states and 7 of 12 outlying areas. In 2008-09, the 

evaluation team received responses from 48 states and 6 of 12 outlying areas. 

In each of the years, there were a number of states for which the evaluation team received 

completed responses from more than one state manager. Exhibit 2.6 shows the number of 

responses to the state manager survey for each of the years. The state was the primary unit of 

analysis in analyzing data from the state manager survey for this report. The state managers’ 

responses were weighted to ensure that each state was equally represented in all summary 

statistics while taking into account the variation in responses within each state. The weighting 

procedure, where each response was weighted by the inverse of the number of managers 

responding from that state, ensured that each state was equally represented when the evaluation 

team aggregated responses across states to describe the distribution of responses. 

Exhibit 2.6. Number of responses to the state manager survey, by year 

Number of responses 

from the states 2006-07

Number of states 

2007-08 2008-09 

Total 56 57 54 

1 36 30 34 

2 15 21 19 

3 4 5 1 

4 1 1 0 

EXHIBIT READS: For the 2006-07 program year, in 36 states the survey of state 

managers was completed by a single respondent. There was one state where the state 

manager survey was completed by four separate respondents. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was 

equally represented in instances where more than one manager from a state responded. 

29 The Centers were expected to serve 62 jurisdictions including the 50 states and 12 other jurisdictions: the District 

of Columbia, Puerto Rico, the Virgin Islands, American Samoa, Commonwealth of Northern Marianas, Federated 

States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap), Guam, Republic of the Marshall Islands, and Republic of 

Palau. For purposes of this report, the term “state” refers to the 50 states and the outlying territories listed here. 

22 
 

 

At the conclusion of the state manager survey, respondents were asked to describe the 

offices or departments within their state department of education they directed. The responses to 

this question show that across the states the offices most often represented by the state managers 

were federal programs and school improvement. Exhibit 2.7 shows the distribution of weighted 

responses to these questions across the three data collection years. 

Exhibit 2.7.	 Office division or department directed by state manager survey 

respondents, by year 

Percent of state managers (weighted) 

Office/department 2006-07 2007-08 2008-09 

Federal programs 68% 64% 65% 

School improvement 64 66 67 

Curriculum and 

instruction 53 50 45 

Assessment and 

accountability 40 45 38 

Special education 30 28 24 

NOTE: Survey respondents were able to select multiple responses to this question and as 

a result, the percentages do not add to 100 percent. 

EXHIBIT READS: For the 2006-07 program year, 68 percent of state managers 

(weighted) indicated that they directed the federal programs office in their state. Sixty-four 

percent of the state managers (weighted) indicated that they headed the school 

improvement office. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was 

equally represented in instances where more than one manager from a state responded. 

Assessment of Quality by Expert Panels 

Given the Centers’ charge to focus on scientifically based research and evidence-based 

practices to improve student achievement and close achievement gaps, one goal of the evaluation 

is to assess the technical quality of work across the Centers using an independent panel of expert 

reviewers. Each sampled project was independently rated by a panel of three experts.30 

Reviewers assigned a score to each of three quality dimensions discussed below, using a 5-point 

rating scale. 

In an effort to maximize interrater reliability, the evaluation team: (1) defined quality and 

developed a detailed scoring rubric that could be applied across all Centers and a range of 

projects; (2) recruited, trained and assigned highly qualified expert panelists; and 

(3) implemented a process for reviewers to discuss their findings with one another when scores 

were discrepant for particular projects. The following sections describe each of these steps, 

followed by a brief discussion of how the final technical quality ratings were calculated. 

30 In each of the years, there were between 2 and 9 projects that were rated by only two panelists. 

23 
Define Technical Quality and Develop Scoring Rubric 

Developing the definition of technical quality and the rubric used to measure it 

incorporated information from many sources, including federal legislation and ED specifications 

on what constituted scientifically based evidence and an example from another federal agency.31 

When defining technical quality and developing the scoring rubric, the evaluation team sought to 

ensure that the definition was relevant to the range of projects Centers would provide–

 recognizing that some projects might have a substantial research knowledge base and others 

might be guided more by promising practices (those that were supported by evidence but not yet 

rigorously studied) or legislative or regulatory requirements. Also, the definition of quality and 

the associated rubric had to be applicable to projects at varying points of development and 

implementation, from early-stage needs assessment and design work to fully-developed products 

and services. The evaluation team developed a quality scoring rubric, included in appendix G, to 

assess quality along the following three dimensions: 

■	 Dimension 1: Demonstrated use of the appropriate documented knowledge base 

■	 Dimension 2: Fidelity of application of the knowledge base to the products and 

services provided 

■	 Dimension 3: Clear and effective delivery 

Reviewers assigned a score to each dimension, using a 5-point rating scale (where 1 

meant “very low quality” and 5 meant “very high quality”), according to the indicators defined 

for each dimension and examples in the scoring booklet. 

Recruit, Train, and Assign Expert Panelists 

To meet the selection criteria for this evaluation, expert panelists had to have current, 

rigorous work in the particular topic of interest (for example, publications in peer-reviewed, 

scholarly journals; presentations at relevant professional organization meetings; recent 

membership on advisory panels or task forces) and be free of conflicts of interest.32 Nominations 

for panelists were made by staff in ED’s Institute of Education Sciences (IES), the Technical 

Work Group for this evaluation, members of the evaluation team, and Center staff. Based on the 

selection criteria, for the 2006-07 data collection cycle the evaluation team selected a total of 70 

expert panelists, 94 percent with a doctorate degree and two-thirds (67 percent) with university 

affiliations, to review the Centers’ 2006-07 sampled projects.33 An additional 14 expert 

reviewers were recruited and trained for the second round of reviews to replace four reviewers 

 

31 Review of Instructional Materials for Middle School Science. (1997, February). National Science Foundation, 

Retrieved from http://www.nsf.gov/pubs/1997/nsf9754/nsf9754.htm?org=NSF. 32 Current Comprehensive Center staff, as well as individuals employed by organizations that had an ongoing
 

financial relationship (for example, a contract or cooperative agreement) with a Comprehensive Center and who 

worked on a Center project, were not eligible to serve as reviewers. 33 The final expert panels for 2006-07 were made up of 67 expert reviewers. One of the initial trainees was removed 

due to a conflict of interest, and two reviewers were later dropped from the review process when they failed to
 

complete their reviews in a timely fashion. 

24 
 

 

who did not return and to add additional depth in certain topical areas, in particular state systems 

of support. All but two of the 77 second round reviewers returned for the third and final round of 

reviews. 

The evaluation team sought to maximize interrater agreement in scoring of quality by 

training expert reviewers to systematically use a standard rubric. During the two-day training, 

experts worked in small groups to discuss how each of the three quality dimensions, and their 

corresponding indicators, applied to sample project descriptions that were provided by the study 

team. Reviewers independently scored one of two projects overnight and submitted their 

dimension-level scores at the beginning of the second day. Scores were posted and analyzed in 

terms of rater agreement at the dimension level. The panelists again worked in small groups to 

discuss their scores and identify possible reasons for any discrepancies in the results. During the 

small group discussions, evaluation staff circulated among the groups to assess whether 

reviewers had adhered to the standards of evidence discussed on Day 1.34 The same training was 

offered to newly recruited reviewers for the second round of reviews, while returning reviewers 

completed a refresher training session offered by Webinar. A similar refresher training session 

was also provided for all returning reviewers in advance of the third and final round of reviews. 

During the actual review process, expert reviewers were asked to score four to eight 

projects in their area(s) of expertise, with no more than three projects from any given Center. 

Since judgments about the state of the available evidence on a given topic and its applicability to 

the project being rated relied heavily upon the knowledge of the expert reviewers, it was 

important that reviewers were assigned projects that matched their area(s) of expertise. The 

evaluation team was also careful when assigning projects to avoid known conflicts of interest.35 

When needed, the evaluation team reassigned projects when reviewers were unable to complete 

their reviews in a timely fashion, identified unforeseen conflicts of interest, or did not feel they 

had the requisite expertise to review the assigned projects. 

Overall, between 93 and 98 percent of all projects sampled were reviewed by three 

panelists. In each year, a small number of projects (9 projects for 2006-07, 2 projects for 2007-

08, and 3 projects for 2008-09) were reviewed by only two panelists because the third assigned 

reviewer did not complete the review within a reasonable timeframe.36 

34 During training, expert reviewers scored a sample project and discussed their scores in small groups, paying 

particular attention to areas where scores were divergent. This approach allowed the evaluation team to identify 

dimensions or indicators within the scoring rubric that seemed to be problematic (resulting in discrepant scores) or 

particular reviewers who needed additional training or appeared to be inappropriate to use. The goal was to have 

panelists leave the training with a common understanding of how to apply the quality rubric; the evaluation team did 

not intend to attempt to establish a specific interrater reliability criterion at training. 

35 For the purpose of assignments, the term “conflict of interest” meant any financial or other interest that appeared 

to conflict with or significantly compromise the service of the individual reviewer because it could significantly 

impair the individual’s objectivity. 

36 In consultation with IES, the evaluation team decided to forgo the third review on these particular projects since 

there was interrater agreement (i.e., a difference less than 2 points) between the two scores that had been submitted 

by the other panelists assigned to these projects. 

25 
 

 

 

 

Address Interrater Reliability 

In addition to the detailed scoring rubric and training provided, a resolution process was 

used to help achieve a high degree of interrater reliability in scoring. If the reviewers’ projectlevel scores (defined as the simple average of their three dimension-level scores) were found to 

differ by 2 or more points from each other for any given project, the evaluation team convened 

the panel by telephone to discuss the ratings.37 The goal of the discrepancy conference calls was 

to give panelists an opportunity to understand the rationale behind their colleagues’ scores and 

consider whether, on the basis of that discussion, any scoring revisions were warranted. The 

evaluation team emphasized to the reviewers that these discussions were not intended to achieve 

consensus among them. 

Overall, between 76 and 83 percent of them had no discrepancies in the original projectlevel scores (exhibit 2.8). After a discrepancy conference call, panelists were given the 

opportunity to submit a revised scoring form and narrative report of the project’s strengths and 

weaknesses. Although panels were not required to reach consensus, discrepancies were resolved 

in all but two to four cases, increasing the number of projects in agreement from 76 percent to 98 

percent in 2006-07, and from 83 percent to 97 percent in 2007-08 and 2008-09. 

To estimate interrater agreement, we calculated the rWG(J) index.38 Before the 

discrepancy calls, the average rWG(J) coefficient ranged from 0.75 to 0.87 across the three years 

of review (exhibit 2.9).39 After the calls were conducted, the value increased to 0.87 for program 

year 2006-07 and 0.88 for 2007-08 and 2008-09. 

37 The 2-point difference threshold was selected in an effort to identify projects with widely divergent scores that cut 

across qualitative categories (i.e., “low” quality versus “high” quality), while at the same time minimizing burden on 

expert panelists. 

38 James, L.R, Demaree R.G., and Wolf G. (1984). Estimating within-group interrater reliability with and without 

response bias. Journal of Applied Psychology, 69(1), 85-98. 39 We estimated interrater agreement using the rWG(J) index. Treating each dimension as an “item,” we first 

calculated the score variance between raters on each dimension and then averaged the three variances. We assumed 

the random error variance to be 2.0 for a 5-point scale, as suggested by James et al. (1984). An rWG value between 

0.71 and 0.90 is generally considered an indicator of “strong agreement” (LeBreton, J.M., and Senter, J.L. (2008, 

October). Answers to 20 questions about interrelated reliability and interrater agreement. Organizational Research 

Methods, 11, 815-852. Retrieved from http://orm.sagepub.com/cgi/content/abstract/11/4/815). 

26 
Exhibit 2.8. Distribution of discrepancies and results of the discrepancy 

call process, by year 

Description	 2006-07 2007-08 2008-09 

Total number of projects 

reviewed by expert panels 122 121 118 

Projects without a projectlevel discrepancy 

93 

(76 percent of 122) 

100 

(83 percent of 121) 

98 

(83 percent of 118) 

Project-level discrepancies 

identified 

29 

(24 percent of 122) 

21 

(17 percent of 121) 

20 

(17 percent of 118) 

Project-level discrepancies 

resolved* by panel calls 

Project-level discrepancies 

remaining after panel calls 

27 

(93 percent of 29 

calls conducted) 

2 

(7 percent of 29 

calls conducted) 

17 

(81 percent of 21 

calls conducted) 

4 

(19 percent of 21 

calls conducted) 

16 

(80 percent of 20 

calls conducted) 

4 

(20 percent of 20 

calls conducted) 

* Discrepancies were considered resolved when the project-level scores differed by less than 2 points. 

EXHIBIT READS: Of the 122 projects reviewed for the 2006-07 program year, 93 (76 percent) had no discrepancies 

in the project level scores. 

Exhibit 2.9.	 Interrater agreement before and after discrepancy calls (original 

versus revised scores), as measured by the rWG(J) index, by year 

rWG(J) index 2006-07 2007-08 2008-09 

Before discrepancy calls 

(original scores) 

After discrepancy calls 

(revised scores) 

0.75 

0.87 

0.81

0.88

0.87

0.88

EXHIBIT READS: The interrater agreement index for the original scores for 2006-07 

project sample was 0.75. This number increased to 0.87 after discrepancy calls were 

conducted. 

Calculate Aggregate and Dimension-Level Measures of Technical Quality 

In order to analyze the technical quality of sampled projects in each year, a series of steps 

was taken to combine individual expert ratings into aggregate scores. The first step was to 

combine the individual dimension-level scores into a single rating from each panelist. For each 

project reviewed, a simple average of the three dimension-level scores was computed to generate 

a project-level score for each reviewer. In the second step, the three reviewers’ scores were 

averaged to determine the overall quality score for each project. In the third step, the overall 

project scores across the set of sample projects were averaged within each Center to calculate a 

Center-level quality score. 

27 
 

 

Center-level scores were aggregated to calculate the average quality rating across 

sampled projects for the Center program as a whole, as well as for two subgroups of interest, the 

16 RCCs and the 5 CCs. Each Center was given equal weight in computing the overall programwide rating for technical quality, as well as the mean ratings for the RCCs and the CCs 

respectively, for the sampled projects; within Centers, each project was given equal weight. 

Survey of Project Participants 

The purpose of the participant survey was to obtain client views of technical assistance 

from the Centers, particularly in the areas of relevance and usefulness. As the primary role of the 

RCCs is to provide technical assistance to the states in their regions, the clients for the RCCs 

included state-level staff. The clients for the CCs included both RCC staff and state-level staff, 

corresponding with the role of the CCs to build the capacity of the RCCs as well as to work with 

RCCs in providing technical assistance to states. 

The evaluation team developed two parallel survey forms for project participants to 

administer each year: one for state-level staff who participated in any Center project, and one for 

RCC staff who participated in a CC project (located in appendix G of this report).40 In sampling 

participants to respond to surveys focused on specific projects, the goal was to identify a sizable 

number of participants, drawn from complete lists of all participants, so that their responses 

would collectively provide a picture of all participants’ views regarding the sampled projects. 

For each survey administration, the evaluation team drew samples of participants in the projects 

that were selected for expert panel review. In this way, expert panel ratings of quality and 

participant ratings of relevance and usefulness were gathered for the same set of projects. A brief 

description of the process used to calculate the measure of relevance and usefulness is provided 

at the end of this section. 

In each data collection year, Centers were asked to furnish full lists of all participants and 

their contact information for each sampled project. These lists made up the sample frame and 

included individuals who had participated in projects in numerous ways including: (1) serving on 

task forces, school support teams, and work groups associated with the project; (2) attending 

conferences, technical assistance retreats, and other meetings held as a part of the project; or (3) 

receiving written materials or other disseminated resources. State-level participants included 

staff who were employed by SEAs as well as employees of intermediate agencies, LEAs, 

schools, or other agencies who had responsibilities for state-level implementation of NCLB; they 

could be participants in both RCC and CC projects. RCC staff, as clients of the CCs, were 

participants solely in CC projects. 

After identifying the sampling frame, the team implemented a sampling strategy that 

combined random sampling from each project and a replacement strategy to minimize 

40 Since the survey asked respondents for their experiences with the Centers in relation to a specific project, the 

survey team provided each respondent with a list of the activities included in that project, based on the content of the 

project cover sheets provided by the Centers. For the on-line survey, this information was displayed on the first 

screen of the survey. For the paper version of the survey, the project-specific information was printed on yellow 

paper and inserted into the front of the booklet. 

28 
respondent burden. A power analysis was not conducted as part of the sampling plan for the 

participant survey. A sampling strategy was developed for selecting a sufficient number of 

participants to obtain fair representation of client views for each project, and across projects, 

while balancing respondent burden and data collection costs. The team drew a simple random 

sample of participants within each sampled project using the following sampling rules, based on 

the number of participants in the project: 

■	 All participants in projects with 12 or fewer participants were sampled. 

■	 A random sample of 12 participants was selected for projects with 13 to 25 

participants. 

■	 A range of 12 to 48 participants were randomly selected to represent 48 percent of 

participants for projects with 26 to 100 participants. 

■	 For each project with more than 100 participants, a random sample of 48 

participants was selected. 

Exhibit 2.10 provides an overview of the participant survey sampling and administration. 

The exhibit shows the overall process used in each year to select the sample in each of the data 

collection years. The number of respondents for each round of data collection was 1,208 project 

participants in 2006-07, 1,319 participants in 2007-08, and 1,035 participants in the final round 

in 2008-09. Details on the sampling frame for each administration and the number of participants 

at each stage of data collection are included in the appendix (see appendix B, exhibit B.4). 

The evaluation team initially administered the participant survey to the sampled 

respondents online. Nonrespondents were contacted by phone, sent reminder cards, and sent 

paper survey forms, all in an effort to obtain completed responses. Nonrespondents included 

participants who did not return a completed survey because they no longer worked for the state 

organization or RCC, as well as participants whose contact information was incorrect. 

Exhibit 2.10. Survey of project participants sampling and survey 

administration summary 

Participants in 

RCC projects 

Participants in CC 

projects 

Sampled participants: 

•State-level 

•RCC staff 

Sampled participants: 

•State-level 

Sample frame 

Study sample 

Survey 

administration 

Completed surveys 

Refusals 

No response 

EXHIBIT READS: The sample frame was made up of participants in RCC projects as well as those 

in CC projects. The resulting sample included state-level participants for RCC projects and a 

combination of RCC and state-level participants for CC projects. 

29 
 

 

 

Calculate Measures of Relevance and Usefulness 

For the relevance and usefulness questions in the participant survey, respondents were 

asked to rate each aspect of relevance and usefulness using a 5-point scale ranging from 5 (a very 

high degree) to 1 (a very low degree).41 

For each of the data collection cycles, the evaluation team analyzed the properties of the 

items included in the survey for constructing indices of relevance and usefulness. Principal 

components analysis with no rotation was conducted on each set of items in the relevance and 

usefulness scales to determine the underlying dimensions represented by these items. For each 

scale, analyses of each year of data resulted in a one-factor solution comprised of the same items 

and with comparable levels of reliability. For the relevance scale, the Cronbach’s alpha was 0.93 

in 2006-07 and 0.94 in 2007-08 and 2008-09; it was 0.95 for the usefulness scale in all three 

years. Additional information on the factor loadings and psychometric properties for the three 

years of data collection is presented in appendix B (exhibits B.5 and B.6). 

In each year, the mean ratings at the respondent level were averaged so that each 

respondent for a given project contributed equally to a project-level rating. Thus, the relevance or 

usefulness rating at the project level was a mean of the ratings provided by sampled participants 

in that project (ranging in number from 1 to 48) who returned surveys.42 Next, the rating of each 

sampled project contributed equally to the computation of the mean rating across projects for 

each Center. Finally, the team calculated an overall mean across the 21 Centers and overall 

means for the 16 RCCs and the 5 CCs, respectively, for the relevance and usefulness ratings of 

the sampled projects. In calculating these overall means, each Center’s mean rating was 

weighted equally. 

Units of Analysis and Use of Statistical Tests in this Report 

In analyzing the quality, relevance, and usefulness data for this study for projects in the 

study sample, there were two important features of the data that had to be accommodated. First, 

the data were aggregated into several different units of analysis including the Center, project, and 

participant. Second, the process used to identify projects both for the sample frame and the study 

sample had an impact on the use of statistical tests in the resulting analyses. Both of these aspects 

of the data analysis are described below. 

41 The response category, “Not able to judge,” on the survey instruments indicated those respondents who were not 

appropriate for addressing the particular item and were, therefore, not included in analyses. In each of the data 

collection years, less than 1 percent of respondents indicated that they were “not able to judge” across either the 

eight items of the relevance scale or the 11 items of the usefulness scale. 

42 In each of the years across the eight items on the relevance scale, the rate of item nonresponse ranged from less 

than 1 percent to 2.5 percent. The rate of missing data among the 11 usefulness scale items was slightly higher with 

the percent of item-level nonresponse ranging from 0.1 to 2.8 percent. For both relevance and usefulness 

calculations, respondents with missing data on more than two of the items were excluded from the calculation, 

resulting in a small number of respondents being excluded from the analysis in each year. 

30 
Units of Analysis for Quality, Relevance, and Usefulness Ratings 

The quality, relevance, and usefulness ratings of the sampled projects were used in 

analyzing and reporting data for several different units of analysis: the projects sampled from the 

program as a whole (all 21 Centers), the projects sampled for the 16 RCCs and the 5 CCs 

respectively, subgroups of the projects, and (for relevance and usefulness) subgroups of the 

participants. For a Center’s mean rating on quality, relevance, or usefulness, the team weighted 

each sampled project from that Center equally; for a mean across the entire program or a set of 

Centers, the team weighted each Center’s mean equally. 

Several analyses were conducted for subgroups of projects across Centers—e.g., those 

projects that included a research synthesis among their products, or included training among their 

services. Each rater (an expert or survey respondent) had equal weight in the individual project’s 

rating, and each project had equal weight in the subgroup mean rating. 

Finally, some analyses were conducted for subgroups of participants across projects. For 

example, the ratings of relevance provided by participants who spent at least three days in a 

project activity were compared with the ratings provided by participants who spent less time. In 

these analyses, the unit was the participant, not the project, and the weighting was designed to 

permit generalization to all the participants across all the sampled projects. Thus the weight for 

each participant’s response was the inverse proportion of the number of respondents compared 

with the total number of participants in the population for that project. 

Number of Projects Varying with the Unit of Analysis. Readers will notice slight 

variations in the total number of projects shown in the exhibits and analyses throughout this 

report. These figures differ for two reasons. 

First, in each of the survey years, there was a small number of projects for which the 

evaluation team did not receive any completed surveys (two in 2006-07, four in 2007-08, and six 

in 2008-09). This meant that for any analysis incorporating participant data, the maximum 

number of projects in the analysis was reduced. For analyses that did not rely on participant data 

(e.g., the quality ratings), the team included all projects. 

Second, as described earlier in this chapter, in both 2006-07 and 2008-09 one project 

included in the sample was conducted jointly by an RCC and a CC. This “combined project” was 

one of the projects counted in analyses specific to RCC projects, and also one of the projects 

counted in analyses specific to CC projects. For analyses across all projects or all participants, 

however, the joint RCC/CC project or its participants were counted only once in the analysis. 

Use of Statistical Tests 

In this report, the presentation of results is sometimes framed by statistical tests, but in 

other instances those tests are not part of the presentation. The description below provides the 

rationale for these differences. 

31 
Analyses of Quality, Relevance, and Usefulness Ratings 

The findings presented in the last section of chapter 5 are based on the ratings of Center 

projects. For analysis by subgroups of project participants (last section of Chapter 5), inferential 

statistics and a level of statistical significance of p<.05 were used to decide which differences 

would be noted in the report. The evaluators adopted inferential statistics for analysis of the 

differences among project participants’ responses because the participants were selected using a 

stratified random sampling process. However, these results should not be generalized beyond the 

sample of projects selected for each year of the evaluation. 

For the interpretation of the analysis of project-level ratings for this study, the evaluation 

team adopted a difference of at least one-half of one standard deviation (calculated as the pooled 

standard deviation for all projects) in size as the minimum threshold for highlighting differences. 

Using Cohen (1988) as a conceptual framework, the evaluation team estimated Cohen’s d (an 

estimate of the effect size defined as the difference in means divided by the pooled standard 

deviation) and adopted the logic of Cohen for what would be considered a moderate difference. 

Although the report highlights general patterns in ratings from year to year, comparisons 

in ratings across year were not conducted. Because each study year’s sample of projects was 

selected through a non-probability procedure, evaluators determined that there was no basis for 

determining what portion of any differences in the distribution of project characteristics or 

ratings across years was the result of sampling bias and what portion reflected true changes in the 

operation of the projects. 

State Managers’ Assessment of Center Technical Assistance 

The data source for the state managers’ assessment of Center technical assistance in 

chapter 4 is the survey of state managers. The responses from this survey comprise a census of 

all state departments of education and not a probability sample. Thus, inferential statistics and 

assessments of statistical significance are not necessary, and any observed difference in the 

distribution of responses to the survey across time is considered a real change. While there may 

be measurement error, the estimate of whether any observed differences might be a result of 

chance incurred as part of a sampling process is not applicable. 

32 
An important context for assessing performance of the Centers is to understand Centers’ 

operations in each year and the changes in operations that occurred in the third and fourth 

program years (2007-08 and 2008-09). As described in chapter 1, ED established structures and 

expectations for the functioning of the Centers, including the division of responsibilities between 

the two types of Centers, the emphasis on applying scientifically based research and promising 

practices to build state capacity to carry out NCLB, and the flow of communication between and 

among the Regional Comprehensive Centers (RCCs), the Content Centers (CCs), and state 

agencies. 

This chapter describes the work of the Centers, addressing the following research 

questions: 

■	 How did the Regional Comprehensive Centers and Content Centers operate as 

part of the Comprehensive Technical Assistance Center program? 

■	 How did Centers develop, refine, and carry out their plans for technical 

assistance? How did they define their clients’ educational needs and 

priorities? 

■	 What were the objectives of the technical assistance the Centers offered? 

What kinds of products and services were provided by the Centers? 

■	 How did the Regional Comprehensive Centers and Content Centers 

coordinate their work? 

For this report a description of the Center operations focuses primarily on the most recent 

program year for which data were collected, the 2008-09 program year. To describe how Center 

operations have evolved over time, data from all three years are presented.43 The chapter begins 

with a discussion of the processes by which Centers negotiated their work, describing ways in 

which they reportedly met the requirement to assess their client organizations’ needs. It provides 

information about the range of technical assistance objectives the Centers addressed and the 

products and services they delivered. Because the two types of Centers were expected to 

coordinate, it discusses ways in which they reportedly did so. The chapter concludes with a 

discussion of procedures for quality assurance that Centers reported. The analysis draws on 

information gathered from the Centers’ submitted Project Inventory Forms (PIFs), Centers’ 

annual management plans, cover sheets that Centers prepared for the expert reviews of their 

 

 

 

3. How the Centers Operated 

43 In general, trend data include all three years of data collection with the following exceptions: management plans 

were reviewed for program years 2006-07 and 2008-09 only; more detailed reviews of the 2008-09 PIFs and 

accompanying cover sheets were performed to better understand the nature of Center work; questions addressing 

Center investments, processes, and working relationships with SEAs and other Centers were added in the last year of 

interview data collection and thus are only available in 2008-09. 

33 
 

 

 

sampled projects, and interviews with Center staff conducted by the evaluation team.44 It also 

draws information about Center technical assistance from the survey of state managers.45 

Throughout the chapter, findings are presented separately for RCCs and CCs. As 

described in more detail in Chapter 1, RCCs were charged with providing direct assistance to 

states on an ongoing basis (“front-line assistance”) while CCs were charged with providing 

sound knowledge about their content areas to both states and RCCs. Because the two types of 

Centers differed in their charge and in their mix of clients, as explained in Chapter 1, one would 

expect differences in their procedures for planning and needs assessment, their objectives, and 

the products and services they delivered. 

Developing Plans: Identifying Client Needs and Priorities 

Before the start of each program year, the Centers were required to deliver a management 

plan to ED outlining the program of technical assistance they planned to provide to clients. A 

key expectation of the Centers was that they would organize their plans around the priorities and 

needs of client organizations. RCCs, charged with providing “front-line” assistance to a set of 

state clients, were expected to communicate with and serve those state agencies directly. The 

CCs served both RCCs and states as clients for their technical assistance. Because their technical 

assistance for states was to be provided in concert with RCCs, they could depend in part on the 

RCCs to alert them to state priorities. In addition to serving the RCCs and states, the CCs also 

served the needs of ED, responding to requests in specific content areas to advance federal 

priorities. The evaluation team gathered Center reports on how they identified client needs. 

According to the Centers, they gathered information to develop and refine their 

management plans for each year in a variety of formal and informal ways. A comparison of 

needs assessment techniques used for the 2006-07 program year with those used for the 2008-09 

program year shows the extent of continuity in these approaches to needs assessment. 

■	 Centers conducted needs assessments annually with their respective primary 

client organizations. In 2008 the two most common modes that RCCs reported 

using to assess state needs were informal communications with state staff (all 16 

RCCs) and designation of a liaison to the SEA (15 of 16 RCCs) (exhibit 3.1). As 

compared with 2006, fewer RCCs reported communicating with chief state school 

officers in 2008 (11 in 2008 vs. 14 in 2006). For CCs, procedures used in 

assessing RCC needs in both 2006 and 2008 included communications with RCC 

staff (5 CCs in 2006 and 2008) and with RCC directors (4 of 5 CCs in 2006 and 

2008), and surveys (4 CCs in 2006 and 5 CCs in 2008). 

44 See chapter 2 for more information about the data sources and the procedures for gathering, coding, and analyzing 

the data reported in this chapter. 

45 A detailed description of the process used to gather and code the information discussed in this chapter can be 

found in appendix C and related supplemental tables are in appendix D. 

34 
Exhibit 3.1. RCC and CC interaction with clients in planning work, by year 

Number of Centers reporting this mode of interaction 

with clients 

RCCs 

(N=16) 

CCs 

(N=5) 

All Centers 

(N=21) 

Mode of interaction 2006-07 2008-09 2006-07 2008-09 2006-07 2008-09 

Informal needs assessment 

through communication with 

state staff/ RCC staff 

15 16 5 5 20 21 

Communication with chief state 

school officers/ RCC directors 14 11 4 4 18 15

Surveys 10 8 4 5 14 13 

Designated liaison to SEA 13 15 NA NA NA NA 

Liaison to SEA with office space 

in SEA building 4 3 NA NA NA NA

NOTE: NA=Not Applicable. CCs were not asked about liaisons to SEAs because they worked nationally 

and were not expected to provide “front line” assistance. 

EXHIBIT READS: For the 2006-07 program year, 15 RCCs reported that they conducted informal needs 

assessment through communication with state staff; 5 CCs reported that they conducted informal needs 

assessment through communication with RCC staff. 

SOURCE: Center responses to binary interview questions. 

■ CCs assessed state-level needs both through direct communication with states 

and through communication with RCCs. For 2008-09, when an interview 

question asked about their means of assessing state needs, all 5 of the CCs 

reported interacting both with states and with RCCs in order to assess state needs 

(exhibit 3.2). Three of them volunteered a response that did not appear on the 

interview protocol: that they anticipated state-level issues and needs by following 

ED requests and national trends. 

■ CCs also reported continuing to respond to ED requests. In 2008, 4 of 5 CCs 

reported responding to requests made by ED; the same number reported doing so 

in 2006. 

35 
 

Exhibit 3.2. CC assessment of state needs, 2008-09 

Number of CCs 

Means of assessing state needs (N=5) 

Ongoing interaction with states 5 

Communication with RCCs about state needs 5 

Anticipation of issues by following ED initiatives 

and national trends 3 

Survey 1 

NOTE: The response, “anticipation of issues...” applies to three answers volunteered in response 

to a probe for “other” sources of input for planning. 

EXHIBIT READS: For the 2008-09 program year, all 5 CCs reported that they assessed the needs 

of the states through ongoing interaction. 

SOURCE: Center responses to binary interview questions. 

36 

As a program providing services to the states, the Center program was expected to show 

responsiveness to needs and requests for technical assistance. However, Centers might not be in 

a position to respond to every client request. Whether in negotiating their management plans 

with clients or in discussing emerging needs over the course of the program year, Centers did not 

always agree to every request. Center staff provided the reasons for which they turned down 

client requests. 

 

■	 In each year, more than half of the Centers reported that they had turned down 

a client request for assistance. The number of Centers that had turned down one 

or more requests increased within a narrow range over the years, from 12 of the 

21 in 2006-07 (9 of 16 RCCs and 3 of 5 CCs) to 13 of 21 in 2007-08 (9 of 16 

RCCs and 4 of 5 CCs), and 14 of 21 (10 of 16 RCCs and 4 of 5 CCs) in 2008-09. 

■	 Among Centers that declined any client request for assistance in 2008-09, the 

substitution of a different type of assistance was reported by more than half of 

the RCCs but none of the CCs. Among the 10 RCCs that declined one or more 

requests, 7 RCCs worked out a different plan of assistance for the client; none of 

the CCs reported working out an alternate plan (exhibit 3.3). Of the 10 RCCs that 

had declined client requests, 5 RCCs reported that they could not honor a request 

because it fell outside their legitimate scope of work. Among the 4 CCs that 

declined requests, 2 CCs turned down requests that did not fit their priorities or 

their available staff time. 
Exhibit 3.3. Reasons reported for declining technical assistance requests, 

2008-09 

RCCs 

(N=16) 

CCs 

(N=5) 

All 

(N=21) 

The Center did not turn down any requests for service 6 1 7

The Center turned down requests for service: 10 4 14 

Turned down request but the Center and the client instead 

agreed on a plan for related, but different, technical assistance 7 0 7

Reasons for turning down request: 

A request fell outside the legitimate scope of work for a 

Center 5 1 6

A request was potentially legitimate Center work but did not 

fit this Center’s priorities for work with the state 2 2 4

Staff time and resources were already fully committed to 

other work 2 2 4

The Center did not have access to the needed expertise to 

carry out the request 0 1 1

EXHIBIT READS: For the 2008-09 program year, 6 of 16 RCCs did not turn down any requests for service. 

SOURCE: Center responses to standard response categories in interviews. 

Adjustments Made in Ongoing Work 

The Centers’ stated objectives could be found in their written annual management plans, 

but the ways in which they refined their plans might result in departures from these objectives. 

Annual management plans written in advance might in some ways be an asset to the Centers, 

enabling them to mount sustained efforts around their intended programs of work. At the same 

time, however, the plans might impose rigidity on Center technical assistance, impeding a 

flexible response to changing circumstances. The evaluation addressed this matter by assessing 

the extent to which Centers followed or adjusted their initial plans with respect to the topical 

coverage of the technical assistance, as stated in the objectives found in each annual management 

plan in the second and fourth program years (2006-07 and 2008-09). 

■	 Across years, Centers continued to address a majority of the topics they had 

planned to address in each year, with some flexibility to make adaptations. The 

content of the Center management plans was compared with the projects Centers 

reported on the project inventory forms (PIFs) that they had conducted. In both 2006-

07 and 2008-09 Centers delivered technical assistance on a particular topic in at least 

80 percent of the instances in which they stated that topic as part of their management 

plan objectives (80 percent in 2006-07, 84 percent in 2008-09) (exhibit 3.4 and 

appendix exhibit D.1). The analysis showed that of 117 topic-related objectives in the 

2008-09 plans, the Centers provided services and products on 98 of them. The fact 

37 
 

that some planned cases46 (117-98=19) were not listed on the PIFs indicates that 

16 percent of the cases (19 of the planned 117) were not conducted. The 

corresponding figure was 20 percent for 2006-07. Looking at the work actually 

conducted, in 36 percent of cases ([152-98]/152) the Center provided technical 

assistance on a topic not initially cited in its planned objectives in 2008-09. The 

corresponding figure was 38 percent for 2006-07. 

■	 There was continuity across years in the topics most often added to or deleted 

from Center agendas during program years; the topics of English language 

learners and response to intervention were added in more instances than other 

topics, while special education curriculum, instruction, and professional 

development was deleted in more instances than other topics . In Exhibit 3.4, 

comparing columns 3 and 2 provides an indication of topics not included in a 

Center’s management plan but in which technical assistance was delivered by that 

Center (additions). Comparison of columns 1 and 2 indicates topic areas in which 

Centers planned but did not deliver a project (deletions). Eight Centers conducted 

work in 2008-09 related to English language learners but had not included that 

topic among their objectives for the year. The other topics added by the nextlargest number of Centers were response to intervention (7 Centers) and migrant 

education (6 Centers). For 2006-07 the additions found in the largest number of 

Centers were response to intervention (7 Centers), English language learners 

(5 Centers), highly qualified teacher provisions (5 Centers), and supplemental 

educational services (5 Centers). There were 12 of the 22 topics on which Centers 

initially stated an objective but did not deliver a project in 2008-09, and 10 such 

topics in 2006-07. The topic for which work was deleted by the largest number of 

Centers in both years was special education curriculum, instruction, and 

professional development (4 Centers in 2008-09, 5 in 2006-07). 

■	 Adaptations in the topical focus of plans were more common among RCCs than 

CCs. For 2008-09, the RCCs added work on 21 of 22 topics while CCs did so on 

4 of 22 topics (exhibit 3.5). The same difference was found in 2006-07, when the 

figures were 19 of 22 topics for RCCs and 7 of 22 for CCs (appendix exhibit D.2). 

Deletions of planned work also spanned more topics among RCCs than CCs 

across years, being observed for 12 of 22 topics among RCCs in 2008-09 (11 of 

22 in 2006-07) and 3 of 22 topics among CCs in 2008-09 (2 of 22 in 2006-07). 

46 Using each Center’s management plan, the evaluation team coded for each Center whether it had specifically stated an 

intention/objective to address work in each of the 22 topics (i.e., the first column of numbers in exhibit 3.4 indicates the number 

of Centers that stated an objective of conducting any work in each topic, beginning with the topic of state systems of support). 

Using the PIFs, the evaluation team coded project topics to determine the number of Centers that actually did work in each topic 

(i.e., the last column of numbers, by topic, in exhibit 3.4). 

38 
Exhibit 3.4. Topics on which Centers stated objectives and/or delivered 

projects, 2008-09 

Topic 

Number of Centers (N=21) 

Objective on 

topic stated in 

management 

plan 

(1) 

Objective on topic stated 

in management plan AND 

at least one project on 

topic reported on PIF 

(2) 

At least one 

project on 

topic reported 

on PIF 

(3) 

Total cases of a Center setting an 

objective and/or reporting a project 117 98 152

Components of effective systems of 

support--state, district, school 

English language learners 

Highly qualified teacher provisions of NCLB 

Teacher preparation and induction 

Teacher professional development 

Data use /data-driven decision making 

Assessment design 

Formative assessment 

Special education--curriculum, instruction, 

and professional development 

Response to Intervention (RtI) 

Special education--assessment 

High school redesign/reform 

Transition to high school 

Dropout prevention 

Mathematics 

Adolescent literacy 

Reading 

Supplemental educational services (SES) 

Parent involvement 

Migrant education 

Data management compliance 

Indian/Native American education 

16 

9 8

7 

7 

4 

7 

7 

5 

5 

5 4

4 

8 

1 

3 

7 

5 

2 

4 

5 

2 

3 

1 

16

7

5

4

5

5

4

1

4

7

1

3

7

5

2

2

4

1

3

0

17 

16

9 

7 

6 

5 

6 

5 

2 

11

6 

9 

3 

5 

9 

8 

3 

4 

7 

7 

5 

2 

EXHIBIT READS: For the 2008-09 program year, for the topic “Components of effective systems of support—state, 

district, school,” 16 Centers had a related objective in their respective annual management plans; of these, all 16 

reported projects on the topic in their PIF; and a total of 17 Centers had projects on the topic, whether or not they had 

stated an objective related to it in their management plan. 

SOURCES: Center management plans for 2008-09 and PIFs prepared by Centers in consultation with evaluation 

team. 

39 
Exhibit 3.5. Topics on which RCCs and CCs stated objectives and/or 

delivered projects, 2008-09 

Number of RCCs (N=16) Number of CCs (N=5) 

Objective 

on topic in 

management plan 

(1) 

Objective on 

topic in plan 

AND project(s) 

on topic on PIF 

(2) 

Project(s) 

on topic 

on PIF 

(3) 

Objective 

on topic in 

management plan 

(4) 

Objective on 

topic in plan 

AND project(s) 

on topic on PIF 

(5) 

Project(s) 

on topic 

on PIF 

(6) 

Total cases 98 82 130 19 16 22 

Components of effective 

systems of support-state, 

district, school 

English language 

learners 

Highly qualified teacher 

provisions of NCLB 

Teacher professional 

development 

Teacher preparation and 

induction 

Data use / data-driven 

decision making 

Assessment design 

Formative assessment 

Special education–

curriculum, instruction 

and professional 

development 

Response to Intervention 

(RtI) 

Special education–
 

assessment 

High school 

redesign/reform 

Transition to high school 

Dropout prevention 

Mathematics 

Adolescent literacy 

Reading 

Supplemental 

educational services 

(SES) 

Parent involvement 

Migrant education 

Data management

compliance 

Indian/Native American 

education 

15 

7 

6 

6 

3 

6 

6 

4 

4 

5 

2 

7 

0 

2 

6 

4 

1 

3 

5 

2 

3 

1 

15 

6 

6 

5 

3 

4 

5 

3 

1 

4 

2 

6 

0 

2 

6 

4 

1 

1 

4 

1 

3 

0 

16 1 

12 2 

8 

7 1 

5 

4 1 

6 

4 1 

2 

9 0 

4 2 

8 1 

2 1 

4 

8 1 

6 1 

2 1 

3 1 

6 0 

7 

5 0 0 

2 

1 

1 

1 

1 

1 

0 

0 

1 

2 

1 

0 

1 

1 

0 

1 

0 

0 

2 

1 

1 

1 

1 

1 

1 

1 

0 

0 

0 

1 

4

1

0

1

1

0 

1 

0

2

2

1

1 

1 

1 

2 

1 

1 

1 

0 

0

0

EXHIBIT READS: For the 2008-09 program year, for the topic “Components of Effective Systems of Support—State, District, 

School,” 15 RCCs reported a related objective in their management plans; of these, all 15 reported projects on the topic; and 

a total of 16 RCCs had projects on the topic, whether or not they had originally stated an objective related to it. 

SOURCES: Center management plans for 2008-09 and PIFs prepared by Centers in consultation with evaluation team. 

40 
 

The Nature of Center Work 

Centers carried out a variety of technical assistance activities across and within projects. 

Their work reflected change and continuity across years as well as decisions about priorities 

negotiated with clients for each year. We discuss here the types of project investments reported 

by Centers in 2008-09, the topics addressed across all projects in all three years, the activities 

carried out in sampled projects in all three years, and the nature of the sampled projects in 2008-

09 on a high-priority topic, that of statewide systems of support. 

Throughout this section we refer to seven distinct types of activities47, which the team 

identified and defined as follows: (1) ongoing consultation and follow-up through multiple 

service contacts over time, either to fulfill repeated requests or to follow up with individuals who 

participated in another assistance activity; (2) research collections and syntheses distributed by a 

Center, whether developed by that Center or elsewhere; (3) engagement of participants in project 

planning, defined as going beyond needs assessment to include opportunities for at least some 

participants to shape the specific assistance; (4) training events, designed to impart skills and 

equip participants to carry out a particular program or strategy; (5) task force meetings and work 

in which a Center supported a group that was itself constituted as a task force to address a state 

purpose; (6) conferences, defined as single events in which multiple speakers or discussants 

presented information; and (7) support for development of a formal plan to implement a program 

or policy, where the Center supported work by participants in a state. 

Center Investments 

For the fourth program year, 2008-09, interviewers asked the Centers to identify their 

investment priorities with respect to project duration and change, as well as to the types of 

technical assistance activities that were priorities for their clients and themselves. Another 

perspective on these matters was provided by the state managers who were asked in surveys 

about the types of technical assistance that their state received from Centers. 

■	 Center directors reported their largest investments to be in multi-year projects 

following “a long term plan.” This response was given by 13 of the 16 RCCs and 

4 of the 5 CCs (exhibit 3.6). For three RCCs, the greatest investment was in multiyear projects that had undergone “substantial shifts from the originally planned 

participants, activities, or purposes.” One CC reported the greatest investment in 

projects that were self-contained within 2008-09. 

47 At the project level, the cover sheets that Centers submitted along with their materials for expert review provided 

data on the types of activities and resources that the project offered to some or all of its participants. See Chapter 2 

for additional details. 

41 
Exhibit 3.6. Reported investment of Center resources in long-term vs. shortterm projects, 2008-09 

Number of Centers reporting as the 

largest investment of resources
 

for 2008-09
 

RCCs 

(N=16) 

CCs 

(N=5) 

All 

Types of Center projects (N=21) 

Multi-year project(s) that followed a long-term plan 

for participants, activities, and purposes 13 4 17

Multi-year project(s) with substantial shifts from the 

originally planned participants, activities, or 

purposes 

3 0 3 

Projects that were self-contained within a single 

year rather than extending across years 0 1 0

EXHIBIT READS: For the 2008-09 program year, 13 RCCs reported that their largest investment of resources 

was in multi-year projects that followed a long-term plan for participants, activities, and purposes. 

SOURCE: Center responses to standard response categories during interviews. 

When asked in interviews about the changes in Center work from 2006 to 2009, Center 

directors from 6 RCCs and 3 CCs described Center assistance as becoming more long term and 

ongoing, whereas it previously was more likely to focus on delivering discrete activities or 

materials. Among the 16 RCCs, 9 reported their work as increasingly focused on systemic and 

structural issues at the state level, with more efforts to build sustainable structures and processes. 

In addition to a shift in focus, 9 RCCs also described providing more “on-call assistance” to SEA 

leaders to think through and help solve pressing problems or plan for improvement. Four of the 5 

CC directors reported an increase in direct assistance to state or RCC clients, such as consultative 

assistance in applying research. 

Centers could address state needs and priorities in a wide range of ways, and in making 

choices among activities they had to strike balances. In negotiating their management plans and 

project activities, they had to recognize clients’ demand for particular modes of technical 

assistance while using their professional judgment about what would be most productive in 

meeting Center objectives. 

■	 Ongoing consultation was the type of project activity cited by the largest 

numbers of RCCs and CCs as (1) most requested by clients, (2) the largest 

investment, and (3) the most important for achieving Center aims in 2008-09. 

Asked to select from a general list of seven types of project activities and 

resources, the directors of 5 of the 16 RCCs and 2 of the 5 CCs said they made the 

greatest investment of resources in “ongoing consultation” in 2008-09 (exhibit 

3.7). Directors also identified this activity as the one most requested by clients (4 

of the 16 RCCs and 4 of the 5 CCs), and the one most important for achieving 

Center objectives (6 of the 16 RCCs and 3 of the 5 CCs). 

42 
 

 

 

Exhibit 3.7. RCC and CC activities and resources by reported client 

demand, investment, and importance to objectives, 2008-09 

Activities and 

resources 

Most requested by 

clients Largest investment 

Most important to 

achieve objectives 

RCCs CCs

All 

Centers RCCs CCs 

All 

Centers RCCs CCs 

All 

Centers 

Ongoing consultation 

and follow-up 4 4 8 5 2 7 6 3 9 

Research collections 

and syntheses 

Engagement of 

participants in project 

planning 

1 1 2 0 1 1 0 0 0 

2 0 2 3 0 3 4 0 4 

Training events 4 0 4 3 0 3 1 0 1 

Task force meetings 

and work 3 0 3 3 0 3 3 0 3 

Conferences 

Support development 

of a formal plan to 

implement a program 

or policy 

0 0 0 0 1 1 0 1 1 

2 0 2 2 1 3 2 1 3 

 EXHIBIT READS: For the 2008-09 program year, four of the 16 RCCs indicated in interviews that ongoing 

consultation and follow-up was the service most requested by their clients. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials and coded by the 

evaluation team. 

■	 State managers’ survey responses across the three years corroborated the 

Centers’ reports of continuity in the work, and indicated that Centers provided a 

combination of hands-on help and delivery of knowledge resources. The largest 

number of state managers, weighted,48 said in each year that the Centers had 

provided “major” assistance with facilitating work groups, with the percentage 

ranging from 43 to 65 percent across years) (exhibit 3.8). The next most often 

reported type of assistance was hands-on help with designing, delivering, or 

convening professional development for local educators (ranging from 29 to 55 

percent), followed by delivery of information about policies and practices in other 

states (ranging from 28 to 46 percent) or about research findings (ranging from 25 

to 45 percent). 

48 For 20 or more states in each year, the evaluation team received completed responses from more than one state 

manager. As the state was the primary unit of analysis in analyzing data from the state manager survey for this 

report, the state managers’ responses were weighted to ensure that each state was equally represented in all summary 

statistics while taking into account the variation in responses within each state. The weighting procedure, where 

each response was weighted by the inverse of the number of managers responding from that state, ensured that each 

state was equally represented when the team aggregated responses across states to describe the distribution of 

responses. See chapter 2 for a detailed description of the sampling and analysis procedures. 

43 
Exhibit 3.8. State managers’ reports of assistance from Centers, by type of 

assistance, by year 

Percent of state managers 

(weighted) reporting “major” 

assistance from any Center 

Type of assistance	 2006-07 2007-08 2008-09 

Facilitating work groups or committees 43% 56% 65% 

Designing, delivering, or convening PD and 

conferences for local educators 29 45 55

Collecting/disseminating information about 

policies and practices in other states 28 33 46

Synthesizing and disseminating research 

findings 25 44 45

Analyzing data or conducting needs 

assessments 16 35 34

Developing tools for monitoring programs 15 35 32 

Reviewing state plans and policies 21 31 29 

Assisting with a response to federal planning 

and reporting requirements 15 20 21

Completing routine tasks more efficiently 13 12 17 

EXHIBIT READS: For the 2006-07 program year, 43 percent of state managers (weighted) reported that 

their state had received “major” assistance from their RCC or any CC that consisted of facilitating work 

groups or committees. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in 

instances where more than one manager from a state responded. 

Topics of Projects 

As discussed earlier in this chapter, the Centers classified all of their projects for each 

year of the study according to the primary topics addressed, using a list of topics developed for 

this evaluation, to complete their annual project inventory forms (PIFs). The number and percent 

of all projects addressing each topic gives an indication of the substantive priorities identified by 

Centers and their clients in each year. Across 23 topics (including “other”), one was the focus for 

the largest share of projects in each year: 

■	 A prioritized topical focus for Center projects in each year was assistance with 

“systems of support,” addressing support for improvement in struggling schools 

and districts. Among all projects, 19 percent addressed this topic in 2006-07, 

25 percent in 2007-08, and 21 percent in 2008-09 (exhibit 3.9). After “systems of 

support”, the topic area with the second largest number of projects each year 

focused on English language learners (7 percent of all projects in both 2006-07 

and 2007-08, and 10 percent in 2008-09). Center services were otherwise widely 

dispersed across topic areas, and no other topic was the focus of as many as 10 

percent of projects in any year. 

44 
Exhibit 3.9. Percent of projects on PIFs, by topic and by year 

2006-07 2007-08 2008-09 

Total number of projects 365 346 332

Topics Percent of total 

Components of effective systems of 

support—state, district, school 

English language learners 

Highly qualified teacher provisions of 

NCLB 

Teacher professional development 

Teacher preparation and induction 

Data use / data-driven decision making 

Assessment design 

Formative assessment 

Special education–curriculum, 

instruction and professional 

development 

Response to Intervention (RtI) 

Special education–assessment 

High school redesign/reform 

Transition to high school 

Dropout prevention 

Mathematics 

Adolescent literacy 

Reading 

Supplemental educational services 

(SES) 

Parent involvement 

Migrant education 

Data management compliance 

Indian/Native American education 

Other 

19% 

7 

4 

4 

2 

2 

1 

3 

2 

3 

1 

4 

2 

1 

5 

5 

7 

5 

3 

3 

4 

1 

13 

25% 

7 

6 

2 

2 

1 

2 

3 

2 

6 

1 

3 

1 

1 

6 

5 

3 

3 

3 

4 

3 

2 

7 

21%

10 

5

2 

2 

2 

2 

2 

1 

6 

3 

5 

1 

2 

8 

4 

2 

2

3 

3 

3 

2 

9 

EXHIBIT READS: For the topic “Components of effective systems of support—state, district, school,” there 

were 71 projects on all PIFs for the 2006-07 program year, which was 19 percent of the total number of 

projects for that year. 

SOURCE: PIFs prepared by Centers in consultation with evaluation team. 

45 
 

Types of Activities and Resources 

A closer look at the Center projects in each year’s sample provided further insight into 

their scope and the activities they encompassed. For each year, the team’s analysis of sampled 

projects (through procedures described in chapter 2) provides information about the nature of 

Center technical assistance activities or resources. Although the sample of projects is not 

statistically representative of the Centers’ work, project sample selection for review of each 

Center’s work favored the most dominant projects and included more than half (56 percent) of 

the designated major or moderate projects. 

Here we focus particularly on comparing activities across RCCs’ and CCs’ sampled 

projects across years. We first remind readers that a project is a unit of analysis that may 

encompass many different activities: 

■	 Projects entailed multiple activities, with RCC projects averaging between 3 and 

4 activities in each year and CC projects averaging between 2 and 3 activities. 

The mean number of activities in RCC projects was 3.2 in 2006-07, and 3.9 in 

2007-08 and 2008-09. In CC projects, the mean was 2.4 in 2006-07, 2.5. in 2007-

08, and 2.8 in 2008-09. 

Exhibit 3.10. Sampled RCC and CC projects by types of activities and 

resources, by year 

Activities and resources 

Percent of sampled RCC 

projects 

Percent of sampled CC 

projects 

2006-07 

(n=96) 

2007-08 

(n=96) 

2008-09 

(n=93) 

2006-07 

(n=27) 

2007-08 

(n=26) 

2008-09 

(n=26) 

Ongoing consultation and followup 

Research collections and 

syntheses 

Engagement of participants in 

project planning 

Training events 

Task force meetings and work 

Conferences 

Support development of a formal 

plan to implement a program or 

policy 

82% 

54 

45 

43 

50 

27 

19 

93% 

53 

63 

55 

58 

35 

32 

91% 

53 

59 

59 

56 

40 

31 

22% 

74 

30 

37 

7 

63 

7 

15% 

85

31

50

8 

42

15 

62% 

77

27

50 

8 

38 

23 

EXHIBIT READS: For the 2006-07program year, 82 percent of sampled RCC projects included ongoing consultation 

and follow-up. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials. In addition to serving 

as resource material for the expert reviewers, these cover sheets were coded by the evaluation team. 

46 
■	 Across years, the kinds of support provided by the RCCs and CCs differed in 

ways that were consistent with the model of technical assistance envisioned by 

ED. The guidance given by ED through the Center grant competition and 

afterwards laid out a particular structure for the Centers’ work: RCCs would 

specialize in interactions with state clients to provide frontline assistance while 

CCs would specialize in activities that required a content focus. In each year, the 

most common activity found in sampled RCC projects was “ongoing consultation 

and follow-up” (82 percent in 2006-07; 93 percent in 2007-08; and 91 percent in 

2008-09) (exhibit 3.10). In CC sampled projects, it was “research collections and 

synthesis” (74 percent in 2006-07, 85 percent in 2007-08, and 77 percent in 2008-

09), an activity that fewer RCC projects included (53 or 54 percent in each year). 

■	 While the two types of Centers each retained a focus on activities distinctly 

associated with the original program design, their ways of working became more 

similar over the years. For example, in a departure from past CC practice, more 

than half of the sampled CC projects in 2008-09 (62 percent) included ongoing 

consultation and follow-up, an activity found in the great majority of RCC projects 

(at least 82 percent in every year). Conferences, which had been more than twice as 

common in the CC project sample as in the RCC project sample in 2006-07 

(63 percent of projects vs. 27 percent) were found in 38 percent of the CC projects 

and 40 percent of the RCC projects in 2008-09 (exhibit 3.10). 

47 
Projects on “Effective Systems of Support – State, District, School” 

In every year, the topic addressed by the largest number of Center projects was “components of 

effective systems of support—state, district, school.” This included technical assistance with the statewide 

systems of support (SSOS) that SEAs are required to mobilize to help struggling schools and districts 

improve student performance. An SSOS comprises an infrastructure for the delivery of onsite help, and 

strategies and materials designed to guide local improvement. For the sampled projects addressing 

statewide systems of support in 2008-09, further detail is provided about patterns of activities conducted 

and resource materials used. Among the 33 sampled projects on systems of support in 2008-09, Centers 

assisted in one or both of two ways: 

■	 Working with SEA decision makers in redesigning their state’s SSOS infrastructure, 

revising strategies for local improvement, or both (24 projects). SSOS redesign efforts 

included reorganization of the SEA and delineation of responsibilities in the SSOS. 

Revising strategies centered on the content to be delivered, for example, approaches to 

instructional coaching or a framework of indicators to guide school improvement. 

■	 Teaching (or helping teach) support providers in carrying out their work in the field (15 

projects). The content of project training centered on using the SEA improvement and 

planning tools with schools and districts (e.g., planning templates, guides, rubrics, selfassessments). In 12 of these projects, Centers also assisted in developing the state 

materials for use in the field. 

Among the materials that RCCs or CCs employed in their work with SEAs or support providers in 

these projects, the three used most often were published by the Content Center on Innovation and 

Improvement (CII) in 2007: 

■	 “Handbook on Statewide Systems of Support” (9 projects). This 294-page handbook 

reviews research related to SSOS and principles for improving schools. It includes 

chapters on the state role, a conceptual framework for an effective SSOS, case studies 

of RCC support, and tools for assessing SSOS. 

■	 “Strengthening the Statewide System of Support: A Manual for the Comprehensive 

Center and the State Education Agency” (7 projects). This 99-page manual is designed 

to guide an RCC in facilitating SSOS self-assessment with an SEA. 

■	 “Handbook on Restructuring and Substantial School Improvement” (6 projects). This 125-

page handbook offers modules with action principles on district frameworks, school board 

and central office roles, school change, learning-focused leadership, monitoring and 

changing instruction, and systems for improved teaching. It also provides approximately 

150 indicators of success in school restructuring. This manual builds on the publication 

School Restructuring Under NCLB: What Works When?, produced by Public Impact and 

the Center for Comprehensive School Reform and Improvement. 

Working Relationships in Technical Assistance Delivery 

The delivery of technical assistance could be affected by several types of working 

relationships: the coordination of work between RCCs and CCs in the two-tier system; the 

Centers’ working relationships with their clients; and the engagement of knowledgeable 

individuals in quality assurance. Based on the design of the center system, the RCCs and CCs 

48 
were expected to work together to serve state clients, and at the same time CCs were to provide 

technical assistance to the RCCs. Impediments could arise in RCC-CC coordination, in the 

provision of technical assistance to states, and in CCs’ assistance to RCCs. Also, in the process 

of preparing technical assistance activities and resources for delivery to clients, Centers drew on 

various sources of quality assurance. These various aspects of coordination and service delivery 

were examined in the evaluation based on the sampled projects across the three years (2006-07, 

2007-08, and 2008-09). Questions addressing Center investments, processes, and working 

relationships with SEAs and other Centers were added in the last year of data interview 

collection, and thus are only available in 2008-09. 

RCC-CC Coordination 

Charged with working together to assist the states, RCCs and CCs had to communicate 

and coordinate. Cooperation across Centers could be manifested not only in communication 

across organizations but also within specific projects: technical assistance projects of each type 

of Center could potentially incorporate help from the other type of Center. The evaluation 

tracked each of these types of coordination. 

For a description of coordination between CCs and RCCs, each CC director was asked 

about types of activities it had carried out in 2008-09 with at least some of the RCCs. 

Corroboration was sought from RCCs, whose directors were asked whether each of the CCs (by 

name) had coordinated in these ways with them. 

■	 All RCCs and CCs reported communicating at least monthly with at least one 

Center of the other type in 2008-09, but working relationships varied across the 

specific Centers. When asked about their coordination practices, all 16 RCCs 

reported receiving knowledge resources from CCs and communicating at least 

monthly with CCs (exhibit 3.11); all 5 of the CCs also reported these coordination 

practices. However, there was variation in the number of RCCs reporting that 

individual CCs had coordinated with them. For example, 15 RCCs reported 

monthly communication with “CC A,” whereas 7 RCCs reported monthly 

communication with “CC D.” 

■	 All but one of the RCCs reported teaming up with a CC to provide assistance to 

states, but this type of coordinated assistance occurred more often with 

particular CCs. When asked about their coordination in providing assistance to 

states, 15 RCCs reported teaming up with CCs and all 5 CCs reported teaming up 

with RCCs. However, the extent to which this type of coordination occurred 

varied by CC. For example, 14 RCCs reported working with 1 CC while 7 RCCs 

reported working with another CC. 

■	 Communication about promising practices in states was the least common type 

of coordination. Two of the 5 CCs reported soliciting information about 

promising state practices from RCCs. Ten of the 16 RCCs reported offering such 

information to CCs (exhibit 3.11), with 2 to 7 RCCs reportedly communicating 

with each specific CC about promising state practices. 

49 
Exhibit 3.11. RCC-CC coordination reported by Center directors, 2008-09 

Type of coordination 

Number of 

RCCs 

reporting this

type of 

coordination 

with one or 

more CCs 

(N=16) 

Number of RCCs reporting this 

type of coordination with 

individual CCs 

(N=16) 

CC 

A 

CC 

B 

CC 

C 

CC 

D 

CC 

E 

Number of 

CCs 

reporting 

this type of 

coordination 

with one or 

more RCCs 

(N=5) 

RCC received CC knowledge 

resources (materials or 

experts); CC provided 

knowledge resources 

16 16 15 13 15 14 5 

RCC and CC communicated 

at least monthly 

RCC and CC teamed up to 

provide technical assistance 

to states 

16 15 14 10 7 11 5 

RCC provided information 

about promising state 

practices to CC(s); CC 

solicited information about 

promising state practices 

from RCCs 

15 14 7 8 12 9 5 

10 7 6 6 2 6 2 

EXHIBIT READS: For the 2008-09 program year, all 16 RCCs reported that they received knowledge 

resources from one or more of the CCs. Sixteen RCCs also reported this type of coordination with CC “A.” 

SOURCE: Center responses to standard response categories during interviews. 

For insight into the nature of joint RCC-CC work at the project level, the study team 

determined whether each RCC project sampled for the evaluation in each year incorporated 

specific types of input from one or more CCs, and vice versa. 

■	 In all three years, more than half of Center projects were completed by a single 

type of Center, although the types of roles that Centers played when involved in 

each others’ projects were consistent with the design of the Center system. The 

CCs, which were expected to focus on the research in a content area, provided 

substantive materials, assistance, or advice in connection with between 32 percent 

and 48 percent of sampled RCC projects across years (exhibit 3.12). The RCCs, 

which were expected to maintain contact with the states in their regions, recruited 

participants or brokered services for 35 to 38 percent of sampled CC projects 

(exhibit 3.13). 

■	 Across years, there were upward trends in CCs and RCCs providing direct 

technical assistance in the other type of Center’s projects. Among sampled RCC 

projects, the percentage that included direct assistance from CC staff was 

18 percent in 2006-07, 22 percent in 2007-08, and 30 percent in 2008-09 (exhibit 

50 
 

3.12). The corresponding figures for RCC delivery of technical assistance in 

sampled CC projects were 11 percent, 12 percent, and 38 percent (exhibit 3.13). 

Exhibit 3.12. Sampled RCC projects by CC contribution, by year 

Percent of sampled RCC 

projects 

2006-07 

(n=96) 

2007-08 

(n=96) 

2008-09 

(n=93) 

RCC project had no CC contribution 52% 67% 52% 

RCC project had a CC contribution 48 32 47 

CC provided materials used in this RCC project 44 28 35 

CC delivered technical assistance to project 

participants 18 22 30 

CC provided advice to the RCC on this project 17 17 26 

EXHIBIT READS: For the 2006-07 program year, there was no CC contribution in 52 percent of sampled 

RCC projects. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials and coded 

by the evaluation team. 

Exhibit 3.13. Sampled CC projects by RCC contribution, by year 

Percent of sampled CC 

projects 

2006-07 

(n=27) 

2007-08 

(n=26) 

2008-09 

(n=26) 

CC project had no RCC contribution 63% 62% 58% 

CC project had an RCC contribution 37 38 42 

RCC recruited participants or brokered service 37 38 35 

RCC delivered technical assistance to project 

participants 11 12 38

EXHIBIT READS: For the 2006-07 program year, there was no CC contribution in 63 percent of sampled 

CC projects. 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials and coded 

by the evaluation team. 

51 
 

Center Reports of Barriers to Achieving Technical Assistance Aims 

The effective delivery of technical assistance depended on the Centers’ working 

effectively with their clients. Centers had the opportunity to describe in interviews the 

circumstances that might have impeded these working relationships. Both RCCs and CCs 

described the barriers that they perceived in achieving their objectives for assisting states. (In 

chapter 4, Centers’ perceptions are compared with those reported by state managers.) Each type 

of Center, from its own perspective, also addressed barriers impeding the delivery of CC 

technical assistance to RCCs as clients. 

■	 Turnover in staff within state offices or intermediary units was the barrier most 

widely reported by both types of Centers, but reported by RCCs to be the greatest 

barrier. Staffing turnover was reported as a barrier by 10 of the 16 RCCs and 3 of 

the 5 CCs (exhibit 3.14). Six RCCs (but no CCs) identified it as the greatest 

barrier. Turnover at the leadership level was a reported barrier for 8 RCCs and 3 

CCs, and the greatest barrier for 2 RCCs. 

■	 The two greatest barriers CCs reported were policy shifts at the state level and a 

mismatch between state priorities and their own scopes of work. Two of the 

5 CCs reported policy or priority shifts as a barrier, and both of them termed this 

the greatest barrier to achieving their objectives with states (exhibit 3.14). Three 

of the 5 CCs reported that “a state’s most important priorities for assistance fell 

outside the Center’s scope of work,” and 2 of them viewed this as their greatest 

barrier. 

■	 Four RCCs raised an issue that had not been on the list of responses: cuts in 

state staffing and budgets. In response to the prompt, “other,” 4 of the 16 RCCs 

brought up the states’ fiscal difficulties as an impediment to technical assistance, 

noting that state layoffs or hiring freezes were challenges for their assistance plans 

(exhibit 3.14).49 

49 Since the Centers were not specifically prompted with this response option, we cannot say how many others 

would have reported it as a barrier if asked. (An additional 4 RCCs and 1 CC gave a range of different responses to 

the “other” prompt, as shown in the exhibit.) 

52 
1 1

 6

 

Exhibit 3.14. Centers’ perception of barriers to achieving objectives in 

delivering technical assistance to states, 2008-09 

Barrier to achieving technical assistance 

objectives 

No barriers reported to achieving objectives in 

serving states 

A state office, division, or intermediary unit 

experienced turnover in staffing 

Number of Centers reporting: 

Barrier Greatest barrier 

RCC 

(N=16) 

CC 

(N=5) 

RCC 

(N=16) 

1 

10 3

CC 

(N=5) 

1

0

A state experienced turnover in leadership 8 3 2 0 

A policy or priority shift at the state level caused the 

Center’s assistance to be less helpful than it might 8 2 1 2

There was a lack of coordination or communication 

within a state agency 5 1 1 0

State staff did not have time to work with the Center 5 0 1 0 

Cuts in state staff and budgets* 4 0 1 0 

A state would have preferred to locate and contract 

directly with experts or consultants rather than 

working with the Centers 

3 0 1 0 

A state’s most important priorities for assistance fell 

outside the Center’s scope of work 2 3 0 2

A state secured most of the technical assistance it 

needed from other sources 2 1 0 0

The process of negotiating a work scope and 

organizing projects took too long 1 0 0 0

Center staff were not able to spend as much time 

working with a state as the state would have liked 1 0 0 0

State clients placed a higher priority on completing 

short-term work than on addressing long-term 

purposes 

1 0 0 0 

The Center was unable to develop a productive 

working relationship with a state 0 1 0 0

The Center did not have the expertise a state 

needed 0 0 0 0

Other 4 1 2 0 

NOTE: *This response was volunteered by multiple respondents in response to the “other” prompt and was 

re-coded as a separate response. 

EXHIBIT READS: For the 2008-09 program year, One RCC reported no barriers to achieving objectives in 

serving states. Ten RCCs reported that a barrier to achieving its technical assistance objectives was that a 

state office, division, or intermediary unit experienced a turnover in staffing. 

SOURCE: Center responses to standard response categories during interviews. 

53 
Turning to the technical assistance relationship between the two types of Centers, the 

CCs described the barriers that they saw as impeding their assistance to RCCs, and the RCCs 

reported on barriers in this same relationship from their perspective as the client. 

■	 The greatest barriers reported in the RCC-CC technical assistance relationship 

were limitations of the CC’s scope of work and staff capacity, and the issue of 

shifts in state policies or priorities. Four of the 5 CCs reported that “RCCs’ most 

important priorities for assistance fell outside the CCs’ scope of work,” and 2 of 

them called this the greatest barrier (exhibit 3.15). Seven of the 16 RCCs reported 

that this was a barrier, with 4 calling it the greatest barrier. Five of the 16 RCCs 

reported that, “CC staff were not able to spend as much time working with us as 

we would have liked,” and 3 of them named it as the greatest barrier. None of the 

CCs reported this as an impediment to achieving its technical assistance 

objectives with RCCs. For 2 CCs, shifts in state policies or priorities were said to 

impede not only their technical assistance to states but also their technical 

assistance to the RCCs working with states. Both reported this barrier as the 

greatest one they encountered in assisting RCCs. One of the 16 RCCs reported 

this as a barrier to receiving assistance from CCs, but not the greatest barrier. 

■	 Four RCCs reported no barriers to receiving technical assistance from CCs. 

However, all 5 CCs did identify barriers (exhibit 3.15). 

54 
Exhibit 3.15. Barriers to CCs’ assistance to RCCs, 2008-09, perceived by RCCs 

and CCs 

Barrier to delivery/ receipt of technical assistance 

Number of Centers reporting 

Barrier Greatest barrier

RCC CC 

(N=16) (N=5) 

RCC 

(N=16) 

CC 

(N=5) 

No barriers reported to receiving assistance 4 0 4 0 

RCCs’ most important priorities for assistance fell 

outside the CCs’ scope of work 7 4 4 2

CC staff were not able to spend as much time working 

with RCCs as the RCCs would have liked 5 0 3 0

The RCC and CC were unable to develop a productive 

working relationship 3 0 0 0

RCCs secured most of the technical assistance they 

needed from other sources 2 1 2 0

CCs did not have the expertise RCCs needed 2 1 0 0 

A policy or priority shift at the state caused the CCs’ 

assistance to RCCs to be less helpful than it might 1 2 0 2

There was a lack of coordination or communication 

within an RCC 1 2 0 0

RCC staff did not have time to work with CCs 1 0 1 0 

RCCs experienced turnover in staffing 0 1 0 0 

RCCs placed a higher priority on completing shortterm work than on addressing long-term purposes 0 1 0 0

The process of negotiating a work scope and 

organizing projects took too long 0 0 0 0

RCCs would have preferred to locate and contract 

directly with experts or consultants rather than working 

with the CCs 

0 0 0 0 

RCCs experienced turnover in leadership 0 0 0 0 

EXHIBIT READS: For the 2008-09 program year, four RCCs reported no barriers to achieving receiving 

technical assistance from CCs. Seven RCCs reported that a barrier to the receipt of technical assistance 

was that the RCCs’ most important priorities for assistance fell outside the CCs’ scope of work. 

SOURCE: Center responses to standard response categories during interviews. 

Quality Assurance Procedures 

Finally, a potentially important aspect of Center operations was the steps taken to ensure 

quality in the technical assistance delivered. Depending on what a Center provided to clients, 

different approaches to quality assurance might be feasible: in particular, materials would lend 

55 
themselves more readily to formal review before delivery, compared with services. Recognizing 

that there were differences in the mix of resources and activities delivered across projects and 

across Centers, the team gathered Centers’ self-reports on the particular sources of expertise 

brought to bear in quality assurance for each project in the 2008-09 sample, selected from a list 

of possible sources. In addition, the interviews with Center directors addressed their standard 

procedures, if any, for quality assurance. 

■	 While most RCC and CC projects were reviewed in-house, many of the CC 

projects also underwent review by external experts and by ED. Review by 

Center staff was reported for more than 90 percent of projects (92 percent of RCC 

projects, 96 percent of CC projects) (exhibit 3.16). Just under half of all projects 

were reviewed by other staff, not regularly employed by the Center, in the lead 

grantee organization (47 percent of RCC projects, 31 percent of CC projects) or in 

subgrantee organizations (38 percent of RCC projects, 58 percent of CC projects). 

For 85 percent of the sampled CC projects, experts were retained to review 

materials; for 81 percent, ED provided a review. The corresponding figures for 

RCC projects were lower, at 44 percent and 17 percent respectively. 

Exhibit 3.16. Sources of expertise used in quality assurance, by Center type, 

2008-09 

Percent of sampled Center projects 

RCC 

projects 

(n=93) 

CC 

projects 

(n=26) 

All 

projects 

Source (n=119) 

Internal Center staff (of lead grantee organization 

and/or subgrantees)
 92% 96% 93%

Other staff in the lead organization, not formally 

employed by the Center 47 31 44

Outside experts retained to review drafts/materials 44 85 53 

Content Center(s) 39 15 34 

Other staff in subgrantee organization(s) 38 58 42 

U.S. Department of Education	 17 81 31 

EXHIBIT READS: For the 2008-09 program year, 92 percent of RCC projects reportedly underwent review by 

internal Center staff prior to delivery 

SOURCE: Project cover sheets prepared by Centers for the expert review of project materials and coded by the 

evaluation team. 

When Center directors were asked in interviews whether they had “a formal process” for 

“quality assurance in technical assistance during 2008-09,” all but 1 of the 21 (an RCC director) 

said yes. The specific procedures Center directors reported using for quality assurance differed 

for products and services. Across the 14 Centers who described their in-house product reviews 

56 
(10 of the 16 RCCs and 4 of the 5 CCs), Centers explained the two types of pre-release reviews 

conducted for the products they distributed in print or electronic form. Most often, in-house 

reviews targeted formatting and style, such as copy editing (7 RCCs and 3 CCs). Centers also 

reported using one or more of the following sources for conducting substantive internal reviews 

(9 RCCs and 3CCs): knowledgeable Center staff members, Center directors personally reviewing 

all products, or staff members from the grantee or subgrantee organization who were not on the 

Center staff. 

For services, a detailed review prior to delivery would not be feasible in the same way 

that it would for a product, and Centers instead described cycles of in-house review informed by 

evaluative feedback from clients. As a part of these cycles of review, client feedback was 

gathered systematically by internal or external evaluators or gathered by the technical assistance 

providers in service reviews with clients. A total of 10 Centers (9 RCCs and 1 CC) cited client 

feedback (e.g., event evaluations) as a source of input to quality assurance. 

57 

 

4. State Managers’ Assessment of Center 

Technical Assistance 

The Centers were expected to provide technical assistance that would build states’ 

capacity to carry out their responsibilities under NCLB, supporting district and school efforts to 

close achievement gaps and raise student achievement. Thus, the perceptions of Center 

assistance expressed by state agency managers provide a relevant perspective on the outputs of 

the overall Center program. These managers were in a position to view the planning, operations, 

and state agency perceptions of the technical assistance provided by both the RCCs and the CCs. 

This evaluation inquired into the extent to which Center technical assistance served state 

purposes and, according to state managers, built state capacity. 

 

This chapter examines how the work of the Centers was regarded by senior managers in 

state education agencies, addressing the following research questions: 

■	 What was the performance of the Comprehensive Centers in addressing state 

needs and priorities? How did their performance change over the period of time 

studied? 

■	 How did the Centers’ state clients define their needs and priorities? 

■	 To what extent, as reported by states, did Center assistance expand state 

capacity to address underlying needs and priorities and meet the goals of 

NCLB? 

■	 To what extent did states rely on other sources of technical assistance 

besides the Centers? What were other sources of technical assistance that 

states used? How did the usefulness of Center assistance compare with the 

usefulness of assistance from other sources? 

The analysis in this chapter draws on information gathered through the survey of state 

managers administered in each of the three data collection year as well as interviews with Center 

staff conducted by the evaluation team. 50 

Extent to Which Center Assistance Served State Purposes 

ED placed a significant emphasis on having the Centers, both RCCs and CCs, deliver 

technical assistance that would advance state efforts to implement NCLB. In developing their 

work plans and delivering technical assistance, the Centers were expected to target their work on 

50 A detailed description of the changes to the state manager survey between the 2006-07 and 2007-08 

administration cycles as well as copies of both survey instruments can be found in appendix E. Supplemental tables 

related to this chapter are in appendix F. 

59 
 

 

 

It did not serve the state’s Percent of state managers (weighted) 

purposes
 

100
 

36

47

56

52

40

36

6 7 

6 6 6 3 It was a good start, but some 

important priorities were 

not addressed 80 

It was a good start 

60 

40 

It served the state’s 

purposes completely 

20 

0 

2006-07 2007-08 2008-09 

(n=56) (n=57) (n=54) 

 

 

 

state concerns and priorities. States’ views of how well the assistance met their own purposes 

provided one perspective on the success of the program. 

All state managers responding to the survey indicated the extent to which Center 

technical assistance had served their state purposes, in each year of the study. Further elaboration 

was provided by those who reported that this assistance had served state purposes less than 

completely: 

■	 More state managers in each year reported that Center technical assistance 

addressed their states’ purposes. The percentage reporting that Center technical 

assistance had “served the state’s purposes completely” was 36 percent for 2006-

07, 47 percent for 2007-08, and 56 percent for 2008-09 (exhibit 4.1).51 

Exhibit 4.1. Extent to which technical assistance from the Centers served state 

purposes, as judged by state managers, by year 

EXHIBIT READS: For the 2006-07 program year, 36 percent of state managers, weighted, reported 

that Center technical assistance served the state’s purposes completely. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally 

represented in instances where more than one manager from a state responded. 

51 The survey of state managers was administered to managers in a census of all state departments of education. 

Therefore, difference in the distribution of state responses to the survey across time is presented without inferential 

statistics or assessments of statistical significance. 

60 
■	 Over the three years, state managers reported greater interest in seeking more 

technical assistance from Centers. The survey posed a series of follow-up 

questions asking why, to the respondents who did not report that Center assistance 

“served the state’s purposes completely” (exhibit 4.2).52 These state managers 

were asked to check off as many as eight possible reasons that the assistance was 

less helpful to their state than it might have been. In 2008-09, the most frequent 

response among the subgroup of state managers whose purposes were not 

completely served was, “Center staff are not able to spend as much time working 

with the state as we would like.” The frequency of this response increased over 

time: it was selected by 17 percent of these respondents in 2006-07, 27 percent in 

2007-08, and 43 percent in 2008-09.53 (These percentages of the subgroup 

represented 9, 10, and 16 percent of all state managers, weighted, respectively.) 

■	 State managers’ reservations about Center technical assistance were not the 

same issues that Center directors saw as barriers to meeting their aims. As 

discussed in chapter 3, the Center directors were asked to indicate what had 

impeded their assistance to states in 2008-09, choosing from a list that included 

all the responses that also appeared on the state manager survey. For the 20 state 

managers (weighted) reporting Center assistance had served the state’s purposes 

less than completely, the issue reported by the highest percentage of state 

managers (43 percent) was that “Center staff are not able to spend as much time 

working with the state as we would like” while the top issue for Center directors 

(48 percent) was that “a policy or priority shift at the state level caused the 

Center’s assistance to be less helpful than it might be” (exhibit 4.2). 

 

 

 

52 In each reporting year, a small number of state managers who indicated they were not fully satisfied with the 

technical assistance they received from the Centers failed to answer the follow-up question asking why they were 

less than satisfied. In 2006-07, 64 percent of state managers were less than fully satisfied and 53 percent provided 

responses to the follow-up question. These percentages were 53 percent and 37 percent for 2007-08 and 44 percent 

and 37 percent for 2008-09.

53 The percentage of state managers reporting that their state’s purposes were not completely served varied by year. 

Thus, for the follow-up question (“reasons the technical assistance has been less helpful than it might be”), 

comparison of percentages from year to year may include variation in responses over time as well as changes in 

respondents addressing the question. 

61 
 

 

 

 

 

 

Exhibit 4.2. Reasons why Center assistance served some states’ purposes less 

than completely, by year, as reported by state managers, and 

barriers reported by Center directors to achieving Center aims, 2008-

09 

Reasons/Barriers 

Percent of those state managers 

(weighted) who reported Center 

assistance had not served their 

purposes completely 

2006-07 

(Total n=30) 

2007-08 

(Total n=21) 

2008-09 

(Total n=20) 

Percent of all 

Center directors 

reporting 

barrier 

2008-09 

(N=21) 

Center staff are not able to spend as much time 

working with the state as we would like 17% 27% 43% 5% 

The state secures most of the technical 

assistance it needs from other sources 31 24 27 14 

The process of negotiating a work scope and 

organizing projects takes too long 33 31 25 5 

The state’s most important priorities for 

assistance fall outside the Center’s scope of 

work 

33 35 22 24 

The Center does not have the expertise the state 

needs 15 31 18 0 

The state has been unable to develop a 

productive working relationship with the Center 3 15 15 5 

The state would prefer to locate and contract 

directly with the experts or consultants from 

whom it needs assistance, rather than working 

through the Centers 

19 36 5 14 

A policy or priority shift at the state level caused 

the Center’s assistance to be less helpful than it 

might 

7 13 4 48 

NOTE: The first three columns display the percentage of state managers (weighted so that each state was equally 

represented) reporting reasons the technical assistance has been less helpful to the state than it might have been. 

Based on Center director interview responses, the last column on the far right displays the percentage of Center 

directors who reported each to be a barrier to the Center achieving its aim. 

EXHIBIT READS: For the 2006-07 program year, of the 30 state managers, weighted, reporting Center assistance 

served their state’s purposes less than completely, 17 percent said a reason was that the Center staff were not able 

to spend as much time working with the state as they would like. 

SOURCE: Survey of State Managers and Center director responses to standard response categories during a phone 

interview. State respondents limited to those who did not answer “It served our purposes completely” to the preceding 

question. Responses to the state manager survey weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

62 
 

 

Reported Assistance Delivery and Capacity Building for State 

NCLB Responsibilities 

The primary aim of the Center program was to deliver technical assistance that would 

both address state priorities in implementing NCLB and build state capacity for NCLB 

implementation. Senior state managers reported their perceptions of the extent to which the 

Centers had delivered assistance and built capacity in the areas of NCLB responsibility that were 

technical assistance priorities for their state. 

■	 State managers consistently placed a priority on assistance with statewide 

systems of support or school support teams. When asked about state 

responsibilities with which they wanted technical assistance from any source (not 

just from the Centers), more than 90 percent of managers, weighted, (95 percent 

in 2007-08 and 94 percent in 2008-09) reported these functions to be a major or 

moderate priority for assistance in 2007-08 and 2008-09 (exhibit 4.3). In 2006-07, 

with a differently worded survey question, the responses were generally 

consistent: 98 percent reported that “building or managing a statewide system of 

support for districts and schools identified for improvement under NCLB” was a 

major or moderate priority for assistance, and 90 percent said the same for 

“training or managing school support teams.” 

■	 The other leading areas of priority for technical assistance were policies and 

practices for English language learners; state assessment and accountability 

systems; and research-based curriculum, instruction, or professional 

development in academic subjects. For each area, between 73 and 74 percent of 

state managers in 2007-08 and between 75 and 79 percent of state managers in 

2008-09 identified these areas of state responsibility as a major or moderate 

priority for technical assistance. (exhibit 4.3). 

■	 State managers reported that Centers delivered technical assistance in an area 

they had identified to be a major or moderate state priority for technical 

assistance. This analysis focused on those managers who reported that their state 

prioritized technical assistance with a particular NCLB responsibility. Looking at 

the top four priorities for technical assistance reported by state managers, in 2008-

09 at least 66 percent of those managers, weighted, who named a particular 

priority also reported that the Centers had delivered assistance with that area (at 

least 77 percent in 2007-08; exhibit 4.3).54 For state systems of support and school 

support teams, the most widely reported priority across states, the percentage 

exceeded 90 percent in each year (94 percent in 2007-08 and 91 percent in 2008-

09). The exceptions to the overall pattern of assistance delivery were found in the 

two least widely reported priorities, supplemental service and choice provisions 

54 The subgroup of state managers who identified a particular area of state responsibility to be a major or moderate 

priority for technical assistance varied by year. Thus, for the follow-up question about the priority areas in which 

states received technical assistance from Centers, comparison of the percentages from year to year may include 

variation in responses over time as well as changes in respondents addressing the question. 

63 
 

 

and communication with parents, where the percentages reporting help from the 

Centers were below 55 percent in both 2007-08 and 2008-09 (exhibit 4.3). 

■	 State reporting of Center’s delivery of technical assistance across state priorities 

under NCLB declined from 2007-08 to 2008-09. With just one exception, 

communication with parents or the public, the state managers who placed at least 

a moderate priority on a particular area of NCLB state responsibility were less 

likely to report Center help with that responsibility in 2008-09 than in the 

previous year. The three largest drops from 2007-08 to 2008-09 were in the 

reports that Centers had provided assistance with educators’ use of assessment 

data (from 83 percent to 58 percent); research-based curriculum, instruction, or 

professional development in academic subjects (from 88 percent to 72 percent); 

and monitoring compliance with NCLB requirements (from 80 percent to 64 

percent) (exhibit 4.3).55 

The drop in NCLB-related technical assistance was not accompanied, however, by an 

overall report that the Centers were failing to address state purposes. As discussed above, the 

weighted proportion of state managers reporting that Center assistance had served state purposes 

“completely” rose from 47 percent to 56 percent over this time (exhibit 4.1). As well, more state 

managers reported at least moderate contributions of Center technical assistance to increasing 

their state’s capacity in areas identified as state priorities: 

■	 State managers’ perceptions of the Centers’ contributions to state capacity 

increased from 2007-08 to 2008-09 in the four NCLB-related areas of highest 

priority. Again, the focus here is on those managers who reported a particular 

responsibility as a technical-assistance priority. When asked to rate the extent to 

which Center assistance had expanded the state’s capacity to carry out state 

NCLB responsibilities, a higher percentage of those state managers who had 

identified each area of responsibility as a priority credited the Centers with great 

or moderate contributions to state capacity in 2008-09 than in 2007-08 in 

statewide systems of support (72 percent in 2007-08 and 82 percent in 2008-09), 

English language learners (59 and 73 percent), state assessment and 

accountability systems (57 and 59 percent), and research-based curriculum, 

instruction, and professional development (64 and 77 percent) (exhibit 4.4).56 

55 Percentages are based on the state manager respondents who identified a particular area of state responsibility to 

be a major or moderate technical assistance priority, which varied by year. Thus, for the follow-up question about 

the priority areas in which states received technical assistance from Centers, comparison of the percentages from 

year to year may include variation in responses over time as well as changes in respondents addressing the question. 56 The subgroup of state managers who identified a particular area of state responsibility to be a major or moderate 

priority for technical assistance varied by year. Thus, for the follow-up question about state reported capacity 

building, comparison of the percentages from year to year may include variation in responses over time as well as 

changes in respondents addressing the question. 

64 
Exhibit 4.3. State priorities for technical assistance and assistance received from 

Centers, 2007-08 and 2008-09 

Area of state responsibility 

under NCLB 

Percent of state managers (weighted) 

2007-08 2008-09 

Reporting 

responsibility as a 

major or 

moderate 

priority 

Responsibility 

is major or 

moderate 

priority and 

Center technical 

assistance 

received in that 

area 

Reporting 

responsibility as a 

major or 

moderate 

priority 

Responsibility 

is major or 

moderate 

priority and 

Center technical 

assistance 

received in that 

area 

Statewide systems of support 

or school support teams 

(n=56; n=52) 

Policies and practices for 

English language learners 

(n=55; n=51) 

State assessment and 

accountability systems 

(n=53; n=50) 

Development or dissemination 

of research-based curriculum, 

instruction, or professional 

development programs in 

academic subject(s) 

(n=54; n=51) 

Assistance with educators’ use 

of assessment data 

(n=54; n=51) 

Monitoring compliance with 

NCLB requirements 

(n=53; n=51) 

Administering supplemental 

educational services (SES) and 

choice provisions 

(n=54; n=50) 

Communication with parents or 

the public 

(n=54; n=50) 

95% 

74 

74 

73 

68 

64

49 

47 

94% 

79 

77 

88 

83 

80 

52 

41 

94% 

79

79 

75

69 

59 

48 

49 

91% 

76 

66 

72 

58 

64 

43 

54 

NOTE: Percentages are based on the state manager respondents who rated each area a major or moderate 

technical assistance priority, which varied by year. 

EXHIBIT READS: For the 2007-08 program year, 95 percent of state managers, weighted, reported that the NCLB 

area of state responsibility of statewide systems of support or school support teams was a major or moderate priority 

for technical assistance for their state. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

65 
Exhibit 4.4. Extent to which Center assistance expanded state capacity in 

priority areas, as judged by state managers who rated the area as 

a major or moderate technical assistance priority, by year 

Percent reporting 

capacity expanded by a 

great or moderate extent 

Area of state responsibility under NCLB 2007-08 2008-09 

Statewide systems of support or school support teams 

(n=56, n=50)
 72% 82%

Policies and practices for English language learners 

(n=43, n=40) 59 73

State assessment and accountability systems 

(n=42, n=39) 57 59

Development or dissemination of research-based 

curriculum, instruction, or professional development 

programs in academic subject(s) 

(n=41, n=39) 

64 77

Assistance with educators’ use of assessment data
 

(n=37, n=36) 62 61

Monitoring compliance with NCLB requirements 

(n=35, n=30) 57 57

Administering supplemental educational services (SES) 

and choice provisions (n=25, n=26) 44 39

Communication with parents or the public 

(n=25, n=26) 48 50

NOTE: Percentages are based on the state manager respondents who rated each area a major or moderate 

technical assistance priority, which varied by year. State managers who chose the response, “Does not 

apply, or not able to judge,” were included in the denominator of the percent calculation. 

EXHIBIT READS: For the 2007-08 program year, among the 56 state managers (weighted) who reported 

that technical assistance in statewide systems of support or school support teams was a major or moderate 

priority for their state, 72 percent reported that technical assistance received from the Centers expanded the 

state’s capacity in this area to a great or moderate extent. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in 

instances where more than one manager from a state responded. 

While the analyses just described focus on reports of technical assistance delivery and 

capacity building in the specific areas of state responsibility that particular state managers 

identified as priorities, it is also useful to know the extent to which managers in all states said 

technical assistance from the Centers had expanded state capacity, irrespective of the priority the 

state accorded to that area. 

■	 The percent of all state managers, weighted, reporting that assistance from the 

Centers had increased their state’s capacity to a “great” extent in at least one 

area of state NCLB responsibilities increased from 62 percent in 2007-08 to 75 

percent in 2008-09. The percent reporting either a “great” or a “moderate” 

66 
expansion of capacity from Center assistance in at least one area rose from 85 

percent to 96 percent. 

Capacity building at the state level can take many forms, and to help illustrate some of 

them the textbox below provides a summary from case studies, “Illustrative Examples of 

Reported Capacity Building Activities.” A description of this data collection is included in 

Appendix E. 

Illustrative Examples of Reported Capacity Building Activities 

As a supplementary data collection, we interviewed SEA staff in 10 states and relevant Center staff 

about the technical assistance provided on SSOS, the most common area for technical assistance. 

The states were selected to maximize the number of Centers included (a total of 10 RCCs and 3 

CCs) and to represent the full range of SEA reported experiences (i.e., cases include SEAs whose 

staff reported capacity was expanded “to a great extent,” “to a moderate extent,” and “to a small 

extent”). The interviews elicited information about the capacity building support provided by the 

Centers and examples of what SEA staff perceived as the benefits of such assistance. The 

interviews focused specifically on three potential capacity building areas: i) technology, systems 

and infrastructure improvements; ii) SEA staff knowledge/skill enhancements; iii) access to external 

support/expertise. The following illustrative examples are not intended to be representative of all 

Center work and provide no indication of the extent to which the following descriptions are 

representative of all states. 

Although SEA respondents commonly highlighted Center support with tool development and use, 

resulting assessments of these efforts varied. Products developed through Center assistance 

included things such as on-line application for districts to submit annual improvement plans that 

SEA staff could access for monitoring purposes, a data system with tools to facilitate district use of 

data for needs-assessment purposes, and a web-based store of resources, training materials, 

research materials, and other SSOS information. A couple of the interviewed SEAs reported that 

the developed tools improved SEA efficiency, while other SEAs reported the tools as useful but 

less integral to subsequent SEA activities in part because staff were not fully implementing the 

tools. 

Working with Centers to refine state processes for supporting districts and schools in need of 

improvement was another Center activity highlighted by some interviewed SEAs. This work 

included things like establishing district and school support teams and training the support 

providers. SEAs also reported working with the Centers to help improve inter-departmental 

communication and coordination through Center facilitated meetings. 

Finally, Centers reported providing SEA staff with knowledge and skills in order to potentially build 

SEA capacity. Such activities included access to synthesized research, best practices, policy briefs, 

and innovative approaches or SSOS models used by other states. Some SEA staff reported 

participation in RCC-led regional professional development workshops. 

State Uses and Perceptions of the Centers Compared with 

Other Sources of Technical Assistance 

With many other sources available to states, the Centers were designed to fill niches—in 

particular, that of building capacity for state implementation of NCLB requirements—rather than 

addressing every purpose for which states might seek technical assistance. The responses of state 

67 
managers confirmed that the states relied upon multiple sources of technical assistance in their 

practice in all three years and that they used these sources for different purposes. 

■	 Among these state managers, the Centers were the source most often reported 

to be used to a “great” or “moderate” extent. More than 80 percent named the 

Centers as a technical assistance resource used at least to a moderate extent in 

both years, 2007-08 and 2008-09 (exhibit 4.5). More than three-quarters also 

cited their Regional Educational Laboratory as at least a moderate source in 

2007-08 (76 percent) and professional associations in 2008-09 (79 percent). In 

2006-07, with a differently worded question, the Centers were also ranked high 

as an assistance source: they were one of the top three sources, along with 

professional associations and Regional Educational Laboratories (appendix 

exhibit F.5). 

Exhibit 4.5. States’ use of external sources of technical assistance, by year 

Percent of state managers 

(weighted) reporting to a great 

or moderate extent: 

2007-08 

(n=57) 

2008-09 

Source of technical assistance (n=50) 

Comprehensive Center network 87% 82% 

Regional Educational Laboratory 76 74 

U.S. Department of Education 64 66 

Professional associations 

(e.g., CCSSO, ASCD) 64 79

Senior managers in other SEAs 59 64 

Consulting firms or private 

contractors 56 51

Colleges and universities 46 52 

EXHIBIT READS: For the 2007-08 program year, 87 percent of state managers, 

weighted, said that they relied on the Centers as a source of technical assistance to a 

great or moderate extent. 

SOURCE: Survey of State Managers. Responses weighted so that each state was 

equally represented in instances where more than one manager from a state 

responded. 

State managers’ reports of the purposes for which they used each source of technical 

assistance shed light on the niches occupied by the Center program—the purposes for which the 

Centers were the most commonly used source. These responses also revealed some purposes for 

which other sources were the most widely used in particular years. 

■	 Across different purposes for technical assistance, state managers identified the 

Centers as the leading source for planning, completing tasks, and developing 

skills. In 2008-09, the Centers were cited as the top source “to plan the initial 

68 
steps in solving a problem” (59 percent), to complete tasks that either “the state 

could do itself if it had more staff or resources” (64 percent) or “for which the 

state lacks expertise” (53 percent), and “to develop the skills of SEA or 

intermediate education agency staff” (55 percent). While not the top source, the 

Centers were cited second to professional associations as a resource states used 

“to gather information or to keep current with new ideas” (73 percent compared 

with 79 percent) (exhibit 4.6). 

Exhibit 4.6. Purposes for which states used external sources of technical 

assistance, 2008-09 

Purpose in seeking technical assistance 

(percent of state managers) 

(n=54) 

External Source 

Comprehensive Center network 

kee

T

p

o

c

g

u

a

r

t

r

h

e

e

n

r

t

i

w

nf

it

o

h

r

n

a

ew

tio

i

n

de

o

a

r

s 

t

m

T

i

o

n solv

o

r o

e

r p

m

th

d

a

n

a

g

h

la

in

p

73% 

h

a

s

ic

t

it

e

h

e

c

t

o

r u

w

le

d

o

p

ld

s

m

u

re

s 

o

o

r

p

c

c

e

o

o

t

f

te

f

s

T

l

a

ta

t

a

m

s

s

ti

e

i

l

e

n

b

i

o

59% 

o

l p

px

AE

m

e

S

o

s

c

k

o

ca

e 

T

l

h

te

t t

ta

a

s

h

it 

t

f

e

s

i

th

k

lf

s

e

s 

64% 

c

p

u

ta

lo

d

s

e

e

y

v

e

c

e

t

d

n

ia

r 

e

o

o

d

g

e

a

f

T

s

m

k

r

s

te

a

e 

n

t

s

i

ti

r

te

r

e

e

53% 

s

d

a

s

rk

d

o

n

w

a s

o

t

f 

T

cri

o

t s

ls

i

il

d

k

n 

s e

io

h

t a

f 

t

f

55% 

d

s

te

i h

ta

t

sr

m

u

o fr

h 

O

it

ec

w

n

y

s 

ta

tl

l

s

c

o

i

e

o

s

h

ir

c

32% 

k ees ’t

e 

n

c

s

r

e

u

o

os

2% 

Regional Educational Laboratory 66 44 34 38 30 19 9 

U.S. Department of Education 58 43 14 18 23 7 11 

Professional associations (e.g., 

CCSSO, ASCD) 79 23 20 20 33 9 4 

Senior managers in other SEAs 63 36 7 7 14 5 10 

Consulting firms or private 29 31 50 42 25 32 21 contractors 

Colleges and universities 50 29 26 21 19 25 20 

EXHIBIT READS: For the 2008-09 program year, 73 percent of state managers (weighted) said that they turned to 

the Centers to gather information or to keep current with new ideas. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

■	 Over time, the Centers were reported by state managers as the top resource for 

more purposes. In 2006-07, state managers reported Centers as the top source in 

two areas: for planning the initial steps in solving a problem (66 percent) and 

developing skills of SEA staff (61 percent). In the following years (2007-08 and 

2008-09), Centers were reported as the top source for two additional purposes: to 

help states complete tasks where they lacked resources (58 percent and 64 

percent, respectively) or expertise (49 percent and 53 percent, respectively). 

69 
State managers also provided a rating of the usefulness of Center technical assistance, in 

comparison with other sources, with respect to particular areas of NCLB state responsibility. 

■	 In the area of statewide systems of support or school support teams, a majority 

of state managers (62 percent, weighted) called the Centers’ technical 

assistance “more useful” than assistance from other sources in 2008-09 (exhibit 

4.7). This was the only area in which more than 80 percent of state managers said 

they could make a judgment on usefulness. The pattern of responses was similar 

in 2007-08 (appendix exhibit F.9). 

Exhibit 4.7. Usefulness of Center assistance compared with 

assistance from other sources, 2008-09 

Percent of state managers, weighted, rating 

Center technical assistance as: 

More 

useful 

About the 

same 

Less 

useful 

Not able to 

Areas of state responsibility under NCLB judge 

Statewide systems of support or school support 

teams (n=49)
 62% 21% 4% 13%

Policies and practices for English language 

learners (n=46)
 35 29 4 32

State assessment and accountability systems 

(n=41) 23 28 6 42

Development or dissemination of research-based 

curriculum, instruction, or professional development 

programs in academic subject(s) (n=45) 

41 35 2 22 

Assistance with educators’ use of assessment data 

(n=41) 28 37 14 21

Monitoring compliance with NCLB requirements 

(n=38) 43 19 7 32

Administering supplemental educational services 

(SES) and choice provisions (n=32)
 16 16 11 58

Communication with parents or the public (n=34) 27 25 3 45
 

NOTE: States that chose the response, “not applicable, state has not sought assistance for this purpose,” were 

excluded from the analysis. 

EXHIBIT READS: For the 2008-09 program year, 62 percent of state managers (weighted) reported that Centers 

were more useful than other sources of technical assistance for the state NCLB responsibility of statewide systems of 

support or school support teams. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

70 
 

 

 

SEA Overall Rating of Center Assistance 

As senior officials in their agencies, the state managers were in a position to provide 

overall judgments of the Center technical assistance their agency had received. The state 

manager survey included scales for the degree to which Center technical assistance, in general 

(not specific to a particular project), was seen as relevant and useful, comprising five indicators 

of relevance and five indicators of usefulness.57 Ratings were based on a 1 to 5 scale.58 

■	 On a scale of 1 (very low degree) to 5 (very high degree), on average, state 

managers rated the Centers’ technical assistance about a 4 (“high” degree) on 

relevance and usefulness in each year. The average relevance scores in each year 

were 3.94 in 2006-07, 3.92 in 2007-08, and 4.07 in 2008-09, and the average 

usefulness scores were 3.86 in 2006-07, 3.99 in 2007-08, and 4.21 in 2008-09.59 

■	 The 2008-09 ratings of relevance and usefulness from the survey of state 

managers suggest that Centers were targeting their assistance more closely on 

key priorities and providing resources that states could continue using. Among 

the individual components of the relevance index, state managers gave the highest 

average rating of 4.30 for Center assistance addressing “an important state 

priority” in 2008-09. On the usefulness index, the highest average rating of 4.40 

in 2008-09 was for Centers’ having “provided state staff with information or 

resources that they will use again” (exhibit 4.8). 

57 The survey items in the relevance and usefulness scales were reduced in number between the 2006-07 and 2007-

08 surveys. Because no distinct sub-dimensions existed in the 2006-07 scales, the scales were simplified to reduce 

burden. Items were selected for deletion based on a detailed analysis of the impact of each item on the overall 

reliability of the scale. In 2007-08 the reliability of the revised relevance scale was .926 (Cronbach’s Alpha) and that 

of the revised usefulness scale was .911. 

58 The response options were: 5 (to a very high degree), 4 (to a high degree), 3 (to a moderate degree), 2 (to a low 

degree), and 1 (to a very low degree). 

59 The survey from 2006-07 was revised for the 2007-08 administration (and remained unchanged for 2008-09), so 

the scores are not directly comparable. The specific changes in the survey are described in appendix E. 

71 
Exhibit 4.8. Mean ratings of the relevance and usefulness of Center 

assistance by component, as judged by state managers, 2008-09 

Component 

Ratings among 

state managers 

(weighted) 

Relevance 

Addressed an important state priority 

Addressed a need or problem that the state faces 

Provided information, advice, and/or resources that could be used to 

guide decisions about policies, programs, and practices 

Addressed a challenge that the state faces related to the implementation 

of NCLB 

Addressed the state’s specific challenges (e.g., policy environment, 

leadership capacity, budget pressures, local politics) 

4.07

4.30 

4.23 

4.09

3.94

3.87

Usefulness 

Provided state staff with information or resources that they will use again 

Was timely 

Helped state staff to develop skills that they will be able to exercise again 

Helped the state to solve a problem 

Helped the state to maintain or change a policy or practice 

4.21

4.40 

4.25 

4.11 

4.05 

3.89 

EXHIBIT READS: For the 2008-09 program year, state managers (weighted) gave a mean relevance rating 

of 4.07 across all specific rating items, and a mean of 4.30 when rating whether the assistance their state 

had received from the Centers addressed an important state priority. 

SOURCE: Surveys of State Managers. Responses weighted so that each state was equally represented in 

instances where more than one manager from a state responded. 

72 
 

 

5. Variation in the Reported Quality, Relevance, and 

Usefulness of the Centers’ Technical Assistance 

While state managers’ reports provide important input to an assessment of the overall 

work of the Center program (chapter 4), additional perspectives are helpful in assessing the 

quality of the technical assistance projects and their relevance and usefulness to the offices and 

the teams of professionals directly served. As the two types of Centers played distinct roles in 

providing technical assistance, ratings were examined for the Center program and for each center 

type in the second through the fourth program years (2006-07, 2007-08, 2008-09). A deeper look 

at variation across Centers and projects and across program years might also provide information 

for program improvement: additional support or oversight might be provided if quality, 

relevance, or usefulness appeared substantially weaker in particular groups of Centers or 

projects, or for particular types of participants. Thus, variation was investigated by features of 

project design (project scope and the type of activities offered) and by participant experiences in 

the project as well as participant background. 

This chapter analyzes the ratings of quality, relevance, and usefulness given to the Center 

projects sampled for the evaluation in each year, addressing the following research questions: 

■	 To what extent was the assistance provided by the Centers of high quality, high 

relevance, and high usefulness? 

■	 Did the quality, relevance, or usefulness of Center assistance change over 

the period of time studied? 

■	 What was the variation in the quality, relevance, and usefulness of Center 

assistance across types of projects and participants? 

Quality was judged on three items called dimensions; relevance was assessed with eight survey 

items and usefulness with 11 items (exhibit 5.1). Sampled projects were rated for technical 

quality by panels of experts with strong knowledge of the content or substantive focus of the 

specific projects they reviewed.60 Ratings of the sampled projects’ relevance and usefulness 

were gathered by surveying a sample of participants—state staff, intermediate agency staff, local 

educators working on behalf of the state, and RCC staff—who were the intended beneficiaries of 

the project and had received at least some of the technical assistance it provided.61 Participant 

ratings focused on the specific project in which respondents had participated whereas the state 

managers rated the relevance and usefulness of the overall Center assistance that the state had 

received.62 Each overall measure (quality, relevance, or usefulness) was calculated as the mean 

60 See chapter 2 for more information about the data sources and procedures for gathering, coding, and analyzing the data
 

reported in this chapter, including discussions of the methods used for expert ratings and participant surveys. 

61 Copies of the expert panel review rating materials and the project participant surveys can be found in appendix G.
 

Supplemental tables related to this chapter are in appendix H. 

62 Additionally, the state manager survey items for the relevance and usefulness scales were reduced in number 

between the 2006-07 and 2007-08 surveys. Thus, measures of relevance and usefulness for the state managers were 

different from the measure administered to project participants. 

73 
 

 

 

 

of ratings assigned to each item. The item-level ratings themselves were based on 5-point rating 

scales (exhibit 5.2).63 

The chapter describes patterns observed in the ratings across the three program years. 

However, changes over time are presented with caution given a number of limitations or caveats 

related to the ratings of quality, relevance, and usefulness. Each year’s sample of projects was a 

purposive one, with unknown differences in the characteristics of the projects across years. For 

the quality ratings, while the same procedures were followed in each year, the cadre of panelists 

was not entirely unchanged.64 Further, the Centers selected and submitted the specific project 

materials that the experts reviewed, and Center staff became aware of the evaluation procedures 

and criteria during 2007. In the first (2006-07) rating cycle, Centers were given instructions in 

May 2007, within two months of the end of the program year, for assembling materials. In the 

subsequent cycles, Centers knew what they would be asked to submit and thus may have 

gathered and prepared materials during the year as projects progressed. Similarly, for the 

relevance and usefulness ratings, the Centers learned in 2007 that they would be asked to provide 

full lists of project participants, from which the evaluation team would select survey respondents, 

and thus they had the opportunity to compile the needed records during the 2007-08 and 2008-09 

program years. Finally, the participants surveyed were a different group of individuals in each 

year, sampled from all participants in the selected projects. 

63 Efforts were made to develop parallel wording and rubrics that would result in similar gradations between rating 

levels (e.g., very high vs. high vs. moderate) across the three measures. However, given the different content of each 

set of items within the three measures and the different contexts for the ratings (experts who underwent training for 

the rating process and reviewed identical packages of materials vs. survey respondents who typically participated in 

different subsets of project activities), the ratings across the three measures are not directly comparable. 64 Sixty-six of the 70 panelists in 2006-07 returned for the 2007-08 rating cycle, and 14 new panelists were added. 

Seventy-eight of the 80 panelists who reviewed materials in 2007-08 returned for 2008-09. 

74 
 

Exhibit 5.1. Quality, relevance, and usefulness items 

From expert panel scoring From project participant surveys 65 

Technical quality Relevance Usefulness 

Reviewers were directed to 

assign a score to each 

dimension and to include the 

basis for their ratings on the 

rating form, including the 

specific artifacts on which their 

score was based. The three 

dimensions are: 

a. Demonstrated use of the 

appropriate documented 

knowledge base–to include 

an accurate portrayal of the 

current state of information 

with prominence to those 

with the most 

accurate/rigorous evidence 

b. Fidelity of application of the 

knowledge base to the 

products and services 

provided–materials are 

consistent with the 

best/accurate information 

available and the 

presentation adequately 

conveys the confidence of 

the information 

c. Clear and effective delivery– 

information is well organized 

and written and accessible to 

the intended audience for 

easy use 

Based on your experience, to 

what degree was this set of 

activities and resources relevant 

to your work, in each of the 

following respects? 

a. Addressed a need or problem 

that my organization faces 

b. Addressed an important 

priority of my organization 

c. Addressed a challenge that 

my organization faces related 

to the implementation of NCLB 

d. Provided information, advice, 

and/or resources that could be 

directly applied to my 

organization’s work 

e. Addressed our particular state 

context 

f. Addressed my organization’s 

specific challenges (e.g., 

policy environment, leadership 

capacity, budget pressures, 

local politics) 

g. Provided information, advice, 

and/or resources that could be 

used to guide decisions about 

policies, programs, or 

practices 

h. Highlighted the implications of 

research findings (or 

information about best 

practice) for policies, 

programs, or practices 

Based on your experience, to what 

degree was this set of activities 

and resources useful to you, in 

each of the following respects? 

a. Provided resources that were 

easy to understand and easy to 

use 

b. Employed an appropriate format 

(e.g., a work group, a 

conference, individual 

consultation, written products) 

c. Provided adequate opportunity 

to learn from colleagues in other 

states 

d. Included adequate follow-up to 

support the use of new 

information and resources 

e. Were timely 

f. Helped my organization solve a 

problem 

g. Helped my organization 

maintain or change a policy or 

practice 

h. Helped my organization take the 

next step in a longer-term 

improvement effort 

i. Provided my organization with 

information or resources that we 

will use again 

j. Helped my organization develop 

a shared expertise or knowledge 

base 

k. Helped individuals in my 

organization to develop skills 

that they will use again 

65 Participant surveys focused on the specific project in which respondents had participated in contrast to the state 

manager surveys that asked about overall Center assistance the state had received. In addition, the participant survey 

items measuring relevance and usefulness were different from those administered to the state manager survey items 

as the state manager survey items were reduced in number between the 2006-07 and 2007-08 surveys. 

75 
 

Exhibit 5.2. Scales for rating of quality by expert panels and relevance 

and usefulness by participants 

Relevance or usefulness 

Quality ratings by content ratings by project 

Rating area experts participants 

Very high—All or almost all of the 

project meets the given indicators 

for a dimension 

5 To a very high degree 

High—Most of the project meets 4 To a high degree the given indicators for a dimension 

Moderate—Some of the project 

3 meets the given indicators for a To a moderate degree 

dimension 

Low—Limited parts of the project

2 meet the given indicators for a To a low degree 

dimension 

Very low—None or almost none of 

1 the project meets the given To a very low degree 

indicators for a dimension 

Average Ratings by Experts and Participants 

Aggregated to the entire program, the ratings of expert panelists and project participants 

provide insight into the quality, relevance, and usefulness of Center assistance. Trends in the 

ratings of sampled projects over time may also be indicative of trends in overall Center technical 

assistance. Thus the data gathered on individual projects were compiled to produce mean ratings 

of quality, relevance, and usefulness for the sampled projects across the program as a whole. In 

these averages, each Center contributed equally to the overall mean for the program.66 Given the 

different roles assigned to the RCCs and CCs,67 it is appropriate to examine whether there were 

differences in ratings between the two types of Centers.68 In addition, any consistent variation in 

the quality, relevance, and usefulness across individual Centers would be useful to understand. 

66 This averaging procedure across Centers and across projects was designed so that each Center contributed equally 

to the overall mean for the program (or for its type of Center, where RCC means were compared with CC means), 

and each project sampled from a Center contributed equally to the Center mean. 67 ED structured the Comprehensive Centers program to consist of 5 Content Centers charged with specializing in 

activities related to specific content areas and 16 Regional Centers charged with specializing in interactions with the 

states in a region. 

68For analyses of ratings presented in this chapter with the project or Center as the unit of analysis, the evaluators 

used a metric derived from Cohen (1988) estimating Cohen's d (an estimate of the effect size defined as the 

differences in means divided by the pooled standard deviation). Adopting the logic of Cohen for what is a moderate 

difference, we adopted a difference in the means of one-half of one standard deviation (analogous to an effect size 

of .5) as our minimum threshold for highlighting differences. The “pooled standard deviation” for each computation 

varied with the unit of analysis. For analyses conducted at the Center level, the pooled standard deviation was 

76 
■	 Center technical assistance, overall, was rated higher on each measure in each 

successive year, with 2008-09 ratings between “moderate” and “high” for 

quality, and “high” for relevance and usefulness (exhibit 5.3). On a scale of 1 to 

5 with a 3 representing “moderate” and a 4 representing “high,” the program-wide 

average rating for the sampled projects rose from 3.34 in 2006-07, to 3.51 in 

2007-08, to 3.57 in 2008-09 for technical quality (scored by panels of content 

experts). Over this same period of time, Centers’ average rating for relevance 

(scored by participants) went up from 3.94 in 2006-07 to 4.08 in 2007-08 and 

4.15 in 2008-09. The greatest changes in participant ratings of relevance were in 

the items related to the tailoring of assistance–addressing the specific challenges 

or particular context of the SEA, or providing information or resources that could 

guide decisions (appendix exhibit H.4). The program-wide participant ratings of 

usefulness rose from 3.69 in 2006-07 to 3.95 in 2007-08, to 3.96 in 2008-09, with 

the largest increases across years in ratings that gave credit to Center technical 

assistance for helping the participant’s organization to take action–maintaining or 

changing a policy or practice, taking next steps in a longer-term improvement 

effort, or solving a problem (appendix exhibit H.5). 

Exhibit 5.3. Mean ratings of technical quality, relevance, and usefulness, by 

center type and by year 

Technical quality Relevance Usefulness 

2006-

07

2007-

08 

2008-

09 

2006-

07	 

2007-

08 

2008-

09 

2006-

07 

2007-

08 

2008-

09 

All Centers (N=21) 3.34 3.51 3.57 3.94	 4.08 4.15 3.69 3.95 3.96 

All RCCs (N=16) 3.21 3.41 3.52 3.99	 4.18 4.15 3.71 3.99 3.94 

All CCs (N=5) 3.73 3.86 3.72 3.78	 3.96 4.17 3.65 3.84 4.01 

Difference of RCC 

and CC means -0.52† -0.45† -0.20† 	 0.21† 0.22† -0.02 0.06	 0.15† -0.07

Pooled standard 

deviation (all Centers)
 0.41 0.41 0.37 0.34 0.22 0.24 0.34 0.23 0.27

Ratio of difference in 

means to pooled 

standard deviation 

-1.28 -1.09 -0.55
 0.62	 1.00 -0.08 0.18	 0.64 -0.26 

 

NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” rating is the mean of the 

ratings for the three quality dimensions. A notation of † indicates that the difference in the mean ratings between the 

CCs and RCCs within that year is at least one-half of one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.34 for the 2006-07 program year. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 

ratings; each project contributed equally to Center ratings; and each Center contributed equally to cross-Center 

ratings. 

computed as the standard deviation of the variable of interest (e.g., relevance) computed at the Center level. For 

analyses using the project as the unit of analysis, the pooled standard deviation was computed at the project level. 

77 
 

■	 The RCCs’ average quality rating was higher in each successive year but 

remained lower than that of the CCs in each year. The average rating of the 

RCCs for the quality of sampled projects, rated by experts, was below that of the 

CCs (3.21 vs. 3.73 in 2006-07, 3.41 vs. 3.86 in 2007-08, and 3.52 vs. 3.72 in 

2008-09). The difference between RCC and CC means decreased from 0.52 in 

2006-07 to 0.20 in 2008-09 (exhibit 5.3). 

■	 The average relevance rating for RCCs was higher than for CCs in the first two 

years, although CC ratings rose in each year. For 2006-07, the relevance rating 

(from project participants) averaged 3.99 for the RCCs and 3.78 for the CCs 

(difference of 0.21 between RCCs and CCs). In the following year the relevance 

ratings averaged 4.18 for RCCs and 3.96 for CCs (difference of 0.22 between 

types of Centers). In 2008-09, the mean was 4.15 for RCCs and 4.17 for CCs 

(exhibit 5.3). 

■	 On the usefulness of sampled projects, there was no consistent difference between 

RCCs and CCs. The 2006-07 average rating for usefulness (from project 

participants) was similar across types of Centers (3.71 and 3.65). In 2007-08 the 

RCC average (3.99) was higher than that of CCs (3.84). In 2008-09 there was not a 

substantial difference between the averages for the two types of Centers (3.94 

and 4.01) (exhibit 5.3). 

Variation by Type of Project 

Analysis of the variation in ratings at the project level could provide information on the 

practices of Centers, indicating particular strengths of the Centers and areas for improvement. 

Perhaps the Centers were more successful, at least from the perspective of the reviewers, at some 

types of projects compared with others. Thus, the evaluation looked at different groups of 

projects across Centers that might be expected to show higher quality, relevance, or usefulness 

than other projects.69 For example, projects that were larger in scope than others in the Center 

might have garnered higher ratings from the experts or the participants. Other potential sources 

of variation examined here included the types of activities or resources offered and the types of 

expertise brought to bear in delivering or reviewing the technical assistance. 

■	 The distributions of ratings across projects were wider than the distributions 

across Centers, which on all three measures were clustered in the “moderate to 

high” range. Some projects were found in the “high to very high” range on each 

measure in each year (exhibit 5.4). There were also projects in each year below 

the “moderate” rating (i.e., in the “low to moderate” and the “low to very low” 

ranges combined) for quality and usefulness. For relevance, no sampled project 

was found below the “moderate” rating in 2007-08 or 2008-09. 

69 For analyses using the project as the unit of analysis, the pooled standard deviation was computed at the project 

level. 

78 
 

Percent of projects 

100 

80 

60 

40 

20 

0 

2006-07 (RCC n=96, CC n=27) 

2007-08 (RCC n=95, CC n=26) 

2008-09 (RCC n=93, CC n=26) 

53	 54 54 51	 52 50 

41 39 

35 

29 

25 

17 

12

8

4 4 0 0 0 0 4 

17 

24 

30 

Very low to Low to Moderate High to very Very low to Low to Moderate High to very 

low moderate to high high low moderate to high high 

(1.0-1.9) (2.0-2.9) (3.0-3.9) (4.0-5.0) (1.0-1.9) (2.0-2.9) (3.0-3.9) (4.0-5.0) 

RCCs CCs 

■	 No sampled project received the lowest possible mean rating (“low to very low”) 

after 2006-07. For 2006-07 both RCCs and CCs had 4 percent of their projects 

rated “low to very low” for quality by experts; in the subsequent years they had 

none. For both relevance and usefulness, 1 percent of RCC projects had a “low to 

very low” rating from participants in 2006-07, but no project of any Center 

received this rating in 2007-08 or 2008-09 (exhibit 5.4). 

Exhibit 5.4. Percent of projects by rating for technical quality, relevance, and 

usefulness, by year 

Technical quality 

79 
Exhibit 5.4. (continued) 

Relevance 

 

 

 

 

Percent of projects 

100 

80 

60 

40 

20 

0 

68 67 

2006-07 (RCC n=93, CC n=27) 

2007-08 (RCC n=93, CC n=25) 

2008-09 (RCC n=90, CC n=23) 

60 61

57 

32 33 

11 

1 0 0 0 0 0 0 0 0 0 3 

36 

52 

37 39 40

Very low to 

low 

(1.0-1.9) 

Low to 

moderate 

(2.0-2.9) 

Moderate 

to high 

(3.0-3.9) 

High to very 

high 

(4.0-5.0) 

RCCs 

Very low to 

low 

(1.0-1.9) 

Low to 

moderate 

(2.0-2.9) 

Moderate 

to high 

(3.0-3.9) 

High to very 

high 

(4.0-5.0) 

CCs 

Usefulness 

 

 

 

 

Percent of projects 

100 

80 

60 

40 

20 

0 

67 

2006-07 (RCC n=94, CC n=27) 

2007-08 (RCC n=92, CC n=25) 

2008-09 (RCC n=90, CC n=23) 

64 

57 

1 

6 

26 

0 

19 

52 

30 

0 3 

48 49 

0 4 

32 

0 1 

51 48 

0 4 

39 

Very low to Low to Moderate High to very Very low to Low to Moderate High to very 

low moderate to high high low moderate to high high 

(1.0-1.9) (2.0-2.9) (3.0-3.9) (4.0-5.0) (1.0-1.9) (2.0-2.9) (3.0-3.9) (4.0-5.0) 

RCCs CCs 

EXHIBIT READS: For the 2006-07 program year, four percent of the projects sampled from RCCs had a mean quality 

rating of very low to low (between 1.0 and 1.9 on a 5-point scale). Twenty-nine percent of the sampled RCC projects 

had a mean quality rating of low to moderate (2.0 to 2.9 on a 5-point scale) in that year. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to project 

ratings. 

80 
 

Ratings by Project Activities 

The technical assistance provided by the Centers varied in delivery methods, including the 

types of activities offered. The following types of activities and resources were found across the 

sampled projects:70 

■ Ongoing consultation and follow-up 

■ Research collections and syntheses 

■ Engagement of participants in project planning 

■ Training events 

■ Task force meetings and work 

■ Conferences 

■ Support for development of a formal plan to implement a program or policy 

We examined whether projects that included certain types of technical assistance activities 

received more favorable project ratings either from experts knowledgeable about the topic or from 

participants in the project. By comparing the ratings of projects that offered each activity with 

ratings of projects that did not, strengths and weaknesses of particular modes of Center work might 

be discernible. However, no clear pattern of differences in ratings by activity type was found across 

all three years (appendix exhibits H.6-H.8). 

Ratings by Whether the Project Addressed Systems of Support 

 Because technical assistance with statewide systems of support or school support teams 

was in high demand from the states (as discussed in chapter 4), projects addressing the topic of 

“effective systems of support” might have been rated higher than others with respect to relevance 

or usefulness. And because it was an area of considerable Center activity, the Centers’ depth of 

experience with the topic might have helped these projects gain relatively high ratings for 

quality. However, no differences were found (exhibit 5.5). 

70 For each sampled project, the Center furnished the team with a cover sheet intended to inform the expert 

reviewers about the project context, purpose, activities, and products. As described in chapter 2 and appendix C, the 

team coded the content of each project cover sheet to describe the activities and resources offered as part of the 

project. When a cover sheet indicated that the project had multiple components, multiple codes were assigned. 

Chapter 3 presents findings on the frequency with which each year’s sampled projects from RCCs and CCs 

respectively included each activity. 

81 
Exhibit 5.5. Technical quality, relevance, and usefulness ratings of SOS 

projects vs. all other projects in the study sample, by year 

 Technical quality Relevance Usefulness

2006-

07 

2007-

08 

2008-

09 

2006-

07 

2007-

08

2008-

09 

2006-

07 

2007-

08

2008-

Project Topic 09 

SOS projects 

(2006-07 n=36, 

2007-08 n=39, 

2008-09 n=33) 

3.41 3.51 3.62 4.00 4.18 4.19 3.71 3.93 3.98

All other projects 

(2006-07 n=87, 

2007-08 n=82, 

2008-09 n=86) 

3.32 3.54 3.58 3.94 4.12 4.15 3.69 3.94 3.97

Difference in means 0.09 -0.03 0.04 0.07 0.06 0.04 0.02 -0.01 0.01 

Pooled standard deviation 0.69 0.65 0.61 0.52 0.43 0.43 0.54 0.48 0.48

Ratio of difference in 

means to pooled standard 

deviation 

0.13 -0.05 0.07 0.13 0.15 0.09 0.03 -0.02 0.02 

† The difference in the mean ratings within that year of SOS projects and all other projects exceeds one-half of the 

pooled standard deviation in the rating. 

EXHIBIT READS: For the 2006-07 program year, the mean technical quality rating for SOS projects was 3.41 while 

the mean technical quality rating for all other projects in the study sample was 3.32. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 

Ratings for RCC Projects by Whether They Drew on CC Contributions 

Cooperation between CCs and RCCs was one element of the design of the Center 

program, as discussed throughout this report. Thus, among the types of resources that RCC 

projects might offer to their participants were materials developed by CCs, activities developed 

with advice from CCs, or technical assistance from CC staff members. One hypothesis for the 

evaluation was that the RCC projects incorporating CC contributions would be rated higher for 

technical quality than RCC projects that were developed without CC input, since the CCs were 

specifically charged with synthesizing the knowledge base in their areas of focus, and since in 

each year the CCs had a higher mean rating of technical quality than the RCCs. 

■	 Compared with other RCC projects, the ones that included CC contributions 

were rated higher for technical quality in one of three years studied. In 2008-09 

the sampled RCC projects that included one or more types of contributions from a 

CC were rated more than one-half of a standard deviation higher than those that 

did not (exhibit 5.6). No differences greater than one-half of a standard deviation 

were found for earlier years. 

82 
Exhibit 5.6. Technical quality, relevance, and usefulness ratings of RCC 

projects, by presence or absence of CC contributions, by year 

CC contribution or no 

CC contribution 

Technical quality Relevance Usefulness 

2006-

07 

2007-

08 

2008-

09 

2006-

07 

2007-

08

2008-

09 

2006-

07 

2007-

08

2008-

09 

CC contribution 

(2006-07 n=46, 

2007-08 n=32, 

2008-09 n=45) 

3.40 3.59 3.72 3.92 4.23 4.13 3.63 3.99 3.97

No CC contribution 

(2006-07 n=50, 

2007-08 n=63, 

2008-09 n=48) 

3.09 3.36 3.39 4.09 4.18 4.19 3.78 3.97 3.94

Difference in means 0.31 0.22 0.33† -0.17 0.05 -0.05 -0.15 0.02 0.02

Pooled standard 

deviation 0.67 0.63 0.61 0.53 0.42 0.43 0.56 0.48 0.47

Ratio of difference in 

means to pooled 

standard deviation 

0.47 0.35 0.54 -0.32 0.12 -0.12 -0.27 0.05 0.05 

† The difference in the mean ratings within that year between the RCC projects with and without CC contributions 

exceeds one-half of the pooled standard deviation in the rating.
 

EXHIBIT READS: For the 2006-07 program year, the mean technical quality rating for RCC projects with a CC 

contribution was 3.40 while the mean technical quality rating for RCC projects with no CC contribution was 3.09. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 

Ratings for CC Projects by Whether They Drew on RCC Contributions 

Also as a part of the cooperation that was expected within the overall Center program, 

some CC projects drew on the RCCs to help engage participants or to provide substantive help 

with technical assistance (as described in chapter 3). The evaluation team hypothesized that the 

CC projects that had RCC involvement might have been rated higher for relevance than other CC 

projects since the RCCs were charged with knowing the specific purposes and priorities of states 

in their regions, and since their projects received higher average ratings for relevance than CC 

projects in 2006-07 and 2007-08. However, the ratings of sampled projects did not show such a 

difference. 

■	 CC projects with RCC contributions were not rated differently than other CC 

projects for relevance or usefulness; they were rated lower than other CC 

projects for quality in one year. Across years, participants’ ratings of CC projects 

with and without RCC contributions showed no differences greater than one-half 

of a standard deviation in relevance or usefulness (exhibit 5.7). CC projects with 

RCC contributions had quality ratings from experts that were more than one-half 

a standard deviation lower in 2007-08 but not in the years before or after. 

83 
 

Exhibit 5.7. Technical quality, relevance, and usefulness ratings of CC 

projects, by presence or absence of RCC contribution, by year 

RCC contribution or no 

RCC contribution 

Technical quality Relevance Usefulness 

2006-

07 

2007-

08 

2008-

09 

2006-

07 

2007-

08

2008-

09 

2006-

07 

2007-

08

2008-

09 

RCC contribution 

(2006-07 n=17, 

2007-08 n=10, 

2008-09 n=11) 

3.84 3.59 3.76 3.72 4.01 4.16 3.58 3.87 4.00

No RCC contribution 

(2006-07 n=10, 

2007-08 n=16, 

2008-09 n=15) 

3.54 4.06 3.69 3.88 3.87 4.18 3.75 3.73 4.09

Difference in means 0.30 -0.47† 0.07 -0.17 0.15 -0.02 -0.17 0.14 -0.09 

Pooled standard deviation 0.62 0.62 0.44 0.46 0.40 0.44 0.47 0.44 0.50

Ratio of difference in 

means to pooled standard 

deviation 

0.49 -0.77 -0.04 -0.36 0.37 -0.04 -0.36 0.32 -0.18 

† The difference in the mean ratings within that year between the CC projects with and without RCC contribution 

exceeds one-half of the pooled standard deviation in the rating.
 

EXHIBIT READS: For the 2006-07 program year, the mean technical quality rating for CC projects with a RCC 

contribution was 3.84 while the mean technical quality rating for CC projects with no RCC contribution was 3.54. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 

Ratings by Source of Quality Assurance 

For 2008-09, the evaluation gathered Centers’ reports on the sources that they used for 

quality assurance on each sampled project. Possible sources of expertise in quality assurance, as 

discussed in chapter 3, included several that were essentially “in-house”: Center staff, other staff 

in the grantee organization, and other staff in subgrantee organizations. The external sources 

included outside experts retained to review drafts, CCs (in the case of a CC project, this would 

be a Center other than the one conducting the project), and the U.S. Department of Education. It 

seemed possible that projects that had undergone external review would receive relatively higher 

ratings for quality from the expert panelists who rated projects for this evaluation. 

■	 Projects that had been reviewed for quality assurance by outside experts or by a 

CC were rated higher for quality than other projects. The mean quality rating for 

projects reported to have been reviewed by experts was 3.73, compared with 3.42 

for other projects (exhibit 5.8). For projects reportedly reviewed by CCs, the 

mean was 3.83, compared with 3.46 for other projects. No differences greater than 

one-half of a standard deviation were found for quality assurance by other 

84 
 

 

 

 

 

 

sources, including sources closely associated with the Center conducting the 

project (i.e., Center staff or other staff of the grantee or subgrantee organization). 

Exhibit 5.8. Technical quality, relevance, and usefulness ratings by source of 

quality assurance, 2008-09 

Source of quality 

assurance 

Technical 

Quality Relevance Usefulness 

Outside experts (n=63) 3.73 4.12 3.94 

No outside experts (n=55) 3.42 4.21 4.02 

Difference in means 0.31† -0.09 -0.08 

Pooled standard deviation 0.61 0.43 0.47 

Ratio of difference in means 

to pooled standard deviation 0.51 -0.21 -0.16 

Content Centers (n=39) 3.83 4.14 3.92 

No Content Centers (n=79) 3.46 4.18 4.01 

Difference in means 0.37† -0.04 -0.08 

Pooled standard deviation 0.61 0.43 0.47 

Ratio of difference in means 

to pooled standard deviation 0.60 -0.09 -0.18 

† The difference in the mean ratings exceeds one-half of the pooled standard deviation in the rating. 

EXHIBIT READS: For the 2008-09 program year, the mean technical quality rating for projects that used 

outside experts was 3.73 while the mean technical quality rating for projects that did not use outside 

experts was 3.42. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project 

participants for relevance and usefulness. Responses weighted so that each panelist or participant 

contributed equally to the project rating, and each project contributed equally to the mean rating for its 

category. 

Relationships between the Relevance, Usefulness, and 

Quality of Center Projects 

In our analysis of the relevance, usefulness, and quality of Center technical assistance, we 

observed that some types of Centers, individual Centers, or groups of projects that received 

relatively high ratings for quality received relatively low ratings for relevance and usefulness, 

and vice versa. We hypothesized that differences in the rating and rankings (the rating relative to 

other groups of projects) could be due to the differences in the types of individuals who reviewed 

and scored projects for quality—content experts—and those who scored the relevance and 

usefulness—project participants. It was expected that these two groups might value and be better 

able to judge different qualities in a Center project, which is why we did not have content experts 

evaluate the projects for their utility or the participants assess the technical quality. 

85 
 

 

 

We examined the associations among the three dimensions more systematically by 

calculating correlation coefficients.71 These statistics indicate the strength and direction of a 

linear relationship between two factors. A correlation coefficient can vary from positive 1.00 

(indicating a perfect positive relationship), through zero (indicating the absence of a 

relationship), to negative 1.00 (indicating a perfect negative relationship). If the correlation is 

statistically significant (p <.05), we can have strong (95 percent) confidence that the level of 

association we calculated is not due to chance. 

■	 Participants’ ratings of relevance and usefulness were closely related 

(exhibit 5.9). The correlation coefficient was +0.84 for 2006-07, +0.79 for 2007-

08, and +0.83 for 2008-09. This indicates that the extent to which participants 

rated the projects as relevant was associated with how they deemed the project to 

be useful to their agency. These coefficients were all statistically significant at 

p<.05. 

■	 There was no statistically significant relationship between ratings for the 

quality of Center projects and ratings for their relevance or usefulness 

(exhibit 5.9). The results indicated correlations ranging from -0.12 to +0.04 

between quality and relevance, and from -0.09 to +0.07 between quality and 

usefulness. Because the coefficients are not statistically significant we cannot be 

sure that they are different from zero (no relationship). Thus the correlations 

suggest that the expert rating of technical quality was not related to the extent to 

which participants deemed the projects to be relevant or useful. 

Exhibit 5.9. Correlations between project-level technical quality, relevance, 

and usefulness ratings, by year 

 Spearman’s Rho 

Combination 2006-07 2007-08 2008-09 

Quality and relevance -0.12 -0.04 +0.04 

Quality and usefulness -0.04 -0.09 +0.07 

Relevance and usefulness +0.84* +0.79* +0.83* 

* The correlation is statistically significant at p<.05. 

EXHIBIT READS: For the 2006-07 program year, the correlation between project-level ratings of 

technical quality and relevance was a negative 0.12. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project 

participants for relevance and usefulness. Responses weighted so that each panelist or participant 

contributed equally to the project rating. 

71 For this analysis, the evaluation team used Spearman’s rank order correlation because this non-parametric rating is 

the appropriate statistical function to describe correlations between two variables where the values of the variables 

are not normally distributed and are on an ordinal, but not interval scale (such as ratings). 

86 
 

 

 

Variation by Participant Experiences and Roles 

The analyses by subgroups of projects, reported above, brought to light few types of 

projects that had consistently higher or lower ratings than other projects. However, it is possible 

that the targeting, tailoring, and engagement in technical assistance experienced by a particular 

participant would affect how he or she regarded it. In other words, the particular way in which 

the Centers delivered assistance to individuals—and the professional roles of the individuals on 

whom assistance focused—might account for differing perceptions of relevance and usefulness. 

If so, a project might have relevance and usefulness similar to those of other projects when the 

ratings are averaged across all participants in that project, but individual participants might give 

it systematically different ratings compared with other participants. 

Thus the team examined whether the rating of relevance or usefulness given by a 

participant was related to that participant’s experience in the project or to his or her job 

responsibilities. These analyses provided an opportunity to follow up on findings from the prior 

evaluation conducted on the Comprehensive Regional Assistance Centers established under the 

1994 ESEA reauthorization (described in chapter 1 of this report).72 As some of the participant 

survey questions changed from the first to the second survey administration, examination of 

participant ratings based on those particular items are presented only for 2007-08 and 2008-09 

when the survey items remained the same. For survey questions that remained the same, all three 

years (2006-07, 2007-08, 2008-09) are presented. The specific participant characteristics 

examined included: 

■	 Whether the participant contributed to the design of the project 

■	 How much time the participant spent with project activities or resources 

■	 The participant’s place of employment, and the extent to which the job focused on 

NCLB 

In the following discussion, the unit of analysis is the participant, not the project. The 

mean ratings are those provided by a group of participants, across all projects, who had a 

particular type of experience or background; within the same project there could be participants 

with different experiences and backgrounds.73 We first describe the variation found in each of 

these participant characteristics and the ways in which ratings did or did not differ according to 

72 U.S. Department of Education, Office of the Under Secretary, Planning and Evaluation Service, Elementary and 

Secondary Education Division. (2000). Comprehensive Regional Assistance Centers Program: Final Report on the 

Evaluation (Volume I). Washington, DC: Author. 73 It is important to note that these participant-level analyses are exploratory, and the relationships described 

between participant characteristics and their ratings are correlational, not causal. The participant responses were 

weighted to represent all project participants in the sample frame for each project. The weights were calculated by 

dividing the total number of sample-eligible participants by the number of respondents completing a survey who 

participated in that project. The weighted figures shown in the tables in this section approach but never exceed the 

sample frame figures shown in appendix exhibit B.4, depending on the number of survey respondents who provided 

valid responses to the specific question(s) comprising each table. See chapter 2 and appendix B for more 

information on survey administration and the process used to weight responses. 

87 
 

each characteristic individually. The section concludes with a multivariate analysis examining 

each participant characteristic while controlling for the presence of the other characteristics. 

Ratings by Participants’ Involvement in Project Design 

An aspect of participation that was of particular interest, given the Centers’ charge to work 

closely with their clients, was the extent to which participants were involved in the projects’ design 

stages and, then, whether perceptions of relevance and usefulness might vary with a participant’s 

involvement. The evaluation of the earlier program of Comprehensive Regional Assistance 

Centers, based on survey results from a sample of participants, concluded: “Comprehensive Center 

technical assistance is more useful to customers if it is tailored to address their needs and 

interests.”74 For the current evaluation, the team analyzed participant responses to a question 

asking respondents whether they were personally involved in determining the goals or designing 

the content or format of the project. 

■	 A higher proportion of participants in the sampled RCC projects reported being 

involved in some aspect of determining project goals or designing the project, 

compared with participants in the CC projects. For 2007-08 the figures were 

39 percent of participants in RCC projects and 20 percent of those in CC projects; 

for 2008-09, 42 percent of participants in RCC projects and 16 percent of those in 

CC projects were involved either in determining goals or designing the projects 

(exhibit 5.10). In 2006-07, when this survey question included a 9-item list of 

ways in which a respondent might have been involved in determining goals or 

designing the project, more respondents indicated that they had been involved 

(57 percent for participants in RCC projects and 42 percent for those in CC 

projects), but the relatively higher rate of involvement in RCC projects was 

observed in that year as well (appendix exhibit H.9). In each of the three years, a 

chi square test found the difference across types of Centers to be statistically 

significant (p<.01).75 

74 U.S. Department of Education, 2000, op cit. 

75 For the analysis of responses of project participants, inferential statistics were used to identify differences in 

responses among groups of participants that were greater than would be expected by chance alone. Because the 

sample of project participants were selected through a random sample of all project participants, the evaluators 

judged it appropriate to use test of statistical significance for these analyses. However, we would caution the reader 

not to generalize the results beyond the sample of projects selected for each year of the study, because the specific 

projects were selected on a non-probability basis. 

88 
Exhibit 5.10. Participants’ involvement in determining project goals or design, 

by type of Center and by year 

Involved in determining 

the goals or content of 

the project 

Percent of participants 

RCC projects CC projects 

2007-08 

(n=1,722) 

2008-09 

(n=1,701) 

2007-08 

(n=2,131) 

2008-09 

(n=1,506) 

Total 100% 100% 100% 100% 

Yes 39 42 20 16 

No 61 58 80 84 

NOTE: Difference in the proportion of participants involved in design by type of Center is 

statistically significant (p<.05, Chi Square) in each year.
 

EXHIBIT READS: for the 2007-08 program year, 39 percent of participants in projects 

conducted by RCCs reported they were involved in the design of the technical assistance.
 

SOURCE: Surveys of project participants. Responses weighted so that respondents 

represented all participants for the project. 

■	 Participants who reported being involved in determining the goals or designing 

the content of the project provided higher relevance and usefulness ratings in 

all years than did other participants. The average ratings in projects sampled for 

the current study corroborate the survey findings from the prior evaluation of the 

Centers. Participants who reported involvement in project design gave the 

projects a 4.06 average score for relevance and a 3.83 score for usefulness in 

2006-07 (exhibit 5.11). In contrast, average scores for participants not involved in 

project design were 3.81 and 3.56, respectively. These differences were 

statistically significant (with p<.01 for differences in both relevance and 

usefulness). 

■	 While the participants who were involved in planning gave higher ratings than 

those who were not, no difference in ratings was found between the projects 

that did or did not offer opportunities for involvement when analyzed at the 

project level. Comparing mean ratings at the project level, as reported earlier in 

this chapter, we did not find a difference greater than one-half of a standard 

deviation in relevance or usefulness ratings among projects that engaged 

participants in planning as opposed to those that did not (appendix exhibits H.6-

H.8).This suggests that involving some participants in the design phase may make 

a difference in the ratings provided by those particular participants, but not in the 

ratings provided by all participants in the project. 

89 
Exhibit 5.11. Mean relevance and usefulness ratings, by respondent 

involvement in determining project goals or design, by year 

Involvement in project 

design 

Relevance Usefulness 

2006-07 2007-08 2008-09 2006-07 2007-08 2008-09 

Involved in project design 

(2006-07 n=1,802, 

2007-08 n=1,092, 

2008-09 n=943) 

4.06 4.27 4.37 3.84 4.14 4.20 

Not involved in project 

design 

(2006-07 n=1,638, 

2007-08 n=2,716, 

2008-09 n=2,134) 

3.81 3.89 3.90 3.56 3.77 3.80 

Difference 0.25* 0.38* 0.47* 0.28* 0.37* 0.40* 

Significance p<.01 p<.01 p<.01 p<.01 p<.01 p<.01 

* Difference statistically significant within each year at p<.05 using a one-way ANOVA. 

EXHIBIT READS: For the 2006-07 program year, participants who indicated that they were involved determining the 

goals or design of the project provided a relevance rating of 4.06 while project participants not involved in determining 

the goals or design of the project provided a relevance rating of 3.81. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all participants for 

the project. 

90 
 

 

 

Ratings by Time Spent on Project Activities 

The literature on professional development suggests that the duration of training is 

associated with better outcomes.76 Similarly, the previous evaluation of the Comprehensive 

Centers program, based on survey data from participants, concluded: “Comprehensive Center 

technical assistance is more useful to customers if it is intensive and if it extends over time.”77 

Therefore, this study explored the possibility of a relationship between the time an individual 

spent in project activities and the ratings he or she provided, assessing whether the same 

relationship persisted. 

■	 Participants who spent more time on project activities tended to give higher 

ratings of relevance and usefulness in 2007-08 and 2008-09. Among all 

participants, those who spent five days or less carrying out activities related to the 

project gave ratings for relevance and usefulness near the halfway point between 

“moderate” and “high”: for relevance, the mean rating given by this group was 

3.59 in 2007-08 and 3.58 in 2008-09; for usefulness the means were 3.46 and 3.42 

(exhibit 5.12). Those who spent more time (i.e., 6 or more days) gave mean 

ratings in the “high” range (near or above 4.0) that were significantly higher than 

ratings by participants who spent 5 days or less (p<.01 for differences). For each 

type of rating in each year, the highest ratings were given by participants who had 

spent more than 10 days on the project.78 

76 Desimone, L., Porter, A.C., Garet, M., Yoon, K.S., and Birman, B. (2002). “Does Professional Development 

Change Teachers’ Instruction? Results from a Three-Year Study.” Educational Evaluation and Policy Analysis, 

24(2), 81-112; Garet, M., Porter, A., Desimone, L., Birman, B., and Yoon, K.S. (2001). “What Makes Professional 

Development Effective? Results from a National Sample of Teachers.” American Educational Research Journal, 

38(4), 915-945. 77U.S. Department of Education, 2000, op cit., p. 9.

78 For 2006-07, respondents indicated the amount of time spent in each of eight activities by selecting from the 

following five options: “More than 5 days,” “3-5 days,” “1-2 days,” “Less than 1 day,” and “Not applicable.” Thus 

the findings are not directly comparable across years. However, a relationship between time spent and ratings of 

relevance and usefulness was found for that year as well. For each of eight project-related activities addressed in the 

survey for 2006-07, participants who reported spending at least three days in the activity rated the project higher on 

both relevance and usefulness than did participants who spent less time in that activity (appendix exhibit H.10). All 

of the differences were statistically significant at the p<.05 level using a one-way ANOVA. In addition, there was a 

statistically significant relationship between the number of activities in which the participant spent three or more 

days and the rating of project relevance. The correlation between the number of different project activities in which 

the respondent participated for three or more days and the rating of relevance was +0.41, while the correlation with 

the rating of usefulness was +0.34. These correlation coefficients were statistically significant at p<.05 

91 
Exhibit 5.12. Relevance and usefulness ratings, by amount of time participants 

spent on project activities, by year 

Time Spent on Project 

Activities 

Relevance Usefulness 

2007-08 

(n=3,820) 

2008-09 

(n=3,045) 

2007-08 

(n=3,791) 

2008-09 

(n=3,029) 

More than 30 days 

(2007-08 n=620, 

2008-09 n=400) 

21 to 30 days 

(2007-08 n=241, 

2008-09 n=223) 

11 to 20 days 

(2007-08 n=690, 

2008-09 n=665) 

6 to 10 days 

(2007-08 n=840, 

2008-09 n=736) 

5 days or less 

(2007-08 n=1,429, 

2008-09 n=1,021) 

4.46 

4.07 

4.33 

4.06 

3.59 

4.35 

4.50 

4.35 

4.13 

3.58 

4.28 

3.89 

4.29 

3.97 

3.46 

4.20 

4.32 

4.24 

4.05 

3.42 

Range (Difference 

between highest and 

lowest rating) 

Significance 

0.87* 

p<.01 

0.92* 

p<.01 

0.83* 

p<.01 

0.90* 

p<.01 

* Within each scale and each year, the difference in ratings by time spent on project activities is 

statistically significant (p<.05, ANOVA). 

EXHIBIT READS: For the 2007-08 program year, project participants involved in project activities 

for more than 30 days had an average relevance rating of 4.46. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented 

all participants for the project. 

Ratings by Participant Roles 

Were the sampled projects rated differently by participants who occupied different roles 

in their day-to-day jobs? The team investigated the possibility of a relationship between the 

participant’s place of employment or job responsibilities and the ratings he or she provided for a 

project’s relevance or usefulness. 

Given the intended emphasis of the Center technical assistance on building capacity at 

the state level, it is important to understand the extent to which the views of participants from 

state agencies might have differed from those of other participants. Of the sampled participants 

in RCC projects, the percent employed by SEAs was 41 percent in 2006-07, 54 percent in 

2007-08, and 56 percent in 2008-09 (exhibit 5.13). The percent employed by other agencies 

92 
 

 

 

 

varied across years.79 In addition to intermediate agencies, local education agencies, and 

schools, other employers included institutions of higher education, nonprofits working with the 

state, and other RCCs. 

Exhibit 5.13. Where participants in RCC projects were employed, by year 

Employed by: 

Total 

Percent of participants 

2006-07 

(n=2,415)

2007-08 

(n=1,603)

2008-09 

(n=1,474) 

100% 100% 100% 

State education agency (SEA) 41 54 56 

Intermediate education agency 14 21 21 

Local education agency (LEA) 15 7 10 

School 20 11 7 

Other 10 7 5 

EXHIBIT READS: Among the surveys completed by participants in RCC projects, 

41 percent were completed by respondents who were employed in a state education 

agency for the 2006-07 program year. 

SOURCE: Surveys of project participants. Responses weighted so that respondents 

represented all participants for the project.
 

■	 In both 2007-08 and 2008-09, participants employed in SEAs gave higher 

ratings for relevance and usefulness to RCC projects than participants 

employed in other agencies. The SEA employees gave relevance ratings 

averaging 4.13 in 2007-08 and 4.16 in 2008-09; other participants’ relevance 

ratings averaged 3.95 in each of those years (exhibit 5.14). For usefulness, the 

SEA employees’ ratings averaged 3.92 in 2007-08 and 4.01 in 2008-09, compared 

with 3.78 and 3.65 for other participants. These differences were statistically 

significant (p<.01) in 2007-08 and 2008-09 for both relevance and usefulness. 

SEA employees’ ratings of relevance and usefulness did not differ significantly 

from those of other participants in 2006-07. 

79 Participants who were employed by agencies other than the SEA were considered “state-level” participants 

because they carried out state-level responsibilities such as membership in a school support team or state-level task 

force (see chapter 2 for more discussion of the definition of “state-level” participants). For this analysis, however, 

we differentiate among participants by their place of employment. 

93 
Exhibit 5.14. Mean relevance and usefulness ratings for RCC projects, by 

agency where respondent worked, by year 

Agency where 

respondent worked 

Relevance Usefulness 

2006-07 2007-08 2008-09 2006-07 2007-08 2008-09 

State education agency 

(SEA) 

(2006-07 n=977, 

2007-08 n=909, 

2008-09 n=826) 

3.99 4.13 4.16 3.68 3.92 4.01 

All other agencies 

(2006-07 n=1,409, 

2007-08 n=850, 

2008-09 n=632) 

3.94 3.95 3.95 3.68 3.78 3.65 

Difference 0.05 0.19* 0.21* 0.00 0.14* 0.36* 

Significance p=.13 p<.01 p<.01 p=.93 p<.01 p<.01 

* Difference statistically significant within each year at p<.05 using a one-way ANOVA. 

EXHIBIT READS: For the 2006-07 program year, the mean relevance rating of RCC projects provided by 

participants in SEAs was 3.99 while the mean relevance rating by participants working at all other agencies 

was 3.94. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all 

participants for the project. 

In CC projects, most participants were employees of either an RCC or an SEA; no more 

than 2 percent were employed by any one other type of agency in any year (exhibit 5.15). The 

relative proportion of RCC and SEA staff participating in CC projects fluctuated, with more 

coming from RCCs in 2006-07 (59 percent RCC), more from SEAs in 2007-08 (54 percent), and 

similar proportions in 2008-09 (52 percent RCC, 48 percent SEA). 

■	 In both 2007-08 and 2008-09, CC projects were deemed more relevant and 

useful by RCC staff than by SEA staff. RCC and SEA staff gave similar 

relevance and usefulness ratings to CC projects in 2006-07, but in each of the 

following years the RCC staff ratings rose while the SEA staff ratings changed 

little (exhibit 5.16). For 2008-09 RCC staff gave mean ratings of 4.17 and 4.14 for 

the relevance and usefulness of CC projects, compared with the 3.87 and 3.76 

ratings given by SEA staff. 

94 
 

Exhibit 5.15. Where participants in CC projects were employed, by year 

Percent of participants 

Employed by: 

2006-07 

(n=1,141) 

2007-08 

(n=2,135) 

2008-09 

(n=1,379) 

Total	 100% 100% 100% 

RCCs 59 41 52 

State education agency (SEA) 

Intermediate education agency 

Local education agency (LEA) 

School 

38 

* 

1 

2 

54

0 * 

* 0 

2 0 

48 

Other * 3 0 

* Less than 0.5 percent.
 

EXHIBIT READS: Among the surveys completed by participants in CC projects, 

59 percent were completed by respondents who were employed by RCCs for the 

2006-07 program year. 

SOURCE: Surveys of project participants. Responses weighted so that 

respondents represented all participants for the project. 

Exhibit 5.16. Mean relevance and usefulness ratings for CC projects, RCC staff 

vs. state-level staff, by year 

RCC or state-level 

staff 

Relevance Usefulness 

2006-07 2007-08 2008-09 2006-07 2007-08 2008-09 

RCC Staff 

(2006-07 n=661, 

2007-08 n=842, 

2008-09 n=712) 

State-level staff 

(2006-07 n=484, 

2007-08 n=1,288, 

2008-09 n=712) 

3.92 

3.91 

4.07 

3.89

4.17

3.87 

3.81 

3.73 

3.95 

3.87

4.14 

3.76 

Difference 

Significance 

-0.01 

p=.83 

0.18*

p<.01

0.30* 

p<.01 

0.08 

p=.12

0.08*

p=.03

0.38* 

p<.01 

* Difference statistically significant within each year at p<.05 using a one-way ANOVA. 

EXHIBIT READS: For the 2006-07 program year, the mean relevance rating of CC projects provided by RCC staff 

was 3.92 while the mean relevance rating of CC projects by state-level staff was 3.91. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all participants for 

the project. 

Finally, because the Centers’ charge focused on assistance related to NCLB, the team 

hypothesized that participants whose regular job responsibilities focused most heavily on that 

law might give relatively higher ratings for relevance and usefulness of the sampled projects. 

95 
 

 

 

 

 

 

 

 

■	 Participants who, in their regular jobs, spent more than 25 percent of their time 

on NCLB-related state responsibilities gave higher ratings to Center projects 

than participants whose jobs focused less on NCLB. We found that those who 

spent more than one quarter of their time at work on NCLB gave significantly 

higher relevance and usefulness ratings, compared with those who spent 25 

percent of their time or less on NCLB (p<.01 for differences in both relevance and 

usefulness using ANOVA) (exhibit 5.17).80 

Exhibit 5.17. Mean relevance and usefulness ratings, by percent of time 

respondent spent on NCLB-related tasks in his/ her job and 

by year 

Percent of time spent on 

NCLB-related tasks 2006-07

Relevance

2007-08 2008-09 2006-07 

Usefulness 

2007-08 2008-09 

More than 75 percent 

(2006-07 n=908, 

2007-08 n=738, 

2008-09 n=924) 

4.05 3.97 4.17 3.87 3.84 4.02 

51 to 75 percent 

(2006-07 n=541, 

2007-08 n=613, 

2008-09 n=587) 

3.98 4.20 4.13 3.78 4.07 4.12 

26 to 50 percent 

(2006-07 n=493, 

2007-08 n=366, 

2008-09 n=549) 

3.94 3.92 4.26 3.72 3.88 4.00 

25 percent or less 

(2006-07 n=476, 

2007-08 n=478, 

2008-09 n=763) 

3.72 3.62 3.74 3.47 3.75 3.59 

Range (Difference between 

highest and lowest rating) 

Significance 

0.33* 

p<.01 

0.58* 

p<.01 

0.51* 

p<.01 

0.40* 

p<.01 

0.32* 

p<.01 

0.53* 

p<.01 

* Difference statistically significant within each year at p<.05 using a one-way ANOVA. 

NOTE: The survey question for this item differed slightly from the 2006-07 administration to the form used for 2007-

08 and 2008-09 in that in the first administration, respondents were first asked to describe the specific aspects of 

NCLB implementation that were included in their job responsibilities and were then asked to describe the percent of 

time they spent on all of those tasks. For the subsequent administrations of the survey respondents were asked just 

to describe the percent of their time was spent on responsibilities related to NCLB. 

EXHIBIT READS: Among project participants for the 2006-07 program year, those whose job responsibilities included 

NCLB-related tasks more than 75 percent of the time provided a relevance rating of 4.05 while participants whose job 

responsibilities included NCLB-related tasks from 51 to 75 percent of the time provided a relevance rating of 3.98. 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all participants for 

the project. 

80 The form of the question changed across years. For 2006-07, participants were asked in the survey to indicate 

what percentage of their hours on the job, in total, they spent working on any of the specific state-level 

responsibilities related to NCLB. In the subsequent years they were simply asked how much time they spent 

working on responsibilities related to NCLB. In each year they were given a choice of four percentage ranges: 0-25 

percent, 26-50 percent, 51-75 percent, and 76-100 percent. 

96 
Because participant characteristics may be interrelated, we conducted multivariate 

analyses to examine the associations of participant characteristics with project ratings of 

relevance and usefulness when holding the other participant characteristics constant. Multiple 

regressions of the four participant characteristics on ratings of project relevance and on ratings of 

project usefulness were consistent with analyses of individual participant characteristics. Each of 

the project characteristics was shown to have a statistically significant relationship with ratings 

of relevance and ratings of usefulness after controlling for the other three predictors (see 

appendix exhibits H.11and H.12). 

97 

Appendix A 

Center Lead Grantee 

and Subgrantee Organizations
 

Center Lead Grantee 

and Subgrantee Organizations 

A-1 

Center name Lead grantee and subgrantees 

Alaska Comprehensive 

Center 

■	 Southeast Regional Resource Center (SERRC of Alaska) 

o Northwest Regional Educational Laboratory (NWREL) 

Appalachia Regional 

Comprehensive Center 

■	 Edvantia 

o	 George Washington University - Center for Equity and Excellence 

in Education (CEEE) 

o Eastern Stream Center on Resources and Training (ESCORT) 

o National Center for Family Literacy (NCFL) 

o University of North Carolina at Greensboro - SERVE Center 

o Southern Regional Education Board (SREB) 

California 

Comprehensive Center 

■	 WestEd 

o American Institutes for Research (AIR) 

o School Services of California 

Florida and Islands 

Comprehensive Center 

■ Educational Testing Service (ETS) 

o University of South Florida - David C. Anchin Center 

o Edvantia 

o Eastern Stream Center on Resources and Training (ESCORT) 

o Florida Association of School Administrators (FASA) 

o International Center for Leadership in Education (ICLE) 

o JLM Professional Education Services 

o Robin Wheeler, LLC 

Great Lakes East 

Comprehensive Center 

■	 Learning Point Associates 

o RMC Research Corporation 

o American Institutes for Research (AIR) 

o Center for Applied Linguistics (CAL) 

o	 University of Michigan - Consortium for Policy Research in 

Education (CPRE) 

Great Lakes West 

Comprehensive Center 

■	 Learning Point Associates 

o American Institutes for Research (AIR) 

o Mid-continent Research for Education and Learning (McREL) 

o	 University of Wisconsin - Wisconsin Center for Educational 

Research (WCER) 

o	 University of Michigan - Consortium for Policy Research in 

Education (CPRE) 

Mid-Atlantic 

Comprehensive Center 

■	 George Washington University - Center for Equity and Excellence in 

Education (CEEE) 

o Edvantia 

o Eastern Stream Center on Resources and Training (ESCORT) 

o Group Jazz 

o Southern Regional Education Board (SREB) 

Mid-Continent 

Comprehensive Center 

■	 University of Oklahoma 

o Northrop Grumman Information Tech 

o Accion Social Comunitaria 

o Mountain Plains Regional Resource Center (MPRRC) 

o North Central Regional Resource Center (NCRRC) 

o Southeast Regional Resource Center (SERRC of Alabama) 

New England 

Comprehensive Center 

■	 RMC Research Corporation 

o Education Development Center (EDC) 

o WestEd 

o Education Alliance at Brown University 
 

Center Lead Grantee 

and Subgrantee Organizations (continued) 

Center name Lead grantee and subgrantees 

New York 

Comprehensive Center 

■ RMC Research Corporation 

o Education Development Center (EDC) 

o WestEd 

o Education Alliance at Brown University 

o United Federation of Teachers Teacher Center (UFTTC) 

North Central 

Comprehensive Center 

■ Mid-continent Research for Education and Learning (McREL) 

Northwest Regional 

Comprehensive Center 

■ Northwest Regional Educational Laboratory (NWREL) 

o RMC Research Corporation 

Pacific Comprehensive 

Center 

■ Pacific Resources for Education and Learning (PREL) 

Southeast 

Comprehensive Center 

■ Southwest Educational Development Laboratory (SEDL) 

o Center for the Education and Study of Diverse Populations 

(CESDP) 

Southwest 

Comprehensive Center 

■ WestED 

o RMC Research Corporation 

o American Institutes for Research (AIR) 

Texas Comprehensive 

Center 

■ Southwest Educational Development Laboratory (SEDL) 

o Center for the Education and Study of Diverse Populations 

(CESDP) 

Assessment and 

Accountability Content 

Center 

■ WestEd 

o National Center for Research on Evaluation Standards, and Student 

Testing (CRESST) 

Center for Innovation 

and Improvement 

■ Academic Development Institute (ADI) 

o Temple University - Center for Research in Human Development 

and Education (CRHDE) 

o Little Planet Learning 

Center on Instruction ■ RMC Research Corporation 

o Florida Center for Reading Research (FCRR) 

o RG Research Group 

o Horizon Research 

o Texas Institute for Measurement, Evaluation and Statistics (TIMES) 

o	 University of Texas at Austin - Vaughn Gross Center for Reading 

and Language Arts 

National Comprehensive 

Center on Teacher 

Quality 

■ Learning Point Associates 

o Education Commission of the States (ECS) 

o Educational Testing Service (ETS) 

o Vanderbilt University 

National High School 

Center 

■ American Institutes for Research (AIR) 

o Learning Point Associates 

o MDRC 

o National Center for Educational Accountability (NCEA) 

o WestEd 

EXHIBIT READS: The Alaska Comprehensive Center was operated by the lead grantee organization named the 

Southeast Regional Resource Center (of Alaska) with support from the subgrantee organization named the 

Northwest Regional Educational Laboratory. 

Source: U.S. Department of Education; individual RCC and CC websites 

A-2 
Appendix B 

Study Sample: 

Center Projects and Project Participants 

Study Sample 

This appendix section presents the Project Inventory Forms (PIFs) that were collected to 

identify the sample of projects to be included in the study as well as the documents sent to 

Centers requesting materials for expert panel review of sampled projects. The appendix 

concludes with historical tables showing the number of projects in the study sample for 2006-07 

and 2007-08 by topic and detailed information describing the administration of the participant 

surveys across all three years. 

Project Inventory Form 

The PIF was developed by the study team as a standard way for Centers to provide an 

inventory of their work for each program year. When asked to complete the inventory forms, 

Center Directors were informed about the specific project year for which information was being 

gathered as well as the purpose of this information to be used in the selection of a sample of 

projects for review of quality by expert panels to be conducted for the national evaluation. When 

completing the PIF, each Center was asked to nominate several projects that they believed best 

represented the work undertaken by that Center. Additional projects were purposefully sampled 

from the remaining projects on each center’s inventory to reflect each center’s overall portfolio 

of work, as well as the work of all regional or content centers in key topic areas. A sample PIF 

was provided including examples of the kinds of projects that should be listed by the centers and 

examples of activities and resources defining each project level of effort – major, moderate, 

minor. Examples of some projects, activities, and deliverables that should not be included on the 

inventory form were also provided, including the following: 

■	 Training or professional development for Center staff 

■	 Work on coordinating committees within the Centers 

■	 Annual needs assessment activity or negotiations with states, unrelated to specific 

projects 

■	 Other internal working meetings or documents 

When completing the PIF, each center provided a list of each project under the 

appropriate topic heading. While the Centers submitted PIFs using 14 topic areas for the 2006-07 

data collection, the evaluation team used the coding process described in appendix C to recode 

the substantive knowledge base of the project into a more detailed set of 22 topics. For the 2007-

08 and 2008-09 PIF submissions, the Centers assigned one of the 22 topic areas to each project 

themselves as part of the submission. 

This appendix includes copies of the materials sent to the Centers to help them complete 

their PIFs including the instructions for completion (page B-6), the list of topic areas used for the 

2007-08 and 2008-09 data collections (page B-14), and a sample section of a PIF (page B-16). 

B-1 
Letters Requesting Materials for Expert Panel Review from Centers 

For each data collection cycle, each Center received a notification letter from the 

evaluation team that listed the projects from their PIF that had been selected for expert panel 

review followed by guidance in assembling the materials for panel review, compiling lists of 

project participants for the participant survey, and completion of the cover sheet for each sampled 

project. The initial request for materials letter and supporting information starts on page B-22. The 

evaluation team reviewed the materials submitted by the Centers for accuracy and completeness 

and in cases where there were missing or incomplete materials or where further explanation was 

needed, followed up with Centers using the follow-up memo provided later in this appendix 

starting on page B-29. 

Decision rules for sampling projects from the PIFs 

Exhibit B.1 below shows the decision rules the evaluation team applied in selecting the 

sample of projects for the evaluation sample. This six-step process was used to draw the 

evaluation in each of the three evaluation cycles. 

Exhibit B.1. Decision rules for sampling projects within each center 

Step 1. Determine the total number of projects to be sampled for that center 

Step 2. Identify the designated number of center-nominated projects, selected with certainty 

Step 3. Determine the number of additional projects to be sampled 

Step 4. Sample Major projects 

 If the number of major projects was less than or equal to number of additional 

projects needed, sample all major projects and go to step 5 

 If the number of major projects is greater than the number of additional projects 

needed, randomly select projects while controlling for topic and state (i.e., do not 

select multiple projects from any given topic or state [for centers serving multiple 

states] until all topics and states have at least one project sampled) – repeat until 

the sample size is met 

Step 5 Determine the number of additional projects to be sampled (if any) 

Step 6. Sample Moderate projects 

 If the number of moderate projects is less than or equal to number of additional 

projects needed, sample all moderate projects 

 If the number of moderate projects is greater than the number of additional projects 

needed, randomly select projects while controlling for topic and state (i.e., do not 

select multiple projects from any given topic or state [for centers serving multiple 

states] until all topics and topics have at least one project sampled) – repeat until 

sample size is met 

B-2 
Distribution of projects and projects in the evaluation sample 

Exhibits B.2 and B.3 included in this appendix (starting on page B-30), show the 

distribution of all major and moderate projects on the project inventory forms and projects in the 

sample by topic for the 2006-07 and 2007-08 data collection cycles. These tables are parallel to a 

similar table in the report for 2008-09 (exhibit 2.5). 

Administration of the Participant Survey 

As described in Chapter 2, the participant survey was administered to project participants. 

This included state-level staff for RCC projects and a combination of state-level and RCC staff 

for CC projects. Exhibit B.4 on page B-32 shows the number of participants at each stage of the 

data collection. Sampled respondents who indicated in their response to the first survey item that 

they did not participate in the project were excluded from analysis. 

For each of the three data collection cycles, the evaluation team analyzed the properties 

of the items included in the participant survey for constructing indices of relevance and 

usefulness. 

Principal components analysis with no rotation was conducted on each set of items in the 

relevance and usefulness scales to determine the underlying dimensions represented by these 

items. For each scale, analyses of each year of data resulted in a one-factor solution comprised of 

the same items and with comparable levels of reliability. For the relevance scale, the Cronbach 

alpha was 0.93 in 2006-07 and 0.94 in 2007-08 and 2008-09, and 0.95 for the usefulness scale in 

all three years. Additional information on the factor loadings and psychometric properties for the 

three years of data collection is provided in exhibits B.5 and B.6 starting on page B-33. 

Units of analysis and weighting for quality, relevance, and usefulness data 

The quality, relevance, and usefulness ratings of the sampled projects were presented in 

chapter 5 using several different units of analysis: the projects sampled from the program as a 

whole (all 21 Centers), the projects sampled for the 16 RCCs and the 5 CCs respectively, 

subgroups of the projects, and (for relevance and usefulness) subgroups of the participants. 

■	 Program-wide and by Center type. The team weighted the mean center-level 

quality, relevance, or usefulness results equally so that each Center contributed 

equally to the mean for the entire program or for a set of Centers. 

■	 Center-level. For a Center’s mean rating on quality, relevance, or usefulness, the 

team weighted each sampled project from that Center equally. 

■	 Project-level. Several analyses were conducted for subgroups of projects across 

Centers—e.g., those projects that included a research synthesis among their 

products, or included training among their services. Each rater (an expert or 

B-3 
survey respondent) had equal weight in the individual project’s rating, and each 

project had equal weight in the subgroup mean rating. 

■	 Participant-level. For some analyses of subgroups of participants across projects, 

the unit was the participant, not the project, and the weighting was designed to 

permit generalization to all the participants across all the sampled projects. Thus 

the weight for each participant’s response was the inverse proportion of the 

number of respondents compared with the total number of participants in the 

population for that project. The n’s shown in the participant-level tables in chapter 

5 reflect the weighted figures. The maximum number is the total number in the 

sample frame shown in exhibit B.4 in this appendix but the actual weighted n’s 

are often somewhat lower because the figures in the table represent a subset of 

participants or because some participants failed to respond to that survey item. 

B-4 
 

Project Inventory Form 

Center Project Inventory Form 

Dear Center Director, 

Enclosed are a Project Inventory Form and instructions to be used to compile a complete 

list of your Center’s projects during the current project year (July 1, 200X–June 30, 200X). 

The national evaluation will use the completed project inventories to select a sample of 

projects for review by expert panels. Please designate several projects that you believe best 

represent the work of your Center; these projects will be included among the sample of projects 

to be reviewed by expert panels. These panels will rate nominated and sampled projects for their 

quality. 

Please complete a draft inventory to review with a member of the national evaluation 

team during the site visit for the evaluation, scheduled to take place between April and June 

200X when [EVALUATION STAFF MEMBER] visits your Center. A final version of the 

project inventory should be completed as soon as possible after the site visit, but no later than 

June 30, 200X. Please return the completed inventory form to your evaluation liaison, [name] at 

_________@___________. 

If you have any questions about the inventory or the instructions for completing it, please 

contact _______________ at 1-xxx-xxxx or by email at _____@_________. 

Thank you for your continued support of the national evaluation of the comprehensive 

center program. 

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless it displays a valid OMB control number. The valid OMB control number for this data collection 

instrument is 1850-8023. The time required to complete these worksheets is estimated to average 16 hours per 

response, including the time to review instructions, search existing data sources, gather the data needed, and fill in 

the form. Send comments regarding this burden estimate or any other aspect of this collection of information, 

including suggestions for reducing the burden, to the U.S. Department of Education, Washington, DC 20202-4651. 

B-5 
Project Inventory Form 

Instructions for Completing an Inventory of Projects 

Each Comprehensive Center will prepare an inventory of all of the projects that were 

active during the current grant period. The first inventory will cover the period from July 1, 2006 

through June 30, 2007. The second inventory will cover materials from July 1, 2007 through 

June 30, 2008, and the third inventory will cover July 1, 2008 through June 30, 2009. 

Projects that began in the current project year but will not be complete by June 30 should 

be entered in the inventory and noted as “ongoing.” 

Purpose 

This inventory will serve as the sampling frame for the expert panel reviews to be 

conducted for the national evaluation. A sample of projects will be selected in two ways. Each 

Center will nominate several projects that they believe best represents the work undertaken by 

that Center. The remaining projects will be purposefully sampled from the remaining projects on 

each Center’s inventory. Samples will be drawn to reflect each Center’s overall portfolio of 

work, as well as the work of all Regional or Content Centers in key topic areas. Items selected 

for review will be rated by an expert panel for their quality. 

If a project is nominated or sampled for expert panel review and rating, the evaluation 

team will ask your Center to collect and transmit all of the documents and other artifacts 

associated with that project (meeting agendas, briefing books, meeting summaries, training 

materials, white papers, web resources, etc.) for the expert panel to review. 

Identifying Projects for Inclusion in the Inventory 

The unit of analysis for the expert review panels will be the project. For the purposes of 

this inventory, a “project” will ordinarily comprise a group of closely related activities and/or 

deliverables designed to achieve a specific outcome for a specific audience. The Content 

Centers have some projects that consist of the development of a single product or deliverable, 

rather than a group of deliverables, but a project that comprises only a single product or 

deliverable will be the exception rather than the rule. The inventory should include all of the 

projects developed or delivered by the Center, including those developed or delivered in 

collaboration with other Centers, during the reporting period. 

Because each project listed in the inventory could potentially be sampled for expert panel 

review, each project shown as an entry (or row) in the inventory form should be a relatively 

complete project that can be understood and rated on its own by expert panel members who may 

not know anything about other aspects of the Center’s work. Although a single project may 

include a number of deliverables and activities, it will be designed to achieve a specific outcome 

and address (in almost all cases) a single topic. Where a group of activities and deliverables can 

be divided up into separate projects, each constituting a complete and coherent whole, the Center 

should list these as separate projects in the inventory. 

The following criteria should guide the Centers as they identify projects (and their 

associated activities and deliverables) for recording in the inventory form. They have been 

B-6 
Project Inventory Form 

developed to ensure that projects constitute units that are large enough for review and rating, but 

focused enough for coherence. Each project entered in the inventory should satisfy all three of 

the following criteria: 

■	 Complete and coherent whole. Each project listed in the inventory should be 

able to stand on its own in an expert panel review. Avoid listing activities and 

deliverables as separate projects in the inventory if they cannot be understood or 

evaluated without information about related activities and deliverables. For 

example, a training event may require extensive planning (e.g., needs assessment, 

materials development) and follow-up activities (e.g., evaluation of the training, 

consultation on action plans). These planning and follow-up activities would be 

very difficult for a panel to rate in the absence of information about the event 

itself. Therefore the Center should list these planning and follow-up activities and 

the event itself as a single project (one row) on the inventory form. Each phase of 

the project—planning, the event itself, and follow-up—will be described briefly 

in the “activities and deliverables” column. Similarly, ongoing work with a statelevel task force should be listed as a single project rather than each meeting of the 

task force being listed as a project. 

■	 Common intended outcome. Where a cluster of activities and deliverables is 

designed by the Center to lead to the same outcome for the same audience(s), 

those activities and deliverables should be grouped as one project in the inventory 

form. On the other hand, where a set of activities is intended to produce more than 

one distinct outcome—for example, helping the state to develop a strategic plan 

for improving reading instruction, and helping the same state recruit and train 

literacy coaches—those two activities should be listed as separate projects. Where 

the Center replicates the same set of activities in each of several different states, 

that set of activities should be listed as a single project if the intended outcomes 

and processes do not differ materially from one state to another. Where the 

intended outcomes do differ substantially from state to state, the work in each 

state should be listed as a separate project. 

■	 Topic area focus. With few exceptions, a project addresses just one of the 14 topic 

areas described in Exhibit A at the end of this document (e.g., state systems of 

support, reading/language arts). Where it is possible to divide a group of related 

activities into two different projects according to the topic area addressed, Centers 

should do so. For example, a regional forum on interventions for low-performing 

students in reading and mathematics that offers separate strands of sessions in each 

subject area should be listed as two different projects, one under reading/language 

arts and one under mathematics. In this case, all of the sessions on reading 

interventions would stand on their own as a complete and coherent whole for rating 

by an expert panel and should be listed as a separate project; the same would be true 

for the sessions on mathematics. (If a project cuts across topic areas and the 

activities and deliverables which it comprises cannot be divided up by topic area 

into complete and coherent units that would make sense to a review panel, the 

B-7 
Project Inventory Form 

project should be listed under the most relevant topic area with a note crossreferencing other topics, following the instructions for column IV below.) 

The sample inventory form at the end of this packet includes examples of the kinds of 

projects that should be listed by the Centers. 

 Some projects, activities, and deliverables should not be included on the inventory form 

at all. These include: 

■	 Training or professional development for Comprehensive Center staff 

■	 Work on coordinating committees within the Comprehensive Center network 

■	 Annual needs assessment activity or negotiations with states, unrelated to specific 

projects 

■	 Other internal working meetings or documents 

Completing the Inventory Form 

Centers should use the attached form to complete their inventory of projects. Sample 

projects and examples of entries can be found in Exhibit B. 

Centers might find it useful to review their annual project plans, technical assistance 

plans, management plans or technical assistance logs as a starting point since those documents 

typically provide an overview of the various projects and activities that were planned for or 

conducted during the year. 

Once the inventory form is complete, Centers should designate which projects they would 

like to nominate for inclusion in the sample of projects reviewed by the expert panels, by inserting 

** after the name of the project in Column II. Centers should designate several projects. 

List each project under the appropriate topic heading. A list of topic headings, with 

definitions, is attached at the end of this document in Exhibit A. 

Directions for completing each column are as follows: 

B-8 
Project Inventory Form 

Topic Area Headings List items under the appropriate topic area heading. Where a project 

fits under more than one topic heading, list it once, under the topic 

heading that is most relevant. Note the project’s relevance to other 

topics in Column IV. Add rows to the table as needed. Leave rows 

blank if your Center does no work in a particular topic area. 

I. Project number Assign consecutive numbers to each item listed in the inventory. 

(Centers may want to complete this column as a final step, after all of 

the items have been entered in the inventory.) 

II. Name Enter the name of the project. Projects nominated by Center staff for 

review should be followed by ** in this column. 

III. Description Provide a concise description of the project. (See examples for 

appropriate level of detail.) 

IV. Additional topics If the project addresses more than one topic area, note that here. Use 

addressed? the list of topic areas provided in the appendix (this list corresponds 

with the row headings in the inventory). Entering the appropriate 

number from the list will save space. 

V. Activities and List all of the activities and deliverables associated with the project. 

Deliverables (For products, include exact title, if applicable. For services, include 

location and type of participant.) Specific activities and deliverables 

may include: 

Meetings/conferences (includes items such as workshops, 

conferences, institutes, forums, webinars) 

Expert consultation/technical assistance (includes items such as 

assistance completing reports or applications, review of state plans, 

needs assessments, audits) 

Facilitation/support of working groups or teams (includes items
 

such as planning meetings, participation in meetings, drafting 

summary documents) 

Guidance/information resources (includes items such as policy or 

issue briefs, fact sheets, congressional testimony, resource guides, 

planning tools, field guides, benchmarking rubrics, handbooks, 

exemplars, literature reviews/summaries, annotated bibliographies, 

case studies, websites) 

Training (includes items such as professional development 

materials/services, software, training materials) 

Other, specify ______________________ 

VI. Start Date Enter the start date for the project, including month and year. 

VII. End Date Enter the end date for the project, including month and year. 

If the project is currently ongoing, enter the note “ongoing.” 

B-9 
VIII: Major, Indicate whether you consider this a major, moderate or minor project 

Moderate or Minor in terms of the relative level of effort and/or resources devoted to it. 

Project 

IX: Target state(s), Regional Comprehensive Centers serving multiple states should note 

region(s), or regional which state(s) participated in each project. Regional Centers serving a 

center (s) single state (Alaska, California, New York, and Texas) should note 

which region(s) within the state participated (e.g., New York City vs. 

rest of state). Content Centers should specify which Regional Centers 

participated in the project. 

X: Collaborations and If the Center used materials developed by one of the Content Centers 

Sources: Other CCs in the course of designing or delivering its own services, list this 

Content Center as a source. 

If the Center collaborated with another Regional Center or Content 

Center on the design, development and/or delivery of products and 

services, list this Center as a collaborator. 

Note collaborations with other Comprehensive Centers only in this 

column. 

XI: Collaborations Note collaborations with other technical assistance providers, such as 

and Sources: Other Regional Educational Laboratories or universities, in this column. 

TA Providers If the Center used materials developed by another TA provider in the 

course of designing or delivering its own services, list this provider as 

a source. 

If the Center collaborated with another TA provider on the design, 

development and/or delivery of products and services, list this 

provider as a collaborator. 

Project Inventory Form 

B-10 
National Evaluation of the Comprehensive Technical Assistance Centers 

Inventory of Projects 

Center Name: ________________________________ 

B-11
 

I. No. II. Name III. Description IV. Additional 

topics addressed? (From list in Exhibit A—enter appropriate number) 

V. Activities and Deliverables 

VI. Start Date VII. End 

Date (enter “ongoing” if project is not complete) 

VIII. Major, moderate or minor project 

IX. Tarstatregwitstatregcen

1. Components of Effective Systems of Support (State, District, School) 2. Data Use / Data-driven Decision Making 3. Formative Assessment 4. Reading 5. Adolescent Literacy 6. Mathematics 7. Drop Out Prevention 8. High School Redesign/Reform 9. Transition to High School 
B-12 

I. No. II. Name III. Description IV. Additional 

topics addressed? (From list in Exhibit A—enter appropriate number) 

V. Activities and Deliverables 

VI. Start Date VII. End 

Date (enter “ongoing” if project is not complete) 

VIII. Major, moderate or minor project 

IX. Tarstatregwitstatregcen

10. Special Education – curriculum, instruction, and professional development 11. Special Education - assessment 12. English Language learners 13. Highly Qualified Teacher Provisions of NCLB 14. Teacher Preparation and Induction 15. Teacher Professional Development 16. Supplemental Education Services (SES) 17. Response to Intervention (RtI) 18. Migrant Education 19. Indian / Native American Education 20. Data Management Compliance 

B-12
 
B-13 

B-13
 

I. No. II. Name III. Description IV. Additional 

topics addressed? (From list in Exhibit A—enter appropriate number) 

V. Activities and Deliverables 

VI. Start Date VII. End 

Date (enter “ongoing” if project is not complete) 

VIII. Major, moderate or minor project 

IX. Tarstatregwitstatregcen

21. Assessment Design 22. Parent Involvement 23. Other 

**Projects nominated by the Center for review by expert panels 
 

 

 

Project Inventory Form 

Exhibit A 

Topic Areas and Definitions 

Topic Definition 

1. Components of effective 

systems of support–state, 

district, school. 

This category included: (1) Design of state systems to meet NCLB 

requirements for statewide systems of support for districts and schools in 

corrective action or identified as in need of improvement [Sec. 1117(a)]; (2) 

Interventions for schools or districts in need of improvement or corrective 

action; (3) For schools, Centers' projects addressed a range of options for 

school leadership and organization, professional development and classroom 

practice, and development and implementation of school improvement plans. 

For districts, Centers' projects typically addressed district leadership and 

development as well as implementation of district improvement plans. 

2. Data use/Data-driven 

decisionmaking. 

Use of data for educational program improvement. 

3. Formative assessment. Guidelines for implementing formative or benchmark assessment, 

comprehensive formative assessment systems. 

4. Reading. Alignment of curriculum, instruction, and assessments with state standards in 

reading; teacher professional development in reading (not specific to 

adolescent literacy). 

5. Adolescent literacy. Policy and practices relevant to literacy at middle school and high school 

levels. 

6. Mathematics. Alignment of curriculum, instruction, and assessments with state standards in 

mathematics as well as teacher professional development in mathematics. 

7. Dropout prevention. Policies and practices specifically designed to keep high school students in 

school through graduation. 

8. High school 

redesign/reform. 

High school renewal, school organization, curriculum, and instructional 

practice. 

9. Transition to high school. Policies and practices designed to improve the preparation of middle-school 

students for success in high school. 

10. Special education– 

curriculum, instruction, 

and professional 

development. 

Curriculum, instruction, and professional development for service to students 

with disabilities. 

11. Special education– 

assessment. 

Implementation and interpretation of assessments for students with 

disabilities, including alternate assessments. 

12. English Language 

Learners. 

Curriculum, instruction, and professional development for service to ELL 

students; Title III improvement plans. This category also includes 

determination of ELLs’ achievement of AYP through assessment of Annual 

Measurable Achievement Objectives; implementation and interpretation of 

ELL assessments. 

13. Highly Qualified Teacher 

provisions of NCLB. 

State plans and policies required in connection with the HQT provisions of 

NCLB, including HOUSSE and equitable distribution. 

14. Teacher preparation and 

induction. 

State policies and practices for pre-service teacher preparation and induction 

of novice teachers as well as activities targeting teacher retention. 

B-14 
 

 

 

Project Inventory Form 

Topic Definition 

15. Teacher professional 

development. 

State policies and local practices for the planning and delivery of 

professional development for teachers (not specific to reading, mathematics, 

high schools, ELLs, or special education). Activities in the area of effective 

teaching are included in this category. 

16. Supplemental 

Educational Services 

(SES). 

Application process to select vendors, monitoring services, and evaluating 

performance of providers. 

17. Response to Intervention 

(RtI). 

RtI implementation, moving RtI into general education, RtI as a strategy for 

school improvement. 

18. Migrant education. Statutorily required state Comprehensive Needs Assessment (CNA) process 

for migrant education. 

19. Indian/Native American 

education. 

Policies and practices to support the educational needs of American Indians, 

Alaska Natives, and Native Hawaiians; includes support of and compliance 

with ESEA Title VII. 

20. Data management 

compliance. 

Title I compliance, EDEN/EDFacts reporting requirements. 

21. Assessment design. Developing new assessments for use by clients; includes classroom, 

benchmark, and state grade-level assessments for federal accountability; 

activities may include design, item development, validation, reliability 

testing, alignment studies. 

22. Parent involvement. Title I and other parent involvement activities. 

23. Other Includes projects that do not fit under any other topic area, including 

websites, membership on SEA committees, and ongoing phone support. 

Includes projects building general skills and knowledge of leaders at all 

levels. Includes projects that assist clients with the review, collection, and 

translation of research, where there is no specific topic. If the research 

addresses a specific topic area, put the project in the appropriate row. Does 

not include internal or network meetings—these types of activities should 

not be included in the project inventory. 

B-15 
B-16
 

Sample Completed Project Inventory 

Center Name: ______________________________ Note: All examples in the table were taken from Year 2 baseline management plans. 

I. No. II. Name III. Description IV. 

Additional 

topics addressed? 

V. Activities and Deliverables VI. 

Start Date 

VII. End Date 

VIII. Major, 

moderate or minor project 

1. State Systems of Support for Schools Identified for Improvement 

1. State TA 

system for PI districts and schools 

Designed an integrated SEA TA system that reaches “program improvement” districts and schools 

• Convened coordinating council; • Created inventory of existing TA 

efforts; 

• Helped create TA plan; • Gave PD and TA to school support 

teams 

7/2006 Ongoing Moderate 

2. Statewide 

system of support** 

Supported development of Regional School Service Centers as a system of support for addressing NCLB 

• Helped create a plan to guide the 

Regional Network Strategy for the 

next five to seven years; 

• Developed the next RFP for the 

RSSCs; 

• Supported RSSC implementation; 

• Developed and helped deliver PD; • Developed a protocol for collecting 

information on implementation 

7/2006 6/2007 Major 

3. “Significant 

Change in School Improvement and Restructuring” 

Developed a modular handbook and workshop on implementation of fast-paced significant school improvement, including restructuring 

2 (Local 

capacity) 

• Drafted 8 modules for handbook to 

be used with SEA systems of support; 

• Presented at institute; • Revised and added workshop 

materials; 

• Consulted at CC A regional 

meeting 

Major 
B-17
 

4. Support to 

School Improvement Team 

Helped the SEA School Improvement Team to identify resource and design training for schools in “monitored” and “priority” statuses 

• Attended meetings organized by 

SIP team and provided reflective syntheses; 

• Brokered resources and services 

available through RRC and CC networks 

7/2006 6/2007 Minor 

2. Building District/Local Capacity to Support School improvement, or Address Corrective Action and Restructuring 

5. District tools 

for monitoring schools 

Helped the SEA develop processes and tools for districts to use in monitoring the implementation of school restructuring plans 

• Prepared draft processes and tools 

for monitoring; 

• Met with SEA school improvement 

staff to review/revise tools 

7/2006 10/2006 Minor 

6. Leadership 

Institute on Helping Districts Assist LowPerforming Schools 

Designed and implemented a PD plan to support the Local Education Agency Assistance Program 

• Convened planning team to design 

the Leadership Institute; 

• Helped conduct institute; • Provided ongoing support to 

participants as they work with districts and schools 

7/2006 6/2007 Moderate 

7. Guidance to 

districts on restructuring schools 

Assisted SEA in developing guidance for districts with schools in restructuring 

• Assisted state in writing guidance 

protocol; 

• Piloted guidance; • Collected and prepared research; • Provided initial training to districts 

with schools in restructuring 

7/2006 12/2006 Moderate 

3. Assessment (excluding assessment of special education students or ELL students) 

8. Growth 

models 

Built SEA understanding of assessment to help in decision making about accountability growth models 

• Identified state needs and create 

plan for growth model; 

• Assisted states accepted as pilot 

states; 

• Assisted ongoing development for 

other states 

7/2006 Ongoing Major 
B-18
 

9. Assessment 

data analysis** 

Built SEA and other staff capacity by helping them train districts and schools in analyzing state assessment results and using them for instructional planning 

• Met with SEA staff to plan PD; • Helped host Statewide Assessment 

Conference; 

• Continued to plan and offered 

training in assessment analysis; 

• Established a cadre of educators to 

provide PD to districts 

5/2006 6/2007 Major 

4. Reading/Language Arts Curriculum, Instruction, and/or Professional Development (includes Adolescent Literacy) 10. Adolescent 

Literacy Cadres 

Organized and delivered training of adolescent literacy cadres made up of SEA staff 

• Convened SEA work teams to 

analyze research; 

• Reviewed literacy diagnostic tools; • Customized a literacy assessment; • Conducted PD for SEA work teams 

on customized assessments 

10/200 6 

6/2007 Moderate 

11. Effective 

Leadership in Literacy 

Built SEA capacity to support “effective leadership in literacy for grades 6-12” 

• Convened work group to discuss 

research; 

• Provided SEA with PD on Strategic 

Instruction Model; 

• Developed state literacy plan 

7/2006 6/2007 Major 

5. Mathematics Curriculum, Instruction, and/or Professional Development 12. K-2 

Mathematics Inventory 

Developed and provided PD to a cadre of educators to help schools implement the state K-2 Mathematics Inventory 

• Helped SEA design PD; • Helped design team pilot 

workshop; 

• Revised PD; • Helped deliver regional trainings 

prioritized for schools in improvement; 

• Reviewed feedback to redesign PD 

7/2006 6/2007 Major 
B-19
 

7. High School Reform 13. Implementing 

High School Task Force Report 

Developed capacity of SEA to implement recommendations of HS Task Force Report related to high school redesign 

• Work group and Steering 

Committee met to consider task force recommendations, research, best practice, and review of prior initiatives; 

• Identified “knowledge 

management” components needed to support state implementation; 

• Identified policies and practices 

needed to promote redesign; 

• Developed multi-year 

implementation plan. 

9/2006 4/2007 Moderate 

8. Special Education 14. Assessing 

special needs students 

Supported SEAs in establishing assessment and accountability systems that include and support special education, ELL, and low-income students 

10 (ELL) • Collected information from Content 

CC A on research- based practices; 

• Conducted SEA Needs 

Assessment; 

• Provided regional training(s) on 

Scientifically Based Research on assessment and accountability systems for special needs students 

7/2006 Ongoing Major 

15. Developing an 

Alternate Assessment 

Assisted SEAs in developing Alternate Academic Assessments for the “1” of students who have significant cognitive disabilities 

3 

(Assessmen 

t) 

• Identified needs and broker 

assistance in developing assessment; 

• Provided TA on development of 

assessment; 

• Provided ongoing TA on 

implementation 

7/2006 6/2007 Moderate 

16. Research 

Synthesis on Special Needs Students 

Published Research Synthesis 1 focusing on teaching specialneeds students 

12 (HQT) • Identified and evaluated existing 

research and syntheses; 

• Conducted gap analysis to identify 

research synthesis needs; 

• Wrote synthesis (50-60 pages); • Posted on on-line resource forum. 

7/2006 9/2007 Major 
B-20
 

9. English Language learners 17. District 

guidance on the education of ELL students 

Helped develop a comprehensive framework for educating English Language Learners to guide district work, including guidance on the use of formative assessments to improve instruction and on family and community engagement 

13 (Parents) • Facilitated meetings of the SEA 

task force responsible for developing and disseminating a comprehensive framework for the education of ELLs; 

• Assisted in integrating effective 

formative assessment practices for ELL students; 

• Assisted in formulating guidance 

on ELL family and community engagement strategies; 

• Provided consultation and 

resources from Content CC A 

7/2006 6/2007 Major 

18. Addressing 

AMAOs 

Assisted SEA in aligning Annual Measurable Achievement Objectives, standards, and assessments for ELL students 

• Conducted a study on alignment of 

English Language Proficiency standards for grades K-12 with the state assessment for ELL students 

• Helped state reconfigure AMAOs 

to align with state test 

8/2006 12/2006 Minor 

10. Highly Qualified Teachers 19. “TQ Source 

Project” 

Developed online resource clearinghouse that provides clients with policy and research data specifically related to teacher quality, as addressed in NCLB 

• Developed “interactive data tool” 

based primarily on NCES data; 

• Updated “policy database” for 

state-by-state policy areas, including policies related to PD and teacher prep, recruitment, retention, certification, etc.; 

• Published next issue of “Tips and 

Tools: Emerging Strategies to Enhance Teacher Quality”; 

• Enhanced “Publications database” 

focusing on teacher quality research 

7/2006 Ongoing Major 
B-21
 

20. Teacher 

Preparation 

and Licensure in literacy 

Assisted SEA in reviewing teacher preparation and licensure requirements related to literacy 

4 

(Language 

arts) 

• Assisted the Task Force on 

Licensure/Professional Development in revisiting preparation and licensure requirements related to literacy instruction for school leaders and teachers; 

• Discussed findings and possible 

next steps related to engagements with representatives from institutes of higher education 

10/200 6 

2/2007 Minor 

12. Supplemental Educational Services 21. “State 

Evaluation of Supplemental Educational Services” Manual 

Updated “State Evaluation of SES” manual and assist with its use by Regional CCs 

• Drafted updates to SES evaluation 

manual 

• Presented at September 2006 

Institute 

• Revised based on evaluations and 

usage 

• Submitted evaluation report to ED 

9/2006 4/2007 Major 

13. Other NCLB-related TA 22. Enhancing 

Schoolwide Planning Materials 

Assisted SEA in updating resources and training for district Title I directors on the topics of schoolwide planning, plan implementation, and district monitoring of school plans 

• Helped update T1 schoolwide 

application and evaluation rubric; 

• Ensured that North Central 

Association Commission on Accreditation and School Improvement school improvement processes include schoolwide plan components; 

• Reviewed need assessment results 

from district/schools; 

• Updated materials based on needs 

7/2006 6/2007 Major 

**Projects nominated by the Center for review by expert panels 
 

 

Request for Materials for Expert Panel Reviews 

REQUEST FOR MATERIALS FOR EXPERT PANEL REVIEWS 

[Date] 

Dear [Center Director]: 

I am happy to report that we have completed the sampling process for your site and have 

selected the following (n) projects for review: 

Project Name Project Description 

These projects will be rated for technical quality by panels of expert reviewers, and for 

relevance and usefulness through surveys of project participants. 

Assembling Panel Review Materials 

Attached you will find a Request for Materials form that includes some basic guidance 

for assembling materials for the panel review process. This form also includes the project 

cover sheet, which should be completed for each project. This cover sheet will be 

provided to the review panelists and is an important source of background and contextual 

information about the project. 

To facilitate the review process, we ask that you do the following: 

•	 Submit 1 hard copy and 1 electronic version of all the materials that comprise the 

review packet for each project. The hard copy version will allow us to assess whether 

the volume of material is appropriate, and will serve as a quick reference source 

should questions arise about the review packet. 

B-22 
 

Request for Materials for Expert Panel Reviews 

•	 If an item is not available in electronic format or if you would prefer the reviewers to 

have a hard copy version of a particular item (for example, a training binder with 

multiple tabs and color handouts included), please provide 3 additional hard copies of 

that item. 

•	 Each review packet should include a cover sheet (using the attached template) that 

lists all of the materials included in the packet. Electronic versions of the materials 

should be filed together in a project folder that is clearly labeled with the project 

name. Feel free to use subfolders within the project folder if you think it will help 

reviewers organize the materials (for example, background/research base, meeting 

notes, training materials, etc.) in a more understandable way. You might also want to 

include the file name of each document referenced in question #5 of the cover sheet 

so that reviewers can easily sort through and match the documents on the cover sheet 

with the files they receive. 

•	 Please send your hard copy materials (bundled by project with a cover sheet on each), 

along with a CD containing separate folders for each of the sampled projects to: 

DIR 

Attn: _______________ 

[address] 

Materials are due no later than [date]. 

I have attached a copy of the quality scoring rubric so that you can see what the panelists 

will be looking for in the review process. As you begin to assemble the materials for 

panel review, we offer the following guidance: 

•	 With the exception of the cover sheet and the participant list, all materials provided in 

the review package should be materials that already exist. We do not expect you to 

create any new materials for the review process. 

•	 Materials included in the packet should focus on work conducted between summer 

200X and summer 200X. Materials developed before summer 200X can be included 

if they provide relevant contextual or background information. Materials developed 

after summer 200X will be considered in the next round of panel reviews (for the 

200X-200X program year), if that particular project is sampled again for review. 

•	 Make sure you have documented the “knowledge base” upon which the project was 

developed. As you can see in the attached rubric, this is a very important factor in the 

quality rating. Depending on the topic area and nature of the project, the knowledge 

B-23 
 

 

 

Request for Materials for Expert Panel Reviews 

base might include empirical research, laws and regulations, or professional 

wisdom81. 

•	 Provide enough materials to allow reviewers to understand the project substantively, 

with a particular focus on the work conducted in the 200X-200X program year. This 

does not mean sending everything ever done or developed in connection with the 

project. You want to provide reviewers with a sufficient basis for their ratings but at 

the same time, not overwhelm them with materials. 

•	 Focus more on the substantive materials than on those dealing with process or 

administrative issues alone. You do not need to include every e-mail message or set 

of meeting notes on a project unless it provides critical contextual information or 

speaks directly to the technical quality of the project. For example, if you are 

assembling materials about an annual forum or institute, you should not include 

meeting notes about conference logistics or attendee evaluation results, but you 

should include notes relating to the development of the agenda and selection of 

presenters. 

If you have any questions about what materials to include in the review packet or how to 

complete the cover sheet, please contact your site visitor. 

Compiling Lists of Project Participants 

For each project selected for the evaluation, a sample of participants will receive a short, 

web-based survey from the national evaluation. This is the reason we are asking each 

Regional Comprehensive Center (RCC) and each Content Center to compile a list of all 

state-level staff who have participated in each project sampled for the evaluation. Content 

Centers should also include all RCC-based staff, as well as all state-level staff, on their 

lists of project participants. Compile a separate list of participants for each project 

sampled, allowing duplication of names across lists. 

Defining “participants.” “Project participants” include all those who have served on task 

forces and work groups associated with the project; state-level or RCC staff who have 

attended conferences, technical assistance retreats, consultations, and other meetings that 

are part of the project; and state-level or RCC staff who have received written materials 

or other resources disseminated under the project. 

If the Comprehensive Center collaborated with state-level staff to provide a service to 

others, and that collaboration was intended to build the capacity of the state in some way, 

then all of the state-level staff who collaborated with the Center on the project should be 

81 Professional wisdom is the judgment that individuals acquire through experience and is reflected through 

consensus views on decisions on implementation. Grover J. Whitehurst. Evidence-Based Education, 

2002. 

B-24 
 

 

 

Request for Materials for Expert Panel Reviews 

included on the list of project participants. These collaborators will be included in the 

pool of potential survey respondents. 

The same logic applies to Content Center projects that involve collaborations with RCC 

staff. Again, if the collaboration was intended to build the capacity of the RCC in some 

way, then all RCC staff who collaborated on the project with the Content Center should 

be included on the list of project participants and will be included in the pool of potential 

survey respondents. 

Defining “state-level.” For the purposes of this evaluation, “state-level” participants in 

Regional Center or Content Center projects may include the following: 

•	 State education agency (SEA) employees and consultants 

•	 Employees of other state agencies, such as governors’ offices 

•	 Employees of intermediate education agencies82 who assist schools on behalf of 

the state 

•	 Individuals who serve on school support teams as part of a statewide system of 

school support under NCLB, including local educators (school district 

administrators, principals, and teachers) 

•	 Individuals who serve on state-level work groups or task forces, including local 

educators 

Local educators who are not serving on school support teams and are not members of a 

state-level work group or task force should not be included in project participant lists. 

RCC staff. Content Centers should include all RCC staff who have participated in each 

project, in addition to state-level project participants. Participant lists for some Content 

Center projects may include no RCC staff, some may include both RCC and state-level 

staff, and some may be made up entirely of RCC staff, depending on the nature of the 

project. 

Contact information. For each project participant, please provide a name, title (if 

available), affiliation, email address, telephone number, and regular mail address. 

Because surveys will be administered via the web, with follow-up by regular mail and by 

telephone, all of this contact information is critical. Send a separate participant list for 

82 Intermediate education agencies are usually established by state statute, but their governance structures 

and funding sources vary from state to state. Depending on the state, they are known as Area Education 

Agencies (AEA), Boards of Cooperative Educational Services (BOCES), Cooperative Education Service 

Agencies (CESA), County Offices of Education (COE), Education Service Centers/Cooperatives (ESC), 

Education Service Districts (ESD), Regional Education Service Agency (RESA), or Regional Education 

Service Centers (RESC). Association of Educational Service Agencies, “Questions Asked About 

Educational Service Agencies,” downloaded from http://www.aesa.us/Q&ABro04.pdf on July 6, 2007. 

B-25 
 

Request for Materials for Expert Panel Reviews 

each sampled project, following the template shown below (using either Word or Excel to 

create the file). 

Project name: Really Important Project 

Last 

name 

First 

name 

Title (if 

available) 

Affiliation/ 

Organization Email address 

Telephone 

number 

Street 

address City State ZIP 

Doe John Title I 

Director 

State Department of 

Public Instruction 

jdoe@sea.k12.us (101) 555-1234 99 State 

Street, 

Room 100 

State 

Capital 

ST 10001 

Etc. 

Participant lists for each project should be included with your sample review materials 

and sent to DIR no later than [date]. 

For additional information. If you have any questions about the participant lists, or if 

there are special circumstances pertaining to a project or to a group of participants that 

you would like to discuss, please contact _______________ at Policy Studies Associates. 

_______________ can be reached at (xxx) xxx-xxxx or by email at _____@_________. 

Thank you for your continued cooperation. We look forward to receiving your materials. 

Sincerely, 

Vice President 

Decision Information Resources 

B-26 
 

Request for Materials for Expert Panel Reviews 

National Evaluation of the Comprehensive Technical Assistance Centers 

Request for Materials for Expert Panel Review
 

The projects below have been sampled from the Project Inventory Form that your Center completed for the most 

recent project year. These projects will be rated for quality by expert review panelists for the national evaluation 

of the Comprehensive Technical Assistance Centers. Surveys of participants in these projects will collect client 

ratings of the projects’ relevance and usefulness. 

Project Activities and Deliverables 

In order to facilitate this process, for each project please provide us with: 

(1) Copies of the materials that together fairly represent each of these items. This should include: documents 

(electronically, wherever possible) leading up to, or resulting from, these efforts–including plans, agendas, 

meeting notes, handouts, presentations, follow-up memos, resolutions, instruments, assessments, summaries, 

syntheses, papers, reports and memoranda of agreement. In other words, we are asking for materials that fully 

describe these products and services for the reviewers and give them a sufficient basis for rating their quality. 

(2) A cover sheet (one per project) that provides background and contextual information about the project. (See 

attached page for specific questions to be answered on the cover sheet.) 

(3) A list of the state-level participants in each project, including names, affiliations, and contact information. 

Questions about what materials to provide and how to complete the cover sheet should be directed to your 

evaluation team liaison. 

These materials will be the basis upon which the reviewers will make their ratings. 

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless it displays a valid OMB control number. The valid OMB control number for this data 

collection instrument is 1850-0823. The time required to complete these worksheets is estimated to average four 

hours per response, including the time to review instructions, search existing data sources, gather the data needed, 

and respond to the questions. Send comments regarding this burden estimate or any other aspect of this collection 

of information, including suggestions for reducing the burden, to the U.S. Department of Education, Washington, 

DC 20202-4651. 

B-27 
Request for Materials for Expert Panel Reviews 

Cover Sheet to Accompany Materials Sent to Review Panel for Sampled Projects 

Name of Comprehensive Center ________________________________ 

Name of person completing this form ____________________________ 

Telephone ________________ Email ____________________ 

Information about Sampled Projects: 

Project Name ______________________________________ 

Project Inventory # _______ 

1.	 Describe the purpose of this project and its intended outcome(s). 

2.	 Who is the customer/client? 

3.	 Describe the circumstances that led to the provision of this project. In this description, 

be sure to indicate whether it was initiated by the Center or requested by the client. 

4.	 List the activities and deliverables associated with this project, and describe how they 

relate to the overall project. 

5.	 Please provide a list of the materials and documents you are sending for the review 

panel. For each, describe the Comprehensive Center’s role and contribution to the 

materials associated with this project (i.e., the materials sent to reviewers for rating 

the quality and relevance of this project.) Please let us know if you used materials 

developed by another source and, if so, provide information about that source. 

6.	 What is the research basis for the product or service? Where appropriate, please 

provide a citation for the research or practice literature upon which it is based. 

B-28 
 

Sample Follow-Up Memo 

Dear [Center Director]: 

Members of the evaluation team met in Houston last week to look over the review 

materials submitted by the Centers. In our cursory review of the PCC projects, we had 

trouble locating certain items that were either described on the project cover sheet(s) or 

that one would expect to see in the project materials. 

Project Name Issue Identified 

If these items were included in your review package, please let us know where in the 

materials we can locate them so that we can make sure that the peer review panelists 

will be able to find them. 

If you didn’t intend to include them in the package, please let us know. 

If you would like to send new copies of these items or additional materials to address 

the issue(s) identified, please forward copies electronically to _____@_________ for hard 

copies to [name], DIR, [address] no later than [date]. 

If you have any questions, please feel free to contact me. 

Best, 

[name] 

B-29 
Exhibit B.2. Distribution of all major and moderate projects and projects 

in the evaluation sample, by topic in 2006-07 

Project topic 

Number of 

major and 

moderate 

projects on 

project 

inventories 

Number of 

projects in the 

sample 

(all major or 

moderate) 

Percent of all 

major and 

moderate 

projects in 

evaluation 

sample 

Total 216 122 56% 

Components of effective systems of 

support—state, district, school 

Data use/data-driven decisionmaking 

Formative assessment 

Reading 

Adolescent literacy 

Mathematics 

Dropout prevention 

High school redesign/reform 

Transition to high school 

Special education—curriculum, instruction, 

and professional development 

Special education—assessment 

English language learners 

Highly qualified teacher provisions of NCLB 

Teacher preparation and induction 

Teacher professional development 

Supplemental educational services (SES) 

Response to Intervention (RtI) 

Migrant education 

Indian/native American education 

Data management compliance 

Assessment design 

Parent involvement 

Other 

54 

5 

5 

17 7 41 

11 

11 

4 

10 

4 

5 

1 

17 

8 

6 

8 

9 

7 

4 

2 

8 

2 

6 

12 

36

2 

1 

5 

6 

2 

5 

3 

3 60

1 

11 

5 

4 

7 

5 

3 

2 

1 

5 

2 

3 

3

67

40 

20 

45 

55 

50 

50 

75 

100 

65 

63 

67 

88 

56 

43 

50 

50 

63 

100 

50 

25 

EXHIBIT READS: For program year 2006-07, there were 54 major and moderate projects on the project 

inventories that focused on components of effective systems of support. Of these, 36 (or 67 percent) were in the 

project sample. 

SOURCE: Project inventory forms submitted by the Centers. 

B-30 
Exhibit B.3. Distribution of all major and moderate projects and projects 

in the evaluation sample, by topic in 2007-08 

Project topic 

Number of 

major and 

moderate 

projects on 

project 

inventories 

Number of 

projects in the 

sample 

(all major or 

moderate) 

Percent of all 

major and 

moderate 

projects in 

evaluation 

sample 

Total 222 122 55%

Components of effective systems of 

support—state, district, school 

Data use/data-driven decisionmaking 

Formative assessment 

Reading 

Adolescent literacy 

Mathematics 

Dropout prevention 

High school redesign/reform 

Transition to high school 

Special education—curriculum, instruction, 

and professional development 

Special education—assessment 

English language learners 

Highly qualified teacher provisions of NCLB 

Teacher preparation and induction 

Teacher professional development 

Supplemental educational services (SES) 

Response to Intervention (RtI) 

Migrant education 

Indian/native American education 

Data management compliance 

Assessment design 

Parent involvement 

Other 

68 

4 

6 

8 2 25

15 

8 

2 

10 

3 

3 1 33

2 

17 

13 

3 

6 

4 

13 

7 

3 

7 

4 

5 

11 

39 

3 

3 

6 

4 

2 

6 

3 100 

2 

11 

7 

2 

4 

2 

6 

3 

0 

6 

1 

2 

7

57

75 

50 

40 

50 

100 

60 

100 

65 

54 

67 

67 

50 

46 

43 

0 

86 

25 

40 

64

EXHIBIT READS: For program year 2007-08, there were 68 major and moderate projects on the project 

inventories that focused on components of effective systems of support. Of these, 39 (or 57 percent) were in the 

project sample. 

SOURCE: Project inventory forms submitted by the Centers. 

B-31 
 

Exhibit B.4. Survey of project participants sampling and survey 

administration summary, by year 

2006-07 2007-08 2008-09
 

Sample frame (a) 

Participants in RCC projects 

Participants in CC projects 

3,904 

2,689 

1,215 

4,011 

1,870 

2,141 

3,417 

1,788

1,629 

Study sample (b) 

Sampled participants from RCC projects 

State-level participants 

Sampled participants from CC projects 

State-level participants 

RCC staff 

1,658 

1,166 

188 

304 

1,710 

1,087 

296 

327 

1,364 

957 

231 

176 

Survey administration 

Completed surveys 

Respondent participated in project 

(c) 

Respondent did not participate in 

project 

Refusals 

1,208 

947 

261 

90 

1,319 

933 

386 

11 

1,035 

750

285

9

No response 360 380 320 

Response Rate 73% 

(1,208/1,658) 

77% 

(1,310/1,710) 

76%

(1,035/1,364) 

NOTES: Sampled respondents who indicated that they did not participate in the project were excluded from the 

analyses. To permit generalization to all participants, the survey responses were weighted for all the sampled 

projects in the following manner: total number participants for that project (a) divided by the number of respondents 

completing a survey who participated in the project (c). For project-level analyses, each respondent had equal weight 

in the individual project’s rating, and for Center-level analysis, the team weighted each sampled project from that 

Center equally.
 

EXHIBIT READS: For the 2006-07 program year, the sample frame included 3,904 participants of which 2,689 were 

participants in RCC projects and 1,215 were participants in CC projects. 

SOURCE: Surveys of project participants. 

B-32 
Exhibit B.5. Factor analysis of relevance and usefulness items from the 

project participant surveys, by year 

2006-07 2007-08 2008-09
 

Completed surveys 

(unweighted) 1,208 1,319 1,035

Relevance 

Eigenvalue 5.4 5.6 5.7 

Percent of variance 

explained
 67.6 69.8 72.1

Reliability coefficient 

(Cronbach’s Alpha) 0.93 0.94 0.94

Usefulness 

Eigenvalue 7.6 7.6 7.7 

Percent of variance 

explained
 69.2 69.3 69.5

Reliability coefficient 

(Cronbach’s Alpha) 0.95 0.95 0.95

EXHIBIT READS: Among the eight components of relevance, one factor was extracted with an 

eigenvalue of 5.4 and explained 67.6 percent of the variance, for the program year 2006-07. 

Among the 11 usefulness items, one factor was extracted for 2006-07 with an eigenvalue of 7.6 

and explained 69.2 percent of the variance. The reliability coefficient for the relevance scale was 

0.93 and it was 0.95 for usefulness. 

SOURCE: Surveys of project participants. 

B-33 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Exhibit B.6. Loading factors for the relevance and usefulness items from the 

project participant surveys, by year 

Relevance items 2006-07 2007-08 2008-09 

Based on your experience, to what degree was this set 

of activities and resources relevant to your work, in 

each of the following respects? 

a. Addressed a need or problem that my organization 

faces 

0.83 0.88 0.87 

b. Addressed an important priority of my organization 0.82 0.84 0.86 

c. Addressed a challenge that my organization faces 

related to the implementation of NCLB 

0.82 0.78 0.82 

d. Provided information, advice, and/or resources that 

could be directly applied to my organization’s work 

0.86 0.87 0.88 

e. Addressed our particular state context 0.80 0.83 0.84 

f. Addressed my organization’s specific challenges 

(e.g., policy environment, leadership capacity, 

budget pressures, local politics) 

0.83 0.84 0.85 

g. Provided information, advice, and/or resources that 

could be used to guide decisions about policies, 

programs, or practices 

0.85 0.86 0.88 

h. Highlighted the implications of research findings (or 

information about best practice) for policies, 

programs, or practices 

0.78 0.77 0.78 

Usefulness items 2006-07 2007-08 2008-09 

Based on your experience, to what degree was this set 

of activities and resources useful to you, in each of the 

following respects? 

a. Provided resources that were easy to understand 

and easy to use 

0.82 0.79 0.81 

b. Employed an appropriate format (e.g., a work group, 

a conference, individual consultation, written 

products) 

0.81 0.81 0.81 

c. Provided adequate opportunity to learn from 

colleagues in other states 

0.64 0.63 0.67 

d. Included adequate follow-up to support the use of 

new information and resources 

0.82 0.83 0.84 

e. Were timely 0.84 0.83 0.83 

f. Helped my organization solve a problem 0.87 0.88 0.87 

g. Helped my organization maintain or change a policy 

or practice 

0.84 0.84 0.87 

h. Helped my organization take the next step in a 

longer-term improvement effort 

0.85 0.88 0.85 

i. Provided my organization with information or 

resources that we will use again 

0.89 0.89 0.89 

j. Helped my organization develop a shared expertise 

or knowledge base 

0.88 0.88 0.87 

k. Helped individuals in my organization to develop 

skills that they will use again 

0.87 0.88 0.85 

B-34 
Appendix C 

Describing Center Operations 

 

Describing Center Operations 

This appendix contains a description of the process used to code the objectives within 

Center management plans, the project inventory forms (PIFs), and the project cover sheets. This 

appendix also includes the protocols used for the interviews with the Centers in person in the 

summers of 2007 and 2008 and by phone in Spring 2010. 

Center Management Plan, Project Inventory, and Cover Sheet 

Coding and Intercoder Reliability 

Data from the Center management plans, project inventory forms (PIFs), and the project 

cover sheets were coded by the evaluation team to build the sample frame and describe Center 

operations. Below is an explanation of the four-step coding process the evaluation team used–(1) 

establish the code sets, (2) train coders, (3) code management plans, PIFs, and cover sheets, (4) 

resolve coding disagreements–as well as a summary of the level of intercoder reliability achieved 

through this process. 

Establish the Code Sets 

Based on initial review of the project cover sheets, the PIFs, and the other materials from 

the Centers (such as the Center management plans), the evaluation team established code sets to 

use during the coding process. One set of codes was used to code the objectives stated in the 

Center management plans and the topics of the projects on the PIFs (for 2006-07). Another set of 

codes was used to code the project cover sheets and a third set of codes was used in a separate 

effort to recode the 33 project cover sheets for the 2008-09 state systems of support projects. 

Center Management Plan and PIF Code Set 

The team developed codes to summarize the primary topics in which Centers were doing 

their work based on a review of Center materials to identify categories of distinct domains or 

topics. These 22 specific topic codes are described below in terms of the substantive knowledge 

base the evaluation team determined would be covered by that topic.83 In the coding process, this 

set of possible codes was used to code the topic areas for the Center objectives as stated in their 

management plans as well as to code the topic area of each project that Centers submitted on 

their PIFs for 2006-07. 

■	 Components of effective systems of support–state, district, school. This category 

included: (1) Design of state systems to meet NCLB requirements for statewide 

systems of support for districts and schools in corrective action or identified as in 

83 Discussion in the report refers primarily to the 22 specific substantive topics. However, as a small number of 

sampled projects fell into the category of “other” topic area, description of all the projects in the study sample by 

topic as displayed in exhibit 2.5 includes the topic area of “other” as the 23rd category. 

C-1 
need of improvement [Sec. 1117(a)]; (2) Interventions for schools or districts in 

need of improvement or corrective action; (3) For schools, Centers' projects 

addressed a range of options for school leadership and organization, professional 

development and classroom practice, and development and implementation of 

school improvement plans. For districts, Centers' projects typically addressed 

district leadership and development as well as implementation of district 

improvement plans. 

■	 Data use/data-driven decisionmaking. Use of data for educational program 

improvement. 

■	 Formative assessment. Guidelines for implementing formative or benchmark 

assessment, comprehensive formative assessment systems. 

■	 Reading. Alignment of curriculum, instruction, and assessments with state 

standards in reading; teacher professional development in reading (not specific to 

adolescent literacy). 

■	 Adolescent literacy. Policy and practices relevant to literacy at middle school and 

high school levels. 

■	 Mathematics. Alignment of curriculum, instruction, and assessments with state 

standards in mathematics as well as teacher professional development in 

mathematics. 

■	 Dropout prevention. Policies and practices specifically designed to keep high 

school students in school through graduation. 

■	 High school redesign/reform. High school renewal, school organization, 

curriculum, and instructional practice. 

■	 Transition to high school. Policies and practices designed to improve the 

preparation of middle-school students for success in high school. 

■	 Special education–curriculum, instruction, and professional development. 

Curriculum, instruction, and professional development for service to students with 

disabilities. 

■	 Special education–assessment. Implementation and interpretation of assessments 

for students with disabilities, including alternate assessments. 

■	 English language learners. Curriculum, instruction, and professional 

development for service to ELL students; Title III improvement plans. This 

category also includes determination of ELLs’ achievement of AYP through 

assessment of Annual Measurable Achievement Objectives; implementation and 

interpretation of ELL assessments. 

■	 Highly qualified teacher provisions of NCLB. State plans and policies required 

in connection with the HQT provisions of NCLB, including HOUSSE and 

equitable distribution. 

■	 Teacher preparation and induction. State policies and practices for pre-service 

teacher preparation and induction of novice teachers as well as activities targeting 

teacher retention. 

■	 Teacher professional development. State policies and local practices for the 

planning and delivery of professional development for teachers (not specific to 

reading, mathematics, high schools, ELLs, or special education). Activities in the 

area of effective teaching are included in this category. 

C-2 
■	 Supplemental educational Services (SES). Application process to select vendors, 

monitoring services, and evaluating performance of providers. 

■	 Response to Intervention (RtI). RtI implementation, moving RtI into general 

education, RtI as a strategy for school improvement. 

■	 Migrant education. Statutorily required state Comprehensive Needs Assessment 

(CNA) process for migrant education. 

■	 Indian/native American education. Policies and practices to support the 

educational needs of American Indians, Alaska Natives, and Native Hawaiians; 

includes support of and compliance with ESEA Title VII. 

■	 Data management compliance. Title I compliance, EDEN/EDFacts reporting 

requirements. 

■	 Assessment design. Developing new assessments for use by clients; includes 

classroom, benchmark, and state grade-level assessments for federal 

accountability; activities may include design, item development, validation, 

reliability testing, alignment studies. 

■	 Parent involvement. Title I and other parent involvement activities. 

Cover Sheet Code Sets 

The team used the information contained in the project cover sheets in two ways. First, in 

each of the three years, the team developed codes to describe project activities and resources, 

contributions CCs made to RCC projects, and contributions RCCs made to CC projects. The 

development of these codes was informed by the team’s review of the Center management plans, 

the transcripts of the interviews conducted in 2007, and a preliminary review of all project cover 

sheets. The code sets used to code the project cover sheets for all three data collection cycles are 

shown below along with the criteria used to assign that code. The 15 possible codes for this 

effort are provided below in exhibit C.1. 

Second, for 2008-09, the team conducted a supplemental analysis of the project cover 

sheets for the 33 sampled projects under the topic State Systems of Support. For these projects, 

the team went into greater depth regarding the components of the project to more closely 

examine their content and purpose. The five possible codes for this effort are provided in exhibit 

C.2. 

C-3 
Exhibit C.1. Project cover sheet codes used across the sampled projects in all 

three data collection cycles 

Types of Project Activities and Resources 

■	 Engagement of participants in project planning was coded for projects that offered some or all 

of their participants the opportunity to take part in designing the project. This involvement went 

beyond merely expressing a need for technical assistance; rather, it included participant 

involvement in shaping the specific assistance that was delivered (who did what, when, and why 

in technical assistance) at various stages of technical assistance delivery). 

Project planning was different from “ongoing consultation and follow-up,” though the same project 

may have had both components. “Ongoing consultation and follow-up” occurred when the Center 

provided service to clients on multiple occasions, whereas “engagement of participants in project 

planning” was specific to the Center working with the client to plan later service. 

■	 Research collections or syntheses were research-based materials distributed to participants in 

a project. In some instances these products were developed by the Center conducting the project, 

but in others they were existing research materials that the Center collected and distributed. 

■	 Training events were designed to impart skills to participants, equipping them to carry out a 

particular program or strategy. The content of trainings was discrete and specialized, and 

participants attended trainings expecting to learn something specific and be able to do something 

differently afterward. Participants and trainers assumed that the primary purpose of the training 

was for the participant to learn something from the trainer. Projects with multiple trainings to the 

same participants over time were also coded as “ongoing consultation and follow-up.” 

■	 Conferences featured presentations of information to audiences from multiple speakers or 

discussants representing a range of perspectives. They focused on informing the participants, not 

training them, and they were single events rather than series of meetings. The conference scope 

was broader than trainings, in that presentations or discussions in a single conference could have 

addressed multiple issues, programs, or solutions; whereas a training focused on a single 

strategy or program. Conference participants varied in their type, interest, and expectation for 

what they wanted to learn at the conference. Conferences also included symposia and forums in 

which participants came at common questions with varying perspectives. 

■	 Support for development of a formal plan to implement a program or policy referred to 

instances in which Center technical assistance supported work by participants that led to a plan 

for implementing a program or policy. A formal state plan codified activities that were expected to 

occur in the state in order to serve state purposes with the intention to guide individuals and units 

toward a common goal. Formal plans may have been required by the federal or state 

government, or may have been initiated within the SEA to address a specific problem. This 

activity did not include service plans. 

■	 Task force meetings and work were engagements over multiple interactions in which a Center 

provided technical assistance to a work group who were themselves constituted as a task force to 

address a state problem. Center support of a task force meant that the Center was a regular 

participant in task force meetings, and Center support ranged from setting the agenda and 

facilitating meetings to providing input upon request. A project was coded as “task force meetings 

and work,” when there was a task force that was trying to accomplish some purpose—not just to 

plan or follow up on Center technical assistance, but to do something for its state (or, 

occasionally, a group of states). Center participation in a task force was also coded “ongoing 

consultation and follow-up” if the Center provided assistance on an ongoing basis over time. If the 

Center was working with the task force to plan service, the project was also coded “engagement 

of participants in project planning.” 

C-4 
Exhibit C.1. Project cover sheet codes used across the sampled projects in all 

three data collection cycles (continued) 

■	 Ongoing consultation and follow-up was either a series of consultations (i.e., fulfilling repeated 

requests from participants for technical assistance) or sustained follow-up with individuals who 

had participated in some other kind of assistance (e.g., a conference or training). Ongoing 

consultation occurred when there were multiple service contacts over time, all related to the 

project goal or topic. The interactions focused on a client question or need. This code did not 

include projects with one-time contacts, such as projects that provided a large conference with no 

follow-up. It also did not include the production of publications that are not accompanied with 

follow-up assistance directly to the client. 

RCC and CC Contributions to RCC and CC Projects 

■	 RCC projects with CC contribution were projects in which the CC explicitly contributed a 

service or resource used by the RCC or its clients as part of RCC project activities. A CC could 

contribute to an RCC project in any of three ways: 

■	 CC provided materials used in this RCC project. The RCC must have used 

CC materials in specific delivery activities or in planning. 

■	 CC provided advice to the RCC on this project. The RCC consulted or 

collaborated with the CC when making decisions about what or how to deliver assistance 

to project participants. Participants may or may not have had direct interaction with the 

CC or CC materials. 

■	 CC delivered technical assistance to RCC project participants. The CC 

provided service directly to the participants, whether face-to-face, by phone, or virtually, 

regardless of what role RCC staff had in this RCC project activity. 

■	 CC projects with RCC contribution were projects in which the RCC explicitly contributed 

assistance used by the CC or its clients as part of CC project activities. An RCC could contribute 

to a CC project in either of two ways: 

■	 RCC recruited participants or brokered service. The RCC connected the CC 

and its clients for the purpose of participation in the project. The RCC may have been 

driven by the CC’s need for participants who could benefit from a specific project, or by a 

client’s need for specific services or products that a CC could deliver. 

■	 RCC delivered technical assistance to CC project participants. The RCC 

provided service directly to participants, whether face to face, by phone, or virtually, 

regardless of what role CC staff had in this CC project activity. 

C-5 
Exhibit C.2. Project cover sheet codes for the 33 state systems of support 

projects in 2008-09 

Purposes of State Systems of Support Projects 

■	 Redesign state SSOS infrastructure or strategies for improvement included projects in which 

the Center worked with SEA leaders who were making decisions about revising design features 

of the state’s SSOS. These projects may have assisted the SEA in making changes to structural 

aspects of the SSOS, such as reorganizing units within the SEA, redefining the responsibilities of 

specific SSOS support agents, and clarifying the processes and timelines associated with 

delivery. These projects also may have assisted the SEA in making changes to the strategies the 

SSOS and its agents deliver to districts and schools to guide their improvement, such as 

intervention models, promising practices based on research, steps for improvement planning, 

indicators of improvement, and other content delivered by school support teams. 

Review of the state’s current SSOS included projects in which the Center helped the SEA review or 

assess the existing system as a step in a plan for its improvement. These projects could include a 

variety of approaches for reviewing SSOS systems, including strategic reviews with SEA task 

forces, needs sensing interviews with SEA leaders and support agents, surveys of districts and 

schools, development of indicators of system functioning, and a formal CC-developed process for 

evaluating the SSOS. 

Deliver a formal SSOS needs assessment report to the SEA was coded when there was evidence that 

the Center, after conducting a needs assessment of evaluation of the SSOS, provided SEA 

leaders with a formal, state-specific, report including what was learned and implications for 

system redesign. 

Develop state-specific SSOS materials included projects in which the Center assisted SEAs in 

developing their own materials for use in SSOS implementation. These materials included 

protocols for SSOS agents to use in the field, tools for monitoring programs, SSOS training and 

communication resources, tools for improvement planning, and state plans and policy documents 

■	 Train SSOS support providers included projects in which the Center assisted the state in 

training, informing, or coaching school support teams or other SSOS agents on how to carry out 

their functions in supporting low-performing districts and schools. These projects may have 

provided training on improvement strategies and models endorsed by the state, as well as on 

how providers might best deliver these strategies and models. For instance, trainings may review 

templates and guidance documents for local improvement planning and implementation. 

Trainings may have also delineated the responsibilities and procedures of support providers and 

how various SSOS components fit together as a system. 

Train Coders 

For each data collection and reporting cycle, all coders were senior members of the 

evaluation team and were trained to use the established code sets. As part of the training, coders 

discussed each of the codes in the code set to ensure all coders had a shared understanding of 

their meaning. They also coded sample items and reviewed, coded, and discussed the final codes 

using representative sample text. The evaluation team used the discussions in the coder training 

to verify that the established codes were clear and well understood. 

C-6 
Code Center Management Plans, PIFs, and Cover Sheets 

All of the objectives in the Center management plans, the PIFs, and cover sheets were 

independently coded by two coders. For all three years, to code the Center management plans, 

each coder reviewed the objectives section of the plan and determined the topic(s) in which the 

Center indicated they planned to work. For the PIFs in 2006-07, each coder reviewed the project 

description and assigned the project subject code for each of the projects (364 projects total). For 

all three years, in order to code the cover sheets, each coder categorized the project information 

submitted by Centers (i.e., project purpose, customer/client, project activities and deliverables, 

roles and contributions of Centers, research basis for products and services) and to assign codes 

according to the coding criteria described earlier (see exhibit C.1 starting on page C-4) for 

identifying the various types of project activities and resources and the ways Centers contributed 

to projects. For the 2008-09 program year, the evaluation team executed a second coding effort 

of the project cover sheets for the 33 projects classified in the topic area state systems of support 

in the general areas of Center activities on SSOS projects to more closely define the purpose of 

this subset of projects (see exhibit C.2 on page C-6). 

Resolve Coding Disagreements 

Members of the evaluation team compared the coding worksheets from each coder to 

determine the final coded responses for each item. In cases where the coders agreed, the final 

response was clear. In cases where the first and second coders did not arrive at the same code, 

they met to review evidence for the codes and choose a final code. If the first and second coder 

were unable to reconcile their codes, a third senior study team member worked with the coders to 

determine the final code for that item. Each year, the product of this step was the final codes for 

the topic areas included in the Center management plan objectives, the topic areas for the all 

projects on the PIFs, and the three areas described above for the project cover sheets. 

Analysis of Intercoder Reliability 

The evaluation team calculated the percent agreement between the first and second coder 

as well as Cohen’s Kappa analysis. The percent agreement calculation used the number of 

agreements divided by total number of possible responses. The Cohen’s Kappa calculation used 

the number observed to agree minus the number expected to agree by chance divided by the 

number of items minus the number expected to agree. The exhibit below shows the results of 

these analyses. 

C-7 
Exhibit C.3. Summary of intercoder reliability for coding, by year 

Coding effort 2006-07 2007-08 2008-09 

Center management plans 

Possible topic area assignments 462 462 462 

Number of agreements 452 433 449 

Cohen’s Kappa .96 .91 .93 

 

Project inventory forms (PIFs) 

Possible topic area assignments 364
 NA NA 

Number of agreements 318
 

Cohen’s Kappa .92 

 

Project cover sheets 

Possible topic area assignments 1,830 1,830 1,770 

Number of agreements 1,590 1,473 1,451 

Cohen’s Kappa .89 .79 .81 

 

Supplemental SSOS coding for 2008-09 

Possible topic area assignments NA NA 165 

Number of agreements 136 

Cohen’s Kappa .84 

NOTE: The evaluation team only coded the topics for the project inventory forms for the 

2006-07 projects. For the subsequent data collection cycles, the Centers submitted their 

PIFs with the projects already coded by topic. The additional coding effort for the SSOS 

projects was done only for the 2008-09 projects. 

EXHIBIT READS: Across all three data collection cycles, there were a total of 462 

possible topic area assignments. In 2006-07, the first and second coder agreed in 452 of 

those cases with a Cohen’s Kappa of .96. 

Center Interviews 

The evaluation team conducted three rounds of interviews with the Center Director and 

other senior Center staff to gather information about the Center operations. The first round of 

interviews covered the 2006-07 program year and took place in summer 2007. These interviews 

were based on open-ended questions posed to RCC and CC staff about Center organization, 

major areas of focus, communication with client organizations, sources of knowledge used, 

approaches taken in quality assurance, modes of delivering technical assistance, and barriers they 

encountered. The second round of interviews took place in summer 2008 and used a protocol that 

included closed-ended, binary questions to follow up specifically about activities that had 

occurred in 2006-07, topics that had been addressed, and work requests that had fallen outside 

their planned scope of work. Additional questions targeted the strategies that Centers had used 

for planning work and engaging clients, the sources for content expertise, and sources they used 

for vetting their products and services. For the third round of interviews, the team asked the 

C-8 
Centers to provide information on Center operations for 2008-09 on their strategies and 

processes for planning and negotiating work with clients, the types of activities most frequently 

requested by their clients, quality assurance, coordination with other Centers, and the evolution 

of their work from 2006 through 2009. The protocol used to interview CC staff for the third 

round of data collection starts on page C-10 of this appendix, the protocol used to interview RCC 

staff starts on page C-15. 

C-9 
2008-09 CC Interview Protocol 

Content Center Interview Protocol, 2008-09 

For these questions, please refer to the Project Inventory Form for 2008-09 as necessary to 

refresh your memory on the full scope of work delivered by your Center in that year. 

Planning and Negotiating Work 

1.	 Which of the following were part of the process of planning your work for the 2008-09 year: 

a.	 Assessment of state needs 

i.	 Through formal needs assessment, such as a survey 

ii.	 Through ongoing interaction with states 

iii. Through communication with RCCs about state needs 

b.	 Assessment of RCC needs 

i.	 Through formal needs assessment, such as a survey 

ii.	 Through ongoing interaction with RCCs 

c.	 Communication with RCC directors 

d.	 Receiving requests from ED 

e.	 Other source(s) of input, please describe: 

2.	 What were your top strategies for building or maintaining working relationships with RCCs 

and/or with state(s) for the 2008-09 year, so that your Center could meet its technical 

assistance objectives? Please describe specifically how you carried out these strategies. 

3.	 Were there instances of your Center receiving a technical-assistance request for 2008-09 

from an RCC or a state that the Center did not carry out? Please describe the types of 

request(s). If there were multiple instances, describe one that was most typical of your 

experience. 

4.	 For 2008-09, did the reasons for not carrying out technical-assistance requests ever include 

the following: 

a.	 A request fell outside the legitimate scope of work for a Content Center 

b.	 A request was potentially legitimate Content Center work but did not fit this Center’s 

priorities for work with the client(s) 

c.	 The Center did not have access to the needed expertise to carry out the request 

d.	 Staff time and resources were already fully committed to other work 

e.	 The request became moot because the Center and the client instead agreed on a plan for 

related, but different, technical assistance 

f.	 Other reason(s), please describe 

5.	 Of the reasons you’ve just identified, which was the one most frequent reason requests were 

not carried out? 

6.	 Were there instances of your Center proposing technical-assistance activities for 2008-09 that 

RCCs, states, or ED declined to accept? Please describe the proposed technical assistance 

C-10 
2008-09 CC Interview Protocol 

and why it was not carried out. If there were multiple instances, describe one that was most 

typical of your experience. 

7.	 Were any of the following a barrier to delivering technical assistance that met your Center’s 

objectives for work with RCCs during 2008-09? 

a.	 The process of negotiating a work scope and organizing projects took too long 

b.	 RCCs’ most important priorities for assistance fell outside the Center’s scope of work 

c.	 RCCs secured most of the technical assistance they needed from other sources 

d.	 RCCs would have preferred to locate and contract directly with experts or consultants 

rather than working with the Content Center 

e.	 A policy or priority shift at the state level caused the Center’s assistance to an RCC to be 

less helpful than it might 

f.	 Center staff were not able to spend as much time working with an RCC as the RCC 

would have liked 

g.	 The Center did not have the expertise an RCC needed 

h.	 The Center was unable to develop a productive working relationship with an RCC 

i.	 An RCC experienced turnover in leadership 

j.	 An RCC experienced turnover in staffing 

k.	 RCCs placed a higher priority on completing short-term work than on addressing longterm purposes 

l.	 RCC staff did not have time to work with the Center 

m. There was a lack of coordination or communication within an RCC 

n.	 Other(s), specify: 

8.	 Now, using the same list including any other items that you added, please identify the one 

barrier in working with RCCs that most impeded the achievement of your Center’s 

objectives, and describe an example. 

C-11 
 

2008-09 CC Interview Protocol 

9.	 Were any of the following a barrier to delivering technical assistance that met your Center’s 

objectives for work with states during 2008-09? 

a.	 The process of negotiating a work scope and organizing projects took too long 

b.	 States’ most important priorities for assistance fell outside the Center’s scope of work 

c.	 States secured most of the technical assistance they needed from other sources 

d.	 States would have preferred to locate and contract directly with experts or consultants 

rather than working with the Content Center 

e.	 A policy or priority shift at the state level caused the Center’s assistance to a state to be 

less helpful than it might 

f.	 Center staff were not able to spend as much time working with a state as the state would 

have liked 

g.	 The Center did not have the expertise a state needed 

h.	 The Center was unable to develop a productive working relationship with a state 

i.	 A state agency experienced turnover in leadership 

j.	 A state office, division, or intermediary unit experienced turnover in staffing 

k.	 States placed a higher priority on completing short-term work than on addressing longterm purposes 

l.	 State staff did not have time to work with the Center 

m. There was a lack of coordination or communication within a state 

n.	 Other(s), specify: 

10. Now, using the same list including any other items that you added, please identify the one 

barrier in working with states that most impeded the achievement of your Center’s 

objectives, and describe an example. 

C-12 
2008-09 CC Interview Protocol 

The Center’s Work During 2008-09 

11. First, I’d like to know about the types of activities and products that your clients asked for 

and that your Center provided during the program year from July 2008 through June 2009. 

Looking at the following types of activities and products, please identify the one type that 

was most requested by RCCs and states during that year. Which one type represented the 

largest investment of Center resources? Which one was most important for achieving your 

Center’s objectives? Were there changes between 2006-07 and 2008-09 in which type was 

most requested, or your largest investment, or the most important type? [If yes,] Please 

describe. 

a.	 Engagement of participants in project planning (more than needs assessment or 

identifying participants)
 

b.	 Research collections and syntheses 

c.	 Conferences (also called symposiums, forums, institutes; highlight a range of
 

perspectives, strategies, or programs) 

d.	 Training events (focused on implementing a specific program or strategy) 

e.	 Task force meetings and work (focused on addressing a specific problem, program or 

policy) 

f.	 Support for development of a formal state plan to implement a program or policy 

g.	 Ongoing consultation and follow-up (multiple contacts with the same participants, that 

were part of a coherent and purposeful whole) 

12. Now I’d like to know about continuity and change in your technical assistance activities over 

the years. Considering all of your Center’s work in 2008-09, which one of the following 

descriptions would characterize the largest share of the total investment of Center resources? 

Please describe an example of this type of work. Was there a change between 2006-07 and 

2008-09 in which type was the largest investment? [If yes,] Please describe. 

a.	 Multi-year project(s) that followed a long-term plan for participants, activities, and 

purposes 

b.	 Multi-year project(s) with substantial shifts from the originally planned participants, 

activities, or purposes 

c.	 Projects that were self-contained within a single year rather than extending across years 

Quality Assurance 

13. For quality assurance in technical assistance during 2008-09, did your Center have a formal 

process? [If yes,] Please describe that process. 

C-13 
2008-09 CC Interview Protocol 

Coordination with Regional Centers 

14. In which of the following ways did your Center work with at least some of the Regional 

Centers during 2008-09: 

a. Maintained communication with liaisons at least monthly 

b. Provided knowledge resources (materials or experts) as requested 

c. Teamed up to work with one or more states in a region 

d. Sought information from Regional Centers about promising practices in their regions 

15. I would like know what strengths and weaknesses you observed in the two-tier system of 

technical assistance from Regional Centers and Content Centers as of the 2008-09 program 

year. Below are possible areas of strength or weakness, and others may come to mind. Please 

identify any of these that were areas of strength in the two-tier Comprehensive Center 

network. Were any of them areas of weakness of the network? Please give examples that 

illustrate strengths and weaknesses. 

a. Clarity in the role of Regional Centers 

b. Clarity in the role of Content Centers 

c. Coordination across Regional and Content Centers 

d. Other area(s) of strength or weakness, please describe: 

Evolution of the Center’s Work 

16. Looking back, what would you describe as the most significant ways in which your Center’s 

work evolved from 2006 through 2009? 

C-14 
2008-09 RCC Interview Protocol 

RCC Interview Protocol, 2008-09 

For these questions, please refer to the Project Inventory Form for 2008-09 as necessary to 

refresh your memory on the full scope of work delivered by your Center in that year. 

Planning and Negotiating Work with State Agencies 

17. Did your Center have one or more staff positions designated for serving as the Center’s 

liaison to specific SEAs for the 2008-09 year? If so, did any of them maintain office space 

within an SEA building? 

18. Which of the following were part of the process of planning your work for the 2008-09 year: 

a.	 Formal needs assessment, such as a survey of client staff 

b.	 Informal needs assessment through communication with client staff 

c.	 Communication with chief state school officer(s) 

d.	 Other source(s) of input, please describe: 

19. What were your top strategies for building or maintaining working relationships with state(s) 

in your region for the 2008-09 year, so that your Center could achieve its technical assistance 

objectives? Please describe specifically how you carried out these strategies. 

20. Were there instances of an SEA making a technical assistance request for 2008-09 that the 

Center did not carry out? Please describe the requests that were not carried out, state by state. 

What types of requests were most common, if any? 

21. For 2008-09, did the Center’s reasons for not carrying out SEA technical assistance requests 

ever include the following: 

a.	 A request fell outside the legitimate scope of work for a Comprehensive Center 

b.	 A request was potentially legitimate Comprehensive Center work but did not fit this 

Center’s priorities for work with the state 

c.	 The Center did not have access to the needed expertise to carry out the request 

d.	 Staff time and resources were already fully committed to other work 

e.	 An SEA request became moot because the Center and SEA instead agreed on a plan for 

related, but different, technical assistance 

f.	 Other reason(s), please describe: 

22. Of the reasons you’ve just identified, which was the one most frequent reason SEA requests 

were not carried out? 

23. Were there instances of the Center proposing technical assistance activities for 2008-09 that 

an SEA declined to accept? Please describe the proposed technical assistance and why it was 

not carried out, state by state. 

24. Were any of the following a barrier to delivering technical assistance that met your Center’s 

objectives during 2008-09? 

C-15 
2008-09 RCC Interview Protocol 

a.	 The process of negotiating a work scope and organizing projects took too long 

b.	 A state’s most important priorities for assistance fell outside the Center’s scope of work 

c.	 A state secured most of the technical assistance it needed from other sources 

d.	 A state would have preferred to locate and contract directly with experts or consultants 

rather than working with the Comprehensive Centers 

e.	 A policy or priority shift at the state level caused the Center’s assistance to be less helpful 

than it might 

f.	 Center staff were not able to spend as much time working with a state as the state would 

have liked 

g.	 The Center did not have the expertise a state needed 

h.	 The Center was unable to develop a productive working relationship with a state 

i.	 A state experienced turnover in leadership 

j.	 A state office, division, or intermediary unit experienced turnover in staffing 

k.	 State clients placed a higher priority on completing short-term work than on addressing 

long-term purposes 

l.	 State staff did not have time to work with the Center 

m. There was a lack of coordination or communication within a state agency 

n.	 Other barrier(s), please describe: 

25. Now, using the same list including any other items that you added, please identify the one 

barrier that most impeded the achievement of your Center’s objectives, and describe an 

example of this barrier. 

The Center’s Work During 2008-09 

26. First, I’d like to know about the types of activities and products that your clients asked for 

and that your Center provided during the program year from July 2008 through June 2009. 

Looking at the following types of activities and products, please identify the one type that 

was most requested by clients during that year. Which one type represented the largest 

investment of Center resources? Which one was most important for achieving your Center’s 

objectives? Were there changes between 2006-07 and 2008-09 in which type was most 

requested, or your largest investment, or the most important type? [If yes,] Please describe. 

a.	 Engagement of participants in project planning (more than needs assessment or 

identifying participants)
 

b.	 Research collections and syntheses 

c.	 Conferences (also called symposiums, forums, institutes; highlight a range of
 

perspectives, strategies, or programs) 

d.	 Training events (focused on implementing a specific program or strategy) 

e.	 Task force meetings and work (focused on addressing a specific problem, program or 

policy) 

f.	 Support for development of a formal state plan to implement a program or policy 

g.	 Ongoing consultation and follow-up (multiple contacts with the same participants, that 

were part of a coherent and purposeful whole) 

27. Now I’d like to know about continuity and change in your technical assistance over the years. 

Considering all of your Center’s work in 2008-09, which one of the following descriptions 

would characterize the largest investment of Center resources? Please describe an example of 

C-16 
2008-09 RCC Interview Protocol 

this type of work. Was there a change between 2006-07 and 2008-09 in which type was the 

largest investment? [If yes,] Please describe. 

a.	 Multi-year project(s) that followed a long-term plan for participants, activities, and 

purposes 

b.	 Multi-year project(s) with substantial shifts from the originally planned participants, 

activities, or purposes 

c.	 Projects that were self-contained within a single year rather than extending across years 

Quality Assurance 

28. For quality assurance in technical assistance during 2008-09, did your Center have a formal 

process? [If yes,] Please describe that process. 

Coordination with Content Centers 

29. With which of the Content Centers did your Center do each of the following during 2008-09: 

a.	 Maintained communication at least monthly through a liaison 

i.	 AACC 

ii.	 CII 

iii. COI 

iv. NCCTQ 

v.	 NHSC 

b.	 Received knowledge resources (materials or experts) that your Center requested 

i.	 AACC 

ii.	 CII 

iii. COI 

iv. NCCTQ 

v.	 NHSC 

c.	 Teamed up to work with one or more states in the region 

i.	 AACC 

ii.	 CII 

iii. COI 

iv. NCCTQ 

v.	 NHSC 

d.	 Informed the Content Center about promising practices in the region 

i.	 AACC 

ii.	 CII 

iii. COI 

iv. NCCTQ 

v.	 NHSC 

C-17 
2008-09 RCC Interview Protocol 

30. Were any of the following a barrier to your Center receiving technical assistance from the 

Content Centers during 2008-09? 

a.	 The process of negotiating a work scope and organizing projects took too long 

b.	 Our most important priorities for assistance fell outside the Content Centers’ scope of 

work 

c.	 We secured most of the technical assistance we needed from other sources 

d.	 We would have preferred to locate and contract directly with experts or consultants rather 

than working with Content Centers 

e.	 A policy or priority shift at the state level caused Content Centers’ assistance to us to be 

less helpful than it might 

f.	 Content Center staff were not able to spend as much time working with us as we would 

have liked 

g.	 Content Centers did not have the expertise we needed 

h.	 Our Center was unable to develop a productive working relationship with a Content 

Center 

i.	 We experienced turnover in leadership 

j.	 We experienced turnover in staffing 

k.	 We placed a higher priority on completing short-term work than on addressing long-term 

purposes 

l.	 Our staff did not have time to work with a Content Center 

m. There was a lack of coordination or communication within our Center 

n.	 Other(s), specify: 

31. Now, using the same list including any other items that you added, please identify the one 

barrier that most impeded your Center receiving technical assistance from Content Centers. 

32. I would like know what strengths and weaknesses you observed in the two-tier system of 

technical assistance from Regional Centers and Content Centers as of the 2008-09 program 

year. Below are possible areas of strength or weakness, and others may come to mind. Please 

identify any of these that were areas of strength in the two-tier Comprehensive Center 

network. Were any of them areas of weakness of the network? Please give examples that 

illustrate strengths and weaknesses. 

a.	 Clarity in the role of Regional Centers 

b.	 Clarity in the role of Content Centers 

c.	 Coordination across Regional and Content Centers 

d.	 Other area(s) of strength or weakness, please describe: 

Evolution of the Center’s Work 

33. Looking back, what would you describe as the most significant ways in which your Center’s 

work evolved from 2006 through 2009? 

C-18 
Appendix D 

Supplemental Tables – Center Operations 

Center Operations – Supplemental Tables 

This appendix serves as a supplement to the information presented in chapter 3 of the 

report and includes tables from the 2006-07 program year that parallel similar 2008-09 tables 

included in the report. 

Exhibit D.1. Topics on which Centers stated objectives and/or delivered 

projects, 2006-07 

Topic 

Objective on 

topic was 

stated in 

management 

plan 

(1) 

Number of Centers 

(N=21) 

Objective on topic was 

stated in management plan 

AND at least one 

project on topic was 

reported on PIF 

(2) 

At least one 

project on 

topic was 

reported on 

PIF 

(3) 

Total cases of a Center setting an 

objective and/or reporting a project 133 107 172 

Components of effective systems of 

support—state, district, school 

English language learners 

Highly qualified teacher provisions of NCLB 

Teacher professional development 

Teacher preparation and induction 

Data use/data-driven decisionmaking 

Assessment design 

Formative assessment 

Special education—curriculum, instruction, 

and professional development 

Response to Intervention (RtI) 

Special education—assessment 

High school redesign/reform 

Transition to high school 

Dropout prevention 

Mathematics 

Adolescent literacy 

Reading 

Supplemental educational services (SES) 

Parent involvement 

Migrant education 

Data management compliance 

Indian/Native American education 

15 

10 

8 

6 

5 

8 

6 

5 

8 

3 

3 

8 

2 

2 

7 

6 

6 

7 

7 

5 

5 

1 

15 

10 

5 

5 

3 

4 

5 

4 

3 

3 

3 

7 

2 

2 

4 

6 

5 

7 

7 

3 

3 

1 

17 

15 

10 

8 

5 

7 

5 

7 

6 

10 

3 

10 

5 

2 

8 

8 

8 

12 

11 

6 

5 

4 

NOTE: Column (1) - column (2) =the number of Centers that deleted work in the topic area as indicated by number of Centers 

that had a topic-related objective in their management plan minus the number of Centers that had the topic area on their PIF 

and their management plan; column (3)- column (2) =the number of Centers that added work in the topic area as indicated by 

number of Centers that reported work on their PIF minus the number of Centers that had the topic area on their PIF and their 

management plan. 

EXHIBIT READS: For the topic “Components of effective systems of support—state, district, school,” 15 Centers had a related 

objective in their respective annual management plans; of these, all 15 reported projects on the topic in their PIF; and a total of 

17 Centers had projects on the topic, whether or not they had stated an objective related to it in their management plan. 

SOURCES: Center management plans for 2006-07; PIFs prepared by Centers in consultation with evaluation team. 

D-1 
Exhibit D.2. Topics on which RCCs and CCs stated objectives and/or delivered 

projects, 2006-07 

Number of RCCs (N=16) Number of CCs (N=5) 

Objective Objective on 

on topic in topic in plan AND 

manage- project(s) on 

ment plan topic on PIF 

(1) (2) 

Project(s) 

on topic on 

PIF 

(3) 

Objective 

on topic in 

management plan 

(4) 

Objective on 

topic in plan AND 

project(s) on 

topic on PIF 

(5) 

Project(s) 

on topic on 

PIF 

(6) 

Total cases 112 89 146 21 18 26 

Components of effective 

systems of support—state, 

district, school 

English language learners 

Highly qualified teacher 

provisions of NCLB 

Teacher professional 

development 

Teacher preparation and 

induction 

Data use/data-driven 

decisionmaking 

Assessment design 

Formative assessment 

Special education – 

curriculum, instruction, 

professional development 

Response to Intervention 

(RtI) 

Special education– 

assessment 

High school 

redesign/reform 

Transition to high school 

Dropout prevention 

Mathematics 

Adolescent literacy 

Reading

Supplemental educational 

services (SES) 

Parent involvement 

Migrant education 

Data management 

compliance 

Indian/Native American 

education 

14 

8 

7 

4 

4 

7 

5 

4 

6 

3 

2 

7 

1 

1 

6 

5 

5 4 7 1 1 1 

6 

6 

5 

5 

1 

14 

8 

4 

4 

2 

3 

4 

3 

3 

3 

2 

6 

1 

1 

3 

5 

6 

6 

3 

3 5 0 0 0 

1 

15 

12 

9 

7 2 1 1 

4 

5 1 1 

4 

5 

5 

8 0 0 

2 1 1 

9 1 1 

4 

1 

7 

6 

11 1 1 

10 

6 

4 

1 

2 

1 1 

1 1 

1 

1 

2 0 

1 

1 

1 

1 

1 

0 

0 

1 

2 

1 

1 

1 

1 

1 

1 

1 

0 

0 

2 

3 

1 

1 

2 

1 

2 

1 

2 

1 

1 

1 

1 

1 

2 

1 

1 

0 

0 

NOTE: column (1)-column (2) =the number of RCCs that deleted work in the topic area as indicated by number of Centers that 

reported work on their management plan minus the number of Centers that had the topic area on their PIF and their management plan 

; column (3)- column (2) =the number of RCCs that added work in the topic area as indicated by number of Centers that reported work 

on their PIF minus the number of Centers that had the topic area on their PIF and their management plan. These calculations are 

analogous for the CCs. 

EXHIBIT READS: For the topic “Components of Effective Systems of Support—State, District, School,” 14 RCCs reported a related 

objective in their management plans; of these, all 14 reported projects on the topic; and a total of 15 RCCs had projects on the topic, 

whether or not they had originally stated an objective related to it. 

SOURCES: Center management plans for 2006-07; PIFs prepared by Centers in consultation with evaluation team. 

D-2 
Appendix E 

Survey of State Managers and Case Study Visits to 10 States 

Survey of State Managers and Case Study Visits to 10 States 

Survey of State Managers 

This appendix describes the survey that was administered to senior state managers 

serving as RCC’s main points of contacts. The survey was developed in the first year of data 

collection to obtain information from state managers about state-level needs and priorities for 

technical assistance, state perspectives of Center technical assistance, and comparative judgment 

of Center assistance in relation to assistance available through other sources. 

After reviewing the responses to the 2006-07 administration of the state manager survey, 

the evaluation team made several revisions to the survey for its administration in 2007-08 and 

2008-09. Exhibit E.1 lists each question that was revised, showing both the 2006-07 question and 

the revised version administered in 2007-08 and 2008-09. Questions that remained unchanged 

from the first year of survey administration are not included in the exhibit. The version of the 

survey used in 2006-07 is included in appendix D of the interim report for this study. 

For several of the survey items, the list of NCLB responsibility areas was revised 

following the 2006-07 survey administration cycle. Exhibit E.2 provides a crosswalk between the 

original list of NCLB responsibility areas used for the 2006-07 administration and the similar list 

used for subsequent survey administration years. 

Case Study Visits to 10 States 

Following the 2007-08 administration of the state manager surveys, ten states were 

selected for site visit interviews to gather information about Center technical assistance provided 

on statewide systems of support (SSOS). The states were selected to maximize the number of 

Centers included (a total of 10 RCCs and 3 CCs). Given that SSOS was reported to be the most 

widespread state priority of NCLB and a major focus of Center assistance in each year, the 

selected 10 states represented the full range of state-reported experience in this topic area. Of the 

10 states selected, state managers reported on their 2007-08 surveys that Center assistance 

expanded state capacity in SSOS “to a great extent” in 5 states, “to a moderate extent” in 1 state, 

“to a small extent” in 3 states, and “does not apply or unable to judge” in 1 state. For the site 

visits, interviews were conducted in each state with SEA administrators (e.g., commissioner, 

deputy commissioner, director) and staff involved in projects focused on SSOS (e.g., senior state 

managers), as well as with RCC leadership and staff (e.g., center director, state liaison, staff 

responsible for SSOS projects). In states that had CC involvement in SSOS projects, the CC 

director and staff working with the RCC were also interviewed. The interviews gathered 

information on the state capacity building support provided by Centers (i.e., Center approaches 

to capacity building, activities and resources for capacity building, evidence of SEA capacity 

built). The code sets used to code the interview data are presented in Exhibit E.3 (page E-10). 

Coding was conducted in pairs with 100 percent consensus established on all coding. 

E-1 
Exhibit E.1. Revisions to State Manager Survey from 2006-07 to 2007-08 

E-2
 

Question 2006-07 administration 2007-08/2008-09 administra

Priorities for technical assistance: When your state requests technical assistance from outside sources from July XXXX to June XXXX, what are its priorities? To what extent is each of the following state responsibilities related to NCLB implementation a priority for the technical assistance the state requests? 

Reporting categories: 

• Major priority • Moderate priority • Minor Priority • Not at all a priority 

a. Formulating or refining state policies to 

respond to NCLB requirements 

b. Building or managing a statewide system of 

support for districts and schools identified for improvement under NCLB 

c. Training or managing the state-level staff or 

school support teams who provide support to districts and schools identified for improvement under NCLB 

d. Designing or implementing state 

assessment and accountability systems 

e. Aligning state accountability systems with 

NCLB accountability systems

f. Supporting use of assessment data by 

schools and districts 

g. Disseminating information on scientificallybased research to districts and schools h. Identifying and/or developing programs or 

models that address district and/or school needs 

i. Providing training and other professional

development to local educators in academic subjects (reading language arts, mathematics, science) 

j. Monitoring compliance with NCLB 

requirements in districts and schools 

k. Communicating with the public about NCLB 

requirements or report cards 

l. Other (Specify: ________________) 

Reporting categories: 

• Major priority • Moderate priority • Minor Priority • Not at all a priority 

a. Statewide systems of 

support teams 

b. State assessment an

systems 

c. Assistance with educ

assessment data 

d. Development or disse

based curriculum, ins

e. or professional develo

academic subject(s) 

f. Policies and practices 

learners 

g. Administering supplem

services (SES) and c

h. Communication with p

i. Monitoring complianc

requirements 

j. State planning or repo

NCLB not covered ab

k. Other (Specify _____
Question 2006-07 administration 2007-08/2008-09 administra

Sources of technical assistance: Reporting categories: Reporting categories: During the XXXX-XX school year (beginning July XXXX and ending June XXXX), to what extent has your state relied on each of the following sources of technical assistance? (Select one response in each row: ) 

• One of the state’s most important resources • To a great extent but not one of the state’s 

most important resources 

• To a moderate extent • Minimally • No contact. 

a. U.S. Department of Education (Specify 

office:_________________________) 

b. Professional associations (e.g., CCSSO, 

ASCD) 

c. Colleges and universities d. Consulting firms or private contractors e. Your counterparts in other SEAs f. Comprehensive Center network g. Regional Educational Laboratory h. Other federally funded technical assistance 

providers (Specify:________________________) 

i. Other (Specify ______________) 

• To a great extent, • To a moderate extent• To a small extent, and • No contact 

a. Comprehensive Centeb. U.S. Department of E

Office:____________

c. Professional associati

ASCD)

d. Colleges and universie. Consulting firms or prf. Your counterparts in og. Regional Educational h. Other federally funded 

providers (Specify:___________

E-3 

i. Other (Specify ______
Question 2006-07 administration 2007-08/2008-09 administra

Technical assistance received by area of state NCLB responsibility 

Reporting categories: 

• Major assistance 

• Mode asrate sistance 

• Minimal ssi a stance 

• No assistance at all • NA, technical assistance not sought by the 

SEA 

• NA, this task is not relevant to my unit’s 

work 

■

a. Formulating or refining state policies to respond 

to NCLB requirements 

b. Building or managing a statewide system of 

support for districts and schools identified for improvement under NCLB 

c. Training or managing the state-level staff or 

school support teams who provide support to districts and schools identified for improvement under NCLB 

d. Designing or implementing state assessment 

and accountability systems 

e. Aligning state accountability systems with NCLB 

accountability systems 

f. Supporting use of assessment data by schools 

and districts 

g. Disseminating information on scientificallybased research to districts and schools h. Identifying and/or developing programs or 

models that address district and/or school needs 

i. Providing training and other professional

development to local educators in academic subjects (reading language arts, mathematics, science) 

j. Monitoring compliance with NCLB requirements 

in districts and schools 

k. Communicating with the public about NCLB 

requirements or report cards 

l. Other (Specify: ________________) 

Reporting categories: 

• Yes • No 

• I don’t know • NA, technical assistan

SEA 

a. Statewide systems of sup

teams 

b. State assessment and accc. Assistance with educators

data 

d. Development or dissemin

based curriculum, instruct

e. or professional developme

academic subject(s) 

f. Policies and practices for 

learners 

g. Administering supplement

services (SES) and choice 

h. Communication with parei. Monitoring compliance witj. State planning or reporting 

NCLB not covered above 

k. Other (Specify _________

During the XXXX-XX s

chool year 

(beginning July XXXX and ending June XXXX), with which areas of state responsibility related to NCLB implementation did your state receive assistance from the Comprehensive Center network (your regional center and any content centers with whom your state has worked)? 

E-4
 
Question 2006-07 administration 2007-08/2008-09 administra

Relevance of technical assistance received Please consider all of the technical assistance that your state has received from the Comprehensive Center network from July XXXX through June XXXX. To what degree were the activities and resources relevant to your state, in each of the following respects? (Circle one response in each row.) 

Reporting categories: 

• To a very high degree • To a high degree • To a moderate degree • To a low degree • To a very low degree • Not able to judge 

a. Addressed a need or problem that the state 

faces 

b. Addressed an important state priority c. Addressed a challenge that the state faces 

related to the implementation of NCLB 

d. Responded flexibly to our state’s changing 

needs 

e. Provided information, advice, and/or resources 

that could be applied to the state’s work 

f. Addressed the particular context in which our 

state operates 

g. Addressed the state’s specific challenges (e.g., 

policy environment, leadership capacity, budget pressures, local politics) 

h. Provided information, advice, and/or resources 

that could be used to guide decisions about policies, programs, and practices 

i. Highlighted the implications of research findings 

(or information about best practice) for policies, pro

grams, or practices 

Reporting categories: 

• To a very high degree • To a high degree • To a moderate degree • To a low degree • To a very low degree • Not able to judge 

■

a. Addressed a need or prob

faces 

b. Addressed an important sc. Addressed a challenge th

related to the implementat

d. Addressed the state’s spe

policy environment, leadepressures, local politics) 

e. Provided information, adv

that could be used to guidpolicies, programs, and pr

E-5
 
Question 2006-07 administration 2007-08/2008-09 administra

Usefulness of technical assistance received 

Please consider all of the 

technical assistance that your state has received from the Comprehensive Center network from July XXXX through June 

XXXX. Considering just this set of products and services, to what degree were the activities and resources useful to your state, in 

each of the following respects? (Circle one response in each row.) 

Reporting categories: 

• To a very high degree 

• To a high degree • To a moderate degree • To a low degree 

• To a very low degree 

• Not able to judge 

a. Provided state staff with resources that were 

easy to understand and easy to use 

b. Employed appropriate formats (e.g., work 

groups, conferences, individual consultation, written products) 

c. Provided adequate opportunity to learn from 

colleagues in other states 

d. Included adequate follow-up to support the use 

of new information and resources 

e. Was timely f. Helped the state to solve a problem g. Helped the state to maintain or change a policy 

or practice 

h. Helped the state take the next step in a longerterm improvement effort i. Provided state staff with information or 

resources that they will use again 

j. Helped state staff to develop skills that they will 

be able to exercise a

gain 

Reporting categories: 

• To a very high degree 

• To a high degree • To a moderate degree • To a low degree 

• To a very low degree • Not able to judge 

a. Was timely b. Helped the state to solve c. Helped the state to mainta

or practice 

d. Provided state staff with in

resources that they will us

e. Helped state staff to deve

be able to exercise again 

E-6
 
Question 2006-07 administration 2007-08/2008-09 administra

Center technical assistance received compared to other 

sources 

Compared with the technical 

assistance your state has received from other sources, (see list of sources in question 2) how would you rate the usefulness of the technical assistance your state received during from July XXXX through 

June XXXX from the Comprehensive Center network for each of the following purposes? (Circle one response in each row.) 

Reporting categories: 

• Much more useful than assistance from 

other resources 

• Somewhat more useful 

• About the same 

• Somewhat less useful • Much less useful than assistance from other 

sources 

• Not able to judge • NA, the state has not sought assistance for 

this purpose 

a. Formulating or refining state policies to respond 

to NCLB requirements 

b. Building or managing a statewide system of 

support for districts and schools identified for improvement under NCLB 

c. Training or managing the state-level staff or 

school support teams who provide support to districts and schools identified for improvement under NCLB 

d. Designing or implementing state assessment 

and accountability systems 

e. Aligning state accountability systems with NCLB 

accountability systems 

f. Supporting use of assessment data by schools 

and districts 

g. Disseminating information on scientificallybased research to districts and schools h. Identifying and/or developing programs or 

models that address district and/or school needs 

i. Providing training and other professional

development to local educators in academic subjects (reading language arts, mathematics, science) 

j. Monitoring compliance with NCLB requirements 

in districts and schools 

k. Communicating with the public about NCLB 

requirements or report cards 

l. Other (Specify: ________________) 

Reporting categories: 

• More useful than assi

resources 

• About the same 

• Less useful than assis

sources 

• Not able to judge • NA, the state has not 

this purpose 

a. Statewide systems of sup

teams 

b. State assessment and accc. Assistance with educators

data 

d. Development or dissemin

based curriculum, instruct

e. or professional developme

academic subject(s) 

f. Policies and practices for 

learners 

g. Administering supplement

services (SES) and choice 

h. Communication with pare

i. Monitoring compliance witj. State planning or reporting 

NCLB not covered above 

k. Other (Specify _________

E-7
 
Question 2006-07 administration 2007-08/2008-09 administra

State capacity to carry out NCLB responsibilities From July XXXX through June XXXX, to what extent has 

assistance from the Comprehensive Center network (your regional center and any of the five content centers)? expanded your state’s capacity to carry out state responsibilities related to NCLB? (Circle one response in each row.) 

Reporting categories: 

• To a very great extent • To a great extent • To a moderate extent • To a small extent 

• To a very small extent 

• Too soon to tell 

• NA, state has not sought assistance for this 

purpose 

a. Formulating or refining state policies to respond 

to NCLB requirements 

b. Building or managing a statewide system of 

support for districts and schools identified for improvement under NCLB 

c. Training or managing the state-level staff or 

school support teams who provide support to districts and schools identified for improvement under NCLB 

d. Designing or implementing state assessment 

and accountability systems 

e. Aligning state accountability systems with NCLB 

accountability systems 

f. Supporting use of assessment data by schools 

and districts 

g. Disseminating information on scientificallybased research to districts and schools h. Identifying and/or developing programs or 

models that address district and/or school needs 

i. Providing training and other professional

development to local educators in academic subjects (reading language arts, mathematics, science) 

j. Monitoring compliance with NCLB requirements 

in districts and schools 

k. Communicating with the public about NCLB 

requirements or report cards 

l. Other (Specify: ________________) 

Reporting categories: 

• To a great extent 

• To a moderate extent • To a small extent • Not at all • Does not apply or not 

a. Statewide systems of sup

teams 

b. State assessment and accc. Assistance with educators

data 

d. Development or dissemin

based curriculum, instruct

e. or professional developme

academic subject(s) 

f. Policies and practices for 

learners 

g. Administering supplement

services (SES) and choice 

h. Communication with parei. Monitoring compliance witj. State planning or reporting 

NCLB not covered above 

k. Other (Specify _________

E-8
 
Used in 2006-07 Summary of change Used in 2007-08 an

Building or managing a statewide system of support for districts and schools identified for improvement under NCLB 

Collapsed two tasks that were not mutually exclusive 

a. Statewide syste

support teams Training or managing the state-level staff or school support teams who provide support to 

districts and schools identified for improvement under NCLB Designing or implementing state assessment and accountability systems 

Collapsed two tasks that were not mutually exclusive b. State assessme

Aligning state accountability systems with NCLB systems accountability systems Supporting use of assessment data by schools and districts 

c. Assistance with 

assessment da

Simplified text

Disseminating information on scientifically-based research to districts and schools 

Collapsed three tasks that were not mutually exclusive 

Identifying and/or developing programs or models that address district and/or school needs 

d. Development o

based curriculuprofessional deacademic subje

Providing training and other professional development to local educators in academic 

subjects (reading language arts, mathematics, science)

Added to reflect an area of state and Center activity 

e. Policies and pra

learners 

Added to reflect an area of state and Center activity 

f. Administering s

services (SES) 

Communicating with the public about NCLB 

requirements or report cards Simplified text g. Communication 

Monitoring compliance with NCLB requirements in districts and schools 

h. Monitoring com

requirements Simplified text

Formulating or refining state policies to respond to NCLB requirements 

Reworded to eliminate overlap with other 

responsibilities 

i. State planning o

NCLB not cove

Other (Specify: ________________) No change j. Other (Specify _

Exhibit E.2. Crosswalk between NCLB state responsibilities used in the 2006-07 and 2007-

and participant surveys 

E-9
 
Exhibit E.3. Codes used for the interviews conducted on state capacity 

building 

1. RCC and CCC Approaches to Capacity Building with SEAs 

• Approaches/philosophies for capacity building with SEAs 

•	 Approaches described as most useful for building capacity with SEAs 

2. Activities and Resources for SEA Capacity Building 

•	 Activities and resources provided to SEAs by RCCs or CCCs 

•	 Activities and resources provided by RCCs or CCs described as most useful for 

building capacity with SEAs 

3. Evidence of SEA Capacity Built by RCCs and CCCs in 3 Dimensions 

•	 Systems, infrastructure and technology [including indicators/measurements of 

SEA capacity] 

•	 Staff expertise [including indicators/measurements of SEA capacity] 

•	 Access to expertise [including indicators/measurements of SEA capacity] 

4. SEA Long-term Capacity 

• Evidence/discussions explicitly stating long-term SEA capacity was built by RCCs 

and CCCs 

• Evidence/discussions explicitly stating long-term SEA capacity was not built by 

RCCs and CCCs 

E-10 
Appendix F 

Supplemental Tables – State Manager Survey Results 

Survey of State Managers Results – Supplemental Tables 

This appendix serves as a supplement to the information presented in chapter 4 of the 

report. It includes survey tabulations from the 2006-07 and 2007-08 administrations of the state 

manager survey as well as more detailed versions of some exhibits provided in the report for all 

three years. 

Exhibit F.1. State priorities for technical assistance and assistance 

received from Centers, 2006-07 

Among state managers reporting 

the responsibility as a major or 

moderate priority, the percent 

reporting they received: 

Major or 

moderate 

assistance from 

the Centers 

related to the 

responsibility 

Any assistance 

from the Centers 

related to the 

Priority area of state responsibility under NCLB responsibility 

Building or managing a statewide system of support for 

districts and schools identified for improvement under NCLB 

(n=46) 

98% 80% 

Training or managing the state-level staff or school support 

teams who provide support to districts and schools identified 

for improvement under NCLB (n=41) 

90 78 

Identifying and/or developing programs or models that 

address district and/or school needs (n=38) 95 79

Disseminating information on scientifically based research to 

districts and schools (n=36) 97 74

Supporting use of assessment data by schools and districts 

(n=36) 86 66

Formulating or refining state policies to respond to NCLB 

requirements (n=34) 94 74

Providing training and other professional development to local 

educators in academic subjects (reading language arts, 

mathematics, science) (n=29) 

79 59 

Designing or implementing state assessment and 

accountability systems (n=25) 80 65

Monitoring compliance with NCLB requirements in districts 

and schools (n=24) 71 50

Aligning state accountability systems with NCLB accountability 

systems (n=21) 81 62

Communicating with the public about NCLB requirements or 

report cards (n=16) 69 33

EXHIBIT READS: For the 2006-07 program year, 46 state managers (weighted) reported a major or moderate priority 

for technical assistance with building or managing a statewide system of support. Of those state mangers, 98 

percent reported receiving technical assistance from the Centers related to that responsibility and 80 percent reported 

that the technical assistance was major or moderate in scope. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

F-1 
Exhibit F.2.	 Extent to which Center Assistance expanded state capacity for 

NCLB implementation, as judged by senior state managers, 2006-

07 

Percent of the state managers (weighted) who rated the area 

as a major or moderate technical assistance priority 

Reporting capacity building in this area NA, state has 

not sought 

To a very To a To a small assistance 

Area of state responsibility great or moderate or very Too soon for this 

under NCLB great extent extent small extent to tell purpose 

Building or managing a statewide system 

of support for districts and schools 

identified for improvement under NCLB 

(n=53) 

53% 14% 18% 8% 7%

Training or managing the state-level staff 

or school support teams who provide 

support to districts and schools identified 

for improvement under NCLB (n=46) 

51 20 12 7 10

Identifying and/or developing programs or 

models that address district and/or school 

needs (n=42) 

47 24 8 11 10

Formulating or refining state policies to 

respond to NCLB requirements (n=40) 43 14 14 11 18 

Supporting use of assessment data by 

schools and districts (n=40) 42 16 11 2 29

Disseminating information on scientifically 

based research to districts and schools 

(n=38) 

39 38 10 5 8

Providing training and other professional 

development to local educators in 

academic subjects (reading language arts, 

mathematics, science) (n=38) 

33 17 11 9 30

Designing or implementing state 

assessment and accountability systems 

(n=35) 

33 25 5 4 33 

Aligning state accountability systems with 

NCLB accountability systems (n=28) 34 24 6 9 27 

Monitoring compliance with NCLB 

requirements in districts and schools 

(n=25) 

34 22 21 8 15 

Communicating with the public about 

NCLB requirements or report cards (n=19) 26 23 7 10 34

EXHIBIT READS: For the 2006-07 program year, the 53 state managers (weighted) reported that technical 

assistance in building or managing a statewide system of support was a major or moderate priority for their state. Of 

those state managers, 53 percent reported that technical assistance received from the Centers expanded the state’s 

capacity in this area to a great or very great extent. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

F-2 
Exhibit F.3. Extent to which Center assistance expanded state capacity in 

priority areas, as judged by senior state managers, 2007-08 

Percent of the state managers (weighted) who rated the area 

as a major or moderate technical assistance priority 

Reporting capacity building in this area 

To a 

moderate

extent 

Does not 

apply or not 

able to judge 

To a great 

extent 

To a small 

extent 

Not at 

Area of state responsibility under NCLB all 

Statewide systems of support or school 

support teams (n=56) 45% 27% 13% 7% 9% 

Policies and practices for English 

language learners (n=43) 26 33 23 5 14 

State assessment and accountability 

systems (n=42) 26 31 17 10 14 

Development or dissemination of 

research-based curriculum, instruction, or 

professional development programs in 

academic subject(s) (n=41) 

37 27 20 2 15 

Assistance with educators’ use of 

assessment data (n=37) 24 38 22 5 11 

Monitoring compliance with NCLB 

requirements (n=35) 31 26 9 9 26 

Administering supplemental educational 

services (SES) and choice provisions 

(n=25) 

24 20 16 8 32 

Communication with parents or the public 

(n=25) 20 28 28 0 24 

EXHIBIT READS: For the 2007-08 program year, 56 state managers (weighted) reported that technical assistance in 

building or managing a statewide system of support was a major or moderate priority for their state. Of those, 

45 percent reported that technical assistance received from the Centers expanded the state’s capacity in this area to 

a great extent. 

F-3 
Exhibit F.4. Extent to which Center assistance expanded state capacity in 

priority areas, as judged by senior state managers, 2008-09 

Percent of the state managers (weighted) who rated the area 

as a major or moderate technical assistance priority 

Reporting capacity building in this area 

To a 

moderate 

extent 

Does not 

apply or not 

able to judge 

To a great 

Area of state responsibility under NCLB extent 

To a small 

extent 

Not at 

all 

Statewide systems of support or school 

support teams (n=50) 56% 26% 14% 0% 4% 

Policies and practices for English 

language learners (n=40) 33 40 10 8 10 

State assessment and accountability 

systems (n=39) 31 28 18 3 21 

Development or dissemination of 

research-based curriculum, instruction, or 

professional development programs in 

academic subject(s) (n=39) 

36 41 10 0 13 

Assistance with educators’ use of 

assessment data (n=36) 22 39 17 3 19 

Monitoring compliance with NCLB 

requirements (n=30) 30 27 17 3 23 

Administering supplemental educational 

services (SES) and choice provisions 

(n=26) 

12 27 23 15 23 

Communication with parents or the public 

(n=26) 19 31 19 12 19 

EXHIBIT READS: For the 2008-09 program year, 50 state managers (weighted) reported that technical assistance in 

building or managing a statewide system of support was a major or moderate priority for their state. Of those, 

56 percent reported that technical assistance received from the Centers expanded the state’s capacity in this area to 

a great extent. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

F-4 
Exhibit F.5. States’ use of external sources of technical assistance, 2006-07 

Percent of state managers (weighted) reporting: 

To a great 

extent, but 

not one of the 

 state's most 

important To a moderate 

resources extent 

One of the 

state's most

important 

External source resources 

Professional associations (e.g., 

CCSSO, ASCD) (n=54) 37% 24% 28% 

Minimally No contact 

9% 2%

Comprehensive Center network 

(n=54) 35 37 14 12 2

Regional Educational 

Laboratory (n=53) 33 29 21 12 5

U.S. Department of Education 

(n=52) 29 16 35 19 1

Colleges and universities 

(n=54) 18 15 34 30 3

Consulting firms or private 

contractors (n=54) 13 29 33 19 6

Senior managers in other SEAs 

(n=54) 9 29 37 23 2

EXHIBIT READS: For the 2006-07 program year, 37 percent of state managers (weighted) said that professional 

associations were one of the state’s most important technical-assistance resources. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded.
 

F-5 
Exhibit F.6. Purposes for which states used external sources of technical 

assistance, 2006-07 

 Purpose in seeking technical assistance 

 (percent of state managers) 

 (n=54) 

 

 

k ees ’t

e 

n

c

s

r

e

u

o

os

d

s

te

i h

ta

t

sr

m

u

o fr

h 

O

it

ec

w

n

y

s 

ta

tl

l

s

c

o

i

e

o

s

h

ir

s

c

a

d

s

kr

d

o

n

w

a s

o

t

f 

T

cri

o

t s

ls

i

il

d

k

n 

s e

io

h

t a

f 

t

c

f

p

u

ta

lo

d

s

e

e

y

v

e

c

External Source 

e

t

d

n

ia

r 

e

o

o

d

g

e

a

f

T

s

 

m

k

r

s

Comprehensive Center network 

te

a

e 

87% 66% 59% 54% 61% 22% 4% 

n

t

s

i

ti

(n=54) 

r

te

r

o

le

e

p

px

AE

m

e

S

o

s

c

k

o

ca

e 

T

l

h

te

t t

ta

a

s

h

it 

t

f

e

s

i

th

k

lf

s

e

h

a

s

s 

ic

t

it

e

h

e

c

t

o

r u

w

le

d

o

p

ld

s

m

u

re

s 

o

o

r

p

c

c

e

o

o

t

te

ff

s

T

l

a

ta

t

a

m

s

s

ti

e

i

l

e

n

b

r

i

o

o

e

r p

m

th

a

d

n

a

g

h

la

in

p

lv

o 

o

o

t

T

r

s

o

in

no

s 

ti

a

a

e id

rm

w

fo

e

n

n

i r

h

e

it

th

w

a

t

g

ne

o

r

T

r uc peek

Regional Educational Laboratory 

(n=55) 84 57 45 54 47 20 4 

U.S. Department of Education 

(n=52) 85 40 22 20 21 12 7 

Professional associations (e.g., 

CCSSO, ASCD) (n=52) 96 51 43 36 57 14 5 

Senior managers in other SEAs 

(n=54) 88 54 10 12 30 3 8 

Consulting firms or private 

contractors (n=54) 30 35 63 47 30 37 15 

Colleges and universities (n=54) 43 31 48 40 30 37 12 

EXHIBIT READS: For the 2006-07 program year, 87 percent of state managers (weighted) said that they turned to 

the Centers to gather information or to keep current with new ideas. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

F-6 
Exhibit F.7. Purposes for which states used external sources of technical 

assistance, 2007-08 

Purpose in seeking technical assistance 

(percent of state managers) 

(n=57) 

External Source 

Comprehensive Center network 

rm

w

fo

e

n

n

i r

h

e

it

th

w

a

t

g

ne

o

r

T

r uc peek

65% 

r o

e

r p

m

th

a

d

n

a

g

h

la

in

p

lv

o 

o

o

t

T

r

s

o

in

no

s 

ti

a

a

e id

60% 

h

a

s

ic

t

it

e

h

e

c

t

o

r u

w

le

d

o

p

ld

s

m

u

re

s 

o

o

r

p

c

c

e

o

o

t

f

te

f

s

T

l

a

ta

t

a

m

s

s

ti

e

i

l

e

n

b

i

o

58% 

o

l p

px

AE

m

e

S

o

s

c

k

o

ca

e 

T

l

h

te

t t

ta

a

s

h

it 

t

f

e

s

i

th

k

lf

s

e

s 

c

p

u

ta

lo

d

s

e

e

y

v

e

c

e

t

d

n

ia

r 

e

o

o

d

g

e

a

f

T

s

m

k

r

s

49% 

te

a

e 

n

t

s

i

ti

r

te

r

e

e

s

d

a

s

rk

d

o

n

w

a s

o

t

f 

T

cri

o

t s

ls

i

il

d

k

n 

s e

io

h

t a

f 

t

f

59% 

d

s

te

i h

ta

t

sr

m

u

o fr

h 

O

it

ec

w

n

y

s 

ta

tl

l

s

c

o

i

e

o

s

h

ir

c

16% 

k ees ’t

e 

n

c

s

r

e

u

o

os

3% 

Regional Educational Laboratory 66 39 41 36 29 13 9 

U.S. Department of Education 61 34 8 10 11 8 16 

Professional associations (e.g., 

CCSSO, ASCD) 67 19 11 16 32 6 16 

Senior managers in other SEAs 71 32 6 8 12 2 8 

Consulting firms or private 

contractors 30 26 46 41 23 21 13 

Colleges and universities 35 28 36 29 19 23 18 

EXHIBIT READS: For the 2007-08 program year, 65 percent of state managers (weighted) said that they turned to 

the Centers to gather information or to keep current with new ideas. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded.
 

F-7 
F-8 

Exhibit F.8. Usefulness of Center assistance compared with assistance from 

other sources, 2006-07 

Percent of state managers (weighted) rating 

Center technical assistance 

Much more or 

somewhat 

more useful 

Much less or 

somewhat 

less useful 

About the 

same 

Not able to 

Areas of state responsibility under NCLB judge

Building or managing a statewide system of 

support for districts and schools identified for 

improvement under NCLB (n=54)

58% 16% 14% 12% 

Identifying and/or developing programs or models 

that address district and/or school needs (n=48) 57 15 14 14 

Training or managing the state-level staff or school 

support teams who provide support to districts and 

schools identified for improvement under NCLB 

(n=49)

55 17 11 17 

Disseminating information on scientifically based 

research to districts and schools (n=51) 50 23 12 15 

Providing training and other professional 

development to local educators in academic 

subjects (reading language arts, mathematics, 

science) (n=40)

45 19 16 20 

Formulating or refining state policies to respond to 

NCLB requirements (n=49) 44 20 12 24 

Supporting use of assessment data by schools 

and districts (n=46) 37 21 21 21 

Monitoring compliance with NCLB requirements in 

districts and schools (n=39) 33 22 18 27 

Designing or implementing state assessment and 

accountability systems (n=39) 29 22 22 27 

Aligning state accountability systems with NCLB 

accountability systems (n=39) 27 27 17 29 

Communicating with the public about NCLB 

requirements or report cards (n=32) 22 26 15 37 

NOTE: States that chose the response, “not applicable, state has not sought assistance for this purpose,” were 

excluded from the analysis. 

EXHIBIT READS: For the 2006-07 program year, 58 percent of state managers (weighted) reported that Centers 

were much more or somewhat more useful than other sources of technical assistance for the state responsibility of 

building or managing a statewide system of support for districts and schools identified for improvement under NCLB. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 
Exhibit F.9. Usefulness of Center assistance compared with 

assistance from other sources, 2007-08 

Percent of state managers, weighted, rating 

Center technical assistance as: 

More 

useful 

About the 

same 

Less 

useful 

Not able to 

Areas of state responsibility under NCLB judge 

Statewide systems of support or school support 

teams (n=54)
 60% 19% 10% 11%

State assessment and accountability systems 

(n=41) 32 36 11 21

Assistance with educators’ use of assessment data 

(n=46) 31 36 14 19

Development or dissemination of research-based 

curriculum, instruction, or professional development 

programs in academic subject(s) (n=48) 

41 34 8 16 

Policies and practices for English language 

learners (n=48)
 42 22 10 26

Administering supplemental educational services
 

(SES) and choice provisions (n=34) 29 19 18 34

Communication with parents or the public (n=38) 18 37 19 26 

Monitoring compliance with NCLB requirements 

(n=35) 41 21 12 25

State planning or reporting requirements of NCLB 

not covered above (n=34) 34 34 10 22

NOTE: States that chose the response, “not applicable, state has not sought assistance for this purpose,” were 

excluded from the analysis. 

EXHIBIT READS: For the 2007-08 program year, 60 percent of state managers, weighted, reported that Centers 

were more useful than other sources of technical assistance for the state NCLB responsibility of statewide systems of 

support or school support teams. 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

F-9 
Exhibit F.10. Relevance and usefulness items from the 2006-07 state manager 

survey 

F-10 

Relevance Usefulness 

a. Addressed a need or problem that the state 

faces 

b. Addressed an important state priority 

c. Addressed a challenge that the state faces 

related to the implementation of NCLB 

d. Responded flexibly to our state’s changing 

needs 

e. Provided information, advice, and/or resources 

that could be applied to the state’s work 

f. Addressed the particular context in which our 

state operates 

g. Addressed the state’s specific challenges (e.g., 

policy environment, leadership capacity, 

budget pressures, local politics) 

h. Provided information, advice, and/or resources 

that could be used to guide decisions about 

policies, programs, and practices 

i. Highlighted the implications of research 

findings (or information about best practice) for 

state-level clients’ policies, programs, or 

practices 

a. Provided state staff with resources that were 

easy to understand and easy to use 

b. Employed appropriate formats (e.g., work 

groups, conferences, individual consultation, 

written products) 

c. Provided adequate opportunity to learn from 

colleagues in other states 

d. Included adequate follow-up to support the use 

of new information and resources 

e. Were timely 

f. Helped the state to solve a problem 

g. Helped the state to maintain or change a policy 

or practice 

h. Helped the state take the next step in a longerterm improvement effort 

i. Provided state staff with information or 

resources that they will use again 

j. Helped state staff to develop skills that they will 

be able to exercise again 
Exhibit F.11. Mean ratings of the relevance and usefulness of Center 

assistance, as judged by state managers, 2006-07 

Mean 

relevance 

rating 

(n=54) 

Mean 

usefulness 

rating 

(n=53) 

Center program	 3.94 3.86 

EXHIBIT READS: For the 2006-07 program year, 54 state managers (weighted) gave 

the Center program a mean relevance rating of 3.94. 

SOURCE: Survey of State Managers. Responses weighted so that each state was 

equally represented in instances where more than one manager from a state 

responded. 

Exhibit F.12. Breakout of ratings of the relevance and usefulness of Center 

assistance, as judged by state managers, 2006-07 

 Relevance Usefulness 

Percent of state managers (weighted) 

giving overall ratings in the following 

ranges: 

Percent of state managers (weighted) 

giving overall ratings in the following 

ranges: 

Very 

low to 

low 

(1.0-1.9) 

High to 

very 

high 

(4.0-5.0) 

High to 

very 

high 

(4.0-5.0) 

Low to 

moderate 

(2.0-2.9) 

Moderate 

to high 

(3.0-3.9) 

Very low 

to low 

(1.0-1.9) 

Low to 

moderate 

(2.0-2.9) 

Moderate 

to high 

(3.0-3.9) 

Center program 3 9 31 57 4 8 38 50 

EXHIBIT READS: For the 2006-07 program year, 3 percent of state managers (weighted) gave the Center program 

an overall relevance rating in the very low to low range (1.0-1.9). 

SOURCE: Survey of State Managers. Responses weighted so that each state was equally represented in instances 

where more than one manager from a state responded. 

F-11 

Appendix G 

Ratings of Quality, Relevance, and Usefulness 

Ratings of Quality, Relevance, and Usefulness 

This appendix section presents the expert panel review rating materials and the 

participant surveys that were collected to determine ratings of technical quality, relevance, and 

usefulness. 

Expert Panel Reviewer Scoring Booklet Used for Quality Ratings 

This appendix section presents the Scoring Booklet that was used by the expert panel 

reviewers for all three cycles of quality ratings. The goal in developing the scoring rubric was to 

provide uniform, objective criteria for rating technical quality. Because the evaluation of each 

project was based on the professional judgment of three panelists, differences among raters were 

inevitable. Providing a well-developed scoring rubric and training the panelists on the use of the 

scoring criteria was intended to maximize interrater agreement and reduce bias. 

For the purposes of this evaluation, technical quality of Center projects was demonstrated 

through the comprehensive and balanced use of scientifically based research and the applicable 

nonresearch knowledge base. The best available research or knowledge base was expected to 

inform the delivery of technical assistance. Where rigorous and consistent evidence was lacking, 

there had to be an acknowledgment in the technical assistance provided of the lack of conclusive 

evidence and advice, and recommendations had to be appropriately tempered. Professional 

wisdom was expected to be integrated with the best available empirical evidence in planning for 

and delivering products and services. To be rated high quality, the materials were expected to be 

accurate, complete, and clear and should support use and implementation of the content. 

Relevance and Usefulness Ratings from RCC and State-Level Participant Surveys 

This appendix section presents the sets of participant surveys that were administered to 

RCC and state-level staff to obtain client views of technical assistance from the Centers, 

particularly in the areas of the relevance and usefulness of Center products. 

In the first year of data collection, the evaluation team developed the two parallel survey 

forms. One version of the survey was written using text appropriate for state-level staff who 

participated in any Center’s project. The second, similar version of the survey was written with 

wording appropriate for RCC staff who participated in a CC project. Both surveys asked 

questions in each of the following areas: 

• Project Participation; 

• Relevance and Usefulness; 

• Priorities for Technical Assistance; 

• Capacity to Carry out Responsibilities Related to NCLB, and 

• Job Responsibilities. 

After reviewing the responses to the 2006-07 administration of the participant survey, the 

evaluation team made several revisions to the survey instrument that remained in place for both 

G-1 
the 2007-08 and 2008-09 survey administration cycles (described in exhibit G.1 below). 

Questions that remained unchanged from the first year of survey administration are not included 

in the exhibit. 

G-2 
Exhibit G.1. Revisions to the participant survey from 2006-07 to 2007-08 

G-3
 

Question 2006-07 administration 2006-07 to 2007-08/2008-09 

Time spent on project by type of activity 

Of this set of activities and resources (described at the beginning of this survey), how much time did you spend participating in each 

of the following types of activities or making use of each of the following types of resources? (Circle one response in each row.) 

Reporting categories: 

• More than 5 days • 3-5 days • 1-2 days • Less than 1 day • Not applicable; not part of this set of activities and 

resources 

Types of activities: 

• Conferences • Training 

• Task force meetings 

• Reviewing general or background information provided by 

the Center 

• Using tools and other resources provided by the Content 

Center 

• Advance planning • Ongoing consultation on this topic • Follow-up and action plans 

During the period from July Xhow much time (in days) did activities and resources desat conferences or trainings, advance planning or follow-uprint or electronic materials. 

• More than 30 days • 21-30 days 

• 11-20 days 

• 6-10 days • 5 days or less 

In your experience with this resources, in which of the fo

components did you participeach row.) 

• Advance planning • Conferences • Training 

• Task force meetings 

• Reviewing general o

provided by the Cen

• Using tools and othe

the Center 

• Ongoing consultatio
Question 2006-07 administration 2006-07 to 2007-08/2008-09 

Respondent involvement in project planning Were you personally involved in determining the goals or designing the content or format of this set of 

activities and resources (described at the beginning of this survey)? In what ways? (Circle all that apply.) 

a. Identifying the problem or need to be addressed b. Selecting or framing the content c. Providing data or other background information during the 

planning phase 

d. Identifying or recruiting project participants e. Identifying or recruiting presenters or resources f. Designing activities g. Planning for or leading dissemination of new ideas and 

information 

h. Coordinating this set of activities with other work that my 

organization does 

i. Planning logistics j. Other (Specify: 

______________________________________) 

k. I did not contribute at all to the design of this set of activities 

and resources 

Were you personally involveor designing the content or factivities and resources (desthis survey)? (Circle Yes or 

G-4
 
Question 2006-07 administration 2006-07 to 2007-08/2008-09 

Respondent’s job responsibilities related to NCLB 

During the period from July 2006 through June 2007, which of the following statements best describes your job responsibilities related to NCLB implementation? (Circle all that apply.) 

a. Formulating or refining state policies to respond to NCLB 

requirements 

b. Building or managing a statewide system of support for districts 

and schools identified for improvement under NCLB 

c. Training or managing the state-level staff or school support 

teams who provide support to districts and schools identified for improvement under NCLB 

d. Designing or implementing state assessment and accountability 

systems 

e. Aligning state accountability systems with NCLB accountability 

systems 

f. Supporting use of assessment data by schools and districts g. Disseminating information on scientifically-based research to 

districts and schools 

h. Identifying and/or developing programs or models that address 

district and/or school needs 

i. Providing training and other professional development to local 

educators in academic subjects (reading language arts, mathematics, science) 

j. Monitoring compliance with NCLB requirements in districts and 

schools 

k. Communicating with the public about NCLB requirements or

report cards 

l. Other (Specify: ________________) 

During the period from July 2006 through June 2007, what percent of your time was spent on all the tasks you selected in Question 13 above, combined? (Circle one.) 

• 0-25 percent • 26-50 percent • 51-75 percent • 76-100 percent 

During the period from July Xwhat percent of your time watechnical assistance to state responsibilities related to NC

• 0-25 percent • 26-50 percent 

• 51-75 percent • 76-100 percent 

G-5
 
G-6
 

National Evaluation of the Comprehensive Ce

Scoring Booklet for Expert Reviewers 

[Date] 

Policy Studies Associates 
Dimension 1: Demonstrated Use of Appropriate Knowledge Base 

Indicators Definitions for Indicators Evidence and Applicability to Project 1A. The knowledge base used is appropriately comprehensive, given the project’s purpose. 

• The most important research and 

knowledge are used when applicable and appropriate for the project. 

• Appropriate legal and regulatory 

guidelines are used when applicable for the project. 

1B. The knowledge base 

used is accurately described. 

• The applicable knowledge base 

is accurately described in the project’s materials. 

• Research findings, regulatory 

guidelines, or professional wisdom used in the project materials are accurately described. 

1C. The knowledge base used for the project is relevant. 

• The research/knowledge base 

used is relevant to the topic. 

• Legal and regulatory guidelines 

that are presented are relevant to the topic. 

G-7
 
Dimension 1: Demonstrated Use of Appropriate Knowledge Base (Continued) 

Indicators Definitions for Indicators Evidence and Applicability to Project 1D. The knowledge base represented in the project is balanced, when appropriate. 

• More than one point of view is 

presented in the knowledge base used, if it is applicable and important. 

• Where the level of evidence 

warrants, the knowledge base used reflects the variety of valid perspectives appropriate for consideration in the field. 

1E. Prominence in findings and recommendations is given to current and best available scientifically based research, knowledge base, and professional wisdom 

• Where available, prominence is 

given to research that is scientifically based. 

• Research and knowledge base 

used and reflected in the project represent the latest generation of findings. 

• Research studies that used 

appropriate designs, methodologies, and measures are given prominence over those with weaker designs, when available. 

1F. Limitations in the knowledge base are acknowledged in the project. 

• Materials clearly note the 

strength (or limitations) of the knowledge or research base. 

• Findings or recommendations 

based on best practice only (rather than more empirical evidence) are accurately identified as such in the materials. 

G-8
 
Criteria for Scoring—Demonstrated Use of Appropriate Knowledge Base Directions: Circle the number that best describes the demonstrated use of the appropriate knowledge base in the project, considering the project’s performanc

identified. 

G-9
 

Very High Quality High Quality Moderate Quality Low Quality 

5 4 3 2 

1A The research/knowledge base 

used or referenced in project materials includes an appropriate set of important sources in this field, pertinent to the project’s purpose. 

There is a research/knowledge base used or referenced in project materials, pertinent to the project’s purpose, but some important sources are not included. 

There is used or rmaterialsbase usematerials importanpertinent 

1B All of the research findings, legal 

and regulatory guidelines, or professional wisdom used in the project’s materials is accurately described. 

Some inaccuracies exist in the description of the research/knowledge base in the project’s materials. 

Inaccuraproject in research

1C All of the information presented 

about the research/ knowledge base in project materials is relevant to the topic. 

Some of the research and/or knowledge base reflected in the materials is relevant; however, some of the material presented is not relevant. 

Most of tthe researelevant 

1D The research/ 

knowledge base used or referenced contains a balanced variety of valid perspectives in this field. 

Some balance is provided in presenting the variety of valid perspectives in the field; however, some imbalance is evident. 

No balanthe variofield. 

1E The most current and rigorous 

research and knowledge available is given the most prominence in project materials. 

The most current and rigorous research/knowledge base available is only partially reflected and given prominence in the project. 

The mosresearchminimallno prom

1F Limitations in the available 

research/knowledge base are clearly described and acknowledged in project materials. 

Limitations in the available research/knowledge base are only partially described and acknowledged in the project materials, although they exist. 

No limitaresearchacknowlematerials
Dimension 2: Fidelity of Application of the Knowledge Base to the Products and Services Pro

G-10
 

Indicators Definitions for Indicators Evidence and Applicability to Project 

2A. The application of theresearch/ knowledge base is clear and accurate. 

• Accurate application of the core 

findings from the available knowledge base is evident in theproject’s products, assistance, oradvice. 

• The project’s products, 

assistance, and advice clearly interpret and apply the research and/or knowledge base. 

2B. There is consistency between the strength of the research/knowledge base and its proposed application. 

• The certainty and strength of 

recommendations in the project’s products, assistance, and advice are consistent with and appropriate for the level of rigor and certainty in the available research/knowledge base. 

2C. Appropriate emphasis is given in application of the most 

rigorous and consistent research and knowledgebase. 

• Prominence is given to products, 

assistance, and advice derived from the most rigorous research 

and knowledge base. 

• Prominence is given to products, 

services, and advice derived from the most consistent 

research and knowledge base. 

2D. Application of nonempirical research and professional wisdom only are appropriately tempered. 

• Products, assistance, and 

advice based on a weak 

research base, limited legal or regulatory guidance, or primarily nonempirical professional wisdom are appropriately 

tempered. 
Criteria for Scoring—Fidelity of Application of the Knowledge Base to the Products and Services Provided Directions: Circle the number that best describes the fidelity of application of the knowledge base to the products and services provided, considering the projeindicators identified. 

G-11
 

Very High Quality High Quality Moderate Quality Low Quality 

5 4 3 2 

2A The products, assistance and 

advice provided through this project are consistently accurate and clear in interpreting and applying the available research and knowledge base used in the project. 

The products, assistance, and advice provided through this project are accurate 

and clear in interpreting and applying the available research and knowledge base in the project used in some respects but not others. 

The advproinainteavaknopro

2B The products, advice, and 

assistance provided through this project are fully consistent with the available research and knowledge base. 

The products, advice, and assistance provided through this project are partially consistent with the available research and knowledge. 

The assprothe avakno

2C The products, assistance, and 

advice provided through this project completely and consistently emphasize the application of findings derived from rigorous and consistent research and knowledge over the application of findings from less rigorous or consistent research or knowledge. 

Some of the products, assistance, and advice provided through this project 

emphasize the application of findings derived from rigorous and consistent research and knowledge over the application of findings from less rigorous or less consistent research or knowledge in some instances, but some products, assistance, and advice provided emphasize less rigorous or 

consistent research and knowledge. 

The advprothe derlesknoof fconkno

2D Products, assistance, and 

advice provided through this project that are based on weak research or knowledge base are consistently acknowledged and appropriately tempered. 

Products, assistance, and advice provided through this project that are based on weak research or knowledge base are partially acknowledged and are tempered only in 

part. 

Proproare 

knoackrar
Dimension 3: Clear and Effective Delivery 

G-12
 

Indicators Definitions for Indicators Evidence and Applicability to Project 3A. Information communicated in the project products and services meets the project’s purpose. 

• Information contained in the project 

materials is clear in meeting the purpose of the project. 

• Information contained in the project 

materials is complete and applicable in meeting the purpose of the project. 

3B. The project contains meaningful learning experiences, appropriate for the intended audience. 

••

Where appropriate, products and services are designed to engage 

participants in effective learning experiences. In the opinion of the reviewer, the audience should be interested in the 

information and recommendations because of the way in which they are delivered. 

3C. The products and services in the project are appropriate for the intended audience. 

• The products and services as presented 

are relevant and well-suited for the intended audience. 

• The products and services as presented 

appear to be useful for the intended audience. 

3D. Ideas are 

effectively communicated. 

• Clear and accessible language is used 

in project materials. 

• Project materials are well-written. 

• Information conveyed will be 

understood by intended audience. 
Criteria for Scoring—Clear and Effective Delivery 

Directions: Circle the number that best describes clear and effective delivery considering the project’s performance on the various indicat

Very High Quality High Quality Moderate Quality Low Quality 

5 4 3 2 

3A Project materials and the 

information presented are complete and applicable for meeting the project’s purpose. 

Project materials and information are partially incomplete or not applicable for meeting the project’s purpose. 

Pinnp

3B The project materials will 

engage participants or users in meaningful learning experiences or will be of interest to them because of the way in which materials are delivered or packaged. 

Some of the project materials will engage participants in learning experiences or will be of interest to them, but some of the materials will not because of the way they are delivered or packaged. 

Tmeupod

3C The products and services 

produced for this project are very appropriate for the intended audience. 

A portion of the products and services produced for this project are not appropriate or relevant for the intended audience. 

Tpnath

3D Ideas and information are 

effectively communicated throughout all of the materials in the project. 

The ideas and information in the project materials are not effectively communicated and are confusing in some instances. 

Tacth

G-13
 
________________________________________________________________________ 

Scoring Booklet for Quality Ratings 

National Evaluation of Comprehensive Centers 

[date] 

Score Reporting Form for Expert Reviews 

Project Title:_____________________________________________________________ 

Comprehensive Center 

Name:___________________________________________________________ 

Please list your scores below: 

Dimension 1 Score: ________________________ 

Dimension 2 Score: ________________________ 

Dimension 3 Score: ________________________ 

I attest that the scores listed on this scoring sheet are based on my thorough review and 

objective assessment of the project cover sheet and project artifacts based on the scoring 

guidance and rubric provided by the evaluation contractor. I attest that I have not 

discussed these materials or this review with anyone other than the designated evaluation 

team contact prior to submission. 

I have also provided a bulleted summary of this project’s strengths and weaknesses. 

I further attest that I do not have any undisclosed conflict of interest for this particular 

project. 

Your Name 

(Print):__________________________________________________________________ 

Your Signature:_______________________________________________________________ 

Date:_____________________________ 

G-14 
Appendix H 

Supplemental Tables – Quality, Relevance, 

and Usefulness Ratings 

Quality Relevance and Usefulness Ratings – 

Supplemental Tables 

This appendix serves as a supplement to the information presented in chapter 5 of the 

report. It includes the quality, relevance, and usefulness ratings for the 2006-07 and 2007-08 

data collection cycles as well as more detailed versions of some exhibits provided in the report 

for all three years. 

 

 

 

 

 

Exhibit H.1. Mean ratings on dimensions of technical quality, 2006-07, by 

center type 

Quality dimensions: 

Dimension 1 Dimension 2 Dimension 3 

(use of (clear and 

Technical knowledge (fidelity of effective 

quality base) application) delivery) 

All Centers 3.34 3.22 3.20 3.60 (N=21) 

All RCCs 3.21 3.05 3.07 3.52 (N=16)
 

All CCs (N=5) 3.73 3.75 3.59 3.85
 

Difference of 

RCC and CC -0.52†

 -0.70† -0.52† -0.33† 

means 

Pooled standard 

deviation(all 0.41 0.49 0.44 0.35 

Centers) 

Ratio of difference 

in means to -1.28 -1.42 -1.18 -0.95 pooled standard 

deviation 

NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” 

rating is the mean of the ratings for the three quality dimensions. A notation of †

 indicates 

that the difference in the mean ratings between the CCs and RCCs is at least one-half of 

one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.34 for 

the program year 2006-07. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of 

project participants for relevance and usefulness. Responses weighted so that each 

panelist or participant contributed equally to project ratings; each project contributed 

equally to Center ratings; and each Center contributed equally to cross-Center ratings. 

H-1 
Exhibit H.2. Mean ratings on dimensions of technical quality, 2007-08, by 

center type 

 Quality dimensions: 

Technical 

quality 

Dimension 1 

(use of 

knowledge 

base) 

Dimension 2 

(fidelity of 

application) 

Dimension 3 

(clear and 

effective 

delivery) 

All Centers 

(N=21) 3.51 3.39 3.44 3.71

All RCCs 

(N=16)
 

All CCs (N=5) 

3.41 

3.86 

3.25 

3.83 

3.32 

3.83 

3.64

3.94
 

Difference of 

RCC and CC 

means 

Pooled standard 

deviation(all 

Centers) 

Ratio of difference 

in means to 

pooled standard 

deviation 

-0.45†

0.41 

-1.09 

-0.58†

0.53 

-1.09 

-0.51†

0.43 

-1.19 

-0.30† 

0.35 

-0.85

NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” 

rating is the mean of the ratings for the three quality dimensions. A notation of † indicates 

that the difference in the mean ratings between the CCs and RCCs is at least one-half of 

one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.51 for 

the program year 2007-08. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of 

project participants for relevance and usefulness. Responses weighted so that each 

panelist or participant contributed equally to project ratings; each project contributed 

equally to Center ratings; and each Center contributed equally to cross-Center ratings. 

H-2 
Exhibit H.3. Mean ratings on dimensions of technical quality, 2008-09, by 

center type 

Quality dimensions: 

Technical 

quality 

Dimension 1 

(use of 

knowledge 

base) 

Dimension 2 

(fidelity of 

application) 

Dimension 3 

(clear and 

effective 

delivery) 

All Centers 

(N=21) 3.57 3.45 3.54 3.72

All RCCs 

(N=16)
 

All CCs (N=5) 

3.52 

3.72 

3.40 

3.63 

3.47 

3.73 

3.70

3.81
 

Difference of 

RCC and CC 

means 

Pooled standard 

deviation(all 

Centers) 

Ratio of difference 

in means to 

pooled standard 

deviation 

-0.20† -0.23†

0.37 

-0.55 

0.43 

-0.54 

-0.26† 

0.37 

-0.71 

-0.11 

0.36 

-0.31

NOTE: All ratings were on a 5-point scale, with 5 as the high value. The “technical quality” 

rating is the mean of the ratings for the three quality dimensions. A notation of † indicates 

that the difference in the mean ratings between the CCs and RCCs is at least one-half of 

one pooled standard deviation in the rating. 

EXHIBIT READS: Among the 21 Centers, the mean technical quality rating was 3.57 for 

the program year 2008-09. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of 

project participants for relevance and usefulness. Responses weighted so that each 

panelist or participant contributed equally to project ratings; each project contributed 

equally to Center ratings; and each Center contributed equally to cross-Center ratings. 

H-3 
Exhibit H.4. Mean relevance ratings, by components of relevance, by year 

Mean relevance component rating 

Components of relevance: 2006-07 2007-08 2008-09 

Addressed a need or problem that my 

organization faces 4.11 4.25 4.26

Addressed an important priority of my 

organization 4.07 4.19 4.23

Provided information, advice, and/or 

resources that could be applied to the work of 

my organization 

4.07 4.26 4.27 

Addressed a challenge that my organization 

faces related to the implementation of NCLB 3.99 4.11 4.07

Provided information, advice, and/or 

resources that could be used to guide 

decisions about policies, programs, and 

practices 

3.92 4.15 4.18

Highlighted the implications of research 

findings (or information about best practice) 

for policies, programs, or practices 

3.89 4.14 4.13 

Addressed our particular state context 3.87 4.05 4.16 

Addressed the specific challenges facing my 

organization (e.g., policy environment, 

leadership capacity, budget pressures, local 

politics) 

3.63 3.94 3.97

EXHIBIT READS: For the 2006-07 program year, the project-level relevance rating was 4.11 for the relevance 

component “addressed a need or problem that their organization faced.” 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist contributed equally to the project rating and 

each project contributed equally to Center ratings. 

H-4 
Exhibit H.5. Mean usefulness ratings, by components of usefulness, by year 

Mean usefulness component rating 

Components of usefulness: 2006-07 2007-08 2008-09 

Provided resources that were easy to 

understand and easy to use 4.05 4.18 4.15

Employed an appropriate format (e.g., a work 

group, a conference, individual consultation, 

written products) 

4.04 4.19 4.23 

Were timely 3.94 4.12 4.13 

Provided my organization with information or 

resources that we will use again 3.92 4.17 4.13

Helped my organization to develop a shared 

expertise or knowledge-base 3.75 4.05 3.98

Helped individuals in my organization 

develop skills that they will use again 3.65 3.93 3.87

Helped my organization take the next step in 

a longer-term improvement effort 3.53 3.89 3.95

Provided adequate opportunity to learn from 

colleagues in other states 3.48 3.54 3.61

Helped my organization to solve a problem 3.47 3.83 3.88 

Included adequate follow-up to support the 

use of new information and resources 3.46 3.80 3.86

Helped my organization to maintain or 

change a policy or practice 3.24 3.73 3.74

EXHIBIT READS: For the 2006-07 program year, the project-level usefulness rating was 4.05 for the usefulness 

component “Provided resources that were easy to understand and easy to use.”
 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist contributed equally to the project rating and 

each project contributed equally to Center ratings. 

H-5 
Exhibit H.6. Technical quality, relevance, and usefulness ratings of projects, 

by type of activity offered in the projects, 2006-07 

Type of assistance activity offered Technical quality Relevance Usefulness 

Ongoing Yes 3.25 4.03 3.74 

consultation and (n=84) 

follow-up No 3.59 3.79 3.60 (n=38) 

Diff SD Ratio -0.34† 0.50 -0.68 0.24† 0.46 -0.52 0.14 0.54 0.26 

Research Yes 3.50 3.94 3.67 

collections and (n=71) 

syntheses No 3.17 3.98 3.73 (n=51) 

Diff SD Ratio 0.33 0.67 0.49 -0.04 0.52 -0.08 -0.06 0.54 -0.11 

Engagement of Yes 3.24 4.00 3.73 

participants in (n=50) 

project planning No 3.44 3.93 3.67 (n=72) 

Diff SD Ratio -0.20 0.67 -0.30 0.07 0.52 0.14 0.06 0.54 0.11 

Training events Yes 3.34 4.09 3.83 (n=50) 

No 3.37 3.86 3.60 (n=72) 

Diff SD Ratio -0.03 0.67 -0.05 0.23 0.52 0.44 0.23 0.54 0.43 

Task force Yes 3.23 4.05 3.71 

meetings and work (n=50) 

No 3.45 3.89 3.68 (n=72) 

Diff SD Ratio -0.22 0.52 -0.42 0.16 0.54 0.30 0.03 0.67 0.04 

Conference Yes 3.60 3.83 3.61 (n=43) 

No 3.21 4.03 3.74 (n=79) 

Diff SD Ratio 0.39† 0.67 0.58 -0.20 0.52 -0.40 -0.13 0.54 -0.24 

Support Yes 3.38 4.02 3.63 

development of a (n=20) 

formal plan to No 3.35 3.94 3.71 implement a (n=102) 

program or policy Diff SD Ratio 0.03 0.67 0.04 0.08 0.54 0.15 -0.08 0.54 -0.15 

NOTE: For each type of assistance, the first two rows show the mean ratings and the number of projects in the 

sample for Yes (the activity or resource was offered) and No (not offered). The third row shows the difference in the 

means, the pooled standard deviation, and the calculated ratio of the difference in means to the pooled standard 

deviation. A notation of † indicates that the difference in the mean ratings is at least one-half of one pooled standard 

deviation.
 

EXHIBIT READS: For the 2006-07 program year, the mean quality rating of projects that offered ongoing consultation
 

and follow-up was 3.25, while the mean quality rating of projects that did not offer this was 3.59. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 

H-6 
Exhibit H.7. Technical quality, relevance, and usefulness ratings of projects, 

by type of activity offered in the projects, 2007-08 

Type of assistance activity offered Technical quality Relevance Usefulness 

Ongoing Yes 3.47 4.19 3.97 

consultation and (n=92) 

follow-up No 3.74 3.96 3.82 (n=29) 

† Diff SD Ratio -0.27 0.65 -0.41 0.23 0.43 0.53 0.15 0.48 0.32 

Research Yes 3.58 4.11 3.88 

collections and (n=73) 

syntheses No 3.46 4.18 4.02 (n=47) 

Diff SD Ratio 0.12 0.65 0.19 -0.08 0.43 -0.18 -0.140.48 -0.29 

Engagement of Yes 3.59 4.14 3.88 

participants in (n=67) 

project planning No 3.46 4.13 4.00 (n=54) 

Diff SD Ratio 0.13 0.65 0.20 0.01 0.43 0.02 -0.11 0.48 -0.24 

Training events Yes 3.51 4.10 3.92 (n=65) 

No 3.56 4.18 3.96 (n=56) 

Diff SD Ratio -0.05 0.65 -0.07 -0.08 0.43 -0.19 -0.04 0.48 -0.08 

Task force Yes 3.45 4.21 3.96 

meetings and work (n=57) 

No 3.61 4.07 3.91 (n=64) 

Diff SD Ratio -0.16 0.65 -0.25 0.14 0.43 0.33 0.05 0.48 0.11 

Conferences Yes 3.56 4.08 3.89 (n=45) 

No 3.52 4.17 3.96 (n=76) 

Diff SD Ratio 0.05 0.65 0.07 -0.10 0.43 -0.22 -0.070.48 -0.15 

Support Yes 3.50 4.30 4.11 

development of a (n=34) 

formal plan to No 3.55 4.07 3.87 implement a (n=87) 

† † program or policy Diff SD Ratio -0.04 0.65 -0.07 0.23 0.43 0.52 0.24 0.48 0.51 

NOTE: For each type of assistance, the first two rows show the mean ratings and the number of projects in the 

sample for Yes (the activity or resource was offered) and No (not offered). The third row shows the difference in the 

means, the pooled standard deviation, and the calculated ratio of the difference in means to the pooled standard 

deviation. A notation of † indicates that the difference in the mean ratings is at least one-half of one pooled standard 

deviation. 

EXHIBIT READS: For the 2007-08 program year, the mean quality rating of projects that included ongoing 

consultation and follow-up was 3.47 while the mean quality rating of projects that did not offer this 3.74. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 

H-7 
Exhibit H.8. Technical quality, relevance, and usefulness ratings of projects, 

by type of activity offered in the projects, 2008-09 

Type of assistance activity offered Technical quality Relevance Usefulness 

Ongoing 

consultation and 

Yes 

(n=100) 3.60 4.18 4.01 

follow-up No 

(n=18) 3.50 4.03 3.81 

Diff SD Ratio 0.10 0.61 0.16 0.15 0.43 0.35 0.19 0.47 0.41 

Research 

collections and 

syntheses 

Yes 

(n=68) 

No 

(n=50) 

3.72 

3.04 

4.14 

4.19

3.98 

3.99 

Diff SD Ratio 0.68† 0.61 1.11 -0.05 0.43 -0.12 -0.01 0.47 -0.02 

Engagement of 

participants in 

project planning 

Yes 

(n=61) 

No 

(n=57) 

3.62

3.54 

4.13 

4.16 

3.93 

4.04 

Diff SD Ratio 0.08 0.61 0.13 -0.03 0.43 -0.07 -0.11 0.47 -0.24 

Training events Yes 

(n=67) 

No 

(n=51) 

3.59 

3.58 

4.23 

4.08 

4.04 

3.90 

Diff SD Ratio 0.00 0.61 0.01 0.16 0.43 0.36 0.15 0.47 0.31 

Task force 

meetings and work 

Yes 

(n=54) 

No 

(n=64) 

3.58

3.59 

4.20 

4.13 

3.98 

3.98 

Diff SD Ratio -0.02 0.61 -0.03 0.07 0.43 0.17 0.00 0.47 0.00 

Conference Yes 

(n=47) 

No 

(n=71) 

3.68

3.53 

3.97

4.29 

3.81 

4.09 

Diff SD Ratio 0.15 0.61 0.25 -0.32† 0.43 -0.73 -0.27† 0.47 -0.58 

Support 

development of a 

formal plan to 

implement a 

program or policy 

Yes 

(n=35) 

No 

(n=83) 

3.60 

3.58 

4.25 

4.12 

4.09 

3.93 

Diff SD Ratio 0.03 0.61 0.05 0.13 0.43 0.30 0.16 0.47 0.34 

NOTE: For each type of assistance, the first two rows show the mean ratings and the number of projects in the 

sample for Yes (the activity or resource was offered) and No (not offered). The third row shows the difference in the 

means, the pooled standard deviation, and the calculated ratio of the difference in means to the pooled standard 

deviation. The standard deviation was calculated across all projects for technical quality, relevance and usefulness 

and therefore the same standard deviation was used in the calculations for each type of assistance offered. A 

notation of †

 indicates that the difference in the mean ratings is at least one-half of one pooled standard deviation.
 

EXHIBIT READS: For the 2008-09 program year, the mean quality rating of projects that ongoing consultation and 

follow-up was 3.60, while the mean quality rating of projects that did not offer this was 3.50. 

SOURCE: Expert panel ratings of sampled projects for technical quality and surveys of project participants for 

relevance and usefulness. Responses weighted so that each panelist or participant contributed equally to the project 

rating, and each project contributed equally to the mean rating for its category. 

H-8 
Percent of participants Involved in determining 

the goals or design of 

the project 

RCC projects 

(n=2,355) 

CC projects 

(n=1,116) 

Total 100% 100%

Yes 57 42

No 43 58

Exhibit H.9. Participants’ involvement in determining project goals or design, 

by type of Center, 2006-07 

NOTE: Difference in the proportion of participants involved in design by type of Center is 

statistically significant (p<.01, Chi Square). 

EXHIBIT READS: For the 2006-07 program year, 57 percent of participants in projects 

conducted by RCCs reported they were involved in determining the goals or design of the 

project. 

SOURCE: Surveys of project participants. Responses weighted so that respondents 

represented all participants for the project. 

H-9 
 

Exhibit H.10. Mean relevance and usefulness ratings, by time respondents 

spent in each activity, 2006-07 

Type of activity or resource Time spent Relevance Usefulness 

Ongoing consultation on 

this topic 

3+ days (n=1,156) 

<3 days (n=1,115)

4.35 

3.81 

4.06

3.59 

Difference 0.54* 0.47* 

Significance p<.01 p<.01
 

Reviewing general or 

background materials 

provided by the Center 

3+ days (n=1,105)

<3 days (n=1,534)

4.32 4.07 

3.78 3.53 

Difference 0.54* 0.54* 

Significance p<.01 p<.01 

Advance planning 3+ days (n=731)

<3 days (n=1,311)

4.37 4.08 

3.91 3.65 

Difference 0.46* 0.40* 

Significance p<.01 p<.01
 

Training 3+ days (n=1,136)

<3 days (n=1,219)

4.13 3.93 

3.94 3.72 

Difference 0.19* 0.21*
 

Significance p<.01 p<.01
 

Task force meetings 3+ days (n=581)

<3 days (n=1,061)

4.29 3.94

3.86 3.61 

Difference 0.43* 0.33* 

Significance p<.01 p<.01
 

Conferences 3+ days (n=1,219) 4.15 3.87 

<3 days (n=1,556) 3.82 3.63

Difference 0.33* 0.24*
 

Significance p<.01 p<.01
 

Follow-up and action plans 3+ days (n=1,138)

<3 days (n=1,307)

4.38 4.04 

3.81 3.57 

Difference 0.57* 0.47* 

Significance p<.01 p<.01
 

Using tools and other 

resources provided by the 

Center 

3+ days (n=1,450)

<3 days (n=1,305)

4.34 4.10 

3.68 3.42 

Difference 0.66* 0.68* 

Significance p<.01 p<.01 

* Difference statistically significant at p<.05 using a one-way ANOVA. 

NOTE: The types of activities and resources shown in this table are based on the types of activities presented in the 

participant survey and do not match categories presented in chapter 3 exhibits and exhibit 5.8, which were coded 

categories from the project cover sheets. 

EXHIBIT READS: For the 2006-07 program year, the relevance ratings among project participants who indicated that 

they spent 3 or more days receiving ongoing consultation or support was 4.35 and statistically significantly higher 

than the ratings from project participants who spent less time receiving ongoing consultation or support (3.81). 

SOURCE: Surveys of project participants. Responses weighted so that respondents represented all participants for 

the project. 

H-10 
Exhibit H.11. Multiple regression of participant characteristics and ratings of 

relevance, by year 

2007-08
 

Adjusted R-Square .186 

Coefficient Beta 

Standard 

Error of 

Beta 

Standardized Beta Significance 

Constant 2.98 0.05 <.01 

Involved in determining the 

goals or content of the 

project 

0.20 0.03 0.11 <.01 

Time participants spent on 

project activities 0.20 0.01 0.35 <.01 

Agency where respondent 

worked 0.18 0.03 0.09 <.01 

Time spent on NCLB-related 

tasks 0.10 0.01 0.14 <.01 

2008-09
 

Adjusted R-Square .187 

Coefficient Beta 

Standard 

Error of 

Beta 

Standardized Beta Significance 

Constant 3.04 0.06 <.01 

Involved in determining the 

goals or content of the 

project 

0.29 0.03 0.17 <.01 

Time participants spent on 

project activities 0.17 0.01 0.29 <.01 

Agency where respondent 

worked 0.25 0.03 0.14 <.01 

Time spent on NCLB-related 

tasks 0.09 0.01 0.14 <.01 

EXHIBIT READS: For the 2007-08 program year, the adjusted R-Square for a multiple regression 

model with these four predictors was .186, indicating the model explained nearly 19 percent of the 

variance in the ratings of project relevance. All four variables were found to be significant predictors of 

the rating of relevance after controlling for the other three factors. Overall, the amount of time a 

participant spent on project activities explained the largest portion of variance, and having a role in role 

in the design of the project the next largest. 

SOURCE: Surveys of sample-eligible project participants. Responses weighted so that respondents 

represented all participants for the project. 

H-11 
Exhibit H.12. Multiple regression of participant characteristics and ratings of 

usefulness, by year 

2007-08
 

Adjusted R-Square .134 

Coefficient Beta 

Standard 

Error of 

Beta 

Standardized Beta Significance 

Constant 3.06 0.05 <.01

Involved in determining the 

goals or content of the 

project 

0.22 0.03 0.12 <.01 

Time participants spent on 

project activities 0.18 0.01 0.31 <.01 

Agency where respondent 

worked 0.18 0.03 0.09 <.01 

Time spent on NCLB-related 

tasks 0.04 0.01 0.06 <.01 

2008-09
 

Adjusted R-Square .185 

Coefficient Beta 

Standard 

Error of 

Beta 

Standardized Beta Significance 

Constant 2.70 0.06 <.01 

Involved in determining the 

goals or content of the 

project 

0.24 0.03 0.13 <.01 

Time participants spent on 

project activities 0.16 0.01 0.27 <.01 

Agency where respondent 

worked 0.38 0.03 0.21 <.01 

Time spent on NCLB-related 

tasks 0.11 0.01 0.17 <.01 

EXHIBIT READS: For the 2007-08 program year, the adjusted R-Square for a multiple regression 

model with these four predictors was .134, indicating the model explained nearly 13 percent of the 

variance in the ratings of project usefulness. All four variables were found to be significant predictors 

of the rating of usefulness after controlling for the other three factors. Overall, the amount of time a 

participant spent on project activities explained the largest portion of variance, and having a role in role 

in the design of the project the next largest. 

SOURCE: Surveys of sample-eligible project participants. Responses weighted so that respondents 

represented all participants for the project. 

H-12