1



# Evaluation of the Comprehensive **Technical Assistance Centers OMB Clearance Request for Data Collection Instruments**  **Part A: Supporting Statement for Paperwork Reduction Act Submission**



## September 1 **8 , 2014** 

**Prepared for:** U.S. Department of Education Contract No. ED

- IES

- 13

- C

- 0059 **Prepared by:** IMPAQ International2



### INTRODUCTION This document has been prepared to support the clearance of data collection instruments for the National Evaluation of the Comprehensive Technical Assistance Centers. The Institute of Education Sciences (IES) within the U.S Department of Education (ED) is conducting this evaluatio n. In the introduction to the supportin g statement, we provide a description of the Comprehensive Technical Assistance Centers program, the evaluation questions and study design. The remaining sections of this document respond to specific instructions of the Office of Management and Budget (OMB ) for the preparation of a supporting statement

. 

### The Comprehensive Technical Assistance Centers Title II of the Educational Technical Assistance Act of 2002 ( F.T AA, Section 203) authorized t he Comprehensive Center (CC) Program, a discretionary grant program establishing technical assistance centers. The CCs were last awarded in 2012, to provide technical assistance to “ help State Education Agencies (SEAs) build their capacity to implement State

- level initiatives and to support d istrict

- a nd school

- level initiatives that improve educational outcomes for all students, close achievement gaps, and improve the quality of instruction” ( 77 FR 33564 ). In 2012, the Department of Education awarded new five

-year grants to 15 Regional Cent ers and 7 C ontent C enters under the CC program . The R egional C enters each serve one to seven U.S. states, territories, and possessions

. They provide technical assistance (TA) that builds the capacity of SEAs to implement, support, scale up, and sustain ini tiatives that help districts and schools improve student outcomes. The CCs focus their work on seven Federal priority areas: 1. Implementing college

- and career

- ready standards and aligned, high

- quality assessments for all students; 2. Identifying, recruiting, developing, and retaining highly effective teachers and leaders; 3. Turning around the lowest

- performing schools; 4. Ensuring the school readiness and success of preschool

- age children and their successful transition to kindergarten; 5. Building rigorous instructional pathways that support the successful transition of all students from secondary education to college without the need for remediation, and careers; 6. Identifying and scaling up innovative approaches to teaching and learning that significantly im prove student outcomes; and 7. Using data

- based decision

- making to improve instructional practices, policies, and student outcomes. The Content Centers provide the Regional Centers and SEAs with in

- depth content knowledge and expertise by providing informati on, publications, tools, and specialized technical assistance. The 7 Content Centers are: 1. Center on Standards and Assessments Implementation 2. Center on Great Teachers and Leaders 3. Center on School Turnaround 4. Center on Enhancing Early Learning Outcomes3

5. Center on College and Career Readiness and Success 6. Center on Building State Capacity and Productivity 7. Center on Innovations in Learning 

### The National Evaluation of the Comprehensive Technical Assistance Centers The National Evaluation is charged with examining and documenting how the individual CCs intend to build SEA capacity ( referred to as theories of action ) and what types of activities they actually conduct to build capacity. It is designed to build on the p revious evaluation of the CCs, which documented the type, extent, and quality of services provided. 

### Evalu **ation Questions** The evaluation will address broad questions in three areas:

 **Program Design: How did the C C** **s design their work?** 1. How did CCs define capacity building? Did their definitions change over time? If so, how? 2. What theories of action did CCs use to guide their general capacity

- building work? Did the theories change over time? If so, how? 3. How did CCs assess the needs of their constituencies? I n addressing these evaluation question s, we seek to identify how the CCs designed their work as TA providers, including the underlying theories of action driving the work. The evaluation will seek to surface the theories of action and definitions of capaci ty building employed by CCs, as well as describe CCs’ plans for assessing the needs of their constituencies and developing TA work plans to address those needs.
 **Program Implementation: How did the C C** **s operate?** 4. What strategies did CCs employ to achieve t heir outcomes? 5. To what extent did CCs implement technical assistance to their constituencies as planned? 6. To what extent and how did CCs collaborate with each other? In addressing these evaluation questions, we seek to identify the various strategies CCs us ed to build capacity, describe the characteristics of those strategies, and document the extent to which CCs implemented the TA as planned. We will also seek to identify common challenges and barriers CCs faced in building capacity, and ways they met those challenges.
 **Program Outcomes: What was the result of the C C** **s’ work?** 7. To what extent did CCs achieve their goals and objectives? In addressing this evaluation question, we seek to identify the extent to which CCs achieved their goals and objectives, particularly as they relate to building their constituents’ capacity. We will also explore the extent to which outcomes aligned with and supported the CCs’ theories of action, and to identify factors that may have contributed to CCs’ success (or failure) in achieving expected outcomes.4



### Focus **o n Two Federal Priority Areas**  As a way to focus the evaluation to gather data in depth rather than breadth, the evaluation will limit data collection on the implementation and outcomes questions to two of the seven federal priority areas: 1. I dentifying, recruiting, developing, and retaining highly effective teachers and leaders , and 2. E nsuring the school readiness and success of preschool

- age children and their successful transition to kindergarten

. These two priority areas were purposefully selected. First, **effective teachers and leaders** is a topic area in which all of the Regional Centers have ongoing projects. In addition, this is a topic area where most SEAs have significant TA and capacity building needs, as many are choosing and implementing educator evaluation systems or supporting districts and schools as they hire and evaluate their professional staff. This priority is also tied to school reform efforts and large federal funding streams such as the Race to the Top initiative, the School Improvement Grants, and the Teacher Incentive Fund. The second priority area, **early learning**, is another high

- profile topic which has recently gained increased attention. In response to federal initiatives and research findings on the benefits of high

- quality early education, many states have increased their funding for state

- supported early childhood education programs over last few years. This evaluation is well poised to examine the role that the CCs play in supporting state e fforts in this priority area. Given the overarching nature of the effective teachers and leaders priority, and the recent policy focus on early learning efforts, we believe that focusing on these two priorities will allow us to learn about how CCs develop SEA capacity (and in the case of Content Centers, both SEA and Regional Center capacity) in these two areas and what difference the CC s’ efforts have made. Furth er, we believe that SEAs’ capacity building needs and the CC s’ approach to providing TA in thes e two priorities may differ across C enters in meaningful ways . These differences are likely to prod uce different types of capacity building outcomes ( i.e ., the needs and approach to building capacity to develop g reat teachers and leaders may be different t han the needs and approach to building capacity related to early learning initiatives). Thus, by selecting these two priorities, we may learn more about the variety of needs and the ways the C Cs address those needs, as well as their outcomes. The selecti on of two priority areas in no way implies that the Department has a preference for these areas over others, or that the centers or SEAs should shift the focus of their efforts to these areas. Rather, this narrowing of focus allows us to target our resourc es in such a way that we are able to learn more about sp ecific capacity building activities and outcomes.5



### Data Sources Data collection for this study will consist of surveys of TA recipients, interviews of CC staff and TA recipients, surveys of CC staff, and observations of TA events. **TA recipient surveys**

 Purpose: To gather information about TA received in selected priority areas from the C Cs, understand the actions resulting from participati ng in TA, and exam ine outcomes related to that TA;  Sample: All S EA and possibly school district staff who received TA from the CCs in the selected priority areas, along with Regional Center staff receiving TA from Content Centers ;  Timing: Yearly, beginning in January 2015 (following OMB approval)

. **C C**  **staff and TA**  **recipient interviews** 

• Purpose: To gather information about the Centers’ capacity building efforts, th eories of action, program implementation, and outcomes ; • Sample: A purposeful sample of Center Directors, Managers, Evaluators, and TA staff; Recipients of CC TA, including SEA and possibly school district staff, and other C Cs ; • Timing: In coordination with site visits

. **C C**  **staff surveys**

 Purpose: To gather information from multiple C C TA staff about the nature , successes and challenges of their work buildi ng SEA capacity or Regional Center capacity, and to help identify high

- leverage projects for more detailed study;  Sample: All Center staff providing TA in the selected priority areas ;  Timing: Yearly, beginning in January 2015 ( immediately following OMB approval)

. **TA event observations** 

 Purpose: To obtain detailed data about the strategies that C Cs use to support capacity building and achieve planned outcomes ;  Sample: Observable services or events that are planned for the selected priority areas, in profil ed projects, or projects that are potential profiled projects ;  Timing: On an ongoing basis beginning in Jan uary 2015, as relevant observable activities are identified

. 

### Evaluation reports The evaluation will produce four reports. The first report will be a formative report focusing on how the C Cs design ed their work as technical assistance providers . The report will describe the C Cs’ underlying theories of action and definitions of “ capacity building,” and explain how the Centers assess ed their constituenc ies’ needs and develop ed work plans to address those needs. This report will be available in November 2015.6

Two interim summative reports (projected date: March 2017) will be produced to address the program implementation and outcomes qu estions, one report for each of the two selected priority areas

. These reports will demonstrate how and to what extent the Comprehensive Technical Assistance Center program has built state capacity in these priority areas. The reports will include descriptions of the strategies used to build SEA capacity, common challenges faced and ways the C Cs sought to address them, the extent to which C Cs achieved their goals and objectives, factors that may have contributed to success (or failure) in achieving expected outcomes, a nd the extent to which C Cs’ outcomes aligned with and supported their theories of action. The interim summative reports will also include six profiles of multi

-year projects, which will be selected and analyzed to illustrate how the C Cs have successfully w orked to build capacity related to project design and planning, implementation, and outcomes. A final report will be produced in September 2018. The final report will summarize findings documented in the interim report s and update findings based on new d ata. The report shall provide a full summary of lessons learned on the CCs’ efforts to build capacity

.7 

### PART A. JUSTIFICATIO **N 1.**  **Circumstances Necessitating the Information Collection** Title II of the Educational Technical Assistance Act of 2002 (Sect ion 204) requires that the National Center for Education Evaluation and Regional Assistance (NCEE), a component of the Department’ s Institute of Education Sciences (IES), provide for ongoing independent evaluation of the Comprehensive Centers. The statute establishes the following specific goals for the evaluation: to analyze the services provided by the Centers; to determine the extent to which each of the Centers meets the objectives of its’ respective plan; and to det ermine whether the services offe red b y each Center meet the educational needs of SEAs, local educational agencies (LEAs), and schools in the region. In October 2013, IES contracted with IMPAQ International to evaluate the Comprehensive Technical Assistance Centers

. 

### 2. **Purpose and Use of Information**  The evaluation will provide the Department of Education and the CCs with a formative report about how the CCs designed and carried out their work, and summative reports on program implementation and outcomes. In doing so, the evaluation seeks t o inform C C staff, the Department of Education, and the larger field about the capacity building strategies and outcomes of the C Cs’ work. The evaluation will inform ED’s staff as they support the CC and the CCs themselves as they implement their work. 

### 3. **Use of Information Technology** Electronic technology will be used whenever possible to reduce the time burden on respondents. TA recipient and C C staff surveys will be administered online using an automated survey administration and da ta collection system
. Online surveys are a proven, cost
- effective data collection methodology that take less time to complete than paper or telephone surveys

. In addition to enabling respondents to complete the survey at a time of their choosing, this method will allow th e project team to monitor the survey response rate in real time and send customized , timely reminder e mails. 

### 4. **Identification of Duplication of Information Collection Efforts** This study will yield unique data to evaluate the CCs. The evaluation will make use of data and documents already collected, produced , or maintained by the CC s and their local evaluators. Whenever possible, information will be collected and reviewed prior to interviews in order to avoid unnecessary interview questions. This stud y involves questions in surveys and interviews that are distinct from those asked by local evaluators. 

### 5. **Impacts on Small Businesses or Other Small Entities** One CC is operated by a small business entity, and several C C s have small business partners. The evaluation team will minimize burden on these entities by scheduling interviews at the convenience of staff and coordinating data collection schedules with local evaluators to avoid multiple collections

.8



### 6. **Consequences of Not Collecting the Data** F ailure to collect the data proposed through this study would breach the legislative mandate in Title II of the Educational Technical Assistance Act of 2002 (Section 204) that requires NCEE to provide for ongoing independent evaluation of the Comprehensive Centers. It would also p revent ED from gaining an in

- depth understanding of what ca pacity building strategies C Cs are using and the outcomes of the C Cs’ work. Understanding the strategies tha t the CC s implemen t and whether the CC s achieved their expected outcomes will enable federal policy makers and program managers to monitor the program and provide useful guida nce to CC s . 

### 7. **Special Data Collection Circumstances** None of the special circu mstances listed apply to this data collection. 

### 8. **Federal Register Notice**
**9. Payments/Gifts to Respondents**  There are no payments or gifts associated with this study. 

### 10. **Assurance of Privacy**  TA recipients and CC staff who are surveyed and interviewed will be assured that their responses will be kept private to the extent permitted by law and by the design of the evaluation . Survey data will be stored on the evaluation contractor’s server that is protected by a firewall that monitors and evaluates all attempted connections from the Internet. Personal information (name, telephone number, and e

- mail address) on each survey response will be maintained in a separate data file apart from the survey data so that individuals outside of the evalu ation team cannot link particular responses to individual respondents. Once the contract is completed, all personal information on each survey respondent will be destroyed. The entire survey database will be encrypted so that any data stored will be furthe r protected. Finally, access to any data with identifying information will be limited only to evaluation team members directly working on the survey. Survey findings will be presented at a level of aggregation such that it will not be possible to link spec ific responses to individual respondents. Everything that is discussed during interviews w ill be used only for the purposes of this study. We will not identify individual respondents by name , and will not share interview information directly with individu als outside of the research team. Findings from the interviews will be reported in summary form . Respondents’ role s and the CCs they work with may be identified in the report, however, which may lead to individuals’ (such as CC directors or SEA chiefs) bei ng identifiable

. We will make respondents aware of these conditions. I nterview notes or recordings will not be shared with ED staff or anyone else outside the study team. Paper copies of interview notes will be secured in a locked file cabinet. Electronic copies of notes will be stored in a SQL Server database located in the contractor’s access

- controlled server room.9



### 11. **Questions of a Sensitive Nature** There are no data of a sensitive, personal, or private nature being collected in the surveys or intervi ews. 

### 12. **Estimate of Annualized Burden Hours and Costs** Exhibit 1 shows the estimated burden hours and estimated monetary burden for respondents for each instrument to be administered in this study. Exhibit 2 displays the estimates of annualized costs by respondent type. Because of the site development work previously accomplished, and given that under their cooperative agreement the CCs are required to provide data for evaluation purposes, the evalu ation team anticipates a 100 percent response rate for the CC staff interviews and a 90 percent response rate for the CC staff surveys. Based on our experience, we expect an 80 percent response rate for the TA recipient surveys and an 8 0 percent response rate for the SEA staff interviews

. The CC staff survey will take approximately 20 minutes to complete, including the time for reading our introductory letter and direct ions

. The TA recipient survey will take about 15 minutes to complete. Interviews with Cen ter and SEA staff and will last about 60 minutes. The total burden hours are estimated at 1052
. To estimate the time required for each type of data collection, we consulted with fewer than 10 IMPAQ staff members or CC staff members. CC staff respondents are private sector employees in nonprofit or for

- profit companies, or in universities. TA recipient respondents are e mployees of a state department of education or a local education agency such as a school district.10

**Exhibit 1**  **: Estimated Burden Hours**

**Note:** The outcomes

-focused interview protocol s will be developed after the first site visit and are not included in this package. The burden table will be updated when the outcomes

-focused protocol s are sent for review. **Exhibit 2: Estimate of Annualized Costs by Respondent Type (per year for 3 years)**



### 13. **Estimates of Annualized Respondent Capital and Maintenance Costs** There are no direct costs to respondents other than that of their time of participation . There will be no start

- up or ongoing financial costs incurred by respondents. There are no record keepers. 

### 14. **Estimates of Annualized Cost to the Government** The est imated cost for this study, including development of a detailed study design and data collection instruments, management of a Technical Working Group, preparation of a justification package, data collection, data analysis, and report preparation, is $7,861 ,244 for the five years, or an average of $ 1,572,244 per year. Estimated cost includes staff and consultant time, travel, and operational expenses

. 

### 15. **Changes in Hour Burden** This request is for a new information collection. There are no changes in hour burden to report. Re sponde nt Ty pe To tal C ost Annua liz ed Cost CC Sta ff $22, 681 $7, 560 TA Re ci pi en t/SEA Sta ff $21, 384 $7, 128 **Tot al**  **$44, 065**  **$14, 688**  Da ta Col lect io n Act iv ity To ta l Sa mp le Si ze Es ti ma ted Resp ons e Ra te Nu mb erof Resp onde nts Ti me

Es ti ma te (i n hours ) Nu mb erof

Ad mi ni st r at io ns Tota l Hour

Bu rde n Hourl y

Ra te Es ti ma ted Mo neta ry Cos t of Burde n Desig n-Foc us ed I nter vi ews wi th CC s ta ff 88 100% 88 1 1 88 $45 $3, 960 Im pl emen ta ti on-Focus ed In ter vi ew wi th CC Staff 114 100% 114 1 2 228 $45 $10, 260 In ter vi ews w it h TA reci pi en ts 88 80% 70 1 2 141 $45 $6, 336 CC Staff Sur vey 264 90% 238 0. 33 3 235 $45 $10, 585 TA Rec ip ie ntSur vey 600 80% 480 0. 25 3 360 $45 $16, 200 **To ta**  **ls 1052**  **$47, 341** 11



### 16. **Plans for Tabulation and Publication 16.1**  **Tabulation** The timeline for project data collection and reporting is in Exhibit 3. Data collection will begin in January 2015 , or as soon as OMB approval is obtained, and conclude in the fourth quarter of 2017. No complex analytical techniques will be used. For quantitative survey and observation data, we will use descriptive statistics and analysis of variance. **16.2 Publication**  The evaluation will produce four reports (see Exhibit 3 for timeline) . The first report will be a formative report focusing on the program design qu estions
– how the Centers design ed their work as technical assistance providers . The report will describe the Centers’ underlying theories of action and definitions of capacity building, and explain how the Centers assess ed their constitue nt s’ needs and develop ed work plans to address those needs. This report will be available in November 2015.

Two interim summative reports (projected date: March 2017) will be produced to address the program implementation and outcomes qu estions for each of the two selected priority areas

. These reports will demonstrate how and to what extent the Comprehensive Technical Assi stance Center program has built state capacity in these priority areas. The reports will include descriptions of the strategies used to build SEA capacity, common challenges faced and ways the Centers sought to address them, the extent to which Centers ach ieved their goals and objectives, factors that may have contributed to success (or failure) in achieving expected outcomes, and the extent to which Centers’ outcomes aligned with and supported their theories of action. The summative reports will also inclu de six profiles of multi

-year projects, which will be selected and analyzed to illustrate how the Centers have successfully worked to build capacity related to project design and planning, implementation, and outcomes. Finally, a final report will be prod uced in September 2018. 

### 17. **Approval to N**  **ot Display the Expiration Date** The OMB approval number and expiration date will be displayed or cited on all information collection instruments. 

### 18. **Exceptions to the Certification Statement** There are no exceptions taken to item 19 of OMB Form 83

- 1.12 2013 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 AdministerTAre ci pient and CC st aff surve ys Conduct in te rv iews of TAre ci pi en ts and CC staff
Conduct obse rvati ons ofprof iled p rojectsa s opportuni ti es ari se Draft re port on Ce nte r d es ig ns/ the ori es of Ma y Fina l re port on Ce nte r d esig ns/the ori es of Nov Draft re ports on im pl em en tati on and outcome s Aug Fina l re ports on i mpl em en tati on and outcome s Ma r Draft f in al summati ve re port Apr Fina l summati ve re port Se p Summati ve re ports

Tas ks a nd Activities 2018 2014 2015 2016 2017 Ye ar 1

Surve ys (T

A re ci pi entan d CC staff)
Int ervi ews( SEA and CC staff)

Formati ve re port Obse rvati ons Ye ar 2 Ye ar 3 Ye ar 4 Ye ar 5 **Exhibit 3**  **: Timeline of Data Collection and Reporting**