## Supporting Statement **for Information Collection Request** **—**

**Part A National E**  **valuation of the**  **Inv esting in**  **Innov ation (i3)**  **Program July**  **24, 2014** Prepared for: **Tracy Rimdzius**  Institute of Education Sciences U.S. Department of Education 555 New Jersey Avenue, N W Wa shington, DC 20208

Submitted by: **Ab t A**  **ssociates** 55 W heeler Street Cambridge, MA 02138

### CONTENTS

**Abt Associ ates**  **National Evaluation of Investing in Innovation (i3) Program ▌**  **pg. i**  **A. Justification**  **................................ ................................**  **................................ ..............................**  **1** Introduction ................................ ................................ ................................ ................................ 1 A.1 Circumstances Requiring the Collection of Data ................................ ............................. 1 A.2 Purposes and Use of the Data ................................ ................................ ........................... 2 A.2.1 Goal One: Describe the intervention implemented by each i3 grantee ............... 3 A.2.2 Goal Two: Present the evidence produced by each i3 evaluation ....................... 3 A.2.3 Goal Three: Assess the strength of the evidence produced by each i3 evaluation ....... 3 A.2. 4 Goal Four: Identify effective and promising interventions ................................ . 4 A.2.5 Goal Five: Assess the results of the i3 Program ................................ .................. 4 A.2.6 Alignment of the data collection goals and the data elements ............................ 5 A.3 Use of Information Technology to Reduce Burden ................................ ....................... 11 A.4 Efforts to Identify Duplication ................................ ................................ ....................... 11 A.5 Small Businesses ................................ ................................ ................................ ............ 12 A.6 Consequences of Not Collecting the Information ................................ .......................... 12 A.7 Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6 ... 12 A.8 Consultation Outside the Agency ................................ ................................ ................... 12 A.8.1 Federal Register announcement ................................ ................................ ........ 12 A.8.2 Consultations Outside the Agency ................................ ................................ .... 13 A.9 Payments or Gifts to Respondents ................................ ................................ ................. 13 A.10 Assurance of Confidentiality ................................ ................................ .......................... 13 A.11 Questions of a Sensitive Nature ................................ ................................ ..................... 14 A.12 Estimates of Response Burden ................................ ................................ ....................... 14 A.13 Estimates of Total Capital and Start-Up Costs/Operation and Maintenance Costs to Respondents or Record Keepers ................................ ................................ .................... 16 A.14 Estimates of Costs to the Federal Government ................................ .............................. 16 A.15 Changes in Burden ................................ ................................ ................................ ......... 16 A.16 Plans for Publication, Analysis, and Schedule ................................ ............................... 16 A.16.1 Approach to i3 analysis and reporting ................................ .............................. 17 A.16.2 Plans for publication and study schedule ................................ .......................... 20 A.17 Approval to Not Display Expiration Date ................................ ................................ ...... 21 A.18 Exceptions to Item 19 of OMB Form 83 - 1 ................................ ................................ .... 21 **References ................................**  **................................ ................................**  **................................ ............**  **22**

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **1** 

## A. **Justification Introduction**  The Institute of Education Sciences (IES), within the U.S. Department of Education (ED), requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995 , clearance for IES to conduct data collection efforts for the National Evaluation of the Investing in Innovation (i3) Program. The i3 Program is designed to support school districts and nonprofit organizations in expanding, developing, and evaluating evidence -based practices and promising efforts to improve outcomes for the nation’ s students, teachers, and schools. Each i3 grantee is required to fund an independent evaluation, in order to document the implementation and outcomes of the educational practices. The National Evaluation of i3 (NEi3) has two goals: 1) to provide technical assistance (TA) to support the independent evaluators in conducting evaluations that are well - designed and well implemented and 2) to summarize the strength of the evidence and findings from the independent evaluations. Data collection will be required to address the second goal of summarizing the strength of the evidence and findings of the individual i3 evaluations being conducted by independent evaluators contracted by each i3 grantee. The required data collection will entail each i3 evaluator completing a Primary Survey; some evaluators will also complete an Early or a Follow -Up Survey (using the exact same survey template as the Primary Survey) . Part A of this request discusses the justification for these data collection activities. Part B , submitted under separate cover, describes the data collection procedures. 

### A. **1 Circumstances**  **Requiring the Collection of Data** To date, ED has awarded over $1 billion in funding through i3. In September 2010, ED awarded approximately $650 million in grants to the Fiscal Year (FY) 2010 cohort of i3 grantees to support the implementation and evaluation of 49 programs (4 Scale -Up grants, 15 Validation grants, and 30 Development grants) aimed at improving educational outcomes for schools, teachers, and students. The following year, ED awarded approximately $150 million to the FY 2011 cohort of i3 grantees, including 23 programs ( 1 Scale -Up grant, 5 Validation grants, and 17 Development grants). Then, in December 2012, ED awarded approximately $140 million in grants to the FY 2012 cohort, including 20 programs ( 8 Validation grants and 12 Development grants). Finally, in December 2013, ED awarded approximately $112 million in grants to the FY 2013 cohort, including 25 programs ( 7 Validation grants and 18 Development Grants). The Scale -Up grants are intended to support taking to scale programs that have strong evidence of their effectiveness in some settings. The Validation grants were given to programs to validate their effectiveness, and the Development grants were given to programs to establish the feasibility of implementation of the program and test the promise for improving outcomes . The i3 Program, overseen by ED’s Office of Innovation and Improvement (OII), requires each grantee to fund an independent evaluation in order to document the implementation and outcomes of the educational practices. While ED’s goal was always that grantees fund evaluations that estimate the impact of the i3 - supported practiced (as implemented at the proposed level of scale) on a relevant outcome , this was explicitly stated as of the FY 2013 i3 competition. The Scale -Up and Validation grantees are expected to fund rigorous impact studies using experimental or quasi - experimental designs (QEDs). These evaluations have the potential to yield strong evidence about a range of approaches to improving educational outcomes. The Development grants, in turn, are also supporting independent evaluations, but

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **2** are not necessarily expected to support rigorous evaluations. ED’s Institute of Education Sciences (IES) has contracted with Abt Associates and its partners 1 (the i3 Evaluation Team) to conduct the NEi3. Achieving the first goal of the NEi3 requires one group within the i3 Evaluation Team, called the Technical Assistance (TA) Team, to take an active role in maximizing the strength of the design and implementation of the evaluations, so that the evaluations yield the strongest evidence possible about what works for whom in efforts to improve educational outcomes. Achieving the second goal of the NEi3 requires a separate group within the i3 Evaluation Team, known as the Analysis and Reporting (AR) Team, to ( 1) assess the evaluation designs and implementation, both to inform ED of the progress of the evaluations 2 and to provide important context for the summary of results, and ( 2) provide clear synopses of what can be learned from this unprecedented investment in educational innovation

—summaries that are cogent and easily understood by a wide range of stakeholders, including educators and school administrators, p olicymakers, ED, and the public. To ensure that the work of the TA Team does not influence the assessments and analysis of the AR Team, the two groups of staff members do not overlap, and are separated by a firewall maintained by the Project Director. The systematic data collection proposed for the NEi3 is necessary to provide the information for the AR Team to review and summarize the findings from the independent evaluations funded by i3. The data collection will be led by the AR team, with oversight from the Project Director. 

### A. **2 Purposes and Use of the**  **Data** This section of the supporting statement provides an overview of the research design and data collection efforts planned to meet the main research questions and overall objectives of the NEi3 . The section begins with an overview of the research design, including the main objectives of the evaluation and key research questions, followed by a description of the data collection activities for which OMB clearance is requested. As noted earlier, the main objectives of the NE i3 are to ( 1) to provide technical assistance (TA) to support the independent evaluators in conducting evaluations that are well - designed and well - implemented and (2) to summarize the strength of the evidence and findings from the independent evaluations. The data collected for this evaluation will address this second objective . Specifically, the data will be used to support reviews and reports to ED that have five goals : 1. Describe the intervention implemented by each i3 grantee. 2. A ssess the strength of the evidence produced by each i3 evaluation. 3. Present the evidence produced by each i3 evaluation, along with assessment of its strength . 4. Identify effective and promising interventions. 5. Assess the results of the i3 Program.

1 The partners are Dillon- Goodson Research Associates, Chesapeake Research Associates, ANALYTICA, Westat, Century Analytics, and American Institutes for Research. 2 It is important to note that the assessment of whether the independent evaluations are well - designed and we ll im plemented forms the basis for some of the Government Performance and Results Act (GPRA) measures for the i3 program.

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **3** These five goals are described in more detail below , along with the research questions that address each goal . **A. 2.1**  **Goal One: Describe the intervention implemented by each i3 grantee**  While the programs are described in the applications submitted by the grantees for funding, it will be important to augment these descriptions with more detailed information than could be provided within the page limits of the applications and to reflect what happened (as opposed to what was planned during the application stage ). This in formation will allow the i3 Evaluation Team to describe key components of each i3 - funded intervention

—for example, in the approach taken to training teachers and delivering instruction to students. An accurate description of the details of each project’s intervention is critical to ED’s understanding of the breadth and depth of the approaches funded, and of the implications for what these specific projects can potentially teach the field about promising approaches to education reform. This goal will be ful filled by answering the following research questions: Q1. What are the key components of the intervention? Q2. What are the student and teacher outcomes that the intervention is designed to affect? Q3. What are the mediators through which the intervention is expected to affect student and teacher outcomes? **A. 2.2**  **Goal Two: As**  **sess the strength of the evidence produced by each i3 evaluation** Periodic assessments of the i3 independent evaluations are essential to the NEi3 . In addition to providing ED with information necessary to report key Government Performance and Reporting Act (GPRA) measures, these assessments provide important context for the interpretation of the findings generated by the i3 independent evaluations and summarized in the reports. Specifically, ED has explicitly articulated the expectation that Scale -Up and Validation grantee s conduct evaluations that are “well - designed and well - implemented,” as defined by the standards of the What Works Clearinghouse (WWC) , because findings generated by such studies provide more convincing evidence of the effectiveness of the programs than evaluations that do not meet this standard. ED expects evaluations of Development grants to provide evidence on the intervention’s promise for improving student outcomes . 3 This goal will be fulfilled by answering the following research questions: Q 4 . For each i3 grant, how strong is the evidence on the causal link between the intervention and its intended student or teacher outcomes? Q 4 a. F or Scale -Up and Validation grants , did the evaluation provide evidence on the effects of the intervention?

3 The i3 Promise criteria operationalize OII’s GPRA requirements, which distinguish between Development and Validation/Scale -Up grants in their evidence requirements. These Promise criteria were developed to assess the quality of evaluations of Development grants that do not meet WWC evidence standards (with or without reservations ). These criteria are intended to determine the extent to which the interventions funded by Development grants provide evidence of “promise” for improving student outcomes and may be ready for a mo re rigorous test of their effectiveness.

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **4**

Q 4b. F or Development grants, did the evaluation produce evidence on whether the intervention warrants a more rigorous study of the intervention’s effects ? For Development grants, if applicable, did the evaluation provide evidence on the outcomes of the intervention? **A. 2.3**  **G oal Three**  **: Present the evidence produced by each i3 evaluation**  Beginning in Spring 2016 , we will present the available findings from both the implementation and impact studies as they emerge from the i3 evaluations. In particular, we will report on the findings on implementation fidelity and the findings from the impact analysis. The NEi3 will report the inf ormation provided by the independent evaluators for each i3 intervention ( after verif ying and processing those data) . This goal will be fulfilled by answering the following research questions: Q5 . For each i3 grant, how faithfully was the intervention imp lemented? Q6 . For each i3 grant, what were the effects of the interventions on /promise of the interventions to improve educational outcomes? **A. 2.4**  **Goal Four: Identify effective and promising interventions**  To help school and district officials identify effective interventions, we will produce lists of interventions with positive effects/outcomes . Since educational interventions are typically designed to improve outcomes in one or more domains, and are targeted for students at particular grade levels, we will list interventions by outcome domain and educational level (e.g., reading achievement for students in elementary school). Each list will include the names of the i3 - funded interventions that produced positive effects, based on i3 studies that meet WWC evidence standards (with or without reservations) . 4 This goal will be fulfilled by answering the following research questions: Q7. Which i3 - funded interventions were found to be effective at improving student or teacher outcomes? Q8. Which i3 - funded interventions were found to be promising at improving student or teacher outcomes? **A. 2.5**  **Goal Five: As**  **sess the results of the i3 Program** To inform federal policymakers, we propose to summarize the effectiveness of the i3 - funded interventions. This goal will be fulfilled by answering the following research questions: Q9. How successful were the i3 - funded interventions? Q9a. What fraction of interventions was found to be effective or promising?

4 See http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_draft_standards_handbook.pdf for WWC evidence standards.

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **5**

Q9b. What fraction of interventions produced evidence that met WWC evidence standards (with or without reservations) and i3 criteria for providing credible evidence of the intervention’s promise for improving e ducational outcomes)? Q9c. What fraction of interventions produced credible evidence of implementation fidelity? **A. 2.6**  **Al ignment of the**  **data collection goals**  **and the data elem ents**  As stated above, the NE i3 has nine research questions that address the five major goals of the evaluation . Exhibit 1 presents a crosswalk of the purposes of the data collection, the research questions , and the individual survey items . Appendix A contains the full data collection survey .

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **6 Exhibit**  **1 .**  **Cross walk A** **ligning Goals, Research Questions, and Survey Items**  **Goal 1: Describe the intervention implemented by each i3 grantee.**  Research Question Survey Items ( Sections and Question Numbers) What are the key components of the intervention? IMPLEMENTATION : A & B , BACKGROUND: 10 What are the student or teacher outcomes that the intervention is designed to affect? IMPACT : C, D , BACKGROUND: 10 What are the mediators through which the intervention is expected to affect student or teacher outcomes? IMPACT: C, D ; IMPLEMENTATION : A & B **Goal 2**  **: Assess the strength of the evidence produced by each i3 evaluation.** Research Question Survey Items (Sections and Question Numbers) For each i3 grant, how strong is the evidence on the causal link between the intervention and its intended student or teacher outcomes? Was the evaluation well - designed and well - implemented (i.e., for Scale - Up and Validation grants, did the evaluation provide evidence on the effects of the intervention)? Di d the evaluation provide evidence on the intervention’s promise for improving student or teacher outcomes (i.e., for Development grants, did the evaluation produce evidence on whether the intervention warrants a more rigorous study of the intervention’s ef fects )? BACKGROUND : 1 - 3, 5 - 9, 11 - 16; IMPACT : A- BS

**Goal 3**  **: Present the evidence produced by each i3 evaluation.** Research Question Survey Items (Sections and Question Numbers) For each i3 grant, how faithfully was the intervention implemented? FIDELITY: C - V For each i3 grant, w hat were the effects of the interventions on educational outcomes? CONTRAST: BL - CD **Goal 4: Identify effective and promising interventions.**  Research Question Survey Items (Sections and Question Numbers) Which i3 -funded interventions were found to be effective at improving student or teacher outcomes? IMPACT : A- BS; BACKGROUND : 1 - 3, 5 - 9, 11 - 16 Which i3 -funded interventions were found to be promising at improving student or teacher outcomes? IMPACT : A- BS; BACKGROUND : 1 - 3, 5 - 9, 11 - 16 **Goal 5: Assess the results of the i3 Program.**  Research Question Survey Items (Sections and Question Numbers) How successful were the i3 -funded interventions? What fraction of interventions was found to be effective or promising? What fraction of interventions produced evidence that met WWC evidence standards and i3 criteria? What fraction of interventions produced credible evidence of implementation fidelity? BACKGROUND 1 - 16; I MPACT A- BS; IMPLEMENTATION : A- Q

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **7 A.2.**  **6 .1**  **Types of data to be collected each year** Data on the i3 evaluations’ designs, activities, and findings will be collected annually beginning in January 20 15 from the evaluators of all of the i3 projects that have findings available that the evaluator determines are ready to be shared with the NEi3 (because these findings are final an d will not be subject to future changes) at the time of our year ly data collection . The types of information that will be available from evaluators each year will vary across the i3 projects depending on the duration of their funding (which ranges from 3 to 5 years) and the cohort to which they belong ( FY 2010, FY 2011, FY 2012 or FY 2013) . Each i3 evaluator will be required to participate in at least one of the five annual data collection periods planned by the NEi3. Three of these data collections are covered by this clearance package, and the remaining two will be included in a future OMB clearance package. While evaluators are permitted to submit data during each of the f ive annual data collection periods, we anticipate that evaluators will submit most of their data during a single period. We refer to this as the evaluators’ Primary Data Collection. We also expect that some evaluators may choose to submit a limited number of early findings prior to that period or additional findings after that period. We refer to these periods as evaluators’ Early Data Collection and Follow -Up Data Collection, respectively. During each grant’s Primary Data Collection (and Early Data Collection , for those grants that choose to submit early findings) , we will ask the independent evaluator t o complete a survey sharing all findings available to date. In subsequent years, we will follow up with evaluators to request any changes/updates to their e valuations by completing a Follow -Up Survey . The Follow -Up Survey will use the same exact survey template as the Primary Survey. Prior to any data collection, we will pre -populate surveys for evaluators based on extant data they share with us (existing design documents , existing reports, conference presentations, etc.) and documents produced to fulfill requirements of ED’s cooperative agreements with the grantees . E ach i3 grant will undergo the following process on a timeline that is unique to its cohort and its award length (i3 grants were funded for periods of anywhere between three and five years), but will include the same key components . **For each grant, we will: 1)**  **Review documents produced by evaluators to fulfill requirements of ED’s cooperative**  **a greements with the grantees**  **. 2)**  **Collect extant data on the study’s finding** **s during the later years of the grant when**  **findings are available (starting in January 2015) .**  **3) Use documents described in 1) and 2) to pre**  **- populate a Primary Survey**  **. 4)**  **Send the evaluator a link to a customized, pre**  **- populated Primary Survey asking them to**  **a) actively confirm pre -**  **populated information and b) enter missing data .**  **5) If needed (because the evaluator produces findings**  **after completing a Primary Survey),**  **we will repeat steps 2), 3) and 4) for data related to the additional findings only**  **.**

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **8** Exhibit 2 , panel 1, below provides a summary of our report release dates associated with each of the three data collections described in this clearance package and the types of data to be collected during Early, Primary and Follow -Up Data Collections ; panel 2 details when we expect each cohort will participate in Early, Primary , and Follow -Up Data Collection s covered in this OMB package . **Exhibit 2: Data Collection and**  **Reporting Plan**  **Overview Panel 1: Description of Reports and Data Collection**  **NEi3 Report Type of**  **Data Collection**  **Timing of**

**Data Collection**  **Type of Data to be Collected NEi3 Findings**  **Report (**  **Expected Release Date:**  **Spring 2016 )**   Describe the interventions as designed and implemented;  Describe the evaluations as implemented;  Assess the strength of the evaluations (as designed and implemented);  Present impact and implementation findings; **Early and Primary**  **Data Collection** January–

March 2015 **Early and Primary Data**  **Collection**

 Characteristics of interventions .  Details of evaluation designs and plans for evaluations .  Details about the implementation of the evaluation to date (e.g., sample selection and assignment, characteristics of measures being used.)  Details of the implementation of the evaluation (e.g., attrition, analytic approach employed) .  Findings from implementation and outcomes studies to date. **Report Addenda ( Expected**  **Release Dates: Spring 2017, Spring 2018**  **)**  Provide updates to prior version of the NEi3 report to reflect additional findings reported by evaluators . **Early, Primary**  and **Follow -**  **Up Data Collection**  January–

March , annually from 201 5 - 20 17 **Early and Primary Data Collection.**  The

NEi3 will collect the data listed i n the five above bullets from additional i3 -funded studies that have not previously participated in Data Collection. **Follow -**  **Up Data Collection.** The NEi3 will collect the following information from grants that have already participated in a Primary Data Collection an d have additional findings to share .  Updates of information collected previously, as needed;  Data on newly available implementation and outcome study fi ndings **Panel 2: Timing and Type of Data Collection by Cohort**

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **9 A.2.6**  **. 2**  **Early and Primary Data Collectio** **n —**  **Information on study design to be pre -**  **populated b**

**y NEi3**  At the time o f Primary D ata C ollection, NEi3 team members will pre -populate the following information into the Primary Survey (see Appendix A) for all i3 independent evaluators . NEi3 team members will us e documents produced by evaluators to fulfill requirements of ED’s cooperative agreements with the grantees .  Information on interventions , including:

 Descriptions of the interventions .

 Roles played by the people and organizations involved with the grant .

 People or a spects of educational organizations that the intervention aims to effect or benefit .

 Logic models of the inputs, mediators, and expected outputs of interventions. (For Scale -Up grants this includes a logic model of the mechanisms and pathways of the scale -up process ) .

 Key elements of intervention implementation .

 Scale of the intervention (in terms of the number of students, teachers, schools , and/or districts intended to be served).  Evaluation plans, including : 5

 Research questions (specified as confirmatory and exploratory) .

 Research design (i.e., randomized control led trials, quasi - experimental designs with comparison groups, regression discontinuity designs, or other types of designs) .

 Treatment and counterfactual conditions (specifically, how the counterfactual differs from the intervention) .

 Characteristics of the potential intervention versus comparison group contrasts to be analyzed .

 Plans for measuring fidelity of intervention implementation .

 Outcomes or dependent variables (e.g., name, timing of measurement for the intervention and the comparison groups , reliability and validity properties , whether outcome data were imputed in the case of missing data, and a description of the measurement instrument if not standard) .

 Independent variable s (e.g., name, timing of measurement, and reliability and validity properties) .

 Analysis methods including descriptions of how impacts in the full sample and for subgroups will be estimated, how the baseline equivalence will be established (in QEDs and RCT s with high attrition), and how clustering of participants and noncompliance to treatment will be accounted for .

 Role of the developers or grantee in the conduct of the evaluation (to assess the independence of the evaluators) .

5 We expect that some projects will use more than one research design. In order to be clear , in this plan we refer only to a single design per evaluation .

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **10**  Description of the study s ample, including:

 Group sizes (overall and per group) .

 Characteristics (e.g., grades or age levels) .

 Location . **A.2.6 .**  **3 Early and**  **Primary Data Collection –**  **Informat ion on study findings to be pre** **- populated**  **by NEi3 whenever possible based on existing reports, conference presentations, etc. shared**  **with the NEi3by i3 independent**  **evaluators** Prior to Primary D ata C ollection, NEi3 team members will pre -populate surveys with the following information for all i3 independent evaluators based on any reports and other documents i3 independent evaluators share with us.  Evaluation findings from the implementation study, including:

 Revised description of the intervention components .

 Level of implementation fidelity .

 Impacts on intermediate outcomes (e.g., teacher knowledge) .  Evaluation findings from impact study for each contrast tested in the full sample or subgroups, plus information that is helpful in interpreting the impact findings, including:

 Impact estimates .

 Standard errors (to allow the NEi3 study team to conduct tests of statistical significance) .

 Standard deviation of the outcome measures (to allow the NEi3 study team to construct effect size measures) .

 Means and standard deviations for baseline variables (to allow the NEi3 study team to test for baseline equivalence) .

 Attrition rate and analysis sample sizes by group for each contrast . **A.2.6 .**  **4 Early and**  **Primary Data Collection –**  **Inform ation to be completed by eva** **luators** Any of the above data that are not available in existing reports will be collected from evaluators via our Primary Data Collection. Evaluators will only be asked to complete Primary Survey items that the NEi3 was not able to pre -populate based on the prior steps. Evaluators will be asked to complete all missing fields, and to actively verify the accuracy of data pre -populated by the NEi3. **A.2. 6**  **. 5**  **Follow**

**- Up**  **Data Collection** During F ollow -Up D ata C ollection, information collected previously will be updated as necessary. Again, we will request existing reports, conference presentations, etc. for use in pre -populating the Follow -Up Survey. Specifically, we will reques t information about any recent findings (not already reported in previous years ) as well as the associated information required for interpreting them. This information will be the same as the information requested during the Primary Data C ollection, except that it will be updated with analyses conducted following the Primary Data C ollection. As previously mentioned, the Follow -Up Survey will use the same exact survey template as the Primary S urvey.

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **11 A.2.**  **6 .**  **6 Data collection procedures and timeline**  The data c ollections covered in this OMB clearance package will occur from January through March of 2015 , January through March of 2016 , and January through March of 2017 . The type of data collection varies between Early, Primary and F ollow -Up D ata C ollection, depending on the cohort and grant (see exhibit 2, panel 2, for more information) . **Exhibit 3: Overview of data**  **collection timelines covered in this OMB Package**  **Data Collection** January– March 2015 January– March 2016 January– March 2017 

### A. **3 Use of Information Technology**  **to Reduce Burden**  The data collection plan for the NE i3 reflects sensitivity to issues of efficiency, accuracy, and respondent burden in several ways. Data collection will take place in a two - stage process in which ( 1) the AR team will pre - fill surveys for evaluators as much as possible based on evaluation documents previously received and ( 2) independent i3 evaluators will respond to a web -based Primary , Early or Follow -Up Survey that includes this pre - filled information for their review and new survey items to respond to that are appropriate for the timeline of their evaluation. This two - stage process will minimize the reporting burden on independent evaluators. In addition, we will reduce burden on evaluators by using web -based data portals. Data will be collected using a combination of Microsoft Word-type survey items and Microsoft Excel -typ e tables and templates on secure website portals accessible only by individual i3 evaluation teams. The survey is designed so that respondents can complete part of the survey, check their records or research answers to questions about which they are unsure , and then complete the survey at a later time. An evaluation - specific username and password will be required each time the survey is accessed. No persons other than those selected to receive usernames and passwords and the study team will have editing acc ess to the website. The study contractor will track completed surveys in real time. 

### A. **4 Efforts to Identify Duplication**  Potential duplications of effort are of two general types: addressing research questions already answered and duplicating data collection. Th e NEi3 will not address research questions already answered . As explained in Section A.1., the i3 grant s were newly funded in FY 2010 , FY 2011, FY 2012 , and FY 2013 and this evaluation is the only federally funded information collection with plans to collect data in order to assess the extent to which the i3 independent evaluations are well - designed and well - implemented , and report the results across the evaluations. To ensure we do not duplicate data collection efforts, independent evaluator s will be encouraged to send study design documents, reports, summaries, and information tables that were constructed in their own work to the AR team prior to the survey each year. These documents will be reviewed by the AR team and information from the documents will be used to pre - fill survey item responses and sections that can be reviewed by the independent evaluator

— rather than asking the evaluators to provide information that has already been reported in existing evaluation documents.

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **12** 

### A. **5 S**  **mall Business e**  **s** The primary entities for this study are independent evaluators employed by both large and small businesses as well as in some universities. Every effort is being made to reduce the burden on the evaluators through the collection of study design documents , reports, summaries, and information tables that evaluators constructed in their own work, a s well as the use of a web -based data collection. The specific plans for reducing burden are described in Section A.3. 

### A. **6 Consequences of Not Collecting the Informati**  **on** The data collection described in this supporting statement is necessary for conducting this evaluation, which is consistent with the goal s of the Investing in Innovation program to identify and document best practices that can be shared and taken to scale based on demonstrated success (American Recovery and Reinvestment Act of 2009 (ARRA), Section 14007(a)(3)(C)) . The information that will be collected through this effort is also necessary to report on the performance measures of the i3 Program, required by the Government Performance and Results Act : **Long -**  **Term Performance Measures**  The percentage of programs, practices, or strategies supported by a [ Scale -Up or Validation] grant that implement a completed well - designed, well - implemented and independent evaluation that provides evidence of their effectiveness at improving student or teacher outcome s .  The p ercentage of programs, practices, or strategies supported by a [ Development] grant with a completed evaluation that provides evidence of their promise for improving student or teacher outcomes .  The percentage of programs, practices, or strategies supported by a [ Scale -Up or Validation] grant with a completed well - designed, well - implemented and independent evaluation that provides information about the key elements and the approach of the project so as to facilitate replication or testing in other settings .  The percentage of programs, practices, or strategies supported by a [ Development] grant with a completed evaluation that provides information about the key elements and the approach of the project so as to facilitate further development, replication or testing in other settings . 

### A. **7 Special Circumstances**  **Justify ing Inconsistencies with Guidelines**  **in 5 CFR 1320.6**  There are no special circumstances required for the collection of this information. 

### A. **8 Consultation Outside the A** **gency A.**  **8.1 Federal Register**  **announcement** In accordance with the Paperwork Reduction Act of 1995 (Pub.L. No. 104 - 13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), IES published a notice in the Federal Register announcing the agency’s intention to request an OMB review of data collection activities for the i3 Evaluation. The notice was published on (date) in volume xx, No. xx, page xx and provided a 60 - day period for public comment .

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **13 A.**  **8.2 Consultations**  **Outside the A gency**  The Abt study team assembled a Technical Work Grou p (TWG) (in consultation with ED) that consists of consultants with various types of expertise relevant to this evaluation. The TWG convened in April 2011 and discussed the overall approach to the NEi3, including providing TA and data collection and reporting . Individuals who served on the expert panel a re listed in Exhib it 4. **Exhibit 4**  **: Expert Panel Members Name**  **Af filiation**  David Francis , Ph.D. University of Houston To m Cook , Ph.D. Northwestern University Brian Jacob , Ph.D. University of Michigan Lawrence Hedges, Ph.D. Northwestern University Carolyn Hill, Ph.D. Georgetown University Chris Lonigan, Ph.D. Florida State University Neal Finkelstein, Ph.D. We stEd Bob Granger, Ph.D. W. T. Grant Foundation 

### A. **9 Pay**  **ments or Gifts to Respondents** Data collection for this study does not involve payments or gifts to respondents. 

### A. **10 As**  **surance of Confidentiality** Abt Associates and its subcontractors follow the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183), which require that all collection, maintenance, use and wide dissemination of data conform to the requirements of the Privacy Act of 1974 (5 U.S.C. 552a), the Family Educational Rights and Privacy Act of 1974 (20 U.S.C. 1232g), and the Protection of Pupil Rights Amendment (20 U.S.C. 1232h). The study team will not be collecting any individually identifiable information and Abt Associates and its subcontractors will protect the confidentiality of all information collected for the study and will use it for research purp oses only. All study staff involved in collecting, reviewing, or analyzing study data will be knowledgeable about data security procedures. The privacy procedures adopted for this study for all data collection, data processing, and analysis activities include the following:  The study will not request any personally identifiable information (PII) data that was collected by independent evaluators. All of the data requested from independent evaluators will be in the form of aggregated reports of the methods, measures, plans, and results in their independent evaluations.  Individual i3 grant s will be identified in this study. Their characteristics, results, and findings reported by independent evaluators, as well as assessments of the quality of the independent evaluations and i3 projects , may potentially be reported. This study cannot, however, associate responses with a specific school or individual participating in i3 independent evaluations in annual reports , since we are not collecting that information .

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **14**  To ensure data security, all individuals hired by Abt Associates Inc. are required to adhere to strict standards and sign an oath of confidentiality as a condition of employment. Abt’s subcontractors will be held to the same standards.  All data files on multi -user systems will be under the control of a database manager, with access limited to project staff on a “need-to -know” basis only.  Identifiable data will be kept for three years after the project end date and then destroyed.  Written records will be shredded and electronic records will be purged. 

### A. **11 Questions of a Sensitive Nature**  The data collection for this study do es not include any questions of a sensitive nature. 

### A. **12 Estimate**  **s of Response Burden**  B urden for the data collection covered by this clearance req uest is 1,5 07 hours , for a total cost to respondents of $56 , 449 . Exhibit 5 presents time estimates of response burden for the data collection activities requested for approval in this submission. The burden estimates are based on the f ollowing assumptions:  The majority (80 percent) of evaluators will submit existing materials (reports, conference presentations, etc.) reporting their findings to the AR team.  The AR team will pre - fill survey item responses for evaluators with as much inf ormation as possible based on the documents they receive from evaluators reporting their findings and documents produced by evaluators to fulfill requirements of ED’s cooperative agreements with the grantees .  Evaluators will respond to a web -based Primary (and possibly also an Early or Follow -Up) Survey that includes pre - filled information for their review and new survey items . Estimated hourly costs to independent evaluators are based on an average hourly wage for social scientists and related workers of $37.45 according to The United States Department of Labor, Bureau of Labor Statistics (U.S. Department of Labor 20 09).

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **15 Exhibit 5. Estimate of Respondent Burden** 

**Data Collection Wave and Activity**  **Total Number of**  **Expected Responses**  **Total**

**Burden Hours**  **per Response**  **Total**

**Burden Hours**  **Hourly Cost**

**per Response f**  **Total**

**Costs P**  **rimary Data Collection : Respond to Survey**  **a 92**  **- -**  **- -**  ... Respondents who submit extant data on findings b 74 7.910 585.3 $ 37.45 $21,921 ...Respondents who do NOT submit extant data on findings b 18 26.333 474.0 $ 37.45 $17,751 **Early or Follow -**  **Up Data Collection : Respond to Survey**  **c 38**  **- -**  **- -**  ... Respondents who submit extant data on findings d 30 7.910 237.3 $ 37.45 $8,887 ...Res p ondents who do NOT submit extant data on findings d 8 26.333 210.7 $ 37.45 $7,889

**Total Number of Responses 130**  **- 1507**  $ 37.45 **$56,449 Total Number of Respondents**  **e** 117 **Average Number of Responses per Respondent** 1.11 **Overall Average Burden Hours per Respondent 13**  **Overall Average Cost per Respondent $482.47**  Notes: This OMB Package covers the period from 1/1/2015 through 12/31/2017. Therefore it includes all data collection activities for the 49 FY2010 grants and 23 FY2011 grants, Early and Primary Data Collection for the 20 FY2012 grants and Early D ata Collection for the 25 FY2013 grants. Responding to the survey (during Primary Data Collection, Early and Follow- up Data Collection) includes reviewing data prepopulated by the NEi3 based on extant data on findings submitted to the NEi3 and confirming a ny information pre - populated by the NEi3 based on review of documents produced by evaluators to fulfill requirements of ED’s cooperative agreeme nts with the grantees. aWe expect 92 grants (all 49 FY2010 grants, all 23 FY2011 grants, and all 20 FY2010 grants) to participate in Primary Data Collection during the period covered by this OMB Package. bWe expect approximately 80 percent (74) of the 92 Primary Data Collection par ticipants to submit extant data prior to their Primary Data Collection. c We expect 38 grants [40 percent (20) of the FY2010 grants, 40 percent (9) of the FY2011 grants, 20 percent (4) of the FY2012 grants, and 20 percent (5) of the FY2013 grants] to parti cipate in Early o r Follow- Up Data Collection. d We expect approximately 80 percent (30) of the 38 Early or Follow- up Data Collection participants to submit extant data prior to their Early or Follow - up Data Collection. e D uring the period covered by this OMB package, we expect a total of 117 survey respondents (49 FY2010 grants, 23 FY2011 grants, 20 F Y2012 grants and 25 FY2013 grants). This number differs from the number of responses because some of these grants will participate in Primary and Early or Follow- up Data Collection. Other grants (FY2013) will only participate in Early Data Collection. f Average hourly wage for “Social Scientists, and Related Workers, All Others” from the Industry - Specific Occupational an d Wage Estimates, U.S. Department of Labor (see http://www.bls.gov/oes/current/oes193099.htm#nat), accessed May 24, 2011

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **16** 

### A. **13 Estimate**  **s of Total Capital and Start**  **- U**  **p Costs/Operation and Maintenance Costs to Respondents or Record**  **Keepers** There are no annualized capital , start -up , or ongoing operation and maintenance costs involved in collecting this information. 

### A. **14 Estimates of Costs to the Federal Government**  The estimated cost to the Federal Government of the activities across all of the NEi3 contracts (FY2010/FY2011 cohort - contract no: ED - IES - 10 - C - 0064 , FY2012 cohort - contract no: ED - IES - 13 - C 0005 , and FY2013 cohort

– contract number ED - IES - 14 - C - 0007) is $22, 322 , 122 for the entire NEi3, including work on all contracts and tasks. These activities began in October, 2010 and will continue until February, 2020 . Thus, the average annual cost to the federal government is approximately $2,391 ,3 66 . 

### A. **15 Changes in Burden**  This is a request for a new collection of information . 

### A. **16 Plans for Publication, A**  **naly sis, and Sch**  **edule** In this section, we present our approach to analyzing the data collected in order to address the five goals of the data collection (and related research questions) introduced in Section A.2: (1) Describe the intervention implemented by each i3 grantee; (2) 

### A ssess the strength of the evidence produced by each i3 evaluation; (3) Present the evidence produced by each i3 evaluation; ( 4) Identify effective and promising interventions; and (5) Assess the results of the i3 Program. After describing our appro ach to addressing the five goals , we discuss our plans for reporting the results. The five goals and research questions we plan to address over the course of our evaluation , presented in Section A.2 above, are reiterated in Exhibit 6.

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **17 Exhibit**  **6 :**  **NEi3’s Goals and**  **Research Questions Goal 1: Describe the intervention implemented by each i3 grantee.**  Q1. Wh at are the key components of the intervention? Q2. Wh at are the student or teacher outcomes that the intervention is designed to affect? Q3. Wh at are the mediators through which the intervention is expected to affect student or teacher outcomes? **Goal 2 : A** **ssess the strength of the evidence produced by each i3 evaluation.** Q 4 . For each i3 grant, how strong is the evidence on the causal link between the intervention and its intended student or teacher outcomes? Q 4 a. For Scale - Up and Validation grants, did the evaluation provide evidence on

the effects of the intervention? Q4b. F or Development grants, did the evaluation produce evidence on whether

t he intervention warrants a more rigorous study of the intervention’s effects ? For Development grants, if applicable, did the evaluation provide evidence on the outcomes of the intervention? **G oal 3**  **: Present the evidence produced by each i3 evaluation.** Q5 . For each i3 grant, how faithfully was the intervention implemented? Q6 . For each i3 grant, w hat were the effects of the interventions on/promise of the interventions to improve educational outcomes ? **Goal 4: Identify effective and promising interventions.**  Q7. Wh ich i3 - funded interventions were found to be effective at improving student or teacher outcomes? Q8. Wh ich i3 - funded interventions were found to be promising at improving student or teacher outcomes? **Goal 5: A ssess the results of**  **the i3 Program.** Q9. How successful were the i3 - funded interventions? Q9a Wh at fraction of interventions was found to be effective or promising? Q9b. Wh at fraction of interventions produced evidence that met WW C evidence

standards and i3 criteria? Q9c. Wh at fraction of interventions produced credible evidence of implementation

fidelity? **A. 16.1**  **Ap proach to i3**  **analysis and re porting**  The nature of our analysis and reporting is to synthesize and review findings, rather than to estimate impacts using statistical techniques. In this section, we provide more information on how we plan to address the research questions posed for the NEi3 by analyzing the data we are requesting permission to collect.

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **18 Goal 1: Describe the intervention implemented by each i3 grantee.**  Q1. Wh at are the key components of the intervention? Ke y com ponents are defined as the activities and inputs that are under the direct control of the individual or organization responsible for program implementation (e.g., program developer, grant recipient), and are considered by the developer to be essenti al in implementing the intervention. Key components may include financial resources, professional development for teachers, curricular materials, or technology products. W e will review the evaluators’ reported key components, possibly rephrase them to ma intain their original meaning but to be consistent with other key com ponents named across other evaluations, and report them in a succinct manner to facilitate easing “browsing” of activities across interventions. Q2. Wh at are the student or teacher outcomes that the intervention is designed to affect? Wewill report the student or teacher outcome domains that were evaluated for each grant. W e m ay

rephrase the outcome domains to maintain their original meaning but to be consistent with ou tcome domains named in other interventions, for example we ma y use “Mathematics” to describe outcome domains labeled by evaluators as “Math Achievement”, “Mathematics Achievement”, “Mathematical Understanding”, etc. Q3. Wh at are the mediators through which the intervention is expected to affect student or teacher outcomes? Wewill report the me diators through which the intervention is expected to affect outcomes. Wem ay rephrase the me diator names to maintain their original meaning but to be consisten t with me diators nam ed in other interventions, for example we ma y use “ Student Engagement ” to describe me diators labeled by evaluators as “ Student Participation ”, “ Student Commitment ”, etc. **Goal 2 : A** **ssess the strength of the evidence produced by each i3 evaluation.** Q4 . For each i3 grant, how strong is the evidence on the causal link between the intervention and its intended student or teacher outcomes? Q4 a. For Scale - Up and Validation grants, did the evaluation provide evidence on

the effects of the intervention? Our WWC - certified reviewers will apply the most recent WWChandbook rules to determine whether the evaluation meets standards without reservations, me ets standards with reservations, or does not meet standards. For this reason, the data collection includes all elements that would be required to complete a WWCStudy Review Guide (SRG). Q4 b. For Development grants, did the evaluation produce eviden ce on whether

the intervention warrants a more rigorous study of the intervention’s effects ? For Development grants, if applicable, did the evaluation provide evidence on the outcome s of the intervention? Our WWC - certified reviewers will apply the most recent WWChandbook rules to determine whether the evaluation meets standards without reservations, meets standards with reservations, or does not meet standards. If it does not meet standards, the WW C - certified reviewers determine that an evaluation prov ides evidence on the intervention’s promise of improving student or teacher outcomes if:  The evaluation uses is a Randomized Controlled Trial, a Regression Discontinuity Design, a Quasi - Experimental Design, and Interrupted Ti meSeries Design without a comp arison group, and a pre - post

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **19** design without a comparison group. 6

 The outcome measure meets WWCstandards. **G oal 3**  **: Present the evidence produced by each i3 evaluation.** Q5 . For each i3 grant, how faithfully was the intervention implemented? Wewill ask evaluators to report their criteria for assessing whether each key com ponent of the intervention was implemented with fidelity. In addition, we will ask evaluators to report annual fidelity estimates so that the NEi3 can assess whether or not t he intervention was implemented with fidelity. W e will state that an intervention was implemented with fidelity in a given year if the evaluator reports that the thresholds for fidelity were met for that year. As most grants are multi - year effort, we will determine overall fidelity as follows:

 **Implemented with fidelity** if the intervention was implemented with fidelity in all years in which the intervention was implemented **;**  **Not implemented with fidelity** if the intervention was not implemented with fidelity in all years in which the intervention was implemented; and  **Implemented with mixed fidelity** if the intervention was implemented with fidelity in some years but not others. Q6. For each i3 grant, w hat were the effects of the interventions on/promise of th e interventions to improve educational outcomes ? Wewill convert all of the reported impact estimates into effect sizes by following WWCguidelines, and report these impact estimates and their statistical significance. **Goal 4: Identify effective and promising interventions.**  Q7. Wh ich i3 - funded interventions were found to be effective at improving student or teacher outcomes?

 Effective interventions will be defined as those with evidence of positive and significant effects based on a study that meets WWC evidence standards. This analysis requires all of the data found in a completed WW C Study Review Guide (SRG). The evidence will indicate a positive effect if for any domain, at least one confirmatory impact estimate in the domain is positive and s tatistically significant, after correcting for multiple comparisons, and no estimated effects for any confirmatory contrasts in the domain are negative and statistically significant. Q8. Wh ich i3 - funded interventions were found to be promising at improvin g student or teacher outcomes? Promising interventions will be defined as Development Grant- funded interventions with evidence of positive and significant outcomes based on a study with a rating of Provides Evidence of the Intervention’ s Promise for Improving Outcomes. The evidence will indicate a positive outcome if for any domain, at least one confirmatory im pact estimate in the domain is positive and statistically significant, after correcting fo r m ultiple comparisons, and no estimated outcomes for any confirmatory contrasts in the domain are negative and statistically significant.

6 The NEi3 will treat an Interrupted Time Series Design or a pre -post design with a com parison group as a Quasi Experimental Design.

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **20 Goal 5: A** **ssess the results of the i3 Program.** Q9. How successful were the i3 - funded interventions? Q9a Wh at fraction of interventions were found to be effective or promising? Wewill use basic tabulation methods to compute the fraction of interventions that were found to be effective or promising. Q9b. Wh at fraction of interventions produced evidence that me t WW C evidence standards and i3 criteria? Wewill use basic tabulation methods to compute the fraction of interventions that were found to meet criteria without reservations, meet criteria with reservations, provide promise of the intervention’s promise for improving outcomes, or not meet criteria . Q9c. Wh at fraction of interventions produced credible evidence of implementation fidelity? Wewill use basic tabulation methods to compute the fraction of interventions that were found to produce credible evidence of implementation fidelity. **A. 16.2**  **Plans for publication and study sch**  **edule** Reports of the NE i3 will summarize information from the evaluations as it becomes availabl e. This means that the content of our reports is driven by the progress made by the independent evaluators. In our first rep ort on study findings , expected to be released in Spring 2016 , we expect that most FY2010 evaluations and some FY2011 evaluations will be reporting findings from their implementation and impact studies based on data collected during the 2013 - 14 school year . This report release date balances the demand for information on the evaluation findings with the need to allow interventions time for full implementation and evaluators’ time to collect and analyze data. In future reports, the main focus of the report will be updating information on the evaluations to reflect progress during the year since the prior data collection. Each of our reports wil l consist of two key sections:  **An individualized Project Profile for each i3 evaluation with findings available that :** 

 Describes the intervention implemented (goal 1).

 Assesses the strength of evidence produced (goal 2).

 Presents the evidence (goal 3 ).  **A series of cross -** **site summary tables that:**

 Identify effective and promising interventions (goal 4)

- Assess the results of the i3 Program (goal 5). The schedule for published reports based on data collected between January 2015 and December 2017 is presented in Exhibit 9 below. The schedule assumes OMB approval will be received by January 1 st , 2015 .

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **21 Exhibit**  **9 :**  **Schedule of Reports**  **Report Expected Release Dates**  First Report on NEi3 findings Spring 2016 First Addenda to Report on NEi3 findings Spring 20 17 Second Addenda to Report on NEi3 findings Spring 20 18 

### A. **17 Ap**  **proval to Not Display Expiration Date**  No exemption is requested. The data collection instruments will display the expiration date. 

### A. **18 Exceptions to Item 19 of OMB Form 83**  **- 1**  The submission describing data collection requires no exemptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **22** 

## References Bloom, H. (2006). The core analytics of randomized experiments for social research. In Alasuutari , P., Bickman , L., & Brannen , J. (eds.), Handbook of Social Research , pp. 1 - 33. Sage Publications. Coalition for Evidence -Based Policy and What Works Clearinghouse. (2005). Key items To get right when conducting a randomized controlled trial in ed ucation. U.S. Education Department’s Institute of Education Sciences (Contract #ED - 02 - CO - 0022). Downloaded from http://coalition4evidence.org/wp - content/uploads/2012/05/Guide -Key- items -to - Get -Right RCT.pdf on March 7, 2014 . Flay, B. (2005). Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prevention Science 6(3) , 1 - 25. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta - analysis . Orlando, FL: Academic Press. Hedges, L. V., & Pigott, T. D. (2001). The power of statistic al tests in meta - analysis. Psychological Methods 4(3), 486 - 504. Higgins, J. P. T. & Green, S. (ed s. ) . (2001). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 . The Cochrane Collaboration, available from www.cochrane handbook.org . Higgins, J. P. T. & Green, S. (ed s. ) . (2009). Cochrane handbook for systematic reviews of interventions version 5.0.2 . The Cochrane Collaboration, available from www.cochrane handbook.org . Konstantopoulos, S. & Hedges, L. V. (2009). Analyzing effect sizes: fixed- effects models . In H. Cooper, L. V. Hedges, & J. C. Valentine ( eds .), The handbook of research synthesis and meta - analysis. New York: Russell Sage Foundation. Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M & Shadish, W. R. (2010). Single- case designs technical documentation . Retrieved from What Works Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf . on December 13, 2010, p 2. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta - a nalysis . Applied Social Research Methods Series (Vol. 49). Thousand Oaks, CA: SAGE Publications. Lipsey, M. (2009). Identifying interesting variables and analysis opp ortunities . In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta - analysis . New York: Russell Sage Foundation. Raudenbush, S. W. (2009). Analyzing effect s izes: Random- effects model s. In Cooper , H. , Hedges, L. V. , & Valentine , J. C. ( e ds.), The handbook of research synthesis and meta - analysis . New York: Ru ssell Sage Foundation. Schochet, P., Cook, T., Deke, J., Imbens, G., Lockwood, J.R., Porter, J., Smith, J. (2010). Standards for regression discontinuity desi gns . Retrieved from What Works Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_rd.pdf on December 13, 2010, p. 1.

### OMB Clearance Package **– Part A** 

**Abt Associates National Evaluation of**  **the Investing in Innovation (i3) Program**  **▌ pg.**  **23** Shadish , W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi- experimental designs for generalized causal inference. Boston : Houghton - Mifflin. Shadish, W. R. & Haddock, C. K. (2009). Comb ining estimates of effect s ize. In Cooper , H. , Hedges, L. V. , & Valentine , J. C. ( e ds.), The handbook of research synthesis and meta - analysis . New York: Russell Sage Foundation. What Works Clearinghouse (20 14 ). What Works Clearinghouse Procedures and Standards Handbook (Version 3 .0) . Retrieved from http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_draft_standards_hand book.pdf , on May 14 , 2014 .