Supporting Statement 

for Information 

Collection Request—

Part A

National Evaluation of 

the Investing in 

Innovation (i3) 

Program

July 24, 2014

Prepared for:

Tracy Rimdzius

Institute of Education Sciences

U.S. Department of Education

555 New Jersey Avenue, NW

Washington, DC 20208

Submitted by:

Abt Associates 

55 Wheeler Street

Cambridge, MA 02138
CONTENTS

Abt Associates National Evaluation of Investing in Innovation (i3) Program ▌pg. i

A. Justification ..............................................................................................................................1

Introduction................................................................................................................................1

A.1 Circumstances Requiring the Collection of Data.............................................................1

A.2 Purposes and Use of the Data...........................................................................................2

A.2.1 Goal One: Describe the intervention implemented by each i3 grantee...............3

A.2.2 Goal Two: Present the evidence produced by each i3 evaluation.......................3

A.2.3 Goal Three: Assess the strength of the evidence produced by each i3 evaluation .......3

A.2.4 Goal Four: Identify effective and promising interventions.................................4

A.2.5 Goal Five: Assess the results of the i3 Program..................................................4

A.2.6 Alignment of the data collection goals and the data elements............................5

A.3 Use of Information Technology to Reduce Burden .......................................................11

A.4 Efforts to Identify Duplication.......................................................................................11

A.5 Small Businesses............................................................................................................12

A.6 Consequences of Not Collecting the Information..........................................................12

A.7 Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 1320.6 ...12

A.8 Consultation Outside the Agency...................................................................................12

A.8.1 Federal Register announcement ........................................................................12

A.8.2 Consultations Outside the Agency....................................................................13

A.9 Payments or Gifts to Respondents .................................................................................13

A.10 Assurance of Confidentiality..........................................................................................13

A.11 Questions of a Sensitive Nature .....................................................................................14

A.12 Estimates of Response Burden.......................................................................................14

A.13 Estimates of Total Capital and Start-Up Costs/Operation and Maintenance Costs to 

Respondents or Record Keepers....................................................................................16

A.14 Estimates of Costs to the Federal Government ..............................................................16

A.15 Changes in Burden.........................................................................................................16

A.16 Plans for Publication, Analysis, and Schedule...............................................................16

A.16.1 Approach to i3 analysis and reporting ..............................................................17

A.16.2 Plans for publication and study schedule ..........................................................20

A.17 Approval to Not Display Expiration Date......................................................................21

A.18 Exceptions to Item 19 of OMB Form 83-1 ....................................................................21

References............................................................................................................................................22
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 1

A. Justification

Introduction

The Institute of Education Sciences (IES), within the U.S. Department of Education (ED), requests that 

the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, 

clearance for IES to conduct data collection efforts for the National Evaluation of the Investing in 

Innovation (i3) Program. The i3 Program is designed to support school districts and nonprofit 

organizations in expanding, developing, and evaluating evidence-based practices and promising efforts to 

improve outcomes for the nation’s students, teachers, and schools. Each i3 grantee is required to fund an 

independent evaluation, in order to document the implementation and outcomes of the educational 

practices. The National Evaluation of i3 (NEi3) has two goals: 1) to provide technical assistance (TA) to 

support the independent evaluators in conducting evaluations that are well-designed and wellimplemented and 2) to summarize the strength of the evidence and findings from the independent 

evaluations.

Data collection will be required to address the second goal of summarizing the strength of the evidence 

and findings of the individual i3 evaluations being conducted by independent evaluators contracted by 

each i3 grantee. The required data collection will entail each i3 evaluator completing a Primary Survey; 

some evaluators will also complete an Early or a Follow-Up Survey (using the exact same survey 

template as the Primary Survey). Part A of this request discusses the justification for these data collection 

activities. Part B, submitted under separate cover, describes the data collection procedures. 

A.1 Circumstances Requiring the Collection of Data

To date, ED has awarded over $1 billion in funding through i3. In September 2010, ED awarded 

approximately $650 million in grants to the Fiscal Year (FY) 2010 cohort of i3 grantees to support the 

implementation and evaluation of 49 programs (4 Scale-Up grants, 15 Validation grants, and 30 

Development grants) aimed at improving educational outcomes for schools, teachers, and students. The 

following year, ED awarded approximately $150 million to the FY 2011 cohort of i3 grantees, including 

23 programs (1 Scale-Up grant, 5 Validation grants, and 17 Development grants). Then, in December 

2012, ED awarded approximately $140 million in grants to the FY 2012 cohort, including 20 programs (8

Validation grants and 12 Development grants). Finally, in December 2013, ED awarded approximately 

$112 million in grants to the FY 2013 cohort, including 25 programs (7 Validation grants and 18 

Development Grants). The Scale-Up grants are intended to support taking to scale programs that have 

strong evidence of their effectiveness in some settings. The Validation grants were given to programs to 

validate their effectiveness, and the Development grants were given to programs to establish the 

feasibility of implementation of the program and test the promise for improving outcomes.

The i3 Program, overseen by ED’s Office of Innovation and Improvement (OII), requires each grantee to 

fund an independent evaluation in order to document the implementation and outcomes of the educational 

practices. While ED’s goal was always that grantees fund evaluations that estimate the impact of the i3-

supported practiced (as implemented at the proposed level of scale) on a relevant outcome, this was 

explicitly stated as of the FY 2013 i3 competition. The Scale-Up and Validation grantees are expected to 

fund rigorous impact studies using experimental or quasi-experimental designs (QEDs). These 

evaluations have the potential to yield strong evidence about a range of approaches to improving 

educational outcomes. The Development grants, in turn, are also supporting independent evaluations, but 
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 2

are not necessarily expected to support rigorous evaluations. ED’s Institute of Education Sciences (IES)

has contracted with Abt Associates and its partners1(the i3 Evaluation Team) to conduct the NEi3. 

Achieving the first goal of the NEi3 requires one group within the i3 Evaluation Team, called the 

Technical Assistance (TA) Team, to take an active role in maximizing the strength of the design and 

implementation of the evaluations, so that the evaluations yield the strongest evidence possible about 

what works for whom in efforts to improve educational outcomes. Achieving the second goal of the NEi3 

requires a separate group within the i3 Evaluation Team, known as the Analysis and Reporting (AR) 

Team, to (1) assess the evaluation designs and implementation, both to inform ED of the progress of the 

evaluations2and to provide important context for the summary of results, and (2) provide clear synopses 

of what can be learned from this unprecedented investment in educational innovation—summaries that 

are cogent and easily understood by a wide range of stakeholders, including educators and school 

administrators, policymakers, ED, and the public. To ensure that the work of the TA Team does not 

influence the assessments and analysis of the AR Team, the two groups of staff members do not overlap, 

and are separated by a firewall maintained by the Project Director. The systematic data collection 

proposed for the NEi3 is necessary to provide the information for the AR Team to review and summarize 

the findings from the independent evaluations funded by i3.The data collection will be led by the AR 

team, with oversight from the Project Director.

A.2 Purposes and Use of the Data

This section of the supporting statement provides an overview of the research design and data collection 

efforts planned to meet the main research questions and overall objectives of the NEi3. The section begins 

with an overview of the research design, including the main objectives of the evaluation and key research 

questions, followed by a description of the data collection activities for which OMB clearance is 

requested. 

As noted earlier, the main objectives of the NEi3 are to (1) to provide technical assistance (TA) to support 

the independent evaluators in conducting evaluations that are well-designed and well-implemented and 

(2) to summarize the strength of the evidence and findings from the independent evaluations. The data 

collected for this evaluation will address this second objective. Specifically, the data will be used to 

support reviews and reports to ED that have five goals:

1. Describe the intervention implemented by each i3 grantee.

2. Assess the strength of the evidence produced by each i3 evaluation.

3. Present the evidence produced by each i3 evaluation, along with assessment of its strength.

4. Identify effective and promising interventions.

5. Assess the results of the i3 Program.

 

1 The partners are Dillon-Goodson Research Associates, Chesapeake Research Associates, ANALYTICA, 

Westat, Century Analytics, and American Institutes for Research.

2

It is important to note that the assessment of whether the independent evaluations are well-designed and wellimplemented forms the basis for some of the Government Performance and Results Act (GPRA) measures for 

the i3 program.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 3

These five goals are described in more detail below, along with the research questions that address each 

goal.

A.2.1 Goal One: Describe the intervention implemented by each i3 grantee

While the programs are described in the applications submitted by the grantees for funding, it will be 

important to augment these descriptions with more detailed information than could be provided within the 

page limits of the applications and to reflect what happened (as opposed to what was planned during the 

application stage). This information will allow the i3 Evaluation Team to describe key components of 

each i3-funded intervention—for example, in the approach taken to training teachers and delivering 

instruction to students. An accurate description of the details of each project’s intervention is critical to 

ED’s understanding of the breadth and depth of the approaches funded, and of the implications for what 

these specific projects can potentially teach the field about promising approaches to education reform. 

This goal will be fulfilled by answering the following research questions:

Q1. What are the key components of the intervention?

Q2. What are the student and teacher outcomes that the intervention is designed to affect? 

Q3. What are the mediators through which the intervention is expected to affect student and 

teacher outcomes?

A.2.2 Goal Two: Assess the strength of the evidence produced by each i3 evaluation

Periodic assessments of the i3 independent evaluations are essential to the NEi3. In addition to providing 

ED with information necessary to report key Government Performance and Reporting Act (GPRA) 

measures, these assessments provide important context for the interpretation of the findings generated by 

the i3 independent evaluations and summarized in the reports. Specifically, ED has explicitly articulated 

the expectation that Scale-Up and Validation grantees conduct evaluations that are “well-designed and 

well-implemented,” as defined by the standards of the What Works Clearinghouse (WWC), because

findings generated by such studies provide more convincing evidence of the effectiveness of the programs 

than evaluations that do not meet this standard. ED expects evaluations of Development grants to provide 

evidence on the intervention’s promise for improving student outcomes.

3

This goal will be fulfilled by answering the following research questions:

Q4. For each i3 grant, how strong is the evidence on the causal link between the intervention and 

its intended student or teacher outcomes?

Q4a. For Scale-Up and Validation grants, did the evaluation provide evidence on the 

effects of the intervention?

 

3 The i3 Promise criteria operationalize OII’s GPRA requirements, which distinguish between Development and 

Validation/Scale-Up grants in their evidence requirements. These Promise criteria were developed to assess the 

quality of evaluations of Development grants that do not meet WWC evidence standards (with or without 

reservations). These criteria are intended to determine the extent to which the interventions funded by 

Development grants provide evidence of “promise” for improving student outcomes and may be ready for a 

more rigorous test of their effectiveness. 
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 4

Q4b. For Development grants, did the evaluation produce evidence on whether the 

intervention warrants a more rigorous study of the intervention’s effects? For 

Development grants, if applicable, did the evaluation provide evidence on the 

outcomes of the intervention? 

A.2.3 Goal Three: Present the evidence produced by each i3 evaluation

Beginning in Spring 2016, we will present the available findings from both the implementation and 

impact studies as they emerge from the i3 evaluations. In particular, we will report on the findings on 

implementation fidelity and the findings from the impact analysis. The NEi3 will report the information 

provided by the independent evaluators for each i3 intervention (after verifying and processing those 

data).

This goal will be fulfilled by answering the following research questions:

Q5. For each i3 grant, how faithfully was the intervention implemented?

Q6. For each i3 grant, what were the effects of the interventions on/promise of the interventions 

to improve educational outcomes?

A.2.4 Goal Four: Identify effective and promising interventions

To help school and district officials identify effective interventions, we will produce lists of interventions 

with positive effects/outcomes. Since educational interventions are typically designed to improve 

outcomes in one or more domains, and are targeted for students at particular grade levels, we will list

interventions by outcome domain and educational level (e.g., reading achievement for students in 

elementary school). Each list will include the names of the i3-funded interventions that produced positive 

effects, based on i3 studies that meet WWC evidence standards (with or without reservations).

4

This goal will be fulfilled by answering the following research questions:

Q7. Which i3-funded interventions were found to be effective at improving student or teacher 

outcomes?

Q8. Which i3-funded interventions were found to be promising at improving student or teacher 

outcomes? 

A.2.5 Goal Five: Assess the results of the i3 Program

To inform federal policymakers, we propose to summarize the effectiveness of the i3-funded 

interventions. This goal will be fulfilled by answering the following research questions:

Q9. How successful were the i3-funded interventions?

Q9a. What fraction of interventions was found to be effective or promising?

 

4

See http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_draft_standards_handbook.pdf for 

WWC evidence standards.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 5

Q9b. What fraction of interventions produced evidence that met WWC evidence 

standards (with or without reservations) and i3 criteria for providing credible 

evidence of the intervention’s promise for improving educational outcomes)?

Q9c. What fraction of interventions produced credible evidence of implementation 

fidelity?

A.2.6 Alignment of the data collection goals and the data elements 

As stated above, the NEi3 has nine research questions that address the five major goals of the evaluation.

Exhibit 1 presents a crosswalk of the purposes of the data collection, the research questions, and the 

individual survey items. Appendix A contains the full data collection survey.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 6

Exhibit 1. Crosswalk Aligning Goals, Research Questions, and Survey Items 

Goal 1: Describe the intervention implemented by each i3 grantee.

Research Question Survey Items (Sections and Question Numbers) 

What are the key components of the intervention? IMPLEMENTATION: A & B, BACKGROUND: 10

What are the student or teacher outcomes that the intervention is designed to affect? IMPACT: C, D, BACKGROUND: 10

What are the mediators through which the intervention is expected to affect student or teacher outcomes? IMPACT: C, D; IMPLEMENTATION: A & B

Goal 2: Assess the strength of the evidence produced by each i3 evaluation.

Research Question Survey Items (Sections and Question Numbers) 

For each i3 grant, how strong is the evidence on the causal link between the intervention and its intended 

student or teacher outcomes?

Was the evaluation well-designed and well-implemented (i.e., for Scale-Up and Validation grants, 

did the evaluation provide evidence on the effects of the intervention)? 

Did the evaluation provide evidence on the intervention’s promise for improving student or 

teacher outcomes (i.e., for Development grants, did the evaluation produce evidence on whether 

the intervention warrants a more rigorous study of the intervention’s effects)?

BACKGROUND: 1-3, 5-9, 11-16; IMPACT: A-BS 

Goal 3: Present the evidence produced by each i3 evaluation.

Research Question Survey Items (Sections and Question Numbers) 

For each i3 grant, how faithfully was the intervention implemented? FIDELITY: C-V

For each i3 grant, what were the effects of the interventions on educational outcomes? CONTRAST: BL-CD

Goal 4: Identify effective and promising interventions. 

Research Question Survey Items (Sections and Question Numbers) 

Which i3-funded interventions were found to be effective at improving student or teacher outcomes? IMPACT: A-BS; BACKGROUND: 1-3, 5-9, 11-16 

Which i3-funded interventions were found to be promising at improving student or teacher outcomes? IMPACT: A-BS; BACKGROUND: 1-3, 5-9, 11-16 

Goal 5: Assess the results of the i3 Program.

Research Question Survey Items (Sections and Question Numbers) 

How successful were the i3-funded interventions?

What fraction of interventions was found to be effective or promising? 

What fraction of interventions produced evidence that met WWC evidence standards and i3 

criteria?

 What fraction of interventions produced credible evidence of implementation fidelity?

BACKGROUND 1-16; IMPACT A-BS; IMPLEMENTATION: A-Q
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 7

A.2.6.1 Types of data to be collected each year

Data on the i3 evaluations’ designs, activities, and findings will be collected annually beginning in 

January 2015 from the evaluators of all of the i3 projects that have findings available that the evaluator 

determines are ready to be shared with the NEi3 (because these findings are final and will not be subject 

to future changes) at the time of our yearly data collection. The types of information that will be available 

from evaluators each year will vary across the i3 projects depending on the duration of their funding 

(which ranges from 3 to 5 years) and the cohort to which they belong (FY 2010, FY 2011, FY 2012 or FY

2013).

Each i3 evaluator will be required to participate in at least one of the five annual data collection periods 

planned by the NEi3. Three of these data collections are covered by this clearance package, and the 

remaining two will be included in a future OMB clearance package. While evaluators are permitted to 

submit data during each of the five annual data collection periods, we anticipate that evaluators will 

submit most of their data during a single period. We refer to this as the evaluators’ Primary Data 

Collection. We also expect that some evaluators may choose to submit a limited number of early findings 

prior to that period or additional findings after that period. We refer to these periods as evaluators’ Early 

Data Collection and Follow-Up Data Collection, respectively. During each grant’s Primary Data 

Collection (and Early Data Collection, for those grants that choose to submit early findings), we will ask

the independent evaluator to complete a survey sharing all findings available to date. In subsequent years, 

we will follow up with evaluators to request any changes/updates to their evaluations by completing a 

Follow-Up Survey. The Follow-Up Survey will use the same exact survey template as the Primary 

Survey.

Prior to any data collection, we will pre-populate surveys for evaluators based on extant data they share 

with us (existing design documents, existing reports, conference presentations, etc.) and documents 

produced to fulfill requirements of ED’s cooperative agreements with the grantees.

Each i3 grant will undergo the following process on a timeline that is unique to its cohort and its award 

length (i3 grants were funded for periods of anywhere between three and five years), but will include the 

same key components.

For each grant, we will:

1) Review documents produced by evaluators to fulfill requirements of ED’s cooperative 

agreements with the grantees.

2) Collect extant data on the study’s findings during the later years of the grant when 

findings are available (starting in January 2015).

3) Use documents described in 1) and 2) to pre-populate a Primary Survey.

4) Send the evaluator a link to a customized, pre-populated Primary Survey asking them to 

a) actively confirm pre-populated information and b) enter missing data.

5) If needed (because the evaluator produces findings after completing a Primary Survey), 

we will repeat steps 2), 3) and 4) for data related to the additional findings only.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 8

Exhibit 2, panel 1, below provides a summary of our report release dates associated with each of the three 

data collections described in this clearance package and the types of data to be collected during Early, 

Primary and Follow-Up Data Collections; panel 2 details when we expect each cohort will participate in 

Early, Primary, and Follow-Up Data Collections covered in this OMB package.

Exhibit 2: Data Collection and Reporting Plan Overview

Panel 1: Description of Reports and Data Collection

NEi3 Report 

Type of Data 

Collection

Timing of 

Data 

Collection Type of Data to be Collected

NEi3 Findings Report (Expected Release Date: 

Spring 2016)

 Describe the interventions as designed and 

implemented;

 Describe the evaluations as implemented;

 Assess the strength of the evaluations (as 

designed and implemented);

 Present impact and implementation findings;

Early and Primary 

Data Collection

January–

March 2015

Early and Primary Data Collection

 Characteristics of interventions.

 Details of evaluation designs and plans for 

evaluations.

 Details about the implementation of the 

evaluation to date (e.g., sample selection 

and assignment, characteristics of 

measures being used.) 

 Details of the implementation of the 

evaluation (e.g., attrition, analytic 

approach employed).

 Findings from implementation and 

outcomes studies to date.

Report Addenda (Expected Release Dates: Spring 

2017, Spring 2018)

 Provide updates to prior version of the NEi3 report 

to reflect additional findings reported by 

evaluators.

Early, Primary and 

Follow-Up Data 

Collection 

January–

March, 

annually from 

2015 - 2017

Early and Primary Data Collection. The 

NEi3 will collect the data listed in the five

above bullets from additional i3-funded 

studies that have not previously participated 

in Data Collection.

Follow-Up Data Collection. The NEi3 will 

collect the following information from grants 

that have already participated in a Primary 

Data Collection and have additional findings 

to share.

 Updates of information collected 

previously, as needed;

 Data on newly available implementation 

and outcome study findings 

Panel 2: Timing and Type of Data Collection by Cohort 
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 9

A.2.6.2 Early and Primary Data Collection—Information on study design to be pre-populated by 

NEi3

At the time of Primary Data Collection, NEi3 team members will pre-populate the following information 

into the Primary Survey (see Appendix A) for all i3 independent evaluators. NEi3 team members will use

documents produced by evaluators to fulfill requirements of ED’s cooperative agreements with the 

grantees.

 Information on interventions, including:

 Descriptions of the interventions.

 Roles played by the people and organizations involved with the grant.

 People or aspects of educational organizations that the intervention aims to effect or benefit.

 Logic models of the inputs, mediators, and expected outputs of interventions. (For Scale-Up

grants this includes a logic model of the mechanisms and pathways of the scale-up process). 

 Key elements of intervention implementation.

 Scale of the intervention (in terms of the number of students, teachers, schools, and/or 

districts intended to be served).

 Evaluation plans, including:

5

 Research questions (specified as confirmatory and exploratory).

 Research design (i.e., randomized controlled trials, quasi-experimental designs with 

comparison groups, regression discontinuity designs, or other types of designs).

 Treatment and counterfactual conditions (specifically, how the counterfactual differs from the 

intervention).

 Characteristics of the potential intervention versus comparison group contrasts to be 

analyzed.

 Plans for measuring fidelity of intervention implementation.

 Outcomes or dependent variables (e.g., name, timing of measurement for the intervention and 

the comparison groups, reliability and validity properties, whether outcome data were 

imputed in the case of missing data, and a description of the measurement instrument if not 

standard).

 Independent variables (e.g., name, timing of measurement, and reliability and validity 

properties).

 Analysis methods including descriptions of how impacts in the full sample and for subgroups 

will be estimated, how the baseline equivalence will be established (in QEDs and RCTs with 

high attrition), and how clustering of participants and noncompliance to treatment will be 

accounted for.

 Role of the developers or grantee in the conduct of the evaluation (to assess the independence 

of the evaluators).

 

5 We expect that some projects will use more than one research design. In order to be clear, in this plan we refer only to a 

single design per evaluation.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 10

 Description of the study sample, including:

 Group sizes (overall and per group).

 Characteristics (e.g., grades or age levels).

 Location.

A.2.6.3 Early and Primary Data Collection – Information on study findings to be pre-populated by 

NEi3 whenever possible based on existing reports, conference presentations, etc. shared 

with the NEi3by i3 independent evaluators

Prior to Primary Data Collection, NEi3 team members will pre-populate surveys with the following 

information for all i3 independent evaluators based on any reports and other documents i3 independent

evaluators share with us. 

 Evaluation findings from the implementation study, including:

 Revised description of the intervention components.

 Level of implementation fidelity.

 Impacts on intermediate outcomes (e.g., teacher knowledge).

 Evaluation findings from impact study for each contrast tested in the full sample or subgroups, 

plus information that is helpful in interpreting the impact findings, including:

 Impact estimates.

 Standard errors (to allow the NEi3 study team to conduct tests of statistical significance).

 Standard deviation of the outcome measures (to allow the NEi3 study team to construct effect 

size measures).

 Means and standard deviations for baseline variables (to allow the NEi3 study team to test for 

baseline equivalence).

 Attrition rate and analysis sample sizes by group for each contrast.

A.2.6.4 Early and Primary Data Collection – Information to be completed by evaluators 

Any of the above data that are not available in existing reports will be collected from evaluators via our 

Primary Data Collection. Evaluators will only be asked to complete Primary Survey items that the NEi3 

was not able to pre-populate based on the prior steps. Evaluators will be asked to complete all missing 

fields, and to actively verify the accuracy of data pre-populated by the NEi3. 

A.2.6.5 Follow-Up Data Collection

During Follow-Up Data Collection, information collected previously will be updated as necessary. Again, 

we will request existing reports, conference presentations, etc. for use in pre-populating the Follow-Up 

Survey. Specifically, we will request information about any recent findings (not already reported in 

previous years) as well as the associated information required for interpreting them. This information will 

be the same as the information requested during the Primary Data Collection, except that it will be 

updated with analyses conducted following the Primary Data Collection. As previously mentioned, the 

Follow-Up Survey will use the same exact survey template as the Primary Survey.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 11

A.2.6.6 Data collection procedures and timeline

The data collections covered in this OMB clearance package will occur from January through March of 

2015, January through March of 2016, and January through March of 2017. The type of data collection 

varies between Early, Primary and Follow-Up Data Collection, depending on the cohort and grant (see 

exhibit 2, panel 2, for more information).

Exhibit 3: Overview of data collection timelines covered in this OMB Package

Data Collection 

January–March 2015

January–March 2016

January–March 2017

A.3 Use of Information Technology to Reduce Burden 

The data collection plan for the NEi3 reflects sensitivity to issues of efficiency, accuracy, and respondent 

burden in several ways. Data collection will take place in a two-stage process in which (1) the AR team 

will pre-fill surveys for evaluators as much as possible based on evaluation documents previously 

received and (2) independent i3 evaluators will respond to a web-based Primary, Early or Follow-Up 

Survey that includes this pre-filled information for their review and new survey items to respond to that 

are appropriate for the timeline of their evaluation. This two-stage process will minimize the reporting 

burden on independent evaluators.

In addition, we will reduce burden on evaluators by using web-based data portals. Data will be collected 

using a combination of Microsoft Word-type survey items and Microsoft Excel-type tables and templates 

on secure website portals accessible only by individual i3 evaluation teams. The survey is designed so 

that respondents can complete part of the survey, check their records or research answers to questions 

about which they are unsure, and then complete the survey at a later time. An evaluation-specific 

username and password will be required each time the survey is accessed. No persons other than those 

selected to receive usernames and passwords and the study team will have editing access to the website.

The study contractor will track completed surveys in real time.

A.4 Efforts to Identify Duplication

Potential duplications of effort are of two general types: addressing research questions already answered 

and duplicating data collection. The NEi3 will not address research questions already answered. As 

explained in Section A.1., the i3 grants were newly funded in FY 2010, FY 2011, FY 2012, and FY 2013

and this evaluation is the only federally funded information collection with plans to collect data in order 

to assess the extent to which the i3 independent evaluations are well-designed and well-implemented, and

report the results across the evaluations. To ensure we do not duplicate data collection efforts, 

independent evaluators will be encouraged to send study design documents, reports, summaries, and 

information tables that were constructed in their own work to the AR team prior to the survey each year.

These documents will be reviewed by the AR team and information from the documents will be used to 

pre-fill survey item responses and sections that can be reviewed by the independent evaluator—rather 

than asking the evaluators to provide information that has already been reported in existing evaluation 

documents.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 12

A.5 Small Businesses

The primary entities for this study are independent evaluators employed by both large and small 

businesses as well as in some universities. Every effort is being made to reduce the burden on the 

evaluators through the collection of study design documents, reports, summaries, and information tables 

that evaluators constructed in their own work, as well as the use of a web-based data collection. The 

specific plans for reducing burden are described in Section A.3.

A.6 Consequences of Not Collecting the Information

The data collection described in this supporting statement is necessary for conducting this evaluation, 

which is consistent with the goals of the Investing in Innovation program to identify and document best

practices that can be shared and taken to scale based on demonstrated success (American Recovery and 

Reinvestment Act of 2009 (ARRA), Section 14007(a)(3)(C)). The information that will be collected 

through this effort is also necessary to report on the performance measures of the i3 Program, required by 

the Government Performance and Results Act: 

Long-Term Performance Measures

 The percentage of programs, practices, or strategies supported by a [Scale-Up or Validation] 

grant that implement a completed well-designed, well-implemented and independent evaluation 

that provides evidence of their effectiveness at improving student or teacher outcomes.

 The percentage of programs, practices, or strategies supported by a [Development] grant with a 

completed evaluation that provides evidence of their promise for improving student or teacher

outcomes.

 The percentage of programs, practices, or strategies supported by a [Scale-Up or Validation] 

grant with a completed well-designed, well-implemented and independent evaluation that 

provides information about the key elements and the approach of the project so as to facilitate 

replication or testing in other settings.

 The percentage of programs, practices, or strategies supported by a [Development] grant with a 

completed evaluation that provides information about the key elements and the approach of the 

project so as to facilitate further development, replication or testing in other settings.

A.7 Special Circumstances Justifying Inconsistencies with Guidelines in 5 CFR 

1320.6

There are no special circumstances required for the collection of this information.

A.8 Consultation Outside the Agency

A.8.1 Federal Register announcement

In accordance with the Paperwork Reduction Act of 1995 (Pub.L. No. 104-13) and Office of Management 

and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), IES published a 

notice in the Federal Register announcing the agency’s intention to request an OMB review of data 

collection activities for the i3 Evaluation. The notice was published on (date) in volume xx, No. xx, page 

xx and provided a 60-day period for public comment.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 13

A.8.2 Consultations Outside the Agency 

The Abt study team assembled a Technical Work Group (TWG) (in consultation with ED) that consists of

consultants with various types of expertise relevant to this evaluation. The TWG convened in April 2011 

and discussed the overall approach to the NEi3, including providing TA and data collection and reporting.

Individuals who served on the expert panel are listed in Exhibit 4.

Exhibit 4: Expert Panel Members 

Name Affiliation

David Francis, Ph.D. University of Houston

Tom Cook, Ph.D. Northwestern University

Brian Jacob, Ph.D. University of Michigan

Lawrence Hedges, Ph.D. Northwestern University

Carolyn Hill, Ph.D. Georgetown University

Chris Lonigan, Ph.D. Florida State University

Neal Finkelstein, Ph.D. WestEd

Bob Granger, Ph.D. W.T. Grant Foundation

A.9 Payments or Gifts to Respondents

Data collection for this study does not involve payments or gifts to respondents.

A.10 Assurance of Confidentiality

Abt Associates and its subcontractors follow the confidentiality and data protection requirements of IES

(The Education Sciences Reform Act of 2002, Title I, Part E, Section 183), which require that all 

collection, maintenance, use and wide dissemination of data conform to the requirements of the Privacy 

Act of 1974 (5 U.S.C. 552a), the Family Educational Rights and Privacy Act of 1974 (20 U.S.C. 1232g), 

and the Protection of Pupil Rights Amendment (20 U.S.C. 1232h). The study team will not be collecting 

any individually identifiable information and Abt Associates and its subcontractors will protect the 

confidentiality of all information collected for the study and will use it for research purposes only.

All study staff involved in collecting, reviewing, or analyzing study data will be knowledgeable about 

data security procedures. The privacy procedures adopted for this study for all data collection, data 

processing, and analysis activities include the following:

 The study will not request any personally identifiable information (PII) data that was collected by 

independent evaluators. All of the data requested from independent evaluators will be in the form 

of aggregated reports of the methods, measures, plans, and results in their independent 

evaluations.

 Individual i3 grants will be identified in this study. Their characteristics, results, and findings 

reported by independent evaluators, as well as assessments of the quality of the independent 

evaluations and i3 projects, may potentially be reported. This study cannot, however, associate 

responses with a specific school or individual participating in i3 independent evaluations in 

annual reports, since we are not collecting that information. 
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 14

 To ensure data security, all individuals hired by Abt Associates Inc. are required to adhere to 

strict standards and sign an oath of confidentiality as a condition of employment. Abt’s 

subcontractors will be held to the same standards.

 All data files on multi-user systems will be under the control of a database manager, with access 

limited to project staff on a “need-to-know” basis only.

 Identifiable data will be kept for three years after the project end date and then destroyed.

 Written records will be shredded and electronic records will be purged.

A.11 Questions of a Sensitive Nature

The data collection for this study does not include any questions of a sensitive nature. 

A.12 Estimates of Response Burden

Burden for the data collection covered by this clearance request is 1,507 hours, for a total cost to 

respondents of $56,449. Exhibit 5 presents time estimates of response burden for the data collection 

activities requested for approval in this submission. The burden estimates are based on the following 

assumptions:

 The majority (80 percent) of evaluators will submit existing materials (reports, conference 

presentations, etc.) reporting their findings to the AR team. 

 The AR team will pre-fill survey item responses for evaluators with as much information as 

possible based on the documents they receive from evaluators reporting their findings and 

documents produced by evaluators to fulfill requirements of ED’s cooperative agreements with 

the grantees.

 Evaluators will respond to a web-based Primary (and possibly also an Early or Follow-Up) 

Survey that includes pre-filled information for their review and new survey items.

Estimated hourly costs to independent evaluators are based on an average hourly wage for social 

scientists and related workers of $37.45 according to The United States Department of Labor, Bureau of 

Labor Statistics (U.S. Department of Labor 2009).
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 15

Exhibit 5. Estimate of Respondent Burden

Data Collection Wave and Activity

Total 

Number of 

Expected 

Responses

Total 

Burden 

Hours per 

Response

Total 

Burden 

Hours

Hourly Cost 

per Responsef

Total 

Costs

Primary Data Collection: Respond to Surveya 92 - - - -

... Respondents who submit extant data on findingsb 74 7.910 585.3 $ 37.45 $21,921

...Respondents who do NOT submit extant data on findingsb 18 26.333 474.0 $ 37.45 $17,751

Early or Follow-Up Data Collection: Respond to Surveyc 38 - - - -

... Respondents who submit extant data on findingsd 30 7.910 237.3 $ 37.45 $8,887

...Respondents who do NOT submit extant data on findingsd 8 26.333 210.7 $ 37.45 $7,889

Total Number of Responses 130 - 1507 $ 37.45 $56,449

Total Number of Respondentse 117

Average Number of Responses per Respondent 1.11

Overall Average Burden Hours per Respondent 13

Overall Average Cost per Respondent $482.47

Notes: 

This OMB Package covers the period from 1/1/2015 through 12/31/2017. Therefore it includes all data collection activities for the 49 FY2010 grants and 23 FY2011 grants, Early and Primary Data 

Collection for the 20 FY2012 grants and Early Data Collection for the 25 FY2013 grants.

Responding to the survey (during Primary Data Collection, Early and Follow-up Data Collection) includes reviewing data prepopulated by the NEi3 based on extant data on findings submitted to the 

NEi3 and confirming any information pre-populated by the NEi3 based on review of documents produced by evaluators to fulfill requirements of ED’s cooperative agreements with the grantees. 

aWe expect 92 grants (all 49 FY2010 grants, all 23 FY2011 grants, and all 20 FY2010 grants) to participate in Primary Data Collection during the period covered by this OMB Package.

bWe expect approximately 80 percent (74) of the 92 Primary Data Collection participants to submit extant data prior to their Primary Data Collection.

c We expect 38 grants [40 percent (20) of the FY2010 grants, 40 percent (9) of the FY2011 grants, 20 percent (4) of the FY2012 grants, and 20 percent (5) of the FY2013 grants] to participate in Early 

or Follow-Up Data Collection. 

dWe expect approximately 80 percent (30) of the 38 Early or Follow-up Data Collection participants to submit extant data prior to their Early or Follow-up Data Collection.

eDuring the period covered by this OMB package, we expect a total of 117 survey respondents (49 FY2010 grants, 23 FY2011 grants, 20 FY2012 grants and 25 FY2013 grants). This number differs 

from the number of responses because some of these grants will participate in Primary and Early or Follow-up Data Collection. Other grants (FY2013) will only participate in Early Data Collection.

fAverage hourly wage for “Social Scientists, and Related Workers, All Others” from the Industry-Specific Occupational and Wage Estimates, U.S. Department of Labor (see 

http://www.bls.gov/oes/current/oes193099.htm#nat), accessed May 24, 2011
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 16

A.13 Estimates of Total Capital and Start-Up Costs/Operation and Maintenance 

Costs to Respondents or Record Keepers

There are no annualized capital, start-up, or ongoing operation and maintenance costs involved in 

collecting this information.

A.14 Estimates of Costs to the Federal Government

The estimated cost to the Federal Government of the activities across all of the NEi3 contracts 

(FY2010/FY2011 cohort -contract no: ED-IES-10-C-0064, FY2012 cohort - contract no: ED-IES-13-C0005, and FY2013 cohort – contract number ED-IES-14-C-0007) is $22,322,122 for the entire NEi3, 

including work on all contracts and tasks. These activities began in October, 2010 and will continue until 

February, 2020. Thus, the average annual cost to the federal government is approximately $2,391,366. 

A.15 Changes in Burden

This is a request for a new collection of information.

A.16 Plans for Publication, Analysis, and Schedule

In this section, we present our approach to analyzing the data collected in order to address the five goals

of the data collection (and related research questions) introduced in Section A.2: (1) Describe the 

intervention implemented by each i3 grantee; (2) Assess the strength of the evidence produced by each i3 

evaluation; (3) Present the evidence produced by each i3 evaluation; (4) Identify effective and promising 

interventions; and (5) Assess the results of the i3 Program.

After describing our approach to addressing the five goals, we discuss our plans for reporting the results.

The five goals and research questions we plan to address over the course of our evaluation, presented in 

Section A.2 above, are reiterated in Exhibit 6.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 17

Exhibit 6: NEi3’s Goals and Research Questions

Goal 1: Describe the intervention implemented by each i3 grantee.

Q1. What are the key components of the intervention?

Q2. What are the student or teacher outcomes that the intervention is designed to affect?

Q3. What are the mediators through which the intervention is expected to affect student

or teacher outcomes?

Goal 2: Assess the strength of the evidence produced by each i3 evaluation.

Q4. For each i3 grant, how strong is the evidence on the causal link between the 

intervention and its intended student or teacher outcomes?

Q4a. For Scale-Up and Validation grants, did the evaluation provide evidence on 

the effects of the intervention? 

Q4b. For Development grants, did the evaluation produce evidence on whether 

the intervention warrants a more rigorous study of the intervention’s effects?

For Development grants, if applicable, did the evaluation provide evidence 

on the outcomes of the intervention? 

Goal 3: Present the evidence produced by each i3 evaluation.

Q5. For each i3 grant, how faithfully was the intervention implemented?

Q6. For each i3 grant, what were the effects of the interventions on/promise of the 

interventions to improve educational outcomes?

Goal 4: Identify effective and promising interventions.

Q7. Which i3-funded interventions were found to be effective at improving student or 

teacher outcomes?

Q8. Which i3-funded interventions were found to be promising at improving student or 

teacher outcomes?

Goal 5: Assess the results of the i3 Program.

Q9. How successful were the i3-funded interventions?

Q9a What fraction of interventions was found to be effective or promising? 

Q9b. What fraction of interventions produced evidence that met WWC evidence 

standards and i3 criteria? 

Q9c. What fraction of interventions produced credible evidence of implementation 

fidelity?

A.16.1 Approach to i3 analysis and reporting 

The nature of our analysis and reporting is to synthesize and review findings, rather than to estimate 

impacts using statistical techniques. In this section, we provide more information on how we plan to 

address the research questions posed for the NEi3 by analyzing the data we are requesting permission to 

collect. 
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 18

Goal 1: Describe the intervention implemented by each i3 grantee. 

Q1. What are the key components of the intervention?

Key components are defined as the activities and inputs that are under the direct 

control of the individual or organization responsible for program implementation (e.g., 

program developer, grant recipient), and are considered by the developer to be 

essential in implementing the intervention. Key components may include financial 

resources, professional development for teachers, curricular materials, or technology 

products. We will review the evaluators’ reported key components, possibly rephrase 

them to maintain their original meaning but to be consistent with other key 

components named across other evaluations, and report them in a succinct manner 

to facilitate easing “browsing” of activities across interventions. 

Q2. What are the student or teacher outcomes that the intervention is designed to affect?

We will report the student or teacher outcome domains that were evaluated for each 

grant. We may rephrase the outcome domains to maintain their original meaning but 

to be consistent with outcome domains named in other interventions, for example we 

may use “Mathematics” to describe outcome domains labeled by evaluators as “Math 

Achievement”, “Mathematics Achievement”, “Mathematical Understanding”, etc. 

Q3. What are the mediators through which the intervention is expected to affect student 

or teacher outcomes?

We will report the mediators through which the intervention is expected to affect 

outcomes. We may rephrase the mediator names to maintain their original meaning 

but to be consistent with mediators named in other interventions, for example we 

may use “Student Engagement” to describe mediators labeled by evaluators as 

“Student Participation”, “Student Commitment”, etc. 

Goal 2: Assess the strength of the evidence produced by each i3 evaluation. 

Q4. For each i3 grant, how strong is the evidence on the causal link between the 

intervention and its intended student or teacher outcomes?

Q4a. For Scale-Up and Validation grants, did the evaluation provide evidence on 

the effects of the intervention? 

Our WWC-certified reviewers will apply the most recent WWC handbook 

rules to determine whether the evaluation meets standards without 

reservations, meets standards with reservations, or does not meet 

standards. For this reason, the data collection includes all elements that 

would be required to complete a WWC Study Review Guide (SRG). 

Q4b. For Development grants, did the evaluation produce evidence on whether 

the intervention warrants a more rigorous study of the intervention’s effects? 

For Development grants, if applicable, did the evaluation provide evidence 

on the outcomes of the intervention? 

Our WWC-certified reviewers will apply the most recent WWC handbook 

rules to determine whether the evaluation meets standards without 

reservations, meets standards with reservations, or does not meet 

standards. If it does not meet standards, the WWC-certified reviewers 

determine that an evaluation provides evidence on the intervention’s 

promise of improving student or teacher outcomes if:

 The evaluation uses is a Randomized Controlled Trial, a Regression 

Discontinuity Design, a Quasi-Experimental Design, and Interrupted 

Time Series Design without a comparison group, and a pre-post 
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 19

design without a comparison group.6

 The outcome measure meets WWC standards. 

Goal 3: Present the evidence produced by each i3 evaluation.

Q5. For each i3 grant, how faithfully was the intervention implemented?

We will ask evaluators to report their criteria for assessing whether each key 

component of the intervention was implemented with fidelity. In addition, we will ask 

evaluators to report annual fidelity estimates so that the NEi3 can assess whether or 

not the intervention was implemented with fidelity. We will state that an intervention 

was implemented with fidelity in a given year if the evaluator reports that the 

thresholds for fidelity were met for that year. As most grants are multi-year effort, we 

will determine overall fidelity as follows:

 Implemented with fidelity if the intervention was implemented with fidelity in 

all years in which the intervention was implemented;

 Not implemented with fidelity if the intervention was not implemented with 

fidelity in all years in which the intervention was implemented; and

 Implemented with mixed fidelity if the intervention was implemented with 

fidelity in some years but not others. 

Q6. For each i3 grant, what were the effects of the interventions on/promise of the 

interventions to improve educational outcomes?

We will convert all of the reported impact estimates into effect sizes by following 

WWC guidelines, and report these impact estimates and their statistical significance.

Goal 4: Identify effective and promising interventions.

Q7. Which i3-funded interventions were found to be effective at improving student or 

teacher outcomes?

 Effective interventions will be defined as those with evidence of positive and 

significant effects based on a study that meets WWC evidence standards. This 

analysis requires all of the data found in a completed WWC Study Review Guide 

(SRG). The evidence will indicate a positive effect if for any domain, at least one 

confirmatory impact estimate in the domain is positive and statistically significant, 

after correcting for multiple comparisons, and no estimated effects for any 

confirmatory contrasts in the domain are negative and statistically significant.

Q8. Which i3-funded interventions were found to be promising at improving student or 

teacher outcomes?

Promising interventions will be defined as Development Grant-funded interventions 

with evidence of positive and significant outcomes based on a study with a rating of 

Provides Evidence of the Intervention’s Promise for Improving Outcomes. The 

evidence will indicate a positive outcome if for any domain, at least one confirmatory 

impact estimate in the domain is positive and statistically significant, after correcting 

for multiple comparisons, and no estimated outcomes for any confirmatory contrasts 

in the domain are negative and statistically significant.

 

6 The NEi3 will treat an Interrupted Time Series Design or a pre-post design with a comparison group as a QuasiExperimental Design.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 20

Goal 5: Assess the results of the i3 Program.

Q9. How successful were the i3-funded interventions?

Q9a What fraction of interventions were found to be effective or promising? 

We will use basic tabulation methods to compute the fraction of interventions 

that were found to be effective or promising. 

Q9b. What fraction of interventions produced evidence that met WWC evidence

standards and i3 criteria? 

We will use basic tabulation methods to compute the fraction of interventions 

that were found to meet criteria without reservations, meet criteria with 

reservations, provide promise of the intervention’s promise for improving 

outcomes, or not meet criteria. 

Q9c. What fraction of interventions produced credible evidence of implementation 

fidelity?

We will use basic tabulation methods to compute the fraction of interventions 

that were found to produce credible evidence of implementation fidelity. 

A.16.2 Plans for publication and study schedule 

Reports of the NEi3 will summarize information from the evaluations as it becomes available. This means 

that the content of our reports is driven by the progress made by the independent evaluators. In our first 

report on study findings, expected to be released in Spring 2016, we expect that most FY2010 evaluations 

and some FY2011 evaluations will be reporting findings from their implementation and impact studies 

based on data collected during the 2013-14 school year. This report release date balances the demand for 

information on the evaluation findings with the need to allow interventions time for full implementation 

and evaluators’ time to collect and analyze data. 

In future reports, the main focus of the report will be updating information on the evaluations to reflect 

progress during the year since the prior data collection. 

Each of our reports will consist of two key sections:

 An individualized Project Profile for each i3 evaluation with findings available that: 

 Describes the intervention implemented (goal 1). 

 Assesses the strength of evidence produced (goal 2).

 Presents the evidence (goal 3).

 A series of cross-site summary tables that: 

 Identify effective and promising interventions (goal 4) 

- Assess the results of the i3 Program (goal 5).

The schedule for published reports based on data collected between January 2015 and December 2017 is 

presented in Exhibit 9 below. The schedule assumes OMB approval will be received by January 1

st

, 2015.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 21

Exhibit 9: Schedule of Reports

Report Expected Release Dates

First Report on NEi3 findings Spring 2016

First Addenda to Report on NEi3 findings Spring 2017

Second Addenda to Report on NEi3 findings Spring 2018

A.17 Approval to Not Display Expiration Date

No exemption is requested. The data collection instruments will display the expiration date. 

A.18 Exceptions to Item 19 of OMB Form 83-1

The submission describing data collection requires no exemptions to the Certificate for Paperwork 

Reduction Act (5 CFR 1320.9).
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 22

References

Bloom, H. (2006). The core analytics of randomized experiments for social research. In Alasuutari, P.,

Bickman, L., & Brannen, J. (eds.), Handbook of Social Research, pp. 1-33. Sage Publications.

Coalition for Evidence-Based Policy and What Works Clearinghouse. (2005). Key items To get right 

when conducting a randomized controlled trial in education. U.S. Education Department’s 

Institute of Education Sciences (Contract #ED- 02-CO-0022). Downloaded from

http://coalition4evidence.org/wp-content/uploads/2012/05/Guide-Key-items-to-Get-RightRCT.pdf on March 7, 2014. 

Flay, B. (2005). Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prevention 

Science 6(3), 1-25.

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological 

Methods 4(3), 486-504.

Higgins, J. P. T. & Green, S. (eds.). (2001). Cochrane Handbook for Systematic Reviews of

Interventions Version 5.0.2. The Cochrane Collaboration, available from www.cochranehandbook.org.

Higgins, J. P. T. & Green, S. (eds.). (2009). Cochrane handbook for systematic reviews of

interventions version 5.0.2. The Cochrane Collaboration, available from www.cochranehandbook.org.

Konstantopoulos, S. & Hedges, L. V. (2009). Analyzing effect sizes: fixed-effects models. In H. Cooper, 

L. V. Hedges, & J. C. Valentine (eds.), The handbook of research synthesis and meta-analysis. 

New York: Russell Sage Foundation.

Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M & Shadish, 

W. R. (2010). Single-case designs technical documentation. Retrieved from What Works 

Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf. on December 13, 2010, p 2. 

Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Applied Social Research Methods 

Series (Vol. 49). Thousand Oaks, CA: SAGE Publications.

Lipsey, M. (2009). Identifying interesting variables and analysis opportunities . In H. Cooper, L. V. 

Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis. New 

York: Russell Sage Foundation.

Raudenbush, S. W. (2009). Analyzing effect sizes: Random-effects models. In Cooper, H., Hedges, L. V., 

& Valentine, J. C. (eds.), The handbook of research synthesis and meta-analysis. New York: 

Russell Sage Foundation.

Schochet, P., Cook, T., Deke, J., Imbens, G., Lockwood, J.R., Porter, J., Smith, J. (2010). Standards for 

regression discontinuity designs. Retrieved from What Works Clearinghouse website: 

http://ies.ed.gov/ncee/wwc/pdf/wwc_rd.pdf on December 13, 2010, p. 1.
OMB Clearance Package – Part A

Abt Associates National Evaluation of the Investing in Innovation (i3) Program ▌pg. 23

Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi- experimental designs for 

generalized causal inference. Boston : Houghton- Mifflin.

Shadish, W. R. & Haddock, C. K. (2009). Combining estimates of effect size. In Cooper, H., Hedges, L. 

V., & Valentine, J. C. (eds.), The handbook of research synthesis and meta-analysis. New York: 

Russell Sage Foundation.

What Works Clearinghouse (2014). What Works Clearinghouse Procedures and Standards Handbook 

(Version 3.0). Retrieved from 

http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_draft_standards_hand

book.pdf, on May 14, 2014.