Anchor

Evidence of Effectiveness in AmeriCorpsFunded Interventions

September 25, 2020

Scott Richman and Andrei Streke

Submitted to:

AmeriCorps

Office of Research and Evaluation

250 E Street, SW

Washington, DC 20525

Project Officer: Lily Zandniapour

Contract Number: GS10F0050L/CNSHQ16F0049

Submitted by:

Mathematica 

111 East Wacker Drive, Suite 3000 

Chicago, IL 60601

Telephone: (312) 994-1002

Facsimile: (312) 994-1003

Project Director: Scott Richman

Reference Number: 50332
i 

This page has been left blank for double-sided copying.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

iii

ACKNOWLEDGMENTS

The authors acknowledge several people who contributed to and supported this report funded by 

AmeriCorps. At AmeriCorps, Lily Zandniapour, Anthony Nerino, Roshni Menon, Sarah Yue, 

Mary Hyde, and Katy Hussey-Sloniker provided advice and direction that greatly enhanced this 

report. We appreciate their thoughtful insights into AmeriCorps programs and their comments on 

our products. We also thank the following study team members at Mathematica and its partner, 

Twin Peaks Partners, LLC, for their assistance in reviewing and extracting information from 

documents that were used as the source of the data analyzed in this report: Mary Anne Anderson, 

Annie Buonaspina, Kylie Cannon, Katie Eddins, Karina Edouard, Alix Gould-Werth, Chris 

Jones, Ellen Kisker (Twin Peaks Partners, LLC), Rebecca Kleinman, Hande Inanc Lenowitz, 

Natalie Reid, Ryan Ruggiero, Noa Sager, Rebekah Selekman, Jill Spielfogel, and Mathew 

Stange. In addition, Cay Bradley, Patricia Del Grosso, Nan Maxwell, Karen Needels, and 

Samina Sattar provided feedback that greatly enhanced the report. Colleen Fitts helped prepare 

it, and Larisa Wiseman provided editorial assistance. 
This page has been left blank for double-sided copying.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

v 

CONTENTS

EXECUTIVE SUMMARY ..........................................................................................................................VII

I. INTRODUCTION ................................................................................................................................ 1

A. Overview of the Scaling Evidence-Based Models project............................................................. 3

B. Overview of the meta-synthesis study.......................................................................................... 4

1. Identifying the interventions that work with the SPREE process ............................................ 4

2. Describing interventions that work ......................................................................................... 8

C. Road map for the report ............................................................................................................... 8

II. AMERICORPS-FUNDED INTERVENTIONS WITH EVIDENCE THEY WORK................................ 11

A. Identifying AmeriCorps-funded interventions with evidence of effectiveness ............................. 12

B. Characteristics of interventions that work ................................................................................... 16

1. Interventions in the education, economic opportunity, and healthy futures 

focus areas had evidence that they work............................................................................. 16

2. The education and economic opportunity focus areas, but not the healthy 

futures focus area, had interventions that were assessed as working in each 

of their priority objectives. .................................................................................................... 17

3. Interventions that work were implemented in urban and rural settings with 

fairly even distribution across geographic regions. .............................................................. 19

4. More than one-quarter (29 percent) of interventions that work had 

AmeriCorps members involved in providing services directly to participants 

or supporting service delivery. ............................................................................................. 19

III. IMPACTS OF INTERVENTIONS WITH EVIDENCE THEY WORK.................................................. 21

A. Magnitude of effects of interventions with evidence they work ................................................... 22

B. Magnitude of effects by intervention characteristics ................................................................... 24

IV. SUMMARY, RECOMMENDATIONS, AND NEXT STEPS................................................................ 29

A. Summary of findings................................................................................................................... 29

B. Recommendations...................................................................................................................... 30

REFERENCES ......................................................................................................................................... 33

APPENDIX A: REVIEW PROCESS AND INTERVENTION REVIEW RUBRIC ......................................A-1

APPENDIX B: CHARACTERISTICS OF INTERVENTIONS WITH EVIDENCE OF 

EFFECTIVENESS ...................................................................................................................................B-1

APPENDIX C: METHODS AND TABLES FOR THE META-ANALYSIS..................................................C-1
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

vi 

TABLES

I.1. AmeriCorps evidence ratings ......................................................................................................... 5

I.2. Standards for categorizing evidence.............................................................................................. 6

II.1. Interventions by focus area .......................................................................................................... 16

FIGURES

I.1. The Scaling Programs with Research Evidence and Effectiveness (SPREE) process......................... 4

II.1. Identifying interventions with evidence of effectiveness..................................................................... 13

II.2. Percentage of interventions with studies that had evidence of effectiveness..................................... 14

II.3. Reasons interventions were without evidence of effectiveness: Percentage with studies that 

did not meet each standard.......................................................................................................... 15

III.1. Overall and intervention average effects for primary outcomes ........................................................ 23

III.2. Intervention average effect sizes by focus area and priority objective .............................................. 25

III.3. Average effects by intervention characteristics ................................................................................. 26

III.4. Average effects by funding year and source..................................................................................... 27
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

vii

EXECUTIVE SUMMARY

Agencies throughout the federal government and other grant-making institutions are 

increasingly conducting systematic evidence reviews and synthesizing the results of wellexecuted research to identify interventions that have evidence of effectiveness. These efforts 

can help funders make strategic decisions on how they can most effectively and efficiently 

address individual and community needs. These reviews and syntheses can also enhance 

funders’ decisions about scaling up specific interventions that show the most promise in order to 

expand the scope of their impact.

AmeriCorps invests significant resources in interventions designed to improve lives and 

strengthen communities and in evaluating the effectiveness of these interventions.

1 To gain 

insights about which funded interventions work and which of those might be ready to scale, the 

agency funded the project known as Scaling Evidence-Based Models in 2016. Conducted by 

Mathematica, this project is designed to help deepen the agency’s understanding of the most 

effective interventions it has funded and its 

knowledge base on issues involved in scaling them. 

This information will also help inform the agency’s 

interest in identifying the intervention components 

that are critical for an intervention’s effectiveness. 

The project has two parts. The first—the subject of 

this report—aims to identify AmeriCorps-funded 

interventions with evidence of effectiveness, which 

we sometimes call interventions that work. We 

define evidence of effectiveness as rigorous 

research that has shown favorable impacts on the 

majority of the targeted outcomes for people 

receiving the intervention services. The second part 

of the process—discussed in the companion report 

“Planned Scaling Activities of AmeriCorps-Funded 

Organizations” (Selekman et al. 2020)—is meant to 

identify how the effective interventions and the 

organizations implementing them might be ready for 

scaling. 

The current report provides the findings for the interventions funded between 2015 and 2019 by 

the AmeriCorps State and National program and between 2010 and 2015 by the Social 

Innovation Fund (SIF), which were the programs and grantee cohorts of interest to AmeriCorps

that had evidence documents available for review for this project. See Appendix A for more 

1 As of September 29th, 2020, the Corporation for National and Community Service is operating under the name 

AmeriCorps. 

Research questions 

1. What interventions consistently 

demonstrate evidence of effectiveness

for their targeted outcomes? 

• Which AmeriCorps-funded 

interventions demonstrate 

evidence of effectiveness on 

participants’ outcomes?

• Among interventions with evidence 

of effectiveness, what are their 

characteristics?

2. What intervention characteristics 

demonstrate evidence of effectiveness 

at a scale that suggests readiness for 

replication more widely?

• How effective are AmeriCorps

interventions with evidence of 

effectiveness?

• What characteristics of the 

interventions might be associated 

with the estimated impacts?
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

viii

information on the process and rubric that Mathematica used to review the evidence 

documents. 

Mathematica used two primary research questions to identify interventions with evidence of 

effectiveness (see sidebar). We answered the first research question by developing and 

applying standards to assess the rigor of the research submitted by AmeriCorps State and 

National 2015–2019 and SIF 2010–2015 grantees as evidence of the funded interventions’ 

effectiveness. We refer to these standards as the Mathematica effectiveness standards 

throughout this report.

2 Mathematica described each intervention in terms of its alignment with 

AmeriCorps’ focus areas and priority 

objectives, funding year, and factors related to 

AmeriCorps’ mission: reaching different 

communities (operationalized as urban/rural 

and region of country) and using AmeriCorps 

members in providing services directly to 

participants or supporting the delivery. We 

answered the second research question using 

a meta-synthesis analysis to uniformly 

measure the magnitude of an intervention’s 

impact and to compare the effectiveness of 

interventions with different participant 

outcomes.

A. Summary of key findings

1. What interventions consistently 

demonstrate evidence of effectiveness? 

Thirty-nine percent of interventions (32 out of 

82 interventions) implemented by grantees 

given a strong or moderate evidence rating in 

a systematic evidence review by independent 

contractors and reviewers for AmeriCorps

(other than Mathematica)3 met the evidence of 

2 Mathematica’s effectiveness standards and AmeriCorps evidence ratings for grantees have similarities but also 

important differences. For example, Mathematica’s effectiveness standards include studies providing detailed 

technical information on the research design’s internal validity and requiring a consistency of positive findings 

across the range of outcomes that were assessed in an evaluation, but these are not AmeriCorps evidence rating

requirements.

3 The third-party AmeriCorps contractors and reviewers (other than Mathematica) assessed the studies submitted by 

grantees for their interventions as potentially being able to produce causal evidence for their interventions. The 

contractor then applied an AmeriCorps evidence rating at the grantee or subgrantee level—not the intervention 

level—based on the quality of evidence that grantees submitted. For this project, Mathematica reviewed the 

evidence submitted by grantees that received a moderate or strong evidence rating.

AmeriCorps focus areas

• Disaster services helps communities across 

the country prepare for, respond to, and 

recover from natural and other disasters.

• Economic opportunity addresses housing, 

financial literacy, and employment needs of 

low-income individuals and families. 

• Education helps communities design and 

implement results-driven, cost-effective 

solutions that support students from cradle to 

career.

• Environmental stewardship (environment) 

trains youth and unemployed and 

underemployed citizens in conservation and 

green jobs, reconnects Americans to the 

outdoors, and supports successful sciencebased conservation strategies. 

• Healthy futures uses a three-pronged

approach of health-focused assistance, 

prevention, and intervention to educate and 

maintain healthy communities. 

• Nonprofit organizational capacity (org 

capacity) helps organizations expand the 

reach, efficiency, or effectiveness of their 

programs.

• Veterans and military families (veterans) 

serves and engages the talents of veterans 

and military families.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

ix

effectiveness standards developed for this project (which, as noted above, we refer to as the 

Mathematica effectiveness standards). Among the assessed criteria, the primary reasons that 

studies did not meet the Mathematica standards were (1) a lack of internal validity that gives 

confidence that the intervention, and not other factors, impacted intervention participants (46

percent); (2) results that were not consistently favorable (44 percent); and (3) a lack of a 

comparison group (40 percent). The 32 interventions that met the Mathematica standards fell 

primarily in the education focus area (56 percent), with 28 percent aligning with the economic 

opportunity focus area and 16 percent falling under the healthy futures focus area (see sidebar 

for AmeriCorps focus areas). These 32 interventions were predominately implemented in urban 

settings (97 percent), although 16 percent were implemented in rural areas. About half (47

percent) were implemented in more than one region of the country, and more than one-quarter 

(29 percent) had AmeriCorps members involved in delivering services or supporting the 

delivery.4

2. What intervention characteristics demonstrate evidence of effectiveness at a scale that 

suggests readiness for replication more widely? 

Participants in the AmeriCorps-funded interventions meeting the Mathematica effectiveness 

standards had better outcomes than 61 percent of participants who did not receive the 

intervention. This represents an 11 percentage point increase in beneficial outcomes. The size 

of this impact varied by focus area. Interventions in the education and healthy futures focus 

areas increased beneficial outcomes for intervention participants by 9 percentage points, 

compared to those not receiving the intervention. For interventions in the economic opportunity

focus area, the impact reached 14 percentage points. The size of the impacts varied by 

AmeriCorps’ priority objective. Interventions aligning with the employability, K–12 success, 

postsecondary education support, school readiness, and supportive family environments priority 

objectives had statistically significant positive impacts. Impacts did not vary by geographic 

region, or the use of AmeriCorps members, which suggests that interventions might be equally 

effective across communities and that AmeriCorps members are as effective as other types of 

personnel at delivering or supporting the delivery of services. Impacts also did not vary by 

intervention funding year, which suggests that AmeriCorps has year-to-year consistency in 

identifying interventions supported by rigorous evidence of effectiveness. 

4 All interventions currently funded by the AmeriCorps State and National program are required to include 

AmeriCorps members in the interventions’ service delivery model. In this report, we describe whether 

AmeriCorps members were part of the service delivery model as described in the evidence documents that 

grantees submitted when applying for funding. Some grantees with AmeriCorps funding before 2015–2019 may 

have had AmeriCorps members deliver services, and the evidence for the intervention’s effectiveness submitted 

by these grantees reflects their use. Other grantees may not have had prior AmeriCorps funding and therefore 

submitted evidence for their intervention’s effectiveness that did not include AmeriCorps member involvement.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

x 

B. Implications of the findings 

These findings lead Mathematica to make three recommendations to help AmeriCorps build a 

larger portfolio of interventions that work. By increasing the proportion of funded interventions 

with evidence of effectiveness—while also funding potentially innovative programs that have not 

yet had time to demonstrate their effectiveness— AmeriCorps would likely be better able to 

improve outcomes of participants, thereby improving lives and strengthening communities. 

Our three recommendations include the following: 

• Help grantees conduct studies that rigorously assess whether their intervention has 

an impact. Of the 82 interventions from funded grants that had a strong or moderate 

AmeriCorps evidence rating, 39 percent met the Mathematica effectiveness standards 

developed for this project. Most of the interventions that did not meet these standards did 

not provide studies that were rigorously designed or executed in such a way that study 

outcomes could be attributed to the intervention and not to other factors. 

• When funding interventions, consider prioritizing those that show impacts across 

multiple outcomes. AmeriCorps funds interventions in different phases of their 

development and with a range of evidence supporting them. Some interventions are likely 

innovative and in the process of building the evidence needed to show that they work. For 

these interventions, AmeriCorps could work with grantees to develop their intervention 

model to build evidence of its effectiveness once the model is fully developed. In most 

cases, however, AmeriCorps funds interventions with a fully developed model. For these 

interventions, rigorous research providing evidence of their effectiveness is needed; without 

such evidence, it is unclear whether these interventions funded by AmeriCorps improve 

participant outcomes. 

• Target funding for scaling at effective interventions. Findings from this study suggest 

that the interventions that AmeriCorps funds have the potential to impact communities, 

especially in the areas of education and economic opportunity. Should AmeriCorps target 

funding for scaling interventions that work and show a readiness to be scaled, the agency’s 

ability to help build strong communities would likely be enhanced. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

1 

I. INTRODUCTION

Agencies throughout the federal government and other grant-making institutions are increasingly 

conducting systematic evidence reviews and synthesizing the results of well-executed research to 

identify interventions that have evidence of effectiveness. These efforts can help funders make 

strategic decisions about how they can most effectively and efficiently address individual and 

community needs. These evidence reviews can also enhance funders’ decisions about scaling up 

interventions that show the most promise in order to expand the scope of their impact. 

AmeriCorps was established as a federal agency in 1993 with a mission to improve lives, 

strengthen communities, and foster civic engagement through service and volunteering.5 The 

AmeriCorps State and National program and the three AmeriCorps Seniors programs (RSVP, 

Senior Companions, and Foster Grandparent) are the main avenues through which the agency 

currently achieves this mission; the Social 

Innovation Fund (SIF) also provided funding 

to innovative community-based programs 

from 2010 to 2016. 

In 2017, an estimated 73,000 AmeriCorps 

members at more than 20,000 locations across 

the country carried out programs in the 

agency’s statutory focus areas (see sidebar).6

Examples of the services members provided 

include the following (AmeriCorps 2017): 

• Supporting youth through teaching, 

tutoring, mentoring, and after-school 

programs 

• Assisting Americans affected by 

hurricanes Harvey, Irma, and Maria 

• Working with nonprofits, police 

departments, and local agencies to prevent 

and reduce opioid abuse 

• Providing veterans and their families with 

employment services, benefits counseling, 

and transportation to medical 

appointments 

5 As of September 29th, 2020, the Corporation for National and Community Service is operating under the name 

AmeriCorps. 

6 Because the three AmeriCorps Seniors programs were not part of the analysis undertaken in this report, we do not 

discuss them. AmeriCorps (2016) provides a more detailed description of AmeriCorps’ programs. 

AmeriCorps focus areas

• Disaster services helps communities across 

the country prepare for, respond to, and 

recover from natural and other disasters.

• Economic opportunity addresses housing, 

financial literacy, and employment needs of 

low-income individuals and families. 

• Education helps communities design and 

implement results-driven, cost-effective 

solutions that support students from cradle to 

career.

• Environmental stewardship (environment) 

trains youth and unemployed and 

underemployed citizens in conservation and 

green jobs, reconnects Americans to the 

outdoors, and supports successful sciencebased conservation strategies. 

• Healthy futures uses a three-pronged 

approach of health-focused assistance, 

prevention, and intervention to educate and 

maintain healthy communities. 

• Nonprofit organizational capacity (org 

capacity) helps organizations expand the 

reach, efficiency, or effectiveness of their 

programs. 

• Veterans and military families (veterans) 

serves and engages the talents of veterans 

and military families.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

2 

• Partnering with mayors, police departments, and local nonprofits to reduce crime, rebuild 

housing, remove blight, and expand economic opportunity in urban areas

Community-based programs supported by the SIF addressed challenging social problems in the 

focus areas of economic opportunity, healthy futures, and youth development (AmeriCorps 

2015). Programs funded include those with the following objectives:

• Increase access to depression treatment in low-income rural communities

• Turn around low-achieving schools 

• Increase tax savings by opening banking accounts for low- to moderate-income families

• Increase employment by providing transitional employment and supports to stabilize lives 

• Use sports to improve body mass indices and waist circumferences of children in 

underserved urban communities 

Because AmeriCorps strives to be a good 

steward of taxpayer dollars and operate 

programs in a cost-effective manner 

(AmeriCorps 2017), it uses third-party 

evaluations, systematic reviews, and metasynthesis analyses to build knowledge of its 

funded interventions with evidence of 

beneficial impacts on participants (Zhang 

and Sun 2016; JBS International 2015). In 

addition, it funded the Scaling EvidenceBased Models project in 2016 to gain 

insights about which of the AmeriCorps 

State and National and SIF interventions 

were effective and might be ready to scale. 

Conducted by Mathematica, this project is 

designed to generate practical knowledge 

about how AmeriCorps might foster the 

successful scaling of effective interventions 

and help support AmeriCorps’ efforts to 

identify which interventions work and how 

they can work for more people. This 

information will also help inform the 

agency’s interest in identifying the 

intervention components that are critical for 

an intervention’s effectiveness. 

Research questions guiding the project 

How do organizations define scaling?

• How is scaling defined and operationalized by 

AmeriCorps-funded organizations? 

What interventions work?

• What interventions consistently demonstrate 

evidence of effectiveness for their targeted 

outcomes?

• What intervention characteristics demonstrate 

evidence of effectiveness at a scale that 

suggests readiness for replication more widely?

How do organizations scale successful 

interventions?

• What organizational resources are necessary to 

successfully scale interventions?

• What funders (philanthropic, public, and private) 

support scaling efforts?

• How are organizations scaling interventions (up 

or down) with fidelity and integrity?

• How are organizations adapting or modifying 

interventions when they scale?

• What facilitates and what hinders scaling?

• What role does evaluation play in scaling efforts 

(process, outcome, or impact data)?
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

3 

In this chapter, we provide an overview of the project (Section A), explain briefly the evidence 

review and meta-synthesis component of the project that is the subject of this report (Section B), 

and outline the structure for the rest of the report (Section C). 

A. Overview of the Scaling Evidence-Based Models project

The Scaling Evidence-Based Models project was funded to deepen AmeriCorps’ understanding 

of how to scale effective interventions and to answer three overarching research questions (in 

bold) and nine subquestions (indicated by bullets) shown in the sidebar on the previous page. A 

review of the evidence—the subject of this report—addresses the second question and its 

subquestions. The first and third research questions are addressed in an analysis of AmeriCorpsfunded organizations’ scaling plans to understand scaling readiness (Selekman et al. 2020) and 

by a process study that examines three AmeriCorps-funded organizations in depth to learn how 

they scaled their interventions and what factors facilitate or hinder scaling.

The project developed and applied the Scaling Programs with Research Evidence and 

Effectiveness (SPREE) process to help AmeriCorps identify which of its funded interventions 

demonstrate evidence of effectiveness and a readiness to scale (Maxwell and Richman 2019). 

The SPREE process has two distinct parts, each of which is represented in Figure I.1. The first 

part—the subject of this report—is designed to identify AmeriCorps-funded interventions with 

evidence of effectiveness, which we define as those that met the project’s effectiveness standards 

(discussed in Section B) and produced favorable findings on the majority of targeted outcomes 

among participants receiving the intervention. We often refer to interventions that have provided 

evidence of effectiveness as interventions that work. The second part—discussed in “Planned 

Scaling Activities of AmeriCorps-Funded Organizations” (Selekman et al. 2020)—is designed to 

determine the extent to which the effective interventions and the organizations implementing 

them demonstrate a readiness for scaling. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

4 

Figure I.1. The Scaling Programs with Research Evidence and Effectiveness (SPREE)

process

B. Overview of the meta-synthesis study

Because interventions that can improve participant outcomes are needed to build stronger 

communities, the project only considers interventions with evidence of effectiveness as 

possibilities for scaling. The research team developed a process for identifying interventions that 

work (see Appendix A) and then analyzed those interventions in terms of the strength of their 

effectiveness and characteristics. 

1. Identifying the interventions that work with the SPREE process 

The SPREE process shown in Figure I.1 is structured so that interventions that do not work (first 

part of the figure) or are not ready to scale (second part) are systematically eliminated from 

consideration for scaling. Taken together, the three stages in the first part of the figure ensure 

that the interventions being considered for scaling work. These three stages are as follows: 

• Collect evidence. To identify interventions that work, one must first compile an inventory of 

potential interventions of interest, which includes information on the intervention services 

and the target population for the intervention, and the supporting documents that present 

evidence of the intervention’s effectiveness (called evidence documents). These evidence 

documents, which typically take the form of research evaluations of the implemented 

interventions, provide the necessary information for evaluating whether the interventions and 

supporting evidence meet the standards that are set in the next stages of the SPREE process.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

5 

AmeriCorps requires its 

AmeriCorps State and 

National- and SIF-funded 

grantees to provide evidence 

documents that present 

evidence of their interventions’ 

effectiveness. We compiled the 

evidence documents for the 

AmeriCorps State and National 

2015–2019 grantees and SIF 

2010–2015 grantees, which 

were the programs and grantee 

cohorts of interest to 

AmeriCorps that had evidence documents available for review for this project. Although 

most evidence documents submitted were in the form of research evaluations, some were 

literature reviews, descriptive survey results, or process or implementation studies. 

We used AmeriCorps’ tiered-evidence framework (AmeriCorps 2016; see Table I.1) for 

rating evidence to determine which evidence documents we should compile from specific 

AmeriCorps State and National 2015–2019 and SIF 2010–2015 grantees.7 The AmeriCorps 

evidence ratings are applied at the grantee or subgrantee level—not the intervention level—

based on the quality of evidence that grantees submit. Independent contractors and reviewers

for AmeriCorps (other than Mathematica) assign moderate or strong ratings to grantees that 

presented evidence generated from at least one study with a design that, if done well, could 

provide confidence that the intervention improves participant outcomes. Such studies assess 

an intervention’s impacts on participants’ outcomes by studying differences in outcomes 

between those receiving the intervention (treatment group) and a comparable group of 

individuals not receiving the intervention (comparison group). Mathematica compiled the 

evidence documents from grantees with a moderate or strong evidence rating and considered 

only these documents eligible for review.

8

 

7

 Both AmeriCorps State and National and SIF have tiered-evidence frameworks to rate the quality of evidence 

submitted to AmeriCorps. Although the programs differ in the evidence categories used (for example, only 

AmeriCorps State and National uses the pre-preliminary evidence rating) and in the language used to describe the 

criteria for the evidence categories, both reserve the moderate and strong evidence ratings for grantees providing 

evidence based on studies that used rigorous research designs to support causal conclusions. Evidence ratings are 

assigned at the grantee level only for AmeriCorps State and National grantees and at either the grantee or 

subgrantee level for the SIF program, depending on which entity is implementing the intervention. Grantees can 

submit more than one evidence document for all interventions they are seeking to implement with AmeriCorps 

funding. 

8 These are the evidence documents that grantees submitted to AmeriCorps. We did not search for or review other 

evidence documents that exist for these interventions.

Table I.1. AmeriCorps evidence ratings

Evidence rating Criteria

Pre-preliminary (used

by AmeriCorps State 

and National only)

Some data collection and testing

Preliminary Outcome study, such as pre- and post-test

with no comparison group or post-test only

for treatment and comparison groups

Moderate Study designs that support causal

conclusions but have limited generalizability

beyond the study context 

Strong Study designs that support causal

conclusions that assess the intervention

nationally, regionally, or at the state level 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

6 

• Categorize evidence. Because variations 

exist in the quality of evidence presented, it is 

important to assess whether the documents 

meet research standards for evidence of 

effectiveness. Assessment can include 

utilizing existing evidence standards, such as 

those developed by federal research 

clearinghouses, or developing one’s own. 

Ultimately, the collected evidence documents 

would be reviewed under these standards to 

determine the extent to which there is 

confidence that the intervention, and not 

other factors, caused the estimated impacts 

for those receiving the interventions. 

Mathematica developed effectiveness 

standards to assess the internal validity of 

evidence documents (Table I.2) using 

standards created by existing clearinghouses 

(What Works Clearinghouse, Clearinghouse 

for Labor Evaluation and Research, and 

Home Visiting Evidence of Effectiveness). 

Studies with internal validity provide greater 

confidence that any differences in outcomes 

between the treatment and comparison groups 

can be attributed to the intervention and not 

to other factors (Shadish et al. 2002 provide a 

detailed discussion).

• We also categorized the evidence documents against other objectives for the project. We 

created effectiveness standards to determine which interventions were examined with studies 

that had designs that used a comparison group, had methods that were adequately described, 

were conducted by an external evaluator, and were for interventions aligned to a grantee’s 

planned scaling activities (Table 1.2). The first three effectiveness standards (study design, 

reporting on methods, and evaluator independence) helped ensure rigor and objectivity so the 

evidence could show that the intervention caused participant outcomes. Finally, alignment 

with the scaling plan, the fourth criterion, helped ensure that grantees were planning to scale 

the intervention with evidence that it works. For example, a grantee planning to scale a 

specific parent education intervention could not present evidence of another parent education 

intervention as proof of its effectiveness.9

• Identify evidence with positive findings. Because the ultimate goal is scaling interventions 

that have evidence that they work, we must also define what constitutes whether an 

9 See Selekman et al. (2020) for the extent to which grantees proposed to modify these interventions when scaling.

Table I.2. Standards for categorizing

evidence

Standards Evidence has ...

Internal validity

Low attrition Few individuals in the treatment or

comparison group who left the

study

No

reassignment

No individuals randomly assigned

to comparison group switched to

the treatment group or vice versa

Baseline

equivalence

Individuals in the treatment and

comparison groups in the final

sample used for analysis of

outcomes that did not differ on at

least some observable

characteristics

No

confounding

factors

A design that precluded factors

other than the intervention from

potentially being responsible for the

estimated impacts 

Project objectives

Study design A comparison group 

Reporting on

methods 

Adequate information on research

design and statistical approach to

gauge impacts

Evaluator

independence

An evaluator who was external to

the grantee

Scaling plan

alignment

Alignment with the intervention

planned for scaling
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

7 

intervention has positive findings by producing favorable impacts on participants. This

entails establishing (1) how much evidence is needed, whether one or multiple evidence 

documents show favorable findings on participant outcomes; and (2) the prevalence of 

positive findings within a given evidence document, such as whether there must be positive 

findings on the majority of assessed outcomes or whether positive findings for at least one 

outcome are sufficient. 

Because the Scaling Evidence-Based Models project is interested in AmeriCorps grantees’ 

scaling of interventions that work, Mathematica developed effectiveness standards to identify 

which interventions have consistently positive findings. An intervention was defined as 

having consistently positive findings if it showed favorable results on the majority of primary 

outcomes that the intervention sought to improve in at least one evidence document.10

Overall, the Mathematica effectiveness standards for interventions—that is, studies having 

evidence with internal validity, aligning evidence to project objectives, and having positive 

findings—build on the AmeriCorps evidence ratings assigned to grantees. Mathematica’s 

effectiveness standards and AmeriCorps evidence ratings for grantees have similarities but also 

important differences. For example, Mathematica’s effectiveness standards include studies 

providing detailed technical information on the research design’s internal validity (for example, 

demonstrating baseline equivalence between the individuals in the treatment and comparison 

groups in the final analytic sample), which is not part of AmeriCorps State and National’s 

evidence rating determinations. The AmeriCorps State and National and SIF programs’ evidence 

ratings only required positive findings on one outcome of interest to quality for a moderate or 

strong rating. Mathematica’s effectiveness standards required consistency of positive findings 

across the range of outcomes that were assessed in an evaluation. Such differences mean that a 

grantee can receive a strong or moderate AmeriCorps evidence rating but its intervention may 

not meet Mathematica’s effectiveness standards. An important similarity between the 

Mathematica effectiveness standards and the AmeriCorps evidence ratings is that both are 

intended to be applied to studies of interventions implemented in a diverse set of focus areas (for 

example, interventions in the areas of education, economic opportunity, and environmental 

stewardship). Although this approach provides consistency in how these studies are reviewed, we 

recognize that conducting rigorous research evaluations or accurately measuring intervention 

impacts might be especially difficult for certain types of outcomes or with specific target 

populations. 

Additionally, since our review was focused on evidence documents from grantees given a 

moderate or strong evidence rating from the AmeriCorps State and National 2015–2019 and SIF 

2010–2015 funding years, we needed to consider whether there were any changes in how either 

of the AmeriCorps programs described the evidence requirements and guidelines for the 

interventions eligible for funding over time. For example, the AmeriCorps State and National 

program modified its Notice of Funding Opportunity (NOFO) requirements and guidelines in

2019 to place a higher value of evaluation points on applicants’ evidence base relative to prior 

10 For AmeriCorps’ evidence review, a grant could receive a strong or moderate evidence rating if the evidence 

submitted achieved a favorable result on only one priority outcome. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

8 

years. Applicants were also instructed to ensure that the proposed intervention clearly aligned

with the evidence base on characteristics of the beneficiary population, characteristics of the 

population delivering the intervention, dosage and design of the intervention, the context in 

which the intervention is delivered, and outcomes of the intervention. Another example is that 

under the 2015 NOFO, applicants could receive a moderate evidence rating using nonexperimental studies, whereas the 2016-2019 NOFOs required experimental or quasiexperimental studies to attain a moderate evidence rating. The effectiveness standards developed 

for this project created an opportunity to apply the same criteria to all grantees, regardless of 

funding year. 

2. Describing interventions that work

Mathematica used the rubric in Appendix A to conduct a systematic review of each evidence 

document provided by grantees with a strong or moderate AmeriCorps evidence rating to 

determine which interventions had evidence that they work. Mathematica extracted information 

from each evidence document to summarize the intervention and provide a high-level assessment 

of its evidence of effectiveness. If multiple evidence documents were submitted for a single 

intervention, summaries were compiled across the evidence documents to determine the overall 

evidence of effectiveness for that intervention. This process was used to answer the research 

question “What interventions consistently demonstrate evidence of effectiveness for their 

targeted outcomes?”

The research question “What intervention characteristics demonstrate evidence of effectiveness 

at a scale that suggests readiness for replication more widely?” is complicated to answer because 

of the diverse set of interventions that AmeriCorps funds. Although this diversity allows 

AmeriCorps to offer communities an opportunity to select from an array of interventions to meet 

their needs, it presents challenges for synthesizing and comparing research findings because 

funded interventions produce a variety of outcomes for participants (for example, improvements 

in test scores, reductions in body mass index, and increases in employment rates). To address this 

issue, we used an effect size index. An effect size index provides a uniform measure of the 

magnitude of outcome differences between the treatment and the comparison groups across 

different types of interventions and quantifies how effective an intervention is. This approach 

affords an opportunity to compare the effectiveness of different interventions with various 

participant outcomes. Appendix C describes the specifics of what effect sizes are, how they are 

calculated, and how they were used in the report’s meta-analysis. 

C. Road map for the report

This report details our findings from analyses of AmeriCorps-funded interventions with evidence 

that they work in improving participant outcomes. In Chapter II, we describe results of the 

process to identify those interventions that work and their characteristics, and in Chapter III, we 

present our findings from the meta-analysis on the magnitude of the impacts of the interventions 

that work. We provide a summary of the main findings and highlight implications from them in 

Chapter IV. We include three appendices to support the discussion in the chapters. In Appendix 

A, we provide a copy of the rubric used to extract information from the evidence documents; in 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

9 

Appendix B, we provide the characteristics of the 32 interventions with evidence that they work; 

and in Appendix C, we provide the technical details (such as how effect sizes were calculated) 

and data tables for the meta-synthesis analysis. 
This page has been left blank for double-sided copying.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

11

II. AMERICORPS-FUNDED INTERVENTIONS WITH 

EVIDENCE THEY WORK

AmeriCorps funds a multitude of interventions that seek to address the needs of local 

communities through volunteerism and national service. Because AmeriCorps is interested in 

fostering the successful scaling of effective interventions to increase the number of individuals 

and communities positively affected by these services, the agency must identify which funded 

interventions work. In this chapter, we present the results from the process used to identify and 

describe the AmeriCorps-funded interventions that met Mathematica’s effectiveness standards. 

We answer the research question “What interventions consistently demonstrate evidence of 

effectiveness for their targeted outcomes?” by answering these two subquestions: 

• Which AmeriCorps-funded interventions demonstrate evidence of effectiveness on 

participants’ outcomes? 

• Among interventions with evidence of effectiveness, what are their characteristics?

We answered the research questions with our systematic review of evidence documents. We 

applied the evidence review process discussed in Chapter I to a set of interventions funded 

through 154 grants with continued funding and a strong or moderate AmeriCorps evidence rating 

as determined by third-party evaluators. These 154 grantees planned to implement 82 

interventions. In Section A of this chapter, we describe the process and corresponding analysis 

used to determine which interventions work and why interventions without evidence of 

effectiveness did not meet evidence standards. For the interventions that do work, we describe 

their characteristics in Section B. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

12

A. Identifying AmeriCorps-funded interventions with evidence of effectiveness

Figure II.1 illustrates the process that produced the assessment that 32 AmeriCorps-funded 

interventions provided evidence that the interventions work: 

• Of the 1,104 AmeriCorps State and National 2015–2019 and SIF 2010–2015 grantees, 154 

(14 percent) had a strong or moderate AmeriCorps evidence rating and continued funding. 

Independent contractors and reviewers for AmeriCorps (other than Mathematica) assessed 

the studies submitted by grantees for their interventions as potentially able to produce causal 

evidence that their interventions work. These 154 grantees planned to implement 82 

interventions. 

Key findings

• Thirty-two interventions (39 percent) out of the 82 interventions from grants with a 

strong or moderate AmeriCorps evidence rating met Mathematica’s effectiveness 

standards that indicate they improved participant outcomes. 

• Interventions did not meet Mathematica’s effectiveness standards primarily for these 

reasons (which were relatively similar across funding years): 

- They did not have studies with internal validity (46 percent): 87 percent of these 

did not show baseline equivalence between the treatment and comparison 

groups.

- They did not have consistently favorable findings for the intervention (44

percent). 

- They did not have studies with a comparison group (40 percent).

• The 32 interventions that were assessed as working had these characteristics:

- They fell primarily in the education focus area (56 percent), with 28 percent and 

16 percent falling into the economic opportunity and the healthy futures focus 

areas, respectively. The predominance of education interventions reflects, at 

least in part, the preponderance of AmeriCorps interventions in the education 

focus area: Of the AmeriCorps State and National 2015–2019 and SIF 2010–

2015 interventions included in the full sample, 54 percent were in the education 

focus area. 

- They were predominately implemented in urban settings (97 percent), with 16 

percent implemented in rural areas (most of these were also implemented in 

urban settings). 

- They were distributed across the geographic regions in the United States, with 

about half implemented in multiple geographic regions (47 percent).

- Some (29 percent) had AmeriCorps members involved in the service delivery 

model. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

13

• Thirty-two (39 percent) of the 82 interventions implemented by grantees with a strong or 

moderate AmeriCorps evidence rating met Mathematica’s effectiveness standards. The other 

50 interventions (61 percent) had studies that did not consistently show favorable findings for 

the intervention; did not have a comparison group; did not have sufficient detail to allow an 

assessment of the study; or did not have internal validity, mainly because of a lack of 

equivalence at baseline between the treatment and comparison groups. 

Figure II.1. Identifying interventions with evidence of effectiveness

SIF = Social Innovation Fund.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

14

Figure II.2 displays the percentage of interventions implemented by grantees with strong or 

moderate AmeriCorps evidence ratings that met Mathematica’s effectiveness standards by the 

grantee funding year. Though the percentage of interventions that met effectiveness standards

ranged from 29 percent (in 2014) to 63 percent (in 2011), we caution readers to interpret these 

numbers with care given the small number of these grants in a given funding year. 

Figure II.2. Percentage of interventions with studies that had evidence of effectiveness

Source: Eighty-two interventions from AmeriCorps State and National 2015–2019 and SIF 2010–2015 grantees with 

a strong or moderate AmeriCorps evidence rating that were reviewed with Mathematica’s effectiveness 

standards. 

Note: The number of interventions that met or did not meet the effectiveness standards in a funding year are 

displayed within the chart bars. The total number of interventions do not add to 82 because an intervention 

could have been funded across multiple years.

Figure II.3 illustrates the reasons that 61 percent of interventions implemented by grantees with 

strong or moderate AmeriCorps evidence ratings did not meet Mathematica’s effectiveness 

standards. As we discussed in Chapter I, we used a two-stage process to assess evidence 

documents, with internal validity assessed at the end of the process, so we caution readers to 

interpret these numbers with care.11 

11 Because grantees could submit multiple evidence documents, an intervention could have had one study that met 

the effectiveness standards and another that failed to do so. Interventions continued through the two-stage review 

process (where we first reviewed documents against the project objective and positive findings criteria before 

applying the internal validity criteria) as long as they had at least one evidence document that met effectiveness 

standards. However, this means that the internal validity criteria were not applied to the evidence documents 

screened out at the first stage of the review process, so it is possible that the percentage of interventions with 

studies not having internal validity could be greater than what is reported in Figure II.2.

0%

20%

40%

60%

80%

100%

Year

2010

(n = 16)

Year

2011

(n = 8)

Year

2012

(n = 9)

Year

2014

(n = 7)

Year

2015

(n = 17)

Year

2016

(n = 24)

Year

2017

(n = 9)

Year

2018

(n = 10)

Year

2019

(n = 19)

Did not meet evidence of effectiveness standards

Met evidence of effectiveness standards
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

15

Figure II.3. Reasons interventions were without evidence of effectiveness: Percentage 

with studies that did not meet each standard

Source: Fifty interventions from AmeriCorps State and National 20152019 and SIF 2010–2015 grantees with a 

strong or moderate AmeriCorps evidence rating that did not provide evidence documents that met 

Mathematica’s effectiveness standards. 

Note: Percentages do not add to 100 percent because interventions could be represented in more than one 

category; multiple evidence documents submitted for an intervention could have failed to meet different 

effectiveness standards, or a single evidence document could have failed to meet more than one 

effectiveness standard. Some of the interventions that did have evidence that they work also had other 

studies that did not meet these effectiveness standards. These interventions are not included in the 

percentages presented in the figure. 

As this figure shows, there were several reasons for not meeting the effectiveness standards: 

• Nearly half (46 percent) of the interventions did not have studies with internal validity. 

Of the 23 interventions that did not have studies with internal validity, 87 percent provided 

evidence documents that did not demonstrate baseline equivalence between the treatment and 

comparison groups. Sometimes, studies used statistical techniques (for example, propensity 

score matching) to create similar treatment and comparison groups but did not conduct 

analyses that showed the groups for whom outcomes were compared were similar before the 

intervention. Another 13 percent had the presence of other factors (confounds) that might 

explain outcome differences between the treatment and comparison groups. 

• Almost half of the interventions (44 percent) did not have studies that showed 

consistently favorable impacts. Although the studies submitted for these interventions 

sometimes showed some statistically significant favorable findings, the majority of findings 

related to the studies’ primary research questions were not statistically significant.12 

• About 40 percent of the interventions did not have studies with a comparison group. 

These studies include those that presented descriptive survey results of program participants, 

process studies, literature reviews, and outcome studies with program participants only.

12 Statistical significance was defined at the p < 0.05 level.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

16

• About 14 percent of interventions had evidence documents that were not aligned with 

the grantees’ proposed scaling. Some grantees proposed scaling a combination of 

interventions but did not submit evidence documents for each of the interventions. For 

example, one grantee intended to use several employment training interventions for its young 

adult target population but only submitted evidence for one of those interventions. 

• About 12 percent of interventions did not have evidence documents with sufficient 

information about their research methods. Such studies often did not discuss in detail how 

the data were collected and analyzed.

The reasons interventions were without evidence of effectiveness did not seem to vary by a 

function of funding year. 

B. Characteristics of interventions that work

Mathematica examined the characteristics of the 32 interventions with evidence of effectiveness 

with respect to how they align to the AmeriCorps focus areas and priority objectives and to the 

agency’s mission to use national service to help all types of communities across the country. In 

this section of the report, we describe the general characteristics of these 32 interventions.13

Appendix B provides a detailed description of each intervention with respect to its focus area, 

priority objective, core elements, implementation setting, and inclusion of AmeriCorps members 

in the service delivery model. 

1. Interventions in the education, economic 

opportunity, and healthy futures focus 

areas had evidence that they work. 

Among the interventions with evidence that they 

work, most were in the focus area of education (56 

percent), 28 percent were in economic opportunity, 

and 16 percent were in healthy futures (see Table 

II.1).14 None of the interventions that focused on 

environmental stewardship, nonprofit 

organizational capacity, veterans and military 

families, and criminal justice had research evidence 

suggesting that they work. Some of these 

differences arise from the distribution of funded interventions in the sample of grants that had a 

13 In this report, we do not identify the 32 individual interventions by their proper names. Instead, we refer to them 

by the AmeriCorps focus area to which they align (for example, Economic Opportunity 1 through 4, Education 1 

through 13, Healthy Futures 1 through 4). We use this type of terminology so that the focus of the report is on the 

broader portfolio of effective interventions that AmeriCorps funds and the evidence grantees submit to support 

their effectiveness, not on the effectiveness of specific interventions per se.

14 One intervention (Education 12) was classified under two AmeriCorps focus areas: education and healthy futures. 

We discuss this intervention under the education focus area because the study submitted evidence for this 

intervention that mostly assessed whether the intervention improved academic outcomes. 

Table II.1. Interventions by focus area

 Percentage of 

interventions

 That work Funded

Education 56% 54% 

Economic opportunity 28% 22% 

Healthy futures 16% 15% 

Environment 0% 5% 

Org capacity 0% 2% 

Veterans 0% 1%

Other (Criminal justice) 0% 1% 

Disaster services 0% 0%

Number 32 82
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

17

strong or moderate AmeriCorps evidence rating. Interventions in economic opportunity 

represented a relatively larger percentage of interventions that work (28 percent) than their share 

of the funded sample (22 percent). Interventions in the focus areas of environmental stewardship 

represented a relatively smaller proportion of those showing evidence of working (0 percent)

than their share of the funded sample (5 percent). The proportion of education and healthy 

futures interventions that work was similar to the proportion of those in the funded sample. There 

were no interventions with research evidence suggesting that they work in the focus areas of 

nonprofit organizational capacity and veterans and military families, though these represented a 

small share of the funded sample. There were no interventions that focused on disaster services 

in the funded sample. Overall, though AmeriCorps’ funded interventions with a strong or 

moderate evidence rating were distributed across six of its focus areas (and included an 

intervention focused on criminal justice, which is not a designated AmeriCorps focus area), the 

interventions with research evidence suggesting that they work were concentrated within three 

focus areas, with the majority of them in the education area.

2. The education and economic opportunity focus areas, but not the healthy futures 

focus area, had interventions that were assessed as working in each of their priority 

objectives. 

To achieve its goals for improving individual and community outcomes across all focus areas, 

the AmeriCorps strategic plan identified priority objectives in each focus area. AmeriCorps sets 

priority objectives to help the agency focus its resources and ensure that the agency and its 

grantees are working toward common goals. Our analysis of the 32 interventions that work 

showed that all education and economic opportunity priority objectives were addressed by these 

interventions, but this was not the case for all healthy futures priority objectives.

15

a. Education 

The 18 interventions in the education focus area that 

provided evidence of working spanned across the three 

priority objectives: 

• Eleven interventions (61 percent) focused on K–12 

success. These interventions provided a wide range of 

services for different age groups, from literacy 

development programs in elementary schools to one-onone tutoring for high school students. 

• Five interventions (28 percent) focused on postsecondary preparation support in high 

schools, such as providing students with information on the financial aid process and support 

during and after the transition from secondary to postsecondary school. 

15 An intervention can be aligned with more than one priority objective. We discuss one intervention (Education 5) 

under the K–12 success priority objective because the majority of outcome measures aligned with this priority 

objective. Although we did not directly examine this issue, it is possible that, as noted in Chapter I, conducting 

rigorous research evaluations or accurately measuring intervention impacts has been especially challenging in 

some priority objective areas.

Education priority 

objectives

Improve

• K–12 success 

• Postsecondary education

support

• School readiness
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

18

• Two interventions (11 percent) focused on school readiness. These interventions included 

literacy development support in preschools and home visiting for parents of young children 

to develop children’s literacy skills. 

b. Economic opportunity 

The nine interventions in the economic opportunity focus area that provided evidence of working 

spanned across the three priority objectives: 

• Six interventions (67 percent) addressed employability 

through employment or education assistance, internships, 

job training, or other employment supports. 

• Three interventions (33 percent) targeted financial 

literacy and used three distinct strategies. One 

intervention (which also targeted employment) provided 

financial counseling and income support counseling. 

Another intervention provided financial assistance in the form of matched savings accounts, 

in which participants could deposit some of their tax refunds and earn an incentive. The third 

intervention provided financial and benefits counseling and tax preparation services. 

• One intervention (11 percent) targeted housing by building, rehabilitating, or repairing

homes. 

c. Healthy futures

Only one intervention in the healthy futures focus area provided evidence of effectiveness that 

aligned to a priority objective. This sports-based intervention targeted childhood obesity by 

promoting physical exercise and healthy nutrition among 

participating children. None of the interventions targeted the 

priority objectives of increasing access to health care or 

facilitating aging in place. Four other interventions that were 

in the healthy futures focus area and had evidence of 

effectiveness did not align with a designated healthy futures 

priority objective. They were designed to foster supportive 

family environments and reduce child maltreatment and 

abuse by providing behavioral skills training with and home 

visits to parents and families.

Although the actual services that each intervention offered varied by focus and topic area—for 

example, education interventions may have offered academic tutoring, whereas economic 

opportunity interventions may have offered job search and placement services—we identified 

two common service delivery modes across all focus areas:16

• Formal education and training. Nearly all interventions—28 of the 32 interventions (88 

percent)—included a formal education or training component, meaning that program 

participants received a standardized curriculum, delivered either individually or in a group 

16 Service delivery modes can also be referred to as intervention strategies or components.

Healthy futures 

priority objectives

Address

• Childhood obesity 

• Access to health care

• Aging in place

Economic opportunity

priority objectives

Improve

• Employability

• Financial literacy 

• Housing
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

19

setting. This component included delivery of academic and test preparation curricula to 

students in the education area, job skills and readiness training for adults in the economic 

opportunity area, and recreational and sports instruction for youth in the healthy futures area.

• One-on-one support. A total of 23 of the 32 interventions (72 percent) included a one-onone service component, meaning that staff met individually with program participants to 

deliver services. This component included mentoring for students in education interventions, 

financial counseling for adults in economic opportunity interventions, and individual home 

visits to deliver parenting support in the healthy futures area. 

3. Interventions that work were implemented in urban and rural settings with fairly 

even distribution across geographic regions. 

AmeriCorps is dedicated to helping address the needs of communities across the United States 

and working to ensure that all types of communities, including underserved rural areas, are 

included. Our analysis suggests that the interventions that work were implemented in both urban 

and rural areas and in communities across the nation: 

• Almost all interventions (97 percent) were implemented in urban areas, with 16 percent 

implemented in rural areas. Four interventions (13 percent) that work implemented in 

urban areas—three in the education focus area and one in the healthy futures focus area—

were also implemented in rural areas. One healthy futures intervention was implemented 

exclusively in rural areas. 

• Interventions were implemented across the country. About half of the interventions with 

evidence that they work (47 percent) were implemented across multiple geographic regions. 

Regional breakdowns indicate that 50 percent were implemented in the Northeast, 41 percent 

in the West, 34 percent in the South, and 28 percent each in the Midwest. The regional 

distribution of interventions did not seem to vary greatly across the focus areas. 

4. More than one-quarter (29 percent) of interventions that work had AmeriCorps 

members involved in providing services directly to participants or supporting service

delivery. 

AmeriCorps’ mission uses national service participants to strengthen communities and improve 

outcomes for individuals.17 Identifying and strengthening effective interventions that use 

national service helps the agency achieve this goal. Although these national service participants

can provide a wide range of services for grantees, our analysis suggests that nine interventions 

(29 percent) that work used AmeriCorps members to deliver services directly to beneficiaries or 

17 This report uses the phrase national service participants to refer to all of those who dedicate time and skills to 

address local community needs through AmeriCorps’ national programs. AmeriCorps refers to individuals serving

through the AmeriCorps State and National program as members and the three AmeriCorps Seniors programs 

(RSVP, Senior Companions, and Foster Grandparent) as volunteers. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

20

support the delivery of essential functions in implementing the intervention.18 This percentage 

varied little across the focus areas. The direct services provided included tutoring, site 

coordination, and college planning supports in the education focus area; employment or financial 

counseling in the employment opportunity area; and home visitations that delivered parent 

education and made referrals to support services. The remaining 71 percent of interventions that 

met Mathematica effectiveness standards were supported by evidence documents in which 

AmeriCorps members were not a part of delivering intervention services.19

18 All interventions currently funded by the AmeriCorps State and National program are required to include 

AmeriCorps members in the interventions’ service delivery model. In this report, we describe whether 

AmeriCorps members were part of the service delivery model as described in the evidence documents that 

grantees submitted when applying for funding. Some grantees with AmeriCorps funding before 2015–2019 may 

have had AmeriCorps members deliver services, and the evidence for the intervention’s effectiveness submitted 

by these grantees reflects their use. Other grantees may not have had prior AmeriCorps funding and therefore 

submitted evidence for their intervention’s effectiveness that did not include AmeriCorps members’ involvement. 

19 Our review determined AmeriCorps members’ involvement in delivering intervention services based on evidence 

documents’ explicitly referencing AmeriCorps members in the description of the intervention. It is possible that 

some evidence documents did not provide sufficient information on the personnel involved in the intervention 

delivery, including the presence of AmeriCorps members. As a result, our finding that 29 percent of interventions 

that work had AmeriCorps members involved in providing services can be viewed as a conservative estimate of 

AmeriCorps member involvement in these interventions. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

21

III. IMPACTS OF INTERVENTIONS WITH EVIDENCE THEY 

WORK

In this chapter, we report results of a meta-analysis that estimated the size of the effects for the 

interventions identified in Chapter II as having evidence of effectiveness. We address the 

project’s research question “What intervention characteristics demonstrate evidence of 

effectiveness at a scale that suggests readiness for replication more widely?” by answering two 

subquestions: 

• How effective are AmeriCorps interventions with evidence of effectiveness?

• What characteristics of the interventions might be associated with the estimated impacts? 

We answered the first question using effect sizes to measure the magnitude of an intervention’s 

impact and to compare the effectiveness of interventions with different participant outcomes 

uniformly. We were able to calculate at least one effect size for 30 of the 32 interventions, which 

include effect sizes for 1,408 outcomes, 419 of which were outcomes that corresponded to an 

evaluation’s stated main focus (often referred to as primary outcomes).20 An effect size is a 

standard measure of the extent to which individuals in the treatment and comparison groups are 

different from each other on various types of study outcomes (for example, reading achievement, 

income levels, and positive parenting behaviors). Without a standard measure, it is difficult to 

compare the relative impacts of interventions across 

different focus areas (for example, comparing education 

interventions to healthy futures interventions). For all 

analyses, we calculated weighted average effect sizes (the 

overall impact estimates across all assessed outcomes) to 

capture the intervention’s impact and used a 95 percent 

confidence interval (CI) to show the range in which the 

actual impact falls with a 95 percent probability. We 

answered the second question by comparing effect sizes 

across the interventions’ characteristics: focus areas, 

priority objectives, geographic locations, and role of 

AmeriCorps members. 

In Section A of this chapter, we present results that answer the first research question, and in 

Section B, we provide results that answer the second question. See Appendix C for details on 

how we calculated effect sizes and conducted the analysis. 

20 The information on two interventions (Economic Opportunity 9 and Healthy Futures 5; see Appendix B) was 

insufficient to calculate effect sizes for this analysis. The study reported effect sizes based on gain scores, which 

are not comparable to those based on means. We also did not have enough information to calculate Hedges’ g or 

the Cox index for 161 outcomes across all of the interventions.

What is an effect size?

An effect size is a standard 

measure of the extent to 

which individuals in the 

treatment and comparison 

groups are different from 

each other on various types 

of study outcomes. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

22

A. Magnitude of effects of interventions with evidence they work 

Figure III.1 illustrates the average effect sizes for the interventions with evidence that they work. 

This figure displays the average effect size across all the identified interventions (displayed as 

“overall”) and for each individual intervention in three AmeriCorps focus areas. This figure, like 

all figures in this chapter, displays effect sizes in different colors and shapes. The confidence 

interval—which shows the range in which the actual effect size is likely to fall with a 95 percent 

probability—is shown by the horizontal line running through each colored shape. When this line 

crosses the 0 point, which is shown by a vertical line from the x-axis, we are not certain if the 

intervention has an impact. When the horizontal line does not cross the 0 point, we are at least 95 

percent certain that the intervention has a beneficial impact. 

Key findings

Participants in AmeriCorps-funded interventions with evidence that they work had 

better outcomes than those who did not participate. The average intervention participant 

outperformed those not receiving the intervention by 11 percentage points in beneficial

outcomes. 

• Different focus areas had different impacts. The average participant in interventions in 

the economic opportunity focus area had better outcomes than 64 percent of those who 

did not participate. The average participant in interventions in the education and healthy

futures focus areas had better outcomes than 59 percent of their comparison group 

counterparts.

• Most priority objectives had impacts. The interventions in the employability, K–12 

success, postsecondary education support, and school readiness priority objectives were 

likely to have positive impacts. Although not an AmeriCorps priority objective, 

interventions classified as focusing on supportive family environments were also likely to 

have a positive impact.

• Impacts did not vary by use of AmeriCorps members. These results suggest that 

AmeriCorps members are as effective as other types of intervention personnel at 

delivering services.

• Impacts did not vary by geographic region or by funding year. These results suggest 

that interventions might be equally effective across the United States. The results also 

suggest that AmeriCorps has year-to-year consistency in identifying interventions 

supported by rigorous evidence of effectiveness.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

23

Figure III.1. Overall and intervention average effects for primary outcomes

Note: Each line, except the overall line, represents a single intervention with the focus area of the intervention 

designated by different colors, shapes, and labeling. Appendix Table C.1 provides detailed information on 

the average effect size for the individual interventions.

Econ = economic opportunity; Educ = education; Health = healthy futures.

The AmeriCorps interventions with evidence that they work had a statistically significant 

average effect size of 0.28. The What Works Clearinghouse (WWC) improvement index (What 

Works Clearinghouse 2020, pp. 17–18) suggests that this effect size means that the average 

person receiving an AmeriCorps-funded intervention (treatment group) would perform better 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

24

than 61 percent of individuals who did not receive the intervention (comparison group). Because 

the average treatment group member without the intervention would outscore 50 percent of the 

comparison group, the 0.28 effect size indicates an 11 percentage point increase in beneficial

outcomes. The average effect size for the individual interventions varied, with most interventions 

(21 of 30) having overall impacts that were not statistically significant, as indicated by the fact 

that the 95 percent confidence interval lines cross zero. Of the nine interventions with significant 

impacts, (1) three were economic opportunity interventions focused on employability; (2) five 

were education interventions, with three focused on K–12 success, one on postsecondary 

education support, and the other one on school readiness; and (3) one was a healthy futures 

intervention focused on supportive family environments.21 

This pattern of findings appears comparable to previous analyses of the overall effectiveness of 

AmeriCorps-funded interventions, with Zhang and Sun (2016) finding that interventions 

implemented by SIF grantees had an overall average effect size of 0.28. However, it is important 

to note how the previous analysis and the one performed for this study differ on the interventions 

examined. The sample of interventions analyzed by Zhang and Sun (2016) were SIF-funded 

programs with AmeriCorps evidence ratings ranging from preliminary through strong. The 

Mathematica findings reported here include interventions funded through AmeriCorps State and 

National and SIF with grantees having a strong or moderate AmeriCorps evidence rating, and 

interventions had rigorous research designs, showed consistently favorable findings, and aligned 

with plans for scaling. Because of these differences, especially those related to the level of 

evidence required to support the interventions’ effectiveness, readers should use caution when 

comparing magnitudes of effects discussed in this study to other meta-analyses performed with 

AmeriCorps-funded interventions. 

B. Magnitude of effects by intervention characteristics

To assess whether the effect sizes significantly varied by any intervention characteristics, we 

compared the average effect sizes for interventions across focus areas, priority objectives, 

geographic region (single versus multiple regions), the involvement of AmeriCorps members in 

delivering services or supporting the delivery, and funding 

year. Rural/urban comparisons could not be made because 

only one intervention operated exclusively in rural areas. 

Figure III.2 displays the average effect sizes for the 

interventions by the focus area and priority objectives under 

which they were classified. The solid lines represent the 

average effect sizes for the AmeriCorps focus areas, and the 

dashed lines correspond to average effect sizes of the priority objectives in the focus area. Some 

focus areas and priority objectives did not have interventions with evidence of effectiveness; 

they are not displayed in the figure. As the figure shows, interventions in the economic 

opportunity area had the largest average effect size (0.36), which translated into a 14 percentage 

21 The average effect size information presented in Figure III.1 and Appendix Table C.1 is based on the analysis of 

the studies’ priority outcome measures. See Appendix Table C.4 for the average effect size information based on 

all outcome measures, which is not represented in Figure III.1. 

Key finding

Interventions in the economic 

opportunity focus area had 

the largest average effect 

size. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

25

point increase in favorable outcomes for participants. This finding is largely driven by the sizable 

impacts found for interventions Econ2, Econ3, and Econ7 (shown in Figure III.1). Effect sizes 

for interventions in the education focus area were also large (0.24); they correspond to a 9 

percentage point increase in favorable outcomes for participants. The healthy futures focus area 

had an effect size of similar magnitude (0.22) that was also statistically significant. 

Figure III.2. Intervention average effect sizes by focus area and priority objective

Note: Solid lines indicate focus area, and dashed lines indicate priority objectives (POs). The parentheses show 

the number of interventions in each characteristic. Focus areas and POs without interventions with evidence 

that they work are not shown. Education 5 was categorized under the K–12 success priority objective, and 

Economic Opportunity 1 was categorized under employability based on the majority of outcome measures 

assessed for these interventions. Although the supportive family environments area is not a designated 

AmeriCorps priority objective, we display the average effect size for the interventions categorized under this 

area.

The size of the effects also varied for interventions aligned with different priority objectives. In 

the economic opportunity focus area, interventions that focus on employability (0.39) appeared 

to outperform those that focus on financial literacy (0.15). In the education focus area, 

interventions that focus on school readiness produced the largest effect sizes (0.37), followed by 

K–12 success interventions (0.23), and then by the postsecondary education support intervention 

(0.19). For healthy futures interventions, the childhood obesity intervention (0.24) had a similar

effect size to supportive family environments interventions (0.22).22 Five objectives—

employability, K–12 success, postsecondary education support, school readiness, and supportive 

family environment—achieved statistically significant average effect sizes. The average effect 

sizes in the remaining priority objectives—financial literacy and childhood obesity—were not 

22 The supportive family environments area is not a designated AmeriCorps priority objective, but we discuss this 

category to provide more information on the interventions in the healthy futures focus area. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

26

statistically significant, perhaps partly because of the small number of respective qualified 

studies. In addition, the difference in the average effect sizes between priority objectives within a 

focus area (results not shown) were not statistically significant from each other.

Figure III.3 shows how average effect sizes varied 

by geography and whether AmeriCorps members 

provided direct services to participants or supported 

the delivery. Interventions implemented nationally 

or in multiple regions had an effect size of 0.32, 

compared to the effect size of 0.25 for interventions 

implemented in a single region. The difference of 

0.07 was not statistically significant (p = 0.23). 

Interventions in which AmeriCorps members 

provided direct services to participants or supported 

the delivery had an effect size of 0.27, which was 

similar to the 0.29 effect size for interventions 

delivered or supported by personnel that did not include AmeriCorps members (the difference of 

0.02 was not statistically significant, p = 0.70). This suggests that AmeriCorps members can be 

as effective in delivering services or supporting the delivery as other types of professionals. 

Figure III.3. Average effects by intervention characteristics

Note: The parentheses show the number of interventions in each characteristic. Appendix Table C.2 provides 

detailed information on the average effect size for the intervention characteristics categories.

Figure III.4 shows how average effect sizes varied by funding year for SIF grantees and 

AmeriCorps State and National grantees (2015–2019). The average effect sizes by funding year 

for interventions funded from 2010 to 2014 (SIF grantees) ranged from 0.22 to 0.33, whereas the 

average effect sizes by funding year for interventions funded from 2015 to 2019 (AmeriCorps 

State and National grantees) ranged from 0.11 to 0.34. Overall, the size of the effects did not 

vary substantially for interventions aligned with the different funding source. The only notable 

exception, the effect size of 0.11 for the 2017 funding year, was not statistically significant. This 

Key finding

Interventions in which 

AmeriCorps members provided 

direct services to participants or 

supported the delivery had a 

similar effect size to 

interventions that did not 

include AmeriCorps members. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

27

suggests that AmeriCorps has year-to-year consistency in identifying interventions supported by 

rigorous evidence of effectiveness, with little variation across the agency’s funding programs. 

Figure III.4. Average effects by funding year and source

Note: There were no interventions funded by SIF in 2013. The 2015 SIF-funded interventions reviewed by the 

project did not meet the project’s evidence of effectiveness standards. The parentheses show the number 

of interventions in each funding year. Interventions were included in the funding year average effect size 

estimate for each year that it was funded (in other words, interventions could appear in multiple funding 

years). Because a specific intervention could have been funded across multiple years, we did not compare 

whether interventions in one funding year had a greater impact on participant outcomes than interventions 

funded in a different funding year. Appendix Table C.3 provides detailed information on the average effect 

size for the funding year categories. 
This page has been left blank for double-sided copying.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

29

IV. SUMMARY, RECOMMENDATIONS, AND NEXT STEPS 

The focus of this report, shown in the first part of the SPREE process in Figure I.1 in Chapter I, 

was to identify the AmeriCorps-funded interventions that have been found through rigorous 

research to improve outcomes for people receiving the intervention services. For these effective 

interventions, we identified their characteristics—their alignment with AmeriCorps focus areas 

and priority objectives, as well as AmeriCorps’ mission in terms of geography and use of 

AmeriCorps members—and estimated how effective they were in improving participants’ 

outcomes. In this chapter, we describe the key findings that resulted from the research (Section 

A) and discuss the implications of these findings for AmeriCorps (Section B). 

A. Summary of findings

Out of 82 interventions from 154 grants with continued funding that had a strong or moderate 

AmeriCorps evidence rating, we identified 32 interventions that met Mathematica’s effectiveness 

standards. The other interventions were eliminated from the review process because their 

evidence documents (1) raised concerns about internal validity, which makes it difficult to 

conclude that the intervention was the cause of improved participant outcomes; (2) lacked 

consistently favorable findings for the intervention; or (3) had study design issues, such as not 

having a comparison group. 

The interventions that met this study’s effectiveness standards aligned with some of the 

AmeriCorps focus areas and priority objectives and somewhat with AmeriCorps’ mission to use 

national service participants to strengthen communities across the country. The interventions 

were confined to the education, economic opportunity, and healthy futures focus areas 

(representing three of the seven AmeriCorps focus areas), with the majority having an education 

focus. The three AmeriCorps priority objectives in the education focus area (school readiness, 

K–12 success, and postsecondary education support) and the three priority objectives in the 

economic opportunity focus area (employability, financial literacy, and housing) each had 

interventions with evidence that they work. One effective healthy futures intervention was 

aligned with the childhood obesity priority objective (other healthy futures interventions focused 

on supportive family environments, although this is not an AmeriCorps priority objective). Most 

of the AmeriCorps-funded interventions were implemented in urban areas (97 percent), with 16 

percent also implemented in rural areas. Four interventions (13 percent) were implemented in 

both urban and rural areas. About half (47 percent) were spread across the different regions of 

the United States, and the remainder (53 percent) were in one region only. AmeriCorps members 

were incorporated into one-quarter (25 percent) of the interventions, with AmeriCorps members 

delivering direct services to participants or supporting the intervention delivery.

The interventions that work had sizable impacts. Interventions in the education, healthy futures, 

and economic opportunity focus areas increased the probability of improving participants’ 

outcomes by 9, 9, and 14 percentage points, respectively, compared to individuals not receiving 

the intervention. The interventions aligned to the different AmeriCorps priority objectives varied 

in the size of their impacts, with interventions focused on participants’ employability and school 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

30

readiness having the largest effects (translated into 15 and 14 percentage points, respectively). 

The interventions also had significant impacts on participant outcomes regardless of whether 

they were implemented within a single geographic region or across multiple regions. 

Interventions that used AmeriCorps members had impacts that were similar to interventions in 

which AmeriCorps members were not involved. 

B. Recommendations 

These findings have several implications for how AmeriCorps can strengthen its efforts to 

identify, develop, and scale effective interventions to meet the needs of local communities. 

Specifically, there are several ways AmeriCorps can support and generate evidence for effective 

interventions and prioritize interventions for scaling. 

• Help grantees conduct studies that rigorously assess whether their intervention has an 

impact. One of AmeriCorps’ core principles in working with grantees is to help advance the 

rigor of the agency’s research evaluations over a grant’s life, with the goal of developing a 

strong evidence base for funded interventions. This principle is best reflected in a progression 

of evidence tiers that range from pre-preliminary through strong evidence for grantees. That 

39 percent of interventions funded by grants with a strong or moderate AmeriCorps evidence 

rating met Mathematica’s effectiveness standards indicates that AmeriCorps might consider 

developing additional resources to help its grantees build stronger evidence of effectiveness 

for their interventions. In particular, our project found that interventions did not meet the 

Mathematica effectiveness standards because of study design and internal validity issues 

(Figure II.2). Grantees delivering these interventions could benefit from additional technical 

assistance in the areas in which their evidence did not meet the project’s standards for 

evidence of effectiveness. Ultimately, these supports have the potential to help increase 

grantees’ awareness of how to produce evidence based upon well-designed and wellimplemented studies with high internal validity. 

• When funding interventions, consider prioritizing those that show impacts across 

multiple outcomes. For this project, interventions were eliminated from the review process 

if their supporting evidence showed that the interventions did not have consistently favorable 

impacts on their participants’ outcomes (see Figure II.2). This suggests that AmeriCorps 

should consider an intervention’s effectiveness when making funding decisions. One way to 

consider effectiveness might be to award more points to interventions that have a consistent, 

positive impact on participants’ outcomes. For example, suppose there were two literacy 

development interventions, both of which had supportive evidence with strong study designs 

that had high internal validity, but one intervention significantly impacted all targeted 

literacy outcomes for its participants, whereas the other had positive impacts on just one 

targeted outcome out of several that were examined. AmeriCorps could prioritize the 

consistently effective intervention for funding by assigning it a higher score. 

AmeriCorps makes a concerted effort to fund interventions across all levels of its rating 

continuum. AmeriCorps funds some new, innovative interventions that are in the process of 

developing their intervention model and evidence of its effectiveness. It also funds more 

established interventions to sustain or scale their implementation. For these fully developed 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

31

interventions, AmeriCorps could emphasize obtaining evidence that the interventions work 

when making strategic decisions about continuing their funding. 

• Target funding for scaling effective interventions. The findings from this study support 

AmeriCorps’ vision for scaling effective interventions to help more individuals across the 

United States through the use of national service participants. Our findings suggest that some 

interventions funded by AmeriCorps are effective in improving participant outcomes in 

urban and rural communities across the various regions in the country. Our findings also 

indicate that AmeriCorps-funded interventions delivered by AmeriCorps members were 

similar in their effectiveness to interventions that did not involve members. AmeriCorps 

could consider focusing more on supporting grantees seeking to scale these effective 

interventions to help more individuals or new communities throughout the country and 

enhance the presence of AmeriCorps members in the service delivery model.
This page has been left blank for double-sided copying.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

33

REFERENCES

AmeriCorps. “Annual Management Report, Fiscal Year 2017.” Washington, DC: AmeriCorps, 

2017. Available at https://www.nationalservice.gov/sites/default/files/documents/CNCSFY17-AMR-508_v2r2.pdf. Accessed June 4, 2019. 

AmeriCorps. “Metadata Glossary.” Washington, DC: AmeriCorps, 2016. Available at 

https://www.nationalservice.gov/sites/default/files/documents/GLOSSARYMetadata%20Categories%20Definitions%20for%20Publication.pdf. Accessed June 4, 2019. 

AmeriCorps. “Summary of Performance and Financial Information, Fiscal Year 2015.” 

Washington, DC: AmeriCorps, 2015. Available at 

https://www.nationalservice.gov/sites/default/files/documents/CNCS_FY_2015_SPFI.pdf. 

Accessed June 4, 2019. 

Borenstein, M., L. V. Hedges, J. P. Higgins, and H. R. Rothstein. Introduction to Meta-Analysis. 

Hoboken, NJ: Wiley, 2009. 

Hedges, L. V. “Fitting Categorical Models to Effect Sizes from a Series of Experiments.”

Journal of Educational Statistics, vol. 7, 1982, pp. 119–137. 

JBS International. “National Service Systematic Review and Synthesis of National Service 

Literature.” Burlingame, CA: JBS International, 2015. Available at 

http://www.nationalservice.gov/sites/default/files/evidenceexchange/FR_NationalServiceSy

nthesisReport.pdf. Accessed June 4, 2019. 

Lipsey, M. W. “Juvenile Delinquency Treatment: A Meta-Analytic Inquiry into the Variability 

of Effects.” In Meta-Analysis for Explanation, edited by T. D. Cook, H. Cooper, D. S. 

Cordray, H. Hartmann, L. V. Hedges, R. J. Light, T. A. Louis, and F. Mosteller, pp. 83–127. 

New York, NY: Russell Sage Foundation, 1992. 

Lipsey, M. W., and D. B. Wilson. Practical Meta-Analysis. Thousand Oaks, CA: Sage 

Publications, 2001. 

Maxwell, N. L., and S. Richman. “Scaling Programs with Research Evidence and Effectiveness 

(SPREE).” Foundation Review, vol. 11, no. 3, November 2019, pp. 55–67. 

Selekman, Rebekah, Katie Eddins, Natalie Reid, and Scott Richman. “Planned Scaling Activities 

of AmeriCorps-Funded Organizations.” Submitted to AmeriCorps by Mathematica.

Chicago, IL, August 2020. 

Shadish, W. R., T. D. Cook, and D. T. Campbell. Experimental and Quasi-Experimental Designs 

for Generalized Causal Inference. Boston, MA: Houghton Mifflin, 2002. 

Slavin, R., and D. Smith. “The Relationship Between Sample Sizes and Effect Sizes in 

Systematic Reviews in Education.” Educational Evaluation and Policy Analysis, vol. 20, no. 

10, 2009, pp. 215–229. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

34

What Works Clearinghouse. Procedures Handbook, Version 4.1. Washington, DC: Institute of 

Education Sciences, U.S. Department of Education, 2020. Available at

https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Procedures-Handbook-v4-1-

508.pdf. Accessed June 4, 2020. 

Zhang, X., and J. Sun. “Meta-Analysis of Evaluations Across the Social Innovation Fund 

Program: Final Report.” Prepared for AmeriCorps, Office of Research and Evaluation. 

Fairfax, VA: ICF International, 2016. Available at 

https://www.nationalservice.gov/sites/default/files/documents/FR_SIF_MetaAnalysis_Final_Report.pdf. Accessed June 4, 2019. 
APPENDIX A: 

REVIEW PROCESS AND INTERVENTION REVIEW 

RUBRIC
This page has been left blank for double-sided copying.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-3 

In this appendix, we describe the review rubric and process that Mathematica used for extracting 

information on the characteristics of the 32 interventions identified as having evidence that they 

work. These interventions were among those proposed by AmeriCorps State and National 2015–

2019 and the Social Innovation Fund 2010–2015 grantees that were judged by an independent, 

third-party evaluator contracted by AmeriCorps as providing strong or moderate evidence for 

their intervention (see Table I.1 in Chapter I). 

Table A.1 describes the criteria we applied to each evidence document to determine eligibility 

for further review. We developed these criteria to align with the overarching goal of the project: 

to identify interventions with evidence of effectiveness that grantees intended to scale to produce 

a wider impact. 

Table A.1. Standards for categorizing evidence of effectiveness 

Criterion Question Yes or No

Project objectives

Study design Did the study use a comparison group in its research design to allow for causal 

inferences to be drawn about the impact of the intervention?

Reporting on 

methods

Did the study sufficiently describe its research design and statistical approach? 

For example, did the study provide adequate information on the formation of its

study groups and the statistical procedures used to gauge the impacts of the 

intervention?

Evaluator 

independence

Was the study conducted by an evaluator external to the grantee?

Impact findings Did the study show consistently favorable findings on the intervention’s 

outcomes of interest? Having consistently favorable findings is defined as having 

intervention impacts that are statistically significant across the majority of 

confirmatory research questions examined in the study.

Scaling plan 

alignment

Is the intervention assessed in the study aligned to the intervention the grantee 

plans to scale up? 

Internal validity

Attrition If the study was a randomized controlled trial (RCT), did the study experience 

high attrition?23

Attrition from the study sample (overall and differential attrition between the study 

groups) can compromise the initial equivalence of the groups and lead to biased 

estimates of an intervention’s effects. To determine the potential for attrition bias, 

we used the What Works Clearinghouse (2020) model,24 which combines the 

overall and differential attrition rates a study may encounter, and generates 

acceptable and unacceptable levels of bias.

23 Attrition is examined only for RCTs and is not applicable to studies using quasi-experimental designs because for 

this study design type, only the analytic sample with post-test outcomes (not the initial sample present at the onset 

of the study) is considered when determining evidence quality. 

24 We used the What Works Clearinghouse’s “optimistic” assumptions for assessing unacceptable levels of expected 

bias. See https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-4 

Criterion Question Yes or No

Reassignment If the study was an RCT, did the study use reassignment?

Reassignment, or switching study participants from the comparison group to the 

intervention group (or vice versa) after random assignment, is another major 

internal validity concern for RCTs. Using reassignment would undermine the 

important assumption that the intervention and comparison groups are similar at 

baseline due to random assignment. 

Baseline 

equivalence

For an RCT study that had high attrition or used reassignment or a study that 

used a quasi-experimental design, did the study demonstrate baseline 

equivalence? That is, did it show that the intervention and comparison groups in 

the final analytic sample are similar on a pre-test outcome measure at baseline?

When intervention and comparison groups are not formed through a 

randomization process (or when RCTs experience high attrition or use 

reassignment), it is possible that groups could differ in unobservable ways at 

baseline even if they appear similar on their measured characteristics. 

Unmeasured baseline differences can bias estimates of the intervention’s 

impact. As a result, studies must demonstrate baseline equivalence, showing 

that the intervention and comparison groups have similar observable 

characteristics at baseline. For this review, equivalence had to be established on 

at least one primary outcome measure, meaning there were no statistically 

significant differences on primary outcome measures for the analytic sample at 

baseline. If a statistical difference occurred, the study would have to statistically 

control for these baseline differences when analyzing post-test outcomes.

Confounding 

factors

Does the study have the presence of a confounding factor that makes it difficult 

to distinguish between the effect of that factor and the intervention, thus making 

it impossible to attribute any potential impacts solely to the intervention?

One common confounding factor is when some aspect of the research design 

lines up with either the intervention or comparison group, also referred to as an 

“n = 1 confound.” For example, if the intervention or comparison group 

comprises a single unit (school, classroom, teacher), the study would not be able 

to isolate the effect of the intervention from other observable or unobservable 

characteristics of that unit. Interventions bundled with other services not being 

studied can also make it impossible to isolate the unique impact of the 

intervention. 

A structured review template was also used to extract intervention impact information from all 

evidence documents that met our evidence quality standards to assess the evidence of 

effectiveness and determine the magnitude of effects. For each evidence document, we recorded 

all outcomes measured at the end of the intervention for each independent study sample. Two 

researchers independently coded relevant information for the meta-analysis from each report 

using the same set of coding guidelines. The coders resolved any discrepancies through 

consensus and consulted a senior coder when consensus could not be reached. 

The intervention review rubric was applied only to the evidence documents that met our study’s 

effectiveness standards. When applying this rubric, we did not consult other external sources (for 

example, we did not query authors to collect more information not available in the documents or 

web searches to find additional information on the interventions). Several interventions 

supported their effectiveness by providing more than one evidence document that met the 

project’s effectiveness standards. A separate assessment was created for each evidence 

document, and the information was then combined to provide an intervention-level summary of 

characteristics that informed the evidence supporting each intervention.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-5 

Reviewing these documents required systematic procedures to ensure that we reliably extracted 

standardized information from each document. Trained reviewers documented the interventions' 

characteristics using the rubric; in addition, a third-party consultant, external to Mathematica, 

reviewed all evidence documents authored by Mathematica to reduce any potential conflicts of 

interest. Reviewers classified each intervention and documented the characteristics of each 

intervention being reviewed, focusing on describing the intervention as implemented and 

evaluated. Study team leaders also reviewed the evidence documents and completed assessments 

to ensure that they contained complete and accurate information. The reviewer and study team 

leader discussed and resolved any discrepancies, and an additional study team leader was 

consulted if a consensus could not be reached. All finalized assessments were also reviewed by a 

Mathematica researcher external to the project as a final quality assurance procedure.

I. INTERVENTION REVIEW SECTION

The intervention review section of the rubric is structured to extract information from documents 

submitted by AmeriCorps grantees that report evaluation findings for AmeriCorps-funded 

interventions. The information extracted through the use of this rubric section describes the 

characteristics of the intervention when it was evaluated. It contains six subsections. The first 

subsection collects information about the context in which an intervention was implemented, and 

each of the next five subsections collects information on one of the five conditions needed for 

scaling. 

A. Context for the intervention

Table A.I.1 provides the first subsection of the rubric, which includes three types of questions 

about the context in which the intervention unfolds: 

1. Description of the intervention, according to AmeriCorps nomenclature. The nomenclature 

includes the intervention’s (1) focus area, (2) topic area, (3) intended outcome domain(s), and 

(4) intended outcome(s). The appendix provides a list of predetermined response options for 

classifying the intervention per the nomenclature, with the “other” response indicating that 

the predetermined areas do not adequately describe the intervention.

2. Local area context, including contextual features that could affect implementation of the 

intervention, including any regional, state, and local contextual features; demand for the 

intervention at the local level; and partners.

3. Measures of implementation, including the percentage of participants who complete the 

intervention and cost per participant. The measures require a definition of components 

(participant, completer, and costs). 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-6 

Table A.I.1. Questions about context for the intervention

Question Response options Additional guidance

I. Description of the intervention 

1. What is the 

intervention’s focus 

area?

Only one focus area. Focus areas include disaster services, economic 

opportunity, education, environmental 

stewardship, healthy futures, nonprofit 

organizational capacity, and veterans and 

military families.

2. What is (are) the 

intervention’s topic 

area(s)?

Up to two topic areas listed in 

Table 1 in the appendix.

The topic area provides greater specificity about 

the intervention’s broad goals than about the 

focus area. For example, an intervention may fall 

into the “economic opportunity” focus area. If it 

helps low-income people obtain employment, 

however, it would fall into the “employability” 

topic area.

3. What is (are) the 

intervention’s outcome 

domain(s)?

Up to two outcome domains 

listed in Table 1 in appendix that 

fall within the topic area selected.

The outcome domain provides greater specificity 

than the topic area. The intervention aimed at 

helping low-income people obtain employment 

may do so by providing industry skill training, 

which is the intervention’s outcome domain. The 

veterans and military families, disaster services, 

and nonprofit organizational capacity focus 

areas and some of the education topic areas do 

not have outcome domains. 

4. What is (are) the 

intervention’s 

outcome(s)?

Up to four outcomes listed in 

Table 1 in appendix. 

Outcomes are the targets that the intervention 

seeks to affect. In the example of the industry 

skill training intervention, an outcome might be 

certifications/skills obtained by participants. 

Outcomes may span several outcome domains 

within a given topic area. 

II. Local area context

1. Might the regional, 

state, and local context 

facilitate or challenge 

implementation of the 

intervention?

Yes, contextual considerations 

might affect implementation.

No, contextual considerations 

are not mentioned.

Regional, state, and local contextual factors can 

challenge or facilitate implementation of an 

intervention. Such factors may include 

community stakeholders, the political 

environment, or the local economy. For example, 

an affordable housing intervention that was 

implemented during a local economic downturn 

could see increased demand for its services.

Question 2 is skipped with a “no” response.

2. What are the major 

features of the regional, 

state, and local context 

that could have affected 

implementation of the 

intervention?

[open-ended] Describe any features of the regional, state, or 

local context that may have been salient in 

implementing the intervention. 

3. Does evidence exist for 

local demand for the 

intervention? 

Yes (full), local demand exists for 

the intervention.

Yes (limited), local demand 

exists, but details are not 

available.

No, local demand for the 

intervention is not mentioned.

Local demand may be identified by conducting a 

needs assessment, landscape analysis, or 

informal data collection through attendance at 

community meetings or conversations with key 

stakeholders. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-7 

Question Response options Additional guidance

4. Do key partners play a 

role in implementing the 

intervention?

Yes, implementation partners are 

identified.

No, partners are not identified.

The organization may use partners to deliver 

services or assist with implementation of the 

intervention. 

Question 5 is skipped with a “no” response. 

5. What role do the key 

partners play in 

implementing the 

intervention?

[open-ended] Describes the partners or contractors that 

assisted with implementation of the intervention, 

including the role played by each.

III. Measures of implementation

1. What is the intended

overall completion rate 

for participants in the 

intervention?

[open-ended] Intended intervention completion rate during a 

specific time period. Interventions often set forth 

completion goals for participants. 

2. What is the actual

overall completion rate 

for the intervention?

[open-ended] Actual rate of participants’ completion of the 

intervention during a specific time period. 

3. Is cost-per-participant 

information discussed?

Yes, cost per participant is 

discussed.

No, cost per participant is not 

mentioned. 

Cost-per-participant information may be 

presented as part of a cost-benefit analysis.

Question 4 is skipped with a “no” response.

4. What is the cost per 

participant for the 

intervention?

[open-ended] The average cost of delivering the intervention 

per participant, including the year the costs were 

captured and information about the resources 

reflected in the calculation (the cost of personnel

time, materials, space rental, partner services, 

among other items).

B. Well-specified intervention

This subsection addresses the first condition of the scaling readiness framework—a wellspecified intervention. It includes questions in six areas (Table A.I.2). The first area describes the 

intervention, and the next four describe a different dimension of each core element. The last area 

describes what it takes a participant to complete the intervention.

1. Core elements, including identification of the intervention by its name. Core elements are 

activities that were part of the intervention and were intended to achieve the intervention’s 

ultimate outcome domain and outcomes.

2. Mode of service delivery, including how each element was designed to be delivered to 

participants. 

3. Intensity, including how often and for how long each element was offered to participants 

and how much of each element each participant received.

4. Personnel needs, including the intervention’s personnel structure. A personnel structure 

includes the job titles of personnel who delivered the intervention, the number of individuals 

involved in delivery of the intervention, reliance on AmeriCorps national service 

participants, and qualifications of personnel. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-8 

5. Setting, including location (geography) and venue (the place where activities occurred, such 

as a community center, home, nonprofit organization, park, school). 

6. Definition of completion, including criteria for being considered a participant in the 

intervention and what a participant needed to do to be considered a “graduate” or completer 

of the intervention. 

In some cases, the rubric refers to a standardized, name-brand intervention (for example, Teach 

For America). A standardized intervention adheres to a specific model or curriculum in terms of 

the scope and sequence of the intervention’s content. It spells out the same type and amount of 

services that each participant should receive. In a nonstandardized intervention, each 

participant’s need determines the type and amount of services delivered; as a result, each 

participant experiences the intervention differently. 

Table A.I.2. Questions about specifying the intervention

Question Response options Additional guidance

I. Core elements

1. What is the name of the 

intervention?

[open-ended] Use of a standardized, name-brand intervention 

rather than the local intervention name. 

2. What are the designed core 

elements of the intervention?

[open-ended] Core elements are the set of activities or 

curricula that result in participants’ receipt of 

services, such as a description of services, the 

content area of the intervention, the existence 

of a standardized intervention model, the 

services or curriculum comprising the 

intervention, and the content area of the 

intervention to be delivered. A standardized 

intervention is considered to have a clearly 

specified intervention. Examples of core 

elements are tutoring, facilitator-led classes or 

workshops, one-on-one coaching, case 

management, electronic or telephone 

communication with participants, and building 

the capacity of an organization or partner 

organization. 

The description must align with the selected 

categorical responses in Table A.1 in the 

appendix. 

3. Were all of the intervention’s 

core elements implemented?

Yes, all core elements were 

implemented.

No, some core elements 

were not implemented.

Question 4 is skipped with a “yes” response.

4. Which core elements of the 

intervention were 

implemented? 

[open-ended] Identify all core elements implemented.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-9 

Question Response options Additional guidance

5. Were any purposeful

changes made to the 

intervention core elements 

implemented as compared to 

an earlier version of the 

model (including adaptations 

and modifications)?

Yes, intervention core 

elements were purposefully 

changed. 

No, intervention core 

elements were not 

purposefully changed. 

Organizations may make changes to the 

intervention core elements prior to 

implementation, referred to as purposeful 

changes. Purposeful changes could extend to 

the intervention content, service delivery mode, 

intensity, and personnel needs (for example, a 

model designed for classroom training of 

service providers may have changed to one-onone training). Implementing a “2.0” version is 

also a purposeful change. Differences between 

intended and actual implementation are not 

considered purposeful changes if, for example, 

the actual duration was shorter than intended, 

but the intended duration did not differ from the 

intervention model. 

Question 6 is skipped with a “no” response. 

6. What purposeful adaptations 

or modifications did the 

organization make?

[open-ended] Describes all purposeful changes made to the 

intervention core elements. 

7. Were any planned changes 

made to the intervention 

during the implementation 

period?

Yes, planned changes were 

made to the intervention 

during implementation. 

No, planned changes were 

not made to the intervention 

during implementation. 

During implementation, an organization may 

decide to make changes to an intervention, 

referred to as planned changes. Planned 

changes may include modifications to activities 

or services, mode of delivery, source of 

delivery, setting, or intensity. For example, an 

organization may have decided to offer an 

intervention in a new setting, extend the period 

of service delivery, or change the originally 

planned activities in order to test whether such 

changes would increase participant outcomes. 

Question 8 is skipped with a “no” response.

8. What were the planned 

changes to the intervention 

and the reasons for them?

[open-ended] Describes all planned changes to intervention 

core elements.

II. Mode of service delivery

1. What is the mode of delivery 

for each element?

[open-ended] The service delivery model specifies how 

services are delivered, who delivers services, 

and the mode through which services are 

delivered. Organizations should provide a 

description of the way in which each core 

element is delivered. For example, peer 

educators or teams of facilitators may deliver 

services in groups or one-on-one meetings and 

through the use of electronic communication or 

in-person meetings.

III. Intensity

1. What is the intended

frequency of delivery of each 

element?

[open-ended] For each element implemented, describes how 

often the element is intended to be offered 

during a specific time period. 

2. What is the frequency with 

which each element was 

actually implemented (on 

average)?

[open-ended] For each element implemented, describes how 

often the element was actually offered during a 

specific time period. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-10

Question Response options Additional guidance

3. What is the intended

duration of attendance for 

each element?

[open-ended] For each element implemented, includes the 

intended length of time participants were 

supposed to receive an element during a 

specific time period.

4. What is the actual duration 

of attendance for each 

element (on average)? 

[open-ended] For each element implemented, describes the 

actual length of time participants received an 

element during a specific time period.

5. What is the intended

number of hours of service 

receipt for each session of 

each element?

[open-ended] For each element implemented, describes how 

many hours participants were supposed to 

receive an element during a specific time 

period.

6. What is the actual number of 

hours of service receipt for 

each session of each element 

(on average)? 

[open-ended] For each element implemented, describes the 

number of hours that participants actually 

received an element during a specific time 

period. 

7. What is the intended total 

number of hours of 

programming for each 

element?

[open-ended] For each element implemented, describes the 

total number of hours of services that 

participants were scheduled to receive. 

8. What is the actual total 

number of hours of 

programming for each 

element (on average)?

[open-ended] For each element implemented, describes the 

total number of hours of services that 

participants actually received, on average, 

during a specific time period. 

IV. Personnel needs

1. How many individuals deliver 

the intervention (overall and 

by intervention element)? 

[open-ended] Describes the number of individuals, by job title, 

who directly delivered the services across all 

locations, both overall and by element. For 

example, if an after-school reading intervention 

used 3 national service participants to lead 

classroom sessions and 9 classroom teachers 

to provide one-on-one tutoring across three 

locations, the response would state, “12 

personnel overall: 3 participants (classroom 

sessions) and 9 teachers (one-on-one 

tutoring).”

2. Who delivers the services per 

intervention element?

[open-ended] For each core element, lists job titles of 

individuals who directly delivered services, 

including several job titles as necessary. For 

example, if an after-school reading intervention 

used national service participants to lead 

classroom sessions and classroom teachers to 

provide one-on-one tutoring, the titles of both 

the participants and teachers are listed, along 

with the service element delivered by each 

group.

3. Does the intervention’s 

personnel structure include 

AmeriCorps service 

participants? 

Yes, AmeriCorps 

participants are included in 

the personnel. 

No, these groups are not 

mentioned.

Question 4 is skipped with a “no” response.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-11

Question Response options Additional guidance

4. What role do AmeriCorps 

participants play?

[open-ended] Describes the role played by AmeriCorps

participants. For example, “participants 

provided tutoring services for an after-school 

intervention.”

5. What qualifications are the 

required or preferred as a 

prerequisite for the personnel

involved in delivering the 

intervention?

[open-ended] Describes qualifications for each job title 

involved in delivering the intervention. 

Qualifications could include experience or 

expertise in delivering the needed services or 

education/training. For example, a nutrition 

education intervention requires educators to 

hold a degree in nutrition and demonstrate at 

least two years of teaching experience. 

6. Does the organization intend 

or plan for a different 

personnel structure or 

personnel qualifications or 

training from what was 

originally intended or 

planned?

Yes, personnel plans differ 

from the personnel as 

implemented.

No, personnel plans do not 

differ from the personnel as 

implemented.

Describes how the intended personnel plan, 

including the job titles of those who delivered 

the intervention, the number of individuals 

involved in service delivery, whether the 

personnel included AmeriCorps national service 

participants, personnel qualifications, and 

training plans for personnel, differed from the 

plan that was implemented. 

Question 7 is skipped with a “no” response.

7. How does the personnel

structure differ from what was 

originally intended or 

planned?

[open-ended] Describes the differences between the planned 

personnel structure and the implemented 

personnel structure. Includes the job titles of 

those who delivered the intervention, the 

number of individuals involved in service 

delivery, and whether the personnel included 

AmeriCorps service participants. 

VII. Setting

1. What are the geographic 

location(s) of the intervention, 

and what sites are included in 

implementation?

[open-ended] For all locations, describes the geographic 

region, location, and whether the intervention 

was implemented in a rural or urban setting (or 

both). For example, an intervention 

implemented in two regions of the United 

States, three cities or areas, and several school 

districts and schools within each district would 

state, "Northeast and Southern regions, 

including New York, NY (Brooklyn): three 

school districts, 10 schools; Washington, DC: 

one school district, 8 schools; eight rural 

counties in southwestern Virginia (two school 

districts, 4 schools).” 

2. In what venue(s) are the 

intervention core elements 

implemented?

[open-ended] The implementation setting is where an 

intervention physically takes place, such as 

community center, home, nonprofit 

organization, park, school, and residential 

facilities. Organizations may implement core 

elements in different settings. For example, an 

intervention may have included job training at 

an American Job Center and job coaching on 

site with employers; both the American Job 

Center and employer settings, along with the 

job training and job coaching activities, are 

listed.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-12

Question Response options Additional guidance

3. Did the evaluation enroll 

participants from a subset of 

intervention locations? 

Yes, the locations for the 

evaluation were a subset of 

places where the 

intervention was 

implemented. 

No, the locations for the 

evaluation and 

implementation were the 

same.

The evaluation of the intervention may be 

conducted with participants from a subset of the 

locations in which the intervention was 

implemented. For example, an intervention may 

have been implemented in two cities, but the 

evaluation included participants in only one city.

Question 4 is skipped with a “no” response.

4. What locations were included 

in the evaluation?

[open-ended] Describes geographic locations indicating 

where the evaluation occurred and the number 

of locations. 

VI. Definition of completion

1. Is intervention participant 

defined?

Yes, participant is defined.

No, participant is not 

defined.

Definition of a participant describes the 

minimum amount of services needed to be 

received in order to be considered a 

“participant” in the intervention. For example, 

individuals may enroll in an intervention and be 

considered a participant even if they never 

receive any services. Alternatively, participants 

may be defined as only those individuals who 

enroll and receive at least one service through 

the intervention.

Question 2 is skipped with a “no” response.

2. What is the definition of an 

intervention participant?

[open-ended] Describes the amount of services an individual 

must receive in order to be considered a 

“participant.”

3. Is completion of the 

intervention defined?

Yes, completion is defined.

No, completion is not 

defined.

Definition of completion indicates what 

participants must do to be considered 

“completers” or “graduates.” For example, 

participants may have to complete a course and 

obtain a certificate in order to complete an 

intervention.

Question 4 is skipped with a “no” response. 

4. What is the definition of 

intervention completion?

[open-ended] Describes what participants must do to be 

considered “completers” or “graduates.”

C. Well-defined target population

This subsection addresses the second condition of the scaling readiness framework—a welldefined target population. It includes questions about the target population along three major 

dimensions (Table A.I.3): 

1. Intervention as it was designed, including the characteristics of the target population as the 

intervention was originally designed. 

2. Intervention as it was implemented, including the characteristics of the population that was 

eligible to participate in the intervention as it was implemented.

3. Evaluation sample for the intervention, including the participants in the evaluation sample.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-13

Table A.I.3. Questions about defining the target population

Question Response options Additional guidance

I. Intervention as it was designed

1. What is the target population 

for the intervention as it was 

designed?

[open-ended] Describes the characteristics of the 

population that the intervention was originally 

designed to serve. Includes risk factors, age 

or grade ranges, participants’ locations, 

gender, and so forth. In the case of several 

target populations, describes each 

population. 

II. Intervention as it was implemented

1. Who is eligible to participate 

in the intervention as it was 

implemented by the 

organization?

[open-ended] Describes who is eligible to participate in the 

intervention as implemented. Includes risk 

factors, age or grade ranges, participants’ 

locations, gender, and so forth, along with 

inclusion and exclusion criteria and 

information for several target populations. 

2. Does the intervention as it is 

implemented enroll 

participants primarily from a 

specific racial or ethnic 

group?

Yes, participants are primarily 

from one of the prespecified 

racial or ethnic categories.

No, participants are not 

primarily from a racial or ethnic

category.

Racial/ethnic groups include African 

American, Hispanic, and other. An even 

distribution of African American, European 

American, and Hispanic participants is 

considered a “no” response. 

3. Does the intervention as it is 

implemented enroll 

participants primarily from a 

certain age group?

Yes, participants are primarily 

from one of the prespecified 

age group(s). 

No, participants are not 

primarily from a certain age 

group(s).

Age categories include 0-5, 6-9, 10-7, 18-24, 

25 55, and 56+. The intervention may be 

designed to serve participants in a specific 

age category, but, during implementation, 

participants from a different age category 

might enroll. Age group categories include all 

ages in that range. For example, if an 

intervention enrolled 4- to 6-year-old children, 

the response would be "0-5 and 6-9." 

4. Does the intervention as it is 

implemented enroll 

participants primarily from any 

of the key populations?

Yes, participants are primarily 

from one of the prespecified 

key populations.

No, participants are not 

primarily from a key 

population.

Key populations include low-income 

households/individuals, developmentally 

disabled individuals, homeless individuals, 

parents/caregivers of young children, 

parolees, and veterans/military families. 

III. Evaluation sample for the intervention

1. Does the evaluation sample 

differ from the target 

population of the intervention 

as it is implemented?

Yes, the evaluation sample 

and target population differ for 

the intervention as 

implemented. 

No, the evaluation sample and 

target population do not differ 

for the intervention as 

implemented.

Differences may arise if the evaluation 

sample was a subset of the population 

served by the intervention. For example, an 

intervention may have targeted 1,000 

students in grades 9 through 12 in Chicago 

and Detroit, but the evaluation may have 

considered the outcomes of only 250 of those 

students in all grades and in both cities. 

Although the evaluation sample might be 

representative of the population it served, it

does not include all of those served. 

Questions 2 through 5 are skipped with a 

“no” response.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-14

Question Response options Additional guidance

2. How does the evaluation 

sample differ from the target 

population for the intervention 

as it is implemented?

[open-ended] Describes how the evaluation sample differs 

from the population intended to receive the 

intervention. 

3. Does the evaluation measure 

outcomes of participants 

primarily from a particular 

racial or ethnic group?

Yes, the evaluation measured 

outcomes of participants 

primarily from a prespecified 

racial or ethnic group(s). 

No, the evaluation did not 

measure outcomes of 

participants primarily from a 

racial or ethnic group(s). 

Racial/ethnic groups include African 

American, Hispanic, and other. An even 

distribution of African American, European 

American, and Hispanic participants is 

considered a “no” response.

4. Does the evaluation measure 

outcomes of participants 

primarily from a certain age 

group?

Yes, the evaluation measured 

outcomes of participants from 

a certain age group(s). 

No, the evaluation did not 

measure outcomes along the 

age dimension.

Age categories include 0-5, 6-9, 10-17, 18-

24, 25-55, and 56+. 

5. Does the evaluation measure 

outcomes of participants from 

any of the key populations? 

Yes, the evaluation measured 

outcomes of participants from 

a prespecified key 

populations.

No, the evaluation did not 

measure outcomes of 

participants from the key 

populations.

Key populations include low-income 

households/individuals, developmentally 

disabled individuals, homeless individuals, 

parents/caregivers of young children, 

parolees, and veterans/military families. 

D. Implementation supports

This section of the rubric addresses the third condition of the scaling readiness framework— 

implementation supports. It includes questions in six major areas, each of which describes the 

supports that help ensure quality in implementation (Table A.I.4). 

1. Implementation monitoring team, including how the organization ensures fidelity to the 

intervention model (whether an organization delivered an intervention as intended). It also 

includes whether implementation issues arose, and whether the organization made any 

purposeful changes to the implementation supports.

2. Performance procedures, including performance goals and benchmarks for how personnel 

deliver the intervention.

3. CQI, including the ongoing, systematic process of identifying, describing, and analyzing 

strengths and challenges during implementation, along with the collection and use of data to 

improve the intervention’s processes. 

4. Preservice and inservice training for the personnel, including initial and ongoing training 

for those who deliver the intervention as well as the identification of those responsible for the 

delivery of training. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-15

5. Communication systems, including the elements that facilitate high quality communication 

among intervention leaders, personnel, and partners. 

6. Data systems, including efforts to track, measure, and store information about 

implementation and to use the information to help make decisions for monitoring and CQI. 

Table A.I.4. Questions about implementation supports

Question Response options Additional guidance

I. Implementation monitoring team

1. Is an implementation 

monitoring team described?

Yes (full), an implementation 

monitoring team is described, 

with details provided. 

Yes (limited), an 

implementation monitoring 

team is described, but with no 

details provided.

No, an implementation 

monitoring team is not 

mentioned.

A monitoring team ensures that 

implementation takes place as planned. 

Team members may play other roles in the 

organization, such as supervisors, project 

managers, and project directors, or they 

may be other personnel dedicated 

exclusively to ensuring implementation of 

the intervention takes place with fidelity to 

the model. 

Question 2 is skipped with a “no” response.

2. Which personnel make up 

the implementation monitoring 

team? 

[open-ended] Describes the team members, including 

their titles and responsibilities, who are 

monitoring implementation of the 

intervention with fidelity to the model. 

3. Does a process exist for 

monitoring service delivery to 

assess fidelity to the 

intervention as planned?

Yes, steps exist for monitoring

fidelity. 

No, steps to monitor fidelity are 

not mentioned.

A process for monitoring fidelity to the 

intervention model ensures that services are 

being delivered as intended. The process 

for monitoring implementation fidelity could 

include collection of service delivery data 

through observations during site visits or 

regular reviews of service data entered into 

a data system. References to processes for 

assessing whether the “intervention is 

delivered as intended” are considered in

assessing fidelity. Discussions of monitoring 

quality or other facets of implementation are 

not considered.

Question 4 is skipped with a “no” response.

4. How is implementation 

fidelity monitored?

[open-ended] Describes the steps taken to monitor 

implementation with fidelity, including 

identification of the individual (by job title) 

responsible for assessing fidelity, frequency 

of implementation monitoring, processes for 

managing/supervising personnel involved in 

service delivery, and frequency of personnel

management or supervisory meetings.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-16

Question Response options Additional guidance

5. Are issues or challenges 

noted regarding 

implementation fidelity? 

Yes, issues were noted. 

No, issues regarding fidelity 

were not mentioned.

Fidelity issues include variation among 

instructors in the manner in which they 

deliver the curriculum across classes or 

cohorts and unplanned modifications to the 

intervention during the evaluation, such as 

changing an implementation site or 

adjusting the curriculum to meet the target 

population’s unexpected needs. 

Question 6 is skipped with a “no” response.

6. What issues or challenges 

are noted regarding 

implementation fidelity?

[open-ended] Describes any issues or challenges reported 

with implementation of the intervention with 

fidelity. 

7. Who was responsible for 

supervising personnel

involved in service delivery? 

[open-ended] Describes, by job title, who supervised the 

personnel involved in service delivery. If 

more than one personnel member 

responsible for supervising the individuals 

involved in service delivery, identifies each 

title and who was supervised. 

8. How were personnel involved 

in service delivery

supervised?

[open-ended] Describes the frequency and format of 

supervision for each job title. For example, 

supervision may take place on a weekly, 

monthly, or as needed basis and may be in 

the form of one-on-one meetings or small 

team meetings. Supervision extends to 

check-in meetings, case note reviews, 

administrative data analysis, 

communication, and supports for 

supervisors. 

9. Did any implementation 

supports purposefully

change from an earlier 

version of the intervention 

model before the start of 

implementation?

Yes, the implementation 

supports changed from an 

earlier version of the model 

before the start of 

implementation.

No, the implementation 

supports did not change from 

an earlier version of the model 

before the start of 

implementation.

Organizations may make changes to 

implementation supports before the start of 

implementation. Purposeful changes to 

implementation supports may include 

changes related to the following: the 

personnel structure, personnel training, 

supervision and performance management, 

implementation monitoring, communication 

processes, and data systems. 

Question 10 is skipped with a “no” 

response.

10. What purposeful changes 

were made to the 

intervention’s implementation 

supports?

[open-ended] Describes implementation supports that 

purposefully differ from an earlier design of 

the intervention. Changes may include any 

support related to the following: steps to 

ensure fidelity, the personnel structure, 

personnel training, supervision and 

performance management, monitoring, 

communication, and data systems. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-17

Question Response options Additional guidance

II. Performance procedures

1. Are performance 

benchmarks established for 

personnel involved in service 

delivery described?

Yes (full), performance

benchmarks established for 

personnel involved in service 

delivery are described, with 

details provided.

Yes (limited), performance

benchmarks established for 

personnel involved in service 

delivery are described, but with 

no details provided.

No, performance benchmarks 

established for personnel

involved in service delivery

were not mentioned.

Performance benchmarks are specific goals 

that personnel involved in service delivery 

must meet as related to delivery of the 

intervention, such as the number of 

participants contacted per personnel

member or the number of participants 

served per personnel member.

Question 2 is skipped with a “no” response. 

2. What were the performance 

benchmarks established for 

personnel involved in service 

delivery? 

[open-ended] Describes the performance benchmarks 

established for personnel involved in service 

delivery, such as the number of participants 

contacted per personnel member or the 

number of participants served per personnel

member. If different performance goals are 

set for specific job titles, goals are listed by 

job title; for example, “case managers are 

required to contact 12 participants per day.” 

3. Are procedures for 

monitoring achievement of 

performance benchmarks 

described?

Yes, procedures for monitoring 

achievement of performance 

benchmarks are described, with 

details provided.

Yes (limited), procedures for 

monitoring achievement of 

performance benchmarks are 

described, but with no details 

provided.

No, procedures for monitoring 

achievement of performance 

benchmarks were not 

mentioned.

Procedures for monitoring achievement of 

performance benchmarks include how 

benchmarks are measured and collected, 

who reviews progress toward benchmarks, 

and the frequency of performance 

monitoring.

Question 4 is skipped with a “no” response.

4. What were the performance 

benchmarks monitoring 

procedures?

[open-ended] Describes the procedures for monitoring 

achievement of performance benchmarks 

including how benchmarks are measured 

and collected, who reviews progress toward 

goals, and how frequently benchmarks for 

goals are captured and reviewed. If different 

performance monitoring procedures are set 

for specific job titles or roles, monitoring 

procedures are listed by job title.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-18

Question Response options Additional guidance

III. Continuous quality improvement (CQI)

1. Are CQI processes for the 

intervention described?

Yes (full), CQI processes are 

described, with details 

provided.

Yes (limited), CQI processes 

are described, but with no 

details provided.

No, CQI processes are not 

mentioned.

Continuous quality improvement processes 

refer to procedures for continuously 

assessing the quality of the intervention as 

implemented to improve implementation 

practice. CQI includes regularly testing the 

intervention and making adjustments as 

needed, with continual retesting of the 

modifications to ensure that the intervention 

is succeeding as planned. 

Question 2 is skipped with a “no” response.

2. What were the CQI 

processes for the 

intervention?

[open-ended] Describes the process for testing the 

intervention, including how frequently CQI 

data is collected. 

3. Were data analyzed to 

support CQI for the 

intervention?

Yes (full), data were analyzed 

and used for CQI, with details 

provided.

Yes (limited), data were 

analyzed and used for CQI, but 

with no details provided.

No, the analysis and use of 

data for CQI was not 

mentioned.

CQI requires the collection of data on 

performance benchmarks and use of the 

data to provide ongoing personnel

development training and the delivery of 

technical assistance to partners. Data 

analysis to support CQI includes examining 

personnel achievement of performance 

benchmarks. 

Question 4 is skipped with a “no” response.

4. How were data analyzed to 

support CQI for the 

intervention?

[open-ended] Describes how performance data were used 

to continuously improve intervention 

implementation.

IV. Preservice and inservice training for personnel

1. Is the initial training to deliver 

the intervention received by 

personnel described?

Yes (full), initial personnel

training to deliver the 

intervention is described, with 

details provided.

Yes (limited), initial personnel

training to deliver the 

intervention is described, but 

with no details provided.

No, initial personnel training to 

deliver the intervention is not 

mentioned.

Training for service delivery given to 

personnel before implementation begins 

includes training on intervention content 

(such as a training on a curriculum used in 

the intervention), methods for service 

delivery (such as motivational interviewing), 

and client processing procedures (such as 

intake procedures).

Question 2 and 3 are skipped with a “no” 

response.

2. What initial training did 

personnel receive to deliver 

the intervention?

[open-ended] Describes, for each job title, initial personnel

training for delivery of the intervention. 

Specifies when training occurred and 

training content and format (such as 

classroom, online). For example, a "summer 

training institute" for teachers occurs in 

person (through classroom sessions and 

practice teaching) in the summer before 

teachers begin teaching. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-19

Question Response options Additional guidance

3. Who delivered the initial 

training?

[open-ended] Describes, by job title, who delivered the 

initial training (for example, supervisors, 

personnel involved in service delivery, and 

partners). Separately identifies people with 

different job titles who delivered different 

types of training. For example, Teach For 

America personnel might deliver a "summer 

training institute" for teachers. 

4. Is the ongoing training to 

deliver the intervention 

received by personnel

described?

Yes (full), ongoing personnel

training to deliver the 

intervention is described, with 

details provided.

Yes (limited), ongoing 

personnel training to deliver the 

intervention is described, but 

with no details provided.

No, ongoing personnel training 

to deliver the intervention is not 

mentioned.

Training for personnel involved in service 

delivery during implementation and 

throughout the service delivery period 

includes refresher trainings on intervention 

content, methods for service, and client 

processing procedures (such as intake 

procedures).

Question 5 and 6 are skipped with a “no” 

response.

5. What is the ongoing training 

needed to deliver the 

intervention?

[open-ended] Describes, by job title, the ongoing training 

that helps service providers deliver the 

intervention. Details include when and how 

often training occurred, content (for 

example, curriculum content, use of data 

systems), and format (for example, 

classroom, online). 

6. Who delivered the ongoing 

training?

[open-ended] Describes, by job title, who delivered the 

ongoing training (for example, supervisors, 

service providers, and partners). 

V. Communication systems

1. Is a communication system 

to support coordination 

among personnel and 

partners the described?

Yes (full), a communication 

system is described, with 

details provided.

Yes (limited), a communication 

system is described, but with no 

details provided.

No, a communication system is 

not mentioned.

Communication systems support 

coordination among personnel and partners 

and specify the frequency and the expected 

duration of communication, the parties 

responsible for communication, and the 

expected duration of communication. 

Systems may vary in the extent to which 

they specify and standardize 

communication. For example, a detailed 

plan may state that, for the first six months 

of the intervention, service providers must 

meet as a group with their supervisor for 

one hour each week to discuss topics 

related to recruitment and engagement of 

participants; for the next six months, 

meetings must take place monthly in the 

form of hour-long sessions to discuss 

service provision. 

Question 2 is skipped with a “no” response.

2. What was the communication 

system for supporting 

coordination personnel and 

partners? 

[open-ended] Describes the communication system, 

including the frequency and duration of the 

communication between personnel and with 

any partners, and which individuals are 

included in the communication system.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-20

Question Response options Additional guidance

3. Is a process for ensuring the 

communication system 

functioned as intended 

described?

Yes (full), a process for 

ensuring the communication 

system functioned is described, 

with details provided.

Yes (limited), a process for 

ensuring the communication 

system functioned is described, 

but with no details provided.

No, a process for ensuring the 

communication system 

functioned is not mentioned

Processes to ensure a communication 

system functions as intended include 

obtaining feedback from personnel

regarding the communication system and 

tracking the communication that takes 

place. Details include how feedback was 

obtained regarding the functioning of the 

communication system. For example, a 

communication system may have included 

logs of how often personnel met together\ or 

used regular check-ins with personnel

eliciting feedback on how easily they feel 

they are able to get in contact with each 

other or key intervention partners.

Question 4 is skipped with a “no” response.

4. What was the process for 

ensuring the communication 

system functioned as 

intended?

[open-ended] Describes the process for ensuring the 

communication system functioned as 

intended.

VI. Data systems

1. Is a data system to support 

data collection, analysis, and 

decision making described?

Yes (full), a data system is in 

place, with details provided.

Yes (limited), a data system is 

mentioned, but with no details 

provided.

No, a data system is not 

mentioned.

Data systems (for example, a management 

information system or a standardized Excel 

spreadsheet) capture enrollment and 

participation data and generate reports on 

participation trends. Such systems support 

data collection, analysis, and decision 

making. A detailed description specifies the 

information captured by the system, the 

organization’s capacity to analyze recorded 

data, and how the organization uses the 

data to support its decision-making 

processes.

Question 2 is skipped with a “no” response.

2. What was the data system 

developed to support data 

collection, analysis, and 

decision making?

[open-ended] Describes the data system developed to 

support data collection, analysis, and 

decision making.

3. Is a process for ensuring 

data quality described?

Yes (full), a process for 

ensuring data quality is 

described, with details 

provided.

Yes (limited), a process for 

ensuring data quality is 

described, but with no details 

provided.

No, a process for ensuring data 

quality is not mentioned.

Processes for ensuring that the data 

collected are of high quality, such as 

checking for the completeness, accuracy, 

consistency, and timeliness of the data. For 

example, there is a plan to check 

participation data entered into a data system 

with paper records of attendance or case 

note file reviews.

Question 4 is skipped with a “no” response.

4. What was the process for 

ensuring the quality of the 

data collected through the 

data system?

[open-ended] Describes the process for ensuring that the 

data collected through the data system was 

of high quality.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-21

Question Response options Additional guidance

5. Is a data system to support 

data collection, analysis, and 

decision making described?

Yes (full), a data system is in 

place, with details provided.

Yes (limited), a data system is 

mentioned, but with no details 

provided.

No, a data system is not 

mentioned.

Data systems (for example, a management 

information system or a standardized Excel 

spreadsheet) capture enrollment and 

participation data and generate reports on 

participation trends. Such systems support 

data collection, analysis, and decision 

making. A detailed description specifies the 

information captured by the system, the 

organization’s capacity to analyze recorded 

data, and how the organization uses the 

data to support its decision-making 

processes.

Question 2 is skipped with a “no” response.

E. Enabling context

This subsection addresses the fourth condition of the scaling readiness framework—the enabling 

context. It includes three aspects of the context in which organizations provide services (Table 

A.I.5): 

1. Support for the intervention, including support from organizational leadership and partner 

agencies for implementing and evaluating the intervention. 

2. Innovation and learning, including earlier activities that the organization might have 

undertaken in an effort to introduce new practices or improve the intervention and the 

organization’s ability to understand the reason for any successes. 

3. Improvement in response to challenges, including the identification of any challenges that 

organizations have encountered during implementation of the intervention and the solutions 

to those challenges. 

Table A.I.5. Questions about the enabling context 

Question Response options Additional guidance

I. Leaders, key stakeholders, and partners support for the intervention

1. Is support from 

organizational leaders for 

the intervention 

described?

Yes (full), support from 

organizational leaders for the 

intervention is described, with 

details provided.

Yes (limited), support from 

organizational leaders for the 

intervention is described, but with no 

details provided.

No, support from organizational 

leaders for the intervention is not 

mentioned. 

Describes organizational leaders and their 

commitment to the intervention. For example, 

organization leaders are described as 

engaging in planning activities or making 

implementation of the intervention a priority 

for the organization. 

Question 2 is skipped with a “no” response.

2. How was support from 

organizational leaders for 

the intervention 

demonstrated?

[open-ended] Describes how support was demonstrated by 

organizational leaders.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-22

Question Response options Additional guidance

3. Is support from 

stakeholders and/or 

partners for the 

intervention described?

Yes (full), support from stakeholders 

and/or partners for the intervention 

is described, with details provided.

Yes (limited), support from 

stakeholders and/or partners for the 

intervention is described, but with no 

details provided.

No, support from stakeholders 

and/or partners for the intervention 

was not mentioned.

Describes stakeholders’ and/or partners’ 

commitment to the intervention. Stakeholders 

and partners include service delivery partners 

or leadership at implementation sites. 

Support may be demonstrated by securing 

space for service delivery at local 

implementation sites or removing 

responsibility for non-intervention related 

workload for personnel involved in service 

delivery. 

Question 4 is skipped with a “no” response.

4. How was support from 

organization stakeholders 

and/or partners for the 

intervention 

demonstrated?

[open-ended] Describes how support was demonstrated by 

stakeholders and/or partners.

II. Innovation and learning

1. Is there any description 

of the organization’s 

earlier efforts to be 

innovative?

Yes (full), efforts to be innovative 

are described, with details provided.

Yes (limited), efforts to be innovative 

are described, but with no details

provided.

No, efforts to be innovative were not 

mentioned.

Describes organizational efforts to identify, 

develop, and implement new ways of 

meeting community needs. For example, 

organizational innovation could include 

offering new products or services or 

identifying novel ways of delivering regularly 

available services. 

Question 2 is skipped with a “no” response.

2. What were the 

innovative efforts 

described?

[open-ended] Describes organization’s efforts to be 

innovative.

3. Is there a description of 

the organization’s efforts 

to improve its 

interventions?

Yes (full), efforts to improve its 

interventions are described, with 

details provided.

Yes (limited), efforts to improve its 

interventions are described, but with 

no details provided.

No, efforts to improve its 

interventions are not mentioned.

Describes the way in which organizations 

have improved interventions. Improvement 

may be demonstrated by a description of how 

the organization advanced its practices and 

enhanced the current service array. For 

instance, the evaluated intervention may 

have improved upon a previous version of 

the intervention by making changes to its 

recruitment strategies. 

Question 4 is skipped with a “no” response. 

4. What were the 

improvement efforts 

described?

[open-ended] Describes organization’s efforts to improve 

interventions.

5. Is there any discussion 

of successes regarding 

implementation of the 

intervention?

Yes (full), implementation 

successes are discussed, with 

details provided.

Yes (limited), implementation 

successes are discussed, but with 

no details provided.

No, implementation successes were 

not mentioned.

Describes successes related to implementing 

intervention core elements, recruiting or 

serving the target population(s), supporting 

implementation, and creating an enabling 

context. 

Question 6 is skipped with a “no” response. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-23

Question Response options Additional guidance

6. What were the 

implementation 

successes of the 

intervention and the 

reasons for those 

successes?

[open-ended] Describes the implementation successes and 

the reasons for those successes. 

III. Improvement in response to challenges

1. Is there any discussion 

of challenges regarding 

implementation of the 

intervention?

Yes, challenges were discussed.

No, challenges were not discussed.

Challenges could be related to intervention 

components, target population(s), 

implementation supports, and the enabling 

context. 

Question 2 is skipped with a “no” response. 

2. What were the 

challenges regarding 

implementation of the 

intervention and 

solutions to those 

challenges?

[open-ended] Includes the implementation challenges and 

the solutions to those challenges, if any. 

F. Implementation infrastructure

This section of the rubric addresses the fifth condition of the scaling readiness framework— 

implementation infrastructure. It includes questions in four major areas, each of which describes 

the organization’s infrastructure in support of implementation of the intervention (Table A.I.6). 

1. Financial resources, including how the organization demonstrates that it or its partners 

successfully supported implementation of the intervention with dedicated financial resources.

2. Sufficient personnel, including whether the organization dedicated the personnel needed 

either to implement or scale the intervention. 

3. Materials, including the standardization of materials and tailoring of materials to meet 

participants’ needs. 

4. Physical space, including the availability of space necessary for service delivery. 

5. Human resource system, including a human resource system that is critical in typically 

overseeing three highly important, recognized implementation drivers—hiring, training, and 

ongoing supervision. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-24

Table A.I.6. Questions about implementation infrastructure

Question Response options Additional guidance

I. Financial resources

1. Does the organization provide 

funding for implementation of 

the intervention?

Yes (full), adequate funding is 

in place, with details provided.

Yes (limited), adequate funding 

is in place, but with no details 

provided. 

No, funding is not mentioned.

Describes the amount and source of funds 

and, if applicable, partners’ funding for 

implementation of the intervention.

Question 2 is skipped with a “no” response. 

2. What funding was provided by 

the organization for 

implementation of the 

intervention?

[open-ended] Describes the funding provided by the 

organization for implementation of the 

intervention.

II. Sufficient personnel

1. Does the organization provide 

personnel to implement the 

intervention (including dedicated 

supervisors and service 

providers)?

Yes (full), the organization 

provides dedicated personnel, 

with details provided.

Yes (limited), the organization 

provides dedicated personnel, 

but with no details provided.

No, dedicated personnel are

not mentioned.

Describes the number of personnel needed 

for implementation, whether personnel

worked exclusively on intervention 

implementation or also had nonintervention 

responsibilities, and, if applicable, partners’ 

role in providing personnel for the 

intervention.

Question 2 is skipped with a “no” response.

2. How many personnel were 

provided by the organization to 

implement the intervention?

[open-ended] Describe the number of personnel needed 

for implementation.

III. Materials 

1. Is there a description of 

materials needed for the 

intervention?

Yes (full), intervention materials 

are described, with details 

provided.

Yes (limited), intervention 

materials are mentioned, but 

with no details provided.

No, needed intervention 

materials are not mentioned.

Describes the materials needed to 

implement the intervention, such as 

workbooks, culturally sensitive documents, 

or other handouts.

Question 2 is skipped with a “no” response. 

2. What were the materials 

needed for the intervention?

[open-ended] Describes the intervention materials needed 

for implementation.

IV. Physical space

1. Is there a description of the 

physical space needed for the 

intervention?

Yes (full), the physical space is 

described, with details 

provided.

Yes (limited), the physical 

space is described, but with no 

details provided.

No, the physical space is not 

mentioned.

Describes the physical space needed for 

implementation, such as the size or number 

of classrooms needed to fit the target 

number of participants or the availability of 

private meeting spaces for one-on-one 

service delivery.

Question 2 is skipped with a “no” response.

2. What physical space was 

needed for implementation of 

the intervention?

[open-ended] Describes the physical space needed for 

implementation.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-25

Question Response options Additional guidance

V. Human resource system

1. Is a human resource system in

place to hire, supervise, and 

develop the personnel? 

Yes (full), a human resource 

system is in place, with details 

provided.

Yes (limited), a human resource 

system is in place, but with no 

details provided.

No, a human resource system 

is not mentioned.

A human resource system supports 

implementation through processes and 

procedures for hiring personnel who meet 

qualifications for implementation with 

fidelity; for defining a supervisory structure 

(including the identification of who reports to 

whom, the frequency of supervisory 

meetings, mechanisms for reporting on 

personnel progress); and for providing 

personnel development (including planned 

training for personnel and opportunities for 

performance monitoring and improvement). 

A clearly defined human resource system 

specifies how these supports are routinized 

and whether procedural guidelines are 

developed to standardize the supports.

Question 2 is skipped with a “no” response.

2. What was the human resource

system put in place for the

intervention?

[open-ended] Describes the human resource system put 

in place for the intervention.

II. SCALING READINESS SECTION

The scaling readiness section of the rubric is used to assess an intervention’s readiness to be 

scaled and the organization’s ability to support that scaling as described in the organization’s 

narrative application for funding or in the organization’s scaling plan. This section is used 

primarily to assess the degree to which the organization specifies details critical for assessing 

that the intervention and the organization will be able to successfully implement the intervention 

and to what degree the intervention proposed for scaling differs from the intervention that was 

evaluated (and described using the previous section of the rubric that was applied to the 

intervention’s evaluation report). The section includes six subsections. One subsection collects 

background information about the approach to scaling and the context in which implementation 

will occur, and the next five subsections collect information on each of the five conditions 

needed for scaling. 

A. Scaling approach and its context

Table A.II.1 provides the first subsection of the rubric, which includes two types of questions 

about the organization’s approach to scaling and the context in which it occurs: 

1. Scaling approach planned for the intervention. Research from implementation science

includes three forms of scaling. (1) Expansion extends the intervention to more people in the

same target population in the same location. Successful expansion requires the intervention

and the organization to serve a larger number of participants with the same service quality

and in a manner demonstrating fidelity to the model’s design. (2) Replication extends the

intervention to the same target population, but in a new location. Successful replication

requires the intervention and the organization to maintain service quality and fidelity to the
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-26

intervention in the new location. (3) Adaptation extends the intervention to a new target 

population. Successful adaption requires the organization to change the intervention in a way 

that maintains service quality. AmeriCorps also funded some AmeriCorps State and National 

grantees to sustain or deepen the intervention services instead of scaling the intervention. 

Some grantees received funds to sustain services, which means that the intervention will 

continue serving the same target population in the current location without any purposeful 

changes to it. Some grantees received funds to deepen services, which means that the

intervention will serve the same target population in the current location, but with enhanced 

services (for example, more hours of job coaching). 

2. Local context and demand associated with the geographic area where the intervention is 

proposed for scaling. Local contextual factors include any regional, state, and local features 

that may affect implementation of the intervention, and local demand factors include 

evidence of demand for the intervention in the local area.

Table A.II.1. Questions about scaling approach and local context

Question Response options Additional guidance

I. Scaling approach

1. What is the organization’s 

proposed intervention?

[open-ended] Describes the proposed intervention, 

including its core components and 

activities and plans for implementation. 

2. How does the organization 

self-categorize its scaling 

approach? 

Expansion

Replication

Adaption

Sustaining services

Deepening services

Not described

Describes the self-categorization of the 

scaling approach.

3. Based on the description of 

the proposed intervention, 

what is the scaling approach? 

Expansion

Replication

Adaption

Sustaining services

Deepening services

Describes the scaling approach as 

defined by implementation science.

4. Will the intervention 

proposed for scaling be 

implemented by the 

organization that developed 

it? 

Yes, the intervention proposed for 

scaling will be implemented by the 

organization developing it.

No, the intervention proposed for 

scaling was developed by a 

different organization.

Describes whether the scaled 

intervention was developed by the 

implementing organization or by another 

entity.

II. Local context and demand

1. Does the organization 

provide evidence of demand 

in the local area for the 

intervention that is being 

scaled? 

Yes (full), evidence exists for local 

demand for the intervention being 

scaled, with details provided.

Yes (limited), demand exists for the 

intervention being scaled, but with 

no details provided.

No, local demand is not mentioned 

or does not exist.

Means of illustrating local demand 

include the results of a needs 

assessment, a landscape analysis 

specifying the scope of the intervention, 

or informal data collection via attendance 

at community meetings or conversations 

with key stakeholders. The results of 

such efforts must motivate the proposed 

scaling approach. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-27

Question Response options Additional guidance

2. Does the organization 

provide evidence that the 

local, regional, or state 

context will be conducive to 

scaling the intervention?

Yes (full), the context will be 

conducive to scaling, with details 

provided. 

Yes (limited), the context will be 

conducive to scaling, but with no 

details provided.

No, the context is not mentioned or 

not conducive to scaling.

Evidence exists that key community 

partners will value the intervention and 

that stakeholders at the local, regional, 

or state level will not impede 

implementation. Such evidence might 

include memorandum of understanding, 

grants, contracts or more formal 

arrangements, as well as tacit

relationships, such as board 

memberships.

B. Well-specified intervention

This subsection addresses the first condition of the scaling readiness framework—the wellspecified intervention—as described in the grantee’s plan for scaling. It includes overall 

questions on the core elements of the intervention proposed for scaling and their dimensions for 

service delivery, as well as the definition of participants completing the intervention (subsection 

A.I.2 describes these dimensions). The subsection asks questions about five dimensions of 

intervention specification (Table A.II.2): (1) core elements (services, the intervention’s content 

area, and whether a standardized intervention model is used), (2) mode of service delivery, (3) 

intensity, (4) personnel needs, (5) setting, and (6) definition of completion. For each, questions 

(1) categorize the specificity of the proposed intervention elements, (2) identify differences 

between the proposed intervention element and the evaluated intervention element (as described 

in the intervention evaluation reports and captured in subsection A.I.2 of the rubric), and, if 

applicable, (3) report the rationale for differences between the proposed intervention element and 

evaluated intervention element. If the organization proposes to expand to a new population or 

replicate the intervention, it may not make any changes to the intervention itself. 

Table A.II.2. Questions about specifying the intervention proposed for scaling

Question Response options Additional guidance

I. Core elements 

1. Are the core elements 

of the intervention well 

specified?

Yes (full), core elements are well 

specified, with details provided.

Yes (limited), core elements are 

specified, but with no details provided.

No, core elements are not mentioned.

Core elements are the set of activities or 

curricula that result in participants’ 

receipt of services, such as a description 

of services, the content area of the 

intervention, the existence of a 

standardized intervention model, the 

services or curriculum comprising the 

intervention, and the content area of the 

intervention to be delivered. A 

standardized intervention is considered 

to have a clearly specified intervention. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-28

Question Response options Additional guidance

2. Do the proposed core 

elements differ from 

those in the evaluation? 

Yes (full), proposed core elements differ 

from those evaluated, with details of the 

differences provided.

Yes (limited), proposed core elements 

differ from those evaluated, but with no 

details provided.

Do not know; proposed core elements 

are not defined.

No, proposed core elements for the 

evaluation and intervention are the 

same. 

The proposed core elements of the 

intervention for scaling may differ from 

the core elements of the intervention that 

were evaluated. Changes to the 

intervention could include the addition of 

services, the modification of existing 

services, the discontinuation of some 

services, the addition of content, or the 

use of a new curriculum, all described in 

sufficient detail to allow an external 

audience to replicate them.

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the core elements? 

Yes, a rationale or support for the 

change(s) is provided.

No, a rationale or support is not 

provided. 

Organizations may provide a reason for 

differences between the proposed 

intervention core elements and the 

evaluated core elements. Support might 

refer, for example, to research showing 

that a proposed change has positive 

impacts on outcomes.

II. Mode of service delivery

1. Is the service delivery 

model well-specified? 

Yes (full), the service delivery model is 

well-specified, with details provided.

Yes (limited), the service delivery model 

is specified, but with no details provided. 

No, the service delivery model is not 

mentioned.

The service delivery model specifies how 

services are delivered, who delivers 

services, and the mode through which 

services are delivered. For example, 

services may be delivered in groups or 

one-on-one meetings, by peer educators 

or teams of facilitators, and the through 

the use of electronic communication or 

in-person meetings. A standardized 

intervention clearly specifies the service 

delivery model. 

2. Does the proposed 

service delivery model 

differ from the service 

delivery model 

implemented during the 

evaluation? 

Yes (full), the proposed service delivery 

model differs from the intervention 

evaluated, with details provided.

Yes (limited), the proposed service 

delivery model differs from the 

intervention evaluated, but with no 

details provided.

Do not know; the service delivery model 

is not defined.

No, the proposed and evaluation service 

delivery models are the same. 

The proposed service delivery model for 

the intervention for scaling may differ 

from the service delivery model of the 

intervention that was evaluated. 

Changes to the service delivery model 

could include the type of delivery (such 

as face-to-face meetings or electronic 

communications) and the nature of 

delivery (such as in a group or one-onone meetings). 

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the service delivery 

model? 

Yes, a rationale or support for the 

change(s) is provided.

No, a rationale or support for the 

proposed change(s) is not provided.

Organizations may provide a reason for 

differences between the proposed 

interventions’ service delivery mode and 

the evaluated service delivery mode. 

Support could include research showing 

that a proposed change has positive 

impacts on outcomes.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-29

Question Response options Additional guidance

III. Intensity

1. Is the intensity of the 

intervention wellspecified?

Yes (full), the intensity is well-specified, 

with details provided.

Yes (limited), the intensity is specified, 

but with no details provided.

No, the intensity is not mentioned.

Describes the total number of hours of 

intervention programming participants 

will receive. This is based on how long 

an intervention will last (duration) and 

the amount of services participants will 

receive (dosage), including the 

frequency of intervention interaction and 

hours of service. 

2. Does the proposed 

intensity of the scaled 

intervention differ from 

the intensity of the 

intervention evaluated? 

Yes (full), the proposed intensity differs 

from the intervention evaluated, with 

details of the differences provided.

Yes (limited), the proposed intensity 

differs from the intervention evaluated, 

but with no details provided.

Do not know; the intensity is not defined.

No, the proposed and evaluated intensity 

is the same.

The proposed intensity of the 

intervention for scaling may differ from 

the intensity of the intervention that were 

evaluated. Differences may include 

changes to how long an intervention

lasts, how many times a participant and 

a service provider meet, and how many 

hours of service content a participant 

receives.

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the intensity of the 

intervention? 

Yes, a rationale for the change(s) is 

included, with details provided.

No, a rationale for the change(s) to 

dosage or duration is not mentioned.

Organizations may provide a reason for 

differences between the proposed 

interventions’ intensity and the evaluated 

intensity. Support might include research 

showing that a proposed change has 

positive impacts on outcomes.

IV. Personnel needs

1. Are the qualifications 

clearly specified for the 

personnel involved in 

delivering the 

intervention?

Yes (full), personnel qualifications are 

clearly specified, with details provided.

Yes (limited), personnel qualifications 

are specified, but with no details 

provided.

No, personnel qualifications are not 

mentioned.

Clearly specified qualifications for each 

job title involved in delivering services for 

the scaled intervention. Qualifications 

could include required training or 

education and experience levels of the 

personnel delivering the intervention 

services. 

2. Do the proposed 

personnel qualifications 

after scaling differ from 

the qualifications of 

personnel delivering 

services as part of the 

evaluation? 

Yes (full), the proposed personnel

qualifications differ from the intervention 

evaluated, with details of the differences 

provided.

Yes (limited), the proposed personnel

qualifications differ, but with no details of 

the differences provided.

Do not know; the personnel

qualifications are not defined.

No, the proposed and evaluation 

personnel are the same.

The proposed personnel of the 

intervention for scaling may differ from 

the personnel of the intervention that 

were evaluated. Differences in personnel

qualifications may include education, 

experience, or various levels of training.

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the personnel

qualifications? 

Yes, a rationale or support exists for the 

personnel qualifications change(s).

No, a rationale for the proposed 

change(s) is not mentioned.

Organizations may provide a reason for 

differences between the proposed 

interventions’ personnel and the 

personnel of the evaluated intervention. 

A rationale could include research 

showing that a proposed change has 

positive impacts on outcomes. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-30

Question Response options Additional guidance

V. Setting 

1. Is the implementation 

setting clearly specified?

Yes (full), the implementation setting is 

clearly specified, with details provided.

Yes (limited), the implementation setting 

is specified, but with no details provided.

No, the implementation setting is not 

mentioned.

Implementation setting includes the 

geographic location of implementation 

and where an intervention takes place. 

The description of the implementation 

setting may include the city and state; 

whether the intervention is being 

implemented in a school, communitybased organization, or workforce center; 

and the number of implementation 

settings. The number of implementation 

sites is not considered a change unless 

the new sites are located in a new city, 

state, or type of setting (such as a 

different implementing agency).

2. Does the proposed 

implementation setting 

differ from the 

evaluation’s 

implementation setting?

Yes (full), the proposed setting differs 

from the evaluation setting, with details 

provided.

Yes (limited), the proposed setting differs 

from the evaluation setting, but with no 

details provided.

Do not know; the setting is not 

described.

No, the proposed and evaluation setting 

are the same.

The proposed implementation setting of 

the intervention for scaling may differ 

from the implementation setting of the 

intervention that was evaluated. 

Differences in the implementation setting 

might include delivering services in a 

new city, state, region, or urban/rural 

area as well as switching from a 

community-based to a school-based 

intervention. 

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the implementation 

setting?

Yes, a rationale exists for the change(s). 

No, a rationale is not provided for the 

proposed change(s).

Organizations may provide a reason for 

differences between the implementation 

setting of the proposed intervention and 

the setting of the evaluated intervention. 

A rationale could include research 

showing that a proposed change has 

positive impacts on outcomes.

VI. Definition of completion

1. Are the criteria for 

completion wellspecified?

Yes (full), the criteria for completion are

well-specified, with details provided.

Yes (limited), the criteria for completion 

are specified, but with no details 

provided.

No, the criteria for completion are not 

mentioned. 

Criteria for completion indicates what 

participants must do to be considered 

“completers” or “graduates” for the 

intervention for scaling. To sufficiently 

specify the completion criteria, there 

must also be a definition of a 

“participant,” including the minimum 

amount of services needed to be 

received in order to be considered a 

“participant.” For example, to be 

considered a participant, a youth must 

attend at least 1 case manager meeting; 

to be considered a completer or 

graduate, participants must attend 10 

case manager meetings.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-31

Question Response options Additional guidance

2. Do the criteria for 

completion differ from 

the criteria described in 

the evaluated 

intervention?

Yes (full), the proposed completion 

criteria differ from the criteria for the 

intervention evaluated, with details 

provided.

Yes (limited), the proposed completion 

criteria differ from the criteria for the 

intervention evaluated, but no details are 

provided.

Do not know; the completion criteria are 

not described.

No, the proposed and evaluated criteria 

for completion are the same.

The completion criteria of the 

intervention for scaling may differ from 

the completion criteria of the evaluated 

intervention. Changes to the criteria for 

completion might include requiring a new 

minimum amount of services to be 

attended to be considered a participant 

or a change in the number of services 

received to be considered a graduate of 

the intervention.

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the participant 

completion definition?

Yes, a rationale or support for the 

change(s) is provided.

No, a rationale or support for the 

proposed change(s) is not provided.

Organizations may provide a reason for 

differences between the completion 

criteria of the proposed intervention and 

the completion criteria of the evaluated 

intervention. A rationale could include 

research showing that a proposed 

change has positive impacts on 

outcomes.

C. Well-defined target population

This subsection of the rubric addresses the second condition of the scaling readiness 

framework—well-defined target population. It includes questions discussed in subsection A.I.3 

about the target population proposed for scaling (Table A.II.3) and describes differences between 

the proposed target population after scaling and the population studied in the evaluation. If the 

organization proposes to expand the population size or replicate the intervention, it may not 

make any changes to the target population. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-32

Table A.II.3. Questions about the target population proposed for scaling

Question Response options Additional guidance

1. Is the target population wellspecified?Yes (full), the target population is 

clearly specified, with details provided.

Yes (limited), the target population is 

specified, but with no details provided.

No, the target population is not 

mentioned.

Describes who is eligible to participate 

in the intervention, including inclusion 

and exclusion criteria. For example, a 

proposed intervention to serve 

unemployed adults should include 

examples of eligible age ranges (for 

example, 18-64), length of 

unemployment, and any other 

inclusion or exclusion criteria such as 

no felony conviction.

2. Does the proposed target 

population differ from the 

evaluation sample? 

Yes (full), the proposed and evaluation 

populations differ, with details of the 

differences provided.

Yes (limited), the proposed and 

evaluation populations differ, but with 

no details of the differences provided.

Do not know; the populations are not 

defined.

No, the proposed and evaluation 

populations are the same. 

Changes to the target population 

include age requirements (for 

example, expanding the eligible age 

range from 11- to 13-year-old children 

to 10- to 14-year-old children), 

personal characteristics (for example, 

risk factors or education levels), or 

eligibility (for example, veterans or 

youth). 

Question 3 is skipped with a “don’t 

know” or “no” response. 

3. Does a rationale or support 

exist for the proposed 

change(s) to the target 

population?

Yes, a rationale exists for the 

change(s), with details provided.

No, no rationale for the change(s) is 

mentioned.

A rationale could include research 

showing that a proposed change has 

had positive impacts on outcomes in 

other studies or that modifications 

could be based on the 

recommendations. 

D. Implementation supports

This section of the rubric addresses the third condition of the scaling readiness framework—

implementation supports. It includes questions about the implementation supports discussed in 

subsection A.I.4 that are available for the scaled intervention (Table A.II.4): (1) implementation 

monitoring team, (2) performance procedures, (3) CQI, (4) preservice and inservice training for 

the personnel, (5) communication system, and (6) data system. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-33

Table A.II.4. Questions about implementation supports for the intervention proposed for 

scaling

Question Response options Additional guidance

I. Implementation monitoring team

1. Is a team proposed to 

monitor implementation of 

the scaled intervention? 

Yes (full), a team is proposed, with 

details provided.

Yes (limited), a team is described, 

but with no details provided,

No, a team is not mentioned,

A monitoring team ensures that 

implementation takes place as planned. 

Team members may play other roles in the 

organization (such as supervisors, project 

managers, and project directors) or they 

may be other personnel members 

dedicated exclusively to ensuring that the 

intervention is implemented with fidelity to 

the model. Details include specific 

information about members of the 

implementation team. 

2. Does a process for 

monitoring fidelity to the 

intervention model exist for 

the scaled intervention?

Yes (full), a process for monitoring 

fidelity to the intervention model is 

described, with details provided.

Yes (limited), a process for 

monitoring fidelity to the 

intervention is described, but with 

no details provided.

No, a process for monitoring fidelity 

is not mentioned

A process for monitoring fidelity to the 

intervention model ensures that services 

are being delivered as intended. The 

process for monitoring implementation 

fidelity could include collecting service 

delivery data through observations during 

site visits or regular reviews of service data 

entered into a data system. Details include 

identification of the individual (by job title) 

responsible for assessing fidelity, 

frequency of implementation monitoring, 

processes for managing/supervising 

personnel members, and frequency of 

personnel management or supervisory 

meetings.

II. Performance procedures

1. Do performance 

benchmarks exist for 

personnel involved in 

service delivery of the 

scaled intervention?

Yes (full), performance 

benchmarks are described, with 

details provided.

Yes (limited), performance 

benchmarks are described, but 

with no details provided.

No, performance benchmarks are 

not mentioned.

Performance benchmarks are specific 

goals that personnel must meet as related 

to delivery of the scaled intervention, such 

as such as the number of participants 

contacted per personnel member or the 

number of participants served per 

personnel member.

2. Do procedures for 

monitoring achievement 

of performance 

benchmarks exist for the 

scaled intervention?

Yes (full), procedures for 

monitoring achievement of 

performance benchmarks are 

described, with details provided.

Yes (limited), procedures for 

monitoring achievement of 

performance benchmarks are 

described, but with no details 

provided.

No, procedures for monitoring 

achievement of performance 

benchmarks are not mentioned. 

Procedures for monitoring achievement of 

performance benchmarks include how 

benchmarks are measured and collected, 

who reviews progress toward benchmarks, 

and the frequency of performance 

monitoring.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-34

Question Response options Additional guidance

III. Continuous quality improvement (CQI)

1. Are CQI processes 

proposed for the scaled 

intervention? 

Yes (full), CQI processes are 

proposed, with details provided

Yes (limited), CQI processes are 

proposed, but with no details 

provided.

No, CQI processes are not 

mentioned.

CQI processes refer to procedures for 

continuously assessing the quality of the 

intervention as implemented to improve 

implementation practice. CQI includes 

regularly testing the intervention and 

making adjustments as needed, with 

continual retesting of the modifications to 

ensure that the intervention is succeeding 

as planned. Details include a description of 

the data collected throughout this process. 

2. Does a plan exist to 

analyze data to support 

CQI for the scaled 

intervention?

Yes (full), data analysis to support 

CQI are described, with details 

provided.

Yes (limited), data analysis to 

support CQI are described, but 

with no details provided.

No, data analysis to support CQI is 

not mentioned.

CQI requires the collection of data on 

performance benchmarks and use of the 

data to provide ongoing personnel

development training and the delivery of 

technical assistance to partners. Describes 

plans for using data to support CQI 

includes examining personnel achievement 

of performance benchmarks. 

IV. Preservice and inservice training for personnel

1. Do plans exist to provide 

initial training to personnel

to deliver the scaled 

intervention? 

Yes (full), initial training plans are 

described, with details provided.

Yes (limited), initial training plans 

are described, but with no details 

provided.

No, initial training plans are not 

mentioned.

Training for service delivery given to 

personnel before implementation begins 

includes training on intervention content 

(such as a training on a curriculum used in 

the intervention), methods for service 

delivery (such as motivational 

interviewing), and client processing 

procedures (such as intake procedures).

2. Do plans exist to provide 

ongoing training to 

personnel to implement the 

scaled intervention?

Yes (full), ongoing training plans 

are described, with details 

provided.

Yes (limited), ongoing training 

plans are described, but with no 

details provided.

No, ongoing training plans are not 

mentioned.

Training for service delivery given to 

personnel during implementation and 

throughout the service delivery period, 

includes refresher trainings on intervention 

content, methods for service, and client 

processing procedures (such as intake 

procedures).
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-35

Question Response options Additional guidance

V. Communication system

1. Does a communication 

system exist to support 

coordination among 

personnel and partners for 

the scaled intervention?

Yes (full), a communication system 

exists, with details provided. 

Yes (limited), a communication 

system is mentioned, but with no 

details provided.

No, a communication system is not 

mentioned.

Communication systems support 

coordination among personnel and 

partners and specify the frequency and 

expected duration of communication and 

the parties responsible for communication. 

Systems may vary in the extent to which 

they specify and standardize 

communication. For example, a detailed 

plan may state that, for the first six months 

of the intervention, service providers must 

meet as a group with their supervisor for 

one hour each week to discuss topics 

related to recruitment and engagement of 

participants; for the next six months, 

meetings must take place monthly in the 

form of hour-long sessions to discuss 

service provision. 

2. Do processes exist for 

ensuring the 

communication system is 

functioning as intended for 

the scaled intervention?

Yes (full), processes for ensuring 

the communication system is 

functioning as intended are 

described, with details provided.

Yes (limited), processes for 

ensuring the communication 

system is functioning as intended 

are described, but with no details 

provided.

No, processes for ensuring the 

communication system is 

functioning as intended are not 

mentioned

Processes to ensure a communication 

system functions as intended include 

obtaining feedback from personnel

regarding the communication system and 

tracking the communication that takes 

place. Details include how feedback was 

obtained regarding the functioning of the 

communication system. For example, a 

communication system may have included 

logs of how often personnel met together 

or used regular check-ins with personnel

eliciting feedback on how easily they feel 

they are able to get in contact with each 

other or key intervention partners.

VI. Data system

1. Does a data system exist 

to support data collection, 

analysis, and decision 

making for the scaled 

intervention?

Yes (full), a data system is 

described, with details provided.

Yes (limited), a data system is 

described, but with no details 

provided.

No, a data system is not 

mentioned.

Data systems capture enrollment and 

participation data and generate reports on 

participation trends. Such systems support 

data collection, analysis, and decision 

making. Details include the information 

captured by the system, the organization’s 

capacity to analyze recorded data, and 

how the organization uses the data to 

support its decision-making processes.

2. Does a process exist for 

ensuring the data quality for 

the scaled intervention?

Yes (full), a process for ensuring 

the data quality is described, with 

details provided.

Yes (limited), a process for 

ensuring the data quality is 

described, but with no details 

provided.

No, a process for ensuring the data 

quality is not mentioned.

Processes for ensuring that the data 

collected are of high quality, such as 

checking for the completeness, accuracy, 

consistency, and timelines of the data. For 

example, there is a plan to check 

participation data entered into a data 

system with paper records of attendance or 

case note file reviews.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-36

E. Enabling context

This subsection addresses the fourth condition of the scaling readiness framework—the enabling 

context. It includes questions about the organizational supports discussed in subsection A.I.5 that 

are available for the scaled intervention (Table A.II.5). Support for scaling pertains to the 

organization’s leaders and key stakeholders, as well as to the organization’s culture and its focus 

on innovation, learning, and improvement. The enabling context involves questions in three 

major areas about the system in which organizations provide services: (1) organizational 

leadership and partner support, (2) innovation and learning and (3) improvements in response to 

challenges. 

Table A.II.5. Questions about the enabling context for the organization proposing the 

scaling

Question Response options Additional guidance

I. Leaders, key stakeholders, and partners support for the intervention

1. Is support from 

organizational leaders for 

scaling the intervention 

described?

Yes (full), support from organizational 

leaders for scaling the intervention is 

described, with details provided.

Yes (limited), support from 

organizational leaders for scaling the 

intervention is described, but with no 

details provided.

No, support from organizational 

leaders for scaling is not mentioned.

Describes organizational leaders and 

their commitment to scaling the 

intervention. For example, 

organization leaders are part of the 

team that is overseeing the 

AmeriCorps State and National grant 

and the scaling plan for the 

intervention.

Question 2 is skipped with a “no” 

response.

2. Is support from 

stakeholders and/or partners 

for scaling the intervention 

described?

Yes (full), support from stakeholders 

and/or partners for scaling the 

intervention is described, with details 

provided.

Yes (limited), support from 

stakeholders and/or partners for 

scaling the intervention is described, 

but with no details provided. 

No, support from stakeholders and/or 

partners for scaling the intervention is 

not mentioned.

Describes stakeholders and/or 

partners’ commitment to scaling the 

intervention. Stakeholders and 

partners include service delivery 

partners or leadership at 

implementation sites. Support may be 

demonstrated by securing space for 

service delivery or removing 

responsibility for non-intervention 

related workload for service providers

so they can focus on scaling the 

intervention.

II. Innovation and learning

1. Has the organization 

previously participated in 

efforts to be innovative? 

Yes (full), efforts to be innovative are 

described, with details provided.

Yes (limited), efforts to be innovative 

are described, but with no details 

provided.

No, efforts to be innovative were not 

mentioned,

Describes organizational efforts to 

identify, develop, and implement new 

ways of meeting community needs. 

For example, organizational innovation 

could include offering new products or 

services or identifying novel ways of 

delivering regularly available services. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-37

Question Response options Additional guidance

2. Has the organization 

previously participated in 

activities to improve its 

interventions? 

Yes (full), efforts to improve its 

interventions were described, with 

details provided.

Yes (limited), efforts to improve its 

interventions were described, but with 

no details provided.

No, efforts to improve its interventions

were not mentioned.

Describes the way in which 

organizations have improved 

interventions. Improvement may be 

demonstrated by a description of how 

the organization advanced its 

practices and enhanced the current 

service array. For instance, the 

evaluated intervention may have 

improved upon a previous version of 

the intervention by making changes to 

its recruitment strategies. 

II. Improvements in response to challenges

1. Did the organization 

previously face challenges in 

supporting intervention 

implementation?

Yes (full), the organization faced 

challenge(s), with details provided.

Yes (limited), the organization faced 

challenges, but with no details 

provided.

No, no challenges are described.

Describes earlier challenges 

experienced by the organization in 

providing organizational support for 

implementation of the intervention. For 

example, organizational leaders might 

have changed or a previous leader 

might not have endorsed the 

intervention and thus refused to 

dedicate personnel to its 

implementation. 

Question 2 is skipped with a “no” 

response. 

2. Did the organization make 

improvements to address 

earlier challenges in 

providing organizational 

support for the scaled 

intervention? 

Yes (full), the organization made 

improvements to address 

challenge(s), with details provided.

Yes (limited), the organization made 

improvements to address challenges, 

but with no details provided.

No, improvements to address 

challenges were not mentioned.

Describes improvements made to 

address challenges that may include a 

change in organizational structure or 

the identification of new funding 

sources.

F. Implementation infrastructure

This subsection addresses the fifth condition of the scaling readiness framework—

implementation infrastructure. It includes questions about the implementation infrastructure

discussed in subsection A.I.6 that are available for the scaled intervention (Table A.II.6): (1) 

financial resources, (2) sufficient personnel to implement the intervention, (3) materials, (4) 

physical space, and (5) a human resource system. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

A-38

Table A.II.6. Questions about implementation infrastructure for the organization 

proposing the scaling

Question Response options Additional guidance

I. Financial resources

1. Will the organization 

provide funding for the 

scaled intervention? 

Yes (full), funding for scaling 

exists, with details provided.

Yes (limited), funding after 

scaling exists, but with no details 

provided.

No, funding for scaling is not 

mentioned.

Describes how the organization and, if 

applicable, its partners will provide funding to 

implement the intervention after scaling. Details 

include the amount and source of funds and the 

plan to continue providing funds in the future.

II. Sufficient personnel

1. Will the organization 

provide dedicated 

personnel to implement 

the scaled intervention? 

Yes (full), personnel plans are 

described, with details provided.

Yes (limited), personnel plans 

exist, but with no details 

provided.

No, personnel plans are not 

mentioned.

Describes the number of personnel who will 

work on implementation (as compared to the 

number required by the intervention model), 

whether personnel will work exclusively on 

intervention implementation, and, if applicable, 

partners’ role in providing personnel to 

implement the intervention.

III. Materials 

1. Will the organization 

have the materials 

needed for the scaled 

intervention?

Yes (full), materials are 

described, with details provided.

Yes (limited), materials are 

mentioned, but with no details 

provided.

No, materials are not mentioned.

Describes how the organization will obtain 

materials needed for implementation after 

scaling, such as workbooks, culturally sensitive 

documents, or other handouts. 

IV. Physical space

1. Will the organization 

have the physical space 

needed for the scaled 

intervention?

Yes (full), the physical space is 

described, with details provided.

Yes (limited), the physical space 

is mentioned, but with no details 

provided.

No, the physical space is not 

mentioned.

Describes how the organization has or will obtain 

the physical space needed for implementation 

after scaling, such as classrooms to fit the target 

number of participants or private meeting spaces 

for one-on-one service delivery.

IV. Human resource system

1. Is a human resource 

system in place to hire, 

supervise, and develop 

the personnel for the 

scaled intervention? 

Yes (full), a human resource 

system is in place, with details 

provided.

Yes (limited), a human resource

system is mentioned, but with no 

details provided.

No, a human resource system is 

not mentioned.

Describes a human resource system to support 

implementation after scaling. Details specify how 

the human resource system supports processes 

and procedures for hiring appropriate personnel, 

describe a supervisory structure, and provide for 

personnel development, including a description 

of how supports are routinized and standardized. 
APPENDIX B:

CHARACTERISTICS OF INTERVENTIONS WITH 

EVIDENCE OF EFFECTIVENESS
This page has been left blank for double-sided copying. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

B-3 

In Table B.1, we provide details about the characteristics of each of the 32 interventions with 

demonstrated evidence that they work. Characteristics include the intervention’s focus area, 

priority objective, core elements, implementation setting, and use of AmeriCorps members in 

service delivery. Our review determined AmeriCorps members’ involvement in delivering 

intervention services based on evidence documents’ explicitly referencing AmeriCorps members 

in the description of the intervention. It is possible that some evidence documents did not provide 

sufficient information on the personnel involved in the intervention delivery, including the 

presence of AmeriCorps members. Information for the interventions was extracted from the 

evidence documents submitted to AmeriCorps using the review rubric described in Appendix A. 

Table B.1. Characteristics of interventions with evidence of effectiveness 

Interventions 

by focus area

(Intervention 

name)

Priority 

objective Core elements

Implementation 

setting

AmeriCorps

members 

referenced 

in evidence 

document?

Economic 

opportunity 1

(Financial 

Opportunity 

Centers)

Employability and 

financial literacy

• Financial counseling

• Employment search/placement 

assistance

• Income support counseling

• Community-based 

organizations

• Urban

• Geographic area 

described as 

nationally or across 

multiple regions

Yes

Economic 

opportunity 2

(National 

Fund for 

Workforce 

Solutions) 

Employability • Assistance obtaining employability 

and training credentials 

• Assistance enrolling in college 

and specialized apprenticeships

• Job readiness training

• Pre-apprenticeship training

• Industry-focused training

• Job search assistance

• Career advancement services

• Community-based 

organization and 

employers

• Urban

• Midwest

No

Economic 

opportunity 3

(Work 

Advance)

Employability • Intensive screening

• Pre-employment and career 

readiness services

• Occupational skills training

• Job development and placement 

services

• Career retention and 

advancement services

• Community-based 

organizations and 

employers

• Urban

• Northeast, Midwest, 

and South

No

Economic 

opportunity 4

(SaveUSA) 

Financial literacy • Matched savings accounts for 

low- and moderate-income tax 

filers 

• Community-based 

organizations

• Urban

• Northeast and South

No

Economic 

opportunity 5

(Professional 

Training 

Corps) 

Employability • Applicant recruitment and 

screening

• Academic and professional skills 

instruction

• College coursework

• Internships

• Career support and mentoring

• Stipends

• Connecting participants to 

employment and education 

opportunities

• College campuses 

and internship sites

• Urban

• Northeast

No
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

B-4 

Interventions 

by focus area

(Intervention 

name)

Priority 

objective Core elements

Implementation 

setting

AmeriCorps 

members 

referenced 

in evidence 

document? 

Economic 

opportunity 6

(Single Stop) 

Financial literacy • Financial counseling 

• Tax preparation

• Benefits screening and counseling 

• Legal counseling 

• Immigration consultations

• College campuses

• Urban

• Northeast

No

Economic 

opportunity 7 

(Genesys 

Works) 

Employability • Summer training program for 

workplace technical and 

communication skills

• Paid internships

• Ongoing college and career 

coaching and support 

• Alumni program

• Training centers and 

internship sites

• Urban

• West

No

Economic 

opportunity 8 

(Year Up) 

Employability • Learning communities

• Advising and mentoring

• Social worker support

• Stipends

• Training for workplace technical

and communication skills

• Paid internships

• Program offices and 

internship sites

• Urban

• Northeast, Midwest, 

South, and West

No

Economic 

opportunity 9 

(Habitat for 

Humanity) 

Housing • Build or repair safe and affordable 

homes

• Housing build sites

• Urban

• South and West

Yes

Education 1

(Blue Engine) 

K–12 success • Tutoring or supplemental 

instruction

• High schools

• Urban

• Northeast

No

Education 2

(Citizen 

Schools) 

K–12 success • Apprenticeships 

• High school application support

• College preparation information

• Leadership skills instruction

• Alumni program

• After-school in middle 

and high schools

• Urban

• Northeast

No

Education 3

(Experience 

Corps)

K–12 success • Literacy tutoring • Elementary schools

• Urban and rural

• Northeast and South

No

Education 4

(Jeffco 

Summer of 

Early Literacy 

Program) 

K–12 success • Literacy instruction

• Professional support for teachers

• Parenting events

• Summer camp in 

elementary schools

• Urban

• West

No

Education 5

(Minnesota 

Reading 

Corps)

K–12 success 

and school 

readiness

• Whole-class literacy enrichment 

activities

• One-on-one or small-group 

literacy tutoring

• Elementary schools, 

early childhood 

centers, and 

community-based 

institutions

• Urban and rural

• Northeast and 

Midwest

Yes
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

B-5 

Interventions 

by focus area

(Intervention 

name)

Priority 

objective Core elements

Implementation 

setting

AmeriCorps 

members 

referenced 

in evidence 

document? 

Education 6

(Playworks) 

K–12 success • Organized recess activities

• Class game time

• Youth leadership program

• After-school activities

• Elementary schools

• Urban

• Geographic area 

described as 

nationally or across 

multiple regions

No

Education 7

(Reading 

Partners) 

K–12 success • One-on-one tutoring 

• Dedicated school space and use 

of materials

• Structured and individualized 

curriculum 

• Student assessments for datadriven instruction

• Rigorous and ongoing training for 

teachers

• Instructional supervision

• Elementary schools

• Urban and rural

• Northeast, South, and 

West

Yes

Education 8

(SEED School 

of 

Washington, 

DC) 

K–12 success • Academic program with individual 

interventions as needed

• Student life program 

• College counseling

• Student support services

• Boarding school for 

middle and high 

school

• Urban

• Northeast

No

Education 9

(SPARK 

Literacy 

Program) 

K–12 success • Individual literacy tutoring

• Family engagement/ 

communication

• Elementary schools 

and parents’ homes

• Urban

• Midwest

No

Education 10

(Teach for 

America) 

K–12 success • Preservice training for teachers

• Training and professional 

development while teaching

• Residential program 

on university 

campuses and 

schools

• Urban

• Geographic area 

described as 

nationally or across 

multiple regions

No

Education 11

(College 

Possible) 

Postsecondary 

education support

• Scholastic Assessment Test 

(SAT) and American College 

Testing (ACT) preparation 

• College admissions 

consulting/support

• Financial aid consulting/support

• High schools

• Urban

• Midwest

Yes

Education 12

(Home 

Instruction for 

Parents of 

Preschool 

Youngsters) 

School readiness • Instructional home visits

• Parent/child literacy activities

• Parent monthly group meetings 

and instruction

• Parents’ homes

• Urban

• West

No

Education 13

(Jumpstart) 

School readiness • Whole-class and small-group 

language and literacy activities

• Individual language and literacy 

activities

• Family involvement/ 

communication

• Volunteer engagement in projects

• Preschools and early 

childhood centers

• Urban

• Northeast and West

Yes
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

B-6 

Interventions 

by focus area

(Intervention 

name)

Priority 

objective Core elements

Implementation 

setting

AmeriCorps 

members 

referenced 

in evidence 

document? 

Education 14

(Bridge 

Project) 

K–12 success • Evidence-based reading 

intervention program 

• Small-group instruction 

• One-on-one tutoring 

• Provision of reading materials to 

students

• Elementary schools

• Urban

• South and West

No

Education 15

(Youth Build) 

Postsecondary 

education support

• Structured curriculum 

• College counseling, planning 

activities, and transition supports 

• Financial aid planning and 

assistance 

• Partnerships with local 

postsecondary institutions

• College campuses 

and local community 

locations

• Urban

• Northeast, Midwest, 

South, and West

No

Education 16

(Afford 

Program) 

Postsecondary 

education support

• Financial aid planning sessions 

and workshops

• Free Application for Federal 

Student Aid (FAFSA)/Dream Act 

certification

• College award letter review 

sessions

• Summer outreach to students

• High schools

• Urban

• West

Yes

Education 17 

(Opportunity 

Works 

Program) 

Postsecondary 

education support

• Accelerated instruction to help 

achieve high school 

diploma/equivalency 

• Personalized guidance and 

mentoring to promote high school 

completion

• Dual enrollment in credit-bearing 

courses

• College and career-ready skills 

development

• Guidance on transitioning to 

college

• Community-based 

organizations, 

community colleges, 

dropout 

reengagement 

centers, and high 

schools

• Urban

• Northeast, South, and 

West 

No

Education 18 

(Success 

Boston 

Coaching) 

Postsecondary 

education support

• Coaching on life skills, study 

skills, help-seeking skills, and 

academic skills

• Job and career mentoring 

• College campuses

and virtual settings

• Urban

• Northeast

Yes

Healthy 

futures 1

(Birth and 

Beyond Home 

Visitation 

Program)

Supportive family 

environments

• Parenting instruction • Parents’ homes

• Urban

• West

Yes

Healthy 

futures 2

(SafeCare 

Home-based 

Services)

Supportive family 

environments

• Behavioral skills training in 

caregiving, household 

management, and parenting skills

• Parents’ homes

• Urban and rural

• Geographic region 

not described

No
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

B-7 

Interventions 

by focus area

(Intervention 

name)

Priority 

objective Core elements

Implementation 

setting

AmeriCorps 

members 

referenced 

in evidence 

document? 

Healthy 

futures 3

(Soccer for 

Success) 

Childhood obesity • Physical activity (soccer skills 

learning and practice)

• Nutrition instruction

• Youth development activities

• Schools, public parks, 

and community 

centers 

• Urban

• Northeast, Midwest, 

South, and West

No

Healthy 

futures 4 

(Access to 

School) 

Supportive family 

environments

• Parenting education

• Parent and child interactive 

learning activities

• Comprehensive care

management for families

• Adult English as a Second 

Language instruction

• Preschools

• Urban

• Midwest

No

Healthy 

futures 5 

(Triple-P 

Parenting 

Program) 

Supportive family 

environments

• Media and information about 

positive parenting

• Individual consultation and 

seminars

• Brief consultations with 

information and active skills 

training

• Individual active skills training 

program 

• Augmented individual active skills 

training program

• Community centers in 

direct contact with 

parents and families 

(schools, health 

centers, social 

services)

• Rural

• South

No

Source: Evidence documents submitted by grantees to AmeriCorps. 

Note: The table shows the 32 interventions funded by the AmeriCorps State and National 2015–2019 and SIF 

2010–2015 programs with evidence of effectiveness, as determined by the standards discussed in Chapter 

I. One grantee proposed to implement Healthy Futures 2 and Healthy Futures 4 together, with separate 

evidence documents submitted for each.
This page has been left blank for double-sided copying
APPENDIX C:

METHODS AND TABLES FOR THE META-ANALYSIS
This page has been left blank for double-sided copying. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

C-3 

In this appendix, we provide details of the approach for conducting the study’s meta-analysis. 

We describe the meta-analytic approach for constructing the effect size metrics used for the 

analysis in Chapter III of the report (Section A), assessing the magnitude of effects for individual 

interventions and by the characteristics of interest (Section B), and conducting analyses on the 

sensitivity of our results to the details of the research design (Section C). 

A. Calculating the effect size indices

The interventions in our analysis had three different focus areas—education, economic 

opportunity, and healthy futures—and the studies that assessed their effectiveness used a variety 

of outcome measures (for example, income levels, achievement test scores, and child 

maltreatment rates). The challenge in evaluating such diverse interventions lies in determining a 

method to synthesize a study’s results in a way that allows us to assess its overall effectiveness 

and compare its effectiveness with that of other interventions. The effect size metric helps 

resolve this challenge by serving as a standard measure of the extent to which individuals in the 

treatment and comparison group differ from each other on study outcomes. This standardization 

allows us to compare the relative impacts of interventions that target increased math achievement 

scores for middle school students to those that focus on improving employment outcomes for 

adults (for example). 

The first step needed to prepare for such an analysis is to collect information from each study on 

the effect size or to use results presented in the study to calculate it. Effect sizes can be 

calculated in two ways, depending on how the outcomes are measured. 

1. Hedges’ g. When outcomes are captured as continuous scales, effect sizes are calculated 

using Hedges’ g (Hedges 1982), which takes the Cohen’s d effect size and adjusts it to 

incorporate small sample sizes. Accordingly, the effect size (d) is calculated, first, as the 

difference between the treatment group outcome mean and the comparison group outcome 

mean divided by the pooled standard deviation (the amount of variation on the outcome that 

exists across both groups in the study). The formula for conducting this calculation is 

where x is the mean, n the sample size, and s the post-intervention standard deviation for the 

treatment (T) and comparison (C) samples. If standard deviations were not reported, we 

estimated the effect sizes using other information, such as t-statistics and p-values. Next, the 

Hedges’ g correction (Hedges 1982) is applied by multiplying the effect size index (d) by a 

factor of  , which is calculated as (1 – 3/[4N – 9]), where N is a combined sample size of

treatment and comparison groups. Whenever possible, the effect size was calculated using 

study outcomes that had been adjusted for other data that had been collected (for example, 

pre-test scores or background characteristics of the study participants). For example, we used 

achievement post-test scores that adjusted for participants’ achievement pre-test scores when 

calculating the effect size. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

C-4 

2. The Cox index. When outcomes are captured as a dichotomous measure, the difference in 

group averages is calculated as the difference in the probability of the occurrence of an event. 

The effect size measure we used for dichotomous outcomes is the Cox’s index, which yields 

effect size values similar to the values of Hedges’ g: 

where  is the correction for small sample sizes;  an   are the probability of 

outcome for sample members in the treatment and comparison groups, respectively; and the 

effect size is given by  When program impacts were reported in an odds ratio,

percentages, or proportions, we converted them to the Cox’s index using well-documented 

formulas (What Works Clearinghouse 2020, p. 23). 

Overall, we included or calculated at least one effect size for 30 of the 32 interventions that were 

assessed to work, with calculations including 1,408 effect sizes.25 A total of 419 of these 

outcomes were classified as “primary outcomes” because they corresponded to an evaluation’s 

stated research questions (that is, the main focus of the study). Outcomes that aligned to other 

evaluation research questions (such as exploratory research questions) were considered to be 

secondary outcomes. 

B. Assessing magnitude of effects for individual interventions and their 

characteristics

We used SPSS macros generated by Lipsey and Wilson (2001) to estimate the average effect 

sizes and their associated standard errors and confidence intervals (that is, how precise the 

average effect size estimate is). We carried out these analyses using the following methods:

• We created a weighted average effect size for each intervention. Because studies analyzed 

multiple outcome differences for each intervention, we wanted the outcome differences with 

the most precision to count more toward the intervention’s average effect size than outcome 

differences with greater variation. This is known as weighting each effect size by the inverse 

of the variance. The precision (variance) is primarily a function of the size of the sample on 

which the effect size is based.

• We included a random effect size component in the weighted average effect size calculation. 

The random effects model allows the effect sizes to vary, a feature that is consistent with the 

diverse set of interventions analyzed (Borenstein et al. 2009).26

25 The information on healthy futures and economic opportunity interventions, Health5 and Econ9, were insufficient 

to calculate effect sizes for this analysis. The study reported effect sizes based on gain scores, which are not 

comparable to those based on means. We also did not have enough information to calculate Hedges’ g or the Cox 

index for 161 outcomes across all of the interventions. 

26 Fixed effects models assume one true effect size across the interventions, which seems unlikely given the 

diversity in interventions. In addition, the homogeneity of the effect size statistic (Q) was rejected across all 

comparisons, which indicated a significant amount of heterogeneity in effect sizes and precluded use of the fixed 

(continued) 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

C-5 

• Many studies reported multiple effect sizes on the same participant samples (for example, 

two measures of reading achievement), and it was not feasible to assume independence of the 

effect size estimates. To control for dependencies, we used the synthetic effect size approach 

(Lipsey and Wilson 2001). That is, we first aggregated a dependent set of effect sizes into a 

single independent average effect size. We then combined that estimate with other 

independent effect sizes reported for the intervention to create the overall average effect size 

for the intervention. 

We used the effect sizes for two types of analyses:

• Estimating an intervention’s overall effect size. Because AmeriCorps is interested in 

identifying interventions that work across the various participant outcomes they target, effect 

sizes for all outcomes were combined to generate a single weighted average effect size for 

the intervention. We synthesized effect sizes within each study to create an overall 

assessment of the effectiveness of each intervention and tested each for statistical 

significance (see Figure III.1 and Appendix Table C.1). This analysis told us the percent 

chance that the intervention had an overall impact on participants’ outcomes, with average 

effect sizes supported by a p-value of less than 0.05 demonstrating evidence that the 

interventions work. 

• Comparing effect sizes across interventions. Because AmeriCorps is also interested in how 

the interventions they fund are successfully addressing the goals set in the agency’s strategic 

plan, we combined and compared an intervention’s effectiveness to interventions with 

different characteristics (focus area, priority objectives, community characteristics, 

involvement of AmeriCorps members, and funding year). For example, we combined the 

average of the effect sizes for all interventions in the education focus area and compared the 

result to the average effect size for all interventions in the economic opportunity focus area 

(see Figures III.2, III.3, and III.4 in Chapter III and Appendix Tables C.1, C.2, and C.3). This 

analysis told us two things: (1) whether specific groups of interventions had an overall, 

statistically significant impact on participants’ outcomes and (2) whether the overall impacts 

from one group had a greater impact on participant outcomes than those of another group. 

Appendix Tables C.1, C.2, and C.3 provide detailed information on the results presented in 

Figures III.1 to III.4 in Chapter III. The terms and acronyms included in these tables (and 

Appendix Table C.4) are as follows: 

• Average effect size is the measure of the average magnitude of the difference between the 

treatment group and the comparison group on the study outcomes. The effect size shows the 

magnitude of the impact (or the difference between the treatment and comparison group) 

relative to the standard deviation of the measure. A positive average effect size value 

indicates that the outcomes for the treatment group exceeded those for the comparison group. 

effects model. In general, random effects models are more conservative than the fixed effects model because they 

result in wider confidence intervals (Borenstein et al. 2009).
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

C-6 

• Standard error measures how precisely or accurately our sample of average effect size 

estimates represents the population. The smaller the standard error, the more precisely 

estimated the average effect size.

• p-values represent the probability that the observed finding was obtained by chance when 

there is actually no true impact of the intervention on participants’ outcomes. A p-value less 

than 0.05 indicates that the average effect size (that is, the average magnitude of the 

difference on the study outcomes between the treatment group and the comparison group) is 

statistically significant.

• -95% confidence interval (CI) and +95% CI show the lower (-95% CI) and upper (+95% 

CI) bound of the range within which we can be 95 percent certain that the true average effect 

size lies.

• The p-value of difference (Qb) represents whether the differences between the average 

effect sizes of intervention characteristic categories (for example, yes or no for AmeriCorps 

members delivering intervention services or supporting the delivery) were statistically 

significant from each other. 

C. Sensitivity analyses

Effect sizes can vary with characteristics of the study, particularly the size of the sample and the 

design of the research (Lipsey 1992; Slavin and Smith 2009). We therefore assessed whether 

differences in the estimated effect sizes varied along these lines. 

• Study sample size. Research has consistently demonstrated that studies with smaller sample 

sizes tend to have much larger positive effects than those with larger sample sizes (Slavin 

and Smith 2009). In our study, the median sample size used in the intervention group was 

293 subjects. Therefore, we categorized intervention evaluations that had sample sizes of 293 

or less as small and those with sample sizes greater than 293 as large. Our analysis showed 

that evaluations with smaller sample sizes had an average effect size of 0.41, whereas 

evaluations with larger sample sizes had an average effect size of 0.20—a statistically 

significant difference. Therefore, our finding indicates that effect sizes produced by 

interventions evaluated with smaller samples were greater than those generated by 

interventions evaluated with larger samples.

• Study research design. Interventions were categorized into two types based on how the 

studies created the treatment and comparison groups for the evaluation: nonrandom (quasiexperimental designs were used in 13 evaluations) and random (experimental designs were 

used in 17 evaluations). The average effect size was 0.27 for evaluations using random 

assignment and 0.30 for evaluations using nonrandom assignment. The difference between 

the average effect sizes was not significant. This finding suggests that research designs used 

to evaluate the interventions we identified did not vary in their ability to detect impacts on 

participant outcomes.

The average effect size for each intervention can also vary based on the type and number of 

outcomes used to calculate it. Our main analyses presented in Chapter III used average effect 

sizes based upon primary outcome measures for each intervention. As shown in Table C.4, we 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

C-7 

replicated our analysis of each intervention by calculating average effect sizes that included all 

outcomes (primary and secondary). In general, the inclusion of secondary outcome measures 

when constructing the average effect sizes did not change the pattern of the results. The average 

effect sizes and their statistical significance were relatively similar at the overall, focus area, 

priority objective, and individual intervention levels to those presented in the main analysis that 

include only primary outcomes. There was an exception with one intervention (Education 2); its 

average effect size was statistically significant when only primary outcomes were included but 

was nonsignificant when based on primary and secondary outcomes. 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

C-8 

Table C.1. Intervention effects: Overall, by focus area, priority objective, and intervention;

primary outcomes

 Average 

effect size

Standard 

error p-value -95% CI +95% CI

Overall (all interventions) 0.28 0.03 0.00 0.23 0.33

Economic opportunity 

interventions

0.36 0.05 0.00 0.26 0.45

Employability 0.39 0.05 0.00 0.30 0.49

Economic opportunity 1 0.03 0.16 0.83 -0.27 0.34 

Economic opportunity 2 0.32 0.06 0.00 0.20 0.45

Economic opportunity 3 0.70 0.09 0.00 0.53 0.86 

Economic opportunity 5 0.20 0.18 0.28 -0.16 0.56 

Economic opportunity 7 0.50 0.18 0.006 0.14 0.86

Economic opportunity 8 0.16 0.15 0.28 -0.14 0.46 

Financial literacy 0.15 0.13 0.25 -0.10 0.40 

Economic opportunity 4 0.08 0.16 0.62 -0.23 0.38 

Economic opportunity 6 0.23 0.17 0.18 -0.10 0.55 

Education interventions 0.24 0.04 0.00 0.16 0.31

K–12 success 0.23 0.05 0.00 0.14 0.33

Education 1 0.37 0.18 0.04 0.01 0.72 

Education 2 0.43 0.18 0.02 0.08 0.77 

Education 3 0.10 0.16 0.55 -0.22 0.41 

Education 4 0.10 0.16 0.53 -0.21 0.41 

Education 5 0.50 0.10 0.00 0.32 0.68

Education 6 0.10 0.15 0.68 -0.20 0.40 

Education 7 0.12 0.11 0.27 -0.10 0.34

Education 8 0.15 0.13 0.27 -0.11 0.40 

Education 9 0.05 0.18 0.77 -0.30 0.40 

Education 10 0.08 0.15 0.59 -0.21 0.37 

Education 14 0.17 0.29 0.56 -0.40 0.73 

Postsecondary education 

support

0.19 0.08 0.01 0.05 0.34 

Education 11 0.21 0.20 0.30 -0.18 0.59

Education 15 0.21 0.11 0.07 -0.02 0.43 

Education 16 0.02 0.17 0.89 -0.30 0.35

Education 17 0.37 0.18 0.04 0.02 0.72

Education 18 0.17 0.15 0.25 -0.12 0.47

School readiness 0.37 0.12 0.002 0.14 0.61 

Education 12 0.37 0.23 0.10 -0.07 0.81

Education 13 0.39 0.13 0.002 0.14 0.64

Healthy futures interventions 0.22 0.10 0.02 0.04 0.41

Supportive family environments 0.22 0.11 0.049 0.00 0.43

Healthy futures 1 0.14 0.16 0.35 -0.16 0.45 

Healthy futures 2 0.11 0.15 0.46 -0.18 0.41 
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

C-9 

 Average 

effect size

Standard 

error p-value -95% CI +95% CI

Healthy futures 4 0.58 0.24 0.01 0.12 1.05 

Childhood obesity 0.24 0.18 0.17 -0.10 0.59

Healthy futures 3 0.24 0.16 0.12 -0.06 0.55

Note: There were 419 primary outcomes across 30 interventions: 46 unique samples contributed effect sizes to 

intervention effects. Each row represents an intervention categorized under a specific focus area and 

priority objective. 

Table C.2. Intervention effects: Overall, by intervention and evaluation characteristics;

primary outcomes

 

Average 

effect 

size 

Standard 

error p-value –95% CI +95% CI

p-value of 

difference 

(Qb)

Multiple geographies 

No (n = 16) 0.25 0.04 0.00 0.16 0.33 0.23

Yes (n = 14) 0.32 0.04 0.00 0.24 0.40 

AmeriCorps members delivering intervention services or supporting the delivery 

No (n = 22) 0.29 0.04 0.00 0.22 0.36 0.70 

Yes (n = 8) 0.27 0.05 0.00 0.16 0.37 

Note: In parentheses, we note the number of interventions that correspond to each intervention characteristic. The 

p-value of difference (Qb) indicates whether the difference across categories is statistically significant. 

Table C.3. Intervention effects: Overall, by funding year and source; primary outcomes

 Average 

effect size

Standard 

error

p-value -95% CI +95% CI

Overall (all interventions) 0.28 0.03 0.00 0.23 0.33

SIF

Year 2010 (n = 8) 0.33 0.05 0.00 0.24 0.42

Year 2011 (n = 5) 0.22 0.09 0.02 0.04 0.40

Year 2012 (n = 5) 0.22 0.09 0.01 0.05 0.40

Year 2014 (n = 2) 0.26 0.14 0.06 -0.01 0.53

AmeriCorps State and National

Year 2015 (n = 6) 0.28 0.06 0.00 0.15 0.40

Year 2016 (n = 9) 0.30 0.06 0.00 0.19 0.41

Year 2017 (n = 4) 0.11 0.10 0.26 -0.08 0.30

Year 2018 (n = 3) 0.34 0.08 0.00 0.18 0.50

Year 2019 (n = 9) 0.26 0.05 0.00 0.16 0.37

Note: In parentheses, we note the number of interventions that correspond to each year. SIF = Social Innovation 

Fund. For Year 2013, n = 0. Because a specific intervention could have been funded across multiple years, 

we did not conduct statistical tests that compare whether interventions in one funding year had a greater 

impact on participant outcomes than interventions funded in a different funding year.
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

C-10

Table C.4. Intervention effects: Overall, by focus area, priority objective, and intervention; 

primary and secondary outcomes

 

Average 

effect size

Standard 

error p-value –95% CI +95% CI

Overall (all interventions) 0.24 0.03 0.00 0.19 0.29

Economic opportunity 

interventions 0.28 0.04 0.00 0.20 0.36

Employability 0.30 0.04 0.00 0.22 0.39

Economic opportunity 1 0.00 0.16 1.00 -0.31 0.31

Economic opportunity 2 0.34 0.06 0.00 0.21 0.46

Economic opportunity 3 0.37 0.09 0.00 0.20 0.53

Economic opportunity 5 0.21 0.22 0.34 -0.22 0.65

Economic opportunity 7 0.50 0.18 0.006 0.14 0.86

Economic opportunity 8 0.11 0.15 0.47 -0.19 0.41

Financial literacy 0.11 0.11 0.29 -0.10 0.33

Economic opportunity 4 0.03 0.16 0.85 -0.27 0.33

Economic opportunity 6 0.20 0.16 0.20 -0.11 0.51

Education interventions 0.22 0.04 0.00 0.15 0.29

K–12 success 0.21 0.04 0.00 0.13 0.30

Education 1 0.37 0.18 0.04 0.02 0.72

Education 2 0.27 0.18 0.14 -0.08 0.62

Education 3 0.10 0.16 0.55 -0.22 0.41

Education 4 0.10 0.16 0.53 -0.21 0.41

Education 5 0.50 0.09 0.00 0.32 0.68

Education 6 0.10 0.15 0.49 -0.19 0.40

Education 7 0.09 0.11 0.39 -0.12 0.31

Education 8 0.12 0.13 0.35 -0.14 0.39

Education 9 0.05 0.18 0.77 -0.29 0.40

Education 10 0.08 0.15 0.59 -0.21 0.37

Education 14 0.17 0.29 0.56 -0.40 0.73

Postsecondary education 

support 0.17 0.07 0.01 0.04 0.30

Education 11 0.23 0.20 0.24 -0.15 0.61

Education 15 0.15 0.11 0.17 -0.07 0.38

Education 16 0.06 0.16 0.73 -0.27 0.38

Education 17 0.29 0.18 0.10 -0.06 0.64

Education 18 0.17 0.15 0.25 -0.12 0.47

School readiness 0.39 0.11 0.00 0.17 0.60

Education 12 0.37 0.23 0.10 -0.07 0.81

Education 13 0.39 0.13 0.00 0.14 0.64

Healthy futures interventions 0.22 0.09 0.01 0.05 0.39

Supportive family environments 0.21 0.10 0.03 0.02 0.39

Healthy futures 1 0.14 0.15 0.35 -0.16 0.45
Evidence of Effectiveness in AmeriCorps-Funded Interventions Mathematica

C-11

 

Average 

effect size

Standard 

error p-value –95% CI +95% CI

Healthy futures 2 0.11 0.15 0.45 -0.18 0.41

Healthy futures 4 0.58 0.24 0.01 0.12 1.05

Childhood obesity 0.24 0.15 0.11 -0.06 0.54

Healthy futures 3 0.24 0.16 0.12 -0.06 0.55

Note: There were 1,408 primary and secondary outcomes across 30 interventions: 46 unique samples 

contributed effect sizes to intervention effects. Each row represents an intervention categorized under a 

specific focus area and priority objectives. 
This page has been left blank for double-sided copying. 
This page has been left blank for double-sided copying.
Mathematica

Princeton, NJ • Ann Arbor, MI • Cambridge, MA 

Chicago, IL • Oakland, CA • Seattle, WA

Tucson, AZ • Woodlawn, MD • Washington, DC 

EDI Global, a Mathematica Company

Bukoba, Tanzania • High Wycombe, United Kingdom

mathematica.org