Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 1

Best Practices in Demonstrating Evidence

Megha Patel, PhD

Research Analyst

Office of Research and Evaluation 

Hi everyone. Welcome to this webinar on Best Practices in Demonstrating Evidence. I’m Megha Patel, a 

research analyst in AmeriCorps’ Office of Research and Evaluation, and I’ll be leading this presentation.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 2

Session Outline

AmeriCorps' approach to evidence

NOFO evidence base criteria

Evidence tiers

Evidence quality

Tips for applicants

Resources

Questions

In this presentation, I’ll start with AmeriCorps’ approach to evidence, then talk about the evidence base 

criteria in the AmeriCorps State and National Notice of Funding Opportunity, or NOFO. I’ll share specific 

details about the evidence tiers and the evidence quality standards. Then I’ll close with a few tips and 

resources for applicants.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 3

Why is Evidence Important?

Achieve a shared goal of offering communities effective solutions that address their 

needs 

Ensure that federal dollars are invested wisely

Inform continuous improvement of programs

• Change what isn’t working

• Do more of what is working

Why is evidence important? Why do we care at AmeriCorps? The reason why evidence is important to 

AmeriCorps State and National can be summed up in just a few words: we want to be confident that 

what we’re funding can work. In other words, it’s important to us that the activities supported by 

AmeriCorps grants are effective in addressing community needs. And this is a shared goal, because we 

know it’s important to you, too, to make a meaningful difference in solving problems within your 

communities. Evidence is the essential link between what you, as the applicant, are proposing to do and 

the outcomes you intend to achieve. Think of evidence as support for your program. When there is solid 

evidence supporting your program model, it gives AmeriCorps confidence that we can invest federal 

dollars and member resources in your project, and that those dollars will be used wisely to make a real 

difference in your community.

Evidence should also be a tool for your program as it grows and matures, to figure out how to make 

continuous improvements to increase the effectiveness of what you do. Even in the most carefully 

designed program model, there are inevitably aspects that are more effective and aspects that are less 

effective. Evidence can help you distinguish between those things and do MORE of what works, and 

LESS of what doesn’t work as well. At AmeriCorps, our grantmaking efforts support the process of 

evidence-building and want your program to learn from this evidence-building feedback loop. Again, 

evidence is support for your program, but also think of it as an opportunity to learn so you can better 

serve your community. 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 4

Evidence in AmeriCorps Grant Applications

Proposed service activities (interventions) must be supported by evidence

Evidence-informed: 

• Uses available knowledge, research, and evaluation to guide program 

design/implementation

• Specific intervention described in application has not been rigorously evaluated

Evidence-based:

• Intervention described in application has been rigorously evaluated and 

demonstrated positive results

Because of the value and importance of evidence, it plays a significant role in AmeriCorps State and 

National grant competitions. All activities supported by AmeriCorps grants – which we also refer to as 

“interventions” -- must be supported by evidence in some way. By intervention, we mean a set of 

activities or practices that you might engage in to serve your communities and bring about change. 

These can be activities around health improvement, employment, or cultural and heritage preservation. 

When we say supported by evidence, this means that your program can be evidence-informed or 

evidence-based. 

The opening section of the NOFO states that “AmeriCorps grants are awarded to eligible organizations 

proposing to engage AmeriCorps members in evidence-based or evidence-informed 

interventions/practices to strengthen communities.” Evidence-informed means that the applicant is 

basing their program design on available knowledge, research, and evaluation from related projects. The 

exact proposed activities themselves might not have been evaluated yet, but evidence from similar 

activities has guided the design of the program. For example, you might have read about and been 

inspired by a program and want to implement it in your own community to address a particular need. 

However, maybe some adaptation to the original intervention is needed to best suit the needs of your 

community. If you use the original program as a model for your program, you can consider your 

program evidence-informed. Evidence-based means that the exact intervention proposed in the 

application has already been rigorously evaluated (I’ll talk more in a moment about what we mean by 

“rigorously”), and that the evaluation shows that the intervention is effective in achieving the outcomes 

the applicant wants to achieve. In other words, your program is evidence-based if the intervention is the 

same and we know that the program is effective at achieving its intended outcomes. 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 5

Stage 1:

Identify a 

promising

program 

design

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

Stage 3:

Assess 

program 

outcomes Stage 2:

Ensure effective 

implementation

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Building Evidence of 

Effectiveness

Evidence 

Informed

Evidence 

Based

Building Evidence of Effectiveness

It’s important to remember that at AmeriCorps, we’re focused on the evidence-building process. We 

don’t expect all applicants to have programs that have been rigorously evaluated from the outset. We 

encourage you to view evidence-building as a continuum, moving from an evidence-informed program 

to an eventually evidence-based program. 

The journey from “evidence-informed” to “evidence based” takes place in stages. 

• The first stage is to identify a promising program design that, as I mentioned, is based on available 

knowledge, research, and evaluation from related projects. We encourage you to explore 

AmeriCorps’ Evidence Exchange to find AmeriCorps-funded programs that work. Other federal 

clearinghouses exist as well, such as What Works Clearinghouse for education-related interventions. 

Another option is to explore academic research on effective interventions using Google Scholar, 

which is a free resource. You may also want to talk to other community organizations implementing 

similar programs to yours. At this stage, we encourage you to begin the evidence-building process by 

exchanging and exploring knowledge and information. 

• The second stage is to make sure this program design is implemented effectively and consistently. A 

process evaluation is a good tool to help you figure out whether or not this is being done well. The 

term “process evaluation” may seem intimidating, but just think of it as studying if your program is 

working the way it’s supposed to. 

• The third stage is to assess program outcomes. You should do this by building high-quality data 

collection protocols that are supported by good instruments (such as surveys, tests, interview 

questions, etc.) and are able to measure outcomes in a valid and consistent way. You’re essentially 

likely doing this already because it’s part of the performance measurement requirements for all 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

AmeriCorps programs. A recorded webinar on Best Practices in Developing Performance Measures is 

available on the AmeriCorps NOFO webpage.

• The fourth and fifth stages of this evidence-building process are to use evaluation to look at whether 

or not you are achieving what you set out to do. You might start out with a non-experimental 

outcome evaluation (stage 4) that gives you a first look at whether your program is achieving what 

you hoped. If you see positive results at this stage, you can move to a type of evaluation that can 

assess causality: in other words, it can examine whether or not your program activities are actually 

causing the outcomes you are seeing. This requires experimental or quasi-experimental evaluation 

designs, which we collectively refer to as impact evaluations. Once an impact evaluation has shown 

meaningful and significant positive results for program outcomes (stage 5) your program can be 

considered evidence-based.

The evidence-building process may seem like a lot but it will all be worth it in the long run. And at 

AmeriCorps, we recognize that it takes time to build evidence, and that depending on where you are in 

your program life cycle, you may not be ready to conduct the type of impact evaluation that will qualify 

you to be considered evidence-based. It is also common for programs to intentionally loop back on this 

cycle as part of continuous learning and improvement: for example, outcome data or the results from an 

initial evaluation may show that the program needs to be modified in order to be more effective. Those 

changes may be significant enough that the program will need to be re-evaluated, which would put an

applicant back at stage 1 or 2. That is OK! There is value in ALL stages of the continuum, and AmeriCorps 

funds grantees at every stage. Always remember that evidence-building is a learning process; any 

negative findings or set-backs could be viewed as learning opportunities that can be used for program 

improvement and eventually strengthening your evidence-base. 

You can leapfrog up to stage 5 if you want to start with a program design that we know already works. 

If you identify an existing evidence-based program design and replicate it, meaning that you implement 

it exactly the way it was designed, you will be considered an evidence-based program too. Be careful, 

though - this is not as easy as it sounds! It takes a lot of work to ensure genuine fidelity to a program 

model. A process evaluation is an essential tool to make sure you are doing it right.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 6

Same Intervention

Evidence-based applicants must provide evidence for the same intervention described 

in the application

“Same intervention” means all of these areas must match:

• Characteristics of beneficiary population 

• Characteristics of population delivering intervention 

• Dosage (frequency, duration) and design of intervention, including all key 

components

• Setting in which intervention is delivered 

• Outcomes of intervention 

6

In order for an applicant to be assessed as evidence-based, the evidence presented by the applicant 

needs to be for the same intervention described in the application. This could mean that it is evidence 

from your own AmeriCorps program, OR, as we just discussed on the last slide, it could mean that it is 

evidence from a different program that you are replicating with fidelity (in other words, doing exactly as 

designed) at your own site. Either way, you need to be able to prove that the intervention described in 

the submitted evidence, and the intervention you are describing in your application, are the same.

This means that the interventions need to match in five characteristics: 

• Characteristics of the beneficiary population: this means that the beneficiaries of your program –

like middle-schoolers or senior citizens – must match those described in the original intervention. If 

your program benefits middle-schoolers, the original intervention also needs to benefit middle 

schoolers. 

• Characteristics of the population delivering the intervention: similar to the point I just made, the 

population delivering the intervention, such as AmeriCorps members or nursing students, need to 

match the population delivering the intervention in the original program. Another point to 

remember is if those delivering the intervention require a specific credential or training, then the 

population delivering your program’s intervention also needs to have the same credentials or 

training. 

• Dosage, or frequency and duration and design of the intervention, including all key components: if 

the original intervention is tutoring middle school children 3 times per week for 1 hour each time, 

then your program model needs to include tutoring middle school children 3 times per week for 1 

hour each time. 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

• The setting in which the intervention is delivered: if the original intervention calls for middle school 

students to be tutored in their home room classroom, then the middle school students in your 

program need to be tutored in their home room classroom. 

• And lastly, the outcomes of the intervention need to be the same. If the original intervention aims to 

improvement literacy, then your program needs to also measure improvement in literacy. 

Why do these dimensions all have to match? It’s to make sure that the evidence you are presenting is 

actually applicable to what you are planning to do. If any of the characteristics on this slide are 

significantly different, it means that the outcomes of your proposed activity can’t be expected to be the 

same as the outcomes shown in the evidence. You could make a reasonable hypothesis that the 

outcomes MIGHT be the same, but the existing evidence doesn’t provide conclusive proof of this. So 

your proposed project would be considered to be evidence-informed, not evidence-based.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 7

Clarifying Program Design

To assess the evidence supporting an application, core components of the 

intervention must be clearly described in both the application and the submitted 

evaluation report(s):

• Characteristics of the beneficiary population 

• Characteristics of population delivering the intervention

• Dosage (frequency, duration) and design of the intervention, including all key 

components and activities

• Context in which the intervention is delivered

• Outcomes of the intervention

7

In order for reviewers to assess whether all of these dimensions match, you need to provide complete 

descriptions of both interventions: the one proposed in the application, and the one that was evaluated 

in the reports or studies you are submitting as evidence. Since the definition of “same intervention” is 

based on five dimensions, you need to describe all five of them clearly. If one or more of these 

dimensions doesn’t sufficiently match, the report won’t count as evidence for the application. 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 8

NOFO Evidence Base Criteria

Evidence Tier (12 points)

Based on: 

• the relative strength of each applicant’s evidence base

• the likelihood that the proposed intervention will lead to outcomes identified in the 

logic model 

Evidence Quality (8 points)

Based on:

• The quality of the applicant’s evidence

• The extent to which the evidence supports the proposed program design

• Exact evidence quality criteria depend on evidence tier

You on the program side have completed the application narrative and presented your evidence. Now 

it’s our turn at AmeriCorps to assess the strength of your evidence. When assessing the strength of the 

evidence supporting an application, AmeriCorps reviewers look at two things: evidence tier and 

evidence quality. Both are described in detail in the NOFO and Mandatory Supplemental Information, 

so these are the guidance documents you should refer to. But here is a quick summary:

Evidence tier refers to the strength, or rigor, of the evidence presented to support the application. This 

is roughly analogous to the different stages of the evidence continuum. The more rigorous the 

evidence, and the higher it would land on the evidence continuum, the more points an application will 

receive for evidence tier. Evidence that supports a cause-effect relationship between the intervention 

and positive program outcomes will receive the most points, because this gives AmeriCorps the highest 

degree of certainty that the federal investment will lead to the desired outcomes.

Evidence rigor needs to be accompanied by evidence quality. There are several dimensions of quality 

that are assessed, and what exactly those dimensions are depends on the applicant’s evidence tier. For 

evidence-informed programs, quality refers to the relevance of the evidence and the extent to which it 

has guided the applicant’s program design. For evidence-based programs, quality refers to how well the 

evaluation of the intervention was designed and implemented. For both types of programs, the 

meaningfulness of the positive results shown by the evidence is an important dimension of evidence 

quality since this supports the value of the program design. The recency of the evidence is also 

important to make sure that it continues to be relevant to the current circumstances.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 9

Evidence Tiers

Pre-Preliminary

Preliminary

Moderate

Strong

Defined in Mandatory Supplemental Information

In the next several slides, we will take a more detailed look at the evidence tiers defined in the 

Mandatory Supplemental Information. Those tiers are:

Pre-Preliminary

Preliminary

Moderate

Strong
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 10

Evidence Tiers: 

Percent of Awarded Applicants, by Year

I mentioned earlier that AmeriCorps funds programs at every stage on the evidence continuum. Just to 

re-emphasize that point, here is a graph showing the evidence tiers for successful new and recompeting 

applicants from 2019-2021. You will notice that the percentage of awarded applicants with Strong and 

Moderate evidence increased significantly during these years, which is consistent with AmeriCorps’ 

commitment to fund what works. However, you’ll also notice that AmeriCorps funds significant 

percentages of applicants in the Pre-preliminary and Preliminary evidence tiers. So if you aren’t yet at a 

stage where you can call yourself an evidence-based program, please don’t let that discourage you from 

applying for funding. AmeriCorps values evidence-informed programs as well, and we also value 

programs that are still on the journey toward becoming evidence-based. As an AmeriCorps grantee, 

you’ll have opportunities to conduct program evaluations that will help grow your evidence. We care 

about what works, but we also care about the evidence-building process, from evidence-informed to 

evidence-based. 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 11

Evidence Tiers: Pre-preliminary 

Applicant has not submitted any outcome/impact evaluations of the intervention 

described in the application

• Applicant must describe in the application narrative how program design is 

evidence-informed

• Applicants may also cite prior performance measure data

Ok. Let’s discuss the evidence tiers. The pre-preliminary evidence tier corresponds to evidence-informed 

program models. These are applicants who don’t have direct evidence for the specific intervention 

described in the application, and so are not able to submit any outcome or impact evaluation reports. 

Instead, the applicant needs to describe how the intervention is guided by available knowledge, 

research, and evaluation from related projects. If previous performance measure data exists for the 

program, this should also be described along with how it has been used to guide the proposed 

intervention.

These descriptions need to be entered in the application narrative itself. If your application is informed 

by evidence but is not actually replicating the intervention described in the evidence, DO NOT submit 

any reports or studies as attachments. Reports that are not of the same intervention described in the 

application will not be reviewed or receive any points.

As stated in the NOFO, all AmeriCorps programs must be either based on or informed by evidence. For 

applicants in the pre-preliminary evidence tier, who don’t yet have evaluation or evidence that supports 

their exact program model, it has to be clear in the application narrative that the program design is 

indeed evidence-informed. This means citing relevant studies or program models you’re familiar with. 

You could even collect evidence from your community using culturally engaged methods and cite that 

data in your application narrative. 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 12

Pre-Preliminary Evidence Tier Example 

Applicant’s Ready to Read program provides small-group tutoring services to 5thgrade students for 30 minutes, twice a week. The program is adapted from Famous 

Tutoring Program’s successful approach, which used the same curriculum to provide 

one-on-one tutoring sessions for 30 minutes every day. A randomized control trial 

conducted last year found that students in the Famous Tutoring Program increased 

their scores on standardized tests by 40% more than the control group. 

Additional documents: none

Here is a (brief) example of a hypothetical applicant that would be assessed as being in the prepreliminary evidence tier. 

Applicant’s Ready to Read program provides small-group tutoring services to 5th-grade students for 30 

minutes, twice a week. The program is adapted from Famous Tutoring Program’s successful approach, 

which used the same curriculum to provide one-on-one tutoring sessions for 30 minutes every day. A 

randomized control trial conducted last year found that students in the Famous Tutoring Program 

increased their scores on standardized tests by 40% more than the control group. 

Additional documents: none

A couple of things to point out here:

• The description says that the program is adapted from the Famous Tutoring Program’s successful 

approach, not replicating their approach. You can see in the description that the type of tutoring 

(small-group) and the frequency (twice a week) is different from the daily one-on-one services 

provided by Famous Tutoring Program. There may be good reasons why the applicant has modified 

the intervention; maybe their school partners have asked them to use a different approach, or maybe 

they are serving a different type of population. However, the changes mean that it is not the same 

intervention, so this is an evidence-informed program, not an evidence based program. 

• Because it is not the same intervention, the applicant did NOT submit the actual randomized 

controlled trial study from Famous Tutoring Program. Instead, they appropriately described it in the 

application narrative. The description makes it clear that the evidence from the Famous Tutoring 

Program informed the applicant’s program design, since they are choosing to use the same 

curriculum. It’s also clear from the description that the evidence is relatively recent (from last year) 

and had a meaningful positive effect on program participants.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 13

Evidence Tiers: Preliminary

Applicant has submitted outcome evaluation report(s) (can be internal or 

independent evaluations) 

• Comparison group may be present, but is not randomly assigned or statistically 

matched

Reports evaluated the same intervention described in the application

Reports show positive results on one or more key desired outcomes in the 

applicant’s logic model

The preliminary evidence tier is for applicants who are progressing toward becoming evidence-based, in 

that they have started evaluating the outcomes of their intervention. Applicants who are assessed as 

being in the Preliminary evidence tier have submitted one or more non-experimental outcome 

evaluation reports to accompany their application. (Non-experimental means that if a comparison 

group was used, it wasn’t randomly assigned or statistically matched, so there isn’t proof that the 

comparison group was equivalent to the treatment group before the intervention.)

To reiterate, the submitted report needs to be of the same intervention described in the application, not 

just an intervention that is kind of similar. And the report also has to show positive results for at least 

one outcome that is part of the applicant’s logic model. A report that shows no effect, or that shows 

positive effects for something unrelated to the applicant’s program design, doesn’t actually provide any 

evidence to support the applicant’s intervention.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 14

Preliminary Evidence Tier Example 

Applicant’s Ready to Read program provides small-group tutoring services to 5thgrade students for 30 minutes, twice a week. Based on pre- and postassessments administered by the Ready to Read program last year, 350 students 

gained at least 1.5 grade levels in reading mastery. The effect sizes were 

moderate and represent a positive result.

Additional Documents: The applicant submitted one internal evaluation report of 

the Ready to Read program describing the results of the pre-post assessment.

Here’s an example of a hypothetical applicant that would be in the Preliminary evidence tier. 

Applicant’s Ready to Read program provides small-group tutoring services to 5th-grade students for 30 

minutes, twice a week. Based on pre- and post- assessments administered by the Ready to Read 

program last year, 350 students gained at least 1.5 grade levels in reading mastery. The effect sizes were 

moderate and represent a positive result.

Additional Documents: The applicant submitted one internal evaluation report of the Ready to Read 

program describing the results of the pre-post assessment.

This applicant submitted an outcome evaluation report from their own program, which means it is the 

same intervention. The report doesn’t have a matched comparison or control group, so it is a nonexperimental evaluation, but it shows positive results on the key outcome of reading mastery. This 

qualifies the applicant for the Preliminary evidence tier. 

A couple of other things the applicant did well in this example:

- They not only submitted the evidence report, but they also described the evidence in the application 

narrative. Describing the full body of evidence, including any submitted reports or studies, in the 

Evidence Base section of the application is explicitly required by the NOFO for all evidence tiers, and 

is important to help reviewers fully assess the applicant’s evidence.

- The report was from last year, which means it was recent.

- The applicant shared information about effect size to demonstrate that the results shown in the 

evaluation report were meaningful.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

One note of caution is that it’s not possible to tell from this description how well the evaluation design 

was actually implemented, or whether the effect size determination was methodologically correct. 

These dimensions of evidence quality would be assessed by looking at the text of the evaluation report 

itself. That’s unfortunately outside the scope of what I can do in a webinar example.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 15

Evidence Tiers: Moderate

Applicant has submitted impact evaluation report(s) (must be independent 

evaluations)

• Experimental (RCT) or quasi-experimental (QED) study designs

• Ability to generalize the findings beyond the study context may be limited (e.g., 

single-site)

Reports evaluated the same intervention described in the application

Reports show positive results on one or more key desired outcomes in the 

applicant’s logic model

The Moderate evidence tier is the first evidence tier that applies to evidence-based applicants. In order 

to qualify for the Moderate evidence tier, applicants have to submit at least one independent (meaning 

externally-conducted) impact evaluation. An impact evaluation, as defined in Mandatory Supplemental 

Information, provides statistical evidence of how well a program achieves its desired outcomes 

compared to what would have happened in the absence of the program. In other words, this type of 

evaluation is designed to show a causal relationship between activities and outcomes. The two types of 

evaluation design that are able to show causality are randomized controlled trial (RCT) and quasiexperimental design (QED) evaluations.

The difference between the Moderate evidence tier and its sister tier, Strong evidence, relates to how 

broadly applicable, or generalizable, the findings are. An applicant with Moderate evidence may have 

conducted an RCT or QED on just a single site, or with one particular population. This means that the 

findings may or may not be applicable in other locations or with other populations. This is different than 

the Strong evidence tier, which you will see shortly.

Just like in the Preliminary evidence tier, it’s essential that the submitted reports are of the same 

intervention described in the application. It’s also essential that the reports show positive results for 

one or more key outcomes in the applicant’s logic model. If either of these two things does not apply to 

a particular report, that report will not contribute to the applicant’s assigned evidence tier.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 16

Moderate Evidence Tier Example 

The applicant’s Ready to Read program uses the same curriculum, program design, 

and dosage as the Famous Tutoring Program and is serving similar students. Based on 

a quasi-experimental evaluation conducted by Famous Tutoring Program at one of 

their program sites, students gained on average 1.3 grade levels on the Famous 

Standardized Literacy Assessment, compared to just 0.8 grade levels for the 

comparison group. The study was conducted by an independent (external) 

evaluator. The results were significant (p < 0.05).

Additional Documents: The applicant submitted one independent evaluation report 

from the Famous Tutoring Program describing the results of the QED study. The 

evaluation was published two years ago.

Here is an example of a hypothetical applicant who would be assessed in the Moderate evidence tier.

The applicant’s Ready to Read program uses the same curriculum, program design, and dosage as the 

Famous Tutoring Program and is serving similar students. Based on a quasi-experimental evaluation 

conducted by Famous Tutoring Program at one of their program sites, students gained on average 1.3 

grade levels on the Famous Standardized Literacy Assessment, compared to just 0.8 grade levels for the 

comparison group. The study was conducted by an independent (external) evaluator. The results were 

significant (p < 0.05).

Additional Documents: The applicant submitted one independent evaluation report from the Famous 

Tutoring Program describing the results of the QED study. The evaluation was published two years ago.

Notice that this applicant didn’t submit an evaluation report from their own program, but rather from 

someone else’s program instead. However, it’s clear from this description that the applicant is 

replicating the Famous Tutoring Program – i.e., implementing it in exactly the same way it was designed 

and evaluated. So the submitted evaluation report is for the same intervention and will be considered in 

determining the evidence tier. This is different from the adaptation we saw in the Pre-Preliminary 

example, where key elements of the intervention had been modified and so it no longer met the “same 

intervention” definition. 

The description here indicates that the report is a recent QED evaluation (i.e., an impact evaluation) that 

showed meaningful and statistically significant positive effects on literacy. However, it was from a single 

program site, which means it may not be broadly generalizable to other sites. This is why this applicant 

would be assessed in the Moderate evidence tier, not the Strong evidence tier. As with the Preliminary 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

evidence tier example, the quality of the evidence will need to be assessed by looking at the text of the 

evaluation report itself.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 17

Evidence Tiers: Strong

Applicant has submitted impact evaluation report(s) (must be independent 

evaluations)

• Experimental (RCT) or quasi-experimental (QED) study designs

• Results are attributable to the intervention and can be generalized beyond the study 

context (e.g., multi-site evaluation or multiple evaluations from different 

sites/populations)

Reports evaluated the same intervention described in the application

Reports show consistently positive results on one or more key desired outcomes in the 

applicant’s logic model

The Strong evidence tier is the most rigorous tier; it includes applicants that are evidence-based and 

whose evidence is broadly applicable in a variety of settings: in other words, it is generalizable beyond

the study context. Applicants reach this level of evidence by conducting independent QED or RCT 

evaluations that are multi-site, or by conducting multiple independent impact evaluations at different 

sites or with different populations. The results need to be consistently positive across all these sites or 

with all these populations. AND, of course, the reports need to be of the same intervention described in 

the application.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 18

Strong Evidence Tier Example

Applicant’s Ready to Read program provides tutoring services in 25 states across the 

country. The program hired an independent evaluator to conduct a randomized 

controlled trial in 16 states, including both rural and urban sites as well as student 

populations with different ethnic/racial backgrounds. The evaluation found that 

students in the Ready to Read program outperformed students in the control group on 

3 specific literacy skills addressed by the program. The results were statistically 

significant with Moderate effect sizes. Subgroup analysis showed positive impacts in 

both rural and urban settings and across multiple ethnic/racial groups. 

Additional Documents: The applicant submitted one independent evaluation report 

from the Ready to Read program describing the results of the RCT study, which was 

published three years ago.

Here is an example of a hypothetical applicant who would be assessed in the Strong evidence tier.

Applicant’s Ready to Read program provides tutoring services in 25 states across the country. The 

program hired an independent evaluator to conduct a randomized controlled trial in 16 states, including 

both rural and urban sites as well as student populations with different ethnic/racial backgrounds. The 

evaluation found that students in the Ready to Read program outperformed students in the control 

group on 3 specific literacy skills addressed by the program. The results were statistically significant with 

Moderate effect sizes. Subgroup analysis showed positive impacts in both rural and urban settings and 

across multiple ethnic/racial groups. 

Additional Documents: The applicant submitted one independent evaluation report from the Ready to 

Read program describing the results of the RCT study, which was published three years ago.

Even though the applicant only submitted a single evaluation report, the report covers multiple sites 

with multiple population variations, and the positive impacts were seen in each of them. This is why the 

applicant is assessed as having Strong evidence rather than Moderate evidence. Evidence quality will be 

assessed based on the details of the submitted evaluation report.

The submitted report is from the applicant’s own project in this example, but it wouldn’t have to be –

the applicant could also have submitted an evaluation report from a program model they are replicating 

with fidelity.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 19

Submitting Evaluation Reports/Studies

Who should submit reports or studies?

• Any applicant wishing to be considered for Preliminary, Moderate or Strong 

evidence → Submit up to 2 reports/studies (or 3 if evaluation report required)

• Any applicant required to submit an evaluation report to meet evaluation 

requirements → Submit evaluation report

Required evaluation report can be submitted in addition to the 2 other studies 

(up to 3 total)

Reports or studies submitted for evidence base consideration MUST be of the 

same intervention proposed in the application

As you’ve seen as we discussed the evidence tiers, applicants must submit reports or studies with their 

application in order to be considered for anything above the Pre-Preliminary evidence tier. All 

applicants are allowed to submit up to 2 studies or evaluation reports. Applicants who need to submit 

an evaluation report to meet their evaluation requirements are allowed to submit up to 3 studies: in 

other words, their required evaluation report can be submitted in addition to the usual 2.

At this point, I probably sound like a broken record when it comes to “same intervention.” But it really 

does bear repeating yet again. Submitted reports or studies will only be considered if the intervention 

in the study matches the intervention proposed in the grant application. Please carefully read the 

definition of “same intervention” in the Mandatory Supplemental Information, and be sure to describe 

in your application how the definition is met in your specific case.

Details about the mechanics of how to submit evidence reports or studies to AmeriCorps are in the 

NOFO and Application Instructions. Please be sure to follow these instructions, and the required 

timeline, carefully.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 20

Submitting Additional Documents

If multiple evaluation briefs/reports/studies are submitted, the most recent (as measured by 

the date of completion or publication) will be reviewed first, and any documents submitted 

beyond the allowable number will not be reviewed. 

Submit Evaluation Reports in Microsoft Word. Include 

• A title page with the AmeriCorps grant number for the project that was evaluated

• The name of the project

• The date of completion of the report

*If any of this required information is missing, the applicant may not receive credit for 

meeting their evaluation requirements.*

After all your hard work in compiling evidence for your application, please make sure that you are 

submitting it correctly. 

Remember, that only the allowable number of evidence documents will be reviewed, which is 2 or 3 

depending on your evaluation requirements. The most recent documents will be reviewed first, and any 

documents submitted beyond that will NOT be reviewed. 

If you are submitting an evaluation report, please submit it in Microsoft Word and include the following 

information. A title page with the AmeriCorps grant number for the project that was evaluated. The 

name of the project, and the date of completion of the report. If any of this information is missing, your 

organization might not receive credit for meeting the evaluation requirements. 

Please check the NOFO for additional guidance on how to submit additional documents. 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 21

Evidence Quality Criteria: 

Pre-Preliminary Evidence Tier

The applicant uses relevant evidence, including past performance measure data 

and/or cited research studies, to inform their proposed program design

The described evidence is relatively recent, preferably from the last six years

The evidence described by the applicant indicates a meaningful positive effect on 

program beneficiaries in at least one key outcome of interest

*Based on content of application narrative*

*Review the NOFO for any footnotes pertaining to Evidence Quality criteria, as 

evidence thresholds differ slightly for each competition*

21

In addition to evidence tier, reviewers also assess the evidence quality for AmeriCorps applicants. As I 

mentioned earlier, the evidence quality criteria differ based on the assessed evidence tier. This slide 

shows the evidence quality criteria for applicants who are in the Pre-Preliminary evidence tier. 

Important words in the criteria are highlighted.

The applicant uses relevant evidence, including past performance measure data and/or cited research 

studies, to inform their proposed program design

The described evidence is relatively recent, preferably from the last six years

The evidence described by the applicant indicates a meaningful positive effect on program beneficiaries 

in at least one key outcome of interest

The criteria are tailored to evidence-informed applicants: i.e., programs that don’t yet have evidence to 

support their specific intervention design. These applicants need to demonstrate in the application 

narrative that they are informed by recent, relevant evidence that shows a meaningful positive effect on 

program beneficiaries. (Notice that this is program beneficiaries – not AmeriCorps members.)

As a reminder, please remember to review the NOFO for any footnotes pertaining to Evidence Quality 

criteria, as evidence thresholds differ slightly for each competition. 
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 22

Evidence Quality Criteria: 

Preliminary/Moderate/Strong Evidence Tier

The submitted reports are of satisfactory methodological quality and rigor for the 

type of evaluation conducted (e.g., adequate sample size and statistical power, 

internal and/or external validity, appropriate use of control or comparison groups, 

etc.)

The submitted reports describe evaluations that were conducted relatively recently, 

preferably within the last six years

The submitted reports show a meaningful and significant positive effect on program 

beneficiaries in at least one key outcome of interest

*Based on submitted reports/studies*

22

This slide shows the evidence quality criteria for applicants who are assessed as being in the Preliminary, 

Moderate, or Strong evidence tier. Again, important words are highlighted in blue.

The submitted reports are of satisfactory methodological quality and rigor for the type of evaluation 

conducted (e.g., adequate sample size and statistical power, internal and/or external validity, 

appropriate use of control or comparison groups, etc.)

The submitted reports describe evaluations that were conducted relatively recently, preferably within 

the last six years

The submitted reports show a meaningful and significant positive effect on program beneficiaries in at 

least one key outcome of interest

The criteria are tailored to evidence-based applicants or applicants that are progressing toward being 

evidence-based: in other words, programs that do have evaluation reports or studies that support their 

specific intervention design. These applicants need to demonstrate in the submitted reports that their 

evidence has satisfactory methodology, is relatively recent, and shows a meaningful and significant 

positive effect on program beneficiaries. (Again, this is program beneficiaries, not AmeriCorps 

members.)
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 23

Evaluation Plans

Additional evaluation-related item that 

must be submitted with the grant 

application:

• Not reviewed until after funding decisions 

are made

• Evaluation plans must be approved by 

AmeriCorps within the first year of the 

grant

Item Who Submits

Evaluation Plan Recompeting Applicants

https://americorps.gov/fundingopportunity/fy-2023-americorps-statenational-grants

I want to digress for a moment to talk about another evaluation-related item that is submitted as part of 

an AmeriCorps grant application. Recompeting competitive applicants are required to submit an 

evaluation plan, following the guidance in the NOFO. 

The evaluation plan is NOT part of the competitive application review process and will not be reviewed 

until after funding decisions are made. So it’s important to remember that the information provided in 

the evaluation plan will NOT be considered as part of the applicant’s evidence base and will NOT affect 

the applicant’s evidence tier or quality. All of the scored evidence criteria must be addressed in the 

main portion of the application narrative and in the submitted evidence reports.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 24

Tips for Applicants

Read the NOFO and Mandatory Supplemental Information carefully

In the application narrative, describe the full body of evidence that exists for your program:

• Summarize the study design and key findings from any submitted reports

• Describe other supporting evidence, for example, past performance measure data or 

other research

Describe how the intervention in the submitted report(s) is the same as the one proposed in 

the application

Do not submit more than the allowable number of studies (either 2 or 3 depending on 

applicant’s evaluation requirements)

Select high quality evidence: rigorous, relevant, recent, meaningful

Before we wrap up today, here are a couple of tips for AmeriCorps grant applicants when it comes to 

addressing the evidence criteria. 

First – READ THE INSTRUCTIONS. The definitions of each evidence tier are carefully and intentionally 

written, and so is the language in the evidence quality criteria. And the instructions for when and how 

to submit evaluation reports or studies are essential to make sure you are considered for the correct 

evidence tier. Please follow this written guidance to the letter.

Second – please follow the instructions in the NOFO and describe the full body of evidence for your 

intervention in the application narrative. Do this even if you are submitting reports to be considered for 

a higher evidence tier. If those studies are determined not to qualify as the same intervention, only the 

content of the application narrative will be considered as evidence. 

Third – please describe in the application narrative how the intervention in your submitted reports or 

studies is the same as the intervention proposed in the application. It may be obvious to you, but 

chances are it probably isn’t obvious to a third-party reviewer reading your application. Use the criteria 

in the “same intervention” definition and explain, point by point, how the interventions match each 

other. If you don’t do that, you are vulnerable to having your submitted reports disregarded as not 

being of the same intervention.

Fourth – please be aware of the limits on the number of studies or reports you can submit. Every year 

there is SOME applicant who submits ten studies, and eight of those studies can’t be reviewed or 

considered. Doing that leaves you vulnerable to having your best-quality studies thrown out. Please 

choose your two (or three) best quality studies and submit only those. And if the studies aren’t of the 

same intervention, as defined in the Mandatory Supplemental Information, PLEASE do not submit them.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Finally, consider evidence quality when you decide which two or three studies to submit. A QED or RCT 

study that isn’t properly done, or that doesn’t show a meaningful positive effect, is not going to do you 

any favors. Stick to evidence that fits the bolded words from the evidence quality criteria: rigorous, 

relevant, recent, and meaningful.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 25

Resources

Notice of Funding Opportunity

Mandatory Supplemental Information

Evaluation Resources on AmeriCorps website

• Evaluation Core Curriculum

• Best Practices in Writing an Evaluation Plan

• Best Practices in Evaluation Reporting

AmeriCorps Evidence Exchange

Email AmericorpsGrants@cns.gov with any questions about evidence base criteria 

for competitive grant applications

In closing, here are some resources that will be helpful to you in demonstrating evidence in your grant 

applications:

- All applicants should carefully read every word of the Notice of Funding Opportunity and Mandatory 

Supplemental Information. These documents have detailed definitions for all of the evidence tiers 

and evidence quality criteria, as well as the instructions for submitting evidence reports.

- The Evaluation Resources page on the AmeriCorps website will help you understand different types 

of evaluation designs and how to use evaluation to build the evidence base for your program. The 

Evaluation Core Curriculum will walk you through the evaluation process from planning to 

implementation to using evaluation results to inform program improvement. Two relevant recorded 

webinars, which are also linked from the NOFO page, cover best practices in writing an evaluation 

plan and reporting on an evaluation.

- The AmeriCorps Evidence Exchange is a searchable repository of evaluation reports from other 

National Service programs. If you are interested in adapting or replicating an existing intervention, 

this is a great place to start.
Best Practices in Demonstrating Evidence - AmeriCorps State and National Grants - November 2022

Slide 26

Resources

Institute of Education Sciences’ National Center for Education Evaluation and 

Regional Assistance What Works Clearinghouse 

Department of Health and Human Services Pathways to Work Evidence 

Clearinghouse 

Department of Labor’s Clearinghouse for Labor Evaluation and Research (CLEAR) 

Email AmericorpsGrants@cns.gov with any questions about evidence base criteria 

for competitive grant applications

Other federal research repositories exist to help you find research to support your program. The 

Institute of Education Sciences’ National Center for Education Evaluation and Regional Assistance has a 

What Works Clearinghouse that contains large-scale evaluations of education programs. 

HHS’ Pathways to Work Evidence Clearinghouse offers evaluations of the effectiveness of interventions 

designed to help job seekers succeed in the labor market. 

The Department of Labor’s Clearinghouse for Labor Evaluation and Research houses research on various 

labor topics including apprenticeship and work-based training, job search assistance, and literacy 

interventions designed to increase employment opportunities for individuals. 

Finally, AmeriCorps State and National has a special mailbox for grant applicants who have questions 

about funding opportunities or the grant application process, including the evidence tier and quality 

review criteria. If you have a question about anything that was said during this presentation, feel free to 

email AmericorpsGrants@cns.gov.