Best Practices in Demonstrating Evidence

Megha Patel, PhD

Research Analyst

Office of Research and Evaluation 
Session Outline

AmeriCorps' approach to evidence

NOFO evidence base criteria

Evidence tiers

Evidence quality

Tips for applicants

Resources

Questions
Why is Evidence Important?

Achieve a shared goal of offering communities effective solutions that address their 

needs 

Ensure that federal dollars are invested wisely

Inform continuous improvement of programs

• Change what isn’t working

• Do more of what is working
Evidence in AmeriCorps Grant Applications

Proposed service activities (interventions) must be supported by evidence

Evidence-informed: 

• Uses available knowledge, research, and evaluation to guide program 

design/implementation

• Specific intervention described in application has not been rigorously evaluated

Evidence-based:

• Intervention described in application has been rigorously evaluated and 

demonstrated positive results
Stage 1:

Identify a 

promising

program 

design

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

Stage 3:

Assess 

program 

outcomes Stage 2:

Ensure effective 

implementation

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Building Evidence of 

Effectiveness

Evidence 

Informed

Evidence 

Based

Building Evidence of Effectiveness
Same Intervention

Evidence-based applicants must provide evidence for the same intervention described 

in the application

“Same intervention” means all of these areas must match:

• Characteristics of beneficiary population 

• Characteristics of population delivering intervention 

• Dosage (frequency, duration) and design of intervention, including all key 

components

• Setting in which intervention is delivered 

• Outcomes of intervention 

6
Clarifying Program Design

To assess the evidence supporting an application, core components of the 

intervention must be clearly described in both the application and the submitted 

evaluation report(s):

• Characteristics of the beneficiary population 

• Characteristics of population delivering the intervention

• Dosage (frequency, duration) and design of the intervention, including all key 

components and activities

• Context in which the intervention is delivered

• Outcomes of the intervention

7
NOFO Evidence Base Criteria

Evidence Tier (12 points)

Based on: 

• the relative strength of each applicant’s evidence base

• the likelihood that the proposed intervention will lead to outcomes identified in the 

logic model 

Evidence Quality (8 points)

Based on:

• The quality of the applicant’s evidence

• The extent to which the evidence supports the proposed program design

• Exact evidence quality criteria depend on evidence tier
Evidence Tiers

Pre-Preliminary

Preliminary

Moderate

Strong

Defined in Mandatory Supplemental Information
Evidence Tiers: 

Percent of Awarded Applicants, by Year
Evidence Tiers: Pre-preliminary 

Applicant has not submitted any outcome/impact evaluations of the intervention 

described in the application

• Applicant must describe in the application narrative how program design is 

evidence-informed

• Applicants may also cite prior performance measure data
Pre-Preliminary Evidence Tier Example 

Applicant’s Ready to Read program provides small-group tutoring services to 5thgrade students for 30 minutes, twice a week. The program is adapted from Famous 

Tutoring Program’s successful approach, which used the same curriculum to provide 

one-on-one tutoring sessions for 30 minutes every day. A randomized control trial 

conducted last year found that students in the Famous Tutoring Program increased 

their scores on standardized tests by 40% more than the control group. 

Additional documents: none
Evidence Tiers: Preliminary

Applicant has submitted outcome evaluation report(s) (can be internal or 

independent evaluations) 

• Comparison group may be present, but is not randomly assigned or statistically 

matched

Reports evaluated the same intervention described in the application

Reports show positive results on one or more key desired outcomes in the 

applicant’s logic model
Preliminary Evidence Tier Example 

Applicant’s Ready to Read program provides small-group tutoring services to 5thgrade students for 30 minutes, twice a week. Based on pre- and postassessments administered by the Ready to Read program last year, 350 students 

gained at least 1.5 grade levels in reading mastery. The effect sizes were 

moderate and represent a positive result.

Additional Documents: The applicant submitted one internal evaluation report of 

the Ready to Read program describing the results of the pre-post assessment.
Evidence Tiers: Moderate

Applicant has submitted impact evaluation report(s) (must be independent 

evaluations)

• Experimental (RCT) or quasi-experimental (QED) study designs

• Ability to generalize the findings beyond the study context may be limited (e.g., 

single-site)

Reports evaluated the same intervention described in the application

Reports show positive results on one or more key desired outcomes in the 

applicant’s logic model
Moderate Evidence Tier Example 

The applicant’s Ready to Read program uses the same curriculum, program design, 

and dosage as the Famous Tutoring Program and is serving similar students. Based on 

a quasi-experimental evaluation conducted by Famous Tutoring Program at one of 

their program sites, students gained on average 1.3 grade levels on the Famous 

Standardized Literacy Assessment, compared to just 0.8 grade levels for the 

comparison group. The study was conducted by an independent (external) 

evaluator. The results were significant (p < 0.05).

Additional Documents: The applicant submitted one independent evaluation report 

from the Famous Tutoring Program describing the results of the QED study. The 

evaluation was published two years ago.
Evidence Tiers: Strong

Applicant has submitted impact evaluation report(s) (must be independent 

evaluations)

• Experimental (RCT) or quasi-experimental (QED) study designs

• Results are attributable to the intervention and can be generalized beyond the study 

context (e.g., multi-site evaluation or multiple evaluations from different 

sites/populations)

Reports evaluated the same intervention described in the application

Reports show consistently positive results on one or more key desired outcomes in the 

applicant’s logic model
Strong Evidence Tier Example

Applicant’s Ready to Read program provides tutoring services in 25 states across the 

country. The program hired an independent evaluator to conduct a randomized 

controlled trial in 16 states, including both rural and urban sites as well as student 

populations with different ethnic/racial backgrounds. The evaluation found that 

students in the Ready to Read program outperformed students in the control group on 

3 specific literacy skills addressed by the program. The results were statistically 

significant with Moderate effect sizes. Subgroup analysis showed positive impacts in 

both rural and urban settings and across multiple ethnic/racial groups. 

Additional Documents: The applicant submitted one independent evaluation report 

from the Ready to Read program describing the results of the RCT study, which was 

published three years ago.
Submitting Evaluation Reports/Studies

Who should submit reports or studies?

• Any applicant wishing to be considered for Preliminary, Moderate or Strong 

evidence  Submit up to 2 reports/studies (or 3 if evaluation report required)

• Any applicant required to submit an evaluation report to meet evaluation 

requirements  Submit evaluation report

Required evaluation report can be submitted in addition to the 2 other studies 

(up to 3 total)

Reports or studies submitted for evidence base consideration MUST be of the 

same intervention proposed in the application
Submitting Additional Documents

If multiple evaluation briefs/reports/studies are submitted, the most recent (as measured by 

the date of completion or publication) will be reviewed first, and any documents submitted 

beyond the allowable number will not be reviewed. 

Submit Evaluation Reports in Microsoft Word. Include 

• A title page with the AmeriCorps grant number for the project that was evaluated

• The name of the project

• The date of completion of the report

*If any of this required information is missing, the applicant may not receive credit for 

meeting their evaluation requirements.*
Evidence Quality Criteria: 

Pre-Preliminary Evidence Tier

The applicant uses relevant evidence, including past performance measure data 

and/or cited research studies, to inform their proposed program design

The described evidence is relatively recent, preferably from the last six years

The evidence described by the applicant indicates a meaningful positive effect on 

program beneficiaries or AmeriCorps members in at least one key outcome of 

interest

*Based on content of application narrative*

*Review the NOFO for any footnotes pertaining to Evidence Quality criteria, as 

evidence thresholds differ slightly for each competition*

21
Evidence Quality Criteria: 

Preliminary/Moderate/Strong Evidence Tier

The submitted reports are of satisfactory methodological quality and rigor for the 

type of evaluation conducted (e.g., adequate sample size and statistical power, 

internal and/or external validity, appropriate use of control or comparison groups, 

etc.)

The submitted reports describe evaluations that were conducted relatively recently, 

preferably within the last six years

The submitted reports show a meaningful and significant positive effect on program 

beneficiaries or AmeriCorps members in at least one key outcome of interest

*Based on submitted reports/studies*

22
Evaluation Plans

Additional evaluation-related item that 

must be submitted with the grant 

application:

• Not reviewed until after funding decisions 

are made

• Evaluation plans must be approved by 

AmeriCorps within the first year of the 

grant

Item Who Submits

Evaluation Plan Recompeting Applicants

https://americorps.gov/fundingopportunity/fy-2024-americorps-statenational-grants
Tips for Applicants

Read the NOFO and Mandatory Supplemental Information carefully

In the application narrative, describe the full body of evidence that exists for your program:

• Summarize the study design and key findings from any submitted reports

• Describe other supporting evidence, for example, past performance measure data or 

other research

Describe how the intervention in the submitted report(s) is the same as the one proposed in 

the application

Do not submit more than the allowable number of studies (either 2 or 3 depending on 

applicant’s evaluation requirements)

Select high quality evidence: rigorous, relevant, recent, meaningful
Resources

Notice of Funding Opportunity

Mandatory Supplemental Information

Evaluation Resources on AmeriCorps website

• Evaluation Core Curriculum

• Best Practices in Writing an Evaluation Plan

• Best Practices in Evaluation Reporting

AmeriCorps Evidence Exchange

Email AmericorpsGrants@cns.gov with any questions about evidence base criteria 

for competitive grant applications
Resources

Institute of Education Sciences’ National Center for Education Evaluation and 

Regional Assistance What Works Clearinghouse 

Department of Health and Human Services Pathways to Work Evidence 

Clearinghouse 

Department of Labor’s Clearinghouse for Labor Evaluation and Research (CLEAR) 

Email AmericorpsGrants@cns.gov with any questions about evidence base criteria 

for competitive grant applications
Questions

27