Best Practices in Demonstrating Evidence

AmeriCorps State and National & CNCS Office of Research and Evaluation | November 2019
Session Outline

• CNCS' approach to evidence

• NOFO evidence base criteria

• Evidence tiers

• Evidence quality

• Tips for applicants

• Q&A
Why is Evidence Important?

• Achieve a shared goal of offering communities 

effective solutions that address their needs 

• Ensure that federal dollars are invested wisely

• Inform continuous improvement of programs

– Change what isn’t working

– Do more of what is working
Evidence in AmeriCorps

Grant Applications

• Proposed service activities (interventions) must 

be supported by evidence

– Threshold requirement for funding

• Evidence-informed: 

– Uses available knowledge, research, and evaluation to 

guide program design/implementation

– Specific intervention described in application has not 

been rigorously evaluated

• Evidence-based:

– Intervention described in application has been 

rigorously evaluated and demonstrated positive results
Building Evidence of 

Effectiveness

Evidence 

Informed

Evidence 

Based

Stage 1:

Identify a 

promising

program design

Stage 2:

Ensure effective 

implementation

Stage 3:

Assess program 

outcomes

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

www.nationalservice.gov/resources/evaluation/evidence-continuum
Same Intervention

• Evidence-based applicants must provide 

evidence for the same intervention described in 

the application

• “Same intervention” means all of these areas 

must match:

– Characteristics of beneficiary population 

– Characteristics of population delivering intervention 

– Dosage (frequency, duration) and design of 

intervention, including all key components

– Setting in which intervention is delivered 

– Outcomes of intervention 

6
Clarifying Program Design

• To assess the evidence supporting an 

application, core components of the intervention 

must be clearly described in both the application 

and the submitted evaluation report(s):

– Characteristics of the beneficiary population 

– Characteristics of population delivering the intervention

– Dosage (frequency, duration) and design of the 

intervention, including all key components and activities

– Context in which the intervention is delivered

– Outcomes of the intervention

7
2020 NOFO:

Evidence Base Criteria

• Evidence Tier (12 points)

– Based on: 

• the relative strength of each applicant’s evidence base

• the likelihood that the proposed intervention will lead to 

outcomes identified in the logic model 

• Evidence Quality (8 points)

– Based on:

• The quality of the applicant’s evidence

• The extent to which the evidence supports the proposed 

program design

– Exact evidence quality criteria depend on evidence tier
2020 Evidence Tiers

• Pre-Preliminary

• Preliminary

• Moderate

• Strong

Defined in 2020 Mandatory Supplemental 

Guidance: 

https://www.nationalservice.gov/sites/default/files/d

ocuments/2020%20MSG%20FINAL.508.pdf
Evidence Tiers: Percent of

Funded Applicants, by Year

44

38

9 9

46

40

5

9

50

26

11 13

0

10

20

30

40

50

60

Pre-preliminary

and below

Preliminary Moderate Strong

2017

2018

2019
Evidence Tiers: Pre-preliminary 

• Applicant has not submitted any outcome/impact 

evaluations of the intervention described in the 

application.

• Applicant must describe in the application 

narrative how program design is evidence- informed. 

• Applicants may also cite prior performance 

measure data.

*Threshold requirement: applicants in this 

evidence tier must have adequate evidence quality 

in order to be considered for funding
Pre-Preliminary 

Evidence Tier Example 

Applicant’s Ready to Read program provides smallgroup tutoring services to 5th-grade students for 30 

minutes, twice a week. The program is adapted 

from Famous Tutoring Program’s successful 

approach, which used the same curriculum to 

provide one-on-one tutoring sessions for 30 

minutes every day. A 2017 randomized control trial 

found that students in the Famous Tutoring 

Program increased their scores on standardized 

tests by 40% more than the control group. 

Additional documents: none
Evidence Tiers: Preliminary

• Applicant has submitted outcome evaluation 

report(s) (can be internal or independent 

evaluations) 

– Comparison group may be present, but is not 

randomly assigned or statistically matched

• Reports evaluated the same intervention 

described in the application

• Reports show positive results on one or more 

key desired outcomes in the applicant’s logic 

model
Preliminary 

Evidence Tier Example 

Applicant’s Ready to Read program provides smallgroup tutoring services to 5th-grade students for 30 

minutes, twice a week. Based on pre- and postassessments administered by the Ready to Read

program in 2017, 350 students gained at least 1.5 

grade levels in reading mastery. The effect sizes were 

significant and represent a positive result.

Additional Documents: The applicant submitted one 

internal evaluation report of the Ready to Read 

program describing the results of the pre-post 

assessment.
Evidence Tiers: Moderate

• Applicant has submitted impact evaluation report(s) 

(must be independent evaluations)

– Experimental (RCT) or quasi-experimental (QED) study 

designs

– Ability to generalize the findings beyond the study context 

may be limited (e.g., single-site)

• Reports evaluated the same intervention described 

in the application

• Reports show positive results on one or more key 

desired outcomes in the applicant’s logic model
Moderate 

Evidence Tier Example 

The applicant’s Ready to Read program uses the same 

curriculum, program design, and dosage as the Famous 

Tutoring Program and is serving similar students. Based on a 

2017 quasi-experimental evaluation conducted by Famous 

Tutoring Program at one of their program sites, students 

gained on average 1.3 grade levels on the Famous 

Standardized Literacy Assessment, compared to just 0.8 grade 

levels for the comparison group. The study was conducted by 

an independent (external) evaluator. The results were 

significant (p < 0.05).

Additional Documents: The applicant submitted one 

independent evaluation report from the Famous Tutoring 

Program describing the results of the QED study.
Evidence Tiers: Strong

• Applicant has submitted impact evaluation report(s) 

(must be independent evaluations)

– Experimental (RCT) or quasi-experimental (QED) study designs

– Results are attributable to the intervention and can be 

generalized beyond the study context (e.g., multi-site 

evaluation or multiple evaluations from different sites/populations)

• Reports evaluated the same intervention described in 

the application

• Reports show consistently positive results on one or 

more key desired outcomes in the applicant’s logic model
Strong

Evidence Tier Example

Applicant’s Ready to Read program provides tutoring services 

in 25 states across the country. The program hired an 

independent evaluator to conduct a randomized controlled trial 

in 16 states, including both rural and urban sites as well as 

student populations with different ethnic/racial backgrounds. 

The evaluation found that students in the Ready to Read 

program outperformed students in the control group on 3 

specific literacy skills addressed by the program. The results 

were statistically significant with Moderate effect sizes. 

Subgroup analysis showed positive impacts in both rural and 

urban settings and across multiple ethnic/racial groups. 

Additional Documents: The applicant submitted one 

independent evaluation report from the Ready to Read 

program describing the results of the RCT study.
Submitting Evaluation 

Reports/Studies

• Who should submit reports or studies?

– In order to be considered for Preliminary, Moderate or 

Strong evidence  Submit up to 2 reports/studies (or 3 if 

evaluation report required)

– Any applicant required to submit an evaluation report to 

meet evaluation requirements  Submit evaluation report

• Required evaluation report can be submitted in 

addition to the 2 other studies (up to 3 total)

• Reports or studies submitted for evidence tier 

MUST be of the same intervention proposed in the 

application
Evidence Quality Criteria:

Pre-Preliminary Evidence Tier

• The applicant uses relevant evidence, including 

past performance measure data and/or cited 

research studies, to inform their proposed 

program design

• The described evidence is relatively recent, 

preferably from the last six years

• The evidence described by the applicant 

indicates a meaningful positive effect on 

program beneficiaries in at least one key 

outcome of interest

*Based on content of application narrative*

20
Evidence Quality Criteria: 

Preliminary/Moderate/Strong 

Evidence Tier

• The submitted reports are of satisfactory

methodological quality and rigor for the type of 

evaluation conducted (e.g., adequate sample size 

and statistical power, internal and/or external validity, 

appropriate use of control or comparison groups, 

etc.)

• The submitted reports describe evaluations that were 

conducted relatively recently, preferably within the 

last six years

• The submitted reports show a meaningful and 

significant positive effect on program beneficiaries 

in at least one key outcome of interest

*Based on submitted reports/studies* 21
Data Collection Plans and 

Evaluation Plans

Additional evaluation-related items that must be 

submitted in the grant application:

• Not reviewed until after funding decisions are 

made

• Evaluation plans must be approved by CNCS 

within the first year of the grant

Type of Plan Who Submits

Data Collection Plan New Applicants

Evaluation Plan Recompeting Applicants
Tips for Applicants

• Read the NOFO and Mandatory Supplemental Guidance 

carefully

• In the application narrative, describe the full body of evidence that 

exists for your program:

– Summarize the study design and key findings from any submitted 

reports

– Describe other supporting evidence, for example, past performance 

measure data or other research

• Describe how the intervention in the submitted report(s) is the 

same as the intervention proposed in the application

• Do not submit more than the allowable number of studies (either 2 or 3 depending on applicant’s evaluation requirements)

• Select high quality evidence: rigorous, relevant, recent, meaningful
Resources

• AmeriCorps State and National FY2020 Notice of 

Funding Opportunity

– https://www.nationalservice.gov/build-yourcapacity/grants/funding-opportunities/2020/americorpsstate-and-national-grants-fy-2020

• Evaluation Resources on the Knowledge Network:

– http://www.nationalservice.gov/resources/evaluation

• CNCS Evidence Exchange

– https://www.nationalservice.gov/impact-ournation/evidence-exchange