**Sample Evaluation Plan Checklist**

Grantees may use this checklist to help guide the development of their evaluation plan. Each of the following sections aligns with the Evaluation Plan Template developed by AmeriCorps State and National (ASN). As you develop your evaluation plan, you can mark off each item to confirm that each of the sections have been fully described. As a note, this checklist serves as a basic guide, so even if all elements are addressed, this does not guarantee that a grantee will obtain approval.

**Theory of Change**

· Statement of the problem

· Narrative account of the program’s planned work (Resources, Activities) and desired results (Outputs, Outcomes)

· What activities do your AmeriCorps members perform?

· How will those activities help the program achieve its desired outcomes?

· Who is the population you are serving?

· Does program description align with your program Logic Model?

· *Optional:*Research evidence supporting the program’s Theory of Change (e.g., prior research on this program model or similar programs)

· *Optional:*Summary of previous evaluations of the program to be evaluated (if any)

**Scope of the Evaluation**

· Purpose and scope of current evaluation

· Which program components or activities described in your Theory of Change will be assessed in the evaluation?

· *Note:* AmeriCorps does not require grantees to evaluate all components of their program; evaluation may focus on a sub-set of program activities

**Evaluation Outcome(s) of Interest**

· For a process evaluation, describe the goals or outputs of the study (e.g., program fidelity).

· Do these align with the program Logic Model and Theory of Change?

· For a non-experimental outcome or impact evaluation (i.e., QED or RCT), describe the outcomes the evaluation will assess.

· Do these align with the program Logic Model and Theory of Change?

· Do the outcomes measure a change in knowledge, attitude, behavior or condition?

· Are these outcomes feasible to measure, based on the source(s) of data needed and level of effort required?

· Will these outcomes be observable within the study period?

****

**Research Questions**

· List of the research question(s) that the evaluation will answer.

· Is each research question:

· Clearly stated and specific?

· Aligned with the program’s Theory of Change and Logic Model?

· Aligned with the selected outcomes of interest?

· Able to produce measurable or observable results?

· Realistic to answer given the resources, time, and experience of the evaluation team?

· Built on results from prior research and evaluations?

· Phrased in accordance with the research design (e.g., reference a control/comparison group for impact designs)

**Evaluation Design**

· Type of evaluation study design(s) clearly stated and fully described:

· Process evaluation;

· Non-experimental outcome evaluation;

· Quasi-experimental evaluation (QED); and/or

· Randomized Control Trial (RCT)

· The design choice aligns with the goals for the evaluation. 

· Description of evaluation study group(s)

· If proposing an impact evaluation (QED or RCT), study groups include a comparison or control group?

**Control or Comparison Group Formation**

· For a QED, consider the following:

· Approach for identifying a pool of similar individuals, organizations, or locations

· Procedures for identifying a matched comparison group from the pool of similar cases (e.g., propensity score matching)

· 

· List of variables (covariates) used for matching the treatment and comparison groups.

· Baseline measure of the outcome(s) or proxy measure(s) (e.g., GPA to estimate future high school graduation)

· For a RCT, consider the following:

· Eligibility criteria for inclusion in the study

· Randomization process to form the treatment and control study groups

· *Optional:* For a non-experimental outcome evaluation, if using a comparison group, how will study participants be selected? Why is this selection appropriate? 

****

**Sample Selection**

· Description of the population from which the sample will be drawn

· 

· Details for selecting the treatment and/or comparison or control group, if applicable

· Sampling procedures**and selection criteria.

· How will the sample be drawn (e.g., simple random, stratified random, purposive, convenience sampling, census of the population)

· *Note:* Provide a numeric estimate, even if data are collected from the full population

· Will there be a consent process?

· Data use agreements?

**Sample Size Justification**

· For a non-experimental outcome evaluation:

· What is the basis for selecting the sample size?

· Is the sample size reasonable for answering the research question(s)?**

· For a QED or RCT, consider the following:

· Was a power analysis calculated and do the findings support the proposed sample size?

· If conducting a subgroup analysis, is the sample size sufficient?

**Data Collection Methods**

· What information will be collected?

· Who/what will be the source of data? 

· How will the information be collected? (data collection tools/instruments)

· How frequently will data be collected (e.g., pre- and post-intervention)?

· When will data be collected and by whom?

· Will AmeriCorps members or staff need to be trained to conduct data collection? If so, when and how?

· Is a description of each data source provided?

· Do the data sources adequately address each of the research questions?

**Analysis Plan**

· How will data be analyzed and is the proposed analysis appropriate for the evaluation’s design and data sources?

· Quantitative data analysis techniques (e.g., descriptive or inferential statistics)

· Qualitative data analysis techniques (e.g., examine data for patterns using triangulation, use qualitative coding software such as NVivo or Excel)

· Does the analysis plan address all of the study’s research question?

· For impact evaluations, is a statistical method proposed for comparing outcomes between study groups (e.g., multiple regression or ANOVA analysis)?

****

**Evaluator Qualifications**

· If using an internal evaluator, provide a description of which program staff will be responsible for specific components of the evaluation and their experience with data analysis and/or evaluation

· If using an external evaluator, provide a description of the evaluator qualifications needed to conduct the evaluation.

· Does the evaluator have experience evaluating similar programs?

· Does the evaluator have experience with similar evaluation designs?

**Timeline**

· The following components are included in the timeline:

· Hire/identify evaluator

· Finalize evaluation plan and data collection tools/instruments (including IRB clearance, if applicable)

· Recruit study participants

· Data collection(s) (baseline, post-test, follow-up periods)

· Analysis

· Report writing (including drafts, if applicable)

· Complete evaluation, including final report, for submission by the next grant application submission date 

· Activities are broken out by evaluation month and year

**Budget**

· Estimate costs based on consultant/evaluator fees, staffing, materials and supplies, equipment, and travel for each major evaluation component

**

**Other Questions to Consider**

*Does the plan provide sufficient detail to determine if the planned evaluation will provide the information needed to address the proposed research questions?*

*For small grantees (i.e., receive less than $500,000 per grant year): Does the plan present an evaluation that will meets the AmeriCorps’ requirements for “small” grantees?*

*For large grantees (i.e., receive $500,000 or more per grant year): Does the plan present an evaluation that will meet the requirements for “large” grantees?*

**

*If a large grantee may be unable to meet the AmeriCorps requirements for an impact evaluation, the grantee may consider submitting an**Alternative Evaluation Approach (AEA) request**for any of the following reasons:*

· Funding Threshold: Grantees may request this option if the total awarded federal share of their budget averages under $1 million per year for the full life of the grant. An external evaluator is NOT required for completing a new evaluation. 

· Previous Impact Evaluation: Grantees that conducted a previous impact evaluation of the same program that achieved a Moderate or Strong evidence rating may qualify for this exemption. An external evaluator would still be required for completing a new evaluation. 

· AmeriCorps National Evaluation: This option may be requested by grantees that are participating in an AmeriCorps evaluation (i.e., bundled evaluation or Return on Investment) that will not be completed during the current grant cycle and for which findings will not be available until a later date. Grantees may also request this option if the design of the AmeriCorps evaluation does not fulfill the evaluation requirements for a large grantee. 

· Program Structure: This option may be requested if a grantee can demonstrate insurmountable challenges to forming a comparison group, or it is not developmentally appropriate to conduct an impact evaluation due to significant changes to the program design. An external evaluator would still be required for completing a new evaluation.

· Replication: Grantees may request this option if they are implementing an evidence-based intervention (i.e., rigorously evaluated through an impact study, and the study obtained a Moderate or Strong evidence rating) with fidelity in a new setting. An external evaluator would still be required for completing a new evaluation.

*More details on these options can be located here:*https://americorps.gov/sites/default/files/document/Alternative_Evaluation_Approach_2023-06_Final_508.pdf*.*

1