NORC at the University of Chicago

How to Write an Evaluation Plan

2021  |

1

1

Understand what an evaluation plan is and the purpose of developing one

Identify key sections of an evaluation plan

Understand what information to include in an evaluation plan

Learning Objectives

2021  |

  2

2

Facilitator notes: For this presentation, we have identified a number of learning objectives.  

By the end of this presentation, you will be able to: 

Understand what an evaluation plan is and the purpose of developing one

Identify key sections of an evaluation plan

Understand what information to include in an evaluation plan

The information contained in this presentation will help you write an evaluation plan for your program, or work with an external evaluator to develop and/or review an evaluation plan. 

Details the program model being evaluated

Describes and justifies the evaluation approach selected

Provides instructions for the evaluation / a guide for each step of the evaluation process

What is an Evaluation Plan?

2021  |

  3

3

Facilitator notes: An evaluation plan does the following:

Details the program model being evaluated. Grantees should review their logic model and theory of change to determine which component or components of their program model they would like to evaluate. It is important to note that grantees do not need to evaluate every component of their program. Rather, you can evaluate one component or element of your program, and then focus on another element in a future evaluation. 

Describes and justifies the evaluation approach selected. The plan should include information about why a certain design and approach was selected. For instance, if you are implementing a new program or activity to your program, a grantee may want to pursue a process evaluation to determine whether this new component is being implemented as intended. This justification would be included in the plan so that AmeriCorps and the evaluator understands why this design was selected.

Provides instructions for the evaluation / a guide for each step of the evaluation process. An evaluation is a written document that states the objectives of the evaluation, the research questions that will be answered, and how you plan to answer those questions. An evaluation plan in many ways is like a detailed instructional manual that captures the overall picture of what your evaluation will address and the evaluation steps and activities that are needed to successfully carry out the evaluation.  It is also important to understand that an evaluation plan should be thought of as a living document, such that it is continually updated as you make progress and finalize decisions about what your evaluation will entail. It is never too early to start your evaluation plan. While you may only be able to describe a few pieces of information at the beginning stages of the evaluation planning process, your plan should be a dynamic tool to help you establish your evaluation goals and objectives, organize your evaluation activities, and determine how the work will get done. 

Additionally, as you may already know, re-compete grantees are required to submit a draft evaluation plan as part of their AmeriCorps application. The evaluation plan that is submitted at the time of application should provide sufficient detail on the evaluation objectives and study methods, as well as plans for identifying a qualified evaluator and an estimated budget. Grantees are then provided with feedback on how to obtain approval for their plan, which is needed within the first year of their grant cycle. Grantees can then work with an experienced evaluator, consultant, State Commission, or other staff familiar with evaluation to further develop and refine their evaluation plan post-award. Thus, it is important for grantee staff to engage an experienced evaluator (either an external evaluator or someone internally who has evaluation experience) early on after receiving their AmeriCorps award to ensure that the evaluator is involved in developing the evaluation plan from the start.

This presentation covers what a well-written, fully developed evaluation plan should look like by the end of the first grant year of a re-compete AmeriCorps award.

Helps decide what information is needed to address the evaluation objectives

Helps identify methods for getting the needed information

Helps determine a reasonable and realistic timeline for the evaluation

Creates a shared understanding between stakeholders (e.g., the grantee staff, evaluator, AmeriCorps staff)

Purpose of an Evaluation Plan

2021  |

  4

4

Facilitator notes: There are many reasons for developing a written evaluation plan. In addition to being a requirement for competitive AmeriCorps State and National (ASN) grantees in their second grant cycle, evaluation plans serve an important purpose. An evaluation plan:

Helps decide what information is needed to address the evaluation objectives

Helps identify the method or methods for getting the needed information and how to analyze it. It can also help you identify potential challenges for approaching the evaluation and how you may be able to address those challenges (e.g., what existing data sources can be used for the evaluation or are the resources available to carry out the type of evaluation we want to pursue). 

Helps determine a reasonable and realistic timeline for the evaluation. Having a detailed plan can lead to better efficiency in approaching each evaluation activities and maximizes your ability to successfully achieve your evaluation objectives.  

Creates a shared understanding of the purpose and use of evaluation results for program stakeholders (e.g., funder, staff, other organizations, community members), grantee staff, and external evaluator(s), if applicable. While an evaluation plan is not a formal contract, it is an opportunity for the grantee to clarify their expectations. 

Small and Large Requirements for an Evaluation Plan
|                                                      | Small Grantees (<$500,000)                                             | Large Grantees (>$500,000)                                             |
|------------------------------------------------------|---------------------------------------------------------------------------|---------------------------------------------------------------------------|
| When are grantees required to conduct an evaluation? | During 2nd competitive grant cycle, and all following competitive cycles. | During 2nd competitive grant cycle, and all following competitive cycles. |
| When must evaluation plans be approved?              | By the end of the first year of the second grant cycle.                   | By the end of the first year of the second grant cycle.                   |
| What evaluation design is required?                  | Any design (e.g., process, non-experimental outcomes).                    | Impact evaluation (experimental or quasi-experimental).                   |
| What type of evaluator?                              | Internal or external evaluator.                                           | External evaluator.                                                       |

Facilitator notes: Depending on the size of the grantee, which is determined by the estimated funding level they receive on an annual basis from AmeriCorps, there are different evaluation expectations or requirements. On this slide, we demonstrate the similarities and differences between these requirements for small grantees (receive less than $500,000 on average per year) and large grantees (receive more than $500,000 on average per year). 

As noted earlier, ASN grantees are required to conduct an evaluation beginning with their second competitive grant cycle. This is expected of all grantees, regardless of size. Additionally, all competitive grantees must have an approved plan by the end of their first grant cycle. This is typically the August of that year. For instance, if you are a grantee that was awarded a grant in 2023, you would have until August 2024 to obtain an approved plan. This timeline may be sooner for State Commission sub-grantees depending on any additional requirements set forth by their Commission. 

The differences emerge when it comes to the evaluation design requirements and the evaluator type. Small grantees have more flexibility in their design choice, and can propose a process, non-experimental outcome evaluation, or even an impact evaluation (a quasi-experimental design, or QED, or a randomized control trial, or RCT). Large grantees, on the other hand, are generally required to conduct an impact evaluation. 

For the evaluator requirements, small grantees can either use an internal evaluator or an external evaluator, but large grantees are required to use an external evaluator. However, large grantees may be able to obtain exemption from these requirements, but this requires a submission of an alternative evaluation approach (AEA) justification form. 

5

Alternative Evaluation Approach (AEA)
| AEA                            | Grantee        | Justification                                                                                                                                                                                                                                                                                                                                   |
|--------------------------------|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Funding threshold              | Large          | - Grantees who receive an average of less than $1 million per year can request to be exempt from the large grantee requirements and conduct an internal non-impact evaluation.                                                                                                                                                                  |
| Previous impact evaluation     | Large          | - Previously conducted an impact evaluation with demonstrated evidence of effectiveness (i.e., Strong or Moderate evidence).                                                                                                                                                                                                                    |
| AmeriCorps National Evaluation | Large or Small | - Grantees participating in an AmeriCorps’ Office of Research and Evaluation national evaluation (i.e., bundled evaluation or Return on Investment) that will not be completed during current grant cycle - Large grantees can also request this if the national evaluation’s design does not fulfill the requirements for a large grantee.     |

Facilitator notes: On these next two slides, we will discuss the different AEA options available to large, as well as small, grantees. The AEA options allow grantees to pursue a different design (for large grantees only) or extend the timeline of their evaluation (for both large and small grantees). These options include, from easiest to hardest to achieve: 

Funding Threshold: Large grantees may request this option if the total awarded federal share of their budget averages under $1 million per year for the full life of the grant. An external evaluator is NOT required for completing a new evaluation. 

Previous Impact Evaluation: Large grantees that conducted a previous impact evaluation of the same program that achieved a Moderate or Strong evidence rating may qualify for this exemption. An external evaluator would still be required for completing a new evaluation. 

AmeriCorps National Evaluation: This option may be requested by large or small grantees that are participating in an AmeriCorps evaluation (i.e., bundled evaluation or Return on Investment) that will not be completed during the current grant cycle and for which findings will not be available until a later date. Grantees may also request this option if the design of the AmeriCorps evaluation does not fulfill the evaluation requirements for a large grantee. 

6

Alternative Evaluation Approach (AEA) – continued 
| AEA                                          | Grantee        | Justification                                                                                                                                                         |
|----------------------------------------------|----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Structure of program or grantee organization | Large          | - Insurmountable challenges forming a comparison group. - Significant changes to program design.                                                                      |
| Replication                                  | Large          | - Implementing an evidence-based intervention with fidelity in a new setting. - A grantee’s application must be assessed at the Strong or Moderate evidence level.    |
| Timing                                       | Large or Small | - Evaluation will not be completed by end of current grant cycle. - AEA approval required only if an interim evaluation report will not meet evaluation requirements. |

Facilitator notes: AEA options continued.

Program Structure: This option may be requested if a large grantee can demonstrate insurmountable challenges to forming a comparison group, or it is not developmentally appropriate to conduct an impact evaluation due to significant changes to the program design. An external evaluator would still be required for completing a new evaluation.

Replication: Large grantees may request this option if they are implementing an evidence-based intervention (i.e., rigorously evaluated through an impact study, and the study obtained a Moderate or Strong evidence rating) with fidelity in a new setting. An external evaluator would still be required for completing a new evaluation.

Timing: Lastly, large or small grantees may request an AEA for timing if they anticipate that their evaluation will not be completed by the end of the current grant cycle. While this may allow grantees to extend the evaluation period, there are two important pieces that grantees should consider before pursuing this option: 1) grantees will be required to submit an implementation report with their next application submission, in lieu of an evaluation report, detailing the progress they’ve made to date on the evaluation; and 2) once the evaluation report is completed, it will not be considered for evidence review until their next grant cycle. That is, if a grantee is recompeting in 2024 and their evaluation report from their previous grant cycle will not be ready until August 2024, past the timeline for the application period, that report would not be available for evidence review until the next application cycle in 2027. Therefore, grantees should work with their AmeriCorps Portfolio Manager and/or State Commission before making this decision.   

More information on the AEA guidance and the request form, which grantees are required to use when pursuing this route, can be found in the Resources for Direct Grantees section. Select the “Evaluate my program” action, then find the tile with the heading “Alternative Evaluation Approach Guidance” and “Alternative Evaluation Approach Request Form.” Links to these specific resources are also available at the end of this presentation. 

7

Key Components of a Plan

2021  |

  8

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

8

Facilitator notes: Based on AmeriCorps’s guidelines in the evaluation plan template located on their website and within the Notice of Funding Opportunity document for competitive grantees and on the Resources for Direct Grantees. An evaluation plan must include the following sections: 

I. Theory of change

II. Scope of the evaluation

III. Evaluation outcome(s) of interest

IV. Research questions

V. Evaluation design 

VI. Sampling methods

VII. Data collection

VIII. Analysis plan 

IX. Evaluator qualifications

X. Timeline

XI. Budget 

Again, the outline presented on this slide can be found on the Notice of Funding Opportunity page. You can also find the template on the AmeriCorps website under the Resources for Direct Grantees section. Select the “Evaluate my program” action, then find the tile with the heading “Evaluation Plan Template.” (The actual direct link for this template will change over time as the template is updated, but the resource tile itself will not change.)

We will talk about each of these sections in more detail on the following slides.

What to include in the...

2021  |

  9

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

9

Facilitator Notes: The first section we will discuss is the Theory of Change. 

Describe how the activities undertaken by your program contribute to a chain of results that lead to the intended outcomes

Your evaluation plan must align with your theory of change and your logic model

Theory of Change

elements

Program

context

Sequence of

required events

Underlying

assumptions

Logic

model

Short-term

outcomes

Intermediate

outcomes

Long-term

outcomes

I. Theory of Change

Facilitator Notes: Your program’s theory of change should describe how the activities undertaken by your program contribute to a chain of results that lead to the intended outcomes. It is also key that the theory of change aligns with your program’s logic model and helps set the stage for the evaluation. For instance, the focus or scope of the evaluation you intend to implement should be clearly connected to the theory of change. One theme that will occur throughout this discussion is the importance of aligning each component of the evaluation.

10

What to include on...

2021  |

  11

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

11

Facilitator notes: The next section we will discuss is the scope of the evaluation, or the components of your program’s theory of change that you are interested in exploring. 

State the goal(s) of the evaluation and specify which program activities will be assessed

Programs are not expected to evaluate every component of their logic model. Instead, the evaluation may focus on a sub-set of program activities.

II. Scope of the Evaluation

2021  |

  12

12

Facilitator notes: For the scope of the evaluation, you’ll want to describe the goals for the evaluation, specifically what is it you want to learn or understand. The stated purpose of the evaluation drives the expectations and sets the boundaries for what the evaluation can and cannot deliver. The evaluation scope should also describe what aspect or aspects of your program your evaluation will focus on and over what time period. Generally speaking, the goals and the scope should align with both your theory of change and program logic model. 

Another key detail, is that programs are not expected to evaluate every component of their program’s logic model. It is more important to develop a quality evaluation that focuses on a few components of a program, and then use findings from that evaluation to explore other program activities. 

What to include on...

2021  |

  13

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

13

Facilitator notes: Following the scope of the evaluation are the evaluation outcomes of interest. As we will discuss on the next slide, the outcomes of interest must align with the program’s theory of change and logic model. 

Describe what outcomes your evaluation will measure

Process / implementation outcomes or outputs

Program beneficiary outcomes

Member outcomes  

Your outcomes of interest should be:

Part of your program’s theory of change and included in your logic model

Feasible for your program to measure given the source(s) of data needed and level of effort required

III. Outcome(s) of Interest

2021  |

  14

14

Facilitator notes: For non-experimental outcome evaluations, outcomes must involve a change in knowledge, attitude, behavior, or condition. Metrics that measure the amount of service provided (e.g., number of students tutored/volunteers recruited/organizations served) should not be listed as outcomes for this type of evaluation. However, for process or implementation evaluations, these metrics may be appropriate.

The outcomes identified should align with the outcomes described in the program's logic mode. Outcomes must also align with the theory of change, the scope of the evaluation, and must be feasible to measure, based on the source(s) of data needed and level of effort required. Given the length of a single grant cycle, it can be challenging to evaluate long-term program outcomes; evaluating short- and medium-term program outcomes may be more practical. 

For a process evaluation, in which outputs or activities are assessed instead. For this section, grantees proposing a process evaluation would instead be describing the outputs, activities, or the intended program goals which the study will assess. 

What to include on...

2021  |

  15

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

15

Facilitator notes: The next section of your evaluation plan should articulate the key research questions that will be investigated. Research questions – the areas you’ll examine as part of an evaluation of your program – are key to defining exactly what it is you’re trying to accomplish. For that reason, research questions should be chosen carefully. When you choose research questions, you're really choosing a research problem - what you want to examine with your research. 

One or more questions that define exactly what your evaluation intends to accomplish

The following are characteristics of a good research question:

Clearly stated and specific

Aligns with your theory of change / logic model

Connect to the outcomes of interest

Measurable and feasible to answer 

Aligns with your chosen evaluation design

IV. Research Questions

2021  |

  16

16

Facilitator notes: There is no set number to how many research questions your evaluation should attempt to answer, however you want to be sure that it is feasible and you have the resources to answer each research question at the end of your evaluation. Thus, it is important that the research questions for your evaluation are stated in such a way that they:

Are clearly stated and specific

Align with your program’s theory of change and logic model. Your questions should match the objectives, activities, and outcomes of the program

Will lead to measurable or observable results

Aligns and is appropriate for the chosen evaluation design. For instance, research questions for an impact evaluation should focus on comparisons or changes between the program’s beneficiaries against individuals that are not part of the program and don’t directly benefit from their services. 

Research questions are worded differently depending on their focus: 

IV. Research Questions

2021  |

  17

17

Facilitator notes: As noted above, research questions should align with the evaluation design being proposed. For process designs, which focus on how the program’s activities are carried out or its planned work – inputs, activities, and outputs (direct products), the goal is to answer questions about what you did in the program, the process side of the logic model will be where you focus your question(s). This could be asking whether the program is being implemented as intended, or identifying the facilitators or challenges of carrying out certain activities. Outcome designs intend to answer questions about what changes the program brought about. 

As this graphic illustrates, process evaluations address questions about program operations, namely the who, what, when, where, why, and how  many of program activities and program outputs. On the other hand, outcome evaluations measure a program’s outcomes and assess program effectiveness. It is important to note that a single evaluation plan can and often does include both process and outcome evaluation questions.

More information about research questions can be found in the “Asking the Right Research Questions” slides on the Evaluation Resources page. 

What to include on...

2021  |

  18

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

18

Facilitator notes: Up to this point in your evaluation plan, you’ve established the framework for your evaluation. After identifying the questions you want to answer, the next step is to explain the approach your evaluation will take to address those questions. 

Description of general categories of evaluation designs:

V. Evaluation Design 

2021  |

  19
| Process Evaluation                                                                                                                                                                                                                                                                  | Outcome Evaluation                                                                                                                                                                       | Impact Evaluation                                                                                                                                                                                                  |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Examines the extent to which a program is operating as intended by assessing ongoing program operations and determining whether the target population is being servedResults may be used to determine what changes and/or improvements should be made to the program’s operations   | Measures changes in knowledge, attitude(s), behavior(s) and/or condition(s) among program beneficiaries or other stakeholder groupsResults may demonstrate what the program has achieved | Measures changes in program beneficiaries or stakeholders relative to a reasonably similar comparison/ control groupResults are an estimate of the program’s impact on beneficiaries or other stakeholder groups   |

19

Facilitator notes: For AmeriCorps, there are three categories of evaluation designs: process evaluations (which looks at how the program is operating and helps identify areas of improvement); outcome evaluation (which measures changes among program beneficiaries and provides details on what the program has achieved); and impact evaluation (which measures and compared changes among program beneficiaries against a reasonably similar comparison or control group). 

Evaluation design is essentially the structure that will provide you the information needed to answer each of your evaluation questions. The appropriate design for evaluating a program will largely depend upon certain considerations. This may include, but is not limited to:  

Your program’s theory of change and logic model

Primary purpose of the evaluation and key research questions

Funder’s evaluation requirements: Many funders require their grantees to complete evaluations of their program. As you consider different evaluation designs, you should also have a clear understanding of whether your funder has any specific evaluation requirements that must be fulfilled. Those requirements may drive the purpose and scope of your evaluation. For example, it is important to note that AmeriCorps has evaluation requirements for large and small re-competing grantees in terms of which evaluation design they may use to assess their programs. Large grantees are those receiving annual AmeriCorps funds of $500,000 or more. Small grantees are those receiving annual AmeriCorps funds of less than $500,000. You should note which type of design is required for your program. As noted earlier, AmeriCorps State and National grantees may seek exemptions from impact designs or requirements if they meet the criteria set forth in the AEA guidance.

Resources available for the evaluation: It is also important to take into consideration what resources (staff time, funding, evaluation expertise) are available to carry out the evaluation. In particular, your evaluation team must possess the skills and experience needed to carry out the type of evaluation design that is chosen. Think about whether you will be able to collect the data required for a given design. More complex evaluation designs require you to collect more data over multiple measurement points. This lengthens the data collection phase of the study and may require more sophisticated analyses to be conducted once the data are collected.

V. AmeriCorps Approved Evaluation Designs 

2021  |

  20
| Type of Design                                        | Category | Details needed on evaluation design                                                                                                                                                                                                                                    |
|-------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Experimental design/Randomized Controlled Trial (RCT) | Impact   | Description of the random assignment procedures that will be used to form treatment and control groupsEligibility criteria for inclusion in the study                                                                                                                  |
| Quasi-experimental Design (QED)                       | Impact   | Description of the approach for identifying a reasonably similar comparison group (e.g., propensity score matching, difference in difference analysis)   List of variables (covariates) to be used to statistically equate treatment and comparison groups at baseline |
| Non-experimental Design                               | Outcome  | Description of whether pre- AND post-test measurements OR post-only measurements will be used                                                                                                                                                                          |
| Process                                               | Process  | Description of the methods that will be used (i.e., qualitative only, quantitative only, or mixed methods)                                                                                                                                                             |

20

Facilitator notes: This slide shows the most common types of evaluation designs for each of the three categories discussed on the previous slide. In addition to the details presented above, the following are characteristics of each design type: 

Experimental design or Randomized Controlled Trial (RCT): An experimental design follows two or more study groups – one is the intervention group and the other is the control group. In an experimental evaluation design, the term control group rather than comparison group is used to describe the study group against which to measure the outcomes of program beneficiaries. 

In an experimental evaluation design, the evaluator employs random assignment techniques (e.g., lottery draw) to assign study participants to either the intervention group or the control group. This means that each study participant has an equal chance of being assigned to either the intervention group or the control group. The control group in an experimental design may receive no services or alternative or delayed services. The use of random assignment in experimental designs creates as equal groups as possible by ensuring that there are no systematic differences between the program and control groups. The process of randomly assigning program applicants for experimental design studies is complex and generally requires specific tailoring to each program’s unique application and intake process. For this reason, random assignment is best conducted using a professional evaluator with experience completing these types of evaluations. 

Quasi-experimental Design (QED): A quasi-experimental evaluation design collects data on two or more study groups – one is the intervention group (i.e., the program’s beneficiaries) and the other is a comparison group (i.e. the study participants that do not receive the intervention or program services and thus serve as the basis for comparison when assessing the effects of an intervention on a program’s beneficiaries). Including a comparison group enables you to answer specific questions related to causality – such as, what would have happened to people if they did not receive the intervention your program offers (i.e., whether the observed changes can be attributed to your intervention). 

In a QED, the intervention and comparison groups are identified from pre-existing or self-selected groups and not through a random assignment process. For example, the intervention group often consists of individuals who self-select to participate in the program being evaluated while the comparison group consists of individuals who have similar characteristics to the intervention group EXCEPT that they are not served by the program or are participating in a different program. The key is that your comparison group does not receive the same intervention that is being evaluated because you want that group to serve as a point of comparison for the group that does receive the intervention. 

It is important to note that you should be thoughtful in your selection of a comparison group to ensure that it is as similar as possible to intervention group. This is key to ensuring that evaluation findings are an accurate estimate of the program’s effect. In most cases, you will want an experienced evaluator to use statistical matching to ensure that your comparison group is as similar as possible to the group of program beneficiaries in your evaluation.

Non-experimental design: A non-experimental outcome design only tracks or collects data on the outcomes for the intervention group (i.e., program beneficiaries). There are variations within the category of non-experimental outcome design, differing only in the number and the timing of outcome measurement points. These variations include:

Single group post-test: Data are collected only once from study participants, immediately after they complete the intervention/program. The major drawback of any single group post-test design is that it does not provide a baseline against which results can be compared. To assess change, you must be able to compare post intervention data with a baseline measure. As shown in the first row of the table.

Single group pre- and post-test: A pre- and post-test design measures outcomes among program beneficiaries before and after the intervention. With this design you simply administer the same measure twice, before and after the intervention. As shown in the second row of the table. The timing of the posttest measure is important. It should allow enough time for your program to have an effect, but not so much time that program effects are diluted or influenced by external factors such as participation in other programs or other environmental circumstances. You can improve upon the single group pre- and post-test design by additional measurement points post-intervention to gain a perspective on your program over time. 

Process: A process evaluation focuses on a program’s inputs, activities, and outputs. 

It can be used to document what a program is doing and to what extent and how consistently the program demonstrates fidelity to the program’s logic model. The results of a process evaluation are most often used to change or improve the program. Process evaluations are able to address research questions about why a project is or is not successful, which can be very helpful for program staff and stakeholders because the results are useful for improving program practices. 

The collection of both qualitative and quantitative data through interviews, surveys, and program administrative data is usually preferred. Additionally, process evaluations mostly rely on simple descriptive statistics (means, frequencies, etc.) and do not require advanced statistical methods. 

More information about evaluation designs can be found in the “Overview of Evaluation Designs” slides on the Evaluation Resources website. 

What to include on...

2021  |

  21

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

21

Facilitator notes: Now that you have presented and described the evaluation design and general framework and approach to your evaluation, there are several additional sections of an evaluation plan that require description, starting with your sampling methods.

For each data source, describe the sample or the target population for the evaluation, including: 

Eligibility criteria that limits the sample or population (e.g., participation level, site/location, age or grade level) 

Sampling procedures (e.g., random, purposeful, or convenience sampling) 

Expected size of the sample or population 

Rationale or justification for sample sizes

For ASN grantees, sample size justification is required for non-experimental (i.e., rationale for sample selection) and impact designs (i.e., a power analysis)

VI. Sampling Methods

2021  |

  22

22

Facilitator notes: In the sampling methods section, the sample or the target population for each data source must be described, including: 

Eligibility criteria that limits the sample or population, such as by participation level, site/location, age or grade level

Sampling procedures, for example, random sampling, purposeful sampling, or convenience sampling 

The expected size of the sample or population. Please note that you must include an estimated number even if you plan to use the entire population. 

And finally you should include a rationale or justification for the sample sizes. If you plan to use the entire population, for example, it may be as simple an explanation as this. If you plan to use a sample, explain how you chose the sample that you will be using and why. For ASN grantees, sample size justification is required for non-experimental and impact evaluation designs. For impact evaluation designs the justification is in the form of a power analysis, which is explained on the next slide. 

A power analysis is used to determine:

How large a sample is needed to enable statistical judgments that are accurate and reliable (i.e., required minimum sample size)

How likely your statistical test will be to detect effects of a given size in a particular situation 

It is recommended that a plan include the results of a power analysis if:

It is an impact evaluation design (i.e., RCT or QED) 

Your analysis involves statistical significance testing

VI. Sampling Methods

2021  |

  23

23

Facilitator notes: In some evaluations, especially impact evaluations, a power analysis is used to determine how large a sample is needed to make accurate and reliable statistical judgments about outcomes. Power helps determine how large a sample size you need in order to obtain reliable evidence of program impacts. It is recommended that a plan include the results of a power analysis if it is an impact evaluation design (i.e., RCT or QED) or if your analysis involves statistical significance testing. For ASN grantees who are conducting an impact evaluation, their evaluation plan must include a power analysis. 

What to include on...

2021  |

  24

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

24

Facilitator notes: The next section of the evaluation plan covers the data collection procedures, data sources, and measurement tools. 

Provide a detailed description of the data that will be collected or extracted to answer the research questions. This section should address:

Who/what will be the source of the data?

What tools/instruments will be used to collect data? 

What is the plan for accessing administrative/extant data?

What information will be collected/compiled? 

When and how often data will data be collected?

Ensure that the data are adequate for addressing all of the study's research questions

VII. Data Collection Procedures, Data Sources, and Measurement Tools

2021  |

  25

25

Facilitator notes: This seventh section of your evaluation plan is really about the data you will collect or compile to answer each of the key evaluation questions that you’ve identified for your evaluation, keeping within the framework of your chosen evaluation design. In this section, it is important that you explain what data will be used in the study, how the data will be used, and how you plan to collect or compile the data. You want to be sure that the data collection methods you’re going that will help to generate accurate information about your program. 

Key information to include in this section is listed here on the slide. More details on selecting and collecting data for your evaluation are available in the “Data Collection for Program Evaluation” slides on the Evaluation Resources web page. 

If there are other activities that may affect how data are collected, you should include that detail here. One example would be information on obtaining Institutional Review Board (IRB) clearance, if that is deemed necessary for your evaluation. IRBs are committees that review research protocols and other materials to ensure the rights, safety, and welfare of human subjects participating in studies. The IRB can certify that the rights of subjects will be protected, that any potential adverse effects on participants will be minimized, and the data will be securely managed and maintained. IRB clearance may be required by your agency leadership or funding source, so it is important to understand what is required before you begin to recruit study participants and collect data. An experienced evaluator should be aware of any IRB clearance requirements that will need to be fulfilled before data collection can begin.

What to include on...

2021  |

  26

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

26

Facilitator notes: After data collection, the next section of your evaluation plan should describe the analytic techniques that will be used to analyze all of the data you plan to collect or compile for the evaluation. 

Explain how each data source will be analyzed to produce findings that address the evaluation's research questions

VIII. Analysis Plan

2021  |

  27
| Type of design                               | Details needed on analysis                                                                                                                                                                                                                                                  |
|----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Non-experimental / Process evaluation design | The quantitative data analysis techniques that will be used to produce the study findings (e.g., Chi-square, t-tests, frequencies, means)The qualitative data analysis techniques that will be used to produce the study findings (e.g., content analysis, thematic coding) |

27

Facilitator notes: Analytical techniques range from basic counts to content analysis to complicated inferential statistical analysis. Depending on the type of data you plan to collect, you might need to conduct quantitative analysis, qualitative analysis, or both. The analytic techniques that are selected should be based on and align with your evaluation questions, the selected evaluation design, and the type of data that will be collected. 

For instance, non-experimental outcome designs and process evaluation designs could use a mixed methods approach, or collect both quantitative and qualitative data. The analytical techniques for these designs tend to be less complex. It is common for non-experimental outcome evaluations to employ simple descriptive statistics (comparing means or frequencies), Chi-square tests, or t-tests for analyzing the quantitative data collected. The qualitative data analysis, which is most common in process evaluation or used to complement quantitative data in non-experimental outcome designs, typically involved organizing and aggregating the data into themes that allow you to identify data trends. To ensure that qualitative data (such as from interviews or field observations) are amenable to analysis and systematically comparable, coding schemes are typically applied to the notes or data. Various approaches are used to interpret meaning or themes from the content of text data. It is important to note that anecdotes or what we sometimes refer to as “great stories” are not the same as qualitative data. Anecdotes – or personal accounts, thoughts, or feelings – collected in an ad hoc fashion cannot tell us whether any improvements occurred in an intervention because no measurements were established. 

VIII. Analysis Plan

2021  |

  28

NOTE: Chi-square tests and t-tests are not adequate for conducting a QED analysis. Instead, a multivariate regression model (e.g., ANOVA) is preferred, so covariates (e.g., pre-test measures and other variables that may affect the outcome of interest) can be controlled for in the analysis. 
| Type of design             | Details needed on analysis                                                                                                                                                                                                                    |
|----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Impact design (RCT or QED) | The statistical test/model that will be used to compare outcomes for the treatment and comparison groupsPlans to assess baseline equivalency of the treatment and comparison groups and any statistical adjustments to be used (if necessary) |

28

Facilitator notes: For impact designs, the statistical techniques tend to be more complex. These designs often require assessing baseline equivalency between the treatment and comparison groups (i.e., are the two groups similar enough at baseline) and could involve multivariate regression models, and other more complex analyses. Therefore, these designs and associated evaluation plans would require experienced evaluators to develop a plan for analysis. As mentioned before, quantitative analysis is generally the only way to provide evidence of program impact.

It is important to note that certain analytical techniques are not appropriate for these more rigorous designs. For instance, Chi-square tests and t-tests, which can be used in a non-experimental outcome evaluation, are not adequate for impact designs. 

What to include on...

2021  |

  29

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

29

Facilitator notes: Grantees will then be expected to describe the evaluator – internal or external – as well as their qualifications for carrying out the study. 

Describe the evaluator(s) who will be carrying out the evaluation activities, including

Whether they are internal or external to the program; 

If using an external evaluator, the description clearly describes how they are independent from the program (e.g., no conflicts of interest); and 

Qualifications for conducting the evaluation

The type of evaluator used may also depend on the size of your program

Small grantees (<$500,000) can use an internal or external evaluator

Large grantees (>$500,000) expected to use an external evaluator, unless a Funding Threshold AEA is pursued

IX. Evaluator Qualifications

2021  |

  30

30

Facilitator notes: For this section, grantees should describe the type of evaluator they intend to use to carry out the evaluation – internal, or someone from the program or host organization, or external, or someone from an external entity such as a college, university, or other organization not associated with the program. Determining whether to use your own internal staff or to rely on an external evaluator to complete an evaluation of your program is a critical decision that will need to be made early on in the planning phase of your evaluation. Generally speaking, the following are key considerations when determining whether to employ an external evaluator/team for an evaluation:

Expertise: Some programs may not have internal staff with the type of technical expertise required to conduct an evaluation. Therefore, depending on the scope and complexity of the evaluation, an individual or team who is external to the program may be needed. This is particularly the case for impact designs in which there are more rigorous analyses (e.g., power analysis, baseline equivalency testing, etc.). It is considered best practice to use an external evaluator for impact evaluations as it helps provide credibility to the study findings; it also helps reduce the likelihood of bias compared to when a program staff person conducts this evaluation. For AmeriCorps State and National grantees, however, large grantees receiving more than $500,000 annually, are required to use an external evaluator. 

Cost: Both internal and external evaluations have cost implications. While using internal program staff to complete an evaluation of your program may be less costly, it also adds to staff workload. It is important to consider this trade-off and whether staff have the capacity to take on additional work to carry out the evaluation activities. Where personnel resources are more scarce than funding, an external evaluator may be the better choice.

Furthermore, the type of evaluator identified for the evaluation may depend onFunders may require that the programs they fund conduct an external evaluation to ensure objectivity in the evaluation’s results. For example, as you may already know, AmeriCorps grantees receiving annual funds of less than $500,000 are NOT required to conduct an external evaluation. However, AmeriCorps grantees receiving annual funds of $500,000 or more are required to conduct an external evaluation.

More information about external evaluators can be found in the “Managing External Evaluations” slides on the Evaluation Resources page. 

What to include on...

2021  |

  31

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

31

Facilitator notes: The final two sections of the evaluation plan are the timeline and budget. While some level of specificity is necessary for the timeline (i.e., ensure that major activities are outlines, and the month and year of each of those activities are captured), the budget section could be a simple line detailing the overall cost (e.g., $5,000 for the entire evaluation, or in-kind resources will be used to fund this study) or more detailed with information about costs per year or activity. However, this level of detail is not necessary for the budget section. 

Provide a detailed timeline of when the major evaluation activities will occur (e.g., evaluation planning, hire evaluator, develop instruments, collect data, analyze data, write report)

Helps determine if the evaluation is on track to be completed on time (i.e., before the next GARP cycle)

Specify the budget allotted for the evaluation

X-IX. Timeline and Budget

2021  |

  32

32

Facilitator notes: As noted on the previous slide, your evaluation plan should include a detailed timeline of when you expect to carry out each of your key evaluation activities specified in your plan. You may find it helpful to lay out a more detailed timeline that lists each of your key evaluation activities and maps out when you expect the activity to start and end and who is responsible for carrying out the activity for added accountability. Importantly, the timeline provides AmeriCorps and your evaluator, if needed or required, an idea for when which activity will be completed and that enough time is set aside for analysis and reporting. AmeriCorps grantees should ensure that the timeline clearly shows that the evaluation report will be completed prior to the next grant cycle. Even though an AmeriCorps State and National grantee operate on a three-year grant cycle, the evaluation needs be completed at the beginning of the third grant cycle and the report is included as part of the application that is submitted for the next grant cycle. 

An estimated budget for your evaluation also should be included in your plan. The cost of evaluations varies widely and will depend on the type of study design, the size of the study, the level of expertise and experience of the evaluator, and data collection expenses. Other common considerations for creating a program evaluation budget are: staff time; materials, equipment, and supplies; travel costs; and data collection. With respect to this last item, evaluations involving more primary data collection tend to be more expensive than those that rely on existing internal program records or external data sources. This is not a comprehensive list of cost consideration. Depending on the program to be evaluated and/or the actual evaluation activities, there may be additional expenses required. 

Know what type of evaluation you must complete

Small vs large grantee requirements

AEA options available

Fully describe each component of the plan

See ASN Notice of Funding Opportunity (NOFO) and Frequently Asked Questions: Evaluation for relevant grant cycle

Review the Evaluation Plan template

Ensure that your description of each of the components aligns with one another (i.e., interrelated)

General Guidelines to Follow

2021  |

  33

33

Facilitator notes: As we conclude today’s presentation, we’ve highlighted some general guidelines to keep in mind as you work to develop and use your evaluation plan:

Know what type of evaluation you must complete based on the average level of funding you receive from AmeriCorps – i.e., small grantees have flexibility in the design they can pursue, whereas large grantees are generally expected to conduct an impact evaluation. However, the AEA options allow large to pursue other designs should they face any particular changes conducting an impact evaluation. 

Fully describe each component of your evaluation plan, and ensure that each are in alignment. For instance, if proposing a process evaluation, the research questions should be appropriate for that design (e.g., focusing on whether the program is being implemented with fidelity, identifying challenges and facilitators, etc.). Details on what is expected for the evaluation plan and each component are available in the NOFO as well as the evaluation plan template, both of which are available on the Funding Opportunity Page of the AmeriCorps website. 

Your AmeriCorps Portfolio Manager or State Commission representative 

ASN – Evaluation Resources

https://www.americorps.gov/grantees-sponsors/evaluation-resources

2021 ASN Grantee Symposium presentation Demystifying Evaluation Plans: https://www.youtube.com/watch?v=GJ0s9HL8Q2A

AmeriCorps Evaluation TA Portal

https://americorpsevaluationta.norc.org

Internal Resources

2021  |

  34

34

Facilitator notes: Here we provide a list of internal resources available to all AmeriCorps programs and grantees. 

Resources for State and National Direct Grantees

https://americorps.gov/grantees-sponsors/directs-territories-tribes?field_document_type_tax_target_id=19756#resources

This page includes information about the following:

AEA Guidance and Request Form

Evaluation FAQs

Evaluation Plan Template

Evaluation Requirements

Internal Resources (continued)

2021  |

  35

35

Facilitator notes: Here are additional internal resources that can be accessed. 

Your external evaluator

The American Evaluation Association

http://www.eval.org

External Resources

2021  |

  36

36

Facilitator notes: In addition to what is available through the AmeriCorps site and staff, you may also attain more information from your external evaluator, if relevant, or through the American Evaluation Association, which includes helpful links and materials about evaluation generally, and is not specific to AmeriCorps. 

Questions?

  37

Facilitator notes: Does anyone have any questions?

37

Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and Evaluation: evaluation@cns.gov

Thank you!

# image2.emf

# image6.png

# image9.png

# image3.png

# image4.emf

# image1.png

# image5.png