NORC at the University of Chicago

Basic Steps in Conducting an Evaluation

2023  |

1

1

Facilitator notes: This presentation provides an overview of the basic steps involved in conducting an evaluation. 

By the end of this presentation, you will be able to:

Describe the basic steps for conducting an evaluation  

Plan for an evaluation

Identify the key components of an evaluation plan

Identify approaches for collecting and analyzing data

Understand how to communicate and apply findings for program improvement

Learning Objectives

Facilitator notes: For this presentation, we have identified a number of learning objectives.  

By the end of this presentation, you will be able to: 

Describe the basic steps for conducting an evaluation

Plan for an evaluation

Identify the key components of an evaluation plan

Identify approaches for collecting and analyzing data

Understand how to communicate and apply findings for program improvement

2

What are the basic steps for conducting an evaluation?

Planning phase

Development phase

Implementation phase

Action and improvement 

Overview of Presentation

2021  |

  3

Facilitator notes: This presentation will cover the following:

What are the basic steps for conducting an evaluation? These include:

Planning phase

Development phase

Implementation phase

Action and improvement

We want to point out that this is a basic overview of the steps to take for conducting an evaluation. We have developed other courses that will address some of the steps and topics introduced today in more depth. 

3

Evaluation Cycle—Four Phases

2021  |

  4

Facilitator notes: Evaluation can be thought of as a set of linked activities and the process for undertaking an evaluation includes four main phases – planning, development, implementation, and action and improvement.

4

Planning

Development

Implementation

Action and Improvement

Basic Steps for Conducting an Evaluation

2021  |

  5
| Phase                  | Step                                                                                                                                          |
|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
| Planning               | Step 1: Build (or Review) a Program Logic Model Step 2: Define Purpose and Scope Step 3: Budget for an Evaluation	Step 4: Select an Evaluator |
| Development            | Step 5: Develop an Evaluation Plan                                                                                                            |
| Implementation         | Step 6: Collect DataStep 7: Analyze Data                                                                                                      |
| Action and Improvement | Step 8: Communicate FindingsStep 9: Apply Findings and Feedback for Program Improvement                                                       |

Facilitator notes: Within each of the 4 broad phases, there are 9 basic steps for conducting an evaluation. The planning phase involves building (or reviewing a program logic model), defining the purpose and scope of the evaluation, budgeting for an evaluation, and selecting an evaluator. The planning phase is followed by the development phase which involves the creation of an evaluation plan. The next phase relates to implementation where data are collected and analyzed. The last phase, action and improvement, involves communicating findings and applying those findings and feedback for program improvement. Together, these steps are designed to help build a strong foundation for your evaluation. 

It should be noted that while some order exists as to how programs generally approach evaluation steps, the process largely depends on a project’s particular circumstances. In addition, the interdependent nature of the activities means that the steps are not necessarily linear but may evolve throughout the development and implementation of your evaluation strategies.  

We’ll talk through each of these steps in more detail over the next several slides.

5

Planning Phase Steps

2021  |

  6

Build a Program Logic Model

Define Purpose and Scope

Select an Evaluator

Budget for an Evaluation

Planning

Facilitator notes: The first four steps in conducting an evaluation are part of the planning phase and involve activities that will help you prepare for an evaluation of your program. We’ll talk about these four planning steps in more detail over the next few slides. Again, we want to emphasize that these planning steps do not need to follow this particular order. Depending on your program, these activities may overlap with one another or follow a different order than what is presented above. 

6

A program logic model is a detailed visual representation of a program and its theory of change.

Communicates how a program works by depicting the intended relationships among program components: 

Inputs or resources

Activities

Outputs

Outcomes

Step 1: Build a Program Logic Model	

2021  |

  7

Outcomes

Facilitator notes: As you begin to plan for an evaluation of a program or intervention, it is essential that there be a clear and comprehensive mapping of the program or intervention itself. Thus, a useful first step in planning an evaluation should be to clarify and confirm your program’s operations or processes and intended outcomes by developing a logic model. If your program has already developed a logic model, then you might only need to review the existing model and possibly update or refine it to reflect your current program operations and goals.

Let’s talk about what a program logic model is:

A program logic model is a detailed visual representation of your program and its theory of change that communicates how your program works, the resources you have to operate your program, the activities you carry out, and the outcomes you hope to achieve. Your program logic model should clearly communicate how your program works by depicting the intended relationships among program components. Key program components consist of:

Inputs or resources - which are considered essential for a program’s activities to occur. They may include any combination of human, financial, organizational, and community-based resources that are available to a program and used to carry out a program’s activities. 

Activities – which are the specific actions that make up your program or intervention. They reflect processes, tools, events, and other actions that are used to bring about your program’s desired changes or results.  

Outputs – what a program’s specific activities will create or produce, providing evidence of service delivery. 

Outcomes - the specific changes that may result from a program’s activities or intervention. A program’s outcomes fall along a continuum, ranging from short- to long-term results. 

The four components outlined on this slide reflect a basic or simple logic model design which aligns with ASN’s logic model guidelines. For this presentation, we use this basic logic model format. 

Also, some of you may be familiar with other terms that are used interchangeably with logic models, such as causal chain, conceptual map, idea map, logical framework, model of change, and roadmap. We use the term “logic model” throughout this presentation. Likewise, there are also a variety of terms for describing these four key components (inputs/resources, activities, outputs, and outcomes) of a logic model, but for this presentation we use the terms shown here on this slide. 

7

Short

Long

Activities

Outputs

Medium

Inputs

Logic models read from left to right

There are two “sides” to a logic model - a process side and an outcomes side.

Step 1: Build a Program Logic Model 

2021  |

  8

Facilitator notes: Logic models are typically read from left to right, employing an if-then sequence among key components. A generic example is shown here. It reads, if your program has these inputs or resources, then it can carry out these activities. If your program carries out these activities, then it can produce these outputs. If your program has produced these outputs, then it will achieve these outcomes. 

In addition, a logic model has two “sides.” The process side focuses on a program’s implementation or its planned work – inputs/resources, activities, and outputs (direct products). The outcomes side of the logic model describes the expected sequence of changes that the program hopes to accomplish, which can be short-term, medium-term, and/or long-term changes. The outcomes side reflects what difference the program intends to make.

8

Step 1: Example Logic Model 

2021  |

  9
| INPUTS                                                                                     | ACTIVITIES                                                                                                                                                          | OUTPUTS                                                                                                                                                                                             | Outcomes                                                                                                                                                                                                                   |                                                                                                                                                                                           |                                                                                                                         |
|--------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
|                                                                                            |                                                                                                                                                                     |                                                                                                                                                                                                     | Short-Term                                                                                                                                                                                                                 | Medium-Term                                                                                                                                                                               | Long-Term                                                                                                               |
| What we invest                                                                             | What we do                                                                                                                                                          | Direct products from program activities                                                                                                                                                             | Changes in knowledge, skills, attitudes, opinions                                                                                                                                                                          | Changes in behavior or action that result from participants’ new knowledge                                                                                                                | Meaningful changes, often in their condition or status in life                                                          |
| Funding Staff100 AmeriCorps State and National members50 non-AmeriCorps volunteersResearch | Conduct job readiness workshopsProvide job search assistanceProvide peer counseling services Provide referrals to transitioning servicesEducate potential employers | # individuals participating in workshops# individuals receiving job search assistance   # individuals receiving counseling services# families receiving referrals  # employers receiving education  | Increased confidence in gaining employmentIncrease in job readiness skillsIncreased knowledge of effective job search strategies Increased knowledge of community servicesIncreased employer knowledge of hiring benefits  | Increase in job placementIncreased capacity of families to manage transition from military to civilian work and family lifeIncreased adoption of military-friendly practices by employers | Individuals maintain stable employmentIncreased family well-beingEmployers routinely hire veterans and military spouses |

                        Process                                                                     Outcomes

Facilitator notes: On this slide we present an example of a logic model for a hypothetical AmeriCorps job readiness program for veterans. 

This hypothetical program is designed to address the disproportionately high rates of unemployment that veterans and their spouses often face compared to their civilian counterparts as well as significant challenges they encounter in transitioning into civilian work and community life. Barriers include the inability to translate military accomplishments into civilian terms, the lack of knowledge regarding available community supports, and employers’ lack of understanding of the unique strengths of veterans as well as tax credits available for their hiring. To combat veteran unemployment, this hypothetical AmeriCorps program supports job readiness and job search skills, transitioning services for families, and education and outreach to potential employers.

9

Why develop a logic model?

Generate clear and shared understanding of how program works

Support program planning and improvement

Serve as foundation for evaluation 

For an overview of logic models, AmeriCorps grantees can refer to the module, “How to Develop a Program Logic Model” located on the Evaluation Resources webpage. 

Step 1: Build a Program Logic Model 

2021  |

  10

Facilitator notes: Now that you have a general understanding of what a logic model is and what it looks like, let’s talk about some of the ways they are useful. 

Developing a logic model has many potential uses for programs. It may be the case that you develop one logic model for your program that serves more than one purpose, or it may be necessary for you to create different versions of a logic model that is tailored for different aims. For example, the logic model you develop for your daily program operations (including performance management and evaluation activities) may be different than the logic model you develop for your GARP application.

On this slide, we present just a few examples of how logic models can be used:

Logic models can be used to ensure that stakeholders have a clear and shared understanding of how a program is intended to work. For example, sharing a program’s logic model with program staff can be beneficial in generating a collective sense of responsibility and accountability as it enables staff to understand how their specific role fits into the broader context of the program. Additionally, funders may require programs to include a logic model in their proposal or application materials because it provides a snapshot or a succinct description of how a program will address a problem by performing specific activities. 

Logic models can be used to support program planning and improvement efforts. For example, logic models can be a useful program management tool to help you determine how to allocate resources effectively as well as serve as a framework to monitor whether the program is being implemented according to plan. 

Last on this slide, logic models can serve as a foundation for evaluation. Logic models can be particularly helpful during the planning stages of an evaluation by helping programs determine when and what to evaluate, so that evaluation resources are used effectively and efficiently. 

10

A logic model can serve as a framework for your written evaluation plan. 

It can help you focus your evaluation by identifying: 

Questions want/need answered 

Aspects of program to evaluate 

Type of evaluation design

Information to collect 

Measures and data collection methods 

Evaluation timeframe

For more information on logic models, AmeriCorps grantees can refer to the module, “How to Develop a Program Logic Model” located on the Evaluation Resources webpage. 

Step 1: Build a Program Logic Model 

2021  |

  11

Facilitator notes: So how can a logic model be used to help in planning for an evaluation? Your logic model can serve as a framework for your evaluation plan by helping you can make informed decisions about what to evaluate, when to evaluate, and how you will evaluate. 

It is important to note that with program evaluation, it is not necessary to evaluate every aspect of your program as depicted in your logic model. Your evaluation can have a narrow focus (e.g., only address questions about one of your program’s service activities and desired outcomes) or it can have a broader focus (e.g., address questions about each of your program’s service activities and desired outcomes), depending on the information you hope to gain from your evaluation. 

Your logic model can be used as a tool to help you focus your evaluation with respect to the following: 

Identify questions you want or need answered about your program

Identify which aspects of your program to evaluate (e.g., will you evaluate a subset or all of your AmeriCorps activities?; will you evaluate your program’s short-term outcomes?) 

Determine the appropriate evaluation design (e.g., will you use a process or an impact evaluation design, or a combination of both?)

Identify what information to collect 

Identify measures and data collection methods

Determine an appropriate timeframe for your evaluation

Throughout this presentation, we will be making several references to a logic model – we consider a logic model to be a planning tool that can be used to help guide you through many of the basic steps involved in planning and conducting an evaluation. 

For more information on developing a program logic model, please see the webinar on How to Develop a Program Logic Model located on the Evaluation Resources webpage.

11

Each evaluation should have a primary purpose around which it can be designed and planned.

Why is the evaluation being done? What do you want to learn? 

How will the results be used? By whom? 

Additional things to consider: 

Specific program requirements

Resources available to carry out the evaluation

Step 2: Define Purpose and Scope

2021  |

  12

Facilitator notes: Let’s turn now to talk about a second step in the evaluation planning phase – defining the purpose and scope of your evaluation. Just as your program needs to have a specific purpose and scope, so does your evaluation. Each evaluation should have a primary purpose around which it can be designed and planned, although it may have several other purposes. The stated purpose of the evaluation drives the expectations and sets the boundaries for what the evaluation can and cannot deliver.

In defining the purpose of the study, it is helpful to identify why the evaluation is being done and how the information collected and reported by the study will actually be used and by whom. 

For example, are program staff trying to understand how to operate the program more efficiently or identify barriers or constraints to implementation? Or does your program need to produce evidence that it is meeting its intended outcomes? Will the results be used by program staff to make changes to the program’s implementation? Will the results be used by the program’s funder to make decisions about future funding opportunities?

In general, defining a specific purpose for your evaluation will allow you to set parameters around the data you collect and methods you will use. 

Questions about why your evaluation is being done and how the information will be used should be discussed among a variety of program staff, and any other individuals who may be involved in the evaluation to ensure there is consensus as to what the evaluation will accomplish. 

As you work to define your evaluation’s purpose and scope, you should also consider:

Whether your funder has any specific evaluation requirements that must be fulfilled. This means that if you are doing an evaluation to fulfill a program requirement set forth by your funder, you want to make sure you understand what exactly those requirements as they will likely drive the purpose and scope of your evaluation. For example, AmeriCorps National Direct grantees and AmeriCorps State Competitive grantees that receive an average annual ASN grant of $500,000 or more must conduct an independent impact evaluation, designed to provide statistical evidence of the impact of the program compared to what would have happened in the absence of the program using a comparison or control group. AmeriCorps National Direct grantees and AmeriCorps State Competitive grantees with average grants of less than $500,000 are also required by ASN to conduct an evaluation. However, the requirement differs in that they may use an internal evaluator rather than an independent one. Another difference in the requirement is that although it is strongly encouraged by ASN to use the most rigorous evaluation design feasible, they are not required to conduct an impact evaluation that uses a comparison or control group. 

It is also important to take into consideration what resources (time, funds, expertise) are available to carry out the evaluation. Because most programs have limited resources that can be put towards an evaluation, it is important to note that it is not necessary to evaluate every aspect of your program as depicted in your logic model. Your evaluation can have a narrow focus (e.g., only address questions about one of your program’s service activities and desired outcomes) or it can have a broader focus (e.g., address questions about each of your program’s service activities and desired outcomes), depending on the information you hope to gain from your evaluation and the resources you have available. 

12

Step 2: Define Purpose and Scope

2021  |

  13

Facilitator notes: Turning back now to a logic model, on this slide we present an example of how your logic model can be used to help you focus your evaluation by narrowing in on the primary question or questions you want to address. A program logic model can help narrow down the vast array of questions you may want to answer about your program, by highlighting the connections between program components and outcomes.

The graphic above provides an example of the types of questions that may be asked of each component in a logic model: 

Questions related to inputs ask, “Are resources adequate to implement the program?” 

Questions related to activities ask, “Are activities delivered as intended?” 

Questions related to outputs ask, “How many, how much was produced?” 

Questions related to outcomes ask, “What changes occurred as a result of the program?”

As you may recall from an earlier slide, a logic model has two “sides.” The process side represents a program’s implementation or its planned work and the outcomes side describes the expected sequence of changes that the program is to accomplish. Depending on which side of the model your question(s) of interest lies, will help you to decide what type of evaluation you will need to conduct – process, outcome, impact, or multiple types. We will talk more about the differences between these types of evaluation at a later point in the presentation.

As you can see at the bottom of this graphic, in order to answer each of these questions, indicators, which are the evidence or information that represents the phenomenon in question, and their data sources will need to be identified. When developing research questions based on your program’s logic model, ensure that questions are stated in a way that is clear and measurable in order for them to be answerable. Again, we will talk more about this at a later point in the presentation.

13

Exercise

The hypothetical veterans program is designed to address unemployment among veterans and their spouses as well as their transition into civilian work and community life. 

Using the logic model developed for the veterans program, what might be some potential research questions? 

Group Exercise: Develop Research Questions for a Veterans Job Readiness Program

2021  |

  14

Facilitator notes: Now that we have presented examples of the types of questions that may be asked for each logic model component, let’s apply these ideas to developing research questions for the hypothetical AmeriCorps veterans program. As you’ll recall, the program is designed to address unemployment and among veterans and their spouses as well as their transition into civilian work and community life. For this exercise, we will develop research questions as a group and have participants use the example logic model to come up with potential research questions. We encourage all of you to participate and provide your input as we develop research questions for this program together. 

Please use the handouts that have been provided on the program’s description and its accompanying logic model as a reference. 

Beginning with the left side of the logic model, a process evaluation asks the broad question, “Is the program being implemented as designed?” What are some potential research questions that we can pose with respect to inputs, activities, and outputs? (At this point, the facilitator may wish to return to the previous slide so participants can see the types of questions that are asked and apply them to the veterans program logic model handout.)

Examples:

Inputs:  	Were resources adequate to implement program activities?

What is the budget for this program? 

How many staff and volunteers does the program have?

Activities:	Were education and outreach activities delivered as intended?

	Is the program running efficiently?  

Outputs:	How many veterans participated in the job readiness workshops?

		Is the program reaching its target population?

		Who is the program serving?

Moving to the right side of the logic model, an outcomes evaluation asks the broad question, “What difference has the program made?” What are some potential research questions that we can pose with respect to short-, medium-, and long-term outcomes?

Examples:

Short-term:		Did veterans demonstrate an improvement in their job readiness skills?

Do employers understand the strengths and benefits of hiring veterans?

Medium-term:	Was there an increase in the number of veterans finding jobs?

Do families adopt more proactive coping strategies to manage the transition from military to civilian life?

Long-term:		Were veterans able to maintain their jobs over the long-term? 

Are families better off?   

Next, we can consider an impact evaluation, which asks the broad question, “What difference has the program made for a treatment group (those who receive an intervention) versus a comparison or control group (those who do not receive the intervention)?” Impact designs include a comparison or control group that are similar to the program group, confirmed via baseline equivalency. What are some potential research questions that we can pose with respect to an impact evaluation?

Examples:

Short-term:	Did veterans who participated in the program demonstrate an improvement in their job readiness skills compared to a control 				group of veterans who did not participate in the program?

Do employers understand the strengths and benefits of hiring veterans?

Medium-term:	Was there an increase in the number of veterans finding jobs who participated in the program versus a control group of veterans who 			will receive the program during the next session?

Do families adopt more proactive coping strategies to manage the transition from military to civilian life among the treatment group versus the comparison group?

Long-term:		Were veterans who participated in the program able to maintain their jobs over the long-term in comparison to a control group of 				veterans who did not patriciate in the program? 

This exercise was intended to help you think through program evaluation questions in terms of the logic model components. To recap, a process evaluation provides information that helps you improve your program and mainly focuses on inputs, activities, and outputs. An outcome evaluation provides information that can be used to demonstrate the results of your program on participants and the community. It focuses on the program’s short-, medium-, and long-term outcomes. Similarly, an impact evaluation focuses demonstrating the results of your program on participants and the community through comparing outcomes for a treatment and comparison or control group.

14

Common cost categories:

Staff time 

Materials, equipment, and supplies

Travel

Data collection

Step 3: Budget for an Evaluation

2021  |

  15

Facilitator notes: The third step in the planning phase involves budgeting for an evaluation. The cost of evaluations varies widely and will depend on the type of study design, the size of the study, the level of expertise and experience of the evaluator, and data collection expenses. 	

Other common considerations for creating a program evaluation budget are: staff time; materials, equipment, and supplies; travel costs; and data collection.  With respect to this last item, evaluations involving more primary data collection tend to be more expensive than those that rely on existing internal program records or external data sources. This is not a comprehensive list of cost consideration. Depending on the program to be evaluated and/or the actual evaluation activities, there may be additional expenses required. 

15

Consider questions of:

Who will conduct it?

If external evaluator, consider what services are and are not included in their cost

If own staff, consider cost of time spent on evaluation relative to programmatic tasks

What will it include and how will it be conducted?

Will it involve new data collection? 

If so, at what time points and where?

Who will manage it? 

Step 3: Budget for an evaluation

2021  |

  16

Facilitator notes: In determining what resources are needed to conduct the evaluation, you should consider, “Who will conduct the evaluation?”, whether it will be an external evaluator or a member of the program staff. If it will be a member of the program staff, you should anticipate the extra hours it will take to complete evaluation activities. 

Some other considerations in estimating resource needs are: 

“What will the evaluation include?” 

“How will it be conducted?” 

“Will the evaluation involve new data collection?” If so, at what time points will data be collected and where will the data collection take place?”

Even when an external evaluator is hired, organizations must also invest staff time in managing an evaluation. Just as you would monitor your program to ensure that it is on track and running smoothly, you want to have staff responsible for monitoring that your evaluation is moving forward as planned. This may require regular meetings (e.g., weekly, monthly, quarterly) with the evaluator to check in on the current status of the evaluation, progress made, whether there have been any setbacks or challenges that need attention, any resource needs, etc. Having program staff invested in the evaluation process ensures that an informed and well-planned evaluation will be produced. 

16

An evaluator is an individual or team of people responsible for leading the evaluation.

Potential options for an evaluator include:

An external source (e.g., consulting firm, college or university personnel, independent consultant)

An internal source - program staff member(s)

Step 4: Select an Evaluator

2021  |

  17

Facilitator notes: We turn now to step 4, selecting an evaluator. An evaluator is an individual or a team of people who are deemed responsible for leading the program evaluation. Your evaluator might be an external source – an individual or a team of people you hire, such as a consulting firm, college or university personnel or an independent consultant. Alternatively, you may decide to use an internal source such as one or more of your program staff members.

17

A key decision is whether to use an internal staff member or to rely on an external evaluator. 

Consider what expertise is needed

Scope and complexity may demand expertise outside of your program staff

Check your funder’s requirements 

Funders may require an external evaluation to ensure objectivity (e.g., Required for AmeriCorps ASN grantees receiving >$500,000 annually)

Consider financial and staff time implications

Both types of evaluations have cost implications, whether financial or staff time

Step 4: Select an Evaluator

2021  |

  18

Facilitator notes: Determining whether to use your own internal staff or to rely on an external evaluator to complete an evaluation of your program is a critical decision that will need to be made early on in the planning phase of your evaluation. Generally speaking, the following are key considerations when determining whether to employ an external evaluator/team for an evaluation:

Expertise: Some programs may not have internal staff with the type of technical expertise required to conduct an evaluation. Therefore, depending on the scope and complexity of the evaluation, an individual or team who is external to the program may be needed. This is particularly the case for impact designs in which there are more rigorous analyses (e.g., power analysis, baseline equivalency testing, etc.). It is considered best practice to use an external evaluator for impact evaluations as it helps provide credibility to the study findings; it also helps reduce the likelihood of bias compared to when a program staff person conducts this evaluation. For AmeriCorps State and National grantees, however, large grantees receiving more than $500,000 annually, are required to use an external evaluator. 

Funder requirements: Funders may require that the programs they fund conduct an external evaluation to ensure objectivity in the evaluation’s results. For example, as you may already know, AmeriCorps grantees receiving annual funds of less than $500,000 are NOT required to conduct an external evaluation. However, AmeriCorps grantees receiving annual funds of $500,000 or more are required to conduct an external evaluation.

Cost: Both internal and external evaluations have cost implications. While using internal program staff to complete an evaluation of your program may be less costly, it also adds to staff workload. It is important to consider this trade-off and whether staff have the capacity to take on additional work to carry out the evaluation activities. Where personnel resources are more scarce than funding, an external evaluator may be the better choice.

18

Certain types of AmeriCorps grantees (i.e., AmeriCorps State and National grantees) may be interested in achieving a specific evidence tier.

Preliminary evidence- internal or external

Moderate evidence- external

Strong evidence- external

Consider taking a hybrid approach

Program and external evaluator share evaluation responsibilities

For more information about these requirements and evidence levels, visit: https://americorps.gov/grantees-sponsors/evaluation-resources

Step 4: Select an Evaluator

2021  |

  19

Facilitator notes: Another consideration is evidence tier. 

Evidence Tier: Some AmeriCorps grantees, specifically AmeriCorps State and National grantees, are required to use either internal or external evaluators in order to achieve certain evidence levels when their evaluations are reviewed as part of the Evidence Review Process that occurs at the end of their grant cycle. Internal or external-> preliminary, but to achieve the higher levels of evidence – Moderate or Strong – an external evaluator must be used in addition to conducting an impact evaluation. For more information about these requirements and evidence levels, please review the information in the Mandatory Supplement which is part of the funding announcement for AmeriCorps State and National Grantees. The 2023 version can be accessed here: https://americorps.gov/funding-opportunity/fy-2023-americorps-state-national-grants.

Hybrid approach: Another option that programs may consider is to take a hybrid approach by hiring an external evaluator to support the more technical aspects of the evaluation and have internal program staff carry out the non-technical aspects of the evaluation. For example, you may hire an external evaluator to design the evaluation, identify or develop your data collection instruments and processes, analyze the data, and write up the evaluation findings. Program staff may play a role in the evaluation by helping to collect data (e.g., administering surveys or tests) and enter or process the data to be used by the external evaluator. This may both help to defray costs and ensure that program staff will remain actively involved in the evaluation process. 

19

How do you find an external evaluator?    

Academic settings

Contact individuals at your local college or university who are not affiliated with the program

Professional settings

American Evaluation Association (AEA) website, click on “Find an Evaluator” tab (https://www.eval.org)

Personal Networks

Ask your funder

Step 4: Select an Evaluator

2021  |

  20

Facilitator notes: How does a program go about identifying an external evaluator? There are several possibilities.  

First, programs can look to academic settings.  Evaluators can often be found in academic entities such as colleges or universities. Formal evaluation is similar to research, and in many cases, evaluators are formally trained in research design and statistics. When looking for an evaluator in an academic setting, you may start by contacting the department that works with your field (e.g., social work, sociology, education or public administration), or see if there’s an evaluation office on campus. 

Second, there are professional evaluation or research firms, as well as individuals who work as independent consultants. Ask other organizations in your community who they have used and recommend working with. Check the American Evaluation Association (AEA) website as it contains a database of individuals who are available for evaluation consulting or to serve on evaluation teams due to specific expertise in particular methodologies. You can search for a listing of individuals in your geographic area and/or your desired area of expertise. 

Personal networks, including other agencies in your community that have used external evaluators. These agencies may be able to recommend a good evaluator, suggest methods for advertising, and provide other useful information.  

Lastly, programs can ask their funders for ideas.  Some large foundations and government agencies have lists of evaluators with expertise in their field. Or you can call agencies who have been funded previously and ask them who they worked with. If a particular name emerges as having worked with the grantees of a particular funding agency, that firm could add credibility to your proposal.

20

Consider whether your potential evaluator has -   

Formal training in evaluation studies

Experience evaluating similar programs/interventions 

Experience that matches the design, methods, and/or approach of your planned evaluation 

Capacity to handle the scale of your planned evaluation

Personal style that fits your program staff or organization

Ability to conduct an objective and unbiased evaluation (e.g., no conflicts of interest, nor appearance of conflicts).

Experience working with/evaluating AmeriCorps programs specifically 

Step 4: Select an Evaluator

2021  |

  21

Facilitator notes: Once you have identified potential evaluators, you will want to ensure that you choose an evaluator that has the capacity to understand your program and the information you expect to gain from your evaluation. Your evaluator must also possess the skills and experience needed to conduct a high quality evaluation. 

When selecting an evaluator or evaluation team, consider whether your potential evaluator has: 

Formal training in evaluation

Experience evaluating similar programs or interventions and experience that matches the type of design, methods, and/or approach of your planned evaluation. You want to find out how much evaluation work they have done, especially in projects similar in content and approach to yours.

Capacity to handle the scale and size of your planned evaluation. Likewise, you want to find out the extent of evaluation work they have done that is of similar scope and size to yours. 

A personal style that fits with your program staff or organization. Make sure that the evaluator can communicate with your staff. If the evaluator only describes the process or approach they will take using highly technical jargon, it may be difficult for your staff to work with them. It may also be a sign that their products/reports will also be highly technical, and hard to interpret and implement recommended changes.

The ability to conduct an objective and unbiased evaluation (e.g., no conflicts of interest, nor appearance of conflicts). The individual is not associated with the program. They can not have a formal or personal relationship with, or stake in, the administration, management, or finances of the grantee or of the program being evaluated

Grantees should consider an evaluator who has experience working with/evaluating AmeriCorps programs, specifically.

Before hiring an evaluator, make sure to interview prospective evaluators, look at their past work (e.g., request a recent report or sample of their work), ask for names of past clients and check their references. Lastly, make sure you feel comfortable that he, she, or the team is credible, competent, and capable of leading an evaluation process that meets your program’s needs and interests. 

21

Development Step: Developing an Evaluation Plan

2021  |

  22

Development

Develop an Evaluation Plan

Facilitator notes: Once the planning phase has been completed, the next phase is the development phase where your evaluation team will develop an evaluation plan.

22

Details the program model being evaluated

Describes and justifies the evaluation approach selected

Provides instructions for the evaluation / a guide for each step of the evaluation process

What is an Evaluation Plan?

2021  |

  23

Facilitator notes: An evaluation plan does the following:

Details the program model being evaluated. Grantees should review their logic model and theory of change to determine which component or components of their program model they would like to evaluate. It is important to note that grantees do not need to evaluate every component of their program. Rather, you can evaluate one component or element of your program, and then focus on another element in a future evaluation. 

Describes and justifies the evaluation approach selected. The plan should include information about why a certain design and approach was selected. For instance, if you are implementing a new program or activity to your program, a grantee may want to pursue a process evaluation to determine whether this new component is being implemented as intended. This justification would be included in the plan so that AmeriCorps and the evaluator understands why this design was selected.

Provides instructions for the evaluation / a guide for each step of the evaluation process. An evaluation is a written document that states the objectives of the evaluation, the research questions that will be answered, and how you plan to answer those questions. An evaluation plan in many ways is like a detailed instructional manual that captures the overall picture of what your evaluation will address and the evaluation steps and activities that are needed to successfully carry out the evaluation.  It is also important to understand that an evaluation plan should be thought of as a living document, such that it is continually updated as you make progress and finalize decisions about what your evaluation will entail. It is never too early to start your evaluation plan. While you may only be able to describe a few pieces of information at the beginning stages of the evaluation planning process, your plan should be a dynamic tool to help you establish your evaluation goals and objectives, organize your evaluation activities, and determine how the work will get done. 

Additionally, as you may already know, re-compete grantees are required to submit a draft evaluation plan as part of their AmeriCorps application. The evaluation plan that is submitted at the time of application should provide sufficient detail on the evaluation objectives and study methods, as well as plans for identifying a qualified evaluator and an estimated budget. Grantees are then provided with feedback on how to obtain approval for their plan, which is needed within the first year of their grant cycle. Grantees can then work with an experienced evaluator, consultant, State Commission, or other staff familiar with evaluation to further develop and refine their evaluation plan post-award. Thus, it is important for grantee staff to engage an experienced evaluator (either an external evaluator or someone internally who has evaluation experience) early on after receiving their AmeriCorps award to ensure that the evaluator is involved in developing the evaluation plan from the start.

We will talk more on what exactly an effective evaluation plan should include on the next few slides. 

23

Helps decide what information is needed to address the evaluation objectives

Helps identify methods for getting the needed information

Helps determine a reasonable and realistic timeline for the evaluation

Creates a shared understanding between stakeholders (e.g., the grantee staff, evaluator, AmeriCorps staff)

Purpose of an Evaluation Plan

2021  |

  24

Facilitator notes: There are many reasons for developing a written evaluation plan. In addition to being a requirement for competitive AmeriCorps State and National (ASN) grantees in their second grant cycle, evaluation plans serve an important purpose. An evaluation plan:

Helps decide what information is needed to address the evaluation objectives

Helps identify the method or methods for getting the needed information and how to analyze it. It can also help you identify potential challenges for approaching the evaluation and how you may be able to address those challenges (e.g., what existing data sources can be used for the evaluation or are the resources available to carry out the type of evaluation we want to pursue). 

Helps determine a reasonable and realistic timeline for the evaluation. Having a detailed plan can lead to better efficiency in approaching each evaluation activities and maximizes your ability to successfully achieve your evaluation objectives.  

Creates a shared understanding of the purpose and use of evaluation results for program stakeholders (e.g., funder, staff, other organizations, community members), grantee staff, and external evaluator(s), if applicable. While an evaluation plan is not a formal contract, it is an opportunity for the grantee to clarify their expectations. 

24

Step 5: Develop an Evaluation Plan

2021  |

  25

What should your evaluation plan include?

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design

Sampling methods

Data collection procedures, data sources, and measurement tools

Analysis plan

Evaluator qualifications

Timeline

Budget

Facilitator notes: Based on AmeriCorps’s guidelines in the evaluation plan template located on their website and within the Notice of Funding Opportunity document for competitive grantees and on the Resources for Direct Grantees. An evaluation plan must include the following sections: 

I. Theory of change

II. Scope of the evaluation

III. Evaluation outcome(s) of interest

IV. Research questions

V. Evaluation design 

VI. Sampling methods

VII. Data collection

VIII. Analysis plan 

IX. Evaluator qualifications

X. Timeline

XI. Budget 

The outline presented on this slide can be found on the Notice of Funding Opportunity page. You can also find the template on the AmeriCorps website under the Resources for Direct Grantees section. Select the “Evaluate my program” action, then find the tile with the heading “Evaluation Plan Template.” (The actual direct link for this template will change over time as the template is updated, but the resource tile itself will not change.)

While there are 11 sections of an evaluation plan, we will highlight some of the major evaluation plan sections here in Step 5. Some of the other sections, such as data collection and analysis, will be addressed in later steps. 

25

I. Theory of Change

Describe how the activities undertaken by your program contribute to a chain of results that lead to the intended outcomes.

Step 5: Develop an Evaluation Plan

2021  |

  26

Theory of Change

elements

Program

context

Sequence of

required events

Underlying

assumptions

Logic

model

Short-term

outcomes

Intermediate

outcomes

Long-term

outcomes

Facilitator notes: Your program’s theory of change should describe how the activities undertaken by your program contribute to a chain of results that lead to the intended outcomes. It is also key that the theory of change aligns with your program’s logic model and helps set the stage for the evaluation. For instance, the focus or scope of the evaluation you intend to implement should be clearly connected to the theory of change. One theme that will occur throughout this discussion is the importance of aligning each component of the evaluation.

26

II. Scope of the Evaluation

State the goal(s) of the evaluation and specify which program activities will be assessed

Programs are not expected to evaluate every component of their logic model. Instead, the evaluation may focus on a sub-set of program activities.

Step 5: Develop an Evaluation Plan

2021  |

  27

Facilitator notes: For the scope of the evaluation, you’ll want to describe the goals for the evaluation, specifically what is it you want to learn or understand. The stated purpose of the evaluation drives the expectations and sets the boundaries for what the evaluation can and cannot deliver. The evaluation scope should also describe what aspect or aspects of your program your evaluation will focus on and over what time period. Generally speaking, the goals and the scope should align with both your theory of change and program logic model. 

Another key detail, is that programs are not expected to evaluate every component of their program’s logic model. It is more important to develop a quality evaluation that focuses on a few components of a program, and then use findings from that evaluation to explore other program activities. 

27

III. Outcome(s) of interest

Describe what outcomes your evaluation will measure

Process / implementation outcomes or outputs

Program beneficiary outcomes

Member outcomes  

Your outcomes of interest should be:

Part of your program’s theory of change and included in your logic model

Feasible for your program to measure given the source(s) of data needed and level of effort required

Step 5: Develop an Evaluation Plan

2021  |

  28

Facilitator notes: For non-experimental outcome evaluations, outcomes must involve a change in knowledge, attitude, behavior, or condition. Metrics that measure the amount of service provided (e.g. number of students tutored/volunteers recruited/organizations served) should not be listed as outcomes for this type of evaluation. However, for process or implementation evaluations, these metrics may be appropriate.

The outcomes identified should align with the outcomes described in the program's logic mode. Outcomes must also align with the theory of change, the scope of the evaluation, and must be feasible to measure, based on the source(s) of data needed and level of effort required. Given the length of a single grant cycle, it can be challenging to evaluate long-term program outcomes; evaluating short- and medium-term program outcomes may be more practical. 

28

IV. Research Questions

One or more questions that define exactly what your evaluation intends to accomplish

The following are characteristics of a good research question:

Clearly stated and specific

Aligns with your theory of change / logic model

Connect to the outcomes of interest

Measurable and feasible to answer 

Aligns with your chosen evaluation design

Step 5: Develop an Evaluation Plan 

2021  |

  29

Facilitator notes: There is no set number to how many research questions your evaluation should attempt to answer, however you want to be sure that it is feasible and you have the resources to answer each research question at the end of your evaluation. Thus, it is important that the research questions for your evaluation are stated in such a way that they:

Are clearly stated and specific

Align with your program’s theory of change and logic model. Your questions should match the objectives, activities, and outcomes of the program

Will lead to measurable or observable results

Aligns and is appropriate for the chosen evaluation design. For instance, research questions for an impact evaluation should focus on comparisons or changes between the program’s beneficiaries against individuals that are not part of the program and don’t directly benefit from their services. 

29

V. Evaluation Design 

Description of general categories of evaluation designs:

Step 5: Develop an evaluation plan	

2021  |

  30
| Process Evaluation                                                                                                                                                                                                                                                                  | Outcome Evaluation                                                                                                                                                                       | Impact Evaluation                                                                                                                                                                                                  |
|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Examines the extent to which a program is operating as intended by assessing ongoing program operations and determining whether the target population is being servedResults may be used to determine what changes and/or improvements should be made to the program’s operations   | Measures changes in knowledge, attitude(s), behavior(s) and/or condition(s) among program beneficiaries or other stakeholder groupsResults may demonstrate what the program has achieved | Measures changes in program beneficiaries or stakeholders relative to a reasonably similar comparison/ control groupResults are an estimate of the program’s impact on beneficiaries or other stakeholder groups   |

Facilitator notes: For AmeriCorps, there are three categories of evaluation designs: process evaluations (which looks at how the program is operating and helps identify areas of improvement); outcome evaluation (which measures changes among program beneficiaries and provides details on what the program has achieved); and impact evaluation (which measures and compared changes among program beneficiaries against a reasonably similar comparison or control group). 

Evaluation design is essentially the structure that will provide you the information needed to answer each of your evaluation questions. The appropriate design for evaluating a program will largely depend upon certain considerations. This may include, but is not limited to:  

Your program’s theory of change and logic model

Primary purpose of the evaluation and key research questions

Funder’s evaluation requirements: Many funders require their grantees to complete evaluations of their program. As you consider different evaluation designs, you should also have a clear understanding of whether your funder has any specific evaluation requirements that must be fulfilled. Those requirements may drive the purpose and scope of your evaluation. For example, it is important to note that AmeriCorps has evaluation requirements for large and small re-competing grantees in terms of which evaluation design they may use to assess their programs. Large grantees are those receiving annual AmeriCorps funds of $500,000 or more. Small grantees are those receiving annual AmeriCorps funds of less than $500,000. You should note which type of design is required for your program. As noted earlier, AmeriCorps State and National grantees may seek exemptions from impact designs or requirements if they meet the criteria set forth in the AEA guidance.

Resources available for the evaluation: It is also important to take into consideration what resources (staff time, funding, evaluation expertise) are available to carry out the evaluation. In particular, your evaluation team must possess the skills and experience needed to carry out the type of evaluation design that is chosen. Think about whether you will be able to collect the data required for a given design. More complex evaluation designs require you to collect more data over multiple measurement points. This lengthens the data collection phase of the study and may require more sophisticated analyses to be conducted once the data are collected.

30

Step 5: Develop an Evaluation Plan	

2021  |

  31

AmeriCorps Approved Evaluation Designs
| Type of Design                                        | Category | Details needed on evaluation design                                                                                                                                                                                                                                    |
|-------------------------------------------------------|----------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Experimental design/Randomized Controlled Trial (RCT) | Impact   | Description of the random assignment procedures that will be used to form treatment and control groupsEligibility criteria for inclusion in the study                                                                                                                  |
| Quasi-experimental Design (QED)                       | Impact   | Description of the approach for identifying a reasonably similar comparison group (e.g., propensity score matching, difference in difference analysis)   List of variables (covariates) to be used to statistically equate treatment and comparison groups at baseline |
| Non-experimental Design                               | Outcome  | Description of whether pre- AND post-test measurements OR post-only measurements will be used                                                                                                                                                                          |
| Process                                               | Process  | Description of the methods that will be used (i.e., qualitative only, quantitative only, or mixed methods)                                                                                                                                                             |

Step 5: Develop an Evaluation Plan	

2021  |

  32

Alternative Evaluation Approach (AEA)
| AEA                            | Grantee        | Justification                                                                                                                                                                                                                                                                                                                                   |
|--------------------------------|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Funding threshold              | Large          | - Grantees who receive an average of less than $1 million per year can request to be exempt from the large grantee requirements and conduct an internal non-impact evaluation.                                                                                                                                                                  |
| Previous impact evaluation     | Large          | - Previously conducted an impact evaluation with demonstrated evidence of effectiveness (i.e., Strong or Moderate evidence).                                                                                                                                                                                                                    |
| AmeriCorps National Evaluation | Large or Small | - Grantees participating in an AmeriCorps’ Office of Research and Evaluation national evaluation (i.e., bundled evaluation or Return on Investment) that will not be completed during current grant cycle - Large grantees can also request this if the national evaluation’s design does not fulfill the requirements for a large grantee.     |

Facilitator notes: If needed, grantees may submit an alternative evaluation approach (AEA) justification form to request various adjustments in their evaluation plan due to particular circumstances. 

On these next two slides, we will discuss the different AEA options available to large, as well as small, grantees. The AEA options allow grantees to pursue a different design (for large grantees only) or extend the timeline of their evaluation (for both large and small grantees). These options include, from easiest to hardest to achieve: 

Funding Threshold: Large grantees may request this option if the total awarded federal share of their budget averages under $1 million per year for the full life of the grant. An external evaluator is NOT required for completing a new evaluation. 

Previous Impact Evaluation: Large grantees that conducted a previous impact evaluation of the same program that achieved a Moderate or Strong evidence rating may qualify for this exemption. An external evaluator would still be required for completing a new evaluation. 

AmeriCorps National Evaluation: This option may be requested by large or small grantees that are participating in an AmeriCorps evaluation (i.e., bundled evaluation or Return on Investment) that will not be completed during the current grant cycle and for which findings will not be available until a later date. Grantees may also request this option if the design of the AmeriCorps evaluation does not fulfill the evaluation requirements for a large grantee. 

32

Step 5: Develop an evaluation plan	

2021  |

  33

Alternative Evaluation Approach (AEA) - continued
| AEA                                          | Grantee        | Justification                                                                                                                                                         |
|----------------------------------------------|----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Structure of program or grantee organization | Large          | - Insurmountable challenges forming a comparison group. - Significant changes to program design.                                                                      |
| Replication                                  | Large          | - Implementing an evidence-based intervention with fidelity in a new setting. - A grantee’s application must be assessed at the Strong or Moderate evidence level.    |
| Timing                                       | Large or Small | - Evaluation will not be completed by end of current grant cycle. - AEA approval required only if an interim evaluation report will not meet evaluation requirements. |

Facilitator notes: AEA options continued.

Program Structure: This option may be requested if a large grantee can demonstrate insurmountable challenges to forming a comparison group, or it is not developmentally appropriate to conduct an impact evaluation due to significant changes to the program design. An external evaluator would still be required for completing a new evaluation.

Replication: Large grantees may request this option if they are implementing an evidence-based intervention (i.e., rigorously evaluated through an impact study, and the study obtained a Moderate or Strong evidence rating) with fidelity in a new setting. An external evaluator would still be required for completing a new evaluation.

Timing: Lastly, large or small grantees may request an AEA for timing if they anticipate that their evaluation will not be completed by the end of the current grant cycle. While this may allow grantees to extend the evaluation period, there are two important pieces that grantees should consider before pursuing this option: 1) grantees will be required to submit an implementation report with their next application submission, in lieu of an evaluation report, detailing the progress they’ve made to date on the evaluation; and 2) once the evaluation report is completed, it will not be considered for evidence review until their next grant cycle. That is, if a grantee is recompeting in 2024 and their evaluation report from their previous grant cycle will not be ready until August 2024, past the timeline for the application period, that report would not be available for evidence review until the next application cycle in 2027. Therefore, grantees should work with their AmeriCorps Portfolio Manager and/or State Commission before making this decision.   

33

Implementation Steps: Collecting and Analyzing Data

2021  |

  34

Implementation

Collect Data

Analyze Data

Facilitator notes: Once the evaluation plan has been completed, the next phase is the implementation phase where you will begin conducting the evaluation. These steps include collecting and analyzing data. It should be noted that this phase in the evaluation cycle can be very time-consuming. Programs need to allow sufficient time for data collection to ensure that data for a full program year are available and where pre/post measures are being used. In addition, programs that will be pre-testing instruments must also factor in this additional time.  

34

What type of data meets your evaluation needs?

Existing data (i.e., secondary data)

Internal program data (e.g., participant records, program logs, performance measurement data)

External datasets / administrative data (e.g., student records, test scores, medical records, test scores, Census data, unemployment insurance claims)

New data (i.e., primary data)

Data from surveys, assessments, interviews, and observations

Step 6: Collect Data

2021  |

  35

Facilitator notes: Data can be obtained in many different ways to address your evaluation needs. Programs often begin by first looking at the data already being collected to understand if existing data can adequately answer the evaluation’s research questions. Existing data (sometimes referred to as secondary data) may pertain to data already collected by the program itself such as performance measures or data that are gathered by external sources, such as administrative data. Administrative data may include student records, test scores, medical records, and Census data.

When existing data are not available to answer research questions, programs need to collect new data (sometimes referred to as primary data). Depending on the research questions being asked, there are several different types of data collection instruments that can be developed and administered to collect new data such as surveys and interviews. 

There are advantages and disadvantages to both existing and new data. Existing data can save resources because time, effort, and financial resources are not spent on collecting new data. The disadvantage, however, is that existing data may be difficult to access, and there may not be a perfect fit between what the evaluation is trying to measure and the purposes for which the existing data were collected. Moreover, it is often difficult to assess how reliable the data are, and how reliably they were recorded and documented. Although more costly, the advantage of using new data is that the evaluator has greater control over the measures, procedures, and implementation and documentation, which can contribute to higher data quality. 

Grantees can build on data that are collected as part of routine program operations, such as performance measures, for their evaluation. Your program may already be collecting data to monitor and report on your program’s progress toward achieving its expected outcomes for program beneficiaries. An example of a routine program activity from which you can gather data is the use of a sign-in log for any member trainings or workshops that are offered. You can use the sign-in log to track members’ attendance in training to measure consistency of attendance as an indicator of members’ commitment to development and learning. If your program decides to conduct an impact evaluation to learn whether your progress on outcomes is caused by your intervention, it may be that you continue to collect the SAME outcomes data, but also include a matched comparison group. Including this comparison group enables you to answer specific questions related to causality – in this case, what would have happened to people if they did not receive the intervention (i.e., whether the observed changes can be attributed to your intervention).

35

Step 6: Collect Data

2021  |

  36
|                 | Quantitative Methods                                                                                                                                                    | Qualitative Methods                                                                                                                                                                                                         |
|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Scope           | Less in-depth data across a larger number of study subjects                                                                                                             | More in-depth data on fewer study subjects                                                                                                                                                                                  |
| Data collection | Standardized instruments with mainly closed-ended questions  (i.e., questions with pre-defined response options) such as  surveys and multiple choice assessments/tests | Standardized instruments and semi-structured interview guides mainly with open-ended questions (i.e., questions with no pre-defined response options) and can be used for interview, focus group, and observation protocols |
| Data format     | Numeric                                                                                                                                                                 | Narrative                                                                                                                                                                                                                   |

Facilitator notes: As we discussed, data collection methods can be either quantitative or qualitative. The table that you see here provides an overview of these two types of methods, highlighting their key differences with regard to the following:

Scope – In general, quantitative methods capture less in-depth data on a larger number of study participants while qualitative methods generate more in-depth data on fewer study participants. This is because quantitative methods are often used to collect data that can be summarized across a larger number of cases (e.g., study participants, program sites) whereas qualitative methods are often used to explore or gain a deeper understanding of a particular subject or topic among a smaller number of cases. Qualitative methods tend to be more labor intensive and time-consuming than quantitative methods approaches, and thus concern fewer cases.

Data collection – Quantitative data collection methods are used to collect data using structured instruments such as a survey with closed-ended questions or items (e.g., questions with pre-defined response options or rating scales). Qualitative data collection methods are intended to collect narrative data using instruments such as surveys or interview protocols that contain open-ended items (e.g., questions that allow the respondent to provide a response in their own words). 

Data format – Quantitative methods yield numeric information, that is, data that can be summed up or counted. Qualitative methods, on the other hand, yield narrative data that provide important insights and context such as a description of the attitudes and perspectives of program beneficiaries. 

36

Step 7: Analyze Data

2021  |

  37
|               | Quantitative Methods                                                                                                 | Qualitative Methods                                                                                                                            |
|---------------|----------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|
| Data analysis | Statistical  approaches are used to summarize the data (frequencies, means, crosstabs, regression )                  | Content analysis is often used in which themes/patterns in the data are identified, categorized, coded, and summarized                         |
| Results       | Results can be compared, and generalized to a larger population; May  provide statistical evidence of program impact | Results  provide meaning, illustrative explanation, and views of study subject(s); NOT able to provide  statistical evidence of program impact |

Facilitator notes: Once the data have been collected, the information must be described, analyzed and interpreted. The steps for preparing the data for analysis and interpretation, however, differ depending on whether the data are quantitative or qualitative.  

For quantitative data, the statistical analyses specified in the evaluation plan are conducted. While we do not attempt to go into detail on the different statistical techniques that might be used for quantitative analysis, most evaluations rely on simple descriptive statistics – means, frequencies, etc. However, when more complex analyses and causal modeling are necessary (particularly in the case of impact evaluations), evaluators will need to use more sophisticated techniques such as analyses of variance, regression analysis, and so forth. This should be handled by an evaluator with expertise in quantitative analysis.

For qualitative data, content analysis is often used in which themes or patterns in the data are identified, categorized, coded, and then summarized. Various approaches are used to interpret meaning or themes from the content of text data. It is important to note that anecdotes or what we sometimes refer to as “great stories” are not the same as qualitative data. Anecdotes – or personal accounts, thoughts, or feelings – collected in an ad hoc fashion cannot tell us whether any improvements occurred in an intervention because no measurements were established. 

In terms of results, quantitative methods produce results that can be generalized to a larger population. In program evaluation, generalizability may be defined as the extent to which you are able to make conclusions about a larger population based on information you have collected from a sample or subset of that population. For example, let’s say a program develops a new curriculum for its workshops and it first tests the curriculum on a sample of low-income families in the community that are assumed to be representative of other families in the community that would be served by the program. If this assumption is correct, the outcomes associated with using the new workshop curriculum on the small number of families can be generalized to estimate the outcomes of using the curriculum on other families in the community who did not participate in the workshop. Quantitative methods are generally the only way to provide statistical evidence of program impact. Qualitative methods produce results that give meaning, illustrative explanation, and views of study subject(s). Results are typically not generalizable to a larger population and cannot provide statistical evidence of program impact. 

Most program evaluations will collect both quantitative data (numbers) and qualitative data (text) in a mixed methods design to produce a more complete understanding of a program – which is called triangulation. A combination of qualitative and quantitative data can improve a program evaluation by ensuring that the limitations of one type of data are balanced by the strengths of another. 

37

Step 7: Example Data Collection and Analysis Crosswalk

2021  |

  38
| Process Evaluation of a Job Readiness Program for Veterans  |                                                                                                            |                                                                                                                                |                                                                    |                                                                                                                                                    |                                                                                                                                                                                                                                                                                        |
|-------------------------------------------------------------|------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Research question                                           | Indicators                                                                                                 | What is collected and how?                                                                                                     | From whom / data sources?                                          | When collected and by whom?                                                                                                                        | How will you analyze the data?                                                                                                                                                                                                                                                         |
| Is the job readiness program being implemented as designed? | a) Member use of program curriculum during workshops b) Duration of workshopsc) Participant workshop rates | a - c) Members report details about workshops in logs with pre-defined categories of reportinga - b) observations of workshops | a - c) Members a - b) Evaluator observes participants in workshops | a - c) External evaluator collects the workshop logs quarterlya) Quarterly observations by the evaluator(s) using structured observation protocols | a - c) Generate frequencies on use of curriculum; average duration of workshops; and average rate of workshop attendancec) Generate frequencies and averages on quantitative data (e.g., ratings scales, frequency scales) and thematically code and analyze open-ended comments/notes |

Facilitator notes: At this point, we’d like to return to the hypothetical AmeriCorps job readiness program for veterans and provide you with an example crosswalk that can be created to help you think through the process for identifying the data collection and analysis for a given research question. For this example, we begin with a research question developed for a process evaluation of the hypothetical veterans program, followed by a research question developed for an impact evaluation.  

The hypothetical program is designed to combat veteran unemployment through a number of different activities. For simplicity, we focus here on the series of workshops delivered by the program.  

The main research question for the process evaluation being conducted for this program concerns whether the job readiness program is being implemented as designed.  Potential indicators for assessing fidelity to the program model include member use of program curriculum during workshops, the duration of workshops, and participant workshops rates. This information can be collected through member logs as well as evaluator observations of the workshops on a quarterly basis.  

Once the data have been gathered, simple descriptive statistics can be generated from the quantitative data such as frequencies on the use of the curriculum and averages on the duration of workshop and participant attendance rates. Meanwhile, qualitative data that have been collected may be thematically coded and analyzed. Taken together, analyses of all the collected data are then used to assess the extent to which the program was implemented as designed.   

38

Step 7: Example Data Collection and Analysis Crosswalk

2021  |

  39
| Impact Evaluation of a Job Readiness Program for Veterans                                                                                                                   |                               |                                                          |                                                                                                                                                                       |                                                                                                                                                      |                                                                                                                                                                                                                     |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------|----------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Research question                                                                                                                                                           | Outcome of interest           | What is collected and how?                               | From whom / data sources?                                                                                                                                             | When collected and by whom?                                                                                                                          | How will you analyze the data?                                                                                                                                                                                      |
| Do veterans who receive the job readiness intervention have a higher average employment rate compared to  a similar group of veterans who did not receive the intervention? | Veterans’ employment status   | Veterans’ employment status is measured with a survey.   | Veterans participating in the program serves as the intervention group.Veterans receiving job assistance services from another program serve as the comparison group. | The evaluator administers the survey at two time points:before the job readiness program begins1 year after the job readiness program is implemented | Calculate the difference in average employment rate in the intervention group minus the difference in average employment rate in the comparison group before and after treatment (difference in differences method) |

Facilitator notes: In this example, we present a research question developed for an impact evaluation of the hypothetical veterans program.

The main research question whether veterans who receive the job readiness intervention have a higher average employment rate compared to a similar group of veterans who did not receive the intervention.  The outcome of interest is veterans’ employment status. One way to collect this information is through surveys. The information, furthermore, must be collected not only on participants but also on an identified comparison group for an impact evaluation. In this example, the comparison group pertains to veterans who are receiving job assistance services through another program. Alternatively, programs may also look within their own program for a comparison group. For example, an impact evaluation may compare participants receiving core as well as supplemental services against participants receiving only core services.  

Once the intervention and comparison groups have been identified, the evaluator will collect the data at two time points. In this example, data will be collected both before the job readiness program begins and a year after the program has been implemented for both the intervention and comparison groups.    

After the data have been gathered, statistical tests (in this case, difference-in-differences methods) are then used to compare program participants with their matched comparison group by subtracting the average outcome (gain) in the comparison group from the average outcome (gain) in the intervention group. Such analyses may show that, on average, veterans participating in the program are more likely to be employed and remain employed than veterans participating in other job assistance programs. It is important to note that the statistical techniques and methods used in an impact evaluation can be sophisticated (involving for example, propensity score matching, identification and inclusion of covariates, etc.) and it is important to have a qualified external evaluator conduct the analyses.   

39

Consider two questions:

What conclusions about the research questions can be drawn from the data that have been analyzed?

What does the data suggest about the program’s theory of change?

Step 7: Analyze Data

2021  |

  40

Facilitator notes: When analyzing the data, it is important to address two key areas: “What conclusions about the research questions can be drawn from the data that have been analyzed?” and  “What does the data suggest about the program’s theory of change?” Does the theory of change require any modifications to better reflect how the program is functioning. 

For the hypothetical veterans program, findings from the process evaluation may reveal that members are faithfully delivering the content of the curriculum during the workshops and that participation in the workshops among veterans remains consistent and high. Based on the data that were collected and analyzed, we would conclude that this particular component of the program faithfully adheres to the program model. 

40

Action and Improvement Steps: Reporting and Utilizing Results

2021  |

  41

Action and Improvement

Communicate Findings

Apply Findings and Feedback for Program Improvement

Facilitator notes: The final phase in the evaluation process includes steps for communicating findings to specific audiences and applying findings for program improvement. The final Action and Improvement Phase is meant to encourage programs to share findings with others to continue to build the knowledge base in “What Works” for AmeriCorps programs and to apply evaluation findings for program improvement.

41

Who are the potential target audiences? 

Program staff, agency personnel, stakeholders, beneficiaries, funders, etc. 

What are potential tools for communicating findings? 

Formal report, shorter memos, PowerPoint briefings, etc. 

Step 8: Communicate Findings

2021  |

  42

Facilitator notes: Before determining how to communicate your findings, first it is important to identify the potential target audience for your findings and the potential tools for communicating these findings. How you communicate your findings and insights will depend on the purpose of the report and your intended audience. 

In writing the final report, it should be a key consideration who the main audience will be for the report. In general, it is best to anticipate the report will be viewed by a wider audience, not just AmeriCorps and program staff. Then the report can be most useful for communicating study findings to outside stakeholders. When writing for a broader audience, the study description and reporting of results should be provided using less technical language, and the more technical aspects of the study should be moved to an appendix. Also, enough detailed information should be provided so that outside parties can gain a full understanding of the program and the evaluation processes. In the instance where the report is mainly intended for internal program use, then the potential audience may be less of a concern and the report can be written to a narrower audience. 

42

AmeriCorps uses evaluation reports to:

Identify training and technical assistance needs for grantees

Identify and share promising practices and models for replication

Strengthen the evidence base for national service

Grantees are encouraged to use their evaluation report to:

Create awareness of and demonstrate success (or lessons learned), and promote sustainability

Identify opportunities for program improvement, adjustment, and future action

Guide the development of an evaluation plan for upcoming grant cycle

Support the evidence base for future grant applications

Step 8: Communicate Findings

2021  |

  43

Facilitator notes: AmeriCorps uses evaluation reports to:

Identify training and technical assistance needs for grantees

Identify and share promising practices and models for replication

Strengthen the evidence base for national service

Grantees are encouraged to use their evaluation report to:

Create awareness of and demonstrate success (or lessons learned), and promote sustainability

Identify opportunities for program improvement, adjustment, and future action

Guide the development of an evaluation plan for upcoming grant cycle

Support the evidence base for future grant applications

43

What is an evaluation report?

A written document that presents the evaluation’s methods and findings 

Provides a transparent basis for: 

Understanding the program’s alignment with its theory of change

Decision-making on policies and programs 

Drawing lessons for program improvement

Step 8: Communicate Findings

2021  |

  44

Facilitator notes: An evaluation report is a written product that objectively describes all of the steps involved in completing the evaluation, including study findings, conclusions, and lessons learned. While the evaluator generally leads the report writing process, several different types of people may be involved in developing and/or reviewing the final report. For example, the program background section is best provided and/or reviewed by program staff, while descriptions requiring more technical explanation, such as evaluation methods, procedures and study limitations, are best written by the program evaluator. Conclusions and recommendations also should be written by the evaluator, so a more independent assessment of the findings is provided. However, lessons learned, including implications for program modification or improvement, are best determined through discussions involving a wider group of program stakeholders, including program staff, funders and program beneficiaries, as appropriate. Questions for future research should be identified through a collaborative approach involving program staff, the evaluator, and other program stakeholders.

An evaluation report also provides a basis for further discussion on:

Understanding the program’s accountability to its theory of change

Decision-making on policies and programs

Drawing lessons for program improvement 

44

Evaluation reporting 

(Summarize and interpret results)

Evaluation planning 

(Define the purpose and  methods to be used)

Evaluation implementation

(Collect and analyze data)

Step 8: Communicate Findings

2021  |

  45

What are the key sections of an evaluation report?

Facilitator notes:  On this slide, we present the key sections commonly included in an evaluation report. We note that you can use some of the information from your evaluation plan to populate the evaluation report. For example, the background and purpose and evaluation methods can be pulled from the plan and enhanced with a level of detail appropriate for the report.

45

Executive Summary

Conclusions & Recommendations

References & Appendices

Background and Purpose

Evaluation Methods

Results/Findings

When reporting findings, it is important to: 

Report positive, as well as negative findings

Present results that are not necessarily conclusive, but show promise and warrant further examination

Be careful not to overstate your findings

Step 8: Communicate Findings

2021  |

  46

Facilitator notes: There is much to be learned from what did work as well as from what did not. Importantly, reporting negative findings does not mean that the evaluation and/or the program failed. It is important to communicate the evidence generated and make changes to programs based on the evaluation findings. CNCS expects grantees to make improvements to their programs in response to evaluation findings. This is a key reason for conducting program evaluation.

Some interventions have a very limited evidence base and any new information greatly increases knowledge in the field. 

Lastly, it is important to communicate the results of your evaluation in the context of your particular program and not generalize too broadly.   

We will provide an example of when negative findings can be useful for program improvement later in the presentation. 

46

Tips to consider: 

Develop an outline for your report before starting the writing process 

Assume the reader has no prior knowledge 

Make the information visually appealing and easy to read 

Use diagrams, graphs or charts to highlight central findings

Take steps to ensure the credibility of your evaluation report

Proofread your report

Step 8: Communicate Findings

2021  |

  47

Also, we have a few general tips for writing an evaluation report and some additional tips for ensuring that your report meets CNCS evaluation requirements. 

Develop an outline for your report before starting the writing process 

Assume the reader has no prior knowledge of your program, the evaluation, or its findings, and provide a thorough description of all three of these areas in the report

It always helps to make the information visually appealing and easy to read (Use professionals, such as a graphic design expert to format your report, if possible)

Similar to short research briefs, Use diagrams, graphs or charts to highlight central findings and put secondary findings in appendices

Take steps to ensure the credibility of your evaluation report. 

Whether commissioning an external evaluation or conducting one internally, credibility is important to build within an evaluation report. Objectivity in evaluations provide integrity and believability to findings. Because your program has an interest in seeing positive results, findings should be presented in as objective a way as possible to ensure the credibility of your evaluation report.  So in your evaluation report, discuss the limitations of your study, including those related to methodology and implementation. Don’t shy away from discussing negative or null findings. Finally, hold recommendations until the end of the report. This allows the reader to come to their own conclusions.

And please proofread your report! This is always an important idea to ensure readability, completeness, and grammatic accuracy. 

47

Other Useful Products for Communication:

Executive summary of final report (5-10 pages)

Short research briefs (2-4 pages)

Graphics and pictures

Bulleted information

Non-technical memos

Step 8: Communicate Findings

2021  |

  48

As mentioned previously, it is important that evaluation findings be disseminated to a wider audience, not just CNCS and program staff. However, a detailed final report is not always the best means for clearly communicating program findings to outside stakeholders. Shorter, less technical documents can be useful for disseminating evaluation findings, especially when developed in tandem with the final evaluation report. An executive summary to the final report is often useful for communicating the most pertinent details of an evaluation without overwhelming potential readers. Also, short research briefs, which contain mainly graphics, pictures, and bulleted information on the evaluation and its findings, can be useful for summarizing evaluation findings in a much more digestible format. Research briefs, which can be as simple as a one page, front-and-back document, can also be more easily distributed at program events and in combination with other program materials. 

For communicating additional or more targeted information beyond that found in the final report, such as implications on program planning or future evaluation plans, non-technical memos can be particularly useful for communicating information.

48

Evaluation findings can support decisions and actions with respect to:

Program design, implementation and effectiveness

Program improvement

Implementing change

Step 9: Apply Findings and Feedback for Program Improvement

2021  |

  49

Facilitator notes: As mentioned earlier in the presentation, ultimately an evaluation must provide usable information that can directly inform program decision-making.

We can use the veterans program as an example of when negative findings can be useful for program improvement. Say a process evaluation of the program is conducted, and it finds that the program is successful in training and placing several veterans in new jobs. However, the most common reason reported by program participants for later leaving a new job placement was a lack of initial support for them and their families to relocate to new communities. What the evaluation found was that many veterans took jobs in new locations, but then, due to moving costs and a lack of familiarity with local services, family members had to wait to join them in their new communities. This unanticipated separation from family was a key reason why veterans tended to voluntarily leave new jobs and return back to their original communities. In response to these findings, the program reported that they have decided to add more transitional and support services for the families of recently placed veterans, so they can more quickly join them in their new communities.  

49

Please visit ASN’s Evaluation Resources for more information:

https://www.nationalservice.gov/resources/evaluation/evaluation-resources

ASN’s Evaluation Resources include course on many topics, such as: 

Developing a Logic Model

Developing Research Questions

Designing an Evaluation

Managing an External Evaluation

AmeriCorps Evaluation TA Portal: 

https://americorpsevaluationta.norc.org/

Resources on Evaluation

2021  |

  50

Facilitator notes: For more information on evaluation, please go to the ASN’s Evaluation Resources webpage located here. 

ASN’s Evaluation Resources include course on many topics, such as: 

Developing a Logic Model

Designing an Evaluation

Managing an External Evaluation

Developing Research Questions

A link to the TA Assistance Portal is also provided on this slide: 

https://americorpsevaluationta.norc.org/

50

Questions?

Facilitator notes: Does anyone have any questions?

51

Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and Evaluation: evaluation@cns.gov

Thank you!

# image2.emf

# image5.png

# image9.jpeg

# image10.jpeg

# image3.png

# image4.emf

# image1.png

# image6.png