Laying the Groundwork Before Your First Evaluation

2021  |

1

AmeriCorps and NORC at the University of Chicago

1

Understand five critical activities for your first grant cycle

Recognize how these activities lay the foundation for future evaluation work

Learn how to plan ahead to ensure that your program achieves key first-cycle milestones

Learning Objectives

-This is a new presentation that we haven’t yet presented as part of our evaluation core curriculum, so you’re in for a treat!

-The learning objectives are for you to:

	-Understand five critical activities for your first grant cycle

	-Recognize how these activities lay the foundation for future evaluation work

	-And learn how to plan ahead to ensure that your program achieves key first-cycle milestones.

AmeriCorps Template 2013

2

Do everything you can to set your program up for success in Year 1 

Think through risks/threats to implementation and mitigate them

Begin to develop data collection systems

Begin to develop performance measurement plans

Pre-Award Time Period

-Pre-award time period,  6-12 months before the program is expected to start.  

-This is the time for you to do everything you can to set your program up for success in its first year. 

-We understand that there will be variation in how programs use this “planning year” – some programs already exist and are just adding AmeriCorps members; some programs are in formula and can make refinements well in advance to set themselves up for success in competitive; some programs actually have planning grants (either commission formula or tribal grantees); some programs are brand new and may have varying levels of resources they can devote to planning for a grant they may or may not get, but if they can devote any time/resources to this, they should, and they should definitely use their time wisely from the time they learn they are funded to when they will begin enrolling members.

-Regardless, the planning period is the time to think through risks/threats to implementation and mitigate them.

-Since programs are expected to begin collecting data immediately in year 1, they should develop their data collection systems and performance measurement plans during the planning period to the extent that is realistic for them.

AmeriCorps Template 2013

3

Program design and implementation

Build and refine data collection systems

Performance measurement

Staff capacity and responsibilities

Evaluation planning

Become a learning organization

Key Foundational Elements

-We’re going to discuss these six activities: (1) program design and implementation, (2) building and refining data collection systems, (3) performance measurement, (4) staff capacity and responsibilities, (5) evaluation planning, and (6) becoming a learning organization. 

-The activities we will discuss today are foundational and should not be afterthoughts in the first three years.

-It is important to budget money and staff time in the first 3 years for the activities we will be outlining in this workshop, just as you would later budget for evaluation activities. 

AmeriCorps Template 2013

4

Your First Three Years: Program Activities Over Time

-All of these activities should be occurring during all three years of the first grant cycle, as shown on this slide. 

-How much you focus on each will obviously ebb and flow over time, but it’s important to keep focused on all of them from the start.

-You want to set yourself up for success in the first grant cycle and beyond. Don’t wait until it’s too late to put these foundational elements in place!

AmeriCorps Template 2013

5

Building evidence of effectiveness

Building Evidence of Effectiveness

Stage 1:Identify a strong program design

Stage 5:Attain causal evidence of positive program outcomes

Stage 3:Assess program outcomes

Stage 2:Ensure effective implementation

Stage 4:Obtain evidence of positive program outcomes

Evidence Informed

Evidence Based

-As an agency, AmeriCorps continues to invest in a portfolio of programs reflecting a range of evidence, from evidence-informed to evidence-based. The diagram on this slide illustrates AmeriCorps’s approach to building evidence which emphasizes that evidence of effectiveness is built over time and falls along a continuum. Evidence should be appropriate for a program’s life cycle and investment of public dollars. This diagram shows different stages to situate a program’s cumulative body of evidence along an evidence continuum. Having a long-term research agenda will help programs progress from stage to stage along the evidence continuum in a resource efficient manner. Programs are not expected to follow the same linear path to building evidence along the continuum. Some programs may accumulate evidence from output performance measurement activities (stage 2) to having causal evidence of effectiveness from an RCT/QED (stage 5). Others may move from collecting output performance measurement data (stage 2) to having pre/post outcome data (stage 3). Programs may also go back and forth between stages as they refine and adjust their program models over time. 

The key stages that a program goes through as they build their evidence base are shown in the diagram. 

Stage 1: Identify a strong program design

Stage 2: Ensure effective implementation

Stage 3: Assess program outcomes 

Stage 4: Obtain evidence of positive program outcomes

Stage 5: Attain causal evidence of positive program outcomes

-It’s important to note that these stages are not mutually exclusive and they do overlap, so consider the continuum to be a guide.  

-For more information check out our new Evidence Continuum video on the Evaluation Resources page!

-The activities we discuss today will mostly fit into Stage 1 and Stage 2.

6

Goal: Improve program design and ensure effective implementation

Key components

Refine logic model

Assess implementation

Conduct process evaluation 

Adjust program model as needed

1. Program Design And Implementation

-First, we will discuss program design and implementation.  The goal of this activity is to improve your program design and ensure that your program is being implemented effectively. 

-This means refining your logic model, assessing implementation, possibly conducting a more formal process evaluation, and using the resulting data to adjust your program model as needed.

AmeriCorps Template 2013

7

Make sure your “model” is actually a model

Achieve shared agreement about what the model is and is not

Ensure all pieces of the model are plausibly connected

Refine Your Logic Model

-Your program’s logic model is much more than just filling out the boxes.  The logic model should clearly explain all the inputs and activities involved in running your program, and it should clearly lay out the outcomes you expect to achieve. 

-A well-defined program model is specific and complete – it leaves little room for question or interpretation about how your program operates and what it intends to achieve. 

-It is critical to understand (and minimize) variation in implementation unless there is a good reason for it. 

-The weaker or less well-defined the program model is, the more difficult it is to ensure effective implementation.  A poorly defined program model is also very difficult to evaluate down the road. 

-As you refine your logic model, you want to discuss it with all relevant stakeholders and ensure that everyone is on the same page about what the program model is, and what it isn’t.

-You also want to ensure that all the pieces of the model are connected, and connected in a plausible and achievable way.

AmeriCorps Template 2013

8

Assessing Your Theory Of Change
|                                                                                                                                                                                                                                                                                                                                                                                               | Indicate to what extent each statement is true. |               |      |                |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------|---------------|------|----------------|
|                                                                                                                                                                                                                                                                                                                                                                                               | Not at all true                                 | Somewhat true | True | Not applicable |
| Theory of Change                                                                                                                                                                                                                                                                                                                                                                              |                                                 |               |      |                |
| There is a coherent, logical program theory. Strategies and activities are designed to address a clearly identified and defined problem or need. There is a logical connection between the program strategies and activities and the intended outcomes or desired changes. Goals and objectives are articulated and attainable with the available resources. (The program has a logic model.) |                                                |              |     |               |
| Program participation is clearly defined and distinguishable from nonparticipation. There is no ambiguity about who is in the program and who is not.                                                                                                                                                                                                                                         |                                                |              |     |               |
| There is a shared understanding among program leadership and staff about the core elements of the program and the context in which the program operates.                                                                                                                                                                                                                                      |                                                |              |     |               |
| There is agreement across the program leadership and staff as to what the expected program outcomes are.                                                                                                                                                                                                                                                                                      |                                                |              |     |               |
| Clear Time Frame for the Program                                                                                                                                                                                                                                                                                                                                                              |                                                 |               |      |                |
| The intervention has a clearly defined timeframe.                                                                                                                                                                                                                                                                                                                                             |                                                |              |     |               |
| There is a reasonable and shared expectation around the timeframe for when observable/measurable outcomes in the short, intermediate or long term will occur.                                                                                                                                                                                                                                 |                                                |              |     |               |

Impact Evaluability Assessment Tool. Source: https://www.americorps.gov/sites/default/files/document/2015_09_03_ImpactEvaluabilityAssessmentTool_ORE.pdf, “Conducting a Needs or Readiness Assessment”. 

-One of your handouts is the Impact Evaluability Assessment Tool, which we will discuss in more depth later in the presentation. Here we have highlighted the sections of the tool related to the theory of change and the program’s timeframe. This is on page 3 of your handout.

-This tool can serve as a guide as you refine your logic model.  Think about the questions posed here: 

	-Is there a coherent, logical program theory? 

	-Are strategies and activities logically connected to the intended outcomes and desired changes?

	-Is program participation clearly defined? 

	-Is there a shared understanding among all stakeholders about the core elements of the program and how it 

	operates, and what the expected outcomes are?

	-Does the intervention have a clearly defined timeframe? 

	-Do all stakeholders have reasonable, shared expectations about when outcomes will occur? 

AmeriCorps Template 2013

9

Implementing your program may lead to changes:

In your logic model

Some parts may not be plausible

Some outcomes may not be realistic

In how your program operates

Application logic model may be ideal  program model may change as it is implemented

Important to document those changes so that you understand what you are actually doing

Document any variation in implementation

Assess Implementation

-It is ok if implementing the program causes you to learn that some parts of your model are not plausible or some outcomes may not be realistic. 

-For example, say that as you implement your program and collect data on that implementation process, and the data show that beneficiaries require a more intensive intervention (more one-on-one time with members, for instance) than originally expected.  Your program may decide to spend more time serving fewer people than you originally proposed in your logic model, in order to achieve the desired comes. 

-The logic model you submit in your application may be your best/ideal version. Know that that the model may change as it is implemented, and it is important to document those changes because you have to evaluate what you are actually doing

- For example, suppose that member time sheets show that sites are providing more training than expected.  The program analyzes the data and learns that sites are having to provide follow-up training to address concepts that members did not master during their initial orientation.  The training component of the program model can be changed to reflect that--either the program can standardize additional training across all sites or change what is offered at orientation.

-If there is variation in implementation that cannot be mitigated for whatever reason, it is important that you document that. Later on, when you plan an evaluation of your program, you will need to understand where and how implementation may be different and account for it in the evaluation design.

AmeriCorps Template 2013

10

Assessing Implementation
|                                                                                                                                                                                                                                                                                                                                                                                                                              | Indicate to what extent each statement is true. |               |               |                |
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------|---------------|---------------|----------------|
|                                                                                                                                                                                                                                                                                                                                                                                                                              | Not at all true                                 | Somewhat true | True          | Not applicable |
| Program Implementation                                                                                                                                                                                                                                                                                                                                                                                                       |                                                 |               |               |                |
| If the program is based on a model or logical program theory, it is implemented with fidelity to that model and has a well-planned sequence of activities.                                                                                                                                                                                                                                                                   |                                                |              |              |               |
| If the program is currently being adapted, it is being adapted using theory/systematically-obtained field-based knowledge, and along lines that can be quantified and documented.                                                                                                                                                                                                                                            |                                                |              |              |               |
| Staff members are qualified and properly trained to operate the program. There are enough qualified staff members on site to implement the planned activities.                                                                                                                                                                                                                                                               |                                                |              |              |               |
| Frontline workers who deliver the services provided by the program have sufficient qualifications to execute the program. There are enough qualified frontline workers on site to successfully execute the program.                                                                                                                                                                                                          |                                                |              |              |               |
| There are systems in place to track program implementation:There are procedures in place to determine if the target population is being served (referral system, intake process).Data that track service usage is collected (attendance lists, case management logs). Input is sought on a regular basis to understand how participants experience the services and to identify and address any problems in a timely manner. |                                              |            |            |             |

Impact Evaluability Assessment Tool. Source: http://www.nationalservice.gov/resources/evaluation/planning-evaluation, “Conducting a Needs or Readiness Assessment”. 

-Here again is another portion of the Impact Evaluability Assessment Tool, on page 4 of your handout, this time the section related to program implementation.  As you assess implementation of your program, ask these questions: 

	-Is the program implemented with fidelity to the logic model? 

	-If the program is being adapted or revised, is that based on systematic data, and can the changes be documented?

	-Are staff and members qualified and properly trained? 

	-Are there enough staff and members to successfully implement the program?

	-Are you tracking service provision and service usage using appropriate data collection methods?

AmeriCorps Template 2013

11

Goal: Build systems to collect and manage high-quality program data

Key components

Select data collection instruments

Build data collection system

Build data management system

Obtain access to administrative data (if applicable)

Ensure data quality

2. Build and Refine Data Collection Systems

-Next, we’ll talk about building and refining data collection systems.  

-From day one (or even before day one!) it is critically important that you have systems in place to collect and manage high-quality program data. 

-You’ll need to figure out how you want to collect data, build data collection systems, put a system in place to manage the data, figure out how you will access other administrative data (if applicable), and in all cases ensure that the data you collect is high-quality. 

AmeriCorps Template 2013

12

First, examine logic model and think through data you need to collect

Outputs

Outcomes

Implementation

Other data beyond performance measures

Then, develop and test instruments

Utilize existing resources

Work with a professional to develop instruments that are valid and reliable

Good instruments can also be used later for evaluation!

Data Collection System

-First, you should look at your logic model and think through what data you need to collect—including data about outputs, outcomes, and implementation processes.

-In addition to output/outcome performance measurement data, you should be tracking implementation data.  This includes output data other than beneficiaries served – for example, data about dosage/delivery of the intervention, data that makes it possible to ensure unduplicated counts, track multiple interventions provided to the same individual; demographic data; privacy concerns, etc.

-Where do you start?  Think about how you want to be able access, analyze, and use the data and by look at your logic model.  Begin with the end in mind.

-Instrument development: don’t just wing it. We have resources online about data collection for both performance measurement and evaluation. There are lots of high-quality instruments out there that you can use, and in most cases you will not have to create your own from scratch.  You may want to work with an expert to develop instruments. 

-Designing valid and reliable instruments means that in many cases they can also be used later for evaluation. 

AmeriCorps Template 2013

13

Data management system should help you to: 

Store data

Access data

Analyze data

Use data

Build (or buy) system to accomplish those goals

Examples include: Efforts to Outcomes, Design Data, Outcomes Results System...

Data Management System

-You also need a data management system to store your data and allow you to access it and use it for program improvement, reporting, and decision making. 

-This can be either a very costly endeavor or not.

-Low cost tools include Google docs, Microsoft Excel, and other widely available resources.

-Excel even has pretty sophisticated data analysis options that will accomplish most of your analytic needs at this stage.

-Eventually you will likely want to invest in a more sophisticated option, especially as your data storage needs increase and you require more complicated analytic tools.

-Talk to other grantees about systems that have worked well for them.

-We put some examples up here, but there are many options out there.

AmeriCorps Template 2013

14

Elements of data quality: 

Valid: data mean what they are supposed to mean

Complete: everyone reports full set of data

Consistent: everyone uses same data collection methods

Accurate: data are free from errors

Verifiable: everyone follows standard practices and checks their data

Ensuring data quality means that you describe your program’s achievements in a trustworthy manner

Ensuring Data Quality 

-You need to ensure that the data you collect is high-quality. This means the data are:

-Valid: the data mean what they are supposed to mean

-Complete: everyone is reporting a full set of data

-Consistent: everyone uses the same data collection methods so that the same data are collected, in the same way

-Accurate: the data are free from errors

-Verifiable: Everyone follows the same standard practices and checks their data

-Data quality is very important. It means that you are describing your program’s achievements in a trustworthy manner. 

-Check out Sarah Yue’s data quality course for more information

AmeriCorps Template 2013

15

Goal: Collect and report performance measures; use results for program improvement

Key components

Select performance measures

Determine measurement frequency

Adjust performance measures

Execute performance management processes

Connect performance management to evaluation 

3. Performance Measurement

-All grantees are required to collect and report performance measures to AmeriCorps, but we really want you to be using performance measures data for program improvement. 

-Performance measures are not a compliance exercise – they should be useful data that will help your program accomplish its goals.

-You will need to select appropriate performance measures, determine how often you will measure, adjust the measures over time as needed, use performance measurement data for program management, and eventually connect performance measurement and management processes to evaluation. 

AmeriCorps Template 2013

16

Select measures that reflect what your program is actually doing

Select measures you can use to understand how your program is working

Select targets that are ambitious yet achievable 

Adjust measures annually if needed

Measure Selection and Adjustment

-We know that many grantees think of performance measurement as just collecting data on the output(s)/outcome(s) in their grant, but really good performance measurement is much more than that.

-Especially with regard to implementation, you should start thinking about what other performance measures you need to be tracking: member enrollment/retention (already should be tracking for compliance purposes), member training outputs/outcomes, member supervision milestones, data on how the intervention is being implemented.  

-Or, you may be tracking key performance measures but don’t know how that data can be used to understand and strengthen implementation, which is what we’ll talk about in the next slide.

-You want to set targets that are ambitious yet achievable, and you may need to adjust your measures or your targets over time. 

AmeriCorps Template 2013

17

Measures you collect and report for AmeriCorps grant

Other measures

Good performance measurement system!

Do not just collect and report your performance measurement data – you need to use it!

Are goals being met?

Use data for program improvement

Continuous feedback loop

Programmatic decisions are

    grounded in data  

Strengthen implementation

Performance Measures Core Curriculum: http://www.nationalservice.gov/resources/performance-measurement

Performance Management

-You should not just be collecting and reporting performance measurement data to AmeriCorps – you need to use that data for program improvement.

-In order to achieve your program goals, you need to know if your goals are being met. One way to do that is by looking at performance measures data.  

-Good measures can help you understand how your program is working. For example, let’s say that you want to understand more about program implementation, specifically member training. You could create a performance measure that looks at member knowledge gains at pre-service and in-service training sessions. These data can help you understand a key component of your program, member training, specifically how it is implemented. If targets are not achieved on this measure, you will know that member training is not being implemented as designed, and you should work to improve it so that targets are met.  

-Performance management should be a continuous feedback loop, where you measure, analyze, identify improvements, make programmatic decisions that are grounded in data, and strengthen implementation as needed. 

-For more information on performance measures, check out the Performance Measures Core Curriculum on the nationalservice.gov Resources page.

AmeriCorps Template 2013

18

Select measures and set targets

Collect performance measures data

Analyze data 

Identify programmatic improvements

Strengthen implementation

AmeriCorps housing assistance program

O5: Number of economically disadvantaged individuals, including homeless individuals, receiving housing services [Target = 500 calls]

O11: Number of economically disadvantaged individuals, including homeless individuals, transitioned into safe, healthy, affordable housing [Target = 600]

Actuals: O5 = 900, O11 = 300

What’s going on? 

Individuals are calling the hotline multiple times

Placement process takes longer than anticipated

Program response? 

Adjust call process – create a second number for repeat clients

Adjust target for O11 to reflect reality of placement process

Performance Management Example

 -Here’s an example of good performance management. A moderate sized AmeriCorps program operates a housing assistance hotline, where those in need of shelter can be connected with available housing units. 

-The program selected the following Economic Opportunity performance measures: O5 (Number of economically disadvantaged individuals, including homeless individuals, receiving housing services) and O11 (Number of economically disadvantaged individuals, including homeless individuals, transitioned into safe, healthy, affordable housing).

-They set a target of fielding 500 calls during the project year, and placing 600 individuals in housing. At the end of the project year, the program reviews its data and discovers that they have exceeded their call target for O5 by 400 calls, fielding 900 calls during the project year. Additionally, they discover that they have only placed 300 individuals in housing.

-Reviewing the member call logs, used to track the number of calls fielded for O5, they find that individuals have called the hotline multiple times in order to secure their placement in housing. The program had anticipated that placement would be completed in the course of one call. Further, the program finds that the lower number of placements than the target of 600 for O11 reflects the fact that the placement process took longer than anticipated, requiring multiple touch points between the member and the client. 

-In response, the program uses this data to make program improvements. First, the program will make adjustments to the call intake process to allow clients to call back multiple times through a separate number, preserving the hotline for first time callers and allowing for triage of cases. In addition, the program will readjust their target for O11 to more accurately reflect the time intensiveness of the placement process and demand for services.

AmeriCorps Template 2013

19

Good performance measurement can be the basis for future evaluation activities

Outcome measures may be extended

Instruments can be utilized 

Data can suggest interesting research questions

Performance measurement does not stop once evaluation begins

Connection to Evaluation

-Performance measures should be collected on an ongoing basis and compared to a target level, whereas an evaluation is designed to answer a specific research question about the goals of the intervention. 

-Despite these differences, the two activities are closely linked and should be aligned. 

-Well chosen performance measures can be the basis for future evaluation activities; for example, outcome measures can be extended or expanded upon, data collection instruments used for performance measures can often be used or adapted for evaluation, and reflecting on performance measures data can help when considering research questions for an evaluation.

-Importantly, performance measurement should be ongoing, all the time, every year. It does not stop once you begin your first evaluation. 

AmeriCorps Template 2013

20

Goal: Build staff capacity to collect, manage, and utilize data

Key components

Develop internal staff skills

Assign responsibilities 

Engage external experts

Become a learning organization

4. Staff Capacity and Responsibilities 

-From day one, you need to be thinking about how you can build your staff’s capacity to collect, manage, and utilize data.

-We acknowledge that this may require a change in both individual mindset – because not a lot of people go into national service because they want to measure things – and in organizational culture.

-However, the sooner you make staff development in this area a priority, the better you will fare down the line, and not just with AmeriCorps. This is a core competency for 21st century managers. 

-Not only do you need to develop staff skills in these areas, but you need to assign responsibilities and sometimes engage external experts for more specialized or technical tasks. 

-We will touch on this last point, becoming a learning organization, toward the end of the presentation.

AmeriCorps Template 2013

21

Develop staff skills in data collection, management, and analysis

Know enough to know what you know, and know what you don’t know

Staff do not have to be experts, but they need to be educated and informed consumers

Assign responsibilities – someone has to take ownership of key tasks

Internal Staff Capacity

-We don’t expect staff to be experts, but they should be reasonably capable.  Staff need to know how to collect high-quality data and manage that data, and at least one person on staff should have basic analytic skills.

-It’s important for program managers to know what they know and what they don’t know, and what their staff know and don’t know. That can help you figure out in which areas you need to seek outside expertise, or where you can target staff development activities and resources. 

-It’s also important that responsibilities for data tasks are clearly defined and assigned. Key tasks need owners so that nothing is missed or overlooked.

-Staff can check out the Evaluation and Performance Measurement Core Curricula, both available online on the Resources page.

AmeriCorps Template 2013

22

Utilize external experts for

Data collection instruments

Complicated analysis

Questions about measurement

Evaluation 

Consider consultants, pro bono experts, cooperative extension, universities

Talk to other programs and stakeholders

Identify and Work with External Experts

-For some technical or complicated data tasks, you’ll want to engage external experts.  

-Experts can be particularly helpful when you are designing data collection instruments, conducting complicated analyses, or when you have questions about measurement or evaluation. 

-Tap into all available resources. Think about finding expertise through cooperative extension programs or universities, volunteer pro bono experts, your board, or consultants. 

-Talk to other programs in your network – tap their brains, see where they have sought out help.

AmeriCorps Template 2013

23

This checklist is a tool designed to help organizations assess and discuss a program’s readiness to participate in a rigorous impact evaluation. It can be used for assessment, planning, and communication purposes. 

The focus areas covered in this checklist are organized into three broad content categories: 

Organization Readiness 

Program Readiness

Evaluation Readiness 

Each category addresses key elements of readiness. Readiness in all three areas covered is important for successful planning and implementation of an evaluation.

Exercise: Evaluability Assessment Checklist

-Now we’re going to take a few minutes to have you start filling out the Impact Evaluability Assessment Tool.

-You can use this tool as a starting point/planning tool – we don’t expert you to have all the answers now, and some of the pages will only be relevant down the road when you start planning an evaluation. 

-You could do the assessment annually, identify growth areas to focus on, make a plan, and assign responsibility for implementation of the plan, and assess again at the end of each year.  

-There is a lot in here, so for now we will highlight some of the areas of focus that would be likely to be the best use of time for programs in first three years: theory of change, and implementation.

-Theory of change is on page 3

-Implementation is on page 4.

AmeriCorps Template 2013

24

Goal: Develop a plan for your first evaluation, which will occur in your second grant cycle (if not earlier).

Key components

Select research questions

Budget for evaluation 

Develop evaluation plan

Make design decisions (process or impact, internal or external) based on grantee size

5. Evaluation Planning

Facilitator notes: To recap, we’ve covered the first of four foundational elements, and now we’ll cover the remaining two: evaluation planning and becoming a learning org. 

-For AmeriCorps State and National grantees, the first year of your competitive grant cycle should be used for developing an evaluation plan, which will be executed in the second year of your grant cycle. However, you can start slightly earlier if you want. It is also important to note that the evaluation requirement applies to grantees in their second grant cycle. Grantees in their very first grant cycle are not expected to have an evaluation plan, but may develop one in preparation for their second grant cycle. 

Planning process has a few core components

Select research questions. These should be questions that you are interested in learning about the program. They should also align with your program’s logic model and theory of change. 

Budget for evaluation 

Develop evaluation plan. This would need to be developed prior to your next grant cycle. You can develop this on your own, in consultation with an evaluator, or with other stakeholders such as your State Commission (if relevant).

Make design decisions. It is important to note that the design selected may be pre-determined by the size of your program. That is, size is based on the average level of funding received each year from AmeriCorps during your three-year grant cycle. If you receive less than $500,000 on average per year, you are considered a small grantee and thus have flexibility in the design option you choose. However, if you received more than $500,000 on average per year, you would be considered large and would be required to conduct an impact evaluation. There are ways for large grantees to be exempt from this requirement (i.e., Alternative Evaluation Approach (AEA)). More information about evaluation designs and AEAs are discussed in more detail in the “How to Write an Evaluation Plan” and “Overview of Evaluation Designs” slides on the Evaluation Resources page on the AmeriCorps website. 

-We will provide an overview of each of these components in this section of the presentation, however, there are also courses available on the Evaluation Resources page that will dive into these topics in much more detail. 

AmeriCorps Template 2013

25

Research questions are a list of questions grantees want to learn about their program

Research questions should be: 

Clear, specific, and well-defined

Focus on a program or program component

Measureable by the evaluation

Appropriate for the design proposed

Aligned with your logic model

Develop Research Questions

One of the fist steps taken as you plan any evaluation is to develop research questions

Research questions are a list of questions that you anticipate being able to answer at the end of the evaluation. 

They focus the evaluation and provide important parameters for the evaluation design

Research questions should test part of your TOC and map back to your LM. 

There are a few things to remember when creating your RQs:

Clear, specific, and well-defined so they can be thoroughly answered after completing the other evaluation activities (e.g., instrument development, data collection, analysis, etc.). 

Also, need to focus on a program or program component. 

As AC grantees, make sure your eval focuses on the AC component of your program, in order for it to meet our evaluation requirements. 

It’s also important that your RQs be measurable by the evaluation, because they eventually need to be answered by your evaluation activities. That is, you should confirm that the data you collect for the evaluation will answer each research question.

You will need to ensure that the research questions are appropriate for the design proposed. For instance, an impact evaluation includes a comparison or control group. Therefore, the research question should be phrased such that its clear that changes in the program beneficiaries will be compared to the changes experienced by a group not receiving the program’s services. 

Finally, RQs need to be aligned with your program’s logic model.

The research questions development process is a great time to engage stakeholders and gives a chance to get buy in and increase a feeling of ownership in the evaluation. 

Creating research questions that follow these guidelines sets your evaluation up for success and helps manage internal and external expectations for the evaluation. If you would like more guidance about developing research questions, you may review the “Developing Research Questions for Evaluation” slides on the Evaluation Resources page on the AmeriCorps website. 

AmeriCorps Template 2013

26

In general, evaluation budgets should be: 

Commensurate with stakeholder expectations and involvement 

Appropriate for the research design used and key questions to be answered 

Adequate for ensuring quality and rigor 

In line with the level of program and organizational resources available 

Budgeting for evaluation is a smart strategic investment

Budget For Evaluation

Facilitator notes: In general, evaluation budgets should:

Reflect the expectations of stakeholders, particularly in terms of scope, duration, and level of rigor of the evaluation. Any requirements or mandated components as a condition of funding are also important here. For example, AmeriCorps State and National grantees receiving over $500k are required to conduct an external impact evaluation covering at least one program year. The use of an external evaluator and the methods needed to implement a comparison or control group design, for example, both bring their own particular cost requirements. So you can see that design requirements, scope of the evaluation, and who will be conducting the evaluation will affect the final budget. The evaluation budget should also reflect any stakeholder investment, be it financial or in another form such as time or technical assistance.

Evaluation budgets should also be appropriate for the research design used and the key research questions you want to answer. Certain evaluation methods or techniques, like primary data collection, simply cost more to execute than others. Relatedly, some key research questions will require different levels of investment, depending on their scope and the depth with which you want to answer them. 

Budgets should also be of sufficient size to ensure that your evaluation will be high quality and rigorous. Underfunding may result in design choices that jeopardize the integrity of your evaluation. It can also lead to last minute workarounds, shortcuts, and quick fixes taken in the midst of your evaluation that can seriously impact the results produced. 

Finally, an evaluation budget should reflect the financial, human capital, and other resources your program or organization has. For example, consider what kind of data collection systems you currently have in place and what you can plan to build for the future.

-Evaluation can provide critical insight into your program. We feel strongly that program evaluation goes beyond meeting requirements and provides important information for program management, decision-making, and improvement. We view evaluation as an smart strategic investment in improving your program, and ultimately, as a stepping stone to serving more beneficiaries more effectively. 

AmeriCorps Template 2013

27

Written document that details all he evaluation steps and activities you plan to conduct

Dynamic tool (i.e., a living document) that should be continually updated as you plan and develop each aspect of the evaluation  

Evaluation Plan Outline

Theory of change

Scope of the evaluation

Evaluation outcome(s) of interest

Research questions

Evaluation design 

Sampling methods

Data collection

Analysis plan 

Evaluator qualifications

Timeline

Budget 

Write An Evaluation Plan

-A key activity in eval planning is formalizing the details of your evaluation in an evaluation plan.

-An evaluation plan is a written document that provides detail on all the evaluation’s steps and activities.

	- Can be rel. short, a few pgs., or quite long, depending on the level of detail needed/desired. 

	- Important to record all info here b/c plan serves as a guide AND an accountability tool. Can be v. important when working with ext. eval or stakeholders.

Dynamic tool that is continually updated as the evaluation is planned and developed. Keep in mind that you can start a draft plan as soon as you like, steps are not linear. However, AmeriCorps State and National grantees are expected to obtain approval of their plan by the end of their first year of their grant cycle. Any significant changes made to the evaluation plan may need to undergo an additional round of review and approval. 

An evaluation plan should include the following items. [read slide]

I. Theory of change

II. Scope of the evaluation

III. Evaluation outcome(s) of interest

IV. Research questions

V. Evaluation design 

VI. Sampling methods

VII. Data collection

VIII. Analysis plan 

IX. Evaluator qualifications

X. Timeline

XI. Budget 

-Again, we have an entire Evaluation Core Curriculum class on developing an evaluation plan, so this slide just briefly shows the key components of a well-developed evaluation plan.

AmeriCorps Template 2013

28

A learning organization:

Reflects on past successes and challenges

Uses data to inform decision making

Makes adjustments to programs and processes based on data

Is not afraid to question assumptions

Thinks strategically about how to improve

Builds evidence of effectiveness

Sets a research agenda for the future

6. Becoming A Learning Organization

Last foundational element to be aware of in 1st 3 years: the concept of learning organizations. Start to implement steps to becoming one.

A learning organization is an organization that creates, acquires, and transfers knowledge, and modifies its behavior to reflect new insights.

Remain relevant, effective b/c can continuously improve their programs based on data.

Data needed to continuously improve are readily available through performance measurement and program evaluation. Ideally you are doing both.

By engaging in perf. Msmt and strategic program evaluation, you can collect the data you need to 

Remain competitive in a scarce funding environment, 

Justify increases in scale, scope, and reach, 

Demonstrate accountability to your stakeholders

Build an evidence base demonstrating program efficacy

The key takeaway here is that by doing PM and eval, you are enhancing your capacity to make data driven decisions, and hopefully to improve your services. 

AmeriCorps Template 2013

29

By the end of your first grant cycle, you should have:

Refined your program and ensured effective implementation

Built and refined data collection systems

Utilized accurate performance measures

Built staff capacity and defined responsibilities

Prepared a plan for your first evaluation

Become a learning organization

And hopefully you’ve also had some fun!

Milestones

-We’ve covered the 6 foundational elements, and to recap, by the end of your first grant cycle you should have met these key milestones:

	-Refined your program and ensured it is being implemented effectively

	-Built, tested, and refined your data collection systems

	-Collected high-quality performance measure data and used that data for program improvement 

	-Develop staff capacity and assigned responsibilities for data tasks

	-Prepared your first evaluation plan

	-And begun the process of becoming a learning organization

If you’ve been able to make good progress on these foundational elements, you’ll be set up for success in coming grant cycles. 

AmeriCorps Template 2013

30

Exercise: Laying the Foundation for Your Program

For this exercise, we are assuming you are all in your first grant cycle, at some point (or you have just seen the light and want to get up to speed on eval). 

Take the next 10-15 min. to go through this worksheet and write down 2-3 activities that you’ve done  or are planning to do in this grant cycle for each of the 5 elements listed here

The idea is that this worksheet can be a roadmap for you to follow. 

If you’re a brand new grantee in your first year, be aspirational. Think about what you want to do over the next 3 years, and how you want to go about that. 

If you’re in your 2nd or 3rd year, write down what you may already be doing currently, then think about the next year or two. 

AmeriCorps Template 2013

31

Go to the Evaluation Resources page for more information:

https://americorps.gov/grantees-sponsors/evaluation-resources

Other courses available:

How to Develop a Program Logic Model

Overview of Evaluation Designs

How to Write an Evaluation Plan

Budgeting for Evaluation

Data Collection for Evaluation

Asking the Right Research Questions

Managing an External Evaluation

And more!

Resources on Evaluation

32

-For more information on evaluation, please go to the National Service Evaluation Resources page

32

Questions?

AmeriCorps Template 2013

33

To contact the Office of Research and Evaluation: evaluation@cns.gov

Thank you!

# image2.emf

# image5.png

# image9.png

# image10.png

# image11.png

# image3.png

# image4.emf

# image1.png

# image6.png