NORC at the University of Chicago

Developing a Long-Term Research Agenda

2023  |

1

1

Facilitator notes: This presentation is about developing a long-term research agenda for your AmeriCorps program. 

By the end of this presentation, you will be able to:

Recognize the importance of building a long-term research agenda

Identify the various stages in building evidence of a program’s effectiveness

Understand the key questions to consider prior to developing a long-term research agenda for your program

Learning Objectives

Facilitator notes: For this presentation, we have identified a number of learning objectives.

By the end of this presentation, you will be able to:

Recognize the importance of building a long-term research agenda

Identify the various stages in building evidence of a program’s effectiveness

Understand the key questions to consider prior to developing a long-term research agenda for your program

2

PART 1Defining a Long-term Research Agenda

Facilitator notes: We begin this presentation with an overview of what a long-term research agenda is, why it is important to develop one, and the relationship between a long-term research agenda and a program’s evidence of effectiveness.

3

A long-term research agenda is a series of intentional or planned program evaluations and research tools that build towards addressing a research goal.

Similar to a strategic plan, a research agenda generally spans over several years.

A research agenda is unique and should be tailored to each individual program.

A research agenda is a dynamic tool (i.e., a living document) that should be revised/updated based on new evidence, shifts in program direction, etc.

What is a Long-term Research Agenda?

Facilitator notes:  

A long-term research agenda is a series of intentional or planned program evaluations and research tools that progressively build towards answering a broad research question or research goal. As each evaluation activity is completed, the findings should be used to support and inform future evaluation activities and goals.

Your program’s research agenda should be similar to its long-term strategic plan, in that your program’s research agenda should include your planned research activities over several future years.

A number of unique factors help to determine a program’s long-term research agenda; therefore, a program’s research agenda should be tailored to the program’s goals, structure, resources, and an organization’s strategic plan. 

A long-term research agenda is a dynamic tool (i.e., a living document) that should be revised or updated based on new evidence, shifts in program direction, or other factors.  

4

Long-term Research Agenda

Research Goal

Program evaluation

Instrument and data collection systems development

Performance measurement

Facilitator notes: Here you see this idea illustrated. 

-The vertical axis shows capacity – as a program develops more capacity, the types of evaluations it implements, and the research tools it uses can become more sophisticated. The horizontal axis shows time – as a program becomes more established and mature, the number and scale of the research activities it can conduct increases. 

-Certain activities, like performance measurement, should happen at all times and from the very beginning of a program’s lifespan. Data collection systems should be developed and utilized from the very beginning. 

-Other activities develop over time, as a program builds capacity and becomes more established. Data collection systems are refined and become more sophisticated, and eventually a program begins to complete formal program evaluation studies.

-This picture also shows that many activities can and should occur simultaneously, no matter how well-established a program is or how much capacity it has. Performance measurement is ongoing, and program evaluation is not simply a one-time task. 

-All these various activities should be aligned and build upon one another in service of a program’s research goal or goals. 

5

A research agenda sets clear goals for what program stakeholders want or need to know about the program years into the future

A research agenda defines your destination, then identifies the supporting steps that will get you there

A research agenda continues to build evidence of program effectiveness

A research agenda demonstrates strategic investment of funds in evaluation activities 

Why is it Important to Have a Long-term Research Agenda?

Facilitator notes: There are a number of reasons why it is beneficial to develop a long-term research agenda for your program. 

A research agenda sets clear goals for what program stakeholders want or need to know about the program years into the future. AmeriCorps does not expect grantees to evaluate their entire program at once, but we recommend that grantees conduct a series of evaluation activities over time. This will help to more strategically commit resources for evaluation by designing studies that build upon one another and by creating complementary evaluation tools and data collection systems that can be deployed multiple times. 

A research agenda defines your destination and then identifies the supporting steps that will get you there. It is important to first lay a foundation for evaluation activities through the completion of important early, developmental work. 

A long-term research agenda should be designed to continue to build evidence of the program’s effectiveness over time. Furthermore, as additional evaluations of the program are designed and planned, the findings from previous evaluations should inform these new efforts.

A research agenda demonstrates strategic investment of funds. Planning evaluation activities such that they comprise a program’s long-term research agenda allows a program to maximize its resources for evaluation and provide information that will benefit its overall development.

6

What does a long-term research agenda look like?  

What do we want to have learned 5 years from now? 10 years from now?

Work backwards: define your destination, then name the supporting steps that will get you there

Each evaluation should build on what you learned previously

If you invest evaluation money strategically, scarce resources can have a big impact

Build a Long-term Research Agenda

7

Facilitator notes: As you begin to build your long-term research agenda, ask program staff and various stakeholders “What does a long-term research agenda look like for this organization?” 

-Figuring out what you want to know 5 or 10 years in the future will help you spend evaluation resources more strategically by laying out studies that build upon one another, and will allow you to make complementary investments that can be used multiple times. 

-Each evaluation should build upon what you learned in earlier steps.  

-As we noted in the last slide, it is important to first lay a foundation for evaluation activities through the completion of some important early, developmental work. For example, if your program’s stakeholders ultimately want to obtain strong causal evidence for the program by implementing a randomized control trial (RCT) or a quasi-experimental design (QED), it is important to identify what supporting steps need to be taken to reach that goal. 

-You may need to begin with descriptive studies and start collecting routine program data. Then you might develop and administer an annual survey, field a process evaluation, and invest in building additional data collection instruments. Then you might use those same data collection instruments again but also collect data from a comparison or control group as you work toward implementing an RCT or QED. 

-By thinking long-term and investing evaluation money strategically, your scarce resources can have a much larger impact.

7

AmeriCorps Housing Assistance Program

Goal: Demonstrate that the program has a positive impact on beneficiaries via an impact evaluation or quasi-experimental evaluation.

Step 1: Collect program data, routinely, on family background characteristics and number of families served. 

Step 2: Process study: Is the program being implemented with fidelity to the model? 

Step 3: Collect pre/post outcome data each year via annual survey.

Step 4: In addition to data collected from Steps 1&2, collect long-term outcomes data via follow-up survey (1 year post- program) 

Step 5: Local demand for program services exceeds the services provided by the program, so identify other local public and private sources providing services to similar families to serve as a comparison group and conduct a quasi-experimental design (QED study).

Example of a Long-term Research Agenda

8

Facilitator notes: For example, consider the long-term research agenda for a hypothetical AmeriCorps program that provides housing assistance for low-income families. The program would eventually like to demonstrate that it has a positive impact on beneficiaries by implementing an impact evaluation or quasi-experimental evaluation. 

-This is a very worthy goal, but the program would be wise to build up to this point by investing in a number of earlier steps first.  For example, during its first grant award period, the program might start by collecting data on the families it serves on a regular basis. Next could be a process study to assess if the program is actually being implemented with fidelity to its logic model. Based on results from the process study, the program would make adjustments to the model as needed. 

-Then it might design an annual survey and collect pre and post outcome data each year. Then, assuming the program is being implemented well, in later program years, it could start to collect longer-term outcome data from families by designing and administering a follow-up survey.  Finally, knowing that demand for services in the area far exceeds the services provided by the program, the program could implement a quasi-experimental design (QED) by comparing the outcomes of families receiving their housing assistance services to the outcomes for families receiving basic housing assistance through other local public and private sources. All the necessary data collection instruments have already been designed and tested in the earlier years, and this data is already being collected routinely from families served by the program. The program would likely need to hire an evaluator to help identify and recruit similar organizations or agencies to serve as a comparison group and collect the same information from the families who are not receiving housing assistance from the program. Importantly, the program has learned and improved at every earlier stage, so by the time they get to the impact evaluation, they have a stable, mature program model to evaluate. 

-By incrementally fielding studies as part of a long-term research agenda, not only has the program gradually built up its evidence base but it has put systems in place that made the final more rigorous evaluation much less costly than it otherwise would have been.

8

Collect routine program data

Process evaluation

Pre-post outcome data via survey (e.g., non-experimental outcome evaluation)

Long term outcome data via follow-up survey

Quasi-experimental design (QED)

Example: Stages in a Long-term Research Agenda

Program: AmeriCorps program provides housing assistance for low-income families.

Research Goal: Demonstrate that the program has a positive impact on beneficiaries via a quasi-experimental design (QED)

9

Program maturity

How long the program has been in operation

Existing evidence base

Evidence that has already been generated on the program that the long-term research agenda should build off

Funder requirements and other stakeholder needs

AmeriCorps has evaluation requirements for some of its grantees, and those requirements should affect a program’s decisions about its long-term research agenda

Sometimes the same evaluation can meet the needs and requirements of multiple funders 

What to Consider When Developing a Long-term Research Agenda

Facilitator notes: The scope and depth of a program’s long-term research agenda is influenced by a number of factors. These factors include:

Program maturity 

This refers to how long the program has been in operation. Key research questions and needs for information will differ based on a program’s maturity. If your program is in an early stage of development, your long-term research agenda may include developing the appropriate data collection systems and identifying baseline information needed to measure program outcomes. On the other hand, a more established program (6+ years in operation) likely already has data collection systems in place and is measuring program outcomes in some capacity. Thus, a long-term research agenda for a mature program may be different then a newer program. Factoring in a program’s level of maturity when developing a long-term research agenda will help ensure that the research agenda is appropriately focused and feasible for the program to carry out over time.

Existing evidence base 

This refers to evidence that has already been generated on the program. As noted earlier, one aspect of a long-term research agenda is that it should be designed to build evidence of the program’s effectiveness over time. Doing so allows a program to maximize its resources for research and evaluation to provide information that will benefit the program’s overall development. Building on past evidence also means that a program doesn’t waste time and resources by repeating the same studies over and over. 

Funder requirements and other stakeholder needs

Often times, a program’s funder has specific evaluation requirements that must be fulfilled and those requirements may affect which evaluation design the program may decide to use to assess their program. For example, AmeriCorps State and National grantees that receive an average annual AmeriCorps grant of $500,000 or more must conduct an evaluation that provides statistical evidence of the impact of their program compared to what would have happened in the absence of the program through the use of a comparison or control group. Small grantees or those receiving annual AmeriCorps funds of less than $500,000 are not required to demonstrate impacts of their program with the use of a comparison or control group.

Also consider the needs and requirements of other stakeholders such as your board and community members, and note that sometimes one evaluation can meet the needs of multiple stakeholder groups.

10

Long-term program goals

A long-term research agenda should be designed to systematically provide information that supports a program’s long-term strategic goals 

Long-term research goals

Programs should have long-term research goals that relate to building evidence of effectiveness over time 

Evaluation budget

The amount of the program’s funding base that will set aside for evaluation activities each year or each grant period

What to Consider When Developing a Long-term Research Agenda

Facilitator notes: It’s important to also consider...

Long-term program goals 

This refers to a program’s long-term strategic goals related to strengthening its program activities, addressing areas for improvement, moving the program forward to new accomplishments, and possibly expanding to new sites or adding new services. A long-term research agenda should be designed to systematically provide information that supports a program’s long-term strategic goals. 

Long-term research goals 

In addition to having long-term program goals, programs should also have long-term research goals that relate to building evidence of effectiveness over time. A long-term research agenda should be designed to help programs accumulate evidence of effectiveness in a resource efficient manner. We will provide more detail on building evidence of effectiveness in Part 2 of this presentation. 

Evaluation budget 

The amount of the program’s funding base that will be set aside for evaluation activities each year or each grant period. A program’s evaluation budget should be considered in developing a long-term research agenda. Because programs may need to raise additional funds to carry out the research activities identified in a long-term research agenda, it is important to plan ahead. 

11

Exercise Part I: Key Considerations in Developing a Long-term Research Agenda for Your AmeriCorps Program
| Your AmeriCorps program  |  |
|--------------------------|--|
| Program maturity         |  |
| Existing evidence        |  |
| Funder requirements      |  |
| Long-term program goals  |  |
| Long-term research goals |  |
| Evaluation budget        |  |

-Facilitator notes: This course has one exercise that we are going to present in phases throughout the presentation. 

-For this first part, fill in these key considerations for developing a long-term research agenda for your program. You have a copy in your handout. 

Program maturity:

Existing evidence: 

Funder requirements: 

Long-term program goals: 

Long-term research goals:.

Evaluation budget:

-Keep this in mind as we go through the next sections of the presentation, and we’ll return to the next part later on. 

12

PART 2Building Evidence of Effectiveness

Facilitator notes: In Part 2 of this presentation, we will focus on building evidence of effectiveness, which is a key factor that influences a program’s long-term research agenda. 

13

Stage 1:Identify a strong program design

Stage 5:Attain causal evidence of positive program outcomes

Stage 3:Assess program outcomes

Stage 2:Ensure effective implementation

Stage 4:Obtain evidence of positive program outcomes

Building evidence of effectiveness

Evidence Informed

Evidence Based

Building Evidence of Effectiveness

Facilitator Notes: As an agency, AmeriCorps continues to invest in a portfolio of programs reflecting a range of evidence, from evidence-informed to evidence-based. The diagram on this slide illustrates AmeriCorps' approach to building evidence which emphasizes that evidence of effectiveness is built over time and falls along a continuum. Evidence should be appropriate for a program’s life cycle and investment of public dollars. This diagram shows different stages to situate a program’s cumulative body of evidence along an evidence continuum. Having a long-term research agenda will help programs progress from stage to stage along the evidence continuum in a resource efficient manner. Programs are not expected to follow the same linear path to building evidence along the continuum. Some programs may accumulate evidence from output performance measurement activities (stage 2) to having causal evidence of effectiveness from an impact evaluation (stage 5). Others may move from collecting output performance measurement data (stage 2) to having pre/post outcome data (stage 3). Programs may also go back and forth between stages as they refine and adjust their program models over time. 

The key stages that a program goes through as they build their evidence base are shown in the diagram. 

Stage 1: Identify a strong program design

Stage 2: Ensure effective implementation

Stage 3: Assess program outcomes 

Stage 4: Obtain evidence of positive program outcomes

Stage 5: Attain causal evidence of positive program outcomes

-It’s important to note that these stages are not mutually exclusive, and they do overlap, so consider the continuum to be a guide.  Next, each of these stages will be discussed in more detail.

14

Stage 1: What is the Program Model and What is it Supposed to do?

Stage 1:Identify a strong program design

Stage 5:Attain causal evidence of positive program outcomes

Stage 3:Assess program outcomes

Stage 2:Ensure effective implementation

Stage 4:Obtain evidence of positive program outcomes

Evidence Informed

Evidence Based

Stage 1:Identify astrong programdesign

Gather evidence to support program design (e.g., conduct a literature review and/or needs assessment)

Develop logic model

Pilot program

Facilitator notes:

The first stage on the evidence continuum is to identify and implement a strong program design by gathering evidence that supports the intervention to be used. At this stage, programs might be looking at evidence from other programs to better define their program model such that it is evidence-informed. This may be done by conducting a literature review or a needs assessment. During this initial stage, you should develop a logic model which clearly communicates the central model of your program. For an overview of logic models, AmeriCorps grantees can refer to the module, “How to Develop a Program Logic Model” located on AmeriCorps’ Evaluation Resources Page. It is also recommended that the program be piloted during this initial step to achieve full implementation prior to expanding the program more widely. 

15

Stage 1: What is the Program Model and What is it Supposed to do?

Stage 2:Ensure effective implementation

Output performance measurement

Process evaluation

Stage 1:Identify a strong program design

Stage 5:Attain causal evidence of positive program outcomes

Stage 3:Assess program outcomes

Stage 2:Ensure effective implementation

Stage 4:Obtain evidence of positive program outcomes

Evidence Informed

Evidence Based

Facilitator notes: Once a strong program design has been identified, the second stage on the evidence continuum is ensuring the effective full implementation of the program. In other words, it is important to gather evidence to confirm that the program is operating as envisioned. Much of these activities can be supported through the identification and regular monitoring of performance measures. In addition, a formal process evaluation should be conducted at this stage to document program processes, ensure fidelity to the program model, evaluate program quality and efficiency, and establish continuous improvement protocols. 

For many programs, this includes collecting data on the specific activities carried out and the direct products and services produced by your program’s activities, referred to as outputs. Outputs are generally measured in terms of the volume of work accomplished, such as amount of service delivered, staff hired, systems developed, sessions conducted, and materials developed. Examples include number of beneficiaries served, number of hours of service provided to beneficiaries, number of staff trained, number of materials distributed, and number of trainings conducted. Collecting and analyzing a program’s output data are useful for keeping implementation of the program on track and also for determining if program implementation meets the quality standards to which the program aspires. This is important so that, if a program does not achieve its intended goals, it can be determined if the program was using the wrong approach or if it was a strong program that simply was not implemented correctly. Output measures also can assess issues such as the costs of operating a program, the numbers of beneficiaries reached, the most successful program locations, or comparisons of the program’s design and activities to other similar programs.

16

Stage 1: What is the Program Model and What is it Supposed to do?

Stage 3:Assessprogram outcomes

Measure program outcomes (e.g., non-experimental evaluation design) 

Stage 1:Identify a strong program design

Stage 5:Attain causal evidence of positive program outcomes

Stage 3:Assess program outcomes

Stage 2:Ensure effective implementation

Stage 4:Obtain evidence of positive program outcomes

Evidence Informed

Evidence Based

Facilitator notes: The third stage in the continuum involves assessing the program’s actual outcomes to better understand whether the program is achieving its expected outcomes. This process involves developing indicators for measuring outcomes and possibly conducting one of the less rigorous and less resource-intensive outcome evaluation designs, such as a non-experimental outcome study. 

An example of a non-experimental outcome design is a single group pre-post design, which examines program beneficiaries both before and after they receive program services. For example, an AmeriCorps program that provides literacy assistance to struggling readers in elementary schools may conduct a non-experimental outcome evaluation by administering a literacy assessment to students who were served by the program at two time points – at the beginning of the school semester and at the end of the school semester. The pre-post outcome data would be analyzed to determine if students’ literacy scores improved over the semester. It is important to point out that a non-experimental evaluation design such as this would not produce results which can be considered attributable or caused by the program. That evidence would be generated by an impact evaluation, which we will discuss later.   

For an overview on evaluation designs, AmeriCorps grantees can refer to the module, “Overview of Evaluation Designs” located on AmeriCorps’ Evaluation Resources page.

17

Stage 1: What is the Program Model and What is it Supposed to do?

Stage 4:Obtainevidence of positiveprogram outcomes

Impact evaluation (e.g., quasi-experimental design, experimental design)

Stage 1:Identify a strong program design

Stage 5:Attain causal evidence of positive program outcomes

Stage 3:Assess program outcomes

Stage 2:Ensure effective implementation

Stage 4:Obtain evidence of positive program outcomes

Evidence Informed

Evidence Based

Facilitator notes: One stage further in the continuum is obtaining evidence of positive program outcomes by examining the relationship between program activities and outcomes to determine if the program’s activities lead to the program’s expected outcomes. Programs at this stage of the continuum will have performed multiple non-experimental outcome evaluations on their program using pre/post measures that yielded positive findings. A program at this stage might utilize an independent external evaluator to conduct a more rigorous outcome study. Or, using the example from the previous slide, at this stage the literacy program might administer a follow-up assessment to students one semester later to see if literacy gains have been sustained. 

Also, by this stage a program has established regular evaluation activities and processes for obtaining continuous program feedback (e.g., annual member surveys and regular data collection and analysis of pre-post measures on their program beneficiaries, etc.).

An outcome evaluation is not the same as collecting and reporting on performance measurement outcome data. Performance measures should be collected on an ongoing basis and compared to a target level, whereas an evaluation is designed to answer a specific research question about the goals of the intervention. Also, an outcome evaluation may in some cases examine different outcomes, or it may involve collecting data on outcomes at an additional time point. 

18

Stage 1: What is the Program Model and What is it Supposed to do?

Stage 5:Attain causal evidence of positiveprogram outcomes

Outcome evaluation

Stage 1:Identify a strong program design

Stage 5:Attain causal evidence of positive program outcomes

Stage 3:Assess program outcomes

Stage 2:Ensure effective implementation

Stage 4:Obtain evidence of positive program outcomes

Evidence Informed

Evidence Based

Facilitator notes: Finally, the highest stage of evidence allows a program to make the claim of being evidence-based after attaining causal evidence of positive program outcomes. At this stage, programs have completed multiple independent evaluations using strong study designs, such as a quasi-experimental evaluation using a comparison group or an experimental, random assignment design study using a control group to establish the causal linkage between program activities and intended outcomes.

These types of evaluation designs are referred to as impact evaluations, and they are distinguished by the use of a comparison or control group, which is a group of individuals that either receive a different intervention than the one being evaluated or no intervention at all. A comparison or control group is necessary for deriving an estimate of the program’s impact by comparing the amount of change or improvement between comparison/control groups and those who participated in the program. An evaluation design that utilizes a comparison/control group provides more reliable evidence that observed changes in program beneficiaries were due to the program or intervention and not caused by something else. 

For this stage on the evidence continuum, an AmeriCorps program that provides literacy assistance to struggling readers in elementary schools may conduct an experimental impact evaluation by administering a literacy assessment to students in the treatment group who were served by the program and a control group of students who were not served by the program. An evaluator would randomly assign students to either the treatment group or the control group. To determine if the program has an impact on students’ literacy achievement, a literacy assessment would be administered to students in both groups at two time points – at the beginning of the school semester when the program starts and at the end of the school semester when the program ends. The pre-post literacy assessment data would be analyzed to calculate the difference in average score of the intervention group minus the difference in average score of the comparison group before and after treatment (difference in differences method). If the data shows that the average score gain of the treatment group is higher than average score gain of the control group at the end of the program, there is reliable causal evidence that shows the program has an effect on students’ literacy achievement. 

AmeriCorps understands that it may not be possible for all programs to obtain evidence at this highest stage on the evidence continuum due to the structure of their program model, their program’s life cycle, and/or resource limitations. Instead, AmeriCorps' expectation is that programs will use the most rigorous evaluation approaches possible as they continue to build their evidence base and obtain the highest level on the evidence continuum that is appropriate for their program.  

19

Building Evidence of Effectiveness: Stages 1-5

Gather evidence that supports program design.

Develop logic model

Pilot program

Outcome Evaluation

Impact Evaluation

Stage 1: Identify a strong program design

Stage 5: Attain causal evidence of positive program outcomes

Stage 3: Assess program outcomes

Stage 2: Ensure effective implementation

Stage 4: Obtain evidence  of positive program outcomes

Evidence Informed

Evidence Based

Facilitator notes: To summarize, programs should have a long-term research agenda that will help them progress from stage to stage along the evidence continuum in a resource efficient manner. Programs should conduct research and evaluation activities with the understanding that these accomplishments are part of a larger continuum of evidence that builds over time and across multiple stages. It is important to note that establishing a strong program design, sound performance measures, and data collection systems, and identifying measureable program outcomes are key accomplishments in the earlier stages of a program’s evaluation activities. These activities also are fundamental to building later stages of evidence of effectiveness through more rigorous evaluation methods such as an impact evaluation design. Impact evaluations are best implemented after a program has had sufficient time to mature and is no longer undergoing refinement of its central model. 

While the evidence continuum is described in terms of stages, a program’s movement along the continuum is not always linear and may be completed in a back-and-forth direction that is more cyclical in nature. For example, a program that is at stage 3 on the continuum may find that the results from a non-experimental outcome evaluation showed no changes in the program’s short-term outcome for beneficiaries. This finding suggests that the relationship between the program’s activities and the short-term outcome may not be working as intended and thus may need to be examined further through a targeted process evaluation to gather more in-depth information about how the program’s activities are being implemented and whether there is fidelity to the program’s central model. While conducting a process evaluation after an outcome evaluation implies moving back to stage 2 on the evidence continuum, the program will still be accumulating new evidence to further its path on the evidence continuum.  

20

Output Performance Measurement

Process evaluation

Outcome Performance Measurement 

Stage 1: Identify a strong program design

Stage 5: Attain strong evidence of positive program outcomes

Stage 3: Assess program outcomes

Stage 2: Ensure effective implementation

Stage 4: Obtain evidence  of positive program outcomes

Exercise Part II: Building evidence of effectiveness for your AmeriCorps program

Evidence Informed

Evidence Based

Exercise Part II: Building Evidence of Effectiveness for Your AmeriCorps Program

-Facilitator notes: Now we’ll return back to the exercise. For this second part, identify where your program currently is on the evidence continuum and then consider where you want to go. Where would you place the stars for your program? 

-Keep this in mind as we go through the next section of the presentation, and we’ll return to the final part of the exercise later on.

21

PART 3Example Scenarios

Facilitator notes: For this next section of the presentation, we present two example scenarios of what a long-term research agenda might look like for different types of funded AmeriCorps programs. 

The first scenario is of a small, new AmeriCorps program, and the second scenario is of a large, established AmeriCorps program. Your program may fit with one of these better than the other, but there are lessons from both that will likely apply to your program and its particular stage of evidence-building. 

22

Stage 1: Identify a strong program design

Stage 5: Attain strong evidence of positive program outcomes

Stage 3: Assess program outcomes

Stage 2: Ensure effective implementation

Stage 4: Obtain evidence  of positive program outcomes

Scenario 1: Building a long-term research agenda for a new Senior Companion Program

Evidence Informed

Evidence Based

Scenario 1: Building a Long-term Research Agenda for a New Senior Companion Program

Facilitator notes: Example scenario 1 is based on an AmeriCorps Seniors grantee implementing a new Senior Companion Program. This slide shows that for a small, new AmeriCorps program, their evidence of effectiveness is initially at stage 1, but the grantee intends to develop a long-term research agenda that will move them to the third stage on the evidence continuum. 

23

Scenario 1: Logic Model for a Senior Companion Program (SCP)

	                    Process                                                                     Outcomes
| INPUTS                                      | ACTIVITIES                                                                                                                                             | OUTPUTS                                                                                                                                                                                                       | OUTCOMES                                                                                                                                                                                   |                                                                                                                                                                                                        |                                                                                                                                                                                  |
|---------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|                                             |                                                                                                                                                        |                                                                                                                                                                                                               | SHORT-TERM                                                                                                                                                                                 | MEDIUM-TERM                                                                                                                                                                                            | LONG-TERM                                                                                                                                                                        |
| What we invest                              | What we do                                                                                                                                             | Direct products from program activities                                                                                                                                                                       | Changes in knowledge, skills, attitudes, opinions                                                                                                                                          | Changes in behavior or action that result from participants’ new knowledge                                                                                                                             | Meaningful changes, often in their condition or status in life                                                                                                                   |
| Funding 4 FT Staff30 SCP volunteersTraining | Provide independent living services (transportation, nutrition/food support, assistance with medical appointments, etc.)Provide companionship services | 50 older adults received transportation services45 older adults received nutrition/food support,30 older adults received assistance with medical appointments 30 older adults received companionship services | Older adults experience improved capacity for independent living.Older adults experience increases in social support.SCP volunteers experience increased feelings of social-connectedness. | Older adults experience increased social and instrumental support.Older adults experience decreases in stress, anxiety, and depression.SCP volunteers experience feelings of making a positive impact. | Older adults experience greater levels of independence.Older adults experience increases in overall health and well-being.SCP volunteers experience increases in overall health. |

Facilitator notes: On this slide we present an example of what a logic model might look like for a small, new Senior Companions Program (SCP) that focuses on providing supportive and individualized services to homebound older adults. SCP volunteers offer companionship to offset feelings of isolation, but also provide other supportive services as necessary such as driving older adults to medical appointments, doing chores around the house, and assisting with shopping for groceries. 

Based on the logic model, the program has the following inputs and resources: funding, full-time program staff, SCP volunteers, and trainings in order to provide independent living and companionship services. By carrying out these activities, the program expects to produce the following outputs each year: 50 older adults receive transportation services, 45 older adults receive nutrition/food support, 30 older adults receive assistance with medical appointments, and 30 older adult volunteers receive companionship services. The program is expected to achieve the following outcomes: 

Short-term

Older adults experience improved capacity for independent living.

Older adults experience increases in social support.

SCP volunteers experience increased feelings of social-connectedness

Medium-term

-   Older adults experience increased social and instrumental support.

-   Older adults experience decreases in stress, anxiety, and depression.

-   SCP volunteers experience feelings of making a positive impact.

Long-term

Older adults experience greater levels of independence.

Older adults experience increases in overall health and well-being.

SCP volunteers experience increases in overall health.

24

Scenario 1: Key Considerations in Developing a Long-term Research Agenda
| New, Senior Companion Program |                                                                                                                                                                                                                                     |
|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Program maturity              | AmeriCorps grantee with no prior years of program implementation.  Operating in only one community site.                                                                                                                            |
| Existing evidence             | The program’s evidence falls in the first stage on the continuum as it has conducted a literature review to determine best practices for implementing core service activities. No evaluations have been conducted on the program.   |
| Funder requirements           | AmeriCorps Seniors grantees are required to report national performance measures; grantees have no explicit evaluation requirements.                                                                                                |
| Long-term program goals       | Achieve full program operation with efficiency and fidelity to the program’s central model. Realize all expected program outcomes.                                                                                                  |
| Long-term research goals      | Generate data to facilitate program improvements and ensure an efficient, full operation of the program’s service activities. Generate data on the program’s short- and medium-term outcomes (see logic model).                     |
| Evaluation budget             | 10% of the program’s annual funding has been set aside for evaluation activities.                                                                                                                                                   |

Facilitator notes: Some key considerations in developing a long-term research agenda for this new AmeriCorps Seniors program that provides senior companion services include: 

Program maturity: The program is new, a first-time AmeriCorps grantee with no prior years of program implementation. In its first year of operation, the program is only operating in one community site. 

Existing evidence: As a new program, the program’s evidence falls in the first stage on the continuum. The program’s is considered evidence-informed based on conducting a literature review to determine best practices for implementing its core service activities.  

Funder requirements: AmeriCorps Seniors requires its grantees to select and report on National Performance Measures. While grantees are encouraged to participate in evaluation activities, there is no explicit evaluation requirement for its grantees. 

Long-term program goals: Within a six-year time frame, the program intends to achieve full program operation with efficiency and fidelity to the program’s central model. It intends to realize all expected short- and medium-term program outcomes.

Long-term research goals: The program’s long-term research goals are aligned with its programmatic goals and involve generating data to facilitate program improvements and ensure an efficient, full operation of the program’s service activities. Additionally, the goal is to generate data on the program’s targeted short- and medium-term outcomes, specifically assessing whether changes occur between beneficiaries’ program entry and exit. 

Evaluation budget: The program has a small evaluation budget and has set aside between 10% of its annual funding for evaluation activities. 

25

Scenario 1: Long-term Research Agenda for a Senior Companion Program (SCP)
| Evaluation activities |                                                                                                                                                                                                                                          | Stage of evidence                                                   |
|-----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|
| 1                     | Develop a logic model and a detailed program implementation plan.                                                                                                                                                                        | 1: Identify strong program design                                   |
| 2                     | Create a data system to routinely collect performance measurement data and background data on program beneficiaries and SCP volunteers. Program staff and SCP volunteers begin routine data collection activities.                       | 2: Ensure effective implementation and develop data collection plan |
| 3                     | Develop a survey to collect short-term outcome data, focusing on beneficiaries’ capacity for independent living and increases in social support. SCP volunteers administer pre/post surveys to program beneficiaries and analyze data.   | 3: Assess program outcomes                                          |
| 4                     | Conduct an internal process evaluation to determine if the program is being implemented with fidelity to the central model. Make data-driven adjustments to the program’s implementation as needed.                                      | 2: Ensure effective implementation                                  |
| 5                     | Conduct a non-experimental outcome evaluation using an external evaluator, measuring both short-term and medium-term outcomes.                                                                                                           | 3: Assess program outcomes                                          |

Facilitator notes: On this slide we present an example of a long-term research agenda for the new SCP grantee that takes into account the program’s logic model and other key characteristics. 

The grantee’s long-term research agenda outlines a set of evaluation activities that are designed to move the program from stage 1 (Identify a strong program design) to stage 3: (Assess program outcomes) on the evidence continuum. Over a six-year time frame, the activities include:

Conduct a literature review to determine best practices for implementing the program’s core service activities –providing supportive and individualized services to homebound older adults. Develop a logic model and a detailed program implementation plan that lays out what and how each of the program’s core service activities are to be implemented, incorporating best practices identified in the literature. This first set of activities aligns with stage 1 on the evidence continuum and should have been accomplished before the program applied for funding.

Create a data system to routinely collect performance measurement data and background data on program beneficiaries, program staff, and SCP volunteers. Collecting background data on all individuals who are involved in the program helps lay the groundwork for a process evaluation. Once the core elements of the data system have been developed, program staff and volunteers will begin to support routine data collection activities. This second set of activities aligns with stage 2 on the evidence continuum. Data collection systems should be in place before the program begins its first year of operation.

Once the program is collecting performance measure routinely and accurately, they would develop a survey to measure the program’s short-term outcomes such as beneficiaries’ capacity for independent living and increases in social support. After the survey has been developed and pilot-tested with a small sample, volunteers will begin to administer the survey to program beneficiaries at program entry and program exit. This third set of activities aligns with stage 3 on the evidence continuum.

Next, for the program’s first evaluation, they could conduct an internal process evaluation to determine if the program is being implemented with fidelity to the central model as depicted by the logic model and program implementation plan. Examples of questions that a process evaluation might address include: 

	- Who is the program serving? Is the program reaching its target population? Are volunteers 	implementing the supportive and individualized services according to the program’s central model? 	Are volunteers using the workshop curriculum according to the central model? What challenges are 	volunteers facing in serving the program’s beneficiaries? The program would then make data-	driven adjustments to the program’s implementation as needed. This fourth set of activities aligns 	with stage 2 on the evidence continuum.

Next, conduct a non-experimental outcome evaluation using an external evaluator, measuring both short-term and medium-term outcomes. Again, the short-term outcomes that will be measured are outcomes on beneficiaries’ capacity for independent living and increases in social support. Also, some medium level outcomes, such as decreases in stress, anxiety, and depression, could be examined. This fifth set of activities aligns with stage 3 on the evidence continuum.

In sum, this example long-term research agenda for a small, new AmeriCorps grantee covers stages 1, 2, and 3 on the evidence continuum with each of the evaluation activities designed to build evidence of program effectiveness.

26

Stage 1: Identify a strong program design

Stage 5: Attain strong evidence of positive program outcomes

Stage 3: Assess program outcomes

Stage 2: Ensure effective implementation

Stage 4: Obtain evidence  of positive program outcomes

Scenario 2: Building a long-term research agenda for a large, established AmeriCorps program

Evidence Informed

Evidence Based

Scenario 2: Building a Long-term Research Agenda for a Large, Established AmeriCorps Program

Facilitator Notes: Example scenario 2 is based on a large, AmeriCorps State and National (ASN) grantee (annual AmeriCorps funds > $500k) in its second three-year AmeriCorps grant cycle. Large ASN grantees are required to complete impact evaluations of their programs during their second or higher three-year grant cycles. This slide shows that for this large, relatively established AmeriCorps program, its evidence falls at stage 3 on the evidence continuum, but the grantee intends to develop a long-term research agenda that will move them to the fifth stage on the evidence continuum. At the same time, the grantee’s research agenda includes evaluation activities to continue to generate evidence that falls under stages 2 and 3 on the continuum. 

27

Scenario 2: Example Logic Model for Large, Established, Environmental Restoration Program 
| INPUTS                                                                                            | ACTIVITIES                                                                                               | OUTPUTS                                                                                                       | Outcomes                                                                                               |                                                                                                 |                                                                      |
|---------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|
|                                                                                                   |                                                                                                          |                                                                                                               | Short-Term                                                                                             | Medium-Term                                                                                     | Long-Term                                                            |
| What we invest                                                                                    | What we do                                                                                               | Direct products from program activities                                                                       | Changes in knowledge, skills, attitudes, opinions                                                      | Changes in behavior or action that result from participants’ new knowledge                      | Meaningful changes, often in their condition or status in life       |
| Funding  Staff 200 AmeriCorps State and National members200 non-AmeriCorps volunteers Research    | Conduct forest enhancement and restorationComplete up-keep activities to enable native plants to survive | Install 100,000 native trees and shrubs on public landRemove 50% of invasive plant species on 10 forest sites | Increase diversity  and coverage of native plant species   Reduce presence of invasive plant species   | Improve habitat spaces for wildlifeIncrease survival rate of native plant species  and wildlife | Maintain conservation of healthy, productive, sustainable ecosystems |

Facilitator notes: 

On this slide we present an example of what a logic model might look like for a large, relatively established ASN AmeriCorps program that recruits members to protect and restore natural habitats. Improving habitat for state and federally listed species is a primary goal of the program and member projects focus on forest enhancement and restoration.

The program has the following inputs and resources: funding, full-time program staff, AmeriCorps members, staff, enhancement and restoration projects on forest and wetland sites, and upkeep activities to enable native plants to survive. By carrying out these activities, the program expects to produce the following outputs in a three-year period: install 100,000 native trees and shrubs on public land and remove 50% of invasive plant species on 10 forest sites (25 acres in size). The program is expected to achieve the following outcomes: 

Short-term

Increase diversity and coverage of native plant species

Reduce presence of invasive plant species

Medium-term

Improve habitat spaces for native plants and wildlife

Increase survival rate of native plants and wildlife

Long-term

Maintain conservation of healthy, productive, sustainable ecosystems

28

Scenario 2: Key Considerations in Developing a Long-term Research Agenda
| Large, Established ASN Environmental Restoration Program |                                                                                                                                                                                                                                                                                                             |
|----------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Program maturity                                         | AmeriCorps grantee in its second three-year AmeriCorps grant cycle. Already operating in multiple sites and expects to add additional service sites.                                                                                                                                                        |
| Existing evidence                                        | Established data collection processes to collect performance measurement output and outcome data. Conducted internal process evaluation yielding evidence that the program is being implemented with fidelity in most service sites.                                                                        |
| Funder requirements                                      | Large grantees must conduct an external impact evaluation by the end of their second grant cycle. Large grantees are required to submit an impact evaluation report AND an evaluation plan for a future evaluation with their re-compete application after completing two or more three-year cycles.        |
| Long-term program goals                                  | Achieve and maintain fidelity of program implementation across all existing sites and any new service sites. Build stronger evidence of effectiveness to support future requests for higher levels of funding to expand program operations.                                                                 |
| Long-term research goals                                 | Conduct an external impact evaluation to assess the program’s short- and medium-term outcomes. Four to six years is the minimum amount of time for program outcomes to be realized . For this reason, the grantee will submit a request for an alternative evaluation approach for timing considerations.   |
| Evaluation budget                                        | 15% of the grantee’s annual funding has been set aside for evaluation activities.  Grantee is seeking additional outside funding for the impact evaluation.                                                                                                                                                 |

Facilitator notes: Some key considerations in developing a long-term research agenda for a large, established AmeriCorps ASN program that focuses on environmental restoration services include: 

Program maturity: The program has completed its first three-year AmeriCorps grant cycle, and has just received a second three-year cycle of funding. Members conduct environmental restoration projects in multiple sites and 2 new sites are to be added in the coming years. 

Existing evidence: During its first grant cycle, the grantee developed a program-wide database and established routine data collection processes for collecting performance measurement output and outcome data across all service sites. This program also conducted a process evaluation in its first grant cycle; while this is good practice, note that it is technically more than AmeriCorps requires for the first grant cycle. 

Funder requirements: For large ASN grantees, AmeriCorps requires an external impact evaluation to be completed by the end of their second three-year grant cycle. Grantees are required to submit an evaluation report AND an evaluation plan with their re-compete application after completing two or more three-year cycles.

Long-term program goals: The program’s long-term goals include maintaining fidelity of program implementation across existing program sites as well as new sites that are added during the second grant cycle. Additionally, the program intends to build up its evidence of effectiveness as part of a larger strategy to secure additional funding to expand program operations in the future.

Long-term research goals: The program’s long-term research goals consist of conducting an impact evaluation that spans a six-year time frame to assess the program’s short- and medium-term outcomes, specifically whether there’s an increase in the diversity and coverage of native plant species, a reduced presence of invasive plant species, improved habitat spaces for wildlife, and an increase in survival rate of native plant species and wildlife. As an environmental restoration program, four to six years is the minimum amount of time needed for the program’s short- and medium-term outcomes to be fully realized. For this reason, the grantee intends to submit a request for an alternative evaluation approach for timing considerations, specifically requesting approval for a three-year extension to complete an external impact evaluation report. If their request is approved, the grantee will submit an “interim” evaluation report with their re-compete application after completing a second three-year grant cycle. More information on AmeriCorps' “Approval of Alternative Evaluation Approaches for ACSN Grantees” can be found on the Evaluation Resources page. 

Evaluation budget: The program has set aside 15% of its annual funding for evaluation activities and is seeking additional outside funding to cover the last three years of the impact evaluation study. 

29

Scenario 2: Long-term research agenda for large, established environmental restoration program
| Evaluation activities |                                                                                                                                                                                                                                                                                                                          | Stage of evidence                               | Grant cycle   |
|-----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------|---------------|
| 1                     | Conduct a quasi-experimental design (QED) study using an external evaluator, measuring all short- and medium-term outcomes over a six-year time frame and relative to a matched comparison group of sites (i.e., adjacent non-serviced areas that are similar to the pre-restoration conditions at the treatment sites). | 5: Obtain evidence of positive program outcomes | 2+3           |
| 2                     | Continue to collect and analyze output and outcome performance measurement data on an annual basis.                                                                                                                                                                                                                      | 3: Assess program outcomes                      | 2, 3, 4, etc. |
| 3                     | Conduct an internal process evaluation focusing on new service sites to determine if the program’s new restoration projects are being implemented with fidelity to the central model. Make data-driven adjustments to the program’s implementation as needed.                                                            | 2: Ensure effective implementation              | 2             |

Facilitator notes: On this slide we present an example of a long-term research agenda for a large, established AmeriCorps grantee that takes into account the program’s logic model and other key characteristics. 

The grantee’s long-term research agenda outlines a set of evaluation activities that are designed to move the program from stage 3 (Assess program outcomes) to stage 5: (Obtain strong evidence of positive program impact) on the evidence continuum while continuing to generate evidence that supports stages 2 and 3. The activities include:

Continue to collect and analyze output and outcome performance measurement data on an annual basis. This set of activities aligns with stage 3 on the evidence continuum, and will occur in the second grant cycle and beyond. 

Simultaneous to those ongoing activities, conduct a quasi-experimental (QED) study using an external evaluator, measuring all short- and medium-term outcomes over a six-year time frame and relative to a matched comparison group of sites (i.e., adjacent non-serviced areas that are similar to the pre-restoration conditions at the treatment sites). Data collection for the QED study may include: 

A pre-restoration site condition survey at both treatment and comparison sites to document the dominant invasive plants at each site, existing native vegetation conditions and coverage, and existing habitat conditions; 

A post-restoration site condition survey administered at multiple time points over a six-year period at both treatment and comparison sites to record native and invasive plant cover estimates, native species richness, plant health and vigor, and habitat conditions. This set of activities aligns with stage 5 on the evidence continuum, and will start in the second grant cycle and continue into the third grant cycle.

Conduct an internal process evaluation focused on new service sites to determine if the new restoration projects are being implemented with fidelity to the central model. Make data-driven adjustments to the program’s implementation as needed. This set of activities aligns with stage 2 on the evidence continuum, and will occur in the second grant cycle.

In sum, this example long-term research agenda for a large, relatively established AmeriCorps grantee shows the grantee moving from stage 3 to stage 5 on the evidence continuum while also carrying out evaluation activities that fall under stages 2 and 3 to assess program implementation across new service sites and generate performance measurement data. 

30

Part 4Exercise

Facilitator notes: Now let’s finish up the exercise, first reviewing the two parts that you completed earlier. 

31

Stage 1: Identify a strong program design

Stage 5: Attain strong evidence of positive program outcomes

Stage 3: Assess program outcomes

Stage 2: Ensure effective implementation

Stage 4: Obtain evidence  of positive program outcomes

Exercise: Building evidence of effectiveness for your AmeriCorps program

Evidence Informed

Evidence Based

Exercise: Building Evidence of Effectiveness for Your AmeriCorps Program

Facilitator Notes: Remember, the goal of the exercise is to begin to build a long-term research agenda for your AmeriCorps program.

32

Exercise Part I: Key Considerations in Developing a Long-term Research Agenda for Your AmeriCorps Program
| Your AmeriCorps program  |  |
|--------------------------|--|
| Program maturity         |  |
| Existing evidence        |  |
| Funder requirements      |  |
| Long-term program goals  |  |
| Long-term research goals |  |
| Evaluation budget        |  |

Facilitator notes: First, we asked you to fill in these key considerations for developing a long-term research agenda for your program: 

Program maturity:

Existing evidence: 

Funder requirements: 

Long-term program goals: 

Long-term research goals:.

Evaluation budget:

33

Stage 1: Identify a strong program design

Stage 5: Attain strong evidence of positive program outcomes

Stage 3: Assess program outcomes

Stage 2: Ensure effective implementation

Stage 4: Obtain evidence  of positive program outcomes

Exercise Part II: Building evidence of effectiveness for your AmeriCorps program

Evidence Informed

Evidence Based

Exercise Part II: Building Evidence of Effectiveness for Your AmeriCorps Program

Facilitator notes: Next we asked you to identify where your program currently is on the evidence continuum and then consider where you want to go. 

34

Exercise Part III: Long-term Research Agenda for Your AmeriCorps Program
| Evaluation activities |  | Stage of evidence | Grant cycle |
|-----------------------|--|-------------------|-------------|
| 1                     |  |                   |             |
| 2                     |  |                   |             |
| 3                     |  |                   |             |
| 4                     |  |                   |             |

Facilitator notes: Now for this last part of the exercise, fill in the evaluation activities that match up with each stage of evidence that you will be building as part of your long-term research agenda.

35

A long-term research agenda is a developmental approach to evaluation whereby evidence of effectiveness is built over time.

A long-term research agenda is unique and should be tailored to fit each individual program.

There is value to building evidence at all stages along the continuum.

A long-term research agenda should reflect an iterative process where evidence is built gradually over time. 

Important Points to Remember

Facilitator notes: There are a few points that are important to remember when developing a long-term research agenda:

A long-term research agenda is a developmental approach to evaluation whereby evidence of effectiveness is built over time, including your planned research activities over several future years.

A number of unique factors help determine a program’s long-term research agenda, so it should be tailored to your program’s goals, structure, and available resources.

There is value to building evidence at all stages along the continuum. While it may not be possible for all programs to obtain evidence at the highest stage on the evidence continuum due to the structure of their program model, their program’s life cycle, and/or resource limitations, AmeriCorps' expectation is that programs will use the most rigorous evaluation approaches possible as they continue to build their evidence base.  

A long-term research agenda should reflect an iterative process where evidence is gradually built up over time. 

36

Key Points to Consider When Developing a Long-term Research Agenda

Facilitator notes: Consequently, there are several key questions to consider prior to developing your program’s long-term research agenda, including: program maturity, your program’s existing evidence base, its funder requirements, its long-term program and research goals, and your program’s evaluation budget.

37

AmeriCorps Resources

https://www.americorps.gov/grantees-sponsors/evaluation-resources

The American Evaluation Association: http://www.eval.org

The Evaluation Center: http://www.wmich.edu/evalctr/

Innovation Network’s Point K Learning Center: http://www.innonet.org

Resources

Facilitator notes: Here we provide a list of additional resources on program evaluation that you may find helpful, including on AmeriCorps’ website. 

38

Questions?

39

Instrument and data collection systems development

Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and Evaluation: evaluation@cns.gov

Thank you!

# image2.emf

# image5.png

# image3.png

# image4.emf

# image9.png

# image1.png

# image6.png