How to Develop a Logic Model

2021  |

1

NORC at the University of Chicago

1

By the end of this presentation, you will be able to:

Describe what a logic model is, and how it can be useful to your daily program operations

Identify the key components of a logic model

Develop a logic model for your program

Use a logic model for evaluation planning

Learning Objectives

2

Facilitator notes: For this presentation, we have a number of learning objectives.  

By the end of this presentation, you will be able to: 

Describe what a logic model is, and how it can be useful to your daily program operations

Identify the key components of a logic model

Develop a logic model for your program

And use a logic model for evaluation planning

2

A program’s theory of change and logic model

Uses of logic models

Components of a logic model

How to read a logic model

How to develop a logic model

How to apply logic models to evaluation

Overview of the Presentation

3

Facilitator notes: This presentation will cover the following:

A program’s theory of change and a program logic model

The different uses of logic models

The key components of a logic model

How to read a logic model

How to develop a logic model

And how to use a logic model to inform your program evaluation activities

3

The general underlying idea of how you believe your intervention will create change

There are three main elements: 

For an overview of theory of change and evidence, ASN grantees can refer to the modules, “Designing Effective Action for Change”. 

Theory of Change

4

Community Problem/Need

Specific Intervention

Intended Outcome

Facilitator notes: A program’s theory of change is the general underlying idea of how you believe your intervention will create change and why the desired change is expected to come about. Your theory of change articulates the assumptions underlying your choice of activities. 

There are three main elements to a theory of change. It begins by identifying the problem or need to be addressed. The second element identifies the specific intervention or set of activities that you have chosen to address the problem or need. And the last element refers to the intended outcome or the change you expect to bring about if you deliver the intervention according to plan. 

Your theory of change – the expected cause and effect relationship between a specific intervention and its anticipated outcome(s) – should be based on evidence or a strong hypothesis of how and why the expected change will occur.

4

Theory of change for a nutrition assistance program:	

Example of a Program’s Theory of Change

Problem/Need

Intervention

Intended Outcome

Families suffer from poor nutrition-related health problems

Nutrition education and referral services

Healthier families

Facilitator notes: On this slide, we present an example of a theory of change for a nutrition assistance program. For this hypothetical program’s theory of change, the problem or need that has been identified is that too many low-income families in the community do not understand the relationship between nutrition and health, and therefore suffer poor health consequences. To address this problem, a research-based nutrition assistance program was developed that consists of a combination of nutrition education and referrals to community services. The intended outcome or goal of this intervention, specified in this theory of change, is an improvement in families’ overall health.

5

A detailed visual representation of a program and its theory of change.

Communicates how a program works by depicting the intended relationships among program components: 

Inputs or resources

Activities

Outputs

Outcomes

What is a Logic Model?

6

Facilitator notes: Now that you have a general understanding of what a program’s theory of change is, let’s talk about what a program logic model is and how it relates to a program’s theory of change. A logic model is a more detailed visual representation of a program’s theory of change. Logic models communicate how a program works by depicting the intended relationships among the resources available to operate the program, the activities the program carries out, and the changes or results the program hopes to achieve which are referred to as outputs and outcomes. We will be talking more about each of these components shortly. 

We want to note that logic models come in many sizes and shapes and also vary in level of detail, ranging from basic/simple to complex. There is no one or “right” way to develop a logic model. It often depends upon your purpose, how you will use the logic model, who will use the logic model, and what your program entails. 

The four components outlined on this slide reflect a basic or simple logic model design which aligns with ASN’s logic model guidelines. For this presentation, we use this basic logic model format. 

Also, some of you may be familiar with other terms that are used interchangeably with logic models, such as causal chain, conceptual map, idea map, logical framework, model of change, and roadmap. We use the term “logic model” throughout this presentation. Likewise, there are also a variety of terms for describing these four key components (inputs/resources, activities, outputs, and outcomes) of a logic model, but for this presentation we use the terms shown here on this slide. 

6

Inputs

Short

Long

Activities

Outputs

Medium

Generate a clear and shared understanding of how a program works

Support program planning and improvement

Serve as foundation for evaluation

Why Develop a Logic Model?

Facilitator notes: Developing a logic model has many potential uses for programs. It may be the case that you develop one logic model for your program that serves more than one purpose, or it may be necessary for you to create different versions of a logic model that is tailored for different aims. For example, the logic model you develop for your daily program operations (including performance management and evaluation activities) may be different than the logic model you develop for your GARP application.

On this slide, we present just a few examples of how logic models can be used:

Logic models can be used to ensure that stakeholders have a clear and shared understanding of how a program is intended to work. For example, sharing a program’s logic model with program staff can be beneficial in generating a collective sense of responsibility and accountability as it enables staff to understand how their specific role fits into the broader context of the program. Additionally, funders may require programs to include a logic model in their proposal or application materials because it provides a snapshot or a succinct description of how a program will address a problem by performing specific activities. 

Logic models can be used to support program planning and improvement efforts. For example, logic models can be a useful program management tool to help you determine how to allocate resources effectively as well as serve as a framework to monitor whether the program is being implemented according to plan. 

Last on this slide, logic models can serve as a foundation for evaluation. Logic models can be particularly helpful during the planning stages of an evaluation by helping programs determine when and what to evaluate, so that evaluation resources are used effectively and efficiently. 

We will be discussing how logic models can be used for evaluation planning in greater detail at a later point in the presentation. At this point, we just want to introduce some of the different ways program logic models are used.

7

Inputs or resources

Activities 

Outputs 

Outcomes (short-, medium- and long-term)

Key Components of a Logic Model

Outcomes

Facilitator notes: As you may recall from an earlier slide, there are four key components of a logic model that describe your planned work and your intended results. These include: inputs or resources, activities, outputs, and outcomes.

We’ll talk through each of these components on the next few slides.  

8

Inputs

Short

Long

Activities

Outputs

Medium

Inputs or resources include the human, financial, organizational, and community resources available for carrying out a program’s activities. 

Examples:

Funding

Program staff 

AmeriCorps Seniors

Volunteers

Training

Research

Key Components of a Logic Model

Facilitator notes: Starting on the far left side of your logic model, we have a program’s inputs (sometimes called resources). Inputs or resources are considered essential for a program’s activities to occur. They may include any combination of human, financial, organizational, and community-based resources that are available to a program and used to carry out a program’s activities. 

For example, inputs or resources for the hypothetical nutrition assistance program might include: 

Funding (e.g., an AmeriCorps grant, a community-based grant)

Program staff

AmeriCorps members

Volunteers

Nutrition assistance research

9

Inputs

Short

Long

Activities

Outputs

Medium

Activities are the processes, tools, events, and actions that are used to bring about a program’s intended changes or results. 

Examples:

Workshops on healthy food options

Food preparation counseling

Referrals to food programs and resources

Key Components of a Logic Model

Facilitator notes: From inputs, we move to the activities component of your logic model. Activities are the specific actions that make up your program or intervention. They reflect processes, tools, events, and other actions that are used to bring about your program’s desired changes or results. 

For example, the hypothetical nutrition assistance program might use its specified resources to carry out the following activities: 

Workshops on healthy food options

Food preparation counseling

Referrals to food programs and resources

10

Inputs

Short

Long

Activities

Outputs

Medium

Outputs are the direct products of a program’s activities and may include types, levels and targets of services to be delivered by the program.

Examples:

# individuals attending workshops

# individuals receiving services

# individuals receiving referrals

Key Components of a Logic Model

Facilitator notes: We move from activities to the outputs component of a logic model. Outputs are what a program’s specific activities will create or produce, providing evidence of service delivery. 

Outputs are often described numerically or quantified in some way. For example, outputs associated with the hypothetical nutrition assistance program might include the following: 

the specific number of individuals who attended a healthy food workshop

the specific number of individuals who received food preparation counseling or services

the specific number of individuals who received a referral to food programs and resources

11

Inputs

Short

Long

Activities

Outputs

Medium

Outcomes are the expected changes in the population served that result from a program’s activities and fall along a continuum, ranging from short to long term results: 

Short-term: changes in knowledge, skills, and/or attitudes (e.g., ↑ knowledge healthy choices)

Medium-term: changes in behavior or action (e.g., ↑ adoption of healthy food practices)

Long-term: changes in condition or status in life (e.g., ↑ food security)

Key Components of a Logic Model

Facilitator notes: From outputs, we move to outcomes. Outcomes are the specific changes that may result from a program’s activities or intervention. A program’s outcomes fall along a continuum, ranging from short- to long-term results. Short-term outcomes are often considered changes in beneficiaries’ knowledge, skills, and/or attitudes. Medium-term outcomes refer to changes in behavior or action that result from beneficiaries’ new knowledge, skills, and/or attitudes. Long-term outcomes are often characterized as changes in condition or status in life that result from beneficiaries’ new behaviors or actions.

It is important to note that describing outcomes as short-, medium-, or long-term depends on the objective, the length of the program, and expectations of the program or intervention. What is identified as a long-term outcome for one program could be an medium-term outcome for another. Your logic model should reflect your program’s theory of change and the sequence of changes necessary to accomplish your program’s long-term results.

Let’s use the hypothetical nutrition assistance program as an example. In the short-term, individuals are expected to increase their knowledge of healthy food choices. This change in knowledge is expected to lead them to actually adopt healthy food practices – a behavioral change that represents a medium-term outcome. This, in turn, is expected to lead to an increase in families’ overall household food security – a long-term outcome of the program. 

12

Inputs

Short

Long

Activities

Outputs

Medium

Difference Between Outputs and Outcomes
| Outputs                                                                                                                                                                                      | Outcomes                                                                                                                                                                                                                                |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Direct products of a program’s activities/servicesOften expressed numerically or quantified in some wayExamples:    # attending workshops    # receiving services    # receiving referrals   | Changes resulting from a program’s activities/servicesOften expressed in terms of change in knowledge, attitude, behavior, or conditionExamples:   ↑ knowledge healthy choices   ↑ adoption healthy practices   ↑ food security         |

Facilitator notes: We want to take a moment to highlight the differences between outputs and outcomes as these two components are often confused with one another.

On the left side of the slide, you can see that program outputs are direct products of a program’s activities or services and are often expressed numerically or quantified in some way. Program outcomes, on the other hand, focus on the changes a program intends to bring about as a result of a program’s activities or intervention. 

While outputs are the essential elements that enable change, they do not represent benefits or changes in and of themselves. 

Are there any questions?

13

Read from left to right

Two “sides” to a logic model - a process side and an outcomes side

Two Major Sides to a Logic Model

Facilitator notes: At this point, you should all be familiar with what a logic model is and its key components. Let’s turn now to how you read a logic model. Logic models are typically read from left to right, employing an if-then sequence among key components. A generic example is shown here. It reads, if your program has these inputs or resources, then it can carry out these activities. If your program carries out these activities, then it can produce these outputs. If your program has produced these outputs, then it will achieve these outcomes. 

In addition, a logic model has two “sides.” The process side focuses on a program’s implementation or its planned work – inputs/resources, activities, and outputs (direct products). The outcomes side of the logic model describes the expected sequence of changes that the program is to accomplish, which can be short-term, medium-term, and/or long-term changes. The outcomes side reflects what difference the program intends to make.

14

Two main approaches are used to create a logic model:

Reverse logic (right to left) – asks “but how” questions

Forward logic (left to right) – uses “if...then” statements

How to Develop a Logic Model

Facilitator notes: Let’s now discuss how to create a logic model. There is no one correct way to create a logic model. The approach you select may depend on the stage of development of your program (i.e., planning, implementation, or maintenance stage) and/or how you plan to use your logic model. There are two main approaches that are used to create a logic model – reverse logic and forward logic. These two approaches can also be used in combination.   

Reverse logic creates a logic model by working backwards. It starts with desired outcomes on the right side and moves backwards to identify a program’s activities and inputs on the left side of the model. This reverse logic approach is often used in the development or planning stages of a program as it ensures that program activities will logically lead to the specific outcomes. Using a reverse logic approach, you will ask the question, “But how?” as you move from right to left in your logic model. 

Alternatively, the forward logic approach starts with your program’s inputs and activities and uses a progressive series of “if...then” statements to establish relationships between inputs, activities, outputs, and outcomes. This forward logic approach is often used when there is a clear understanding of what the inputs and activities are or will be (e.g., a program that is in the implementation or maintenance stage rather than the design or development stage). 

Regardless of which approach you take (forward or reverse), working out sequences of “if...then” relationships or the “how” about a program enables one to uncover gaps in logic, clarify assumptions, and better understand how investments and a program’s activities are likely to produce intended results/outcomes.  

15

What is the desired long-term outcome?

Increase # of healthy families. But how?

What is the desired intermediate outcome?

Increase # of families using healthy food practices. But how?

What is the desired short-term outcome?

Individuals gain knowledge of healthy food choices. But how?

What outputs are needed to achieve the outcomes?

200 families complete an educational workshop. But how?

What activities are needed to achieve the outcomes?

Conduct four educational workshops per month. But how?

What inputs are needed to achieve the outcomes?

Funding, program staff, AmeriCorps Seniors, volunteers, research.

How to Create a Logic Model Using Reverse Logic – Sample Nutrition Program

Facilitator notes: Let’s walk through how to create a logic model using a reverse logic approach. Again, this approach starts with desired outcomes and requires you to work backwards to develop activities and inputs. Reverse logic asks the question, “But how?” as you move from desired outcomes to identifying the inputs that lead to these outcomes in your logic model. 

Using the nutrition assistance program as an example, this reverse logic approach (shown here) starts with the program’s long-term outcome of increasing the number of healthy families and works backwards to determine how the program will achieve this outcome. 

16

How to Create a Logic Model Using Forward Logic  – Sample Nutrition Program

Forward logic uses “if-then” statements. 

Source: W.K. Kellogg Foundation Evaluation Handbook (2004), Adapted

Facilitator notes: Let’s now walk through how to create a logic model using forward logic which is demonstrated on this slide here. Again, forward logic uses “if...then” statements, moving from the left side of the logic model with resources/inputs and progressively builds the links as we move across to the outcomes side of the logic model. 

Using the nutrition assistance program as an example, this forward logic approach would be, If we have access to AmeriCorps members, then we can provide food education workshops and referral services. If we carry out these activities, then we will deliver services to 200 families.  If families receive these services, then they will learn about healthy food choices and improve their skills in preparing healthy foods. If families gain this knowledge and acquire these skills, then they will adopt healthy food practices. If families adopt healthy food practices, then they will be healthier.

17

Questions to Consider as You Create a Logic Model
| Component |                   | Questions to consider                                                                          |
|-----------|-------------------|------------------------------------------------------------------------------------------------|
|           | Inputs/ Resources | What resources do you need to implement your program?                                          |
|           | Activities        | What activities will be or are being carried out to achieve your program’s desired outcomes?   |
|           | Outputs           | What are the direct products of your program’s activities?                                     |
| Outcomes  | Short-term        | What changes in knowledge, skills, and/or attitudes do you expect from your program?           |
|           | Medium-term       | What changes in behavior or actions do you expect from your program?                           |
|           | Long-term         | What changes in status or condition do you expect from your program?                           |

Facilitator notes: We’ve provided some questions on this slide here to help you think through the modeling process for each component. Similar questions are also found on the template itself. 

18

Theory of Change. A wildlife conservation program is designed to create healthy, productive, and sustainable ecosystems for the benefit of wildlife in areas of need.

What might this program’s logic model look like?

Exercise: Develop a Logic Model for a Wildlife Conservation Program

Exercise #1

Facilitator notes: Now, we’re going to work on building a logic model together as a group. In this group exercise, let’s try to apply a forward logic approach in developing a logic model for a hypothetical wildlife conservation program. A brief description of the program is provided here. We encourage all of you to participate and provide your input as we build a logic model for this program together. 

19

Example Logic Model for a Wildlife Conservation Program 
| PROBLEM                   | INPUTS         | ACTIVITIES | OUTPUTS                                 | Outcomes                                          |                                                                              |                                                                |
|---------------------------|----------------|------------|-----------------------------------------|---------------------------------------------------|------------------------------------------------------------------------------|----------------------------------------------------------------|
|                           |                |            |                                         | Short-Term                                        | Medium-Term                                                                  | Long-Term                                                      |
| Community problem or need | What we invest | What we do | Direct products from program activities | Changes in knowledge, skills, attitudes, opinions | Changes in behavior or action that result from participants’ new knowledge   | Meaningful changes, often in their condition or status in life |
|                           |                |            |                                         |                                                   |                                                                              |                                                                |

The presence of invasive species and waste (e.g., trash from hikers or visitors) has made it difficult for wildlife to prosper, thus resulting in the reduction of native species (plant and wildlife) and negatively affecting the area’s ecosystem

Funding 

Staff

200 AmeriCorps State and National members

200 non-AmeriCorps volunteers

Research

Conduct waste removal projects

Conduct habitat development projects

Conduct invasive species removal

Develop habitat corridors

Plant native plant species on 30 sites

Remove invasive plant species on 30 sites

Remove toxic waste on 50 acres of wetlands

Develop habitat corridors on 10 sites

Increase in food and clean water supply for native wildlife

Increase in available shelter for native wildlife

Increase in habitat connectivity

Improve habitat space for native wildlife

Increase in native wildlife population sizes

Increase in biodiversity 

Conservation of healthy, productive, sustainable ecosystems for the benefit of wildlife

Facilitator notes: For this exercise, we are using ASN’s NOFO template for a program logic model. 

Before we populate the columns, we want to identify the overarching problem that the program is trying to address. The theory of change can help us define this problem or need. 

- Once we know what problem the program is intended to address, we can ask ourselves, “What might be some potential investments for this program?” The first column, inputs, refers to the resources invested in a program. Funding, program staff, AmeriCorps members, and wildlife conservation research are just some of the potential investments for this program. Other possible resources include additional volunteers who are not AmeriCorps members.  

The next column refers to the activities that the program carries out. If the program has these resources, then what are some potential activities or services that it would be able to provide? Some potential activities include:

 AmeriCorps members and volunteers conduct waste removal projects

 AmeriCorps members and volunteers conduct habitat development projects 

 AmeriCorps members and volunteers conduct invasive species removal

- AmeriCorps members and volunteers develop habitat corridors

The next column refers to a program’s outputs. If we are able to carry out some of the activities we’ve discussed, then what would be some direct products of the activities? Outputs associated with the activities listed above may include:

Plant native trees on x sites

Remove invasive species on x sites

Remove toxic waste on x acres of wetlands 

Develop habitat corridors on x sites (e.g., habitat bridge)

The next column refers to the short-term outcomes column of the program. Based on the activities we’ve discussed (reiterate key activities from the audience), then what might be some short-term changes that may result from these program services and activities? Facilitator may want to note that the guiding text under short-term, medium-term, and long-term outcomes is more applicable for programs that target “people” as opposed to the environment/ecosystem such as this program. However, it is still important to keep in mind that with outcomes, it is about change – so the question is still the same, what changes do you expect will occur because of the program’s services or activities?  Some examples of short-term outcomes (based on the activities listed above) may include: 

Increase in food and clean water supply for native wildlife

Increase in available shelter for native wildlife

Increase in habitat connectivity

Improve habitat space for wildlife

Moving to the medium-term outcomes column, what would be some changes in behavior or action that that might occur because of the activities we’ve discussed (these are changes that you would expect to observe after short-term outcomes have been achieved)? Some examples of medium-term outcomes (based on the activities listed above) may include: 

Increase in wildlife population sizes

Increase in biodiversity

The last column refers to long-term outcomes. If participants change their behavior, then what changes in condition or status can be expected as a result? An example of an expected long-term outcome for this program is:

Conservation of healthy, productive, sustainable ecosystems for the benefit of wildlife    

Just to recap, this exercise was intended to give you an overview of the type of information to include and how to organize it as you develop a logic model for your program.  This example we’ve just done demonstrates a forward logic approach, since we developed our model from left to right. If we were to use a reverse logic approach, we would have started with the long-term goal of the program and worked backwards to answer questions about how the program will achieve its desired outcomes. 

20

In each column of the logic model template, identify the following key components for your program:

Inputs

Activities

Outputs 

Outcomes (short-, medium- and long-term)

Developing a Logic Model

Exercise #2

Facilitator notes: Now that we’ve completed a logic model as a group, we’d like you to develop a logic model that is specific to your program and one that can be useful to you in your daily program operations. Keep in mind that this exercise is about thinking through the logic of your program or intervention. You should include all components of your program and the relationships among them, even if they are not necessarily described in your AmeriCorps application. Likewise, you should include all of your program’s expected outcomes even if you do not plan to measure or collect data on them. 

As we talked about earlier, you may take a forward logic or reverse logic approach, or a combination of the two to create your logic model.

21

Consider asking the following questions:

Level of detail: Does your model contain an appropriate amount of detail for its intended use? Does it include all key program components? 

Plausible: Does the logic of the model seem correct? Are there any gaps in the logic of the program? 

Realistic: Is it reasonable to assume that the program can achieve the expected outcomes?

Consensus: Do program staff and external stakeholders agree that the model accurately depicts the program and its intended results?

Verify Your Logic Model

Facilitator notes: At this point, let’s just pause with where you’re at in creating a logic model of your program.  

An important step in finalizing your logic model is to verify or assess whether the model you’ve created reflects an appropriate level of detail, is plausible, is realistic, and that there is consensus among key stakeholders that the model accurately represents your program. One way to verify that your model meets these criteria is to run through a series of questions, such as the ones shown here on this slide.

In particular, you want to make sure that your logic model: 

Contains the appropriate amount of detail for its intended use, and includes all key program components.

Depicts plausible relationships between program components.

Is realistic. One thing to note is that it’s okay to identify longer-term outcomes for your program even though it may not seem feasible for you to measure the cause and effect relationship between your intervention and your longer-term outcomes. In verifying your logic model, it’s more important to ask whether it’s reasonable to assume that the program can achieve the expected short- to long-term outcomes. 

Achieves consensus among your program’s stakeholders that the model accurately depicts the program and its intended results.

Also, you may want to involve other program staff who were not involved in developing the logic model as well as other external stakeholders to help you verify your model.

EXERCISE #2 – Continued:

To practice verifying your logic model, we want you to exchange logic models with a partner. Each of you should review your partner’s logic model and consider whether the logic model contains an appropriate level of detail, is plausible, and is realistic.  

Any questions before we move on? 

22

Performance Measurement and Program Evaluation
| Performance Measurement                                                                                                               | Program Evaluation                                                                                                                                                                                                                                                        |
|---------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Ongoing monitoring and reporting of program accomplishments and progressExplains what level of performance is achieved by the program | In-depth research activity conducted periodically or on an ad-hoc basisAnswers questions or tests hypotheses about program processes and/or outcomesUsed to assess whether or not a program works as expected and why (e.g., did the program cause the observed changes?) |

Facilitator notes: Now that you’ve had an opportunity to draft a logic model of your program, we want to transition to talking about how your logic model can be used to help with both performance measurement and program evaluation activities. However, before we do so, we want to first provide you with a brief overview of performance measurement and program evaluation and, in particular, how these activities differ from one another. While both performance measurement and program evaluation are considered measurement activities, the two activities serve different purposes.

Some of you may already know what performance measurement is because it’s an activity that you may are already doing in your program. Performance measurement is ongoing monitoring and reporting of program accomplishments and progress toward its pre-established goals. For many programs, this includes collecting data on the specific activities carried out and the direct products and services produced by your program’s activities (outputs). Performance measurement data help you understand what level of performance is achieved by the program/intervention.

Program evaluation, on the other hand, is an in-depth research activity that answers specific questions or tests hypotheses about program processes and/or program outcomes. The results enable you to arrive at a judgment of whether the intervention or a specific component of the intervention works or does not work as expected, and also what adjustments may be needed to improve the program. A key difference between performance measurement and program evaluation is that program evaluation helps you understand and explain why you’re seeing the program results.

It is important to keep in mind that performance measurement and program evaluation are not mutually exclusive. Grantees already engaged in performance measurement activities can build on that work as they plan for a program evaluation. For example, let’s say your program is already collecting data to monitor and report on your program’s progress toward achieving its expected outcomes for program beneficiaries. If your program decides to conduct an impact evaluation to learn whether your progress on outcomes is caused by your intervention, it may be that you continue to collect the SAME outcomes data, but also include a matched comparison group. Including this comparison group enables you to answer specific questions related to causality – in this case, what would have happened to people if they did not receive the intervention (i.e., whether the observed changes can be attributed to your intervention).

Grantees who want to learn more about performance measurement and program evaluation can refer to the Evaluation Resources page.

https://www.nationalserviceresources.gov/evaluation-americorps

23

A logic model can serve as a framework for planning performance measurement activities. It can help to:

Identify components of your program to include in performance measurement

Identify indicators and the measures of progress/performance that align with program components

Logic Models as a Performance Measurement Tool

Facilitator notes: Now that you have an understanding of what performance measurement entails, let’s talk about how logic models can be used as a tool in performance measurement. A few examples of how logic models can help you plan your performance measurement activities are listed here:  

Identify which components of your program (resources, activities, outputs, outcomes) to include in your performance measurement

Identify indicators and measures of progress or performance that align with program components

Just as your logic model can help inform your performance measurement activities, it can also help you plan for an evaluation. We’ll be discussing this in more detail on the next few slides. Any questions before we move on? 

24

A logic model can serve as a framework for your evaluation plan. It can help you focus your evaluation by identifying: 

Questions want/need answered 

Aspects of program to evaluate 

Type of evaluation design

Information to collect 

Measures and data collection methods 

Evaluation timeframe

Logic Models as an Evaluation Tool

Facilitator notes: At this point, we turn our attention to how logic models can inform program evaluation. Your logic model can be used as a tool to help you plan for your evaluation. It can serve as a framework for your evaluation plan by helping you can make informed decisions about what to evaluate, when to evaluate, and how you will evaluate. 

It is important to note that with program evaluation, it is not necessary to evaluate every aspect of your program as depicted in your logic model. Your evaluation can have a narrow focus (e.g., only address questions about one of your program’s service activities and desired outcomes) or it can have a broader focus (e.g., address questions about each of your program’s service activities and desired outcomes), depending on the information you hope to gain from your evaluation. 

Your logic model can be used as a tool to help you focus your evaluation with respect to the following: 

Identify questions you want or need answered about your program

Identify which aspects of your program to evaluate (e.g., will you evaluate a subset or all of your AmeriCorps activities?; will you evaluate your program’s short-term outcomes?) 

Determine the appropriate evaluation design (e.g., will you use a process or an impact evaluation design, or a combination of both?)

Identify what information to collect 

Identify measures and data collection methods

Determine an appropriate timeframe for your evaluation

We’ll walk through a few examples of how to actually use your logic model to focus your evaluation on the next few slides. 

25

Determining What to Evaluate

Facilitator notes: Let’s start with an example of how a logic model can be used to help you draft or narrow in on the questions that you want your evaluation to answer. The graphic on this slide provides an example of the types of questions that may be asked of each component in a logic model: 

Questions related to inputs ask, “Are resources adequate to implement the program?” 

Questions related to activities ask, “Are activities delivered as intended?” 

Questions related to outputs ask, “How many, how much was produced?” 

Questions related to outcomes ask, “What changes occurred as a result of the program?”

As you can see at the bottom of this graphic, in order to answer each of these questions, indicators (i.e., the evidence or information that represents the phenomenon in question) and their data sources will need to be identified. When developing research questions based on your program’s logic model, ensure that questions are stated in a way that is clear and measurable in order for them to be answerable.      

26

Determining What to Evaluate – Sample Nutrition Program
| PROBLEM                                                                                                                                                                                                         | INPUTS                                                               | ACTIVITIES                                                                                                                  | OUTPUTS                                                                                                | Outcomes                                                                                                                                                                                                |                                                                                       |                                                            |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------|
|                                                                                                                                                                                                                 |                                                                      |                                                                                                                             |                                                                                                        | Short-Term                                                                                                                                                                                              | Medium-Term                                                                           | Long-Term                                                  |
| Families suffer from poor nutrition-related health problems and there is limited services available to better educate families and individuals on the importance of integrating healthy foods into their diets. | Funding  Staff 200 AmeriCorps State and National members Research    | Conduct educational workshops Provide nutrition and food prep counseling Provide referrals to food programs and resources   | # individuals receiving education # individuals receiving services # individuals receiving referrals   | Increased knowledge of healthy food choices Improved attitudes about healthy eating Improved skill in preparation of healthy foods Increased knowledge of food programs and community food resources    | Increased adoption of healthy food practices  Increased access to more food options   | Families are healthier  Increased household food security  |

 Process                                                                  Outcomes 

Facilitator notes: As you narrow in on the questions you want answered about your program, this in turn, will help you to determine the type of evaluation you’ll need to conduct and what elements of your logic model are to be incorporated into your evaluation. As noted earlier, a logic model has two sides – a process side and an outcomes side. Questions related to the process side of the model are generally different than questions related to the outcomes side of the model, and therefore require different evaluation approaches to address them. 

Let’s return to the nutrition assistance program we discussed earlier in the presentation as an example. Program staff may be interested in answering the question, “Was the nutrition assistance program implemented as planned and, if not, how and why was it altered for implementation?” These questions concern the key components on the left side of the logic model (the process side), so the type of evaluation needed to answer these questions would be a process evaluation. This means the data collected will focus on understanding the relationship between the resources or inputs that were employed and what activities were accomplished with these resources. Process evaluations tend to be more useful in the beginning stages of a program’s implementation to help determine whether target populations are being reached, people are receiving the intended services, and if staff are adequately qualified, among other things. For example, let’s say the process evaluation data show that the nutrition and food prep counseling was only implemented in a few sites rather than across all sites, as planned, because of staffing constraints. Program staff will be able to use these findings to make adjustments to the program and deploy more staffing resources to sites that were unable to deliver the nutrition and food prep counseling activity. 

27

Determining What to Evaluate – Sample Nutrition Program

 Process                                                                  Outcomes 
| PROBLEM                                                                                                                                                                                                         | INPUTS                                                               | ACTIVITIES                                                                                                                  | OUTPUTS                                                                                                | Outcomes                                                                                                                                                                                                |                                                                                       |                                                            |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------|
|                                                                                                                                                                                                                 |                                                                      |                                                                                                                             |                                                                                                        | Short-Term                                                                                                                                                                                              | Medium-Term                                                                           | Long-Term                                                  |
| Families suffer from poor nutrition-related health problems and there is limited services available to better educate families and individuals on the importance of integrating healthy foods into their diets. | Funding  Staff 200 AmeriCorps State and National members Research    | Conduct educational workshops Provide nutrition and food prep counseling Provide referrals to food programs and resources   | # individuals receiving education # individuals receiving services # individuals receiving referrals   | Increased knowledge of healthy food choices Improved attitudes about healthy eating Improved skill in preparation of healthy foods Increased knowledge of food programs and community food resources    | Increased adoption of healthy food practices  Increased access to more food options   | Families are healthier  Increased household food security  |

Facilitator notes: Alternatively, program staff may be interested in understanding whether their program (or specific components of their program) is effective or not. This type of question calls for an impact evaluation design where the focus is on understanding whether the program was able to produce (all or a subset of) the desired outcomes specified on the right side of the logic model. 

Let’s again use the hypothetical nutrition assistance program as an example. If program staff are interested in finding out “what difference the nutrition assistance program has had on individuals’ knowledge of or attitude about healthy food choices?,” an impact evaluation design that includes a comparison or control group would need to be employed to determine what would have happened to people if they did not receive the intervention. In general, an impact evaluation assesses the extent to which a program achieves its desired outcomes and whether those outcomes can be attributed to the program. This type of evaluation typically occurs after a program has reached a steady state of implementation (e.g., after conducting a process evaluation that showed the program is being implemented as designed).

28

Determining What to Evaluate – Sample Nutrition Program

 Process                                                                  Outcomes 
| PROBLEM                                                                                                                                                                                                         | INPUTS                                                               | ACTIVITIES                                                                                                                  | OUTPUTS                                                                                                | Outcomes                                                                                                                                                                                                |                                                                                       |                                                            |
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|------------------------------------------------------------|
|                                                                                                                                                                                                                 |                                                                      |                                                                                                                             |                                                                                                        | Short-Term                                                                                                                                                                                              | Medium-Term                                                                           | Long-Term                                                  |
| Families suffer from poor nutrition-related health problems and there is limited services available to better educate families and individuals on the importance of integrating healthy foods into their diets. | Funding  Staff 200 AmeriCorps State and National members Research    | Conduct educational workshops Provide nutrition and food prep counseling Provide referrals to food programs and resources   | # individuals receiving education # individuals receiving services # individuals receiving referrals   | Increased knowledge of healthy food choices Improved attitudes about healthy eating Improved skill in preparation of healthy foods Increased knowledge of food programs and community food resources    | Increased adoption of healthy food practices  Increased access to more food options   | Families are healthier  Increased household food security  |

Facilitator notes: For the nutrition assistance program, several short-term and medium-term outcomes have been identified for an evaluation. These are highlighted in red. Depending on the particular intervention, a different nutrition assistance program may determine other outcomes to be more appropriate for an evaluation.  

It is important to note that programs conducting an impact evaluation do not need to evaluate ALL program outcomes at ONE time or even over a period of time. In deciding which outcomes to include in an evaluation, it is important to consider what questions your evaluation seeks answers to and the timeframe of the evaluation and outcome measures (i.e., the amount of time you have available to collect your data as well as the time you can expect changes or program effects to occur, among other factors). A program’s influence is most directly related to proximal outcomes, such as short- and medium-term outcomes. The more time that elapses between your intervention and the outcome that is measured by your evaluation, the more external variables can come into play that are beyond the program’s control. It is important to note that all program evaluations are not designed to measure long-term outcomes and ASN is not expecting programs to measure them in all cases. 

29

Examples of Outcome Measures and Data Sources
|             | Outcomes                                                                              |                                                        |                                                                                              |
|-------------|---------------------------------------------------------------------------------------|--------------------------------------------------------|----------------------------------------------------------------------------------------------|
|             | Short-Term                                                                            | Medium-Term                                            | Long-Term                                                                                    |
| Outcomes    | Increased knowledge of healthy food choices                                           | Increased access to more food options                  | Families are healthier                                                                       |
| Measure     | % ↑ individuals demonstrating greater understanding of benefits of good nutrition     | % ↑ individuals enrolled in food assistance programs   | % ↓risk factors for nutrition related problems and chronic diseases                          |
| Data Source | Pre/post surveys of beneficiaries and a matched comparison group of non-beneficiaries | Administrative data records                            | Pre/post health records of beneficiaries and a matched comparison group of non-beneficiaries |

Facilitator notes: The table above lists some outcomes identified in the nutrition assistance program logic model. We provide an example measure for these selected short-, medium- and long-term outcomes. Keep in mind that each of the outcomes can be measured in other ways as well. The particular measure will depend on and should be tailored to the unique characteristics of the program itself.   

Below each measure is a potential data source. These data sources may be existing or new. The data sources that refer to a matched comparison group of non-beneficiaries pertain to impact evaluations and would not be necessary for programs conducting a process evaluation. 

30

Developing a logic model is not completed in one session or alone. 

There is no one best logic model or model development process.

Logic models represent intention.

A program logic model can change and be refined as the program changes and develops. 

Logic models play a critical role in building the evidence base for a program.

Final Thoughts on Logic Models

  31

Facilitator notes: As we conclude today’s presentation, we’ve highlighted some things to keep in mind as you work to develop and use your program logic model:

Developing a logic model is not completed in one session or alone. 

There is no one best logic model.

Logic models represent intention. We want to underscore that logic models reflect planned activities and outcomes and should be supported by an underlying theory of change. In practice, programs may not only operate quite differently, but also produce unintended results. An evaluation helps identify how faithfully a program actually adheres to its model.    

A program logic model is not static; it can and should be changed and refined as the program changes and develops. 

Programs do not need to evaluate every aspect of a logic model.

Logic models play a critical role in informing evaluation and building the evidence base for a program.

31

AmeriCorps Evaluation Resources page (Logic Model Course, and other evaluation topics)

https://americorps.gov/grantees-sponsors/evaluation-resources

W.K. Kellogg Foundation Logic Model Development Guide

http://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

Innovation Network Logic Model Workbook 	https://innonet.org/media/logic_model_workbook_0.pdf

Resources for Logic Model Development

32

Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and Evaluation: evaluation@cns.gov

Thank you!

# image2.emf

# image5.png

# image9.jpeg

# image10.emf

# image11.jpeg

# image3.png

# image4.emf

# image1.png

# image6.png