# Presentation: 788751

## Barb Pryor, M.S. R.DJolene DeFiore-Hyrmer, B.S.Ohio Heart Disease and Stroke Prevention ProgramOhio Department of Health Matt GladdenApplied Research and Evaluation, HDSP 

**Barb Pryor, M.S. R.D****Jolene DeFiore-Hyrmer, B.S.****Ohio Heart Disease and Stroke Prevention Program****Ohio Department of Health**** Matt Gladden****Applied Research and Evaluation, HDSP **

_**Logic Models for Complex Programs**_
- HDSP State Management and Evaluation Training
- September, 2007

## The findings and conclusions in this presentation are those of the author and do not necessarily represent the views of the Centers for Disease Control and Prevention

## ALL MODELS ARE LIMITED, BUT SOME ARE USEFUL

**How to make logic models useful to you?**

**Notes:**
Logic models have limitations and you may encounter some frustration applying them.  Logic models, however, can be a valuable support to your program.  This training strives to make logic models useful to you.

## Objectives for Today

- Explain the components and principles of a logic model
- Discuss strategies for constructing logic models of complex programs
- Identify limitations of logic models
- Talk about a state example from Ohio
*Engage in hands on exercises*
*Provide you useful tools*

**Notes:**
Go through first four objectives.  Click on the final two bullet points and explain that the presentation includes hands-on exercises that can also be used as useful tools when you return to your site. 

## Objective 1Explain the components and principles of a logic model

**Notes:**
We start by describing the key elements of logic models.

## What is a Logic Model? 

-    Graphic depiction (A VISUAL MAP) of the relationship between program activities and intended effects

**Notes:**
Stress that the logic model is a map and should be looked as a program blueprint.

## Why Bother with Logic Models?

- Improve planning process
- Clarify goals
- Identify priorities, assumptions, and challenges
- Buy-in stakeholders
- Evidence based thinking
- Enhance ability to detect difficulties
- Facilitate evaluation
- Realistic expectations
- Clear articulation of program

**Notes:**
Improve planning process:  The process of creating a logic model has a number of benefits in that it clarifies the goals and structure of the program.

Evidence based thinking:  By outlining the program, a logic helps you know where to look to detect challenges and obstacles and facilitates evaluation.

Realistic expectations:  Systems change initiatives and complex programs take time to make an impact.  A logic model can help set realistic expectations while laying out a compelling argument for the program.

Clear articulation of program:  You should be able to describe your program in 1-minute, 10-minute, and 30-minute to stakeholders, participants, and potential funders.  The logic model helps create a compelling one-minute speech 

## Five Major Components

**INPUTS**

**ACTIVITIES **

**OUTPUTS**

**OUTCOMES**

**What we put in**

**What we do**

**Direct products of activities**

**Benefits or changes**

**CONTEXT AND ASSUMPTIONS**

**1**

**2**

**3**

**4**

**5**

**Notes:**
Quickly describe.  Stress outputs can be tricky because it is often difficult to separate our hopes for the impact of the program from what we are producing.

## Input

- Human, financial, organizational, and relationship resources that a program has available to do the work.

- The resources that are needed to support the program’s activities.

**INPUT**

- Key Question: What supports our program activities?

**Notes:**
Inputs are what we have.  What we already possess.  Inputs included financial resources, but include any resource or asset that helps with the program.  For instance, strong relationships, previous collaboration, or political support.

## Logic Model: Activities

**Create products**  - promotional materials, educational curricula, and media ads
**Provide services** - training, advocacy, and resources
**Support infrastructure** – build relationships, build capacity, and planning

- What the program and staff do to bring about the desired results.   Events and actions that are an intentional part of the program.

**ACTIVITIES**

- Key Question: How are we going to structure our intervention?

**Notes:**
Describe slide.

## Logic Model: Output

- Number of train the trainer classes taught, coalition meetings held, electronic medical records materials produced
- Participation rates
- Hours of service

**OUTPUT**

- Key Question: What were the tangible, direct “products” that are produced by the activities?

**Notes:**
Outputs are what are activities produce such as products or the training of individuals.  They are not the impact of our activities.  Important to make this distinction because we need to monitor whether are outputs lead to the outcomes we want.

## Logic model terminology

- Individual changes in attitude, knowledge, skills, health status, environment etc.
- Organizational or system changes – increased capacity, improved population health, changes in policy
- Usually expressed as short term, intermediate, and long term

- The specific changes that result in someone or something

**Outcomes**

- Key Question: Why we did the intervention?

**Notes:**
These are the impact of our interventions and the driving reason for doing the intervention.  Outcomes are usually divided into short, intermediate, and long-term.  Short-term outcomes often deal with attitudinal change while intermediate outcomes relate to behavior change and long-term outcomes relate to public health impact.  Have to adapt to fit your program.

## Criteria for a Good Logic Model

- Process
- Reasonable stakeholder input
- Audience specific
- Change over time
- Technical
- Logical
- Tells a story
- Appropriate level of detail
- Visually engaging
-  Easy to read

**Notes:**
Good logic models emerge from a good process as well as being technically sound.  Reasonable stakeholder input implies you need appropriate input from stakeholders, but you do not necessarily need to include everybody.  This could bog the process down.  Pick the key people you really need.

Audience specific refers to you should know your target audience so you can customize the model to them.  

All programs change over time and you should change the logic model to fit these changes.

Technical

A logic model should tell a compelling story that makes sense to people.  The level of the detail of the logic model should respond to the needs of your audience.  Finally, making your logic visually engaging and easy to read increase the chance people will use it.

## Criteria for a Good Logic Model

- Useful
- Guiding the creation of evaluation questions
- Assist monitoring
- Help describe program to stakeholders

**Notes:**
The ultimate test of a good logic model is whether its used.  Logic models have a variety of uses guiding evaluation questions, assisting in monitoring programs, and describing a program to stakeholders.

## Exercise

- Develop a media campaign to educate people about the signs and symptoms of a stroke and to call 911
- Please organize the cards describing the program into inputs, activities, outputs, and outcomes
- This is just an example!
- Time: 7 Minutes

**Notes:**
Now we will apply our knowledge of logic models in an exercise.  Go through directions and stress this is not an example of an “ideal” program.

## Major Components of a Logic Model

**INPUTS**

**ACTIVITIES **

**OUTPUTS**

**OUTCOMES**

**What we put in**

**What we do**

**Direct products of activities**

**Benefits or changes**

**CONTEXT AND ASSUMPTIONS**

**Notes:**
Show while people are doing the exercise

## INPUTS

**INPUTS**

**Funding from partners**
**Program staffing**
**Affordable media outlets that reach target audience**
**CDC funding**

**Notes:**
Go through answers and discuss differences between my opinions and attendees.  There are compelling rationales to place some of the cards in different places.  Logic and rationale are the most important.

## INPUTS

**INPUTS**

**ACTIVITIES **

**Identify target audience**
**Identify PSA or create stroke signs & symptoms message **
**Contract media firm to deliver the message **
**Distribute message to partners**
**Evaluate media campaign**

**Funding from partners**
**Program staffing**
**Affordable media outlets that reach target audience**
**CDC funding**

**Notes:**
Go through answers and discuss differences between my opinions and attendees.  There are compelling rationales to place some of the cards in different places.  Logic and rationale are the most important.

## INPUTS

**INPUTS**

**ACTIVITIES **

**OUTPUTS**

**Report evaluating the media campaign**

**Signs & symptoms media campaign completed**
**Effective signs & symptoms message identified or developed**

**Identify target audience**
**Identify PSA or create stroke signs & symptoms message **
**Contract media firm to deliver the message **
**Distribute message to partners**
**Evaluate media campaign**

**Funding from partners**
**Program staffing**
**Affordable media outlets that reach target audience**
**CDC funding**

**Notes:**
Go through answers and discuss differences between my opinions and attendees.  There are compelling rationales to place some of the cards in different places.  Logic and rationale are the most important.

I highlight evaluation because the output from evaluations are often not listed on a logic model even though evaluation is listed as an activity.  Later in the presentation we will discuss feedback loops as a way of displaying this relationship.

## INPUTS

**INPUTS**

**ACTIVITIES **

**OUTPUTS**

**OUTCOMES**

**SHORT**
**High percent of target audience viewed signs & symptoms message multiple times**

**INTERMEDIATE**

**Target audience more likely to recognize signs & symptoms of stroke and call 911**

**Decrease mortality**

**LONG**

**PROCESS**

**Stroke treated more quickly**

**Report evaluating the media campaign**

**Signs & symptoms media campaign completed**
**Effective signs & symptoms message identified or developed**

**Identify target audience**
**Identify PSA or create stroke signs & symptoms message **
**Contract media firm to deliver the message **
**Distribute message to partners**
**Evaluate media campaign**

**Funding from partners**
**Program staffing**
**Affordable media outlets that reach target audience**
**CDC funding**

**Notes:**
Go through answers and discuss differences between my opinions and attendees.  There are compelling rationales to place some of the cards in different places.  Logic and rationale are the most important.

## Objective 2Discuss strategies for constructing logic models of complex programs

## Moving Beyond a Simple Program

- Multiple components
- Large number of inputs, activities, and outputs
- Objectives are unclear or debated
- Activities or outputs are interrelated
- Planning phase has yet to identify target group or activities

**Notes:**
Our previous discussion is helpful in laying out the basic components of a logic model, but fails to capture the complexity of your programs and the difficulty of crafting logic models for your programs.  In your programs, (list and describe challenges).  Ask if there are other challenges?

## Doing Logic Models in the Real Word

**Common themes**
- Level of detail (10,000 foot versus 1,000 foot)
- An important tool to support evaluation
**Identifying relationships among activities, outputs, and outcomes**
- The theory of action
**Assumptions and contextual factors**
**Modify with time**
*Looking at all 4 elements at once*

**Notes:**
In order to create a more complex logic model, you need to ensure that it addresses four major elements (list elements).  As discussed before, a logic model needs to be at the appropriate level of detail.  In order to determine what is appropriate, you need to know your audience.  In general, logic models are at a much more general level of detail, what is referred to as a 10,000 foot view.  They tend not to include all details.

Identify relationships:  A logic model should delineate or describe the relationships among the columns in the logic models.  Specifically, how do you think your activities will related to your outputs and your outputs to your outcome.  A reader should be able to understand why you think your program will make an impact.

Assumptions:  You need to assess your assumptions and identify contextual factors that can impact your program.

Modify with time:  A logic model should grow and change with your program.  Not every year, but when your program makes changes or moves into a new stage.

Have to address each of these four elements simultaneously.

## Doing Logic Models in the Real Word

- I am presenting basic principles
- Have to be uniquely applied to your situation
- Do not interpret as requirements
- Work through these elements by expanding our previous example
- An example not meant to be an ideal program

**Notes:**
I am providing tools and basic principals to help you create your logic model. These have to be applied to your program and are not strict requirements.  Each logic model has its own challenges.

Now we are going to work through an example program that addresses stroke mortality and disability.  I have added a number of activities to our previous example.  Again, this is just an example and should not be viewed as an ideal program.

## INPUTS

**ACTIVITIES **

**OUTPUTS**

**OUTCOMES**

**Build partnership among EMS, fire department, and hospitals**
**Identify target audience for stroke signs and symptoms media campaign**
**Ensure state EMS tracking system compatible with NEMSIS**
**Identify PSA or create stroke signs and symptoms message**
**Provide trainings on the utility of NEMSIS**
**Contract media firm to delivery the stroke signs & symptoms message**
**Develop a stroke protocol for 911 dispatchers**
**Evaluate media campaign**
**Distribute stroke signs and symptoms message to partners**
**Develop state system to track EMS service**
**Train 911 dispatchers in use of stroke protocol**
**Convene a workgroup of experts on stroke rehabilitation to make recommendations for improvement**
**Educate stakeholders on the importance of stroke rehabilitation**
**Hold summit that includes leadership in EMS, fire department, and major hospitals to identify partnership opportunities**

**Stroke signs and symptoms media campaign completed**
**Strong partnership among EMS, fire department and hospitals formed**
**Developed an effective signs & symptoms message**
**State EMS tracking system compatible with NEMSIS**
**Rehabilitation expert workgroup meets monthly**
**A retreat including EMS, fire department, and hospital leadership held**
**A stroke protocol for 911 dispatchers developed**
**Report evaluating media campaign**
**State EMS tracking system developed**
**Majority of 911 dispatchers trained in stroke protocol**
**Rehabilitation expert workgroup makes concrete recommendations for improvement**

**Short**

**Interm.**

**Long**

**911 dispatchers appropriately respond to 911 calls responding strokes**
**Implement expert workgroup recommendations on stroke rehabilitation**
**High percent of target audience reached multiple times by signs and symptoms stroke campaign**
**System changes in the EMS response to stroke implemented**
**Accountability for EMS performance increased**
**Target audience knowledge of stroke signs and symptoms increased**

**Lower stroke mortality**
**Lower disability from stroke**
**Decreases in health disparities in the target group**

**Funding from partners**
**Governor support**
**Program staffing**
**CDC technical assistance**
**CDC funding**
**Affordable media outlets that reach target audience**
**Partners (Academic, Community, Health Care)**
**Technical assistance from a local university**

**Improved EMS response time to stroke victims**
**Higher quality stroke rehabilitation**
**Higher medication compliance during stroke rehabilitation**
**Target group call 911 quicker in response to a stroke**
**EMS delivers patients to stroke certified hospitals**
**Lifestyle changes made by people recovering from stroke**
**Improved EMS delivery time to hospital**

**Notes:**
Please take thirty-seconds to look at the model. What do you notice?

Strength:  It is laid out correctly.

Weakness:  It is very hard to quickly understand how the program works and connect activities to outcomes.

## Element 1: Common Themes

- Appropriate level of detail
- 10,000 foot vs 1,000 foot
- Enough detail to meet audience needs
- Meaningfully group activities, outputs, and outcomes
- Resist listing all activities
- Provide more detail about current activities
- More general description of future activities that are ill-defined

**Notes:**
First, we will look how we can improve the model by searching for common themes.  You need enough information to explain why the program is important and identify its key elements, but you do not want to overwhelm the reader with detail.  More general view is usually better than over-detailed.  In order to get to the appropriate level of detail, you need to meaningfully group your activities by common theme.  This can be very difficult because you are rightfully broad of all your activities.  By listing all your activities, however, you make it difficult to figure exactly what your program is doing.  Usually, four or five major points is what a reader can easily interpret.  Finally, provide more detail about your current activities.  For planning activities, it is okay to display your process for planning or your goals because you do not have detail yet

## Where We Start

**Where We Start**

**Where We Hope to End**

**Notes:**
Go from having a rich listing of activities to grouping that enable a reader to quickly understand your program.

## Themes to Group Activities

- Goal oriented
- CDC 6 priority areas (e.g., signs and symptoms)
- Major goals of state program
- Location
- Community, health care, and business
- Larger themes
- Public education
- Treatment
- Infrastructure
- Other strategies that fit your program

**Notes:**
A number of possible themes can be used to group activities.  I provide a few examples here.  Go through examples and stress that the audience can use other themes to group activities.  Key indicator is they make sense to your audience and you.

## Exercise: Grouping Activity 

- Group 14 activities into 5 or fewer groups
- Activities grouped together should be meaningfully related
- Report groupings and why
- Sorting technique is a tool
- Time
- 10 minutes for activity
-   5 minutes to report out groupings

**Notes:**
Describe activity

## My Groupings

- Media campaign
- Accountability
- NEMSIS tracking system
- Infrastructure for EMS improvement
- Partnership
- Improve 911 response to stroke
- Design protocol and train
- Improve stroke rehabilitation
- Expert workgroup

_**EMS Quality Improvement**_

**Notes:**
After the groups present their groupings, I will present my five groupings.  My grouping reflect the five major initiative the program is taking.  I could of reduced by grouping to three, but lumping the EMS quality improvement initiatives together.

## ACTIVITIES 

**Identify PSA or create stroke signs &  symptoms message**
**Contract media firm to delivery the stroke signs & symptoms message**
**Evaluate media campaign**
**Distribute symptoms message to partners**

**ACTIVITIES **

**Build partnership among EMS, fire department, and hospitals**
**Hold summit that includes leadership in EMS, fire department, and major hospitals to identify partnership opportunities**

**Develop state system to track EMS service**
**Ensure state EMS tracking system compatible with NEMSIS**
**Provide trainings on the utility of NEMSIS**

**Develop a stroke protocol for 911 dispatchers**
**Train 911 dispatchers in the use of stroke protocol**

**Convene workgroup of experts on stroke rehabilitation to make recommendations for improvement**
**Educate stakeholders on the importance of stroke rehabilitation**

**Notes:**
This shows how each activity falls into the five groups or three groups.  Approach would again depend on my audience.

## ACTIVITIES 

**ACTIVITIES **

**OUTPUTS**

_**Media campaign to educate people about stroke signs and symptoms and to call 911**_
**Develop message**
**Contract to company to run media campaign**
**Define target audience**

_**Convene stroke rehabilitation expert workgroup**_

_**EMS tracking**_
**Local tracking system compatible with NEMSIS**

_**Completed stroke awareness  media campaign**_
**Used effective message**
**Reached target audience**

_**Expert recommendations on how to improve rehabilitation for stroke victims**_

_**EMS tracking**_
**Local system that is NEMSIS compatible**

_**Build partnership among EMS, fire department, and hospitals**_

_**Strong partnership built among EMS, fire department and hospitals**_

_**Majority of 911 dispatchers are using a stroke protocol**_

_**Implement stroke protocol among 911 dispatchers**_

**Does this belong in this model?**

**Use bullet points when want to show sub-activities**

**Less detail needed because do not know the planning results**

**Arrows within column can show relationship between activities**

**Notes:**
Now that we have grouped activities, similar grouping could be created for activities.  Sometime activities or outputs will be related to one another.  These can be shown be placing arrows in-between boxes.  An double arrow can shows that the activities effect one another.  For instance, building a partnership may assist efforts to construct an EMS tracking system, while success building an EMS tracking system may support a partnerhsip.

When you are collapsing activities, sometimes bullet points are a good way of showing sub-activities that you think are important.  This lets the reader than just look at the major points or explore the model deeper based on their need.

For the planning activities, less detail is needed because the objective is to make a plan.  Also, the focus of this initiative if very different than the others.  Its inclusion in the model again depends on your purpose.  To display all efforts to address stroke burden or to focus on your EMS efforts.

## Lessons on Themes

- No right answer
- Philosophy of the program
- What are your key goals?
- Be able to describe your program at different levels of detail
- What is the most appropriate level?
- Seek connections and structure among activities
- Bullet points under larger topic
- Easy read

**Notes:**
I want to stress there are not “right” groupings.  However, there are clearer and more logical groupings.  These depend on your program philosophy and goals.

It is always good to be able to describe your program at different levels.  Bullet points may help display multiple level in the same model.  Be careful to keep the model readable.

## Element 2: Theory of Action

- Outline the theory of action
- When possible, identify specific relationships among the activities, outputs, and outcomes
- Key monitoring, evaluation, and explanation
- Map

**Notes:**
Grouping the activities now makes it easier to identify key relationships between activities and outputs and outputs and outcomes.  It is critical to be specific about these relationships when possible.  Being specific makes the argument more compelling and helps monitor the program and identify keep evaluation questions.  Remember a logic model should be a map of your activities.

## Types of Relationships
Many to one (multiple activities are targeting same output or outcome)
One to many (planning activity leads to many initiatives)
Many to many (multiple inputs combine to produce multiple activities)
Relationships “within a column”
One activity dependent on the other
Activities strongly related

- Many to one (multiple activities are targeting same output or outcome)
- One to many (planning activity leads to many initiatives)
- Many to many (multiple inputs combine to produce multiple activities)
- Relationships “within a column”
- One activity dependent on the other
- Activities strongly related

- Theory of Action

**Notes:**
Describe the 3 relationships and “within column” relationships.  Create handout

Note the logic model is very poor and displaying many to many activities.

## ACTIVITIES 

**ACTIVITIES **

**OUTPUTS**

_**Media campaign to educate people about stroke signs and symptoms and to call 911**_
**Develop message**
**Contract to company to run media campaign**
**Define target audience**

_**Convene stroke rehabilitation expert workgroup**_

_**EMS tracking**_
**Local tracking system compatible with NEMSIS**

_**Completed stroke awareness  media campaign**_
**Used effective message**
**Reached target audience**

_**Expert recommendations on how to improve rehabilitation for stroke victims**_

_**EMS tracking**_
**Local system that is NEMSIS compatible**

_**Build partnership among EMS, fire department, and hospitals**_

_**Strong partnership built among EMS, fire department and hospitals**_

_**Majority of 911 dispatchers are using a stroke protocol**_

_**Implement stroke protocol among 911 dispatchers**_

**Notes:**
In this example, each activity is associated with an output.  Very clear to reader.  I also track the “within” column relationships we discussed earlier.  Lets move to the relationship between outputs and outcomes.

## OUTPUTS

_**Expert recommendations on how to improve rehabilitation for stroke victims**_

_**Majority of 911 dispatchers are using a stroke protocol**_

_**Strong partnership built among EMS, fire department and hospitals**_

**OUTCOMES**

**Short**

**Intermediate**

**High percent of target audience reached multiple times**

**Dispatchers appropriately respond to 911 calls involving strokes**

**System changes in EMS response to stroke**

**Improved EMS response time to stroke victims**

**Improved EMS delivery time to hospitals**

**Implement recommendations on stroke rehabilitation**

**Higher quality stroke rehabilitation**

**Higher medication compliance**

**Lifestyle changes**

_**Completed stroke awareness media campaign**_
**Used effective message**
**Reached target audience**

**Target audience knowledge of stroke signs and symptoms increased**

**Target group call 911 quicker in response to a stroke**

_**EMS tracking**_
**Local system that is NEMSIS compatible**

**Strong accountability for EMS response**

**EMS delivers patients to stroke certified hospitals**

**Notes:**
The outputs to outcomes relationship is more complex.  In the media campaign, we see an example of one to many relationships.  We have two major goals for the campaign, exposure and increase knowledge.  Knowledge could have been an intermediate goals, but I choose to make it short because it fit better.  Media reach and knowledge combine to increase the target group calling 911 quicker in response to a stroke.

For EMS improvements, we see a many to many relationship.  We are hoping that the EMS tracking, partnerships and 911 dispatcher protocol combine to improve 3 steps in the EMS response to strokes.  Also, I should note that both the accountability and dispatcher efforts are focused on spurring and supporting larger systems changes.

In the final example, we see how the outcomes of the planning effort represent what we hope to change through planning.  In the future, we will need to substitute in the actual activities that we are pursuing.

## OUTPUTS

**OUTCOMES**

**Short**

**Intermediate**

**Higher quality stroke rehabilitation**

**Higher medication compliance**

**Lifestyle changes**

**ACTIVITIES**** **

_**Convene stroke rehabilitation expert workgroup**_

**Lower stroke  mortality**

**Lower disability from stroke**

**Decreases in health disparities in the target group**

**Long**

**INPUT**

_**Funding**_
**Private foundation**
**CDC**
_**Political**_
**Governor support**
_**Partners**_
**Media outlets**
**Program partners (academic, community, health care)**
_**Technical**_
**CDC technical assistance**
**Local university**
_**Program staffing**_

_**Media campaign to educate people about stroke signs and symptoms and to call 911**_
**Develop message**
**Contract to company to run media campaign**
**Define target audience**

_**EMS tracking**_
**Local tracking system compatible with NEMSIS**

_**Build partnership among EMS, fire department, and hospitals**_

_**Implement stroke protocol among 911 dispatchers**_

_**Completed stroke awareness  media campaign**_
**Used effective message**
**Reached target audience**

_**Expert recommendations on how to improve rehabilitation for stroke victims**_

_**EMS tracking**_
**Local system that is NEMSIS compatible**

_**Strong partnership built among EMS, fire department and hospitals**_

_**Majority of 911 dispatchers are using a stroke protocol**_

**High percent of target audience reached multiple times**

**Dispatchers appropriately respond to 911 calls involving strokes**

**System changes in EMS response to stroke**

**Implement recommendations on stroke rehabilitation**

**Target audience knowledge of stroke signs and symptoms increased**

**Strong accountability for EMS response**

**Improved EMS response time to stroke victims**

**Improved EMS delivery time to hospitals**

**Target group call 911 quicker in response to a stroke**

**EMS delivers patients to stroke certified hospitals**

**Notes:**
Now we put the full model together.  This model is clearer and easier to read then the first model and also highlights testable hypotheses.  This model may have too many arrows between boxes and we may choose to remove these depending on our audience.

## OUTPUTS

**OUTCOMES**

**Short**

**Intermediate**

**Higher quality stroke rehabilitation**

**Higher medication compliance**

**Lifestyle changes**

**ACTIVITIES**** **

_**Convene stroke rehabilitation expert workgroup**_

**Lower stroke  mortality**

**Lower disability from stroke**

**Decreases in health disparities in the target group**

**INPUT**

_**Funding**_
**Private foundation**
**CDC**
_**Political**_
**Governor support**
_**Partners**_
**Media outlets**
**Program partners (academic, community, health care)**
_**Technical**_
**CDC technical assistance**
**Local university**
_**Program staffing**_

_**Media campaign to educate people about stroke signs and symptoms and to call 911**_
**Develop message**
**Contract to company to run media campaign**
**Define target audience**

_**Improve EMS System**_
**Create EMS tracking system  that is compatible with NEMSIS system**
**Build partnership among EMS, fire department, and hospitals**
**Implement 911 stroke protocol for dispatchers**

_**Completed stroke awareness  media campaign**_
**Used effective message**
**Reached target audience**

_**Expert recommendations on how to improve rehabilitation for stroke victims**_

_**Improvements in EMS infrastructure**_
**Tracking system that is NEMSIS compatible**
**Strong partnership between EMS, fire department, and hospitals**
**911 dispatchers using stroke protocol**

**High percent of target audience reached multiple times**

_**System changes in EMS response to stroke**_
**Improved dispatcher response to 911 calls involving strokes**

**Implement recommendations on stroke rehabilitation**

**Target audience knowledge of stroke signs and symptoms increased**

**Strong accountability for EMS response**

**Improved EMS response time to stroke victims**

**Improved EMS delivery time to hospitals**

**Target group call 911 quicker in response to a stroke**

**EMS delivers patients to stroke certified hospitals**

**Long**

## INPUTS

**ACTIVITIES **

**OUTPUTS**

**OUTCOMES**

**Build partnership among EMS, fire department, and hospitals**
**Identify target audience for stroke signs and symptoms media campaign**
**Ensure state EMS tracking system compatible with NEMSIS**
**Identify PSA or create stroke signs and symptoms message**
**Provide trainings on the utility of NEMSIS**
**Contract media firm to delivery the stroke signs & symptoms message**
**Develop a stroke protocol for 911 dispatchers**
**Evaluate media campaign**
**Distribute stroke signs and symptoms message to partners**
**Develop state system to track EMS service**
**Train 911 dispatchers in use of stroke protocol**
**Convene a workgroup of experts on stroke rehabilitation to make recommendations for improvement**
**Educate stakeholders on the importance of stroke rehabilitation**
**Hold summit that includes leadership in EMS, fire department, and major hospitals to identify partnership opportunities**

**Stroke signs and symptoms media campaign completed**
**Strong partnership among EMS, fire department and hospitals formed**
**Developed an effective signs & symptoms message**
**State EMS tracking system compatible with NEMSIS**
**Rehabilitation expert workgroup meets monthly**
**A retreat including EMS, fire department, and hospital leadership held**
**A stroke protocol for 911 dispatchers developed**
**Report evaluating media campaign**
**State EMS tracking system developed**
**Majority of 911 dispatchers trained in stroke protocol**
**Rehabilitation expert workgroup makes concrete recommendations for improvement**

**Short**

**Interm.**

**Long**

**911 dispatchers appropriately respond to 911 calls responding strokes**
**Implement expert workgroup recommendations on stroke rehabilitation**
**High percent of target audience reached multiple times by signs and symptoms stroke campaign**
**System changes in the EMS response to stroke implemented**
**Accountability for EMS performance increased**
**Target audience knowledge of stroke signs and symptoms increased**

**Lower stroke mortality**
**Lower disability from stroke**
**Decreases in health disparities in the target group**

**Funding from partners**
**Governor support**
**Program Staffing**
**CDC technical assistance**
**CDC funding**
**Affordable media outlets that reach target audience**
**Partners (Academic, Community, Health Care)**
**Technical assistance from a local university**

**Improved EMS response time to stroke victims**
**Higher quality stroke rehabilitation**
**Higher medication compliance during stroke rehabilitation**
**Target group call 911 quicker in response to a stroke**
**EMS delivers patients to stroke certified hospitals**
**Lifestyle changes made by people recovering from stroke**
**Improved EMS delivery time to hospital**

**Notes:**
Show new versus old model

## Links between inputs and activities are often too complex to model simply

**OUTCOMES**

**Short**

**Intermediate**

**Higher quality stroke rehabilitation**

**Higher medication compliance**

**Lifestyle changes**

**ACTIVITIES**** **

_**Convene stroke rehabilitation expert workgroup**_

**Lower stroke  mortality**

**Lower disability from stroke**

**Decreases in health disparities in the target group**

**Long**

**INPUT**

_**Funding**_
**Private foundation**
**CDC**
_**Political**_
**Governor support**
_**Partners**_
**Media outlets**
**Program partners (academic, community, health care)**
_**Technical**_
**CDC technical assistance**
**Local university**
_**Program staffing**_

_**Media campaign to educate people about stroke signs and symptoms and to call 911**_
**Develop message**
**Contract to company to run media campaign**
**Define target audience**

_**EMS quality improvement**_
**Local tracking system compatible with NEMSIS**

_**Build partnership among EMS, fire department, and hospitals**_

_**Implement stroke protocol among 911 dispatchers**_

_**Completed stroke awareness  media campaign**_
**Used effective message**
**Reached target audience**

_**Expert recommendations on how to improve rehabilitation for stroke victims**_

_**EMS quality improvement**_
**Local system that is NEMSIS compatible**

_**Strong partnership built among EMS, fire department and hospitals**_

_**Majority of 911 dispatchers are using a stroke protocol**_

**High percent of target audience reached multiple times**

**Dispatchers appropriately respond to 911 calls involving strokes**

**System changes in EMS response to stroke**

**Implement recommendations on stroke rehabilitation**

**Target audience knowledge of stroke signs and symptoms increased**

**Strong accountability for EMS response**

**Improved EMS response time to stroke victims**

**Improved EMS delivery time to hospitals**

**Target group call 911 quicker in response to a stroke**

**EMS delivers patients to stroke certified hospitals**

**Links between inputs and activities are often too complex to model simply**

**Often intermediate outcomes converge to produce long-term outcomes**

**Notes:**
I also want to note that the connections between resources and activities is often displayed as general relationship because linkages are too complex to display in a logic model.  Similarly, you often see this between intermediate and long-term outcomes because the impact of programs often combine to effect public health.  There are instances where you require more detail.

## Feedback Loop

- An output or outcome that affects an input or activity
- Evaluation
- Needs assessment
- Fundraising
- Political support/opposition
- Indicated by an arrow pointing backwards

**Notes:**
I would like to end by discussing feedback loops.  Sometimes and output or an outcomes impacts your input or activity.  For instance, evaluation findings may impact how an activity is done.  A fundraising campaign may increase the input of financial resources.  Success may also trigger political opposition that changes the context of your program.  A feedback loop is graphically represented by an arrow pointing backwards.  To see what this looks like lets turn to the next slide.

## Example of Feedback Loop

**ACTIVITIES **

**Contract media campaign on stroke signs and symptoms**

**OUTPUTS**

**Media campaign completed**

**SHORT OUTCOMES**

**People’s knowledge of signs and symptoms increased**

**Evaluation of Campaign**

**Evaluate campaign**

**Recommendations for media campaign improvements**

**Notes:**
Describe feedback loop

## Caution!

- Too many arrows are confusing!
- Focus major relationships
- Be very careful about feedback loops
- Place related activities next to each other in the model

**Notes:**
I want to caution you that too many arrows can be confusing as too few.  I fall into this trap and it is easy to fall into because you want to capture the complex relationships you face in reality.  Be especially careful with feedback loops.  Finally, placing similar activities next to each other in a model often help make the logic model readable because they make it easier for the activities to combine to point to similar outputs or outcomes.

## Element 3: Assumptions and Context Factors

- Critical to always be thinking about your assumptions
- Arrows should help identify assumptions
- What assumptions are untested
- What are the unintended consequences of your program?
- Identify major contextual factors
- Be aware
- Respond quickly to problems
- Help prevent misinterpreting program effects

**Notes:**
Something that is often left out of discussion of logic models are a discussion of the assumptions of your program and contextual factors that may impact it.  Identifying assumptions is critical because it enables you to identify possible problems early and may identify an evaluation need.  The arrows in your models highlight assumptions and thus I have pushed to be specific when possible.  You should also look into which of your assumptions are well-tested by previous research or programs and which are not.  Finally, you should look at unintended consequences.  What are some of the unintended positive or negative contributions of your program.

It is also important to examine contextual factors that impact your program.  Understanding context allows you to respond more quickly to problems and helps preventing misinterpreting effects.  Your program may be having a positive effect while the public health burden you are trying to impact is going up.  This can happen the otherways too.

## Example of Assumptions and Context Factors

- Assumptions
- Media campaign will be effective
- Untested: Partnerships will lead to system improvements
- Unintended consequence: Partnership could heighten tensions
- Context factors
- Political change (e.g., recession, city council, etc.)
- Contentious relationship between fire department and EMS

**Notes:**
This provides examples.

## Usually placed underneath the logic model in two boxes
Sometimes left out of logic models

- Sometimes left out of logic models

- Assumptions and Context Factors in Logic Models

**Notes:**
Assumptions and context are usually placed underneath the logic model, but are sometimes left out due to space.  You should always consider them regardless of whether they are on the logic model.

## Element 4: Adjust with Time

- Program changes
- Component added
- Component dropped
- Integrate what you have learned
- Over time the results of activities may become inputs
- Model shifts to the left
- Update logic model with the results of planning activities
- Specific initiative
- A good logic model today may not be good tomorrow

**Notes:**
The logic model also needs to adjust with time in order to incorporate new program components and integrate what you have learned.  Also, over time, the model shifts to the left where your outputs of activities may become inputs for future activities.  This is especially true as states transition from capacity building to basic implementation.  Also, planning activities result in specific activities and those activities should be put in instead of the planning activities once they are implemented.

## Technical Checklist

- Identify and classify program inputs, outputs, and outcomes
- Appropriate level of detail is used
- Groupings are meaningful
- Logic model persuasive to targeted audience
- Clearly articulate “theory of change” by mapping important relationships among activities, output, and outcomes
- All activities are associated with an output
- Feedback loops are indicated

**Notes:**
This goes through a logic model checklist to help you construct your logic model.

## Identify your assumptions
Unanticipated consequences
Identify major contextual factors that impact your program
Factors that impact implementation or outcomes
Update model over time
Verify logic model:
Is easy to read
Understood by new reader
Integrates input and feedback from your stakeholders

- Unanticipated consequences
- Identify major contextual factors that impact your program
- Factors that impact implementation or outcomes
- Update model over time
- Verify logic model:
- Is easy to read
- Understood by new reader
- Integrates input and feedback from your stakeholders

- Technical Checklist Continued

**Notes:**
Checklist continued.

## Objective 3Identify limitations of logic models

**Notes:**
I would like to end with limitations in the hope that understanding the limits of logic models will limit your frustration while creating logic models.

## Ignoring limitations can produce logic models that do not make sense
Lessen frustration 
Need to look to other models when modeling strong interconnections or very complex relationships
Systems thinking

- Lessen frustration
- Need to look to other models when modeling strong interconnections or very complex relationships
- Systems thinking

- Why Worry about Limitations?

- LOGIC MODELS ARE STILL USEFUL!

**Notes:**
As stated at the beginning, logic models have some limitations and trying to exceed these limitations can produce confusing models and cause frustration.  In general, logic models have trouble displaying strong and complex interconnections among factors.  There are other ways of modeling this type of relationships.  Even with these problems, logic models are still useful to tell the story of your program.

## Limitations of a Logic Model

- The logic model progresses in one direction through time
- Changes can often occur across the model
- Change in the real world often does not occur in a predictable stepwise fashion
- Logic model is unable to represent strong interconnections among activities, outputs, and outcomes
- Reinforces assumptions and diminishes context

**Notes:**
Discuss limitations.

## Conference
Roundtable sessions to work on your logic model Wednesday at 11:00 PM

After the conference
mgladden@cdc.gov
(770) 488-8007

CDC HDSP Logic Model Evaluation Guide
http://0-www.cdc.gov.mill1.sjlibrary.org/dhdsp/state_program/evaluation_guides/logic_model.htm

**Roundtable sessions to work on your logic model Wednesday at 11:00 PM**
**After the conference**
**mgladden**_**[@cdc.gov](mailto:gkv7@cdc.gov)**_
**(770) 488-8007**
**CDC HDSP Logic Model Evaluation Guide**
**http://0-www.cdc.gov.mill1.sjlibrary.org/dhdsp/state_program/evaluation_guides/logic_model.htm**

**Contact Information**

**Notes:**
If you want to talk in more detail about your logic model, please feel free to come to a roundtable session tomorrow or feel free to call me after the conference.

## Objective 4Talk about a state example from Ohio 

**Notes:**
Now, I would like to introduce Barb Pryor and Jolene DeFiore-Hyrmer who will describe the logic model they created for Ohio,

## Logic Models in the Real World

- CDC HDSP Training
- September 2007

- Barb Pryor, M.S. R.D
- Jolene DeFiore-Hyrmer, B.S.
- Ohio Heart Disease and Stroke Prevention Program
- Ohio Department of Health

## Incorporating the Theory of Logic Models into Reality....a state’s perspective

**Ohio’s HDSP Program had established structure and ongoing activities**
**CDC HDSP Program’s RFP for new 5-year cycle defined expectations, program priorities and outcomes**
**2007-12 Workplan completed prior to development of Ohio‘s logic model**
**Logic model built by working toward the center from Inputs on left side and Long-term Outcomes on right side**

## Building Ohio’s Logic Model

**Identify what must happen in each step to accomplish Long-term Outcomes**
**Align activities to Outcomes**
**Activities and Outcomes mirror Ohio HDSP work plan**
**Long-term Outcomes align with CDC’s HDSP State Program Logic Model**
**Intermediate Outcomes reflect measurable Impact Objectives in Ohio’s HDSP 2007-2012 workplan  **
**Only major activities and relationships illustrated**

## Ohio Heart Disease and Stroke Prevention Program Logic Model-- 2007-2012

**Short term outcomes**

**Intermediate Outcomes**

**Long term Outcomes**

**Inputs**

- CDC funding

- CDC technical assistance and training

- Epidemiology and scientific capacity

- Reduction of disability and death from heart disease and stroke

- System changes in the Health care, Worksite, and Community settings that promote heart healthy and stroke free lives

- CDC-provided best practices

- CVD burden surveillance
- Indicator surveillance

- Technical assistance and training to partners

- Assessment of current systems of care

- Partnership development

- Impact Evaluation

- CVH State plan updated

- Heart Disease and Stroke Prevention Council formed

- Partnership with Ohio Commission on Minority Health, Ohio Commission on African-American Males

- Capacity of health care providers strengthened

- Integration among chronic disease prevention programs

- Ohio Chronic Disease Plan developed

- Effective interventions and strategies implemented within settings and priority populations

- Reduction of racial, gender, geographic,  and SES disparities.

- Increase in the percentage of Ohioans who can identify the symptoms of stroke and heart attack and the immediate response to call 9-1-1

- Increase in the percentage of Ohioans whose cholesterol levels are controlled

- Increase in the percentage of Ohioans whose high blood pressure
- is controlled

**Outcomes**

**Activities**

**Outputs**

**Ohio Heart Disease and Stroke Prevention Program Logic Model-- 2007-2012**

- HDSP program partners

## Expansion of Ohio’s Logic Model

**Expanded to integrate new project into Ohio HDSP Program**
**Paul Coverdell Acute Stroke Registry**
**Long-term Outcomes remain constant**
**Process Inputs and Activities similar**
**Outputs, Short-term and Intermediate Outcomes expanded to incorporate new Coverdell components**

## Slide 61

## Benefits of Logic Model

**Clearly describes Ohio HDSP Program to partners**
**Identifies key indicators to measure for program evaluation**
**Provides a “reality check” for Work Plan**
**Keeps staff focused on what needs to be done to achieve outcomes**
**Allows for change as Program grows**

## How do you think you can improve your logic model?
Should activities be categorized at a higher level?
Are relationships clearly specified?
Is the model easy to read?
How can the model be used?
Thank you for your time.

- Should activities be categorized at a higher level?
- Are relationships clearly specified?
- Is the model easy to read?
- How can the model be used?
- Thank you for your time.

- Ideas to Improve Your Logic Model

**Notes:**
Before we go, please take a look at your logic model and think of ways that it can be improved.