# Presentation: 184019

## Logic Model Development

** **

## Why are we doing this?

- As part of a planning process
- As a way to focus more on what we want to *accomplish* rather than on what we *do*
- Set us up to be meaningfully evaluated

**Notes:**
Show intended program impacts, tell our story
Help determine what to monitor and evaluate based on what YOU determine is important

## What logic models should do:

- Make your job easier in the long term
- Show intended program results, tell your story
- Help determine what to monitor and evaluate based on what YOU determine is important

**Notes:**
Easier = streamline the program planning process and simpler reporting

## What is a Logic Model?

- Planning tool
- Flow diagram of your program with defined goals, inputs, outputs, and outcomes connected through causal links
- Visual representation of what and how a program produces its outcomes

**Notes:**
Everything you need to know is in one flow chart.

Logic models link project outcomes with project activities, outputs, and inputs (or resources).

Logic models provide a road map, a visual representation, of your project and how it is expected to work. 

Other terms include program theory or logic, change mapping, outcome sequence charts, outcome mapping, and logic framework

## The Pieces of a Logic Model

**Long Term Outcomes** **– **describe the intended effect of the program on the issues.  They might be social, economic, environmental, or individual consequences.** **
**Mid Term Outcomes – **describe expected changes in the  environment or audience’s behavior based on the continuation of the program or a number of projects.**  **
**Short Term Outcomes – **describe the expected immediate  effect of the program or project (audience reactions or abilities, change in the environment).** **

**Notes:**
effect

## More Pieces

**Outputs – **physical products resulting from activities needed to achieve the desired outcomes.
**Activities – **what you spend your time doing to achieve desired outcomes, produce necessary outputs, or obtain resources.
**Resources** **– **time, money, human resources, office space, utilities, equipment, supplies, management and partner support, etc. needed to accomplish project.

**Notes:**
We must also consider the above when developing our program and creating our logic model.

If resources are not available and cannot be obtained, you may want to rethink your objectives and activities...amend them to be realistic and to be possible with resources that are already on hand

## Program Logic Model

- Resources

- Activities or inputs

- Products or outputs

**Short-Term (immediate) Outcomes** (knowledge, skills, and abilities, changes in the environment)

**Mid-Term Outcomes**
- (behavior change, application of new skills/tools, impacts on environment)

**Long-Term Outcomes**
- (results or change/improvement in issue or effectiveness)

- Planning

- Implementation and Management

**Notes:**
Long-term Outcomes –the intended ultimate impacts of the project on the issue
Mid-term Outcomes – the expected impacts on the audience’s behavior because of the project.  
Short-term Outcomes –the immediate impacts of the project activities (reactions, and change in KSAs or attitudes immediately following exposure to the project/activity). 
Outputs – physical products or services needed to support achieving the desired outcomes
LOGICAL PROGRESSION – SALAD DRESSING STORY
Activities – what YOU have to do or provide in order to achieve the desired outcomes
Resources –time, money, people, office space, utilities, equipment, supplies, management and partner support, etc. needed to achieve the intended impact  
Helpful to develop the logic model at the same time that you consider how you will evaluate the process
Logic model can help to identify appropriate points in the project design process, project activities and outcomes for evaluation to occur

## Points to Remember 

- It’s a tool, not an end
- Outcomes drive activities
- Strength of the logic model is in causal links (if – then statements)
- Basic tool is flexible, make it work for you...
- doesn’t have to have 3 levels of outcomes
- can have iterative loops
- can have feedback loops
- can recognize others’ contributions to the results
- Be consistent in the level you are working within; program level LM does not capture the same detail as project level LM
- Outputs are what we produce, outcomes are what happen as a result of something we’ve produced

## Ginger’s Project Logic Model

- Time
- Dog treats
- Car
- Obedience classes

_**Activities**_
- Teach dog proper behavior for various commands
- Practice sessions in different settings
- Implement progressive reward system

_**Outputs**_
- Practice sessions
- Dog who understands the link between command and desired behavior

_**Short-term outcome**_
- Bogs understands commands and can respond properly when rewarded consistently.

_**Mid-term outcome**_
- Bogs obeys commands in a variety of settings and can respond properly when rewarded consistently.

_**Long-term outcome**_
- Bogs obeys commands without the promise of an immediate reward.

_**Goal:**_  Well-behaved dog
_**Objective:**_ Given that he understands what Ginger wants him to do, Bogs will obey commands at least 90% of the time within six months.

_**External Influences:**_
**Husband**
**Behavior learned from other dogs**
**Distractions**

_**Internal Influences:**_
**Travel schedule**
**Other priorities**

**Ginger’s Project Logic Model**

## Good Dog!

## Project Logic Model

- Resources

- Activities

- Outputs

- Short-Term outcomes

- Mid-Term outcomes

- Long-Term outcomes

** ****Marsh Restoration Workshop Series**

- Coordinator
- Speakers
- Facilities
- Equipment
- Refreshments
- Partners

- Take registration

- Put together presentations

- Dev. support materials

- Advertise Workshop

- Secure speakers

- Contact list and list serve on regional restoration experts

- four workshops

- Support materials

- Participants learn new restoration techniques

- Participants are satisfied with the quality of the workshop

- Participants know whom to go to for assistance with their restoration projects

- Participants continue in discussions and share information learned to colleagues within region

- Participants apply restoration techniques

- Regional restoration efforts are more effective

## Program Logic Model

- Research

- Information on natural and social science

- Protection of resources

- Application of new knowledge to implement  BMP

- Community support for BMPs

- Increased understanding of natural and anthropogenic threats and processes

- Public awareness of issues and management practices

- Public awareness of regulations

- Habitat restoration

- Education

- Enforcement

- Stewardship

- Increased public understanding of issues and management practices

- Increased adherence to regulations

- Increased value of natural function

- Resources

- ActivitiesOutputs

- Short-Term outcomes

- Mid-Term outcomes

- Long-Term outcomes

## From Projects to Programs

- 						Short, Mid, Long-term
- Goal   Objectives   Resources   Activities   Outputs   	Outcomes

- AGENCY

- 						Short, Mid, Long-term
- Goal   Objectives   Resources   Activities   Outputs   	Outcomes

- PROGRAM

- 						Short, Mid, Long-term
- Goal   Objectives   Resources   Activities   Outputs   	Outcomes

- PROJECT

- Objectives	Resources	Outputs		Outcomes

- ACTIVITY

**Notes:**
Same thing from the bottom up

Every part has its own logic model

Project is activity within program; program is activity for agency

## How To Construct a Logic Model

**Step 1**  Identify the effect of the program on the              intended audience and/or issue (as short-, mid-,             and long-term outcomes).
**Step 2  **List the activities and outputs that will support              achievement of the objectives/outcomes.
**Step 3**  List the resources needed and/or available.
**Step 4**  Restate the outcomes as SMART objectives.
**Step 5**  Read the model left-to-right as a series of  “If . . .               then . . .” statements.

**Notes:**
We are always tempted to begin developing our projects with the activities:  “I’m going to do a workshop!”  We can’t do this.  We must first decide on the impacts we want to achieve and THEN how to achieve them.

When identifying impacts, start with the “ultimate” or largest change in the audience and issue that your project can expect to achieve.  Then think about what actions or behaviors will change for the audience.  Then, what change is needed in the audiences’ skills, knowledge, behavior, and attitudes for these outcomes to occur.

Make sure that EACH activity and output supports an objective.  Looking back to the needs assessment helps with this.

Are resources sufficient?  If not, it may be necessary to amend the objectives to be realistic.  We must match up both ends of the logic model.  We cannot expect to improve the water quality of the entire east coast with $3,000.

Are the statements logical?  If not, start over.

We will discuss SMART objectives in the next section of the course.

## Step 1What Are the Desired or Expected Outcomes for the Audience or Issue?

- What do you want to happen long-term as a result of your efforts?
- What are the steps in the mid- and short-term that demonstrate that you are making progress towards that long-term outcome?

## Step 2  List the Activities and Outputs that Support the Objectives and Outcomes

- What activities, materials, or information are needed to move the audience or issue from its current state to the desired state (your expected impact)?

**Notes:**
What do WE have to do to see these outcomes happen?

## Step 3  List the Resources Needed and/or Available

- What will be needed to achieve the desired impact?
- What is available?
- How can you address the gap?
- If you can’t, then go back to step 1.

**Notes:**
What is necessary to reach outcomes?

## ACTIVITYConstruct a Logic Model

**Notes:**
Give each participant a large logic model sheet of paper.

Give each participant post-it papers

Instruct participants to:
Use a current or upcoming project to develop a logic model
Fold back the Objectives column
Begin with a single long-term outcome and construct a simple model (additional outcomes can be added once you’ve developed your strategy and model to achieve one long-term outcome).

Point out that the back of the Logic Model Sheet is an ADDIE “cheat sheet”.

## Step 4  Rewrite the Expected Outcomes asSMART Objectives

**S**pecific
**M**easurable
**A**udience/issue-directed
**R**ealistic but ambitious
**T**ime-bound

**Notes:**
Objectives provide the specific end points and measurable outcomes.

In order to measure the impacts on participants and/or issues of a project it needs objectives that are not vague and that produce an observable action.

Letting audience know what is expected of them and they can therefore organize their learning to accomplish goals.

Objectives specify what the audience will be able to do after being exposed to the project.

Objectives are written as “the audience will...” and describe the intended impacts of the project.

The best objectives are the ones that exclude the greatest number of possible meanings other than your intent.  In other words, it succeeds in communicating your intent of instruction yet avoids misinterpretation.

Well written objectives are SMART objectives.  They have the following characteristics:

## Objectives

*“**Sixty percent of the coastal zone management (CZM) programs will have 3 or more staff members                using GIS by 2009. *
_**S**__pecific_ = “capable of using”
_**M**__easurable_ = 60 percent of programs will have 3 or more (you can go count it)
_**A**__udience-directed_ = what the CZM program staff will get from it
_**R**__ealistic and ambitious_ = cost-effective GIS training is available at NOAA Coastal Services Center and currently <20% have this capability
_**T**__ime-bound_ = 2009

*capable of*

*capable of*

**Notes:**
A good point to make at this time is the fact that you can use percentages of audiences that will be capable, “50% of the CZM programs will have 3 or more” etc...  We must be realistic and realize that we can’t impact the people we can’t reach.  Don’t bite off more that you can chew.

Also, these objectives should be well written, and not subject to misinterpretation, as we are being held accountable. e.g. if it was written as ...staff members will be using.... That would imply that CSC would have to provide hardware and software as well as instruction\

Who will be able to do what, how (and how well), by when?

## Activity

**Write SMART objectives for the outcomes in your logic model.  **

## Objectives

- After viewing the orientation video and exhibits, 80 percent of the visitors to Hanauma Bay will be able to describe reef ecology, and practice the basics of reef etiquette, which include recreating in the area’s sandy bottoms, not removing anything from the marine environment, not feeding the fish, and not harassing the turtles.
- After 5 years of this program, there will be decreased occurrence of fish swarming visitors and turtle aggression within a given transect area.
- After 5 years of this program, there will be a decrease in the number of new impacts observed within a given transect area.
- Sixty percent of visitors will practice the basics of reef etiquette at the other reef systems that they visit.
- After 10 years of this program, the Hanauma Bay reef will begin to regain health, evidenced by increased percentages of healthy coral coverage and increased numbers of species and trophic levels.

_**Resources**_
- Education Center and displays
- Staff
- Volunteers
- Equipment
- $$$

_**Activities**_
- Host orientation session with video
- Staff information desk in Center and on beach
- Educational tours
- Maintain interactive exhibits

_**Outputs**_
- Visitors who have seen orientation video that describes reef ecology
- PowerPoint presentation
- Handouts and brochures for educational tours

_**Short-term outcomes**_
- Visitors to Hanauma Bay will . . .
- be able to describe reef ecology
- understand how humans impact the reef by feeding fish, harassing turtles, removing flora, fauna or substrate from the reef, and “irresponsible” interactions

_**Long-term outcomes**_
- Visitors to Hanauma Bay will apply this knowledge to the other reef systems that they visit
- Hanauma Bay reef begins to regain health

_**Mid-term outcomes**_
- Visitors to Hanauma Bay will . . .
- stop feeding the fish and curb their harassment of turtles
- observe the no-take guidelines while on the reef
- practice “responsible interactions”
- animal behavior will return to normal
- the reef at Hanauma Bay will suffer less physical damage

**Notes:**
Confusing! Huh?

Just shows that everything should be linked: resources – activities – outputs – outcomes – objectives etc.

## Step 5  Read the model

- Do the statements (boxes) in the model make sense when read from left to right?
- Can you make logical connections between each activityand output and an intended outcome?
- Do the outcomes reflect your objectives?
- Are sufficient resources available to achieve the objectives?
- If you achieve each of your objectives (i.e., have the impacts you expect) will you achieve your goal?

- If you answer NO to any of these, review the model determine what isn’t logical, and amend as needed

## Benefits of a Logic Model

- Shows how all the components fit together
- Demonstrates how elements and individuals contribute to the mission
- Helps connect how resources are used and impacts from their use
- Helps identify appropriate performance measures
- Serves as basis of planning, evaluation, and    management decisions

**Notes:**
Logic models must make sense.  When they do, they are an excellent way to communicate to colleagues and superiors the specifics and expectations of a project.

Logic models help us to remain on track and force us to think things through completely.

With outcomes developed, the ways to measure them are easier to attain (performance measures).

Logic models are also a great way to justify more resources ($$) for project and to impress funding agencies.

Logic models help define the endpoint.  

Logic models are also effective in helping to break down projects and get rid of necessary components.

When there isn’t a logic model for a project, often it morphs into something unrecognizable!

Bullet #2 is an important one because it shows staff where there role is in the project.  Give a personal example (like Ginger’s).

Participants work individually, or in pairs, to construct a logic model for their project.

## Common Pitfalls in Creating and Using Logic Models

- Get hung up on the language
- Can work in columns/rows and forget the connections are what make it LOGICAL
- Confuse logic model and performance measures with evaluation
- View (or use) as an academic exercise
- View as linear
- Mix scales (different level of detail at different levels of organization)

## Limitations of a Logic Model

- Model does not tell you if you are doing the “right thing”.  If you do not know what you want to accomplish, the logic model process will be confusing
- Not a strategic plan or an implementation plan, it helps plan how to accomplish to your outcomes
- Model is less important than the process and elements that it addresses
- Snapshot of key elements and logic, not the whole story

## Performance Measurement

- The collection, reporting, and interpretation of performance indicators related to how well a program performs, particularly regarding the delivery of services (process and outputs) and achievement of results (outcomes).

**Notes:**
How many of you are collecting performance measures?

How many of you do a questionnaire at the end of  a workshop, asking “was it useful?”  “Did you like it?” etc.?

“Do you and your organization systematically collect data to tell people how your program is doing/performing?”  

If so, then you are practicing performance measurement.  Analyzing data makes that practice performance measurement.  Most of you are probably already doing it but may be calling it something else: performance indicators, evaluation, etc...    

It is an essential evaluation activity and the principal tool for formative evaluation designed to provide feedback for project improvement.

Performance measurement shows us whether we will meet our objectives or not. 

## Moving from Logic Models to Performance Measures

- You must understand the overall program logic in order to identify what needs to be measured.
- Performance measures can and should be selected from all levels of outputs and outcomes.
- Logic models can show how individual programs can contribute to the larger scale goals.

**Notes:**
Pick PM from different levels
Way to get to pm that is more meaniful
Justification

## Program Logic Model

- Research

- Information on natural and social science

- Protection of resources

- Application of new knowledge to implement  BMP

- Community support for BMPs

- Increased understanding of natural and anthropogenic threats and processes

- Public awareness of issues and management practices

- Public awareness of regulations

- Habitat restoration

- Education

- Enforcement

- Stewardship

- Increased public understanding of issues and management practices

- Increased adherence to regulations

- Increased value of natural function

- Resources

- ActivitiesOutputs

- Short-Term outcomes

- Mid-Term outcomes

- Long-Term outcomes

## Program Performance Measures

- _LT Outcome _Protection of Resources

- _MT Outcome _Community support for BMPs

- _Outputs_
- Public awareness of issues and management practices

- _Education_
- Workshops
- Field trips
- Classroom
- Web site

- _Resources_
- Staff
- Equipment
- Facilities
- Partnerships

- _ST Outcome _Increased understanding of issues and management practices

- How many staff doing education?
- Cost of ed facilities maintenance
- Time spent maintaining partnerships

- How many workshops implemented?
- How many people/hours of participants?
- Satisfaction level with education

- Percent of participants able to identify priority issues
- Percent of community recognizing management options to issues

- Number of times BMPs are discussed in public forums
- Percent of participants that can describe BMP

- Percent of participants that engage in community discussion about priority issues
- Percent of community that votes in support of referendum

- Percent of habitat viable for native species
- Percent reduction of threats

## What Can We Measure?The Friedman Model

- _Quantity_

- _Quality_

- _Outputs_

- _Outcomes_

**Output Quantity**

**Output Quality**

**Outcome**** ****Quantity**

**Outcome Quality**

*How many research projects have been completed?*

*How many results have been published in scientific journals?*

*How many management practices have used the results of the research?*

*How many management practices have been improved by the results of research?*

**Notes:**
When selecting performance measures, it is important to have a balance between quantity and quality

Measuring quality is important if trying to develop relationships not just how many research projects were completed but how often have the results been published in scientific journals.  How many management practices have used the results of the research (quantity outcome), are the management practices being improved by the research results (quality outcome)?

Depending on who you are you may be interested in different types of measures

## What’s the measure?

- Number of contacts from extraterrestrials

**Notes:**
What is the performance measurement in this cartoon? Number of contacts from extraterrestrials.

This is not “rocket science” it is fairly simple.

Sometimes we have to measure something that isn’t happening ... that may be hard to do.  In those cases we use a proximate/proxy measure

For a project dealing with the restoration of a watershed, for example, reduced erosion might be chosen as an indicator. Because it is difficult to measure the level of erosion,
   the level of sedimentation in the river may be used as a proxy.

## Analyzing Performance Data

- Over time
- Against targets
- Across units
- Against benchmarks
- Other (what is meaningful to your program?)

*Always considering external influences!*

**Notes:**
As mentioned earlier, it is not sufficient to just collect data, we must analyze it.

Trends over time

Against other targets – what have other education groups done and what have their success rates been?

Across units – aquaculture dept to water quality dept.  Or across NERRs, Sea Grants, etc...

Against the benchmarks that have been set for the industry.  For example, in education.
Other (what is meaningful to your  program? effectiveness, efficiency, economy, etc.)

## Benefits of Performance Measurement

- Provides a clearer project focus on agency mission and strategy
- Improved project management and decision making
- Provides early warning system enabling decisionmakers to undertake preventative action
- Improved performance
- Increased accountability

**Notes:**
Remember the slide at the beginning where we talked about legislation over the years that speaks to showing the impact of your projects?  PMs help us be accountable, improve performance.

It ensures the project will not fail, or at least not if the PMs were developed by the same individuals implementing the project.

Should we modify anything?

If there are necessary modifications we have a more precise idea of what needs to be modified

If the PMs are off, it doesn’t mean that the project needs to be discontinued, just modified.

We want to be held accountable so that we can attain more resources (money and support)

## Limitations of Performance Measurement

- Selected measures may receive more emphasis than other project components
- Corruptibility of measures
- Incorrect interpretation

## Another Chamber Victory for our Members and Community!Women in Business Conference

**GOAL**
- The Chamber partners with the Center for Women on the annual Women in Business Conference to **accelerate & integrate business women into mainstream networks.**

**ACTION TAKEN**
- The Third Annual Women in Business **Conference was held** on Saturday, March 29 at the Charleston Area Convention Center.

**RESULT**
**More than 300 women attended the conference** along with 42 exhibitors, 26 sponsors and 30 business community leaders.

**WHAT DOES THIS MEAN FOR YOU**
**In 2002, there were** **13,644**** **women owned businesses & self employed women in the tri-county area.  Census estimates that there are **now more than 20,000.**
- That’s why the conf is important, women can **gain insight into common challenges** & **build connections with other women business leaders. **

**OUTPUT**

**GOAL**

**ACTIVITY**

**OUTCOMES**

**PERF MEAS**

## ACTIVITYBased on your Logic Model and SMART Objectives, develop Performance Measures.

**Notes:**
Objectives are outcomes written smartly; PMs are the measurable parts of the objectives.  Good place to start, get thoughts going for brainstorming PMs.  Caution: can be low hanging fruit.  Choose PMs that are meaningful to your program.

## Evaluation

- Evaluation is the systematic collection of information about activities, characteristics, and outcomes of projects to make judgments about the project, improve effectiveness, and/or inform decisions about future programming.

- adapted from Patton, 1997

**Notes:**
Evaluation helps you make decisions.  The key decision = is this project needed?  Will we continue with this project?  We don’t want to spend the time and the money if it is not going to be used.

What are some reasons that we wouldn’t need to do an evaluation?
If program is on track and no further information is needed
If the evaluation is not going to make or change  a decision
If results are just going to be put in a binder on a shelf (filed away) and not used 

## Evaluation

- Analyze the effectiveness, efficiency, and appropriateness of programs, strategies, and investments in achieving the goals and outcomes of your program
- Identify outcomes, including unintended outcomes
- Establish why actions lead to or fail intended outcomes.

## Evaluation Continuum

- Anecdotal
- Performance Monitoring
- Evaluation
- Evaluation Research

**Evaluation Continuum**

**Notes:**
There is a continuum.... And the objective is to be at the appropriate place for the your project/program.  Most sites are collecting information (monitoring if systematic) – how can you best use that?
Some sites are getting into evaluation (example... Estuary Net??, )
Some sites are doing evaluation research

Lowest is no feedback or the evaluator not listening

One of the negative aspects of anecdotal evaluation is that there is no systematic way to evaluate the responses.  You can still have an evaluation form but it can be used for the wrong reasons.  For example, if I would like to include more activities in our trainings, I might single out the one evaluation/individual who felt the same way and say “see, our participants think we need more activities”!  Do not change aspects of a program due to one evaluation.  Another difficulty with anecdotal evaluation may be that a representative sample was not chosen resulting in bias.

Performance Monitoring is simply to find out if we are meeting our objectives it does not give us any information on causation.  The evaluation does that.  We cannot make drastic decisions about our program from performance measures.

Ex. Water quality monitoring::water quality research as performance measurement:: evaluation

Collecting PMs/data points is to evaluation as monitoring is to research.

Water quality monitoring shows you trends, consistencies...it doesn’t give you a cause or outcome, it just tells you what is happening and if you are on the right track.  The data points that you are collecting are the measures.  In order to find out why the results are showing what they are, you must do research, evaluate, and analyze the data...this is performance measurement.

Data gathered systematically so that we can observe trends.  

Research tells you the reason: dissolved oxygen levels declined  why?  Research, just like to find out why your performance measures weren’t on the right track, you’d have to conduct an evaluation.
The Evaluation Research component we are familiar with due to academia.  They are usually the ones to evaluate results.  Scientific evaluation.  They are usually the ones to do this because it takes a lot of time, effort, and money and they have the students to work on it and the money from grants.  E.R. is a key resource and helps to achieve and create benchmarks.  There are a variety of social science and education journals where the results can be found.  This shows causality

## Determines if you have met your objectives

Documents results that can be used to justify programs or build support

Identifies programs that need improvement

Reviews and reinforces essential program points

- Determines if you have met your objectives
- Documents results that can be used to justify programs or build support
- Identifies programs that need improvement
- Reviews and reinforces essential program points

## Take- Home Message

- Don’t make it more complicated than it is
- Use logic models as a planning tool to articulate how we are going to accomplish our goals
- Power is in the if-then statements
- Outcomes need to be within your sphere of _influence_ but not your _control_