# Presentation: 046590

**Measuring the Difference: Strategies for Improving and Evaluating Health Information Outreach Programs**
**Cathy Burroughs**
**Maryanne Blake**
**NN/LM, PNR**
***May 22, 2004***
***Annual Meeting***
***Medical Library Association***
***Washington D.C.***

- NN/LM-MR, Salt Lake City

**Notes:**
Good Morning!

Thank you for coming—some of you have come a long distance!  This will be a great class because you all bring different perspectives about outreach and we’d really like to give you a chance to learn from each other as we go through the class] 

We’re going to be talking about Strategies for Planning and Evaluation HIO and we have lots to cover

First, start with you--  introduce yourself, what you do, background with outreach,  interest in workshop

## Outreach Evaluation Resource Center 

- _[http://nnlm.gov/evaluation](http://nnlm.gov/evaluation)_
- NN/LM, Pacific Northwest Region
- University of Washington, Box 357155
- Seattle, WA  98195-7155
- _[cburroug@u.washington.edu](mailto:cburroug@u.washington.edu)_
- --206-543-9261
- _[blakema@u.washington.edu](mailto:Blakema@u.washington.edu)_
- --206-221-3405

**Notes:**
Introduce ourselves--- 

Me--  at the RML since 1997—hired to be a project manager for study with NLM about best practices in outreach evaluation.  

Measuring the Difference– resulted from this project.  

-- Approach to evaluation study --find out what other disciplines do for evaluation and what they’ve learned from their research.

Stellar advisory board and learned a lot from them.  More about that later.

Currently, help with evaluation planning for outreach funded by NLM.  Am doing  this workshop with Maryanne in all the NLM regions.  

Also, I’d like to be of assistance to individual projects for questions about your evaluation

Let’s talk about the workshop today--  NEXT SLIDE 

## Workshop Objectives

- Identify where and how evaluation fits in program development
- Select theories to strengthen outreach strategies
- Determine what to consider when planning outreach evaluation

**Notes:**
Workshop purpose --an introduction to outreach evaluation.  

Looks at the big picture of what can be done and tips on planning it into your outreach process. 

Also--  introduce theories dopted from health education—that helps to know how people learn, change and adopt new ideas .  

Will talk about the planning considerations-- evaluation objectives, design, and data collection.

And the class will brainstorm together on some evaluation exercises

All this will be done with reference to the evaluation guide–

Please look at the packets you have--  1) class slides, several exercises, and a few handouts

The day’s agenda will be......

Bathrooms are......

## Mission of Outreach

- 	To affect capacities of  target communities to address issues of health and health care through  effective access, use, and exchange of  health information resources.

**Notes:**
First, let’s talk about some assumptions we may bring to the topic of outreach and evaluation

Ask Class (before show definition)—What do you think are the goals of outreach---what’s the big picture goal? 

## Outreach Activities

***Promotional:*** exhibits, brochures, fact sheets
***Logistical:*** providing equip, facilitating connections, developing local resources, intermediated search services, ILL
***Educational:*** training classes, demos
***Collaborative: ***partnerships

**Notes:**
Given the general goal--here are some ways of categorizing what we do

TALK FROM SLIDE 

Ask if class can add others? 

## Why Evaluate?

- * Benefits *
- Justify program with management or funder
- Improve the outreach program
- Other??

**Notes:**
Why evaluate– isn’t  it enough to list what we’ve done?

What is the use of all the work?

Show evidence  of need for additional funding

Keeping track of  “lessons learned”

Other? 

Find out -- are we helping the community– from their perspective

Is this worth doing again?

## Why Not Evaluate?

- *Fears*
- Anxiety about  time and resources
- Unreasonable expectations
- Discourages program ideas that may “fail”
- Others?

**Notes:**
Fears:  

*too much funding and time

* it sets me up --  I have to say I’ll do everything and make a ton of difference

*keeps me from trying a new idea (solution:  call it exploratory)

Others?
Sensitive about “testing” clients

MAB--Ask Class to Share some of the Evaluation Experiences—what were some of the worst and some of their best experiences? 

## Best Practices in Outreach

- What Works?
- How Do We Know?

## Behavior Change Theories

- Success in attaining outreach
- 	objectives can be challenging
- Changing behavior patterns requires outreach activities that provide more than just information
- 		***Strategies are needed to help motivate, ******	******facilitate, and reinforce change***

**Notes:**
Outreach is a lot about influencing change

--getting people to select and use credible information resources is the ultimate hope

But a lot that goes into making that happen

We’ve learned that just giving out information won’t necessarily make a difference

Strategies are needed to help motivate, facilitate, and reinforce change

Theories give some guidance to the strategies you might use—they are sort of the underpinnings for your activities

Theories provide the backgrnd and context for understanding how and why people behave the way we do, and what helps us make changes.

E.g. you might be developing a website

How will you design get people to use or value it in their work (that’s your outcome)   Use theories!  

## Outcomes=Results or Changes

- By individual/organization/community:
- Awareness
- Knowledge
- Attitudes and perceptions
- Skills
- Behavior
- Priorities, policy, support

**Notes:**
Briefly—these are the general sort of outcomes that are important in outreach if we want to facilitate change

Outcomes address what will change, or hope to be different, on the part of the audience or community we work with.  

Having access to the Internet is not an outcome... it’s a mechanism

Increased awareness, knowledge, and skill in using a resource on the Internet is one kind of outcome

Changes in how web resources are used (web-use behavior)  are outcomes.... E.g.  Frequency, type of resource used, reason for access

Ways that awareness, skill, use of web resources might impact decisions and action by individuals or an organizaton are the long range outcomes

There are theories that will help understand why people make changes.... NEXT

## Factors Affecting Behavior

***Predisposing******:*** provide the motivation or reason behind a behavior (knowledge, attitudes, beliefs)
***Enabling******:*** make it possible for a motivation to be realized (available resources, skills, info services, ability to obtain resources)
***Reinforcing******:*** reward a behavior (opinion leaders, community level participation)

**Notes:**
Theories talk about factors that affect behavior change

These can be grouped in to 3 categories 

## 4 Theories

- Social Learning Theory
- Stages of Change Model
- _[http://nnlm.gov/pnr/eval/witte.html](http://nnlm.gov/pnr/eval/witte.html)_
- Diffusion of Innovations Model
- _http://nnlm.gov/pnr/eval/rogers.html_
- Community Organization

**Notes:**
These theories identify variables that explain a behavior – how things work

SO, theories become a guide to use in our strategies because we can better understand what we do that might influence change

## Social Learning Theory

- Albert Bandura (1970s), also called social cognitive theory
- Factors that affect behavior change include:
- 	- behavioral capability
- 	- outcome expectations
- 	- self-efficacy
- 	- observational learning

**Notes:**
Cathy --Explain the Chart above

Behavioral capability--  knowledge of what to do and how to do it

Outcome expectations—beliefs about likely results

Self efficacy– confidence in ability to take action

Observational learning– beliefs based on observing others like self

Bandura believes self-efficacy is the most important factor
It influences:
The choices we make
The effort we put forth
How long we persist when there are obstacles
How we feel

Refer class to page 25  for self-efficacy techniques

## Stages of Change Model

- 5-stage framework for explaining how behavior change occurs
- People at different points in the process of change can benefit from
- 	different interventions, matched
- 	to their stage at that time

**Notes:**
Idea is that 

Giving out info won’t work if it’s not the “right stage”

## 5 Stages

***Precontemplation:******	***Not thinking of it
***Contemplation:******	***Thinks about it
***Preparation:******	***Makes plans
***Action:******	******	***Uses [it]
***Maintenance:******	***Continues new behavior

**Notes:**
How many of you know someone who refuses—won’t even think about—learning to use some kind of technology?  
My uncle ---  refused to get a computer and be connected to his family via email for several years after his wife died
Maryanne’s mom didn’t even want to consider getting money from an ATM
But, they might change—my uncle, in fact, did.  But, it won’t be from just telling them about email or the ATM. 
There are stages of change.
Think about a change of behavior you’ve made in your life.  It might be health related—stopping smoking or losing weight.  Maybe it was a decision to change careers.
Anyone want to share what change they are thinking about?
Whatever it was—there were some stages you can probably remember.
Thinking about whatever change you made, can anyone recall what made you want to –why did you even start thinking about it? 
Others of you—when you were making a change, did you have a phase where you gathered information—where you needed to prepare yourself?  What information did you neeed?
Once you had educated yourself, what made you take the next step—to actually take action?
Others—thinking about the change you’ve made, what has helped you keep it up and not revert back to the old days?
[reinforcement]
These stages of change can be useful when thinking about your outreach audience--  don’t expect too much and try to accommodate the level of info that they can take in or are ready for.  
A demo might be more helpful than instruction if you are trying to get them to think about using technology
The idea of getting to them at the time of need—just in time—might mean you will be offering several classes at different levels—and repeat them, rather than a one-shot deal.

## Diffusion of Innovations

- Everett Rogers (1940s)
- Developed the theory of how new ideas or products spread within a society or from one society to another

**Notes:**
Slide 18 – Everett Rogers developed the theory working with farmers and new varieties of corn and how the use of these varieties diffused through that part of society.  

Story to tell:.

## Key Diffusion Principles

- People adopt an innovation because people they respect endorse it (not due to experts)
- Innovators are first to adopt (2.5% of pop)
- Early Adopters are second (13.5% of pop)
- Early Majority (34)
- Late Majority (34%)
- Laggards (16%)

**Notes:**
Slide 19 – Key Diffusion Principles
People adopt innovations not on basis of scientific research by experts (nerds) but because people they know and respect endorse it.  These people are the opinion leaders or the early adopters.  
Adopted first by innovators (2.5%); then by early adopters (13.5%.)
The triangle is: 
innovators = 2.5%
Early adopters = 13.5%
Early majority = 34%
Late majority = 34%
Laggards = 16%
When critical mass is reached enough people have adopted the technology so that the rate of adoption becomes self-sustaining.  
The idea in outreach is to target the early adopters and the opinion leaders; they will spread the technology (or idea) for you.

## People are more likely to adopt an innovation if it has:

- Advantage
- Compatibility
- Complexity
- Trialability
- Observability

**Notes:**
Slide 20 – Influences to adoption of technology – what are the outreach applications
Advantage (relative advantage) – idea is better than the one it replaces
Compatibility – how it fits in with the lifestyle, values, habits, experience and needs of potential adopters (many physicians don’t adopt new ideas since they don’t know how to fit it into their practice)
Complexity – how difficult is the technology to use or understand and learn
Trialability – how much can you experiment with it before committing to adopt
Observability – how much does this technology provide tangible results
What does this mean when you plan an activity? 
1 – target key people
2 – show advantages
3 – show how it can be compatible with people’s lifestyles
5 – let people observe results by trying it

## Community Organization

- Empowerment
- Community competence
- Participation and relevance
- Issue selection

**Notes:**
Community Organization is not a theory in itself

It’s helpful for outreach goals to influence changes in attitudes, perceptions, and decisions about supporting health information access at the organizational or community level

It has a grass roots context... you as the outreach director may have your own opinions for  how health information resources might benefit a  community or organization.  But, your thinking about why a community or organization “should” know about, promote, or support access to health resources will not be as effective as when the community or organization perceives that type of thinking as relevant to issues and priorities of importance to them.

Strictly speaking, CO is a process by which groups within a community identify common problems or goals, mobilize resources, and implement strategies for reaching those goals

The conceptual framework for community organization is based on public health initiatives that recognize that changes in behavior depend on more than just factors related to individuals (such as motivation, readiness to change, or other attitudes and beliefs).

Individual changes are activated, supported, focused, or otherwise validated by a sense of community involvement.  

Outreach programs may not be about grassroots community organization.

However, if you are able to involve  programs or groups with similar overall interests in ways that leverage  their participation, opportunities will emerge to make outcomes relevant and sustainable.

Empowerment:  gives communities tools and responsibility for making decisions that affect them (you’re not giving them the solution.. )

Community competence:  as groups w/ similar or cross cutting interests see each their and others’ roles toward problem solving, they gain a sense of mastery (e.g. partnerships between patient records, patient education, clinical services, and the library—the IAMS model)

Participation & relevance:  goals and activities are integrated into already existing programs or initiatives (e.g. information skills training w/ in existing medical school curriculum)

Issue selection:  Identifying winnable, simple, specific concerns...I.e. being realistic and getting to success.  

## Phases of Evaluation

- Community Assessment
- Audience Assessment
- Process Evaluation
- Summative Evaluation

**Notes:**

Having talked about the ultimate changes we hope to make– the outcomes we’d like to see– as a result of outreach.

We’ve also talked some about strategies to make those changes... the underlying theories.

Now we’re ready to plan the outreach and evaluate the outcomes, right?

Well... evaluation is more than evaluating outcomes.  There are different phases that will help you to plan your program, track it’s progress, and collect data that will help to improve and demonstrate its success.  

It’s never too late to work in evaluation—but it’s best to begin evaluation planning at the start:

community assessment--to understand the context of the group being reached, and to develop mutual goals for ways that outreach can help.

Audience assessment—provides information about the specific group targeted to an outreach activity

Process eval—helps to spot areas for improvement

document what really is happening

Check on progress toward objectives

Summative eval—overall assessment of the whole program, such as whether objectives were reached and what happened as a result

You don’t have to collect a lot of data in all of these phases --  some types of data will be easy (such as tracking the number of trainings you conduct).  Other times, depending on the project, you may need to spend a lot more time on the community assessment before you can even make relevant plans.

It’s up to you to decide what you most want to find out.  

## Planning and Evaluating Outreach

- Identify Target Community

- Conduct Community Profile

- Establish Goals & Objectives

- Develop Activities and Strategies

- Conduct Audience Assessment to Tailor Strategies

- Develop a Plan to Implement Activities

- Establish Evaluation Objectives (Process or Summative)
- Select Design and Data Collection Methods
- Develop Evaluation Instruments

- Stage Two

- Stage Three

- Carry out Plan       to Implement Outreach Activities

- Carry out Plan for Process Evaluation to Monitor and Improve Activities

- Carry out Plan for Summative Evaluation to Assess Progress and Impact

- Analyze Results for Answers to Evaluation Objectives

- Share Results and Modify Program as Needed

- Stage Five

- Stage Six

_**Planning and Evaluating Outreach**_

- Stage Four

**Notes:**

AFTER THRU FLOW CHART

The entire evaluation process we cover does not apply to all programs.  

You will need to extrapolate to your situations--  

You might be testing something new –

Or, conducting a usability study to plan and refine a new website 

Each of these could be an evaluation in itself

But, at the overview level, we’ll follow evaluation in all stages of program development in the manual. 

## Evaluation Starts with Assessing and UnderstandingCommunity Needs

**Notes:**
MAB Start Here

An overall look at your outreach community is the first step in program development and the first type of evaluation you will use

This is the community assessment phase--  on page 1 of the guide is a flowchart

## Planning and Evaluating Outreach

- Identify Target Community

- Conduct Community Profile

- Establish Goals & Objectives

- Develop Activities and Strategies

- Conduct Audience Assessment to Tailor Strategies

- Develop a Plan to Implement Activities

- Establish Evaluation Objectives (Process or Summative)
- Select Design and Data Collection Methods
- Develop Evaluation Instruments

- Stage Two

- Stage Three

- Carry out Plan       to Implement Outreach Activities

- Carry out Plan for Process Evaluation to Monitor and Improve Activities

- Carry out Plan for Summative Evaluation to Assess Progress and Impact

- Analyze Results for Answers to Evaluation Objectives

- Share Results and Modify Program as Needed

- Stage Five

- Stage Six

- _Planning and Evaluating Outreach_

- Stage Four

## Identify Target Community

- 	***Community:*** group of individuals who share certain characteristics, e.g., occupation, culture, geographic location

**Notes:**

Community assessment --  Needs Assessment;  similar meanings---CA is broader

The idea of a community assessment ---to understand your clients and their situation and environment

What do we mean by “community”?

 Community is a broad concept, but loosely it is a group of individuals who share something in common:

Live in the same area
Share the same culture
Have common interest or concern—AIDS, e.g.
Share an occupation or sense of identity

## Types of Communities

- Rural or urban underserved areas
- Tribal community
- Rural primary care settings
- AIDS workers
- Public health workers
- K-12 nurses, librarians, teachers

**Notes:**
When you are identifying your community—

It may be a  place (rural county) or cultural community (tribal reservation or Hispanic neighborhood)

Or, your community might be a group that  share  an occupation or —such as  health aides or school nurses

Or, maybe it’s youth or seniors

Whatever group it is—the idea of a community assessment is to look at the factors that help to define their situation, needs, and problems in accessing health information.   

## Identify Target Communities

- What populations would you like to reach?
- Who can you best reach and influence?
- What groups could benefit most from health information access?
- What community is “ready” for outreach?
- What community is most in need?
- Are there potential partners to work with you?

**Notes:**
Think about
	 Groups that you are not serving that you would like to reach;

Who could benefit most:
Readiness in terms of:  
technology, computers, connections, etc
structure and availability of health and social services in place, 
information resources availability

Partners may be a target community as well, they may be the recipients of your information outreach and 
Their clients get the benefit of the outreach you have done with the partner 

Partners are a distribution channel
Partners could be 
groups of health professionals,
community based organizations
public libraries

## Topics of Interest  

- Demographic, economic, social and health issues
- Current information needs
- Barriers to accessing health information
- Cultural characteristics
- Resources available

**Notes:**

It’s important not only to identify your community, but find out things about them that will help to plan the outreach project.

The topics that a community assessment focuses on is similar to a needs assessment, but you gain a broader understanding about the issues that may influence what or how you conduct your project.

You want to identify what’s important to them, what they have going for them that might be incorporated into your program, their values and beliefs, as well as the health topics they deal with and may need help in accessing.

In this process you may not only identify the barriers to information access, but also the causes or reasons for  those barriers.  

You also are involving the community and avoiding planning from the desk...  people will be more likely to participate if they have a say in what is needed and should be done.

Can Anyone think of things about your audience you were glad to find out, or wish you had known?

Example:  Native Americans have a historical relationships with the govt that bring up major issues of trust.  
Feel over-researched, but that not much is given back.  
Health information specifically for or created by Indians  is  spotty – 
Iit’s important not to think and  come across like we know best what they need—
Rather we should learn from them about ways we can help them ---which could be to create or develop their own resources for support

## Use secondary data from national/local sources
Use the literature
Obtain user input

- Use the literature
- Obtain user input

- Conduct Research

**Notes:**
How do you conduct a community assessment

Think about what data already exists
--Census data for demographics down to the county level
--Maybe surveys that have been done by other local organizations
--do lit searches to see what else is out there 

User input--   important to get perspectives first hand

## Rigorous Approach

- To make generalizations
- Survey – typical instrument
- Reliability, validity, random sampling

**Notes:**
Extensive data collection means you are hoping to make some generalizations—e.g. teens look to their peers for health information

This type of research requires rigor to be sure its statistically valid and reliable

Random sampling is also important, so that all people being researched have an equal chance of responding. 

This is really major research about a community’s needs—not needed or appropriate in most situations. 

For tips on conducting surveys, see pp. 12-13 in guide

## Practical Approach

- To gain an overall  understanding of targeted group and their environment
- Key informant interviews
- Focus groups with stakeholders
- Open-ended survey questions

**Notes:**
Intensive—is the most practical way to get data—you aren’t making generalizations—not getting an overall impression

You are talking to key contacts—stakeholders who have an interest in the problems of outreach or health care 

Provide the Handout on Questions from Behavior Change Theory

“Who” talk to depends on your community

E.g. health providers, administrators, professional health association, the public—

For TCP, we talked to tribal leaders, system admin, health providers, dept heads—large variety

People who help to give you an overall picture of the topics of interest

## Utilizing the Results

- Understand causes for information problems and needs
- Reach a shared vision
- Develop hoped for outcomes
- Look for measures to pre-assess

**Notes:**
Talk from slides

What will you do with this information?

 Figure out how outreach could help—plus what is important to your audience 
  Get feedback and ideas from your target audience
   Create a shared outreach vision 
  Become aware of factors to consider in your plannning
  Identify causes of barriers--  such as attitudes of administration 

Perhaps identify what might be needed to better understand the information needs or behaviors that you hope to address...e.g. with Public Librarians, we’ve learned from previous outreach projects, from talking to librarians at exhibits, and from a general overall sense of things, that public librarians may at first need and want training and skills in providing health reference service. 

Whereas, in trying determine what outreach program would best fit to the public health workforce, the direction is not as clear.  So, there may be a need for more extensive needs assessment in this situation.

We’ve developed a case scenario that we’ll build on throughout the class

It’s a pretend outreach scenario, but based on some actual outreach experience– so some of this may sound familiar.  

MAB?

## Case Scenario

- Outreach to School Nurses

**Notes:**
After MAB introduces the case scenario is introduced, ask class to suggest 

Questions that might be asked to find the information presented in the scenario

Also, where to find this info---what sources (where, what, who)

Don’t ask for outcomes, yet...wait until just before the Breakout Session

## Planning and Evaluating Outreach

- Identify Target Community

- Conduct Community Profile

- Establish Goals & Objectives

- Develop Activities and Strategies

- Conduct Audience Assessment to Tailor Strategies

- Develop a Plan to Implement Activities

- Establish Evaluation Objectives (Process or Summative)
- Select Design and Data Collection Methods
- Develop Evaluation Instruments

- Stage Two

- Stage Three

- Stage Four

- Carry out Plan       to Implement Outreach Activities

- Carry out Plan for Process Evaluation to Monitor and Improve Activities

- Carry out Plan for Summative Evaluation to Assess Progress and Impact

- Analyze Results for Answers to Evaluation Objectives

- Share Results and Modify Program as Needed

- Stage Five

- Stage Six

- _Planning and Evaluating Outreach_

## Developing Goals and Objectives

- “If you don’t know where you’re going, you’ll end up somewhere else.”
-  Yogi Berra

## Setting Goals

- What effect will the program  have?
- What overall improvement is needed?
- What are the goals of my targeted audience?

**Notes:**
WHAT IS A GOAL
A goal is a broad definition of what will ideally happen as a result of your program

   usually long term and a little out of reach
   represent the ideal.  

In the long run, what effect do I hope to have on information access problems for this community? 

What is the overall improvement I want to achieve?

Example Goals on NEXT SLIDE
          

## Sample Goal

- To improve the quality of health care provided by school nurses via effective access and use of credible health information resources.

## Objectives

- Steps important to reaching the goals
- Define what will be done or the outcomes that will hopefully result
- Should be measurable

**Notes:**
Objectives are the steps considered essential to realizing the goals.  

The key is to make them measurable– to think ahead about how – if you carry them out--- they will make a difference

If objectives are unrealistically optimistic or are not measurable, it is harder to know how successful you are, 

Objectives are the steps considered essential to realizing the goals.  

General objectives provide an overall direction, and focus the priorities

Assists in determining program priorities

The key is to make them measurable– to think ahead about how – if you carry them out--- you will be able to tell

If objectives are unrealistically optimistic or are not measurable, it is harder to know how successful you are, 

## Two Types of Objectives

- Process objective:
- Specifies what outreach staff will do – is a process statement
- Outcome objective:
-  Includes statements of change – such as what is hoped to change on the part of outreach participants or in the community

**Notes:**
There are two main categories of objectives—known as process and outcomes 

Process objectives state specific and measurable activities, products, services to be accomplished

Outcomes objectives include statements of change

Process objectives are the ones that say what staff will do —i.e. conduct X number of skills training workshops.  

Process obj—sort of the map of activities to follow along

Outcomes-based objectives measure the results or differences that outreach has made on the targeted audience.  

LOOK AT THE HANDOUT –”Typical Objectives”

## Sample Process Objective

- Within six months, project staff will develop and maintain a website for school nurses.

## Making Process Objectives Measurable

- Process objectives list specific items that can be tracked or measured, such as:
*What: *The action, product, service to be accomplished, developed, or maintained.  Sometimes called the “*output*.”
*When: *A date or time period
*Target population:  *To or for whom

## Outcomes=Results or Changes

- By individual/organization/community:
- Awareness
- Knowledge
- Attitudes and perceptions
- Skills
- Behavior
- Priorities, policy, support

**Notes:**
Briefly—these are the general sort of outcomes that are important in outreach if we want to facilitate change

Outcomes address what will change, or hope to be different, on the part of the audience or community we work with.  

Having access to the Internet is not an outcome... it’s a mechanism

Increased awareness, knowledge, and skill in using a resource on the Internet is one kind of outcome

Changes in how web resources are used (web-use behavior)  are outcomes.... E.g.  Frequency, type of resource used, reason for access

Ways that awareness, skill, use of web resources might impact decisions and action by individuals or an organizaton are the long range outcomes

There are theories that will help understand why people make changes.... NEXT

## Outcome Objective

- Focuses on the intended results of the program’s activities
- Statement of intended benefits to people: achievements or changes in skill, knowledge, attitude, behavior, condition, or life status for program participants

## Types of Outcomes Objectives

- _Change Statements_ are outcomes that identify the change or desired effect in what or by whom.
-  _Targets_ are “change statements” with specific levels of achievement
-  _Benchmarks_ make comparisons to other time periods or organizations

**Notes:**

Change statements clarify what results or difference are intended due to the intervention or activity done or effort made.  

LOOK AT THE HANDOUT –”Typical Objectives”

## Example Types of Outcomes:

- _Change statement_  *More school nurse participants use the school nurse website for health information*
- _Target_  *75% more school nurse participants use the school nurse website for health information*
- _Benchmark_ *Compared to school nurses in the 2 county area, 80% more school nurse participants prefer to use the school nurse website*

**Notes:**
Change Statements are outcomes that identify the change or desired effect in what or by whom.  

 Targets are “change statements” with specific levels of achievement

 Benchmarks make comparisons to other time periods or organizations

## To Measure an Outcome—Identify its “Indicator”!

- When figuring out what to measure, determine the desired effect (change) stated in the outcome objective
- Then, gather data on indicators that provide some type of evidence that the change has occurred.

**Notes:**
NEED SOME EXAMPLES!!!

## Example Indicator

- Outcome: *More school nurses use the school nurse website for health information*
- Indicator(s) for use:
- Visits to school nurse website
- Number of nurses who have bookmarked  the school nurse website
- Frequency of visits to website

## Drafting the Objectives

- Review Community Assessment—look at data to determine what you think needs to change.  Remember to consider what the community wants to get from an outreach program

**Notes:**
NEXT—create slide about developing statements of activities.  Discuss how they can be made very concrete by adding project milestones.  

## Drafting the Objectives, cont.

- Then, select priority outcomes – the hoped for changes in knowledge, behavior, attitude, or skill by program participants-- *write out the outcomes objectives*
- Think about what activities, products, or services are needed to effect that *change—then write out the process objectives*

## BREAKOUT GROUPS -- EXERCISE #1

**Notes:**
BEFORE BREAKOUTS

Do SECOND Part of Case Scenario (having class list outcomes and outputs)

THEN, ASSIGN OUTCOMES FOR BREAKOUT GROUPS

## Planning and Evaluating Outreach

- Identify Target Community

- Conduct Community Profile

- Establish Goals & Objectives

- Develop Activities and Strategies

- Conduct Audience Assessment to Tailor Strategies

- Develop a Plan to Implement Activities

- Establish Evaluation Objectives (Process or Summative)
- Select Design and Data Collection Methods
- Develop Evaluation Instruments

- Stage Two

- Stage Three

- Carry out Plan       to Implement Outreach Activities

- Carry out Plan for Process Evaluation to Monitor and Improve Activities

- Carry out Plan for Summative Evaluation to Assess Progress and Impact

- Analyze Results for Answers to Evaluation Objectives

- Share Results and Modify Program as Needed

- Stage Five

- Stage Six

- _Planning and Evaluating Outreach_

- Stage Four

## Planning for Activities

- Review what you will do– e.g. your process or activities
- Think about ways that theories may help you be more effective (Stage 3 Tool Kit, page 37 )
- Review the literature for other best practices (Appendix K)

**Notes:**
Theories suggest both
 program strategies
Classroom techniques

Program strategies example:
Recruit opinion leaders to help promote and advocate use of health information resources (--endorse training classes, maybe even participate as co-teachers)

Classroom technique:
Demonstrate searches that are very relevant to audience needs (Self-efficacy)

Refer class to Best Practices in Appendix K on Page XX

## Think Through Theory + Strategy = Outcome

- Reach outcomes using  strategies based on theory and best practice
- See Appendix I

**Notes:**
Theories suggest both
 program strategies
Classroom techniques
Program strategies example:
To link these theories to your strategies– you are simply looking ahead at what outcomes you will expect and then applying theory based strategies to get there

USE OUTCOMES ON FLIP CHART

ASK CLASS TO PAIR UP AND THINK ABOUT THEIR GROUP”s OUTCOME FROM EXERCISE 1.  WHAT THEORIES MIGHT BE USED TO  HELP ACHIEVE THEM AND HOW?

EXAMPLE:  Desired outcome is Increased Awareness of Credible Health Information Resources:  Using Stages of Change, determine sense of need and readiness for making health information choices;  work w/ community advisors to develop messages that promote awareness and spark interest in action (such as signing up for a class).  

## Implementation Plan

- A blueprint to clarify, document, and schedule your objectives and plans

**Notes:**
Provides a blueprint
Helps clarify
Useful for scheduling
Document for future use
Good communication tool

## Creating an Implementation Plan

- Describe the overall community and its needs
- List program goals
- List hoped for outcomes
- List objectives
- Describe strategies to be used
- Specify activities
- Include a timeline

## Introducing the Logic Model

- Framework for program and evaluation
- Fluid
- Changes over time
- Makes evaluation easier

**Notes:**

Template, framework for describing a proposed or existing project or program

Logic models can be fluid – used to monitor progress and process; 

observations may suggest need to change, new approach to ensure desired
 results will be received.

Used for program planning; evaluation plans are built on the program logic model

## Logic Models May Change Over Time

**Notes:**
logic models as fluid and evolving

Changing resources may influence appropriate activities which in turn will affect outputs and outcomes. As outcomes are realized new resources and or activities may be identified.

## Plan BackwardImplement Forward

- Outcomes
- Activities
- Resources

**Notes:**
You need to know where you want to go or what you want to do (hike? Sun? sail? Visit museums) before you can decide how to get there. 

Much like planning a vacation. You hve to know the destination before you decide whether to drive or fly, when, where you’ll stay.

## Parts of the Logic Model

- Inputs or Resources
- What will you need to carry out your activities?
- Activities
- What will you do?
- Outputs
- How many of what will your activities produce?
- Outcomes
- So what – the difference your program makes, the benefits that accrue because of your program

**Notes:**
Resources – what you already have and what you will need. If you don’t have the resources you need how will you get them?

Activities – be very specific. While it seems tedious to actually write all the activities remember that logic models can be used to support your plans to readers who may not know the details that you do.

Outputs: Librarians are good at counting and outputs are just that – how many – questions answers, books purchased, shelved, loaned, ILLs filled.
Think outside the traditional statistics – what about orders submitted, checks written, ethernet cables run, accounts created

Outcomes – this is the so what. Again, be specific and think about how you’ll know that you succeeded. What changed because you conducted this program or provided this service. Who will benefit? How? Who do you WANT to benefit? Your activities will be designed to accomplish these outcomes

This is the Plan backward, implement forward If you know what outcomes you are looking for – the why of doing this program – then you can plan the activities, figure out the outputs, and decide what resources you’ll need to be able to pull it all off.

## Outcomes

- Outcomes should be
- specific
- measurable
- action oriented
- realistic
- timed

**Notes:**
Desired outcome:

Health professionals will use published literature more often to answer questions about patient care when electronic journals are available.

The hospital will spend less on binding and replacing lost copies.

Does it meet the criteria?

## Outputs

- Outputs are what the program “puts out”!
- They are measures of accomplishment
- E.g.-- counts of services or programs delivered, produced or implemented
- Outputs are sometimes confused with *outcomes* (which are different behaviors, decisions, actions, attitudes or other changes by outreach participants)

## Sample Logic Model

| RESOURCES | ACTIVITIES/PROCESS | OUTPUTS | IMMEDIATE
OUTCOMES |
| --- | --- | --- | --- |
| Project librarians
School nurse colleagues on email
School district librarians
Reference librarians at the project library
NN/LM PNR librarians | a. To prepare, a skills assessment and survey will be done to better understand the scope needed for the class, what will be taught, and what skills will be evaluated to assess immediate outcomes
b. Project librarians will develop and present Web searching skills training sessions and additional training materials* for school nurses.  One aspect of the training will be information and exercises in how to tell high quality information on the Web, especially health information.
*These materials would include a set of selective, quality health information bookmarks in various media, and a “cheat sheet” of indicators for recognizing quality Web resources.
c. Meet with school district librarians to explain the project. Invite them to the class, if they are interested, explaining that the emphasis will be on finding health information.
d. Analyze post-training evaluations and feedback; indicate modifications to the training tools if necessary | Contacts with school district librarians
Class materials developed
Set of health information bookmarks on CD, disk, paper and Web developed
Web resource quality indicators “cheat sheet” developed
X school nurses trained to find high quality health information on the Web | Nurses can demonstrate the ability to recognize at least 4 indicators of quality when searching for health information resources on the Web. |

**Notes:**
Remember  - it’s built backwards. What do you want to change or what benefit are you looking to provide. Then, what will it take to get there?
Outcomes are 
	specific
	Measurable
	Action oriented
	realistic

## Planning and Evaluating Outreach

- Identify Target Community

- Conduct Community Profile

- Establish Goals & Objectives

- Develop Activities and Strategies

- Conduct Audience Assessment to Tailor Strategies

- Develop a Plan to Implement Activities

- Establish Evaluation Objectives (Process or Summative)
- Select Design and Data Collection Methods
- Develop Evaluation Instruments

- Stage Two

- Stage Three

- Stage Four

- Carry out Plan       to Implement Outreach Activities

- Carry out Plan for Process Evaluation to Monitor and Improve Activities

- Carry out Plan for Summative Evaluation to Assess Progress and Impact

- Analyze Results for Answers to Evaluation Objectives

- Share Results and Modify Program as Needed

- Stage Five

- Stage Six

- _Planning and Evaluating Outreach_

## How Do I Evaluate My Program?

- Different evaluation phases address different questions (Figure 11, page 43)

## Basics of an Evaluation Plan

- Questions or issues you will address in the evaluation
- What you will measure, how, when?

**Notes:**
One of the most challenging aspects of evaluation is figuring out your evaluation objectives—the questions you want to address

Because, evaluation can be resource intensive, you want to think ahead and prioritize your evaluation questions.  

The goal is to avoid collecting data that doesn’t in the end provide what you need.

LOOK AT PROGRAM FLOW CHART—p. 43

## Consider the Evaluation Interests of Stakeholders

- Funding agency
- Targeted community
- Outreach staff

**Notes:**
What questions will you ask?  

Consider who is most likely to need and use the information collected and for what purposes. 

## Interests of Funder

- Were objectives reached?
- What else happened re: benefits, changes or impact (positive or negative) ?
- What worked the best ?
- What was least successful?
- How satisfied were participants?
- How much time and resources were involved?

**Notes:**
What describes the community, its needs, resources and context 

What was hoped to be accomplished by the program or its services and products? 

What is the impact of the program, services, or products?

How much time and resources were involved?  

What lessons were learned?

How will data impact program decisions?

## Interests of Outreach Staff

- Were process objectives realistic?
- If not, why not?
- What worked the best ?
- What was least successful?
- How successful was the program from the perspective of the community?

**Notes:**
Did plans use realistic assumptions?\]

What difference did the program make?   

What outcomes were unexpected ?  

How can the program, products, services improve?  

How satisfied were participants?

How successful was the program from the perspective of the community?

## Interests of Community

- How successful is the program, product, service for me or my community?
- What are the benefits of participating in this program or using this service?
- What improvements will be made?

## Process (Formative) Evaluation

- How are we doing in reaching our objectives?
- Is outreach working as intended?
- How can it be improved?

**Notes:**
Keeps track of a program as it is happening so that modifications and improvements can be made on an ongoing basis 

Ask Class for suggestions

Answers the OVERALL Questions, such as Are Process Objectives Realistic?; Are we ON Track?  What Worked the Best, the Least, What Improvements are Needed

Are activities being scheduled and promoted?
Is equipment being ordered and delivered on time?
How much is being spent?
What is taking the most time?

Are educational objectives being met?
Are participant counts on track?

ALLOWS FROM IMPROVEMENT MID-PROGRAM, e.g. when doing a class satisfaction survey, or a mid-program meeting of the partners

What can be improved?
What is the most valuable– and the least valuable 

How can the program improve?

TALK FROM SLIDE

## Summative (Outcome) Evaluation

- Helps determine what outreach accomplished
- Evaluation questions address:
- Were process objectives met?
- What difference (outcomes) resulted?

**Notes:**
Looks at original objectives to see what was accomplished

Helps answer “So What?”– trying to show ways the project made an impact

This is where thinking about outcomes in advance will help answer that question  

Are the outcomes beneficial or not? 

Any unintended outcomes?

TO DO---  List the Outcomes Questions from the OERL Site

## Choosing the Important Evaluation Questions

- Prioritize by figuring out “I need to know ____because I need to decide_____.”
- Consider the benefits and costs of answering each question

**Notes:**

There is rarely enough time or resources to answer all questions about program practice and effects

Prioritize by asking “I need to know ____because I need to decide_____.”

E.g need to know what was most valuable and what was least because I will be revising the class or approach

E.g. need to know how many think the website if useful so we can justify maintaining it

E.g. need to know what the impact of success is on our partners (e.g. public libraries) so we can justify the need for more resource sharing

## What Will You Measure & How?

- Decisions about data collection, such as:
- What *indicators* and *outputs* need to be measured
- What type of data will be gathered
- When and from whom (or what) measurements are gathered

**Notes:**

Work on examples of Indicators and outputs

OUTCOME—might be changing the perception of time wasted using the Internet.   Indicator...rating a site as valuable to work performance.  

Regardless of the purposes for your questions, data should be collected in a systematic way.  

There are two considerations for evaluation design:

1. What type of data is collected
2. How data is collected 

Data may be quantitative or qualitative (as described next)

## Quantitative Data

- Quantitative--precoded responses
- Produce hard data such as counts, ratings or scores
- For example
- numbers of outreach participants
-  Percentage of users satisfied with class instruction
- Number of website hits by school nurses
- Percentages of users who indicate increased use in a follow-up survey

**Notes:**
Quantitative research methods were originally developed in the natural sciences to study natural phenomena. 

Often, our studies are exploratory... trying to understand how or why

Quantitative is useful often for tracking something concrete that has some meaning for comparisons.  

E.g. educational assessment use reading scores as a benchmark for comparing different teaching methods or curriculum

Libraries don’t have that same kind of quantitative measure.... Turnstile counts, website hits... they do indicate something—people are coming into the library or finding your website.  But, they do not give you much information on the quality of impact on their information need

LOOK AT OERL RE: Questions about When to Use a Questionnaire

Written questionnaires are typically used to gather quantitative data, whether informally via a feedback questionnaire, or through a statistically valid survey. 

TALK FROM SLIDE

## Qualitative Data

- In-depth, open-ended interviews
- Direct observation
- Open-ended survey questions, diaries, records
- Develops an understanding--what is happening
- E.g.  How is the school nurse website being used?
- What kind of impact does its use have in the school nurse program?

**Notes:**
Qualitative research methods were developed in the social sciences to enable researchers to study social and cultural phenomena. 

Qualitative methods are concerned with recording feelings, experiences, and impressions according to the subjects’ own words, either spoken or written. 

Qualitative data sources include observation, interviews and questionnaires, documents and texts, and the researcher’s impressions and reactions. 

(Examples of qualitative
methods are action research, case study research and ethnography.)

## What Are the Data Sources?

- Self Report
- Written questionnaires
- Interviews
- Observations
- Focus groups
- Meetings
- Records

**Notes:**
Another consideration for evaluation planning is the type of data sources you’ll use.  

Quick review:  

Questionnaires—challenge is to write questions that get the information you want.  Lots of room for misinterpretation.  Always good to pretest your questions on a few people.  More tips on questionnaires are in Stage 1 Toolkit and Appendix B

Mention Dillman

Interviews - questions may be predetermined, but the interviewer is free to pursue interesting responses.  

Focus group interviews take advantage of small group dynamics (usually eight to twelve individuals).  

The open-ended nature of interviews or focus groups allows participants to provide answers in their own words and allows researchers to better understand issues from the perspective of the audience.

Mention Glitz and Berg resources

Observations—advantage is data is collected in a natural setting. There can be protocols about who or what to observe, when and how long, and the method of recording the information (e.g., a questionnaire or tally sheet).  Or, and observer can simply record what he or she notices.

Records could be sign-in sheets, interlibrary loan tallies, document service requests, computer log files.

Meetings are a good source of information for planning stages of a program.  For example, a meeting with contacts of the targeted audience and outreach staff will be helpful for effective planning of the implementation and evaluation.  

Turn to Figure 15 on page 61

## Quality Control

- Validity—does the instrument measure what was intended
- Reliability—is the answer always consistent
- Cultural appropriateness

**Notes:**
Part of the quality of research you conduct depends on the validity and reliability of the instruments used to collect data 

Validity refers to whether the instrument accurately measures what was intended. 

An issue of validity might be whether you think a follow-up questionnaire can accurately measure use of PubMed for clinical decision making.  Respondents may want to answer in a way that will reflect well on themselves, while not being very realistic.  (Halo effect)

Reliability is a measure of the consistency of the data collection instrument.  A reliable instrument gives the same (or nearly the same) result every time.   

You might question the reliability of a question to remember the number of searches done last Fall quarter

There are other challenged to collecting data, such as cultural factors.

Many ethnic groups feel over-researched, such as  American Indians/Alaska Natives and African Americans.  

They are  skeptical or mistrustful of the evaluation process.  
	Their experience -- social scientists enter their communities and collect data, but frequently fail to share their findings or take visible and beneficial action. 

 In Hispanic communities, evaluators are viewed with suspicion as outsiders who conduct sterile research only to justify the shutdown of needed projects or services (5).

build confidence in the purpose and benefits of the research results for the community.  
Try to involve respected community members and leaders in evaluation planning (e.g. to review a questionnaire and data collection strategy).   
Ask their cooperation in helping you to recruit participation.
.  Be sure to share your findings, if possible as early as the draft stage, for their review and comment.
.  

## When to Collect Data?                                                                                                                                                                                                                                                    

- Go back to the questions the evaluation will address
- Will you document or track process and progress?

**Notes:**
Another question has to do with design—when you’ll collect and from whom

Talk from slide

Monitoring or documenting a project in  process is like gathering data to create a snapshot or profile of what is happening. You might review records (class attendance sheets),  or gather class satisfaction surveys.   Or, check up on how your program is being perceived by doing some mid-course interviews.

When trying to asses whether outreach  impact or influence changes in behavior or the environment,  there are several data collection designs that vary in complexity and resources.

To explain these, first let’s review evaluation design variables are—NEXT SLIDE

## Will you determine the amount or types of  difference outreach is making?

- Pre-activity/post-activity
- Post-activity only

**Notes:**
START HERE:

So, an evaluation design helps to figure out when you will measure the variables you’d like to study.

Pre/post design—implies looking for a change:

Increase in skill
2. Different use of sources/dbases
3.Different reasons for use/non-use

Post-only
Satisfaction with class?
Recommended Med+ to others?
Use of new listserv/website

## BREAKOUT GROUPS -- EXERCISE # 2

**Notes:**
LOOK AT THE EXERCISE.

## Evaluation Issues

- Outreach goals and objectives
- Evaluation questions to be addressed
- Types of info/data (quantitative, qualitative)
- Design: when & from whom collect data
- Data collection: what and how collect data
- Resources
- Timeline

**Notes:**
REVIEW OF ISSUES---TALK FROM SLIDE

## Planning and Evaluating Outreach

- Identify Target Community

- Conduct Community Profile

- Establish Goals & Objectives

- Develop Activities and Strategies

- Conduct Audience Assessment to Tailor Strategies

- Develop a Plan to Implement Activities

- Establish Evaluation Objectives (Process or Summative)
- Select Design and Data Collection Methods
- Develop Evaluation Instruments

- Stage Two

- Stage Three

- Carry out Plan       to Implement Outreach Activities

- Carry out Plan for Process Evaluation to Monitor and Improve Activities

- Carry out Plan for Summative Evaluation to Assess Progress and Impact

- Analyze Results for Answers to Evaluation Objectives

- Share Results and Modify Program as Needed

- Stage Five

- Stage Six

- _Planning and Evaluating Outreach_

- Stage Four

## Making Sense of the Data

- Data analysis should be thought about before collecting data
- Be sure to leave enough time
- If needed, limit the evaluation goals and reduce the number of data collected

**Notes:**
In a series on the Research Process in the Research Section newsletter, Nancy Woelfl provides a practical and helpful article about data analysis. 

She cautions that giving thought to the statistical methods to be used to analyze the data should be thought about before the data is collected.  

Unless you have training in statistical methods, your best bet is to seek help from outside sources.  MLA Research Section is doing a lot to help inform on this topic.

NEXT SLIDE

## Where to Get Help

- Handbooks, and websites
- Continuing education courses, workshops, courses in other departments
- Collaborate with faculty at your local university

## Resources

***Rule of thumb:***
- 	5% or more of a program’s budget should be allotted to evaluation activities
- 	Different designs require different levels of resources

**Notes:**
Table of design resource need intensity (from low to high) on pages 46-47 

## Resource Factors

- # of participants
- Frequency of data collection & travel
- Length of time
- # and type of data collection instruments
- Availability of existing data
- Staff expertise with research methods
- Ease of administering instruments
- Willingness of outreach participants

## Planning and Evaluating Outreach

- Identify Target Community

- Conduct Community Profile

- Establish Goals & Objectives

- Develop Activities and Strategies

- Conduct Audience Assessment to Tailor Strategies

- Develop a Plan to Implement Activities

- Establish Evaluation Objectives (Process or Summative)
- Select Design and Data Collection Methods
- Develop Evaluation Instruments

- Stage Two

- Stage Three

- Stage Four

- Carry out Plan       to Implement Outreach Activities

- Carry out Plan for Process Evaluation to Monitor and Improve Activities

- Carry out Plan for Summative Evaluation to Assess Progress and Impact

- Analyze Results for Answers to Evaluation Objectives

- Share Results and Modify Program as Needed

- Stage Five

- Stage Six

- _Planning and Evaluating Outreach_

## Using & Reporting Results

## ReassessGoals & Objectives

- Has anything changed with your audience or your organization’s mission to require revisions in the original goals and objectives?
- Are some objectives not being met? Why?
- Are there strategies or activities that did not succeed? Why?

**Notes:**
What were the relative costs (including staff time) and results of different aspects of the program?

Are there some activities that appear to work as well but cost less than others?

## Reporting

- Oral or written
- Summary of results
- Provide purpose, background, methodology of the evaluation
- Conclusions and recommendations
- Attachments

## Disseminating Results

- Identify audiences (funding agency, targeted community, staff, professional colleagues)
- Look for other publishing, presentation, or promotion opportunities
- Professional meetings and activities, websites, listservs, print and e-journals

## Planning and Evaluating Outreach

- Identify Target Community

- Conduct Community Profile

- Establish Goals & Objectives

- Develop Activities and Strategies

- Conduct Audience Assessment to Tailor Strategies

- Develop a Plan to Implement Activities

- Establish Evaluation Objectives (Process or Summative)
- Select Design and Data Collection Methods
- Develop Evaluation Instruments

- Stage Two

- Stage Three

- Stage Four

- Carry out Plan       to Implement Outreach Activities

- Carry out Plan for Process Evaluation to Monitor and Improve Activities

- Carry out Plan for Summative Evaluation to Assess Progress and Impact

- Analyze Results for Answers to Evaluation Objectives

- Share Results and Modify Program as Needed

- Stage Five

- Stage Six

- _Planning and Evaluating Outreach_

## CDC Evaluation Working Group Resources http://www.cdc.gov/eval/resources.htm 
Institute on Museum & Library Services Outcomes Based Evaluation
http://www.imls.gov/grants/current/crnt_obe.htm

National Cancer Institute’s Usability Site http://usability.gov/ 
Outreach Evaluation Resource Center
	http://nnlm.gov/evaluation

- Institute on Museum & Library Services Outcomes Based Evaluation
- _[http://www.imls.gov/grants/current/crnt_obe.htm](http://www.imls.gov/grants/current/crnt_obe.htm)_
- National Cancer Institute’s Usability Site http://usability.gov/
- Outreach Evaluation Resource Center
- 	http://nnlm.gov/evaluation

- Web Resources