# Presentation: 146986

## Measuring Your Impact: Using Evaluation for Library Advocacy

- Chicago, Illinois
- May 16, 2008
- Sponsored by the
- Medical Library Association

## Instructors

- Susan Barnes
- Assistant Director
- Outreach Evaluation Resource Center (OERC)
- sjbarnes@u.washington.edu
- and
- Maryanne Blake
- Evaluation/Outreach Coordinator
- NN/LM PNR, University of Washington, Seattle, WA
- blakema@u.washington.edu

## Today’s plan

- Agenda
- Resources/Packet
- Exercises
- Housekeeping

##                 outcomesWorkshop objectives

- By the end of the workshop participants can:
- Understand the library’s value in terms of the mission of the larger organization
- Describe some tools used to assess the library, its users and stakeholders

##                outcomesWorkshop objectives

- Identify the tools and methods used for data collection and analysis
- Design an evaluation plan for a service in a library
- Communicate evaluation results

## What this workshop is not...

- Custom-tailored
- A quick fix
- One size fits all

## Demonstrating the library’s value

## Why demonstrate value?

- To show the impact of the library on the larger organization’s mission and goals
- To show accountability for your resources; the library’s contribution to the bottom line
- To demonstrate that the library provides value-added services to solve staff problems
- To use as an advocacy and marketing tool

## How & what to evaluate

- What you decide to evaluate depends on:
- What you need or want to know
- What your users feel is important
- What certain stakeholders want to have evaluated
- Use “smallball” evaluation

## Yesterday’s Librarian

- “That was one of the delightful things about her job. It brought her in touch with so many people, all—or almost all—of whom made her not only feel welcome but that she was doing something really worthwhile.
- She gave a little sigh of pleasure as she unlocked the library door. She loved working here and she loved the work itself.”
- From:  _Jan Marlowe, Hospital Librarian_. Margaret Malcolm. Toronto; New York : Harlequin, 1976, (c)1960. Harlequin's collection ed.

## 

“Libraries are not ends in themselves, and they should not be supported because they have intrinsic value.”

- “Libraries are not ends in themselves, and they should not be supported because they have intrinsic value.”

- Plutchak TS. Means, not ends. J Med Libr Assoc 2004 Jul;92(3):294.

- Means, not ends

## The bottom line

- You are either generating revenue, or supporting those who do
***OR ...***
- You are helping to control operating expenses or supporting those who do
***OR ...***
- You are creating expenses that add recognized value
***OR ...***
- You are creating expenses that must be controlled or eliminated to reduce overhead

- Will Welton, PHD, Director, MHA Program, University of Washington

## Library service is value-added 

- Medical Staff Support
- Evidence-based clinical decision-making
- Resource for patient education
- Marketing
- Health information outreach to community
- Legal/Risk Management
- Accreditation
- Education and Research Support

## The big question!

- What is the library doing to support the bottom line?

## What you need is a plan

- Know the vision
- Look at the environment
- Create the plans
- Collect and analyze data
- Communicate the value

## The vision

## Start with the organization’s  mission

- Align the library’s mission, vision  & goals with the organization’s mission, vision & goals
- The library’s value = library’s contribution to achieving organization’s mission & goals

## Provide excellent clinical care

Provide services, resources needed for teaching and learning

Reduce corporate risk
Increase profitability

Improve the lives of patients and their families

- Provide services, resources needed for teaching and learning
- Reduce corporate risk
- Increase profitability
- Improve the lives of patients and their families

- Abels EG, Cogdill KW, Zach L. Identifying and communicating the contributions of library and information services in hospitals and academic health sciences centers  J Med Libr Assoc. 2004 Jan;92(1):46-55.

- Clinical Care
- Education
- Management of operations
- Service

- Mission Concept	 Organizational Goal

****

## The environment

## Why look at the environment? 

- Understand needs, desires and problems in context
- Validate assumptions about your contributions and services
- Provide a baseline for future evaluation
- Help to develop the blueprint to plan and evaluate your contributions and services

## Components of environment

- The organization
- Your library
- Clients/users
- Stakeholders
- The community

## Assessing the environment

- User/stakeholder input
- Surveys
- Focus groups
- Interviews
- Unsolicited Feedback
- Observation
- Assessment by walking around
- Library statistics and records

## Get out of the library!

- Talk to people
- Discuss information problems they are trying to solve
- Show how the library can help

## SWOT analysis

**Strengths**
**Internal**
**Positive statements about your library**

**Weaknesses**
**Internal**
**Statements about what is lacking in your library**

**Opportunities**
**External**
**What do clients want you to do that you are not doing?**

**Threats**
**External**
**Factors that can adversely impact your library’s goals**

****

**Notes:**
SWOT Analysis exercise for the whole class –
	Strengths & weaknesses are INTERNAL
	Opportunities & threats are EXTERNAL
Use wall mounted paper, have students put post-its on each square
Examples of strengths:
	Collection, 	knowledgeable staff
Weakness
	Staff recruitment and retention
	condition of physical facilities
Opportunities
	Library website used heavily
Threats
	budget cuts

## Wake up!  Lunch time!

**Wake up!  Lunch time!**

## Plan Backward, Implement Forward

## Where to go & how to get there

- Goals
- Outcomes
- Activities and outputs
- Resources

## Goals

- Statement(s) of Purpose – why we do what we do and for whom
- Examples:
- Patients and their families will have improved health information literacy
- Staff will have access to information for timely clinical decision-making
- Acquisition of library materials will be cost effective

## Goals are based on...

- The library’s contribution to the organization’s mission and goals
- Environment
- SWOT analysis threats and weaknesses as well as the strengths and opportunities
- Other assessments

**Notes:**
You will be deciding what you want to do to address information you found out in your assessment. 

If we look at the SWOT analysis and other assessments we do for the library these will point to what we want to 
Make our goal
	It will address threats and weaknesses and take advantage of strengths and opportunities.

## Outcomes

- Outcomes are changes in attitude, behavior, skills, knowledge or situation
- Short, medium and/or long term
- Intended or unintended
- Positive as well as negative
- S.M.A.R.T.

## SMART Outcomes

_**S**__pecific_: one or more web pages on the library website
_**M**__easurable_: done or not; do staff access resources
_**A**__ction-oriented_: created and posted; staff access resources
_**R**__ealistic_: it is possible within the timeframe
_**T**__imed_: “when” is articulated

## Example

- Goal: Staff will have access to information for timely clinical decision making.
**Intermediate Outcome**: Health resources Web pages are available on the library website by the end of the project
**Long Term Outcome:** Staff can access health information through the library website

## Exercise

**Goal:**
- What is your goal?
**Outcome:**
- What outcomes do you expect?

****

## Activities and outputs

- Activities
- What will you do?
- Outputs
- How many did you do?
- How many attended?
- How many were distributed?
- How many times was it used?

## Resources

- What you have
- Income
- Equipment
- Collection
- Etc.
- What you need
- Operating expenses (e.g., personnel, acquisitions, maintenance, etc.)
- Funds for new initiatives or services
- Etc.

## A POP QUIZ!

## POP QUIZ: What is a logic model?

- A beautiful person who thinks deep thoughts

## POP QUIZ:  What is a logic model?

- A set of plastic pieces to assemble step by step

## A planning tool that links activities to results

- POP QUIZ:  What is a logic model?

## The logic model

- Helps organize your thoughts – both before and during a project/program
- Provides a framework for planning and evaluating programs
- Clarifies intended outcomes
- Acts as a communication tool
- Good source for more information:
**W.K. Kellogg Foundation Logic Model Development Guide (see class bibliography for link)**

## 

| Resources | Activity | Outputs | Outcomes |
| --- | --- | --- | --- |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

- Logic model worksheet

- Goal:_________________________________________________

****

## Sample logic model

- Goal:
- Acquire and manage information resources in a cost-effective manner

## Logic model towards this goal

| Resources | Activity | Outputs | Outcomes |
| --- | --- | --- | --- |
| Personnel, money, expertise needed | What you will do | What your activity will produce – data, classes, brochures, etc. | The SO WHAT – the benefits that accrue as a result of your program |
| Knowledgeable librarian | Select books and 
negotiate discounts | Selection list and optimal pricing
 | Reduce hospital costs by buying shared resources. |
| Budget | Purchase books | Enter PO and  ensure access is active  | Increase value of hospital expenditures  by ensuring that ordered materials are received  |
| Record keeping system and trained staff  | Purchase books | Catalog records | Users know that hospital owns title |

## The model may change over time

## Logic model group activity

- Choose a goal
- Identify outcome(s)
- List
- activities
- outputs
- resources

****

## Sample goals

- Use one of your own goals
- Or use one of ours:
- Patients and their families will have improved health information literacy
- Staff will have information for timely clinical decision-making
- Acquisition of library materials will be cost effective

## The evaluation plan

## The evaluation plan

- Builds on the logic model
- Evaluates success
- Assesses value

**Notes:**
If you are planning a specific project you must do the evaluation plan at the beginning not at the end

## Consider the purpose

- Who is your audience? This may be different than the beneficiaries of your services
- Your users
- Administrators
- Fund raisers
- How will the information be used?
- Financial savings or justification
- Intangible or non-monetary value of program benefits to community
- Marketing and advocating for the library

## What do you want to know?

- Have you achieved your stated outcome?
-  “Reduce institutional costs by buying shared resources”

## Pieces of the evaluation plan

- Indicators
- How will you know you have achieved the outcomes?
- Data
- Sources
- Methods of collection
- Frequency of collection
- Resources
- Expertise or tools needed to collect and analyze data

## Indicators 

- “Indicators are the measures you select as markers of your success”**Logic Model Development Guide W.K. Kellogg Foundation**** **
- Observable and measurable signs of reaching an outcome
- Indicators are usually quantitative

## Existing records and statistics
Observation
User and stakeholder input

Survey results
Formal
Informal 
The literature 
Specialist or expert input

- Observation
- User and stakeholder input
- Survey results
- Formal
- Informal
- The literature
- Specialist or expert input

- Sources of evaluation data

## Data collection methods 

- Savings: statistics (output, use, financial)
- Attitudes and beliefs: surveys, interviews, focus groups
- Awareness: proxy measures, surveys, interviews, focus groups
- Behavior: proxy measures, interviews, focus groups, observation
- Knowledge: written or oral tests
- Skills: observations, hands-on tests

## Consider before collecting data...

- Indicators and outputs to be measured
- Types of data to be gathered
- Qualitative vs. quantitative
- When and from whom
- Focused on the users’ perspective

## Evaluation plan

- Outcome (from logic model):________________________________________

| Activity | Affects Whom
 | Indicator | Target | Data Source
 | Data Analysis Methods | Data Collection Frequency |
| --- | --- | --- | --- | --- | --- | --- |
| What is done to achieve outcome – from Logic Model | People involved in or influenced by activity

 | Measurable result of activity | Specific indicator measurement desired | Origin of indicator measurements | Organize, examine, learn from the data | Date, time and intervals
 |
|  |  |  |  |  |  |  |

## Example part 1

| Activity | Affects Whom | Indicator | Target |
| --- | --- | --- | --- |
| What is done to achieve the outcome | People involved in or influenced by activity | Measurable result of activity | Specific desired  result |
|  |  |  |  |

## Example part 1 (filled in)

| Activity | Affects Whom | Indicator | Target |
| --- | --- | --- | --- |
| What is done to achieve the outcome | People involved in or influenced by activity | Measurable result of activity | Specific desired  result |
| Select resources and 
negotiate discounts | Users, CFO, Educators... | Discount negotiated from list price | 20% average discount
 |

## Example part 2

| Data Source | Data Analysis Methods | Data Collection Frequency |
| --- | --- | --- |
| Origin of indicator measurements | Organize, examine, learn from the data | Date, time and intervals |
|  |  |  |

## Example part 2 (filled in)

| Data Source | Data Collection Frequency | Data Analysis Methods |
| --- | --- | --- |
| Origin of indicator measurements | Date, time and intervals | Organize, examine, learn from the data |
| CFO, librarian, Accounting

 | Data recorded monthly throughout fiscal year | Spreadsheets |

## Now design your own evaluation plan

## From your logic model...

- Select an outcome and describe:
- An activity that will help achieve the outcome
- Audience(s) affected by the activities
- Indicators of outcome results
- Targets for indicators
- Data source
- Data collection frequency
- Data analysis methods

****

## Collection and analysis

## “Not everything that can be counted counts and not everything that counts can be counted.”

Supposedly a sign in Albert Einstein’s office

- Supposedly a sign in Albert Einstein’s office

- What to measure

**Notes:**
Think about stories – a wonderful way to communicate value but they can’t really be counted.  

## Make sense of the data

- Think about data analysis ***before*** collecting the data
- Identify experts that can help with data analysis
- Test your collection and analysis tools
- Leave enough time
- Go “smallball” if needed

## Some analysis tools

- Benchmarking
- How you compare to others of similar size and circumstance
- A common measuring stick to evaluate process performance
- Calculators
- Use to illustrate the retail value of your resources and services
- Cost/Benefit Analysis
- Use to evaluate the benefits of a program or service
- Return on Investment
- How much your investment in the program or project earned for the company

## Benchmarking

- Can improve your library's performance
- Can help you gain upper management support
- Can help prove the value of your library

_**[http://www.mlahq.org/members/benchmark/](http://www.mlahq.org/members/benchmark/)**_
- (from the MLA Benchmarking Network web page)

## Benchmarking resources

- Your own library
- The competition
- Other hospitals/libraries in your system
- Other hospitals/libraries in your area
- MLA Benchmarking
- AAHSL Survey Statistics

## Library value calculators

- List of resources and services provided by the library
- Estimated value of each item
- Number of uses per month
- Total is the sum of uses x value for each resource and service

## A calculator from the MCRML

## Value of one month of services

## Your list

**Enter the number of uses for each service or resource; enter the retail value you place on each resource or service. The value of services will be calculated for you. You may enter additional resource or services in the blank cells**

## Calculator applied

- I just gave the calculator a try. I think it’s very easy to use. I pulled out my fiscal year 2007 annual report for the library and just started plugging in numbers. Some of my statistics don’t fit exactly, but you’ve provided the ability to add our own line items. I think it’s a great start. My first quick total came out to $1,366,065.00. My budget for fiscal year 2007 was just under $380,000.00. I’m meeting with my VP about next year’s budget on Friday. I’ll be taking this calculation along.

## CBA: cost/benefit analysis

- The ratio showing dollar value of benefits gained for dollar value of costs
- Use to evaluate the benefits of a program or service
- Benefits divided by costs gives the value realized by transaction

## CBA

- BENEFITS ÷ COSTS
*Or*
- _BENEFITS_
- COSTS

## ROI: return on investment

- Percentage showing the return (increase in value) on dollars spent to achieve a benefit
- Amount the company’s investment in a program earned for the company
- Earnings on money spent

## ROI

- ((BENEFITS – COSTS) ÷ COSTS) × 100
*Or*

- _BENEFITS – COSTS _
- COSTS

- X  100

## Preparing for CBA and/or ROI

- Select the program or service to value
- Journal subscriptions
- Identify benefits derived from program/service
- Journal articles on staff desktops
- Identify who gets the benefit
- Library users
- Convert the benefits to quantifiable terms (based on surveys and/or records)
- # of journals used; # of articles accessed

## Convert benefits to quantifiable terms

- Identify basis for valuing the benefits
- Calculate $ value of benefits: cost of a single personal subscription $99 X # of subscriptions
- Identify and calculate costs related to producing the benefits
- Cost of total number of subscriptions
- Staff salaries, space costs, etc. to support the online subscriptions (overhead)

- Matthews JR Internet Outsourcing Using an Application Service Provider: A How-To-Do-It Manual for Librarians. How-To-Do-It Manuals for Librarians, Number 110.  New York: Neal-Schuman, 2002: 71-72.

**Notes:**
Additional benefits:

No mail room personnel to handle receipt
No library staff time to receive, process, keep track of physical copy
Time for staff to walk to library
Income generated during the time that staff would have walked to the library
Fewer checks written for multiple subscriptions
Clerical time to enter and track multiple subscriptions
Increased knowledge due to availability of literature for all staff

## Costing tips

- Consider tangible vs. intangible benefits
- Be conservative in estimating benefits and liberal in assessing costs
- Obtain administrative approval of the value of the benefits for each alternative; involve financial staff
- Express benefits and costs in a value system shared by all parties – $$
- Think about inflation and depreciation if study extends over several years

## A CBA/ROI Example

- Subscription to online
-  *Fictional Medical Journal*
- at Anytown Institution

## Fictional Medical Journal Online

- The library spends $1600 for an institutional subscription to *Fictional Medical Journal* (FMJ) online
- 7800 articles are viewed per year using the library’s subscription

## Assumption (for teaching only)

- Each person reads on average 2 articles per week
- 2x52=104 articles per user
- 7800 uses represents 75 users
-  7800/104=75
- Without a library, each person who needed FMJ would subscribe at a cost of $99 for online only

## Benefit to the institution

-  **$7,425** (cost to buy 75 personal subscriptions: 75 x $99)
_**-$1,600**__ _(cost for library to buy institutional subscription)
**=$5,825** **savings for online institutional subscription**

****

## Value of the benefit

- $7425/$1600=$4.64
- (cost of personal subscriptions /institutional subscription)
- $4.64 of benefit for each dollar spent by library on* *FMJ online

## Summary: benefit/cost ratio

- Cost: $1,600 spent on FMJ
- Benefit to the users: $7,425 (value of 75 subscriptions at $99/subscription)
- The ratio of benefits to costs is $7425K:$1600K, or 4.64:1, or $4.64 in benefits to users for every $1 spent by the library

## CBA of document delivery

- Document Delivery services
- Librarian’s time – hourly salary x .25
- Fee charged by lending library - $15
- Pay per view
- Patron’s time – hourly salary x .3
- Provider’s fee - $35 - $65

## Benefit to the institution

-  **$85** (cost for pay per view for patron’s time and article cost)
_**-$20**__ _(cost for one ILL)
**=$65** **savings for obtaining an article through ILL**

****

## Value of the benefit

- $85/$20=$4.25
- (cost of pay per view /one ILL)
- $4.25 of benefit for each dollar spent by library on* *ILL service

## Calculating ROI

- ROI reflects the money realized by making an investment
- ROI should be larger than the likely return on a bank account or certificate of deposit or some other investment instrument
- Superior stock market return is 10-12%

## ROI 

- Benefits
- Less costs of carrying out the service or program (subscription cost plus staff cost)
- Divided by costs
- Multiplied by 100

- Costs

- Benefits – Costs

- X 100

## Simple benefits and costs

- Benefits = **$7,425**
**	****	**(cost to buy 75 subscriptions)
- Net cost = **$1,800**
**	****	**(price of an institutional subscription plus 	$200 overhead)

## Calculating ROI

- 7,425 – 1,800

- 1,800

- = 312.5%

- X 100

## ROI of document delivery

- Document Delivery services
- Librarian’s time – hourly salary x .25
- Fee charged by lending library - $15
- Pay per view
- Patron’s time – hourly salary x .3
- Provider’s fee - $35 - $65

## Calculating ROI

- (($60/3)+$65) – (($20/4)+$15) or
- $85 - $20

- $20

- = 325%

- X 100

## CBA/ROI exercise

- Select one service or resource
- List all the costs associated with providing the service once or obtaining one resource
- List alternative sources for that service or resource
- List the costs associated with that alternative source
- Plug the numbers into the formula

## Communicating the value

## Use the results...

- To communicate your value
- To promote and advocate for the library
- To  maintain or get additional funding
- As baseline information to show improvement over time
- To improve library services and programs

## Some communication methods

- Written and/or oral reports
- Formal
- Informal
- Executive summary
- Press release for newsletter or local paper
- Promotional materials for the library
- Staff meetings
- Informal hallway or elevator talk

## Communication tips 

- Have others review the communications
- Frame message in terms of added value to the mission and goals of the organization
- Use figures, tables, graphs to make your point
- Should be able to stand alone
- Have a narrative to give more details

## More tips

- Know and rehearse your message
- Talk to administrators one-on-one when possible
- Focus on benefits
- Don’t use library jargon
- Educate, but don’t bore

## What sticks? 

**SUCCESs**
- Simple
- Unexpected
- Concrete
- Credible
- Emotional
- Stories

- (From:  ***Made to Stick***. Heath, Chip.  New York:  Random House, 2007)

**Notes:**
“Stickiness” from Malcolm Gladwell’s Tipping Point, expanded in Made to Stick

SUCCESs:

Simple (find the core, single most important thing) WHAT TO SAY
Unexpected (get attention, surprise; hold attention, interest) PAY ATTENTION
Concrete (understand, remember, tangible) UNDERSTAND AND REMEMBER
Credible (accessible, testable) BELIEVE AND AGREE
Emotional (make people care, power of association, self-interest and identity) CARE
Stories (get people to act; tell people how to act—simulation) ACT

## “Elevator talk”

- Simple, memorable mini-speech
- Ready for any brief encounter
- What your library contributes and why that matters
- 	 “Sadly and too often, the people who have the most influence over our future are really not all that interested in libraries.”

- Gardner Reed S. Making the Case for Your Library: A How-To-Do-It Manual. How-To-Do-It Manuals for Librarians, Number 104.  New York: Neal-Schuman, 2001: 6

## Elevator talk exercise

- Select an outcome
- Invent results
- Determine the audience
- Create a sound bite that highlights the value of the outcome to that audience

****

## Go forth & evaluate

- Do smallball evaluation
- Relate your value to the mission and goals of your organization
- Plan your evaluation before you begin
- Share your value with many audiences
- Don’t wait for a crisis – *start now!*** **

## Want more information?

- Resources list for this class
- The OERC Web site
- _[http://nnlm.gov/evaluation/](http://nnlm.gov/evaluation/)__ _
- Web resources for evaluation
- _[http://nnlm.gov/evaluation/tools/](http://nnlm.gov/evaluation/tools/)_