# Presentation: 100763

- Measuring Your Impact:
**Using Evaluation ****to Demonstrate Value**
**Phoenix, Arizona**
**	****May 19, 2006**
**	****Medical Library Association**

- NN/LM-MR, Salt Lake City

**Notes:**
Reminder to instructors

What do you want to get out of this class? – write down as Maryanne and Susan/Betsy introduce; then share during their personal intro.
Share personal goals, ahas 
AHA sheet – write your Ahas!
Creating groups, koosh ball toss

## Instructors

**Betsy Kelly**
**Evaluation and Assessment Coordinator, NN/LM MCR**
**Associate Director, Becker Library, Washington University**
**kellyb@msnotes.wustl.edu**
**and **
**Maryanne Blake**
**Evaluation & Outreach Coordinator, NN/LM PNR**
**blakema@u.washington.edu**

**Notes:**
Reminders to instructors:

Do the logistics talk – bathrooms, breaks, lunch, etc
Do the go-through-the-packet talk

## Workshop objectives

**By the end of the workshop participants will:**
- Understand the library’s value in terms of the mission of the larger organization
- Describe the tools used to assess the library, its users and stakeholders

**Notes:**

And the class will brainstorm together on some evaluation exercises

Please look at the packets you have--  1) class slides, several exercises, and a few handouts

The day’s agenda will be......

Bathrooms are......

## Workshop objectives

- Identify the tools and methods used for data collection and analysis
- Design an evaluation plan for a service in a library
- Communicate evaluation results

**Notes:**
AHA moments

Before we get to the nitty gritty of developing an evaluation plan for the library Lets talk basics.

Showing the library’s and the librarian’s value isn’t about sending some numbers to administration when budget time rolls around.  
It is about rethinking the way you do business in the library and the way you plan for the library.
Many of you may already be thinking this way.  Let’s see... 

## Why demonstrate the library’s value?

- To show the impact of the library on the larger organization’s mission and goals
- To show accountability for your resources; the library’s contribution to the bottom line
- To demonstrate that the library provides value-added services to solve staff problems
- To use as an advocacy and marketing tool

**Notes:**

Be proactive – don’t wait for the call from administration asking for justification of the library’s existence or cuts in staff
You need to talk about how the library contributes to the overall mission and goals of the hospital.  Tie your services to the mission, values and goals

Helps you document your value with data not just say that it is valuable

Brings awareness and attention to the library

## I can’t believe I have to evaluate the whole thing!

**Use “smallball” evaluation**** **
**What you decide to evaluate depends on:**
- What you need or want to know
- What your users feel is important
- What certain stakeholders want to have evaluated

- Environment

**Notes:**
Use comment from Boston class about one person libraries and evaluation for dummies 

Mom’s single ‘em to death theory

Charles Friedman’s paper on smallball evaluation

Include the house building concept – painting one room at a time, not all of them; planting a tree not a forest

## Yesterday’s Librarian

- “That was one of the delightful things about her job. It brought her in touch with so many people, all—or almost all—of whom made her not only feel welcome but that she was doing something really worthwhile.
- She gave a little sigh of pleasure as she unlocked the library door. She loved working here and she loved the work itself.”
- From:  _Jan Marlowe, Hospital Librarian_. Margaret Malcolm. Toronto; New York : Harlequin, 1976, (c)1960. Harlequin's collection ed.

**Notes:**
Read the quote and ask how many are still using this model of hospital librarianship.  It is so 1960s!

## Today’s librarians...

**Must be:**
- Proactive
- Political
- Practical
- Business–like
**Must reinvent librarianship**

**Notes:**
You need to change the way you do business in your hospital library. 
A hospital, even a non-profit, is a business and you must run your library like a cost-center in a business.  You have to justify your existence.  
You should be writing a business plan for the library.  
You need to be out in the hospital advocating for the library (yes, marketing), making friends and influencing people.  
You need to find out who has the power in the hospital – power over what?  Tell the VA finance department story.

## Reinventing librarianship

**T. Scott Pluchak, ****JMLA: July 2004 Editorial **
**Align the library’s priorities and services with the vision, mission and values of the larger organization to show its value ...**
**Remember, the library is a tool; not an end in itself**
- It will not be supported simply because it has intrinsic value

**Notes:**
This applies to hospital librarians as well as other librarians, including academic medical  librarians
Working from your institution’s mission is critical in terms of how you talk about your value to the larger organization – see Cogdill’s articles.
Do a summary of Keith’s article and bring it up here. 
Tell them forum story of how Martha and Dave Bjornson both mentioned this as VERY important
People don’t want to hear about what you have in the library they want answers to their problems – show them how you can help them.
We know we are good – but that is not enough.  Just because people say they love libraries doesn’t mean they will support them.  You have to show your value.
Libraries have talents, resources and skills that are essential for the success of your organizations

## Get started – get out of the library!

**Talk to people**

**Talk about information problems they are trying to solve**

**Show how the library can help them**

**Notes:**
Librarians attraction to one person operations
Self confidence, assertiveness, the MBA personality - me

## Solving problems = value-added For hospital libraries

**Medical Staff Support**
- Evidence-based clinical decision-making
**Marketing**
- Research
- Patient/family education and support
- Community support
**Legal/Risk Management**** **
**Human Resources Support**
- Staff education and development
**Planning & Program Development**
- Evidence-based management support (competitive intelligence)

**Notes:**
Partner with other departments to support them in their problem solving.  
This is added value to your services.  
Will Welton, Director Masters in Health Administration Program at University of Washington

## Educational Support
Resources for students and faculty
Liaisons to academic programs 
Evidence-based clinical decision-making 
Accreditation 
Outreach
Patients/families
Greater community 
Human Resources Support
Staff education and development

- Resources for students and faculty
- Liaisons to academic programs
- Evidence-based clinical decision-making
**Accreditation**** **
**Outreach**
- Patients/families
- Greater community
**Human Resources Support**
- Staff education and development

- Solving problems = value-added For academic libraries

## The bottom lineWill Welton

**You are either generating revenue, or supporting those who do****OR...**
**You are helping to control operating expenses or supporting those who do**** ****OR ...**
**You are creating expenses that add recognized value **
**	****OR...****	**
**You are creating expenses that must be controlled or eliminated to reduce overhead**

**Notes:**
Put in no-nonsense terms here is the bottom line from one who knows
Will Welton heads a program at the University of Washington – a Masters in Health Administration.
He works with hospital administrators all the time.  Here are words of wisdom that he recently shared with a group of medical librarians.

By the way his wife is a medical librarian.

## So, ask yourself“What am I doing to support the bottom line?”And remember...

**Notes:**
Valuating Information  Intangibles by Frank Portugal 

## “There are no sacred departments ...  Any function can be contracted to other professional agencies that want the business.  For a department director an important question to consider is: What would we do differently if we had to compete with outside agencies to keep the internal client we have come to take for granted?”

If Disney Ran Your Hospital: 9 1⁄2 Things You Would Do Differently.  Fred Lee, Bozeman, MT: Second River Healthcare Press, 2004.

***If Disney Ran Your Hospital: 9 1⁄2 Things You Would Do Differently.  Fred Lee, Bozeman, MT: Second River Healthcare Press, 2004******.***

## A Plan To Show Your Value IsLike Building A House

## Before you dig...

**The vision **
**The environment**
**The blueprint **
**The tools**
- And then...** **
**Build it **
**The housewarming**

**Notes:**
The vision = what kind of house do you see for yourself and your family
The environment = what are the things internally and externally that will influence the building of the house
(or not building the house but scaling down your vision)
The blueprint = the logic model -- 

## The Vision

## The vision  - know what you want to build and why

**Align library mission and goals with organization mission and goals**** **
**Library’s value = library’s contribution to achieving organization mission and goals**

- Vision

**Notes:**
Keith Cogdill’s article in JMLA 2004 with Eileen Abels and Lisl Zach on communicating the value of the hospital library in terms of  its contributions to achieving the hospitals mission and goals.
They came up with a taxonomy of contributions of the library that reflected the mission and goals common in most  hospitals – went out and spoke to hospital administrators 

Library’s contributions to the mission and goals of the organization
As indicators of value

##    Mission Concept

Clinical care

Education
Management of operations
Service

- Clinical care
- Education
- Management of operations
- Service

_**Organizational Goal**_
**Provide excellent clinical care**
**Provide services, resources needed for teaching and learning**** **
**Reduce corporate risk**** **
**Increase profitability**** **
**Improve the lives of patients and their families**

- Abels EG, Cogdill KW, Zach L. Identifying and communicating the contributions of library and information services
-  in hospitals and academic health sciences centers  J Med Libr Assoc. 2004 Jan;92(1):46-55.]

- Vision

**Notes:**
Lead a whole audience brainstorming exercise here to decide on what contributions the library
 might make and what services the library has to support this or might need to support this.

Use Keith’s article chart as help in this

Whole class exercise to fill in the chart using Cogdill article as a guide
Flip chart fill in 
Organizational goal and what library services (activities) can help the organization achieve that goal

## The Environment

## The environment

**The organization**** **
**Your library**** **
**Clients/users**** **
**Stakeholders**** **
**The community**

- Environment

## Why look at the environment? 

**Understand needs, desires and problems in context**** **
**Validate assumptions about your contributions and services **** **
**Provide a baseline for future evaluation**
**Help to develop the blueprint to plan and evaluate your contributions and services**

- Environment

**Notes:**
Talk from slides

What will you do with this information?

 Figure out how the library helps—plus what is important to your audience 
  Get feedback and ideas from your target audience
   Create a shared vision 
  Become aware of factors to consider in your plannning
  Identify causes of barriers--  such as attitudes of administration 

## SWOT analysis

- Strengths
- Internal
**Positive statements about your library**

- Weaknesses
- Internal
**Statements about what is lacking in your library**

- Opportunities
- External
**What do clients want you to do that you are not doing?**

- Threats
- External
**Factors that can adversely impact your library’s goals**

- Environment

**Notes:**
SWOT Analysis exercise for the whole class –
Use wall mounted paper, have students put post-its on each square

Examples of strengths
Collection, knowledgeable staff
Service 

Weakness

Opportunities

Threats

## Assessing the environment

**User/stakeholder input**
- Surveys
- Focus groups
- Interviews
- Unsolicited Feedback
**Observation**
- Assessment by walking around
**Library statistics and records**

- Environment

**Notes:**
Show handout that talks about different assessment tools and their strengths and weaknesses

## Break time for Gobi (and for you too!)

## What You Need Is A Blueprint!

## The blueprint

**What you will build comes from analyzing:**
**What do you need?  (Goal)**
- Comfort and convenience
**What would meet your need? (Outcome)**
- A renovation of existing space?
- An addition?
- A new house?

- Blueprint

## The blueprint

**What actions must be taken?  (Activities)**
**How much of what must be produced?  (Outputs)**
**What resources will be required? (Resources)**

- Blueprint

**Notes:**
Activities:
Determine specifications
Find architect and/or contractor
Obtain permits
Etc., etc.
Outputs:
   New rooms
   Reconfigured space
Resources:
   Money!
   Manpower
   Building materials

## Plan backward;Implement forward

**Goals**
**Outcomes**
**Activities and outputs**
**Resources**

- Blueprint

**Notes:**
You need to know where you want to go or what you want to do (hike? Sun? sail? Visit museums) before you can decide how to get there. 

Much like planning a vacation. You have to know the destination before you decide whether to drive or fly, when, where you’ll stay.

## Goals

**Statement(s) of Purpose – why we do what we do and for whom**
**Examples:**
- Patients and their families will have improved health information literacy
- Staff will have information for timely clinical decision-making
- Acquisition of library materials will be cost effective

- Blueprint

**Notes:**
Goals are long-range statements describing a desired condition or future that outreach is working toward fulfilling.
Set goals with input from the audience/community—base on needs and concerns rather than a personal agenda

## Outcomes

**Goal-related outcomes should be ****SMART**
- Specific
- Measurable
- Action-oriented
- Realistic
- Timed
**Outcomes may be Short, Medium and/or Long Term**

- Blueprint

**Notes:**
While goals describe an ultimate ideal, smaller steps are needed to achieve them
Outcomes – this is the so what. Again, be specific and think about how you’ll know that you succeeded. What changed because you conducted this program or provided this service. Who will benefit? How? Who do you WANT to benefit? Your activities will be designed to accomplish these outcomes, how do the outcomes address the mission and goals of the organization

Desired outcome:

Health professionals will use published literature more often to answer questions about patient care when electronic journals are available.

The hospital will spend less on binding and replacing lost copies.

Does it meet the criteria?

## Exercise model

**Goal: Nurses will have access to health information**
**Intermediate Outcome**: Health resources web pages are available on the library website by the end of the program
**Long Term Outcome:** Nurses access health information through the library website
**S**: one or more web pages on the library website
**M**: done or not; do nurses access resources
**A**: created and posted; nurses access resources
**R**: it is possible within the timeframe
**T**: “when” is articulated

- Blueprint

## Exercise – you do it now

- Goal:
- The library will cost effectively acquire and manage information resources
- Outcome:
- What outcomes do you expect?

- Blueprint

**Notes:**
Articulate outcomes and note whether short, medium or long term, and whether obtainable solely through this project or by bringing in others at some point

## Activities and outputs

**Activities **
- What will you do?
**Outputs**
- How many did you do?
- How many attended?
- How many were distributed?
- How many times was it used?

- Blueprint

**Notes:**
Activities – be very specific. While it seems tedious to actually write all the activities remember that this can be used to support your plans to readers who may not know the details that you do.

Outputs: Librarians are good at counting and outputs are just that – how many – questions answers, books purchased, shelved, loaned, ILLs filled.
Think outside the traditional statistics – what about orders submitted, checks written, ethernet cables run, accounts created

## Resources

**What you have**
- Income
- Equipment
- Collection
- Etc.
**What you need**
- Operating expenses (e.g., personnel, acquisitions, maintenance, etc.)
- Funds for new initiatives or services
- Etc.

- Blueprint

**Notes:**
Resources – what you already have and what you will need. If you don’t have the resources you need how will you get them?

## The logic model

**A tool for organizing your thoughts – both before and during a project/program**** **
**Framework for planning and evaluating programs**** **
**Makes evaluation easier**** **
**Good source for more information:**** **
- W.K. Kellogg Foundation Logic Model Development Guide at  http://www.wkkf.org/DesktopModules/WKF_DmaItem/
- ViewDoc.aspx?LanguageID=0&CID=284&ListID=28&ItemID=2813669&fld=PDFFe

- Blueprint

**Notes:**
 A logic model is a planning tool to clarify and graphically display what your project intends to do and what it hopes to accomplish and impact.
A logic model:
Summarizes key program elements; 
Explains rationale behind program activities; 
Clarifies intended outcomes; 
Provides a communication tool 
Think of a logic model as a map that you develop to clarify and communicate what your project intends to do and its presumed impact. 
The components of logic models vary because there is no one single logic model format.

Template, framework for describing a proposed or existing project or program

Logic models can be fluid – used to monitor progress and process; blueprints change, too!  But keep track of your original goals, even if they have to change.  

observations may suggest need to change, new approach to ensure desired
 results will be received.

Used for program planning; evaluation plans are built on the program logic model

## Pieces of the logic model

**Resources**
- What will you need to carry out your activities?
**Activities**
- What will you do?
**Outputs**
- How many of what will your activities produce?
**Outcomes**
- So what – the difference your program makes, the benefits that accrue because of your program

- Blueprint

**Notes:**
Resources – what you already have and what you will need. If you don’t have the resources you need how will you get them?

Activities – be very specific. While it seems tedious to actually write all the activities remember that logic models can be used to support your plans to readers who may not know the details that you do.

Outputs: Librarians are good at counting and outputs are just that – how many – questions answers, books purchased, shelved, loaned, ILLs filled.
Think outside the traditional statistics – what about orders submitted, checks written, ethernet cables run, accounts created

Outcomes – this is the so what. Again, be specific and think about how you’ll know that you succeeded. What changed because you conducted this program or provided this service. Who will benefit? How? Who do you WANT to benefit? Your activities will be designed to accomplish these outcomes, how do the outcomes address the mission and goals of the organization

This is the Plan backward, implement forward If you know what outcomes you are looking for – the why of doing this program – then you can plan the activities, figure out the outputs, and decide what resources you’ll need to be able to pull it all off.

## Sample logic model

| Resources | Activity | Outputs | Outcomes |
| --- | --- | --- | --- |
| Personnel, money, expertise needed | What you will do | What your activity will produce – data, classes, brochures, etc | The SO WHAT – the benefits that accrue as a result of your program |

| Reduce hospital costs by buying shared resources. |
| --- |

| Budget | Purchase Books | Enter PO and  ensure access is active  | Increase value of hospital expenditures  by ensuring that ordered materials are received  |
| --- | --- | --- | --- |
| Record keeping system and trained staff  | Process Books | Catalog records  | Users know that hospital owns title |

- Goal: Cost effectively acquire and manage information resources

| Knowledgeable Librarian | Select Books and 
Negotiate discounts | Selection list and optimal pricing |
| --- | --- | --- |

**Notes:**
Steps for developing a logic model:
-Usually a logic model is used to organize thoughts about how to achieve a goal
-Start with the column you know best.  Often you’ll have an idea about what you want to do, so the activities column would be a good starting point
-After listing an activity or two, think about the results you hope to accomplish, and write outputs and outcomes
-After completing activities, outputs, and outcomes: think about what you need and list in resources column

## Logic model group activity

**Chose a service to evaluate**
**Articulate the goal**
**Identify outcome(s)**
**List **
- activities
- resources
- outputs

- Blueprint

**Notes:**
Ask for a non traditional library service, a non collection service

## Logic model worksheet

| Resources | Activity | Outputs | Outcomes |
| --- | --- | --- | --- |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |

- Goal:_________________________________________________

## Logic models may change over time

- Blueprint

**Notes:**
logic models as fluid and evolving

Changing resources may influence appropriate activities which in turn will affect outputs and outcomes. As outcomes are realized new resources and or activities may be identified.

## The blueprint part II:The evaluation plan

**Builds on the logic model**
**Consider the purpose**
- Who is your audience?
- Your users
- Administrators
- Fund raisers
- How will the information be used?
- Financial savings or justification
- Intangible or non-monetary value of program benefits to community
- Marketing and advocating for the library

- Blueprint

**Notes:**
If you don’t know what you want to do how can you evaluate whether you succeeded?
Who will be interested in your results?
	Just you and your staff?
	Your users?
	Your administrators?
	Your development/fund raising office?
You need to be very clear on this because what data you collect will depend on and influence how you design your reports

## The evaluation plan - continued

**Questions**
- What do you want to know about your program?
**Indicators**
- How will you know you have achieved the outcomes?
**Data**
- Sources
- Methods
- Frequency
**Resources**
- Expertise or tools needed to collect and analyze data

- Blueprint

**Notes:**
In evaluation speak – what is the evaluand? Think back to our section on Choosing what to evaluate.
Indicators are usually written as % of # that 

Data
	Can you get the financial data you need?
	Do you need before and after – skills, opinions, usage
	Who has the information you need? Users? Accounting? Computer records? Online Journal vendors?

Resources –
	Can you do it yourself? 
	Do you have staff? 
	Do you have access to experts – statisticians? Others with experience creating surveys? 
	Do you need to do any work anonymously – so your respondents don’t hedge their answers or color things to you get what they think you want to hear

## Evaluation plan

- Outcome (from logic model):________________________________________

| Activity | Affects Whom | Indicator | Target | Data Source | Data Collection Frequency | Data Analysis Methods |
| --- | --- | --- | --- | --- | --- | --- |
| What is done to achieve outcome – from Logic Model | People involved in or influenced by activity | Measurable result of activity | Specific indicator measurement desired | Origin of indicator measurements | Date, time and intervals | Organize, examine, learn from the data |
|  |  |  |  |  |  |  |

## Evaluation plan

| Activity | Affects Whom | Indicator | Target |
| --- | --- | --- | --- |
| What is done to achieve the outcome | People involved in or influenced by activity | Measurable result of activity | Specific desired  result |

- Blueprint

- Outcome: Reduce institutional costs by buying shared resources.

| Select Resources and 
Negotiate discounts | CFO, CEO, librarian, users | Discount negotiated from List Price | 20% average discount |
| --- | --- | --- | --- |

## Evaluation plan II

| Vendor List Price, contracts, POs | Data recorded monthly throughout fiscal year | Spreadsheets |
| --- | --- | --- |

- Blueprint

- Outcome: Reduce hospital costs by buying shared resources.

| Data Source | Data Collection Frequency | Data Analysis Methods |
| --- | --- | --- |
| Origin of indicator measurements | Date, time and intervals | Organize, examine, learn from the data |

## Design Your Own Evaluation Plan

- Blueprint

**Notes:**
This is a group activity

## Evaluation plan group activity

**Select an outcome from your logic model and list:**
- An activity from your logic model that will help achieve the outcome
- Audience(s) affected by the activities
- Indicators of outcome results
- Targets for indicators
- Data source
- Data collection frequency
- Data analysis methods

- Blueprint

**Notes:**
Ask for a non traditional library service, a non collection service

Add a timer

## Evaluation plan worksheet

- Outcome (from Logic Model):_______________________

| Activity | Affects Whom | Indicator | Target | Data Source | Data Collection Frequency | Data Analysis Methods |
| --- | --- | --- | --- | --- | --- | --- |
| What is done to achieve outcome – from Logic Model | People involved in or influenced by activity | Measurable result of activity | Specific indicator measurement desired | Origin of indicator measurements | Date, time and intervals | Organize, examine, learn from the data |
|  |  |  |  |  |  |  |

## Wake up!  Time for “Librarian Chow”

## The Tools To Build It

## “Not everything that can be counted counts and not everything that counts can be counted”
Albert Einstein

**Albert Einstein**

- Build It

## Methods and tools for collecting data

**What to measure and how  **
- What indicators and outputs need to be measured
- What types of data need to be gathered
- Qualitative vs. quantitative
- When and from whom will it be gathered
- Focus on the users’ perspective

- Build It

**Notes:**
For example, time saved by library users, money saved by using library instead of alternatives, costs avoided through library services 
Value of services, importance of information

Combine quantitative with qualitative to provide objective, dollar-oriented bottom line data plus the story behind the data

“Library managers must determine what the organization’s leaders look for in judging value.  This can be accomplished through executive interviews or an executive library advisory board.  Another tactic is to look around at the best funded or best-valued departments and find out what they do to demonstrate value.  At any rate, before jumping into any data collection activities, it’s critical that library managers make sure the information they’re planning to collect will provide metrics that resonate with funders.” p. 3, 

Above info from: “Demonstrating value and return on investment...” Roger Stouse.  Information Outlook, March 2003. 7(3)14-19.

Don’t concentrate exclusively on existing services
Don’t use instruments developed for other contexts without making them your own
Don’t assume that similar people have similar needs, issues

Looking for participant behavior and feedback

## Tools for measuring ....

**Savings – statistics (output, use, financial)**
**Attitudes and beliefs – surveys, interviews, focus groups**
**Awareness – proxy measures, surveys, interviews, focus groups**
**Behavior – proxy measures, interviews, focus groups, observation**
**Knowledge – written or oral tests**
**Skills – observations, hands-on tests**

- Build It

**Notes:**
Focus groups: sort of a group interview of 6 to 10 people from similar backgrounds; in-depth attitudes, beliefs, anecdotes; group dynamics generate more ideas than individual
Interviews: question-and-answer sessions with an individual (in person, telephone, online); in-depth attitudes, beliefs, anecdotes; personal contact encourage more detailed responses
Observation: observe and record behaviors; discover behaviors in library setting
Surveys: brief responses, can be close-ended or open-ended, written or online –can be used to identify benefits of using library and/or what users would pay if purchase for themselves
Follow-up interviews: follow up on findings from other approaches
Proxy measures – numbers like # of pamphlets picked up, # of people who visited the Web site, # of ILLs done

## Existing records and statistics
Observation
Obtain user and stakeholder input – focus groups, interviews with key informants 
Surveys
Formal
Informal 
The literature 
Specialists or consultants

**Observation**
**Obtain user and stakeholder input – focus groups, interviews with key informants**** **
**Surveys**
- Formal
- Informal
**The literature**** **
**Specialists or consultants**

- Sources of evaluation data

- Build It

**Notes:**
How do you conduct a community assessment

Think about what data already exists
--Census data for demographics down to the county level
--BENCHMARKING

--Maybe surveys that have been done by other local organizations
--do lit searches to see what else is out there 

User input--   important to get perspectives first hand

## Make sense of the data

**Data analysis should be thought about ***before*** collecting data**** **
**Identify experts that can help with data analysis**** **
**Test your collection and analysis tools**
**Be sure to leave enough time**
**If needed, limit the evaluation goals and reduce the amount of data collected (“smallball”)**

- Build It

**Notes:**
In a series on the Research Process in the Research Section newsletter, Nancy Woelfl provides a practical and helpful article about data analysis. 

She cautions that giving thought to the statistical methods to be used to analyze the data should be thought about before the data is collected.  

Unless you have training in statistical methods, your best bet is to seek help from outside sources.  MLA Research Section is doing a lot to help inform on this topic.

## “Hot” analysis tools

**Benchmarking**
- How you compare to others of similar size and circumstance
- A common measuring stick to evaluate process performance
**CBA is Cost/Benefit Analysis**
- Use to evaluate the benefits of a program or service
**ROI is Return on Investment**
- How much your investment in the program or project earned for the company

- Build It

## Benchmarking

**Benchmarking, conducted properly, can **
- improve your library's performance;
- help you to gain or improve upper management support; and
- help to prove the value of your library
- _[http://www.mlahq.org/members/benchmark/](http://www.mlahq.org/members/benchmark/)_
**(from the MLA Benchmarking Network web page)**

- Build It

## Benchmarking resources

**Your own library**** **
**The competition**** **
**Other hospitals in your system**** **
**Other hospitals in your area**** **
**MLA Benchmarking**
**AAHSL Survey Statistics**

- Build It

## CBA (cost/benefit analysis)

**CBA – ratio showing dollar value of benefits gained for dollar value of costs**** **
**Use to evaluate the benefits of a program or service**
**Benefits ****÷ Costs**

- Build It

**Notes:**
Compare costs of an alternative with associated benefits 

Benefit of a library service

Cost of same service from another source

Benefit - Cost = 
	Value realized per service transaction

## ROI (return on investment)

**ROI – percentage showing the return (increase in value) on dollars spent to achieve a benefit**
**Amount the company’s investment in a program or project earned for the company**
** ****(****(Benefits – Costs) ****÷ Costs****) ****× 100**

- Build It

**Notes:**
Ratio of net profits to total assets

Business tool to determine earnings on money spent

CFO will select option with best ROI 

ROI of a library must show that the expense provides the highest return of any potential use of the budget dollars

percentage that shows the return (increase in value) on dollars spent to achieve a benefit 
CFO’s are charged with getting the most return from every dollar invested or spent

ROI works best in for-profit organizations that track tangible assets (income, capital, stockholder’s equity)
Libraries that are not themselves for-profit, or are in nonprofit organizations, can use ROI by converting their activities into revenue by multiplying expenses associated with those activities by the appropriate markup values.
Valuating Information Intangibles:  Measuring the Bottom Line Contribution of Librarians and Information Professionals.  Frank H. Portugal.  Washington, DC:  Special Libraries Association, 2000.

Maximize the internal rate of return on the investment—libraries in the for-profit sector often asked to do this; an organization makes investment decisions based on analysis of ROI and the option with the best ROI is selected.
The Bottom Line:  Determining and Communicating the Value of the Special Library.  Joseph R. Matthews.  Westport, Connecticut:  Libraries Unlimited, 2002.

## Preparing for CBA and/or ROI

**Select the program or service you will value**** **
- Journal subscriptions
**Identify benefits derived from them****  **
- Journal articles on staff desktops or for in-library use
**Identify who gets the benefit **
- library users
**Convert the benefits to quantifiable terms (based on surveys and/or records) **
- # of journals used; # of articles accessed

- Build It

**Notes:**
Find all cost data for operating information services
Collect user estimates of the value of beneficial library services
Record narrative accounts of library impact
Analyze the cost and benefit information gathered
	“A Bottom-Line Adventure” Barbara Weiner.  Behavioral & Social Sciences Librarian 18(2)2000:27-31

Reduce all benefits to quantifiable terms.  If cannot quantify, don’t include; treat separately

Design data collection method
Collect data
Convert data to monetary values
Isolate the effects of the program
Costs of the program must be included
Difficult to use with qualitative data

## Convert benefits to quantifiable terms

**Identify basis for value the benefits:**
- calculate $ value of benefits – cost of a single personal subscription $99 X # of subscriptions
**Identify and calculate costs related to producing the benefits:**
- paying for x subscriptions, staff salaries, space

**Internet Outsourcing Using an Application Service Provider: A How-To-Do-It Manual for Librarians**.  Joseph R. Matthews.  How-To-Do-It Manuals for Librarians, Number 110.  New York: Neal-Schuman, 2002. pp. 71-72.

- Build It

## Costing tips

**Consider tangible vs. intangible benefits**** **
**Be conservative in estimating benefits and liberal in assessing costs**** **
**Obtain administrative approval of the value of the benefits for each alternative; involve your financial staff**** **
**Benefits and costs must be expressed in a value system shared by all parties – $$**** **
**If your study extends over several years you may need to think about inflation and depreciation**

- Build It

**Notes:**
Intangible benefits—qualitative factors—feelings, opinions, perceptions
Identify alternatives and then positive and negative attributes of each.
Prepare a “profit-and-loss statement” showing which alternative has the most desirable features and fewest undesirables.  
Qualitative analysis often has less influence than quantitative.

Internet Outsourcing Using an Application Service Provider: A How-To-Do-It Manual for Librarians.  Joseph R. Matthews.  How-To-Do-It Manuals for Librarians, Number 110.  New York: Neal-Schuman, 2002. p. 77.

## An example: Electronic journal subscriptions at Anytown Regional Medical Center

**The Library spends $1600 for an institutional subscription to NEJM online.**** **
**7800 articles are viewed per year using the Library’s subscription. **** **
***Assumption (for teaching purposes only)*****: **
- Each person reads on average 2 articles per week
- 7800 uses represents 75 users (7800/104)
- Without a library each person who needed NEJM would subscribe at a cost of $99 for online only.

- Build It

**Notes:**
How do we know that a book is used 2 times a year?  Circulation records or survey

How do we know that a person would buy it?  Survey – willingness to pay

## The example continued... Electronic journal purchase benefit

**Benefit to the institution: **
** **$7,425 (cost to buy 75 personal subscriptions)
- _-$1,600 _(cost for library to buy institutional subscription)
- =$5,825 savings for online institutional subscription
**Value of the benefit **
**= **$7425/$1600 = $4.64 of benefit for each $ spent by library on NEJM online

- Build It

**Notes:**
How do we know that a book is used 2 times a year?  Circulation records or survey

How do we know that a person would buy it?  Survey

## The example continued... Simple one year benefit/cost ratio

**Cost: $1,600 spent on NEJM**** **
**Benefit to the users: $7,425 (value of 75 subscriptions at $99/subscription)**
**The ratio of benefits to costs is $7425K:$1600K, or ****4.64:1****, or $****4.64**** in benefits to users for every $1 spent by the library.**

- Build It

##  The example continued... Calculating ROI

**The ROI reflects the $$ realized by making an investment****ROI should be larger than the likely return on a bank account or certificate of deposit or some other investment instrument**
**Superior stock market return is 10-12%.**

- Build It

##  The example continued...ROI 

**Benefit (cash not spent)**
**Less costs of carrying out the service or program (subscription ****plus**** staff)**
**Divided by costs**
**Multiplied by 100**

- Costs

- Benefits – Costs

- X 100

- Build It

## The example continued...Simple benefits and costs

**Benefit = **$7,425
- 		**(cost to buy 75 subscriptions)**
**Net cost = **$1,800
- 		**(price of an institutional subscription plus ****	****personnel and administrative costs)**

## The example continued... Calculating ROI

**7,425 – 1,800**

- 1,800

- = 312.5%

- X 100

- Build It

**Notes:**
Usually ROI is over a period of ;years and the costs and benefits are figured in present day values. There is a formula to do Present Value but not going into that today. It’s available in many resources.

Some considerations:
	Who realizes the benefit?
		The docs who don’t buy the subscriptions? If the institution wouldn’t have paid, then they don’t see the value
		But if the docs don’t have to come to the library, and have more time to bill for services, then the institution can realize a value – additional income for the time that was made available for billing
			And salaries of secretaries who aren’t spending time entering and tracking subscriptions and bills for the docs

What else might be considered in figuring the value of the benefit of subscribing?

## The Housewarming

## The housewarming

**Invite your friends**
- Share evaluation results with stakeholders
**Update the décor**
- Introduce new products or improve services identified in the evaluation
**Enjoy your (new) house**
- Continue to provide services knowing that you are giving your institution the highest value for its money

- Housewarming

## Use the results...

**To communicate your value**** **
**To promote and advocate for the library**** **
**To  maintain or get additional funding**** **
**As baseline information to show improvement over time**** **
**To improve library services and programs**

- Housewarming

## Communicate the results

**What is the purpose of the communication?**** **
**What communication strategy serves that purpose?**
**Who needs to know the results?**
- Stakeholders like administrators, staff and other partners, library users, community groups
**What is the best strategy to reach them?**

- Housewarming

**Notes:**
Ask the journalists –
	who will use the information
Purpose  could be publicity, funding, advocacy, survival, promoting a  service, 
	what and why do they need to know
	how will they use the information
	When do they need it

Choose the best type of presentation to meet their needs

Purpose or use - 

## Some communication strategies

**Written and/or oral reports**
- Formal
- Informal
**Executive summary**** **
**Press release for newsletter or local paper**** **
**Promotional materials for the library**** **
**Staff meetings**** **
**Informal hallway or elevator talk**** **

- Housewarming

**Notes:**
Elevator talk

Do you know the library subscribes to nejm online?
When I see Mr. Smith – Do you know the library is subscribing to... or 

Who are you comfortable chatting with? Administrators? Docs? Deans? Department Chairs

## Communication tips 

**Have others review the communications**** **
**When you write an Executive Summary**
- Make it brief, understandable, descriptive
- Frame it in terms of added value to and the mission and goals of the organization
**Use figures, tables, graphs to make your point**
- Should be able to stand alone
- Have a narrative to give more details

- Housewarming

**Notes:**
Some may want a 5 minute oral summary of your findings. Can you digest so the highlights – the most persuasive results are apparent?
Talk about the results in terms of the mission, vision, value of the organization; how the library adds value to the organization, how it solved a problem

If you give an oral report always give them a written report with it.  

Some may read only the executive summary. Decisions are made based on summaries – make sure your results are clear and convincing

## “Elevator talk”

**Simple, memorable mini-speech**** **
**What does your library contribute and why does that matter? **
**Ready for any brief encounter**
- 	“... we forget that other people have other agendas.  Sadly and too often, the people who have the most influence over our future are really not all that interested in libraries.” 1

- 1**Making the Case for Your Library: A How-To-Do-It Manual**.     Sally Gardner Reed.  How-To-Do-It Manuals for Librarians, Number 104.  New York: Neal-Schuman, 2001. p.6

- Housewarming

**Notes:**
Communicate not just what your library is doing, but why it matters.  Focus on “what” attracts users; focus on “why” attracts supporters.
Making a case for your library requires you to develop a strong message that conveys what your library’s mission is and why that mission is important to the larger organization. 
If I were to single out the one major mistake librarians make in trying to persuade others to support libraries it is that we forget that other people have other agendas.  Sadly and too often, the people who have the most influence over our future are really not all that interested in libraries.” 

“War stories”—storytelling puts information in context and brings facts and figures to life (The Bottom Line:  Determining and Communicating the Value of the Special Library.  Joseph R. Matthews.  Westport, Connecticut:  Libraries Unlimited, 2002.)

## More tips

**Know and rehearse your message**** **
**Talk to administrators one-on-one when possible**** **
**Focus on benefits**** **
**Don’t use library jargon**** **
**Educate, but don’t bore**

- Housewarming

## Go forth and evaluate and remember...

**Do smallball evaluation**** **
**Relate your value to the mission and goals of your organization **** **
**Plan your evaluation before you begin**** **
**Share your value with many audiences**** **
**Don’t wait for a crisis – start now!**

## Want more information?

**Resources List for this class**** **
**The OERC Web Site**
- http://nnlm.gov/evaluation/
**Web Resources for Evaluation**
- http://nnlm.gov/evaluation/webresources.htm

## Now, let’s all relax and enjoy Phoenix!