# Presentation: 602965

## EVALUATION:  GETTING STARTEDFirst Steps Toward Finishing Strong

**Candace J. Chitty, RN, MBA, CPHQ, LHRM**
**President**

**Where Results Begin**

## Never doubt that a small group of thoughtful, 

**committed people can change the world.  **
**Indeed it is the only thing that**
**ever has.**
-                                             -**Margaret Mead**

## Why Evaluate?

**Collecting information about how things are done and the results helps us understand how community initiatives develop and disseminate lessons other groups can profit from.**
**Providing **_**ongoing feedback**_** can improve community work by encouraging continuous** **adjustments of programs, policies and activities.**
**By involving community members, people who haven’t had a voice may gain the opportunity to better understand and contribute to program efforts.**
**Finally, evaluation can help hold groups accountable to the community and to the grant makers who provide funding.**

## Institute of Medicine’s 2000 Report

**Described our healthcare safety net as “intact, but endangered”.  One of the key recommendations in the report focused on the need for data systems and measures to **_**evaluate**_** the performance of the safety net and health outcomes of vulnerable populations.**
*2000, Institute of Medicine’s “Crossing the Quality Chasm”*

## HCAP Legislation: Grantee Program Monitoring & Evaluation Expectations

**Grantees will:**
- Describe a plan for evaluation of the activities carried out under the grant, including _**measurement of progress toward the goals and objectives**_ of the program and the _**use of evaluation findings to improve program performance**_ (quality improvement).

## What does the Legislation Require?

- HCAP Grantees are required to _**report data at least every six months**_ that will show the extent to which HCAP activities have:
- Improved the _**effectiveness**_**,** _**efficiency**_, and _**coordination of services**_ for uninsured and underinsured individuals in the communities or geographic areas served by each grantee

## What does the Legislation Require?

- And whether these activities resulted in:
- The _**provision of better quality health care**_ for such individuals and;
- The provision of health care to such individuals was at a _**lower cost**_ than would be possible in the absence of the activities conducted [under HCAP].

## Key Evaluation Principles

***First Steps:***
- Realize that evaluation is a critical and ongoing task of successful programs.
- Get started early.
- Implement your evaluation at the same time as your initiatives.
- Support collaborative evaluation planning.
- Use evaluation as a catalyst for community change and sustainability – **Finishing strong!**

## Slide 9

## Evaluation FrameworkSix Steps

**Step 1:  Engage stakeholders**
**The evaluation cycle begins by engaging stakeholders (i.e. persons or organizations having an investment in what will be learned from the evaluation and what will be done with the knowledge)**
** ****Helps increase chances that the evaluation will be    useful**
** ****Can improve evaluation’s credibility**
** ****Clarifies evaluation roles and responsibilities**

## Evaluation FrameworkSix Steps

**Step 2: Describe the program**
**Program descriptions convey the mission and objectives of the program being evaluated.  They should be sufficiently detailed to ensure understanding of goals and strategies.**
** ****Improves evaluation’s fairness and accuracy**
** ****Permits a balanced assessment of strengths and weaknesses**
** ****Helps stakeholders understand how program features fit together and relate to a larger context**

## Why Use the Logic Model?  

**How will it help you?**
**Helps to clarify what is appropriate to evaluate, and when, so that evaluation resources are used wisely.**
**Summarizes complex programs to communicate with stakeholders, funders, audiences.**
**Enables effective competition for resources. (Many funders request logic models in their grant requests.)**

## Sample Logic Model with Evaluation Questions

## Evaluation FrameworkSix Steps

**Step 3: Focus the evaluation design**
**The evaluation must be focused to assess the issues of greatest concern to stakeholders while using time and resources as efficiently as possible.**
** ****Provides investment in quality**
** ****Understanding how evaluation results are to be used**
** ****Describes practical methods for sampling, data collection, data analysis, interpretation and judgment.**
** ****Provides a written protocol or agreement that summarizes evaluation procedures with clear roles and responsibilities of stakeholders**
** ****Revisions of parts or all of the plan when critical circumstances change**

## Evaluation FrameworkSix Steps

**Step 4: Gather credible evidence**
**An evaluation should strive to collect information that will convey a well-rounded picture of the program so that the information is seen as credible by primary users.**
** ****Improves evaluation’s fairness and accuracy**
** ****Permits a balanced assessment of strengths and weaknesses**
** ****Helps stakeholders understand how program features fit together and relate to a larger context**

## Evaluation FrameworkSix Steps

**Step 5: Justifying conclusions**
**Evaluation conclusions are justified when they are linked to the evidence gathered and judged against agreed-upon values or standards set by the stakeholders.**
** ****Reinforces conclusions central to the evaluations utility and accuracy**
** ****Involves analysis and synthesis of qualitative and quantitative data, systematic interpretation, and appropriate comparison against relevant standards**

## Evaluation FrameworkSix Steps

**Step 6: Ensuring use and sharing lessons learned**
**Preparing for use involves strategic thinking and continued vigilance, both of which begin in the earliest stages of stakeholder engagement and continue throughout the evaluation process.**
** ****Ensures that evaluation achieves its primary purpose –**_**being useful**_

##  Standards for Effective EvaluationJoint Committee for Standards for Educational Evaluation

- Standards help to avoid creating an unbalanced evaluation ( e.g. one that is accurate and feasible but not useful or one that would be useful and accurate but is not feasible).
- Standards are grouped into four categories and include a total of 30 specific standards.
*Source:  www.eval.org/EvaluationDocuments/progeval.html*

## Standards for Effective Evaluation

**Category One:  Utility Standards**
**Ensures that information needs of evaluation users are satisfied.  **
**Stakeholder identification**
**Evaluator credibility**
**Information scope and selection**
**Values identification**
**Report clarity**
**Report timeliness and dissemination**
**Evaluation impact**

## Standards for Effective Evaluation

**Category Two:  Feasibility Standards**
**Ensures the evaluation is viable and pragmatic.  **
**Practical procedures**
**Political viability**
**Cost effectiveness**

## Standards for Effective Evaluation

**Category Three:  Proprietary Standards**
**Ensures the evaluation is ethical.  **
**Service orientation**
**Formal agreements**
**Rights of population served**
**Human interactions**
**Complete and fair assessments**
**Disclosure of findings**
**Conflict of interest**
**Fiscal responsibility**

## Standards for Effective Evaluation

**Category Four:  Accuracy Standards**
**Ensures the evaluation conveys technically adequate information regarding the determining features of merit of the program.  **
**Program documentation**
**Context analysis**
**Described purposes and procedures**
**Defensible information sources**
**Valid information**
**Reliable information**

## Standards for Effective Evaluation

**Category Four:  Accuracy Standards**
**Systematic information**
**Analysis of quantitative information**
**Analysis of qualitative information**
**Justified conclusions**
**Impartial reporting**
**Metaevaluation**

## Defensible Information Sources  

**Reliable. **
**Valid.**
**Operational definitions.**
**Defined populations.**
**Measurements over time.**

## Reliability  

**Reliability is concerned with the accuracy of the measurement instrument or procedure.**
**Reliable data provides stable results over time. **

## Validity

**Validity is concerned with the success at measuring what is intended to be measured.**
**Validity is concerned with how a measure or procedure appears. **
**Does it seem like a reasonable way to gain the information the program is attempting to obtain? **
**Does the measure seem well designed? **
**Does the measure seem as though it will work reliably?**
**Are the results generalizable?**

## Operational Definitions

**Maps the data collection process to ensure consistency.**
**Measurement specifications (what, when, how)**

## Clearly Defined PopulationsIMPORTANT 

**Uninsured**

**Underinsured**

**Age Category**

**FPL Served**

## Measurements Over Time

**Baseline**
**Re-measurement Trends**

## Comprehensive Analysis

**Performance Goals and/or Benchmarks.**
**Data Analysis and Synthesis**

## Performance Goals and/or Benchmarking

**If you don’t know your destination any path will do.**

## Data Analysis and Synthesis

***Data are assessed in order to determine:***
**What were the findings? **
**What outcomes went well ? **
**What are the potential causes contributing to less than desirable outcomes? **
**How do these causes effect the outcomes? **
**What could have been done better? **
**What would we change and do differently? **

## Who Should Conduct Your Evaluation?

**Evaluation is a **_**team effort**_** that works to:**
**Determine the focus and design of an evaluation**
**Develop the evaluation plan, performance indicators, and data collection instruments**
**Collect, analyze, and interpret the data**
**Prepare reports on evaluation findings**

## Types of Evaluation Teams

**Option 1:  Hiring an outside evaluator**
**Option 2:  Using an in house evaluation team supported by an outside consultant and program staff**
**Option 3:  Using an in house evaluation team supported by program staff**

## Which Evaluation Team Option is Best?  

**Does your HCAP program have funds designated for evaluation purposes?**
**Have you successfully conducted previous evaluations of similar programs, components, or services?**
**Are there existing measures or performance indicators in place?**
**Can you collect evaluation information as part of your regular program activities?**
**Are there program staff who have training and experience in evaluation-related tasks?**

## Hiring an Outside Evaluator

**Before begin the finding and hiring an outside evaluator develop a statement of work (SOW) which details the general and specific requirements for the evaluator.  **
**In creating the SOW you will need to know the type of evaluation activities you want the evaluator to perform. **

## Hiring an Outside Evaluator

**Consider the following:**
**What level of experience does the evaluator have in the area of program evaluation?**
**Does the evaluator have experience conducting evaluations in the area of healthcare access programs?**
**Can the evaluator offer assistance in the full range of program evaluation activities?**

## Hiring an Outside Evaluator

**Consider the following:**
**Will the evaluator present information in a way that is useful to you?**
**How much will it cost?**
**Is the evaluator willing to work closely with you?**

## A Good Evaluator:

**Is willing to work collaboratively to develop an evaluation plan that meets your needs.**
**Is able to communicate in simple, practical terms**
**Has experience evaluating similar programs and working with similar levels of resources**
**Has experience with statistical methods**

## A Good Evaluator:

**Has the time available to do the evaluation**
**Has experience developing data collection forms or using standardized instruments**
**Is willing to work with a national evaluation team (if there is one)**
**Will treat data confidentiality**

## Managing Evaluation by an Outside Evaluator

**One mechanism for effectively managing the evaluation is to prepare a written contract.  The contract should include:**
**Who will perform evaluation tasks**
**Who owns the evaluation data**
**Your expectations about contacts with the evaluator and the program**

## Finishing StrongUsing Evaluation in Sustainability:  Policy Change and Investment Influencers

**Where Results Begin**

## In business, consumer and supplier buying decisions are heavily influenced on a seller’s past performance.  The seller must demonstrate proven and effective delivery of products and/or services.

## Ideally, in healthcare, policymakers and  community stakeholders should be able to make policy  and/or investment decisions based on available, understandable, and reliable outcome performance measures.

## The ideal may be difficult to realize in today’s healthcare system due to:
Proven and effective outcome measures remain few.
Data for developing outcome measures are often hard to come by.
The population’s are often transient in nature.

**Proven and effective outcome measures remain few.**
**Data for developing outcome measures are often hard to come by.**
**The population’s are often transient in nature.**

## Data Influences Policy

**Outcome performance measures derived from sound methodology presented in a clear and understandable way cannot ensure the desired outcome but it has been **_**proven to be a compelling strategy for getting the attention of key policy and decision-makers.**_
*Source: Agency for Healthcare Research and Quality*

## Reliable  outcome data allows key policy and decision-makers to:
Assess the effects of health care program and policy choices.
Guide future health care policy making.
Accurately measure outcomes, community access to care, utilization, and costs.

**Assess the effects of health care program and policy choices.**
**Guide future health care policy making.**
**Accurately measure outcomes, community access to care, utilization, and costs.**

**One of the Keys to “influencing” a policy or community change is to **_**clearly deliver the message**_**. This means getting the message straight by presenting information which is **_**understandable, concise and factually correct**_**.**
*Lawrence and Lewin, Presenting Information to Decision-makers: A Guide for Policy Analysts*

## Using Evaluation to Deliver a Clear Message

**Tells a Good Story.**
**Incorporate information from performance measurement into the story.**

## Value of Evaluation Data in Telling The Story

**Data is generally used for one of these purposes:**
**To Persuade (that is, you are giving your opinions about what *****you *****think should be done given the facts you present); *****or*****  **
**To Inform about trends (“just the facts ma’am”) so *****your reader *****can decide what they think should be done given the facts you present.**

## A good story tells us what we didn’t know before, and after reading it, we realize we really, really wanted to know it.

## Elements of A Good Story

_**Enticing**_** – Hey-this story looks interesting. **
_**Captivating**_** – I can’t put this down.  Focuses on things people will remember after hearing your story.**
_**Persuasive**_** – This is for real – Data-driven performance based on sound methodology.**
_**Stimulating**_** – The WOW Factor. **

## To Sum It Up....

- Evaluation is an early and central part of your HCAP program.
- You have taken your first steps toward  finishing strong by understanding key evaluation principles, framework and standards and applying them as guidelines in your evaluation plan.
-  **Good luck.**

## Slide 54