# Presentation: 103847

## Effective Program Evaluation: Using the Quality Evolution framework to build a successful program.

- Environmental Protection Agency
- National Air Quality Conference
- February 12-17, 2005
- Presented By: Heidi Adams

## Presentation Overview

- What are the definitions that support QA/QI?
- What is program evaluation?
- What methods do we use to evaluate our programs effectively?
- How do we do it?

## Quality Rules to Live By

- Today’s problems come from yesterday’s solutions
- The harder you push, the harder the system pushes back
- Behavior grows better before it grows worse
- The easy way out usually leads back in
- The cure can be worse than the disease

- Faster is slower
- Cause and effect are not closely related in time and space
- Small changes can produce big results
- You can have your cake and eat it too – but not all at once
- Dividing the elephant in half does not produce two small elephants
- There is no blame

**Source: The Fifth Discipline – Peter M. Senge**

**Notes:**
A quality assurance/quality improvement program developed at the beginning of a process rather than at the end will ensure that some issues are avoided – 
Quality is learned and must be intertwined in the overall system.  Quality cannot be forced it must be seen as a welcomed addition 
Systemic problems are hard to recognize – it is essential that we gather all of the information before trying to solve or correct a problem
Sometimes the solution is right in front of us if we learn to ask the right questions and sometimes the solution is hidden which requires more work, fact finding, and slouthing to find the source of the problem and the proper solution – ongoing quality improvement has these features built in
Sometimes we have to take something apart in order to fix it
Developing a quality system takes time if it is going to make a positive impact
Identification of the symptoms of a problem do not always interelate to the problem itself
Even the smallest adjustment to a process could yield large results in quality of care
It is possible with planning everyone could get what they need – it will take a gradual phasing in process and may require that use a couple of different systems to yield the results
Understanding challenging issues requires that we look at the whole system – by splitting it into smaller parts we are only diluting the outcome
True systems thinking tells us that there are no outside parties to blame – we must be able to look at our own system and partner accordingly to create a sound process that benefits all

## What is Quality?

- It is program specific

## What Is Quality Assurance?

- Developing tools that will assist with internal and external compliance with policies, regulations, and outcome expectations.
- Ongoing process of continuous internal process audits.
- Developing the most effective system and process available.

**Notes:**
We will develop policies and standards that will aid in the system running smoothly 

## What Is Quality Improvement

- Identification of areas that are in need of further development and enhancement.
- Continuous critical look at the system.
- Utilizing data resources to target areas

**Notes:**
Ensuring that if there is a break down in the system or if a break down is pending then we will proactively target the areas in need of improvement and set goals and establish plan for correction.

## Program Evaluation

- Why do we do it?
- Understand impact on the greater society
- Program Growth
- Understand where changes are needed

## Program Evaluation

- How do we do it?
- Collect data
- Customer satisfaction
- Special population surveys
- Impact on community (media, online, hotline, education/outreach)
- Question our motives
- Develop strategic planning tools
- Set goals
- Research environment – changes in the industry, geographic changes, changes in regulations...

## Program Evaluation

- How do we do it well?
- Begin with a plan which includes goals, objectives, mission, and vision
- Move into program analysis
- What are we doing?
- Is it effective?
- What should we be doing?

## Quality Evolution Framework

**Business Plan**

**Program evaluation is evolutionary not circular – it is a blending of ideas**
**Begins at the beginning – it evolves out of a business plan/a strategic plan/a long range plan. **
**Go back to the beginning**

- People want to target and fix problems
- Rarely look at the system as a whole before applying bandage
- Ending up with a pyramid of issues

- Often times people take the circular method and never accomplish anything ending up with the same frustration
- Never going anywhere

## Step 1

**5 point initial assessment**
- Who do you want to be?
- What do you want to do?
- How do you want to get there?
- Who is involved?
- When do you want to do it?

## Step 2

- Reassess
- Vision and Mission
- Goals and Objectives – long and short term
- Achievements
- Current – Future – Gaps

## Step 3

- What does the data say?

## Step 4

- Develop a work program

- Organizational Recruitment

- Membership Recruitment
- and Retention

- Community Outreach

- Fundraising

- Program Evaluation

## Step 5

- Identify challenges utilizing gap analysis and develop mechanisms to address challenges
- Develop a monitoring guide and tool that will assist with continuous self evaluation

## Developing a Monitoring Guide

**Each Standard Includes:**
**Objective Statement** – A goal statement or acceptable performance statement
**Intent Statement** – a sentence that describes the importance of the standard from a consumer viewpoint
**Element** – specific questions that help to identify an organization’s performance that will be evaluated
**Method of Evaluation/Data Source** – types of documentation and evidence used to support compliance of a standard
- _Documented process_ – policies and procedures, documented flow charts, protocols, and other mechanisms that describe an actual process used by the organization.
- _Reports_ – sources of evidence
- _Materials_ – prepared materials the organization develops to aid in educating and disseminating information
- _Records or files_ – any additional evidence such as interview documentation that proves compliance
**Scoring** – The level of performance of the organization for each standard element

## Step 6

- ACT!!

## Organizational Recruitment

**Identify Point of **
**Contact**

**Identify Organizations **
**Most Likely to Care **
**About Environment**

**Compile corporate **
**lists**

**Develop Preliminary **
**Work Plan**

**Develop **
**Documents/Tools**

**Develop Script**

**Plan Meetings/**
**Direct Contact**

**Identify nationwide **
**best practices **

## Case Study

## A Bi Regional Program

- Program covers two major metropolitan areas with two different views of air quality and its impact
- The mission of the program stays the same, however, the goals and objectives change each year which may not match the mission
- Each person in a leadership role has their own agenda which may not be the original intent of the program
- The program has not grown truly in seven years
- Staff changes frequently

## Issues

**Inconsistent**
**Priorities**

**Mission creep**
**lack of understanding**
**of program development**

**Inconsistency in staffing**

**Growth has been**
**stagnate**

## Evaluation Components

**Board Retreats**

**Professional**
**Surveys**

**Evaluation of**
**Manager**

**Rarely to never implement**
**recommendations**
**of surveyor**

**Personal evaluation**
**by only a few individuals**
**never established**
**evaluation goals**

**Nice get together**
**where the same issues and**
**frustrations are voiced**
**each year with no resolution**

## How effective was the evaluation?

- Alienated board members who wanted to make a difference
- Frustrated contractors
- Confused the external community with different messages (outreach and education program or a regulator??)

## What should they do?

**Programmatic Plans**
**For Each Area**

**Outcomes**

**Sources of Information**

**Deliverables**

**Try to alleviate dysfunction by revisiting program**
**origins**
**They may want an external review of program**
**to help evaluate effectiveness at all levels**
**Go back to the data and use it **

**Individuals Involved In Process**
**Board of Directors**
**Staff**
**Organization Members**
**Focus Group Participants**

## Quality Evolution Framework

**Business Plan**

- Program evaluation is evolutionary not circular – blending process
- Begins at the beginning – it evolves out of a business plan/a strategic plan/a long range plan.
- Go back to the beginning

## Questions?