# Presentation: 066119

## Drinking Water AcademyWeb Conference The Public Water System Supervision Logic Model

**Thursday, February 24, 2005**
**11:30 – 2:00 EST**

- Drinking Water AcademyWeb Conference** ****The Public Water System Supervision Logic Model**

## Opening Remarks from Steve Heare, DWPD Director

## Logic Model Web Conference Agenda

- Presentation 				11:30 – 12:30
- Break					12:30 – 12:40
- Q&A:  First Round			  12:40 – 1:15
- Break					    1:15 – 1:25
- Q&A:  Second Round			    1:25 – 2:00

## Presentation Overview

- Brief Recap of Logic Models:  What and Why?
- Summary of Concerns and DWPD Responses
- Remaining Project Milestones
- Closing Thoughts

**Presentation Overview**

## Logic Model Development Work Group

- Region 2 – Robert Poon
- Region 3 – Michelle Hoover
- Region 4 – Janine Morris
- Region 5- Alicia Brown,
- 	Joe Janczy

**Logic Model Development Work Group**

- Region 6 – Blake Atkins
- Region 7 – Doug Brune
- Region 8 – Anthony Deloach
- Region 9 – Barry Pollock
- Region 10 – Craig Paulsen

- Other Headquarters Participants:  Evelyn Washington, 	Elizabeth McDermott and Ray Enyeart

## I.  Brief Recap of Logic Models:  What and Why?

## What is a Logic Model?

- A Logic Model is:
- a diagram or flow chart that shows how a program should work in theory
- 	“Logic” :  * ****how***** **do resources and activities lead to 				results?
- 	“Model” :   ***what***** **does our program look like?

## Background:   Components of a Logic Model

**Inputs** – human resources, funding, technology
**Activities** – ongoing work on products and services to achieve results
**Outputs** – most direct results of our efforts – evidence of products and 	service delivery.
**Outcomes **– changes in behavior, knowledge, skills or attitudes
**Impacts** – changes in conditions, such as improvements in health and 	safety.

## A Basic Logic Model

## Why Create a Logic Model?

## Why create a logic model?

- To meet external demands to explain our program...
-   Increasing demands for program performance data
-   Increasing interest in logic models and indicators
-   EPA Inspector General:  logic models for evaluation of
- drinking water programs and practices
-   OMB:  program evaluation for budget decision making**
- To improve program management and oversight relationship
-   Develop shared frame of reference for understanding the program
-   Develop better indicators of program health, status and trends
-   Create framework for systematic assessment

## Why Create a Logic Model?External Demands

**“****The principle here is clear:  Taxpayer dollars must be spent wisely, or not at all.”**
- 		- President Bush, State of the Union Address, 2/2/05
**“****Program assessments [by OMB] factor into Bush plan to trim deficit”**
- 		- Government Executive, 2/8/05
**“ ****Big Role for PART Ratings in 2006 Budget Cuts”**
- 		- Fedweek, 2/22/05
**“ ‘ ****People on Capitol Hill don’t always understand what PART does and how it can be useful to ****	****	****us.’ “**
- 		- Will Hart, Spokesman for Senate Environment and Public Works 				Committee Chairman 	Inhofe

## The New Budget Reality

- President’s Management Agenda:  Principles
-  Programs [**read:** **WE**] bear the burden of proof to show achievement of goals
-  Programs supported in the past will not necessarily be supported today
-  Funding not just on the basis of need, but also on accomplishments
-  Mismanaged, wasteful or duplicative programs will receive less funding,
- be redesigned, or eliminated altogether
- Budget decision-making criteria
-   Does the program meet the Nation’s priorities?
-   Does the program meet the President’s principles?
-   Does the program produce the intended results?
- Budget Risks are Real:  Proposed FY 2006 budget cuts
-  99 discretionary programs eliminated totaling  _$8.6 billion_
-  55 discretionary programs’ funding reduced totaling  _$6.5 billion*_
- *source:  OMB. “Major Savings and Reforms in the FY 2006 President’s Budget.”  February 11, 2005

## Use the PWSS Logic Model toHelp Manage the Budget Risk

- What does this mean?
-   ***Information demands*** increasing even as our budgets are flat or decreasing
-  *** ******Drinking water budget at increased risk. ***We must improve our ability to justify 	our program or we increase the chances of a budget cut.
- Our response?  _Budget risk management_.** **Let’s meet these demands and manage the budget risk by developing a logic model and indicators.
-   ***Make the best possible case*** for our program with existing information
-   ***Describe our performance in context*** to ensure that external factors and unique 	aspects of programs are acknowledged

## Why Create a Logic Model?  Improve Program Management and Oversight

## Characteristics of the PWSS Oversight and Management Relationship Today

-  we redirect limited resources to react to, rather than anticipate, implementation problems
**Rule-Focused**
-  we do assessments or attempt to diagnose problems without a sense of the bigger picture
**Anecdotal**
-  we do not use indicators to understand relationships between inputs, activities and results
**Assumptions Implicit**
-  we rely mostly on unstated assumptions about what’s working, what’s not, and why
**Ad Hoc Assessments**
-  we do not do program assessments using a common framework
**Context Secondary to Meeting Performance Targets**
-  we do not fully appreciate the importance of the story behind the results

**Characteristics of the PWSS Oversight and Management Relationship Today**

## Potential Characteristics of Oversight and Management Relationship in the Future

**Proactive**
** **let’s limit fire drills by anticipating implementation challenges
**Program-Focused**
-  let’s do program assessments to see how the pieces fit together
**Fact-based**
** **let’s** **use qualitative and quantitative information to support better decision making
**Explicit**
** **let’s test our assumptions based on a common framework
**Systematic**
** **let’s** **do program assessments regularly using the model and indicators
**Context Seen as Critical to Performance Measures**
-  let’s put our results in context to fully explain the performance story

## Why Develop a Logic Model? Improve Oversight and Management Relationship

- Let’s change how we:
- Understand
- Measure
- Assess
- Communicate

- Current approach:
- Fire-drills
- Rule-focused
- Anecdotal
- Implicit
- Ad Hoc
- Context Secondary

- Logic model approach:
- Proactive
- Program-focused
- Fact-based
- Explicit
- Systematic
- Context *Critical*

**Gradual Transition**

## Using the Logic Model and PWSS Indicators:  Limitations

** **We know that** **a logic model and indicators cannot tell the whole story.	Every program has a story to tell and context is important.
-  Let’s remember that although indicators can give us a sense of what’s going on, 	they can’t tell us what to do about it.
-  Sustained leadership is essential to making this process work.  DWPD is fully 	committed to using the model and indicators to improve management 	and oversight for the long term.

## II.  Comment Summary and DWPD Responses

## Comment Summary:  Regions and States Generally OK with Model

- Nearly all Region and State comments affirmed that:
-  The model has all the core program elements
-  The model flow is (with some minor refinements) logical and plausible
-  Some concerns about using it as shared frame of reference
- Recommended changes were:
-  Update or add definitions to the logic model boxes
-  Reword activities and outputs
-  Clarify relationships of activities to outputs
- Given these comments:
-  DWPD will issue a revised Model by the end of March
-  New release will include descriptions and rationale for changes

## How does DWPD envision using the model?

**Understand through Program Evaluation**
- Identify problem areas and their root causes
- Identify best practices in program operations
- Find barriers to implementation
**Measure Our Progress**
- Develop range of indicators to support management, planning and decision making
**Assess Where We Are**
- Assess state of program based on indicators (quantitative and qualitative)
**Communicate Internally and Externally**
- Improve dialogue within the program
- Protect our budget in the short run, make a case for additional resources in the long 	run
**Take Action**
- Share best practices in program operations
- Remove barriers to lessen implementation burden on states and utilities
- Set training and technical assistance priorities

## Using the Logic Model:  A Brief  Example

## Using the Logic Model:AWOP Example

***If ****TSC, Regions and States engage in:*
-  _Outreach_ to small systems to make them aware of available expertise
-  _Technical assistance_ for small systems on filtration technology
-  _Training_ for small systems on using filtration technology to meet rule requirements
***Then...***
-  _Information_ about AWOP expertise _is available_ to small systems 	to encourage enrollment in this voluntary program
-  _Systems are trained_ to optimize filtration through PBT
-  _Systems receive on-site technical assistance_ to optimize their 	system

## AWOP Example (continued...)

***If**** systems enroll in the program and receive performance-based training and on-site technical assistance, ****then...***
-  _Systems improve their understanding_ of how AWOP training and 	technical assistance can help them meet rule requirements
-   _Systems maintain adequate (technical) capacity_ to meet (or even 	exceed) turbidity standards
***If ****systems optimize filtration to maintain or enhance their technical capacity to comply with, or even exceed, turbidity requirements, ****then...***
-  Where *Cryptosporidium* is found in the source water, risk of 	exposure decreases as turbidity in finished water decreases 	(_human health protection via safe water_)

## AWOP Example (continued...)

- The logic model flow can help managers and engineers improve AWOP, 	focusing on the following kinds of questions:
- What assumptions did we make about what’s working, what’s not working 	and why?
- What do our data suggest about the relationships between AWOP 	technical assistance and training, systems’ increased understanding of 	optimization, and systems’ ability to reduce turbidity?
- Did the training and technical assistance reach the right people at the right 	time, and in the right way?
- Do we see the expected reductions in turbidity violations, or potential 	turbidity violations found in sanitary surveys, for systems that have 	received Performance-based Training?

## Will the Model Limit Flexibility?

- The Logic Model focuses on core programs common to all Regions and States
- Balancing flexibility with budget risk management and meeting program 	needs.
***How*** we do oversight varies across Regions
***What*** **national core indicators** are would be the same:  “common denominators”
- Remember:  The model is scalable for Region and State use
- How could a modified version of the model help you manage issues unique to each 	Region?
- Core indicators would not preclude additional indicators suited to management 	needs within each Region

## What is the relationship between the Model and the Strategic Plan?

- Today we use the following to communicate our progress in the 	Strategic Plan:
-  Strategic Plan Targets A through D (population served, systems in compliance, in 	terms of old and new rules)
-  Sanitary Survey program activity measure (PAM)
- Key Questions:
- Are these 4 Strategic Targets and 1 PAM enough to explain the program?
- Are these 4 Targets and PAM enough for us to manage the budget risk?

- Answer:  No.  Let’s use the logic-model to influence how future Strategic Plans portray the drinking water program
-    Develop better measures for portraying PWSS accomplishments
-    Develop better baselines for performance
-    Clarify how external factors (things outside our control or influence) affect our 	performance

## What is EPA doing to minimize the potential burden of using the model and indicators?

- Minimizing the burden one of DWPD’s _top priorities_.   To do so we will:
- Use existing data flows (Data Verifications, SDWIS, and Regional 	Reviews) to support the pilot
- Use existing reporting relationships to collect the data
- Test the model and indicators through a pilot in FY 2006.
- Conduct post-pilot assessment to assess whether we were able to use 	only existing data and reporting relationships.

## How does the model relate to other drinking water programs?

- Need to strike the right balance between:
-  “Macro-level” for Strategic Plan
-  “Operational level” for Programs *within *PWSS (e.g. AWOP)
- PWSS model intended for managers of core PWSS program
**  **Focus on the core program elements that describe PWSS
-   Core program lies between Macro and Operational levels
- Let’s pilot the PWSS logic model, consider other programs based on experience
- Good approximation for purposes of a pilot in FY 2006
-  All models subject to change through experience

## Slide 31

## Can you describe the Model Pilot in more detail?

- Piloting means we will:
-  Collect data using existing data flows and reporting arrangements
-  Assess value of model and indicators for future use
-  Assess effects of model and indicators on work load
- Logic Model Work Group will clarify in Summer 2005:
-  Expected roles and responsibilities
-  Final pilot indicators and associated data sources
-  Process for post-pilot assessment

## III.  Next Steps

## Project Schedule: Remaining Milestones

**Project Schedule: Remaining Milestones**

## IV.  Closing Thoughts

## Putting Our Model in Perspective

**“****It’s better to be roughly right than precisely ignorant”**
**We  can – and are expected to – know more about the relationship between ****	****what we do and the outcomes we want to achieve**
**Let’s focus on using indicators as “marker buoys” rather than “highway lanes”**
**Even the best indicators can only provide us direction; they cannot explain ****	****our program definitively**
**Let’s not let “the perfect be the enemy of the good”**
**The model we have is a starting point.  It is not set in stone.**
**The more we use the model and indicators, the greater their potential value.**
**The usefulness of the model and indicators will improve with sustained ****	****leadership and through our shared experience.**