Using Evaluation to Improve O

Work:

A Resource Guide

December 16, 2010
Audience

Survey: Who is on this call? (pick your primary)

VACO/national program leaders or staffFacility‐based leaders/staffVISN‐based leaders/staffResearchersOtherPlease write in your questions: CIPRS@va.go

submit your questions questions via LiveMeeting. LiveMeeting.
Course Outline

• Why is evaluation important in VHA?• How is evaluation being used in Systems Red• Using Evaluation to Improve Our Work: A Res

Guide

• After Action Review

• Team Development Measure• Q&A
Presenters

• Joe Francis, MD, MPH, Chief Officer, VHA Office of Quali

Performance

• Michael Davies, MD, Director, VHA Office of Systems Re

• Dede Ordin, MD, MPH, Director, Special Studies, VA Offi

Quality and Performance

• Susanne Salem‐Schatz, ScD, Investigator, VA Center for

Implementation Practice and Research Support

• Richard Richard Martinello, Martinello, MD, Acting Senior Medical Medical Advisor, Advisor, V

of Public Health and Environmental Hazards

• Craig Roth, MD, Co‐Chief, General Medicine, Minneapoli

Medical Center

Moderator: Brian Mittman, PhD, VA Center for Implementa

Practice and Research Support
Why is evaluation importa

in VHA?

Joe Francis, MD, MPH

Chief Officer

VHA Office of Quality and Performan
What is Evaluation?

Evaluation:• Is a planned, purposeful and systematic proc

• Involves collecting data on work, programs, a

organization, which

• Enhances knowledge and decision‐making an

• Differs from:

– measurement for accountability or reward

– measurement for research– measurement for quality improvement
What is Evaluation?

Evaluation can be useful in VHA:• Evaluations can be conducted at the pro

office, policy, and practice practice levels

• Evaluations can aid communication wit

veterans veterans (i e. ., evaluation evaluation of My Healthe

Healthe
Why Evaluation?

• Evaluation helps us to learn (internal).• Evaluation helps us to communicate and to

motivate/persuade (external)

• The most effective learning and communicat

should engage the entire brain:

– Left brain: numbers

– Right brain: stories
What is Evaluation?

There are two basic types of evaluation:• Evaluations can guide the ongoing developm

evolution of a program or improvement initia

(helps us to improve while we are doing)

• Evaluations can also provide information abo

program eff i ect veness and impact (h l e ps us t

improve for the next time)
Evaluation in VHA

Evaluation for learning and improvement is already em

into many aspects aspects of our work:

• Office of Emergency Management in the Center for

Environmental and Occupational Safety and Health (

provides guidance for the After Action Reviews that 

conducted following each real and simulated emergeevent.

• Root cause analyses to evaluate problems and priori

solutions following adverse events.

• The VHA System‐Wide Ongoing Ongoing Assessment Assessment and Rev

Strategy Program (SOARS) provides tracer and self‐assessment tools and consultation
How is evaluation being us

in Systems Rd i ? e esign?

Michael Davies, MD,

Director

VHA Office of Systems Redesign
Evaluation in Systems Redesig

VHA Mission: Mission: Provide Provide exceptional exceptional healthcare healthcare th

improves our health and well being

SR Tagline: Improving our work is our workVHA aims to create a culture of improvement“Thousands of teams and millions of changes”
Evaluation in VHA

Evaluation is heart of culture of improvementMany kinds of evaluation/feedback

– Achievement of a goal– Process or outcome– Concurrent or retrospective

Evaluation helps align and inform our overall

improvement effort
Background of Evaluation Resource Gu

• Systems Systems Redesign Redesign Steering Steering Committee Committee please

please

support, align and connect the people doing 

work with those interested in evaluation.

– Systems Improvement Subcommittee

• Future Vision: Incorporate into guidebook so 

all speak with one voice to “operations operations” whil

improving our knowledge behind the scenes
Q&A

Moderated by:

Brian Mittman, PhD

Director

VA Center for Implementation Practic

and Research Support
Using Evaluation to Impro

Our Work: A Resource Gui

Dede Ordin, MD, MPH

Director, Special Studies Studies

VA Office of Quality and Performan
Improving the Work of Improvemen

• Funded by OQP and QUERI• Focus on eval i uat on tools

• Needs assessment to prioritize tool developm

– SR POCs– Cancer Care Collaborative coaches– QMOs

• Identified priorities:

– Structured approach for reflective learning

– Tool for team self‐assessment and strengt
Using Evaluation to Improve Our Work

A Resource Guide

• Designed to help VHA staff and leaders expan

A Resource Guide

• Designed to help VHA staff and leaders expan

deepen current evaluation activities.

• To be used for self‐evaluation and improvem

rather than accountability.

• Resources can be useful at any point in the lif

project

18
Using Evaluation to Improve Our Work

A Resource Guide

Overview of evaluation

A Resource Guide

After Action Review:

• Structured reflection

Team Development Measure:

• Assess development

Additional evaluation resources:

• General Program Evaluation Resources

• Resources Rltd e a e to E l ti va uation in VA• Examples of Evaluations Conducted in VA

19
After Action Review

Richard Martinello, MD

Acting Senior Medical Advisor

VHA Office of Public Health and Environment

Environment

Hazards

Susanne Salem‐Schatz, ScD

Investigator

VA Center for Implementation Practice and Rese

Support
After Action Review

(AAR) Survey

Have you used an AAR before?
Guide to the After Action Review

A tool to assess the strengths and weaknesses of past even

project to inform future teams and work

• 1‐page overview overview• Planning an After Action Review

– Logistics and facilitation

– Organizing the AAR conversation by key events, them

issues

• Conducting an After Action Review

– Introduction including meeting context and purpose

– Sample ground rules and facilitation guidance

• Framework and plan for sharing AAR findings

– AAR Report Template
After Action Report

What is it?• A report generated in response to the observed act

outcomes from an event.

• Brief summary of what happened

• What worked well?• What did not work well?• What needs to be accomplished accomplished or further further explored explored 

ensure a better response during the next similar ev

Ultimately, make recommendations to improve the resp

the next event.
What does the AAR look like?

Framework

HEICS National Framework for Influenza Prepar•Logistics•Operations•Fiscal

Response•Surveillance

Fiscal •Mitigation

•Planning •Vaccination

•Communication and Education

Depth & Breadth: depends on...

•Needs/goals

•Resources•Examples:Single page, focused on one aspect

Hundreds of pages (ex: WH Katrina AAR
Production of the AAR

– Should start when the event starts

• Take notes• Take pictures (both good and bad)

• Consider what could be better, what additional res

would help, etc. during the event

– Assemble a team and a plan

• Individuals involved with event response• Stakeholders

– Internal Internal (i e. ., part of your organization) organization)– External (i.e., outside organization, community)

• What is the goal and the focus?
Production of the AAR (cont’

– Assemble data

• Methods

– Brief discussion among responders/stake

responders/stake

– More extensive survey

• Maintain Maintain an open atmosphere atmosphere, encourage encourage o

thinking, brainstorming and a no blame

environment

– Organize data and draft the report
AAR Recommendations

• Perhaps the most critical critical section section• List actionable items based on the analysis

• Specific action/task

• Example: purchase more ventilators• Further analysis recommended recommended

• Successful recommendation characteristics

• Specific Specific, assign task to an individual/group individual/group

date, follow‐up
AAR Recommendations (cont

• Integration of AAR findings findings and

recommendations

• Revision Revision of emergency emergency operations operations pla• Integration of certain findings into no

emergency operations processes

• Improve likelihood of sustainability

• Opportunities to further refine pro
Team Dl t eve opment Measu

C i ra g R th o , MD

Co‐Chief, General Medicine

Minneapolis VA Medical Medical Center

Susanne Salem‐Schatz, ScD

Investigator

VA Center for Implementation Practice a

Research Support
Team Development Measure©

(TDM) Survey

Have you used the TDM before?
Guide to the Team Development Meas

A tool to track team development and supp

improvement

• 1‐page overview

• About team development• Implementing the TDM Survey

– Automated by VSSC

– Facilitation guidance

• Action plan worksheet• Tips for successful teamwork
TDM © Background

• Used with permission from PeaceHealt• Used in healthcare delivery teams,

administrative, project project, and executive executive t

• Valid and Reliable• User‐friendly

• Useful
Team Development Measure©

• Point in time assessment of Team Development• 31 questions, 4 key aspects

– cohesion

– communi ti ca on

– clarity of team roles

– clarity clarity of team goals and the means to achieve achieve th

• Single TDM Score can be tracked over time• Creates the opp y ortunity for reflection

– External facilitator is recommended– Teams can review on their own if needed
VHA Use of the TDM©

VISN Name VISN Total Teams

(V01) VA New England

Healthcare System

19

(V07) VA Southeast Network 3

(V08) VA Sunshine Healthcare

Network

2

(V10) VA Healthcare System of

Ohio

1

(V11) Veterans Veterans In Partnership Partnership 8(V12) VA Great Lakes HealthCare System

5

(V19) Rocky MountainNetwork

1

(V23) VA Midwest Health CareNetwork

37

TOTAL 76
Sample TDM© Report
Sample TDM© Report
Case Study

Setting

Minneapolis VA Medical Center

Large, primary and tertiary care
Teams

• 2 pilot primary primary care Patient Patient Aligned Aligned Care Tea

(PACTs).

• Formed March 2010.

• Members:

– Selected Selected by GIM Chiefs and Nursing Nursing Direc

Direc

– Attended VA Learning Sessions in Las Vega

2010) + 2 regional learning collaboratives

(principles of PACT).
Team 1

• Physician

• RN• LPN

• MSA
Approach we took

• I told leaders leaders about TDM©, and was invited invited t

with teams.

• Teams embraced the idea

• I arranged to have e‐mail links set up

• Members completed anonymously

• I gathered and distributed results to all mem

• I facilitated facilitated discussions discussions ( 1~ hr)
Why we tried the TDM©

• Assess stage of team development development (baseline) (baseline)• Help team clarify what to work on (instructiv

• Create/stimulate safe dialogue dialogue for:

– Team formation (cohesiveness)– Communication– Role & Goal clarity
Overall Score
Cohesiveness

June, Baseline October

Craig Roth 2010
Communication

June, Baseline October
Role Clarity

June Baseline O b , cto er
Goals and Means Clarity

June, Baseline October
Reactions from team

• Very lively discussions. Full participation (mo

previ l) ous y .

• Some surprises:

– Diversity of answers

– Not as development as hoped

• Identified: strengths, areas to work on and fe

• Eager to take again to assess progress.
What teams liked

• Easy access, short completion time, eas

questions, data display friendly, quick t

around, tracking answers, storing data (

• Anonymous (safe)• No cost• Stimulated discussion: data
Strengths

• Instructive: demonstrated:

– elements/competencies/stages of te– where team sh ld ou be going

• Useful to appreciate strengths, items to

celebrate
Future Plans

• All teams will use: baseline baseline, 3, 6, 12 mo

• Use trained TDM facilitators, using high

structured and standardized standardized approach approach.

• Monitor more objectively how teams a

progressing and where success is occur

and to support/change where needed.

• Will consider putting some items on slid

project and discuss.
Q&A

Moderated by:

Brian Mittman, PhD

Director

VA Center for Implementation Practic

and Research Support
How to Access the Evaluation Resource G

Available Available on CIPRS and Office of Systems Systems Redes

Redes

SharePoint sites:

• http://vaww.portal.gla.med.va.gov/sites/Res

SRD/CIPRS/Evaluation_Resource_Guide/defa

• https://srd.vssc.med.va.gov/Pages/UsingEval

Imp p roveOurWork.aspx
Fielding the TDM Survey

Automated system by VSSC

1. Email CIPRS@va.gov to request access to the sy

2. Automated by VSSC

3. Once access is granted granted, need the following following info

info

to set up a survey:

• Name of team

• Names of team members

4. Online survey link sent to team members

5. Can check on the number of respondents respondents, close 

survey, produce a report, and distribute report members
Any additional additional questions questions can be sent 

CIPRS@va.gov