# Presentation: 453360

## Working Group:Can Six Blind Men FindApples & Oranges? Measuring Variable Implementation of QI Interventions Using Multiple Data Sources

## Alexander S. Young, MD, MSHS
Elizabeth (Becky) Yano, PhD, MSPH
Lisa V. Rubenstein, MD, MSPH
Alison Hamilton, PhD

- Alexander S. Young, MD, MSHS
- Elizabeth (Becky) Yano, PhD, MSPH
- Lisa V. Rubenstein, MD, MSPH
- Alison Hamilton, PhD

## 90 minutes: presentations
60 minutes: group discussion and breakout groups
30 minutes: group consensus on priorities, suggested next steps, directions
Working group moves to plenary
5 minute summary presented 

- 90 minutes: presentations
- 60 minutes: group discussion and breakout groups
- 30 minutes: group consensus on priorities, suggested next steps, directions
- Working group moves to plenary
- 5 minute summary presented

## Presentations
Introduction and overview (Alex)
QUITS smoking cessation trial (Becky)
TIDES depression collaborative care (Lisa)
EQUIP evidence-based practice in schizophrenia (Alison)

- Presentations
- Introduction and overview (Alex)
- QUITS smoking cessation trial (Becky)
- TIDES depression collaborative care (Lisa)
- EQUIP evidence-based practice in schizophrenia (Alison)

## Brief orientation to example QI intervention 
How context matters  sets us up for variable QI intervention deployment
Process for intentional adaptation of evidence into context of local practice
Types of data sources brought to bear on measuring implementation
including development of a fidelity score
Triangulation of data sources to tell story 

- Brief orientation to example QI intervention
- How context matters  sets us up for variable QI intervention deployment
- Process for intentional adaptation of evidence into context of local practice
- Types of data sources brought to bear on measuring implementation
- including development of a fidelity score
- Triangulation of data sources to tell story

## QUITS (Quality Improvement Trial for Smoking cessation)
evidence-based quality improvement to implement smoking cessation guidelines 
Scott Sherman MD & Becky Yano PhD (co-PIs)
TIDES (Translating Interventions for Depression into Evidence-based Solutions)
depression collaborative care model 
Lisa Rubenstein MD & Ed Chaney PhD (co-PIs)

- QUITS (Quality Improvement Trial for Smoking cessation)
- evidence-based quality improvement to implement smoking cessation guidelines
- Scott Sherman MD & Becky Yano PhD (co-PIs)
- TIDES (Translating Interventions for Depression into Evidence-based Solutions)
- depression collaborative care model
- Lisa Rubenstein MD & Ed Chaney PhD (co-PIs)

## EQUIP (Enhancing QUality of care In Psychosis)
evidence-based quality improvement to implement effective treatments in schizophrenia
Alex Young MD & Amy Cohen PhD (co-PIs)

- EQUIP (Enhancing QUality of care In Psychosis)
- evidence-based quality improvement to implement effective treatments in schizophrenia
- Alex Young MD & Amy Cohen PhD (co-PIs)

## QUITS Smoking Cessation Trial

**EBQI**

- Education “toolkit”

- Local QI plan development

- Expert review/feedback

- Performance feedback

- Leadership support

- “local buy-in”
- “priority-setting”

- _Evidence base:_
-  SC clinic referrals
-  Tobacco quitlines
-  PC-based intn’s

## TIDES Depression Collaborative Care

**EBQI**

- Provider/patient education

- Depression care manager

- QI Informatics support

- Performance feedback

- Leadership support

- “adaptation”
- “priority-setting”

- _Evidence base:_
-  >20 RCTs
-  Depression
-   toolkit

## EQUIP Effective Schizophrenia Care

**EBQI**

- Provider/patient education

- Quality manager

- QI Informatics support

- Performance feedback

- Leadership support

- “infrastructure”
- “priority-setting”

- _Evidence base:_
-  TMAP
-  EQUIP-1

## TIDES
2:1 intervention-to-control sites x 3 VISNs (6 intervention + 3 control sites total)
VISN leaders chose sites, we randomized within network (block on network characteristics)
QUITS
regional concentration in southwest (3 VISNs)
matched on size/academic affiliation within VISN
we chose sites and randomized within network
EQUIP
1:1 intervention-to-control sites x 4 diverse VISNs
sites chosen based on leadership interest

- TIDES
- 2:1 intervention-to-control sites x 3 VISNs (6 intervention + 3 control sites total)
- VISN leaders chose sites, we randomized within network (block on network characteristics)
- QUITS
- regional concentration in southwest (3 VISNs)
- matched on size/academic affiliation within VISN
- we chose sites and randomized within network
- EQUIP
- 1:1 intervention-to-control sites x 4 diverse VISNs
- sites chosen based on leadership interest

## Attitudes / beliefs / experiences
perceived need for the intervention
competing demands
staff openness to innovation
Resources
perceived time to use program and participate in implementation
organizational structure, staffing, prior QI experience, informatics support

- Attitudes / beliefs / experiences
- perceived need for the intervention
- competing demands
- staff openness to innovation
- Resources
- perceived time to use program and participate in implementation
- organizational structure, staffing, prior QI experience, informatics support

*Source*:  Kirchner JE, Parker LE, Yano EM, COVES evaluation (2007).

## Multiple Data Sources: Measuring Implementation

|  | TIDES | QUITS | EQUIP | Examples |
| --- | --- | --- | --- | --- |
| Semi-structured interviews:leaders, clinicians, mgrs |  | -- |  | participation, level of implementation |
| Organizational site surveys:
PC/MH leaders, SC mgrs |  |  |  | clinic structure, processes, change |
| Field journals | -- | -- |  | group-level dynamics, implementation details |
| Administrative data |  |  |  | visits, Rxs, costs |
| Patient surveys |  |  |  | PHQ9, BASIS, quits |
| Clinician surveys & activity logs |  |  |  | knowledge, attitudes, behaviors |
| Practice checklists (experts) | -- |  | -- | QII components |

**Notes:**
Qualitative semi-structured interviews.  Quantitative organizational site surveys.  

## Multiple Data Sources:Strengths and Challenges 

|  | Strengths | Challenges |
| --- | --- | --- |
| Semi-structured interviews:leaders, clinicians, mgrs | rich data, diverse perspectives  | expensive, time-consuming |
| Organizational site surveys:
PC/MH leaders, SC mgrs | site profiles, fast, easier to analyze | limited discovery, key informant view |
| Field journals | detailed contextual data | variation between observers |
| Administrative data | readily available, historical value | not QII-specific, local coding differences |
| Patient surveys | experience, exposure, outcomes | expensive, sensitive to sample |
| Clinician surveys & activity logs | implementers! | difficult responders |
| Practice checklists (experts) | face validity, eyes on the ground | variation between observers |

## Critical to collect information about implementation from multiple sources
be prepared for disagreement
perspectives and opportunities for observation differ for managers, providers vs. patients
Recognize differences between “exposed” sample and practice population
does the “enrolled” group represent the practice?
did the intervention penetrate among all providers?

- Critical to collect information about implementation from multiple sources
- be prepared for disagreement
- perspectives and opportunities for observation differ for managers, providers vs. patients
- Recognize differences between “exposed” sample and practice population
- does the “enrolled” group represent the practice?
- did the intervention penetrate among all providers?