# Presentation: 235031

## Lessons Learned in Clinical Decision Support: Over-alerting

- Tejal K. Gandhi, MD MPH
- Director of Patient Safety
- Brigham and Women’s Hospital
- Assistant Professor in Medicine
- Harvard Medical School

## Overriding of Alerts

- Studies have shown that MDs override clinical decision support alerts a large percent of the time
- 88% of inpatient DDI alerts overridden (Payne et al. Proc AMIA 2002)
- 83% of inpatient drug-allergy alerts (Abookire et al. Proc AMIA 2000)
- 89% of outpatient high severity DDI alerts and 91% of outpatient drug-allergy alerts (Weingart et al. Arch Intern Med 2003)

## Are Overrides Appropriate?

- 7,761 drug-allergy alerts in BWH inpatients, Aug-Oct 2002
- Alerts were overridden 80% of the time
- In chart review of 300 overrides, all were clinically justified
- Evidence that we are over-alerting
- Only 6% of alerts were triggered by an exact match between drug ordered and drug in allergy list (lots of potential noise)

- Hsieh et al. JAMIA 2004

## Alerts Triggered by Exact Drug-Allergy Matches are Rare  

**	**_**Ordered drug**_** ****	**
- Hydromorphone (DILAUDID®) (N=18)

**	**_**Drug in allergy list**_
- Codeine (39%)
- Oxycodone (22%)
- Meperidine (22%)
- Morphine (17%)
- Hydromorphone (0%)

**Notes:**
Examples of drugs in patients’ allergy lists that “matched” with ordered drugs to trigger allergy alerts.   Dilaudid, Lasix, and HCTZ are given as examples to illustrate how rare exact matches are.

## Alerts Triggered by Exact Drug-Allergy Matches are Rare

**	**_**Ordered drug**_** ****	**
- Furosemide (LASIX®) (N=18)
- Hydrochlorothiazide (N=4)

**	**_**Drug in allergy list**_
- ‘Sulfa’ (95%)
- Furosemide (5%)
- ‘Sulfa’ (100%)
- Hydrochlorothiazide (0%)

## Allergy Alerting Recommendations

- Need better specificity of alerts to avoid false positives
- E.g. requiring exact matches for certain classes
- Good news: FDB recently removed lasix/HCTZ – sulfa interaction

## Overall Alerting Issues

- Need more studies to maximize effectiveness of alerts/ minimize over-alerting
- Issue of how best to display the messages
- Need to learn from other industries (industrial engineering)

## Drug-Pregnancy Level 1

## Slide 9

## Potential Strategies to Improve Alerting

- Creation of streamlined knowledge bases
- Only essential content
- Balance between sensitivity and specificity
- Tiering of alerts is also a possibility
- Hard stop
- Interruptive
- Non-interruptive
- Minimizing interruptions

## Impact of Reduced Alerting on Override Rates

- Study in the ambulatory setting
- Decision support included
- Duplicate drug
- Drug-disease
- Drug-drug
- Drug-lab
- Drug-pregnancy

- Shah et al. JAMIA 2006

## Knowledge base streamlining

- Expert panel
- Physicians, pharmacists, informaticians
- Reviewed sources
- Vendor knowledge-bases, pre-existing locally created KBs, literature
- Removed certain alerts and tiered the rest

## Alert tiers

- Level 1 – Potentially life-threatening
- E.g., erythromycin - diltiazem -> V-fib
- “Hard stop” – couldn’t proceed
- Level 2 – Potential for serious injury
- Rizatriptan - linezolid -> serotonin syndrome
- Interruptive, required a reason
- Level 3 – Use w/ caution
- Warfarin – levofloxacin -> increased PT
- Noninterruptive

## Slide 14

## Actions w/ level 2 alerts

- Cancel
- Do not proceed with order
- Modify, examples include
- Alter the problem list
- Discontinue the pre-existing drug
- Hold the other medication
- “Accept” defined as “cancel” or “modify”
- “Override” defined as not accept

## Results

- Final knowledge base
- 2% level 1 63% level 2, 35% level 3
- 18,115 alerts
- 12,933 non-interruptive (71%)
- 5,182 interruptive (29%)
- Of 5,182 interruptive alerts
- 3475 (67%) accepted

## Interruptive alerts

| Alert | N | Accepted | Overridden |
| --- | --- | --- | --- |
| Duplicate class | 3,875 | 2,695 (77%) | 910 (23%) |
| Drug-drug | 1078 | 451 (42%) | 627 (58%) |
| Drug-disease | 19 | 10 (53%) | 9 (47%) |
| Drug-lab | 92 | 37 (40%) | 55 (60%) |
| Drug-pregnancy | 118 | 12 (10%) | 106 (90%) |
| Total | 5182 | 3,475 (67%) | 1,707 (33%) |

## Summary of Reduced Alerting Study

- Can reduce alert burden by streamlining and tiering the knowledge base
- Concept of a “non-interruptive” alert may be helpful
- Still need more research on what is optimal level of alerting
- “Are we missing things” is always the worry

## Non-interruptive alerts

| 
Alert | N (% of all alerts in that category) |
| --- | --- |
| Duplicate drug | 0 (0%) |
| Drug-drug | 3547 (77%) |
| Drug-disease | 24 (56%) |
| Drug-lab | 4,444 (98%) |
| Drug-pregnancy | 4,918 (98%) |
| Total | 12,933 (71%) |

## Slide 21

## Impact of Non-Interruptive Alerts

- Drilled down into the drug-lab alerts
- _No difference_ in lab ordering between control and intervention groups
- Is it a user interface issue or are non-interruptive alerts just not effective?
- Cost- benefit issue
- Minimize interruptions so higher risk alerts are accepted more often... yet little benefit seen with non-interruptive alerts

## Impact of Tiering on Inpatient DDI Alerts

- Two academic medical centers
- Same knowledge base
- Site A used 3 tiers
- Site B had all of the alerts as interruptive (Level 2)
- Overall alert acceptance higher at tiered site (29% vs 10%, p<.001)

- Paterno, et al. Unpublished data.

## Tiered Inpatient DDI Acceptance Rates

- Level 1 Acceptance rates
- 100% (hard stop) vs 34% (not a hard stop)
- Level 2 Acceptance rates
- 29% vs 11%
- Likely higher at tiered site since less alert fatigue because fewer interruptive alerts

## Conclusions

- Streamlined knowledge bases and tiered alerting have higher acceptance rates
- Especially for very high risk alerts
- Non-interruptive alerts may have little value
- What is our ideal acceptance rate?? Sensitivity/specificity? Best way to display?
- More work needs to be done to maximize the clinical benefits
- Sharing of streamlined knowledge should be widespread
- No need to reinvent the wheel